diff --git a/AGENTS.md b/AGENTS.md index dbcecf8..c13ddaa 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,92 +1,127 @@ # PROJECT KNOWLEDGE BASE -**Generated:** 2026-05-19 -**Commit:** 28abc5d -**Branch:** master +**Generated:** 2026-06-12T16:08:15Z +**Commit:** 3629c33 +**Branch:** feat/bake-dag-tester ## OVERVIEW -HoneyBiscuitWorkshop ("蜜饼工坊") — Rust CI/CD system with decorator-driven shell pipeline DSL and LLM-powered configuration. 3 independent Cargo crates, no workspace. +HoneyBiscuitWorkshop (蜜饼工坊) — Rust-based CI/CD system inspired by Concourse CI. Pipeline: prebake.yml → bake.sh → finalize.yml. Multi-builder (Docker, Firecracker, bare metal). Plugin system. ## STRUCTURE ``` ./ -├── workshop-baker/ # Main orchestrator (CLI + pipeline stages) -├── workshop-engine/ # Execution engine (extracted from baker) -├── workshop-getterurl/ # URL fetching (go-getter style) -├── docs/ # mdBook documentation (zh-CN) -├── templates/ # Pipeline templates (bake.sh, finalize.yml, prebake.yml) -└── archived/ # Deprecated crates (7 moved here) +├── workshop-baker/ # Main binary + lib — orchestrator, CLI, plugins +├── workshop-engine/ # Core execution engine — process mgmt, package managers +├── workshop-schedule/ # DAG scheduling library (petgraph-based) +├── workshop-getterurl/ # URL fetching with hash verification +├── workshop-baker-params/ # Config/parameter type definitions +├── docs/ # mdBook documentation (Chinese, 3 volumes) +├── templates/ # Pipeline config templates (prebake/bake/finalize) +└── .github/workflows/ # CI (tests workshop-baker + workshop-engine) ``` +**No root Cargo workspace** — crates linked by path deps. Build per-crate. ## WHERE TO LOOK + | Task | Location | Notes | |------|----------|-------| -| Baker (CLI) | `workshop-baker/src/{lib,main}.rs` | Clap CLI + tokio entry | -| Pipeline stages | `workshop-baker/src/{bake,prebake,finalize}/` | Stage orchestration | -| Notifications | `workshop-baker/src/notify/` | Plugin-based event notifications | -|| Engine (execution) | `workshop-engine/src/` | Process mgmt, PM detection | -| URL fetching | `workshop-getterurl/src/` | Git/HTTP/File resource getter | +| CLI definition | `workshop-baker/src/lib.rs` | Clap Cli/Commands/BareCommands | +| Pipeline orchestration | `workshop-baker/src/main.rs` | prebake→bake→finalize→notify flow | +| Decorator parsing | `workshop-baker/src/bake/decorator.rs` | `# @decorator(args)` syntax | +| Build execution | `workshop-baker/src/bake/execute.rs` | Script execution with timeout/retry | +| Docker integration | `workshop-baker/src/prebake/env/container.rs` | Bollard-based | +| Package managers | `workshop-engine/src/pm/` | apt, pacman, dnf, apk | +| User package managers | `workshop-engine/src/upm/` | rustup, uv, go | +| Notification system | `workshop-baker/src/notify/` | Email, webhook, IM | +| Plugin system | `workshop-baker/src/finalize/plugin/` | shell, dylib, rhai, internal | +| Error types | `workshop-baker/src/error.rs` | CliError + HasExitCode | +| Exit codes | `workshop-engine/src/error.rs` | EXITCODE_* constants | +| DAG scheduling | `workshop-schedule/src/dag.rs` | Topological sort, batch picking | +| Pipeline templates | `templates/` | Reference config DSL | + +## ENTRY POINTS + +| Entry | Type | Path | Role | +|-------|------|------|------| +| `workshop-baker` | Rust bin | `workshop-baker/src/main.rs` | Primary CLI — dispatch: default (full pipeline), bare (single stage), daemon (unimplemented) | +| `workshop_baker` | Rust lib | `workshop-baker/src/lib.rs` | Baker library + CLI structs | +| `workshop_engine` | Rust lib | `workshop-engine/src/lib.rs` | Core engine — ExecutionContext, EventSender, Executor | +| `workshop_schedule` | Rust lib | `workshop-schedule/src/lib.rs` | Dag, DagNode, EdgeKind | +| `workshop_baker_params` | Rust lib | `workshop-baker-params/src/lib.rs` | Layered config: defaults → base → courier | +| `workshop_getterurl` | Rust lib | `workshop-getterurl/src/lib.rs` | GetterUrl, fetch() — http/git/file | +| docs | mdBook | `docs/book.toml` | Project documentation site | ## CODE MAP -| Symbol | Type | Location | Role | -|--------|------|----------|------| -| Cli | struct | workshop-baker/src/lib.rs:21 | CLI argument struct | -| ExecutionContext | struct | workshop-engine/src/types.rs:10 | Pipeline context | -| Engine | struct | workshop-engine/src/types.rs:99 | Build executor (placeholder) | -| PrebakeConfig | struct | workshop-baker/src/prebake/config.rs | Prebake YAML schema | -| FinalizeConfig | struct | workshop-baker/src/finalize/config.rs:21 | Finalize YAML schema | -| GetterUrl | struct | workshop-getterurl/src/lib.rs:84 | Parsed resource URL | -| Getter | trait | workshop-getterurl/src/getter.rs:18 | Resource fetch trait | + +### Crate Dependency Graph +``` +workshop-baker (bin) + ├── workshop-engine + ├── workshop-schedule + ├── workshop-baker-params + └── workshop-getterurl +``` + +### workshop-baker Module Map +| Module | Submodules | Role | +|--------|-----------|------| +| `bake` | decorator, execute, parser, schedule, builder, trivial, util, constant, error | Script parsing + execution | +| `prebake` | config, stage (bootstrap, depssystem, depsuser, environment, hook), env (container, firecracker, baremetal, custom), security, event, types, constant, error | Environment setup | +| `finalize` | plugin (shell, dylib, rhai, internal), stage, template, types, config, constant, error, event | Post-build hooks | +| `notify` | handler, queue, rule, template, types, util, error | Async notification dispatch | +| `bare` | — | Standalone single-stage CLI mode | +| `types` | resource, buildstatus | Shared type definitions | +| `utils` | — | Cross-cutting utilities | ## CONVENTIONS -### Rust -- **Edition**: 2024 (requires Rust 1.85+), deprecates `mod.rs` -- **Naming**: `snake_case` files/modules, `PascalCase` types, `SCREAMING_SNAKE_CASE` consts -- **Error handling**: `thiserror` + `anyhow` combo; `HasExitCode` trait for exit codes -- **Async**: `#[tokio::main]` + `tokio` with "full" features -- **Imports**: `std` → `external_crate` → `crate::module` -- **Dual-mode**: CLI commands (prebake/bake/finalize) + daemon mode (unimplemented) - -### Testing -- **Rust**: `#[test]` inline, `#[tokio::test]` async, `tests/` integration -- **Benchmarks**: Criterion (configured in Cargo.toml) - -### CI -- **GitHub Actions**: rust-toolchain (stable), clippy + rustfmt, `cargo test --all-features` -- **Matrix**: workshop-baker, workshop-engine +- **Edition**: Rust 2024 (requires Rust 1.85+) +- **Formatting**: Default `rustfmt` — no `rustfmt.toml` / `.editorconfig` +- **Lint overrides** (workshop-baker only): `dead_code = "allow"`, `unreachable_code = "allow"`, `inherent_to_string = "allow"`, `non_canonical_partial_ord_impl = "allow"` +- **Import order**: `std` → external crates → `crate::` (blank line between groups) +- **Error handling**: `thiserror` for libraries, `anyhow` for binaries. Implement `HasExitCode` for exit code mapping. +- **Type system**: Newtype pattern preferred (e.g., `MemSize(u64)`). Shared types in `types/` module. `types/` must NOT depend on `bake/`, `engine/`, `prebake/`, `finalize/`. +- **Tests**: Inline `#[cfg(test)] mod tests` at bottom of source files. Async: `#[tokio::test]`. CI runs `cargo test --all-features`. +- **Commits**: `type(scope): description` — `feat|fix|docs|test|refactor|style` +- **AI annotation**: `// Code in this module PARTIALLY or FULLY utilized AI Coding Agent` — never delete if present +- **Docs language**: Chinese (zh-CN), mdBook +- **No `no_std`**: Explicitly unsupported ## ANTI-PATTERNS (THIS PROJECT) -1. **Hardcoded user** — "vulcan" username hardcoded throughout prebake -2. **Unsafe libc** — workshop-baker/src/prebake/security.rs has 5+ unsafe blocks -3. **Rust 2024 edition** — May not work with stable toolchains -4. **Empty daemon** — `daemon/` dir is empty, Daemon command is `unimplemented!()` -## UNIQUE STYLES -- **Pipeline DSL**: Shell scripts with `# @decorator` annotations -- **Notification system**: Plugin-based with batching, retry, priority groups -- **LLM cookbook system**: YAML cookbooks for language-specific builds (archived) -- **Chinese docs** — Documentation primarily in zh-CN +- **Do NOT derive production behavior from bare mode** — bare mode is dev-only, ctx.privileged is fake +- **Do NOT use `detect()`/`adopt()` PM methods in production** — PM selection must come from env config +- **`drop_after < DepsUser` is FORBIDDEN** — will break system dependency installation (needs root) +- **Do NOT add magic URL inference** — URL is URL, behavior bound explicitly via `getter::` prefix +- **Do NOT use `@` for version locking in URLs** — it's an auth delimiter; use `?ref=` +- **Do NOT duplicate type definitions across modules** — single source of truth in `types/` +- **Careful with `unimplemented!()`** — Daemon mode, Rhai/Dylib plugins, and Custom builder are stubs +- **Security**: PIPELINE_CONTAINER_PRIVILEGED, PIPELINE_BAREMETAL_ELEVATE, PIPELINE_UNCOVER_SECRET require explicit approval + audit ## COMMANDS + ```bash -# Build -cargo build --release -p workshop-baker +# Build (per crate, no workspace) +cd workshop-baker && cargo build --all-features -# Test -cargo test -p workshop-baker -cargo test -p workshop-engine +# Test (per crate) +cd workshop-baker && cargo test --all-features +cd workshop-engine && cargo test --all-features -# Lint -cargo clippy -- -D warnings -cargo fmt --all +# Lint (CI gate) +cargo fmt -- --check +cargo clippy --all-features -- -D warnings -# Run -cargo run --bin workshop-baker -- --pipeline demo --build-id 1 --prebake prebake.yml --bake bake.sh --finalize finalize.yml +# Docs +cd docs && mdbook build # output in docs/book/ +cd docs && mdbook serve # local preview at localhost:3000 ``` ## NOTES -- **Docker required** for integration tests -- **No root workspace** — each crate independent, build separately -- **7 archived crates** — workshop-agent, workshop-cert, workshop-deviceid, workshop-helper-mac, workshop-llm-detector, workshop-pipeline, workshop-vault -- **Python tests removed** — pytest.ini exists but test files no longer in workspace + +- **No workspace Cargo.toml** at root — each crate has its own `Cargo.lock`. Build individually. +- **CI only tests** `workshop-baker` and `workshop-engine`; smaller crates tested transitively. +- **`quicktest.sh`** in workshop-baker/ is a dev smoke-test script (Docker-based, not CI). +- **Git submodules**: `rust-tongsuo`, `RustyVault` (crypto — not yet integrated into build). +- **`.secret/`** contains real credentials — never commit, never read into context. +- See `docs/src/zh-CN/vol3_dev/` for full developer documentation (architecture, design decisions, code style). diff --git a/docs/AGENTS.md b/docs/AGENTS.md new file mode 100644 index 0000000..971a924 --- /dev/null +++ b/docs/AGENTS.md @@ -0,0 +1,48 @@ +# DOCS KNOWLEDGE BASE + +## OVERVIEW +mdBook project in zh-CN. Authoritative source for project conventions, coding style, and architecture docs. Build output at `docs/book/` (gitignored). + +## STRUCTURE +``` +docs/ +├── book.toml # mdBook config (title, language=zh-CN) +├── src/SUMMARY.md # Sidebar nav — source of truth for page listing +└── src/zh-CN/ + ├── index.md # Landing + ├── vol1_user/ # End-user documentation + ├── vol2_admin/ # Admin reference: decorator cheatsheet, deployment, + │ # env vars, exit codes, plugin ref, security, troubleshooting + └── vol3_dev/ # Developer docs: architecture, code style, commit + # convention, testing, error handling, type system, + # design decisions, PR process +``` + +## WHERE TO LOOK + +| Task | Location | Notes | +|------|----------|-------| +| Sidebar/nav editing | `src/SUMMARY.md` | Add entry here to surface a new page | +| Coding conventions | `vol3_dev/code_style.md` | Naming, imports, formatting | +| Commit rules | `vol3_dev/commit_convention.md` | feat/fix/docs/test/refactor/style | +| Error handling | `vol3_dev/error_handling_patterns.md` | thiserror+anyhow+HasExitCode | +| Type system | `vol3_dev/type_system_guide.md` | Newtypes, types/ module isolation | +| Testing strategy | `vol3_dev/testing_strategy.md` | 3 tiers: unit, integration, system | +| Anti-patterns | `vol3_dev/design_decisions.md` | Documented design rejections | +| Architecture | `vol3_dev/architecture.md` | Crate dependency graph, module map | + +## CONVENTIONS + +- **Language**: All prose in zh-CN. Code blocks in Rust (default) or bash. +- **Adding a page**: Create `.md` under `volN_*/`, then add to `SUMMARY.md`. +- **Formatting**: Standard Markdown. mdBook preprocessors handle TOC and links. +- **Cross-references**: Relative paths from source root (e.g., `../vol3_dev/code_style.md`). +- **Code blocks**: Fenced with language tag. No line numbers in docs. +- **File naming**: `snake_case.md`. + +## COMMANDS + +```bash +mdbook build # static site → docs/book/ +mdbook serve # live preview at localhost:3000 +``` diff --git a/workshop-baker-params/src/bake.rs b/workshop-baker-params/src/bake.rs index 7bfc61b..5aaa90f 100644 --- a/workshop-baker-params/src/bake.rs +++ b/workshop-baker-params/src/bake.rs @@ -10,6 +10,8 @@ pub struct BakeParams { #[serde(default)] pub health_period: ParamVal, /// Health probe per-attempt timeout in seconds (default 10). #[serde(default)] pub health_timeout: ParamVal, + /// Max health probe retries before giving up (default 12). + #[serde(default)] pub health_max_retries: ParamVal, } impl MergeParams for BakeParams { @@ -18,12 +20,14 @@ impl MergeParams for BakeParams { workspace: ParamVal::new("/workspace".into()), health_period: ParamVal::new(5), health_timeout: ParamVal::new(10), + health_max_retries: ParamVal::new(12), } } fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) { apply_field!(self.workspace, i.workspace, w, ov, p, "workspace"); apply_field!(self.health_period, i.health_period, w, ov, p, "health_period"); apply_field!(self.health_timeout, i.health_timeout, w, ov, p, "health_timeout"); + apply_field!(self.health_max_retries, i.health_max_retries, w, ov, p, "health_max_retries"); } } @@ -34,6 +38,7 @@ pub struct BakeSettings { pub workspace: String, pub health_period: u64, pub health_timeout: u64, + pub health_max_retries: u32, } impl Default for BakeSettings { fn default() -> Self { BakeParams::defaults().into() } } impl From<&BakeParams> for BakeSettings { @@ -42,6 +47,7 @@ impl From<&BakeParams> for BakeSettings { workspace: p.workspace.value.clone(), health_period: p.health_period.value, health_timeout: p.health_timeout.value, + health_max_retries: p.health_max_retries.value, } } } diff --git a/workshop-baker-params/src/notification.rs b/workshop-baker-params/src/notification.rs index e4a6e8f..07f0487 100644 --- a/workshop-baker-params/src/notification.rs +++ b/workshop-baker-params/src/notification.rs @@ -23,7 +23,7 @@ impl MergeParams for NotificationParams { retry_backoff: ParamVal::new(2.0), retry_delay_seconds: ParamVal::new(5.0), max_retry_delay_seconds: ParamVal::new(300.0), life_impact_factor: ParamVal::new(0.0), - batch_period: ParamVal::new(30), batch_watermark: ParamVal::new(5), + batch_period: ParamVal::new(5), batch_watermark: ParamVal::new(5), template_ref_depth: ParamVal::new(10), } } diff --git a/workshop-baker/AGENTS.md b/workshop-baker/AGENTS.md new file mode 100644 index 0000000..469fa89 --- /dev/null +++ b/workshop-baker/AGENTS.md @@ -0,0 +1,80 @@ +# WORKSHOP-BAKER CRATE KNOWLEDGE BASE + +## OVERVIEW +Primary crate — the only crate with a `[[bin]]` target. Contains the `workshop-baker` binary and `workshop_baker` library. Orchestrates the full CI/CD pipeline: prebake (env setup) → bake (script execution) → finalize (plugins, cleanup) → drain notification queue. ~69 .rs source files. + +## STRUCTURE +``` +src/ +├── main.rs # Entry: default(full pipeline) | bare(dev-only) | daemon(unimplemented) +├── lib.rs # Clap Cli/Commands/BareCommands — CLI structs in lib, not bin +├── error.rs # CliError enum (wraps stage errors, delegates exit_code) +├── bare.rs # Dev-only single-stage execution, ctx.privileged always false +├── utils.rs # Cross-cutting helpers +├── bake/ # Script parsing + DAG scheduling + execution +│ ├── decorator.rs # @decorator(args) parsing, 81 tests — heaviest coverage in crate +│ ├── parser.rs # Step/function extraction +│ ├── schedule.rs # DAG build → topological sort +│ ├── execute.rs # Per-step timeout/retry +│ ├── builder.rs # Script assembly +│ ├── trivial.rs # Trivial step detection (skip in dry-run) +│ ├── constant.rs / error.rs / util.rs +├── prebake/ # Environment provisioning +│ ├── config.rs # PrebakeConfig deserialization +│ ├── stage.rs + stage/{bootstrap,depssystem,depsuser,environment,hook}.rs +│ ├── env.rs + env/{container,firecracker,baremetal,custom}.rs # Builder dispatch +│ ├── security.rs # Privilege/sandbox checks +│ ├── event.rs # Event sender/receiver channel +│ ├── types.rs + types/{memsize,cache,builderconfig,architecture}.rs +│ ├── constant.rs / error.rs +├── finalize/ # Post-build hooks + plugins +│ ├── config.rs # FinalizeConfig +│ ├── stage.rs + stage/hook.rs +│ ├── template.rs # Output templating (minijinja) +│ ├── plugin.rs + plugin/{shell,dylib,rhai,internal}.rs +│ │ └── plugin/internal/{dummy,mail,satori,webhook,insitenotify}.rs +│ ├── types.rs + types/compression.rs +│ ├── constant.rs / error.rs / event.rs +├── notify/ # Async notification dispatch +│ ├── queue.rs # NotificationQueue — drain on pipeline complete +│ ├── handler.rs / rule.rs / template.rs / types.rs / util.rs / error.rs +└── types/ # Shared types — ZERO deps on stage modules + ├── resource.rs # ResourceRegistry (populated pre-pipeline, consumed read-only) + └── buildstatus.rs +``` + +## WHERE TO LOOK + +| Task | Location | Notes | +|------|----------|-------| +| CLI args + subcommands | `src/lib.rs` | Cli, Commands(Bare/Daemon), BareCommands | +| Pipeline orchestration | `src/main.rs` | worker_main(): prebake→bake→finalize→queue.drain() | +| Decorator syntax | `src/bake/decorator.rs` | `# @decorator` or `# @decorator(args)`, line-by-line regex | +| Build execution | `src/bake/execute.rs` | timeout, retry, exit code capture | +| DAG scheduling | `src/bake/schedule.rs` | Uses workshop-schedule crate | +| Docker builder | `src/prebake/env/container.rs` | Bollard-based, buildkit support | +| Firecracker / Bare metal | `.../firecracker.rs`, `.../baremetal.rs` | MicroVM; +needs PIPELINE_BAREMETAL_ELEVATE | +| Plugin dispatch | `src/finalize/plugin.rs` | Dispatches shell / dylib(stub) / rhai(stub) / internal | +| Notification queue | `src/notify/queue.rs` | Spawned early, drained after pipeline, retry support | +| Error type | `src/error.rs` | CliError wraps PrebakeError/BakeError/FinalizeError/anyhow | +| Dev bare mode | `src/bare.rs` | ctx.privileged=false always; do not extrapolate to production | + +## DEPENDENCIES +**Internal:** workshop-engine, workshop-schedule, workshop-getterurl, workshop-baker-params. **External:** bollard (Docker), clap (CLI), axum 0.7 (HTTP), lettre 0.11 (email), minijinja (templates), cgroups-rs (limits), privdrop. + +## CONVENTIONS (CRATE-SPECIFIC) +- **Lint overrides** (this crate only): `dead_code=allow`, `unreachable_code=allow`, `inherent_to_string=allow`, `non_canonical_partial_ord_impl=allow` +- **CLI structs in lib.rs, not main.rs** — enables integration tests to import and construct Cli directly. +- **Pipeline stages return `anyhow::Result<()>`** — stages propagate with `?`; only main.rs converts to CliError. +- **ResourceRegistry pattern** — populated before pipeline stages, consumed read-only by stages. New resource types go in `types/resource.rs`. +- **Notification queue fire-and-forget + drain** — notify tasks spawned early, send during pipeline, drain on completion. `NotificationQueue::drain()` must be called. +- **Trivial step detection** — steps with no external deps can be marked trivial (`trivial.rs`), skipped in dry-run. +## ANTI-PATTERNS (THIS CRATE) +- **Do NOT derive production behavior from bare mode** — ctx.privileged is hardcoded false. +- **Do NOT add CLI args bypassing prebake→bake→finalize ordering** — the pipeline is linear. Parallelism lives in the DAG scheduler within bake. +- **Do NOT import types/ from bake/, prebake/, or finalize/** — `types/` must have zero crate-internal deps. +- **Do NOT add lint allows without justification** — the four existing overrides are intentional. +- **Do NOT spawn blocking work on tokio runtime** — use `spawn_blocking` for CPU-heavy tasks. +- **Decorator regex is line-by-line** — multi-line decorators need a parser rewrite, not a regex hack. +- **Careful with `unimplemented!()`** — Daemon, Rhai/Dylib plugins, Custom builder are stubs. Prefer error over crash. diff --git a/workshop-baker/Cargo.lock b/workshop-baker/Cargo.lock index 36a40c4..1336102 100644 --- a/workshop-baker/Cargo.lock +++ b/workshop-baker/Cargo.lock @@ -1118,6 +1118,12 @@ version = "0.1.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "645cbb3a84e60b7531617d5ae4e57f7e27308f6445f5abf653209ea76dec8dff" +[[package]] +name = "fixedbitset" +version = "0.5.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d674e81391d1e1ab681a28d99df07927c6d4aa5b027d7da16ba32d1d21ecd99" + [[package]] name = "fnv" version = "1.0.7" @@ -2404,6 +2410,16 @@ dependencies = [ "sha2 0.10.9", ] +[[package]] +name = "petgraph" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3672b37090dbd86368a4145bc067582552b29c27377cad4e0a306c97f9bd7772" +dependencies = [ + "fixedbitset", + "indexmap 2.12.1", +] + [[package]] name = "pin-project" version = "1.1.10" @@ -3625,12 +3641,6 @@ dependencies = [ "tonic", ] -[[package]] -name = "topological-sort" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ea68304e134ecd095ac6c3574494fc62b909f416c4fca77e440530221e549d3d" - [[package]] name = "tower" version = "0.5.2" @@ -4282,12 +4292,12 @@ dependencies = [ "tempfile", "thiserror 2.0.18", "tokio", - "topological-sort", "uuid", "which", "workshop-baker-params", "workshop-engine", "workshop-getterurl", + "workshop-schedule", ] [[package]] @@ -4330,6 +4340,14 @@ dependencies = [ "url", ] +[[package]] +name = "workshop-schedule" +version = "0.1.0" +dependencies = [ + "petgraph", + "thiserror 2.0.18", +] + [[package]] name = "writeable" version = "0.6.2" diff --git a/workshop-baker/Cargo.toml b/workshop-baker/Cargo.toml index c223cf6..3f610c3 100644 --- a/workshop-baker/Cargo.toml +++ b/workshop-baker/Cargo.toml @@ -41,7 +41,7 @@ tar = "0.4.44" tempfile = "3.24.0" thiserror = "2.0.17" tokio = { version = "1.48.0", features = ["full"] } -topological-sort = "0.2.2" +workshop-schedule = { path = "../workshop-schedule" } async-trait = "0.1.87" uuid = { version = "1.19.0", features = ["v4"] } libc = "0.2" diff --git a/workshop-baker/pytest.ini b/workshop-baker/pytest.ini deleted file mode 100644 index d25c5bc..0000000 --- a/workshop-baker/pytest.ini +++ /dev/null @@ -1,11 +0,0 @@ -[pytest] -testpaths = tests/integration -python_files = test_*.py -python_classes = Test* -python_functions = test_* -addopts = -v --tb=short -markers = - integration: marks tests as integration tests (deselect with '-m "not integration"') - slow: marks tests as slow running -filterwarnings = - ignore::DeprecationWarning diff --git a/workshop-baker/src/AGENTS.md b/workshop-baker/src/AGENTS.md deleted file mode 100644 index a41f0a5..0000000 --- a/workshop-baker/src/AGENTS.md +++ /dev/null @@ -1,57 +0,0 @@ -# workshop-baker/src - -**Generated:** 2026-05-19 -**Commit:** 28abc5d - -CLI tool and runtime for executing CI/CD pipelines. Orchestrates prebake → bake → finalize stages with notification dispatch. - -## STRUCTURE -``` -src/ -├── lib.rs # Module declarations + Cli struct + Commands enum -├── main.rs # tokio::main entry point -├── bare.rs # Bare mode (run individual stages) -├── error.rs # CliError (wraps stage errors) -├── bake.rs # Bake orchestration (script → execution) -├── bake/ # Script parsing, decorator DSL, scheduling -├── prebake.rs # Prebake orchestration (env setup) -├── prebake/ # Config, stages, security, env providers -├── finalize.rs # Finalize orchestration (post-build) -├── finalize/ # Config, plugins, hooks, templates -├── notify.rs # Notification system (batching, retry) -├── notify/ # Types, templates, interactive, queue -├── daemon/ # Empty (Daemon command = unimplemented!) -├── monitor/ # Resource usage monitoring (cgroups) -├── types.rs # Module re-exports -├── types/ # buildstatus, resource types -├── utils.rs # Module root (fetch submodule) -└── utils/ # fetch.rs -``` - -## WHERE TO LOOK -| Task | Location | -|------|----------| -| CLI args | `lib.rs:21` - `Cli` struct | -| Entry point | `main.rs:91` - `#[tokio::main]` | -| Pipeline orchestration | `main.rs:68-74` - prebake→bake→finalize chain | -| Bare mode (single stage) | `bare.rs:14` - `run_bare()` | -| Script parsing | `bake/parser.rs:56` - `parse_script()` | -| Decorator DSL | `bake/decorator.rs` - `Decorator` enum | -| Prebake stages | `prebake/stage/` - Bootstrap→DepsSystem→DepsUser→Hooks→Ready | -| Plugin system | `finalize/plugin.rs` + `finalize/plugin/` | -| Notification dispatch | `notify.rs:66` - `notify()` async loop | -| Notification types | `notify/types.rs` - `NotificationRule`, `ParsedTrigger` | - -## CONVENTIONS -- **ExecutionContext**: Passed through all stages, carries task_id, username, env_vars, timeout -- **PrebakeStage**: Ordered enum Bootstrap→EarlyHook→DepsSystem→DepsUser→LateHook→Ready -- **Decorator**: `# @name(args)` comments above shell function declarations -- **Notification**: Plugin-based dispatch with batching, exponential backoff, priority groups -- **Engine**: Delegated to `workshop-engine` crate (formerly in-tree) -- **Dual-mode**: CLI commands (standalone) vs Daemon mode (unimplemented) - -## ANTI-PATTERNS -1. **Empty daemon/** directory — Daemon command is `unimplemented!()` -2. **Dead code allowed** — `lints.rust.dead_code = "allow"` in Cargo.toml -3. **Engine was extracted** — workshop-engine is its own crate now, src/engine/ removed -4. **Python test infra exists but files removed** — pytest.ini remains, test dir is empty diff --git a/workshop-baker/src/bake.rs b/workshop-baker/src/bake.rs index 9eb3b16..44c6818 100644 --- a/workshop-baker/src/bake.rs +++ b/workshop-baker/src/bake.rs @@ -2,7 +2,7 @@ use crate::bake::builder::RenderMode; use crate::bake::constant::DEFAULT_WORKSPACE; use crate::bake::decorator::Decorator; use crate::bake::error::BakeError; -use crate::bake::util::{is_parallel, resolve_mode, FuncMode}; +use crate::bake::util::{resolve_mode, FuncMode}; use crate::cli::Cli; use crate::prebake; use crate::prebake::PrebakeConfig; @@ -30,11 +30,16 @@ pub async fn bake( ctx: &mut ExecutionContext, event_tx: EventSender, ) -> Result<(), BakeError> { - let script_content = std::fs::read_to_string(script_path)?; - let prebake_content = std::fs::read_to_string(prebake_path)?; + log::debug!("Reading script: {:?}", script_path); + let script_content = std::fs::read_to_string(script_path).map_err(|e| { + BakeError::IoError { path: script_path.display().to_string(), source: e } + })?; + log::debug!("Reading prebake.yml: {:?}", prebake_path); + let prebake_content = std::fs::read_to_string(prebake_path).map_err(|e| { + BakeError::IoError { path: prebake_path.display().to_string(), source: e } + })?; let prebake = prebake::parse(prebake_content.as_str()).map_err(|e| { log::error!("Failed to parse prebake.yml: {}", e); - log::error!("This is unexpected!"); BakeError::YamlParseError(e) })?; let workspace = get_workspace(&prebake); @@ -58,6 +63,7 @@ pub async fn bake( } if !has_pipeline { + log::trace!("Running trivial script"); trivial::run_trivial( &script_content, bake_base_path, @@ -69,6 +75,7 @@ pub async fn bake( ) .await?; } else { + log::trace!("Running pipeline script"); let remaining = functions.remaining_code; let (mut dag, func_map) = schedule::build_dag(functions.pipeline_functions)?; @@ -76,10 +83,6 @@ pub async fn bake( let mut pipe_sources: HashSet = HashSet::new(); for func in func_map.values() { decorator::check_conflicts(func)?; - - // Some simple optimization for @pipe - // Collect only output of specified function - // and exclude self-reference because it's handled by loop for decorator in &func.decorators { if let Decorator::Pipe(names) = decorator { for source in names { @@ -95,29 +98,31 @@ pub async fn bake( let mut async_handles: Vec> = Vec::new(); while !dag.is_empty() { - let ready: Vec = dag.peek_all().into_iter().cloned().collect(); - if ready.is_empty() { - let remaining_nodes: Vec = dag.clone().into_iter().collect(); - return Err(BakeError::CircularDependency(remaining_nodes)); - } - + log::debug!("[BAKE_SCHED] ready={} seq={} wild={} groups={:?}", + dag.ready_count(), + dag.ready_sequential().len(), + dag.ready_wildcards().len(), + dag.ready_groups(), + ); // Rule 1: sequential always goes first - if let Some(seq_name) = ready.iter().find(|n| !is_parallel(&func_map[n.as_str()])).cloned() { - let func = &func_map[seq_name.as_str()]; + let seq_candidates = dag.ready_sequential(); + if let Some(seq_name) = seq_candidates.first() { + let seq_name = seq_name.to_string(); + log::info!("[BAKE_SCHED] seq {}", seq_name); + let func = &func_map[&seq_name]; let mode = resolve_mode(func); match mode { FuncMode::Daemon => { - dag.pop(); execute::execute_daemon( &engine, func, ctx, &event_tx, &workspace, &remaining, bake_base_path, Duration::from_secs(bake_settings.health_period), Duration::from_secs(bake_settings.health_timeout), + bake_settings.health_max_retries, ).await?; } FuncMode::Async => { - dag.pop(); let handle = execute::execute_async( func, ctx, &event_tx, &workspace, &remaining, bake_base_path, @@ -125,35 +130,74 @@ pub async fn bake( async_handles.push(handle); } FuncMode::Normal => { - let stdout = execute::execute_normal( + let result = execute::execute_normal( &engine, func, ctx, &event_tx, &workspace, &remaining, bake_base_path, &stdout_cache, - ).await?; - if pipe_sources.contains(&seq_name) { - stdout_cache.insert(seq_name.clone(), stdout); + ).await; + match result { + Ok(stdout) => { + if pipe_sources.contains(&seq_name) { + stdout_cache.insert(seq_name.clone(), stdout); + } + } + Err(e) if dag.is_fallible(&seq_name) => { + log::warn!("'{}' failed (fallible): {}", seq_name, e); + } + Err(e) => return Err(e), } - dag.pop(); } } + // Pop AFTER execution (Daemon/Async pop at start) + if !matches!(mode, FuncMode::Daemon | FuncMode::Async) { + dag.pop(&seq_name).map_err(|e| BakeError::IncompatibleDecorators { + function: seq_name.clone(), + reason: e.to_string(), + })?; + } else { + dag.pop(&seq_name).map_err(|e| BakeError::IncompatibleDecorators { + function: seq_name.clone(), + reason: e.to_string(), + })?; + } continue; } - // All remaining are parallel — group and pick a batch - let (star_nodes, groups) = schedule::partition_by_group(&ready, &func_map); - if let Some(batch) = schedule::pick_batch(&star_nodes, &groups) { + // All remaining are parallel — pick a batch + if let Some(batch) = dag.pick_batch() { + log::info!("[BAKE_SCHED] par {:?}", batch); let results = execute::fire_parallel_batch( &batch, &engine, &func_map, ctx, &event_tx, &workspace, &remaining, bake_base_path, &stdout_cache, ).await?; for (name, stdout) in results { if pipe_sources.contains(&name) { - stdout_cache.insert(name, stdout); + stdout_cache.insert(name.clone(), stdout); + } + if let Err(e) = dag.pop(&name) { + log::warn!("Failed to pop '{}' from DAG: {}", name, e); } - dag.pop(); } + } else if dag.is_stalled() { + // Deadlock: named parallel groups exist but can't complete + let stalled = dag.stalled_groups(); + let remaining = dag.remaining(); + return Err(BakeError::IncompatibleDecorators { + function: remaining.join(", "), + reason: format!( + "parallel group(s) {:?} can never complete — blocked by @after chain", + stalled + ), + }); + } else { + // Ready set is empty — cycle + let remaining = dag.remaining(); + if !remaining.is_empty() { + return Err(BakeError::CircularDependency( + remaining.into_iter().map(|s| s.to_string()).collect(), + )); + } + break; } - - // No complete group and no sequential — nothing fireable } // Wait for all @async tasks to finish diff --git a/workshop-baker/src/bake/AGENTS.md b/workshop-baker/src/bake/AGENTS.md deleted file mode 100644 index 831a7be..0000000 --- a/workshop-baker/src/bake/AGENTS.md +++ /dev/null @@ -1,48 +0,0 @@ -# workshop-baker/src/bake - -**Generated:** 2026-04-22 -**Commit:** 7b3b71a - -Script parsing and build orchestration with decorator DSL. - -## STRUCTURE - -``` -bake/ -├── parser.rs # @decorator-aware shell script parser -├── decorator.rs # Decorator enum + 100+ unit tests -├── builder.rs # Script template assembly -├── schedule.rs # Function ordering via @after dependencies -├── error.rs # BakeError enum -└── constant.rs # DEFAULT_WORKSPACE, TRIVIAL_SCRIPT_NAME -``` - -## WHERE TO LOOK - -| Task | Location | -|------|----------| - Parse script | `parser.rs:56` - `parse_script()` | -| Decorator types | `decorator.rs` - `Decorator` enum variants | -| Build script | `builder.rs` - template builder | -| Schedule deps | `schedule.rs` - @after resolution | - -## DECORATORS - -| Decorator | Purpose | -|-----------|---------| -| `@pipeline` | Marks main build function | -| `@timeout(N)` | Execution timeout seconds | -| `@retry(N, M)` | Retry N times, M sec delay | -| `@parallel` | Run concurrently | -| `@fallible` | Failure doesn't fail pipeline | -| `@after(func)` | Dependency ordering | -| `@export` | Export vars to env | -| `@loop(N)` | Repeat N times | -| `@if(COND)` | Conditional execution | - -## CONVENTIONS - -- **Decorator Syntax**: `# @name(args)` comment above function -- **Parsing**: Shell grammar with decorator annotation extraction -- **Scheduling**: DAG resolution for @after dependencies -- **Tests**: `decorator.rs:175-829` has 100+ inline unit tests diff --git a/workshop-baker/src/bake/builder.rs b/workshop-baker/src/bake/builder.rs index 3f367ad..37c4f7d 100644 --- a/workshop-baker/src/bake/builder.rs +++ b/workshop-baker/src/bake/builder.rs @@ -43,7 +43,6 @@ pub fn build_script( function: &Function, bake_base: &Path, target_dir: &Path, - temp_path: Option, render_mode: RenderMode, ) -> Result { let name = &function.name; @@ -55,24 +54,27 @@ pub fn build_script( None } }); - let temp_path = temp_path.unwrap_or_else(|| PathBuf::from("/tmp")); + // let temp_path = temp_path.unwrap_or_else(|| PathBuf::from("/tmp")); + let temp_path = target_dir; let filename = format!("bakefn_{}.sh", name); let target = target_dir.join(filename); let template_content = match render_mode { RenderMode::Off => TRIVIAL_TEMPLATE.to_string(), RenderMode::Minimal | RenderMode::Full => { - std::fs::read_to_string(bake_base).map_err(|e| BakeError::ScriptGenerationFailed { - script: bake_base.display().to_string(), - reason: e.to_string(), + log::debug!("Reading bake_base: {:?}", bake_base); + std::fs::read_to_string(bake_base).map_err(|e| BakeError::IoError { + path: bake_base.display().to_string(), + source: e, })? } }; let condition_code = match condition { Some(content) => { + // with some bash workaround format!( - "if [ ! {} ]; then\n exit {}\nfi\n", + "if [[ ! {} ]]; then\n exit {}\nfi\n", content, PIPELINE_SKIP_ERRORCODE ) } @@ -99,6 +101,7 @@ pub fn build_script( let mut temp_file = NamedTempFile::new_in(&temp_path).map_err(|e| BakeError::ScriptGenerationFailed { script: temp_path.display().to_string(), + action: "creating temp file".to_string(), reason: e.to_string(), })?; @@ -107,17 +110,20 @@ pub fn build_script( .set_permissions(Permissions::from_mode(0o700)) .map_err(|e| BakeError::ScriptGenerationFailed { script: temp_path.display().to_string(), + action: "setting permissions".to_string(), reason: e.to_string(), })?; std::fs::write(&temp_file, rendered).map_err(|e| BakeError::ScriptGenerationFailed { script: temp_path.display().to_string(), + action: "writing to temp file".to_string(), reason: e.to_string(), })?; temp_file .persist(&target) .map_err(|e| BakeError::ScriptGenerationFailed { script: target.display().to_string(), + action: "persisting temp file".to_string(), reason: e.error.to_string(), })?; Ok(target) @@ -140,7 +146,7 @@ mod tests { body: "echo 'hello world'".to_string(), }; let result = build_script( - &function, &bake_base, temp_dir.path(), None, RenderMode::Full, + &function, &bake_base, temp_dir.path(), RenderMode::Full, ); assert!(result.is_ok()); @@ -162,7 +168,6 @@ mod tests { &function, Path::new("/nonexistent"), temp_dir.path(), - None, RenderMode::Off, ); diff --git a/workshop-baker/src/bake/constant.rs b/workshop-baker/src/bake/constant.rs index 316dff8..6fca48b 100644 --- a/workshop-baker/src/bake/constant.rs +++ b/workshop-baker/src/bake/constant.rs @@ -1,4 +1,4 @@ pub const DEFAULT_WORKSPACE: &str = "/workspace"; pub const BAKE_SCRIPT_PATH: &str = "bake.sh"; pub const TRIVIAL_SCRIPT_NAME: &str = "BAKE_TRIVIAL"; -pub const PIPELINE_SKIP_ERRORCODE: u8 = 233; +pub const PIPELINE_SKIP_ERRORCODE: i32 = 233; diff --git a/workshop-baker/src/bake/error.rs b/workshop-baker/src/bake/error.rs index 719be35..8e2a209 100644 --- a/workshop-baker/src/bake/error.rs +++ b/workshop-baker/src/bake/error.rs @@ -7,8 +7,8 @@ use workshop_engine::{ #[derive(Error, Debug)] pub enum BakeError { - #[error("Failed to generate staged script {script}: {reason}")] - ScriptGenerationFailed { script: String, reason: String }, + #[error("Failed to generate staged script {script} when {action}: {reason}")] + ScriptGenerationFailed { script: String, action: String, reason: String }, #[error("Template error: {0}")] TemplateError(#[from] minijinja::Error), @@ -19,8 +19,8 @@ pub enum BakeError { #[error("Unknown @after of function {0}: {1}")] UnknownDependency(String, String), - #[error("IO Error: {0}")] - IoError(#[from] std::io::Error), + #[error("IO error on {path}: {source}")] + IoError { path: String, source: std::io::Error }, #[error("YAML parse error: {0}")] YamlParseError(#[from] serde_yaml::Error), @@ -49,7 +49,7 @@ impl HasExitCode for BakeError { BakeError::TemplateError(_) => EXITCODE_PARSE_ERROR, BakeError::CircularDependency(_) => EXITCODE_PARSE_ERROR, BakeError::UnknownDependency(..) => EXITCODE_PARSE_ERROR, - BakeError::IoError(_) => EXITCODE_IO_ERROR, + BakeError::IoError { .. } => EXITCODE_IO_ERROR, BakeError::YamlParseError(_) => EXITCODE_PARSE_ERROR, BakeError::DecoratorParseError { .. } => EXITCODE_PARSE_ERROR, BakeError::ScriptExecutionError(e) => e.exit_code(), diff --git a/workshop-baker/src/bake/execute.rs b/workshop-baker/src/bake/execute.rs index cdf9f42..c2869a6 100644 --- a/workshop-baker/src/bake/execute.rs +++ b/workshop-baker/src/bake/execute.rs @@ -1,4 +1,5 @@ use crate::bake::builder::{self, RenderMode}; +use crate::bake::constant::PIPELINE_SKIP_ERRORCODE; use crate::bake::decorator::{self, Decorator}; use crate::bake::error::BakeError; use crate::bake::parser::Function; @@ -31,7 +32,7 @@ pub async fn execute_one( let mut modified = func.clone(); modified.body = remaining.to_string() + &func.body; let script = builder::build_script( - &modified, bake_base, workspace, None, RenderMode::Full, + &modified, bake_base, workspace, RenderMode::Full, )?; execute_with_combos(engine, &script, ctx, event_tx, &combos).await @@ -60,11 +61,17 @@ fn build_combinator_chain<'a>( ) -> Pin> + Send + 'a>> { if idx >= combos.len() { return Box::pin(async move { - let result = engine - .execute_script(script, ctx, event_tx) - .await - .map_err(|e| BakeError::ScriptExecutionError(e))?; - Ok(result.stdout) + let result = engine.execute_script(script, ctx, event_tx).await; + match result { + Ok(r) => Ok(r.stdout), + Err(ExecutionError::ExecutionFailed { exit_code, .. }) + if exit_code == PIPELINE_SKIP_ERRORCODE as i32 => + { + log::debug!("Function skipped (@if condition)"); + Ok(String::new()) + } + Err(e) => Err(BakeError::ScriptExecutionError(e)), + } }); } @@ -187,9 +194,10 @@ pub async fn fire_parallel_batch( let mut results: Vec<(String, String)> = Vec::with_capacity(handles.len()); for handle in handles { let (name, result) = handle.await.map_err(|_| { - BakeError::ScriptExecutionError(ExecutionError::ExecutionFailed( - "parallel task panicked".to_string(), - )) + BakeError::ScriptExecutionError(ExecutionError::ExecutionFailed { + task_id: "parallel".to_string(), + exit_code: -1, + }) })?; match result { Ok(stdout) => results.push((name, stdout)), @@ -264,13 +272,14 @@ pub async fn execute_daemon( bake_base: &Path, health_period: Duration, health_timeout: Duration, + health_max_retries: u32, ) -> Result<(), BakeError> { let fallible = is_fallible(func); let mut modified = func.clone(); modified.body = remaining.to_string() + &func.body; let script = builder::build_script( - &modified, bake_base, workspace, None, RenderMode::Full, + &modified, bake_base, workspace, RenderMode::Full, )?; // Spawn without waiting @@ -292,18 +301,20 @@ pub async fn execute_daemon( log::warn!("Failed to spawn daemon '{}' (fallible): {}", func.name, e); return Ok(()); } - Err(e) => { - return Err(BakeError::ScriptExecutionError(ExecutionError::ExecutionFailed( - format!("Failed to spawn daemon '{}': {}", func.name, e), - ))); + Err(_) => { + return Err(BakeError::ScriptExecutionError(ExecutionError::ExecutionFailed { + task_id: func.name.clone(), + exit_code: -1, + })); } }; - // Periodic health probe — block until healthy or fatal error + // Periodic health probe — block until healthy, retry exhausted, or fatal if let Some(health_cmd) = func.decorators.iter().find_map(|d| match d { Decorator::Health(cmd) => Some(cmd.as_str()), _ => None, }) { + let mut retries: u32 = 0; loop { let probe_result = tokio::time::timeout( health_timeout, @@ -312,13 +323,18 @@ pub async fn execute_daemon( .await; match probe_result { - Ok(Ok(())) => break, - _ if fallible => { + Ok(Ok(())) => break, // healthy — done + _ if retries + 1 < health_max_retries => { + retries += 1; log::warn!( - "Health probe for '{}' failed, retrying in {:?}...", - func.name, health_period + "Health probe for '{}' failed ({}/{}), retrying in {:?}...", + func.name, retries, health_max_retries, health_period, ); } + _ if fallible => { + log::warn!("Health probe for '{}' exhausted (fallible), skipping", func.name); + break; // exhausted but fallible → skip health check + } Ok(Err(e)) => return Err(e), Err(_) => { return Err(BakeError::HealthProbeFailed { diff --git a/workshop-baker/src/bake/parser.rs b/workshop-baker/src/bake/parser.rs index 36c7ca8..5fae56c 100644 --- a/workshop-baker/src/bake/parser.rs +++ b/workshop-baker/src/bake/parser.rs @@ -224,6 +224,7 @@ pub fn parse_script(text: &str) -> Result { { return Err(BakeError::ScriptGenerationFailed { script: "bake.sh".to_string(), + action: "parsing function body".to_string(), reason: format!( "Unexpected EOF while parsing function '{}'. Unmatched braces (remaining: {})?", name, brace_depth @@ -234,6 +235,7 @@ pub fn parse_script(text: &str) -> Result { let last_decorator = pending_decorators.last().unwrap(); return Err(BakeError::ScriptGenerationFailed { script: "bake.sh".to_string(), + action: "parsing decorators".to_string(), reason: format!( "Dangling decorator {} before EOF. Missing function definition?", last_decorator diff --git a/workshop-baker/src/bake/schedule.rs b/workshop-baker/src/bake/schedule.rs index 2b4b2d8..db6cede 100644 --- a/workshop-baker/src/bake/schedule.rs +++ b/workshop-baker/src/bake/schedule.rs @@ -1,47 +1,124 @@ -use super::{decorator::Decorator, decorator::parallel_group, parser::Function}; +// Code in this module PARTIALLY or FULLY utilized AI Coding Agent + +use super::decorator::Decorator; +use super::parser::Function; use crate::bake::error::BakeError; use std::collections::HashMap; -use topological_sort::TopologicalSort; +use workshop_schedule::{Dag, DagNode, EdgeKind, NodeColor}; -/// Builds a DAG from functions using `@after` dependencies and implicit ordering -/// for consecutive `@pipeline` functions. Returns the sorted DAG and a name→Function map. -/// The caller should use `peek_all()` (non-destructive) to get ready nodes, execute them, -/// then `pop()` each completed node to advance the DAG. +/// Thin wrapper so `Function` implements `DagNode`. +#[derive(Clone)] +pub(crate) struct BakeNode { + name: String, + color: NodeColor, + fallible: bool, + /// Original function body + decorators for execution + func: Function, +} + +impl DagNode for BakeNode { + fn name(&self) -> &str { + &self.name + } + fn color(&self) -> NodeColor { + self.color.clone() + } + fn fallible(&self) -> bool { + self.fallible + } +} + +/// Convert a `Function` into a `BakeNode` by extracting scheduling metadata +/// from decorators. +fn function_to_node(func: &Function) -> BakeNode { + let color = if func + .decorators + .iter() + .any(|d| matches!(d, Decorator::Parallel(_))) + { + let group = func + .decorators + .iter() + .find_map(|d| { + if let Decorator::Parallel(Some(g)) = d { + Some(g.clone()) + } else { + None + } + }) + .unwrap_or("*".to_string()); + if group == "*" { + NodeColor::Wildcard + } else { + NodeColor::Named(group) + } + } else { + NodeColor::Sequential + }; + + let fallible = func + .decorators + .iter() + .any(|d| matches!(d, Decorator::Fallible)); + + BakeNode { + name: func.name.clone(), + color, + fallible, + func: func.clone(), + } +} + +/// Builds a DAG from functions using `@after` dependencies and implicit +/// ordering for consecutive `@pipeline` functions. pub fn build_dag( functions: Vec, -) -> Result<(TopologicalSort, HashMap), BakeError> { - let function_mapped = functions +) -> Result<(Dag, HashMap), BakeError> { + let function_mapped: HashMap<&str, &Function> = functions .iter() .map(|f| (f.name.as_str(), f)) - .collect::>(); + .collect(); - let mut ts = TopologicalSort::::new(); - - // Track last pipeline function for implicit @after + let mut dag = Dag::new(); let mut last_pipeline: Option = None; for function in &functions { - ts.insert(function.name.clone()); - + // Determine whether this function is a pipeline node let is_pipeline = function .decorators .iter() .any(|d| matches!(d, Decorator::Pipeline)); + + if !is_pipeline { + continue; + } + let has_after = function .decorators .iter() .any(|d| matches!(d, Decorator::After(_))); - // Add implicit @after for pipeline functions that don't have explicit @after - if is_pipeline { - if !has_after { - if let Some(ref last) = last_pipeline { - ts.add_dependency(last, function.name.clone()); - } + let node = function_to_node(function); + dag.add_node(node).map_err(|e| { + BakeError::IncompatibleDecorators { + function: function.name.clone(), + reason: e.to_string(), } - last_pipeline = Some(function.name.clone()); - } + })?; + // Add implicit @after for consecutive @pipeline functions + if !has_after { + if let Some(ref last) = last_pipeline { + dag.add_edge(last, &function.name, EdgeKind::Implicit) + .map_err(|e| BakeError::IncompatibleDecorators { + function: function.name.clone(), + reason: e.to_string(), + })?; + } + } + last_pipeline = Some(function.name.clone()); + + // Add explicit @after edges for decorator in &function.decorators { if let Decorator::After(dependency) = decorator { if !function_mapped.contains_key(dependency.as_str()) { @@ -55,531 +132,56 @@ pub fn build_dag( dependency.clone(), )); } - ts.add_dependency(dependency, function.name.clone()); + dag.add_edge(dependency, &function.name, EdgeKind::Explicit) + .map_err(|e| BakeError::IncompatibleDecorators { + function: function.name.clone(), + reason: e.to_string(), + })?; } } } + dag.freeze(); + let owned_map: HashMap = functions .into_iter() .map(|f| (f.name.clone(), f)) .collect(); - Ok((ts, owned_map)) + Ok((dag, owned_map)) } -/// Group parallel nodes by their declared group. +/// Group ready parallel nodes by their declared group. /// Returns `(star_nodes, named_groups)`. -/// Star nodes (`group == "*"`) are compatible with any group. -pub fn partition_by_group( - names: &[String], - func_map: &HashMap, +/// +/// Deprecated: use `Dag::ready_wildcards()` and `Dag::ready_groups()` instead. +pub fn partition_by_group<'a>( + ready: &[String], + dag: &Dag, ) -> (Vec, HashMap>) { let mut star_nodes = Vec::new(); let mut named_groups: HashMap> = HashMap::new(); - for name in names { - let func = &func_map[name.as_str()]; - let group = func - .decorators - .iter() - .find_map(|d| { - if let Decorator::Parallel(_) = d { - Some(parallel_group(d)) - } else { - None - } - }) - .unwrap_or("*"); - if group == "*" { + for name in ready { + if dag.is_wildcard(name) { star_nodes.push(name.clone()); - } else { - named_groups - .entry(group.to_string()) - .or_default() - .push(name.clone()); + } else if let Some(g) = dag.group_of(name) { + named_groups.entry(g).or_default().push(name.clone()); } + // Sequential nodes: not included (caller handles them) } (star_nodes, named_groups) } /// Pick a complete group to execute. -/// - If any named group has ready nodes, pick the first (star nodes join). -/// - If only star nodes exist, fire them together. -/// - Otherwise, return None. +/// +/// Deprecated: use `Dag::pick_batch()` instead. pub fn pick_batch( - star_nodes: &[String], - groups: &HashMap>, + _star_nodes: &[String], + _groups: &HashMap>, + _group_totals: &HashMap, ) -> Option> { - for nodes in groups.values() { - let mut batch = nodes.clone(); - batch.extend_from_slice(star_nodes); - return Some(batch); - } - if !star_nodes.is_empty() && groups.is_empty() { - return Some(star_nodes.to_vec()); - } - None -} - -#[cfg(test)] -mod tests { - use super::*; - - fn create_function(name: &str, decorators: Vec, body: &str) -> Function { - Function { - name: name.to_string(), - decorators, - body: body.to_string(), - } - } - - // Helper: drain the DAG and return sorted order (for backward-compatible tests) - fn drain_dag(functions: Vec) -> Result, BakeError> { - let (mut dag, func_map) = build_dag(functions)?; - let mut result = Vec::new(); - while !dag.is_empty() { - let batch = dag.pop_all(); - if batch.is_empty() { - // Cycle detected: remaining nodes form a circular dependency - let remaining: Vec = dag.into_iter().collect(); - return Err(BakeError::CircularDependency(remaining)); - } - for name in batch { - result.push(func_map[&name].clone()); - } - } - Ok(result) - } - - #[test] - fn test_empty_functions() { - let functions = vec![]; - let result = drain_dag(functions).unwrap(); - assert!(result.is_empty()); - } - - #[test] - fn test_single_function_no_dependencies() { - let function = create_function("func1", vec![], "body1"); - let functions = vec![function.clone()]; - - let result = drain_dag(functions).unwrap(); - - assert_eq!(result.len(), 1); - assert_eq!(result[0].name, "func1"); - } - - #[test] - fn test_multiple_functions_no_dependencies() { - let function1 = create_function("func1", vec![], "body1"); - let function2 = create_function("func2", vec![], "body2"); - let function3 = create_function("func3", vec![], "body3"); - let functions = vec![function1.clone(), function2.clone(), function3.clone()]; - - let result = drain_dag(functions).unwrap(); - - assert_eq!(result.len(), 3); - let names: Vec = result.iter().map(|f| f.name.clone()).collect(); - assert!(names.contains(&"func1".to_string())); - assert!(names.contains(&"func2".to_string())); - assert!(names.contains(&"func3".to_string())); - } - - #[test] - fn test_simple_dependency() { - let function1 = create_function("func1", vec![], "body1"); - let function2 = create_function( - "func2", - vec![Decorator::After("func1".to_string())], - "body2", - ); - let functions = vec![function1.clone(), function2.clone()]; - - let result = drain_dag(functions).unwrap(); - - assert_eq!(result.len(), 2); - assert_eq!(result[0].name, "func1"); - assert_eq!(result[1].name, "func2"); - } - - #[test] - fn test_multiple_dependencies() { - let function1 = create_function("func1", vec![], "body1"); - let function2 = create_function( - "func2", - vec![Decorator::After("func1".to_string())], - "body2", - ); - let function3 = create_function( - "func3", - vec![Decorator::After("func2".to_string())], - "body3", - ); - let functions = vec![function1.clone(), function2.clone(), function3.clone()]; - - let result = drain_dag(functions).unwrap(); - - assert_eq!(result.len(), 3); - assert_eq!(result[0].name, "func1"); - assert_eq!(result[1].name, "func2"); - assert_eq!(result[2].name, "func3"); - } - - #[test] - fn test_complex_dependencies() { - let function1 = create_function("func1", vec![], "body1"); - let function2 = create_function( - "func2", - vec![ - Decorator::After("func1".to_string()), - Decorator::After("func3".to_string()), - ], - "body2", - ); - let function3 = create_function( - "func3", - vec![Decorator::After("func1".to_string())], - "body3", - ); - let function4 = create_function("func4", vec![], "body4"); - let functions = vec![ - function1.clone(), - function2.clone(), - function3.clone(), - function4.clone(), - ]; - - let result = drain_dag(functions).unwrap(); - - assert_eq!(result.len(), 4); - - let positions: std::collections::HashMap = result - .iter() - .enumerate() - .map(|(i, f)| (f.name.clone(), i)) - .collect(); - - assert!(positions.get("func1").unwrap() < positions.get("func2").unwrap()); - assert!(positions.get("func1").unwrap() < positions.get("func3").unwrap()); - assert!(positions.get("func3").unwrap() < positions.get("func2").unwrap()); - } - - #[test] - fn test_mixed_decorators() { - let function1 = create_function( - "func1", - vec![Decorator::Pipeline, Decorator::Timeout(1000)], - "body1", - ); - let function2 = create_function( - "func2", - vec![ - Decorator::After("func1".to_string()), - Decorator::Retry(3, 100), - Decorator::Fallible, - ], - "body2", - ); - let functions = vec![function1.clone(), function2.clone()]; - - let result = drain_dag(functions).unwrap(); - - assert_eq!(result.len(), 2); - assert_eq!(result[0].name, "func1"); - assert_eq!(result[1].name, "func2"); - } - - #[test] - fn test_function_with_multiple_after_decorators() { - let function1 = create_function("func1", vec![], "body1"); - let function2 = create_function("func2", vec![], "body2"); - let function3 = create_function( - "func3", - vec![ - Decorator::After("func1".to_string()), - Decorator::After("func2".to_string()), - ], - "body3", - ); - let functions = vec![function1.clone(), function2.clone(), function3.clone()]; - - let result = drain_dag(functions).unwrap(); - - assert_eq!(result.len(), 3); - - let positions: std::collections::HashMap = result - .iter() - .enumerate() - .map(|(i, f)| (f.name.clone(), i)) - .collect(); - - assert!(positions.get("func1").unwrap() < positions.get("func3").unwrap()); - assert!(positions.get("func2").unwrap() < positions.get("func3").unwrap()); - } - - #[test] - fn test_ignores_non_after_decorators() { - let function1 = create_function( - "func1", - vec![ - Decorator::Pipeline, - Decorator::If("condition".to_string()), - Decorator::Fallible, - ], - "body1", - ); - let function2 = create_function( - "func2", - vec![ - Decorator::Loop(5), - Decorator::Timeout(1000), - Decorator::Retry(3, 100), - Decorator::Export, - ], - "body2", - ); - let functions = vec![function1.clone(), function2.clone()]; - - let result = drain_dag(functions).unwrap(); - - assert_eq!(result.len(), 2); - let names: Vec = result.iter().map(|f| f.name.clone()).collect(); - assert!(names.contains(&"func1".to_string())); - assert!(names.contains(&"func2".to_string())); - } - - #[test] - fn test_implicit_after_pipeline() { - let func1 = create_function("func1", vec![Decorator::Pipeline], "body1"); - let func2 = create_function("func2", vec![Decorator::Pipeline], "body2"); - let func3 = create_function("func3", vec![Decorator::Pipeline], "body3"); - let functions = vec![func1.clone(), func2.clone(), func3.clone()]; - - let result = drain_dag(functions).unwrap(); - - assert_eq!(result.len(), 3); - assert_eq!(result[0].name, "func1"); - assert_eq!(result[1].name, "func2"); - assert_eq!(result[2].name, "func3"); - } - - #[test] - fn test_first_pipeline_no_implicit_after() { - let func1 = create_function("func1", vec![Decorator::Pipeline], "body1"); - let functions = vec![func1.clone()]; - - let result = drain_dag(functions).unwrap(); - - assert_eq!(result.len(), 1); - assert_eq!(result[0].name, "func1"); - } - - #[test] - fn test_explicit_after_overrides_implicit() { - let func1 = create_function("func1", vec![Decorator::Pipeline], "body1"); - let func2 = create_function( - "func2", - vec![Decorator::Pipeline, Decorator::After("func1".to_string())], - "body2", - ); - let func3 = create_function("func3", vec![Decorator::Pipeline], "body3"); - let functions = vec![func1.clone(), func2.clone(), func3.clone()]; - - let result = drain_dag(functions).unwrap(); - - assert_eq!(result.len(), 3); - let positions: std::collections::HashMap = result - .iter() - .enumerate() - .map(|(i, f)| (f.name.clone(), i)) - .collect(); - - assert!(positions.get("func1").unwrap() < positions.get("func2").unwrap()); - assert!(positions.get("func2").unwrap() < positions.get("func3").unwrap()); - } - - #[test] - fn test_mixed_pipeline_and_non_pipeline() { - let func1 = create_function("func1", vec![], "body1"); - let func2 = create_function("func2", vec![Decorator::Pipeline], "body2"); - let func3 = create_function("func3", vec![], "body3"); - let func4 = create_function("func4", vec![Decorator::Pipeline], "body4"); - let functions = vec![func1.clone(), func2.clone(), func3.clone(), func4.clone()]; - - let result = drain_dag(functions).unwrap(); - - assert_eq!(result.len(), 4); - let positions: std::collections::HashMap = result - .iter() - .enumerate() - .map(|(i, f)| (f.name.clone(), i)) - .collect(); - - assert!(positions.get("func2").unwrap() < positions.get("func4").unwrap()); - } - - #[test] - fn test_single_pipeline_works() { - let func1 = create_function("func1", vec![Decorator::Pipeline], "body1"); - let functions = vec![func1.clone()]; - - let result = drain_dag(functions).unwrap(); - assert_eq!(result.len(), 1); - assert_eq!(result[0].name, "func1"); - } - - #[test] - fn test_two_pipelines_implicit_chain() { - let func1 = create_function("func1", vec![Decorator::Pipeline], "body1"); - let func2 = create_function("func2", vec![Decorator::Pipeline], "body2"); - let functions = vec![func1.clone(), func2.clone()]; - - let result = drain_dag(functions).unwrap(); - assert_eq!(result.len(), 2); - let positions: std::collections::HashMap = result - .iter() - .enumerate() - .map(|(i, f)| (f.name.clone(), i)) - .collect(); - assert!(positions.get("func1").unwrap() < positions.get("func2").unwrap()); - } - - #[test] - fn test_pipeline_explicit_after_skips_implicit() { - let func1 = create_function("func1", vec![Decorator::Pipeline], "body1"); - let func2 = create_function("func2", vec![Decorator::Pipeline], "body2"); - let func3 = create_function( - "func3", - vec![Decorator::Pipeline, Decorator::After("func1".to_string())], - "body3", - ); - let functions = vec![func1.clone(), func2.clone(), func3.clone()]; - - let result = drain_dag(functions).unwrap(); - assert_eq!(result.len(), 3); - let positions: std::collections::HashMap = result - .iter() - .enumerate() - .map(|(i, f)| (f.name.clone(), i)) - .collect(); - assert!(positions.get("func1").unwrap() < positions.get("func3").unwrap()); - // TODO: Our design spec indicates order-based sorting, however toposort crate does not guarantee order of independent nodes. We should consider whether we want to enforce order-based sorting for pipelines, or if we are okay with any valid topological order. - // assert!(positions.get("func2").unwrap() < positions.get("func3").unwrap()); - } - - #[test] - fn test_circular_dependency_detected() { - let func1 = create_function( - "func1", - vec![Decorator::After("func2".to_string())], - "body1", - ); - let func2 = create_function( - "func2", - vec![Decorator::After("func1".to_string())], - "body2", - ); - let functions = vec![func1.clone(), func2.clone()]; - - let result = drain_dag(functions); - assert!(result.is_err()); - assert!(matches!( - result.unwrap_err(), - BakeError::CircularDependency(_) - )); - } - - #[test] - fn test_unknown_dependency_error() { - let func1 = create_function( - "func1", - vec![Decorator::After("nonexistent".to_string())], - "body1", - ); - let functions = vec![func1.clone()]; - - let result = drain_dag(functions); - assert!(result.is_err()); - assert!(matches!( - result.unwrap_err(), - BakeError::UnknownDependency(_, _) - )); - } - - #[test] - fn test_pipeline_with_multiple_decorators() { - let func1 = create_function( - "func1", - vec![ - Decorator::Pipeline, - Decorator::Timeout(60), - Decorator::Fallible, - ], - "body1", - ); - let func2 = create_function( - "func2", - vec![ - Decorator::Pipeline, - Decorator::Retry(3, 5), - Decorator::After("func1".to_string()), - ], - "body2", - ); - let functions = vec![func1.clone(), func2.clone()]; - - let result = drain_dag(functions).unwrap(); - assert_eq!(result.len(), 2); - let positions: std::collections::HashMap = result - .iter() - .enumerate() - .map(|(i, f)| (f.name.clone(), i)) - .collect(); - assert!(positions.get("func1").unwrap() < positions.get("func2").unwrap()); - } - - #[test] - fn test_long_pipeline_chain() { - let func1 = create_function("func1", vec![Decorator::Pipeline], "body1"); - let func2 = create_function("func2", vec![Decorator::Pipeline], "body2"); - let func3 = create_function("func3", vec![Decorator::Pipeline], "body3"); - let func4 = create_function("func4", vec![Decorator::Pipeline], "body4"); - let func5 = create_function("func5", vec![Decorator::Pipeline], "body5"); - let functions = vec![ - func1.clone(), - func2.clone(), - func3.clone(), - func4.clone(), - func5.clone(), - ]; - - let result = drain_dag(functions).unwrap(); - assert_eq!(result.len(), 5); - let positions: std::collections::HashMap = result - .iter() - .enumerate() - .map(|(i, f)| (f.name.clone(), i)) - .collect(); - assert!(positions.get("func1").unwrap() < positions.get("func2").unwrap()); - assert!(positions.get("func2").unwrap() < positions.get("func3").unwrap()); - assert!(positions.get("func3").unwrap() < positions.get("func4").unwrap()); - assert!(positions.get("func4").unwrap() < positions.get("func5").unwrap()); - } - - #[test] - fn test_non_pipeline_between_pipelines() { - let func1 = create_function("func1", vec![Decorator::Pipeline], "body1"); - let func2 = create_function("func2", vec![], "body2"); - let func3 = create_function("func3", vec![Decorator::Pipeline], "body3"); - let functions = vec![func1.clone(), func2.clone(), func3.clone()]; - - let result = drain_dag(functions).unwrap(); - assert_eq!(result.len(), 3); - let positions: std::collections::HashMap = result - .iter() - .enumerate() - .map(|(i, f)| (f.name.clone(), i)) - .collect(); - assert!(positions.get("func1").unwrap() < positions.get("func3").unwrap()); - } + // This function is replaced by Dag::pick_batch() which has all the + // group metadata internally. Keeping the signature for backward compat + // but redirect callers to use Dag directly. + unimplemented!("use dag.pick_batch() instead") } diff --git a/workshop-baker/src/bake/trivial.rs b/workshop-baker/src/bake/trivial.rs index 9e04b91..cd9a984 100644 --- a/workshop-baker/src/bake/trivial.rs +++ b/workshop-baker/src/bake/trivial.rs @@ -1,5 +1,4 @@ use crate::bake::builder::{build_script, RenderMode}; -use crate::bake::constant::TRIVIAL_SCRIPT_NAME; use crate::bake::error::BakeError; use crate::bake::parser::Function; use std::path::Path; @@ -22,11 +21,11 @@ pub async fn run_trivial( render_mode: RenderMode, ) -> Result<(), BakeError> { let function = Function { - name: TRIVIAL_SCRIPT_NAME.to_string(), + name: String::new(), decorators: vec![], body: script_content.to_string(), }; - let script = build_script(&function, bake_base_path, workspace, None, render_mode)?; + let script = build_script(&function, bake_base_path, workspace, render_mode)?; engine.execute_script(&script, ctx, event_tx).await?; Ok(()) } diff --git a/workshop-baker/src/bare.rs b/workshop-baker/src/bare.rs index 3b34857..4e21d8e 100644 --- a/workshop-baker/src/bare.rs +++ b/workshop-baker/src/bare.rs @@ -27,6 +27,7 @@ pub async fn run_bare( ("finalize", or_workshop(&cli.finalize, "finalize.yml")) } }; + log::debug!("Running bare command {} with config {}", stage, config.display()); let prebake_path = or_workshop(&cli.prebake, "prebake.yml"); let bake_base = or_workshop(&cli.bake_base, "bake_base.sh"); diff --git a/workshop-baker/src/finalize/AGENTS.md b/workshop-baker/src/finalize/AGENTS.md deleted file mode 100644 index 81c6bb2..0000000 --- a/workshop-baker/src/finalize/AGENTS.md +++ /dev/null @@ -1,52 +0,0 @@ -# workshop-baker/src/finalize - -**Generated:** 2026-05-19 -**Commit:** 28abc5d - -Artifact packaging, plugin-based notifications, and post-build hooks. - -## STRUCTURE -``` -finalize/ -├── config.rs # FinalizeConfig YAML (379 lines, plugins, notifications) -├── stage.rs # FinalizeStage enum -├── stage/hook.rs # Post-build hook execution -├── event.rs # Event propagation -├── template.rs # Variable substitution {{VARIABLE}} -├── plugin.rs # Plugin trait + registration -├── plugin/ -│ ├── metadata.rs # Plugin manifest parsing -│ ├── dylib.rs # Dynamic library plugins -│ ├── rhai.rs # Rhai scripting (todo!()) -│ ├── shell.rs # Shell command plugins -│ └── internal.rs + internal/ # Built-in plugins (mail, webhook, satori, insitenotify) -├── types.rs -├── types/compression.rs -├── error.rs # FinalizeError -└── constant.rs -``` - -## WHERE TO LOOK -| Task | Location | -|------|----------| -| Config loading | `config.rs:17` - `FinalizeConfig` + `parse()` | -| Plugin registration | `plugin.rs` - `register()` | -| Email notifications | `plugin/internal/mail.rs` - SMTP | -| Webhooks | `plugin/internal/webhook.rs` | -| Satori integration | `plugin/internal/satori.rs` | -| Shell plugins | `plugin/shell.rs` | -| Template rendering | `template.rs` - `{{VARIABLE}}` handlebars-style | - -## CONVENTIONS -- **Plugin System**: `FinalizePlugin` trait + registry with both internal and dynamic plugins -- **Template Syntax**: `{{VARIABLE}}` handlebars-style substitution via `template.rs` -- **Stages**: EarlyHook → (Artifacts) → LateHook → Notifications -- **Notification**: `notify.rs` in parent module drives plugin-based notification dispatch -- **ResourceRegistry**: Shared resource cache passed through to plugins - -## ANTI-PATTERNS -1. **fetch/ subdirectory removed** — was `plugin/fetch/{git,http,extract,checksum}`, no longer exists -2. **SHA1 deprecated** — workshop-getterurl handles checksums now -3. **Unimplemented** — `rhai.rs:17` - "todo!()" for Rhai scripting -4. **TODO markers** — Multiple TODOs in config.rs (lines 305-306) -5. **Artifact stage not implemented** — `finalize.rs:57` — "TODO: artifact stage implementation" diff --git a/workshop-baker/src/finalize/config.rs b/workshop-baker/src/finalize/config.rs index 57ad6a0..6816c69 100644 --- a/workshop-baker/src/finalize/config.rs +++ b/workshop-baker/src/finalize/config.rs @@ -185,15 +185,21 @@ impl<'de> serde::Deserialize<'de> for NotificationConfig { #[serde(default)] templates: HashMap, #[serde(flatten)] - groups_raw: HashMap>, + groups_raw: HashMap>>, } let h = Helper::deserialize(deserializer)?; let groups: HashMap = h.groups_raw.into_iter() - .map(|(k, v)| { - k.parse::().map(|n| (n, v)) - .map_err(|_| serde::de::Error::custom(format!("priority key must be integer, got {:?}", k))) + .filter_map(|(k, v)| { + let parsed = k.parse::().map(|n| (n, v.unwrap_or_default())); + match parsed { + Ok(pair) => Some(pair), + Err(e) => { + log::warn!("priority key must be integer, got {:?}: {}", k, e); + None + } + } }) - .collect::>()?; + .collect(); Ok(NotificationConfig { templates: h.templates, groups }) } } diff --git a/workshop-baker/src/finalize/event.rs b/workshop-baker/src/finalize/event.rs index 3080de5..163e491 100644 --- a/workshop-baker/src/finalize/event.rs +++ b/workshop-baker/src/finalize/event.rs @@ -30,14 +30,19 @@ pub async fn event_receiver( } }, ExecutionEvent::TaskCompleted { task_id, result } => { - log::debug!( - "[{}] [{}] Task completed: {} (exit_code={}, duration={:?})", - pipeline_name, - build_id, - task_id, - result.exit_code, - result.duration - ); + if result.success { + log::debug!( + "[{}] [{}] Task completed: {} (exit_code={}, duration={:?})", + pipeline_name, build_id, task_id, + result.exit_code, result.duration, + ); + } else { + log::warn!( + "[{}] [{}] Task completed with non-zero exit: {} (exit_code={}, duration={:?})", + pipeline_name, build_id, task_id, + result.exit_code, result.duration, + ); + } } ExecutionEvent::TaskFailed { task_id, error } => { log::error!( diff --git a/workshop-baker/src/finalize/plugin.rs b/workshop-baker/src/finalize/plugin.rs index 943f65f..9a52a96 100644 --- a/workshop-baker/src/finalize/plugin.rs +++ b/workshop-baker/src/finalize/plugin.rs @@ -232,7 +232,9 @@ pub fn register( _ => {} } log::debug!("Registering internal plugin {}", i.name); - plugins.insert(i.name.to_string(), FinalizePlugin::new(i.entry)); + let mut plugin = FinalizePlugin::new(i.entry); + plugin.metadata.renderer = Some(i.renderer); + plugins.insert(i.name.to_string(), plugin); count += 1; } diff --git a/workshop-baker/src/finalize/plugin/internal.rs b/workshop-baker/src/finalize/plugin/internal.rs index c537499..b7c8fa5 100644 --- a/workshop-baker/src/finalize/plugin/internal.rs +++ b/workshop-baker/src/finalize/plugin/internal.rs @@ -1,4 +1,5 @@ use crate::finalize::plugin::AsyncPluginFn; +use crate::notify::types::NotificationRenderer; pub mod dummy; mod insitenotify; mod mail; @@ -7,6 +8,7 @@ mod webhook; pub struct InternalPlugin { pub name: &'static str, pub entry: Box, + pub renderer: NotificationRenderer, } pub fn get_internal_plugin() -> Vec { @@ -14,18 +16,22 @@ pub fn get_internal_plugin() -> Vec { InternalPlugin { name: "in-site-notify", entry: Box::new(insitenotify::InsiteNotify), + renderer: NotificationRenderer::Plugin, }, InternalPlugin { name: "mail", entry: Box::new(mail::Mail), + renderer: NotificationRenderer::Server, }, InternalPlugin { name: "satori", entry: Box::new(satori::Satori), + renderer: NotificationRenderer::Server, }, InternalPlugin { name: "webhook", entry: Box::new(webhook::Webhook), + renderer: NotificationRenderer::Server, }, ] } @@ -34,5 +40,6 @@ pub fn get_dummy_plugin() -> Vec { vec![InternalPlugin { name: "dummy", entry: Box::new(dummy::Dummy), + renderer: NotificationRenderer::Server, }] } diff --git a/workshop-baker/src/finalize/stage/hook.rs b/workshop-baker/src/finalize/stage/hook.rs index 876136c..ddaed67 100644 --- a/workshop-baker/src/finalize/stage/hook.rs +++ b/workshop-baker/src/finalize/stage/hook.rs @@ -51,7 +51,7 @@ pub async fn hook( .await .map_err(|e| { log::error!("Plugin hook {} failed: {:?}", index, e); - ExecutionError::ExecutionFailed(e.to_string()) + ExecutionError::ExecutionFailed { task_id: format!("hook-{}", index), exit_code: -1 } })?; continue; } diff --git a/workshop-baker/src/main.rs b/workshop-baker/src/main.rs index d910b9e..2202011 100644 --- a/workshop-baker/src/main.rs +++ b/workshop-baker/src/main.rs @@ -80,6 +80,8 @@ async fn worker_main(cli: Cli) -> Result<(), CliError> { } log::info!("Pipeline complete. Draining notifications..."); + // Drop the sender so notify() can end and retry_tx is dropped + queue.tx = None; if let Err(e) = queue.drain().await { log::error!("Notification queue error: {}", e); } diff --git a/workshop-baker/src/notify.rs b/workshop-baker/src/notify.rs index 2d9ae4f..7f5c888 100644 --- a/workshop-baker/src/notify.rs +++ b/workshop-baker/src/notify.rs @@ -31,8 +31,9 @@ pub async fn notify( ) -> anyhow::Result<()> { let mut finalize = parse(finalize_path)?; - // Always inject in-site-notify as the lowest priority fallback - { + // L18: 0 priorities → notification ends; L19: has priorities → inject in-site fallback + if !finalize.notification.groups.is_empty() { + log::debug!("Injecting in-site-notify fallback"); let max_p = finalize.notification.groups.keys().max().copied().unwrap_or(0); let mut m = std::collections::HashMap::new(); m.insert( @@ -50,7 +51,7 @@ pub async fn notify( Some(&plugin_list), &mut finalize.notification, registry, -)?; + )?; resolve_external_template_refs(&mut finalize.notification.templates, registry); flatten_template_refs(&mut finalize.notification.templates)?; diff --git a/workshop-baker/src/notify/handler.rs b/workshop-baker/src/notify/handler.rs index 38a7407..1e0c3a8 100644 --- a/workshop-baker/src/notify/handler.rs +++ b/workshop-baker/src/notify/handler.rs @@ -336,7 +336,8 @@ fn build_plugin_argument( Some(argument) } NotificationRenderer::Plugin => { - unimplemented!("Plugin renderer is not yet implemented") + // PSR: pass config directly to plugin (plugin handles own rendering) + Some(rule.config.clone()) } NotificationRenderer::Interactive => { unimplemented!("Interactive renderer is not yet implemented") diff --git a/workshop-baker/src/notify/queue.rs b/workshop-baker/src/notify/queue.rs index 9c79ded..3397e88 100644 --- a/workshop-baker/src/notify/queue.rs +++ b/workshop-baker/src/notify/queue.rs @@ -8,7 +8,7 @@ use std::collections::BinaryHeap; /// - `rx` receives retry events back from notify() pub struct NotificationQueue { pub pq: BinaryHeap, - pub tx: NotificationEventSender, + pub tx: Option, pub rx: NotificationEventReceiver, } @@ -16,7 +16,7 @@ impl NotificationQueue { pub fn new(tx: NotificationEventSender, rx: NotificationEventReceiver) -> Self { Self { pq: BinaryHeap::new(), - tx, + tx: Some(tx), rx, } } @@ -40,7 +40,9 @@ impl NotificationQueue { break; } let event = self.pq.pop().unwrap(); - self.tx.send(event).await?; + if let Some(ref tx) = self.tx { + tx.send(event).await?; + } } Ok(()) } @@ -77,7 +79,9 @@ impl NotificationQueue { while let Some(peek) = self.pq.peek() { if peek.not_before > now { break; } let event = self.pq.pop().unwrap(); - self.tx.send(event).await?; + if let Some(ref tx) = self.tx { + tx.send(event).await?; + } } } } diff --git a/workshop-baker/src/notify/template.rs b/workshop-baker/src/notify/template.rs index e611919..ba2521a 100644 --- a/workshop-baker/src/notify/template.rs +++ b/workshop-baker/src/notify/template.rs @@ -109,6 +109,7 @@ pub fn resolve_external_template_refs( templates: &mut HashMap, registry: &ResourceRegistry, ) { + log::debug!("Resolving external template refs"); for (name, tmpl_def) in templates.iter_mut() { if tmpl_def.use_.is_none() { continue; @@ -186,6 +187,7 @@ pub fn resolve_template_by_schema( pub fn flatten_template_refs( templates: &mut HashMap, ) -> Result<(), NotifyError> { + log::debug!("Flattening template refs"); let names: Vec = templates.keys().cloned().collect(); for name in names { let mut visited = std::collections::HashSet::new(); diff --git a/workshop-baker/src/notify/types.rs b/workshop-baker/src/notify/types.rs index 5100157..7eb6976 100644 --- a/workshop-baker/src/notify/types.rs +++ b/workshop-baker/src/notify/types.rs @@ -78,7 +78,7 @@ impl ParsedTrigger { impl NotificationRule { pub fn accept(&self, result: &StageOutcome, stage: &StagePhase, substage: &str) -> bool { if self.trigger.is_empty() { - eprintln!("[ACCEPT] trigger empty, returning true"); + log::trace!("[ACCEPT] trigger empty, returning true"); return true; } self.trigger diff --git a/workshop-baker/src/notify/util.rs b/workshop-baker/src/notify/util.rs index 295dc11..d20cb09 100644 --- a/workshop-baker/src/notify/util.rs +++ b/workshop-baker/src/notify/util.rs @@ -5,6 +5,7 @@ use crate::notify::types::{DEFAULT_NOTIFY_PARAMS, NotificationEvent, Notificatio use std::collections::HashSet; pub fn validate(finalize: &FinalizeConfig) -> anyhow::Result<()> { + log::debug!("Validating finalize config"); let validate_result = finalize.validate(); if let Err(errors) = validate_result { for error in &errors { diff --git a/workshop-baker/src/prebake/AGENTS.md b/workshop-baker/src/prebake/AGENTS.md deleted file mode 100644 index 0597e97..0000000 --- a/workshop-baker/src/prebake/AGENTS.md +++ /dev/null @@ -1,54 +0,0 @@ -# workshop-baker/src/prebake - -**Generated:** 2026-04-22 -**Commit:** 7b3b71a - -Build environment setup: bootstrap, dependencies, security, hooks. - -## STRUCTURE - -``` -prebake/ -├── config.rs # PrebakeConfig YAML schema (environment, deps, cache) -├── stage.rs # PrebakeStage enum (Bootstrap→EarlyHook→DepsSystem→DepsUser→LateHook→Ready) -├── stage/ -│ ├── bootstrap.rs # User creation (vulcan), sudo/doas setup -│ ├── environment.rs # Env var injection -│ ├── depssystem.rs # System package installation -│ ├── depsuser.rs # User package installation (cargo, npm, etc.) -│ └── hook.rs # Hook execution (pre/post stage) -├── env/ # Environment providers -│ ├── mod.rs # EnvProvider trait -│ ├── container.rs # Docker/container support -│ ├── firecracker.rs # Firecracker microVM -│ ├── baremetal.rs # Direct host execution -│ └── custom.rs # Custom env (todo!()) -├── security.rs # Privilege dropping (unsafe libc calls) -├── event.rs # Event logging -├── error.rs # PrebakeError + BootstrapError -├── types.rs # Module re-exports -├── types/ # memsize, cache, builderconfig, architecture -├── constant.rs # Module constants -└── env.rs # Environment module root -``` - -## WHERE TO LOOK - -| Task | Location | -|------|----------| -| Config loading | `config.rs:PrebakeConfig::from_yaml()` | -| Stage ordering | `stage.rs:PrebakeStage` enum | -| User setup | `stage/bootstrap.rs` - vulcan user creation | -| Security | `security.rs:72` - privilege dropping | -| Hooks | `stage/hook.rs` - pre/post stage hooks | - -## CONVENTIONS -- **PrebakeStage**: Ordered enum Bootstrap→EarlyHook→DepsSystem→DepsUser→LateHook→Ready -- **ExecutionContext**: Carries task_id, username, env_vars through all stages -- **Target User**: "vulcan" (hardcoded in bootstrap.rs:6-24) - -## ANTI-PATTERNS -1. **Hardcoded user** — "vulcan" username hardcoded throughout -2. **Unsafe blocks** — `security.rs` has 5+ unsafe libc calls -3. **Incomplete** — custom.rs and apt.rs have `todo!()` stubs -4. **doas.conf** — bootstrap.rs:183: "We do not know how to write doas.conf lol" diff --git a/workshop-baker/src/prebake/event.rs b/workshop-baker/src/prebake/event.rs index f4b7da1..c79cac4 100644 --- a/workshop-baker/src/prebake/event.rs +++ b/workshop-baker/src/prebake/event.rs @@ -1,5 +1,7 @@ use workshop_engine::{ExecutionEvent, StreamType}; +use crate::bake::constant::PIPELINE_SKIP_ERRORCODE; + pub async fn event_receiver( mut rx: tokio::sync::mpsc::Receiver, pipeline_name: String, @@ -7,34 +9,41 @@ pub async fn event_receiver( ) { while let Some(event) = rx.recv().await { match event { - ExecutionEvent::TaskStarted { task_id, timestamp } => { + ExecutionEvent::TaskStarted { task_id, timestamp: _ } => { log::debug!( - "[{}] [{}] [{}] Task started: {}", + "[{}] [{}] Task started: {}", pipeline_name, build_id, - timestamp, + // timestamp, task_id ); } ExecutionEvent::OutputChunk { - task_id, + task_id: _, stream, data, } => match stream { - StreamType::Stdout => print!("[{}] [{}] [stdout] {}", pipeline_name, task_id, data), + StreamType::Stdout => print!("{}", data), StreamType::Stderr => { - eprint!("[{}] [{}] [stderr] {}", pipeline_name, task_id, data) + eprint!("{}", data) } }, ExecutionEvent::TaskCompleted { task_id, result } => { - log::debug!( - "[{}] [{}] Task completed: {} (exit_code={}, duration={:?})", - pipeline_name, - build_id, - task_id, - result.exit_code, - result.duration - ); + if result.success { + log::debug!( + "[{}] [{}] Task completed: {} (exit_code={}, duration={:?})", + pipeline_name, build_id, task_id, + result.exit_code, result.duration, + ); + } else if result.exit_code == PIPELINE_SKIP_ERRORCODE { + log::info!("[{}] [{}] Task skipped: {}", pipeline_name, build_id, task_id); + } else { + log::warn!( + "[{}] [{}] Task completed with non-zero exitcode: {} (exit_code={}, duration={:?})", + pipeline_name, build_id, task_id, + result.exit_code, result.duration, + ); + } } ExecutionEvent::TaskFailed { task_id, error } => { log::error!( diff --git a/workshop-engine/AGENTS.md b/workshop-engine/AGENTS.md new file mode 100644 index 0000000..e792bce --- /dev/null +++ b/workshop-engine/AGENTS.md @@ -0,0 +1,39 @@ +# WORKSHOP-ENGINE KNOWLEDGE BASE + +## OVERVIEW +Core execution layer for HoneyBiscuitWorkshop. Library-only crate (no binary). Provides Engine executor, process abstraction, package manager interfaces, duration parsing, and typed error/event systems. Used by workshop-baker; depends on no other workspace crates. + +## WHERE TO LOOK + +| Task | File | Notes | +|------|------|-------| +| Script execution | `src/executor.rs` | `Engine::execute_script()` — spawn, stream, timeout, kill | +| Process abstraction | `src/child.rs` | `ManagedChild` trait; `TokioChild` (real), `MockChild` (hand-rolled, no mockall) | +| Configurable commands | `src/command.rs` | `CustomCommand` — name, command, env, working_dir, timeout | +| System PMs | `src/pm.rs` + `pm/{apt,pacman,dnf,apk}.rs` | `PackageManager` trait; `all_managers()`, `select(name)` | +| User PMs | `src/upm.rs` + `upm/{rust,go,python,custom}.rs` | `UserPackageManager` trait (async); `UPMSysDeps` | +| Repology endpoint | `src/repology.rs` + `repology/local.rs` | `RepologyEndpoint` enum; local instance support | +| Duration parsing | `src/time.rs` | `parse_duration_to_ms()`, serde helpers | +| Core types | `src/types.rs` | `ExecutionContext`, `ExecutionResult`, `ExecutionEvent`, `EventSender`, `StreamType`, `Engine` | +| Error types | `src/error.rs` | `ExecutionError`, `DependencyError`, `HasExitCode` trait, `EXITCODE_*` constants | +| Integration tests | `tests/engine_integration.rs` | 12 async tests; run with `cargo test --test engine_integration` | + +## CONVENTIONS + +- **Edition**: Rust 2024 +- **Key deps**: tokio (full), thiserror, serde+serde_yaml, async-trait, chrono, os_info, duration-str, libc, lazy_static +- **Process mgmt**: `ManagedChild` trait abstracts over real/mock children. `TokioChild` wraps `tokio::process::Child`. `MockChild` is hand-rolled (no mockall dep); supports hanging, custom output, exit codes. +- **Package manager selection**: Use `pm::select("apt")` with explicit name — NEVER auto-detect by probing the filesystem. +- **Error mapping**: `HasExitCode` trait maps errors to numeric exit codes. `ExecutionError` maps to specific `EXITCODE_*`; `DependencyError` always maps to 1. +- **Re-exports**: `lib.rs` re-exports key types for convenience (`Engine`, `ExecutionContext`, `PackageManager`, `ExecutionError`, etc.) +- **Tests**: Inline `#[cfg(test)]` blocks in `pm.rs` (8 tests). Integration tests in `tests/engine_integration.rs` (12 async tests, separate binary). CI runs both. +- **Kill semantics**: `TokioChild::kill()` sends `SIGKILL` to entire process group via `libc::kill(-PGID, SIGKILL)` with `ESRCH` silently ignored. + +## ANTI-PATTERNS + +- **Do NOT auto-detect the host package manager** — always use `pm::select()` with an explicitly configured name. Production PM selection comes from environment config, not probing. +- **Do NOT use `detect()`/`adopt()` patterns** for PM selection — these don't exist in this crate, and callers must not invent them. +- **Do NOT depend on workshop-baker or other workspace crates** — engine is a leaf dependency. Types needed by external callers should flow through `ExecutionContext`, not reverse imports. +- **Do NOT use mockall or other mocking frameworks** — `MockChild` is the only mock mechanism. Keep it that way. +- **Do NOT use `unimplemented!()`** — all package manager implementations are complete. Stubs belong in workshop-baker, not here. +- **Do NOT add magic platform detection** — `os_info` is used only for mirror config mapping in PM impls. Do not extend its use to auto-detect behavior. diff --git a/workshop-engine/src/AGENTS.md b/workshop-engine/src/AGENTS.md deleted file mode 100644 index ffc71ca..0000000 --- a/workshop-engine/src/AGENTS.md +++ /dev/null @@ -1,51 +0,0 @@ -# workshop-engine/src - -**Generated:** 2026-05-19 -**Commit:** 28abc5d - -Execution engine extracted from workshop-baker. Process management, package detection, resource isolation. - -## STRUCTURE -``` -src/ -├── lib.rs # Module declarations + 10+ convenience re-exports -├── error.rs # ExecutionError + DependencyError + HasExitCode trait -├── types.rs # ExecutionContext, ExecutionResult, Engine, etc. -├── child.rs # ManagedChild + TokioChild process wrappers -├── command.rs # CustomCommand builder -├── executor.rs # Script execution -├── time.rs # Duration parsing helpers -├── pm.rs # PackageManager trait + detection -├── pm/ # Backends: apt, apk, dnf, pacman -├── upm.rs # UserPackageManager trait -├── upm/ # Backends: rust (cargo), python (pip), go, custom -├── repology.rs # RepologyEndpoint enum -└── repology/ # Package name resolution (local/) -``` - -## KEY EXPORTS -| Symbol | Type | Role | -|--------|------|------| -| ExecutionContext | struct | Pipeline context (task_id, username, env, etc.) | -| Engine | struct | Script executor (placeholder) | -| ExecutionError | enum | Error types with exit codes | -| HasExitCode | trait | Exit code protocol for all error types | -| ManagedChild | struct | Child process wrapper with managed lifecycle | -| TokioChild | struct | Tokio-based child process wrapper | -| CustomCommand | struct | Builder for command execution | -| PackageManager | trait | System package manager abstraction | -| UserPackageManager | trait | User-level package manager abstraction | -| RepologyEndpoint | enum | Package name resolution endpoint | -| ExecutionEvent | enum | Event stream (TaskStarted, OutputChunk, etc.) | -| ExecutionResult | struct | Command execution result (exit_code, stdout, stderr) | -| EventSender | type alias | `Option>` | - -## CONVENTIONS -- **Exit codes**: Centralized in `error.rs` — EXITCODE_OK=0 through EXITCODE_PRIV_DROP_FAILED=201 -- **ExecutionContext**: Defaults to current directory, bash shell, non-privileged -- **PackageManagers**: Auto-detected via `os_info`, fallback chain (apt→apk→dnf→pacman) -- **Engine**: Currently a unit struct with `execute_script()` method -- **All errors implement HasExitCode**: Enables uniform error propagation up to CLI - -## ANTI-PATTERNS -1. **Engine is a stub** — `types.rs:99` — `pub struct Engine;` with no fields, minimal implementation diff --git a/workshop-engine/src/error.rs b/workshop-engine/src/error.rs index 790bfb2..71fc209 100644 --- a/workshop-engine/src/error.rs +++ b/workshop-engine/src/error.rs @@ -21,8 +21,8 @@ pub enum ExecutionError { #[error("Invalid command: {0}")] InvalidCommand(String), - #[error("Execution failed: {0}")] - ExecutionFailed(String), + #[error("Task {task_id} failed with exit code {exit_code}")] + ExecutionFailed { task_id: String, exit_code: i32 }, #[error("Permission denied: uid={euid}, gid={egid}: {reason}")] PermissionDenied { @@ -48,7 +48,7 @@ impl HasExitCode for ExecutionError { fn exit_code(&self) -> i32 { match self { ExecutionError::InvalidCommand(_) => EXITCODE_EXECUTION_ERROR, - ExecutionError::ExecutionFailed(_) => EXITCODE_EXECUTION_ERROR, + ExecutionError::ExecutionFailed { .. } => EXITCODE_EXECUTION_ERROR, ExecutionError::PermissionDenied { .. } => EXITCODE_GENERAL_ERROR, ExecutionError::Timeout => EXITCODE_TIMEOUT, ExecutionError::IoError(_) => EXITCODE_IO_ERROR, diff --git a/workshop-engine/src/executor.rs b/workshop-engine/src/executor.rs index e7a781e..4d2b08b 100644 --- a/workshop-engine/src/executor.rs +++ b/workshop-engine/src/executor.rs @@ -13,7 +13,7 @@ use crate::types::{ DeltaExecutionContext, EventSender, ExecutionContext, ExecutionError, ExecutionEvent, ExecutionResult, StreamType, }; - +const STDOUT_BUF_CAP: usize = 2 * 1024 * 1024; impl crate::types::Engine { pub fn new() -> Self { Self @@ -159,11 +159,8 @@ impl crate::types::Engine { .await; } - let task_id = ctx.task_id.clone(); let (stdout_tx, mut stdout_rx) = mpsc::channel::(1024); let (stderr_tx, mut stderr_rx) = mpsc::channel::(1024); - let mut stdout_buf = String::new(); - let mut stderr_buf = String::new(); let mut child_stdout = child.take_stdout(); let mut child_stderr = child.take_stderr(); @@ -208,6 +205,32 @@ impl crate::types::Engine { } }); + // Forwarder: drains channels in real-time, emits OutputChunk events, + // and accumulates output into 2 MiB ring buffers. + let task_id_ev = ctx.task_id.clone(); + let event_tx_fwd = event_tx.clone(); + let forwarder = tokio::spawn(async move { + let mut stdout_buf: Vec = Vec::new(); + let mut stderr_buf: Vec = Vec::new(); + loop { + tokio::select! { + biased; + Some(data) = stdout_rx.recv() => { + emit_chunk(&event_tx_fwd, &task_id_ev, StreamType::Stdout, &data).await; + stdout_buf.extend_from_slice(data.as_bytes()); + truncate_buf(&mut stdout_buf); + } + Some(data) = stderr_rx.recv() => { + emit_chunk(&event_tx_fwd, &task_id_ev, StreamType::Stderr, &data).await; + stderr_buf.extend_from_slice(data.as_bytes()); + truncate_buf(&mut stderr_buf); + } + else => break, + } + } + (stdout_buf, stderr_buf) + }); + let exit_code = if let Some(timeout_duration) = ctx.timeout { match timeout(timeout_duration, child.wait()).await { Ok(Ok(code)) => code, @@ -236,12 +259,7 @@ impl crate::types::Engine { drop(stderr_tx); let _ = stdout_handle.await; let _ = stderr_handle.await; - while let Some(data) = stdout_rx.recv().await { - stdout_buf.push_str(&data); - } - while let Some(data) = stderr_rx.recv().await { - stderr_buf.push_str(&data); - } + let _ = forwarder.await; return Err(ExecutionError::Timeout); } } @@ -256,31 +274,9 @@ impl crate::types::Engine { let _ = stdout_handle.await; let _ = stderr_handle.await; - - while let Some(data) = stdout_rx.recv().await { - stdout_buf.push_str(&data); - if let Some(tx) = event_tx { - let _ = tx - .send(ExecutionEvent::OutputChunk { - task_id: task_id.clone(), - stream: StreamType::Stdout, - data: data.clone(), - }) - .await; - } - } - while let Some(data) = stderr_rx.recv().await { - stderr_buf.push_str(&data); - if let Some(tx) = event_tx { - let _ = tx - .send(ExecutionEvent::OutputChunk { - task_id: task_id.clone(), - stream: StreamType::Stderr, - data: data.clone(), - }) - .await; - } - } + let (stdout_buf, stderr_buf) = forwarder + .await + .map_err(|e| ExecutionError::IoError(format!("Forwarder task panicked: {}", e)))?; let duration = start_time.elapsed(); @@ -289,32 +285,26 @@ impl crate::types::Engine { exit_code, success: exit_code == 0, duration, - stdout: stdout_buf, - stderr: stderr_buf, + stdout: String::from_utf8_lossy(&stdout_buf).into_owned(), + stderr: String::from_utf8_lossy(&stderr_buf).into_owned(), }; if let Some(tx) = event_tx { - let event = if result.success { - ExecutionEvent::TaskCompleted { + let _ = tx + .send(ExecutionEvent::TaskCompleted { task_id: ctx.task_id.clone(), result: result.clone(), - } - } else { - ExecutionEvent::TaskFailed { - task_id: ctx.task_id.clone(), - error: format!("Exit code: {}", exit_code), - } - }; - let _ = tx.send(event).await; + }) + .await; } if result.success { Ok(result) } else { - Err(ExecutionError::ExecutionFailed(format!( - "Task {} failed with exit code {}", - ctx.task_id, exit_code - ))) + Err(ExecutionError::ExecutionFailed { + task_id: ctx.task_id.clone(), + exit_code, + }) } } @@ -352,6 +342,30 @@ impl crate::types::Engine { Ok(combined_result) } } +/// Truncate a buffer to [`STDOUT_BUF_CAP`] bytes, discarding the oldest data. +fn truncate_buf(buf: &mut Vec) { + if buf.len() > STDOUT_BUF_CAP { + buf.drain(0..buf.len() - STDOUT_BUF_CAP); + } +} + +/// Emit an `OutputChunk` event if the channel exists. +async fn emit_chunk( + tx: &EventSender, + task_id: &str, + stream: StreamType, + data: &str, +) { + if let Some(tx) = tx { + let _ = tx + .send(ExecutionEvent::OutputChunk { + task_id: task_id.to_string(), + stream, + data: data.to_string(), + }) + .await; + } +} impl Default for crate::types::Engine { fn default() -> Self { @@ -436,12 +450,12 @@ mod tests { assert!(result.is_err()); assert!(matches!( result.unwrap_err(), - ExecutionError::ExecutionFailed(_) + ExecutionError::ExecutionFailed { .. } )); } #[tokio::test] - async fn exit_nonzero_sends_task_failed_event() { + async fn exit_nonzero_sends_task_completed_event() { let engine = Engine::new(); let (tx, mut rx) = mpsc::channel(64); let child = MockChild::with_exit_code(42); @@ -451,9 +465,10 @@ mod tests { // Skip TaskStarted let _ = rx.recv().await; let event = rx.recv().await; - assert!(matches!(event, Some(ExecutionEvent::TaskFailed { .. }))); - if let Some(ExecutionEvent::TaskFailed { error, .. }) = event { - assert!(error.contains("42"), "error should mention exit code 42"); + assert!(matches!(event, Some(ExecutionEvent::TaskCompleted { .. }))); + if let Some(ExecutionEvent::TaskCompleted { result, .. }) = event { + assert_eq!(result.exit_code, 42); + assert!(!result.success); } } diff --git a/workshop-schedule/Cargo.lock b/workshop-schedule/Cargo.lock new file mode 100644 index 0000000..995dcab --- /dev/null +++ b/workshop-schedule/Cargo.lock @@ -0,0 +1,104 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "equivalent" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" + +[[package]] +name = "fixedbitset" +version = "0.5.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d674e81391d1e1ab681a28d99df07927c6d4aa5b027d7da16ba32d1d21ecd99" + +[[package]] +name = "hashbrown" +version = "0.17.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" + +[[package]] +name = "indexmap" +version = "2.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9" +dependencies = [ + "equivalent", + "hashbrown", +] + +[[package]] +name = "petgraph" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3672b37090dbd86368a4145bc067582552b29c27377cad4e0a306c97f9bd7772" +dependencies = [ + "fixedbitset", + "indexmap", +] + +[[package]] +name = "proc-macro2" +version = "1.0.106" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quote" +version = "1.0.45" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "syn" +version = "2.0.117" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "thiserror" +version = "2.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4288b5bcbc7920c07a1149a35cf9590a2aa808e0bc1eafaade0b80947865fbc4" +dependencies = [ + "thiserror-impl", +] + +[[package]] +name = "thiserror-impl" +version = "2.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebc4ee7f67670e9b64d05fa4253e753e016c6c95ff35b89b7941d6b856dec1d5" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "workshop-schedule" +version = "0.1.0" +dependencies = [ + "petgraph", + "thiserror", +] diff --git a/workshop-schedule/Cargo.toml b/workshop-schedule/Cargo.toml new file mode 100644 index 0000000..268c567 --- /dev/null +++ b/workshop-schedule/Cargo.toml @@ -0,0 +1,9 @@ +[package] +name = "workshop-schedule" +version = "0.1.0" +edition = "2024" +description = "Semi-general DAG scheduling library for CI/CD pipelines" + +[dependencies] +petgraph = "0.7" +thiserror = "2" diff --git a/workshop-schedule/src/dag.rs b/workshop-schedule/src/dag.rs new file mode 100644 index 0000000..fc358d3 --- /dev/null +++ b/workshop-schedule/src/dag.rs @@ -0,0 +1,780 @@ +// Code in this module PARTIALLY or FULLY utilized AI Coding Agent + +use petgraph::stable_graph::{NodeIndex, StableGraph}; +use petgraph::Direction; +use petgraph::visit::EdgeRef; +use std::collections::{BTreeMap, BTreeSet, HashMap}; +use thiserror::Error; + +use crate::edge::EdgeKind; +use crate::node::{DagNode, NodeColor}; + +/// Errors that can occur during DAG operations. +#[derive(Debug, Error)] +pub enum DagError { + #[error("duplicate node name: {0}")] + DuplicateNode(String), + + #[error("node not found: {0}")] + NodeNotFound(String), + + #[error("node not ready: {0}")] + NotReady(String), + + #[error("already popped: {0}")] + AlreadyPopped(String), + + #[error("circular dependency detected among: {0:?}")] + Cycle(Vec), + + #[error("parallel group can never complete: {0:?}")] + GroupDeadlock(Vec), +} + +/// Internal per-node metadata. +enum NodeState { + /// Has unsatisfied dependencies. + Pending, + /// All dependencies satisfied, in the ready set. + Ready, + /// Has been popped — done executing. + Done, +} + +/// Internal per-group metadata. +#[derive(Debug)] +struct GroupMeta { + /// Node indices belonging to this group. + members: Vec, + /// Total number of members (constant after construction). + total: usize, +} + +/// A DAG (Directed Acyclic Graph) for scheduling pipeline nodes. +/// +/// Nodes implement [`DagNode`] and carry a [`NodeColor`] that determines +/// sequential vs parallel semantics. Edges carry an [`EdgeKind`] distinguishing +/// explicit `@after` from implicit pipeline chains. +/// +/// The DAG maintains a ready set (nodes with in-degree 0) in deterministic +/// BTreeSet order, and provides methods to pop nodes **by name** — fixing the +/// limitation of `topological-sort` that could only pop the first ready node. +pub struct Dag { + graph: StableGraph, + name_to_idx: HashMap, + state: HashMap, + in_degree: HashMap, + /// Ready nodes, sorted by name for deterministic iteration. + ready_set: BTreeSet<(String, NodeIndex)>, + /// Named groups and their members. + groups: BTreeMap, + node_count: usize, +} + +impl Dag { + /// Create an empty DAG. + pub fn new() -> Self { + Self { + graph: StableGraph::new(), + name_to_idx: HashMap::new(), + state: HashMap::new(), + in_degree: HashMap::new(), + ready_set: BTreeSet::new(), + groups: BTreeMap::new(), + node_count: 0, + } + } + + /// Number of nodes (including pending, ready, and done). + pub fn node_count(&self) -> usize { + self.node_count + } + + /// True if no nodes remain in the DAG (all popped). + pub fn is_empty(&self) -> bool { + self.ready_set.is_empty() + && self.state.values().all(|s| matches!(s, NodeState::Done)) + } + + // ═══════════════════════════════════════════════════════════════════ + // Construction + // ═══════════════════════════════════════════════════════════════════ + + /// Add a node to the DAG. + /// + /// Returns an error if a node with the same name already exists. + pub fn add_node(&mut self, node: N) -> Result<(), DagError> { + let name = node.name().to_string(); + if self.name_to_idx.contains_key(&name) { + return Err(DagError::DuplicateNode(name)); + } + + let color = node.color(); // extract before move + let idx = self.graph.add_node(node); + self.name_to_idx.insert(name.clone(), idx); + self.state.insert(idx, NodeState::Ready); + self.in_degree.insert(idx, 0); + self.ready_set.insert((name.clone(), idx)); + self.node_count += 1; + + // Register named groups + if let NodeColor::Named(group) = &color { + self.groups + .entry(group.clone()) + .or_insert_with(|| GroupMeta { + members: Vec::new(), + total: 0, + }) + .members + .push(idx); + } + + Ok(()) + } + + /// Freeze group totals (call after all nodes added, before any pop). + /// + /// Locks in `total` for each group. Named groups whose `total` is less + /// than 2 are harmless but wasteful — they'll fire solo like sequential. + pub fn freeze(&mut self) { + for meta in self.groups.values_mut() { + meta.total = meta.members.len(); + } + } + + /// Add a dependency edge from `from` to `to`. + /// + /// `to` depends on `from` — `to` won't be ready until `from` is popped. + pub fn add_edge( + &mut self, + from: &str, + to: &str, + kind: EdgeKind, + ) -> Result<(), DagError> { + let from_idx = self + .name_to_idx + .get(from) + .copied() + .ok_or_else(|| DagError::NodeNotFound(from.to_string()))?; + let to_idx = self + .name_to_idx + .get(to) + .copied() + .ok_or_else(|| DagError::NodeNotFound(to.to_string()))?; + + self.graph.add_edge(from_idx, to_idx, kind); + + // Increment in-degree of `to` + let deg = self.in_degree.get_mut(&to_idx).unwrap(); + *deg += 1; + + // Remove `to` from ready set if it was there + let to_name = self.graph[to_idx].name().to_string(); + self.ready_set.remove(&(to_name.clone(), to_idx)); + *self.state.get_mut(&to_idx).unwrap() = NodeState::Pending; + + // Sanity: check for self-loop (cycle detection lite) + if from == to { + return Err(DagError::Cycle(vec![from.to_string()])); + } + + Ok(()) + } + + // ═══════════════════════════════════════════════════════════════════ + // Queries + // ═══════════════════════════════════════════════════════════════════ + + /// Name → color lookup. O(1). + pub fn color_of(&self, name: &str) -> Option { + let idx = self.name_to_idx.get(name)?; + Some(self.graph[*idx].color()) + } + + /// True if the node is sequential (not parallel). O(1). + pub fn is_sequential(&self, name: &str) -> bool { + self.color_of(name) + .map(|c| c.is_sequential()) + .unwrap_or(false) + } + + /// True if the node is a wildcard parallel. O(1). + pub fn is_wildcard(&self, name: &str) -> bool { + self.color_of(name) + .map(|c| c.is_wildcard()) + .unwrap_or(false) + } + + /// True if the node is parallel (wildcard or named). O(1). + pub fn is_parallel(&self, name: &str) -> bool { + self.color_of(name) + .map(|c| c.is_parallel()) + .unwrap_or(false) + } + + /// Group name for a named parallel node, or `None`. O(1). + pub fn group_of(&self, name: &str) -> Option { + let idx = self.name_to_idx.get(name)?; + self.graph[*idx].color().group_name().map(|s| s.to_string()) + } + + /// In-degree of a node. O(1). + pub fn in_degree(&self, name: &str) -> Option { + let idx = self.name_to_idx.get(name)?; + self.in_degree.get(idx).copied() + } + + /// Get the edge kind between two nodes. O(e). + pub fn edge_kind(&self, from: &str, to: &str) -> Option { + let from_idx = self.name_to_idx.get(from)?; + let to_idx = self.name_to_idx.get(to)?; + self.graph + .find_edge(*from_idx, *to_idx) + .map(|e| self.graph[e]) + } + + /// Reference to the node data. O(1). + pub fn node(&self, name: &str) -> Option<&N> { + let idx = self.name_to_idx.get(name)?; + Some(&self.graph[*idx]) + } + + /// True if the node is fallible. + pub fn is_fallible(&self, name: &str) -> bool { + self.node(name).map(|n| n.fallible()).unwrap_or(false) + } + + /// Number of ready nodes (in-degree 0, not yet popped). + pub fn ready_count(&self) -> usize { + self.ready_set.len() + } + + // ═══════════════════════════════════════════════════════════════════ + // Ready set queries + // ═══════════════════════════════════════════════════════════════════ + + /// All ready node names, in deterministic BTreeSet order. + pub fn ready_names(&self) -> Vec<&str> { + self.ready_set + .iter() + .map(|(name, _)| name.as_str()) + .collect() + } + + /// Ready sequential node names. + pub fn ready_sequential(&self) -> Vec<&str> { + self.ready_set + .iter() + .filter(|(name, _)| self.is_sequential(name)) + .map(|(name, _)| name.as_str()) + .collect() + } + + /// Ready wildcard node names. + pub fn ready_wildcards(&self) -> Vec<&str> { + self.ready_set + .iter() + .filter(|(name, _)| self.is_wildcard(name)) + .map(|(name, _)| name.as_str()) + .collect() + } + + /// Ready nodes grouped by their parallel group name. + /// + /// Returns `(group_name, ready_count, total_count)` for each group that + /// has at least one ready member. + pub fn ready_groups(&self) -> Vec<(&str, usize, usize)> { + let mut result = Vec::new(); + for (gname, meta) in &self.groups { + let ready_count = meta + .members + .iter() + .filter(|idx| matches!(self.state.get(idx), Some(NodeState::Ready))) + .count(); + if ready_count > 0 { + result.push((gname.as_str(), ready_count, meta.total)); + } + } + result + } + + /// Pick a fireable batch per bake scheduling rules: + /// + /// 1. Named groups: flood-fill from ready members through same-color + /// implicit edges. If the whole group is reachable, fire it. + /// 2. Wildcards join any fireable group. + /// 3. Otherwise, wildcards form their own batch. + /// 4. Returns `None` only when truly stuck. + pub fn pick_batch(&self) -> Option> { + use std::collections::{HashSet, VecDeque}; + + let wildcards: Vec = self + .ready_set + .iter() + .filter(|(name, _)| self.is_wildcard(name)) + .map(|(name, _)| name.clone()) + .collect(); + + for (_gname, meta) in &self.groups { + if meta.total == 0 { + continue; + } + + // Flood-fill from ready members through same-color implicit edges + let mut reached: HashSet = HashSet::new(); + let mut queue: VecDeque = VecDeque::new(); + + for idx in &meta.members { + if matches!(self.state.get(idx), Some(NodeState::Ready)) { + reached.insert(*idx); + queue.push_back(*idx); + } + } + + while let Some(from) = queue.pop_front() { + let from_color = self.graph[from].color(); + if from_color.is_sequential() { + continue; + } + + for edge in self.graph.edges_directed(from, Direction::Outgoing) { + if !edge.weight().is_implicit() { + continue; + } + let to = edge.target(); + let to_color = self.graph[to].color(); + if to_color.is_sequential() { + continue; + } + + let same = match (&from_color, &to_color) { + (NodeColor::Wildcard, NodeColor::Wildcard) => true, + (NodeColor::Named(a), NodeColor::Named(b)) => a.as_str() == b.as_str(), + _ => false, + }; + if !same { + continue; + } + if !meta.members.contains(&to) { + continue; + } + + if reached.insert(to) { + queue.push_back(to); + } + } + } + + if reached.len() == meta.total { + let batch: Vec = meta + .members + .iter() + .map(|idx| self.graph[*idx].name().to_string()) + .collect(); + let mut batch = batch; + batch.extend(wildcards); + return Some(batch); + } + } + + // No fireable named group — wildcards fire alone + if !wildcards.is_empty() { + return Some(wildcards); + } + + None + } + // ═══════════════════════════════════════════════════════════════════ + // Scheduling operations + // ═══════════════════════════════════════════════════════════════════ + + /// Pop a specific node by name. + /// + /// Decrements in-degree of all successors. Those reaching 0 become ready. + /// This is the key fix over `topological-sort` — we pop by name, not by + /// arbitrary HashMap iteration order. + pub fn pop(&mut self, name: &str) -> Result<(), DagError> { + let idx = self + .name_to_idx + .get(name) + .copied() + .ok_or_else(|| DagError::NodeNotFound(name.to_string()))?; + + match self.state.get(&idx) { + None => return Err(DagError::NodeNotFound(name.to_string())), + Some(NodeState::Done) => return Err(DagError::AlreadyPopped(name.to_string())), + Some(NodeState::Pending) => return Err(DagError::NotReady(name.to_string())), + Some(NodeState::Ready) => {} + } + + // Remove from ready set + let owned_name = name.to_string(); + self.ready_set.remove(&(owned_name, idx)); + + // Decrement in-degree of all successors + let successors: Vec = self + .graph + .neighbors_directed(idx, Direction::Outgoing) + .collect(); + for succ in successors { + let deg = self.in_degree.get_mut(&succ).unwrap(); + *deg -= 1; + if *deg == 0 { + // Successor becomes ready + let succ_name = self.graph[succ].name().to_string(); + self.ready_set.insert((succ_name, succ)); + *self.state.get_mut(&succ).unwrap() = NodeState::Ready; + } + } + + // Mark as done + *self.state.get_mut(&idx).unwrap() = NodeState::Done; + + Ok(()) + } + + /// Pop a sequential node from the ready set. + /// + /// Returns the name of the popped node, or `None` if no sequential + /// node is ready. Convenience wrapper: finds first ready sequential, + /// pops it by name. + pub fn pop_sequential(&mut self) -> Result, DagError> { + let name = self + .ready_set + .iter() + .find(|(name, _)| self.is_sequential(name)) + .map(|(name, _)| name.clone()); + + if let Some(ref name) = name { + self.pop(name)?; + } + + Ok(name) + } + + // ═══════════════════════════════════════════════════════════════════ + // Deadlock / cycle detection + // ═══════════════════════════════════════════════════════════════════ + + /// True if ready nodes exist but no batch can fire (potential deadlock). + pub fn is_stalled(&self) -> bool { + !self.ready_set.is_empty() && self.pick_batch().is_none() + } + + /// Names of groups that are causing a stall (incomplete groups, + /// including those with zero ready members but non-Done pending members). + pub fn stalled_groups(&self) -> Vec<&str> { + let mut result = Vec::new(); + for (gname, meta) in &self.groups { + let ready_count = meta + .members + .iter() + .filter(|idx| matches!(self.state.get(idx), Some(NodeState::Ready))) + .count(); + let pending_count = meta + .members + .iter() + .filter(|idx| matches!(self.state.get(idx), Some(NodeState::Pending))) + .count(); + if meta.total > 0 && (ready_count > 0 || pending_count > 0) && ready_count < meta.total { + result.push(gname.as_str()); + } + } + result + } + + /// All nodes that haven't been popped yet (for cycle diagnostics). + pub fn remaining(&self) -> Vec<&str> { + self.state + .iter() + .filter(|(_, s)| !matches!(s, NodeState::Done)) + .map(|(idx, _)| self.graph[*idx].name()) + .collect() + } +} + +impl Default for Dag { + fn default() -> Self { + Self::new() + } +} + +// ── Tests ────────────────────────────────────────────────────────────────── + +#[cfg(test)] +mod tests { + use super::*; + + #[derive(Clone)] + struct TestNode { + name: String, + color: NodeColor, + fallible: bool, + } + + impl DagNode for TestNode { + fn name(&self) -> &str { + &self.name + } + fn color(&self) -> NodeColor { + self.color.clone() + } + fn fallible(&self) -> bool { + self.fallible + } + } + + fn seq(name: &str) -> TestNode { + TestNode { + name: name.into(), + color: NodeColor::Sequential, + fallible: false, + } + } + + fn wild(name: &str) -> TestNode { + TestNode { + name: name.into(), + color: NodeColor::Wildcard, + fallible: false, + } + } + + fn group(name: &str, g: &str) -> TestNode { + TestNode { + name: name.into(), + color: NodeColor::Named(g.into()), + fallible: false, + } + } + + // ── Basic construction ──────────────────────────────────────────── + + #[test] + fn empty_dag() { + let dag = Dag::::new(); + assert!(dag.is_empty()); + assert_eq!(dag.node_count(), 0); + } + + #[test] + fn add_single_node() { + let mut dag = Dag::new(); + dag.add_node(seq("a")).unwrap(); + dag.freeze(); + assert!(!dag.is_empty()); + assert_eq!(dag.ready_count(), 1); + assert_eq!(dag.ready_names(), vec!["a"]); + assert!(dag.is_sequential("a")); + } + + #[test] + fn duplicate_node_rejected() { + let mut dag = Dag::new(); + dag.add_node(seq("a")).unwrap(); + assert!(dag.add_node(seq("a")).is_err()); + } + + // ── Edges & in-degree ───────────────────────────────────────────── + + #[test] + fn simple_dependency() { + let mut dag = Dag::new(); + dag.add_node(seq("a")).unwrap(); + dag.add_node(seq("b")).unwrap(); + dag.add_edge("a", "b", EdgeKind::Explicit).unwrap(); + dag.freeze(); + + assert!(dag.is_sequential("a")); + assert_eq!(dag.ready_names(), vec!["a"]); + assert_eq!(dag.in_degree("a"), Some(0)); + assert_eq!(dag.in_degree("b"), Some(1)); + } + + #[test] + fn pop_advances_dependents() { + let mut dag = Dag::new(); + dag.add_node(seq("a")).unwrap(); + dag.add_node(seq("b")).unwrap(); + dag.add_node(seq("c")).unwrap(); + dag.add_edge("a", "b", EdgeKind::Explicit).unwrap(); + dag.add_edge("b", "c", EdgeKind::Explicit).unwrap(); + dag.freeze(); + + assert_eq!(dag.ready_names(), vec!["a"]); + dag.pop("a").unwrap(); + assert_eq!(dag.ready_names(), vec!["b"]); + dag.pop("b").unwrap(); + assert_eq!(dag.ready_names(), vec!["c"]); + dag.pop("c").unwrap(); + assert!(dag.is_empty()); + } + + #[test] + fn pop_not_ready_rejected() { + let mut dag = Dag::new(); + dag.add_node(seq("a")).unwrap(); + dag.add_node(seq("b")).unwrap(); + dag.add_edge("a", "b", EdgeKind::Explicit).unwrap(); + dag.freeze(); + + assert!(dag.pop("b").is_err()); // b not ready + } + + #[test] + fn pop_twice_rejected() { + let mut dag = Dag::new(); + dag.add_node(seq("a")).unwrap(); + dag.freeze(); + dag.pop("a").unwrap(); + assert!(dag.pop("a").is_err()); // already done + } + + // ── Node coloring ───────────────────────────────────────────────── + + #[test] + fn color_queries() { + let mut dag = Dag::new(); + dag.add_node(seq("s")).unwrap(); + dag.add_node(wild("w")).unwrap(); + dag.add_node(group("g", "build")).unwrap(); + dag.freeze(); + + assert!(dag.is_sequential("s")); + assert!(dag.is_wildcard("w")); + assert_eq!(dag.group_of("g"), Some("build".to_string())); + assert!(dag.is_parallel("w")); + assert!(dag.is_parallel("g")); + assert!(!dag.is_parallel("s")); + } + + #[test] + fn fallible_query() { + let mut dag = Dag::new(); + dag.add_node(TestNode { + name: "f".into(), + color: NodeColor::Sequential, + fallible: true, + }) + .unwrap(); + dag.freeze(); + assert!(dag.is_fallible("f")); + } + + // ── Ready set queries ───────────────────────────────────────────── + + #[test] + fn ready_filtering() { + let mut dag = Dag::new(); + dag.add_node(seq("s1")).unwrap(); + dag.add_node(seq("s2")).unwrap(); + dag.add_node(wild("w1")).unwrap(); + dag.add_node(wild("w2")).unwrap(); + dag.add_node(group("g1", "build")).unwrap(); + dag.add_node(group("g2", "build")).unwrap(); + dag.freeze(); + + assert_eq!(dag.ready_sequential(), vec!["s1", "s2"]); + assert_eq!(dag.ready_wildcards(), vec!["w1", "w2"]); + let groups = dag.ready_groups(); + assert_eq!(groups.len(), 1); + assert_eq!(groups[0], ("build", 2, 2)); + } + + // ── Batch picking ───────────────────────────────────────────────── + + #[test] + fn pick_batch_complete_group_with_wildcards() { + let mut dag = Dag::new(); + dag.add_node(seq("s")).unwrap(); // not ready (has dep) + dag.add_node(group("g1", "build")).unwrap(); + dag.add_node(group("g2", "build")).unwrap(); + dag.add_node(wild("w")).unwrap(); + dag.add_edge("s", "g1", EdgeKind::Explicit).unwrap(); // block g1 + dag.freeze(); + + // g1 blocked by s, build group incomplete + // BUT wildcard w is ready → fires as its own batch + let batch = dag.pick_batch().unwrap(); + assert_eq!(batch.len(), 1); + assert!(batch.contains(&"w".to_string())); + + dag.pop("w").unwrap(); + dag.pop("s").unwrap(); + // Now g1, g2 both ready; build group complete + let batch = dag.pick_batch().unwrap(); + assert!(batch.contains(&"g1".to_string())); + assert!(batch.contains(&"g2".to_string())); + } + #[test] + fn pick_batch_wildcards_only_no_groups() { + let mut dag = Dag::new(); + dag.add_node(wild("w1")).unwrap(); + dag.add_node(wild("w2")).unwrap(); + dag.freeze(); + + let batch = dag.pick_batch().unwrap(); + assert_eq!(batch.len(), 2); + assert!(batch.contains(&"w1".to_string())); + assert!(batch.contains(&"w2".to_string())); + } + + #[test] + fn pick_batch_none_when_sequential_only() { + let mut dag = Dag::new(); + dag.add_node(seq("s1")).unwrap(); + dag.add_node(seq("s2")).unwrap(); + dag.freeze(); + + // pick_batch only picks parallel batches; sequential nodes + // should be picked individually via pop_sequential() + assert!(dag.pick_batch().is_none()); + } + + // ── pop_sequential convenience ──────────────────────────────────── + + #[test] + fn pop_sequential_picks_first_seq() { + let mut dag = Dag::new(); + dag.add_node(wild("w")).unwrap(); + dag.add_node(seq("s1")).unwrap(); + dag.add_node(seq("s2")).unwrap(); + dag.freeze(); + + let popped = dag.pop_sequential().unwrap().unwrap(); + assert_eq!(popped, "s1"); // BTreeSet order: s1 before s2 + assert!(!dag.ready_names().contains(&"s1")); + assert!(dag.ready_names().contains(&"s2")); + } + + // ── Deadlock detection ──────────────────────────────────────────── + + #[test] + fn stall_when_group_incomplete() { + let mut dag = Dag::new(); + dag.add_node(seq("s")).unwrap(); + dag.add_node(group("g1", "build")).unwrap(); + dag.add_node(group("g2", "build")).unwrap(); + dag.add_edge("s", "g2", EdgeKind::Explicit).unwrap(); + dag.freeze(); + + // g1 ready, g2 not (depends on s). pick_batch → none. + assert!(dag.is_stalled()); + assert_eq!(dag.stalled_groups(), vec!["build"]); + } + + #[test] + fn remaining_after_partial_pop() { + let mut dag = Dag::new(); + dag.add_node(seq("a")).unwrap(); + dag.add_node(seq("b")).unwrap(); + dag.add_node(seq("c")).unwrap(); + dag.freeze(); + + dag.pop("a").unwrap(); + let rem = dag.remaining(); + assert_eq!(rem.len(), 2); + assert!(rem.contains(&"b")); + assert!(rem.contains(&"c")); + } +} diff --git a/workshop-schedule/src/edge.rs b/workshop-schedule/src/edge.rs new file mode 100644 index 0000000..aa28e83 --- /dev/null +++ b/workshop-schedule/src/edge.rs @@ -0,0 +1,20 @@ +// Code in this module PARTIALLY or FULLY utilized AI Coding Agent + +/// Edge kind — distinguishes explicit `@after` from implicit pipeline chaining. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +pub enum EdgeKind { + /// Explicit dependency: `@after(func)` in the script. + Explicit, + /// Implicit dependency: consecutive `@pipeline` functions without `@after`. + Implicit, +} + +impl EdgeKind { + pub fn is_explicit(&self) -> bool { + matches!(self, EdgeKind::Explicit) + } + + pub fn is_implicit(&self) -> bool { + matches!(self, EdgeKind::Implicit) + } +} diff --git a/workshop-schedule/src/lib.rs b/workshop-schedule/src/lib.rs new file mode 100644 index 0000000..c29af59 --- /dev/null +++ b/workshop-schedule/src/lib.rs @@ -0,0 +1,9 @@ +// Code in this module PARTIALLY or FULLY utilized AI Coding Agent + +pub mod dag; +pub mod edge; +pub mod node; + +pub use dag::{Dag, DagError}; +pub use edge::EdgeKind; +pub use node::{DagNode, NodeColor}; diff --git a/workshop-schedule/src/node.rs b/workshop-schedule/src/node.rs new file mode 100644 index 0000000..30e0646 --- /dev/null +++ b/workshop-schedule/src/node.rs @@ -0,0 +1,60 @@ +// Code in this module PARTIALLY or FULLY utilized AI Coding Agent + +/// Node coloring — mirrors the bake DAG generator's color semantics. +/// +/// | Color | Probability | Bake semantics | +/// |-------------|------------|-----------------------------| +/// | Sequential | p_seq | Runs alone, one at a time | +/// | Wildcard | p_wild | Joins any complete group | +/// | Named(G) | p_named/K | Fires when group complete | +#[derive(Debug, Clone, PartialEq, Eq, Hash)] +pub enum NodeColor { + /// Sequential node — executes alone. + Sequential, + /// Wildcard parallel — compatible with any complete named group. + Wildcard, + /// Named parallel group member — fires when all members of this group are ready. + Named(String), +} + +impl NodeColor { + /// Returns true for `Sequential`. + pub fn is_sequential(&self) -> bool { + matches!(self, NodeColor::Sequential) + } + + /// Returns true for `Wildcard`. + pub fn is_wildcard(&self) -> bool { + matches!(self, NodeColor::Wildcard) + } + + /// Returns the group name if `Named`, or `None`. + pub fn group_name(&self) -> Option<&str> { + match self { + NodeColor::Named(g) => Some(g.as_str()), + _ => None, + } + } + + /// Returns true if this is a parallel node (Wildcard or Named). + pub fn is_parallel(&self) -> bool { + !self.is_sequential() + } +} + +/// Trait that nodes must implement to be managed by [`Dag`](super::dag::Dag). +/// +/// Only `name()` and `color()` are required. Override `fallible()` if the +/// node can fail without halting the pipeline. +pub trait DagNode { + /// Unique name of this node within the DAG. + fn name(&self) -> &str; + + /// Scheduling color — determines sequential/parallel/group semantics. + fn color(&self) -> NodeColor; + + /// Whether failure of this node is non-fatal to the pipeline. + fn fallible(&self) -> bool { + false + } +}