Compare commits

5 Commits

Author SHA1 Message Date
Catty Steve fdc00f0635 feat(baker): add debug feature flags and resource prefetching
Add DebugFeature bitflags for controlling notify.dummy, events,
scheduler,
and script debug behaviors. Add prefetch module to fetch plugins and
templates before pipeline runs. Add cleanup and artifact handling to
finalize stage. Add build_id to ExecutionContext.
2026-06-17 20:43:17 +08:00
Catty Steve 26809df720 feat(bake): DAG scheduler, decorator execution, notify refactor, workshop-schedule crate
- Add workshop-schedule crate (petgraph-based DAG scheduling)
- Implement all bake decorators with combinator chain execution
- Refactor notify module: extract handler, rule, template, util, queue
- Add NotificationQueue with priority/drain semantics
- Enhance finalize plugin system with internal plugin registration
- Update ExecutionContext and event types
- Add per-crate AGENTS.md knowledge base files
- Remove inline AGENTS.md files (consolidated into per-crate docs)
2026-06-15 20:45:06 +08:00
Catty Steve 3629c33fc4 feat(bake): support daemon, health and async decorator
Add `@async` decorator for fire-and-forget tasks, `@pipe` for stdout
routing between functions, and `@daemon` with configurable health
probes. Functions now resolve execution mode and track stdout cache for
inter-function communication.
2026-06-11 16:26:44 +08:00
Catty Steve 6d9ddbca56 refactor(bake/baker): tidy up bake.rs 2026-06-09 23:36:58 +08:00
Catty Steve 9ff9073fce feat(baker/bake): implement all available decorators 2026-06-09 23:09:57 +08:00
70 changed files with 4181 additions and 1477 deletions
+97 -62
View File
@@ -1,92 +1,127 @@
# PROJECT KNOWLEDGE BASE
**Generated:** 2026-05-19
**Commit:** 28abc5d
**Branch:** master
**Generated:** 2026-06-12T16:08:15Z
**Commit:** 3629c33
**Branch:** feat/bake-dag-tester
## OVERVIEW
HoneyBiscuitWorkshop ("蜜饼工坊") — Rust CI/CD system with decorator-driven shell pipeline DSL and LLM-powered configuration. 3 independent Cargo crates, no workspace.
HoneyBiscuitWorkshop (蜜饼工坊) — Rust-based CI/CD system inspired by Concourse CI. Pipeline: prebake.yml → bake.sh → finalize.yml. Multi-builder (Docker, Firecracker, bare metal). Plugin system.
## STRUCTURE
```
./
├── workshop-baker/ # Main orchestrator (CLI + pipeline stages)
├── workshop-engine/ # Execution engine (extracted from baker)
├── workshop-getterurl/ # URL fetching (go-getter style)
├── docs/ # mdBook documentation (zh-CN)
├── templates/ # Pipeline templates (bake.sh, finalize.yml, prebake.yml)
── archived/ # Deprecated crates (7 moved here)
├── workshop-baker/ # Main binary + lib — orchestrator, CLI, plugins
├── workshop-engine/ # Core execution engine — process mgmt, package managers
├── workshop-schedule/ # DAG scheduling library (petgraph-based)
├── workshop-getterurl/ # URL fetching with hash verification
├── workshop-baker-params/ # Config/parameter type definitions
── docs/ # mdBook documentation (Chinese, 3 volumes)
├── templates/ # Pipeline config templates (prebake/bake/finalize)
└── .github/workflows/ # CI (tests workshop-baker + workshop-engine)
```
**No root Cargo workspace** — crates linked by path deps. Build per-crate.
## WHERE TO LOOK
| Task | Location | Notes |
|------|----------|-------|
| Baker (CLI) | `workshop-baker/src/{lib,main}.rs` | Clap CLI + tokio entry |
| Pipeline stages | `workshop-baker/src/{bake,prebake,finalize}/` | Stage orchestration |
| Notifications | `workshop-baker/src/notify/` | Plugin-based event notifications |
|| Engine (execution) | `workshop-engine/src/` | Process mgmt, PM detection |
| URL fetching | `workshop-getterurl/src/` | Git/HTTP/File resource getter |
| CLI definition | `workshop-baker/src/lib.rs` | Clap Cli/Commands/BareCommands |
| Pipeline orchestration | `workshop-baker/src/main.rs` | prebake→bake→finalize→notify flow |
| Decorator parsing | `workshop-baker/src/bake/decorator.rs` | `# @decorator(args)` syntax |
| Build execution | `workshop-baker/src/bake/execute.rs` | Script execution with timeout/retry |
| Docker integration | `workshop-baker/src/prebake/env/container.rs` | Bollard-based |
| Package managers | `workshop-engine/src/pm/` | apt, pacman, dnf, apk |
| User package managers | `workshop-engine/src/upm/` | rustup, uv, go |
| Notification system | `workshop-baker/src/notify/` | Email, webhook, IM |
| Plugin system | `workshop-baker/src/finalize/plugin/` | shell, dylib, rhai, internal |
| Error types | `workshop-baker/src/error.rs` | CliError + HasExitCode |
| Exit codes | `workshop-engine/src/error.rs` | EXITCODE_* constants |
| DAG scheduling | `workshop-schedule/src/dag.rs` | Topological sort, batch picking |
| Pipeline templates | `templates/` | Reference config DSL |
## ENTRY POINTS
| Entry | Type | Path | Role |
|-------|------|------|------|
| `workshop-baker` | Rust bin | `workshop-baker/src/main.rs` | Primary CLI — dispatch: default (full pipeline), bare (single stage), daemon (unimplemented) |
| `workshop_baker` | Rust lib | `workshop-baker/src/lib.rs` | Baker library + CLI structs |
| `workshop_engine` | Rust lib | `workshop-engine/src/lib.rs` | Core engine — ExecutionContext, EventSender, Executor |
| `workshop_schedule` | Rust lib | `workshop-schedule/src/lib.rs` | Dag, DagNode, EdgeKind |
| `workshop_baker_params` | Rust lib | `workshop-baker-params/src/lib.rs` | Layered config: defaults → base → courier |
| `workshop_getterurl` | Rust lib | `workshop-getterurl/src/lib.rs` | GetterUrl, fetch() — http/git/file |
| docs | mdBook | `docs/book.toml` | Project documentation site |
## CODE MAP
| Symbol | Type | Location | Role |
|--------|------|----------|------|
| Cli | struct | workshop-baker/src/lib.rs:21 | CLI argument struct |
| ExecutionContext | struct | workshop-engine/src/types.rs:10 | Pipeline context |
| Engine | struct | workshop-engine/src/types.rs:99 | Build executor (placeholder) |
| PrebakeConfig | struct | workshop-baker/src/prebake/config.rs | Prebake YAML schema |
| FinalizeConfig | struct | workshop-baker/src/finalize/config.rs:21 | Finalize YAML schema |
| GetterUrl | struct | workshop-getterurl/src/lib.rs:84 | Parsed resource URL |
| Getter | trait | workshop-getterurl/src/getter.rs:18 | Resource fetch trait |
### Crate Dependency Graph
```
workshop-baker (bin)
├── workshop-engine
├── workshop-schedule
├── workshop-baker-params
└── workshop-getterurl
```
### workshop-baker Module Map
| Module | Submodules | Role |
|--------|-----------|------|
| `bake` | decorator, execute, parser, schedule, builder, trivial, util, constant, error | Script parsing + execution |
| `prebake` | config, stage (bootstrap, depssystem, depsuser, environment, hook), env (container, firecracker, baremetal, custom), security, event, types, constant, error | Environment setup |
| `finalize` | plugin (shell, dylib, rhai, internal), stage, template, types, config, constant, error, event | Post-build hooks |
| `notify` | handler, queue, rule, template, types, util, error | Async notification dispatch |
| `bare` | — | Standalone single-stage CLI mode |
| `types` | resource, buildstatus | Shared type definitions |
| `utils` | — | Cross-cutting utilities |
## CONVENTIONS
### Rust
- **Edition**: 2024 (requires Rust 1.85+), deprecates `mod.rs`
- **Naming**: `snake_case` files/modules, `PascalCase` types, `SCREAMING_SNAKE_CASE` consts
- **Error handling**: `thiserror` + `anyhow` combo; `HasExitCode` trait for exit codes
- **Async**: `#[tokio::main]` + `tokio` with "full" features
- **Imports**: `std``external_crate``crate::module`
- **Dual-mode**: CLI commands (prebake/bake/finalize) + daemon mode (unimplemented)
### Testing
- **Rust**: `#[test]` inline, `#[tokio::test]` async, `tests/` integration
- **Benchmarks**: Criterion (configured in Cargo.toml)
### CI
- **GitHub Actions**: rust-toolchain (stable), clippy + rustfmt, `cargo test --all-features`
- **Matrix**: workshop-baker, workshop-engine
- **Edition**: Rust 2024 (requires Rust 1.85+)
- **Formatting**: Default `rustfmt` — no `rustfmt.toml` / `.editorconfig`
- **Lint overrides** (workshop-baker only): `dead_code = "allow"`, `unreachable_code = "allow"`, `inherent_to_string = "allow"`, `non_canonical_partial_ord_impl = "allow"`
- **Import order**: `std` → external crates → `crate::` (blank line between groups)
- **Error handling**: `thiserror` for libraries, `anyhow` for binaries. Implement `HasExitCode` for exit code mapping.
- **Type system**: Newtype pattern preferred (e.g., `MemSize(u64)`). Shared types in `types/` module. `types/` must NOT depend on `bake/`, `engine/`, `prebake/`, `finalize/`.
- **Tests**: Inline `#[cfg(test)] mod tests` at bottom of source files. Async: `#[tokio::test]`. CI runs `cargo test --all-features`.
- **Commits**: `type(scope): description``feat|fix|docs|test|refactor|style`
- **AI annotation**: `// Code in this module PARTIALLY or FULLY utilized AI Coding Agent` — never delete if present
- **Docs language**: Chinese (zh-CN), mdBook
- **No `no_std`**: Explicitly unsupported
## ANTI-PATTERNS (THIS PROJECT)
1. **Hardcoded user** — "vulcan" username hardcoded throughout prebake
2. **Unsafe libc** — workshop-baker/src/prebake/security.rs has 5+ unsafe blocks
3. **Rust 2024 edition** — May not work with stable toolchains
4. **Empty daemon**`daemon/` dir is empty, Daemon command is `unimplemented!()`
## UNIQUE STYLES
- **Pipeline DSL**: Shell scripts with `# @decorator` annotations
- **Notification system**: Plugin-based with batching, retry, priority groups
- **LLM cookbook system**: YAML cookbooks for language-specific builds (archived)
- **Chinese docs** — Documentation primarily in zh-CN
- **Do NOT derive production behavior from bare mode** — bare mode is dev-only, ctx.privileged is fake
- **Do NOT use `detect()`/`adopt()` PM methods in production** — PM selection must come from env config
- **`drop_after < DepsUser` is FORBIDDEN** — will break system dependency installation (needs root)
- **Do NOT add magic URL inference** — URL is URL, behavior bound explicitly via `getter::` prefix
- **Do NOT use `@` for version locking in URLs** — it's an auth delimiter; use `?ref=`
- **Do NOT duplicate type definitions across modules** — single source of truth in `types/`
- **Careful with `unimplemented!()`** — Daemon mode, Rhai/Dylib plugins, and Custom builder are stubs
- **Security**: PIPELINE_CONTAINER_PRIVILEGED, PIPELINE_BAREMETAL_ELEVATE, PIPELINE_UNCOVER_SECRET require explicit approval + audit
## COMMANDS
```bash
# Build
cargo build --release -p workshop-baker
# Build (per crate, no workspace)
cd workshop-baker && cargo build --all-features
# Test
cargo test -p workshop-baker
cargo test -p workshop-engine
# Test (per crate)
cd workshop-baker && cargo test --all-features
cd workshop-engine && cargo test --all-features
# Lint
cargo clippy -- -D warnings
cargo fmt --all
# Lint (CI gate)
cargo fmt -- --check
cargo clippy --all-features -- -D warnings
# Run
cargo run --bin workshop-baker -- --pipeline demo --build-id 1 --prebake prebake.yml --bake bake.sh --finalize finalize.yml
# Docs
cd docs && mdbook build # output in docs/book/
cd docs && mdbook serve # local preview at localhost:3000
```
## NOTES
- **Docker required** for integration tests
- **No root workspace** — each crate independent, build separately
- **7 archived crates** workshop-agent, workshop-cert, workshop-deviceid, workshop-helper-mac, workshop-llm-detector, workshop-pipeline, workshop-vault
- **Python tests removed** — pytest.ini exists but test files no longer in workspace
- **No workspace Cargo.toml** at root — each crate has its own `Cargo.lock`. Build individually.
- **CI only tests** `workshop-baker` and `workshop-engine`; smaller crates tested transitively.
- **`quicktest.sh`** in workshop-baker/ is a dev smoke-test script (Docker-based, not CI).
- **Git submodules**: `rust-tongsuo`, `RustyVault` (crypto — not yet integrated into build).
- **`.secret/`** contains real credentials — never commit, never read into context.
- See `docs/src/zh-CN/vol3_dev/` for full developer documentation (architecture, design decisions, code style).
+48
View File
@@ -0,0 +1,48 @@
# DOCS KNOWLEDGE BASE
## OVERVIEW
mdBook project in zh-CN. Authoritative source for project conventions, coding style, and architecture docs. Build output at `docs/book/` (gitignored).
## STRUCTURE
```
docs/
├── book.toml # mdBook config (title, language=zh-CN)
├── src/SUMMARY.md # Sidebar nav — source of truth for page listing
└── src/zh-CN/
├── index.md # Landing
├── vol1_user/ # End-user documentation
├── vol2_admin/ # Admin reference: decorator cheatsheet, deployment,
│ # env vars, exit codes, plugin ref, security, troubleshooting
└── vol3_dev/ # Developer docs: architecture, code style, commit
# convention, testing, error handling, type system,
# design decisions, PR process
```
## WHERE TO LOOK
| Task | Location | Notes |
|------|----------|-------|
| Sidebar/nav editing | `src/SUMMARY.md` | Add entry here to surface a new page |
| Coding conventions | `vol3_dev/code_style.md` | Naming, imports, formatting |
| Commit rules | `vol3_dev/commit_convention.md` | feat/fix/docs/test/refactor/style |
| Error handling | `vol3_dev/error_handling_patterns.md` | thiserror+anyhow+HasExitCode |
| Type system | `vol3_dev/type_system_guide.md` | Newtypes, types/ module isolation |
| Testing strategy | `vol3_dev/testing_strategy.md` | 3 tiers: unit, integration, system |
| Anti-patterns | `vol3_dev/design_decisions.md` | Documented design rejections |
| Architecture | `vol3_dev/architecture.md` | Crate dependency graph, module map |
## CONVENTIONS
- **Language**: All prose in zh-CN. Code blocks in Rust (default) or bash.
- **Adding a page**: Create `.md` under `volN_*/`, then add to `SUMMARY.md`.
- **Formatting**: Standard Markdown. mdBook preprocessors handle TOC and links.
- **Cross-references**: Relative paths from source root (e.g., `../vol3_dev/code_style.md`).
- **Code blocks**: Fenced with language tag. No line numbers in docs.
- **File naming**: `snake_case.md`.
## COMMANDS
```bash
mdbook build # static site → docs/book/
mdbook serve # live preview at localhost:3000
```
+33 -3
View File
@@ -6,19 +6,49 @@ use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
#[serde(deny_unknown_fields)]
pub struct BakeParams {
#[serde(default)] pub workspace: ParamVal<String>,
/// Health probe interval in seconds (default 5).
#[serde(default)] pub health_period: ParamVal<u64>,
/// Health probe per-attempt timeout in seconds (default 10).
#[serde(default)] pub health_timeout: ParamVal<u64>,
/// Max health probe retries before giving up (default 12).
#[serde(default)] pub health_max_retries: ParamVal<u32>,
}
impl MergeParams for BakeParams {
fn defaults() -> Self { Self { workspace: ParamVal::new("/workspace".into()) } }
fn defaults() -> Self {
Self {
workspace: ParamVal::new("/workspace".into()),
health_period: ParamVal::new(5),
health_timeout: ParamVal::new(10),
health_max_retries: ParamVal::new(12),
}
}
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
apply_field!(self.workspace, i.workspace, w, ov, p, "workspace");
apply_field!(self.health_period, i.health_period, w, ov, p, "health_period");
apply_field!(self.health_timeout, i.health_timeout, w, ov, p, "health_timeout");
apply_field!(self.health_max_retries, i.health_max_retries, w, ov, p, "health_max_retries");
}
}
impl Default for BakeParams { fn default() -> Self { Self::defaults() } }
#[derive(Debug, Clone)]
pub struct BakeSettings { pub workspace: String }
pub struct BakeSettings {
pub workspace: String,
pub health_period: u64,
pub health_timeout: u64,
pub health_max_retries: u32,
}
impl Default for BakeSettings { fn default() -> Self { BakeParams::defaults().into() } }
impl From<&BakeParams> for BakeSettings { fn from(p: &BakeParams) -> Self { Self { workspace: p.workspace.value.clone() } } }
impl From<&BakeParams> for BakeSettings {
fn from(p: &BakeParams) -> Self {
Self {
workspace: p.workspace.value.clone(),
health_period: p.health_period.value,
health_timeout: p.health_timeout.value,
health_max_retries: p.health_max_retries.value,
}
}
}
impl From<BakeParams> for BakeSettings { fn from(p: BakeParams) -> Self { (&p).into() } }
+1 -1
View File
@@ -23,7 +23,7 @@ impl MergeParams for NotificationParams {
retry_backoff: ParamVal::new(2.0), retry_delay_seconds: ParamVal::new(5.0),
max_retry_delay_seconds: ParamVal::new(300.0),
life_impact_factor: ParamVal::new(0.0),
batch_period: ParamVal::new(30), batch_watermark: ParamVal::new(5),
batch_period: ParamVal::new(5), batch_watermark: ParamVal::new(1),
template_ref_depth: ParamVal::new(10),
}
}
+80
View File
@@ -0,0 +1,80 @@
# WORKSHOP-BAKER CRATE KNOWLEDGE BASE
## OVERVIEW
Primary crate — the only crate with a `[[bin]]` target. Contains the `workshop-baker` binary and `workshop_baker` library. Orchestrates the full CI/CD pipeline: prebake (env setup) → bake (script execution) → finalize (plugins, cleanup) → drain notification queue. ~69 .rs source files.
## STRUCTURE
```
src/
├── main.rs # Entry: default(full pipeline) | bare(dev-only) | daemon(unimplemented)
├── lib.rs # Clap Cli/Commands/BareCommands — CLI structs in lib, not bin
├── error.rs # CliError enum (wraps stage errors, delegates exit_code)
├── bare.rs # Dev-only single-stage execution, ctx.privileged always false
├── utils.rs # Cross-cutting helpers
├── bake/ # Script parsing + DAG scheduling + execution
│ ├── decorator.rs # @decorator(args) parsing, 81 tests — heaviest coverage in crate
│ ├── parser.rs # Step/function extraction
│ ├── schedule.rs # DAG build → topological sort
│ ├── execute.rs # Per-step timeout/retry
│ ├── builder.rs # Script assembly
│ ├── trivial.rs # Trivial step detection (skip in dry-run)
│ ├── constant.rs / error.rs / util.rs
├── prebake/ # Environment provisioning
│ ├── config.rs # PrebakeConfig deserialization
│ ├── stage.rs + stage/{bootstrap,depssystem,depsuser,environment,hook}.rs
│ ├── env.rs + env/{container,firecracker,baremetal,custom}.rs # Builder dispatch
│ ├── security.rs # Privilege/sandbox checks
│ ├── event.rs # Event sender/receiver channel
│ ├── types.rs + types/{memsize,cache,builderconfig,architecture}.rs
│ ├── constant.rs / error.rs
├── finalize/ # Post-build hooks + plugins
│ ├── config.rs # FinalizeConfig
│ ├── stage.rs + stage/hook.rs
│ ├── template.rs # Output templating (minijinja)
│ ├── plugin.rs + plugin/{shell,dylib,rhai,internal}.rs
│ │ └── plugin/internal/{dummy,mail,satori,webhook,insitenotify}.rs
│ ├── types.rs + types/compression.rs
│ ├── constant.rs / error.rs / event.rs
├── notify/ # Async notification dispatch
│ ├── queue.rs # NotificationQueue — drain on pipeline complete
│ ├── handler.rs / rule.rs / template.rs / types.rs / util.rs / error.rs
└── types/ # Shared types — ZERO deps on stage modules
├── resource.rs # ResourceRegistry (populated pre-pipeline, consumed read-only)
└── buildstatus.rs
```
## WHERE TO LOOK
| Task | Location | Notes |
|------|----------|-------|
| CLI args + subcommands | `src/lib.rs` | Cli, Commands(Bare/Daemon), BareCommands |
| Pipeline orchestration | `src/main.rs` | worker_main(): prebake→bake→finalize→queue.drain() |
| Decorator syntax | `src/bake/decorator.rs` | `# @decorator` or `# @decorator(args)`, line-by-line regex |
| Build execution | `src/bake/execute.rs` | timeout, retry, exit code capture |
| DAG scheduling | `src/bake/schedule.rs` | Uses workshop-schedule crate |
| Docker builder | `src/prebake/env/container.rs` | Bollard-based, buildkit support |
| Firecracker / Bare metal | `.../firecracker.rs`, `.../baremetal.rs` | MicroVM;
needs PIPELINE_BAREMETAL_ELEVATE |
| Plugin dispatch | `src/finalize/plugin.rs` | Dispatches shell / dylib(stub) / rhai(stub) / internal |
| Notification queue | `src/notify/queue.rs` | Spawned early, drained after pipeline, retry support |
| Error type | `src/error.rs` | CliError wraps PrebakeError/BakeError/FinalizeError/anyhow |
| Dev bare mode | `src/bare.rs` | ctx.privileged=false always; do not extrapolate to production |
## DEPENDENCIES
**Internal:** workshop-engine, workshop-schedule, workshop-getterurl, workshop-baker-params. **External:** bollard (Docker), clap (CLI), axum 0.7 (HTTP), lettre 0.11 (email), minijinja (templates), cgroups-rs (limits), privdrop.
## CONVENTIONS (CRATE-SPECIFIC)
- **Lint overrides** (this crate only): `dead_code=allow`, `unreachable_code=allow`, `inherent_to_string=allow`, `non_canonical_partial_ord_impl=allow`
- **CLI structs in lib.rs, not main.rs** — enables integration tests to import and construct Cli directly.
- **Pipeline stages return `anyhow::Result<()>`** — stages propagate with `?`; only main.rs converts to CliError.
- **ResourceRegistry pattern** — populated before pipeline stages, consumed read-only by stages. New resource types go in `types/resource.rs`.
- **Notification queue fire-and-forget + drain** — notify tasks spawned early, send during pipeline, drain on completion. `NotificationQueue::drain()` must be called.
- **Trivial step detection** — steps with no external deps can be marked trivial (`trivial.rs`), skipped in dry-run.
## ANTI-PATTERNS (THIS CRATE)
- **Do NOT derive production behavior from bare mode** — ctx.privileged is hardcoded false.
- **Do NOT add CLI args bypassing prebake→bake→finalize ordering** — the pipeline is linear. Parallelism lives in the DAG scheduler within bake.
- **Do NOT import types/ from bake/, prebake/, or finalize/** — `types/` must have zero crate-internal deps.
- **Do NOT add lint allows without justification** — the four existing overrides are intentional.
- **Do NOT spawn blocking work on tokio runtime** — use `spawn_blocking` for CPU-heavy tasks.
- **Decorator regex is line-by-line** — multi-line decorators need a parser rewrite, not a regex hack.
- **Careful with `unimplemented!()`** — Daemon, Rhai/Dylib plugins, Custom builder are stubs. Prefer error over crash.
+589 -27
View File
@@ -2,6 +2,12 @@
# It is not intended for manual editing.
version = 4
[[package]]
name = "adler2"
version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa"
[[package]]
name = "aho-corasick"
version = "1.1.4"
@@ -259,6 +265,28 @@ version = "1.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8"
[[package]]
name = "aws-lc-rs"
version = "1.17.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5ec2f1fc3ec205783a5da9a7e6c1509cc69dedf09a1949e412c1e18469326d00"
dependencies = [
"aws-lc-sys",
"zeroize",
]
[[package]]
name = "aws-lc-sys"
version = "0.41.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1a2f9779ce85b93ab6170dd940ad0169b5766ff848247aff13bb788b832fe3f4"
dependencies = [
"cc",
"cmake",
"dunce",
"fs_extra",
]
[[package]]
name = "axum"
version = "0.7.9"
@@ -393,6 +421,15 @@ dependencies = [
"generic-array",
]
[[package]]
name = "block-buffer"
version = "0.12.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cdd35008169921d80bc60d3d0ab416eecb028c4cd653352907921d95084790be"
dependencies = [
"hybrid-array",
]
[[package]]
name = "block2"
version = "0.6.2"
@@ -514,6 +551,17 @@ version = "1.11.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1e748733b7cbc798e1434b6ac524f0c1ff2ab456fe201501e6497c8417a4fc33"
[[package]]
name = "capctl"
version = "0.2.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4a6e71767585f51c2a33fed6d67147ec0343725fc3c03bf4b89fe67fede56aa5"
dependencies = [
"bitflags 1.3.2",
"cfg-if",
"libc",
]
[[package]]
name = "cast"
version = "0.3.0"
@@ -527,6 +575,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7a0aeaff4ff1a90589618835a598e545176939b97874f7abc7851caa0618f203"
dependencies = [
"find-msvc-tools",
"jobserver",
"libc",
"shlex",
]
@@ -637,12 +687,31 @@ version = "0.7.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a1d728cc89cf3aee9ff92b05e62b19ee65a02b5702cff7d5a377e32c6ae29d8d"
[[package]]
name = "cmake"
version = "0.1.58"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c0f78a02292a74a88ac736019ab962ece0bc380e3f977bf72e376c5d78ff0678"
dependencies = [
"cc",
]
[[package]]
name = "colorchoice"
version = "1.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b05b61dc5112cbb17e4b6cd61790d9845d13888356391624cbe7e41efeac1e75"
[[package]]
name = "combine"
version = "4.6.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ba5a308b75df32fe02788e748662718f03fde005016435c444eea572398219fd"
dependencies = [
"bytes",
"memchr",
]
[[package]]
name = "concurrent-queue"
version = "2.5.0"
@@ -672,6 +741,12 @@ dependencies = [
"yaml-rust2",
]
[[package]]
name = "const-oid"
version = "0.10.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c"
[[package]]
name = "const-random"
version = "0.1.18"
@@ -726,6 +801,24 @@ dependencies = [
"libc",
]
[[package]]
name = "cpufeatures"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201"
dependencies = [
"libc",
]
[[package]]
name = "crc32fast"
version = "1.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9481c1c90cbf2ac953f07c8d4a58aa3945c425b7185c9154d67a65e4230da511"
dependencies = [
"cfg-if",
]
[[package]]
name = "criterion"
version = "0.5.1"
@@ -803,6 +896,15 @@ dependencies = [
"typenum",
]
[[package]]
name = "crypto-common"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ce6e4c961d6cd6c9a86db418387425e8bdeaf05b3c8bc1411e6dca4c252f1453"
dependencies = [
"hybrid-array",
]
[[package]]
name = "deranged"
version = "0.5.5"
@@ -819,8 +921,19 @@ version = "0.10.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292"
dependencies = [
"block-buffer",
"crypto-common",
"block-buffer 0.10.4",
"crypto-common 0.1.7",
]
[[package]]
name = "digest"
version = "0.11.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2"
dependencies = [
"block-buffer 0.12.0",
"const-oid",
"crypto-common 0.2.2",
]
[[package]]
@@ -853,6 +966,12 @@ dependencies = [
"const-random",
]
[[package]]
name = "dunce"
version = "1.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "92773504d58c093f6de2459af4af33faa518c13451eb8f2b5698ed3d36e7c813"
[[package]]
name = "duration-str"
version = "0.21.0"
@@ -1025,6 +1144,22 @@ version = "0.1.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "645cbb3a84e60b7531617d5ae4e57f7e27308f6445f5abf653209ea76dec8dff"
[[package]]
name = "fixedbitset"
version = "0.5.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1d674e81391d1e1ab681a28d99df07927c6d4aa5b027d7da16ba32d1d21ecd99"
[[package]]
name = "flate2"
version = "1.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "843fba2746e448b37e26a819579957415c8cef339bf08564fe8b7ddbd959573c"
dependencies = [
"crc32fast",
"miniz_oxide",
]
[[package]]
name = "fnv"
version = "1.0.7"
@@ -1046,6 +1181,12 @@ dependencies = [
"percent-encoding",
]
[[package]]
name = "fs_extra"
version = "1.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c"
[[package]]
name = "futures-channel"
version = "0.3.31"
@@ -1136,8 +1277,10 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "335ff9f135e4384c8150d6f27c6daed433577f86b4750418338c01a1a2528592"
dependencies = [
"cfg-if",
"js-sys",
"libc",
"wasi",
"wasm-bindgen",
]
[[package]]
@@ -1147,9 +1290,11 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd"
dependencies = [
"cfg-if",
"js-sys",
"libc",
"r-efi",
"wasip2",
"wasm-bindgen",
]
[[package]]
@@ -1309,6 +1454,15 @@ version = "1.0.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "df3b46402a9d5adb4c86a0cf463f42e19994e3ee891101b1841f30a545cb49a9"
[[package]]
name = "hybrid-array"
version = "0.4.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9155a582abd142abc056962c29e3ce5ff2ad5469f4246b537ed42c5deba857da"
dependencies = [
"typenum",
]
[[package]]
name = "hyper"
version = "1.8.1"
@@ -1382,6 +1536,7 @@ version = "0.1.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "727805d60e7938b76b826a6ef209eb70eaa1812794f9424d4a4e2d740662df5f"
dependencies = [
"base64",
"bytes",
"futures-channel",
"futures-core",
@@ -1389,7 +1544,9 @@ dependencies = [
"http",
"http-body",
"hyper",
"ipnet",
"libc",
"percent-encoding",
"pin-project-lite",
"socket2",
"tokio",
@@ -1561,6 +1718,12 @@ dependencies = [
"serde_core",
]
[[package]]
name = "ipnet"
version = "2.12.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d98f6fed1fde3f8c21bc40a1abb88dd75e67924f9cffc3ef95607bad8017f8e2"
[[package]]
name = "is-terminal"
version = "0.4.17"
@@ -1626,6 +1789,65 @@ dependencies = [
"syn",
]
[[package]]
name = "jni"
version = "0.22.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5efd9a482cf3a427f00d6b35f14332adc7902ce91efb778580e180ff90fa3498"
dependencies = [
"cfg-if",
"combine",
"jni-macros",
"jni-sys",
"log",
"simd_cesu8",
"thiserror 2.0.18",
"walkdir",
"windows-link",
]
[[package]]
name = "jni-macros"
version = "0.22.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a00109accc170f0bdb141fed3e393c565b6f5e072365c3bd58f5b062591560a3"
dependencies = [
"proc-macro2",
"quote",
"rustc_version",
"simd_cesu8",
"syn",
]
[[package]]
name = "jni-sys"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c6377a88cb3910bee9b0fa88d4f42e1d2da8e79915598f65fb0c7ee14c878af2"
dependencies = [
"jni-sys-macros",
]
[[package]]
name = "jni-sys-macros"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "38c0b942f458fe50cdac086d2f946512305e5631e720728f2a61aabcd47a6264"
dependencies = [
"quote",
"syn",
]
[[package]]
name = "jobserver"
version = "0.1.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9afb3de4395d6b3e67a780b6de64b51c978ecf11cb9a462c66be7d4ca9039d33"
dependencies = [
"getrandom 0.3.4",
"libc",
]
[[package]]
name = "js-sys"
version = "0.3.83"
@@ -1721,9 +1943,15 @@ dependencies = [
[[package]]
name = "log"
version = "0.4.28"
version = "0.4.32"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "34080505efa8e45a4b816c349525ebe327ceaa8559756f0356cba97ef3bf7432"
checksum = "953f07c43838f8e6f9758cab68bf5bed85465e7587ebe0b823f1bcd81978ad3a"
[[package]]
name = "lru-slab"
version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154"
[[package]]
name = "matchit"
@@ -1737,6 +1965,16 @@ version = "0.8.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "47e1ffaa40ddd1f3ed91f717a33c8c0ee23fff369e3aa8772b9605cc1d22f4c3"
[[package]]
name = "md-5"
version = "0.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "69b6441f590336821bb897fb28fc622898ccceb1d6cea3fde5ea86b090c4de98"
dependencies = [
"cfg-if",
"digest 0.11.3",
]
[[package]]
name = "memchr"
version = "2.7.6"
@@ -1768,10 +2006,20 @@ dependencies = [
]
[[package]]
name = "mio"
version = "1.1.0"
name = "miniz_oxide"
version = "0.8.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "69d83b0086dc8ecf3ce9ae2874b2d1290252e2a30720bea58a5c6639b0092873"
checksum = "1fa76a2c86f704bdb222d66965fb3d63269ce38518b83cb0575fca855ebb6316"
dependencies = [
"adler2",
"simd-adler32",
]
[[package]]
name = "mio"
version = "1.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "02bd0af71c67b473010cbbc60715ee815645a4dc942899111f494b4b737d6fda"
dependencies = [
"libc",
"wasi",
@@ -2205,7 +2453,17 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bf1d70880e76bdc13ba52eafa6239ce793d85c8e43896507e43dd8984ff05b82"
dependencies = [
"pest",
"sha2",
"sha2 0.10.9",
]
[[package]]
name = "petgraph"
version = "0.7.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3672b37090dbd86368a4145bc067582552b29c27377cad4e0a306c97f9bd7772"
dependencies = [
"fixedbitset",
"indexmap 2.12.1",
]
[[package]]
@@ -2251,6 +2509,12 @@ dependencies = [
"futures-io",
]
[[package]]
name = "pkg-config"
version = "0.3.33"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e"
[[package]]
name = "plain"
version = "0.2.3"
@@ -2398,6 +2662,62 @@ dependencies = [
"prost",
]
[[package]]
name = "quinn"
version = "0.11.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b9e20a958963c291dc322d98411f541009df2ced7b5a4f2bd52337638cfccf20"
dependencies = [
"bytes",
"cfg_aliases",
"pin-project-lite",
"quinn-proto",
"quinn-udp",
"rustc-hash",
"rustls",
"socket2",
"thiserror 2.0.18",
"tokio",
"tracing",
"web-time",
]
[[package]]
name = "quinn-proto"
version = "0.11.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "434b42fec591c96ef50e21e886936e66d3cc3f737104fdb9b737c40ffb94c098"
dependencies = [
"aws-lc-rs",
"bytes",
"getrandom 0.3.4",
"lru-slab",
"rand",
"ring",
"rustc-hash",
"rustls",
"rustls-pki-types",
"slab",
"thiserror 2.0.18",
"tinyvec",
"tracing",
"web-time",
]
[[package]]
name = "quinn-udp"
version = "0.5.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "addec6a0dcad8a8d96a771f815f0eaf55f9d1805756410b39f5fa81332574cbd"
dependencies = [
"cfg_aliases",
"libc",
"once_cell",
"socket2",
"tracing",
"windows-sys 0.60.2",
]
[[package]]
name = "quote"
version = "1.0.42"
@@ -2535,6 +2855,41 @@ version = "0.8.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7a2d987857b319362043e95f5353c0535c1f58eec5336fdfcf626430af7def58"
[[package]]
name = "reqwest"
version = "0.13.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "219c5811de6525e5416c7d5d53bb656d3afdbc6c5af816e0802bcfa42dbdc1c3"
dependencies = [
"base64",
"bytes",
"futures-core",
"http",
"http-body",
"http-body-util",
"hyper",
"hyper-rustls",
"hyper-util",
"js-sys",
"log",
"percent-encoding",
"pin-project-lite",
"quinn",
"rustls",
"rustls-pki-types",
"rustls-platform-verifier",
"sync_wrapper",
"tokio",
"tokio-rustls",
"tower",
"tower-http",
"tower-service",
"url",
"wasm-bindgen",
"wasm-bindgen-futures",
"web-sys",
]
[[package]]
name = "ring"
version = "0.17.14"
@@ -2583,6 +2938,21 @@ dependencies = [
"num-traits",
]
[[package]]
name = "rustc-hash"
version = "2.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "94300abf3f1ae2e2b8ffb7b58043de3d399c73fa6f4b73826402a5c457614dbe"
[[package]]
name = "rustc_version"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92"
dependencies = [
"semver",
]
[[package]]
name = "rustix"
version = "1.1.4"
@@ -2602,6 +2972,7 @@ version = "0.23.35"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "533f54bc6a7d4f647e46ad909549eda97bf5afc1585190ef692b4286b198bd8f"
dependencies = [
"aws-lc-rs",
"log",
"once_cell",
"ring",
@@ -2629,15 +3000,44 @@ version = "1.13.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "21e6f2ab2928ca4291b86736a8bd920a277a399bba1589409d72154ff87c1282"
dependencies = [
"web-time",
"zeroize",
]
[[package]]
name = "rustls-platform-verifier"
version = "0.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "26d1e2536ce4f35f4846aa13bff16bd0ff40157cdb14cc056c7b14ba41233ba0"
dependencies = [
"core-foundation",
"core-foundation-sys",
"jni",
"log",
"once_cell",
"rustls",
"rustls-native-certs",
"rustls-platform-verifier-android",
"rustls-webpki",
"security-framework",
"security-framework-sys",
"webpki-root-certs",
"windows-sys 0.61.2",
]
[[package]]
name = "rustls-platform-verifier-android"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f87165f0995f63a9fbeea62b64d10b4d9d8e78ec6d7d51fb2125fda7bb36788f"
[[package]]
name = "rustls-webpki"
version = "0.103.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2ffdfa2f5286e2247234e03f680868ac2815974dc39e00ea15adc445d0aafe52"
dependencies = [
"aws-lc-rs",
"ring",
"rustls-pki-types",
"untrusted",
@@ -2861,6 +3261,17 @@ dependencies = [
"unsafe-libyaml",
]
[[package]]
name = "sha1"
version = "0.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "aacc4cc499359472b4abe1bf11d0b12e688af9a805fa5e3016f9a386dc2d0214"
dependencies = [
"cfg-if",
"cpufeatures 0.3.0",
"digest 0.11.3",
]
[[package]]
name = "sha2"
version = "0.10.9"
@@ -2868,8 +3279,19 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283"
dependencies = [
"cfg-if",
"cpufeatures",
"digest",
"cpufeatures 0.2.17",
"digest 0.10.7",
]
[[package]]
name = "sha2"
version = "0.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "446ba717509524cb3f22f17ecc096f10f4822d76ab5c0b9822c5f9c284e825f4"
dependencies = [
"cfg-if",
"cpufeatures 0.3.0",
"digest 0.11.3",
]
[[package]]
@@ -2894,6 +3316,28 @@ dependencies = [
"libc",
]
[[package]]
name = "simd-adler32"
version = "0.3.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "703d5c7ef118737c72f1af64ad2f6f8c5e1921f818cdcb97b8fe6fc69bf66214"
[[package]]
name = "simd_cesu8"
version = "1.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "94f90157bb87cddf702797c5dadfa0be7d266cdf49e22da2fcaa32eff75b2c33"
dependencies = [
"rustc_version",
"simdutf8",
]
[[package]]
name = "simdutf8"
version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e"
[[package]]
name = "slab"
version = "0.4.11"
@@ -2908,12 +3352,12 @@ checksum = "67b1b7a3b5fe4f1376887184045fcf45c69e92af734b7aaddc05fb777b6fbd03"
[[package]]
name = "socket2"
version = "0.6.1"
version = "0.6.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "17129e116933cf371d018bb80ae557e889637989d8638274fb25622827b03881"
checksum = "52d1cfed4120b4d927bf7c0f86d2087a4a7d6027c906d9f9d525a80573b9be51"
dependencies = [
"libc",
"windows-sys 0.60.2",
"windows-sys 0.61.2",
]
[[package]]
@@ -2956,6 +3400,9 @@ name = "sync_wrapper"
version = "1.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0bf256ce5efdfa370213c1dabab5935a12e49f2c58d15e9eac2870d3b4f27263"
dependencies = [
"futures-core",
]
[[package]]
name = "synstructure"
@@ -3093,10 +3540,25 @@ dependencies = [
]
[[package]]
name = "tokio"
version = "1.48.0"
name = "tinyvec"
version = "1.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ff360e02eab121e0bc37a2d3b4d4dc622e6eda3a8e5253d5435ecf5bd4c68408"
checksum = "3e61e67053d25a4e82c844e8424039d9745781b3fc4f32b8d55ed50f5f667ef3"
dependencies = [
"tinyvec_macros",
]
[[package]]
name = "tinyvec_macros"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20"
[[package]]
name = "tokio"
version = "1.52.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8fc7f01b389ac15039e4dc9531aa973a135d7a4135281b12d7c1bc79fd57fffe"
dependencies = [
"bytes",
"libc",
@@ -3111,9 +3573,9 @@ dependencies = [
[[package]]
name = "tokio-macros"
version = "2.6.0"
version = "2.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "af407857209536a95c8e56f8231ef2c2e2aff839b22e07a1ffcbc617e9db9fa5"
checksum = "385a6cb71ab9ab790c5fe8d67f1645e6c450a7ce006a33de03daa956cf70a496"
dependencies = [
"proc-macro2",
"quote",
@@ -3237,12 +3699,6 @@ dependencies = [
"tonic",
]
[[package]]
name = "topological-sort"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ea68304e134ecd095ac6c3574494fc62b909f416c4fca77e440530221e549d3d"
[[package]]
name = "tower"
version = "0.5.2"
@@ -3262,6 +3718,24 @@ dependencies = [
"tracing",
]
[[package]]
name = "tower-http"
version = "0.6.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4cfcf7e2740e6fc6d4d688b4ef00650406bb94adf4731e43c096c3a19fe40840"
dependencies = [
"bitflags 2.10.0",
"bytes",
"futures-util",
"http",
"http-body",
"pin-project-lite",
"tower",
"tower-layer",
"tower-service",
"url",
]
[[package]]
name = "tower-layer"
version = "0.3.3"
@@ -3320,9 +3794,9 @@ checksum = "bc7d623258602320d5c55d1bc22793b57daff0ec7efc270ea7d55ce1d5f5471c"
[[package]]
name = "typenum"
version = "1.19.0"
version = "1.20.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "562d481066bde0658276a35467c4af00bdc6ee726305698a55b86e61d7ad82bb"
checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20"
[[package]]
name = "ucd-trie"
@@ -3488,6 +3962,19 @@ dependencies = [
"wasm-bindgen-shared",
]
[[package]]
name = "wasm-bindgen-futures"
version = "0.4.56"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "836d9622d604feee9e5de25ac10e3ea5f2d65b41eac0d9ce72eb5deae707ce7c"
dependencies = [
"cfg-if",
"js-sys",
"once_cell",
"wasm-bindgen",
"web-sys",
]
[[package]]
name = "wasm-bindgen-macro"
version = "0.2.106"
@@ -3530,6 +4017,25 @@ dependencies = [
"wasm-bindgen",
]
[[package]]
name = "web-time"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5a6580f308b1fad9207618087a65c04e7a10bc77e02c8e84e9b00dd4b12fa0bb"
dependencies = [
"js-sys",
"wasm-bindgen",
]
[[package]]
name = "webpki-root-certs"
version = "1.0.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f31141ce3fc3e300ae89b78c0dd67f9708061d1d2eda54b8209346fd6be9a92c"
dependencies = [
"rustls-pki-types",
]
[[package]]
name = "webpki-roots"
version = "1.0.4"
@@ -3816,7 +4322,9 @@ dependencies = [
"anyhow",
"async-trait",
"axum 0.7.9",
"bitflags 2.10.0",
"bollard",
"capctl",
"cgroups-rs",
"chrono",
"clap",
@@ -3824,6 +4332,7 @@ dependencies = [
"criterion",
"duration-str",
"env_logger",
"flate2",
"futures-util",
"glob",
"globset",
@@ -3844,11 +4353,14 @@ dependencies = [
"tempfile",
"thiserror 2.0.18",
"tokio",
"topological-sort",
"url",
"uuid",
"which",
"workshop-baker-params",
"workshop-engine",
"workshop-getterurl",
"workshop-schedule",
"zstd",
]
[[package]]
@@ -3877,6 +4389,28 @@ dependencies = [
"tokio",
]
[[package]]
name = "workshop-getterurl"
version = "0.1.0"
dependencies = [
"log",
"md-5",
"reqwest",
"sha1",
"sha2 0.11.0",
"thiserror 2.0.18",
"tokio",
"url",
]
[[package]]
name = "workshop-schedule"
version = "0.1.0"
dependencies = [
"petgraph",
"thiserror 2.0.18",
]
[[package]]
name = "writeable"
version = "0.6.2"
@@ -4074,6 +4608,34 @@ version = "1.0.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e9747e91771f56fd7893e1164abd78febd14a670ceec257caad15e051de35f06"
[[package]]
name = "zstd"
version = "0.13.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e91ee311a569c327171651566e07972200e76fcfe2242a4fa446149a3881c08a"
dependencies = [
"zstd-safe",
]
[[package]]
name = "zstd-safe"
version = "7.2.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8f49c4d5f0abb602a93fb8736af2a4f4dd9512e36f7f570d66e65ff867ed3b9d"
dependencies = [
"zstd-sys",
]
[[package]]
name = "zstd-sys"
version = "2.0.16+zstd.1.5.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "91e19ebc2adc8f83e43039e79776e3fda8ca919132d68a1fed6a5faca2683748"
dependencies = [
"cc",
"pkg-config",
]
[[package]]
name = "zvariant"
version = "5.8.0"
+7 -1
View File
@@ -41,7 +41,7 @@ tar = "0.4.44"
tempfile = "3.24.0"
thiserror = "2.0.17"
tokio = { version = "1.48.0", features = ["full"] }
topological-sort = "0.2.2"
workshop-schedule = { path = "../workshop-schedule" }
async-trait = "0.1.87"
uuid = { version = "1.19.0", features = ["v4"] }
libc = "0.2"
@@ -50,8 +50,14 @@ privdrop = "0.5.6"
which = "8.0.2"
globset = "0.4.18"
axum = "0.7"
workshop-getterurl = { path = "../workshop-getterurl" }
workshop-baker-params = { version = "0.1.0", path = "../workshop-baker-params" }
bitflags = "2"
url = "2.5.8"
zstd = "0.13.3"
flate2 = "1.1.9"
capctl = "0.2.4"
[dev-dependencies]
criterion = "0.5"
-11
View File
@@ -1,11 +0,0 @@
[pytest]
testpaths = tests/integration
python_files = test_*.py
python_classes = Test*
python_functions = test_*
addopts = -v --tb=short
markers =
integration: marks tests as integration tests (deselect with '-m "not integration"')
slow: marks tests as slow running
filterwarnings =
ignore::DeprecationWarning
-57
View File
@@ -1,57 +0,0 @@
# workshop-baker/src
**Generated:** 2026-05-19
**Commit:** 28abc5d
CLI tool and runtime for executing CI/CD pipelines. Orchestrates prebake → bake → finalize stages with notification dispatch.
## STRUCTURE
```
src/
├── lib.rs # Module declarations + Cli struct + Commands enum
├── main.rs # tokio::main entry point
├── bare.rs # Bare mode (run individual stages)
├── error.rs # CliError (wraps stage errors)
├── bake.rs # Bake orchestration (script → execution)
├── bake/ # Script parsing, decorator DSL, scheduling
├── prebake.rs # Prebake orchestration (env setup)
├── prebake/ # Config, stages, security, env providers
├── finalize.rs # Finalize orchestration (post-build)
├── finalize/ # Config, plugins, hooks, templates
├── notify.rs # Notification system (batching, retry)
├── notify/ # Types, templates, interactive, queue
├── daemon/ # Empty (Daemon command = unimplemented!)
├── monitor/ # Resource usage monitoring (cgroups)
├── types.rs # Module re-exports
├── types/ # buildstatus, resource types
├── utils.rs # Module root (fetch submodule)
└── utils/ # fetch.rs
```
## WHERE TO LOOK
| Task | Location |
|------|----------|
| CLI args | `lib.rs:21` - `Cli` struct |
| Entry point | `main.rs:91` - `#[tokio::main]` |
| Pipeline orchestration | `main.rs:68-74` - prebake→bake→finalize chain |
| Bare mode (single stage) | `bare.rs:14` - `run_bare()` |
| Script parsing | `bake/parser.rs:56` - `parse_script()` |
| Decorator DSL | `bake/decorator.rs` - `Decorator` enum |
| Prebake stages | `prebake/stage/` - Bootstrap→DepsSystem→DepsUser→Hooks→Ready |
| Plugin system | `finalize/plugin.rs` + `finalize/plugin/` |
| Notification dispatch | `notify.rs:66` - `notify()` async loop |
| Notification types | `notify/types.rs` - `NotificationRule`, `ParsedTrigger` |
## CONVENTIONS
- **ExecutionContext**: Passed through all stages, carries task_id, username, env_vars, timeout
- **PrebakeStage**: Ordered enum Bootstrap→EarlyHook→DepsSystem→DepsUser→LateHook→Ready
- **Decorator**: `# @name(args)` comments above shell function declarations
- **Notification**: Plugin-based dispatch with batching, exponential backoff, priority groups
- **Engine**: Delegated to `workshop-engine` crate (formerly in-tree)
- **Dual-mode**: CLI commands (standalone) vs Daemon mode (unimplemented)
## ANTI-PATTERNS
1. **Empty daemon/** directory — Daemon command is `unimplemented!()`
2. **Dead code allowed**`lints.rust.dead_code = "allow"` in Cargo.toml
3. **Engine was extracted** — workshop-engine is its own crate now, src/engine/ removed
4. **Python test infra exists but files removed** — pytest.ini remains, test dir is empty
+177 -55
View File
@@ -1,20 +1,27 @@
use crate::bake::constant::{DEFAULT_WORKSPACE, TRIVIAL_SCRIPT_NAME};
use crate::types::debug::DebugFeature;
use crate::bake::builder::RenderMode;
use crate::bake::constant::DEFAULT_WORKSPACE;
use crate::bake::decorator::Decorator;
use crate::bake::error::BakeError;
use crate::bake::parser::Function;
use crate::bake::util::{resolve_mode, FuncMode};
use crate::cli::Cli;
use crate::prebake;
use crate::prebake::PrebakeConfig;
use crate::prebake::security::get_drop_after;
use crate::prebake::stage::PrebakeStage;
use std::collections::{HashMap, HashSet};
use std::path::{Path, PathBuf};
use std::time::Duration;
use workshop_baker_params::BakeSettings;
use workshop_engine::{Engine, EventSender, ExecutionContext};
mod builder;
pub mod constant;
mod decorator;
pub mod error;
mod execute;
pub mod parser;
mod schedule;
pub mod trivial;
pub mod util;
pub async fn bake(
script_path: &Path,
@@ -23,12 +30,18 @@ pub async fn bake(
_cli: &Cli,
ctx: &mut ExecutionContext,
event_tx: EventSender,
nevent_tx: Option<crate::notify::types::NotificationEventSender>,
) -> Result<(), BakeError> {
let script_content = std::fs::read_to_string(script_path)?;
let prebake_content = std::fs::read_to_string(prebake_path)?;
log::debug!("Reading script: {:?}", script_path);
let script_content = std::fs::read_to_string(script_path).map_err(|e| {
BakeError::IoError { path: script_path.display().to_string(), source: e }
})?;
log::debug!("Reading prebake.yml: {:?}", prebake_path);
let prebake_content = std::fs::read_to_string(prebake_path).map_err(|e| {
BakeError::IoError { path: prebake_path.display().to_string(), source: e }
})?;
let prebake = prebake::parse(prebake_content.as_str()).map_err(|e| {
log::error!("Failed to parse prebake.yml: {}", e);
log::error!("This is unexpected!");
BakeError::YamlParseError(e)
})?;
let workspace = get_workspace(&prebake);
@@ -39,44 +52,174 @@ pub async fn bake(
.iter()
.any(|d| matches!(d, decorator::Decorator::Pipeline))
});
let trivial = false;
let use_template = true;
// TODO: if PIP(pipeline implicit parameter).trivial is set, always use RenderMode::Off
let render_mode = RenderMode::Full;
let engine = Engine::new();
// TODO: resolve from merged param layers (PIP integration)
let bake_settings = BakeSettings::default();
if let Some(env) = &prebake.envvars
&& let Some(prebake_env) = &env.bake
{
ctx.env_vars.extend(prebake_env.clone());
}
if !has_pipeline || trivial {
let function = Function {
name: TRIVIAL_SCRIPT_NAME.to_string(),
decorators: vec![],
body: script_content,
};
let script =
builder::build_script(&function, bake_base_path, &workspace, None, use_template)?;
let result = engine.execute_script(&script, ctx, &event_tx).await;
result?;
if !has_pipeline {
log::trace!("Running trivial script");
trivial::run_trivial(
&script_content,
bake_base_path,
&workspace,
&engine,
ctx,
&event_tx,
render_mode,
)
.await?;
} else {
let sorted_functions = schedule::sort_function(functions.pipeline_functions)?;
log::trace!("Running pipeline script");
let remaining = functions.remaining_code;
for func in sorted_functions {
let mut modified_func = func.clone();
modified_func.body = remaining.clone() + &func.body;
let script = builder::build_script(
&modified_func,
bake_base_path,
&workspace,
None,
use_template,
)?;
let result = engine.execute_script(&script, ctx, &event_tx).await;
result?;
let (mut dag, func_map) = schedule::build_dag(functions.pipeline_functions)?;
// Pre-scan: pipe sources + decorator conflict checks.
let mut pipe_sources: HashSet<String> = HashSet::new();
for func in func_map.values() {
decorator::check_conflicts(func)?;
for decorator in &func.decorators {
if let Decorator::Pipe(names) = decorator {
for source in names {
if *source != func.name {
pipe_sources.insert(source.clone());
}
}
}
}
}
let mut stdout_cache: HashMap<String, String> = HashMap::new();
let mut async_handles: Vec<tokio::task::JoinHandle<()>> = Vec::new();
while !dag.is_empty() {
let scheduler_debug = DebugFeature::from_bits_retain(ctx.debug_flags).contains(DebugFeature::Scheduler);
if scheduler_debug {
log::info!("[BAKE_SCHED] ready={} seq={} wild={} groups={:?}",
dag.ready_count(),
dag.ready_sequential().len(),
dag.ready_wildcards().len(),
dag.ready_groups(),
);
} else {
log::debug!("[BAKE_SCHED] ready={} seq={} wild={} groups={:?}",
dag.ready_count(),
dag.ready_sequential().len(),
dag.ready_wildcards().len(),
dag.ready_groups(),
);
}
// Rule 1: sequential always goes first
let seq_candidates = dag.ready_sequential();
if let Some(seq_name) = seq_candidates.first() {
let seq_name = seq_name.to_string();
log::info!("[BAKE_SCHED] seq {}", seq_name);
let func = &func_map[&seq_name];
let mode = resolve_mode(func);
match mode {
FuncMode::Daemon => {
execute::execute_daemon(
&engine, func, ctx, &event_tx,
&workspace, &remaining, bake_base_path,
Duration::from_secs(bake_settings.health_period),
Duration::from_secs(bake_settings.health_timeout),
bake_settings.health_max_retries,
).await?;
}
FuncMode::Async => {
let handle = execute::execute_async(
func, ctx, &event_tx,
&workspace, &remaining, bake_base_path,
);
async_handles.push(handle);
}
FuncMode::Normal => {
let result = execute::execute_normal(
&engine, func, ctx, &event_tx,
&workspace, &remaining, bake_base_path, &stdout_cache,
).await;
match result {
Ok(stdout) => {
if pipe_sources.contains(&seq_name) {
stdout_cache.insert(seq_name.clone(), stdout);
}
}
Err(e) if dag.is_fallible(&seq_name) => {
log::warn!("'{}' failed (fallible): {}", seq_name, e);
}
Err(e) => return Err(e),
}
}
}
// Pop AFTER execution (Daemon/Async pop at start)
if !matches!(mode, FuncMode::Daemon | FuncMode::Async) {
dag.pop(&seq_name).map_err(|e| BakeError::IncompatibleDecorators {
function: seq_name.clone(),
reason: e.to_string(),
})?;
} else {
dag.pop(&seq_name).map_err(|e| BakeError::IncompatibleDecorators {
function: seq_name.clone(),
reason: e.to_string(),
})?;
}
continue;
}
// All remaining are parallel — pick a batch
if let Some(batch) = dag.pick_batch() {
log::info!("[BAKE_SCHED] par {:?}", batch);
let results = execute::fire_parallel_batch(
&batch, &engine, &func_map, ctx, &event_tx,
&workspace, &remaining, bake_base_path, &stdout_cache,
).await?;
for (name, stdout) in results {
if pipe_sources.contains(&name) {
stdout_cache.insert(name.clone(), stdout);
}
if let Err(e) = dag.pop(&name) {
log::warn!("Failed to pop '{}' from DAG: {}", name, e);
}
}
} else if dag.is_stalled() {
// Deadlock: named parallel groups exist but can't complete
let stalled = dag.stalled_groups();
let remaining = dag.remaining();
return Err(BakeError::IncompatibleDecorators {
function: remaining.join(", "),
reason: format!(
"parallel group(s) {:?} can never complete — blocked by @after chain",
stalled
),
});
} else {
// Ready set is empty — cycle
let remaining = dag.remaining();
if !remaining.is_empty() {
return Err(BakeError::CircularDependency(
remaining.into_iter().map(|s| s.to_string()).collect(),
));
}
break;
}
}
// Wait for all @async tasks to finish
for handle in async_handles {
let _ = handle.await;
}
}
crate::notify::types::send_stage_event(&nevent_tx, crate::types::buildstatus::StagePhase::Bake, "bake", crate::types::buildstatus::StageOutcome::Success).await;
Ok(())
}
@@ -95,26 +238,5 @@ fn get_workspace(prebake_config: &PrebakeConfig) -> PathBuf {
.unwrap_or_else(|| PathBuf::from(DEFAULT_WORKSPACE))
}
fn privileged(prebake_config: &PrebakeConfig) -> bool {
if let Some(bootstrap) = &prebake_config.bootstrap
&& bootstrap.user == "root"
{
return true;
}
let drop_after = get_drop_after(&prebake_config.security).unwrap_or_default();
PrebakeStage::Never == drop_after
}
type TaskResult = Result<(), BakeError>;
type TaskFn = Box<dyn FnOnce() -> TaskResult + Send>;
// TODO: Learn closure and continue
// async fn run<F>(f:F) -> TaskFn {
// let engine = Engine::new();
// engine.execute_script
// }
// fn with_timeout<F>(f: F, duration: i32) -> TaskFn {
// }
// ── AI Coding Agent marker ──
// Code in this module PARTIALLY or FULLY utilized AI Coding Agent
-48
View File
@@ -1,48 +0,0 @@
# workshop-baker/src/bake
**Generated:** 2026-04-22
**Commit:** 7b3b71a
Script parsing and build orchestration with decorator DSL.
## STRUCTURE
```
bake/
├── parser.rs # @decorator-aware shell script parser
├── decorator.rs # Decorator enum + 100+ unit tests
├── builder.rs # Script template assembly
├── schedule.rs # Function ordering via @after dependencies
├── error.rs # BakeError enum
└── constant.rs # DEFAULT_WORKSPACE, TRIVIAL_SCRIPT_NAME
```
## WHERE TO LOOK
| Task | Location |
|------|----------|
Parse script | `parser.rs:56` - `parse_script()` |
| Decorator types | `decorator.rs` - `Decorator` enum variants |
| Build script | `builder.rs` - template builder |
| Schedule deps | `schedule.rs` - @after resolution |
## DECORATORS
| Decorator | Purpose |
|-----------|---------|
| `@pipeline` | Marks main build function |
| `@timeout(N)` | Execution timeout seconds |
| `@retry(N, M)` | Retry N times, M sec delay |
| `@parallel` | Run concurrently |
| `@fallible` | Failure doesn't fail pipeline |
| `@after(func)` | Dependency ordering |
| `@export` | Export vars to env |
| `@loop(N)` | Repeat N times |
| `@if(COND)` | Conditional execution |
## CONVENTIONS
- **Decorator Syntax**: `# @name(args)` comment above function
- **Parsing**: Shell grammar with decorator annotation extraction
- **Scheduling**: DAG resolution for @after dependencies
- **Tests**: `decorator.rs:175-829` has 100+ inline unit tests
+75 -46
View File
@@ -1,4 +1,5 @@
use crate::bake::constant::{PIPELINE_SKIP_ERRORCODE, TRIVIAL_SCRIPT_NAME};
use crate::types::debug::DebugFeature;
use crate::bake::constant::PIPELINE_SKIP_ERRORCODE;
use crate::bake::decorator::Decorator;
use crate::bake::error::BakeError;
use crate::bake::parser::Function;
@@ -9,24 +10,45 @@ use std::path::Path;
use std::path::PathBuf;
use tempfile::NamedTempFile;
/// Template for use_template=false, which just executes the main body without any wrapping.
/// Three-level rendering control.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum RenderMode {
/// No template, hardcoded "{{ main }}". Debug escape hatch.
Off,
/// Uses bake_base.sh with essential execution skeleton only.
Minimal,
/// Uses bake_base.sh with helpers (logs, colors, etc.).
Full,
}
impl RenderMode {
pub fn as_str(&self) -> &'static str {
match self {
RenderMode::Off => "off",
RenderMode::Minimal => "minimal",
RenderMode::Full => "full",
}
}
}
/// Hardcoded template for RenderMode::Off.
const TRIVIAL_TEMPLATE: &str = "{{ main }}";
/// Renders a `Function` into an executable shell script using a minijinja template.
/// If `use_template` is true, wraps the body in `bake_base`; otherwise uses a trivial template.
///
/// `render_mode` controls the template source:
/// - `Off`: hardcoded `{{ main }}`, no bake_base file needed.
/// - `Minimal` / `Full`: reads bake_base.sh; the template distinguishes the two
/// via the `render_mode` context variable.
pub fn build_script(
function: &Function,
bake_base: &Path,
target_dir: &Path,
temp_path: Option<PathBuf>,
use_template: bool,
render_mode: RenderMode,
debug_flags: u32,
) -> Result<PathBuf, BakeError> {
let name = &function.name;
let trivial = name == TRIVIAL_SCRIPT_NAME;
let body = &function.body;
let export = function
.find_decorator(|d| matches!(d, Decorator::Export).then_some(()))
.is_some();
let condition = function.find_decorator(|d| {
if let Decorator::If(content) = d {
Some(content.clone())
@@ -34,57 +56,59 @@ pub fn build_script(
None
}
});
let temp_path = temp_path.unwrap_or_else(|| PathBuf::from("/tmp"));
// let temp_path = temp_path.unwrap_or_else(|| PathBuf::from("/tmp"));
let temp_path = target_dir;
let filename = format!("bakefn_{}.sh", name);
let target = target_dir.join(filename);
let bake_base_content = if use_template {
std::fs::read_to_string(bake_base).map_err(|e| BakeError::ScriptGenerationFailed {
script: bake_base.display().to_string(),
reason: e.to_string(),
})?
} else {
TRIVIAL_TEMPLATE.to_string()
let template_content = match render_mode {
RenderMode::Off => TRIVIAL_TEMPLATE.to_string(),
RenderMode::Minimal | RenderMode::Full => {
let script_debug = DebugFeature::from_bits_retain(debug_flags).contains(DebugFeature::Script);
if script_debug {
log::info!("Reading bake_base: {:?}", bake_base);
} else {
log::debug!("Reading bake_base: {:?}", bake_base);
}
std::fs::read_to_string(bake_base).map_err(|e| BakeError::IoError {
path: bake_base.display().to_string(),
source: e,
})?
}
};
let condition_code = match condition {
Some(content) => {
// with some bash workaround
format!(
"if [ ! {} ]; then\n exit {}\nfi\n",
"if [[ ! {} ]]; then\n exit {}\nfi\n",
content, PIPELINE_SKIP_ERRORCODE
)
}
None => String::new(),
};
let mut preexport_code = String::new();
let mut export_code = String::new();
if !trivial {
if export {
preexport_code = format!("declare -xp > /dev/shm/bakeenv.before.{}\n", name);
export_code = format!(
"declare -xp > /dev/shm/bakeenv.after.{}\ndiff -w --new-line-format='%L' --old-line-format='' --unchanged-line-format='' /dev/shm/bakeenv.before.{} /dev/shm/bakeenv.after.{} > /dev/shm/bakeexport.{}\n",
name, name, name, name
);
}
preexport_code.push_str(format!(". /dev/shm/bakeexport.{}\n", name).as_str());
export_code.push_str(format!("rm -f /dev/shm/bakeenv.*.{}\n", name).as_str());
}
// preexport/export: stubs for the future IPC-based @export.
let preexport_code = String::new();
let export_code = String::new();
let mut env = Environment::new();
env.add_template("script", bake_base_content.as_str())?;
let script = env.get_template("script")?;
let rendered = if use_template {
script.render(
context! {main => body,condition => condition_code, preexport => preexport_code, name => name, export => export_code},
)?
} else {
script.render(context! {main => body})?
};
env.add_template("script", &template_content)?;
let rendered = env.get_template("script")?.render(
context! {
main => body,
condition => condition_code,
preexport => preexport_code,
name => name,
export => export_code,
render_mode => render_mode.as_str(),
},
)?;
let mut temp_file =
NamedTempFile::new_in(&temp_path).map_err(|e| BakeError::ScriptGenerationFailed {
script: temp_path.display().to_string(),
action: "creating temp file".to_string(),
reason: e.to_string(),
})?;
@@ -93,17 +117,20 @@ pub fn build_script(
.set_permissions(Permissions::from_mode(0o700))
.map_err(|e| BakeError::ScriptGenerationFailed {
script: temp_path.display().to_string(),
action: "setting permissions".to_string(),
reason: e.to_string(),
})?;
std::fs::write(&temp_file, rendered).map_err(|e| BakeError::ScriptGenerationFailed {
script: temp_path.display().to_string(),
action: "writing to temp file".to_string(),
reason: e.to_string(),
})?;
temp_file
.persist(&target)
.map_err(|e| BakeError::ScriptGenerationFailed {
script: target.display().to_string(),
action: "persisting temp file".to_string(),
reason: e.error.to_string(),
})?;
Ok(target)
@@ -114,7 +141,7 @@ mod tests {
use super::*;
#[test]
fn test_build_script_with_template() {
fn test_build_script_full_mode() {
let temp_dir = tempfile::tempdir().unwrap();
let bake_base = temp_dir.path().join("bake_base.sh");
@@ -125,7 +152,9 @@ mod tests {
decorators: vec![],
body: "echo 'hello world'".to_string(),
};
let result = build_script(&function, &bake_base, temp_dir.path(), None, true);
let result = build_script(
&function, &bake_base, temp_dir.path(), RenderMode::Full, 0,
);
assert!(result.is_ok());
let script_path = result.unwrap();
@@ -134,10 +163,10 @@ mod tests {
}
#[test]
fn test_build_script_trivial_mode() {
fn test_build_script_off_mode() {
let temp_dir = tempfile::tempdir().unwrap();
let function = Function {
name: "trivial".to_string(),
name: "test".to_string(),
decorators: vec![],
body: "echo 'direct output'".to_string(),
};
@@ -146,8 +175,8 @@ mod tests {
&function,
Path::new("/nonexistent"),
temp_dir.path(),
None,
false,
RenderMode::Off,
0,
);
assert!(result.is_ok());
+1 -1
View File
@@ -1,4 +1,4 @@
pub const DEFAULT_WORKSPACE: &str = "/workspace";
pub const BAKE_SCRIPT_PATH: &str = "bake.sh";
pub const TRIVIAL_SCRIPT_NAME: &str = "BAKE_TRIVIAL";
pub const PIPELINE_SKIP_ERRORCODE: u8 = 233;
pub const PIPELINE_SKIP_ERRORCODE: i32 = 233;
+153 -10
View File
@@ -4,11 +4,11 @@ use std::fmt::Display;
use std::fmt::Formatter;
use crate::bake::error::BakeError;
use crate::bake::parser::Function;
lazy_static! {
// "# @decorator" or "# @decorator(parameters)"
static ref DECORATOR_REGEX: Regex = Regex::new(r"^\s*#\s+@(\w+)\s?(\((.*)\))?").unwrap()
;
static ref DECORATOR_REGEX: Regex = Regex::new(r"^\s*#\s+@(\w+)\s?(\((.*)\))?").unwrap();
static ref UNEXPECTED_ARGUMENT: &'static str = "decorator does not accept arguments";
static ref MISSING_ARGUMENT: &'static str = "decorator requires an argument";
static ref UNKNOWN_DECORATOR: &'static str = "unknown decorator";
@@ -39,11 +39,121 @@ pub enum Decorator {
/// Chain functions via pipe with comma-separated names.
Pipe(Vec<String>),
/// Run concurrently with other parallel functions.
Parallel,
/// `None` = wildcard ("*" group, compatible with any group).
/// `Some(name)` = restricted to the named group.
Parallel(Option<String>),
/// Run as a background daemon process.
Daemon,
/// Health check endpoint for daemon verification.
/// Health check command for daemon verification (shell statement).
Health(String),
/// Fire-and-forget execution; DAG pop happens at start, not end.
Async,
}
/// Returns true if the decorator is a flag (position-independent, consumed after execution).
pub fn is_flag(d: &Decorator) -> bool {
matches!(
d,
Decorator::Pipeline
| Decorator::Fallible
| Decorator::Export
| Decorator::If(_)
| Decorator::After(_)
| Decorator::Parallel(_)
| Decorator::Health(_)
| Decorator::Async
| Decorator::Unknown
)
}
/// Returns true if the decorator is a combinator (position-dependent, nestable).
pub fn is_combinator(d: &Decorator) -> bool {
matches!(d, Decorator::Loop(_) | Decorator::Timeout(_) | Decorator::Retry(..))
}
/// Returns true if the decorator is a mode (at most one, alters what gets executed).
pub fn is_mode(d: &Decorator) -> bool {
matches!(d, Decorator::Pipe(_) | Decorator::Daemon | Decorator::Async)
}
/// Extract the parallel group name. Returns "*" for wildcard (no group specified).
pub fn parallel_group(d: &Decorator) -> &str {
match d {
Decorator::Parallel(Some(g)) => g.as_str(),
_ => "*",
}
}
/// Check for incompatible decorator combinations on a function.
/// Returns `Err(IncompatibleDecorators)` on conflict, `Ok(())` otherwise.
pub fn check_conflicts(func: &Function) -> Result<(), BakeError> {
let has_daemon = func.decorators.iter().any(|d| matches!(d, Decorator::Daemon));
let has_async = func.decorators.iter().any(|d| matches!(d, Decorator::Async));
let has_pipe = func.decorators.iter().any(|d| matches!(d, Decorator::Pipe(_)));
let has_parallel = func.decorators.iter().any(|d| matches!(d, Decorator::Parallel(_)));
let has_health = func.decorators.iter().any(|d| matches!(d, Decorator::Health(_)));
let has_export = func.decorators.iter().any(|d| matches!(d, Decorator::Export));
let has_combinator = func.decorators.iter().any(|d| is_combinator(d));
let mode_count = [has_daemon, has_async, has_pipe].iter().filter(|&&x| x).count();
if mode_count > 1 {
return Err(BakeError::IncompatibleDecorators {
function: func.name.clone(),
reason: "@daemon, @async, and @pipe are mutually exclusive".into(),
});
}
if has_daemon {
if has_parallel {
return Err(BakeError::IncompatibleDecorators {
function: func.name.clone(),
reason: "@daemon cannot be combined with @parallel".into(),
});
}
if has_combinator {
return Err(BakeError::IncompatibleDecorators {
function: func.name.clone(),
reason: "@daemon cannot be combined with @loop/@retry/@timeout".into(),
});
}
if has_export {
return Err(BakeError::IncompatibleDecorators {
function: func.name.clone(),
reason: "@daemon cannot be combined with @export".into(),
});
}
}
if has_health && !has_daemon {
return Err(BakeError::IncompatibleDecorators {
function: func.name.clone(),
reason: "@health requires @daemon".into(),
});
}
if has_async {
if has_parallel {
return Err(BakeError::IncompatibleDecorators {
function: func.name.clone(),
reason: "@async cannot be combined with @parallel".into(),
});
}
if has_export {
return Err(BakeError::IncompatibleDecorators {
function: func.name.clone(),
reason: "@async cannot be combined with @export".into(),
});
}
}
if has_pipe && has_parallel {
return Err(BakeError::IncompatibleDecorators {
function: func.name.clone(),
reason: "@pipe cannot be combined with @parallel".into(),
});
}
Ok(())
}
impl Display for Decorator {
@@ -59,9 +169,13 @@ impl Display for Decorator {
Decorator::Retry(count, delay) => write!(f, "@retry({}, {})", count, delay),
Decorator::Export => write!(f, "@export"),
Decorator::Pipe(funcs) => write!(f, "@pipe({})", funcs.join(", ")),
Decorator::Parallel => write!(f, "@parallel"),
Decorator::Parallel(group) => match group {
None => write!(f, "@parallel"),
Some(g) => write!(f, "@parallel({})", g),
},
Decorator::Daemon => write!(f, "@daemon"),
Decorator::Health(check) => write!(f, "@health({})", check),
Decorator::Async => write!(f, "@async"),
}
}
}
@@ -161,9 +275,17 @@ pub fn parse_decorator(line: &str, line_num: usize) -> Result<Option<Decorator>,
}
}),
"export" => decorator_noarg(Decorator::Export),
"parallel" => decorator_noarg(Decorator::Parallel),
"parallel" => {
if has_arg {
let group = required_arg()?;
Ok(Some(Decorator::Parallel(Some(group.to_string()))))
} else {
Ok(Some(Decorator::Parallel(None)))
}
}
"daemon" => decorator_noarg(Decorator::Daemon),
"health" => required_arg().map(|a| Some(Decorator::Health(a.to_string()))),
"async" => decorator_noarg(Decorator::Async),
"pipe" => required_arg().and_then(|a| {
let functions: Vec<String> = a
.split(',')
@@ -290,7 +412,11 @@ mod tests {
#[test]
fn test_decorator_display_parallel() {
assert_eq!(format!("{}", Decorator::Parallel), "@parallel");
assert_eq!(format!("{}", Decorator::Parallel(None)), "@parallel");
assert_eq!(
format!("{}", Decorator::Parallel(Some("build".to_string()))),
"@parallel(build)"
);
}
#[test]
@@ -310,6 +436,11 @@ mod tests {
);
}
#[test]
fn test_decorator_display_async() {
assert_eq!(format!("{}", Decorator::Async), "@async");
}
#[test]
fn test_decorator_display_unknown() {
assert_eq!(format!("{}", Decorator::Unknown), "@<unknown>");
@@ -340,7 +471,13 @@ mod tests {
#[test]
fn test_parse_decorator_parallel() {
let result = parse_decorator("# @parallel", 1).unwrap();
assert_eq!(result, Some(Decorator::Parallel));
assert_eq!(result, Some(Decorator::Parallel(None)));
}
#[test]
fn test_parse_decorator_parallel_with_group() {
let result = parse_decorator("# @parallel(build)", 1).unwrap();
assert_eq!(result, Some(Decorator::Parallel(Some("build".to_string()))));
}
#[test]
@@ -349,6 +486,12 @@ mod tests {
assert_eq!(result, Some(Decorator::Daemon));
}
#[test]
fn test_parse_decorator_async() {
let result = parse_decorator("# @async", 1).unwrap();
assert_eq!(result, Some(Decorator::Async));
}
// =====================================================================
// Tests for parse_decorator - Valid single-argument decorators
// =====================================================================
@@ -576,8 +719,8 @@ mod tests {
}
#[test]
fn test_parse_decorator_parallel_with_arg() {
let result = parse_decorator("# @parallel(true)", 1);
fn test_parse_decorator_parallel_empty_args() {
let result = parse_decorator("# @parallel()", 1);
assert!(result.is_err());
}
+13 -5
View File
@@ -7,8 +7,8 @@ use workshop_engine::{
#[derive(Error, Debug)]
pub enum BakeError {
#[error("Failed to generate staged script {script}: {reason}")]
ScriptGenerationFailed { script: String, reason: String },
#[error("Failed to generate staged script {script} when {action}: {reason}")]
ScriptGenerationFailed { script: String, action: String, reason: String },
#[error("Template error: {0}")]
TemplateError(#[from] minijinja::Error),
@@ -19,8 +19,8 @@ pub enum BakeError {
#[error("Unknown @after of function {0}: {1}")]
UnknownDependency(String, String),
#[error("IO Error: {0}")]
IoError(#[from] std::io::Error),
#[error("IO error on {path}: {source}")]
IoError { path: String, source: std::io::Error },
#[error("YAML parse error: {0}")]
YamlParseError(#[from] serde_yaml::Error),
@@ -34,6 +34,12 @@ pub enum BakeError {
#[error("Script execution error: {0}")]
ScriptExecutionError(#[from] ExecutionError),
#[error("Incompatible decorators on function '{function}': {reason}")]
IncompatibleDecorators { function: String, reason: String },
#[error("Health probe failed for daemon '{command}': {error}")]
HealthProbeFailed { command: String, error: String },
}
impl HasExitCode for BakeError {
@@ -43,10 +49,12 @@ impl HasExitCode for BakeError {
BakeError::TemplateError(_) => EXITCODE_PARSE_ERROR,
BakeError::CircularDependency(_) => EXITCODE_PARSE_ERROR,
BakeError::UnknownDependency(..) => EXITCODE_PARSE_ERROR,
BakeError::IoError(_) => EXITCODE_IO_ERROR,
BakeError::IoError { .. } => EXITCODE_IO_ERROR,
BakeError::YamlParseError(_) => EXITCODE_PARSE_ERROR,
BakeError::DecoratorParseError { .. } => EXITCODE_PARSE_ERROR,
BakeError::ScriptExecutionError(e) => e.exit_code(),
BakeError::IncompatibleDecorators { .. } => EXITCODE_PARSE_ERROR,
BakeError::HealthProbeFailed { .. } => EXITCODE_IO_ERROR,
}
}
}
+383
View File
@@ -0,0 +1,383 @@
use crate::types::debug::DebugFeature;
use crate::bake::builder::{self, RenderMode};
use crate::bake::constant::PIPELINE_SKIP_ERRORCODE;
use crate::bake::decorator::{self, Decorator};
use crate::bake::error::BakeError;
use crate::bake::parser::Function;
use crate::bake::util::{is_fallible, resolve_pipe_stdin};
use std::collections::HashMap;
use std::future::Future;
use std::path::{Path, PathBuf};
use std::pin::Pin;
use std::process::Stdio;
use std::time::Duration;
use workshop_engine::{Engine, EventSender, ExecutionContext, ExecutionError};
/// Execute a single function with combinator wrapping.
/// Returns the captured stdout (for `@pipe` consumers).
pub async fn execute_one(
engine: &Engine,
func: &Function,
ctx: &mut ExecutionContext,
event_tx: &EventSender,
workspace: &Path,
remaining: &str,
bake_base: &Path,
) -> Result<String, BakeError> {
let combos: Vec<&Decorator> = func
.decorators
.iter()
.filter(|d| decorator::is_combinator(d))
.collect();
let mut modified = func.clone();
modified.body = remaining.to_string() + &func.body;
let script = builder::build_script(
&modified, bake_base, workspace, RenderMode::Full, ctx.debug_flags,
)?;
execute_with_combos(engine, &script, ctx, event_tx, &combos).await
}
async fn execute_with_combos(
engine: &Engine,
script: &PathBuf,
ctx: &mut ExecutionContext,
event_tx: &EventSender,
combos: &[&Decorator],
) -> Result<String, BakeError> {
build_combinator_chain(engine, script, ctx, event_tx, combos, 0).await
}
/// Builds a combinator execution chain. Returns the captured stdout.
/// For `@loop` + `@pipe(self)`, each iteration feeds its stdout back as
/// the next iteration's stdin via `ctx.stdin`.
fn build_combinator_chain<'a>(
engine: &'a Engine,
script: &'a PathBuf,
ctx: &'a mut ExecutionContext,
event_tx: &'a EventSender,
combos: &'a [&'a Decorator],
idx: usize,
) -> Pin<Box<dyn Future<Output = Result<String, BakeError>> + Send + 'a>> {
if idx >= combos.len() {
return Box::pin(async move {
let result = engine.execute_script(script, ctx, event_tx).await;
match result {
Ok(r) => Ok(r.stdout),
Err(ExecutionError::ExecutionFailed { exit_code, .. })
if exit_code == PIPELINE_SKIP_ERRORCODE as i32 =>
{
log::debug!("Function skipped (@if condition)");
Ok(String::new())
}
Err(e) => Err(BakeError::ScriptExecutionError(e)),
}
});
}
match combos[idx] {
Decorator::Loop(count) => Box::pin(async move {
let mut last_stdout = String::new();
for i in 0..*count {
let script_debug = DebugFeature::from_bits_retain(ctx.debug_flags).contains(DebugFeature::Script);
if script_debug {
log::info!("Loop {}/{}", i + 1, count);
} else {
log::debug!("Loop {}/{}", i + 1, count);
}
last_stdout = build_combinator_chain(
engine, script, ctx, event_tx, combos, idx + 1,
)
.await?;
// Feed stdout as stdin for the next iteration (@pipe(self))
if i + 1 < *count {
ctx.stdin = Some(last_stdout.clone().into_bytes());
}
}
Ok(last_stdout)
}),
Decorator::Timeout(secs) => Box::pin(async move {
tokio::time::timeout(
Duration::from_secs(*secs as u64),
build_combinator_chain(
engine, script, ctx, event_tx, combos, idx + 1,
),
)
.await
.map_err(|_| BakeError::ScriptExecutionError(ExecutionError::Timeout))?
}),
Decorator::Retry(retries, delay_secs) => Box::pin(async move {
let max_attempts = *retries as usize + 1;
let delay = Duration::from_secs(*delay_secs as u64);
let mut last_err = None;
for attempt in 0..max_attempts {
match build_combinator_chain(
engine, script, ctx, event_tx, combos, idx + 1,
)
.await
{
Ok(stdout) => return Ok(stdout),
Err(e) => {
if matches!(
&e,
BakeError::ScriptExecutionError(ExecutionError::Timeout)
) {
return Err(e);
}
if attempt < max_attempts - 1 {
last_err = Some(e);
log::warn!(
"Retry {}/{} after {}s",
attempt + 1,
retries,
delay_secs
);
tokio::time::sleep(delay).await;
} else {
return Err(e);
}
}
}
}
Err(last_err.unwrap())
}),
_ => build_combinator_chain(
engine, script, ctx, event_tx, combos, idx + 1,
),
}
}
/// Fire a batch of parallel nodes concurrently.
///
/// All nodes in the batch are waited on before returning. A non-`@fallible`
/// failure is surfaced only after all nodes complete (GitHub/GitLab/Concourse
/// convention).
///
/// Returns a `(name, stdout)` pair for each completed node so the caller can
/// populate the stdout cache for `@pipe`.
pub async fn fire_parallel_batch(
batch: &[String],
_engine: &Engine,
func_map: &HashMap<String, Function>,
ctx: &mut ExecutionContext,
event_tx: &EventSender,
workspace: &Path,
remaining: &str,
bake_base: &Path,
stdout_cache: &HashMap<String, String>,
) -> Result<Vec<(String, String)>, BakeError> {
let handles: Vec<_> = batch
.iter()
.map(|name| {
let function = func_map[name.as_str()].clone();
let mut task_ctx = ctx.clone();
// Resolve per-task stdin from @pipe + stdout_cache
task_ctx.stdin = resolve_pipe_stdin(&function, stdout_cache);
let task_events = event_tx.clone();
let task_workspace = workspace.to_path_buf();
let task_remaining = remaining.to_string();
let task_bake_base = bake_base.to_path_buf();
let task_name = name.clone();
tokio::spawn(async move {
let task_engine = Engine::new();
let result = execute_one(
&task_engine,
&function,
&mut task_ctx,
&task_events,
&task_workspace,
&task_remaining,
&task_bake_base,
)
.await;
(task_name, result)
})
})
.collect();
let mut first_fatal: Option<BakeError> = None;
let mut results: Vec<(String, String)> = Vec::with_capacity(handles.len());
for handle in handles {
let (name, result) = handle.await.map_err(|_| {
BakeError::ScriptExecutionError(ExecutionError::ExecutionFailed {
task_id: "parallel".to_string(),
exit_code: -1,
})
})?;
match result {
Ok(stdout) => results.push((name, stdout)),
Err(err) if is_fallible(&func_map[&name]) => {
log::warn!("'{}' failed (fallible, non-fatal): {}", name, err);
}
Err(err) => {
log::error!("'{}' failed (fatal): {}", name, err);
first_fatal.get_or_insert(err);
}
}
}
if let Some(err) = first_fatal {
return Err(err);
}
Ok(results)
}
/// Spawn an `@async` function in the background.
/// Returns a `JoinHandle` that the caller must await before returning.
pub fn execute_async(
func: &Function,
ctx: &ExecutionContext,
event_tx: &EventSender,
workspace: &Path,
remaining: &str,
bake_base: &Path,
) -> tokio::task::JoinHandle<()> {
let mut task_ctx = ctx.clone();
let task_events = event_tx.clone();
let task_workspace = workspace.to_path_buf();
let task_remaining = remaining.to_string();
let task_bake_base = bake_base.to_path_buf();
let task_func = func.clone();
tokio::spawn(async move {
let task_engine = Engine::new();
let _ = execute_one(
&task_engine, &task_func, &mut task_ctx,
&task_events, &task_workspace,
&task_remaining, &task_bake_base,
).await;
})
}
/// Execute a Normal-mode function, resolving pipe stdin first.
pub async fn execute_normal(
engine: &Engine,
func: &Function,
ctx: &mut ExecutionContext,
event_tx: &EventSender,
workspace: &Path,
remaining: &str,
bake_base: &Path,
stdout_cache: &HashMap<String, String>,
) -> Result<String, BakeError> {
ctx.stdin = resolve_pipe_stdin(func, stdout_cache);
execute_one(engine, func, ctx, event_tx, workspace, remaining, bake_base).await
}
// ── Daemon / Health ─────────────────────────────────────────────────────────
/// Execute a `@daemon` function: build script, spawn, periodic health-probe
/// (blocking until healthy), then return.
pub async fn execute_daemon(
engine: &Engine,
func: &Function,
ctx: &ExecutionContext,
event_tx: &EventSender,
workspace: &Path,
remaining: &str,
bake_base: &Path,
health_period: Duration,
health_timeout: Duration,
health_max_retries: u32,
) -> Result<(), BakeError> {
let fallible = is_fallible(func);
let mut modified = func.clone();
modified.body = remaining.to_string() + &func.body;
let script = builder::build_script(
&modified, bake_base, workspace, RenderMode::Full, ctx.debug_flags,
)?;
// Spawn without waiting
let mut command = tokio::process::Command::new(ctx.shell.as_str());
command.arg("-c");
command.arg(script.to_str().unwrap());
command.current_dir(&ctx.working_dir);
command.envs(&ctx.env_vars);
command.stdin(Stdio::null());
command.stdout(Stdio::null());
command.stderr(Stdio::null());
#[cfg(unix)]
command.process_group(0);
//let child = // Necessary?
match command.spawn() {
Ok(c) => c,
Err(e) if fallible => {
log::warn!("Failed to spawn daemon '{}' (fallible): {}", func.name, e);
return Ok(());
}
Err(_) => {
return Err(BakeError::ScriptExecutionError(ExecutionError::ExecutionFailed {
task_id: func.name.clone(),
exit_code: -1,
}));
}
};
// Periodic health probe — block until healthy, retry exhausted, or fatal
if let Some(health_cmd) = func.decorators.iter().find_map(|d| match d {
Decorator::Health(cmd) => Some(cmd.as_str()),
_ => None,
}) {
let mut retries: u32 = 0;
loop {
let probe_result = tokio::time::timeout(
health_timeout,
health_probe_one(engine, health_cmd, remaining, ctx, event_tx),
)
.await;
match probe_result {
Ok(Ok(())) => break, // healthy — done
_ if retries + 1 < health_max_retries => {
retries += 1;
log::warn!(
"Health probe for '{}' failed ({}/{}), retrying in {:?}...",
func.name, retries, health_max_retries, health_period,
);
}
_ if fallible => {
log::warn!("Health probe for '{}' exhausted (fallible), skipping", func.name);
break; // exhausted but fallible → skip health check
}
Ok(Err(e)) => return Err(e),
Err(_) => {
return Err(BakeError::HealthProbeFailed {
command: health_cmd.to_string(),
error: "timeout".to_string(),
});
}
}
tokio::time::sleep(health_period).await;
}
}
// Cleanup is handled by finalize.
Ok(())
}
/// Single health-probe attempt. Exit 0 = healthy.
async fn health_probe_one(
engine: &Engine,
cmd: &str,
remaining: &str,
ctx: &ExecutionContext,
event_tx: &EventSender,
) -> Result<(), BakeError> {
let full_cmd = format!("{}\n{}", remaining, cmd);
let result = engine
.execute_command(&full_cmd, ctx, event_tx)
.await
.map_err(|e| BakeError::HealthProbeFailed {
command: cmd.to_string(),
error: format!("{}", e),
})?;
if !result.success {
return Err(BakeError::HealthProbeFailed {
command: cmd.to_string(),
error: format!("exit code {}", result.exit_code),
});
}
Ok(())
}
+3 -1
View File
@@ -174,7 +174,7 @@ pub fn parse_script(text: &str) -> Result<ParsedScript, BakeError> {
}
continue;
}
eprintln!(
log::warn!(
"Warning: Dangling decorator(s) at line {}. Treating as comments.",
line_number - 1
);
@@ -224,6 +224,7 @@ pub fn parse_script(text: &str) -> Result<ParsedScript, BakeError> {
{
return Err(BakeError::ScriptGenerationFailed {
script: "bake.sh".to_string(),
action: "parsing function body".to_string(),
reason: format!(
"Unexpected EOF while parsing function '{}'. Unmatched braces (remaining: {})?",
name, brace_depth
@@ -234,6 +235,7 @@ pub fn parse_script(text: &str) -> Result<ParsedScript, BakeError> {
let last_decorator = pending_decorators.last().unwrap();
return Err(BakeError::ScriptGenerationFailed {
script: "bake.sh".to_string(),
action: "parsing decorators".to_string(),
reason: format!(
"Dangling decorator {} before EOF. Missing function definition?",
last_decorator
+143 -471
View File
@@ -1,42 +1,124 @@
use super::{decorator::Decorator, parser::Function};
// Code in this module PARTIALLY or FULLY utilized AI Coding Agent
use super::decorator::Decorator;
use super::parser::Function;
use crate::bake::error::BakeError;
use std::collections::HashMap;
use topological_sort::TopologicalSort;
use workshop_schedule::{Dag, DagNode, EdgeKind, NodeColor};
/// Topologically sorts functions by explicit `@after` dependencies and implicit ordering
/// for consecutive `@pipeline` functions (each pipeline function implicitly depends on
/// the previous one unless it has an explicit `@after`).
pub fn sort_function(functions: Vec<Function>) -> Result<Vec<Function>, BakeError> {
let function_mapped = functions
/// Thin wrapper so `Function` implements `DagNode`.
#[derive(Clone)]
pub(crate) struct BakeNode {
name: String,
color: NodeColor,
fallible: bool,
/// Original function body + decorators for execution
func: Function,
}
impl DagNode for BakeNode {
fn name(&self) -> &str {
&self.name
}
fn color(&self) -> NodeColor {
self.color.clone()
}
fn fallible(&self) -> bool {
self.fallible
}
}
/// Convert a `Function` into a `BakeNode` by extracting scheduling metadata
/// from decorators.
fn function_to_node(func: &Function) -> BakeNode {
let color = if func
.decorators
.iter()
.any(|d| matches!(d, Decorator::Parallel(_)))
{
let group = func
.decorators
.iter()
.find_map(|d| {
if let Decorator::Parallel(Some(g)) = d {
Some(g.clone())
} else {
None
}
})
.unwrap_or("*".to_string());
if group == "*" {
NodeColor::Wildcard
} else {
NodeColor::Named(group)
}
} else {
NodeColor::Sequential
};
let fallible = func
.decorators
.iter()
.any(|d| matches!(d, Decorator::Fallible));
BakeNode {
name: func.name.clone(),
color,
fallible,
func: func.clone(),
}
}
/// Builds a DAG from functions using `@after` dependencies and implicit
/// ordering for consecutive `@pipeline` functions.
pub fn build_dag(
functions: Vec<Function>,
) -> Result<(Dag<BakeNode>, HashMap<String, Function>), BakeError> {
let function_mapped: HashMap<&str, &Function> = functions
.iter()
.map(|f| (f.name.as_str(), f))
.collect::<HashMap<&str, &Function>>();
.collect();
let mut ts = TopologicalSort::<String>::new();
// Track last pipeline function for implicit @after
let mut dag = Dag::new();
let mut last_pipeline: Option<String> = None;
for function in &functions {
ts.insert(function.name.clone());
// Determine whether this function is a pipeline node
let is_pipeline = function
.decorators
.iter()
.any(|d| matches!(d, Decorator::Pipeline));
if !is_pipeline {
continue;
}
let has_after = function
.decorators
.iter()
.any(|d| matches!(d, Decorator::After(_)));
// Add implicit @after for pipeline functions that don't have explicit @after
if is_pipeline {
if !has_after && let Some(ref last) = last_pipeline {
ts.add_dependency(last, function.name.clone());
let node = function_to_node(function);
dag.add_node(node).map_err(|e| {
BakeError::IncompatibleDecorators {
function: function.name.clone(),
reason: e.to_string(),
}
last_pipeline = Some(function.name.clone());
}
})?;
// Add implicit @after for consecutive @pipeline functions
if !has_after {
if let Some(ref last) = last_pipeline {
dag.add_edge(last, &function.name, EdgeKind::Implicit)
.map_err(|e| BakeError::IncompatibleDecorators {
function: function.name.clone(),
reason: e.to_string(),
})?;
}
}
last_pipeline = Some(function.name.clone());
// Add explicit @after edges
for decorator in &function.decorators {
if let Decorator::After(dependency) = decorator {
if !function_mapped.contains_key(dependency.as_str()) {
@@ -50,466 +132,56 @@ pub fn sort_function(functions: Vec<Function>) -> Result<Vec<Function>, BakeErro
dependency.clone(),
));
}
ts.add_dependency(dependency, function.name.clone());
dag.add_edge(dependency, &function.name, EdgeKind::Explicit)
.map_err(|e| BakeError::IncompatibleDecorators {
function: function.name.clone(),
reason: e.to_string(),
})?;
}
}
}
let mut functions_sorted = Vec::<Function>::new();
while let Some(name) = ts.pop() {
let function = function_mapped[name.as_str()];
functions_sorted.push(function.clone());
}
dag.freeze();
if !ts.is_empty() {
let remaining: Vec<String> = ts.collect();
return Err(BakeError::CircularDependency(remaining));
}
let owned_map: HashMap<String, Function> = functions
.into_iter()
.map(|f| (f.name.clone(), f))
.collect();
Ok(functions_sorted)
Ok((dag, owned_map))
}
#[cfg(test)]
mod tests {
use super::*;
fn create_function(name: &str, decorators: Vec<Decorator>, body: &str) -> Function {
Function {
name: name.to_string(),
decorators,
body: body.to_string(),
/// Group ready parallel nodes by their declared group.
/// Returns `(star_nodes, named_groups)`.
///
/// Deprecated: use `Dag::ready_wildcards()` and `Dag::ready_groups()` instead.
pub fn partition_by_group<'a>(
ready: &[String],
dag: &Dag<BakeNode>,
) -> (Vec<String>, HashMap<String, Vec<String>>) {
let mut star_nodes = Vec::new();
let mut named_groups: HashMap<String, Vec<String>> = HashMap::new();
for name in ready {
if dag.is_wildcard(name) {
star_nodes.push(name.clone());
} else if let Some(g) = dag.group_of(name) {
named_groups.entry(g).or_default().push(name.clone());
}
// Sequential nodes: not included (caller handles them)
}
#[test]
fn test_empty_functions() {
let functions = vec![];
let result = sort_function(functions).unwrap();
assert!(result.is_empty());
}
#[test]
fn test_single_function_no_dependencies() {
let function = create_function("func1", vec![], "body1");
let functions = vec![function.clone()];
let result = sort_function(functions).unwrap();
assert_eq!(result.len(), 1);
assert_eq!(result[0].name, "func1");
}
#[test]
fn test_multiple_functions_no_dependencies() {
let function1 = create_function("func1", vec![], "body1");
let function2 = create_function("func2", vec![], "body2");
let function3 = create_function("func3", vec![], "body3");
let functions = vec![function1.clone(), function2.clone(), function3.clone()];
let result = sort_function(functions).unwrap();
assert_eq!(result.len(), 3);
let names: Vec<String> = result.iter().map(|f| f.name.clone()).collect();
assert!(names.contains(&"func1".to_string()));
assert!(names.contains(&"func2".to_string()));
assert!(names.contains(&"func3".to_string()));
}
#[test]
fn test_simple_dependency() {
let function1 = create_function("func1", vec![], "body1");
let function2 = create_function(
"func2",
vec![Decorator::After("func1".to_string())],
"body2",
);
let functions = vec![function1.clone(), function2.clone()];
let result = sort_function(functions).unwrap();
assert_eq!(result.len(), 2);
assert_eq!(result[0].name, "func1");
assert_eq!(result[1].name, "func2");
}
#[test]
fn test_multiple_dependencies() {
let function1 = create_function("func1", vec![], "body1");
let function2 = create_function(
"func2",
vec![Decorator::After("func1".to_string())],
"body2",
);
let function3 = create_function(
"func3",
vec![Decorator::After("func2".to_string())],
"body3",
);
let functions = vec![function1.clone(), function2.clone(), function3.clone()];
let result = sort_function(functions).unwrap();
assert_eq!(result.len(), 3);
assert_eq!(result[0].name, "func1");
assert_eq!(result[1].name, "func2");
assert_eq!(result[2].name, "func3");
}
#[test]
fn test_complex_dependencies() {
let function1 = create_function("func1", vec![], "body1");
let function2 = create_function(
"func2",
vec![
Decorator::After("func1".to_string()),
Decorator::After("func3".to_string()),
],
"body2",
);
let function3 = create_function(
"func3",
vec![Decorator::After("func1".to_string())],
"body3",
);
let function4 = create_function("func4", vec![], "body4");
let functions = vec![
function1.clone(),
function2.clone(),
function3.clone(),
function4.clone(),
];
let result = sort_function(functions).unwrap();
assert_eq!(result.len(), 4);
let positions: std::collections::HashMap<String, usize> = result
.iter()
.enumerate()
.map(|(i, f)| (f.name.clone(), i))
.collect();
assert!(positions.get("func1").unwrap() < positions.get("func2").unwrap());
assert!(positions.get("func1").unwrap() < positions.get("func3").unwrap());
assert!(positions.get("func3").unwrap() < positions.get("func2").unwrap());
}
#[test]
fn test_mixed_decorators() {
let function1 = create_function(
"func1",
vec![Decorator::Pipeline, Decorator::Timeout(1000)],
"body1",
);
let function2 = create_function(
"func2",
vec![
Decorator::After("func1".to_string()),
Decorator::Retry(3, 100),
Decorator::Fallible,
],
"body2",
);
let functions = vec![function1.clone(), function2.clone()];
let result = sort_function(functions).unwrap();
assert_eq!(result.len(), 2);
assert_eq!(result[0].name, "func1");
assert_eq!(result[1].name, "func2");
}
#[test]
fn test_function_with_multiple_after_decorators() {
let function1 = create_function("func1", vec![], "body1");
let function2 = create_function("func2", vec![], "body2");
let function3 = create_function(
"func3",
vec![
Decorator::After("func1".to_string()),
Decorator::After("func2".to_string()),
],
"body3",
);
let functions = vec![function1.clone(), function2.clone(), function3.clone()];
let result = sort_function(functions).unwrap();
assert_eq!(result.len(), 3);
let positions: std::collections::HashMap<String, usize> = result
.iter()
.enumerate()
.map(|(i, f)| (f.name.clone(), i))
.collect();
assert!(positions.get("func1").unwrap() < positions.get("func3").unwrap());
assert!(positions.get("func2").unwrap() < positions.get("func3").unwrap());
}
#[test]
fn test_ignores_non_after_decorators() {
let function1 = create_function(
"func1",
vec![
Decorator::Pipeline,
Decorator::If("condition".to_string()),
Decorator::Fallible,
],
"body1",
);
let function2 = create_function(
"func2",
vec![
Decorator::Loop(5),
Decorator::Timeout(1000),
Decorator::Retry(3, 100),
Decorator::Export,
],
"body2",
);
let functions = vec![function1.clone(), function2.clone()];
let result = sort_function(functions).unwrap();
assert_eq!(result.len(), 2);
let names: Vec<String> = result.iter().map(|f| f.name.clone()).collect();
assert!(names.contains(&"func1".to_string()));
assert!(names.contains(&"func2".to_string()));
}
#[test]
fn test_implicit_after_pipeline() {
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
let func2 = create_function("func2", vec![Decorator::Pipeline], "body2");
let func3 = create_function("func3", vec![Decorator::Pipeline], "body3");
let functions = vec![func1.clone(), func2.clone(), func3.clone()];
let result = sort_function(functions).unwrap();
assert_eq!(result.len(), 3);
assert_eq!(result[0].name, "func1");
assert_eq!(result[1].name, "func2");
assert_eq!(result[2].name, "func3");
}
#[test]
fn test_first_pipeline_no_implicit_after() {
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
let functions = vec![func1.clone()];
let result = sort_function(functions).unwrap();
assert_eq!(result.len(), 1);
assert_eq!(result[0].name, "func1");
}
#[test]
fn test_explicit_after_overrides_implicit() {
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
let func2 = create_function(
"func2",
vec![Decorator::Pipeline, Decorator::After("func1".to_string())],
"body2",
);
let func3 = create_function("func3", vec![Decorator::Pipeline], "body3");
let functions = vec![func1.clone(), func2.clone(), func3.clone()];
let result = sort_function(functions).unwrap();
assert_eq!(result.len(), 3);
let positions: std::collections::HashMap<String, usize> = result
.iter()
.enumerate()
.map(|(i, f)| (f.name.clone(), i))
.collect();
assert!(positions.get("func1").unwrap() < positions.get("func2").unwrap());
assert!(positions.get("func2").unwrap() < positions.get("func3").unwrap());
}
#[test]
fn test_mixed_pipeline_and_non_pipeline() {
let func1 = create_function("func1", vec![], "body1");
let func2 = create_function("func2", vec![Decorator::Pipeline], "body2");
let func3 = create_function("func3", vec![], "body3");
let func4 = create_function("func4", vec![Decorator::Pipeline], "body4");
let functions = vec![func1.clone(), func2.clone(), func3.clone(), func4.clone()];
let result = sort_function(functions).unwrap();
assert_eq!(result.len(), 4);
let positions: std::collections::HashMap<String, usize> = result
.iter()
.enumerate()
.map(|(i, f)| (f.name.clone(), i))
.collect();
assert!(positions.get("func2").unwrap() < positions.get("func4").unwrap());
}
#[test]
fn test_single_pipeline_works() {
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
let functions = vec![func1.clone()];
let result = sort_function(functions).unwrap();
assert_eq!(result.len(), 1);
assert_eq!(result[0].name, "func1");
}
#[test]
fn test_two_pipelines_implicit_chain() {
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
let func2 = create_function("func2", vec![Decorator::Pipeline], "body2");
let functions = vec![func1.clone(), func2.clone()];
let result = sort_function(functions).unwrap();
assert_eq!(result.len(), 2);
let positions: std::collections::HashMap<String, usize> = result
.iter()
.enumerate()
.map(|(i, f)| (f.name.clone(), i))
.collect();
assert!(positions.get("func1").unwrap() < positions.get("func2").unwrap());
}
#[test]
fn test_pipeline_explicit_after_skips_implicit() {
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
let func2 = create_function("func2", vec![Decorator::Pipeline], "body2");
let func3 = create_function(
"func3",
vec![Decorator::Pipeline, Decorator::After("func1".to_string())],
"body3",
);
let functions = vec![func1.clone(), func2.clone(), func3.clone()];
let result = sort_function(functions).unwrap();
assert_eq!(result.len(), 3);
let positions: std::collections::HashMap<String, usize> = result
.iter()
.enumerate()
.map(|(i, f)| (f.name.clone(), i))
.collect();
assert!(positions.get("func1").unwrap() < positions.get("func3").unwrap());
// TODO: Our design spec indicates order-based sorting, however toposort crate does not guarantee order of independent nodes. We should consider whether we want to enforce order-based sorting for pipelines, or if we are okay with any valid topological order.
// assert!(positions.get("func2").unwrap() < positions.get("func3").unwrap());
}
#[test]
fn test_circular_dependency_detected() {
let func1 = create_function(
"func1",
vec![Decorator::After("func2".to_string())],
"body1",
);
let func2 = create_function(
"func2",
vec![Decorator::After("func1".to_string())],
"body2",
);
let functions = vec![func1.clone(), func2.clone()];
let result = sort_function(functions);
assert!(result.is_err());
assert!(matches!(
result.unwrap_err(),
BakeError::CircularDependency(_)
));
}
#[test]
fn test_unknown_dependency_error() {
let func1 = create_function(
"func1",
vec![Decorator::After("nonexistent".to_string())],
"body1",
);
let functions = vec![func1.clone()];
let result = sort_function(functions);
assert!(result.is_err());
assert!(matches!(
result.unwrap_err(),
BakeError::UnknownDependency(_, _)
));
}
#[test]
fn test_pipeline_with_multiple_decorators() {
let func1 = create_function(
"func1",
vec![
Decorator::Pipeline,
Decorator::Timeout(60),
Decorator::Fallible,
],
"body1",
);
let func2 = create_function(
"func2",
vec![
Decorator::Pipeline,
Decorator::Retry(3, 5),
Decorator::After("func1".to_string()),
],
"body2",
);
let functions = vec![func1.clone(), func2.clone()];
let result = sort_function(functions).unwrap();
assert_eq!(result.len(), 2);
let positions: std::collections::HashMap<String, usize> = result
.iter()
.enumerate()
.map(|(i, f)| (f.name.clone(), i))
.collect();
assert!(positions.get("func1").unwrap() < positions.get("func2").unwrap());
}
#[test]
fn test_long_pipeline_chain() {
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
let func2 = create_function("func2", vec![Decorator::Pipeline], "body2");
let func3 = create_function("func3", vec![Decorator::Pipeline], "body3");
let func4 = create_function("func4", vec![Decorator::Pipeline], "body4");
let func5 = create_function("func5", vec![Decorator::Pipeline], "body5");
let functions = vec![
func1.clone(),
func2.clone(),
func3.clone(),
func4.clone(),
func5.clone(),
];
let result = sort_function(functions).unwrap();
assert_eq!(result.len(), 5);
let positions: std::collections::HashMap<String, usize> = result
.iter()
.enumerate()
.map(|(i, f)| (f.name.clone(), i))
.collect();
assert!(positions.get("func1").unwrap() < positions.get("func2").unwrap());
assert!(positions.get("func2").unwrap() < positions.get("func3").unwrap());
assert!(positions.get("func3").unwrap() < positions.get("func4").unwrap());
assert!(positions.get("func4").unwrap() < positions.get("func5").unwrap());
}
#[test]
fn test_non_pipeline_between_pipelines() {
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
let func2 = create_function("func2", vec![], "body2");
let func3 = create_function("func3", vec![Decorator::Pipeline], "body3");
let functions = vec![func1.clone(), func2.clone(), func3.clone()];
let result = sort_function(functions).unwrap();
assert_eq!(result.len(), 3);
let positions: std::collections::HashMap<String, usize> = result
.iter()
.enumerate()
.map(|(i, f)| (f.name.clone(), i))
.collect();
assert!(positions.get("func1").unwrap() < positions.get("func3").unwrap());
}
(star_nodes, named_groups)
}
/// Pick a complete group to execute.
///
/// Deprecated: use `Dag::pick_batch()` instead.
pub fn pick_batch(
_star_nodes: &[String],
_groups: &HashMap<String, Vec<String>>,
_group_totals: &HashMap<String, usize>,
) -> Option<Vec<String>> {
// This function is replaced by Dag::pick_batch() which has all the
// group metadata internally. Keeping the signature for backward compat
// but redirect callers to use Dag directly.
unimplemented!("use dag.pick_batch() instead")
}
+31
View File
@@ -0,0 +1,31 @@
use crate::bake::builder::{build_script, RenderMode};
use crate::bake::error::BakeError;
use crate::bake::parser::Function;
use std::path::Path;
use workshop_engine::{Engine, EventSender, ExecutionContext};
/// Executes `script_content` as a single function in trivial mode.
///
/// Two entry paths lead here:
/// - **Auto-detection** (`!has_pipeline`): bake.sh has no `@pipeline` functions,
/// the entire content is treated as one trivial function.
/// - **Explicit PIP** (future): `trivial` parameter forces trivial mode even
/// when `@pipeline` functions exist, typically with `RenderMode::Off`.
pub async fn run_trivial(
script_content: &str,
bake_base_path: &Path,
workspace: &Path,
engine: &Engine,
ctx: &mut ExecutionContext,
event_tx: &EventSender,
render_mode: RenderMode,
) -> Result<(), BakeError> {
let function = Function {
name: String::new(),
decorators: vec![],
body: script_content.to_string(),
};
let script = build_script(&function, bake_base_path, workspace, render_mode, ctx.debug_flags)?;
engine.execute_script(&script, ctx, event_tx).await?;
Ok(())
}
+61
View File
@@ -0,0 +1,61 @@
use crate::bake::decorator::Decorator;
use crate::bake::parser::Function;
use std::collections::HashMap;
/// Execution mode for a pipeline function.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum FuncMode {
Normal,
Daemon,
Async,
}
/// Determine the execution mode from decorators.
/// `@daemon`, `@async`, `@pipe` are mutually exclusive modes.
pub fn resolve_mode(func: &Function) -> FuncMode {
for d in &func.decorators {
match d {
Decorator::Daemon => return FuncMode::Daemon,
Decorator::Async => return FuncMode::Async,
_ => {}
}
}
FuncMode::Normal
}
/// True if the function is decorated with `@parallel`.
pub fn is_parallel(func: &Function) -> bool {
func.decorators
.iter()
.any(|d| matches!(d, Decorator::Parallel(_)))
}
/// True if the function is decorated with `@fallible`.
pub fn is_fallible(func: &Function) -> bool {
func.decorators
.iter()
.any(|d| matches!(d, Decorator::Fallible))
}
/// Resolve stdin for a `@pipe`-decorated function from the stdout cache.
///
/// Rule 1: self-references are skipped (handled by loop combinator).
/// Rule 3: sources present in the cache are concatenated.
/// Returns `None` if any source is missing (caller decides based on `@fallible`).
pub fn resolve_pipe_stdin(
func: &Function,
cache: &HashMap<String, String>,
) -> Option<Vec<u8>> {
let sources = func.decorators.iter().find_map(|d| match d {
Decorator::Pipe(names) => Some(names.as_slice()),
_ => None,
})?;
let mut combined = String::new();
for source in sources {
if *source == func.name {
continue; // Rule 1: skip self
}
combined.push_str(cache.get(source)?); // Rule 3
}
Some(combined.into_bytes())
}
+7 -13
View File
@@ -1,15 +1,8 @@
use std::path::PathBuf;
use crate::cli::{BareCommands, Cli};
use crate::error::CliError;
use crate::types::resource::ResourceRegistry;
use crate::{bake::bake, finalize::finalize, prebake::prebake};
use workshop_engine::{EventSender, ExecutionContext};
fn or_workshop(path: &Option<PathBuf>, name: &str) -> PathBuf {
path
.clone()
.unwrap_or_else(|| PathBuf::from(format!(".workshop/{}", name)))
}
pub async fn run_bare(
cli: &Cli,
@@ -21,15 +14,16 @@ pub async fn run_bare(
registry: &ResourceRegistry,
) -> Result<(), CliError> {
let (stage, config) = match command {
BareCommands::Prebake { .. } => ("prebake", or_workshop(&cli.prebake, "prebake.yml")),
BareCommands::Bake { .. } => ("bake", or_workshop(&cli.bake, "bake.sh")),
BareCommands::Prebake { .. } => ("prebake", &cli.prebake),
BareCommands::Bake { .. } => ("bake", &cli.bake),
BareCommands::Finalize { .. } => {
("finalize", or_workshop(&cli.finalize, "finalize.yml"))
("finalize", &cli.finalize)
}
};
log::debug!("Running bare command {} with config {}", stage, config.display());
let prebake_path = or_workshop(&cli.prebake, "prebake.yml");
let bake_base = or_workshop(&cli.bake_base, "bake_base.sh");
let prebake_path = &cli.prebake;
let bake_base = &cli.bake_base;
ctx.task_id = format!("{}-{}-{}", pipeline, build_id, stage);
ctx.pipeline_name = pipeline.to_string();
@@ -40,7 +34,7 @@ pub async fn run_bare(
prebake(&config, cli, None, ctx, event_tx, None).await?;
}
BareCommands::Bake { .. } => {
bake(&config, &bake_base, &prebake_path, cli, ctx, event_tx).await?;
bake(&config, &bake_base, &prebake_path, cli, ctx, event_tx, None).await?;
}
BareCommands::Finalize { .. } => {
finalize(&config, cli, ctx, event_tx, registry).await?;
View File
+13 -3
View File
@@ -2,6 +2,8 @@ pub mod config;
pub mod constant;
pub mod error;
pub mod event;
pub mod cleanup;
pub mod artifact;
pub mod plugin;
pub mod stage;
pub mod template;
@@ -52,9 +54,17 @@ pub async fn finalize(
// Determine if we should continue with artifacts/latehook
// If prior stages failed, we still send notification but skip artifacts
let _prior_failed = earlyhook_result.is_err();
// Prepare artifacts
// TODO: artifact stage implementation
// Artifact collection and publishing
if !_prior_failed {
log::info!("Processing {} artifacts", finalize.artifact.len());
artifact::collect_and_publish(
&finalize.artifact,
&registered_plugins,
ctx,
event_tx.clone(),
)
.await?;
}
// Late hook
stage::hook::hook(
-52
View File
@@ -1,52 +0,0 @@
# workshop-baker/src/finalize
**Generated:** 2026-05-19
**Commit:** 28abc5d
Artifact packaging, plugin-based notifications, and post-build hooks.
## STRUCTURE
```
finalize/
├── config.rs # FinalizeConfig YAML (379 lines, plugins, notifications)
├── stage.rs # FinalizeStage enum
├── stage/hook.rs # Post-build hook execution
├── event.rs # Event propagation
├── template.rs # Variable substitution {{VARIABLE}}
├── plugin.rs # Plugin trait + registration
├── plugin/
│ ├── metadata.rs # Plugin manifest parsing
│ ├── dylib.rs # Dynamic library plugins
│ ├── rhai.rs # Rhai scripting (todo!())
│ ├── shell.rs # Shell command plugins
│ └── internal.rs + internal/ # Built-in plugins (mail, webhook, satori, insitenotify)
├── types.rs
├── types/compression.rs
├── error.rs # FinalizeError
└── constant.rs
```
## WHERE TO LOOK
| Task | Location |
|------|----------|
| Config loading | `config.rs:17` - `FinalizeConfig` + `parse()` |
| Plugin registration | `plugin.rs` - `register()` |
| Email notifications | `plugin/internal/mail.rs` - SMTP |
| Webhooks | `plugin/internal/webhook.rs` |
| Satori integration | `plugin/internal/satori.rs` |
| Shell plugins | `plugin/shell.rs` |
| Template rendering | `template.rs` - `{{VARIABLE}}` handlebars-style |
## CONVENTIONS
- **Plugin System**: `FinalizePlugin` trait + registry with both internal and dynamic plugins
- **Template Syntax**: `{{VARIABLE}}` handlebars-style substitution via `template.rs`
- **Stages**: EarlyHook → (Artifacts) → LateHook → Notifications
- **Notification**: `notify.rs` in parent module drives plugin-based notification dispatch
- **ResourceRegistry**: Shared resource cache passed through to plugins
## ANTI-PATTERNS
1. **fetch/ subdirectory removed** — was `plugin/fetch/{git,http,extract,checksum}`, no longer exists
2. **SHA1 deprecated** — workshop-getterurl handles checksums now
3. **Unimplemented**`rhai.rs:17` - "todo!()" for Rhai scripting
4. **TODO markers** — Multiple TODOs in config.rs (lines 305-306)
5. **Artifact stage not implemented**`finalize.rs:57` — "TODO: artifact stage implementation"
+139
View File
@@ -0,0 +1,139 @@
//! Artifact collection and publishing.
//!
//! Collects build artifacts (via glob patterns), optionally compresses them,
//! and dispatches each artifact through its configured publish plugins.
//! The publish plugin receives the artifact path as a system-injected
//! `artifact` field in its argument.
use std::path::{Path, PathBuf};
use crate::finalize::config::ArtifactDef;
use crate::finalize::error::FinalizeError;
use crate::finalize::plugin::PluginMap;
use crate::finalize::types::compression::CompressionMethod;
use workshop_engine::{EventSender, ExecutionContext};
/// Collect artifacts and execute their publish plugins.
///
/// For each artifact:
/// 1. Resolve source files via glob
/// 2. Optionally compress (zstd, gzip, dir)
/// 3. Inject `artifact` key into each publish plugin's config
/// 4. Call the plugin
pub async fn collect_and_publish(
artifacts: &[ArtifactDef],
plugins: &PluginMap,
ctx: &ExecutionContext,
event_tx: EventSender,
) -> Result<(), FinalizeError> {
if artifacts.is_empty() {
return Ok(());
}
for artifact in artifacts {
let artifact_path = process_artifact(artifact).await?;
if let Some(ref path) = artifact_path {
log::info!("Artifact processed: {} -> {}", artifact.path, path.display());
}
for (plugin_name, publish_config) in &artifact.publish {
let plugin = plugins.get(plugin_name).ok_or_else(|| {
FinalizeError::PluginExecutionFailed(format!(
"publish plugin '{}' not found for artifact '{}'",
plugin_name, artifact.path,
))
})?;
// Inject artifact path into the plugin argument
let argument = if let Some(ref path) = artifact_path {
let mut arg = publish_config.clone();
if let Some(map) = arg.as_mapping_mut() {
map.insert(
serde_yaml::Value::String("artifact".into()),
serde_yaml::Value::String(path.to_string_lossy().into()),
);
}
arg
} else {
publish_config.clone()
};
log::info!(
"Publishing artifact '{}' via plugin '{}'",
artifact.path, plugin_name,
);
plugin.call(argument, ctx, &event_tx).await?;
}
}
Ok(())
}
/// Resolve and optionally compress an artifact.
/// Returns the local path to the processed artifact, or `None` if non-file artifact.
async fn process_artifact(artifact: &ArtifactDef) -> Result<Option<PathBuf>, FinalizeError> {
if artifact.path.is_empty() {
return Ok(None); // non-filesystem artifact (e.g. Docker image)
}
let src = PathBuf::from(&artifact.path);
if !src.exists() {
if artifact.path.contains('*') || artifact.path.contains('?') {
return Err(FinalizeError::PluginExecutionFailed(
format!("glob patterns not supported: '{}'; use a directory path", artifact.path),
));
}
return Err(FinalizeError::ArtifactNotFound(src));
}
match artifact.compression {
CompressionMethod::None => Ok(Some(src)),
CompressionMethod::Dir => Ok(Some(src)),
CompressionMethod::Gzip => {
let dst = std::env::temp_dir().join(format!("artifact-{}.tar.gz", artifact.id.as_deref().unwrap_or("unknown")));
compress_tar_zstd_or_gzip(&src, &dst, false).await?;
Ok(Some(dst))
}
CompressionMethod::Zstd => {
let dst = std::env::temp_dir().join(format!("artifact-{}.tar.zst", artifact.id.as_deref().unwrap_or("unknown")));
compress_tar_zstd_or_gzip(&src, &dst, true).await?;
Ok(Some(dst))
}
}
}
async fn compress_tar_zstd_or_gzip(
src: &Path,
dst: &Path,
zstd: bool,
) -> Result<(), FinalizeError> {
let _archive = tokio::task::spawn_blocking({
let src = src.to_path_buf();
let dst = dst.to_path_buf();
move || -> Result<(), String> {
let file = std::fs::File::create(&dst)
.map_err(|e| format!("cannot create {}: {}", dst.display(), e))?;
let writer: Box<dyn std::io::Write> = if zstd {
Box::new(zstd::stream::write::Encoder::new(file, 0).map_err(|e| e.to_string())?)
} else {
Box::new(flate2::write::GzEncoder::new(file, flate2::Compression::default()))
};
let mut tar = tar::Builder::new(writer);
if src.is_dir() {
tar.append_dir_all(".", &src)
.map_err(|e| format!("tar error: {}", e))?;
} else {
tar.append_path(&src)
.map_err(|e| format!("tar error: {}", e))?;
}
tar.into_inner().map_err(|e| e.to_string())?;
Ok(())
}
})
.await
.map_err(|e| FinalizeError::PluginExecutionFailed(format!("spawn_blocking: {}", e)))?
.map_err(FinalizeError::PluginExecutionFailed)?;
Ok(())
}
+67
View File
@@ -0,0 +1,67 @@
//! Pipeline cleanup and resource reclamation.
//!
//! Runs after all pipeline stages and notification dispatch complete.
//! Checks the configured [`CleanupPolicy`] and performs the following:
//!
//! 1. Temp file / cache directory removal
//! 2. Policy-based skip (never, manual, on_success)
use crate::finalize::config::{CleanupConfig, CleanupPolicy};
use crate::finalize::error::FinalizeError;
use workshop_engine::ExecutionContext;
/// Execute cleanup according to the pipeline's [`CleanupPolicy`].
///
/// Returns `Ok(())` when cleanup completes (or is skipped per policy).
/// Returns `Err` only if a cleanup action itself fails (not if the
/// pipeline failed; pipeline outcome is checked via `pipeline_ok`).
pub fn cleanup(
config: &CleanupConfig,
_ctx: &ExecutionContext,
pipeline_ok: bool,
) -> Result<(), FinalizeError> {
match config.policy {
CleanupPolicy::Never => {
log::info!("[cleanup] policy=never, skipping all cleanup");
return Ok(());
}
CleanupPolicy::Manual => {
log::info!("[cleanup] policy=manual, hanging for bakerd signal");
// TODO(daemon): hang on bakerd signal
// For now (standalone/bare) manual == Never
return Ok(());
}
CleanupPolicy::OnSuccess if !pipeline_ok => {
log::info!("[cleanup] policy=on_success, but pipeline failed — skipping");
// TODO(daemon): hang for bakerd
return Ok(());
}
_ => {}
}
log::info!("[cleanup] starting");
cleanup_temp_files();
subprocess_reclaim();
log::info!("[cleanup] complete");
Ok(())
}
/// Reclaim child processes (placeholder for daemon-mode PID tracking).
/// In standalone mode, children are reaped by the kernel on process exit.
fn subprocess_reclaim() {
// TODO(daemon): track spawned PIDs during pipeline and signal them here.
}
/// Remove plugin cache and temporary directories.
fn cleanup_temp_files() {
let cache = std::path::Path::new("/tmp/baker");
if cache.exists() {
match std::fs::remove_dir_all(cache) {
Ok(()) => log::debug!("[cleanup] removed cache: {}", cache.display()),
Err(e) => log::warn!("[cleanup] failed to remove {}: {}", cache.display(), e),
}
}
let workshop_templates = std::env::temp_dir().join("workshop-templates");
if workshop_templates.exists() {
let _ = std::fs::remove_dir_all(&workshop_templates);
}
}
+15 -7
View File
@@ -185,15 +185,21 @@ impl<'de> serde::Deserialize<'de> for NotificationConfig {
#[serde(default)]
templates: HashMap<String, NotificationTemplateDef>,
#[serde(flatten)]
groups_raw: HashMap<String, HashMap<String, NotificationMethod>>,
groups_raw: HashMap<String, Option<HashMap<String, NotificationMethod>>>,
}
let h = Helper::deserialize(deserializer)?;
let groups: HashMap<u32, _> = h.groups_raw.into_iter()
.map(|(k, v)| {
k.parse::<u32>().map(|n| (n, v))
.map_err(|_| serde::de::Error::custom(format!("priority key must be integer, got {:?}", k)))
.filter_map(|(k, v)| {
let parsed = k.parse::<u32>().map(|n| (n, v.unwrap_or_default()));
match parsed {
Ok(pair) => Some(pair),
Err(e) => {
log::warn!("priority key must be integer, got {:?}: {}", k, e);
None
}
}
})
.collect::<Result<_, _>>()?;
.collect();
Ok(NotificationConfig { templates: h.templates, groups })
}
}
@@ -336,8 +342,9 @@ pub struct ArtifactDef {
/// Compression method: "zstd", "gzip", "none", "dir"
#[serde(default)]
pub compression: CompressionMethod,
// Publishing targets
// TODO: under heavy refactor
/// Publish targets: plugin name → configuration (artifact path injected by system)
#[serde(default)]
pub publish: HashMap<String, serde_yaml::Value>
}
impl Default for ArtifactDef {
@@ -347,6 +354,7 @@ impl Default for ArtifactDef {
path: String::new(),
retention_ms: default_retention_ms(),
compression: CompressionMethod::default(),
publish: HashMap::new(),
}
}
}
+41 -21
View File
@@ -1,4 +1,5 @@
use workshop_engine::{ExecutionEvent, StreamType};
use crate::types::debug::DebugFeature;
/// Receives and processes execution events for the finalize stage.
/// Logs task lifecycle events (started, completed, failed) and streaming output (stdout/stderr)
@@ -7,43 +8,62 @@ pub async fn event_receiver(
mut rx: tokio::sync::mpsc::Receiver<ExecutionEvent>,
pipeline_name: String,
build_id: String,
debug_flags: u32,
) {
let events_debug = DebugFeature::from_bits_retain(debug_flags).contains(DebugFeature::Events);
while let Some(event) = rx.recv().await {
match event {
ExecutionEvent::TaskStarted { task_id, timestamp } => {
log::debug!(
"[{}] [{}] [{}] Task started: {}",
pipeline_name,
build_id,
timestamp,
task_id
);
if events_debug {
log::info!(
"[{}] [{}] [{}] Task started: {}",
pipeline_name,
build_id,
timestamp,
task_id
);
} else {
log::debug!(
"Task started: {}",
task_id
);
}
}
ExecutionEvent::OutputChunk {
task_id,
task_id: _,
stream,
data,
} => match stream {
StreamType::Stdout => print!("[{}] [{}] [stdout] {}", pipeline_name, task_id, data),
StreamType::Stdout => print!("{}", data),
StreamType::Stderr => {
eprint!("[{}] [{}] [stderr] {}", pipeline_name, task_id, data)
eprint!("{}", data)
}
},
ExecutionEvent::TaskCompleted { task_id, result } => {
log::debug!(
"[{}] [{}] Task completed: {} (exit_code={}, duration={:?})",
pipeline_name,
build_id,
task_id,
result.exit_code,
result.duration
);
if result.success {
if events_debug {
log::info!(
"[{}] [{}] Task completed: {} (exit_code={}, duration={:?})",
pipeline_name, build_id, task_id,
result.exit_code, result.duration,
);
} else {
log::debug!(
"Task completed: {} (exit_code={}, duration={:?})",
task_id, result.exit_code, result.duration,
);
}
} else {
log::warn!(
"Task completed with non-zero exit: {} (exit_code={}, duration={:?})",
task_id,
result.exit_code, result.duration,
);
}
}
ExecutionEvent::TaskFailed { task_id, error } => {
log::error!(
"[{}] [{}] Task failed: {} (error={})",
pipeline_name,
build_id,
"Task failed: {} (error={})",
task_id,
error
);
+4 -2
View File
@@ -10,7 +10,7 @@ use std::path::{Path, PathBuf};
use std::sync::Arc;
mod dylib;
mod internal;
pub mod internal;
mod metadata;
mod rhai;
mod shell;
@@ -232,7 +232,9 @@ pub fn register(
_ => {}
}
log::debug!("Registering internal plugin {}", i.name);
plugins.insert(i.name.to_string(), FinalizePlugin::new(i.entry));
let mut plugin = FinalizePlugin::new(i.entry);
plugin.metadata.renderer = Some(i.renderer);
plugins.insert(i.name.to_string(), plugin);
count += 1;
}
+10 -1
View File
@@ -1,4 +1,5 @@
use crate::finalize::plugin::AsyncPluginFn;
use crate::notify::types::NotificationRenderer;
pub mod dummy;
mod insitenotify;
mod mail;
@@ -7,6 +8,7 @@ mod webhook;
pub struct InternalPlugin {
pub name: &'static str,
pub entry: Box<dyn AsyncPluginFn>,
pub renderer: NotificationRenderer,
}
pub fn get_internal_plugin() -> Vec<InternalPlugin> {
@@ -14,25 +16,32 @@ pub fn get_internal_plugin() -> Vec<InternalPlugin> {
InternalPlugin {
name: "in-site-notify",
entry: Box::new(insitenotify::InsiteNotify),
renderer: NotificationRenderer::Plugin,
},
InternalPlugin {
name: "mail",
entry: Box::new(mail::Mail),
renderer: NotificationRenderer::Server,
},
InternalPlugin {
name: "satori",
entry: Box::new(satori::Satori),
renderer: NotificationRenderer::Server,
},
InternalPlugin {
name: "webhook",
entry: Box::new(webhook::Webhook),
renderer: NotificationRenderer::Server,
},
]
}
pub fn get_dummy_plugin() -> Vec<InternalPlugin> {
/// Returns dummy only (for `--debug notify.dummy` mode).
/// All real adapters and in-site-notify are excluded.
pub fn get_debug_plugin() -> Vec<InternalPlugin> {
vec![InternalPlugin {
name: "dummy",
entry: Box::new(dummy::Dummy),
renderer: NotificationRenderer::Plugin,
}]
}
@@ -10,12 +10,12 @@ pub struct Dummy;
impl AsyncPluginFn for Dummy {
async fn call(
&self,
_metadata: &PluginMetadata,
_argument: serde_yaml::Value,
metadata: &PluginMetadata,
argument: serde_yaml::Value,
_ctx: &ExecutionContext,
_event_tx: &EventSender,
) -> Result<(), PluginError> {
log::debug!("[NOTIFY_DEBUG] Successfully registered dummy plugin.");
log::info!("[Dummy] metadata={:?}, arg={:?}", metadata, argument);
Ok(())
}
}
@@ -4,7 +4,7 @@ use serde::Deserialize;
use std::path::PathBuf;
/// Plugin manifest metadata containing identification and configuration.
#[derive(Deserialize, Clone)]
#[derive(Deserialize, Clone, Debug)]
pub struct PluginMetadata {
/// Plugin name identifier.
pub name: String,
+1 -1
View File
@@ -51,7 +51,7 @@ pub async fn hook(
.await
.map_err(|e| {
log::error!("Plugin hook {} failed: {:?}", index, e);
ExecutionError::ExecutionFailed(e.to_string())
ExecutionError::ExecutionFailed { task_id: format!("hook-{}", index), exit_code: -1 }
})?;
continue;
}
+19 -12
View File
@@ -3,9 +3,11 @@ pub mod bake;
pub mod error;
pub mod finalize;
pub mod notify;
pub mod prefetch;
pub mod prebake;
pub mod types;
pub mod utils;
pub mod daemon;
pub use clap::{Parser, Subcommand};
pub use serde::Deserialize;
@@ -21,37 +23,42 @@ pub mod cli {
pub struct Cli {
/// Username of the pipeline executor
#[arg(short, long, env = "HBW_USERNAME")]
pub username: String,
pub username: Option<String>,
/// Pipeline ID of this pipeline
#[arg(short, long, env = "HBW_PIPELINE")]
pub pipeline: String,
pub pipeline: Option<String>,
/// Build ID of this build
#[arg(short, long, env = "HBW_BUILD_ID")]
pub build_id: String,
pub build_id: Option<String>,
/// Parse the script without executing
#[arg(long, default_value = "false", env = "HBW_DRY_RUN")]
pub dry_run: bool,
/// Override prebake config path
#[arg(long)]
pub prebake: Option<PathBuf>,
#[arg(long, default_value = ".workshop/prebake.yml")]
pub prebake: PathBuf,
/// Override bake script path
#[arg(long)]
pub bake: Option<PathBuf>,
#[arg(long, default_value = ".workshop/bake.sh")]
pub bake: PathBuf,
/// Override bake_base.sh path
#[arg(long)]
pub bake_base: Option<PathBuf>,
#[arg(long, default_value = ".workshop/bake_base.sh")]
pub bake_base: PathBuf,
/// Override finalize config path
#[arg(long)]
pub finalize: Option<PathBuf>,
#[arg(long, default_value = ".workshop/finalize.yml")]
pub finalize: PathBuf,
/// Debug features (comma-separated). "notify" enables notification debug logging.
/// Debug features. Use `--debug help` for details.
#[arg(long, env = "HBW_DEBUG")]
pub debug: Option<String>,
/// JSON string of pre-resolved resources: `{"name":"/path",...}`
/// Only valid in `bare` mode.
#[arg(long, value_name = "JSON")]
pub resource_registry: Option<String>,
#[command(subcommand)]
pub command: Option<Commands>,
}
+173 -49
View File
@@ -1,89 +1,164 @@
use workshop_baker::types::resource::ResourceRegistry;
use std::path::PathBuf;
use workshop_baker::{
bake::bake, cli::{Cli, Commands, Parser},
error::CliError,
finalize::finalize, notify::notify, prebake::prebake,
};
use capctl;
use workshop_baker::bare::run_bare;
use workshop_baker::notify::queue::NotificationQueue;
use workshop_baker::types::debug::DebugFeature;
use workshop_baker::types::resource::ResourceRegistry;
use workshop_baker::{
bake::bake,
cli::{Cli, Commands, Parser},
error::CliError,
finalize::finalize,
notify::notify,
prebake::prebake,
};
use workshop_engine::{EventSender, ExecutionContext};
fn or_workshop(path: &Option<PathBuf>, name: &str) -> PathBuf {
path.clone().unwrap_or_else(|| PathBuf::from(format!(".workshop/{}", name)))
}
async fn worker_main(cli: Cli) -> Result<(), CliError> {
let prebake_path = cli.prebake.clone().unwrap_or_else(|| {
log::error!("--prebake is required in worker standalone mode");
std::process::exit(1);
});
let bake_path = cli.bake.clone().unwrap_or_else(|| {
log::error!("--bake is required in worker standalone mode");
std::process::exit(1);
});
let finalize_path = cli.finalize.clone().unwrap_or_else(|| {
log::error!("--finalize is required in worker standalone mode");
std::process::exit(1);
});
// a magic closure hack
let require = |name: &str, val: &Option<String>| -> String {
val.clone().unwrap_or_else(|| {
log::error!("--{} is required", name);
std::process::exit(1);
})
};
// worker depends on these information to work
let pipeline = require("pipeline", &cli.pipeline);
let build_id = require("build-id", &cli.build_id);
let username = require("username", &cli.username);
log::info!("Worker standalone mode");
println!("── worker standalone mode ───────────────────────");
println!(" {:<14} {:?}", "prebake:", cli.prebake);
println!(" {:<14} {:?}", "bake:", cli.bake);
println!(" {:<14} {:?}", "bake-base:", cli.bake_base);
println!(" {:<14} {:?}", "finalize:", cli.finalize);
println!("──────────────────────────────────────────────────");
let debug_flags = DebugFeature::parse(cli.debug.as_deref());
if !debug_flags.is_empty() {
log::info!("Debug features enabled: {:?}", debug_flags);
}
let mut ctx = ExecutionContext {
standalone: true,
task_id: format!("{}-{}-worker", cli.pipeline, cli.build_id),
pipeline_name: cli.pipeline.clone(),
username: cli.username.clone(),
task_id: format!("{}-{}-worker", pipeline, build_id),
pipeline_name: pipeline.clone(),
build_id: build_id.clone(),
username: username.clone(),
dry_run: cli.dry_run,
privileged: false,
debug_flags: debug_flags.bits(),
..Default::default()
};
let (tx, event_rx) = tokio::sync::mpsc::channel(256);
let event_tx: EventSender = Some(tx);
let _event_handle = tokio::spawn(workshop_baker::prebake::event::event_receiver(
event_rx, cli.pipeline.clone(), cli.build_id.clone(),
event_rx,
pipeline.clone(),
build_id.clone(),
ctx.debug_flags,
));
let (to_tx, to_rx) = tokio::sync::mpsc::channel(256);
let (retry_tx, retry_rx) = tokio::sync::mpsc::channel(256);
// Resource registry — populated before stages, consumed by stages
let registry = ResourceRegistry::new();
// TODO: pre-fetch plugins and templates into registry
// Resource registry — fetch plugins/templates before pipeline stages
let mut registry = ResourceRegistry::new();
if let Ok(finalize_cfg) = workshop_baker::finalize::parse(&cli.finalize) {
let _ = workshop_baker::prefetch::prefetch_resources(&finalize_cfg, &mut registry).await;
}
let registry = registry; // freeze mut → immut
let notify_path = finalize_path.clone();
let notify_finalize = cli.finalize.clone();
let notify_ctx = ctx.clone();
let notify_etx = event_tx.clone();
let debug = cli.debug.clone();
let notify_registry = registry.clone();
let notify_debug = ctx.debug_flags;
let notify_handle = tokio::spawn(async move {
notify(&notify_path, &mut notify_ctx.clone(), notify_etx, to_rx, retry_tx, debug.map(|d| d.contains("notify")).unwrap_or(false), &notify_registry).await
notify(
&notify_finalize,
&mut notify_ctx.clone(),
notify_etx,
to_rx,
retry_tx,
notify_debug,
&notify_registry,
)
.await
});
let nevent_tx = to_tx.clone();
let mut queue = NotificationQueue::new(to_tx, retry_rx);
let mut queue = NotificationQueue::new(to_tx, retry_rx, notify_debug);
let pipe_result: anyhow::Result<()> = async {
prebake(&prebake_path, &cli, None, &mut ctx, event_tx.clone(), Some(nevent_tx)).await?;
let bake_base_path = or_workshop(&cli.bake_base, "bake_base.sh");
let pb_path = or_workshop(&cli.prebake, "prebake.yml");
bake(&bake_path, &bake_base_path, &pb_path, &cli, &mut ctx, event_tx.clone()).await?;
finalize(&finalize_path, &cli, &mut ctx, event_tx.clone(), &registry).await?;
prebake(
&cli.prebake,
&cli,
None,
&mut ctx,
event_tx.clone(),
Some(nevent_tx.clone()),
)
.await?;
let bake_result = bake(
&cli.bake,
&cli.bake_base,
&cli.prebake,
&cli,
&mut ctx,
event_tx.clone(),
Some(nevent_tx.clone()),
)
.await;
if bake_result.is_err() {
workshop_baker::notify::types::send_stage_event(
&Some(nevent_tx.clone()),
workshop_baker::types::buildstatus::StagePhase::Bake,
"bake",
workshop_baker::types::buildstatus::StageOutcome::Failure,
)
.await;
}
bake_result?;
finalize(&cli.finalize, &cli, &mut ctx, event_tx.clone(), &registry).await?;
Ok(())
}.await;
}
.await;
if let Err(e) = pipe_result {
// Notify pipeline failure before exiting
workshop_baker::notify::types::send_stage_event(
&Some(nevent_tx.clone()),
workshop_baker::types::buildstatus::StagePhase::Bake,
"pipeline",
workshop_baker::types::buildstatus::StageOutcome::PipelineFailure,
)
.await;
log::error!("Pipeline stage failed: {}", e);
return Err(CliError::General(e));
}
// Notify pipeline success
workshop_baker::notify::types::send_stage_event(
&Some(nevent_tx.clone()),
workshop_baker::types::buildstatus::StagePhase::Bake,
"pipeline",
workshop_baker::types::buildstatus::StageOutcome::PipelineSuccess,
)
.await;
log::info!("Pipeline complete. Draining notifications...");
drop(nevent_tx);
queue.tx = None;
if let Err(e) = queue.drain().await {
log::error!("Notification queue error: {}", e);
}
let _ = notify_handle.await;
// Cleanup after notifications are fully drained
let pipeline_ok = pipe_result.is_ok();
let _ = workshop_baker::finalize::parse(&cli.finalize).map(|cfg| {
let _ = workshop_baker::finalize::cleanup::cleanup(&cfg.cleanup, &ctx, pipeline_ok);
});
Ok(())
}
@@ -91,32 +166,81 @@ async fn worker_main(cli: Cli) -> Result<(), CliError> {
async fn main() {
env_logger::init();
let cli = Cli::parse();
if std::process::id() == 1 {
log::error!("Baker is not meant to be run as PID=1");
std::process::exit(1);
}
if let Err(e) = capctl::prctl::set_subreaper(true){
log::error!("Failed to set self as subreaper: {}", e);
log::error!("finalize.cleanup might not work properly");
}
// --debug help: print help and exit immediately
if let Some(ref raw) = cli.debug {
if DebugFeature::print_help(raw) {
return;
}
}
let debug_flags = DebugFeature::parse(cli.debug.as_deref());
if cli.dry_run {
log::warn!("Dry run enabled, no changes will be made.");
}
let pipeline = cli.pipeline.clone();
let build_id = cli.build_id.clone();
// --resource-registry is only allowed in bare mode
if cli.resource_registry.is_some() && !matches!(cli.command, Some(Commands::Bare { .. })) {
log::error!("--resource-registry is only supported in bare mode");
std::process::exit(1);
}
match &cli.command {
Some(Commands::Bare { command }) => {
// --pipeline, --build-id, --username are optional in bare mode
// and they will be defaulted to "BarePipeline", "0", "bare" respectively
let pipeline = cli
.pipeline
.as_deref()
.unwrap_or("BarePipeline")
.to_string();
let build_id = cli.build_id.as_deref().unwrap_or("0").to_string();
let username = cli.username.as_deref().unwrap_or("bare").to_string();
let mut ctx = ExecutionContext {
standalone: true,
task_id: format!("{}-{}-bare", pipeline, build_id),
pipeline_name: pipeline.clone(),
username: cli.username.clone(),
build_id: build_id.clone(),
username: username.clone(),
dry_run: cli.dry_run,
privileged: false,
debug_flags: debug_flags.bits(),
..Default::default()
};
let (tx, event_rx) = tokio::sync::mpsc::channel(256);
let event_tx: EventSender = Some(tx);
let event_handle = tokio::spawn(workshop_baker::prebake::event::event_receiver(
event_rx, pipeline.clone(), build_id.clone(),
event_rx,
pipeline.clone(),
build_id.clone(),
ctx.debug_flags,
));
let registry = ResourceRegistry::new();
let result = run_bare(&cli, command, &mut ctx, event_tx, &pipeline, &build_id, &registry).await;
let registry = if let Some(ref json_str) = cli.resource_registry {
match workshop_baker::types::resource::ResourceRegistry::from_json_str(json_str) {
Ok(r) => {
log::info!("Loaded {} resources from --resource-registry", r.len());
r
}
Err(e) => {
log::error!("Failed to parse --resource-registry: {}", e);
std::process::exit(1);
}
}
} else {
ResourceRegistry::new()
};
let result = run_bare(
&cli, command, &mut ctx, event_tx, &pipeline, &build_id, &registry,
)
.await;
drop(ctx);
let _ = event_handle.await;
-118
View File
@@ -1,118 +0,0 @@
use crate::monitor::UsageStats;
use cgroups_rs;
use cgroups_rs::fs::Cgroup;
use cgroups_rs::fs::{cpu::CpuController, memory::MemController, pid::PidController};
use log;
use std::time::{Duration, Instant};
#[cfg(target_os = "linux")]
pub fn get_usage(
name: &String,
usage_prev: crate::monitor::UsageStats,
) -> anyhow::Result<crate::monitor::UsageStats> {
let hierarchy = cgroups_rs::fs::hierarchies::auto();
if !hierarchy.v2() {
return Err(anyhow::anyhow!(
"Unsupported cgroup hierarchy, cgroups v2 required."
));
}
let cgroup = Cgroup::load(hierarchy, name);
let mut cpu_usage_us: u64 = 0;
let mut cpu_usage_percent: f64 = 0.0;
let mut memory_usage_bytes: u64 = 0;
let mut memory_limit_bytes: i64 = 0;
let mut process_count: u64 = 0;
if let Some(cpu) = cgroup.controller_of::<CpuController>() {
let stats = parse_cpustat(cpu.cpu().stat.as_str())?;
cpu_usage_us = stats.usage_usec;
} else {
log::warn!("Failed to fetch CPU usage info!");
}
let current = Instant::now();
let delta_usage = cpu_usage_us - usage_prev.cpu_usage_us;
let delta_wall = current - usage_prev._timestamp;
if usage_prev._timestamp > current {
log::warn!("Usage stats are from the future!");
} else if usage_prev.cpu_usage_us == 0 {
log::info!("Previous CPU usage is zero, probably a fresh start.");
} else if delta_wall < Duration::from_millis(1) {
log::warn!("Usage stats are too close together!");
} else {
cpu_usage_percent = delta_usage as f64 / delta_wall.as_micros() as f64 * 100.0;
// dbg!(&delta_usage, &delta_wall);
}
if let Some(mem) = cgroup.controller_of::<MemController>() {
let stats = mem.memory_stat();
memory_usage_bytes = stats.usage_in_bytes;
memory_limit_bytes = stats.limit_in_bytes;
}
if let Some(proc) = cgroup.controller_of::<PidController>() {
process_count = proc.get_pid_current()?;
}
Ok(UsageStats {
cpu_usage_us,
cpu_usage_percent,
memory_usage_bytes,
// memory_usage_percent: 0.0,
memory_limit_bytes,
process_count,
_timestamp: Instant::now(),
})
// todo!();
}
#[derive(Debug, Default, Clone)]
pub struct CgroupCpuStat {
pub usage_usec: u64,
pub user_usec: u64,
pub system_usec: u64,
pub nice_usec: u64,
pub core_sched_force_idle_usec: u64,
pub nr_periods: u64,
pub nr_throttled: u64,
pub throttled_usec: u64,
pub nr_bursts: u64,
pub burst_usec: u64,
}
fn parse_cpustat(stat: &str) -> anyhow::Result<CgroupCpuStat> {
let mut cpu_stat = CgroupCpuStat::default();
for line in stat.lines() {
let line = line.trim();
if line.is_empty() {
continue;
}
let (key, value_str) = line
.split_once(' ')
.ok_or_else(|| anyhow::anyhow!("Invalid format:{}", line))?;
let value = value_str.parse::<u64>().map_err(|e| {
anyhow::anyhow!(
"Cannot convert value {}({}) into integer: {}",
key,
value_str,
e
)
})?;
match key {
"usage_usec" => cpu_stat.usage_usec = value,
"user_usec" => cpu_stat.user_usec = value,
"system_usec" => cpu_stat.system_usec = value,
"nice_usec" => cpu_stat.nice_usec = value,
"core_sched.force_idle_usec" => cpu_stat.core_sched_force_idle_usec = value,
"nr_periods" => cpu_stat.nr_periods = value,
"nr_throttled" => cpu_stat.nr_throttled = value,
"throttled_usec" => cpu_stat.throttled_usec = value,
"nr_bursts" => cpu_stat.nr_bursts = value,
"burst_usec" => cpu_stat.burst_usec = value,
_unknown_key => {} // Ignore unknown keys
}
}
Ok(cpu_stat)
}
-4
View File
@@ -1,4 +0,0 @@
#[cfg(target_os = "windows")]
pub fn get_usage() -> anyhow::Result<f64> {
unimplemented!("Not supporting windows JobObject at this time!");
}
+52 -14
View File
@@ -1,10 +1,12 @@
use crate::finalize::parse;
use crate::finalize::plugin;
use crate::finalize::plugin::internal;
use crate::notify::types::DEFAULT_NOTIFY_PARAMS;
use crate::notify::types::NotificationEventReceiver;
use crate::notify::types::NotificationEventSender;
use crate::notify::handler::NotificationHandler;
use crate::types::resource::ResourceRegistry;
use crate::types::debug::DebugFeature;
use std::path::Path;
use std::sync::Arc;
use workshop_engine::{EventSender, ExecutionContext};
@@ -26,17 +28,23 @@ pub async fn notify(
_event_tx: EventSender,
mut rx: NotificationEventReceiver,
retry_tx: NotificationEventSender,
_debug: bool,
debug_flags: u32,
registry: &ResourceRegistry,
) -> anyhow::Result<()> {
let debug = DebugFeature::from_bits_retain(debug_flags);
let notify_debug = debug.contains(DebugFeature::NotifyDummy);
let mut finalize = parse(finalize_path)?;
// Always inject in-site-notify as the lowest priority fallback
{
// L18: 0 priorities → notification ends; L19: has priorities → inject fallback
// In notify-debug mode, inject dummy instead of in-site-notify.
if !finalize.notification.groups.is_empty() {
let fallback_name = if notify_debug { "dummy" } else { "in-site-notify" };
log::debug!("Injecting {} fallback", fallback_name);
let max_p = finalize.notification.groups.keys().max().copied().unwrap_or(0);
let mut m = std::collections::HashMap::new();
m.insert(
"in-site-notify".to_string(),
fallback_name.to_string(),
crate::finalize::NotificationMethod::default(),
);
finalize.notification.groups.insert(max_p + 1, m);
@@ -45,34 +53,64 @@ pub async fn notify(
validate(&finalize)?;
let plugin_list = extract_plugin(&finalize);
let notify_plugins = plugin::register(
finalize.plugin,
Some(&plugin_list),
&mut finalize.notification,
registry,
)?;
// Register plugins: use dummy-only set in notify-debug mode
let notify_plugins = if notify_debug {
log::info!("[DEBUG-NOTIFY] Replacing notification plugins with debug set");
let mut plugins: crate::finalize::plugin::PluginMap = std::collections::HashMap::new();
for i in internal::get_debug_plugin() {
log::info!("[DEBUG-NOTIFY] Registered: {}", i.name);
let mut plugin = plugin::FinalizePlugin::new(i.entry);
plugin.metadata.renderer = Some(i.renderer);
plugins.insert(i.name.to_string(), plugin);
}
plugins
} else {
plugin::register(
finalize.plugin,
Some(&plugin_list),
&mut finalize.notification,
registry,
)?
};
resolve_external_template_refs(&mut finalize.notification.templates, registry);
flatten_template_refs(&mut finalize.notification.templates)?;
let notification_rules = parse_rules(&finalize.notification);
// Debug mode: eliminate cooldown so bucket dispatches immediately
let (period, watermark) = if notify_debug {
(0, 0)
} else {
(DEFAULT_NOTIFY_PARAMS.batch_period.value, DEFAULT_NOTIFY_PARAMS.batch_watermark.value)
};
let handler = Arc::new(NotificationHandler::new(
Arc::new(notification_rules),
Arc::new(notify_plugins),
Arc::new(finalize.notification.templates),
Arc::new(ctx.clone()),
retry_tx,
DEFAULT_NOTIFY_PARAMS.batch_period.value,
DEFAULT_NOTIFY_PARAMS.batch_watermark.value,
period,
watermark,
));
let mut handles: Vec<tokio::task::JoinHandle<()>> = Vec::new();
loop {
let Some(event) = rx.recv().await else { break };
if notify_debug {
log::info!(
"[DEBUG-NOTIFY] event: id={} stage={:?} substage={} outcome={:?}",
event.id, event.stage, event.substage, event.outcome);
}
let h = handler.clone();
tokio::spawn(async move {
handles.push(tokio::spawn(async move {
h.process_event(event).await;
});
}));
}
// Wait for all bucket dispatches to complete before returning
for handle in handles {
let _ = handle.await;
}
Ok(())
}
+6 -1
View File
@@ -73,8 +73,10 @@ impl NotificationHandler {
/// Process a single event: route to buckets, await completion,
/// then handle retry/escalation for each bucket result.
pub async fn process_event(self: Arc<Self>, event: NotificationEvent) {
log::info!("[TRACE] process_event: stage={:?} substage={} outcome={:?}", event.stage, event.substage, event.outcome);
let entries = self.route_event(&event);
if entries.is_empty() {
log::warn!("[TRACE] process_event: no matching rules, event dropped");
return;
}
@@ -142,6 +144,8 @@ impl NotificationHandler {
) -> Vec<((String, usize), Arc<Notify>)> {
let mut matches: Vec<(String, usize)> = Vec::new();
let config_groups: Vec<usize> = self.config.values().map(|r| r.len()).collect();
log::info!("[TRACE] route_event: groups={:?}", config_groups);
for ruleset in self.config.values() {
for (method, rules) in ruleset.iter() {
if let Some(ref method_rules) = event.target.rules {
@@ -336,7 +340,8 @@ fn build_plugin_argument(
Some(argument)
}
NotificationRenderer::Plugin => {
unimplemented!("Plugin renderer is not yet implemented")
// PSR: pass config directly to plugin (plugin handles own rendering)
Some(rule.config.clone())
}
NotificationRenderer::Interactive => {
unimplemented!("Interactive renderer is not yet implemented")
+32 -6
View File
@@ -1,3 +1,4 @@
use crate::types::debug::DebugFeature;
use crate::notify::types::{NotificationEvent, NotificationEventReceiver, NotificationEventSender};
use std::collections::BinaryHeap;
@@ -8,16 +9,18 @@ use std::collections::BinaryHeap;
/// - `rx` receives retry events back from notify()
pub struct NotificationQueue {
pub pq: BinaryHeap<NotificationEvent>,
pub tx: NotificationEventSender,
pub tx: Option<NotificationEventSender>,
pub rx: NotificationEventReceiver,
pub debug_flags: u32,
}
impl NotificationQueue {
pub fn new(tx: NotificationEventSender, rx: NotificationEventReceiver) -> Self {
pub fn new(tx: NotificationEventSender, rx: NotificationEventReceiver, debug_flags: u32) -> Self {
Self {
pq: BinaryHeap::new(),
tx,
tx: Some(tx),
rx,
debug_flags,
}
}
@@ -29,8 +32,12 @@ impl NotificationQueue {
/// Non-blocking single round: recycle retry events into PQ, then
/// pop and send all ready events (not_before ≤ now).
pub async fn flush_now(&mut self) -> anyhow::Result<()> {
let events_debug = DebugFeature::from_bits_retain(self.debug_flags).contains(DebugFeature::Events);
// Recycle retry events
while let Ok(event) = self.rx.try_recv() {
if events_debug {
log::info!("[EVENTS] flush_now: recycle event id={} retrying", event.id);
}
self.pq.push(event);
}
// Send ready events
@@ -40,7 +47,12 @@ impl NotificationQueue {
break;
}
let event = self.pq.pop().unwrap();
self.tx.send(event).await?;
if events_debug {
log::info!("[EVENTS] flush_now: sending event id={}", event.id);
}
if let Some(ref tx) = self.tx {
tx.send(event).await?;
}
}
Ok(())
}
@@ -49,9 +61,13 @@ impl NotificationQueue {
/// For standalone mode: called after prebake/bake/finalize to wait for completion.
/// For c/d mode: the main execution loop.
pub async fn drain(&mut self) -> anyhow::Result<()> {
let events_debug = DebugFeature::from_bits_retain(self.debug_flags).contains(DebugFeature::Events);
loop {
// Drain retry channel
while let Ok(event) = self.rx.try_recv() {
if events_debug {
log::info!("[EVENTS] drain: recycle event id={} retrying", event.id);
}
self.pq.push(event);
}
// Determine sleep deadline
@@ -68,7 +84,12 @@ impl NotificationQueue {
tokio::select! {
event = self.rx.recv() => {
match event {
Some(event) => self.pq.push(event),
Some(event) => {
if events_debug {
log::info!("[EVENTS] drain: recv event id={}", event.id);
}
self.pq.push(event);
}
None => break Ok(()),
}
}
@@ -77,7 +98,12 @@ impl NotificationQueue {
while let Some(peek) = self.pq.peek() {
if peek.not_before > now { break; }
let event = self.pq.pop().unwrap();
self.tx.send(event).await?;
if events_debug {
log::info!("[EVENTS] drain: sending event id={}", event.id);
}
if let Some(ref tx) = self.tx {
tx.send(event).await?;
}
}
}
}
+7 -11
View File
@@ -109,6 +109,7 @@ pub fn resolve_external_template_refs(
templates: &mut HashMap<String, NotificationTemplateDef>,
registry: &ResourceRegistry,
) {
log::debug!("Resolving external template refs");
for (name, tmpl_def) in templates.iter_mut() {
if tmpl_def.use_.is_none() {
continue;
@@ -139,17 +140,11 @@ pub fn resolve_template(
_standalone: bool,
server_templates: Option<&HashMap<String, NotificationTemplateDef>>,
) -> Result<NotificationTemplateDef, NotifyError> {
// 1. Try the initial schema directly
if let Some(schema_name) = schema.as_deref() {
if let Some(template) =
resolve_template_by_schema(schema_name, method_name, templates, server_templates)
{
return Ok(template);
}
}
// 2. Walk the fallback chain
let mut schema_next = resolve_template_next_schema(schema);
let mut schema_next = if schema.is_none() {
resolve_template_next_schema(schema)
} else {
schema.clone()
};
while let Some(ref schema_name) = schema_next {
if let Some(template) =
resolve_template_by_schema(schema_name, method_name, templates, server_templates)
@@ -192,6 +187,7 @@ pub fn resolve_template_by_schema(
pub fn flatten_template_refs(
templates: &mut HashMap<String, NotificationTemplateDef>,
) -> Result<(), NotifyError> {
log::debug!("Flattening template refs");
let names: Vec<String> = templates.keys().cloned().collect();
for name in names {
let mut visited = std::collections::HashSet::new();
+34 -36
View File
@@ -1,4 +1,3 @@
use crate::finalize::plugin::FinalizePlugin;
use crate::types::buildstatus::StageOutcome;
use crate::types::buildstatus::StagePhase;
use chrono::{DateTime, Utc};
@@ -13,29 +12,21 @@ use std::time::Duration;
use uuid::Uuid;
use crate::finalize::NotificationTemplate;
use crate::finalize::plugin::FinalizePlugin;
pub type NotificationEventReceiver = tokio::sync::mpsc::Receiver<NotificationEvent>;
pub type NotificationEventSender = tokio::sync::mpsc::Sender<NotificationEvent>;
pub type NotificationPluginMap = HashMap<String, FinalizePlugin>;
/// Final runtime configuration for notification, restored and flattened from yaml
pub struct NotificationRule {
pub enabled: bool,
pub trigger: Vec<ParsedTrigger>,
pub template: NotificationTemplate,
/// Whether notification failure should not trigger next priority group
pub fallible: bool,
/// Maximum number of attempts
pub max_lives: u32,
/// Timeout for notification execution
pub timeout: Duration,
/// Plugin-specific configuration
pub config: serde_yaml::Value,
/// Factor of exponential backoff for retry delay
// Not provided for external use
pub retry_backoff: f64,
}
@@ -56,7 +47,6 @@ pub enum Matchable {
impl Matchable {
fn matches(&self, stage: &StagePhase, substage: &str) -> bool {
let stagename = [stage.as_str(), ".", substage].concat();
match self {
Matchable::Glob(s) => s.is_match(stagename.as_str()),
Matchable::Regex(r) => r.is_match(stagename.as_str()),
@@ -69,7 +59,6 @@ impl ParsedTrigger {
match self {
ParsedTrigger::OnSuccess => outcome == &StageOutcome::PipelineSuccess,
ParsedTrigger::OnFailure => outcome == &StageOutcome::PipelineFailure,
// TODO: Implement condition checking for OnFinishOf
ParsedTrigger::OnFinishOf(m) => m.matches(stage, substage),
}
}
@@ -78,12 +67,9 @@ impl ParsedTrigger {
impl NotificationRule {
pub fn accept(&self, result: &StageOutcome, stage: &StagePhase, substage: &str) -> bool {
if self.trigger.is_empty() {
eprintln!("[ACCEPT] trigger empty, returning true");
return true;
}
self.trigger
.iter()
.any(|t| t.accept(result, stage, substage))
self.trigger.iter().any(|t| t.accept(result, stage, substage))
}
}
@@ -113,10 +99,6 @@ impl Default for NotificationTarget {
pub type Priority = u32;
pub type Method = String;
pub type NotificationRulesMap = HashMap<Priority, HashMap<Method, Vec<NotificationRule>>>;
// pub type BundledNotificationEvent = Vec<NotificationEvent>;
pub type NotificationEventReceiver = tokio::sync::mpsc::Receiver<NotificationEvent>;
pub type NotificationEventSender = tokio::sync::mpsc::Sender<NotificationEvent>;
pub type NotificationPluginMap = HashMap<String, FinalizePlugin>;
#[derive(Debug, Clone, Serialize)]
pub struct NotificationEvent {
@@ -127,9 +109,9 @@ pub struct NotificationEvent {
pub not_before: DateTime<Utc>,
pub priority: u32,
pub effective_priority: u32,
pub retry_backoff: f64, // Exponential backoff factor
pub attempted: u32, // Cumulative cross-group retry count
pub target: NotificationTarget, // Current audience
pub retry_backoff: f64,
pub attempted: u32,
pub target: NotificationTarget,
}
impl Hash for NotificationEvent {
@@ -188,23 +170,39 @@ impl Display for NotificationRenderer {
}
/// Notification batching configuration.
/// Controls how notification events are grouped before dispatch.
#[derive(Debug, Clone)]
pub struct BatchConfig {
/// Whether batching is enabled
pub enabled: bool,
/// Group period in seconds (the maximum wait time for a batch)
pub period: u32,
/// Group watermark threshold (the minimum number of events before batching)
pub watermark: u32,
}
impl Default for BatchConfig {
fn default() -> Self {
Self {
enabled: true,
period: 30,
watermark: 5,
}
Self { enabled: true, period: 30, watermark: 5 }
}
}
/// Send a stage completion event to the notification queue.
pub async fn send_stage_event(
nevent_tx: &Option<NotificationEventSender>,
stage: StagePhase,
substage: &str,
outcome: StageOutcome,
) {
if let Some(tx) = nevent_tx {
let event = NotificationEvent {
id: Uuid::new_v4(),
stage,
substage: substage.to_string(),
outcome,
not_before: Utc::now(),
priority: 0,
effective_priority: 0,
retry_backoff: 1.0,
attempted: 0,
target: Default::default(),
};
let _ = tx.send(event).await;
}
}
+1
View File
@@ -5,6 +5,7 @@ use crate::notify::types::{DEFAULT_NOTIFY_PARAMS, NotificationEvent, Notificatio
use std::collections::HashSet;
pub fn validate(finalize: &FinalizeConfig) -> anyhow::Result<()> {
log::debug!("Validating finalize config");
let validate_result = finalize.validate();
if let Err(errors) = validate_result {
for error in &errors {
+52 -41
View File
@@ -22,20 +22,19 @@
pub mod config;
pub mod constant;
pub mod env;
pub mod error;
pub mod event;
pub mod security;
pub mod stage;
pub mod types;
use crate::notify::types::NotificationEventSender;
use crate::notify::types::send_stage_event;
use crate::types::buildstatus::{StageOutcome, StagePhase};
use crate::{
cli::Cli,
prebake::{error::PrebakeError, security::get_drop_after, stage::PrebakeStage},
};
use crate::notify::types::NotificationEventSender;
use crate::types::buildstatus::{StageOutcome, StagePhase};
use uuid::Uuid;
use chrono::Utc;
use std::path::Path;
use workshop_engine::{EventSender, ExecutionContext};
@@ -91,7 +90,7 @@ pub fn parse(yaml_content: &str) -> Result<PrebakeConfig, serde_yaml::Error> {
/// - Any stage execution fails
pub async fn prebake(
prebake_path: &Path,
cli: &Cli,
_cli: &Cli,
stage: Option<PrebakeStage>,
ctx: &mut ExecutionContext,
event_tx: EventSender,
@@ -130,19 +129,25 @@ pub async fn prebake(
let _stage_start = Utc::now();
log::info!("Running PBStage: Bootstrap");
prebake_drop_privilege(PrebakeStage::Bootstrap, dropafter, target_user, ctx)?;
ctx.task_id = format!("{}-{}-bootstrap", cli.pipeline, cli.build_id);
ctx.task_id = format!("{}-{}-bootstrap", ctx.pipeline_name, ctx.build_id);
let osinfo = os_info::get();
let result = stage::bootstrap::bootstrap(&prebake, &osinfo, ctx, &event_tx).await;
result?;
}
send_stage_event(&nevent_tx, StagePhase::Prebake, "bootstrap", StageOutcome::Success).await;
send_stage_event(
&nevent_tx,
StagePhase::Prebake,
"bootstrap",
StageOutcome::Success,
)
.await;
// EarlyHook stage
if stage <= PrebakeStage::EarlyHook {
let _stage_start = Utc::now();
log::info!("Running PBStage: EarlyHook");
prebake_drop_privilege(PrebakeStage::EarlyHook, dropafter, target_user, ctx)?;
ctx.task_id = format!("{}-{}-earlyhook", cli.pipeline, cli.build_id);
ctx.task_id = format!("{}-{}-earlyhook", ctx.pipeline_name, ctx.build_id);
let earlyhook = match &prebake.hooks {
Some(hooks) => hooks.early.clone(),
None => None,
@@ -150,7 +155,13 @@ pub async fn prebake(
let result = stage::hook::hook(earlyhook, ctx, event_tx.clone()).await;
result?;
}
send_stage_event(&nevent_tx, StagePhase::Prebake, "earlyhook", StageOutcome::Success).await;
send_stage_event(
&nevent_tx,
StagePhase::Prebake,
"earlyhook",
StageOutcome::Success,
)
.await;
if let Some(dependencies) = prebake.dependencies {
let endpoint = dependencies
@@ -217,11 +228,17 @@ pub async fn prebake(
let _stage_start = Utc::now();
log::info!("Running PBStage: DepsSystem");
prebake_drop_privilege(PrebakeStage::DepsSystem, dropafter, target_user, ctx)?;
ctx.task_id = format!("{}-{}-depssystem", cli.pipeline, cli.build_id);
ctx.task_id = format!("{}-{}-depssystem", ctx.pipeline_name, ctx.build_id);
let result = stage::depssystem::depssystem(&system_config, ctx, event_tx.clone()).await;
result?;
}
send_stage_event(&nevent_tx, StagePhase::Prebake, "depssystem", StageOutcome::Success).await;
send_stage_event(
&nevent_tx,
StagePhase::Prebake,
"depssystem",
StageOutcome::Success,
)
.await;
if let Some(user) = dependencies.user
&& stage <= PrebakeStage::DepsUser
@@ -229,56 +246,50 @@ pub async fn prebake(
let _stage_start = Utc::now();
log::info!("Running PBStage: DepsUser");
prebake_drop_privilege(PrebakeStage::DepsUser, dropafter, target_user, ctx)?;
ctx.task_id = format!("{}-{}-depsuser", cli.pipeline, cli.build_id);
ctx.task_id = format!("{}-{}-depsuser", ctx.pipeline_name, ctx.build_id);
let result = stage::depsuser::depsuser(&user, ctx, event_tx.clone(), &endpoint).await;
result?;
}
}
send_stage_event(&nevent_tx, StagePhase::Prebake, "depsuser", StageOutcome::Success).await;
send_stage_event(
&nevent_tx,
StagePhase::Prebake,
"depsuser",
StageOutcome::Success,
)
.await;
// LateHook stage
if stage <= PrebakeStage::LateHook {
let _stage_start = Utc::now();
log::info!("Running PBStage: LateHook");
prebake_drop_privilege(PrebakeStage::LateHook, dropafter, target_user, ctx)?;
ctx.task_id = format!("{}-{}-latehook", cli.pipeline, cli.build_id);
ctx.task_id = format!("{}-{}-latehook", ctx.pipeline_name, ctx.build_id);
let latehook = match &prebake.hooks {
Some(hooks) => hooks.late.clone(),
None => None,
};
let result = stage::hook::hook(latehook, ctx, event_tx.clone()).await;
result?;
send_stage_event(&nevent_tx, StagePhase::Prebake, "latehook", StageOutcome::Success).await;
send_stage_event(
&nevent_tx,
StagePhase::Prebake,
"latehook",
StageOutcome::Success,
)
.await;
}
// Ready stage
ctx.task_id = format!("{}-{}-ready", cli.pipeline, cli.build_id);
send_stage_event(&nevent_tx, StagePhase::Prebake, "ready", StageOutcome::Success).await;
ctx.task_id = format!("{}-{}-ready", ctx.pipeline_name, ctx.build_id);
send_stage_event(
&nevent_tx,
StagePhase::Prebake,
"ready",
StageOutcome::Success,
)
.await;
log::info!("Prebake done!");
Ok(())
}
/// Send a stage completion event to the notification queue.
async fn send_stage_event(
nevent_tx: &Option<NotificationEventSender>,
stage: StagePhase,
substage: &str,
outcome: StageOutcome,
) {
if let Some(tx) = nevent_tx {
let event = crate::notify::types::NotificationEvent {
id: Uuid::new_v4(),
stage,
substage: substage.to_string(),
outcome,
not_before: chrono::Utc::now(),
priority: 0,
effective_priority: 0,
retry_backoff: 1.0,
attempted: 0,
target: Default::default(),
};
let _ = tx.send(event).await;
}
}
-54
View File
@@ -1,54 +0,0 @@
# workshop-baker/src/prebake
**Generated:** 2026-04-22
**Commit:** 7b3b71a
Build environment setup: bootstrap, dependencies, security, hooks.
## STRUCTURE
```
prebake/
├── config.rs # PrebakeConfig YAML schema (environment, deps, cache)
├── stage.rs # PrebakeStage enum (Bootstrap→EarlyHook→DepsSystem→DepsUser→LateHook→Ready)
├── stage/
│ ├── bootstrap.rs # User creation (vulcan), sudo/doas setup
│ ├── environment.rs # Env var injection
│ ├── depssystem.rs # System package installation
│ ├── depsuser.rs # User package installation (cargo, npm, etc.)
│ └── hook.rs # Hook execution (pre/post stage)
├── env/ # Environment providers
│ ├── mod.rs # EnvProvider trait
│ ├── container.rs # Docker/container support
│ ├── firecracker.rs # Firecracker microVM
│ ├── baremetal.rs # Direct host execution
│ └── custom.rs # Custom env (todo!())
├── security.rs # Privilege dropping (unsafe libc calls)
├── event.rs # Event logging
├── error.rs # PrebakeError + BootstrapError
├── types.rs # Module re-exports
├── types/ # memsize, cache, builderconfig, architecture
├── constant.rs # Module constants
└── env.rs # Environment module root
```
## WHERE TO LOOK
| Task | Location |
|------|----------|
| Config loading | `config.rs:PrebakeConfig::from_yaml()` |
| Stage ordering | `stage.rs:PrebakeStage` enum |
| User setup | `stage/bootstrap.rs` - vulcan user creation |
| Security | `security.rs:72` - privilege dropping |
| Hooks | `stage/hook.rs` - pre/post stage hooks |
## CONVENTIONS
- **PrebakeStage**: Ordered enum Bootstrap→EarlyHook→DepsSystem→DepsUser→LateHook→Ready
- **ExecutionContext**: Carries task_id, username, env_vars through all stages
- **Target User**: "vulcan" (hardcoded in bootstrap.rs:6-24)
## ANTI-PATTERNS
1. **Hardcoded user** — "vulcan" username hardcoded throughout
2. **Unsafe blocks**`security.rs` has 5+ unsafe libc calls
3. **Incomplete** — custom.rs and apt.rs have `todo!()` stubs
4. **doas.conf** — bootstrap.rs:183: "We do not know how to write doas.conf lol"
+47 -19
View File
@@ -1,40 +1,68 @@
use workshop_engine::{ExecutionEvent, StreamType};
use crate::types::debug::DebugFeature;
use crate::bake::constant::PIPELINE_SKIP_ERRORCODE;
pub async fn event_receiver(
mut rx: tokio::sync::mpsc::Receiver<ExecutionEvent>,
pipeline_name: String,
build_id: String,
debug_flags: u32,
) {
let events_debug = DebugFeature::from_bits_retain(debug_flags).contains(DebugFeature::Events);
while let Some(event) = rx.recv().await {
match event {
ExecutionEvent::TaskStarted { task_id, timestamp } => {
log::debug!(
"[{}] [{}] [{}] Task started: {}",
pipeline_name,
build_id,
timestamp,
task_id
);
ExecutionEvent::TaskStarted { task_id, timestamp: _ } => {
if events_debug {
log::info!(
"[{}] [{}] Task started: {}",
pipeline_name,
build_id,
task_id
);
} else {
log::debug!(
"[{}] [{}] Task started: {}",
pipeline_name,
build_id,
task_id
);
}
}
ExecutionEvent::OutputChunk {
task_id,
task_id: _,
stream,
data,
} => match stream {
StreamType::Stdout => print!("[{}] [{}] [stdout] {}", pipeline_name, task_id, data),
StreamType::Stdout => print!("{}", data),
StreamType::Stderr => {
eprint!("[{}] [{}] [stderr] {}", pipeline_name, task_id, data)
eprint!("{}", data)
}
},
ExecutionEvent::TaskCompleted { task_id, result } => {
log::debug!(
"[{}] [{}] Task completed: {} (exit_code={}, duration={:?})",
pipeline_name,
build_id,
task_id,
result.exit_code,
result.duration
);
if result.success {
if events_debug {
log::info!(
"[{}] [{}] Task completed: {} (exit_code={}, duration={:?})",
pipeline_name, build_id, task_id,
result.exit_code, result.duration,
);
} else {
log::debug!(
"[{}] [{}] Task completed: {} (exit_code={}, duration={:?})",
pipeline_name, build_id, task_id,
result.exit_code, result.duration,
);
}
} else if result.exit_code == PIPELINE_SKIP_ERRORCODE {
log::info!("[{}] [{}] Task skipped: {}", pipeline_name, build_id, task_id);
} else {
log::warn!(
"[{}] [{}] Task completed with non-zero exitcode: {} (exit_code={}, duration={:?})",
pipeline_name, build_id, task_id,
result.exit_code, result.duration,
);
}
}
ExecutionEvent::TaskFailed { task_id, error } => {
log::error!(
+4 -1
View File
@@ -113,7 +113,7 @@ pub fn lookup_uid_by_name(username: &str) -> Result<u32, PrebakeError> {
}
}
// TODO: originally unsafe {pwent} here, maybe to check and remove
// NOTE: originally unsafe {pwent} here
// let pwent = pwent ;
if pwent.is_null() {
if let Ok(uid) = username.parse::<u32>() {
@@ -231,6 +231,9 @@ pub fn prebake_drop_privilege(
user
);
ctx.privileged = false;
let home = format!("/home/{}", user);
//unsafe { std::env::set_var("HOME", &home); }
ctx.env_vars.insert("HOME".to_string(), home);
}
Ok(())
}
+42 -91
View File
@@ -54,58 +54,56 @@ pub async fn bootstrap(
let bootstrap_config = config.bootstrap.clone().unwrap_or_default();
if let Some(raw) = bootstrap_config.custom {
match BootstrapSource::parse(&raw) {
BootstrapSource::Inline(script) => {
if ctx.dry_run {
log::info!(
"[DRY_RUN] User provided inline bootstrap script:\n{}",
script
);
return Ok(());
// Workshop-specific schemes requiring bakerd infrastructure
if let Some(rest) = raw.strip_prefix("workspace://") {
log::info!("Custom bootstrap from workspace: {}", rest);
unimplemented!("Workspace-relative bootstrap requires bakerd's workspace manager");
} else if let Some(name) = raw.strip_prefix("server://") {
log::info!("Custom bootstrap from server: {}", name);
unimplemented!("Server-provided bootstrap requires bakerd's script registry");
}
// Use getterURL for standard URL schemes: http, https, file, git, ...
else if workshop_getterurl::GetterUrl::parse(&raw).is_ok() {
log::info!("Fetching custom bootstrap via getterURL: {}", raw);
if ctx.standalone {
let temp_dir = tempfile::tempdir().map_err(BootstrapError::IOError)?;
let fetched = workshop_getterurl::fetch(&raw, temp_dir.path())
.await
.map_err(|e| BootstrapError::InvalidConfig(e.to_string()))?;
if fetched.is_dir() {
return Err(BootstrapError::InvalidConfig(
format!("getterURL returned a directory, expected a single file: {}", fetched.display())
));
}
log::debug!("Using inline bootstrap script ({} bytes)", script.len());
if ctx.standalone {
write_bootstrap_content(&bootstrap_path, script)?;
} else {
// Requires PIP(Pipeline Implicit Parameters) to be implemented
unimplemented!("Permission check is not implemented, failing...");
}
}
BootstrapSource::Workspace(relpath) => {
// Project-relative path, resolved by bakerd against the project root.
// Routing path relpath through bakerd's workspace manager ensures
// proper sandboxing and access control.
log::info!("Custom bootstrap from workspace: {}", relpath);
unimplemented!("Workspace-relative bootstrap requires bakerd's workspace manager");
}
BootstrapSource::Server(name) => {
// Script provided by the bake server (bakerd), keyed by name.
// The server maintains a registry of trusted bootstrap scripts.
log::info!("Custom bootstrap from server: {}", name);
unimplemented!("Server-provided bootstrap requires bakerd's script registry");
}
BootstrapSource::Http(url) => {
// HTTP download, similar to finalize::plugin::fetch::fetch_http_plugin.
// Should support: checksum verification, retry, timeout.
log::info!("Custom bootstrap from URL: {}", url);
log::info!(" Target: {}", bootstrap_path.display());
unimplemented!(
"HTTP bootstrap fetching requires bakerd-level network infrastructure"
);
}
BootstrapSource::File(path) => {
// Local file copy. This is the simplest case — we can implement
// this directly without bakerd infrastructure.
log::info!("Custom bootstrap from local file: {}", path);
tokio::fs::copy(&path, &bootstrap_path)
let content = tokio::fs::read_to_string(&fetched)
.await
.map_err(BootstrapError::IOError)?;
if ctx.dry_run {
log::debug!("[DRY_RUN] Fetched bootstrap script:\n{}", content);
return Ok(());
}
write_bootstrap_content(&bootstrap_path, content)?;
} else {
unimplemented!("Permission check is not implemented, failing...");
}
}
// Inline content (not a valid getterURL pattern)
else {
if ctx.dry_run {
log::debug!("[DRY_RUN] User provided inline bootstrap script:\n{}", raw);
return Ok(());
}
log::debug!("Using inline bootstrap script ({} bytes)", raw.len());
if ctx.standalone {
write_bootstrap_content(&bootstrap_path, raw.to_string())?;
} else {
unimplemented!("Permission check is not implemented, failing...");
}
}
} else {
let script = generate_bootstrap(&bootstrap_config, config, osinfo, ctx.standalone)?;
if ctx.dry_run {
log::info!("[DRY_RUN] Generated bootstrap script:\n{}", script);
log::debug!("[DRY_RUN] Generated bootstrap script:\n{}", script);
return Ok(());
}
log::debug!("Generated bootstrap script:\n{}", script);
@@ -135,53 +133,6 @@ fn write_bootstrap_content(bootstrap_path: &Path, content: String) -> Result<(),
Ok(())
}
/// Source type for a custom bootstrap script.
///
/// Mirrors the scheme-based resolution in `finalize::plugin::fetch` URL parsing,
/// and supports the schemes defined in the prebake.yml.tmpl spec:
/// `workspace://`, `file://`, `server://`, `https?://`, plus inline content.
enum BootstrapSource {
/// Inline script content (no scheme match).
Inline(String),
/// Project-relative file: `workspace://path/to/bootstrap.sh`
Workspace(String),
/// Script provided by bake server: `server://script-name`
Server(String),
/// HTTP(S) download: `https://example.com/bootstrap.sh`
Http(String),
/// Local file path: `file:///bin/bootstrap.sh`
File(String),
}
impl BootstrapSource {
/// Parses a raw custom bootstrap string into its source type.
///
/// Scheme detection follows the template spec:
/// - `workspace://` → Workspace
/// - `file://` → File (local path)
/// - `server://` → Server (bakerd registry)
/// - `http://`/`https://`→ Http
/// - No recognized scheme → Inline
fn parse(raw: &str) -> Self {
if let Some(rest) = raw.strip_prefix("workspace://") {
return BootstrapSource::Workspace(rest.to_string());
}
if let Some(path) = raw.strip_prefix("file://") {
return BootstrapSource::File(path.to_string());
}
if let Some(name) = raw.strip_prefix("server://") {
return BootstrapSource::Server(name.to_string());
}
if let Some(url) = raw
.strip_prefix("https://")
.or_else(|| raw.strip_prefix("http://"))
{
return BootstrapSource::Http(url.to_string());
}
BootstrapSource::Inline(raw.to_string())
}
}
/// Generates a bootstrap script based on configuration and target OS.
///
/// The generated script performs the following setup tasks:
-2
View File
@@ -57,7 +57,6 @@ pub async fn hook(
ctx: &ExecutionContext,
event_tx: EventSender,
) -> Result<(), ExecutionError> {
// dbg!(ctx);
let hook = match hook {
Some(h) => h,
None => {
@@ -79,7 +78,6 @@ pub async fn hook(
let result = engine
.execute_command_with_delta(&hook_cmd.command, ctx, &event_tx, &deltactx)
.await;
// dbg!(&result);
if let Err(e) = result {
log::error!("Hook {} failed: {:?}", index, e);
return Err(e);
+206
View File
@@ -0,0 +1,206 @@
//! Resource prefetching for standalone mode.
//!
//! Populates a [`ResourceRegistry`] by fetching plugins and templates
//! from their external URLs **before** the pipeline runs.
//!
//! In worker mode this is the baker's responsibility; in daemon mode
//! the bakerd would call the same lower-level primitives (or replace
//! this module entirely).
use std::path::{Path, PathBuf};
use crate::finalize::{FinalizeConfig, NotificationTemplateDef, PluginConfig, RawPluginMap};
use crate::types::resource::ResourceRegistry;
use workshop_getterurl;
/// Errors raised during resource prefetching.
#[derive(Debug, thiserror::Error)]
pub enum PrefetchError {
#[error("failed to construct resource URL for plugin '{name}': {detail}")]
UrlConstruction { name: String, detail: String },
#[error("failed to fetch '{url}': {source}")]
Fetch {
url: String,
#[source]
source: workshop_getterurl::GetterError,
},
#[error("file not found after resolving workspace path: {0}")]
FileNotFound(String),
#[error(transparent)]
Io(#[from] std::io::Error),
}
/// Fetch all external plugins and templates defined in [`FinalizeConfig`]
/// and register them into the given [`ResourceRegistry`].
///
/// * Plugins with the `late_binding` flag are **skipped** — they are
/// expected to be produced by the bake stage itself.
/// * Templates with a `use_` field (external URL) are fetched and
/// registered; inline templates are left as-is.
pub async fn prefetch_resources(
config: &FinalizeConfig,
registry: &mut ResourceRegistry,
) -> Result<(), PrefetchError> {
let _dir = tempfile::tempdir()?;
let target_base = _dir.path().to_path_buf();
prefetch_plugins(&config.plugin, registry, &target_base).await?;
prefetch_templates(&config.notification.templates, registry, &target_base).await?;
Ok(())
}
// ── plugins ──────────────────────────────────────────────────────────────
async fn prefetch_plugins(
plugins: &RawPluginMap,
registry: &mut ResourceRegistry,
cache: &Path,
) -> Result<(), PrefetchError> {
for (name, sources) in plugins {
let plugin_cfg = sources.values().next().ok_or_else(|| {
PrefetchError::UrlConstruction {
name: name.clone(),
detail: "no source URL configured".into(),
}
})?;
if plugin_cfg.late_binding {
log::debug!("[prefetch] Skipping late-binding plugin '{}'", name);
continue;
}
let source_url = sources.keys().next().unwrap(); // the URL string
let dest = fetch_plugin_resource(source_url, plugin_cfg, cache).await?;
registry.register(name, dest);
}
Ok(())
}
/// Build a full getter-url from a plugin source string and its associated
/// config, fetch it, and return the local path.
async fn fetch_plugin_resource(
source: &str,
cfg: &PluginConfig,
cache: &Path,
) -> Result<PathBuf, PrefetchError> {
// file:// — already local, just resolve and register
if let Some(path_str) = source.strip_prefix("file://") {
let abs = std::env::current_dir()?.join(path_str);
if !abs.exists() {
return Err(PrefetchError::FileNotFound(abs.display().to_string()));
}
return Ok(abs);
}
// workspace:// — resolve relative to cwd then treat as file
if let Some(rel) = source.strip_prefix("workspace://") {
let abs = std::env::current_dir()?.join(rel);
if !abs.exists() {
return Err(PrefetchError::FileNotFound(abs.display().to_string()));
}
return Ok(abs);
}
// Remote URL — add query params and fetch via getterurl
let url = enrich_url(source, cfg).map_err(|e| PrefetchError::UrlConstruction {
name: String::new(),
detail: e,
})?;
workshop_getterurl::fetch(&url, cache)
.await
.map_err(|source| PrefetchError::Fetch {
url: url.clone(),
source,
})
}
// ── templates ────────────────────────────────────────────────────────────
async fn prefetch_templates(
templates: &std::collections::HashMap<String, NotificationTemplateDef>,
registry: &mut ResourceRegistry,
cache: &Path,
) -> Result<(), PrefetchError> {
for (name, tmpl) in templates {
let Some(ref url) = tmpl.use_ else {
continue; // inline template, no fetch needed
};
// file:// / workspace:// — resolve locally
if let Some(path_str) = url.strip_prefix("file://") {
let abs = std::env::current_dir()?.join(path_str);
if abs.exists() {
registry.register(name, abs);
} else {
log::warn!("[prefetch] template '{}' file not found: {}", name, abs.display());
}
continue;
}
if let Some(rel) = url.strip_prefix("workspace://") {
let abs = std::env::current_dir()?.join(rel);
if abs.exists() {
registry.register(name, abs);
} else {
log::warn!("[prefetch] template '{}' workspace file not found: {}", name, abs.display());
}
continue;
}
// Remote template URL
match workshop_getterurl::fetch(url, cache).await {
Ok(path) => registry.register(name, path),
Err(e) => log::warn!("[prefetch] failed to fetch template '{}': {}", name, e),
}
}
Ok(())
}
// ── helpers ──────────────────────────────────────────────────────────────
/// Append `?checksum=&depth=` to a plugin source URL when the
/// corresponding [`PluginConfig`] fields are set.
fn enrich_url(source: &str, cfg: &PluginConfig) -> Result<String, String> {
// Validate the URL is parseable first
url::Url::parse(source).map_err(|e| format!("invalid URL '{}': {}", source, e))?;
let mut result = source.to_string();
let mut sep = if source.contains('?') { "&" } else { "?" };
if let Some(ref cs) = cfg.checksum {
result.push_str(sep);
result.push_str(&format!("checksum={}", cs));
sep = "&";
}
if let Some(true) = cfg.shallow {
result.push_str(sep);
result.push_str("depth=1");
}
Ok(result)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_enrich_url_checksum() {
let cfg = PluginConfig {
fallible: false,
retry: 0,
timeout_ms: 0,
late_binding: false,
checksum: Some("sha256:abc".into()),
shallow: None,
runtime: Default::default(),
config: serde_yaml::Value::Mapping(Default::default()),
};
let url = enrich_url("https://example.com/plugin.tar.gz", &cfg).unwrap();
assert_eq!(url, "https://example.com/plugin.tar.gz?checksum=sha256:abc");
}
}
+1
View File
@@ -1,2 +1,3 @@
pub mod buildstatus;
pub mod resource;
pub mod debug;
+176
View File
@@ -0,0 +1,176 @@
// Code in this module PARTIALLY or FULLY utilized AI Coding Agent
//! Debug mode feature flags for the HoneyBiscuitWorkshop pipeline.
//!
//! These flags are controlled by `--debug <FEATURES>` (comma-separated)
//! or the `HBW_DEBUG` environment variable. Each flag enables specific
//! behavioral changes that aid development and debugging.
//!
//! Unlike `RUST_LOG` (which controls log visibility), `DebugFeature`
//! controls **program behavior** — replacing real notifications with
//! dummy plugins, tracing internal event flow, etc.
use bitflags::bitflags;
bitflags! {
/// Bitmask of debug features enabled at runtime.
///
/// Propagated through [`ExecutionContext`](workshop_engine::ExecutionContext)
/// so all pipeline stages can conditionally alter their behavior.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct DebugFeature: u32 {
/// Notification debug: replace real adapters with `Dummy` (no emails/webhooks).
const NotifyDummy = 1 << 0;
/// Events: trace all `ExecutionEvent` (process lifecycle) and
/// `NotificationEvent` (stage outcomes) through the system.
const Events = 1 << 1;
/// Scheduler: log DAG topology, batch selection, and scheduling
/// decisions at `info!` level (normally `debug!`/gated).
const Scheduler = 1 << 2;
/// Script: display parsed script functions, generated final
/// scripts, and combinator chain details before execution.
const Script = 1 << 3;
}
}
impl DebugFeature {
/// Parse a comma-separated `--debug` string into a `DebugFeature` bitmask.
///
/// Unknown tokens are silently ignored. `"all"` enables all known features.
///
/// # Examples
///
/// ```
/// use workshop_baker::types::debug::DebugFeature;
///
/// let flags = DebugFeature::parse("notify.dummy,events");
/// assert!(flags.contains(DebugFeature::NotifyDummy));
/// assert!(flags.contains(DebugFeature::Events));
/// assert!(!flags.contains(DebugFeature::Scheduler));
///
/// let all = DebugFeature::parse("all");
/// assert!(all.contains(DebugFeature::all()));
/// ```
pub fn _parse(raw: &str) -> Self {
if raw.contains("all") {
return Self::all();
}
let mut flags = Self::empty();
for token in raw.split(',') {
let token = token.trim();
match token {
"notify.dummy" => flags.insert(Self::NotifyDummy),
"events" => flags.insert(Self::Events),
"scheduler" => flags.insert(Self::Scheduler),
"script" => flags.insert(Self::Script),
"" => {}
unknown => log::warn!("Unknown debug feature: '{}'", unknown),
}
}
flags
}
/// Print help for `--debug` features and return `true` if `help` was requested.
///
/// Called before `env_logger::init()`, so output goes to stdout directly.
/// The caller should exit immediately when this returns `true`.
pub fn print_help(raw: &str) -> bool {
if raw.trim() != "help" {
return false;
}
println!("Debug features (comma-separated, use with --debug <FEATURES>):");
println!();
println!(" notify.dummy Replace real notification adapters with Dummy");
println!(" (no emails/webhooks sent during debug)");
println!(" events Trace all ExecutionEvent and NotificationEvent");
println!(" through the event queue at info log level");
println!(" scheduler Show DAG topology, batch selection, and scheduling");
println!(" decisions each tick");
println!(" script Display parsed bake.sh functions, generated final");
println!(" scripts, and combinator chain details");
println!(" all Enable all features above");
println!();
println!("Examples:");
println!(" workshop-baker --debug notify.dummy,events ...");
println!(" workshop-baker --debug all ...");
println!(" HBW_DEBUG=scheduler,script workshop-baker ...");
true
}
/// Parse from an `Option<&str>`, returning empty flags for `None`.
pub fn parse(raw: Option<&str>) -> Self {
match raw {
Some(s) if !s.is_empty() => Self::_parse(s),
_ => Self::empty(),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn parse_empty() {
assert_eq!(DebugFeature::_parse(""), DebugFeature::empty());
}
#[test]
fn parse_single() {
let f = DebugFeature::_parse("notify.dummy");
assert!(f.contains(DebugFeature::NotifyDummy));
assert!(!f.contains(DebugFeature::Events));
assert!(!f.contains(DebugFeature::Scheduler));
}
#[test]
fn parse_multiple() {
let f = DebugFeature::_parse("notify.dummy,events");
assert!(f.contains(DebugFeature::NotifyDummy));
assert!(f.contains(DebugFeature::Events));
assert!(!f.contains(DebugFeature::Scheduler));
}
#[test]
fn parse_all() {
let f = DebugFeature::_parse("all");
assert!(f.contains(DebugFeature::NotifyDummy));
assert!(f.contains(DebugFeature::Events));
assert!(f.contains(DebugFeature::Scheduler));
assert!(f.contains(DebugFeature::Script));
}
#[test]
fn parse_whitespace() {
let f = DebugFeature::_parse("notify.dummy , scheduler");
assert!(f.contains(DebugFeature::NotifyDummy));
assert!(f.contains(DebugFeature::Scheduler));
}
#[test]
fn parse_unknown_is_ignored() {
let f = DebugFeature::_parse("notify.dummy,foobar,events");
assert!(f.contains(DebugFeature::NotifyDummy));
assert!(f.contains(DebugFeature::Events));
// unknown "foobar" just logged and skipped
}
#[test]
fn parse_optional_none() {
assert_eq!(DebugFeature::parse(None), DebugFeature::empty());
}
#[test]
fn parse_optional_empty_str() {
assert_eq!(DebugFeature::parse(Some("")), DebugFeature::empty());
}
#[test]
fn parse_optional_some() {
let f = DebugFeature::parse(Some("events"));
assert!(f.contains(DebugFeature::Events));
}
}
+14 -3
View File
@@ -6,15 +6,15 @@
//! exclusively through `resolve()` — the registry is treated as read-only
//! once the pipeline is running.
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::path::PathBuf;
/// A locally resolved resource with a logical name and filesystem path.
///
/// Created by the resource fetcher (bakerd in worker mode, or the baker
/// itself in standalone mode) and consumed by pipeline stages via the
/// [`ResourceRegistry`].
#[derive(Debug, Clone, PartialEq, Eq)]
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct Resource {
/// Logical name used for lookup (e.g., `"plugin/docker-push"`).
pub name: String,
@@ -36,7 +36,7 @@ impl Resource {
///
/// Populated during the fetch phase (before prebake) and then treated as
/// immutable for the remainder of the pipeline.
#[derive(Debug, Clone, Default)]
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct ResourceRegistry {
map: HashMap<String, Resource>,
}
@@ -78,6 +78,17 @@ impl ResourceRegistry {
pub fn is_empty(&self) -> bool {
self.map.is_empty()
}
/// Deserialize from a flat JSON: `{ "name": "/path", ... }`
pub fn from_json_str(json: &str) -> Result<Self, String> {
let flat: HashMap<String, PathBuf> = serde_json::from_str(json)
.map_err(|e| format!("invalid resource registry JSON: {}", e))?;
let mut reg = ResourceRegistry::new();
for (name, path) in flat {
reg.register(&name, path);
}
Ok(reg)
}
}
#[cfg(test)]
+39
View File
@@ -0,0 +1,39 @@
# WORKSHOP-ENGINE KNOWLEDGE BASE
## OVERVIEW
Core execution layer for HoneyBiscuitWorkshop. Library-only crate (no binary). Provides Engine executor, process abstraction, package manager interfaces, duration parsing, and typed error/event systems. Used by workshop-baker; depends on no other workspace crates.
## WHERE TO LOOK
| Task | File | Notes |
|------|------|-------|
| Script execution | `src/executor.rs` | `Engine::execute_script()` — spawn, stream, timeout, kill |
| Process abstraction | `src/child.rs` | `ManagedChild` trait; `TokioChild` (real), `MockChild` (hand-rolled, no mockall) |
| Configurable commands | `src/command.rs` | `CustomCommand` — name, command, env, working_dir, timeout |
| System PMs | `src/pm.rs` + `pm/{apt,pacman,dnf,apk}.rs` | `PackageManager` trait; `all_managers()`, `select(name)` |
| User PMs | `src/upm.rs` + `upm/{rust,go,python,custom}.rs` | `UserPackageManager` trait (async); `UPMSysDeps` |
| Repology endpoint | `src/repology.rs` + `repology/local.rs` | `RepologyEndpoint` enum; local instance support |
| Duration parsing | `src/time.rs` | `parse_duration_to_ms()`, serde helpers |
| Core types | `src/types.rs` | `ExecutionContext`, `ExecutionResult`, `ExecutionEvent`, `EventSender`, `StreamType`, `Engine` |
| Error types | `src/error.rs` | `ExecutionError`, `DependencyError`, `HasExitCode` trait, `EXITCODE_*` constants |
| Integration tests | `tests/engine_integration.rs` | 12 async tests; run with `cargo test --test engine_integration` |
## CONVENTIONS
- **Edition**: Rust 2024
- **Key deps**: tokio (full), thiserror, serde+serde_yaml, async-trait, chrono, os_info, duration-str, libc, lazy_static
- **Process mgmt**: `ManagedChild` trait abstracts over real/mock children. `TokioChild` wraps `tokio::process::Child`. `MockChild` is hand-rolled (no mockall dep); supports hanging, custom output, exit codes.
- **Package manager selection**: Use `pm::select("apt")` with explicit name — NEVER auto-detect by probing the filesystem.
- **Error mapping**: `HasExitCode` trait maps errors to numeric exit codes. `ExecutionError` maps to specific `EXITCODE_*`; `DependencyError` always maps to 1.
- **Re-exports**: `lib.rs` re-exports key types for convenience (`Engine`, `ExecutionContext`, `PackageManager`, `ExecutionError`, etc.)
- **Tests**: Inline `#[cfg(test)]` blocks in `pm.rs` (8 tests). Integration tests in `tests/engine_integration.rs` (12 async tests, separate binary). CI runs both.
- **Kill semantics**: `TokioChild::kill()` sends `SIGKILL` to entire process group via `libc::kill(-PGID, SIGKILL)` with `ESRCH` silently ignored.
## ANTI-PATTERNS
- **Do NOT auto-detect the host package manager** — always use `pm::select()` with an explicitly configured name. Production PM selection comes from environment config, not probing.
- **Do NOT use `detect()`/`adopt()` patterns** for PM selection — these don't exist in this crate, and callers must not invent them.
- **Do NOT depend on workshop-baker or other workspace crates** — engine is a leaf dependency. Types needed by external callers should flow through `ExecutionContext`, not reverse imports.
- **Do NOT use mockall or other mocking frameworks** — `MockChild` is the only mock mechanism. Keep it that way.
- **Do NOT use `unimplemented!()`** — all package manager implementations are complete. Stubs belong in workshop-baker, not here.
- **Do NOT add magic platform detection** — `os_info` is used only for mirror config mapping in PM impls. Do not extend its use to auto-detect behavior.
-51
View File
@@ -1,51 +0,0 @@
# workshop-engine/src
**Generated:** 2026-05-19
**Commit:** 28abc5d
Execution engine extracted from workshop-baker. Process management, package detection, resource isolation.
## STRUCTURE
```
src/
├── lib.rs # Module declarations + 10+ convenience re-exports
├── error.rs # ExecutionError + DependencyError + HasExitCode trait
├── types.rs # ExecutionContext, ExecutionResult, Engine, etc.
├── child.rs # ManagedChild + TokioChild process wrappers
├── command.rs # CustomCommand builder
├── executor.rs # Script execution
├── time.rs # Duration parsing helpers
├── pm.rs # PackageManager trait + detection
├── pm/ # Backends: apt, apk, dnf, pacman
├── upm.rs # UserPackageManager trait
├── upm/ # Backends: rust (cargo), python (pip), go, custom
├── repology.rs # RepologyEndpoint enum
└── repology/ # Package name resolution (local/)
```
## KEY EXPORTS
| Symbol | Type | Role |
|--------|------|------|
| ExecutionContext | struct | Pipeline context (task_id, username, env, etc.) |
| Engine | struct | Script executor (placeholder) |
| ExecutionError | enum | Error types with exit codes |
| HasExitCode | trait | Exit code protocol for all error types |
| ManagedChild | struct | Child process wrapper with managed lifecycle |
| TokioChild | struct | Tokio-based child process wrapper |
| CustomCommand | struct | Builder for command execution |
| PackageManager | trait | System package manager abstraction |
| UserPackageManager | trait | User-level package manager abstraction |
| RepologyEndpoint | enum | Package name resolution endpoint |
| ExecutionEvent | enum | Event stream (TaskStarted, OutputChunk, etc.) |
| ExecutionResult | struct | Command execution result (exit_code, stdout, stderr) |
| EventSender | type alias | `Option<mpsc::Sender<ExecutionEvent>>` |
## CONVENTIONS
- **Exit codes**: Centralized in `error.rs` — EXITCODE_OK=0 through EXITCODE_PRIV_DROP_FAILED=201
- **ExecutionContext**: Defaults to current directory, bash shell, non-privileged
- **PackageManagers**: Auto-detected via `os_info`, fallback chain (apt→apk→dnf→pacman)
- **Engine**: Currently a unit struct with `execute_script()` method
- **All errors implement HasExitCode**: Enables uniform error propagation up to CLI
## ANTI-PATTERNS
1. **Engine is a stub**`types.rs:99``pub struct Engine;` with no fields, minimal implementation
+3 -3
View File
@@ -21,8 +21,8 @@ pub enum ExecutionError {
#[error("Invalid command: {0}")]
InvalidCommand(String),
#[error("Execution failed: {0}")]
ExecutionFailed(String),
#[error("Task {task_id} failed with exit code {exit_code}")]
ExecutionFailed { task_id: String, exit_code: i32 },
#[error("Permission denied: uid={euid}, gid={egid}: {reason}")]
PermissionDenied {
@@ -48,7 +48,7 @@ impl HasExitCode for ExecutionError {
fn exit_code(&self) -> i32 {
match self {
ExecutionError::InvalidCommand(_) => EXITCODE_EXECUTION_ERROR,
ExecutionError::ExecutionFailed(_) => EXITCODE_EXECUTION_ERROR,
ExecutionError::ExecutionFailed { .. } => EXITCODE_EXECUTION_ERROR,
ExecutionError::PermissionDenied { .. } => EXITCODE_GENERAL_ERROR,
ExecutionError::Timeout => EXITCODE_TIMEOUT,
ExecutionError::IoError(_) => EXITCODE_IO_ERROR,
+84 -58
View File
@@ -2,6 +2,7 @@ use chrono::Utc;
use std::path::PathBuf;
use std::process::Stdio;
use tokio::io::AsyncReadExt;
use tokio::io::AsyncWriteExt;
use tokio::process::Command;
use tokio::sync::mpsc;
use tokio::time::timeout;
@@ -12,7 +13,7 @@ use crate::types::{
DeltaExecutionContext, EventSender, ExecutionContext, ExecutionError, ExecutionEvent,
ExecutionResult, StreamType,
};
const STDOUT_BUF_CAP: usize = 2 * 1024 * 1024;
impl crate::types::Engine {
pub fn new() -> Self {
Self
@@ -87,7 +88,7 @@ impl crate::types::Engine {
.env("HBW_TASKID", &ctx.task_id)
.env("HBW_PIPELINE", &ctx.pipeline_name)
.env("HBW_USERNAME", &ctx.username)
.stdin(Stdio::null())
.stdin(if ctx.stdin.is_some() { Stdio::piped() } else { Stdio::null() })
.stdout(Stdio::piped())
.stderr(Stdio::piped());
}
@@ -121,10 +122,19 @@ impl crate::types::Engine {
#[cfg(unix)]
command.process_group(0);
let child = command.spawn().map_err(|e| {
let mut child = command.spawn().map_err(|e| {
ExecutionError::InvalidCommand(format!("Failed to spawn command: {}", e))
})?;
// Write piped stdin (e.g. from @pipe). Consumed once per execution.
if let Some(ref data) = ctx.stdin {
if let Some(mut stdin_pipe) = child.stdin.take() {
stdin_pipe.write_all(data).await.map_err(|e| {
ExecutionError::IoError(format!("Failed to write stdin: {}", e))
})?;
}
}
self.execute_child(TokioChild(child), ctx, event_tx).await
}
@@ -149,11 +159,8 @@ impl crate::types::Engine {
.await;
}
let task_id = ctx.task_id.clone();
let (stdout_tx, mut stdout_rx) = mpsc::channel::<String>(1024);
let (stderr_tx, mut stderr_rx) = mpsc::channel::<String>(1024);
let mut stdout_buf = String::new();
let mut stderr_buf = String::new();
let mut child_stdout = child.take_stdout();
let mut child_stderr = child.take_stderr();
@@ -198,6 +205,32 @@ impl crate::types::Engine {
}
});
// Forwarder: drains channels in real-time, emits OutputChunk events,
// and accumulates output into 2 MiB ring buffers.
let task_id_ev = ctx.task_id.clone();
let event_tx_fwd = event_tx.clone();
let forwarder = tokio::spawn(async move {
let mut stdout_buf: Vec<u8> = Vec::new();
let mut stderr_buf: Vec<u8> = Vec::new();
loop {
tokio::select! {
biased;
Some(data) = stdout_rx.recv() => {
emit_chunk(&event_tx_fwd, &task_id_ev, StreamType::Stdout, &data).await;
stdout_buf.extend_from_slice(data.as_bytes());
truncate_buf(&mut stdout_buf);
}
Some(data) = stderr_rx.recv() => {
emit_chunk(&event_tx_fwd, &task_id_ev, StreamType::Stderr, &data).await;
stderr_buf.extend_from_slice(data.as_bytes());
truncate_buf(&mut stderr_buf);
}
else => break,
}
}
(stdout_buf, stderr_buf)
});
let exit_code = if let Some(timeout_duration) = ctx.timeout {
match timeout(timeout_duration, child.wait()).await {
Ok(Ok(code)) => code,
@@ -226,12 +259,7 @@ impl crate::types::Engine {
drop(stderr_tx);
let _ = stdout_handle.await;
let _ = stderr_handle.await;
while let Some(data) = stdout_rx.recv().await {
stdout_buf.push_str(&data);
}
while let Some(data) = stderr_rx.recv().await {
stderr_buf.push_str(&data);
}
let _ = forwarder.await;
return Err(ExecutionError::Timeout);
}
}
@@ -246,31 +274,9 @@ impl crate::types::Engine {
let _ = stdout_handle.await;
let _ = stderr_handle.await;
while let Some(data) = stdout_rx.recv().await {
stdout_buf.push_str(&data);
if let Some(tx) = event_tx {
let _ = tx
.send(ExecutionEvent::OutputChunk {
task_id: task_id.clone(),
stream: StreamType::Stdout,
data: data.clone(),
})
.await;
}
}
while let Some(data) = stderr_rx.recv().await {
stderr_buf.push_str(&data);
if let Some(tx) = event_tx {
let _ = tx
.send(ExecutionEvent::OutputChunk {
task_id: task_id.clone(),
stream: StreamType::Stderr,
data: data.clone(),
})
.await;
}
}
let (stdout_buf, stderr_buf) = forwarder
.await
.map_err(|e| ExecutionError::IoError(format!("Forwarder task panicked: {}", e)))?;
let duration = start_time.elapsed();
@@ -279,32 +285,26 @@ impl crate::types::Engine {
exit_code,
success: exit_code == 0,
duration,
stdout: stdout_buf,
stderr: stderr_buf,
stdout: String::from_utf8_lossy(&stdout_buf).into_owned(),
stderr: String::from_utf8_lossy(&stderr_buf).into_owned(),
};
if let Some(tx) = event_tx {
let event = if result.success {
ExecutionEvent::TaskCompleted {
let _ = tx
.send(ExecutionEvent::TaskCompleted {
task_id: ctx.task_id.clone(),
result: result.clone(),
}
} else {
ExecutionEvent::TaskFailed {
task_id: ctx.task_id.clone(),
error: format!("Exit code: {}", exit_code),
}
};
let _ = tx.send(event).await;
})
.await;
}
if result.success {
Ok(result)
} else {
Err(ExecutionError::ExecutionFailed(format!(
"Task {} failed with exit code {}",
ctx.task_id, exit_code
)))
Err(ExecutionError::ExecutionFailed {
task_id: ctx.task_id.clone(),
exit_code,
})
}
}
@@ -342,6 +342,30 @@ impl crate::types::Engine {
Ok(combined_result)
}
}
/// Truncate a buffer to [`STDOUT_BUF_CAP`] bytes, discarding the oldest data.
fn truncate_buf(buf: &mut Vec<u8>) {
if buf.len() > STDOUT_BUF_CAP {
buf.drain(0..buf.len() - STDOUT_BUF_CAP);
}
}
/// Emit an `OutputChunk` event if the channel exists.
async fn emit_chunk(
tx: &EventSender,
task_id: &str,
stream: StreamType,
data: &str,
) {
if let Some(tx) = tx {
let _ = tx
.send(ExecutionEvent::OutputChunk {
task_id: task_id.to_string(),
stream,
data: data.to_string(),
})
.await;
}
}
impl Default for crate::types::Engine {
fn default() -> Self {
@@ -369,6 +393,7 @@ mod tests {
dry_run: false,
privileged: false,
standalone: false,
stdin: None,
}
}
@@ -425,12 +450,12 @@ mod tests {
assert!(result.is_err());
assert!(matches!(
result.unwrap_err(),
ExecutionError::ExecutionFailed(_)
ExecutionError::ExecutionFailed { .. }
));
}
#[tokio::test]
async fn exit_nonzero_sends_task_failed_event() {
async fn exit_nonzero_sends_task_completed_event() {
let engine = Engine::new();
let (tx, mut rx) = mpsc::channel(64);
let child = MockChild::with_exit_code(42);
@@ -440,9 +465,10 @@ mod tests {
// Skip TaskStarted
let _ = rx.recv().await;
let event = rx.recv().await;
assert!(matches!(event, Some(ExecutionEvent::TaskFailed { .. })));
if let Some(ExecutionEvent::TaskFailed { error, .. }) = event {
assert!(error.contains("42"), "error should mention exit code 42");
assert!(matches!(event, Some(ExecutionEvent::TaskCompleted { .. })));
if let Some(ExecutionEvent::TaskCompleted { result, .. }) = event {
assert_eq!(result.exit_code, 42);
assert!(!result.success);
}
}
+10
View File
@@ -11,6 +11,7 @@ pub struct ExecutionContext {
pub task_id: String,
pub pipeline_name: String,
pub username: String,
pub build_id: String,
pub working_dir: PathBuf,
pub env_vars: HashMap<String, String>,
pub timeout: Option<Duration>,
@@ -18,6 +19,12 @@ pub struct ExecutionContext {
pub dry_run: bool,
pub privileged: bool,
pub standalone: bool,
/// stdin data piped by the scheduler (e.g. via `@pipe`).
/// Consumed (taken) by engine during execution.
pub stdin: Option<Vec<u8>>,
/// Bitmask of active debug features, decoded by `workshop-baker`.
/// 0 means no debug features. See `DebugFeature` for supported flags.
pub debug_flags: u32,
}
#[derive(Debug, Clone, Default)]
@@ -33,6 +40,7 @@ impl Default for ExecutionContext {
task_id: String::new(),
pipeline_name: String::new(),
username: String::new(),
build_id: String::new(),
working_dir: std::env::current_dir().unwrap_or_else(|_| PathBuf::from("/")),
env_vars: HashMap::new(),
timeout: None,
@@ -40,6 +48,8 @@ impl Default for ExecutionContext {
dry_run: false,
privileged: false,
standalone: false,
debug_flags: 0,
stdin: None,
}
}
}
@@ -27,6 +27,7 @@ fn create_context() -> ExecutionContext {
dry_run: false,
privileged: false,
standalone: false,
stdin: None,
}
}
+104
View File
@@ -0,0 +1,104 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 4
[[package]]
name = "equivalent"
version = "1.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f"
[[package]]
name = "fixedbitset"
version = "0.5.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1d674e81391d1e1ab681a28d99df07927c6d4aa5b027d7da16ba32d1d21ecd99"
[[package]]
name = "hashbrown"
version = "0.17.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a"
[[package]]
name = "indexmap"
version = "2.14.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9"
dependencies = [
"equivalent",
"hashbrown",
]
[[package]]
name = "petgraph"
version = "0.7.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3672b37090dbd86368a4145bc067582552b29c27377cad4e0a306c97f9bd7772"
dependencies = [
"fixedbitset",
"indexmap",
]
[[package]]
name = "proc-macro2"
version = "1.0.106"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934"
dependencies = [
"unicode-ident",
]
[[package]]
name = "quote"
version = "1.0.45"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924"
dependencies = [
"proc-macro2",
]
[[package]]
name = "syn"
version = "2.0.117"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99"
dependencies = [
"proc-macro2",
"quote",
"unicode-ident",
]
[[package]]
name = "thiserror"
version = "2.0.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4288b5bcbc7920c07a1149a35cf9590a2aa808e0bc1eafaade0b80947865fbc4"
dependencies = [
"thiserror-impl",
]
[[package]]
name = "thiserror-impl"
version = "2.0.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ebc4ee7f67670e9b64d05fa4253e753e016c6c95ff35b89b7941d6b856dec1d5"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "unicode-ident"
version = "1.0.24"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75"
[[package]]
name = "workshop-schedule"
version = "0.1.0"
dependencies = [
"petgraph",
"thiserror",
]
+9
View File
@@ -0,0 +1,9 @@
[package]
name = "workshop-schedule"
version = "0.1.0"
edition = "2024"
description = "Semi-general DAG scheduling library for CI/CD pipelines"
[dependencies]
petgraph = "0.7"
thiserror = "2"
+780
View File
@@ -0,0 +1,780 @@
// Code in this module PARTIALLY or FULLY utilized AI Coding Agent
use petgraph::stable_graph::{NodeIndex, StableGraph};
use petgraph::Direction;
use petgraph::visit::EdgeRef;
use std::collections::{BTreeMap, BTreeSet, HashMap};
use thiserror::Error;
use crate::edge::EdgeKind;
use crate::node::{DagNode, NodeColor};
/// Errors that can occur during DAG operations.
#[derive(Debug, Error)]
pub enum DagError {
#[error("duplicate node name: {0}")]
DuplicateNode(String),
#[error("node not found: {0}")]
NodeNotFound(String),
#[error("node not ready: {0}")]
NotReady(String),
#[error("already popped: {0}")]
AlreadyPopped(String),
#[error("circular dependency detected among: {0:?}")]
Cycle(Vec<String>),
#[error("parallel group can never complete: {0:?}")]
GroupDeadlock(Vec<String>),
}
/// Internal per-node metadata.
enum NodeState {
/// Has unsatisfied dependencies.
Pending,
/// All dependencies satisfied, in the ready set.
Ready,
/// Has been popped — done executing.
Done,
}
/// Internal per-group metadata.
#[derive(Debug)]
struct GroupMeta {
/// Node indices belonging to this group.
members: Vec<NodeIndex>,
/// Total number of members (constant after construction).
total: usize,
}
/// A DAG (Directed Acyclic Graph) for scheduling pipeline nodes.
///
/// Nodes implement [`DagNode`] and carry a [`NodeColor`] that determines
/// sequential vs parallel semantics. Edges carry an [`EdgeKind`] distinguishing
/// explicit `@after` from implicit pipeline chains.
///
/// The DAG maintains a ready set (nodes with in-degree 0) in deterministic
/// BTreeSet order, and provides methods to pop nodes **by name** — fixing the
/// limitation of `topological-sort` that could only pop the first ready node.
pub struct Dag<N: DagNode> {
graph: StableGraph<N, EdgeKind>,
name_to_idx: HashMap<String, NodeIndex>,
state: HashMap<NodeIndex, NodeState>,
in_degree: HashMap<NodeIndex, usize>,
/// Ready nodes, sorted by name for deterministic iteration.
ready_set: BTreeSet<(String, NodeIndex)>,
/// Named groups and their members.
groups: BTreeMap<String, GroupMeta>,
node_count: usize,
}
impl<N: DagNode> Dag<N> {
/// Create an empty DAG.
pub fn new() -> Self {
Self {
graph: StableGraph::new(),
name_to_idx: HashMap::new(),
state: HashMap::new(),
in_degree: HashMap::new(),
ready_set: BTreeSet::new(),
groups: BTreeMap::new(),
node_count: 0,
}
}
/// Number of nodes (including pending, ready, and done).
pub fn node_count(&self) -> usize {
self.node_count
}
/// True if no nodes remain in the DAG (all popped).
pub fn is_empty(&self) -> bool {
self.ready_set.is_empty()
&& self.state.values().all(|s| matches!(s, NodeState::Done))
}
// ═══════════════════════════════════════════════════════════════════
// Construction
// ═══════════════════════════════════════════════════════════════════
/// Add a node to the DAG.
///
/// Returns an error if a node with the same name already exists.
pub fn add_node(&mut self, node: N) -> Result<(), DagError> {
let name = node.name().to_string();
if self.name_to_idx.contains_key(&name) {
return Err(DagError::DuplicateNode(name));
}
let color = node.color(); // extract before move
let idx = self.graph.add_node(node);
self.name_to_idx.insert(name.clone(), idx);
self.state.insert(idx, NodeState::Ready);
self.in_degree.insert(idx, 0);
self.ready_set.insert((name.clone(), idx));
self.node_count += 1;
// Register named groups
if let NodeColor::Named(group) = &color {
self.groups
.entry(group.clone())
.or_insert_with(|| GroupMeta {
members: Vec::new(),
total: 0,
})
.members
.push(idx);
}
Ok(())
}
/// Freeze group totals (call after all nodes added, before any pop).
///
/// Locks in `total` for each group. Named groups whose `total` is less
/// than 2 are harmless but wasteful — they'll fire solo like sequential.
pub fn freeze(&mut self) {
for meta in self.groups.values_mut() {
meta.total = meta.members.len();
}
}
/// Add a dependency edge from `from` to `to`.
///
/// `to` depends on `from` — `to` won't be ready until `from` is popped.
pub fn add_edge(
&mut self,
from: &str,
to: &str,
kind: EdgeKind,
) -> Result<(), DagError> {
let from_idx = self
.name_to_idx
.get(from)
.copied()
.ok_or_else(|| DagError::NodeNotFound(from.to_string()))?;
let to_idx = self
.name_to_idx
.get(to)
.copied()
.ok_or_else(|| DagError::NodeNotFound(to.to_string()))?;
self.graph.add_edge(from_idx, to_idx, kind);
// Increment in-degree of `to`
let deg = self.in_degree.get_mut(&to_idx).unwrap();
*deg += 1;
// Remove `to` from ready set if it was there
let to_name = self.graph[to_idx].name().to_string();
self.ready_set.remove(&(to_name.clone(), to_idx));
*self.state.get_mut(&to_idx).unwrap() = NodeState::Pending;
// Sanity: check for self-loop (cycle detection lite)
if from == to {
return Err(DagError::Cycle(vec![from.to_string()]));
}
Ok(())
}
// ═══════════════════════════════════════════════════════════════════
// Queries
// ═══════════════════════════════════════════════════════════════════
/// Name → color lookup. O(1).
pub fn color_of(&self, name: &str) -> Option<NodeColor> {
let idx = self.name_to_idx.get(name)?;
Some(self.graph[*idx].color())
}
/// True if the node is sequential (not parallel). O(1).
pub fn is_sequential(&self, name: &str) -> bool {
self.color_of(name)
.map(|c| c.is_sequential())
.unwrap_or(false)
}
/// True if the node is a wildcard parallel. O(1).
pub fn is_wildcard(&self, name: &str) -> bool {
self.color_of(name)
.map(|c| c.is_wildcard())
.unwrap_or(false)
}
/// True if the node is parallel (wildcard or named). O(1).
pub fn is_parallel(&self, name: &str) -> bool {
self.color_of(name)
.map(|c| c.is_parallel())
.unwrap_or(false)
}
/// Group name for a named parallel node, or `None`. O(1).
pub fn group_of(&self, name: &str) -> Option<String> {
let idx = self.name_to_idx.get(name)?;
self.graph[*idx].color().group_name().map(|s| s.to_string())
}
/// In-degree of a node. O(1).
pub fn in_degree(&self, name: &str) -> Option<usize> {
let idx = self.name_to_idx.get(name)?;
self.in_degree.get(idx).copied()
}
/// Get the edge kind between two nodes. O(e).
pub fn edge_kind(&self, from: &str, to: &str) -> Option<EdgeKind> {
let from_idx = self.name_to_idx.get(from)?;
let to_idx = self.name_to_idx.get(to)?;
self.graph
.find_edge(*from_idx, *to_idx)
.map(|e| self.graph[e])
}
/// Reference to the node data. O(1).
pub fn node(&self, name: &str) -> Option<&N> {
let idx = self.name_to_idx.get(name)?;
Some(&self.graph[*idx])
}
/// True if the node is fallible.
pub fn is_fallible(&self, name: &str) -> bool {
self.node(name).map(|n| n.fallible()).unwrap_or(false)
}
/// Number of ready nodes (in-degree 0, not yet popped).
pub fn ready_count(&self) -> usize {
self.ready_set.len()
}
// ═══════════════════════════════════════════════════════════════════
// Ready set queries
// ═══════════════════════════════════════════════════════════════════
/// All ready node names, in deterministic BTreeSet order.
pub fn ready_names(&self) -> Vec<&str> {
self.ready_set
.iter()
.map(|(name, _)| name.as_str())
.collect()
}
/// Ready sequential node names.
pub fn ready_sequential(&self) -> Vec<&str> {
self.ready_set
.iter()
.filter(|(name, _)| self.is_sequential(name))
.map(|(name, _)| name.as_str())
.collect()
}
/// Ready wildcard node names.
pub fn ready_wildcards(&self) -> Vec<&str> {
self.ready_set
.iter()
.filter(|(name, _)| self.is_wildcard(name))
.map(|(name, _)| name.as_str())
.collect()
}
/// Ready nodes grouped by their parallel group name.
///
/// Returns `(group_name, ready_count, total_count)` for each group that
/// has at least one ready member.
pub fn ready_groups(&self) -> Vec<(&str, usize, usize)> {
let mut result = Vec::new();
for (gname, meta) in &self.groups {
let ready_count = meta
.members
.iter()
.filter(|idx| matches!(self.state.get(idx), Some(NodeState::Ready)))
.count();
if ready_count > 0 {
result.push((gname.as_str(), ready_count, meta.total));
}
}
result
}
/// Pick a fireable batch per bake scheduling rules:
///
/// 1. Named groups: flood-fill from ready members through same-color
/// implicit edges. If the whole group is reachable, fire it.
/// 2. Wildcards join any fireable group.
/// 3. Otherwise, wildcards form their own batch.
/// 4. Returns `None` only when truly stuck.
pub fn pick_batch(&self) -> Option<Vec<String>> {
use std::collections::{HashSet, VecDeque};
let wildcards: Vec<String> = self
.ready_set
.iter()
.filter(|(name, _)| self.is_wildcard(name))
.map(|(name, _)| name.clone())
.collect();
for (_gname, meta) in &self.groups {
if meta.total == 0 {
continue;
}
// Flood-fill from ready members through same-color implicit edges
let mut reached: HashSet<NodeIndex> = HashSet::new();
let mut queue: VecDeque<NodeIndex> = VecDeque::new();
for idx in &meta.members {
if matches!(self.state.get(idx), Some(NodeState::Ready)) {
reached.insert(*idx);
queue.push_back(*idx);
}
}
while let Some(from) = queue.pop_front() {
let from_color = self.graph[from].color();
if from_color.is_sequential() {
continue;
}
for edge in self.graph.edges_directed(from, Direction::Outgoing) {
if !edge.weight().is_implicit() {
continue;
}
let to = edge.target();
let to_color = self.graph[to].color();
if to_color.is_sequential() {
continue;
}
let same = match (&from_color, &to_color) {
(NodeColor::Wildcard, NodeColor::Wildcard) => true,
(NodeColor::Named(a), NodeColor::Named(b)) => a.as_str() == b.as_str(),
_ => false,
};
if !same {
continue;
}
if !meta.members.contains(&to) {
continue;
}
if reached.insert(to) {
queue.push_back(to);
}
}
}
if reached.len() == meta.total {
let batch: Vec<String> = meta
.members
.iter()
.map(|idx| self.graph[*idx].name().to_string())
.collect();
let mut batch = batch;
batch.extend(wildcards);
return Some(batch);
}
}
// No fireable named group — wildcards fire alone
if !wildcards.is_empty() {
return Some(wildcards);
}
None
}
// ═══════════════════════════════════════════════════════════════════
// Scheduling operations
// ═══════════════════════════════════════════════════════════════════
/// Pop a specific node by name.
///
/// Decrements in-degree of all successors. Those reaching 0 become ready.
/// This is the key fix over `topological-sort` — we pop by name, not by
/// arbitrary HashMap iteration order.
pub fn pop(&mut self, name: &str) -> Result<(), DagError> {
let idx = self
.name_to_idx
.get(name)
.copied()
.ok_or_else(|| DagError::NodeNotFound(name.to_string()))?;
match self.state.get(&idx) {
None => return Err(DagError::NodeNotFound(name.to_string())),
Some(NodeState::Done) => return Err(DagError::AlreadyPopped(name.to_string())),
Some(NodeState::Pending) => return Err(DagError::NotReady(name.to_string())),
Some(NodeState::Ready) => {}
}
// Remove from ready set
let owned_name = name.to_string();
self.ready_set.remove(&(owned_name, idx));
// Decrement in-degree of all successors
let successors: Vec<NodeIndex> = self
.graph
.neighbors_directed(idx, Direction::Outgoing)
.collect();
for succ in successors {
let deg = self.in_degree.get_mut(&succ).unwrap();
*deg -= 1;
if *deg == 0 {
// Successor becomes ready
let succ_name = self.graph[succ].name().to_string();
self.ready_set.insert((succ_name, succ));
*self.state.get_mut(&succ).unwrap() = NodeState::Ready;
}
}
// Mark as done
*self.state.get_mut(&idx).unwrap() = NodeState::Done;
Ok(())
}
/// Pop a sequential node from the ready set.
///
/// Returns the name of the popped node, or `None` if no sequential
/// node is ready. Convenience wrapper: finds first ready sequential,
/// pops it by name.
pub fn pop_sequential(&mut self) -> Result<Option<String>, DagError> {
let name = self
.ready_set
.iter()
.find(|(name, _)| self.is_sequential(name))
.map(|(name, _)| name.clone());
if let Some(ref name) = name {
self.pop(name)?;
}
Ok(name)
}
// ═══════════════════════════════════════════════════════════════════
// Deadlock / cycle detection
// ═══════════════════════════════════════════════════════════════════
/// True if ready nodes exist but no batch can fire (potential deadlock).
pub fn is_stalled(&self) -> bool {
!self.ready_set.is_empty() && self.pick_batch().is_none()
}
/// Names of groups that are causing a stall (incomplete groups,
/// including those with zero ready members but non-Done pending members).
pub fn stalled_groups(&self) -> Vec<&str> {
let mut result = Vec::new();
for (gname, meta) in &self.groups {
let ready_count = meta
.members
.iter()
.filter(|idx| matches!(self.state.get(idx), Some(NodeState::Ready)))
.count();
let pending_count = meta
.members
.iter()
.filter(|idx| matches!(self.state.get(idx), Some(NodeState::Pending)))
.count();
if meta.total > 0 && (ready_count > 0 || pending_count > 0) && ready_count < meta.total {
result.push(gname.as_str());
}
}
result
}
/// All nodes that haven't been popped yet (for cycle diagnostics).
pub fn remaining(&self) -> Vec<&str> {
self.state
.iter()
.filter(|(_, s)| !matches!(s, NodeState::Done))
.map(|(idx, _)| self.graph[*idx].name())
.collect()
}
}
impl<N: DagNode> Default for Dag<N> {
fn default() -> Self {
Self::new()
}
}
// ── Tests ──────────────────────────────────────────────────────────────────
#[cfg(test)]
mod tests {
use super::*;
#[derive(Clone)]
struct TestNode {
name: String,
color: NodeColor,
fallible: bool,
}
impl DagNode for TestNode {
fn name(&self) -> &str {
&self.name
}
fn color(&self) -> NodeColor {
self.color.clone()
}
fn fallible(&self) -> bool {
self.fallible
}
}
fn seq(name: &str) -> TestNode {
TestNode {
name: name.into(),
color: NodeColor::Sequential,
fallible: false,
}
}
fn wild(name: &str) -> TestNode {
TestNode {
name: name.into(),
color: NodeColor::Wildcard,
fallible: false,
}
}
fn group(name: &str, g: &str) -> TestNode {
TestNode {
name: name.into(),
color: NodeColor::Named(g.into()),
fallible: false,
}
}
// ── Basic construction ────────────────────────────────────────────
#[test]
fn empty_dag() {
let dag = Dag::<TestNode>::new();
assert!(dag.is_empty());
assert_eq!(dag.node_count(), 0);
}
#[test]
fn add_single_node() {
let mut dag = Dag::new();
dag.add_node(seq("a")).unwrap();
dag.freeze();
assert!(!dag.is_empty());
assert_eq!(dag.ready_count(), 1);
assert_eq!(dag.ready_names(), vec!["a"]);
assert!(dag.is_sequential("a"));
}
#[test]
fn duplicate_node_rejected() {
let mut dag = Dag::new();
dag.add_node(seq("a")).unwrap();
assert!(dag.add_node(seq("a")).is_err());
}
// ── Edges & in-degree ─────────────────────────────────────────────
#[test]
fn simple_dependency() {
let mut dag = Dag::new();
dag.add_node(seq("a")).unwrap();
dag.add_node(seq("b")).unwrap();
dag.add_edge("a", "b", EdgeKind::Explicit).unwrap();
dag.freeze();
assert!(dag.is_sequential("a"));
assert_eq!(dag.ready_names(), vec!["a"]);
assert_eq!(dag.in_degree("a"), Some(0));
assert_eq!(dag.in_degree("b"), Some(1));
}
#[test]
fn pop_advances_dependents() {
let mut dag = Dag::new();
dag.add_node(seq("a")).unwrap();
dag.add_node(seq("b")).unwrap();
dag.add_node(seq("c")).unwrap();
dag.add_edge("a", "b", EdgeKind::Explicit).unwrap();
dag.add_edge("b", "c", EdgeKind::Explicit).unwrap();
dag.freeze();
assert_eq!(dag.ready_names(), vec!["a"]);
dag.pop("a").unwrap();
assert_eq!(dag.ready_names(), vec!["b"]);
dag.pop("b").unwrap();
assert_eq!(dag.ready_names(), vec!["c"]);
dag.pop("c").unwrap();
assert!(dag.is_empty());
}
#[test]
fn pop_not_ready_rejected() {
let mut dag = Dag::new();
dag.add_node(seq("a")).unwrap();
dag.add_node(seq("b")).unwrap();
dag.add_edge("a", "b", EdgeKind::Explicit).unwrap();
dag.freeze();
assert!(dag.pop("b").is_err()); // b not ready
}
#[test]
fn pop_twice_rejected() {
let mut dag = Dag::new();
dag.add_node(seq("a")).unwrap();
dag.freeze();
dag.pop("a").unwrap();
assert!(dag.pop("a").is_err()); // already done
}
// ── Node coloring ─────────────────────────────────────────────────
#[test]
fn color_queries() {
let mut dag = Dag::new();
dag.add_node(seq("s")).unwrap();
dag.add_node(wild("w")).unwrap();
dag.add_node(group("g", "build")).unwrap();
dag.freeze();
assert!(dag.is_sequential("s"));
assert!(dag.is_wildcard("w"));
assert_eq!(dag.group_of("g"), Some("build".to_string()));
assert!(dag.is_parallel("w"));
assert!(dag.is_parallel("g"));
assert!(!dag.is_parallel("s"));
}
#[test]
fn fallible_query() {
let mut dag = Dag::new();
dag.add_node(TestNode {
name: "f".into(),
color: NodeColor::Sequential,
fallible: true,
})
.unwrap();
dag.freeze();
assert!(dag.is_fallible("f"));
}
// ── Ready set queries ─────────────────────────────────────────────
#[test]
fn ready_filtering() {
let mut dag = Dag::new();
dag.add_node(seq("s1")).unwrap();
dag.add_node(seq("s2")).unwrap();
dag.add_node(wild("w1")).unwrap();
dag.add_node(wild("w2")).unwrap();
dag.add_node(group("g1", "build")).unwrap();
dag.add_node(group("g2", "build")).unwrap();
dag.freeze();
assert_eq!(dag.ready_sequential(), vec!["s1", "s2"]);
assert_eq!(dag.ready_wildcards(), vec!["w1", "w2"]);
let groups = dag.ready_groups();
assert_eq!(groups.len(), 1);
assert_eq!(groups[0], ("build", 2, 2));
}
// ── Batch picking ─────────────────────────────────────────────────
#[test]
fn pick_batch_complete_group_with_wildcards() {
let mut dag = Dag::new();
dag.add_node(seq("s")).unwrap(); // not ready (has dep)
dag.add_node(group("g1", "build")).unwrap();
dag.add_node(group("g2", "build")).unwrap();
dag.add_node(wild("w")).unwrap();
dag.add_edge("s", "g1", EdgeKind::Explicit).unwrap(); // block g1
dag.freeze();
// g1 blocked by s, build group incomplete
// BUT wildcard w is ready → fires as its own batch
let batch = dag.pick_batch().unwrap();
assert_eq!(batch.len(), 1);
assert!(batch.contains(&"w".to_string()));
dag.pop("w").unwrap();
dag.pop("s").unwrap();
// Now g1, g2 both ready; build group complete
let batch = dag.pick_batch().unwrap();
assert!(batch.contains(&"g1".to_string()));
assert!(batch.contains(&"g2".to_string()));
}
#[test]
fn pick_batch_wildcards_only_no_groups() {
let mut dag = Dag::new();
dag.add_node(wild("w1")).unwrap();
dag.add_node(wild("w2")).unwrap();
dag.freeze();
let batch = dag.pick_batch().unwrap();
assert_eq!(batch.len(), 2);
assert!(batch.contains(&"w1".to_string()));
assert!(batch.contains(&"w2".to_string()));
}
#[test]
fn pick_batch_none_when_sequential_only() {
let mut dag = Dag::new();
dag.add_node(seq("s1")).unwrap();
dag.add_node(seq("s2")).unwrap();
dag.freeze();
// pick_batch only picks parallel batches; sequential nodes
// should be picked individually via pop_sequential()
assert!(dag.pick_batch().is_none());
}
// ── pop_sequential convenience ────────────────────────────────────
#[test]
fn pop_sequential_picks_first_seq() {
let mut dag = Dag::new();
dag.add_node(wild("w")).unwrap();
dag.add_node(seq("s1")).unwrap();
dag.add_node(seq("s2")).unwrap();
dag.freeze();
let popped = dag.pop_sequential().unwrap().unwrap();
assert_eq!(popped, "s1"); // BTreeSet order: s1 before s2
assert!(!dag.ready_names().contains(&"s1"));
assert!(dag.ready_names().contains(&"s2"));
}
// ── Deadlock detection ────────────────────────────────────────────
#[test]
fn stall_when_group_incomplete() {
let mut dag = Dag::new();
dag.add_node(seq("s")).unwrap();
dag.add_node(group("g1", "build")).unwrap();
dag.add_node(group("g2", "build")).unwrap();
dag.add_edge("s", "g2", EdgeKind::Explicit).unwrap();
dag.freeze();
// g1 ready, g2 not (depends on s). pick_batch → none.
assert!(dag.is_stalled());
assert_eq!(dag.stalled_groups(), vec!["build"]);
}
#[test]
fn remaining_after_partial_pop() {
let mut dag = Dag::new();
dag.add_node(seq("a")).unwrap();
dag.add_node(seq("b")).unwrap();
dag.add_node(seq("c")).unwrap();
dag.freeze();
dag.pop("a").unwrap();
let rem = dag.remaining();
assert_eq!(rem.len(), 2);
assert!(rem.contains(&"b"));
assert!(rem.contains(&"c"));
}
}
+20
View File
@@ -0,0 +1,20 @@
// Code in this module PARTIALLY or FULLY utilized AI Coding Agent
/// Edge kind — distinguishes explicit `@after` from implicit pipeline chaining.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum EdgeKind {
/// Explicit dependency: `@after(func)` in the script.
Explicit,
/// Implicit dependency: consecutive `@pipeline` functions without `@after`.
Implicit,
}
impl EdgeKind {
pub fn is_explicit(&self) -> bool {
matches!(self, EdgeKind::Explicit)
}
pub fn is_implicit(&self) -> bool {
matches!(self, EdgeKind::Implicit)
}
}
+9
View File
@@ -0,0 +1,9 @@
// Code in this module PARTIALLY or FULLY utilized AI Coding Agent
pub mod dag;
pub mod edge;
pub mod node;
pub use dag::{Dag, DagError};
pub use edge::EdgeKind;
pub use node::{DagNode, NodeColor};
+60
View File
@@ -0,0 +1,60 @@
// Code in this module PARTIALLY or FULLY utilized AI Coding Agent
/// Node coloring — mirrors the bake DAG generator's color semantics.
///
/// | Color | Probability | Bake semantics |
/// |-------------|------------|-----------------------------|
/// | Sequential | p_seq | Runs alone, one at a time |
/// | Wildcard | p_wild | Joins any complete group |
/// | Named(G) | p_named/K | Fires when group complete |
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub enum NodeColor {
/// Sequential node — executes alone.
Sequential,
/// Wildcard parallel — compatible with any complete named group.
Wildcard,
/// Named parallel group member — fires when all members of this group are ready.
Named(String),
}
impl NodeColor {
/// Returns true for `Sequential`.
pub fn is_sequential(&self) -> bool {
matches!(self, NodeColor::Sequential)
}
/// Returns true for `Wildcard`.
pub fn is_wildcard(&self) -> bool {
matches!(self, NodeColor::Wildcard)
}
/// Returns the group name if `Named`, or `None`.
pub fn group_name(&self) -> Option<&str> {
match self {
NodeColor::Named(g) => Some(g.as_str()),
_ => None,
}
}
/// Returns true if this is a parallel node (Wildcard or Named).
pub fn is_parallel(&self) -> bool {
!self.is_sequential()
}
}
/// Trait that nodes must implement to be managed by [`Dag`](super::dag::Dag).
///
/// Only `name()` and `color()` are required. Override `fallible()` if the
/// node can fail without halting the pipeline.
pub trait DagNode {
/// Unique name of this node within the DAG.
fn name(&self) -> &str;
/// Scheduling color — determines sequential/parallel/group semantics.
fn color(&self) -> NodeColor;
/// Whether failure of this node is non-fatal to the pipeline.
fn fallible(&self) -> bool {
false
}
}