Compare commits
5 Commits
df5026cdbe
...
fdc00f0635
| Author | SHA1 | Date | |
|---|---|---|---|
| fdc00f0635 | |||
| 26809df720 | |||
| 3629c33fc4 | |||
| 6d9ddbca56 | |||
| 9ff9073fce |
@@ -1,92 +1,127 @@
|
|||||||
# PROJECT KNOWLEDGE BASE
|
# PROJECT KNOWLEDGE BASE
|
||||||
|
|
||||||
**Generated:** 2026-05-19
|
**Generated:** 2026-06-12T16:08:15Z
|
||||||
**Commit:** 28abc5d
|
**Commit:** 3629c33
|
||||||
**Branch:** master
|
**Branch:** feat/bake-dag-tester
|
||||||
|
|
||||||
## OVERVIEW
|
## OVERVIEW
|
||||||
HoneyBiscuitWorkshop ("蜜饼工坊") — Rust CI/CD system with decorator-driven shell pipeline DSL and LLM-powered configuration. 3 independent Cargo crates, no workspace.
|
HoneyBiscuitWorkshop (蜜饼工坊) — Rust-based CI/CD system inspired by Concourse CI. Pipeline: prebake.yml → bake.sh → finalize.yml. Multi-builder (Docker, Firecracker, bare metal). Plugin system.
|
||||||
|
|
||||||
## STRUCTURE
|
## STRUCTURE
|
||||||
```
|
```
|
||||||
./
|
./
|
||||||
├── workshop-baker/ # Main orchestrator (CLI + pipeline stages)
|
├── workshop-baker/ # Main binary + lib — orchestrator, CLI, plugins
|
||||||
├── workshop-engine/ # Execution engine (extracted from baker)
|
├── workshop-engine/ # Core execution engine — process mgmt, package managers
|
||||||
├── workshop-getterurl/ # URL fetching (go-getter style)
|
├── workshop-schedule/ # DAG scheduling library (petgraph-based)
|
||||||
├── docs/ # mdBook documentation (zh-CN)
|
├── workshop-getterurl/ # URL fetching with hash verification
|
||||||
├── templates/ # Pipeline templates (bake.sh, finalize.yml, prebake.yml)
|
├── workshop-baker-params/ # Config/parameter type definitions
|
||||||
└── archived/ # Deprecated crates (7 moved here)
|
├── docs/ # mdBook documentation (Chinese, 3 volumes)
|
||||||
|
├── templates/ # Pipeline config templates (prebake/bake/finalize)
|
||||||
|
└── .github/workflows/ # CI (tests workshop-baker + workshop-engine)
|
||||||
```
|
```
|
||||||
|
**No root Cargo workspace** — crates linked by path deps. Build per-crate.
|
||||||
|
|
||||||
## WHERE TO LOOK
|
## WHERE TO LOOK
|
||||||
|
|
||||||
| Task | Location | Notes |
|
| Task | Location | Notes |
|
||||||
|------|----------|-------|
|
|------|----------|-------|
|
||||||
| Baker (CLI) | `workshop-baker/src/{lib,main}.rs` | Clap CLI + tokio entry |
|
| CLI definition | `workshop-baker/src/lib.rs` | Clap Cli/Commands/BareCommands |
|
||||||
| Pipeline stages | `workshop-baker/src/{bake,prebake,finalize}/` | Stage orchestration |
|
| Pipeline orchestration | `workshop-baker/src/main.rs` | prebake→bake→finalize→notify flow |
|
||||||
| Notifications | `workshop-baker/src/notify/` | Plugin-based event notifications |
|
| Decorator parsing | `workshop-baker/src/bake/decorator.rs` | `# @decorator(args)` syntax |
|
||||||
|| Engine (execution) | `workshop-engine/src/` | Process mgmt, PM detection |
|
| Build execution | `workshop-baker/src/bake/execute.rs` | Script execution with timeout/retry |
|
||||||
| URL fetching | `workshop-getterurl/src/` | Git/HTTP/File resource getter |
|
| Docker integration | `workshop-baker/src/prebake/env/container.rs` | Bollard-based |
|
||||||
|
| Package managers | `workshop-engine/src/pm/` | apt, pacman, dnf, apk |
|
||||||
|
| User package managers | `workshop-engine/src/upm/` | rustup, uv, go |
|
||||||
|
| Notification system | `workshop-baker/src/notify/` | Email, webhook, IM |
|
||||||
|
| Plugin system | `workshop-baker/src/finalize/plugin/` | shell, dylib, rhai, internal |
|
||||||
|
| Error types | `workshop-baker/src/error.rs` | CliError + HasExitCode |
|
||||||
|
| Exit codes | `workshop-engine/src/error.rs` | EXITCODE_* constants |
|
||||||
|
| DAG scheduling | `workshop-schedule/src/dag.rs` | Topological sort, batch picking |
|
||||||
|
| Pipeline templates | `templates/` | Reference config DSL |
|
||||||
|
|
||||||
|
## ENTRY POINTS
|
||||||
|
|
||||||
|
| Entry | Type | Path | Role |
|
||||||
|
|-------|------|------|------|
|
||||||
|
| `workshop-baker` | Rust bin | `workshop-baker/src/main.rs` | Primary CLI — dispatch: default (full pipeline), bare (single stage), daemon (unimplemented) |
|
||||||
|
| `workshop_baker` | Rust lib | `workshop-baker/src/lib.rs` | Baker library + CLI structs |
|
||||||
|
| `workshop_engine` | Rust lib | `workshop-engine/src/lib.rs` | Core engine — ExecutionContext, EventSender, Executor |
|
||||||
|
| `workshop_schedule` | Rust lib | `workshop-schedule/src/lib.rs` | Dag, DagNode, EdgeKind |
|
||||||
|
| `workshop_baker_params` | Rust lib | `workshop-baker-params/src/lib.rs` | Layered config: defaults → base → courier |
|
||||||
|
| `workshop_getterurl` | Rust lib | `workshop-getterurl/src/lib.rs` | GetterUrl, fetch() — http/git/file |
|
||||||
|
| docs | mdBook | `docs/book.toml` | Project documentation site |
|
||||||
|
|
||||||
## CODE MAP
|
## CODE MAP
|
||||||
| Symbol | Type | Location | Role |
|
|
||||||
|--------|------|----------|------|
|
### Crate Dependency Graph
|
||||||
| Cli | struct | workshop-baker/src/lib.rs:21 | CLI argument struct |
|
```
|
||||||
| ExecutionContext | struct | workshop-engine/src/types.rs:10 | Pipeline context |
|
workshop-baker (bin)
|
||||||
| Engine | struct | workshop-engine/src/types.rs:99 | Build executor (placeholder) |
|
├── workshop-engine
|
||||||
| PrebakeConfig | struct | workshop-baker/src/prebake/config.rs | Prebake YAML schema |
|
├── workshop-schedule
|
||||||
| FinalizeConfig | struct | workshop-baker/src/finalize/config.rs:21 | Finalize YAML schema |
|
├── workshop-baker-params
|
||||||
| GetterUrl | struct | workshop-getterurl/src/lib.rs:84 | Parsed resource URL |
|
└── workshop-getterurl
|
||||||
| Getter | trait | workshop-getterurl/src/getter.rs:18 | Resource fetch trait |
|
```
|
||||||
|
|
||||||
|
### workshop-baker Module Map
|
||||||
|
| Module | Submodules | Role |
|
||||||
|
|--------|-----------|------|
|
||||||
|
| `bake` | decorator, execute, parser, schedule, builder, trivial, util, constant, error | Script parsing + execution |
|
||||||
|
| `prebake` | config, stage (bootstrap, depssystem, depsuser, environment, hook), env (container, firecracker, baremetal, custom), security, event, types, constant, error | Environment setup |
|
||||||
|
| `finalize` | plugin (shell, dylib, rhai, internal), stage, template, types, config, constant, error, event | Post-build hooks |
|
||||||
|
| `notify` | handler, queue, rule, template, types, util, error | Async notification dispatch |
|
||||||
|
| `bare` | — | Standalone single-stage CLI mode |
|
||||||
|
| `types` | resource, buildstatus | Shared type definitions |
|
||||||
|
| `utils` | — | Cross-cutting utilities |
|
||||||
|
|
||||||
## CONVENTIONS
|
## CONVENTIONS
|
||||||
|
|
||||||
### Rust
|
- **Edition**: Rust 2024 (requires Rust 1.85+)
|
||||||
- **Edition**: 2024 (requires Rust 1.85+), deprecates `mod.rs`
|
- **Formatting**: Default `rustfmt` — no `rustfmt.toml` / `.editorconfig`
|
||||||
- **Naming**: `snake_case` files/modules, `PascalCase` types, `SCREAMING_SNAKE_CASE` consts
|
- **Lint overrides** (workshop-baker only): `dead_code = "allow"`, `unreachable_code = "allow"`, `inherent_to_string = "allow"`, `non_canonical_partial_ord_impl = "allow"`
|
||||||
- **Error handling**: `thiserror` + `anyhow` combo; `HasExitCode` trait for exit codes
|
- **Import order**: `std` → external crates → `crate::` (blank line between groups)
|
||||||
- **Async**: `#[tokio::main]` + `tokio` with "full" features
|
- **Error handling**: `thiserror` for libraries, `anyhow` for binaries. Implement `HasExitCode` for exit code mapping.
|
||||||
- **Imports**: `std` → `external_crate` → `crate::module`
|
- **Type system**: Newtype pattern preferred (e.g., `MemSize(u64)`). Shared types in `types/` module. `types/` must NOT depend on `bake/`, `engine/`, `prebake/`, `finalize/`.
|
||||||
- **Dual-mode**: CLI commands (prebake/bake/finalize) + daemon mode (unimplemented)
|
- **Tests**: Inline `#[cfg(test)] mod tests` at bottom of source files. Async: `#[tokio::test]`. CI runs `cargo test --all-features`.
|
||||||
|
- **Commits**: `type(scope): description` — `feat|fix|docs|test|refactor|style`
|
||||||
### Testing
|
- **AI annotation**: `// Code in this module PARTIALLY or FULLY utilized AI Coding Agent` — never delete if present
|
||||||
- **Rust**: `#[test]` inline, `#[tokio::test]` async, `tests/` integration
|
- **Docs language**: Chinese (zh-CN), mdBook
|
||||||
- **Benchmarks**: Criterion (configured in Cargo.toml)
|
- **No `no_std`**: Explicitly unsupported
|
||||||
|
|
||||||
### CI
|
|
||||||
- **GitHub Actions**: rust-toolchain (stable), clippy + rustfmt, `cargo test --all-features`
|
|
||||||
- **Matrix**: workshop-baker, workshop-engine
|
|
||||||
|
|
||||||
## ANTI-PATTERNS (THIS PROJECT)
|
## ANTI-PATTERNS (THIS PROJECT)
|
||||||
1. **Hardcoded user** — "vulcan" username hardcoded throughout prebake
|
|
||||||
2. **Unsafe libc** — workshop-baker/src/prebake/security.rs has 5+ unsafe blocks
|
|
||||||
3. **Rust 2024 edition** — May not work with stable toolchains
|
|
||||||
4. **Empty daemon** — `daemon/` dir is empty, Daemon command is `unimplemented!()`
|
|
||||||
|
|
||||||
## UNIQUE STYLES
|
- **Do NOT derive production behavior from bare mode** — bare mode is dev-only, ctx.privileged is fake
|
||||||
- **Pipeline DSL**: Shell scripts with `# @decorator` annotations
|
- **Do NOT use `detect()`/`adopt()` PM methods in production** — PM selection must come from env config
|
||||||
- **Notification system**: Plugin-based with batching, retry, priority groups
|
- **`drop_after < DepsUser` is FORBIDDEN** — will break system dependency installation (needs root)
|
||||||
- **LLM cookbook system**: YAML cookbooks for language-specific builds (archived)
|
- **Do NOT add magic URL inference** — URL is URL, behavior bound explicitly via `getter::` prefix
|
||||||
- **Chinese docs** — Documentation primarily in zh-CN
|
- **Do NOT use `@` for version locking in URLs** — it's an auth delimiter; use `?ref=`
|
||||||
|
- **Do NOT duplicate type definitions across modules** — single source of truth in `types/`
|
||||||
|
- **Careful with `unimplemented!()`** — Daemon mode, Rhai/Dylib plugins, and Custom builder are stubs
|
||||||
|
- **Security**: PIPELINE_CONTAINER_PRIVILEGED, PIPELINE_BAREMETAL_ELEVATE, PIPELINE_UNCOVER_SECRET require explicit approval + audit
|
||||||
|
|
||||||
## COMMANDS
|
## COMMANDS
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Build
|
# Build (per crate, no workspace)
|
||||||
cargo build --release -p workshop-baker
|
cd workshop-baker && cargo build --all-features
|
||||||
|
|
||||||
# Test
|
# Test (per crate)
|
||||||
cargo test -p workshop-baker
|
cd workshop-baker && cargo test --all-features
|
||||||
cargo test -p workshop-engine
|
cd workshop-engine && cargo test --all-features
|
||||||
|
|
||||||
# Lint
|
# Lint (CI gate)
|
||||||
cargo clippy -- -D warnings
|
cargo fmt -- --check
|
||||||
cargo fmt --all
|
cargo clippy --all-features -- -D warnings
|
||||||
|
|
||||||
# Run
|
# Docs
|
||||||
cargo run --bin workshop-baker -- --pipeline demo --build-id 1 --prebake prebake.yml --bake bake.sh --finalize finalize.yml
|
cd docs && mdbook build # output in docs/book/
|
||||||
|
cd docs && mdbook serve # local preview at localhost:3000
|
||||||
```
|
```
|
||||||
|
|
||||||
## NOTES
|
## NOTES
|
||||||
- **Docker required** for integration tests
|
|
||||||
- **No root workspace** — each crate independent, build separately
|
- **No workspace Cargo.toml** at root — each crate has its own `Cargo.lock`. Build individually.
|
||||||
- **7 archived crates** — workshop-agent, workshop-cert, workshop-deviceid, workshop-helper-mac, workshop-llm-detector, workshop-pipeline, workshop-vault
|
- **CI only tests** `workshop-baker` and `workshop-engine`; smaller crates tested transitively.
|
||||||
- **Python tests removed** — pytest.ini exists but test files no longer in workspace
|
- **`quicktest.sh`** in workshop-baker/ is a dev smoke-test script (Docker-based, not CI).
|
||||||
|
- **Git submodules**: `rust-tongsuo`, `RustyVault` (crypto — not yet integrated into build).
|
||||||
|
- **`.secret/`** contains real credentials — never commit, never read into context.
|
||||||
|
- See `docs/src/zh-CN/vol3_dev/` for full developer documentation (architecture, design decisions, code style).
|
||||||
|
|||||||
@@ -0,0 +1,48 @@
|
|||||||
|
# DOCS KNOWLEDGE BASE
|
||||||
|
|
||||||
|
## OVERVIEW
|
||||||
|
mdBook project in zh-CN. Authoritative source for project conventions, coding style, and architecture docs. Build output at `docs/book/` (gitignored).
|
||||||
|
|
||||||
|
## STRUCTURE
|
||||||
|
```
|
||||||
|
docs/
|
||||||
|
├── book.toml # mdBook config (title, language=zh-CN)
|
||||||
|
├── src/SUMMARY.md # Sidebar nav — source of truth for page listing
|
||||||
|
└── src/zh-CN/
|
||||||
|
├── index.md # Landing
|
||||||
|
├── vol1_user/ # End-user documentation
|
||||||
|
├── vol2_admin/ # Admin reference: decorator cheatsheet, deployment,
|
||||||
|
│ # env vars, exit codes, plugin ref, security, troubleshooting
|
||||||
|
└── vol3_dev/ # Developer docs: architecture, code style, commit
|
||||||
|
# convention, testing, error handling, type system,
|
||||||
|
# design decisions, PR process
|
||||||
|
```
|
||||||
|
|
||||||
|
## WHERE TO LOOK
|
||||||
|
|
||||||
|
| Task | Location | Notes |
|
||||||
|
|------|----------|-------|
|
||||||
|
| Sidebar/nav editing | `src/SUMMARY.md` | Add entry here to surface a new page |
|
||||||
|
| Coding conventions | `vol3_dev/code_style.md` | Naming, imports, formatting |
|
||||||
|
| Commit rules | `vol3_dev/commit_convention.md` | feat/fix/docs/test/refactor/style |
|
||||||
|
| Error handling | `vol3_dev/error_handling_patterns.md` | thiserror+anyhow+HasExitCode |
|
||||||
|
| Type system | `vol3_dev/type_system_guide.md` | Newtypes, types/ module isolation |
|
||||||
|
| Testing strategy | `vol3_dev/testing_strategy.md` | 3 tiers: unit, integration, system |
|
||||||
|
| Anti-patterns | `vol3_dev/design_decisions.md` | Documented design rejections |
|
||||||
|
| Architecture | `vol3_dev/architecture.md` | Crate dependency graph, module map |
|
||||||
|
|
||||||
|
## CONVENTIONS
|
||||||
|
|
||||||
|
- **Language**: All prose in zh-CN. Code blocks in Rust (default) or bash.
|
||||||
|
- **Adding a page**: Create `.md` under `volN_*/`, then add to `SUMMARY.md`.
|
||||||
|
- **Formatting**: Standard Markdown. mdBook preprocessors handle TOC and links.
|
||||||
|
- **Cross-references**: Relative paths from source root (e.g., `../vol3_dev/code_style.md`).
|
||||||
|
- **Code blocks**: Fenced with language tag. No line numbers in docs.
|
||||||
|
- **File naming**: `snake_case.md`.
|
||||||
|
|
||||||
|
## COMMANDS
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mdbook build # static site → docs/book/
|
||||||
|
mdbook serve # live preview at localhost:3000
|
||||||
|
```
|
||||||
@@ -6,19 +6,49 @@ use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
|
|||||||
#[serde(deny_unknown_fields)]
|
#[serde(deny_unknown_fields)]
|
||||||
pub struct BakeParams {
|
pub struct BakeParams {
|
||||||
#[serde(default)] pub workspace: ParamVal<String>,
|
#[serde(default)] pub workspace: ParamVal<String>,
|
||||||
|
/// Health probe interval in seconds (default 5).
|
||||||
|
#[serde(default)] pub health_period: ParamVal<u64>,
|
||||||
|
/// Health probe per-attempt timeout in seconds (default 10).
|
||||||
|
#[serde(default)] pub health_timeout: ParamVal<u64>,
|
||||||
|
/// Max health probe retries before giving up (default 12).
|
||||||
|
#[serde(default)] pub health_max_retries: ParamVal<u32>,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl MergeParams for BakeParams {
|
impl MergeParams for BakeParams {
|
||||||
fn defaults() -> Self { Self { workspace: ParamVal::new("/workspace".into()) } }
|
fn defaults() -> Self {
|
||||||
|
Self {
|
||||||
|
workspace: ParamVal::new("/workspace".into()),
|
||||||
|
health_period: ParamVal::new(5),
|
||||||
|
health_timeout: ParamVal::new(10),
|
||||||
|
health_max_retries: ParamVal::new(12),
|
||||||
|
}
|
||||||
|
}
|
||||||
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
|
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
|
||||||
apply_field!(self.workspace, i.workspace, w, ov, p, "workspace");
|
apply_field!(self.workspace, i.workspace, w, ov, p, "workspace");
|
||||||
|
apply_field!(self.health_period, i.health_period, w, ov, p, "health_period");
|
||||||
|
apply_field!(self.health_timeout, i.health_timeout, w, ov, p, "health_timeout");
|
||||||
|
apply_field!(self.health_max_retries, i.health_max_retries, w, ov, p, "health_max_retries");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Default for BakeParams { fn default() -> Self { Self::defaults() } }
|
impl Default for BakeParams { fn default() -> Self { Self::defaults() } }
|
||||||
|
|
||||||
#[derive(Debug, Clone)]
|
#[derive(Debug, Clone)]
|
||||||
pub struct BakeSettings { pub workspace: String }
|
pub struct BakeSettings {
|
||||||
|
pub workspace: String,
|
||||||
|
pub health_period: u64,
|
||||||
|
pub health_timeout: u64,
|
||||||
|
pub health_max_retries: u32,
|
||||||
|
}
|
||||||
impl Default for BakeSettings { fn default() -> Self { BakeParams::defaults().into() } }
|
impl Default for BakeSettings { fn default() -> Self { BakeParams::defaults().into() } }
|
||||||
impl From<&BakeParams> for BakeSettings { fn from(p: &BakeParams) -> Self { Self { workspace: p.workspace.value.clone() } } }
|
impl From<&BakeParams> for BakeSettings {
|
||||||
|
fn from(p: &BakeParams) -> Self {
|
||||||
|
Self {
|
||||||
|
workspace: p.workspace.value.clone(),
|
||||||
|
health_period: p.health_period.value,
|
||||||
|
health_timeout: p.health_timeout.value,
|
||||||
|
health_max_retries: p.health_max_retries.value,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
impl From<BakeParams> for BakeSettings { fn from(p: BakeParams) -> Self { (&p).into() } }
|
impl From<BakeParams> for BakeSettings { fn from(p: BakeParams) -> Self { (&p).into() } }
|
||||||
|
|||||||
@@ -23,7 +23,7 @@ impl MergeParams for NotificationParams {
|
|||||||
retry_backoff: ParamVal::new(2.0), retry_delay_seconds: ParamVal::new(5.0),
|
retry_backoff: ParamVal::new(2.0), retry_delay_seconds: ParamVal::new(5.0),
|
||||||
max_retry_delay_seconds: ParamVal::new(300.0),
|
max_retry_delay_seconds: ParamVal::new(300.0),
|
||||||
life_impact_factor: ParamVal::new(0.0),
|
life_impact_factor: ParamVal::new(0.0),
|
||||||
batch_period: ParamVal::new(30), batch_watermark: ParamVal::new(5),
|
batch_period: ParamVal::new(5), batch_watermark: ParamVal::new(1),
|
||||||
template_ref_depth: ParamVal::new(10),
|
template_ref_depth: ParamVal::new(10),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,80 @@
|
|||||||
|
# WORKSHOP-BAKER CRATE KNOWLEDGE BASE
|
||||||
|
|
||||||
|
## OVERVIEW
|
||||||
|
Primary crate — the only crate with a `[[bin]]` target. Contains the `workshop-baker` binary and `workshop_baker` library. Orchestrates the full CI/CD pipeline: prebake (env setup) → bake (script execution) → finalize (plugins, cleanup) → drain notification queue. ~69 .rs source files.
|
||||||
|
|
||||||
|
## STRUCTURE
|
||||||
|
```
|
||||||
|
src/
|
||||||
|
├── main.rs # Entry: default(full pipeline) | bare(dev-only) | daemon(unimplemented)
|
||||||
|
├── lib.rs # Clap Cli/Commands/BareCommands — CLI structs in lib, not bin
|
||||||
|
├── error.rs # CliError enum (wraps stage errors, delegates exit_code)
|
||||||
|
├── bare.rs # Dev-only single-stage execution, ctx.privileged always false
|
||||||
|
├── utils.rs # Cross-cutting helpers
|
||||||
|
├── bake/ # Script parsing + DAG scheduling + execution
|
||||||
|
│ ├── decorator.rs # @decorator(args) parsing, 81 tests — heaviest coverage in crate
|
||||||
|
│ ├── parser.rs # Step/function extraction
|
||||||
|
│ ├── schedule.rs # DAG build → topological sort
|
||||||
|
│ ├── execute.rs # Per-step timeout/retry
|
||||||
|
│ ├── builder.rs # Script assembly
|
||||||
|
│ ├── trivial.rs # Trivial step detection (skip in dry-run)
|
||||||
|
│ ├── constant.rs / error.rs / util.rs
|
||||||
|
├── prebake/ # Environment provisioning
|
||||||
|
│ ├── config.rs # PrebakeConfig deserialization
|
||||||
|
│ ├── stage.rs + stage/{bootstrap,depssystem,depsuser,environment,hook}.rs
|
||||||
|
│ ├── env.rs + env/{container,firecracker,baremetal,custom}.rs # Builder dispatch
|
||||||
|
│ ├── security.rs # Privilege/sandbox checks
|
||||||
|
│ ├── event.rs # Event sender/receiver channel
|
||||||
|
│ ├── types.rs + types/{memsize,cache,builderconfig,architecture}.rs
|
||||||
|
│ ├── constant.rs / error.rs
|
||||||
|
├── finalize/ # Post-build hooks + plugins
|
||||||
|
│ ├── config.rs # FinalizeConfig
|
||||||
|
│ ├── stage.rs + stage/hook.rs
|
||||||
|
│ ├── template.rs # Output templating (minijinja)
|
||||||
|
│ ├── plugin.rs + plugin/{shell,dylib,rhai,internal}.rs
|
||||||
|
│ │ └── plugin/internal/{dummy,mail,satori,webhook,insitenotify}.rs
|
||||||
|
│ ├── types.rs + types/compression.rs
|
||||||
|
│ ├── constant.rs / error.rs / event.rs
|
||||||
|
├── notify/ # Async notification dispatch
|
||||||
|
│ ├── queue.rs # NotificationQueue — drain on pipeline complete
|
||||||
|
│ ├── handler.rs / rule.rs / template.rs / types.rs / util.rs / error.rs
|
||||||
|
└── types/ # Shared types — ZERO deps on stage modules
|
||||||
|
├── resource.rs # ResourceRegistry (populated pre-pipeline, consumed read-only)
|
||||||
|
└── buildstatus.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
## WHERE TO LOOK
|
||||||
|
|
||||||
|
| Task | Location | Notes |
|
||||||
|
|------|----------|-------|
|
||||||
|
| CLI args + subcommands | `src/lib.rs` | Cli, Commands(Bare/Daemon), BareCommands |
|
||||||
|
| Pipeline orchestration | `src/main.rs` | worker_main(): prebake→bake→finalize→queue.drain() |
|
||||||
|
| Decorator syntax | `src/bake/decorator.rs` | `# @decorator` or `# @decorator(args)`, line-by-line regex |
|
||||||
|
| Build execution | `src/bake/execute.rs` | timeout, retry, exit code capture |
|
||||||
|
| DAG scheduling | `src/bake/schedule.rs` | Uses workshop-schedule crate |
|
||||||
|
| Docker builder | `src/prebake/env/container.rs` | Bollard-based, buildkit support |
|
||||||
|
| Firecracker / Bare metal | `.../firecracker.rs`, `.../baremetal.rs` | MicroVM;
|
||||||
|
needs PIPELINE_BAREMETAL_ELEVATE |
|
||||||
|
| Plugin dispatch | `src/finalize/plugin.rs` | Dispatches shell / dylib(stub) / rhai(stub) / internal |
|
||||||
|
| Notification queue | `src/notify/queue.rs` | Spawned early, drained after pipeline, retry support |
|
||||||
|
| Error type | `src/error.rs` | CliError wraps PrebakeError/BakeError/FinalizeError/anyhow |
|
||||||
|
| Dev bare mode | `src/bare.rs` | ctx.privileged=false always; do not extrapolate to production |
|
||||||
|
|
||||||
|
## DEPENDENCIES
|
||||||
|
**Internal:** workshop-engine, workshop-schedule, workshop-getterurl, workshop-baker-params. **External:** bollard (Docker), clap (CLI), axum 0.7 (HTTP), lettre 0.11 (email), minijinja (templates), cgroups-rs (limits), privdrop.
|
||||||
|
|
||||||
|
## CONVENTIONS (CRATE-SPECIFIC)
|
||||||
|
- **Lint overrides** (this crate only): `dead_code=allow`, `unreachable_code=allow`, `inherent_to_string=allow`, `non_canonical_partial_ord_impl=allow`
|
||||||
|
- **CLI structs in lib.rs, not main.rs** — enables integration tests to import and construct Cli directly.
|
||||||
|
- **Pipeline stages return `anyhow::Result<()>`** — stages propagate with `?`; only main.rs converts to CliError.
|
||||||
|
- **ResourceRegistry pattern** — populated before pipeline stages, consumed read-only by stages. New resource types go in `types/resource.rs`.
|
||||||
|
- **Notification queue fire-and-forget + drain** — notify tasks spawned early, send during pipeline, drain on completion. `NotificationQueue::drain()` must be called.
|
||||||
|
- **Trivial step detection** — steps with no external deps can be marked trivial (`trivial.rs`), skipped in dry-run.
|
||||||
|
## ANTI-PATTERNS (THIS CRATE)
|
||||||
|
- **Do NOT derive production behavior from bare mode** — ctx.privileged is hardcoded false.
|
||||||
|
- **Do NOT add CLI args bypassing prebake→bake→finalize ordering** — the pipeline is linear. Parallelism lives in the DAG scheduler within bake.
|
||||||
|
- **Do NOT import types/ from bake/, prebake/, or finalize/** — `types/` must have zero crate-internal deps.
|
||||||
|
- **Do NOT add lint allows without justification** — the four existing overrides are intentional.
|
||||||
|
- **Do NOT spawn blocking work on tokio runtime** — use `spawn_blocking` for CPU-heavy tasks.
|
||||||
|
- **Decorator regex is line-by-line** — multi-line decorators need a parser rewrite, not a regex hack.
|
||||||
|
- **Careful with `unimplemented!()`** — Daemon, Rhai/Dylib plugins, Custom builder are stubs. Prefer error over crash.
|
||||||
Generated
+589
-27
@@ -2,6 +2,12 @@
|
|||||||
# It is not intended for manual editing.
|
# It is not intended for manual editing.
|
||||||
version = 4
|
version = 4
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "adler2"
|
||||||
|
version = "2.0.1"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa"
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "aho-corasick"
|
name = "aho-corasick"
|
||||||
version = "1.1.4"
|
version = "1.1.4"
|
||||||
@@ -259,6 +265,28 @@ version = "1.5.0"
|
|||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8"
|
checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "aws-lc-rs"
|
||||||
|
version = "1.17.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "5ec2f1fc3ec205783a5da9a7e6c1509cc69dedf09a1949e412c1e18469326d00"
|
||||||
|
dependencies = [
|
||||||
|
"aws-lc-sys",
|
||||||
|
"zeroize",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "aws-lc-sys"
|
||||||
|
version = "0.41.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "1a2f9779ce85b93ab6170dd940ad0169b5766ff848247aff13bb788b832fe3f4"
|
||||||
|
dependencies = [
|
||||||
|
"cc",
|
||||||
|
"cmake",
|
||||||
|
"dunce",
|
||||||
|
"fs_extra",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "axum"
|
name = "axum"
|
||||||
version = "0.7.9"
|
version = "0.7.9"
|
||||||
@@ -393,6 +421,15 @@ dependencies = [
|
|||||||
"generic-array",
|
"generic-array",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "block-buffer"
|
||||||
|
version = "0.12.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "cdd35008169921d80bc60d3d0ab416eecb028c4cd653352907921d95084790be"
|
||||||
|
dependencies = [
|
||||||
|
"hybrid-array",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "block2"
|
name = "block2"
|
||||||
version = "0.6.2"
|
version = "0.6.2"
|
||||||
@@ -514,6 +551,17 @@ version = "1.11.1"
|
|||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "1e748733b7cbc798e1434b6ac524f0c1ff2ab456fe201501e6497c8417a4fc33"
|
checksum = "1e748733b7cbc798e1434b6ac524f0c1ff2ab456fe201501e6497c8417a4fc33"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "capctl"
|
||||||
|
version = "0.2.4"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "4a6e71767585f51c2a33fed6d67147ec0343725fc3c03bf4b89fe67fede56aa5"
|
||||||
|
dependencies = [
|
||||||
|
"bitflags 1.3.2",
|
||||||
|
"cfg-if",
|
||||||
|
"libc",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "cast"
|
name = "cast"
|
||||||
version = "0.3.0"
|
version = "0.3.0"
|
||||||
@@ -527,6 +575,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
|||||||
checksum = "7a0aeaff4ff1a90589618835a598e545176939b97874f7abc7851caa0618f203"
|
checksum = "7a0aeaff4ff1a90589618835a598e545176939b97874f7abc7851caa0618f203"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"find-msvc-tools",
|
"find-msvc-tools",
|
||||||
|
"jobserver",
|
||||||
|
"libc",
|
||||||
"shlex",
|
"shlex",
|
||||||
]
|
]
|
||||||
|
|
||||||
@@ -637,12 +687,31 @@ version = "0.7.6"
|
|||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "a1d728cc89cf3aee9ff92b05e62b19ee65a02b5702cff7d5a377e32c6ae29d8d"
|
checksum = "a1d728cc89cf3aee9ff92b05e62b19ee65a02b5702cff7d5a377e32c6ae29d8d"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "cmake"
|
||||||
|
version = "0.1.58"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "c0f78a02292a74a88ac736019ab962ece0bc380e3f977bf72e376c5d78ff0678"
|
||||||
|
dependencies = [
|
||||||
|
"cc",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "colorchoice"
|
name = "colorchoice"
|
||||||
version = "1.0.4"
|
version = "1.0.4"
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "b05b61dc5112cbb17e4b6cd61790d9845d13888356391624cbe7e41efeac1e75"
|
checksum = "b05b61dc5112cbb17e4b6cd61790d9845d13888356391624cbe7e41efeac1e75"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "combine"
|
||||||
|
version = "4.6.7"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "ba5a308b75df32fe02788e748662718f03fde005016435c444eea572398219fd"
|
||||||
|
dependencies = [
|
||||||
|
"bytes",
|
||||||
|
"memchr",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "concurrent-queue"
|
name = "concurrent-queue"
|
||||||
version = "2.5.0"
|
version = "2.5.0"
|
||||||
@@ -672,6 +741,12 @@ dependencies = [
|
|||||||
"yaml-rust2",
|
"yaml-rust2",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "const-oid"
|
||||||
|
version = "0.10.2"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c"
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "const-random"
|
name = "const-random"
|
||||||
version = "0.1.18"
|
version = "0.1.18"
|
||||||
@@ -726,6 +801,24 @@ dependencies = [
|
|||||||
"libc",
|
"libc",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "cpufeatures"
|
||||||
|
version = "0.3.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201"
|
||||||
|
dependencies = [
|
||||||
|
"libc",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "crc32fast"
|
||||||
|
version = "1.5.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "9481c1c90cbf2ac953f07c8d4a58aa3945c425b7185c9154d67a65e4230da511"
|
||||||
|
dependencies = [
|
||||||
|
"cfg-if",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "criterion"
|
name = "criterion"
|
||||||
version = "0.5.1"
|
version = "0.5.1"
|
||||||
@@ -803,6 +896,15 @@ dependencies = [
|
|||||||
"typenum",
|
"typenum",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "crypto-common"
|
||||||
|
version = "0.2.2"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "ce6e4c961d6cd6c9a86db418387425e8bdeaf05b3c8bc1411e6dca4c252f1453"
|
||||||
|
dependencies = [
|
||||||
|
"hybrid-array",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "deranged"
|
name = "deranged"
|
||||||
version = "0.5.5"
|
version = "0.5.5"
|
||||||
@@ -819,8 +921,19 @@ version = "0.10.7"
|
|||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292"
|
checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"block-buffer",
|
"block-buffer 0.10.4",
|
||||||
"crypto-common",
|
"crypto-common 0.1.7",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "digest"
|
||||||
|
version = "0.11.3"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2"
|
||||||
|
dependencies = [
|
||||||
|
"block-buffer 0.12.0",
|
||||||
|
"const-oid",
|
||||||
|
"crypto-common 0.2.2",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
@@ -853,6 +966,12 @@ dependencies = [
|
|||||||
"const-random",
|
"const-random",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "dunce"
|
||||||
|
version = "1.0.5"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "92773504d58c093f6de2459af4af33faa518c13451eb8f2b5698ed3d36e7c813"
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "duration-str"
|
name = "duration-str"
|
||||||
version = "0.21.0"
|
version = "0.21.0"
|
||||||
@@ -1025,6 +1144,22 @@ version = "0.1.6"
|
|||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "645cbb3a84e60b7531617d5ae4e57f7e27308f6445f5abf653209ea76dec8dff"
|
checksum = "645cbb3a84e60b7531617d5ae4e57f7e27308f6445f5abf653209ea76dec8dff"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "fixedbitset"
|
||||||
|
version = "0.5.7"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "1d674e81391d1e1ab681a28d99df07927c6d4aa5b027d7da16ba32d1d21ecd99"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "flate2"
|
||||||
|
version = "1.1.9"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "843fba2746e448b37e26a819579957415c8cef339bf08564fe8b7ddbd959573c"
|
||||||
|
dependencies = [
|
||||||
|
"crc32fast",
|
||||||
|
"miniz_oxide",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "fnv"
|
name = "fnv"
|
||||||
version = "1.0.7"
|
version = "1.0.7"
|
||||||
@@ -1046,6 +1181,12 @@ dependencies = [
|
|||||||
"percent-encoding",
|
"percent-encoding",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "fs_extra"
|
||||||
|
version = "1.3.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c"
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "futures-channel"
|
name = "futures-channel"
|
||||||
version = "0.3.31"
|
version = "0.3.31"
|
||||||
@@ -1136,8 +1277,10 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
|||||||
checksum = "335ff9f135e4384c8150d6f27c6daed433577f86b4750418338c01a1a2528592"
|
checksum = "335ff9f135e4384c8150d6f27c6daed433577f86b4750418338c01a1a2528592"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"cfg-if",
|
"cfg-if",
|
||||||
|
"js-sys",
|
||||||
"libc",
|
"libc",
|
||||||
"wasi",
|
"wasi",
|
||||||
|
"wasm-bindgen",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
@@ -1147,9 +1290,11 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
|||||||
checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd"
|
checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"cfg-if",
|
"cfg-if",
|
||||||
|
"js-sys",
|
||||||
"libc",
|
"libc",
|
||||||
"r-efi",
|
"r-efi",
|
||||||
"wasip2",
|
"wasip2",
|
||||||
|
"wasm-bindgen",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
@@ -1309,6 +1454,15 @@ version = "1.0.3"
|
|||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "df3b46402a9d5adb4c86a0cf463f42e19994e3ee891101b1841f30a545cb49a9"
|
checksum = "df3b46402a9d5adb4c86a0cf463f42e19994e3ee891101b1841f30a545cb49a9"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "hybrid-array"
|
||||||
|
version = "0.4.12"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "9155a582abd142abc056962c29e3ce5ff2ad5469f4246b537ed42c5deba857da"
|
||||||
|
dependencies = [
|
||||||
|
"typenum",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "hyper"
|
name = "hyper"
|
||||||
version = "1.8.1"
|
version = "1.8.1"
|
||||||
@@ -1382,6 +1536,7 @@ version = "0.1.19"
|
|||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "727805d60e7938b76b826a6ef209eb70eaa1812794f9424d4a4e2d740662df5f"
|
checksum = "727805d60e7938b76b826a6ef209eb70eaa1812794f9424d4a4e2d740662df5f"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
|
"base64",
|
||||||
"bytes",
|
"bytes",
|
||||||
"futures-channel",
|
"futures-channel",
|
||||||
"futures-core",
|
"futures-core",
|
||||||
@@ -1389,7 +1544,9 @@ dependencies = [
|
|||||||
"http",
|
"http",
|
||||||
"http-body",
|
"http-body",
|
||||||
"hyper",
|
"hyper",
|
||||||
|
"ipnet",
|
||||||
"libc",
|
"libc",
|
||||||
|
"percent-encoding",
|
||||||
"pin-project-lite",
|
"pin-project-lite",
|
||||||
"socket2",
|
"socket2",
|
||||||
"tokio",
|
"tokio",
|
||||||
@@ -1561,6 +1718,12 @@ dependencies = [
|
|||||||
"serde_core",
|
"serde_core",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "ipnet"
|
||||||
|
version = "2.12.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "d98f6fed1fde3f8c21bc40a1abb88dd75e67924f9cffc3ef95607bad8017f8e2"
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "is-terminal"
|
name = "is-terminal"
|
||||||
version = "0.4.17"
|
version = "0.4.17"
|
||||||
@@ -1626,6 +1789,65 @@ dependencies = [
|
|||||||
"syn",
|
"syn",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "jni"
|
||||||
|
version = "0.22.4"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "5efd9a482cf3a427f00d6b35f14332adc7902ce91efb778580e180ff90fa3498"
|
||||||
|
dependencies = [
|
||||||
|
"cfg-if",
|
||||||
|
"combine",
|
||||||
|
"jni-macros",
|
||||||
|
"jni-sys",
|
||||||
|
"log",
|
||||||
|
"simd_cesu8",
|
||||||
|
"thiserror 2.0.18",
|
||||||
|
"walkdir",
|
||||||
|
"windows-link",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "jni-macros"
|
||||||
|
version = "0.22.4"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "a00109accc170f0bdb141fed3e393c565b6f5e072365c3bd58f5b062591560a3"
|
||||||
|
dependencies = [
|
||||||
|
"proc-macro2",
|
||||||
|
"quote",
|
||||||
|
"rustc_version",
|
||||||
|
"simd_cesu8",
|
||||||
|
"syn",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "jni-sys"
|
||||||
|
version = "0.4.1"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "c6377a88cb3910bee9b0fa88d4f42e1d2da8e79915598f65fb0c7ee14c878af2"
|
||||||
|
dependencies = [
|
||||||
|
"jni-sys-macros",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "jni-sys-macros"
|
||||||
|
version = "0.4.1"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "38c0b942f458fe50cdac086d2f946512305e5631e720728f2a61aabcd47a6264"
|
||||||
|
dependencies = [
|
||||||
|
"quote",
|
||||||
|
"syn",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "jobserver"
|
||||||
|
version = "0.1.34"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "9afb3de4395d6b3e67a780b6de64b51c978ecf11cb9a462c66be7d4ca9039d33"
|
||||||
|
dependencies = [
|
||||||
|
"getrandom 0.3.4",
|
||||||
|
"libc",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "js-sys"
|
name = "js-sys"
|
||||||
version = "0.3.83"
|
version = "0.3.83"
|
||||||
@@ -1721,9 +1943,15 @@ dependencies = [
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "log"
|
name = "log"
|
||||||
version = "0.4.28"
|
version = "0.4.32"
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "34080505efa8e45a4b816c349525ebe327ceaa8559756f0356cba97ef3bf7432"
|
checksum = "953f07c43838f8e6f9758cab68bf5bed85465e7587ebe0b823f1bcd81978ad3a"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "lru-slab"
|
||||||
|
version = "0.1.2"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154"
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "matchit"
|
name = "matchit"
|
||||||
@@ -1737,6 +1965,16 @@ version = "0.8.4"
|
|||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "47e1ffaa40ddd1f3ed91f717a33c8c0ee23fff369e3aa8772b9605cc1d22f4c3"
|
checksum = "47e1ffaa40ddd1f3ed91f717a33c8c0ee23fff369e3aa8772b9605cc1d22f4c3"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "md-5"
|
||||||
|
version = "0.11.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "69b6441f590336821bb897fb28fc622898ccceb1d6cea3fde5ea86b090c4de98"
|
||||||
|
dependencies = [
|
||||||
|
"cfg-if",
|
||||||
|
"digest 0.11.3",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "memchr"
|
name = "memchr"
|
||||||
version = "2.7.6"
|
version = "2.7.6"
|
||||||
@@ -1768,10 +2006,20 @@ dependencies = [
|
|||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "mio"
|
name = "miniz_oxide"
|
||||||
version = "1.1.0"
|
version = "0.8.9"
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "69d83b0086dc8ecf3ce9ae2874b2d1290252e2a30720bea58a5c6639b0092873"
|
checksum = "1fa76a2c86f704bdb222d66965fb3d63269ce38518b83cb0575fca855ebb6316"
|
||||||
|
dependencies = [
|
||||||
|
"adler2",
|
||||||
|
"simd-adler32",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "mio"
|
||||||
|
version = "1.2.1"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "02bd0af71c67b473010cbbc60715ee815645a4dc942899111f494b4b737d6fda"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"libc",
|
"libc",
|
||||||
"wasi",
|
"wasi",
|
||||||
@@ -2205,7 +2453,17 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
|||||||
checksum = "bf1d70880e76bdc13ba52eafa6239ce793d85c8e43896507e43dd8984ff05b82"
|
checksum = "bf1d70880e76bdc13ba52eafa6239ce793d85c8e43896507e43dd8984ff05b82"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"pest",
|
"pest",
|
||||||
"sha2",
|
"sha2 0.10.9",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "petgraph"
|
||||||
|
version = "0.7.1"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "3672b37090dbd86368a4145bc067582552b29c27377cad4e0a306c97f9bd7772"
|
||||||
|
dependencies = [
|
||||||
|
"fixedbitset",
|
||||||
|
"indexmap 2.12.1",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
@@ -2251,6 +2509,12 @@ dependencies = [
|
|||||||
"futures-io",
|
"futures-io",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "pkg-config"
|
||||||
|
version = "0.3.33"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e"
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "plain"
|
name = "plain"
|
||||||
version = "0.2.3"
|
version = "0.2.3"
|
||||||
@@ -2398,6 +2662,62 @@ dependencies = [
|
|||||||
"prost",
|
"prost",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "quinn"
|
||||||
|
version = "0.11.9"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "b9e20a958963c291dc322d98411f541009df2ced7b5a4f2bd52337638cfccf20"
|
||||||
|
dependencies = [
|
||||||
|
"bytes",
|
||||||
|
"cfg_aliases",
|
||||||
|
"pin-project-lite",
|
||||||
|
"quinn-proto",
|
||||||
|
"quinn-udp",
|
||||||
|
"rustc-hash",
|
||||||
|
"rustls",
|
||||||
|
"socket2",
|
||||||
|
"thiserror 2.0.18",
|
||||||
|
"tokio",
|
||||||
|
"tracing",
|
||||||
|
"web-time",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "quinn-proto"
|
||||||
|
version = "0.11.14"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "434b42fec591c96ef50e21e886936e66d3cc3f737104fdb9b737c40ffb94c098"
|
||||||
|
dependencies = [
|
||||||
|
"aws-lc-rs",
|
||||||
|
"bytes",
|
||||||
|
"getrandom 0.3.4",
|
||||||
|
"lru-slab",
|
||||||
|
"rand",
|
||||||
|
"ring",
|
||||||
|
"rustc-hash",
|
||||||
|
"rustls",
|
||||||
|
"rustls-pki-types",
|
||||||
|
"slab",
|
||||||
|
"thiserror 2.0.18",
|
||||||
|
"tinyvec",
|
||||||
|
"tracing",
|
||||||
|
"web-time",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "quinn-udp"
|
||||||
|
version = "0.5.14"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "addec6a0dcad8a8d96a771f815f0eaf55f9d1805756410b39f5fa81332574cbd"
|
||||||
|
dependencies = [
|
||||||
|
"cfg_aliases",
|
||||||
|
"libc",
|
||||||
|
"once_cell",
|
||||||
|
"socket2",
|
||||||
|
"tracing",
|
||||||
|
"windows-sys 0.60.2",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "quote"
|
name = "quote"
|
||||||
version = "1.0.42"
|
version = "1.0.42"
|
||||||
@@ -2535,6 +2855,41 @@ version = "0.8.8"
|
|||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "7a2d987857b319362043e95f5353c0535c1f58eec5336fdfcf626430af7def58"
|
checksum = "7a2d987857b319362043e95f5353c0535c1f58eec5336fdfcf626430af7def58"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "reqwest"
|
||||||
|
version = "0.13.4"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "219c5811de6525e5416c7d5d53bb656d3afdbc6c5af816e0802bcfa42dbdc1c3"
|
||||||
|
dependencies = [
|
||||||
|
"base64",
|
||||||
|
"bytes",
|
||||||
|
"futures-core",
|
||||||
|
"http",
|
||||||
|
"http-body",
|
||||||
|
"http-body-util",
|
||||||
|
"hyper",
|
||||||
|
"hyper-rustls",
|
||||||
|
"hyper-util",
|
||||||
|
"js-sys",
|
||||||
|
"log",
|
||||||
|
"percent-encoding",
|
||||||
|
"pin-project-lite",
|
||||||
|
"quinn",
|
||||||
|
"rustls",
|
||||||
|
"rustls-pki-types",
|
||||||
|
"rustls-platform-verifier",
|
||||||
|
"sync_wrapper",
|
||||||
|
"tokio",
|
||||||
|
"tokio-rustls",
|
||||||
|
"tower",
|
||||||
|
"tower-http",
|
||||||
|
"tower-service",
|
||||||
|
"url",
|
||||||
|
"wasm-bindgen",
|
||||||
|
"wasm-bindgen-futures",
|
||||||
|
"web-sys",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "ring"
|
name = "ring"
|
||||||
version = "0.17.14"
|
version = "0.17.14"
|
||||||
@@ -2583,6 +2938,21 @@ dependencies = [
|
|||||||
"num-traits",
|
"num-traits",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "rustc-hash"
|
||||||
|
version = "2.1.2"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "94300abf3f1ae2e2b8ffb7b58043de3d399c73fa6f4b73826402a5c457614dbe"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "rustc_version"
|
||||||
|
version = "0.4.1"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92"
|
||||||
|
dependencies = [
|
||||||
|
"semver",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "rustix"
|
name = "rustix"
|
||||||
version = "1.1.4"
|
version = "1.1.4"
|
||||||
@@ -2602,6 +2972,7 @@ version = "0.23.35"
|
|||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "533f54bc6a7d4f647e46ad909549eda97bf5afc1585190ef692b4286b198bd8f"
|
checksum = "533f54bc6a7d4f647e46ad909549eda97bf5afc1585190ef692b4286b198bd8f"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
|
"aws-lc-rs",
|
||||||
"log",
|
"log",
|
||||||
"once_cell",
|
"once_cell",
|
||||||
"ring",
|
"ring",
|
||||||
@@ -2629,15 +3000,44 @@ version = "1.13.2"
|
|||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "21e6f2ab2928ca4291b86736a8bd920a277a399bba1589409d72154ff87c1282"
|
checksum = "21e6f2ab2928ca4291b86736a8bd920a277a399bba1589409d72154ff87c1282"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
|
"web-time",
|
||||||
"zeroize",
|
"zeroize",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "rustls-platform-verifier"
|
||||||
|
version = "0.7.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "26d1e2536ce4f35f4846aa13bff16bd0ff40157cdb14cc056c7b14ba41233ba0"
|
||||||
|
dependencies = [
|
||||||
|
"core-foundation",
|
||||||
|
"core-foundation-sys",
|
||||||
|
"jni",
|
||||||
|
"log",
|
||||||
|
"once_cell",
|
||||||
|
"rustls",
|
||||||
|
"rustls-native-certs",
|
||||||
|
"rustls-platform-verifier-android",
|
||||||
|
"rustls-webpki",
|
||||||
|
"security-framework",
|
||||||
|
"security-framework-sys",
|
||||||
|
"webpki-root-certs",
|
||||||
|
"windows-sys 0.61.2",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "rustls-platform-verifier-android"
|
||||||
|
version = "0.1.1"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "f87165f0995f63a9fbeea62b64d10b4d9d8e78ec6d7d51fb2125fda7bb36788f"
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "rustls-webpki"
|
name = "rustls-webpki"
|
||||||
version = "0.103.8"
|
version = "0.103.8"
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "2ffdfa2f5286e2247234e03f680868ac2815974dc39e00ea15adc445d0aafe52"
|
checksum = "2ffdfa2f5286e2247234e03f680868ac2815974dc39e00ea15adc445d0aafe52"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
|
"aws-lc-rs",
|
||||||
"ring",
|
"ring",
|
||||||
"rustls-pki-types",
|
"rustls-pki-types",
|
||||||
"untrusted",
|
"untrusted",
|
||||||
@@ -2861,6 +3261,17 @@ dependencies = [
|
|||||||
"unsafe-libyaml",
|
"unsafe-libyaml",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "sha1"
|
||||||
|
version = "0.11.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "aacc4cc499359472b4abe1bf11d0b12e688af9a805fa5e3016f9a386dc2d0214"
|
||||||
|
dependencies = [
|
||||||
|
"cfg-if",
|
||||||
|
"cpufeatures 0.3.0",
|
||||||
|
"digest 0.11.3",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "sha2"
|
name = "sha2"
|
||||||
version = "0.10.9"
|
version = "0.10.9"
|
||||||
@@ -2868,8 +3279,19 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
|||||||
checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283"
|
checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"cfg-if",
|
"cfg-if",
|
||||||
"cpufeatures",
|
"cpufeatures 0.2.17",
|
||||||
"digest",
|
"digest 0.10.7",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "sha2"
|
||||||
|
version = "0.11.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "446ba717509524cb3f22f17ecc096f10f4822d76ab5c0b9822c5f9c284e825f4"
|
||||||
|
dependencies = [
|
||||||
|
"cfg-if",
|
||||||
|
"cpufeatures 0.3.0",
|
||||||
|
"digest 0.11.3",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
@@ -2894,6 +3316,28 @@ dependencies = [
|
|||||||
"libc",
|
"libc",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "simd-adler32"
|
||||||
|
version = "0.3.9"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "703d5c7ef118737c72f1af64ad2f6f8c5e1921f818cdcb97b8fe6fc69bf66214"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "simd_cesu8"
|
||||||
|
version = "1.1.1"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "94f90157bb87cddf702797c5dadfa0be7d266cdf49e22da2fcaa32eff75b2c33"
|
||||||
|
dependencies = [
|
||||||
|
"rustc_version",
|
||||||
|
"simdutf8",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "simdutf8"
|
||||||
|
version = "0.1.5"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e"
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "slab"
|
name = "slab"
|
||||||
version = "0.4.11"
|
version = "0.4.11"
|
||||||
@@ -2908,12 +3352,12 @@ checksum = "67b1b7a3b5fe4f1376887184045fcf45c69e92af734b7aaddc05fb777b6fbd03"
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "socket2"
|
name = "socket2"
|
||||||
version = "0.6.1"
|
version = "0.6.4"
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "17129e116933cf371d018bb80ae557e889637989d8638274fb25622827b03881"
|
checksum = "52d1cfed4120b4d927bf7c0f86d2087a4a7d6027c906d9f9d525a80573b9be51"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"libc",
|
"libc",
|
||||||
"windows-sys 0.60.2",
|
"windows-sys 0.61.2",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
@@ -2956,6 +3400,9 @@ name = "sync_wrapper"
|
|||||||
version = "1.0.2"
|
version = "1.0.2"
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "0bf256ce5efdfa370213c1dabab5935a12e49f2c58d15e9eac2870d3b4f27263"
|
checksum = "0bf256ce5efdfa370213c1dabab5935a12e49f2c58d15e9eac2870d3b4f27263"
|
||||||
|
dependencies = [
|
||||||
|
"futures-core",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "synstructure"
|
name = "synstructure"
|
||||||
@@ -3093,10 +3540,25 @@ dependencies = [
|
|||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "tokio"
|
name = "tinyvec"
|
||||||
version = "1.48.0"
|
version = "1.11.0"
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "ff360e02eab121e0bc37a2d3b4d4dc622e6eda3a8e5253d5435ecf5bd4c68408"
|
checksum = "3e61e67053d25a4e82c844e8424039d9745781b3fc4f32b8d55ed50f5f667ef3"
|
||||||
|
dependencies = [
|
||||||
|
"tinyvec_macros",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "tinyvec_macros"
|
||||||
|
version = "0.1.1"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "tokio"
|
||||||
|
version = "1.52.3"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "8fc7f01b389ac15039e4dc9531aa973a135d7a4135281b12d7c1bc79fd57fffe"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"bytes",
|
"bytes",
|
||||||
"libc",
|
"libc",
|
||||||
@@ -3111,9 +3573,9 @@ dependencies = [
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "tokio-macros"
|
name = "tokio-macros"
|
||||||
version = "2.6.0"
|
version = "2.7.0"
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "af407857209536a95c8e56f8231ef2c2e2aff839b22e07a1ffcbc617e9db9fa5"
|
checksum = "385a6cb71ab9ab790c5fe8d67f1645e6c450a7ce006a33de03daa956cf70a496"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"proc-macro2",
|
"proc-macro2",
|
||||||
"quote",
|
"quote",
|
||||||
@@ -3237,12 +3699,6 @@ dependencies = [
|
|||||||
"tonic",
|
"tonic",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "topological-sort"
|
|
||||||
version = "0.2.2"
|
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
||||||
checksum = "ea68304e134ecd095ac6c3574494fc62b909f416c4fca77e440530221e549d3d"
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "tower"
|
name = "tower"
|
||||||
version = "0.5.2"
|
version = "0.5.2"
|
||||||
@@ -3262,6 +3718,24 @@ dependencies = [
|
|||||||
"tracing",
|
"tracing",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "tower-http"
|
||||||
|
version = "0.6.11"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "4cfcf7e2740e6fc6d4d688b4ef00650406bb94adf4731e43c096c3a19fe40840"
|
||||||
|
dependencies = [
|
||||||
|
"bitflags 2.10.0",
|
||||||
|
"bytes",
|
||||||
|
"futures-util",
|
||||||
|
"http",
|
||||||
|
"http-body",
|
||||||
|
"pin-project-lite",
|
||||||
|
"tower",
|
||||||
|
"tower-layer",
|
||||||
|
"tower-service",
|
||||||
|
"url",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "tower-layer"
|
name = "tower-layer"
|
||||||
version = "0.3.3"
|
version = "0.3.3"
|
||||||
@@ -3320,9 +3794,9 @@ checksum = "bc7d623258602320d5c55d1bc22793b57daff0ec7efc270ea7d55ce1d5f5471c"
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "typenum"
|
name = "typenum"
|
||||||
version = "1.19.0"
|
version = "1.20.1"
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "562d481066bde0658276a35467c4af00bdc6ee726305698a55b86e61d7ad82bb"
|
checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20"
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "ucd-trie"
|
name = "ucd-trie"
|
||||||
@@ -3488,6 +3962,19 @@ dependencies = [
|
|||||||
"wasm-bindgen-shared",
|
"wasm-bindgen-shared",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "wasm-bindgen-futures"
|
||||||
|
version = "0.4.56"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "836d9622d604feee9e5de25ac10e3ea5f2d65b41eac0d9ce72eb5deae707ce7c"
|
||||||
|
dependencies = [
|
||||||
|
"cfg-if",
|
||||||
|
"js-sys",
|
||||||
|
"once_cell",
|
||||||
|
"wasm-bindgen",
|
||||||
|
"web-sys",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "wasm-bindgen-macro"
|
name = "wasm-bindgen-macro"
|
||||||
version = "0.2.106"
|
version = "0.2.106"
|
||||||
@@ -3530,6 +4017,25 @@ dependencies = [
|
|||||||
"wasm-bindgen",
|
"wasm-bindgen",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "web-time"
|
||||||
|
version = "1.1.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "5a6580f308b1fad9207618087a65c04e7a10bc77e02c8e84e9b00dd4b12fa0bb"
|
||||||
|
dependencies = [
|
||||||
|
"js-sys",
|
||||||
|
"wasm-bindgen",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "webpki-root-certs"
|
||||||
|
version = "1.0.7"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "f31141ce3fc3e300ae89b78c0dd67f9708061d1d2eda54b8209346fd6be9a92c"
|
||||||
|
dependencies = [
|
||||||
|
"rustls-pki-types",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "webpki-roots"
|
name = "webpki-roots"
|
||||||
version = "1.0.4"
|
version = "1.0.4"
|
||||||
@@ -3816,7 +4322,9 @@ dependencies = [
|
|||||||
"anyhow",
|
"anyhow",
|
||||||
"async-trait",
|
"async-trait",
|
||||||
"axum 0.7.9",
|
"axum 0.7.9",
|
||||||
|
"bitflags 2.10.0",
|
||||||
"bollard",
|
"bollard",
|
||||||
|
"capctl",
|
||||||
"cgroups-rs",
|
"cgroups-rs",
|
||||||
"chrono",
|
"chrono",
|
||||||
"clap",
|
"clap",
|
||||||
@@ -3824,6 +4332,7 @@ dependencies = [
|
|||||||
"criterion",
|
"criterion",
|
||||||
"duration-str",
|
"duration-str",
|
||||||
"env_logger",
|
"env_logger",
|
||||||
|
"flate2",
|
||||||
"futures-util",
|
"futures-util",
|
||||||
"glob",
|
"glob",
|
||||||
"globset",
|
"globset",
|
||||||
@@ -3844,11 +4353,14 @@ dependencies = [
|
|||||||
"tempfile",
|
"tempfile",
|
||||||
"thiserror 2.0.18",
|
"thiserror 2.0.18",
|
||||||
"tokio",
|
"tokio",
|
||||||
"topological-sort",
|
"url",
|
||||||
"uuid",
|
"uuid",
|
||||||
"which",
|
"which",
|
||||||
"workshop-baker-params",
|
"workshop-baker-params",
|
||||||
"workshop-engine",
|
"workshop-engine",
|
||||||
|
"workshop-getterurl",
|
||||||
|
"workshop-schedule",
|
||||||
|
"zstd",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
@@ -3877,6 +4389,28 @@ dependencies = [
|
|||||||
"tokio",
|
"tokio",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "workshop-getterurl"
|
||||||
|
version = "0.1.0"
|
||||||
|
dependencies = [
|
||||||
|
"log",
|
||||||
|
"md-5",
|
||||||
|
"reqwest",
|
||||||
|
"sha1",
|
||||||
|
"sha2 0.11.0",
|
||||||
|
"thiserror 2.0.18",
|
||||||
|
"tokio",
|
||||||
|
"url",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "workshop-schedule"
|
||||||
|
version = "0.1.0"
|
||||||
|
dependencies = [
|
||||||
|
"petgraph",
|
||||||
|
"thiserror 2.0.18",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "writeable"
|
name = "writeable"
|
||||||
version = "0.6.2"
|
version = "0.6.2"
|
||||||
@@ -4074,6 +4608,34 @@ version = "1.0.3"
|
|||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "e9747e91771f56fd7893e1164abd78febd14a670ceec257caad15e051de35f06"
|
checksum = "e9747e91771f56fd7893e1164abd78febd14a670ceec257caad15e051de35f06"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "zstd"
|
||||||
|
version = "0.13.3"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "e91ee311a569c327171651566e07972200e76fcfe2242a4fa446149a3881c08a"
|
||||||
|
dependencies = [
|
||||||
|
"zstd-safe",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "zstd-safe"
|
||||||
|
version = "7.2.4"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "8f49c4d5f0abb602a93fb8736af2a4f4dd9512e36f7f570d66e65ff867ed3b9d"
|
||||||
|
dependencies = [
|
||||||
|
"zstd-sys",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "zstd-sys"
|
||||||
|
version = "2.0.16+zstd.1.5.7"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "91e19ebc2adc8f83e43039e79776e3fda8ca919132d68a1fed6a5faca2683748"
|
||||||
|
dependencies = [
|
||||||
|
"cc",
|
||||||
|
"pkg-config",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "zvariant"
|
name = "zvariant"
|
||||||
version = "5.8.0"
|
version = "5.8.0"
|
||||||
|
|||||||
@@ -41,7 +41,7 @@ tar = "0.4.44"
|
|||||||
tempfile = "3.24.0"
|
tempfile = "3.24.0"
|
||||||
thiserror = "2.0.17"
|
thiserror = "2.0.17"
|
||||||
tokio = { version = "1.48.0", features = ["full"] }
|
tokio = { version = "1.48.0", features = ["full"] }
|
||||||
topological-sort = "0.2.2"
|
workshop-schedule = { path = "../workshop-schedule" }
|
||||||
async-trait = "0.1.87"
|
async-trait = "0.1.87"
|
||||||
uuid = { version = "1.19.0", features = ["v4"] }
|
uuid = { version = "1.19.0", features = ["v4"] }
|
||||||
libc = "0.2"
|
libc = "0.2"
|
||||||
@@ -50,8 +50,14 @@ privdrop = "0.5.6"
|
|||||||
which = "8.0.2"
|
which = "8.0.2"
|
||||||
globset = "0.4.18"
|
globset = "0.4.18"
|
||||||
axum = "0.7"
|
axum = "0.7"
|
||||||
|
workshop-getterurl = { path = "../workshop-getterurl" }
|
||||||
workshop-baker-params = { version = "0.1.0", path = "../workshop-baker-params" }
|
workshop-baker-params = { version = "0.1.0", path = "../workshop-baker-params" }
|
||||||
|
|
||||||
|
bitflags = "2"
|
||||||
|
url = "2.5.8"
|
||||||
|
zstd = "0.13.3"
|
||||||
|
flate2 = "1.1.9"
|
||||||
|
capctl = "0.2.4"
|
||||||
[dev-dependencies]
|
[dev-dependencies]
|
||||||
criterion = "0.5"
|
criterion = "0.5"
|
||||||
|
|
||||||
|
|||||||
@@ -1,11 +0,0 @@
|
|||||||
[pytest]
|
|
||||||
testpaths = tests/integration
|
|
||||||
python_files = test_*.py
|
|
||||||
python_classes = Test*
|
|
||||||
python_functions = test_*
|
|
||||||
addopts = -v --tb=short
|
|
||||||
markers =
|
|
||||||
integration: marks tests as integration tests (deselect with '-m "not integration"')
|
|
||||||
slow: marks tests as slow running
|
|
||||||
filterwarnings =
|
|
||||||
ignore::DeprecationWarning
|
|
||||||
@@ -1,57 +0,0 @@
|
|||||||
# workshop-baker/src
|
|
||||||
|
|
||||||
**Generated:** 2026-05-19
|
|
||||||
**Commit:** 28abc5d
|
|
||||||
|
|
||||||
CLI tool and runtime for executing CI/CD pipelines. Orchestrates prebake → bake → finalize stages with notification dispatch.
|
|
||||||
|
|
||||||
## STRUCTURE
|
|
||||||
```
|
|
||||||
src/
|
|
||||||
├── lib.rs # Module declarations + Cli struct + Commands enum
|
|
||||||
├── main.rs # tokio::main entry point
|
|
||||||
├── bare.rs # Bare mode (run individual stages)
|
|
||||||
├── error.rs # CliError (wraps stage errors)
|
|
||||||
├── bake.rs # Bake orchestration (script → execution)
|
|
||||||
├── bake/ # Script parsing, decorator DSL, scheduling
|
|
||||||
├── prebake.rs # Prebake orchestration (env setup)
|
|
||||||
├── prebake/ # Config, stages, security, env providers
|
|
||||||
├── finalize.rs # Finalize orchestration (post-build)
|
|
||||||
├── finalize/ # Config, plugins, hooks, templates
|
|
||||||
├── notify.rs # Notification system (batching, retry)
|
|
||||||
├── notify/ # Types, templates, interactive, queue
|
|
||||||
├── daemon/ # Empty (Daemon command = unimplemented!)
|
|
||||||
├── monitor/ # Resource usage monitoring (cgroups)
|
|
||||||
├── types.rs # Module re-exports
|
|
||||||
├── types/ # buildstatus, resource types
|
|
||||||
├── utils.rs # Module root (fetch submodule)
|
|
||||||
└── utils/ # fetch.rs
|
|
||||||
```
|
|
||||||
|
|
||||||
## WHERE TO LOOK
|
|
||||||
| Task | Location |
|
|
||||||
|------|----------|
|
|
||||||
| CLI args | `lib.rs:21` - `Cli` struct |
|
|
||||||
| Entry point | `main.rs:91` - `#[tokio::main]` |
|
|
||||||
| Pipeline orchestration | `main.rs:68-74` - prebake→bake→finalize chain |
|
|
||||||
| Bare mode (single stage) | `bare.rs:14` - `run_bare()` |
|
|
||||||
| Script parsing | `bake/parser.rs:56` - `parse_script()` |
|
|
||||||
| Decorator DSL | `bake/decorator.rs` - `Decorator` enum |
|
|
||||||
| Prebake stages | `prebake/stage/` - Bootstrap→DepsSystem→DepsUser→Hooks→Ready |
|
|
||||||
| Plugin system | `finalize/plugin.rs` + `finalize/plugin/` |
|
|
||||||
| Notification dispatch | `notify.rs:66` - `notify()` async loop |
|
|
||||||
| Notification types | `notify/types.rs` - `NotificationRule`, `ParsedTrigger` |
|
|
||||||
|
|
||||||
## CONVENTIONS
|
|
||||||
- **ExecutionContext**: Passed through all stages, carries task_id, username, env_vars, timeout
|
|
||||||
- **PrebakeStage**: Ordered enum Bootstrap→EarlyHook→DepsSystem→DepsUser→LateHook→Ready
|
|
||||||
- **Decorator**: `# @name(args)` comments above shell function declarations
|
|
||||||
- **Notification**: Plugin-based dispatch with batching, exponential backoff, priority groups
|
|
||||||
- **Engine**: Delegated to `workshop-engine` crate (formerly in-tree)
|
|
||||||
- **Dual-mode**: CLI commands (standalone) vs Daemon mode (unimplemented)
|
|
||||||
|
|
||||||
## ANTI-PATTERNS
|
|
||||||
1. **Empty daemon/** directory — Daemon command is `unimplemented!()`
|
|
||||||
2. **Dead code allowed** — `lints.rust.dead_code = "allow"` in Cargo.toml
|
|
||||||
3. **Engine was extracted** — workshop-engine is its own crate now, src/engine/ removed
|
|
||||||
4. **Python test infra exists but files removed** — pytest.ini remains, test dir is empty
|
|
||||||
+177
-55
@@ -1,20 +1,27 @@
|
|||||||
use crate::bake::constant::{DEFAULT_WORKSPACE, TRIVIAL_SCRIPT_NAME};
|
use crate::types::debug::DebugFeature;
|
||||||
|
use crate::bake::builder::RenderMode;
|
||||||
|
use crate::bake::constant::DEFAULT_WORKSPACE;
|
||||||
|
use crate::bake::decorator::Decorator;
|
||||||
use crate::bake::error::BakeError;
|
use crate::bake::error::BakeError;
|
||||||
use crate::bake::parser::Function;
|
use crate::bake::util::{resolve_mode, FuncMode};
|
||||||
use crate::cli::Cli;
|
use crate::cli::Cli;
|
||||||
use crate::prebake;
|
use crate::prebake;
|
||||||
use crate::prebake::PrebakeConfig;
|
use crate::prebake::PrebakeConfig;
|
||||||
use crate::prebake::security::get_drop_after;
|
use std::collections::{HashMap, HashSet};
|
||||||
use crate::prebake::stage::PrebakeStage;
|
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
|
use std::time::Duration;
|
||||||
|
use workshop_baker_params::BakeSettings;
|
||||||
use workshop_engine::{Engine, EventSender, ExecutionContext};
|
use workshop_engine::{Engine, EventSender, ExecutionContext};
|
||||||
|
|
||||||
mod builder;
|
mod builder;
|
||||||
pub mod constant;
|
pub mod constant;
|
||||||
mod decorator;
|
mod decorator;
|
||||||
pub mod error;
|
pub mod error;
|
||||||
|
mod execute;
|
||||||
pub mod parser;
|
pub mod parser;
|
||||||
mod schedule;
|
mod schedule;
|
||||||
|
pub mod trivial;
|
||||||
|
pub mod util;
|
||||||
|
|
||||||
pub async fn bake(
|
pub async fn bake(
|
||||||
script_path: &Path,
|
script_path: &Path,
|
||||||
@@ -23,12 +30,18 @@ pub async fn bake(
|
|||||||
_cli: &Cli,
|
_cli: &Cli,
|
||||||
ctx: &mut ExecutionContext,
|
ctx: &mut ExecutionContext,
|
||||||
event_tx: EventSender,
|
event_tx: EventSender,
|
||||||
|
nevent_tx: Option<crate::notify::types::NotificationEventSender>,
|
||||||
) -> Result<(), BakeError> {
|
) -> Result<(), BakeError> {
|
||||||
let script_content = std::fs::read_to_string(script_path)?;
|
log::debug!("Reading script: {:?}", script_path);
|
||||||
let prebake_content = std::fs::read_to_string(prebake_path)?;
|
let script_content = std::fs::read_to_string(script_path).map_err(|e| {
|
||||||
|
BakeError::IoError { path: script_path.display().to_string(), source: e }
|
||||||
|
})?;
|
||||||
|
log::debug!("Reading prebake.yml: {:?}", prebake_path);
|
||||||
|
let prebake_content = std::fs::read_to_string(prebake_path).map_err(|e| {
|
||||||
|
BakeError::IoError { path: prebake_path.display().to_string(), source: e }
|
||||||
|
})?;
|
||||||
let prebake = prebake::parse(prebake_content.as_str()).map_err(|e| {
|
let prebake = prebake::parse(prebake_content.as_str()).map_err(|e| {
|
||||||
log::error!("Failed to parse prebake.yml: {}", e);
|
log::error!("Failed to parse prebake.yml: {}", e);
|
||||||
log::error!("This is unexpected!");
|
|
||||||
BakeError::YamlParseError(e)
|
BakeError::YamlParseError(e)
|
||||||
})?;
|
})?;
|
||||||
let workspace = get_workspace(&prebake);
|
let workspace = get_workspace(&prebake);
|
||||||
@@ -39,44 +52,174 @@ pub async fn bake(
|
|||||||
.iter()
|
.iter()
|
||||||
.any(|d| matches!(d, decorator::Decorator::Pipeline))
|
.any(|d| matches!(d, decorator::Decorator::Pipeline))
|
||||||
});
|
});
|
||||||
let trivial = false;
|
// TODO: if PIP(pipeline implicit parameter).trivial is set, always use RenderMode::Off
|
||||||
let use_template = true;
|
let render_mode = RenderMode::Full;
|
||||||
|
|
||||||
let engine = Engine::new();
|
let engine = Engine::new();
|
||||||
|
// TODO: resolve from merged param layers (PIP integration)
|
||||||
|
let bake_settings = BakeSettings::default();
|
||||||
if let Some(env) = &prebake.envvars
|
if let Some(env) = &prebake.envvars
|
||||||
&& let Some(prebake_env) = &env.bake
|
&& let Some(prebake_env) = &env.bake
|
||||||
{
|
{
|
||||||
ctx.env_vars.extend(prebake_env.clone());
|
ctx.env_vars.extend(prebake_env.clone());
|
||||||
}
|
}
|
||||||
|
|
||||||
if !has_pipeline || trivial {
|
if !has_pipeline {
|
||||||
let function = Function {
|
log::trace!("Running trivial script");
|
||||||
name: TRIVIAL_SCRIPT_NAME.to_string(),
|
trivial::run_trivial(
|
||||||
decorators: vec![],
|
&script_content,
|
||||||
body: script_content,
|
bake_base_path,
|
||||||
};
|
&workspace,
|
||||||
let script =
|
&engine,
|
||||||
builder::build_script(&function, bake_base_path, &workspace, None, use_template)?;
|
ctx,
|
||||||
let result = engine.execute_script(&script, ctx, &event_tx).await;
|
&event_tx,
|
||||||
result?;
|
render_mode,
|
||||||
|
)
|
||||||
|
.await?;
|
||||||
} else {
|
} else {
|
||||||
let sorted_functions = schedule::sort_function(functions.pipeline_functions)?;
|
log::trace!("Running pipeline script");
|
||||||
let remaining = functions.remaining_code;
|
let remaining = functions.remaining_code;
|
||||||
for func in sorted_functions {
|
let (mut dag, func_map) = schedule::build_dag(functions.pipeline_functions)?;
|
||||||
let mut modified_func = func.clone();
|
|
||||||
modified_func.body = remaining.clone() + &func.body;
|
// Pre-scan: pipe sources + decorator conflict checks.
|
||||||
let script = builder::build_script(
|
let mut pipe_sources: HashSet<String> = HashSet::new();
|
||||||
&modified_func,
|
for func in func_map.values() {
|
||||||
bake_base_path,
|
decorator::check_conflicts(func)?;
|
||||||
&workspace,
|
for decorator in &func.decorators {
|
||||||
None,
|
if let Decorator::Pipe(names) = decorator {
|
||||||
use_template,
|
for source in names {
|
||||||
)?;
|
if *source != func.name {
|
||||||
let result = engine.execute_script(&script, ctx, &event_tx).await;
|
pipe_sources.insert(source.clone());
|
||||||
result?;
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut stdout_cache: HashMap<String, String> = HashMap::new();
|
||||||
|
let mut async_handles: Vec<tokio::task::JoinHandle<()>> = Vec::new();
|
||||||
|
|
||||||
|
while !dag.is_empty() {
|
||||||
|
let scheduler_debug = DebugFeature::from_bits_retain(ctx.debug_flags).contains(DebugFeature::Scheduler);
|
||||||
|
if scheduler_debug {
|
||||||
|
log::info!("[BAKE_SCHED] ready={} seq={} wild={} groups={:?}",
|
||||||
|
dag.ready_count(),
|
||||||
|
dag.ready_sequential().len(),
|
||||||
|
dag.ready_wildcards().len(),
|
||||||
|
dag.ready_groups(),
|
||||||
|
);
|
||||||
|
} else {
|
||||||
|
log::debug!("[BAKE_SCHED] ready={} seq={} wild={} groups={:?}",
|
||||||
|
dag.ready_count(),
|
||||||
|
dag.ready_sequential().len(),
|
||||||
|
dag.ready_wildcards().len(),
|
||||||
|
dag.ready_groups(),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
// Rule 1: sequential always goes first
|
||||||
|
let seq_candidates = dag.ready_sequential();
|
||||||
|
if let Some(seq_name) = seq_candidates.first() {
|
||||||
|
let seq_name = seq_name.to_string();
|
||||||
|
log::info!("[BAKE_SCHED] seq {}", seq_name);
|
||||||
|
let func = &func_map[&seq_name];
|
||||||
|
let mode = resolve_mode(func);
|
||||||
|
|
||||||
|
match mode {
|
||||||
|
FuncMode::Daemon => {
|
||||||
|
execute::execute_daemon(
|
||||||
|
&engine, func, ctx, &event_tx,
|
||||||
|
&workspace, &remaining, bake_base_path,
|
||||||
|
Duration::from_secs(bake_settings.health_period),
|
||||||
|
Duration::from_secs(bake_settings.health_timeout),
|
||||||
|
bake_settings.health_max_retries,
|
||||||
|
).await?;
|
||||||
|
}
|
||||||
|
FuncMode::Async => {
|
||||||
|
let handle = execute::execute_async(
|
||||||
|
func, ctx, &event_tx,
|
||||||
|
&workspace, &remaining, bake_base_path,
|
||||||
|
);
|
||||||
|
async_handles.push(handle);
|
||||||
|
}
|
||||||
|
FuncMode::Normal => {
|
||||||
|
let result = execute::execute_normal(
|
||||||
|
&engine, func, ctx, &event_tx,
|
||||||
|
&workspace, &remaining, bake_base_path, &stdout_cache,
|
||||||
|
).await;
|
||||||
|
match result {
|
||||||
|
Ok(stdout) => {
|
||||||
|
if pipe_sources.contains(&seq_name) {
|
||||||
|
stdout_cache.insert(seq_name.clone(), stdout);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Err(e) if dag.is_fallible(&seq_name) => {
|
||||||
|
log::warn!("'{}' failed (fallible): {}", seq_name, e);
|
||||||
|
}
|
||||||
|
Err(e) => return Err(e),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Pop AFTER execution (Daemon/Async pop at start)
|
||||||
|
if !matches!(mode, FuncMode::Daemon | FuncMode::Async) {
|
||||||
|
dag.pop(&seq_name).map_err(|e| BakeError::IncompatibleDecorators {
|
||||||
|
function: seq_name.clone(),
|
||||||
|
reason: e.to_string(),
|
||||||
|
})?;
|
||||||
|
} else {
|
||||||
|
dag.pop(&seq_name).map_err(|e| BakeError::IncompatibleDecorators {
|
||||||
|
function: seq_name.clone(),
|
||||||
|
reason: e.to_string(),
|
||||||
|
})?;
|
||||||
|
}
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
// All remaining are parallel — pick a batch
|
||||||
|
if let Some(batch) = dag.pick_batch() {
|
||||||
|
log::info!("[BAKE_SCHED] par {:?}", batch);
|
||||||
|
let results = execute::fire_parallel_batch(
|
||||||
|
&batch, &engine, &func_map, ctx, &event_tx,
|
||||||
|
&workspace, &remaining, bake_base_path, &stdout_cache,
|
||||||
|
).await?;
|
||||||
|
for (name, stdout) in results {
|
||||||
|
if pipe_sources.contains(&name) {
|
||||||
|
stdout_cache.insert(name.clone(), stdout);
|
||||||
|
}
|
||||||
|
if let Err(e) = dag.pop(&name) {
|
||||||
|
log::warn!("Failed to pop '{}' from DAG: {}", name, e);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else if dag.is_stalled() {
|
||||||
|
// Deadlock: named parallel groups exist but can't complete
|
||||||
|
let stalled = dag.stalled_groups();
|
||||||
|
let remaining = dag.remaining();
|
||||||
|
return Err(BakeError::IncompatibleDecorators {
|
||||||
|
function: remaining.join(", "),
|
||||||
|
reason: format!(
|
||||||
|
"parallel group(s) {:?} can never complete — blocked by @after chain",
|
||||||
|
stalled
|
||||||
|
),
|
||||||
|
});
|
||||||
|
} else {
|
||||||
|
// Ready set is empty — cycle
|
||||||
|
let remaining = dag.remaining();
|
||||||
|
if !remaining.is_empty() {
|
||||||
|
return Err(BakeError::CircularDependency(
|
||||||
|
remaining.into_iter().map(|s| s.to_string()).collect(),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Wait for all @async tasks to finish
|
||||||
|
for handle in async_handles {
|
||||||
|
let _ = handle.await;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
crate::notify::types::send_stage_event(&nevent_tx, crate::types::buildstatus::StagePhase::Bake, "bake", crate::types::buildstatus::StageOutcome::Success).await;
|
||||||
|
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -95,26 +238,5 @@ fn get_workspace(prebake_config: &PrebakeConfig) -> PathBuf {
|
|||||||
.unwrap_or_else(|| PathBuf::from(DEFAULT_WORKSPACE))
|
.unwrap_or_else(|| PathBuf::from(DEFAULT_WORKSPACE))
|
||||||
}
|
}
|
||||||
|
|
||||||
fn privileged(prebake_config: &PrebakeConfig) -> bool {
|
// ── AI Coding Agent marker ──
|
||||||
if let Some(bootstrap) = &prebake_config.bootstrap
|
// Code in this module PARTIALLY or FULLY utilized AI Coding Agent
|
||||||
&& bootstrap.user == "root"
|
|
||||||
{
|
|
||||||
return true;
|
|
||||||
}
|
|
||||||
let drop_after = get_drop_after(&prebake_config.security).unwrap_or_default();
|
|
||||||
PrebakeStage::Never == drop_after
|
|
||||||
}
|
|
||||||
|
|
||||||
type TaskResult = Result<(), BakeError>;
|
|
||||||
|
|
||||||
type TaskFn = Box<dyn FnOnce() -> TaskResult + Send>;
|
|
||||||
|
|
||||||
// TODO: Learn closure and continue
|
|
||||||
// async fn run<F>(f:F) -> TaskFn {
|
|
||||||
// let engine = Engine::new();
|
|
||||||
// engine.execute_script
|
|
||||||
// }
|
|
||||||
|
|
||||||
// fn with_timeout<F>(f: F, duration: i32) -> TaskFn {
|
|
||||||
|
|
||||||
// }
|
|
||||||
|
|||||||
@@ -1,48 +0,0 @@
|
|||||||
# workshop-baker/src/bake
|
|
||||||
|
|
||||||
**Generated:** 2026-04-22
|
|
||||||
**Commit:** 7b3b71a
|
|
||||||
|
|
||||||
Script parsing and build orchestration with decorator DSL.
|
|
||||||
|
|
||||||
## STRUCTURE
|
|
||||||
|
|
||||||
```
|
|
||||||
bake/
|
|
||||||
├── parser.rs # @decorator-aware shell script parser
|
|
||||||
├── decorator.rs # Decorator enum + 100+ unit tests
|
|
||||||
├── builder.rs # Script template assembly
|
|
||||||
├── schedule.rs # Function ordering via @after dependencies
|
|
||||||
├── error.rs # BakeError enum
|
|
||||||
└── constant.rs # DEFAULT_WORKSPACE, TRIVIAL_SCRIPT_NAME
|
|
||||||
```
|
|
||||||
|
|
||||||
## WHERE TO LOOK
|
|
||||||
|
|
||||||
| Task | Location |
|
|
||||||
|------|----------|
|
|
||||||
Parse script | `parser.rs:56` - `parse_script()` |
|
|
||||||
| Decorator types | `decorator.rs` - `Decorator` enum variants |
|
|
||||||
| Build script | `builder.rs` - template builder |
|
|
||||||
| Schedule deps | `schedule.rs` - @after resolution |
|
|
||||||
|
|
||||||
## DECORATORS
|
|
||||||
|
|
||||||
| Decorator | Purpose |
|
|
||||||
|-----------|---------|
|
|
||||||
| `@pipeline` | Marks main build function |
|
|
||||||
| `@timeout(N)` | Execution timeout seconds |
|
|
||||||
| `@retry(N, M)` | Retry N times, M sec delay |
|
|
||||||
| `@parallel` | Run concurrently |
|
|
||||||
| `@fallible` | Failure doesn't fail pipeline |
|
|
||||||
| `@after(func)` | Dependency ordering |
|
|
||||||
| `@export` | Export vars to env |
|
|
||||||
| `@loop(N)` | Repeat N times |
|
|
||||||
| `@if(COND)` | Conditional execution |
|
|
||||||
|
|
||||||
## CONVENTIONS
|
|
||||||
|
|
||||||
- **Decorator Syntax**: `# @name(args)` comment above function
|
|
||||||
- **Parsing**: Shell grammar with decorator annotation extraction
|
|
||||||
- **Scheduling**: DAG resolution for @after dependencies
|
|
||||||
- **Tests**: `decorator.rs:175-829` has 100+ inline unit tests
|
|
||||||
@@ -1,4 +1,5 @@
|
|||||||
use crate::bake::constant::{PIPELINE_SKIP_ERRORCODE, TRIVIAL_SCRIPT_NAME};
|
use crate::types::debug::DebugFeature;
|
||||||
|
use crate::bake::constant::PIPELINE_SKIP_ERRORCODE;
|
||||||
use crate::bake::decorator::Decorator;
|
use crate::bake::decorator::Decorator;
|
||||||
use crate::bake::error::BakeError;
|
use crate::bake::error::BakeError;
|
||||||
use crate::bake::parser::Function;
|
use crate::bake::parser::Function;
|
||||||
@@ -9,24 +10,45 @@ use std::path::Path;
|
|||||||
use std::path::PathBuf;
|
use std::path::PathBuf;
|
||||||
use tempfile::NamedTempFile;
|
use tempfile::NamedTempFile;
|
||||||
|
|
||||||
/// Template for use_template=false, which just executes the main body without any wrapping.
|
/// Three-level rendering control.
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||||
|
pub enum RenderMode {
|
||||||
|
/// No template, hardcoded "{{ main }}". Debug escape hatch.
|
||||||
|
Off,
|
||||||
|
/// Uses bake_base.sh with essential execution skeleton only.
|
||||||
|
Minimal,
|
||||||
|
/// Uses bake_base.sh with helpers (logs, colors, etc.).
|
||||||
|
Full,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl RenderMode {
|
||||||
|
pub fn as_str(&self) -> &'static str {
|
||||||
|
match self {
|
||||||
|
RenderMode::Off => "off",
|
||||||
|
RenderMode::Minimal => "minimal",
|
||||||
|
RenderMode::Full => "full",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Hardcoded template for RenderMode::Off.
|
||||||
const TRIVIAL_TEMPLATE: &str = "{{ main }}";
|
const TRIVIAL_TEMPLATE: &str = "{{ main }}";
|
||||||
|
|
||||||
/// Renders a `Function` into an executable shell script using a minijinja template.
|
/// Renders a `Function` into an executable shell script using a minijinja template.
|
||||||
/// If `use_template` is true, wraps the body in `bake_base`; otherwise uses a trivial template.
|
///
|
||||||
|
/// `render_mode` controls the template source:
|
||||||
|
/// - `Off`: hardcoded `{{ main }}`, no bake_base file needed.
|
||||||
|
/// - `Minimal` / `Full`: reads bake_base.sh; the template distinguishes the two
|
||||||
|
/// via the `render_mode` context variable.
|
||||||
pub fn build_script(
|
pub fn build_script(
|
||||||
function: &Function,
|
function: &Function,
|
||||||
bake_base: &Path,
|
bake_base: &Path,
|
||||||
target_dir: &Path,
|
target_dir: &Path,
|
||||||
temp_path: Option<PathBuf>,
|
render_mode: RenderMode,
|
||||||
use_template: bool,
|
debug_flags: u32,
|
||||||
) -> Result<PathBuf, BakeError> {
|
) -> Result<PathBuf, BakeError> {
|
||||||
let name = &function.name;
|
let name = &function.name;
|
||||||
let trivial = name == TRIVIAL_SCRIPT_NAME;
|
|
||||||
let body = &function.body;
|
let body = &function.body;
|
||||||
let export = function
|
|
||||||
.find_decorator(|d| matches!(d, Decorator::Export).then_some(()))
|
|
||||||
.is_some();
|
|
||||||
let condition = function.find_decorator(|d| {
|
let condition = function.find_decorator(|d| {
|
||||||
if let Decorator::If(content) = d {
|
if let Decorator::If(content) = d {
|
||||||
Some(content.clone())
|
Some(content.clone())
|
||||||
@@ -34,57 +56,59 @@ pub fn build_script(
|
|||||||
None
|
None
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
let temp_path = temp_path.unwrap_or_else(|| PathBuf::from("/tmp"));
|
// let temp_path = temp_path.unwrap_or_else(|| PathBuf::from("/tmp"));
|
||||||
|
let temp_path = target_dir;
|
||||||
let filename = format!("bakefn_{}.sh", name);
|
let filename = format!("bakefn_{}.sh", name);
|
||||||
let target = target_dir.join(filename);
|
let target = target_dir.join(filename);
|
||||||
|
|
||||||
let bake_base_content = if use_template {
|
let template_content = match render_mode {
|
||||||
std::fs::read_to_string(bake_base).map_err(|e| BakeError::ScriptGenerationFailed {
|
RenderMode::Off => TRIVIAL_TEMPLATE.to_string(),
|
||||||
script: bake_base.display().to_string(),
|
RenderMode::Minimal | RenderMode::Full => {
|
||||||
reason: e.to_string(),
|
let script_debug = DebugFeature::from_bits_retain(debug_flags).contains(DebugFeature::Script);
|
||||||
})?
|
if script_debug {
|
||||||
} else {
|
log::info!("Reading bake_base: {:?}", bake_base);
|
||||||
TRIVIAL_TEMPLATE.to_string()
|
} else {
|
||||||
|
log::debug!("Reading bake_base: {:?}", bake_base);
|
||||||
|
}
|
||||||
|
std::fs::read_to_string(bake_base).map_err(|e| BakeError::IoError {
|
||||||
|
path: bake_base.display().to_string(),
|
||||||
|
source: e,
|
||||||
|
})?
|
||||||
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
let condition_code = match condition {
|
let condition_code = match condition {
|
||||||
Some(content) => {
|
Some(content) => {
|
||||||
|
// with some bash workaround
|
||||||
format!(
|
format!(
|
||||||
"if [ ! {} ]; then\n exit {}\nfi\n",
|
"if [[ ! {} ]]; then\n exit {}\nfi\n",
|
||||||
content, PIPELINE_SKIP_ERRORCODE
|
content, PIPELINE_SKIP_ERRORCODE
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
None => String::new(),
|
None => String::new(),
|
||||||
};
|
};
|
||||||
|
|
||||||
let mut preexport_code = String::new();
|
// preexport/export: stubs for the future IPC-based @export.
|
||||||
let mut export_code = String::new();
|
let preexport_code = String::new();
|
||||||
if !trivial {
|
let export_code = String::new();
|
||||||
if export {
|
|
||||||
preexport_code = format!("declare -xp > /dev/shm/bakeenv.before.{}\n", name);
|
|
||||||
export_code = format!(
|
|
||||||
"declare -xp > /dev/shm/bakeenv.after.{}\ndiff -w --new-line-format='%L' --old-line-format='' --unchanged-line-format='' /dev/shm/bakeenv.before.{} /dev/shm/bakeenv.after.{} > /dev/shm/bakeexport.{}\n",
|
|
||||||
name, name, name, name
|
|
||||||
);
|
|
||||||
}
|
|
||||||
preexport_code.push_str(format!(". /dev/shm/bakeexport.{}\n", name).as_str());
|
|
||||||
export_code.push_str(format!("rm -f /dev/shm/bakeenv.*.{}\n", name).as_str());
|
|
||||||
}
|
|
||||||
|
|
||||||
let mut env = Environment::new();
|
let mut env = Environment::new();
|
||||||
env.add_template("script", bake_base_content.as_str())?;
|
env.add_template("script", &template_content)?;
|
||||||
let script = env.get_template("script")?;
|
let rendered = env.get_template("script")?.render(
|
||||||
let rendered = if use_template {
|
context! {
|
||||||
script.render(
|
main => body,
|
||||||
context! {main => body,condition => condition_code, preexport => preexport_code, name => name, export => export_code},
|
condition => condition_code,
|
||||||
)?
|
preexport => preexport_code,
|
||||||
} else {
|
name => name,
|
||||||
script.render(context! {main => body})?
|
export => export_code,
|
||||||
};
|
render_mode => render_mode.as_str(),
|
||||||
|
},
|
||||||
|
)?;
|
||||||
|
|
||||||
let mut temp_file =
|
let mut temp_file =
|
||||||
NamedTempFile::new_in(&temp_path).map_err(|e| BakeError::ScriptGenerationFailed {
|
NamedTempFile::new_in(&temp_path).map_err(|e| BakeError::ScriptGenerationFailed {
|
||||||
script: temp_path.display().to_string(),
|
script: temp_path.display().to_string(),
|
||||||
|
action: "creating temp file".to_string(),
|
||||||
reason: e.to_string(),
|
reason: e.to_string(),
|
||||||
})?;
|
})?;
|
||||||
|
|
||||||
@@ -93,17 +117,20 @@ pub fn build_script(
|
|||||||
.set_permissions(Permissions::from_mode(0o700))
|
.set_permissions(Permissions::from_mode(0o700))
|
||||||
.map_err(|e| BakeError::ScriptGenerationFailed {
|
.map_err(|e| BakeError::ScriptGenerationFailed {
|
||||||
script: temp_path.display().to_string(),
|
script: temp_path.display().to_string(),
|
||||||
|
action: "setting permissions".to_string(),
|
||||||
reason: e.to_string(),
|
reason: e.to_string(),
|
||||||
})?;
|
})?;
|
||||||
|
|
||||||
std::fs::write(&temp_file, rendered).map_err(|e| BakeError::ScriptGenerationFailed {
|
std::fs::write(&temp_file, rendered).map_err(|e| BakeError::ScriptGenerationFailed {
|
||||||
script: temp_path.display().to_string(),
|
script: temp_path.display().to_string(),
|
||||||
|
action: "writing to temp file".to_string(),
|
||||||
reason: e.to_string(),
|
reason: e.to_string(),
|
||||||
})?;
|
})?;
|
||||||
temp_file
|
temp_file
|
||||||
.persist(&target)
|
.persist(&target)
|
||||||
.map_err(|e| BakeError::ScriptGenerationFailed {
|
.map_err(|e| BakeError::ScriptGenerationFailed {
|
||||||
script: target.display().to_string(),
|
script: target.display().to_string(),
|
||||||
|
action: "persisting temp file".to_string(),
|
||||||
reason: e.error.to_string(),
|
reason: e.error.to_string(),
|
||||||
})?;
|
})?;
|
||||||
Ok(target)
|
Ok(target)
|
||||||
@@ -114,7 +141,7 @@ mod tests {
|
|||||||
use super::*;
|
use super::*;
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_build_script_with_template() {
|
fn test_build_script_full_mode() {
|
||||||
let temp_dir = tempfile::tempdir().unwrap();
|
let temp_dir = tempfile::tempdir().unwrap();
|
||||||
let bake_base = temp_dir.path().join("bake_base.sh");
|
let bake_base = temp_dir.path().join("bake_base.sh");
|
||||||
|
|
||||||
@@ -125,7 +152,9 @@ mod tests {
|
|||||||
decorators: vec![],
|
decorators: vec![],
|
||||||
body: "echo 'hello world'".to_string(),
|
body: "echo 'hello world'".to_string(),
|
||||||
};
|
};
|
||||||
let result = build_script(&function, &bake_base, temp_dir.path(), None, true);
|
let result = build_script(
|
||||||
|
&function, &bake_base, temp_dir.path(), RenderMode::Full, 0,
|
||||||
|
);
|
||||||
|
|
||||||
assert!(result.is_ok());
|
assert!(result.is_ok());
|
||||||
let script_path = result.unwrap();
|
let script_path = result.unwrap();
|
||||||
@@ -134,10 +163,10 @@ mod tests {
|
|||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_build_script_trivial_mode() {
|
fn test_build_script_off_mode() {
|
||||||
let temp_dir = tempfile::tempdir().unwrap();
|
let temp_dir = tempfile::tempdir().unwrap();
|
||||||
let function = Function {
|
let function = Function {
|
||||||
name: "trivial".to_string(),
|
name: "test".to_string(),
|
||||||
decorators: vec![],
|
decorators: vec![],
|
||||||
body: "echo 'direct output'".to_string(),
|
body: "echo 'direct output'".to_string(),
|
||||||
};
|
};
|
||||||
@@ -146,8 +175,8 @@ mod tests {
|
|||||||
&function,
|
&function,
|
||||||
Path::new("/nonexistent"),
|
Path::new("/nonexistent"),
|
||||||
temp_dir.path(),
|
temp_dir.path(),
|
||||||
None,
|
RenderMode::Off,
|
||||||
false,
|
0,
|
||||||
);
|
);
|
||||||
|
|
||||||
assert!(result.is_ok());
|
assert!(result.is_ok());
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
pub const DEFAULT_WORKSPACE: &str = "/workspace";
|
pub const DEFAULT_WORKSPACE: &str = "/workspace";
|
||||||
pub const BAKE_SCRIPT_PATH: &str = "bake.sh";
|
pub const BAKE_SCRIPT_PATH: &str = "bake.sh";
|
||||||
pub const TRIVIAL_SCRIPT_NAME: &str = "BAKE_TRIVIAL";
|
pub const TRIVIAL_SCRIPT_NAME: &str = "BAKE_TRIVIAL";
|
||||||
pub const PIPELINE_SKIP_ERRORCODE: u8 = 233;
|
pub const PIPELINE_SKIP_ERRORCODE: i32 = 233;
|
||||||
|
|||||||
@@ -4,11 +4,11 @@ use std::fmt::Display;
|
|||||||
use std::fmt::Formatter;
|
use std::fmt::Formatter;
|
||||||
|
|
||||||
use crate::bake::error::BakeError;
|
use crate::bake::error::BakeError;
|
||||||
|
use crate::bake::parser::Function;
|
||||||
|
|
||||||
lazy_static! {
|
lazy_static! {
|
||||||
// "# @decorator" or "# @decorator(parameters)"
|
// "# @decorator" or "# @decorator(parameters)"
|
||||||
static ref DECORATOR_REGEX: Regex = Regex::new(r"^\s*#\s+@(\w+)\s?(\((.*)\))?").unwrap()
|
static ref DECORATOR_REGEX: Regex = Regex::new(r"^\s*#\s+@(\w+)\s?(\((.*)\))?").unwrap();
|
||||||
;
|
|
||||||
static ref UNEXPECTED_ARGUMENT: &'static str = "decorator does not accept arguments";
|
static ref UNEXPECTED_ARGUMENT: &'static str = "decorator does not accept arguments";
|
||||||
static ref MISSING_ARGUMENT: &'static str = "decorator requires an argument";
|
static ref MISSING_ARGUMENT: &'static str = "decorator requires an argument";
|
||||||
static ref UNKNOWN_DECORATOR: &'static str = "unknown decorator";
|
static ref UNKNOWN_DECORATOR: &'static str = "unknown decorator";
|
||||||
@@ -39,11 +39,121 @@ pub enum Decorator {
|
|||||||
/// Chain functions via pipe with comma-separated names.
|
/// Chain functions via pipe with comma-separated names.
|
||||||
Pipe(Vec<String>),
|
Pipe(Vec<String>),
|
||||||
/// Run concurrently with other parallel functions.
|
/// Run concurrently with other parallel functions.
|
||||||
Parallel,
|
/// `None` = wildcard ("*" group, compatible with any group).
|
||||||
|
/// `Some(name)` = restricted to the named group.
|
||||||
|
Parallel(Option<String>),
|
||||||
/// Run as a background daemon process.
|
/// Run as a background daemon process.
|
||||||
Daemon,
|
Daemon,
|
||||||
/// Health check endpoint for daemon verification.
|
/// Health check command for daemon verification (shell statement).
|
||||||
Health(String),
|
Health(String),
|
||||||
|
/// Fire-and-forget execution; DAG pop happens at start, not end.
|
||||||
|
Async,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns true if the decorator is a flag (position-independent, consumed after execution).
|
||||||
|
pub fn is_flag(d: &Decorator) -> bool {
|
||||||
|
matches!(
|
||||||
|
d,
|
||||||
|
Decorator::Pipeline
|
||||||
|
| Decorator::Fallible
|
||||||
|
| Decorator::Export
|
||||||
|
| Decorator::If(_)
|
||||||
|
| Decorator::After(_)
|
||||||
|
| Decorator::Parallel(_)
|
||||||
|
| Decorator::Health(_)
|
||||||
|
| Decorator::Async
|
||||||
|
| Decorator::Unknown
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns true if the decorator is a combinator (position-dependent, nestable).
|
||||||
|
pub fn is_combinator(d: &Decorator) -> bool {
|
||||||
|
matches!(d, Decorator::Loop(_) | Decorator::Timeout(_) | Decorator::Retry(..))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns true if the decorator is a mode (at most one, alters what gets executed).
|
||||||
|
pub fn is_mode(d: &Decorator) -> bool {
|
||||||
|
matches!(d, Decorator::Pipe(_) | Decorator::Daemon | Decorator::Async)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Extract the parallel group name. Returns "*" for wildcard (no group specified).
|
||||||
|
pub fn parallel_group(d: &Decorator) -> &str {
|
||||||
|
match d {
|
||||||
|
Decorator::Parallel(Some(g)) => g.as_str(),
|
||||||
|
_ => "*",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Check for incompatible decorator combinations on a function.
|
||||||
|
/// Returns `Err(IncompatibleDecorators)` on conflict, `Ok(())` otherwise.
|
||||||
|
pub fn check_conflicts(func: &Function) -> Result<(), BakeError> {
|
||||||
|
let has_daemon = func.decorators.iter().any(|d| matches!(d, Decorator::Daemon));
|
||||||
|
let has_async = func.decorators.iter().any(|d| matches!(d, Decorator::Async));
|
||||||
|
let has_pipe = func.decorators.iter().any(|d| matches!(d, Decorator::Pipe(_)));
|
||||||
|
let has_parallel = func.decorators.iter().any(|d| matches!(d, Decorator::Parallel(_)));
|
||||||
|
let has_health = func.decorators.iter().any(|d| matches!(d, Decorator::Health(_)));
|
||||||
|
let has_export = func.decorators.iter().any(|d| matches!(d, Decorator::Export));
|
||||||
|
let has_combinator = func.decorators.iter().any(|d| is_combinator(d));
|
||||||
|
|
||||||
|
let mode_count = [has_daemon, has_async, has_pipe].iter().filter(|&&x| x).count();
|
||||||
|
if mode_count > 1 {
|
||||||
|
return Err(BakeError::IncompatibleDecorators {
|
||||||
|
function: func.name.clone(),
|
||||||
|
reason: "@daemon, @async, and @pipe are mutually exclusive".into(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
if has_daemon {
|
||||||
|
if has_parallel {
|
||||||
|
return Err(BakeError::IncompatibleDecorators {
|
||||||
|
function: func.name.clone(),
|
||||||
|
reason: "@daemon cannot be combined with @parallel".into(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
if has_combinator {
|
||||||
|
return Err(BakeError::IncompatibleDecorators {
|
||||||
|
function: func.name.clone(),
|
||||||
|
reason: "@daemon cannot be combined with @loop/@retry/@timeout".into(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
if has_export {
|
||||||
|
return Err(BakeError::IncompatibleDecorators {
|
||||||
|
function: func.name.clone(),
|
||||||
|
reason: "@daemon cannot be combined with @export".into(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if has_health && !has_daemon {
|
||||||
|
return Err(BakeError::IncompatibleDecorators {
|
||||||
|
function: func.name.clone(),
|
||||||
|
reason: "@health requires @daemon".into(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
if has_async {
|
||||||
|
if has_parallel {
|
||||||
|
return Err(BakeError::IncompatibleDecorators {
|
||||||
|
function: func.name.clone(),
|
||||||
|
reason: "@async cannot be combined with @parallel".into(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
if has_export {
|
||||||
|
return Err(BakeError::IncompatibleDecorators {
|
||||||
|
function: func.name.clone(),
|
||||||
|
reason: "@async cannot be combined with @export".into(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if has_pipe && has_parallel {
|
||||||
|
return Err(BakeError::IncompatibleDecorators {
|
||||||
|
function: func.name.clone(),
|
||||||
|
reason: "@pipe cannot be combined with @parallel".into(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Display for Decorator {
|
impl Display for Decorator {
|
||||||
@@ -59,9 +169,13 @@ impl Display for Decorator {
|
|||||||
Decorator::Retry(count, delay) => write!(f, "@retry({}, {})", count, delay),
|
Decorator::Retry(count, delay) => write!(f, "@retry({}, {})", count, delay),
|
||||||
Decorator::Export => write!(f, "@export"),
|
Decorator::Export => write!(f, "@export"),
|
||||||
Decorator::Pipe(funcs) => write!(f, "@pipe({})", funcs.join(", ")),
|
Decorator::Pipe(funcs) => write!(f, "@pipe({})", funcs.join(", ")),
|
||||||
Decorator::Parallel => write!(f, "@parallel"),
|
Decorator::Parallel(group) => match group {
|
||||||
|
None => write!(f, "@parallel"),
|
||||||
|
Some(g) => write!(f, "@parallel({})", g),
|
||||||
|
},
|
||||||
Decorator::Daemon => write!(f, "@daemon"),
|
Decorator::Daemon => write!(f, "@daemon"),
|
||||||
Decorator::Health(check) => write!(f, "@health({})", check),
|
Decorator::Health(check) => write!(f, "@health({})", check),
|
||||||
|
Decorator::Async => write!(f, "@async"),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -161,9 +275,17 @@ pub fn parse_decorator(line: &str, line_num: usize) -> Result<Option<Decorator>,
|
|||||||
}
|
}
|
||||||
}),
|
}),
|
||||||
"export" => decorator_noarg(Decorator::Export),
|
"export" => decorator_noarg(Decorator::Export),
|
||||||
"parallel" => decorator_noarg(Decorator::Parallel),
|
"parallel" => {
|
||||||
|
if has_arg {
|
||||||
|
let group = required_arg()?;
|
||||||
|
Ok(Some(Decorator::Parallel(Some(group.to_string()))))
|
||||||
|
} else {
|
||||||
|
Ok(Some(Decorator::Parallel(None)))
|
||||||
|
}
|
||||||
|
}
|
||||||
"daemon" => decorator_noarg(Decorator::Daemon),
|
"daemon" => decorator_noarg(Decorator::Daemon),
|
||||||
"health" => required_arg().map(|a| Some(Decorator::Health(a.to_string()))),
|
"health" => required_arg().map(|a| Some(Decorator::Health(a.to_string()))),
|
||||||
|
"async" => decorator_noarg(Decorator::Async),
|
||||||
"pipe" => required_arg().and_then(|a| {
|
"pipe" => required_arg().and_then(|a| {
|
||||||
let functions: Vec<String> = a
|
let functions: Vec<String> = a
|
||||||
.split(',')
|
.split(',')
|
||||||
@@ -290,7 +412,11 @@ mod tests {
|
|||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_decorator_display_parallel() {
|
fn test_decorator_display_parallel() {
|
||||||
assert_eq!(format!("{}", Decorator::Parallel), "@parallel");
|
assert_eq!(format!("{}", Decorator::Parallel(None)), "@parallel");
|
||||||
|
assert_eq!(
|
||||||
|
format!("{}", Decorator::Parallel(Some("build".to_string()))),
|
||||||
|
"@parallel(build)"
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
@@ -310,6 +436,11 @@ mod tests {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_decorator_display_async() {
|
||||||
|
assert_eq!(format!("{}", Decorator::Async), "@async");
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_decorator_display_unknown() {
|
fn test_decorator_display_unknown() {
|
||||||
assert_eq!(format!("{}", Decorator::Unknown), "@<unknown>");
|
assert_eq!(format!("{}", Decorator::Unknown), "@<unknown>");
|
||||||
@@ -340,7 +471,13 @@ mod tests {
|
|||||||
#[test]
|
#[test]
|
||||||
fn test_parse_decorator_parallel() {
|
fn test_parse_decorator_parallel() {
|
||||||
let result = parse_decorator("# @parallel", 1).unwrap();
|
let result = parse_decorator("# @parallel", 1).unwrap();
|
||||||
assert_eq!(result, Some(Decorator::Parallel));
|
assert_eq!(result, Some(Decorator::Parallel(None)));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_parse_decorator_parallel_with_group() {
|
||||||
|
let result = parse_decorator("# @parallel(build)", 1).unwrap();
|
||||||
|
assert_eq!(result, Some(Decorator::Parallel(Some("build".to_string()))));
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
@@ -349,6 +486,12 @@ mod tests {
|
|||||||
assert_eq!(result, Some(Decorator::Daemon));
|
assert_eq!(result, Some(Decorator::Daemon));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_parse_decorator_async() {
|
||||||
|
let result = parse_decorator("# @async", 1).unwrap();
|
||||||
|
assert_eq!(result, Some(Decorator::Async));
|
||||||
|
}
|
||||||
|
|
||||||
// =====================================================================
|
// =====================================================================
|
||||||
// Tests for parse_decorator - Valid single-argument decorators
|
// Tests for parse_decorator - Valid single-argument decorators
|
||||||
// =====================================================================
|
// =====================================================================
|
||||||
@@ -576,8 +719,8 @@ mod tests {
|
|||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_parse_decorator_parallel_with_arg() {
|
fn test_parse_decorator_parallel_empty_args() {
|
||||||
let result = parse_decorator("# @parallel(true)", 1);
|
let result = parse_decorator("# @parallel()", 1);
|
||||||
assert!(result.is_err());
|
assert!(result.is_err());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -7,8 +7,8 @@ use workshop_engine::{
|
|||||||
|
|
||||||
#[derive(Error, Debug)]
|
#[derive(Error, Debug)]
|
||||||
pub enum BakeError {
|
pub enum BakeError {
|
||||||
#[error("Failed to generate staged script {script}: {reason}")]
|
#[error("Failed to generate staged script {script} when {action}: {reason}")]
|
||||||
ScriptGenerationFailed { script: String, reason: String },
|
ScriptGenerationFailed { script: String, action: String, reason: String },
|
||||||
|
|
||||||
#[error("Template error: {0}")]
|
#[error("Template error: {0}")]
|
||||||
TemplateError(#[from] minijinja::Error),
|
TemplateError(#[from] minijinja::Error),
|
||||||
@@ -19,8 +19,8 @@ pub enum BakeError {
|
|||||||
#[error("Unknown @after of function {0}: {1}")]
|
#[error("Unknown @after of function {0}: {1}")]
|
||||||
UnknownDependency(String, String),
|
UnknownDependency(String, String),
|
||||||
|
|
||||||
#[error("IO Error: {0}")]
|
#[error("IO error on {path}: {source}")]
|
||||||
IoError(#[from] std::io::Error),
|
IoError { path: String, source: std::io::Error },
|
||||||
|
|
||||||
#[error("YAML parse error: {0}")]
|
#[error("YAML parse error: {0}")]
|
||||||
YamlParseError(#[from] serde_yaml::Error),
|
YamlParseError(#[from] serde_yaml::Error),
|
||||||
@@ -34,6 +34,12 @@ pub enum BakeError {
|
|||||||
|
|
||||||
#[error("Script execution error: {0}")]
|
#[error("Script execution error: {0}")]
|
||||||
ScriptExecutionError(#[from] ExecutionError),
|
ScriptExecutionError(#[from] ExecutionError),
|
||||||
|
|
||||||
|
#[error("Incompatible decorators on function '{function}': {reason}")]
|
||||||
|
IncompatibleDecorators { function: String, reason: String },
|
||||||
|
|
||||||
|
#[error("Health probe failed for daemon '{command}': {error}")]
|
||||||
|
HealthProbeFailed { command: String, error: String },
|
||||||
}
|
}
|
||||||
|
|
||||||
impl HasExitCode for BakeError {
|
impl HasExitCode for BakeError {
|
||||||
@@ -43,10 +49,12 @@ impl HasExitCode for BakeError {
|
|||||||
BakeError::TemplateError(_) => EXITCODE_PARSE_ERROR,
|
BakeError::TemplateError(_) => EXITCODE_PARSE_ERROR,
|
||||||
BakeError::CircularDependency(_) => EXITCODE_PARSE_ERROR,
|
BakeError::CircularDependency(_) => EXITCODE_PARSE_ERROR,
|
||||||
BakeError::UnknownDependency(..) => EXITCODE_PARSE_ERROR,
|
BakeError::UnknownDependency(..) => EXITCODE_PARSE_ERROR,
|
||||||
BakeError::IoError(_) => EXITCODE_IO_ERROR,
|
BakeError::IoError { .. } => EXITCODE_IO_ERROR,
|
||||||
BakeError::YamlParseError(_) => EXITCODE_PARSE_ERROR,
|
BakeError::YamlParseError(_) => EXITCODE_PARSE_ERROR,
|
||||||
BakeError::DecoratorParseError { .. } => EXITCODE_PARSE_ERROR,
|
BakeError::DecoratorParseError { .. } => EXITCODE_PARSE_ERROR,
|
||||||
BakeError::ScriptExecutionError(e) => e.exit_code(),
|
BakeError::ScriptExecutionError(e) => e.exit_code(),
|
||||||
|
BakeError::IncompatibleDecorators { .. } => EXITCODE_PARSE_ERROR,
|
||||||
|
BakeError::HealthProbeFailed { .. } => EXITCODE_IO_ERROR,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,383 @@
|
|||||||
|
use crate::types::debug::DebugFeature;
|
||||||
|
use crate::bake::builder::{self, RenderMode};
|
||||||
|
use crate::bake::constant::PIPELINE_SKIP_ERRORCODE;
|
||||||
|
use crate::bake::decorator::{self, Decorator};
|
||||||
|
use crate::bake::error::BakeError;
|
||||||
|
use crate::bake::parser::Function;
|
||||||
|
use crate::bake::util::{is_fallible, resolve_pipe_stdin};
|
||||||
|
use std::collections::HashMap;
|
||||||
|
use std::future::Future;
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
use std::pin::Pin;
|
||||||
|
use std::process::Stdio;
|
||||||
|
use std::time::Duration;
|
||||||
|
use workshop_engine::{Engine, EventSender, ExecutionContext, ExecutionError};
|
||||||
|
|
||||||
|
/// Execute a single function with combinator wrapping.
|
||||||
|
/// Returns the captured stdout (for `@pipe` consumers).
|
||||||
|
pub async fn execute_one(
|
||||||
|
engine: &Engine,
|
||||||
|
func: &Function,
|
||||||
|
ctx: &mut ExecutionContext,
|
||||||
|
event_tx: &EventSender,
|
||||||
|
workspace: &Path,
|
||||||
|
remaining: &str,
|
||||||
|
bake_base: &Path,
|
||||||
|
) -> Result<String, BakeError> {
|
||||||
|
let combos: Vec<&Decorator> = func
|
||||||
|
.decorators
|
||||||
|
.iter()
|
||||||
|
.filter(|d| decorator::is_combinator(d))
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
let mut modified = func.clone();
|
||||||
|
modified.body = remaining.to_string() + &func.body;
|
||||||
|
let script = builder::build_script(
|
||||||
|
&modified, bake_base, workspace, RenderMode::Full, ctx.debug_flags,
|
||||||
|
)?;
|
||||||
|
|
||||||
|
execute_with_combos(engine, &script, ctx, event_tx, &combos).await
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn execute_with_combos(
|
||||||
|
engine: &Engine,
|
||||||
|
script: &PathBuf,
|
||||||
|
ctx: &mut ExecutionContext,
|
||||||
|
event_tx: &EventSender,
|
||||||
|
combos: &[&Decorator],
|
||||||
|
) -> Result<String, BakeError> {
|
||||||
|
build_combinator_chain(engine, script, ctx, event_tx, combos, 0).await
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Builds a combinator execution chain. Returns the captured stdout.
|
||||||
|
/// For `@loop` + `@pipe(self)`, each iteration feeds its stdout back as
|
||||||
|
/// the next iteration's stdin via `ctx.stdin`.
|
||||||
|
fn build_combinator_chain<'a>(
|
||||||
|
engine: &'a Engine,
|
||||||
|
script: &'a PathBuf,
|
||||||
|
ctx: &'a mut ExecutionContext,
|
||||||
|
event_tx: &'a EventSender,
|
||||||
|
combos: &'a [&'a Decorator],
|
||||||
|
idx: usize,
|
||||||
|
) -> Pin<Box<dyn Future<Output = Result<String, BakeError>> + Send + 'a>> {
|
||||||
|
if idx >= combos.len() {
|
||||||
|
return Box::pin(async move {
|
||||||
|
let result = engine.execute_script(script, ctx, event_tx).await;
|
||||||
|
match result {
|
||||||
|
Ok(r) => Ok(r.stdout),
|
||||||
|
Err(ExecutionError::ExecutionFailed { exit_code, .. })
|
||||||
|
if exit_code == PIPELINE_SKIP_ERRORCODE as i32 =>
|
||||||
|
{
|
||||||
|
log::debug!("Function skipped (@if condition)");
|
||||||
|
Ok(String::new())
|
||||||
|
}
|
||||||
|
Err(e) => Err(BakeError::ScriptExecutionError(e)),
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
match combos[idx] {
|
||||||
|
Decorator::Loop(count) => Box::pin(async move {
|
||||||
|
let mut last_stdout = String::new();
|
||||||
|
for i in 0..*count {
|
||||||
|
let script_debug = DebugFeature::from_bits_retain(ctx.debug_flags).contains(DebugFeature::Script);
|
||||||
|
if script_debug {
|
||||||
|
log::info!("Loop {}/{}", i + 1, count);
|
||||||
|
} else {
|
||||||
|
log::debug!("Loop {}/{}", i + 1, count);
|
||||||
|
}
|
||||||
|
last_stdout = build_combinator_chain(
|
||||||
|
engine, script, ctx, event_tx, combos, idx + 1,
|
||||||
|
)
|
||||||
|
.await?;
|
||||||
|
// Feed stdout as stdin for the next iteration (@pipe(self))
|
||||||
|
if i + 1 < *count {
|
||||||
|
ctx.stdin = Some(last_stdout.clone().into_bytes());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(last_stdout)
|
||||||
|
}),
|
||||||
|
Decorator::Timeout(secs) => Box::pin(async move {
|
||||||
|
tokio::time::timeout(
|
||||||
|
Duration::from_secs(*secs as u64),
|
||||||
|
build_combinator_chain(
|
||||||
|
engine, script, ctx, event_tx, combos, idx + 1,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.map_err(|_| BakeError::ScriptExecutionError(ExecutionError::Timeout))?
|
||||||
|
}),
|
||||||
|
Decorator::Retry(retries, delay_secs) => Box::pin(async move {
|
||||||
|
let max_attempts = *retries as usize + 1;
|
||||||
|
let delay = Duration::from_secs(*delay_secs as u64);
|
||||||
|
let mut last_err = None;
|
||||||
|
for attempt in 0..max_attempts {
|
||||||
|
match build_combinator_chain(
|
||||||
|
engine, script, ctx, event_tx, combos, idx + 1,
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
{
|
||||||
|
Ok(stdout) => return Ok(stdout),
|
||||||
|
Err(e) => {
|
||||||
|
if matches!(
|
||||||
|
&e,
|
||||||
|
BakeError::ScriptExecutionError(ExecutionError::Timeout)
|
||||||
|
) {
|
||||||
|
return Err(e);
|
||||||
|
}
|
||||||
|
if attempt < max_attempts - 1 {
|
||||||
|
last_err = Some(e);
|
||||||
|
log::warn!(
|
||||||
|
"Retry {}/{} after {}s",
|
||||||
|
attempt + 1,
|
||||||
|
retries,
|
||||||
|
delay_secs
|
||||||
|
);
|
||||||
|
tokio::time::sleep(delay).await;
|
||||||
|
} else {
|
||||||
|
return Err(e);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Err(last_err.unwrap())
|
||||||
|
}),
|
||||||
|
_ => build_combinator_chain(
|
||||||
|
engine, script, ctx, event_tx, combos, idx + 1,
|
||||||
|
),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Fire a batch of parallel nodes concurrently.
|
||||||
|
///
|
||||||
|
/// All nodes in the batch are waited on before returning. A non-`@fallible`
|
||||||
|
/// failure is surfaced only after all nodes complete (GitHub/GitLab/Concourse
|
||||||
|
/// convention).
|
||||||
|
///
|
||||||
|
/// Returns a `(name, stdout)` pair for each completed node so the caller can
|
||||||
|
/// populate the stdout cache for `@pipe`.
|
||||||
|
pub async fn fire_parallel_batch(
|
||||||
|
batch: &[String],
|
||||||
|
_engine: &Engine,
|
||||||
|
func_map: &HashMap<String, Function>,
|
||||||
|
ctx: &mut ExecutionContext,
|
||||||
|
event_tx: &EventSender,
|
||||||
|
workspace: &Path,
|
||||||
|
remaining: &str,
|
||||||
|
bake_base: &Path,
|
||||||
|
stdout_cache: &HashMap<String, String>,
|
||||||
|
) -> Result<Vec<(String, String)>, BakeError> {
|
||||||
|
let handles: Vec<_> = batch
|
||||||
|
.iter()
|
||||||
|
.map(|name| {
|
||||||
|
let function = func_map[name.as_str()].clone();
|
||||||
|
let mut task_ctx = ctx.clone();
|
||||||
|
// Resolve per-task stdin from @pipe + stdout_cache
|
||||||
|
task_ctx.stdin = resolve_pipe_stdin(&function, stdout_cache);
|
||||||
|
let task_events = event_tx.clone();
|
||||||
|
let task_workspace = workspace.to_path_buf();
|
||||||
|
let task_remaining = remaining.to_string();
|
||||||
|
let task_bake_base = bake_base.to_path_buf();
|
||||||
|
let task_name = name.clone();
|
||||||
|
tokio::spawn(async move {
|
||||||
|
let task_engine = Engine::new();
|
||||||
|
let result = execute_one(
|
||||||
|
&task_engine,
|
||||||
|
&function,
|
||||||
|
&mut task_ctx,
|
||||||
|
&task_events,
|
||||||
|
&task_workspace,
|
||||||
|
&task_remaining,
|
||||||
|
&task_bake_base,
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
(task_name, result)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
let mut first_fatal: Option<BakeError> = None;
|
||||||
|
let mut results: Vec<(String, String)> = Vec::with_capacity(handles.len());
|
||||||
|
for handle in handles {
|
||||||
|
let (name, result) = handle.await.map_err(|_| {
|
||||||
|
BakeError::ScriptExecutionError(ExecutionError::ExecutionFailed {
|
||||||
|
task_id: "parallel".to_string(),
|
||||||
|
exit_code: -1,
|
||||||
|
})
|
||||||
|
})?;
|
||||||
|
match result {
|
||||||
|
Ok(stdout) => results.push((name, stdout)),
|
||||||
|
Err(err) if is_fallible(&func_map[&name]) => {
|
||||||
|
log::warn!("'{}' failed (fallible, non-fatal): {}", name, err);
|
||||||
|
}
|
||||||
|
Err(err) => {
|
||||||
|
log::error!("'{}' failed (fatal): {}", name, err);
|
||||||
|
first_fatal.get_or_insert(err);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if let Some(err) = first_fatal {
|
||||||
|
return Err(err);
|
||||||
|
}
|
||||||
|
Ok(results)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Spawn an `@async` function in the background.
|
||||||
|
/// Returns a `JoinHandle` that the caller must await before returning.
|
||||||
|
pub fn execute_async(
|
||||||
|
func: &Function,
|
||||||
|
ctx: &ExecutionContext,
|
||||||
|
event_tx: &EventSender,
|
||||||
|
workspace: &Path,
|
||||||
|
remaining: &str,
|
||||||
|
bake_base: &Path,
|
||||||
|
) -> tokio::task::JoinHandle<()> {
|
||||||
|
let mut task_ctx = ctx.clone();
|
||||||
|
let task_events = event_tx.clone();
|
||||||
|
let task_workspace = workspace.to_path_buf();
|
||||||
|
let task_remaining = remaining.to_string();
|
||||||
|
let task_bake_base = bake_base.to_path_buf();
|
||||||
|
let task_func = func.clone();
|
||||||
|
tokio::spawn(async move {
|
||||||
|
let task_engine = Engine::new();
|
||||||
|
let _ = execute_one(
|
||||||
|
&task_engine, &task_func, &mut task_ctx,
|
||||||
|
&task_events, &task_workspace,
|
||||||
|
&task_remaining, &task_bake_base,
|
||||||
|
).await;
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Execute a Normal-mode function, resolving pipe stdin first.
|
||||||
|
pub async fn execute_normal(
|
||||||
|
engine: &Engine,
|
||||||
|
func: &Function,
|
||||||
|
ctx: &mut ExecutionContext,
|
||||||
|
event_tx: &EventSender,
|
||||||
|
workspace: &Path,
|
||||||
|
remaining: &str,
|
||||||
|
bake_base: &Path,
|
||||||
|
stdout_cache: &HashMap<String, String>,
|
||||||
|
) -> Result<String, BakeError> {
|
||||||
|
ctx.stdin = resolve_pipe_stdin(func, stdout_cache);
|
||||||
|
execute_one(engine, func, ctx, event_tx, workspace, remaining, bake_base).await
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Daemon / Health ─────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// Execute a `@daemon` function: build script, spawn, periodic health-probe
|
||||||
|
/// (blocking until healthy), then return.
|
||||||
|
pub async fn execute_daemon(
|
||||||
|
engine: &Engine,
|
||||||
|
func: &Function,
|
||||||
|
ctx: &ExecutionContext,
|
||||||
|
event_tx: &EventSender,
|
||||||
|
workspace: &Path,
|
||||||
|
remaining: &str,
|
||||||
|
bake_base: &Path,
|
||||||
|
health_period: Duration,
|
||||||
|
health_timeout: Duration,
|
||||||
|
health_max_retries: u32,
|
||||||
|
) -> Result<(), BakeError> {
|
||||||
|
let fallible = is_fallible(func);
|
||||||
|
|
||||||
|
let mut modified = func.clone();
|
||||||
|
modified.body = remaining.to_string() + &func.body;
|
||||||
|
let script = builder::build_script(
|
||||||
|
&modified, bake_base, workspace, RenderMode::Full, ctx.debug_flags,
|
||||||
|
)?;
|
||||||
|
|
||||||
|
// Spawn without waiting
|
||||||
|
let mut command = tokio::process::Command::new(ctx.shell.as_str());
|
||||||
|
command.arg("-c");
|
||||||
|
command.arg(script.to_str().unwrap());
|
||||||
|
command.current_dir(&ctx.working_dir);
|
||||||
|
command.envs(&ctx.env_vars);
|
||||||
|
command.stdin(Stdio::null());
|
||||||
|
command.stdout(Stdio::null());
|
||||||
|
command.stderr(Stdio::null());
|
||||||
|
#[cfg(unix)]
|
||||||
|
command.process_group(0);
|
||||||
|
|
||||||
|
//let child = // Necessary?
|
||||||
|
match command.spawn() {
|
||||||
|
Ok(c) => c,
|
||||||
|
Err(e) if fallible => {
|
||||||
|
log::warn!("Failed to spawn daemon '{}' (fallible): {}", func.name, e);
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
Err(_) => {
|
||||||
|
return Err(BakeError::ScriptExecutionError(ExecutionError::ExecutionFailed {
|
||||||
|
task_id: func.name.clone(),
|
||||||
|
exit_code: -1,
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
// Periodic health probe — block until healthy, retry exhausted, or fatal
|
||||||
|
if let Some(health_cmd) = func.decorators.iter().find_map(|d| match d {
|
||||||
|
Decorator::Health(cmd) => Some(cmd.as_str()),
|
||||||
|
_ => None,
|
||||||
|
}) {
|
||||||
|
let mut retries: u32 = 0;
|
||||||
|
loop {
|
||||||
|
let probe_result = tokio::time::timeout(
|
||||||
|
health_timeout,
|
||||||
|
health_probe_one(engine, health_cmd, remaining, ctx, event_tx),
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
|
||||||
|
match probe_result {
|
||||||
|
Ok(Ok(())) => break, // healthy — done
|
||||||
|
_ if retries + 1 < health_max_retries => {
|
||||||
|
retries += 1;
|
||||||
|
log::warn!(
|
||||||
|
"Health probe for '{}' failed ({}/{}), retrying in {:?}...",
|
||||||
|
func.name, retries, health_max_retries, health_period,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
_ if fallible => {
|
||||||
|
log::warn!("Health probe for '{}' exhausted (fallible), skipping", func.name);
|
||||||
|
break; // exhausted but fallible → skip health check
|
||||||
|
}
|
||||||
|
Ok(Err(e)) => return Err(e),
|
||||||
|
Err(_) => {
|
||||||
|
return Err(BakeError::HealthProbeFailed {
|
||||||
|
command: health_cmd.to_string(),
|
||||||
|
error: "timeout".to_string(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
tokio::time::sleep(health_period).await;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Cleanup is handled by finalize.
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Single health-probe attempt. Exit 0 = healthy.
|
||||||
|
async fn health_probe_one(
|
||||||
|
engine: &Engine,
|
||||||
|
cmd: &str,
|
||||||
|
remaining: &str,
|
||||||
|
ctx: &ExecutionContext,
|
||||||
|
event_tx: &EventSender,
|
||||||
|
) -> Result<(), BakeError> {
|
||||||
|
let full_cmd = format!("{}\n{}", remaining, cmd);
|
||||||
|
let result = engine
|
||||||
|
.execute_command(&full_cmd, ctx, event_tx)
|
||||||
|
.await
|
||||||
|
.map_err(|e| BakeError::HealthProbeFailed {
|
||||||
|
command: cmd.to_string(),
|
||||||
|
error: format!("{}", e),
|
||||||
|
})?;
|
||||||
|
if !result.success {
|
||||||
|
return Err(BakeError::HealthProbeFailed {
|
||||||
|
command: cmd.to_string(),
|
||||||
|
error: format!("exit code {}", result.exit_code),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
@@ -174,7 +174,7 @@ pub fn parse_script(text: &str) -> Result<ParsedScript, BakeError> {
|
|||||||
}
|
}
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
eprintln!(
|
log::warn!(
|
||||||
"Warning: Dangling decorator(s) at line {}. Treating as comments.",
|
"Warning: Dangling decorator(s) at line {}. Treating as comments.",
|
||||||
line_number - 1
|
line_number - 1
|
||||||
);
|
);
|
||||||
@@ -224,6 +224,7 @@ pub fn parse_script(text: &str) -> Result<ParsedScript, BakeError> {
|
|||||||
{
|
{
|
||||||
return Err(BakeError::ScriptGenerationFailed {
|
return Err(BakeError::ScriptGenerationFailed {
|
||||||
script: "bake.sh".to_string(),
|
script: "bake.sh".to_string(),
|
||||||
|
action: "parsing function body".to_string(),
|
||||||
reason: format!(
|
reason: format!(
|
||||||
"Unexpected EOF while parsing function '{}'. Unmatched braces (remaining: {})?",
|
"Unexpected EOF while parsing function '{}'. Unmatched braces (remaining: {})?",
|
||||||
name, brace_depth
|
name, brace_depth
|
||||||
@@ -234,6 +235,7 @@ pub fn parse_script(text: &str) -> Result<ParsedScript, BakeError> {
|
|||||||
let last_decorator = pending_decorators.last().unwrap();
|
let last_decorator = pending_decorators.last().unwrap();
|
||||||
return Err(BakeError::ScriptGenerationFailed {
|
return Err(BakeError::ScriptGenerationFailed {
|
||||||
script: "bake.sh".to_string(),
|
script: "bake.sh".to_string(),
|
||||||
|
action: "parsing decorators".to_string(),
|
||||||
reason: format!(
|
reason: format!(
|
||||||
"Dangling decorator {} before EOF. Missing function definition?",
|
"Dangling decorator {} before EOF. Missing function definition?",
|
||||||
last_decorator
|
last_decorator
|
||||||
|
|||||||
+143
-471
@@ -1,42 +1,124 @@
|
|||||||
use super::{decorator::Decorator, parser::Function};
|
// Code in this module PARTIALLY or FULLY utilized AI Coding Agent
|
||||||
|
|
||||||
|
use super::decorator::Decorator;
|
||||||
|
use super::parser::Function;
|
||||||
use crate::bake::error::BakeError;
|
use crate::bake::error::BakeError;
|
||||||
use std::collections::HashMap;
|
use std::collections::HashMap;
|
||||||
use topological_sort::TopologicalSort;
|
use workshop_schedule::{Dag, DagNode, EdgeKind, NodeColor};
|
||||||
|
|
||||||
/// Topologically sorts functions by explicit `@after` dependencies and implicit ordering
|
/// Thin wrapper so `Function` implements `DagNode`.
|
||||||
/// for consecutive `@pipeline` functions (each pipeline function implicitly depends on
|
#[derive(Clone)]
|
||||||
/// the previous one unless it has an explicit `@after`).
|
pub(crate) struct BakeNode {
|
||||||
pub fn sort_function(functions: Vec<Function>) -> Result<Vec<Function>, BakeError> {
|
name: String,
|
||||||
let function_mapped = functions
|
color: NodeColor,
|
||||||
|
fallible: bool,
|
||||||
|
/// Original function body + decorators for execution
|
||||||
|
func: Function,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl DagNode for BakeNode {
|
||||||
|
fn name(&self) -> &str {
|
||||||
|
&self.name
|
||||||
|
}
|
||||||
|
fn color(&self) -> NodeColor {
|
||||||
|
self.color.clone()
|
||||||
|
}
|
||||||
|
fn fallible(&self) -> bool {
|
||||||
|
self.fallible
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Convert a `Function` into a `BakeNode` by extracting scheduling metadata
|
||||||
|
/// from decorators.
|
||||||
|
fn function_to_node(func: &Function) -> BakeNode {
|
||||||
|
let color = if func
|
||||||
|
.decorators
|
||||||
|
.iter()
|
||||||
|
.any(|d| matches!(d, Decorator::Parallel(_)))
|
||||||
|
{
|
||||||
|
let group = func
|
||||||
|
.decorators
|
||||||
|
.iter()
|
||||||
|
.find_map(|d| {
|
||||||
|
if let Decorator::Parallel(Some(g)) = d {
|
||||||
|
Some(g.clone())
|
||||||
|
} else {
|
||||||
|
None
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.unwrap_or("*".to_string());
|
||||||
|
if group == "*" {
|
||||||
|
NodeColor::Wildcard
|
||||||
|
} else {
|
||||||
|
NodeColor::Named(group)
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
NodeColor::Sequential
|
||||||
|
};
|
||||||
|
|
||||||
|
let fallible = func
|
||||||
|
.decorators
|
||||||
|
.iter()
|
||||||
|
.any(|d| matches!(d, Decorator::Fallible));
|
||||||
|
|
||||||
|
BakeNode {
|
||||||
|
name: func.name.clone(),
|
||||||
|
color,
|
||||||
|
fallible,
|
||||||
|
func: func.clone(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Builds a DAG from functions using `@after` dependencies and implicit
|
||||||
|
/// ordering for consecutive `@pipeline` functions.
|
||||||
|
pub fn build_dag(
|
||||||
|
functions: Vec<Function>,
|
||||||
|
) -> Result<(Dag<BakeNode>, HashMap<String, Function>), BakeError> {
|
||||||
|
let function_mapped: HashMap<&str, &Function> = functions
|
||||||
.iter()
|
.iter()
|
||||||
.map(|f| (f.name.as_str(), f))
|
.map(|f| (f.name.as_str(), f))
|
||||||
.collect::<HashMap<&str, &Function>>();
|
.collect();
|
||||||
|
|
||||||
let mut ts = TopologicalSort::<String>::new();
|
let mut dag = Dag::new();
|
||||||
|
|
||||||
// Track last pipeline function for implicit @after
|
|
||||||
let mut last_pipeline: Option<String> = None;
|
let mut last_pipeline: Option<String> = None;
|
||||||
|
|
||||||
for function in &functions {
|
for function in &functions {
|
||||||
ts.insert(function.name.clone());
|
// Determine whether this function is a pipeline node
|
||||||
|
|
||||||
let is_pipeline = function
|
let is_pipeline = function
|
||||||
.decorators
|
.decorators
|
||||||
.iter()
|
.iter()
|
||||||
.any(|d| matches!(d, Decorator::Pipeline));
|
.any(|d| matches!(d, Decorator::Pipeline));
|
||||||
|
|
||||||
|
if !is_pipeline {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
let has_after = function
|
let has_after = function
|
||||||
.decorators
|
.decorators
|
||||||
.iter()
|
.iter()
|
||||||
.any(|d| matches!(d, Decorator::After(_)));
|
.any(|d| matches!(d, Decorator::After(_)));
|
||||||
|
|
||||||
// Add implicit @after for pipeline functions that don't have explicit @after
|
let node = function_to_node(function);
|
||||||
if is_pipeline {
|
dag.add_node(node).map_err(|e| {
|
||||||
if !has_after && let Some(ref last) = last_pipeline {
|
BakeError::IncompatibleDecorators {
|
||||||
ts.add_dependency(last, function.name.clone());
|
function: function.name.clone(),
|
||||||
|
reason: e.to_string(),
|
||||||
}
|
}
|
||||||
last_pipeline = Some(function.name.clone());
|
})?;
|
||||||
}
|
|
||||||
|
|
||||||
|
// Add implicit @after for consecutive @pipeline functions
|
||||||
|
if !has_after {
|
||||||
|
if let Some(ref last) = last_pipeline {
|
||||||
|
dag.add_edge(last, &function.name, EdgeKind::Implicit)
|
||||||
|
.map_err(|e| BakeError::IncompatibleDecorators {
|
||||||
|
function: function.name.clone(),
|
||||||
|
reason: e.to_string(),
|
||||||
|
})?;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
last_pipeline = Some(function.name.clone());
|
||||||
|
|
||||||
|
// Add explicit @after edges
|
||||||
for decorator in &function.decorators {
|
for decorator in &function.decorators {
|
||||||
if let Decorator::After(dependency) = decorator {
|
if let Decorator::After(dependency) = decorator {
|
||||||
if !function_mapped.contains_key(dependency.as_str()) {
|
if !function_mapped.contains_key(dependency.as_str()) {
|
||||||
@@ -50,466 +132,56 @@ pub fn sort_function(functions: Vec<Function>) -> Result<Vec<Function>, BakeErro
|
|||||||
dependency.clone(),
|
dependency.clone(),
|
||||||
));
|
));
|
||||||
}
|
}
|
||||||
ts.add_dependency(dependency, function.name.clone());
|
dag.add_edge(dependency, &function.name, EdgeKind::Explicit)
|
||||||
|
.map_err(|e| BakeError::IncompatibleDecorators {
|
||||||
|
function: function.name.clone(),
|
||||||
|
reason: e.to_string(),
|
||||||
|
})?;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
let mut functions_sorted = Vec::<Function>::new();
|
dag.freeze();
|
||||||
while let Some(name) = ts.pop() {
|
|
||||||
let function = function_mapped[name.as_str()];
|
|
||||||
functions_sorted.push(function.clone());
|
|
||||||
}
|
|
||||||
|
|
||||||
if !ts.is_empty() {
|
let owned_map: HashMap<String, Function> = functions
|
||||||
let remaining: Vec<String> = ts.collect();
|
.into_iter()
|
||||||
return Err(BakeError::CircularDependency(remaining));
|
.map(|f| (f.name.clone(), f))
|
||||||
}
|
.collect();
|
||||||
|
|
||||||
Ok(functions_sorted)
|
Ok((dag, owned_map))
|
||||||
}
|
}
|
||||||
|
|
||||||
#[cfg(test)]
|
/// Group ready parallel nodes by their declared group.
|
||||||
mod tests {
|
/// Returns `(star_nodes, named_groups)`.
|
||||||
use super::*;
|
///
|
||||||
|
/// Deprecated: use `Dag::ready_wildcards()` and `Dag::ready_groups()` instead.
|
||||||
fn create_function(name: &str, decorators: Vec<Decorator>, body: &str) -> Function {
|
pub fn partition_by_group<'a>(
|
||||||
Function {
|
ready: &[String],
|
||||||
name: name.to_string(),
|
dag: &Dag<BakeNode>,
|
||||||
decorators,
|
) -> (Vec<String>, HashMap<String, Vec<String>>) {
|
||||||
body: body.to_string(),
|
let mut star_nodes = Vec::new();
|
||||||
|
let mut named_groups: HashMap<String, Vec<String>> = HashMap::new();
|
||||||
|
for name in ready {
|
||||||
|
if dag.is_wildcard(name) {
|
||||||
|
star_nodes.push(name.clone());
|
||||||
|
} else if let Some(g) = dag.group_of(name) {
|
||||||
|
named_groups.entry(g).or_default().push(name.clone());
|
||||||
}
|
}
|
||||||
|
// Sequential nodes: not included (caller handles them)
|
||||||
}
|
}
|
||||||
|
(star_nodes, named_groups)
|
||||||
#[test]
|
}
|
||||||
fn test_empty_functions() {
|
|
||||||
let functions = vec![];
|
/// Pick a complete group to execute.
|
||||||
let result = sort_function(functions).unwrap();
|
///
|
||||||
assert!(result.is_empty());
|
/// Deprecated: use `Dag::pick_batch()` instead.
|
||||||
}
|
pub fn pick_batch(
|
||||||
|
_star_nodes: &[String],
|
||||||
#[test]
|
_groups: &HashMap<String, Vec<String>>,
|
||||||
fn test_single_function_no_dependencies() {
|
_group_totals: &HashMap<String, usize>,
|
||||||
let function = create_function("func1", vec![], "body1");
|
) -> Option<Vec<String>> {
|
||||||
let functions = vec![function.clone()];
|
// This function is replaced by Dag::pick_batch() which has all the
|
||||||
|
// group metadata internally. Keeping the signature for backward compat
|
||||||
let result = sort_function(functions).unwrap();
|
// but redirect callers to use Dag directly.
|
||||||
|
unimplemented!("use dag.pick_batch() instead")
|
||||||
assert_eq!(result.len(), 1);
|
|
||||||
assert_eq!(result[0].name, "func1");
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_multiple_functions_no_dependencies() {
|
|
||||||
let function1 = create_function("func1", vec![], "body1");
|
|
||||||
let function2 = create_function("func2", vec![], "body2");
|
|
||||||
let function3 = create_function("func3", vec![], "body3");
|
|
||||||
let functions = vec![function1.clone(), function2.clone(), function3.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
|
|
||||||
assert_eq!(result.len(), 3);
|
|
||||||
let names: Vec<String> = result.iter().map(|f| f.name.clone()).collect();
|
|
||||||
assert!(names.contains(&"func1".to_string()));
|
|
||||||
assert!(names.contains(&"func2".to_string()));
|
|
||||||
assert!(names.contains(&"func3".to_string()));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_simple_dependency() {
|
|
||||||
let function1 = create_function("func1", vec![], "body1");
|
|
||||||
let function2 = create_function(
|
|
||||||
"func2",
|
|
||||||
vec![Decorator::After("func1".to_string())],
|
|
||||||
"body2",
|
|
||||||
);
|
|
||||||
let functions = vec![function1.clone(), function2.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
|
|
||||||
assert_eq!(result.len(), 2);
|
|
||||||
assert_eq!(result[0].name, "func1");
|
|
||||||
assert_eq!(result[1].name, "func2");
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_multiple_dependencies() {
|
|
||||||
let function1 = create_function("func1", vec![], "body1");
|
|
||||||
let function2 = create_function(
|
|
||||||
"func2",
|
|
||||||
vec![Decorator::After("func1".to_string())],
|
|
||||||
"body2",
|
|
||||||
);
|
|
||||||
let function3 = create_function(
|
|
||||||
"func3",
|
|
||||||
vec![Decorator::After("func2".to_string())],
|
|
||||||
"body3",
|
|
||||||
);
|
|
||||||
let functions = vec![function1.clone(), function2.clone(), function3.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
|
|
||||||
assert_eq!(result.len(), 3);
|
|
||||||
assert_eq!(result[0].name, "func1");
|
|
||||||
assert_eq!(result[1].name, "func2");
|
|
||||||
assert_eq!(result[2].name, "func3");
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_complex_dependencies() {
|
|
||||||
let function1 = create_function("func1", vec![], "body1");
|
|
||||||
let function2 = create_function(
|
|
||||||
"func2",
|
|
||||||
vec![
|
|
||||||
Decorator::After("func1".to_string()),
|
|
||||||
Decorator::After("func3".to_string()),
|
|
||||||
],
|
|
||||||
"body2",
|
|
||||||
);
|
|
||||||
let function3 = create_function(
|
|
||||||
"func3",
|
|
||||||
vec![Decorator::After("func1".to_string())],
|
|
||||||
"body3",
|
|
||||||
);
|
|
||||||
let function4 = create_function("func4", vec![], "body4");
|
|
||||||
let functions = vec![
|
|
||||||
function1.clone(),
|
|
||||||
function2.clone(),
|
|
||||||
function3.clone(),
|
|
||||||
function4.clone(),
|
|
||||||
];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
|
|
||||||
assert_eq!(result.len(), 4);
|
|
||||||
|
|
||||||
let positions: std::collections::HashMap<String, usize> = result
|
|
||||||
.iter()
|
|
||||||
.enumerate()
|
|
||||||
.map(|(i, f)| (f.name.clone(), i))
|
|
||||||
.collect();
|
|
||||||
|
|
||||||
assert!(positions.get("func1").unwrap() < positions.get("func2").unwrap());
|
|
||||||
assert!(positions.get("func1").unwrap() < positions.get("func3").unwrap());
|
|
||||||
assert!(positions.get("func3").unwrap() < positions.get("func2").unwrap());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_mixed_decorators() {
|
|
||||||
let function1 = create_function(
|
|
||||||
"func1",
|
|
||||||
vec![Decorator::Pipeline, Decorator::Timeout(1000)],
|
|
||||||
"body1",
|
|
||||||
);
|
|
||||||
let function2 = create_function(
|
|
||||||
"func2",
|
|
||||||
vec![
|
|
||||||
Decorator::After("func1".to_string()),
|
|
||||||
Decorator::Retry(3, 100),
|
|
||||||
Decorator::Fallible,
|
|
||||||
],
|
|
||||||
"body2",
|
|
||||||
);
|
|
||||||
let functions = vec![function1.clone(), function2.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
|
|
||||||
assert_eq!(result.len(), 2);
|
|
||||||
assert_eq!(result[0].name, "func1");
|
|
||||||
assert_eq!(result[1].name, "func2");
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_function_with_multiple_after_decorators() {
|
|
||||||
let function1 = create_function("func1", vec![], "body1");
|
|
||||||
let function2 = create_function("func2", vec![], "body2");
|
|
||||||
let function3 = create_function(
|
|
||||||
"func3",
|
|
||||||
vec![
|
|
||||||
Decorator::After("func1".to_string()),
|
|
||||||
Decorator::After("func2".to_string()),
|
|
||||||
],
|
|
||||||
"body3",
|
|
||||||
);
|
|
||||||
let functions = vec![function1.clone(), function2.clone(), function3.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
|
|
||||||
assert_eq!(result.len(), 3);
|
|
||||||
|
|
||||||
let positions: std::collections::HashMap<String, usize> = result
|
|
||||||
.iter()
|
|
||||||
.enumerate()
|
|
||||||
.map(|(i, f)| (f.name.clone(), i))
|
|
||||||
.collect();
|
|
||||||
|
|
||||||
assert!(positions.get("func1").unwrap() < positions.get("func3").unwrap());
|
|
||||||
assert!(positions.get("func2").unwrap() < positions.get("func3").unwrap());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_ignores_non_after_decorators() {
|
|
||||||
let function1 = create_function(
|
|
||||||
"func1",
|
|
||||||
vec![
|
|
||||||
Decorator::Pipeline,
|
|
||||||
Decorator::If("condition".to_string()),
|
|
||||||
Decorator::Fallible,
|
|
||||||
],
|
|
||||||
"body1",
|
|
||||||
);
|
|
||||||
let function2 = create_function(
|
|
||||||
"func2",
|
|
||||||
vec![
|
|
||||||
Decorator::Loop(5),
|
|
||||||
Decorator::Timeout(1000),
|
|
||||||
Decorator::Retry(3, 100),
|
|
||||||
Decorator::Export,
|
|
||||||
],
|
|
||||||
"body2",
|
|
||||||
);
|
|
||||||
let functions = vec![function1.clone(), function2.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
|
|
||||||
assert_eq!(result.len(), 2);
|
|
||||||
let names: Vec<String> = result.iter().map(|f| f.name.clone()).collect();
|
|
||||||
assert!(names.contains(&"func1".to_string()));
|
|
||||||
assert!(names.contains(&"func2".to_string()));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_implicit_after_pipeline() {
|
|
||||||
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
|
|
||||||
let func2 = create_function("func2", vec![Decorator::Pipeline], "body2");
|
|
||||||
let func3 = create_function("func3", vec![Decorator::Pipeline], "body3");
|
|
||||||
let functions = vec![func1.clone(), func2.clone(), func3.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
|
|
||||||
assert_eq!(result.len(), 3);
|
|
||||||
assert_eq!(result[0].name, "func1");
|
|
||||||
assert_eq!(result[1].name, "func2");
|
|
||||||
assert_eq!(result[2].name, "func3");
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_first_pipeline_no_implicit_after() {
|
|
||||||
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
|
|
||||||
let functions = vec![func1.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
|
|
||||||
assert_eq!(result.len(), 1);
|
|
||||||
assert_eq!(result[0].name, "func1");
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_explicit_after_overrides_implicit() {
|
|
||||||
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
|
|
||||||
let func2 = create_function(
|
|
||||||
"func2",
|
|
||||||
vec![Decorator::Pipeline, Decorator::After("func1".to_string())],
|
|
||||||
"body2",
|
|
||||||
);
|
|
||||||
let func3 = create_function("func3", vec![Decorator::Pipeline], "body3");
|
|
||||||
let functions = vec![func1.clone(), func2.clone(), func3.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
|
|
||||||
assert_eq!(result.len(), 3);
|
|
||||||
let positions: std::collections::HashMap<String, usize> = result
|
|
||||||
.iter()
|
|
||||||
.enumerate()
|
|
||||||
.map(|(i, f)| (f.name.clone(), i))
|
|
||||||
.collect();
|
|
||||||
|
|
||||||
assert!(positions.get("func1").unwrap() < positions.get("func2").unwrap());
|
|
||||||
assert!(positions.get("func2").unwrap() < positions.get("func3").unwrap());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_mixed_pipeline_and_non_pipeline() {
|
|
||||||
let func1 = create_function("func1", vec![], "body1");
|
|
||||||
let func2 = create_function("func2", vec![Decorator::Pipeline], "body2");
|
|
||||||
let func3 = create_function("func3", vec![], "body3");
|
|
||||||
let func4 = create_function("func4", vec![Decorator::Pipeline], "body4");
|
|
||||||
let functions = vec![func1.clone(), func2.clone(), func3.clone(), func4.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
|
|
||||||
assert_eq!(result.len(), 4);
|
|
||||||
let positions: std::collections::HashMap<String, usize> = result
|
|
||||||
.iter()
|
|
||||||
.enumerate()
|
|
||||||
.map(|(i, f)| (f.name.clone(), i))
|
|
||||||
.collect();
|
|
||||||
|
|
||||||
assert!(positions.get("func2").unwrap() < positions.get("func4").unwrap());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_single_pipeline_works() {
|
|
||||||
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
|
|
||||||
let functions = vec![func1.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
assert_eq!(result.len(), 1);
|
|
||||||
assert_eq!(result[0].name, "func1");
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_two_pipelines_implicit_chain() {
|
|
||||||
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
|
|
||||||
let func2 = create_function("func2", vec![Decorator::Pipeline], "body2");
|
|
||||||
let functions = vec![func1.clone(), func2.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
assert_eq!(result.len(), 2);
|
|
||||||
let positions: std::collections::HashMap<String, usize> = result
|
|
||||||
.iter()
|
|
||||||
.enumerate()
|
|
||||||
.map(|(i, f)| (f.name.clone(), i))
|
|
||||||
.collect();
|
|
||||||
assert!(positions.get("func1").unwrap() < positions.get("func2").unwrap());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_pipeline_explicit_after_skips_implicit() {
|
|
||||||
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
|
|
||||||
let func2 = create_function("func2", vec![Decorator::Pipeline], "body2");
|
|
||||||
let func3 = create_function(
|
|
||||||
"func3",
|
|
||||||
vec![Decorator::Pipeline, Decorator::After("func1".to_string())],
|
|
||||||
"body3",
|
|
||||||
);
|
|
||||||
let functions = vec![func1.clone(), func2.clone(), func3.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
assert_eq!(result.len(), 3);
|
|
||||||
let positions: std::collections::HashMap<String, usize> = result
|
|
||||||
.iter()
|
|
||||||
.enumerate()
|
|
||||||
.map(|(i, f)| (f.name.clone(), i))
|
|
||||||
.collect();
|
|
||||||
assert!(positions.get("func1").unwrap() < positions.get("func3").unwrap());
|
|
||||||
// TODO: Our design spec indicates order-based sorting, however toposort crate does not guarantee order of independent nodes. We should consider whether we want to enforce order-based sorting for pipelines, or if we are okay with any valid topological order.
|
|
||||||
// assert!(positions.get("func2").unwrap() < positions.get("func3").unwrap());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_circular_dependency_detected() {
|
|
||||||
let func1 = create_function(
|
|
||||||
"func1",
|
|
||||||
vec![Decorator::After("func2".to_string())],
|
|
||||||
"body1",
|
|
||||||
);
|
|
||||||
let func2 = create_function(
|
|
||||||
"func2",
|
|
||||||
vec![Decorator::After("func1".to_string())],
|
|
||||||
"body2",
|
|
||||||
);
|
|
||||||
let functions = vec![func1.clone(), func2.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions);
|
|
||||||
assert!(result.is_err());
|
|
||||||
assert!(matches!(
|
|
||||||
result.unwrap_err(),
|
|
||||||
BakeError::CircularDependency(_)
|
|
||||||
));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_unknown_dependency_error() {
|
|
||||||
let func1 = create_function(
|
|
||||||
"func1",
|
|
||||||
vec![Decorator::After("nonexistent".to_string())],
|
|
||||||
"body1",
|
|
||||||
);
|
|
||||||
let functions = vec![func1.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions);
|
|
||||||
assert!(result.is_err());
|
|
||||||
assert!(matches!(
|
|
||||||
result.unwrap_err(),
|
|
||||||
BakeError::UnknownDependency(_, _)
|
|
||||||
));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_pipeline_with_multiple_decorators() {
|
|
||||||
let func1 = create_function(
|
|
||||||
"func1",
|
|
||||||
vec![
|
|
||||||
Decorator::Pipeline,
|
|
||||||
Decorator::Timeout(60),
|
|
||||||
Decorator::Fallible,
|
|
||||||
],
|
|
||||||
"body1",
|
|
||||||
);
|
|
||||||
let func2 = create_function(
|
|
||||||
"func2",
|
|
||||||
vec![
|
|
||||||
Decorator::Pipeline,
|
|
||||||
Decorator::Retry(3, 5),
|
|
||||||
Decorator::After("func1".to_string()),
|
|
||||||
],
|
|
||||||
"body2",
|
|
||||||
);
|
|
||||||
let functions = vec![func1.clone(), func2.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
assert_eq!(result.len(), 2);
|
|
||||||
let positions: std::collections::HashMap<String, usize> = result
|
|
||||||
.iter()
|
|
||||||
.enumerate()
|
|
||||||
.map(|(i, f)| (f.name.clone(), i))
|
|
||||||
.collect();
|
|
||||||
assert!(positions.get("func1").unwrap() < positions.get("func2").unwrap());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_long_pipeline_chain() {
|
|
||||||
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
|
|
||||||
let func2 = create_function("func2", vec![Decorator::Pipeline], "body2");
|
|
||||||
let func3 = create_function("func3", vec![Decorator::Pipeline], "body3");
|
|
||||||
let func4 = create_function("func4", vec![Decorator::Pipeline], "body4");
|
|
||||||
let func5 = create_function("func5", vec![Decorator::Pipeline], "body5");
|
|
||||||
let functions = vec![
|
|
||||||
func1.clone(),
|
|
||||||
func2.clone(),
|
|
||||||
func3.clone(),
|
|
||||||
func4.clone(),
|
|
||||||
func5.clone(),
|
|
||||||
];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
assert_eq!(result.len(), 5);
|
|
||||||
let positions: std::collections::HashMap<String, usize> = result
|
|
||||||
.iter()
|
|
||||||
.enumerate()
|
|
||||||
.map(|(i, f)| (f.name.clone(), i))
|
|
||||||
.collect();
|
|
||||||
assert!(positions.get("func1").unwrap() < positions.get("func2").unwrap());
|
|
||||||
assert!(positions.get("func2").unwrap() < positions.get("func3").unwrap());
|
|
||||||
assert!(positions.get("func3").unwrap() < positions.get("func4").unwrap());
|
|
||||||
assert!(positions.get("func4").unwrap() < positions.get("func5").unwrap());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_non_pipeline_between_pipelines() {
|
|
||||||
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
|
|
||||||
let func2 = create_function("func2", vec![], "body2");
|
|
||||||
let func3 = create_function("func3", vec![Decorator::Pipeline], "body3");
|
|
||||||
let functions = vec![func1.clone(), func2.clone(), func3.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
assert_eq!(result.len(), 3);
|
|
||||||
let positions: std::collections::HashMap<String, usize> = result
|
|
||||||
.iter()
|
|
||||||
.enumerate()
|
|
||||||
.map(|(i, f)| (f.name.clone(), i))
|
|
||||||
.collect();
|
|
||||||
assert!(positions.get("func1").unwrap() < positions.get("func3").unwrap());
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,31 @@
|
|||||||
|
use crate::bake::builder::{build_script, RenderMode};
|
||||||
|
use crate::bake::error::BakeError;
|
||||||
|
use crate::bake::parser::Function;
|
||||||
|
use std::path::Path;
|
||||||
|
use workshop_engine::{Engine, EventSender, ExecutionContext};
|
||||||
|
|
||||||
|
/// Executes `script_content` as a single function in trivial mode.
|
||||||
|
///
|
||||||
|
/// Two entry paths lead here:
|
||||||
|
/// - **Auto-detection** (`!has_pipeline`): bake.sh has no `@pipeline` functions,
|
||||||
|
/// the entire content is treated as one trivial function.
|
||||||
|
/// - **Explicit PIP** (future): `trivial` parameter forces trivial mode even
|
||||||
|
/// when `@pipeline` functions exist, typically with `RenderMode::Off`.
|
||||||
|
pub async fn run_trivial(
|
||||||
|
script_content: &str,
|
||||||
|
bake_base_path: &Path,
|
||||||
|
workspace: &Path,
|
||||||
|
engine: &Engine,
|
||||||
|
ctx: &mut ExecutionContext,
|
||||||
|
event_tx: &EventSender,
|
||||||
|
render_mode: RenderMode,
|
||||||
|
) -> Result<(), BakeError> {
|
||||||
|
let function = Function {
|
||||||
|
name: String::new(),
|
||||||
|
decorators: vec![],
|
||||||
|
body: script_content.to_string(),
|
||||||
|
};
|
||||||
|
let script = build_script(&function, bake_base_path, workspace, render_mode, ctx.debug_flags)?;
|
||||||
|
engine.execute_script(&script, ctx, event_tx).await?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
@@ -0,0 +1,61 @@
|
|||||||
|
use crate::bake::decorator::Decorator;
|
||||||
|
use crate::bake::parser::Function;
|
||||||
|
use std::collections::HashMap;
|
||||||
|
|
||||||
|
/// Execution mode for a pipeline function.
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||||
|
pub enum FuncMode {
|
||||||
|
Normal,
|
||||||
|
Daemon,
|
||||||
|
Async,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Determine the execution mode from decorators.
|
||||||
|
/// `@daemon`, `@async`, `@pipe` are mutually exclusive modes.
|
||||||
|
pub fn resolve_mode(func: &Function) -> FuncMode {
|
||||||
|
for d in &func.decorators {
|
||||||
|
match d {
|
||||||
|
Decorator::Daemon => return FuncMode::Daemon,
|
||||||
|
Decorator::Async => return FuncMode::Async,
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
FuncMode::Normal
|
||||||
|
}
|
||||||
|
|
||||||
|
/// True if the function is decorated with `@parallel`.
|
||||||
|
pub fn is_parallel(func: &Function) -> bool {
|
||||||
|
func.decorators
|
||||||
|
.iter()
|
||||||
|
.any(|d| matches!(d, Decorator::Parallel(_)))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// True if the function is decorated with `@fallible`.
|
||||||
|
pub fn is_fallible(func: &Function) -> bool {
|
||||||
|
func.decorators
|
||||||
|
.iter()
|
||||||
|
.any(|d| matches!(d, Decorator::Fallible))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Resolve stdin for a `@pipe`-decorated function from the stdout cache.
|
||||||
|
///
|
||||||
|
/// Rule 1: self-references are skipped (handled by loop combinator).
|
||||||
|
/// Rule 3: sources present in the cache are concatenated.
|
||||||
|
/// Returns `None` if any source is missing (caller decides based on `@fallible`).
|
||||||
|
pub fn resolve_pipe_stdin(
|
||||||
|
func: &Function,
|
||||||
|
cache: &HashMap<String, String>,
|
||||||
|
) -> Option<Vec<u8>> {
|
||||||
|
let sources = func.decorators.iter().find_map(|d| match d {
|
||||||
|
Decorator::Pipe(names) => Some(names.as_slice()),
|
||||||
|
_ => None,
|
||||||
|
})?;
|
||||||
|
let mut combined = String::new();
|
||||||
|
for source in sources {
|
||||||
|
if *source == func.name {
|
||||||
|
continue; // Rule 1: skip self
|
||||||
|
}
|
||||||
|
combined.push_str(cache.get(source)?); // Rule 3
|
||||||
|
}
|
||||||
|
Some(combined.into_bytes())
|
||||||
|
}
|
||||||
@@ -1,15 +1,8 @@
|
|||||||
use std::path::PathBuf;
|
|
||||||
|
|
||||||
use crate::cli::{BareCommands, Cli};
|
use crate::cli::{BareCommands, Cli};
|
||||||
use crate::error::CliError;
|
use crate::error::CliError;
|
||||||
use crate::types::resource::ResourceRegistry;
|
use crate::types::resource::ResourceRegistry;
|
||||||
use crate::{bake::bake, finalize::finalize, prebake::prebake};
|
use crate::{bake::bake, finalize::finalize, prebake::prebake};
|
||||||
use workshop_engine::{EventSender, ExecutionContext};
|
use workshop_engine::{EventSender, ExecutionContext};
|
||||||
fn or_workshop(path: &Option<PathBuf>, name: &str) -> PathBuf {
|
|
||||||
path
|
|
||||||
.clone()
|
|
||||||
.unwrap_or_else(|| PathBuf::from(format!(".workshop/{}", name)))
|
|
||||||
}
|
|
||||||
|
|
||||||
pub async fn run_bare(
|
pub async fn run_bare(
|
||||||
cli: &Cli,
|
cli: &Cli,
|
||||||
@@ -21,15 +14,16 @@ pub async fn run_bare(
|
|||||||
registry: &ResourceRegistry,
|
registry: &ResourceRegistry,
|
||||||
) -> Result<(), CliError> {
|
) -> Result<(), CliError> {
|
||||||
let (stage, config) = match command {
|
let (stage, config) = match command {
|
||||||
BareCommands::Prebake { .. } => ("prebake", or_workshop(&cli.prebake, "prebake.yml")),
|
BareCommands::Prebake { .. } => ("prebake", &cli.prebake),
|
||||||
BareCommands::Bake { .. } => ("bake", or_workshop(&cli.bake, "bake.sh")),
|
BareCommands::Bake { .. } => ("bake", &cli.bake),
|
||||||
BareCommands::Finalize { .. } => {
|
BareCommands::Finalize { .. } => {
|
||||||
("finalize", or_workshop(&cli.finalize, "finalize.yml"))
|
("finalize", &cli.finalize)
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
log::debug!("Running bare command {} with config {}", stage, config.display());
|
||||||
|
|
||||||
let prebake_path = or_workshop(&cli.prebake, "prebake.yml");
|
let prebake_path = &cli.prebake;
|
||||||
let bake_base = or_workshop(&cli.bake_base, "bake_base.sh");
|
let bake_base = &cli.bake_base;
|
||||||
|
|
||||||
ctx.task_id = format!("{}-{}-{}", pipeline, build_id, stage);
|
ctx.task_id = format!("{}-{}-{}", pipeline, build_id, stage);
|
||||||
ctx.pipeline_name = pipeline.to_string();
|
ctx.pipeline_name = pipeline.to_string();
|
||||||
@@ -40,7 +34,7 @@ pub async fn run_bare(
|
|||||||
prebake(&config, cli, None, ctx, event_tx, None).await?;
|
prebake(&config, cli, None, ctx, event_tx, None).await?;
|
||||||
}
|
}
|
||||||
BareCommands::Bake { .. } => {
|
BareCommands::Bake { .. } => {
|
||||||
bake(&config, &bake_base, &prebake_path, cli, ctx, event_tx).await?;
|
bake(&config, &bake_base, &prebake_path, cli, ctx, event_tx, None).await?;
|
||||||
}
|
}
|
||||||
BareCommands::Finalize { .. } => {
|
BareCommands::Finalize { .. } => {
|
||||||
finalize(&config, cli, ctx, event_tx, registry).await?;
|
finalize(&config, cli, ctx, event_tx, registry).await?;
|
||||||
|
|||||||
Vendored
@@ -2,6 +2,8 @@ pub mod config;
|
|||||||
pub mod constant;
|
pub mod constant;
|
||||||
pub mod error;
|
pub mod error;
|
||||||
pub mod event;
|
pub mod event;
|
||||||
|
pub mod cleanup;
|
||||||
|
pub mod artifact;
|
||||||
pub mod plugin;
|
pub mod plugin;
|
||||||
pub mod stage;
|
pub mod stage;
|
||||||
pub mod template;
|
pub mod template;
|
||||||
@@ -52,9 +54,17 @@ pub async fn finalize(
|
|||||||
// Determine if we should continue with artifacts/latehook
|
// Determine if we should continue with artifacts/latehook
|
||||||
// If prior stages failed, we still send notification but skip artifacts
|
// If prior stages failed, we still send notification but skip artifacts
|
||||||
let _prior_failed = earlyhook_result.is_err();
|
let _prior_failed = earlyhook_result.is_err();
|
||||||
|
// Artifact collection and publishing
|
||||||
// Prepare artifacts
|
if !_prior_failed {
|
||||||
// TODO: artifact stage implementation
|
log::info!("Processing {} artifacts", finalize.artifact.len());
|
||||||
|
artifact::collect_and_publish(
|
||||||
|
&finalize.artifact,
|
||||||
|
®istered_plugins,
|
||||||
|
ctx,
|
||||||
|
event_tx.clone(),
|
||||||
|
)
|
||||||
|
.await?;
|
||||||
|
}
|
||||||
|
|
||||||
// Late hook
|
// Late hook
|
||||||
stage::hook::hook(
|
stage::hook::hook(
|
||||||
|
|||||||
@@ -1,52 +0,0 @@
|
|||||||
# workshop-baker/src/finalize
|
|
||||||
|
|
||||||
**Generated:** 2026-05-19
|
|
||||||
**Commit:** 28abc5d
|
|
||||||
|
|
||||||
Artifact packaging, plugin-based notifications, and post-build hooks.
|
|
||||||
|
|
||||||
## STRUCTURE
|
|
||||||
```
|
|
||||||
finalize/
|
|
||||||
├── config.rs # FinalizeConfig YAML (379 lines, plugins, notifications)
|
|
||||||
├── stage.rs # FinalizeStage enum
|
|
||||||
├── stage/hook.rs # Post-build hook execution
|
|
||||||
├── event.rs # Event propagation
|
|
||||||
├── template.rs # Variable substitution {{VARIABLE}}
|
|
||||||
├── plugin.rs # Plugin trait + registration
|
|
||||||
├── plugin/
|
|
||||||
│ ├── metadata.rs # Plugin manifest parsing
|
|
||||||
│ ├── dylib.rs # Dynamic library plugins
|
|
||||||
│ ├── rhai.rs # Rhai scripting (todo!())
|
|
||||||
│ ├── shell.rs # Shell command plugins
|
|
||||||
│ └── internal.rs + internal/ # Built-in plugins (mail, webhook, satori, insitenotify)
|
|
||||||
├── types.rs
|
|
||||||
├── types/compression.rs
|
|
||||||
├── error.rs # FinalizeError
|
|
||||||
└── constant.rs
|
|
||||||
```
|
|
||||||
|
|
||||||
## WHERE TO LOOK
|
|
||||||
| Task | Location |
|
|
||||||
|------|----------|
|
|
||||||
| Config loading | `config.rs:17` - `FinalizeConfig` + `parse()` |
|
|
||||||
| Plugin registration | `plugin.rs` - `register()` |
|
|
||||||
| Email notifications | `plugin/internal/mail.rs` - SMTP |
|
|
||||||
| Webhooks | `plugin/internal/webhook.rs` |
|
|
||||||
| Satori integration | `plugin/internal/satori.rs` |
|
|
||||||
| Shell plugins | `plugin/shell.rs` |
|
|
||||||
| Template rendering | `template.rs` - `{{VARIABLE}}` handlebars-style |
|
|
||||||
|
|
||||||
## CONVENTIONS
|
|
||||||
- **Plugin System**: `FinalizePlugin` trait + registry with both internal and dynamic plugins
|
|
||||||
- **Template Syntax**: `{{VARIABLE}}` handlebars-style substitution via `template.rs`
|
|
||||||
- **Stages**: EarlyHook → (Artifacts) → LateHook → Notifications
|
|
||||||
- **Notification**: `notify.rs` in parent module drives plugin-based notification dispatch
|
|
||||||
- **ResourceRegistry**: Shared resource cache passed through to plugins
|
|
||||||
|
|
||||||
## ANTI-PATTERNS
|
|
||||||
1. **fetch/ subdirectory removed** — was `plugin/fetch/{git,http,extract,checksum}`, no longer exists
|
|
||||||
2. **SHA1 deprecated** — workshop-getterurl handles checksums now
|
|
||||||
3. **Unimplemented** — `rhai.rs:17` - "todo!()" for Rhai scripting
|
|
||||||
4. **TODO markers** — Multiple TODOs in config.rs (lines 305-306)
|
|
||||||
5. **Artifact stage not implemented** — `finalize.rs:57` — "TODO: artifact stage implementation"
|
|
||||||
@@ -0,0 +1,139 @@
|
|||||||
|
//! Artifact collection and publishing.
|
||||||
|
//!
|
||||||
|
//! Collects build artifacts (via glob patterns), optionally compresses them,
|
||||||
|
//! and dispatches each artifact through its configured publish plugins.
|
||||||
|
//! The publish plugin receives the artifact path as a system-injected
|
||||||
|
//! `artifact` field in its argument.
|
||||||
|
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
use crate::finalize::config::ArtifactDef;
|
||||||
|
use crate::finalize::error::FinalizeError;
|
||||||
|
use crate::finalize::plugin::PluginMap;
|
||||||
|
use crate::finalize::types::compression::CompressionMethod;
|
||||||
|
use workshop_engine::{EventSender, ExecutionContext};
|
||||||
|
|
||||||
|
/// Collect artifacts and execute their publish plugins.
|
||||||
|
///
|
||||||
|
/// For each artifact:
|
||||||
|
/// 1. Resolve source files via glob
|
||||||
|
/// 2. Optionally compress (zstd, gzip, dir)
|
||||||
|
/// 3. Inject `artifact` key into each publish plugin's config
|
||||||
|
/// 4. Call the plugin
|
||||||
|
pub async fn collect_and_publish(
|
||||||
|
artifacts: &[ArtifactDef],
|
||||||
|
plugins: &PluginMap,
|
||||||
|
ctx: &ExecutionContext,
|
||||||
|
event_tx: EventSender,
|
||||||
|
) -> Result<(), FinalizeError> {
|
||||||
|
if artifacts.is_empty() {
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
for artifact in artifacts {
|
||||||
|
let artifact_path = process_artifact(artifact).await?;
|
||||||
|
|
||||||
|
if let Some(ref path) = artifact_path {
|
||||||
|
log::info!("Artifact processed: {} -> {}", artifact.path, path.display());
|
||||||
|
}
|
||||||
|
|
||||||
|
for (plugin_name, publish_config) in &artifact.publish {
|
||||||
|
let plugin = plugins.get(plugin_name).ok_or_else(|| {
|
||||||
|
FinalizeError::PluginExecutionFailed(format!(
|
||||||
|
"publish plugin '{}' not found for artifact '{}'",
|
||||||
|
plugin_name, artifact.path,
|
||||||
|
))
|
||||||
|
})?;
|
||||||
|
|
||||||
|
// Inject artifact path into the plugin argument
|
||||||
|
let argument = if let Some(ref path) = artifact_path {
|
||||||
|
let mut arg = publish_config.clone();
|
||||||
|
if let Some(map) = arg.as_mapping_mut() {
|
||||||
|
map.insert(
|
||||||
|
serde_yaml::Value::String("artifact".into()),
|
||||||
|
serde_yaml::Value::String(path.to_string_lossy().into()),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
arg
|
||||||
|
} else {
|
||||||
|
publish_config.clone()
|
||||||
|
};
|
||||||
|
|
||||||
|
log::info!(
|
||||||
|
"Publishing artifact '{}' via plugin '{}'",
|
||||||
|
artifact.path, plugin_name,
|
||||||
|
);
|
||||||
|
|
||||||
|
plugin.call(argument, ctx, &event_tx).await?;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Resolve and optionally compress an artifact.
|
||||||
|
/// Returns the local path to the processed artifact, or `None` if non-file artifact.
|
||||||
|
async fn process_artifact(artifact: &ArtifactDef) -> Result<Option<PathBuf>, FinalizeError> {
|
||||||
|
if artifact.path.is_empty() {
|
||||||
|
return Ok(None); // non-filesystem artifact (e.g. Docker image)
|
||||||
|
}
|
||||||
|
|
||||||
|
let src = PathBuf::from(&artifact.path);
|
||||||
|
if !src.exists() {
|
||||||
|
if artifact.path.contains('*') || artifact.path.contains('?') {
|
||||||
|
return Err(FinalizeError::PluginExecutionFailed(
|
||||||
|
format!("glob patterns not supported: '{}'; use a directory path", artifact.path),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
return Err(FinalizeError::ArtifactNotFound(src));
|
||||||
|
}
|
||||||
|
|
||||||
|
match artifact.compression {
|
||||||
|
CompressionMethod::None => Ok(Some(src)),
|
||||||
|
CompressionMethod::Dir => Ok(Some(src)),
|
||||||
|
CompressionMethod::Gzip => {
|
||||||
|
let dst = std::env::temp_dir().join(format!("artifact-{}.tar.gz", artifact.id.as_deref().unwrap_or("unknown")));
|
||||||
|
compress_tar_zstd_or_gzip(&src, &dst, false).await?;
|
||||||
|
Ok(Some(dst))
|
||||||
|
}
|
||||||
|
CompressionMethod::Zstd => {
|
||||||
|
let dst = std::env::temp_dir().join(format!("artifact-{}.tar.zst", artifact.id.as_deref().unwrap_or("unknown")));
|
||||||
|
compress_tar_zstd_or_gzip(&src, &dst, true).await?;
|
||||||
|
Ok(Some(dst))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn compress_tar_zstd_or_gzip(
|
||||||
|
src: &Path,
|
||||||
|
dst: &Path,
|
||||||
|
zstd: bool,
|
||||||
|
) -> Result<(), FinalizeError> {
|
||||||
|
let _archive = tokio::task::spawn_blocking({
|
||||||
|
let src = src.to_path_buf();
|
||||||
|
let dst = dst.to_path_buf();
|
||||||
|
move || -> Result<(), String> {
|
||||||
|
let file = std::fs::File::create(&dst)
|
||||||
|
.map_err(|e| format!("cannot create {}: {}", dst.display(), e))?;
|
||||||
|
let writer: Box<dyn std::io::Write> = if zstd {
|
||||||
|
Box::new(zstd::stream::write::Encoder::new(file, 0).map_err(|e| e.to_string())?)
|
||||||
|
} else {
|
||||||
|
Box::new(flate2::write::GzEncoder::new(file, flate2::Compression::default()))
|
||||||
|
};
|
||||||
|
let mut tar = tar::Builder::new(writer);
|
||||||
|
if src.is_dir() {
|
||||||
|
tar.append_dir_all(".", &src)
|
||||||
|
.map_err(|e| format!("tar error: {}", e))?;
|
||||||
|
} else {
|
||||||
|
tar.append_path(&src)
|
||||||
|
.map_err(|e| format!("tar error: {}", e))?;
|
||||||
|
}
|
||||||
|
tar.into_inner().map_err(|e| e.to_string())?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.map_err(|e| FinalizeError::PluginExecutionFailed(format!("spawn_blocking: {}", e)))?
|
||||||
|
.map_err(FinalizeError::PluginExecutionFailed)?;
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
@@ -0,0 +1,67 @@
|
|||||||
|
//! Pipeline cleanup and resource reclamation.
|
||||||
|
//!
|
||||||
|
//! Runs after all pipeline stages and notification dispatch complete.
|
||||||
|
//! Checks the configured [`CleanupPolicy`] and performs the following:
|
||||||
|
//!
|
||||||
|
//! 1. Temp file / cache directory removal
|
||||||
|
//! 2. Policy-based skip (never, manual, on_success)
|
||||||
|
|
||||||
|
use crate::finalize::config::{CleanupConfig, CleanupPolicy};
|
||||||
|
use crate::finalize::error::FinalizeError;
|
||||||
|
use workshop_engine::ExecutionContext;
|
||||||
|
|
||||||
|
/// Execute cleanup according to the pipeline's [`CleanupPolicy`].
|
||||||
|
///
|
||||||
|
/// Returns `Ok(())` when cleanup completes (or is skipped per policy).
|
||||||
|
/// Returns `Err` only if a cleanup action itself fails (not if the
|
||||||
|
/// pipeline failed; pipeline outcome is checked via `pipeline_ok`).
|
||||||
|
pub fn cleanup(
|
||||||
|
config: &CleanupConfig,
|
||||||
|
_ctx: &ExecutionContext,
|
||||||
|
pipeline_ok: bool,
|
||||||
|
) -> Result<(), FinalizeError> {
|
||||||
|
match config.policy {
|
||||||
|
CleanupPolicy::Never => {
|
||||||
|
log::info!("[cleanup] policy=never, skipping all cleanup");
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
CleanupPolicy::Manual => {
|
||||||
|
log::info!("[cleanup] policy=manual, hanging for bakerd signal");
|
||||||
|
// TODO(daemon): hang on bakerd signal
|
||||||
|
// For now (standalone/bare) manual == Never
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
CleanupPolicy::OnSuccess if !pipeline_ok => {
|
||||||
|
log::info!("[cleanup] policy=on_success, but pipeline failed — skipping");
|
||||||
|
// TODO(daemon): hang for bakerd
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
log::info!("[cleanup] starting");
|
||||||
|
cleanup_temp_files();
|
||||||
|
subprocess_reclaim();
|
||||||
|
log::info!("[cleanup] complete");
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Reclaim child processes (placeholder for daemon-mode PID tracking).
|
||||||
|
/// In standalone mode, children are reaped by the kernel on process exit.
|
||||||
|
fn subprocess_reclaim() {
|
||||||
|
// TODO(daemon): track spawned PIDs during pipeline and signal them here.
|
||||||
|
}
|
||||||
|
/// Remove plugin cache and temporary directories.
|
||||||
|
fn cleanup_temp_files() {
|
||||||
|
let cache = std::path::Path::new("/tmp/baker");
|
||||||
|
if cache.exists() {
|
||||||
|
match std::fs::remove_dir_all(cache) {
|
||||||
|
Ok(()) => log::debug!("[cleanup] removed cache: {}", cache.display()),
|
||||||
|
Err(e) => log::warn!("[cleanup] failed to remove {}: {}", cache.display(), e),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let workshop_templates = std::env::temp_dir().join("workshop-templates");
|
||||||
|
if workshop_templates.exists() {
|
||||||
|
let _ = std::fs::remove_dir_all(&workshop_templates);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -185,15 +185,21 @@ impl<'de> serde::Deserialize<'de> for NotificationConfig {
|
|||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
templates: HashMap<String, NotificationTemplateDef>,
|
templates: HashMap<String, NotificationTemplateDef>,
|
||||||
#[serde(flatten)]
|
#[serde(flatten)]
|
||||||
groups_raw: HashMap<String, HashMap<String, NotificationMethod>>,
|
groups_raw: HashMap<String, Option<HashMap<String, NotificationMethod>>>,
|
||||||
}
|
}
|
||||||
let h = Helper::deserialize(deserializer)?;
|
let h = Helper::deserialize(deserializer)?;
|
||||||
let groups: HashMap<u32, _> = h.groups_raw.into_iter()
|
let groups: HashMap<u32, _> = h.groups_raw.into_iter()
|
||||||
.map(|(k, v)| {
|
.filter_map(|(k, v)| {
|
||||||
k.parse::<u32>().map(|n| (n, v))
|
let parsed = k.parse::<u32>().map(|n| (n, v.unwrap_or_default()));
|
||||||
.map_err(|_| serde::de::Error::custom(format!("priority key must be integer, got {:?}", k)))
|
match parsed {
|
||||||
|
Ok(pair) => Some(pair),
|
||||||
|
Err(e) => {
|
||||||
|
log::warn!("priority key must be integer, got {:?}: {}", k, e);
|
||||||
|
None
|
||||||
|
}
|
||||||
|
}
|
||||||
})
|
})
|
||||||
.collect::<Result<_, _>>()?;
|
.collect();
|
||||||
Ok(NotificationConfig { templates: h.templates, groups })
|
Ok(NotificationConfig { templates: h.templates, groups })
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -336,8 +342,9 @@ pub struct ArtifactDef {
|
|||||||
/// Compression method: "zstd", "gzip", "none", "dir"
|
/// Compression method: "zstd", "gzip", "none", "dir"
|
||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
pub compression: CompressionMethod,
|
pub compression: CompressionMethod,
|
||||||
// Publishing targets
|
/// Publish targets: plugin name → configuration (artifact path injected by system)
|
||||||
// TODO: under heavy refactor
|
#[serde(default)]
|
||||||
|
pub publish: HashMap<String, serde_yaml::Value>
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Default for ArtifactDef {
|
impl Default for ArtifactDef {
|
||||||
@@ -347,6 +354,7 @@ impl Default for ArtifactDef {
|
|||||||
path: String::new(),
|
path: String::new(),
|
||||||
retention_ms: default_retention_ms(),
|
retention_ms: default_retention_ms(),
|
||||||
compression: CompressionMethod::default(),
|
compression: CompressionMethod::default(),
|
||||||
|
publish: HashMap::new(),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,4 +1,5 @@
|
|||||||
use workshop_engine::{ExecutionEvent, StreamType};
|
use workshop_engine::{ExecutionEvent, StreamType};
|
||||||
|
use crate::types::debug::DebugFeature;
|
||||||
|
|
||||||
/// Receives and processes execution events for the finalize stage.
|
/// Receives and processes execution events for the finalize stage.
|
||||||
/// Logs task lifecycle events (started, completed, failed) and streaming output (stdout/stderr)
|
/// Logs task lifecycle events (started, completed, failed) and streaming output (stdout/stderr)
|
||||||
@@ -7,43 +8,62 @@ pub async fn event_receiver(
|
|||||||
mut rx: tokio::sync::mpsc::Receiver<ExecutionEvent>,
|
mut rx: tokio::sync::mpsc::Receiver<ExecutionEvent>,
|
||||||
pipeline_name: String,
|
pipeline_name: String,
|
||||||
build_id: String,
|
build_id: String,
|
||||||
|
debug_flags: u32,
|
||||||
) {
|
) {
|
||||||
|
let events_debug = DebugFeature::from_bits_retain(debug_flags).contains(DebugFeature::Events);
|
||||||
while let Some(event) = rx.recv().await {
|
while let Some(event) = rx.recv().await {
|
||||||
match event {
|
match event {
|
||||||
ExecutionEvent::TaskStarted { task_id, timestamp } => {
|
ExecutionEvent::TaskStarted { task_id, timestamp } => {
|
||||||
log::debug!(
|
if events_debug {
|
||||||
"[{}] [{}] [{}] Task started: {}",
|
log::info!(
|
||||||
pipeline_name,
|
"[{}] [{}] [{}] Task started: {}",
|
||||||
build_id,
|
pipeline_name,
|
||||||
timestamp,
|
build_id,
|
||||||
task_id
|
timestamp,
|
||||||
);
|
task_id
|
||||||
|
);
|
||||||
|
} else {
|
||||||
|
log::debug!(
|
||||||
|
"Task started: {}",
|
||||||
|
task_id
|
||||||
|
);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
ExecutionEvent::OutputChunk {
|
ExecutionEvent::OutputChunk {
|
||||||
task_id,
|
task_id: _,
|
||||||
stream,
|
stream,
|
||||||
data,
|
data,
|
||||||
} => match stream {
|
} => match stream {
|
||||||
StreamType::Stdout => print!("[{}] [{}] [stdout] {}", pipeline_name, task_id, data),
|
StreamType::Stdout => print!("{}", data),
|
||||||
StreamType::Stderr => {
|
StreamType::Stderr => {
|
||||||
eprint!("[{}] [{}] [stderr] {}", pipeline_name, task_id, data)
|
eprint!("{}", data)
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
ExecutionEvent::TaskCompleted { task_id, result } => {
|
ExecutionEvent::TaskCompleted { task_id, result } => {
|
||||||
log::debug!(
|
if result.success {
|
||||||
"[{}] [{}] Task completed: {} (exit_code={}, duration={:?})",
|
if events_debug {
|
||||||
pipeline_name,
|
log::info!(
|
||||||
build_id,
|
"[{}] [{}] Task completed: {} (exit_code={}, duration={:?})",
|
||||||
task_id,
|
pipeline_name, build_id, task_id,
|
||||||
result.exit_code,
|
result.exit_code, result.duration,
|
||||||
result.duration
|
);
|
||||||
);
|
} else {
|
||||||
|
log::debug!(
|
||||||
|
"Task completed: {} (exit_code={}, duration={:?})",
|
||||||
|
task_id, result.exit_code, result.duration,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
log::warn!(
|
||||||
|
"Task completed with non-zero exit: {} (exit_code={}, duration={:?})",
|
||||||
|
task_id,
|
||||||
|
result.exit_code, result.duration,
|
||||||
|
);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
ExecutionEvent::TaskFailed { task_id, error } => {
|
ExecutionEvent::TaskFailed { task_id, error } => {
|
||||||
log::error!(
|
log::error!(
|
||||||
"[{}] [{}] Task failed: {} (error={})",
|
"Task failed: {} (error={})",
|
||||||
pipeline_name,
|
|
||||||
build_id,
|
|
||||||
task_id,
|
task_id,
|
||||||
error
|
error
|
||||||
);
|
);
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ use std::path::{Path, PathBuf};
|
|||||||
use std::sync::Arc;
|
use std::sync::Arc;
|
||||||
|
|
||||||
mod dylib;
|
mod dylib;
|
||||||
mod internal;
|
pub mod internal;
|
||||||
mod metadata;
|
mod metadata;
|
||||||
mod rhai;
|
mod rhai;
|
||||||
mod shell;
|
mod shell;
|
||||||
@@ -232,7 +232,9 @@ pub fn register(
|
|||||||
_ => {}
|
_ => {}
|
||||||
}
|
}
|
||||||
log::debug!("Registering internal plugin {}", i.name);
|
log::debug!("Registering internal plugin {}", i.name);
|
||||||
plugins.insert(i.name.to_string(), FinalizePlugin::new(i.entry));
|
let mut plugin = FinalizePlugin::new(i.entry);
|
||||||
|
plugin.metadata.renderer = Some(i.renderer);
|
||||||
|
plugins.insert(i.name.to_string(), plugin);
|
||||||
count += 1;
|
count += 1;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -1,4 +1,5 @@
|
|||||||
use crate::finalize::plugin::AsyncPluginFn;
|
use crate::finalize::plugin::AsyncPluginFn;
|
||||||
|
use crate::notify::types::NotificationRenderer;
|
||||||
pub mod dummy;
|
pub mod dummy;
|
||||||
mod insitenotify;
|
mod insitenotify;
|
||||||
mod mail;
|
mod mail;
|
||||||
@@ -7,6 +8,7 @@ mod webhook;
|
|||||||
pub struct InternalPlugin {
|
pub struct InternalPlugin {
|
||||||
pub name: &'static str,
|
pub name: &'static str,
|
||||||
pub entry: Box<dyn AsyncPluginFn>,
|
pub entry: Box<dyn AsyncPluginFn>,
|
||||||
|
pub renderer: NotificationRenderer,
|
||||||
}
|
}
|
||||||
|
|
||||||
pub fn get_internal_plugin() -> Vec<InternalPlugin> {
|
pub fn get_internal_plugin() -> Vec<InternalPlugin> {
|
||||||
@@ -14,25 +16,32 @@ pub fn get_internal_plugin() -> Vec<InternalPlugin> {
|
|||||||
InternalPlugin {
|
InternalPlugin {
|
||||||
name: "in-site-notify",
|
name: "in-site-notify",
|
||||||
entry: Box::new(insitenotify::InsiteNotify),
|
entry: Box::new(insitenotify::InsiteNotify),
|
||||||
|
renderer: NotificationRenderer::Plugin,
|
||||||
},
|
},
|
||||||
InternalPlugin {
|
InternalPlugin {
|
||||||
name: "mail",
|
name: "mail",
|
||||||
entry: Box::new(mail::Mail),
|
entry: Box::new(mail::Mail),
|
||||||
|
renderer: NotificationRenderer::Server,
|
||||||
},
|
},
|
||||||
InternalPlugin {
|
InternalPlugin {
|
||||||
name: "satori",
|
name: "satori",
|
||||||
entry: Box::new(satori::Satori),
|
entry: Box::new(satori::Satori),
|
||||||
|
renderer: NotificationRenderer::Server,
|
||||||
},
|
},
|
||||||
InternalPlugin {
|
InternalPlugin {
|
||||||
name: "webhook",
|
name: "webhook",
|
||||||
entry: Box::new(webhook::Webhook),
|
entry: Box::new(webhook::Webhook),
|
||||||
|
renderer: NotificationRenderer::Server,
|
||||||
},
|
},
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
|
|
||||||
pub fn get_dummy_plugin() -> Vec<InternalPlugin> {
|
/// Returns dummy only (for `--debug notify.dummy` mode).
|
||||||
|
/// All real adapters and in-site-notify are excluded.
|
||||||
|
pub fn get_debug_plugin() -> Vec<InternalPlugin> {
|
||||||
vec![InternalPlugin {
|
vec![InternalPlugin {
|
||||||
name: "dummy",
|
name: "dummy",
|
||||||
entry: Box::new(dummy::Dummy),
|
entry: Box::new(dummy::Dummy),
|
||||||
|
renderer: NotificationRenderer::Plugin,
|
||||||
}]
|
}]
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -10,12 +10,12 @@ pub struct Dummy;
|
|||||||
impl AsyncPluginFn for Dummy {
|
impl AsyncPluginFn for Dummy {
|
||||||
async fn call(
|
async fn call(
|
||||||
&self,
|
&self,
|
||||||
_metadata: &PluginMetadata,
|
metadata: &PluginMetadata,
|
||||||
_argument: serde_yaml::Value,
|
argument: serde_yaml::Value,
|
||||||
_ctx: &ExecutionContext,
|
_ctx: &ExecutionContext,
|
||||||
_event_tx: &EventSender,
|
_event_tx: &EventSender,
|
||||||
) -> Result<(), PluginError> {
|
) -> Result<(), PluginError> {
|
||||||
log::debug!("[NOTIFY_DEBUG] Successfully registered dummy plugin.");
|
log::info!("[Dummy] metadata={:?}, arg={:?}", metadata, argument);
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ use serde::Deserialize;
|
|||||||
use std::path::PathBuf;
|
use std::path::PathBuf;
|
||||||
|
|
||||||
/// Plugin manifest metadata containing identification and configuration.
|
/// Plugin manifest metadata containing identification and configuration.
|
||||||
#[derive(Deserialize, Clone)]
|
#[derive(Deserialize, Clone, Debug)]
|
||||||
pub struct PluginMetadata {
|
pub struct PluginMetadata {
|
||||||
/// Plugin name identifier.
|
/// Plugin name identifier.
|
||||||
pub name: String,
|
pub name: String,
|
||||||
|
|||||||
@@ -51,7 +51,7 @@ pub async fn hook(
|
|||||||
.await
|
.await
|
||||||
.map_err(|e| {
|
.map_err(|e| {
|
||||||
log::error!("Plugin hook {} failed: {:?}", index, e);
|
log::error!("Plugin hook {} failed: {:?}", index, e);
|
||||||
ExecutionError::ExecutionFailed(e.to_string())
|
ExecutionError::ExecutionFailed { task_id: format!("hook-{}", index), exit_code: -1 }
|
||||||
})?;
|
})?;
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|||||||
+19
-12
@@ -3,9 +3,11 @@ pub mod bake;
|
|||||||
pub mod error;
|
pub mod error;
|
||||||
pub mod finalize;
|
pub mod finalize;
|
||||||
pub mod notify;
|
pub mod notify;
|
||||||
|
pub mod prefetch;
|
||||||
pub mod prebake;
|
pub mod prebake;
|
||||||
pub mod types;
|
pub mod types;
|
||||||
pub mod utils;
|
pub mod utils;
|
||||||
|
pub mod daemon;
|
||||||
|
|
||||||
pub use clap::{Parser, Subcommand};
|
pub use clap::{Parser, Subcommand};
|
||||||
pub use serde::Deserialize;
|
pub use serde::Deserialize;
|
||||||
@@ -21,37 +23,42 @@ pub mod cli {
|
|||||||
pub struct Cli {
|
pub struct Cli {
|
||||||
/// Username of the pipeline executor
|
/// Username of the pipeline executor
|
||||||
#[arg(short, long, env = "HBW_USERNAME")]
|
#[arg(short, long, env = "HBW_USERNAME")]
|
||||||
pub username: String,
|
pub username: Option<String>,
|
||||||
|
|
||||||
/// Pipeline ID of this pipeline
|
/// Pipeline ID of this pipeline
|
||||||
#[arg(short, long, env = "HBW_PIPELINE")]
|
#[arg(short, long, env = "HBW_PIPELINE")]
|
||||||
pub pipeline: String,
|
pub pipeline: Option<String>,
|
||||||
|
|
||||||
/// Build ID of this build
|
/// Build ID of this build
|
||||||
#[arg(short, long, env = "HBW_BUILD_ID")]
|
#[arg(short, long, env = "HBW_BUILD_ID")]
|
||||||
pub build_id: String,
|
pub build_id: Option<String>,
|
||||||
|
|
||||||
/// Parse the script without executing
|
/// Parse the script without executing
|
||||||
#[arg(long, default_value = "false", env = "HBW_DRY_RUN")]
|
#[arg(long, default_value = "false", env = "HBW_DRY_RUN")]
|
||||||
pub dry_run: bool,
|
pub dry_run: bool,
|
||||||
|
|
||||||
/// Override prebake config path
|
/// Override prebake config path
|
||||||
#[arg(long)]
|
#[arg(long, default_value = ".workshop/prebake.yml")]
|
||||||
pub prebake: Option<PathBuf>,
|
pub prebake: PathBuf,
|
||||||
/// Override bake script path
|
/// Override bake script path
|
||||||
#[arg(long)]
|
#[arg(long, default_value = ".workshop/bake.sh")]
|
||||||
pub bake: Option<PathBuf>,
|
pub bake: PathBuf,
|
||||||
/// Override bake_base.sh path
|
/// Override bake_base.sh path
|
||||||
#[arg(long)]
|
#[arg(long, default_value = ".workshop/bake_base.sh")]
|
||||||
pub bake_base: Option<PathBuf>,
|
pub bake_base: PathBuf,
|
||||||
/// Override finalize config path
|
/// Override finalize config path
|
||||||
#[arg(long)]
|
#[arg(long, default_value = ".workshop/finalize.yml")]
|
||||||
pub finalize: Option<PathBuf>,
|
pub finalize: PathBuf,
|
||||||
|
|
||||||
/// Debug features (comma-separated). "notify" enables notification debug logging.
|
/// Debug features. Use `--debug help` for details.
|
||||||
#[arg(long, env = "HBW_DEBUG")]
|
#[arg(long, env = "HBW_DEBUG")]
|
||||||
pub debug: Option<String>,
|
pub debug: Option<String>,
|
||||||
|
|
||||||
|
/// JSON string of pre-resolved resources: `{"name":"/path",...}`
|
||||||
|
/// Only valid in `bare` mode.
|
||||||
|
#[arg(long, value_name = "JSON")]
|
||||||
|
pub resource_registry: Option<String>,
|
||||||
|
|
||||||
#[command(subcommand)]
|
#[command(subcommand)]
|
||||||
pub command: Option<Commands>,
|
pub command: Option<Commands>,
|
||||||
}
|
}
|
||||||
|
|||||||
+173
-49
@@ -1,89 +1,164 @@
|
|||||||
use workshop_baker::types::resource::ResourceRegistry;
|
use capctl;
|
||||||
use std::path::PathBuf;
|
|
||||||
|
|
||||||
use workshop_baker::{
|
|
||||||
bake::bake, cli::{Cli, Commands, Parser},
|
|
||||||
error::CliError,
|
|
||||||
finalize::finalize, notify::notify, prebake::prebake,
|
|
||||||
};
|
|
||||||
use workshop_baker::bare::run_bare;
|
use workshop_baker::bare::run_bare;
|
||||||
use workshop_baker::notify::queue::NotificationQueue;
|
use workshop_baker::notify::queue::NotificationQueue;
|
||||||
|
use workshop_baker::types::debug::DebugFeature;
|
||||||
|
use workshop_baker::types::resource::ResourceRegistry;
|
||||||
|
use workshop_baker::{
|
||||||
|
bake::bake,
|
||||||
|
cli::{Cli, Commands, Parser},
|
||||||
|
error::CliError,
|
||||||
|
finalize::finalize,
|
||||||
|
notify::notify,
|
||||||
|
prebake::prebake,
|
||||||
|
};
|
||||||
use workshop_engine::{EventSender, ExecutionContext};
|
use workshop_engine::{EventSender, ExecutionContext};
|
||||||
|
|
||||||
fn or_workshop(path: &Option<PathBuf>, name: &str) -> PathBuf {
|
|
||||||
path.clone().unwrap_or_else(|| PathBuf::from(format!(".workshop/{}", name)))
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn worker_main(cli: Cli) -> Result<(), CliError> {
|
async fn worker_main(cli: Cli) -> Result<(), CliError> {
|
||||||
let prebake_path = cli.prebake.clone().unwrap_or_else(|| {
|
// a magic closure hack
|
||||||
log::error!("--prebake is required in worker standalone mode");
|
let require = |name: &str, val: &Option<String>| -> String {
|
||||||
std::process::exit(1);
|
val.clone().unwrap_or_else(|| {
|
||||||
});
|
log::error!("--{} is required", name);
|
||||||
let bake_path = cli.bake.clone().unwrap_or_else(|| {
|
std::process::exit(1);
|
||||||
log::error!("--bake is required in worker standalone mode");
|
})
|
||||||
std::process::exit(1);
|
};
|
||||||
});
|
// worker depends on these information to work
|
||||||
let finalize_path = cli.finalize.clone().unwrap_or_else(|| {
|
let pipeline = require("pipeline", &cli.pipeline);
|
||||||
log::error!("--finalize is required in worker standalone mode");
|
let build_id = require("build-id", &cli.build_id);
|
||||||
std::process::exit(1);
|
let username = require("username", &cli.username);
|
||||||
});
|
|
||||||
|
|
||||||
log::info!("Worker standalone mode");
|
println!("── worker standalone mode ───────────────────────");
|
||||||
|
println!(" {:<14} {:?}", "prebake:", cli.prebake);
|
||||||
|
println!(" {:<14} {:?}", "bake:", cli.bake);
|
||||||
|
println!(" {:<14} {:?}", "bake-base:", cli.bake_base);
|
||||||
|
println!(" {:<14} {:?}", "finalize:", cli.finalize);
|
||||||
|
println!("──────────────────────────────────────────────────");
|
||||||
|
|
||||||
|
let debug_flags = DebugFeature::parse(cli.debug.as_deref());
|
||||||
|
if !debug_flags.is_empty() {
|
||||||
|
log::info!("Debug features enabled: {:?}", debug_flags);
|
||||||
|
}
|
||||||
|
|
||||||
let mut ctx = ExecutionContext {
|
let mut ctx = ExecutionContext {
|
||||||
standalone: true,
|
standalone: true,
|
||||||
task_id: format!("{}-{}-worker", cli.pipeline, cli.build_id),
|
task_id: format!("{}-{}-worker", pipeline, build_id),
|
||||||
pipeline_name: cli.pipeline.clone(),
|
pipeline_name: pipeline.clone(),
|
||||||
username: cli.username.clone(),
|
build_id: build_id.clone(),
|
||||||
|
username: username.clone(),
|
||||||
dry_run: cli.dry_run,
|
dry_run: cli.dry_run,
|
||||||
privileged: false,
|
privileged: false,
|
||||||
|
debug_flags: debug_flags.bits(),
|
||||||
..Default::default()
|
..Default::default()
|
||||||
};
|
};
|
||||||
|
|
||||||
let (tx, event_rx) = tokio::sync::mpsc::channel(256);
|
let (tx, event_rx) = tokio::sync::mpsc::channel(256);
|
||||||
let event_tx: EventSender = Some(tx);
|
let event_tx: EventSender = Some(tx);
|
||||||
let _event_handle = tokio::spawn(workshop_baker::prebake::event::event_receiver(
|
let _event_handle = tokio::spawn(workshop_baker::prebake::event::event_receiver(
|
||||||
event_rx, cli.pipeline.clone(), cli.build_id.clone(),
|
event_rx,
|
||||||
|
pipeline.clone(),
|
||||||
|
build_id.clone(),
|
||||||
|
ctx.debug_flags,
|
||||||
));
|
));
|
||||||
|
|
||||||
let (to_tx, to_rx) = tokio::sync::mpsc::channel(256);
|
let (to_tx, to_rx) = tokio::sync::mpsc::channel(256);
|
||||||
let (retry_tx, retry_rx) = tokio::sync::mpsc::channel(256);
|
let (retry_tx, retry_rx) = tokio::sync::mpsc::channel(256);
|
||||||
|
|
||||||
// Resource registry — populated before stages, consumed by stages
|
// Resource registry — fetch plugins/templates before pipeline stages
|
||||||
let registry = ResourceRegistry::new();
|
let mut registry = ResourceRegistry::new();
|
||||||
// TODO: pre-fetch plugins and templates into registry
|
if let Ok(finalize_cfg) = workshop_baker::finalize::parse(&cli.finalize) {
|
||||||
|
let _ = workshop_baker::prefetch::prefetch_resources(&finalize_cfg, &mut registry).await;
|
||||||
|
}
|
||||||
|
let registry = registry; // freeze mut → immut
|
||||||
|
|
||||||
let notify_path = finalize_path.clone();
|
let notify_finalize = cli.finalize.clone();
|
||||||
let notify_ctx = ctx.clone();
|
let notify_ctx = ctx.clone();
|
||||||
let notify_etx = event_tx.clone();
|
let notify_etx = event_tx.clone();
|
||||||
let debug = cli.debug.clone();
|
|
||||||
let notify_registry = registry.clone();
|
let notify_registry = registry.clone();
|
||||||
|
let notify_debug = ctx.debug_flags;
|
||||||
let notify_handle = tokio::spawn(async move {
|
let notify_handle = tokio::spawn(async move {
|
||||||
notify(¬ify_path, &mut notify_ctx.clone(), notify_etx, to_rx, retry_tx, debug.map(|d| d.contains("notify")).unwrap_or(false), ¬ify_registry).await
|
notify(
|
||||||
|
¬ify_finalize,
|
||||||
|
&mut notify_ctx.clone(),
|
||||||
|
notify_etx,
|
||||||
|
to_rx,
|
||||||
|
retry_tx,
|
||||||
|
notify_debug,
|
||||||
|
¬ify_registry,
|
||||||
|
)
|
||||||
|
.await
|
||||||
});
|
});
|
||||||
|
|
||||||
let nevent_tx = to_tx.clone();
|
let nevent_tx = to_tx.clone();
|
||||||
let mut queue = NotificationQueue::new(to_tx, retry_rx);
|
let mut queue = NotificationQueue::new(to_tx, retry_rx, notify_debug);
|
||||||
|
|
||||||
let pipe_result: anyhow::Result<()> = async {
|
let pipe_result: anyhow::Result<()> = async {
|
||||||
prebake(&prebake_path, &cli, None, &mut ctx, event_tx.clone(), Some(nevent_tx)).await?;
|
prebake(
|
||||||
let bake_base_path = or_workshop(&cli.bake_base, "bake_base.sh");
|
&cli.prebake,
|
||||||
let pb_path = or_workshop(&cli.prebake, "prebake.yml");
|
&cli,
|
||||||
bake(&bake_path, &bake_base_path, &pb_path, &cli, &mut ctx, event_tx.clone()).await?;
|
None,
|
||||||
finalize(&finalize_path, &cli, &mut ctx, event_tx.clone(), ®istry).await?;
|
&mut ctx,
|
||||||
|
event_tx.clone(),
|
||||||
|
Some(nevent_tx.clone()),
|
||||||
|
)
|
||||||
|
.await?;
|
||||||
|
let bake_result = bake(
|
||||||
|
&cli.bake,
|
||||||
|
&cli.bake_base,
|
||||||
|
&cli.prebake,
|
||||||
|
&cli,
|
||||||
|
&mut ctx,
|
||||||
|
event_tx.clone(),
|
||||||
|
Some(nevent_tx.clone()),
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
if bake_result.is_err() {
|
||||||
|
workshop_baker::notify::types::send_stage_event(
|
||||||
|
&Some(nevent_tx.clone()),
|
||||||
|
workshop_baker::types::buildstatus::StagePhase::Bake,
|
||||||
|
"bake",
|
||||||
|
workshop_baker::types::buildstatus::StageOutcome::Failure,
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
}
|
||||||
|
bake_result?;
|
||||||
|
finalize(&cli.finalize, &cli, &mut ctx, event_tx.clone(), ®istry).await?;
|
||||||
Ok(())
|
Ok(())
|
||||||
}.await;
|
}
|
||||||
|
.await;
|
||||||
|
|
||||||
if let Err(e) = pipe_result {
|
if let Err(e) = pipe_result {
|
||||||
|
// Notify pipeline failure before exiting
|
||||||
|
workshop_baker::notify::types::send_stage_event(
|
||||||
|
&Some(nevent_tx.clone()),
|
||||||
|
workshop_baker::types::buildstatus::StagePhase::Bake,
|
||||||
|
"pipeline",
|
||||||
|
workshop_baker::types::buildstatus::StageOutcome::PipelineFailure,
|
||||||
|
)
|
||||||
|
.await;
|
||||||
log::error!("Pipeline stage failed: {}", e);
|
log::error!("Pipeline stage failed: {}", e);
|
||||||
return Err(CliError::General(e));
|
return Err(CliError::General(e));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Notify pipeline success
|
||||||
|
workshop_baker::notify::types::send_stage_event(
|
||||||
|
&Some(nevent_tx.clone()),
|
||||||
|
workshop_baker::types::buildstatus::StagePhase::Bake,
|
||||||
|
"pipeline",
|
||||||
|
workshop_baker::types::buildstatus::StageOutcome::PipelineSuccess,
|
||||||
|
)
|
||||||
|
.await;
|
||||||
log::info!("Pipeline complete. Draining notifications...");
|
log::info!("Pipeline complete. Draining notifications...");
|
||||||
|
drop(nevent_tx);
|
||||||
|
queue.tx = None;
|
||||||
if let Err(e) = queue.drain().await {
|
if let Err(e) = queue.drain().await {
|
||||||
log::error!("Notification queue error: {}", e);
|
log::error!("Notification queue error: {}", e);
|
||||||
}
|
}
|
||||||
let _ = notify_handle.await;
|
let _ = notify_handle.await;
|
||||||
|
|
||||||
|
// Cleanup after notifications are fully drained
|
||||||
|
let pipeline_ok = pipe_result.is_ok();
|
||||||
|
let _ = workshop_baker::finalize::parse(&cli.finalize).map(|cfg| {
|
||||||
|
let _ = workshop_baker::finalize::cleanup::cleanup(&cfg.cleanup, &ctx, pipeline_ok);
|
||||||
|
});
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -91,32 +166,81 @@ async fn worker_main(cli: Cli) -> Result<(), CliError> {
|
|||||||
async fn main() {
|
async fn main() {
|
||||||
env_logger::init();
|
env_logger::init();
|
||||||
let cli = Cli::parse();
|
let cli = Cli::parse();
|
||||||
|
if std::process::id() == 1 {
|
||||||
|
log::error!("Baker is not meant to be run as PID=1");
|
||||||
|
std::process::exit(1);
|
||||||
|
}
|
||||||
|
if let Err(e) = capctl::prctl::set_subreaper(true){
|
||||||
|
log::error!("Failed to set self as subreaper: {}", e);
|
||||||
|
log::error!("finalize.cleanup might not work properly");
|
||||||
|
}
|
||||||
|
|
||||||
|
// --debug help: print help and exit immediately
|
||||||
|
if let Some(ref raw) = cli.debug {
|
||||||
|
if DebugFeature::print_help(raw) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let debug_flags = DebugFeature::parse(cli.debug.as_deref());
|
||||||
if cli.dry_run {
|
if cli.dry_run {
|
||||||
log::warn!("Dry run enabled, no changes will be made.");
|
log::warn!("Dry run enabled, no changes will be made.");
|
||||||
}
|
}
|
||||||
|
|
||||||
let pipeline = cli.pipeline.clone();
|
// --resource-registry is only allowed in bare mode
|
||||||
let build_id = cli.build_id.clone();
|
if cli.resource_registry.is_some() && !matches!(cli.command, Some(Commands::Bare { .. })) {
|
||||||
|
log::error!("--resource-registry is only supported in bare mode");
|
||||||
|
std::process::exit(1);
|
||||||
|
}
|
||||||
match &cli.command {
|
match &cli.command {
|
||||||
Some(Commands::Bare { command }) => {
|
Some(Commands::Bare { command }) => {
|
||||||
|
// --pipeline, --build-id, --username are optional in bare mode
|
||||||
|
// and they will be defaulted to "BarePipeline", "0", "bare" respectively
|
||||||
|
let pipeline = cli
|
||||||
|
.pipeline
|
||||||
|
.as_deref()
|
||||||
|
.unwrap_or("BarePipeline")
|
||||||
|
.to_string();
|
||||||
|
let build_id = cli.build_id.as_deref().unwrap_or("0").to_string();
|
||||||
|
let username = cli.username.as_deref().unwrap_or("bare").to_string();
|
||||||
|
|
||||||
let mut ctx = ExecutionContext {
|
let mut ctx = ExecutionContext {
|
||||||
standalone: true,
|
standalone: true,
|
||||||
task_id: format!("{}-{}-bare", pipeline, build_id),
|
task_id: format!("{}-{}-bare", pipeline, build_id),
|
||||||
pipeline_name: pipeline.clone(),
|
pipeline_name: pipeline.clone(),
|
||||||
username: cli.username.clone(),
|
build_id: build_id.clone(),
|
||||||
|
username: username.clone(),
|
||||||
dry_run: cli.dry_run,
|
dry_run: cli.dry_run,
|
||||||
privileged: false,
|
privileged: false,
|
||||||
|
debug_flags: debug_flags.bits(),
|
||||||
..Default::default()
|
..Default::default()
|
||||||
};
|
};
|
||||||
let (tx, event_rx) = tokio::sync::mpsc::channel(256);
|
let (tx, event_rx) = tokio::sync::mpsc::channel(256);
|
||||||
let event_tx: EventSender = Some(tx);
|
let event_tx: EventSender = Some(tx);
|
||||||
let event_handle = tokio::spawn(workshop_baker::prebake::event::event_receiver(
|
let event_handle = tokio::spawn(workshop_baker::prebake::event::event_receiver(
|
||||||
event_rx, pipeline.clone(), build_id.clone(),
|
event_rx,
|
||||||
|
pipeline.clone(),
|
||||||
|
build_id.clone(),
|
||||||
|
ctx.debug_flags,
|
||||||
));
|
));
|
||||||
|
|
||||||
let registry = ResourceRegistry::new();
|
let registry = if let Some(ref json_str) = cli.resource_registry {
|
||||||
let result = run_bare(&cli, command, &mut ctx, event_tx, &pipeline, &build_id, ®istry).await;
|
match workshop_baker::types::resource::ResourceRegistry::from_json_str(json_str) {
|
||||||
|
Ok(r) => {
|
||||||
|
log::info!("Loaded {} resources from --resource-registry", r.len());
|
||||||
|
r
|
||||||
|
}
|
||||||
|
Err(e) => {
|
||||||
|
log::error!("Failed to parse --resource-registry: {}", e);
|
||||||
|
std::process::exit(1);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
ResourceRegistry::new()
|
||||||
|
};
|
||||||
|
let result = run_bare(
|
||||||
|
&cli, command, &mut ctx, event_tx, &pipeline, &build_id, ®istry,
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
|
||||||
drop(ctx);
|
drop(ctx);
|
||||||
let _ = event_handle.await;
|
let _ = event_handle.await;
|
||||||
|
|||||||
@@ -1,118 +0,0 @@
|
|||||||
use crate::monitor::UsageStats;
|
|
||||||
use cgroups_rs;
|
|
||||||
use cgroups_rs::fs::Cgroup;
|
|
||||||
use cgroups_rs::fs::{cpu::CpuController, memory::MemController, pid::PidController};
|
|
||||||
use log;
|
|
||||||
use std::time::{Duration, Instant};
|
|
||||||
#[cfg(target_os = "linux")]
|
|
||||||
pub fn get_usage(
|
|
||||||
name: &String,
|
|
||||||
usage_prev: crate::monitor::UsageStats,
|
|
||||||
) -> anyhow::Result<crate::monitor::UsageStats> {
|
|
||||||
let hierarchy = cgroups_rs::fs::hierarchies::auto();
|
|
||||||
if !hierarchy.v2() {
|
|
||||||
return Err(anyhow::anyhow!(
|
|
||||||
"Unsupported cgroup hierarchy, cgroups v2 required."
|
|
||||||
));
|
|
||||||
}
|
|
||||||
let cgroup = Cgroup::load(hierarchy, name);
|
|
||||||
let mut cpu_usage_us: u64 = 0;
|
|
||||||
let mut cpu_usage_percent: f64 = 0.0;
|
|
||||||
let mut memory_usage_bytes: u64 = 0;
|
|
||||||
let mut memory_limit_bytes: i64 = 0;
|
|
||||||
let mut process_count: u64 = 0;
|
|
||||||
|
|
||||||
if let Some(cpu) = cgroup.controller_of::<CpuController>() {
|
|
||||||
let stats = parse_cpustat(cpu.cpu().stat.as_str())?;
|
|
||||||
cpu_usage_us = stats.usage_usec;
|
|
||||||
} else {
|
|
||||||
log::warn!("Failed to fetch CPU usage info!");
|
|
||||||
}
|
|
||||||
|
|
||||||
let current = Instant::now();
|
|
||||||
let delta_usage = cpu_usage_us - usage_prev.cpu_usage_us;
|
|
||||||
let delta_wall = current - usage_prev._timestamp;
|
|
||||||
if usage_prev._timestamp > current {
|
|
||||||
log::warn!("Usage stats are from the future!");
|
|
||||||
} else if usage_prev.cpu_usage_us == 0 {
|
|
||||||
log::info!("Previous CPU usage is zero, probably a fresh start.");
|
|
||||||
} else if delta_wall < Duration::from_millis(1) {
|
|
||||||
log::warn!("Usage stats are too close together!");
|
|
||||||
} else {
|
|
||||||
cpu_usage_percent = delta_usage as f64 / delta_wall.as_micros() as f64 * 100.0;
|
|
||||||
// dbg!(&delta_usage, &delta_wall);
|
|
||||||
}
|
|
||||||
|
|
||||||
if let Some(mem) = cgroup.controller_of::<MemController>() {
|
|
||||||
let stats = mem.memory_stat();
|
|
||||||
memory_usage_bytes = stats.usage_in_bytes;
|
|
||||||
memory_limit_bytes = stats.limit_in_bytes;
|
|
||||||
}
|
|
||||||
|
|
||||||
if let Some(proc) = cgroup.controller_of::<PidController>() {
|
|
||||||
process_count = proc.get_pid_current()?;
|
|
||||||
}
|
|
||||||
Ok(UsageStats {
|
|
||||||
cpu_usage_us,
|
|
||||||
cpu_usage_percent,
|
|
||||||
memory_usage_bytes,
|
|
||||||
// memory_usage_percent: 0.0,
|
|
||||||
memory_limit_bytes,
|
|
||||||
process_count,
|
|
||||||
_timestamp: Instant::now(),
|
|
||||||
})
|
|
||||||
|
|
||||||
// todo!();
|
|
||||||
}
|
|
||||||
|
|
||||||
#[derive(Debug, Default, Clone)]
|
|
||||||
pub struct CgroupCpuStat {
|
|
||||||
pub usage_usec: u64,
|
|
||||||
pub user_usec: u64,
|
|
||||||
pub system_usec: u64,
|
|
||||||
pub nice_usec: u64,
|
|
||||||
pub core_sched_force_idle_usec: u64,
|
|
||||||
pub nr_periods: u64,
|
|
||||||
pub nr_throttled: u64,
|
|
||||||
pub throttled_usec: u64,
|
|
||||||
pub nr_bursts: u64,
|
|
||||||
pub burst_usec: u64,
|
|
||||||
}
|
|
||||||
|
|
||||||
fn parse_cpustat(stat: &str) -> anyhow::Result<CgroupCpuStat> {
|
|
||||||
let mut cpu_stat = CgroupCpuStat::default();
|
|
||||||
for line in stat.lines() {
|
|
||||||
let line = line.trim();
|
|
||||||
if line.is_empty() {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
|
|
||||||
let (key, value_str) = line
|
|
||||||
.split_once(' ')
|
|
||||||
.ok_or_else(|| anyhow::anyhow!("Invalid format:{}", line))?;
|
|
||||||
|
|
||||||
let value = value_str.parse::<u64>().map_err(|e| {
|
|
||||||
anyhow::anyhow!(
|
|
||||||
"Cannot convert value {}({}) into integer: {}",
|
|
||||||
key,
|
|
||||||
value_str,
|
|
||||||
e
|
|
||||||
)
|
|
||||||
})?;
|
|
||||||
|
|
||||||
match key {
|
|
||||||
"usage_usec" => cpu_stat.usage_usec = value,
|
|
||||||
"user_usec" => cpu_stat.user_usec = value,
|
|
||||||
"system_usec" => cpu_stat.system_usec = value,
|
|
||||||
"nice_usec" => cpu_stat.nice_usec = value,
|
|
||||||
"core_sched.force_idle_usec" => cpu_stat.core_sched_force_idle_usec = value,
|
|
||||||
"nr_periods" => cpu_stat.nr_periods = value,
|
|
||||||
"nr_throttled" => cpu_stat.nr_throttled = value,
|
|
||||||
"throttled_usec" => cpu_stat.throttled_usec = value,
|
|
||||||
"nr_bursts" => cpu_stat.nr_bursts = value,
|
|
||||||
"burst_usec" => cpu_stat.burst_usec = value,
|
|
||||||
_unknown_key => {} // Ignore unknown keys
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Ok(cpu_stat)
|
|
||||||
}
|
|
||||||
@@ -1,4 +0,0 @@
|
|||||||
#[cfg(target_os = "windows")]
|
|
||||||
pub fn get_usage() -> anyhow::Result<f64> {
|
|
||||||
unimplemented!("Not supporting windows JobObject at this time!");
|
|
||||||
}
|
|
||||||
@@ -1,10 +1,12 @@
|
|||||||
use crate::finalize::parse;
|
use crate::finalize::parse;
|
||||||
use crate::finalize::plugin;
|
use crate::finalize::plugin;
|
||||||
|
use crate::finalize::plugin::internal;
|
||||||
use crate::notify::types::DEFAULT_NOTIFY_PARAMS;
|
use crate::notify::types::DEFAULT_NOTIFY_PARAMS;
|
||||||
use crate::notify::types::NotificationEventReceiver;
|
use crate::notify::types::NotificationEventReceiver;
|
||||||
use crate::notify::types::NotificationEventSender;
|
use crate::notify::types::NotificationEventSender;
|
||||||
use crate::notify::handler::NotificationHandler;
|
use crate::notify::handler::NotificationHandler;
|
||||||
use crate::types::resource::ResourceRegistry;
|
use crate::types::resource::ResourceRegistry;
|
||||||
|
use crate::types::debug::DebugFeature;
|
||||||
use std::path::Path;
|
use std::path::Path;
|
||||||
use std::sync::Arc;
|
use std::sync::Arc;
|
||||||
use workshop_engine::{EventSender, ExecutionContext};
|
use workshop_engine::{EventSender, ExecutionContext};
|
||||||
@@ -26,17 +28,23 @@ pub async fn notify(
|
|||||||
_event_tx: EventSender,
|
_event_tx: EventSender,
|
||||||
mut rx: NotificationEventReceiver,
|
mut rx: NotificationEventReceiver,
|
||||||
retry_tx: NotificationEventSender,
|
retry_tx: NotificationEventSender,
|
||||||
_debug: bool,
|
debug_flags: u32,
|
||||||
registry: &ResourceRegistry,
|
registry: &ResourceRegistry,
|
||||||
) -> anyhow::Result<()> {
|
) -> anyhow::Result<()> {
|
||||||
|
let debug = DebugFeature::from_bits_retain(debug_flags);
|
||||||
|
let notify_debug = debug.contains(DebugFeature::NotifyDummy);
|
||||||
|
|
||||||
let mut finalize = parse(finalize_path)?;
|
let mut finalize = parse(finalize_path)?;
|
||||||
|
|
||||||
// Always inject in-site-notify as the lowest priority fallback
|
// L18: 0 priorities → notification ends; L19: has priorities → inject fallback
|
||||||
{
|
// In notify-debug mode, inject dummy instead of in-site-notify.
|
||||||
|
if !finalize.notification.groups.is_empty() {
|
||||||
|
let fallback_name = if notify_debug { "dummy" } else { "in-site-notify" };
|
||||||
|
log::debug!("Injecting {} fallback", fallback_name);
|
||||||
let max_p = finalize.notification.groups.keys().max().copied().unwrap_or(0);
|
let max_p = finalize.notification.groups.keys().max().copied().unwrap_or(0);
|
||||||
let mut m = std::collections::HashMap::new();
|
let mut m = std::collections::HashMap::new();
|
||||||
m.insert(
|
m.insert(
|
||||||
"in-site-notify".to_string(),
|
fallback_name.to_string(),
|
||||||
crate::finalize::NotificationMethod::default(),
|
crate::finalize::NotificationMethod::default(),
|
||||||
);
|
);
|
||||||
finalize.notification.groups.insert(max_p + 1, m);
|
finalize.notification.groups.insert(max_p + 1, m);
|
||||||
@@ -45,34 +53,64 @@ pub async fn notify(
|
|||||||
validate(&finalize)?;
|
validate(&finalize)?;
|
||||||
let plugin_list = extract_plugin(&finalize);
|
let plugin_list = extract_plugin(&finalize);
|
||||||
|
|
||||||
let notify_plugins = plugin::register(
|
// Register plugins: use dummy-only set in notify-debug mode
|
||||||
finalize.plugin,
|
let notify_plugins = if notify_debug {
|
||||||
Some(&plugin_list),
|
log::info!("[DEBUG-NOTIFY] Replacing notification plugins with debug set");
|
||||||
&mut finalize.notification,
|
let mut plugins: crate::finalize::plugin::PluginMap = std::collections::HashMap::new();
|
||||||
registry,
|
for i in internal::get_debug_plugin() {
|
||||||
)?;
|
log::info!("[DEBUG-NOTIFY] Registered: {}", i.name);
|
||||||
|
let mut plugin = plugin::FinalizePlugin::new(i.entry);
|
||||||
|
plugin.metadata.renderer = Some(i.renderer);
|
||||||
|
plugins.insert(i.name.to_string(), plugin);
|
||||||
|
}
|
||||||
|
plugins
|
||||||
|
} else {
|
||||||
|
plugin::register(
|
||||||
|
finalize.plugin,
|
||||||
|
Some(&plugin_list),
|
||||||
|
&mut finalize.notification,
|
||||||
|
registry,
|
||||||
|
)?
|
||||||
|
};
|
||||||
|
|
||||||
resolve_external_template_refs(&mut finalize.notification.templates, registry);
|
resolve_external_template_refs(&mut finalize.notification.templates, registry);
|
||||||
flatten_template_refs(&mut finalize.notification.templates)?;
|
flatten_template_refs(&mut finalize.notification.templates)?;
|
||||||
|
|
||||||
let notification_rules = parse_rules(&finalize.notification);
|
let notification_rules = parse_rules(&finalize.notification);
|
||||||
|
|
||||||
|
// Debug mode: eliminate cooldown so bucket dispatches immediately
|
||||||
|
let (period, watermark) = if notify_debug {
|
||||||
|
(0, 0)
|
||||||
|
} else {
|
||||||
|
(DEFAULT_NOTIFY_PARAMS.batch_period.value, DEFAULT_NOTIFY_PARAMS.batch_watermark.value)
|
||||||
|
};
|
||||||
|
|
||||||
let handler = Arc::new(NotificationHandler::new(
|
let handler = Arc::new(NotificationHandler::new(
|
||||||
Arc::new(notification_rules),
|
Arc::new(notification_rules),
|
||||||
Arc::new(notify_plugins),
|
Arc::new(notify_plugins),
|
||||||
Arc::new(finalize.notification.templates),
|
Arc::new(finalize.notification.templates),
|
||||||
Arc::new(ctx.clone()),
|
Arc::new(ctx.clone()),
|
||||||
retry_tx,
|
retry_tx,
|
||||||
DEFAULT_NOTIFY_PARAMS.batch_period.value,
|
period,
|
||||||
DEFAULT_NOTIFY_PARAMS.batch_watermark.value,
|
watermark,
|
||||||
));
|
));
|
||||||
|
|
||||||
|
let mut handles: Vec<tokio::task::JoinHandle<()>> = Vec::new();
|
||||||
loop {
|
loop {
|
||||||
let Some(event) = rx.recv().await else { break };
|
let Some(event) = rx.recv().await else { break };
|
||||||
|
if notify_debug {
|
||||||
|
log::info!(
|
||||||
|
"[DEBUG-NOTIFY] event: id={} stage={:?} substage={} outcome={:?}",
|
||||||
|
event.id, event.stage, event.substage, event.outcome);
|
||||||
|
}
|
||||||
let h = handler.clone();
|
let h = handler.clone();
|
||||||
tokio::spawn(async move {
|
handles.push(tokio::spawn(async move {
|
||||||
h.process_event(event).await;
|
h.process_event(event).await;
|
||||||
});
|
}));
|
||||||
|
}
|
||||||
|
// Wait for all bucket dispatches to complete before returning
|
||||||
|
for handle in handles {
|
||||||
|
let _ = handle.await;
|
||||||
}
|
}
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -73,8 +73,10 @@ impl NotificationHandler {
|
|||||||
/// Process a single event: route to buckets, await completion,
|
/// Process a single event: route to buckets, await completion,
|
||||||
/// then handle retry/escalation for each bucket result.
|
/// then handle retry/escalation for each bucket result.
|
||||||
pub async fn process_event(self: Arc<Self>, event: NotificationEvent) {
|
pub async fn process_event(self: Arc<Self>, event: NotificationEvent) {
|
||||||
|
log::info!("[TRACE] process_event: stage={:?} substage={} outcome={:?}", event.stage, event.substage, event.outcome);
|
||||||
let entries = self.route_event(&event);
|
let entries = self.route_event(&event);
|
||||||
if entries.is_empty() {
|
if entries.is_empty() {
|
||||||
|
log::warn!("[TRACE] process_event: no matching rules, event dropped");
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -142,6 +144,8 @@ impl NotificationHandler {
|
|||||||
) -> Vec<((String, usize), Arc<Notify>)> {
|
) -> Vec<((String, usize), Arc<Notify>)> {
|
||||||
let mut matches: Vec<(String, usize)> = Vec::new();
|
let mut matches: Vec<(String, usize)> = Vec::new();
|
||||||
|
|
||||||
|
let config_groups: Vec<usize> = self.config.values().map(|r| r.len()).collect();
|
||||||
|
log::info!("[TRACE] route_event: groups={:?}", config_groups);
|
||||||
for ruleset in self.config.values() {
|
for ruleset in self.config.values() {
|
||||||
for (method, rules) in ruleset.iter() {
|
for (method, rules) in ruleset.iter() {
|
||||||
if let Some(ref method_rules) = event.target.rules {
|
if let Some(ref method_rules) = event.target.rules {
|
||||||
@@ -336,7 +340,8 @@ fn build_plugin_argument(
|
|||||||
Some(argument)
|
Some(argument)
|
||||||
}
|
}
|
||||||
NotificationRenderer::Plugin => {
|
NotificationRenderer::Plugin => {
|
||||||
unimplemented!("Plugin renderer is not yet implemented")
|
// PSR: pass config directly to plugin (plugin handles own rendering)
|
||||||
|
Some(rule.config.clone())
|
||||||
}
|
}
|
||||||
NotificationRenderer::Interactive => {
|
NotificationRenderer::Interactive => {
|
||||||
unimplemented!("Interactive renderer is not yet implemented")
|
unimplemented!("Interactive renderer is not yet implemented")
|
||||||
|
|||||||
@@ -1,3 +1,4 @@
|
|||||||
|
use crate::types::debug::DebugFeature;
|
||||||
use crate::notify::types::{NotificationEvent, NotificationEventReceiver, NotificationEventSender};
|
use crate::notify::types::{NotificationEvent, NotificationEventReceiver, NotificationEventSender};
|
||||||
use std::collections::BinaryHeap;
|
use std::collections::BinaryHeap;
|
||||||
|
|
||||||
@@ -8,16 +9,18 @@ use std::collections::BinaryHeap;
|
|||||||
/// - `rx` receives retry events back from notify()
|
/// - `rx` receives retry events back from notify()
|
||||||
pub struct NotificationQueue {
|
pub struct NotificationQueue {
|
||||||
pub pq: BinaryHeap<NotificationEvent>,
|
pub pq: BinaryHeap<NotificationEvent>,
|
||||||
pub tx: NotificationEventSender,
|
pub tx: Option<NotificationEventSender>,
|
||||||
pub rx: NotificationEventReceiver,
|
pub rx: NotificationEventReceiver,
|
||||||
|
pub debug_flags: u32,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl NotificationQueue {
|
impl NotificationQueue {
|
||||||
pub fn new(tx: NotificationEventSender, rx: NotificationEventReceiver) -> Self {
|
pub fn new(tx: NotificationEventSender, rx: NotificationEventReceiver, debug_flags: u32) -> Self {
|
||||||
Self {
|
Self {
|
||||||
pq: BinaryHeap::new(),
|
pq: BinaryHeap::new(),
|
||||||
tx,
|
tx: Some(tx),
|
||||||
rx,
|
rx,
|
||||||
|
debug_flags,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -29,8 +32,12 @@ impl NotificationQueue {
|
|||||||
/// Non-blocking single round: recycle retry events into PQ, then
|
/// Non-blocking single round: recycle retry events into PQ, then
|
||||||
/// pop and send all ready events (not_before ≤ now).
|
/// pop and send all ready events (not_before ≤ now).
|
||||||
pub async fn flush_now(&mut self) -> anyhow::Result<()> {
|
pub async fn flush_now(&mut self) -> anyhow::Result<()> {
|
||||||
|
let events_debug = DebugFeature::from_bits_retain(self.debug_flags).contains(DebugFeature::Events);
|
||||||
// Recycle retry events
|
// Recycle retry events
|
||||||
while let Ok(event) = self.rx.try_recv() {
|
while let Ok(event) = self.rx.try_recv() {
|
||||||
|
if events_debug {
|
||||||
|
log::info!("[EVENTS] flush_now: recycle event id={} retrying", event.id);
|
||||||
|
}
|
||||||
self.pq.push(event);
|
self.pq.push(event);
|
||||||
}
|
}
|
||||||
// Send ready events
|
// Send ready events
|
||||||
@@ -40,7 +47,12 @@ impl NotificationQueue {
|
|||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
let event = self.pq.pop().unwrap();
|
let event = self.pq.pop().unwrap();
|
||||||
self.tx.send(event).await?;
|
if events_debug {
|
||||||
|
log::info!("[EVENTS] flush_now: sending event id={}", event.id);
|
||||||
|
}
|
||||||
|
if let Some(ref tx) = self.tx {
|
||||||
|
tx.send(event).await?;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
@@ -49,9 +61,13 @@ impl NotificationQueue {
|
|||||||
/// For standalone mode: called after prebake/bake/finalize to wait for completion.
|
/// For standalone mode: called after prebake/bake/finalize to wait for completion.
|
||||||
/// For c/d mode: the main execution loop.
|
/// For c/d mode: the main execution loop.
|
||||||
pub async fn drain(&mut self) -> anyhow::Result<()> {
|
pub async fn drain(&mut self) -> anyhow::Result<()> {
|
||||||
|
let events_debug = DebugFeature::from_bits_retain(self.debug_flags).contains(DebugFeature::Events);
|
||||||
loop {
|
loop {
|
||||||
// Drain retry channel
|
// Drain retry channel
|
||||||
while let Ok(event) = self.rx.try_recv() {
|
while let Ok(event) = self.rx.try_recv() {
|
||||||
|
if events_debug {
|
||||||
|
log::info!("[EVENTS] drain: recycle event id={} retrying", event.id);
|
||||||
|
}
|
||||||
self.pq.push(event);
|
self.pq.push(event);
|
||||||
}
|
}
|
||||||
// Determine sleep deadline
|
// Determine sleep deadline
|
||||||
@@ -68,7 +84,12 @@ impl NotificationQueue {
|
|||||||
tokio::select! {
|
tokio::select! {
|
||||||
event = self.rx.recv() => {
|
event = self.rx.recv() => {
|
||||||
match event {
|
match event {
|
||||||
Some(event) => self.pq.push(event),
|
Some(event) => {
|
||||||
|
if events_debug {
|
||||||
|
log::info!("[EVENTS] drain: recv event id={}", event.id);
|
||||||
|
}
|
||||||
|
self.pq.push(event);
|
||||||
|
}
|
||||||
None => break Ok(()),
|
None => break Ok(()),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -77,7 +98,12 @@ impl NotificationQueue {
|
|||||||
while let Some(peek) = self.pq.peek() {
|
while let Some(peek) = self.pq.peek() {
|
||||||
if peek.not_before > now { break; }
|
if peek.not_before > now { break; }
|
||||||
let event = self.pq.pop().unwrap();
|
let event = self.pq.pop().unwrap();
|
||||||
self.tx.send(event).await?;
|
if events_debug {
|
||||||
|
log::info!("[EVENTS] drain: sending event id={}", event.id);
|
||||||
|
}
|
||||||
|
if let Some(ref tx) = self.tx {
|
||||||
|
tx.send(event).await?;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -109,6 +109,7 @@ pub fn resolve_external_template_refs(
|
|||||||
templates: &mut HashMap<String, NotificationTemplateDef>,
|
templates: &mut HashMap<String, NotificationTemplateDef>,
|
||||||
registry: &ResourceRegistry,
|
registry: &ResourceRegistry,
|
||||||
) {
|
) {
|
||||||
|
log::debug!("Resolving external template refs");
|
||||||
for (name, tmpl_def) in templates.iter_mut() {
|
for (name, tmpl_def) in templates.iter_mut() {
|
||||||
if tmpl_def.use_.is_none() {
|
if tmpl_def.use_.is_none() {
|
||||||
continue;
|
continue;
|
||||||
@@ -139,17 +140,11 @@ pub fn resolve_template(
|
|||||||
_standalone: bool,
|
_standalone: bool,
|
||||||
server_templates: Option<&HashMap<String, NotificationTemplateDef>>,
|
server_templates: Option<&HashMap<String, NotificationTemplateDef>>,
|
||||||
) -> Result<NotificationTemplateDef, NotifyError> {
|
) -> Result<NotificationTemplateDef, NotifyError> {
|
||||||
// 1. Try the initial schema directly
|
let mut schema_next = if schema.is_none() {
|
||||||
if let Some(schema_name) = schema.as_deref() {
|
resolve_template_next_schema(schema)
|
||||||
if let Some(template) =
|
} else {
|
||||||
resolve_template_by_schema(schema_name, method_name, templates, server_templates)
|
schema.clone()
|
||||||
{
|
};
|
||||||
return Ok(template);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// 2. Walk the fallback chain
|
|
||||||
let mut schema_next = resolve_template_next_schema(schema);
|
|
||||||
while let Some(ref schema_name) = schema_next {
|
while let Some(ref schema_name) = schema_next {
|
||||||
if let Some(template) =
|
if let Some(template) =
|
||||||
resolve_template_by_schema(schema_name, method_name, templates, server_templates)
|
resolve_template_by_schema(schema_name, method_name, templates, server_templates)
|
||||||
@@ -192,6 +187,7 @@ pub fn resolve_template_by_schema(
|
|||||||
pub fn flatten_template_refs(
|
pub fn flatten_template_refs(
|
||||||
templates: &mut HashMap<String, NotificationTemplateDef>,
|
templates: &mut HashMap<String, NotificationTemplateDef>,
|
||||||
) -> Result<(), NotifyError> {
|
) -> Result<(), NotifyError> {
|
||||||
|
log::debug!("Flattening template refs");
|
||||||
let names: Vec<String> = templates.keys().cloned().collect();
|
let names: Vec<String> = templates.keys().cloned().collect();
|
||||||
for name in names {
|
for name in names {
|
||||||
let mut visited = std::collections::HashSet::new();
|
let mut visited = std::collections::HashSet::new();
|
||||||
|
|||||||
@@ -1,4 +1,3 @@
|
|||||||
use crate::finalize::plugin::FinalizePlugin;
|
|
||||||
use crate::types::buildstatus::StageOutcome;
|
use crate::types::buildstatus::StageOutcome;
|
||||||
use crate::types::buildstatus::StagePhase;
|
use crate::types::buildstatus::StagePhase;
|
||||||
use chrono::{DateTime, Utc};
|
use chrono::{DateTime, Utc};
|
||||||
@@ -13,29 +12,21 @@ use std::time::Duration;
|
|||||||
use uuid::Uuid;
|
use uuid::Uuid;
|
||||||
|
|
||||||
use crate::finalize::NotificationTemplate;
|
use crate::finalize::NotificationTemplate;
|
||||||
|
use crate::finalize::plugin::FinalizePlugin;
|
||||||
|
|
||||||
|
pub type NotificationEventReceiver = tokio::sync::mpsc::Receiver<NotificationEvent>;
|
||||||
|
pub type NotificationEventSender = tokio::sync::mpsc::Sender<NotificationEvent>;
|
||||||
|
pub type NotificationPluginMap = HashMap<String, FinalizePlugin>;
|
||||||
|
|
||||||
/// Final runtime configuration for notification, restored and flattened from yaml
|
/// Final runtime configuration for notification, restored and flattened from yaml
|
||||||
pub struct NotificationRule {
|
pub struct NotificationRule {
|
||||||
pub enabled: bool,
|
pub enabled: bool,
|
||||||
|
|
||||||
pub trigger: Vec<ParsedTrigger>,
|
pub trigger: Vec<ParsedTrigger>,
|
||||||
|
|
||||||
pub template: NotificationTemplate,
|
pub template: NotificationTemplate,
|
||||||
|
|
||||||
/// Whether notification failure should not trigger next priority group
|
|
||||||
pub fallible: bool,
|
pub fallible: bool,
|
||||||
|
|
||||||
/// Maximum number of attempts
|
|
||||||
pub max_lives: u32,
|
pub max_lives: u32,
|
||||||
|
|
||||||
/// Timeout for notification execution
|
|
||||||
pub timeout: Duration,
|
pub timeout: Duration,
|
||||||
|
|
||||||
/// Plugin-specific configuration
|
|
||||||
pub config: serde_yaml::Value,
|
pub config: serde_yaml::Value,
|
||||||
|
|
||||||
/// Factor of exponential backoff for retry delay
|
|
||||||
// Not provided for external use
|
|
||||||
pub retry_backoff: f64,
|
pub retry_backoff: f64,
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -56,7 +47,6 @@ pub enum Matchable {
|
|||||||
impl Matchable {
|
impl Matchable {
|
||||||
fn matches(&self, stage: &StagePhase, substage: &str) -> bool {
|
fn matches(&self, stage: &StagePhase, substage: &str) -> bool {
|
||||||
let stagename = [stage.as_str(), ".", substage].concat();
|
let stagename = [stage.as_str(), ".", substage].concat();
|
||||||
|
|
||||||
match self {
|
match self {
|
||||||
Matchable::Glob(s) => s.is_match(stagename.as_str()),
|
Matchable::Glob(s) => s.is_match(stagename.as_str()),
|
||||||
Matchable::Regex(r) => r.is_match(stagename.as_str()),
|
Matchable::Regex(r) => r.is_match(stagename.as_str()),
|
||||||
@@ -69,7 +59,6 @@ impl ParsedTrigger {
|
|||||||
match self {
|
match self {
|
||||||
ParsedTrigger::OnSuccess => outcome == &StageOutcome::PipelineSuccess,
|
ParsedTrigger::OnSuccess => outcome == &StageOutcome::PipelineSuccess,
|
||||||
ParsedTrigger::OnFailure => outcome == &StageOutcome::PipelineFailure,
|
ParsedTrigger::OnFailure => outcome == &StageOutcome::PipelineFailure,
|
||||||
// TODO: Implement condition checking for OnFinishOf
|
|
||||||
ParsedTrigger::OnFinishOf(m) => m.matches(stage, substage),
|
ParsedTrigger::OnFinishOf(m) => m.matches(stage, substage),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -78,12 +67,9 @@ impl ParsedTrigger {
|
|||||||
impl NotificationRule {
|
impl NotificationRule {
|
||||||
pub fn accept(&self, result: &StageOutcome, stage: &StagePhase, substage: &str) -> bool {
|
pub fn accept(&self, result: &StageOutcome, stage: &StagePhase, substage: &str) -> bool {
|
||||||
if self.trigger.is_empty() {
|
if self.trigger.is_empty() {
|
||||||
eprintln!("[ACCEPT] trigger empty, returning true");
|
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
self.trigger
|
self.trigger.iter().any(|t| t.accept(result, stage, substage))
|
||||||
.iter()
|
|
||||||
.any(|t| t.accept(result, stage, substage))
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -113,10 +99,6 @@ impl Default for NotificationTarget {
|
|||||||
pub type Priority = u32;
|
pub type Priority = u32;
|
||||||
pub type Method = String;
|
pub type Method = String;
|
||||||
pub type NotificationRulesMap = HashMap<Priority, HashMap<Method, Vec<NotificationRule>>>;
|
pub type NotificationRulesMap = HashMap<Priority, HashMap<Method, Vec<NotificationRule>>>;
|
||||||
// pub type BundledNotificationEvent = Vec<NotificationEvent>;
|
|
||||||
pub type NotificationEventReceiver = tokio::sync::mpsc::Receiver<NotificationEvent>;
|
|
||||||
pub type NotificationEventSender = tokio::sync::mpsc::Sender<NotificationEvent>;
|
|
||||||
pub type NotificationPluginMap = HashMap<String, FinalizePlugin>;
|
|
||||||
|
|
||||||
#[derive(Debug, Clone, Serialize)]
|
#[derive(Debug, Clone, Serialize)]
|
||||||
pub struct NotificationEvent {
|
pub struct NotificationEvent {
|
||||||
@@ -127,9 +109,9 @@ pub struct NotificationEvent {
|
|||||||
pub not_before: DateTime<Utc>,
|
pub not_before: DateTime<Utc>,
|
||||||
pub priority: u32,
|
pub priority: u32,
|
||||||
pub effective_priority: u32,
|
pub effective_priority: u32,
|
||||||
pub retry_backoff: f64, // Exponential backoff factor
|
pub retry_backoff: f64,
|
||||||
pub attempted: u32, // Cumulative cross-group retry count
|
pub attempted: u32,
|
||||||
pub target: NotificationTarget, // Current audience
|
pub target: NotificationTarget,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Hash for NotificationEvent {
|
impl Hash for NotificationEvent {
|
||||||
@@ -188,23 +170,39 @@ impl Display for NotificationRenderer {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// Notification batching configuration.
|
/// Notification batching configuration.
|
||||||
/// Controls how notification events are grouped before dispatch.
|
|
||||||
#[derive(Debug, Clone)]
|
#[derive(Debug, Clone)]
|
||||||
pub struct BatchConfig {
|
pub struct BatchConfig {
|
||||||
/// Whether batching is enabled
|
|
||||||
pub enabled: bool,
|
pub enabled: bool,
|
||||||
/// Group period in seconds (the maximum wait time for a batch)
|
|
||||||
pub period: u32,
|
pub period: u32,
|
||||||
/// Group watermark threshold (the minimum number of events before batching)
|
|
||||||
pub watermark: u32,
|
pub watermark: u32,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Default for BatchConfig {
|
impl Default for BatchConfig {
|
||||||
fn default() -> Self {
|
fn default() -> Self {
|
||||||
Self {
|
Self { enabled: true, period: 30, watermark: 5 }
|
||||||
enabled: true,
|
}
|
||||||
period: 30,
|
}
|
||||||
watermark: 5,
|
|
||||||
}
|
/// Send a stage completion event to the notification queue.
|
||||||
|
pub async fn send_stage_event(
|
||||||
|
nevent_tx: &Option<NotificationEventSender>,
|
||||||
|
stage: StagePhase,
|
||||||
|
substage: &str,
|
||||||
|
outcome: StageOutcome,
|
||||||
|
) {
|
||||||
|
if let Some(tx) = nevent_tx {
|
||||||
|
let event = NotificationEvent {
|
||||||
|
id: Uuid::new_v4(),
|
||||||
|
stage,
|
||||||
|
substage: substage.to_string(),
|
||||||
|
outcome,
|
||||||
|
not_before: Utc::now(),
|
||||||
|
priority: 0,
|
||||||
|
effective_priority: 0,
|
||||||
|
retry_backoff: 1.0,
|
||||||
|
attempted: 0,
|
||||||
|
target: Default::default(),
|
||||||
|
};
|
||||||
|
let _ = tx.send(event).await;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ use crate::notify::types::{DEFAULT_NOTIFY_PARAMS, NotificationEvent, Notificatio
|
|||||||
use std::collections::HashSet;
|
use std::collections::HashSet;
|
||||||
|
|
||||||
pub fn validate(finalize: &FinalizeConfig) -> anyhow::Result<()> {
|
pub fn validate(finalize: &FinalizeConfig) -> anyhow::Result<()> {
|
||||||
|
log::debug!("Validating finalize config");
|
||||||
let validate_result = finalize.validate();
|
let validate_result = finalize.validate();
|
||||||
if let Err(errors) = validate_result {
|
if let Err(errors) = validate_result {
|
||||||
for error in &errors {
|
for error in &errors {
|
||||||
|
|||||||
@@ -22,20 +22,19 @@
|
|||||||
|
|
||||||
pub mod config;
|
pub mod config;
|
||||||
pub mod constant;
|
pub mod constant;
|
||||||
pub mod env;
|
|
||||||
pub mod error;
|
pub mod error;
|
||||||
pub mod event;
|
pub mod event;
|
||||||
pub mod security;
|
pub mod security;
|
||||||
pub mod stage;
|
pub mod stage;
|
||||||
pub mod types;
|
pub mod types;
|
||||||
|
|
||||||
|
use crate::notify::types::NotificationEventSender;
|
||||||
|
use crate::notify::types::send_stage_event;
|
||||||
|
use crate::types::buildstatus::{StageOutcome, StagePhase};
|
||||||
use crate::{
|
use crate::{
|
||||||
cli::Cli,
|
cli::Cli,
|
||||||
prebake::{error::PrebakeError, security::get_drop_after, stage::PrebakeStage},
|
prebake::{error::PrebakeError, security::get_drop_after, stage::PrebakeStage},
|
||||||
};
|
};
|
||||||
use crate::notify::types::NotificationEventSender;
|
|
||||||
use crate::types::buildstatus::{StageOutcome, StagePhase};
|
|
||||||
use uuid::Uuid;
|
|
||||||
use chrono::Utc;
|
use chrono::Utc;
|
||||||
use std::path::Path;
|
use std::path::Path;
|
||||||
use workshop_engine::{EventSender, ExecutionContext};
|
use workshop_engine::{EventSender, ExecutionContext};
|
||||||
@@ -91,7 +90,7 @@ pub fn parse(yaml_content: &str) -> Result<PrebakeConfig, serde_yaml::Error> {
|
|||||||
/// - Any stage execution fails
|
/// - Any stage execution fails
|
||||||
pub async fn prebake(
|
pub async fn prebake(
|
||||||
prebake_path: &Path,
|
prebake_path: &Path,
|
||||||
cli: &Cli,
|
_cli: &Cli,
|
||||||
stage: Option<PrebakeStage>,
|
stage: Option<PrebakeStage>,
|
||||||
ctx: &mut ExecutionContext,
|
ctx: &mut ExecutionContext,
|
||||||
event_tx: EventSender,
|
event_tx: EventSender,
|
||||||
@@ -130,19 +129,25 @@ pub async fn prebake(
|
|||||||
let _stage_start = Utc::now();
|
let _stage_start = Utc::now();
|
||||||
log::info!("Running PBStage: Bootstrap");
|
log::info!("Running PBStage: Bootstrap");
|
||||||
prebake_drop_privilege(PrebakeStage::Bootstrap, dropafter, target_user, ctx)?;
|
prebake_drop_privilege(PrebakeStage::Bootstrap, dropafter, target_user, ctx)?;
|
||||||
ctx.task_id = format!("{}-{}-bootstrap", cli.pipeline, cli.build_id);
|
ctx.task_id = format!("{}-{}-bootstrap", ctx.pipeline_name, ctx.build_id);
|
||||||
let osinfo = os_info::get();
|
let osinfo = os_info::get();
|
||||||
let result = stage::bootstrap::bootstrap(&prebake, &osinfo, ctx, &event_tx).await;
|
let result = stage::bootstrap::bootstrap(&prebake, &osinfo, ctx, &event_tx).await;
|
||||||
result?;
|
result?;
|
||||||
}
|
}
|
||||||
send_stage_event(&nevent_tx, StagePhase::Prebake, "bootstrap", StageOutcome::Success).await;
|
send_stage_event(
|
||||||
|
&nevent_tx,
|
||||||
|
StagePhase::Prebake,
|
||||||
|
"bootstrap",
|
||||||
|
StageOutcome::Success,
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
|
||||||
// EarlyHook stage
|
// EarlyHook stage
|
||||||
if stage <= PrebakeStage::EarlyHook {
|
if stage <= PrebakeStage::EarlyHook {
|
||||||
let _stage_start = Utc::now();
|
let _stage_start = Utc::now();
|
||||||
log::info!("Running PBStage: EarlyHook");
|
log::info!("Running PBStage: EarlyHook");
|
||||||
prebake_drop_privilege(PrebakeStage::EarlyHook, dropafter, target_user, ctx)?;
|
prebake_drop_privilege(PrebakeStage::EarlyHook, dropafter, target_user, ctx)?;
|
||||||
ctx.task_id = format!("{}-{}-earlyhook", cli.pipeline, cli.build_id);
|
ctx.task_id = format!("{}-{}-earlyhook", ctx.pipeline_name, ctx.build_id);
|
||||||
let earlyhook = match &prebake.hooks {
|
let earlyhook = match &prebake.hooks {
|
||||||
Some(hooks) => hooks.early.clone(),
|
Some(hooks) => hooks.early.clone(),
|
||||||
None => None,
|
None => None,
|
||||||
@@ -150,7 +155,13 @@ pub async fn prebake(
|
|||||||
let result = stage::hook::hook(earlyhook, ctx, event_tx.clone()).await;
|
let result = stage::hook::hook(earlyhook, ctx, event_tx.clone()).await;
|
||||||
result?;
|
result?;
|
||||||
}
|
}
|
||||||
send_stage_event(&nevent_tx, StagePhase::Prebake, "earlyhook", StageOutcome::Success).await;
|
send_stage_event(
|
||||||
|
&nevent_tx,
|
||||||
|
StagePhase::Prebake,
|
||||||
|
"earlyhook",
|
||||||
|
StageOutcome::Success,
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
|
||||||
if let Some(dependencies) = prebake.dependencies {
|
if let Some(dependencies) = prebake.dependencies {
|
||||||
let endpoint = dependencies
|
let endpoint = dependencies
|
||||||
@@ -217,11 +228,17 @@ pub async fn prebake(
|
|||||||
let _stage_start = Utc::now();
|
let _stage_start = Utc::now();
|
||||||
log::info!("Running PBStage: DepsSystem");
|
log::info!("Running PBStage: DepsSystem");
|
||||||
prebake_drop_privilege(PrebakeStage::DepsSystem, dropafter, target_user, ctx)?;
|
prebake_drop_privilege(PrebakeStage::DepsSystem, dropafter, target_user, ctx)?;
|
||||||
ctx.task_id = format!("{}-{}-depssystem", cli.pipeline, cli.build_id);
|
ctx.task_id = format!("{}-{}-depssystem", ctx.pipeline_name, ctx.build_id);
|
||||||
let result = stage::depssystem::depssystem(&system_config, ctx, event_tx.clone()).await;
|
let result = stage::depssystem::depssystem(&system_config, ctx, event_tx.clone()).await;
|
||||||
result?;
|
result?;
|
||||||
}
|
}
|
||||||
send_stage_event(&nevent_tx, StagePhase::Prebake, "depssystem", StageOutcome::Success).await;
|
send_stage_event(
|
||||||
|
&nevent_tx,
|
||||||
|
StagePhase::Prebake,
|
||||||
|
"depssystem",
|
||||||
|
StageOutcome::Success,
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
|
||||||
if let Some(user) = dependencies.user
|
if let Some(user) = dependencies.user
|
||||||
&& stage <= PrebakeStage::DepsUser
|
&& stage <= PrebakeStage::DepsUser
|
||||||
@@ -229,56 +246,50 @@ pub async fn prebake(
|
|||||||
let _stage_start = Utc::now();
|
let _stage_start = Utc::now();
|
||||||
log::info!("Running PBStage: DepsUser");
|
log::info!("Running PBStage: DepsUser");
|
||||||
prebake_drop_privilege(PrebakeStage::DepsUser, dropafter, target_user, ctx)?;
|
prebake_drop_privilege(PrebakeStage::DepsUser, dropafter, target_user, ctx)?;
|
||||||
ctx.task_id = format!("{}-{}-depsuser", cli.pipeline, cli.build_id);
|
ctx.task_id = format!("{}-{}-depsuser", ctx.pipeline_name, ctx.build_id);
|
||||||
let result = stage::depsuser::depsuser(&user, ctx, event_tx.clone(), &endpoint).await;
|
let result = stage::depsuser::depsuser(&user, ctx, event_tx.clone(), &endpoint).await;
|
||||||
result?;
|
result?;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
send_stage_event(&nevent_tx, StagePhase::Prebake, "depsuser", StageOutcome::Success).await;
|
send_stage_event(
|
||||||
|
&nevent_tx,
|
||||||
|
StagePhase::Prebake,
|
||||||
|
"depsuser",
|
||||||
|
StageOutcome::Success,
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
|
||||||
// LateHook stage
|
// LateHook stage
|
||||||
if stage <= PrebakeStage::LateHook {
|
if stage <= PrebakeStage::LateHook {
|
||||||
let _stage_start = Utc::now();
|
let _stage_start = Utc::now();
|
||||||
log::info!("Running PBStage: LateHook");
|
log::info!("Running PBStage: LateHook");
|
||||||
prebake_drop_privilege(PrebakeStage::LateHook, dropafter, target_user, ctx)?;
|
prebake_drop_privilege(PrebakeStage::LateHook, dropafter, target_user, ctx)?;
|
||||||
ctx.task_id = format!("{}-{}-latehook", cli.pipeline, cli.build_id);
|
ctx.task_id = format!("{}-{}-latehook", ctx.pipeline_name, ctx.build_id);
|
||||||
let latehook = match &prebake.hooks {
|
let latehook = match &prebake.hooks {
|
||||||
Some(hooks) => hooks.late.clone(),
|
Some(hooks) => hooks.late.clone(),
|
||||||
None => None,
|
None => None,
|
||||||
};
|
};
|
||||||
let result = stage::hook::hook(latehook, ctx, event_tx.clone()).await;
|
let result = stage::hook::hook(latehook, ctx, event_tx.clone()).await;
|
||||||
result?;
|
result?;
|
||||||
send_stage_event(&nevent_tx, StagePhase::Prebake, "latehook", StageOutcome::Success).await;
|
send_stage_event(
|
||||||
|
&nevent_tx,
|
||||||
|
StagePhase::Prebake,
|
||||||
|
"latehook",
|
||||||
|
StageOutcome::Success,
|
||||||
|
)
|
||||||
|
.await;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Ready stage
|
// Ready stage
|
||||||
ctx.task_id = format!("{}-{}-ready", cli.pipeline, cli.build_id);
|
ctx.task_id = format!("{}-{}-ready", ctx.pipeline_name, ctx.build_id);
|
||||||
send_stage_event(&nevent_tx, StagePhase::Prebake, "ready", StageOutcome::Success).await;
|
send_stage_event(
|
||||||
|
&nevent_tx,
|
||||||
|
StagePhase::Prebake,
|
||||||
|
"ready",
|
||||||
|
StageOutcome::Success,
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
|
||||||
log::info!("Prebake done!");
|
log::info!("Prebake done!");
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
|
||||||
|
|
||||||
/// Send a stage completion event to the notification queue.
|
|
||||||
async fn send_stage_event(
|
|
||||||
nevent_tx: &Option<NotificationEventSender>,
|
|
||||||
stage: StagePhase,
|
|
||||||
substage: &str,
|
|
||||||
outcome: StageOutcome,
|
|
||||||
) {
|
|
||||||
if let Some(tx) = nevent_tx {
|
|
||||||
let event = crate::notify::types::NotificationEvent {
|
|
||||||
id: Uuid::new_v4(),
|
|
||||||
stage,
|
|
||||||
substage: substage.to_string(),
|
|
||||||
outcome,
|
|
||||||
not_before: chrono::Utc::now(),
|
|
||||||
priority: 0,
|
|
||||||
effective_priority: 0,
|
|
||||||
retry_backoff: 1.0,
|
|
||||||
attempted: 0,
|
|
||||||
target: Default::default(),
|
|
||||||
};
|
|
||||||
let _ = tx.send(event).await;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,54 +0,0 @@
|
|||||||
# workshop-baker/src/prebake
|
|
||||||
|
|
||||||
**Generated:** 2026-04-22
|
|
||||||
**Commit:** 7b3b71a
|
|
||||||
|
|
||||||
Build environment setup: bootstrap, dependencies, security, hooks.
|
|
||||||
|
|
||||||
## STRUCTURE
|
|
||||||
|
|
||||||
```
|
|
||||||
prebake/
|
|
||||||
├── config.rs # PrebakeConfig YAML schema (environment, deps, cache)
|
|
||||||
├── stage.rs # PrebakeStage enum (Bootstrap→EarlyHook→DepsSystem→DepsUser→LateHook→Ready)
|
|
||||||
├── stage/
|
|
||||||
│ ├── bootstrap.rs # User creation (vulcan), sudo/doas setup
|
|
||||||
│ ├── environment.rs # Env var injection
|
|
||||||
│ ├── depssystem.rs # System package installation
|
|
||||||
│ ├── depsuser.rs # User package installation (cargo, npm, etc.)
|
|
||||||
│ └── hook.rs # Hook execution (pre/post stage)
|
|
||||||
├── env/ # Environment providers
|
|
||||||
│ ├── mod.rs # EnvProvider trait
|
|
||||||
│ ├── container.rs # Docker/container support
|
|
||||||
│ ├── firecracker.rs # Firecracker microVM
|
|
||||||
│ ├── baremetal.rs # Direct host execution
|
|
||||||
│ └── custom.rs # Custom env (todo!())
|
|
||||||
├── security.rs # Privilege dropping (unsafe libc calls)
|
|
||||||
├── event.rs # Event logging
|
|
||||||
├── error.rs # PrebakeError + BootstrapError
|
|
||||||
├── types.rs # Module re-exports
|
|
||||||
├── types/ # memsize, cache, builderconfig, architecture
|
|
||||||
├── constant.rs # Module constants
|
|
||||||
└── env.rs # Environment module root
|
|
||||||
```
|
|
||||||
|
|
||||||
## WHERE TO LOOK
|
|
||||||
|
|
||||||
| Task | Location |
|
|
||||||
|------|----------|
|
|
||||||
| Config loading | `config.rs:PrebakeConfig::from_yaml()` |
|
|
||||||
| Stage ordering | `stage.rs:PrebakeStage` enum |
|
|
||||||
| User setup | `stage/bootstrap.rs` - vulcan user creation |
|
|
||||||
| Security | `security.rs:72` - privilege dropping |
|
|
||||||
| Hooks | `stage/hook.rs` - pre/post stage hooks |
|
|
||||||
|
|
||||||
## CONVENTIONS
|
|
||||||
- **PrebakeStage**: Ordered enum Bootstrap→EarlyHook→DepsSystem→DepsUser→LateHook→Ready
|
|
||||||
- **ExecutionContext**: Carries task_id, username, env_vars through all stages
|
|
||||||
- **Target User**: "vulcan" (hardcoded in bootstrap.rs:6-24)
|
|
||||||
|
|
||||||
## ANTI-PATTERNS
|
|
||||||
1. **Hardcoded user** — "vulcan" username hardcoded throughout
|
|
||||||
2. **Unsafe blocks** — `security.rs` has 5+ unsafe libc calls
|
|
||||||
3. **Incomplete** — custom.rs and apt.rs have `todo!()` stubs
|
|
||||||
4. **doas.conf** — bootstrap.rs:183: "We do not know how to write doas.conf lol"
|
|
||||||
@@ -1,40 +1,68 @@
|
|||||||
use workshop_engine::{ExecutionEvent, StreamType};
|
use workshop_engine::{ExecutionEvent, StreamType};
|
||||||
|
|
||||||
|
use crate::types::debug::DebugFeature;
|
||||||
|
use crate::bake::constant::PIPELINE_SKIP_ERRORCODE;
|
||||||
|
|
||||||
pub async fn event_receiver(
|
pub async fn event_receiver(
|
||||||
mut rx: tokio::sync::mpsc::Receiver<ExecutionEvent>,
|
mut rx: tokio::sync::mpsc::Receiver<ExecutionEvent>,
|
||||||
pipeline_name: String,
|
pipeline_name: String,
|
||||||
build_id: String,
|
build_id: String,
|
||||||
|
debug_flags: u32,
|
||||||
) {
|
) {
|
||||||
|
let events_debug = DebugFeature::from_bits_retain(debug_flags).contains(DebugFeature::Events);
|
||||||
while let Some(event) = rx.recv().await {
|
while let Some(event) = rx.recv().await {
|
||||||
match event {
|
match event {
|
||||||
ExecutionEvent::TaskStarted { task_id, timestamp } => {
|
ExecutionEvent::TaskStarted { task_id, timestamp: _ } => {
|
||||||
log::debug!(
|
if events_debug {
|
||||||
"[{}] [{}] [{}] Task started: {}",
|
log::info!(
|
||||||
pipeline_name,
|
"[{}] [{}] Task started: {}",
|
||||||
build_id,
|
pipeline_name,
|
||||||
timestamp,
|
build_id,
|
||||||
task_id
|
task_id
|
||||||
);
|
);
|
||||||
|
} else {
|
||||||
|
log::debug!(
|
||||||
|
"[{}] [{}] Task started: {}",
|
||||||
|
pipeline_name,
|
||||||
|
build_id,
|
||||||
|
task_id
|
||||||
|
);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
ExecutionEvent::OutputChunk {
|
ExecutionEvent::OutputChunk {
|
||||||
task_id,
|
task_id: _,
|
||||||
stream,
|
stream,
|
||||||
data,
|
data,
|
||||||
} => match stream {
|
} => match stream {
|
||||||
StreamType::Stdout => print!("[{}] [{}] [stdout] {}", pipeline_name, task_id, data),
|
StreamType::Stdout => print!("{}", data),
|
||||||
StreamType::Stderr => {
|
StreamType::Stderr => {
|
||||||
eprint!("[{}] [{}] [stderr] {}", pipeline_name, task_id, data)
|
eprint!("{}", data)
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
ExecutionEvent::TaskCompleted { task_id, result } => {
|
ExecutionEvent::TaskCompleted { task_id, result } => {
|
||||||
log::debug!(
|
if result.success {
|
||||||
"[{}] [{}] Task completed: {} (exit_code={}, duration={:?})",
|
if events_debug {
|
||||||
pipeline_name,
|
log::info!(
|
||||||
build_id,
|
"[{}] [{}] Task completed: {} (exit_code={}, duration={:?})",
|
||||||
task_id,
|
pipeline_name, build_id, task_id,
|
||||||
result.exit_code,
|
result.exit_code, result.duration,
|
||||||
result.duration
|
);
|
||||||
);
|
} else {
|
||||||
|
log::debug!(
|
||||||
|
"[{}] [{}] Task completed: {} (exit_code={}, duration={:?})",
|
||||||
|
pipeline_name, build_id, task_id,
|
||||||
|
result.exit_code, result.duration,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
} else if result.exit_code == PIPELINE_SKIP_ERRORCODE {
|
||||||
|
log::info!("[{}] [{}] Task skipped: {}", pipeline_name, build_id, task_id);
|
||||||
|
} else {
|
||||||
|
log::warn!(
|
||||||
|
"[{}] [{}] Task completed with non-zero exitcode: {} (exit_code={}, duration={:?})",
|
||||||
|
pipeline_name, build_id, task_id,
|
||||||
|
result.exit_code, result.duration,
|
||||||
|
);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
ExecutionEvent::TaskFailed { task_id, error } => {
|
ExecutionEvent::TaskFailed { task_id, error } => {
|
||||||
log::error!(
|
log::error!(
|
||||||
|
|||||||
@@ -113,7 +113,7 @@ pub fn lookup_uid_by_name(username: &str) -> Result<u32, PrebakeError> {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TODO: originally unsafe {pwent} here, maybe to check and remove
|
// NOTE: originally unsafe {pwent} here
|
||||||
// let pwent = pwent ;
|
// let pwent = pwent ;
|
||||||
if pwent.is_null() {
|
if pwent.is_null() {
|
||||||
if let Ok(uid) = username.parse::<u32>() {
|
if let Ok(uid) = username.parse::<u32>() {
|
||||||
@@ -231,6 +231,9 @@ pub fn prebake_drop_privilege(
|
|||||||
user
|
user
|
||||||
);
|
);
|
||||||
ctx.privileged = false;
|
ctx.privileged = false;
|
||||||
|
let home = format!("/home/{}", user);
|
||||||
|
//unsafe { std::env::set_var("HOME", &home); }
|
||||||
|
ctx.env_vars.insert("HOME".to_string(), home);
|
||||||
}
|
}
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -54,58 +54,56 @@ pub async fn bootstrap(
|
|||||||
let bootstrap_config = config.bootstrap.clone().unwrap_or_default();
|
let bootstrap_config = config.bootstrap.clone().unwrap_or_default();
|
||||||
|
|
||||||
if let Some(raw) = bootstrap_config.custom {
|
if let Some(raw) = bootstrap_config.custom {
|
||||||
match BootstrapSource::parse(&raw) {
|
// Workshop-specific schemes requiring bakerd infrastructure
|
||||||
BootstrapSource::Inline(script) => {
|
if let Some(rest) = raw.strip_prefix("workspace://") {
|
||||||
if ctx.dry_run {
|
log::info!("Custom bootstrap from workspace: {}", rest);
|
||||||
log::info!(
|
unimplemented!("Workspace-relative bootstrap requires bakerd's workspace manager");
|
||||||
"[DRY_RUN] User provided inline bootstrap script:\n{}",
|
} else if let Some(name) = raw.strip_prefix("server://") {
|
||||||
script
|
log::info!("Custom bootstrap from server: {}", name);
|
||||||
);
|
unimplemented!("Server-provided bootstrap requires bakerd's script registry");
|
||||||
return Ok(());
|
}
|
||||||
|
// Use getterURL for standard URL schemes: http, https, file, git, ...
|
||||||
|
else if workshop_getterurl::GetterUrl::parse(&raw).is_ok() {
|
||||||
|
log::info!("Fetching custom bootstrap via getterURL: {}", raw);
|
||||||
|
if ctx.standalone {
|
||||||
|
let temp_dir = tempfile::tempdir().map_err(BootstrapError::IOError)?;
|
||||||
|
let fetched = workshop_getterurl::fetch(&raw, temp_dir.path())
|
||||||
|
.await
|
||||||
|
.map_err(|e| BootstrapError::InvalidConfig(e.to_string()))?;
|
||||||
|
if fetched.is_dir() {
|
||||||
|
return Err(BootstrapError::InvalidConfig(
|
||||||
|
format!("getterURL returned a directory, expected a single file: {}", fetched.display())
|
||||||
|
));
|
||||||
}
|
}
|
||||||
log::debug!("Using inline bootstrap script ({} bytes)", script.len());
|
let content = tokio::fs::read_to_string(&fetched)
|
||||||
if ctx.standalone {
|
|
||||||
write_bootstrap_content(&bootstrap_path, script)?;
|
|
||||||
} else {
|
|
||||||
// Requires PIP(Pipeline Implicit Parameters) to be implemented
|
|
||||||
unimplemented!("Permission check is not implemented, failing...");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
BootstrapSource::Workspace(relpath) => {
|
|
||||||
// Project-relative path, resolved by bakerd against the project root.
|
|
||||||
// Routing path relpath through bakerd's workspace manager ensures
|
|
||||||
// proper sandboxing and access control.
|
|
||||||
log::info!("Custom bootstrap from workspace: {}", relpath);
|
|
||||||
unimplemented!("Workspace-relative bootstrap requires bakerd's workspace manager");
|
|
||||||
}
|
|
||||||
BootstrapSource::Server(name) => {
|
|
||||||
// Script provided by the bake server (bakerd), keyed by name.
|
|
||||||
// The server maintains a registry of trusted bootstrap scripts.
|
|
||||||
log::info!("Custom bootstrap from server: {}", name);
|
|
||||||
unimplemented!("Server-provided bootstrap requires bakerd's script registry");
|
|
||||||
}
|
|
||||||
BootstrapSource::Http(url) => {
|
|
||||||
// HTTP download, similar to finalize::plugin::fetch::fetch_http_plugin.
|
|
||||||
// Should support: checksum verification, retry, timeout.
|
|
||||||
log::info!("Custom bootstrap from URL: {}", url);
|
|
||||||
log::info!(" Target: {}", bootstrap_path.display());
|
|
||||||
unimplemented!(
|
|
||||||
"HTTP bootstrap fetching requires bakerd-level network infrastructure"
|
|
||||||
);
|
|
||||||
}
|
|
||||||
BootstrapSource::File(path) => {
|
|
||||||
// Local file copy. This is the simplest case — we can implement
|
|
||||||
// this directly without bakerd infrastructure.
|
|
||||||
log::info!("Custom bootstrap from local file: {}", path);
|
|
||||||
tokio::fs::copy(&path, &bootstrap_path)
|
|
||||||
.await
|
.await
|
||||||
.map_err(BootstrapError::IOError)?;
|
.map_err(BootstrapError::IOError)?;
|
||||||
|
if ctx.dry_run {
|
||||||
|
log::debug!("[DRY_RUN] Fetched bootstrap script:\n{}", content);
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
write_bootstrap_content(&bootstrap_path, content)?;
|
||||||
|
} else {
|
||||||
|
unimplemented!("Permission check is not implemented, failing...");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Inline content (not a valid getterURL pattern)
|
||||||
|
else {
|
||||||
|
if ctx.dry_run {
|
||||||
|
log::debug!("[DRY_RUN] User provided inline bootstrap script:\n{}", raw);
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
log::debug!("Using inline bootstrap script ({} bytes)", raw.len());
|
||||||
|
if ctx.standalone {
|
||||||
|
write_bootstrap_content(&bootstrap_path, raw.to_string())?;
|
||||||
|
} else {
|
||||||
|
unimplemented!("Permission check is not implemented, failing...");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
let script = generate_bootstrap(&bootstrap_config, config, osinfo, ctx.standalone)?;
|
let script = generate_bootstrap(&bootstrap_config, config, osinfo, ctx.standalone)?;
|
||||||
if ctx.dry_run {
|
if ctx.dry_run {
|
||||||
log::info!("[DRY_RUN] Generated bootstrap script:\n{}", script);
|
log::debug!("[DRY_RUN] Generated bootstrap script:\n{}", script);
|
||||||
return Ok(());
|
return Ok(());
|
||||||
}
|
}
|
||||||
log::debug!("Generated bootstrap script:\n{}", script);
|
log::debug!("Generated bootstrap script:\n{}", script);
|
||||||
@@ -135,53 +133,6 @@ fn write_bootstrap_content(bootstrap_path: &Path, content: String) -> Result<(),
|
|||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Source type for a custom bootstrap script.
|
|
||||||
///
|
|
||||||
/// Mirrors the scheme-based resolution in `finalize::plugin::fetch` URL parsing,
|
|
||||||
/// and supports the schemes defined in the prebake.yml.tmpl spec:
|
|
||||||
/// `workspace://`, `file://`, `server://`, `https?://`, plus inline content.
|
|
||||||
enum BootstrapSource {
|
|
||||||
/// Inline script content (no scheme match).
|
|
||||||
Inline(String),
|
|
||||||
/// Project-relative file: `workspace://path/to/bootstrap.sh`
|
|
||||||
Workspace(String),
|
|
||||||
/// Script provided by bake server: `server://script-name`
|
|
||||||
Server(String),
|
|
||||||
/// HTTP(S) download: `https://example.com/bootstrap.sh`
|
|
||||||
Http(String),
|
|
||||||
/// Local file path: `file:///bin/bootstrap.sh`
|
|
||||||
File(String),
|
|
||||||
}
|
|
||||||
|
|
||||||
impl BootstrapSource {
|
|
||||||
/// Parses a raw custom bootstrap string into its source type.
|
|
||||||
///
|
|
||||||
/// Scheme detection follows the template spec:
|
|
||||||
/// - `workspace://` → Workspace
|
|
||||||
/// - `file://` → File (local path)
|
|
||||||
/// - `server://` → Server (bakerd registry)
|
|
||||||
/// - `http://`/`https://`→ Http
|
|
||||||
/// - No recognized scheme → Inline
|
|
||||||
fn parse(raw: &str) -> Self {
|
|
||||||
if let Some(rest) = raw.strip_prefix("workspace://") {
|
|
||||||
return BootstrapSource::Workspace(rest.to_string());
|
|
||||||
}
|
|
||||||
if let Some(path) = raw.strip_prefix("file://") {
|
|
||||||
return BootstrapSource::File(path.to_string());
|
|
||||||
}
|
|
||||||
if let Some(name) = raw.strip_prefix("server://") {
|
|
||||||
return BootstrapSource::Server(name.to_string());
|
|
||||||
}
|
|
||||||
if let Some(url) = raw
|
|
||||||
.strip_prefix("https://")
|
|
||||||
.or_else(|| raw.strip_prefix("http://"))
|
|
||||||
{
|
|
||||||
return BootstrapSource::Http(url.to_string());
|
|
||||||
}
|
|
||||||
BootstrapSource::Inline(raw.to_string())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Generates a bootstrap script based on configuration and target OS.
|
/// Generates a bootstrap script based on configuration and target OS.
|
||||||
///
|
///
|
||||||
/// The generated script performs the following setup tasks:
|
/// The generated script performs the following setup tasks:
|
||||||
|
|||||||
@@ -57,7 +57,6 @@ pub async fn hook(
|
|||||||
ctx: &ExecutionContext,
|
ctx: &ExecutionContext,
|
||||||
event_tx: EventSender,
|
event_tx: EventSender,
|
||||||
) -> Result<(), ExecutionError> {
|
) -> Result<(), ExecutionError> {
|
||||||
// dbg!(ctx);
|
|
||||||
let hook = match hook {
|
let hook = match hook {
|
||||||
Some(h) => h,
|
Some(h) => h,
|
||||||
None => {
|
None => {
|
||||||
@@ -79,7 +78,6 @@ pub async fn hook(
|
|||||||
let result = engine
|
let result = engine
|
||||||
.execute_command_with_delta(&hook_cmd.command, ctx, &event_tx, &deltactx)
|
.execute_command_with_delta(&hook_cmd.command, ctx, &event_tx, &deltactx)
|
||||||
.await;
|
.await;
|
||||||
// dbg!(&result);
|
|
||||||
if let Err(e) = result {
|
if let Err(e) = result {
|
||||||
log::error!("Hook {} failed: {:?}", index, e);
|
log::error!("Hook {} failed: {:?}", index, e);
|
||||||
return Err(e);
|
return Err(e);
|
||||||
|
|||||||
@@ -0,0 +1,206 @@
|
|||||||
|
//! Resource prefetching for standalone mode.
|
||||||
|
//!
|
||||||
|
//! Populates a [`ResourceRegistry`] by fetching plugins and templates
|
||||||
|
//! from their external URLs **before** the pipeline runs.
|
||||||
|
//!
|
||||||
|
//! In worker mode this is the baker's responsibility; in daemon mode
|
||||||
|
//! the bakerd would call the same lower-level primitives (or replace
|
||||||
|
//! this module entirely).
|
||||||
|
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
|
||||||
|
use crate::finalize::{FinalizeConfig, NotificationTemplateDef, PluginConfig, RawPluginMap};
|
||||||
|
use crate::types::resource::ResourceRegistry;
|
||||||
|
use workshop_getterurl;
|
||||||
|
|
||||||
|
/// Errors raised during resource prefetching.
|
||||||
|
#[derive(Debug, thiserror::Error)]
|
||||||
|
pub enum PrefetchError {
|
||||||
|
#[error("failed to construct resource URL for plugin '{name}': {detail}")]
|
||||||
|
UrlConstruction { name: String, detail: String },
|
||||||
|
|
||||||
|
#[error("failed to fetch '{url}': {source}")]
|
||||||
|
Fetch {
|
||||||
|
url: String,
|
||||||
|
#[source]
|
||||||
|
source: workshop_getterurl::GetterError,
|
||||||
|
},
|
||||||
|
|
||||||
|
#[error("file not found after resolving workspace path: {0}")]
|
||||||
|
FileNotFound(String),
|
||||||
|
|
||||||
|
#[error(transparent)]
|
||||||
|
Io(#[from] std::io::Error),
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Fetch all external plugins and templates defined in [`FinalizeConfig`]
|
||||||
|
/// and register them into the given [`ResourceRegistry`].
|
||||||
|
///
|
||||||
|
/// * Plugins with the `late_binding` flag are **skipped** — they are
|
||||||
|
/// expected to be produced by the bake stage itself.
|
||||||
|
/// * Templates with a `use_` field (external URL) are fetched and
|
||||||
|
/// registered; inline templates are left as-is.
|
||||||
|
pub async fn prefetch_resources(
|
||||||
|
config: &FinalizeConfig,
|
||||||
|
registry: &mut ResourceRegistry,
|
||||||
|
) -> Result<(), PrefetchError> {
|
||||||
|
let _dir = tempfile::tempdir()?;
|
||||||
|
let target_base = _dir.path().to_path_buf();
|
||||||
|
|
||||||
|
prefetch_plugins(&config.plugin, registry, &target_base).await?;
|
||||||
|
prefetch_templates(&config.notification.templates, registry, &target_base).await?;
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── plugins ──────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
async fn prefetch_plugins(
|
||||||
|
plugins: &RawPluginMap,
|
||||||
|
registry: &mut ResourceRegistry,
|
||||||
|
cache: &Path,
|
||||||
|
) -> Result<(), PrefetchError> {
|
||||||
|
for (name, sources) in plugins {
|
||||||
|
let plugin_cfg = sources.values().next().ok_or_else(|| {
|
||||||
|
PrefetchError::UrlConstruction {
|
||||||
|
name: name.clone(),
|
||||||
|
detail: "no source URL configured".into(),
|
||||||
|
}
|
||||||
|
})?;
|
||||||
|
|
||||||
|
if plugin_cfg.late_binding {
|
||||||
|
log::debug!("[prefetch] Skipping late-binding plugin '{}'", name);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
let source_url = sources.keys().next().unwrap(); // the URL string
|
||||||
|
let dest = fetch_plugin_resource(source_url, plugin_cfg, cache).await?;
|
||||||
|
registry.register(name, dest);
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Build a full getter-url from a plugin source string and its associated
|
||||||
|
/// config, fetch it, and return the local path.
|
||||||
|
async fn fetch_plugin_resource(
|
||||||
|
source: &str,
|
||||||
|
cfg: &PluginConfig,
|
||||||
|
cache: &Path,
|
||||||
|
) -> Result<PathBuf, PrefetchError> {
|
||||||
|
// file:// — already local, just resolve and register
|
||||||
|
if let Some(path_str) = source.strip_prefix("file://") {
|
||||||
|
let abs = std::env::current_dir()?.join(path_str);
|
||||||
|
if !abs.exists() {
|
||||||
|
return Err(PrefetchError::FileNotFound(abs.display().to_string()));
|
||||||
|
}
|
||||||
|
return Ok(abs);
|
||||||
|
}
|
||||||
|
|
||||||
|
// workspace:// — resolve relative to cwd then treat as file
|
||||||
|
if let Some(rel) = source.strip_prefix("workspace://") {
|
||||||
|
let abs = std::env::current_dir()?.join(rel);
|
||||||
|
if !abs.exists() {
|
||||||
|
return Err(PrefetchError::FileNotFound(abs.display().to_string()));
|
||||||
|
}
|
||||||
|
return Ok(abs);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Remote URL — add query params and fetch via getterurl
|
||||||
|
let url = enrich_url(source, cfg).map_err(|e| PrefetchError::UrlConstruction {
|
||||||
|
name: String::new(),
|
||||||
|
detail: e,
|
||||||
|
})?;
|
||||||
|
|
||||||
|
workshop_getterurl::fetch(&url, cache)
|
||||||
|
.await
|
||||||
|
.map_err(|source| PrefetchError::Fetch {
|
||||||
|
url: url.clone(),
|
||||||
|
source,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── templates ────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
async fn prefetch_templates(
|
||||||
|
templates: &std::collections::HashMap<String, NotificationTemplateDef>,
|
||||||
|
registry: &mut ResourceRegistry,
|
||||||
|
cache: &Path,
|
||||||
|
) -> Result<(), PrefetchError> {
|
||||||
|
for (name, tmpl) in templates {
|
||||||
|
let Some(ref url) = tmpl.use_ else {
|
||||||
|
continue; // inline template, no fetch needed
|
||||||
|
};
|
||||||
|
|
||||||
|
// file:// / workspace:// — resolve locally
|
||||||
|
if let Some(path_str) = url.strip_prefix("file://") {
|
||||||
|
let abs = std::env::current_dir()?.join(path_str);
|
||||||
|
if abs.exists() {
|
||||||
|
registry.register(name, abs);
|
||||||
|
} else {
|
||||||
|
log::warn!("[prefetch] template '{}' file not found: {}", name, abs.display());
|
||||||
|
}
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if let Some(rel) = url.strip_prefix("workspace://") {
|
||||||
|
let abs = std::env::current_dir()?.join(rel);
|
||||||
|
if abs.exists() {
|
||||||
|
registry.register(name, abs);
|
||||||
|
} else {
|
||||||
|
log::warn!("[prefetch] template '{}' workspace file not found: {}", name, abs.display());
|
||||||
|
}
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Remote template URL
|
||||||
|
match workshop_getterurl::fetch(url, cache).await {
|
||||||
|
Ok(path) => registry.register(name, path),
|
||||||
|
Err(e) => log::warn!("[prefetch] failed to fetch template '{}': {}", name, e),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── helpers ──────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// Append `?checksum=&depth=` to a plugin source URL when the
|
||||||
|
/// corresponding [`PluginConfig`] fields are set.
|
||||||
|
fn enrich_url(source: &str, cfg: &PluginConfig) -> Result<String, String> {
|
||||||
|
// Validate the URL is parseable first
|
||||||
|
url::Url::parse(source).map_err(|e| format!("invalid URL '{}': {}", source, e))?;
|
||||||
|
|
||||||
|
let mut result = source.to_string();
|
||||||
|
let mut sep = if source.contains('?') { "&" } else { "?" };
|
||||||
|
|
||||||
|
if let Some(ref cs) = cfg.checksum {
|
||||||
|
result.push_str(sep);
|
||||||
|
result.push_str(&format!("checksum={}", cs));
|
||||||
|
sep = "&";
|
||||||
|
}
|
||||||
|
if let Some(true) = cfg.shallow {
|
||||||
|
result.push_str(sep);
|
||||||
|
result.push_str("depth=1");
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(result)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_enrich_url_checksum() {
|
||||||
|
let cfg = PluginConfig {
|
||||||
|
fallible: false,
|
||||||
|
retry: 0,
|
||||||
|
timeout_ms: 0,
|
||||||
|
late_binding: false,
|
||||||
|
checksum: Some("sha256:abc".into()),
|
||||||
|
shallow: None,
|
||||||
|
runtime: Default::default(),
|
||||||
|
config: serde_yaml::Value::Mapping(Default::default()),
|
||||||
|
};
|
||||||
|
let url = enrich_url("https://example.com/plugin.tar.gz", &cfg).unwrap();
|
||||||
|
assert_eq!(url, "https://example.com/plugin.tar.gz?checksum=sha256:abc");
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,2 +1,3 @@
|
|||||||
pub mod buildstatus;
|
pub mod buildstatus;
|
||||||
pub mod resource;
|
pub mod resource;
|
||||||
|
pub mod debug;
|
||||||
|
|||||||
@@ -0,0 +1,176 @@
|
|||||||
|
// Code in this module PARTIALLY or FULLY utilized AI Coding Agent
|
||||||
|
//! Debug mode feature flags for the HoneyBiscuitWorkshop pipeline.
|
||||||
|
//!
|
||||||
|
//! These flags are controlled by `--debug <FEATURES>` (comma-separated)
|
||||||
|
//! or the `HBW_DEBUG` environment variable. Each flag enables specific
|
||||||
|
//! behavioral changes that aid development and debugging.
|
||||||
|
//!
|
||||||
|
//! Unlike `RUST_LOG` (which controls log visibility), `DebugFeature`
|
||||||
|
//! controls **program behavior** — replacing real notifications with
|
||||||
|
//! dummy plugins, tracing internal event flow, etc.
|
||||||
|
|
||||||
|
use bitflags::bitflags;
|
||||||
|
|
||||||
|
bitflags! {
|
||||||
|
/// Bitmask of debug features enabled at runtime.
|
||||||
|
///
|
||||||
|
/// Propagated through [`ExecutionContext`](workshop_engine::ExecutionContext)
|
||||||
|
/// so all pipeline stages can conditionally alter their behavior.
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
|
||||||
|
pub struct DebugFeature: u32 {
|
||||||
|
/// Notification debug: replace real adapters with `Dummy` (no emails/webhooks).
|
||||||
|
const NotifyDummy = 1 << 0;
|
||||||
|
|
||||||
|
/// Events: trace all `ExecutionEvent` (process lifecycle) and
|
||||||
|
/// `NotificationEvent` (stage outcomes) through the system.
|
||||||
|
const Events = 1 << 1;
|
||||||
|
|
||||||
|
/// Scheduler: log DAG topology, batch selection, and scheduling
|
||||||
|
/// decisions at `info!` level (normally `debug!`/gated).
|
||||||
|
const Scheduler = 1 << 2;
|
||||||
|
|
||||||
|
/// Script: display parsed script functions, generated final
|
||||||
|
/// scripts, and combinator chain details before execution.
|
||||||
|
const Script = 1 << 3;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl DebugFeature {
|
||||||
|
/// Parse a comma-separated `--debug` string into a `DebugFeature` bitmask.
|
||||||
|
///
|
||||||
|
/// Unknown tokens are silently ignored. `"all"` enables all known features.
|
||||||
|
///
|
||||||
|
/// # Examples
|
||||||
|
///
|
||||||
|
/// ```
|
||||||
|
/// use workshop_baker::types::debug::DebugFeature;
|
||||||
|
///
|
||||||
|
/// let flags = DebugFeature::parse("notify.dummy,events");
|
||||||
|
/// assert!(flags.contains(DebugFeature::NotifyDummy));
|
||||||
|
/// assert!(flags.contains(DebugFeature::Events));
|
||||||
|
/// assert!(!flags.contains(DebugFeature::Scheduler));
|
||||||
|
///
|
||||||
|
/// let all = DebugFeature::parse("all");
|
||||||
|
/// assert!(all.contains(DebugFeature::all()));
|
||||||
|
/// ```
|
||||||
|
pub fn _parse(raw: &str) -> Self {
|
||||||
|
if raw.contains("all") {
|
||||||
|
return Self::all();
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut flags = Self::empty();
|
||||||
|
for token in raw.split(',') {
|
||||||
|
let token = token.trim();
|
||||||
|
match token {
|
||||||
|
"notify.dummy" => flags.insert(Self::NotifyDummy),
|
||||||
|
"events" => flags.insert(Self::Events),
|
||||||
|
"scheduler" => flags.insert(Self::Scheduler),
|
||||||
|
"script" => flags.insert(Self::Script),
|
||||||
|
"" => {}
|
||||||
|
unknown => log::warn!("Unknown debug feature: '{}'", unknown),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
flags
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Print help for `--debug` features and return `true` if `help` was requested.
|
||||||
|
///
|
||||||
|
/// Called before `env_logger::init()`, so output goes to stdout directly.
|
||||||
|
/// The caller should exit immediately when this returns `true`.
|
||||||
|
pub fn print_help(raw: &str) -> bool {
|
||||||
|
if raw.trim() != "help" {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
println!("Debug features (comma-separated, use with --debug <FEATURES>):");
|
||||||
|
println!();
|
||||||
|
println!(" notify.dummy Replace real notification adapters with Dummy");
|
||||||
|
println!(" (no emails/webhooks sent during debug)");
|
||||||
|
println!(" events Trace all ExecutionEvent and NotificationEvent");
|
||||||
|
println!(" through the event queue at info log level");
|
||||||
|
println!(" scheduler Show DAG topology, batch selection, and scheduling");
|
||||||
|
println!(" decisions each tick");
|
||||||
|
println!(" script Display parsed bake.sh functions, generated final");
|
||||||
|
println!(" scripts, and combinator chain details");
|
||||||
|
println!(" all Enable all features above");
|
||||||
|
println!();
|
||||||
|
println!("Examples:");
|
||||||
|
println!(" workshop-baker --debug notify.dummy,events ...");
|
||||||
|
println!(" workshop-baker --debug all ...");
|
||||||
|
println!(" HBW_DEBUG=scheduler,script workshop-baker ...");
|
||||||
|
true
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Parse from an `Option<&str>`, returning empty flags for `None`.
|
||||||
|
pub fn parse(raw: Option<&str>) -> Self {
|
||||||
|
match raw {
|
||||||
|
Some(s) if !s.is_empty() => Self::_parse(s),
|
||||||
|
_ => Self::empty(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn parse_empty() {
|
||||||
|
assert_eq!(DebugFeature::_parse(""), DebugFeature::empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn parse_single() {
|
||||||
|
let f = DebugFeature::_parse("notify.dummy");
|
||||||
|
assert!(f.contains(DebugFeature::NotifyDummy));
|
||||||
|
assert!(!f.contains(DebugFeature::Events));
|
||||||
|
assert!(!f.contains(DebugFeature::Scheduler));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn parse_multiple() {
|
||||||
|
let f = DebugFeature::_parse("notify.dummy,events");
|
||||||
|
assert!(f.contains(DebugFeature::NotifyDummy));
|
||||||
|
assert!(f.contains(DebugFeature::Events));
|
||||||
|
assert!(!f.contains(DebugFeature::Scheduler));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn parse_all() {
|
||||||
|
let f = DebugFeature::_parse("all");
|
||||||
|
assert!(f.contains(DebugFeature::NotifyDummy));
|
||||||
|
assert!(f.contains(DebugFeature::Events));
|
||||||
|
assert!(f.contains(DebugFeature::Scheduler));
|
||||||
|
assert!(f.contains(DebugFeature::Script));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn parse_whitespace() {
|
||||||
|
let f = DebugFeature::_parse("notify.dummy , scheduler");
|
||||||
|
assert!(f.contains(DebugFeature::NotifyDummy));
|
||||||
|
assert!(f.contains(DebugFeature::Scheduler));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn parse_unknown_is_ignored() {
|
||||||
|
let f = DebugFeature::_parse("notify.dummy,foobar,events");
|
||||||
|
assert!(f.contains(DebugFeature::NotifyDummy));
|
||||||
|
assert!(f.contains(DebugFeature::Events));
|
||||||
|
// unknown "foobar" just logged and skipped
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn parse_optional_none() {
|
||||||
|
assert_eq!(DebugFeature::parse(None), DebugFeature::empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn parse_optional_empty_str() {
|
||||||
|
assert_eq!(DebugFeature::parse(Some("")), DebugFeature::empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn parse_optional_some() {
|
||||||
|
let f = DebugFeature::parse(Some("events"));
|
||||||
|
assert!(f.contains(DebugFeature::Events));
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -6,15 +6,15 @@
|
|||||||
//! exclusively through `resolve()` — the registry is treated as read-only
|
//! exclusively through `resolve()` — the registry is treated as read-only
|
||||||
//! once the pipeline is running.
|
//! once the pipeline is running.
|
||||||
|
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
use std::collections::HashMap;
|
use std::collections::HashMap;
|
||||||
use std::path::PathBuf;
|
use std::path::PathBuf;
|
||||||
|
|
||||||
/// A locally resolved resource with a logical name and filesystem path.
|
/// A locally resolved resource with a logical name and filesystem path.
|
||||||
///
|
///
|
||||||
/// Created by the resource fetcher (bakerd in worker mode, or the baker
|
/// Created by the resource fetcher (bakerd in worker mode, or the baker
|
||||||
/// itself in standalone mode) and consumed by pipeline stages via the
|
/// itself in standalone mode) and consumed by pipeline stages via the
|
||||||
/// [`ResourceRegistry`].
|
/// [`ResourceRegistry`].
|
||||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||||
pub struct Resource {
|
pub struct Resource {
|
||||||
/// Logical name used for lookup (e.g., `"plugin/docker-push"`).
|
/// Logical name used for lookup (e.g., `"plugin/docker-push"`).
|
||||||
pub name: String,
|
pub name: String,
|
||||||
@@ -36,7 +36,7 @@ impl Resource {
|
|||||||
///
|
///
|
||||||
/// Populated during the fetch phase (before prebake) and then treated as
|
/// Populated during the fetch phase (before prebake) and then treated as
|
||||||
/// immutable for the remainder of the pipeline.
|
/// immutable for the remainder of the pipeline.
|
||||||
#[derive(Debug, Clone, Default)]
|
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
|
||||||
pub struct ResourceRegistry {
|
pub struct ResourceRegistry {
|
||||||
map: HashMap<String, Resource>,
|
map: HashMap<String, Resource>,
|
||||||
}
|
}
|
||||||
@@ -78,6 +78,17 @@ impl ResourceRegistry {
|
|||||||
pub fn is_empty(&self) -> bool {
|
pub fn is_empty(&self) -> bool {
|
||||||
self.map.is_empty()
|
self.map.is_empty()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Deserialize from a flat JSON: `{ "name": "/path", ... }`
|
||||||
|
pub fn from_json_str(json: &str) -> Result<Self, String> {
|
||||||
|
let flat: HashMap<String, PathBuf> = serde_json::from_str(json)
|
||||||
|
.map_err(|e| format!("invalid resource registry JSON: {}", e))?;
|
||||||
|
let mut reg = ResourceRegistry::new();
|
||||||
|
for (name, path) in flat {
|
||||||
|
reg.register(&name, path);
|
||||||
|
}
|
||||||
|
Ok(reg)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
|
|||||||
@@ -0,0 +1,39 @@
|
|||||||
|
# WORKSHOP-ENGINE KNOWLEDGE BASE
|
||||||
|
|
||||||
|
## OVERVIEW
|
||||||
|
Core execution layer for HoneyBiscuitWorkshop. Library-only crate (no binary). Provides Engine executor, process abstraction, package manager interfaces, duration parsing, and typed error/event systems. Used by workshop-baker; depends on no other workspace crates.
|
||||||
|
|
||||||
|
## WHERE TO LOOK
|
||||||
|
|
||||||
|
| Task | File | Notes |
|
||||||
|
|------|------|-------|
|
||||||
|
| Script execution | `src/executor.rs` | `Engine::execute_script()` — spawn, stream, timeout, kill |
|
||||||
|
| Process abstraction | `src/child.rs` | `ManagedChild` trait; `TokioChild` (real), `MockChild` (hand-rolled, no mockall) |
|
||||||
|
| Configurable commands | `src/command.rs` | `CustomCommand` — name, command, env, working_dir, timeout |
|
||||||
|
| System PMs | `src/pm.rs` + `pm/{apt,pacman,dnf,apk}.rs` | `PackageManager` trait; `all_managers()`, `select(name)` |
|
||||||
|
| User PMs | `src/upm.rs` + `upm/{rust,go,python,custom}.rs` | `UserPackageManager` trait (async); `UPMSysDeps` |
|
||||||
|
| Repology endpoint | `src/repology.rs` + `repology/local.rs` | `RepologyEndpoint` enum; local instance support |
|
||||||
|
| Duration parsing | `src/time.rs` | `parse_duration_to_ms()`, serde helpers |
|
||||||
|
| Core types | `src/types.rs` | `ExecutionContext`, `ExecutionResult`, `ExecutionEvent`, `EventSender`, `StreamType`, `Engine` |
|
||||||
|
| Error types | `src/error.rs` | `ExecutionError`, `DependencyError`, `HasExitCode` trait, `EXITCODE_*` constants |
|
||||||
|
| Integration tests | `tests/engine_integration.rs` | 12 async tests; run with `cargo test --test engine_integration` |
|
||||||
|
|
||||||
|
## CONVENTIONS
|
||||||
|
|
||||||
|
- **Edition**: Rust 2024
|
||||||
|
- **Key deps**: tokio (full), thiserror, serde+serde_yaml, async-trait, chrono, os_info, duration-str, libc, lazy_static
|
||||||
|
- **Process mgmt**: `ManagedChild` trait abstracts over real/mock children. `TokioChild` wraps `tokio::process::Child`. `MockChild` is hand-rolled (no mockall dep); supports hanging, custom output, exit codes.
|
||||||
|
- **Package manager selection**: Use `pm::select("apt")` with explicit name — NEVER auto-detect by probing the filesystem.
|
||||||
|
- **Error mapping**: `HasExitCode` trait maps errors to numeric exit codes. `ExecutionError` maps to specific `EXITCODE_*`; `DependencyError` always maps to 1.
|
||||||
|
- **Re-exports**: `lib.rs` re-exports key types for convenience (`Engine`, `ExecutionContext`, `PackageManager`, `ExecutionError`, etc.)
|
||||||
|
- **Tests**: Inline `#[cfg(test)]` blocks in `pm.rs` (8 tests). Integration tests in `tests/engine_integration.rs` (12 async tests, separate binary). CI runs both.
|
||||||
|
- **Kill semantics**: `TokioChild::kill()` sends `SIGKILL` to entire process group via `libc::kill(-PGID, SIGKILL)` with `ESRCH` silently ignored.
|
||||||
|
|
||||||
|
## ANTI-PATTERNS
|
||||||
|
|
||||||
|
- **Do NOT auto-detect the host package manager** — always use `pm::select()` with an explicitly configured name. Production PM selection comes from environment config, not probing.
|
||||||
|
- **Do NOT use `detect()`/`adopt()` patterns** for PM selection — these don't exist in this crate, and callers must not invent them.
|
||||||
|
- **Do NOT depend on workshop-baker or other workspace crates** — engine is a leaf dependency. Types needed by external callers should flow through `ExecutionContext`, not reverse imports.
|
||||||
|
- **Do NOT use mockall or other mocking frameworks** — `MockChild` is the only mock mechanism. Keep it that way.
|
||||||
|
- **Do NOT use `unimplemented!()`** — all package manager implementations are complete. Stubs belong in workshop-baker, not here.
|
||||||
|
- **Do NOT add magic platform detection** — `os_info` is used only for mirror config mapping in PM impls. Do not extend its use to auto-detect behavior.
|
||||||
@@ -1,51 +0,0 @@
|
|||||||
# workshop-engine/src
|
|
||||||
|
|
||||||
**Generated:** 2026-05-19
|
|
||||||
**Commit:** 28abc5d
|
|
||||||
|
|
||||||
Execution engine extracted from workshop-baker. Process management, package detection, resource isolation.
|
|
||||||
|
|
||||||
## STRUCTURE
|
|
||||||
```
|
|
||||||
src/
|
|
||||||
├── lib.rs # Module declarations + 10+ convenience re-exports
|
|
||||||
├── error.rs # ExecutionError + DependencyError + HasExitCode trait
|
|
||||||
├── types.rs # ExecutionContext, ExecutionResult, Engine, etc.
|
|
||||||
├── child.rs # ManagedChild + TokioChild process wrappers
|
|
||||||
├── command.rs # CustomCommand builder
|
|
||||||
├── executor.rs # Script execution
|
|
||||||
├── time.rs # Duration parsing helpers
|
|
||||||
├── pm.rs # PackageManager trait + detection
|
|
||||||
├── pm/ # Backends: apt, apk, dnf, pacman
|
|
||||||
├── upm.rs # UserPackageManager trait
|
|
||||||
├── upm/ # Backends: rust (cargo), python (pip), go, custom
|
|
||||||
├── repology.rs # RepologyEndpoint enum
|
|
||||||
└── repology/ # Package name resolution (local/)
|
|
||||||
```
|
|
||||||
|
|
||||||
## KEY EXPORTS
|
|
||||||
| Symbol | Type | Role |
|
|
||||||
|--------|------|------|
|
|
||||||
| ExecutionContext | struct | Pipeline context (task_id, username, env, etc.) |
|
|
||||||
| Engine | struct | Script executor (placeholder) |
|
|
||||||
| ExecutionError | enum | Error types with exit codes |
|
|
||||||
| HasExitCode | trait | Exit code protocol for all error types |
|
|
||||||
| ManagedChild | struct | Child process wrapper with managed lifecycle |
|
|
||||||
| TokioChild | struct | Tokio-based child process wrapper |
|
|
||||||
| CustomCommand | struct | Builder for command execution |
|
|
||||||
| PackageManager | trait | System package manager abstraction |
|
|
||||||
| UserPackageManager | trait | User-level package manager abstraction |
|
|
||||||
| RepologyEndpoint | enum | Package name resolution endpoint |
|
|
||||||
| ExecutionEvent | enum | Event stream (TaskStarted, OutputChunk, etc.) |
|
|
||||||
| ExecutionResult | struct | Command execution result (exit_code, stdout, stderr) |
|
|
||||||
| EventSender | type alias | `Option<mpsc::Sender<ExecutionEvent>>` |
|
|
||||||
|
|
||||||
## CONVENTIONS
|
|
||||||
- **Exit codes**: Centralized in `error.rs` — EXITCODE_OK=0 through EXITCODE_PRIV_DROP_FAILED=201
|
|
||||||
- **ExecutionContext**: Defaults to current directory, bash shell, non-privileged
|
|
||||||
- **PackageManagers**: Auto-detected via `os_info`, fallback chain (apt→apk→dnf→pacman)
|
|
||||||
- **Engine**: Currently a unit struct with `execute_script()` method
|
|
||||||
- **All errors implement HasExitCode**: Enables uniform error propagation up to CLI
|
|
||||||
|
|
||||||
## ANTI-PATTERNS
|
|
||||||
1. **Engine is a stub** — `types.rs:99` — `pub struct Engine;` with no fields, minimal implementation
|
|
||||||
@@ -21,8 +21,8 @@ pub enum ExecutionError {
|
|||||||
#[error("Invalid command: {0}")]
|
#[error("Invalid command: {0}")]
|
||||||
InvalidCommand(String),
|
InvalidCommand(String),
|
||||||
|
|
||||||
#[error("Execution failed: {0}")]
|
#[error("Task {task_id} failed with exit code {exit_code}")]
|
||||||
ExecutionFailed(String),
|
ExecutionFailed { task_id: String, exit_code: i32 },
|
||||||
|
|
||||||
#[error("Permission denied: uid={euid}, gid={egid}: {reason}")]
|
#[error("Permission denied: uid={euid}, gid={egid}: {reason}")]
|
||||||
PermissionDenied {
|
PermissionDenied {
|
||||||
@@ -48,7 +48,7 @@ impl HasExitCode for ExecutionError {
|
|||||||
fn exit_code(&self) -> i32 {
|
fn exit_code(&self) -> i32 {
|
||||||
match self {
|
match self {
|
||||||
ExecutionError::InvalidCommand(_) => EXITCODE_EXECUTION_ERROR,
|
ExecutionError::InvalidCommand(_) => EXITCODE_EXECUTION_ERROR,
|
||||||
ExecutionError::ExecutionFailed(_) => EXITCODE_EXECUTION_ERROR,
|
ExecutionError::ExecutionFailed { .. } => EXITCODE_EXECUTION_ERROR,
|
||||||
ExecutionError::PermissionDenied { .. } => EXITCODE_GENERAL_ERROR,
|
ExecutionError::PermissionDenied { .. } => EXITCODE_GENERAL_ERROR,
|
||||||
ExecutionError::Timeout => EXITCODE_TIMEOUT,
|
ExecutionError::Timeout => EXITCODE_TIMEOUT,
|
||||||
ExecutionError::IoError(_) => EXITCODE_IO_ERROR,
|
ExecutionError::IoError(_) => EXITCODE_IO_ERROR,
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ use chrono::Utc;
|
|||||||
use std::path::PathBuf;
|
use std::path::PathBuf;
|
||||||
use std::process::Stdio;
|
use std::process::Stdio;
|
||||||
use tokio::io::AsyncReadExt;
|
use tokio::io::AsyncReadExt;
|
||||||
|
use tokio::io::AsyncWriteExt;
|
||||||
use tokio::process::Command;
|
use tokio::process::Command;
|
||||||
use tokio::sync::mpsc;
|
use tokio::sync::mpsc;
|
||||||
use tokio::time::timeout;
|
use tokio::time::timeout;
|
||||||
@@ -12,7 +13,7 @@ use crate::types::{
|
|||||||
DeltaExecutionContext, EventSender, ExecutionContext, ExecutionError, ExecutionEvent,
|
DeltaExecutionContext, EventSender, ExecutionContext, ExecutionError, ExecutionEvent,
|
||||||
ExecutionResult, StreamType,
|
ExecutionResult, StreamType,
|
||||||
};
|
};
|
||||||
|
const STDOUT_BUF_CAP: usize = 2 * 1024 * 1024;
|
||||||
impl crate::types::Engine {
|
impl crate::types::Engine {
|
||||||
pub fn new() -> Self {
|
pub fn new() -> Self {
|
||||||
Self
|
Self
|
||||||
@@ -87,7 +88,7 @@ impl crate::types::Engine {
|
|||||||
.env("HBW_TASKID", &ctx.task_id)
|
.env("HBW_TASKID", &ctx.task_id)
|
||||||
.env("HBW_PIPELINE", &ctx.pipeline_name)
|
.env("HBW_PIPELINE", &ctx.pipeline_name)
|
||||||
.env("HBW_USERNAME", &ctx.username)
|
.env("HBW_USERNAME", &ctx.username)
|
||||||
.stdin(Stdio::null())
|
.stdin(if ctx.stdin.is_some() { Stdio::piped() } else { Stdio::null() })
|
||||||
.stdout(Stdio::piped())
|
.stdout(Stdio::piped())
|
||||||
.stderr(Stdio::piped());
|
.stderr(Stdio::piped());
|
||||||
}
|
}
|
||||||
@@ -121,10 +122,19 @@ impl crate::types::Engine {
|
|||||||
#[cfg(unix)]
|
#[cfg(unix)]
|
||||||
command.process_group(0);
|
command.process_group(0);
|
||||||
|
|
||||||
let child = command.spawn().map_err(|e| {
|
let mut child = command.spawn().map_err(|e| {
|
||||||
ExecutionError::InvalidCommand(format!("Failed to spawn command: {}", e))
|
ExecutionError::InvalidCommand(format!("Failed to spawn command: {}", e))
|
||||||
})?;
|
})?;
|
||||||
|
|
||||||
|
// Write piped stdin (e.g. from @pipe). Consumed once per execution.
|
||||||
|
if let Some(ref data) = ctx.stdin {
|
||||||
|
if let Some(mut stdin_pipe) = child.stdin.take() {
|
||||||
|
stdin_pipe.write_all(data).await.map_err(|e| {
|
||||||
|
ExecutionError::IoError(format!("Failed to write stdin: {}", e))
|
||||||
|
})?;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
self.execute_child(TokioChild(child), ctx, event_tx).await
|
self.execute_child(TokioChild(child), ctx, event_tx).await
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -149,11 +159,8 @@ impl crate::types::Engine {
|
|||||||
.await;
|
.await;
|
||||||
}
|
}
|
||||||
|
|
||||||
let task_id = ctx.task_id.clone();
|
|
||||||
let (stdout_tx, mut stdout_rx) = mpsc::channel::<String>(1024);
|
let (stdout_tx, mut stdout_rx) = mpsc::channel::<String>(1024);
|
||||||
let (stderr_tx, mut stderr_rx) = mpsc::channel::<String>(1024);
|
let (stderr_tx, mut stderr_rx) = mpsc::channel::<String>(1024);
|
||||||
let mut stdout_buf = String::new();
|
|
||||||
let mut stderr_buf = String::new();
|
|
||||||
|
|
||||||
let mut child_stdout = child.take_stdout();
|
let mut child_stdout = child.take_stdout();
|
||||||
let mut child_stderr = child.take_stderr();
|
let mut child_stderr = child.take_stderr();
|
||||||
@@ -198,6 +205,32 @@ impl crate::types::Engine {
|
|||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// Forwarder: drains channels in real-time, emits OutputChunk events,
|
||||||
|
// and accumulates output into 2 MiB ring buffers.
|
||||||
|
let task_id_ev = ctx.task_id.clone();
|
||||||
|
let event_tx_fwd = event_tx.clone();
|
||||||
|
let forwarder = tokio::spawn(async move {
|
||||||
|
let mut stdout_buf: Vec<u8> = Vec::new();
|
||||||
|
let mut stderr_buf: Vec<u8> = Vec::new();
|
||||||
|
loop {
|
||||||
|
tokio::select! {
|
||||||
|
biased;
|
||||||
|
Some(data) = stdout_rx.recv() => {
|
||||||
|
emit_chunk(&event_tx_fwd, &task_id_ev, StreamType::Stdout, &data).await;
|
||||||
|
stdout_buf.extend_from_slice(data.as_bytes());
|
||||||
|
truncate_buf(&mut stdout_buf);
|
||||||
|
}
|
||||||
|
Some(data) = stderr_rx.recv() => {
|
||||||
|
emit_chunk(&event_tx_fwd, &task_id_ev, StreamType::Stderr, &data).await;
|
||||||
|
stderr_buf.extend_from_slice(data.as_bytes());
|
||||||
|
truncate_buf(&mut stderr_buf);
|
||||||
|
}
|
||||||
|
else => break,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
(stdout_buf, stderr_buf)
|
||||||
|
});
|
||||||
|
|
||||||
let exit_code = if let Some(timeout_duration) = ctx.timeout {
|
let exit_code = if let Some(timeout_duration) = ctx.timeout {
|
||||||
match timeout(timeout_duration, child.wait()).await {
|
match timeout(timeout_duration, child.wait()).await {
|
||||||
Ok(Ok(code)) => code,
|
Ok(Ok(code)) => code,
|
||||||
@@ -226,12 +259,7 @@ impl crate::types::Engine {
|
|||||||
drop(stderr_tx);
|
drop(stderr_tx);
|
||||||
let _ = stdout_handle.await;
|
let _ = stdout_handle.await;
|
||||||
let _ = stderr_handle.await;
|
let _ = stderr_handle.await;
|
||||||
while let Some(data) = stdout_rx.recv().await {
|
let _ = forwarder.await;
|
||||||
stdout_buf.push_str(&data);
|
|
||||||
}
|
|
||||||
while let Some(data) = stderr_rx.recv().await {
|
|
||||||
stderr_buf.push_str(&data);
|
|
||||||
}
|
|
||||||
return Err(ExecutionError::Timeout);
|
return Err(ExecutionError::Timeout);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -246,31 +274,9 @@ impl crate::types::Engine {
|
|||||||
|
|
||||||
let _ = stdout_handle.await;
|
let _ = stdout_handle.await;
|
||||||
let _ = stderr_handle.await;
|
let _ = stderr_handle.await;
|
||||||
|
let (stdout_buf, stderr_buf) = forwarder
|
||||||
while let Some(data) = stdout_rx.recv().await {
|
.await
|
||||||
stdout_buf.push_str(&data);
|
.map_err(|e| ExecutionError::IoError(format!("Forwarder task panicked: {}", e)))?;
|
||||||
if let Some(tx) = event_tx {
|
|
||||||
let _ = tx
|
|
||||||
.send(ExecutionEvent::OutputChunk {
|
|
||||||
task_id: task_id.clone(),
|
|
||||||
stream: StreamType::Stdout,
|
|
||||||
data: data.clone(),
|
|
||||||
})
|
|
||||||
.await;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
while let Some(data) = stderr_rx.recv().await {
|
|
||||||
stderr_buf.push_str(&data);
|
|
||||||
if let Some(tx) = event_tx {
|
|
||||||
let _ = tx
|
|
||||||
.send(ExecutionEvent::OutputChunk {
|
|
||||||
task_id: task_id.clone(),
|
|
||||||
stream: StreamType::Stderr,
|
|
||||||
data: data.clone(),
|
|
||||||
})
|
|
||||||
.await;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
let duration = start_time.elapsed();
|
let duration = start_time.elapsed();
|
||||||
|
|
||||||
@@ -279,32 +285,26 @@ impl crate::types::Engine {
|
|||||||
exit_code,
|
exit_code,
|
||||||
success: exit_code == 0,
|
success: exit_code == 0,
|
||||||
duration,
|
duration,
|
||||||
stdout: stdout_buf,
|
stdout: String::from_utf8_lossy(&stdout_buf).into_owned(),
|
||||||
stderr: stderr_buf,
|
stderr: String::from_utf8_lossy(&stderr_buf).into_owned(),
|
||||||
};
|
};
|
||||||
|
|
||||||
if let Some(tx) = event_tx {
|
if let Some(tx) = event_tx {
|
||||||
let event = if result.success {
|
let _ = tx
|
||||||
ExecutionEvent::TaskCompleted {
|
.send(ExecutionEvent::TaskCompleted {
|
||||||
task_id: ctx.task_id.clone(),
|
task_id: ctx.task_id.clone(),
|
||||||
result: result.clone(),
|
result: result.clone(),
|
||||||
}
|
})
|
||||||
} else {
|
.await;
|
||||||
ExecutionEvent::TaskFailed {
|
|
||||||
task_id: ctx.task_id.clone(),
|
|
||||||
error: format!("Exit code: {}", exit_code),
|
|
||||||
}
|
|
||||||
};
|
|
||||||
let _ = tx.send(event).await;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
if result.success {
|
if result.success {
|
||||||
Ok(result)
|
Ok(result)
|
||||||
} else {
|
} else {
|
||||||
Err(ExecutionError::ExecutionFailed(format!(
|
Err(ExecutionError::ExecutionFailed {
|
||||||
"Task {} failed with exit code {}",
|
task_id: ctx.task_id.clone(),
|
||||||
ctx.task_id, exit_code
|
exit_code,
|
||||||
)))
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -342,6 +342,30 @@ impl crate::types::Engine {
|
|||||||
Ok(combined_result)
|
Ok(combined_result)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
/// Truncate a buffer to [`STDOUT_BUF_CAP`] bytes, discarding the oldest data.
|
||||||
|
fn truncate_buf(buf: &mut Vec<u8>) {
|
||||||
|
if buf.len() > STDOUT_BUF_CAP {
|
||||||
|
buf.drain(0..buf.len() - STDOUT_BUF_CAP);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Emit an `OutputChunk` event if the channel exists.
|
||||||
|
async fn emit_chunk(
|
||||||
|
tx: &EventSender,
|
||||||
|
task_id: &str,
|
||||||
|
stream: StreamType,
|
||||||
|
data: &str,
|
||||||
|
) {
|
||||||
|
if let Some(tx) = tx {
|
||||||
|
let _ = tx
|
||||||
|
.send(ExecutionEvent::OutputChunk {
|
||||||
|
task_id: task_id.to_string(),
|
||||||
|
stream,
|
||||||
|
data: data.to_string(),
|
||||||
|
})
|
||||||
|
.await;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
impl Default for crate::types::Engine {
|
impl Default for crate::types::Engine {
|
||||||
fn default() -> Self {
|
fn default() -> Self {
|
||||||
@@ -369,6 +393,7 @@ mod tests {
|
|||||||
dry_run: false,
|
dry_run: false,
|
||||||
privileged: false,
|
privileged: false,
|
||||||
standalone: false,
|
standalone: false,
|
||||||
|
stdin: None,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -425,12 +450,12 @@ mod tests {
|
|||||||
assert!(result.is_err());
|
assert!(result.is_err());
|
||||||
assert!(matches!(
|
assert!(matches!(
|
||||||
result.unwrap_err(),
|
result.unwrap_err(),
|
||||||
ExecutionError::ExecutionFailed(_)
|
ExecutionError::ExecutionFailed { .. }
|
||||||
));
|
));
|
||||||
}
|
}
|
||||||
|
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
async fn exit_nonzero_sends_task_failed_event() {
|
async fn exit_nonzero_sends_task_completed_event() {
|
||||||
let engine = Engine::new();
|
let engine = Engine::new();
|
||||||
let (tx, mut rx) = mpsc::channel(64);
|
let (tx, mut rx) = mpsc::channel(64);
|
||||||
let child = MockChild::with_exit_code(42);
|
let child = MockChild::with_exit_code(42);
|
||||||
@@ -440,9 +465,10 @@ mod tests {
|
|||||||
// Skip TaskStarted
|
// Skip TaskStarted
|
||||||
let _ = rx.recv().await;
|
let _ = rx.recv().await;
|
||||||
let event = rx.recv().await;
|
let event = rx.recv().await;
|
||||||
assert!(matches!(event, Some(ExecutionEvent::TaskFailed { .. })));
|
assert!(matches!(event, Some(ExecutionEvent::TaskCompleted { .. })));
|
||||||
if let Some(ExecutionEvent::TaskFailed { error, .. }) = event {
|
if let Some(ExecutionEvent::TaskCompleted { result, .. }) = event {
|
||||||
assert!(error.contains("42"), "error should mention exit code 42");
|
assert_eq!(result.exit_code, 42);
|
||||||
|
assert!(!result.success);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ pub struct ExecutionContext {
|
|||||||
pub task_id: String,
|
pub task_id: String,
|
||||||
pub pipeline_name: String,
|
pub pipeline_name: String,
|
||||||
pub username: String,
|
pub username: String,
|
||||||
|
pub build_id: String,
|
||||||
pub working_dir: PathBuf,
|
pub working_dir: PathBuf,
|
||||||
pub env_vars: HashMap<String, String>,
|
pub env_vars: HashMap<String, String>,
|
||||||
pub timeout: Option<Duration>,
|
pub timeout: Option<Duration>,
|
||||||
@@ -18,6 +19,12 @@ pub struct ExecutionContext {
|
|||||||
pub dry_run: bool,
|
pub dry_run: bool,
|
||||||
pub privileged: bool,
|
pub privileged: bool,
|
||||||
pub standalone: bool,
|
pub standalone: bool,
|
||||||
|
/// stdin data piped by the scheduler (e.g. via `@pipe`).
|
||||||
|
/// Consumed (taken) by engine during execution.
|
||||||
|
pub stdin: Option<Vec<u8>>,
|
||||||
|
/// Bitmask of active debug features, decoded by `workshop-baker`.
|
||||||
|
/// 0 means no debug features. See `DebugFeature` for supported flags.
|
||||||
|
pub debug_flags: u32,
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Debug, Clone, Default)]
|
#[derive(Debug, Clone, Default)]
|
||||||
@@ -33,6 +40,7 @@ impl Default for ExecutionContext {
|
|||||||
task_id: String::new(),
|
task_id: String::new(),
|
||||||
pipeline_name: String::new(),
|
pipeline_name: String::new(),
|
||||||
username: String::new(),
|
username: String::new(),
|
||||||
|
build_id: String::new(),
|
||||||
working_dir: std::env::current_dir().unwrap_or_else(|_| PathBuf::from("/")),
|
working_dir: std::env::current_dir().unwrap_or_else(|_| PathBuf::from("/")),
|
||||||
env_vars: HashMap::new(),
|
env_vars: HashMap::new(),
|
||||||
timeout: None,
|
timeout: None,
|
||||||
@@ -40,6 +48,8 @@ impl Default for ExecutionContext {
|
|||||||
dry_run: false,
|
dry_run: false,
|
||||||
privileged: false,
|
privileged: false,
|
||||||
standalone: false,
|
standalone: false,
|
||||||
|
debug_flags: 0,
|
||||||
|
stdin: None,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -27,6 +27,7 @@ fn create_context() -> ExecutionContext {
|
|||||||
dry_run: false,
|
dry_run: false,
|
||||||
privileged: false,
|
privileged: false,
|
||||||
standalone: false,
|
standalone: false,
|
||||||
|
stdin: None,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
Generated
+104
@@ -0,0 +1,104 @@
|
|||||||
|
# This file is automatically @generated by Cargo.
|
||||||
|
# It is not intended for manual editing.
|
||||||
|
version = 4
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "equivalent"
|
||||||
|
version = "1.0.2"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "fixedbitset"
|
||||||
|
version = "0.5.7"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "1d674e81391d1e1ab681a28d99df07927c6d4aa5b027d7da16ba32d1d21ecd99"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "hashbrown"
|
||||||
|
version = "0.17.1"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "indexmap"
|
||||||
|
version = "2.14.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9"
|
||||||
|
dependencies = [
|
||||||
|
"equivalent",
|
||||||
|
"hashbrown",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "petgraph"
|
||||||
|
version = "0.7.1"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "3672b37090dbd86368a4145bc067582552b29c27377cad4e0a306c97f9bd7772"
|
||||||
|
dependencies = [
|
||||||
|
"fixedbitset",
|
||||||
|
"indexmap",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "proc-macro2"
|
||||||
|
version = "1.0.106"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934"
|
||||||
|
dependencies = [
|
||||||
|
"unicode-ident",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "quote"
|
||||||
|
version = "1.0.45"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924"
|
||||||
|
dependencies = [
|
||||||
|
"proc-macro2",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "syn"
|
||||||
|
version = "2.0.117"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99"
|
||||||
|
dependencies = [
|
||||||
|
"proc-macro2",
|
||||||
|
"quote",
|
||||||
|
"unicode-ident",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "thiserror"
|
||||||
|
version = "2.0.18"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "4288b5bcbc7920c07a1149a35cf9590a2aa808e0bc1eafaade0b80947865fbc4"
|
||||||
|
dependencies = [
|
||||||
|
"thiserror-impl",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "thiserror-impl"
|
||||||
|
version = "2.0.18"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "ebc4ee7f67670e9b64d05fa4253e753e016c6c95ff35b89b7941d6b856dec1d5"
|
||||||
|
dependencies = [
|
||||||
|
"proc-macro2",
|
||||||
|
"quote",
|
||||||
|
"syn",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "unicode-ident"
|
||||||
|
version = "1.0.24"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "workshop-schedule"
|
||||||
|
version = "0.1.0"
|
||||||
|
dependencies = [
|
||||||
|
"petgraph",
|
||||||
|
"thiserror",
|
||||||
|
]
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
[package]
|
||||||
|
name = "workshop-schedule"
|
||||||
|
version = "0.1.0"
|
||||||
|
edition = "2024"
|
||||||
|
description = "Semi-general DAG scheduling library for CI/CD pipelines"
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
petgraph = "0.7"
|
||||||
|
thiserror = "2"
|
||||||
@@ -0,0 +1,780 @@
|
|||||||
|
// Code in this module PARTIALLY or FULLY utilized AI Coding Agent
|
||||||
|
|
||||||
|
use petgraph::stable_graph::{NodeIndex, StableGraph};
|
||||||
|
use petgraph::Direction;
|
||||||
|
use petgraph::visit::EdgeRef;
|
||||||
|
use std::collections::{BTreeMap, BTreeSet, HashMap};
|
||||||
|
use thiserror::Error;
|
||||||
|
|
||||||
|
use crate::edge::EdgeKind;
|
||||||
|
use crate::node::{DagNode, NodeColor};
|
||||||
|
|
||||||
|
/// Errors that can occur during DAG operations.
|
||||||
|
#[derive(Debug, Error)]
|
||||||
|
pub enum DagError {
|
||||||
|
#[error("duplicate node name: {0}")]
|
||||||
|
DuplicateNode(String),
|
||||||
|
|
||||||
|
#[error("node not found: {0}")]
|
||||||
|
NodeNotFound(String),
|
||||||
|
|
||||||
|
#[error("node not ready: {0}")]
|
||||||
|
NotReady(String),
|
||||||
|
|
||||||
|
#[error("already popped: {0}")]
|
||||||
|
AlreadyPopped(String),
|
||||||
|
|
||||||
|
#[error("circular dependency detected among: {0:?}")]
|
||||||
|
Cycle(Vec<String>),
|
||||||
|
|
||||||
|
#[error("parallel group can never complete: {0:?}")]
|
||||||
|
GroupDeadlock(Vec<String>),
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Internal per-node metadata.
|
||||||
|
enum NodeState {
|
||||||
|
/// Has unsatisfied dependencies.
|
||||||
|
Pending,
|
||||||
|
/// All dependencies satisfied, in the ready set.
|
||||||
|
Ready,
|
||||||
|
/// Has been popped — done executing.
|
||||||
|
Done,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Internal per-group metadata.
|
||||||
|
#[derive(Debug)]
|
||||||
|
struct GroupMeta {
|
||||||
|
/// Node indices belonging to this group.
|
||||||
|
members: Vec<NodeIndex>,
|
||||||
|
/// Total number of members (constant after construction).
|
||||||
|
total: usize,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A DAG (Directed Acyclic Graph) for scheduling pipeline nodes.
|
||||||
|
///
|
||||||
|
/// Nodes implement [`DagNode`] and carry a [`NodeColor`] that determines
|
||||||
|
/// sequential vs parallel semantics. Edges carry an [`EdgeKind`] distinguishing
|
||||||
|
/// explicit `@after` from implicit pipeline chains.
|
||||||
|
///
|
||||||
|
/// The DAG maintains a ready set (nodes with in-degree 0) in deterministic
|
||||||
|
/// BTreeSet order, and provides methods to pop nodes **by name** — fixing the
|
||||||
|
/// limitation of `topological-sort` that could only pop the first ready node.
|
||||||
|
pub struct Dag<N: DagNode> {
|
||||||
|
graph: StableGraph<N, EdgeKind>,
|
||||||
|
name_to_idx: HashMap<String, NodeIndex>,
|
||||||
|
state: HashMap<NodeIndex, NodeState>,
|
||||||
|
in_degree: HashMap<NodeIndex, usize>,
|
||||||
|
/// Ready nodes, sorted by name for deterministic iteration.
|
||||||
|
ready_set: BTreeSet<(String, NodeIndex)>,
|
||||||
|
/// Named groups and their members.
|
||||||
|
groups: BTreeMap<String, GroupMeta>,
|
||||||
|
node_count: usize,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl<N: DagNode> Dag<N> {
|
||||||
|
/// Create an empty DAG.
|
||||||
|
pub fn new() -> Self {
|
||||||
|
Self {
|
||||||
|
graph: StableGraph::new(),
|
||||||
|
name_to_idx: HashMap::new(),
|
||||||
|
state: HashMap::new(),
|
||||||
|
in_degree: HashMap::new(),
|
||||||
|
ready_set: BTreeSet::new(),
|
||||||
|
groups: BTreeMap::new(),
|
||||||
|
node_count: 0,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Number of nodes (including pending, ready, and done).
|
||||||
|
pub fn node_count(&self) -> usize {
|
||||||
|
self.node_count
|
||||||
|
}
|
||||||
|
|
||||||
|
/// True if no nodes remain in the DAG (all popped).
|
||||||
|
pub fn is_empty(&self) -> bool {
|
||||||
|
self.ready_set.is_empty()
|
||||||
|
&& self.state.values().all(|s| matches!(s, NodeState::Done))
|
||||||
|
}
|
||||||
|
|
||||||
|
// ═══════════════════════════════════════════════════════════════════
|
||||||
|
// Construction
|
||||||
|
// ═══════════════════════════════════════════════════════════════════
|
||||||
|
|
||||||
|
/// Add a node to the DAG.
|
||||||
|
///
|
||||||
|
/// Returns an error if a node with the same name already exists.
|
||||||
|
pub fn add_node(&mut self, node: N) -> Result<(), DagError> {
|
||||||
|
let name = node.name().to_string();
|
||||||
|
if self.name_to_idx.contains_key(&name) {
|
||||||
|
return Err(DagError::DuplicateNode(name));
|
||||||
|
}
|
||||||
|
|
||||||
|
let color = node.color(); // extract before move
|
||||||
|
let idx = self.graph.add_node(node);
|
||||||
|
self.name_to_idx.insert(name.clone(), idx);
|
||||||
|
self.state.insert(idx, NodeState::Ready);
|
||||||
|
self.in_degree.insert(idx, 0);
|
||||||
|
self.ready_set.insert((name.clone(), idx));
|
||||||
|
self.node_count += 1;
|
||||||
|
|
||||||
|
// Register named groups
|
||||||
|
if let NodeColor::Named(group) = &color {
|
||||||
|
self.groups
|
||||||
|
.entry(group.clone())
|
||||||
|
.or_insert_with(|| GroupMeta {
|
||||||
|
members: Vec::new(),
|
||||||
|
total: 0,
|
||||||
|
})
|
||||||
|
.members
|
||||||
|
.push(idx);
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Freeze group totals (call after all nodes added, before any pop).
|
||||||
|
///
|
||||||
|
/// Locks in `total` for each group. Named groups whose `total` is less
|
||||||
|
/// than 2 are harmless but wasteful — they'll fire solo like sequential.
|
||||||
|
pub fn freeze(&mut self) {
|
||||||
|
for meta in self.groups.values_mut() {
|
||||||
|
meta.total = meta.members.len();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Add a dependency edge from `from` to `to`.
|
||||||
|
///
|
||||||
|
/// `to` depends on `from` — `to` won't be ready until `from` is popped.
|
||||||
|
pub fn add_edge(
|
||||||
|
&mut self,
|
||||||
|
from: &str,
|
||||||
|
to: &str,
|
||||||
|
kind: EdgeKind,
|
||||||
|
) -> Result<(), DagError> {
|
||||||
|
let from_idx = self
|
||||||
|
.name_to_idx
|
||||||
|
.get(from)
|
||||||
|
.copied()
|
||||||
|
.ok_or_else(|| DagError::NodeNotFound(from.to_string()))?;
|
||||||
|
let to_idx = self
|
||||||
|
.name_to_idx
|
||||||
|
.get(to)
|
||||||
|
.copied()
|
||||||
|
.ok_or_else(|| DagError::NodeNotFound(to.to_string()))?;
|
||||||
|
|
||||||
|
self.graph.add_edge(from_idx, to_idx, kind);
|
||||||
|
|
||||||
|
// Increment in-degree of `to`
|
||||||
|
let deg = self.in_degree.get_mut(&to_idx).unwrap();
|
||||||
|
*deg += 1;
|
||||||
|
|
||||||
|
// Remove `to` from ready set if it was there
|
||||||
|
let to_name = self.graph[to_idx].name().to_string();
|
||||||
|
self.ready_set.remove(&(to_name.clone(), to_idx));
|
||||||
|
*self.state.get_mut(&to_idx).unwrap() = NodeState::Pending;
|
||||||
|
|
||||||
|
// Sanity: check for self-loop (cycle detection lite)
|
||||||
|
if from == to {
|
||||||
|
return Err(DagError::Cycle(vec![from.to_string()]));
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
// ═══════════════════════════════════════════════════════════════════
|
||||||
|
// Queries
|
||||||
|
// ═══════════════════════════════════════════════════════════════════
|
||||||
|
|
||||||
|
/// Name → color lookup. O(1).
|
||||||
|
pub fn color_of(&self, name: &str) -> Option<NodeColor> {
|
||||||
|
let idx = self.name_to_idx.get(name)?;
|
||||||
|
Some(self.graph[*idx].color())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// True if the node is sequential (not parallel). O(1).
|
||||||
|
pub fn is_sequential(&self, name: &str) -> bool {
|
||||||
|
self.color_of(name)
|
||||||
|
.map(|c| c.is_sequential())
|
||||||
|
.unwrap_or(false)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// True if the node is a wildcard parallel. O(1).
|
||||||
|
pub fn is_wildcard(&self, name: &str) -> bool {
|
||||||
|
self.color_of(name)
|
||||||
|
.map(|c| c.is_wildcard())
|
||||||
|
.unwrap_or(false)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// True if the node is parallel (wildcard or named). O(1).
|
||||||
|
pub fn is_parallel(&self, name: &str) -> bool {
|
||||||
|
self.color_of(name)
|
||||||
|
.map(|c| c.is_parallel())
|
||||||
|
.unwrap_or(false)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Group name for a named parallel node, or `None`. O(1).
|
||||||
|
pub fn group_of(&self, name: &str) -> Option<String> {
|
||||||
|
let idx = self.name_to_idx.get(name)?;
|
||||||
|
self.graph[*idx].color().group_name().map(|s| s.to_string())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// In-degree of a node. O(1).
|
||||||
|
pub fn in_degree(&self, name: &str) -> Option<usize> {
|
||||||
|
let idx = self.name_to_idx.get(name)?;
|
||||||
|
self.in_degree.get(idx).copied()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Get the edge kind between two nodes. O(e).
|
||||||
|
pub fn edge_kind(&self, from: &str, to: &str) -> Option<EdgeKind> {
|
||||||
|
let from_idx = self.name_to_idx.get(from)?;
|
||||||
|
let to_idx = self.name_to_idx.get(to)?;
|
||||||
|
self.graph
|
||||||
|
.find_edge(*from_idx, *to_idx)
|
||||||
|
.map(|e| self.graph[e])
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Reference to the node data. O(1).
|
||||||
|
pub fn node(&self, name: &str) -> Option<&N> {
|
||||||
|
let idx = self.name_to_idx.get(name)?;
|
||||||
|
Some(&self.graph[*idx])
|
||||||
|
}
|
||||||
|
|
||||||
|
/// True if the node is fallible.
|
||||||
|
pub fn is_fallible(&self, name: &str) -> bool {
|
||||||
|
self.node(name).map(|n| n.fallible()).unwrap_or(false)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Number of ready nodes (in-degree 0, not yet popped).
|
||||||
|
pub fn ready_count(&self) -> usize {
|
||||||
|
self.ready_set.len()
|
||||||
|
}
|
||||||
|
|
||||||
|
// ═══════════════════════════════════════════════════════════════════
|
||||||
|
// Ready set queries
|
||||||
|
// ═══════════════════════════════════════════════════════════════════
|
||||||
|
|
||||||
|
/// All ready node names, in deterministic BTreeSet order.
|
||||||
|
pub fn ready_names(&self) -> Vec<&str> {
|
||||||
|
self.ready_set
|
||||||
|
.iter()
|
||||||
|
.map(|(name, _)| name.as_str())
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Ready sequential node names.
|
||||||
|
pub fn ready_sequential(&self) -> Vec<&str> {
|
||||||
|
self.ready_set
|
||||||
|
.iter()
|
||||||
|
.filter(|(name, _)| self.is_sequential(name))
|
||||||
|
.map(|(name, _)| name.as_str())
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Ready wildcard node names.
|
||||||
|
pub fn ready_wildcards(&self) -> Vec<&str> {
|
||||||
|
self.ready_set
|
||||||
|
.iter()
|
||||||
|
.filter(|(name, _)| self.is_wildcard(name))
|
||||||
|
.map(|(name, _)| name.as_str())
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Ready nodes grouped by their parallel group name.
|
||||||
|
///
|
||||||
|
/// Returns `(group_name, ready_count, total_count)` for each group that
|
||||||
|
/// has at least one ready member.
|
||||||
|
pub fn ready_groups(&self) -> Vec<(&str, usize, usize)> {
|
||||||
|
let mut result = Vec::new();
|
||||||
|
for (gname, meta) in &self.groups {
|
||||||
|
let ready_count = meta
|
||||||
|
.members
|
||||||
|
.iter()
|
||||||
|
.filter(|idx| matches!(self.state.get(idx), Some(NodeState::Ready)))
|
||||||
|
.count();
|
||||||
|
if ready_count > 0 {
|
||||||
|
result.push((gname.as_str(), ready_count, meta.total));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
result
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Pick a fireable batch per bake scheduling rules:
|
||||||
|
///
|
||||||
|
/// 1. Named groups: flood-fill from ready members through same-color
|
||||||
|
/// implicit edges. If the whole group is reachable, fire it.
|
||||||
|
/// 2. Wildcards join any fireable group.
|
||||||
|
/// 3. Otherwise, wildcards form their own batch.
|
||||||
|
/// 4. Returns `None` only when truly stuck.
|
||||||
|
pub fn pick_batch(&self) -> Option<Vec<String>> {
|
||||||
|
use std::collections::{HashSet, VecDeque};
|
||||||
|
|
||||||
|
let wildcards: Vec<String> = self
|
||||||
|
.ready_set
|
||||||
|
.iter()
|
||||||
|
.filter(|(name, _)| self.is_wildcard(name))
|
||||||
|
.map(|(name, _)| name.clone())
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
for (_gname, meta) in &self.groups {
|
||||||
|
if meta.total == 0 {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Flood-fill from ready members through same-color implicit edges
|
||||||
|
let mut reached: HashSet<NodeIndex> = HashSet::new();
|
||||||
|
let mut queue: VecDeque<NodeIndex> = VecDeque::new();
|
||||||
|
|
||||||
|
for idx in &meta.members {
|
||||||
|
if matches!(self.state.get(idx), Some(NodeState::Ready)) {
|
||||||
|
reached.insert(*idx);
|
||||||
|
queue.push_back(*idx);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
while let Some(from) = queue.pop_front() {
|
||||||
|
let from_color = self.graph[from].color();
|
||||||
|
if from_color.is_sequential() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
for edge in self.graph.edges_directed(from, Direction::Outgoing) {
|
||||||
|
if !edge.weight().is_implicit() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let to = edge.target();
|
||||||
|
let to_color = self.graph[to].color();
|
||||||
|
if to_color.is_sequential() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
let same = match (&from_color, &to_color) {
|
||||||
|
(NodeColor::Wildcard, NodeColor::Wildcard) => true,
|
||||||
|
(NodeColor::Named(a), NodeColor::Named(b)) => a.as_str() == b.as_str(),
|
||||||
|
_ => false,
|
||||||
|
};
|
||||||
|
if !same {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if !meta.members.contains(&to) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if reached.insert(to) {
|
||||||
|
queue.push_back(to);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if reached.len() == meta.total {
|
||||||
|
let batch: Vec<String> = meta
|
||||||
|
.members
|
||||||
|
.iter()
|
||||||
|
.map(|idx| self.graph[*idx].name().to_string())
|
||||||
|
.collect();
|
||||||
|
let mut batch = batch;
|
||||||
|
batch.extend(wildcards);
|
||||||
|
return Some(batch);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// No fireable named group — wildcards fire alone
|
||||||
|
if !wildcards.is_empty() {
|
||||||
|
return Some(wildcards);
|
||||||
|
}
|
||||||
|
|
||||||
|
None
|
||||||
|
}
|
||||||
|
// ═══════════════════════════════════════════════════════════════════
|
||||||
|
// Scheduling operations
|
||||||
|
// ═══════════════════════════════════════════════════════════════════
|
||||||
|
|
||||||
|
/// Pop a specific node by name.
|
||||||
|
///
|
||||||
|
/// Decrements in-degree of all successors. Those reaching 0 become ready.
|
||||||
|
/// This is the key fix over `topological-sort` — we pop by name, not by
|
||||||
|
/// arbitrary HashMap iteration order.
|
||||||
|
pub fn pop(&mut self, name: &str) -> Result<(), DagError> {
|
||||||
|
let idx = self
|
||||||
|
.name_to_idx
|
||||||
|
.get(name)
|
||||||
|
.copied()
|
||||||
|
.ok_or_else(|| DagError::NodeNotFound(name.to_string()))?;
|
||||||
|
|
||||||
|
match self.state.get(&idx) {
|
||||||
|
None => return Err(DagError::NodeNotFound(name.to_string())),
|
||||||
|
Some(NodeState::Done) => return Err(DagError::AlreadyPopped(name.to_string())),
|
||||||
|
Some(NodeState::Pending) => return Err(DagError::NotReady(name.to_string())),
|
||||||
|
Some(NodeState::Ready) => {}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Remove from ready set
|
||||||
|
let owned_name = name.to_string();
|
||||||
|
self.ready_set.remove(&(owned_name, idx));
|
||||||
|
|
||||||
|
// Decrement in-degree of all successors
|
||||||
|
let successors: Vec<NodeIndex> = self
|
||||||
|
.graph
|
||||||
|
.neighbors_directed(idx, Direction::Outgoing)
|
||||||
|
.collect();
|
||||||
|
for succ in successors {
|
||||||
|
let deg = self.in_degree.get_mut(&succ).unwrap();
|
||||||
|
*deg -= 1;
|
||||||
|
if *deg == 0 {
|
||||||
|
// Successor becomes ready
|
||||||
|
let succ_name = self.graph[succ].name().to_string();
|
||||||
|
self.ready_set.insert((succ_name, succ));
|
||||||
|
*self.state.get_mut(&succ).unwrap() = NodeState::Ready;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Mark as done
|
||||||
|
*self.state.get_mut(&idx).unwrap() = NodeState::Done;
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Pop a sequential node from the ready set.
|
||||||
|
///
|
||||||
|
/// Returns the name of the popped node, or `None` if no sequential
|
||||||
|
/// node is ready. Convenience wrapper: finds first ready sequential,
|
||||||
|
/// pops it by name.
|
||||||
|
pub fn pop_sequential(&mut self) -> Result<Option<String>, DagError> {
|
||||||
|
let name = self
|
||||||
|
.ready_set
|
||||||
|
.iter()
|
||||||
|
.find(|(name, _)| self.is_sequential(name))
|
||||||
|
.map(|(name, _)| name.clone());
|
||||||
|
|
||||||
|
if let Some(ref name) = name {
|
||||||
|
self.pop(name)?;
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(name)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ═══════════════════════════════════════════════════════════════════
|
||||||
|
// Deadlock / cycle detection
|
||||||
|
// ═══════════════════════════════════════════════════════════════════
|
||||||
|
|
||||||
|
/// True if ready nodes exist but no batch can fire (potential deadlock).
|
||||||
|
pub fn is_stalled(&self) -> bool {
|
||||||
|
!self.ready_set.is_empty() && self.pick_batch().is_none()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Names of groups that are causing a stall (incomplete groups,
|
||||||
|
/// including those with zero ready members but non-Done pending members).
|
||||||
|
pub fn stalled_groups(&self) -> Vec<&str> {
|
||||||
|
let mut result = Vec::new();
|
||||||
|
for (gname, meta) in &self.groups {
|
||||||
|
let ready_count = meta
|
||||||
|
.members
|
||||||
|
.iter()
|
||||||
|
.filter(|idx| matches!(self.state.get(idx), Some(NodeState::Ready)))
|
||||||
|
.count();
|
||||||
|
let pending_count = meta
|
||||||
|
.members
|
||||||
|
.iter()
|
||||||
|
.filter(|idx| matches!(self.state.get(idx), Some(NodeState::Pending)))
|
||||||
|
.count();
|
||||||
|
if meta.total > 0 && (ready_count > 0 || pending_count > 0) && ready_count < meta.total {
|
||||||
|
result.push(gname.as_str());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
result
|
||||||
|
}
|
||||||
|
|
||||||
|
/// All nodes that haven't been popped yet (for cycle diagnostics).
|
||||||
|
pub fn remaining(&self) -> Vec<&str> {
|
||||||
|
self.state
|
||||||
|
.iter()
|
||||||
|
.filter(|(_, s)| !matches!(s, NodeState::Done))
|
||||||
|
.map(|(idx, _)| self.graph[*idx].name())
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl<N: DagNode> Default for Dag<N> {
|
||||||
|
fn default() -> Self {
|
||||||
|
Self::new()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Tests ──────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[derive(Clone)]
|
||||||
|
struct TestNode {
|
||||||
|
name: String,
|
||||||
|
color: NodeColor,
|
||||||
|
fallible: bool,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl DagNode for TestNode {
|
||||||
|
fn name(&self) -> &str {
|
||||||
|
&self.name
|
||||||
|
}
|
||||||
|
fn color(&self) -> NodeColor {
|
||||||
|
self.color.clone()
|
||||||
|
}
|
||||||
|
fn fallible(&self) -> bool {
|
||||||
|
self.fallible
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn seq(name: &str) -> TestNode {
|
||||||
|
TestNode {
|
||||||
|
name: name.into(),
|
||||||
|
color: NodeColor::Sequential,
|
||||||
|
fallible: false,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn wild(name: &str) -> TestNode {
|
||||||
|
TestNode {
|
||||||
|
name: name.into(),
|
||||||
|
color: NodeColor::Wildcard,
|
||||||
|
fallible: false,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn group(name: &str, g: &str) -> TestNode {
|
||||||
|
TestNode {
|
||||||
|
name: name.into(),
|
||||||
|
color: NodeColor::Named(g.into()),
|
||||||
|
fallible: false,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Basic construction ────────────────────────────────────────────
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn empty_dag() {
|
||||||
|
let dag = Dag::<TestNode>::new();
|
||||||
|
assert!(dag.is_empty());
|
||||||
|
assert_eq!(dag.node_count(), 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn add_single_node() {
|
||||||
|
let mut dag = Dag::new();
|
||||||
|
dag.add_node(seq("a")).unwrap();
|
||||||
|
dag.freeze();
|
||||||
|
assert!(!dag.is_empty());
|
||||||
|
assert_eq!(dag.ready_count(), 1);
|
||||||
|
assert_eq!(dag.ready_names(), vec!["a"]);
|
||||||
|
assert!(dag.is_sequential("a"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn duplicate_node_rejected() {
|
||||||
|
let mut dag = Dag::new();
|
||||||
|
dag.add_node(seq("a")).unwrap();
|
||||||
|
assert!(dag.add_node(seq("a")).is_err());
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Edges & in-degree ─────────────────────────────────────────────
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn simple_dependency() {
|
||||||
|
let mut dag = Dag::new();
|
||||||
|
dag.add_node(seq("a")).unwrap();
|
||||||
|
dag.add_node(seq("b")).unwrap();
|
||||||
|
dag.add_edge("a", "b", EdgeKind::Explicit).unwrap();
|
||||||
|
dag.freeze();
|
||||||
|
|
||||||
|
assert!(dag.is_sequential("a"));
|
||||||
|
assert_eq!(dag.ready_names(), vec!["a"]);
|
||||||
|
assert_eq!(dag.in_degree("a"), Some(0));
|
||||||
|
assert_eq!(dag.in_degree("b"), Some(1));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pop_advances_dependents() {
|
||||||
|
let mut dag = Dag::new();
|
||||||
|
dag.add_node(seq("a")).unwrap();
|
||||||
|
dag.add_node(seq("b")).unwrap();
|
||||||
|
dag.add_node(seq("c")).unwrap();
|
||||||
|
dag.add_edge("a", "b", EdgeKind::Explicit).unwrap();
|
||||||
|
dag.add_edge("b", "c", EdgeKind::Explicit).unwrap();
|
||||||
|
dag.freeze();
|
||||||
|
|
||||||
|
assert_eq!(dag.ready_names(), vec!["a"]);
|
||||||
|
dag.pop("a").unwrap();
|
||||||
|
assert_eq!(dag.ready_names(), vec!["b"]);
|
||||||
|
dag.pop("b").unwrap();
|
||||||
|
assert_eq!(dag.ready_names(), vec!["c"]);
|
||||||
|
dag.pop("c").unwrap();
|
||||||
|
assert!(dag.is_empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pop_not_ready_rejected() {
|
||||||
|
let mut dag = Dag::new();
|
||||||
|
dag.add_node(seq("a")).unwrap();
|
||||||
|
dag.add_node(seq("b")).unwrap();
|
||||||
|
dag.add_edge("a", "b", EdgeKind::Explicit).unwrap();
|
||||||
|
dag.freeze();
|
||||||
|
|
||||||
|
assert!(dag.pop("b").is_err()); // b not ready
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pop_twice_rejected() {
|
||||||
|
let mut dag = Dag::new();
|
||||||
|
dag.add_node(seq("a")).unwrap();
|
||||||
|
dag.freeze();
|
||||||
|
dag.pop("a").unwrap();
|
||||||
|
assert!(dag.pop("a").is_err()); // already done
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Node coloring ─────────────────────────────────────────────────
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn color_queries() {
|
||||||
|
let mut dag = Dag::new();
|
||||||
|
dag.add_node(seq("s")).unwrap();
|
||||||
|
dag.add_node(wild("w")).unwrap();
|
||||||
|
dag.add_node(group("g", "build")).unwrap();
|
||||||
|
dag.freeze();
|
||||||
|
|
||||||
|
assert!(dag.is_sequential("s"));
|
||||||
|
assert!(dag.is_wildcard("w"));
|
||||||
|
assert_eq!(dag.group_of("g"), Some("build".to_string()));
|
||||||
|
assert!(dag.is_parallel("w"));
|
||||||
|
assert!(dag.is_parallel("g"));
|
||||||
|
assert!(!dag.is_parallel("s"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn fallible_query() {
|
||||||
|
let mut dag = Dag::new();
|
||||||
|
dag.add_node(TestNode {
|
||||||
|
name: "f".into(),
|
||||||
|
color: NodeColor::Sequential,
|
||||||
|
fallible: true,
|
||||||
|
})
|
||||||
|
.unwrap();
|
||||||
|
dag.freeze();
|
||||||
|
assert!(dag.is_fallible("f"));
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Ready set queries ─────────────────────────────────────────────
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn ready_filtering() {
|
||||||
|
let mut dag = Dag::new();
|
||||||
|
dag.add_node(seq("s1")).unwrap();
|
||||||
|
dag.add_node(seq("s2")).unwrap();
|
||||||
|
dag.add_node(wild("w1")).unwrap();
|
||||||
|
dag.add_node(wild("w2")).unwrap();
|
||||||
|
dag.add_node(group("g1", "build")).unwrap();
|
||||||
|
dag.add_node(group("g2", "build")).unwrap();
|
||||||
|
dag.freeze();
|
||||||
|
|
||||||
|
assert_eq!(dag.ready_sequential(), vec!["s1", "s2"]);
|
||||||
|
assert_eq!(dag.ready_wildcards(), vec!["w1", "w2"]);
|
||||||
|
let groups = dag.ready_groups();
|
||||||
|
assert_eq!(groups.len(), 1);
|
||||||
|
assert_eq!(groups[0], ("build", 2, 2));
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Batch picking ─────────────────────────────────────────────────
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pick_batch_complete_group_with_wildcards() {
|
||||||
|
let mut dag = Dag::new();
|
||||||
|
dag.add_node(seq("s")).unwrap(); // not ready (has dep)
|
||||||
|
dag.add_node(group("g1", "build")).unwrap();
|
||||||
|
dag.add_node(group("g2", "build")).unwrap();
|
||||||
|
dag.add_node(wild("w")).unwrap();
|
||||||
|
dag.add_edge("s", "g1", EdgeKind::Explicit).unwrap(); // block g1
|
||||||
|
dag.freeze();
|
||||||
|
|
||||||
|
// g1 blocked by s, build group incomplete
|
||||||
|
// BUT wildcard w is ready → fires as its own batch
|
||||||
|
let batch = dag.pick_batch().unwrap();
|
||||||
|
assert_eq!(batch.len(), 1);
|
||||||
|
assert!(batch.contains(&"w".to_string()));
|
||||||
|
|
||||||
|
dag.pop("w").unwrap();
|
||||||
|
dag.pop("s").unwrap();
|
||||||
|
// Now g1, g2 both ready; build group complete
|
||||||
|
let batch = dag.pick_batch().unwrap();
|
||||||
|
assert!(batch.contains(&"g1".to_string()));
|
||||||
|
assert!(batch.contains(&"g2".to_string()));
|
||||||
|
}
|
||||||
|
#[test]
|
||||||
|
fn pick_batch_wildcards_only_no_groups() {
|
||||||
|
let mut dag = Dag::new();
|
||||||
|
dag.add_node(wild("w1")).unwrap();
|
||||||
|
dag.add_node(wild("w2")).unwrap();
|
||||||
|
dag.freeze();
|
||||||
|
|
||||||
|
let batch = dag.pick_batch().unwrap();
|
||||||
|
assert_eq!(batch.len(), 2);
|
||||||
|
assert!(batch.contains(&"w1".to_string()));
|
||||||
|
assert!(batch.contains(&"w2".to_string()));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pick_batch_none_when_sequential_only() {
|
||||||
|
let mut dag = Dag::new();
|
||||||
|
dag.add_node(seq("s1")).unwrap();
|
||||||
|
dag.add_node(seq("s2")).unwrap();
|
||||||
|
dag.freeze();
|
||||||
|
|
||||||
|
// pick_batch only picks parallel batches; sequential nodes
|
||||||
|
// should be picked individually via pop_sequential()
|
||||||
|
assert!(dag.pick_batch().is_none());
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── pop_sequential convenience ────────────────────────────────────
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pop_sequential_picks_first_seq() {
|
||||||
|
let mut dag = Dag::new();
|
||||||
|
dag.add_node(wild("w")).unwrap();
|
||||||
|
dag.add_node(seq("s1")).unwrap();
|
||||||
|
dag.add_node(seq("s2")).unwrap();
|
||||||
|
dag.freeze();
|
||||||
|
|
||||||
|
let popped = dag.pop_sequential().unwrap().unwrap();
|
||||||
|
assert_eq!(popped, "s1"); // BTreeSet order: s1 before s2
|
||||||
|
assert!(!dag.ready_names().contains(&"s1"));
|
||||||
|
assert!(dag.ready_names().contains(&"s2"));
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Deadlock detection ────────────────────────────────────────────
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn stall_when_group_incomplete() {
|
||||||
|
let mut dag = Dag::new();
|
||||||
|
dag.add_node(seq("s")).unwrap();
|
||||||
|
dag.add_node(group("g1", "build")).unwrap();
|
||||||
|
dag.add_node(group("g2", "build")).unwrap();
|
||||||
|
dag.add_edge("s", "g2", EdgeKind::Explicit).unwrap();
|
||||||
|
dag.freeze();
|
||||||
|
|
||||||
|
// g1 ready, g2 not (depends on s). pick_batch → none.
|
||||||
|
assert!(dag.is_stalled());
|
||||||
|
assert_eq!(dag.stalled_groups(), vec!["build"]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn remaining_after_partial_pop() {
|
||||||
|
let mut dag = Dag::new();
|
||||||
|
dag.add_node(seq("a")).unwrap();
|
||||||
|
dag.add_node(seq("b")).unwrap();
|
||||||
|
dag.add_node(seq("c")).unwrap();
|
||||||
|
dag.freeze();
|
||||||
|
|
||||||
|
dag.pop("a").unwrap();
|
||||||
|
let rem = dag.remaining();
|
||||||
|
assert_eq!(rem.len(), 2);
|
||||||
|
assert!(rem.contains(&"b"));
|
||||||
|
assert!(rem.contains(&"c"));
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,20 @@
|
|||||||
|
// Code in this module PARTIALLY or FULLY utilized AI Coding Agent
|
||||||
|
|
||||||
|
/// Edge kind — distinguishes explicit `@after` from implicit pipeline chaining.
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
|
||||||
|
pub enum EdgeKind {
|
||||||
|
/// Explicit dependency: `@after(func)` in the script.
|
||||||
|
Explicit,
|
||||||
|
/// Implicit dependency: consecutive `@pipeline` functions without `@after`.
|
||||||
|
Implicit,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl EdgeKind {
|
||||||
|
pub fn is_explicit(&self) -> bool {
|
||||||
|
matches!(self, EdgeKind::Explicit)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn is_implicit(&self) -> bool {
|
||||||
|
matches!(self, EdgeKind::Implicit)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
// Code in this module PARTIALLY or FULLY utilized AI Coding Agent
|
||||||
|
|
||||||
|
pub mod dag;
|
||||||
|
pub mod edge;
|
||||||
|
pub mod node;
|
||||||
|
|
||||||
|
pub use dag::{Dag, DagError};
|
||||||
|
pub use edge::EdgeKind;
|
||||||
|
pub use node::{DagNode, NodeColor};
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
// Code in this module PARTIALLY or FULLY utilized AI Coding Agent
|
||||||
|
|
||||||
|
/// Node coloring — mirrors the bake DAG generator's color semantics.
|
||||||
|
///
|
||||||
|
/// | Color | Probability | Bake semantics |
|
||||||
|
/// |-------------|------------|-----------------------------|
|
||||||
|
/// | Sequential | p_seq | Runs alone, one at a time |
|
||||||
|
/// | Wildcard | p_wild | Joins any complete group |
|
||||||
|
/// | Named(G) | p_named/K | Fires when group complete |
|
||||||
|
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
|
||||||
|
pub enum NodeColor {
|
||||||
|
/// Sequential node — executes alone.
|
||||||
|
Sequential,
|
||||||
|
/// Wildcard parallel — compatible with any complete named group.
|
||||||
|
Wildcard,
|
||||||
|
/// Named parallel group member — fires when all members of this group are ready.
|
||||||
|
Named(String),
|
||||||
|
}
|
||||||
|
|
||||||
|
impl NodeColor {
|
||||||
|
/// Returns true for `Sequential`.
|
||||||
|
pub fn is_sequential(&self) -> bool {
|
||||||
|
matches!(self, NodeColor::Sequential)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns true for `Wildcard`.
|
||||||
|
pub fn is_wildcard(&self) -> bool {
|
||||||
|
matches!(self, NodeColor::Wildcard)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the group name if `Named`, or `None`.
|
||||||
|
pub fn group_name(&self) -> Option<&str> {
|
||||||
|
match self {
|
||||||
|
NodeColor::Named(g) => Some(g.as_str()),
|
||||||
|
_ => None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns true if this is a parallel node (Wildcard or Named).
|
||||||
|
pub fn is_parallel(&self) -> bool {
|
||||||
|
!self.is_sequential()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Trait that nodes must implement to be managed by [`Dag`](super::dag::Dag).
|
||||||
|
///
|
||||||
|
/// Only `name()` and `color()` are required. Override `fallible()` if the
|
||||||
|
/// node can fail without halting the pipeline.
|
||||||
|
pub trait DagNode {
|
||||||
|
/// Unique name of this node within the DAG.
|
||||||
|
fn name(&self) -> &str;
|
||||||
|
|
||||||
|
/// Scheduling color — determines sequential/parallel/group semantics.
|
||||||
|
fn color(&self) -> NodeColor;
|
||||||
|
|
||||||
|
/// Whether failure of this node is non-fatal to the pipeline.
|
||||||
|
fn fallible(&self) -> bool {
|
||||||
|
false
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user