Compare commits
17 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| fdc00f0635 | |||
| 26809df720 | |||
| 3629c33fc4 | |||
| 6d9ddbca56 | |||
| 9ff9073fce | |||
| df5026cdbe | |||
| d1ad08ef3a | |||
| cf2968e720 | |||
| 28abc5d207 | |||
| 6e7b96cd66 | |||
| 3e80825d58 | |||
| c97eafab29 | |||
| f737b6a24d | |||
| 681b7d1333 | |||
| f07588a716 | |||
| b0530fb858 | |||
| fd18b547e5 |
+3
-1
@@ -9,7 +9,7 @@ archived
|
|||||||
*.bak
|
*.bak
|
||||||
*.clean
|
*.clean
|
||||||
session*
|
session*
|
||||||
llm.env
|
.secret
|
||||||
|
|
||||||
# System test artifacts
|
# System test artifacts
|
||||||
tests/results/
|
tests/results/
|
||||||
@@ -18,3 +18,5 @@ tests/**/__pycache__/
|
|||||||
tests/.pytest_cache/
|
tests/.pytest_cache/
|
||||||
tests/llm/.cache/
|
tests/llm/.cache/
|
||||||
*.pyc
|
*.pyc
|
||||||
|
.staging
|
||||||
|
.pytest_cache
|
||||||
|
|||||||
@@ -1,108 +1,127 @@
|
|||||||
# PROJECT KNOWLEDGE BASE
|
# PROJECT KNOWLEDGE BASE
|
||||||
|
|
||||||
**Generated:** 2026-04-22
|
**Generated:** 2026-06-12T16:08:15Z
|
||||||
**Commit:** 7b3b71a
|
**Commit:** 3629c33
|
||||||
**Branch:** master
|
**Branch:** feat/bake-dag-tester
|
||||||
|
|
||||||
## OVERVIEW
|
## OVERVIEW
|
||||||
HoneyBiscuitWorkshop is a Rust-based CI/CD system ("蜜饼工坊") with LLM-powered pipeline auto-configuration. 9 independent Cargo crates, no workspace.
|
HoneyBiscuitWorkshop (蜜饼工坊) — Rust-based CI/CD system inspired by Concourse CI. Pipeline: prebake.yml → bake.sh → finalize.yml. Multi-builder (Docker, Firecracker, bare metal). Plugin system.
|
||||||
|
|
||||||
## STRUCTURE
|
## STRUCTURE
|
||||||
```
|
```
|
||||||
./
|
./
|
||||||
├── workshop-baker/ # Main orchestrator (CLI + daemon)
|
├── workshop-baker/ # Main binary + lib — orchestrator, CLI, plugins
|
||||||
├── workshop-agent/ # HTTP agent server
|
├── workshop-engine/ # Core execution engine — process mgmt, package managers
|
||||||
├── workshop-pipeline/ # Config library (prebake/finalize)
|
├── workshop-schedule/ # DAG scheduling library (petgraph-based)
|
||||||
├── workshop-llm-detector/ # LLM repo analyzer
|
├── workshop-getterurl/ # URL fetching with hash verification
|
||||||
├── workshop-vault/ # Secret management client
|
├── workshop-baker-params/ # Config/parameter type definitions
|
||||||
├── workshop-cert/ # TLS certificate generator
|
├── docs/ # mdBook documentation (Chinese, 3 volumes)
|
||||||
├── workshop-deviceid/ # Device ID library
|
├── templates/ # Pipeline config templates (prebake/bake/finalize)
|
||||||
├── workshop-builder-native/ # Stub (unused)
|
└── .github/workflows/ # CI (tests workshop-baker + workshop-engine)
|
||||||
├── workshop-helper-mac/ # MAC address utility
|
|
||||||
├── docs/ # mdBook documentation (zh-CN)
|
|
||||||
├── playground/ # Experimentation
|
|
||||||
└── third_party/ # Empty
|
|
||||||
```
|
```
|
||||||
|
**No root Cargo workspace** — crates linked by path deps. Build per-crate.
|
||||||
|
|
||||||
## WHERE TO LOOK
|
## WHERE TO LOOK
|
||||||
|
|
||||||
| Task | Location | Notes |
|
| Task | Location | Notes |
|
||||||
|------|----------|-------|
|
|------|----------|-------|
|
||||||
| Baker (main) | `workshop-baker/src/{bake,prebake,engine}/` | Core pipeline logic |
|
| CLI definition | `workshop-baker/src/lib.rs` | Clap Cli/Commands/BareCommands |
|
||||||
| Agent server | `workshop-agent/src/` | Actix-web HTTP + TLS |
|
| Pipeline orchestration | `workshop-baker/src/main.rs` | prebake→bake→finalize→notify flow |
|
||||||
| Pipeline config | `workshop-pipeline/src/config/` | prebake/finalize YAML |
|
| Decorator parsing | `workshop-baker/src/bake/decorator.rs` | `# @decorator(args)` syntax |
|
||||||
| LLM integration | `workshop-llm-detector/src/llm/` | Ollama/OpenAI clients |
|
| Build execution | `workshop-baker/src/bake/execute.rs` | Script execution with timeout/retry |
|
||||||
| Secrets | `workshop-vault/src/` | Vault client wrapper |
|
| Docker integration | `workshop-baker/src/prebake/env/container.rs` | Bollard-based |
|
||||||
|
| Package managers | `workshop-engine/src/pm/` | apt, pacman, dnf, apk |
|
||||||
|
| User package managers | `workshop-engine/src/upm/` | rustup, uv, go |
|
||||||
|
| Notification system | `workshop-baker/src/notify/` | Email, webhook, IM |
|
||||||
|
| Plugin system | `workshop-baker/src/finalize/plugin/` | shell, dylib, rhai, internal |
|
||||||
|
| Error types | `workshop-baker/src/error.rs` | CliError + HasExitCode |
|
||||||
|
| Exit codes | `workshop-engine/src/error.rs` | EXITCODE_* constants |
|
||||||
|
| DAG scheduling | `workshop-schedule/src/dag.rs` | Topological sort, batch picking |
|
||||||
|
| Pipeline templates | `templates/` | Reference config DSL |
|
||||||
|
|
||||||
|
## ENTRY POINTS
|
||||||
|
|
||||||
|
| Entry | Type | Path | Role |
|
||||||
|
|-------|------|------|------|
|
||||||
|
| `workshop-baker` | Rust bin | `workshop-baker/src/main.rs` | Primary CLI — dispatch: default (full pipeline), bare (single stage), daemon (unimplemented) |
|
||||||
|
| `workshop_baker` | Rust lib | `workshop-baker/src/lib.rs` | Baker library + CLI structs |
|
||||||
|
| `workshop_engine` | Rust lib | `workshop-engine/src/lib.rs` | Core engine — ExecutionContext, EventSender, Executor |
|
||||||
|
| `workshop_schedule` | Rust lib | `workshop-schedule/src/lib.rs` | Dag, DagNode, EdgeKind |
|
||||||
|
| `workshop_baker_params` | Rust lib | `workshop-baker-params/src/lib.rs` | Layered config: defaults → base → courier |
|
||||||
|
| `workshop_getterurl` | Rust lib | `workshop-getterurl/src/lib.rs` | GetterUrl, fetch() — http/git/file |
|
||||||
|
| docs | mdBook | `docs/book.toml` | Project documentation site |
|
||||||
|
|
||||||
## CODE MAP
|
## CODE MAP
|
||||||
| Symbol | Type | Location | Role |
|
|
||||||
|--------|------|----------|------|
|
### Crate Dependency Graph
|
||||||
| PipelineConfig | struct | workshop-pipeline/src/lib.rs:27 | Full pipeline config |
|
```
|
||||||
| CICDLLMHelper | struct | workshop-llm-detector/src/lib.rs:16 | LLM analysis entry |
|
workshop-baker (bin)
|
||||||
| Engine | struct | workshop-baker/src/engine.rs | Build executor |
|
├── workshop-engine
|
||||||
| VaultClient | struct | workshop-vault/src/client.rs | Secret ops |
|
├── workshop-schedule
|
||||||
|
├── workshop-baker-params
|
||||||
|
└── workshop-getterurl
|
||||||
|
```
|
||||||
|
|
||||||
|
### workshop-baker Module Map
|
||||||
|
| Module | Submodules | Role |
|
||||||
|
|--------|-----------|------|
|
||||||
|
| `bake` | decorator, execute, parser, schedule, builder, trivial, util, constant, error | Script parsing + execution |
|
||||||
|
| `prebake` | config, stage (bootstrap, depssystem, depsuser, environment, hook), env (container, firecracker, baremetal, custom), security, event, types, constant, error | Environment setup |
|
||||||
|
| `finalize` | plugin (shell, dylib, rhai, internal), stage, template, types, config, constant, error, event | Post-build hooks |
|
||||||
|
| `notify` | handler, queue, rule, template, types, util, error | Async notification dispatch |
|
||||||
|
| `bare` | — | Standalone single-stage CLI mode |
|
||||||
|
| `types` | resource, buildstatus | Shared type definitions |
|
||||||
|
| `utils` | — | Cross-cutting utilities |
|
||||||
|
|
||||||
## CONVENTIONS
|
## CONVENTIONS
|
||||||
|
|
||||||
### Rust
|
- **Edition**: Rust 2024 (requires Rust 1.85+)
|
||||||
- **Edition**: 2024 (non-standard, requires Rust 1.85+)
|
- **Formatting**: Default `rustfmt` — no `rustfmt.toml` / `.editorconfig`
|
||||||
- Deprecate `mod.rs`
|
- **Lint overrides** (workshop-baker only): `dead_code = "allow"`, `unreachable_code = "allow"`, `inherent_to_string = "allow"`, `non_canonical_partial_ord_impl = "allow"`
|
||||||
- **Naming**: `snake_case` files/modules, `PascalCase` types, `SCREAMING_SNAKE_CASE` consts
|
- **Import order**: `std` → external crates → `crate::` (blank line between groups)
|
||||||
- **Error handling**: `thiserror` + `anyhow` combo
|
- **Error handling**: `thiserror` for libraries, `anyhow` for binaries. Implement `HasExitCode` for exit code mapping.
|
||||||
- **Async**: `#[tokio::main]` + `tokio` with "full" features
|
- **Type system**: Newtype pattern preferred (e.g., `MemSize(u64)`). Shared types in `types/` module. `types/` must NOT depend on `bake/`, `engine/`, `prebake/`, `finalize/`.
|
||||||
- **Imports**: `std` → `external_crate` → `crate::module`
|
- **Tests**: Inline `#[cfg(test)] mod tests` at bottom of source files. Async: `#[tokio::test]`. CI runs `cargo test --all-features`.
|
||||||
|
- **Commits**: `type(scope): description` — `feat|fix|docs|test|refactor|style`
|
||||||
### Testing
|
- **AI annotation**: `// Code in this module PARTIALLY or FULLY utilized AI Coding Agent` — never delete if present
|
||||||
- **Rust**: `#[test]` inline, `#[tokio::test]` async, `tests/*.rs` integration
|
- **Docs language**: Chinese (zh-CN), mdBook
|
||||||
- **Python**: `pytest` with `tests/integration/test_*.py` (Docker-based)
|
- **No `no_std`**: Explicitly unsupported
|
||||||
- **Benchmarks**: Criterion with `harness = false`
|
|
||||||
|
|
||||||
### Build Pipeline
|
|
||||||
- **prebake.yml**: Environment setup (docker/firecracker/baremetal)
|
|
||||||
- **bake.sh**: Build execution with `# @pipeline` decorators
|
|
||||||
- **finalize.yml**: Packaging, deployment, notifications
|
|
||||||
- **Annotations**: `@timeout()`, `@retry()`, `@parallel`, `@fallible`
|
|
||||||
|
|
||||||
## ANTI-PATTERNS (THIS PROJECT)
|
## ANTI-PATTERNS (THIS PROJECT)
|
||||||
|
|
||||||
1. **Python in Rust project** — workshop-baker has Python tests (pytest.ini, test_prebake.py)
|
- **Do NOT derive production behavior from bare mode** — bare mode is dev-only, ctx.privileged is fake
|
||||||
2. **temp/ directory** — Non-standard in Rust projects
|
- **Do NOT use `detect()`/`adopt()` PM methods in production** — PM selection must come from env config
|
||||||
3. **Certificates in source** — artifact workshop-cert/certs/ not excluded
|
- **`drop_after < DepsUser` is FORBIDDEN** — will break system dependency installation (needs root)
|
||||||
4. **Deprecated projects** — workshop-executor.old, workshop-monitor.old still present
|
- **Do NOT add magic URL inference** — URL is URL, behavior bound explicitly via `getter::` prefix
|
||||||
5. **Hardcoded mirrors** — Tsinghua University mirrors in configs
|
- **Do NOT use `@` for version locking in URLs** — it's an auth delimiter; use `?ref=`
|
||||||
6. **Rust 2024 edition** — May not work with stable toolchains
|
- **Do NOT duplicate type definitions across modules** — single source of truth in `types/`
|
||||||
|
- **Careful with `unimplemented!()`** — Daemon mode, Rhai/Dylib plugins, and Custom builder are stubs
|
||||||
## UNIQUE STYLES
|
- **Security**: PIPELINE_CONTAINER_PRIVILEGED, PIPELINE_BAREMETAL_ELEVATE, PIPELINE_UNCOVER_SECRET require explicit approval + audit
|
||||||
|
|
||||||
- **Pipeline DSL**: Shell scripts with `# @decorator` annotations
|
|
||||||
- **Dual-mode baker**: CLI commands + daemon mode
|
|
||||||
- **LLM cookbook system**: YAML cookbooks for language-specific builds
|
|
||||||
- **Chinese docs** — Documentation primarily in zh-CN
|
|
||||||
- **User "vulcan"** — Custom build user (not root/runner)
|
|
||||||
|
|
||||||
## COMMANDS
|
## COMMANDS
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Build
|
# Build (per crate, no workspace)
|
||||||
cargo build --release -p workshop-baker
|
cd workshop-baker && cargo build --all-features
|
||||||
|
|
||||||
# Test
|
# Test (per crate)
|
||||||
cargo test -p workshop-baker
|
cd workshop-baker && cargo test --all-features
|
||||||
pytest tests/integration/test_prebake.py -v
|
cd workshop-engine && cargo test --all-features
|
||||||
|
|
||||||
# Lint
|
# Lint (CI gate)
|
||||||
cargo clippy -- -D warnings
|
cargo fmt -- --check
|
||||||
cargo fmt --all
|
cargo clippy --all-features -- -D warnings
|
||||||
|
|
||||||
# Run baker
|
# Docs
|
||||||
cargo run --bin workshop-baker -- prebake config.yml
|
cd docs && mdbook build # output in docs/book/
|
||||||
cargo run --bin workshop-baker -- bake script.sh
|
cd docs && mdbook serve # local preview at localhost:3000
|
||||||
|
|
||||||
# Run agent
|
|
||||||
cargo run --bin workshop-agent
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## NOTES
|
## NOTES
|
||||||
|
|
||||||
- **Docker required** for integration tests
|
- **No workspace Cargo.toml** at root — each crate has its own `Cargo.lock`. Build individually.
|
||||||
- **workshop-builder-native** is a stub (3-line Hello World)
|
- **CI only tests** `workshop-baker` and `workshop-engine`; smaller crates tested transitively.
|
||||||
- No root workspace Cargo.toml — each crate is independent
|
- **`quicktest.sh`** in workshop-baker/ is a dev smoke-test script (Docker-based, not CI).
|
||||||
- `AGENTS.md.old` at root — legacy documentation
|
- **Git submodules**: `rust-tongsuo`, `RustyVault` (crypto — not yet integrated into build).
|
||||||
|
- **`.secret/`** contains real credentials — never commit, never read into context.
|
||||||
|
- See `docs/src/zh-CN/vol3_dev/` for full developer documentation (architecture, design decisions, code style).
|
||||||
|
|||||||
@@ -0,0 +1,48 @@
|
|||||||
|
# DOCS KNOWLEDGE BASE
|
||||||
|
|
||||||
|
## OVERVIEW
|
||||||
|
mdBook project in zh-CN. Authoritative source for project conventions, coding style, and architecture docs. Build output at `docs/book/` (gitignored).
|
||||||
|
|
||||||
|
## STRUCTURE
|
||||||
|
```
|
||||||
|
docs/
|
||||||
|
├── book.toml # mdBook config (title, language=zh-CN)
|
||||||
|
├── src/SUMMARY.md # Sidebar nav — source of truth for page listing
|
||||||
|
└── src/zh-CN/
|
||||||
|
├── index.md # Landing
|
||||||
|
├── vol1_user/ # End-user documentation
|
||||||
|
├── vol2_admin/ # Admin reference: decorator cheatsheet, deployment,
|
||||||
|
│ # env vars, exit codes, plugin ref, security, troubleshooting
|
||||||
|
└── vol3_dev/ # Developer docs: architecture, code style, commit
|
||||||
|
# convention, testing, error handling, type system,
|
||||||
|
# design decisions, PR process
|
||||||
|
```
|
||||||
|
|
||||||
|
## WHERE TO LOOK
|
||||||
|
|
||||||
|
| Task | Location | Notes |
|
||||||
|
|------|----------|-------|
|
||||||
|
| Sidebar/nav editing | `src/SUMMARY.md` | Add entry here to surface a new page |
|
||||||
|
| Coding conventions | `vol3_dev/code_style.md` | Naming, imports, formatting |
|
||||||
|
| Commit rules | `vol3_dev/commit_convention.md` | feat/fix/docs/test/refactor/style |
|
||||||
|
| Error handling | `vol3_dev/error_handling_patterns.md` | thiserror+anyhow+HasExitCode |
|
||||||
|
| Type system | `vol3_dev/type_system_guide.md` | Newtypes, types/ module isolation |
|
||||||
|
| Testing strategy | `vol3_dev/testing_strategy.md` | 3 tiers: unit, integration, system |
|
||||||
|
| Anti-patterns | `vol3_dev/design_decisions.md` | Documented design rejections |
|
||||||
|
| Architecture | `vol3_dev/architecture.md` | Crate dependency graph, module map |
|
||||||
|
|
||||||
|
## CONVENTIONS
|
||||||
|
|
||||||
|
- **Language**: All prose in zh-CN. Code blocks in Rust (default) or bash.
|
||||||
|
- **Adding a page**: Create `.md` under `volN_*/`, then add to `SUMMARY.md`.
|
||||||
|
- **Formatting**: Standard Markdown. mdBook preprocessors handle TOC and links.
|
||||||
|
- **Cross-references**: Relative paths from source root (e.g., `../vol3_dev/code_style.md`).
|
||||||
|
- **Code blocks**: Fenced with language tag. No line numbers in docs.
|
||||||
|
- **File naming**: `snake_case.md`.
|
||||||
|
|
||||||
|
## COMMANDS
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mdbook build # static site → docs/book/
|
||||||
|
mdbook serve # live preview at localhost:3000
|
||||||
|
```
|
||||||
@@ -35,3 +35,4 @@
|
|||||||
- [自定义 Decorator](zh-CN/vol3_dev/custom_decorator.md)
|
- [自定义 Decorator](zh-CN/vol3_dev/custom_decorator.md)
|
||||||
- [模板变量](zh-CN/vol3_dev/template_variables.md)
|
- [模板变量](zh-CN/vol3_dev/template_variables.md)
|
||||||
- [邮件模板](zh-CN/vol3_dev/mail_template.md)
|
- [邮件模板](zh-CN/vol3_dev/mail_template.md)
|
||||||
|
- [Resource 模型](zh-CN/vol3_dev/resource_model.md)
|
||||||
|
|||||||
@@ -0,0 +1,88 @@
|
|||||||
|
# Resource 模型
|
||||||
|
|
||||||
|
## 核心思想
|
||||||
|
|
||||||
|
整个系统只关心两件事:东西从哪来(fetch),东西到哪去(publish)。
|
||||||
|
|
||||||
|
fetch 方向:URL → 本地路径 → 使用。
|
||||||
|
publish 方向:本地路径 → 变换(插件)→ URL。
|
||||||
|
|
||||||
|
两个方向共用同一套寻址方式,但 Resource 中间态只存在于 fetch 方向。
|
||||||
|
|
||||||
|
## URL 格式
|
||||||
|
|
||||||
|
```
|
||||||
|
getter::url?param1=value1¶m2=value2
|
||||||
|
```
|
||||||
|
|
||||||
|
getter 显式声明意图。不是"我猜这是个 git 仓库",而是"使用者告诉我这是什么"。
|
||||||
|
|
||||||
|
只有三个 getter:
|
||||||
|
|
||||||
|
- `git` — 克隆仓库,必要时 checkout 指定版本
|
||||||
|
- `http` / `https` — 下载文件,必要时校验 checksum
|
||||||
|
- `file` — 复制本地文件或目录
|
||||||
|
|
||||||
|
版本锁定不用 `@v1.0`(`@` 在 URL 里是认证分隔符),用 `?ref=v1.0`。
|
||||||
|
|
||||||
|
没有 `github.com/user/repo` 看起来应该 clone 它这种黑魔法。
|
||||||
|
没有 `.tar.gz` 看起来应该解压它的惊喜。
|
||||||
|
URL 是 URL,行为是行为,两者用 getter:: 显式绑定。
|
||||||
|
|
||||||
|
## Resource 不是网络地址
|
||||||
|
|
||||||
|
流水线阶段(prebake / finalize / notify)只接触 Resource:一个名字 + 一个本地路径,已就绪。它们不经由网络、不解析 URL、不碰 getter。
|
||||||
|
|
||||||
|
所有 fetch 行为在阶段启动前完成:
|
||||||
|
|
||||||
|
- worker 模式:bakerd fetch → 填入 ResourceRegistry → 启动 baker
|
||||||
|
- standalone 模式:baker 自己 fetch → 填入 ResourceRegistry → 执行阶段
|
||||||
|
- bare 模式:不 fetch,传入的必须是 Resource
|
||||||
|
|
||||||
|
ResourceRegistry 是名字到路径的映射。填进去就只读了。
|
||||||
|
|
||||||
|
## getterurl 是独立的 crate
|
||||||
|
|
||||||
|
URL 格式解析 + 基本 fetch 放在 workshop-getterurl,和 baker 本身没有耦合。任何人都可以用。
|
||||||
|
|
||||||
|
crate 分两层:
|
||||||
|
|
||||||
|
```
|
||||||
|
结构层(parser):
|
||||||
|
"git::https://host/repo.git?ref=v1.0" → { getter: "git", url: Url, params: Params }
|
||||||
|
|
||||||
|
获取层(builtin getters):
|
||||||
|
getter.fetch(url, params, dst) → local_path
|
||||||
|
git: git clone + optional checkout
|
||||||
|
http: HTTP GET + optional checksum
|
||||||
|
file: std::fs::copy
|
||||||
|
```
|
||||||
|
|
||||||
|
获取层是字面意义的——不解压、不推断、不猜意图。
|
||||||
|
|
||||||
|
## Checksum
|
||||||
|
|
||||||
|
checksum 作为 URL query 参数传递。三态语义:
|
||||||
|
|
||||||
|
- `?checksum=sha256:abc` — 必须匹配
|
||||||
|
- `?checksum=` — 显式禁用校验
|
||||||
|
- 无 checksum — 如果服务端返回 Content-Digest 头,校验它
|
||||||
|
|
||||||
|
getter 不主动校验,它只执行 URL 里指定的操作。
|
||||||
|
|
||||||
|
## Artifact 不经过 Resource
|
||||||
|
|
||||||
|
artifact 是 bake 的产物,自产生就开始被使用,不存在中间态。它的路径就是 bake 产出的本地路径,不走 ResourceRegistry。
|
||||||
|
|
||||||
|
artifact 的 publish 方向由插件定义行为——可能上传、可能扫描、可能只记录。插件定义最终的行为,只要是自洽的。
|
||||||
|
|
||||||
|
## 三种运行模式
|
||||||
|
|
||||||
|
```
|
||||||
|
fetch 执行者 fetch 在阶段内
|
||||||
|
daemon 模式 bakerd 否
|
||||||
|
standalone baker 自身 否(阶段启动前已完成)
|
||||||
|
bare 模式 无 否(必须传入 Resource)
|
||||||
|
```
|
||||||
|
|
||||||
|
bare 模式下传入的必须是已经就绪的 Resource。外部 URL 是 daemon 或 standalone 的事,bare 不负责。
|
||||||
@@ -1,101 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
set -euo pipefail
|
|
||||||
|
|
||||||
# HoneyBiscuitWorkshop System Test Runner
|
|
||||||
#
|
|
||||||
# Usage:
|
|
||||||
# ./runner.sh # Arch baker tests (hbw-system-test:latest)
|
|
||||||
# ./runner.sh --image hbw-test-pm:test-apt # Debian engine PM tests
|
|
||||||
# ./runner.sh -k test_install # Arch baker, filtered
|
|
||||||
|
|
||||||
MODE="auto"
|
|
||||||
TEST_IMAGE=""
|
|
||||||
TEST_DIR="tests/baker"
|
|
||||||
PYTEST_ARGS=""
|
|
||||||
|
|
||||||
while [[ $# -gt 0 ]]; do
|
|
||||||
case "$1" in
|
|
||||||
--image) TEST_IMAGE="$2"; TEST_DIR="tests/engine"; shift 2 ;;
|
|
||||||
manual) MODE="manual"; shift ;;
|
|
||||||
auto) MODE="auto"; shift ;;
|
|
||||||
--) shift; PYTEST_ARGS="$@"; break ;;
|
|
||||||
*) PYTEST_ARGS="$PYTEST_ARGS $1"; shift ;;
|
|
||||||
esac
|
|
||||||
done
|
|
||||||
|
|
||||||
# Default image: Arch-based baker test image
|
|
||||||
: ${TEST_IMAGE:="hbw-system-test:latest"}
|
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
|
||||||
CONTAINER_NAME="hbw-system-test-container"
|
|
||||||
STAGING_DIR="/tmp/hbw-system-test"
|
|
||||||
|
|
||||||
cleanup() {
|
|
||||||
docker rm -f "$CONTAINER_NAME" 2>/dev/null || true
|
|
||||||
[ "$TEST_DIR" = "tests/baker" ] && rm -rf "$STAGING_DIR"
|
|
||||||
}
|
|
||||||
trap cleanup EXIT
|
|
||||||
|
|
||||||
build_binary() {
|
|
||||||
local target="x86_64-unknown-linux-musl"
|
|
||||||
local binary="$SCRIPT_DIR/workshop-baker/target/$target/debug/workshop-baker"
|
|
||||||
if [ ! -f "$binary" ]; then
|
|
||||||
echo "[runner] Building static (musl) baker binary..."
|
|
||||||
cd "$SCRIPT_DIR/workshop-baker"
|
|
||||||
cargo build -p workshop-baker --target "$target" 2>&1 | tail -n 1
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
prepare_staging() {
|
|
||||||
# Only needed for baker tests (hbw-system-test:latest mounts /workshop)
|
|
||||||
if [ "$TEST_DIR" = "tests/baker" ]; then
|
|
||||||
mkdir -p "$STAGING_DIR"
|
|
||||||
cp "$SCRIPT_DIR/workshop-baker/target/x86_64-unknown-linux-musl/debug/workshop-baker" "$STAGING_DIR/"
|
|
||||||
chmod +x "$STAGING_DIR/workshop-baker"
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
run_container() {
|
|
||||||
local tag="${TEST_IMAGE##*:}"
|
|
||||||
local filter=""
|
|
||||||
case "$tag" in
|
|
||||||
test-arch) filter="-k pacman" ;;
|
|
||||||
test-apt) filter="-k apt" ;;
|
|
||||||
test-dnf) filter="-k dnf" ;;
|
|
||||||
test-apk) filter="-k apk" ;;
|
|
||||||
esac
|
|
||||||
|
|
||||||
echo "[runner] Starting: $TEST_IMAGE (tests=$TEST_DIR${filter:+, filter=$filter})"
|
|
||||||
mkdir -p "$SCRIPT_DIR/tests/results"
|
|
||||||
|
|
||||||
local mounts="-v $SCRIPT_DIR:/src:ro"
|
|
||||||
local entrypoint=""
|
|
||||||
|
|
||||||
if [ "$TEST_DIR" = "tests/baker" ]; then
|
|
||||||
mounts="$mounts -v $STAGING_DIR:/workshop:ro"
|
|
||||||
entrypoint='--entrypoint ""'
|
|
||||||
fi
|
|
||||||
|
|
||||||
local cmd="sh -c \"python3 -m venv /tmp/v && . /tmp/v/bin/activate && \
|
|
||||||
pip install pytest pytest-timeout -q && \
|
|
||||||
exec python -m pytest $TEST_DIR $PYTEST_ARGS $filter\""
|
|
||||||
|
|
||||||
if [ "$MODE" = "manual" ]; then
|
|
||||||
docker run -it --rm --name "$CONTAINER_NAME" --privileged $entrypoint \
|
|
||||||
$mounts "$TEST_IMAGE"
|
|
||||||
else
|
|
||||||
docker run --rm --name "$CONTAINER_NAME" --privileged $entrypoint \
|
|
||||||
$mounts -w /src \
|
|
||||||
"$TEST_IMAGE" "$cmd"
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
echo "============================================"
|
|
||||||
echo " HBW System Test Runner"
|
|
||||||
echo " Image: $TEST_IMAGE"
|
|
||||||
echo " Tests: $TEST_DIR"
|
|
||||||
echo "============================================"
|
|
||||||
build_binary
|
|
||||||
prepare_staging
|
|
||||||
run_container
|
|
||||||
echo "[runner] Done."
|
|
||||||
@@ -1,32 +0,0 @@
|
|||||||
FROM archlinux:latest
|
|
||||||
|
|
||||||
RUN sed -i 's|https://geo.mirror.pkgbuild.com|https://mirrors.tuna.tsinghua.edu.cn/archlinux|g' /etc/pacman.d/mirrorlist && \
|
|
||||||
sed -i 's|https://mirror.rackspace.com/archlinux|https://mirrors.tuna.tsinghua.edu.cn/archlinux|g' /etc/pacman.d/mirrorlist && \
|
|
||||||
echo 'Server = https://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch' >> /etc/pacman.d/mirrorlist
|
|
||||||
|
|
||||||
RUN pacman -Syu --noconfirm && \
|
|
||||||
pacman -S --noconfirm \
|
|
||||||
python python-pip base-devel git curl wget sudo \
|
|
||||||
which procps-ng shadow && \
|
|
||||||
pacman -Scc --noconfirm
|
|
||||||
|
|
||||||
# Create test user
|
|
||||||
RUN useradd -m -s /bin/bash testuser
|
|
||||||
|
|
||||||
# Python environment
|
|
||||||
WORKDIR /app
|
|
||||||
COPY tests/requirements.txt ./requirements.txt
|
|
||||||
RUN python -m venv /opt/test-venv && \
|
|
||||||
/opt/test-venv/bin/pip install -r requirements.txt
|
|
||||||
|
|
||||||
# Copy test suite
|
|
||||||
COPY tests/ ./tests/
|
|
||||||
COPY tests/entrypoint.sh ./entrypoint.sh
|
|
||||||
RUN chmod +x entrypoint.sh
|
|
||||||
|
|
||||||
# Environment
|
|
||||||
ENV PATH="/opt/test-venv/bin:$PATH"
|
|
||||||
ENV PYTHONUNBUFFERED=1
|
|
||||||
ENV PYTHONDONTWRITEBYTECODE=1
|
|
||||||
|
|
||||||
ENTRYPOINT ["./entrypoint.sh"]
|
|
||||||
@@ -1,129 +0,0 @@
|
|||||||
import os
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
|
|
||||||
FIXTURES = os.path.join(os.path.dirname(__file__), "fixtures", "bake")
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.bake
|
|
||||||
class TestBakeSimpleScript:
|
|
||||||
|
|
||||||
def test_simple_script_runs(self, run_baker, bake_workspace, work_dir):
|
|
||||||
prebake_cfg = os.path.join(
|
|
||||||
os.path.dirname(__file__), "fixtures", "prebake", "minimal.yml"
|
|
||||||
)
|
|
||||||
script = os.path.join(FIXTURES, "simple.sh")
|
|
||||||
bake_base = os.path.join(FIXTURES, "bake_base.sh")
|
|
||||||
|
|
||||||
result = run_baker(
|
|
||||||
"bake", script,
|
|
||||||
"--bake-base", bake_base,
|
|
||||||
"--prebake", prebake_cfg,
|
|
||||||
cwd=work_dir,
|
|
||||||
timeout=60,
|
|
||||||
)
|
|
||||||
assert result["success"], f"Bake failed: {result['stderr']}"
|
|
||||||
assert "Hello from simple bake script" in result["stdout"]
|
|
||||||
|
|
||||||
def test_simple_script_sets_env_vars(self, run_baker, bake_workspace, work_dir):
|
|
||||||
prebake_cfg = os.path.join(
|
|
||||||
os.path.dirname(__file__), "fixtures", "prebake", "minimal.yml"
|
|
||||||
)
|
|
||||||
script = os.path.join(FIXTURES, "simple.sh")
|
|
||||||
bake_base = os.path.join(FIXTURES, "bake_base.sh")
|
|
||||||
|
|
||||||
result = run_baker(
|
|
||||||
"bake", script,
|
|
||||||
"--bake-base", bake_base,
|
|
||||||
"--prebake", prebake_cfg,
|
|
||||||
cwd=work_dir,
|
|
||||||
timeout=60,
|
|
||||||
)
|
|
||||||
assert result["success"]
|
|
||||||
assert "test-pipeline" in result["stdout"]
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.bake
|
|
||||||
class TestBakePipeline:
|
|
||||||
|
|
||||||
def test_pipeline_functions_execute(self, run_baker, bake_workspace, work_dir):
|
|
||||||
prebake_cfg = os.path.join(
|
|
||||||
os.path.dirname(__file__), "fixtures", "prebake", "minimal.yml"
|
|
||||||
)
|
|
||||||
script = os.path.join(FIXTURES, "pipeline.sh")
|
|
||||||
bake_base = os.path.join(FIXTURES, "bake_base.sh")
|
|
||||||
|
|
||||||
result = run_baker(
|
|
||||||
"bake", script,
|
|
||||||
"--bake-base", bake_base,
|
|
||||||
"--prebake", prebake_cfg,
|
|
||||||
cwd=work_dir,
|
|
||||||
timeout=60,
|
|
||||||
)
|
|
||||||
assert result["success"], f"Pipeline bake failed: {result['stderr']}"
|
|
||||||
assert os.path.exists("/tmp/bake-test-output/step1.txt")
|
|
||||||
assert os.path.exists("/tmp/bake-test-output/step2.txt")
|
|
||||||
|
|
||||||
def test_pipeline_step_outputs_correct(self, run_baker, bake_workspace, work_dir):
|
|
||||||
prebake_cfg = os.path.join(
|
|
||||||
os.path.dirname(__file__), "fixtures", "prebake", "minimal.yml"
|
|
||||||
)
|
|
||||||
script = os.path.join(FIXTURES, "pipeline.sh")
|
|
||||||
bake_base = os.path.join(FIXTURES, "bake_base.sh")
|
|
||||||
|
|
||||||
result = run_baker(
|
|
||||||
"bake", script,
|
|
||||||
"--bake-base", bake_base,
|
|
||||||
"--prebake", prebake_cfg,
|
|
||||||
cwd=work_dir,
|
|
||||||
timeout=60,
|
|
||||||
)
|
|
||||||
assert result["success"]
|
|
||||||
with open("/tmp/bake-test-output/step1.txt") as f:
|
|
||||||
assert f.read().strip() == "step1-done"
|
|
||||||
with open("/tmp/bake-test-output/step2.txt") as f:
|
|
||||||
assert f.read().strip() == "step2-done"
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.bake
|
|
||||||
class TestBakeAfterDependencies:
|
|
||||||
|
|
||||||
def test_after_ordering(self, run_baker, bake_workspace, work_dir):
|
|
||||||
prebake_cfg = os.path.join(
|
|
||||||
os.path.dirname(__file__), "fixtures", "prebake", "minimal.yml"
|
|
||||||
)
|
|
||||||
script = os.path.join(FIXTURES, "with_after.sh")
|
|
||||||
bake_base = os.path.join(FIXTURES, "bake_base.sh")
|
|
||||||
|
|
||||||
result = run_baker(
|
|
||||||
"bake", script,
|
|
||||||
"--bake-base", bake_base,
|
|
||||||
"--prebake", prebake_cfg,
|
|
||||||
cwd=work_dir,
|
|
||||||
timeout=60,
|
|
||||||
)
|
|
||||||
assert result["success"], f"@after bake failed: {result['stderr']}"
|
|
||||||
assert os.path.exists("/tmp/bake-after-test/status.txt")
|
|
||||||
assert os.path.exists("/tmp/bake-after-test/source.txt")
|
|
||||||
assert os.path.exists("/tmp/bake-after-test/build.txt")
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.bake
|
|
||||||
class TestBakeStatusFiles:
|
|
||||||
|
|
||||||
def test_status_written_after_bake(self, run_baker, bake_workspace, work_dir):
|
|
||||||
prebake_cfg = os.path.join(
|
|
||||||
os.path.dirname(__file__), "fixtures", "prebake", "minimal.yml"
|
|
||||||
)
|
|
||||||
script = os.path.join(FIXTURES, "simple.sh")
|
|
||||||
bake_base = os.path.join(FIXTURES, "bake_base.sh")
|
|
||||||
|
|
||||||
result = run_baker(
|
|
||||||
"bake", script,
|
|
||||||
"--bake-base", bake_base,
|
|
||||||
"--prebake", prebake_cfg,
|
|
||||||
cwd=work_dir,
|
|
||||||
timeout=60,
|
|
||||||
)
|
|
||||||
assert result["success"]
|
|
||||||
assert os.path.exists("/tmp/.hbwstatus")
|
|
||||||
@@ -1,77 +0,0 @@
|
|||||||
import os
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from probes import file
|
|
||||||
from llm import judge_log, LLMClient
|
|
||||||
|
|
||||||
|
|
||||||
FIXTURES = os.path.join(os.path.dirname(__file__), "fixtures", "finalize")
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.finalize
|
|
||||||
class TestFinalizeMinimal:
|
|
||||||
|
|
||||||
def test_minimal_finalize_succeeds(self, run_baker, work_dir):
|
|
||||||
config = os.path.join(FIXTURES, "minimal.yml")
|
|
||||||
result = run_baker("finalize", config, cwd=work_dir, timeout=60)
|
|
||||||
assert result["success"], f"Minimal finalize failed: {result['stderr']}"
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.finalize
|
|
||||||
class TestFinalizeHooks:
|
|
||||||
|
|
||||||
def test_early_hook_executes(self, run_baker, work_dir):
|
|
||||||
config = os.path.join(FIXTURES, "with_hooks.yml")
|
|
||||||
result = run_baker("finalize", config, cwd=work_dir, timeout=60)
|
|
||||||
assert result["success"]
|
|
||||||
assert "finalize early hook" in result["stdout"] or "finalize early hook" in result["stderr"]
|
|
||||||
|
|
||||||
def test_late_hook_executes(self, run_baker, work_dir):
|
|
||||||
config = os.path.join(FIXTURES, "with_hooks.yml")
|
|
||||||
result = run_baker("finalize", config, cwd=work_dir, timeout=60)
|
|
||||||
assert result["success"]
|
|
||||||
assert "finalize late hook" in result["stdout"] or "finalize late hook" in result["stderr"]
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.finalize
|
|
||||||
class TestFinalizeStatus:
|
|
||||||
|
|
||||||
def test_finalize_writes_status(self, run_baker, work_dir):
|
|
||||||
config = os.path.join(FIXTURES, "minimal.yml")
|
|
||||||
result = run_baker("finalize", config, cwd=work_dir, timeout=60)
|
|
||||||
assert result["success"]
|
|
||||||
assert os.path.exists("/tmp/.hbwstatus")
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.finalize
|
|
||||||
@pytest.mark.llm
|
|
||||||
class TestFinalizeWithLLM:
|
|
||||||
|
|
||||||
def test_finalize_hooks_via_llm(self, run_baker, llm_env, work_dir):
|
|
||||||
if not llm_env:
|
|
||||||
pytest.skip("LLM environment not configured")
|
|
||||||
|
|
||||||
config = os.path.join(FIXTURES, "with_hooks.yml")
|
|
||||||
result = run_baker("finalize", config, cwd=work_dir, timeout=60)
|
|
||||||
|
|
||||||
combined_log = result["stdout"] + "\n" + result["stderr"]
|
|
||||||
|
|
||||||
judgment = judge_log(
|
|
||||||
operation="finalize with early and late hooks",
|
|
||||||
expected_behavior="Both early and late hooks executed successfully, "
|
|
||||||
"printing 'finalize early hook' and 'finalize late hook'",
|
|
||||||
actual_log=combined_log,
|
|
||||||
)
|
|
||||||
assert judgment.passed, f"LLM judgment failed: {judgment.reason}"
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.finalize
|
|
||||||
class TestFinalizePlugins:
|
|
||||||
|
|
||||||
@pytest.mark.xfail(reason="Plugin download requires network/git access", strict=False)
|
|
||||||
def test_plugin_download(self, run_baker, work_dir):
|
|
||||||
pass
|
|
||||||
|
|
||||||
@pytest.mark.xfail(reason="Artifact stage not yet implemented", strict=False)
|
|
||||||
def test_artifact_collection(self, run_baker, work_dir):
|
|
||||||
pass
|
|
||||||
@@ -1,261 +0,0 @@
|
|||||||
import os
|
|
||||||
import json
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from probes import user, file, package
|
|
||||||
|
|
||||||
|
|
||||||
FIXTURES = os.path.join(os.path.dirname(__file__), "fixtures", "prebake")
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.prebake
|
|
||||||
class TestPrebakeConfigValidation:
|
|
||||||
|
|
||||||
def test_invalid_version_fails(self, run_baker, work_dir):
|
|
||||||
config = os.path.join(FIXTURES, "invalid_version.yml")
|
|
||||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
|
||||||
assert not result["success"], f"Expected prebake to fail with invalid version"
|
|
||||||
assert "version" in result["stderr"].lower() or "Version" in result["stderr"]
|
|
||||||
|
|
||||||
def test_missing_required_field_fails(self, run_baker, work_dir):
|
|
||||||
missing_env = os.path.join(work_dir, "missing_env.yml")
|
|
||||||
with open(missing_env, "w") as f:
|
|
||||||
f.write('version: "1.0"\n')
|
|
||||||
result = run_baker("prebake", missing_env, cwd=work_dir, timeout=60)
|
|
||||||
assert not result["success"]
|
|
||||||
|
|
||||||
def test_minimal_config_succeeds(self, run_baker, work_dir):
|
|
||||||
config = os.path.join(FIXTURES, "minimal.yml")
|
|
||||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
|
||||||
assert result["success"], f"Minimal prebake failed: {result['stderr']}"
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.prebake
|
|
||||||
class TestPrebakeBootstrap:
|
|
||||||
|
|
||||||
def test_bootstrap_creates_vulcan_user(self, run_baker, work_dir):
|
|
||||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
|
||||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
|
||||||
assert result["success"], f"prebake failed: {result['stderr']}"
|
|
||||||
assert user.user_exists("vulcan")
|
|
||||||
|
|
||||||
def test_bootstrap_creates_workspace(self, run_baker, work_dir):
|
|
||||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
|
||||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
|
||||||
assert result["success"]
|
|
||||||
assert file.dir_exists("/home/vulcan/workspace")
|
|
||||||
|
|
||||||
def test_bootstrap_creates_symlink(self, run_baker, work_dir):
|
|
||||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
|
||||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
|
||||||
assert result["success"]
|
|
||||||
assert os.path.islink("/workspace")
|
|
||||||
|
|
||||||
def test_bootstrap_generates_bootstrap_script(self, run_baker, work_dir):
|
|
||||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
|
||||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
|
||||||
assert result["success"]
|
|
||||||
assert file.file_exists("/bootstrap.sh")
|
|
||||||
|
|
||||||
def test_bootstrap_script_permissions(self, run_baker, work_dir):
|
|
||||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
|
||||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
|
||||||
assert result["success"]
|
|
||||||
mode = file.file_permissions("/bootstrap.sh")
|
|
||||||
assert mode & 0o111, f"/bootstrap.sh not executable, mode: {oct(mode)}"
|
|
||||||
|
|
||||||
def test_bootstrap_script_contains_user_creation(self, run_baker, work_dir):
|
|
||||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
|
||||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
|
||||||
assert result["success"]
|
|
||||||
assert file.file_exists("/bootstrap.sh")
|
|
||||||
assert file.file_contains("/bootstrap.sh", "useradd")
|
|
||||||
|
|
||||||
def test_bootstrap_script_contains_workspace_setup(self, run_baker, work_dir):
|
|
||||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
|
||||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
|
||||||
assert result["success"]
|
|
||||||
assert file.file_contains("/bootstrap.sh", "mkdir -p /home/vulcan/workspace")
|
|
||||||
|
|
||||||
def test_bootstrap_with_pacman_creates_sudoers(self, run_baker, work_dir):
|
|
||||||
config = os.path.join(FIXTURES, "with_pacman.yml")
|
|
||||||
result = run_baker("prebake", config, cwd=work_dir, timeout=120)
|
|
||||||
assert result["success"], f"prebake failed: {result['stderr']}"
|
|
||||||
assert file.file_exists("/etc/sudoers.d/workshop")
|
|
||||||
|
|
||||||
def test_custom_bootstrap_executes(self, run_baker, work_dir):
|
|
||||||
config = os.path.join(FIXTURES, "with_custom_bootstrap.yml")
|
|
||||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
|
||||||
assert result["success"], f"Custom bootstrap failed: {result['stderr']}"
|
|
||||||
assert file.dir_exists("/custom-workspace")
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.prebake
|
|
||||||
class TestPrebakeDryRun:
|
|
||||||
|
|
||||||
def test_dry_run_does_not_create_user(self, run_baker, work_dir):
|
|
||||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
|
||||||
env = {"HBW_DRY_RUN": "true"}
|
|
||||||
result = run_baker("prebake", config, cwd=work_dir, env=env, timeout=60)
|
|
||||||
assert result["success"], f"Dry-run failed: {result['stderr']}"
|
|
||||||
|
|
||||||
def test_dry_run_logs_script_content(self, run_baker, work_dir):
|
|
||||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
|
||||||
env = {"HBW_DRY_RUN": "true", "RUST_LOG": "info"}
|
|
||||||
result = run_baker("prebake", config, cwd=work_dir, env=env, timeout=60)
|
|
||||||
assert result["success"]
|
|
||||||
combined = result["stdout"] + result["stderr"]
|
|
||||||
assert "DRY_RUN" in combined or "dry_run" in combined.lower()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.prebake
|
|
||||||
class TestPrebakeHooks:
|
|
||||||
|
|
||||||
#@pytest.mark.xfail(reason="Hook execution fails with Permission denied in container environment", strict=False)
|
|
||||||
def test_early_hook_executes(self, run_baker, work_dir):
|
|
||||||
config = os.path.join(FIXTURES, "with_hooks.yml")
|
|
||||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
|
||||||
assert result["success"], f"prebake with hooks failed: {result['stderr']}"
|
|
||||||
combined = result["stdout"] + result["stderr"]
|
|
||||||
assert "early hook executed" in combined
|
|
||||||
|
|
||||||
#@pytest.mark.xfail(reason="Hook execution fails with Permission denied in container environment", strict=False)
|
|
||||||
def test_late_hook_executes(self, run_baker, work_dir):
|
|
||||||
config = os.path.join(FIXTURES, "with_hooks.yml")
|
|
||||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
|
||||||
assert result["success"], f"prebake with hooks failed: {result['stderr']}"
|
|
||||||
combined = result["stdout"] + result["stderr"]
|
|
||||||
assert "late hook executed" in combined
|
|
||||||
|
|
||||||
#@pytest.mark.xfail(reason="Hook execution fails with Permission denied in container environment", strict=False)
|
|
||||||
def test_hooks_run_in_order(self, run_baker, work_dir):
|
|
||||||
config = os.path.join(FIXTURES, "with_hooks.yml")
|
|
||||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
|
||||||
assert result["success"]
|
|
||||||
combined = result["stdout"] + result["stderr"]
|
|
||||||
early_pos = combined.find("early hook executed")
|
|
||||||
late_pos = combined.find("late hook executed")
|
|
||||||
assert early_pos != -1 and late_pos != -1
|
|
||||||
assert early_pos < late_pos, "Early hook should execute before late hook"
|
|
||||||
|
|
||||||
def test_empty_hooks_succeed(self, run_baker, work_dir):
|
|
||||||
config = os.path.join(FIXTURES, "minimal.yml")
|
|
||||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
|
||||||
assert result["success"], f"prebake with no hooks failed: {result['stderr']}"
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.prebake
|
|
||||||
class TestPrebakeDepsSystem:
|
|
||||||
|
|
||||||
@pytest.mark.xfail(reason="Package installation may fail in test environment", strict=False)
|
|
||||||
def test_pacman_packages_installed(self, run_baker, work_dir):
|
|
||||||
config = os.path.join(FIXTURES, "with_pacman.yml")
|
|
||||||
result = run_baker("prebake", config, cwd=work_dir, timeout=120)
|
|
||||||
assert result["success"], f"prebake failed: {result['stderr']}"
|
|
||||||
assert package.package_installed("curl")
|
|
||||||
assert package.package_installed("wget")
|
|
||||||
|
|
||||||
def test_pacman_mirror_change(self, run_baker, work_dir):
|
|
||||||
mirror_config = os.path.join(work_dir, "mirror.yml")
|
|
||||||
with open(mirror_config, "w") as f:
|
|
||||||
f.write('''
|
|
||||||
version: "1.0"
|
|
||||||
environment:
|
|
||||||
builder: "baremetal"
|
|
||||||
bootstrap:
|
|
||||||
user: "vulcan"
|
|
||||||
dependencies:
|
|
||||||
system:
|
|
||||||
pacman:
|
|
||||||
packages: ["which"]
|
|
||||||
mirror: "https://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch"
|
|
||||||
''')
|
|
||||||
result = run_baker("prebake", mirror_config, cwd=work_dir, timeout=120)
|
|
||||||
assert result["success"], f"prebake with mirror failed: {result['stderr']}"
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.prebake
|
|
||||||
class TestPrebakeEnvVars:
|
|
||||||
|
|
||||||
def test_prebake_envvars_injected(self, run_baker, work_dir):
|
|
||||||
hook_config = os.path.join(work_dir, "envvar_hook.yml")
|
|
||||||
with open(hook_config, "w") as f:
|
|
||||||
f.write('''
|
|
||||||
version: "1.0"
|
|
||||||
environment:
|
|
||||||
builder: "baremetal"
|
|
||||||
bootstrap:
|
|
||||||
user: "vulcan"
|
|
||||||
envvars:
|
|
||||||
prebake:
|
|
||||||
TEST_INJECTED: "injected_value"
|
|
||||||
hooks:
|
|
||||||
early:
|
|
||||||
- name: "check-env"
|
|
||||||
command: "echo $TEST_INJECTED"
|
|
||||||
''')
|
|
||||||
result = run_baker("prebake", hook_config, cwd=work_dir, timeout=60)
|
|
||||||
assert result["success"], f"prebake with envvars failed: {result['stderr']}"
|
|
||||||
combined = result["stdout"] + result["stderr"]
|
|
||||||
assert "injected_value" in combined
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.prebake
|
|
||||||
class TestPrebakeSecurity:
|
|
||||||
|
|
||||||
#@pytest.mark.xfail(reason="Privilege drop may fail in container without proper setup", strict=False)
|
|
||||||
def test_privilege_drop_after_bootstrap(self, run_baker, work_dir):
|
|
||||||
config = os.path.join(FIXTURES, "with_security.yml")
|
|
||||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
|
||||||
assert result["success"], f"prebake with security failed: {result['stderr']}"
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.prebake
|
|
||||||
class TestPrebakeStatus:
|
|
||||||
|
|
||||||
def test_status_files_written(self, run_baker, work_dir):
|
|
||||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
|
||||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
|
||||||
assert result["success"]
|
|
||||||
assert os.path.exists("/tmp/.hbwstatus")
|
|
||||||
|
|
||||||
def test_status_contains_bootstrap_stage(self, run_baker, work_dir):
|
|
||||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
|
||||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
|
||||||
assert result["success"]
|
|
||||||
with open("/tmp/.hbwstatus", "r") as f:
|
|
||||||
lines = f.readlines()
|
|
||||||
assert len(lines) > 0
|
|
||||||
bootstrap_lines = [l for l in lines if json.loads(l).get("name") == "bootstrap"]
|
|
||||||
assert len(bootstrap_lines) > 0, "No bootstrap stage found in status file"
|
|
||||||
data = json.loads(bootstrap_lines[-1])
|
|
||||||
assert data.get("phase") == "Prebake"
|
|
||||||
assert data.get("result") == "Success"
|
|
||||||
|
|
||||||
def test_status_contains_duration(self, run_baker, work_dir):
|
|
||||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
|
||||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
|
||||||
assert result["success"]
|
|
||||||
with open("/tmp/.hbwstatus", "r") as f:
|
|
||||||
data = json.loads(f.readline())
|
|
||||||
assert "duration_ms" in data
|
|
||||||
assert isinstance(data["duration_ms"], int)
|
|
||||||
|
|
||||||
def test_status_contains_timestamps(self, run_baker, work_dir):
|
|
||||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
|
||||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
|
||||||
assert result["success"]
|
|
||||||
with open("/tmp/.hbwstatus", "r") as f:
|
|
||||||
data = json.loads(f.readline())
|
|
||||||
assert "started_at" in data
|
|
||||||
assert "finished_at" in data
|
|
||||||
|
|
||||||
def test_status_written_for_each_stage(self, run_baker, work_dir):
|
|
||||||
config = os.path.join(FIXTURES, "with_pacman.yml")
|
|
||||||
result = run_baker("prebake", config, cwd=work_dir, timeout=120)
|
|
||||||
assert result["success"], f"prebake failed: {result['stderr']}"
|
|
||||||
with open("/tmp/.hbwstatus", "r") as f:
|
|
||||||
lines = f.readlines()
|
|
||||||
names = [json.loads(line).get("name") for line in lines]
|
|
||||||
assert "bootstrap" in names
|
|
||||||
assert "depssystem" in names
|
|
||||||
@@ -1,164 +0,0 @@
|
|||||||
"""
|
|
||||||
Root conftest for HoneyBiscuitWorkshop system tests.
|
|
||||||
|
|
||||||
Provides core fixtures for:
|
|
||||||
- Locating the baker binary and examples
|
|
||||||
- Running subprocess commands with output capture
|
|
||||||
- Managing temporary working directories
|
|
||||||
"""
|
|
||||||
import os
|
|
||||||
import subprocess
|
|
||||||
import shutil
|
|
||||||
import tempfile
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
|
|
||||||
def pytest_configure(config):
|
|
||||||
"""Register custom markers."""
|
|
||||||
config.addinivalue_line("markers", "prebake: prebake stage tests")
|
|
||||||
config.addinivalue_line("markers", "bake: bake stage tests")
|
|
||||||
config.addinivalue_line("markers", "finalize: finalize stage tests")
|
|
||||||
config.addinivalue_line("markers", "llm: tests requiring LLM API")
|
|
||||||
config.addinivalue_line("markers", "slow: slow running tests")
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture(scope="session")
|
|
||||||
def baker_bin():
|
|
||||||
"""Path to the workshop-baker binary."""
|
|
||||||
path = os.environ.get("BAKER_BIN", "/workshop/workshop-baker")
|
|
||||||
if not os.path.isfile(path):
|
|
||||||
pytest.skip(f"Baker binary not found at {path}")
|
|
||||||
if not os.access(path, os.X_OK):
|
|
||||||
pytest.skip(f"Baker binary not executable at {path}")
|
|
||||||
return path
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture(scope="session")
|
|
||||||
def examples_dir():
|
|
||||||
"""Path to the examples directory (may be None)."""
|
|
||||||
path = os.environ.get("EXAMPLES_DIR", "/workshop/examples")
|
|
||||||
if not os.path.isdir(path):
|
|
||||||
return None
|
|
||||||
return path
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture(scope="session")
|
|
||||||
def llm_env():
|
|
||||||
"""LLM environment variables dict (from llm.env)."""
|
|
||||||
env = {}
|
|
||||||
for var in [
|
|
||||||
"OPENAI_BASE_URL",
|
|
||||||
"OPENAI_API_KEY",
|
|
||||||
"ANTHROPIC_BASE_URL",
|
|
||||||
"ANTHROPIC_API_KEY",
|
|
||||||
"MODEL",
|
|
||||||
"REASONING_EFFORT",
|
|
||||||
]:
|
|
||||||
val = os.environ.get(var)
|
|
||||||
if val:
|
|
||||||
env[var] = val
|
|
||||||
return env
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def work_dir():
|
|
||||||
"""Create a temporary working directory for a test, cleaned up after."""
|
|
||||||
d = tempfile.mkdtemp(prefix="hbw-test-")
|
|
||||||
yield d
|
|
||||||
shutil.rmtree(d, ignore_errors=True)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def run_cmd():
|
|
||||||
"""Fixture that provides a command runner function.
|
|
||||||
|
|
||||||
Returns a function that runs a command and returns
|
|
||||||
(stdout, stderr, returncode, success).
|
|
||||||
"""
|
|
||||||
|
|
||||||
def _run(args, cwd=None, env=None, timeout=120):
|
|
||||||
"""Run a command and capture output.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
args: Command as list of strings.
|
|
||||||
cwd: Working directory.
|
|
||||||
env: Environment variables (merged with os.environ).
|
|
||||||
timeout: Timeout in seconds.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
dict with keys: stdout, stderr, returncode, success
|
|
||||||
"""
|
|
||||||
full_env = {**os.environ}
|
|
||||||
if env:
|
|
||||||
full_env.update(env)
|
|
||||||
|
|
||||||
try:
|
|
||||||
result = subprocess.run(
|
|
||||||
args,
|
|
||||||
cwd=cwd,
|
|
||||||
env=full_env,
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
timeout=timeout,
|
|
||||||
)
|
|
||||||
return {
|
|
||||||
"stdout": result.stdout,
|
|
||||||
"stderr": result.stderr,
|
|
||||||
"returncode": result.returncode,
|
|
||||||
"success": result.returncode == 0,
|
|
||||||
}
|
|
||||||
except subprocess.TimeoutExpired:
|
|
||||||
return {
|
|
||||||
"stdout": "",
|
|
||||||
"stderr": f"Command timed out after {timeout}s",
|
|
||||||
"returncode": -1,
|
|
||||||
"success": False,
|
|
||||||
}
|
|
||||||
except Exception as e:
|
|
||||||
return {
|
|
||||||
"stdout": "",
|
|
||||||
"stderr": str(e),
|
|
||||||
"returncode": -1,
|
|
||||||
"success": False,
|
|
||||||
}
|
|
||||||
|
|
||||||
return _run
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def run_baker(baker_bin, run_cmd):
|
|
||||||
"""Fixture that provides a baker command runner.
|
|
||||||
|
|
||||||
Returns a function that runs workshop-baker with common flags.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def _run(subcommand, *args, cwd=None, env=None, timeout=120):
|
|
||||||
cmd = [
|
|
||||||
baker_bin,
|
|
||||||
"-u", "testuser",
|
|
||||||
"-p", "test-pipeline",
|
|
||||||
"-b", "test-build-001",
|
|
||||||
"bare",
|
|
||||||
subcommand,
|
|
||||||
]
|
|
||||||
cmd.extend(args)
|
|
||||||
return run_cmd(cmd, cwd=cwd, env=env, timeout=timeout)
|
|
||||||
|
|
||||||
return _run
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def fixtures_dir():
|
|
||||||
"""Path to the test fixtures directory."""
|
|
||||||
return os.path.join(os.path.dirname(__file__), "fixtures")
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def bake_workspace():
|
|
||||||
"""Create /workspace for bake tests (baker writes temp scripts there)."""
|
|
||||||
existed = os.path.exists("/workspace")
|
|
||||||
if not existed:
|
|
||||||
os.makedirs("/workspace", exist_ok=True)
|
|
||||||
yield
|
|
||||||
if not existed:
|
|
||||||
shutil.rmtree("/workspace", ignore_errors=True)
|
|
||||||
@@ -1,102 +0,0 @@
|
|||||||
"""PM system tests driven by bare workshop-baker prebake."""
|
|
||||||
|
|
||||||
import os
|
|
||||||
import pytest
|
|
||||||
from probes import binary_exists
|
|
||||||
|
|
||||||
PACMAN_MIRROR = "https://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch"
|
|
||||||
|
|
||||||
|
|
||||||
def _yml(pm, pkgs, mirror="", repo=None):
|
|
||||||
"""Build prebake.yml. repo is a list of lines (without lead indent)."""
|
|
||||||
m = ""
|
|
||||||
if mirror:
|
|
||||||
m += ' mirror: "' + mirror + '"\n'
|
|
||||||
if repo:
|
|
||||||
m += " repositories:\n"
|
|
||||||
for line in repo:
|
|
||||||
m += " " + line + "\n"
|
|
||||||
return """\
|
|
||||||
version: "1.0"
|
|
||||||
environment:
|
|
||||||
builder: "baremetal"
|
|
||||||
bootstrap:
|
|
||||||
user: "vulcan"
|
|
||||||
dependencies:
|
|
||||||
system:
|
|
||||||
{pm}:
|
|
||||||
packages: [{pkgs}]
|
|
||||||
{m}""".format(pm=pm, pkgs=pkgs, m=m)
|
|
||||||
|
|
||||||
|
|
||||||
# ── install ──────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
@pytest.mark.parametrize("pm_name,pkgs", [
|
|
||||||
("pacman", '"curl", "git"'),
|
|
||||||
("apt", '"curl", "git"'),
|
|
||||||
("dnf", '"curl", "git"'),
|
|
||||||
("apk", '"curl", "git"'),
|
|
||||||
])
|
|
||||||
def test_install(run_cmd, work_dir, pm_name, pkgs):
|
|
||||||
cfg = os.path.join(work_dir, "pm.yml")
|
|
||||||
mirror = PACMAN_MIRROR if pm_name == "pacman" else ""
|
|
||||||
with open(cfg, "w") as f:
|
|
||||||
f.write(_yml(pm_name, pkgs, mirror=mirror))
|
|
||||||
r = run_cmd(["/baker", "-p", "t", "-b", "1", "-u", "testuser",
|
|
||||||
"bare", "prebake", cfg], timeout=120)
|
|
||||||
assert r["success"], "[{}] prebake: {}".format(pm_name, r["stderr"][:300])
|
|
||||||
for pkg in ["curl", "git"]:
|
|
||||||
assert binary_exists("/usr/bin/" + pkg), "[{}] {} missing".format(pm_name, pkg)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize("pm_name", ["pacman", "apt", "dnf", "apk"])
|
|
||||||
def test_reinstall(run_cmd, work_dir, pm_name):
|
|
||||||
cfg = os.path.join(work_dir, "pm.yml")
|
|
||||||
mirror = PACMAN_MIRROR if pm_name == "pacman" else ""
|
|
||||||
with open(cfg, "w") as f:
|
|
||||||
f.write(_yml(pm_name, '"curl"', mirror=mirror))
|
|
||||||
cmd = ["/baker", "-p", "t", "-b", "1", "-u", "testuser", "bare", "prebake", cfg]
|
|
||||||
r1 = run_cmd(cmd, timeout=120)
|
|
||||||
assert r1["success"], "[{}] first install failed".format(pm_name)
|
|
||||||
r2 = run_cmd(cmd, timeout=120)
|
|
||||||
assert r2["success"], "[{}] reinstall failed".format(pm_name)
|
|
||||||
|
|
||||||
|
|
||||||
# ── change_mirror ────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
@pytest.mark.parametrize("pm_name", ["pacman", "apt", "dnf", "apk"])
|
|
||||||
def test_mirror(run_cmd, work_dir, pm_name):
|
|
||||||
cfg = os.path.join(work_dir, "pm.yml")
|
|
||||||
mirror = PACMAN_MIRROR if pm_name == "pacman" else ""
|
|
||||||
with open(cfg, "w") as f:
|
|
||||||
f.write(_yml(pm_name, '"which"', mirror=mirror))
|
|
||||||
r = run_cmd(["/baker", "-p", "t", "-b", "1", "-u", "testuser",
|
|
||||||
"bare", "prebake", cfg], timeout=120)
|
|
||||||
assert r["success"], "[{}] mirror: {}".format(pm_name, r["stderr"][:300])
|
|
||||||
assert binary_exists("/usr/bin/which"), "[{}] which not installed".format(pm_name)
|
|
||||||
|
|
||||||
|
|
||||||
# ── add_repository ───────────────────────────────────────────────────
|
|
||||||
|
|
||||||
@pytest.mark.parametrize("pm_name,repo_lines,result_file", [
|
|
||||||
("pacman", ['- name: custom', ' server: "https://repo.example.com/$arch"'],
|
|
||||||
"/etc/pacman.conf"),
|
|
||||||
("apt", ['- url: "https://repo.example.com/debian"', ' distro: bookworm'],
|
|
||||||
"/etc/apt/sources.list.d/custom.list"),
|
|
||||||
("dnf", ['- name: custom', ' baseurl: "https://repo.example.com/fedora"'],
|
|
||||||
"/etc/yum.repos.d/custom.repo"),
|
|
||||||
("apk", ['- url: "https://repo.example.com/v3.19/main"'],
|
|
||||||
"/etc/apk/repositories"),
|
|
||||||
])
|
|
||||||
@pytest.mark.xfail(reason="fake repo URL, package install may fail", strict=False)
|
|
||||||
def test_add_repo(run_cmd, work_dir, pm_name, repo_lines, result_file):
|
|
||||||
cfg = os.path.join(work_dir, "pm.yml")
|
|
||||||
mirror = PACMAN_MIRROR if pm_name == "pacman" else ""
|
|
||||||
with open(cfg, "w") as f:
|
|
||||||
f.write(_yml(pm_name, '"curl"', mirror=mirror, repo=repo_lines))
|
|
||||||
r = run_cmd(["/baker", "-p", "t", "-b", "1", "-u", "testuser",
|
|
||||||
"bare", "prebake", cfg], timeout=120)
|
|
||||||
# Prebake may fail if repo URL is unreachable — check only the config file
|
|
||||||
# was written by add_repository before the (possibly failing) install step.
|
|
||||||
assert binary_exists(result_file), \
|
|
||||||
"[{}] repo file missing: {}".format(pm_name, result_file)
|
|
||||||
@@ -1,36 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
set -e
|
|
||||||
|
|
||||||
# Load LLM environment if available
|
|
||||||
if [ -f /workshop/llm.env ]; then
|
|
||||||
set -a
|
|
||||||
source /workshop/llm.env
|
|
||||||
set +a
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Export test configuration
|
|
||||||
export BAKER_BIN=/workshop/workshop-baker
|
|
||||||
export EXAMPLES_DIR=/workshop/examples
|
|
||||||
export TEST_MODE=${TEST_MODE:-auto}
|
|
||||||
|
|
||||||
if [ "$TEST_MODE" = "manual" ]; then
|
|
||||||
echo "========================================"
|
|
||||||
echo " HoneyBiscuitWorkshop System Tests"
|
|
||||||
echo " Mode: MANUAL (interactive shell)"
|
|
||||||
echo "========================================"
|
|
||||||
echo ""
|
|
||||||
echo "Binary: $BAKER_BIN"
|
|
||||||
echo "Examples: $EXAMPLES_DIR"
|
|
||||||
echo ""
|
|
||||||
echo "Run tests manually:"
|
|
||||||
echo " cd /app && pytest tests/ -v"
|
|
||||||
echo ""
|
|
||||||
exec /bin/bash
|
|
||||||
else
|
|
||||||
echo "========================================"
|
|
||||||
echo " HoneyBiscuitWorkshop System Tests"
|
|
||||||
echo " Mode: AUTO (running pytest)"
|
|
||||||
echo "========================================"
|
|
||||||
cd /app
|
|
||||||
exec python -m pytest tests/ -v --tb=short --junitxml=/results/report.xml "$@"
|
|
||||||
fi
|
|
||||||
Vendored
-6
@@ -1,6 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
[ -f /dev/shm/bakeexport.{{ name }} ] && . /dev/shm/bakeexport.{{ name }} || true
|
|
||||||
{{ condition }}
|
|
||||||
{{ main }}
|
|
||||||
{{ name }}
|
|
||||||
{{ export }}
|
|
||||||
Vendored
-25
@@ -1,25 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
|
|
||||||
# @pipeline
|
|
||||||
step_one() {
|
|
||||||
echo "Step 1: Setup"
|
|
||||||
mkdir -p /tmp/bake-test-output
|
|
||||||
echo "step1-done" > /tmp/bake-test-output/step1.txt
|
|
||||||
}
|
|
||||||
|
|
||||||
# @pipeline
|
|
||||||
step_two() {
|
|
||||||
echo "Step 2: Build"
|
|
||||||
echo "step2-done" > /tmp/bake-test-output/step2.txt
|
|
||||||
}
|
|
||||||
|
|
||||||
# @pipeline
|
|
||||||
step_three() {
|
|
||||||
echo "Step 3: Verify"
|
|
||||||
if [ -f /tmp/bake-test-output/step1.txt ] && [ -f /tmp/bake-test-output/step2.txt ]; then
|
|
||||||
echo "All steps completed successfully"
|
|
||||||
else
|
|
||||||
echo "ERROR: Missing step outputs"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
Vendored
-5
@@ -1,5 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
echo "Hello from simple bake script"
|
|
||||||
echo "Task ID: $HBW_TASKID"
|
|
||||||
echo "Pipeline: $HBW_PIPELINE"
|
|
||||||
exit 0
|
|
||||||
Vendored
-22
@@ -1,22 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
|
|
||||||
# @pipeline
|
|
||||||
init_workspace() {
|
|
||||||
echo "Initializing workspace"
|
|
||||||
mkdir -p /tmp/bake-after-test
|
|
||||||
echo "init" > /tmp/bake-after-test/status.txt
|
|
||||||
}
|
|
||||||
|
|
||||||
# @pipeline
|
|
||||||
# @after(init_workspace)
|
|
||||||
fetch_source() {
|
|
||||||
echo "Fetching source (depends on init_workspace)"
|
|
||||||
echo "fetched" > /tmp/bake-after-test/source.txt
|
|
||||||
}
|
|
||||||
|
|
||||||
# @pipeline
|
|
||||||
# @after(fetch_source)
|
|
||||||
build_project() {
|
|
||||||
echo "Building project (depends on fetch_source)"
|
|
||||||
echo "built" > /tmp/bake-after-test/build.txt
|
|
||||||
}
|
|
||||||
-4
@@ -1,4 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
echo "EARLY_HOOK_EXECUTED"
|
|
||||||
echo "Working directory: $(pwd)"
|
|
||||||
echo "User: $(whoami)"
|
|
||||||
Vendored
-3
@@ -1,3 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
echo "LATE_HOOK_EXECUTED"
|
|
||||||
echo "Working directory: $(pwd)"
|
|
||||||
Vendored
-6
@@ -1,6 +0,0 @@
|
|||||||
version: "0.0.1"
|
|
||||||
plugin: {}
|
|
||||||
notification: {}
|
|
||||||
artifact: []
|
|
||||||
cleanup:
|
|
||||||
policy: "auto"
|
|
||||||
-13
@@ -1,13 +0,0 @@
|
|||||||
version: "0.0.1"
|
|
||||||
plugin: {}
|
|
||||||
notification: {}
|
|
||||||
artifact: []
|
|
||||||
cleanup:
|
|
||||||
policy: "auto"
|
|
||||||
hooks:
|
|
||||||
early:
|
|
||||||
- name: "finalize-early"
|
|
||||||
command: "echo 'finalize early hook'"
|
|
||||||
late:
|
|
||||||
- name: "finalize-late"
|
|
||||||
command: "echo 'finalize late hook'"
|
|
||||||
Vendored
-6
@@ -1,6 +0,0 @@
|
|||||||
FROM docker.1ms.run/library/alpine:3.19
|
|
||||||
RUN sed -i 's|dl-cdn.alpinelinux.org|mirrors.tuna.tsinghua.edu.cn|g' /etc/apk/repositories
|
|
||||||
RUN apk add --no-cache python3 py3-pip bash
|
|
||||||
COPY workshop-baker/target/x86_64-unknown-linux-musl/debug/workshop-baker /baker
|
|
||||||
RUN chmod +x /baker
|
|
||||||
ENTRYPOINT ["/bin/sh", "-c"]
|
|
||||||
Vendored
-6
@@ -1,6 +0,0 @@
|
|||||||
FROM docker.1ms.run/library/debian:bookworm-slim
|
|
||||||
RUN sed -i 's|deb.debian.org|mirrors.tuna.tsinghua.edu.cn|g' /etc/apt/sources.list.d/debian.sources 2>/dev/null; sed -i 's|security.debian.org|mirrors.tuna.tsinghua.edu.cn/debian-security|g' /etc/apt/sources.list.d/debian.sources 2>/dev/null; true
|
|
||||||
RUN apt-get update -qq && apt-get install -y -qq --no-install-recommends python3 python3-venv ca-certificates 2>&1 | tail -n 2
|
|
||||||
COPY workshop-baker/target/x86_64-unknown-linux-musl/debug/workshop-baker /baker
|
|
||||||
RUN chmod +x /baker
|
|
||||||
ENTRYPOINT ["/bin/sh", "-c"]
|
|
||||||
Vendored
-6
@@ -1,6 +0,0 @@
|
|||||||
FROM docker.1ms.run/library/fedora:43
|
|
||||||
RUN sed -e 's|^metalink=|#metalink=|g' -e 's|^#baseurl=http://download.example/pub/fedora/linux|baseurl=https://mirrors.tuna.tsinghua.edu.cn/fedora|g' -i.bak /etc/yum.repos.d/fedora.repo /etc/yum.repos.d/fedora-updates.repo
|
|
||||||
RUN dnf install -y python3 2>&1 | tail -n 2
|
|
||||||
COPY workshop-baker/target/x86_64-unknown-linux-musl/debug/workshop-baker /baker
|
|
||||||
RUN chmod +x /baker
|
|
||||||
ENTRYPOINT ["/bin/sh", "-c"]
|
|
||||||
Vendored
-2
@@ -1,2 +0,0 @@
|
|||||||
FROM docker.1ms.run/library/archlinux:latest
|
|
||||||
RUN echo 'Server = https://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch' > /etc/pacman.d/mirrorlist
|
|
||||||
-8
@@ -1,8 +0,0 @@
|
|||||||
version: "1.0"
|
|
||||||
environment:
|
|
||||||
builder: "baremetal"
|
|
||||||
bootstrap:
|
|
||||||
user: "vulcan"
|
|
||||||
workspace:
|
|
||||||
path: "/home/vulcan/workspace"
|
|
||||||
fallback: true
|
|
||||||
Vendored
-15
@@ -1,15 +0,0 @@
|
|||||||
version: "1.0"
|
|
||||||
environment:
|
|
||||||
builder: "baremetal"
|
|
||||||
bootstrap:
|
|
||||||
user: "vulcan"
|
|
||||||
workspace:
|
|
||||||
path: "/home/vulcan/workspace"
|
|
||||||
fallback: true
|
|
||||||
hooks:
|
|
||||||
early:
|
|
||||||
- name: "test-early-hook"
|
|
||||||
command: "echo 'early hook executed'"
|
|
||||||
late:
|
|
||||||
- name: "test-late-hook"
|
|
||||||
command: "echo 'late hook executed'"
|
|
||||||
-3
@@ -1,3 +0,0 @@
|
|||||||
version: "0.5"
|
|
||||||
environment:
|
|
||||||
builder: "baremetal"
|
|
||||||
Vendored
-8
@@ -1,8 +0,0 @@
|
|||||||
version: "1.0"
|
|
||||||
environment:
|
|
||||||
builder: "baremetal"
|
|
||||||
bootstrap:
|
|
||||||
user: "vulcan"
|
|
||||||
workspace:
|
|
||||||
path: "/home/vulcan/workspace"
|
|
||||||
fallback: true
|
|
||||||
@@ -1,12 +0,0 @@
|
|||||||
version: "1.0"
|
|
||||||
environment:
|
|
||||||
builder: "baremetal"
|
|
||||||
bootstrap:
|
|
||||||
user: "vulcan"
|
|
||||||
workspace:
|
|
||||||
path: "/home/vulcan/workspace"
|
|
||||||
fallback: true
|
|
||||||
custom: |
|
|
||||||
#!/bin/sh
|
|
||||||
echo 'custom bootstrap executed'
|
|
||||||
mkdir -p /custom-workspace
|
|
||||||
-16
@@ -1,16 +0,0 @@
|
|||||||
version: "1.0"
|
|
||||||
environment:
|
|
||||||
builder: "baremetal"
|
|
||||||
bootstrap:
|
|
||||||
user: "vulcan"
|
|
||||||
workspace:
|
|
||||||
path: "/home/vulcan/workspace"
|
|
||||||
fallback: true
|
|
||||||
dependencies:
|
|
||||||
system:
|
|
||||||
pacman:
|
|
||||||
packages:
|
|
||||||
- "which"
|
|
||||||
user:
|
|
||||||
cargo:
|
|
||||||
- "ripgrep"
|
|
||||||
-13
@@ -1,13 +0,0 @@
|
|||||||
version: "1.0"
|
|
||||||
environment:
|
|
||||||
builder: "baremetal"
|
|
||||||
bootstrap:
|
|
||||||
user: "vulcan"
|
|
||||||
workspace:
|
|
||||||
path: "/home/vulcan/workspace"
|
|
||||||
fallback: true
|
|
||||||
envvars:
|
|
||||||
prebake:
|
|
||||||
TEST_PREBAKE_VAR: "prebake_value"
|
|
||||||
bake:
|
|
||||||
TEST_BAKE_VAR: "bake_value"
|
|
||||||
-17
@@ -1,17 +0,0 @@
|
|||||||
version: "1.0"
|
|
||||||
environment:
|
|
||||||
builder: "baremetal"
|
|
||||||
bootstrap:
|
|
||||||
user: "vulcan"
|
|
||||||
workspace:
|
|
||||||
path: "/home/vulcan/workspace"
|
|
||||||
fallback: true
|
|
||||||
hooks:
|
|
||||||
early:
|
|
||||||
- name: "test-early-hook"
|
|
||||||
command: "echo 'early hook executed'"
|
|
||||||
working_dir: "/workspace"
|
|
||||||
late:
|
|
||||||
- name: "test-late-hook"
|
|
||||||
command: "echo 'late hook executed'"
|
|
||||||
working_dir: "/workspace"
|
|
||||||
-14
@@ -1,14 +0,0 @@
|
|||||||
version: "1.0"
|
|
||||||
environment:
|
|
||||||
builder: "baremetal"
|
|
||||||
bootstrap:
|
|
||||||
user: "vulcan"
|
|
||||||
workspace:
|
|
||||||
path: "/home/vulcan/workspace"
|
|
||||||
fallback: true
|
|
||||||
dependencies:
|
|
||||||
system:
|
|
||||||
pacman:
|
|
||||||
packages:
|
|
||||||
- "curl"
|
|
||||||
- "wget"
|
|
||||||
-10
@@ -1,10 +0,0 @@
|
|||||||
version: "1.0"
|
|
||||||
environment:
|
|
||||||
builder: "baremetal"
|
|
||||||
bootstrap:
|
|
||||||
user: "vulcan"
|
|
||||||
workspace:
|
|
||||||
path: "/home/vulcan/workspace"
|
|
||||||
fallback: true
|
|
||||||
security:
|
|
||||||
drop_after: "Bootstrap"
|
|
||||||
@@ -1,12 +0,0 @@
|
|||||||
"""
|
|
||||||
LLM-based log judgment system for system tests.
|
|
||||||
|
|
||||||
Uses LLM APIs to evaluate non-deterministic test outputs (logs, stderr, etc.)
|
|
||||||
where simple string matching is insufficient.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from .client import LLMClient
|
|
||||||
from .cache import LLMCache
|
|
||||||
from .judge import judge_log, JudgmentResult
|
|
||||||
|
|
||||||
__all__ = ["LLMClient", "LLMCache", "judge_log", "JudgmentResult"]
|
|
||||||
@@ -1,56 +0,0 @@
|
|||||||
"""
|
|
||||||
Disk-based cache for LLM API responses.
|
|
||||||
|
|
||||||
Uses SHA256 of (prompt + model + system_prompt) as cache key.
|
|
||||||
Cache files stored as JSON in ``.cache/`` directory.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import hashlib
|
|
||||||
import json
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
|
|
||||||
class LLMCache:
|
|
||||||
"""SHA256-based disk cache for LLM API responses."""
|
|
||||||
|
|
||||||
def __init__(self, cache_dir: str | None = None):
|
|
||||||
if cache_dir is None:
|
|
||||||
import os
|
|
||||||
|
|
||||||
cache_dir = os.path.join(os.path.dirname(__file__), ".cache")
|
|
||||||
self.cache_dir = Path(cache_dir)
|
|
||||||
self.cache_dir.mkdir(parents=True, exist_ok=True)
|
|
||||||
|
|
||||||
def _make_key(self, prompt: str, model: str, system_prompt: str = "") -> str:
|
|
||||||
"""Create deterministic cache key from request parameters."""
|
|
||||||
data = json.dumps(
|
|
||||||
{"prompt": prompt, "model": model, "system": system_prompt},
|
|
||||||
sort_keys=True,
|
|
||||||
)
|
|
||||||
return hashlib.sha256(data.encode()).hexdigest()
|
|
||||||
|
|
||||||
def get(self, prompt: str, model: str, system_prompt: str = "") -> dict | None:
|
|
||||||
"""Retrieve cached response if exists."""
|
|
||||||
key = self._make_key(prompt, model, system_prompt)
|
|
||||||
path = self.cache_dir / f"{key}.json"
|
|
||||||
if path.exists():
|
|
||||||
try:
|
|
||||||
with open(path) as f:
|
|
||||||
return json.load(f)
|
|
||||||
except (json.JSONDecodeError, IOError):
|
|
||||||
return None
|
|
||||||
return None
|
|
||||||
|
|
||||||
def set(
|
|
||||||
self, prompt: str, model: str, system_prompt: str, response: dict
|
|
||||||
) -> None:
|
|
||||||
"""Store response in cache."""
|
|
||||||
key = self._make_key(prompt, model, system_prompt)
|
|
||||||
path = self.cache_dir / f"{key}.json"
|
|
||||||
with open(path, "w") as f:
|
|
||||||
json.dump(response, f, ensure_ascii=False)
|
|
||||||
|
|
||||||
def clear(self) -> None:
|
|
||||||
"""Clear all cached responses."""
|
|
||||||
for f in self.cache_dir.glob("*.json"):
|
|
||||||
f.unlink()
|
|
||||||
@@ -1,187 +0,0 @@
|
|||||||
"""
|
|
||||||
LLM client supporting both Anthropic and OpenAI APIs.
|
|
||||||
|
|
||||||
Prefers Anthropic format (per project convention).
|
|
||||||
Falls back to OpenAI format if Anthropic is unavailable.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import os
|
|
||||||
import logging
|
|
||||||
from .cache import LLMCache
|
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
|
|
||||||
class LLMClient:
|
|
||||||
"""LLM client with caching support. Prefers Anthropic API."""
|
|
||||||
|
|
||||||
def __init__(self, cache: LLMCache | None = None):
|
|
||||||
self.cache = cache or LLMCache()
|
|
||||||
self._anthropic_client = None
|
|
||||||
self._openai_client = None
|
|
||||||
self._init_clients()
|
|
||||||
|
|
||||||
def _init_clients(self):
|
|
||||||
"""Initialize API clients based on available environment variables."""
|
|
||||||
# Anthropic (preferred)
|
|
||||||
anthropic_key = os.environ.get("ANTHROPIC_API_KEY")
|
|
||||||
anthropic_base = os.environ.get("ANTHROPIC_BASE_URL")
|
|
||||||
if anthropic_key:
|
|
||||||
try:
|
|
||||||
import anthropic
|
|
||||||
|
|
||||||
kwargs: dict = {"api_key": anthropic_key}
|
|
||||||
if anthropic_base:
|
|
||||||
kwargs["base_url"] = anthropic_base
|
|
||||||
self._anthropic_client = anthropic.Anthropic(**kwargs)
|
|
||||||
logger.info("Anthropic client initialized")
|
|
||||||
except ImportError:
|
|
||||||
logger.warning("anthropic package not installed")
|
|
||||||
|
|
||||||
# OpenAI (fallback)
|
|
||||||
openai_key = os.environ.get("OPENAI_API_KEY")
|
|
||||||
openai_base = os.environ.get("OPENAI_BASE_URL")
|
|
||||||
if openai_key:
|
|
||||||
try:
|
|
||||||
import openai
|
|
||||||
|
|
||||||
kwargs = {"api_key": openai_key}
|
|
||||||
if openai_base:
|
|
||||||
kwargs["base_url"] = openai_base
|
|
||||||
self._openai_client = openai.OpenAI(**kwargs)
|
|
||||||
logger.info("OpenAI client initialized")
|
|
||||||
except ImportError:
|
|
||||||
logger.warning("openai package not installed")
|
|
||||||
|
|
||||||
@property
|
|
||||||
def available(self) -> bool:
|
|
||||||
"""Check if any LLM client is available."""
|
|
||||||
return (
|
|
||||||
self._anthropic_client is not None or self._openai_client is not None
|
|
||||||
)
|
|
||||||
|
|
||||||
def complete(
|
|
||||||
self, prompt: str, system_prompt: str = "", max_tokens: int = 2000
|
|
||||||
) -> str:
|
|
||||||
"""Send a completion request, with caching.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
prompt: User message.
|
|
||||||
system_prompt: System message (sets role / persona).
|
|
||||||
max_tokens: Max response tokens.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
LLM response text.
|
|
||||||
|
|
||||||
Raises:
|
|
||||||
RuntimeError: If no client is available.
|
|
||||||
"""
|
|
||||||
model = os.environ.get("MODEL", "deepseek-v4-flash")
|
|
||||||
|
|
||||||
# Check cache
|
|
||||||
cached = self.cache.get(prompt, model, system_prompt)
|
|
||||||
if cached:
|
|
||||||
logger.info("Cache hit for LLM request")
|
|
||||||
return cached["content"]
|
|
||||||
|
|
||||||
# Try Anthropic first
|
|
||||||
if self._anthropic_client:
|
|
||||||
content = self._call_anthropic(
|
|
||||||
prompt, system_prompt, model, max_tokens
|
|
||||||
)
|
|
||||||
elif self._openai_client:
|
|
||||||
content = self._call_openai(prompt, system_prompt, model, max_tokens)
|
|
||||||
else:
|
|
||||||
raise RuntimeError(
|
|
||||||
"No LLM client available. "
|
|
||||||
"Set ANTHROPIC_API_KEY or OPENAI_API_KEY."
|
|
||||||
)
|
|
||||||
|
|
||||||
# Cache the response
|
|
||||||
self.cache.set(prompt, model, system_prompt, {"content": content})
|
|
||||||
return content
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
# Private helpers
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
|
|
||||||
def _call_anthropic(
|
|
||||||
self, prompt: str, system_prompt: str, model: str, max_tokens: int
|
|
||||||
) -> str:
|
|
||||||
"""Call Anthropic API."""
|
|
||||||
kwargs: dict = {
|
|
||||||
"model": model,
|
|
||||||
"max_tokens": max_tokens,
|
|
||||||
"messages": [{"role": "user", "content": prompt}],
|
|
||||||
}
|
|
||||||
if system_prompt:
|
|
||||||
kwargs["system"] = system_prompt
|
|
||||||
|
|
||||||
# Only add reasoning_effort if the model supports it
|
|
||||||
reasoning_effort = os.environ.get("REASONING_EFFORT")
|
|
||||||
if reasoning_effort:
|
|
||||||
try:
|
|
||||||
kwargs["reasoning_effort"] = reasoning_effort
|
|
||||||
response = self._anthropic_client.messages.create(**kwargs)
|
|
||||||
except Exception as e:
|
|
||||||
if (
|
|
||||||
"reasoning_effort" in str(e).lower()
|
|
||||||
or "unexpected" in str(e).lower()
|
|
||||||
):
|
|
||||||
del kwargs["reasoning_effort"]
|
|
||||||
response = self._anthropic_client.messages.create(**kwargs)
|
|
||||||
else:
|
|
||||||
raise
|
|
||||||
else:
|
|
||||||
response = self._anthropic_client.messages.create(**kwargs)
|
|
||||||
|
|
||||||
# Extract text from response content blocks
|
|
||||||
if hasattr(response, "content") and response.content:
|
|
||||||
block = response.content[0]
|
|
||||||
if hasattr(block, "text"):
|
|
||||||
return block.text
|
|
||||||
if hasattr(block, "type") and block.type == "thinking":
|
|
||||||
for b in response.content:
|
|
||||||
if hasattr(b, "text"):
|
|
||||||
return b.text
|
|
||||||
return str(block)
|
|
||||||
return str(response)
|
|
||||||
|
|
||||||
def _call_openai(
|
|
||||||
self, prompt: str, system_prompt: str, model: str, max_tokens: int
|
|
||||||
) -> str:
|
|
||||||
"""Call OpenAI-compatible API."""
|
|
||||||
messages: list[dict] = []
|
|
||||||
if system_prompt:
|
|
||||||
messages.append({"role": "system", "content": system_prompt})
|
|
||||||
messages.append({"role": "user", "content": prompt})
|
|
||||||
|
|
||||||
kwargs: dict = {
|
|
||||||
"model": model,
|
|
||||||
"messages": messages,
|
|
||||||
"max_tokens": max_tokens,
|
|
||||||
}
|
|
||||||
|
|
||||||
# Only add reasoning_effort if supported
|
|
||||||
reasoning_effort = os.environ.get("REASONING_EFFORT")
|
|
||||||
if reasoning_effort:
|
|
||||||
try:
|
|
||||||
kwargs["reasoning_effort"] = reasoning_effort
|
|
||||||
response = self._openai_client.chat.completions.create(
|
|
||||||
**kwargs
|
|
||||||
)
|
|
||||||
except Exception as e:
|
|
||||||
if (
|
|
||||||
"reasoning_effort" in str(e).lower()
|
|
||||||
or "unexpected" in str(e).lower()
|
|
||||||
):
|
|
||||||
del kwargs["reasoning_effort"]
|
|
||||||
response = self._openai_client.chat.completions.create(
|
|
||||||
**kwargs
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
raise
|
|
||||||
else:
|
|
||||||
response = self._openai_client.chat.completions.create(**kwargs)
|
|
||||||
|
|
||||||
return response.choices[0].message.content
|
|
||||||
@@ -1,149 +0,0 @@
|
|||||||
"""
|
|
||||||
LLM-based log judgment for system tests.
|
|
||||||
|
|
||||||
Evaluates logs against expected behavior using LLM semantic analysis.
|
|
||||||
Returns structured pass/fail results with reasoning.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import json
|
|
||||||
import re
|
|
||||||
import logging
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from .client import LLMClient
|
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
JUDGE_SYSTEM_PROMPT = """\
|
|
||||||
You are a test result judge for a CI/CD system called HoneyBiscuitWorkshop.
|
|
||||||
|
|
||||||
Your job: Evaluate whether actual command output/logs match expected behavior.
|
|
||||||
|
|
||||||
RULES:
|
|
||||||
1. You MUST respond with valid JSON in this exact format:
|
|
||||||
{"verdict": "pass" | "fail", "reason": "brief explanation"}
|
|
||||||
2. "pass" = the output confirms the expected behavior occurred
|
|
||||||
3. "fail" = the output shows the expected behavior did NOT occur, or shows an error
|
|
||||||
4. Ignore irrelevant noise (timestamps, debug lines, unrelated warnings)
|
|
||||||
5. Focus ONLY on whether the specific expected behavior is confirmed or contradicted
|
|
||||||
6. Be strict: if the evidence is ambiguous, verdict is "fail"
|
|
||||||
7. Do NOT include any text outside the JSON object"""
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class JudgmentResult:
|
|
||||||
"""Result of an LLM log judgment."""
|
|
||||||
|
|
||||||
verdict: str # "pass" or "fail"
|
|
||||||
reason: str
|
|
||||||
raw_response: str
|
|
||||||
|
|
||||||
@property
|
|
||||||
def passed(self) -> bool:
|
|
||||||
return self.verdict == "pass"
|
|
||||||
|
|
||||||
|
|
||||||
def _extract_relevant_logs(full_log: str, max_chars: int = 4000) -> str:
|
|
||||||
"""Extract relevant portions from logs, removing noise.
|
|
||||||
|
|
||||||
Keeps the last *max_chars* characters (most recent / relevant).
|
|
||||||
Removes common noise patterns.
|
|
||||||
"""
|
|
||||||
lines = full_log.strip().split("\n")
|
|
||||||
|
|
||||||
noise_patterns = [
|
|
||||||
r"^\d{4}-\d{2}-\d{2}T",
|
|
||||||
r"^DEBUG ",
|
|
||||||
r"^TRACE ",
|
|
||||||
]
|
|
||||||
filtered = [
|
|
||||||
line
|
|
||||||
for line in lines
|
|
||||||
if not any(re.match(p, line) for p in noise_patterns)
|
|
||||||
]
|
|
||||||
|
|
||||||
text = "\n".join(filtered)
|
|
||||||
if len(text) > max_chars:
|
|
||||||
text = "...\n" + text[-max_chars:]
|
|
||||||
return text
|
|
||||||
|
|
||||||
|
|
||||||
def judge_log(
|
|
||||||
operation: str,
|
|
||||||
expected_behavior: str,
|
|
||||||
actual_log: str,
|
|
||||||
client: LLMClient | None = None,
|
|
||||||
) -> JudgmentResult:
|
|
||||||
"""Judge whether actual logs match expected behavior.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
operation: Description of what was attempted
|
|
||||||
(e.g. ``"prebake bootstrap stage"``).
|
|
||||||
expected_behavior: What should have happened
|
|
||||||
(e.g. ``"user vulcan was created"``).
|
|
||||||
actual_log: The actual stdout / stderr output.
|
|
||||||
client: ``LLMClient`` instance (creates default if ``None``).
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
``JudgmentResult`` with verdict and reason.
|
|
||||||
"""
|
|
||||||
if client is None:
|
|
||||||
client = LLMClient()
|
|
||||||
|
|
||||||
if not client.available:
|
|
||||||
logger.warning("No LLM client available, returning fail verdict")
|
|
||||||
return JudgmentResult(
|
|
||||||
verdict="fail",
|
|
||||||
reason="No LLM client available for judgment",
|
|
||||||
raw_response="",
|
|
||||||
)
|
|
||||||
|
|
||||||
relevant_log = _extract_relevant_logs(actual_log)
|
|
||||||
|
|
||||||
prompt = f"""\
|
|
||||||
## Operation
|
|
||||||
{operation}
|
|
||||||
|
|
||||||
## Expected Behavior
|
|
||||||
{expected_behavior}
|
|
||||||
|
|
||||||
## Actual Output (relevant excerpts)
|
|
||||||
```
|
|
||||||
{relevant_log}
|
|
||||||
```
|
|
||||||
|
|
||||||
Evaluate: Does the actual output confirm the expected behavior occurred?
|
|
||||||
Respond with JSON only."""
|
|
||||||
|
|
||||||
try:
|
|
||||||
response = client.complete(
|
|
||||||
prompt=prompt,
|
|
||||||
system_prompt=JUDGE_SYSTEM_PROMPT,
|
|
||||||
max_tokens=500,
|
|
||||||
)
|
|
||||||
|
|
||||||
# Parse JSON response — handle potential markdown code blocks
|
|
||||||
text = response.strip()
|
|
||||||
if text.startswith("```"):
|
|
||||||
lines = text.split("\n")
|
|
||||||
text = "\n".join(lines[1:-1]) if len(lines) > 2 else text
|
|
||||||
|
|
||||||
result = json.loads(text)
|
|
||||||
return JudgmentResult(
|
|
||||||
verdict=result.get("verdict", "fail"),
|
|
||||||
reason=result.get("reason", "No reason provided"),
|
|
||||||
raw_response=response,
|
|
||||||
)
|
|
||||||
except json.JSONDecodeError as e:
|
|
||||||
logger.error(f"Failed to parse LLM response as JSON: {e}")
|
|
||||||
return JudgmentResult(
|
|
||||||
verdict="fail",
|
|
||||||
reason=f"LLM response was not valid JSON: {response[:200]}",
|
|
||||||
raw_response=response,
|
|
||||||
)
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"LLM judgment failed: {e}")
|
|
||||||
return JudgmentResult(
|
|
||||||
verdict="fail",
|
|
||||||
reason=f"LLM call failed: {str(e)}",
|
|
||||||
raw_response="",
|
|
||||||
)
|
|
||||||
@@ -1,45 +0,0 @@
|
|||||||
"""
|
|
||||||
Environment probes for verifying system state in workshop-baker tests.
|
|
||||||
|
|
||||||
Each probe is a standalone function that can be called from any test.
|
|
||||||
Probes return bool or Optional[int/str]; they do not raise on failure.
|
|
||||||
|
|
||||||
A probe accepts an optional *runner* callable (default ``subprocess.run``)
|
|
||||||
that decouples it from the execution context (host or Docker).
|
|
||||||
"""
|
|
||||||
|
|
||||||
from .user import user_exists, user_uid, user_gid, user_in_group, user_shell
|
|
||||||
from .file import (
|
|
||||||
file_exists,
|
|
||||||
file_absent,
|
|
||||||
dir_exists,
|
|
||||||
dir_absent,
|
|
||||||
file_permissions,
|
|
||||||
file_contains,
|
|
||||||
file_owned_by,
|
|
||||||
file_group,
|
|
||||||
)
|
|
||||||
from .package import binary_exists, binary_on_path, package_installed, package_version
|
|
||||||
from .process import process_running, process_count
|
|
||||||
|
|
||||||
__all__ = [
|
|
||||||
"user_exists",
|
|
||||||
"user_uid",
|
|
||||||
"user_gid",
|
|
||||||
"user_in_group",
|
|
||||||
"user_shell",
|
|
||||||
"file_exists",
|
|
||||||
"file_absent",
|
|
||||||
"dir_exists",
|
|
||||||
"dir_absent",
|
|
||||||
"file_permissions",
|
|
||||||
"file_contains",
|
|
||||||
"file_owned_by",
|
|
||||||
"file_group",
|
|
||||||
"package_installed",
|
|
||||||
"package_version",
|
|
||||||
"binary_exists",
|
|
||||||
"binary_on_path",
|
|
||||||
"process_running",
|
|
||||||
"process_count",
|
|
||||||
]
|
|
||||||
@@ -1,53 +0,0 @@
|
|||||||
"""File and directory existence / permission probes."""
|
|
||||||
|
|
||||||
import os
|
|
||||||
import stat
|
|
||||||
import grp
|
|
||||||
import pwd
|
|
||||||
|
|
||||||
|
|
||||||
def file_exists(path: str) -> bool:
|
|
||||||
"""Check if *path* exists and is a regular file."""
|
|
||||||
return os.path.isfile(path)
|
|
||||||
|
|
||||||
|
|
||||||
def dir_exists(path: str) -> bool:
|
|
||||||
"""Check if *path* exists and is a directory."""
|
|
||||||
return os.path.isdir(path)
|
|
||||||
|
|
||||||
|
|
||||||
def file_permissions(path: str) -> int:
|
|
||||||
"""Return the permission mode of *path* as an octal int (e.g. ``0o755``).
|
|
||||||
|
|
||||||
Raises ``FileNotFoundError`` if *path* doesn't exist.
|
|
||||||
"""
|
|
||||||
return stat.S_IMODE(os.stat(path).st_mode)
|
|
||||||
|
|
||||||
|
|
||||||
def file_contains(path: str, text: str) -> bool:
|
|
||||||
"""Check if a file contains the specified text."""
|
|
||||||
with open(path, "r", errors="replace") as f:
|
|
||||||
return text in f.read()
|
|
||||||
|
|
||||||
|
|
||||||
def file_owned_by(path: str, username: str) -> bool:
|
|
||||||
"""Check if a file is owned by the specified user."""
|
|
||||||
st = os.stat(path)
|
|
||||||
owner = pwd.getpwuid(st.st_uid).pw_name
|
|
||||||
return owner == username
|
|
||||||
|
|
||||||
|
|
||||||
def file_group(path: str) -> str:
|
|
||||||
"""Get the group owner of a file."""
|
|
||||||
st = os.stat(path)
|
|
||||||
return grp.getgrgid(st.st_gid).gr_name
|
|
||||||
|
|
||||||
|
|
||||||
def file_absent(path: str) -> bool:
|
|
||||||
"""Check that *path* does not exist."""
|
|
||||||
return not os.path.lexists(path)
|
|
||||||
|
|
||||||
|
|
||||||
def dir_absent(path: str) -> bool:
|
|
||||||
"""Check that *path* does not exist or is not a directory."""
|
|
||||||
return not os.path.isdir(path)
|
|
||||||
@@ -1,42 +0,0 @@
|
|||||||
"""Package / binary existence probes (cross-distro).
|
|
||||||
|
|
||||||
Each probe accepts an optional *runner* callable with the same signature
|
|
||||||
as ``subprocess.run``. Defaults to running on the host; for Docker tests,
|
|
||||||
pass a runner that wraps ``docker run --rm <image>``.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import subprocess
|
|
||||||
from typing import Any, Callable
|
|
||||||
|
|
||||||
Runner = Callable[..., Any]
|
|
||||||
|
|
||||||
|
|
||||||
def _run(cmd: list[str], runner: Runner = subprocess.run, **kw: Any) -> Any:
|
|
||||||
opts = {"capture_output": True, "text": True, "timeout": 30}
|
|
||||||
opts.update(kw)
|
|
||||||
return runner(cmd, **opts)
|
|
||||||
|
|
||||||
|
|
||||||
def _succeeded(r: Any) -> bool:
|
|
||||||
""".returncode==0 for CompletedProcess, or ["success"] for dict."""
|
|
||||||
if hasattr(r, "returncode"):
|
|
||||||
return r.returncode == 0
|
|
||||||
return bool(r.get("success", False))
|
|
||||||
|
|
||||||
|
|
||||||
def binary_exists(path: str, runner: Runner = subprocess.run) -> bool:
|
|
||||||
"""Check if a binary exists at *path* (e.g. ``/usr/bin/curl``).
|
|
||||||
|
|
||||||
Uses ``test -f`` (POSIX, guaranteed available).
|
|
||||||
"""
|
|
||||||
return _succeeded(_run(["test", "-f", path], runner))
|
|
||||||
|
|
||||||
|
|
||||||
def binary_on_path(name: str, runner: Runner = subprocess.run) -> bool:
|
|
||||||
"""Check if a command is on PATH (via ``command -v``)."""
|
|
||||||
return _succeeded(_run(["sh", "-c", "command -v " + name], runner))
|
|
||||||
|
|
||||||
|
|
||||||
# Backward-compatible aliases
|
|
||||||
package_installed = binary_exists
|
|
||||||
package_version = binary_on_path # used in __init__ export, now binary-based
|
|
||||||
@@ -1,44 +0,0 @@
|
|||||||
"""Process existence probes."""
|
|
||||||
|
|
||||||
import os
|
|
||||||
import subprocess
|
|
||||||
|
|
||||||
|
|
||||||
def process_running(process_name: str) -> bool:
|
|
||||||
"""Check if a process with the given name is running."""
|
|
||||||
try:
|
|
||||||
result = subprocess.run(
|
|
||||||
["pgrep", "-x", process_name],
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
)
|
|
||||||
return result.returncode == 0
|
|
||||||
except FileNotFoundError:
|
|
||||||
# Fall back to /proc scan if pgrep not available
|
|
||||||
try:
|
|
||||||
for pid_dir in os.listdir("/proc"):
|
|
||||||
if pid_dir.isdigit():
|
|
||||||
try:
|
|
||||||
with open(f"/proc/{pid_dir}/comm") as f:
|
|
||||||
if f.read().strip() == process_name:
|
|
||||||
return True
|
|
||||||
except (IOError, OSError):
|
|
||||||
continue
|
|
||||||
except OSError:
|
|
||||||
pass
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
def process_count(process_name: str) -> int:
|
|
||||||
"""Count processes with the given name."""
|
|
||||||
try:
|
|
||||||
result = subprocess.run(
|
|
||||||
["pgrep", "-c", "-x", process_name],
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
)
|
|
||||||
if result.returncode == 0:
|
|
||||||
return int(result.stdout.strip())
|
|
||||||
return 0
|
|
||||||
except (FileNotFoundError, ValueError):
|
|
||||||
return 0
|
|
||||||
@@ -1,41 +0,0 @@
|
|||||||
"""User existence and attribute probes."""
|
|
||||||
|
|
||||||
import pwd
|
|
||||||
import grp
|
|
||||||
|
|
||||||
|
|
||||||
def user_exists(username: str) -> bool:
|
|
||||||
"""Check if a system user exists."""
|
|
||||||
try:
|
|
||||||
pwd.getpwnam(username)
|
|
||||||
return True
|
|
||||||
except KeyError:
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
def user_uid(username: str) -> int | None:
|
|
||||||
try:
|
|
||||||
return pwd.getpwnam(username).pw_uid
|
|
||||||
except KeyError:
|
|
||||||
return None
|
|
||||||
|
|
||||||
def user_gid(username: str) -> int | None:
|
|
||||||
try:
|
|
||||||
return pwd.getpwnam(username).pw_gid
|
|
||||||
except KeyError:
|
|
||||||
return None
|
|
||||||
|
|
||||||
def user_in_group(username: str, groupname: str) -> bool:
|
|
||||||
"""Check if a user is a member of a group."""
|
|
||||||
try:
|
|
||||||
group = grp.getgrnam(groupname)
|
|
||||||
return username in group.gr_mem
|
|
||||||
except KeyError:
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
def user_shell(username: str) -> str | None:
|
|
||||||
try:
|
|
||||||
return pwd.getpwnam(username).pw_shell
|
|
||||||
except KeyError:
|
|
||||||
return None
|
|
||||||
@@ -1,6 +0,0 @@
|
|||||||
pytest>=8.0.0
|
|
||||||
pytest-timeout>=2.2.0
|
|
||||||
pytest-xdist>=3.5.0
|
|
||||||
anthropic>=0.40.0,<0.90.0
|
|
||||||
openai>=1.50.0,<2.0.0
|
|
||||||
pyyaml>=6.0
|
|
||||||
Generated
+107
@@ -0,0 +1,107 @@
|
|||||||
|
# This file is automatically @generated by Cargo.
|
||||||
|
# It is not intended for manual editing.
|
||||||
|
version = 4
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "itoa"
|
||||||
|
version = "1.0.18"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "memchr"
|
||||||
|
version = "2.8.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "f8ca58f447f06ed17d5fc4043ce1b10dd205e060fb3ce5b979b8ed8e59ff3f79"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "proc-macro2"
|
||||||
|
version = "1.0.106"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934"
|
||||||
|
dependencies = [
|
||||||
|
"unicode-ident",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "quote"
|
||||||
|
version = "1.0.45"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924"
|
||||||
|
dependencies = [
|
||||||
|
"proc-macro2",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "serde"
|
||||||
|
version = "1.0.228"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e"
|
||||||
|
dependencies = [
|
||||||
|
"serde_core",
|
||||||
|
"serde_derive",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "serde_core"
|
||||||
|
version = "1.0.228"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad"
|
||||||
|
dependencies = [
|
||||||
|
"serde_derive",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "serde_derive"
|
||||||
|
version = "1.0.228"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79"
|
||||||
|
dependencies = [
|
||||||
|
"proc-macro2",
|
||||||
|
"quote",
|
||||||
|
"syn",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "serde_json"
|
||||||
|
version = "1.0.149"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "83fc039473c5595ace860d8c4fafa220ff474b3fc6bfdb4293327f1a37e94d86"
|
||||||
|
dependencies = [
|
||||||
|
"itoa",
|
||||||
|
"memchr",
|
||||||
|
"serde",
|
||||||
|
"serde_core",
|
||||||
|
"zmij",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "syn"
|
||||||
|
version = "2.0.117"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99"
|
||||||
|
dependencies = [
|
||||||
|
"proc-macro2",
|
||||||
|
"quote",
|
||||||
|
"unicode-ident",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "unicode-ident"
|
||||||
|
version = "1.0.24"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "workshop-baker-params"
|
||||||
|
version = "0.1.0"
|
||||||
|
dependencies = [
|
||||||
|
"serde",
|
||||||
|
"serde_json",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "zmij"
|
||||||
|
version = "1.0.21"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa"
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
[package]
|
||||||
|
name = "workshop-baker-params"
|
||||||
|
version = "0.1.0"
|
||||||
|
edition = "2024"
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
serde = { version = "1.0.228", features = ["derive"] }
|
||||||
|
serde_json = "1.0.149"
|
||||||
@@ -0,0 +1,54 @@
|
|||||||
|
use crate::apply_field;
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
#[serde(deny_unknown_fields)]
|
||||||
|
pub struct BakeParams {
|
||||||
|
#[serde(default)] pub workspace: ParamVal<String>,
|
||||||
|
/// Health probe interval in seconds (default 5).
|
||||||
|
#[serde(default)] pub health_period: ParamVal<u64>,
|
||||||
|
/// Health probe per-attempt timeout in seconds (default 10).
|
||||||
|
#[serde(default)] pub health_timeout: ParamVal<u64>,
|
||||||
|
/// Max health probe retries before giving up (default 12).
|
||||||
|
#[serde(default)] pub health_max_retries: ParamVal<u32>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl MergeParams for BakeParams {
|
||||||
|
fn defaults() -> Self {
|
||||||
|
Self {
|
||||||
|
workspace: ParamVal::new("/workspace".into()),
|
||||||
|
health_period: ParamVal::new(5),
|
||||||
|
health_timeout: ParamVal::new(10),
|
||||||
|
health_max_retries: ParamVal::new(12),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
|
||||||
|
apply_field!(self.workspace, i.workspace, w, ov, p, "workspace");
|
||||||
|
apply_field!(self.health_period, i.health_period, w, ov, p, "health_period");
|
||||||
|
apply_field!(self.health_timeout, i.health_timeout, w, ov, p, "health_timeout");
|
||||||
|
apply_field!(self.health_max_retries, i.health_max_retries, w, ov, p, "health_max_retries");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Default for BakeParams { fn default() -> Self { Self::defaults() } }
|
||||||
|
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct BakeSettings {
|
||||||
|
pub workspace: String,
|
||||||
|
pub health_period: u64,
|
||||||
|
pub health_timeout: u64,
|
||||||
|
pub health_max_retries: u32,
|
||||||
|
}
|
||||||
|
impl Default for BakeSettings { fn default() -> Self { BakeParams::defaults().into() } }
|
||||||
|
impl From<&BakeParams> for BakeSettings {
|
||||||
|
fn from(p: &BakeParams) -> Self {
|
||||||
|
Self {
|
||||||
|
workspace: p.workspace.value.clone(),
|
||||||
|
health_period: p.health_period.value,
|
||||||
|
health_timeout: p.health_timeout.value,
|
||||||
|
health_max_retries: p.health_max_retries.value,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
impl From<BakeParams> for BakeSettings { fn from(p: BakeParams) -> Self { (&p).into() } }
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
use crate::apply_field;
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
#[serde(deny_unknown_fields)]
|
||||||
|
pub struct FinalizeParams {
|
||||||
|
#[serde(default)] pub artifact_retention_days: ParamVal<u32>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl MergeParams for FinalizeParams {
|
||||||
|
fn defaults() -> Self { Self { artifact_retention_days: ParamVal::new(7) } }
|
||||||
|
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
|
||||||
|
apply_field!(self.artifact_retention_days, i.artifact_retention_days, w, ov, p, "artifact_retention_days");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
impl Default for FinalizeParams { fn default() -> Self { Self::defaults() } }
|
||||||
|
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct FinalizeSettings { pub artifact_retention_days: u32 }
|
||||||
|
impl Default for FinalizeSettings { fn default() -> Self { FinalizeParams::defaults().into() } }
|
||||||
|
impl From<&FinalizeParams> for FinalizeSettings { fn from(p: &FinalizeParams) -> Self { Self { artifact_retention_days: p.artifact_retention_days.value } } }
|
||||||
|
impl From<FinalizeParams> for FinalizeSettings { fn from(p: FinalizeParams) -> Self { (&p).into() } }
|
||||||
@@ -0,0 +1,168 @@
|
|||||||
|
pub mod notification;
|
||||||
|
pub mod prebake;
|
||||||
|
pub mod bake;
|
||||||
|
pub mod finalize;
|
||||||
|
pub mod writer;
|
||||||
|
pub mod param;
|
||||||
|
pub mod traits;
|
||||||
|
|
||||||
|
pub use notification::{NotificationParams, NotificationSettings};
|
||||||
|
pub use prebake::{PrebakeParams, PrebakeSettings};
|
||||||
|
pub use bake::{BakeParams, BakeSettings};
|
||||||
|
pub use finalize::{FinalizeParams, FinalizeSettings};
|
||||||
|
pub use writer::Writer;
|
||||||
|
pub use param::ParamVal;
|
||||||
|
pub use traits::MergeParams;
|
||||||
|
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
use std::collections::HashMap;
|
||||||
|
|
||||||
|
/// Root parameter tree. Each component gets its own subtree.
|
||||||
|
/// Used for both IPC (Serialized to JSON) and in-process merging.
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||||
|
pub struct Params {
|
||||||
|
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||||
|
pub notification: Option<NotificationParams>,
|
||||||
|
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||||
|
pub prebake: Option<PrebakeParams>,
|
||||||
|
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||||
|
pub bake: Option<BakeParams>,
|
||||||
|
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||||
|
pub finalize: Option<FinalizeParams>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Apply and merge helpers.
|
||||||
|
impl Params {
|
||||||
|
/// Merge a layer from a specific writer into the current params.
|
||||||
|
/// Returns entries that were overridden (for debug).
|
||||||
|
pub fn apply(&mut self, incoming: Params, writer: Writer) -> Overridden {
|
||||||
|
let mut ov = Overridden::new();
|
||||||
|
if let Some(l) = incoming.notification {
|
||||||
|
if let Some(ref mut b) = self.notification {
|
||||||
|
b.apply(l, writer, &mut ov, "notification.");
|
||||||
|
} else {
|
||||||
|
self.notification = Some(l);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if let Some(l) = incoming.prebake {
|
||||||
|
if let Some(ref mut b) = self.prebake {
|
||||||
|
b.apply(l, writer, &mut ov, "prebake.");
|
||||||
|
} else {
|
||||||
|
self.prebake = Some(l);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if let Some(l) = incoming.bake {
|
||||||
|
if let Some(ref mut b) = self.bake {
|
||||||
|
b.apply(l, writer, &mut ov, "bake.");
|
||||||
|
} else {
|
||||||
|
self.bake = Some(l);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if let Some(l) = incoming.finalize {
|
||||||
|
if let Some(ref mut b) = self.finalize {
|
||||||
|
b.apply(l, writer, &mut ov, "finalize.");
|
||||||
|
} else {
|
||||||
|
self.finalize = Some(l);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
ov
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Convert to baker-consumable settings. All params guaranteed to have values.
|
||||||
|
pub fn to_settings(&self) -> Result<AllSettings, String> {
|
||||||
|
Ok(AllSettings {
|
||||||
|
notification: self.notification.as_ref().map(NotificationSettings::from).unwrap_or_default(),
|
||||||
|
prebake: self.prebake.as_ref().map(PrebakeSettings::from).unwrap_or_default(),
|
||||||
|
bake: self.bake.as_ref().map(BakeSettings::from).unwrap_or_default(),
|
||||||
|
finalize: self.finalize.as_ref().map(FinalizeSettings::from).unwrap_or_default(),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Plain-value settings for baker consumption. No metadata, no Option.
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct AllSettings {
|
||||||
|
pub notification: NotificationSettings,
|
||||||
|
pub prebake: PrebakeSettings,
|
||||||
|
pub bake: BakeSettings,
|
||||||
|
pub finalize: FinalizeSettings,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Debug — values that were overridden during merge, keyed by dot-path.
|
||||||
|
#[derive(Debug, Clone, Default)]
|
||||||
|
pub struct Overridden {
|
||||||
|
pub entries: std::collections::HashMap<String, serde_json::Value>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Overridden {
|
||||||
|
pub fn new() -> Self { Self { entries: HashMap::new() } }
|
||||||
|
pub fn is_empty(&self) -> bool { self.entries.is_empty() }
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_empty_params_serializes_empty() {
|
||||||
|
let p = Params::default();
|
||||||
|
let json = serde_json::to_string(&p).unwrap();
|
||||||
|
assert_eq!(json, "{}");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_notification_params_serialize() {
|
||||||
|
let mut p = Params::default();
|
||||||
|
p.notification = Some(NotificationParams::defaults());
|
||||||
|
let json = serde_json::to_string_pretty(&p).unwrap();
|
||||||
|
assert!(json.contains("max_lives"));
|
||||||
|
assert!(json.contains("3"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_apply_courier_overrides_default() {
|
||||||
|
let mut base = Params::default();
|
||||||
|
base.notification = Some(NotificationParams::defaults());
|
||||||
|
let incoming = Params {
|
||||||
|
notification: Some(NotificationParams {
|
||||||
|
max_lives: ParamVal::with(5, Writer::Courier),
|
||||||
|
..Default::default()
|
||||||
|
}),
|
||||||
|
..Default::default()
|
||||||
|
};
|
||||||
|
let ov = base.apply(incoming, Writer::Courier);
|
||||||
|
let s = base.to_settings().unwrap();
|
||||||
|
assert_eq!(s.notification.max_lives, 5);
|
||||||
|
assert!(ov.is_empty()); // default had no writer, not an "override"
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_apply_upstream_wins() {
|
||||||
|
let mut base = Params::default();
|
||||||
|
base.notification = Some(NotificationParams {
|
||||||
|
max_lives: ParamVal::with(3, Writer::Base),
|
||||||
|
..Default::default()
|
||||||
|
});
|
||||||
|
let incoming = Params {
|
||||||
|
notification: Some(NotificationParams {
|
||||||
|
max_lives: ParamVal::with(5, Writer::Courier),
|
||||||
|
..Default::default()
|
||||||
|
}),
|
||||||
|
..Default::default()
|
||||||
|
};
|
||||||
|
let ov = base.apply(incoming, Writer::Courier);
|
||||||
|
let s = base.to_settings().unwrap();
|
||||||
|
// base(3) > courier(2) — rejected, base stays
|
||||||
|
assert_eq!(s.notification.max_lives, 3);
|
||||||
|
assert!(ov.is_empty()); // nothing overridden
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_deny_unknown_fields() {
|
||||||
|
let bad = r#"{"max_lives":{"value":5},"ghost_param":{"value":1}}"#;
|
||||||
|
let result: Result<NotificationParams, _> = serde_json::from_str(bad);
|
||||||
|
assert!(result.is_err()); // unknown field should be denied
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,65 @@
|
|||||||
|
use crate::apply_field;
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
#[serde(deny_unknown_fields)]
|
||||||
|
pub struct NotificationParams {
|
||||||
|
#[serde(default)] pub max_lives: ParamVal<u32>,
|
||||||
|
#[serde(default)] pub timeout_ms: ParamVal<u64>,
|
||||||
|
#[serde(default)] pub retry_backoff: ParamVal<f64>,
|
||||||
|
#[serde(default)] pub retry_delay_seconds: ParamVal<f64>,
|
||||||
|
#[serde(default)] pub max_retry_delay_seconds: ParamVal<f64>,
|
||||||
|
#[serde(default)] pub life_impact_factor: ParamVal<f64>,
|
||||||
|
#[serde(default)] pub batch_period: ParamVal<u32>,
|
||||||
|
#[serde(default)] pub batch_watermark: ParamVal<u32>,
|
||||||
|
#[serde(default)] pub template_ref_depth: ParamVal<u32>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl MergeParams for NotificationParams {
|
||||||
|
fn defaults() -> Self {
|
||||||
|
Self {
|
||||||
|
max_lives: ParamVal::new(3), timeout_ms: ParamVal::new(30000),
|
||||||
|
retry_backoff: ParamVal::new(2.0), retry_delay_seconds: ParamVal::new(5.0),
|
||||||
|
max_retry_delay_seconds: ParamVal::new(300.0),
|
||||||
|
life_impact_factor: ParamVal::new(0.0),
|
||||||
|
batch_period: ParamVal::new(5), batch_watermark: ParamVal::new(1),
|
||||||
|
template_ref_depth: ParamVal::new(10),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
fn apply(&mut self, incoming: Self, writer: Writer, ov: &mut Overridden, prefix: &str) {
|
||||||
|
apply_field!(self.max_lives, incoming.max_lives, writer, ov, prefix, "max_lives");
|
||||||
|
apply_field!(self.timeout_ms, incoming.timeout_ms, writer, ov, prefix, "timeout_ms");
|
||||||
|
apply_field!(self.retry_backoff, incoming.retry_backoff, writer, ov, prefix, "retry_backoff");
|
||||||
|
apply_field!(self.retry_delay_seconds, incoming.retry_delay_seconds, writer, ov, prefix, "retry_delay_seconds");
|
||||||
|
apply_field!(self.max_retry_delay_seconds, incoming.max_retry_delay_seconds, writer, ov, prefix, "max_retry_delay_seconds");
|
||||||
|
apply_field!(self.life_impact_factor, incoming.life_impact_factor, writer, ov, prefix, "life_impact_factor");
|
||||||
|
apply_field!(self.batch_period, incoming.batch_period, writer, ov, prefix, "batch_period");
|
||||||
|
apply_field!(self.batch_watermark, incoming.batch_watermark, writer, ov, prefix, "batch_watermark");
|
||||||
|
apply_field!(self.template_ref_depth, incoming.template_ref_depth, writer, ov, prefix, "template_ref_depth");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Default for NotificationParams { fn default() -> Self { Self::defaults() } }
|
||||||
|
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct NotificationSettings {
|
||||||
|
pub max_lives: u32, pub timeout_ms: u64, pub retry_backoff: f64,
|
||||||
|
pub retry_delay_seconds: f64, pub max_retry_delay_seconds: f64,
|
||||||
|
pub life_impact_factor: f64,
|
||||||
|
pub batch_period: u32, pub batch_watermark: u32, pub template_ref_depth: u32,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Default for NotificationSettings { fn default() -> Self { NotificationParams::defaults().into() } }
|
||||||
|
|
||||||
|
impl From<&NotificationParams> for NotificationSettings {
|
||||||
|
fn from(p: &NotificationParams) -> Self { Self {
|
||||||
|
max_lives: p.max_lives.value, timeout_ms: p.timeout_ms.value,
|
||||||
|
retry_backoff: p.retry_backoff.value, retry_delay_seconds: p.retry_delay_seconds.value,
|
||||||
|
max_retry_delay_seconds: p.max_retry_delay_seconds.value,
|
||||||
|
life_impact_factor: p.life_impact_factor.value,
|
||||||
|
batch_period: p.batch_period.value, batch_watermark: p.batch_watermark.value,
|
||||||
|
template_ref_depth: p.template_ref_depth.value,
|
||||||
|
}}
|
||||||
|
}
|
||||||
|
impl From<NotificationParams> for NotificationSettings { fn from(p: NotificationParams) -> Self { (&p).into() } }
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
use crate::Writer;
|
||||||
|
|
||||||
|
/// A strongly-typed parameter value with writer tracking.
|
||||||
|
///
|
||||||
|
/// Serializes as `{ "value": 3, "writer": "base" }`.
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct ParamVal<T> {
|
||||||
|
pub value: T,
|
||||||
|
#[serde(skip_serializing_if = "Option::is_none")]
|
||||||
|
pub writer: Option<Writer>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl<T: Default> Default for ParamVal<T> {
|
||||||
|
fn default() -> Self {
|
||||||
|
ParamVal { value: T::default(), writer: None }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl<T> ParamVal<T> {
|
||||||
|
pub fn new(value: T) -> Self {
|
||||||
|
ParamVal { value, writer: None }
|
||||||
|
}
|
||||||
|
pub fn with(value: T, writer: Writer) -> Self {
|
||||||
|
ParamVal { value, writer: Some(writer) }
|
||||||
|
}
|
||||||
|
/// Whether an incoming value from `incoming_writer` should override this one.
|
||||||
|
pub fn should_override(&self, incoming: Writer) -> bool {
|
||||||
|
let current = self.writer.map_or(0, |w| w.priority());
|
||||||
|
incoming.priority() > current
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,54 @@
|
|||||||
|
pub mod network;
|
||||||
|
pub mod cache;
|
||||||
|
pub mod engine;
|
||||||
|
|
||||||
|
use crate::apply_field;
|
||||||
|
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
#[serde(deny_unknown_fields)]
|
||||||
|
pub struct PrebakeParams {
|
||||||
|
#[serde(default)] pub bootstrap_user: ParamVal<String>,
|
||||||
|
#[serde(default)] pub workspace: ParamVal<String>,
|
||||||
|
#[serde(default)] pub network: network::NetworkParams,
|
||||||
|
#[serde(default)] pub cache: cache::CacheParams,
|
||||||
|
#[serde(default)] pub engine: engine::EngineParams,
|
||||||
|
#[serde(default)] pub repology_endpoint: ParamVal<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl MergeParams for PrebakeParams {
|
||||||
|
fn defaults() -> Self {
|
||||||
|
Self {
|
||||||
|
bootstrap_user: ParamVal::new("vulcan".into()), workspace: ParamVal::new("/home/vulcan/workspace".into()),
|
||||||
|
network: network::NetworkParams::defaults(), cache: cache::CacheParams::defaults(),
|
||||||
|
engine: engine::EngineParams::defaults(),
|
||||||
|
repology_endpoint: ParamVal::new("default".into()),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
|
||||||
|
apply_field!(self.bootstrap_user, i.bootstrap_user, w, ov, p, "bootstrap_user");
|
||||||
|
apply_field!(self.workspace, i.workspace, w, ov, p, "workspace");
|
||||||
|
apply_field!(self.repology_endpoint, i.repology_endpoint, w, ov, p, "repology_endpoint");
|
||||||
|
self.network.apply(i.network, w, ov, &format!("{}network.", p));
|
||||||
|
self.cache.apply(i.cache, w, ov, &format!("{}cache.", p));
|
||||||
|
self.engine.apply(i.engine, w, ov, &format!("{}engine.", p));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
impl Default for PrebakeParams { fn default() -> Self { Self::defaults() } }
|
||||||
|
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct PrebakeSettings {
|
||||||
|
pub bootstrap_user: String, pub workspace: String,
|
||||||
|
pub network: network::NetworkSettings, pub cache: cache::CacheSettings,
|
||||||
|
pub engine: engine::EngineSettings, pub repology_endpoint: String,
|
||||||
|
}
|
||||||
|
impl Default for PrebakeSettings { fn default() -> Self { PrebakeParams::defaults().into() } }
|
||||||
|
impl From<&PrebakeParams> for PrebakeSettings {
|
||||||
|
fn from(p: &PrebakeParams) -> Self { Self {
|
||||||
|
bootstrap_user: p.bootstrap_user.value.clone(), workspace: p.workspace.value.clone(),
|
||||||
|
network: (&p.network).into(), cache: (&p.cache).into(), engine: (&p.engine).into(),
|
||||||
|
repology_endpoint: p.repology_endpoint.value.clone(),
|
||||||
|
}}
|
||||||
|
}
|
||||||
|
impl From<PrebakeParams> for PrebakeSettings { fn from(p: PrebakeParams) -> Self { (&p).into() } }
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
use crate::apply_field;
|
||||||
|
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct CacheParams {
|
||||||
|
#[serde(default)] pub strategy: ParamVal<String>,
|
||||||
|
#[serde(default)] pub mode: ParamVal<String>,
|
||||||
|
#[serde(default)] pub ttl_days: ParamVal<u32>,
|
||||||
|
#[serde(default)] pub max_size_gb: ParamVal<u32>,
|
||||||
|
#[serde(default)] pub cleanup_policy: ParamVal<String>,
|
||||||
|
}
|
||||||
|
impl MergeParams for CacheParams {
|
||||||
|
fn defaults() -> Self {
|
||||||
|
Self {
|
||||||
|
strategy: ParamVal::new("always".into()), mode: ParamVal::new("zstd".into()),
|
||||||
|
ttl_days: ParamVal::new(30), max_size_gb: ParamVal::new(20),
|
||||||
|
cleanup_policy: ParamVal::new("lru".into()),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
|
||||||
|
apply_field!(self.strategy, i.strategy, w, ov, p, "strategy");
|
||||||
|
apply_field!(self.mode, i.mode, w, ov, p, "mode");
|
||||||
|
apply_field!(self.ttl_days, i.ttl_days, w, ov, p, "ttl_days");
|
||||||
|
apply_field!(self.max_size_gb, i.max_size_gb, w, ov, p, "max_size_gb");
|
||||||
|
apply_field!(self.cleanup_policy, i.cleanup_policy, w, ov, p, "cleanup_policy");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
impl Default for CacheParams { fn default() -> Self { Self::defaults() } }
|
||||||
|
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct CacheSettings { pub strategy: String, pub mode: String, pub ttl_days: u32, pub max_size_gb: u32, pub cleanup_policy: String }
|
||||||
|
impl Default for CacheSettings { fn default() -> Self { CacheParams::defaults().into() } }
|
||||||
|
impl From<&CacheParams> for CacheSettings { fn from(p: &CacheParams) -> Self {
|
||||||
|
Self { strategy: p.strategy.value.clone(), mode: p.mode.value.clone(), ttl_days: p.ttl_days.value, max_size_gb: p.max_size_gb.value, cleanup_policy: p.cleanup_policy.value.clone() }
|
||||||
|
} }
|
||||||
|
impl From<CacheParams> for CacheSettings { fn from(p: CacheParams) -> Self { (&p).into() } }
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
use crate::apply_field;
|
||||||
|
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct EngineParams {
|
||||||
|
#[serde(default)] pub timeout_ms: ParamVal<u64>,
|
||||||
|
}
|
||||||
|
impl MergeParams for EngineParams {
|
||||||
|
fn defaults() -> Self { Self { timeout_ms: ParamVal::new(300000) } }
|
||||||
|
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
|
||||||
|
apply_field!(self.timeout_ms, i.timeout_ms, w, ov, p, "timeout_ms");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
impl Default for EngineParams { fn default() -> Self { Self::defaults() } }
|
||||||
|
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct EngineSettings { pub timeout_ms: u64 }
|
||||||
|
impl Default for EngineSettings { fn default() -> Self { EngineParams::defaults().into() } }
|
||||||
|
impl From<&EngineParams> for EngineSettings { fn from(p: &EngineParams) -> Self { Self { timeout_ms: p.timeout_ms.value } } }
|
||||||
|
impl From<EngineParams> for EngineSettings { fn from(p: EngineParams) -> Self { (&p).into() } }
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
use crate::apply_field;
|
||||||
|
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct NetworkParams {
|
||||||
|
#[serde(default)] pub enabled: ParamVal<bool>,
|
||||||
|
#[serde(default)] pub outbound: ParamVal<bool>,
|
||||||
|
}
|
||||||
|
impl MergeParams for NetworkParams {
|
||||||
|
fn defaults() -> Self { Self { enabled: ParamVal::new(true), outbound: ParamVal::new(true) } }
|
||||||
|
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
|
||||||
|
apply_field!(self.enabled, i.enabled, w, ov, p, "enabled");
|
||||||
|
apply_field!(self.outbound, i.outbound, w, ov, p, "outbound");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
impl Default for NetworkParams { fn default() -> Self { Self::defaults() } }
|
||||||
|
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct NetworkSettings { pub enabled: bool, pub outbound: bool }
|
||||||
|
impl Default for NetworkSettings { fn default() -> Self { NetworkParams::defaults().into() } }
|
||||||
|
impl From<&NetworkParams> for NetworkSettings { fn from(p: &NetworkParams) -> Self { Self { enabled: p.enabled.value, outbound: p.outbound.value } } }
|
||||||
|
impl From<NetworkParams> for NetworkSettings { fn from(p: NetworkParams) -> Self { (&p).into() } }
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
use crate::{Writer, Overridden};
|
||||||
|
|
||||||
|
/// A parameter tree node that can merge itself with an incoming layer.
|
||||||
|
pub trait MergeParams: Sized {
|
||||||
|
fn apply(&mut self, incoming: Self, writer: Writer, ov: &mut Overridden, prefix: &str);
|
||||||
|
fn defaults() -> Self;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Apply one field's merge logic.
|
||||||
|
/// ```ignore
|
||||||
|
/// apply_field!(self.foo, incoming.foo, writer, ov, prefix, "foo");
|
||||||
|
/// ```
|
||||||
|
#[macro_export]
|
||||||
|
macro_rules! apply_field {
|
||||||
|
($self:ident.$field:ident, $incoming:ident.$field2:ident, $writer:expr, $ov:ident, $prefix:expr, $name:expr) => {
|
||||||
|
let old_w = $self.$field.writer;
|
||||||
|
if $self.$field.should_override($writer) {
|
||||||
|
let old_v = ::std::mem::replace(&mut $self.$field.value, $incoming.$field2.value);
|
||||||
|
$self.$field.writer = ::std::option::Option::Some($writer);
|
||||||
|
if let ::std::option::Option::Some(w) = old_w {
|
||||||
|
$ov.entries.insert(
|
||||||
|
format!("{}{}", $prefix, $name),
|
||||||
|
::serde_json::json!({"value": old_v, "writer": w.as_str()}),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
}
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
||||||
|
#[serde(rename_all = "lowercase")]
|
||||||
|
pub enum Writer {
|
||||||
|
Base,
|
||||||
|
Courier,
|
||||||
|
Bakerd,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Writer {
|
||||||
|
/// Priority: Base=3 > Courier=2 > Bakerd=1
|
||||||
|
pub fn priority(self) -> u8 {
|
||||||
|
match self {
|
||||||
|
Writer::Base => 3,
|
||||||
|
Writer::Courier => 2,
|
||||||
|
Writer::Bakerd => 1,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
pub fn as_str(self) -> &'static str {
|
||||||
|
match self {
|
||||||
|
Writer::Base => "base", Writer::Courier => "courier", Writer::Bakerd => "bakerd",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
impl Default for Writer { fn default() -> Self { Writer::Base } }
|
||||||
@@ -1,3 +1,4 @@
|
|||||||
temp
|
temp
|
||||||
examples
|
examples
|
||||||
quicktest.sh
|
quicktest.sh
|
||||||
|
_env.sh
|
||||||
|
|||||||
@@ -0,0 +1,80 @@
|
|||||||
|
# WORKSHOP-BAKER CRATE KNOWLEDGE BASE
|
||||||
|
|
||||||
|
## OVERVIEW
|
||||||
|
Primary crate — the only crate with a `[[bin]]` target. Contains the `workshop-baker` binary and `workshop_baker` library. Orchestrates the full CI/CD pipeline: prebake (env setup) → bake (script execution) → finalize (plugins, cleanup) → drain notification queue. ~69 .rs source files.
|
||||||
|
|
||||||
|
## STRUCTURE
|
||||||
|
```
|
||||||
|
src/
|
||||||
|
├── main.rs # Entry: default(full pipeline) | bare(dev-only) | daemon(unimplemented)
|
||||||
|
├── lib.rs # Clap Cli/Commands/BareCommands — CLI structs in lib, not bin
|
||||||
|
├── error.rs # CliError enum (wraps stage errors, delegates exit_code)
|
||||||
|
├── bare.rs # Dev-only single-stage execution, ctx.privileged always false
|
||||||
|
├── utils.rs # Cross-cutting helpers
|
||||||
|
├── bake/ # Script parsing + DAG scheduling + execution
|
||||||
|
│ ├── decorator.rs # @decorator(args) parsing, 81 tests — heaviest coverage in crate
|
||||||
|
│ ├── parser.rs # Step/function extraction
|
||||||
|
│ ├── schedule.rs # DAG build → topological sort
|
||||||
|
│ ├── execute.rs # Per-step timeout/retry
|
||||||
|
│ ├── builder.rs # Script assembly
|
||||||
|
│ ├── trivial.rs # Trivial step detection (skip in dry-run)
|
||||||
|
│ ├── constant.rs / error.rs / util.rs
|
||||||
|
├── prebake/ # Environment provisioning
|
||||||
|
│ ├── config.rs # PrebakeConfig deserialization
|
||||||
|
│ ├── stage.rs + stage/{bootstrap,depssystem,depsuser,environment,hook}.rs
|
||||||
|
│ ├── env.rs + env/{container,firecracker,baremetal,custom}.rs # Builder dispatch
|
||||||
|
│ ├── security.rs # Privilege/sandbox checks
|
||||||
|
│ ├── event.rs # Event sender/receiver channel
|
||||||
|
│ ├── types.rs + types/{memsize,cache,builderconfig,architecture}.rs
|
||||||
|
│ ├── constant.rs / error.rs
|
||||||
|
├── finalize/ # Post-build hooks + plugins
|
||||||
|
│ ├── config.rs # FinalizeConfig
|
||||||
|
│ ├── stage.rs + stage/hook.rs
|
||||||
|
│ ├── template.rs # Output templating (minijinja)
|
||||||
|
│ ├── plugin.rs + plugin/{shell,dylib,rhai,internal}.rs
|
||||||
|
│ │ └── plugin/internal/{dummy,mail,satori,webhook,insitenotify}.rs
|
||||||
|
│ ├── types.rs + types/compression.rs
|
||||||
|
│ ├── constant.rs / error.rs / event.rs
|
||||||
|
├── notify/ # Async notification dispatch
|
||||||
|
│ ├── queue.rs # NotificationQueue — drain on pipeline complete
|
||||||
|
│ ├── handler.rs / rule.rs / template.rs / types.rs / util.rs / error.rs
|
||||||
|
└── types/ # Shared types — ZERO deps on stage modules
|
||||||
|
├── resource.rs # ResourceRegistry (populated pre-pipeline, consumed read-only)
|
||||||
|
└── buildstatus.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
## WHERE TO LOOK
|
||||||
|
|
||||||
|
| Task | Location | Notes |
|
||||||
|
|------|----------|-------|
|
||||||
|
| CLI args + subcommands | `src/lib.rs` | Cli, Commands(Bare/Daemon), BareCommands |
|
||||||
|
| Pipeline orchestration | `src/main.rs` | worker_main(): prebake→bake→finalize→queue.drain() |
|
||||||
|
| Decorator syntax | `src/bake/decorator.rs` | `# @decorator` or `# @decorator(args)`, line-by-line regex |
|
||||||
|
| Build execution | `src/bake/execute.rs` | timeout, retry, exit code capture |
|
||||||
|
| DAG scheduling | `src/bake/schedule.rs` | Uses workshop-schedule crate |
|
||||||
|
| Docker builder | `src/prebake/env/container.rs` | Bollard-based, buildkit support |
|
||||||
|
| Firecracker / Bare metal | `.../firecracker.rs`, `.../baremetal.rs` | MicroVM;
|
||||||
|
needs PIPELINE_BAREMETAL_ELEVATE |
|
||||||
|
| Plugin dispatch | `src/finalize/plugin.rs` | Dispatches shell / dylib(stub) / rhai(stub) / internal |
|
||||||
|
| Notification queue | `src/notify/queue.rs` | Spawned early, drained after pipeline, retry support |
|
||||||
|
| Error type | `src/error.rs` | CliError wraps PrebakeError/BakeError/FinalizeError/anyhow |
|
||||||
|
| Dev bare mode | `src/bare.rs` | ctx.privileged=false always; do not extrapolate to production |
|
||||||
|
|
||||||
|
## DEPENDENCIES
|
||||||
|
**Internal:** workshop-engine, workshop-schedule, workshop-getterurl, workshop-baker-params. **External:** bollard (Docker), clap (CLI), axum 0.7 (HTTP), lettre 0.11 (email), minijinja (templates), cgroups-rs (limits), privdrop.
|
||||||
|
|
||||||
|
## CONVENTIONS (CRATE-SPECIFIC)
|
||||||
|
- **Lint overrides** (this crate only): `dead_code=allow`, `unreachable_code=allow`, `inherent_to_string=allow`, `non_canonical_partial_ord_impl=allow`
|
||||||
|
- **CLI structs in lib.rs, not main.rs** — enables integration tests to import and construct Cli directly.
|
||||||
|
- **Pipeline stages return `anyhow::Result<()>`** — stages propagate with `?`; only main.rs converts to CliError.
|
||||||
|
- **ResourceRegistry pattern** — populated before pipeline stages, consumed read-only by stages. New resource types go in `types/resource.rs`.
|
||||||
|
- **Notification queue fire-and-forget + drain** — notify tasks spawned early, send during pipeline, drain on completion. `NotificationQueue::drain()` must be called.
|
||||||
|
- **Trivial step detection** — steps with no external deps can be marked trivial (`trivial.rs`), skipped in dry-run.
|
||||||
|
## ANTI-PATTERNS (THIS CRATE)
|
||||||
|
- **Do NOT derive production behavior from bare mode** — ctx.privileged is hardcoded false.
|
||||||
|
- **Do NOT add CLI args bypassing prebake→bake→finalize ordering** — the pipeline is linear. Parallelism lives in the DAG scheduler within bake.
|
||||||
|
- **Do NOT import types/ from bake/, prebake/, or finalize/** — `types/` must have zero crate-internal deps.
|
||||||
|
- **Do NOT add lint allows without justification** — the four existing overrides are intentional.
|
||||||
|
- **Do NOT spawn blocking work on tokio runtime** — use `spawn_blocking` for CPU-heavy tasks.
|
||||||
|
- **Decorator regex is line-by-line** — multi-line decorators need a parser rewrite, not a regex hack.
|
||||||
|
- **Careful with `unimplemented!()`** — Daemon, Rhai/Dylib plugins, Custom builder are stubs. Prefer error over crash.
|
||||||
Generated
+376
-1201
File diff suppressed because it is too large
Load Diff
@@ -41,20 +41,23 @@ tar = "0.4.44"
|
|||||||
tempfile = "3.24.0"
|
tempfile = "3.24.0"
|
||||||
thiserror = "2.0.17"
|
thiserror = "2.0.17"
|
||||||
tokio = { version = "1.48.0", features = ["full"] }
|
tokio = { version = "1.48.0", features = ["full"] }
|
||||||
topological-sort = "0.2.2"
|
workshop-schedule = { path = "../workshop-schedule" }
|
||||||
async-trait = "0.1.87"
|
async-trait = "0.1.87"
|
||||||
uuid = { version = "1.19.0", features = ["v4"] }
|
uuid = { version = "1.19.0", features = ["v4"] }
|
||||||
libc = "0.2"
|
libc = "0.2"
|
||||||
os_info = "3.14.0"
|
os_info = "3.14.0"
|
||||||
privdrop = "0.5.6"
|
privdrop = "0.5.6"
|
||||||
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
|
|
||||||
which = "8.0.2"
|
which = "8.0.2"
|
||||||
gix = "0.81.0"
|
|
||||||
sha2 = "0.10"
|
|
||||||
flate2 = "1.0"
|
|
||||||
zstd = "0.13"
|
|
||||||
globset = "0.4.18"
|
globset = "0.4.18"
|
||||||
|
axum = "0.7"
|
||||||
|
workshop-getterurl = { path = "../workshop-getterurl" }
|
||||||
|
workshop-baker-params = { version = "0.1.0", path = "../workshop-baker-params" }
|
||||||
|
|
||||||
|
bitflags = "2"
|
||||||
|
url = "2.5.8"
|
||||||
|
zstd = "0.13.3"
|
||||||
|
flate2 = "1.1.9"
|
||||||
|
capctl = "0.2.4"
|
||||||
[dev-dependencies]
|
[dev-dependencies]
|
||||||
criterion = "0.5"
|
criterion = "0.5"
|
||||||
|
|
||||||
@@ -66,6 +69,3 @@ unreachable_code = "allow"
|
|||||||
inherent_to_string = "allow"
|
inherent_to_string = "allow"
|
||||||
non_canonical_partial_ord_impl = "allow"
|
non_canonical_partial_ord_impl = "allow"
|
||||||
|
|
||||||
[[bench]]
|
|
||||||
name = "parser_bench"
|
|
||||||
harness = false
|
|
||||||
|
|||||||
@@ -1,93 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
import os
|
|
||||||
import random
|
|
||||||
import string
|
|
||||||
|
|
||||||
OUTPUT_DIR = "/tmp"
|
|
||||||
DECORATORS = [
|
|
||||||
"# @pipeline",
|
|
||||||
"# @fallible",
|
|
||||||
"# @parallel",
|
|
||||||
"# @export",
|
|
||||||
"# @timeout(60)",
|
|
||||||
"# @retry(3, 5)",
|
|
||||||
"# @if(condition)",
|
|
||||||
"# @after(dep)",
|
|
||||||
"# @pipe(step1, step2)",
|
|
||||||
"# @health(/health)",
|
|
||||||
"# @loop(10)",
|
|
||||||
]
|
|
||||||
FUNCTION_BODIES = [
|
|
||||||
"echo 'Processing...';",
|
|
||||||
"local var=$(date +%s);",
|
|
||||||
"if [ -f /tmp/test ]; then echo 'exists'; fi;",
|
|
||||||
"for i in $(seq 1 10); do echo $i; done;",
|
|
||||||
"case $1 in start) echo 'starting';; stop) echo 'stopping';; esac;",
|
|
||||||
"read -r line < /dev/stdin;",
|
|
||||||
"export PATH=$PATH:/usr/local/bin;",
|
|
||||||
"set -e; set -u;",
|
|
||||||
"trap 'echo error' ERR;",
|
|
||||||
"cd /tmp || exit 1;",
|
|
||||||
"local result=$(grep -r 'pattern' /var/log/ 2>/dev/null);",
|
|
||||||
'for f in /tmp/*.tmp; do rm -f "$f"; done',
|
|
||||||
"if [[ $var -gt 100 ]]; then echo 'large'; fi;",
|
|
||||||
'while read -r line; do echo "$line"; done < /etc/passwd;',
|
|
||||||
'select opt in a b c; do echo "Selected $opt"; break; done;',
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
def random_string(length=8):
|
|
||||||
return "".join(random.choices(string.ascii_letters + string.digits, k=length))
|
|
||||||
|
|
||||||
|
|
||||||
def generate_function(body_lines):
|
|
||||||
lines = []
|
|
||||||
func_name = f"func_{random_string(6)}"
|
|
||||||
for _ in range(random.randint(0, 3)):
|
|
||||||
lines.append(random.choice(DECORATORS))
|
|
||||||
lines.append(f"{func_name}() {{")
|
|
||||||
for _ in range(body_lines):
|
|
||||||
lines.append(" " + random.choice(FUNCTION_BODIES))
|
|
||||||
lines.append("}")
|
|
||||||
lines.append("")
|
|
||||||
return lines
|
|
||||||
|
|
||||||
|
|
||||||
def generate_script(target_lines, num_functions, target_bytes):
|
|
||||||
lines = []
|
|
||||||
total_lines = 0
|
|
||||||
body_per_func = max(1, (target_lines - num_functions * 3) // num_functions)
|
|
||||||
|
|
||||||
for _ in range(num_functions):
|
|
||||||
func_lines = generate_function(body_per_func)
|
|
||||||
lines.extend(func_lines)
|
|
||||||
total_lines += len(func_lines)
|
|
||||||
|
|
||||||
while (
|
|
||||||
total_lines < target_lines
|
|
||||||
or len("\n".join(lines).encode("utf-8")) < target_bytes
|
|
||||||
):
|
|
||||||
lines.append(random.choice(FUNCTION_BODIES))
|
|
||||||
total_lines += 1
|
|
||||||
|
|
||||||
return "\n".join(lines)
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
sizes = [
|
|
||||||
("small", 1000, 10, 50_000),
|
|
||||||
("medium", 5000, 50, 500_000),
|
|
||||||
("large", 10000, 100, 950_000),
|
|
||||||
]
|
|
||||||
for name, lines, funcs, target_bytes in sizes:
|
|
||||||
script = generate_script(lines, funcs, target_bytes)
|
|
||||||
filepath = os.path.join(OUTPUT_DIR, f"bench_{name}.sh")
|
|
||||||
with open(filepath, "w") as f:
|
|
||||||
f.write(script)
|
|
||||||
actual_bytes = len(script.encode("utf-8"))
|
|
||||||
actual_lines = len(script.splitlines())
|
|
||||||
print(f"Generated {filepath}: lines={actual_lines}, bytes={actual_bytes}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,61 +0,0 @@
|
|||||||
use criterion::{Criterion, black_box, criterion_group, criterion_main};
|
|
||||||
use std::fs;
|
|
||||||
use std::path::Path;
|
|
||||||
|
|
||||||
fn parse_large_script(c: &mut Criterion) {
|
|
||||||
let script_path = Path::new("/tmp/bench_large.sh");
|
|
||||||
|
|
||||||
let script_content = fs::read_to_string(script_path)
|
|
||||||
.expect("Failed to read benchmark script. Run `python3 benches/generate.py` first.");
|
|
||||||
|
|
||||||
let byte_size = script_content.len();
|
|
||||||
let line_count = script_content.lines().count();
|
|
||||||
|
|
||||||
println!("Benchmark file: {} lines, {} bytes", line_count, byte_size);
|
|
||||||
|
|
||||||
c.bench_function("parse_script_large", |b| {
|
|
||||||
b.iter(|| {
|
|
||||||
let result =
|
|
||||||
workshop_baker::bake::parser::parse_script(black_box(&script_content.clone()));
|
|
||||||
black_box(result)
|
|
||||||
});
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
fn parse_medium_script(c: &mut Criterion) {
|
|
||||||
let script_path = Path::new("/tmp/bench_medium.sh");
|
|
||||||
|
|
||||||
let script_content = fs::read_to_string(script_path)
|
|
||||||
.expect("Failed to read benchmark script. Run `python3 benches/generate.py` first.");
|
|
||||||
|
|
||||||
c.bench_function("parse_script_medium", |b| {
|
|
||||||
b.iter(|| {
|
|
||||||
let result =
|
|
||||||
workshop_baker::bake::parser::parse_script(black_box(&script_content.clone()));
|
|
||||||
black_box(result)
|
|
||||||
});
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
fn parse_small_script(c: &mut Criterion) {
|
|
||||||
let script_path = Path::new("/tmp/bench_small.sh");
|
|
||||||
|
|
||||||
let script_content = fs::read_to_string(script_path)
|
|
||||||
.expect("Failed to read benchmark script. Run `python3 benches/generate.py` first.");
|
|
||||||
|
|
||||||
c.bench_function("parse_script_small", |b| {
|
|
||||||
b.iter(|| {
|
|
||||||
let result =
|
|
||||||
workshop_baker::bake::parser::parse_script(black_box(&script_content.clone()));
|
|
||||||
black_box(result)
|
|
||||||
});
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
criterion_group!(
|
|
||||||
benches,
|
|
||||||
parse_small_script,
|
|
||||||
parse_medium_script,
|
|
||||||
parse_large_script
|
|
||||||
);
|
|
||||||
criterion_main!(benches);
|
|
||||||
@@ -1,11 +0,0 @@
|
|||||||
[pytest]
|
|
||||||
testpaths = tests/integration
|
|
||||||
python_files = test_*.py
|
|
||||||
python_classes = Test*
|
|
||||||
python_functions = test_*
|
|
||||||
addopts = -v --tb=short
|
|
||||||
markers =
|
|
||||||
integration: marks tests as integration tests (deselect with '-m "not integration"')
|
|
||||||
slow: marks tests as slow running
|
|
||||||
filterwarnings =
|
|
||||||
ignore::DeprecationWarning
|
|
||||||
@@ -1,62 +0,0 @@
|
|||||||
# workshop-baker/src
|
|
||||||
|
|
||||||
**Generated:** 2026-04-22
|
|
||||||
**Commit:** 7b3b71a
|
|
||||||
|
|
||||||
Baker CLI + execution engine for HoneyBiscuitWorkshop pipelines.
|
|
||||||
|
|
||||||
## OVERVIEW
|
|
||||||
|
|
||||||
CLI tool and runtime for executing CI/CD pipelines with resource isolation, privilege dropping, and package management integration.
|
|
||||||
|
|
||||||
## STRUCTURE
|
|
||||||
|
|
||||||
```
|
|
||||||
src/
|
|
||||||
├── bake.rs # Top-level bake orchestration
|
|
||||||
├── bake/ # Script parsing and building
|
|
||||||
│ ├── parser.rs # @decorator-aware shell parser
|
|
||||||
│ ├── decorator.rs # @pipeline, @retry, @parallel, etc.
|
|
||||||
│ ├── builder.rs # Script template builder
|
|
||||||
│ └── schedule.rs # Function ordering via @after deps
|
|
||||||
├── engine/ # Execution runtime
|
|
||||||
│ ├── executor.rs # Script execution with cgroups
|
|
||||||
│ ├── cgroups.rs # Linux cgroup resource limits
|
|
||||||
│ ├── repology.rs # Package name resolution
|
|
||||||
│ ├── pm.rs # Package manager detection
|
|
||||||
│ └── upm.rs # User package manager (Nix/Guix)
|
|
||||||
├── prebake/ # Build environment setup
|
|
||||||
│ ├── config.rs # PrebakeConfig YAML schema
|
|
||||||
│ ├── stage/ # Bootstrap, DepsSystem, DepsUser, Hooks
|
|
||||||
│ └── security.rs # Privilege dropping logic
|
|
||||||
├── types/ # Shared type definitions
|
|
||||||
│ ├── repology.rs # RepologyEndpoint enum
|
|
||||||
│ ├── builderconfig.rs
|
|
||||||
│ ├── cache.rs
|
|
||||||
│ └── memsize.rs
|
|
||||||
├── monitor/ # Resource usage monitoring
|
|
||||||
│ ├── cgroups.rs # cgroup-based usage collection
|
|
||||||
│ └── jobobject.rs # Windows job objects (stub)
|
|
||||||
├── daemon.rs # Unix socket daemon (TODO: refactor)
|
|
||||||
└── lib.rs # CLI struct + module exports
|
|
||||||
```
|
|
||||||
|
|
||||||
## WHERE TO LOOK
|
|
||||||
|
|
||||||
| Task | Location |
|
|
||||||
|------|----------|
|
|
||||||
| Script parsing with @decorators | `bake/parser.rs:29` - `parse_script()` |
|
|
||||||
| Decorator types | `bake/decorator.rs` - `Decorator` enum |
|
|
||||||
| Build execution | `engine/executor.rs` - `Executor::execute_script()` |
|
|
||||||
| Resource limits | `engine/types.rs:56` - `ResourceLimits` struct |
|
|
||||||
| Prebake stages | `prebake/stage/` subdirectories |
|
|
||||||
| Package detection | `engine/pm.rs:detection()` |
|
|
||||||
| CLI entry | `lib.rs:26` - `cli::Cli` struct |
|
|
||||||
|
|
||||||
## CONVENTIONS
|
|
||||||
|
|
||||||
- **ExecutionContext**: Passed through all stages, carries task_id, username, env_vars, timeout
|
|
||||||
- **PrebakeStage**: Ordered enum controlling stage execution sequence (Bootstrap -> Ready)
|
|
||||||
- **Decorator**: Attached to shell functions via `# @name(args)` comments above function declarations
|
|
||||||
- **Engine**: Lightweight wrapper around cgroup manager for resource isolation
|
|
||||||
- **Dual-mode**: CLI commands (prebake, bake, finalize) vs daemon mode via Unix socket
|
|
||||||
+177
-55
@@ -1,20 +1,27 @@
|
|||||||
use crate::bake::constant::{DEFAULT_WORKSPACE, TRIVIAL_SCRIPT_NAME};
|
use crate::types::debug::DebugFeature;
|
||||||
|
use crate::bake::builder::RenderMode;
|
||||||
|
use crate::bake::constant::DEFAULT_WORKSPACE;
|
||||||
|
use crate::bake::decorator::Decorator;
|
||||||
use crate::bake::error::BakeError;
|
use crate::bake::error::BakeError;
|
||||||
use crate::bake::parser::Function;
|
use crate::bake::util::{resolve_mode, FuncMode};
|
||||||
use crate::cli::Cli;
|
use crate::cli::Cli;
|
||||||
use crate::prebake;
|
use crate::prebake;
|
||||||
use crate::prebake::PrebakeConfig;
|
use crate::prebake::PrebakeConfig;
|
||||||
use crate::prebake::security::get_drop_after;
|
use std::collections::{HashMap, HashSet};
|
||||||
use crate::prebake::stage::PrebakeStage;
|
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
|
use std::time::Duration;
|
||||||
|
use workshop_baker_params::BakeSettings;
|
||||||
use workshop_engine::{Engine, EventSender, ExecutionContext};
|
use workshop_engine::{Engine, EventSender, ExecutionContext};
|
||||||
|
|
||||||
mod builder;
|
mod builder;
|
||||||
pub mod constant;
|
pub mod constant;
|
||||||
mod decorator;
|
mod decorator;
|
||||||
pub mod error;
|
pub mod error;
|
||||||
|
mod execute;
|
||||||
pub mod parser;
|
pub mod parser;
|
||||||
mod schedule;
|
mod schedule;
|
||||||
|
pub mod trivial;
|
||||||
|
pub mod util;
|
||||||
|
|
||||||
pub async fn bake(
|
pub async fn bake(
|
||||||
script_path: &Path,
|
script_path: &Path,
|
||||||
@@ -23,12 +30,18 @@ pub async fn bake(
|
|||||||
_cli: &Cli,
|
_cli: &Cli,
|
||||||
ctx: &mut ExecutionContext,
|
ctx: &mut ExecutionContext,
|
||||||
event_tx: EventSender,
|
event_tx: EventSender,
|
||||||
|
nevent_tx: Option<crate::notify::types::NotificationEventSender>,
|
||||||
) -> Result<(), BakeError> {
|
) -> Result<(), BakeError> {
|
||||||
let script_content = std::fs::read_to_string(script_path)?;
|
log::debug!("Reading script: {:?}", script_path);
|
||||||
let prebake_content = std::fs::read_to_string(prebake_path)?;
|
let script_content = std::fs::read_to_string(script_path).map_err(|e| {
|
||||||
|
BakeError::IoError { path: script_path.display().to_string(), source: e }
|
||||||
|
})?;
|
||||||
|
log::debug!("Reading prebake.yml: {:?}", prebake_path);
|
||||||
|
let prebake_content = std::fs::read_to_string(prebake_path).map_err(|e| {
|
||||||
|
BakeError::IoError { path: prebake_path.display().to_string(), source: e }
|
||||||
|
})?;
|
||||||
let prebake = prebake::parse(prebake_content.as_str()).map_err(|e| {
|
let prebake = prebake::parse(prebake_content.as_str()).map_err(|e| {
|
||||||
log::error!("Failed to parse prebake.yml: {}", e);
|
log::error!("Failed to parse prebake.yml: {}", e);
|
||||||
log::error!("This is unexpected!");
|
|
||||||
BakeError::YamlParseError(e)
|
BakeError::YamlParseError(e)
|
||||||
})?;
|
})?;
|
||||||
let workspace = get_workspace(&prebake);
|
let workspace = get_workspace(&prebake);
|
||||||
@@ -39,44 +52,174 @@ pub async fn bake(
|
|||||||
.iter()
|
.iter()
|
||||||
.any(|d| matches!(d, decorator::Decorator::Pipeline))
|
.any(|d| matches!(d, decorator::Decorator::Pipeline))
|
||||||
});
|
});
|
||||||
let trivial = false;
|
// TODO: if PIP(pipeline implicit parameter).trivial is set, always use RenderMode::Off
|
||||||
let use_template = true;
|
let render_mode = RenderMode::Full;
|
||||||
|
|
||||||
let engine = Engine::new();
|
let engine = Engine::new();
|
||||||
|
// TODO: resolve from merged param layers (PIP integration)
|
||||||
|
let bake_settings = BakeSettings::default();
|
||||||
if let Some(env) = &prebake.envvars
|
if let Some(env) = &prebake.envvars
|
||||||
&& let Some(prebake_env) = &env.bake
|
&& let Some(prebake_env) = &env.bake
|
||||||
{
|
{
|
||||||
ctx.env_vars.extend(prebake_env.clone());
|
ctx.env_vars.extend(prebake_env.clone());
|
||||||
}
|
}
|
||||||
|
|
||||||
if !has_pipeline || trivial {
|
if !has_pipeline {
|
||||||
let function = Function {
|
log::trace!("Running trivial script");
|
||||||
name: TRIVIAL_SCRIPT_NAME.to_string(),
|
trivial::run_trivial(
|
||||||
decorators: vec![],
|
&script_content,
|
||||||
body: script_content,
|
bake_base_path,
|
||||||
};
|
&workspace,
|
||||||
let script =
|
&engine,
|
||||||
builder::build_script(&function, bake_base_path, &workspace, None, use_template)?;
|
ctx,
|
||||||
let result = engine.execute_script(&script, ctx, &event_tx).await;
|
&event_tx,
|
||||||
result?;
|
render_mode,
|
||||||
|
)
|
||||||
|
.await?;
|
||||||
} else {
|
} else {
|
||||||
let sorted_functions = schedule::sort_function(functions.pipeline_functions)?;
|
log::trace!("Running pipeline script");
|
||||||
let remaining = functions.remaining_code;
|
let remaining = functions.remaining_code;
|
||||||
for func in sorted_functions {
|
let (mut dag, func_map) = schedule::build_dag(functions.pipeline_functions)?;
|
||||||
let mut modified_func = func.clone();
|
|
||||||
modified_func.body = remaining.clone() + &func.body;
|
// Pre-scan: pipe sources + decorator conflict checks.
|
||||||
let script = builder::build_script(
|
let mut pipe_sources: HashSet<String> = HashSet::new();
|
||||||
&modified_func,
|
for func in func_map.values() {
|
||||||
bake_base_path,
|
decorator::check_conflicts(func)?;
|
||||||
&workspace,
|
for decorator in &func.decorators {
|
||||||
None,
|
if let Decorator::Pipe(names) = decorator {
|
||||||
use_template,
|
for source in names {
|
||||||
)?;
|
if *source != func.name {
|
||||||
let result = engine.execute_script(&script, ctx, &event_tx).await;
|
pipe_sources.insert(source.clone());
|
||||||
result?;
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut stdout_cache: HashMap<String, String> = HashMap::new();
|
||||||
|
let mut async_handles: Vec<tokio::task::JoinHandle<()>> = Vec::new();
|
||||||
|
|
||||||
|
while !dag.is_empty() {
|
||||||
|
let scheduler_debug = DebugFeature::from_bits_retain(ctx.debug_flags).contains(DebugFeature::Scheduler);
|
||||||
|
if scheduler_debug {
|
||||||
|
log::info!("[BAKE_SCHED] ready={} seq={} wild={} groups={:?}",
|
||||||
|
dag.ready_count(),
|
||||||
|
dag.ready_sequential().len(),
|
||||||
|
dag.ready_wildcards().len(),
|
||||||
|
dag.ready_groups(),
|
||||||
|
);
|
||||||
|
} else {
|
||||||
|
log::debug!("[BAKE_SCHED] ready={} seq={} wild={} groups={:?}",
|
||||||
|
dag.ready_count(),
|
||||||
|
dag.ready_sequential().len(),
|
||||||
|
dag.ready_wildcards().len(),
|
||||||
|
dag.ready_groups(),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
// Rule 1: sequential always goes first
|
||||||
|
let seq_candidates = dag.ready_sequential();
|
||||||
|
if let Some(seq_name) = seq_candidates.first() {
|
||||||
|
let seq_name = seq_name.to_string();
|
||||||
|
log::info!("[BAKE_SCHED] seq {}", seq_name);
|
||||||
|
let func = &func_map[&seq_name];
|
||||||
|
let mode = resolve_mode(func);
|
||||||
|
|
||||||
|
match mode {
|
||||||
|
FuncMode::Daemon => {
|
||||||
|
execute::execute_daemon(
|
||||||
|
&engine, func, ctx, &event_tx,
|
||||||
|
&workspace, &remaining, bake_base_path,
|
||||||
|
Duration::from_secs(bake_settings.health_period),
|
||||||
|
Duration::from_secs(bake_settings.health_timeout),
|
||||||
|
bake_settings.health_max_retries,
|
||||||
|
).await?;
|
||||||
|
}
|
||||||
|
FuncMode::Async => {
|
||||||
|
let handle = execute::execute_async(
|
||||||
|
func, ctx, &event_tx,
|
||||||
|
&workspace, &remaining, bake_base_path,
|
||||||
|
);
|
||||||
|
async_handles.push(handle);
|
||||||
|
}
|
||||||
|
FuncMode::Normal => {
|
||||||
|
let result = execute::execute_normal(
|
||||||
|
&engine, func, ctx, &event_tx,
|
||||||
|
&workspace, &remaining, bake_base_path, &stdout_cache,
|
||||||
|
).await;
|
||||||
|
match result {
|
||||||
|
Ok(stdout) => {
|
||||||
|
if pipe_sources.contains(&seq_name) {
|
||||||
|
stdout_cache.insert(seq_name.clone(), stdout);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Err(e) if dag.is_fallible(&seq_name) => {
|
||||||
|
log::warn!("'{}' failed (fallible): {}", seq_name, e);
|
||||||
|
}
|
||||||
|
Err(e) => return Err(e),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Pop AFTER execution (Daemon/Async pop at start)
|
||||||
|
if !matches!(mode, FuncMode::Daemon | FuncMode::Async) {
|
||||||
|
dag.pop(&seq_name).map_err(|e| BakeError::IncompatibleDecorators {
|
||||||
|
function: seq_name.clone(),
|
||||||
|
reason: e.to_string(),
|
||||||
|
})?;
|
||||||
|
} else {
|
||||||
|
dag.pop(&seq_name).map_err(|e| BakeError::IncompatibleDecorators {
|
||||||
|
function: seq_name.clone(),
|
||||||
|
reason: e.to_string(),
|
||||||
|
})?;
|
||||||
|
}
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
// All remaining are parallel — pick a batch
|
||||||
|
if let Some(batch) = dag.pick_batch() {
|
||||||
|
log::info!("[BAKE_SCHED] par {:?}", batch);
|
||||||
|
let results = execute::fire_parallel_batch(
|
||||||
|
&batch, &engine, &func_map, ctx, &event_tx,
|
||||||
|
&workspace, &remaining, bake_base_path, &stdout_cache,
|
||||||
|
).await?;
|
||||||
|
for (name, stdout) in results {
|
||||||
|
if pipe_sources.contains(&name) {
|
||||||
|
stdout_cache.insert(name.clone(), stdout);
|
||||||
|
}
|
||||||
|
if let Err(e) = dag.pop(&name) {
|
||||||
|
log::warn!("Failed to pop '{}' from DAG: {}", name, e);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else if dag.is_stalled() {
|
||||||
|
// Deadlock: named parallel groups exist but can't complete
|
||||||
|
let stalled = dag.stalled_groups();
|
||||||
|
let remaining = dag.remaining();
|
||||||
|
return Err(BakeError::IncompatibleDecorators {
|
||||||
|
function: remaining.join(", "),
|
||||||
|
reason: format!(
|
||||||
|
"parallel group(s) {:?} can never complete — blocked by @after chain",
|
||||||
|
stalled
|
||||||
|
),
|
||||||
|
});
|
||||||
|
} else {
|
||||||
|
// Ready set is empty — cycle
|
||||||
|
let remaining = dag.remaining();
|
||||||
|
if !remaining.is_empty() {
|
||||||
|
return Err(BakeError::CircularDependency(
|
||||||
|
remaining.into_iter().map(|s| s.to_string()).collect(),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Wait for all @async tasks to finish
|
||||||
|
for handle in async_handles {
|
||||||
|
let _ = handle.await;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
crate::notify::types::send_stage_event(&nevent_tx, crate::types::buildstatus::StagePhase::Bake, "bake", crate::types::buildstatus::StageOutcome::Success).await;
|
||||||
|
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -95,26 +238,5 @@ fn get_workspace(prebake_config: &PrebakeConfig) -> PathBuf {
|
|||||||
.unwrap_or_else(|| PathBuf::from(DEFAULT_WORKSPACE))
|
.unwrap_or_else(|| PathBuf::from(DEFAULT_WORKSPACE))
|
||||||
}
|
}
|
||||||
|
|
||||||
fn privileged(prebake_config: &PrebakeConfig) -> bool {
|
// ── AI Coding Agent marker ──
|
||||||
if let Some(bootstrap) = &prebake_config.bootstrap
|
// Code in this module PARTIALLY or FULLY utilized AI Coding Agent
|
||||||
&& bootstrap.user == "root"
|
|
||||||
{
|
|
||||||
return true;
|
|
||||||
}
|
|
||||||
let drop_after = get_drop_after(&prebake_config.security).unwrap_or_default();
|
|
||||||
PrebakeStage::Never == drop_after
|
|
||||||
}
|
|
||||||
|
|
||||||
type TaskResult = Result<(), BakeError>;
|
|
||||||
|
|
||||||
type TaskFn = Box<dyn FnOnce() -> TaskResult + Send>;
|
|
||||||
|
|
||||||
// TODO: Learn closure and continue
|
|
||||||
// async fn run<F>(f:F) -> TaskFn {
|
|
||||||
// let engine = Engine::new();
|
|
||||||
// engine.execute_script
|
|
||||||
// }
|
|
||||||
|
|
||||||
// fn with_timeout<F>(f: F, duration: i32) -> TaskFn {
|
|
||||||
|
|
||||||
// }
|
|
||||||
|
|||||||
@@ -1,46 +0,0 @@
|
|||||||
# workshop-baker/src/bake
|
|
||||||
|
|
||||||
**Generated:** 2026-04-22
|
|
||||||
**Commit:** 7b3b71a
|
|
||||||
|
|
||||||
Script parsing and build orchestration with decorator DSL.
|
|
||||||
|
|
||||||
## STRUCTURE
|
|
||||||
|
|
||||||
```
|
|
||||||
bake/
|
|
||||||
├── parser.rs # @decorator-aware shell script parser
|
|
||||||
├── decorator.rs # Decorator enum + 100+ unit tests
|
|
||||||
├── builder.rs # Script template assembly
|
|
||||||
└── schedule.rs # Function ordering via @after dependencies
|
|
||||||
```
|
|
||||||
|
|
||||||
## WHERE TO LOOK
|
|
||||||
|
|
||||||
| Task | Location |
|
|
||||||
|------|----------|
|
|
||||||
| Parse script | `parser.rs:29` - `parse_script()` |
|
|
||||||
| Decorator types | `decorator.rs` - `Decorator` enum variants |
|
|
||||||
| Build script | `builder.rs` - template builder |
|
|
||||||
| Schedule deps | `schedule.rs` - @after resolution |
|
|
||||||
|
|
||||||
## DECORATORS
|
|
||||||
|
|
||||||
| Decorator | Purpose |
|
|
||||||
|-----------|---------|
|
|
||||||
| `@pipeline` | Marks main build function |
|
|
||||||
| `@timeout(N)` | Execution timeout seconds |
|
|
||||||
| `@retry(N, M)` | Retry N times, M sec delay |
|
|
||||||
| `@parallel` | Run concurrently |
|
|
||||||
| `@fallible` | Failure doesn't fail pipeline |
|
|
||||||
| `@after(func)` | Dependency ordering |
|
|
||||||
| `@export` | Export vars to env |
|
|
||||||
| `@loop(N)` | Repeat N times |
|
|
||||||
| `@if(COND)` | Conditional execution |
|
|
||||||
|
|
||||||
## CONVENTIONS
|
|
||||||
|
|
||||||
- **Decorator Syntax**: `# @name(args)` comment above function
|
|
||||||
- **Parsing**: Shell grammar with decorator annotation extraction
|
|
||||||
- **Scheduling**: DAG resolution for @after dependencies
|
|
||||||
- **Tests**: `decorator.rs:175-829` has 100+ inline unit tests
|
|
||||||
@@ -1,4 +1,5 @@
|
|||||||
use crate::bake::constant::{PIPELINE_SKIP_ERRORCODE, TRIVIAL_SCRIPT_NAME};
|
use crate::types::debug::DebugFeature;
|
||||||
|
use crate::bake::constant::PIPELINE_SKIP_ERRORCODE;
|
||||||
use crate::bake::decorator::Decorator;
|
use crate::bake::decorator::Decorator;
|
||||||
use crate::bake::error::BakeError;
|
use crate::bake::error::BakeError;
|
||||||
use crate::bake::parser::Function;
|
use crate::bake::parser::Function;
|
||||||
@@ -9,24 +10,45 @@ use std::path::Path;
|
|||||||
use std::path::PathBuf;
|
use std::path::PathBuf;
|
||||||
use tempfile::NamedTempFile;
|
use tempfile::NamedTempFile;
|
||||||
|
|
||||||
/// Template for use_template=false, which just executes the main body without any wrapping.
|
/// Three-level rendering control.
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||||
|
pub enum RenderMode {
|
||||||
|
/// No template, hardcoded "{{ main }}". Debug escape hatch.
|
||||||
|
Off,
|
||||||
|
/// Uses bake_base.sh with essential execution skeleton only.
|
||||||
|
Minimal,
|
||||||
|
/// Uses bake_base.sh with helpers (logs, colors, etc.).
|
||||||
|
Full,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl RenderMode {
|
||||||
|
pub fn as_str(&self) -> &'static str {
|
||||||
|
match self {
|
||||||
|
RenderMode::Off => "off",
|
||||||
|
RenderMode::Minimal => "minimal",
|
||||||
|
RenderMode::Full => "full",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Hardcoded template for RenderMode::Off.
|
||||||
const TRIVIAL_TEMPLATE: &str = "{{ main }}";
|
const TRIVIAL_TEMPLATE: &str = "{{ main }}";
|
||||||
|
|
||||||
/// Renders a `Function` into an executable shell script using a minijinja template.
|
/// Renders a `Function` into an executable shell script using a minijinja template.
|
||||||
/// If `use_template` is true, wraps the body in `bake_base`; otherwise uses a trivial template.
|
///
|
||||||
|
/// `render_mode` controls the template source:
|
||||||
|
/// - `Off`: hardcoded `{{ main }}`, no bake_base file needed.
|
||||||
|
/// - `Minimal` / `Full`: reads bake_base.sh; the template distinguishes the two
|
||||||
|
/// via the `render_mode` context variable.
|
||||||
pub fn build_script(
|
pub fn build_script(
|
||||||
function: &Function,
|
function: &Function,
|
||||||
bake_base: &Path,
|
bake_base: &Path,
|
||||||
target_dir: &Path,
|
target_dir: &Path,
|
||||||
temp_path: Option<PathBuf>,
|
render_mode: RenderMode,
|
||||||
use_template: bool,
|
debug_flags: u32,
|
||||||
) -> Result<PathBuf, BakeError> {
|
) -> Result<PathBuf, BakeError> {
|
||||||
let name = &function.name;
|
let name = &function.name;
|
||||||
let trivial = name == TRIVIAL_SCRIPT_NAME;
|
|
||||||
let body = &function.body;
|
let body = &function.body;
|
||||||
let export = function
|
|
||||||
.find_decorator(|d| matches!(d, Decorator::Export).then_some(()))
|
|
||||||
.is_some();
|
|
||||||
let condition = function.find_decorator(|d| {
|
let condition = function.find_decorator(|d| {
|
||||||
if let Decorator::If(content) = d {
|
if let Decorator::If(content) = d {
|
||||||
Some(content.clone())
|
Some(content.clone())
|
||||||
@@ -34,57 +56,59 @@ pub fn build_script(
|
|||||||
None
|
None
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
let temp_path = temp_path.unwrap_or_else(|| PathBuf::from("/tmp"));
|
// let temp_path = temp_path.unwrap_or_else(|| PathBuf::from("/tmp"));
|
||||||
|
let temp_path = target_dir;
|
||||||
let filename = format!("bakefn_{}.sh", name);
|
let filename = format!("bakefn_{}.sh", name);
|
||||||
let target = target_dir.join(filename);
|
let target = target_dir.join(filename);
|
||||||
|
|
||||||
let bake_base_content = if use_template {
|
let template_content = match render_mode {
|
||||||
std::fs::read_to_string(bake_base).map_err(|e| BakeError::ScriptGenerationFailed {
|
RenderMode::Off => TRIVIAL_TEMPLATE.to_string(),
|
||||||
script: bake_base.display().to_string(),
|
RenderMode::Minimal | RenderMode::Full => {
|
||||||
reason: e.to_string(),
|
let script_debug = DebugFeature::from_bits_retain(debug_flags).contains(DebugFeature::Script);
|
||||||
})?
|
if script_debug {
|
||||||
} else {
|
log::info!("Reading bake_base: {:?}", bake_base);
|
||||||
TRIVIAL_TEMPLATE.to_string()
|
} else {
|
||||||
|
log::debug!("Reading bake_base: {:?}", bake_base);
|
||||||
|
}
|
||||||
|
std::fs::read_to_string(bake_base).map_err(|e| BakeError::IoError {
|
||||||
|
path: bake_base.display().to_string(),
|
||||||
|
source: e,
|
||||||
|
})?
|
||||||
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
let condition_code = match condition {
|
let condition_code = match condition {
|
||||||
Some(content) => {
|
Some(content) => {
|
||||||
|
// with some bash workaround
|
||||||
format!(
|
format!(
|
||||||
"if [ ! {} ]; then\n exit {}\nfi\n",
|
"if [[ ! {} ]]; then\n exit {}\nfi\n",
|
||||||
content, PIPELINE_SKIP_ERRORCODE
|
content, PIPELINE_SKIP_ERRORCODE
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
None => String::new(),
|
None => String::new(),
|
||||||
};
|
};
|
||||||
|
|
||||||
let mut preexport_code = String::new();
|
// preexport/export: stubs for the future IPC-based @export.
|
||||||
let mut export_code = String::new();
|
let preexport_code = String::new();
|
||||||
if !trivial {
|
let export_code = String::new();
|
||||||
if export {
|
|
||||||
preexport_code = format!("declare -xp > /dev/shm/bakeenv.before.{}\n", name);
|
|
||||||
export_code = format!(
|
|
||||||
"declare -xp > /dev/shm/bakeenv.after.{}\ndiff -w --new-line-format='%L' --old-line-format='' --unchanged-line-format='' /dev/shm/bakeenv.before.{} /dev/shm/bakeenv.after.{} > /dev/shm/bakeexport.{}\n",
|
|
||||||
name, name, name, name
|
|
||||||
);
|
|
||||||
}
|
|
||||||
preexport_code.push_str(format!(". /dev/shm/bakeexport.{}\n", name).as_str());
|
|
||||||
export_code.push_str(format!("rm -f /dev/shm/bakeenv.*.{}\n", name).as_str());
|
|
||||||
}
|
|
||||||
|
|
||||||
let mut env = Environment::new();
|
let mut env = Environment::new();
|
||||||
env.add_template("script", bake_base_content.as_str())?;
|
env.add_template("script", &template_content)?;
|
||||||
let script = env.get_template("script")?;
|
let rendered = env.get_template("script")?.render(
|
||||||
let rendered = if use_template {
|
context! {
|
||||||
script.render(
|
main => body,
|
||||||
context! {main => body,condition => condition_code, preexport => preexport_code, name => name, export => export_code},
|
condition => condition_code,
|
||||||
)?
|
preexport => preexport_code,
|
||||||
} else {
|
name => name,
|
||||||
script.render(context! {main => body})?
|
export => export_code,
|
||||||
};
|
render_mode => render_mode.as_str(),
|
||||||
|
},
|
||||||
|
)?;
|
||||||
|
|
||||||
let mut temp_file =
|
let mut temp_file =
|
||||||
NamedTempFile::new_in(&temp_path).map_err(|e| BakeError::ScriptGenerationFailed {
|
NamedTempFile::new_in(&temp_path).map_err(|e| BakeError::ScriptGenerationFailed {
|
||||||
script: temp_path.display().to_string(),
|
script: temp_path.display().to_string(),
|
||||||
|
action: "creating temp file".to_string(),
|
||||||
reason: e.to_string(),
|
reason: e.to_string(),
|
||||||
})?;
|
})?;
|
||||||
|
|
||||||
@@ -93,17 +117,20 @@ pub fn build_script(
|
|||||||
.set_permissions(Permissions::from_mode(0o700))
|
.set_permissions(Permissions::from_mode(0o700))
|
||||||
.map_err(|e| BakeError::ScriptGenerationFailed {
|
.map_err(|e| BakeError::ScriptGenerationFailed {
|
||||||
script: temp_path.display().to_string(),
|
script: temp_path.display().to_string(),
|
||||||
|
action: "setting permissions".to_string(),
|
||||||
reason: e.to_string(),
|
reason: e.to_string(),
|
||||||
})?;
|
})?;
|
||||||
|
|
||||||
std::fs::write(&temp_file, rendered).map_err(|e| BakeError::ScriptGenerationFailed {
|
std::fs::write(&temp_file, rendered).map_err(|e| BakeError::ScriptGenerationFailed {
|
||||||
script: temp_path.display().to_string(),
|
script: temp_path.display().to_string(),
|
||||||
|
action: "writing to temp file".to_string(),
|
||||||
reason: e.to_string(),
|
reason: e.to_string(),
|
||||||
})?;
|
})?;
|
||||||
temp_file
|
temp_file
|
||||||
.persist(&target)
|
.persist(&target)
|
||||||
.map_err(|e| BakeError::ScriptGenerationFailed {
|
.map_err(|e| BakeError::ScriptGenerationFailed {
|
||||||
script: target.display().to_string(),
|
script: target.display().to_string(),
|
||||||
|
action: "persisting temp file".to_string(),
|
||||||
reason: e.error.to_string(),
|
reason: e.error.to_string(),
|
||||||
})?;
|
})?;
|
||||||
Ok(target)
|
Ok(target)
|
||||||
@@ -114,7 +141,7 @@ mod tests {
|
|||||||
use super::*;
|
use super::*;
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_build_script_with_template() {
|
fn test_build_script_full_mode() {
|
||||||
let temp_dir = tempfile::tempdir().unwrap();
|
let temp_dir = tempfile::tempdir().unwrap();
|
||||||
let bake_base = temp_dir.path().join("bake_base.sh");
|
let bake_base = temp_dir.path().join("bake_base.sh");
|
||||||
|
|
||||||
@@ -125,7 +152,9 @@ mod tests {
|
|||||||
decorators: vec![],
|
decorators: vec![],
|
||||||
body: "echo 'hello world'".to_string(),
|
body: "echo 'hello world'".to_string(),
|
||||||
};
|
};
|
||||||
let result = build_script(&function, &bake_base, temp_dir.path(), None, true);
|
let result = build_script(
|
||||||
|
&function, &bake_base, temp_dir.path(), RenderMode::Full, 0,
|
||||||
|
);
|
||||||
|
|
||||||
assert!(result.is_ok());
|
assert!(result.is_ok());
|
||||||
let script_path = result.unwrap();
|
let script_path = result.unwrap();
|
||||||
@@ -134,10 +163,10 @@ mod tests {
|
|||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_build_script_trivial_mode() {
|
fn test_build_script_off_mode() {
|
||||||
let temp_dir = tempfile::tempdir().unwrap();
|
let temp_dir = tempfile::tempdir().unwrap();
|
||||||
let function = Function {
|
let function = Function {
|
||||||
name: "trivial".to_string(),
|
name: "test".to_string(),
|
||||||
decorators: vec![],
|
decorators: vec![],
|
||||||
body: "echo 'direct output'".to_string(),
|
body: "echo 'direct output'".to_string(),
|
||||||
};
|
};
|
||||||
@@ -146,8 +175,8 @@ mod tests {
|
|||||||
&function,
|
&function,
|
||||||
Path::new("/nonexistent"),
|
Path::new("/nonexistent"),
|
||||||
temp_dir.path(),
|
temp_dir.path(),
|
||||||
None,
|
RenderMode::Off,
|
||||||
false,
|
0,
|
||||||
);
|
);
|
||||||
|
|
||||||
assert!(result.is_ok());
|
assert!(result.is_ok());
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
pub const DEFAULT_WORKSPACE: &str = "/workspace";
|
pub const DEFAULT_WORKSPACE: &str = "/workspace";
|
||||||
pub const BAKE_SCRIPT_PATH: &str = "bake.sh";
|
pub const BAKE_SCRIPT_PATH: &str = "bake.sh";
|
||||||
pub const TRIVIAL_SCRIPT_NAME: &str = "BAKE_TRIVIAL";
|
pub const TRIVIAL_SCRIPT_NAME: &str = "BAKE_TRIVIAL";
|
||||||
pub const PIPELINE_SKIP_ERRORCODE: u8 = 233;
|
pub const PIPELINE_SKIP_ERRORCODE: i32 = 233;
|
||||||
|
|||||||
@@ -4,11 +4,11 @@ use std::fmt::Display;
|
|||||||
use std::fmt::Formatter;
|
use std::fmt::Formatter;
|
||||||
|
|
||||||
use crate::bake::error::BakeError;
|
use crate::bake::error::BakeError;
|
||||||
|
use crate::bake::parser::Function;
|
||||||
|
|
||||||
lazy_static! {
|
lazy_static! {
|
||||||
// "# @decorator" or "# @decorator(parameters)"
|
// "# @decorator" or "# @decorator(parameters)"
|
||||||
static ref DECORATOR_REGEX: Regex = Regex::new(r"^\s*#\s+@(\w+)\s?(\((.*)\))?").unwrap()
|
static ref DECORATOR_REGEX: Regex = Regex::new(r"^\s*#\s+@(\w+)\s?(\((.*)\))?").unwrap();
|
||||||
;
|
|
||||||
static ref UNEXPECTED_ARGUMENT: &'static str = "decorator does not accept arguments";
|
static ref UNEXPECTED_ARGUMENT: &'static str = "decorator does not accept arguments";
|
||||||
static ref MISSING_ARGUMENT: &'static str = "decorator requires an argument";
|
static ref MISSING_ARGUMENT: &'static str = "decorator requires an argument";
|
||||||
static ref UNKNOWN_DECORATOR: &'static str = "unknown decorator";
|
static ref UNKNOWN_DECORATOR: &'static str = "unknown decorator";
|
||||||
@@ -39,11 +39,121 @@ pub enum Decorator {
|
|||||||
/// Chain functions via pipe with comma-separated names.
|
/// Chain functions via pipe with comma-separated names.
|
||||||
Pipe(Vec<String>),
|
Pipe(Vec<String>),
|
||||||
/// Run concurrently with other parallel functions.
|
/// Run concurrently with other parallel functions.
|
||||||
Parallel,
|
/// `None` = wildcard ("*" group, compatible with any group).
|
||||||
|
/// `Some(name)` = restricted to the named group.
|
||||||
|
Parallel(Option<String>),
|
||||||
/// Run as a background daemon process.
|
/// Run as a background daemon process.
|
||||||
Daemon,
|
Daemon,
|
||||||
/// Health check endpoint for daemon verification.
|
/// Health check command for daemon verification (shell statement).
|
||||||
Health(String),
|
Health(String),
|
||||||
|
/// Fire-and-forget execution; DAG pop happens at start, not end.
|
||||||
|
Async,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns true if the decorator is a flag (position-independent, consumed after execution).
|
||||||
|
pub fn is_flag(d: &Decorator) -> bool {
|
||||||
|
matches!(
|
||||||
|
d,
|
||||||
|
Decorator::Pipeline
|
||||||
|
| Decorator::Fallible
|
||||||
|
| Decorator::Export
|
||||||
|
| Decorator::If(_)
|
||||||
|
| Decorator::After(_)
|
||||||
|
| Decorator::Parallel(_)
|
||||||
|
| Decorator::Health(_)
|
||||||
|
| Decorator::Async
|
||||||
|
| Decorator::Unknown
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns true if the decorator is a combinator (position-dependent, nestable).
|
||||||
|
pub fn is_combinator(d: &Decorator) -> bool {
|
||||||
|
matches!(d, Decorator::Loop(_) | Decorator::Timeout(_) | Decorator::Retry(..))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns true if the decorator is a mode (at most one, alters what gets executed).
|
||||||
|
pub fn is_mode(d: &Decorator) -> bool {
|
||||||
|
matches!(d, Decorator::Pipe(_) | Decorator::Daemon | Decorator::Async)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Extract the parallel group name. Returns "*" for wildcard (no group specified).
|
||||||
|
pub fn parallel_group(d: &Decorator) -> &str {
|
||||||
|
match d {
|
||||||
|
Decorator::Parallel(Some(g)) => g.as_str(),
|
||||||
|
_ => "*",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Check for incompatible decorator combinations on a function.
|
||||||
|
/// Returns `Err(IncompatibleDecorators)` on conflict, `Ok(())` otherwise.
|
||||||
|
pub fn check_conflicts(func: &Function) -> Result<(), BakeError> {
|
||||||
|
let has_daemon = func.decorators.iter().any(|d| matches!(d, Decorator::Daemon));
|
||||||
|
let has_async = func.decorators.iter().any(|d| matches!(d, Decorator::Async));
|
||||||
|
let has_pipe = func.decorators.iter().any(|d| matches!(d, Decorator::Pipe(_)));
|
||||||
|
let has_parallel = func.decorators.iter().any(|d| matches!(d, Decorator::Parallel(_)));
|
||||||
|
let has_health = func.decorators.iter().any(|d| matches!(d, Decorator::Health(_)));
|
||||||
|
let has_export = func.decorators.iter().any(|d| matches!(d, Decorator::Export));
|
||||||
|
let has_combinator = func.decorators.iter().any(|d| is_combinator(d));
|
||||||
|
|
||||||
|
let mode_count = [has_daemon, has_async, has_pipe].iter().filter(|&&x| x).count();
|
||||||
|
if mode_count > 1 {
|
||||||
|
return Err(BakeError::IncompatibleDecorators {
|
||||||
|
function: func.name.clone(),
|
||||||
|
reason: "@daemon, @async, and @pipe are mutually exclusive".into(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
if has_daemon {
|
||||||
|
if has_parallel {
|
||||||
|
return Err(BakeError::IncompatibleDecorators {
|
||||||
|
function: func.name.clone(),
|
||||||
|
reason: "@daemon cannot be combined with @parallel".into(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
if has_combinator {
|
||||||
|
return Err(BakeError::IncompatibleDecorators {
|
||||||
|
function: func.name.clone(),
|
||||||
|
reason: "@daemon cannot be combined with @loop/@retry/@timeout".into(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
if has_export {
|
||||||
|
return Err(BakeError::IncompatibleDecorators {
|
||||||
|
function: func.name.clone(),
|
||||||
|
reason: "@daemon cannot be combined with @export".into(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if has_health && !has_daemon {
|
||||||
|
return Err(BakeError::IncompatibleDecorators {
|
||||||
|
function: func.name.clone(),
|
||||||
|
reason: "@health requires @daemon".into(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
if has_async {
|
||||||
|
if has_parallel {
|
||||||
|
return Err(BakeError::IncompatibleDecorators {
|
||||||
|
function: func.name.clone(),
|
||||||
|
reason: "@async cannot be combined with @parallel".into(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
if has_export {
|
||||||
|
return Err(BakeError::IncompatibleDecorators {
|
||||||
|
function: func.name.clone(),
|
||||||
|
reason: "@async cannot be combined with @export".into(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if has_pipe && has_parallel {
|
||||||
|
return Err(BakeError::IncompatibleDecorators {
|
||||||
|
function: func.name.clone(),
|
||||||
|
reason: "@pipe cannot be combined with @parallel".into(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Display for Decorator {
|
impl Display for Decorator {
|
||||||
@@ -59,9 +169,13 @@ impl Display for Decorator {
|
|||||||
Decorator::Retry(count, delay) => write!(f, "@retry({}, {})", count, delay),
|
Decorator::Retry(count, delay) => write!(f, "@retry({}, {})", count, delay),
|
||||||
Decorator::Export => write!(f, "@export"),
|
Decorator::Export => write!(f, "@export"),
|
||||||
Decorator::Pipe(funcs) => write!(f, "@pipe({})", funcs.join(", ")),
|
Decorator::Pipe(funcs) => write!(f, "@pipe({})", funcs.join(", ")),
|
||||||
Decorator::Parallel => write!(f, "@parallel"),
|
Decorator::Parallel(group) => match group {
|
||||||
|
None => write!(f, "@parallel"),
|
||||||
|
Some(g) => write!(f, "@parallel({})", g),
|
||||||
|
},
|
||||||
Decorator::Daemon => write!(f, "@daemon"),
|
Decorator::Daemon => write!(f, "@daemon"),
|
||||||
Decorator::Health(check) => write!(f, "@health({})", check),
|
Decorator::Health(check) => write!(f, "@health({})", check),
|
||||||
|
Decorator::Async => write!(f, "@async"),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -161,9 +275,17 @@ pub fn parse_decorator(line: &str, line_num: usize) -> Result<Option<Decorator>,
|
|||||||
}
|
}
|
||||||
}),
|
}),
|
||||||
"export" => decorator_noarg(Decorator::Export),
|
"export" => decorator_noarg(Decorator::Export),
|
||||||
"parallel" => decorator_noarg(Decorator::Parallel),
|
"parallel" => {
|
||||||
|
if has_arg {
|
||||||
|
let group = required_arg()?;
|
||||||
|
Ok(Some(Decorator::Parallel(Some(group.to_string()))))
|
||||||
|
} else {
|
||||||
|
Ok(Some(Decorator::Parallel(None)))
|
||||||
|
}
|
||||||
|
}
|
||||||
"daemon" => decorator_noarg(Decorator::Daemon),
|
"daemon" => decorator_noarg(Decorator::Daemon),
|
||||||
"health" => required_arg().map(|a| Some(Decorator::Health(a.to_string()))),
|
"health" => required_arg().map(|a| Some(Decorator::Health(a.to_string()))),
|
||||||
|
"async" => decorator_noarg(Decorator::Async),
|
||||||
"pipe" => required_arg().and_then(|a| {
|
"pipe" => required_arg().and_then(|a| {
|
||||||
let functions: Vec<String> = a
|
let functions: Vec<String> = a
|
||||||
.split(',')
|
.split(',')
|
||||||
@@ -290,7 +412,11 @@ mod tests {
|
|||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_decorator_display_parallel() {
|
fn test_decorator_display_parallel() {
|
||||||
assert_eq!(format!("{}", Decorator::Parallel), "@parallel");
|
assert_eq!(format!("{}", Decorator::Parallel(None)), "@parallel");
|
||||||
|
assert_eq!(
|
||||||
|
format!("{}", Decorator::Parallel(Some("build".to_string()))),
|
||||||
|
"@parallel(build)"
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
@@ -310,6 +436,11 @@ mod tests {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_decorator_display_async() {
|
||||||
|
assert_eq!(format!("{}", Decorator::Async), "@async");
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_decorator_display_unknown() {
|
fn test_decorator_display_unknown() {
|
||||||
assert_eq!(format!("{}", Decorator::Unknown), "@<unknown>");
|
assert_eq!(format!("{}", Decorator::Unknown), "@<unknown>");
|
||||||
@@ -340,7 +471,13 @@ mod tests {
|
|||||||
#[test]
|
#[test]
|
||||||
fn test_parse_decorator_parallel() {
|
fn test_parse_decorator_parallel() {
|
||||||
let result = parse_decorator("# @parallel", 1).unwrap();
|
let result = parse_decorator("# @parallel", 1).unwrap();
|
||||||
assert_eq!(result, Some(Decorator::Parallel));
|
assert_eq!(result, Some(Decorator::Parallel(None)));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_parse_decorator_parallel_with_group() {
|
||||||
|
let result = parse_decorator("# @parallel(build)", 1).unwrap();
|
||||||
|
assert_eq!(result, Some(Decorator::Parallel(Some("build".to_string()))));
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
@@ -349,6 +486,12 @@ mod tests {
|
|||||||
assert_eq!(result, Some(Decorator::Daemon));
|
assert_eq!(result, Some(Decorator::Daemon));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_parse_decorator_async() {
|
||||||
|
let result = parse_decorator("# @async", 1).unwrap();
|
||||||
|
assert_eq!(result, Some(Decorator::Async));
|
||||||
|
}
|
||||||
|
|
||||||
// =====================================================================
|
// =====================================================================
|
||||||
// Tests for parse_decorator - Valid single-argument decorators
|
// Tests for parse_decorator - Valid single-argument decorators
|
||||||
// =====================================================================
|
// =====================================================================
|
||||||
@@ -576,8 +719,8 @@ mod tests {
|
|||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_parse_decorator_parallel_with_arg() {
|
fn test_parse_decorator_parallel_empty_args() {
|
||||||
let result = parse_decorator("# @parallel(true)", 1);
|
let result = parse_decorator("# @parallel()", 1);
|
||||||
assert!(result.is_err());
|
assert!(result.is_err());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -7,8 +7,8 @@ use workshop_engine::{
|
|||||||
|
|
||||||
#[derive(Error, Debug)]
|
#[derive(Error, Debug)]
|
||||||
pub enum BakeError {
|
pub enum BakeError {
|
||||||
#[error("Failed to generate staged script {script}: {reason}")]
|
#[error("Failed to generate staged script {script} when {action}: {reason}")]
|
||||||
ScriptGenerationFailed { script: String, reason: String },
|
ScriptGenerationFailed { script: String, action: String, reason: String },
|
||||||
|
|
||||||
#[error("Template error: {0}")]
|
#[error("Template error: {0}")]
|
||||||
TemplateError(#[from] minijinja::Error),
|
TemplateError(#[from] minijinja::Error),
|
||||||
@@ -19,8 +19,8 @@ pub enum BakeError {
|
|||||||
#[error("Unknown @after of function {0}: {1}")]
|
#[error("Unknown @after of function {0}: {1}")]
|
||||||
UnknownDependency(String, String),
|
UnknownDependency(String, String),
|
||||||
|
|
||||||
#[error("IO Error: {0}")]
|
#[error("IO error on {path}: {source}")]
|
||||||
IoError(#[from] std::io::Error),
|
IoError { path: String, source: std::io::Error },
|
||||||
|
|
||||||
#[error("YAML parse error: {0}")]
|
#[error("YAML parse error: {0}")]
|
||||||
YamlParseError(#[from] serde_yaml::Error),
|
YamlParseError(#[from] serde_yaml::Error),
|
||||||
@@ -34,6 +34,12 @@ pub enum BakeError {
|
|||||||
|
|
||||||
#[error("Script execution error: {0}")]
|
#[error("Script execution error: {0}")]
|
||||||
ScriptExecutionError(#[from] ExecutionError),
|
ScriptExecutionError(#[from] ExecutionError),
|
||||||
|
|
||||||
|
#[error("Incompatible decorators on function '{function}': {reason}")]
|
||||||
|
IncompatibleDecorators { function: String, reason: String },
|
||||||
|
|
||||||
|
#[error("Health probe failed for daemon '{command}': {error}")]
|
||||||
|
HealthProbeFailed { command: String, error: String },
|
||||||
}
|
}
|
||||||
|
|
||||||
impl HasExitCode for BakeError {
|
impl HasExitCode for BakeError {
|
||||||
@@ -43,10 +49,12 @@ impl HasExitCode for BakeError {
|
|||||||
BakeError::TemplateError(_) => EXITCODE_PARSE_ERROR,
|
BakeError::TemplateError(_) => EXITCODE_PARSE_ERROR,
|
||||||
BakeError::CircularDependency(_) => EXITCODE_PARSE_ERROR,
|
BakeError::CircularDependency(_) => EXITCODE_PARSE_ERROR,
|
||||||
BakeError::UnknownDependency(..) => EXITCODE_PARSE_ERROR,
|
BakeError::UnknownDependency(..) => EXITCODE_PARSE_ERROR,
|
||||||
BakeError::IoError(_) => EXITCODE_IO_ERROR,
|
BakeError::IoError { .. } => EXITCODE_IO_ERROR,
|
||||||
BakeError::YamlParseError(_) => EXITCODE_PARSE_ERROR,
|
BakeError::YamlParseError(_) => EXITCODE_PARSE_ERROR,
|
||||||
BakeError::DecoratorParseError { .. } => EXITCODE_PARSE_ERROR,
|
BakeError::DecoratorParseError { .. } => EXITCODE_PARSE_ERROR,
|
||||||
BakeError::ScriptExecutionError(e) => e.exit_code(),
|
BakeError::ScriptExecutionError(e) => e.exit_code(),
|
||||||
|
BakeError::IncompatibleDecorators { .. } => EXITCODE_PARSE_ERROR,
|
||||||
|
BakeError::HealthProbeFailed { .. } => EXITCODE_IO_ERROR,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,383 @@
|
|||||||
|
use crate::types::debug::DebugFeature;
|
||||||
|
use crate::bake::builder::{self, RenderMode};
|
||||||
|
use crate::bake::constant::PIPELINE_SKIP_ERRORCODE;
|
||||||
|
use crate::bake::decorator::{self, Decorator};
|
||||||
|
use crate::bake::error::BakeError;
|
||||||
|
use crate::bake::parser::Function;
|
||||||
|
use crate::bake::util::{is_fallible, resolve_pipe_stdin};
|
||||||
|
use std::collections::HashMap;
|
||||||
|
use std::future::Future;
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
use std::pin::Pin;
|
||||||
|
use std::process::Stdio;
|
||||||
|
use std::time::Duration;
|
||||||
|
use workshop_engine::{Engine, EventSender, ExecutionContext, ExecutionError};
|
||||||
|
|
||||||
|
/// Execute a single function with combinator wrapping.
|
||||||
|
/// Returns the captured stdout (for `@pipe` consumers).
|
||||||
|
pub async fn execute_one(
|
||||||
|
engine: &Engine,
|
||||||
|
func: &Function,
|
||||||
|
ctx: &mut ExecutionContext,
|
||||||
|
event_tx: &EventSender,
|
||||||
|
workspace: &Path,
|
||||||
|
remaining: &str,
|
||||||
|
bake_base: &Path,
|
||||||
|
) -> Result<String, BakeError> {
|
||||||
|
let combos: Vec<&Decorator> = func
|
||||||
|
.decorators
|
||||||
|
.iter()
|
||||||
|
.filter(|d| decorator::is_combinator(d))
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
let mut modified = func.clone();
|
||||||
|
modified.body = remaining.to_string() + &func.body;
|
||||||
|
let script = builder::build_script(
|
||||||
|
&modified, bake_base, workspace, RenderMode::Full, ctx.debug_flags,
|
||||||
|
)?;
|
||||||
|
|
||||||
|
execute_with_combos(engine, &script, ctx, event_tx, &combos).await
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn execute_with_combos(
|
||||||
|
engine: &Engine,
|
||||||
|
script: &PathBuf,
|
||||||
|
ctx: &mut ExecutionContext,
|
||||||
|
event_tx: &EventSender,
|
||||||
|
combos: &[&Decorator],
|
||||||
|
) -> Result<String, BakeError> {
|
||||||
|
build_combinator_chain(engine, script, ctx, event_tx, combos, 0).await
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Builds a combinator execution chain. Returns the captured stdout.
|
||||||
|
/// For `@loop` + `@pipe(self)`, each iteration feeds its stdout back as
|
||||||
|
/// the next iteration's stdin via `ctx.stdin`.
|
||||||
|
fn build_combinator_chain<'a>(
|
||||||
|
engine: &'a Engine,
|
||||||
|
script: &'a PathBuf,
|
||||||
|
ctx: &'a mut ExecutionContext,
|
||||||
|
event_tx: &'a EventSender,
|
||||||
|
combos: &'a [&'a Decorator],
|
||||||
|
idx: usize,
|
||||||
|
) -> Pin<Box<dyn Future<Output = Result<String, BakeError>> + Send + 'a>> {
|
||||||
|
if idx >= combos.len() {
|
||||||
|
return Box::pin(async move {
|
||||||
|
let result = engine.execute_script(script, ctx, event_tx).await;
|
||||||
|
match result {
|
||||||
|
Ok(r) => Ok(r.stdout),
|
||||||
|
Err(ExecutionError::ExecutionFailed { exit_code, .. })
|
||||||
|
if exit_code == PIPELINE_SKIP_ERRORCODE as i32 =>
|
||||||
|
{
|
||||||
|
log::debug!("Function skipped (@if condition)");
|
||||||
|
Ok(String::new())
|
||||||
|
}
|
||||||
|
Err(e) => Err(BakeError::ScriptExecutionError(e)),
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
match combos[idx] {
|
||||||
|
Decorator::Loop(count) => Box::pin(async move {
|
||||||
|
let mut last_stdout = String::new();
|
||||||
|
for i in 0..*count {
|
||||||
|
let script_debug = DebugFeature::from_bits_retain(ctx.debug_flags).contains(DebugFeature::Script);
|
||||||
|
if script_debug {
|
||||||
|
log::info!("Loop {}/{}", i + 1, count);
|
||||||
|
} else {
|
||||||
|
log::debug!("Loop {}/{}", i + 1, count);
|
||||||
|
}
|
||||||
|
last_stdout = build_combinator_chain(
|
||||||
|
engine, script, ctx, event_tx, combos, idx + 1,
|
||||||
|
)
|
||||||
|
.await?;
|
||||||
|
// Feed stdout as stdin for the next iteration (@pipe(self))
|
||||||
|
if i + 1 < *count {
|
||||||
|
ctx.stdin = Some(last_stdout.clone().into_bytes());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(last_stdout)
|
||||||
|
}),
|
||||||
|
Decorator::Timeout(secs) => Box::pin(async move {
|
||||||
|
tokio::time::timeout(
|
||||||
|
Duration::from_secs(*secs as u64),
|
||||||
|
build_combinator_chain(
|
||||||
|
engine, script, ctx, event_tx, combos, idx + 1,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.map_err(|_| BakeError::ScriptExecutionError(ExecutionError::Timeout))?
|
||||||
|
}),
|
||||||
|
Decorator::Retry(retries, delay_secs) => Box::pin(async move {
|
||||||
|
let max_attempts = *retries as usize + 1;
|
||||||
|
let delay = Duration::from_secs(*delay_secs as u64);
|
||||||
|
let mut last_err = None;
|
||||||
|
for attempt in 0..max_attempts {
|
||||||
|
match build_combinator_chain(
|
||||||
|
engine, script, ctx, event_tx, combos, idx + 1,
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
{
|
||||||
|
Ok(stdout) => return Ok(stdout),
|
||||||
|
Err(e) => {
|
||||||
|
if matches!(
|
||||||
|
&e,
|
||||||
|
BakeError::ScriptExecutionError(ExecutionError::Timeout)
|
||||||
|
) {
|
||||||
|
return Err(e);
|
||||||
|
}
|
||||||
|
if attempt < max_attempts - 1 {
|
||||||
|
last_err = Some(e);
|
||||||
|
log::warn!(
|
||||||
|
"Retry {}/{} after {}s",
|
||||||
|
attempt + 1,
|
||||||
|
retries,
|
||||||
|
delay_secs
|
||||||
|
);
|
||||||
|
tokio::time::sleep(delay).await;
|
||||||
|
} else {
|
||||||
|
return Err(e);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Err(last_err.unwrap())
|
||||||
|
}),
|
||||||
|
_ => build_combinator_chain(
|
||||||
|
engine, script, ctx, event_tx, combos, idx + 1,
|
||||||
|
),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Fire a batch of parallel nodes concurrently.
|
||||||
|
///
|
||||||
|
/// All nodes in the batch are waited on before returning. A non-`@fallible`
|
||||||
|
/// failure is surfaced only after all nodes complete (GitHub/GitLab/Concourse
|
||||||
|
/// convention).
|
||||||
|
///
|
||||||
|
/// Returns a `(name, stdout)` pair for each completed node so the caller can
|
||||||
|
/// populate the stdout cache for `@pipe`.
|
||||||
|
pub async fn fire_parallel_batch(
|
||||||
|
batch: &[String],
|
||||||
|
_engine: &Engine,
|
||||||
|
func_map: &HashMap<String, Function>,
|
||||||
|
ctx: &mut ExecutionContext,
|
||||||
|
event_tx: &EventSender,
|
||||||
|
workspace: &Path,
|
||||||
|
remaining: &str,
|
||||||
|
bake_base: &Path,
|
||||||
|
stdout_cache: &HashMap<String, String>,
|
||||||
|
) -> Result<Vec<(String, String)>, BakeError> {
|
||||||
|
let handles: Vec<_> = batch
|
||||||
|
.iter()
|
||||||
|
.map(|name| {
|
||||||
|
let function = func_map[name.as_str()].clone();
|
||||||
|
let mut task_ctx = ctx.clone();
|
||||||
|
// Resolve per-task stdin from @pipe + stdout_cache
|
||||||
|
task_ctx.stdin = resolve_pipe_stdin(&function, stdout_cache);
|
||||||
|
let task_events = event_tx.clone();
|
||||||
|
let task_workspace = workspace.to_path_buf();
|
||||||
|
let task_remaining = remaining.to_string();
|
||||||
|
let task_bake_base = bake_base.to_path_buf();
|
||||||
|
let task_name = name.clone();
|
||||||
|
tokio::spawn(async move {
|
||||||
|
let task_engine = Engine::new();
|
||||||
|
let result = execute_one(
|
||||||
|
&task_engine,
|
||||||
|
&function,
|
||||||
|
&mut task_ctx,
|
||||||
|
&task_events,
|
||||||
|
&task_workspace,
|
||||||
|
&task_remaining,
|
||||||
|
&task_bake_base,
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
(task_name, result)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
let mut first_fatal: Option<BakeError> = None;
|
||||||
|
let mut results: Vec<(String, String)> = Vec::with_capacity(handles.len());
|
||||||
|
for handle in handles {
|
||||||
|
let (name, result) = handle.await.map_err(|_| {
|
||||||
|
BakeError::ScriptExecutionError(ExecutionError::ExecutionFailed {
|
||||||
|
task_id: "parallel".to_string(),
|
||||||
|
exit_code: -1,
|
||||||
|
})
|
||||||
|
})?;
|
||||||
|
match result {
|
||||||
|
Ok(stdout) => results.push((name, stdout)),
|
||||||
|
Err(err) if is_fallible(&func_map[&name]) => {
|
||||||
|
log::warn!("'{}' failed (fallible, non-fatal): {}", name, err);
|
||||||
|
}
|
||||||
|
Err(err) => {
|
||||||
|
log::error!("'{}' failed (fatal): {}", name, err);
|
||||||
|
first_fatal.get_or_insert(err);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if let Some(err) = first_fatal {
|
||||||
|
return Err(err);
|
||||||
|
}
|
||||||
|
Ok(results)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Spawn an `@async` function in the background.
|
||||||
|
/// Returns a `JoinHandle` that the caller must await before returning.
|
||||||
|
pub fn execute_async(
|
||||||
|
func: &Function,
|
||||||
|
ctx: &ExecutionContext,
|
||||||
|
event_tx: &EventSender,
|
||||||
|
workspace: &Path,
|
||||||
|
remaining: &str,
|
||||||
|
bake_base: &Path,
|
||||||
|
) -> tokio::task::JoinHandle<()> {
|
||||||
|
let mut task_ctx = ctx.clone();
|
||||||
|
let task_events = event_tx.clone();
|
||||||
|
let task_workspace = workspace.to_path_buf();
|
||||||
|
let task_remaining = remaining.to_string();
|
||||||
|
let task_bake_base = bake_base.to_path_buf();
|
||||||
|
let task_func = func.clone();
|
||||||
|
tokio::spawn(async move {
|
||||||
|
let task_engine = Engine::new();
|
||||||
|
let _ = execute_one(
|
||||||
|
&task_engine, &task_func, &mut task_ctx,
|
||||||
|
&task_events, &task_workspace,
|
||||||
|
&task_remaining, &task_bake_base,
|
||||||
|
).await;
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Execute a Normal-mode function, resolving pipe stdin first.
|
||||||
|
pub async fn execute_normal(
|
||||||
|
engine: &Engine,
|
||||||
|
func: &Function,
|
||||||
|
ctx: &mut ExecutionContext,
|
||||||
|
event_tx: &EventSender,
|
||||||
|
workspace: &Path,
|
||||||
|
remaining: &str,
|
||||||
|
bake_base: &Path,
|
||||||
|
stdout_cache: &HashMap<String, String>,
|
||||||
|
) -> Result<String, BakeError> {
|
||||||
|
ctx.stdin = resolve_pipe_stdin(func, stdout_cache);
|
||||||
|
execute_one(engine, func, ctx, event_tx, workspace, remaining, bake_base).await
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Daemon / Health ─────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// Execute a `@daemon` function: build script, spawn, periodic health-probe
|
||||||
|
/// (blocking until healthy), then return.
|
||||||
|
pub async fn execute_daemon(
|
||||||
|
engine: &Engine,
|
||||||
|
func: &Function,
|
||||||
|
ctx: &ExecutionContext,
|
||||||
|
event_tx: &EventSender,
|
||||||
|
workspace: &Path,
|
||||||
|
remaining: &str,
|
||||||
|
bake_base: &Path,
|
||||||
|
health_period: Duration,
|
||||||
|
health_timeout: Duration,
|
||||||
|
health_max_retries: u32,
|
||||||
|
) -> Result<(), BakeError> {
|
||||||
|
let fallible = is_fallible(func);
|
||||||
|
|
||||||
|
let mut modified = func.clone();
|
||||||
|
modified.body = remaining.to_string() + &func.body;
|
||||||
|
let script = builder::build_script(
|
||||||
|
&modified, bake_base, workspace, RenderMode::Full, ctx.debug_flags,
|
||||||
|
)?;
|
||||||
|
|
||||||
|
// Spawn without waiting
|
||||||
|
let mut command = tokio::process::Command::new(ctx.shell.as_str());
|
||||||
|
command.arg("-c");
|
||||||
|
command.arg(script.to_str().unwrap());
|
||||||
|
command.current_dir(&ctx.working_dir);
|
||||||
|
command.envs(&ctx.env_vars);
|
||||||
|
command.stdin(Stdio::null());
|
||||||
|
command.stdout(Stdio::null());
|
||||||
|
command.stderr(Stdio::null());
|
||||||
|
#[cfg(unix)]
|
||||||
|
command.process_group(0);
|
||||||
|
|
||||||
|
//let child = // Necessary?
|
||||||
|
match command.spawn() {
|
||||||
|
Ok(c) => c,
|
||||||
|
Err(e) if fallible => {
|
||||||
|
log::warn!("Failed to spawn daemon '{}' (fallible): {}", func.name, e);
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
Err(_) => {
|
||||||
|
return Err(BakeError::ScriptExecutionError(ExecutionError::ExecutionFailed {
|
||||||
|
task_id: func.name.clone(),
|
||||||
|
exit_code: -1,
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
// Periodic health probe — block until healthy, retry exhausted, or fatal
|
||||||
|
if let Some(health_cmd) = func.decorators.iter().find_map(|d| match d {
|
||||||
|
Decorator::Health(cmd) => Some(cmd.as_str()),
|
||||||
|
_ => None,
|
||||||
|
}) {
|
||||||
|
let mut retries: u32 = 0;
|
||||||
|
loop {
|
||||||
|
let probe_result = tokio::time::timeout(
|
||||||
|
health_timeout,
|
||||||
|
health_probe_one(engine, health_cmd, remaining, ctx, event_tx),
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
|
||||||
|
match probe_result {
|
||||||
|
Ok(Ok(())) => break, // healthy — done
|
||||||
|
_ if retries + 1 < health_max_retries => {
|
||||||
|
retries += 1;
|
||||||
|
log::warn!(
|
||||||
|
"Health probe for '{}' failed ({}/{}), retrying in {:?}...",
|
||||||
|
func.name, retries, health_max_retries, health_period,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
_ if fallible => {
|
||||||
|
log::warn!("Health probe for '{}' exhausted (fallible), skipping", func.name);
|
||||||
|
break; // exhausted but fallible → skip health check
|
||||||
|
}
|
||||||
|
Ok(Err(e)) => return Err(e),
|
||||||
|
Err(_) => {
|
||||||
|
return Err(BakeError::HealthProbeFailed {
|
||||||
|
command: health_cmd.to_string(),
|
||||||
|
error: "timeout".to_string(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
tokio::time::sleep(health_period).await;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Cleanup is handled by finalize.
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Single health-probe attempt. Exit 0 = healthy.
|
||||||
|
async fn health_probe_one(
|
||||||
|
engine: &Engine,
|
||||||
|
cmd: &str,
|
||||||
|
remaining: &str,
|
||||||
|
ctx: &ExecutionContext,
|
||||||
|
event_tx: &EventSender,
|
||||||
|
) -> Result<(), BakeError> {
|
||||||
|
let full_cmd = format!("{}\n{}", remaining, cmd);
|
||||||
|
let result = engine
|
||||||
|
.execute_command(&full_cmd, ctx, event_tx)
|
||||||
|
.await
|
||||||
|
.map_err(|e| BakeError::HealthProbeFailed {
|
||||||
|
command: cmd.to_string(),
|
||||||
|
error: format!("{}", e),
|
||||||
|
})?;
|
||||||
|
if !result.success {
|
||||||
|
return Err(BakeError::HealthProbeFailed {
|
||||||
|
command: cmd.to_string(),
|
||||||
|
error: format!("exit code {}", result.exit_code),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
@@ -174,7 +174,7 @@ pub fn parse_script(text: &str) -> Result<ParsedScript, BakeError> {
|
|||||||
}
|
}
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
eprintln!(
|
log::warn!(
|
||||||
"Warning: Dangling decorator(s) at line {}. Treating as comments.",
|
"Warning: Dangling decorator(s) at line {}. Treating as comments.",
|
||||||
line_number - 1
|
line_number - 1
|
||||||
);
|
);
|
||||||
@@ -224,6 +224,7 @@ pub fn parse_script(text: &str) -> Result<ParsedScript, BakeError> {
|
|||||||
{
|
{
|
||||||
return Err(BakeError::ScriptGenerationFailed {
|
return Err(BakeError::ScriptGenerationFailed {
|
||||||
script: "bake.sh".to_string(),
|
script: "bake.sh".to_string(),
|
||||||
|
action: "parsing function body".to_string(),
|
||||||
reason: format!(
|
reason: format!(
|
||||||
"Unexpected EOF while parsing function '{}'. Unmatched braces (remaining: {})?",
|
"Unexpected EOF while parsing function '{}'. Unmatched braces (remaining: {})?",
|
||||||
name, brace_depth
|
name, brace_depth
|
||||||
@@ -234,6 +235,7 @@ pub fn parse_script(text: &str) -> Result<ParsedScript, BakeError> {
|
|||||||
let last_decorator = pending_decorators.last().unwrap();
|
let last_decorator = pending_decorators.last().unwrap();
|
||||||
return Err(BakeError::ScriptGenerationFailed {
|
return Err(BakeError::ScriptGenerationFailed {
|
||||||
script: "bake.sh".to_string(),
|
script: "bake.sh".to_string(),
|
||||||
|
action: "parsing decorators".to_string(),
|
||||||
reason: format!(
|
reason: format!(
|
||||||
"Dangling decorator {} before EOF. Missing function definition?",
|
"Dangling decorator {} before EOF. Missing function definition?",
|
||||||
last_decorator
|
last_decorator
|
||||||
|
|||||||
+143
-471
@@ -1,42 +1,124 @@
|
|||||||
use super::{decorator::Decorator, parser::Function};
|
// Code in this module PARTIALLY or FULLY utilized AI Coding Agent
|
||||||
|
|
||||||
|
use super::decorator::Decorator;
|
||||||
|
use super::parser::Function;
|
||||||
use crate::bake::error::BakeError;
|
use crate::bake::error::BakeError;
|
||||||
use std::collections::HashMap;
|
use std::collections::HashMap;
|
||||||
use topological_sort::TopologicalSort;
|
use workshop_schedule::{Dag, DagNode, EdgeKind, NodeColor};
|
||||||
|
|
||||||
/// Topologically sorts functions by explicit `@after` dependencies and implicit ordering
|
/// Thin wrapper so `Function` implements `DagNode`.
|
||||||
/// for consecutive `@pipeline` functions (each pipeline function implicitly depends on
|
#[derive(Clone)]
|
||||||
/// the previous one unless it has an explicit `@after`).
|
pub(crate) struct BakeNode {
|
||||||
pub fn sort_function(functions: Vec<Function>) -> Result<Vec<Function>, BakeError> {
|
name: String,
|
||||||
let function_mapped = functions
|
color: NodeColor,
|
||||||
|
fallible: bool,
|
||||||
|
/// Original function body + decorators for execution
|
||||||
|
func: Function,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl DagNode for BakeNode {
|
||||||
|
fn name(&self) -> &str {
|
||||||
|
&self.name
|
||||||
|
}
|
||||||
|
fn color(&self) -> NodeColor {
|
||||||
|
self.color.clone()
|
||||||
|
}
|
||||||
|
fn fallible(&self) -> bool {
|
||||||
|
self.fallible
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Convert a `Function` into a `BakeNode` by extracting scheduling metadata
|
||||||
|
/// from decorators.
|
||||||
|
fn function_to_node(func: &Function) -> BakeNode {
|
||||||
|
let color = if func
|
||||||
|
.decorators
|
||||||
|
.iter()
|
||||||
|
.any(|d| matches!(d, Decorator::Parallel(_)))
|
||||||
|
{
|
||||||
|
let group = func
|
||||||
|
.decorators
|
||||||
|
.iter()
|
||||||
|
.find_map(|d| {
|
||||||
|
if let Decorator::Parallel(Some(g)) = d {
|
||||||
|
Some(g.clone())
|
||||||
|
} else {
|
||||||
|
None
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.unwrap_or("*".to_string());
|
||||||
|
if group == "*" {
|
||||||
|
NodeColor::Wildcard
|
||||||
|
} else {
|
||||||
|
NodeColor::Named(group)
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
NodeColor::Sequential
|
||||||
|
};
|
||||||
|
|
||||||
|
let fallible = func
|
||||||
|
.decorators
|
||||||
|
.iter()
|
||||||
|
.any(|d| matches!(d, Decorator::Fallible));
|
||||||
|
|
||||||
|
BakeNode {
|
||||||
|
name: func.name.clone(),
|
||||||
|
color,
|
||||||
|
fallible,
|
||||||
|
func: func.clone(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Builds a DAG from functions using `@after` dependencies and implicit
|
||||||
|
/// ordering for consecutive `@pipeline` functions.
|
||||||
|
pub fn build_dag(
|
||||||
|
functions: Vec<Function>,
|
||||||
|
) -> Result<(Dag<BakeNode>, HashMap<String, Function>), BakeError> {
|
||||||
|
let function_mapped: HashMap<&str, &Function> = functions
|
||||||
.iter()
|
.iter()
|
||||||
.map(|f| (f.name.as_str(), f))
|
.map(|f| (f.name.as_str(), f))
|
||||||
.collect::<HashMap<&str, &Function>>();
|
.collect();
|
||||||
|
|
||||||
let mut ts = TopologicalSort::<String>::new();
|
let mut dag = Dag::new();
|
||||||
|
|
||||||
// Track last pipeline function for implicit @after
|
|
||||||
let mut last_pipeline: Option<String> = None;
|
let mut last_pipeline: Option<String> = None;
|
||||||
|
|
||||||
for function in &functions {
|
for function in &functions {
|
||||||
ts.insert(function.name.clone());
|
// Determine whether this function is a pipeline node
|
||||||
|
|
||||||
let is_pipeline = function
|
let is_pipeline = function
|
||||||
.decorators
|
.decorators
|
||||||
.iter()
|
.iter()
|
||||||
.any(|d| matches!(d, Decorator::Pipeline));
|
.any(|d| matches!(d, Decorator::Pipeline));
|
||||||
|
|
||||||
|
if !is_pipeline {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
let has_after = function
|
let has_after = function
|
||||||
.decorators
|
.decorators
|
||||||
.iter()
|
.iter()
|
||||||
.any(|d| matches!(d, Decorator::After(_)));
|
.any(|d| matches!(d, Decorator::After(_)));
|
||||||
|
|
||||||
// Add implicit @after for pipeline functions that don't have explicit @after
|
let node = function_to_node(function);
|
||||||
if is_pipeline {
|
dag.add_node(node).map_err(|e| {
|
||||||
if !has_after && let Some(ref last) = last_pipeline {
|
BakeError::IncompatibleDecorators {
|
||||||
ts.add_dependency(last, function.name.clone());
|
function: function.name.clone(),
|
||||||
|
reason: e.to_string(),
|
||||||
}
|
}
|
||||||
last_pipeline = Some(function.name.clone());
|
})?;
|
||||||
}
|
|
||||||
|
|
||||||
|
// Add implicit @after for consecutive @pipeline functions
|
||||||
|
if !has_after {
|
||||||
|
if let Some(ref last) = last_pipeline {
|
||||||
|
dag.add_edge(last, &function.name, EdgeKind::Implicit)
|
||||||
|
.map_err(|e| BakeError::IncompatibleDecorators {
|
||||||
|
function: function.name.clone(),
|
||||||
|
reason: e.to_string(),
|
||||||
|
})?;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
last_pipeline = Some(function.name.clone());
|
||||||
|
|
||||||
|
// Add explicit @after edges
|
||||||
for decorator in &function.decorators {
|
for decorator in &function.decorators {
|
||||||
if let Decorator::After(dependency) = decorator {
|
if let Decorator::After(dependency) = decorator {
|
||||||
if !function_mapped.contains_key(dependency.as_str()) {
|
if !function_mapped.contains_key(dependency.as_str()) {
|
||||||
@@ -50,466 +132,56 @@ pub fn sort_function(functions: Vec<Function>) -> Result<Vec<Function>, BakeErro
|
|||||||
dependency.clone(),
|
dependency.clone(),
|
||||||
));
|
));
|
||||||
}
|
}
|
||||||
ts.add_dependency(dependency, function.name.clone());
|
dag.add_edge(dependency, &function.name, EdgeKind::Explicit)
|
||||||
|
.map_err(|e| BakeError::IncompatibleDecorators {
|
||||||
|
function: function.name.clone(),
|
||||||
|
reason: e.to_string(),
|
||||||
|
})?;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
let mut functions_sorted = Vec::<Function>::new();
|
dag.freeze();
|
||||||
while let Some(name) = ts.pop() {
|
|
||||||
let function = function_mapped[name.as_str()];
|
|
||||||
functions_sorted.push(function.clone());
|
|
||||||
}
|
|
||||||
|
|
||||||
if !ts.is_empty() {
|
let owned_map: HashMap<String, Function> = functions
|
||||||
let remaining: Vec<String> = ts.collect();
|
.into_iter()
|
||||||
return Err(BakeError::CircularDependency(remaining));
|
.map(|f| (f.name.clone(), f))
|
||||||
}
|
.collect();
|
||||||
|
|
||||||
Ok(functions_sorted)
|
Ok((dag, owned_map))
|
||||||
}
|
}
|
||||||
|
|
||||||
#[cfg(test)]
|
/// Group ready parallel nodes by their declared group.
|
||||||
mod tests {
|
/// Returns `(star_nodes, named_groups)`.
|
||||||
use super::*;
|
///
|
||||||
|
/// Deprecated: use `Dag::ready_wildcards()` and `Dag::ready_groups()` instead.
|
||||||
fn create_function(name: &str, decorators: Vec<Decorator>, body: &str) -> Function {
|
pub fn partition_by_group<'a>(
|
||||||
Function {
|
ready: &[String],
|
||||||
name: name.to_string(),
|
dag: &Dag<BakeNode>,
|
||||||
decorators,
|
) -> (Vec<String>, HashMap<String, Vec<String>>) {
|
||||||
body: body.to_string(),
|
let mut star_nodes = Vec::new();
|
||||||
|
let mut named_groups: HashMap<String, Vec<String>> = HashMap::new();
|
||||||
|
for name in ready {
|
||||||
|
if dag.is_wildcard(name) {
|
||||||
|
star_nodes.push(name.clone());
|
||||||
|
} else if let Some(g) = dag.group_of(name) {
|
||||||
|
named_groups.entry(g).or_default().push(name.clone());
|
||||||
}
|
}
|
||||||
|
// Sequential nodes: not included (caller handles them)
|
||||||
}
|
}
|
||||||
|
(star_nodes, named_groups)
|
||||||
#[test]
|
}
|
||||||
fn test_empty_functions() {
|
|
||||||
let functions = vec![];
|
/// Pick a complete group to execute.
|
||||||
let result = sort_function(functions).unwrap();
|
///
|
||||||
assert!(result.is_empty());
|
/// Deprecated: use `Dag::pick_batch()` instead.
|
||||||
}
|
pub fn pick_batch(
|
||||||
|
_star_nodes: &[String],
|
||||||
#[test]
|
_groups: &HashMap<String, Vec<String>>,
|
||||||
fn test_single_function_no_dependencies() {
|
_group_totals: &HashMap<String, usize>,
|
||||||
let function = create_function("func1", vec![], "body1");
|
) -> Option<Vec<String>> {
|
||||||
let functions = vec![function.clone()];
|
// This function is replaced by Dag::pick_batch() which has all the
|
||||||
|
// group metadata internally. Keeping the signature for backward compat
|
||||||
let result = sort_function(functions).unwrap();
|
// but redirect callers to use Dag directly.
|
||||||
|
unimplemented!("use dag.pick_batch() instead")
|
||||||
assert_eq!(result.len(), 1);
|
|
||||||
assert_eq!(result[0].name, "func1");
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_multiple_functions_no_dependencies() {
|
|
||||||
let function1 = create_function("func1", vec![], "body1");
|
|
||||||
let function2 = create_function("func2", vec![], "body2");
|
|
||||||
let function3 = create_function("func3", vec![], "body3");
|
|
||||||
let functions = vec![function1.clone(), function2.clone(), function3.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
|
|
||||||
assert_eq!(result.len(), 3);
|
|
||||||
let names: Vec<String> = result.iter().map(|f| f.name.clone()).collect();
|
|
||||||
assert!(names.contains(&"func1".to_string()));
|
|
||||||
assert!(names.contains(&"func2".to_string()));
|
|
||||||
assert!(names.contains(&"func3".to_string()));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_simple_dependency() {
|
|
||||||
let function1 = create_function("func1", vec![], "body1");
|
|
||||||
let function2 = create_function(
|
|
||||||
"func2",
|
|
||||||
vec![Decorator::After("func1".to_string())],
|
|
||||||
"body2",
|
|
||||||
);
|
|
||||||
let functions = vec![function1.clone(), function2.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
|
|
||||||
assert_eq!(result.len(), 2);
|
|
||||||
assert_eq!(result[0].name, "func1");
|
|
||||||
assert_eq!(result[1].name, "func2");
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_multiple_dependencies() {
|
|
||||||
let function1 = create_function("func1", vec![], "body1");
|
|
||||||
let function2 = create_function(
|
|
||||||
"func2",
|
|
||||||
vec![Decorator::After("func1".to_string())],
|
|
||||||
"body2",
|
|
||||||
);
|
|
||||||
let function3 = create_function(
|
|
||||||
"func3",
|
|
||||||
vec![Decorator::After("func2".to_string())],
|
|
||||||
"body3",
|
|
||||||
);
|
|
||||||
let functions = vec![function1.clone(), function2.clone(), function3.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
|
|
||||||
assert_eq!(result.len(), 3);
|
|
||||||
assert_eq!(result[0].name, "func1");
|
|
||||||
assert_eq!(result[1].name, "func2");
|
|
||||||
assert_eq!(result[2].name, "func3");
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_complex_dependencies() {
|
|
||||||
let function1 = create_function("func1", vec![], "body1");
|
|
||||||
let function2 = create_function(
|
|
||||||
"func2",
|
|
||||||
vec![
|
|
||||||
Decorator::After("func1".to_string()),
|
|
||||||
Decorator::After("func3".to_string()),
|
|
||||||
],
|
|
||||||
"body2",
|
|
||||||
);
|
|
||||||
let function3 = create_function(
|
|
||||||
"func3",
|
|
||||||
vec![Decorator::After("func1".to_string())],
|
|
||||||
"body3",
|
|
||||||
);
|
|
||||||
let function4 = create_function("func4", vec![], "body4");
|
|
||||||
let functions = vec![
|
|
||||||
function1.clone(),
|
|
||||||
function2.clone(),
|
|
||||||
function3.clone(),
|
|
||||||
function4.clone(),
|
|
||||||
];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
|
|
||||||
assert_eq!(result.len(), 4);
|
|
||||||
|
|
||||||
let positions: std::collections::HashMap<String, usize> = result
|
|
||||||
.iter()
|
|
||||||
.enumerate()
|
|
||||||
.map(|(i, f)| (f.name.clone(), i))
|
|
||||||
.collect();
|
|
||||||
|
|
||||||
assert!(positions.get("func1").unwrap() < positions.get("func2").unwrap());
|
|
||||||
assert!(positions.get("func1").unwrap() < positions.get("func3").unwrap());
|
|
||||||
assert!(positions.get("func3").unwrap() < positions.get("func2").unwrap());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_mixed_decorators() {
|
|
||||||
let function1 = create_function(
|
|
||||||
"func1",
|
|
||||||
vec![Decorator::Pipeline, Decorator::Timeout(1000)],
|
|
||||||
"body1",
|
|
||||||
);
|
|
||||||
let function2 = create_function(
|
|
||||||
"func2",
|
|
||||||
vec![
|
|
||||||
Decorator::After("func1".to_string()),
|
|
||||||
Decorator::Retry(3, 100),
|
|
||||||
Decorator::Fallible,
|
|
||||||
],
|
|
||||||
"body2",
|
|
||||||
);
|
|
||||||
let functions = vec![function1.clone(), function2.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
|
|
||||||
assert_eq!(result.len(), 2);
|
|
||||||
assert_eq!(result[0].name, "func1");
|
|
||||||
assert_eq!(result[1].name, "func2");
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_function_with_multiple_after_decorators() {
|
|
||||||
let function1 = create_function("func1", vec![], "body1");
|
|
||||||
let function2 = create_function("func2", vec![], "body2");
|
|
||||||
let function3 = create_function(
|
|
||||||
"func3",
|
|
||||||
vec![
|
|
||||||
Decorator::After("func1".to_string()),
|
|
||||||
Decorator::After("func2".to_string()),
|
|
||||||
],
|
|
||||||
"body3",
|
|
||||||
);
|
|
||||||
let functions = vec![function1.clone(), function2.clone(), function3.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
|
|
||||||
assert_eq!(result.len(), 3);
|
|
||||||
|
|
||||||
let positions: std::collections::HashMap<String, usize> = result
|
|
||||||
.iter()
|
|
||||||
.enumerate()
|
|
||||||
.map(|(i, f)| (f.name.clone(), i))
|
|
||||||
.collect();
|
|
||||||
|
|
||||||
assert!(positions.get("func1").unwrap() < positions.get("func3").unwrap());
|
|
||||||
assert!(positions.get("func2").unwrap() < positions.get("func3").unwrap());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_ignores_non_after_decorators() {
|
|
||||||
let function1 = create_function(
|
|
||||||
"func1",
|
|
||||||
vec![
|
|
||||||
Decorator::Pipeline,
|
|
||||||
Decorator::If("condition".to_string()),
|
|
||||||
Decorator::Fallible,
|
|
||||||
],
|
|
||||||
"body1",
|
|
||||||
);
|
|
||||||
let function2 = create_function(
|
|
||||||
"func2",
|
|
||||||
vec![
|
|
||||||
Decorator::Loop(5),
|
|
||||||
Decorator::Timeout(1000),
|
|
||||||
Decorator::Retry(3, 100),
|
|
||||||
Decorator::Export,
|
|
||||||
],
|
|
||||||
"body2",
|
|
||||||
);
|
|
||||||
let functions = vec![function1.clone(), function2.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
|
|
||||||
assert_eq!(result.len(), 2);
|
|
||||||
let names: Vec<String> = result.iter().map(|f| f.name.clone()).collect();
|
|
||||||
assert!(names.contains(&"func1".to_string()));
|
|
||||||
assert!(names.contains(&"func2".to_string()));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_implicit_after_pipeline() {
|
|
||||||
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
|
|
||||||
let func2 = create_function("func2", vec![Decorator::Pipeline], "body2");
|
|
||||||
let func3 = create_function("func3", vec![Decorator::Pipeline], "body3");
|
|
||||||
let functions = vec![func1.clone(), func2.clone(), func3.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
|
|
||||||
assert_eq!(result.len(), 3);
|
|
||||||
assert_eq!(result[0].name, "func1");
|
|
||||||
assert_eq!(result[1].name, "func2");
|
|
||||||
assert_eq!(result[2].name, "func3");
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_first_pipeline_no_implicit_after() {
|
|
||||||
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
|
|
||||||
let functions = vec![func1.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
|
|
||||||
assert_eq!(result.len(), 1);
|
|
||||||
assert_eq!(result[0].name, "func1");
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_explicit_after_overrides_implicit() {
|
|
||||||
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
|
|
||||||
let func2 = create_function(
|
|
||||||
"func2",
|
|
||||||
vec![Decorator::Pipeline, Decorator::After("func1".to_string())],
|
|
||||||
"body2",
|
|
||||||
);
|
|
||||||
let func3 = create_function("func3", vec![Decorator::Pipeline], "body3");
|
|
||||||
let functions = vec![func1.clone(), func2.clone(), func3.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
|
|
||||||
assert_eq!(result.len(), 3);
|
|
||||||
let positions: std::collections::HashMap<String, usize> = result
|
|
||||||
.iter()
|
|
||||||
.enumerate()
|
|
||||||
.map(|(i, f)| (f.name.clone(), i))
|
|
||||||
.collect();
|
|
||||||
|
|
||||||
assert!(positions.get("func1").unwrap() < positions.get("func2").unwrap());
|
|
||||||
assert!(positions.get("func2").unwrap() < positions.get("func3").unwrap());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_mixed_pipeline_and_non_pipeline() {
|
|
||||||
let func1 = create_function("func1", vec![], "body1");
|
|
||||||
let func2 = create_function("func2", vec![Decorator::Pipeline], "body2");
|
|
||||||
let func3 = create_function("func3", vec![], "body3");
|
|
||||||
let func4 = create_function("func4", vec![Decorator::Pipeline], "body4");
|
|
||||||
let functions = vec![func1.clone(), func2.clone(), func3.clone(), func4.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
|
|
||||||
assert_eq!(result.len(), 4);
|
|
||||||
let positions: std::collections::HashMap<String, usize> = result
|
|
||||||
.iter()
|
|
||||||
.enumerate()
|
|
||||||
.map(|(i, f)| (f.name.clone(), i))
|
|
||||||
.collect();
|
|
||||||
|
|
||||||
assert!(positions.get("func2").unwrap() < positions.get("func4").unwrap());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_single_pipeline_works() {
|
|
||||||
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
|
|
||||||
let functions = vec![func1.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
assert_eq!(result.len(), 1);
|
|
||||||
assert_eq!(result[0].name, "func1");
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_two_pipelines_implicit_chain() {
|
|
||||||
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
|
|
||||||
let func2 = create_function("func2", vec![Decorator::Pipeline], "body2");
|
|
||||||
let functions = vec![func1.clone(), func2.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
assert_eq!(result.len(), 2);
|
|
||||||
let positions: std::collections::HashMap<String, usize> = result
|
|
||||||
.iter()
|
|
||||||
.enumerate()
|
|
||||||
.map(|(i, f)| (f.name.clone(), i))
|
|
||||||
.collect();
|
|
||||||
assert!(positions.get("func1").unwrap() < positions.get("func2").unwrap());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_pipeline_explicit_after_skips_implicit() {
|
|
||||||
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
|
|
||||||
let func2 = create_function("func2", vec![Decorator::Pipeline], "body2");
|
|
||||||
let func3 = create_function(
|
|
||||||
"func3",
|
|
||||||
vec![Decorator::Pipeline, Decorator::After("func1".to_string())],
|
|
||||||
"body3",
|
|
||||||
);
|
|
||||||
let functions = vec![func1.clone(), func2.clone(), func3.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
assert_eq!(result.len(), 3);
|
|
||||||
let positions: std::collections::HashMap<String, usize> = result
|
|
||||||
.iter()
|
|
||||||
.enumerate()
|
|
||||||
.map(|(i, f)| (f.name.clone(), i))
|
|
||||||
.collect();
|
|
||||||
assert!(positions.get("func1").unwrap() < positions.get("func3").unwrap());
|
|
||||||
// TODO: Our design spec indicates order-based sorting, however toposort crate does not guarantee order of independent nodes. We should consider whether we want to enforce order-based sorting for pipelines, or if we are okay with any valid topological order.
|
|
||||||
// assert!(positions.get("func2").unwrap() < positions.get("func3").unwrap());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_circular_dependency_detected() {
|
|
||||||
let func1 = create_function(
|
|
||||||
"func1",
|
|
||||||
vec![Decorator::After("func2".to_string())],
|
|
||||||
"body1",
|
|
||||||
);
|
|
||||||
let func2 = create_function(
|
|
||||||
"func2",
|
|
||||||
vec![Decorator::After("func1".to_string())],
|
|
||||||
"body2",
|
|
||||||
);
|
|
||||||
let functions = vec![func1.clone(), func2.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions);
|
|
||||||
assert!(result.is_err());
|
|
||||||
assert!(matches!(
|
|
||||||
result.unwrap_err(),
|
|
||||||
BakeError::CircularDependency(_)
|
|
||||||
));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_unknown_dependency_error() {
|
|
||||||
let func1 = create_function(
|
|
||||||
"func1",
|
|
||||||
vec![Decorator::After("nonexistent".to_string())],
|
|
||||||
"body1",
|
|
||||||
);
|
|
||||||
let functions = vec![func1.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions);
|
|
||||||
assert!(result.is_err());
|
|
||||||
assert!(matches!(
|
|
||||||
result.unwrap_err(),
|
|
||||||
BakeError::UnknownDependency(_, _)
|
|
||||||
));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_pipeline_with_multiple_decorators() {
|
|
||||||
let func1 = create_function(
|
|
||||||
"func1",
|
|
||||||
vec![
|
|
||||||
Decorator::Pipeline,
|
|
||||||
Decorator::Timeout(60),
|
|
||||||
Decorator::Fallible,
|
|
||||||
],
|
|
||||||
"body1",
|
|
||||||
);
|
|
||||||
let func2 = create_function(
|
|
||||||
"func2",
|
|
||||||
vec![
|
|
||||||
Decorator::Pipeline,
|
|
||||||
Decorator::Retry(3, 5),
|
|
||||||
Decorator::After("func1".to_string()),
|
|
||||||
],
|
|
||||||
"body2",
|
|
||||||
);
|
|
||||||
let functions = vec![func1.clone(), func2.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
assert_eq!(result.len(), 2);
|
|
||||||
let positions: std::collections::HashMap<String, usize> = result
|
|
||||||
.iter()
|
|
||||||
.enumerate()
|
|
||||||
.map(|(i, f)| (f.name.clone(), i))
|
|
||||||
.collect();
|
|
||||||
assert!(positions.get("func1").unwrap() < positions.get("func2").unwrap());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_long_pipeline_chain() {
|
|
||||||
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
|
|
||||||
let func2 = create_function("func2", vec![Decorator::Pipeline], "body2");
|
|
||||||
let func3 = create_function("func3", vec![Decorator::Pipeline], "body3");
|
|
||||||
let func4 = create_function("func4", vec![Decorator::Pipeline], "body4");
|
|
||||||
let func5 = create_function("func5", vec![Decorator::Pipeline], "body5");
|
|
||||||
let functions = vec![
|
|
||||||
func1.clone(),
|
|
||||||
func2.clone(),
|
|
||||||
func3.clone(),
|
|
||||||
func4.clone(),
|
|
||||||
func5.clone(),
|
|
||||||
];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
assert_eq!(result.len(), 5);
|
|
||||||
let positions: std::collections::HashMap<String, usize> = result
|
|
||||||
.iter()
|
|
||||||
.enumerate()
|
|
||||||
.map(|(i, f)| (f.name.clone(), i))
|
|
||||||
.collect();
|
|
||||||
assert!(positions.get("func1").unwrap() < positions.get("func2").unwrap());
|
|
||||||
assert!(positions.get("func2").unwrap() < positions.get("func3").unwrap());
|
|
||||||
assert!(positions.get("func3").unwrap() < positions.get("func4").unwrap());
|
|
||||||
assert!(positions.get("func4").unwrap() < positions.get("func5").unwrap());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_non_pipeline_between_pipelines() {
|
|
||||||
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
|
|
||||||
let func2 = create_function("func2", vec![], "body2");
|
|
||||||
let func3 = create_function("func3", vec![Decorator::Pipeline], "body3");
|
|
||||||
let functions = vec![func1.clone(), func2.clone(), func3.clone()];
|
|
||||||
|
|
||||||
let result = sort_function(functions).unwrap();
|
|
||||||
assert_eq!(result.len(), 3);
|
|
||||||
let positions: std::collections::HashMap<String, usize> = result
|
|
||||||
.iter()
|
|
||||||
.enumerate()
|
|
||||||
.map(|(i, f)| (f.name.clone(), i))
|
|
||||||
.collect();
|
|
||||||
assert!(positions.get("func1").unwrap() < positions.get("func3").unwrap());
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,31 @@
|
|||||||
|
use crate::bake::builder::{build_script, RenderMode};
|
||||||
|
use crate::bake::error::BakeError;
|
||||||
|
use crate::bake::parser::Function;
|
||||||
|
use std::path::Path;
|
||||||
|
use workshop_engine::{Engine, EventSender, ExecutionContext};
|
||||||
|
|
||||||
|
/// Executes `script_content` as a single function in trivial mode.
|
||||||
|
///
|
||||||
|
/// Two entry paths lead here:
|
||||||
|
/// - **Auto-detection** (`!has_pipeline`): bake.sh has no `@pipeline` functions,
|
||||||
|
/// the entire content is treated as one trivial function.
|
||||||
|
/// - **Explicit PIP** (future): `trivial` parameter forces trivial mode even
|
||||||
|
/// when `@pipeline` functions exist, typically with `RenderMode::Off`.
|
||||||
|
pub async fn run_trivial(
|
||||||
|
script_content: &str,
|
||||||
|
bake_base_path: &Path,
|
||||||
|
workspace: &Path,
|
||||||
|
engine: &Engine,
|
||||||
|
ctx: &mut ExecutionContext,
|
||||||
|
event_tx: &EventSender,
|
||||||
|
render_mode: RenderMode,
|
||||||
|
) -> Result<(), BakeError> {
|
||||||
|
let function = Function {
|
||||||
|
name: String::new(),
|
||||||
|
decorators: vec![],
|
||||||
|
body: script_content.to_string(),
|
||||||
|
};
|
||||||
|
let script = build_script(&function, bake_base_path, workspace, render_mode, ctx.debug_flags)?;
|
||||||
|
engine.execute_script(&script, ctx, event_tx).await?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
@@ -0,0 +1,61 @@
|
|||||||
|
use crate::bake::decorator::Decorator;
|
||||||
|
use crate::bake::parser::Function;
|
||||||
|
use std::collections::HashMap;
|
||||||
|
|
||||||
|
/// Execution mode for a pipeline function.
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||||
|
pub enum FuncMode {
|
||||||
|
Normal,
|
||||||
|
Daemon,
|
||||||
|
Async,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Determine the execution mode from decorators.
|
||||||
|
/// `@daemon`, `@async`, `@pipe` are mutually exclusive modes.
|
||||||
|
pub fn resolve_mode(func: &Function) -> FuncMode {
|
||||||
|
for d in &func.decorators {
|
||||||
|
match d {
|
||||||
|
Decorator::Daemon => return FuncMode::Daemon,
|
||||||
|
Decorator::Async => return FuncMode::Async,
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
FuncMode::Normal
|
||||||
|
}
|
||||||
|
|
||||||
|
/// True if the function is decorated with `@parallel`.
|
||||||
|
pub fn is_parallel(func: &Function) -> bool {
|
||||||
|
func.decorators
|
||||||
|
.iter()
|
||||||
|
.any(|d| matches!(d, Decorator::Parallel(_)))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// True if the function is decorated with `@fallible`.
|
||||||
|
pub fn is_fallible(func: &Function) -> bool {
|
||||||
|
func.decorators
|
||||||
|
.iter()
|
||||||
|
.any(|d| matches!(d, Decorator::Fallible))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Resolve stdin for a `@pipe`-decorated function from the stdout cache.
|
||||||
|
///
|
||||||
|
/// Rule 1: self-references are skipped (handled by loop combinator).
|
||||||
|
/// Rule 3: sources present in the cache are concatenated.
|
||||||
|
/// Returns `None` if any source is missing (caller decides based on `@fallible`).
|
||||||
|
pub fn resolve_pipe_stdin(
|
||||||
|
func: &Function,
|
||||||
|
cache: &HashMap<String, String>,
|
||||||
|
) -> Option<Vec<u8>> {
|
||||||
|
let sources = func.decorators.iter().find_map(|d| match d {
|
||||||
|
Decorator::Pipe(names) => Some(names.as_slice()),
|
||||||
|
_ => None,
|
||||||
|
})?;
|
||||||
|
let mut combined = String::new();
|
||||||
|
for source in sources {
|
||||||
|
if *source == func.name {
|
||||||
|
continue; // Rule 1: skip self
|
||||||
|
}
|
||||||
|
combined.push_str(cache.get(source)?); // Rule 3
|
||||||
|
}
|
||||||
|
Some(combined.into_bytes())
|
||||||
|
}
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
use crate::cli::{BareCommands, Cli};
|
||||||
|
use crate::error::CliError;
|
||||||
|
use crate::types::resource::ResourceRegistry;
|
||||||
|
use crate::{bake::bake, finalize::finalize, prebake::prebake};
|
||||||
|
use workshop_engine::{EventSender, ExecutionContext};
|
||||||
|
|
||||||
|
pub async fn run_bare(
|
||||||
|
cli: &Cli,
|
||||||
|
command: &BareCommands,
|
||||||
|
ctx: &mut ExecutionContext,
|
||||||
|
event_tx: EventSender,
|
||||||
|
pipeline: &str,
|
||||||
|
build_id: &str,
|
||||||
|
registry: &ResourceRegistry,
|
||||||
|
) -> Result<(), CliError> {
|
||||||
|
let (stage, config) = match command {
|
||||||
|
BareCommands::Prebake { .. } => ("prebake", &cli.prebake),
|
||||||
|
BareCommands::Bake { .. } => ("bake", &cli.bake),
|
||||||
|
BareCommands::Finalize { .. } => {
|
||||||
|
("finalize", &cli.finalize)
|
||||||
|
}
|
||||||
|
};
|
||||||
|
log::debug!("Running bare command {} with config {}", stage, config.display());
|
||||||
|
|
||||||
|
let prebake_path = &cli.prebake;
|
||||||
|
let bake_base = &cli.bake_base;
|
||||||
|
|
||||||
|
ctx.task_id = format!("{}-{}-{}", pipeline, build_id, stage);
|
||||||
|
ctx.pipeline_name = pipeline.to_string();
|
||||||
|
ctx.standalone = true;
|
||||||
|
|
||||||
|
match command {
|
||||||
|
BareCommands::Prebake { .. } => {
|
||||||
|
prebake(&config, cli, None, ctx, event_tx, None).await?;
|
||||||
|
}
|
||||||
|
BareCommands::Bake { .. } => {
|
||||||
|
bake(&config, &bake_base, &prebake_path, cli, ctx, event_tx, None).await?;
|
||||||
|
}
|
||||||
|
BareCommands::Finalize { .. } => {
|
||||||
|
finalize(&config, cli, ctx, event_tx, registry).await?;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
@@ -1,193 +0,0 @@
|
|||||||
use crate::cli::Cli;
|
|
||||||
// use serde::Serialize;
|
|
||||||
use serde_json::Value;
|
|
||||||
// use std::path::PathBuf;
|
|
||||||
// use std::sync::Arc;
|
|
||||||
use tokio::io::{AsyncBufReadExt, BufReader};
|
|
||||||
use tokio::net::UnixStream;
|
|
||||||
// use tokio::process::{Child, Command};
|
|
||||||
// use tokio::signal;
|
|
||||||
// use tokio::sync::RwLock;
|
|
||||||
// use tokio::time::{Duration, interval};
|
|
||||||
|
|
||||||
// #[derive(Debug, Clone, Serialize)]
|
|
||||||
// struct Heartbeat {
|
|
||||||
// pipeline_id: String,
|
|
||||||
// build_id: String,
|
|
||||||
// stage: String,
|
|
||||||
// timestamp: u64,
|
|
||||||
// status: String,
|
|
||||||
// subtasks: Vec<String>,
|
|
||||||
// }
|
|
||||||
|
|
||||||
pub async fn daemon(_cli: &Cli) {
|
|
||||||
todo!("Daemon should be refactored");
|
|
||||||
// let socket = PathBuf::from(settings.get_string("socket").unwrap());
|
|
||||||
// let agentsocket = settings.get_string("agentsocket").unwrap();
|
|
||||||
// let _ = std::fs::remove_file(&socket);
|
|
||||||
|
|
||||||
// let heartbeat = Arc::new(RwLock::new(Heartbeat {
|
|
||||||
// pipeline_id: cli.pipeline.clone(),
|
|
||||||
// build_id: cli.build_id.clone(),
|
|
||||||
// stage: String::new(),
|
|
||||||
// timestamp: 0,
|
|
||||||
// status: String::new(),
|
|
||||||
// subtasks: Vec::new(),
|
|
||||||
// }));
|
|
||||||
|
|
||||||
// // let heartbeat_clone = heartbeat.clone();
|
|
||||||
// let heartbeat_task = {
|
|
||||||
// let hb = heartbeat.clone();
|
|
||||||
// tokio::spawn(async move {
|
|
||||||
// let mut tick = interval(Duration::from_secs(1));
|
|
||||||
// loop {
|
|
||||||
// tick.tick().await;
|
|
||||||
// let mut hb = hb.write().await;
|
|
||||||
// hb.timestamp = std::time::SystemTime::now()
|
|
||||||
// .duration_since(std::time::UNIX_EPOCH)
|
|
||||||
// .unwrap()
|
|
||||||
// .as_millis() as u64;
|
|
||||||
|
|
||||||
// // 只输出到stdout(模拟未来发送给Agent)
|
|
||||||
// println!("[HEARTBEAT] {}", serde_json::to_string(&*hb).unwrap());
|
|
||||||
// }
|
|
||||||
// })
|
|
||||||
// };
|
|
||||||
|
|
||||||
// let socket_task = {
|
|
||||||
// let socket = socket.clone();
|
|
||||||
// tokio::spawn(async move {
|
|
||||||
// log::info!("Creating Unix socket at {}", socket.display());
|
|
||||||
// let listener =
|
|
||||||
// UnixListener::bind(&socket).expect("Failed to bind Unix socket");
|
|
||||||
|
|
||||||
// loop {
|
|
||||||
// match listener.accept().await {
|
|
||||||
// Ok((stream, _)) => {
|
|
||||||
// tokio::spawn(handle_connection(stream));
|
|
||||||
// }
|
|
||||||
// Err(e) => {
|
|
||||||
// log::error!("Accept error: {}", e);
|
|
||||||
// break;
|
|
||||||
// }
|
|
||||||
// }
|
|
||||||
// }
|
|
||||||
// })
|
|
||||||
// };
|
|
||||||
|
|
||||||
// let mut tasks: Vec<TaskSpec> = Vec::new();
|
|
||||||
|
|
||||||
// let execution_task = {
|
|
||||||
// let hb = heartbeat.clone();
|
|
||||||
// tokio::spawn(async move {
|
|
||||||
// let mut hb = hb.write().await;
|
|
||||||
// let mut children: Vec<(String, Child)> = vec![];
|
|
||||||
|
|
||||||
// let executable = std::env::current_exe().unwrap();
|
|
||||||
|
|
||||||
// for (index, task) in tasks.iter().enumerate() {
|
|
||||||
// let mut command = task.command(&executable);
|
|
||||||
// command.stdout(std::process::Stdio::piped());
|
|
||||||
// command.stderr(std::process::Stdio::piped());
|
|
||||||
// let child = command
|
|
||||||
// .spawn()
|
|
||||||
// .expect(format!("Failed to spawn child {}", index).as_str());
|
|
||||||
|
|
||||||
// let pid = child.id().unwrap_or(0).to_string();
|
|
||||||
// children.push((pid.clone(), child));
|
|
||||||
|
|
||||||
// hb.subtasks.push(pid.clone());
|
|
||||||
// }
|
|
||||||
|
|
||||||
// for (pid, mut child) in children {
|
|
||||||
// let status = child.wait().await.expect("Failed to wait child");
|
|
||||||
// log::info!("Subtask {} exited with: {}", pid, status);
|
|
||||||
|
|
||||||
// hb.subtasks.retain(|p| p != &pid);
|
|
||||||
// }
|
|
||||||
|
|
||||||
// hb.stage = "Success".to_string();
|
|
||||||
// })
|
|
||||||
// };
|
|
||||||
|
|
||||||
// tokio::select! {
|
|
||||||
// _ = heartbeat_task => {
|
|
||||||
// log::error!("Heartbeat task died");
|
|
||||||
// }
|
|
||||||
// _ = execution_task => {
|
|
||||||
// log::info!("All subtasks completed");
|
|
||||||
// }
|
|
||||||
// _ = socket_task => {
|
|
||||||
// log::error!("Socket task died");
|
|
||||||
// }
|
|
||||||
// _ = signal::ctrl_c() => {
|
|
||||||
// log::info!("Shutting down...");
|
|
||||||
// }
|
|
||||||
// }
|
|
||||||
// for pid in heartbeat.read().await.subtasks.clone() {
|
|
||||||
// let _ = Command::new("kill").arg("-TERM").arg(&pid).status().await;
|
|
||||||
// }
|
|
||||||
|
|
||||||
// let _ = std::fs::remove_file(&socket);
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn _handle_connection(stream: UnixStream) {
|
|
||||||
let peer_addr = match stream.peer_addr() {
|
|
||||||
Ok(addr) => format!("{:?}", addr),
|
|
||||||
Err(_) => "unknown".to_string(),
|
|
||||||
};
|
|
||||||
|
|
||||||
log::debug!("New connection from: {}", peer_addr);
|
|
||||||
|
|
||||||
let mut reader = BufReader::new(stream);
|
|
||||||
let mut line = String::new();
|
|
||||||
|
|
||||||
loop {
|
|
||||||
line.clear();
|
|
||||||
match reader.read_line(&mut line).await {
|
|
||||||
Ok(0) => {
|
|
||||||
log::debug!("Connection closed by peer: {}", peer_addr);
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
Ok(_) => {
|
|
||||||
// 简单打印原始JSON(预留字段解析空间)
|
|
||||||
match serde_json::from_str::<Value>(&line) {
|
|
||||||
Ok(json) => {
|
|
||||||
// 只提取关键字段打印(字段可能变化)
|
|
||||||
let msgtype = json
|
|
||||||
.get("msgtype")
|
|
||||||
.and_then(|v| v.as_str())
|
|
||||||
.unwrap_or("unknown");
|
|
||||||
let name = json
|
|
||||||
.get("name")
|
|
||||||
.and_then(|v| v.as_str())
|
|
||||||
.unwrap_or("unnamed");
|
|
||||||
|
|
||||||
log::info!("[{}] Message from '{}': {}", msgtype, name, line.trim());
|
|
||||||
|
|
||||||
// TODO: 字段稳定后,可扩展为结构化打印
|
|
||||||
// match msgtype {
|
|
||||||
// "ResourceUsage" => print_resource(&json),
|
|
||||||
// "Log" => print_log(&json),
|
|
||||||
// _ => log::debug!("Unknown type: {}", msgtype),
|
|
||||||
// }
|
|
||||||
}
|
|
||||||
Err(e) => {
|
|
||||||
log::warn!(
|
|
||||||
"Invalid JSON from {}: {} - Error: {}",
|
|
||||||
peer_addr,
|
|
||||||
line.trim(),
|
|
||||||
e
|
|
||||||
);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Err(e) => {
|
|
||||||
log::error!("Failed to read from {}: {}", peer_addr, e);
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
log::debug!("Connection handler for {} exiting", peer_addr);
|
|
||||||
}
|
|
||||||
|
|||||||
Vendored
@@ -0,0 +1,30 @@
|
|||||||
|
use thiserror::Error;
|
||||||
|
use workshop_engine::HasExitCode;
|
||||||
|
|
||||||
|
#[derive(Error, Debug)]
|
||||||
|
pub enum CliError {
|
||||||
|
#[error("{0}")]
|
||||||
|
Prebake(#[from] crate::prebake::error::PrebakeError),
|
||||||
|
#[error("{0}")]
|
||||||
|
Bake(#[from] crate::bake::error::BakeError),
|
||||||
|
#[error("{0}")]
|
||||||
|
Finalize(#[from] crate::finalize::FinalizeError),
|
||||||
|
#[error("{0}")]
|
||||||
|
General(anyhow::Error),
|
||||||
|
}
|
||||||
|
|
||||||
|
impl From<anyhow::Error> for CliError {
|
||||||
|
fn from(e: anyhow::Error) -> Self {
|
||||||
|
CliError::General(e)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
impl CliError {
|
||||||
|
pub fn exit_code(&self) -> i32 {
|
||||||
|
match self {
|
||||||
|
CliError::Prebake(e) => e.exit_code(),
|
||||||
|
CliError::Bake(e) => e.exit_code(),
|
||||||
|
CliError::Finalize(e) => e.exit_code(),
|
||||||
|
CliError::General(_) => 1,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -2,18 +2,18 @@ pub mod config;
|
|||||||
pub mod constant;
|
pub mod constant;
|
||||||
pub mod error;
|
pub mod error;
|
||||||
pub mod event;
|
pub mod event;
|
||||||
|
pub mod cleanup;
|
||||||
|
pub mod artifact;
|
||||||
pub mod plugin;
|
pub mod plugin;
|
||||||
pub mod stage;
|
pub mod stage;
|
||||||
pub mod template;
|
pub mod template;
|
||||||
pub mod types;
|
pub mod types;
|
||||||
|
|
||||||
use crate::cli::Cli;
|
use crate::cli::Cli;
|
||||||
use crate::finalize::constant::FINALIZE_STANDALONE_PLUGIN_PATH;
|
use crate::types::resource::ResourceRegistry;
|
||||||
use crate::finalize::plugin::fetch::{self, FetchArgument};
|
|
||||||
pub use config::*;
|
pub use config::*;
|
||||||
pub use error::FinalizeError;
|
pub use error::FinalizeError;
|
||||||
use std::path::Path;
|
use std::path::Path;
|
||||||
use std::path::PathBuf;
|
|
||||||
use workshop_engine::EventSender;
|
use workshop_engine::EventSender;
|
||||||
|
|
||||||
pub fn parse(path: &Path) -> Result<FinalizeConfig, FinalizeError> {
|
pub fn parse(path: &Path) -> Result<FinalizeConfig, FinalizeError> {
|
||||||
@@ -27,6 +27,7 @@ pub async fn finalize(
|
|||||||
_cli: &Cli,
|
_cli: &Cli,
|
||||||
ctx: &mut workshop_engine::ExecutionContext,
|
ctx: &mut workshop_engine::ExecutionContext,
|
||||||
event_tx: EventSender,
|
event_tx: EventSender,
|
||||||
|
registry: &ResourceRegistry,
|
||||||
) -> Result<(), FinalizeError> {
|
) -> Result<(), FinalizeError> {
|
||||||
let mut finalize = parse(finalize_path)?;
|
let mut finalize = parse(finalize_path)?;
|
||||||
let validate_result = finalize.validate();
|
let validate_result = finalize.validate();
|
||||||
@@ -36,33 +37,10 @@ pub async fn finalize(
|
|||||||
}
|
}
|
||||||
return Err(FinalizeError::ValidateError(errors));
|
return Err(FinalizeError::ValidateError(errors));
|
||||||
};
|
};
|
||||||
if ctx.standalone {
|
|
||||||
// In standalone mode, finalize should download plugin by itself
|
|
||||||
log::debug!("Downloading {} plugins", finalize.plugin.len());
|
|
||||||
let plugin_path = PathBuf::from(FINALIZE_STANDALONE_PLUGIN_PATH);
|
|
||||||
// if plugin_path.exists() {
|
|
||||||
// std::fs::remove_dir_all(&plugin_path).map_err(FinalizeError::IoError)?;
|
|
||||||
// }
|
|
||||||
for (name, data) in finalize.plugin.iter_mut() {
|
|
||||||
log::info!("Downloading plugin {}", name);
|
|
||||||
let (source, config) = data.iter_mut().next().unwrap_or_else(|| {
|
|
||||||
panic!(
|
|
||||||
"Internal Error: Plugin {} has no source configured, this should have been caught by validation!",
|
|
||||||
name
|
|
||||||
)
|
|
||||||
});
|
|
||||||
let fetch_argument = FetchArgument {
|
|
||||||
checksum: config.checksum.clone(),
|
|
||||||
shallow: config.shallow,
|
|
||||||
};
|
|
||||||
fetch::fetch_plugin(source, name, &plugin_path, fetch_argument).await?;
|
|
||||||
config.runtime.fspath = Some(plugin_path.join(name));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Register Plugins
|
// Register Plugins (paths resolved from registry)
|
||||||
log::info!("Registering {} plugins", finalize.plugin.len());
|
log::info!("Registering {} plugins", finalize.plugin.len());
|
||||||
let registered_plugins = plugin::register(finalize.plugin, None)?;
|
let registered_plugins = plugin::register(finalize.plugin, None, &mut finalize.notification, registry)?;
|
||||||
|
|
||||||
// Early hook
|
// Early hook
|
||||||
let earlyhook_result = stage::hook::hook(
|
let earlyhook_result = stage::hook::hook(
|
||||||
@@ -76,9 +54,17 @@ pub async fn finalize(
|
|||||||
// Determine if we should continue with artifacts/latehook
|
// Determine if we should continue with artifacts/latehook
|
||||||
// If prior stages failed, we still send notification but skip artifacts
|
// If prior stages failed, we still send notification but skip artifacts
|
||||||
let _prior_failed = earlyhook_result.is_err();
|
let _prior_failed = earlyhook_result.is_err();
|
||||||
|
// Artifact collection and publishing
|
||||||
// Prepare artifacts
|
if !_prior_failed {
|
||||||
// TODO: artifact stage implementation
|
log::info!("Processing {} artifacts", finalize.artifact.len());
|
||||||
|
artifact::collect_and_publish(
|
||||||
|
&finalize.artifact,
|
||||||
|
®istered_plugins,
|
||||||
|
ctx,
|
||||||
|
event_tx.clone(),
|
||||||
|
)
|
||||||
|
.await?;
|
||||||
|
}
|
||||||
|
|
||||||
// Late hook
|
// Late hook
|
||||||
stage::hook::hook(
|
stage::hook::hook(
|
||||||
|
|||||||
@@ -1,56 +0,0 @@
|
|||||||
# workshop-baker/src/finalize
|
|
||||||
|
|
||||||
**Generated:** 2026-04-22
|
|
||||||
**Commit:** 7b3b71a
|
|
||||||
|
|
||||||
Artifact packaging, distribution, deployment, and notifications.
|
|
||||||
|
|
||||||
## STRUCTURE
|
|
||||||
|
|
||||||
```
|
|
||||||
finalize/
|
|
||||||
├── config.rs # FinalizeConfig YAML (artifacts, deploy, notify)
|
|
||||||
├── stage.rs # FinalizeStage enum
|
|
||||||
├── stage/hook.rs # Post-build hooks
|
|
||||||
├── event.rs # Event propagation
|
|
||||||
├── template.rs # Variable substitution {{VAR}}
|
|
||||||
├── plugin.rs # Plugin trait + registry
|
|
||||||
├── plugin/
|
|
||||||
│ ├── metadata.rs # Plugin manifest parsing
|
|
||||||
│ ├── dylib.rs # Dynamic library plugins
|
|
||||||
│ ├── rhai.rs # Rhai scripting (todo!())
|
|
||||||
│ ├── shell.rs # Shell command plugins
|
|
||||||
│ └── internal/ # Built-in plugins
|
|
||||||
│ ├── mail/ # Email notifications (SMTP)
|
|
||||||
│ ├── webhook.rs # HTTP webhooks
|
|
||||||
│ ├── satori.rs # Satori integration
|
|
||||||
│ └── insitenotify.rs # In-site notifications
|
|
||||||
│ └── fetch/ # Artifact fetching
|
|
||||||
│ ├── git.rs # Git repository fetch
|
|
||||||
│ ├── http.rs # HTTP download
|
|
||||||
│ ├── extract.rs # Archive extraction
|
|
||||||
│ └── checksum.rs# SHA256 verification (SHA1 deprecated)
|
|
||||||
```
|
|
||||||
|
|
||||||
## WHERE TO LOOK
|
|
||||||
|
|
||||||
| Task | Location |
|
|
||||||
|------|----------|
|
|
||||||
| Config loading | `config.rs` - FinalizeConfig |
|
|
||||||
| Email notify | `plugin/internal/mail/` - SMTP templates |
|
|
||||||
| Webhooks | `plugin/internal/webhook.rs` |
|
|
||||||
| Artifact fetch | `plugin/fetch/` - git/http/extract |
|
|
||||||
| Checksums | `plugin/fetch/checksum.rs:13` - SHA1 deprecated |
|
|
||||||
|
|
||||||
## CONVENTIONS
|
|
||||||
|
|
||||||
- **Plugin System**: Dynamic + internal plugins via Plugin trait
|
|
||||||
- **Template Syntax**: `{{VARIABLE}}` handlebars-style
|
|
||||||
- **Stages**: Hook → Fetch → Build → Publish → Notify → Cleanup
|
|
||||||
- **Fetch**: Supports git, http, with extraction and checksum verify
|
|
||||||
|
|
||||||
## ANTI-PATTERNS
|
|
||||||
|
|
||||||
1. **SHA1 deprecated** — `checksum.rs:13` warns SHA1 is deprecated
|
|
||||||
2. **Unimplemented** — `rhai.rs:17` - "todo!()" for Rhai scripting
|
|
||||||
3. **TODO markers** — Multiple TODOs in config.rs (lines 305-306)
|
|
||||||
@@ -0,0 +1,139 @@
|
|||||||
|
//! Artifact collection and publishing.
|
||||||
|
//!
|
||||||
|
//! Collects build artifacts (via glob patterns), optionally compresses them,
|
||||||
|
//! and dispatches each artifact through its configured publish plugins.
|
||||||
|
//! The publish plugin receives the artifact path as a system-injected
|
||||||
|
//! `artifact` field in its argument.
|
||||||
|
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
use crate::finalize::config::ArtifactDef;
|
||||||
|
use crate::finalize::error::FinalizeError;
|
||||||
|
use crate::finalize::plugin::PluginMap;
|
||||||
|
use crate::finalize::types::compression::CompressionMethod;
|
||||||
|
use workshop_engine::{EventSender, ExecutionContext};
|
||||||
|
|
||||||
|
/// Collect artifacts and execute their publish plugins.
|
||||||
|
///
|
||||||
|
/// For each artifact:
|
||||||
|
/// 1. Resolve source files via glob
|
||||||
|
/// 2. Optionally compress (zstd, gzip, dir)
|
||||||
|
/// 3. Inject `artifact` key into each publish plugin's config
|
||||||
|
/// 4. Call the plugin
|
||||||
|
pub async fn collect_and_publish(
|
||||||
|
artifacts: &[ArtifactDef],
|
||||||
|
plugins: &PluginMap,
|
||||||
|
ctx: &ExecutionContext,
|
||||||
|
event_tx: EventSender,
|
||||||
|
) -> Result<(), FinalizeError> {
|
||||||
|
if artifacts.is_empty() {
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
for artifact in artifacts {
|
||||||
|
let artifact_path = process_artifact(artifact).await?;
|
||||||
|
|
||||||
|
if let Some(ref path) = artifact_path {
|
||||||
|
log::info!("Artifact processed: {} -> {}", artifact.path, path.display());
|
||||||
|
}
|
||||||
|
|
||||||
|
for (plugin_name, publish_config) in &artifact.publish {
|
||||||
|
let plugin = plugins.get(plugin_name).ok_or_else(|| {
|
||||||
|
FinalizeError::PluginExecutionFailed(format!(
|
||||||
|
"publish plugin '{}' not found for artifact '{}'",
|
||||||
|
plugin_name, artifact.path,
|
||||||
|
))
|
||||||
|
})?;
|
||||||
|
|
||||||
|
// Inject artifact path into the plugin argument
|
||||||
|
let argument = if let Some(ref path) = artifact_path {
|
||||||
|
let mut arg = publish_config.clone();
|
||||||
|
if let Some(map) = arg.as_mapping_mut() {
|
||||||
|
map.insert(
|
||||||
|
serde_yaml::Value::String("artifact".into()),
|
||||||
|
serde_yaml::Value::String(path.to_string_lossy().into()),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
arg
|
||||||
|
} else {
|
||||||
|
publish_config.clone()
|
||||||
|
};
|
||||||
|
|
||||||
|
log::info!(
|
||||||
|
"Publishing artifact '{}' via plugin '{}'",
|
||||||
|
artifact.path, plugin_name,
|
||||||
|
);
|
||||||
|
|
||||||
|
plugin.call(argument, ctx, &event_tx).await?;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Resolve and optionally compress an artifact.
|
||||||
|
/// Returns the local path to the processed artifact, or `None` if non-file artifact.
|
||||||
|
async fn process_artifact(artifact: &ArtifactDef) -> Result<Option<PathBuf>, FinalizeError> {
|
||||||
|
if artifact.path.is_empty() {
|
||||||
|
return Ok(None); // non-filesystem artifact (e.g. Docker image)
|
||||||
|
}
|
||||||
|
|
||||||
|
let src = PathBuf::from(&artifact.path);
|
||||||
|
if !src.exists() {
|
||||||
|
if artifact.path.contains('*') || artifact.path.contains('?') {
|
||||||
|
return Err(FinalizeError::PluginExecutionFailed(
|
||||||
|
format!("glob patterns not supported: '{}'; use a directory path", artifact.path),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
return Err(FinalizeError::ArtifactNotFound(src));
|
||||||
|
}
|
||||||
|
|
||||||
|
match artifact.compression {
|
||||||
|
CompressionMethod::None => Ok(Some(src)),
|
||||||
|
CompressionMethod::Dir => Ok(Some(src)),
|
||||||
|
CompressionMethod::Gzip => {
|
||||||
|
let dst = std::env::temp_dir().join(format!("artifact-{}.tar.gz", artifact.id.as_deref().unwrap_or("unknown")));
|
||||||
|
compress_tar_zstd_or_gzip(&src, &dst, false).await?;
|
||||||
|
Ok(Some(dst))
|
||||||
|
}
|
||||||
|
CompressionMethod::Zstd => {
|
||||||
|
let dst = std::env::temp_dir().join(format!("artifact-{}.tar.zst", artifact.id.as_deref().unwrap_or("unknown")));
|
||||||
|
compress_tar_zstd_or_gzip(&src, &dst, true).await?;
|
||||||
|
Ok(Some(dst))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn compress_tar_zstd_or_gzip(
|
||||||
|
src: &Path,
|
||||||
|
dst: &Path,
|
||||||
|
zstd: bool,
|
||||||
|
) -> Result<(), FinalizeError> {
|
||||||
|
let _archive = tokio::task::spawn_blocking({
|
||||||
|
let src = src.to_path_buf();
|
||||||
|
let dst = dst.to_path_buf();
|
||||||
|
move || -> Result<(), String> {
|
||||||
|
let file = std::fs::File::create(&dst)
|
||||||
|
.map_err(|e| format!("cannot create {}: {}", dst.display(), e))?;
|
||||||
|
let writer: Box<dyn std::io::Write> = if zstd {
|
||||||
|
Box::new(zstd::stream::write::Encoder::new(file, 0).map_err(|e| e.to_string())?)
|
||||||
|
} else {
|
||||||
|
Box::new(flate2::write::GzEncoder::new(file, flate2::Compression::default()))
|
||||||
|
};
|
||||||
|
let mut tar = tar::Builder::new(writer);
|
||||||
|
if src.is_dir() {
|
||||||
|
tar.append_dir_all(".", &src)
|
||||||
|
.map_err(|e| format!("tar error: {}", e))?;
|
||||||
|
} else {
|
||||||
|
tar.append_path(&src)
|
||||||
|
.map_err(|e| format!("tar error: {}", e))?;
|
||||||
|
}
|
||||||
|
tar.into_inner().map_err(|e| e.to_string())?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.map_err(|e| FinalizeError::PluginExecutionFailed(format!("spawn_blocking: {}", e)))?
|
||||||
|
.map_err(FinalizeError::PluginExecutionFailed)?;
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
@@ -0,0 +1,67 @@
|
|||||||
|
//! Pipeline cleanup and resource reclamation.
|
||||||
|
//!
|
||||||
|
//! Runs after all pipeline stages and notification dispatch complete.
|
||||||
|
//! Checks the configured [`CleanupPolicy`] and performs the following:
|
||||||
|
//!
|
||||||
|
//! 1. Temp file / cache directory removal
|
||||||
|
//! 2. Policy-based skip (never, manual, on_success)
|
||||||
|
|
||||||
|
use crate::finalize::config::{CleanupConfig, CleanupPolicy};
|
||||||
|
use crate::finalize::error::FinalizeError;
|
||||||
|
use workshop_engine::ExecutionContext;
|
||||||
|
|
||||||
|
/// Execute cleanup according to the pipeline's [`CleanupPolicy`].
|
||||||
|
///
|
||||||
|
/// Returns `Ok(())` when cleanup completes (or is skipped per policy).
|
||||||
|
/// Returns `Err` only if a cleanup action itself fails (not if the
|
||||||
|
/// pipeline failed; pipeline outcome is checked via `pipeline_ok`).
|
||||||
|
pub fn cleanup(
|
||||||
|
config: &CleanupConfig,
|
||||||
|
_ctx: &ExecutionContext,
|
||||||
|
pipeline_ok: bool,
|
||||||
|
) -> Result<(), FinalizeError> {
|
||||||
|
match config.policy {
|
||||||
|
CleanupPolicy::Never => {
|
||||||
|
log::info!("[cleanup] policy=never, skipping all cleanup");
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
CleanupPolicy::Manual => {
|
||||||
|
log::info!("[cleanup] policy=manual, hanging for bakerd signal");
|
||||||
|
// TODO(daemon): hang on bakerd signal
|
||||||
|
// For now (standalone/bare) manual == Never
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
CleanupPolicy::OnSuccess if !pipeline_ok => {
|
||||||
|
log::info!("[cleanup] policy=on_success, but pipeline failed — skipping");
|
||||||
|
// TODO(daemon): hang for bakerd
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
log::info!("[cleanup] starting");
|
||||||
|
cleanup_temp_files();
|
||||||
|
subprocess_reclaim();
|
||||||
|
log::info!("[cleanup] complete");
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Reclaim child processes (placeholder for daemon-mode PID tracking).
|
||||||
|
/// In standalone mode, children are reaped by the kernel on process exit.
|
||||||
|
fn subprocess_reclaim() {
|
||||||
|
// TODO(daemon): track spawned PIDs during pipeline and signal them here.
|
||||||
|
}
|
||||||
|
/// Remove plugin cache and temporary directories.
|
||||||
|
fn cleanup_temp_files() {
|
||||||
|
let cache = std::path::Path::new("/tmp/baker");
|
||||||
|
if cache.exists() {
|
||||||
|
match std::fs::remove_dir_all(cache) {
|
||||||
|
Ok(()) => log::debug!("[cleanup] removed cache: {}", cache.display()),
|
||||||
|
Err(e) => log::warn!("[cleanup] failed to remove {}: {}", cache.display(), e),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let workshop_templates = std::env::temp_dir().join("workshop-templates");
|
||||||
|
if workshop_templates.exists() {
|
||||||
|
let _ = std::fs::remove_dir_all(&workshop_templates);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -115,6 +115,17 @@ pub struct NotificationMethod {
|
|||||||
pub policy: Vec<NotifyPolicy>,
|
pub policy: Vec<NotifyPolicy>,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
impl Default for NotificationMethod {
|
||||||
|
fn default() -> Self {
|
||||||
|
Self {
|
||||||
|
template: NotificationTemplate::default(),
|
||||||
|
trigger: Vec::new(),
|
||||||
|
config: Value::default(),
|
||||||
|
policy: Vec::new(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
#[derive(Debug, Deserialize, Serialize, Clone, Default)]
|
#[derive(Debug, Deserialize, Serialize, Clone, Default)]
|
||||||
pub struct NotificationTemplate {
|
pub struct NotificationTemplate {
|
||||||
/// Payload schema: "default" (plugin-defined) or "custom" (requires on_* templates)
|
/// Payload schema: "default" (plugin-defined) or "custom" (requires on_* templates)
|
||||||
@@ -134,17 +145,65 @@ pub struct NotificationTemplate {
|
|||||||
pub on_finish_of: Option<OnFinishOf>,
|
pub on_finish_of: Option<OnFinishOf>,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
impl std::ops::AddAssign for NotificationTemplate {
|
||||||
|
fn add_assign(&mut self, other: Self) {
|
||||||
|
if self.schema.is_none() {
|
||||||
|
self.schema = other.schema;
|
||||||
|
}
|
||||||
|
if self.on_success.is_none() {
|
||||||
|
self.on_success = other.on_success;
|
||||||
|
}
|
||||||
|
if self.on_failure.is_none() {
|
||||||
|
self.on_failure = other.on_failure;
|
||||||
|
}
|
||||||
|
if self.on_finish_of.is_none() {
|
||||||
|
self.on_finish_of = other.on_finish_of;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Notification configuration container
|
/// Notification configuration container
|
||||||
#[derive(Debug, Deserialize, Serialize, Clone, Default)]
|
#[derive(Debug, Serialize, Clone, Default)]
|
||||||
pub struct NotificationConfig {
|
pub struct NotificationConfig {
|
||||||
/// Template definitions for notification content
|
/// Template definitions for notification content
|
||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
pub templates: HashMap<String, NotificationTemplateDef>,
|
pub templates: HashMap<String, NotificationTemplateDef>,
|
||||||
|
|
||||||
#[serde(flatten)]
|
/// Priority groups (YAML keys like `0`, `1` are parsed as u32).
|
||||||
|
/// Custom Deserialize handles serde_yaml string→u32 conversion.
|
||||||
|
#[serde(default)]
|
||||||
pub groups: HashMap<u32, HashMap<String, NotificationMethod>>,
|
pub groups: HashMap<u32, HashMap<String, NotificationMethod>>,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Serde helper: YAML map keys are always strings, convert to u32.
|
||||||
|
impl<'de> serde::Deserialize<'de> for NotificationConfig {
|
||||||
|
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
|
||||||
|
where D: serde::Deserializer<'de>
|
||||||
|
{
|
||||||
|
#[derive(serde::Deserialize)]
|
||||||
|
struct Helper {
|
||||||
|
#[serde(default)]
|
||||||
|
templates: HashMap<String, NotificationTemplateDef>,
|
||||||
|
#[serde(flatten)]
|
||||||
|
groups_raw: HashMap<String, Option<HashMap<String, NotificationMethod>>>,
|
||||||
|
}
|
||||||
|
let h = Helper::deserialize(deserializer)?;
|
||||||
|
let groups: HashMap<u32, _> = h.groups_raw.into_iter()
|
||||||
|
.filter_map(|(k, v)| {
|
||||||
|
let parsed = k.parse::<u32>().map(|n| (n, v.unwrap_or_default()));
|
||||||
|
match parsed {
|
||||||
|
Ok(pair) => Some(pair),
|
||||||
|
Err(e) => {
|
||||||
|
log::warn!("priority key must be integer, got {:?}: {}", k, e);
|
||||||
|
None
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
Ok(NotificationConfig { templates: h.templates, groups })
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Stage-triggered notification configuration
|
/// Stage-triggered notification configuration
|
||||||
#[derive(Debug, Deserialize, Serialize, Clone)]
|
#[derive(Debug, Deserialize, Serialize, Clone)]
|
||||||
pub struct OnFinishOf {
|
pub struct OnFinishOf {
|
||||||
@@ -158,7 +217,7 @@ pub struct OnFinishOf {
|
|||||||
|
|
||||||
/// Notification template definition.
|
/// Notification template definition.
|
||||||
/// Supports inline content (on_success/on_failure/on_finish_of) or external reference (use).
|
/// Supports inline content (on_success/on_failure/on_finish_of) or external reference (use).
|
||||||
#[derive(Debug, Deserialize, Serialize, Clone)]
|
#[derive(Debug, Deserialize, Serialize, Clone, Default)]
|
||||||
pub struct NotificationTemplateDef {
|
pub struct NotificationTemplateDef {
|
||||||
/// External template reference (git URL).
|
/// External template reference (git URL).
|
||||||
/// Cannot be combined with on_* fields in MVP (validation will reject).
|
/// Cannot be combined with on_* fields in MVP (validation will reject).
|
||||||
@@ -178,6 +237,53 @@ pub struct NotificationTemplateDef {
|
|||||||
pub on_finish_of: Option<String>,
|
pub on_finish_of: Option<String>,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
impl NotificationTemplateDef {
|
||||||
|
/// Merge `other` into `self`, keeping self's values when present.
|
||||||
|
///
|
||||||
|
/// `other` fills in gaps where `self` is `None`.
|
||||||
|
/// Chaining: `a.apply(b).apply(c)` = a wins, then b fills gaps, then c fills gaps.
|
||||||
|
pub fn apply(mut self, other: Self) -> Self {
|
||||||
|
if self.on_success.is_none() {
|
||||||
|
self.on_success = other.on_success;
|
||||||
|
}
|
||||||
|
if self.on_failure.is_none() {
|
||||||
|
self.on_failure = other.on_failure;
|
||||||
|
}
|
||||||
|
if self.on_finish_of.is_none() {
|
||||||
|
self.on_finish_of = other.on_finish_of;
|
||||||
|
}
|
||||||
|
self
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Builder for [`NotificationTemplateDef`].
|
||||||
|
pub struct NotificationTemplateDefBuilder(NotificationTemplateDef);
|
||||||
|
|
||||||
|
impl NotificationTemplateDefBuilder {
|
||||||
|
pub fn new() -> Self {
|
||||||
|
Self(NotificationTemplateDef::default())
|
||||||
|
}
|
||||||
|
pub fn with_use(mut self, val: impl Into<String>) -> Self {
|
||||||
|
self.0.use_ = Some(val.into());
|
||||||
|
self
|
||||||
|
}
|
||||||
|
pub fn with_on_success(mut self, val: impl Into<String>) -> Self {
|
||||||
|
self.0.on_success = Some(val.into());
|
||||||
|
self
|
||||||
|
}
|
||||||
|
pub fn with_on_failure(mut self, val: impl Into<String>) -> Self {
|
||||||
|
self.0.on_failure = Some(val.into());
|
||||||
|
self
|
||||||
|
}
|
||||||
|
pub fn with_on_finish_of(mut self, val: impl Into<String>) -> Self {
|
||||||
|
self.0.on_finish_of = Some(val.into());
|
||||||
|
self
|
||||||
|
}
|
||||||
|
pub fn build(self) -> NotificationTemplateDef {
|
||||||
|
self.0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Notification trigger type.
|
/// Notification trigger type.
|
||||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||||
pub enum Trigger {
|
pub enum Trigger {
|
||||||
@@ -236,8 +342,9 @@ pub struct ArtifactDef {
|
|||||||
/// Compression method: "zstd", "gzip", "none", "dir"
|
/// Compression method: "zstd", "gzip", "none", "dir"
|
||||||
#[serde(default)]
|
#[serde(default)]
|
||||||
pub compression: CompressionMethod,
|
pub compression: CompressionMethod,
|
||||||
// Publishing targets
|
/// Publish targets: plugin name → configuration (artifact path injected by system)
|
||||||
// TODO: under heavy refactor
|
#[serde(default)]
|
||||||
|
pub publish: HashMap<String, serde_yaml::Value>
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Default for ArtifactDef {
|
impl Default for ArtifactDef {
|
||||||
@@ -247,6 +354,7 @@ impl Default for ArtifactDef {
|
|||||||
path: String::new(),
|
path: String::new(),
|
||||||
retention_ms: default_retention_ms(),
|
retention_ms: default_retention_ms(),
|
||||||
compression: CompressionMethod::default(),
|
compression: CompressionMethod::default(),
|
||||||
|
publish: HashMap::new(),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -336,15 +444,18 @@ impl FinalizeConfig {
|
|||||||
fn validate_notification(&self, errors: &mut Vec<String>) {
|
fn validate_notification(&self, errors: &mut Vec<String>) {
|
||||||
// Validate templates
|
// Validate templates
|
||||||
for (name, tmpl) in &self.notification.templates {
|
for (name, tmpl) in &self.notification.templates {
|
||||||
if tmpl.use_.is_some()
|
if let Some(ref u) = tmpl.use_ {
|
||||||
&& (tmpl.on_success.is_some()
|
if u.contains("://") && (
|
||||||
|| tmpl.on_failure.is_some()
|
tmpl.on_success.is_some() ||
|
||||||
|| tmpl.on_finish_of.is_some())
|
tmpl.on_failure.is_some() ||
|
||||||
{
|
tmpl.on_finish_of.is_some()
|
||||||
errors.push(format!(
|
) {
|
||||||
"Template '{}' cannot combine 'use' with 'on_*' fields (MVP restriction)",
|
errors.push(format!(
|
||||||
name
|
"Template '{}' cannot combine external 'use' with 'on_*' fields",
|
||||||
));
|
name
|
||||||
|
));
|
||||||
|
}
|
||||||
|
// Local references (no ://) are allowed to have on_* overrides
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,4 +1,5 @@
|
|||||||
use workshop_engine::{ExecutionEvent, StreamType};
|
use workshop_engine::{ExecutionEvent, StreamType};
|
||||||
|
use crate::types::debug::DebugFeature;
|
||||||
|
|
||||||
/// Receives and processes execution events for the finalize stage.
|
/// Receives and processes execution events for the finalize stage.
|
||||||
/// Logs task lifecycle events (started, completed, failed) and streaming output (stdout/stderr)
|
/// Logs task lifecycle events (started, completed, failed) and streaming output (stdout/stderr)
|
||||||
@@ -7,43 +8,62 @@ pub async fn event_receiver(
|
|||||||
mut rx: tokio::sync::mpsc::Receiver<ExecutionEvent>,
|
mut rx: tokio::sync::mpsc::Receiver<ExecutionEvent>,
|
||||||
pipeline_name: String,
|
pipeline_name: String,
|
||||||
build_id: String,
|
build_id: String,
|
||||||
|
debug_flags: u32,
|
||||||
) {
|
) {
|
||||||
|
let events_debug = DebugFeature::from_bits_retain(debug_flags).contains(DebugFeature::Events);
|
||||||
while let Some(event) = rx.recv().await {
|
while let Some(event) = rx.recv().await {
|
||||||
match event {
|
match event {
|
||||||
ExecutionEvent::TaskStarted { task_id, timestamp } => {
|
ExecutionEvent::TaskStarted { task_id, timestamp } => {
|
||||||
log::debug!(
|
if events_debug {
|
||||||
"[{}] [{}] [{}] Task started: {}",
|
log::info!(
|
||||||
pipeline_name,
|
"[{}] [{}] [{}] Task started: {}",
|
||||||
build_id,
|
pipeline_name,
|
||||||
timestamp,
|
build_id,
|
||||||
task_id
|
timestamp,
|
||||||
);
|
task_id
|
||||||
|
);
|
||||||
|
} else {
|
||||||
|
log::debug!(
|
||||||
|
"Task started: {}",
|
||||||
|
task_id
|
||||||
|
);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
ExecutionEvent::OutputChunk {
|
ExecutionEvent::OutputChunk {
|
||||||
task_id,
|
task_id: _,
|
||||||
stream,
|
stream,
|
||||||
data,
|
data,
|
||||||
} => match stream {
|
} => match stream {
|
||||||
StreamType::Stdout => print!("[{}] [{}] [stdout] {}", pipeline_name, task_id, data),
|
StreamType::Stdout => print!("{}", data),
|
||||||
StreamType::Stderr => {
|
StreamType::Stderr => {
|
||||||
eprint!("[{}] [{}] [stderr] {}", pipeline_name, task_id, data)
|
eprint!("{}", data)
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
ExecutionEvent::TaskCompleted { task_id, result } => {
|
ExecutionEvent::TaskCompleted { task_id, result } => {
|
||||||
log::debug!(
|
if result.success {
|
||||||
"[{}] [{}] Task completed: {} (exit_code={}, duration={:?})",
|
if events_debug {
|
||||||
pipeline_name,
|
log::info!(
|
||||||
build_id,
|
"[{}] [{}] Task completed: {} (exit_code={}, duration={:?})",
|
||||||
task_id,
|
pipeline_name, build_id, task_id,
|
||||||
result.exit_code,
|
result.exit_code, result.duration,
|
||||||
result.duration
|
);
|
||||||
);
|
} else {
|
||||||
|
log::debug!(
|
||||||
|
"Task completed: {} (exit_code={}, duration={:?})",
|
||||||
|
task_id, result.exit_code, result.duration,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
log::warn!(
|
||||||
|
"Task completed with non-zero exit: {} (exit_code={}, duration={:?})",
|
||||||
|
task_id,
|
||||||
|
result.exit_code, result.duration,
|
||||||
|
);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
ExecutionEvent::TaskFailed { task_id, error } => {
|
ExecutionEvent::TaskFailed { task_id, error } => {
|
||||||
log::error!(
|
log::error!(
|
||||||
"[{}] [{}] Task failed: {} (error={})",
|
"Task failed: {} (error={})",
|
||||||
pipeline_name,
|
|
||||||
build_id,
|
|
||||||
task_id,
|
task_id,
|
||||||
error
|
error
|
||||||
);
|
);
|
||||||
|
|||||||
@@ -1,15 +1,16 @@
|
|||||||
use crate::finalize::RawPluginMap;
|
use crate::finalize::RawPluginMap;
|
||||||
use crate::finalize::constant::{FINALIZE_PLUGIN_MANIFEST, FINALIZE_PLUGIN_MANIFEST_EXTENSION};
|
use crate::finalize::constant::{FINALIZE_PLUGIN_MANIFEST, FINALIZE_PLUGIN_MANIFEST_EXTENSION};
|
||||||
use crate::finalize::plugin::metadata::{PluginMetadata, PluginType};
|
use crate::finalize::plugin::metadata::{PluginMetadata, PluginType};
|
||||||
use crate::{EventSender, ExecutionContext, finalize::error::PluginError};
|
use workshop_engine::{EventSender, ExecutionContext};
|
||||||
|
use crate::{ finalize::error::PluginError};
|
||||||
|
use crate::types::resource::ResourceRegistry;
|
||||||
use async_trait::async_trait;
|
use async_trait::async_trait;
|
||||||
use std::collections::{HashMap, HashSet};
|
use std::collections::{HashMap, HashSet};
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
use std::sync::Arc;
|
use std::sync::Arc;
|
||||||
|
|
||||||
mod dylib;
|
mod dylib;
|
||||||
pub mod fetch;
|
pub mod internal;
|
||||||
mod internal;
|
|
||||||
mod metadata;
|
mod metadata;
|
||||||
mod rhai;
|
mod rhai;
|
||||||
mod shell;
|
mod shell;
|
||||||
@@ -214,6 +215,8 @@ fn get_entry(plugin_type: PluginType, name: &str) -> Result<Box<dyn AsyncPluginF
|
|||||||
pub fn register(
|
pub fn register(
|
||||||
external_plugins: RawPluginMap,
|
external_plugins: RawPluginMap,
|
||||||
list: Option<&HashSet<String>>,
|
list: Option<&HashSet<String>>,
|
||||||
|
notification_config: &mut crate::finalize::NotificationConfig,
|
||||||
|
registry: &ResourceRegistry,
|
||||||
) -> Result<PluginMap, PluginError> {
|
) -> Result<PluginMap, PluginError> {
|
||||||
let mut plugins: PluginMap = HashMap::new();
|
let mut plugins: PluginMap = HashMap::new();
|
||||||
let mut count = 0;
|
let mut count = 0;
|
||||||
@@ -229,7 +232,9 @@ pub fn register(
|
|||||||
_ => {}
|
_ => {}
|
||||||
}
|
}
|
||||||
log::debug!("Registering internal plugin {}", i.name);
|
log::debug!("Registering internal plugin {}", i.name);
|
||||||
plugins.insert(i.name.to_string(), FinalizePlugin::new(i.entry));
|
let mut plugin = FinalizePlugin::new(i.entry);
|
||||||
|
plugin.metadata.renderer = Some(i.renderer);
|
||||||
|
plugins.insert(i.name.to_string(), plugin);
|
||||||
count += 1;
|
count += 1;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -255,7 +260,14 @@ pub fn register(
|
|||||||
log::warn!("Plugin {} is not available: {}", name, reason);
|
log::warn!("Plugin {} is not available: {}", name, reason);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
let metadata = read_manifest(config.runtime.fspath, &name)?;
|
// Resolve plugin path from registry instead of config.runtime.fspath
|
||||||
|
let plugin_path = registry.resolve(&name).ok_or_else(|| {
|
||||||
|
PluginError::PluginUnavailable {
|
||||||
|
name: name.clone(),
|
||||||
|
reason: "plugin not found in resource registry".to_string(),
|
||||||
|
}
|
||||||
|
})?;
|
||||||
|
let metadata = read_manifest(Some(plugin_path), &name)?;
|
||||||
let plugin_type = get_plugin_type(metadata.plugin_type.clone(), &metadata.entrypoint);
|
let plugin_type = get_plugin_type(metadata.plugin_type.clone(), &metadata.entrypoint);
|
||||||
let entry = get_entry(plugin_type, &name)?;
|
let entry = get_entry(plugin_type, &name)?;
|
||||||
plugins.insert(
|
plugins.insert(
|
||||||
@@ -267,8 +279,18 @@ pub fn register(
|
|||||||
count += 1;
|
count += 1;
|
||||||
}
|
}
|
||||||
|
|
||||||
log::info!("Registered {} plugins", count);
|
// Inject plugin-declared templates as .fallback
|
||||||
|
for (name, plugin) in &plugins {
|
||||||
|
if let Some(ref templates) = plugin.metadata.templates {
|
||||||
|
let key = format!("{}.fallback", name);
|
||||||
|
for tmpl in templates {
|
||||||
|
notification_config.templates.insert(key.clone(), tmpl.clone());
|
||||||
|
}
|
||||||
|
log::debug!("Registered templates from plugin '{}' as '{}'", name, key);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
log::info!("Registered {} plugins", count);
|
||||||
Ok(plugins)
|
Ok(plugins)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
use crate::finalize::plugin::PluginError;
|
use crate::finalize::plugin::PluginError;
|
||||||
use crate::finalize::plugin::PluginMetadata;
|
use crate::finalize::plugin::PluginMetadata;
|
||||||
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn};
|
use workshop_engine::{EventSender, ExecutionContext};
|
||||||
|
use crate::{ finalize::plugin::AsyncPluginFn};
|
||||||
use async_trait::async_trait;
|
use async_trait::async_trait;
|
||||||
|
|
||||||
pub struct Dylib;
|
pub struct Dylib;
|
||||||
|
|||||||
@@ -1,109 +0,0 @@
|
|||||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
|
||||||
use crate::finalize::fetch;
|
|
||||||
use crate::finalize::plugin::{PluginError, RawPluginMap};
|
|
||||||
use std::collections::HashSet;
|
|
||||||
use std::path::Path;
|
|
||||||
|
|
||||||
mod checksum;
|
|
||||||
mod extract;
|
|
||||||
mod git;
|
|
||||||
mod http;
|
|
||||||
mod types;
|
|
||||||
|
|
||||||
pub use types::FetchArgument;
|
|
||||||
|
|
||||||
enum URLScheme {
|
|
||||||
Unknown,
|
|
||||||
Git,
|
|
||||||
Http,
|
|
||||||
Https,
|
|
||||||
File,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl URLScheme {
|
|
||||||
fn parse(url: &str) -> (Self, String) {
|
|
||||||
match url.split_once("://") {
|
|
||||||
Some((prefix, rest)) => {
|
|
||||||
let scheme = match prefix {
|
|
||||||
"git" => URLScheme::Git,
|
|
||||||
"https" => URLScheme::Https,
|
|
||||||
"http" => URLScheme::Http,
|
|
||||||
"file" => URLScheme::File,
|
|
||||||
_ => URLScheme::Unknown,
|
|
||||||
};
|
|
||||||
(scheme, rest.to_string())
|
|
||||||
}
|
|
||||||
None => (URLScheme::Unknown, url.to_string()),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
pub async fn fetch_plugin(
|
|
||||||
url: &str,
|
|
||||||
name: &str,
|
|
||||||
basedir: &Path,
|
|
||||||
argument: FetchArgument,
|
|
||||||
) -> Result<(), PluginError> {
|
|
||||||
let (scheme, path) = URLScheme::parse(url);
|
|
||||||
|
|
||||||
match scheme {
|
|
||||||
URLScheme::Git => {
|
|
||||||
// NOTE: as a practical compromise, we use git command for git clone
|
|
||||||
// gix will be supported later, with ONLY https scheme
|
|
||||||
// TODO: Implement gix
|
|
||||||
log::info!("Fetching git plugin: {}", url);
|
|
||||||
git::fetch_git_plugin(&path, name, basedir, argument.shallow).await
|
|
||||||
}
|
|
||||||
URLScheme::Http | URLScheme::Https => {
|
|
||||||
log::info!("Fetching http plugin: {}", url);
|
|
||||||
let checksum = argument.get_checksum();
|
|
||||||
http::fetch_http_plugin(url, name, basedir, checksum).await
|
|
||||||
}
|
|
||||||
URLScheme::File => {
|
|
||||||
// TODO: Support client mode
|
|
||||||
// NOTE: In standalone mode, we just assume path is well-defined and utilize
|
|
||||||
log::info!("Fetching file plugin: {}", url);
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
URLScheme::Unknown => {
|
|
||||||
log::error!("Unknown URL scheme in plugin URL: {}", url);
|
|
||||||
Err(PluginError::GeneralError(format!(
|
|
||||||
"Unknown URL scheme in plugin URL: {}",
|
|
||||||
url
|
|
||||||
)))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
pub async fn fetch_plugins(
|
|
||||||
plugin_path: &Path,
|
|
||||||
plugin_config: &mut RawPluginMap,
|
|
||||||
list: Option<&HashSet<String>>,
|
|
||||||
) -> Result<(), PluginError> {
|
|
||||||
log::info!("Preparing to download {} plugins", plugin_config.len());
|
|
||||||
for (name, data) in plugin_config.iter_mut() {
|
|
||||||
match list {
|
|
||||||
Some(list) if !list.contains(name) => {
|
|
||||||
log::debug!("Plugin {} not in list, skipping", name);
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
_ => {}
|
|
||||||
}
|
|
||||||
log::debug!("Downloading plugin {}", name);
|
|
||||||
let (source, config) =
|
|
||||||
data.iter_mut()
|
|
||||||
.next()
|
|
||||||
.ok_or_else(|| PluginError::PluginUnavailable {
|
|
||||||
name: name.clone(),
|
|
||||||
reason: "no source configured".to_string(),
|
|
||||||
})?;
|
|
||||||
let fetch_argument = FetchArgument {
|
|
||||||
checksum: config.checksum.clone(),
|
|
||||||
shallow: config.shallow,
|
|
||||||
};
|
|
||||||
fetch::fetch_plugin(source, name, plugin_path, fetch_argument).await?;
|
|
||||||
log::debug!("Storing plugin {} to {}", name, plugin_path.display());
|
|
||||||
config.runtime.fspath = Some(plugin_path.join(name));
|
|
||||||
}
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
@@ -1,137 +0,0 @@
|
|||||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
|
||||||
use super::types::ChecksumType;
|
|
||||||
use sha2::{Digest, Sha256, Sha512};
|
|
||||||
|
|
||||||
/// Detects the checksum type based on the length of the checksum string.
|
|
||||||
///
|
|
||||||
/// Returns `Ok(None)` for empty string (checksum disabled).
|
|
||||||
/// Returns `Ok(Some(ChecksumType))` for valid SHA256 (64 chars) or SHA512 (128 chars).
|
|
||||||
/// Returns `Err` for invalid lengths (32=MD5, 40=SHA1) or unsupported lengths.
|
|
||||||
pub fn detect_checksum_type(checksum: &str) -> Result<Option<ChecksumType>, String> {
|
|
||||||
match checksum.len() {
|
|
||||||
0 => Ok(None),
|
|
||||||
32 => Err(format!(
|
|
||||||
"MD5 (first 8 chars: {}) is not supported for security reasons. Use SHA256 (64 chars) or SHA512 (128 chars)",
|
|
||||||
&checksum[..8.min(checksum.len())]
|
|
||||||
)),
|
|
||||||
40 => Err(format!(
|
|
||||||
"SHA1 (first 8 chars: {}) is deprecated and not supported. Use SHA256 (64 chars) or SHA512 (128 chars)",
|
|
||||||
&checksum[..8.min(checksum.len())]
|
|
||||||
)),
|
|
||||||
64 => Ok(Some(ChecksumType::Sha256)),
|
|
||||||
128 => Ok(Some(ChecksumType::Sha512)),
|
|
||||||
_ => Err(format!(
|
|
||||||
"Invalid checksum length: {}. Expected 0 (disabled), 64 (SHA256), or 128 (SHA512). First 8 chars: {}",
|
|
||||||
checksum.len(),
|
|
||||||
&checksum[..8.min(checksum.len())]
|
|
||||||
)),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Verifies that the data matches the expected checksum.
|
|
||||||
///
|
|
||||||
/// Compares the computed hash of `data` against `expected` using the specified `checksum_type`.
|
|
||||||
/// Returns `Ok(())` if the checksums match, `Err` otherwise.
|
|
||||||
pub fn verify_checksum(
|
|
||||||
data: &[u8],
|
|
||||||
expected: &str,
|
|
||||||
checksum_type: ChecksumType,
|
|
||||||
) -> Result<(), String> {
|
|
||||||
let actual = match checksum_type {
|
|
||||||
ChecksumType::Sha256 => {
|
|
||||||
let mut hasher = Sha256::new();
|
|
||||||
hasher.update(data);
|
|
||||||
format!("{:x}", hasher.finalize())
|
|
||||||
}
|
|
||||||
ChecksumType::Sha512 => {
|
|
||||||
let mut hasher = Sha512::new();
|
|
||||||
hasher.update(data);
|
|
||||||
format!("{:x}", hasher.finalize())
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
if actual.eq_ignore_ascii_case(expected) {
|
|
||||||
Ok(())
|
|
||||||
} else {
|
|
||||||
Err(format!(
|
|
||||||
"Checksum mismatch: expected {}..., got {}...",
|
|
||||||
&expected[..8.min(expected.len())],
|
|
||||||
&actual[..8.min(actual.len())]
|
|
||||||
))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod tests {
|
|
||||||
use super::*;
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_detect_checksum_type_empty() {
|
|
||||||
assert_eq!(detect_checksum_type("").unwrap(), None);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_detect_checksum_type_md5_rejected() {
|
|
||||||
let result = detect_checksum_type("a".repeat(32).as_str());
|
|
||||||
assert!(result.is_err());
|
|
||||||
assert!(result.unwrap_err().contains("MD5"));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_detect_checksum_type_sha1_rejected() {
|
|
||||||
let result = detect_checksum_type("a".repeat(40).as_str());
|
|
||||||
assert!(result.is_err());
|
|
||||||
assert!(result.unwrap_err().contains("SHA1"));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_detect_checksum_type_sha256() {
|
|
||||||
assert_eq!(
|
|
||||||
detect_checksum_type("a".repeat(64).as_str()).unwrap(),
|
|
||||||
Some(ChecksumType::Sha256)
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_detect_checksum_type_sha512() {
|
|
||||||
assert_eq!(
|
|
||||||
detect_checksum_type("a".repeat(128).as_str()).unwrap(),
|
|
||||||
Some(ChecksumType::Sha512)
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_detect_checksum_type_invalid_length() {
|
|
||||||
let result = detect_checksum_type("a".repeat(50).as_str());
|
|
||||||
assert!(result.is_err());
|
|
||||||
assert!(result.unwrap_err().contains("Invalid checksum length"));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_verify_checksum_sha256_valid() {
|
|
||||||
let data = b"hello";
|
|
||||||
let expected = "2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824";
|
|
||||||
assert!(verify_checksum(data, expected, ChecksumType::Sha256).is_ok());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_verify_checksum_sha256_invalid() {
|
|
||||||
let data = b"hello";
|
|
||||||
let expected = "0000000000000000000000000000000000000000000000000000000000000000";
|
|
||||||
assert!(verify_checksum(data, expected, ChecksumType::Sha256).is_err());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_verify_checksum_sha512_valid() {
|
|
||||||
let data = b"hello";
|
|
||||||
let expected = "9b71d224bd62f3785d96d46ad3ea3d73319bfbc2890caadae2dff72519673ca72323c3d99ba5c11d7c7acc6e14b8c5da0c4663475c2e5c3adef46f73bcdec043";
|
|
||||||
assert!(verify_checksum(data, expected, ChecksumType::Sha512).is_ok());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_verify_checksum_sha512_invalid() {
|
|
||||||
let data = b"hello";
|
|
||||||
let expected = "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000";
|
|
||||||
assert!(verify_checksum(data, expected, ChecksumType::Sha512).is_err());
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,169 +0,0 @@
|
|||||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
|
||||||
use crate::finalize::plugin::PluginError;
|
|
||||||
use std::path::{Path, PathBuf};
|
|
||||||
use tokio::task;
|
|
||||||
|
|
||||||
/// Compression type for tar archives.
|
|
||||||
#[derive(Debug, Clone, Copy, PartialEq)]
|
|
||||||
pub enum CompressionType {
|
|
||||||
/// Gzip compressed (.tar.gz, .tgz)
|
|
||||||
Gzip,
|
|
||||||
/// Zstd compressed (.tar.zst, .tar.zstd, .tzst)
|
|
||||||
Zstd,
|
|
||||||
/// Uncompressed tar (.tar)
|
|
||||||
None,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl CompressionType {
|
|
||||||
/// Detects compression type from file path extension.
|
|
||||||
///
|
|
||||||
/// Supports: .tar.gz, .tgz (Gzip), .tar.zst, .tar.zstd, .tzst (Zstd), .tar (None).
|
|
||||||
pub fn from_path(path: &std::path::Path) -> Result<Self, String> {
|
|
||||||
let name = path
|
|
||||||
.file_name()
|
|
||||||
.and_then(|n| n.to_str())
|
|
||||||
.unwrap_or("")
|
|
||||||
.to_lowercase();
|
|
||||||
|
|
||||||
if name.ends_with(".tar.gz") || name.ends_with(".tgz") {
|
|
||||||
Ok(CompressionType::Gzip)
|
|
||||||
} else if name.ends_with(".tar.zst")
|
|
||||||
|| name.ends_with(".tar.zstd")
|
|
||||||
|| name.ends_with(".tzst")
|
|
||||||
{
|
|
||||||
Ok(CompressionType::Zstd)
|
|
||||||
} else if name.ends_with(".tar") {
|
|
||||||
Ok(CompressionType::None)
|
|
||||||
} else {
|
|
||||||
Err(format!(
|
|
||||||
"Unsupported archive format: '{}'. Supported formats: .tar, .tar.gz, .tgz, .tar.zst, .tar.zstd, .tzst",
|
|
||||||
name
|
|
||||||
))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Extracts a tar archive to the destination directory.
|
|
||||||
///
|
|
||||||
/// Removes existing destination directory if present, creates a fresh directory,
|
|
||||||
/// then unpacks the archive using the appropriate decompression based on `compression`.
|
|
||||||
pub async fn extract_tar(
|
|
||||||
archive: &Path,
|
|
||||||
destdir: &PathBuf,
|
|
||||||
compression: CompressionType,
|
|
||||||
) -> Result<(), PluginError> {
|
|
||||||
if destdir.exists() {
|
|
||||||
tokio::fs::remove_dir_all(destdir).await.map_err(|e| {
|
|
||||||
PluginError::GeneralError(format!("Failed to remove existing directory: {}", e))
|
|
||||||
})?;
|
|
||||||
}
|
|
||||||
tokio::fs::create_dir_all(destdir)
|
|
||||||
.await
|
|
||||||
.map_err(|e| PluginError::GeneralError(format!("Failed to create directory: {}", e)))?;
|
|
||||||
|
|
||||||
let archive = archive.to_path_buf();
|
|
||||||
let destdir = destdir.clone();
|
|
||||||
|
|
||||||
task::spawn_blocking(move || {
|
|
||||||
let file = std::fs::File::open(&archive)
|
|
||||||
.map_err(|e| PluginError::GeneralError(format!("Failed to open archive: {}", e)))?;
|
|
||||||
|
|
||||||
match compression {
|
|
||||||
CompressionType::Gzip => {
|
|
||||||
let decoder = flate2::read::GzDecoder::new(file);
|
|
||||||
let mut archive = tar::Archive::new(decoder);
|
|
||||||
archive.unpack(&destdir).map_err(|e| {
|
|
||||||
PluginError::GeneralError(format!("Failed to unpack archive: {}", e))
|
|
||||||
})?;
|
|
||||||
}
|
|
||||||
CompressionType::Zstd => {
|
|
||||||
let decoder = zstd::stream::read::Decoder::new(file).map_err(|e| {
|
|
||||||
PluginError::GeneralError(format!("Failed to create zstd decoder: {}", e))
|
|
||||||
})?;
|
|
||||||
let mut archive = tar::Archive::new(decoder);
|
|
||||||
archive.unpack(&destdir).map_err(|e| {
|
|
||||||
PluginError::GeneralError(format!("Failed to unpack archive: {}", e))
|
|
||||||
})?;
|
|
||||||
}
|
|
||||||
CompressionType::None => {
|
|
||||||
let mut archive = tar::Archive::new(file);
|
|
||||||
archive.unpack(&destdir).map_err(|e| {
|
|
||||||
PluginError::GeneralError(format!("Failed to unpack archive: {}", e))
|
|
||||||
})?;
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
Ok::<(), PluginError>(())
|
|
||||||
})
|
|
||||||
.await
|
|
||||||
.map_err(|e| PluginError::GeneralError(format!("Task join error: {}", e)))??;
|
|
||||||
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod tests {
|
|
||||||
use super::*;
|
|
||||||
use std::path::Path;
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_from_path_tar_gz() {
|
|
||||||
assert_eq!(
|
|
||||||
CompressionType::from_path(Path::new("archive.tar.gz")).unwrap(),
|
|
||||||
CompressionType::Gzip
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_from_path_tgz() {
|
|
||||||
assert_eq!(
|
|
||||||
CompressionType::from_path(Path::new("archive.tgz")).unwrap(),
|
|
||||||
CompressionType::Gzip
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_from_path_tar_zst() {
|
|
||||||
assert_eq!(
|
|
||||||
CompressionType::from_path(Path::new("archive.tar.zst")).unwrap(),
|
|
||||||
CompressionType::Zstd
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_from_path_tar_zstd() {
|
|
||||||
assert_eq!(
|
|
||||||
CompressionType::from_path(Path::new("archive.tar.zstd")).unwrap(),
|
|
||||||
CompressionType::Zstd
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_from_path_tzst() {
|
|
||||||
assert_eq!(
|
|
||||||
CompressionType::from_path(Path::new("archive.tzst")).unwrap(),
|
|
||||||
CompressionType::Zstd
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_from_path_tar() {
|
|
||||||
assert_eq!(
|
|
||||||
CompressionType::from_path(Path::new("archive.tar")).unwrap(),
|
|
||||||
CompressionType::None
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_from_path_unsupported() {
|
|
||||||
assert!(CompressionType::from_path(Path::new("archive.zip")).is_err());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_from_path_case_insensitive() {
|
|
||||||
assert_eq!(
|
|
||||||
CompressionType::from_path(Path::new("archive.TAR.GZ")).unwrap(),
|
|
||||||
CompressionType::Gzip
|
|
||||||
);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,185 +0,0 @@
|
|||||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
|
||||||
use crate::finalize::plugin::PluginError;
|
|
||||||
use std::path::Path;
|
|
||||||
use tokio::process::Command;
|
|
||||||
|
|
||||||
/// Version specification for a git repository.
|
|
||||||
#[derive(Debug, PartialEq)]
|
|
||||||
pub enum GitVersion {
|
|
||||||
/// No version specified (invalid for fetch operations)
|
|
||||||
None,
|
|
||||||
/// Single version string (tag or commit hash)
|
|
||||||
Single(String),
|
|
||||||
/// Mixed: try tag first, fallback to commit hash
|
|
||||||
Mixed(String, String),
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Parses a git URL with optional version specification.
|
|
||||||
///
|
|
||||||
/// Format: `url` (no version), `url@tag` (single version), `url@tag:commit` (mixed version).
|
|
||||||
/// Returns (base_url, GitVersion).
|
|
||||||
pub fn parse_git_url(url: &str) -> (String, GitVersion) {
|
|
||||||
if let Some((base, version)) = url.rsplit_once('@') {
|
|
||||||
if version.contains(":") {
|
|
||||||
let (tag, commit) = version.split_once(':').unwrap();
|
|
||||||
return (
|
|
||||||
base.to_string(),
|
|
||||||
GitVersion::Mixed(tag.to_string(), commit.to_string()),
|
|
||||||
);
|
|
||||||
}
|
|
||||||
return (base.to_string(), GitVersion::Single(version.to_string()));
|
|
||||||
}
|
|
||||||
(url.to_string(), GitVersion::None)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Clones a git repository and checks out the specified version.
|
|
||||||
///
|
|
||||||
/// Removes existing destination directory, performs shallow or full clone,
|
|
||||||
/// then checks out the given version (tag or commit).
|
|
||||||
pub async fn git_clone_and_checkout(
|
|
||||||
baseurl: &str,
|
|
||||||
version: &str,
|
|
||||||
dest: &Path,
|
|
||||||
name: &str,
|
|
||||||
shallow: bool,
|
|
||||||
) -> Result<(), PluginError> {
|
|
||||||
if dest.exists() {
|
|
||||||
tokio::fs::remove_dir_all(dest)
|
|
||||||
.await
|
|
||||||
.map_err(|e| PluginError::FetchError {
|
|
||||||
name: name.to_string(),
|
|
||||||
url: baseurl.to_string(),
|
|
||||||
reason: format!("Failed to remove existing directory: {}", e),
|
|
||||||
})?;
|
|
||||||
}
|
|
||||||
let mut cmd = Command::new("git");
|
|
||||||
cmd.arg("clone");
|
|
||||||
if shallow {
|
|
||||||
cmd.arg("--depth").arg("1");
|
|
||||||
}
|
|
||||||
cmd.arg(baseurl).arg(dest);
|
|
||||||
let output = cmd.output().await.map_err(|e| PluginError::FetchError {
|
|
||||||
name: name.to_string(),
|
|
||||||
url: baseurl.to_string(),
|
|
||||||
reason: e.to_string(),
|
|
||||||
})?;
|
|
||||||
if !output.status.success() {
|
|
||||||
return Err(PluginError::FetchError {
|
|
||||||
name: name.to_string(),
|
|
||||||
url: baseurl.to_string(),
|
|
||||||
reason: String::from_utf8_lossy(&output.stderr).to_string(),
|
|
||||||
});
|
|
||||||
}
|
|
||||||
let mut cmd = Command::new("git");
|
|
||||||
cmd.arg("checkout").arg(version).current_dir(dest);
|
|
||||||
let output = cmd.output().await.map_err(|e| PluginError::FetchError {
|
|
||||||
name: name.to_string(),
|
|
||||||
url: baseurl.to_string(),
|
|
||||||
reason: e.to_string(),
|
|
||||||
})?;
|
|
||||||
if !output.status.success() {
|
|
||||||
return Err(PluginError::FetchError {
|
|
||||||
name: name.to_string(),
|
|
||||||
url: baseurl.to_string(),
|
|
||||||
reason: String::from_utf8_lossy(&output.stderr).to_string(),
|
|
||||||
});
|
|
||||||
}
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Fetches a plugin from a git repository.
|
|
||||||
///
|
|
||||||
/// Parses the URL for version info, then clones and checks out the appropriate ref.
|
|
||||||
/// For Mixed version, tries the tag first and falls back to the commit hash.
|
|
||||||
pub async fn fetch_git_plugin(
|
|
||||||
path: &str,
|
|
||||||
name: &str,
|
|
||||||
basedir: &Path,
|
|
||||||
shallow: Option<bool>,
|
|
||||||
) -> Result<(), PluginError> {
|
|
||||||
let (baseurl, version) = parse_git_url(path);
|
|
||||||
match version {
|
|
||||||
GitVersion::None => {
|
|
||||||
log::error!("Version should be specified for git plugin URL: {}", path);
|
|
||||||
Err(PluginError::FetchError {
|
|
||||||
name: name.to_string(),
|
|
||||||
url: path.to_string(),
|
|
||||||
reason: "Version should be specified as either a tag or a commit hash.".to_string(),
|
|
||||||
})
|
|
||||||
}
|
|
||||||
GitVersion::Single(version) => {
|
|
||||||
log::debug!("Fetching git plugin: {}@{}", name, version);
|
|
||||||
let shallow_flag = shallow.unwrap_or(false);
|
|
||||||
git_clone_and_checkout(
|
|
||||||
&baseurl,
|
|
||||||
&version.to_string(),
|
|
||||||
&basedir.join(name),
|
|
||||||
name,
|
|
||||||
shallow_flag,
|
|
||||||
)
|
|
||||||
.await
|
|
||||||
}
|
|
||||||
GitVersion::Mixed(tag, commit) => {
|
|
||||||
log::debug!("Fetching git plugin: {}@{}/{}", name, tag, commit);
|
|
||||||
let shallow_flag = shallow.unwrap_or(false);
|
|
||||||
let result =
|
|
||||||
git_clone_and_checkout(&baseurl, &tag, &basedir.join(name), name, shallow_flag)
|
|
||||||
.await;
|
|
||||||
if result.is_err() {
|
|
||||||
log::warn!(
|
|
||||||
"Failed to clone with tag, trying commit hash. URL: {}",
|
|
||||||
path
|
|
||||||
);
|
|
||||||
return git_clone_and_checkout(&baseurl, &commit, &basedir.join(name), name, false)
|
|
||||||
.await;
|
|
||||||
}
|
|
||||||
result
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod tests {
|
|
||||||
use super::*;
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_parse_git_url_no_version() {
|
|
||||||
let (url, version) = parse_git_url("https://github.com/user/repo");
|
|
||||||
assert_eq!(url, "https://github.com/user/repo");
|
|
||||||
assert_eq!(version, GitVersion::None);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_parse_git_url_single_tag() {
|
|
||||||
let (url, version) = parse_git_url("https://github.com/user/repo@v1.0.0");
|
|
||||||
assert_eq!(url, "https://github.com/user/repo");
|
|
||||||
assert_eq!(version, GitVersion::Single("v1.0.0".to_string()));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_parse_git_url_mixed_version() {
|
|
||||||
let (url, version) = parse_git_url("https://github.com/user/repo@main:abc123");
|
|
||||||
assert_eq!(url, "https://github.com/user/repo");
|
|
||||||
assert_eq!(
|
|
||||||
version,
|
|
||||||
GitVersion::Mixed("main".to_string(), "abc123".to_string())
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_parse_git_url_complex_url() {
|
|
||||||
let (url, version) = parse_git_url("git@github.com:user/repo.git@tag:commit");
|
|
||||||
assert_eq!(url, "git@github.com:user/repo.git");
|
|
||||||
assert_eq!(
|
|
||||||
version,
|
|
||||||
GitVersion::Mixed("tag".to_string(), "commit".to_string())
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_parse_git_url_single_with_at_in_path() {
|
|
||||||
let (url, version) = parse_git_url("https://github.com/user/repo@my-repo@v1.0");
|
|
||||||
assert_eq!(url, "https://github.com/user/repo@my-repo");
|
|
||||||
assert_eq!(version, GitVersion::Single("v1.0".to_string()));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,110 +0,0 @@
|
|||||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
|
||||||
use super::checksum::{detect_checksum_type, verify_checksum};
|
|
||||||
use super::extract::{CompressionType, extract_tar};
|
|
||||||
use crate::finalize::plugin::PluginError;
|
|
||||||
use std::path::Path;
|
|
||||||
|
|
||||||
pub async fn download_to_file(url: &str, dest: &Path, name: &str) -> Result<(), PluginError> {
|
|
||||||
let client = reqwest::Client::builder()
|
|
||||||
.timeout(std::time::Duration::from_secs(300))
|
|
||||||
.build()
|
|
||||||
.map_err(|e| PluginError::GeneralError(format!("Failed to build HTTP client: {}", e)))?;
|
|
||||||
|
|
||||||
let response = client
|
|
||||||
.get(url)
|
|
||||||
.send()
|
|
||||||
.await
|
|
||||||
.map_err(|e| PluginError::FetchError {
|
|
||||||
name: name.to_string(),
|
|
||||||
url: url.to_string(),
|
|
||||||
reason: format!("HTTP request failed: {}", e),
|
|
||||||
})?;
|
|
||||||
|
|
||||||
let status = response.status();
|
|
||||||
if !status.is_success() {
|
|
||||||
return Err(PluginError::FetchError {
|
|
||||||
name: name.to_string(),
|
|
||||||
url: url.to_string(),
|
|
||||||
reason: format!("HTTP error: {}", status),
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
let bytes = response
|
|
||||||
.bytes()
|
|
||||||
.await
|
|
||||||
.map_err(|e| PluginError::FetchError {
|
|
||||||
name: name.to_string(),
|
|
||||||
url: url.to_string(),
|
|
||||||
reason: format!("Failed to read response body: {}", e),
|
|
||||||
})?;
|
|
||||||
|
|
||||||
tokio::fs::write(dest, &bytes)
|
|
||||||
.await
|
|
||||||
.map_err(|e| PluginError::FetchError {
|
|
||||||
name: name.to_string(),
|
|
||||||
url: url.to_string(),
|
|
||||||
reason: format!("Failed to write file: {}", e),
|
|
||||||
})?;
|
|
||||||
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
fn extract_archive_extension(url: &str) -> &str {
|
|
||||||
let path_part = url
|
|
||||||
.split('?')
|
|
||||||
.next()
|
|
||||||
.unwrap_or(url)
|
|
||||||
.split('#')
|
|
||||||
.next()
|
|
||||||
.unwrap_or(url);
|
|
||||||
let file_part = path_part.rsplit('/').next().unwrap_or("");
|
|
||||||
let lower = file_part.to_lowercase();
|
|
||||||
if lower.ends_with(".tar.gz") || lower.ends_with(".tgz") {
|
|
||||||
".tar.gz"
|
|
||||||
} else if lower.ends_with(".tar.zst")
|
|
||||||
|| lower.ends_with(".tar.zstd")
|
|
||||||
|| lower.ends_with(".tzst")
|
|
||||||
{
|
|
||||||
".tar.zst"
|
|
||||||
} else if lower.ends_with(".tar") {
|
|
||||||
".tar"
|
|
||||||
} else {
|
|
||||||
".unknown"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
pub async fn fetch_http_plugin(
|
|
||||||
url: &str,
|
|
||||||
name: &str,
|
|
||||||
basedir: &Path,
|
|
||||||
checksum_str: Option<&str>,
|
|
||||||
) -> Result<(), PluginError> {
|
|
||||||
let temp_dir = std::env::temp_dir().join("workshop-plugins");
|
|
||||||
tokio::fs::create_dir_all(&temp_dir).await.ok();
|
|
||||||
let ext = extract_archive_extension(url);
|
|
||||||
let temp_file = temp_dir.join(format!("{}-{}{}", name, uuid::Uuid::new_v4(), ext));
|
|
||||||
|
|
||||||
download_to_file(url, &temp_file, name).await?;
|
|
||||||
|
|
||||||
if let Some(checksum) = checksum_str
|
|
||||||
&& !checksum.is_empty()
|
|
||||||
{
|
|
||||||
let bytes = tokio::fs::read(&temp_file).await.map_err(|e| {
|
|
||||||
PluginError::GeneralError(format!("Failed to read downloaded file: {}", e))
|
|
||||||
})?;
|
|
||||||
|
|
||||||
let checksum_type = detect_checksum_type(checksum).map_err(PluginError::GeneralError)?;
|
|
||||||
|
|
||||||
if let Some(ct) = checksum_type {
|
|
||||||
verify_checksum(&bytes, checksum, ct).map_err(PluginError::GeneralError)?;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
let compression = CompressionType::from_path(&temp_file).map_err(PluginError::GeneralError)?;
|
|
||||||
let destdir = basedir.join(name);
|
|
||||||
extract_tar(&temp_file, &destdir, compression).await?;
|
|
||||||
|
|
||||||
tokio::fs::remove_file(&temp_file).await.ok();
|
|
||||||
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
@@ -1,84 +0,0 @@
|
|||||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
|
||||||
use serde::{Deserialize, Serialize};
|
|
||||||
|
|
||||||
/// Supported checksum types for verifying downloaded artifacts.
|
|
||||||
#[derive(Debug, Clone, PartialEq)]
|
|
||||||
pub enum ChecksumType {
|
|
||||||
/// SHA-256 hash (64 hex characters)
|
|
||||||
Sha256,
|
|
||||||
/// SHA-512 hash (128 hex characters)
|
|
||||||
Sha512,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Arguments for fetching a plugin artifact.
|
|
||||||
///
|
|
||||||
/// Contains optional checksum verification and shallow clone settings.
|
|
||||||
#[derive(Debug, Deserialize, Serialize, Clone, Default)]
|
|
||||||
pub struct FetchArgument {
|
|
||||||
/// SHA256 or SHA512 hex string for integrity verification.
|
|
||||||
#[serde(default)]
|
|
||||||
pub checksum: Option<String>,
|
|
||||||
/// Perform a shallow clone (single revision only).
|
|
||||||
#[serde(default)]
|
|
||||||
pub shallow: Option<bool>,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl FetchArgument {
|
|
||||||
/// Returns the checksum if present and non-empty.
|
|
||||||
pub fn get_checksum(&self) -> Option<&str> {
|
|
||||||
self.checksum.as_deref().filter(|s| !s.is_empty())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod tests {
|
|
||||||
use super::*;
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_get_checksum_some_non_empty() {
|
|
||||||
let arg = FetchArgument {
|
|
||||||
checksum: Some("abc123".to_string()),
|
|
||||||
shallow: None,
|
|
||||||
};
|
|
||||||
assert_eq!(arg.get_checksum(), Some("abc123"));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_get_checksum_empty_string() {
|
|
||||||
let arg = FetchArgument {
|
|
||||||
checksum: Some("".to_string()),
|
|
||||||
shallow: None,
|
|
||||||
};
|
|
||||||
assert_eq!(arg.get_checksum(), None);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_get_checksum_none() {
|
|
||||||
let arg = FetchArgument {
|
|
||||||
checksum: None,
|
|
||||||
shallow: None,
|
|
||||||
};
|
|
||||||
assert_eq!(arg.get_checksum(), None);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_serde_roundtrip() {
|
|
||||||
let arg = FetchArgument {
|
|
||||||
checksum: Some("deadbeef".to_string()),
|
|
||||||
shallow: Some(true),
|
|
||||||
};
|
|
||||||
let json = serde_json::to_string(&arg).unwrap();
|
|
||||||
let parsed: FetchArgument = serde_json::from_str(&json).unwrap();
|
|
||||||
assert_eq!(parsed.checksum, arg.checksum);
|
|
||||||
assert_eq!(parsed.shallow, arg.shallow);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn test_serde_roundtrip_empty() {
|
|
||||||
let arg = FetchArgument::default();
|
|
||||||
let json = serde_json::to_string(&arg).unwrap();
|
|
||||||
let parsed: FetchArgument = serde_json::from_str(&json).unwrap();
|
|
||||||
assert_eq!(parsed.checksum, arg.checksum);
|
|
||||||
assert_eq!(parsed.shallow, arg.shallow);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,4 +1,5 @@
|
|||||||
use crate::finalize::plugin::AsyncPluginFn;
|
use crate::finalize::plugin::AsyncPluginFn;
|
||||||
|
use crate::notify::types::NotificationRenderer;
|
||||||
pub mod dummy;
|
pub mod dummy;
|
||||||
mod insitenotify;
|
mod insitenotify;
|
||||||
mod mail;
|
mod mail;
|
||||||
@@ -7,6 +8,7 @@ mod webhook;
|
|||||||
pub struct InternalPlugin {
|
pub struct InternalPlugin {
|
||||||
pub name: &'static str,
|
pub name: &'static str,
|
||||||
pub entry: Box<dyn AsyncPluginFn>,
|
pub entry: Box<dyn AsyncPluginFn>,
|
||||||
|
pub renderer: NotificationRenderer,
|
||||||
}
|
}
|
||||||
|
|
||||||
pub fn get_internal_plugin() -> Vec<InternalPlugin> {
|
pub fn get_internal_plugin() -> Vec<InternalPlugin> {
|
||||||
@@ -14,18 +16,32 @@ pub fn get_internal_plugin() -> Vec<InternalPlugin> {
|
|||||||
InternalPlugin {
|
InternalPlugin {
|
||||||
name: "in-site-notify",
|
name: "in-site-notify",
|
||||||
entry: Box::new(insitenotify::InsiteNotify),
|
entry: Box::new(insitenotify::InsiteNotify),
|
||||||
|
renderer: NotificationRenderer::Plugin,
|
||||||
},
|
},
|
||||||
InternalPlugin {
|
InternalPlugin {
|
||||||
name: "mail",
|
name: "mail",
|
||||||
entry: Box::new(mail::Mail),
|
entry: Box::new(mail::Mail),
|
||||||
|
renderer: NotificationRenderer::Server,
|
||||||
},
|
},
|
||||||
InternalPlugin {
|
InternalPlugin {
|
||||||
name: "satori",
|
name: "satori",
|
||||||
entry: Box::new(satori::Satori),
|
entry: Box::new(satori::Satori),
|
||||||
|
renderer: NotificationRenderer::Server,
|
||||||
},
|
},
|
||||||
InternalPlugin {
|
InternalPlugin {
|
||||||
name: "webhook",
|
name: "webhook",
|
||||||
entry: Box::new(webhook::Webhook),
|
entry: Box::new(webhook::Webhook),
|
||||||
|
renderer: NotificationRenderer::Server,
|
||||||
},
|
},
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Returns dummy only (for `--debug notify.dummy` mode).
|
||||||
|
/// All real adapters and in-site-notify are excluded.
|
||||||
|
pub fn get_debug_plugin() -> Vec<InternalPlugin> {
|
||||||
|
vec![InternalPlugin {
|
||||||
|
name: "dummy",
|
||||||
|
entry: Box::new(dummy::Dummy),
|
||||||
|
renderer: NotificationRenderer::Plugin,
|
||||||
|
}]
|
||||||
|
}
|
||||||
|
|||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user