Compare commits
17 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Catty Steve
|
fdc00f0635 | ||
|
Catty Steve
|
26809df720 | ||
|
Catty Steve
|
3629c33fc4 | ||
|
Catty Steve
|
6d9ddbca56 | ||
|
Catty Steve
|
9ff9073fce | ||
|
Catty Steve
|
df5026cdbe | ||
|
Catty Steve
|
d1ad08ef3a | ||
|
Catty Steve
|
cf2968e720 | ||
|
Catty Steve
|
28abc5d207 | ||
|
Catty Steve
|
6e7b96cd66 | ||
|
Catty Steve
|
3e80825d58 | ||
|
Catty Steve
|
c97eafab29 | ||
|
Catty Steve
|
f737b6a24d | ||
|
Catty Steve
|
681b7d1333 | ||
|
Catty Steve
|
f07588a716 | ||
|
Catty Steve
|
b0530fb858 | ||
|
Catty Steve
|
fd18b547e5 |
+3
-1
@@ -9,7 +9,7 @@ archived
|
||||
*.bak
|
||||
*.clean
|
||||
session*
|
||||
llm.env
|
||||
.secret
|
||||
|
||||
# System test artifacts
|
||||
tests/results/
|
||||
@@ -18,3 +18,5 @@ tests/**/__pycache__/
|
||||
tests/.pytest_cache/
|
||||
tests/llm/.cache/
|
||||
*.pyc
|
||||
.staging
|
||||
.pytest_cache
|
||||
|
||||
@@ -1,108 +1,127 @@
|
||||
# PROJECT KNOWLEDGE BASE
|
||||
|
||||
**Generated:** 2026-04-22
|
||||
**Commit:** 7b3b71a
|
||||
**Branch:** master
|
||||
**Generated:** 2026-06-12T16:08:15Z
|
||||
**Commit:** 3629c33
|
||||
**Branch:** feat/bake-dag-tester
|
||||
|
||||
## OVERVIEW
|
||||
HoneyBiscuitWorkshop is a Rust-based CI/CD system ("蜜饼工坊") with LLM-powered pipeline auto-configuration. 9 independent Cargo crates, no workspace.
|
||||
HoneyBiscuitWorkshop (蜜饼工坊) — Rust-based CI/CD system inspired by Concourse CI. Pipeline: prebake.yml → bake.sh → finalize.yml. Multi-builder (Docker, Firecracker, bare metal). Plugin system.
|
||||
|
||||
## STRUCTURE
|
||||
```
|
||||
./
|
||||
├── workshop-baker/ # Main orchestrator (CLI + daemon)
|
||||
├── workshop-agent/ # HTTP agent server
|
||||
├── workshop-pipeline/ # Config library (prebake/finalize)
|
||||
├── workshop-llm-detector/ # LLM repo analyzer
|
||||
├── workshop-vault/ # Secret management client
|
||||
├── workshop-cert/ # TLS certificate generator
|
||||
├── workshop-deviceid/ # Device ID library
|
||||
├── workshop-builder-native/ # Stub (unused)
|
||||
├── workshop-helper-mac/ # MAC address utility
|
||||
├── docs/ # mdBook documentation (zh-CN)
|
||||
├── playground/ # Experimentation
|
||||
└── third_party/ # Empty
|
||||
├── workshop-baker/ # Main binary + lib — orchestrator, CLI, plugins
|
||||
├── workshop-engine/ # Core execution engine — process mgmt, package managers
|
||||
├── workshop-schedule/ # DAG scheduling library (petgraph-based)
|
||||
├── workshop-getterurl/ # URL fetching with hash verification
|
||||
├── workshop-baker-params/ # Config/parameter type definitions
|
||||
├── docs/ # mdBook documentation (Chinese, 3 volumes)
|
||||
├── templates/ # Pipeline config templates (prebake/bake/finalize)
|
||||
└── .github/workflows/ # CI (tests workshop-baker + workshop-engine)
|
||||
```
|
||||
**No root Cargo workspace** — crates linked by path deps. Build per-crate.
|
||||
|
||||
## WHERE TO LOOK
|
||||
|
||||
| Task | Location | Notes |
|
||||
|------|----------|-------|
|
||||
| Baker (main) | `workshop-baker/src/{bake,prebake,engine}/` | Core pipeline logic |
|
||||
| Agent server | `workshop-agent/src/` | Actix-web HTTP + TLS |
|
||||
| Pipeline config | `workshop-pipeline/src/config/` | prebake/finalize YAML |
|
||||
| LLM integration | `workshop-llm-detector/src/llm/` | Ollama/OpenAI clients |
|
||||
| Secrets | `workshop-vault/src/` | Vault client wrapper |
|
||||
| CLI definition | `workshop-baker/src/lib.rs` | Clap Cli/Commands/BareCommands |
|
||||
| Pipeline orchestration | `workshop-baker/src/main.rs` | prebake→bake→finalize→notify flow |
|
||||
| Decorator parsing | `workshop-baker/src/bake/decorator.rs` | `# @decorator(args)` syntax |
|
||||
| Build execution | `workshop-baker/src/bake/execute.rs` | Script execution with timeout/retry |
|
||||
| Docker integration | `workshop-baker/src/prebake/env/container.rs` | Bollard-based |
|
||||
| Package managers | `workshop-engine/src/pm/` | apt, pacman, dnf, apk |
|
||||
| User package managers | `workshop-engine/src/upm/` | rustup, uv, go |
|
||||
| Notification system | `workshop-baker/src/notify/` | Email, webhook, IM |
|
||||
| Plugin system | `workshop-baker/src/finalize/plugin/` | shell, dylib, rhai, internal |
|
||||
| Error types | `workshop-baker/src/error.rs` | CliError + HasExitCode |
|
||||
| Exit codes | `workshop-engine/src/error.rs` | EXITCODE_* constants |
|
||||
| DAG scheduling | `workshop-schedule/src/dag.rs` | Topological sort, batch picking |
|
||||
| Pipeline templates | `templates/` | Reference config DSL |
|
||||
|
||||
## ENTRY POINTS
|
||||
|
||||
| Entry | Type | Path | Role |
|
||||
|-------|------|------|------|
|
||||
| `workshop-baker` | Rust bin | `workshop-baker/src/main.rs` | Primary CLI — dispatch: default (full pipeline), bare (single stage), daemon (unimplemented) |
|
||||
| `workshop_baker` | Rust lib | `workshop-baker/src/lib.rs` | Baker library + CLI structs |
|
||||
| `workshop_engine` | Rust lib | `workshop-engine/src/lib.rs` | Core engine — ExecutionContext, EventSender, Executor |
|
||||
| `workshop_schedule` | Rust lib | `workshop-schedule/src/lib.rs` | Dag, DagNode, EdgeKind |
|
||||
| `workshop_baker_params` | Rust lib | `workshop-baker-params/src/lib.rs` | Layered config: defaults → base → courier |
|
||||
| `workshop_getterurl` | Rust lib | `workshop-getterurl/src/lib.rs` | GetterUrl, fetch() — http/git/file |
|
||||
| docs | mdBook | `docs/book.toml` | Project documentation site |
|
||||
|
||||
## CODE MAP
|
||||
| Symbol | Type | Location | Role |
|
||||
|--------|------|----------|------|
|
||||
| PipelineConfig | struct | workshop-pipeline/src/lib.rs:27 | Full pipeline config |
|
||||
| CICDLLMHelper | struct | workshop-llm-detector/src/lib.rs:16 | LLM analysis entry |
|
||||
| Engine | struct | workshop-baker/src/engine.rs | Build executor |
|
||||
| VaultClient | struct | workshop-vault/src/client.rs | Secret ops |
|
||||
|
||||
### Crate Dependency Graph
|
||||
```
|
||||
workshop-baker (bin)
|
||||
├── workshop-engine
|
||||
├── workshop-schedule
|
||||
├── workshop-baker-params
|
||||
└── workshop-getterurl
|
||||
```
|
||||
|
||||
### workshop-baker Module Map
|
||||
| Module | Submodules | Role |
|
||||
|--------|-----------|------|
|
||||
| `bake` | decorator, execute, parser, schedule, builder, trivial, util, constant, error | Script parsing + execution |
|
||||
| `prebake` | config, stage (bootstrap, depssystem, depsuser, environment, hook), env (container, firecracker, baremetal, custom), security, event, types, constant, error | Environment setup |
|
||||
| `finalize` | plugin (shell, dylib, rhai, internal), stage, template, types, config, constant, error, event | Post-build hooks |
|
||||
| `notify` | handler, queue, rule, template, types, util, error | Async notification dispatch |
|
||||
| `bare` | — | Standalone single-stage CLI mode |
|
||||
| `types` | resource, buildstatus | Shared type definitions |
|
||||
| `utils` | — | Cross-cutting utilities |
|
||||
|
||||
## CONVENTIONS
|
||||
|
||||
### Rust
|
||||
- **Edition**: 2024 (non-standard, requires Rust 1.85+)
|
||||
- Deprecate `mod.rs`
|
||||
- **Naming**: `snake_case` files/modules, `PascalCase` types, `SCREAMING_SNAKE_CASE` consts
|
||||
- **Error handling**: `thiserror` + `anyhow` combo
|
||||
- **Async**: `#[tokio::main]` + `tokio` with "full" features
|
||||
- **Imports**: `std` → `external_crate` → `crate::module`
|
||||
|
||||
### Testing
|
||||
- **Rust**: `#[test]` inline, `#[tokio::test]` async, `tests/*.rs` integration
|
||||
- **Python**: `pytest` with `tests/integration/test_*.py` (Docker-based)
|
||||
- **Benchmarks**: Criterion with `harness = false`
|
||||
|
||||
### Build Pipeline
|
||||
- **prebake.yml**: Environment setup (docker/firecracker/baremetal)
|
||||
- **bake.sh**: Build execution with `# @pipeline` decorators
|
||||
- **finalize.yml**: Packaging, deployment, notifications
|
||||
- **Annotations**: `@timeout()`, `@retry()`, `@parallel`, `@fallible`
|
||||
- **Edition**: Rust 2024 (requires Rust 1.85+)
|
||||
- **Formatting**: Default `rustfmt` — no `rustfmt.toml` / `.editorconfig`
|
||||
- **Lint overrides** (workshop-baker only): `dead_code = "allow"`, `unreachable_code = "allow"`, `inherent_to_string = "allow"`, `non_canonical_partial_ord_impl = "allow"`
|
||||
- **Import order**: `std` → external crates → `crate::` (blank line between groups)
|
||||
- **Error handling**: `thiserror` for libraries, `anyhow` for binaries. Implement `HasExitCode` for exit code mapping.
|
||||
- **Type system**: Newtype pattern preferred (e.g., `MemSize(u64)`). Shared types in `types/` module. `types/` must NOT depend on `bake/`, `engine/`, `prebake/`, `finalize/`.
|
||||
- **Tests**: Inline `#[cfg(test)] mod tests` at bottom of source files. Async: `#[tokio::test]`. CI runs `cargo test --all-features`.
|
||||
- **Commits**: `type(scope): description` — `feat|fix|docs|test|refactor|style`
|
||||
- **AI annotation**: `// Code in this module PARTIALLY or FULLY utilized AI Coding Agent` — never delete if present
|
||||
- **Docs language**: Chinese (zh-CN), mdBook
|
||||
- **No `no_std`**: Explicitly unsupported
|
||||
|
||||
## ANTI-PATTERNS (THIS PROJECT)
|
||||
|
||||
1. **Python in Rust project** — workshop-baker has Python tests (pytest.ini, test_prebake.py)
|
||||
2. **temp/ directory** — Non-standard in Rust projects
|
||||
3. **Certificates in source** — artifact workshop-cert/certs/ not excluded
|
||||
4. **Deprecated projects** — workshop-executor.old, workshop-monitor.old still present
|
||||
5. **Hardcoded mirrors** — Tsinghua University mirrors in configs
|
||||
6. **Rust 2024 edition** — May not work with stable toolchains
|
||||
|
||||
## UNIQUE STYLES
|
||||
|
||||
- **Pipeline DSL**: Shell scripts with `# @decorator` annotations
|
||||
- **Dual-mode baker**: CLI commands + daemon mode
|
||||
- **LLM cookbook system**: YAML cookbooks for language-specific builds
|
||||
- **Chinese docs** — Documentation primarily in zh-CN
|
||||
- **User "vulcan"** — Custom build user (not root/runner)
|
||||
- **Do NOT derive production behavior from bare mode** — bare mode is dev-only, ctx.privileged is fake
|
||||
- **Do NOT use `detect()`/`adopt()` PM methods in production** — PM selection must come from env config
|
||||
- **`drop_after < DepsUser` is FORBIDDEN** — will break system dependency installation (needs root)
|
||||
- **Do NOT add magic URL inference** — URL is URL, behavior bound explicitly via `getter::` prefix
|
||||
- **Do NOT use `@` for version locking in URLs** — it's an auth delimiter; use `?ref=`
|
||||
- **Do NOT duplicate type definitions across modules** — single source of truth in `types/`
|
||||
- **Careful with `unimplemented!()`** — Daemon mode, Rhai/Dylib plugins, and Custom builder are stubs
|
||||
- **Security**: PIPELINE_CONTAINER_PRIVILEGED, PIPELINE_BAREMETAL_ELEVATE, PIPELINE_UNCOVER_SECRET require explicit approval + audit
|
||||
|
||||
## COMMANDS
|
||||
|
||||
```bash
|
||||
# Build
|
||||
cargo build --release -p workshop-baker
|
||||
# Build (per crate, no workspace)
|
||||
cd workshop-baker && cargo build --all-features
|
||||
|
||||
# Test
|
||||
cargo test -p workshop-baker
|
||||
pytest tests/integration/test_prebake.py -v
|
||||
# Test (per crate)
|
||||
cd workshop-baker && cargo test --all-features
|
||||
cd workshop-engine && cargo test --all-features
|
||||
|
||||
# Lint
|
||||
cargo clippy -- -D warnings
|
||||
cargo fmt --all
|
||||
# Lint (CI gate)
|
||||
cargo fmt -- --check
|
||||
cargo clippy --all-features -- -D warnings
|
||||
|
||||
# Run baker
|
||||
cargo run --bin workshop-baker -- prebake config.yml
|
||||
cargo run --bin workshop-baker -- bake script.sh
|
||||
|
||||
# Run agent
|
||||
cargo run --bin workshop-agent
|
||||
# Docs
|
||||
cd docs && mdbook build # output in docs/book/
|
||||
cd docs && mdbook serve # local preview at localhost:3000
|
||||
```
|
||||
|
||||
## NOTES
|
||||
|
||||
- **Docker required** for integration tests
|
||||
- **workshop-builder-native** is a stub (3-line Hello World)
|
||||
- No root workspace Cargo.toml — each crate is independent
|
||||
- `AGENTS.md.old` at root — legacy documentation
|
||||
- **No workspace Cargo.toml** at root — each crate has its own `Cargo.lock`. Build individually.
|
||||
- **CI only tests** `workshop-baker` and `workshop-engine`; smaller crates tested transitively.
|
||||
- **`quicktest.sh`** in workshop-baker/ is a dev smoke-test script (Docker-based, not CI).
|
||||
- **Git submodules**: `rust-tongsuo`, `RustyVault` (crypto — not yet integrated into build).
|
||||
- **`.secret/`** contains real credentials — never commit, never read into context.
|
||||
- See `docs/src/zh-CN/vol3_dev/` for full developer documentation (architecture, design decisions, code style).
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
# DOCS KNOWLEDGE BASE
|
||||
|
||||
## OVERVIEW
|
||||
mdBook project in zh-CN. Authoritative source for project conventions, coding style, and architecture docs. Build output at `docs/book/` (gitignored).
|
||||
|
||||
## STRUCTURE
|
||||
```
|
||||
docs/
|
||||
├── book.toml # mdBook config (title, language=zh-CN)
|
||||
├── src/SUMMARY.md # Sidebar nav — source of truth for page listing
|
||||
└── src/zh-CN/
|
||||
├── index.md # Landing
|
||||
├── vol1_user/ # End-user documentation
|
||||
├── vol2_admin/ # Admin reference: decorator cheatsheet, deployment,
|
||||
│ # env vars, exit codes, plugin ref, security, troubleshooting
|
||||
└── vol3_dev/ # Developer docs: architecture, code style, commit
|
||||
# convention, testing, error handling, type system,
|
||||
# design decisions, PR process
|
||||
```
|
||||
|
||||
## WHERE TO LOOK
|
||||
|
||||
| Task | Location | Notes |
|
||||
|------|----------|-------|
|
||||
| Sidebar/nav editing | `src/SUMMARY.md` | Add entry here to surface a new page |
|
||||
| Coding conventions | `vol3_dev/code_style.md` | Naming, imports, formatting |
|
||||
| Commit rules | `vol3_dev/commit_convention.md` | feat/fix/docs/test/refactor/style |
|
||||
| Error handling | `vol3_dev/error_handling_patterns.md` | thiserror+anyhow+HasExitCode |
|
||||
| Type system | `vol3_dev/type_system_guide.md` | Newtypes, types/ module isolation |
|
||||
| Testing strategy | `vol3_dev/testing_strategy.md` | 3 tiers: unit, integration, system |
|
||||
| Anti-patterns | `vol3_dev/design_decisions.md` | Documented design rejections |
|
||||
| Architecture | `vol3_dev/architecture.md` | Crate dependency graph, module map |
|
||||
|
||||
## CONVENTIONS
|
||||
|
||||
- **Language**: All prose in zh-CN. Code blocks in Rust (default) or bash.
|
||||
- **Adding a page**: Create `.md` under `volN_*/`, then add to `SUMMARY.md`.
|
||||
- **Formatting**: Standard Markdown. mdBook preprocessors handle TOC and links.
|
||||
- **Cross-references**: Relative paths from source root (e.g., `../vol3_dev/code_style.md`).
|
||||
- **Code blocks**: Fenced with language tag. No line numbers in docs.
|
||||
- **File naming**: `snake_case.md`.
|
||||
|
||||
## COMMANDS
|
||||
|
||||
```bash
|
||||
mdbook build # static site → docs/book/
|
||||
mdbook serve # live preview at localhost:3000
|
||||
```
|
||||
@@ -35,3 +35,4 @@
|
||||
- [自定义 Decorator](zh-CN/vol3_dev/custom_decorator.md)
|
||||
- [模板变量](zh-CN/vol3_dev/template_variables.md)
|
||||
- [邮件模板](zh-CN/vol3_dev/mail_template.md)
|
||||
- [Resource 模型](zh-CN/vol3_dev/resource_model.md)
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
# Resource 模型
|
||||
|
||||
## 核心思想
|
||||
|
||||
整个系统只关心两件事：东西从哪来（fetch），东西到哪去（publish）。
|
||||
|
||||
fetch 方向：URL → 本地路径 → 使用。
|
||||
publish 方向：本地路径 → 变换（插件）→ URL。
|
||||
|
||||
两个方向共用同一套寻址方式，但 Resource 中间态只存在于 fetch 方向。
|
||||
|
||||
## URL 格式
|
||||
|
||||
```
|
||||
getter::url?param1=value1¶m2=value2
|
||||
```
|
||||
|
||||
getter 显式声明意图。不是"我猜这是个 git 仓库"，而是"使用者告诉我这是什么"。
|
||||
|
||||
只有三个 getter：
|
||||
|
||||
- `git` — 克隆仓库，必要时 checkout 指定版本
|
||||
- `http` / `https` — 下载文件，必要时校验 checksum
|
||||
- `file` — 复制本地文件或目录
|
||||
|
||||
版本锁定不用 `@v1.0`（`@` 在 URL 里是认证分隔符），用 `?ref=v1.0`。
|
||||
|
||||
没有 `github.com/user/repo` 看起来应该 clone 它这种黑魔法。
|
||||
没有 `.tar.gz` 看起来应该解压它的惊喜。
|
||||
URL 是 URL，行为是行为，两者用 getter:: 显式绑定。
|
||||
|
||||
## Resource 不是网络地址
|
||||
|
||||
流水线阶段（prebake / finalize / notify）只接触 Resource：一个名字 + 一个本地路径，已就绪。它们不经由网络、不解析 URL、不碰 getter。
|
||||
|
||||
所有 fetch 行为在阶段启动前完成：
|
||||
|
||||
- worker 模式：bakerd fetch → 填入 ResourceRegistry → 启动 baker
|
||||
- standalone 模式：baker 自己 fetch → 填入 ResourceRegistry → 执行阶段
|
||||
- bare 模式：不 fetch，传入的必须是 Resource
|
||||
|
||||
ResourceRegistry 是名字到路径的映射。填进去就只读了。
|
||||
|
||||
## getterurl 是独立的 crate
|
||||
|
||||
URL 格式解析 + 基本 fetch 放在 workshop-getterurl，和 baker 本身没有耦合。任何人都可以用。
|
||||
|
||||
crate 分两层：
|
||||
|
||||
```
|
||||
结构层（parser）：
|
||||
"git::https://host/repo.git?ref=v1.0" → { getter: "git", url: Url, params: Params }
|
||||
|
||||
获取层（builtin getters）：
|
||||
getter.fetch(url, params, dst) → local_path
|
||||
git: git clone + optional checkout
|
||||
http: HTTP GET + optional checksum
|
||||
file: std::fs::copy
|
||||
```
|
||||
|
||||
获取层是字面意义的——不解压、不推断、不猜意图。
|
||||
|
||||
## Checksum
|
||||
|
||||
checksum 作为 URL query 参数传递。三态语义：
|
||||
|
||||
- `?checksum=sha256:abc` — 必须匹配
|
||||
- `?checksum=` — 显式禁用校验
|
||||
- 无 checksum — 如果服务端返回 Content-Digest 头，校验它
|
||||
|
||||
getter 不主动校验，它只执行 URL 里指定的操作。
|
||||
|
||||
## Artifact 不经过 Resource
|
||||
|
||||
artifact 是 bake 的产物，自产生就开始被使用，不存在中间态。它的路径就是 bake 产出的本地路径，不走 ResourceRegistry。
|
||||
|
||||
artifact 的 publish 方向由插件定义行为——可能上传、可能扫描、可能只记录。插件定义最终的行为，只要是自洽的。
|
||||
|
||||
## 三种运行模式
|
||||
|
||||
```
|
||||
fetch 执行者 fetch 在阶段内
|
||||
daemon 模式 bakerd 否
|
||||
standalone baker 自身 否（阶段启动前已完成）
|
||||
bare 模式 无 否（必须传入 Resource）
|
||||
```
|
||||
|
||||
bare 模式下传入的必须是已经就绪的 Resource。外部 URL 是 daemon 或 standalone 的事，bare 不负责。
|
||||
@@ -1,101 +0,0 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
# HoneyBiscuitWorkshop System Test Runner
|
||||
#
|
||||
# Usage:
|
||||
# ./runner.sh # Arch baker tests (hbw-system-test:latest)
|
||||
# ./runner.sh --image hbw-test-pm:test-apt # Debian engine PM tests
|
||||
# ./runner.sh -k test_install # Arch baker, filtered
|
||||
|
||||
MODE="auto"
|
||||
TEST_IMAGE=""
|
||||
TEST_DIR="tests/baker"
|
||||
PYTEST_ARGS=""
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--image) TEST_IMAGE="$2"; TEST_DIR="tests/engine"; shift 2 ;;
|
||||
manual) MODE="manual"; shift ;;
|
||||
auto) MODE="auto"; shift ;;
|
||||
--) shift; PYTEST_ARGS="$@"; break ;;
|
||||
*) PYTEST_ARGS="$PYTEST_ARGS $1"; shift ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Default image: Arch-based baker test image
|
||||
: ${TEST_IMAGE:="hbw-system-test:latest"}
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
CONTAINER_NAME="hbw-system-test-container"
|
||||
STAGING_DIR="/tmp/hbw-system-test"
|
||||
|
||||
cleanup() {
|
||||
docker rm -f "$CONTAINER_NAME" 2>/dev/null || true
|
||||
[ "$TEST_DIR" = "tests/baker" ] && rm -rf "$STAGING_DIR"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
build_binary() {
|
||||
local target="x86_64-unknown-linux-musl"
|
||||
local binary="$SCRIPT_DIR/workshop-baker/target/$target/debug/workshop-baker"
|
||||
if [ ! -f "$binary" ]; then
|
||||
echo "[runner] Building static (musl) baker binary..."
|
||||
cd "$SCRIPT_DIR/workshop-baker"
|
||||
cargo build -p workshop-baker --target "$target" 2>&1 | tail -n 1
|
||||
fi
|
||||
}
|
||||
|
||||
prepare_staging() {
|
||||
# Only needed for baker tests (hbw-system-test:latest mounts /workshop)
|
||||
if [ "$TEST_DIR" = "tests/baker" ]; then
|
||||
mkdir -p "$STAGING_DIR"
|
||||
cp "$SCRIPT_DIR/workshop-baker/target/x86_64-unknown-linux-musl/debug/workshop-baker" "$STAGING_DIR/"
|
||||
chmod +x "$STAGING_DIR/workshop-baker"
|
||||
fi
|
||||
}
|
||||
|
||||
run_container() {
|
||||
local tag="${TEST_IMAGE##*:}"
|
||||
local filter=""
|
||||
case "$tag" in
|
||||
test-arch) filter="-k pacman" ;;
|
||||
test-apt) filter="-k apt" ;;
|
||||
test-dnf) filter="-k dnf" ;;
|
||||
test-apk) filter="-k apk" ;;
|
||||
esac
|
||||
|
||||
echo "[runner] Starting: $TEST_IMAGE (tests=$TEST_DIR${filter:+, filter=$filter})"
|
||||
mkdir -p "$SCRIPT_DIR/tests/results"
|
||||
|
||||
local mounts="-v $SCRIPT_DIR:/src:ro"
|
||||
local entrypoint=""
|
||||
|
||||
if [ "$TEST_DIR" = "tests/baker" ]; then
|
||||
mounts="$mounts -v $STAGING_DIR:/workshop:ro"
|
||||
entrypoint='--entrypoint ""'
|
||||
fi
|
||||
|
||||
local cmd="sh -c \"python3 -m venv /tmp/v && . /tmp/v/bin/activate && \
|
||||
pip install pytest pytest-timeout -q && \
|
||||
exec python -m pytest $TEST_DIR $PYTEST_ARGS $filter\""
|
||||
|
||||
if [ "$MODE" = "manual" ]; then
|
||||
docker run -it --rm --name "$CONTAINER_NAME" --privileged $entrypoint \
|
||||
$mounts "$TEST_IMAGE"
|
||||
else
|
||||
docker run --rm --name "$CONTAINER_NAME" --privileged $entrypoint \
|
||||
$mounts -w /src \
|
||||
"$TEST_IMAGE" "$cmd"
|
||||
fi
|
||||
}
|
||||
|
||||
echo "============================================"
|
||||
echo " HBW System Test Runner"
|
||||
echo " Image: $TEST_IMAGE"
|
||||
echo " Tests: $TEST_DIR"
|
||||
echo "============================================"
|
||||
build_binary
|
||||
prepare_staging
|
||||
run_container
|
||||
echo "[runner] Done."
|
||||
@@ -1,32 +0,0 @@
|
||||
FROM archlinux:latest
|
||||
|
||||
RUN sed -i 's|https://geo.mirror.pkgbuild.com|https://mirrors.tuna.tsinghua.edu.cn/archlinux|g' /etc/pacman.d/mirrorlist && \
|
||||
sed -i 's|https://mirror.rackspace.com/archlinux|https://mirrors.tuna.tsinghua.edu.cn/archlinux|g' /etc/pacman.d/mirrorlist && \
|
||||
echo 'Server = https://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch' >> /etc/pacman.d/mirrorlist
|
||||
|
||||
RUN pacman -Syu --noconfirm && \
|
||||
pacman -S --noconfirm \
|
||||
python python-pip base-devel git curl wget sudo \
|
||||
which procps-ng shadow && \
|
||||
pacman -Scc --noconfirm
|
||||
|
||||
# Create test user
|
||||
RUN useradd -m -s /bin/bash testuser
|
||||
|
||||
# Python environment
|
||||
WORKDIR /app
|
||||
COPY tests/requirements.txt ./requirements.txt
|
||||
RUN python -m venv /opt/test-venv && \
|
||||
/opt/test-venv/bin/pip install -r requirements.txt
|
||||
|
||||
# Copy test suite
|
||||
COPY tests/ ./tests/
|
||||
COPY tests/entrypoint.sh ./entrypoint.sh
|
||||
RUN chmod +x entrypoint.sh
|
||||
|
||||
# Environment
|
||||
ENV PATH="/opt/test-venv/bin:$PATH"
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
ENV PYTHONDONTWRITEBYTECODE=1
|
||||
|
||||
ENTRYPOINT ["./entrypoint.sh"]
|
||||
@@ -1,129 +0,0 @@
|
||||
import os
|
||||
import pytest
|
||||
|
||||
|
||||
FIXTURES = os.path.join(os.path.dirname(__file__), "fixtures", "bake")
|
||||
|
||||
|
||||
@pytest.mark.bake
|
||||
class TestBakeSimpleScript:
|
||||
|
||||
def test_simple_script_runs(self, run_baker, bake_workspace, work_dir):
|
||||
prebake_cfg = os.path.join(
|
||||
os.path.dirname(__file__), "fixtures", "prebake", "minimal.yml"
|
||||
)
|
||||
script = os.path.join(FIXTURES, "simple.sh")
|
||||
bake_base = os.path.join(FIXTURES, "bake_base.sh")
|
||||
|
||||
result = run_baker(
|
||||
"bake", script,
|
||||
"--bake-base", bake_base,
|
||||
"--prebake", prebake_cfg,
|
||||
cwd=work_dir,
|
||||
timeout=60,
|
||||
)
|
||||
assert result["success"], f"Bake failed: {result['stderr']}"
|
||||
assert "Hello from simple bake script" in result["stdout"]
|
||||
|
||||
def test_simple_script_sets_env_vars(self, run_baker, bake_workspace, work_dir):
|
||||
prebake_cfg = os.path.join(
|
||||
os.path.dirname(__file__), "fixtures", "prebake", "minimal.yml"
|
||||
)
|
||||
script = os.path.join(FIXTURES, "simple.sh")
|
||||
bake_base = os.path.join(FIXTURES, "bake_base.sh")
|
||||
|
||||
result = run_baker(
|
||||
"bake", script,
|
||||
"--bake-base", bake_base,
|
||||
"--prebake", prebake_cfg,
|
||||
cwd=work_dir,
|
||||
timeout=60,
|
||||
)
|
||||
assert result["success"]
|
||||
assert "test-pipeline" in result["stdout"]
|
||||
|
||||
|
||||
@pytest.mark.bake
|
||||
class TestBakePipeline:
|
||||
|
||||
def test_pipeline_functions_execute(self, run_baker, bake_workspace, work_dir):
|
||||
prebake_cfg = os.path.join(
|
||||
os.path.dirname(__file__), "fixtures", "prebake", "minimal.yml"
|
||||
)
|
||||
script = os.path.join(FIXTURES, "pipeline.sh")
|
||||
bake_base = os.path.join(FIXTURES, "bake_base.sh")
|
||||
|
||||
result = run_baker(
|
||||
"bake", script,
|
||||
"--bake-base", bake_base,
|
||||
"--prebake", prebake_cfg,
|
||||
cwd=work_dir,
|
||||
timeout=60,
|
||||
)
|
||||
assert result["success"], f"Pipeline bake failed: {result['stderr']}"
|
||||
assert os.path.exists("/tmp/bake-test-output/step1.txt")
|
||||
assert os.path.exists("/tmp/bake-test-output/step2.txt")
|
||||
|
||||
def test_pipeline_step_outputs_correct(self, run_baker, bake_workspace, work_dir):
|
||||
prebake_cfg = os.path.join(
|
||||
os.path.dirname(__file__), "fixtures", "prebake", "minimal.yml"
|
||||
)
|
||||
script = os.path.join(FIXTURES, "pipeline.sh")
|
||||
bake_base = os.path.join(FIXTURES, "bake_base.sh")
|
||||
|
||||
result = run_baker(
|
||||
"bake", script,
|
||||
"--bake-base", bake_base,
|
||||
"--prebake", prebake_cfg,
|
||||
cwd=work_dir,
|
||||
timeout=60,
|
||||
)
|
||||
assert result["success"]
|
||||
with open("/tmp/bake-test-output/step1.txt") as f:
|
||||
assert f.read().strip() == "step1-done"
|
||||
with open("/tmp/bake-test-output/step2.txt") as f:
|
||||
assert f.read().strip() == "step2-done"
|
||||
|
||||
|
||||
@pytest.mark.bake
|
||||
class TestBakeAfterDependencies:
|
||||
|
||||
def test_after_ordering(self, run_baker, bake_workspace, work_dir):
|
||||
prebake_cfg = os.path.join(
|
||||
os.path.dirname(__file__), "fixtures", "prebake", "minimal.yml"
|
||||
)
|
||||
script = os.path.join(FIXTURES, "with_after.sh")
|
||||
bake_base = os.path.join(FIXTURES, "bake_base.sh")
|
||||
|
||||
result = run_baker(
|
||||
"bake", script,
|
||||
"--bake-base", bake_base,
|
||||
"--prebake", prebake_cfg,
|
||||
cwd=work_dir,
|
||||
timeout=60,
|
||||
)
|
||||
assert result["success"], f"@after bake failed: {result['stderr']}"
|
||||
assert os.path.exists("/tmp/bake-after-test/status.txt")
|
||||
assert os.path.exists("/tmp/bake-after-test/source.txt")
|
||||
assert os.path.exists("/tmp/bake-after-test/build.txt")
|
||||
|
||||
|
||||
@pytest.mark.bake
|
||||
class TestBakeStatusFiles:
|
||||
|
||||
def test_status_written_after_bake(self, run_baker, bake_workspace, work_dir):
|
||||
prebake_cfg = os.path.join(
|
||||
os.path.dirname(__file__), "fixtures", "prebake", "minimal.yml"
|
||||
)
|
||||
script = os.path.join(FIXTURES, "simple.sh")
|
||||
bake_base = os.path.join(FIXTURES, "bake_base.sh")
|
||||
|
||||
result = run_baker(
|
||||
"bake", script,
|
||||
"--bake-base", bake_base,
|
||||
"--prebake", prebake_cfg,
|
||||
cwd=work_dir,
|
||||
timeout=60,
|
||||
)
|
||||
assert result["success"]
|
||||
assert os.path.exists("/tmp/.hbwstatus")
|
||||
@@ -1,77 +0,0 @@
|
||||
import os
|
||||
import pytest
|
||||
|
||||
from probes import file
|
||||
from llm import judge_log, LLMClient
|
||||
|
||||
|
||||
FIXTURES = os.path.join(os.path.dirname(__file__), "fixtures", "finalize")
|
||||
|
||||
|
||||
@pytest.mark.finalize
|
||||
class TestFinalizeMinimal:
|
||||
|
||||
def test_minimal_finalize_succeeds(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "minimal.yml")
|
||||
result = run_baker("finalize", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"], f"Minimal finalize failed: {result['stderr']}"
|
||||
|
||||
|
||||
@pytest.mark.finalize
|
||||
class TestFinalizeHooks:
|
||||
|
||||
def test_early_hook_executes(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "with_hooks.yml")
|
||||
result = run_baker("finalize", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
assert "finalize early hook" in result["stdout"] or "finalize early hook" in result["stderr"]
|
||||
|
||||
def test_late_hook_executes(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "with_hooks.yml")
|
||||
result = run_baker("finalize", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
assert "finalize late hook" in result["stdout"] or "finalize late hook" in result["stderr"]
|
||||
|
||||
|
||||
@pytest.mark.finalize
|
||||
class TestFinalizeStatus:
|
||||
|
||||
def test_finalize_writes_status(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "minimal.yml")
|
||||
result = run_baker("finalize", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
assert os.path.exists("/tmp/.hbwstatus")
|
||||
|
||||
|
||||
@pytest.mark.finalize
|
||||
@pytest.mark.llm
|
||||
class TestFinalizeWithLLM:
|
||||
|
||||
def test_finalize_hooks_via_llm(self, run_baker, llm_env, work_dir):
|
||||
if not llm_env:
|
||||
pytest.skip("LLM environment not configured")
|
||||
|
||||
config = os.path.join(FIXTURES, "with_hooks.yml")
|
||||
result = run_baker("finalize", config, cwd=work_dir, timeout=60)
|
||||
|
||||
combined_log = result["stdout"] + "\n" + result["stderr"]
|
||||
|
||||
judgment = judge_log(
|
||||
operation="finalize with early and late hooks",
|
||||
expected_behavior="Both early and late hooks executed successfully, "
|
||||
"printing 'finalize early hook' and 'finalize late hook'",
|
||||
actual_log=combined_log,
|
||||
)
|
||||
assert judgment.passed, f"LLM judgment failed: {judgment.reason}"
|
||||
|
||||
|
||||
@pytest.mark.finalize
|
||||
class TestFinalizePlugins:
|
||||
|
||||
@pytest.mark.xfail(reason="Plugin download requires network/git access", strict=False)
|
||||
def test_plugin_download(self, run_baker, work_dir):
|
||||
pass
|
||||
|
||||
@pytest.mark.xfail(reason="Artifact stage not yet implemented", strict=False)
|
||||
def test_artifact_collection(self, run_baker, work_dir):
|
||||
pass
|
||||
@@ -1,261 +0,0 @@
|
||||
import os
|
||||
import json
|
||||
import pytest
|
||||
|
||||
from probes import user, file, package
|
||||
|
||||
|
||||
FIXTURES = os.path.join(os.path.dirname(__file__), "fixtures", "prebake")
|
||||
|
||||
|
||||
@pytest.mark.prebake
|
||||
class TestPrebakeConfigValidation:
|
||||
|
||||
def test_invalid_version_fails(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "invalid_version.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert not result["success"], f"Expected prebake to fail with invalid version"
|
||||
assert "version" in result["stderr"].lower() or "Version" in result["stderr"]
|
||||
|
||||
def test_missing_required_field_fails(self, run_baker, work_dir):
|
||||
missing_env = os.path.join(work_dir, "missing_env.yml")
|
||||
with open(missing_env, "w") as f:
|
||||
f.write('version: "1.0"\n')
|
||||
result = run_baker("prebake", missing_env, cwd=work_dir, timeout=60)
|
||||
assert not result["success"]
|
||||
|
||||
def test_minimal_config_succeeds(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "minimal.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"], f"Minimal prebake failed: {result['stderr']}"
|
||||
|
||||
|
||||
@pytest.mark.prebake
|
||||
class TestPrebakeBootstrap:
|
||||
|
||||
def test_bootstrap_creates_vulcan_user(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"], f"prebake failed: {result['stderr']}"
|
||||
assert user.user_exists("vulcan")
|
||||
|
||||
def test_bootstrap_creates_workspace(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
assert file.dir_exists("/home/vulcan/workspace")
|
||||
|
||||
def test_bootstrap_creates_symlink(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
assert os.path.islink("/workspace")
|
||||
|
||||
def test_bootstrap_generates_bootstrap_script(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
assert file.file_exists("/bootstrap.sh")
|
||||
|
||||
def test_bootstrap_script_permissions(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
mode = file.file_permissions("/bootstrap.sh")
|
||||
assert mode & 0o111, f"/bootstrap.sh not executable, mode: {oct(mode)}"
|
||||
|
||||
def test_bootstrap_script_contains_user_creation(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
assert file.file_exists("/bootstrap.sh")
|
||||
assert file.file_contains("/bootstrap.sh", "useradd")
|
||||
|
||||
def test_bootstrap_script_contains_workspace_setup(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
assert file.file_contains("/bootstrap.sh", "mkdir -p /home/vulcan/workspace")
|
||||
|
||||
def test_bootstrap_with_pacman_creates_sudoers(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "with_pacman.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=120)
|
||||
assert result["success"], f"prebake failed: {result['stderr']}"
|
||||
assert file.file_exists("/etc/sudoers.d/workshop")
|
||||
|
||||
def test_custom_bootstrap_executes(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "with_custom_bootstrap.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"], f"Custom bootstrap failed: {result['stderr']}"
|
||||
assert file.dir_exists("/custom-workspace")
|
||||
|
||||
|
||||
@pytest.mark.prebake
|
||||
class TestPrebakeDryRun:
|
||||
|
||||
def test_dry_run_does_not_create_user(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
env = {"HBW_DRY_RUN": "true"}
|
||||
result = run_baker("prebake", config, cwd=work_dir, env=env, timeout=60)
|
||||
assert result["success"], f"Dry-run failed: {result['stderr']}"
|
||||
|
||||
def test_dry_run_logs_script_content(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
env = {"HBW_DRY_RUN": "true", "RUST_LOG": "info"}
|
||||
result = run_baker("prebake", config, cwd=work_dir, env=env, timeout=60)
|
||||
assert result["success"]
|
||||
combined = result["stdout"] + result["stderr"]
|
||||
assert "DRY_RUN" in combined or "dry_run" in combined.lower()
|
||||
|
||||
|
||||
@pytest.mark.prebake
|
||||
class TestPrebakeHooks:
|
||||
|
||||
#@pytest.mark.xfail(reason="Hook execution fails with Permission denied in container environment", strict=False)
|
||||
def test_early_hook_executes(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "with_hooks.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"], f"prebake with hooks failed: {result['stderr']}"
|
||||
combined = result["stdout"] + result["stderr"]
|
||||
assert "early hook executed" in combined
|
||||
|
||||
#@pytest.mark.xfail(reason="Hook execution fails with Permission denied in container environment", strict=False)
|
||||
def test_late_hook_executes(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "with_hooks.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"], f"prebake with hooks failed: {result['stderr']}"
|
||||
combined = result["stdout"] + result["stderr"]
|
||||
assert "late hook executed" in combined
|
||||
|
||||
#@pytest.mark.xfail(reason="Hook execution fails with Permission denied in container environment", strict=False)
|
||||
def test_hooks_run_in_order(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "with_hooks.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
combined = result["stdout"] + result["stderr"]
|
||||
early_pos = combined.find("early hook executed")
|
||||
late_pos = combined.find("late hook executed")
|
||||
assert early_pos != -1 and late_pos != -1
|
||||
assert early_pos < late_pos, "Early hook should execute before late hook"
|
||||
|
||||
def test_empty_hooks_succeed(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "minimal.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"], f"prebake with no hooks failed: {result['stderr']}"
|
||||
|
||||
|
||||
@pytest.mark.prebake
|
||||
class TestPrebakeDepsSystem:
|
||||
|
||||
@pytest.mark.xfail(reason="Package installation may fail in test environment", strict=False)
|
||||
def test_pacman_packages_installed(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "with_pacman.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=120)
|
||||
assert result["success"], f"prebake failed: {result['stderr']}"
|
||||
assert package.package_installed("curl")
|
||||
assert package.package_installed("wget")
|
||||
|
||||
def test_pacman_mirror_change(self, run_baker, work_dir):
|
||||
mirror_config = os.path.join(work_dir, "mirror.yml")
|
||||
with open(mirror_config, "w") as f:
|
||||
f.write('''
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
dependencies:
|
||||
system:
|
||||
pacman:
|
||||
packages: ["which"]
|
||||
mirror: "https://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch"
|
||||
''')
|
||||
result = run_baker("prebake", mirror_config, cwd=work_dir, timeout=120)
|
||||
assert result["success"], f"prebake with mirror failed: {result['stderr']}"
|
||||
|
||||
|
||||
@pytest.mark.prebake
|
||||
class TestPrebakeEnvVars:
|
||||
|
||||
def test_prebake_envvars_injected(self, run_baker, work_dir):
|
||||
hook_config = os.path.join(work_dir, "envvar_hook.yml")
|
||||
with open(hook_config, "w") as f:
|
||||
f.write('''
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
envvars:
|
||||
prebake:
|
||||
TEST_INJECTED: "injected_value"
|
||||
hooks:
|
||||
early:
|
||||
- name: "check-env"
|
||||
command: "echo $TEST_INJECTED"
|
||||
''')
|
||||
result = run_baker("prebake", hook_config, cwd=work_dir, timeout=60)
|
||||
assert result["success"], f"prebake with envvars failed: {result['stderr']}"
|
||||
combined = result["stdout"] + result["stderr"]
|
||||
assert "injected_value" in combined
|
||||
|
||||
|
||||
@pytest.mark.prebake
|
||||
class TestPrebakeSecurity:
|
||||
|
||||
#@pytest.mark.xfail(reason="Privilege drop may fail in container without proper setup", strict=False)
|
||||
def test_privilege_drop_after_bootstrap(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "with_security.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"], f"prebake with security failed: {result['stderr']}"
|
||||
|
||||
|
||||
@pytest.mark.prebake
|
||||
class TestPrebakeStatus:
|
||||
|
||||
def test_status_files_written(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
assert os.path.exists("/tmp/.hbwstatus")
|
||||
|
||||
def test_status_contains_bootstrap_stage(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
with open("/tmp/.hbwstatus", "r") as f:
|
||||
lines = f.readlines()
|
||||
assert len(lines) > 0
|
||||
bootstrap_lines = [l for l in lines if json.loads(l).get("name") == "bootstrap"]
|
||||
assert len(bootstrap_lines) > 0, "No bootstrap stage found in status file"
|
||||
data = json.loads(bootstrap_lines[-1])
|
||||
assert data.get("phase") == "Prebake"
|
||||
assert data.get("result") == "Success"
|
||||
|
||||
def test_status_contains_duration(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
with open("/tmp/.hbwstatus", "r") as f:
|
||||
data = json.loads(f.readline())
|
||||
assert "duration_ms" in data
|
||||
assert isinstance(data["duration_ms"], int)
|
||||
|
||||
def test_status_contains_timestamps(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
with open("/tmp/.hbwstatus", "r") as f:
|
||||
data = json.loads(f.readline())
|
||||
assert "started_at" in data
|
||||
assert "finished_at" in data
|
||||
|
||||
def test_status_written_for_each_stage(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "with_pacman.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=120)
|
||||
assert result["success"], f"prebake failed: {result['stderr']}"
|
||||
with open("/tmp/.hbwstatus", "r") as f:
|
||||
lines = f.readlines()
|
||||
names = [json.loads(line).get("name") for line in lines]
|
||||
assert "bootstrap" in names
|
||||
assert "depssystem" in names
|
||||
@@ -1,164 +0,0 @@
|
||||
"""
|
||||
Root conftest for HoneyBiscuitWorkshop system tests.
|
||||
|
||||
Provides core fixtures for:
|
||||
- Locating the baker binary and examples
|
||||
- Running subprocess commands with output capture
|
||||
- Managing temporary working directories
|
||||
"""
|
||||
import os
|
||||
import subprocess
|
||||
import shutil
|
||||
import tempfile
|
||||
import pytest
|
||||
|
||||
|
||||
def pytest_configure(config):
|
||||
"""Register custom markers."""
|
||||
config.addinivalue_line("markers", "prebake: prebake stage tests")
|
||||
config.addinivalue_line("markers", "bake: bake stage tests")
|
||||
config.addinivalue_line("markers", "finalize: finalize stage tests")
|
||||
config.addinivalue_line("markers", "llm: tests requiring LLM API")
|
||||
config.addinivalue_line("markers", "slow: slow running tests")
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def baker_bin():
|
||||
"""Path to the workshop-baker binary."""
|
||||
path = os.environ.get("BAKER_BIN", "/workshop/workshop-baker")
|
||||
if not os.path.isfile(path):
|
||||
pytest.skip(f"Baker binary not found at {path}")
|
||||
if not os.access(path, os.X_OK):
|
||||
pytest.skip(f"Baker binary not executable at {path}")
|
||||
return path
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def examples_dir():
|
||||
"""Path to the examples directory (may be None)."""
|
||||
path = os.environ.get("EXAMPLES_DIR", "/workshop/examples")
|
||||
if not os.path.isdir(path):
|
||||
return None
|
||||
return path
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def llm_env():
|
||||
"""LLM environment variables dict (from llm.env)."""
|
||||
env = {}
|
||||
for var in [
|
||||
"OPENAI_BASE_URL",
|
||||
"OPENAI_API_KEY",
|
||||
"ANTHROPIC_BASE_URL",
|
||||
"ANTHROPIC_API_KEY",
|
||||
"MODEL",
|
||||
"REASONING_EFFORT",
|
||||
]:
|
||||
val = os.environ.get(var)
|
||||
if val:
|
||||
env[var] = val
|
||||
return env
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def work_dir():
|
||||
"""Create a temporary working directory for a test, cleaned up after."""
|
||||
d = tempfile.mkdtemp(prefix="hbw-test-")
|
||||
yield d
|
||||
shutil.rmtree(d, ignore_errors=True)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def run_cmd():
|
||||
"""Fixture that provides a command runner function.
|
||||
|
||||
Returns a function that runs a command and returns
|
||||
(stdout, stderr, returncode, success).
|
||||
"""
|
||||
|
||||
def _run(args, cwd=None, env=None, timeout=120):
|
||||
"""Run a command and capture output.
|
||||
|
||||
Args:
|
||||
args: Command as list of strings.
|
||||
cwd: Working directory.
|
||||
env: Environment variables (merged with os.environ).
|
||||
timeout: Timeout in seconds.
|
||||
|
||||
Returns:
|
||||
dict with keys: stdout, stderr, returncode, success
|
||||
"""
|
||||
full_env = {**os.environ}
|
||||
if env:
|
||||
full_env.update(env)
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
args,
|
||||
cwd=cwd,
|
||||
env=full_env,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=timeout,
|
||||
)
|
||||
return {
|
||||
"stdout": result.stdout,
|
||||
"stderr": result.stderr,
|
||||
"returncode": result.returncode,
|
||||
"success": result.returncode == 0,
|
||||
}
|
||||
except subprocess.TimeoutExpired:
|
||||
return {
|
||||
"stdout": "",
|
||||
"stderr": f"Command timed out after {timeout}s",
|
||||
"returncode": -1,
|
||||
"success": False,
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"stdout": "",
|
||||
"stderr": str(e),
|
||||
"returncode": -1,
|
||||
"success": False,
|
||||
}
|
||||
|
||||
return _run
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def run_baker(baker_bin, run_cmd):
|
||||
"""Fixture that provides a baker command runner.
|
||||
|
||||
Returns a function that runs workshop-baker with common flags.
|
||||
"""
|
||||
|
||||
def _run(subcommand, *args, cwd=None, env=None, timeout=120):
|
||||
cmd = [
|
||||
baker_bin,
|
||||
"-u", "testuser",
|
||||
"-p", "test-pipeline",
|
||||
"-b", "test-build-001",
|
||||
"bare",
|
||||
subcommand,
|
||||
]
|
||||
cmd.extend(args)
|
||||
return run_cmd(cmd, cwd=cwd, env=env, timeout=timeout)
|
||||
|
||||
return _run
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def fixtures_dir():
|
||||
"""Path to the test fixtures directory."""
|
||||
return os.path.join(os.path.dirname(__file__), "fixtures")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def bake_workspace():
|
||||
"""Create /workspace for bake tests (baker writes temp scripts there)."""
|
||||
existed = os.path.exists("/workspace")
|
||||
if not existed:
|
||||
os.makedirs("/workspace", exist_ok=True)
|
||||
yield
|
||||
if not existed:
|
||||
shutil.rmtree("/workspace", ignore_errors=True)
|
||||
@@ -1,102 +0,0 @@
|
||||
"""PM system tests driven by bare workshop-baker prebake."""
|
||||
|
||||
import os
|
||||
import pytest
|
||||
from probes import binary_exists
|
||||
|
||||
PACMAN_MIRROR = "https://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch"
|
||||
|
||||
|
||||
def _yml(pm, pkgs, mirror="", repo=None):
|
||||
"""Build prebake.yml. repo is a list of lines (without lead indent)."""
|
||||
m = ""
|
||||
if mirror:
|
||||
m += ' mirror: "' + mirror + '"\n'
|
||||
if repo:
|
||||
m += " repositories:\n"
|
||||
for line in repo:
|
||||
m += " " + line + "\n"
|
||||
return """\
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
dependencies:
|
||||
system:
|
||||
{pm}:
|
||||
packages: [{pkgs}]
|
||||
{m}""".format(pm=pm, pkgs=pkgs, m=m)
|
||||
|
||||
|
||||
# ── install ──────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.parametrize("pm_name,pkgs", [
|
||||
("pacman", '"curl", "git"'),
|
||||
("apt", '"curl", "git"'),
|
||||
("dnf", '"curl", "git"'),
|
||||
("apk", '"curl", "git"'),
|
||||
])
|
||||
def test_install(run_cmd, work_dir, pm_name, pkgs):
|
||||
cfg = os.path.join(work_dir, "pm.yml")
|
||||
mirror = PACMAN_MIRROR if pm_name == "pacman" else ""
|
||||
with open(cfg, "w") as f:
|
||||
f.write(_yml(pm_name, pkgs, mirror=mirror))
|
||||
r = run_cmd(["/baker", "-p", "t", "-b", "1", "-u", "testuser",
|
||||
"bare", "prebake", cfg], timeout=120)
|
||||
assert r["success"], "[{}] prebake: {}".format(pm_name, r["stderr"][:300])
|
||||
for pkg in ["curl", "git"]:
|
||||
assert binary_exists("/usr/bin/" + pkg), "[{}] {} missing".format(pm_name, pkg)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("pm_name", ["pacman", "apt", "dnf", "apk"])
|
||||
def test_reinstall(run_cmd, work_dir, pm_name):
|
||||
cfg = os.path.join(work_dir, "pm.yml")
|
||||
mirror = PACMAN_MIRROR if pm_name == "pacman" else ""
|
||||
with open(cfg, "w") as f:
|
||||
f.write(_yml(pm_name, '"curl"', mirror=mirror))
|
||||
cmd = ["/baker", "-p", "t", "-b", "1", "-u", "testuser", "bare", "prebake", cfg]
|
||||
r1 = run_cmd(cmd, timeout=120)
|
||||
assert r1["success"], "[{}] first install failed".format(pm_name)
|
||||
r2 = run_cmd(cmd, timeout=120)
|
||||
assert r2["success"], "[{}] reinstall failed".format(pm_name)
|
||||
|
||||
|
||||
# ── change_mirror ────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.parametrize("pm_name", ["pacman", "apt", "dnf", "apk"])
|
||||
def test_mirror(run_cmd, work_dir, pm_name):
|
||||
cfg = os.path.join(work_dir, "pm.yml")
|
||||
mirror = PACMAN_MIRROR if pm_name == "pacman" else ""
|
||||
with open(cfg, "w") as f:
|
||||
f.write(_yml(pm_name, '"which"', mirror=mirror))
|
||||
r = run_cmd(["/baker", "-p", "t", "-b", "1", "-u", "testuser",
|
||||
"bare", "prebake", cfg], timeout=120)
|
||||
assert r["success"], "[{}] mirror: {}".format(pm_name, r["stderr"][:300])
|
||||
assert binary_exists("/usr/bin/which"), "[{}] which not installed".format(pm_name)
|
||||
|
||||
|
||||
# ── add_repository ───────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.parametrize("pm_name,repo_lines,result_file", [
|
||||
("pacman", ['- name: custom', ' server: "https://repo.example.com/$arch"'],
|
||||
"/etc/pacman.conf"),
|
||||
("apt", ['- url: "https://repo.example.com/debian"', ' distro: bookworm'],
|
||||
"/etc/apt/sources.list.d/custom.list"),
|
||||
("dnf", ['- name: custom', ' baseurl: "https://repo.example.com/fedora"'],
|
||||
"/etc/yum.repos.d/custom.repo"),
|
||||
("apk", ['- url: "https://repo.example.com/v3.19/main"'],
|
||||
"/etc/apk/repositories"),
|
||||
])
|
||||
@pytest.mark.xfail(reason="fake repo URL, package install may fail", strict=False)
|
||||
def test_add_repo(run_cmd, work_dir, pm_name, repo_lines, result_file):
|
||||
cfg = os.path.join(work_dir, "pm.yml")
|
||||
mirror = PACMAN_MIRROR if pm_name == "pacman" else ""
|
||||
with open(cfg, "w") as f:
|
||||
f.write(_yml(pm_name, '"curl"', mirror=mirror, repo=repo_lines))
|
||||
r = run_cmd(["/baker", "-p", "t", "-b", "1", "-u", "testuser",
|
||||
"bare", "prebake", cfg], timeout=120)
|
||||
# Prebake may fail if repo URL is unreachable — check only the config file
|
||||
# was written by add_repository before the (possibly failing) install step.
|
||||
assert binary_exists(result_file), \
|
||||
"[{}] repo file missing: {}".format(pm_name, result_file)
|
||||
@@ -1,36 +0,0 @@
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
# Load LLM environment if available
|
||||
if [ -f /workshop/llm.env ]; then
|
||||
set -a
|
||||
source /workshop/llm.env
|
||||
set +a
|
||||
fi
|
||||
|
||||
# Export test configuration
|
||||
export BAKER_BIN=/workshop/workshop-baker
|
||||
export EXAMPLES_DIR=/workshop/examples
|
||||
export TEST_MODE=${TEST_MODE:-auto}
|
||||
|
||||
if [ "$TEST_MODE" = "manual" ]; then
|
||||
echo "========================================"
|
||||
echo " HoneyBiscuitWorkshop System Tests"
|
||||
echo " Mode: MANUAL (interactive shell)"
|
||||
echo "========================================"
|
||||
echo ""
|
||||
echo "Binary: $BAKER_BIN"
|
||||
echo "Examples: $EXAMPLES_DIR"
|
||||
echo ""
|
||||
echo "Run tests manually:"
|
||||
echo " cd /app && pytest tests/ -v"
|
||||
echo ""
|
||||
exec /bin/bash
|
||||
else
|
||||
echo "========================================"
|
||||
echo " HoneyBiscuitWorkshop System Tests"
|
||||
echo " Mode: AUTO (running pytest)"
|
||||
echo "========================================"
|
||||
cd /app
|
||||
exec python -m pytest tests/ -v --tb=short --junitxml=/results/report.xml "$@"
|
||||
fi
|
||||
Vendored
-6
@@ -1,6 +0,0 @@
|
||||
#!/bin/bash
|
||||
[ -f /dev/shm/bakeexport.{{ name }} ] && . /dev/shm/bakeexport.{{ name }} || true
|
||||
{{ condition }}
|
||||
{{ main }}
|
||||
{{ name }}
|
||||
{{ export }}
|
||||
Vendored
-25
@@ -1,25 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
# @pipeline
|
||||
step_one() {
|
||||
echo "Step 1: Setup"
|
||||
mkdir -p /tmp/bake-test-output
|
||||
echo "step1-done" > /tmp/bake-test-output/step1.txt
|
||||
}
|
||||
|
||||
# @pipeline
|
||||
step_two() {
|
||||
echo "Step 2: Build"
|
||||
echo "step2-done" > /tmp/bake-test-output/step2.txt
|
||||
}
|
||||
|
||||
# @pipeline
|
||||
step_three() {
|
||||
echo "Step 3: Verify"
|
||||
if [ -f /tmp/bake-test-output/step1.txt ] && [ -f /tmp/bake-test-output/step2.txt ]; then
|
||||
echo "All steps completed successfully"
|
||||
else
|
||||
echo "ERROR: Missing step outputs"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
Vendored
-5
@@ -1,5 +0,0 @@
|
||||
#!/bin/bash
|
||||
echo "Hello from simple bake script"
|
||||
echo "Task ID: $HBW_TASKID"
|
||||
echo "Pipeline: $HBW_PIPELINE"
|
||||
exit 0
|
||||
Vendored
-22
@@ -1,22 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
# @pipeline
|
||||
init_workspace() {
|
||||
echo "Initializing workspace"
|
||||
mkdir -p /tmp/bake-after-test
|
||||
echo "init" > /tmp/bake-after-test/status.txt
|
||||
}
|
||||
|
||||
# @pipeline
|
||||
# @after(init_workspace)
|
||||
fetch_source() {
|
||||
echo "Fetching source (depends on init_workspace)"
|
||||
echo "fetched" > /tmp/bake-after-test/source.txt
|
||||
}
|
||||
|
||||
# @pipeline
|
||||
# @after(fetch_source)
|
||||
build_project() {
|
||||
echo "Building project (depends on fetch_source)"
|
||||
echo "built" > /tmp/bake-after-test/build.txt
|
||||
}
|
||||
-4
@@ -1,4 +0,0 @@
|
||||
#!/bin/bash
|
||||
echo "EARLY_HOOK_EXECUTED"
|
||||
echo "Working directory: $(pwd)"
|
||||
echo "User: $(whoami)"
|
||||
Vendored
-3
@@ -1,3 +0,0 @@
|
||||
#!/bin/bash
|
||||
echo "LATE_HOOK_EXECUTED"
|
||||
echo "Working directory: $(pwd)"
|
||||
Vendored
-6
@@ -1,6 +0,0 @@
|
||||
version: "0.0.1"
|
||||
plugin: {}
|
||||
notification: {}
|
||||
artifact: []
|
||||
cleanup:
|
||||
policy: "auto"
|
||||
-13
@@ -1,13 +0,0 @@
|
||||
version: "0.0.1"
|
||||
plugin: {}
|
||||
notification: {}
|
||||
artifact: []
|
||||
cleanup:
|
||||
policy: "auto"
|
||||
hooks:
|
||||
early:
|
||||
- name: "finalize-early"
|
||||
command: "echo 'finalize early hook'"
|
||||
late:
|
||||
- name: "finalize-late"
|
||||
command: "echo 'finalize late hook'"
|
||||
Vendored
-6
@@ -1,6 +0,0 @@
|
||||
FROM docker.1ms.run/library/alpine:3.19
|
||||
RUN sed -i 's|dl-cdn.alpinelinux.org|mirrors.tuna.tsinghua.edu.cn|g' /etc/apk/repositories
|
||||
RUN apk add --no-cache python3 py3-pip bash
|
||||
COPY workshop-baker/target/x86_64-unknown-linux-musl/debug/workshop-baker /baker
|
||||
RUN chmod +x /baker
|
||||
ENTRYPOINT ["/bin/sh", "-c"]
|
||||
Vendored
-6
@@ -1,6 +0,0 @@
|
||||
FROM docker.1ms.run/library/debian:bookworm-slim
|
||||
RUN sed -i 's|deb.debian.org|mirrors.tuna.tsinghua.edu.cn|g' /etc/apt/sources.list.d/debian.sources 2>/dev/null; sed -i 's|security.debian.org|mirrors.tuna.tsinghua.edu.cn/debian-security|g' /etc/apt/sources.list.d/debian.sources 2>/dev/null; true
|
||||
RUN apt-get update -qq && apt-get install -y -qq --no-install-recommends python3 python3-venv ca-certificates 2>&1 | tail -n 2
|
||||
COPY workshop-baker/target/x86_64-unknown-linux-musl/debug/workshop-baker /baker
|
||||
RUN chmod +x /baker
|
||||
ENTRYPOINT ["/bin/sh", "-c"]
|
||||
Vendored
-6
@@ -1,6 +0,0 @@
|
||||
FROM docker.1ms.run/library/fedora:43
|
||||
RUN sed -e 's|^metalink=|#metalink=|g' -e 's|^#baseurl=http://download.example/pub/fedora/linux|baseurl=https://mirrors.tuna.tsinghua.edu.cn/fedora|g' -i.bak /etc/yum.repos.d/fedora.repo /etc/yum.repos.d/fedora-updates.repo
|
||||
RUN dnf install -y python3 2>&1 | tail -n 2
|
||||
COPY workshop-baker/target/x86_64-unknown-linux-musl/debug/workshop-baker /baker
|
||||
RUN chmod +x /baker
|
||||
ENTRYPOINT ["/bin/sh", "-c"]
|
||||
Vendored
-2
@@ -1,2 +0,0 @@
|
||||
FROM docker.1ms.run/library/archlinux:latest
|
||||
RUN echo 'Server = https://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch' > /etc/pacman.d/mirrorlist
|
||||
-8
@@ -1,8 +0,0 @@
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
workspace:
|
||||
path: "/home/vulcan/workspace"
|
||||
fallback: true
|
||||
Vendored
-15
@@ -1,15 +0,0 @@
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
workspace:
|
||||
path: "/home/vulcan/workspace"
|
||||
fallback: true
|
||||
hooks:
|
||||
early:
|
||||
- name: "test-early-hook"
|
||||
command: "echo 'early hook executed'"
|
||||
late:
|
||||
- name: "test-late-hook"
|
||||
command: "echo 'late hook executed'"
|
||||
-3
@@ -1,3 +0,0 @@
|
||||
version: "0.5"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
Vendored
-8
@@ -1,8 +0,0 @@
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
workspace:
|
||||
path: "/home/vulcan/workspace"
|
||||
fallback: true
|
||||
@@ -1,12 +0,0 @@
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
workspace:
|
||||
path: "/home/vulcan/workspace"
|
||||
fallback: true
|
||||
custom: |
|
||||
#!/bin/sh
|
||||
echo 'custom bootstrap executed'
|
||||
mkdir -p /custom-workspace
|
||||
-16
@@ -1,16 +0,0 @@
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
workspace:
|
||||
path: "/home/vulcan/workspace"
|
||||
fallback: true
|
||||
dependencies:
|
||||
system:
|
||||
pacman:
|
||||
packages:
|
||||
- "which"
|
||||
user:
|
||||
cargo:
|
||||
- "ripgrep"
|
||||
-13
@@ -1,13 +0,0 @@
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
workspace:
|
||||
path: "/home/vulcan/workspace"
|
||||
fallback: true
|
||||
envvars:
|
||||
prebake:
|
||||
TEST_PREBAKE_VAR: "prebake_value"
|
||||
bake:
|
||||
TEST_BAKE_VAR: "bake_value"
|
||||
-17
@@ -1,17 +0,0 @@
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
workspace:
|
||||
path: "/home/vulcan/workspace"
|
||||
fallback: true
|
||||
hooks:
|
||||
early:
|
||||
- name: "test-early-hook"
|
||||
command: "echo 'early hook executed'"
|
||||
working_dir: "/workspace"
|
||||
late:
|
||||
- name: "test-late-hook"
|
||||
command: "echo 'late hook executed'"
|
||||
working_dir: "/workspace"
|
||||
-14
@@ -1,14 +0,0 @@
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
workspace:
|
||||
path: "/home/vulcan/workspace"
|
||||
fallback: true
|
||||
dependencies:
|
||||
system:
|
||||
pacman:
|
||||
packages:
|
||||
- "curl"
|
||||
- "wget"
|
||||
-10
@@ -1,10 +0,0 @@
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
workspace:
|
||||
path: "/home/vulcan/workspace"
|
||||
fallback: true
|
||||
security:
|
||||
drop_after: "Bootstrap"
|
||||
@@ -1,12 +0,0 @@
|
||||
"""
|
||||
LLM-based log judgment system for system tests.
|
||||
|
||||
Uses LLM APIs to evaluate non-deterministic test outputs (logs, stderr, etc.)
|
||||
where simple string matching is insufficient.
|
||||
"""
|
||||
|
||||
from .client import LLMClient
|
||||
from .cache import LLMCache
|
||||
from .judge import judge_log, JudgmentResult
|
||||
|
||||
__all__ = ["LLMClient", "LLMCache", "judge_log", "JudgmentResult"]
|
||||
@@ -1,56 +0,0 @@
|
||||
"""
|
||||
Disk-based cache for LLM API responses.
|
||||
|
||||
Uses SHA256 of (prompt + model + system_prompt) as cache key.
|
||||
Cache files stored as JSON in ``.cache/`` directory.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
class LLMCache:
|
||||
"""SHA256-based disk cache for LLM API responses."""
|
||||
|
||||
def __init__(self, cache_dir: str | None = None):
|
||||
if cache_dir is None:
|
||||
import os
|
||||
|
||||
cache_dir = os.path.join(os.path.dirname(__file__), ".cache")
|
||||
self.cache_dir = Path(cache_dir)
|
||||
self.cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def _make_key(self, prompt: str, model: str, system_prompt: str = "") -> str:
|
||||
"""Create deterministic cache key from request parameters."""
|
||||
data = json.dumps(
|
||||
{"prompt": prompt, "model": model, "system": system_prompt},
|
||||
sort_keys=True,
|
||||
)
|
||||
return hashlib.sha256(data.encode()).hexdigest()
|
||||
|
||||
def get(self, prompt: str, model: str, system_prompt: str = "") -> dict | None:
|
||||
"""Retrieve cached response if exists."""
|
||||
key = self._make_key(prompt, model, system_prompt)
|
||||
path = self.cache_dir / f"{key}.json"
|
||||
if path.exists():
|
||||
try:
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
except (json.JSONDecodeError, IOError):
|
||||
return None
|
||||
return None
|
||||
|
||||
def set(
|
||||
self, prompt: str, model: str, system_prompt: str, response: dict
|
||||
) -> None:
|
||||
"""Store response in cache."""
|
||||
key = self._make_key(prompt, model, system_prompt)
|
||||
path = self.cache_dir / f"{key}.json"
|
||||
with open(path, "w") as f:
|
||||
json.dump(response, f, ensure_ascii=False)
|
||||
|
||||
def clear(self) -> None:
|
||||
"""Clear all cached responses."""
|
||||
for f in self.cache_dir.glob("*.json"):
|
||||
f.unlink()
|
||||
@@ -1,187 +0,0 @@
|
||||
"""
|
||||
LLM client supporting both Anthropic and OpenAI APIs.
|
||||
|
||||
Prefers Anthropic format (per project convention).
|
||||
Falls back to OpenAI format if Anthropic is unavailable.
|
||||
"""
|
||||
|
||||
import os
|
||||
import logging
|
||||
from .cache import LLMCache
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class LLMClient:
|
||||
"""LLM client with caching support. Prefers Anthropic API."""
|
||||
|
||||
def __init__(self, cache: LLMCache | None = None):
|
||||
self.cache = cache or LLMCache()
|
||||
self._anthropic_client = None
|
||||
self._openai_client = None
|
||||
self._init_clients()
|
||||
|
||||
def _init_clients(self):
|
||||
"""Initialize API clients based on available environment variables."""
|
||||
# Anthropic (preferred)
|
||||
anthropic_key = os.environ.get("ANTHROPIC_API_KEY")
|
||||
anthropic_base = os.environ.get("ANTHROPIC_BASE_URL")
|
||||
if anthropic_key:
|
||||
try:
|
||||
import anthropic
|
||||
|
||||
kwargs: dict = {"api_key": anthropic_key}
|
||||
if anthropic_base:
|
||||
kwargs["base_url"] = anthropic_base
|
||||
self._anthropic_client = anthropic.Anthropic(**kwargs)
|
||||
logger.info("Anthropic client initialized")
|
||||
except ImportError:
|
||||
logger.warning("anthropic package not installed")
|
||||
|
||||
# OpenAI (fallback)
|
||||
openai_key = os.environ.get("OPENAI_API_KEY")
|
||||
openai_base = os.environ.get("OPENAI_BASE_URL")
|
||||
if openai_key:
|
||||
try:
|
||||
import openai
|
||||
|
||||
kwargs = {"api_key": openai_key}
|
||||
if openai_base:
|
||||
kwargs["base_url"] = openai_base
|
||||
self._openai_client = openai.OpenAI(**kwargs)
|
||||
logger.info("OpenAI client initialized")
|
||||
except ImportError:
|
||||
logger.warning("openai package not installed")
|
||||
|
||||
@property
|
||||
def available(self) -> bool:
|
||||
"""Check if any LLM client is available."""
|
||||
return (
|
||||
self._anthropic_client is not None or self._openai_client is not None
|
||||
)
|
||||
|
||||
def complete(
|
||||
self, prompt: str, system_prompt: str = "", max_tokens: int = 2000
|
||||
) -> str:
|
||||
"""Send a completion request, with caching.
|
||||
|
||||
Args:
|
||||
prompt: User message.
|
||||
system_prompt: System message (sets role / persona).
|
||||
max_tokens: Max response tokens.
|
||||
|
||||
Returns:
|
||||
LLM response text.
|
||||
|
||||
Raises:
|
||||
RuntimeError: If no client is available.
|
||||
"""
|
||||
model = os.environ.get("MODEL", "deepseek-v4-flash")
|
||||
|
||||
# Check cache
|
||||
cached = self.cache.get(prompt, model, system_prompt)
|
||||
if cached:
|
||||
logger.info("Cache hit for LLM request")
|
||||
return cached["content"]
|
||||
|
||||
# Try Anthropic first
|
||||
if self._anthropic_client:
|
||||
content = self._call_anthropic(
|
||||
prompt, system_prompt, model, max_tokens
|
||||
)
|
||||
elif self._openai_client:
|
||||
content = self._call_openai(prompt, system_prompt, model, max_tokens)
|
||||
else:
|
||||
raise RuntimeError(
|
||||
"No LLM client available. "
|
||||
"Set ANTHROPIC_API_KEY or OPENAI_API_KEY."
|
||||
)
|
||||
|
||||
# Cache the response
|
||||
self.cache.set(prompt, model, system_prompt, {"content": content})
|
||||
return content
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Private helpers
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _call_anthropic(
|
||||
self, prompt: str, system_prompt: str, model: str, max_tokens: int
|
||||
) -> str:
|
||||
"""Call Anthropic API."""
|
||||
kwargs: dict = {
|
||||
"model": model,
|
||||
"max_tokens": max_tokens,
|
||||
"messages": [{"role": "user", "content": prompt}],
|
||||
}
|
||||
if system_prompt:
|
||||
kwargs["system"] = system_prompt
|
||||
|
||||
# Only add reasoning_effort if the model supports it
|
||||
reasoning_effort = os.environ.get("REASONING_EFFORT")
|
||||
if reasoning_effort:
|
||||
try:
|
||||
kwargs["reasoning_effort"] = reasoning_effort
|
||||
response = self._anthropic_client.messages.create(**kwargs)
|
||||
except Exception as e:
|
||||
if (
|
||||
"reasoning_effort" in str(e).lower()
|
||||
or "unexpected" in str(e).lower()
|
||||
):
|
||||
del kwargs["reasoning_effort"]
|
||||
response = self._anthropic_client.messages.create(**kwargs)
|
||||
else:
|
||||
raise
|
||||
else:
|
||||
response = self._anthropic_client.messages.create(**kwargs)
|
||||
|
||||
# Extract text from response content blocks
|
||||
if hasattr(response, "content") and response.content:
|
||||
block = response.content[0]
|
||||
if hasattr(block, "text"):
|
||||
return block.text
|
||||
if hasattr(block, "type") and block.type == "thinking":
|
||||
for b in response.content:
|
||||
if hasattr(b, "text"):
|
||||
return b.text
|
||||
return str(block)
|
||||
return str(response)
|
||||
|
||||
def _call_openai(
|
||||
self, prompt: str, system_prompt: str, model: str, max_tokens: int
|
||||
) -> str:
|
||||
"""Call OpenAI-compatible API."""
|
||||
messages: list[dict] = []
|
||||
if system_prompt:
|
||||
messages.append({"role": "system", "content": system_prompt})
|
||||
messages.append({"role": "user", "content": prompt})
|
||||
|
||||
kwargs: dict = {
|
||||
"model": model,
|
||||
"messages": messages,
|
||||
"max_tokens": max_tokens,
|
||||
}
|
||||
|
||||
# Only add reasoning_effort if supported
|
||||
reasoning_effort = os.environ.get("REASONING_EFFORT")
|
||||
if reasoning_effort:
|
||||
try:
|
||||
kwargs["reasoning_effort"] = reasoning_effort
|
||||
response = self._openai_client.chat.completions.create(
|
||||
**kwargs
|
||||
)
|
||||
except Exception as e:
|
||||
if (
|
||||
"reasoning_effort" in str(e).lower()
|
||||
or "unexpected" in str(e).lower()
|
||||
):
|
||||
del kwargs["reasoning_effort"]
|
||||
response = self._openai_client.chat.completions.create(
|
||||
**kwargs
|
||||
)
|
||||
else:
|
||||
raise
|
||||
else:
|
||||
response = self._openai_client.chat.completions.create(**kwargs)
|
||||
|
||||
return response.choices[0].message.content
|
||||
@@ -1,149 +0,0 @@
|
||||
"""
|
||||
LLM-based log judgment for system tests.
|
||||
|
||||
Evaluates logs against expected behavior using LLM semantic analysis.
|
||||
Returns structured pass/fail results with reasoning.
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
from .client import LLMClient
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
JUDGE_SYSTEM_PROMPT = """\
|
||||
You are a test result judge for a CI/CD system called HoneyBiscuitWorkshop.
|
||||
|
||||
Your job: Evaluate whether actual command output/logs match expected behavior.
|
||||
|
||||
RULES:
|
||||
1. You MUST respond with valid JSON in this exact format:
|
||||
{"verdict": "pass" | "fail", "reason": "brief explanation"}
|
||||
2. "pass" = the output confirms the expected behavior occurred
|
||||
3. "fail" = the output shows the expected behavior did NOT occur, or shows an error
|
||||
4. Ignore irrelevant noise (timestamps, debug lines, unrelated warnings)
|
||||
5. Focus ONLY on whether the specific expected behavior is confirmed or contradicted
|
||||
6. Be strict: if the evidence is ambiguous, verdict is "fail"
|
||||
7. Do NOT include any text outside the JSON object"""
|
||||
|
||||
|
||||
@dataclass
|
||||
class JudgmentResult:
|
||||
"""Result of an LLM log judgment."""
|
||||
|
||||
verdict: str # "pass" or "fail"
|
||||
reason: str
|
||||
raw_response: str
|
||||
|
||||
@property
|
||||
def passed(self) -> bool:
|
||||
return self.verdict == "pass"
|
||||
|
||||
|
||||
def _extract_relevant_logs(full_log: str, max_chars: int = 4000) -> str:
|
||||
"""Extract relevant portions from logs, removing noise.
|
||||
|
||||
Keeps the last *max_chars* characters (most recent / relevant).
|
||||
Removes common noise patterns.
|
||||
"""
|
||||
lines = full_log.strip().split("\n")
|
||||
|
||||
noise_patterns = [
|
||||
r"^\d{4}-\d{2}-\d{2}T",
|
||||
r"^DEBUG ",
|
||||
r"^TRACE ",
|
||||
]
|
||||
filtered = [
|
||||
line
|
||||
for line in lines
|
||||
if not any(re.match(p, line) for p in noise_patterns)
|
||||
]
|
||||
|
||||
text = "\n".join(filtered)
|
||||
if len(text) > max_chars:
|
||||
text = "...\n" + text[-max_chars:]
|
||||
return text
|
||||
|
||||
|
||||
def judge_log(
|
||||
operation: str,
|
||||
expected_behavior: str,
|
||||
actual_log: str,
|
||||
client: LLMClient | None = None,
|
||||
) -> JudgmentResult:
|
||||
"""Judge whether actual logs match expected behavior.
|
||||
|
||||
Args:
|
||||
operation: Description of what was attempted
|
||||
(e.g. ``"prebake bootstrap stage"``).
|
||||
expected_behavior: What should have happened
|
||||
(e.g. ``"user vulcan was created"``).
|
||||
actual_log: The actual stdout / stderr output.
|
||||
client: ``LLMClient`` instance (creates default if ``None``).
|
||||
|
||||
Returns:
|
||||
``JudgmentResult`` with verdict and reason.
|
||||
"""
|
||||
if client is None:
|
||||
client = LLMClient()
|
||||
|
||||
if not client.available:
|
||||
logger.warning("No LLM client available, returning fail verdict")
|
||||
return JudgmentResult(
|
||||
verdict="fail",
|
||||
reason="No LLM client available for judgment",
|
||||
raw_response="",
|
||||
)
|
||||
|
||||
relevant_log = _extract_relevant_logs(actual_log)
|
||||
|
||||
prompt = f"""\
|
||||
## Operation
|
||||
{operation}
|
||||
|
||||
## Expected Behavior
|
||||
{expected_behavior}
|
||||
|
||||
## Actual Output (relevant excerpts)
|
||||
```
|
||||
{relevant_log}
|
||||
```
|
||||
|
||||
Evaluate: Does the actual output confirm the expected behavior occurred?
|
||||
Respond with JSON only."""
|
||||
|
||||
try:
|
||||
response = client.complete(
|
||||
prompt=prompt,
|
||||
system_prompt=JUDGE_SYSTEM_PROMPT,
|
||||
max_tokens=500,
|
||||
)
|
||||
|
||||
# Parse JSON response — handle potential markdown code blocks
|
||||
text = response.strip()
|
||||
if text.startswith("```"):
|
||||
lines = text.split("\n")
|
||||
text = "\n".join(lines[1:-1]) if len(lines) > 2 else text
|
||||
|
||||
result = json.loads(text)
|
||||
return JudgmentResult(
|
||||
verdict=result.get("verdict", "fail"),
|
||||
reason=result.get("reason", "No reason provided"),
|
||||
raw_response=response,
|
||||
)
|
||||
except json.JSONDecodeError as e:
|
||||
logger.error(f"Failed to parse LLM response as JSON: {e}")
|
||||
return JudgmentResult(
|
||||
verdict="fail",
|
||||
reason=f"LLM response was not valid JSON: {response[:200]}",
|
||||
raw_response=response,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"LLM judgment failed: {e}")
|
||||
return JudgmentResult(
|
||||
verdict="fail",
|
||||
reason=f"LLM call failed: {str(e)}",
|
||||
raw_response="",
|
||||
)
|
||||
@@ -1,45 +0,0 @@
|
||||
"""
|
||||
Environment probes for verifying system state in workshop-baker tests.
|
||||
|
||||
Each probe is a standalone function that can be called from any test.
|
||||
Probes return bool or Optional[int/str]; they do not raise on failure.
|
||||
|
||||
A probe accepts an optional *runner* callable (default ``subprocess.run``)
|
||||
that decouples it from the execution context (host or Docker).
|
||||
"""
|
||||
|
||||
from .user import user_exists, user_uid, user_gid, user_in_group, user_shell
|
||||
from .file import (
|
||||
file_exists,
|
||||
file_absent,
|
||||
dir_exists,
|
||||
dir_absent,
|
||||
file_permissions,
|
||||
file_contains,
|
||||
file_owned_by,
|
||||
file_group,
|
||||
)
|
||||
from .package import binary_exists, binary_on_path, package_installed, package_version
|
||||
from .process import process_running, process_count
|
||||
|
||||
__all__ = [
|
||||
"user_exists",
|
||||
"user_uid",
|
||||
"user_gid",
|
||||
"user_in_group",
|
||||
"user_shell",
|
||||
"file_exists",
|
||||
"file_absent",
|
||||
"dir_exists",
|
||||
"dir_absent",
|
||||
"file_permissions",
|
||||
"file_contains",
|
||||
"file_owned_by",
|
||||
"file_group",
|
||||
"package_installed",
|
||||
"package_version",
|
||||
"binary_exists",
|
||||
"binary_on_path",
|
||||
"process_running",
|
||||
"process_count",
|
||||
]
|
||||
@@ -1,53 +0,0 @@
|
||||
"""File and directory existence / permission probes."""
|
||||
|
||||
import os
|
||||
import stat
|
||||
import grp
|
||||
import pwd
|
||||
|
||||
|
||||
def file_exists(path: str) -> bool:
|
||||
"""Check if *path* exists and is a regular file."""
|
||||
return os.path.isfile(path)
|
||||
|
||||
|
||||
def dir_exists(path: str) -> bool:
|
||||
"""Check if *path* exists and is a directory."""
|
||||
return os.path.isdir(path)
|
||||
|
||||
|
||||
def file_permissions(path: str) -> int:
|
||||
"""Return the permission mode of *path* as an octal int (e.g. ``0o755``).
|
||||
|
||||
Raises ``FileNotFoundError`` if *path* doesn't exist.
|
||||
"""
|
||||
return stat.S_IMODE(os.stat(path).st_mode)
|
||||
|
||||
|
||||
def file_contains(path: str, text: str) -> bool:
|
||||
"""Check if a file contains the specified text."""
|
||||
with open(path, "r", errors="replace") as f:
|
||||
return text in f.read()
|
||||
|
||||
|
||||
def file_owned_by(path: str, username: str) -> bool:
|
||||
"""Check if a file is owned by the specified user."""
|
||||
st = os.stat(path)
|
||||
owner = pwd.getpwuid(st.st_uid).pw_name
|
||||
return owner == username
|
||||
|
||||
|
||||
def file_group(path: str) -> str:
|
||||
"""Get the group owner of a file."""
|
||||
st = os.stat(path)
|
||||
return grp.getgrgid(st.st_gid).gr_name
|
||||
|
||||
|
||||
def file_absent(path: str) -> bool:
|
||||
"""Check that *path* does not exist."""
|
||||
return not os.path.lexists(path)
|
||||
|
||||
|
||||
def dir_absent(path: str) -> bool:
|
||||
"""Check that *path* does not exist or is not a directory."""
|
||||
return not os.path.isdir(path)
|
||||
@@ -1,42 +0,0 @@
|
||||
"""Package / binary existence probes (cross-distro).
|
||||
|
||||
Each probe accepts an optional *runner* callable with the same signature
|
||||
as ``subprocess.run``. Defaults to running on the host; for Docker tests,
|
||||
pass a runner that wraps ``docker run --rm <image>``.
|
||||
"""
|
||||
|
||||
import subprocess
|
||||
from typing import Any, Callable
|
||||
|
||||
Runner = Callable[..., Any]
|
||||
|
||||
|
||||
def _run(cmd: list[str], runner: Runner = subprocess.run, **kw: Any) -> Any:
|
||||
opts = {"capture_output": True, "text": True, "timeout": 30}
|
||||
opts.update(kw)
|
||||
return runner(cmd, **opts)
|
||||
|
||||
|
||||
def _succeeded(r: Any) -> bool:
|
||||
""".returncode==0 for CompletedProcess, or ["success"] for dict."""
|
||||
if hasattr(r, "returncode"):
|
||||
return r.returncode == 0
|
||||
return bool(r.get("success", False))
|
||||
|
||||
|
||||
def binary_exists(path: str, runner: Runner = subprocess.run) -> bool:
|
||||
"""Check if a binary exists at *path* (e.g. ``/usr/bin/curl``).
|
||||
|
||||
Uses ``test -f`` (POSIX, guaranteed available).
|
||||
"""
|
||||
return _succeeded(_run(["test", "-f", path], runner))
|
||||
|
||||
|
||||
def binary_on_path(name: str, runner: Runner = subprocess.run) -> bool:
|
||||
"""Check if a command is on PATH (via ``command -v``)."""
|
||||
return _succeeded(_run(["sh", "-c", "command -v " + name], runner))
|
||||
|
||||
|
||||
# Backward-compatible aliases
|
||||
package_installed = binary_exists
|
||||
package_version = binary_on_path # used in __init__ export, now binary-based
|
||||
@@ -1,44 +0,0 @@
|
||||
"""Process existence probes."""
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
|
||||
|
||||
def process_running(process_name: str) -> bool:
|
||||
"""Check if a process with the given name is running."""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["pgrep", "-x", process_name],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
return result.returncode == 0
|
||||
except FileNotFoundError:
|
||||
# Fall back to /proc scan if pgrep not available
|
||||
try:
|
||||
for pid_dir in os.listdir("/proc"):
|
||||
if pid_dir.isdigit():
|
||||
try:
|
||||
with open(f"/proc/{pid_dir}/comm") as f:
|
||||
if f.read().strip() == process_name:
|
||||
return True
|
||||
except (IOError, OSError):
|
||||
continue
|
||||
except OSError:
|
||||
pass
|
||||
return False
|
||||
|
||||
|
||||
def process_count(process_name: str) -> int:
|
||||
"""Count processes with the given name."""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["pgrep", "-c", "-x", process_name],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
if result.returncode == 0:
|
||||
return int(result.stdout.strip())
|
||||
return 0
|
||||
except (FileNotFoundError, ValueError):
|
||||
return 0
|
||||
@@ -1,41 +0,0 @@
|
||||
"""User existence and attribute probes."""
|
||||
|
||||
import pwd
|
||||
import grp
|
||||
|
||||
|
||||
def user_exists(username: str) -> bool:
|
||||
"""Check if a system user exists."""
|
||||
try:
|
||||
pwd.getpwnam(username)
|
||||
return True
|
||||
except KeyError:
|
||||
return False
|
||||
|
||||
|
||||
def user_uid(username: str) -> int | None:
|
||||
try:
|
||||
return pwd.getpwnam(username).pw_uid
|
||||
except KeyError:
|
||||
return None
|
||||
|
||||
def user_gid(username: str) -> int | None:
|
||||
try:
|
||||
return pwd.getpwnam(username).pw_gid
|
||||
except KeyError:
|
||||
return None
|
||||
|
||||
def user_in_group(username: str, groupname: str) -> bool:
|
||||
"""Check if a user is a member of a group."""
|
||||
try:
|
||||
group = grp.getgrnam(groupname)
|
||||
return username in group.gr_mem
|
||||
except KeyError:
|
||||
return False
|
||||
|
||||
|
||||
def user_shell(username: str) -> str | None:
|
||||
try:
|
||||
return pwd.getpwnam(username).pw_shell
|
||||
except KeyError:
|
||||
return None
|
||||
@@ -1,6 +0,0 @@
|
||||
pytest>=8.0.0
|
||||
pytest-timeout>=2.2.0
|
||||
pytest-xdist>=3.5.0
|
||||
anthropic>=0.40.0,<0.90.0
|
||||
openai>=1.50.0,<2.0.0
|
||||
pyyaml>=6.0
|
||||
Generated
+107
@@ -0,0 +1,107 @@
|
||||
# This file is automatically @generated by Cargo.
|
||||
# It is not intended for manual editing.
|
||||
version = 4
|
||||
|
||||
[[package]]
|
||||
name = "itoa"
|
||||
version = "1.0.18"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
|
||||
|
||||
[[package]]
|
||||
name = "memchr"
|
||||
version = "2.8.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f8ca58f447f06ed17d5fc4043ce1b10dd205e060fb3ce5b979b8ed8e59ff3f79"
|
||||
|
||||
[[package]]
|
||||
name = "proc-macro2"
|
||||
version = "1.0.106"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934"
|
||||
dependencies = [
|
||||
"unicode-ident",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "quote"
|
||||
version = "1.0.45"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde"
|
||||
version = "1.0.228"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e"
|
||||
dependencies = [
|
||||
"serde_core",
|
||||
"serde_derive",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde_core"
|
||||
version = "1.0.228"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad"
|
||||
dependencies = [
|
||||
"serde_derive",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde_derive"
|
||||
version = "1.0.228"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde_json"
|
||||
version = "1.0.149"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "83fc039473c5595ace860d8c4fafa220ff474b3fc6bfdb4293327f1a37e94d86"
|
||||
dependencies = [
|
||||
"itoa",
|
||||
"memchr",
|
||||
"serde",
|
||||
"serde_core",
|
||||
"zmij",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "syn"
|
||||
version = "2.0.117"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"unicode-ident",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "unicode-ident"
|
||||
version = "1.0.24"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75"
|
||||
|
||||
[[package]]
|
||||
name = "workshop-baker-params"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"serde",
|
||||
"serde_json",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "zmij"
|
||||
version = "1.0.21"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa"
|
||||
@@ -0,0 +1,8 @@
|
||||
[package]
|
||||
name = "workshop-baker-params"
|
||||
version = "0.1.0"
|
||||
edition = "2024"
|
||||
|
||||
[dependencies]
|
||||
serde = { version = "1.0.228", features = ["derive"] }
|
||||
serde_json = "1.0.149"
|
||||
@@ -0,0 +1,54 @@
|
||||
use crate::apply_field;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct BakeParams {
|
||||
#[serde(default)] pub workspace: ParamVal<String>,
|
||||
/// Health probe interval in seconds (default 5).
|
||||
#[serde(default)] pub health_period: ParamVal<u64>,
|
||||
/// Health probe per-attempt timeout in seconds (default 10).
|
||||
#[serde(default)] pub health_timeout: ParamVal<u64>,
|
||||
/// Max health probe retries before giving up (default 12).
|
||||
#[serde(default)] pub health_max_retries: ParamVal<u32>,
|
||||
}
|
||||
|
||||
impl MergeParams for BakeParams {
|
||||
fn defaults() -> Self {
|
||||
Self {
|
||||
workspace: ParamVal::new("/workspace".into()),
|
||||
health_period: ParamVal::new(5),
|
||||
health_timeout: ParamVal::new(10),
|
||||
health_max_retries: ParamVal::new(12),
|
||||
}
|
||||
}
|
||||
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
|
||||
apply_field!(self.workspace, i.workspace, w, ov, p, "workspace");
|
||||
apply_field!(self.health_period, i.health_period, w, ov, p, "health_period");
|
||||
apply_field!(self.health_timeout, i.health_timeout, w, ov, p, "health_timeout");
|
||||
apply_field!(self.health_max_retries, i.health_max_retries, w, ov, p, "health_max_retries");
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for BakeParams { fn default() -> Self { Self::defaults() } }
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct BakeSettings {
|
||||
pub workspace: String,
|
||||
pub health_period: u64,
|
||||
pub health_timeout: u64,
|
||||
pub health_max_retries: u32,
|
||||
}
|
||||
impl Default for BakeSettings { fn default() -> Self { BakeParams::defaults().into() } }
|
||||
impl From<&BakeParams> for BakeSettings {
|
||||
fn from(p: &BakeParams) -> Self {
|
||||
Self {
|
||||
workspace: p.workspace.value.clone(),
|
||||
health_period: p.health_period.value,
|
||||
health_timeout: p.health_timeout.value,
|
||||
health_max_retries: p.health_max_retries.value,
|
||||
}
|
||||
}
|
||||
}
|
||||
impl From<BakeParams> for BakeSettings { fn from(p: BakeParams) -> Self { (&p).into() } }
|
||||
@@ -0,0 +1,23 @@
|
||||
use crate::apply_field;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct FinalizeParams {
|
||||
#[serde(default)] pub artifact_retention_days: ParamVal<u32>,
|
||||
}
|
||||
|
||||
impl MergeParams for FinalizeParams {
|
||||
fn defaults() -> Self { Self { artifact_retention_days: ParamVal::new(7) } }
|
||||
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
|
||||
apply_field!(self.artifact_retention_days, i.artifact_retention_days, w, ov, p, "artifact_retention_days");
|
||||
}
|
||||
}
|
||||
impl Default for FinalizeParams { fn default() -> Self { Self::defaults() } }
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct FinalizeSettings { pub artifact_retention_days: u32 }
|
||||
impl Default for FinalizeSettings { fn default() -> Self { FinalizeParams::defaults().into() } }
|
||||
impl From<&FinalizeParams> for FinalizeSettings { fn from(p: &FinalizeParams) -> Self { Self { artifact_retention_days: p.artifact_retention_days.value } } }
|
||||
impl From<FinalizeParams> for FinalizeSettings { fn from(p: FinalizeParams) -> Self { (&p).into() } }
|
||||
@@ -0,0 +1,168 @@
|
||||
pub mod notification;
|
||||
pub mod prebake;
|
||||
pub mod bake;
|
||||
pub mod finalize;
|
||||
pub mod writer;
|
||||
pub mod param;
|
||||
pub mod traits;
|
||||
|
||||
pub use notification::{NotificationParams, NotificationSettings};
|
||||
pub use prebake::{PrebakeParams, PrebakeSettings};
|
||||
pub use bake::{BakeParams, BakeSettings};
|
||||
pub use finalize::{FinalizeParams, FinalizeSettings};
|
||||
pub use writer::Writer;
|
||||
pub use param::ParamVal;
|
||||
pub use traits::MergeParams;
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::collections::HashMap;
|
||||
|
||||
/// Root parameter tree. Each component gets its own subtree.
|
||||
/// Used for both IPC (Serialized to JSON) and in-process merging.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||
pub struct Params {
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub notification: Option<NotificationParams>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub prebake: Option<PrebakeParams>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub bake: Option<BakeParams>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub finalize: Option<FinalizeParams>,
|
||||
}
|
||||
|
||||
/// Apply and merge helpers.
|
||||
impl Params {
|
||||
/// Merge a layer from a specific writer into the current params.
|
||||
/// Returns entries that were overridden (for debug).
|
||||
pub fn apply(&mut self, incoming: Params, writer: Writer) -> Overridden {
|
||||
let mut ov = Overridden::new();
|
||||
if let Some(l) = incoming.notification {
|
||||
if let Some(ref mut b) = self.notification {
|
||||
b.apply(l, writer, &mut ov, "notification.");
|
||||
} else {
|
||||
self.notification = Some(l);
|
||||
}
|
||||
}
|
||||
if let Some(l) = incoming.prebake {
|
||||
if let Some(ref mut b) = self.prebake {
|
||||
b.apply(l, writer, &mut ov, "prebake.");
|
||||
} else {
|
||||
self.prebake = Some(l);
|
||||
}
|
||||
}
|
||||
if let Some(l) = incoming.bake {
|
||||
if let Some(ref mut b) = self.bake {
|
||||
b.apply(l, writer, &mut ov, "bake.");
|
||||
} else {
|
||||
self.bake = Some(l);
|
||||
}
|
||||
}
|
||||
if let Some(l) = incoming.finalize {
|
||||
if let Some(ref mut b) = self.finalize {
|
||||
b.apply(l, writer, &mut ov, "finalize.");
|
||||
} else {
|
||||
self.finalize = Some(l);
|
||||
}
|
||||
}
|
||||
ov
|
||||
}
|
||||
|
||||
/// Convert to baker-consumable settings. All params guaranteed to have values.
|
||||
pub fn to_settings(&self) -> Result<AllSettings, String> {
|
||||
Ok(AllSettings {
|
||||
notification: self.notification.as_ref().map(NotificationSettings::from).unwrap_or_default(),
|
||||
prebake: self.prebake.as_ref().map(PrebakeSettings::from).unwrap_or_default(),
|
||||
bake: self.bake.as_ref().map(BakeSettings::from).unwrap_or_default(),
|
||||
finalize: self.finalize.as_ref().map(FinalizeSettings::from).unwrap_or_default(),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// Plain-value settings for baker consumption. No metadata, no Option.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct AllSettings {
|
||||
pub notification: NotificationSettings,
|
||||
pub prebake: PrebakeSettings,
|
||||
pub bake: BakeSettings,
|
||||
pub finalize: FinalizeSettings,
|
||||
}
|
||||
|
||||
/// Debug — values that were overridden during merge, keyed by dot-path.
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct Overridden {
|
||||
pub entries: std::collections::HashMap<String, serde_json::Value>,
|
||||
}
|
||||
|
||||
impl Overridden {
|
||||
pub fn new() -> Self { Self { entries: HashMap::new() } }
|
||||
pub fn is_empty(&self) -> bool { self.entries.is_empty() }
|
||||
}
|
||||
|
||||
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_empty_params_serializes_empty() {
|
||||
let p = Params::default();
|
||||
let json = serde_json::to_string(&p).unwrap();
|
||||
assert_eq!(json, "{}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_notification_params_serialize() {
|
||||
let mut p = Params::default();
|
||||
p.notification = Some(NotificationParams::defaults());
|
||||
let json = serde_json::to_string_pretty(&p).unwrap();
|
||||
assert!(json.contains("max_lives"));
|
||||
assert!(json.contains("3"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_apply_courier_overrides_default() {
|
||||
let mut base = Params::default();
|
||||
base.notification = Some(NotificationParams::defaults());
|
||||
let incoming = Params {
|
||||
notification: Some(NotificationParams {
|
||||
max_lives: ParamVal::with(5, Writer::Courier),
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
};
|
||||
let ov = base.apply(incoming, Writer::Courier);
|
||||
let s = base.to_settings().unwrap();
|
||||
assert_eq!(s.notification.max_lives, 5);
|
||||
assert!(ov.is_empty()); // default had no writer, not an "override"
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_apply_upstream_wins() {
|
||||
let mut base = Params::default();
|
||||
base.notification = Some(NotificationParams {
|
||||
max_lives: ParamVal::with(3, Writer::Base),
|
||||
..Default::default()
|
||||
});
|
||||
let incoming = Params {
|
||||
notification: Some(NotificationParams {
|
||||
max_lives: ParamVal::with(5, Writer::Courier),
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
};
|
||||
let ov = base.apply(incoming, Writer::Courier);
|
||||
let s = base.to_settings().unwrap();
|
||||
// base(3) > courier(2) — rejected, base stays
|
||||
assert_eq!(s.notification.max_lives, 3);
|
||||
assert!(ov.is_empty()); // nothing overridden
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_deny_unknown_fields() {
|
||||
let bad = r#"{"max_lives":{"value":5},"ghost_param":{"value":1}}"#;
|
||||
let result: Result<NotificationParams, _> = serde_json::from_str(bad);
|
||||
assert!(result.is_err()); // unknown field should be denied
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
use crate::apply_field;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct NotificationParams {
|
||||
#[serde(default)] pub max_lives: ParamVal<u32>,
|
||||
#[serde(default)] pub timeout_ms: ParamVal<u64>,
|
||||
#[serde(default)] pub retry_backoff: ParamVal<f64>,
|
||||
#[serde(default)] pub retry_delay_seconds: ParamVal<f64>,
|
||||
#[serde(default)] pub max_retry_delay_seconds: ParamVal<f64>,
|
||||
#[serde(default)] pub life_impact_factor: ParamVal<f64>,
|
||||
#[serde(default)] pub batch_period: ParamVal<u32>,
|
||||
#[serde(default)] pub batch_watermark: ParamVal<u32>,
|
||||
#[serde(default)] pub template_ref_depth: ParamVal<u32>,
|
||||
}
|
||||
|
||||
impl MergeParams for NotificationParams {
|
||||
fn defaults() -> Self {
|
||||
Self {
|
||||
max_lives: ParamVal::new(3), timeout_ms: ParamVal::new(30000),
|
||||
retry_backoff: ParamVal::new(2.0), retry_delay_seconds: ParamVal::new(5.0),
|
||||
max_retry_delay_seconds: ParamVal::new(300.0),
|
||||
life_impact_factor: ParamVal::new(0.0),
|
||||
batch_period: ParamVal::new(5), batch_watermark: ParamVal::new(1),
|
||||
template_ref_depth: ParamVal::new(10),
|
||||
}
|
||||
}
|
||||
fn apply(&mut self, incoming: Self, writer: Writer, ov: &mut Overridden, prefix: &str) {
|
||||
apply_field!(self.max_lives, incoming.max_lives, writer, ov, prefix, "max_lives");
|
||||
apply_field!(self.timeout_ms, incoming.timeout_ms, writer, ov, prefix, "timeout_ms");
|
||||
apply_field!(self.retry_backoff, incoming.retry_backoff, writer, ov, prefix, "retry_backoff");
|
||||
apply_field!(self.retry_delay_seconds, incoming.retry_delay_seconds, writer, ov, prefix, "retry_delay_seconds");
|
||||
apply_field!(self.max_retry_delay_seconds, incoming.max_retry_delay_seconds, writer, ov, prefix, "max_retry_delay_seconds");
|
||||
apply_field!(self.life_impact_factor, incoming.life_impact_factor, writer, ov, prefix, "life_impact_factor");
|
||||
apply_field!(self.batch_period, incoming.batch_period, writer, ov, prefix, "batch_period");
|
||||
apply_field!(self.batch_watermark, incoming.batch_watermark, writer, ov, prefix, "batch_watermark");
|
||||
apply_field!(self.template_ref_depth, incoming.template_ref_depth, writer, ov, prefix, "template_ref_depth");
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for NotificationParams { fn default() -> Self { Self::defaults() } }
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct NotificationSettings {
|
||||
pub max_lives: u32, pub timeout_ms: u64, pub retry_backoff: f64,
|
||||
pub retry_delay_seconds: f64, pub max_retry_delay_seconds: f64,
|
||||
pub life_impact_factor: f64,
|
||||
pub batch_period: u32, pub batch_watermark: u32, pub template_ref_depth: u32,
|
||||
}
|
||||
|
||||
impl Default for NotificationSettings { fn default() -> Self { NotificationParams::defaults().into() } }
|
||||
|
||||
impl From<&NotificationParams> for NotificationSettings {
|
||||
fn from(p: &NotificationParams) -> Self { Self {
|
||||
max_lives: p.max_lives.value, timeout_ms: p.timeout_ms.value,
|
||||
retry_backoff: p.retry_backoff.value, retry_delay_seconds: p.retry_delay_seconds.value,
|
||||
max_retry_delay_seconds: p.max_retry_delay_seconds.value,
|
||||
life_impact_factor: p.life_impact_factor.value,
|
||||
batch_period: p.batch_period.value, batch_watermark: p.batch_watermark.value,
|
||||
template_ref_depth: p.template_ref_depth.value,
|
||||
}}
|
||||
}
|
||||
impl From<NotificationParams> for NotificationSettings { fn from(p: NotificationParams) -> Self { (&p).into() } }
|
||||
@@ -0,0 +1,32 @@
|
||||
use serde::{Deserialize, Serialize};
|
||||
use crate::Writer;
|
||||
|
||||
/// A strongly-typed parameter value with writer tracking.
|
||||
///
|
||||
/// Serializes as `{ "value": 3, "writer": "base" }`.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct ParamVal<T> {
|
||||
pub value: T,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub writer: Option<Writer>,
|
||||
}
|
||||
|
||||
impl<T: Default> Default for ParamVal<T> {
|
||||
fn default() -> Self {
|
||||
ParamVal { value: T::default(), writer: None }
|
||||
}
|
||||
}
|
||||
|
||||
impl<T> ParamVal<T> {
|
||||
pub fn new(value: T) -> Self {
|
||||
ParamVal { value, writer: None }
|
||||
}
|
||||
pub fn with(value: T, writer: Writer) -> Self {
|
||||
ParamVal { value, writer: Some(writer) }
|
||||
}
|
||||
/// Whether an incoming value from `incoming_writer` should override this one.
|
||||
pub fn should_override(&self, incoming: Writer) -> bool {
|
||||
let current = self.writer.map_or(0, |w| w.priority());
|
||||
incoming.priority() > current
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
pub mod network;
|
||||
pub mod cache;
|
||||
pub mod engine;
|
||||
|
||||
use crate::apply_field;
|
||||
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct PrebakeParams {
|
||||
#[serde(default)] pub bootstrap_user: ParamVal<String>,
|
||||
#[serde(default)] pub workspace: ParamVal<String>,
|
||||
#[serde(default)] pub network: network::NetworkParams,
|
||||
#[serde(default)] pub cache: cache::CacheParams,
|
||||
#[serde(default)] pub engine: engine::EngineParams,
|
||||
#[serde(default)] pub repology_endpoint: ParamVal<String>,
|
||||
}
|
||||
|
||||
impl MergeParams for PrebakeParams {
|
||||
fn defaults() -> Self {
|
||||
Self {
|
||||
bootstrap_user: ParamVal::new("vulcan".into()), workspace: ParamVal::new("/home/vulcan/workspace".into()),
|
||||
network: network::NetworkParams::defaults(), cache: cache::CacheParams::defaults(),
|
||||
engine: engine::EngineParams::defaults(),
|
||||
repology_endpoint: ParamVal::new("default".into()),
|
||||
}
|
||||
}
|
||||
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
|
||||
apply_field!(self.bootstrap_user, i.bootstrap_user, w, ov, p, "bootstrap_user");
|
||||
apply_field!(self.workspace, i.workspace, w, ov, p, "workspace");
|
||||
apply_field!(self.repology_endpoint, i.repology_endpoint, w, ov, p, "repology_endpoint");
|
||||
self.network.apply(i.network, w, ov, &format!("{}network.", p));
|
||||
self.cache.apply(i.cache, w, ov, &format!("{}cache.", p));
|
||||
self.engine.apply(i.engine, w, ov, &format!("{}engine.", p));
|
||||
}
|
||||
}
|
||||
impl Default for PrebakeParams { fn default() -> Self { Self::defaults() } }
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct PrebakeSettings {
|
||||
pub bootstrap_user: String, pub workspace: String,
|
||||
pub network: network::NetworkSettings, pub cache: cache::CacheSettings,
|
||||
pub engine: engine::EngineSettings, pub repology_endpoint: String,
|
||||
}
|
||||
impl Default for PrebakeSettings { fn default() -> Self { PrebakeParams::defaults().into() } }
|
||||
impl From<&PrebakeParams> for PrebakeSettings {
|
||||
fn from(p: &PrebakeParams) -> Self { Self {
|
||||
bootstrap_user: p.bootstrap_user.value.clone(), workspace: p.workspace.value.clone(),
|
||||
network: (&p.network).into(), cache: (&p.cache).into(), engine: (&p.engine).into(),
|
||||
repology_endpoint: p.repology_endpoint.value.clone(),
|
||||
}}
|
||||
}
|
||||
impl From<PrebakeParams> for PrebakeSettings { fn from(p: PrebakeParams) -> Self { (&p).into() } }
|
||||
@@ -0,0 +1,37 @@
|
||||
use crate::apply_field;
|
||||
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct CacheParams {
|
||||
#[serde(default)] pub strategy: ParamVal<String>,
|
||||
#[serde(default)] pub mode: ParamVal<String>,
|
||||
#[serde(default)] pub ttl_days: ParamVal<u32>,
|
||||
#[serde(default)] pub max_size_gb: ParamVal<u32>,
|
||||
#[serde(default)] pub cleanup_policy: ParamVal<String>,
|
||||
}
|
||||
impl MergeParams for CacheParams {
|
||||
fn defaults() -> Self {
|
||||
Self {
|
||||
strategy: ParamVal::new("always".into()), mode: ParamVal::new("zstd".into()),
|
||||
ttl_days: ParamVal::new(30), max_size_gb: ParamVal::new(20),
|
||||
cleanup_policy: ParamVal::new("lru".into()),
|
||||
}
|
||||
}
|
||||
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
|
||||
apply_field!(self.strategy, i.strategy, w, ov, p, "strategy");
|
||||
apply_field!(self.mode, i.mode, w, ov, p, "mode");
|
||||
apply_field!(self.ttl_days, i.ttl_days, w, ov, p, "ttl_days");
|
||||
apply_field!(self.max_size_gb, i.max_size_gb, w, ov, p, "max_size_gb");
|
||||
apply_field!(self.cleanup_policy, i.cleanup_policy, w, ov, p, "cleanup_policy");
|
||||
}
|
||||
}
|
||||
impl Default for CacheParams { fn default() -> Self { Self::defaults() } }
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct CacheSettings { pub strategy: String, pub mode: String, pub ttl_days: u32, pub max_size_gb: u32, pub cleanup_policy: String }
|
||||
impl Default for CacheSettings { fn default() -> Self { CacheParams::defaults().into() } }
|
||||
impl From<&CacheParams> for CacheSettings { fn from(p: &CacheParams) -> Self {
|
||||
Self { strategy: p.strategy.value.clone(), mode: p.mode.value.clone(), ttl_days: p.ttl_days.value, max_size_gb: p.max_size_gb.value, cleanup_policy: p.cleanup_policy.value.clone() }
|
||||
} }
|
||||
impl From<CacheParams> for CacheSettings { fn from(p: CacheParams) -> Self { (&p).into() } }
|
||||
@@ -0,0 +1,21 @@
|
||||
use crate::apply_field;
|
||||
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct EngineParams {
|
||||
#[serde(default)] pub timeout_ms: ParamVal<u64>,
|
||||
}
|
||||
impl MergeParams for EngineParams {
|
||||
fn defaults() -> Self { Self { timeout_ms: ParamVal::new(300000) } }
|
||||
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
|
||||
apply_field!(self.timeout_ms, i.timeout_ms, w, ov, p, "timeout_ms");
|
||||
}
|
||||
}
|
||||
impl Default for EngineParams { fn default() -> Self { Self::defaults() } }
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct EngineSettings { pub timeout_ms: u64 }
|
||||
impl Default for EngineSettings { fn default() -> Self { EngineParams::defaults().into() } }
|
||||
impl From<&EngineParams> for EngineSettings { fn from(p: &EngineParams) -> Self { Self { timeout_ms: p.timeout_ms.value } } }
|
||||
impl From<EngineParams> for EngineSettings { fn from(p: EngineParams) -> Self { (&p).into() } }
|
||||
@@ -0,0 +1,23 @@
|
||||
use crate::apply_field;
|
||||
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct NetworkParams {
|
||||
#[serde(default)] pub enabled: ParamVal<bool>,
|
||||
#[serde(default)] pub outbound: ParamVal<bool>,
|
||||
}
|
||||
impl MergeParams for NetworkParams {
|
||||
fn defaults() -> Self { Self { enabled: ParamVal::new(true), outbound: ParamVal::new(true) } }
|
||||
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
|
||||
apply_field!(self.enabled, i.enabled, w, ov, p, "enabled");
|
||||
apply_field!(self.outbound, i.outbound, w, ov, p, "outbound");
|
||||
}
|
||||
}
|
||||
impl Default for NetworkParams { fn default() -> Self { Self::defaults() } }
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct NetworkSettings { pub enabled: bool, pub outbound: bool }
|
||||
impl Default for NetworkSettings { fn default() -> Self { NetworkParams::defaults().into() } }
|
||||
impl From<&NetworkParams> for NetworkSettings { fn from(p: &NetworkParams) -> Self { Self { enabled: p.enabled.value, outbound: p.outbound.value } } }
|
||||
impl From<NetworkParams> for NetworkSettings { fn from(p: NetworkParams) -> Self { (&p).into() } }
|
||||
@@ -0,0 +1,28 @@
|
||||
use crate::{Writer, Overridden};
|
||||
|
||||
/// A parameter tree node that can merge itself with an incoming layer.
|
||||
pub trait MergeParams: Sized {
|
||||
fn apply(&mut self, incoming: Self, writer: Writer, ov: &mut Overridden, prefix: &str);
|
||||
fn defaults() -> Self;
|
||||
}
|
||||
|
||||
/// Apply one field's merge logic.
|
||||
/// ```ignore
|
||||
/// apply_field!(self.foo, incoming.foo, writer, ov, prefix, "foo");
|
||||
/// ```
|
||||
#[macro_export]
|
||||
macro_rules! apply_field {
|
||||
($self:ident.$field:ident, $incoming:ident.$field2:ident, $writer:expr, $ov:ident, $prefix:expr, $name:expr) => {
|
||||
let old_w = $self.$field.writer;
|
||||
if $self.$field.should_override($writer) {
|
||||
let old_v = ::std::mem::replace(&mut $self.$field.value, $incoming.$field2.value);
|
||||
$self.$field.writer = ::std::option::Option::Some($writer);
|
||||
if let ::std::option::Option::Some(w) = old_w {
|
||||
$ov.entries.insert(
|
||||
format!("{}{}", $prefix, $name),
|
||||
::serde_json::json!({"value": old_v, "writer": w.as_str()}),
|
||||
);
|
||||
}
|
||||
}
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub enum Writer {
|
||||
Base,
|
||||
Courier,
|
||||
Bakerd,
|
||||
}
|
||||
|
||||
impl Writer {
|
||||
/// Priority: Base=3 > Courier=2 > Bakerd=1
|
||||
pub fn priority(self) -> u8 {
|
||||
match self {
|
||||
Writer::Base => 3,
|
||||
Writer::Courier => 2,
|
||||
Writer::Bakerd => 1,
|
||||
}
|
||||
}
|
||||
pub fn as_str(self) -> &'static str {
|
||||
match self {
|
||||
Writer::Base => "base", Writer::Courier => "courier", Writer::Bakerd => "bakerd",
|
||||
}
|
||||
}
|
||||
}
|
||||
impl Default for Writer { fn default() -> Self { Writer::Base } }
|
||||
@@ -1,3 +1,4 @@
|
||||
temp
|
||||
examples
|
||||
quicktest.sh
|
||||
_env.sh
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
# WORKSHOP-BAKER CRATE KNOWLEDGE BASE
|
||||
|
||||
## OVERVIEW
|
||||
Primary crate — the only crate with a `[[bin]]` target. Contains the `workshop-baker` binary and `workshop_baker` library. Orchestrates the full CI/CD pipeline: prebake (env setup) → bake (script execution) → finalize (plugins, cleanup) → drain notification queue. ~69 .rs source files.
|
||||
|
||||
## STRUCTURE
|
||||
```
|
||||
src/
|
||||
├── main.rs # Entry: default(full pipeline) | bare(dev-only) | daemon(unimplemented)
|
||||
├── lib.rs # Clap Cli/Commands/BareCommands — CLI structs in lib, not bin
|
||||
├── error.rs # CliError enum (wraps stage errors, delegates exit_code)
|
||||
├── bare.rs # Dev-only single-stage execution, ctx.privileged always false
|
||||
├── utils.rs # Cross-cutting helpers
|
||||
├── bake/ # Script parsing + DAG scheduling + execution
|
||||
│ ├── decorator.rs # @decorator(args) parsing, 81 tests — heaviest coverage in crate
|
||||
│ ├── parser.rs # Step/function extraction
|
||||
│ ├── schedule.rs # DAG build → topological sort
|
||||
│ ├── execute.rs # Per-step timeout/retry
|
||||
│ ├── builder.rs # Script assembly
|
||||
│ ├── trivial.rs # Trivial step detection (skip in dry-run)
|
||||
│ ├── constant.rs / error.rs / util.rs
|
||||
├── prebake/ # Environment provisioning
|
||||
│ ├── config.rs # PrebakeConfig deserialization
|
||||
│ ├── stage.rs + stage/{bootstrap,depssystem,depsuser,environment,hook}.rs
|
||||
│ ├── env.rs + env/{container,firecracker,baremetal,custom}.rs # Builder dispatch
|
||||
│ ├── security.rs # Privilege/sandbox checks
|
||||
│ ├── event.rs # Event sender/receiver channel
|
||||
│ ├── types.rs + types/{memsize,cache,builderconfig,architecture}.rs
|
||||
│ ├── constant.rs / error.rs
|
||||
├── finalize/ # Post-build hooks + plugins
|
||||
│ ├── config.rs # FinalizeConfig
|
||||
│ ├── stage.rs + stage/hook.rs
|
||||
│ ├── template.rs # Output templating (minijinja)
|
||||
│ ├── plugin.rs + plugin/{shell,dylib,rhai,internal}.rs
|
||||
│ │ └── plugin/internal/{dummy,mail,satori,webhook,insitenotify}.rs
|
||||
│ ├── types.rs + types/compression.rs
|
||||
│ ├── constant.rs / error.rs / event.rs
|
||||
├── notify/ # Async notification dispatch
|
||||
│ ├── queue.rs # NotificationQueue — drain on pipeline complete
|
||||
│ ├── handler.rs / rule.rs / template.rs / types.rs / util.rs / error.rs
|
||||
└── types/ # Shared types — ZERO deps on stage modules
|
||||
├── resource.rs # ResourceRegistry (populated pre-pipeline, consumed read-only)
|
||||
└── buildstatus.rs
|
||||
```
|
||||
|
||||
## WHERE TO LOOK
|
||||
|
||||
| Task | Location | Notes |
|
||||
|------|----------|-------|
|
||||
| CLI args + subcommands | `src/lib.rs` | Cli, Commands(Bare/Daemon), BareCommands |
|
||||
| Pipeline orchestration | `src/main.rs` | worker_main(): prebake→bake→finalize→queue.drain() |
|
||||
| Decorator syntax | `src/bake/decorator.rs` | `# @decorator` or `# @decorator(args)`, line-by-line regex |
|
||||
| Build execution | `src/bake/execute.rs` | timeout, retry, exit code capture |
|
||||
| DAG scheduling | `src/bake/schedule.rs` | Uses workshop-schedule crate |
|
||||
| Docker builder | `src/prebake/env/container.rs` | Bollard-based, buildkit support |
|
||||
| Firecracker / Bare metal | `.../firecracker.rs`, `.../baremetal.rs` | MicroVM;
|
||||
needs PIPELINE_BAREMETAL_ELEVATE |
|
||||
| Plugin dispatch | `src/finalize/plugin.rs` | Dispatches shell / dylib(stub) / rhai(stub) / internal |
|
||||
| Notification queue | `src/notify/queue.rs` | Spawned early, drained after pipeline, retry support |
|
||||
| Error type | `src/error.rs` | CliError wraps PrebakeError/BakeError/FinalizeError/anyhow |
|
||||
| Dev bare mode | `src/bare.rs` | ctx.privileged=false always; do not extrapolate to production |
|
||||
|
||||
## DEPENDENCIES
|
||||
**Internal:** workshop-engine, workshop-schedule, workshop-getterurl, workshop-baker-params. **External:** bollard (Docker), clap (CLI), axum 0.7 (HTTP), lettre 0.11 (email), minijinja (templates), cgroups-rs (limits), privdrop.
|
||||
|
||||
## CONVENTIONS (CRATE-SPECIFIC)
|
||||
- **Lint overrides** (this crate only): `dead_code=allow`, `unreachable_code=allow`, `inherent_to_string=allow`, `non_canonical_partial_ord_impl=allow`
|
||||
- **CLI structs in lib.rs, not main.rs** — enables integration tests to import and construct Cli directly.
|
||||
- **Pipeline stages return `anyhow::Result<()>`** — stages propagate with `?`; only main.rs converts to CliError.
|
||||
- **ResourceRegistry pattern** — populated before pipeline stages, consumed read-only by stages. New resource types go in `types/resource.rs`.
|
||||
- **Notification queue fire-and-forget + drain** — notify tasks spawned early, send during pipeline, drain on completion. `NotificationQueue::drain()` must be called.
|
||||
- **Trivial step detection** — steps with no external deps can be marked trivial (`trivial.rs`), skipped in dry-run.
|
||||
## ANTI-PATTERNS (THIS CRATE)
|
||||
- **Do NOT derive production behavior from bare mode** — ctx.privileged is hardcoded false.
|
||||
- **Do NOT add CLI args bypassing prebake→bake→finalize ordering** — the pipeline is linear. Parallelism lives in the DAG scheduler within bake.
|
||||
- **Do NOT import types/ from bake/, prebake/, or finalize/** — `types/` must have zero crate-internal deps.
|
||||
- **Do NOT add lint allows without justification** — the four existing overrides are intentional.
|
||||
- **Do NOT spawn blocking work on tokio runtime** — use `spawn_blocking` for CPU-heavy tasks.
|
||||
- **Decorator regex is line-by-line** — multi-line decorators need a parser rewrite, not a regex hack.
|
||||
- **Careful with `unimplemented!()`** — Daemon, Rhai/Dylib plugins, Custom builder are stubs. Prefer error over crash.
|
||||
Generated
+376
-1201
File diff suppressed because it is too large
Load Diff
@@ -41,20 +41,23 @@ tar = "0.4.44"
|
||||
tempfile = "3.24.0"
|
||||
thiserror = "2.0.17"
|
||||
tokio = { version = "1.48.0", features = ["full"] }
|
||||
topological-sort = "0.2.2"
|
||||
workshop-schedule = { path = "../workshop-schedule" }
|
||||
async-trait = "0.1.87"
|
||||
uuid = { version = "1.19.0", features = ["v4"] }
|
||||
libc = "0.2"
|
||||
os_info = "3.14.0"
|
||||
privdrop = "0.5.6"
|
||||
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
|
||||
which = "8.0.2"
|
||||
gix = "0.81.0"
|
||||
sha2 = "0.10"
|
||||
flate2 = "1.0"
|
||||
zstd = "0.13"
|
||||
globset = "0.4.18"
|
||||
axum = "0.7"
|
||||
workshop-getterurl = { path = "../workshop-getterurl" }
|
||||
workshop-baker-params = { version = "0.1.0", path = "../workshop-baker-params" }
|
||||
|
||||
bitflags = "2"
|
||||
url = "2.5.8"
|
||||
zstd = "0.13.3"
|
||||
flate2 = "1.1.9"
|
||||
capctl = "0.2.4"
|
||||
[dev-dependencies]
|
||||
criterion = "0.5"
|
||||
|
||||
@@ -66,6 +69,3 @@ unreachable_code = "allow"
|
||||
inherent_to_string = "allow"
|
||||
non_canonical_partial_ord_impl = "allow"
|
||||
|
||||
[[bench]]
|
||||
name = "parser_bench"
|
||||
harness = false
|
||||
|
||||
@@ -1,93 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
import os
|
||||
import random
|
||||
import string
|
||||
|
||||
OUTPUT_DIR = "/tmp"
|
||||
DECORATORS = [
|
||||
"# @pipeline",
|
||||
"# @fallible",
|
||||
"# @parallel",
|
||||
"# @export",
|
||||
"# @timeout(60)",
|
||||
"# @retry(3, 5)",
|
||||
"# @if(condition)",
|
||||
"# @after(dep)",
|
||||
"# @pipe(step1, step2)",
|
||||
"# @health(/health)",
|
||||
"# @loop(10)",
|
||||
]
|
||||
FUNCTION_BODIES = [
|
||||
"echo 'Processing...';",
|
||||
"local var=$(date +%s);",
|
||||
"if [ -f /tmp/test ]; then echo 'exists'; fi;",
|
||||
"for i in $(seq 1 10); do echo $i; done;",
|
||||
"case $1 in start) echo 'starting';; stop) echo 'stopping';; esac;",
|
||||
"read -r line < /dev/stdin;",
|
||||
"export PATH=$PATH:/usr/local/bin;",
|
||||
"set -e; set -u;",
|
||||
"trap 'echo error' ERR;",
|
||||
"cd /tmp || exit 1;",
|
||||
"local result=$(grep -r 'pattern' /var/log/ 2>/dev/null);",
|
||||
'for f in /tmp/*.tmp; do rm -f "$f"; done',
|
||||
"if [[ $var -gt 100 ]]; then echo 'large'; fi;",
|
||||
'while read -r line; do echo "$line"; done < /etc/passwd;',
|
||||
'select opt in a b c; do echo "Selected $opt"; break; done;',
|
||||
]
|
||||
|
||||
|
||||
def random_string(length=8):
|
||||
return "".join(random.choices(string.ascii_letters + string.digits, k=length))
|
||||
|
||||
|
||||
def generate_function(body_lines):
|
||||
lines = []
|
||||
func_name = f"func_{random_string(6)}"
|
||||
for _ in range(random.randint(0, 3)):
|
||||
lines.append(random.choice(DECORATORS))
|
||||
lines.append(f"{func_name}() {{")
|
||||
for _ in range(body_lines):
|
||||
lines.append(" " + random.choice(FUNCTION_BODIES))
|
||||
lines.append("}")
|
||||
lines.append("")
|
||||
return lines
|
||||
|
||||
|
||||
def generate_script(target_lines, num_functions, target_bytes):
|
||||
lines = []
|
||||
total_lines = 0
|
||||
body_per_func = max(1, (target_lines - num_functions * 3) // num_functions)
|
||||
|
||||
for _ in range(num_functions):
|
||||
func_lines = generate_function(body_per_func)
|
||||
lines.extend(func_lines)
|
||||
total_lines += len(func_lines)
|
||||
|
||||
while (
|
||||
total_lines < target_lines
|
||||
or len("\n".join(lines).encode("utf-8")) < target_bytes
|
||||
):
|
||||
lines.append(random.choice(FUNCTION_BODIES))
|
||||
total_lines += 1
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main():
|
||||
sizes = [
|
||||
("small", 1000, 10, 50_000),
|
||||
("medium", 5000, 50, 500_000),
|
||||
("large", 10000, 100, 950_000),
|
||||
]
|
||||
for name, lines, funcs, target_bytes in sizes:
|
||||
script = generate_script(lines, funcs, target_bytes)
|
||||
filepath = os.path.join(OUTPUT_DIR, f"bench_{name}.sh")
|
||||
with open(filepath, "w") as f:
|
||||
f.write(script)
|
||||
actual_bytes = len(script.encode("utf-8"))
|
||||
actual_lines = len(script.splitlines())
|
||||
print(f"Generated {filepath}: lines={actual_lines}, bytes={actual_bytes}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,61 +0,0 @@
|
||||
use criterion::{Criterion, black_box, criterion_group, criterion_main};
|
||||
use std::fs;
|
||||
use std::path::Path;
|
||||
|
||||
fn parse_large_script(c: &mut Criterion) {
|
||||
let script_path = Path::new("/tmp/bench_large.sh");
|
||||
|
||||
let script_content = fs::read_to_string(script_path)
|
||||
.expect("Failed to read benchmark script. Run `python3 benches/generate.py` first.");
|
||||
|
||||
let byte_size = script_content.len();
|
||||
let line_count = script_content.lines().count();
|
||||
|
||||
println!("Benchmark file: {} lines, {} bytes", line_count, byte_size);
|
||||
|
||||
c.bench_function("parse_script_large", |b| {
|
||||
b.iter(|| {
|
||||
let result =
|
||||
workshop_baker::bake::parser::parse_script(black_box(&script_content.clone()));
|
||||
black_box(result)
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
fn parse_medium_script(c: &mut Criterion) {
|
||||
let script_path = Path::new("/tmp/bench_medium.sh");
|
||||
|
||||
let script_content = fs::read_to_string(script_path)
|
||||
.expect("Failed to read benchmark script. Run `python3 benches/generate.py` first.");
|
||||
|
||||
c.bench_function("parse_script_medium", |b| {
|
||||
b.iter(|| {
|
||||
let result =
|
||||
workshop_baker::bake::parser::parse_script(black_box(&script_content.clone()));
|
||||
black_box(result)
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
fn parse_small_script(c: &mut Criterion) {
|
||||
let script_path = Path::new("/tmp/bench_small.sh");
|
||||
|
||||
let script_content = fs::read_to_string(script_path)
|
||||
.expect("Failed to read benchmark script. Run `python3 benches/generate.py` first.");
|
||||
|
||||
c.bench_function("parse_script_small", |b| {
|
||||
b.iter(|| {
|
||||
let result =
|
||||
workshop_baker::bake::parser::parse_script(black_box(&script_content.clone()));
|
||||
black_box(result)
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
criterion_group!(
|
||||
benches,
|
||||
parse_small_script,
|
||||
parse_medium_script,
|
||||
parse_large_script
|
||||
);
|
||||
criterion_main!(benches);
|
||||
@@ -1,11 +0,0 @@
|
||||
[pytest]
|
||||
testpaths = tests/integration
|
||||
python_files = test_*.py
|
||||
python_classes = Test*
|
||||
python_functions = test_*
|
||||
addopts = -v --tb=short
|
||||
markers =
|
||||
integration: marks tests as integration tests (deselect with '-m "not integration"')
|
||||
slow: marks tests as slow running
|
||||
filterwarnings =
|
||||
ignore::DeprecationWarning
|
||||
@@ -1,62 +0,0 @@
|
||||
# workshop-baker/src
|
||||
|
||||
**Generated:** 2026-04-22
|
||||
**Commit:** 7b3b71a
|
||||
|
||||
Baker CLI + execution engine for HoneyBiscuitWorkshop pipelines.
|
||||
|
||||
## OVERVIEW
|
||||
|
||||
CLI tool and runtime for executing CI/CD pipelines with resource isolation, privilege dropping, and package management integration.
|
||||
|
||||
## STRUCTURE
|
||||
|
||||
```
|
||||
src/
|
||||
├── bake.rs # Top-level bake orchestration
|
||||
├── bake/ # Script parsing and building
|
||||
│ ├── parser.rs # @decorator-aware shell parser
|
||||
│ ├── decorator.rs # @pipeline, @retry, @parallel, etc.
|
||||
│ ├── builder.rs # Script template builder
|
||||
│ └── schedule.rs # Function ordering via @after deps
|
||||
├── engine/ # Execution runtime
|
||||
│ ├── executor.rs # Script execution with cgroups
|
||||
│ ├── cgroups.rs # Linux cgroup resource limits
|
||||
│ ├── repology.rs # Package name resolution
|
||||
│ ├── pm.rs # Package manager detection
|
||||
│ └── upm.rs # User package manager (Nix/Guix)
|
||||
├── prebake/ # Build environment setup
|
||||
│ ├── config.rs # PrebakeConfig YAML schema
|
||||
│ ├── stage/ # Bootstrap, DepsSystem, DepsUser, Hooks
|
||||
│ └── security.rs # Privilege dropping logic
|
||||
├── types/ # Shared type definitions
|
||||
│ ├── repology.rs # RepologyEndpoint enum
|
||||
│ ├── builderconfig.rs
|
||||
│ ├── cache.rs
|
||||
│ └── memsize.rs
|
||||
├── monitor/ # Resource usage monitoring
|
||||
│ ├── cgroups.rs # cgroup-based usage collection
|
||||
│ └── jobobject.rs # Windows job objects (stub)
|
||||
├── daemon.rs # Unix socket daemon (TODO: refactor)
|
||||
└── lib.rs # CLI struct + module exports
|
||||
```
|
||||
|
||||
## WHERE TO LOOK
|
||||
|
||||
| Task | Location |
|
||||
|------|----------|
|
||||
| Script parsing with @decorators | `bake/parser.rs:29` - `parse_script()` |
|
||||
| Decorator types | `bake/decorator.rs` - `Decorator` enum |
|
||||
| Build execution | `engine/executor.rs` - `Executor::execute_script()` |
|
||||
| Resource limits | `engine/types.rs:56` - `ResourceLimits` struct |
|
||||
| Prebake stages | `prebake/stage/` subdirectories |
|
||||
| Package detection | `engine/pm.rs:detection()` |
|
||||
| CLI entry | `lib.rs:26` - `cli::Cli` struct |
|
||||
|
||||
## CONVENTIONS
|
||||
|
||||
- **ExecutionContext**: Passed through all stages, carries task_id, username, env_vars, timeout
|
||||
- **PrebakeStage**: Ordered enum controlling stage execution sequence (Bootstrap -> Ready)
|
||||
- **Decorator**: Attached to shell functions via `# @name(args)` comments above function declarations
|
||||
- **Engine**: Lightweight wrapper around cgroup manager for resource isolation
|
||||
- **Dual-mode**: CLI commands (prebake, bake, finalize) vs daemon mode via Unix socket
|
||||
+177
-55
@@ -1,20 +1,27 @@
|
||||
use crate::bake::constant::{DEFAULT_WORKSPACE, TRIVIAL_SCRIPT_NAME};
|
||||
use crate::types::debug::DebugFeature;
|
||||
use crate::bake::builder::RenderMode;
|
||||
use crate::bake::constant::DEFAULT_WORKSPACE;
|
||||
use crate::bake::decorator::Decorator;
|
||||
use crate::bake::error::BakeError;
|
||||
use crate::bake::parser::Function;
|
||||
use crate::bake::util::{resolve_mode, FuncMode};
|
||||
use crate::cli::Cli;
|
||||
use crate::prebake;
|
||||
use crate::prebake::PrebakeConfig;
|
||||
use crate::prebake::security::get_drop_after;
|
||||
use crate::prebake::stage::PrebakeStage;
|
||||
use std::collections::{HashMap, HashSet};
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::time::Duration;
|
||||
use workshop_baker_params::BakeSettings;
|
||||
use workshop_engine::{Engine, EventSender, ExecutionContext};
|
||||
|
||||
mod builder;
|
||||
pub mod constant;
|
||||
mod decorator;
|
||||
pub mod error;
|
||||
mod execute;
|
||||
pub mod parser;
|
||||
mod schedule;
|
||||
pub mod trivial;
|
||||
pub mod util;
|
||||
|
||||
pub async fn bake(
|
||||
script_path: &Path,
|
||||
@@ -23,12 +30,18 @@ pub async fn bake(
|
||||
_cli: &Cli,
|
||||
ctx: &mut ExecutionContext,
|
||||
event_tx: EventSender,
|
||||
nevent_tx: Option<crate::notify::types::NotificationEventSender>,
|
||||
) -> Result<(), BakeError> {
|
||||
let script_content = std::fs::read_to_string(script_path)?;
|
||||
let prebake_content = std::fs::read_to_string(prebake_path)?;
|
||||
log::debug!("Reading script: {:?}", script_path);
|
||||
let script_content = std::fs::read_to_string(script_path).map_err(|e| {
|
||||
BakeError::IoError { path: script_path.display().to_string(), source: e }
|
||||
})?;
|
||||
log::debug!("Reading prebake.yml: {:?}", prebake_path);
|
||||
let prebake_content = std::fs::read_to_string(prebake_path).map_err(|e| {
|
||||
BakeError::IoError { path: prebake_path.display().to_string(), source: e }
|
||||
})?;
|
||||
let prebake = prebake::parse(prebake_content.as_str()).map_err(|e| {
|
||||
log::error!("Failed to parse prebake.yml: {}", e);
|
||||
log::error!("This is unexpected!");
|
||||
BakeError::YamlParseError(e)
|
||||
})?;
|
||||
let workspace = get_workspace(&prebake);
|
||||
@@ -39,44 +52,174 @@ pub async fn bake(
|
||||
.iter()
|
||||
.any(|d| matches!(d, decorator::Decorator::Pipeline))
|
||||
});
|
||||
let trivial = false;
|
||||
let use_template = true;
|
||||
// TODO: if PIP(pipeline implicit parameter).trivial is set, always use RenderMode::Off
|
||||
let render_mode = RenderMode::Full;
|
||||
|
||||
let engine = Engine::new();
|
||||
// TODO: resolve from merged param layers (PIP integration)
|
||||
let bake_settings = BakeSettings::default();
|
||||
if let Some(env) = &prebake.envvars
|
||||
&& let Some(prebake_env) = &env.bake
|
||||
{
|
||||
ctx.env_vars.extend(prebake_env.clone());
|
||||
}
|
||||
|
||||
if !has_pipeline || trivial {
|
||||
let function = Function {
|
||||
name: TRIVIAL_SCRIPT_NAME.to_string(),
|
||||
decorators: vec![],
|
||||
body: script_content,
|
||||
};
|
||||
let script =
|
||||
builder::build_script(&function, bake_base_path, &workspace, None, use_template)?;
|
||||
let result = engine.execute_script(&script, ctx, &event_tx).await;
|
||||
result?;
|
||||
if !has_pipeline {
|
||||
log::trace!("Running trivial script");
|
||||
trivial::run_trivial(
|
||||
&script_content,
|
||||
bake_base_path,
|
||||
&workspace,
|
||||
&engine,
|
||||
ctx,
|
||||
&event_tx,
|
||||
render_mode,
|
||||
)
|
||||
.await?;
|
||||
} else {
|
||||
let sorted_functions = schedule::sort_function(functions.pipeline_functions)?;
|
||||
log::trace!("Running pipeline script");
|
||||
let remaining = functions.remaining_code;
|
||||
for func in sorted_functions {
|
||||
let mut modified_func = func.clone();
|
||||
modified_func.body = remaining.clone() + &func.body;
|
||||
let script = builder::build_script(
|
||||
&modified_func,
|
||||
bake_base_path,
|
||||
&workspace,
|
||||
None,
|
||||
use_template,
|
||||
)?;
|
||||
let result = engine.execute_script(&script, ctx, &event_tx).await;
|
||||
result?;
|
||||
let (mut dag, func_map) = schedule::build_dag(functions.pipeline_functions)?;
|
||||
|
||||
// Pre-scan: pipe sources + decorator conflict checks.
|
||||
let mut pipe_sources: HashSet<String> = HashSet::new();
|
||||
for func in func_map.values() {
|
||||
decorator::check_conflicts(func)?;
|
||||
for decorator in &func.decorators {
|
||||
if let Decorator::Pipe(names) = decorator {
|
||||
for source in names {
|
||||
if *source != func.name {
|
||||
pipe_sources.insert(source.clone());
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let mut stdout_cache: HashMap<String, String> = HashMap::new();
|
||||
let mut async_handles: Vec<tokio::task::JoinHandle<()>> = Vec::new();
|
||||
|
||||
while !dag.is_empty() {
|
||||
let scheduler_debug = DebugFeature::from_bits_retain(ctx.debug_flags).contains(DebugFeature::Scheduler);
|
||||
if scheduler_debug {
|
||||
log::info!("[BAKE_SCHED] ready={} seq={} wild={} groups={:?}",
|
||||
dag.ready_count(),
|
||||
dag.ready_sequential().len(),
|
||||
dag.ready_wildcards().len(),
|
||||
dag.ready_groups(),
|
||||
);
|
||||
} else {
|
||||
log::debug!("[BAKE_SCHED] ready={} seq={} wild={} groups={:?}",
|
||||
dag.ready_count(),
|
||||
dag.ready_sequential().len(),
|
||||
dag.ready_wildcards().len(),
|
||||
dag.ready_groups(),
|
||||
);
|
||||
}
|
||||
// Rule 1: sequential always goes first
|
||||
let seq_candidates = dag.ready_sequential();
|
||||
if let Some(seq_name) = seq_candidates.first() {
|
||||
let seq_name = seq_name.to_string();
|
||||
log::info!("[BAKE_SCHED] seq {}", seq_name);
|
||||
let func = &func_map[&seq_name];
|
||||
let mode = resolve_mode(func);
|
||||
|
||||
match mode {
|
||||
FuncMode::Daemon => {
|
||||
execute::execute_daemon(
|
||||
&engine, func, ctx, &event_tx,
|
||||
&workspace, &remaining, bake_base_path,
|
||||
Duration::from_secs(bake_settings.health_period),
|
||||
Duration::from_secs(bake_settings.health_timeout),
|
||||
bake_settings.health_max_retries,
|
||||
).await?;
|
||||
}
|
||||
FuncMode::Async => {
|
||||
let handle = execute::execute_async(
|
||||
func, ctx, &event_tx,
|
||||
&workspace, &remaining, bake_base_path,
|
||||
);
|
||||
async_handles.push(handle);
|
||||
}
|
||||
FuncMode::Normal => {
|
||||
let result = execute::execute_normal(
|
||||
&engine, func, ctx, &event_tx,
|
||||
&workspace, &remaining, bake_base_path, &stdout_cache,
|
||||
).await;
|
||||
match result {
|
||||
Ok(stdout) => {
|
||||
if pipe_sources.contains(&seq_name) {
|
||||
stdout_cache.insert(seq_name.clone(), stdout);
|
||||
}
|
||||
}
|
||||
Err(e) if dag.is_fallible(&seq_name) => {
|
||||
log::warn!("'{}' failed (fallible): {}", seq_name, e);
|
||||
}
|
||||
Err(e) => return Err(e),
|
||||
}
|
||||
}
|
||||
}
|
||||
// Pop AFTER execution (Daemon/Async pop at start)
|
||||
if !matches!(mode, FuncMode::Daemon | FuncMode::Async) {
|
||||
dag.pop(&seq_name).map_err(|e| BakeError::IncompatibleDecorators {
|
||||
function: seq_name.clone(),
|
||||
reason: e.to_string(),
|
||||
})?;
|
||||
} else {
|
||||
dag.pop(&seq_name).map_err(|e| BakeError::IncompatibleDecorators {
|
||||
function: seq_name.clone(),
|
||||
reason: e.to_string(),
|
||||
})?;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
// All remaining are parallel — pick a batch
|
||||
if let Some(batch) = dag.pick_batch() {
|
||||
log::info!("[BAKE_SCHED] par {:?}", batch);
|
||||
let results = execute::fire_parallel_batch(
|
||||
&batch, &engine, &func_map, ctx, &event_tx,
|
||||
&workspace, &remaining, bake_base_path, &stdout_cache,
|
||||
).await?;
|
||||
for (name, stdout) in results {
|
||||
if pipe_sources.contains(&name) {
|
||||
stdout_cache.insert(name.clone(), stdout);
|
||||
}
|
||||
if let Err(e) = dag.pop(&name) {
|
||||
log::warn!("Failed to pop '{}' from DAG: {}", name, e);
|
||||
}
|
||||
}
|
||||
} else if dag.is_stalled() {
|
||||
// Deadlock: named parallel groups exist but can't complete
|
||||
let stalled = dag.stalled_groups();
|
||||
let remaining = dag.remaining();
|
||||
return Err(BakeError::IncompatibleDecorators {
|
||||
function: remaining.join(", "),
|
||||
reason: format!(
|
||||
"parallel group(s) {:?} can never complete — blocked by @after chain",
|
||||
stalled
|
||||
),
|
||||
});
|
||||
} else {
|
||||
// Ready set is empty — cycle
|
||||
let remaining = dag.remaining();
|
||||
if !remaining.is_empty() {
|
||||
return Err(BakeError::CircularDependency(
|
||||
remaining.into_iter().map(|s| s.to_string()).collect(),
|
||||
));
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
// Wait for all @async tasks to finish
|
||||
for handle in async_handles {
|
||||
let _ = handle.await;
|
||||
}
|
||||
}
|
||||
|
||||
crate::notify::types::send_stage_event(&nevent_tx, crate::types::buildstatus::StagePhase::Bake, "bake", crate::types::buildstatus::StageOutcome::Success).await;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -95,26 +238,5 @@ fn get_workspace(prebake_config: &PrebakeConfig) -> PathBuf {
|
||||
.unwrap_or_else(|| PathBuf::from(DEFAULT_WORKSPACE))
|
||||
}
|
||||
|
||||
fn privileged(prebake_config: &PrebakeConfig) -> bool {
|
||||
if let Some(bootstrap) = &prebake_config.bootstrap
|
||||
&& bootstrap.user == "root"
|
||||
{
|
||||
return true;
|
||||
}
|
||||
let drop_after = get_drop_after(&prebake_config.security).unwrap_or_default();
|
||||
PrebakeStage::Never == drop_after
|
||||
}
|
||||
|
||||
type TaskResult = Result<(), BakeError>;
|
||||
|
||||
type TaskFn = Box<dyn FnOnce() -> TaskResult + Send>;
|
||||
|
||||
// TODO: Learn closure and continue
|
||||
// async fn run<F>(f:F) -> TaskFn {
|
||||
// let engine = Engine::new();
|
||||
// engine.execute_script
|
||||
// }
|
||||
|
||||
// fn with_timeout<F>(f: F, duration: i32) -> TaskFn {
|
||||
|
||||
// }
|
||||
// ── AI Coding Agent marker ──
|
||||
// Code in this module PARTIALLY or FULLY utilized AI Coding Agent
|
||||
|
||||
@@ -1,46 +0,0 @@
|
||||
# workshop-baker/src/bake
|
||||
|
||||
**Generated:** 2026-04-22
|
||||
**Commit:** 7b3b71a
|
||||
|
||||
Script parsing and build orchestration with decorator DSL.
|
||||
|
||||
## STRUCTURE
|
||||
|
||||
```
|
||||
bake/
|
||||
├── parser.rs # @decorator-aware shell script parser
|
||||
├── decorator.rs # Decorator enum + 100+ unit tests
|
||||
├── builder.rs # Script template assembly
|
||||
└── schedule.rs # Function ordering via @after dependencies
|
||||
```
|
||||
|
||||
## WHERE TO LOOK
|
||||
|
||||
| Task | Location |
|
||||
|------|----------|
|
||||
| Parse script | `parser.rs:29` - `parse_script()` |
|
||||
| Decorator types | `decorator.rs` - `Decorator` enum variants |
|
||||
| Build script | `builder.rs` - template builder |
|
||||
| Schedule deps | `schedule.rs` - @after resolution |
|
||||
|
||||
## DECORATORS
|
||||
|
||||
| Decorator | Purpose |
|
||||
|-----------|---------|
|
||||
| `@pipeline` | Marks main build function |
|
||||
| `@timeout(N)` | Execution timeout seconds |
|
||||
| `@retry(N, M)` | Retry N times, M sec delay |
|
||||
| `@parallel` | Run concurrently |
|
||||
| `@fallible` | Failure doesn't fail pipeline |
|
||||
| `@after(func)` | Dependency ordering |
|
||||
| `@export` | Export vars to env |
|
||||
| `@loop(N)` | Repeat N times |
|
||||
| `@if(COND)` | Conditional execution |
|
||||
|
||||
## CONVENTIONS
|
||||
|
||||
- **Decorator Syntax**: `# @name(args)` comment above function
|
||||
- **Parsing**: Shell grammar with decorator annotation extraction
|
||||
- **Scheduling**: DAG resolution for @after dependencies
|
||||
- **Tests**: `decorator.rs:175-829` has 100+ inline unit tests
|
||||
@@ -1,4 +1,5 @@
|
||||
use crate::bake::constant::{PIPELINE_SKIP_ERRORCODE, TRIVIAL_SCRIPT_NAME};
|
||||
use crate::types::debug::DebugFeature;
|
||||
use crate::bake::constant::PIPELINE_SKIP_ERRORCODE;
|
||||
use crate::bake::decorator::Decorator;
|
||||
use crate::bake::error::BakeError;
|
||||
use crate::bake::parser::Function;
|
||||
@@ -9,24 +10,45 @@ use std::path::Path;
|
||||
use std::path::PathBuf;
|
||||
use tempfile::NamedTempFile;
|
||||
|
||||
/// Template for use_template=false, which just executes the main body without any wrapping.
|
||||
/// Three-level rendering control.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum RenderMode {
|
||||
/// No template, hardcoded "{{ main }}". Debug escape hatch.
|
||||
Off,
|
||||
/// Uses bake_base.sh with essential execution skeleton only.
|
||||
Minimal,
|
||||
/// Uses bake_base.sh with helpers (logs, colors, etc.).
|
||||
Full,
|
||||
}
|
||||
|
||||
impl RenderMode {
|
||||
pub fn as_str(&self) -> &'static str {
|
||||
match self {
|
||||
RenderMode::Off => "off",
|
||||
RenderMode::Minimal => "minimal",
|
||||
RenderMode::Full => "full",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Hardcoded template for RenderMode::Off.
|
||||
const TRIVIAL_TEMPLATE: &str = "{{ main }}";
|
||||
|
||||
/// Renders a `Function` into an executable shell script using a minijinja template.
|
||||
/// If `use_template` is true, wraps the body in `bake_base`; otherwise uses a trivial template.
|
||||
///
|
||||
/// `render_mode` controls the template source:
|
||||
/// - `Off`: hardcoded `{{ main }}`, no bake_base file needed.
|
||||
/// - `Minimal` / `Full`: reads bake_base.sh; the template distinguishes the two
|
||||
/// via the `render_mode` context variable.
|
||||
pub fn build_script(
|
||||
function: &Function,
|
||||
bake_base: &Path,
|
||||
target_dir: &Path,
|
||||
temp_path: Option<PathBuf>,
|
||||
use_template: bool,
|
||||
render_mode: RenderMode,
|
||||
debug_flags: u32,
|
||||
) -> Result<PathBuf, BakeError> {
|
||||
let name = &function.name;
|
||||
let trivial = name == TRIVIAL_SCRIPT_NAME;
|
||||
let body = &function.body;
|
||||
let export = function
|
||||
.find_decorator(|d| matches!(d, Decorator::Export).then_some(()))
|
||||
.is_some();
|
||||
let condition = function.find_decorator(|d| {
|
||||
if let Decorator::If(content) = d {
|
||||
Some(content.clone())
|
||||
@@ -34,57 +56,59 @@ pub fn build_script(
|
||||
None
|
||||
}
|
||||
});
|
||||
let temp_path = temp_path.unwrap_or_else(|| PathBuf::from("/tmp"));
|
||||
// let temp_path = temp_path.unwrap_or_else(|| PathBuf::from("/tmp"));
|
||||
let temp_path = target_dir;
|
||||
let filename = format!("bakefn_{}.sh", name);
|
||||
let target = target_dir.join(filename);
|
||||
|
||||
let bake_base_content = if use_template {
|
||||
std::fs::read_to_string(bake_base).map_err(|e| BakeError::ScriptGenerationFailed {
|
||||
script: bake_base.display().to_string(),
|
||||
reason: e.to_string(),
|
||||
})?
|
||||
} else {
|
||||
TRIVIAL_TEMPLATE.to_string()
|
||||
let template_content = match render_mode {
|
||||
RenderMode::Off => TRIVIAL_TEMPLATE.to_string(),
|
||||
RenderMode::Minimal | RenderMode::Full => {
|
||||
let script_debug = DebugFeature::from_bits_retain(debug_flags).contains(DebugFeature::Script);
|
||||
if script_debug {
|
||||
log::info!("Reading bake_base: {:?}", bake_base);
|
||||
} else {
|
||||
log::debug!("Reading bake_base: {:?}", bake_base);
|
||||
}
|
||||
std::fs::read_to_string(bake_base).map_err(|e| BakeError::IoError {
|
||||
path: bake_base.display().to_string(),
|
||||
source: e,
|
||||
})?
|
||||
}
|
||||
};
|
||||
|
||||
let condition_code = match condition {
|
||||
Some(content) => {
|
||||
// with some bash workaround
|
||||
format!(
|
||||
"if [ ! {} ]; then\n exit {}\nfi\n",
|
||||
"if [[ ! {} ]]; then\n exit {}\nfi\n",
|
||||
content, PIPELINE_SKIP_ERRORCODE
|
||||
)
|
||||
}
|
||||
None => String::new(),
|
||||
};
|
||||
|
||||
let mut preexport_code = String::new();
|
||||
let mut export_code = String::new();
|
||||
if !trivial {
|
||||
if export {
|
||||
preexport_code = format!("declare -xp > /dev/shm/bakeenv.before.{}\n", name);
|
||||
export_code = format!(
|
||||
"declare -xp > /dev/shm/bakeenv.after.{}\ndiff -w --new-line-format='%L' --old-line-format='' --unchanged-line-format='' /dev/shm/bakeenv.before.{} /dev/shm/bakeenv.after.{} > /dev/shm/bakeexport.{}\n",
|
||||
name, name, name, name
|
||||
);
|
||||
}
|
||||
preexport_code.push_str(format!(". /dev/shm/bakeexport.{}\n", name).as_str());
|
||||
export_code.push_str(format!("rm -f /dev/shm/bakeenv.*.{}\n", name).as_str());
|
||||
}
|
||||
// preexport/export: stubs for the future IPC-based @export.
|
||||
let preexport_code = String::new();
|
||||
let export_code = String::new();
|
||||
|
||||
let mut env = Environment::new();
|
||||
env.add_template("script", bake_base_content.as_str())?;
|
||||
let script = env.get_template("script")?;
|
||||
let rendered = if use_template {
|
||||
script.render(
|
||||
context! {main => body,condition => condition_code, preexport => preexport_code, name => name, export => export_code},
|
||||
)?
|
||||
} else {
|
||||
script.render(context! {main => body})?
|
||||
};
|
||||
env.add_template("script", &template_content)?;
|
||||
let rendered = env.get_template("script")?.render(
|
||||
context! {
|
||||
main => body,
|
||||
condition => condition_code,
|
||||
preexport => preexport_code,
|
||||
name => name,
|
||||
export => export_code,
|
||||
render_mode => render_mode.as_str(),
|
||||
},
|
||||
)?;
|
||||
|
||||
let mut temp_file =
|
||||
NamedTempFile::new_in(&temp_path).map_err(|e| BakeError::ScriptGenerationFailed {
|
||||
script: temp_path.display().to_string(),
|
||||
action: "creating temp file".to_string(),
|
||||
reason: e.to_string(),
|
||||
})?;
|
||||
|
||||
@@ -93,17 +117,20 @@ pub fn build_script(
|
||||
.set_permissions(Permissions::from_mode(0o700))
|
||||
.map_err(|e| BakeError::ScriptGenerationFailed {
|
||||
script: temp_path.display().to_string(),
|
||||
action: "setting permissions".to_string(),
|
||||
reason: e.to_string(),
|
||||
})?;
|
||||
|
||||
std::fs::write(&temp_file, rendered).map_err(|e| BakeError::ScriptGenerationFailed {
|
||||
script: temp_path.display().to_string(),
|
||||
action: "writing to temp file".to_string(),
|
||||
reason: e.to_string(),
|
||||
})?;
|
||||
temp_file
|
||||
.persist(&target)
|
||||
.map_err(|e| BakeError::ScriptGenerationFailed {
|
||||
script: target.display().to_string(),
|
||||
action: "persisting temp file".to_string(),
|
||||
reason: e.error.to_string(),
|
||||
})?;
|
||||
Ok(target)
|
||||
@@ -114,7 +141,7 @@ mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_build_script_with_template() {
|
||||
fn test_build_script_full_mode() {
|
||||
let temp_dir = tempfile::tempdir().unwrap();
|
||||
let bake_base = temp_dir.path().join("bake_base.sh");
|
||||
|
||||
@@ -125,7 +152,9 @@ mod tests {
|
||||
decorators: vec![],
|
||||
body: "echo 'hello world'".to_string(),
|
||||
};
|
||||
let result = build_script(&function, &bake_base, temp_dir.path(), None, true);
|
||||
let result = build_script(
|
||||
&function, &bake_base, temp_dir.path(), RenderMode::Full, 0,
|
||||
);
|
||||
|
||||
assert!(result.is_ok());
|
||||
let script_path = result.unwrap();
|
||||
@@ -134,10 +163,10 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_build_script_trivial_mode() {
|
||||
fn test_build_script_off_mode() {
|
||||
let temp_dir = tempfile::tempdir().unwrap();
|
||||
let function = Function {
|
||||
name: "trivial".to_string(),
|
||||
name: "test".to_string(),
|
||||
decorators: vec![],
|
||||
body: "echo 'direct output'".to_string(),
|
||||
};
|
||||
@@ -146,8 +175,8 @@ mod tests {
|
||||
&function,
|
||||
Path::new("/nonexistent"),
|
||||
temp_dir.path(),
|
||||
None,
|
||||
false,
|
||||
RenderMode::Off,
|
||||
0,
|
||||
);
|
||||
|
||||
assert!(result.is_ok());
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
pub const DEFAULT_WORKSPACE: &str = "/workspace";
|
||||
pub const BAKE_SCRIPT_PATH: &str = "bake.sh";
|
||||
pub const TRIVIAL_SCRIPT_NAME: &str = "BAKE_TRIVIAL";
|
||||
pub const PIPELINE_SKIP_ERRORCODE: u8 = 233;
|
||||
pub const PIPELINE_SKIP_ERRORCODE: i32 = 233;
|
||||
|
||||
@@ -4,11 +4,11 @@ use std::fmt::Display;
|
||||
use std::fmt::Formatter;
|
||||
|
||||
use crate::bake::error::BakeError;
|
||||
use crate::bake::parser::Function;
|
||||
|
||||
lazy_static! {
|
||||
// "# @decorator" or "# @decorator(parameters)"
|
||||
static ref DECORATOR_REGEX: Regex = Regex::new(r"^\s*#\s+@(\w+)\s?(\((.*)\))?").unwrap()
|
||||
;
|
||||
static ref DECORATOR_REGEX: Regex = Regex::new(r"^\s*#\s+@(\w+)\s?(\((.*)\))?").unwrap();
|
||||
static ref UNEXPECTED_ARGUMENT: &'static str = "decorator does not accept arguments";
|
||||
static ref MISSING_ARGUMENT: &'static str = "decorator requires an argument";
|
||||
static ref UNKNOWN_DECORATOR: &'static str = "unknown decorator";
|
||||
@@ -39,11 +39,121 @@ pub enum Decorator {
|
||||
/// Chain functions via pipe with comma-separated names.
|
||||
Pipe(Vec<String>),
|
||||
/// Run concurrently with other parallel functions.
|
||||
Parallel,
|
||||
/// `None` = wildcard ("*" group, compatible with any group).
|
||||
/// `Some(name)` = restricted to the named group.
|
||||
Parallel(Option<String>),
|
||||
/// Run as a background daemon process.
|
||||
Daemon,
|
||||
/// Health check endpoint for daemon verification.
|
||||
/// Health check command for daemon verification (shell statement).
|
||||
Health(String),
|
||||
/// Fire-and-forget execution; DAG pop happens at start, not end.
|
||||
Async,
|
||||
}
|
||||
|
||||
/// Returns true if the decorator is a flag (position-independent, consumed after execution).
|
||||
pub fn is_flag(d: &Decorator) -> bool {
|
||||
matches!(
|
||||
d,
|
||||
Decorator::Pipeline
|
||||
| Decorator::Fallible
|
||||
| Decorator::Export
|
||||
| Decorator::If(_)
|
||||
| Decorator::After(_)
|
||||
| Decorator::Parallel(_)
|
||||
| Decorator::Health(_)
|
||||
| Decorator::Async
|
||||
| Decorator::Unknown
|
||||
)
|
||||
}
|
||||
|
||||
/// Returns true if the decorator is a combinator (position-dependent, nestable).
|
||||
pub fn is_combinator(d: &Decorator) -> bool {
|
||||
matches!(d, Decorator::Loop(_) | Decorator::Timeout(_) | Decorator::Retry(..))
|
||||
}
|
||||
|
||||
/// Returns true if the decorator is a mode (at most one, alters what gets executed).
|
||||
pub fn is_mode(d: &Decorator) -> bool {
|
||||
matches!(d, Decorator::Pipe(_) | Decorator::Daemon | Decorator::Async)
|
||||
}
|
||||
|
||||
/// Extract the parallel group name. Returns "*" for wildcard (no group specified).
|
||||
pub fn parallel_group(d: &Decorator) -> &str {
|
||||
match d {
|
||||
Decorator::Parallel(Some(g)) => g.as_str(),
|
||||
_ => "*",
|
||||
}
|
||||
}
|
||||
|
||||
/// Check for incompatible decorator combinations on a function.
|
||||
/// Returns `Err(IncompatibleDecorators)` on conflict, `Ok(())` otherwise.
|
||||
pub fn check_conflicts(func: &Function) -> Result<(), BakeError> {
|
||||
let has_daemon = func.decorators.iter().any(|d| matches!(d, Decorator::Daemon));
|
||||
let has_async = func.decorators.iter().any(|d| matches!(d, Decorator::Async));
|
||||
let has_pipe = func.decorators.iter().any(|d| matches!(d, Decorator::Pipe(_)));
|
||||
let has_parallel = func.decorators.iter().any(|d| matches!(d, Decorator::Parallel(_)));
|
||||
let has_health = func.decorators.iter().any(|d| matches!(d, Decorator::Health(_)));
|
||||
let has_export = func.decorators.iter().any(|d| matches!(d, Decorator::Export));
|
||||
let has_combinator = func.decorators.iter().any(|d| is_combinator(d));
|
||||
|
||||
let mode_count = [has_daemon, has_async, has_pipe].iter().filter(|&&x| x).count();
|
||||
if mode_count > 1 {
|
||||
return Err(BakeError::IncompatibleDecorators {
|
||||
function: func.name.clone(),
|
||||
reason: "@daemon, @async, and @pipe are mutually exclusive".into(),
|
||||
});
|
||||
}
|
||||
|
||||
if has_daemon {
|
||||
if has_parallel {
|
||||
return Err(BakeError::IncompatibleDecorators {
|
||||
function: func.name.clone(),
|
||||
reason: "@daemon cannot be combined with @parallel".into(),
|
||||
});
|
||||
}
|
||||
if has_combinator {
|
||||
return Err(BakeError::IncompatibleDecorators {
|
||||
function: func.name.clone(),
|
||||
reason: "@daemon cannot be combined with @loop/@retry/@timeout".into(),
|
||||
});
|
||||
}
|
||||
if has_export {
|
||||
return Err(BakeError::IncompatibleDecorators {
|
||||
function: func.name.clone(),
|
||||
reason: "@daemon cannot be combined with @export".into(),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if has_health && !has_daemon {
|
||||
return Err(BakeError::IncompatibleDecorators {
|
||||
function: func.name.clone(),
|
||||
reason: "@health requires @daemon".into(),
|
||||
});
|
||||
}
|
||||
|
||||
if has_async {
|
||||
if has_parallel {
|
||||
return Err(BakeError::IncompatibleDecorators {
|
||||
function: func.name.clone(),
|
||||
reason: "@async cannot be combined with @parallel".into(),
|
||||
});
|
||||
}
|
||||
if has_export {
|
||||
return Err(BakeError::IncompatibleDecorators {
|
||||
function: func.name.clone(),
|
||||
reason: "@async cannot be combined with @export".into(),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if has_pipe && has_parallel {
|
||||
return Err(BakeError::IncompatibleDecorators {
|
||||
function: func.name.clone(),
|
||||
reason: "@pipe cannot be combined with @parallel".into(),
|
||||
});
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
impl Display for Decorator {
|
||||
@@ -59,9 +169,13 @@ impl Display for Decorator {
|
||||
Decorator::Retry(count, delay) => write!(f, "@retry({}, {})", count, delay),
|
||||
Decorator::Export => write!(f, "@export"),
|
||||
Decorator::Pipe(funcs) => write!(f, "@pipe({})", funcs.join(", ")),
|
||||
Decorator::Parallel => write!(f, "@parallel"),
|
||||
Decorator::Parallel(group) => match group {
|
||||
None => write!(f, "@parallel"),
|
||||
Some(g) => write!(f, "@parallel({})", g),
|
||||
},
|
||||
Decorator::Daemon => write!(f, "@daemon"),
|
||||
Decorator::Health(check) => write!(f, "@health({})", check),
|
||||
Decorator::Async => write!(f, "@async"),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -161,9 +275,17 @@ pub fn parse_decorator(line: &str, line_num: usize) -> Result<Option<Decorator>,
|
||||
}
|
||||
}),
|
||||
"export" => decorator_noarg(Decorator::Export),
|
||||
"parallel" => decorator_noarg(Decorator::Parallel),
|
||||
"parallel" => {
|
||||
if has_arg {
|
||||
let group = required_arg()?;
|
||||
Ok(Some(Decorator::Parallel(Some(group.to_string()))))
|
||||
} else {
|
||||
Ok(Some(Decorator::Parallel(None)))
|
||||
}
|
||||
}
|
||||
"daemon" => decorator_noarg(Decorator::Daemon),
|
||||
"health" => required_arg().map(|a| Some(Decorator::Health(a.to_string()))),
|
||||
"async" => decorator_noarg(Decorator::Async),
|
||||
"pipe" => required_arg().and_then(|a| {
|
||||
let functions: Vec<String> = a
|
||||
.split(',')
|
||||
@@ -290,7 +412,11 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn test_decorator_display_parallel() {
|
||||
assert_eq!(format!("{}", Decorator::Parallel), "@parallel");
|
||||
assert_eq!(format!("{}", Decorator::Parallel(None)), "@parallel");
|
||||
assert_eq!(
|
||||
format!("{}", Decorator::Parallel(Some("build".to_string()))),
|
||||
"@parallel(build)"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -310,6 +436,11 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_decorator_display_async() {
|
||||
assert_eq!(format!("{}", Decorator::Async), "@async");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_decorator_display_unknown() {
|
||||
assert_eq!(format!("{}", Decorator::Unknown), "@<unknown>");
|
||||
@@ -340,7 +471,13 @@ mod tests {
|
||||
#[test]
|
||||
fn test_parse_decorator_parallel() {
|
||||
let result = parse_decorator("# @parallel", 1).unwrap();
|
||||
assert_eq!(result, Some(Decorator::Parallel));
|
||||
assert_eq!(result, Some(Decorator::Parallel(None)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_parse_decorator_parallel_with_group() {
|
||||
let result = parse_decorator("# @parallel(build)", 1).unwrap();
|
||||
assert_eq!(result, Some(Decorator::Parallel(Some("build".to_string()))));
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -349,6 +486,12 @@ mod tests {
|
||||
assert_eq!(result, Some(Decorator::Daemon));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_parse_decorator_async() {
|
||||
let result = parse_decorator("# @async", 1).unwrap();
|
||||
assert_eq!(result, Some(Decorator::Async));
|
||||
}
|
||||
|
||||
// =====================================================================
|
||||
// Tests for parse_decorator - Valid single-argument decorators
|
||||
// =====================================================================
|
||||
@@ -576,8 +719,8 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_parse_decorator_parallel_with_arg() {
|
||||
let result = parse_decorator("# @parallel(true)", 1);
|
||||
fn test_parse_decorator_parallel_empty_args() {
|
||||
let result = parse_decorator("# @parallel()", 1);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
|
||||
|
||||
@@ -7,8 +7,8 @@ use workshop_engine::{
|
||||
|
||||
#[derive(Error, Debug)]
|
||||
pub enum BakeError {
|
||||
#[error("Failed to generate staged script {script}: {reason}")]
|
||||
ScriptGenerationFailed { script: String, reason: String },
|
||||
#[error("Failed to generate staged script {script} when {action}: {reason}")]
|
||||
ScriptGenerationFailed { script: String, action: String, reason: String },
|
||||
|
||||
#[error("Template error: {0}")]
|
||||
TemplateError(#[from] minijinja::Error),
|
||||
@@ -19,8 +19,8 @@ pub enum BakeError {
|
||||
#[error("Unknown @after of function {0}: {1}")]
|
||||
UnknownDependency(String, String),
|
||||
|
||||
#[error("IO Error: {0}")]
|
||||
IoError(#[from] std::io::Error),
|
||||
#[error("IO error on {path}: {source}")]
|
||||
IoError { path: String, source: std::io::Error },
|
||||
|
||||
#[error("YAML parse error: {0}")]
|
||||
YamlParseError(#[from] serde_yaml::Error),
|
||||
@@ -34,6 +34,12 @@ pub enum BakeError {
|
||||
|
||||
#[error("Script execution error: {0}")]
|
||||
ScriptExecutionError(#[from] ExecutionError),
|
||||
|
||||
#[error("Incompatible decorators on function '{function}': {reason}")]
|
||||
IncompatibleDecorators { function: String, reason: String },
|
||||
|
||||
#[error("Health probe failed for daemon '{command}': {error}")]
|
||||
HealthProbeFailed { command: String, error: String },
|
||||
}
|
||||
|
||||
impl HasExitCode for BakeError {
|
||||
@@ -43,10 +49,12 @@ impl HasExitCode for BakeError {
|
||||
BakeError::TemplateError(_) => EXITCODE_PARSE_ERROR,
|
||||
BakeError::CircularDependency(_) => EXITCODE_PARSE_ERROR,
|
||||
BakeError::UnknownDependency(..) => EXITCODE_PARSE_ERROR,
|
||||
BakeError::IoError(_) => EXITCODE_IO_ERROR,
|
||||
BakeError::IoError { .. } => EXITCODE_IO_ERROR,
|
||||
BakeError::YamlParseError(_) => EXITCODE_PARSE_ERROR,
|
||||
BakeError::DecoratorParseError { .. } => EXITCODE_PARSE_ERROR,
|
||||
BakeError::ScriptExecutionError(e) => e.exit_code(),
|
||||
BakeError::IncompatibleDecorators { .. } => EXITCODE_PARSE_ERROR,
|
||||
BakeError::HealthProbeFailed { .. } => EXITCODE_IO_ERROR,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,383 @@
|
||||
use crate::types::debug::DebugFeature;
|
||||
use crate::bake::builder::{self, RenderMode};
|
||||
use crate::bake::constant::PIPELINE_SKIP_ERRORCODE;
|
||||
use crate::bake::decorator::{self, Decorator};
|
||||
use crate::bake::error::BakeError;
|
||||
use crate::bake::parser::Function;
|
||||
use crate::bake::util::{is_fallible, resolve_pipe_stdin};
|
||||
use std::collections::HashMap;
|
||||
use std::future::Future;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::pin::Pin;
|
||||
use std::process::Stdio;
|
||||
use std::time::Duration;
|
||||
use workshop_engine::{Engine, EventSender, ExecutionContext, ExecutionError};
|
||||
|
||||
/// Execute a single function with combinator wrapping.
|
||||
/// Returns the captured stdout (for `@pipe` consumers).
|
||||
pub async fn execute_one(
|
||||
engine: &Engine,
|
||||
func: &Function,
|
||||
ctx: &mut ExecutionContext,
|
||||
event_tx: &EventSender,
|
||||
workspace: &Path,
|
||||
remaining: &str,
|
||||
bake_base: &Path,
|
||||
) -> Result<String, BakeError> {
|
||||
let combos: Vec<&Decorator> = func
|
||||
.decorators
|
||||
.iter()
|
||||
.filter(|d| decorator::is_combinator(d))
|
||||
.collect();
|
||||
|
||||
let mut modified = func.clone();
|
||||
modified.body = remaining.to_string() + &func.body;
|
||||
let script = builder::build_script(
|
||||
&modified, bake_base, workspace, RenderMode::Full, ctx.debug_flags,
|
||||
)?;
|
||||
|
||||
execute_with_combos(engine, &script, ctx, event_tx, &combos).await
|
||||
}
|
||||
|
||||
async fn execute_with_combos(
|
||||
engine: &Engine,
|
||||
script: &PathBuf,
|
||||
ctx: &mut ExecutionContext,
|
||||
event_tx: &EventSender,
|
||||
combos: &[&Decorator],
|
||||
) -> Result<String, BakeError> {
|
||||
build_combinator_chain(engine, script, ctx, event_tx, combos, 0).await
|
||||
}
|
||||
|
||||
/// Builds a combinator execution chain. Returns the captured stdout.
|
||||
/// For `@loop` + `@pipe(self)`, each iteration feeds its stdout back as
|
||||
/// the next iteration's stdin via `ctx.stdin`.
|
||||
fn build_combinator_chain<'a>(
|
||||
engine: &'a Engine,
|
||||
script: &'a PathBuf,
|
||||
ctx: &'a mut ExecutionContext,
|
||||
event_tx: &'a EventSender,
|
||||
combos: &'a [&'a Decorator],
|
||||
idx: usize,
|
||||
) -> Pin<Box<dyn Future<Output = Result<String, BakeError>> + Send + 'a>> {
|
||||
if idx >= combos.len() {
|
||||
return Box::pin(async move {
|
||||
let result = engine.execute_script(script, ctx, event_tx).await;
|
||||
match result {
|
||||
Ok(r) => Ok(r.stdout),
|
||||
Err(ExecutionError::ExecutionFailed { exit_code, .. })
|
||||
if exit_code == PIPELINE_SKIP_ERRORCODE as i32 =>
|
||||
{
|
||||
log::debug!("Function skipped (@if condition)");
|
||||
Ok(String::new())
|
||||
}
|
||||
Err(e) => Err(BakeError::ScriptExecutionError(e)),
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
match combos[idx] {
|
||||
Decorator::Loop(count) => Box::pin(async move {
|
||||
let mut last_stdout = String::new();
|
||||
for i in 0..*count {
|
||||
let script_debug = DebugFeature::from_bits_retain(ctx.debug_flags).contains(DebugFeature::Script);
|
||||
if script_debug {
|
||||
log::info!("Loop {}/{}", i + 1, count);
|
||||
} else {
|
||||
log::debug!("Loop {}/{}", i + 1, count);
|
||||
}
|
||||
last_stdout = build_combinator_chain(
|
||||
engine, script, ctx, event_tx, combos, idx + 1,
|
||||
)
|
||||
.await?;
|
||||
// Feed stdout as stdin for the next iteration (@pipe(self))
|
||||
if i + 1 < *count {
|
||||
ctx.stdin = Some(last_stdout.clone().into_bytes());
|
||||
}
|
||||
}
|
||||
Ok(last_stdout)
|
||||
}),
|
||||
Decorator::Timeout(secs) => Box::pin(async move {
|
||||
tokio::time::timeout(
|
||||
Duration::from_secs(*secs as u64),
|
||||
build_combinator_chain(
|
||||
engine, script, ctx, event_tx, combos, idx + 1,
|
||||
),
|
||||
)
|
||||
.await
|
||||
.map_err(|_| BakeError::ScriptExecutionError(ExecutionError::Timeout))?
|
||||
}),
|
||||
Decorator::Retry(retries, delay_secs) => Box::pin(async move {
|
||||
let max_attempts = *retries as usize + 1;
|
||||
let delay = Duration::from_secs(*delay_secs as u64);
|
||||
let mut last_err = None;
|
||||
for attempt in 0..max_attempts {
|
||||
match build_combinator_chain(
|
||||
engine, script, ctx, event_tx, combos, idx + 1,
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(stdout) => return Ok(stdout),
|
||||
Err(e) => {
|
||||
if matches!(
|
||||
&e,
|
||||
BakeError::ScriptExecutionError(ExecutionError::Timeout)
|
||||
) {
|
||||
return Err(e);
|
||||
}
|
||||
if attempt < max_attempts - 1 {
|
||||
last_err = Some(e);
|
||||
log::warn!(
|
||||
"Retry {}/{} after {}s",
|
||||
attempt + 1,
|
||||
retries,
|
||||
delay_secs
|
||||
);
|
||||
tokio::time::sleep(delay).await;
|
||||
} else {
|
||||
return Err(e);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(last_err.unwrap())
|
||||
}),
|
||||
_ => build_combinator_chain(
|
||||
engine, script, ctx, event_tx, combos, idx + 1,
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
/// Fire a batch of parallel nodes concurrently.
|
||||
///
|
||||
/// All nodes in the batch are waited on before returning. A non-`@fallible`
|
||||
/// failure is surfaced only after all nodes complete (GitHub/GitLab/Concourse
|
||||
/// convention).
|
||||
///
|
||||
/// Returns a `(name, stdout)` pair for each completed node so the caller can
|
||||
/// populate the stdout cache for `@pipe`.
|
||||
pub async fn fire_parallel_batch(
|
||||
batch: &[String],
|
||||
_engine: &Engine,
|
||||
func_map: &HashMap<String, Function>,
|
||||
ctx: &mut ExecutionContext,
|
||||
event_tx: &EventSender,
|
||||
workspace: &Path,
|
||||
remaining: &str,
|
||||
bake_base: &Path,
|
||||
stdout_cache: &HashMap<String, String>,
|
||||
) -> Result<Vec<(String, String)>, BakeError> {
|
||||
let handles: Vec<_> = batch
|
||||
.iter()
|
||||
.map(|name| {
|
||||
let function = func_map[name.as_str()].clone();
|
||||
let mut task_ctx = ctx.clone();
|
||||
// Resolve per-task stdin from @pipe + stdout_cache
|
||||
task_ctx.stdin = resolve_pipe_stdin(&function, stdout_cache);
|
||||
let task_events = event_tx.clone();
|
||||
let task_workspace = workspace.to_path_buf();
|
||||
let task_remaining = remaining.to_string();
|
||||
let task_bake_base = bake_base.to_path_buf();
|
||||
let task_name = name.clone();
|
||||
tokio::spawn(async move {
|
||||
let task_engine = Engine::new();
|
||||
let result = execute_one(
|
||||
&task_engine,
|
||||
&function,
|
||||
&mut task_ctx,
|
||||
&task_events,
|
||||
&task_workspace,
|
||||
&task_remaining,
|
||||
&task_bake_base,
|
||||
)
|
||||
.await;
|
||||
(task_name, result)
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
|
||||
let mut first_fatal: Option<BakeError> = None;
|
||||
let mut results: Vec<(String, String)> = Vec::with_capacity(handles.len());
|
||||
for handle in handles {
|
||||
let (name, result) = handle.await.map_err(|_| {
|
||||
BakeError::ScriptExecutionError(ExecutionError::ExecutionFailed {
|
||||
task_id: "parallel".to_string(),
|
||||
exit_code: -1,
|
||||
})
|
||||
})?;
|
||||
match result {
|
||||
Ok(stdout) => results.push((name, stdout)),
|
||||
Err(err) if is_fallible(&func_map[&name]) => {
|
||||
log::warn!("'{}' failed (fallible, non-fatal): {}", name, err);
|
||||
}
|
||||
Err(err) => {
|
||||
log::error!("'{}' failed (fatal): {}", name, err);
|
||||
first_fatal.get_or_insert(err);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if let Some(err) = first_fatal {
|
||||
return Err(err);
|
||||
}
|
||||
Ok(results)
|
||||
}
|
||||
|
||||
/// Spawn an `@async` function in the background.
|
||||
/// Returns a `JoinHandle` that the caller must await before returning.
|
||||
pub fn execute_async(
|
||||
func: &Function,
|
||||
ctx: &ExecutionContext,
|
||||
event_tx: &EventSender,
|
||||
workspace: &Path,
|
||||
remaining: &str,
|
||||
bake_base: &Path,
|
||||
) -> tokio::task::JoinHandle<()> {
|
||||
let mut task_ctx = ctx.clone();
|
||||
let task_events = event_tx.clone();
|
||||
let task_workspace = workspace.to_path_buf();
|
||||
let task_remaining = remaining.to_string();
|
||||
let task_bake_base = bake_base.to_path_buf();
|
||||
let task_func = func.clone();
|
||||
tokio::spawn(async move {
|
||||
let task_engine = Engine::new();
|
||||
let _ = execute_one(
|
||||
&task_engine, &task_func, &mut task_ctx,
|
||||
&task_events, &task_workspace,
|
||||
&task_remaining, &task_bake_base,
|
||||
).await;
|
||||
})
|
||||
}
|
||||
|
||||
/// Execute a Normal-mode function, resolving pipe stdin first.
|
||||
pub async fn execute_normal(
|
||||
engine: &Engine,
|
||||
func: &Function,
|
||||
ctx: &mut ExecutionContext,
|
||||
event_tx: &EventSender,
|
||||
workspace: &Path,
|
||||
remaining: &str,
|
||||
bake_base: &Path,
|
||||
stdout_cache: &HashMap<String, String>,
|
||||
) -> Result<String, BakeError> {
|
||||
ctx.stdin = resolve_pipe_stdin(func, stdout_cache);
|
||||
execute_one(engine, func, ctx, event_tx, workspace, remaining, bake_base).await
|
||||
}
|
||||
|
||||
// ── Daemon / Health ─────────────────────────────────────────────────────────
|
||||
|
||||
/// Execute a `@daemon` function: build script, spawn, periodic health-probe
|
||||
/// (blocking until healthy), then return.
|
||||
pub async fn execute_daemon(
|
||||
engine: &Engine,
|
||||
func: &Function,
|
||||
ctx: &ExecutionContext,
|
||||
event_tx: &EventSender,
|
||||
workspace: &Path,
|
||||
remaining: &str,
|
||||
bake_base: &Path,
|
||||
health_period: Duration,
|
||||
health_timeout: Duration,
|
||||
health_max_retries: u32,
|
||||
) -> Result<(), BakeError> {
|
||||
let fallible = is_fallible(func);
|
||||
|
||||
let mut modified = func.clone();
|
||||
modified.body = remaining.to_string() + &func.body;
|
||||
let script = builder::build_script(
|
||||
&modified, bake_base, workspace, RenderMode::Full, ctx.debug_flags,
|
||||
)?;
|
||||
|
||||
// Spawn without waiting
|
||||
let mut command = tokio::process::Command::new(ctx.shell.as_str());
|
||||
command.arg("-c");
|
||||
command.arg(script.to_str().unwrap());
|
||||
command.current_dir(&ctx.working_dir);
|
||||
command.envs(&ctx.env_vars);
|
||||
command.stdin(Stdio::null());
|
||||
command.stdout(Stdio::null());
|
||||
command.stderr(Stdio::null());
|
||||
#[cfg(unix)]
|
||||
command.process_group(0);
|
||||
|
||||
//let child = // Necessary?
|
||||
match command.spawn() {
|
||||
Ok(c) => c,
|
||||
Err(e) if fallible => {
|
||||
log::warn!("Failed to spawn daemon '{}' (fallible): {}", func.name, e);
|
||||
return Ok(());
|
||||
}
|
||||
Err(_) => {
|
||||
return Err(BakeError::ScriptExecutionError(ExecutionError::ExecutionFailed {
|
||||
task_id: func.name.clone(),
|
||||
exit_code: -1,
|
||||
}));
|
||||
}
|
||||
};
|
||||
|
||||
// Periodic health probe — block until healthy, retry exhausted, or fatal
|
||||
if let Some(health_cmd) = func.decorators.iter().find_map(|d| match d {
|
||||
Decorator::Health(cmd) => Some(cmd.as_str()),
|
||||
_ => None,
|
||||
}) {
|
||||
let mut retries: u32 = 0;
|
||||
loop {
|
||||
let probe_result = tokio::time::timeout(
|
||||
health_timeout,
|
||||
health_probe_one(engine, health_cmd, remaining, ctx, event_tx),
|
||||
)
|
||||
.await;
|
||||
|
||||
match probe_result {
|
||||
Ok(Ok(())) => break, // healthy — done
|
||||
_ if retries + 1 < health_max_retries => {
|
||||
retries += 1;
|
||||
log::warn!(
|
||||
"Health probe for '{}' failed ({}/{}), retrying in {:?}...",
|
||||
func.name, retries, health_max_retries, health_period,
|
||||
);
|
||||
}
|
||||
_ if fallible => {
|
||||
log::warn!("Health probe for '{}' exhausted (fallible), skipping", func.name);
|
||||
break; // exhausted but fallible → skip health check
|
||||
}
|
||||
Ok(Err(e)) => return Err(e),
|
||||
Err(_) => {
|
||||
return Err(BakeError::HealthProbeFailed {
|
||||
command: health_cmd.to_string(),
|
||||
error: "timeout".to_string(),
|
||||
});
|
||||
}
|
||||
}
|
||||
tokio::time::sleep(health_period).await;
|
||||
}
|
||||
}
|
||||
|
||||
// Cleanup is handled by finalize.
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Single health-probe attempt. Exit 0 = healthy.
|
||||
async fn health_probe_one(
|
||||
engine: &Engine,
|
||||
cmd: &str,
|
||||
remaining: &str,
|
||||
ctx: &ExecutionContext,
|
||||
event_tx: &EventSender,
|
||||
) -> Result<(), BakeError> {
|
||||
let full_cmd = format!("{}\n{}", remaining, cmd);
|
||||
let result = engine
|
||||
.execute_command(&full_cmd, ctx, event_tx)
|
||||
.await
|
||||
.map_err(|e| BakeError::HealthProbeFailed {
|
||||
command: cmd.to_string(),
|
||||
error: format!("{}", e),
|
||||
})?;
|
||||
if !result.success {
|
||||
return Err(BakeError::HealthProbeFailed {
|
||||
command: cmd.to_string(),
|
||||
error: format!("exit code {}", result.exit_code),
|
||||
});
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
@@ -174,7 +174,7 @@ pub fn parse_script(text: &str) -> Result<ParsedScript, BakeError> {
|
||||
}
|
||||
continue;
|
||||
}
|
||||
eprintln!(
|
||||
log::warn!(
|
||||
"Warning: Dangling decorator(s) at line {}. Treating as comments.",
|
||||
line_number - 1
|
||||
);
|
||||
@@ -224,6 +224,7 @@ pub fn parse_script(text: &str) -> Result<ParsedScript, BakeError> {
|
||||
{
|
||||
return Err(BakeError::ScriptGenerationFailed {
|
||||
script: "bake.sh".to_string(),
|
||||
action: "parsing function body".to_string(),
|
||||
reason: format!(
|
||||
"Unexpected EOF while parsing function '{}'. Unmatched braces (remaining: {})?",
|
||||
name, brace_depth
|
||||
@@ -234,6 +235,7 @@ pub fn parse_script(text: &str) -> Result<ParsedScript, BakeError> {
|
||||
let last_decorator = pending_decorators.last().unwrap();
|
||||
return Err(BakeError::ScriptGenerationFailed {
|
||||
script: "bake.sh".to_string(),
|
||||
action: "parsing decorators".to_string(),
|
||||
reason: format!(
|
||||
"Dangling decorator {} before EOF. Missing function definition?",
|
||||
last_decorator
|
||||
|
||||
+143
-471
@@ -1,42 +1,124 @@
|
||||
use super::{decorator::Decorator, parser::Function};
|
||||
// Code in this module PARTIALLY or FULLY utilized AI Coding Agent
|
||||
|
||||
use super::decorator::Decorator;
|
||||
use super::parser::Function;
|
||||
use crate::bake::error::BakeError;
|
||||
use std::collections::HashMap;
|
||||
use topological_sort::TopologicalSort;
|
||||
use workshop_schedule::{Dag, DagNode, EdgeKind, NodeColor};
|
||||
|
||||
/// Topologically sorts functions by explicit `@after` dependencies and implicit ordering
|
||||
/// for consecutive `@pipeline` functions (each pipeline function implicitly depends on
|
||||
/// the previous one unless it has an explicit `@after`).
|
||||
pub fn sort_function(functions: Vec<Function>) -> Result<Vec<Function>, BakeError> {
|
||||
let function_mapped = functions
|
||||
/// Thin wrapper so `Function` implements `DagNode`.
|
||||
#[derive(Clone)]
|
||||
pub(crate) struct BakeNode {
|
||||
name: String,
|
||||
color: NodeColor,
|
||||
fallible: bool,
|
||||
/// Original function body + decorators for execution
|
||||
func: Function,
|
||||
}
|
||||
|
||||
impl DagNode for BakeNode {
|
||||
fn name(&self) -> &str {
|
||||
&self.name
|
||||
}
|
||||
fn color(&self) -> NodeColor {
|
||||
self.color.clone()
|
||||
}
|
||||
fn fallible(&self) -> bool {
|
||||
self.fallible
|
||||
}
|
||||
}
|
||||
|
||||
/// Convert a `Function` into a `BakeNode` by extracting scheduling metadata
|
||||
/// from decorators.
|
||||
fn function_to_node(func: &Function) -> BakeNode {
|
||||
let color = if func
|
||||
.decorators
|
||||
.iter()
|
||||
.any(|d| matches!(d, Decorator::Parallel(_)))
|
||||
{
|
||||
let group = func
|
||||
.decorators
|
||||
.iter()
|
||||
.find_map(|d| {
|
||||
if let Decorator::Parallel(Some(g)) = d {
|
||||
Some(g.clone())
|
||||
} else {
|
||||
None
|
||||
}
|
||||
})
|
||||
.unwrap_or("*".to_string());
|
||||
if group == "*" {
|
||||
NodeColor::Wildcard
|
||||
} else {
|
||||
NodeColor::Named(group)
|
||||
}
|
||||
} else {
|
||||
NodeColor::Sequential
|
||||
};
|
||||
|
||||
let fallible = func
|
||||
.decorators
|
||||
.iter()
|
||||
.any(|d| matches!(d, Decorator::Fallible));
|
||||
|
||||
BakeNode {
|
||||
name: func.name.clone(),
|
||||
color,
|
||||
fallible,
|
||||
func: func.clone(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Builds a DAG from functions using `@after` dependencies and implicit
|
||||
/// ordering for consecutive `@pipeline` functions.
|
||||
pub fn build_dag(
|
||||
functions: Vec<Function>,
|
||||
) -> Result<(Dag<BakeNode>, HashMap<String, Function>), BakeError> {
|
||||
let function_mapped: HashMap<&str, &Function> = functions
|
||||
.iter()
|
||||
.map(|f| (f.name.as_str(), f))
|
||||
.collect::<HashMap<&str, &Function>>();
|
||||
.collect();
|
||||
|
||||
let mut ts = TopologicalSort::<String>::new();
|
||||
|
||||
// Track last pipeline function for implicit @after
|
||||
let mut dag = Dag::new();
|
||||
let mut last_pipeline: Option<String> = None;
|
||||
|
||||
for function in &functions {
|
||||
ts.insert(function.name.clone());
|
||||
|
||||
// Determine whether this function is a pipeline node
|
||||
let is_pipeline = function
|
||||
.decorators
|
||||
.iter()
|
||||
.any(|d| matches!(d, Decorator::Pipeline));
|
||||
|
||||
if !is_pipeline {
|
||||
continue;
|
||||
}
|
||||
|
||||
let has_after = function
|
||||
.decorators
|
||||
.iter()
|
||||
.any(|d| matches!(d, Decorator::After(_)));
|
||||
|
||||
// Add implicit @after for pipeline functions that don't have explicit @after
|
||||
if is_pipeline {
|
||||
if !has_after && let Some(ref last) = last_pipeline {
|
||||
ts.add_dependency(last, function.name.clone());
|
||||
let node = function_to_node(function);
|
||||
dag.add_node(node).map_err(|e| {
|
||||
BakeError::IncompatibleDecorators {
|
||||
function: function.name.clone(),
|
||||
reason: e.to_string(),
|
||||
}
|
||||
last_pipeline = Some(function.name.clone());
|
||||
}
|
||||
})?;
|
||||
|
||||
// Add implicit @after for consecutive @pipeline functions
|
||||
if !has_after {
|
||||
if let Some(ref last) = last_pipeline {
|
||||
dag.add_edge(last, &function.name, EdgeKind::Implicit)
|
||||
.map_err(|e| BakeError::IncompatibleDecorators {
|
||||
function: function.name.clone(),
|
||||
reason: e.to_string(),
|
||||
})?;
|
||||
}
|
||||
}
|
||||
last_pipeline = Some(function.name.clone());
|
||||
|
||||
// Add explicit @after edges
|
||||
for decorator in &function.decorators {
|
||||
if let Decorator::After(dependency) = decorator {
|
||||
if !function_mapped.contains_key(dependency.as_str()) {
|
||||
@@ -50,466 +132,56 @@ pub fn sort_function(functions: Vec<Function>) -> Result<Vec<Function>, BakeErro
|
||||
dependency.clone(),
|
||||
));
|
||||
}
|
||||
ts.add_dependency(dependency, function.name.clone());
|
||||
dag.add_edge(dependency, &function.name, EdgeKind::Explicit)
|
||||
.map_err(|e| BakeError::IncompatibleDecorators {
|
||||
function: function.name.clone(),
|
||||
reason: e.to_string(),
|
||||
})?;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let mut functions_sorted = Vec::<Function>::new();
|
||||
while let Some(name) = ts.pop() {
|
||||
let function = function_mapped[name.as_str()];
|
||||
functions_sorted.push(function.clone());
|
||||
}
|
||||
dag.freeze();
|
||||
|
||||
if !ts.is_empty() {
|
||||
let remaining: Vec<String> = ts.collect();
|
||||
return Err(BakeError::CircularDependency(remaining));
|
||||
}
|
||||
let owned_map: HashMap<String, Function> = functions
|
||||
.into_iter()
|
||||
.map(|f| (f.name.clone(), f))
|
||||
.collect();
|
||||
|
||||
Ok(functions_sorted)
|
||||
Ok((dag, owned_map))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn create_function(name: &str, decorators: Vec<Decorator>, body: &str) -> Function {
|
||||
Function {
|
||||
name: name.to_string(),
|
||||
decorators,
|
||||
body: body.to_string(),
|
||||
/// Group ready parallel nodes by their declared group.
|
||||
/// Returns `(star_nodes, named_groups)`.
|
||||
///
|
||||
/// Deprecated: use `Dag::ready_wildcards()` and `Dag::ready_groups()` instead.
|
||||
pub fn partition_by_group<'a>(
|
||||
ready: &[String],
|
||||
dag: &Dag<BakeNode>,
|
||||
) -> (Vec<String>, HashMap<String, Vec<String>>) {
|
||||
let mut star_nodes = Vec::new();
|
||||
let mut named_groups: HashMap<String, Vec<String>> = HashMap::new();
|
||||
for name in ready {
|
||||
if dag.is_wildcard(name) {
|
||||
star_nodes.push(name.clone());
|
||||
} else if let Some(g) = dag.group_of(name) {
|
||||
named_groups.entry(g).or_default().push(name.clone());
|
||||
}
|
||||
// Sequential nodes: not included (caller handles them)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_empty_functions() {
|
||||
let functions = vec![];
|
||||
let result = sort_function(functions).unwrap();
|
||||
assert!(result.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_single_function_no_dependencies() {
|
||||
let function = create_function("func1", vec![], "body1");
|
||||
let functions = vec![function.clone()];
|
||||
|
||||
let result = sort_function(functions).unwrap();
|
||||
|
||||
assert_eq!(result.len(), 1);
|
||||
assert_eq!(result[0].name, "func1");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_multiple_functions_no_dependencies() {
|
||||
let function1 = create_function("func1", vec![], "body1");
|
||||
let function2 = create_function("func2", vec![], "body2");
|
||||
let function3 = create_function("func3", vec![], "body3");
|
||||
let functions = vec![function1.clone(), function2.clone(), function3.clone()];
|
||||
|
||||
let result = sort_function(functions).unwrap();
|
||||
|
||||
assert_eq!(result.len(), 3);
|
||||
let names: Vec<String> = result.iter().map(|f| f.name.clone()).collect();
|
||||
assert!(names.contains(&"func1".to_string()));
|
||||
assert!(names.contains(&"func2".to_string()));
|
||||
assert!(names.contains(&"func3".to_string()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_simple_dependency() {
|
||||
let function1 = create_function("func1", vec![], "body1");
|
||||
let function2 = create_function(
|
||||
"func2",
|
||||
vec![Decorator::After("func1".to_string())],
|
||||
"body2",
|
||||
);
|
||||
let functions = vec![function1.clone(), function2.clone()];
|
||||
|
||||
let result = sort_function(functions).unwrap();
|
||||
|
||||
assert_eq!(result.len(), 2);
|
||||
assert_eq!(result[0].name, "func1");
|
||||
assert_eq!(result[1].name, "func2");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_multiple_dependencies() {
|
||||
let function1 = create_function("func1", vec![], "body1");
|
||||
let function2 = create_function(
|
||||
"func2",
|
||||
vec![Decorator::After("func1".to_string())],
|
||||
"body2",
|
||||
);
|
||||
let function3 = create_function(
|
||||
"func3",
|
||||
vec![Decorator::After("func2".to_string())],
|
||||
"body3",
|
||||
);
|
||||
let functions = vec![function1.clone(), function2.clone(), function3.clone()];
|
||||
|
||||
let result = sort_function(functions).unwrap();
|
||||
|
||||
assert_eq!(result.len(), 3);
|
||||
assert_eq!(result[0].name, "func1");
|
||||
assert_eq!(result[1].name, "func2");
|
||||
assert_eq!(result[2].name, "func3");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_complex_dependencies() {
|
||||
let function1 = create_function("func1", vec![], "body1");
|
||||
let function2 = create_function(
|
||||
"func2",
|
||||
vec![
|
||||
Decorator::After("func1".to_string()),
|
||||
Decorator::After("func3".to_string()),
|
||||
],
|
||||
"body2",
|
||||
);
|
||||
let function3 = create_function(
|
||||
"func3",
|
||||
vec![Decorator::After("func1".to_string())],
|
||||
"body3",
|
||||
);
|
||||
let function4 = create_function("func4", vec![], "body4");
|
||||
let functions = vec![
|
||||
function1.clone(),
|
||||
function2.clone(),
|
||||
function3.clone(),
|
||||
function4.clone(),
|
||||
];
|
||||
|
||||
let result = sort_function(functions).unwrap();
|
||||
|
||||
assert_eq!(result.len(), 4);
|
||||
|
||||
let positions: std::collections::HashMap<String, usize> = result
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(i, f)| (f.name.clone(), i))
|
||||
.collect();
|
||||
|
||||
assert!(positions.get("func1").unwrap() < positions.get("func2").unwrap());
|
||||
assert!(positions.get("func1").unwrap() < positions.get("func3").unwrap());
|
||||
assert!(positions.get("func3").unwrap() < positions.get("func2").unwrap());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_mixed_decorators() {
|
||||
let function1 = create_function(
|
||||
"func1",
|
||||
vec![Decorator::Pipeline, Decorator::Timeout(1000)],
|
||||
"body1",
|
||||
);
|
||||
let function2 = create_function(
|
||||
"func2",
|
||||
vec![
|
||||
Decorator::After("func1".to_string()),
|
||||
Decorator::Retry(3, 100),
|
||||
Decorator::Fallible,
|
||||
],
|
||||
"body2",
|
||||
);
|
||||
let functions = vec![function1.clone(), function2.clone()];
|
||||
|
||||
let result = sort_function(functions).unwrap();
|
||||
|
||||
assert_eq!(result.len(), 2);
|
||||
assert_eq!(result[0].name, "func1");
|
||||
assert_eq!(result[1].name, "func2");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_function_with_multiple_after_decorators() {
|
||||
let function1 = create_function("func1", vec![], "body1");
|
||||
let function2 = create_function("func2", vec![], "body2");
|
||||
let function3 = create_function(
|
||||
"func3",
|
||||
vec![
|
||||
Decorator::After("func1".to_string()),
|
||||
Decorator::After("func2".to_string()),
|
||||
],
|
||||
"body3",
|
||||
);
|
||||
let functions = vec![function1.clone(), function2.clone(), function3.clone()];
|
||||
|
||||
let result = sort_function(functions).unwrap();
|
||||
|
||||
assert_eq!(result.len(), 3);
|
||||
|
||||
let positions: std::collections::HashMap<String, usize> = result
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(i, f)| (f.name.clone(), i))
|
||||
.collect();
|
||||
|
||||
assert!(positions.get("func1").unwrap() < positions.get("func3").unwrap());
|
||||
assert!(positions.get("func2").unwrap() < positions.get("func3").unwrap());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_ignores_non_after_decorators() {
|
||||
let function1 = create_function(
|
||||
"func1",
|
||||
vec![
|
||||
Decorator::Pipeline,
|
||||
Decorator::If("condition".to_string()),
|
||||
Decorator::Fallible,
|
||||
],
|
||||
"body1",
|
||||
);
|
||||
let function2 = create_function(
|
||||
"func2",
|
||||
vec![
|
||||
Decorator::Loop(5),
|
||||
Decorator::Timeout(1000),
|
||||
Decorator::Retry(3, 100),
|
||||
Decorator::Export,
|
||||
],
|
||||
"body2",
|
||||
);
|
||||
let functions = vec![function1.clone(), function2.clone()];
|
||||
|
||||
let result = sort_function(functions).unwrap();
|
||||
|
||||
assert_eq!(result.len(), 2);
|
||||
let names: Vec<String> = result.iter().map(|f| f.name.clone()).collect();
|
||||
assert!(names.contains(&"func1".to_string()));
|
||||
assert!(names.contains(&"func2".to_string()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_implicit_after_pipeline() {
|
||||
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
|
||||
let func2 = create_function("func2", vec![Decorator::Pipeline], "body2");
|
||||
let func3 = create_function("func3", vec![Decorator::Pipeline], "body3");
|
||||
let functions = vec![func1.clone(), func2.clone(), func3.clone()];
|
||||
|
||||
let result = sort_function(functions).unwrap();
|
||||
|
||||
assert_eq!(result.len(), 3);
|
||||
assert_eq!(result[0].name, "func1");
|
||||
assert_eq!(result[1].name, "func2");
|
||||
assert_eq!(result[2].name, "func3");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_first_pipeline_no_implicit_after() {
|
||||
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
|
||||
let functions = vec![func1.clone()];
|
||||
|
||||
let result = sort_function(functions).unwrap();
|
||||
|
||||
assert_eq!(result.len(), 1);
|
||||
assert_eq!(result[0].name, "func1");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_explicit_after_overrides_implicit() {
|
||||
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
|
||||
let func2 = create_function(
|
||||
"func2",
|
||||
vec![Decorator::Pipeline, Decorator::After("func1".to_string())],
|
||||
"body2",
|
||||
);
|
||||
let func3 = create_function("func3", vec![Decorator::Pipeline], "body3");
|
||||
let functions = vec![func1.clone(), func2.clone(), func3.clone()];
|
||||
|
||||
let result = sort_function(functions).unwrap();
|
||||
|
||||
assert_eq!(result.len(), 3);
|
||||
let positions: std::collections::HashMap<String, usize> = result
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(i, f)| (f.name.clone(), i))
|
||||
.collect();
|
||||
|
||||
assert!(positions.get("func1").unwrap() < positions.get("func2").unwrap());
|
||||
assert!(positions.get("func2").unwrap() < positions.get("func3").unwrap());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_mixed_pipeline_and_non_pipeline() {
|
||||
let func1 = create_function("func1", vec![], "body1");
|
||||
let func2 = create_function("func2", vec![Decorator::Pipeline], "body2");
|
||||
let func3 = create_function("func3", vec![], "body3");
|
||||
let func4 = create_function("func4", vec![Decorator::Pipeline], "body4");
|
||||
let functions = vec![func1.clone(), func2.clone(), func3.clone(), func4.clone()];
|
||||
|
||||
let result = sort_function(functions).unwrap();
|
||||
|
||||
assert_eq!(result.len(), 4);
|
||||
let positions: std::collections::HashMap<String, usize> = result
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(i, f)| (f.name.clone(), i))
|
||||
.collect();
|
||||
|
||||
assert!(positions.get("func2").unwrap() < positions.get("func4").unwrap());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_single_pipeline_works() {
|
||||
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
|
||||
let functions = vec![func1.clone()];
|
||||
|
||||
let result = sort_function(functions).unwrap();
|
||||
assert_eq!(result.len(), 1);
|
||||
assert_eq!(result[0].name, "func1");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_two_pipelines_implicit_chain() {
|
||||
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
|
||||
let func2 = create_function("func2", vec![Decorator::Pipeline], "body2");
|
||||
let functions = vec![func1.clone(), func2.clone()];
|
||||
|
||||
let result = sort_function(functions).unwrap();
|
||||
assert_eq!(result.len(), 2);
|
||||
let positions: std::collections::HashMap<String, usize> = result
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(i, f)| (f.name.clone(), i))
|
||||
.collect();
|
||||
assert!(positions.get("func1").unwrap() < positions.get("func2").unwrap());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_pipeline_explicit_after_skips_implicit() {
|
||||
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
|
||||
let func2 = create_function("func2", vec![Decorator::Pipeline], "body2");
|
||||
let func3 = create_function(
|
||||
"func3",
|
||||
vec![Decorator::Pipeline, Decorator::After("func1".to_string())],
|
||||
"body3",
|
||||
);
|
||||
let functions = vec![func1.clone(), func2.clone(), func3.clone()];
|
||||
|
||||
let result = sort_function(functions).unwrap();
|
||||
assert_eq!(result.len(), 3);
|
||||
let positions: std::collections::HashMap<String, usize> = result
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(i, f)| (f.name.clone(), i))
|
||||
.collect();
|
||||
assert!(positions.get("func1").unwrap() < positions.get("func3").unwrap());
|
||||
// TODO: Our design spec indicates order-based sorting, however toposort crate does not guarantee order of independent nodes. We should consider whether we want to enforce order-based sorting for pipelines, or if we are okay with any valid topological order.
|
||||
// assert!(positions.get("func2").unwrap() < positions.get("func3").unwrap());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_circular_dependency_detected() {
|
||||
let func1 = create_function(
|
||||
"func1",
|
||||
vec![Decorator::After("func2".to_string())],
|
||||
"body1",
|
||||
);
|
||||
let func2 = create_function(
|
||||
"func2",
|
||||
vec![Decorator::After("func1".to_string())],
|
||||
"body2",
|
||||
);
|
||||
let functions = vec![func1.clone(), func2.clone()];
|
||||
|
||||
let result = sort_function(functions);
|
||||
assert!(result.is_err());
|
||||
assert!(matches!(
|
||||
result.unwrap_err(),
|
||||
BakeError::CircularDependency(_)
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_unknown_dependency_error() {
|
||||
let func1 = create_function(
|
||||
"func1",
|
||||
vec![Decorator::After("nonexistent".to_string())],
|
||||
"body1",
|
||||
);
|
||||
let functions = vec![func1.clone()];
|
||||
|
||||
let result = sort_function(functions);
|
||||
assert!(result.is_err());
|
||||
assert!(matches!(
|
||||
result.unwrap_err(),
|
||||
BakeError::UnknownDependency(_, _)
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_pipeline_with_multiple_decorators() {
|
||||
let func1 = create_function(
|
||||
"func1",
|
||||
vec![
|
||||
Decorator::Pipeline,
|
||||
Decorator::Timeout(60),
|
||||
Decorator::Fallible,
|
||||
],
|
||||
"body1",
|
||||
);
|
||||
let func2 = create_function(
|
||||
"func2",
|
||||
vec![
|
||||
Decorator::Pipeline,
|
||||
Decorator::Retry(3, 5),
|
||||
Decorator::After("func1".to_string()),
|
||||
],
|
||||
"body2",
|
||||
);
|
||||
let functions = vec![func1.clone(), func2.clone()];
|
||||
|
||||
let result = sort_function(functions).unwrap();
|
||||
assert_eq!(result.len(), 2);
|
||||
let positions: std::collections::HashMap<String, usize> = result
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(i, f)| (f.name.clone(), i))
|
||||
.collect();
|
||||
assert!(positions.get("func1").unwrap() < positions.get("func2").unwrap());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_long_pipeline_chain() {
|
||||
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
|
||||
let func2 = create_function("func2", vec![Decorator::Pipeline], "body2");
|
||||
let func3 = create_function("func3", vec![Decorator::Pipeline], "body3");
|
||||
let func4 = create_function("func4", vec![Decorator::Pipeline], "body4");
|
||||
let func5 = create_function("func5", vec![Decorator::Pipeline], "body5");
|
||||
let functions = vec![
|
||||
func1.clone(),
|
||||
func2.clone(),
|
||||
func3.clone(),
|
||||
func4.clone(),
|
||||
func5.clone(),
|
||||
];
|
||||
|
||||
let result = sort_function(functions).unwrap();
|
||||
assert_eq!(result.len(), 5);
|
||||
let positions: std::collections::HashMap<String, usize> = result
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(i, f)| (f.name.clone(), i))
|
||||
.collect();
|
||||
assert!(positions.get("func1").unwrap() < positions.get("func2").unwrap());
|
||||
assert!(positions.get("func2").unwrap() < positions.get("func3").unwrap());
|
||||
assert!(positions.get("func3").unwrap() < positions.get("func4").unwrap());
|
||||
assert!(positions.get("func4").unwrap() < positions.get("func5").unwrap());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_non_pipeline_between_pipelines() {
|
||||
let func1 = create_function("func1", vec![Decorator::Pipeline], "body1");
|
||||
let func2 = create_function("func2", vec![], "body2");
|
||||
let func3 = create_function("func3", vec![Decorator::Pipeline], "body3");
|
||||
let functions = vec![func1.clone(), func2.clone(), func3.clone()];
|
||||
|
||||
let result = sort_function(functions).unwrap();
|
||||
assert_eq!(result.len(), 3);
|
||||
let positions: std::collections::HashMap<String, usize> = result
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(i, f)| (f.name.clone(), i))
|
||||
.collect();
|
||||
assert!(positions.get("func1").unwrap() < positions.get("func3").unwrap());
|
||||
}
|
||||
(star_nodes, named_groups)
|
||||
}
|
||||
|
||||
/// Pick a complete group to execute.
|
||||
///
|
||||
/// Deprecated: use `Dag::pick_batch()` instead.
|
||||
pub fn pick_batch(
|
||||
_star_nodes: &[String],
|
||||
_groups: &HashMap<String, Vec<String>>,
|
||||
_group_totals: &HashMap<String, usize>,
|
||||
) -> Option<Vec<String>> {
|
||||
// This function is replaced by Dag::pick_batch() which has all the
|
||||
// group metadata internally. Keeping the signature for backward compat
|
||||
// but redirect callers to use Dag directly.
|
||||
unimplemented!("use dag.pick_batch() instead")
|
||||
}
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
use crate::bake::builder::{build_script, RenderMode};
|
||||
use crate::bake::error::BakeError;
|
||||
use crate::bake::parser::Function;
|
||||
use std::path::Path;
|
||||
use workshop_engine::{Engine, EventSender, ExecutionContext};
|
||||
|
||||
/// Executes `script_content` as a single function in trivial mode.
|
||||
///
|
||||
/// Two entry paths lead here:
|
||||
/// - **Auto-detection** (`!has_pipeline`): bake.sh has no `@pipeline` functions,
|
||||
/// the entire content is treated as one trivial function.
|
||||
/// - **Explicit PIP** (future): `trivial` parameter forces trivial mode even
|
||||
/// when `@pipeline` functions exist, typically with `RenderMode::Off`.
|
||||
pub async fn run_trivial(
|
||||
script_content: &str,
|
||||
bake_base_path: &Path,
|
||||
workspace: &Path,
|
||||
engine: &Engine,
|
||||
ctx: &mut ExecutionContext,
|
||||
event_tx: &EventSender,
|
||||
render_mode: RenderMode,
|
||||
) -> Result<(), BakeError> {
|
||||
let function = Function {
|
||||
name: String::new(),
|
||||
decorators: vec![],
|
||||
body: script_content.to_string(),
|
||||
};
|
||||
let script = build_script(&function, bake_base_path, workspace, render_mode, ctx.debug_flags)?;
|
||||
engine.execute_script(&script, ctx, event_tx).await?;
|
||||
Ok(())
|
||||
}
|
||||
@@ -0,0 +1,61 @@
|
||||
use crate::bake::decorator::Decorator;
|
||||
use crate::bake::parser::Function;
|
||||
use std::collections::HashMap;
|
||||
|
||||
/// Execution mode for a pipeline function.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum FuncMode {
|
||||
Normal,
|
||||
Daemon,
|
||||
Async,
|
||||
}
|
||||
|
||||
/// Determine the execution mode from decorators.
|
||||
/// `@daemon`, `@async`, `@pipe` are mutually exclusive modes.
|
||||
pub fn resolve_mode(func: &Function) -> FuncMode {
|
||||
for d in &func.decorators {
|
||||
match d {
|
||||
Decorator::Daemon => return FuncMode::Daemon,
|
||||
Decorator::Async => return FuncMode::Async,
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
FuncMode::Normal
|
||||
}
|
||||
|
||||
/// True if the function is decorated with `@parallel`.
|
||||
pub fn is_parallel(func: &Function) -> bool {
|
||||
func.decorators
|
||||
.iter()
|
||||
.any(|d| matches!(d, Decorator::Parallel(_)))
|
||||
}
|
||||
|
||||
/// True if the function is decorated with `@fallible`.
|
||||
pub fn is_fallible(func: &Function) -> bool {
|
||||
func.decorators
|
||||
.iter()
|
||||
.any(|d| matches!(d, Decorator::Fallible))
|
||||
}
|
||||
|
||||
/// Resolve stdin for a `@pipe`-decorated function from the stdout cache.
|
||||
///
|
||||
/// Rule 1: self-references are skipped (handled by loop combinator).
|
||||
/// Rule 3: sources present in the cache are concatenated.
|
||||
/// Returns `None` if any source is missing (caller decides based on `@fallible`).
|
||||
pub fn resolve_pipe_stdin(
|
||||
func: &Function,
|
||||
cache: &HashMap<String, String>,
|
||||
) -> Option<Vec<u8>> {
|
||||
let sources = func.decorators.iter().find_map(|d| match d {
|
||||
Decorator::Pipe(names) => Some(names.as_slice()),
|
||||
_ => None,
|
||||
})?;
|
||||
let mut combined = String::new();
|
||||
for source in sources {
|
||||
if *source == func.name {
|
||||
continue; // Rule 1: skip self
|
||||
}
|
||||
combined.push_str(cache.get(source)?); // Rule 3
|
||||
}
|
||||
Some(combined.into_bytes())
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
use crate::cli::{BareCommands, Cli};
|
||||
use crate::error::CliError;
|
||||
use crate::types::resource::ResourceRegistry;
|
||||
use crate::{bake::bake, finalize::finalize, prebake::prebake};
|
||||
use workshop_engine::{EventSender, ExecutionContext};
|
||||
|
||||
pub async fn run_bare(
|
||||
cli: &Cli,
|
||||
command: &BareCommands,
|
||||
ctx: &mut ExecutionContext,
|
||||
event_tx: EventSender,
|
||||
pipeline: &str,
|
||||
build_id: &str,
|
||||
registry: &ResourceRegistry,
|
||||
) -> Result<(), CliError> {
|
||||
let (stage, config) = match command {
|
||||
BareCommands::Prebake { .. } => ("prebake", &cli.prebake),
|
||||
BareCommands::Bake { .. } => ("bake", &cli.bake),
|
||||
BareCommands::Finalize { .. } => {
|
||||
("finalize", &cli.finalize)
|
||||
}
|
||||
};
|
||||
log::debug!("Running bare command {} with config {}", stage, config.display());
|
||||
|
||||
let prebake_path = &cli.prebake;
|
||||
let bake_base = &cli.bake_base;
|
||||
|
||||
ctx.task_id = format!("{}-{}-{}", pipeline, build_id, stage);
|
||||
ctx.pipeline_name = pipeline.to_string();
|
||||
ctx.standalone = true;
|
||||
|
||||
match command {
|
||||
BareCommands::Prebake { .. } => {
|
||||
prebake(&config, cli, None, ctx, event_tx, None).await?;
|
||||
}
|
||||
BareCommands::Bake { .. } => {
|
||||
bake(&config, &bake_base, &prebake_path, cli, ctx, event_tx, None).await?;
|
||||
}
|
||||
BareCommands::Finalize { .. } => {
|
||||
finalize(&config, cli, ctx, event_tx, registry).await?;
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
@@ -1,193 +0,0 @@
|
||||
use crate::cli::Cli;
|
||||
// use serde::Serialize;
|
||||
use serde_json::Value;
|
||||
// use std::path::PathBuf;
|
||||
// use std::sync::Arc;
|
||||
use tokio::io::{AsyncBufReadExt, BufReader};
|
||||
use tokio::net::UnixStream;
|
||||
// use tokio::process::{Child, Command};
|
||||
// use tokio::signal;
|
||||
// use tokio::sync::RwLock;
|
||||
// use tokio::time::{Duration, interval};
|
||||
|
||||
// #[derive(Debug, Clone, Serialize)]
|
||||
// struct Heartbeat {
|
||||
// pipeline_id: String,
|
||||
// build_id: String,
|
||||
// stage: String,
|
||||
// timestamp: u64,
|
||||
// status: String,
|
||||
// subtasks: Vec<String>,
|
||||
// }
|
||||
|
||||
pub async fn daemon(_cli: &Cli) {
|
||||
todo!("Daemon should be refactored");
|
||||
// let socket = PathBuf::from(settings.get_string("socket").unwrap());
|
||||
// let agentsocket = settings.get_string("agentsocket").unwrap();
|
||||
// let _ = std::fs::remove_file(&socket);
|
||||
|
||||
// let heartbeat = Arc::new(RwLock::new(Heartbeat {
|
||||
// pipeline_id: cli.pipeline.clone(),
|
||||
// build_id: cli.build_id.clone(),
|
||||
// stage: String::new(),
|
||||
// timestamp: 0,
|
||||
// status: String::new(),
|
||||
// subtasks: Vec::new(),
|
||||
// }));
|
||||
|
||||
// // let heartbeat_clone = heartbeat.clone();
|
||||
// let heartbeat_task = {
|
||||
// let hb = heartbeat.clone();
|
||||
// tokio::spawn(async move {
|
||||
// let mut tick = interval(Duration::from_secs(1));
|
||||
// loop {
|
||||
// tick.tick().await;
|
||||
// let mut hb = hb.write().await;
|
||||
// hb.timestamp = std::time::SystemTime::now()
|
||||
// .duration_since(std::time::UNIX_EPOCH)
|
||||
// .unwrap()
|
||||
// .as_millis() as u64;
|
||||
|
||||
// // 只输出到stdout（模拟未来发送给Agent）
|
||||
// println!("[HEARTBEAT] {}", serde_json::to_string(&*hb).unwrap());
|
||||
// }
|
||||
// })
|
||||
// };
|
||||
|
||||
// let socket_task = {
|
||||
// let socket = socket.clone();
|
||||
// tokio::spawn(async move {
|
||||
// log::info!("Creating Unix socket at {}", socket.display());
|
||||
// let listener =
|
||||
// UnixListener::bind(&socket).expect("Failed to bind Unix socket");
|
||||
|
||||
// loop {
|
||||
// match listener.accept().await {
|
||||
// Ok((stream, _)) => {
|
||||
// tokio::spawn(handle_connection(stream));
|
||||
// }
|
||||
// Err(e) => {
|
||||
// log::error!("Accept error: {}", e);
|
||||
// break;
|
||||
// }
|
||||
// }
|
||||
// }
|
||||
// })
|
||||
// };
|
||||
|
||||
// let mut tasks: Vec<TaskSpec> = Vec::new();
|
||||
|
||||
// let execution_task = {
|
||||
// let hb = heartbeat.clone();
|
||||
// tokio::spawn(async move {
|
||||
// let mut hb = hb.write().await;
|
||||
// let mut children: Vec<(String, Child)> = vec![];
|
||||
|
||||
// let executable = std::env::current_exe().unwrap();
|
||||
|
||||
// for (index, task) in tasks.iter().enumerate() {
|
||||
// let mut command = task.command(&executable);
|
||||
// command.stdout(std::process::Stdio::piped());
|
||||
// command.stderr(std::process::Stdio::piped());
|
||||
// let child = command
|
||||
// .spawn()
|
||||
// .expect(format!("Failed to spawn child {}", index).as_str());
|
||||
|
||||
// let pid = child.id().unwrap_or(0).to_string();
|
||||
// children.push((pid.clone(), child));
|
||||
|
||||
// hb.subtasks.push(pid.clone());
|
||||
// }
|
||||
|
||||
// for (pid, mut child) in children {
|
||||
// let status = child.wait().await.expect("Failed to wait child");
|
||||
// log::info!("Subtask {} exited with: {}", pid, status);
|
||||
|
||||
// hb.subtasks.retain(|p| p != &pid);
|
||||
// }
|
||||
|
||||
// hb.stage = "Success".to_string();
|
||||
// })
|
||||
// };
|
||||
|
||||
// tokio::select! {
|
||||
// _ = heartbeat_task => {
|
||||
// log::error!("Heartbeat task died");
|
||||
// }
|
||||
// _ = execution_task => {
|
||||
// log::info!("All subtasks completed");
|
||||
// }
|
||||
// _ = socket_task => {
|
||||
// log::error!("Socket task died");
|
||||
// }
|
||||
// _ = signal::ctrl_c() => {
|
||||
// log::info!("Shutting down...");
|
||||
// }
|
||||
// }
|
||||
// for pid in heartbeat.read().await.subtasks.clone() {
|
||||
// let _ = Command::new("kill").arg("-TERM").arg(&pid).status().await;
|
||||
// }
|
||||
|
||||
// let _ = std::fs::remove_file(&socket);
|
||||
}
|
||||
|
||||
async fn _handle_connection(stream: UnixStream) {
|
||||
let peer_addr = match stream.peer_addr() {
|
||||
Ok(addr) => format!("{:?}", addr),
|
||||
Err(_) => "unknown".to_string(),
|
||||
};
|
||||
|
||||
log::debug!("New connection from: {}", peer_addr);
|
||||
|
||||
let mut reader = BufReader::new(stream);
|
||||
let mut line = String::new();
|
||||
|
||||
loop {
|
||||
line.clear();
|
||||
match reader.read_line(&mut line).await {
|
||||
Ok(0) => {
|
||||
log::debug!("Connection closed by peer: {}", peer_addr);
|
||||
break;
|
||||
}
|
||||
Ok(_) => {
|
||||
// 简单打印原始JSON（预留字段解析空间）
|
||||
match serde_json::from_str::<Value>(&line) {
|
||||
Ok(json) => {
|
||||
// 只提取关键字段打印（字段可能变化）
|
||||
let msgtype = json
|
||||
.get("msgtype")
|
||||
.and_then(|v| v.as_str())
|
||||
.unwrap_or("unknown");
|
||||
let name = json
|
||||
.get("name")
|
||||
.and_then(|v| v.as_str())
|
||||
.unwrap_or("unnamed");
|
||||
|
||||
log::info!("[{}] Message from '{}': {}", msgtype, name, line.trim());
|
||||
|
||||
// TODO: 字段稳定后，可扩展为结构化打印
|
||||
// match msgtype {
|
||||
// "ResourceUsage" => print_resource(&json),
|
||||
// "Log" => print_log(&json),
|
||||
// _ => log::debug!("Unknown type: {}", msgtype),
|
||||
// }
|
||||
}
|
||||
Err(e) => {
|
||||
log::warn!(
|
||||
"Invalid JSON from {}: {} - Error: {}",
|
||||
peer_addr,
|
||||
line.trim(),
|
||||
e
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(e) => {
|
||||
log::error!("Failed to read from {}: {}", peer_addr, e);
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
log::debug!("Connection handler for {} exiting", peer_addr);
|
||||
}
|
||||
|
||||
Vendored
@@ -0,0 +1,30 @@
|
||||
use thiserror::Error;
|
||||
use workshop_engine::HasExitCode;
|
||||
|
||||
#[derive(Error, Debug)]
|
||||
pub enum CliError {
|
||||
#[error("{0}")]
|
||||
Prebake(#[from] crate::prebake::error::PrebakeError),
|
||||
#[error("{0}")]
|
||||
Bake(#[from] crate::bake::error::BakeError),
|
||||
#[error("{0}")]
|
||||
Finalize(#[from] crate::finalize::FinalizeError),
|
||||
#[error("{0}")]
|
||||
General(anyhow::Error),
|
||||
}
|
||||
|
||||
impl From<anyhow::Error> for CliError {
|
||||
fn from(e: anyhow::Error) -> Self {
|
||||
CliError::General(e)
|
||||
}
|
||||
}
|
||||
impl CliError {
|
||||
pub fn exit_code(&self) -> i32 {
|
||||
match self {
|
||||
CliError::Prebake(e) => e.exit_code(),
|
||||
CliError::Bake(e) => e.exit_code(),
|
||||
CliError::Finalize(e) => e.exit_code(),
|
||||
CliError::General(_) => 1,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -2,18 +2,18 @@ pub mod config;
|
||||
pub mod constant;
|
||||
pub mod error;
|
||||
pub mod event;
|
||||
pub mod cleanup;
|
||||
pub mod artifact;
|
||||
pub mod plugin;
|
||||
pub mod stage;
|
||||
pub mod template;
|
||||
pub mod types;
|
||||
|
||||
use crate::cli::Cli;
|
||||
use crate::finalize::constant::FINALIZE_STANDALONE_PLUGIN_PATH;
|
||||
use crate::finalize::plugin::fetch::{self, FetchArgument};
|
||||
use crate::types::resource::ResourceRegistry;
|
||||
pub use config::*;
|
||||
pub use error::FinalizeError;
|
||||
use std::path::Path;
|
||||
use std::path::PathBuf;
|
||||
use workshop_engine::EventSender;
|
||||
|
||||
pub fn parse(path: &Path) -> Result<FinalizeConfig, FinalizeError> {
|
||||
@@ -27,6 +27,7 @@ pub async fn finalize(
|
||||
_cli: &Cli,
|
||||
ctx: &mut workshop_engine::ExecutionContext,
|
||||
event_tx: EventSender,
|
||||
registry: &ResourceRegistry,
|
||||
) -> Result<(), FinalizeError> {
|
||||
let mut finalize = parse(finalize_path)?;
|
||||
let validate_result = finalize.validate();
|
||||
@@ -36,33 +37,10 @@ pub async fn finalize(
|
||||
}
|
||||
return Err(FinalizeError::ValidateError(errors));
|
||||
};
|
||||
if ctx.standalone {
|
||||
// In standalone mode, finalize should download plugin by itself
|
||||
log::debug!("Downloading {} plugins", finalize.plugin.len());
|
||||
let plugin_path = PathBuf::from(FINALIZE_STANDALONE_PLUGIN_PATH);
|
||||
// if plugin_path.exists() {
|
||||
// std::fs::remove_dir_all(&plugin_path).map_err(FinalizeError::IoError)?;
|
||||
// }
|
||||
for (name, data) in finalize.plugin.iter_mut() {
|
||||
log::info!("Downloading plugin {}", name);
|
||||
let (source, config) = data.iter_mut().next().unwrap_or_else(|| {
|
||||
panic!(
|
||||
"Internal Error: Plugin {} has no source configured, this should have been caught by validation!",
|
||||
name
|
||||
)
|
||||
});
|
||||
let fetch_argument = FetchArgument {
|
||||
checksum: config.checksum.clone(),
|
||||
shallow: config.shallow,
|
||||
};
|
||||
fetch::fetch_plugin(source, name, &plugin_path, fetch_argument).await?;
|
||||
config.runtime.fspath = Some(plugin_path.join(name));
|
||||
}
|
||||
}
|
||||
|
||||
// Register Plugins
|
||||
// Register Plugins (paths resolved from registry)
|
||||
log::info!("Registering {} plugins", finalize.plugin.len());
|
||||
let registered_plugins = plugin::register(finalize.plugin, None)?;
|
||||
let registered_plugins = plugin::register(finalize.plugin, None, &mut finalize.notification, registry)?;
|
||||
|
||||
// Early hook
|
||||
let earlyhook_result = stage::hook::hook(
|
||||
@@ -76,9 +54,17 @@ pub async fn finalize(
|
||||
// Determine if we should continue with artifacts/latehook
|
||||
// If prior stages failed, we still send notification but skip artifacts
|
||||
let _prior_failed = earlyhook_result.is_err();
|
||||
|
||||
// Prepare artifacts
|
||||
// TODO: artifact stage implementation
|
||||
// Artifact collection and publishing
|
||||
if !_prior_failed {
|
||||
log::info!("Processing {} artifacts", finalize.artifact.len());
|
||||
artifact::collect_and_publish(
|
||||
&finalize.artifact,
|
||||
®istered_plugins,
|
||||
ctx,
|
||||
event_tx.clone(),
|
||||
)
|
||||
.await?;
|
||||
}
|
||||
|
||||
// Late hook
|
||||
stage::hook::hook(
|
||||
|
||||
@@ -1,56 +0,0 @@
|
||||
# workshop-baker/src/finalize
|
||||
|
||||
**Generated:** 2026-04-22
|
||||
**Commit:** 7b3b71a
|
||||
|
||||
Artifact packaging, distribution, deployment, and notifications.
|
||||
|
||||
## STRUCTURE
|
||||
|
||||
```
|
||||
finalize/
|
||||
├── config.rs # FinalizeConfig YAML (artifacts, deploy, notify)
|
||||
├── stage.rs # FinalizeStage enum
|
||||
├── stage/hook.rs # Post-build hooks
|
||||
├── event.rs # Event propagation
|
||||
├── template.rs # Variable substitution {{VAR}}
|
||||
├── plugin.rs # Plugin trait + registry
|
||||
├── plugin/
|
||||
│ ├── metadata.rs # Plugin manifest parsing
|
||||
│ ├── dylib.rs # Dynamic library plugins
|
||||
│ ├── rhai.rs # Rhai scripting (todo!())
|
||||
│ ├── shell.rs # Shell command plugins
|
||||
│ └── internal/ # Built-in plugins
|
||||
│ ├── mail/ # Email notifications (SMTP)
|
||||
│ ├── webhook.rs # HTTP webhooks
|
||||
│ ├── satori.rs # Satori integration
|
||||
│ └── insitenotify.rs # In-site notifications
|
||||
│ └── fetch/ # Artifact fetching
|
||||
│ ├── git.rs # Git repository fetch
|
||||
│ ├── http.rs # HTTP download
|
||||
│ ├── extract.rs # Archive extraction
|
||||
│ └── checksum.rs# SHA256 verification (SHA1 deprecated)
|
||||
```
|
||||
|
||||
## WHERE TO LOOK
|
||||
|
||||
| Task | Location |
|
||||
|------|----------|
|
||||
| Config loading | `config.rs` - FinalizeConfig |
|
||||
| Email notify | `plugin/internal/mail/` - SMTP templates |
|
||||
| Webhooks | `plugin/internal/webhook.rs` |
|
||||
| Artifact fetch | `plugin/fetch/` - git/http/extract |
|
||||
| Checksums | `plugin/fetch/checksum.rs:13` - SHA1 deprecated |
|
||||
|
||||
## CONVENTIONS
|
||||
|
||||
- **Plugin System**: Dynamic + internal plugins via Plugin trait
|
||||
- **Template Syntax**: `{{VARIABLE}}` handlebars-style
|
||||
- **Stages**: Hook → Fetch → Build → Publish → Notify → Cleanup
|
||||
- **Fetch**: Supports git, http, with extraction and checksum verify
|
||||
|
||||
## ANTI-PATTERNS
|
||||
|
||||
1. **SHA1 deprecated** — `checksum.rs:13` warns SHA1 is deprecated
|
||||
2. **Unimplemented** — `rhai.rs:17` - "todo!()" for Rhai scripting
|
||||
3. **TODO markers** — Multiple TODOs in config.rs (lines 305-306)
|
||||
@@ -0,0 +1,139 @@
|
||||
//! Artifact collection and publishing.
|
||||
//!
|
||||
//! Collects build artifacts (via glob patterns), optionally compresses them,
|
||||
//! and dispatches each artifact through its configured publish plugins.
|
||||
//! The publish plugin receives the artifact path as a system-injected
|
||||
//! `artifact` field in its argument.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
use crate::finalize::config::ArtifactDef;
|
||||
use crate::finalize::error::FinalizeError;
|
||||
use crate::finalize::plugin::PluginMap;
|
||||
use crate::finalize::types::compression::CompressionMethod;
|
||||
use workshop_engine::{EventSender, ExecutionContext};
|
||||
|
||||
/// Collect artifacts and execute their publish plugins.
|
||||
///
|
||||
/// For each artifact:
|
||||
/// 1. Resolve source files via glob
|
||||
/// 2. Optionally compress (zstd, gzip, dir)
|
||||
/// 3. Inject `artifact` key into each publish plugin's config
|
||||
/// 4. Call the plugin
|
||||
pub async fn collect_and_publish(
|
||||
artifacts: &[ArtifactDef],
|
||||
plugins: &PluginMap,
|
||||
ctx: &ExecutionContext,
|
||||
event_tx: EventSender,
|
||||
) -> Result<(), FinalizeError> {
|
||||
if artifacts.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
for artifact in artifacts {
|
||||
let artifact_path = process_artifact(artifact).await?;
|
||||
|
||||
if let Some(ref path) = artifact_path {
|
||||
log::info!("Artifact processed: {} -> {}", artifact.path, path.display());
|
||||
}
|
||||
|
||||
for (plugin_name, publish_config) in &artifact.publish {
|
||||
let plugin = plugins.get(plugin_name).ok_or_else(|| {
|
||||
FinalizeError::PluginExecutionFailed(format!(
|
||||
"publish plugin '{}' not found for artifact '{}'",
|
||||
plugin_name, artifact.path,
|
||||
))
|
||||
})?;
|
||||
|
||||
// Inject artifact path into the plugin argument
|
||||
let argument = if let Some(ref path) = artifact_path {
|
||||
let mut arg = publish_config.clone();
|
||||
if let Some(map) = arg.as_mapping_mut() {
|
||||
map.insert(
|
||||
serde_yaml::Value::String("artifact".into()),
|
||||
serde_yaml::Value::String(path.to_string_lossy().into()),
|
||||
);
|
||||
}
|
||||
arg
|
||||
} else {
|
||||
publish_config.clone()
|
||||
};
|
||||
|
||||
log::info!(
|
||||
"Publishing artifact '{}' via plugin '{}'",
|
||||
artifact.path, plugin_name,
|
||||
);
|
||||
|
||||
plugin.call(argument, ctx, &event_tx).await?;
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Resolve and optionally compress an artifact.
|
||||
/// Returns the local path to the processed artifact, or `None` if non-file artifact.
|
||||
async fn process_artifact(artifact: &ArtifactDef) -> Result<Option<PathBuf>, FinalizeError> {
|
||||
if artifact.path.is_empty() {
|
||||
return Ok(None); // non-filesystem artifact (e.g. Docker image)
|
||||
}
|
||||
|
||||
let src = PathBuf::from(&artifact.path);
|
||||
if !src.exists() {
|
||||
if artifact.path.contains('*') || artifact.path.contains('?') {
|
||||
return Err(FinalizeError::PluginExecutionFailed(
|
||||
format!("glob patterns not supported: '{}'; use a directory path", artifact.path),
|
||||
));
|
||||
}
|
||||
return Err(FinalizeError::ArtifactNotFound(src));
|
||||
}
|
||||
|
||||
match artifact.compression {
|
||||
CompressionMethod::None => Ok(Some(src)),
|
||||
CompressionMethod::Dir => Ok(Some(src)),
|
||||
CompressionMethod::Gzip => {
|
||||
let dst = std::env::temp_dir().join(format!("artifact-{}.tar.gz", artifact.id.as_deref().unwrap_or("unknown")));
|
||||
compress_tar_zstd_or_gzip(&src, &dst, false).await?;
|
||||
Ok(Some(dst))
|
||||
}
|
||||
CompressionMethod::Zstd => {
|
||||
let dst = std::env::temp_dir().join(format!("artifact-{}.tar.zst", artifact.id.as_deref().unwrap_or("unknown")));
|
||||
compress_tar_zstd_or_gzip(&src, &dst, true).await?;
|
||||
Ok(Some(dst))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn compress_tar_zstd_or_gzip(
|
||||
src: &Path,
|
||||
dst: &Path,
|
||||
zstd: bool,
|
||||
) -> Result<(), FinalizeError> {
|
||||
let _archive = tokio::task::spawn_blocking({
|
||||
let src = src.to_path_buf();
|
||||
let dst = dst.to_path_buf();
|
||||
move || -> Result<(), String> {
|
||||
let file = std::fs::File::create(&dst)
|
||||
.map_err(|e| format!("cannot create {}: {}", dst.display(), e))?;
|
||||
let writer: Box<dyn std::io::Write> = if zstd {
|
||||
Box::new(zstd::stream::write::Encoder::new(file, 0).map_err(|e| e.to_string())?)
|
||||
} else {
|
||||
Box::new(flate2::write::GzEncoder::new(file, flate2::Compression::default()))
|
||||
};
|
||||
let mut tar = tar::Builder::new(writer);
|
||||
if src.is_dir() {
|
||||
tar.append_dir_all(".", &src)
|
||||
.map_err(|e| format!("tar error: {}", e))?;
|
||||
} else {
|
||||
tar.append_path(&src)
|
||||
.map_err(|e| format!("tar error: {}", e))?;
|
||||
}
|
||||
tar.into_inner().map_err(|e| e.to_string())?;
|
||||
Ok(())
|
||||
}
|
||||
})
|
||||
.await
|
||||
.map_err(|e| FinalizeError::PluginExecutionFailed(format!("spawn_blocking: {}", e)))?
|
||||
.map_err(FinalizeError::PluginExecutionFailed)?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
//! Pipeline cleanup and resource reclamation.
|
||||
//!
|
||||
//! Runs after all pipeline stages and notification dispatch complete.
|
||||
//! Checks the configured [`CleanupPolicy`] and performs the following:
|
||||
//!
|
||||
//! 1. Temp file / cache directory removal
|
||||
//! 2. Policy-based skip (never, manual, on_success)
|
||||
|
||||
use crate::finalize::config::{CleanupConfig, CleanupPolicy};
|
||||
use crate::finalize::error::FinalizeError;
|
||||
use workshop_engine::ExecutionContext;
|
||||
|
||||
/// Execute cleanup according to the pipeline's [`CleanupPolicy`].
|
||||
///
|
||||
/// Returns `Ok(())` when cleanup completes (or is skipped per policy).
|
||||
/// Returns `Err` only if a cleanup action itself fails (not if the
|
||||
/// pipeline failed; pipeline outcome is checked via `pipeline_ok`).
|
||||
pub fn cleanup(
|
||||
config: &CleanupConfig,
|
||||
_ctx: &ExecutionContext,
|
||||
pipeline_ok: bool,
|
||||
) -> Result<(), FinalizeError> {
|
||||
match config.policy {
|
||||
CleanupPolicy::Never => {
|
||||
log::info!("[cleanup] policy=never, skipping all cleanup");
|
||||
return Ok(());
|
||||
}
|
||||
CleanupPolicy::Manual => {
|
||||
log::info!("[cleanup] policy=manual, hanging for bakerd signal");
|
||||
// TODO(daemon): hang on bakerd signal
|
||||
// For now (standalone/bare) manual == Never
|
||||
return Ok(());
|
||||
}
|
||||
CleanupPolicy::OnSuccess if !pipeline_ok => {
|
||||
log::info!("[cleanup] policy=on_success, but pipeline failed — skipping");
|
||||
// TODO(daemon): hang for bakerd
|
||||
return Ok(());
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
log::info!("[cleanup] starting");
|
||||
cleanup_temp_files();
|
||||
subprocess_reclaim();
|
||||
log::info!("[cleanup] complete");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Reclaim child processes (placeholder for daemon-mode PID tracking).
|
||||
/// In standalone mode, children are reaped by the kernel on process exit.
|
||||
fn subprocess_reclaim() {
|
||||
// TODO(daemon): track spawned PIDs during pipeline and signal them here.
|
||||
}
|
||||
/// Remove plugin cache and temporary directories.
|
||||
fn cleanup_temp_files() {
|
||||
let cache = std::path::Path::new("/tmp/baker");
|
||||
if cache.exists() {
|
||||
match std::fs::remove_dir_all(cache) {
|
||||
Ok(()) => log::debug!("[cleanup] removed cache: {}", cache.display()),
|
||||
Err(e) => log::warn!("[cleanup] failed to remove {}: {}", cache.display(), e),
|
||||
}
|
||||
}
|
||||
|
||||
let workshop_templates = std::env::temp_dir().join("workshop-templates");
|
||||
if workshop_templates.exists() {
|
||||
let _ = std::fs::remove_dir_all(&workshop_templates);
|
||||
}
|
||||
}
|
||||
@@ -115,6 +115,17 @@ pub struct NotificationMethod {
|
||||
pub policy: Vec<NotifyPolicy>,
|
||||
}
|
||||
|
||||
impl Default for NotificationMethod {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
template: NotificationTemplate::default(),
|
||||
trigger: Vec::new(),
|
||||
config: Value::default(),
|
||||
policy: Vec::new(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize, Serialize, Clone, Default)]
|
||||
pub struct NotificationTemplate {
|
||||
/// Payload schema: "default" (plugin-defined) or "custom" (requires on_* templates)
|
||||
@@ -134,17 +145,65 @@ pub struct NotificationTemplate {
|
||||
pub on_finish_of: Option<OnFinishOf>,
|
||||
}
|
||||
|
||||
impl std::ops::AddAssign for NotificationTemplate {
|
||||
fn add_assign(&mut self, other: Self) {
|
||||
if self.schema.is_none() {
|
||||
self.schema = other.schema;
|
||||
}
|
||||
if self.on_success.is_none() {
|
||||
self.on_success = other.on_success;
|
||||
}
|
||||
if self.on_failure.is_none() {
|
||||
self.on_failure = other.on_failure;
|
||||
}
|
||||
if self.on_finish_of.is_none() {
|
||||
self.on_finish_of = other.on_finish_of;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Notification configuration container
|
||||
#[derive(Debug, Deserialize, Serialize, Clone, Default)]
|
||||
#[derive(Debug, Serialize, Clone, Default)]
|
||||
pub struct NotificationConfig {
|
||||
/// Template definitions for notification content
|
||||
#[serde(default)]
|
||||
pub templates: HashMap<String, NotificationTemplateDef>,
|
||||
|
||||
#[serde(flatten)]
|
||||
/// Priority groups (YAML keys like `0`, `1` are parsed as u32).
|
||||
/// Custom Deserialize handles serde_yaml string→u32 conversion.
|
||||
#[serde(default)]
|
||||
pub groups: HashMap<u32, HashMap<String, NotificationMethod>>,
|
||||
}
|
||||
|
||||
/// Serde helper: YAML map keys are always strings, convert to u32.
|
||||
impl<'de> serde::Deserialize<'de> for NotificationConfig {
|
||||
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
|
||||
where D: serde::Deserializer<'de>
|
||||
{
|
||||
#[derive(serde::Deserialize)]
|
||||
struct Helper {
|
||||
#[serde(default)]
|
||||
templates: HashMap<String, NotificationTemplateDef>,
|
||||
#[serde(flatten)]
|
||||
groups_raw: HashMap<String, Option<HashMap<String, NotificationMethod>>>,
|
||||
}
|
||||
let h = Helper::deserialize(deserializer)?;
|
||||
let groups: HashMap<u32, _> = h.groups_raw.into_iter()
|
||||
.filter_map(|(k, v)| {
|
||||
let parsed = k.parse::<u32>().map(|n| (n, v.unwrap_or_default()));
|
||||
match parsed {
|
||||
Ok(pair) => Some(pair),
|
||||
Err(e) => {
|
||||
log::warn!("priority key must be integer, got {:?}: {}", k, e);
|
||||
None
|
||||
}
|
||||
}
|
||||
})
|
||||
.collect();
|
||||
Ok(NotificationConfig { templates: h.templates, groups })
|
||||
}
|
||||
}
|
||||
|
||||
/// Stage-triggered notification configuration
|
||||
#[derive(Debug, Deserialize, Serialize, Clone)]
|
||||
pub struct OnFinishOf {
|
||||
@@ -158,7 +217,7 @@ pub struct OnFinishOf {
|
||||
|
||||
/// Notification template definition.
|
||||
/// Supports inline content (on_success/on_failure/on_finish_of) or external reference (use).
|
||||
#[derive(Debug, Deserialize, Serialize, Clone)]
|
||||
#[derive(Debug, Deserialize, Serialize, Clone, Default)]
|
||||
pub struct NotificationTemplateDef {
|
||||
/// External template reference (git URL).
|
||||
/// Cannot be combined with on_* fields in MVP (validation will reject).
|
||||
@@ -178,6 +237,53 @@ pub struct NotificationTemplateDef {
|
||||
pub on_finish_of: Option<String>,
|
||||
}
|
||||
|
||||
impl NotificationTemplateDef {
|
||||
/// Merge `other` into `self`, keeping self's values when present.
|
||||
///
|
||||
/// `other` fills in gaps where `self` is `None`.
|
||||
/// Chaining: `a.apply(b).apply(c)` = a wins, then b fills gaps, then c fills gaps.
|
||||
pub fn apply(mut self, other: Self) -> Self {
|
||||
if self.on_success.is_none() {
|
||||
self.on_success = other.on_success;
|
||||
}
|
||||
if self.on_failure.is_none() {
|
||||
self.on_failure = other.on_failure;
|
||||
}
|
||||
if self.on_finish_of.is_none() {
|
||||
self.on_finish_of = other.on_finish_of;
|
||||
}
|
||||
self
|
||||
}
|
||||
}
|
||||
|
||||
/// Builder for [`NotificationTemplateDef`].
|
||||
pub struct NotificationTemplateDefBuilder(NotificationTemplateDef);
|
||||
|
||||
impl NotificationTemplateDefBuilder {
|
||||
pub fn new() -> Self {
|
||||
Self(NotificationTemplateDef::default())
|
||||
}
|
||||
pub fn with_use(mut self, val: impl Into<String>) -> Self {
|
||||
self.0.use_ = Some(val.into());
|
||||
self
|
||||
}
|
||||
pub fn with_on_success(mut self, val: impl Into<String>) -> Self {
|
||||
self.0.on_success = Some(val.into());
|
||||
self
|
||||
}
|
||||
pub fn with_on_failure(mut self, val: impl Into<String>) -> Self {
|
||||
self.0.on_failure = Some(val.into());
|
||||
self
|
||||
}
|
||||
pub fn with_on_finish_of(mut self, val: impl Into<String>) -> Self {
|
||||
self.0.on_finish_of = Some(val.into());
|
||||
self
|
||||
}
|
||||
pub fn build(self) -> NotificationTemplateDef {
|
||||
self.0
|
||||
}
|
||||
}
|
||||
|
||||
/// Notification trigger type.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum Trigger {
|
||||
@@ -236,8 +342,9 @@ pub struct ArtifactDef {
|
||||
/// Compression method: "zstd", "gzip", "none", "dir"
|
||||
#[serde(default)]
|
||||
pub compression: CompressionMethod,
|
||||
// Publishing targets
|
||||
// TODO: under heavy refactor
|
||||
/// Publish targets: plugin name → configuration (artifact path injected by system)
|
||||
#[serde(default)]
|
||||
pub publish: HashMap<String, serde_yaml::Value>
|
||||
}
|
||||
|
||||
impl Default for ArtifactDef {
|
||||
@@ -247,6 +354,7 @@ impl Default for ArtifactDef {
|
||||
path: String::new(),
|
||||
retention_ms: default_retention_ms(),
|
||||
compression: CompressionMethod::default(),
|
||||
publish: HashMap::new(),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -336,15 +444,18 @@ impl FinalizeConfig {
|
||||
fn validate_notification(&self, errors: &mut Vec<String>) {
|
||||
// Validate templates
|
||||
for (name, tmpl) in &self.notification.templates {
|
||||
if tmpl.use_.is_some()
|
||||
&& (tmpl.on_success.is_some()
|
||||
|| tmpl.on_failure.is_some()
|
||||
|| tmpl.on_finish_of.is_some())
|
||||
{
|
||||
errors.push(format!(
|
||||
"Template '{}' cannot combine 'use' with 'on_*' fields (MVP restriction)",
|
||||
name
|
||||
));
|
||||
if let Some(ref u) = tmpl.use_ {
|
||||
if u.contains("://") && (
|
||||
tmpl.on_success.is_some() ||
|
||||
tmpl.on_failure.is_some() ||
|
||||
tmpl.on_finish_of.is_some()
|
||||
) {
|
||||
errors.push(format!(
|
||||
"Template '{}' cannot combine external 'use' with 'on_*' fields",
|
||||
name
|
||||
));
|
||||
}
|
||||
// Local references (no ://) are allowed to have on_* overrides
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
use workshop_engine::{ExecutionEvent, StreamType};
|
||||
use crate::types::debug::DebugFeature;
|
||||
|
||||
/// Receives and processes execution events for the finalize stage.
|
||||
/// Logs task lifecycle events (started, completed, failed) and streaming output (stdout/stderr)
|
||||
@@ -7,43 +8,62 @@ pub async fn event_receiver(
|
||||
mut rx: tokio::sync::mpsc::Receiver<ExecutionEvent>,
|
||||
pipeline_name: String,
|
||||
build_id: String,
|
||||
debug_flags: u32,
|
||||
) {
|
||||
let events_debug = DebugFeature::from_bits_retain(debug_flags).contains(DebugFeature::Events);
|
||||
while let Some(event) = rx.recv().await {
|
||||
match event {
|
||||
ExecutionEvent::TaskStarted { task_id, timestamp } => {
|
||||
log::debug!(
|
||||
"[{}] [{}] [{}] Task started: {}",
|
||||
pipeline_name,
|
||||
build_id,
|
||||
timestamp,
|
||||
task_id
|
||||
);
|
||||
if events_debug {
|
||||
log::info!(
|
||||
"[{}] [{}] [{}] Task started: {}",
|
||||
pipeline_name,
|
||||
build_id,
|
||||
timestamp,
|
||||
task_id
|
||||
);
|
||||
} else {
|
||||
log::debug!(
|
||||
"Task started: {}",
|
||||
task_id
|
||||
);
|
||||
}
|
||||
}
|
||||
ExecutionEvent::OutputChunk {
|
||||
task_id,
|
||||
task_id: _,
|
||||
stream,
|
||||
data,
|
||||
} => match stream {
|
||||
StreamType::Stdout => print!("[{}] [{}] [stdout] {}", pipeline_name, task_id, data),
|
||||
StreamType::Stdout => print!("{}", data),
|
||||
StreamType::Stderr => {
|
||||
eprint!("[{}] [{}] [stderr] {}", pipeline_name, task_id, data)
|
||||
eprint!("{}", data)
|
||||
}
|
||||
},
|
||||
ExecutionEvent::TaskCompleted { task_id, result } => {
|
||||
log::debug!(
|
||||
"[{}] [{}] Task completed: {} (exit_code={}, duration={:?})",
|
||||
pipeline_name,
|
||||
build_id,
|
||||
task_id,
|
||||
result.exit_code,
|
||||
result.duration
|
||||
);
|
||||
if result.success {
|
||||
if events_debug {
|
||||
log::info!(
|
||||
"[{}] [{}] Task completed: {} (exit_code={}, duration={:?})",
|
||||
pipeline_name, build_id, task_id,
|
||||
result.exit_code, result.duration,
|
||||
);
|
||||
} else {
|
||||
log::debug!(
|
||||
"Task completed: {} (exit_code={}, duration={:?})",
|
||||
task_id, result.exit_code, result.duration,
|
||||
);
|
||||
}
|
||||
} else {
|
||||
log::warn!(
|
||||
"Task completed with non-zero exit: {} (exit_code={}, duration={:?})",
|
||||
task_id,
|
||||
result.exit_code, result.duration,
|
||||
);
|
||||
}
|
||||
}
|
||||
ExecutionEvent::TaskFailed { task_id, error } => {
|
||||
log::error!(
|
||||
"[{}] [{}] Task failed: {} (error={})",
|
||||
pipeline_name,
|
||||
build_id,
|
||||
"Task failed: {} (error={})",
|
||||
task_id,
|
||||
error
|
||||
);
|
||||
|
||||
@@ -1,15 +1,16 @@
|
||||
use crate::finalize::RawPluginMap;
|
||||
use crate::finalize::constant::{FINALIZE_PLUGIN_MANIFEST, FINALIZE_PLUGIN_MANIFEST_EXTENSION};
|
||||
use crate::finalize::plugin::metadata::{PluginMetadata, PluginType};
|
||||
use crate::{EventSender, ExecutionContext, finalize::error::PluginError};
|
||||
use workshop_engine::{EventSender, ExecutionContext};
|
||||
use crate::{ finalize::error::PluginError};
|
||||
use crate::types::resource::ResourceRegistry;
|
||||
use async_trait::async_trait;
|
||||
use std::collections::{HashMap, HashSet};
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::Arc;
|
||||
|
||||
mod dylib;
|
||||
pub mod fetch;
|
||||
mod internal;
|
||||
pub mod internal;
|
||||
mod metadata;
|
||||
mod rhai;
|
||||
mod shell;
|
||||
@@ -214,6 +215,8 @@ fn get_entry(plugin_type: PluginType, name: &str) -> Result<Box<dyn AsyncPluginF
|
||||
pub fn register(
|
||||
external_plugins: RawPluginMap,
|
||||
list: Option<&HashSet<String>>,
|
||||
notification_config: &mut crate::finalize::NotificationConfig,
|
||||
registry: &ResourceRegistry,
|
||||
) -> Result<PluginMap, PluginError> {
|
||||
let mut plugins: PluginMap = HashMap::new();
|
||||
let mut count = 0;
|
||||
@@ -229,7 +232,9 @@ pub fn register(
|
||||
_ => {}
|
||||
}
|
||||
log::debug!("Registering internal plugin {}", i.name);
|
||||
plugins.insert(i.name.to_string(), FinalizePlugin::new(i.entry));
|
||||
let mut plugin = FinalizePlugin::new(i.entry);
|
||||
plugin.metadata.renderer = Some(i.renderer);
|
||||
plugins.insert(i.name.to_string(), plugin);
|
||||
count += 1;
|
||||
}
|
||||
|
||||
@@ -255,7 +260,14 @@ pub fn register(
|
||||
log::warn!("Plugin {} is not available: {}", name, reason);
|
||||
continue;
|
||||
}
|
||||
let metadata = read_manifest(config.runtime.fspath, &name)?;
|
||||
// Resolve plugin path from registry instead of config.runtime.fspath
|
||||
let plugin_path = registry.resolve(&name).ok_or_else(|| {
|
||||
PluginError::PluginUnavailable {
|
||||
name: name.clone(),
|
||||
reason: "plugin not found in resource registry".to_string(),
|
||||
}
|
||||
})?;
|
||||
let metadata = read_manifest(Some(plugin_path), &name)?;
|
||||
let plugin_type = get_plugin_type(metadata.plugin_type.clone(), &metadata.entrypoint);
|
||||
let entry = get_entry(plugin_type, &name)?;
|
||||
plugins.insert(
|
||||
@@ -267,8 +279,18 @@ pub fn register(
|
||||
count += 1;
|
||||
}
|
||||
|
||||
log::info!("Registered {} plugins", count);
|
||||
// Inject plugin-declared templates as .fallback
|
||||
for (name, plugin) in &plugins {
|
||||
if let Some(ref templates) = plugin.metadata.templates {
|
||||
let key = format!("{}.fallback", name);
|
||||
for tmpl in templates {
|
||||
notification_config.templates.insert(key.clone(), tmpl.clone());
|
||||
}
|
||||
log::debug!("Registered templates from plugin '{}' as '{}'", name, key);
|
||||
}
|
||||
}
|
||||
|
||||
log::info!("Registered {} plugins", count);
|
||||
Ok(plugins)
|
||||
}
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
use crate::finalize::plugin::PluginError;
|
||||
use crate::finalize::plugin::PluginMetadata;
|
||||
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn};
|
||||
use workshop_engine::{EventSender, ExecutionContext};
|
||||
use crate::{ finalize::plugin::AsyncPluginFn};
|
||||
use async_trait::async_trait;
|
||||
|
||||
pub struct Dylib;
|
||||
|
||||
@@ -1,109 +0,0 @@
|
||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
||||
use crate::finalize::fetch;
|
||||
use crate::finalize::plugin::{PluginError, RawPluginMap};
|
||||
use std::collections::HashSet;
|
||||
use std::path::Path;
|
||||
|
||||
mod checksum;
|
||||
mod extract;
|
||||
mod git;
|
||||
mod http;
|
||||
mod types;
|
||||
|
||||
pub use types::FetchArgument;
|
||||
|
||||
enum URLScheme {
|
||||
Unknown,
|
||||
Git,
|
||||
Http,
|
||||
Https,
|
||||
File,
|
||||
}
|
||||
|
||||
impl URLScheme {
|
||||
fn parse(url: &str) -> (Self, String) {
|
||||
match url.split_once("://") {
|
||||
Some((prefix, rest)) => {
|
||||
let scheme = match prefix {
|
||||
"git" => URLScheme::Git,
|
||||
"https" => URLScheme::Https,
|
||||
"http" => URLScheme::Http,
|
||||
"file" => URLScheme::File,
|
||||
_ => URLScheme::Unknown,
|
||||
};
|
||||
(scheme, rest.to_string())
|
||||
}
|
||||
None => (URLScheme::Unknown, url.to_string()),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn fetch_plugin(
|
||||
url: &str,
|
||||
name: &str,
|
||||
basedir: &Path,
|
||||
argument: FetchArgument,
|
||||
) -> Result<(), PluginError> {
|
||||
let (scheme, path) = URLScheme::parse(url);
|
||||
|
||||
match scheme {
|
||||
URLScheme::Git => {
|
||||
// NOTE: as a practical compromise, we use git command for git clone
|
||||
// gix will be supported later, with ONLY https scheme
|
||||
// TODO: Implement gix
|
||||
log::info!("Fetching git plugin: {}", url);
|
||||
git::fetch_git_plugin(&path, name, basedir, argument.shallow).await
|
||||
}
|
||||
URLScheme::Http | URLScheme::Https => {
|
||||
log::info!("Fetching http plugin: {}", url);
|
||||
let checksum = argument.get_checksum();
|
||||
http::fetch_http_plugin(url, name, basedir, checksum).await
|
||||
}
|
||||
URLScheme::File => {
|
||||
// TODO: Support client mode
|
||||
// NOTE: In standalone mode, we just assume path is well-defined and utilize
|
||||
log::info!("Fetching file plugin: {}", url);
|
||||
Ok(())
|
||||
}
|
||||
URLScheme::Unknown => {
|
||||
log::error!("Unknown URL scheme in plugin URL: {}", url);
|
||||
Err(PluginError::GeneralError(format!(
|
||||
"Unknown URL scheme in plugin URL: {}",
|
||||
url
|
||||
)))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn fetch_plugins(
|
||||
plugin_path: &Path,
|
||||
plugin_config: &mut RawPluginMap,
|
||||
list: Option<&HashSet<String>>,
|
||||
) -> Result<(), PluginError> {
|
||||
log::info!("Preparing to download {} plugins", plugin_config.len());
|
||||
for (name, data) in plugin_config.iter_mut() {
|
||||
match list {
|
||||
Some(list) if !list.contains(name) => {
|
||||
log::debug!("Plugin {} not in list, skipping", name);
|
||||
continue;
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
log::debug!("Downloading plugin {}", name);
|
||||
let (source, config) =
|
||||
data.iter_mut()
|
||||
.next()
|
||||
.ok_or_else(|| PluginError::PluginUnavailable {
|
||||
name: name.clone(),
|
||||
reason: "no source configured".to_string(),
|
||||
})?;
|
||||
let fetch_argument = FetchArgument {
|
||||
checksum: config.checksum.clone(),
|
||||
shallow: config.shallow,
|
||||
};
|
||||
fetch::fetch_plugin(source, name, plugin_path, fetch_argument).await?;
|
||||
log::debug!("Storing plugin {} to {}", name, plugin_path.display());
|
||||
config.runtime.fspath = Some(plugin_path.join(name));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
@@ -1,137 +0,0 @@
|
||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
||||
use super::types::ChecksumType;
|
||||
use sha2::{Digest, Sha256, Sha512};
|
||||
|
||||
/// Detects the checksum type based on the length of the checksum string.
|
||||
///
|
||||
/// Returns `Ok(None)` for empty string (checksum disabled).
|
||||
/// Returns `Ok(Some(ChecksumType))` for valid SHA256 (64 chars) or SHA512 (128 chars).
|
||||
/// Returns `Err` for invalid lengths (32=MD5, 40=SHA1) or unsupported lengths.
|
||||
pub fn detect_checksum_type(checksum: &str) -> Result<Option<ChecksumType>, String> {
|
||||
match checksum.len() {
|
||||
0 => Ok(None),
|
||||
32 => Err(format!(
|
||||
"MD5 (first 8 chars: {}) is not supported for security reasons. Use SHA256 (64 chars) or SHA512 (128 chars)",
|
||||
&checksum[..8.min(checksum.len())]
|
||||
)),
|
||||
40 => Err(format!(
|
||||
"SHA1 (first 8 chars: {}) is deprecated and not supported. Use SHA256 (64 chars) or SHA512 (128 chars)",
|
||||
&checksum[..8.min(checksum.len())]
|
||||
)),
|
||||
64 => Ok(Some(ChecksumType::Sha256)),
|
||||
128 => Ok(Some(ChecksumType::Sha512)),
|
||||
_ => Err(format!(
|
||||
"Invalid checksum length: {}. Expected 0 (disabled), 64 (SHA256), or 128 (SHA512). First 8 chars: {}",
|
||||
checksum.len(),
|
||||
&checksum[..8.min(checksum.len())]
|
||||
)),
|
||||
}
|
||||
}
|
||||
|
||||
/// Verifies that the data matches the expected checksum.
|
||||
///
|
||||
/// Compares the computed hash of `data` against `expected` using the specified `checksum_type`.
|
||||
/// Returns `Ok(())` if the checksums match, `Err` otherwise.
|
||||
pub fn verify_checksum(
|
||||
data: &[u8],
|
||||
expected: &str,
|
||||
checksum_type: ChecksumType,
|
||||
) -> Result<(), String> {
|
||||
let actual = match checksum_type {
|
||||
ChecksumType::Sha256 => {
|
||||
let mut hasher = Sha256::new();
|
||||
hasher.update(data);
|
||||
format!("{:x}", hasher.finalize())
|
||||
}
|
||||
ChecksumType::Sha512 => {
|
||||
let mut hasher = Sha512::new();
|
||||
hasher.update(data);
|
||||
format!("{:x}", hasher.finalize())
|
||||
}
|
||||
};
|
||||
|
||||
if actual.eq_ignore_ascii_case(expected) {
|
||||
Ok(())
|
||||
} else {
|
||||
Err(format!(
|
||||
"Checksum mismatch: expected {}..., got {}...",
|
||||
&expected[..8.min(expected.len())],
|
||||
&actual[..8.min(actual.len())]
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_detect_checksum_type_empty() {
|
||||
assert_eq!(detect_checksum_type("").unwrap(), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_detect_checksum_type_md5_rejected() {
|
||||
let result = detect_checksum_type("a".repeat(32).as_str());
|
||||
assert!(result.is_err());
|
||||
assert!(result.unwrap_err().contains("MD5"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_detect_checksum_type_sha1_rejected() {
|
||||
let result = detect_checksum_type("a".repeat(40).as_str());
|
||||
assert!(result.is_err());
|
||||
assert!(result.unwrap_err().contains("SHA1"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_detect_checksum_type_sha256() {
|
||||
assert_eq!(
|
||||
detect_checksum_type("a".repeat(64).as_str()).unwrap(),
|
||||
Some(ChecksumType::Sha256)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_detect_checksum_type_sha512() {
|
||||
assert_eq!(
|
||||
detect_checksum_type("a".repeat(128).as_str()).unwrap(),
|
||||
Some(ChecksumType::Sha512)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_detect_checksum_type_invalid_length() {
|
||||
let result = detect_checksum_type("a".repeat(50).as_str());
|
||||
assert!(result.is_err());
|
||||
assert!(result.unwrap_err().contains("Invalid checksum length"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_verify_checksum_sha256_valid() {
|
||||
let data = b"hello";
|
||||
let expected = "2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824";
|
||||
assert!(verify_checksum(data, expected, ChecksumType::Sha256).is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_verify_checksum_sha256_invalid() {
|
||||
let data = b"hello";
|
||||
let expected = "0000000000000000000000000000000000000000000000000000000000000000";
|
||||
assert!(verify_checksum(data, expected, ChecksumType::Sha256).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_verify_checksum_sha512_valid() {
|
||||
let data = b"hello";
|
||||
let expected = "9b71d224bd62f3785d96d46ad3ea3d73319bfbc2890caadae2dff72519673ca72323c3d99ba5c11d7c7acc6e14b8c5da0c4663475c2e5c3adef46f73bcdec043";
|
||||
assert!(verify_checksum(data, expected, ChecksumType::Sha512).is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_verify_checksum_sha512_invalid() {
|
||||
let data = b"hello";
|
||||
let expected = "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000";
|
||||
assert!(verify_checksum(data, expected, ChecksumType::Sha512).is_err());
|
||||
}
|
||||
}
|
||||
@@ -1,169 +0,0 @@
|
||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
||||
use crate::finalize::plugin::PluginError;
|
||||
use std::path::{Path, PathBuf};
|
||||
use tokio::task;
|
||||
|
||||
/// Compression type for tar archives.
|
||||
#[derive(Debug, Clone, Copy, PartialEq)]
|
||||
pub enum CompressionType {
|
||||
/// Gzip compressed (.tar.gz, .tgz)
|
||||
Gzip,
|
||||
/// Zstd compressed (.tar.zst, .tar.zstd, .tzst)
|
||||
Zstd,
|
||||
/// Uncompressed tar (.tar)
|
||||
None,
|
||||
}
|
||||
|
||||
impl CompressionType {
|
||||
/// Detects compression type from file path extension.
|
||||
///
|
||||
/// Supports: .tar.gz, .tgz (Gzip), .tar.zst, .tar.zstd, .tzst (Zstd), .tar (None).
|
||||
pub fn from_path(path: &std::path::Path) -> Result<Self, String> {
|
||||
let name = path
|
||||
.file_name()
|
||||
.and_then(|n| n.to_str())
|
||||
.unwrap_or("")
|
||||
.to_lowercase();
|
||||
|
||||
if name.ends_with(".tar.gz") || name.ends_with(".tgz") {
|
||||
Ok(CompressionType::Gzip)
|
||||
} else if name.ends_with(".tar.zst")
|
||||
|| name.ends_with(".tar.zstd")
|
||||
|| name.ends_with(".tzst")
|
||||
{
|
||||
Ok(CompressionType::Zstd)
|
||||
} else if name.ends_with(".tar") {
|
||||
Ok(CompressionType::None)
|
||||
} else {
|
||||
Err(format!(
|
||||
"Unsupported archive format: '{}'. Supported formats: .tar, .tar.gz, .tgz, .tar.zst, .tar.zstd, .tzst",
|
||||
name
|
||||
))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Extracts a tar archive to the destination directory.
|
||||
///
|
||||
/// Removes existing destination directory if present, creates a fresh directory,
|
||||
/// then unpacks the archive using the appropriate decompression based on `compression`.
|
||||
pub async fn extract_tar(
|
||||
archive: &Path,
|
||||
destdir: &PathBuf,
|
||||
compression: CompressionType,
|
||||
) -> Result<(), PluginError> {
|
||||
if destdir.exists() {
|
||||
tokio::fs::remove_dir_all(destdir).await.map_err(|e| {
|
||||
PluginError::GeneralError(format!("Failed to remove existing directory: {}", e))
|
||||
})?;
|
||||
}
|
||||
tokio::fs::create_dir_all(destdir)
|
||||
.await
|
||||
.map_err(|e| PluginError::GeneralError(format!("Failed to create directory: {}", e)))?;
|
||||
|
||||
let archive = archive.to_path_buf();
|
||||
let destdir = destdir.clone();
|
||||
|
||||
task::spawn_blocking(move || {
|
||||
let file = std::fs::File::open(&archive)
|
||||
.map_err(|e| PluginError::GeneralError(format!("Failed to open archive: {}", e)))?;
|
||||
|
||||
match compression {
|
||||
CompressionType::Gzip => {
|
||||
let decoder = flate2::read::GzDecoder::new(file);
|
||||
let mut archive = tar::Archive::new(decoder);
|
||||
archive.unpack(&destdir).map_err(|e| {
|
||||
PluginError::GeneralError(format!("Failed to unpack archive: {}", e))
|
||||
})?;
|
||||
}
|
||||
CompressionType::Zstd => {
|
||||
let decoder = zstd::stream::read::Decoder::new(file).map_err(|e| {
|
||||
PluginError::GeneralError(format!("Failed to create zstd decoder: {}", e))
|
||||
})?;
|
||||
let mut archive = tar::Archive::new(decoder);
|
||||
archive.unpack(&destdir).map_err(|e| {
|
||||
PluginError::GeneralError(format!("Failed to unpack archive: {}", e))
|
||||
})?;
|
||||
}
|
||||
CompressionType::None => {
|
||||
let mut archive = tar::Archive::new(file);
|
||||
archive.unpack(&destdir).map_err(|e| {
|
||||
PluginError::GeneralError(format!("Failed to unpack archive: {}", e))
|
||||
})?;
|
||||
}
|
||||
};
|
||||
|
||||
Ok::<(), PluginError>(())
|
||||
})
|
||||
.await
|
||||
.map_err(|e| PluginError::GeneralError(format!("Task join error: {}", e)))??;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::path::Path;
|
||||
|
||||
#[test]
|
||||
fn test_from_path_tar_gz() {
|
||||
assert_eq!(
|
||||
CompressionType::from_path(Path::new("archive.tar.gz")).unwrap(),
|
||||
CompressionType::Gzip
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_from_path_tgz() {
|
||||
assert_eq!(
|
||||
CompressionType::from_path(Path::new("archive.tgz")).unwrap(),
|
||||
CompressionType::Gzip
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_from_path_tar_zst() {
|
||||
assert_eq!(
|
||||
CompressionType::from_path(Path::new("archive.tar.zst")).unwrap(),
|
||||
CompressionType::Zstd
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_from_path_tar_zstd() {
|
||||
assert_eq!(
|
||||
CompressionType::from_path(Path::new("archive.tar.zstd")).unwrap(),
|
||||
CompressionType::Zstd
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_from_path_tzst() {
|
||||
assert_eq!(
|
||||
CompressionType::from_path(Path::new("archive.tzst")).unwrap(),
|
||||
CompressionType::Zstd
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_from_path_tar() {
|
||||
assert_eq!(
|
||||
CompressionType::from_path(Path::new("archive.tar")).unwrap(),
|
||||
CompressionType::None
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_from_path_unsupported() {
|
||||
assert!(CompressionType::from_path(Path::new("archive.zip")).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_from_path_case_insensitive() {
|
||||
assert_eq!(
|
||||
CompressionType::from_path(Path::new("archive.TAR.GZ")).unwrap(),
|
||||
CompressionType::Gzip
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -1,185 +0,0 @@
|
||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
||||
use crate::finalize::plugin::PluginError;
|
||||
use std::path::Path;
|
||||
use tokio::process::Command;
|
||||
|
||||
/// Version specification for a git repository.
|
||||
#[derive(Debug, PartialEq)]
|
||||
pub enum GitVersion {
|
||||
/// No version specified (invalid for fetch operations)
|
||||
None,
|
||||
/// Single version string (tag or commit hash)
|
||||
Single(String),
|
||||
/// Mixed: try tag first, fallback to commit hash
|
||||
Mixed(String, String),
|
||||
}
|
||||
|
||||
/// Parses a git URL with optional version specification.
|
||||
///
|
||||
/// Format: `url` (no version), `url@tag` (single version), `url@tag:commit` (mixed version).
|
||||
/// Returns (base_url, GitVersion).
|
||||
pub fn parse_git_url(url: &str) -> (String, GitVersion) {
|
||||
if let Some((base, version)) = url.rsplit_once('@') {
|
||||
if version.contains(":") {
|
||||
let (tag, commit) = version.split_once(':').unwrap();
|
||||
return (
|
||||
base.to_string(),
|
||||
GitVersion::Mixed(tag.to_string(), commit.to_string()),
|
||||
);
|
||||
}
|
||||
return (base.to_string(), GitVersion::Single(version.to_string()));
|
||||
}
|
||||
(url.to_string(), GitVersion::None)
|
||||
}
|
||||
|
||||
/// Clones a git repository and checks out the specified version.
|
||||
///
|
||||
/// Removes existing destination directory, performs shallow or full clone,
|
||||
/// then checks out the given version (tag or commit).
|
||||
pub async fn git_clone_and_checkout(
|
||||
baseurl: &str,
|
||||
version: &str,
|
||||
dest: &Path,
|
||||
name: &str,
|
||||
shallow: bool,
|
||||
) -> Result<(), PluginError> {
|
||||
if dest.exists() {
|
||||
tokio::fs::remove_dir_all(dest)
|
||||
.await
|
||||
.map_err(|e| PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: baseurl.to_string(),
|
||||
reason: format!("Failed to remove existing directory: {}", e),
|
||||
})?;
|
||||
}
|
||||
let mut cmd = Command::new("git");
|
||||
cmd.arg("clone");
|
||||
if shallow {
|
||||
cmd.arg("--depth").arg("1");
|
||||
}
|
||||
cmd.arg(baseurl).arg(dest);
|
||||
let output = cmd.output().await.map_err(|e| PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: baseurl.to_string(),
|
||||
reason: e.to_string(),
|
||||
})?;
|
||||
if !output.status.success() {
|
||||
return Err(PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: baseurl.to_string(),
|
||||
reason: String::from_utf8_lossy(&output.stderr).to_string(),
|
||||
});
|
||||
}
|
||||
let mut cmd = Command::new("git");
|
||||
cmd.arg("checkout").arg(version).current_dir(dest);
|
||||
let output = cmd.output().await.map_err(|e| PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: baseurl.to_string(),
|
||||
reason: e.to_string(),
|
||||
})?;
|
||||
if !output.status.success() {
|
||||
return Err(PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: baseurl.to_string(),
|
||||
reason: String::from_utf8_lossy(&output.stderr).to_string(),
|
||||
});
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Fetches a plugin from a git repository.
|
||||
///
|
||||
/// Parses the URL for version info, then clones and checks out the appropriate ref.
|
||||
/// For Mixed version, tries the tag first and falls back to the commit hash.
|
||||
pub async fn fetch_git_plugin(
|
||||
path: &str,
|
||||
name: &str,
|
||||
basedir: &Path,
|
||||
shallow: Option<bool>,
|
||||
) -> Result<(), PluginError> {
|
||||
let (baseurl, version) = parse_git_url(path);
|
||||
match version {
|
||||
GitVersion::None => {
|
||||
log::error!("Version should be specified for git plugin URL: {}", path);
|
||||
Err(PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: path.to_string(),
|
||||
reason: "Version should be specified as either a tag or a commit hash.".to_string(),
|
||||
})
|
||||
}
|
||||
GitVersion::Single(version) => {
|
||||
log::debug!("Fetching git plugin: {}@{}", name, version);
|
||||
let shallow_flag = shallow.unwrap_or(false);
|
||||
git_clone_and_checkout(
|
||||
&baseurl,
|
||||
&version.to_string(),
|
||||
&basedir.join(name),
|
||||
name,
|
||||
shallow_flag,
|
||||
)
|
||||
.await
|
||||
}
|
||||
GitVersion::Mixed(tag, commit) => {
|
||||
log::debug!("Fetching git plugin: {}@{}/{}", name, tag, commit);
|
||||
let shallow_flag = shallow.unwrap_or(false);
|
||||
let result =
|
||||
git_clone_and_checkout(&baseurl, &tag, &basedir.join(name), name, shallow_flag)
|
||||
.await;
|
||||
if result.is_err() {
|
||||
log::warn!(
|
||||
"Failed to clone with tag, trying commit hash. URL: {}",
|
||||
path
|
||||
);
|
||||
return git_clone_and_checkout(&baseurl, &commit, &basedir.join(name), name, false)
|
||||
.await;
|
||||
}
|
||||
result
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_parse_git_url_no_version() {
|
||||
let (url, version) = parse_git_url("https://github.com/user/repo");
|
||||
assert_eq!(url, "https://github.com/user/repo");
|
||||
assert_eq!(version, GitVersion::None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_parse_git_url_single_tag() {
|
||||
let (url, version) = parse_git_url("https://github.com/user/repo@v1.0.0");
|
||||
assert_eq!(url, "https://github.com/user/repo");
|
||||
assert_eq!(version, GitVersion::Single("v1.0.0".to_string()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_parse_git_url_mixed_version() {
|
||||
let (url, version) = parse_git_url("https://github.com/user/repo@main:abc123");
|
||||
assert_eq!(url, "https://github.com/user/repo");
|
||||
assert_eq!(
|
||||
version,
|
||||
GitVersion::Mixed("main".to_string(), "abc123".to_string())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_parse_git_url_complex_url() {
|
||||
let (url, version) = parse_git_url("git@github.com:user/repo.git@tag:commit");
|
||||
assert_eq!(url, "git@github.com:user/repo.git");
|
||||
assert_eq!(
|
||||
version,
|
||||
GitVersion::Mixed("tag".to_string(), "commit".to_string())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_parse_git_url_single_with_at_in_path() {
|
||||
let (url, version) = parse_git_url("https://github.com/user/repo@my-repo@v1.0");
|
||||
assert_eq!(url, "https://github.com/user/repo@my-repo");
|
||||
assert_eq!(version, GitVersion::Single("v1.0".to_string()));
|
||||
}
|
||||
}
|
||||
@@ -1,110 +0,0 @@
|
||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
||||
use super::checksum::{detect_checksum_type, verify_checksum};
|
||||
use super::extract::{CompressionType, extract_tar};
|
||||
use crate::finalize::plugin::PluginError;
|
||||
use std::path::Path;
|
||||
|
||||
pub async fn download_to_file(url: &str, dest: &Path, name: &str) -> Result<(), PluginError> {
|
||||
let client = reqwest::Client::builder()
|
||||
.timeout(std::time::Duration::from_secs(300))
|
||||
.build()
|
||||
.map_err(|e| PluginError::GeneralError(format!("Failed to build HTTP client: {}", e)))?;
|
||||
|
||||
let response = client
|
||||
.get(url)
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: url.to_string(),
|
||||
reason: format!("HTTP request failed: {}", e),
|
||||
})?;
|
||||
|
||||
let status = response.status();
|
||||
if !status.is_success() {
|
||||
return Err(PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: url.to_string(),
|
||||
reason: format!("HTTP error: {}", status),
|
||||
});
|
||||
}
|
||||
|
||||
let bytes = response
|
||||
.bytes()
|
||||
.await
|
||||
.map_err(|e| PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: url.to_string(),
|
||||
reason: format!("Failed to read response body: {}", e),
|
||||
})?;
|
||||
|
||||
tokio::fs::write(dest, &bytes)
|
||||
.await
|
||||
.map_err(|e| PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: url.to_string(),
|
||||
reason: format!("Failed to write file: {}", e),
|
||||
})?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn extract_archive_extension(url: &str) -> &str {
|
||||
let path_part = url
|
||||
.split('?')
|
||||
.next()
|
||||
.unwrap_or(url)
|
||||
.split('#')
|
||||
.next()
|
||||
.unwrap_or(url);
|
||||
let file_part = path_part.rsplit('/').next().unwrap_or("");
|
||||
let lower = file_part.to_lowercase();
|
||||
if lower.ends_with(".tar.gz") || lower.ends_with(".tgz") {
|
||||
".tar.gz"
|
||||
} else if lower.ends_with(".tar.zst")
|
||||
|| lower.ends_with(".tar.zstd")
|
||||
|| lower.ends_with(".tzst")
|
||||
{
|
||||
".tar.zst"
|
||||
} else if lower.ends_with(".tar") {
|
||||
".tar"
|
||||
} else {
|
||||
".unknown"
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn fetch_http_plugin(
|
||||
url: &str,
|
||||
name: &str,
|
||||
basedir: &Path,
|
||||
checksum_str: Option<&str>,
|
||||
) -> Result<(), PluginError> {
|
||||
let temp_dir = std::env::temp_dir().join("workshop-plugins");
|
||||
tokio::fs::create_dir_all(&temp_dir).await.ok();
|
||||
let ext = extract_archive_extension(url);
|
||||
let temp_file = temp_dir.join(format!("{}-{}{}", name, uuid::Uuid::new_v4(), ext));
|
||||
|
||||
download_to_file(url, &temp_file, name).await?;
|
||||
|
||||
if let Some(checksum) = checksum_str
|
||||
&& !checksum.is_empty()
|
||||
{
|
||||
let bytes = tokio::fs::read(&temp_file).await.map_err(|e| {
|
||||
PluginError::GeneralError(format!("Failed to read downloaded file: {}", e))
|
||||
})?;
|
||||
|
||||
let checksum_type = detect_checksum_type(checksum).map_err(PluginError::GeneralError)?;
|
||||
|
||||
if let Some(ct) = checksum_type {
|
||||
verify_checksum(&bytes, checksum, ct).map_err(PluginError::GeneralError)?;
|
||||
}
|
||||
}
|
||||
|
||||
let compression = CompressionType::from_path(&temp_file).map_err(PluginError::GeneralError)?;
|
||||
let destdir = basedir.join(name);
|
||||
extract_tar(&temp_file, &destdir, compression).await?;
|
||||
|
||||
tokio::fs::remove_file(&temp_file).await.ok();
|
||||
|
||||
Ok(())
|
||||
}
|
||||
@@ -1,84 +0,0 @@
|
||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
/// Supported checksum types for verifying downloaded artifacts.
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub enum ChecksumType {
|
||||
/// SHA-256 hash (64 hex characters)
|
||||
Sha256,
|
||||
/// SHA-512 hash (128 hex characters)
|
||||
Sha512,
|
||||
}
|
||||
|
||||
/// Arguments for fetching a plugin artifact.
|
||||
///
|
||||
/// Contains optional checksum verification and shallow clone settings.
|
||||
#[derive(Debug, Deserialize, Serialize, Clone, Default)]
|
||||
pub struct FetchArgument {
|
||||
/// SHA256 or SHA512 hex string for integrity verification.
|
||||
#[serde(default)]
|
||||
pub checksum: Option<String>,
|
||||
/// Perform a shallow clone (single revision only).
|
||||
#[serde(default)]
|
||||
pub shallow: Option<bool>,
|
||||
}
|
||||
|
||||
impl FetchArgument {
|
||||
/// Returns the checksum if present and non-empty.
|
||||
pub fn get_checksum(&self) -> Option<&str> {
|
||||
self.checksum.as_deref().filter(|s| !s.is_empty())
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_get_checksum_some_non_empty() {
|
||||
let arg = FetchArgument {
|
||||
checksum: Some("abc123".to_string()),
|
||||
shallow: None,
|
||||
};
|
||||
assert_eq!(arg.get_checksum(), Some("abc123"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_get_checksum_empty_string() {
|
||||
let arg = FetchArgument {
|
||||
checksum: Some("".to_string()),
|
||||
shallow: None,
|
||||
};
|
||||
assert_eq!(arg.get_checksum(), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_get_checksum_none() {
|
||||
let arg = FetchArgument {
|
||||
checksum: None,
|
||||
shallow: None,
|
||||
};
|
||||
assert_eq!(arg.get_checksum(), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_serde_roundtrip() {
|
||||
let arg = FetchArgument {
|
||||
checksum: Some("deadbeef".to_string()),
|
||||
shallow: Some(true),
|
||||
};
|
||||
let json = serde_json::to_string(&arg).unwrap();
|
||||
let parsed: FetchArgument = serde_json::from_str(&json).unwrap();
|
||||
assert_eq!(parsed.checksum, arg.checksum);
|
||||
assert_eq!(parsed.shallow, arg.shallow);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_serde_roundtrip_empty() {
|
||||
let arg = FetchArgument::default();
|
||||
let json = serde_json::to_string(&arg).unwrap();
|
||||
let parsed: FetchArgument = serde_json::from_str(&json).unwrap();
|
||||
assert_eq!(parsed.checksum, arg.checksum);
|
||||
assert_eq!(parsed.shallow, arg.shallow);
|
||||
}
|
||||
}
|
||||
@@ -1,4 +1,5 @@
|
||||
use crate::finalize::plugin::AsyncPluginFn;
|
||||
use crate::notify::types::NotificationRenderer;
|
||||
pub mod dummy;
|
||||
mod insitenotify;
|
||||
mod mail;
|
||||
@@ -7,6 +8,7 @@ mod webhook;
|
||||
pub struct InternalPlugin {
|
||||
pub name: &'static str,
|
||||
pub entry: Box<dyn AsyncPluginFn>,
|
||||
pub renderer: NotificationRenderer,
|
||||
}
|
||||
|
||||
pub fn get_internal_plugin() -> Vec<InternalPlugin> {
|
||||
@@ -14,18 +16,32 @@ pub fn get_internal_plugin() -> Vec<InternalPlugin> {
|
||||
InternalPlugin {
|
||||
name: "in-site-notify",
|
||||
entry: Box::new(insitenotify::InsiteNotify),
|
||||
renderer: NotificationRenderer::Plugin,
|
||||
},
|
||||
InternalPlugin {
|
||||
name: "mail",
|
||||
entry: Box::new(mail::Mail),
|
||||
renderer: NotificationRenderer::Server,
|
||||
},
|
||||
InternalPlugin {
|
||||
name: "satori",
|
||||
entry: Box::new(satori::Satori),
|
||||
renderer: NotificationRenderer::Server,
|
||||
},
|
||||
InternalPlugin {
|
||||
name: "webhook",
|
||||
entry: Box::new(webhook::Webhook),
|
||||
renderer: NotificationRenderer::Server,
|
||||
},
|
||||
]
|
||||
}
|
||||
|
||||
/// Returns dummy only (for `--debug notify.dummy` mode).
|
||||
/// All real adapters and in-site-notify are excluded.
|
||||
pub fn get_debug_plugin() -> Vec<InternalPlugin> {
|
||||
vec![InternalPlugin {
|
||||
name: "dummy",
|
||||
entry: Box::new(dummy::Dummy),
|
||||
renderer: NotificationRenderer::Plugin,
|
||||
}]
|
||||
}
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user