Compare commits

...

8 Commits

Author SHA1 Message Date
Catty Steve df5026cdbe refactor(notify): extract handler, rule, util
CI / ${{ matrix.crate }} (workshop-engine) (push) Failing after 3m54s
CI / ${{ matrix.crate }} (workshop-baker) (push) Failing after 27m50s
Split the monolithic notify module into three submodules:
- `handler`: async bucket dispatch with cooldown, retry, escalation
- `rule`: rule parsing from YAML config, template merging, overrides
- `util`: validation, event timing, helper functions
  `NotificationTemplateDef` replaces `AddAssign` with `apply()` for
  non-destructive merging (self wins, other fills gaps).
  `insitenotify` logs instead of panicking.
2026-05-22 00:16:28 +08:00
Catty Steve d1ad08ef3a feat: add workshop-baker-params for config layering
Implement a priority-based merge system for pipeline configuration.
Parameters carry their source (Base, Courier, Bakerd) and only
higher-priority writers override. Migrate notification, prebake, bake,
and finalize config to use ParamVal wrappers. Remove hardcoded
constants and the InteractiveServer component.

BREAKING CHANGE: Remove NotAvailable variant from MethodStatus enum
2026-05-21 20:04:47 +08:00
Catty Steve cf2968e720 refactor: update AGENTS.md and remove osbolete modules
Archive 7 deprecated crates. Add Resource model documentation.
Remove internal fetch module from workshop-baker.
2026-05-20 16:30:50 +08:00
Catty Steve 28abc5d207 refactor: replace internal fetch utilities with ResourceRegistry
Remove the monolithic utils::fetch module (HTTP download, git clone,
checksum, extraction) and utils::fetch_plugin module from the baker.
Introduce a ResourceRegistry type to centrally track pipeline
resources by name and path.

The actual fetching logic is now delegated to the workshop-getterurl
crate, which gains optional md5 and sha1 hash support behind default
features. Also flatten the builtin module in workshop-getterurl.
2026-05-19 23:22:59 +08:00
Catty Steve 6e7b96cd66 refactor: remove unused crates, refactor fetch
- Move fetch_plugin from finalize::plugin::fetch to utils::fetch_plugin
- Delete unused crates: workshop-cert, workshop-deviceid, workshop-vault
- Remove parser benchmarks and empty daemon/socket modules
- Add new workshop-getterurl crate for resource fetching
2026-05-19 21:53:41 +08:00
Catty Steve 3e80825d58 refactor(plugin)!: remove dedicated plugin fetch subsystem
Drop the entire plugin fetch module (HTTP, git, checksums, extraction).
Simplify mail notification plugin to a bare stub.
Remove outdated metadata tests.

BREAKING CHANGE: Not a buildable version!
2026-05-19 16:24:19 +08:00
Catty Steve c97eafab29 refactor(baker): Restructure CLI, drop legacy tests
- Remove Python-based test framework, Dockerfiles, and runner scripts
- Add bare subcommand for standalone pipeline stage execution
- Implement unified worker mode and error module
- Refactor NotificationConfig to support YAML string keys
- Add dummy internal plugin and clean up mail template
2026-05-19 16:19:28 +08:00
Catty Steve f737b6a24d feat(baker): add axum HTTP server and refactor plugin fetching
Refactor plugin imports to use workshop_engine types, consolidate
resource fetching into utils/fetch module with checksum verification,
and add interactive notification server using axum.
2026-05-18 10:01:55 +08:00
156 changed files with 5500 additions and 14760 deletions
+1 -1
View File
@@ -9,7 +9,7 @@ archived
*.bak *.bak
*.clean *.clean
session* session*
llm.env .secret
# System test artifacts # System test artifacts
tests/results/ tests/results/
+40 -56
View File
@@ -1,83 +1,72 @@
# PROJECT KNOWLEDGE BASE # PROJECT KNOWLEDGE BASE
**Generated:** 2026-04-22 **Generated:** 2026-05-19
**Commit:** 7b3b71a **Commit:** 28abc5d
**Branch:** master **Branch:** master
## OVERVIEW ## OVERVIEW
HoneyBiscuitWorkshop is a Rust-based CI/CD system ("蜜饼工坊") with LLM-powered pipeline auto-configuration. 9 independent Cargo crates, no workspace. HoneyBiscuitWorkshop ("蜜饼工坊") — Rust CI/CD system with decorator-driven shell pipeline DSL and LLM-powered configuration. 3 independent Cargo crates, no workspace.
## STRUCTURE ## STRUCTURE
``` ```
./ ./
├── workshop-baker/ # Main orchestrator (CLI + daemon) ├── workshop-baker/ # Main orchestrator (CLI + pipeline stages)
├── workshop-agent/ # HTTP agent server ├── workshop-engine/ # Execution engine (extracted from baker)
├── workshop-pipeline/ # Config library (prebake/finalize) ├── workshop-getterurl/ # URL fetching (go-getter style)
├── workshop-llm-detector/ # LLM repo analyzer
├── workshop-vault/ # Secret management client
├── workshop-cert/ # TLS certificate generator
├── workshop-deviceid/ # Device ID library
├── workshop-builder-native/ # Stub (unused)
├── workshop-helper-mac/ # MAC address utility
├── docs/ # mdBook documentation (zh-CN) ├── docs/ # mdBook documentation (zh-CN)
├── playground/ # Experimentation ├── templates/ # Pipeline templates (bake.sh, finalize.yml, prebake.yml)
└── third_party/ # Empty └── archived/ # Deprecated crates (7 moved here)
``` ```
## WHERE TO LOOK ## WHERE TO LOOK
| Task | Location | Notes | | Task | Location | Notes |
|------|----------|-------| |------|----------|-------|
| Baker (main) | `workshop-baker/src/{bake,prebake,engine}/` | Core pipeline logic | | Baker (CLI) | `workshop-baker/src/{lib,main}.rs` | Clap CLI + tokio entry |
| Agent server | `workshop-agent/src/` | Actix-web HTTP + TLS | | Pipeline stages | `workshop-baker/src/{bake,prebake,finalize}/` | Stage orchestration |
| Pipeline config | `workshop-pipeline/src/config/` | prebake/finalize YAML | | Notifications | `workshop-baker/src/notify/` | Plugin-based event notifications |
| LLM integration | `workshop-llm-detector/src/llm/` | Ollama/OpenAI clients | || Engine (execution) | `workshop-engine/src/` | Process mgmt, PM detection |
| Secrets | `workshop-vault/src/` | Vault client wrapper | | URL fetching | `workshop-getterurl/src/` | Git/HTTP/File resource getter |
## CODE MAP ## CODE MAP
| Symbol | Type | Location | Role | | Symbol | Type | Location | Role |
|--------|------|----------|------| |--------|------|----------|------|
| PipelineConfig | struct | workshop-pipeline/src/lib.rs:27 | Full pipeline config | | Cli | struct | workshop-baker/src/lib.rs:21 | CLI argument struct |
| CICDLLMHelper | struct | workshop-llm-detector/src/lib.rs:16 | LLM analysis entry | | ExecutionContext | struct | workshop-engine/src/types.rs:10 | Pipeline context |
| Engine | struct | workshop-baker/src/engine.rs | Build executor | | Engine | struct | workshop-engine/src/types.rs:99 | Build executor (placeholder) |
| VaultClient | struct | workshop-vault/src/client.rs | Secret ops | | PrebakeConfig | struct | workshop-baker/src/prebake/config.rs | Prebake YAML schema |
| FinalizeConfig | struct | workshop-baker/src/finalize/config.rs:21 | Finalize YAML schema |
| GetterUrl | struct | workshop-getterurl/src/lib.rs:84 | Parsed resource URL |
| Getter | trait | workshop-getterurl/src/getter.rs:18 | Resource fetch trait |
## CONVENTIONS ## CONVENTIONS
### Rust ### Rust
- **Edition**: 2024 (non-standard, requires Rust 1.85+) - **Edition**: 2024 (requires Rust 1.85+), deprecates `mod.rs`
- Deprecate `mod.rs`
- **Naming**: `snake_case` files/modules, `PascalCase` types, `SCREAMING_SNAKE_CASE` consts - **Naming**: `snake_case` files/modules, `PascalCase` types, `SCREAMING_SNAKE_CASE` consts
- **Error handling**: `thiserror` + `anyhow` combo - **Error handling**: `thiserror` + `anyhow` combo; `HasExitCode` trait for exit codes
- **Async**: `#[tokio::main]` + `tokio` with "full" features - **Async**: `#[tokio::main]` + `tokio` with "full" features
- **Imports**: `std``external_crate``crate::module` - **Imports**: `std``external_crate``crate::module`
- **Dual-mode**: CLI commands (prebake/bake/finalize) + daemon mode (unimplemented)
### Testing ### Testing
- **Rust**: `#[test]` inline, `#[tokio::test]` async, `tests/*.rs` integration - **Rust**: `#[test]` inline, `#[tokio::test]` async, `tests/` integration
- **Python**: `pytest` with `tests/integration/test_*.py` (Docker-based) - **Benchmarks**: Criterion (configured in Cargo.toml)
- **Benchmarks**: Criterion with `harness = false`
### Build Pipeline ### CI
- **prebake.yml**: Environment setup (docker/firecracker/baremetal) - **GitHub Actions**: rust-toolchain (stable), clippy + rustfmt, `cargo test --all-features`
- **bake.sh**: Build execution with `# @pipeline` decorators - **Matrix**: workshop-baker, workshop-engine
- **finalize.yml**: Packaging, deployment, notifications
- **Annotations**: `@timeout()`, `@retry()`, `@parallel`, `@fallible`
## ANTI-PATTERNS (THIS PROJECT) ## ANTI-PATTERNS (THIS PROJECT)
1. **Hardcoded user** — "vulcan" username hardcoded throughout prebake
1. **Python in Rust project** — workshop-baker has Python tests (pytest.ini, test_prebake.py) 2. **Unsafe libc** — workshop-baker/src/prebake/security.rs has 5+ unsafe blocks
2. **temp/ directory** — Non-standard in Rust projects 3. **Rust 2024 edition** — May not work with stable toolchains
3. **Certificates in source** — artifact workshop-cert/certs/ not excluded 4. **Empty daemon**`daemon/` dir is empty, Daemon command is `unimplemented!()`
4. **Deprecated projects** — workshop-executor.old, workshop-monitor.old still present
5. **Hardcoded mirrors** — Tsinghua University mirrors in configs
6. **Rust 2024 edition** — May not work with stable toolchains
## UNIQUE STYLES ## UNIQUE STYLES
- **Pipeline DSL**: Shell scripts with `# @decorator` annotations - **Pipeline DSL**: Shell scripts with `# @decorator` annotations
- **Dual-mode baker**: CLI commands + daemon mode - **Notification system**: Plugin-based with batching, retry, priority groups
- **LLM cookbook system**: YAML cookbooks for language-specific builds - **LLM cookbook system**: YAML cookbooks for language-specific builds (archived)
- **Chinese docs** — Documentation primarily in zh-CN - **Chinese docs** — Documentation primarily in zh-CN
- **User "vulcan"** — Custom build user (not root/runner)
## COMMANDS ## COMMANDS
```bash ```bash
@@ -86,23 +75,18 @@ cargo build --release -p workshop-baker
# Test # Test
cargo test -p workshop-baker cargo test -p workshop-baker
pytest tests/integration/test_prebake.py -v cargo test -p workshop-engine
# Lint # Lint
cargo clippy -- -D warnings cargo clippy -- -D warnings
cargo fmt --all cargo fmt --all
# Run baker # Run
cargo run --bin workshop-baker -- prebake config.yml cargo run --bin workshop-baker -- --pipeline demo --build-id 1 --prebake prebake.yml --bake bake.sh --finalize finalize.yml
cargo run --bin workshop-baker -- bake script.sh
# Run agent
cargo run --bin workshop-agent
``` ```
## NOTES ## NOTES
- **Docker required** for integration tests - **Docker required** for integration tests
- **workshop-builder-native** is a stub (3-line Hello World) - **No root workspace** — each crate independent, build separately
- No root workspace Cargo.toml — each crate is independent - **7 archived crates** — workshop-agent, workshop-cert, workshop-deviceid, workshop-helper-mac, workshop-llm-detector, workshop-pipeline, workshop-vault
- `AGENTS.md.old` at root — legacy documentation - **Python tests removed** — pytest.ini exists but test files no longer in workspace
+1
View File
@@ -35,3 +35,4 @@
- [自定义 Decorator](zh-CN/vol3_dev/custom_decorator.md) - [自定义 Decorator](zh-CN/vol3_dev/custom_decorator.md)
- [模板变量](zh-CN/vol3_dev/template_variables.md) - [模板变量](zh-CN/vol3_dev/template_variables.md)
- [邮件模板](zh-CN/vol3_dev/mail_template.md) - [邮件模板](zh-CN/vol3_dev/mail_template.md)
- [Resource 模型](zh-CN/vol3_dev/resource_model.md)
+88
View File
@@ -0,0 +1,88 @@
# Resource 模型
## 核心思想
整个系统只关心两件事:东西从哪来(fetch),东西到哪去(publish)。
fetch 方向:URL → 本地路径 → 使用。
publish 方向:本地路径 → 变换(插件)→ URL。
两个方向共用同一套寻址方式,但 Resource 中间态只存在于 fetch 方向。
## URL 格式
```
getter::url?param1=value1&param2=value2
```
getter 显式声明意图。不是"我猜这是个 git 仓库",而是"使用者告诉我这是什么"。
只有三个 getter
- `git` — 克隆仓库,必要时 checkout 指定版本
- `http` / `https` — 下载文件,必要时校验 checksum
- `file` — 复制本地文件或目录
版本锁定不用 `@v1.0``@` 在 URL 里是认证分隔符),用 `?ref=v1.0`
没有 `github.com/user/repo` 看起来应该 clone 它这种黑魔法。
没有 `.tar.gz` 看起来应该解压它的惊喜。
URL 是 URL,行为是行为,两者用 getter:: 显式绑定。
## Resource 不是网络地址
流水线阶段(prebake / finalize / notify)只接触 Resource:一个名字 + 一个本地路径,已就绪。它们不经由网络、不解析 URL、不碰 getter。
所有 fetch 行为在阶段启动前完成:
- worker 模式:bakerd fetch → 填入 ResourceRegistry → 启动 baker
- standalone 模式:baker 自己 fetch → 填入 ResourceRegistry → 执行阶段
- bare 模式:不 fetch,传入的必须是 Resource
ResourceRegistry 是名字到路径的映射。填进去就只读了。
## getterurl 是独立的 crate
URL 格式解析 + 基本 fetch 放在 workshop-getterurl,和 baker 本身没有耦合。任何人都可以用。
crate 分两层:
```
结构层(parser):
"git::https://host/repo.git?ref=v1.0" → { getter: "git", url: Url, params: Params }
获取层(builtin getters):
getter.fetch(url, params, dst) → local_path
git: git clone + optional checkout
http: HTTP GET + optional checksum
file: std::fs::copy
```
获取层是字面意义的——不解压、不推断、不猜意图。
## Checksum
checksum 作为 URL query 参数传递。三态语义:
- `?checksum=sha256:abc` — 必须匹配
- `?checksum=` — 显式禁用校验
- 无 checksum — 如果服务端返回 Content-Digest 头,校验它
getter 不主动校验,它只执行 URL 里指定的操作。
## Artifact 不经过 Resource
artifact 是 bake 的产物,自产生就开始被使用,不存在中间态。它的路径就是 bake 产出的本地路径,不走 ResourceRegistry。
artifact 的 publish 方向由插件定义行为——可能上传、可能扫描、可能只记录。插件定义最终的行为,只要是自洽的。
## 三种运行模式
```
fetch 执行者 fetch 在阶段内
daemon 模式 bakerd 否
standalone baker 自身 否(阶段启动前已完成)
bare 模式 无 否(必须传入 Resource
```
bare 模式下传入的必须是已经就绪的 Resource。外部 URL 是 daemon 或 standalone 的事,bare 不负责。
-217
View File
@@ -1,217 +0,0 @@
#!/bin/bash
set -euo pipefail
# HoneyBiscuitWorkshop System Test Runner
#
# Usage:
# ./runner.sh # baker tests (default)
# ./runner.sh -c upm:rust # Rust UPM tests
# ./runner.sh -c pm:pacman # pacman PM engine tests
# ./runner.sh -c pm:apt -k test_mirror # apt mirror test only
# ./runner.sh -c upm:rust --network # include network tests
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
CONTAINER_NAME="hbw-system-test-container"
STAGING_DIR="$SCRIPT_DIR/.staging"
# ── defaults ────────────────────────────────────────────────────────
MODE="auto"
CATEGORY="baker" # baker | upm:<name> | pm:<name>
TEST_DIR=""
TEST_IMAGE=""
PYTEST_ARGS=()
NETWORK=0
BAKER_LOG=1
# ── parse args ──────────────────────────────────────────────────────
show_help() {
cat << 'HELP'
Usage: ./runner.sh [OPTIONS]
Categories (-c):
baker Baker system tests [default]
upm:rust Rust UPM tests
pm:pacman Pacman PM engine tests
pm:apt APT PM engine tests
pm:dnf DNF PM engine tests
pm:apk APK PM engine tests
Options:
-c, --category CAT Test category (see above)
--network Allow network-dependent tests (toolchain/crate downloads)
-k FILTER pytest -k expression
--timeout SECONDS Per-test timeout (default 120)
--log / --no-log Stream baker output to terminal [default: --log]
-h, --help This message
HELP
exit 0
}
while [[ $# -gt 0 ]]; do
case "$1" in
-c|--category)
CATEGORY="$2"; shift 2 ;;
--network)
NETWORK=1; shift ;;
-k)
PYTEST_ARGS+=(-k "$2"); shift 2 ;;
--timeout)
PYTEST_ARGS+=(--timeout "$2"); shift 2 ;;
--log) BAKER_LOG=1; shift ;;
--no-log) BAKER_LOG=0; shift ;;
manual) MODE="manual"; shift ;;
auto) MODE="auto"; shift ;;
-h|--help)
show_help ;;
--)
shift; PYTEST_ARGS+=("$@"); break ;;
*)
PYTEST_ARGS+=("$1"); shift ;;
esac
done
# ── resolve category → test dir + image ─────────────────────────────
case "$CATEGORY" in
baker)
TEST_DIR="tests/baker"
TEST_IMAGE="hbw-system-test:latest"
;;
upm:rust)
TEST_DIR="tests/upm"
TEST_IMAGE="hbw-system-test:latest"
;;
pm:pacman|pm:arch)
TEST_DIR="tests/engine"
TEST_IMAGE="hbw-test-pm:test-arch"
;;
pm:apt)
TEST_DIR="tests/engine"
TEST_IMAGE="hbw-test-pm:test-apt"
;;
pm:dnf)
TEST_DIR="tests/engine"
TEST_IMAGE="hbw-test-pm:test-dnf"
;;
pm:apk)
TEST_DIR="tests/engine"
TEST_IMAGE="hbw-test-pm:test-apk"
;;
*)
echo "Unknown category: $CATEGORY" >&2
echo "Run './runner.sh --help' for usage." >&2
exit 1
;;
esac
# ── env vars ──────────────────────────────────────────────────────
DOCKER_ENV=()
DOCKER_ENV+=(-e RUST_LOG=${RUST_LOG:-info})
DOCKER_ENV+=(-e BAKER_LOG=${BAKER_LOG:-1})
# Load llm.env if present (for LLM-based test evaluation)
if [ -f "$SCRIPT_DIR/llm.env" ]; then
while IFS='=' read -r key value; do
[ -z "$key" ] && continue
[[ "$key" =~ ^# ]] && continue
DOCKER_ENV+=(-e "$key=$value")
done < "$SCRIPT_DIR/llm.env"
fi
if [ "$NETWORK" -eq 1 ]; then
DOCKER_ENV+=(-e RUST_UPM_NETWORK=1)
fi
# ── cleanup ─────────────────────────────────────────────────────────
cleanup() {
docker rm -f "$CONTAINER_NAME" 2>/dev/null || true
}
trap cleanup EXIT
# ── build baker binary ──────────────────────────────────────────────
build_binary() {
local target="x86_64-unknown-linux-musl"
local binary="$SCRIPT_DIR/workshop-baker/target/$target/debug/workshop-baker"
if [ ! -f "$binary" ]; then
echo "[runner] Building static (musl) baker binary..."
cargo build -p workshop-baker --target "$target" 2>&1 | tail -n 1
fi
}
# ── staging ─────────────────────────────────────────────────────────
prepare_staging() {
if [ "$TEST_DIR" != "tests/engine" ]; then
mkdir -p "$STAGING_DIR"
cp "$SCRIPT_DIR/workshop-baker/target/x86_64-unknown-linux-musl/debug/workshop-baker" "$STAGING_DIR/"
chmod +x "$STAGING_DIR/workshop-baker"
fi
}
# ── pm filter ───────────────────────────────────────────────────────
pm_filter() {
case "${TEST_IMAGE##*:}" in
test-arch) echo "-k pacman" ;;
test-apt) echo "-k apt" ;;
test-dnf) echo "-k dnf" ;;
test-apk) echo "-k apk" ;;
*) echo "" ;;
esac
}
# ── run ─────────────────────────────────────────────────────────────
run_container() {
local filter=""
if [ "$TEST_DIR" = "tests/engine" ]; then
filter="$(pm_filter)"
fi
echo "============================================"
echo " HBW System Test Runner"
echo " Category: $TEST_DIR Image: $TEST_IMAGE"
echo "============================================"
mkdir -p "$SCRIPT_DIR/tests/results"
local mounts=(-v "$SCRIPT_DIR:/src:ro")
if [ "$TEST_DIR" != "tests/engine" ]; then
mounts+=(-v "$STAGING_DIR:/workshop") # writable: baker writes temp scripts here
fi
local pytest_cmd="python3 -m venv /tmp/v && . /tmp/v/bin/activate && \
pip install -i https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple \
-r $TEST_DIR/../requirements.txt -q && \
exec python -m pytest -v -s $TEST_DIR ${PYTEST_ARGS[*]} $filter"
if [ "$MODE" = "manual" ]; then
docker run -it --rm --name "$CONTAINER_NAME" --privileged \
${DOCKER_ENV[@]} "${mounts[@]}" "$TEST_IMAGE"
else
# Baker/UPM images (hbw-system-test) have ./entrypoint.sh which
# doesn't exist when /src is the workdir → override to empty.
# PM images (hbw-test-pm:*) have /bin/sh -c → CMD passed directly.
if [ "$TEST_DIR" = "tests/engine" ]; then
docker run --rm --name "$CONTAINER_NAME" --privileged \
${DOCKER_ENV[@]} "${mounts[@]}" -w /src \
"$TEST_IMAGE" "$pytest_cmd"
else
docker run --rm --name "$CONTAINER_NAME" --privileged \
--entrypoint "" ${DOCKER_ENV[@]} "${mounts[@]}" -w /src \
"$TEST_IMAGE" /bin/sh -c "$pytest_cmd"
fi
fi
}
# ── main ────────────────────────────────────────────────────────────
build_binary
prepare_staging
run_container
echo "[runner] Done."
-32
View File
@@ -1,32 +0,0 @@
FROM archlinux:latest
RUN sed -i 's|https://geo.mirror.pkgbuild.com|https://mirrors.tuna.tsinghua.edu.cn/archlinux|g' /etc/pacman.d/mirrorlist && \
sed -i 's|https://mirror.rackspace.com/archlinux|https://mirrors.tuna.tsinghua.edu.cn/archlinux|g' /etc/pacman.d/mirrorlist && \
echo 'Server = https://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch' >> /etc/pacman.d/mirrorlist
RUN pacman -Syu --noconfirm && \
pacman -S --noconfirm \
python python-pip base-devel git curl wget sudo \
which procps-ng shadow && \
pacman -Scc --noconfirm
# Create test user
RUN useradd -m -s /bin/bash testuser
# Python environment
WORKDIR /app
COPY tests/requirements.txt ./requirements.txt
RUN python -m venv /opt/test-venv && \
/opt/test-venv/bin/pip install -r requirements.txt
# Copy test suite
COPY tests/ ./tests/
COPY tests/entrypoint.sh ./entrypoint.sh
RUN chmod +x entrypoint.sh
# Environment
ENV PATH="/opt/test-venv/bin:$PATH"
ENV PYTHONUNBUFFERED=1
ENV PYTHONDONTWRITEBYTECODE=1
ENTRYPOINT ["./entrypoint.sh"]
-105
View File
@@ -1,105 +0,0 @@
import os
import pytest
FIXTURES = os.path.join(os.path.dirname(os.path.dirname(__file__)), "fixtures", "bake")
@pytest.mark.bake
class TestBakeSimpleScript:
def test_simple_script_runs(self, run_baker, bake_workspace, work_dir):
prebake_cfg = os.path.join(
os.path.dirname(os.path.dirname(__file__)), "fixtures", "prebake", "minimal.yml"
)
script = os.path.join(FIXTURES, "simple.sh")
bake_base = os.path.join(FIXTURES, "bake_base.sh")
result = run_baker(
"bake", script,
"--bake-base", bake_base,
"--prebake", prebake_cfg,
cwd=work_dir,
timeout=60,
)
assert result["success"], f"Bake failed: {result['stderr']}"
assert "Hello from simple bake script" in result["stdout"]
def test_simple_script_sets_env_vars(self, run_baker, bake_workspace, work_dir):
prebake_cfg = os.path.join(
os.path.dirname(os.path.dirname(__file__)), "fixtures", "prebake", "minimal.yml"
)
script = os.path.join(FIXTURES, "simple.sh")
bake_base = os.path.join(FIXTURES, "bake_base.sh")
result = run_baker(
"bake", script,
"--bake-base", bake_base,
"--prebake", prebake_cfg,
cwd=work_dir,
timeout=60,
)
assert result["success"]
assert "test-pipeline" in result["stdout"]
@pytest.mark.bake
class TestBakePipeline:
def test_pipeline_functions_execute(self, run_baker, bake_workspace, work_dir):
prebake_cfg = os.path.join(
os.path.dirname(os.path.dirname(__file__)), "fixtures", "prebake", "minimal.yml"
)
script = os.path.join(FIXTURES, "pipeline.sh")
bake_base = os.path.join(FIXTURES, "bake_base.sh")
result = run_baker(
"bake", script,
"--bake-base", bake_base,
"--prebake", prebake_cfg,
cwd=work_dir,
timeout=60,
)
assert result["success"], f"Pipeline bake failed: {result['stderr']}"
assert os.path.exists("/tmp/bake-test-output/step1.txt")
assert os.path.exists("/tmp/bake-test-output/step2.txt")
def test_pipeline_step_outputs_correct(self, run_baker, bake_workspace, work_dir):
prebake_cfg = os.path.join(
os.path.dirname(os.path.dirname(__file__)), "fixtures", "prebake", "minimal.yml"
)
script = os.path.join(FIXTURES, "pipeline.sh")
bake_base = os.path.join(FIXTURES, "bake_base.sh")
result = run_baker(
"bake", script,
"--bake-base", bake_base,
"--prebake", prebake_cfg,
cwd=work_dir,
timeout=60,
)
assert result["success"]
with open("/tmp/bake-test-output/step1.txt") as f:
assert f.read().strip() == "step1-done"
with open("/tmp/bake-test-output/step2.txt") as f:
assert f.read().strip() == "step2-done"
@pytest.mark.bake
class TestBakeAfterDependencies:
def test_after_ordering(self, run_baker, bake_workspace, work_dir):
prebake_cfg = os.path.join(
os.path.dirname(os.path.dirname(__file__)), "fixtures", "prebake", "minimal.yml"
)
script = os.path.join(FIXTURES, "with_after.sh")
bake_base = os.path.join(FIXTURES, "bake_base.sh")
result = run_baker(
"bake", script,
"--bake-base", bake_base,
"--prebake", prebake_cfg,
cwd=work_dir,
timeout=60,
)
assert result["success"], f"@after bake failed: {result['stderr']}"
assert os.path.exists("/tmp/bake-after-test/status.txt")
assert os.path.exists("/tmp/bake-after-test/source.txt")
assert os.path.exists("/tmp/bake-after-test/build.txt")
-62
View File
@@ -1,62 +0,0 @@
import os
import pytest
from probes import file
from llm import judge_log, LLMClient
FIXTURES = os.path.join(os.path.dirname(os.path.dirname(__file__)), "fixtures", "finalize")
@pytest.mark.finalize
class TestFinalizeMinimal:
def test_minimal_finalize_succeeds(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "minimal.yml")
result = run_baker("finalize", config, cwd=work_dir, timeout=60)
assert result["success"], f"Minimal finalize failed: {result['stderr']}"
@pytest.mark.finalize
class TestFinalizeHooks:
def test_early_hook_executes(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "with_hooks.yml")
result = run_baker("finalize", config, cwd=work_dir, timeout=60)
assert result["success"]
assert "finalize early hook" in result["stdout"] or "finalize early hook" in result["stderr"]
def test_late_hook_executes(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "with_hooks.yml")
result = run_baker("finalize", config, cwd=work_dir, timeout=60)
assert result["success"]
assert "finalize late hook" in result["stdout"] or "finalize late hook" in result["stderr"]
@pytest.mark.finalize
@pytest.mark.llm
class TestFinalizeWithLLM:
def test_finalize_hooks_via_llm(self, run_baker, llm_env, work_dir):
if not llm_env:
pytest.skip("LLM environment not configured")
config = os.path.join(FIXTURES, "with_hooks.yml")
result = run_baker("finalize", config, cwd=work_dir, timeout=60)
combined_log = result["stdout"] + "\n" + result["stderr"]
judgment = judge_log(
operation="finalize with early and late hooks",
expected_behavior="Both early and late hooks executed successfully, "
"printing 'finalize early hook' and 'finalize late hook'",
actual_log=combined_log,
)
assert judgment.passed, f"LLM judgment failed: {judgment.reason}"
@pytest.mark.finalize
class TestFinalizePlugins:
@pytest.mark.skip(reason="Plugin download test not yet implemented")
def test_plugin_download(self, run_baker, work_dir):
pass
@pytest.mark.skip(reason="Artifact stage not yet implemented")
def test_artifact_collection(self, run_baker, work_dir):
pass
-202
View File
@@ -1,202 +0,0 @@
import os
import json
import pytest
from probes import user, file, package
FIXTURES = os.path.join(os.path.dirname(os.path.dirname(__file__)), "fixtures", "prebake")
@pytest.mark.prebake
class TestPrebakeConfigValidation:
def test_invalid_version_fails(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "invalid_version.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert not result["success"], f"Expected prebake to fail with invalid version"
assert "version" in result["stderr"].lower() or "Version" in result["stderr"]
def test_missing_required_field_fails(self, run_baker, work_dir):
missing_env = os.path.join(work_dir, "missing_env.yml")
with open(missing_env, "w") as f:
f.write('version: "1.0"\n')
result = run_baker("prebake", missing_env, cwd=work_dir, timeout=60)
assert not result["success"]
def test_minimal_config_succeeds(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "minimal.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"], f"Minimal prebake failed: {result['stderr']}"
@pytest.mark.prebake
class TestPrebakeBootstrap:
def test_bootstrap_creates_vulcan_user(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "bootstrap_only.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"], f"prebake failed: {result['stderr']}"
assert user.user_exists("vulcan")
def test_bootstrap_creates_workspace(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "bootstrap_only.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"]
assert file.dir_exists("/home/vulcan/workspace")
def test_bootstrap_creates_symlink(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "bootstrap_only.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"]
assert os.path.islink("/workspace")
def test_bootstrap_generates_bootstrap_script(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "bootstrap_only.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"]
assert file.file_exists("/bootstrap.sh")
def test_bootstrap_script_permissions(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "bootstrap_only.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"]
mode = file.file_permissions("/bootstrap.sh")
assert mode & 0o111, f"/bootstrap.sh not executable, mode: {oct(mode)}"
def test_bootstrap_script_contains_user_creation(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "bootstrap_only.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"]
assert file.file_exists("/bootstrap.sh")
assert file.file_contains("/bootstrap.sh", "useradd")
def test_bootstrap_script_contains_workspace_setup(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "bootstrap_only.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"]
assert file.file_contains("/bootstrap.sh", "mkdir -p /home/vulcan/workspace")
def test_bootstrap_with_pacman_creates_sudoers(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "with_pacman.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=120)
assert result["success"], f"prebake failed: {result['stderr']}"
assert file.file_exists("/etc/sudoers.d/workshop")
def test_custom_bootstrap_executes(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "with_custom_bootstrap.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"], f"Custom bootstrap failed: {result['stderr']}"
assert file.dir_exists("/custom-workspace")
@pytest.mark.prebake
class TestPrebakeDryRun:
def test_dry_run_does_not_create_user(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "bootstrap_only.yml")
env = {"HBW_DRY_RUN": "true"}
result = run_baker("prebake", config, cwd=work_dir, env=env, timeout=60)
assert result["success"], f"Dry-run failed: {result['stderr']}"
def test_dry_run_logs_script_content(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "bootstrap_only.yml")
env = {"HBW_DRY_RUN": "true", "RUST_LOG": "info"}
result = run_baker("prebake", config, cwd=work_dir, env=env, timeout=60)
assert result["success"]
combined = result["stdout"] + result["stderr"]
assert "DRY_RUN" in combined or "dry_run" in combined.lower()
@pytest.mark.prebake
class TestPrebakeHooks:
#@pytest.mark.xfail(reason="Hook execution fails with Permission denied in container environment", strict=False)
def test_early_hook_executes(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "with_hooks.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"], f"prebake with hooks failed: {result['stderr']}"
combined = result["stdout"] + result["stderr"]
assert "early hook executed" in combined
#@pytest.mark.xfail(reason="Hook execution fails with Permission denied in container environment", strict=False)
def test_late_hook_executes(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "with_hooks.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"], f"prebake with hooks failed: {result['stderr']}"
combined = result["stdout"] + result["stderr"]
assert "late hook executed" in combined
#@pytest.mark.xfail(reason="Hook execution fails with Permission denied in container environment", strict=False)
def test_hooks_run_in_order(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "with_hooks.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"]
combined = result["stdout"] + result["stderr"]
early_pos = combined.find("early hook executed")
late_pos = combined.find("late hook executed")
assert early_pos != -1 and late_pos != -1
assert early_pos < late_pos, "Early hook should execute before late hook"
def test_empty_hooks_succeed(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "minimal.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"], f"prebake with no hooks failed: {result['stderr']}"
@pytest.mark.prebake
class TestPrebakeDepsSystem:
def test_pacman_packages_installed(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "with_pacman.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=120)
assert result["success"], f"prebake failed: {result['stderr']}"
assert package.binary_exists("/usr/bin/curl")
assert package.binary_exists("/usr/bin/wget")
def test_pacman_mirror_change(self, run_baker, work_dir):
mirror_config = os.path.join(work_dir, "mirror.yml")
with open(mirror_config, "w") as f:
f.write('''
version: "1.0"
environment:
builder: "baremetal"
bootstrap:
user: "vulcan"
dependencies:
system:
pacman:
packages: ["which"]
mirror: "https://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch"
''')
result = run_baker("prebake", mirror_config, cwd=work_dir, timeout=120)
assert result["success"], f"prebake with mirror failed: {result['stderr']}"
@pytest.mark.prebake
class TestPrebakeEnvVars:
def test_prebake_envvars_injected(self, run_baker, work_dir):
hook_config = os.path.join(work_dir, "envvar_hook.yml")
with open(hook_config, "w") as f:
f.write('''
version: "1.0"
environment:
builder: "baremetal"
bootstrap:
user: "vulcan"
envvars:
prebake:
TEST_INJECTED: "injected_value"
hooks:
early:
- name: "check-env"
command: "echo $TEST_INJECTED"
''')
result = run_baker("prebake", hook_config, cwd=work_dir, timeout=60)
assert result["success"], f"prebake with envvars failed: {result['stderr']}"
combined = result["stdout"] + result["stderr"]
assert "injected_value" in combined
@pytest.mark.prebake
class TestPrebakeSecurity:
#@pytest.mark.xfail(reason="Privilege drop may fail in container without proper setup", strict=False)
def test_privilege_drop_after_bootstrap(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "with_security.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"], f"prebake with security failed: {result['stderr']}"
-180
View File
@@ -1,180 +0,0 @@
"""
Root conftest for HoneyBiscuitWorkshop system tests.
Provides core fixtures for:
- Locating the baker binary and examples
- Running subprocess commands with output capture
- Managing temporary working directories
"""
import os
import subprocess
import shutil
import tempfile
import pytest
def pytest_configure(config):
"""Register custom markers."""
config.addinivalue_line("markers", "prebake: prebake stage tests")
config.addinivalue_line("markers", "bake: bake stage tests")
config.addinivalue_line("markers", "finalize: finalize stage tests")
config.addinivalue_line("markers", "llm: tests requiring LLM API")
config.addinivalue_line("markers", "slow: slow running tests")
@pytest.fixture(scope="session")
def baker_bin():
"""Path to the workshop-baker binary."""
path = os.environ.get("BAKER_BIN", "/workshop/workshop-baker")
if not os.path.isfile(path):
pytest.skip(f"Baker binary not found at {path}")
if not os.access(path, os.X_OK):
pytest.skip(f"Baker binary not executable at {path}")
return path
@pytest.fixture(scope="session")
def examples_dir():
"""Path to the examples directory (may be None)."""
path = os.environ.get("EXAMPLES_DIR", "/workshop/examples")
if not os.path.isdir(path):
return None
return path
@pytest.fixture(scope="session")
def llm_env():
"""LLM environment variables dict (from llm.env)."""
env = {}
for var in [
"OPENAI_BASE_URL",
"OPENAI_API_KEY",
"ANTHROPIC_BASE_URL",
"ANTHROPIC_API_KEY",
"MODEL",
"REASONING_EFFORT",
]:
val = os.environ.get(var)
if val:
env[var] = val
return env
@pytest.fixture
def work_dir():
"""Create a temporary working directory for a test, cleaned up after."""
d = tempfile.mkdtemp(prefix="hbw-test-")
yield d
shutil.rmtree(d, ignore_errors=True)
@pytest.fixture
def run_cmd():
"""Fixture that provides a command runner function.
Returns a function that runs a command and returns
(stdout, stderr, returncode, success).
"""
def _run(args, cwd=None, env=None, timeout=120):
full_env = {**os.environ}
if env:
full_env.update(env)
show_logs = os.environ.get("BAKER_LOG", "0") == "1"
try:
if show_logs:
# Stream output to terminal in real-time while capturing
import sys, select
p = subprocess.Popen(
args, cwd=cwd, env=full_env,
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
text=True,
)
stdout_chunks, stderr_chunks = [], []
try:
p.wait(timeout=timeout)
except subprocess.TimeoutExpired:
p.kill()
p.wait()
return {"stdout": "", "stderr": f"Command timed out after {timeout}s",
"returncode": -1, "success": False}
for line in p.stdout:
sys.stderr.write("[baker] " + line)
stdout_chunks.append(line)
for line in p.stderr:
sys.stderr.write("[baker:err] " + line)
stderr_chunks.append(line)
return {
"stdout": "".join(stdout_chunks),
"stderr": "".join(stderr_chunks),
"returncode": p.returncode,
"success": p.returncode == 0,
}
else:
result = subprocess.run(
args, cwd=cwd, env=full_env,
capture_output=True, text=True, timeout=timeout,
)
return {
"stdout": result.stdout,
"stderr": result.stderr,
"returncode": result.returncode,
"success": result.returncode == 0,
}
except subprocess.TimeoutExpired:
return {
"stdout": "",
"stderr": f"Command timed out after {timeout}s",
"returncode": -1,
"success": False,
}
except Exception as e:
return {
"stdout": "",
"stderr": str(e),
"returncode": -1,
"success": False,
}
return _run
@pytest.fixture
def run_baker(baker_bin, run_cmd):
"""Fixture that provides a baker command runner.
Returns a function that runs workshop-baker with common flags.
"""
def _run(subcommand, *args, cwd=None, env=None, timeout=120):
cmd = [
baker_bin,
"-u", "testuser",
"-p", "test-pipeline",
"-b", "test-build-001",
"bare",
subcommand,
]
cmd.extend(args)
return run_cmd(cmd, cwd=cwd, env=env, timeout=timeout)
return _run
@pytest.fixture
def fixtures_dir():
"""Path to the test fixtures directory."""
return os.path.join(os.path.dirname(__file__), "fixtures")
@pytest.fixture
def bake_workspace():
"""Create /workspace for bake tests (baker writes temp scripts there)."""
existed = os.path.exists("/workspace")
if not existed:
os.makedirs("/workspace", exist_ok=True)
yield
if not existed:
shutil.rmtree("/workspace", ignore_errors=True)
-102
View File
@@ -1,102 +0,0 @@
"""PM system tests driven by bare workshop-baker prebake."""
import os
import pytest
from probes import binary_exists
PACMAN_MIRROR = "https://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch"
def _yml(pm, pkgs, mirror="", repo=None):
"""Build prebake.yml. repo is a list of lines (without lead indent)."""
m = ""
if mirror:
m += ' mirror: "' + mirror + '"\n'
if repo:
m += " repositories:\n"
for line in repo:
m += " " + line + "\n"
return """\
version: "1.0"
environment:
builder: "baremetal"
bootstrap:
user: "vulcan"
dependencies:
system:
{pm}:
packages: [{pkgs}]
{m}""".format(pm=pm, pkgs=pkgs, m=m)
# ── install ──────────────────────────────────────────────────────────
@pytest.mark.parametrize("pm_name,pkgs", [
("pacman", '"curl", "git"'),
("apt", '"curl", "git"'),
("dnf", '"curl", "git"'),
("apk", '"curl", "git"'),
])
def test_install(run_cmd, work_dir, pm_name, pkgs):
cfg = os.path.join(work_dir, "pm.yml")
mirror = PACMAN_MIRROR if pm_name == "pacman" else ""
with open(cfg, "w") as f:
f.write(_yml(pm_name, pkgs, mirror=mirror))
r = run_cmd(["/baker", "-p", "t", "-b", "1", "-u", "testuser",
"bare", "prebake", cfg], timeout=120)
assert r["success"], "[{}] prebake: {}".format(pm_name, r["stderr"][:300])
for pkg in ["curl", "git"]:
assert binary_exists("/usr/bin/" + pkg), "[{}] {} missing".format(pm_name, pkg)
@pytest.mark.parametrize("pm_name", ["pacman", "apt", "dnf", "apk"])
def test_reinstall(run_cmd, work_dir, pm_name):
cfg = os.path.join(work_dir, "pm.yml")
mirror = PACMAN_MIRROR if pm_name == "pacman" else ""
with open(cfg, "w") as f:
f.write(_yml(pm_name, '"curl"', mirror=mirror))
cmd = ["/baker", "-p", "t", "-b", "1", "-u", "testuser", "bare", "prebake", cfg]
r1 = run_cmd(cmd, timeout=120)
assert r1["success"], "[{}] first install failed".format(pm_name)
r2 = run_cmd(cmd, timeout=120)
assert r2["success"], "[{}] reinstall failed".format(pm_name)
# ── change_mirror ────────────────────────────────────────────────────
@pytest.mark.parametrize("pm_name", ["pacman", "apt", "dnf", "apk"])
def test_mirror(run_cmd, work_dir, pm_name):
cfg = os.path.join(work_dir, "pm.yml")
mirror = PACMAN_MIRROR if pm_name == "pacman" else ""
with open(cfg, "w") as f:
f.write(_yml(pm_name, '"which"', mirror=mirror))
r = run_cmd(["/baker", "-p", "t", "-b", "1", "-u", "testuser",
"bare", "prebake", cfg], timeout=120)
assert r["success"], "[{}] mirror: {}".format(pm_name, r["stderr"][:300])
assert binary_exists("/usr/bin/which"), "[{}] which not installed".format(pm_name)
# ── add_repository ───────────────────────────────────────────────────
@pytest.mark.parametrize("pm_name,repo_lines,result_file", [
("pacman", ['- name: custom', ' server: "https://repo.example.com/$arch"'],
"/etc/pacman.conf"),
("apt", ['- url: "https://repo.example.com/debian"', ' distro: bookworm'],
"/etc/apt/sources.list.d/custom.list"),
("dnf", ['- name: custom', ' baseurl: "https://repo.example.com/fedora"'],
"/etc/yum.repos.d/custom.repo"),
("apk", ['- url: "https://repo.example.com/v3.19/main"'],
"/etc/apk/repositories"),
])
@pytest.mark.xfail(reason="fake repo URL, package install may fail", strict=False)
def test_add_repo(run_cmd, work_dir, pm_name, repo_lines, result_file):
cfg = os.path.join(work_dir, "pm.yml")
mirror = PACMAN_MIRROR if pm_name == "pacman" else ""
with open(cfg, "w") as f:
f.write(_yml(pm_name, '"curl"', mirror=mirror, repo=repo_lines))
r = run_cmd(["/baker", "-p", "t", "-b", "1", "-u", "testuser",
"bare", "prebake", cfg], timeout=120)
# Prebake may fail if repo URL is unreachable — check only the config file
# was written by add_repository before the (possibly failing) install step.
assert binary_exists(result_file), \
"[{}] repo file missing: {}".format(pm_name, result_file)
-36
View File
@@ -1,36 +0,0 @@
#!/bin/bash
set -e
# Load LLM environment if available
if [ -f /workshop/llm.env ]; then
set -a
source /workshop/llm.env
set +a
fi
# Export test configuration
export BAKER_BIN=/workshop/workshop-baker
export EXAMPLES_DIR=/workshop/examples
export TEST_MODE=${TEST_MODE:-auto}
if [ "$TEST_MODE" = "manual" ]; then
echo "========================================"
echo " HoneyBiscuitWorkshop System Tests"
echo " Mode: MANUAL (interactive shell)"
echo "========================================"
echo ""
echo "Binary: $BAKER_BIN"
echo "Examples: $EXAMPLES_DIR"
echo ""
echo "Run tests manually:"
echo " cd /app && pytest tests/ -v"
echo ""
exec /bin/bash
else
echo "========================================"
echo " HoneyBiscuitWorkshop System Tests"
echo " Mode: AUTO (running pytest)"
echo "========================================"
cd /app
exec python -m pytest tests/ -v --tb=short --junitxml=/results/report.xml "$@"
fi
-6
View File
@@ -1,6 +0,0 @@
#!/bin/bash
[ -f /dev/shm/bakeexport.{{ name }} ] && . /dev/shm/bakeexport.{{ name }} || true
{{ condition }}
{{ main }}
{{ name }}
{{ export }}
-25
View File
@@ -1,25 +0,0 @@
#!/bin/bash
# @pipeline
step_one() {
echo "Step 1: Setup"
mkdir -p /tmp/bake-test-output
echo "step1-done" > /tmp/bake-test-output/step1.txt
}
# @pipeline
step_two() {
echo "Step 2: Build"
echo "step2-done" > /tmp/bake-test-output/step2.txt
}
# @pipeline
step_three() {
echo "Step 3: Verify"
if [ -f /tmp/bake-test-output/step1.txt ] && [ -f /tmp/bake-test-output/step2.txt ]; then
echo "All steps completed successfully"
else
echo "ERROR: Missing step outputs"
exit 1
fi
}
-5
View File
@@ -1,5 +0,0 @@
#!/bin/bash
echo "Hello from simple bake script"
echo "Task ID: $HBW_TASKID"
echo "Pipeline: $HBW_PIPELINE"
exit 0
-22
View File
@@ -1,22 +0,0 @@
#!/bin/bash
# @pipeline
init_workspace() {
echo "Initializing workspace"
mkdir -p /tmp/bake-after-test
echo "init" > /tmp/bake-after-test/status.txt
}
# @pipeline
# @after(init_workspace)
fetch_source() {
echo "Fetching source (depends on init_workspace)"
echo "fetched" > /tmp/bake-after-test/source.txt
}
# @pipeline
# @after(fetch_source)
build_project() {
echo "Building project (depends on fetch_source)"
echo "built" > /tmp/bake-after-test/build.txt
}
-4
View File
@@ -1,4 +0,0 @@
#!/bin/bash
echo "EARLY_HOOK_EXECUTED"
echo "Working directory: $(pwd)"
echo "User: $(whoami)"
-3
View File
@@ -1,3 +0,0 @@
#!/bin/bash
echo "LATE_HOOK_EXECUTED"
echo "Working directory: $(pwd)"
-6
View File
@@ -1,6 +0,0 @@
version: "0.0.1"
plugin: {}
notification: {}
artifact: []
cleanup:
policy: "auto"
-13
View File
@@ -1,13 +0,0 @@
version: "0.0.1"
plugin: {}
notification: {}
artifact: []
cleanup:
policy: "auto"
hooks:
early:
- name: "finalize-early"
command: "echo 'finalize early hook'"
late:
- name: "finalize-late"
command: "echo 'finalize late hook'"
-6
View File
@@ -1,6 +0,0 @@
FROM docker.1ms.run/library/alpine:3.19
RUN sed -i 's|dl-cdn.alpinelinux.org|mirrors.tuna.tsinghua.edu.cn|g' /etc/apk/repositories
RUN apk add --no-cache python3 py3-pip bash
COPY workshop-baker/target/x86_64-unknown-linux-musl/debug/workshop-baker /baker
RUN chmod +x /baker
ENTRYPOINT ["/bin/sh", "-c"]
-6
View File
@@ -1,6 +0,0 @@
FROM docker.1ms.run/library/debian:bookworm-slim
RUN sed -i 's|deb.debian.org|mirrors.tuna.tsinghua.edu.cn|g' /etc/apt/sources.list.d/debian.sources 2>/dev/null; sed -i 's|security.debian.org|mirrors.tuna.tsinghua.edu.cn/debian-security|g' /etc/apt/sources.list.d/debian.sources 2>/dev/null; true
RUN apt-get update -qq && apt-get install -y -qq --no-install-recommends python3 python3-venv ca-certificates 2>&1 | tail -n 2
COPY workshop-baker/target/x86_64-unknown-linux-musl/debug/workshop-baker /baker
RUN chmod +x /baker
ENTRYPOINT ["/bin/sh", "-c"]
-6
View File
@@ -1,6 +0,0 @@
FROM docker.1ms.run/library/fedora:43
RUN sed -e 's|^metalink=|#metalink=|g' -e 's|^#baseurl=http://download.example/pub/fedora/linux|baseurl=https://mirrors.tuna.tsinghua.edu.cn/fedora|g' -i.bak /etc/yum.repos.d/fedora.repo /etc/yum.repos.d/fedora-updates.repo
RUN dnf install -y python3 2>&1 | tail -n 2
COPY workshop-baker/target/x86_64-unknown-linux-musl/debug/workshop-baker /baker
RUN chmod +x /baker
ENTRYPOINT ["/bin/sh", "-c"]
-2
View File
@@ -1,2 +0,0 @@
FROM docker.1ms.run/library/archlinux:latest
RUN echo 'Server = https://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch' > /etc/pacman.d/mirrorlist
-8
View File
@@ -1,8 +0,0 @@
version: "1.0"
environment:
builder: "baremetal"
bootstrap:
user: "vulcan"
workspace:
path: "/home/vulcan/workspace"
fallback: true
-15
View File
@@ -1,15 +0,0 @@
version: "1.0"
environment:
builder: "baremetal"
bootstrap:
user: "vulcan"
workspace:
path: "/home/vulcan/workspace"
fallback: true
hooks:
early:
- name: "test-early-hook"
command: "echo 'early hook executed'"
late:
- name: "test-late-hook"
command: "echo 'late hook executed'"
-3
View File
@@ -1,3 +0,0 @@
version: "0.5"
environment:
builder: "baremetal"
-8
View File
@@ -1,8 +0,0 @@
version: "1.0"
environment:
builder: "baremetal"
bootstrap:
user: "vulcan"
workspace:
path: "/home/vulcan/workspace"
fallback: true
-12
View File
@@ -1,12 +0,0 @@
version: "1.0"
environment:
builder: "baremetal"
bootstrap:
user: "vulcan"
workspace:
path: "/home/vulcan/workspace"
fallback: true
custom: |
#!/bin/sh
echo 'custom bootstrap executed'
mkdir -p /custom-workspace
-16
View File
@@ -1,16 +0,0 @@
version: "1.0"
environment:
builder: "baremetal"
bootstrap:
user: "vulcan"
workspace:
path: "/home/vulcan/workspace"
fallback: true
dependencies:
system:
pacman:
packages:
- "which"
user:
cargo:
- "ripgrep"
-13
View File
@@ -1,13 +0,0 @@
version: "1.0"
environment:
builder: "baremetal"
bootstrap:
user: "vulcan"
workspace:
path: "/home/vulcan/workspace"
fallback: true
envvars:
prebake:
TEST_PREBAKE_VAR: "prebake_value"
bake:
TEST_BAKE_VAR: "bake_value"
-17
View File
@@ -1,17 +0,0 @@
version: "1.0"
environment:
builder: "baremetal"
bootstrap:
user: "vulcan"
workspace:
path: "/home/vulcan/workspace"
fallback: true
hooks:
early:
- name: "test-early-hook"
command: "echo 'early hook executed'"
working_dir: "/workspace"
late:
- name: "test-late-hook"
command: "echo 'late hook executed'"
working_dir: "/workspace"
-15
View File
@@ -1,15 +0,0 @@
version: "1.0"
environment:
builder: "baremetal"
bootstrap:
user: "vulcan"
workspace:
path: "/home/vulcan/workspace"
fallback: true
dependencies:
system:
pacman:
mirror: "https://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch"
packages:
- "curl"
- "wget"
-10
View File
@@ -1,10 +0,0 @@
version: "1.0"
environment:
builder: "baremetal"
bootstrap:
user: "vulcan"
workspace:
path: "/home/vulcan/workspace"
fallback: true
security:
drop_after: "Bootstrap"
-12
View File
@@ -1,12 +0,0 @@
"""
LLM-based log judgment system for system tests.
Uses LLM APIs to evaluate non-deterministic test outputs (logs, stderr, etc.)
where simple string matching is insufficient.
"""
from .client import LLMClient
from .cache import LLMCache
from .judge import judge_log, JudgmentResult
__all__ = ["LLMClient", "LLMCache", "judge_log", "JudgmentResult"]
-54
View File
@@ -1,54 +0,0 @@
"""
Disk-based cache for LLM API responses.
Uses SHA256 of (prompt + model + system_prompt) as cache key.
Cache files stored as JSON in ``.cache/`` directory.
"""
import hashlib
import json
from pathlib import Path
class LLMCache:
"""SHA256-based disk cache for LLM API responses."""
def __init__(self, cache_dir: str | None = None):
if cache_dir is None:
cache_dir = "/tmp/llm-cache"
self.cache_dir = Path(cache_dir)
self.cache_dir.mkdir(parents=True, exist_ok=True)
def _make_key(self, prompt: str, model: str, system_prompt: str = "") -> str:
"""Create deterministic cache key from request parameters."""
data = json.dumps(
{"prompt": prompt, "model": model, "system": system_prompt},
sort_keys=True,
)
return hashlib.sha256(data.encode()).hexdigest()
def get(self, prompt: str, model: str, system_prompt: str = "") -> dict | None:
"""Retrieve cached response if exists."""
key = self._make_key(prompt, model, system_prompt)
path = self.cache_dir / f"{key}.json"
if path.exists():
try:
with open(path) as f:
return json.load(f)
except (json.JSONDecodeError, IOError):
return None
return None
def set(
self, prompt: str, model: str, system_prompt: str, response: dict
) -> None:
"""Store response in cache."""
key = self._make_key(prompt, model, system_prompt)
path = self.cache_dir / f"{key}.json"
with open(path, "w") as f:
json.dump(response, f, ensure_ascii=False)
def clear(self) -> None:
"""Clear all cached responses."""
for f in self.cache_dir.glob("*.json"):
f.unlink()
-187
View File
@@ -1,187 +0,0 @@
"""
LLM client supporting both Anthropic and OpenAI APIs.
Prefers Anthropic format (per project convention).
Falls back to OpenAI format if Anthropic is unavailable.
"""
import os
import logging
from .cache import LLMCache
logger = logging.getLogger(__name__)
class LLMClient:
"""LLM client with caching support. Prefers Anthropic API."""
def __init__(self, cache: LLMCache | None = None):
self.cache = cache or LLMCache()
self._anthropic_client = None
self._openai_client = None
self._init_clients()
def _init_clients(self):
"""Initialize API clients based on available environment variables."""
# Anthropic (preferred)
anthropic_key = os.environ.get("ANTHROPIC_API_KEY")
anthropic_base = os.environ.get("ANTHROPIC_BASE_URL")
if anthropic_key:
try:
import anthropic
kwargs: dict = {"api_key": anthropic_key}
if anthropic_base:
kwargs["base_url"] = anthropic_base
self._anthropic_client = anthropic.Anthropic(**kwargs)
logger.info("Anthropic client initialized")
except ImportError:
logger.warning("anthropic package not installed")
# OpenAI (fallback)
openai_key = os.environ.get("OPENAI_API_KEY")
openai_base = os.environ.get("OPENAI_BASE_URL")
if openai_key:
try:
import openai
kwargs = {"api_key": openai_key}
if openai_base:
kwargs["base_url"] = openai_base
self._openai_client = openai.OpenAI(**kwargs)
logger.info("OpenAI client initialized")
except ImportError:
logger.warning("openai package not installed")
@property
def available(self) -> bool:
"""Check if any LLM client is available."""
return (
self._anthropic_client is not None or self._openai_client is not None
)
def complete(
self, prompt: str, system_prompt: str = "", max_tokens: int = 2000
) -> str:
"""Send a completion request, with caching.
Args:
prompt: User message.
system_prompt: System message (sets role / persona).
max_tokens: Max response tokens.
Returns:
LLM response text.
Raises:
RuntimeError: If no client is available.
"""
model = os.environ.get("MODEL", "deepseek-v4-flash")
# Check cache
cached = self.cache.get(prompt, model, system_prompt)
if cached:
logger.info("Cache hit for LLM request")
return cached["content"]
# Try Anthropic first
if self._anthropic_client:
content = self._call_anthropic(
prompt, system_prompt, model, max_tokens
)
elif self._openai_client:
content = self._call_openai(prompt, system_prompt, model, max_tokens)
else:
raise RuntimeError(
"No LLM client available. "
"Set ANTHROPIC_API_KEY or OPENAI_API_KEY."
)
# Cache the response
self.cache.set(prompt, model, system_prompt, {"content": content})
return content
# ------------------------------------------------------------------
# Private helpers
# ------------------------------------------------------------------
def _call_anthropic(
self, prompt: str, system_prompt: str, model: str, max_tokens: int
) -> str:
"""Call Anthropic API."""
kwargs: dict = {
"model": model,
"max_tokens": max_tokens,
"messages": [{"role": "user", "content": prompt}],
}
if system_prompt:
kwargs["system"] = system_prompt
# Only add reasoning_effort if the model supports it
reasoning_effort = os.environ.get("REASONING_EFFORT")
if reasoning_effort:
try:
kwargs["reasoning_effort"] = reasoning_effort
response = self._anthropic_client.messages.create(**kwargs)
except Exception as e:
if (
"reasoning_effort" in str(e).lower()
or "unexpected" in str(e).lower()
):
del kwargs["reasoning_effort"]
response = self._anthropic_client.messages.create(**kwargs)
else:
raise
else:
response = self._anthropic_client.messages.create(**kwargs)
# Extract text from response content blocks
if hasattr(response, "content") and response.content:
block = response.content[0]
if hasattr(block, "text"):
return block.text
if hasattr(block, "type") and block.type == "thinking":
for b in response.content:
if hasattr(b, "text"):
return b.text
return str(block)
return str(response)
def _call_openai(
self, prompt: str, system_prompt: str, model: str, max_tokens: int
) -> str:
"""Call OpenAI-compatible API."""
messages: list[dict] = []
if system_prompt:
messages.append({"role": "system", "content": system_prompt})
messages.append({"role": "user", "content": prompt})
kwargs: dict = {
"model": model,
"messages": messages,
"max_tokens": max_tokens,
}
# Only add reasoning_effort if supported
reasoning_effort = os.environ.get("REASONING_EFFORT")
if reasoning_effort:
try:
kwargs["reasoning_effort"] = reasoning_effort
response = self._openai_client.chat.completions.create(
**kwargs
)
except Exception as e:
if (
"reasoning_effort" in str(e).lower()
or "unexpected" in str(e).lower()
):
del kwargs["reasoning_effort"]
response = self._openai_client.chat.completions.create(
**kwargs
)
else:
raise
else:
response = self._openai_client.chat.completions.create(**kwargs)
return response.choices[0].message.content
-149
View File
@@ -1,149 +0,0 @@
"""
LLM-based log judgment for system tests.
Evaluates logs against expected behavior using LLM semantic analysis.
Returns structured pass/fail results with reasoning.
"""
import json
import re
import logging
from dataclasses import dataclass
from .client import LLMClient
logger = logging.getLogger(__name__)
JUDGE_SYSTEM_PROMPT = """\
You are a test result judge for a CI/CD system called HoneyBiscuitWorkshop.
Your job: Evaluate whether actual command output/logs match expected behavior.
RULES:
1. You MUST respond with valid JSON in this exact format:
{"verdict": "pass" | "fail", "reason": "brief explanation"}
2. "pass" = the output confirms the expected behavior occurred
3. "fail" = the output shows the expected behavior did NOT occur, or shows an error
4. Ignore irrelevant noise (timestamps, debug lines, unrelated warnings)
5. Focus ONLY on whether the specific expected behavior is confirmed or contradicted
6. Be strict: if the evidence is ambiguous, verdict is "fail"
7. Do NOT include any text outside the JSON object"""
@dataclass
class JudgmentResult:
"""Result of an LLM log judgment."""
verdict: str # "pass" or "fail"
reason: str
raw_response: str
@property
def passed(self) -> bool:
return self.verdict == "pass"
def _extract_relevant_logs(full_log: str, max_chars: int = 4000) -> str:
"""Extract relevant portions from logs, removing noise.
Keeps the last *max_chars* characters (most recent / relevant).
Removes common noise patterns.
"""
lines = full_log.strip().split("\n")
noise_patterns = [
r"^\d{4}-\d{2}-\d{2}T",
r"^DEBUG ",
r"^TRACE ",
]
filtered = [
line
for line in lines
if not any(re.match(p, line) for p in noise_patterns)
]
text = "\n".join(filtered)
if len(text) > max_chars:
text = "...\n" + text[-max_chars:]
return text
def judge_log(
operation: str,
expected_behavior: str,
actual_log: str,
client: LLMClient | None = None,
) -> JudgmentResult:
"""Judge whether actual logs match expected behavior.
Args:
operation: Description of what was attempted
(e.g. ``"prebake bootstrap stage"``).
expected_behavior: What should have happened
(e.g. ``"user vulcan was created"``).
actual_log: The actual stdout / stderr output.
client: ``LLMClient`` instance (creates default if ``None``).
Returns:
``JudgmentResult`` with verdict and reason.
"""
if client is None:
client = LLMClient()
if not client.available:
logger.warning("No LLM client available, returning fail verdict")
return JudgmentResult(
verdict="fail",
reason="No LLM client available for judgment",
raw_response="",
)
relevant_log = _extract_relevant_logs(actual_log)
prompt = f"""\
## Operation
{operation}
## Expected Behavior
{expected_behavior}
## Actual Output (relevant excerpts)
```
{relevant_log}
```
Evaluate: Does the actual output confirm the expected behavior occurred?
Respond with JSON only."""
try:
response = client.complete(
prompt=prompt,
system_prompt=JUDGE_SYSTEM_PROMPT,
max_tokens=500,
)
# Parse JSON response — handle potential markdown code blocks
text = response.strip()
if text.startswith("```"):
lines = text.split("\n")
text = "\n".join(lines[1:-1]) if len(lines) > 2 else text
result = json.loads(text)
return JudgmentResult(
verdict=result.get("verdict", "fail"),
reason=result.get("reason", "No reason provided"),
raw_response=response,
)
except json.JSONDecodeError as e:
logger.error(f"Failed to parse LLM response as JSON: {e}")
return JudgmentResult(
verdict="fail",
reason=f"LLM response was not valid JSON: {response[:200]}",
raw_response=response,
)
except Exception as e:
logger.error(f"LLM judgment failed: {e}")
return JudgmentResult(
verdict="fail",
reason=f"LLM call failed: {str(e)}",
raw_response="",
)
-45
View File
@@ -1,45 +0,0 @@
"""
Environment probes for verifying system state in workshop-baker tests.
Each probe is a standalone function that can be called from any test.
Probes return bool or Optional[int/str]; they do not raise on failure.
A probe accepts an optional *runner* callable (default ``subprocess.run``)
that decouples it from the execution context (host or Docker).
"""
from .user import user_exists, user_uid, user_gid, user_in_group, user_shell
from .file import (
file_exists,
file_absent,
dir_exists,
dir_absent,
file_permissions,
file_contains,
file_owned_by,
file_group,
)
from .package import binary_exists, binary_on_path, package_installed, package_version
from .process import process_running, process_count
__all__ = [
"user_exists",
"user_uid",
"user_gid",
"user_in_group",
"user_shell",
"file_exists",
"file_absent",
"dir_exists",
"dir_absent",
"file_permissions",
"file_contains",
"file_owned_by",
"file_group",
"package_installed",
"package_version",
"binary_exists",
"binary_on_path",
"process_running",
"process_count",
]
-53
View File
@@ -1,53 +0,0 @@
"""File and directory existence / permission probes."""
import os
import stat
import grp
import pwd
def file_exists(path: str) -> bool:
"""Check if *path* exists and is a regular file."""
return os.path.isfile(path)
def dir_exists(path: str) -> bool:
"""Check if *path* exists and is a directory."""
return os.path.isdir(path)
def file_permissions(path: str) -> int:
"""Return the permission mode of *path* as an octal int (e.g. ``0o755``).
Raises ``FileNotFoundError`` if *path* doesn't exist.
"""
return stat.S_IMODE(os.stat(path).st_mode)
def file_contains(path: str, text: str) -> bool:
"""Check if a file contains the specified text."""
with open(path, "r", errors="replace") as f:
return text in f.read()
def file_owned_by(path: str, username: str) -> bool:
"""Check if a file is owned by the specified user."""
st = os.stat(path)
owner = pwd.getpwuid(st.st_uid).pw_name
return owner == username
def file_group(path: str) -> str:
"""Get the group owner of a file."""
st = os.stat(path)
return grp.getgrgid(st.st_gid).gr_name
def file_absent(path: str) -> bool:
"""Check that *path* does not exist."""
return not os.path.lexists(path)
def dir_absent(path: str) -> bool:
"""Check that *path* does not exist or is not a directory."""
return not os.path.isdir(path)
-42
View File
@@ -1,42 +0,0 @@
"""Package / binary existence probes (cross-distro).
Each probe accepts an optional *runner* callable with the same signature
as ``subprocess.run``. Defaults to running on the host; for Docker tests,
pass a runner that wraps ``docker run --rm <image>``.
"""
import subprocess
from typing import Any, Callable
Runner = Callable[..., Any]
def _run(cmd: list[str], runner: Runner = subprocess.run, **kw: Any) -> Any:
opts = {"capture_output": True, "text": True, "timeout": 30}
opts.update(kw)
return runner(cmd, **opts)
def _succeeded(r: Any) -> bool:
""".returncode==0 for CompletedProcess, or ["success"] for dict."""
if hasattr(r, "returncode"):
return r.returncode == 0
return bool(r.get("success", False))
def binary_exists(path: str, runner: Runner = subprocess.run) -> bool:
"""Check if a binary exists at *path* (e.g. ``/usr/bin/curl``).
Uses ``test -f`` (POSIX, guaranteed available).
"""
return _succeeded(_run(["test", "-f", path], runner))
def binary_on_path(name: str, runner: Runner = subprocess.run) -> bool:
"""Check if a command is on PATH (via ``command -v``)."""
return _succeeded(_run(["sh", "-c", "command -v " + name], runner))
# Backward-compatible aliases
package_installed = binary_exists
package_version = binary_on_path # used in __init__ export, now binary-based
-44
View File
@@ -1,44 +0,0 @@
"""Process existence probes."""
import os
import subprocess
def process_running(process_name: str) -> bool:
"""Check if a process with the given name is running."""
try:
result = subprocess.run(
["pgrep", "-x", process_name],
capture_output=True,
text=True,
)
return result.returncode == 0
except FileNotFoundError:
# Fall back to /proc scan if pgrep not available
try:
for pid_dir in os.listdir("/proc"):
if pid_dir.isdigit():
try:
with open(f"/proc/{pid_dir}/comm") as f:
if f.read().strip() == process_name:
return True
except (IOError, OSError):
continue
except OSError:
pass
return False
def process_count(process_name: str) -> int:
"""Count processes with the given name."""
try:
result = subprocess.run(
["pgrep", "-c", "-x", process_name],
capture_output=True,
text=True,
)
if result.returncode == 0:
return int(result.stdout.strip())
return 0
except (FileNotFoundError, ValueError):
return 0
-41
View File
@@ -1,41 +0,0 @@
"""User existence and attribute probes."""
import pwd
import grp
def user_exists(username: str) -> bool:
"""Check if a system user exists."""
try:
pwd.getpwnam(username)
return True
except KeyError:
return False
def user_uid(username: str) -> int | None:
try:
return pwd.getpwnam(username).pw_uid
except KeyError:
return None
def user_gid(username: str) -> int | None:
try:
return pwd.getpwnam(username).pw_gid
except KeyError:
return None
def user_in_group(username: str, groupname: str) -> bool:
"""Check if a user is a member of a group."""
try:
group = grp.getgrnam(groupname)
return username in group.gr_mem
except KeyError:
return False
def user_shell(username: str) -> str | None:
try:
return pwd.getpwnam(username).pw_shell
except KeyError:
return None
-6
View File
@@ -1,6 +0,0 @@
pytest>=8.0.0
pytest-timeout>=2.2.0
pytest-xdist>=3.5.0
anthropic>=0.40.0,<0.90.0
openai>=1.50.0,<2.0.0
pyyaml>=6.0
-150
View File
@@ -1,150 +0,0 @@
"""Go UPM system tests — go install + go mod download via bare prebake."""
import os
import subprocess
import pytest
from probes import binary_exists
def _yml(actions_yaml):
return """\
version: "1.0"
environment:
builder: "baremetal"
bootstrap:
user: "vulcan"
security:
drop_after: never
dependencies:
config:
repology_endpoint: none
system:
pacman:
packages:
user:
go:
""" + _indent(actions_yaml, 6)
def _indent(text, spaces):
pad = " " * spaces
return "\n".join(pad + line if line.strip() else line
for line in text.splitlines())
def _query(cmd, timeout=60):
try:
r = subprocess.run(cmd, capture_output=True, text=True,
timeout=timeout, env={**os.environ, "HOME": "/root"})
return {"success": r.returncode == 0, "stdout": r.stdout, "stderr": r.stderr}
except Exception as e:
return {"success": False, "stdout": "", "stderr": str(e)}
def _run_prebake(run_baker, config_path, timeout, operation, expected):
r = run_baker("prebake", config_path, timeout=timeout,
env={"RUST_LOG": os.environ.get("RUST_LOG", "info")})
full_log = "[stdout]\n{}\n[stderr]\n{}".format(r["stdout"], r["stderr"])
return r, full_log
def _check(r, full_log, operation, expected):
if r["success"]:
return
diagnosis = ""
try:
from llm import judge_log
j = judge_log(operation, expected, full_log)
diagnosis = "\n LLM diagnosis: [{}] {}".format(j.verdict, j.reason)
except Exception:
pass
pytest.fail("{} failed ({} chars):\n{}\n{}".format(
operation, len(full_log), full_log, diagnosis))
def _gopath_bin(name):
"""Return the expected $GOPATH/bin path or $HOME/go/bin fallback."""
gopath = os.environ.get("GOPATH", "")
home = os.environ.get("HOME", "/root")
if not gopath:
gopath = os.path.join(home, "go")
return os.path.join(gopath, "bin", name)
# ── go install ──────────────────────────────────────────────────────
def test_go_install(run_baker, work_dir):
"""go install a small known tool, verify binary exists."""
cfg = os.path.join(work_dir, "go.yml")
with open(cfg, "w") as f:
f.write(_yml("""\
proxy: "https://goproxy.cn,direct"
packages: ["golang.org/x/tools/cmd/goimports@latest"]
"""))
r, log = _run_prebake(run_baker, cfg, timeout=300,
operation="go install goimports",
expected="goimports binary appears in GOPATH/bin")
_check(r, log, "go install", "goimports installed")
bin_path = _gopath_bin("goimports")
assert binary_exists(bin_path), f"goimports not found at {bin_path}"
# ── go mod download ─────────────────────────────────────────────────
def test_go_mod_download(run_baker, work_dir):
"""Create a minimal go.mod, download deps, verify cache populated."""
mod_dir = os.path.join(work_dir, "testmod")
os.makedirs(mod_dir, exist_ok=True)
with open(os.path.join(mod_dir, "go.mod"), "w") as f:
f.write("module test\n\ngo 1.21\n\nrequire golang.org/x/text v0.14.0\n")
cfg = os.path.join(work_dir, "go.yml")
with open(cfg, "w") as f:
f.write(_yml("""\
proxy: "https://goproxy.cn,direct"
modules: ["{mod}"]
""".format(mod=os.path.join(mod_dir, "go.mod"))))
r, log = _run_prebake(run_baker, cfg, timeout=180,
operation="go mod download",
expected="GOMODCACHE has x/text cached")
_check(r, log, "go mod download", "dependencies downloaded to module cache")
# Check GOMODCACHE or default $GOPATH/pkg/mod
gomodcache = os.environ.get("GOMODCACHE", "")
if not gomodcache:
gopath = os.environ.get("GOPATH",
os.path.join(os.environ.get("HOME", "/root"), "go"))
gomodcache = os.path.join(gopath, "pkg", "mod")
cache_dir = os.path.join(gomodcache, "golang.org", "x")
assert os.path.isdir(cache_dir), f"no module cache at {cache_dir}"
# ── go install + mod download ───────────────────────────────────────
def test_go_full(run_baker, work_dir):
"""go install a tool AND download project deps."""
mod_dir = os.path.join(work_dir, "testmod2")
os.makedirs(mod_dir, exist_ok=True)
with open(os.path.join(mod_dir, "go.mod"), "w") as f:
f.write("module test\n\ngo 1.21\n")
cfg = os.path.join(work_dir, "go.yml")
with open(cfg, "w") as f:
f.write(_yml("""\
proxy: "https://goproxy.cn,direct"
packages: ["golang.org/x/tools/cmd/goimports@latest"]
modules: ["{mod}"]
env:
GOMODCACHE: "/tmp/gocache"
""".format(mod=os.path.join(mod_dir, "go.mod"))))
r, log = _run_prebake(run_baker, cfg, timeout=300,
operation="go install + go mod download",
expected="goimports installed, module cache populated")
_check(r, log, "go full", "tool installed + deps downloaded")
assert binary_exists(_gopath_bin("goimports")), "goimports missing"
assert os.path.isdir("/tmp/gocache"), "GOMODCACHE not created"
-163
View File
@@ -1,163 +0,0 @@
"""Python UPM system tests — pip, uv, poetry, pipx via bare prebake."""
import os
import pytest
from probes import binary_exists
RUST_LOG = os.environ.get("RUST_LOG", "info")
def _yml(actions_yaml):
return """\
version: "1.0"
environment:
builder: "baremetal"
bootstrap:
user: "vulcan"
security:
drop_after: never
dependencies:
config:
repology_endpoint: none
system:
pacman:
packages:
user:
python:
""" + _indent(actions_yaml, 6)
def _indent(text, spaces):
pad = " " * spaces
return "\n".join(pad + line if line.strip() else line
for line in text.splitlines())
def _query(cmd, timeout=60):
import subprocess
try:
r = subprocess.run(cmd, capture_output=True, text=True,
timeout=timeout, env={**os.environ, "HOME": "/root"})
return {"success": r.returncode == 0, "stdout": r.stdout, "stderr": r.stderr}
except Exception as e:
return {"success": False, "stdout": "", "stderr": str(e)}
def _run_prebake(run_baker, config_path, timeout, operation, expected):
env_vars = {"RUST_LOG": RUST_LOG}
r = run_baker("prebake", config_path, timeout=timeout, env=env_vars)
full_log = "[stdout]\n{}\n[stderr]\n{}".format(r["stdout"], r["stderr"])
return r, full_log
def _check(r, full_log, operation, expected):
if r["success"]:
return
diagnosis = ""
try:
from llm import judge_log
j = judge_log(operation, expected, full_log)
diagnosis = "\n LLM diagnosis: [{}] {}".format(j.verdict, j.reason)
except Exception:
pass
pytest.fail("{} failed ({} chars):\n{}\n{}".format(
operation, len(full_log), full_log, diagnosis))
# ── pip ─────────────────────────────────────────────────────────────
def test_pip_install_package(run_baker, work_dir):
"""Install a package via pip."""
cfg = os.path.join(work_dir, "python.yml")
with open(cfg, "w") as f:
f.write(_yml("""\
- type: pip
packages: [requests]
index: "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple"
args: ["--break-system-packages"]
"""))
r, log = _run_prebake(run_baker, cfg, timeout=120,
operation="pip install requests", expected="requests module importable")
_check(r, log, "pip install", "pip installs requests successfully")
r2 = _query(["python3", "-c", "import requests"])
assert r2["success"], f"import requests failed: {r2['stderr']}"
# ── pip + venv ──────────────────────────────────────────────────────
def test_pip_venv_install(run_baker, work_dir):
"""Create venv and install inside it."""
cfg = os.path.join(work_dir, "python.yml")
with open(cfg, "w") as f:
f.write(_yml("""\
- type: pip
directory: "/tmp/test-venv"
index: "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple"
packages: [requests]
"""))
r, log = _run_prebake(run_baker, cfg, timeout=120,
operation="venv + pip install",
expected="requests installed inside venv")
_check(r, log, "pip venv install", "venv created, requests installed")
r2 = _query(["/tmp/test-venv/bin/python3", "-c", "import requests"])
assert r2["success"], f"import from venv failed: {r2['stderr']}"
# ── pip + requirements ──────────────────────────────────────────────
def test_pip_requirements_file(run_baker, work_dir):
"""Install from a requirements.txt file."""
req_path = os.path.join(work_dir, "requirements.txt")
with open(req_path, "w") as f:
f.write("requests\n")
cfg = os.path.join(work_dir, "python.yml")
with open(cfg, "w") as f:
f.write(_yml("""\
- type: pip
directory: "/tmp/test-venv2"
index: "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple"
manifest: "{}"
""".format(req_path)))
r, log = _run_prebake(run_baker, cfg, timeout=120,
operation="pip install -r requirements.txt",
expected="requests installed from requirements file")
_check(r, log, "pip -r requirements.txt", "pip installs from requirements file")
r2 = _query(["/tmp/test-venv2/bin/python3", "-c", "import requests"])
assert r2["success"], f"import from venv failed: {r2['stderr']}"
# ── multiple actions ────────────────────────────────────────────────
def test_multiple_actions(run_baker, work_dir):
"""Two pip actions in one config."""
cfg = os.path.join(work_dir, "python.yml")
with open(cfg, "w") as f:
f.write(_yml("""\
- type: pip
directory: "/tmp/test-venv-a"
index: "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple"
packages: [requests]
args: ["--break-system-packages"]
- type: pip
directory: "/tmp/test-venv-b"
index: "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple"
packages: [pytest]
"""))
r, log = _run_prebake(run_baker, cfg, timeout=180,
operation="two pip actions",
expected="both venvs have their packages")
_check(r, log, "multiple pip actions", "both requests and pytest installed")
r2 = _query(["/tmp/test-venv-a/bin/python3", "-c", "import requests"])
assert r2["success"], f"venv-a missing requests: {r2['stderr']}"
r3 = _query(["/tmp/test-venv-b/bin/python3", "-c", "import pytest"])
assert r3["success"], f"venv-b missing pytest: {r3['stderr']}"
-275
View File
@@ -1,275 +0,0 @@
"""Rust UPM system tests driven by bare workshop-baker prebake.
These tests exercise the ``user.rust`` UPM through its full lifecycle:
1. sysdep injection (rustup via pacman, repology_endpoint: none)
2. rustup toolchain / component / target installation
3. cargo fetch (project dependency pre-fetch)
4. cargo install (global crate tools)
Observability
* ``RUST_LOG=info`` is injected into every baker invocation.
* On failure the *full* stdout + stderr is printed (no truncation).
* An LLM judge (``tests/llm/judge.py``) diagnoses failures with a
human-readable reason, cached per-log-hash to save tokens.
Network-dependent tests (toolchain download) are skipped by default.
Set ``RUST_UPM_NETWORK=1`` to run them.
"""
import os
import subprocess
import pytest
from probes import binary_exists
# ── helpers ─────────────────────────────────────────────────────────
RUST_LOG = os.environ.get("RUST_LOG", "info")
def _yml(user_section):
"""Build a minimal prebake.yml focused on the rust UPM."""
return """\
version: "1.0"
environment:
builder: "baremetal"
bootstrap:
user: "vulcan"
security:
drop_after: never
envvars:
prebake:
RUSTUP_DIST_SERVER: "https://mirrors.tuna.tsinghua.edu.cn/rustup"
dependencies:
config:
repology_endpoint: none
system:
pacman:
packages:
user:
rust:
""" + _indent(user_section, 6)
def _indent(text, spaces):
pad = " " * spaces
return "\n".join(pad + line if line.strip() else line
for line in text.splitlines())
def _cargo_toml(dir_path):
path = os.path.join(dir_path, "Cargo.toml")
with open(path, "w") as f:
f.write('[package]\nname = "test"\nversion = "0.1.0"\nedition = "2021"\n')
return path
def _query(cmd, timeout=60):
try:
r = subprocess.run(cmd, capture_output=True, text=True,
timeout=timeout, env={**os.environ, "HOME": "/root"})
return {"success": r.returncode == 0, "stdout": r.stdout,
"stderr": r.stderr}
except Exception as e:
return {"success": False, "stdout": "", "stderr": str(e)}
def _run_prebake(run_baker, config_path, timeout, operation, expected):
"""Run prebake with RUST_LOG, return (result, full_log)."""
env_vars = {"RUST_LOG": RUST_LOG}
r = run_baker("prebake", config_path, timeout=timeout, env=env_vars)
full_log = "[stdout]\n{}\n[stderr]\n{}".format(r["stdout"], r["stderr"])
return r, full_log
def _check(r, full_log, operation, expected):
"""Assert success; on failure call LLM judge and show full output."""
if r["success"]:
return
diagnosis = ""
try:
from llm import judge_log
j = judge_log(operation, expected, full_log)
diagnosis = "\n LLM diagnosis: [{}] {}".format(j.verdict, j.reason)
except Exception:
pass
pytest.fail(
"{} failed ({} chars):\n{}\n{}".format(
operation, len(full_log), full_log, diagnosis)
)
# ── system dependency resolution ────────────────────────────────────
def test_repology_none_config(run_baker, work_dir):
cfg = os.path.join(work_dir, "rust.yml")
with open(cfg, "w") as f:
f.write(_yml("""\
cargo: {}
"""))
r, log = _run_prebake(run_baker, cfg, timeout=120,
operation="UPM sysdep: repology_endpoint=none",
expected="rustup gets installed via pacman")
_check(r, log, "repology-none sysdep resolution",
"pacman installs rustup, empty cargo config")
assert binary_exists("/usr/bin/rustup"), "rustup not installed via sysdep"
# ── rustup: toolchain & components ──────────────────────────────────
@pytest.mark.skipif(
"RUST_UPM_NETWORK" not in os.environ,
reason="set RUST_UPM_NETWORK=1 for tests that download toolchain/crates"
)
def test_toolchain_and_component(run_baker, work_dir):
cfg = os.path.join(work_dir, "rust.yml")
with open(cfg, "w") as f:
f.write(_yml("""\
toolchain: "stable"
components: ["rustfmt"]
"""))
r, log = _run_prebake(run_baker, cfg, timeout=600,
operation="rustup default stable + component add rustfmt",
expected="stable toolchain active, rustfmt installed")
_check(r, log, "toolchain + rustfmt installation",
"rustup default stable succeeds; rustfmt available")
assert binary_exists("/usr/bin/rustup"), "rustup not installed"
r2 = _query(["rustup", "which", "rustc"])
assert r2["success"], "rustc not found via rustup which"
assert os.path.isfile(r2["stdout"].strip()), "rustc binary missing"
# ── cross-compilation targets ───────────────────────────────────────
@pytest.mark.skipif(
"RUST_UPM_NETWORK" not in os.environ,
reason="set RUST_UPM_NETWORK=1 for tests that download toolchain/crates"
)
def test_cross_compile_target(run_baker, work_dir):
cfg = os.path.join(work_dir, "rust.yml")
with open(cfg, "w") as f:
f.write(_yml("""\
toolchain: "stable"
targets: ["wasm32-unknown-unknown"]
"""))
r, log = _run_prebake(run_baker, cfg, timeout=600,
operation="rustup target add wasm32-unknown-unknown",
expected="wasm32 target appears in installed list")
_check(r, log, "cross-compilation target installation",
"wasm32-unknown-unknown is installed")
r2 = _query(["rustup", "target", "list", "--installed"])
assert r2["success"], "rustup target list failed"
assert "wasm32-unknown-unknown" in r2["stdout"], "wasm32 target missing"
# ── cargo fetch ─────────────────────────────────────────────────────
@pytest.mark.skipif(
"RUST_UPM_NETWORK" not in os.environ,
reason="set RUST_UPM_NETWORK=1 for tests that download toolchain/crates"
)
def test_cargo_fetch(run_baker, work_dir):
manifest = _cargo_toml(work_dir)
cfg = os.path.join(work_dir, "rust.yml")
with open(cfg, "w") as f:
f.write(_yml("""\
toolchain: "stable"
cargo:
fetch: true
locked: true
manifests: ["{manifest}"]
""".format(manifest=manifest)))
r, log = _run_prebake(run_baker, cfg, timeout=600,
operation="cargo fetch on minimal Cargo.toml",
expected="~/.cargo/registry/cache populated")
_check(r, log, "cargo fetch dependency pre-fetch",
"cargo fetch succeeds, registry cache non-empty")
cache = os.path.expanduser("~/.cargo/registry/cache")
assert os.path.isdir(cache), "registry/cache missing"
assert len(os.listdir(cache)) > 0, "registry/cache empty after fetch"
# ── cargo install ───────────────────────────────────────────────────
@pytest.mark.skipif(
"RUST_UPM_NETWORK" not in os.environ,
reason="set RUST_UPM_NETWORK=1 for tests that download toolchain/crates"
)
def test_cargo_install(run_baker, work_dir):
cfg = os.path.join(work_dir, "rust.yml")
with open(cfg, "w") as f:
f.write(_yml("""\
toolchain: "stable"
cargo:
packages: ["cargo-audit"]
locked: true
"""))
r, log = _run_prebake(run_baker, cfg, timeout=1200,
operation="cargo install cargo-audit",
expected="cargo-audit binary in ~/.cargo/bin/")
_check(r, log, "cargo install crate",
"cargo-audit compiled and installed")
assert binary_exists(
os.path.expanduser("~/.cargo/bin/cargo-audit")
), "cargo-audit not installed"
# ── full flow ───────────────────────────────────────────────────────
@pytest.mark.skipif(
"RUST_UPM_NETWORK" not in os.environ,
reason="set RUST_UPM_NETWORK=1 for tests that download toolchain/crates"
)
def test_full_flow(run_baker, work_dir):
manifest = _cargo_toml(work_dir)
cfg = os.path.join(work_dir, "rust.yml")
with open(cfg, "w") as f:
f.write(_yml("""\
toolchain: "stable"
components: ["rustfmt"]
targets: ["wasm32-unknown-unknown"]
cargo:
fetch: true
locked: true
manifests: ["{manifest}"]
packages: ["cargo-watch"]
""".format(manifest=manifest)))
r, log = _run_prebake(run_baker, cfg, timeout=1200,
operation="full Rust UPM flow",
expected="all stages succeed")
_check(r, log, "full Rust UPM end-to-end",
"toolchain + cross target + cargo fetch + cargo install all pass")
assert binary_exists("/usr/bin/rustup")
r2 = _query(["rustup", "which", "rustc"])
assert r2["success"]
r3 = _query(["rustup", "target", "list", "--installed"])
assert "wasm32-unknown-unknown" in r3["stdout"]
cache = os.path.expanduser("~/.cargo/registry/cache")
assert os.path.isdir(cache) and len(os.listdir(cache)) > 0
assert binary_exists(os.path.expanduser("~/.cargo/bin/cargo-watch"))
+107
View File
@@ -0,0 +1,107 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 4
[[package]]
name = "itoa"
version = "1.0.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
[[package]]
name = "memchr"
version = "2.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8ca58f447f06ed17d5fc4043ce1b10dd205e060fb3ce5b979b8ed8e59ff3f79"
[[package]]
name = "proc-macro2"
version = "1.0.106"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934"
dependencies = [
"unicode-ident",
]
[[package]]
name = "quote"
version = "1.0.45"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924"
dependencies = [
"proc-macro2",
]
[[package]]
name = "serde"
version = "1.0.228"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e"
dependencies = [
"serde_core",
"serde_derive",
]
[[package]]
name = "serde_core"
version = "1.0.228"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad"
dependencies = [
"serde_derive",
]
[[package]]
name = "serde_derive"
version = "1.0.228"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "serde_json"
version = "1.0.149"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "83fc039473c5595ace860d8c4fafa220ff474b3fc6bfdb4293327f1a37e94d86"
dependencies = [
"itoa",
"memchr",
"serde",
"serde_core",
"zmij",
]
[[package]]
name = "syn"
version = "2.0.117"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99"
dependencies = [
"proc-macro2",
"quote",
"unicode-ident",
]
[[package]]
name = "unicode-ident"
version = "1.0.24"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75"
[[package]]
name = "workshop-baker-params"
version = "0.1.0"
dependencies = [
"serde",
"serde_json",
]
[[package]]
name = "zmij"
version = "1.0.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa"
+8
View File
@@ -0,0 +1,8 @@
[package]
name = "workshop-baker-params"
version = "0.1.0"
edition = "2024"
[dependencies]
serde = { version = "1.0.228", features = ["derive"] }
serde_json = "1.0.149"
+24
View File
@@ -0,0 +1,24 @@
use crate::apply_field;
use serde::{Deserialize, Serialize};
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct BakeParams {
#[serde(default)] pub workspace: ParamVal<String>,
}
impl MergeParams for BakeParams {
fn defaults() -> Self { Self { workspace: ParamVal::new("/workspace".into()) } }
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
apply_field!(self.workspace, i.workspace, w, ov, p, "workspace");
}
}
impl Default for BakeParams { fn default() -> Self { Self::defaults() } }
#[derive(Debug, Clone)]
pub struct BakeSettings { pub workspace: String }
impl Default for BakeSettings { fn default() -> Self { BakeParams::defaults().into() } }
impl From<&BakeParams> for BakeSettings { fn from(p: &BakeParams) -> Self { Self { workspace: p.workspace.value.clone() } } }
impl From<BakeParams> for BakeSettings { fn from(p: BakeParams) -> Self { (&p).into() } }
+23
View File
@@ -0,0 +1,23 @@
use crate::apply_field;
use serde::{Deserialize, Serialize};
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct FinalizeParams {
#[serde(default)] pub artifact_retention_days: ParamVal<u32>,
}
impl MergeParams for FinalizeParams {
fn defaults() -> Self { Self { artifact_retention_days: ParamVal::new(7) } }
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
apply_field!(self.artifact_retention_days, i.artifact_retention_days, w, ov, p, "artifact_retention_days");
}
}
impl Default for FinalizeParams { fn default() -> Self { Self::defaults() } }
#[derive(Debug, Clone)]
pub struct FinalizeSettings { pub artifact_retention_days: u32 }
impl Default for FinalizeSettings { fn default() -> Self { FinalizeParams::defaults().into() } }
impl From<&FinalizeParams> for FinalizeSettings { fn from(p: &FinalizeParams) -> Self { Self { artifact_retention_days: p.artifact_retention_days.value } } }
impl From<FinalizeParams> for FinalizeSettings { fn from(p: FinalizeParams) -> Self { (&p).into() } }
+168
View File
@@ -0,0 +1,168 @@
pub mod notification;
pub mod prebake;
pub mod bake;
pub mod finalize;
pub mod writer;
pub mod param;
pub mod traits;
pub use notification::{NotificationParams, NotificationSettings};
pub use prebake::{PrebakeParams, PrebakeSettings};
pub use bake::{BakeParams, BakeSettings};
pub use finalize::{FinalizeParams, FinalizeSettings};
pub use writer::Writer;
pub use param::ParamVal;
pub use traits::MergeParams;
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
/// Root parameter tree. Each component gets its own subtree.
/// Used for both IPC (Serialized to JSON) and in-process merging.
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct Params {
#[serde(default, skip_serializing_if = "Option::is_none")]
pub notification: Option<NotificationParams>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub prebake: Option<PrebakeParams>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub bake: Option<BakeParams>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub finalize: Option<FinalizeParams>,
}
/// Apply and merge helpers.
impl Params {
/// Merge a layer from a specific writer into the current params.
/// Returns entries that were overridden (for debug).
pub fn apply(&mut self, incoming: Params, writer: Writer) -> Overridden {
let mut ov = Overridden::new();
if let Some(l) = incoming.notification {
if let Some(ref mut b) = self.notification {
b.apply(l, writer, &mut ov, "notification.");
} else {
self.notification = Some(l);
}
}
if let Some(l) = incoming.prebake {
if let Some(ref mut b) = self.prebake {
b.apply(l, writer, &mut ov, "prebake.");
} else {
self.prebake = Some(l);
}
}
if let Some(l) = incoming.bake {
if let Some(ref mut b) = self.bake {
b.apply(l, writer, &mut ov, "bake.");
} else {
self.bake = Some(l);
}
}
if let Some(l) = incoming.finalize {
if let Some(ref mut b) = self.finalize {
b.apply(l, writer, &mut ov, "finalize.");
} else {
self.finalize = Some(l);
}
}
ov
}
/// Convert to baker-consumable settings. All params guaranteed to have values.
pub fn to_settings(&self) -> Result<AllSettings, String> {
Ok(AllSettings {
notification: self.notification.as_ref().map(NotificationSettings::from).unwrap_or_default(),
prebake: self.prebake.as_ref().map(PrebakeSettings::from).unwrap_or_default(),
bake: self.bake.as_ref().map(BakeSettings::from).unwrap_or_default(),
finalize: self.finalize.as_ref().map(FinalizeSettings::from).unwrap_or_default(),
})
}
}
/// Plain-value settings for baker consumption. No metadata, no Option.
#[derive(Debug, Clone)]
pub struct AllSettings {
pub notification: NotificationSettings,
pub prebake: PrebakeSettings,
pub bake: BakeSettings,
pub finalize: FinalizeSettings,
}
/// Debug — values that were overridden during merge, keyed by dot-path.
#[derive(Debug, Clone, Default)]
pub struct Overridden {
pub entries: std::collections::HashMap<String, serde_json::Value>,
}
impl Overridden {
pub fn new() -> Self { Self { entries: HashMap::new() } }
pub fn is_empty(&self) -> bool { self.entries.is_empty() }
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_empty_params_serializes_empty() {
let p = Params::default();
let json = serde_json::to_string(&p).unwrap();
assert_eq!(json, "{}");
}
#[test]
fn test_notification_params_serialize() {
let mut p = Params::default();
p.notification = Some(NotificationParams::defaults());
let json = serde_json::to_string_pretty(&p).unwrap();
assert!(json.contains("max_lives"));
assert!(json.contains("3"));
}
#[test]
fn test_apply_courier_overrides_default() {
let mut base = Params::default();
base.notification = Some(NotificationParams::defaults());
let incoming = Params {
notification: Some(NotificationParams {
max_lives: ParamVal::with(5, Writer::Courier),
..Default::default()
}),
..Default::default()
};
let ov = base.apply(incoming, Writer::Courier);
let s = base.to_settings().unwrap();
assert_eq!(s.notification.max_lives, 5);
assert!(ov.is_empty()); // default had no writer, not an "override"
}
#[test]
fn test_apply_upstream_wins() {
let mut base = Params::default();
base.notification = Some(NotificationParams {
max_lives: ParamVal::with(3, Writer::Base),
..Default::default()
});
let incoming = Params {
notification: Some(NotificationParams {
max_lives: ParamVal::with(5, Writer::Courier),
..Default::default()
}),
..Default::default()
};
let ov = base.apply(incoming, Writer::Courier);
let s = base.to_settings().unwrap();
// base(3) > courier(2) — rejected, base stays
assert_eq!(s.notification.max_lives, 3);
assert!(ov.is_empty()); // nothing overridden
}
#[test]
fn test_deny_unknown_fields() {
let bad = r#"{"max_lives":{"value":5},"ghost_param":{"value":1}}"#;
let result: Result<NotificationParams, _> = serde_json::from_str(bad);
assert!(result.is_err()); // unknown field should be denied
}
}
+65
View File
@@ -0,0 +1,65 @@
use crate::apply_field;
use serde::{Deserialize, Serialize};
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct NotificationParams {
#[serde(default)] pub max_lives: ParamVal<u32>,
#[serde(default)] pub timeout_ms: ParamVal<u64>,
#[serde(default)] pub retry_backoff: ParamVal<f64>,
#[serde(default)] pub retry_delay_seconds: ParamVal<f64>,
#[serde(default)] pub max_retry_delay_seconds: ParamVal<f64>,
#[serde(default)] pub life_impact_factor: ParamVal<f64>,
#[serde(default)] pub batch_period: ParamVal<u32>,
#[serde(default)] pub batch_watermark: ParamVal<u32>,
#[serde(default)] pub template_ref_depth: ParamVal<u32>,
}
impl MergeParams for NotificationParams {
fn defaults() -> Self {
Self {
max_lives: ParamVal::new(3), timeout_ms: ParamVal::new(30000),
retry_backoff: ParamVal::new(2.0), retry_delay_seconds: ParamVal::new(5.0),
max_retry_delay_seconds: ParamVal::new(300.0),
life_impact_factor: ParamVal::new(0.0),
batch_period: ParamVal::new(30), batch_watermark: ParamVal::new(5),
template_ref_depth: ParamVal::new(10),
}
}
fn apply(&mut self, incoming: Self, writer: Writer, ov: &mut Overridden, prefix: &str) {
apply_field!(self.max_lives, incoming.max_lives, writer, ov, prefix, "max_lives");
apply_field!(self.timeout_ms, incoming.timeout_ms, writer, ov, prefix, "timeout_ms");
apply_field!(self.retry_backoff, incoming.retry_backoff, writer, ov, prefix, "retry_backoff");
apply_field!(self.retry_delay_seconds, incoming.retry_delay_seconds, writer, ov, prefix, "retry_delay_seconds");
apply_field!(self.max_retry_delay_seconds, incoming.max_retry_delay_seconds, writer, ov, prefix, "max_retry_delay_seconds");
apply_field!(self.life_impact_factor, incoming.life_impact_factor, writer, ov, prefix, "life_impact_factor");
apply_field!(self.batch_period, incoming.batch_period, writer, ov, prefix, "batch_period");
apply_field!(self.batch_watermark, incoming.batch_watermark, writer, ov, prefix, "batch_watermark");
apply_field!(self.template_ref_depth, incoming.template_ref_depth, writer, ov, prefix, "template_ref_depth");
}
}
impl Default for NotificationParams { fn default() -> Self { Self::defaults() } }
#[derive(Debug, Clone)]
pub struct NotificationSettings {
pub max_lives: u32, pub timeout_ms: u64, pub retry_backoff: f64,
pub retry_delay_seconds: f64, pub max_retry_delay_seconds: f64,
pub life_impact_factor: f64,
pub batch_period: u32, pub batch_watermark: u32, pub template_ref_depth: u32,
}
impl Default for NotificationSettings { fn default() -> Self { NotificationParams::defaults().into() } }
impl From<&NotificationParams> for NotificationSettings {
fn from(p: &NotificationParams) -> Self { Self {
max_lives: p.max_lives.value, timeout_ms: p.timeout_ms.value,
retry_backoff: p.retry_backoff.value, retry_delay_seconds: p.retry_delay_seconds.value,
max_retry_delay_seconds: p.max_retry_delay_seconds.value,
life_impact_factor: p.life_impact_factor.value,
batch_period: p.batch_period.value, batch_watermark: p.batch_watermark.value,
template_ref_depth: p.template_ref_depth.value,
}}
}
impl From<NotificationParams> for NotificationSettings { fn from(p: NotificationParams) -> Self { (&p).into() } }
+32
View File
@@ -0,0 +1,32 @@
use serde::{Deserialize, Serialize};
use crate::Writer;
/// A strongly-typed parameter value with writer tracking.
///
/// Serializes as `{ "value": 3, "writer": "base" }`.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ParamVal<T> {
pub value: T,
#[serde(skip_serializing_if = "Option::is_none")]
pub writer: Option<Writer>,
}
impl<T: Default> Default for ParamVal<T> {
fn default() -> Self {
ParamVal { value: T::default(), writer: None }
}
}
impl<T> ParamVal<T> {
pub fn new(value: T) -> Self {
ParamVal { value, writer: None }
}
pub fn with(value: T, writer: Writer) -> Self {
ParamVal { value, writer: Some(writer) }
}
/// Whether an incoming value from `incoming_writer` should override this one.
pub fn should_override(&self, incoming: Writer) -> bool {
let current = self.writer.map_or(0, |w| w.priority());
incoming.priority() > current
}
}
+54
View File
@@ -0,0 +1,54 @@
pub mod network;
pub mod cache;
pub mod engine;
use crate::apply_field;
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct PrebakeParams {
#[serde(default)] pub bootstrap_user: ParamVal<String>,
#[serde(default)] pub workspace: ParamVal<String>,
#[serde(default)] pub network: network::NetworkParams,
#[serde(default)] pub cache: cache::CacheParams,
#[serde(default)] pub engine: engine::EngineParams,
#[serde(default)] pub repology_endpoint: ParamVal<String>,
}
impl MergeParams for PrebakeParams {
fn defaults() -> Self {
Self {
bootstrap_user: ParamVal::new("vulcan".into()), workspace: ParamVal::new("/home/vulcan/workspace".into()),
network: network::NetworkParams::defaults(), cache: cache::CacheParams::defaults(),
engine: engine::EngineParams::defaults(),
repology_endpoint: ParamVal::new("default".into()),
}
}
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
apply_field!(self.bootstrap_user, i.bootstrap_user, w, ov, p, "bootstrap_user");
apply_field!(self.workspace, i.workspace, w, ov, p, "workspace");
apply_field!(self.repology_endpoint, i.repology_endpoint, w, ov, p, "repology_endpoint");
self.network.apply(i.network, w, ov, &format!("{}network.", p));
self.cache.apply(i.cache, w, ov, &format!("{}cache.", p));
self.engine.apply(i.engine, w, ov, &format!("{}engine.", p));
}
}
impl Default for PrebakeParams { fn default() -> Self { Self::defaults() } }
#[derive(Debug, Clone)]
pub struct PrebakeSettings {
pub bootstrap_user: String, pub workspace: String,
pub network: network::NetworkSettings, pub cache: cache::CacheSettings,
pub engine: engine::EngineSettings, pub repology_endpoint: String,
}
impl Default for PrebakeSettings { fn default() -> Self { PrebakeParams::defaults().into() } }
impl From<&PrebakeParams> for PrebakeSettings {
fn from(p: &PrebakeParams) -> Self { Self {
bootstrap_user: p.bootstrap_user.value.clone(), workspace: p.workspace.value.clone(),
network: (&p.network).into(), cache: (&p.cache).into(), engine: (&p.engine).into(),
repology_endpoint: p.repology_endpoint.value.clone(),
}}
}
impl From<PrebakeParams> for PrebakeSettings { fn from(p: PrebakeParams) -> Self { (&p).into() } }
@@ -0,0 +1,37 @@
use crate::apply_field;
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CacheParams {
#[serde(default)] pub strategy: ParamVal<String>,
#[serde(default)] pub mode: ParamVal<String>,
#[serde(default)] pub ttl_days: ParamVal<u32>,
#[serde(default)] pub max_size_gb: ParamVal<u32>,
#[serde(default)] pub cleanup_policy: ParamVal<String>,
}
impl MergeParams for CacheParams {
fn defaults() -> Self {
Self {
strategy: ParamVal::new("always".into()), mode: ParamVal::new("zstd".into()),
ttl_days: ParamVal::new(30), max_size_gb: ParamVal::new(20),
cleanup_policy: ParamVal::new("lru".into()),
}
}
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
apply_field!(self.strategy, i.strategy, w, ov, p, "strategy");
apply_field!(self.mode, i.mode, w, ov, p, "mode");
apply_field!(self.ttl_days, i.ttl_days, w, ov, p, "ttl_days");
apply_field!(self.max_size_gb, i.max_size_gb, w, ov, p, "max_size_gb");
apply_field!(self.cleanup_policy, i.cleanup_policy, w, ov, p, "cleanup_policy");
}
}
impl Default for CacheParams { fn default() -> Self { Self::defaults() } }
#[derive(Debug, Clone)]
pub struct CacheSettings { pub strategy: String, pub mode: String, pub ttl_days: u32, pub max_size_gb: u32, pub cleanup_policy: String }
impl Default for CacheSettings { fn default() -> Self { CacheParams::defaults().into() } }
impl From<&CacheParams> for CacheSettings { fn from(p: &CacheParams) -> Self {
Self { strategy: p.strategy.value.clone(), mode: p.mode.value.clone(), ttl_days: p.ttl_days.value, max_size_gb: p.max_size_gb.value, cleanup_policy: p.cleanup_policy.value.clone() }
} }
impl From<CacheParams> for CacheSettings { fn from(p: CacheParams) -> Self { (&p).into() } }
@@ -0,0 +1,21 @@
use crate::apply_field;
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct EngineParams {
#[serde(default)] pub timeout_ms: ParamVal<u64>,
}
impl MergeParams for EngineParams {
fn defaults() -> Self { Self { timeout_ms: ParamVal::new(300000) } }
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
apply_field!(self.timeout_ms, i.timeout_ms, w, ov, p, "timeout_ms");
}
}
impl Default for EngineParams { fn default() -> Self { Self::defaults() } }
#[derive(Debug, Clone)]
pub struct EngineSettings { pub timeout_ms: u64 }
impl Default for EngineSettings { fn default() -> Self { EngineParams::defaults().into() } }
impl From<&EngineParams> for EngineSettings { fn from(p: &EngineParams) -> Self { Self { timeout_ms: p.timeout_ms.value } } }
impl From<EngineParams> for EngineSettings { fn from(p: EngineParams) -> Self { (&p).into() } }
@@ -0,0 +1,23 @@
use crate::apply_field;
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct NetworkParams {
#[serde(default)] pub enabled: ParamVal<bool>,
#[serde(default)] pub outbound: ParamVal<bool>,
}
impl MergeParams for NetworkParams {
fn defaults() -> Self { Self { enabled: ParamVal::new(true), outbound: ParamVal::new(true) } }
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
apply_field!(self.enabled, i.enabled, w, ov, p, "enabled");
apply_field!(self.outbound, i.outbound, w, ov, p, "outbound");
}
}
impl Default for NetworkParams { fn default() -> Self { Self::defaults() } }
#[derive(Debug, Clone)]
pub struct NetworkSettings { pub enabled: bool, pub outbound: bool }
impl Default for NetworkSettings { fn default() -> Self { NetworkParams::defaults().into() } }
impl From<&NetworkParams> for NetworkSettings { fn from(p: &NetworkParams) -> Self { Self { enabled: p.enabled.value, outbound: p.outbound.value } } }
impl From<NetworkParams> for NetworkSettings { fn from(p: NetworkParams) -> Self { (&p).into() } }
+28
View File
@@ -0,0 +1,28 @@
use crate::{Writer, Overridden};
/// A parameter tree node that can merge itself with an incoming layer.
pub trait MergeParams: Sized {
fn apply(&mut self, incoming: Self, writer: Writer, ov: &mut Overridden, prefix: &str);
fn defaults() -> Self;
}
/// Apply one field's merge logic.
/// ```ignore
/// apply_field!(self.foo, incoming.foo, writer, ov, prefix, "foo");
/// ```
#[macro_export]
macro_rules! apply_field {
($self:ident.$field:ident, $incoming:ident.$field2:ident, $writer:expr, $ov:ident, $prefix:expr, $name:expr) => {
let old_w = $self.$field.writer;
if $self.$field.should_override($writer) {
let old_v = ::std::mem::replace(&mut $self.$field.value, $incoming.$field2.value);
$self.$field.writer = ::std::option::Option::Some($writer);
if let ::std::option::Option::Some(w) = old_w {
$ov.entries.insert(
format!("{}{}", $prefix, $name),
::serde_json::json!({"value": old_v, "writer": w.as_str()}),
);
}
}
};
}
+26
View File
@@ -0,0 +1,26 @@
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum Writer {
Base,
Courier,
Bakerd,
}
impl Writer {
/// Priority: Base=3 > Courier=2 > Bakerd=1
pub fn priority(self) -> u8 {
match self {
Writer::Base => 3,
Writer::Courier => 2,
Writer::Bakerd => 1,
}
}
pub fn as_str(self) -> &'static str {
match self {
Writer::Base => "base", Writer::Courier => "courier", Writer::Bakerd => "bakerd",
}
}
}
impl Default for Writer { fn default() -> Self { Writer::Base } }
+1
View File
@@ -1,3 +1,4 @@
temp temp
examples examples
quicktest.sh quicktest.sh
_env.sh
+84 -1471
View File
File diff suppressed because it is too large Load Diff
+2 -8
View File
@@ -47,13 +47,10 @@ uuid = { version = "1.19.0", features = ["v4"] }
libc = "0.2" libc = "0.2"
os_info = "3.14.0" os_info = "3.14.0"
privdrop = "0.5.6" privdrop = "0.5.6"
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
which = "8.0.2" which = "8.0.2"
gix = "0.81.0"
sha2 = "0.10"
flate2 = "1.0"
zstd = "0.13"
globset = "0.4.18" globset = "0.4.18"
axum = "0.7"
workshop-baker-params = { version = "0.1.0", path = "../workshop-baker-params" }
[dev-dependencies] [dev-dependencies]
criterion = "0.5" criterion = "0.5"
@@ -66,6 +63,3 @@ unreachable_code = "allow"
inherent_to_string = "allow" inherent_to_string = "allow"
non_canonical_partial_ord_impl = "allow" non_canonical_partial_ord_impl = "allow"
[[bench]]
name = "parser_bench"
harness = false
-93
View File
@@ -1,93 +0,0 @@
#!/usr/bin/env python3
import os
import random
import string
OUTPUT_DIR = "/tmp"
DECORATORS = [
"# @pipeline",
"# @fallible",
"# @parallel",
"# @export",
"# @timeout(60)",
"# @retry(3, 5)",
"# @if(condition)",
"# @after(dep)",
"# @pipe(step1, step2)",
"# @health(/health)",
"# @loop(10)",
]
FUNCTION_BODIES = [
"echo 'Processing...';",
"local var=$(date +%s);",
"if [ -f /tmp/test ]; then echo 'exists'; fi;",
"for i in $(seq 1 10); do echo $i; done;",
"case $1 in start) echo 'starting';; stop) echo 'stopping';; esac;",
"read -r line < /dev/stdin;",
"export PATH=$PATH:/usr/local/bin;",
"set -e; set -u;",
"trap 'echo error' ERR;",
"cd /tmp || exit 1;",
"local result=$(grep -r 'pattern' /var/log/ 2>/dev/null);",
'for f in /tmp/*.tmp; do rm -f "$f"; done',
"if [[ $var -gt 100 ]]; then echo 'large'; fi;",
'while read -r line; do echo "$line"; done < /etc/passwd;',
'select opt in a b c; do echo "Selected $opt"; break; done;',
]
def random_string(length=8):
return "".join(random.choices(string.ascii_letters + string.digits, k=length))
def generate_function(body_lines):
lines = []
func_name = f"func_{random_string(6)}"
for _ in range(random.randint(0, 3)):
lines.append(random.choice(DECORATORS))
lines.append(f"{func_name}() {{")
for _ in range(body_lines):
lines.append(" " + random.choice(FUNCTION_BODIES))
lines.append("}")
lines.append("")
return lines
def generate_script(target_lines, num_functions, target_bytes):
lines = []
total_lines = 0
body_per_func = max(1, (target_lines - num_functions * 3) // num_functions)
for _ in range(num_functions):
func_lines = generate_function(body_per_func)
lines.extend(func_lines)
total_lines += len(func_lines)
while (
total_lines < target_lines
or len("\n".join(lines).encode("utf-8")) < target_bytes
):
lines.append(random.choice(FUNCTION_BODIES))
total_lines += 1
return "\n".join(lines)
def main():
sizes = [
("small", 1000, 10, 50_000),
("medium", 5000, 50, 500_000),
("large", 10000, 100, 950_000),
]
for name, lines, funcs, target_bytes in sizes:
script = generate_script(lines, funcs, target_bytes)
filepath = os.path.join(OUTPUT_DIR, f"bench_{name}.sh")
with open(filepath, "w") as f:
f.write(script)
actual_bytes = len(script.encode("utf-8"))
actual_lines = len(script.splitlines())
print(f"Generated {filepath}: lines={actual_lines}, bytes={actual_bytes}")
if __name__ == "__main__":
main()
-61
View File
@@ -1,61 +0,0 @@
use criterion::{Criterion, black_box, criterion_group, criterion_main};
use std::fs;
use std::path::Path;
fn parse_large_script(c: &mut Criterion) {
let script_path = Path::new("/tmp/bench_large.sh");
let script_content = fs::read_to_string(script_path)
.expect("Failed to read benchmark script. Run `python3 benches/generate.py` first.");
let byte_size = script_content.len();
let line_count = script_content.lines().count();
println!("Benchmark file: {} lines, {} bytes", line_count, byte_size);
c.bench_function("parse_script_large", |b| {
b.iter(|| {
let result =
workshop_baker::bake::parser::parse_script(black_box(&script_content.clone()));
black_box(result)
});
});
}
fn parse_medium_script(c: &mut Criterion) {
let script_path = Path::new("/tmp/bench_medium.sh");
let script_content = fs::read_to_string(script_path)
.expect("Failed to read benchmark script. Run `python3 benches/generate.py` first.");
c.bench_function("parse_script_medium", |b| {
b.iter(|| {
let result =
workshop_baker::bake::parser::parse_script(black_box(&script_content.clone()));
black_box(result)
});
});
}
fn parse_small_script(c: &mut Criterion) {
let script_path = Path::new("/tmp/bench_small.sh");
let script_content = fs::read_to_string(script_path)
.expect("Failed to read benchmark script. Run `python3 benches/generate.py` first.");
c.bench_function("parse_script_small", |b| {
b.iter(|| {
let result =
workshop_baker::bake::parser::parse_script(black_box(&script_content.clone()));
black_box(result)
});
});
}
criterion_group!(
benches,
parse_small_script,
parse_medium_script,
parse_large_script
);
criterion_main!(benches);
+42 -47
View File
@@ -1,62 +1,57 @@
# workshop-baker/src # workshop-baker/src
**Generated:** 2026-04-22 **Generated:** 2026-05-19
**Commit:** 7b3b71a **Commit:** 28abc5d
Baker CLI + execution engine for HoneyBiscuitWorkshop pipelines. CLI tool and runtime for executing CI/CD pipelines. Orchestrates prebake → bake → finalize stages with notification dispatch.
## OVERVIEW
CLI tool and runtime for executing CI/CD pipelines with resource isolation, privilege dropping, and package management integration.
## STRUCTURE ## STRUCTURE
``` ```
src/ src/
├── bake.rs # Top-level bake orchestration ├── lib.rs # Module declarations + Cli struct + Commands enum
├── bake/ # Script parsing and building ├── main.rs # tokio::main entry point
├── parser.rs # @decorator-aware shell parser ├── bare.rs # Bare mode (run individual stages)
│ ├── decorator.rs # @pipeline, @retry, @parallel, etc. ├── error.rs # CliError (wraps stage errors)
├── builder.rs # Script template builder ├── bake.rs # Bake orchestration (script → execution)
│ └── schedule.rs # Function ordering via @after deps ├── bake/ # Script parsing, decorator DSL, scheduling
├── engine/ # Execution runtime ├── prebake.rs # Prebake orchestration (env setup)
│ ├── executor.rs # Script execution with cgroups ├── prebake/ # Config, stages, security, env providers
│ ├── cgroups.rs # Linux cgroup resource limits ├── finalize.rs # Finalize orchestration (post-build)
│ ├── repology.rs # Package name resolution ├── finalize/ # Config, plugins, hooks, templates
│ ├── pm.rs # Package manager detection ├── notify.rs # Notification system (batching, retry)
│ └── upm.rs # User package manager (Nix/Guix) ├── notify/ # Types, templates, interactive, queue
├── prebake/ # Build environment setup ├── daemon/ # Empty (Daemon command = unimplemented!)
├── config.rs # PrebakeConfig YAML schema ├── monitor/ # Resource usage monitoring (cgroups)
│ ├── stage/ # Bootstrap, DepsSystem, DepsUser, Hooks ├── types.rs # Module re-exports
│ └── security.rs # Privilege dropping logic ├── types/ # buildstatus, resource types
├── types/ # Shared type definitions ├── utils.rs # Module root (fetch submodule)
│ ├── repology.rs # RepologyEndpoint enum └── utils/ # fetch.rs
│ ├── builderconfig.rs
│ ├── cache.rs
│ └── memsize.rs
├── monitor/ # Resource usage monitoring
│ ├── cgroups.rs # cgroup-based usage collection
│ └── jobobject.rs # Windows job objects (stub)
├── daemon.rs # Unix socket daemon (TODO: refactor)
└── lib.rs # CLI struct + module exports
``` ```
## WHERE TO LOOK ## WHERE TO LOOK
| Task | Location | | Task | Location |
|------|----------| |------|----------|
| Script parsing with @decorators | `bake/parser.rs:29` - `parse_script()` | | CLI args | `lib.rs:21` - `Cli` struct |
| Decorator types | `bake/decorator.rs` - `Decorator` enum | | Entry point | `main.rs:91` - `#[tokio::main]` |
| Build execution | `engine/executor.rs` - `Executor::execute_script()` | | Pipeline orchestration | `main.rs:68-74` - prebake→bake→finalize chain |
| Resource limits | `engine/types.rs:56` - `ResourceLimits` struct | | Bare mode (single stage) | `bare.rs:14` - `run_bare()` |
| Prebake stages | `prebake/stage/` subdirectories | | Script parsing | `bake/parser.rs:56` - `parse_script()` |
| Package detection | `engine/pm.rs:detection()` | | Decorator DSL | `bake/decorator.rs` - `Decorator` enum |
| CLI entry | `lib.rs:26` - `cli::Cli` struct | | Prebake stages | `prebake/stage/` - Bootstrap→DepsSystem→DepsUser→Hooks→Ready |
| Plugin system | `finalize/plugin.rs` + `finalize/plugin/` |
| Notification dispatch | `notify.rs:66` - `notify()` async loop |
| Notification types | `notify/types.rs` - `NotificationRule`, `ParsedTrigger` |
## CONVENTIONS ## CONVENTIONS
- **ExecutionContext**: Passed through all stages, carries task_id, username, env_vars, timeout - **ExecutionContext**: Passed through all stages, carries task_id, username, env_vars, timeout
- **PrebakeStage**: Ordered enum controlling stage execution sequence (Bootstrap -> Ready) - **PrebakeStage**: Ordered enum Bootstrap→EarlyHook→DepsSystem→DepsUser→LateHook→Ready
- **Decorator**: Attached to shell functions via `# @name(args)` comments above function declarations - **Decorator**: `# @name(args)` comments above shell function declarations
- **Engine**: Lightweight wrapper around cgroup manager for resource isolation - **Notification**: Plugin-based dispatch with batching, exponential backoff, priority groups
- **Dual-mode**: CLI commands (prebake, bake, finalize) vs daemon mode via Unix socket - **Engine**: Delegated to `workshop-engine` crate (formerly in-tree)
- **Dual-mode**: CLI commands (standalone) vs Daemon mode (unimplemented)
## ANTI-PATTERNS
1. **Empty daemon/** directory — Daemon command is `unimplemented!()`
2. **Dead code allowed**`lints.rust.dead_code = "allow"` in Cargo.toml
3. **Engine was extracted** — workshop-engine is its own crate now, src/engine/ removed
4. **Python test infra exists but files removed** — pytest.ini remains, test dir is empty
+4 -2
View File
@@ -12,14 +12,16 @@ bake/
├── parser.rs # @decorator-aware shell script parser ├── parser.rs # @decorator-aware shell script parser
├── decorator.rs # Decorator enum + 100+ unit tests ├── decorator.rs # Decorator enum + 100+ unit tests
├── builder.rs # Script template assembly ├── builder.rs # Script template assembly
── schedule.rs # Function ordering via @after dependencies ── schedule.rs # Function ordering via @after dependencies
├── error.rs # BakeError enum
└── constant.rs # DEFAULT_WORKSPACE, TRIVIAL_SCRIPT_NAME
``` ```
## WHERE TO LOOK ## WHERE TO LOOK
| Task | Location | | Task | Location |
|------|----------| |------|----------|
| Parse script | `parser.rs:29` - `parse_script()` | Parse script | `parser.rs:56` - `parse_script()` |
| Decorator types | `decorator.rs` - `Decorator` enum variants | | Decorator types | `decorator.rs` - `Decorator` enum variants |
| Build script | `builder.rs` - template builder | | Build script | `builder.rs` - template builder |
| Schedule deps | `schedule.rs` - @after resolution | | Schedule deps | `schedule.rs` - @after resolution |
+50
View File
@@ -0,0 +1,50 @@
use std::path::PathBuf;
use crate::cli::{BareCommands, Cli};
use crate::error::CliError;
use crate::types::resource::ResourceRegistry;
use crate::{bake::bake, finalize::finalize, prebake::prebake};
use workshop_engine::{EventSender, ExecutionContext};
fn or_workshop(path: &Option<PathBuf>, name: &str) -> PathBuf {
path
.clone()
.unwrap_or_else(|| PathBuf::from(format!(".workshop/{}", name)))
}
pub async fn run_bare(
cli: &Cli,
command: &BareCommands,
ctx: &mut ExecutionContext,
event_tx: EventSender,
pipeline: &str,
build_id: &str,
registry: &ResourceRegistry,
) -> Result<(), CliError> {
let (stage, config) = match command {
BareCommands::Prebake { .. } => ("prebake", or_workshop(&cli.prebake, "prebake.yml")),
BareCommands::Bake { .. } => ("bake", or_workshop(&cli.bake, "bake.sh")),
BareCommands::Finalize { .. } => {
("finalize", or_workshop(&cli.finalize, "finalize.yml"))
}
};
let prebake_path = or_workshop(&cli.prebake, "prebake.yml");
let bake_base = or_workshop(&cli.bake_base, "bake_base.sh");
ctx.task_id = format!("{}-{}-{}", pipeline, build_id, stage);
ctx.pipeline_name = pipeline.to_string();
ctx.standalone = true;
match command {
BareCommands::Prebake { .. } => {
prebake(&config, cli, None, ctx, event_tx, None).await?;
}
BareCommands::Bake { .. } => {
bake(&config, &bake_base, &prebake_path, cli, ctx, event_tx).await?;
}
BareCommands::Finalize { .. } => {
finalize(&config, cli, ctx, event_tx, registry).await?;
}
}
Ok(())
}
-193
View File
@@ -1,193 +0,0 @@
use crate::cli::Cli;
// use serde::Serialize;
use serde_json::Value;
// use std::path::PathBuf;
// use std::sync::Arc;
use tokio::io::{AsyncBufReadExt, BufReader};
use tokio::net::UnixStream;
// use tokio::process::{Child, Command};
// use tokio::signal;
// use tokio::sync::RwLock;
// use tokio::time::{Duration, interval};
// #[derive(Debug, Clone, Serialize)]
// struct Heartbeat {
// pipeline_id: String,
// build_id: String,
// stage: String,
// timestamp: u64,
// status: String,
// subtasks: Vec<String>,
// }
pub async fn daemon(_cli: &Cli) {
todo!("Daemon should be refactored");
// let socket = PathBuf::from(settings.get_string("socket").unwrap());
// let agentsocket = settings.get_string("agentsocket").unwrap();
// let _ = std::fs::remove_file(&socket);
// let heartbeat = Arc::new(RwLock::new(Heartbeat {
// pipeline_id: cli.pipeline.clone(),
// build_id: cli.build_id.clone(),
// stage: String::new(),
// timestamp: 0,
// status: String::new(),
// subtasks: Vec::new(),
// }));
// // let heartbeat_clone = heartbeat.clone();
// let heartbeat_task = {
// let hb = heartbeat.clone();
// tokio::spawn(async move {
// let mut tick = interval(Duration::from_secs(1));
// loop {
// tick.tick().await;
// let mut hb = hb.write().await;
// hb.timestamp = std::time::SystemTime::now()
// .duration_since(std::time::UNIX_EPOCH)
// .unwrap()
// .as_millis() as u64;
// // 只输出到stdout(模拟未来发送给Agent
// println!("[HEARTBEAT] {}", serde_json::to_string(&*hb).unwrap());
// }
// })
// };
// let socket_task = {
// let socket = socket.clone();
// tokio::spawn(async move {
// log::info!("Creating Unix socket at {}", socket.display());
// let listener =
// UnixListener::bind(&socket).expect("Failed to bind Unix socket");
// loop {
// match listener.accept().await {
// Ok((stream, _)) => {
// tokio::spawn(handle_connection(stream));
// }
// Err(e) => {
// log::error!("Accept error: {}", e);
// break;
// }
// }
// }
// })
// };
// let mut tasks: Vec<TaskSpec> = Vec::new();
// let execution_task = {
// let hb = heartbeat.clone();
// tokio::spawn(async move {
// let mut hb = hb.write().await;
// let mut children: Vec<(String, Child)> = vec![];
// let executable = std::env::current_exe().unwrap();
// for (index, task) in tasks.iter().enumerate() {
// let mut command = task.command(&executable);
// command.stdout(std::process::Stdio::piped());
// command.stderr(std::process::Stdio::piped());
// let child = command
// .spawn()
// .expect(format!("Failed to spawn child {}", index).as_str());
// let pid = child.id().unwrap_or(0).to_string();
// children.push((pid.clone(), child));
// hb.subtasks.push(pid.clone());
// }
// for (pid, mut child) in children {
// let status = child.wait().await.expect("Failed to wait child");
// log::info!("Subtask {} exited with: {}", pid, status);
// hb.subtasks.retain(|p| p != &pid);
// }
// hb.stage = "Success".to_string();
// })
// };
// tokio::select! {
// _ = heartbeat_task => {
// log::error!("Heartbeat task died");
// }
// _ = execution_task => {
// log::info!("All subtasks completed");
// }
// _ = socket_task => {
// log::error!("Socket task died");
// }
// _ = signal::ctrl_c() => {
// log::info!("Shutting down...");
// }
// }
// for pid in heartbeat.read().await.subtasks.clone() {
// let _ = Command::new("kill").arg("-TERM").arg(&pid).status().await;
// }
// let _ = std::fs::remove_file(&socket);
}
async fn _handle_connection(stream: UnixStream) {
let peer_addr = match stream.peer_addr() {
Ok(addr) => format!("{:?}", addr),
Err(_) => "unknown".to_string(),
};
log::debug!("New connection from: {}", peer_addr);
let mut reader = BufReader::new(stream);
let mut line = String::new();
loop {
line.clear();
match reader.read_line(&mut line).await {
Ok(0) => {
log::debug!("Connection closed by peer: {}", peer_addr);
break;
}
Ok(_) => {
// 简单打印原始JSON(预留字段解析空间)
match serde_json::from_str::<Value>(&line) {
Ok(json) => {
// 只提取关键字段打印(字段可能变化)
let msgtype = json
.get("msgtype")
.and_then(|v| v.as_str())
.unwrap_or("unknown");
let name = json
.get("name")
.and_then(|v| v.as_str())
.unwrap_or("unnamed");
log::info!("[{}] Message from '{}': {}", msgtype, name, line.trim());
// TODO: 字段稳定后,可扩展为结构化打印
// match msgtype {
// "ResourceUsage" => print_resource(&json),
// "Log" => print_log(&json),
// _ => log::debug!("Unknown type: {}", msgtype),
// }
}
Err(e) => {
log::warn!(
"Invalid JSON from {}: {} - Error: {}",
peer_addr,
line.trim(),
e
);
}
}
}
Err(e) => {
log::error!("Failed to read from {}: {}", peer_addr, e);
break;
}
}
}
log::debug!("Connection handler for {} exiting", peer_addr);
}
+30
View File
@@ -0,0 +1,30 @@
use thiserror::Error;
use workshop_engine::HasExitCode;
#[derive(Error, Debug)]
pub enum CliError {
#[error("{0}")]
Prebake(#[from] crate::prebake::error::PrebakeError),
#[error("{0}")]
Bake(#[from] crate::bake::error::BakeError),
#[error("{0}")]
Finalize(#[from] crate::finalize::FinalizeError),
#[error("{0}")]
General(anyhow::Error),
}
impl From<anyhow::Error> for CliError {
fn from(e: anyhow::Error) -> Self {
CliError::General(e)
}
}
impl CliError {
pub fn exit_code(&self) -> i32 {
match self {
CliError::Prebake(e) => e.exit_code(),
CliError::Bake(e) => e.exit_code(),
CliError::Finalize(e) => e.exit_code(),
CliError::General(_) => 1,
}
}
}
+4 -28
View File
@@ -8,12 +8,10 @@ pub mod template;
pub mod types; pub mod types;
use crate::cli::Cli; use crate::cli::Cli;
use crate::finalize::constant::FINALIZE_STANDALONE_PLUGIN_PATH; use crate::types::resource::ResourceRegistry;
use crate::finalize::plugin::fetch::{self, FetchArgument};
pub use config::*; pub use config::*;
pub use error::FinalizeError; pub use error::FinalizeError;
use std::path::Path; use std::path::Path;
use std::path::PathBuf;
use workshop_engine::EventSender; use workshop_engine::EventSender;
pub fn parse(path: &Path) -> Result<FinalizeConfig, FinalizeError> { pub fn parse(path: &Path) -> Result<FinalizeConfig, FinalizeError> {
@@ -27,6 +25,7 @@ pub async fn finalize(
_cli: &Cli, _cli: &Cli,
ctx: &mut workshop_engine::ExecutionContext, ctx: &mut workshop_engine::ExecutionContext,
event_tx: EventSender, event_tx: EventSender,
registry: &ResourceRegistry,
) -> Result<(), FinalizeError> { ) -> Result<(), FinalizeError> {
let mut finalize = parse(finalize_path)?; let mut finalize = parse(finalize_path)?;
let validate_result = finalize.validate(); let validate_result = finalize.validate();
@@ -36,33 +35,10 @@ pub async fn finalize(
} }
return Err(FinalizeError::ValidateError(errors)); return Err(FinalizeError::ValidateError(errors));
}; };
if ctx.standalone {
// In standalone mode, finalize should download plugin by itself
log::debug!("Downloading {} plugins", finalize.plugin.len());
let plugin_path = PathBuf::from(FINALIZE_STANDALONE_PLUGIN_PATH);
// if plugin_path.exists() {
// std::fs::remove_dir_all(&plugin_path).map_err(FinalizeError::IoError)?;
// }
for (name, data) in finalize.plugin.iter_mut() {
log::info!("Downloading plugin {}", name);
let (source, config) = data.iter_mut().next().unwrap_or_else(|| {
panic!(
"Internal Error: Plugin {} has no source configured, this should have been caught by validation!",
name
)
});
let fetch_argument = FetchArgument {
checksum: config.checksum.clone(),
shallow: config.shallow,
};
fetch::fetch_plugin(source, name, &plugin_path, fetch_argument).await?;
config.runtime.fspath = Some(plugin_path.join(name));
}
}
// Register Plugins // Register Plugins (paths resolved from registry)
log::info!("Registering {} plugins", finalize.plugin.len()); log::info!("Registering {} plugins", finalize.plugin.len());
let registered_plugins = plugin::register(finalize.plugin, None)?; let registered_plugins = plugin::register(finalize.plugin, None, &mut finalize.notification, registry)?;
// Early hook // Early hook
let earlyhook_result = stage::hook::hook( let earlyhook_result = stage::hook::hook(
+28 -32
View File
@@ -1,56 +1,52 @@
# workshop-baker/src/finalize # workshop-baker/src/finalize
**Generated:** 2026-04-22 **Generated:** 2026-05-19
**Commit:** 7b3b71a **Commit:** 28abc5d
Artifact packaging, distribution, deployment, and notifications. Artifact packaging, plugin-based notifications, and post-build hooks.
## STRUCTURE ## STRUCTURE
``` ```
finalize/ finalize/
├── config.rs # FinalizeConfig YAML (artifacts, deploy, notify) ├── config.rs # FinalizeConfig YAML (379 lines, plugins, notifications)
├── stage.rs # FinalizeStage enum ├── stage.rs # FinalizeStage enum
├── stage/hook.rs # Post-build hooks ├── stage/hook.rs # Post-build hook execution
├── event.rs # Event propagation ├── event.rs # Event propagation
├── template.rs # Variable substitution {{VAR}} ├── template.rs # Variable substitution {{VARIABLE}}
├── plugin.rs # Plugin trait + registry ├── plugin.rs # Plugin trait + registration
├── plugin/ ├── plugin/
│ ├── metadata.rs # Plugin manifest parsing │ ├── metadata.rs # Plugin manifest parsing
│ ├── dylib.rs # Dynamic library plugins │ ├── dylib.rs # Dynamic library plugins
│ ├── rhai.rs # Rhai scripting (todo!()) │ ├── rhai.rs # Rhai scripting (todo!())
│ ├── shell.rs # Shell command plugins │ ├── shell.rs # Shell command plugins
│ └── internal/ # Built-in plugins │ └── internal.rs + internal/ # Built-in plugins (mail, webhook, satori, insitenotify)
│ ├── mail/ # Email notifications (SMTP) ├── types.rs
│ ├── webhook.rs # HTTP webhooks ├── types/compression.rs
├── satori.rs # Satori integration ├── error.rs # FinalizeError
│ └── insitenotify.rs # In-site notifications └── constant.rs
│ └── fetch/ # Artifact fetching
│ ├── git.rs # Git repository fetch
│ ├── http.rs # HTTP download
│ ├── extract.rs # Archive extraction
│ └── checksum.rs# SHA256 verification (SHA1 deprecated)
``` ```
## WHERE TO LOOK ## WHERE TO LOOK
| Task | Location | | Task | Location |
|------|----------| |------|----------|
| Config loading | `config.rs` - FinalizeConfig | | Config loading | `config.rs:17` - `FinalizeConfig` + `parse()` |
| Email notify | `plugin/internal/mail/` - SMTP templates | | Plugin registration | `plugin.rs` - `register()` |
| Email notifications | `plugin/internal/mail.rs` - SMTP |
| Webhooks | `plugin/internal/webhook.rs` | | Webhooks | `plugin/internal/webhook.rs` |
| Artifact fetch | `plugin/fetch/` - git/http/extract | | Satori integration | `plugin/internal/satori.rs` |
| Checksums | `plugin/fetch/checksum.rs:13` - SHA1 deprecated | | Shell plugins | `plugin/shell.rs` |
| Template rendering | `template.rs` - `{{VARIABLE}}` handlebars-style |
## CONVENTIONS ## CONVENTIONS
- **Plugin System**: `FinalizePlugin` trait + registry with both internal and dynamic plugins
- **Plugin System**: Dynamic + internal plugins via Plugin trait - **Template Syntax**: `{{VARIABLE}}` handlebars-style substitution via `template.rs`
- **Template Syntax**: `{{VARIABLE}}` handlebars-style - **Stages**: EarlyHook → (Artifacts) → LateHook → Notifications
- **Stages**: Hook → Fetch → Build → Publish → Notify → Cleanup - **Notification**: `notify.rs` in parent module drives plugin-based notification dispatch
- **Fetch**: Supports git, http, with extraction and checksum verify - **ResourceRegistry**: Shared resource cache passed through to plugins
## ANTI-PATTERNS ## ANTI-PATTERNS
1. **fetch/ subdirectory removed** — was `plugin/fetch/{git,http,extract,checksum}`, no longer exists
1. **SHA1 deprecated**`checksum.rs:13` warns SHA1 is deprecated 2. **SHA1 deprecated**workshop-getterurl handles checksums now
2. **Unimplemented**`rhai.rs:17` - "todo!()" for Rhai scripting 3. **Unimplemented**`rhai.rs:17` - "todo!()" for Rhai scripting
3. **TODO markers** — Multiple TODOs in config.rs (lines 305-306) 4. **TODO markers** — Multiple TODOs in config.rs (lines 305-306)
5. **Artifact stage not implemented**`finalize.rs:57` — "TODO: artifact stage implementation"
+112 -9
View File
@@ -115,6 +115,17 @@ pub struct NotificationMethod {
pub policy: Vec<NotifyPolicy>, pub policy: Vec<NotifyPolicy>,
} }
impl Default for NotificationMethod {
fn default() -> Self {
Self {
template: NotificationTemplate::default(),
trigger: Vec::new(),
config: Value::default(),
policy: Vec::new(),
}
}
}
#[derive(Debug, Deserialize, Serialize, Clone, Default)] #[derive(Debug, Deserialize, Serialize, Clone, Default)]
pub struct NotificationTemplate { pub struct NotificationTemplate {
/// Payload schema: "default" (plugin-defined) or "custom" (requires on_* templates) /// Payload schema: "default" (plugin-defined) or "custom" (requires on_* templates)
@@ -134,17 +145,59 @@ pub struct NotificationTemplate {
pub on_finish_of: Option<OnFinishOf>, pub on_finish_of: Option<OnFinishOf>,
} }
impl std::ops::AddAssign for NotificationTemplate {
fn add_assign(&mut self, other: Self) {
if self.schema.is_none() {
self.schema = other.schema;
}
if self.on_success.is_none() {
self.on_success = other.on_success;
}
if self.on_failure.is_none() {
self.on_failure = other.on_failure;
}
if self.on_finish_of.is_none() {
self.on_finish_of = other.on_finish_of;
}
}
}
/// Notification configuration container /// Notification configuration container
#[derive(Debug, Deserialize, Serialize, Clone, Default)] #[derive(Debug, Serialize, Clone, Default)]
pub struct NotificationConfig { pub struct NotificationConfig {
/// Template definitions for notification content /// Template definitions for notification content
#[serde(default)] #[serde(default)]
pub templates: HashMap<String, NotificationTemplateDef>, pub templates: HashMap<String, NotificationTemplateDef>,
#[serde(flatten)] /// Priority groups (YAML keys like `0`, `1` are parsed as u32).
/// Custom Deserialize handles serde_yaml string→u32 conversion.
#[serde(default)]
pub groups: HashMap<u32, HashMap<String, NotificationMethod>>, pub groups: HashMap<u32, HashMap<String, NotificationMethod>>,
} }
/// Serde helper: YAML map keys are always strings, convert to u32.
impl<'de> serde::Deserialize<'de> for NotificationConfig {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where D: serde::Deserializer<'de>
{
#[derive(serde::Deserialize)]
struct Helper {
#[serde(default)]
templates: HashMap<String, NotificationTemplateDef>,
#[serde(flatten)]
groups_raw: HashMap<String, HashMap<String, NotificationMethod>>,
}
let h = Helper::deserialize(deserializer)?;
let groups: HashMap<u32, _> = h.groups_raw.into_iter()
.map(|(k, v)| {
k.parse::<u32>().map(|n| (n, v))
.map_err(|_| serde::de::Error::custom(format!("priority key must be integer, got {:?}", k)))
})
.collect::<Result<_, _>>()?;
Ok(NotificationConfig { templates: h.templates, groups })
}
}
/// Stage-triggered notification configuration /// Stage-triggered notification configuration
#[derive(Debug, Deserialize, Serialize, Clone)] #[derive(Debug, Deserialize, Serialize, Clone)]
pub struct OnFinishOf { pub struct OnFinishOf {
@@ -158,7 +211,7 @@ pub struct OnFinishOf {
/// Notification template definition. /// Notification template definition.
/// Supports inline content (on_success/on_failure/on_finish_of) or external reference (use). /// Supports inline content (on_success/on_failure/on_finish_of) or external reference (use).
#[derive(Debug, Deserialize, Serialize, Clone)] #[derive(Debug, Deserialize, Serialize, Clone, Default)]
pub struct NotificationTemplateDef { pub struct NotificationTemplateDef {
/// External template reference (git URL). /// External template reference (git URL).
/// Cannot be combined with on_* fields in MVP (validation will reject). /// Cannot be combined with on_* fields in MVP (validation will reject).
@@ -178,6 +231,53 @@ pub struct NotificationTemplateDef {
pub on_finish_of: Option<String>, pub on_finish_of: Option<String>,
} }
impl NotificationTemplateDef {
/// Merge `other` into `self`, keeping self's values when present.
///
/// `other` fills in gaps where `self` is `None`.
/// Chaining: `a.apply(b).apply(c)` = a wins, then b fills gaps, then c fills gaps.
pub fn apply(mut self, other: Self) -> Self {
if self.on_success.is_none() {
self.on_success = other.on_success;
}
if self.on_failure.is_none() {
self.on_failure = other.on_failure;
}
if self.on_finish_of.is_none() {
self.on_finish_of = other.on_finish_of;
}
self
}
}
/// Builder for [`NotificationTemplateDef`].
pub struct NotificationTemplateDefBuilder(NotificationTemplateDef);
impl NotificationTemplateDefBuilder {
pub fn new() -> Self {
Self(NotificationTemplateDef::default())
}
pub fn with_use(mut self, val: impl Into<String>) -> Self {
self.0.use_ = Some(val.into());
self
}
pub fn with_on_success(mut self, val: impl Into<String>) -> Self {
self.0.on_success = Some(val.into());
self
}
pub fn with_on_failure(mut self, val: impl Into<String>) -> Self {
self.0.on_failure = Some(val.into());
self
}
pub fn with_on_finish_of(mut self, val: impl Into<String>) -> Self {
self.0.on_finish_of = Some(val.into());
self
}
pub fn build(self) -> NotificationTemplateDef {
self.0
}
}
/// Notification trigger type. /// Notification trigger type.
#[derive(Debug, Clone, PartialEq, Eq)] #[derive(Debug, Clone, PartialEq, Eq)]
pub enum Trigger { pub enum Trigger {
@@ -336,16 +436,19 @@ impl FinalizeConfig {
fn validate_notification(&self, errors: &mut Vec<String>) { fn validate_notification(&self, errors: &mut Vec<String>) {
// Validate templates // Validate templates
for (name, tmpl) in &self.notification.templates { for (name, tmpl) in &self.notification.templates {
if tmpl.use_.is_some() if let Some(ref u) = tmpl.use_ {
&& (tmpl.on_success.is_some() if u.contains("://") && (
|| tmpl.on_failure.is_some() tmpl.on_success.is_some() ||
|| tmpl.on_finish_of.is_some()) tmpl.on_failure.is_some() ||
{ tmpl.on_finish_of.is_some()
) {
errors.push(format!( errors.push(format!(
"Template '{}' cannot combine 'use' with 'on_*' fields (MVP restriction)", "Template '{}' cannot combine external 'use' with 'on_*' fields",
name name
)); ));
} }
// Local references (no ://) are allowed to have on_* overrides
}
} }
} }
} }
+24 -4
View File
@@ -1,14 +1,15 @@
use crate::finalize::RawPluginMap; use crate::finalize::RawPluginMap;
use crate::finalize::constant::{FINALIZE_PLUGIN_MANIFEST, FINALIZE_PLUGIN_MANIFEST_EXTENSION}; use crate::finalize::constant::{FINALIZE_PLUGIN_MANIFEST, FINALIZE_PLUGIN_MANIFEST_EXTENSION};
use crate::finalize::plugin::metadata::{PluginMetadata, PluginType}; use crate::finalize::plugin::metadata::{PluginMetadata, PluginType};
use crate::{EventSender, ExecutionContext, finalize::error::PluginError}; use workshop_engine::{EventSender, ExecutionContext};
use crate::{ finalize::error::PluginError};
use crate::types::resource::ResourceRegistry;
use async_trait::async_trait; use async_trait::async_trait;
use std::collections::{HashMap, HashSet}; use std::collections::{HashMap, HashSet};
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use std::sync::Arc; use std::sync::Arc;
mod dylib; mod dylib;
pub mod fetch;
mod internal; mod internal;
mod metadata; mod metadata;
mod rhai; mod rhai;
@@ -214,6 +215,8 @@ fn get_entry(plugin_type: PluginType, name: &str) -> Result<Box<dyn AsyncPluginF
pub fn register( pub fn register(
external_plugins: RawPluginMap, external_plugins: RawPluginMap,
list: Option<&HashSet<String>>, list: Option<&HashSet<String>>,
notification_config: &mut crate::finalize::NotificationConfig,
registry: &ResourceRegistry,
) -> Result<PluginMap, PluginError> { ) -> Result<PluginMap, PluginError> {
let mut plugins: PluginMap = HashMap::new(); let mut plugins: PluginMap = HashMap::new();
let mut count = 0; let mut count = 0;
@@ -255,7 +258,14 @@ pub fn register(
log::warn!("Plugin {} is not available: {}", name, reason); log::warn!("Plugin {} is not available: {}", name, reason);
continue; continue;
} }
let metadata = read_manifest(config.runtime.fspath, &name)?; // Resolve plugin path from registry instead of config.runtime.fspath
let plugin_path = registry.resolve(&name).ok_or_else(|| {
PluginError::PluginUnavailable {
name: name.clone(),
reason: "plugin not found in resource registry".to_string(),
}
})?;
let metadata = read_manifest(Some(plugin_path), &name)?;
let plugin_type = get_plugin_type(metadata.plugin_type.clone(), &metadata.entrypoint); let plugin_type = get_plugin_type(metadata.plugin_type.clone(), &metadata.entrypoint);
let entry = get_entry(plugin_type, &name)?; let entry = get_entry(plugin_type, &name)?;
plugins.insert( plugins.insert(
@@ -267,8 +277,18 @@ pub fn register(
count += 1; count += 1;
} }
log::info!("Registered {} plugins", count); // Inject plugin-declared templates as .fallback
for (name, plugin) in &plugins {
if let Some(ref templates) = plugin.metadata.templates {
let key = format!("{}.fallback", name);
for tmpl in templates {
notification_config.templates.insert(key.clone(), tmpl.clone());
}
log::debug!("Registered templates from plugin '{}' as '{}'", name, key);
}
}
log::info!("Registered {} plugins", count);
Ok(plugins) Ok(plugins)
} }
+2 -1
View File
@@ -1,6 +1,7 @@
use crate::finalize::plugin::PluginError; use crate::finalize::plugin::PluginError;
use crate::finalize::plugin::PluginMetadata; use crate::finalize::plugin::PluginMetadata;
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn}; use workshop_engine::{EventSender, ExecutionContext};
use crate::{ finalize::plugin::AsyncPluginFn};
use async_trait::async_trait; use async_trait::async_trait;
pub struct Dylib; pub struct Dylib;
-109
View File
@@ -1,109 +0,0 @@
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
use crate::finalize::fetch;
use crate::finalize::plugin::{PluginError, RawPluginMap};
use std::collections::HashSet;
use std::path::Path;
mod checksum;
mod extract;
mod git;
mod http;
mod types;
pub use types::FetchArgument;
enum URLScheme {
Unknown,
Git,
Http,
Https,
File,
}
impl URLScheme {
fn parse(url: &str) -> (Self, String) {
match url.split_once("://") {
Some((prefix, rest)) => {
let scheme = match prefix {
"git" => URLScheme::Git,
"https" => URLScheme::Https,
"http" => URLScheme::Http,
"file" => URLScheme::File,
_ => URLScheme::Unknown,
};
(scheme, rest.to_string())
}
None => (URLScheme::Unknown, url.to_string()),
}
}
}
pub async fn fetch_plugin(
url: &str,
name: &str,
basedir: &Path,
argument: FetchArgument,
) -> Result<(), PluginError> {
let (scheme, path) = URLScheme::parse(url);
match scheme {
URLScheme::Git => {
// NOTE: as a practical compromise, we use git command for git clone
// gix will be supported later, with ONLY https scheme
// TODO: Implement gix
log::info!("Fetching git plugin: {}", url);
git::fetch_git_plugin(&path, name, basedir, argument.shallow).await
}
URLScheme::Http | URLScheme::Https => {
log::info!("Fetching http plugin: {}", url);
let checksum = argument.get_checksum();
http::fetch_http_plugin(url, name, basedir, checksum).await
}
URLScheme::File => {
// TODO: Support client mode
// NOTE: In standalone mode, we just assume path is well-defined and utilize
log::info!("Fetching file plugin: {}", url);
Ok(())
}
URLScheme::Unknown => {
log::error!("Unknown URL scheme in plugin URL: {}", url);
Err(PluginError::GeneralError(format!(
"Unknown URL scheme in plugin URL: {}",
url
)))
}
}
}
pub async fn fetch_plugins(
plugin_path: &Path,
plugin_config: &mut RawPluginMap,
list: Option<&HashSet<String>>,
) -> Result<(), PluginError> {
log::info!("Preparing to download {} plugins", plugin_config.len());
for (name, data) in plugin_config.iter_mut() {
match list {
Some(list) if !list.contains(name) => {
log::debug!("Plugin {} not in list, skipping", name);
continue;
}
_ => {}
}
log::debug!("Downloading plugin {}", name);
let (source, config) =
data.iter_mut()
.next()
.ok_or_else(|| PluginError::PluginUnavailable {
name: name.clone(),
reason: "no source configured".to_string(),
})?;
let fetch_argument = FetchArgument {
checksum: config.checksum.clone(),
shallow: config.shallow,
};
fetch::fetch_plugin(source, name, plugin_path, fetch_argument).await?;
log::debug!("Storing plugin {} to {}", name, plugin_path.display());
config.runtime.fspath = Some(plugin_path.join(name));
}
Ok(())
}
@@ -1,137 +0,0 @@
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
use super::types::ChecksumType;
use sha2::{Digest, Sha256, Sha512};
/// Detects the checksum type based on the length of the checksum string.
///
/// Returns `Ok(None)` for empty string (checksum disabled).
/// Returns `Ok(Some(ChecksumType))` for valid SHA256 (64 chars) or SHA512 (128 chars).
/// Returns `Err` for invalid lengths (32=MD5, 40=SHA1) or unsupported lengths.
pub fn detect_checksum_type(checksum: &str) -> Result<Option<ChecksumType>, String> {
match checksum.len() {
0 => Ok(None),
32 => Err(format!(
"MD5 (first 8 chars: {}) is not supported for security reasons. Use SHA256 (64 chars) or SHA512 (128 chars)",
&checksum[..8.min(checksum.len())]
)),
40 => Err(format!(
"SHA1 (first 8 chars: {}) is deprecated and not supported. Use SHA256 (64 chars) or SHA512 (128 chars)",
&checksum[..8.min(checksum.len())]
)),
64 => Ok(Some(ChecksumType::Sha256)),
128 => Ok(Some(ChecksumType::Sha512)),
_ => Err(format!(
"Invalid checksum length: {}. Expected 0 (disabled), 64 (SHA256), or 128 (SHA512). First 8 chars: {}",
checksum.len(),
&checksum[..8.min(checksum.len())]
)),
}
}
/// Verifies that the data matches the expected checksum.
///
/// Compares the computed hash of `data` against `expected` using the specified `checksum_type`.
/// Returns `Ok(())` if the checksums match, `Err` otherwise.
pub fn verify_checksum(
data: &[u8],
expected: &str,
checksum_type: ChecksumType,
) -> Result<(), String> {
let actual = match checksum_type {
ChecksumType::Sha256 => {
let mut hasher = Sha256::new();
hasher.update(data);
format!("{:x}", hasher.finalize())
}
ChecksumType::Sha512 => {
let mut hasher = Sha512::new();
hasher.update(data);
format!("{:x}", hasher.finalize())
}
};
if actual.eq_ignore_ascii_case(expected) {
Ok(())
} else {
Err(format!(
"Checksum mismatch: expected {}..., got {}...",
&expected[..8.min(expected.len())],
&actual[..8.min(actual.len())]
))
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_detect_checksum_type_empty() {
assert_eq!(detect_checksum_type("").unwrap(), None);
}
#[test]
fn test_detect_checksum_type_md5_rejected() {
let result = detect_checksum_type("a".repeat(32).as_str());
assert!(result.is_err());
assert!(result.unwrap_err().contains("MD5"));
}
#[test]
fn test_detect_checksum_type_sha1_rejected() {
let result = detect_checksum_type("a".repeat(40).as_str());
assert!(result.is_err());
assert!(result.unwrap_err().contains("SHA1"));
}
#[test]
fn test_detect_checksum_type_sha256() {
assert_eq!(
detect_checksum_type("a".repeat(64).as_str()).unwrap(),
Some(ChecksumType::Sha256)
);
}
#[test]
fn test_detect_checksum_type_sha512() {
assert_eq!(
detect_checksum_type("a".repeat(128).as_str()).unwrap(),
Some(ChecksumType::Sha512)
);
}
#[test]
fn test_detect_checksum_type_invalid_length() {
let result = detect_checksum_type("a".repeat(50).as_str());
assert!(result.is_err());
assert!(result.unwrap_err().contains("Invalid checksum length"));
}
#[test]
fn test_verify_checksum_sha256_valid() {
let data = b"hello";
let expected = "2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824";
assert!(verify_checksum(data, expected, ChecksumType::Sha256).is_ok());
}
#[test]
fn test_verify_checksum_sha256_invalid() {
let data = b"hello";
let expected = "0000000000000000000000000000000000000000000000000000000000000000";
assert!(verify_checksum(data, expected, ChecksumType::Sha256).is_err());
}
#[test]
fn test_verify_checksum_sha512_valid() {
let data = b"hello";
let expected = "9b71d224bd62f3785d96d46ad3ea3d73319bfbc2890caadae2dff72519673ca72323c3d99ba5c11d7c7acc6e14b8c5da0c4663475c2e5c3adef46f73bcdec043";
assert!(verify_checksum(data, expected, ChecksumType::Sha512).is_ok());
}
#[test]
fn test_verify_checksum_sha512_invalid() {
let data = b"hello";
let expected = "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000";
assert!(verify_checksum(data, expected, ChecksumType::Sha512).is_err());
}
}
@@ -1,169 +0,0 @@
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
use crate::finalize::plugin::PluginError;
use std::path::{Path, PathBuf};
use tokio::task;
/// Compression type for tar archives.
#[derive(Debug, Clone, Copy, PartialEq)]
pub enum CompressionType {
/// Gzip compressed (.tar.gz, .tgz)
Gzip,
/// Zstd compressed (.tar.zst, .tar.zstd, .tzst)
Zstd,
/// Uncompressed tar (.tar)
None,
}
impl CompressionType {
/// Detects compression type from file path extension.
///
/// Supports: .tar.gz, .tgz (Gzip), .tar.zst, .tar.zstd, .tzst (Zstd), .tar (None).
pub fn from_path(path: &std::path::Path) -> Result<Self, String> {
let name = path
.file_name()
.and_then(|n| n.to_str())
.unwrap_or("")
.to_lowercase();
if name.ends_with(".tar.gz") || name.ends_with(".tgz") {
Ok(CompressionType::Gzip)
} else if name.ends_with(".tar.zst")
|| name.ends_with(".tar.zstd")
|| name.ends_with(".tzst")
{
Ok(CompressionType::Zstd)
} else if name.ends_with(".tar") {
Ok(CompressionType::None)
} else {
Err(format!(
"Unsupported archive format: '{}'. Supported formats: .tar, .tar.gz, .tgz, .tar.zst, .tar.zstd, .tzst",
name
))
}
}
}
/// Extracts a tar archive to the destination directory.
///
/// Removes existing destination directory if present, creates a fresh directory,
/// then unpacks the archive using the appropriate decompression based on `compression`.
pub async fn extract_tar(
archive: &Path,
destdir: &PathBuf,
compression: CompressionType,
) -> Result<(), PluginError> {
if destdir.exists() {
tokio::fs::remove_dir_all(destdir).await.map_err(|e| {
PluginError::GeneralError(format!("Failed to remove existing directory: {}", e))
})?;
}
tokio::fs::create_dir_all(destdir)
.await
.map_err(|e| PluginError::GeneralError(format!("Failed to create directory: {}", e)))?;
let archive = archive.to_path_buf();
let destdir = destdir.clone();
task::spawn_blocking(move || {
let file = std::fs::File::open(&archive)
.map_err(|e| PluginError::GeneralError(format!("Failed to open archive: {}", e)))?;
match compression {
CompressionType::Gzip => {
let decoder = flate2::read::GzDecoder::new(file);
let mut archive = tar::Archive::new(decoder);
archive.unpack(&destdir).map_err(|e| {
PluginError::GeneralError(format!("Failed to unpack archive: {}", e))
})?;
}
CompressionType::Zstd => {
let decoder = zstd::stream::read::Decoder::new(file).map_err(|e| {
PluginError::GeneralError(format!("Failed to create zstd decoder: {}", e))
})?;
let mut archive = tar::Archive::new(decoder);
archive.unpack(&destdir).map_err(|e| {
PluginError::GeneralError(format!("Failed to unpack archive: {}", e))
})?;
}
CompressionType::None => {
let mut archive = tar::Archive::new(file);
archive.unpack(&destdir).map_err(|e| {
PluginError::GeneralError(format!("Failed to unpack archive: {}", e))
})?;
}
};
Ok::<(), PluginError>(())
})
.await
.map_err(|e| PluginError::GeneralError(format!("Task join error: {}", e)))??;
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use std::path::Path;
#[test]
fn test_from_path_tar_gz() {
assert_eq!(
CompressionType::from_path(Path::new("archive.tar.gz")).unwrap(),
CompressionType::Gzip
);
}
#[test]
fn test_from_path_tgz() {
assert_eq!(
CompressionType::from_path(Path::new("archive.tgz")).unwrap(),
CompressionType::Gzip
);
}
#[test]
fn test_from_path_tar_zst() {
assert_eq!(
CompressionType::from_path(Path::new("archive.tar.zst")).unwrap(),
CompressionType::Zstd
);
}
#[test]
fn test_from_path_tar_zstd() {
assert_eq!(
CompressionType::from_path(Path::new("archive.tar.zstd")).unwrap(),
CompressionType::Zstd
);
}
#[test]
fn test_from_path_tzst() {
assert_eq!(
CompressionType::from_path(Path::new("archive.tzst")).unwrap(),
CompressionType::Zstd
);
}
#[test]
fn test_from_path_tar() {
assert_eq!(
CompressionType::from_path(Path::new("archive.tar")).unwrap(),
CompressionType::None
);
}
#[test]
fn test_from_path_unsupported() {
assert!(CompressionType::from_path(Path::new("archive.zip")).is_err());
}
#[test]
fn test_from_path_case_insensitive() {
assert_eq!(
CompressionType::from_path(Path::new("archive.TAR.GZ")).unwrap(),
CompressionType::Gzip
);
}
}
@@ -1,185 +0,0 @@
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
use crate::finalize::plugin::PluginError;
use std::path::Path;
use tokio::process::Command;
/// Version specification for a git repository.
#[derive(Debug, PartialEq)]
pub enum GitVersion {
/// No version specified (invalid for fetch operations)
None,
/// Single version string (tag or commit hash)
Single(String),
/// Mixed: try tag first, fallback to commit hash
Mixed(String, String),
}
/// Parses a git URL with optional version specification.
///
/// Format: `url` (no version), `url@tag` (single version), `url@tag:commit` (mixed version).
/// Returns (base_url, GitVersion).
pub fn parse_git_url(url: &str) -> (String, GitVersion) {
if let Some((base, version)) = url.rsplit_once('@') {
if version.contains(":") {
let (tag, commit) = version.split_once(':').unwrap();
return (
base.to_string(),
GitVersion::Mixed(tag.to_string(), commit.to_string()),
);
}
return (base.to_string(), GitVersion::Single(version.to_string()));
}
(url.to_string(), GitVersion::None)
}
/// Clones a git repository and checks out the specified version.
///
/// Removes existing destination directory, performs shallow or full clone,
/// then checks out the given version (tag or commit).
pub async fn git_clone_and_checkout(
baseurl: &str,
version: &str,
dest: &Path,
name: &str,
shallow: bool,
) -> Result<(), PluginError> {
if dest.exists() {
tokio::fs::remove_dir_all(dest)
.await
.map_err(|e| PluginError::FetchError {
name: name.to_string(),
url: baseurl.to_string(),
reason: format!("Failed to remove existing directory: {}", e),
})?;
}
let mut cmd = Command::new("git");
cmd.arg("clone");
if shallow {
cmd.arg("--depth").arg("1");
}
cmd.arg(baseurl).arg(dest);
let output = cmd.output().await.map_err(|e| PluginError::FetchError {
name: name.to_string(),
url: baseurl.to_string(),
reason: e.to_string(),
})?;
if !output.status.success() {
return Err(PluginError::FetchError {
name: name.to_string(),
url: baseurl.to_string(),
reason: String::from_utf8_lossy(&output.stderr).to_string(),
});
}
let mut cmd = Command::new("git");
cmd.arg("checkout").arg(version).current_dir(dest);
let output = cmd.output().await.map_err(|e| PluginError::FetchError {
name: name.to_string(),
url: baseurl.to_string(),
reason: e.to_string(),
})?;
if !output.status.success() {
return Err(PluginError::FetchError {
name: name.to_string(),
url: baseurl.to_string(),
reason: String::from_utf8_lossy(&output.stderr).to_string(),
});
}
Ok(())
}
/// Fetches a plugin from a git repository.
///
/// Parses the URL for version info, then clones and checks out the appropriate ref.
/// For Mixed version, tries the tag first and falls back to the commit hash.
pub async fn fetch_git_plugin(
path: &str,
name: &str,
basedir: &Path,
shallow: Option<bool>,
) -> Result<(), PluginError> {
let (baseurl, version) = parse_git_url(path);
match version {
GitVersion::None => {
log::error!("Version should be specified for git plugin URL: {}", path);
Err(PluginError::FetchError {
name: name.to_string(),
url: path.to_string(),
reason: "Version should be specified as either a tag or a commit hash.".to_string(),
})
}
GitVersion::Single(version) => {
log::debug!("Fetching git plugin: {}@{}", name, version);
let shallow_flag = shallow.unwrap_or(false);
git_clone_and_checkout(
&baseurl,
&version.to_string(),
&basedir.join(name),
name,
shallow_flag,
)
.await
}
GitVersion::Mixed(tag, commit) => {
log::debug!("Fetching git plugin: {}@{}/{}", name, tag, commit);
let shallow_flag = shallow.unwrap_or(false);
let result =
git_clone_and_checkout(&baseurl, &tag, &basedir.join(name), name, shallow_flag)
.await;
if result.is_err() {
log::warn!(
"Failed to clone with tag, trying commit hash. URL: {}",
path
);
return git_clone_and_checkout(&baseurl, &commit, &basedir.join(name), name, false)
.await;
}
result
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_parse_git_url_no_version() {
let (url, version) = parse_git_url("https://github.com/user/repo");
assert_eq!(url, "https://github.com/user/repo");
assert_eq!(version, GitVersion::None);
}
#[test]
fn test_parse_git_url_single_tag() {
let (url, version) = parse_git_url("https://github.com/user/repo@v1.0.0");
assert_eq!(url, "https://github.com/user/repo");
assert_eq!(version, GitVersion::Single("v1.0.0".to_string()));
}
#[test]
fn test_parse_git_url_mixed_version() {
let (url, version) = parse_git_url("https://github.com/user/repo@main:abc123");
assert_eq!(url, "https://github.com/user/repo");
assert_eq!(
version,
GitVersion::Mixed("main".to_string(), "abc123".to_string())
);
}
#[test]
fn test_parse_git_url_complex_url() {
let (url, version) = parse_git_url("git@github.com:user/repo.git@tag:commit");
assert_eq!(url, "git@github.com:user/repo.git");
assert_eq!(
version,
GitVersion::Mixed("tag".to_string(), "commit".to_string())
);
}
#[test]
fn test_parse_git_url_single_with_at_in_path() {
let (url, version) = parse_git_url("https://github.com/user/repo@my-repo@v1.0");
assert_eq!(url, "https://github.com/user/repo@my-repo");
assert_eq!(version, GitVersion::Single("v1.0".to_string()));
}
}
@@ -1,110 +0,0 @@
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
use super::checksum::{detect_checksum_type, verify_checksum};
use super::extract::{CompressionType, extract_tar};
use crate::finalize::plugin::PluginError;
use std::path::Path;
pub async fn download_to_file(url: &str, dest: &Path, name: &str) -> Result<(), PluginError> {
let client = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(300))
.build()
.map_err(|e| PluginError::GeneralError(format!("Failed to build HTTP client: {}", e)))?;
let response = client
.get(url)
.send()
.await
.map_err(|e| PluginError::FetchError {
name: name.to_string(),
url: url.to_string(),
reason: format!("HTTP request failed: {}", e),
})?;
let status = response.status();
if !status.is_success() {
return Err(PluginError::FetchError {
name: name.to_string(),
url: url.to_string(),
reason: format!("HTTP error: {}", status),
});
}
let bytes = response
.bytes()
.await
.map_err(|e| PluginError::FetchError {
name: name.to_string(),
url: url.to_string(),
reason: format!("Failed to read response body: {}", e),
})?;
tokio::fs::write(dest, &bytes)
.await
.map_err(|e| PluginError::FetchError {
name: name.to_string(),
url: url.to_string(),
reason: format!("Failed to write file: {}", e),
})?;
Ok(())
}
fn extract_archive_extension(url: &str) -> &str {
let path_part = url
.split('?')
.next()
.unwrap_or(url)
.split('#')
.next()
.unwrap_or(url);
let file_part = path_part.rsplit('/').next().unwrap_or("");
let lower = file_part.to_lowercase();
if lower.ends_with(".tar.gz") || lower.ends_with(".tgz") {
".tar.gz"
} else if lower.ends_with(".tar.zst")
|| lower.ends_with(".tar.zstd")
|| lower.ends_with(".tzst")
{
".tar.zst"
} else if lower.ends_with(".tar") {
".tar"
} else {
".unknown"
}
}
pub async fn fetch_http_plugin(
url: &str,
name: &str,
basedir: &Path,
checksum_str: Option<&str>,
) -> Result<(), PluginError> {
let temp_dir = std::env::temp_dir().join("workshop-plugins");
tokio::fs::create_dir_all(&temp_dir).await.ok();
let ext = extract_archive_extension(url);
let temp_file = temp_dir.join(format!("{}-{}{}", name, uuid::Uuid::new_v4(), ext));
download_to_file(url, &temp_file, name).await?;
if let Some(checksum) = checksum_str
&& !checksum.is_empty()
{
let bytes = tokio::fs::read(&temp_file).await.map_err(|e| {
PluginError::GeneralError(format!("Failed to read downloaded file: {}", e))
})?;
let checksum_type = detect_checksum_type(checksum).map_err(PluginError::GeneralError)?;
if let Some(ct) = checksum_type {
verify_checksum(&bytes, checksum, ct).map_err(PluginError::GeneralError)?;
}
}
let compression = CompressionType::from_path(&temp_file).map_err(PluginError::GeneralError)?;
let destdir = basedir.join(name);
extract_tar(&temp_file, &destdir, compression).await?;
tokio::fs::remove_file(&temp_file).await.ok();
Ok(())
}
@@ -1,84 +0,0 @@
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
use serde::{Deserialize, Serialize};
/// Supported checksum types for verifying downloaded artifacts.
#[derive(Debug, Clone, PartialEq)]
pub enum ChecksumType {
/// SHA-256 hash (64 hex characters)
Sha256,
/// SHA-512 hash (128 hex characters)
Sha512,
}
/// Arguments for fetching a plugin artifact.
///
/// Contains optional checksum verification and shallow clone settings.
#[derive(Debug, Deserialize, Serialize, Clone, Default)]
pub struct FetchArgument {
/// SHA256 or SHA512 hex string for integrity verification.
#[serde(default)]
pub checksum: Option<String>,
/// Perform a shallow clone (single revision only).
#[serde(default)]
pub shallow: Option<bool>,
}
impl FetchArgument {
/// Returns the checksum if present and non-empty.
pub fn get_checksum(&self) -> Option<&str> {
self.checksum.as_deref().filter(|s| !s.is_empty())
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_get_checksum_some_non_empty() {
let arg = FetchArgument {
checksum: Some("abc123".to_string()),
shallow: None,
};
assert_eq!(arg.get_checksum(), Some("abc123"));
}
#[test]
fn test_get_checksum_empty_string() {
let arg = FetchArgument {
checksum: Some("".to_string()),
shallow: None,
};
assert_eq!(arg.get_checksum(), None);
}
#[test]
fn test_get_checksum_none() {
let arg = FetchArgument {
checksum: None,
shallow: None,
};
assert_eq!(arg.get_checksum(), None);
}
#[test]
fn test_serde_roundtrip() {
let arg = FetchArgument {
checksum: Some("deadbeef".to_string()),
shallow: Some(true),
};
let json = serde_json::to_string(&arg).unwrap();
let parsed: FetchArgument = serde_json::from_str(&json).unwrap();
assert_eq!(parsed.checksum, arg.checksum);
assert_eq!(parsed.shallow, arg.shallow);
}
#[test]
fn test_serde_roundtrip_empty() {
let arg = FetchArgument::default();
let json = serde_json::to_string(&arg).unwrap();
let parsed: FetchArgument = serde_json::from_str(&json).unwrap();
assert_eq!(parsed.checksum, arg.checksum);
assert_eq!(parsed.shallow, arg.shallow);
}
}
@@ -29,3 +29,10 @@ pub fn get_internal_plugin() -> Vec<InternalPlugin> {
}, },
] ]
} }
pub fn get_dummy_plugin() -> Vec<InternalPlugin> {
vec![InternalPlugin {
name: "dummy",
entry: Box::new(dummy::Dummy),
}]
}
@@ -1,12 +1,13 @@
use crate::finalize::plugin::PluginError; use crate::finalize::plugin::PluginError;
use crate::finalize::plugin::PluginMetadata; use crate::finalize::plugin::PluginMetadata;
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn}; use workshop_engine::{EventSender, ExecutionContext};
use crate::{ finalize::plugin::AsyncPluginFn};
use async_trait::async_trait; use async_trait::async_trait;
pub struct _Dummy; pub struct Dummy;
#[async_trait] #[async_trait]
impl AsyncPluginFn for _Dummy { impl AsyncPluginFn for Dummy {
async fn call( async fn call(
&self, &self,
_metadata: &PluginMetadata, _metadata: &PluginMetadata,
@@ -14,6 +15,7 @@ impl AsyncPluginFn for _Dummy {
_ctx: &ExecutionContext, _ctx: &ExecutionContext,
_event_tx: &EventSender, _event_tx: &EventSender,
) -> Result<(), PluginError> { ) -> Result<(), PluginError> {
log::debug!("[NOTIFY_DEBUG] Successfully registered dummy plugin.");
Ok(()) Ok(())
} }
} }
@@ -1,12 +1,10 @@
use crate::finalize::plugin::PluginMetadata; use crate::finalize::plugin::PluginMetadata;
use crate::{ use crate::{
EventSender, ExecutionContext,
finalize::{error::PluginError, plugin::AsyncPluginFn}, finalize::{error::PluginError, plugin::AsyncPluginFn},
}; };
use async_trait::async_trait; use async_trait::async_trait;
use workshop_engine::{EventSender, ExecutionContext};
pub struct InsiteNotify; pub struct InsiteNotify;
#[async_trait] #[async_trait]
impl AsyncPluginFn for InsiteNotify { impl AsyncPluginFn for InsiteNotify {
async fn call( async fn call(
@@ -16,6 +14,7 @@ impl AsyncPluginFn for InsiteNotify {
_ctx: &ExecutionContext, _ctx: &ExecutionContext,
_event_tx: &EventSender, _event_tx: &EventSender,
) -> Result<(), PluginError> { ) -> Result<(), PluginError> {
unimplemented!("Requires workshop-base to be implemented!"); log::info!("in-site-notify: placeholder (always succeeds)");
Ok(())
} }
} }
@@ -1,18 +1,9 @@
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
use crate::finalize::plugin::PluginError;
use crate::finalize::plugin::PluginMetadata; use crate::finalize::plugin::PluginMetadata;
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn}; use crate::{
finalize::{error::PluginError, plugin::AsyncPluginFn},
};
use async_trait::async_trait; use async_trait::async_trait;
use lettre::message::{MultiPart, SinglePart, header}; use workshop_engine::{EventSender, ExecutionContext};
use lettre::{AsyncSmtpTransport, AsyncTransport, Message, Tokio1Executor};
use std::time::Duration;
use tokio::time::timeout;
mod config;
mod template;
pub use config::*;
use template::*;
pub struct Mail; pub struct Mail;
@@ -21,182 +12,10 @@ impl AsyncPluginFn for Mail {
async fn call( async fn call(
&self, &self,
_metadata: &PluginMetadata, _metadata: &PluginMetadata,
argument: serde_yaml::Value, _argument: serde_yaml::Value,
ctx: &ExecutionContext, _ctx: &ExecutionContext,
_event_tx: &EventSender, _event_tx: &EventSender,
) -> Result<(), PluginError> { ) -> Result<(), PluginError> {
let config: MailConfig = serde_yaml::from_value(argument) todo!();
.map_err(|e| PluginError::GeneralError(format!("Invalid mail config: {}", e)))?;
if let Err(errors) = config.validate() {
return Err(PluginError::GeneralError(format!(
"Mail config validation failed: {:?}",
errors
)));
}
let template_data = prepare_template_data(ctx);
let (subject_template, body_template_html, body_template_text) = match config.schema {
Schema::Default => {
let subject = config
.subject
.as_deref()
.unwrap_or(DefaultTemplates::SUBJECT);
let body_html = config
.body
.as_deref()
.unwrap_or(DefaultTemplates::BODY_HTML);
(
subject.to_string(),
body_html.to_string(),
DefaultTemplates::BODY_TEXT.to_string(),
)
}
Schema::Custom => {
let subject = config.subject.clone().unwrap();
let body_html = config.body.clone().unwrap();
(subject, body_html.clone(), body_html)
}
};
let subject = render_template(&subject_template, &template_data)
.map_err(|e| PluginError::GeneralError(format!("Failed to render subject: {}", e)))?;
let body_html = render_template(&body_template_html, &template_data)
.map_err(|e| PluginError::GeneralError(format!("Failed to render body_html: {}", e)))?;
let body_text = render_template(&body_template_text, &template_data).map_err(|e| {
PluginError::GeneralError(format!("Failed to render body_text(?): {}", e))
})?;
let email = build_email(&config, subject, body_html, body_text)?;
send_email(&config, email).await
} }
} }
fn build_email(
config: &MailConfig,
subject: String,
body_html: String,
body_text: String,
) -> Result<Message, PluginError> {
let from_str = config.from.to_string();
let from: lettre::message::Mailbox = from_str.parse().map_err(|e| {
PluginError::GeneralError(format!("Invalid from address '{}': {}", from_str, e))
})?;
let to_addrs: Vec<String> = config.to.to_vec();
if to_addrs.is_empty() {
return Err(PluginError::GeneralError(
"No recipients specified".to_string(),
));
}
let mut message_builder = Message::builder().from(from).subject(subject);
for addr in &to_addrs {
let mailbox: lettre::message::Mailbox = addr.parse().map_err(|e| {
PluginError::GeneralError(format!("Invalid to address '{}': {}", addr, e))
})?;
message_builder = message_builder.to(mailbox);
}
if let Some(cc) = &config.cc {
for addr in cc.to_vec() {
let mailbox: lettre::message::Mailbox = addr.parse().map_err(|e| {
PluginError::GeneralError(format!("Invalid cc address '{}': {}", addr, e))
})?;
message_builder = message_builder.cc(mailbox);
}
}
if let Some(bcc) = &config.bcc {
for addr in bcc.to_vec() {
let mailbox: lettre::message::Mailbox = addr.parse().map_err(|e| {
PluginError::GeneralError(format!("Invalid bcc address '{}': {}", addr, e))
})?;
message_builder = message_builder.bcc(mailbox);
}
}
if let Some(reply_to) = &config.reply_to {
let mailbox: lettre::message::Mailbox = reply_to.parse().map_err(|e| {
PluginError::GeneralError(format!("Invalid reply-to address '{}': {}", reply_to, e))
})?;
message_builder = message_builder.reply_to(mailbox);
}
let message = message_builder
.multipart(
MultiPart::alternative()
.singlepart(
SinglePart::builder()
.header(header::ContentType::TEXT_PLAIN)
.body(body_text),
)
.singlepart(
SinglePart::builder()
.header(header::ContentType::TEXT_HTML)
.body(body_html),
),
)
.map_err(|e| PluginError::GeneralError(format!("Failed to build email: {}", e)))?;
Ok(message)
}
async fn send_email(config: &MailConfig, email: Message) -> Result<(), PluginError> {
let connect_timeout = if config.smtp.connect_timeout_ms == 0 {
None
} else {
Some(Duration::from_millis(config.smtp.connect_timeout_ms))
};
let mut transport_builder = match config.smtp.encryption {
Encryption::Tls => {
log::debug!("Using TLS encryption");
AsyncSmtpTransport::<Tokio1Executor>::relay(&config.smtp.host)
.map_err(|e| PluginError::GeneralError(format!("Invalid SMTP host: {}", e)))?
}
Encryption::Starttls => {
log::debug!("Using StartTLS encryption");
AsyncSmtpTransport::<Tokio1Executor>::starttls_relay(&config.smtp.host)
.map_err(|e| PluginError::GeneralError(format!("Invalid SMTP host: {}", e)))?
}
Encryption::None => {
log::debug!("Using NO encryption");
AsyncSmtpTransport::<Tokio1Executor>::builder_dangerous(&config.smtp.host)
}
};
transport_builder = transport_builder.port(config.smtp.port);
match config.smtp.auth.auth_type {
AuthType::Password => {
log::debug!("Using password auth");
transport_builder = transport_builder.credentials(
(
config.smtp.auth.username.clone(),
config.smtp.auth.password.clone(),
)
.into(),
);
}
AuthType::None => {
log::debug!("Using no auth");
}
}
let transport = transport_builder.build();
let result = match connect_timeout {
Some(t) => timeout(t, transport.send(email)).await,
None => Ok(transport.send(email).await),
};
result
.map_err(|_| PluginError::GeneralError("SMTP connection timed out".to_string()))?
.map_err(|e| PluginError::GeneralError(format!("SMTP error: {}", e)))?;
Ok(())
}
@@ -1,490 +0,0 @@
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
use serde::Deserialize;
use workshop_engine::deserialize_duration_ms;
/// Email address representation supporting plain string or structured object.
#[derive(Debug, Deserialize, Clone)]
#[serde(untagged)]
pub enum EmailAddress {
/// Plain email address string.
String(String),
/// Structured email with display name.
Object { name: String, email: String },
}
impl EmailAddress {
/// Returns the formatted address string (e.g., "user@example.com" or "Name <user@example.com>").
pub fn to_string(&self) -> String {
match self {
EmailAddress::String(email) => email.clone(),
EmailAddress::Object { name, email } => {
if name.is_empty() {
email.clone()
} else {
format!("{} <{}>", name, email)
}
}
}
}
/// Returns the raw email address without formatting.
pub fn email(&self) -> &str {
match self {
EmailAddress::String(email) => email.as_str(),
EmailAddress::Object { email, .. } => email.as_str(),
}
}
}
/// Recipients container supporting single or multiple addresses.
#[derive(Debug, Deserialize, Clone)]
#[serde(untagged)]
pub enum Recipients {
/// Single recipient.
Single(String),
/// Multiple recipients.
Multiple(Vec<String>),
}
impl Recipients {
/// Converts recipients to a flat vector of email strings.
pub fn to_vec(&self) -> Vec<String> {
match self {
Recipients::Single(s) => vec![s.clone()],
Recipients::Multiple(v) => v.clone(),
}
}
}
/// SMTP authentication mechanism.
#[derive(Debug, Deserialize, Clone)]
#[serde(rename_all = "lowercase")]
pub enum AuthType {
/// No authentication.
None,
/// Password-based authentication.
Password,
}
/// SMTP authentication configuration.
#[derive(Debug, Deserialize, Clone)]
pub struct AuthConfig {
/// Authentication mechanism type.
#[serde(rename = "type")]
pub auth_type: AuthType,
/// Username for authentication.
pub username: String,
/// Password for authentication.
pub password: String,
}
/// SMTP connection encryption mode.
#[derive(Debug, Deserialize, Clone)]
#[serde(rename_all = "lowercase")]
pub enum Encryption {
/// Implicit TLS/SSL on connection.
Tls,
/// STARTTLS upgrade after initial plaintext connection.
Starttls,
/// No encryption.
None,
}
/// SMTP server connection configuration.
#[derive(Debug, Deserialize, Clone)]
pub struct SmtpConfig {
/// SMTP server hostname.
pub host: String,
/// SMTP server port.
pub port: u16,
/// Authentication configuration.
pub auth: AuthConfig,
/// Encryption mode.
pub encryption: Encryption,
/// Connection timeout in milliseconds.
#[serde(
default = "default_connect_timeout_ms",
deserialize_with = "deserialize_duration_ms"
)]
pub connect_timeout_ms: u64,
}
fn default_connect_timeout_ms() -> u64 {
30_000
}
/// Mail schema type determining required fields.
#[derive(Debug, Deserialize, Clone)]
#[serde(rename_all = "lowercase")]
pub enum Schema {
/// Default schema with minimal required fields.
Default,
/// Custom schema requiring explicit subject and body.
Custom,
}
fn default_schema() -> Schema {
Schema::Default
}
/// Complete mail notification configuration.
#[derive(Debug, Deserialize, Clone)]
pub struct MailConfig {
/// Primary recipients.
pub to: Recipients,
/// Carbon copy recipients.
#[serde(default)]
pub cc: Option<Recipients>,
/// Blind carbon copy recipients.
#[serde(default)]
pub bcc: Option<Recipients>,
/// Sender address.
pub from: EmailAddress,
/// Reply-to address.
#[serde(default)]
pub reply_to: Option<String>,
/// Schema type determining validation rules.
#[serde(default = "default_schema")]
pub schema: Schema,
/// Email subject (required for custom schema).
pub subject: Option<String>,
/// Email body content (required for custom schema).
pub body: Option<String>,
/// SMTP server configuration.
pub smtp: SmtpConfig,
}
impl MailConfig {
/// Validates the mail configuration according to schema rules.
/// Returns Ok(()) if valid, or Err(vec) with error messages if invalid.
pub fn validate(&self) -> Result<(), Vec<String>> {
let mut errors = Vec::new();
if matches!(self.schema, Schema::Custom) {
if self.subject.is_none() {
errors.push("subject: required when schema=custom".to_string());
}
if self.body.is_none() {
errors.push("body: required when schema=custom".to_string());
}
}
if self.to.to_vec().is_empty() {
errors.push("to: must have at least one recipient".to_string());
}
if errors.is_empty() {
Ok(())
} else {
Err(errors)
}
}
}
/// Built-in email notification templates.
pub struct DefaultTemplates;
impl DefaultTemplates {
/// Default subject template with build status.
pub const SUBJECT: &'static str =
"[{{ build.status | upper }}] {{ pipeline.name }} #{{ build.id }}";
/// Default HTML email body template.
pub const BODY_HTML: &'static str = r#"<!DOCTYPE html>
<html>
<body style="font-family: Arial, sans-serif;">
<h2 style="color: {{ build.status_color }};">
Build {{ build.status | title }}
</h2>
<table>
<tr><td><strong>Pipeline:</strong></td><td>{{ pipeline.name }}</td></tr>
<tr><td><strong>Build ID:</strong></td><td>{{ build.id }}</td></tr>
<tr><td><strong>Status:</strong></td><td>{{ build.status }}</td></tr>
<tr><td><strong>Duration:</strong></td><td>{{ build.duration }}</td></tr>
<tr><td><strong>Author:</strong></td><td>{{ commit.author }}</td></tr>
<tr><td><strong>Message:</strong></td><td>{{ commit.message }}</td></tr>
</table>
<p>
<a href="{{ build.url }}" style="background: #007bff; color: white; padding: 10px 20px; text-decoration: none; border-radius: 5px;">
View Build Details
</a>
</p>
</body>
</html>"#;
/// Default plain text email body template.
pub const BODY_TEXT: &'static str = r#"Build {{ build.status | title }}
==============================
Pipeline: {{ pipeline.name }}
Build ID: {{ build.id }}
Status: {{ build.status }}
Duration: {{ build.duration }}
Commit: {{ commit.hash }}
Author: {{ commit.author }}
Message: {{ commit.message }}
View details: {{ build.url }}"#;
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_email_address_string_to_string() {
let addr = EmailAddress::String("test@example.com".to_string());
assert_eq!(addr.to_string(), "test@example.com");
}
#[test]
fn test_email_address_object_with_name_to_string() {
let addr = EmailAddress::Object {
name: "Test User".to_string(),
email: "test@example.com".to_string(),
};
assert_eq!(addr.to_string(), "Test User <test@example.com>");
}
#[test]
fn test_email_address_object_without_name_to_string() {
let addr = EmailAddress::Object {
name: "".to_string(),
email: "test@example.com".to_string(),
};
assert_eq!(addr.to_string(), "test@example.com");
}
#[test]
fn test_email_address_string_email() {
let addr = EmailAddress::String("test@example.com".to_string());
assert_eq!(addr.email(), "test@example.com");
}
#[test]
fn test_email_address_object_email() {
let addr = EmailAddress::Object {
name: "Test User".to_string(),
email: "test@example.com".to_string(),
};
assert_eq!(addr.email(), "test@example.com");
}
#[test]
fn test_recipients_single_to_vec() {
let recipients = Recipients::Single("test@example.com".to_string());
let vec = recipients.to_vec();
assert_eq!(vec, vec!["test@example.com"]);
}
#[test]
fn test_recipients_multiple_to_vec() {
let recipients = Recipients::Multiple(vec![
"a@example.com".to_string(),
"b@example.com".to_string(),
]);
let vec = recipients.to_vec();
assert_eq!(vec, vec!["a@example.com", "b@example.com"]);
}
#[test]
fn test_mail_config_validate_valid_default_schema() {
let config = MailConfig {
to: Recipients::Single("to@example.com".to_string()),
cc: None,
bcc: None,
from: EmailAddress::String("from@example.com".to_string()),
reply_to: None,
schema: Schema::Default,
subject: None,
body: None,
smtp: SmtpConfig {
host: "smtp.example.com".to_string(),
port: 587,
auth: AuthConfig {
auth_type: AuthType::None,
username: "".to_string(),
password: "".to_string(),
},
encryption: Encryption::Tls,
connect_timeout_ms: 30_000,
},
};
assert!(config.validate().is_ok());
}
#[test]
fn test_mail_config_validate_valid_custom_schema() {
let config = MailConfig {
to: Recipients::Single("to@example.com".to_string()),
cc: None,
bcc: None,
from: EmailAddress::String("from@example.com".to_string()),
reply_to: None,
schema: Schema::Custom,
subject: Some("Test Subject".to_string()),
body: Some("Test Body".to_string()),
smtp: SmtpConfig {
host: "smtp.example.com".to_string(),
port: 587,
auth: AuthConfig {
auth_type: AuthType::None,
username: "".to_string(),
password: "".to_string(),
},
encryption: Encryption::Tls,
connect_timeout_ms: 30_000,
},
};
assert!(config.validate().is_ok());
}
#[test]
fn test_mail_config_validate_custom_schema_missing_subject() {
let config = MailConfig {
to: Recipients::Single("to@example.com".to_string()),
cc: None,
bcc: None,
from: EmailAddress::String("from@example.com".to_string()),
reply_to: None,
schema: Schema::Custom,
subject: None,
body: Some("Test Body".to_string()),
smtp: SmtpConfig {
host: "smtp.example.com".to_string(),
port: 587,
auth: AuthConfig {
auth_type: AuthType::None,
username: "".to_string(),
password: "".to_string(),
},
encryption: Encryption::Tls,
connect_timeout_ms: 30_000,
},
};
let result = config.validate();
assert!(result.is_err());
assert!(result.unwrap_err().iter().any(|e| e.contains("subject")));
}
#[test]
fn test_mail_config_validate_custom_schema_missing_body() {
let config = MailConfig {
to: Recipients::Single("to@example.com".to_string()),
cc: None,
bcc: None,
from: EmailAddress::String("from@example.com".to_string()),
reply_to: None,
schema: Schema::Custom,
subject: Some("Test Subject".to_string()),
body: None,
smtp: SmtpConfig {
host: "smtp.example.com".to_string(),
port: 587,
auth: AuthConfig {
auth_type: AuthType::None,
username: "".to_string(),
password: "".to_string(),
},
encryption: Encryption::Tls,
connect_timeout_ms: 30_000,
},
};
let result = config.validate();
assert!(result.is_err());
assert!(result.unwrap_err().iter().any(|e| e.contains("body")));
}
#[test]
fn test_mail_config_validate_empty_recipients() {
let config = MailConfig {
to: Recipients::Multiple(vec![]),
cc: None,
bcc: None,
from: EmailAddress::String("from@example.com".to_string()),
reply_to: None,
schema: Schema::Default,
subject: None,
body: None,
smtp: SmtpConfig {
host: "smtp.example.com".to_string(),
port: 587,
auth: AuthConfig {
auth_type: AuthType::None,
username: "".to_string(),
password: "".to_string(),
},
encryption: Encryption::Tls,
connect_timeout_ms: 30_000,
},
};
let result = config.validate();
assert!(result.is_err());
assert!(result.unwrap_err().iter().any(|e| e.contains("recipient")));
}
#[test]
fn test_email_address_serde_string() {
let json = r#""test@example.com""#;
let addr: EmailAddress = serde_json::from_str(json).unwrap();
assert!(matches!(addr, EmailAddress::String(s) if s == "test@example.com"));
}
#[test]
fn test_email_address_serde_object() {
let json = r#"{"name":"Test User","email":"test@example.com"}"#;
let addr: EmailAddress = serde_json::from_str(json).unwrap();
assert!(
matches!(addr, EmailAddress::Object { name, email } if name == "Test User" && email == "test@example.com")
);
}
#[test]
fn test_recipients_serde_single() {
let json = r#""single@example.com""#;
let recipients: Recipients = serde_json::from_str(json).unwrap();
assert!(matches!(recipients, Recipients::Single(s) if s == "single@example.com"));
}
#[test]
fn test_recipients_serde_multiple() {
let json = r#"["a@example.com","b@example.com"]"#;
let recipients: Recipients = serde_json::from_str(json).unwrap();
assert!(
matches!(recipients, Recipients::Multiple(v) if v == vec!["a@example.com".to_string(), "b@example.com".to_string()])
);
}
#[test]
fn test_auth_type_serde() {
#[derive(serde::Deserialize, Debug)]
struct Test {
#[serde(rename = "type")]
auth_type: AuthType,
}
let json = r#"{"type":"password"}"#;
let t: Test = serde_json::from_str(json).unwrap();
assert!(matches!(t.auth_type, AuthType::Password));
}
#[test]
fn test_encryption_serde() {
#[derive(serde::Deserialize, Debug)]
struct Test {
encryption: Encryption,
}
let json = r#"{"encryption":"tls"}"#;
let t: Test = serde_json::from_str(json).unwrap();
assert!(matches!(t.encryption, Encryption::Tls));
}
#[test]
fn test_schema_serde() {
#[derive(serde::Deserialize, Debug)]
struct Test {
schema: Schema,
}
let json = r#"{"schema":"custom"}"#;
let t: Test = serde_json::from_str(json).unwrap();
assert!(matches!(t.schema, Schema::Custom));
}
}
@@ -1,101 +0,0 @@
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
use crate::ExecutionContext;
use minijinja::Environment;
use std::collections::HashMap;
/// Builds template context data from execution context.
/// Populates pipeline, build, and commit information for template rendering.
pub fn prepare_template_data(ctx: &ExecutionContext) -> HashMap<String, serde_json::Value> {
let mut data = HashMap::new();
data.insert(
"pipeline".to_string(),
serde_json::json!({
"name": ctx.pipeline_name,
}),
);
data.insert(
"build".to_string(),
serde_json::json!({
"id": ctx.task_id,
"status": "success",
"status_color": "green",
"duration": "0s",
"url": "",
}),
);
data.insert(
"commit".to_string(),
serde_json::json!({
"hash": "",
"short_hash": "",
"message": "",
"author": "",
"author_email": "",
}),
);
data
}
/// Renders a minijinja template string with the provided data map.
/// Returns the rendered string on success, or a String error message on failure.
pub fn render_template(
template: &str,
data: &HashMap<String, serde_json::Value>,
) -> Result<String, String> {
let env = Environment::new();
let tmpl = env
.template_from_str(template)
.map_err(|e| format!("Template parse error: {}", e))?;
let result = tmpl
.render(data)
.map_err(|e| format!("Template render error: {}", e))?;
Ok(result)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_render_template_simple() {
let template = "Hello {{ name }}!";
let mut data = HashMap::new();
data.insert(
"name".to_string(),
serde_json::Value::String("World".to_string()),
);
let result = render_template(template, &data).unwrap();
assert_eq!(result, "Hello World!");
}
#[test]
fn test_render_template_with_build_status() {
let template = "Build {{ build.status }} for pipeline {{ pipeline.name }}";
let mut data = HashMap::new();
data.insert(
"build".to_string(),
serde_json::json!({ "status": "success" }),
);
data.insert(
"pipeline".to_string(),
serde_json::json!({ "name": "test-pipeline" }),
);
let result = render_template(template, &data).unwrap();
assert_eq!(result, "Build success for pipeline test-pipeline");
}
#[test]
fn test_render_template_invalid_syntax() {
let template = "{{ Unterminated";
let data = HashMap::new();
let result = render_template(template, &data);
assert!(result.is_err());
}
}
@@ -1,6 +1,7 @@
use crate::finalize::plugin::PluginError; use crate::finalize::plugin::PluginError;
use crate::finalize::plugin::PluginMetadata; use crate::finalize::plugin::PluginMetadata;
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn}; use workshop_engine::{EventSender, ExecutionContext};
use crate::{ finalize::plugin::AsyncPluginFn};
use async_trait::async_trait; use async_trait::async_trait;
pub struct Satori; pub struct Satori;
@@ -1,6 +1,7 @@
use crate::finalize::plugin::PluginError; use crate::finalize::plugin::PluginError;
use crate::finalize::plugin::PluginMetadata; use crate::finalize::plugin::PluginMetadata;
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn}; use workshop_engine::{EventSender, ExecutionContext};
use crate::{ finalize::plugin::AsyncPluginFn};
use async_trait::async_trait; use async_trait::async_trait;
pub struct Webhook; pub struct Webhook;
+7 -74
View File
@@ -1,3 +1,4 @@
use crate::finalize::NotificationTemplateDef;
use crate::notify::types::NotificationRenderer; use crate::notify::types::NotificationRenderer;
use serde::Deserialize; use serde::Deserialize;
use std::path::PathBuf; use std::path::PathBuf;
@@ -18,10 +19,14 @@ pub struct PluginMetadata {
#[serde(default)] #[serde(default)]
pub renderer: Option<NotificationRenderer>, pub renderer: Option<NotificationRenderer>,
/// Additional YAML fields not captured by struct. /// Additional YAML fields.
#[serde(flatten)] #[serde(flatten)]
pub value: serde_yaml::Value, pub value: serde_yaml::Value,
/// Plugin-defined default templates, auto-registered as <name>.fallback
#[serde(default)]
pub templates: Option<Vec<NotificationTemplateDef>>,
/// Filesystem path to plugin definition. /// Filesystem path to plugin definition.
#[serde(skip)] #[serde(skip)]
pub fspath: PathBuf, pub fspath: PathBuf,
@@ -36,6 +41,7 @@ impl Default for PluginMetadata {
plugin_type: PluginType::Unknown, plugin_type: PluginType::Unknown,
value: serde_yaml::Value::Null, value: serde_yaml::Value::Null,
renderer: None, renderer: None,
templates: None,
fspath: PathBuf::new(), fspath: PathBuf::new(),
} }
} }
@@ -50,76 +56,3 @@ pub enum PluginType {
Rhai, Rhai,
Dylib, Dylib,
} }
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_plugin_metadata_default() {
let meta = PluginMetadata::default();
assert_eq!(meta.name, "");
assert_eq!(meta.version, "");
assert_eq!(meta.entrypoint, "");
assert_eq!(meta.plugin_type, PluginType::Unknown);
}
#[test]
fn test_plugin_type_default() {
let pt = PluginType::default();
assert_eq!(pt, PluginType::Unknown);
}
#[test]
fn test_plugin_type_equality() {
assert_eq!(PluginType::Shell, PluginType::Shell);
assert_eq!(PluginType::Dylib, PluginType::Dylib);
assert_ne!(PluginType::Shell, PluginType::Dylib);
}
#[test]
fn test_plugin_metadata_clone() {
let meta = PluginMetadata {
name: "test".to_string(),
version: "1.0".to_string(),
entrypoint: "/bin/test".to_string(),
plugin_type: PluginType::Shell,
value: serde_yaml::Value::Null,
fspath: PathBuf::from("/path/to/plugin"),
renderer: None,
};
let cloned = meta.clone();
assert_eq!(cloned.name, meta.name);
assert_eq!(cloned.version, meta.version);
assert_eq!(cloned.plugin_type, meta.plugin_type);
}
#[test]
fn test_plugin_type_serde_unknown() {
let yaml = "Unknown";
let pt: PluginType = serde_yaml::from_str(yaml).unwrap();
assert_eq!(pt, PluginType::Unknown);
}
#[test]
fn test_plugin_type_serde_shell() {
let yaml = "Shell";
let pt: PluginType = serde_yaml::from_str(yaml).unwrap();
assert_eq!(pt, PluginType::Shell);
}
#[test]
fn test_plugin_metadata_serde() {
let yaml = r#"
name: test-plugin
version: "2.0"
entrypoint: ./bin/test
plugin_type: Shell
"#;
let meta: PluginMetadata = serde_yaml::from_str(yaml).unwrap();
assert_eq!(meta.name, "test-plugin");
assert_eq!(meta.version, "2.0");
assert_eq!(meta.entrypoint, "./bin/test");
assert_eq!(meta.plugin_type, PluginType::Shell);
}
}
+2 -1
View File
@@ -1,6 +1,7 @@
use crate::finalize::plugin::PluginError; use crate::finalize::plugin::PluginError;
use crate::finalize::plugin::PluginMetadata; use crate::finalize::plugin::PluginMetadata;
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn}; use workshop_engine::{EventSender, ExecutionContext};
use crate::{ finalize::plugin::AsyncPluginFn};
use async_trait::async_trait; use async_trait::async_trait;
pub struct Rhai; pub struct Rhai;
+3 -2
View File
@@ -1,8 +1,9 @@
use crate::Engine; use workshop_engine::Engine;
use crate::finalize::constant::FINALIZE_SHELL_ENVPREFIX; use crate::finalize::constant::FINALIZE_SHELL_ENVPREFIX;
use crate::finalize::plugin::PluginError; use crate::finalize::plugin::PluginError;
use crate::finalize::plugin::PluginMetadata; use crate::finalize::plugin::PluginMetadata;
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn}; use workshop_engine::{EventSender, ExecutionContext};
use crate::{ finalize::plugin::AsyncPluginFn};
use async_trait::async_trait; use async_trait::async_trait;
use std::collections::HashMap; use std::collections::HashMap;
+1 -1
View File
@@ -1,4 +1,4 @@
use crate::ExecutionContext; use workshop_engine::ExecutionContext;
use crate::finalize::plugin::{PluginMap, call_named_plugin}; use crate::finalize::plugin::{PluginMap, call_named_plugin};
use std::path::PathBuf; use std::path::PathBuf;
use workshop_engine::{CustomCommand, DeltaExecutionContext, Engine, EventSender, ExecutionError}; use workshop_engine::{CustomCommand, DeltaExecutionContext, Engine, EventSender, ExecutionError};
+25 -57
View File
@@ -1,27 +1,16 @@
// Library crate for workshop-executor pub mod bare;
// Selective exports: only modules used by external callers or tests
pub mod bake; pub mod bake;
pub mod daemon; pub mod error;
pub mod finalize; pub mod finalize;
pub mod monitor;
pub mod notify; pub mod notify;
pub mod prebake; pub mod prebake;
pub mod socket;
pub mod types; pub mod types;
pub mod utils;
// External crates re-exports
pub use config;
// Re-export commonly used types from workshop-engine
pub use workshop_engine::{Engine, EventSender, ExecutionContext, ExecutionError, ExecutionResult};
// CLI types used by main (keep here since they're used in daemon)
pub use clap::{Parser, Subcommand}; pub use clap::{Parser, Subcommand};
pub use serde::Deserialize; pub use serde::Deserialize;
pub use std::collections::HashMap; pub use std::collections::HashMap;
pub use std::path::PathBuf; pub use std::path::PathBuf;
pub mod cli { pub mod cli {
pub use super::{Deserialize, HashMap, Parser, PathBuf, Subcommand}; pub use super::{Deserialize, HashMap, Parser, PathBuf, Subcommand};
pub use clap::Command; pub use clap::Command;
@@ -46,64 +35,43 @@ pub mod cli {
#[arg(long, default_value = "false", env = "HBW_DRY_RUN")] #[arg(long, default_value = "false", env = "HBW_DRY_RUN")]
pub dry_run: bool, pub dry_run: bool,
/// Override prebake config path
#[arg(long)]
pub prebake: Option<PathBuf>,
/// Override bake script path
#[arg(long)]
pub bake: Option<PathBuf>,
/// Override bake_base.sh path
#[arg(long)]
pub bake_base: Option<PathBuf>,
/// Override finalize config path
#[arg(long)]
pub finalize: Option<PathBuf>,
/// Debug features (comma-separated). "notify" enables notification debug logging.
#[arg(long, env = "HBW_DEBUG")]
pub debug: Option<String>,
#[command(subcommand)] #[command(subcommand)]
pub command: Option<Commands>, pub command: Option<Commands>,
} }
#[derive(Subcommand, Debug)] #[derive(Subcommand, Debug)]
pub enum Commands { pub enum Commands {
/// Run pipeline stages in standalone (bare) mode /// Run individual pipeline stages
Bare { Bare {
#[command(subcommand)] #[command(subcommand)]
command: BareCommands, command: BareCommands,
}, },
/// Run in client mode Daemon {},
Client { config: String },
} }
#[derive(Subcommand, Debug)] #[derive(Subcommand, Debug)]
pub enum BareCommands { pub enum BareCommands {
/// Prepare a task Prebake {},
Prebake { config: PathBuf }, Bake {},
Finalize {},
/// Run a task
Bake {
/// Path to the script to execute
script: PathBuf,
/// Path to bake_base.sh
#[arg(short, long, default_value = "./bake_base.sh")]
bake_base: PathBuf,
#[arg(short, long, default_value = "./prebake.yml")]
prebake: PathBuf,
},
/// Finish a task
Finalize { config: String },
} }
#[derive(Debug, Deserialize)]
pub struct TaskSpec {
pub subcommand: String,
pub arguments: Vec<String>,
pub options: HashMap<String, String>,
}
impl TaskSpec {
pub fn command(&self, executable: &PathBuf) -> tokio::process::Command {
let mut command = tokio::process::Command::new(executable);
command.arg(&self.subcommand);
for arg in &self.arguments {
command.arg(arg);
}
for (flag, value) in self.options.iter() {
if value.is_empty() {
command.arg(format!("--{}", flag));
} else {
command.arg(format!("--{}={}", flag, value));
}
}
command
}
}
} }
+110 -63
View File
@@ -1,13 +1,91 @@
use std::path::Path; use workshop_baker::types::resource::ResourceRegistry;
use std::path::PathBuf;
use workshop_baker::{ use workshop_baker::{
bake::bake, bake::bake, cli::{Cli, Commands, Parser},
cli::{BareCommands, Cli, Commands, Parser}, error::CliError,
daemon, finalize::finalize, notify::notify, prebake::prebake,
finalize::finalize,
prebake::prebake,
}; };
use workshop_engine::{EventSender, ExecutionContext, error::HasExitCode}; use workshop_baker::bare::run_bare;
use workshop_baker::notify::queue::NotificationQueue;
use workshop_engine::{EventSender, ExecutionContext};
fn or_workshop(path: &Option<PathBuf>, name: &str) -> PathBuf {
path.clone().unwrap_or_else(|| PathBuf::from(format!(".workshop/{}", name)))
}
async fn worker_main(cli: Cli) -> Result<(), CliError> {
let prebake_path = cli.prebake.clone().unwrap_or_else(|| {
log::error!("--prebake is required in worker standalone mode");
std::process::exit(1);
});
let bake_path = cli.bake.clone().unwrap_or_else(|| {
log::error!("--bake is required in worker standalone mode");
std::process::exit(1);
});
let finalize_path = cli.finalize.clone().unwrap_or_else(|| {
log::error!("--finalize is required in worker standalone mode");
std::process::exit(1);
});
log::info!("Worker standalone mode");
let mut ctx = ExecutionContext {
standalone: true,
task_id: format!("{}-{}-worker", cli.pipeline, cli.build_id),
pipeline_name: cli.pipeline.clone(),
username: cli.username.clone(),
dry_run: cli.dry_run,
privileged: false,
..Default::default()
};
let (tx, event_rx) = tokio::sync::mpsc::channel(256);
let event_tx: EventSender = Some(tx);
let _event_handle = tokio::spawn(workshop_baker::prebake::event::event_receiver(
event_rx, cli.pipeline.clone(), cli.build_id.clone(),
));
let (to_tx, to_rx) = tokio::sync::mpsc::channel(256);
let (retry_tx, retry_rx) = tokio::sync::mpsc::channel(256);
// Resource registry — populated before stages, consumed by stages
let registry = ResourceRegistry::new();
// TODO: pre-fetch plugins and templates into registry
let notify_path = finalize_path.clone();
let notify_ctx = ctx.clone();
let notify_etx = event_tx.clone();
let debug = cli.debug.clone();
let notify_registry = registry.clone();
let notify_handle = tokio::spawn(async move {
notify(&notify_path, &mut notify_ctx.clone(), notify_etx, to_rx, retry_tx, debug.map(|d| d.contains("notify")).unwrap_or(false), &notify_registry).await
});
let nevent_tx = to_tx.clone();
let mut queue = NotificationQueue::new(to_tx, retry_rx);
let pipe_result: anyhow::Result<()> = async {
prebake(&prebake_path, &cli, None, &mut ctx, event_tx.clone(), Some(nevent_tx)).await?;
let bake_base_path = or_workshop(&cli.bake_base, "bake_base.sh");
let pb_path = or_workshop(&cli.prebake, "prebake.yml");
bake(&bake_path, &bake_base_path, &pb_path, &cli, &mut ctx, event_tx.clone()).await?;
finalize(&finalize_path, &cli, &mut ctx, event_tx.clone(), &registry).await?;
Ok(())
}.await;
if let Err(e) = pipe_result {
log::error!("Pipeline stage failed: {}", e);
return Err(CliError::General(e));
}
log::info!("Pipeline complete. Draining notifications...");
if let Err(e) = queue.drain().await {
log::error!("Notification queue error: {}", e);
}
let _ = notify_handle.await;
Ok(())
}
#[tokio::main] #[tokio::main]
async fn main() { async fn main() {
@@ -16,77 +94,46 @@ async fn main() {
if cli.dry_run { if cli.dry_run {
log::warn!("Dry run enabled, no changes will be made."); log::warn!("Dry run enabled, no changes will be made.");
} }
match &cli.command {
Some(Commands::Client { config: _ }) => {
unimplemented!("Client mode not yet implemented.")
}
Some(Commands::Bare { command }) => {
log::info!("Running in standalone mode.");
let (stage, privileged) = match command { let pipeline = cli.pipeline.clone();
BareCommands::Prebake { .. } => ("init", true), let build_id = cli.build_id.clone();
BareCommands::Bake { .. } => ("bake", false),
BareCommands::Finalize { .. } => ("finalize", false), match &cli.command {
}; Some(Commands::Bare { command }) => {
let mut ctx = ExecutionContext { let mut ctx = ExecutionContext {
standalone: true, standalone: true,
task_id: format!("{}-{}-{}", cli.pipeline, cli.build_id, stage), task_id: format!("{}-{}-bare", pipeline, build_id),
pipeline_name: cli.pipeline.clone(), pipeline_name: pipeline.clone(),
username: cli.username.clone(), username: cli.username.clone(),
dry_run: cli.dry_run, dry_run: cli.dry_run,
privileged, privileged: false,
..Default::default() ..Default::default()
}; };
let (tx, event_rx) = tokio::sync::mpsc::channel(256); let (tx, event_rx) = tokio::sync::mpsc::channel(256);
let event_tx: EventSender = Some(tx); let event_tx: EventSender = Some(tx);
let handle = tokio::spawn(workshop_baker::prebake::event::event_receiver( let event_handle = tokio::spawn(workshop_baker::prebake::event::event_receiver(
event_rx, event_rx, pipeline.clone(), build_id.clone(),
cli.pipeline.clone(),
cli.build_id.clone(),
)); ));
let result: anyhow::Result<()> = match command { let registry = ResourceRegistry::new();
BareCommands::Prebake { config } => { let result = run_bare(&cli, command, &mut ctx, event_tx, &pipeline, &build_id, &registry).await;
prebake(config, &cli, None, &mut ctx, event_tx).await
}
BareCommands::Bake {
script,
bake_base,
prebake: prebake_path,
} => bake(script, bake_base, prebake_path, &cli, &mut ctx, event_tx)
.await
.map_err(|e| e.into()),
BareCommands::Finalize { config } => {
finalize(Path::new(config), &cli, &mut ctx, event_tx)
.await
.map_err(|e| e.into())
}
};
drop(ctx); drop(ctx);
let _ = handle.await; let _ = event_handle.await;
dbg!(&result);
if let Err(ref e) = result { if let Err(e) = result {
let code = match command { log::error!("{}", e);
BareCommands::Prebake { .. } => e std::process::exit(e.exit_code());
.downcast_ref::<workshop_baker::prebake::error::PrebakeError>()
.map(|pe| pe.exit_code())
.unwrap_or(1),
BareCommands::Bake { .. } => e
.downcast_ref::<workshop_baker::bake::error::BakeError>()
.map(|be| be.exit_code())
.unwrap_or(1),
BareCommands::Finalize { .. } => e
.downcast_ref::<workshop_baker::finalize::FinalizeError>()
.map(|fe| fe.exit_code())
.unwrap_or(1),
};
log::error!("{} stage failed: {}", stage, e);
std::process::exit(code);
} }
} }
Some(Commands::Daemon {}) => {
unimplemented!("Daemon mode not implemented");
}
None => { None => {
log::info!("Executor will be running in daemon mode."); if let Err(e) = worker_main(cli).await {
let _result = daemon::daemon(&cli).await; log::error!("{}", e);
std::process::exit(e.exit_code());
}
} }
} }
} }
-89
View File
@@ -1,89 +0,0 @@
mod cgroups;
mod jobobject;
use crate::socket::{establish_connection, get_socket_addr};
use config::Config;
use serde::{Deserialize, Serialize};
use std::time::Instant;
use tokio::io::AsyncWriteExt;
#[derive(Debug, Serialize, Deserialize)]
pub struct ResourceUsageInfo {
pub msgtype: String,
pub name: String,
pub os_type: OsType,
pub statistics: UsageStats,
pub timestamp: u64,
}
#[derive(Debug, Serialize, Deserialize)]
pub enum OsType {
Linux,
Windows,
MacOS,
}
#[derive(Debug, Serialize, Deserialize, Clone)]
pub struct UsageStats {
pub cpu_usage_percent: f64,
pub memory_usage_bytes: u64,
pub memory_limit_bytes: i64,
// pub memory_usage_percent: f64,
pub process_count: u64,
pub cpu_usage_us: u64,
/// Internal timestamp, using std::Instant
#[serde(skip, default = "default_internal_timestamp")]
_timestamp: Instant,
}
fn default_internal_timestamp() -> Instant {
Instant::now()
}
pub async fn monitor(
name: &String,
interval_ms: Option<u64>,
settings: Config,
) -> anyhow::Result<()> {
// let socket_addr = get_wsagent_socket_addr()?;
let socket_addr = get_socket_addr(settings.get_string("socket")?)?;
let interval = tokio::time::Duration::from_millis(interval_ms.unwrap_or(1000));
let mut socket = establish_connection(&socket_addr).await?;
let mut prev_usage = UsageStats {
cpu_usage_percent: 0.0,
memory_usage_bytes: 0,
memory_limit_bytes: 0,
process_count: 0,
cpu_usage_us: 0,
_timestamp: default_internal_timestamp(),
};
loop {
let usage: UsageStats = cgroups::get_usage(name, prev_usage)?; // TODO: 多系统支持?
prev_usage = usage.clone();
// dbg!(&usage);
let usage_info = ResourceUsageInfo {
msgtype: "ResourceUsage".to_string(),
name: name.clone(),
os_type: OsType::Linux, // TODO: 多系统支持?
statistics: usage,
timestamp: std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)?
.as_millis() as u64,
};
let usage_json = serde_json::to_string(&usage_info)?;
send_usage_info(&mut socket, &usage_json).await?;
// TODO: 增加消息接收,提供(伪)双向通信功能
tokio::time::sleep(interval).await;
}
}
async fn send_usage_info<S>(stream: &mut S, usage_json: &str) -> anyhow::Result<()>
where
S: tokio::io::AsyncWrite + Unpin,
{
stream.write_all(usage_json.as_bytes()).await?;
stream.write_all(b"\n").await?;
Ok(())
}
+424 -356
View File
@@ -1,400 +1,468 @@
use crate::finalize::FinalizeConfig;
use crate::finalize::NotificationConfig;
use crate::finalize::NotificationTemplate;
use crate::finalize::config::TriggerItemRaw;
use crate::finalize::constant::FINALIZE_STANDALONE_PLUGIN_PATH;
use crate::finalize::parse; use crate::finalize::parse;
use crate::finalize::plugin; use crate::finalize::plugin;
use crate::finalize::plugin::fetch; use crate::notify::types::DEFAULT_NOTIFY_PARAMS;
use crate::notify::types::Matchable;
use crate::notify::types::Method;
use crate::notify::types::NOTIFICATION_DEFAULT_FALLIBLE;
use crate::notify::types::NOTIFICATION_DEFAULT_MAX_LIVES;
use crate::notify::types::NOTIFICATION_DEFAULT_RETRY_BACKOFF;
use crate::notify::types::NOTIFICATION_DEFAULT_TIMEOUT_MS;
use crate::notify::types::NOTIFICATION_NOTREADY_DELAY;
use crate::notify::types::NOTIFICATION_TEMPORARY_DELAY_FACTOR;
use crate::notify::types::NOTIFICATION_TEMPORARY_DELAY_TIME;
use crate::notify::types::NOTIFICATION_TEMPORARY_MAX_TIME;
use crate::notify::types::NotificationEvent;
use crate::notify::types::NotificationEventReceiver; use crate::notify::types::NotificationEventReceiver;
use crate::notify::types::NotificationEventSender; use crate::notify::types::NotificationEventSender;
use crate::notify::types::NotificationPluginMap; use crate::notify::handler::NotificationHandler;
use crate::notify::types::NotificationRenderer; use crate::types::resource::ResourceRegistry;
use crate::notify::types::NotificationRule;
use crate::notify::types::NotificationRulesMap;
use crate::notify::types::ParsedTrigger;
use crate::{EventSender, ExecutionContext};
use chrono::Utc;
use globset::{Glob, GlobSetBuilder};
use std::collections::HashMap;
use std::collections::HashSet;
use std::path::Path; use std::path::Path;
use std::path::PathBuf; use std::sync::Arc;
use workshop_engine::{EventSender, ExecutionContext};
mod error; mod error;
pub mod handler;
pub mod queue;
pub mod rule;
pub mod template;
pub mod types; pub mod types;
use error::NotifyError; pub mod util;
use rule::parse_rules;
fn validate(finalize: &FinalizeConfig) -> anyhow::Result<()> { use template::{flatten_template_refs, resolve_external_template_refs};
let validate_result = finalize.validate(); use util::{extract_plugin, validate};
if let Err(errors) = validate_result {
for error in &errors {
log::error!("Error: {}", error);
}
anyhow::bail!(NotifyError::FinalizeValidationError(errors))
};
Ok(())
}
fn extract_plugin(finalize: &FinalizeConfig) -> HashSet<String> {
finalize
.notification
.groups
.values()
.flat_map(|methods| methods.keys())
.cloned()
.collect()
}
pub async fn notify( pub async fn notify(
finalize_path: &Path, finalize_path: &Path,
ctx: &mut ExecutionContext, ctx: &mut ExecutionContext,
_event_tx: EventSender, _event_tx: EventSender,
notifyevent_rx: &mut NotificationEventReceiver, mut rx: NotificationEventReceiver,
notifyevent_tx: NotificationEventSender, retry_tx: NotificationEventSender,
_debug: bool,
registry: &ResourceRegistry,
) -> anyhow::Result<()> { ) -> anyhow::Result<()> {
let mut finalize = parse(finalize_path)?; let mut finalize = parse(finalize_path)?;
// Always inject in-site-notify as the lowest priority fallback
{
let max_p = finalize.notification.groups.keys().max().copied().unwrap_or(0);
let mut m = std::collections::HashMap::new();
m.insert(
"in-site-notify".to_string(),
crate::finalize::NotificationMethod::default(),
);
finalize.notification.groups.insert(max_p + 1, m);
}
validate(&finalize)?; validate(&finalize)?;
let plugin_list = extract_plugin(&finalize); let plugin_list = extract_plugin(&finalize);
if plugin_list.is_empty() { let notify_plugins = plugin::register(
log::warn!("No notification method provided, skipping notification"); finalize.plugin,
// TODO: send event to event_tx
return Ok(());
}
if ctx.standalone {
fetch::fetch_plugins(
&PathBuf::from(FINALIZE_STANDALONE_PLUGIN_PATH),
&mut finalize.plugin,
Some(&plugin_list), Some(&plugin_list),
) &mut finalize.notification,
.await?; registry,
} )?;
let notify_plugins = plugin::register(finalize.plugin, Some(&plugin_list))?;
resolve_external_template_refs(&mut finalize.notification.templates, registry);
flatten_template_refs(&mut finalize.notification.templates)?;
let notification_rules = parse_rules(&finalize.notification); let notification_rules = parse_rules(&finalize.notification);
loop { let handler = Arc::new(NotificationHandler::new(
tokio::select! { Arc::new(notification_rules),
Some(event) = notifyevent_rx.recv() => { Arc::new(notify_plugins),
handle_notification_event( Arc::new(finalize.notification.templates),
event, Arc::new(ctx.clone()),
&notify_plugins, retry_tx,
&notification_rules, DEFAULT_NOTIFY_PARAMS.batch_period.value,
&notifyevent_tx, DEFAULT_NOTIFY_PARAMS.batch_watermark.value,
) ));
.await?;
}
}
}
}
/// Process a single notification event. loop {
/// let Some(event) = rx.recv().await else { break };
/// Delegates to [`notification_handler`] and handles errors: let h = handler.clone();
/// - [`NotifyError::PluginNotAvailable`] — reschedule with static delay, lower priority tokio::spawn(async move {
/// - [`NotifyError::TemporaryError`] — consume one life, reschedule with exponential backoff h.process_event(event).await;
/// - [`NotifyError::PermanentError`] — drop the event });
/// - other — log and drop
async fn handle_notification_event(
event: NotificationEvent,
plugins: &NotificationPluginMap,
rules: &NotificationRulesMap,
notifyevent_tx: &NotificationEventSender,
) -> anyhow::Result<()> {
match notification_handler(plugins, rules, &event) {
Ok(_) => {}
Err(e) => {
log::debug!("Failed to send notification: {:?}", e);
match e {
NotifyError::PluginNotAvailable(_) => {
let mut event = event;
event.not_before = Utc::now() + NOTIFICATION_NOTREADY_DELAY;
event.effective_priority = event.effective_priority.saturating_add(1);
notifyevent_tx.send(event).await?;
}
NotifyError::TemporaryError(_) => {
let mut event = event;
event.lives -= 1;
event.not_before = Utc::now() + notify_delay(&event);
notifyevent_tx.send(event).await?;
}
NotifyError::PermanentError(_) => {
log::error!("None of the configured notification plugins works.");
log::error!("Message will be dropped.");
}
error => {
log::error!(
"Internal error: notify_handler returned unexpected error: {:?}",
error
);
}
}
}
} }
Ok(()) Ok(())
} }
fn notification_handler( // TODO: PIPELINE_NOTIFICATION_EXHAUSTED — set when any method returns Exhausted
plugins: &NotificationPluginMap, // TODO: PIPELINE_NOTIFICATION_FAILED — set when any method returns PermanentError
config: &NotificationRulesMap, // Markers sent via event_tx as ExecutionEvent variants for future WebUI/CLI display.
event: &NotificationEvent,
) -> Result<(), NotifyError> {
for (_priority, ruleset) in config.iter() {
for (method, rules) in ruleset.iter() {
if !event.target.contains(method) {
log::trace!("target does not contain method name: {}", method);
continue;
}
for (rule_id, rule) in rules.iter().enumerate() {
if !rule.enabled {
log::trace!("rule is not enabled: {}", rule_id);
continue;
}
if rule.accept(&event.outcome, &event.stage, &event.substage) {
let plugin = plugins
.get(method)
.ok_or(NotifyError::UnknownPlugin(method.clone()))?;
let renderer = plugin.metadata.renderer.clone().unwrap_or_default();
match renderer {
NotificationRenderer::Server => {
// ...
todo!();
}
NotificationRenderer::Plugin => {
unimplemented!();
return Err(NotifyError::UnsupportedRenderer(method.clone(), renderer));
}
NotificationRenderer::Interactive => {
unimplemented!();
return Err(NotifyError::UnsupportedRenderer(method.clone(), renderer));
}
}
}
}
}
}
todo!(); #[cfg(test)]
} mod tests {
use super::*;
use crate::finalize::NotificationTemplateDef;
use crate::notify::types::NotificationRule;
use crate::notify::types::NotificationTarget;
use std::collections::HashMap;
use crate::notify::types::NotificationEvent;
use crate::notify::types::NotificationRulesMap;
use crate::notify::error::NotifyError;
use crate::notify::template::resolve_template;
use crate::notify::util::{find_rule, notify_delay, wait_time};
fn parse_trigger_item_raw(item: &TriggerItemRaw) -> Vec<ParsedTrigger> { // ── helper ──
match item {
TriggerItemRaw::Simple(s) => match s.as_str() { fn make_tmpl(on_success: &str, on_failure: &str) -> NotificationTemplateDef {
"on_success" => vec![ParsedTrigger::OnSuccess], NotificationTemplateDef {
"on_failure" => vec![ParsedTrigger::OnFailure], use_: None,
s if s.starts_with("on_finish_of:") => { on_success: if on_success.is_empty() {
let pattern = s.trim_start_matches("on_finish_of:").trim(); None
let mut builder = GlobSetBuilder::new(); } else {
builder.add(Glob::new(pattern).expect("Invalid glob pattern")); Some(on_success.into())
let glob_set = builder.build().expect("Failed to compile glob set");
vec![ParsedTrigger::OnFinishOf(Matchable::Glob(glob_set))]
}
_ => {
log::warn!("Unknown trigger string: {}", s);
vec![]
}
}, },
TriggerItemRaw::Map(map) => { on_failure: if on_failure.is_empty() {
let mut triggers = Vec::new(); None
for (key, patterns) in map {
match key.as_str() {
"on_finish_of" => {
for pattern in patterns {
let mut builder = GlobSetBuilder::new();
builder.add(Glob::new(pattern).expect("Invalid glob pattern"));
let glob_set = builder.build().expect("Failed to compile glob set");
triggers.push(ParsedTrigger::OnFinishOf(Matchable::Glob(glob_set)));
}
}
"on_success" => triggers.push(ParsedTrigger::OnSuccess),
"on_failure" => triggers.push(ParsedTrigger::OnFailure),
_ => {
log::warn!("Unknown trigger key: {}", key);
}
}
}
triggers
}
}
}
fn merge_templates(
method_template: &NotificationTemplate,
policy_template: &Option<serde_yaml::Value>,
) -> NotificationTemplate {
let mut merged = method_template.clone();
if let Some(policy_tmpl) = policy_template
&& let Ok(policy_template) =
serde_yaml::from_value::<NotificationTemplate>(policy_tmpl.clone())
{
if policy_template.schema.is_some() {
merged.schema = policy_template.schema;
}
if policy_template.on_success.is_some() {
merged.on_success = policy_template.on_success;
}
if policy_template.on_failure.is_some() {
merged.on_failure = policy_template.on_failure;
}
if policy_template.on_finish_of.is_some() {
merged.on_finish_of = policy_template.on_finish_of;
}
}
merged
}
fn extract_overrides(overrides: &serde_yaml::Value) -> (bool, u32, std::time::Duration, f64) {
let fallible = overrides
.get("fallible")
.and_then(|v| v.as_bool())
.unwrap_or(NOTIFICATION_DEFAULT_FALLIBLE);
let max_lives = overrides
.get("max_lives")
.and_then(|v| v.as_u64())
.unwrap_or(NOTIFICATION_DEFAULT_MAX_LIVES as u64) as u32;
let timeout_ms = overrides
.get("timeout_ms")
.and_then(|v| v.as_u64())
.unwrap_or(NOTIFICATION_DEFAULT_TIMEOUT_MS);
let retry_backoff = overrides
.get("retry_backoff")
.and_then(|v| v.as_f64())
.unwrap_or(NOTIFICATION_DEFAULT_RETRY_BACKOFF);
(
fallible,
max_lives,
std::time::Duration::from_millis(timeout_ms),
retry_backoff,
)
}
fn parse_rules(config: &NotificationConfig) -> NotificationRulesMap {
let mut rules = NotificationRulesMap::default();
for (priority, group) in &config.groups {
let mut ruleset: HashMap<Method, Vec<NotificationRule>> = HashMap::new();
for (method_name, method) in group {
let mut method_rules: Vec<NotificationRule> = Vec::new();
if method.policy.is_empty() {
let trigger: Vec<ParsedTrigger> = method
.trigger
.iter()
.flat_map(parse_trigger_item_raw)
.collect();
let template = method.template.clone();
let (fallible, max_lives, timeout, retry_backoff) =
extract_overrides(&method.config);
method_rules.push(NotificationRule {
enabled: true,
trigger,
template,
fallible,
max_lives,
timeout,
config: method.config.clone(),
retry_backoff,
});
} else { } else {
for policy in &method.policy { Some(on_failure.into())
let trigger: Vec<ParsedTrigger> = if policy.trigger.is_empty() { },
method on_finish_of: None,
.trigger
.iter()
.flat_map(parse_trigger_item_raw)
.collect()
} else {
policy
.trigger
.iter()
.flat_map(parse_trigger_item_raw)
.collect()
};
let template =
merge_templates(&method.template, &Some(policy.template.clone()));
let method_overrides = extract_overrides(&method.config);
let policy_overrides = extract_overrides(&policy.overrides);
let fallible = if policy.overrides.get("fallible").is_some() {
policy_overrides.0
} else {
method_overrides.0
};
let max_lives = if policy.overrides.get("max_lives").is_some() {
policy_overrides.1
} else {
method_overrides.1
};
let timeout = if policy.overrides.get("timeout_ms").is_some() {
policy_overrides.2
} else {
method_overrides.2
};
let retry_backoff = if policy.overrides.get("retry_backoff").is_some() {
policy_overrides.3
} else {
method_overrides.3
};
method_rules.push(NotificationRule {
enabled: true,
trigger,
template,
fallible,
max_lives,
timeout,
config: method.config.clone(),
retry_backoff,
});
} }
} }
ruleset.insert(method_name.clone(), method_rules); fn make_tmpl_with_use(use_ref: &str) -> NotificationTemplateDef {
NotificationTemplateDef {
use_: Some(use_ref.into()),
on_success: None,
on_failure: None,
on_finish_of: None,
} }
rules.insert(*priority, ruleset);
}
rules
} }
// calculate min(1, k/x)*t as notification grouping period // ═══════════════════════════════════════════
// to avoid excessive notification frequency // resolve_template_content
fn wait_time(group_period: u32, group_watermark: u32, count: u32) -> f32 { // ═══════════════════════════════════════════
if count == 0 {
log::warn!("Unexpected notification count: {}", count); #[test]
return group_period as f32; fn resolve_none_cd_default_found() {
} let mut ws = HashMap::new();
f32::min(1.0, group_watermark as f32 / count as f32) * group_period as f32 ws.insert("mail".into(), make_tmpl("OK", "FAIL"));
let templates = HashMap::new();
let result = resolve_template(&None, "mail", &templates, false, Some(&ws));
assert!(result.is_ok());
assert_eq!(result.unwrap().on_success.unwrap(), "OK");
} }
// calculate p^(m-l)*t as notification delay #[test]
fn notify_delay(event: &NotificationEvent) -> std::time::Duration { fn resolve_none_cd_default_missing_fallback() {
let exponent = (event.max_lives - event.lives) as i32; let ws: HashMap<String, NotificationTemplateDef> = HashMap::new();
if exponent < 0 { let mut templates = HashMap::new();
log::warn!( templates.insert("mail.fallback".into(), make_tmpl("FALLBACK", ""));
"Unexpected lives in notification: lives({}) > max_lives({})", let result = resolve_template(&None, "mail", &templates, false, Some(&ws));
event.lives, assert!(result.is_ok());
event.max_lives assert_eq!(result.unwrap().on_success.unwrap(), "FALLBACK");
}
#[test]
fn resolve_none_cd_no_default_no_fallback_error() {
let ws: HashMap<String, NotificationTemplateDef> = HashMap::new();
let templates = HashMap::new();
let result = resolve_template(&None, "mail", &templates, false, Some(&ws));
assert!(result.is_err());
assert!(format!("{}", result.unwrap_err()).contains("failed"));
}
#[test]
fn resolve_none_standalone_fallback() {
let mut templates = HashMap::new();
templates.insert("mail.fallback".into(), make_tmpl("STANDALONE", ""));
let result = resolve_template(&None, "mail", &templates, true, None);
assert!(result.is_ok());
assert_eq!(result.unwrap().on_success.unwrap(), "STANDALONE");
}
#[test]
fn resolve_none_standalone_no_fallback_error() {
let result = resolve_template(&None, "mail", &HashMap::new(), true, None);
assert!(result.is_err());
}
#[test]
fn resolve_custom() {
let result = resolve_template(&Some("custom".into()), "mail", &HashMap::new(), false, None);
assert!(result.is_ok());
// "custom" returns an empty default — caller reads rule.template.on_* directly
}
#[test]
fn resolve_default_standalone_fallback() {
let mut templates = HashMap::new();
templates.insert("mail.fallback".into(), make_tmpl("FB", ""));
let result = resolve_template(&Some("default".into()), "mail", &templates, true, None);
assert!(result.is_ok());
assert_eq!(result.unwrap().on_success.unwrap(), "FB");
}
#[test]
fn resolve_default_standalone_no_fallback_error() {
let result = resolve_template(&Some("default".into()), "mail", &HashMap::new(), true, None);
assert!(result.is_err());
}
#[test]
fn resolve_default_cd_wsbase() {
let mut ws = HashMap::new();
ws.insert("mail".into(), make_tmpl("WS_OK", ""));
let result = resolve_template(
&Some("default".into()),
"mail",
&HashMap::new(),
false,
Some(&ws),
); );
return std::time::Duration::from_secs_f64(NOTIFICATION_TEMPORARY_MAX_TIME); assert!(result.is_ok());
assert_eq!(result.unwrap().on_success.unwrap(), "WS_OK");
} }
let multiplier = NOTIFICATION_TEMPORARY_DELAY_FACTOR.powi(exponent);
let delay = multiplier * NOTIFICATION_TEMPORARY_DELAY_TIME; #[test]
if !delay.is_finite() || delay < 0.0 { fn resolve_default_cd_no_wsbase_error() {
log::warn!( let result = resolve_template(
"Unexpected calculated delay: {}, treating as infinite", &Some("default".into()),
delay "mail",
&HashMap::new(),
false,
None,
); );
return std::time::Duration::from_secs_f64(NOTIFICATION_TEMPORARY_MAX_TIME); assert!(result.is_err());
}
#[test]
fn resolve_named_found() {
let mut templates = HashMap::new();
templates.insert("sms-tmpl1".into(), make_tmpl("NAMED", ""));
let result = resolve_template(&Some("sms-tmpl1".into()), "mail", &templates, false, None);
assert!(result.is_ok());
assert_eq!(result.unwrap().on_success.unwrap(), "NAMED");
}
#[test]
fn resolve_named_not_found_method_match_fallback() {
let mut templates = HashMap::new();
templates.insert("mail.fallback".into(), make_tmpl("FB", ""));
let result = resolve_template(&Some("mail".into()), "mail", &templates, false, None);
assert!(result.is_ok());
assert_eq!(result.unwrap().on_success.unwrap(), "FB");
}
#[test]
fn resolve_named_not_found_no_match_error() {
let result = resolve_template(&Some("ghost".into()), "mail", &HashMap::new(), false, None);
assert!(result.is_err());
}
// ═══════════════════════════════════════════
// flatten_template_refs
// ═══════════════════════════════════════════
#[test]
fn flatten_no_refs_unchanged() {
let mut templates = HashMap::new();
templates.insert("a".into(), make_tmpl("A_OK", "A_FAIL"));
let result = flatten_template_refs(&mut templates);
assert!(result.is_ok());
assert_eq!(templates["a"].on_success.as_deref(), Some("A_OK"));
assert!(templates["a"].use_.is_none());
}
#[test]
fn flatten_single_ref_merged() {
let mut templates = HashMap::new();
templates.insert("base".into(), make_tmpl("BASE_OK", "BASE_FAIL"));
templates.insert("child".into(), make_tmpl_with_use("base"));
let result = flatten_template_refs(&mut templates);
assert!(result.is_ok());
assert_eq!(templates["child"].on_success.as_deref(), Some("BASE_OK"));
assert_eq!(templates["child"].on_failure.as_deref(), Some("BASE_FAIL"));
assert!(templates["child"].use_.is_none());
}
#[test]
fn flatten_child_overrides_parent() {
let mut templates = HashMap::new();
templates.insert("base".into(), make_tmpl("BASE_OK", "BASE_FAIL"));
templates.insert(
"child".into(),
NotificationTemplateDef {
use_: Some("base".into()),
on_success: Some("MY_OK".into()),
on_failure: None,
on_finish_of: None,
},
);
let result = flatten_template_refs(&mut templates);
assert!(result.is_ok());
assert_eq!(templates["child"].on_success.as_deref(), Some("MY_OK"));
assert_eq!(templates["child"].on_failure.as_deref(), Some("BASE_FAIL"));
}
#[test]
fn flatten_chain_a_b_c() {
let mut templates = HashMap::new();
templates.insert("c".into(), make_tmpl("C_OK", "C_FAIL"));
templates.insert("b".into(), make_tmpl_with_use("c"));
templates.insert("a".into(), make_tmpl_with_use("b"));
let result = flatten_template_refs(&mut templates);
assert!(result.is_ok());
assert_eq!(templates["a"].on_success.as_deref(), Some("C_OK"));
assert_eq!(templates["b"].on_success.as_deref(), Some("C_OK"));
}
#[test]
fn flatten_cycle_detected() {
let mut templates = HashMap::new();
templates.insert("x".into(), make_tmpl_with_use("y"));
templates.insert("y".into(), make_tmpl_with_use("x"));
let result = flatten_template_refs(&mut templates);
assert!(result.is_err());
match result.unwrap_err() {
NotifyError::TemplateLoop(_) => {}
other => panic!("expected TemplateLoop, got {:?}", other),
}
}
#[test]
fn flatten_self_ref_detected() {
let mut templates = HashMap::new();
templates.insert("self".into(), make_tmpl_with_use("self"));
let result = flatten_template_refs(&mut templates);
assert!(result.is_err());
match result.unwrap_err() {
NotifyError::TemplateLoop(_) => {}
other => panic!("expected TemplateLoop, got {:?}", other),
}
}
#[test]
fn flatten_nonexistent_ref_error() {
let mut templates = HashMap::new();
templates.insert("orphan".into(), make_tmpl_with_use("ghost"));
let result = flatten_template_refs(&mut templates);
assert!(result.is_err());
}
#[test]
fn flatten_external_url_preserved() {
let mut templates = HashMap::new();
templates.insert(
"ext".into(),
NotificationTemplateDef {
use_: Some("git://example.com/repo@v1".into()),
on_success: None,
on_failure: None,
on_finish_of: None,
},
);
let result = flatten_template_refs(&mut templates);
assert!(result.is_ok());
// External URLs are NOT flattened — use_ preserved
assert_eq!(
templates["ext"].use_.as_deref(),
Some("git://example.com/repo@v1")
);
}
// ═══════════════════════════════════════════
// wait_time
// ═══════════════════════════════════════════
#[test]
fn wait_time_zero_count() {
let w = wait_time(30, 5, 0);
assert_eq!(w, 30.0);
}
#[test]
fn wait_time_at_watermark() {
let w = wait_time(30, 5, 5);
assert_eq!(w, 30.0);
}
#[test]
fn wait_time_below_watermark() {
let w = wait_time(30, 5, 2);
assert!((w - 30.0).abs() < 0.01);
}
#[test]
fn wait_time_above_watermark() {
let w = wait_time(30, 5, 10);
assert!((w - 15.0).abs() < 0.01);
}
#[test]
fn wait_time_far_above_watermark() {
let w = wait_time(30, 5, 100);
assert!((w - 1.5).abs() < 0.01);
}
// ═══════════════════════════════════════════
// notify_delay
// ═══════════════════════════════════════════
fn make_event(count: u32, retry_backoff: f64) -> NotificationEvent {
NotificationEvent {
id: uuid::Uuid::new_v4(),
stage: crate::types::buildstatus::StagePhase::Bake,
substage: "build".into(),
outcome: crate::types::buildstatus::StageOutcome::PipelineSuccess,
not_before: chrono::Utc::now(),
priority: 0,
effective_priority: 0,
attempted: 0,
retry_backoff,
target: NotificationTarget {
priority: 0,
rules: None,
count,
},
}
}
#[test]
fn notify_delay_default_factor() {
let event = make_event(1, 2.0);
let delay = notify_delay(&event);
assert!(delay.as_secs_f64() > 0.0);
}
#[test]
fn notify_delay_larger_factor() {
let event1 = make_event(1, 2.0);
let event2 = make_event(1, 4.0);
let d1 = notify_delay(&event1).as_secs_f64();
let d2 = notify_delay(&event2).as_secs_f64();
assert!(d2 > d1, "larger retry_backoff should produce longer delay");
}
// ═══════════════════════════════════════════
// find_rule
// ═══════════════════════════════════════════
#[test]
fn find_rule_found() {
use crate::finalize::NotificationTemplate;
let rule = NotificationRule {
enabled: true,
trigger: vec![],
template: NotificationTemplate {
schema: None,
on_success: None,
on_failure: None,
on_finish_of: None,
},
fallible: false,
max_lives: 3,
timeout: std::time::Duration::from_secs(30),
config: serde_yaml::Value::Null,
retry_backoff: 2.0,
};
let mut rules_map: NotificationRulesMap = HashMap::new();
rules_map.insert(0, {
let mut m: HashMap<String, Vec<NotificationRule>> = HashMap::new();
m.insert("mail".into(), vec![rule]);
m
});
let result = find_rule(&rules_map, "mail", 0);
assert!(result.is_ok());
}
#[test]
fn find_rule_not_found() {
let rules_map: NotificationRulesMap = HashMap::new();
let result = find_rule(&rules_map, "mail", 0);
assert!(result.is_err());
} }
std::time::Duration::from_secs_f64(delay)
} }
+13 -1
View File
@@ -1,4 +1,4 @@
use crate::notify::NotificationRenderer; use crate::notify::types::NotificationRenderer;
use thiserror::Error; use thiserror::Error;
#[derive(Error, Debug)] #[derive(Error, Debug)]
@@ -23,4 +23,16 @@ pub enum NotifyError {
#[error("Unsupported renderer for {0}: {1}")] #[error("Unsupported renderer for {0}: {1}")]
UnsupportedRenderer(String, NotificationRenderer), UnsupportedRenderer(String, NotificationRenderer),
#[error("Schema error: {0}")]
SchemaError(String),
#[error("Template render error: {0}")]
TemplateRenderError(String),
#[error("Circular template reference: {0}")]
TemplateLoop(String),
#[error("Template reference depth exceeded (max {max}): {name}")]
TemplateDepthExceeded { name: String, max: u32 },
} }
+345
View File
@@ -0,0 +1,345 @@
use std::collections::HashMap;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex};
use std::time::Duration;
use chrono::Utc;
use tokio::sync::Notify;
use crate::finalize::NotificationTemplateDef;
use crate::notify::template::{
build_batch_template_context, render_template, resolve_template,
select_template_str,
};
use crate::notify::types::{
MethodStatus, NotificationEvent,
NotificationEventSender, NotificationPluginMap, NotificationRenderer,
NotificationRulesMap,
};
use crate::notify::util::{
find_rule, get_fallible, is_temporary_error, notify_delay, wait_time,
};
/// Shared state for a single dispatch bucket.
struct BucketEntry {
events: Arc<Mutex<Vec<NotificationEvent>>>,
result: Arc<Mutex<Option<(MethodStatus, Vec<NotificationEvent>)>>>,
done: Arc<AtomicBool>,
notify: Arc<Notify>,
method: String,
rule_id: usize,
_handle: tokio::task::JoinHandle<()>,
}
/// Concurrent notification handler with bucket-based dispatch.
///
/// Events are routed to matching buckets. Buckets cool down and dispatch
/// plugins concurrently. The caller awaits bucket completion, then handles
/// retry/escalation decisions.
pub struct NotificationHandler {
buckets: Mutex<HashMap<(String, usize), BucketEntry>>,
pub config: Arc<NotificationRulesMap>,
plugins: Arc<NotificationPluginMap>,
templates: Arc<HashMap<String, NotificationTemplateDef>>,
ctx: Arc<workshop_engine::ExecutionContext>,
retry_tx: NotificationEventSender,
period: u32,
watermark: u32,
}
impl NotificationHandler {
#[allow(clippy::too_many_arguments)]
pub fn new(
config: Arc<NotificationRulesMap>,
plugins: Arc<NotificationPluginMap>,
templates: Arc<HashMap<String, NotificationTemplateDef>>,
ctx: Arc<workshop_engine::ExecutionContext>,
retry_tx: NotificationEventSender,
period: u32,
watermark: u32,
) -> Self {
NotificationHandler {
buckets: Mutex::new(HashMap::new()),
config,
plugins,
templates,
ctx,
retry_tx,
period,
watermark,
}
}
/// Process a single event: route to buckets, await completion,
/// then handle retry/escalation for each bucket result.
pub async fn process_event(self: Arc<Self>, event: NotificationEvent) {
let entries = self.route_event(&event);
if entries.is_empty() {
return;
}
// Await all buckets — yields CPU
for (_, notify) in &entries {
notify.notified().await;
}
// Collect results and remove entries
let mut results: Vec<(String, usize, Option<(MethodStatus, Vec<NotificationEvent>)>)> =
Vec::new();
{
let mut buckets = self.buckets.lock().unwrap();
for (key, _) in &entries {
if let Some(entry) = buckets.remove(key) {
let r = entry.result.lock().unwrap().take();
results.push((entry.method, entry.rule_id, r));
}
}
}
// Handle retry/escalation per method
for (method, _rule_id, result) in results {
let Some((status, mut events)) = result else { continue };
match &status {
MethodStatus::Success => {
log::debug!("[NOTIFY] {}: success ({} events)", method, events.len());
}
MethodStatus::TemporaryError(_) => {
log::warn!("[NOTIFY] {}: temporary error, retrying", method);
for event in events.iter_mut() {
event.target.rules = Some(vec![method.clone()]);
event.target.count += 1;
event.not_before = Utc::now() + notify_delay(event);
let _ = self.retry_tx.send(event.clone()).await;
}
}
MethodStatus::PermanentError(_) | MethodStatus::Exhausted(_) => {
let fallible = get_fallible(&self.config, &method);
if fallible {
log::error!("[NOTIFY] {}: discard (fallible)", method);
} else {
log::warn!("[NOTIFY] {}: escalating priority", method);
for event in events.iter_mut() {
event.target.priority += 1;
event.target.rules = None;
event.attempted = event.target.count;
event.target.count = 0;
event.effective_priority = event.priority;
let _ = self.retry_tx.send(event.clone()).await;
}
}
}
}
}
}
/// Route a single event to all matching buckets.
///
/// Returns Notify handles for awaiting bucket completion.
fn route_event(
&self,
event: &NotificationEvent,
) -> Vec<((String, usize), Arc<Notify>)> {
let mut matches: Vec<(String, usize)> = Vec::new();
for ruleset in self.config.values() {
for (method, rules) in ruleset.iter() {
if let Some(ref method_rules) = event.target.rules {
if !method_rules.contains(method) {
continue;
}
}
for (rule_id, rule) in rules.iter().enumerate() {
if !rule.enabled {
continue;
}
if rule.accept(&event.outcome, &event.stage, &event.substage) {
matches.push((method.clone(), rule_id));
}
}
}
}
let mut buckets = self.buckets.lock().unwrap();
let mut result: Vec<((String, usize), Arc<Notify>)> = Vec::new();
for key in matches {
if let Some(entry) = buckets.get(&key) {
if entry.done.load(Ordering::Acquire) {
buckets.remove(&key);
}
}
if let Some(entry) = buckets.get(&key) {
entry.events.lock().unwrap().push(event.clone());
result.push((key, entry.notify.clone()));
} else {
let (entry, notify) = Self::spawn_bucket(
&key.0,
key.1,
&self.config,
&self.plugins,
&self.templates,
&self.ctx,
self.period,
self.watermark,
);
entry.events.lock().unwrap().push(event.clone());
result.push((key.clone(), notify));
buckets.insert(key, entry);
}
}
result
}
/// Spawn a bucket task: cooldown → take → dispatch → store → notify.
fn spawn_bucket(
method: &str,
rule_id: usize,
config: &Arc<NotificationRulesMap>,
plugins: &Arc<NotificationPluginMap>,
templates: &Arc<HashMap<String, NotificationTemplateDef>>,
ctx: &Arc<workshop_engine::ExecutionContext>,
period: u32,
watermark: u32,
) -> (BucketEntry, Arc<Notify>) {
let events: Arc<Mutex<Vec<NotificationEvent>>> = Arc::new(Mutex::new(Vec::new()));
let result: Arc<Mutex<Option<(MethodStatus, Vec<NotificationEvent>)>>> =
Arc::new(Mutex::new(None));
let notify = Arc::new(Notify::new());
let done = Arc::new(AtomicBool::new(false));
let ec = events.clone();
let rc = result.clone();
let nc = notify.clone();
let dc = done.clone();
let m = method.to_string();
let c = config.clone();
let p = plugins.clone();
let t = templates.clone();
let x = ctx.clone();
let handle = tokio::spawn(async move {
// Phase 1: Cooldown
let count = ec.lock().unwrap().len() as u32;
let wait_secs = wait_time(period, watermark, count);
tokio::time::sleep(Duration::from_secs_f32(wait_secs)).await;
// Phase 2: Take events
let batch = std::mem::take(&mut *ec.lock().unwrap());
if batch.is_empty() {
dc.store(true, Ordering::Release);
nc.notify_waiters();
return;
}
// Phase 3: Dispatch
let rule = match find_rule(&c, &m, rule_id) {
Ok(r) => r,
Err(_) => {
*rc.lock().unwrap() = Some((MethodStatus::PermanentError("rule not found".into()), batch));
dc.store(true, Ordering::Release);
nc.notify_waiters();
return;
}
};
let plugin = match p.get(&m) {
Some(p) => p,
None => {
*rc.lock().unwrap() = Some((MethodStatus::PermanentError("plugin not found".into()), batch));
dc.store(true, Ordering::Release);
nc.notify_waiters();
return;
}
};
let renderer = plugin.metadata.renderer.clone().unwrap_or_default();
let argument = build_plugin_argument(&renderer, rule, &m, &t, &x, &batch);
let status = match argument {
Some(arg) => {
match tokio::time::timeout(rule.timeout, plugin.call(arg, &x, &None)).await {
Ok(Ok(())) => MethodStatus::Success,
Ok(Err(e)) if is_temporary_error(&e) => {
MethodStatus::TemporaryError(format!("{}", e))
}
Ok(Err(e)) => MethodStatus::PermanentError(format!("{}", e)),
Err(_) => MethodStatus::TemporaryError("timeout".into()),
}
}
None => MethodStatus::PermanentError("argument build failed".into()),
};
*rc.lock().unwrap() = Some((status, batch));
dc.store(true, Ordering::Release);
nc.notify_waiters();
});
let entry = BucketEntry {
events,
result,
done,
notify: notify.clone(),
method: method.to_string(),
rule_id,
_handle: handle,
};
(entry, notify)
}
}
/// Build the YAML argument for a plugin call based on renderer mode.
fn build_plugin_argument(
renderer: &NotificationRenderer,
rule: &crate::notify::types::NotificationRule,
method: &str,
templates: &HashMap<String, NotificationTemplateDef>,
ctx: &workshop_engine::ExecutionContext,
events: &[NotificationEvent],
) -> Option<serde_yaml::Value> {
let first = events.first()?;
match renderer {
NotificationRenderer::Server => {
let template_def = resolve_template(
&rule.template.schema,
method,
templates,
ctx.standalone,
None,
)
.ok()?;
let template_str = select_template_str(&template_def, &first.outcome)
.or_else(|| rule.template.on_success.as_deref())
.or_else(|| rule.template.on_failure.as_deref())
.or_else(|| {
rule.template
.on_finish_of
.as_ref()
.map(|o| o.content.as_str())
})?;
let data = build_batch_template_context(ctx, events);
let rendered = render_template(template_str, &data).ok()?;
let mut argument = rule.config.clone();
if let serde_yaml::Value::Mapping(ref mut map) = argument {
map.insert(
serde_yaml::Value::String("_rendered_content".into()),
serde_yaml::Value::String(rendered),
);
}
log::info!("SSR: rendered content for method '{}'", method);
Some(argument)
}
NotificationRenderer::Plugin => {
unimplemented!("Plugin renderer is not yet implemented")
}
NotificationRenderer::Interactive => {
unimplemented!("Interactive renderer is not yet implemented")
}
}
}
+86
View File
@@ -0,0 +1,86 @@
use crate::notify::types::{NotificationEvent, NotificationEventReceiver, NotificationEventSender};
use std::collections::BinaryHeap;
/// Priority queue for notification events.
///
/// Integrates with `notify()`:
/// - `tx` sends events to notify() for processing
/// - `rx` receives retry events back from notify()
pub struct NotificationQueue {
pub pq: BinaryHeap<NotificationEvent>,
pub tx: NotificationEventSender,
pub rx: NotificationEventReceiver,
}
impl NotificationQueue {
pub fn new(tx: NotificationEventSender, rx: NotificationEventReceiver) -> Self {
Self {
pq: BinaryHeap::new(),
tx,
rx,
}
}
/// Push an event into the priority queue.
pub fn enqueue(&mut self, event: NotificationEvent) {
self.pq.push(event);
}
/// Non-blocking single round: recycle retry events into PQ, then
/// pop and send all ready events (not_before ≤ now).
pub async fn flush_now(&mut self) -> anyhow::Result<()> {
// Recycle retry events
while let Ok(event) = self.rx.try_recv() {
self.pq.push(event);
}
// Send ready events
let now = chrono::Utc::now();
while let Some(peek) = self.pq.peek() {
if peek.not_before > now {
break;
}
let event = self.pq.pop().unwrap();
self.tx.send(event).await?;
}
Ok(())
}
/// Blocking drain until all events are processed.
/// For standalone mode: called after prebake/bake/finalize to wait for completion.
/// For c/d mode: the main execution loop.
pub async fn drain(&mut self) -> anyhow::Result<()> {
loop {
// Drain retry channel
while let Ok(event) = self.rx.try_recv() {
self.pq.push(event);
}
// Determine sleep deadline
let until = self
.pq
.peek()
.map(|e| e.not_before)
.unwrap_or_else(|| chrono::Utc::now() + chrono::Duration::seconds(3600));
let std_until = (until - chrono::Utc::now())
.to_std()
.unwrap_or(std::time::Duration::from_secs(1));
tokio::select! {
event = self.rx.recv() => {
match event {
Some(event) => self.pq.push(event),
None => break Ok(()),
}
}
_ = tokio::time::sleep(std_until) => {
let now = chrono::Utc::now();
while let Some(peek) = self.pq.peek() {
if peek.not_before > now { break; }
let event = self.pq.pop().unwrap();
self.tx.send(event).await?;
}
}
}
}
}
}

Some files were not shown because too many files have changed in this diff Show More