Compare commits

12 Commits

Author SHA1 Message Date
Catty Steve df5026cdbe refactor(notify): extract handler, rule, util
CI / ${{ matrix.crate }} (workshop-engine) (push) Failing after 3m54s
CI / ${{ matrix.crate }} (workshop-baker) (push) Failing after 27m50s
Split the monolithic notify module into three submodules:
- `handler`: async bucket dispatch with cooldown, retry, escalation
- `rule`: rule parsing from YAML config, template merging, overrides
- `util`: validation, event timing, helper functions
  `NotificationTemplateDef` replaces `AddAssign` with `apply()` for
  non-destructive merging (self wins, other fills gaps).
  `insitenotify` logs instead of panicking.
2026-05-22 00:16:28 +08:00
Catty Steve d1ad08ef3a feat: add workshop-baker-params for config layering
Implement a priority-based merge system for pipeline configuration.
Parameters carry their source (Base, Courier, Bakerd) and only
higher-priority writers override. Migrate notification, prebake, bake,
and finalize config to use ParamVal wrappers. Remove hardcoded
constants and the InteractiveServer component.

BREAKING CHANGE: Remove NotAvailable variant from MethodStatus enum
2026-05-21 20:04:47 +08:00
Catty Steve cf2968e720 refactor: update AGENTS.md and remove osbolete modules
Archive 7 deprecated crates. Add Resource model documentation.
Remove internal fetch module from workshop-baker.
2026-05-20 16:30:50 +08:00
Catty Steve 28abc5d207 refactor: replace internal fetch utilities with ResourceRegistry
Remove the monolithic utils::fetch module (HTTP download, git clone,
checksum, extraction) and utils::fetch_plugin module from the baker.
Introduce a ResourceRegistry type to centrally track pipeline
resources by name and path.

The actual fetching logic is now delegated to the workshop-getterurl
crate, which gains optional md5 and sha1 hash support behind default
features. Also flatten the builtin module in workshop-getterurl.
2026-05-19 23:22:59 +08:00
Catty Steve 6e7b96cd66 refactor: remove unused crates, refactor fetch
- Move fetch_plugin from finalize::plugin::fetch to utils::fetch_plugin
- Delete unused crates: workshop-cert, workshop-deviceid, workshop-vault
- Remove parser benchmarks and empty daemon/socket modules
- Add new workshop-getterurl crate for resource fetching
2026-05-19 21:53:41 +08:00
Catty Steve 3e80825d58 refactor(plugin)!: remove dedicated plugin fetch subsystem
Drop the entire plugin fetch module (HTTP, git, checksums, extraction).
Simplify mail notification plugin to a bare stub.
Remove outdated metadata tests.

BREAKING CHANGE: Not a buildable version!
2026-05-19 16:24:19 +08:00
Catty Steve c97eafab29 refactor(baker): Restructure CLI, drop legacy tests
- Remove Python-based test framework, Dockerfiles, and runner scripts
- Add bare subcommand for standalone pipeline stage execution
- Implement unified worker mode and error module
- Refactor NotificationConfig to support YAML string keys
- Add dummy internal plugin and clean up mail template
2026-05-19 16:19:28 +08:00
Catty Steve f737b6a24d feat(baker): add axum HTTP server and refactor plugin fetching
Refactor plugin imports to use workshop_engine types, consolidate
resource fetching into utils/fetch module with checksum verification,
and add interactive notification server using axum.
2026-05-18 10:01:55 +08:00
Catty Steve 681b7d1333 style: format all code to comply with cargo fmt check
CI / ${{ matrix.crate }} (workshop-engine) (push) Failing after 36m14s
CI / ${{ matrix.crate }} (workshop-baker) (push) Successful in 1h35m51s
2026-04-28 19:17:56 +08:00
Catty Steve f07588a716 test: add Go UPM system tests for bare prebake 2026-04-28 19:12:53 +08:00
Catty Steve b0530fb858 feat(upm): add Python and Go package manager support 2026-04-28 16:56:53 +08:00
Catty Steve fd18b547e5 feat(upm): implement Rust user package manager
Adds the `user.rust` UPM to workshop-engine for managing Rust
toolchains,
components, cross-compilation targets, and cargo operations:

- `rustup default <toolchain>` / `rustup component add` / `rustup target
  add`
- `cargo fetch` for dependency pre-fetch with optional `--locked`
- `cargo install` for global crate tools with registry support

Includes 275-line test suite (`tests/upm/test_rust.py`) covering the
full
lifecycle with optional network tests.
2026-04-28 12:20:05 +08:00
164 changed files with 6404 additions and 14201 deletions
+3 -1
View File
@@ -9,7 +9,7 @@ archived
*.bak *.bak
*.clean *.clean
session* session*
llm.env .secret
# System test artifacts # System test artifacts
tests/results/ tests/results/
@@ -18,3 +18,5 @@ tests/**/__pycache__/
tests/.pytest_cache/ tests/.pytest_cache/
tests/llm/.cache/ tests/llm/.cache/
*.pyc *.pyc
.staging
.pytest_cache
+41 -57
View File
@@ -1,83 +1,72 @@
# PROJECT KNOWLEDGE BASE # PROJECT KNOWLEDGE BASE
**Generated:** 2026-04-22 **Generated:** 2026-05-19
**Commit:** 7b3b71a **Commit:** 28abc5d
**Branch:** master **Branch:** master
## OVERVIEW ## OVERVIEW
HoneyBiscuitWorkshop is a Rust-based CI/CD system ("蜜饼工坊") with LLM-powered pipeline auto-configuration. 9 independent Cargo crates, no workspace. HoneyBiscuitWorkshop ("蜜饼工坊") — Rust CI/CD system with decorator-driven shell pipeline DSL and LLM-powered configuration. 3 independent Cargo crates, no workspace.
## STRUCTURE ## STRUCTURE
``` ```
./ ./
├── workshop-baker/ # Main orchestrator (CLI + daemon) ├── workshop-baker/ # Main orchestrator (CLI + pipeline stages)
├── workshop-agent/ # HTTP agent server ├── workshop-engine/ # Execution engine (extracted from baker)
├── workshop-pipeline/ # Config library (prebake/finalize) ├── workshop-getterurl/ # URL fetching (go-getter style)
├── workshop-llm-detector/ # LLM repo analyzer ├── docs/ # mdBook documentation (zh-CN)
├── workshop-vault/ # Secret management client ├── templates/ # Pipeline templates (bake.sh, finalize.yml, prebake.yml)
── workshop-cert/ # TLS certificate generator ── archived/ # Deprecated crates (7 moved here)
├── workshop-deviceid/ # Device ID library
├── workshop-builder-native/ # Stub (unused)
├── workshop-helper-mac/ # MAC address utility
├── docs/ # mdBook documentation (zh-CN)
├── playground/ # Experimentation
└── third_party/ # Empty
``` ```
## WHERE TO LOOK ## WHERE TO LOOK
| Task | Location | Notes | | Task | Location | Notes |
|------|----------|-------| |------|----------|-------|
| Baker (main) | `workshop-baker/src/{bake,prebake,engine}/` | Core pipeline logic | | Baker (CLI) | `workshop-baker/src/{lib,main}.rs` | Clap CLI + tokio entry |
| Agent server | `workshop-agent/src/` | Actix-web HTTP + TLS | | Pipeline stages | `workshop-baker/src/{bake,prebake,finalize}/` | Stage orchestration |
| Pipeline config | `workshop-pipeline/src/config/` | prebake/finalize YAML | | Notifications | `workshop-baker/src/notify/` | Plugin-based event notifications |
| LLM integration | `workshop-llm-detector/src/llm/` | Ollama/OpenAI clients | || Engine (execution) | `workshop-engine/src/` | Process mgmt, PM detection |
| Secrets | `workshop-vault/src/` | Vault client wrapper | | URL fetching | `workshop-getterurl/src/` | Git/HTTP/File resource getter |
## CODE MAP ## CODE MAP
| Symbol | Type | Location | Role | | Symbol | Type | Location | Role |
|--------|------|----------|------| |--------|------|----------|------|
| PipelineConfig | struct | workshop-pipeline/src/lib.rs:27 | Full pipeline config | | Cli | struct | workshop-baker/src/lib.rs:21 | CLI argument struct |
| CICDLLMHelper | struct | workshop-llm-detector/src/lib.rs:16 | LLM analysis entry | | ExecutionContext | struct | workshop-engine/src/types.rs:10 | Pipeline context |
| Engine | struct | workshop-baker/src/engine.rs | Build executor | | Engine | struct | workshop-engine/src/types.rs:99 | Build executor (placeholder) |
| VaultClient | struct | workshop-vault/src/client.rs | Secret ops | | PrebakeConfig | struct | workshop-baker/src/prebake/config.rs | Prebake YAML schema |
| FinalizeConfig | struct | workshop-baker/src/finalize/config.rs:21 | Finalize YAML schema |
| GetterUrl | struct | workshop-getterurl/src/lib.rs:84 | Parsed resource URL |
| Getter | trait | workshop-getterurl/src/getter.rs:18 | Resource fetch trait |
## CONVENTIONS ## CONVENTIONS
### Rust ### Rust
- **Edition**: 2024 (non-standard, requires Rust 1.85+) - **Edition**: 2024 (requires Rust 1.85+), deprecates `mod.rs`
- Deprecate `mod.rs`
- **Naming**: `snake_case` files/modules, `PascalCase` types, `SCREAMING_SNAKE_CASE` consts - **Naming**: `snake_case` files/modules, `PascalCase` types, `SCREAMING_SNAKE_CASE` consts
- **Error handling**: `thiserror` + `anyhow` combo - **Error handling**: `thiserror` + `anyhow` combo; `HasExitCode` trait for exit codes
- **Async**: `#[tokio::main]` + `tokio` with "full" features - **Async**: `#[tokio::main]` + `tokio` with "full" features
- **Imports**: `std``external_crate``crate::module` - **Imports**: `std``external_crate``crate::module`
- **Dual-mode**: CLI commands (prebake/bake/finalize) + daemon mode (unimplemented)
### Testing ### Testing
- **Rust**: `#[test]` inline, `#[tokio::test]` async, `tests/*.rs` integration - **Rust**: `#[test]` inline, `#[tokio::test]` async, `tests/` integration
- **Python**: `pytest` with `tests/integration/test_*.py` (Docker-based) - **Benchmarks**: Criterion (configured in Cargo.toml)
- **Benchmarks**: Criterion with `harness = false`
### Build Pipeline ### CI
- **prebake.yml**: Environment setup (docker/firecracker/baremetal) - **GitHub Actions**: rust-toolchain (stable), clippy + rustfmt, `cargo test --all-features`
- **bake.sh**: Build execution with `# @pipeline` decorators - **Matrix**: workshop-baker, workshop-engine
- **finalize.yml**: Packaging, deployment, notifications
- **Annotations**: `@timeout()`, `@retry()`, `@parallel`, `@fallible`
## ANTI-PATTERNS (THIS PROJECT) ## ANTI-PATTERNS (THIS PROJECT)
1. **Hardcoded user** — "vulcan" username hardcoded throughout prebake
1. **Python in Rust project** — workshop-baker has Python tests (pytest.ini, test_prebake.py) 2. **Unsafe libc** — workshop-baker/src/prebake/security.rs has 5+ unsafe blocks
2. **temp/ directory** — Non-standard in Rust projects 3. **Rust 2024 edition** — May not work with stable toolchains
3. **Certificates in source** — artifact workshop-cert/certs/ not excluded 4. **Empty daemon**`daemon/` dir is empty, Daemon command is `unimplemented!()`
4. **Deprecated projects** — workshop-executor.old, workshop-monitor.old still present
5. **Hardcoded mirrors** — Tsinghua University mirrors in configs
6. **Rust 2024 edition** — May not work with stable toolchains
## UNIQUE STYLES ## UNIQUE STYLES
- **Pipeline DSL**: Shell scripts with `# @decorator` annotations - **Pipeline DSL**: Shell scripts with `# @decorator` annotations
- **Dual-mode baker**: CLI commands + daemon mode - **Notification system**: Plugin-based with batching, retry, priority groups
- **LLM cookbook system**: YAML cookbooks for language-specific builds - **LLM cookbook system**: YAML cookbooks for language-specific builds (archived)
- **Chinese docs** — Documentation primarily in zh-CN - **Chinese docs** — Documentation primarily in zh-CN
- **User "vulcan"** — Custom build user (not root/runner)
## COMMANDS ## COMMANDS
```bash ```bash
@@ -86,23 +75,18 @@ cargo build --release -p workshop-baker
# Test # Test
cargo test -p workshop-baker cargo test -p workshop-baker
pytest tests/integration/test_prebake.py -v cargo test -p workshop-engine
# Lint # Lint
cargo clippy -- -D warnings cargo clippy -- -D warnings
cargo fmt --all cargo fmt --all
# Run baker # Run
cargo run --bin workshop-baker -- prebake config.yml cargo run --bin workshop-baker -- --pipeline demo --build-id 1 --prebake prebake.yml --bake bake.sh --finalize finalize.yml
cargo run --bin workshop-baker -- bake script.sh
# Run agent
cargo run --bin workshop-agent
``` ```
## NOTES ## NOTES
- **Docker required** for integration tests - **Docker required** for integration tests
- **workshop-builder-native** is a stub (3-line Hello World) - **No root workspace** — each crate independent, build separately
- No root workspace Cargo.toml — each crate is independent - **7 archived crates** — workshop-agent, workshop-cert, workshop-deviceid, workshop-helper-mac, workshop-llm-detector, workshop-pipeline, workshop-vault
- `AGENTS.md.old` at root — legacy documentation - **Python tests removed** — pytest.ini exists but test files no longer in workspace
+1
View File
@@ -35,3 +35,4 @@
- [自定义 Decorator](zh-CN/vol3_dev/custom_decorator.md) - [自定义 Decorator](zh-CN/vol3_dev/custom_decorator.md)
- [模板变量](zh-CN/vol3_dev/template_variables.md) - [模板变量](zh-CN/vol3_dev/template_variables.md)
- [邮件模板](zh-CN/vol3_dev/mail_template.md) - [邮件模板](zh-CN/vol3_dev/mail_template.md)
- [Resource 模型](zh-CN/vol3_dev/resource_model.md)
+88
View File
@@ -0,0 +1,88 @@
# Resource 模型
## 核心思想
整个系统只关心两件事:东西从哪来(fetch),东西到哪去(publish)。
fetch 方向:URL → 本地路径 → 使用。
publish 方向:本地路径 → 变换(插件)→ URL。
两个方向共用同一套寻址方式,但 Resource 中间态只存在于 fetch 方向。
## URL 格式
```
getter::url?param1=value1&param2=value2
```
getter 显式声明意图。不是"我猜这是个 git 仓库",而是"使用者告诉我这是什么"。
只有三个 getter
- `git` — 克隆仓库,必要时 checkout 指定版本
- `http` / `https` — 下载文件,必要时校验 checksum
- `file` — 复制本地文件或目录
版本锁定不用 `@v1.0``@` 在 URL 里是认证分隔符),用 `?ref=v1.0`
没有 `github.com/user/repo` 看起来应该 clone 它这种黑魔法。
没有 `.tar.gz` 看起来应该解压它的惊喜。
URL 是 URL,行为是行为,两者用 getter:: 显式绑定。
## Resource 不是网络地址
流水线阶段(prebake / finalize / notify)只接触 Resource:一个名字 + 一个本地路径,已就绪。它们不经由网络、不解析 URL、不碰 getter。
所有 fetch 行为在阶段启动前完成:
- worker 模式:bakerd fetch → 填入 ResourceRegistry → 启动 baker
- standalone 模式:baker 自己 fetch → 填入 ResourceRegistry → 执行阶段
- bare 模式:不 fetch,传入的必须是 Resource
ResourceRegistry 是名字到路径的映射。填进去就只读了。
## getterurl 是独立的 crate
URL 格式解析 + 基本 fetch 放在 workshop-getterurl,和 baker 本身没有耦合。任何人都可以用。
crate 分两层:
```
结构层(parser):
"git::https://host/repo.git?ref=v1.0" → { getter: "git", url: Url, params: Params }
获取层(builtin getters):
getter.fetch(url, params, dst) → local_path
git: git clone + optional checkout
http: HTTP GET + optional checksum
file: std::fs::copy
```
获取层是字面意义的——不解压、不推断、不猜意图。
## Checksum
checksum 作为 URL query 参数传递。三态语义:
- `?checksum=sha256:abc` — 必须匹配
- `?checksum=` — 显式禁用校验
- 无 checksum — 如果服务端返回 Content-Digest 头,校验它
getter 不主动校验,它只执行 URL 里指定的操作。
## Artifact 不经过 Resource
artifact 是 bake 的产物,自产生就开始被使用,不存在中间态。它的路径就是 bake 产出的本地路径,不走 ResourceRegistry。
artifact 的 publish 方向由插件定义行为——可能上传、可能扫描、可能只记录。插件定义最终的行为,只要是自洽的。
## 三种运行模式
```
fetch 执行者 fetch 在阶段内
daemon 模式 bakerd 否
standalone baker 自身 否(阶段启动前已完成)
bare 模式 无 否(必须传入 Resource
```
bare 模式下传入的必须是已经就绪的 Resource。外部 URL 是 daemon 或 standalone 的事,bare 不负责。
-101
View File
@@ -1,101 +0,0 @@
#!/bin/bash
set -euo pipefail
# HoneyBiscuitWorkshop System Test Runner
#
# Usage:
# ./runner.sh # Arch baker tests (hbw-system-test:latest)
# ./runner.sh --image hbw-test-pm:test-apt # Debian engine PM tests
# ./runner.sh -k test_install # Arch baker, filtered
MODE="auto"
TEST_IMAGE=""
TEST_DIR="tests/baker"
PYTEST_ARGS=""
while [[ $# -gt 0 ]]; do
case "$1" in
--image) TEST_IMAGE="$2"; TEST_DIR="tests/engine"; shift 2 ;;
manual) MODE="manual"; shift ;;
auto) MODE="auto"; shift ;;
--) shift; PYTEST_ARGS="$@"; break ;;
*) PYTEST_ARGS="$PYTEST_ARGS $1"; shift ;;
esac
done
# Default image: Arch-based baker test image
: ${TEST_IMAGE:="hbw-system-test:latest"}
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
CONTAINER_NAME="hbw-system-test-container"
STAGING_DIR="/tmp/hbw-system-test"
cleanup() {
docker rm -f "$CONTAINER_NAME" 2>/dev/null || true
[ "$TEST_DIR" = "tests/baker" ] && rm -rf "$STAGING_DIR"
}
trap cleanup EXIT
build_binary() {
local target="x86_64-unknown-linux-musl"
local binary="$SCRIPT_DIR/workshop-baker/target/$target/debug/workshop-baker"
if [ ! -f "$binary" ]; then
echo "[runner] Building static (musl) baker binary..."
cd "$SCRIPT_DIR/workshop-baker"
cargo build -p workshop-baker --target "$target" 2>&1 | tail -n 1
fi
}
prepare_staging() {
# Only needed for baker tests (hbw-system-test:latest mounts /workshop)
if [ "$TEST_DIR" = "tests/baker" ]; then
mkdir -p "$STAGING_DIR"
cp "$SCRIPT_DIR/workshop-baker/target/x86_64-unknown-linux-musl/debug/workshop-baker" "$STAGING_DIR/"
chmod +x "$STAGING_DIR/workshop-baker"
fi
}
run_container() {
local tag="${TEST_IMAGE##*:}"
local filter=""
case "$tag" in
test-arch) filter="-k pacman" ;;
test-apt) filter="-k apt" ;;
test-dnf) filter="-k dnf" ;;
test-apk) filter="-k apk" ;;
esac
echo "[runner] Starting: $TEST_IMAGE (tests=$TEST_DIR${filter:+, filter=$filter})"
mkdir -p "$SCRIPT_DIR/tests/results"
local mounts="-v $SCRIPT_DIR:/src:ro"
local entrypoint=""
if [ "$TEST_DIR" = "tests/baker" ]; then
mounts="$mounts -v $STAGING_DIR:/workshop:ro"
entrypoint='--entrypoint ""'
fi
local cmd="sh -c \"python3 -m venv /tmp/v && . /tmp/v/bin/activate && \
pip install pytest pytest-timeout -q && \
exec python -m pytest $TEST_DIR $PYTEST_ARGS $filter\""
if [ "$MODE" = "manual" ]; then
docker run -it --rm --name "$CONTAINER_NAME" --privileged $entrypoint \
$mounts "$TEST_IMAGE"
else
docker run --rm --name "$CONTAINER_NAME" --privileged $entrypoint \
$mounts -w /src \
"$TEST_IMAGE" "$cmd"
fi
}
echo "============================================"
echo " HBW System Test Runner"
echo " Image: $TEST_IMAGE"
echo " Tests: $TEST_DIR"
echo "============================================"
build_binary
prepare_staging
run_container
echo "[runner] Done."
-32
View File
@@ -1,32 +0,0 @@
FROM archlinux:latest
RUN sed -i 's|https://geo.mirror.pkgbuild.com|https://mirrors.tuna.tsinghua.edu.cn/archlinux|g' /etc/pacman.d/mirrorlist && \
sed -i 's|https://mirror.rackspace.com/archlinux|https://mirrors.tuna.tsinghua.edu.cn/archlinux|g' /etc/pacman.d/mirrorlist && \
echo 'Server = https://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch' >> /etc/pacman.d/mirrorlist
RUN pacman -Syu --noconfirm && \
pacman -S --noconfirm \
python python-pip base-devel git curl wget sudo \
which procps-ng shadow && \
pacman -Scc --noconfirm
# Create test user
RUN useradd -m -s /bin/bash testuser
# Python environment
WORKDIR /app
COPY tests/requirements.txt ./requirements.txt
RUN python -m venv /opt/test-venv && \
/opt/test-venv/bin/pip install -r requirements.txt
# Copy test suite
COPY tests/ ./tests/
COPY tests/entrypoint.sh ./entrypoint.sh
RUN chmod +x entrypoint.sh
# Environment
ENV PATH="/opt/test-venv/bin:$PATH"
ENV PYTHONUNBUFFERED=1
ENV PYTHONDONTWRITEBYTECODE=1
ENTRYPOINT ["./entrypoint.sh"]
-129
View File
@@ -1,129 +0,0 @@
import os
import pytest
FIXTURES = os.path.join(os.path.dirname(__file__), "fixtures", "bake")
@pytest.mark.bake
class TestBakeSimpleScript:
def test_simple_script_runs(self, run_baker, bake_workspace, work_dir):
prebake_cfg = os.path.join(
os.path.dirname(__file__), "fixtures", "prebake", "minimal.yml"
)
script = os.path.join(FIXTURES, "simple.sh")
bake_base = os.path.join(FIXTURES, "bake_base.sh")
result = run_baker(
"bake", script,
"--bake-base", bake_base,
"--prebake", prebake_cfg,
cwd=work_dir,
timeout=60,
)
assert result["success"], f"Bake failed: {result['stderr']}"
assert "Hello from simple bake script" in result["stdout"]
def test_simple_script_sets_env_vars(self, run_baker, bake_workspace, work_dir):
prebake_cfg = os.path.join(
os.path.dirname(__file__), "fixtures", "prebake", "minimal.yml"
)
script = os.path.join(FIXTURES, "simple.sh")
bake_base = os.path.join(FIXTURES, "bake_base.sh")
result = run_baker(
"bake", script,
"--bake-base", bake_base,
"--prebake", prebake_cfg,
cwd=work_dir,
timeout=60,
)
assert result["success"]
assert "test-pipeline" in result["stdout"]
@pytest.mark.bake
class TestBakePipeline:
def test_pipeline_functions_execute(self, run_baker, bake_workspace, work_dir):
prebake_cfg = os.path.join(
os.path.dirname(__file__), "fixtures", "prebake", "minimal.yml"
)
script = os.path.join(FIXTURES, "pipeline.sh")
bake_base = os.path.join(FIXTURES, "bake_base.sh")
result = run_baker(
"bake", script,
"--bake-base", bake_base,
"--prebake", prebake_cfg,
cwd=work_dir,
timeout=60,
)
assert result["success"], f"Pipeline bake failed: {result['stderr']}"
assert os.path.exists("/tmp/bake-test-output/step1.txt")
assert os.path.exists("/tmp/bake-test-output/step2.txt")
def test_pipeline_step_outputs_correct(self, run_baker, bake_workspace, work_dir):
prebake_cfg = os.path.join(
os.path.dirname(__file__), "fixtures", "prebake", "minimal.yml"
)
script = os.path.join(FIXTURES, "pipeline.sh")
bake_base = os.path.join(FIXTURES, "bake_base.sh")
result = run_baker(
"bake", script,
"--bake-base", bake_base,
"--prebake", prebake_cfg,
cwd=work_dir,
timeout=60,
)
assert result["success"]
with open("/tmp/bake-test-output/step1.txt") as f:
assert f.read().strip() == "step1-done"
with open("/tmp/bake-test-output/step2.txt") as f:
assert f.read().strip() == "step2-done"
@pytest.mark.bake
class TestBakeAfterDependencies:
def test_after_ordering(self, run_baker, bake_workspace, work_dir):
prebake_cfg = os.path.join(
os.path.dirname(__file__), "fixtures", "prebake", "minimal.yml"
)
script = os.path.join(FIXTURES, "with_after.sh")
bake_base = os.path.join(FIXTURES, "bake_base.sh")
result = run_baker(
"bake", script,
"--bake-base", bake_base,
"--prebake", prebake_cfg,
cwd=work_dir,
timeout=60,
)
assert result["success"], f"@after bake failed: {result['stderr']}"
assert os.path.exists("/tmp/bake-after-test/status.txt")
assert os.path.exists("/tmp/bake-after-test/source.txt")
assert os.path.exists("/tmp/bake-after-test/build.txt")
@pytest.mark.bake
class TestBakeStatusFiles:
def test_status_written_after_bake(self, run_baker, bake_workspace, work_dir):
prebake_cfg = os.path.join(
os.path.dirname(__file__), "fixtures", "prebake", "minimal.yml"
)
script = os.path.join(FIXTURES, "simple.sh")
bake_base = os.path.join(FIXTURES, "bake_base.sh")
result = run_baker(
"bake", script,
"--bake-base", bake_base,
"--prebake", prebake_cfg,
cwd=work_dir,
timeout=60,
)
assert result["success"]
assert os.path.exists("/tmp/.hbwstatus")
-77
View File
@@ -1,77 +0,0 @@
import os
import pytest
from probes import file
from llm import judge_log, LLMClient
FIXTURES = os.path.join(os.path.dirname(__file__), "fixtures", "finalize")
@pytest.mark.finalize
class TestFinalizeMinimal:
def test_minimal_finalize_succeeds(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "minimal.yml")
result = run_baker("finalize", config, cwd=work_dir, timeout=60)
assert result["success"], f"Minimal finalize failed: {result['stderr']}"
@pytest.mark.finalize
class TestFinalizeHooks:
def test_early_hook_executes(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "with_hooks.yml")
result = run_baker("finalize", config, cwd=work_dir, timeout=60)
assert result["success"]
assert "finalize early hook" in result["stdout"] or "finalize early hook" in result["stderr"]
def test_late_hook_executes(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "with_hooks.yml")
result = run_baker("finalize", config, cwd=work_dir, timeout=60)
assert result["success"]
assert "finalize late hook" in result["stdout"] or "finalize late hook" in result["stderr"]
@pytest.mark.finalize
class TestFinalizeStatus:
def test_finalize_writes_status(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "minimal.yml")
result = run_baker("finalize", config, cwd=work_dir, timeout=60)
assert result["success"]
assert os.path.exists("/tmp/.hbwstatus")
@pytest.mark.finalize
@pytest.mark.llm
class TestFinalizeWithLLM:
def test_finalize_hooks_via_llm(self, run_baker, llm_env, work_dir):
if not llm_env:
pytest.skip("LLM environment not configured")
config = os.path.join(FIXTURES, "with_hooks.yml")
result = run_baker("finalize", config, cwd=work_dir, timeout=60)
combined_log = result["stdout"] + "\n" + result["stderr"]
judgment = judge_log(
operation="finalize with early and late hooks",
expected_behavior="Both early and late hooks executed successfully, "
"printing 'finalize early hook' and 'finalize late hook'",
actual_log=combined_log,
)
assert judgment.passed, f"LLM judgment failed: {judgment.reason}"
@pytest.mark.finalize
class TestFinalizePlugins:
@pytest.mark.xfail(reason="Plugin download requires network/git access", strict=False)
def test_plugin_download(self, run_baker, work_dir):
pass
@pytest.mark.xfail(reason="Artifact stage not yet implemented", strict=False)
def test_artifact_collection(self, run_baker, work_dir):
pass
-261
View File
@@ -1,261 +0,0 @@
import os
import json
import pytest
from probes import user, file, package
FIXTURES = os.path.join(os.path.dirname(__file__), "fixtures", "prebake")
@pytest.mark.prebake
class TestPrebakeConfigValidation:
def test_invalid_version_fails(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "invalid_version.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert not result["success"], f"Expected prebake to fail with invalid version"
assert "version" in result["stderr"].lower() or "Version" in result["stderr"]
def test_missing_required_field_fails(self, run_baker, work_dir):
missing_env = os.path.join(work_dir, "missing_env.yml")
with open(missing_env, "w") as f:
f.write('version: "1.0"\n')
result = run_baker("prebake", missing_env, cwd=work_dir, timeout=60)
assert not result["success"]
def test_minimal_config_succeeds(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "minimal.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"], f"Minimal prebake failed: {result['stderr']}"
@pytest.mark.prebake
class TestPrebakeBootstrap:
def test_bootstrap_creates_vulcan_user(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "bootstrap_only.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"], f"prebake failed: {result['stderr']}"
assert user.user_exists("vulcan")
def test_bootstrap_creates_workspace(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "bootstrap_only.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"]
assert file.dir_exists("/home/vulcan/workspace")
def test_bootstrap_creates_symlink(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "bootstrap_only.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"]
assert os.path.islink("/workspace")
def test_bootstrap_generates_bootstrap_script(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "bootstrap_only.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"]
assert file.file_exists("/bootstrap.sh")
def test_bootstrap_script_permissions(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "bootstrap_only.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"]
mode = file.file_permissions("/bootstrap.sh")
assert mode & 0o111, f"/bootstrap.sh not executable, mode: {oct(mode)}"
def test_bootstrap_script_contains_user_creation(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "bootstrap_only.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"]
assert file.file_exists("/bootstrap.sh")
assert file.file_contains("/bootstrap.sh", "useradd")
def test_bootstrap_script_contains_workspace_setup(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "bootstrap_only.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"]
assert file.file_contains("/bootstrap.sh", "mkdir -p /home/vulcan/workspace")
def test_bootstrap_with_pacman_creates_sudoers(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "with_pacman.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=120)
assert result["success"], f"prebake failed: {result['stderr']}"
assert file.file_exists("/etc/sudoers.d/workshop")
def test_custom_bootstrap_executes(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "with_custom_bootstrap.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"], f"Custom bootstrap failed: {result['stderr']}"
assert file.dir_exists("/custom-workspace")
@pytest.mark.prebake
class TestPrebakeDryRun:
def test_dry_run_does_not_create_user(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "bootstrap_only.yml")
env = {"HBW_DRY_RUN": "true"}
result = run_baker("prebake", config, cwd=work_dir, env=env, timeout=60)
assert result["success"], f"Dry-run failed: {result['stderr']}"
def test_dry_run_logs_script_content(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "bootstrap_only.yml")
env = {"HBW_DRY_RUN": "true", "RUST_LOG": "info"}
result = run_baker("prebake", config, cwd=work_dir, env=env, timeout=60)
assert result["success"]
combined = result["stdout"] + result["stderr"]
assert "DRY_RUN" in combined or "dry_run" in combined.lower()
@pytest.mark.prebake
class TestPrebakeHooks:
#@pytest.mark.xfail(reason="Hook execution fails with Permission denied in container environment", strict=False)
def test_early_hook_executes(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "with_hooks.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"], f"prebake with hooks failed: {result['stderr']}"
combined = result["stdout"] + result["stderr"]
assert "early hook executed" in combined
#@pytest.mark.xfail(reason="Hook execution fails with Permission denied in container environment", strict=False)
def test_late_hook_executes(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "with_hooks.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"], f"prebake with hooks failed: {result['stderr']}"
combined = result["stdout"] + result["stderr"]
assert "late hook executed" in combined
#@pytest.mark.xfail(reason="Hook execution fails with Permission denied in container environment", strict=False)
def test_hooks_run_in_order(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "with_hooks.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"]
combined = result["stdout"] + result["stderr"]
early_pos = combined.find("early hook executed")
late_pos = combined.find("late hook executed")
assert early_pos != -1 and late_pos != -1
assert early_pos < late_pos, "Early hook should execute before late hook"
def test_empty_hooks_succeed(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "minimal.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"], f"prebake with no hooks failed: {result['stderr']}"
@pytest.mark.prebake
class TestPrebakeDepsSystem:
@pytest.mark.xfail(reason="Package installation may fail in test environment", strict=False)
def test_pacman_packages_installed(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "with_pacman.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=120)
assert result["success"], f"prebake failed: {result['stderr']}"
assert package.package_installed("curl")
assert package.package_installed("wget")
def test_pacman_mirror_change(self, run_baker, work_dir):
mirror_config = os.path.join(work_dir, "mirror.yml")
with open(mirror_config, "w") as f:
f.write('''
version: "1.0"
environment:
builder: "baremetal"
bootstrap:
user: "vulcan"
dependencies:
system:
pacman:
packages: ["which"]
mirror: "https://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch"
''')
result = run_baker("prebake", mirror_config, cwd=work_dir, timeout=120)
assert result["success"], f"prebake with mirror failed: {result['stderr']}"
@pytest.mark.prebake
class TestPrebakeEnvVars:
def test_prebake_envvars_injected(self, run_baker, work_dir):
hook_config = os.path.join(work_dir, "envvar_hook.yml")
with open(hook_config, "w") as f:
f.write('''
version: "1.0"
environment:
builder: "baremetal"
bootstrap:
user: "vulcan"
envvars:
prebake:
TEST_INJECTED: "injected_value"
hooks:
early:
- name: "check-env"
command: "echo $TEST_INJECTED"
''')
result = run_baker("prebake", hook_config, cwd=work_dir, timeout=60)
assert result["success"], f"prebake with envvars failed: {result['stderr']}"
combined = result["stdout"] + result["stderr"]
assert "injected_value" in combined
@pytest.mark.prebake
class TestPrebakeSecurity:
#@pytest.mark.xfail(reason="Privilege drop may fail in container without proper setup", strict=False)
def test_privilege_drop_after_bootstrap(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "with_security.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"], f"prebake with security failed: {result['stderr']}"
@pytest.mark.prebake
class TestPrebakeStatus:
def test_status_files_written(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "bootstrap_only.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"]
assert os.path.exists("/tmp/.hbwstatus")
def test_status_contains_bootstrap_stage(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "bootstrap_only.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"]
with open("/tmp/.hbwstatus", "r") as f:
lines = f.readlines()
assert len(lines) > 0
bootstrap_lines = [l for l in lines if json.loads(l).get("name") == "bootstrap"]
assert len(bootstrap_lines) > 0, "No bootstrap stage found in status file"
data = json.loads(bootstrap_lines[-1])
assert data.get("phase") == "Prebake"
assert data.get("result") == "Success"
def test_status_contains_duration(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "bootstrap_only.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"]
with open("/tmp/.hbwstatus", "r") as f:
data = json.loads(f.readline())
assert "duration_ms" in data
assert isinstance(data["duration_ms"], int)
def test_status_contains_timestamps(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "bootstrap_only.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
assert result["success"]
with open("/tmp/.hbwstatus", "r") as f:
data = json.loads(f.readline())
assert "started_at" in data
assert "finished_at" in data
def test_status_written_for_each_stage(self, run_baker, work_dir):
config = os.path.join(FIXTURES, "with_pacman.yml")
result = run_baker("prebake", config, cwd=work_dir, timeout=120)
assert result["success"], f"prebake failed: {result['stderr']}"
with open("/tmp/.hbwstatus", "r") as f:
lines = f.readlines()
names = [json.loads(line).get("name") for line in lines]
assert "bootstrap" in names
assert "depssystem" in names
-164
View File
@@ -1,164 +0,0 @@
"""
Root conftest for HoneyBiscuitWorkshop system tests.
Provides core fixtures for:
- Locating the baker binary and examples
- Running subprocess commands with output capture
- Managing temporary working directories
"""
import os
import subprocess
import shutil
import tempfile
import pytest
def pytest_configure(config):
"""Register custom markers."""
config.addinivalue_line("markers", "prebake: prebake stage tests")
config.addinivalue_line("markers", "bake: bake stage tests")
config.addinivalue_line("markers", "finalize: finalize stage tests")
config.addinivalue_line("markers", "llm: tests requiring LLM API")
config.addinivalue_line("markers", "slow: slow running tests")
@pytest.fixture(scope="session")
def baker_bin():
"""Path to the workshop-baker binary."""
path = os.environ.get("BAKER_BIN", "/workshop/workshop-baker")
if not os.path.isfile(path):
pytest.skip(f"Baker binary not found at {path}")
if not os.access(path, os.X_OK):
pytest.skip(f"Baker binary not executable at {path}")
return path
@pytest.fixture(scope="session")
def examples_dir():
"""Path to the examples directory (may be None)."""
path = os.environ.get("EXAMPLES_DIR", "/workshop/examples")
if not os.path.isdir(path):
return None
return path
@pytest.fixture(scope="session")
def llm_env():
"""LLM environment variables dict (from llm.env)."""
env = {}
for var in [
"OPENAI_BASE_URL",
"OPENAI_API_KEY",
"ANTHROPIC_BASE_URL",
"ANTHROPIC_API_KEY",
"MODEL",
"REASONING_EFFORT",
]:
val = os.environ.get(var)
if val:
env[var] = val
return env
@pytest.fixture
def work_dir():
"""Create a temporary working directory for a test, cleaned up after."""
d = tempfile.mkdtemp(prefix="hbw-test-")
yield d
shutil.rmtree(d, ignore_errors=True)
@pytest.fixture
def run_cmd():
"""Fixture that provides a command runner function.
Returns a function that runs a command and returns
(stdout, stderr, returncode, success).
"""
def _run(args, cwd=None, env=None, timeout=120):
"""Run a command and capture output.
Args:
args: Command as list of strings.
cwd: Working directory.
env: Environment variables (merged with os.environ).
timeout: Timeout in seconds.
Returns:
dict with keys: stdout, stderr, returncode, success
"""
full_env = {**os.environ}
if env:
full_env.update(env)
try:
result = subprocess.run(
args,
cwd=cwd,
env=full_env,
capture_output=True,
text=True,
timeout=timeout,
)
return {
"stdout": result.stdout,
"stderr": result.stderr,
"returncode": result.returncode,
"success": result.returncode == 0,
}
except subprocess.TimeoutExpired:
return {
"stdout": "",
"stderr": f"Command timed out after {timeout}s",
"returncode": -1,
"success": False,
}
except Exception as e:
return {
"stdout": "",
"stderr": str(e),
"returncode": -1,
"success": False,
}
return _run
@pytest.fixture
def run_baker(baker_bin, run_cmd):
"""Fixture that provides a baker command runner.
Returns a function that runs workshop-baker with common flags.
"""
def _run(subcommand, *args, cwd=None, env=None, timeout=120):
cmd = [
baker_bin,
"-u", "testuser",
"-p", "test-pipeline",
"-b", "test-build-001",
"bare",
subcommand,
]
cmd.extend(args)
return run_cmd(cmd, cwd=cwd, env=env, timeout=timeout)
return _run
@pytest.fixture
def fixtures_dir():
"""Path to the test fixtures directory."""
return os.path.join(os.path.dirname(__file__), "fixtures")
@pytest.fixture
def bake_workspace():
"""Create /workspace for bake tests (baker writes temp scripts there)."""
existed = os.path.exists("/workspace")
if not existed:
os.makedirs("/workspace", exist_ok=True)
yield
if not existed:
shutil.rmtree("/workspace", ignore_errors=True)
-102
View File
@@ -1,102 +0,0 @@
"""PM system tests driven by bare workshop-baker prebake."""
import os
import pytest
from probes import binary_exists
PACMAN_MIRROR = "https://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch"
def _yml(pm, pkgs, mirror="", repo=None):
"""Build prebake.yml. repo is a list of lines (without lead indent)."""
m = ""
if mirror:
m += ' mirror: "' + mirror + '"\n'
if repo:
m += " repositories:\n"
for line in repo:
m += " " + line + "\n"
return """\
version: "1.0"
environment:
builder: "baremetal"
bootstrap:
user: "vulcan"
dependencies:
system:
{pm}:
packages: [{pkgs}]
{m}""".format(pm=pm, pkgs=pkgs, m=m)
# ── install ──────────────────────────────────────────────────────────
@pytest.mark.parametrize("pm_name,pkgs", [
("pacman", '"curl", "git"'),
("apt", '"curl", "git"'),
("dnf", '"curl", "git"'),
("apk", '"curl", "git"'),
])
def test_install(run_cmd, work_dir, pm_name, pkgs):
cfg = os.path.join(work_dir, "pm.yml")
mirror = PACMAN_MIRROR if pm_name == "pacman" else ""
with open(cfg, "w") as f:
f.write(_yml(pm_name, pkgs, mirror=mirror))
r = run_cmd(["/baker", "-p", "t", "-b", "1", "-u", "testuser",
"bare", "prebake", cfg], timeout=120)
assert r["success"], "[{}] prebake: {}".format(pm_name, r["stderr"][:300])
for pkg in ["curl", "git"]:
assert binary_exists("/usr/bin/" + pkg), "[{}] {} missing".format(pm_name, pkg)
@pytest.mark.parametrize("pm_name", ["pacman", "apt", "dnf", "apk"])
def test_reinstall(run_cmd, work_dir, pm_name):
cfg = os.path.join(work_dir, "pm.yml")
mirror = PACMAN_MIRROR if pm_name == "pacman" else ""
with open(cfg, "w") as f:
f.write(_yml(pm_name, '"curl"', mirror=mirror))
cmd = ["/baker", "-p", "t", "-b", "1", "-u", "testuser", "bare", "prebake", cfg]
r1 = run_cmd(cmd, timeout=120)
assert r1["success"], "[{}] first install failed".format(pm_name)
r2 = run_cmd(cmd, timeout=120)
assert r2["success"], "[{}] reinstall failed".format(pm_name)
# ── change_mirror ────────────────────────────────────────────────────
@pytest.mark.parametrize("pm_name", ["pacman", "apt", "dnf", "apk"])
def test_mirror(run_cmd, work_dir, pm_name):
cfg = os.path.join(work_dir, "pm.yml")
mirror = PACMAN_MIRROR if pm_name == "pacman" else ""
with open(cfg, "w") as f:
f.write(_yml(pm_name, '"which"', mirror=mirror))
r = run_cmd(["/baker", "-p", "t", "-b", "1", "-u", "testuser",
"bare", "prebake", cfg], timeout=120)
assert r["success"], "[{}] mirror: {}".format(pm_name, r["stderr"][:300])
assert binary_exists("/usr/bin/which"), "[{}] which not installed".format(pm_name)
# ── add_repository ───────────────────────────────────────────────────
@pytest.mark.parametrize("pm_name,repo_lines,result_file", [
("pacman", ['- name: custom', ' server: "https://repo.example.com/$arch"'],
"/etc/pacman.conf"),
("apt", ['- url: "https://repo.example.com/debian"', ' distro: bookworm'],
"/etc/apt/sources.list.d/custom.list"),
("dnf", ['- name: custom', ' baseurl: "https://repo.example.com/fedora"'],
"/etc/yum.repos.d/custom.repo"),
("apk", ['- url: "https://repo.example.com/v3.19/main"'],
"/etc/apk/repositories"),
])
@pytest.mark.xfail(reason="fake repo URL, package install may fail", strict=False)
def test_add_repo(run_cmd, work_dir, pm_name, repo_lines, result_file):
cfg = os.path.join(work_dir, "pm.yml")
mirror = PACMAN_MIRROR if pm_name == "pacman" else ""
with open(cfg, "w") as f:
f.write(_yml(pm_name, '"curl"', mirror=mirror, repo=repo_lines))
r = run_cmd(["/baker", "-p", "t", "-b", "1", "-u", "testuser",
"bare", "prebake", cfg], timeout=120)
# Prebake may fail if repo URL is unreachable — check only the config file
# was written by add_repository before the (possibly failing) install step.
assert binary_exists(result_file), \
"[{}] repo file missing: {}".format(pm_name, result_file)
-36
View File
@@ -1,36 +0,0 @@
#!/bin/bash
set -e
# Load LLM environment if available
if [ -f /workshop/llm.env ]; then
set -a
source /workshop/llm.env
set +a
fi
# Export test configuration
export BAKER_BIN=/workshop/workshop-baker
export EXAMPLES_DIR=/workshop/examples
export TEST_MODE=${TEST_MODE:-auto}
if [ "$TEST_MODE" = "manual" ]; then
echo "========================================"
echo " HoneyBiscuitWorkshop System Tests"
echo " Mode: MANUAL (interactive shell)"
echo "========================================"
echo ""
echo "Binary: $BAKER_BIN"
echo "Examples: $EXAMPLES_DIR"
echo ""
echo "Run tests manually:"
echo " cd /app && pytest tests/ -v"
echo ""
exec /bin/bash
else
echo "========================================"
echo " HoneyBiscuitWorkshop System Tests"
echo " Mode: AUTO (running pytest)"
echo "========================================"
cd /app
exec python -m pytest tests/ -v --tb=short --junitxml=/results/report.xml "$@"
fi
-6
View File
@@ -1,6 +0,0 @@
#!/bin/bash
[ -f /dev/shm/bakeexport.{{ name }} ] && . /dev/shm/bakeexport.{{ name }} || true
{{ condition }}
{{ main }}
{{ name }}
{{ export }}
-25
View File
@@ -1,25 +0,0 @@
#!/bin/bash
# @pipeline
step_one() {
echo "Step 1: Setup"
mkdir -p /tmp/bake-test-output
echo "step1-done" > /tmp/bake-test-output/step1.txt
}
# @pipeline
step_two() {
echo "Step 2: Build"
echo "step2-done" > /tmp/bake-test-output/step2.txt
}
# @pipeline
step_three() {
echo "Step 3: Verify"
if [ -f /tmp/bake-test-output/step1.txt ] && [ -f /tmp/bake-test-output/step2.txt ]; then
echo "All steps completed successfully"
else
echo "ERROR: Missing step outputs"
exit 1
fi
}
-5
View File
@@ -1,5 +0,0 @@
#!/bin/bash
echo "Hello from simple bake script"
echo "Task ID: $HBW_TASKID"
echo "Pipeline: $HBW_PIPELINE"
exit 0
-22
View File
@@ -1,22 +0,0 @@
#!/bin/bash
# @pipeline
init_workspace() {
echo "Initializing workspace"
mkdir -p /tmp/bake-after-test
echo "init" > /tmp/bake-after-test/status.txt
}
# @pipeline
# @after(init_workspace)
fetch_source() {
echo "Fetching source (depends on init_workspace)"
echo "fetched" > /tmp/bake-after-test/source.txt
}
# @pipeline
# @after(fetch_source)
build_project() {
echo "Building project (depends on fetch_source)"
echo "built" > /tmp/bake-after-test/build.txt
}
-4
View File
@@ -1,4 +0,0 @@
#!/bin/bash
echo "EARLY_HOOK_EXECUTED"
echo "Working directory: $(pwd)"
echo "User: $(whoami)"
-3
View File
@@ -1,3 +0,0 @@
#!/bin/bash
echo "LATE_HOOK_EXECUTED"
echo "Working directory: $(pwd)"
-6
View File
@@ -1,6 +0,0 @@
version: "0.0.1"
plugin: {}
notification: {}
artifact: []
cleanup:
policy: "auto"
-13
View File
@@ -1,13 +0,0 @@
version: "0.0.1"
plugin: {}
notification: {}
artifact: []
cleanup:
policy: "auto"
hooks:
early:
- name: "finalize-early"
command: "echo 'finalize early hook'"
late:
- name: "finalize-late"
command: "echo 'finalize late hook'"
-6
View File
@@ -1,6 +0,0 @@
FROM docker.1ms.run/library/alpine:3.19
RUN sed -i 's|dl-cdn.alpinelinux.org|mirrors.tuna.tsinghua.edu.cn|g' /etc/apk/repositories
RUN apk add --no-cache python3 py3-pip bash
COPY workshop-baker/target/x86_64-unknown-linux-musl/debug/workshop-baker /baker
RUN chmod +x /baker
ENTRYPOINT ["/bin/sh", "-c"]
-6
View File
@@ -1,6 +0,0 @@
FROM docker.1ms.run/library/debian:bookworm-slim
RUN sed -i 's|deb.debian.org|mirrors.tuna.tsinghua.edu.cn|g' /etc/apt/sources.list.d/debian.sources 2>/dev/null; sed -i 's|security.debian.org|mirrors.tuna.tsinghua.edu.cn/debian-security|g' /etc/apt/sources.list.d/debian.sources 2>/dev/null; true
RUN apt-get update -qq && apt-get install -y -qq --no-install-recommends python3 python3-venv ca-certificates 2>&1 | tail -n 2
COPY workshop-baker/target/x86_64-unknown-linux-musl/debug/workshop-baker /baker
RUN chmod +x /baker
ENTRYPOINT ["/bin/sh", "-c"]
-6
View File
@@ -1,6 +0,0 @@
FROM docker.1ms.run/library/fedora:43
RUN sed -e 's|^metalink=|#metalink=|g' -e 's|^#baseurl=http://download.example/pub/fedora/linux|baseurl=https://mirrors.tuna.tsinghua.edu.cn/fedora|g' -i.bak /etc/yum.repos.d/fedora.repo /etc/yum.repos.d/fedora-updates.repo
RUN dnf install -y python3 2>&1 | tail -n 2
COPY workshop-baker/target/x86_64-unknown-linux-musl/debug/workshop-baker /baker
RUN chmod +x /baker
ENTRYPOINT ["/bin/sh", "-c"]
-2
View File
@@ -1,2 +0,0 @@
FROM docker.1ms.run/library/archlinux:latest
RUN echo 'Server = https://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch' > /etc/pacman.d/mirrorlist
-8
View File
@@ -1,8 +0,0 @@
version: "1.0"
environment:
builder: "baremetal"
bootstrap:
user: "vulcan"
workspace:
path: "/home/vulcan/workspace"
fallback: true
-15
View File
@@ -1,15 +0,0 @@
version: "1.0"
environment:
builder: "baremetal"
bootstrap:
user: "vulcan"
workspace:
path: "/home/vulcan/workspace"
fallback: true
hooks:
early:
- name: "test-early-hook"
command: "echo 'early hook executed'"
late:
- name: "test-late-hook"
command: "echo 'late hook executed'"
-3
View File
@@ -1,3 +0,0 @@
version: "0.5"
environment:
builder: "baremetal"
-8
View File
@@ -1,8 +0,0 @@
version: "1.0"
environment:
builder: "baremetal"
bootstrap:
user: "vulcan"
workspace:
path: "/home/vulcan/workspace"
fallback: true
-12
View File
@@ -1,12 +0,0 @@
version: "1.0"
environment:
builder: "baremetal"
bootstrap:
user: "vulcan"
workspace:
path: "/home/vulcan/workspace"
fallback: true
custom: |
#!/bin/sh
echo 'custom bootstrap executed'
mkdir -p /custom-workspace
-16
View File
@@ -1,16 +0,0 @@
version: "1.0"
environment:
builder: "baremetal"
bootstrap:
user: "vulcan"
workspace:
path: "/home/vulcan/workspace"
fallback: true
dependencies:
system:
pacman:
packages:
- "which"
user:
cargo:
- "ripgrep"
-13
View File
@@ -1,13 +0,0 @@
version: "1.0"
environment:
builder: "baremetal"
bootstrap:
user: "vulcan"
workspace:
path: "/home/vulcan/workspace"
fallback: true
envvars:
prebake:
TEST_PREBAKE_VAR: "prebake_value"
bake:
TEST_BAKE_VAR: "bake_value"
-17
View File
@@ -1,17 +0,0 @@
version: "1.0"
environment:
builder: "baremetal"
bootstrap:
user: "vulcan"
workspace:
path: "/home/vulcan/workspace"
fallback: true
hooks:
early:
- name: "test-early-hook"
command: "echo 'early hook executed'"
working_dir: "/workspace"
late:
- name: "test-late-hook"
command: "echo 'late hook executed'"
working_dir: "/workspace"
-14
View File
@@ -1,14 +0,0 @@
version: "1.0"
environment:
builder: "baremetal"
bootstrap:
user: "vulcan"
workspace:
path: "/home/vulcan/workspace"
fallback: true
dependencies:
system:
pacman:
packages:
- "curl"
- "wget"
-10
View File
@@ -1,10 +0,0 @@
version: "1.0"
environment:
builder: "baremetal"
bootstrap:
user: "vulcan"
workspace:
path: "/home/vulcan/workspace"
fallback: true
security:
drop_after: "Bootstrap"
-12
View File
@@ -1,12 +0,0 @@
"""
LLM-based log judgment system for system tests.
Uses LLM APIs to evaluate non-deterministic test outputs (logs, stderr, etc.)
where simple string matching is insufficient.
"""
from .client import LLMClient
from .cache import LLMCache
from .judge import judge_log, JudgmentResult
__all__ = ["LLMClient", "LLMCache", "judge_log", "JudgmentResult"]
-56
View File
@@ -1,56 +0,0 @@
"""
Disk-based cache for LLM API responses.
Uses SHA256 of (prompt + model + system_prompt) as cache key.
Cache files stored as JSON in ``.cache/`` directory.
"""
import hashlib
import json
from pathlib import Path
class LLMCache:
"""SHA256-based disk cache for LLM API responses."""
def __init__(self, cache_dir: str | None = None):
if cache_dir is None:
import os
cache_dir = os.path.join(os.path.dirname(__file__), ".cache")
self.cache_dir = Path(cache_dir)
self.cache_dir.mkdir(parents=True, exist_ok=True)
def _make_key(self, prompt: str, model: str, system_prompt: str = "") -> str:
"""Create deterministic cache key from request parameters."""
data = json.dumps(
{"prompt": prompt, "model": model, "system": system_prompt},
sort_keys=True,
)
return hashlib.sha256(data.encode()).hexdigest()
def get(self, prompt: str, model: str, system_prompt: str = "") -> dict | None:
"""Retrieve cached response if exists."""
key = self._make_key(prompt, model, system_prompt)
path = self.cache_dir / f"{key}.json"
if path.exists():
try:
with open(path) as f:
return json.load(f)
except (json.JSONDecodeError, IOError):
return None
return None
def set(
self, prompt: str, model: str, system_prompt: str, response: dict
) -> None:
"""Store response in cache."""
key = self._make_key(prompt, model, system_prompt)
path = self.cache_dir / f"{key}.json"
with open(path, "w") as f:
json.dump(response, f, ensure_ascii=False)
def clear(self) -> None:
"""Clear all cached responses."""
for f in self.cache_dir.glob("*.json"):
f.unlink()
-187
View File
@@ -1,187 +0,0 @@
"""
LLM client supporting both Anthropic and OpenAI APIs.
Prefers Anthropic format (per project convention).
Falls back to OpenAI format if Anthropic is unavailable.
"""
import os
import logging
from .cache import LLMCache
logger = logging.getLogger(__name__)
class LLMClient:
"""LLM client with caching support. Prefers Anthropic API."""
def __init__(self, cache: LLMCache | None = None):
self.cache = cache or LLMCache()
self._anthropic_client = None
self._openai_client = None
self._init_clients()
def _init_clients(self):
"""Initialize API clients based on available environment variables."""
# Anthropic (preferred)
anthropic_key = os.environ.get("ANTHROPIC_API_KEY")
anthropic_base = os.environ.get("ANTHROPIC_BASE_URL")
if anthropic_key:
try:
import anthropic
kwargs: dict = {"api_key": anthropic_key}
if anthropic_base:
kwargs["base_url"] = anthropic_base
self._anthropic_client = anthropic.Anthropic(**kwargs)
logger.info("Anthropic client initialized")
except ImportError:
logger.warning("anthropic package not installed")
# OpenAI (fallback)
openai_key = os.environ.get("OPENAI_API_KEY")
openai_base = os.environ.get("OPENAI_BASE_URL")
if openai_key:
try:
import openai
kwargs = {"api_key": openai_key}
if openai_base:
kwargs["base_url"] = openai_base
self._openai_client = openai.OpenAI(**kwargs)
logger.info("OpenAI client initialized")
except ImportError:
logger.warning("openai package not installed")
@property
def available(self) -> bool:
"""Check if any LLM client is available."""
return (
self._anthropic_client is not None or self._openai_client is not None
)
def complete(
self, prompt: str, system_prompt: str = "", max_tokens: int = 2000
) -> str:
"""Send a completion request, with caching.
Args:
prompt: User message.
system_prompt: System message (sets role / persona).
max_tokens: Max response tokens.
Returns:
LLM response text.
Raises:
RuntimeError: If no client is available.
"""
model = os.environ.get("MODEL", "deepseek-v4-flash")
# Check cache
cached = self.cache.get(prompt, model, system_prompt)
if cached:
logger.info("Cache hit for LLM request")
return cached["content"]
# Try Anthropic first
if self._anthropic_client:
content = self._call_anthropic(
prompt, system_prompt, model, max_tokens
)
elif self._openai_client:
content = self._call_openai(prompt, system_prompt, model, max_tokens)
else:
raise RuntimeError(
"No LLM client available. "
"Set ANTHROPIC_API_KEY or OPENAI_API_KEY."
)
# Cache the response
self.cache.set(prompt, model, system_prompt, {"content": content})
return content
# ------------------------------------------------------------------
# Private helpers
# ------------------------------------------------------------------
def _call_anthropic(
self, prompt: str, system_prompt: str, model: str, max_tokens: int
) -> str:
"""Call Anthropic API."""
kwargs: dict = {
"model": model,
"max_tokens": max_tokens,
"messages": [{"role": "user", "content": prompt}],
}
if system_prompt:
kwargs["system"] = system_prompt
# Only add reasoning_effort if the model supports it
reasoning_effort = os.environ.get("REASONING_EFFORT")
if reasoning_effort:
try:
kwargs["reasoning_effort"] = reasoning_effort
response = self._anthropic_client.messages.create(**kwargs)
except Exception as e:
if (
"reasoning_effort" in str(e).lower()
or "unexpected" in str(e).lower()
):
del kwargs["reasoning_effort"]
response = self._anthropic_client.messages.create(**kwargs)
else:
raise
else:
response = self._anthropic_client.messages.create(**kwargs)
# Extract text from response content blocks
if hasattr(response, "content") and response.content:
block = response.content[0]
if hasattr(block, "text"):
return block.text
if hasattr(block, "type") and block.type == "thinking":
for b in response.content:
if hasattr(b, "text"):
return b.text
return str(block)
return str(response)
def _call_openai(
self, prompt: str, system_prompt: str, model: str, max_tokens: int
) -> str:
"""Call OpenAI-compatible API."""
messages: list[dict] = []
if system_prompt:
messages.append({"role": "system", "content": system_prompt})
messages.append({"role": "user", "content": prompt})
kwargs: dict = {
"model": model,
"messages": messages,
"max_tokens": max_tokens,
}
# Only add reasoning_effort if supported
reasoning_effort = os.environ.get("REASONING_EFFORT")
if reasoning_effort:
try:
kwargs["reasoning_effort"] = reasoning_effort
response = self._openai_client.chat.completions.create(
**kwargs
)
except Exception as e:
if (
"reasoning_effort" in str(e).lower()
or "unexpected" in str(e).lower()
):
del kwargs["reasoning_effort"]
response = self._openai_client.chat.completions.create(
**kwargs
)
else:
raise
else:
response = self._openai_client.chat.completions.create(**kwargs)
return response.choices[0].message.content
-149
View File
@@ -1,149 +0,0 @@
"""
LLM-based log judgment for system tests.
Evaluates logs against expected behavior using LLM semantic analysis.
Returns structured pass/fail results with reasoning.
"""
import json
import re
import logging
from dataclasses import dataclass
from .client import LLMClient
logger = logging.getLogger(__name__)
JUDGE_SYSTEM_PROMPT = """\
You are a test result judge for a CI/CD system called HoneyBiscuitWorkshop.
Your job: Evaluate whether actual command output/logs match expected behavior.
RULES:
1. You MUST respond with valid JSON in this exact format:
{"verdict": "pass" | "fail", "reason": "brief explanation"}
2. "pass" = the output confirms the expected behavior occurred
3. "fail" = the output shows the expected behavior did NOT occur, or shows an error
4. Ignore irrelevant noise (timestamps, debug lines, unrelated warnings)
5. Focus ONLY on whether the specific expected behavior is confirmed or contradicted
6. Be strict: if the evidence is ambiguous, verdict is "fail"
7. Do NOT include any text outside the JSON object"""
@dataclass
class JudgmentResult:
"""Result of an LLM log judgment."""
verdict: str # "pass" or "fail"
reason: str
raw_response: str
@property
def passed(self) -> bool:
return self.verdict == "pass"
def _extract_relevant_logs(full_log: str, max_chars: int = 4000) -> str:
"""Extract relevant portions from logs, removing noise.
Keeps the last *max_chars* characters (most recent / relevant).
Removes common noise patterns.
"""
lines = full_log.strip().split("\n")
noise_patterns = [
r"^\d{4}-\d{2}-\d{2}T",
r"^DEBUG ",
r"^TRACE ",
]
filtered = [
line
for line in lines
if not any(re.match(p, line) for p in noise_patterns)
]
text = "\n".join(filtered)
if len(text) > max_chars:
text = "...\n" + text[-max_chars:]
return text
def judge_log(
operation: str,
expected_behavior: str,
actual_log: str,
client: LLMClient | None = None,
) -> JudgmentResult:
"""Judge whether actual logs match expected behavior.
Args:
operation: Description of what was attempted
(e.g. ``"prebake bootstrap stage"``).
expected_behavior: What should have happened
(e.g. ``"user vulcan was created"``).
actual_log: The actual stdout / stderr output.
client: ``LLMClient`` instance (creates default if ``None``).
Returns:
``JudgmentResult`` with verdict and reason.
"""
if client is None:
client = LLMClient()
if not client.available:
logger.warning("No LLM client available, returning fail verdict")
return JudgmentResult(
verdict="fail",
reason="No LLM client available for judgment",
raw_response="",
)
relevant_log = _extract_relevant_logs(actual_log)
prompt = f"""\
## Operation
{operation}
## Expected Behavior
{expected_behavior}
## Actual Output (relevant excerpts)
```
{relevant_log}
```
Evaluate: Does the actual output confirm the expected behavior occurred?
Respond with JSON only."""
try:
response = client.complete(
prompt=prompt,
system_prompt=JUDGE_SYSTEM_PROMPT,
max_tokens=500,
)
# Parse JSON response — handle potential markdown code blocks
text = response.strip()
if text.startswith("```"):
lines = text.split("\n")
text = "\n".join(lines[1:-1]) if len(lines) > 2 else text
result = json.loads(text)
return JudgmentResult(
verdict=result.get("verdict", "fail"),
reason=result.get("reason", "No reason provided"),
raw_response=response,
)
except json.JSONDecodeError as e:
logger.error(f"Failed to parse LLM response as JSON: {e}")
return JudgmentResult(
verdict="fail",
reason=f"LLM response was not valid JSON: {response[:200]}",
raw_response=response,
)
except Exception as e:
logger.error(f"LLM judgment failed: {e}")
return JudgmentResult(
verdict="fail",
reason=f"LLM call failed: {str(e)}",
raw_response="",
)
-45
View File
@@ -1,45 +0,0 @@
"""
Environment probes for verifying system state in workshop-baker tests.
Each probe is a standalone function that can be called from any test.
Probes return bool or Optional[int/str]; they do not raise on failure.
A probe accepts an optional *runner* callable (default ``subprocess.run``)
that decouples it from the execution context (host or Docker).
"""
from .user import user_exists, user_uid, user_gid, user_in_group, user_shell
from .file import (
file_exists,
file_absent,
dir_exists,
dir_absent,
file_permissions,
file_contains,
file_owned_by,
file_group,
)
from .package import binary_exists, binary_on_path, package_installed, package_version
from .process import process_running, process_count
__all__ = [
"user_exists",
"user_uid",
"user_gid",
"user_in_group",
"user_shell",
"file_exists",
"file_absent",
"dir_exists",
"dir_absent",
"file_permissions",
"file_contains",
"file_owned_by",
"file_group",
"package_installed",
"package_version",
"binary_exists",
"binary_on_path",
"process_running",
"process_count",
]
-53
View File
@@ -1,53 +0,0 @@
"""File and directory existence / permission probes."""
import os
import stat
import grp
import pwd
def file_exists(path: str) -> bool:
"""Check if *path* exists and is a regular file."""
return os.path.isfile(path)
def dir_exists(path: str) -> bool:
"""Check if *path* exists and is a directory."""
return os.path.isdir(path)
def file_permissions(path: str) -> int:
"""Return the permission mode of *path* as an octal int (e.g. ``0o755``).
Raises ``FileNotFoundError`` if *path* doesn't exist.
"""
return stat.S_IMODE(os.stat(path).st_mode)
def file_contains(path: str, text: str) -> bool:
"""Check if a file contains the specified text."""
with open(path, "r", errors="replace") as f:
return text in f.read()
def file_owned_by(path: str, username: str) -> bool:
"""Check if a file is owned by the specified user."""
st = os.stat(path)
owner = pwd.getpwuid(st.st_uid).pw_name
return owner == username
def file_group(path: str) -> str:
"""Get the group owner of a file."""
st = os.stat(path)
return grp.getgrgid(st.st_gid).gr_name
def file_absent(path: str) -> bool:
"""Check that *path* does not exist."""
return not os.path.lexists(path)
def dir_absent(path: str) -> bool:
"""Check that *path* does not exist or is not a directory."""
return not os.path.isdir(path)
-42
View File
@@ -1,42 +0,0 @@
"""Package / binary existence probes (cross-distro).
Each probe accepts an optional *runner* callable with the same signature
as ``subprocess.run``. Defaults to running on the host; for Docker tests,
pass a runner that wraps ``docker run --rm <image>``.
"""
import subprocess
from typing import Any, Callable
Runner = Callable[..., Any]
def _run(cmd: list[str], runner: Runner = subprocess.run, **kw: Any) -> Any:
opts = {"capture_output": True, "text": True, "timeout": 30}
opts.update(kw)
return runner(cmd, **opts)
def _succeeded(r: Any) -> bool:
""".returncode==0 for CompletedProcess, or ["success"] for dict."""
if hasattr(r, "returncode"):
return r.returncode == 0
return bool(r.get("success", False))
def binary_exists(path: str, runner: Runner = subprocess.run) -> bool:
"""Check if a binary exists at *path* (e.g. ``/usr/bin/curl``).
Uses ``test -f`` (POSIX, guaranteed available).
"""
return _succeeded(_run(["test", "-f", path], runner))
def binary_on_path(name: str, runner: Runner = subprocess.run) -> bool:
"""Check if a command is on PATH (via ``command -v``)."""
return _succeeded(_run(["sh", "-c", "command -v " + name], runner))
# Backward-compatible aliases
package_installed = binary_exists
package_version = binary_on_path # used in __init__ export, now binary-based
-44
View File
@@ -1,44 +0,0 @@
"""Process existence probes."""
import os
import subprocess
def process_running(process_name: str) -> bool:
"""Check if a process with the given name is running."""
try:
result = subprocess.run(
["pgrep", "-x", process_name],
capture_output=True,
text=True,
)
return result.returncode == 0
except FileNotFoundError:
# Fall back to /proc scan if pgrep not available
try:
for pid_dir in os.listdir("/proc"):
if pid_dir.isdigit():
try:
with open(f"/proc/{pid_dir}/comm") as f:
if f.read().strip() == process_name:
return True
except (IOError, OSError):
continue
except OSError:
pass
return False
def process_count(process_name: str) -> int:
"""Count processes with the given name."""
try:
result = subprocess.run(
["pgrep", "-c", "-x", process_name],
capture_output=True,
text=True,
)
if result.returncode == 0:
return int(result.stdout.strip())
return 0
except (FileNotFoundError, ValueError):
return 0
-41
View File
@@ -1,41 +0,0 @@
"""User existence and attribute probes."""
import pwd
import grp
def user_exists(username: str) -> bool:
"""Check if a system user exists."""
try:
pwd.getpwnam(username)
return True
except KeyError:
return False
def user_uid(username: str) -> int | None:
try:
return pwd.getpwnam(username).pw_uid
except KeyError:
return None
def user_gid(username: str) -> int | None:
try:
return pwd.getpwnam(username).pw_gid
except KeyError:
return None
def user_in_group(username: str, groupname: str) -> bool:
"""Check if a user is a member of a group."""
try:
group = grp.getgrnam(groupname)
return username in group.gr_mem
except KeyError:
return False
def user_shell(username: str) -> str | None:
try:
return pwd.getpwnam(username).pw_shell
except KeyError:
return None
-6
View File
@@ -1,6 +0,0 @@
pytest>=8.0.0
pytest-timeout>=2.2.0
pytest-xdist>=3.5.0
anthropic>=0.40.0,<0.90.0
openai>=1.50.0,<2.0.0
pyyaml>=6.0
+107
View File
@@ -0,0 +1,107 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 4
[[package]]
name = "itoa"
version = "1.0.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
[[package]]
name = "memchr"
version = "2.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8ca58f447f06ed17d5fc4043ce1b10dd205e060fb3ce5b979b8ed8e59ff3f79"
[[package]]
name = "proc-macro2"
version = "1.0.106"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934"
dependencies = [
"unicode-ident",
]
[[package]]
name = "quote"
version = "1.0.45"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924"
dependencies = [
"proc-macro2",
]
[[package]]
name = "serde"
version = "1.0.228"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e"
dependencies = [
"serde_core",
"serde_derive",
]
[[package]]
name = "serde_core"
version = "1.0.228"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad"
dependencies = [
"serde_derive",
]
[[package]]
name = "serde_derive"
version = "1.0.228"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "serde_json"
version = "1.0.149"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "83fc039473c5595ace860d8c4fafa220ff474b3fc6bfdb4293327f1a37e94d86"
dependencies = [
"itoa",
"memchr",
"serde",
"serde_core",
"zmij",
]
[[package]]
name = "syn"
version = "2.0.117"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99"
dependencies = [
"proc-macro2",
"quote",
"unicode-ident",
]
[[package]]
name = "unicode-ident"
version = "1.0.24"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75"
[[package]]
name = "workshop-baker-params"
version = "0.1.0"
dependencies = [
"serde",
"serde_json",
]
[[package]]
name = "zmij"
version = "1.0.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa"
+8
View File
@@ -0,0 +1,8 @@
[package]
name = "workshop-baker-params"
version = "0.1.0"
edition = "2024"
[dependencies]
serde = { version = "1.0.228", features = ["derive"] }
serde_json = "1.0.149"
+24
View File
@@ -0,0 +1,24 @@
use crate::apply_field;
use serde::{Deserialize, Serialize};
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct BakeParams {
#[serde(default)] pub workspace: ParamVal<String>,
}
impl MergeParams for BakeParams {
fn defaults() -> Self { Self { workspace: ParamVal::new("/workspace".into()) } }
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
apply_field!(self.workspace, i.workspace, w, ov, p, "workspace");
}
}
impl Default for BakeParams { fn default() -> Self { Self::defaults() } }
#[derive(Debug, Clone)]
pub struct BakeSettings { pub workspace: String }
impl Default for BakeSettings { fn default() -> Self { BakeParams::defaults().into() } }
impl From<&BakeParams> for BakeSettings { fn from(p: &BakeParams) -> Self { Self { workspace: p.workspace.value.clone() } } }
impl From<BakeParams> for BakeSettings { fn from(p: BakeParams) -> Self { (&p).into() } }
+23
View File
@@ -0,0 +1,23 @@
use crate::apply_field;
use serde::{Deserialize, Serialize};
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct FinalizeParams {
#[serde(default)] pub artifact_retention_days: ParamVal<u32>,
}
impl MergeParams for FinalizeParams {
fn defaults() -> Self { Self { artifact_retention_days: ParamVal::new(7) } }
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
apply_field!(self.artifact_retention_days, i.artifact_retention_days, w, ov, p, "artifact_retention_days");
}
}
impl Default for FinalizeParams { fn default() -> Self { Self::defaults() } }
#[derive(Debug, Clone)]
pub struct FinalizeSettings { pub artifact_retention_days: u32 }
impl Default for FinalizeSettings { fn default() -> Self { FinalizeParams::defaults().into() } }
impl From<&FinalizeParams> for FinalizeSettings { fn from(p: &FinalizeParams) -> Self { Self { artifact_retention_days: p.artifact_retention_days.value } } }
impl From<FinalizeParams> for FinalizeSettings { fn from(p: FinalizeParams) -> Self { (&p).into() } }
+168
View File
@@ -0,0 +1,168 @@
pub mod notification;
pub mod prebake;
pub mod bake;
pub mod finalize;
pub mod writer;
pub mod param;
pub mod traits;
pub use notification::{NotificationParams, NotificationSettings};
pub use prebake::{PrebakeParams, PrebakeSettings};
pub use bake::{BakeParams, BakeSettings};
pub use finalize::{FinalizeParams, FinalizeSettings};
pub use writer::Writer;
pub use param::ParamVal;
pub use traits::MergeParams;
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
/// Root parameter tree. Each component gets its own subtree.
/// Used for both IPC (Serialized to JSON) and in-process merging.
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct Params {
#[serde(default, skip_serializing_if = "Option::is_none")]
pub notification: Option<NotificationParams>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub prebake: Option<PrebakeParams>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub bake: Option<BakeParams>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub finalize: Option<FinalizeParams>,
}
/// Apply and merge helpers.
impl Params {
/// Merge a layer from a specific writer into the current params.
/// Returns entries that were overridden (for debug).
pub fn apply(&mut self, incoming: Params, writer: Writer) -> Overridden {
let mut ov = Overridden::new();
if let Some(l) = incoming.notification {
if let Some(ref mut b) = self.notification {
b.apply(l, writer, &mut ov, "notification.");
} else {
self.notification = Some(l);
}
}
if let Some(l) = incoming.prebake {
if let Some(ref mut b) = self.prebake {
b.apply(l, writer, &mut ov, "prebake.");
} else {
self.prebake = Some(l);
}
}
if let Some(l) = incoming.bake {
if let Some(ref mut b) = self.bake {
b.apply(l, writer, &mut ov, "bake.");
} else {
self.bake = Some(l);
}
}
if let Some(l) = incoming.finalize {
if let Some(ref mut b) = self.finalize {
b.apply(l, writer, &mut ov, "finalize.");
} else {
self.finalize = Some(l);
}
}
ov
}
/// Convert to baker-consumable settings. All params guaranteed to have values.
pub fn to_settings(&self) -> Result<AllSettings, String> {
Ok(AllSettings {
notification: self.notification.as_ref().map(NotificationSettings::from).unwrap_or_default(),
prebake: self.prebake.as_ref().map(PrebakeSettings::from).unwrap_or_default(),
bake: self.bake.as_ref().map(BakeSettings::from).unwrap_or_default(),
finalize: self.finalize.as_ref().map(FinalizeSettings::from).unwrap_or_default(),
})
}
}
/// Plain-value settings for baker consumption. No metadata, no Option.
#[derive(Debug, Clone)]
pub struct AllSettings {
pub notification: NotificationSettings,
pub prebake: PrebakeSettings,
pub bake: BakeSettings,
pub finalize: FinalizeSettings,
}
/// Debug — values that were overridden during merge, keyed by dot-path.
#[derive(Debug, Clone, Default)]
pub struct Overridden {
pub entries: std::collections::HashMap<String, serde_json::Value>,
}
impl Overridden {
pub fn new() -> Self { Self { entries: HashMap::new() } }
pub fn is_empty(&self) -> bool { self.entries.is_empty() }
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_empty_params_serializes_empty() {
let p = Params::default();
let json = serde_json::to_string(&p).unwrap();
assert_eq!(json, "{}");
}
#[test]
fn test_notification_params_serialize() {
let mut p = Params::default();
p.notification = Some(NotificationParams::defaults());
let json = serde_json::to_string_pretty(&p).unwrap();
assert!(json.contains("max_lives"));
assert!(json.contains("3"));
}
#[test]
fn test_apply_courier_overrides_default() {
let mut base = Params::default();
base.notification = Some(NotificationParams::defaults());
let incoming = Params {
notification: Some(NotificationParams {
max_lives: ParamVal::with(5, Writer::Courier),
..Default::default()
}),
..Default::default()
};
let ov = base.apply(incoming, Writer::Courier);
let s = base.to_settings().unwrap();
assert_eq!(s.notification.max_lives, 5);
assert!(ov.is_empty()); // default had no writer, not an "override"
}
#[test]
fn test_apply_upstream_wins() {
let mut base = Params::default();
base.notification = Some(NotificationParams {
max_lives: ParamVal::with(3, Writer::Base),
..Default::default()
});
let incoming = Params {
notification: Some(NotificationParams {
max_lives: ParamVal::with(5, Writer::Courier),
..Default::default()
}),
..Default::default()
};
let ov = base.apply(incoming, Writer::Courier);
let s = base.to_settings().unwrap();
// base(3) > courier(2) — rejected, base stays
assert_eq!(s.notification.max_lives, 3);
assert!(ov.is_empty()); // nothing overridden
}
#[test]
fn test_deny_unknown_fields() {
let bad = r#"{"max_lives":{"value":5},"ghost_param":{"value":1}}"#;
let result: Result<NotificationParams, _> = serde_json::from_str(bad);
assert!(result.is_err()); // unknown field should be denied
}
}
+65
View File
@@ -0,0 +1,65 @@
use crate::apply_field;
use serde::{Deserialize, Serialize};
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct NotificationParams {
#[serde(default)] pub max_lives: ParamVal<u32>,
#[serde(default)] pub timeout_ms: ParamVal<u64>,
#[serde(default)] pub retry_backoff: ParamVal<f64>,
#[serde(default)] pub retry_delay_seconds: ParamVal<f64>,
#[serde(default)] pub max_retry_delay_seconds: ParamVal<f64>,
#[serde(default)] pub life_impact_factor: ParamVal<f64>,
#[serde(default)] pub batch_period: ParamVal<u32>,
#[serde(default)] pub batch_watermark: ParamVal<u32>,
#[serde(default)] pub template_ref_depth: ParamVal<u32>,
}
impl MergeParams for NotificationParams {
fn defaults() -> Self {
Self {
max_lives: ParamVal::new(3), timeout_ms: ParamVal::new(30000),
retry_backoff: ParamVal::new(2.0), retry_delay_seconds: ParamVal::new(5.0),
max_retry_delay_seconds: ParamVal::new(300.0),
life_impact_factor: ParamVal::new(0.0),
batch_period: ParamVal::new(30), batch_watermark: ParamVal::new(5),
template_ref_depth: ParamVal::new(10),
}
}
fn apply(&mut self, incoming: Self, writer: Writer, ov: &mut Overridden, prefix: &str) {
apply_field!(self.max_lives, incoming.max_lives, writer, ov, prefix, "max_lives");
apply_field!(self.timeout_ms, incoming.timeout_ms, writer, ov, prefix, "timeout_ms");
apply_field!(self.retry_backoff, incoming.retry_backoff, writer, ov, prefix, "retry_backoff");
apply_field!(self.retry_delay_seconds, incoming.retry_delay_seconds, writer, ov, prefix, "retry_delay_seconds");
apply_field!(self.max_retry_delay_seconds, incoming.max_retry_delay_seconds, writer, ov, prefix, "max_retry_delay_seconds");
apply_field!(self.life_impact_factor, incoming.life_impact_factor, writer, ov, prefix, "life_impact_factor");
apply_field!(self.batch_period, incoming.batch_period, writer, ov, prefix, "batch_period");
apply_field!(self.batch_watermark, incoming.batch_watermark, writer, ov, prefix, "batch_watermark");
apply_field!(self.template_ref_depth, incoming.template_ref_depth, writer, ov, prefix, "template_ref_depth");
}
}
impl Default for NotificationParams { fn default() -> Self { Self::defaults() } }
#[derive(Debug, Clone)]
pub struct NotificationSettings {
pub max_lives: u32, pub timeout_ms: u64, pub retry_backoff: f64,
pub retry_delay_seconds: f64, pub max_retry_delay_seconds: f64,
pub life_impact_factor: f64,
pub batch_period: u32, pub batch_watermark: u32, pub template_ref_depth: u32,
}
impl Default for NotificationSettings { fn default() -> Self { NotificationParams::defaults().into() } }
impl From<&NotificationParams> for NotificationSettings {
fn from(p: &NotificationParams) -> Self { Self {
max_lives: p.max_lives.value, timeout_ms: p.timeout_ms.value,
retry_backoff: p.retry_backoff.value, retry_delay_seconds: p.retry_delay_seconds.value,
max_retry_delay_seconds: p.max_retry_delay_seconds.value,
life_impact_factor: p.life_impact_factor.value,
batch_period: p.batch_period.value, batch_watermark: p.batch_watermark.value,
template_ref_depth: p.template_ref_depth.value,
}}
}
impl From<NotificationParams> for NotificationSettings { fn from(p: NotificationParams) -> Self { (&p).into() } }
+32
View File
@@ -0,0 +1,32 @@
use serde::{Deserialize, Serialize};
use crate::Writer;
/// A strongly-typed parameter value with writer tracking.
///
/// Serializes as `{ "value": 3, "writer": "base" }`.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ParamVal<T> {
pub value: T,
#[serde(skip_serializing_if = "Option::is_none")]
pub writer: Option<Writer>,
}
impl<T: Default> Default for ParamVal<T> {
fn default() -> Self {
ParamVal { value: T::default(), writer: None }
}
}
impl<T> ParamVal<T> {
pub fn new(value: T) -> Self {
ParamVal { value, writer: None }
}
pub fn with(value: T, writer: Writer) -> Self {
ParamVal { value, writer: Some(writer) }
}
/// Whether an incoming value from `incoming_writer` should override this one.
pub fn should_override(&self, incoming: Writer) -> bool {
let current = self.writer.map_or(0, |w| w.priority());
incoming.priority() > current
}
}
+54
View File
@@ -0,0 +1,54 @@
pub mod network;
pub mod cache;
pub mod engine;
use crate::apply_field;
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct PrebakeParams {
#[serde(default)] pub bootstrap_user: ParamVal<String>,
#[serde(default)] pub workspace: ParamVal<String>,
#[serde(default)] pub network: network::NetworkParams,
#[serde(default)] pub cache: cache::CacheParams,
#[serde(default)] pub engine: engine::EngineParams,
#[serde(default)] pub repology_endpoint: ParamVal<String>,
}
impl MergeParams for PrebakeParams {
fn defaults() -> Self {
Self {
bootstrap_user: ParamVal::new("vulcan".into()), workspace: ParamVal::new("/home/vulcan/workspace".into()),
network: network::NetworkParams::defaults(), cache: cache::CacheParams::defaults(),
engine: engine::EngineParams::defaults(),
repology_endpoint: ParamVal::new("default".into()),
}
}
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
apply_field!(self.bootstrap_user, i.bootstrap_user, w, ov, p, "bootstrap_user");
apply_field!(self.workspace, i.workspace, w, ov, p, "workspace");
apply_field!(self.repology_endpoint, i.repology_endpoint, w, ov, p, "repology_endpoint");
self.network.apply(i.network, w, ov, &format!("{}network.", p));
self.cache.apply(i.cache, w, ov, &format!("{}cache.", p));
self.engine.apply(i.engine, w, ov, &format!("{}engine.", p));
}
}
impl Default for PrebakeParams { fn default() -> Self { Self::defaults() } }
#[derive(Debug, Clone)]
pub struct PrebakeSettings {
pub bootstrap_user: String, pub workspace: String,
pub network: network::NetworkSettings, pub cache: cache::CacheSettings,
pub engine: engine::EngineSettings, pub repology_endpoint: String,
}
impl Default for PrebakeSettings { fn default() -> Self { PrebakeParams::defaults().into() } }
impl From<&PrebakeParams> for PrebakeSettings {
fn from(p: &PrebakeParams) -> Self { Self {
bootstrap_user: p.bootstrap_user.value.clone(), workspace: p.workspace.value.clone(),
network: (&p.network).into(), cache: (&p.cache).into(), engine: (&p.engine).into(),
repology_endpoint: p.repology_endpoint.value.clone(),
}}
}
impl From<PrebakeParams> for PrebakeSettings { fn from(p: PrebakeParams) -> Self { (&p).into() } }
@@ -0,0 +1,37 @@
use crate::apply_field;
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CacheParams {
#[serde(default)] pub strategy: ParamVal<String>,
#[serde(default)] pub mode: ParamVal<String>,
#[serde(default)] pub ttl_days: ParamVal<u32>,
#[serde(default)] pub max_size_gb: ParamVal<u32>,
#[serde(default)] pub cleanup_policy: ParamVal<String>,
}
impl MergeParams for CacheParams {
fn defaults() -> Self {
Self {
strategy: ParamVal::new("always".into()), mode: ParamVal::new("zstd".into()),
ttl_days: ParamVal::new(30), max_size_gb: ParamVal::new(20),
cleanup_policy: ParamVal::new("lru".into()),
}
}
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
apply_field!(self.strategy, i.strategy, w, ov, p, "strategy");
apply_field!(self.mode, i.mode, w, ov, p, "mode");
apply_field!(self.ttl_days, i.ttl_days, w, ov, p, "ttl_days");
apply_field!(self.max_size_gb, i.max_size_gb, w, ov, p, "max_size_gb");
apply_field!(self.cleanup_policy, i.cleanup_policy, w, ov, p, "cleanup_policy");
}
}
impl Default for CacheParams { fn default() -> Self { Self::defaults() } }
#[derive(Debug, Clone)]
pub struct CacheSettings { pub strategy: String, pub mode: String, pub ttl_days: u32, pub max_size_gb: u32, pub cleanup_policy: String }
impl Default for CacheSettings { fn default() -> Self { CacheParams::defaults().into() } }
impl From<&CacheParams> for CacheSettings { fn from(p: &CacheParams) -> Self {
Self { strategy: p.strategy.value.clone(), mode: p.mode.value.clone(), ttl_days: p.ttl_days.value, max_size_gb: p.max_size_gb.value, cleanup_policy: p.cleanup_policy.value.clone() }
} }
impl From<CacheParams> for CacheSettings { fn from(p: CacheParams) -> Self { (&p).into() } }
@@ -0,0 +1,21 @@
use crate::apply_field;
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct EngineParams {
#[serde(default)] pub timeout_ms: ParamVal<u64>,
}
impl MergeParams for EngineParams {
fn defaults() -> Self { Self { timeout_ms: ParamVal::new(300000) } }
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
apply_field!(self.timeout_ms, i.timeout_ms, w, ov, p, "timeout_ms");
}
}
impl Default for EngineParams { fn default() -> Self { Self::defaults() } }
#[derive(Debug, Clone)]
pub struct EngineSettings { pub timeout_ms: u64 }
impl Default for EngineSettings { fn default() -> Self { EngineParams::defaults().into() } }
impl From<&EngineParams> for EngineSettings { fn from(p: &EngineParams) -> Self { Self { timeout_ms: p.timeout_ms.value } } }
impl From<EngineParams> for EngineSettings { fn from(p: EngineParams) -> Self { (&p).into() } }
@@ -0,0 +1,23 @@
use crate::apply_field;
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct NetworkParams {
#[serde(default)] pub enabled: ParamVal<bool>,
#[serde(default)] pub outbound: ParamVal<bool>,
}
impl MergeParams for NetworkParams {
fn defaults() -> Self { Self { enabled: ParamVal::new(true), outbound: ParamVal::new(true) } }
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
apply_field!(self.enabled, i.enabled, w, ov, p, "enabled");
apply_field!(self.outbound, i.outbound, w, ov, p, "outbound");
}
}
impl Default for NetworkParams { fn default() -> Self { Self::defaults() } }
#[derive(Debug, Clone)]
pub struct NetworkSettings { pub enabled: bool, pub outbound: bool }
impl Default for NetworkSettings { fn default() -> Self { NetworkParams::defaults().into() } }
impl From<&NetworkParams> for NetworkSettings { fn from(p: &NetworkParams) -> Self { Self { enabled: p.enabled.value, outbound: p.outbound.value } } }
impl From<NetworkParams> for NetworkSettings { fn from(p: NetworkParams) -> Self { (&p).into() } }
+28
View File
@@ -0,0 +1,28 @@
use crate::{Writer, Overridden};
/// A parameter tree node that can merge itself with an incoming layer.
pub trait MergeParams: Sized {
fn apply(&mut self, incoming: Self, writer: Writer, ov: &mut Overridden, prefix: &str);
fn defaults() -> Self;
}
/// Apply one field's merge logic.
/// ```ignore
/// apply_field!(self.foo, incoming.foo, writer, ov, prefix, "foo");
/// ```
#[macro_export]
macro_rules! apply_field {
($self:ident.$field:ident, $incoming:ident.$field2:ident, $writer:expr, $ov:ident, $prefix:expr, $name:expr) => {
let old_w = $self.$field.writer;
if $self.$field.should_override($writer) {
let old_v = ::std::mem::replace(&mut $self.$field.value, $incoming.$field2.value);
$self.$field.writer = ::std::option::Option::Some($writer);
if let ::std::option::Option::Some(w) = old_w {
$ov.entries.insert(
format!("{}{}", $prefix, $name),
::serde_json::json!({"value": old_v, "writer": w.as_str()}),
);
}
}
};
}
+26
View File
@@ -0,0 +1,26 @@
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum Writer {
Base,
Courier,
Bakerd,
}
impl Writer {
/// Priority: Base=3 > Courier=2 > Bakerd=1
pub fn priority(self) -> u8 {
match self {
Writer::Base => 3,
Writer::Courier => 2,
Writer::Bakerd => 1,
}
}
pub fn as_str(self) -> &'static str {
match self {
Writer::Base => "base", Writer::Courier => "courier", Writer::Bakerd => "bakerd",
}
}
}
impl Default for Writer { fn default() -> Self { Writer::Base } }
+1
View File
@@ -1,3 +1,4 @@
temp temp
examples examples
quicktest.sh quicktest.sh
_env.sh
+84 -1471
View File
File diff suppressed because it is too large Load Diff
+2 -8
View File
@@ -47,13 +47,10 @@ uuid = { version = "1.19.0", features = ["v4"] }
libc = "0.2" libc = "0.2"
os_info = "3.14.0" os_info = "3.14.0"
privdrop = "0.5.6" privdrop = "0.5.6"
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
which = "8.0.2" which = "8.0.2"
gix = "0.81.0"
sha2 = "0.10"
flate2 = "1.0"
zstd = "0.13"
globset = "0.4.18" globset = "0.4.18"
axum = "0.7"
workshop-baker-params = { version = "0.1.0", path = "../workshop-baker-params" }
[dev-dependencies] [dev-dependencies]
criterion = "0.5" criterion = "0.5"
@@ -66,6 +63,3 @@ unreachable_code = "allow"
inherent_to_string = "allow" inherent_to_string = "allow"
non_canonical_partial_ord_impl = "allow" non_canonical_partial_ord_impl = "allow"
[[bench]]
name = "parser_bench"
harness = false
-93
View File
@@ -1,93 +0,0 @@
#!/usr/bin/env python3
import os
import random
import string
OUTPUT_DIR = "/tmp"
DECORATORS = [
"# @pipeline",
"# @fallible",
"# @parallel",
"# @export",
"# @timeout(60)",
"# @retry(3, 5)",
"# @if(condition)",
"# @after(dep)",
"# @pipe(step1, step2)",
"# @health(/health)",
"# @loop(10)",
]
FUNCTION_BODIES = [
"echo 'Processing...';",
"local var=$(date +%s);",
"if [ -f /tmp/test ]; then echo 'exists'; fi;",
"for i in $(seq 1 10); do echo $i; done;",
"case $1 in start) echo 'starting';; stop) echo 'stopping';; esac;",
"read -r line < /dev/stdin;",
"export PATH=$PATH:/usr/local/bin;",
"set -e; set -u;",
"trap 'echo error' ERR;",
"cd /tmp || exit 1;",
"local result=$(grep -r 'pattern' /var/log/ 2>/dev/null);",
'for f in /tmp/*.tmp; do rm -f "$f"; done',
"if [[ $var -gt 100 ]]; then echo 'large'; fi;",
'while read -r line; do echo "$line"; done < /etc/passwd;',
'select opt in a b c; do echo "Selected $opt"; break; done;',
]
def random_string(length=8):
return "".join(random.choices(string.ascii_letters + string.digits, k=length))
def generate_function(body_lines):
lines = []
func_name = f"func_{random_string(6)}"
for _ in range(random.randint(0, 3)):
lines.append(random.choice(DECORATORS))
lines.append(f"{func_name}() {{")
for _ in range(body_lines):
lines.append(" " + random.choice(FUNCTION_BODIES))
lines.append("}")
lines.append("")
return lines
def generate_script(target_lines, num_functions, target_bytes):
lines = []
total_lines = 0
body_per_func = max(1, (target_lines - num_functions * 3) // num_functions)
for _ in range(num_functions):
func_lines = generate_function(body_per_func)
lines.extend(func_lines)
total_lines += len(func_lines)
while (
total_lines < target_lines
or len("\n".join(lines).encode("utf-8")) < target_bytes
):
lines.append(random.choice(FUNCTION_BODIES))
total_lines += 1
return "\n".join(lines)
def main():
sizes = [
("small", 1000, 10, 50_000),
("medium", 5000, 50, 500_000),
("large", 10000, 100, 950_000),
]
for name, lines, funcs, target_bytes in sizes:
script = generate_script(lines, funcs, target_bytes)
filepath = os.path.join(OUTPUT_DIR, f"bench_{name}.sh")
with open(filepath, "w") as f:
f.write(script)
actual_bytes = len(script.encode("utf-8"))
actual_lines = len(script.splitlines())
print(f"Generated {filepath}: lines={actual_lines}, bytes={actual_bytes}")
if __name__ == "__main__":
main()
-61
View File
@@ -1,61 +0,0 @@
use criterion::{Criterion, black_box, criterion_group, criterion_main};
use std::fs;
use std::path::Path;
fn parse_large_script(c: &mut Criterion) {
let script_path = Path::new("/tmp/bench_large.sh");
let script_content = fs::read_to_string(script_path)
.expect("Failed to read benchmark script. Run `python3 benches/generate.py` first.");
let byte_size = script_content.len();
let line_count = script_content.lines().count();
println!("Benchmark file: {} lines, {} bytes", line_count, byte_size);
c.bench_function("parse_script_large", |b| {
b.iter(|| {
let result =
workshop_baker::bake::parser::parse_script(black_box(&script_content.clone()));
black_box(result)
});
});
}
fn parse_medium_script(c: &mut Criterion) {
let script_path = Path::new("/tmp/bench_medium.sh");
let script_content = fs::read_to_string(script_path)
.expect("Failed to read benchmark script. Run `python3 benches/generate.py` first.");
c.bench_function("parse_script_medium", |b| {
b.iter(|| {
let result =
workshop_baker::bake::parser::parse_script(black_box(&script_content.clone()));
black_box(result)
});
});
}
fn parse_small_script(c: &mut Criterion) {
let script_path = Path::new("/tmp/bench_small.sh");
let script_content = fs::read_to_string(script_path)
.expect("Failed to read benchmark script. Run `python3 benches/generate.py` first.");
c.bench_function("parse_script_small", |b| {
b.iter(|| {
let result =
workshop_baker::bake::parser::parse_script(black_box(&script_content.clone()));
black_box(result)
});
});
}
criterion_group!(
benches,
parse_small_script,
parse_medium_script,
parse_large_script
);
criterion_main!(benches);
+42 -47
View File
@@ -1,62 +1,57 @@
# workshop-baker/src # workshop-baker/src
**Generated:** 2026-04-22 **Generated:** 2026-05-19
**Commit:** 7b3b71a **Commit:** 28abc5d
Baker CLI + execution engine for HoneyBiscuitWorkshop pipelines. CLI tool and runtime for executing CI/CD pipelines. Orchestrates prebake → bake → finalize stages with notification dispatch.
## OVERVIEW
CLI tool and runtime for executing CI/CD pipelines with resource isolation, privilege dropping, and package management integration.
## STRUCTURE ## STRUCTURE
``` ```
src/ src/
├── bake.rs # Top-level bake orchestration ├── lib.rs # Module declarations + Cli struct + Commands enum
├── bake/ # Script parsing and building ├── main.rs # tokio::main entry point
├── parser.rs # @decorator-aware shell parser ├── bare.rs # Bare mode (run individual stages)
│ ├── decorator.rs # @pipeline, @retry, @parallel, etc. ├── error.rs # CliError (wraps stage errors)
├── builder.rs # Script template builder ├── bake.rs # Bake orchestration (script → execution)
│ └── schedule.rs # Function ordering via @after deps ├── bake/ # Script parsing, decorator DSL, scheduling
├── engine/ # Execution runtime ├── prebake.rs # Prebake orchestration (env setup)
│ ├── executor.rs # Script execution with cgroups ├── prebake/ # Config, stages, security, env providers
│ ├── cgroups.rs # Linux cgroup resource limits ├── finalize.rs # Finalize orchestration (post-build)
│ ├── repology.rs # Package name resolution ├── finalize/ # Config, plugins, hooks, templates
│ ├── pm.rs # Package manager detection ├── notify.rs # Notification system (batching, retry)
│ └── upm.rs # User package manager (Nix/Guix) ├── notify/ # Types, templates, interactive, queue
├── prebake/ # Build environment setup ├── daemon/ # Empty (Daemon command = unimplemented!)
├── config.rs # PrebakeConfig YAML schema ├── monitor/ # Resource usage monitoring (cgroups)
│ ├── stage/ # Bootstrap, DepsSystem, DepsUser, Hooks ├── types.rs # Module re-exports
│ └── security.rs # Privilege dropping logic ├── types/ # buildstatus, resource types
├── types/ # Shared type definitions ├── utils.rs # Module root (fetch submodule)
│ ├── repology.rs # RepologyEndpoint enum └── utils/ # fetch.rs
│ ├── builderconfig.rs
│ ├── cache.rs
│ └── memsize.rs
├── monitor/ # Resource usage monitoring
│ ├── cgroups.rs # cgroup-based usage collection
│ └── jobobject.rs # Windows job objects (stub)
├── daemon.rs # Unix socket daemon (TODO: refactor)
└── lib.rs # CLI struct + module exports
``` ```
## WHERE TO LOOK ## WHERE TO LOOK
| Task | Location | | Task | Location |
|------|----------| |------|----------|
| Script parsing with @decorators | `bake/parser.rs:29` - `parse_script()` | | CLI args | `lib.rs:21` - `Cli` struct |
| Decorator types | `bake/decorator.rs` - `Decorator` enum | | Entry point | `main.rs:91` - `#[tokio::main]` |
| Build execution | `engine/executor.rs` - `Executor::execute_script()` | | Pipeline orchestration | `main.rs:68-74` - prebake→bake→finalize chain |
| Resource limits | `engine/types.rs:56` - `ResourceLimits` struct | | Bare mode (single stage) | `bare.rs:14` - `run_bare()` |
| Prebake stages | `prebake/stage/` subdirectories | | Script parsing | `bake/parser.rs:56` - `parse_script()` |
| Package detection | `engine/pm.rs:detection()` | | Decorator DSL | `bake/decorator.rs` - `Decorator` enum |
| CLI entry | `lib.rs:26` - `cli::Cli` struct | | Prebake stages | `prebake/stage/` - Bootstrap→DepsSystem→DepsUser→Hooks→Ready |
| Plugin system | `finalize/plugin.rs` + `finalize/plugin/` |
| Notification dispatch | `notify.rs:66` - `notify()` async loop |
| Notification types | `notify/types.rs` - `NotificationRule`, `ParsedTrigger` |
## CONVENTIONS ## CONVENTIONS
- **ExecutionContext**: Passed through all stages, carries task_id, username, env_vars, timeout - **ExecutionContext**: Passed through all stages, carries task_id, username, env_vars, timeout
- **PrebakeStage**: Ordered enum controlling stage execution sequence (Bootstrap -> Ready) - **PrebakeStage**: Ordered enum Bootstrap→EarlyHook→DepsSystem→DepsUser→LateHook→Ready
- **Decorator**: Attached to shell functions via `# @name(args)` comments above function declarations - **Decorator**: `# @name(args)` comments above shell function declarations
- **Engine**: Lightweight wrapper around cgroup manager for resource isolation - **Notification**: Plugin-based dispatch with batching, exponential backoff, priority groups
- **Dual-mode**: CLI commands (prebake, bake, finalize) vs daemon mode via Unix socket - **Engine**: Delegated to `workshop-engine` crate (formerly in-tree)
- **Dual-mode**: CLI commands (standalone) vs Daemon mode (unimplemented)
## ANTI-PATTERNS
1. **Empty daemon/** directory — Daemon command is `unimplemented!()`
2. **Dead code allowed**`lints.rust.dead_code = "allow"` in Cargo.toml
3. **Engine was extracted** — workshop-engine is its own crate now, src/engine/ removed
4. **Python test infra exists but files removed** — pytest.ini remains, test dir is empty
+4 -2
View File
@@ -12,14 +12,16 @@ bake/
├── parser.rs # @decorator-aware shell script parser ├── parser.rs # @decorator-aware shell script parser
├── decorator.rs # Decorator enum + 100+ unit tests ├── decorator.rs # Decorator enum + 100+ unit tests
├── builder.rs # Script template assembly ├── builder.rs # Script template assembly
── schedule.rs # Function ordering via @after dependencies ── schedule.rs # Function ordering via @after dependencies
├── error.rs # BakeError enum
└── constant.rs # DEFAULT_WORKSPACE, TRIVIAL_SCRIPT_NAME
``` ```
## WHERE TO LOOK ## WHERE TO LOOK
| Task | Location | | Task | Location |
|------|----------| |------|----------|
| Parse script | `parser.rs:29` - `parse_script()` | Parse script | `parser.rs:56` - `parse_script()` |
| Decorator types | `decorator.rs` - `Decorator` enum variants | | Decorator types | `decorator.rs` - `Decorator` enum variants |
| Build script | `builder.rs` - template builder | | Build script | `builder.rs` - template builder |
| Schedule deps | `schedule.rs` - @after resolution | | Schedule deps | `schedule.rs` - @after resolution |
+50
View File
@@ -0,0 +1,50 @@
use std::path::PathBuf;
use crate::cli::{BareCommands, Cli};
use crate::error::CliError;
use crate::types::resource::ResourceRegistry;
use crate::{bake::bake, finalize::finalize, prebake::prebake};
use workshop_engine::{EventSender, ExecutionContext};
fn or_workshop(path: &Option<PathBuf>, name: &str) -> PathBuf {
path
.clone()
.unwrap_or_else(|| PathBuf::from(format!(".workshop/{}", name)))
}
pub async fn run_bare(
cli: &Cli,
command: &BareCommands,
ctx: &mut ExecutionContext,
event_tx: EventSender,
pipeline: &str,
build_id: &str,
registry: &ResourceRegistry,
) -> Result<(), CliError> {
let (stage, config) = match command {
BareCommands::Prebake { .. } => ("prebake", or_workshop(&cli.prebake, "prebake.yml")),
BareCommands::Bake { .. } => ("bake", or_workshop(&cli.bake, "bake.sh")),
BareCommands::Finalize { .. } => {
("finalize", or_workshop(&cli.finalize, "finalize.yml"))
}
};
let prebake_path = or_workshop(&cli.prebake, "prebake.yml");
let bake_base = or_workshop(&cli.bake_base, "bake_base.sh");
ctx.task_id = format!("{}-{}-{}", pipeline, build_id, stage);
ctx.pipeline_name = pipeline.to_string();
ctx.standalone = true;
match command {
BareCommands::Prebake { .. } => {
prebake(&config, cli, None, ctx, event_tx, None).await?;
}
BareCommands::Bake { .. } => {
bake(&config, &bake_base, &prebake_path, cli, ctx, event_tx).await?;
}
BareCommands::Finalize { .. } => {
finalize(&config, cli, ctx, event_tx, registry).await?;
}
}
Ok(())
}
-193
View File
@@ -1,193 +0,0 @@
use crate::cli::Cli;
// use serde::Serialize;
use serde_json::Value;
// use std::path::PathBuf;
// use std::sync::Arc;
use tokio::io::{AsyncBufReadExt, BufReader};
use tokio::net::UnixStream;
// use tokio::process::{Child, Command};
// use tokio::signal;
// use tokio::sync::RwLock;
// use tokio::time::{Duration, interval};
// #[derive(Debug, Clone, Serialize)]
// struct Heartbeat {
// pipeline_id: String,
// build_id: String,
// stage: String,
// timestamp: u64,
// status: String,
// subtasks: Vec<String>,
// }
pub async fn daemon(_cli: &Cli) {
todo!("Daemon should be refactored");
// let socket = PathBuf::from(settings.get_string("socket").unwrap());
// let agentsocket = settings.get_string("agentsocket").unwrap();
// let _ = std::fs::remove_file(&socket);
// let heartbeat = Arc::new(RwLock::new(Heartbeat {
// pipeline_id: cli.pipeline.clone(),
// build_id: cli.build_id.clone(),
// stage: String::new(),
// timestamp: 0,
// status: String::new(),
// subtasks: Vec::new(),
// }));
// // let heartbeat_clone = heartbeat.clone();
// let heartbeat_task = {
// let hb = heartbeat.clone();
// tokio::spawn(async move {
// let mut tick = interval(Duration::from_secs(1));
// loop {
// tick.tick().await;
// let mut hb = hb.write().await;
// hb.timestamp = std::time::SystemTime::now()
// .duration_since(std::time::UNIX_EPOCH)
// .unwrap()
// .as_millis() as u64;
// // 只输出到stdout(模拟未来发送给Agent
// println!("[HEARTBEAT] {}", serde_json::to_string(&*hb).unwrap());
// }
// })
// };
// let socket_task = {
// let socket = socket.clone();
// tokio::spawn(async move {
// log::info!("Creating Unix socket at {}", socket.display());
// let listener =
// UnixListener::bind(&socket).expect("Failed to bind Unix socket");
// loop {
// match listener.accept().await {
// Ok((stream, _)) => {
// tokio::spawn(handle_connection(stream));
// }
// Err(e) => {
// log::error!("Accept error: {}", e);
// break;
// }
// }
// }
// })
// };
// let mut tasks: Vec<TaskSpec> = Vec::new();
// let execution_task = {
// let hb = heartbeat.clone();
// tokio::spawn(async move {
// let mut hb = hb.write().await;
// let mut children: Vec<(String, Child)> = vec![];
// let executable = std::env::current_exe().unwrap();
// for (index, task) in tasks.iter().enumerate() {
// let mut command = task.command(&executable);
// command.stdout(std::process::Stdio::piped());
// command.stderr(std::process::Stdio::piped());
// let child = command
// .spawn()
// .expect(format!("Failed to spawn child {}", index).as_str());
// let pid = child.id().unwrap_or(0).to_string();
// children.push((pid.clone(), child));
// hb.subtasks.push(pid.clone());
// }
// for (pid, mut child) in children {
// let status = child.wait().await.expect("Failed to wait child");
// log::info!("Subtask {} exited with: {}", pid, status);
// hb.subtasks.retain(|p| p != &pid);
// }
// hb.stage = "Success".to_string();
// })
// };
// tokio::select! {
// _ = heartbeat_task => {
// log::error!("Heartbeat task died");
// }
// _ = execution_task => {
// log::info!("All subtasks completed");
// }
// _ = socket_task => {
// log::error!("Socket task died");
// }
// _ = signal::ctrl_c() => {
// log::info!("Shutting down...");
// }
// }
// for pid in heartbeat.read().await.subtasks.clone() {
// let _ = Command::new("kill").arg("-TERM").arg(&pid).status().await;
// }
// let _ = std::fs::remove_file(&socket);
}
async fn _handle_connection(stream: UnixStream) {
let peer_addr = match stream.peer_addr() {
Ok(addr) => format!("{:?}", addr),
Err(_) => "unknown".to_string(),
};
log::debug!("New connection from: {}", peer_addr);
let mut reader = BufReader::new(stream);
let mut line = String::new();
loop {
line.clear();
match reader.read_line(&mut line).await {
Ok(0) => {
log::debug!("Connection closed by peer: {}", peer_addr);
break;
}
Ok(_) => {
// 简单打印原始JSON(预留字段解析空间)
match serde_json::from_str::<Value>(&line) {
Ok(json) => {
// 只提取关键字段打印(字段可能变化)
let msgtype = json
.get("msgtype")
.and_then(|v| v.as_str())
.unwrap_or("unknown");
let name = json
.get("name")
.and_then(|v| v.as_str())
.unwrap_or("unnamed");
log::info!("[{}] Message from '{}': {}", msgtype, name, line.trim());
// TODO: 字段稳定后,可扩展为结构化打印
// match msgtype {
// "ResourceUsage" => print_resource(&json),
// "Log" => print_log(&json),
// _ => log::debug!("Unknown type: {}", msgtype),
// }
}
Err(e) => {
log::warn!(
"Invalid JSON from {}: {} - Error: {}",
peer_addr,
line.trim(),
e
);
}
}
}
Err(e) => {
log::error!("Failed to read from {}: {}", peer_addr, e);
break;
}
}
}
log::debug!("Connection handler for {} exiting", peer_addr);
}
+30
View File
@@ -0,0 +1,30 @@
use thiserror::Error;
use workshop_engine::HasExitCode;
#[derive(Error, Debug)]
pub enum CliError {
#[error("{0}")]
Prebake(#[from] crate::prebake::error::PrebakeError),
#[error("{0}")]
Bake(#[from] crate::bake::error::BakeError),
#[error("{0}")]
Finalize(#[from] crate::finalize::FinalizeError),
#[error("{0}")]
General(anyhow::Error),
}
impl From<anyhow::Error> for CliError {
fn from(e: anyhow::Error) -> Self {
CliError::General(e)
}
}
impl CliError {
pub fn exit_code(&self) -> i32 {
match self {
CliError::Prebake(e) => e.exit_code(),
CliError::Bake(e) => e.exit_code(),
CliError::Finalize(e) => e.exit_code(),
CliError::General(_) => 1,
}
}
}
+4 -28
View File
@@ -8,12 +8,10 @@ pub mod template;
pub mod types; pub mod types;
use crate::cli::Cli; use crate::cli::Cli;
use crate::finalize::constant::FINALIZE_STANDALONE_PLUGIN_PATH; use crate::types::resource::ResourceRegistry;
use crate::finalize::plugin::fetch::{self, FetchArgument};
pub use config::*; pub use config::*;
pub use error::FinalizeError; pub use error::FinalizeError;
use std::path::Path; use std::path::Path;
use std::path::PathBuf;
use workshop_engine::EventSender; use workshop_engine::EventSender;
pub fn parse(path: &Path) -> Result<FinalizeConfig, FinalizeError> { pub fn parse(path: &Path) -> Result<FinalizeConfig, FinalizeError> {
@@ -27,6 +25,7 @@ pub async fn finalize(
_cli: &Cli, _cli: &Cli,
ctx: &mut workshop_engine::ExecutionContext, ctx: &mut workshop_engine::ExecutionContext,
event_tx: EventSender, event_tx: EventSender,
registry: &ResourceRegistry,
) -> Result<(), FinalizeError> { ) -> Result<(), FinalizeError> {
let mut finalize = parse(finalize_path)?; let mut finalize = parse(finalize_path)?;
let validate_result = finalize.validate(); let validate_result = finalize.validate();
@@ -36,33 +35,10 @@ pub async fn finalize(
} }
return Err(FinalizeError::ValidateError(errors)); return Err(FinalizeError::ValidateError(errors));
}; };
if ctx.standalone {
// In standalone mode, finalize should download plugin by itself
log::debug!("Downloading {} plugins", finalize.plugin.len());
let plugin_path = PathBuf::from(FINALIZE_STANDALONE_PLUGIN_PATH);
// if plugin_path.exists() {
// std::fs::remove_dir_all(&plugin_path).map_err(FinalizeError::IoError)?;
// }
for (name, data) in finalize.plugin.iter_mut() {
log::info!("Downloading plugin {}", name);
let (source, config) = data.iter_mut().next().unwrap_or_else(|| {
panic!(
"Internal Error: Plugin {} has no source configured, this should have been caught by validation!",
name
)
});
let fetch_argument = FetchArgument {
checksum: config.checksum.clone(),
shallow: config.shallow,
};
fetch::fetch_plugin(source, name, &plugin_path, fetch_argument).await?;
config.runtime.fspath = Some(plugin_path.join(name));
}
}
// Register Plugins // Register Plugins (paths resolved from registry)
log::info!("Registering {} plugins", finalize.plugin.len()); log::info!("Registering {} plugins", finalize.plugin.len());
let registered_plugins = plugin::register(finalize.plugin, None)?; let registered_plugins = plugin::register(finalize.plugin, None, &mut finalize.notification, registry)?;
// Early hook // Early hook
let earlyhook_result = stage::hook::hook( let earlyhook_result = stage::hook::hook(
+28 -32
View File
@@ -1,56 +1,52 @@
# workshop-baker/src/finalize # workshop-baker/src/finalize
**Generated:** 2026-04-22 **Generated:** 2026-05-19
**Commit:** 7b3b71a **Commit:** 28abc5d
Artifact packaging, distribution, deployment, and notifications. Artifact packaging, plugin-based notifications, and post-build hooks.
## STRUCTURE ## STRUCTURE
``` ```
finalize/ finalize/
├── config.rs # FinalizeConfig YAML (artifacts, deploy, notify) ├── config.rs # FinalizeConfig YAML (379 lines, plugins, notifications)
├── stage.rs # FinalizeStage enum ├── stage.rs # FinalizeStage enum
├── stage/hook.rs # Post-build hooks ├── stage/hook.rs # Post-build hook execution
├── event.rs # Event propagation ├── event.rs # Event propagation
├── template.rs # Variable substitution {{VAR}} ├── template.rs # Variable substitution {{VARIABLE}}
├── plugin.rs # Plugin trait + registry ├── plugin.rs # Plugin trait + registration
├── plugin/ ├── plugin/
│ ├── metadata.rs # Plugin manifest parsing │ ├── metadata.rs # Plugin manifest parsing
│ ├── dylib.rs # Dynamic library plugins │ ├── dylib.rs # Dynamic library plugins
│ ├── rhai.rs # Rhai scripting (todo!()) │ ├── rhai.rs # Rhai scripting (todo!())
│ ├── shell.rs # Shell command plugins │ ├── shell.rs # Shell command plugins
│ └── internal/ # Built-in plugins │ └── internal.rs + internal/ # Built-in plugins (mail, webhook, satori, insitenotify)
│ ├── mail/ # Email notifications (SMTP) ├── types.rs
│ ├── webhook.rs # HTTP webhooks ├── types/compression.rs
├── satori.rs # Satori integration ├── error.rs # FinalizeError
│ └── insitenotify.rs # In-site notifications └── constant.rs
│ └── fetch/ # Artifact fetching
│ ├── git.rs # Git repository fetch
│ ├── http.rs # HTTP download
│ ├── extract.rs # Archive extraction
│ └── checksum.rs# SHA256 verification (SHA1 deprecated)
``` ```
## WHERE TO LOOK ## WHERE TO LOOK
| Task | Location | | Task | Location |
|------|----------| |------|----------|
| Config loading | `config.rs` - FinalizeConfig | | Config loading | `config.rs:17` - `FinalizeConfig` + `parse()` |
| Email notify | `plugin/internal/mail/` - SMTP templates | | Plugin registration | `plugin.rs` - `register()` |
| Email notifications | `plugin/internal/mail.rs` - SMTP |
| Webhooks | `plugin/internal/webhook.rs` | | Webhooks | `plugin/internal/webhook.rs` |
| Artifact fetch | `plugin/fetch/` - git/http/extract | | Satori integration | `plugin/internal/satori.rs` |
| Checksums | `plugin/fetch/checksum.rs:13` - SHA1 deprecated | | Shell plugins | `plugin/shell.rs` |
| Template rendering | `template.rs` - `{{VARIABLE}}` handlebars-style |
## CONVENTIONS ## CONVENTIONS
- **Plugin System**: `FinalizePlugin` trait + registry with both internal and dynamic plugins
- **Plugin System**: Dynamic + internal plugins via Plugin trait - **Template Syntax**: `{{VARIABLE}}` handlebars-style substitution via `template.rs`
- **Template Syntax**: `{{VARIABLE}}` handlebars-style - **Stages**: EarlyHook → (Artifacts) → LateHook → Notifications
- **Stages**: Hook → Fetch → Build → Publish → Notify → Cleanup - **Notification**: `notify.rs` in parent module drives plugin-based notification dispatch
- **Fetch**: Supports git, http, with extraction and checksum verify - **ResourceRegistry**: Shared resource cache passed through to plugins
## ANTI-PATTERNS ## ANTI-PATTERNS
1. **fetch/ subdirectory removed** — was `plugin/fetch/{git,http,extract,checksum}`, no longer exists
1. **SHA1 deprecated**`checksum.rs:13` warns SHA1 is deprecated 2. **SHA1 deprecated**workshop-getterurl handles checksums now
2. **Unimplemented**`rhai.rs:17` - "todo!()" for Rhai scripting 3. **Unimplemented**`rhai.rs:17` - "todo!()" for Rhai scripting
3. **TODO markers** — Multiple TODOs in config.rs (lines 305-306) 4. **TODO markers** — Multiple TODOs in config.rs (lines 305-306)
5. **Artifact stage not implemented**`finalize.rs:57` — "TODO: artifact stage implementation"
+115 -12
View File
@@ -115,6 +115,17 @@ pub struct NotificationMethod {
pub policy: Vec<NotifyPolicy>, pub policy: Vec<NotifyPolicy>,
} }
impl Default for NotificationMethod {
fn default() -> Self {
Self {
template: NotificationTemplate::default(),
trigger: Vec::new(),
config: Value::default(),
policy: Vec::new(),
}
}
}
#[derive(Debug, Deserialize, Serialize, Clone, Default)] #[derive(Debug, Deserialize, Serialize, Clone, Default)]
pub struct NotificationTemplate { pub struct NotificationTemplate {
/// Payload schema: "default" (plugin-defined) or "custom" (requires on_* templates) /// Payload schema: "default" (plugin-defined) or "custom" (requires on_* templates)
@@ -134,17 +145,59 @@ pub struct NotificationTemplate {
pub on_finish_of: Option<OnFinishOf>, pub on_finish_of: Option<OnFinishOf>,
} }
impl std::ops::AddAssign for NotificationTemplate {
fn add_assign(&mut self, other: Self) {
if self.schema.is_none() {
self.schema = other.schema;
}
if self.on_success.is_none() {
self.on_success = other.on_success;
}
if self.on_failure.is_none() {
self.on_failure = other.on_failure;
}
if self.on_finish_of.is_none() {
self.on_finish_of = other.on_finish_of;
}
}
}
/// Notification configuration container /// Notification configuration container
#[derive(Debug, Deserialize, Serialize, Clone, Default)] #[derive(Debug, Serialize, Clone, Default)]
pub struct NotificationConfig { pub struct NotificationConfig {
/// Template definitions for notification content /// Template definitions for notification content
#[serde(default)] #[serde(default)]
pub templates: HashMap<String, NotificationTemplateDef>, pub templates: HashMap<String, NotificationTemplateDef>,
#[serde(flatten)] /// Priority groups (YAML keys like `0`, `1` are parsed as u32).
/// Custom Deserialize handles serde_yaml string→u32 conversion.
#[serde(default)]
pub groups: HashMap<u32, HashMap<String, NotificationMethod>>, pub groups: HashMap<u32, HashMap<String, NotificationMethod>>,
} }
/// Serde helper: YAML map keys are always strings, convert to u32.
impl<'de> serde::Deserialize<'de> for NotificationConfig {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where D: serde::Deserializer<'de>
{
#[derive(serde::Deserialize)]
struct Helper {
#[serde(default)]
templates: HashMap<String, NotificationTemplateDef>,
#[serde(flatten)]
groups_raw: HashMap<String, HashMap<String, NotificationMethod>>,
}
let h = Helper::deserialize(deserializer)?;
let groups: HashMap<u32, _> = h.groups_raw.into_iter()
.map(|(k, v)| {
k.parse::<u32>().map(|n| (n, v))
.map_err(|_| serde::de::Error::custom(format!("priority key must be integer, got {:?}", k)))
})
.collect::<Result<_, _>>()?;
Ok(NotificationConfig { templates: h.templates, groups })
}
}
/// Stage-triggered notification configuration /// Stage-triggered notification configuration
#[derive(Debug, Deserialize, Serialize, Clone)] #[derive(Debug, Deserialize, Serialize, Clone)]
pub struct OnFinishOf { pub struct OnFinishOf {
@@ -158,7 +211,7 @@ pub struct OnFinishOf {
/// Notification template definition. /// Notification template definition.
/// Supports inline content (on_success/on_failure/on_finish_of) or external reference (use). /// Supports inline content (on_success/on_failure/on_finish_of) or external reference (use).
#[derive(Debug, Deserialize, Serialize, Clone)] #[derive(Debug, Deserialize, Serialize, Clone, Default)]
pub struct NotificationTemplateDef { pub struct NotificationTemplateDef {
/// External template reference (git URL). /// External template reference (git URL).
/// Cannot be combined with on_* fields in MVP (validation will reject). /// Cannot be combined with on_* fields in MVP (validation will reject).
@@ -178,6 +231,53 @@ pub struct NotificationTemplateDef {
pub on_finish_of: Option<String>, pub on_finish_of: Option<String>,
} }
impl NotificationTemplateDef {
/// Merge `other` into `self`, keeping self's values when present.
///
/// `other` fills in gaps where `self` is `None`.
/// Chaining: `a.apply(b).apply(c)` = a wins, then b fills gaps, then c fills gaps.
pub fn apply(mut self, other: Self) -> Self {
if self.on_success.is_none() {
self.on_success = other.on_success;
}
if self.on_failure.is_none() {
self.on_failure = other.on_failure;
}
if self.on_finish_of.is_none() {
self.on_finish_of = other.on_finish_of;
}
self
}
}
/// Builder for [`NotificationTemplateDef`].
pub struct NotificationTemplateDefBuilder(NotificationTemplateDef);
impl NotificationTemplateDefBuilder {
pub fn new() -> Self {
Self(NotificationTemplateDef::default())
}
pub fn with_use(mut self, val: impl Into<String>) -> Self {
self.0.use_ = Some(val.into());
self
}
pub fn with_on_success(mut self, val: impl Into<String>) -> Self {
self.0.on_success = Some(val.into());
self
}
pub fn with_on_failure(mut self, val: impl Into<String>) -> Self {
self.0.on_failure = Some(val.into());
self
}
pub fn with_on_finish_of(mut self, val: impl Into<String>) -> Self {
self.0.on_finish_of = Some(val.into());
self
}
pub fn build(self) -> NotificationTemplateDef {
self.0
}
}
/// Notification trigger type. /// Notification trigger type.
#[derive(Debug, Clone, PartialEq, Eq)] #[derive(Debug, Clone, PartialEq, Eq)]
pub enum Trigger { pub enum Trigger {
@@ -336,15 +436,18 @@ impl FinalizeConfig {
fn validate_notification(&self, errors: &mut Vec<String>) { fn validate_notification(&self, errors: &mut Vec<String>) {
// Validate templates // Validate templates
for (name, tmpl) in &self.notification.templates { for (name, tmpl) in &self.notification.templates {
if tmpl.use_.is_some() if let Some(ref u) = tmpl.use_ {
&& (tmpl.on_success.is_some() if u.contains("://") && (
|| tmpl.on_failure.is_some() tmpl.on_success.is_some() ||
|| tmpl.on_finish_of.is_some()) tmpl.on_failure.is_some() ||
{ tmpl.on_finish_of.is_some()
errors.push(format!( ) {
"Template '{}' cannot combine 'use' with 'on_*' fields (MVP restriction)", errors.push(format!(
name "Template '{}' cannot combine external 'use' with 'on_*' fields",
)); name
));
}
// Local references (no ://) are allowed to have on_* overrides
} }
} }
} }
+24 -4
View File
@@ -1,14 +1,15 @@
use crate::finalize::RawPluginMap; use crate::finalize::RawPluginMap;
use crate::finalize::constant::{FINALIZE_PLUGIN_MANIFEST, FINALIZE_PLUGIN_MANIFEST_EXTENSION}; use crate::finalize::constant::{FINALIZE_PLUGIN_MANIFEST, FINALIZE_PLUGIN_MANIFEST_EXTENSION};
use crate::finalize::plugin::metadata::{PluginMetadata, PluginType}; use crate::finalize::plugin::metadata::{PluginMetadata, PluginType};
use crate::{EventSender, ExecutionContext, finalize::error::PluginError}; use workshop_engine::{EventSender, ExecutionContext};
use crate::{ finalize::error::PluginError};
use crate::types::resource::ResourceRegistry;
use async_trait::async_trait; use async_trait::async_trait;
use std::collections::{HashMap, HashSet}; use std::collections::{HashMap, HashSet};
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use std::sync::Arc; use std::sync::Arc;
mod dylib; mod dylib;
pub mod fetch;
mod internal; mod internal;
mod metadata; mod metadata;
mod rhai; mod rhai;
@@ -214,6 +215,8 @@ fn get_entry(plugin_type: PluginType, name: &str) -> Result<Box<dyn AsyncPluginF
pub fn register( pub fn register(
external_plugins: RawPluginMap, external_plugins: RawPluginMap,
list: Option<&HashSet<String>>, list: Option<&HashSet<String>>,
notification_config: &mut crate::finalize::NotificationConfig,
registry: &ResourceRegistry,
) -> Result<PluginMap, PluginError> { ) -> Result<PluginMap, PluginError> {
let mut plugins: PluginMap = HashMap::new(); let mut plugins: PluginMap = HashMap::new();
let mut count = 0; let mut count = 0;
@@ -255,7 +258,14 @@ pub fn register(
log::warn!("Plugin {} is not available: {}", name, reason); log::warn!("Plugin {} is not available: {}", name, reason);
continue; continue;
} }
let metadata = read_manifest(config.runtime.fspath, &name)?; // Resolve plugin path from registry instead of config.runtime.fspath
let plugin_path = registry.resolve(&name).ok_or_else(|| {
PluginError::PluginUnavailable {
name: name.clone(),
reason: "plugin not found in resource registry".to_string(),
}
})?;
let metadata = read_manifest(Some(plugin_path), &name)?;
let plugin_type = get_plugin_type(metadata.plugin_type.clone(), &metadata.entrypoint); let plugin_type = get_plugin_type(metadata.plugin_type.clone(), &metadata.entrypoint);
let entry = get_entry(plugin_type, &name)?; let entry = get_entry(plugin_type, &name)?;
plugins.insert( plugins.insert(
@@ -267,8 +277,18 @@ pub fn register(
count += 1; count += 1;
} }
log::info!("Registered {} plugins", count); // Inject plugin-declared templates as .fallback
for (name, plugin) in &plugins {
if let Some(ref templates) = plugin.metadata.templates {
let key = format!("{}.fallback", name);
for tmpl in templates {
notification_config.templates.insert(key.clone(), tmpl.clone());
}
log::debug!("Registered templates from plugin '{}' as '{}'", name, key);
}
}
log::info!("Registered {} plugins", count);
Ok(plugins) Ok(plugins)
} }
+2 -1
View File
@@ -1,6 +1,7 @@
use crate::finalize::plugin::PluginError; use crate::finalize::plugin::PluginError;
use crate::finalize::plugin::PluginMetadata; use crate::finalize::plugin::PluginMetadata;
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn}; use workshop_engine::{EventSender, ExecutionContext};
use crate::{ finalize::plugin::AsyncPluginFn};
use async_trait::async_trait; use async_trait::async_trait;
pub struct Dylib; pub struct Dylib;
-109
View File
@@ -1,109 +0,0 @@
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
use crate::finalize::fetch;
use crate::finalize::plugin::{PluginError, RawPluginMap};
use std::collections::HashSet;
use std::path::Path;
mod checksum;
mod extract;
mod git;
mod http;
mod types;
pub use types::FetchArgument;
enum URLScheme {
Unknown,
Git,
Http,
Https,
File,
}
impl URLScheme {
fn parse(url: &str) -> (Self, String) {
match url.split_once("://") {
Some((prefix, rest)) => {
let scheme = match prefix {
"git" => URLScheme::Git,
"https" => URLScheme::Https,
"http" => URLScheme::Http,
"file" => URLScheme::File,
_ => URLScheme::Unknown,
};
(scheme, rest.to_string())
}
None => (URLScheme::Unknown, url.to_string()),
}
}
}
pub async fn fetch_plugin(
url: &str,
name: &str,
basedir: &Path,
argument: FetchArgument,
) -> Result<(), PluginError> {
let (scheme, path) = URLScheme::parse(url);
match scheme {
URLScheme::Git => {
// NOTE: as a practical compromise, we use git command for git clone
// gix will be supported later, with ONLY https scheme
// TODO: Implement gix
log::info!("Fetching git plugin: {}", url);
git::fetch_git_plugin(&path, name, basedir, argument.shallow).await
}
URLScheme::Http | URLScheme::Https => {
log::info!("Fetching http plugin: {}", url);
let checksum = argument.get_checksum();
http::fetch_http_plugin(url, name, basedir, checksum).await
}
URLScheme::File => {
// TODO: Support client mode
// NOTE: In standalone mode, we just assume path is well-defined and utilize
log::info!("Fetching file plugin: {}", url);
Ok(())
}
URLScheme::Unknown => {
log::error!("Unknown URL scheme in plugin URL: {}", url);
Err(PluginError::GeneralError(format!(
"Unknown URL scheme in plugin URL: {}",
url
)))
}
}
}
pub async fn fetch_plugins(
plugin_path: &Path,
plugin_config: &mut RawPluginMap,
list: Option<&HashSet<String>>,
) -> Result<(), PluginError> {
log::info!("Preparing to download {} plugins", plugin_config.len());
for (name, data) in plugin_config.iter_mut() {
match list {
Some(list) if !list.contains(name) => {
log::debug!("Plugin {} not in list, skipping", name);
continue;
}
_ => {}
}
log::debug!("Downloading plugin {}", name);
let (source, config) =
data.iter_mut()
.next()
.ok_or_else(|| PluginError::PluginUnavailable {
name: name.clone(),
reason: "no source configured".to_string(),
})?;
let fetch_argument = FetchArgument {
checksum: config.checksum.clone(),
shallow: config.shallow,
};
fetch::fetch_plugin(source, name, plugin_path, fetch_argument).await?;
log::debug!("Storing plugin {} to {}", name, plugin_path.display());
config.runtime.fspath = Some(plugin_path.join(name));
}
Ok(())
}
@@ -1,137 +0,0 @@
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
use super::types::ChecksumType;
use sha2::{Digest, Sha256, Sha512};
/// Detects the checksum type based on the length of the checksum string.
///
/// Returns `Ok(None)` for empty string (checksum disabled).
/// Returns `Ok(Some(ChecksumType))` for valid SHA256 (64 chars) or SHA512 (128 chars).
/// Returns `Err` for invalid lengths (32=MD5, 40=SHA1) or unsupported lengths.
pub fn detect_checksum_type(checksum: &str) -> Result<Option<ChecksumType>, String> {
match checksum.len() {
0 => Ok(None),
32 => Err(format!(
"MD5 (first 8 chars: {}) is not supported for security reasons. Use SHA256 (64 chars) or SHA512 (128 chars)",
&checksum[..8.min(checksum.len())]
)),
40 => Err(format!(
"SHA1 (first 8 chars: {}) is deprecated and not supported. Use SHA256 (64 chars) or SHA512 (128 chars)",
&checksum[..8.min(checksum.len())]
)),
64 => Ok(Some(ChecksumType::Sha256)),
128 => Ok(Some(ChecksumType::Sha512)),
_ => Err(format!(
"Invalid checksum length: {}. Expected 0 (disabled), 64 (SHA256), or 128 (SHA512). First 8 chars: {}",
checksum.len(),
&checksum[..8.min(checksum.len())]
)),
}
}
/// Verifies that the data matches the expected checksum.
///
/// Compares the computed hash of `data` against `expected` using the specified `checksum_type`.
/// Returns `Ok(())` if the checksums match, `Err` otherwise.
pub fn verify_checksum(
data: &[u8],
expected: &str,
checksum_type: ChecksumType,
) -> Result<(), String> {
let actual = match checksum_type {
ChecksumType::Sha256 => {
let mut hasher = Sha256::new();
hasher.update(data);
format!("{:x}", hasher.finalize())
}
ChecksumType::Sha512 => {
let mut hasher = Sha512::new();
hasher.update(data);
format!("{:x}", hasher.finalize())
}
};
if actual.eq_ignore_ascii_case(expected) {
Ok(())
} else {
Err(format!(
"Checksum mismatch: expected {}..., got {}...",
&expected[..8.min(expected.len())],
&actual[..8.min(actual.len())]
))
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_detect_checksum_type_empty() {
assert_eq!(detect_checksum_type("").unwrap(), None);
}
#[test]
fn test_detect_checksum_type_md5_rejected() {
let result = detect_checksum_type("a".repeat(32).as_str());
assert!(result.is_err());
assert!(result.unwrap_err().contains("MD5"));
}
#[test]
fn test_detect_checksum_type_sha1_rejected() {
let result = detect_checksum_type("a".repeat(40).as_str());
assert!(result.is_err());
assert!(result.unwrap_err().contains("SHA1"));
}
#[test]
fn test_detect_checksum_type_sha256() {
assert_eq!(
detect_checksum_type("a".repeat(64).as_str()).unwrap(),
Some(ChecksumType::Sha256)
);
}
#[test]
fn test_detect_checksum_type_sha512() {
assert_eq!(
detect_checksum_type("a".repeat(128).as_str()).unwrap(),
Some(ChecksumType::Sha512)
);
}
#[test]
fn test_detect_checksum_type_invalid_length() {
let result = detect_checksum_type("a".repeat(50).as_str());
assert!(result.is_err());
assert!(result.unwrap_err().contains("Invalid checksum length"));
}
#[test]
fn test_verify_checksum_sha256_valid() {
let data = b"hello";
let expected = "2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824";
assert!(verify_checksum(data, expected, ChecksumType::Sha256).is_ok());
}
#[test]
fn test_verify_checksum_sha256_invalid() {
let data = b"hello";
let expected = "0000000000000000000000000000000000000000000000000000000000000000";
assert!(verify_checksum(data, expected, ChecksumType::Sha256).is_err());
}
#[test]
fn test_verify_checksum_sha512_valid() {
let data = b"hello";
let expected = "9b71d224bd62f3785d96d46ad3ea3d73319bfbc2890caadae2dff72519673ca72323c3d99ba5c11d7c7acc6e14b8c5da0c4663475c2e5c3adef46f73bcdec043";
assert!(verify_checksum(data, expected, ChecksumType::Sha512).is_ok());
}
#[test]
fn test_verify_checksum_sha512_invalid() {
let data = b"hello";
let expected = "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000";
assert!(verify_checksum(data, expected, ChecksumType::Sha512).is_err());
}
}
@@ -1,169 +0,0 @@
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
use crate::finalize::plugin::PluginError;
use std::path::{Path, PathBuf};
use tokio::task;
/// Compression type for tar archives.
#[derive(Debug, Clone, Copy, PartialEq)]
pub enum CompressionType {
/// Gzip compressed (.tar.gz, .tgz)
Gzip,
/// Zstd compressed (.tar.zst, .tar.zstd, .tzst)
Zstd,
/// Uncompressed tar (.tar)
None,
}
impl CompressionType {
/// Detects compression type from file path extension.
///
/// Supports: .tar.gz, .tgz (Gzip), .tar.zst, .tar.zstd, .tzst (Zstd), .tar (None).
pub fn from_path(path: &std::path::Path) -> Result<Self, String> {
let name = path
.file_name()
.and_then(|n| n.to_str())
.unwrap_or("")
.to_lowercase();
if name.ends_with(".tar.gz") || name.ends_with(".tgz") {
Ok(CompressionType::Gzip)
} else if name.ends_with(".tar.zst")
|| name.ends_with(".tar.zstd")
|| name.ends_with(".tzst")
{
Ok(CompressionType::Zstd)
} else if name.ends_with(".tar") {
Ok(CompressionType::None)
} else {
Err(format!(
"Unsupported archive format: '{}'. Supported formats: .tar, .tar.gz, .tgz, .tar.zst, .tar.zstd, .tzst",
name
))
}
}
}
/// Extracts a tar archive to the destination directory.
///
/// Removes existing destination directory if present, creates a fresh directory,
/// then unpacks the archive using the appropriate decompression based on `compression`.
pub async fn extract_tar(
archive: &Path,
destdir: &PathBuf,
compression: CompressionType,
) -> Result<(), PluginError> {
if destdir.exists() {
tokio::fs::remove_dir_all(destdir).await.map_err(|e| {
PluginError::GeneralError(format!("Failed to remove existing directory: {}", e))
})?;
}
tokio::fs::create_dir_all(destdir)
.await
.map_err(|e| PluginError::GeneralError(format!("Failed to create directory: {}", e)))?;
let archive = archive.to_path_buf();
let destdir = destdir.clone();
task::spawn_blocking(move || {
let file = std::fs::File::open(&archive)
.map_err(|e| PluginError::GeneralError(format!("Failed to open archive: {}", e)))?;
match compression {
CompressionType::Gzip => {
let decoder = flate2::read::GzDecoder::new(file);
let mut archive = tar::Archive::new(decoder);
archive.unpack(&destdir).map_err(|e| {
PluginError::GeneralError(format!("Failed to unpack archive: {}", e))
})?;
}
CompressionType::Zstd => {
let decoder = zstd::stream::read::Decoder::new(file).map_err(|e| {
PluginError::GeneralError(format!("Failed to create zstd decoder: {}", e))
})?;
let mut archive = tar::Archive::new(decoder);
archive.unpack(&destdir).map_err(|e| {
PluginError::GeneralError(format!("Failed to unpack archive: {}", e))
})?;
}
CompressionType::None => {
let mut archive = tar::Archive::new(file);
archive.unpack(&destdir).map_err(|e| {
PluginError::GeneralError(format!("Failed to unpack archive: {}", e))
})?;
}
};
Ok::<(), PluginError>(())
})
.await
.map_err(|e| PluginError::GeneralError(format!("Task join error: {}", e)))??;
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use std::path::Path;
#[test]
fn test_from_path_tar_gz() {
assert_eq!(
CompressionType::from_path(Path::new("archive.tar.gz")).unwrap(),
CompressionType::Gzip
);
}
#[test]
fn test_from_path_tgz() {
assert_eq!(
CompressionType::from_path(Path::new("archive.tgz")).unwrap(),
CompressionType::Gzip
);
}
#[test]
fn test_from_path_tar_zst() {
assert_eq!(
CompressionType::from_path(Path::new("archive.tar.zst")).unwrap(),
CompressionType::Zstd
);
}
#[test]
fn test_from_path_tar_zstd() {
assert_eq!(
CompressionType::from_path(Path::new("archive.tar.zstd")).unwrap(),
CompressionType::Zstd
);
}
#[test]
fn test_from_path_tzst() {
assert_eq!(
CompressionType::from_path(Path::new("archive.tzst")).unwrap(),
CompressionType::Zstd
);
}
#[test]
fn test_from_path_tar() {
assert_eq!(
CompressionType::from_path(Path::new("archive.tar")).unwrap(),
CompressionType::None
);
}
#[test]
fn test_from_path_unsupported() {
assert!(CompressionType::from_path(Path::new("archive.zip")).is_err());
}
#[test]
fn test_from_path_case_insensitive() {
assert_eq!(
CompressionType::from_path(Path::new("archive.TAR.GZ")).unwrap(),
CompressionType::Gzip
);
}
}
@@ -1,185 +0,0 @@
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
use crate::finalize::plugin::PluginError;
use std::path::Path;
use tokio::process::Command;
/// Version specification for a git repository.
#[derive(Debug, PartialEq)]
pub enum GitVersion {
/// No version specified (invalid for fetch operations)
None,
/// Single version string (tag or commit hash)
Single(String),
/// Mixed: try tag first, fallback to commit hash
Mixed(String, String),
}
/// Parses a git URL with optional version specification.
///
/// Format: `url` (no version), `url@tag` (single version), `url@tag:commit` (mixed version).
/// Returns (base_url, GitVersion).
pub fn parse_git_url(url: &str) -> (String, GitVersion) {
if let Some((base, version)) = url.rsplit_once('@') {
if version.contains(":") {
let (tag, commit) = version.split_once(':').unwrap();
return (
base.to_string(),
GitVersion::Mixed(tag.to_string(), commit.to_string()),
);
}
return (base.to_string(), GitVersion::Single(version.to_string()));
}
(url.to_string(), GitVersion::None)
}
/// Clones a git repository and checks out the specified version.
///
/// Removes existing destination directory, performs shallow or full clone,
/// then checks out the given version (tag or commit).
pub async fn git_clone_and_checkout(
baseurl: &str,
version: &str,
dest: &Path,
name: &str,
shallow: bool,
) -> Result<(), PluginError> {
if dest.exists() {
tokio::fs::remove_dir_all(dest)
.await
.map_err(|e| PluginError::FetchError {
name: name.to_string(),
url: baseurl.to_string(),
reason: format!("Failed to remove existing directory: {}", e),
})?;
}
let mut cmd = Command::new("git");
cmd.arg("clone");
if shallow {
cmd.arg("--depth").arg("1");
}
cmd.arg(baseurl).arg(dest);
let output = cmd.output().await.map_err(|e| PluginError::FetchError {
name: name.to_string(),
url: baseurl.to_string(),
reason: e.to_string(),
})?;
if !output.status.success() {
return Err(PluginError::FetchError {
name: name.to_string(),
url: baseurl.to_string(),
reason: String::from_utf8_lossy(&output.stderr).to_string(),
});
}
let mut cmd = Command::new("git");
cmd.arg("checkout").arg(version).current_dir(dest);
let output = cmd.output().await.map_err(|e| PluginError::FetchError {
name: name.to_string(),
url: baseurl.to_string(),
reason: e.to_string(),
})?;
if !output.status.success() {
return Err(PluginError::FetchError {
name: name.to_string(),
url: baseurl.to_string(),
reason: String::from_utf8_lossy(&output.stderr).to_string(),
});
}
Ok(())
}
/// Fetches a plugin from a git repository.
///
/// Parses the URL for version info, then clones and checks out the appropriate ref.
/// For Mixed version, tries the tag first and falls back to the commit hash.
pub async fn fetch_git_plugin(
path: &str,
name: &str,
basedir: &Path,
shallow: Option<bool>,
) -> Result<(), PluginError> {
let (baseurl, version) = parse_git_url(path);
match version {
GitVersion::None => {
log::error!("Version should be specified for git plugin URL: {}", path);
Err(PluginError::FetchError {
name: name.to_string(),
url: path.to_string(),
reason: "Version should be specified as either a tag or a commit hash.".to_string(),
})
}
GitVersion::Single(version) => {
log::debug!("Fetching git plugin: {}@{}", name, version);
let shallow_flag = shallow.unwrap_or(false);
git_clone_and_checkout(
&baseurl,
&version.to_string(),
&basedir.join(name),
name,
shallow_flag,
)
.await
}
GitVersion::Mixed(tag, commit) => {
log::debug!("Fetching git plugin: {}@{}/{}", name, tag, commit);
let shallow_flag = shallow.unwrap_or(false);
let result =
git_clone_and_checkout(&baseurl, &tag, &basedir.join(name), name, shallow_flag)
.await;
if result.is_err() {
log::warn!(
"Failed to clone with tag, trying commit hash. URL: {}",
path
);
return git_clone_and_checkout(&baseurl, &commit, &basedir.join(name), name, false)
.await;
}
result
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_parse_git_url_no_version() {
let (url, version) = parse_git_url("https://github.com/user/repo");
assert_eq!(url, "https://github.com/user/repo");
assert_eq!(version, GitVersion::None);
}
#[test]
fn test_parse_git_url_single_tag() {
let (url, version) = parse_git_url("https://github.com/user/repo@v1.0.0");
assert_eq!(url, "https://github.com/user/repo");
assert_eq!(version, GitVersion::Single("v1.0.0".to_string()));
}
#[test]
fn test_parse_git_url_mixed_version() {
let (url, version) = parse_git_url("https://github.com/user/repo@main:abc123");
assert_eq!(url, "https://github.com/user/repo");
assert_eq!(
version,
GitVersion::Mixed("main".to_string(), "abc123".to_string())
);
}
#[test]
fn test_parse_git_url_complex_url() {
let (url, version) = parse_git_url("git@github.com:user/repo.git@tag:commit");
assert_eq!(url, "git@github.com:user/repo.git");
assert_eq!(
version,
GitVersion::Mixed("tag".to_string(), "commit".to_string())
);
}
#[test]
fn test_parse_git_url_single_with_at_in_path() {
let (url, version) = parse_git_url("https://github.com/user/repo@my-repo@v1.0");
assert_eq!(url, "https://github.com/user/repo@my-repo");
assert_eq!(version, GitVersion::Single("v1.0".to_string()));
}
}
@@ -1,110 +0,0 @@
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
use super::checksum::{detect_checksum_type, verify_checksum};
use super::extract::{CompressionType, extract_tar};
use crate::finalize::plugin::PluginError;
use std::path::Path;
pub async fn download_to_file(url: &str, dest: &Path, name: &str) -> Result<(), PluginError> {
let client = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(300))
.build()
.map_err(|e| PluginError::GeneralError(format!("Failed to build HTTP client: {}", e)))?;
let response = client
.get(url)
.send()
.await
.map_err(|e| PluginError::FetchError {
name: name.to_string(),
url: url.to_string(),
reason: format!("HTTP request failed: {}", e),
})?;
let status = response.status();
if !status.is_success() {
return Err(PluginError::FetchError {
name: name.to_string(),
url: url.to_string(),
reason: format!("HTTP error: {}", status),
});
}
let bytes = response
.bytes()
.await
.map_err(|e| PluginError::FetchError {
name: name.to_string(),
url: url.to_string(),
reason: format!("Failed to read response body: {}", e),
})?;
tokio::fs::write(dest, &bytes)
.await
.map_err(|e| PluginError::FetchError {
name: name.to_string(),
url: url.to_string(),
reason: format!("Failed to write file: {}", e),
})?;
Ok(())
}
fn extract_archive_extension(url: &str) -> &str {
let path_part = url
.split('?')
.next()
.unwrap_or(url)
.split('#')
.next()
.unwrap_or(url);
let file_part = path_part.rsplit('/').next().unwrap_or("");
let lower = file_part.to_lowercase();
if lower.ends_with(".tar.gz") || lower.ends_with(".tgz") {
".tar.gz"
} else if lower.ends_with(".tar.zst")
|| lower.ends_with(".tar.zstd")
|| lower.ends_with(".tzst")
{
".tar.zst"
} else if lower.ends_with(".tar") {
".tar"
} else {
".unknown"
}
}
pub async fn fetch_http_plugin(
url: &str,
name: &str,
basedir: &Path,
checksum_str: Option<&str>,
) -> Result<(), PluginError> {
let temp_dir = std::env::temp_dir().join("workshop-plugins");
tokio::fs::create_dir_all(&temp_dir).await.ok();
let ext = extract_archive_extension(url);
let temp_file = temp_dir.join(format!("{}-{}{}", name, uuid::Uuid::new_v4(), ext));
download_to_file(url, &temp_file, name).await?;
if let Some(checksum) = checksum_str
&& !checksum.is_empty()
{
let bytes = tokio::fs::read(&temp_file).await.map_err(|e| {
PluginError::GeneralError(format!("Failed to read downloaded file: {}", e))
})?;
let checksum_type = detect_checksum_type(checksum).map_err(PluginError::GeneralError)?;
if let Some(ct) = checksum_type {
verify_checksum(&bytes, checksum, ct).map_err(PluginError::GeneralError)?;
}
}
let compression = CompressionType::from_path(&temp_file).map_err(PluginError::GeneralError)?;
let destdir = basedir.join(name);
extract_tar(&temp_file, &destdir, compression).await?;
tokio::fs::remove_file(&temp_file).await.ok();
Ok(())
}
@@ -1,84 +0,0 @@
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
use serde::{Deserialize, Serialize};
/// Supported checksum types for verifying downloaded artifacts.
#[derive(Debug, Clone, PartialEq)]
pub enum ChecksumType {
/// SHA-256 hash (64 hex characters)
Sha256,
/// SHA-512 hash (128 hex characters)
Sha512,
}
/// Arguments for fetching a plugin artifact.
///
/// Contains optional checksum verification and shallow clone settings.
#[derive(Debug, Deserialize, Serialize, Clone, Default)]
pub struct FetchArgument {
/// SHA256 or SHA512 hex string for integrity verification.
#[serde(default)]
pub checksum: Option<String>,
/// Perform a shallow clone (single revision only).
#[serde(default)]
pub shallow: Option<bool>,
}
impl FetchArgument {
/// Returns the checksum if present and non-empty.
pub fn get_checksum(&self) -> Option<&str> {
self.checksum.as_deref().filter(|s| !s.is_empty())
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_get_checksum_some_non_empty() {
let arg = FetchArgument {
checksum: Some("abc123".to_string()),
shallow: None,
};
assert_eq!(arg.get_checksum(), Some("abc123"));
}
#[test]
fn test_get_checksum_empty_string() {
let arg = FetchArgument {
checksum: Some("".to_string()),
shallow: None,
};
assert_eq!(arg.get_checksum(), None);
}
#[test]
fn test_get_checksum_none() {
let arg = FetchArgument {
checksum: None,
shallow: None,
};
assert_eq!(arg.get_checksum(), None);
}
#[test]
fn test_serde_roundtrip() {
let arg = FetchArgument {
checksum: Some("deadbeef".to_string()),
shallow: Some(true),
};
let json = serde_json::to_string(&arg).unwrap();
let parsed: FetchArgument = serde_json::from_str(&json).unwrap();
assert_eq!(parsed.checksum, arg.checksum);
assert_eq!(parsed.shallow, arg.shallow);
}
#[test]
fn test_serde_roundtrip_empty() {
let arg = FetchArgument::default();
let json = serde_json::to_string(&arg).unwrap();
let parsed: FetchArgument = serde_json::from_str(&json).unwrap();
assert_eq!(parsed.checksum, arg.checksum);
assert_eq!(parsed.shallow, arg.shallow);
}
}
@@ -29,3 +29,10 @@ pub fn get_internal_plugin() -> Vec<InternalPlugin> {
}, },
] ]
} }
pub fn get_dummy_plugin() -> Vec<InternalPlugin> {
vec![InternalPlugin {
name: "dummy",
entry: Box::new(dummy::Dummy),
}]
}
@@ -1,12 +1,13 @@
use crate::finalize::plugin::PluginError; use crate::finalize::plugin::PluginError;
use crate::finalize::plugin::PluginMetadata; use crate::finalize::plugin::PluginMetadata;
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn}; use workshop_engine::{EventSender, ExecutionContext};
use crate::{ finalize::plugin::AsyncPluginFn};
use async_trait::async_trait; use async_trait::async_trait;
pub struct _Dummy; pub struct Dummy;
#[async_trait] #[async_trait]
impl AsyncPluginFn for _Dummy { impl AsyncPluginFn for Dummy {
async fn call( async fn call(
&self, &self,
_metadata: &PluginMetadata, _metadata: &PluginMetadata,
@@ -14,6 +15,7 @@ impl AsyncPluginFn for _Dummy {
_ctx: &ExecutionContext, _ctx: &ExecutionContext,
_event_tx: &EventSender, _event_tx: &EventSender,
) -> Result<(), PluginError> { ) -> Result<(), PluginError> {
log::debug!("[NOTIFY_DEBUG] Successfully registered dummy plugin.");
Ok(()) Ok(())
} }
} }
@@ -1,12 +1,10 @@
use crate::finalize::plugin::PluginMetadata; use crate::finalize::plugin::PluginMetadata;
use crate::{ use crate::{
EventSender, ExecutionContext,
finalize::{error::PluginError, plugin::AsyncPluginFn}, finalize::{error::PluginError, plugin::AsyncPluginFn},
}; };
use async_trait::async_trait; use async_trait::async_trait;
use workshop_engine::{EventSender, ExecutionContext};
pub struct InsiteNotify; pub struct InsiteNotify;
#[async_trait] #[async_trait]
impl AsyncPluginFn for InsiteNotify { impl AsyncPluginFn for InsiteNotify {
async fn call( async fn call(
@@ -16,6 +14,7 @@ impl AsyncPluginFn for InsiteNotify {
_ctx: &ExecutionContext, _ctx: &ExecutionContext,
_event_tx: &EventSender, _event_tx: &EventSender,
) -> Result<(), PluginError> { ) -> Result<(), PluginError> {
unimplemented!("Requires workshop-base to be implemented!"); log::info!("in-site-notify: placeholder (always succeeds)");
Ok(())
} }
} }
@@ -1,18 +1,9 @@
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
use crate::finalize::plugin::PluginError;
use crate::finalize::plugin::PluginMetadata; use crate::finalize::plugin::PluginMetadata;
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn}; use crate::{
finalize::{error::PluginError, plugin::AsyncPluginFn},
};
use async_trait::async_trait; use async_trait::async_trait;
use lettre::message::{MultiPart, SinglePart, header}; use workshop_engine::{EventSender, ExecutionContext};
use lettre::{AsyncSmtpTransport, AsyncTransport, Message, Tokio1Executor};
use std::time::Duration;
use tokio::time::timeout;
mod config;
mod template;
pub use config::*;
use template::*;
pub struct Mail; pub struct Mail;
@@ -21,182 +12,10 @@ impl AsyncPluginFn for Mail {
async fn call( async fn call(
&self, &self,
_metadata: &PluginMetadata, _metadata: &PluginMetadata,
argument: serde_yaml::Value, _argument: serde_yaml::Value,
ctx: &ExecutionContext, _ctx: &ExecutionContext,
_event_tx: &EventSender, _event_tx: &EventSender,
) -> Result<(), PluginError> { ) -> Result<(), PluginError> {
let config: MailConfig = serde_yaml::from_value(argument) todo!();
.map_err(|e| PluginError::GeneralError(format!("Invalid mail config: {}", e)))?;
if let Err(errors) = config.validate() {
return Err(PluginError::GeneralError(format!(
"Mail config validation failed: {:?}",
errors
)));
}
let template_data = prepare_template_data(ctx);
let (subject_template, body_template_html, body_template_text) = match config.schema {
Schema::Default => {
let subject = config
.subject
.as_deref()
.unwrap_or(DefaultTemplates::SUBJECT);
let body_html = config
.body
.as_deref()
.unwrap_or(DefaultTemplates::BODY_HTML);
(
subject.to_string(),
body_html.to_string(),
DefaultTemplates::BODY_TEXT.to_string(),
)
}
Schema::Custom => {
let subject = config.subject.clone().unwrap();
let body_html = config.body.clone().unwrap();
(subject, body_html.clone(), body_html)
}
};
let subject = render_template(&subject_template, &template_data)
.map_err(|e| PluginError::GeneralError(format!("Failed to render subject: {}", e)))?;
let body_html = render_template(&body_template_html, &template_data)
.map_err(|e| PluginError::GeneralError(format!("Failed to render body_html: {}", e)))?;
let body_text = render_template(&body_template_text, &template_data).map_err(|e| {
PluginError::GeneralError(format!("Failed to render body_text(?): {}", e))
})?;
let email = build_email(&config, subject, body_html, body_text)?;
send_email(&config, email).await
} }
} }
fn build_email(
config: &MailConfig,
subject: String,
body_html: String,
body_text: String,
) -> Result<Message, PluginError> {
let from_str = config.from.to_string();
let from: lettre::message::Mailbox = from_str.parse().map_err(|e| {
PluginError::GeneralError(format!("Invalid from address '{}': {}", from_str, e))
})?;
let to_addrs: Vec<String> = config.to.to_vec();
if to_addrs.is_empty() {
return Err(PluginError::GeneralError(
"No recipients specified".to_string(),
));
}
let mut message_builder = Message::builder().from(from).subject(subject);
for addr in &to_addrs {
let mailbox: lettre::message::Mailbox = addr.parse().map_err(|e| {
PluginError::GeneralError(format!("Invalid to address '{}': {}", addr, e))
})?;
message_builder = message_builder.to(mailbox);
}
if let Some(cc) = &config.cc {
for addr in cc.to_vec() {
let mailbox: lettre::message::Mailbox = addr.parse().map_err(|e| {
PluginError::GeneralError(format!("Invalid cc address '{}': {}", addr, e))
})?;
message_builder = message_builder.cc(mailbox);
}
}
if let Some(bcc) = &config.bcc {
for addr in bcc.to_vec() {
let mailbox: lettre::message::Mailbox = addr.parse().map_err(|e| {
PluginError::GeneralError(format!("Invalid bcc address '{}': {}", addr, e))
})?;
message_builder = message_builder.bcc(mailbox);
}
}
if let Some(reply_to) = &config.reply_to {
let mailbox: lettre::message::Mailbox = reply_to.parse().map_err(|e| {
PluginError::GeneralError(format!("Invalid reply-to address '{}': {}", reply_to, e))
})?;
message_builder = message_builder.reply_to(mailbox);
}
let message = message_builder
.multipart(
MultiPart::alternative()
.singlepart(
SinglePart::builder()
.header(header::ContentType::TEXT_PLAIN)
.body(body_text),
)
.singlepart(
SinglePart::builder()
.header(header::ContentType::TEXT_HTML)
.body(body_html),
),
)
.map_err(|e| PluginError::GeneralError(format!("Failed to build email: {}", e)))?;
Ok(message)
}
async fn send_email(config: &MailConfig, email: Message) -> Result<(), PluginError> {
let connect_timeout = if config.smtp.connect_timeout_ms == 0 {
None
} else {
Some(Duration::from_millis(config.smtp.connect_timeout_ms))
};
let mut transport_builder = match config.smtp.encryption {
Encryption::Tls => {
log::debug!("Using TLS encryption");
AsyncSmtpTransport::<Tokio1Executor>::relay(&config.smtp.host)
.map_err(|e| PluginError::GeneralError(format!("Invalid SMTP host: {}", e)))?
}
Encryption::Starttls => {
log::debug!("Using StartTLS encryption");
AsyncSmtpTransport::<Tokio1Executor>::starttls_relay(&config.smtp.host)
.map_err(|e| PluginError::GeneralError(format!("Invalid SMTP host: {}", e)))?
}
Encryption::None => {
log::debug!("Using NO encryption");
AsyncSmtpTransport::<Tokio1Executor>::builder_dangerous(&config.smtp.host)
}
};
transport_builder = transport_builder.port(config.smtp.port);
match config.smtp.auth.auth_type {
AuthType::Password => {
log::debug!("Using password auth");
transport_builder = transport_builder.credentials(
(
config.smtp.auth.username.clone(),
config.smtp.auth.password.clone(),
)
.into(),
);
}
AuthType::None => {
log::debug!("Using no auth");
}
}
let transport = transport_builder.build();
let result = match connect_timeout {
Some(t) => timeout(t, transport.send(email)).await,
None => Ok(transport.send(email).await),
};
result
.map_err(|_| PluginError::GeneralError("SMTP connection timed out".to_string()))?
.map_err(|e| PluginError::GeneralError(format!("SMTP error: {}", e)))?;
Ok(())
}
@@ -1,490 +0,0 @@
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
use serde::Deserialize;
use workshop_engine::deserialize_duration_ms;
/// Email address representation supporting plain string or structured object.
#[derive(Debug, Deserialize, Clone)]
#[serde(untagged)]
pub enum EmailAddress {
/// Plain email address string.
String(String),
/// Structured email with display name.
Object { name: String, email: String },
}
impl EmailAddress {
/// Returns the formatted address string (e.g., "user@example.com" or "Name <user@example.com>").
pub fn to_string(&self) -> String {
match self {
EmailAddress::String(email) => email.clone(),
EmailAddress::Object { name, email } => {
if name.is_empty() {
email.clone()
} else {
format!("{} <{}>", name, email)
}
}
}
}
/// Returns the raw email address without formatting.
pub fn email(&self) -> &str {
match self {
EmailAddress::String(email) => email.as_str(),
EmailAddress::Object { email, .. } => email.as_str(),
}
}
}
/// Recipients container supporting single or multiple addresses.
#[derive(Debug, Deserialize, Clone)]
#[serde(untagged)]
pub enum Recipients {
/// Single recipient.
Single(String),
/// Multiple recipients.
Multiple(Vec<String>),
}
impl Recipients {
/// Converts recipients to a flat vector of email strings.
pub fn to_vec(&self) -> Vec<String> {
match self {
Recipients::Single(s) => vec![s.clone()],
Recipients::Multiple(v) => v.clone(),
}
}
}
/// SMTP authentication mechanism.
#[derive(Debug, Deserialize, Clone)]
#[serde(rename_all = "lowercase")]
pub enum AuthType {
/// No authentication.
None,
/// Password-based authentication.
Password,
}
/// SMTP authentication configuration.
#[derive(Debug, Deserialize, Clone)]
pub struct AuthConfig {
/// Authentication mechanism type.
#[serde(rename = "type")]
pub auth_type: AuthType,
/// Username for authentication.
pub username: String,
/// Password for authentication.
pub password: String,
}
/// SMTP connection encryption mode.
#[derive(Debug, Deserialize, Clone)]
#[serde(rename_all = "lowercase")]
pub enum Encryption {
/// Implicit TLS/SSL on connection.
Tls,
/// STARTTLS upgrade after initial plaintext connection.
Starttls,
/// No encryption.
None,
}
/// SMTP server connection configuration.
#[derive(Debug, Deserialize, Clone)]
pub struct SmtpConfig {
/// SMTP server hostname.
pub host: String,
/// SMTP server port.
pub port: u16,
/// Authentication configuration.
pub auth: AuthConfig,
/// Encryption mode.
pub encryption: Encryption,
/// Connection timeout in milliseconds.
#[serde(
default = "default_connect_timeout_ms",
deserialize_with = "deserialize_duration_ms"
)]
pub connect_timeout_ms: u64,
}
fn default_connect_timeout_ms() -> u64 {
30_000
}
/// Mail schema type determining required fields.
#[derive(Debug, Deserialize, Clone)]
#[serde(rename_all = "lowercase")]
pub enum Schema {
/// Default schema with minimal required fields.
Default,
/// Custom schema requiring explicit subject and body.
Custom,
}
fn default_schema() -> Schema {
Schema::Default
}
/// Complete mail notification configuration.
#[derive(Debug, Deserialize, Clone)]
pub struct MailConfig {
/// Primary recipients.
pub to: Recipients,
/// Carbon copy recipients.
#[serde(default)]
pub cc: Option<Recipients>,
/// Blind carbon copy recipients.
#[serde(default)]
pub bcc: Option<Recipients>,
/// Sender address.
pub from: EmailAddress,
/// Reply-to address.
#[serde(default)]
pub reply_to: Option<String>,
/// Schema type determining validation rules.
#[serde(default = "default_schema")]
pub schema: Schema,
/// Email subject (required for custom schema).
pub subject: Option<String>,
/// Email body content (required for custom schema).
pub body: Option<String>,
/// SMTP server configuration.
pub smtp: SmtpConfig,
}
impl MailConfig {
/// Validates the mail configuration according to schema rules.
/// Returns Ok(()) if valid, or Err(vec) with error messages if invalid.
pub fn validate(&self) -> Result<(), Vec<String>> {
let mut errors = Vec::new();
if matches!(self.schema, Schema::Custom) {
if self.subject.is_none() {
errors.push("subject: required when schema=custom".to_string());
}
if self.body.is_none() {
errors.push("body: required when schema=custom".to_string());
}
}
if self.to.to_vec().is_empty() {
errors.push("to: must have at least one recipient".to_string());
}
if errors.is_empty() {
Ok(())
} else {
Err(errors)
}
}
}
/// Built-in email notification templates.
pub struct DefaultTemplates;
impl DefaultTemplates {
/// Default subject template with build status.
pub const SUBJECT: &'static str =
"[{{ build.status | upper }}] {{ pipeline.name }} #{{ build.id }}";
/// Default HTML email body template.
pub const BODY_HTML: &'static str = r#"<!DOCTYPE html>
<html>
<body style="font-family: Arial, sans-serif;">
<h2 style="color: {{ build.status_color }};">
Build {{ build.status | title }}
</h2>
<table>
<tr><td><strong>Pipeline:</strong></td><td>{{ pipeline.name }}</td></tr>
<tr><td><strong>Build ID:</strong></td><td>{{ build.id }}</td></tr>
<tr><td><strong>Status:</strong></td><td>{{ build.status }}</td></tr>
<tr><td><strong>Duration:</strong></td><td>{{ build.duration }}</td></tr>
<tr><td><strong>Author:</strong></td><td>{{ commit.author }}</td></tr>
<tr><td><strong>Message:</strong></td><td>{{ commit.message }}</td></tr>
</table>
<p>
<a href="{{ build.url }}" style="background: #007bff; color: white; padding: 10px 20px; text-decoration: none; border-radius: 5px;">
View Build Details
</a>
</p>
</body>
</html>"#;
/// Default plain text email body template.
pub const BODY_TEXT: &'static str = r#"Build {{ build.status | title }}
==============================
Pipeline: {{ pipeline.name }}
Build ID: {{ build.id }}
Status: {{ build.status }}
Duration: {{ build.duration }}
Commit: {{ commit.hash }}
Author: {{ commit.author }}
Message: {{ commit.message }}
View details: {{ build.url }}"#;
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_email_address_string_to_string() {
let addr = EmailAddress::String("test@example.com".to_string());
assert_eq!(addr.to_string(), "test@example.com");
}
#[test]
fn test_email_address_object_with_name_to_string() {
let addr = EmailAddress::Object {
name: "Test User".to_string(),
email: "test@example.com".to_string(),
};
assert_eq!(addr.to_string(), "Test User <test@example.com>");
}
#[test]
fn test_email_address_object_without_name_to_string() {
let addr = EmailAddress::Object {
name: "".to_string(),
email: "test@example.com".to_string(),
};
assert_eq!(addr.to_string(), "test@example.com");
}
#[test]
fn test_email_address_string_email() {
let addr = EmailAddress::String("test@example.com".to_string());
assert_eq!(addr.email(), "test@example.com");
}
#[test]
fn test_email_address_object_email() {
let addr = EmailAddress::Object {
name: "Test User".to_string(),
email: "test@example.com".to_string(),
};
assert_eq!(addr.email(), "test@example.com");
}
#[test]
fn test_recipients_single_to_vec() {
let recipients = Recipients::Single("test@example.com".to_string());
let vec = recipients.to_vec();
assert_eq!(vec, vec!["test@example.com"]);
}
#[test]
fn test_recipients_multiple_to_vec() {
let recipients = Recipients::Multiple(vec![
"a@example.com".to_string(),
"b@example.com".to_string(),
]);
let vec = recipients.to_vec();
assert_eq!(vec, vec!["a@example.com", "b@example.com"]);
}
#[test]
fn test_mail_config_validate_valid_default_schema() {
let config = MailConfig {
to: Recipients::Single("to@example.com".to_string()),
cc: None,
bcc: None,
from: EmailAddress::String("from@example.com".to_string()),
reply_to: None,
schema: Schema::Default,
subject: None,
body: None,
smtp: SmtpConfig {
host: "smtp.example.com".to_string(),
port: 587,
auth: AuthConfig {
auth_type: AuthType::None,
username: "".to_string(),
password: "".to_string(),
},
encryption: Encryption::Tls,
connect_timeout_ms: 30_000,
},
};
assert!(config.validate().is_ok());
}
#[test]
fn test_mail_config_validate_valid_custom_schema() {
let config = MailConfig {
to: Recipients::Single("to@example.com".to_string()),
cc: None,
bcc: None,
from: EmailAddress::String("from@example.com".to_string()),
reply_to: None,
schema: Schema::Custom,
subject: Some("Test Subject".to_string()),
body: Some("Test Body".to_string()),
smtp: SmtpConfig {
host: "smtp.example.com".to_string(),
port: 587,
auth: AuthConfig {
auth_type: AuthType::None,
username: "".to_string(),
password: "".to_string(),
},
encryption: Encryption::Tls,
connect_timeout_ms: 30_000,
},
};
assert!(config.validate().is_ok());
}
#[test]
fn test_mail_config_validate_custom_schema_missing_subject() {
let config = MailConfig {
to: Recipients::Single("to@example.com".to_string()),
cc: None,
bcc: None,
from: EmailAddress::String("from@example.com".to_string()),
reply_to: None,
schema: Schema::Custom,
subject: None,
body: Some("Test Body".to_string()),
smtp: SmtpConfig {
host: "smtp.example.com".to_string(),
port: 587,
auth: AuthConfig {
auth_type: AuthType::None,
username: "".to_string(),
password: "".to_string(),
},
encryption: Encryption::Tls,
connect_timeout_ms: 30_000,
},
};
let result = config.validate();
assert!(result.is_err());
assert!(result.unwrap_err().iter().any(|e| e.contains("subject")));
}
#[test]
fn test_mail_config_validate_custom_schema_missing_body() {
let config = MailConfig {
to: Recipients::Single("to@example.com".to_string()),
cc: None,
bcc: None,
from: EmailAddress::String("from@example.com".to_string()),
reply_to: None,
schema: Schema::Custom,
subject: Some("Test Subject".to_string()),
body: None,
smtp: SmtpConfig {
host: "smtp.example.com".to_string(),
port: 587,
auth: AuthConfig {
auth_type: AuthType::None,
username: "".to_string(),
password: "".to_string(),
},
encryption: Encryption::Tls,
connect_timeout_ms: 30_000,
},
};
let result = config.validate();
assert!(result.is_err());
assert!(result.unwrap_err().iter().any(|e| e.contains("body")));
}
#[test]
fn test_mail_config_validate_empty_recipients() {
let config = MailConfig {
to: Recipients::Multiple(vec![]),
cc: None,
bcc: None,
from: EmailAddress::String("from@example.com".to_string()),
reply_to: None,
schema: Schema::Default,
subject: None,
body: None,
smtp: SmtpConfig {
host: "smtp.example.com".to_string(),
port: 587,
auth: AuthConfig {
auth_type: AuthType::None,
username: "".to_string(),
password: "".to_string(),
},
encryption: Encryption::Tls,
connect_timeout_ms: 30_000,
},
};
let result = config.validate();
assert!(result.is_err());
assert!(result.unwrap_err().iter().any(|e| e.contains("recipient")));
}
#[test]
fn test_email_address_serde_string() {
let json = r#""test@example.com""#;
let addr: EmailAddress = serde_json::from_str(json).unwrap();
assert!(matches!(addr, EmailAddress::String(s) if s == "test@example.com"));
}
#[test]
fn test_email_address_serde_object() {
let json = r#"{"name":"Test User","email":"test@example.com"}"#;
let addr: EmailAddress = serde_json::from_str(json).unwrap();
assert!(
matches!(addr, EmailAddress::Object { name, email } if name == "Test User" && email == "test@example.com")
);
}
#[test]
fn test_recipients_serde_single() {
let json = r#""single@example.com""#;
let recipients: Recipients = serde_json::from_str(json).unwrap();
assert!(matches!(recipients, Recipients::Single(s) if s == "single@example.com"));
}
#[test]
fn test_recipients_serde_multiple() {
let json = r#"["a@example.com","b@example.com"]"#;
let recipients: Recipients = serde_json::from_str(json).unwrap();
assert!(
matches!(recipients, Recipients::Multiple(v) if v == vec!["a@example.com".to_string(), "b@example.com".to_string()])
);
}
#[test]
fn test_auth_type_serde() {
#[derive(serde::Deserialize, Debug)]
struct Test {
#[serde(rename = "type")]
auth_type: AuthType,
}
let json = r#"{"type":"password"}"#;
let t: Test = serde_json::from_str(json).unwrap();
assert!(matches!(t.auth_type, AuthType::Password));
}
#[test]
fn test_encryption_serde() {
#[derive(serde::Deserialize, Debug)]
struct Test {
encryption: Encryption,
}
let json = r#"{"encryption":"tls"}"#;
let t: Test = serde_json::from_str(json).unwrap();
assert!(matches!(t.encryption, Encryption::Tls));
}
#[test]
fn test_schema_serde() {
#[derive(serde::Deserialize, Debug)]
struct Test {
schema: Schema,
}
let json = r#"{"schema":"custom"}"#;
let t: Test = serde_json::from_str(json).unwrap();
assert!(matches!(t.schema, Schema::Custom));
}
}
@@ -1,101 +0,0 @@
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
use crate::ExecutionContext;
use minijinja::Environment;
use std::collections::HashMap;
/// Builds template context data from execution context.
/// Populates pipeline, build, and commit information for template rendering.
pub fn prepare_template_data(ctx: &ExecutionContext) -> HashMap<String, serde_json::Value> {
let mut data = HashMap::new();
data.insert(
"pipeline".to_string(),
serde_json::json!({
"name": ctx.pipeline_name,
}),
);
data.insert(
"build".to_string(),
serde_json::json!({
"id": ctx.task_id,
"status": "success",
"status_color": "green",
"duration": "0s",
"url": "",
}),
);
data.insert(
"commit".to_string(),
serde_json::json!({
"hash": "",
"short_hash": "",
"message": "",
"author": "",
"author_email": "",
}),
);
data
}
/// Renders a minijinja template string with the provided data map.
/// Returns the rendered string on success, or a String error message on failure.
pub fn render_template(
template: &str,
data: &HashMap<String, serde_json::Value>,
) -> Result<String, String> {
let env = Environment::new();
let tmpl = env
.template_from_str(template)
.map_err(|e| format!("Template parse error: {}", e))?;
let result = tmpl
.render(data)
.map_err(|e| format!("Template render error: {}", e))?;
Ok(result)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_render_template_simple() {
let template = "Hello {{ name }}!";
let mut data = HashMap::new();
data.insert(
"name".to_string(),
serde_json::Value::String("World".to_string()),
);
let result = render_template(template, &data).unwrap();
assert_eq!(result, "Hello World!");
}
#[test]
fn test_render_template_with_build_status() {
let template = "Build {{ build.status }} for pipeline {{ pipeline.name }}";
let mut data = HashMap::new();
data.insert(
"build".to_string(),
serde_json::json!({ "status": "success" }),
);
data.insert(
"pipeline".to_string(),
serde_json::json!({ "name": "test-pipeline" }),
);
let result = render_template(template, &data).unwrap();
assert_eq!(result, "Build success for pipeline test-pipeline");
}
#[test]
fn test_render_template_invalid_syntax() {
let template = "{{ Unterminated";
let data = HashMap::new();
let result = render_template(template, &data);
assert!(result.is_err());
}
}
@@ -1,6 +1,7 @@
use crate::finalize::plugin::PluginError; use crate::finalize::plugin::PluginError;
use crate::finalize::plugin::PluginMetadata; use crate::finalize::plugin::PluginMetadata;
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn}; use workshop_engine::{EventSender, ExecutionContext};
use crate::{ finalize::plugin::AsyncPluginFn};
use async_trait::async_trait; use async_trait::async_trait;
pub struct Satori; pub struct Satori;
@@ -1,6 +1,7 @@
use crate::finalize::plugin::PluginError; use crate::finalize::plugin::PluginError;
use crate::finalize::plugin::PluginMetadata; use crate::finalize::plugin::PluginMetadata;
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn}; use workshop_engine::{EventSender, ExecutionContext};
use crate::{ finalize::plugin::AsyncPluginFn};
use async_trait::async_trait; use async_trait::async_trait;
pub struct Webhook; pub struct Webhook;
+7 -74
View File
@@ -1,3 +1,4 @@
use crate::finalize::NotificationTemplateDef;
use crate::notify::types::NotificationRenderer; use crate::notify::types::NotificationRenderer;
use serde::Deserialize; use serde::Deserialize;
use std::path::PathBuf; use std::path::PathBuf;
@@ -18,10 +19,14 @@ pub struct PluginMetadata {
#[serde(default)] #[serde(default)]
pub renderer: Option<NotificationRenderer>, pub renderer: Option<NotificationRenderer>,
/// Additional YAML fields not captured by struct. /// Additional YAML fields.
#[serde(flatten)] #[serde(flatten)]
pub value: serde_yaml::Value, pub value: serde_yaml::Value,
/// Plugin-defined default templates, auto-registered as <name>.fallback
#[serde(default)]
pub templates: Option<Vec<NotificationTemplateDef>>,
/// Filesystem path to plugin definition. /// Filesystem path to plugin definition.
#[serde(skip)] #[serde(skip)]
pub fspath: PathBuf, pub fspath: PathBuf,
@@ -36,6 +41,7 @@ impl Default for PluginMetadata {
plugin_type: PluginType::Unknown, plugin_type: PluginType::Unknown,
value: serde_yaml::Value::Null, value: serde_yaml::Value::Null,
renderer: None, renderer: None,
templates: None,
fspath: PathBuf::new(), fspath: PathBuf::new(),
} }
} }
@@ -50,76 +56,3 @@ pub enum PluginType {
Rhai, Rhai,
Dylib, Dylib,
} }
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_plugin_metadata_default() {
let meta = PluginMetadata::default();
assert_eq!(meta.name, "");
assert_eq!(meta.version, "");
assert_eq!(meta.entrypoint, "");
assert_eq!(meta.plugin_type, PluginType::Unknown);
}
#[test]
fn test_plugin_type_default() {
let pt = PluginType::default();
assert_eq!(pt, PluginType::Unknown);
}
#[test]
fn test_plugin_type_equality() {
assert_eq!(PluginType::Shell, PluginType::Shell);
assert_eq!(PluginType::Dylib, PluginType::Dylib);
assert_ne!(PluginType::Shell, PluginType::Dylib);
}
#[test]
fn test_plugin_metadata_clone() {
let meta = PluginMetadata {
name: "test".to_string(),
version: "1.0".to_string(),
entrypoint: "/bin/test".to_string(),
plugin_type: PluginType::Shell,
value: serde_yaml::Value::Null,
fspath: PathBuf::from("/path/to/plugin"),
renderer: None,
};
let cloned = meta.clone();
assert_eq!(cloned.name, meta.name);
assert_eq!(cloned.version, meta.version);
assert_eq!(cloned.plugin_type, meta.plugin_type);
}
#[test]
fn test_plugin_type_serde_unknown() {
let yaml = "Unknown";
let pt: PluginType = serde_yaml::from_str(yaml).unwrap();
assert_eq!(pt, PluginType::Unknown);
}
#[test]
fn test_plugin_type_serde_shell() {
let yaml = "Shell";
let pt: PluginType = serde_yaml::from_str(yaml).unwrap();
assert_eq!(pt, PluginType::Shell);
}
#[test]
fn test_plugin_metadata_serde() {
let yaml = r#"
name: test-plugin
version: "2.0"
entrypoint: ./bin/test
plugin_type: Shell
"#;
let meta: PluginMetadata = serde_yaml::from_str(yaml).unwrap();
assert_eq!(meta.name, "test-plugin");
assert_eq!(meta.version, "2.0");
assert_eq!(meta.entrypoint, "./bin/test");
assert_eq!(meta.plugin_type, PluginType::Shell);
}
}
+2 -1
View File
@@ -1,6 +1,7 @@
use crate::finalize::plugin::PluginError; use crate::finalize::plugin::PluginError;
use crate::finalize::plugin::PluginMetadata; use crate::finalize::plugin::PluginMetadata;
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn}; use workshop_engine::{EventSender, ExecutionContext};
use crate::{ finalize::plugin::AsyncPluginFn};
use async_trait::async_trait; use async_trait::async_trait;
pub struct Rhai; pub struct Rhai;
+3 -2
View File
@@ -1,8 +1,9 @@
use crate::Engine; use workshop_engine::Engine;
use crate::finalize::constant::FINALIZE_SHELL_ENVPREFIX; use crate::finalize::constant::FINALIZE_SHELL_ENVPREFIX;
use crate::finalize::plugin::PluginError; use crate::finalize::plugin::PluginError;
use crate::finalize::plugin::PluginMetadata; use crate::finalize::plugin::PluginMetadata;
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn}; use workshop_engine::{EventSender, ExecutionContext};
use crate::{ finalize::plugin::AsyncPluginFn};
use async_trait::async_trait; use async_trait::async_trait;
use std::collections::HashMap; use std::collections::HashMap;
+1 -1
View File
@@ -1,4 +1,4 @@
use crate::ExecutionContext; use workshop_engine::ExecutionContext;
use crate::finalize::plugin::{PluginMap, call_named_plugin}; use crate::finalize::plugin::{PluginMap, call_named_plugin};
use std::path::PathBuf; use std::path::PathBuf;
use workshop_engine::{CustomCommand, DeltaExecutionContext, Engine, EventSender, ExecutionError}; use workshop_engine::{CustomCommand, DeltaExecutionContext, Engine, EventSender, ExecutionError};
+25 -57
View File
@@ -1,27 +1,16 @@
// Library crate for workshop-executor pub mod bare;
// Selective exports: only modules used by external callers or tests
pub mod bake; pub mod bake;
pub mod daemon; pub mod error;
pub mod finalize; pub mod finalize;
pub mod monitor;
pub mod notify; pub mod notify;
pub mod prebake; pub mod prebake;
pub mod socket;
pub mod types; pub mod types;
pub mod utils;
// External crates re-exports
pub use config;
// Re-export commonly used types from workshop-engine
pub use workshop_engine::{Engine, EventSender, ExecutionContext, ExecutionError, ExecutionResult};
// CLI types used by main (keep here since they're used in daemon)
pub use clap::{Parser, Subcommand}; pub use clap::{Parser, Subcommand};
pub use serde::Deserialize; pub use serde::Deserialize;
pub use std::collections::HashMap; pub use std::collections::HashMap;
pub use std::path::PathBuf; pub use std::path::PathBuf;
pub mod cli { pub mod cli {
pub use super::{Deserialize, HashMap, Parser, PathBuf, Subcommand}; pub use super::{Deserialize, HashMap, Parser, PathBuf, Subcommand};
pub use clap::Command; pub use clap::Command;
@@ -46,64 +35,43 @@ pub mod cli {
#[arg(long, default_value = "false", env = "HBW_DRY_RUN")] #[arg(long, default_value = "false", env = "HBW_DRY_RUN")]
pub dry_run: bool, pub dry_run: bool,
/// Override prebake config path
#[arg(long)]
pub prebake: Option<PathBuf>,
/// Override bake script path
#[arg(long)]
pub bake: Option<PathBuf>,
/// Override bake_base.sh path
#[arg(long)]
pub bake_base: Option<PathBuf>,
/// Override finalize config path
#[arg(long)]
pub finalize: Option<PathBuf>,
/// Debug features (comma-separated). "notify" enables notification debug logging.
#[arg(long, env = "HBW_DEBUG")]
pub debug: Option<String>,
#[command(subcommand)] #[command(subcommand)]
pub command: Option<Commands>, pub command: Option<Commands>,
} }
#[derive(Subcommand, Debug)] #[derive(Subcommand, Debug)]
pub enum Commands { pub enum Commands {
/// Run pipeline stages in standalone (bare) mode /// Run individual pipeline stages
Bare { Bare {
#[command(subcommand)] #[command(subcommand)]
command: BareCommands, command: BareCommands,
}, },
/// Run in client mode Daemon {},
Client { config: String },
} }
#[derive(Subcommand, Debug)] #[derive(Subcommand, Debug)]
pub enum BareCommands { pub enum BareCommands {
/// Prepare a task Prebake {},
Prebake { config: PathBuf }, Bake {},
Finalize {},
/// Run a task
Bake {
/// Path to the script to execute
script: PathBuf,
/// Path to bake_base.sh
#[arg(short, long, default_value = "./bake_base.sh")]
bake_base: PathBuf,
#[arg(short, long, default_value = "./prebake.yml")]
prebake: PathBuf,
},
/// Finish a task
Finalize { config: String },
} }
#[derive(Debug, Deserialize)]
pub struct TaskSpec {
pub subcommand: String,
pub arguments: Vec<String>,
pub options: HashMap<String, String>,
}
impl TaskSpec {
pub fn command(&self, executable: &PathBuf) -> tokio::process::Command {
let mut command = tokio::process::Command::new(executable);
command.arg(&self.subcommand);
for arg in &self.arguments {
command.arg(arg);
}
for (flag, value) in self.options.iter() {
if value.is_empty() {
command.arg(format!("--{}", flag));
} else {
command.arg(format!("--{}={}", flag, value));
}
}
command
}
}
} }
+110 -63
View File
@@ -1,13 +1,91 @@
use std::path::Path; use workshop_baker::types::resource::ResourceRegistry;
use std::path::PathBuf;
use workshop_baker::{ use workshop_baker::{
bake::bake, bake::bake, cli::{Cli, Commands, Parser},
cli::{BareCommands, Cli, Commands, Parser}, error::CliError,
daemon, finalize::finalize, notify::notify, prebake::prebake,
finalize::finalize,
prebake::prebake,
}; };
use workshop_engine::{EventSender, ExecutionContext, error::HasExitCode}; use workshop_baker::bare::run_bare;
use workshop_baker::notify::queue::NotificationQueue;
use workshop_engine::{EventSender, ExecutionContext};
fn or_workshop(path: &Option<PathBuf>, name: &str) -> PathBuf {
path.clone().unwrap_or_else(|| PathBuf::from(format!(".workshop/{}", name)))
}
async fn worker_main(cli: Cli) -> Result<(), CliError> {
let prebake_path = cli.prebake.clone().unwrap_or_else(|| {
log::error!("--prebake is required in worker standalone mode");
std::process::exit(1);
});
let bake_path = cli.bake.clone().unwrap_or_else(|| {
log::error!("--bake is required in worker standalone mode");
std::process::exit(1);
});
let finalize_path = cli.finalize.clone().unwrap_or_else(|| {
log::error!("--finalize is required in worker standalone mode");
std::process::exit(1);
});
log::info!("Worker standalone mode");
let mut ctx = ExecutionContext {
standalone: true,
task_id: format!("{}-{}-worker", cli.pipeline, cli.build_id),
pipeline_name: cli.pipeline.clone(),
username: cli.username.clone(),
dry_run: cli.dry_run,
privileged: false,
..Default::default()
};
let (tx, event_rx) = tokio::sync::mpsc::channel(256);
let event_tx: EventSender = Some(tx);
let _event_handle = tokio::spawn(workshop_baker::prebake::event::event_receiver(
event_rx, cli.pipeline.clone(), cli.build_id.clone(),
));
let (to_tx, to_rx) = tokio::sync::mpsc::channel(256);
let (retry_tx, retry_rx) = tokio::sync::mpsc::channel(256);
// Resource registry — populated before stages, consumed by stages
let registry = ResourceRegistry::new();
// TODO: pre-fetch plugins and templates into registry
let notify_path = finalize_path.clone();
let notify_ctx = ctx.clone();
let notify_etx = event_tx.clone();
let debug = cli.debug.clone();
let notify_registry = registry.clone();
let notify_handle = tokio::spawn(async move {
notify(&notify_path, &mut notify_ctx.clone(), notify_etx, to_rx, retry_tx, debug.map(|d| d.contains("notify")).unwrap_or(false), &notify_registry).await
});
let nevent_tx = to_tx.clone();
let mut queue = NotificationQueue::new(to_tx, retry_rx);
let pipe_result: anyhow::Result<()> = async {
prebake(&prebake_path, &cli, None, &mut ctx, event_tx.clone(), Some(nevent_tx)).await?;
let bake_base_path = or_workshop(&cli.bake_base, "bake_base.sh");
let pb_path = or_workshop(&cli.prebake, "prebake.yml");
bake(&bake_path, &bake_base_path, &pb_path, &cli, &mut ctx, event_tx.clone()).await?;
finalize(&finalize_path, &cli, &mut ctx, event_tx.clone(), &registry).await?;
Ok(())
}.await;
if let Err(e) = pipe_result {
log::error!("Pipeline stage failed: {}", e);
return Err(CliError::General(e));
}
log::info!("Pipeline complete. Draining notifications...");
if let Err(e) = queue.drain().await {
log::error!("Notification queue error: {}", e);
}
let _ = notify_handle.await;
Ok(())
}
#[tokio::main] #[tokio::main]
async fn main() { async fn main() {
@@ -16,77 +94,46 @@ async fn main() {
if cli.dry_run { if cli.dry_run {
log::warn!("Dry run enabled, no changes will be made."); log::warn!("Dry run enabled, no changes will be made.");
} }
match &cli.command {
Some(Commands::Client { config: _ }) => {
unimplemented!("Client mode not yet implemented.")
}
Some(Commands::Bare { command }) => {
log::info!("Running in standalone mode.");
let (stage, privileged) = match command { let pipeline = cli.pipeline.clone();
BareCommands::Prebake { .. } => ("init", true), let build_id = cli.build_id.clone();
BareCommands::Bake { .. } => ("bake", false),
BareCommands::Finalize { .. } => ("finalize", false), match &cli.command {
}; Some(Commands::Bare { command }) => {
let mut ctx = ExecutionContext { let mut ctx = ExecutionContext {
standalone: true, standalone: true,
task_id: format!("{}-{}-{}", cli.pipeline, cli.build_id, stage), task_id: format!("{}-{}-bare", pipeline, build_id),
pipeline_name: cli.pipeline.clone(), pipeline_name: pipeline.clone(),
username: cli.username.clone(), username: cli.username.clone(),
dry_run: cli.dry_run, dry_run: cli.dry_run,
privileged, privileged: false,
..Default::default() ..Default::default()
}; };
let (tx, event_rx) = tokio::sync::mpsc::channel(256); let (tx, event_rx) = tokio::sync::mpsc::channel(256);
let event_tx: EventSender = Some(tx); let event_tx: EventSender = Some(tx);
let handle = tokio::spawn(workshop_baker::prebake::event::event_receiver( let event_handle = tokio::spawn(workshop_baker::prebake::event::event_receiver(
event_rx, event_rx, pipeline.clone(), build_id.clone(),
cli.pipeline.clone(),
cli.build_id.clone(),
)); ));
let result: anyhow::Result<()> = match command { let registry = ResourceRegistry::new();
BareCommands::Prebake { config } => { let result = run_bare(&cli, command, &mut ctx, event_tx, &pipeline, &build_id, &registry).await;
prebake(config, &cli, None, &mut ctx, event_tx).await
}
BareCommands::Bake {
script,
bake_base,
prebake: prebake_path,
} => bake(script, bake_base, prebake_path, &cli, &mut ctx, event_tx)
.await
.map_err(|e| e.into()),
BareCommands::Finalize { config } => {
finalize(Path::new(config), &cli, &mut ctx, event_tx)
.await
.map_err(|e| e.into())
}
};
drop(ctx); drop(ctx);
let _ = handle.await; let _ = event_handle.await;
dbg!(&result);
if let Err(ref e) = result { if let Err(e) = result {
let code = match command { log::error!("{}", e);
BareCommands::Prebake { .. } => e std::process::exit(e.exit_code());
.downcast_ref::<workshop_baker::prebake::error::PrebakeError>()
.map(|pe| pe.exit_code())
.unwrap_or(1),
BareCommands::Bake { .. } => e
.downcast_ref::<workshop_baker::bake::error::BakeError>()
.map(|be| be.exit_code())
.unwrap_or(1),
BareCommands::Finalize { .. } => e
.downcast_ref::<workshop_baker::finalize::FinalizeError>()
.map(|fe| fe.exit_code())
.unwrap_or(1),
};
log::error!("{} stage failed: {}", stage, e);
std::process::exit(code);
} }
} }
Some(Commands::Daemon {}) => {
unimplemented!("Daemon mode not implemented");
}
None => { None => {
log::info!("Executor will be running in daemon mode."); if let Err(e) = worker_main(cli).await {
let _result = daemon::daemon(&cli).await; log::error!("{}", e);
std::process::exit(e.exit_code());
}
} }
} }
} }
-89
View File
@@ -1,89 +0,0 @@
mod cgroups;
mod jobobject;
use crate::socket::{establish_connection, get_socket_addr};
use config::Config;
use serde::{Deserialize, Serialize};
use std::time::Instant;
use tokio::io::AsyncWriteExt;
#[derive(Debug, Serialize, Deserialize)]
pub struct ResourceUsageInfo {
pub msgtype: String,
pub name: String,
pub os_type: OsType,
pub statistics: UsageStats,
pub timestamp: u64,
}
#[derive(Debug, Serialize, Deserialize)]
pub enum OsType {
Linux,
Windows,
MacOS,
}
#[derive(Debug, Serialize, Deserialize, Clone)]
pub struct UsageStats {
pub cpu_usage_percent: f64,
pub memory_usage_bytes: u64,
pub memory_limit_bytes: i64,
// pub memory_usage_percent: f64,
pub process_count: u64,
pub cpu_usage_us: u64,
/// Internal timestamp, using std::Instant
#[serde(skip, default = "default_internal_timestamp")]
_timestamp: Instant,
}
fn default_internal_timestamp() -> Instant {
Instant::now()
}
pub async fn monitor(
name: &String,
interval_ms: Option<u64>,
settings: Config,
) -> anyhow::Result<()> {
// let socket_addr = get_wsagent_socket_addr()?;
let socket_addr = get_socket_addr(settings.get_string("socket")?)?;
let interval = tokio::time::Duration::from_millis(interval_ms.unwrap_or(1000));
let mut socket = establish_connection(&socket_addr).await?;
let mut prev_usage = UsageStats {
cpu_usage_percent: 0.0,
memory_usage_bytes: 0,
memory_limit_bytes: 0,
process_count: 0,
cpu_usage_us: 0,
_timestamp: default_internal_timestamp(),
};
loop {
let usage: UsageStats = cgroups::get_usage(name, prev_usage)?; // TODO: 多系统支持?
prev_usage = usage.clone();
// dbg!(&usage);
let usage_info = ResourceUsageInfo {
msgtype: "ResourceUsage".to_string(),
name: name.clone(),
os_type: OsType::Linux, // TODO: 多系统支持?
statistics: usage,
timestamp: std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)?
.as_millis() as u64,
};
let usage_json = serde_json::to_string(&usage_info)?;
send_usage_info(&mut socket, &usage_json).await?;
// TODO: 增加消息接收,提供(伪)双向通信功能
tokio::time::sleep(interval).await;
}
}
async fn send_usage_info<S>(stream: &mut S, usage_json: &str) -> anyhow::Result<()>
where
S: tokio::io::AsyncWrite + Unpin,
{
stream.write_all(usage_json.as_bytes()).await?;
stream.write_all(b"\n").await?;
Ok(())
}
+429 -361
View File
@@ -1,400 +1,468 @@
use crate::finalize::FinalizeConfig;
use crate::finalize::NotificationConfig;
use crate::finalize::NotificationTemplate;
use crate::finalize::config::TriggerItemRaw;
use crate::finalize::constant::FINALIZE_STANDALONE_PLUGIN_PATH;
use crate::finalize::parse; use crate::finalize::parse;
use crate::finalize::plugin; use crate::finalize::plugin;
use crate::finalize::plugin::fetch; use crate::notify::types::DEFAULT_NOTIFY_PARAMS;
use crate::notify::types::Matchable;
use crate::notify::types::Method;
use crate::notify::types::NOTIFICATION_DEFAULT_FALLIBLE;
use crate::notify::types::NOTIFICATION_DEFAULT_MAX_LIVES;
use crate::notify::types::NOTIFICATION_DEFAULT_RETRY_BACKOFF;
use crate::notify::types::NOTIFICATION_DEFAULT_TIMEOUT_MS;
use crate::notify::types::NOTIFICATION_NOTREADY_DELAY;
use crate::notify::types::NOTIFICATION_TEMPORARY_DELAY_FACTOR;
use crate::notify::types::NOTIFICATION_TEMPORARY_DELAY_TIME;
use crate::notify::types::NOTIFICATION_TEMPORARY_MAX_TIME;
use crate::notify::types::NotificationEvent;
use crate::notify::types::NotificationEventReceiver; use crate::notify::types::NotificationEventReceiver;
use crate::notify::types::NotificationEventSender; use crate::notify::types::NotificationEventSender;
use crate::notify::types::NotificationPluginMap; use crate::notify::handler::NotificationHandler;
use crate::notify::types::NotificationRenderer; use crate::types::resource::ResourceRegistry;
use crate::notify::types::NotificationRule;
use crate::notify::types::NotificationRulesMap;
use crate::notify::types::ParsedTrigger;
use crate::{EventSender, ExecutionContext};
use chrono::Utc;
use globset::{Glob, GlobSetBuilder};
use std::collections::HashMap;
use std::collections::HashSet;
use std::path::Path; use std::path::Path;
use std::path::PathBuf; use std::sync::Arc;
use workshop_engine::{EventSender, ExecutionContext};
mod error; mod error;
pub mod handler;
pub mod queue;
pub mod rule;
pub mod template;
pub mod types; pub mod types;
use error::NotifyError; pub mod util;
use rule::parse_rules;
fn validate(finalize: &FinalizeConfig) -> anyhow::Result<()> { use template::{flatten_template_refs, resolve_external_template_refs};
let validate_result = finalize.validate(); use util::{extract_plugin, validate};
if let Err(errors) = validate_result {
for error in &errors {
log::error!("Error: {}", error);
}
anyhow::bail!(NotifyError::FinalizeValidationError(errors))
};
Ok(())
}
fn extract_plugin(finalize: &FinalizeConfig) -> HashSet<String> {
finalize
.notification
.groups
.values()
.flat_map(|methods| methods.keys())
.cloned()
.collect()
}
pub async fn notify( pub async fn notify(
finalize_path: &Path, finalize_path: &Path,
ctx: &mut ExecutionContext, ctx: &mut ExecutionContext,
_event_tx: EventSender, _event_tx: EventSender,
notifyevent_rx: &mut NotificationEventReceiver, mut rx: NotificationEventReceiver,
notifyevent_tx: NotificationEventSender, retry_tx: NotificationEventSender,
_debug: bool,
registry: &ResourceRegistry,
) -> anyhow::Result<()> { ) -> anyhow::Result<()> {
let mut finalize = parse(finalize_path)?; let mut finalize = parse(finalize_path)?;
// Always inject in-site-notify as the lowest priority fallback
{
let max_p = finalize.notification.groups.keys().max().copied().unwrap_or(0);
let mut m = std::collections::HashMap::new();
m.insert(
"in-site-notify".to_string(),
crate::finalize::NotificationMethod::default(),
);
finalize.notification.groups.insert(max_p + 1, m);
}
validate(&finalize)?; validate(&finalize)?;
let plugin_list = extract_plugin(&finalize); let plugin_list = extract_plugin(&finalize);
if plugin_list.is_empty() { let notify_plugins = plugin::register(
log::warn!("No notification method provided, skipping notification"); finalize.plugin,
// TODO: send event to event_tx Some(&plugin_list),
return Ok(()); &mut finalize.notification,
} registry,
)?;
if ctx.standalone { resolve_external_template_refs(&mut finalize.notification.templates, registry);
fetch::fetch_plugins( flatten_template_refs(&mut finalize.notification.templates)?;
&PathBuf::from(FINALIZE_STANDALONE_PLUGIN_PATH),
&mut finalize.plugin,
Some(&plugin_list),
)
.await?;
}
let notify_plugins = plugin::register(finalize.plugin, Some(&plugin_list))?;
let notification_rules = parse_rules(&finalize.notification); let notification_rules = parse_rules(&finalize.notification);
loop { let handler = Arc::new(NotificationHandler::new(
tokio::select! { Arc::new(notification_rules),
Some(event) = notifyevent_rx.recv() => { Arc::new(notify_plugins),
handle_notification_event( Arc::new(finalize.notification.templates),
event, Arc::new(ctx.clone()),
&notify_plugins, retry_tx,
&notification_rules, DEFAULT_NOTIFY_PARAMS.batch_period.value,
&notifyevent_tx, DEFAULT_NOTIFY_PARAMS.batch_watermark.value,
) ));
.await?;
}
}
}
}
/// Process a single notification event. loop {
/// let Some(event) = rx.recv().await else { break };
/// Delegates to [`notification_handler`] and handles errors: let h = handler.clone();
/// - [`NotifyError::PluginNotAvailable`] — reschedule with static delay, lower priority tokio::spawn(async move {
/// - [`NotifyError::TemporaryError`] — consume one life, reschedule with exponential backoff h.process_event(event).await;
/// - [`NotifyError::PermanentError`] — drop the event });
/// - other — log and drop
async fn handle_notification_event(
event: NotificationEvent,
plugins: &NotificationPluginMap,
rules: &NotificationRulesMap,
notifyevent_tx: &NotificationEventSender,
) -> anyhow::Result<()> {
match notification_handler(plugins, rules, &event) {
Ok(_) => {}
Err(e) => {
log::debug!("Failed to send notification: {:?}", e);
match e {
NotifyError::PluginNotAvailable(_) => {
let mut event = event;
event.not_before = Utc::now() + NOTIFICATION_NOTREADY_DELAY;
event.effective_priority = event.effective_priority.saturating_add(1);
notifyevent_tx.send(event).await?;
}
NotifyError::TemporaryError(_) => {
let mut event = event;
event.lives -= 1;
event.not_before = Utc::now() + notify_delay(&event);
notifyevent_tx.send(event).await?;
}
NotifyError::PermanentError(_) => {
log::error!("None of the configured notification plugins works.");
log::error!("Message will be dropped.");
}
error => {
log::error!(
"Internal error: notify_handler returned unexpected error: {:?}",
error
);
}
}
}
} }
Ok(()) Ok(())
} }
fn notification_handler( // TODO: PIPELINE_NOTIFICATION_EXHAUSTED — set when any method returns Exhausted
plugins: &NotificationPluginMap, // TODO: PIPELINE_NOTIFICATION_FAILED — set when any method returns PermanentError
config: &NotificationRulesMap, // Markers sent via event_tx as ExecutionEvent variants for future WebUI/CLI display.
event: &NotificationEvent,
) -> Result<(), NotifyError> {
for (_priority, ruleset) in config.iter() {
for (method, rules) in ruleset.iter() {
if !event.target.contains(method) {
log::trace!("target does not contain method name: {}", method);
continue;
}
for (rule_id, rule) in rules.iter().enumerate() {
if !rule.enabled {
log::trace!("rule is not enabled: {}", rule_id);
continue;
}
if rule.accept(&event.outcome, &event.stage, &event.substage) {
let plugin = plugins
.get(method)
.ok_or(NotifyError::UnknownPlugin(method.clone()))?;
let renderer = plugin.metadata.renderer.clone().unwrap_or_default();
match renderer {
NotificationRenderer::Server => {
// ...
todo!();
}
NotificationRenderer::Plugin => {
unimplemented!();
return Err(NotifyError::UnsupportedRenderer(method.clone(), renderer));
}
NotificationRenderer::Interactive => {
unimplemented!();
return Err(NotifyError::UnsupportedRenderer(method.clone(), renderer));
}
}
}
}
}
}
todo!(); #[cfg(test)]
} mod tests {
use super::*;
use crate::finalize::NotificationTemplateDef;
use crate::notify::types::NotificationRule;
use crate::notify::types::NotificationTarget;
use std::collections::HashMap;
use crate::notify::types::NotificationEvent;
use crate::notify::types::NotificationRulesMap;
use crate::notify::error::NotifyError;
use crate::notify::template::resolve_template;
use crate::notify::util::{find_rule, notify_delay, wait_time};
fn parse_trigger_item_raw(item: &TriggerItemRaw) -> Vec<ParsedTrigger> { // ── helper ──
match item {
TriggerItemRaw::Simple(s) => match s.as_str() {
"on_success" => vec![ParsedTrigger::OnSuccess],
"on_failure" => vec![ParsedTrigger::OnFailure],
s if s.starts_with("on_finish_of:") => {
let pattern = s.trim_start_matches("on_finish_of:").trim();
let mut builder = GlobSetBuilder::new();
builder.add(Glob::new(pattern).expect("Invalid glob pattern"));
let glob_set = builder.build().expect("Failed to compile glob set");
vec![ParsedTrigger::OnFinishOf(Matchable::Glob(glob_set))]
}
_ => {
log::warn!("Unknown trigger string: {}", s);
vec![]
}
},
TriggerItemRaw::Map(map) => {
let mut triggers = Vec::new();
for (key, patterns) in map {
match key.as_str() {
"on_finish_of" => {
for pattern in patterns {
let mut builder = GlobSetBuilder::new();
builder.add(Glob::new(pattern).expect("Invalid glob pattern"));
let glob_set = builder.build().expect("Failed to compile glob set");
triggers.push(ParsedTrigger::OnFinishOf(Matchable::Glob(glob_set)));
}
}
"on_success" => triggers.push(ParsedTrigger::OnSuccess),
"on_failure" => triggers.push(ParsedTrigger::OnFailure),
_ => {
log::warn!("Unknown trigger key: {}", key);
}
}
}
triggers
}
}
}
fn merge_templates( fn make_tmpl(on_success: &str, on_failure: &str) -> NotificationTemplateDef {
method_template: &NotificationTemplate, NotificationTemplateDef {
policy_template: &Option<serde_yaml::Value>, use_: None,
) -> NotificationTemplate { on_success: if on_success.is_empty() {
let mut merged = method_template.clone(); None
if let Some(policy_tmpl) = policy_template
&& let Ok(policy_template) =
serde_yaml::from_value::<NotificationTemplate>(policy_tmpl.clone())
{
if policy_template.schema.is_some() {
merged.schema = policy_template.schema;
}
if policy_template.on_success.is_some() {
merged.on_success = policy_template.on_success;
}
if policy_template.on_failure.is_some() {
merged.on_failure = policy_template.on_failure;
}
if policy_template.on_finish_of.is_some() {
merged.on_finish_of = policy_template.on_finish_of;
}
}
merged
}
fn extract_overrides(overrides: &serde_yaml::Value) -> (bool, u32, std::time::Duration, f64) {
let fallible = overrides
.get("fallible")
.and_then(|v| v.as_bool())
.unwrap_or(NOTIFICATION_DEFAULT_FALLIBLE);
let max_lives = overrides
.get("max_lives")
.and_then(|v| v.as_u64())
.unwrap_or(NOTIFICATION_DEFAULT_MAX_LIVES as u64) as u32;
let timeout_ms = overrides
.get("timeout_ms")
.and_then(|v| v.as_u64())
.unwrap_or(NOTIFICATION_DEFAULT_TIMEOUT_MS);
let retry_backoff = overrides
.get("retry_backoff")
.and_then(|v| v.as_f64())
.unwrap_or(NOTIFICATION_DEFAULT_RETRY_BACKOFF);
(
fallible,
max_lives,
std::time::Duration::from_millis(timeout_ms),
retry_backoff,
)
}
fn parse_rules(config: &NotificationConfig) -> NotificationRulesMap {
let mut rules = NotificationRulesMap::default();
for (priority, group) in &config.groups {
let mut ruleset: HashMap<Method, Vec<NotificationRule>> = HashMap::new();
for (method_name, method) in group {
let mut method_rules: Vec<NotificationRule> = Vec::new();
if method.policy.is_empty() {
let trigger: Vec<ParsedTrigger> = method
.trigger
.iter()
.flat_map(parse_trigger_item_raw)
.collect();
let template = method.template.clone();
let (fallible, max_lives, timeout, retry_backoff) =
extract_overrides(&method.config);
method_rules.push(NotificationRule {
enabled: true,
trigger,
template,
fallible,
max_lives,
timeout,
config: method.config.clone(),
retry_backoff,
});
} else { } else {
for policy in &method.policy { Some(on_success.into())
let trigger: Vec<ParsedTrigger> = if policy.trigger.is_empty() { },
method on_failure: if on_failure.is_empty() {
.trigger None
.iter() } else {
.flat_map(parse_trigger_item_raw) Some(on_failure.into())
.collect() },
} else { on_finish_of: None,
policy
.trigger
.iter()
.flat_map(parse_trigger_item_raw)
.collect()
};
let template =
merge_templates(&method.template, &Some(policy.template.clone()));
let method_overrides = extract_overrides(&method.config);
let policy_overrides = extract_overrides(&policy.overrides);
let fallible = if policy.overrides.get("fallible").is_some() {
policy_overrides.0
} else {
method_overrides.0
};
let max_lives = if policy.overrides.get("max_lives").is_some() {
policy_overrides.1
} else {
method_overrides.1
};
let timeout = if policy.overrides.get("timeout_ms").is_some() {
policy_overrides.2
} else {
method_overrides.2
};
let retry_backoff = if policy.overrides.get("retry_backoff").is_some() {
policy_overrides.3
} else {
method_overrides.3
};
method_rules.push(NotificationRule {
enabled: true,
trigger,
template,
fallible,
max_lives,
timeout,
config: method.config.clone(),
retry_backoff,
});
}
}
ruleset.insert(method_name.clone(), method_rules);
} }
rules.insert(*priority, ruleset);
} }
rules
}
// calculate min(1, k/x)*t as notification grouping period fn make_tmpl_with_use(use_ref: &str) -> NotificationTemplateDef {
// to avoid excessive notification frequency NotificationTemplateDef {
fn wait_time(group_period: u32, group_watermark: u32, count: u32) -> f32 { use_: Some(use_ref.into()),
if count == 0 { on_success: None,
log::warn!("Unexpected notification count: {}", count); on_failure: None,
return group_period as f32; on_finish_of: None,
}
} }
f32::min(1.0, group_watermark as f32 / count as f32) * group_period as f32
}
// calculate p^(m-l)*t as notification delay // ═══════════════════════════════════════════
fn notify_delay(event: &NotificationEvent) -> std::time::Duration { // resolve_template_content
let exponent = (event.max_lives - event.lives) as i32; // ═══════════════════════════════════════════
if exponent < 0 {
log::warn!( #[test]
"Unexpected lives in notification: lives({}) > max_lives({})", fn resolve_none_cd_default_found() {
event.lives, let mut ws = HashMap::new();
event.max_lives ws.insert("mail".into(), make_tmpl("OK", "FAIL"));
); let templates = HashMap::new();
return std::time::Duration::from_secs_f64(NOTIFICATION_TEMPORARY_MAX_TIME); let result = resolve_template(&None, "mail", &templates, false, Some(&ws));
assert!(result.is_ok());
assert_eq!(result.unwrap().on_success.unwrap(), "OK");
} }
let multiplier = NOTIFICATION_TEMPORARY_DELAY_FACTOR.powi(exponent);
let delay = multiplier * NOTIFICATION_TEMPORARY_DELAY_TIME; #[test]
if !delay.is_finite() || delay < 0.0 { fn resolve_none_cd_default_missing_fallback() {
log::warn!( let ws: HashMap<String, NotificationTemplateDef> = HashMap::new();
"Unexpected calculated delay: {}, treating as infinite", let mut templates = HashMap::new();
delay templates.insert("mail.fallback".into(), make_tmpl("FALLBACK", ""));
); let result = resolve_template(&None, "mail", &templates, false, Some(&ws));
return std::time::Duration::from_secs_f64(NOTIFICATION_TEMPORARY_MAX_TIME); assert!(result.is_ok());
assert_eq!(result.unwrap().on_success.unwrap(), "FALLBACK");
}
#[test]
fn resolve_none_cd_no_default_no_fallback_error() {
let ws: HashMap<String, NotificationTemplateDef> = HashMap::new();
let templates = HashMap::new();
let result = resolve_template(&None, "mail", &templates, false, Some(&ws));
assert!(result.is_err());
assert!(format!("{}", result.unwrap_err()).contains("failed"));
}
#[test]
fn resolve_none_standalone_fallback() {
let mut templates = HashMap::new();
templates.insert("mail.fallback".into(), make_tmpl("STANDALONE", ""));
let result = resolve_template(&None, "mail", &templates, true, None);
assert!(result.is_ok());
assert_eq!(result.unwrap().on_success.unwrap(), "STANDALONE");
}
#[test]
fn resolve_none_standalone_no_fallback_error() {
let result = resolve_template(&None, "mail", &HashMap::new(), true, None);
assert!(result.is_err());
}
#[test]
fn resolve_custom() {
let result = resolve_template(&Some("custom".into()), "mail", &HashMap::new(), false, None);
assert!(result.is_ok());
// "custom" returns an empty default — caller reads rule.template.on_* directly
}
#[test]
fn resolve_default_standalone_fallback() {
let mut templates = HashMap::new();
templates.insert("mail.fallback".into(), make_tmpl("FB", ""));
let result = resolve_template(&Some("default".into()), "mail", &templates, true, None);
assert!(result.is_ok());
assert_eq!(result.unwrap().on_success.unwrap(), "FB");
}
#[test]
fn resolve_default_standalone_no_fallback_error() {
let result = resolve_template(&Some("default".into()), "mail", &HashMap::new(), true, None);
assert!(result.is_err());
}
#[test]
fn resolve_default_cd_wsbase() {
let mut ws = HashMap::new();
ws.insert("mail".into(), make_tmpl("WS_OK", ""));
let result = resolve_template(
&Some("default".into()),
"mail",
&HashMap::new(),
false,
Some(&ws),
);
assert!(result.is_ok());
assert_eq!(result.unwrap().on_success.unwrap(), "WS_OK");
}
#[test]
fn resolve_default_cd_no_wsbase_error() {
let result = resolve_template(
&Some("default".into()),
"mail",
&HashMap::new(),
false,
None,
);
assert!(result.is_err());
}
#[test]
fn resolve_named_found() {
let mut templates = HashMap::new();
templates.insert("sms-tmpl1".into(), make_tmpl("NAMED", ""));
let result = resolve_template(&Some("sms-tmpl1".into()), "mail", &templates, false, None);
assert!(result.is_ok());
assert_eq!(result.unwrap().on_success.unwrap(), "NAMED");
}
#[test]
fn resolve_named_not_found_method_match_fallback() {
let mut templates = HashMap::new();
templates.insert("mail.fallback".into(), make_tmpl("FB", ""));
let result = resolve_template(&Some("mail".into()), "mail", &templates, false, None);
assert!(result.is_ok());
assert_eq!(result.unwrap().on_success.unwrap(), "FB");
}
#[test]
fn resolve_named_not_found_no_match_error() {
let result = resolve_template(&Some("ghost".into()), "mail", &HashMap::new(), false, None);
assert!(result.is_err());
}
// ═══════════════════════════════════════════
// flatten_template_refs
// ═══════════════════════════════════════════
#[test]
fn flatten_no_refs_unchanged() {
let mut templates = HashMap::new();
templates.insert("a".into(), make_tmpl("A_OK", "A_FAIL"));
let result = flatten_template_refs(&mut templates);
assert!(result.is_ok());
assert_eq!(templates["a"].on_success.as_deref(), Some("A_OK"));
assert!(templates["a"].use_.is_none());
}
#[test]
fn flatten_single_ref_merged() {
let mut templates = HashMap::new();
templates.insert("base".into(), make_tmpl("BASE_OK", "BASE_FAIL"));
templates.insert("child".into(), make_tmpl_with_use("base"));
let result = flatten_template_refs(&mut templates);
assert!(result.is_ok());
assert_eq!(templates["child"].on_success.as_deref(), Some("BASE_OK"));
assert_eq!(templates["child"].on_failure.as_deref(), Some("BASE_FAIL"));
assert!(templates["child"].use_.is_none());
}
#[test]
fn flatten_child_overrides_parent() {
let mut templates = HashMap::new();
templates.insert("base".into(), make_tmpl("BASE_OK", "BASE_FAIL"));
templates.insert(
"child".into(),
NotificationTemplateDef {
use_: Some("base".into()),
on_success: Some("MY_OK".into()),
on_failure: None,
on_finish_of: None,
},
);
let result = flatten_template_refs(&mut templates);
assert!(result.is_ok());
assert_eq!(templates["child"].on_success.as_deref(), Some("MY_OK"));
assert_eq!(templates["child"].on_failure.as_deref(), Some("BASE_FAIL"));
}
#[test]
fn flatten_chain_a_b_c() {
let mut templates = HashMap::new();
templates.insert("c".into(), make_tmpl("C_OK", "C_FAIL"));
templates.insert("b".into(), make_tmpl_with_use("c"));
templates.insert("a".into(), make_tmpl_with_use("b"));
let result = flatten_template_refs(&mut templates);
assert!(result.is_ok());
assert_eq!(templates["a"].on_success.as_deref(), Some("C_OK"));
assert_eq!(templates["b"].on_success.as_deref(), Some("C_OK"));
}
#[test]
fn flatten_cycle_detected() {
let mut templates = HashMap::new();
templates.insert("x".into(), make_tmpl_with_use("y"));
templates.insert("y".into(), make_tmpl_with_use("x"));
let result = flatten_template_refs(&mut templates);
assert!(result.is_err());
match result.unwrap_err() {
NotifyError::TemplateLoop(_) => {}
other => panic!("expected TemplateLoop, got {:?}", other),
}
}
#[test]
fn flatten_self_ref_detected() {
let mut templates = HashMap::new();
templates.insert("self".into(), make_tmpl_with_use("self"));
let result = flatten_template_refs(&mut templates);
assert!(result.is_err());
match result.unwrap_err() {
NotifyError::TemplateLoop(_) => {}
other => panic!("expected TemplateLoop, got {:?}", other),
}
}
#[test]
fn flatten_nonexistent_ref_error() {
let mut templates = HashMap::new();
templates.insert("orphan".into(), make_tmpl_with_use("ghost"));
let result = flatten_template_refs(&mut templates);
assert!(result.is_err());
}
#[test]
fn flatten_external_url_preserved() {
let mut templates = HashMap::new();
templates.insert(
"ext".into(),
NotificationTemplateDef {
use_: Some("git://example.com/repo@v1".into()),
on_success: None,
on_failure: None,
on_finish_of: None,
},
);
let result = flatten_template_refs(&mut templates);
assert!(result.is_ok());
// External URLs are NOT flattened — use_ preserved
assert_eq!(
templates["ext"].use_.as_deref(),
Some("git://example.com/repo@v1")
);
}
// ═══════════════════════════════════════════
// wait_time
// ═══════════════════════════════════════════
#[test]
fn wait_time_zero_count() {
let w = wait_time(30, 5, 0);
assert_eq!(w, 30.0);
}
#[test]
fn wait_time_at_watermark() {
let w = wait_time(30, 5, 5);
assert_eq!(w, 30.0);
}
#[test]
fn wait_time_below_watermark() {
let w = wait_time(30, 5, 2);
assert!((w - 30.0).abs() < 0.01);
}
#[test]
fn wait_time_above_watermark() {
let w = wait_time(30, 5, 10);
assert!((w - 15.0).abs() < 0.01);
}
#[test]
fn wait_time_far_above_watermark() {
let w = wait_time(30, 5, 100);
assert!((w - 1.5).abs() < 0.01);
}
// ═══════════════════════════════════════════
// notify_delay
// ═══════════════════════════════════════════
fn make_event(count: u32, retry_backoff: f64) -> NotificationEvent {
NotificationEvent {
id: uuid::Uuid::new_v4(),
stage: crate::types::buildstatus::StagePhase::Bake,
substage: "build".into(),
outcome: crate::types::buildstatus::StageOutcome::PipelineSuccess,
not_before: chrono::Utc::now(),
priority: 0,
effective_priority: 0,
attempted: 0,
retry_backoff,
target: NotificationTarget {
priority: 0,
rules: None,
count,
},
}
}
#[test]
fn notify_delay_default_factor() {
let event = make_event(1, 2.0);
let delay = notify_delay(&event);
assert!(delay.as_secs_f64() > 0.0);
}
#[test]
fn notify_delay_larger_factor() {
let event1 = make_event(1, 2.0);
let event2 = make_event(1, 4.0);
let d1 = notify_delay(&event1).as_secs_f64();
let d2 = notify_delay(&event2).as_secs_f64();
assert!(d2 > d1, "larger retry_backoff should produce longer delay");
}
// ═══════════════════════════════════════════
// find_rule
// ═══════════════════════════════════════════
#[test]
fn find_rule_found() {
use crate::finalize::NotificationTemplate;
let rule = NotificationRule {
enabled: true,
trigger: vec![],
template: NotificationTemplate {
schema: None,
on_success: None,
on_failure: None,
on_finish_of: None,
},
fallible: false,
max_lives: 3,
timeout: std::time::Duration::from_secs(30),
config: serde_yaml::Value::Null,
retry_backoff: 2.0,
};
let mut rules_map: NotificationRulesMap = HashMap::new();
rules_map.insert(0, {
let mut m: HashMap<String, Vec<NotificationRule>> = HashMap::new();
m.insert("mail".into(), vec![rule]);
m
});
let result = find_rule(&rules_map, "mail", 0);
assert!(result.is_ok());
}
#[test]
fn find_rule_not_found() {
let rules_map: NotificationRulesMap = HashMap::new();
let result = find_rule(&rules_map, "mail", 0);
assert!(result.is_err());
} }
std::time::Duration::from_secs_f64(delay)
} }
+13 -1
View File
@@ -1,4 +1,4 @@
use crate::notify::NotificationRenderer; use crate::notify::types::NotificationRenderer;
use thiserror::Error; use thiserror::Error;
#[derive(Error, Debug)] #[derive(Error, Debug)]
@@ -23,4 +23,16 @@ pub enum NotifyError {
#[error("Unsupported renderer for {0}: {1}")] #[error("Unsupported renderer for {0}: {1}")]
UnsupportedRenderer(String, NotificationRenderer), UnsupportedRenderer(String, NotificationRenderer),
#[error("Schema error: {0}")]
SchemaError(String),
#[error("Template render error: {0}")]
TemplateRenderError(String),
#[error("Circular template reference: {0}")]
TemplateLoop(String),
#[error("Template reference depth exceeded (max {max}): {name}")]
TemplateDepthExceeded { name: String, max: u32 },
} }
+345
View File
@@ -0,0 +1,345 @@
use std::collections::HashMap;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex};
use std::time::Duration;
use chrono::Utc;
use tokio::sync::Notify;
use crate::finalize::NotificationTemplateDef;
use crate::notify::template::{
build_batch_template_context, render_template, resolve_template,
select_template_str,
};
use crate::notify::types::{
MethodStatus, NotificationEvent,
NotificationEventSender, NotificationPluginMap, NotificationRenderer,
NotificationRulesMap,
};
use crate::notify::util::{
find_rule, get_fallible, is_temporary_error, notify_delay, wait_time,
};
/// Shared state for a single dispatch bucket.
struct BucketEntry {
events: Arc<Mutex<Vec<NotificationEvent>>>,
result: Arc<Mutex<Option<(MethodStatus, Vec<NotificationEvent>)>>>,
done: Arc<AtomicBool>,
notify: Arc<Notify>,
method: String,
rule_id: usize,
_handle: tokio::task::JoinHandle<()>,
}
/// Concurrent notification handler with bucket-based dispatch.
///
/// Events are routed to matching buckets. Buckets cool down and dispatch
/// plugins concurrently. The caller awaits bucket completion, then handles
/// retry/escalation decisions.
pub struct NotificationHandler {
buckets: Mutex<HashMap<(String, usize), BucketEntry>>,
pub config: Arc<NotificationRulesMap>,
plugins: Arc<NotificationPluginMap>,
templates: Arc<HashMap<String, NotificationTemplateDef>>,
ctx: Arc<workshop_engine::ExecutionContext>,
retry_tx: NotificationEventSender,
period: u32,
watermark: u32,
}
impl NotificationHandler {
#[allow(clippy::too_many_arguments)]
pub fn new(
config: Arc<NotificationRulesMap>,
plugins: Arc<NotificationPluginMap>,
templates: Arc<HashMap<String, NotificationTemplateDef>>,
ctx: Arc<workshop_engine::ExecutionContext>,
retry_tx: NotificationEventSender,
period: u32,
watermark: u32,
) -> Self {
NotificationHandler {
buckets: Mutex::new(HashMap::new()),
config,
plugins,
templates,
ctx,
retry_tx,
period,
watermark,
}
}
/// Process a single event: route to buckets, await completion,
/// then handle retry/escalation for each bucket result.
pub async fn process_event(self: Arc<Self>, event: NotificationEvent) {
let entries = self.route_event(&event);
if entries.is_empty() {
return;
}
// Await all buckets — yields CPU
for (_, notify) in &entries {
notify.notified().await;
}
// Collect results and remove entries
let mut results: Vec<(String, usize, Option<(MethodStatus, Vec<NotificationEvent>)>)> =
Vec::new();
{
let mut buckets = self.buckets.lock().unwrap();
for (key, _) in &entries {
if let Some(entry) = buckets.remove(key) {
let r = entry.result.lock().unwrap().take();
results.push((entry.method, entry.rule_id, r));
}
}
}
// Handle retry/escalation per method
for (method, _rule_id, result) in results {
let Some((status, mut events)) = result else { continue };
match &status {
MethodStatus::Success => {
log::debug!("[NOTIFY] {}: success ({} events)", method, events.len());
}
MethodStatus::TemporaryError(_) => {
log::warn!("[NOTIFY] {}: temporary error, retrying", method);
for event in events.iter_mut() {
event.target.rules = Some(vec![method.clone()]);
event.target.count += 1;
event.not_before = Utc::now() + notify_delay(event);
let _ = self.retry_tx.send(event.clone()).await;
}
}
MethodStatus::PermanentError(_) | MethodStatus::Exhausted(_) => {
let fallible = get_fallible(&self.config, &method);
if fallible {
log::error!("[NOTIFY] {}: discard (fallible)", method);
} else {
log::warn!("[NOTIFY] {}: escalating priority", method);
for event in events.iter_mut() {
event.target.priority += 1;
event.target.rules = None;
event.attempted = event.target.count;
event.target.count = 0;
event.effective_priority = event.priority;
let _ = self.retry_tx.send(event.clone()).await;
}
}
}
}
}
}
/// Route a single event to all matching buckets.
///
/// Returns Notify handles for awaiting bucket completion.
fn route_event(
&self,
event: &NotificationEvent,
) -> Vec<((String, usize), Arc<Notify>)> {
let mut matches: Vec<(String, usize)> = Vec::new();
for ruleset in self.config.values() {
for (method, rules) in ruleset.iter() {
if let Some(ref method_rules) = event.target.rules {
if !method_rules.contains(method) {
continue;
}
}
for (rule_id, rule) in rules.iter().enumerate() {
if !rule.enabled {
continue;
}
if rule.accept(&event.outcome, &event.stage, &event.substage) {
matches.push((method.clone(), rule_id));
}
}
}
}
let mut buckets = self.buckets.lock().unwrap();
let mut result: Vec<((String, usize), Arc<Notify>)> = Vec::new();
for key in matches {
if let Some(entry) = buckets.get(&key) {
if entry.done.load(Ordering::Acquire) {
buckets.remove(&key);
}
}
if let Some(entry) = buckets.get(&key) {
entry.events.lock().unwrap().push(event.clone());
result.push((key, entry.notify.clone()));
} else {
let (entry, notify) = Self::spawn_bucket(
&key.0,
key.1,
&self.config,
&self.plugins,
&self.templates,
&self.ctx,
self.period,
self.watermark,
);
entry.events.lock().unwrap().push(event.clone());
result.push((key.clone(), notify));
buckets.insert(key, entry);
}
}
result
}
/// Spawn a bucket task: cooldown → take → dispatch → store → notify.
fn spawn_bucket(
method: &str,
rule_id: usize,
config: &Arc<NotificationRulesMap>,
plugins: &Arc<NotificationPluginMap>,
templates: &Arc<HashMap<String, NotificationTemplateDef>>,
ctx: &Arc<workshop_engine::ExecutionContext>,
period: u32,
watermark: u32,
) -> (BucketEntry, Arc<Notify>) {
let events: Arc<Mutex<Vec<NotificationEvent>>> = Arc::new(Mutex::new(Vec::new()));
let result: Arc<Mutex<Option<(MethodStatus, Vec<NotificationEvent>)>>> =
Arc::new(Mutex::new(None));
let notify = Arc::new(Notify::new());
let done = Arc::new(AtomicBool::new(false));
let ec = events.clone();
let rc = result.clone();
let nc = notify.clone();
let dc = done.clone();
let m = method.to_string();
let c = config.clone();
let p = plugins.clone();
let t = templates.clone();
let x = ctx.clone();
let handle = tokio::spawn(async move {
// Phase 1: Cooldown
let count = ec.lock().unwrap().len() as u32;
let wait_secs = wait_time(period, watermark, count);
tokio::time::sleep(Duration::from_secs_f32(wait_secs)).await;
// Phase 2: Take events
let batch = std::mem::take(&mut *ec.lock().unwrap());
if batch.is_empty() {
dc.store(true, Ordering::Release);
nc.notify_waiters();
return;
}
// Phase 3: Dispatch
let rule = match find_rule(&c, &m, rule_id) {
Ok(r) => r,
Err(_) => {
*rc.lock().unwrap() = Some((MethodStatus::PermanentError("rule not found".into()), batch));
dc.store(true, Ordering::Release);
nc.notify_waiters();
return;
}
};
let plugin = match p.get(&m) {
Some(p) => p,
None => {
*rc.lock().unwrap() = Some((MethodStatus::PermanentError("plugin not found".into()), batch));
dc.store(true, Ordering::Release);
nc.notify_waiters();
return;
}
};
let renderer = plugin.metadata.renderer.clone().unwrap_or_default();
let argument = build_plugin_argument(&renderer, rule, &m, &t, &x, &batch);
let status = match argument {
Some(arg) => {
match tokio::time::timeout(rule.timeout, plugin.call(arg, &x, &None)).await {
Ok(Ok(())) => MethodStatus::Success,
Ok(Err(e)) if is_temporary_error(&e) => {
MethodStatus::TemporaryError(format!("{}", e))
}
Ok(Err(e)) => MethodStatus::PermanentError(format!("{}", e)),
Err(_) => MethodStatus::TemporaryError("timeout".into()),
}
}
None => MethodStatus::PermanentError("argument build failed".into()),
};
*rc.lock().unwrap() = Some((status, batch));
dc.store(true, Ordering::Release);
nc.notify_waiters();
});
let entry = BucketEntry {
events,
result,
done,
notify: notify.clone(),
method: method.to_string(),
rule_id,
_handle: handle,
};
(entry, notify)
}
}
/// Build the YAML argument for a plugin call based on renderer mode.
fn build_plugin_argument(
renderer: &NotificationRenderer,
rule: &crate::notify::types::NotificationRule,
method: &str,
templates: &HashMap<String, NotificationTemplateDef>,
ctx: &workshop_engine::ExecutionContext,
events: &[NotificationEvent],
) -> Option<serde_yaml::Value> {
let first = events.first()?;
match renderer {
NotificationRenderer::Server => {
let template_def = resolve_template(
&rule.template.schema,
method,
templates,
ctx.standalone,
None,
)
.ok()?;
let template_str = select_template_str(&template_def, &first.outcome)
.or_else(|| rule.template.on_success.as_deref())
.or_else(|| rule.template.on_failure.as_deref())
.or_else(|| {
rule.template
.on_finish_of
.as_ref()
.map(|o| o.content.as_str())
})?;
let data = build_batch_template_context(ctx, events);
let rendered = render_template(template_str, &data).ok()?;
let mut argument = rule.config.clone();
if let serde_yaml::Value::Mapping(ref mut map) = argument {
map.insert(
serde_yaml::Value::String("_rendered_content".into()),
serde_yaml::Value::String(rendered),
);
}
log::info!("SSR: rendered content for method '{}'", method);
Some(argument)
}
NotificationRenderer::Plugin => {
unimplemented!("Plugin renderer is not yet implemented")
}
NotificationRenderer::Interactive => {
unimplemented!("Interactive renderer is not yet implemented")
}
}
}
+86
View File
@@ -0,0 +1,86 @@
use crate::notify::types::{NotificationEvent, NotificationEventReceiver, NotificationEventSender};
use std::collections::BinaryHeap;
/// Priority queue for notification events.
///
/// Integrates with `notify()`:
/// - `tx` sends events to notify() for processing
/// - `rx` receives retry events back from notify()
pub struct NotificationQueue {
pub pq: BinaryHeap<NotificationEvent>,
pub tx: NotificationEventSender,
pub rx: NotificationEventReceiver,
}
impl NotificationQueue {
pub fn new(tx: NotificationEventSender, rx: NotificationEventReceiver) -> Self {
Self {
pq: BinaryHeap::new(),
tx,
rx,
}
}
/// Push an event into the priority queue.
pub fn enqueue(&mut self, event: NotificationEvent) {
self.pq.push(event);
}
/// Non-blocking single round: recycle retry events into PQ, then
/// pop and send all ready events (not_before ≤ now).
pub async fn flush_now(&mut self) -> anyhow::Result<()> {
// Recycle retry events
while let Ok(event) = self.rx.try_recv() {
self.pq.push(event);
}
// Send ready events
let now = chrono::Utc::now();
while let Some(peek) = self.pq.peek() {
if peek.not_before > now {
break;
}
let event = self.pq.pop().unwrap();
self.tx.send(event).await?;
}
Ok(())
}
/// Blocking drain until all events are processed.
/// For standalone mode: called after prebake/bake/finalize to wait for completion.
/// For c/d mode: the main execution loop.
pub async fn drain(&mut self) -> anyhow::Result<()> {
loop {
// Drain retry channel
while let Ok(event) = self.rx.try_recv() {
self.pq.push(event);
}
// Determine sleep deadline
let until = self
.pq
.peek()
.map(|e| e.not_before)
.unwrap_or_else(|| chrono::Utc::now() + chrono::Duration::seconds(3600));
let std_until = (until - chrono::Utc::now())
.to_std()
.unwrap_or(std::time::Duration::from_secs(1));
tokio::select! {
event = self.rx.recv() => {
match event {
Some(event) => self.pq.push(event),
None => break Ok(()),
}
}
_ = tokio::time::sleep(std_until) => {
let now = chrono::Utc::now();
while let Some(peek) = self.pq.peek() {
if peek.not_before > now { break; }
let event = self.pq.pop().unwrap();
self.tx.send(event).await?;
}
}
}
}
}
}
+148
View File
@@ -0,0 +1,148 @@
use crate::finalize::NotificationConfig;
use crate::finalize::NotificationTemplate;
use crate::finalize::config::{NotifyPolicy, TriggerItemRaw};
use crate::notify::types::{DEFAULT_NOTIFY_PARAMS, Matchable, Method, NotificationRule, NotificationRulesMap, ParsedTrigger};
use globset::{Glob, GlobSetBuilder};
use std::collections::HashMap;
/// Parse a single `on_finish_of` pattern string into a ParsedTrigger, or None.
/// Supports `re:` prefix for regex patterns, otherwise treated as glob.
fn parse_on_finish_of(pattern: &str) -> Option<ParsedTrigger> {
let pattern = pattern.trim();
if let Some(re_str) = pattern.strip_prefix("re:") {
regex::Regex::new(re_str).ok().map(|r| ParsedTrigger::OnFinishOf(Matchable::Regex(r)))
} else {
Glob::new(pattern).ok().and_then(|g| {
let mut builder = GlobSetBuilder::new();
builder.add(g);
builder.build().ok()
}).map(|gs| ParsedTrigger::OnFinishOf(Matchable::Glob(gs)))
}
}
fn parse_trigger_item_raw(item: &TriggerItemRaw) -> Vec<ParsedTrigger> {
// Normalize Simple → single-entry Vec, then unify both branches
let entries: Vec<(&str, Vec<&str>)> = match item {
TriggerItemRaw::Simple(s) => {
if let Some(pattern) = s.strip_prefix("on_finish_of:") {
vec![("on_finish_of", vec![pattern.trim()])]
} else {
vec![(s.as_str(), vec![])]
}
}
TriggerItemRaw::Map(map) => map.iter().map(|(k, v)|
(k.as_str(), v.iter().map(|s| s.as_str()).collect())
).collect(),
};
let mut triggers = Vec::new();
for (key, patterns) in entries {
match key {
"on_finish_of" => {
for pattern in patterns {
if let Some(t) = parse_on_finish_of(pattern) {
triggers.push(t);
}
}
}
"on_success" => triggers.push(ParsedTrigger::OnSuccess),
"on_failure" => triggers.push(ParsedTrigger::OnFailure),
_ => log::warn!("Unknown trigger key: {}", key),
}
}
triggers
}
fn merge_templates(
method_template: &NotificationTemplate,
policy_template: &Option<serde_yaml::Value>,
) -> NotificationTemplate {
let mut merged = method_template.clone();
if let Some(policy_tmpl) = policy_template
&& let Ok(policy) = serde_yaml::from_value::<NotificationTemplate>(policy_tmpl.clone())
{
merged += policy;
}
merged
}
struct RuleOverrides {
fallible: bool,
max_lives: u32,
timeout: std::time::Duration,
retry_backoff: f64,
}
impl RuleOverrides {
fn from_value(v: &serde_yaml::Value) -> Self {
RuleOverrides {
fallible: v.get("fallible").and_then(|v| v.as_bool()).unwrap_or(false),
max_lives: v.get("max_lives").and_then(|v| v.as_u64()).unwrap_or(DEFAULT_NOTIFY_PARAMS.max_lives.value as u64) as u32,
timeout: std::time::Duration::from_millis(
v.get("timeout_ms").and_then(|v| v.as_u64()).unwrap_or(DEFAULT_NOTIFY_PARAMS.timeout_ms.value)
),
retry_backoff: v.get("retry_backoff").and_then(|v| v.as_f64()).unwrap_or(DEFAULT_NOTIFY_PARAMS.retry_backoff.value),
}
}
/// Chain: apply `source` atop existing values (source wins when present).
fn apply(self, source: &serde_yaml::Value) -> Self {
RuleOverrides {
fallible: source.get("fallible").and_then(|v| v.as_bool()).unwrap_or(self.fallible),
max_lives: source.get("max_lives").and_then(|v| v.as_u64()).unwrap_or(self.max_lives as u64) as u32,
timeout: source.get("timeout_ms").and_then(|v| v.as_u64()).map(std::time::Duration::from_millis).unwrap_or(self.timeout),
retry_backoff: source.get("retry_backoff").and_then(|v| v.as_f64()).unwrap_or(self.retry_backoff),
}
}
}
pub fn parse_rules(config: &NotificationConfig) -> NotificationRulesMap {
log::info!("[NOTIFY] parse_rules: groups={}", config.groups.len());
let mut rules = NotificationRulesMap::default();
for (priority, group) in &config.groups {
let mut ruleset: HashMap<Method, Vec<NotificationRule>> = HashMap::new();
for (method_name, method) in group {
let mut method_rules = Vec::new();
// Unify: no policy → one empty policy (same merge path)
let empty_policy = NotifyPolicy {
trigger: vec![],
template: serde_yaml::Value::Null,
overrides: serde_yaml::Value::Null,
};
let policies: &[NotifyPolicy] = if method.policy.is_empty() {
std::slice::from_ref(&empty_policy)
} else {
&method.policy
};
for policy in policies {
let trigger: Vec<ParsedTrigger> = if policy.trigger.is_empty() {
method.trigger.iter().flat_map(parse_trigger_item_raw).collect()
} else {
policy.trigger.iter().flat_map(parse_trigger_item_raw).collect()
};
let template =
merge_templates(&method.template, &Some(policy.template.clone()));
let vals = RuleOverrides::from_value(&serde_yaml::Value::Null)
.apply(&method.config)
.apply(&policy.overrides);
method_rules.push(NotificationRule {
enabled: true,
trigger,
template,
fallible: vals.fallible,
max_lives: vals.max_lives,
timeout: vals.timeout,
config: method.config.clone(),
retry_backoff: vals.retry_backoff,
});
}
ruleset.insert(method_name.clone(), method_rules);
}
rules.insert(*priority, ruleset);
}
rules
}
+281
View File
@@ -0,0 +1,281 @@
use std::collections::HashMap;
use minijinja::Environment;
use workshop_engine::ExecutionContext;
use crate::finalize::NotificationTemplateDef;
use crate::notify::error::NotifyError;
use crate::notify::types::NotificationEvent;
use crate::types::buildstatus::StageOutcome;
use crate::types::resource::ResourceRegistry;
/// Maximum depth for template cross-references (prevent infinite recursion)
pub const TEMPLATE_REFERENCE_DEPTH: u32 = 10;
/// Build template context for a batch of notification events.
pub fn build_batch_template_context(
ctx: &ExecutionContext,
events: &[NotificationEvent],
) -> HashMap<String, serde_json::Value> {
let mut data = HashMap::new();
data.insert("pipeline".into(), serde_json::json!({
"name": ctx.pipeline_name,
}));
data.insert("count".into(), serde_json::json!(events.len()));
let event_list: Vec<_> = events
.iter()
.map(|e| {
serde_json::json!({
"stage": format!("{}.{}", e.stage.as_str(), e.substage),
"status": e.outcome.to_string(),
})
})
.collect();
data.insert("events".into(), serde_json::json!(event_list));
// Single event compatibility: provide direct access
if events.len() == 1 {
let e = &events[0];
data.insert("build".into(), serde_json::json!({
"id": ctx.task_id,
"status": e.outcome.to_string(),
"stage": format!("{}.{}", e.stage.as_str(), e.substage),
"duration": "0s",
"url": "",
}));
} else {
data.insert("build".into(), serde_json::json!({
"id": ctx.task_id,
"status": "batched",
"stage": "batch",
"duration": "0s",
"url": "",
}));
}
data.insert("commit".into(), serde_json::json!({
"hash": "",
"short_hash": "",
"message": "",
"author": "",
"author_email": "",
}));
data
}
/// Build template context for a single notification event (convenience wrapper).
pub fn build_template_context(
ctx: &ExecutionContext,
event: &NotificationEvent,
) -> HashMap<String, serde_json::Value> {
build_batch_template_context(ctx, std::slice::from_ref(event))
}
/// Render a minijinja template string with the provided data.
pub fn render_template(
template: &str,
data: &HashMap<String, serde_json::Value>,
) -> Result<String, String> {
let env = Environment::new();
let tmpl = env
.template_from_str(template)
.map_err(|e| format!("Template parse error: {}", e))?;
let result = tmpl
.render(data)
.map_err(|e| format!("Template render error: {}", e))?;
Ok(result)
}
/// Select template string from NotificationTemplateDef based on outcome.
pub fn select_template_str<'a>(
tmpl: &'a NotificationTemplateDef,
outcome: &StageOutcome,
) -> Option<&'a str> {
match outcome {
StageOutcome::PipelineSuccess => tmpl.on_success.as_deref(),
StageOutcome::PipelineFailure => tmpl.on_failure.as_deref(),
_ => None,
}
}
/// Resolve external template references from the resource registry.
pub fn resolve_external_template_refs(
templates: &mut HashMap<String, NotificationTemplateDef>,
registry: &ResourceRegistry,
) {
for (name, tmpl_def) in templates.iter_mut() {
if tmpl_def.use_.is_none() {
continue;
}
let Some(path) = registry.resolve(name) else {
continue;
};
let Ok(content) = std::fs::read_to_string(path.join("template.yml"))
.or_else(|_| std::fs::read_to_string(path.join("template.yaml")))
else {
continue;
};
let Ok(fetched) =
serde_yaml::from_str::<NotificationTemplateDef>(&content)
else {
continue;
};
*tmpl_def = fetched.apply(std::mem::take(tmpl_def));
tmpl_def.use_ = None;
}
}
/// Resolve a schema reference into a NotificationTemplateDef by walking a fallback chain.
pub fn resolve_template(
schema: &Option<String>,
method_name: &str,
templates: &HashMap<String, NotificationTemplateDef>,
_standalone: bool,
server_templates: Option<&HashMap<String, NotificationTemplateDef>>,
) -> Result<NotificationTemplateDef, NotifyError> {
// 1. Try the initial schema directly
if let Some(schema_name) = schema.as_deref() {
if let Some(template) =
resolve_template_by_schema(schema_name, method_name, templates, server_templates)
{
return Ok(template);
}
}
// 2. Walk the fallback chain
let mut schema_next = resolve_template_next_schema(schema);
while let Some(ref schema_name) = schema_next {
if let Some(template) =
resolve_template_by_schema(schema_name, method_name, templates, server_templates)
{
return Ok(template);
}
log::warn!("Failed to resolve template for schema {}.", schema_name);
schema_next = resolve_template_next_schema(&schema_next);
}
Err(NotifyError::SchemaError(format!("All template resolution attempts beginning from {} failed.", schema.as_deref().unwrap_or("[system default]"))))
}
/// Return the next schema name in the fallback chain, or None to terminate.
pub fn resolve_template_next_schema(schema: &Option<String>) -> Option<String> {
match schema.as_deref() {
Some("custom") => None,
Some("default") => Some("fallback".to_string()),
Some("fallback") => None,
Some(_) => Some("default".to_string()),
None => Some("default".to_string()),
}
}
/// Perform a single lookup for a given schema name. Returns None if not found.
pub fn resolve_template_by_schema(
schema: &str,
method_name: &str,
templates: &HashMap<String, NotificationTemplateDef>,
server_templates: Option<&HashMap<String, NotificationTemplateDef>>,
) -> Option<NotificationTemplateDef> {
match schema {
"custom" => Some(NotificationTemplateDef::default()),
"fallback" => templates.get(&format!("{}.fallback", method_name)).cloned(),
"default" => server_templates.and_then(|b| b.get(method_name)).cloned(),
name => templates.get(name).cloned(),
}
}
/// Flatten all local template cross-references (use: "other-name").
pub fn flatten_template_refs(
templates: &mut HashMap<String, NotificationTemplateDef>,
) -> Result<(), NotifyError> {
let names: Vec<String> = templates.keys().cloned().collect();
for name in names {
let mut visited = std::collections::HashSet::new();
resolve_chain(&name, templates, 0, &mut visited)?;
}
Ok(())
}
fn resolve_chain(
name: &str,
templates: &mut HashMap<String, NotificationTemplateDef>,
depth: u32,
visited: &mut std::collections::HashSet<String>,
) -> Result<(), NotifyError> {
if depth > TEMPLATE_REFERENCE_DEPTH {
return Err(NotifyError::TemplateDepthExceeded {
name: name.to_string(),
max: TEMPLATE_REFERENCE_DEPTH,
});
}
if !visited.insert(name.to_string()) {
return Err(NotifyError::TemplateLoop(name.to_string()));
}
// Only process local references (no ://)
if let Some(ref_name) = templates.get(name).and_then(|t| t.use_.clone()) {
if !ref_name.contains("://") {
if !templates.contains_key(&ref_name) {
return Err(NotifyError::SchemaError(format!(
"template '{}' references non-existent template '{}'",
name, ref_name
)));
}
resolve_chain(&ref_name, templates, depth + 1, visited)?;
// Merge: base provides defaults, referencing template overrides
let base = templates[&ref_name].clone();
let template = templates.get_mut(name).unwrap();
*template = std::mem::take(template).apply(base);
template.use_ = None;
}
}
visited.remove(name);
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_render_template_simple() {
let template = "Hello {{ name }}!";
let mut data = HashMap::new();
data.insert(
"name".to_string(),
serde_json::Value::String("World".to_string()),
);
let result = render_template(template, &data).unwrap();
assert_eq!(result, "Hello World!");
}
#[test]
fn test_render_template_with_build_status() {
let template = "Build {{ build.status }} for pipeline {{ pipeline.name }}";
let mut data = HashMap::new();
data.insert(
"build".to_string(),
serde_json::json!({ "status": "success" }),
);
data.insert(
"pipeline".to_string(),
serde_json::json!({ "name": "test-pipeline" }),
);
let result = render_template(template, &data).unwrap();
assert_eq!(result, "Build success for pipeline test-pipeline");
}
#[test]
fn test_render_template_invalid_syntax() {
let template = "{{ Unterminated";
let data = HashMap::new();
let result = render_template(template, &data);
assert!(result.is_err());
}
}
+56 -22
View File
@@ -7,7 +7,6 @@ use regex::Regex;
use serde::Deserialize; use serde::Deserialize;
use serde::Serialize; use serde::Serialize;
use std::collections::HashMap; use std::collections::HashMap;
use std::collections::HashSet;
use std::fmt::Display; use std::fmt::Display;
use std::hash::Hash; use std::hash::Hash;
use std::time::Duration; use std::time::Duration;
@@ -78,12 +77,39 @@ impl ParsedTrigger {
impl NotificationRule { impl NotificationRule {
pub fn accept(&self, result: &StageOutcome, stage: &StagePhase, substage: &str) -> bool { pub fn accept(&self, result: &StageOutcome, stage: &StagePhase, substage: &str) -> bool {
if self.trigger.is_empty() {
eprintln!("[ACCEPT] trigger empty, returning true");
return true;
}
self.trigger self.trigger
.iter() .iter()
.any(|t| t.accept(result, stage, substage)) .any(|t| t.accept(result, stage, substage))
} }
} }
/// Status of a single notification method after plugin execution.
#[derive(Debug)]
pub enum MethodStatus {
Success,
TemporaryError(String),
PermanentError(String),
Exhausted(String),
}
/// Tracks which notification methods still need attention within a priority group.
#[derive(Debug, Clone, Serialize)]
pub struct NotificationTarget {
pub priority: u32,
pub rules: Option<Vec<String>>,
pub count: u32,
}
impl Default for NotificationTarget {
fn default() -> Self {
Self { priority: 0, rules: None, count: 0 }
}
}
pub type Priority = u32; pub type Priority = u32;
pub type Method = String; pub type Method = String;
pub type NotificationRulesMap = HashMap<Priority, HashMap<Method, Vec<NotificationRule>>>; pub type NotificationRulesMap = HashMap<Priority, HashMap<Method, Vec<NotificationRule>>>;
@@ -101,9 +127,9 @@ pub struct NotificationEvent {
pub not_before: DateTime<Utc>, pub not_before: DateTime<Utc>,
pub priority: u32, pub priority: u32,
pub effective_priority: u32, pub effective_priority: u32,
pub lives: u32, pub retry_backoff: f64, // Exponential backoff factor
pub max_lives: u32, pub attempted: u32, // Cumulative cross-group retry count
pub target: HashSet<String>, // Target audience pub target: NotificationTarget, // Current audience
} }
impl Hash for NotificationEvent { impl Hash for NotificationEvent {
@@ -137,25 +163,11 @@ impl Ord for NotificationEvent {
} }
} }
/// Delay before retrying an unavailable plugin (5 seconds) use std::sync::LazyLock;
pub const NOTIFICATION_NOTREADY_DELAY: std::time::Duration = std::time::Duration::from_secs(5); use workshop_baker_params::{NotificationParams, MergeParams};
/// Exponential backoff base (2.0 = double each retry) /// Default parameters used when no external config is provided.
pub const NOTIFICATION_TEMPORARY_DELAY_FACTOR: f64 = 2.0; pub static DEFAULT_NOTIFY_PARAMS: LazyLock<NotificationParams> = LazyLock::new(|| NotificationParams::defaults());
/// Initial backoff delay for temporary errors (5 seconds)
pub const NOTIFICATION_TEMPORARY_DELAY_TIME: f64 = 5.0;
/// Upper bound for backoff delay (unbounded)
pub const NOTIFICATION_TEMPORARY_MAX_TIME: f64 = f64::MAX;
/// Lives recovered after downgrade as fraction of max_lives (0.0 = none)
pub const NOTIFICATION_LIFE_RECOVERY_FACTOR: f64 = 0.0;
pub const NOTIFICATION_DEFAULT_FALLIBLE: bool = false;
pub const NOTIFICATION_DEFAULT_MAX_LIVES: u32 = 3;
pub const NOTIFICATION_DEFAULT_TIMEOUT_MS: u64 = 30000;
pub const NOTIFICATION_DEFAULT_RETRY_BACKOFF: f64 = 2.0;
#[derive(Default, Debug, Deserialize, Clone)] #[derive(Default, Debug, Deserialize, Clone)]
pub enum NotificationRenderer { pub enum NotificationRenderer {
@@ -174,3 +186,25 @@ impl Display for NotificationRenderer {
} }
} }
} }
/// Notification batching configuration.
/// Controls how notification events are grouped before dispatch.
#[derive(Debug, Clone)]
pub struct BatchConfig {
/// Whether batching is enabled
pub enabled: bool,
/// Group period in seconds (the maximum wait time for a batch)
pub period: u32,
/// Group watermark threshold (the minimum number of events before batching)
pub watermark: u32,
}
impl Default for BatchConfig {
fn default() -> Self {
Self {
enabled: true,
period: 30,
watermark: 5,
}
}
}

Some files were not shown because too many files have changed in this diff Show More