Compare commits
8 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| df5026cdbe | |||
| d1ad08ef3a | |||
| cf2968e720 | |||
| 28abc5d207 | |||
| 6e7b96cd66 | |||
| 3e80825d58 | |||
| c97eafab29 | |||
| f737b6a24d |
+1
-1
@@ -9,7 +9,7 @@ archived
|
||||
*.bak
|
||||
*.clean
|
||||
session*
|
||||
llm.env
|
||||
.secret
|
||||
|
||||
# System test artifacts
|
||||
tests/results/
|
||||
|
||||
@@ -1,83 +1,72 @@
|
||||
# PROJECT KNOWLEDGE BASE
|
||||
|
||||
**Generated:** 2026-04-22
|
||||
**Commit:** 7b3b71a
|
||||
**Generated:** 2026-05-19
|
||||
**Commit:** 28abc5d
|
||||
**Branch:** master
|
||||
|
||||
## OVERVIEW
|
||||
HoneyBiscuitWorkshop is a Rust-based CI/CD system ("蜜饼工坊") with LLM-powered pipeline auto-configuration. 9 independent Cargo crates, no workspace.
|
||||
HoneyBiscuitWorkshop ("蜜饼工坊") — Rust CI/CD system with decorator-driven shell pipeline DSL and LLM-powered configuration. 3 independent Cargo crates, no workspace.
|
||||
|
||||
## STRUCTURE
|
||||
```
|
||||
./
|
||||
├── workshop-baker/ # Main orchestrator (CLI + daemon)
|
||||
├── workshop-agent/ # HTTP agent server
|
||||
├── workshop-pipeline/ # Config library (prebake/finalize)
|
||||
├── workshop-llm-detector/ # LLM repo analyzer
|
||||
├── workshop-vault/ # Secret management client
|
||||
├── workshop-cert/ # TLS certificate generator
|
||||
├── workshop-deviceid/ # Device ID library
|
||||
├── workshop-builder-native/ # Stub (unused)
|
||||
├── workshop-helper-mac/ # MAC address utility
|
||||
├── docs/ # mdBook documentation (zh-CN)
|
||||
├── playground/ # Experimentation
|
||||
└── third_party/ # Empty
|
||||
├── workshop-baker/ # Main orchestrator (CLI + pipeline stages)
|
||||
├── workshop-engine/ # Execution engine (extracted from baker)
|
||||
├── workshop-getterurl/ # URL fetching (go-getter style)
|
||||
├── docs/ # mdBook documentation (zh-CN)
|
||||
├── templates/ # Pipeline templates (bake.sh, finalize.yml, prebake.yml)
|
||||
└── archived/ # Deprecated crates (7 moved here)
|
||||
```
|
||||
|
||||
## WHERE TO LOOK
|
||||
| Task | Location | Notes |
|
||||
|------|----------|-------|
|
||||
| Baker (main) | `workshop-baker/src/{bake,prebake,engine}/` | Core pipeline logic |
|
||||
| Agent server | `workshop-agent/src/` | Actix-web HTTP + TLS |
|
||||
| Pipeline config | `workshop-pipeline/src/config/` | prebake/finalize YAML |
|
||||
| LLM integration | `workshop-llm-detector/src/llm/` | Ollama/OpenAI clients |
|
||||
| Secrets | `workshop-vault/src/` | Vault client wrapper |
|
||||
| Baker (CLI) | `workshop-baker/src/{lib,main}.rs` | Clap CLI + tokio entry |
|
||||
| Pipeline stages | `workshop-baker/src/{bake,prebake,finalize}/` | Stage orchestration |
|
||||
| Notifications | `workshop-baker/src/notify/` | Plugin-based event notifications |
|
||||
|| Engine (execution) | `workshop-engine/src/` | Process mgmt, PM detection |
|
||||
| URL fetching | `workshop-getterurl/src/` | Git/HTTP/File resource getter |
|
||||
|
||||
## CODE MAP
|
||||
| Symbol | Type | Location | Role |
|
||||
|--------|------|----------|------|
|
||||
| PipelineConfig | struct | workshop-pipeline/src/lib.rs:27 | Full pipeline config |
|
||||
| CICDLLMHelper | struct | workshop-llm-detector/src/lib.rs:16 | LLM analysis entry |
|
||||
| Engine | struct | workshop-baker/src/engine.rs | Build executor |
|
||||
| VaultClient | struct | workshop-vault/src/client.rs | Secret ops |
|
||||
| Cli | struct | workshop-baker/src/lib.rs:21 | CLI argument struct |
|
||||
| ExecutionContext | struct | workshop-engine/src/types.rs:10 | Pipeline context |
|
||||
| Engine | struct | workshop-engine/src/types.rs:99 | Build executor (placeholder) |
|
||||
| PrebakeConfig | struct | workshop-baker/src/prebake/config.rs | Prebake YAML schema |
|
||||
| FinalizeConfig | struct | workshop-baker/src/finalize/config.rs:21 | Finalize YAML schema |
|
||||
| GetterUrl | struct | workshop-getterurl/src/lib.rs:84 | Parsed resource URL |
|
||||
| Getter | trait | workshop-getterurl/src/getter.rs:18 | Resource fetch trait |
|
||||
|
||||
## CONVENTIONS
|
||||
|
||||
### Rust
|
||||
- **Edition**: 2024 (non-standard, requires Rust 1.85+)
|
||||
- Deprecate `mod.rs`
|
||||
- **Edition**: 2024 (requires Rust 1.85+), deprecates `mod.rs`
|
||||
- **Naming**: `snake_case` files/modules, `PascalCase` types, `SCREAMING_SNAKE_CASE` consts
|
||||
- **Error handling**: `thiserror` + `anyhow` combo
|
||||
- **Error handling**: `thiserror` + `anyhow` combo; `HasExitCode` trait for exit codes
|
||||
- **Async**: `#[tokio::main]` + `tokio` with "full" features
|
||||
- **Imports**: `std` → `external_crate` → `crate::module`
|
||||
- **Dual-mode**: CLI commands (prebake/bake/finalize) + daemon mode (unimplemented)
|
||||
|
||||
### Testing
|
||||
- **Rust**: `#[test]` inline, `#[tokio::test]` async, `tests/*.rs` integration
|
||||
- **Python**: `pytest` with `tests/integration/test_*.py` (Docker-based)
|
||||
- **Benchmarks**: Criterion with `harness = false`
|
||||
- **Rust**: `#[test]` inline, `#[tokio::test]` async, `tests/` integration
|
||||
- **Benchmarks**: Criterion (configured in Cargo.toml)
|
||||
|
||||
### Build Pipeline
|
||||
- **prebake.yml**: Environment setup (docker/firecracker/baremetal)
|
||||
- **bake.sh**: Build execution with `# @pipeline` decorators
|
||||
- **finalize.yml**: Packaging, deployment, notifications
|
||||
- **Annotations**: `@timeout()`, `@retry()`, `@parallel`, `@fallible`
|
||||
### CI
|
||||
- **GitHub Actions**: rust-toolchain (stable), clippy + rustfmt, `cargo test --all-features`
|
||||
- **Matrix**: workshop-baker, workshop-engine
|
||||
|
||||
## ANTI-PATTERNS (THIS PROJECT)
|
||||
|
||||
1. **Python in Rust project** — workshop-baker has Python tests (pytest.ini, test_prebake.py)
|
||||
2. **temp/ directory** — Non-standard in Rust projects
|
||||
3. **Certificates in source** — artifact workshop-cert/certs/ not excluded
|
||||
4. **Deprecated projects** — workshop-executor.old, workshop-monitor.old still present
|
||||
5. **Hardcoded mirrors** — Tsinghua University mirrors in configs
|
||||
6. **Rust 2024 edition** — May not work with stable toolchains
|
||||
1. **Hardcoded user** — "vulcan" username hardcoded throughout prebake
|
||||
2. **Unsafe libc** — workshop-baker/src/prebake/security.rs has 5+ unsafe blocks
|
||||
3. **Rust 2024 edition** — May not work with stable toolchains
|
||||
4. **Empty daemon** — `daemon/` dir is empty, Daemon command is `unimplemented!()`
|
||||
|
||||
## UNIQUE STYLES
|
||||
|
||||
- **Pipeline DSL**: Shell scripts with `# @decorator` annotations
|
||||
- **Dual-mode baker**: CLI commands + daemon mode
|
||||
- **LLM cookbook system**: YAML cookbooks for language-specific builds
|
||||
- **Notification system**: Plugin-based with batching, retry, priority groups
|
||||
- **LLM cookbook system**: YAML cookbooks for language-specific builds (archived)
|
||||
- **Chinese docs** — Documentation primarily in zh-CN
|
||||
- **User "vulcan"** — Custom build user (not root/runner)
|
||||
|
||||
## COMMANDS
|
||||
```bash
|
||||
@@ -86,23 +75,18 @@ cargo build --release -p workshop-baker
|
||||
|
||||
# Test
|
||||
cargo test -p workshop-baker
|
||||
pytest tests/integration/test_prebake.py -v
|
||||
cargo test -p workshop-engine
|
||||
|
||||
# Lint
|
||||
cargo clippy -- -D warnings
|
||||
cargo fmt --all
|
||||
|
||||
# Run baker
|
||||
cargo run --bin workshop-baker -- prebake config.yml
|
||||
cargo run --bin workshop-baker -- bake script.sh
|
||||
|
||||
# Run agent
|
||||
cargo run --bin workshop-agent
|
||||
# Run
|
||||
cargo run --bin workshop-baker -- --pipeline demo --build-id 1 --prebake prebake.yml --bake bake.sh --finalize finalize.yml
|
||||
```
|
||||
|
||||
## NOTES
|
||||
|
||||
- **Docker required** for integration tests
|
||||
- **workshop-builder-native** is a stub (3-line Hello World)
|
||||
- No root workspace Cargo.toml — each crate is independent
|
||||
- `AGENTS.md.old` at root — legacy documentation
|
||||
- **No root workspace** — each crate independent, build separately
|
||||
- **7 archived crates** — workshop-agent, workshop-cert, workshop-deviceid, workshop-helper-mac, workshop-llm-detector, workshop-pipeline, workshop-vault
|
||||
- **Python tests removed** — pytest.ini exists but test files no longer in workspace
|
||||
|
||||
@@ -35,3 +35,4 @@
|
||||
- [自定义 Decorator](zh-CN/vol3_dev/custom_decorator.md)
|
||||
- [模板变量](zh-CN/vol3_dev/template_variables.md)
|
||||
- [邮件模板](zh-CN/vol3_dev/mail_template.md)
|
||||
- [Resource 模型](zh-CN/vol3_dev/resource_model.md)
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
# Resource 模型
|
||||
|
||||
## 核心思想
|
||||
|
||||
整个系统只关心两件事:东西从哪来(fetch),东西到哪去(publish)。
|
||||
|
||||
fetch 方向:URL → 本地路径 → 使用。
|
||||
publish 方向:本地路径 → 变换(插件)→ URL。
|
||||
|
||||
两个方向共用同一套寻址方式,但 Resource 中间态只存在于 fetch 方向。
|
||||
|
||||
## URL 格式
|
||||
|
||||
```
|
||||
getter::url?param1=value1¶m2=value2
|
||||
```
|
||||
|
||||
getter 显式声明意图。不是"我猜这是个 git 仓库",而是"使用者告诉我这是什么"。
|
||||
|
||||
只有三个 getter:
|
||||
|
||||
- `git` — 克隆仓库,必要时 checkout 指定版本
|
||||
- `http` / `https` — 下载文件,必要时校验 checksum
|
||||
- `file` — 复制本地文件或目录
|
||||
|
||||
版本锁定不用 `@v1.0`(`@` 在 URL 里是认证分隔符),用 `?ref=v1.0`。
|
||||
|
||||
没有 `github.com/user/repo` 看起来应该 clone 它这种黑魔法。
|
||||
没有 `.tar.gz` 看起来应该解压它的惊喜。
|
||||
URL 是 URL,行为是行为,两者用 getter:: 显式绑定。
|
||||
|
||||
## Resource 不是网络地址
|
||||
|
||||
流水线阶段(prebake / finalize / notify)只接触 Resource:一个名字 + 一个本地路径,已就绪。它们不经由网络、不解析 URL、不碰 getter。
|
||||
|
||||
所有 fetch 行为在阶段启动前完成:
|
||||
|
||||
- worker 模式:bakerd fetch → 填入 ResourceRegistry → 启动 baker
|
||||
- standalone 模式:baker 自己 fetch → 填入 ResourceRegistry → 执行阶段
|
||||
- bare 模式:不 fetch,传入的必须是 Resource
|
||||
|
||||
ResourceRegistry 是名字到路径的映射。填进去就只读了。
|
||||
|
||||
## getterurl 是独立的 crate
|
||||
|
||||
URL 格式解析 + 基本 fetch 放在 workshop-getterurl,和 baker 本身没有耦合。任何人都可以用。
|
||||
|
||||
crate 分两层:
|
||||
|
||||
```
|
||||
结构层(parser):
|
||||
"git::https://host/repo.git?ref=v1.0" → { getter: "git", url: Url, params: Params }
|
||||
|
||||
获取层(builtin getters):
|
||||
getter.fetch(url, params, dst) → local_path
|
||||
git: git clone + optional checkout
|
||||
http: HTTP GET + optional checksum
|
||||
file: std::fs::copy
|
||||
```
|
||||
|
||||
获取层是字面意义的——不解压、不推断、不猜意图。
|
||||
|
||||
## Checksum
|
||||
|
||||
checksum 作为 URL query 参数传递。三态语义:
|
||||
|
||||
- `?checksum=sha256:abc` — 必须匹配
|
||||
- `?checksum=` — 显式禁用校验
|
||||
- 无 checksum — 如果服务端返回 Content-Digest 头,校验它
|
||||
|
||||
getter 不主动校验,它只执行 URL 里指定的操作。
|
||||
|
||||
## Artifact 不经过 Resource
|
||||
|
||||
artifact 是 bake 的产物,自产生就开始被使用,不存在中间态。它的路径就是 bake 产出的本地路径,不走 ResourceRegistry。
|
||||
|
||||
artifact 的 publish 方向由插件定义行为——可能上传、可能扫描、可能只记录。插件定义最终的行为,只要是自洽的。
|
||||
|
||||
## 三种运行模式
|
||||
|
||||
```
|
||||
fetch 执行者 fetch 在阶段内
|
||||
daemon 模式 bakerd 否
|
||||
standalone baker 自身 否(阶段启动前已完成)
|
||||
bare 模式 无 否(必须传入 Resource)
|
||||
```
|
||||
|
||||
bare 模式下传入的必须是已经就绪的 Resource。外部 URL 是 daemon 或 standalone 的事,bare 不负责。
|
||||
@@ -1,217 +0,0 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
# HoneyBiscuitWorkshop System Test Runner
|
||||
#
|
||||
# Usage:
|
||||
# ./runner.sh # baker tests (default)
|
||||
# ./runner.sh -c upm:rust # Rust UPM tests
|
||||
# ./runner.sh -c pm:pacman # pacman PM engine tests
|
||||
# ./runner.sh -c pm:apt -k test_mirror # apt mirror test only
|
||||
# ./runner.sh -c upm:rust --network # include network tests
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
CONTAINER_NAME="hbw-system-test-container"
|
||||
STAGING_DIR="$SCRIPT_DIR/.staging"
|
||||
|
||||
# ── defaults ────────────────────────────────────────────────────────
|
||||
|
||||
MODE="auto"
|
||||
CATEGORY="baker" # baker | upm:<name> | pm:<name>
|
||||
TEST_DIR=""
|
||||
TEST_IMAGE=""
|
||||
PYTEST_ARGS=()
|
||||
NETWORK=0
|
||||
BAKER_LOG=1
|
||||
|
||||
# ── parse args ──────────────────────────────────────────────────────
|
||||
|
||||
show_help() {
|
||||
cat << 'HELP'
|
||||
Usage: ./runner.sh [OPTIONS]
|
||||
|
||||
Categories (-c):
|
||||
baker Baker system tests [default]
|
||||
upm:rust Rust UPM tests
|
||||
pm:pacman Pacman PM engine tests
|
||||
pm:apt APT PM engine tests
|
||||
pm:dnf DNF PM engine tests
|
||||
pm:apk APK PM engine tests
|
||||
|
||||
Options:
|
||||
-c, --category CAT Test category (see above)
|
||||
--network Allow network-dependent tests (toolchain/crate downloads)
|
||||
-k FILTER pytest -k expression
|
||||
--timeout SECONDS Per-test timeout (default 120)
|
||||
--log / --no-log Stream baker output to terminal [default: --log]
|
||||
-h, --help This message
|
||||
HELP
|
||||
exit 0
|
||||
}
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
-c|--category)
|
||||
CATEGORY="$2"; shift 2 ;;
|
||||
--network)
|
||||
NETWORK=1; shift ;;
|
||||
-k)
|
||||
PYTEST_ARGS+=(-k "$2"); shift 2 ;;
|
||||
--timeout)
|
||||
PYTEST_ARGS+=(--timeout "$2"); shift 2 ;;
|
||||
--log) BAKER_LOG=1; shift ;;
|
||||
--no-log) BAKER_LOG=0; shift ;;
|
||||
manual) MODE="manual"; shift ;;
|
||||
auto) MODE="auto"; shift ;;
|
||||
-h|--help)
|
||||
show_help ;;
|
||||
--)
|
||||
shift; PYTEST_ARGS+=("$@"); break ;;
|
||||
*)
|
||||
PYTEST_ARGS+=("$1"); shift ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# ── resolve category → test dir + image ─────────────────────────────
|
||||
|
||||
case "$CATEGORY" in
|
||||
baker)
|
||||
TEST_DIR="tests/baker"
|
||||
TEST_IMAGE="hbw-system-test:latest"
|
||||
;;
|
||||
upm:rust)
|
||||
TEST_DIR="tests/upm"
|
||||
TEST_IMAGE="hbw-system-test:latest"
|
||||
;;
|
||||
pm:pacman|pm:arch)
|
||||
TEST_DIR="tests/engine"
|
||||
TEST_IMAGE="hbw-test-pm:test-arch"
|
||||
;;
|
||||
pm:apt)
|
||||
TEST_DIR="tests/engine"
|
||||
TEST_IMAGE="hbw-test-pm:test-apt"
|
||||
;;
|
||||
pm:dnf)
|
||||
TEST_DIR="tests/engine"
|
||||
TEST_IMAGE="hbw-test-pm:test-dnf"
|
||||
;;
|
||||
pm:apk)
|
||||
TEST_DIR="tests/engine"
|
||||
TEST_IMAGE="hbw-test-pm:test-apk"
|
||||
;;
|
||||
*)
|
||||
echo "Unknown category: $CATEGORY" >&2
|
||||
echo "Run './runner.sh --help' for usage." >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
# ── env vars ──────────────────────────────────────────────────────
|
||||
|
||||
DOCKER_ENV=()
|
||||
DOCKER_ENV+=(-e RUST_LOG=${RUST_LOG:-info})
|
||||
DOCKER_ENV+=(-e BAKER_LOG=${BAKER_LOG:-1})
|
||||
|
||||
# Load llm.env if present (for LLM-based test evaluation)
|
||||
if [ -f "$SCRIPT_DIR/llm.env" ]; then
|
||||
while IFS='=' read -r key value; do
|
||||
[ -z "$key" ] && continue
|
||||
[[ "$key" =~ ^# ]] && continue
|
||||
DOCKER_ENV+=(-e "$key=$value")
|
||||
done < "$SCRIPT_DIR/llm.env"
|
||||
fi
|
||||
|
||||
if [ "$NETWORK" -eq 1 ]; then
|
||||
DOCKER_ENV+=(-e RUST_UPM_NETWORK=1)
|
||||
fi
|
||||
|
||||
# ── cleanup ─────────────────────────────────────────────────────────
|
||||
|
||||
cleanup() {
|
||||
docker rm -f "$CONTAINER_NAME" 2>/dev/null || true
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
# ── build baker binary ──────────────────────────────────────────────
|
||||
|
||||
build_binary() {
|
||||
local target="x86_64-unknown-linux-musl"
|
||||
local binary="$SCRIPT_DIR/workshop-baker/target/$target/debug/workshop-baker"
|
||||
if [ ! -f "$binary" ]; then
|
||||
echo "[runner] Building static (musl) baker binary..."
|
||||
cargo build -p workshop-baker --target "$target" 2>&1 | tail -n 1
|
||||
fi
|
||||
}
|
||||
|
||||
# ── staging ─────────────────────────────────────────────────────────
|
||||
|
||||
prepare_staging() {
|
||||
if [ "$TEST_DIR" != "tests/engine" ]; then
|
||||
mkdir -p "$STAGING_DIR"
|
||||
cp "$SCRIPT_DIR/workshop-baker/target/x86_64-unknown-linux-musl/debug/workshop-baker" "$STAGING_DIR/"
|
||||
chmod +x "$STAGING_DIR/workshop-baker"
|
||||
fi
|
||||
}
|
||||
|
||||
# ── pm filter ───────────────────────────────────────────────────────
|
||||
|
||||
pm_filter() {
|
||||
case "${TEST_IMAGE##*:}" in
|
||||
test-arch) echo "-k pacman" ;;
|
||||
test-apt) echo "-k apt" ;;
|
||||
test-dnf) echo "-k dnf" ;;
|
||||
test-apk) echo "-k apk" ;;
|
||||
*) echo "" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# ── run ─────────────────────────────────────────────────────────────
|
||||
|
||||
run_container() {
|
||||
local filter=""
|
||||
if [ "$TEST_DIR" = "tests/engine" ]; then
|
||||
filter="$(pm_filter)"
|
||||
fi
|
||||
|
||||
echo "============================================"
|
||||
echo " HBW System Test Runner"
|
||||
echo " Category: $TEST_DIR Image: $TEST_IMAGE"
|
||||
echo "============================================"
|
||||
|
||||
mkdir -p "$SCRIPT_DIR/tests/results"
|
||||
|
||||
local mounts=(-v "$SCRIPT_DIR:/src:ro")
|
||||
if [ "$TEST_DIR" != "tests/engine" ]; then
|
||||
mounts+=(-v "$STAGING_DIR:/workshop") # writable: baker writes temp scripts here
|
||||
fi
|
||||
|
||||
local pytest_cmd="python3 -m venv /tmp/v && . /tmp/v/bin/activate && \
|
||||
pip install -i https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple \
|
||||
-r $TEST_DIR/../requirements.txt -q && \
|
||||
exec python -m pytest -v -s $TEST_DIR ${PYTEST_ARGS[*]} $filter"
|
||||
|
||||
if [ "$MODE" = "manual" ]; then
|
||||
docker run -it --rm --name "$CONTAINER_NAME" --privileged \
|
||||
${DOCKER_ENV[@]} "${mounts[@]}" "$TEST_IMAGE"
|
||||
else
|
||||
# Baker/UPM images (hbw-system-test) have ./entrypoint.sh which
|
||||
# doesn't exist when /src is the workdir → override to empty.
|
||||
# PM images (hbw-test-pm:*) have /bin/sh -c → CMD passed directly.
|
||||
if [ "$TEST_DIR" = "tests/engine" ]; then
|
||||
docker run --rm --name "$CONTAINER_NAME" --privileged \
|
||||
${DOCKER_ENV[@]} "${mounts[@]}" -w /src \
|
||||
"$TEST_IMAGE" "$pytest_cmd"
|
||||
else
|
||||
docker run --rm --name "$CONTAINER_NAME" --privileged \
|
||||
--entrypoint "" ${DOCKER_ENV[@]} "${mounts[@]}" -w /src \
|
||||
"$TEST_IMAGE" /bin/sh -c "$pytest_cmd"
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
# ── main ────────────────────────────────────────────────────────────
|
||||
|
||||
build_binary
|
||||
prepare_staging
|
||||
run_container
|
||||
echo "[runner] Done."
|
||||
@@ -1,32 +0,0 @@
|
||||
FROM archlinux:latest
|
||||
|
||||
RUN sed -i 's|https://geo.mirror.pkgbuild.com|https://mirrors.tuna.tsinghua.edu.cn/archlinux|g' /etc/pacman.d/mirrorlist && \
|
||||
sed -i 's|https://mirror.rackspace.com/archlinux|https://mirrors.tuna.tsinghua.edu.cn/archlinux|g' /etc/pacman.d/mirrorlist && \
|
||||
echo 'Server = https://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch' >> /etc/pacman.d/mirrorlist
|
||||
|
||||
RUN pacman -Syu --noconfirm && \
|
||||
pacman -S --noconfirm \
|
||||
python python-pip base-devel git curl wget sudo \
|
||||
which procps-ng shadow && \
|
||||
pacman -Scc --noconfirm
|
||||
|
||||
# Create test user
|
||||
RUN useradd -m -s /bin/bash testuser
|
||||
|
||||
# Python environment
|
||||
WORKDIR /app
|
||||
COPY tests/requirements.txt ./requirements.txt
|
||||
RUN python -m venv /opt/test-venv && \
|
||||
/opt/test-venv/bin/pip install -r requirements.txt
|
||||
|
||||
# Copy test suite
|
||||
COPY tests/ ./tests/
|
||||
COPY tests/entrypoint.sh ./entrypoint.sh
|
||||
RUN chmod +x entrypoint.sh
|
||||
|
||||
# Environment
|
||||
ENV PATH="/opt/test-venv/bin:$PATH"
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
ENV PYTHONDONTWRITEBYTECODE=1
|
||||
|
||||
ENTRYPOINT ["./entrypoint.sh"]
|
||||
@@ -1,105 +0,0 @@
|
||||
import os
|
||||
import pytest
|
||||
|
||||
FIXTURES = os.path.join(os.path.dirname(os.path.dirname(__file__)), "fixtures", "bake")
|
||||
|
||||
@pytest.mark.bake
|
||||
class TestBakeSimpleScript:
|
||||
|
||||
def test_simple_script_runs(self, run_baker, bake_workspace, work_dir):
|
||||
prebake_cfg = os.path.join(
|
||||
os.path.dirname(os.path.dirname(__file__)), "fixtures", "prebake", "minimal.yml"
|
||||
)
|
||||
script = os.path.join(FIXTURES, "simple.sh")
|
||||
bake_base = os.path.join(FIXTURES, "bake_base.sh")
|
||||
|
||||
result = run_baker(
|
||||
"bake", script,
|
||||
"--bake-base", bake_base,
|
||||
"--prebake", prebake_cfg,
|
||||
cwd=work_dir,
|
||||
timeout=60,
|
||||
)
|
||||
assert result["success"], f"Bake failed: {result['stderr']}"
|
||||
assert "Hello from simple bake script" in result["stdout"]
|
||||
|
||||
def test_simple_script_sets_env_vars(self, run_baker, bake_workspace, work_dir):
|
||||
prebake_cfg = os.path.join(
|
||||
os.path.dirname(os.path.dirname(__file__)), "fixtures", "prebake", "minimal.yml"
|
||||
)
|
||||
script = os.path.join(FIXTURES, "simple.sh")
|
||||
bake_base = os.path.join(FIXTURES, "bake_base.sh")
|
||||
|
||||
result = run_baker(
|
||||
"bake", script,
|
||||
"--bake-base", bake_base,
|
||||
"--prebake", prebake_cfg,
|
||||
cwd=work_dir,
|
||||
timeout=60,
|
||||
)
|
||||
assert result["success"]
|
||||
assert "test-pipeline" in result["stdout"]
|
||||
|
||||
@pytest.mark.bake
|
||||
class TestBakePipeline:
|
||||
|
||||
def test_pipeline_functions_execute(self, run_baker, bake_workspace, work_dir):
|
||||
prebake_cfg = os.path.join(
|
||||
os.path.dirname(os.path.dirname(__file__)), "fixtures", "prebake", "minimal.yml"
|
||||
)
|
||||
script = os.path.join(FIXTURES, "pipeline.sh")
|
||||
bake_base = os.path.join(FIXTURES, "bake_base.sh")
|
||||
|
||||
result = run_baker(
|
||||
"bake", script,
|
||||
"--bake-base", bake_base,
|
||||
"--prebake", prebake_cfg,
|
||||
cwd=work_dir,
|
||||
timeout=60,
|
||||
)
|
||||
assert result["success"], f"Pipeline bake failed: {result['stderr']}"
|
||||
assert os.path.exists("/tmp/bake-test-output/step1.txt")
|
||||
assert os.path.exists("/tmp/bake-test-output/step2.txt")
|
||||
|
||||
def test_pipeline_step_outputs_correct(self, run_baker, bake_workspace, work_dir):
|
||||
prebake_cfg = os.path.join(
|
||||
os.path.dirname(os.path.dirname(__file__)), "fixtures", "prebake", "minimal.yml"
|
||||
)
|
||||
script = os.path.join(FIXTURES, "pipeline.sh")
|
||||
bake_base = os.path.join(FIXTURES, "bake_base.sh")
|
||||
|
||||
result = run_baker(
|
||||
"bake", script,
|
||||
"--bake-base", bake_base,
|
||||
"--prebake", prebake_cfg,
|
||||
cwd=work_dir,
|
||||
timeout=60,
|
||||
)
|
||||
assert result["success"]
|
||||
with open("/tmp/bake-test-output/step1.txt") as f:
|
||||
assert f.read().strip() == "step1-done"
|
||||
with open("/tmp/bake-test-output/step2.txt") as f:
|
||||
assert f.read().strip() == "step2-done"
|
||||
|
||||
@pytest.mark.bake
|
||||
class TestBakeAfterDependencies:
|
||||
|
||||
def test_after_ordering(self, run_baker, bake_workspace, work_dir):
|
||||
prebake_cfg = os.path.join(
|
||||
os.path.dirname(os.path.dirname(__file__)), "fixtures", "prebake", "minimal.yml"
|
||||
)
|
||||
script = os.path.join(FIXTURES, "with_after.sh")
|
||||
bake_base = os.path.join(FIXTURES, "bake_base.sh")
|
||||
|
||||
result = run_baker(
|
||||
"bake", script,
|
||||
"--bake-base", bake_base,
|
||||
"--prebake", prebake_cfg,
|
||||
cwd=work_dir,
|
||||
timeout=60,
|
||||
)
|
||||
assert result["success"], f"@after bake failed: {result['stderr']}"
|
||||
assert os.path.exists("/tmp/bake-after-test/status.txt")
|
||||
assert os.path.exists("/tmp/bake-after-test/source.txt")
|
||||
assert os.path.exists("/tmp/bake-after-test/build.txt")
|
||||
|
||||
@@ -1,62 +0,0 @@
|
||||
import os
|
||||
import pytest
|
||||
|
||||
from probes import file
|
||||
from llm import judge_log, LLMClient
|
||||
|
||||
FIXTURES = os.path.join(os.path.dirname(os.path.dirname(__file__)), "fixtures", "finalize")
|
||||
|
||||
@pytest.mark.finalize
|
||||
class TestFinalizeMinimal:
|
||||
|
||||
def test_minimal_finalize_succeeds(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "minimal.yml")
|
||||
result = run_baker("finalize", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"], f"Minimal finalize failed: {result['stderr']}"
|
||||
|
||||
@pytest.mark.finalize
|
||||
class TestFinalizeHooks:
|
||||
|
||||
def test_early_hook_executes(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "with_hooks.yml")
|
||||
result = run_baker("finalize", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
assert "finalize early hook" in result["stdout"] or "finalize early hook" in result["stderr"]
|
||||
|
||||
def test_late_hook_executes(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "with_hooks.yml")
|
||||
result = run_baker("finalize", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
assert "finalize late hook" in result["stdout"] or "finalize late hook" in result["stderr"]
|
||||
|
||||
@pytest.mark.finalize
|
||||
@pytest.mark.llm
|
||||
class TestFinalizeWithLLM:
|
||||
|
||||
def test_finalize_hooks_via_llm(self, run_baker, llm_env, work_dir):
|
||||
if not llm_env:
|
||||
pytest.skip("LLM environment not configured")
|
||||
|
||||
config = os.path.join(FIXTURES, "with_hooks.yml")
|
||||
result = run_baker("finalize", config, cwd=work_dir, timeout=60)
|
||||
|
||||
combined_log = result["stdout"] + "\n" + result["stderr"]
|
||||
|
||||
judgment = judge_log(
|
||||
operation="finalize with early and late hooks",
|
||||
expected_behavior="Both early and late hooks executed successfully, "
|
||||
"printing 'finalize early hook' and 'finalize late hook'",
|
||||
actual_log=combined_log,
|
||||
)
|
||||
assert judgment.passed, f"LLM judgment failed: {judgment.reason}"
|
||||
|
||||
@pytest.mark.finalize
|
||||
class TestFinalizePlugins:
|
||||
|
||||
@pytest.mark.skip(reason="Plugin download test not yet implemented")
|
||||
def test_plugin_download(self, run_baker, work_dir):
|
||||
pass
|
||||
|
||||
@pytest.mark.skip(reason="Artifact stage not yet implemented")
|
||||
def test_artifact_collection(self, run_baker, work_dir):
|
||||
pass
|
||||
@@ -1,202 +0,0 @@
|
||||
import os
|
||||
import json
|
||||
import pytest
|
||||
|
||||
from probes import user, file, package
|
||||
|
||||
FIXTURES = os.path.join(os.path.dirname(os.path.dirname(__file__)), "fixtures", "prebake")
|
||||
|
||||
@pytest.mark.prebake
|
||||
class TestPrebakeConfigValidation:
|
||||
|
||||
def test_invalid_version_fails(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "invalid_version.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert not result["success"], f"Expected prebake to fail with invalid version"
|
||||
assert "version" in result["stderr"].lower() or "Version" in result["stderr"]
|
||||
|
||||
def test_missing_required_field_fails(self, run_baker, work_dir):
|
||||
missing_env = os.path.join(work_dir, "missing_env.yml")
|
||||
with open(missing_env, "w") as f:
|
||||
f.write('version: "1.0"\n')
|
||||
result = run_baker("prebake", missing_env, cwd=work_dir, timeout=60)
|
||||
assert not result["success"]
|
||||
|
||||
def test_minimal_config_succeeds(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "minimal.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"], f"Minimal prebake failed: {result['stderr']}"
|
||||
|
||||
@pytest.mark.prebake
|
||||
class TestPrebakeBootstrap:
|
||||
|
||||
def test_bootstrap_creates_vulcan_user(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"], f"prebake failed: {result['stderr']}"
|
||||
assert user.user_exists("vulcan")
|
||||
|
||||
def test_bootstrap_creates_workspace(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
assert file.dir_exists("/home/vulcan/workspace")
|
||||
|
||||
def test_bootstrap_creates_symlink(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
assert os.path.islink("/workspace")
|
||||
|
||||
def test_bootstrap_generates_bootstrap_script(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
assert file.file_exists("/bootstrap.sh")
|
||||
|
||||
def test_bootstrap_script_permissions(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
mode = file.file_permissions("/bootstrap.sh")
|
||||
assert mode & 0o111, f"/bootstrap.sh not executable, mode: {oct(mode)}"
|
||||
|
||||
def test_bootstrap_script_contains_user_creation(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
assert file.file_exists("/bootstrap.sh")
|
||||
assert file.file_contains("/bootstrap.sh", "useradd")
|
||||
|
||||
def test_bootstrap_script_contains_workspace_setup(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
assert file.file_contains("/bootstrap.sh", "mkdir -p /home/vulcan/workspace")
|
||||
|
||||
def test_bootstrap_with_pacman_creates_sudoers(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "with_pacman.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=120)
|
||||
assert result["success"], f"prebake failed: {result['stderr']}"
|
||||
assert file.file_exists("/etc/sudoers.d/workshop")
|
||||
|
||||
def test_custom_bootstrap_executes(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "with_custom_bootstrap.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"], f"Custom bootstrap failed: {result['stderr']}"
|
||||
assert file.dir_exists("/custom-workspace")
|
||||
|
||||
@pytest.mark.prebake
|
||||
class TestPrebakeDryRun:
|
||||
|
||||
def test_dry_run_does_not_create_user(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
env = {"HBW_DRY_RUN": "true"}
|
||||
result = run_baker("prebake", config, cwd=work_dir, env=env, timeout=60)
|
||||
assert result["success"], f"Dry-run failed: {result['stderr']}"
|
||||
|
||||
def test_dry_run_logs_script_content(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
env = {"HBW_DRY_RUN": "true", "RUST_LOG": "info"}
|
||||
result = run_baker("prebake", config, cwd=work_dir, env=env, timeout=60)
|
||||
assert result["success"]
|
||||
combined = result["stdout"] + result["stderr"]
|
||||
assert "DRY_RUN" in combined or "dry_run" in combined.lower()
|
||||
|
||||
@pytest.mark.prebake
|
||||
class TestPrebakeHooks:
|
||||
|
||||
#@pytest.mark.xfail(reason="Hook execution fails with Permission denied in container environment", strict=False)
|
||||
def test_early_hook_executes(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "with_hooks.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"], f"prebake with hooks failed: {result['stderr']}"
|
||||
combined = result["stdout"] + result["stderr"]
|
||||
assert "early hook executed" in combined
|
||||
|
||||
#@pytest.mark.xfail(reason="Hook execution fails with Permission denied in container environment", strict=False)
|
||||
def test_late_hook_executes(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "with_hooks.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"], f"prebake with hooks failed: {result['stderr']}"
|
||||
combined = result["stdout"] + result["stderr"]
|
||||
assert "late hook executed" in combined
|
||||
|
||||
#@pytest.mark.xfail(reason="Hook execution fails with Permission denied in container environment", strict=False)
|
||||
def test_hooks_run_in_order(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "with_hooks.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
combined = result["stdout"] + result["stderr"]
|
||||
early_pos = combined.find("early hook executed")
|
||||
late_pos = combined.find("late hook executed")
|
||||
assert early_pos != -1 and late_pos != -1
|
||||
assert early_pos < late_pos, "Early hook should execute before late hook"
|
||||
|
||||
def test_empty_hooks_succeed(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "minimal.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"], f"prebake with no hooks failed: {result['stderr']}"
|
||||
|
||||
@pytest.mark.prebake
|
||||
class TestPrebakeDepsSystem:
|
||||
|
||||
def test_pacman_packages_installed(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "with_pacman.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=120)
|
||||
assert result["success"], f"prebake failed: {result['stderr']}"
|
||||
assert package.binary_exists("/usr/bin/curl")
|
||||
assert package.binary_exists("/usr/bin/wget")
|
||||
|
||||
def test_pacman_mirror_change(self, run_baker, work_dir):
|
||||
mirror_config = os.path.join(work_dir, "mirror.yml")
|
||||
with open(mirror_config, "w") as f:
|
||||
f.write('''
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
dependencies:
|
||||
system:
|
||||
pacman:
|
||||
packages: ["which"]
|
||||
mirror: "https://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch"
|
||||
''')
|
||||
result = run_baker("prebake", mirror_config, cwd=work_dir, timeout=120)
|
||||
assert result["success"], f"prebake with mirror failed: {result['stderr']}"
|
||||
|
||||
@pytest.mark.prebake
|
||||
class TestPrebakeEnvVars:
|
||||
|
||||
def test_prebake_envvars_injected(self, run_baker, work_dir):
|
||||
hook_config = os.path.join(work_dir, "envvar_hook.yml")
|
||||
with open(hook_config, "w") as f:
|
||||
f.write('''
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
envvars:
|
||||
prebake:
|
||||
TEST_INJECTED: "injected_value"
|
||||
hooks:
|
||||
early:
|
||||
- name: "check-env"
|
||||
command: "echo $TEST_INJECTED"
|
||||
''')
|
||||
result = run_baker("prebake", hook_config, cwd=work_dir, timeout=60)
|
||||
assert result["success"], f"prebake with envvars failed: {result['stderr']}"
|
||||
combined = result["stdout"] + result["stderr"]
|
||||
assert "injected_value" in combined
|
||||
|
||||
@pytest.mark.prebake
|
||||
class TestPrebakeSecurity:
|
||||
|
||||
#@pytest.mark.xfail(reason="Privilege drop may fail in container without proper setup", strict=False)
|
||||
def test_privilege_drop_after_bootstrap(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "with_security.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"], f"prebake with security failed: {result['stderr']}"
|
||||
|
||||
@@ -1,180 +0,0 @@
|
||||
"""
|
||||
Root conftest for HoneyBiscuitWorkshop system tests.
|
||||
|
||||
Provides core fixtures for:
|
||||
- Locating the baker binary and examples
|
||||
- Running subprocess commands with output capture
|
||||
- Managing temporary working directories
|
||||
"""
|
||||
import os
|
||||
import subprocess
|
||||
import shutil
|
||||
import tempfile
|
||||
import pytest
|
||||
|
||||
|
||||
def pytest_configure(config):
|
||||
"""Register custom markers."""
|
||||
config.addinivalue_line("markers", "prebake: prebake stage tests")
|
||||
config.addinivalue_line("markers", "bake: bake stage tests")
|
||||
config.addinivalue_line("markers", "finalize: finalize stage tests")
|
||||
config.addinivalue_line("markers", "llm: tests requiring LLM API")
|
||||
config.addinivalue_line("markers", "slow: slow running tests")
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def baker_bin():
|
||||
"""Path to the workshop-baker binary."""
|
||||
path = os.environ.get("BAKER_BIN", "/workshop/workshop-baker")
|
||||
if not os.path.isfile(path):
|
||||
pytest.skip(f"Baker binary not found at {path}")
|
||||
if not os.access(path, os.X_OK):
|
||||
pytest.skip(f"Baker binary not executable at {path}")
|
||||
return path
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def examples_dir():
|
||||
"""Path to the examples directory (may be None)."""
|
||||
path = os.environ.get("EXAMPLES_DIR", "/workshop/examples")
|
||||
if not os.path.isdir(path):
|
||||
return None
|
||||
return path
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def llm_env():
|
||||
"""LLM environment variables dict (from llm.env)."""
|
||||
env = {}
|
||||
for var in [
|
||||
"OPENAI_BASE_URL",
|
||||
"OPENAI_API_KEY",
|
||||
"ANTHROPIC_BASE_URL",
|
||||
"ANTHROPIC_API_KEY",
|
||||
"MODEL",
|
||||
"REASONING_EFFORT",
|
||||
]:
|
||||
val = os.environ.get(var)
|
||||
if val:
|
||||
env[var] = val
|
||||
return env
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def work_dir():
|
||||
"""Create a temporary working directory for a test, cleaned up after."""
|
||||
d = tempfile.mkdtemp(prefix="hbw-test-")
|
||||
yield d
|
||||
shutil.rmtree(d, ignore_errors=True)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def run_cmd():
|
||||
"""Fixture that provides a command runner function.
|
||||
|
||||
Returns a function that runs a command and returns
|
||||
(stdout, stderr, returncode, success).
|
||||
"""
|
||||
|
||||
def _run(args, cwd=None, env=None, timeout=120):
|
||||
full_env = {**os.environ}
|
||||
if env:
|
||||
full_env.update(env)
|
||||
|
||||
show_logs = os.environ.get("BAKER_LOG", "0") == "1"
|
||||
|
||||
try:
|
||||
if show_logs:
|
||||
# Stream output to terminal in real-time while capturing
|
||||
import sys, select
|
||||
p = subprocess.Popen(
|
||||
args, cwd=cwd, env=full_env,
|
||||
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
|
||||
text=True,
|
||||
)
|
||||
stdout_chunks, stderr_chunks = [], []
|
||||
try:
|
||||
p.wait(timeout=timeout)
|
||||
except subprocess.TimeoutExpired:
|
||||
p.kill()
|
||||
p.wait()
|
||||
return {"stdout": "", "stderr": f"Command timed out after {timeout}s",
|
||||
"returncode": -1, "success": False}
|
||||
for line in p.stdout:
|
||||
sys.stderr.write("[baker] " + line)
|
||||
stdout_chunks.append(line)
|
||||
for line in p.stderr:
|
||||
sys.stderr.write("[baker:err] " + line)
|
||||
stderr_chunks.append(line)
|
||||
return {
|
||||
"stdout": "".join(stdout_chunks),
|
||||
"stderr": "".join(stderr_chunks),
|
||||
"returncode": p.returncode,
|
||||
"success": p.returncode == 0,
|
||||
}
|
||||
else:
|
||||
result = subprocess.run(
|
||||
args, cwd=cwd, env=full_env,
|
||||
capture_output=True, text=True, timeout=timeout,
|
||||
)
|
||||
return {
|
||||
"stdout": result.stdout,
|
||||
"stderr": result.stderr,
|
||||
"returncode": result.returncode,
|
||||
"success": result.returncode == 0,
|
||||
}
|
||||
except subprocess.TimeoutExpired:
|
||||
return {
|
||||
"stdout": "",
|
||||
"stderr": f"Command timed out after {timeout}s",
|
||||
"returncode": -1,
|
||||
"success": False,
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"stdout": "",
|
||||
"stderr": str(e),
|
||||
"returncode": -1,
|
||||
"success": False,
|
||||
}
|
||||
|
||||
return _run
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def run_baker(baker_bin, run_cmd):
|
||||
"""Fixture that provides a baker command runner.
|
||||
|
||||
Returns a function that runs workshop-baker with common flags.
|
||||
"""
|
||||
|
||||
def _run(subcommand, *args, cwd=None, env=None, timeout=120):
|
||||
cmd = [
|
||||
baker_bin,
|
||||
"-u", "testuser",
|
||||
"-p", "test-pipeline",
|
||||
"-b", "test-build-001",
|
||||
"bare",
|
||||
subcommand,
|
||||
]
|
||||
cmd.extend(args)
|
||||
return run_cmd(cmd, cwd=cwd, env=env, timeout=timeout)
|
||||
|
||||
return _run
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def fixtures_dir():
|
||||
"""Path to the test fixtures directory."""
|
||||
return os.path.join(os.path.dirname(__file__), "fixtures")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def bake_workspace():
|
||||
"""Create /workspace for bake tests (baker writes temp scripts there)."""
|
||||
existed = os.path.exists("/workspace")
|
||||
if not existed:
|
||||
os.makedirs("/workspace", exist_ok=True)
|
||||
yield
|
||||
if not existed:
|
||||
shutil.rmtree("/workspace", ignore_errors=True)
|
||||
@@ -1,102 +0,0 @@
|
||||
"""PM system tests driven by bare workshop-baker prebake."""
|
||||
|
||||
import os
|
||||
import pytest
|
||||
from probes import binary_exists
|
||||
|
||||
PACMAN_MIRROR = "https://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch"
|
||||
|
||||
|
||||
def _yml(pm, pkgs, mirror="", repo=None):
|
||||
"""Build prebake.yml. repo is a list of lines (without lead indent)."""
|
||||
m = ""
|
||||
if mirror:
|
||||
m += ' mirror: "' + mirror + '"\n'
|
||||
if repo:
|
||||
m += " repositories:\n"
|
||||
for line in repo:
|
||||
m += " " + line + "\n"
|
||||
return """\
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
dependencies:
|
||||
system:
|
||||
{pm}:
|
||||
packages: [{pkgs}]
|
||||
{m}""".format(pm=pm, pkgs=pkgs, m=m)
|
||||
|
||||
|
||||
# ── install ──────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.parametrize("pm_name,pkgs", [
|
||||
("pacman", '"curl", "git"'),
|
||||
("apt", '"curl", "git"'),
|
||||
("dnf", '"curl", "git"'),
|
||||
("apk", '"curl", "git"'),
|
||||
])
|
||||
def test_install(run_cmd, work_dir, pm_name, pkgs):
|
||||
cfg = os.path.join(work_dir, "pm.yml")
|
||||
mirror = PACMAN_MIRROR if pm_name == "pacman" else ""
|
||||
with open(cfg, "w") as f:
|
||||
f.write(_yml(pm_name, pkgs, mirror=mirror))
|
||||
r = run_cmd(["/baker", "-p", "t", "-b", "1", "-u", "testuser",
|
||||
"bare", "prebake", cfg], timeout=120)
|
||||
assert r["success"], "[{}] prebake: {}".format(pm_name, r["stderr"][:300])
|
||||
for pkg in ["curl", "git"]:
|
||||
assert binary_exists("/usr/bin/" + pkg), "[{}] {} missing".format(pm_name, pkg)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("pm_name", ["pacman", "apt", "dnf", "apk"])
|
||||
def test_reinstall(run_cmd, work_dir, pm_name):
|
||||
cfg = os.path.join(work_dir, "pm.yml")
|
||||
mirror = PACMAN_MIRROR if pm_name == "pacman" else ""
|
||||
with open(cfg, "w") as f:
|
||||
f.write(_yml(pm_name, '"curl"', mirror=mirror))
|
||||
cmd = ["/baker", "-p", "t", "-b", "1", "-u", "testuser", "bare", "prebake", cfg]
|
||||
r1 = run_cmd(cmd, timeout=120)
|
||||
assert r1["success"], "[{}] first install failed".format(pm_name)
|
||||
r2 = run_cmd(cmd, timeout=120)
|
||||
assert r2["success"], "[{}] reinstall failed".format(pm_name)
|
||||
|
||||
|
||||
# ── change_mirror ────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.parametrize("pm_name", ["pacman", "apt", "dnf", "apk"])
|
||||
def test_mirror(run_cmd, work_dir, pm_name):
|
||||
cfg = os.path.join(work_dir, "pm.yml")
|
||||
mirror = PACMAN_MIRROR if pm_name == "pacman" else ""
|
||||
with open(cfg, "w") as f:
|
||||
f.write(_yml(pm_name, '"which"', mirror=mirror))
|
||||
r = run_cmd(["/baker", "-p", "t", "-b", "1", "-u", "testuser",
|
||||
"bare", "prebake", cfg], timeout=120)
|
||||
assert r["success"], "[{}] mirror: {}".format(pm_name, r["stderr"][:300])
|
||||
assert binary_exists("/usr/bin/which"), "[{}] which not installed".format(pm_name)
|
||||
|
||||
|
||||
# ── add_repository ───────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.parametrize("pm_name,repo_lines,result_file", [
|
||||
("pacman", ['- name: custom', ' server: "https://repo.example.com/$arch"'],
|
||||
"/etc/pacman.conf"),
|
||||
("apt", ['- url: "https://repo.example.com/debian"', ' distro: bookworm'],
|
||||
"/etc/apt/sources.list.d/custom.list"),
|
||||
("dnf", ['- name: custom', ' baseurl: "https://repo.example.com/fedora"'],
|
||||
"/etc/yum.repos.d/custom.repo"),
|
||||
("apk", ['- url: "https://repo.example.com/v3.19/main"'],
|
||||
"/etc/apk/repositories"),
|
||||
])
|
||||
@pytest.mark.xfail(reason="fake repo URL, package install may fail", strict=False)
|
||||
def test_add_repo(run_cmd, work_dir, pm_name, repo_lines, result_file):
|
||||
cfg = os.path.join(work_dir, "pm.yml")
|
||||
mirror = PACMAN_MIRROR if pm_name == "pacman" else ""
|
||||
with open(cfg, "w") as f:
|
||||
f.write(_yml(pm_name, '"curl"', mirror=mirror, repo=repo_lines))
|
||||
r = run_cmd(["/baker", "-p", "t", "-b", "1", "-u", "testuser",
|
||||
"bare", "prebake", cfg], timeout=120)
|
||||
# Prebake may fail if repo URL is unreachable — check only the config file
|
||||
# was written by add_repository before the (possibly failing) install step.
|
||||
assert binary_exists(result_file), \
|
||||
"[{}] repo file missing: {}".format(pm_name, result_file)
|
||||
@@ -1,36 +0,0 @@
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
# Load LLM environment if available
|
||||
if [ -f /workshop/llm.env ]; then
|
||||
set -a
|
||||
source /workshop/llm.env
|
||||
set +a
|
||||
fi
|
||||
|
||||
# Export test configuration
|
||||
export BAKER_BIN=/workshop/workshop-baker
|
||||
export EXAMPLES_DIR=/workshop/examples
|
||||
export TEST_MODE=${TEST_MODE:-auto}
|
||||
|
||||
if [ "$TEST_MODE" = "manual" ]; then
|
||||
echo "========================================"
|
||||
echo " HoneyBiscuitWorkshop System Tests"
|
||||
echo " Mode: MANUAL (interactive shell)"
|
||||
echo "========================================"
|
||||
echo ""
|
||||
echo "Binary: $BAKER_BIN"
|
||||
echo "Examples: $EXAMPLES_DIR"
|
||||
echo ""
|
||||
echo "Run tests manually:"
|
||||
echo " cd /app && pytest tests/ -v"
|
||||
echo ""
|
||||
exec /bin/bash
|
||||
else
|
||||
echo "========================================"
|
||||
echo " HoneyBiscuitWorkshop System Tests"
|
||||
echo " Mode: AUTO (running pytest)"
|
||||
echo "========================================"
|
||||
cd /app
|
||||
exec python -m pytest tests/ -v --tb=short --junitxml=/results/report.xml "$@"
|
||||
fi
|
||||
Vendored
-6
@@ -1,6 +0,0 @@
|
||||
#!/bin/bash
|
||||
[ -f /dev/shm/bakeexport.{{ name }} ] && . /dev/shm/bakeexport.{{ name }} || true
|
||||
{{ condition }}
|
||||
{{ main }}
|
||||
{{ name }}
|
||||
{{ export }}
|
||||
Vendored
-25
@@ -1,25 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
# @pipeline
|
||||
step_one() {
|
||||
echo "Step 1: Setup"
|
||||
mkdir -p /tmp/bake-test-output
|
||||
echo "step1-done" > /tmp/bake-test-output/step1.txt
|
||||
}
|
||||
|
||||
# @pipeline
|
||||
step_two() {
|
||||
echo "Step 2: Build"
|
||||
echo "step2-done" > /tmp/bake-test-output/step2.txt
|
||||
}
|
||||
|
||||
# @pipeline
|
||||
step_three() {
|
||||
echo "Step 3: Verify"
|
||||
if [ -f /tmp/bake-test-output/step1.txt ] && [ -f /tmp/bake-test-output/step2.txt ]; then
|
||||
echo "All steps completed successfully"
|
||||
else
|
||||
echo "ERROR: Missing step outputs"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
Vendored
-5
@@ -1,5 +0,0 @@
|
||||
#!/bin/bash
|
||||
echo "Hello from simple bake script"
|
||||
echo "Task ID: $HBW_TASKID"
|
||||
echo "Pipeline: $HBW_PIPELINE"
|
||||
exit 0
|
||||
Vendored
-22
@@ -1,22 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
# @pipeline
|
||||
init_workspace() {
|
||||
echo "Initializing workspace"
|
||||
mkdir -p /tmp/bake-after-test
|
||||
echo "init" > /tmp/bake-after-test/status.txt
|
||||
}
|
||||
|
||||
# @pipeline
|
||||
# @after(init_workspace)
|
||||
fetch_source() {
|
||||
echo "Fetching source (depends on init_workspace)"
|
||||
echo "fetched" > /tmp/bake-after-test/source.txt
|
||||
}
|
||||
|
||||
# @pipeline
|
||||
# @after(fetch_source)
|
||||
build_project() {
|
||||
echo "Building project (depends on fetch_source)"
|
||||
echo "built" > /tmp/bake-after-test/build.txt
|
||||
}
|
||||
-4
@@ -1,4 +0,0 @@
|
||||
#!/bin/bash
|
||||
echo "EARLY_HOOK_EXECUTED"
|
||||
echo "Working directory: $(pwd)"
|
||||
echo "User: $(whoami)"
|
||||
Vendored
-3
@@ -1,3 +0,0 @@
|
||||
#!/bin/bash
|
||||
echo "LATE_HOOK_EXECUTED"
|
||||
echo "Working directory: $(pwd)"
|
||||
Vendored
-6
@@ -1,6 +0,0 @@
|
||||
version: "0.0.1"
|
||||
plugin: {}
|
||||
notification: {}
|
||||
artifact: []
|
||||
cleanup:
|
||||
policy: "auto"
|
||||
-13
@@ -1,13 +0,0 @@
|
||||
version: "0.0.1"
|
||||
plugin: {}
|
||||
notification: {}
|
||||
artifact: []
|
||||
cleanup:
|
||||
policy: "auto"
|
||||
hooks:
|
||||
early:
|
||||
- name: "finalize-early"
|
||||
command: "echo 'finalize early hook'"
|
||||
late:
|
||||
- name: "finalize-late"
|
||||
command: "echo 'finalize late hook'"
|
||||
Vendored
-6
@@ -1,6 +0,0 @@
|
||||
FROM docker.1ms.run/library/alpine:3.19
|
||||
RUN sed -i 's|dl-cdn.alpinelinux.org|mirrors.tuna.tsinghua.edu.cn|g' /etc/apk/repositories
|
||||
RUN apk add --no-cache python3 py3-pip bash
|
||||
COPY workshop-baker/target/x86_64-unknown-linux-musl/debug/workshop-baker /baker
|
||||
RUN chmod +x /baker
|
||||
ENTRYPOINT ["/bin/sh", "-c"]
|
||||
Vendored
-6
@@ -1,6 +0,0 @@
|
||||
FROM docker.1ms.run/library/debian:bookworm-slim
|
||||
RUN sed -i 's|deb.debian.org|mirrors.tuna.tsinghua.edu.cn|g' /etc/apt/sources.list.d/debian.sources 2>/dev/null; sed -i 's|security.debian.org|mirrors.tuna.tsinghua.edu.cn/debian-security|g' /etc/apt/sources.list.d/debian.sources 2>/dev/null; true
|
||||
RUN apt-get update -qq && apt-get install -y -qq --no-install-recommends python3 python3-venv ca-certificates 2>&1 | tail -n 2
|
||||
COPY workshop-baker/target/x86_64-unknown-linux-musl/debug/workshop-baker /baker
|
||||
RUN chmod +x /baker
|
||||
ENTRYPOINT ["/bin/sh", "-c"]
|
||||
Vendored
-6
@@ -1,6 +0,0 @@
|
||||
FROM docker.1ms.run/library/fedora:43
|
||||
RUN sed -e 's|^metalink=|#metalink=|g' -e 's|^#baseurl=http://download.example/pub/fedora/linux|baseurl=https://mirrors.tuna.tsinghua.edu.cn/fedora|g' -i.bak /etc/yum.repos.d/fedora.repo /etc/yum.repos.d/fedora-updates.repo
|
||||
RUN dnf install -y python3 2>&1 | tail -n 2
|
||||
COPY workshop-baker/target/x86_64-unknown-linux-musl/debug/workshop-baker /baker
|
||||
RUN chmod +x /baker
|
||||
ENTRYPOINT ["/bin/sh", "-c"]
|
||||
Vendored
-2
@@ -1,2 +0,0 @@
|
||||
FROM docker.1ms.run/library/archlinux:latest
|
||||
RUN echo 'Server = https://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch' > /etc/pacman.d/mirrorlist
|
||||
-8
@@ -1,8 +0,0 @@
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
workspace:
|
||||
path: "/home/vulcan/workspace"
|
||||
fallback: true
|
||||
Vendored
-15
@@ -1,15 +0,0 @@
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
workspace:
|
||||
path: "/home/vulcan/workspace"
|
||||
fallback: true
|
||||
hooks:
|
||||
early:
|
||||
- name: "test-early-hook"
|
||||
command: "echo 'early hook executed'"
|
||||
late:
|
||||
- name: "test-late-hook"
|
||||
command: "echo 'late hook executed'"
|
||||
-3
@@ -1,3 +0,0 @@
|
||||
version: "0.5"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
Vendored
-8
@@ -1,8 +0,0 @@
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
workspace:
|
||||
path: "/home/vulcan/workspace"
|
||||
fallback: true
|
||||
@@ -1,12 +0,0 @@
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
workspace:
|
||||
path: "/home/vulcan/workspace"
|
||||
fallback: true
|
||||
custom: |
|
||||
#!/bin/sh
|
||||
echo 'custom bootstrap executed'
|
||||
mkdir -p /custom-workspace
|
||||
-16
@@ -1,16 +0,0 @@
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
workspace:
|
||||
path: "/home/vulcan/workspace"
|
||||
fallback: true
|
||||
dependencies:
|
||||
system:
|
||||
pacman:
|
||||
packages:
|
||||
- "which"
|
||||
user:
|
||||
cargo:
|
||||
- "ripgrep"
|
||||
-13
@@ -1,13 +0,0 @@
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
workspace:
|
||||
path: "/home/vulcan/workspace"
|
||||
fallback: true
|
||||
envvars:
|
||||
prebake:
|
||||
TEST_PREBAKE_VAR: "prebake_value"
|
||||
bake:
|
||||
TEST_BAKE_VAR: "bake_value"
|
||||
-17
@@ -1,17 +0,0 @@
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
workspace:
|
||||
path: "/home/vulcan/workspace"
|
||||
fallback: true
|
||||
hooks:
|
||||
early:
|
||||
- name: "test-early-hook"
|
||||
command: "echo 'early hook executed'"
|
||||
working_dir: "/workspace"
|
||||
late:
|
||||
- name: "test-late-hook"
|
||||
command: "echo 'late hook executed'"
|
||||
working_dir: "/workspace"
|
||||
-15
@@ -1,15 +0,0 @@
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
workspace:
|
||||
path: "/home/vulcan/workspace"
|
||||
fallback: true
|
||||
dependencies:
|
||||
system:
|
||||
pacman:
|
||||
mirror: "https://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch"
|
||||
packages:
|
||||
- "curl"
|
||||
- "wget"
|
||||
-10
@@ -1,10 +0,0 @@
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
workspace:
|
||||
path: "/home/vulcan/workspace"
|
||||
fallback: true
|
||||
security:
|
||||
drop_after: "Bootstrap"
|
||||
@@ -1,12 +0,0 @@
|
||||
"""
|
||||
LLM-based log judgment system for system tests.
|
||||
|
||||
Uses LLM APIs to evaluate non-deterministic test outputs (logs, stderr, etc.)
|
||||
where simple string matching is insufficient.
|
||||
"""
|
||||
|
||||
from .client import LLMClient
|
||||
from .cache import LLMCache
|
||||
from .judge import judge_log, JudgmentResult
|
||||
|
||||
__all__ = ["LLMClient", "LLMCache", "judge_log", "JudgmentResult"]
|
||||
@@ -1,54 +0,0 @@
|
||||
"""
|
||||
Disk-based cache for LLM API responses.
|
||||
|
||||
Uses SHA256 of (prompt + model + system_prompt) as cache key.
|
||||
Cache files stored as JSON in ``.cache/`` directory.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
class LLMCache:
|
||||
"""SHA256-based disk cache for LLM API responses."""
|
||||
|
||||
def __init__(self, cache_dir: str | None = None):
|
||||
if cache_dir is None:
|
||||
cache_dir = "/tmp/llm-cache"
|
||||
self.cache_dir = Path(cache_dir)
|
||||
self.cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def _make_key(self, prompt: str, model: str, system_prompt: str = "") -> str:
|
||||
"""Create deterministic cache key from request parameters."""
|
||||
data = json.dumps(
|
||||
{"prompt": prompt, "model": model, "system": system_prompt},
|
||||
sort_keys=True,
|
||||
)
|
||||
return hashlib.sha256(data.encode()).hexdigest()
|
||||
|
||||
def get(self, prompt: str, model: str, system_prompt: str = "") -> dict | None:
|
||||
"""Retrieve cached response if exists."""
|
||||
key = self._make_key(prompt, model, system_prompt)
|
||||
path = self.cache_dir / f"{key}.json"
|
||||
if path.exists():
|
||||
try:
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
except (json.JSONDecodeError, IOError):
|
||||
return None
|
||||
return None
|
||||
|
||||
def set(
|
||||
self, prompt: str, model: str, system_prompt: str, response: dict
|
||||
) -> None:
|
||||
"""Store response in cache."""
|
||||
key = self._make_key(prompt, model, system_prompt)
|
||||
path = self.cache_dir / f"{key}.json"
|
||||
with open(path, "w") as f:
|
||||
json.dump(response, f, ensure_ascii=False)
|
||||
|
||||
def clear(self) -> None:
|
||||
"""Clear all cached responses."""
|
||||
for f in self.cache_dir.glob("*.json"):
|
||||
f.unlink()
|
||||
@@ -1,187 +0,0 @@
|
||||
"""
|
||||
LLM client supporting both Anthropic and OpenAI APIs.
|
||||
|
||||
Prefers Anthropic format (per project convention).
|
||||
Falls back to OpenAI format if Anthropic is unavailable.
|
||||
"""
|
||||
|
||||
import os
|
||||
import logging
|
||||
from .cache import LLMCache
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class LLMClient:
|
||||
"""LLM client with caching support. Prefers Anthropic API."""
|
||||
|
||||
def __init__(self, cache: LLMCache | None = None):
|
||||
self.cache = cache or LLMCache()
|
||||
self._anthropic_client = None
|
||||
self._openai_client = None
|
||||
self._init_clients()
|
||||
|
||||
def _init_clients(self):
|
||||
"""Initialize API clients based on available environment variables."""
|
||||
# Anthropic (preferred)
|
||||
anthropic_key = os.environ.get("ANTHROPIC_API_KEY")
|
||||
anthropic_base = os.environ.get("ANTHROPIC_BASE_URL")
|
||||
if anthropic_key:
|
||||
try:
|
||||
import anthropic
|
||||
|
||||
kwargs: dict = {"api_key": anthropic_key}
|
||||
if anthropic_base:
|
||||
kwargs["base_url"] = anthropic_base
|
||||
self._anthropic_client = anthropic.Anthropic(**kwargs)
|
||||
logger.info("Anthropic client initialized")
|
||||
except ImportError:
|
||||
logger.warning("anthropic package not installed")
|
||||
|
||||
# OpenAI (fallback)
|
||||
openai_key = os.environ.get("OPENAI_API_KEY")
|
||||
openai_base = os.environ.get("OPENAI_BASE_URL")
|
||||
if openai_key:
|
||||
try:
|
||||
import openai
|
||||
|
||||
kwargs = {"api_key": openai_key}
|
||||
if openai_base:
|
||||
kwargs["base_url"] = openai_base
|
||||
self._openai_client = openai.OpenAI(**kwargs)
|
||||
logger.info("OpenAI client initialized")
|
||||
except ImportError:
|
||||
logger.warning("openai package not installed")
|
||||
|
||||
@property
|
||||
def available(self) -> bool:
|
||||
"""Check if any LLM client is available."""
|
||||
return (
|
||||
self._anthropic_client is not None or self._openai_client is not None
|
||||
)
|
||||
|
||||
def complete(
|
||||
self, prompt: str, system_prompt: str = "", max_tokens: int = 2000
|
||||
) -> str:
|
||||
"""Send a completion request, with caching.
|
||||
|
||||
Args:
|
||||
prompt: User message.
|
||||
system_prompt: System message (sets role / persona).
|
||||
max_tokens: Max response tokens.
|
||||
|
||||
Returns:
|
||||
LLM response text.
|
||||
|
||||
Raises:
|
||||
RuntimeError: If no client is available.
|
||||
"""
|
||||
model = os.environ.get("MODEL", "deepseek-v4-flash")
|
||||
|
||||
# Check cache
|
||||
cached = self.cache.get(prompt, model, system_prompt)
|
||||
if cached:
|
||||
logger.info("Cache hit for LLM request")
|
||||
return cached["content"]
|
||||
|
||||
# Try Anthropic first
|
||||
if self._anthropic_client:
|
||||
content = self._call_anthropic(
|
||||
prompt, system_prompt, model, max_tokens
|
||||
)
|
||||
elif self._openai_client:
|
||||
content = self._call_openai(prompt, system_prompt, model, max_tokens)
|
||||
else:
|
||||
raise RuntimeError(
|
||||
"No LLM client available. "
|
||||
"Set ANTHROPIC_API_KEY or OPENAI_API_KEY."
|
||||
)
|
||||
|
||||
# Cache the response
|
||||
self.cache.set(prompt, model, system_prompt, {"content": content})
|
||||
return content
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Private helpers
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _call_anthropic(
|
||||
self, prompt: str, system_prompt: str, model: str, max_tokens: int
|
||||
) -> str:
|
||||
"""Call Anthropic API."""
|
||||
kwargs: dict = {
|
||||
"model": model,
|
||||
"max_tokens": max_tokens,
|
||||
"messages": [{"role": "user", "content": prompt}],
|
||||
}
|
||||
if system_prompt:
|
||||
kwargs["system"] = system_prompt
|
||||
|
||||
# Only add reasoning_effort if the model supports it
|
||||
reasoning_effort = os.environ.get("REASONING_EFFORT")
|
||||
if reasoning_effort:
|
||||
try:
|
||||
kwargs["reasoning_effort"] = reasoning_effort
|
||||
response = self._anthropic_client.messages.create(**kwargs)
|
||||
except Exception as e:
|
||||
if (
|
||||
"reasoning_effort" in str(e).lower()
|
||||
or "unexpected" in str(e).lower()
|
||||
):
|
||||
del kwargs["reasoning_effort"]
|
||||
response = self._anthropic_client.messages.create(**kwargs)
|
||||
else:
|
||||
raise
|
||||
else:
|
||||
response = self._anthropic_client.messages.create(**kwargs)
|
||||
|
||||
# Extract text from response content blocks
|
||||
if hasattr(response, "content") and response.content:
|
||||
block = response.content[0]
|
||||
if hasattr(block, "text"):
|
||||
return block.text
|
||||
if hasattr(block, "type") and block.type == "thinking":
|
||||
for b in response.content:
|
||||
if hasattr(b, "text"):
|
||||
return b.text
|
||||
return str(block)
|
||||
return str(response)
|
||||
|
||||
def _call_openai(
|
||||
self, prompt: str, system_prompt: str, model: str, max_tokens: int
|
||||
) -> str:
|
||||
"""Call OpenAI-compatible API."""
|
||||
messages: list[dict] = []
|
||||
if system_prompt:
|
||||
messages.append({"role": "system", "content": system_prompt})
|
||||
messages.append({"role": "user", "content": prompt})
|
||||
|
||||
kwargs: dict = {
|
||||
"model": model,
|
||||
"messages": messages,
|
||||
"max_tokens": max_tokens,
|
||||
}
|
||||
|
||||
# Only add reasoning_effort if supported
|
||||
reasoning_effort = os.environ.get("REASONING_EFFORT")
|
||||
if reasoning_effort:
|
||||
try:
|
||||
kwargs["reasoning_effort"] = reasoning_effort
|
||||
response = self._openai_client.chat.completions.create(
|
||||
**kwargs
|
||||
)
|
||||
except Exception as e:
|
||||
if (
|
||||
"reasoning_effort" in str(e).lower()
|
||||
or "unexpected" in str(e).lower()
|
||||
):
|
||||
del kwargs["reasoning_effort"]
|
||||
response = self._openai_client.chat.completions.create(
|
||||
**kwargs
|
||||
)
|
||||
else:
|
||||
raise
|
||||
else:
|
||||
response = self._openai_client.chat.completions.create(**kwargs)
|
||||
|
||||
return response.choices[0].message.content
|
||||
@@ -1,149 +0,0 @@
|
||||
"""
|
||||
LLM-based log judgment for system tests.
|
||||
|
||||
Evaluates logs against expected behavior using LLM semantic analysis.
|
||||
Returns structured pass/fail results with reasoning.
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
from .client import LLMClient
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
JUDGE_SYSTEM_PROMPT = """\
|
||||
You are a test result judge for a CI/CD system called HoneyBiscuitWorkshop.
|
||||
|
||||
Your job: Evaluate whether actual command output/logs match expected behavior.
|
||||
|
||||
RULES:
|
||||
1. You MUST respond with valid JSON in this exact format:
|
||||
{"verdict": "pass" | "fail", "reason": "brief explanation"}
|
||||
2. "pass" = the output confirms the expected behavior occurred
|
||||
3. "fail" = the output shows the expected behavior did NOT occur, or shows an error
|
||||
4. Ignore irrelevant noise (timestamps, debug lines, unrelated warnings)
|
||||
5. Focus ONLY on whether the specific expected behavior is confirmed or contradicted
|
||||
6. Be strict: if the evidence is ambiguous, verdict is "fail"
|
||||
7. Do NOT include any text outside the JSON object"""
|
||||
|
||||
|
||||
@dataclass
|
||||
class JudgmentResult:
|
||||
"""Result of an LLM log judgment."""
|
||||
|
||||
verdict: str # "pass" or "fail"
|
||||
reason: str
|
||||
raw_response: str
|
||||
|
||||
@property
|
||||
def passed(self) -> bool:
|
||||
return self.verdict == "pass"
|
||||
|
||||
|
||||
def _extract_relevant_logs(full_log: str, max_chars: int = 4000) -> str:
|
||||
"""Extract relevant portions from logs, removing noise.
|
||||
|
||||
Keeps the last *max_chars* characters (most recent / relevant).
|
||||
Removes common noise patterns.
|
||||
"""
|
||||
lines = full_log.strip().split("\n")
|
||||
|
||||
noise_patterns = [
|
||||
r"^\d{4}-\d{2}-\d{2}T",
|
||||
r"^DEBUG ",
|
||||
r"^TRACE ",
|
||||
]
|
||||
filtered = [
|
||||
line
|
||||
for line in lines
|
||||
if not any(re.match(p, line) for p in noise_patterns)
|
||||
]
|
||||
|
||||
text = "\n".join(filtered)
|
||||
if len(text) > max_chars:
|
||||
text = "...\n" + text[-max_chars:]
|
||||
return text
|
||||
|
||||
|
||||
def judge_log(
|
||||
operation: str,
|
||||
expected_behavior: str,
|
||||
actual_log: str,
|
||||
client: LLMClient | None = None,
|
||||
) -> JudgmentResult:
|
||||
"""Judge whether actual logs match expected behavior.
|
||||
|
||||
Args:
|
||||
operation: Description of what was attempted
|
||||
(e.g. ``"prebake bootstrap stage"``).
|
||||
expected_behavior: What should have happened
|
||||
(e.g. ``"user vulcan was created"``).
|
||||
actual_log: The actual stdout / stderr output.
|
||||
client: ``LLMClient`` instance (creates default if ``None``).
|
||||
|
||||
Returns:
|
||||
``JudgmentResult`` with verdict and reason.
|
||||
"""
|
||||
if client is None:
|
||||
client = LLMClient()
|
||||
|
||||
if not client.available:
|
||||
logger.warning("No LLM client available, returning fail verdict")
|
||||
return JudgmentResult(
|
||||
verdict="fail",
|
||||
reason="No LLM client available for judgment",
|
||||
raw_response="",
|
||||
)
|
||||
|
||||
relevant_log = _extract_relevant_logs(actual_log)
|
||||
|
||||
prompt = f"""\
|
||||
## Operation
|
||||
{operation}
|
||||
|
||||
## Expected Behavior
|
||||
{expected_behavior}
|
||||
|
||||
## Actual Output (relevant excerpts)
|
||||
```
|
||||
{relevant_log}
|
||||
```
|
||||
|
||||
Evaluate: Does the actual output confirm the expected behavior occurred?
|
||||
Respond with JSON only."""
|
||||
|
||||
try:
|
||||
response = client.complete(
|
||||
prompt=prompt,
|
||||
system_prompt=JUDGE_SYSTEM_PROMPT,
|
||||
max_tokens=500,
|
||||
)
|
||||
|
||||
# Parse JSON response — handle potential markdown code blocks
|
||||
text = response.strip()
|
||||
if text.startswith("```"):
|
||||
lines = text.split("\n")
|
||||
text = "\n".join(lines[1:-1]) if len(lines) > 2 else text
|
||||
|
||||
result = json.loads(text)
|
||||
return JudgmentResult(
|
||||
verdict=result.get("verdict", "fail"),
|
||||
reason=result.get("reason", "No reason provided"),
|
||||
raw_response=response,
|
||||
)
|
||||
except json.JSONDecodeError as e:
|
||||
logger.error(f"Failed to parse LLM response as JSON: {e}")
|
||||
return JudgmentResult(
|
||||
verdict="fail",
|
||||
reason=f"LLM response was not valid JSON: {response[:200]}",
|
||||
raw_response=response,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"LLM judgment failed: {e}")
|
||||
return JudgmentResult(
|
||||
verdict="fail",
|
||||
reason=f"LLM call failed: {str(e)}",
|
||||
raw_response="",
|
||||
)
|
||||
@@ -1,45 +0,0 @@
|
||||
"""
|
||||
Environment probes for verifying system state in workshop-baker tests.
|
||||
|
||||
Each probe is a standalone function that can be called from any test.
|
||||
Probes return bool or Optional[int/str]; they do not raise on failure.
|
||||
|
||||
A probe accepts an optional *runner* callable (default ``subprocess.run``)
|
||||
that decouples it from the execution context (host or Docker).
|
||||
"""
|
||||
|
||||
from .user import user_exists, user_uid, user_gid, user_in_group, user_shell
|
||||
from .file import (
|
||||
file_exists,
|
||||
file_absent,
|
||||
dir_exists,
|
||||
dir_absent,
|
||||
file_permissions,
|
||||
file_contains,
|
||||
file_owned_by,
|
||||
file_group,
|
||||
)
|
||||
from .package import binary_exists, binary_on_path, package_installed, package_version
|
||||
from .process import process_running, process_count
|
||||
|
||||
__all__ = [
|
||||
"user_exists",
|
||||
"user_uid",
|
||||
"user_gid",
|
||||
"user_in_group",
|
||||
"user_shell",
|
||||
"file_exists",
|
||||
"file_absent",
|
||||
"dir_exists",
|
||||
"dir_absent",
|
||||
"file_permissions",
|
||||
"file_contains",
|
||||
"file_owned_by",
|
||||
"file_group",
|
||||
"package_installed",
|
||||
"package_version",
|
||||
"binary_exists",
|
||||
"binary_on_path",
|
||||
"process_running",
|
||||
"process_count",
|
||||
]
|
||||
@@ -1,53 +0,0 @@
|
||||
"""File and directory existence / permission probes."""
|
||||
|
||||
import os
|
||||
import stat
|
||||
import grp
|
||||
import pwd
|
||||
|
||||
|
||||
def file_exists(path: str) -> bool:
|
||||
"""Check if *path* exists and is a regular file."""
|
||||
return os.path.isfile(path)
|
||||
|
||||
|
||||
def dir_exists(path: str) -> bool:
|
||||
"""Check if *path* exists and is a directory."""
|
||||
return os.path.isdir(path)
|
||||
|
||||
|
||||
def file_permissions(path: str) -> int:
|
||||
"""Return the permission mode of *path* as an octal int (e.g. ``0o755``).
|
||||
|
||||
Raises ``FileNotFoundError`` if *path* doesn't exist.
|
||||
"""
|
||||
return stat.S_IMODE(os.stat(path).st_mode)
|
||||
|
||||
|
||||
def file_contains(path: str, text: str) -> bool:
|
||||
"""Check if a file contains the specified text."""
|
||||
with open(path, "r", errors="replace") as f:
|
||||
return text in f.read()
|
||||
|
||||
|
||||
def file_owned_by(path: str, username: str) -> bool:
|
||||
"""Check if a file is owned by the specified user."""
|
||||
st = os.stat(path)
|
||||
owner = pwd.getpwuid(st.st_uid).pw_name
|
||||
return owner == username
|
||||
|
||||
|
||||
def file_group(path: str) -> str:
|
||||
"""Get the group owner of a file."""
|
||||
st = os.stat(path)
|
||||
return grp.getgrgid(st.st_gid).gr_name
|
||||
|
||||
|
||||
def file_absent(path: str) -> bool:
|
||||
"""Check that *path* does not exist."""
|
||||
return not os.path.lexists(path)
|
||||
|
||||
|
||||
def dir_absent(path: str) -> bool:
|
||||
"""Check that *path* does not exist or is not a directory."""
|
||||
return not os.path.isdir(path)
|
||||
@@ -1,42 +0,0 @@
|
||||
"""Package / binary existence probes (cross-distro).
|
||||
|
||||
Each probe accepts an optional *runner* callable with the same signature
|
||||
as ``subprocess.run``. Defaults to running on the host; for Docker tests,
|
||||
pass a runner that wraps ``docker run --rm <image>``.
|
||||
"""
|
||||
|
||||
import subprocess
|
||||
from typing import Any, Callable
|
||||
|
||||
Runner = Callable[..., Any]
|
||||
|
||||
|
||||
def _run(cmd: list[str], runner: Runner = subprocess.run, **kw: Any) -> Any:
|
||||
opts = {"capture_output": True, "text": True, "timeout": 30}
|
||||
opts.update(kw)
|
||||
return runner(cmd, **opts)
|
||||
|
||||
|
||||
def _succeeded(r: Any) -> bool:
|
||||
""".returncode==0 for CompletedProcess, or ["success"] for dict."""
|
||||
if hasattr(r, "returncode"):
|
||||
return r.returncode == 0
|
||||
return bool(r.get("success", False))
|
||||
|
||||
|
||||
def binary_exists(path: str, runner: Runner = subprocess.run) -> bool:
|
||||
"""Check if a binary exists at *path* (e.g. ``/usr/bin/curl``).
|
||||
|
||||
Uses ``test -f`` (POSIX, guaranteed available).
|
||||
"""
|
||||
return _succeeded(_run(["test", "-f", path], runner))
|
||||
|
||||
|
||||
def binary_on_path(name: str, runner: Runner = subprocess.run) -> bool:
|
||||
"""Check if a command is on PATH (via ``command -v``)."""
|
||||
return _succeeded(_run(["sh", "-c", "command -v " + name], runner))
|
||||
|
||||
|
||||
# Backward-compatible aliases
|
||||
package_installed = binary_exists
|
||||
package_version = binary_on_path # used in __init__ export, now binary-based
|
||||
@@ -1,44 +0,0 @@
|
||||
"""Process existence probes."""
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
|
||||
|
||||
def process_running(process_name: str) -> bool:
|
||||
"""Check if a process with the given name is running."""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["pgrep", "-x", process_name],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
return result.returncode == 0
|
||||
except FileNotFoundError:
|
||||
# Fall back to /proc scan if pgrep not available
|
||||
try:
|
||||
for pid_dir in os.listdir("/proc"):
|
||||
if pid_dir.isdigit():
|
||||
try:
|
||||
with open(f"/proc/{pid_dir}/comm") as f:
|
||||
if f.read().strip() == process_name:
|
||||
return True
|
||||
except (IOError, OSError):
|
||||
continue
|
||||
except OSError:
|
||||
pass
|
||||
return False
|
||||
|
||||
|
||||
def process_count(process_name: str) -> int:
|
||||
"""Count processes with the given name."""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["pgrep", "-c", "-x", process_name],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
if result.returncode == 0:
|
||||
return int(result.stdout.strip())
|
||||
return 0
|
||||
except (FileNotFoundError, ValueError):
|
||||
return 0
|
||||
@@ -1,41 +0,0 @@
|
||||
"""User existence and attribute probes."""
|
||||
|
||||
import pwd
|
||||
import grp
|
||||
|
||||
|
||||
def user_exists(username: str) -> bool:
|
||||
"""Check if a system user exists."""
|
||||
try:
|
||||
pwd.getpwnam(username)
|
||||
return True
|
||||
except KeyError:
|
||||
return False
|
||||
|
||||
|
||||
def user_uid(username: str) -> int | None:
|
||||
try:
|
||||
return pwd.getpwnam(username).pw_uid
|
||||
except KeyError:
|
||||
return None
|
||||
|
||||
def user_gid(username: str) -> int | None:
|
||||
try:
|
||||
return pwd.getpwnam(username).pw_gid
|
||||
except KeyError:
|
||||
return None
|
||||
|
||||
def user_in_group(username: str, groupname: str) -> bool:
|
||||
"""Check if a user is a member of a group."""
|
||||
try:
|
||||
group = grp.getgrnam(groupname)
|
||||
return username in group.gr_mem
|
||||
except KeyError:
|
||||
return False
|
||||
|
||||
|
||||
def user_shell(username: str) -> str | None:
|
||||
try:
|
||||
return pwd.getpwnam(username).pw_shell
|
||||
except KeyError:
|
||||
return None
|
||||
@@ -1,6 +0,0 @@
|
||||
pytest>=8.0.0
|
||||
pytest-timeout>=2.2.0
|
||||
pytest-xdist>=3.5.0
|
||||
anthropic>=0.40.0,<0.90.0
|
||||
openai>=1.50.0,<2.0.0
|
||||
pyyaml>=6.0
|
||||
@@ -1,150 +0,0 @@
|
||||
"""Go UPM system tests — go install + go mod download via bare prebake."""
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import pytest
|
||||
from probes import binary_exists
|
||||
|
||||
|
||||
def _yml(actions_yaml):
|
||||
return """\
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
security:
|
||||
drop_after: never
|
||||
dependencies:
|
||||
config:
|
||||
repology_endpoint: none
|
||||
system:
|
||||
pacman:
|
||||
packages:
|
||||
user:
|
||||
go:
|
||||
""" + _indent(actions_yaml, 6)
|
||||
|
||||
|
||||
def _indent(text, spaces):
|
||||
pad = " " * spaces
|
||||
return "\n".join(pad + line if line.strip() else line
|
||||
for line in text.splitlines())
|
||||
|
||||
|
||||
def _query(cmd, timeout=60):
|
||||
try:
|
||||
r = subprocess.run(cmd, capture_output=True, text=True,
|
||||
timeout=timeout, env={**os.environ, "HOME": "/root"})
|
||||
return {"success": r.returncode == 0, "stdout": r.stdout, "stderr": r.stderr}
|
||||
except Exception as e:
|
||||
return {"success": False, "stdout": "", "stderr": str(e)}
|
||||
|
||||
|
||||
def _run_prebake(run_baker, config_path, timeout, operation, expected):
|
||||
r = run_baker("prebake", config_path, timeout=timeout,
|
||||
env={"RUST_LOG": os.environ.get("RUST_LOG", "info")})
|
||||
full_log = "[stdout]\n{}\n[stderr]\n{}".format(r["stdout"], r["stderr"])
|
||||
return r, full_log
|
||||
|
||||
|
||||
def _check(r, full_log, operation, expected):
|
||||
if r["success"]:
|
||||
return
|
||||
diagnosis = ""
|
||||
try:
|
||||
from llm import judge_log
|
||||
j = judge_log(operation, expected, full_log)
|
||||
diagnosis = "\n LLM diagnosis: [{}] {}".format(j.verdict, j.reason)
|
||||
except Exception:
|
||||
pass
|
||||
pytest.fail("{} failed ({} chars):\n{}\n{}".format(
|
||||
operation, len(full_log), full_log, diagnosis))
|
||||
|
||||
|
||||
def _gopath_bin(name):
|
||||
"""Return the expected $GOPATH/bin path or $HOME/go/bin fallback."""
|
||||
gopath = os.environ.get("GOPATH", "")
|
||||
home = os.environ.get("HOME", "/root")
|
||||
if not gopath:
|
||||
gopath = os.path.join(home, "go")
|
||||
return os.path.join(gopath, "bin", name)
|
||||
|
||||
|
||||
# ── go install ──────────────────────────────────────────────────────
|
||||
|
||||
def test_go_install(run_baker, work_dir):
|
||||
"""go install a small known tool, verify binary exists."""
|
||||
cfg = os.path.join(work_dir, "go.yml")
|
||||
with open(cfg, "w") as f:
|
||||
f.write(_yml("""\
|
||||
proxy: "https://goproxy.cn,direct"
|
||||
packages: ["golang.org/x/tools/cmd/goimports@latest"]
|
||||
"""))
|
||||
|
||||
r, log = _run_prebake(run_baker, cfg, timeout=300,
|
||||
operation="go install goimports",
|
||||
expected="goimports binary appears in GOPATH/bin")
|
||||
_check(r, log, "go install", "goimports installed")
|
||||
|
||||
bin_path = _gopath_bin("goimports")
|
||||
assert binary_exists(bin_path), f"goimports not found at {bin_path}"
|
||||
|
||||
|
||||
# ── go mod download ─────────────────────────────────────────────────
|
||||
|
||||
def test_go_mod_download(run_baker, work_dir):
|
||||
"""Create a minimal go.mod, download deps, verify cache populated."""
|
||||
mod_dir = os.path.join(work_dir, "testmod")
|
||||
os.makedirs(mod_dir, exist_ok=True)
|
||||
with open(os.path.join(mod_dir, "go.mod"), "w") as f:
|
||||
f.write("module test\n\ngo 1.21\n\nrequire golang.org/x/text v0.14.0\n")
|
||||
|
||||
cfg = os.path.join(work_dir, "go.yml")
|
||||
with open(cfg, "w") as f:
|
||||
f.write(_yml("""\
|
||||
proxy: "https://goproxy.cn,direct"
|
||||
modules: ["{mod}"]
|
||||
""".format(mod=os.path.join(mod_dir, "go.mod"))))
|
||||
|
||||
r, log = _run_prebake(run_baker, cfg, timeout=180,
|
||||
operation="go mod download",
|
||||
expected="GOMODCACHE has x/text cached")
|
||||
_check(r, log, "go mod download", "dependencies downloaded to module cache")
|
||||
|
||||
# Check GOMODCACHE or default $GOPATH/pkg/mod
|
||||
gomodcache = os.environ.get("GOMODCACHE", "")
|
||||
if not gomodcache:
|
||||
gopath = os.environ.get("GOPATH",
|
||||
os.path.join(os.environ.get("HOME", "/root"), "go"))
|
||||
gomodcache = os.path.join(gopath, "pkg", "mod")
|
||||
cache_dir = os.path.join(gomodcache, "golang.org", "x")
|
||||
assert os.path.isdir(cache_dir), f"no module cache at {cache_dir}"
|
||||
|
||||
|
||||
# ── go install + mod download ───────────────────────────────────────
|
||||
|
||||
def test_go_full(run_baker, work_dir):
|
||||
"""go install a tool AND download project deps."""
|
||||
mod_dir = os.path.join(work_dir, "testmod2")
|
||||
os.makedirs(mod_dir, exist_ok=True)
|
||||
with open(os.path.join(mod_dir, "go.mod"), "w") as f:
|
||||
f.write("module test\n\ngo 1.21\n")
|
||||
|
||||
cfg = os.path.join(work_dir, "go.yml")
|
||||
with open(cfg, "w") as f:
|
||||
f.write(_yml("""\
|
||||
proxy: "https://goproxy.cn,direct"
|
||||
packages: ["golang.org/x/tools/cmd/goimports@latest"]
|
||||
modules: ["{mod}"]
|
||||
env:
|
||||
GOMODCACHE: "/tmp/gocache"
|
||||
""".format(mod=os.path.join(mod_dir, "go.mod"))))
|
||||
|
||||
r, log = _run_prebake(run_baker, cfg, timeout=300,
|
||||
operation="go install + go mod download",
|
||||
expected="goimports installed, module cache populated")
|
||||
_check(r, log, "go full", "tool installed + deps downloaded")
|
||||
|
||||
assert binary_exists(_gopath_bin("goimports")), "goimports missing"
|
||||
assert os.path.isdir("/tmp/gocache"), "GOMODCACHE not created"
|
||||
@@ -1,163 +0,0 @@
|
||||
"""Python UPM system tests — pip, uv, poetry, pipx via bare prebake."""
|
||||
|
||||
import os
|
||||
import pytest
|
||||
from probes import binary_exists
|
||||
|
||||
RUST_LOG = os.environ.get("RUST_LOG", "info")
|
||||
|
||||
|
||||
def _yml(actions_yaml):
|
||||
return """\
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
security:
|
||||
drop_after: never
|
||||
dependencies:
|
||||
config:
|
||||
repology_endpoint: none
|
||||
system:
|
||||
pacman:
|
||||
packages:
|
||||
user:
|
||||
python:
|
||||
""" + _indent(actions_yaml, 6)
|
||||
|
||||
|
||||
def _indent(text, spaces):
|
||||
pad = " " * spaces
|
||||
return "\n".join(pad + line if line.strip() else line
|
||||
for line in text.splitlines())
|
||||
|
||||
|
||||
def _query(cmd, timeout=60):
|
||||
import subprocess
|
||||
try:
|
||||
r = subprocess.run(cmd, capture_output=True, text=True,
|
||||
timeout=timeout, env={**os.environ, "HOME": "/root"})
|
||||
return {"success": r.returncode == 0, "stdout": r.stdout, "stderr": r.stderr}
|
||||
except Exception as e:
|
||||
return {"success": False, "stdout": "", "stderr": str(e)}
|
||||
|
||||
|
||||
def _run_prebake(run_baker, config_path, timeout, operation, expected):
|
||||
env_vars = {"RUST_LOG": RUST_LOG}
|
||||
r = run_baker("prebake", config_path, timeout=timeout, env=env_vars)
|
||||
full_log = "[stdout]\n{}\n[stderr]\n{}".format(r["stdout"], r["stderr"])
|
||||
return r, full_log
|
||||
|
||||
|
||||
def _check(r, full_log, operation, expected):
|
||||
if r["success"]:
|
||||
return
|
||||
diagnosis = ""
|
||||
try:
|
||||
from llm import judge_log
|
||||
j = judge_log(operation, expected, full_log)
|
||||
diagnosis = "\n LLM diagnosis: [{}] {}".format(j.verdict, j.reason)
|
||||
except Exception:
|
||||
pass
|
||||
pytest.fail("{} failed ({} chars):\n{}\n{}".format(
|
||||
operation, len(full_log), full_log, diagnosis))
|
||||
|
||||
|
||||
# ── pip ─────────────────────────────────────────────────────────────
|
||||
|
||||
def test_pip_install_package(run_baker, work_dir):
|
||||
"""Install a package via pip."""
|
||||
cfg = os.path.join(work_dir, "python.yml")
|
||||
with open(cfg, "w") as f:
|
||||
f.write(_yml("""\
|
||||
- type: pip
|
||||
packages: [requests]
|
||||
index: "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple"
|
||||
args: ["--break-system-packages"]
|
||||
"""))
|
||||
|
||||
r, log = _run_prebake(run_baker, cfg, timeout=120,
|
||||
operation="pip install requests", expected="requests module importable")
|
||||
_check(r, log, "pip install", "pip installs requests successfully")
|
||||
|
||||
r2 = _query(["python3", "-c", "import requests"])
|
||||
assert r2["success"], f"import requests failed: {r2['stderr']}"
|
||||
|
||||
|
||||
# ── pip + venv ──────────────────────────────────────────────────────
|
||||
|
||||
def test_pip_venv_install(run_baker, work_dir):
|
||||
"""Create venv and install inside it."""
|
||||
cfg = os.path.join(work_dir, "python.yml")
|
||||
with open(cfg, "w") as f:
|
||||
f.write(_yml("""\
|
||||
- type: pip
|
||||
directory: "/tmp/test-venv"
|
||||
index: "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple"
|
||||
packages: [requests]
|
||||
"""))
|
||||
|
||||
r, log = _run_prebake(run_baker, cfg, timeout=120,
|
||||
operation="venv + pip install",
|
||||
expected="requests installed inside venv")
|
||||
_check(r, log, "pip venv install", "venv created, requests installed")
|
||||
|
||||
r2 = _query(["/tmp/test-venv/bin/python3", "-c", "import requests"])
|
||||
assert r2["success"], f"import from venv failed: {r2['stderr']}"
|
||||
|
||||
|
||||
# ── pip + requirements ──────────────────────────────────────────────
|
||||
|
||||
def test_pip_requirements_file(run_baker, work_dir):
|
||||
"""Install from a requirements.txt file."""
|
||||
req_path = os.path.join(work_dir, "requirements.txt")
|
||||
with open(req_path, "w") as f:
|
||||
f.write("requests\n")
|
||||
|
||||
cfg = os.path.join(work_dir, "python.yml")
|
||||
with open(cfg, "w") as f:
|
||||
f.write(_yml("""\
|
||||
- type: pip
|
||||
directory: "/tmp/test-venv2"
|
||||
index: "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple"
|
||||
manifest: "{}"
|
||||
""".format(req_path)))
|
||||
|
||||
r, log = _run_prebake(run_baker, cfg, timeout=120,
|
||||
operation="pip install -r requirements.txt",
|
||||
expected="requests installed from requirements file")
|
||||
_check(r, log, "pip -r requirements.txt", "pip installs from requirements file")
|
||||
|
||||
r2 = _query(["/tmp/test-venv2/bin/python3", "-c", "import requests"])
|
||||
assert r2["success"], f"import from venv failed: {r2['stderr']}"
|
||||
|
||||
|
||||
# ── multiple actions ────────────────────────────────────────────────
|
||||
|
||||
def test_multiple_actions(run_baker, work_dir):
|
||||
"""Two pip actions in one config."""
|
||||
cfg = os.path.join(work_dir, "python.yml")
|
||||
with open(cfg, "w") as f:
|
||||
f.write(_yml("""\
|
||||
- type: pip
|
||||
directory: "/tmp/test-venv-a"
|
||||
index: "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple"
|
||||
packages: [requests]
|
||||
args: ["--break-system-packages"]
|
||||
- type: pip
|
||||
directory: "/tmp/test-venv-b"
|
||||
index: "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple"
|
||||
packages: [pytest]
|
||||
"""))
|
||||
|
||||
r, log = _run_prebake(run_baker, cfg, timeout=180,
|
||||
operation="two pip actions",
|
||||
expected="both venvs have their packages")
|
||||
_check(r, log, "multiple pip actions", "both requests and pytest installed")
|
||||
|
||||
r2 = _query(["/tmp/test-venv-a/bin/python3", "-c", "import requests"])
|
||||
assert r2["success"], f"venv-a missing requests: {r2['stderr']}"
|
||||
|
||||
r3 = _query(["/tmp/test-venv-b/bin/python3", "-c", "import pytest"])
|
||||
assert r3["success"], f"venv-b missing pytest: {r3['stderr']}"
|
||||
@@ -1,275 +0,0 @@
|
||||
"""Rust UPM system tests driven by bare workshop-baker prebake.
|
||||
|
||||
These tests exercise the ``user.rust`` UPM through its full lifecycle:
|
||||
1. sysdep injection (rustup via pacman, repology_endpoint: none)
|
||||
2. rustup toolchain / component / target installation
|
||||
3. cargo fetch (project dependency pre-fetch)
|
||||
4. cargo install (global crate tools)
|
||||
|
||||
Observability
|
||||
* ``RUST_LOG=info`` is injected into every baker invocation.
|
||||
* On failure the *full* stdout + stderr is printed (no truncation).
|
||||
* An LLM judge (``tests/llm/judge.py``) diagnoses failures with a
|
||||
human-readable reason, cached per-log-hash to save tokens.
|
||||
|
||||
Network-dependent tests (toolchain download) are skipped by default.
|
||||
Set ``RUST_UPM_NETWORK=1`` to run them.
|
||||
"""
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import pytest
|
||||
from probes import binary_exists
|
||||
|
||||
# ── helpers ─────────────────────────────────────────────────────────
|
||||
|
||||
RUST_LOG = os.environ.get("RUST_LOG", "info")
|
||||
|
||||
|
||||
def _yml(user_section):
|
||||
"""Build a minimal prebake.yml focused on the rust UPM."""
|
||||
return """\
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
security:
|
||||
drop_after: never
|
||||
envvars:
|
||||
prebake:
|
||||
RUSTUP_DIST_SERVER: "https://mirrors.tuna.tsinghua.edu.cn/rustup"
|
||||
|
||||
dependencies:
|
||||
config:
|
||||
repology_endpoint: none
|
||||
system:
|
||||
pacman:
|
||||
packages:
|
||||
user:
|
||||
rust:
|
||||
""" + _indent(user_section, 6)
|
||||
|
||||
|
||||
def _indent(text, spaces):
|
||||
pad = " " * spaces
|
||||
return "\n".join(pad + line if line.strip() else line
|
||||
for line in text.splitlines())
|
||||
|
||||
|
||||
def _cargo_toml(dir_path):
|
||||
path = os.path.join(dir_path, "Cargo.toml")
|
||||
with open(path, "w") as f:
|
||||
f.write('[package]\nname = "test"\nversion = "0.1.0"\nedition = "2021"\n')
|
||||
return path
|
||||
|
||||
|
||||
def _query(cmd, timeout=60):
|
||||
try:
|
||||
r = subprocess.run(cmd, capture_output=True, text=True,
|
||||
timeout=timeout, env={**os.environ, "HOME": "/root"})
|
||||
return {"success": r.returncode == 0, "stdout": r.stdout,
|
||||
"stderr": r.stderr}
|
||||
except Exception as e:
|
||||
return {"success": False, "stdout": "", "stderr": str(e)}
|
||||
|
||||
|
||||
def _run_prebake(run_baker, config_path, timeout, operation, expected):
|
||||
"""Run prebake with RUST_LOG, return (result, full_log)."""
|
||||
env_vars = {"RUST_LOG": RUST_LOG}
|
||||
r = run_baker("prebake", config_path, timeout=timeout, env=env_vars)
|
||||
full_log = "[stdout]\n{}\n[stderr]\n{}".format(r["stdout"], r["stderr"])
|
||||
return r, full_log
|
||||
|
||||
|
||||
def _check(r, full_log, operation, expected):
|
||||
"""Assert success; on failure call LLM judge and show full output."""
|
||||
if r["success"]:
|
||||
return
|
||||
|
||||
diagnosis = ""
|
||||
try:
|
||||
from llm import judge_log
|
||||
j = judge_log(operation, expected, full_log)
|
||||
diagnosis = "\n LLM diagnosis: [{}] {}".format(j.verdict, j.reason)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
pytest.fail(
|
||||
"{} failed ({} chars):\n{}\n{}".format(
|
||||
operation, len(full_log), full_log, diagnosis)
|
||||
)
|
||||
|
||||
|
||||
# ── system dependency resolution ────────────────────────────────────
|
||||
|
||||
|
||||
def test_repology_none_config(run_baker, work_dir):
|
||||
cfg = os.path.join(work_dir, "rust.yml")
|
||||
with open(cfg, "w") as f:
|
||||
f.write(_yml("""\
|
||||
cargo: {}
|
||||
"""))
|
||||
|
||||
r, log = _run_prebake(run_baker, cfg, timeout=120,
|
||||
operation="UPM sysdep: repology_endpoint=none",
|
||||
expected="rustup gets installed via pacman")
|
||||
|
||||
_check(r, log, "repology-none sysdep resolution",
|
||||
"pacman installs rustup, empty cargo config")
|
||||
|
||||
assert binary_exists("/usr/bin/rustup"), "rustup not installed via sysdep"
|
||||
|
||||
|
||||
# ── rustup: toolchain & components ──────────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
"RUST_UPM_NETWORK" not in os.environ,
|
||||
reason="set RUST_UPM_NETWORK=1 for tests that download toolchain/crates"
|
||||
)
|
||||
def test_toolchain_and_component(run_baker, work_dir):
|
||||
cfg = os.path.join(work_dir, "rust.yml")
|
||||
with open(cfg, "w") as f:
|
||||
f.write(_yml("""\
|
||||
toolchain: "stable"
|
||||
components: ["rustfmt"]
|
||||
"""))
|
||||
|
||||
r, log = _run_prebake(run_baker, cfg, timeout=600,
|
||||
operation="rustup default stable + component add rustfmt",
|
||||
expected="stable toolchain active, rustfmt installed")
|
||||
|
||||
_check(r, log, "toolchain + rustfmt installation",
|
||||
"rustup default stable succeeds; rustfmt available")
|
||||
|
||||
assert binary_exists("/usr/bin/rustup"), "rustup not installed"
|
||||
r2 = _query(["rustup", "which", "rustc"])
|
||||
assert r2["success"], "rustc not found via rustup which"
|
||||
assert os.path.isfile(r2["stdout"].strip()), "rustc binary missing"
|
||||
|
||||
|
||||
# ── cross-compilation targets ───────────────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
"RUST_UPM_NETWORK" not in os.environ,
|
||||
reason="set RUST_UPM_NETWORK=1 for tests that download toolchain/crates"
|
||||
)
|
||||
def test_cross_compile_target(run_baker, work_dir):
|
||||
cfg = os.path.join(work_dir, "rust.yml")
|
||||
with open(cfg, "w") as f:
|
||||
f.write(_yml("""\
|
||||
toolchain: "stable"
|
||||
targets: ["wasm32-unknown-unknown"]
|
||||
"""))
|
||||
|
||||
r, log = _run_prebake(run_baker, cfg, timeout=600,
|
||||
operation="rustup target add wasm32-unknown-unknown",
|
||||
expected="wasm32 target appears in installed list")
|
||||
|
||||
_check(r, log, "cross-compilation target installation",
|
||||
"wasm32-unknown-unknown is installed")
|
||||
|
||||
r2 = _query(["rustup", "target", "list", "--installed"])
|
||||
assert r2["success"], "rustup target list failed"
|
||||
assert "wasm32-unknown-unknown" in r2["stdout"], "wasm32 target missing"
|
||||
|
||||
|
||||
# ── cargo fetch ─────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
"RUST_UPM_NETWORK" not in os.environ,
|
||||
reason="set RUST_UPM_NETWORK=1 for tests that download toolchain/crates"
|
||||
)
|
||||
def test_cargo_fetch(run_baker, work_dir):
|
||||
manifest = _cargo_toml(work_dir)
|
||||
cfg = os.path.join(work_dir, "rust.yml")
|
||||
with open(cfg, "w") as f:
|
||||
f.write(_yml("""\
|
||||
toolchain: "stable"
|
||||
cargo:
|
||||
fetch: true
|
||||
locked: true
|
||||
manifests: ["{manifest}"]
|
||||
""".format(manifest=manifest)))
|
||||
|
||||
r, log = _run_prebake(run_baker, cfg, timeout=600,
|
||||
operation="cargo fetch on minimal Cargo.toml",
|
||||
expected="~/.cargo/registry/cache populated")
|
||||
|
||||
_check(r, log, "cargo fetch dependency pre-fetch",
|
||||
"cargo fetch succeeds, registry cache non-empty")
|
||||
|
||||
cache = os.path.expanduser("~/.cargo/registry/cache")
|
||||
assert os.path.isdir(cache), "registry/cache missing"
|
||||
assert len(os.listdir(cache)) > 0, "registry/cache empty after fetch"
|
||||
|
||||
|
||||
# ── cargo install ───────────────────────────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
"RUST_UPM_NETWORK" not in os.environ,
|
||||
reason="set RUST_UPM_NETWORK=1 for tests that download toolchain/crates"
|
||||
)
|
||||
def test_cargo_install(run_baker, work_dir):
|
||||
cfg = os.path.join(work_dir, "rust.yml")
|
||||
with open(cfg, "w") as f:
|
||||
f.write(_yml("""\
|
||||
toolchain: "stable"
|
||||
cargo:
|
||||
packages: ["cargo-audit"]
|
||||
locked: true
|
||||
"""))
|
||||
|
||||
r, log = _run_prebake(run_baker, cfg, timeout=1200,
|
||||
operation="cargo install cargo-audit",
|
||||
expected="cargo-audit binary in ~/.cargo/bin/")
|
||||
|
||||
_check(r, log, "cargo install crate",
|
||||
"cargo-audit compiled and installed")
|
||||
|
||||
assert binary_exists(
|
||||
os.path.expanduser("~/.cargo/bin/cargo-audit")
|
||||
), "cargo-audit not installed"
|
||||
|
||||
|
||||
# ── full flow ───────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
"RUST_UPM_NETWORK" not in os.environ,
|
||||
reason="set RUST_UPM_NETWORK=1 for tests that download toolchain/crates"
|
||||
)
|
||||
def test_full_flow(run_baker, work_dir):
|
||||
manifest = _cargo_toml(work_dir)
|
||||
cfg = os.path.join(work_dir, "rust.yml")
|
||||
with open(cfg, "w") as f:
|
||||
f.write(_yml("""\
|
||||
toolchain: "stable"
|
||||
components: ["rustfmt"]
|
||||
targets: ["wasm32-unknown-unknown"]
|
||||
cargo:
|
||||
fetch: true
|
||||
locked: true
|
||||
manifests: ["{manifest}"]
|
||||
packages: ["cargo-watch"]
|
||||
""".format(manifest=manifest)))
|
||||
|
||||
r, log = _run_prebake(run_baker, cfg, timeout=1200,
|
||||
operation="full Rust UPM flow",
|
||||
expected="all stages succeed")
|
||||
|
||||
_check(r, log, "full Rust UPM end-to-end",
|
||||
"toolchain + cross target + cargo fetch + cargo install all pass")
|
||||
|
||||
assert binary_exists("/usr/bin/rustup")
|
||||
r2 = _query(["rustup", "which", "rustc"])
|
||||
assert r2["success"]
|
||||
r3 = _query(["rustup", "target", "list", "--installed"])
|
||||
assert "wasm32-unknown-unknown" in r3["stdout"]
|
||||
cache = os.path.expanduser("~/.cargo/registry/cache")
|
||||
assert os.path.isdir(cache) and len(os.listdir(cache)) > 0
|
||||
assert binary_exists(os.path.expanduser("~/.cargo/bin/cargo-watch"))
|
||||
Generated
+107
@@ -0,0 +1,107 @@
|
||||
# This file is automatically @generated by Cargo.
|
||||
# It is not intended for manual editing.
|
||||
version = 4
|
||||
|
||||
[[package]]
|
||||
name = "itoa"
|
||||
version = "1.0.18"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
|
||||
|
||||
[[package]]
|
||||
name = "memchr"
|
||||
version = "2.8.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f8ca58f447f06ed17d5fc4043ce1b10dd205e060fb3ce5b979b8ed8e59ff3f79"
|
||||
|
||||
[[package]]
|
||||
name = "proc-macro2"
|
||||
version = "1.0.106"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934"
|
||||
dependencies = [
|
||||
"unicode-ident",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "quote"
|
||||
version = "1.0.45"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde"
|
||||
version = "1.0.228"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e"
|
||||
dependencies = [
|
||||
"serde_core",
|
||||
"serde_derive",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde_core"
|
||||
version = "1.0.228"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad"
|
||||
dependencies = [
|
||||
"serde_derive",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde_derive"
|
||||
version = "1.0.228"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde_json"
|
||||
version = "1.0.149"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "83fc039473c5595ace860d8c4fafa220ff474b3fc6bfdb4293327f1a37e94d86"
|
||||
dependencies = [
|
||||
"itoa",
|
||||
"memchr",
|
||||
"serde",
|
||||
"serde_core",
|
||||
"zmij",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "syn"
|
||||
version = "2.0.117"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"unicode-ident",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "unicode-ident"
|
||||
version = "1.0.24"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75"
|
||||
|
||||
[[package]]
|
||||
name = "workshop-baker-params"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"serde",
|
||||
"serde_json",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "zmij"
|
||||
version = "1.0.21"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa"
|
||||
@@ -0,0 +1,8 @@
|
||||
[package]
|
||||
name = "workshop-baker-params"
|
||||
version = "0.1.0"
|
||||
edition = "2024"
|
||||
|
||||
[dependencies]
|
||||
serde = { version = "1.0.228", features = ["derive"] }
|
||||
serde_json = "1.0.149"
|
||||
@@ -0,0 +1,24 @@
|
||||
use crate::apply_field;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct BakeParams {
|
||||
#[serde(default)] pub workspace: ParamVal<String>,
|
||||
}
|
||||
|
||||
impl MergeParams for BakeParams {
|
||||
fn defaults() -> Self { Self { workspace: ParamVal::new("/workspace".into()) } }
|
||||
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
|
||||
apply_field!(self.workspace, i.workspace, w, ov, p, "workspace");
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for BakeParams { fn default() -> Self { Self::defaults() } }
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct BakeSettings { pub workspace: String }
|
||||
impl Default for BakeSettings { fn default() -> Self { BakeParams::defaults().into() } }
|
||||
impl From<&BakeParams> for BakeSettings { fn from(p: &BakeParams) -> Self { Self { workspace: p.workspace.value.clone() } } }
|
||||
impl From<BakeParams> for BakeSettings { fn from(p: BakeParams) -> Self { (&p).into() } }
|
||||
@@ -0,0 +1,23 @@
|
||||
use crate::apply_field;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct FinalizeParams {
|
||||
#[serde(default)] pub artifact_retention_days: ParamVal<u32>,
|
||||
}
|
||||
|
||||
impl MergeParams for FinalizeParams {
|
||||
fn defaults() -> Self { Self { artifact_retention_days: ParamVal::new(7) } }
|
||||
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
|
||||
apply_field!(self.artifact_retention_days, i.artifact_retention_days, w, ov, p, "artifact_retention_days");
|
||||
}
|
||||
}
|
||||
impl Default for FinalizeParams { fn default() -> Self { Self::defaults() } }
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct FinalizeSettings { pub artifact_retention_days: u32 }
|
||||
impl Default for FinalizeSettings { fn default() -> Self { FinalizeParams::defaults().into() } }
|
||||
impl From<&FinalizeParams> for FinalizeSettings { fn from(p: &FinalizeParams) -> Self { Self { artifact_retention_days: p.artifact_retention_days.value } } }
|
||||
impl From<FinalizeParams> for FinalizeSettings { fn from(p: FinalizeParams) -> Self { (&p).into() } }
|
||||
@@ -0,0 +1,168 @@
|
||||
pub mod notification;
|
||||
pub mod prebake;
|
||||
pub mod bake;
|
||||
pub mod finalize;
|
||||
pub mod writer;
|
||||
pub mod param;
|
||||
pub mod traits;
|
||||
|
||||
pub use notification::{NotificationParams, NotificationSettings};
|
||||
pub use prebake::{PrebakeParams, PrebakeSettings};
|
||||
pub use bake::{BakeParams, BakeSettings};
|
||||
pub use finalize::{FinalizeParams, FinalizeSettings};
|
||||
pub use writer::Writer;
|
||||
pub use param::ParamVal;
|
||||
pub use traits::MergeParams;
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::collections::HashMap;
|
||||
|
||||
/// Root parameter tree. Each component gets its own subtree.
|
||||
/// Used for both IPC (Serialized to JSON) and in-process merging.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||
pub struct Params {
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub notification: Option<NotificationParams>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub prebake: Option<PrebakeParams>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub bake: Option<BakeParams>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub finalize: Option<FinalizeParams>,
|
||||
}
|
||||
|
||||
/// Apply and merge helpers.
|
||||
impl Params {
|
||||
/// Merge a layer from a specific writer into the current params.
|
||||
/// Returns entries that were overridden (for debug).
|
||||
pub fn apply(&mut self, incoming: Params, writer: Writer) -> Overridden {
|
||||
let mut ov = Overridden::new();
|
||||
if let Some(l) = incoming.notification {
|
||||
if let Some(ref mut b) = self.notification {
|
||||
b.apply(l, writer, &mut ov, "notification.");
|
||||
} else {
|
||||
self.notification = Some(l);
|
||||
}
|
||||
}
|
||||
if let Some(l) = incoming.prebake {
|
||||
if let Some(ref mut b) = self.prebake {
|
||||
b.apply(l, writer, &mut ov, "prebake.");
|
||||
} else {
|
||||
self.prebake = Some(l);
|
||||
}
|
||||
}
|
||||
if let Some(l) = incoming.bake {
|
||||
if let Some(ref mut b) = self.bake {
|
||||
b.apply(l, writer, &mut ov, "bake.");
|
||||
} else {
|
||||
self.bake = Some(l);
|
||||
}
|
||||
}
|
||||
if let Some(l) = incoming.finalize {
|
||||
if let Some(ref mut b) = self.finalize {
|
||||
b.apply(l, writer, &mut ov, "finalize.");
|
||||
} else {
|
||||
self.finalize = Some(l);
|
||||
}
|
||||
}
|
||||
ov
|
||||
}
|
||||
|
||||
/// Convert to baker-consumable settings. All params guaranteed to have values.
|
||||
pub fn to_settings(&self) -> Result<AllSettings, String> {
|
||||
Ok(AllSettings {
|
||||
notification: self.notification.as_ref().map(NotificationSettings::from).unwrap_or_default(),
|
||||
prebake: self.prebake.as_ref().map(PrebakeSettings::from).unwrap_or_default(),
|
||||
bake: self.bake.as_ref().map(BakeSettings::from).unwrap_or_default(),
|
||||
finalize: self.finalize.as_ref().map(FinalizeSettings::from).unwrap_or_default(),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// Plain-value settings for baker consumption. No metadata, no Option.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct AllSettings {
|
||||
pub notification: NotificationSettings,
|
||||
pub prebake: PrebakeSettings,
|
||||
pub bake: BakeSettings,
|
||||
pub finalize: FinalizeSettings,
|
||||
}
|
||||
|
||||
/// Debug — values that were overridden during merge, keyed by dot-path.
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct Overridden {
|
||||
pub entries: std::collections::HashMap<String, serde_json::Value>,
|
||||
}
|
||||
|
||||
impl Overridden {
|
||||
pub fn new() -> Self { Self { entries: HashMap::new() } }
|
||||
pub fn is_empty(&self) -> bool { self.entries.is_empty() }
|
||||
}
|
||||
|
||||
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_empty_params_serializes_empty() {
|
||||
let p = Params::default();
|
||||
let json = serde_json::to_string(&p).unwrap();
|
||||
assert_eq!(json, "{}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_notification_params_serialize() {
|
||||
let mut p = Params::default();
|
||||
p.notification = Some(NotificationParams::defaults());
|
||||
let json = serde_json::to_string_pretty(&p).unwrap();
|
||||
assert!(json.contains("max_lives"));
|
||||
assert!(json.contains("3"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_apply_courier_overrides_default() {
|
||||
let mut base = Params::default();
|
||||
base.notification = Some(NotificationParams::defaults());
|
||||
let incoming = Params {
|
||||
notification: Some(NotificationParams {
|
||||
max_lives: ParamVal::with(5, Writer::Courier),
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
};
|
||||
let ov = base.apply(incoming, Writer::Courier);
|
||||
let s = base.to_settings().unwrap();
|
||||
assert_eq!(s.notification.max_lives, 5);
|
||||
assert!(ov.is_empty()); // default had no writer, not an "override"
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_apply_upstream_wins() {
|
||||
let mut base = Params::default();
|
||||
base.notification = Some(NotificationParams {
|
||||
max_lives: ParamVal::with(3, Writer::Base),
|
||||
..Default::default()
|
||||
});
|
||||
let incoming = Params {
|
||||
notification: Some(NotificationParams {
|
||||
max_lives: ParamVal::with(5, Writer::Courier),
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
};
|
||||
let ov = base.apply(incoming, Writer::Courier);
|
||||
let s = base.to_settings().unwrap();
|
||||
// base(3) > courier(2) — rejected, base stays
|
||||
assert_eq!(s.notification.max_lives, 3);
|
||||
assert!(ov.is_empty()); // nothing overridden
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_deny_unknown_fields() {
|
||||
let bad = r#"{"max_lives":{"value":5},"ghost_param":{"value":1}}"#;
|
||||
let result: Result<NotificationParams, _> = serde_json::from_str(bad);
|
||||
assert!(result.is_err()); // unknown field should be denied
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
use crate::apply_field;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct NotificationParams {
|
||||
#[serde(default)] pub max_lives: ParamVal<u32>,
|
||||
#[serde(default)] pub timeout_ms: ParamVal<u64>,
|
||||
#[serde(default)] pub retry_backoff: ParamVal<f64>,
|
||||
#[serde(default)] pub retry_delay_seconds: ParamVal<f64>,
|
||||
#[serde(default)] pub max_retry_delay_seconds: ParamVal<f64>,
|
||||
#[serde(default)] pub life_impact_factor: ParamVal<f64>,
|
||||
#[serde(default)] pub batch_period: ParamVal<u32>,
|
||||
#[serde(default)] pub batch_watermark: ParamVal<u32>,
|
||||
#[serde(default)] pub template_ref_depth: ParamVal<u32>,
|
||||
}
|
||||
|
||||
impl MergeParams for NotificationParams {
|
||||
fn defaults() -> Self {
|
||||
Self {
|
||||
max_lives: ParamVal::new(3), timeout_ms: ParamVal::new(30000),
|
||||
retry_backoff: ParamVal::new(2.0), retry_delay_seconds: ParamVal::new(5.0),
|
||||
max_retry_delay_seconds: ParamVal::new(300.0),
|
||||
life_impact_factor: ParamVal::new(0.0),
|
||||
batch_period: ParamVal::new(30), batch_watermark: ParamVal::new(5),
|
||||
template_ref_depth: ParamVal::new(10),
|
||||
}
|
||||
}
|
||||
fn apply(&mut self, incoming: Self, writer: Writer, ov: &mut Overridden, prefix: &str) {
|
||||
apply_field!(self.max_lives, incoming.max_lives, writer, ov, prefix, "max_lives");
|
||||
apply_field!(self.timeout_ms, incoming.timeout_ms, writer, ov, prefix, "timeout_ms");
|
||||
apply_field!(self.retry_backoff, incoming.retry_backoff, writer, ov, prefix, "retry_backoff");
|
||||
apply_field!(self.retry_delay_seconds, incoming.retry_delay_seconds, writer, ov, prefix, "retry_delay_seconds");
|
||||
apply_field!(self.max_retry_delay_seconds, incoming.max_retry_delay_seconds, writer, ov, prefix, "max_retry_delay_seconds");
|
||||
apply_field!(self.life_impact_factor, incoming.life_impact_factor, writer, ov, prefix, "life_impact_factor");
|
||||
apply_field!(self.batch_period, incoming.batch_period, writer, ov, prefix, "batch_period");
|
||||
apply_field!(self.batch_watermark, incoming.batch_watermark, writer, ov, prefix, "batch_watermark");
|
||||
apply_field!(self.template_ref_depth, incoming.template_ref_depth, writer, ov, prefix, "template_ref_depth");
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for NotificationParams { fn default() -> Self { Self::defaults() } }
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct NotificationSettings {
|
||||
pub max_lives: u32, pub timeout_ms: u64, pub retry_backoff: f64,
|
||||
pub retry_delay_seconds: f64, pub max_retry_delay_seconds: f64,
|
||||
pub life_impact_factor: f64,
|
||||
pub batch_period: u32, pub batch_watermark: u32, pub template_ref_depth: u32,
|
||||
}
|
||||
|
||||
impl Default for NotificationSettings { fn default() -> Self { NotificationParams::defaults().into() } }
|
||||
|
||||
impl From<&NotificationParams> for NotificationSettings {
|
||||
fn from(p: &NotificationParams) -> Self { Self {
|
||||
max_lives: p.max_lives.value, timeout_ms: p.timeout_ms.value,
|
||||
retry_backoff: p.retry_backoff.value, retry_delay_seconds: p.retry_delay_seconds.value,
|
||||
max_retry_delay_seconds: p.max_retry_delay_seconds.value,
|
||||
life_impact_factor: p.life_impact_factor.value,
|
||||
batch_period: p.batch_period.value, batch_watermark: p.batch_watermark.value,
|
||||
template_ref_depth: p.template_ref_depth.value,
|
||||
}}
|
||||
}
|
||||
impl From<NotificationParams> for NotificationSettings { fn from(p: NotificationParams) -> Self { (&p).into() } }
|
||||
@@ -0,0 +1,32 @@
|
||||
use serde::{Deserialize, Serialize};
|
||||
use crate::Writer;
|
||||
|
||||
/// A strongly-typed parameter value with writer tracking.
|
||||
///
|
||||
/// Serializes as `{ "value": 3, "writer": "base" }`.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct ParamVal<T> {
|
||||
pub value: T,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub writer: Option<Writer>,
|
||||
}
|
||||
|
||||
impl<T: Default> Default for ParamVal<T> {
|
||||
fn default() -> Self {
|
||||
ParamVal { value: T::default(), writer: None }
|
||||
}
|
||||
}
|
||||
|
||||
impl<T> ParamVal<T> {
|
||||
pub fn new(value: T) -> Self {
|
||||
ParamVal { value, writer: None }
|
||||
}
|
||||
pub fn with(value: T, writer: Writer) -> Self {
|
||||
ParamVal { value, writer: Some(writer) }
|
||||
}
|
||||
/// Whether an incoming value from `incoming_writer` should override this one.
|
||||
pub fn should_override(&self, incoming: Writer) -> bool {
|
||||
let current = self.writer.map_or(0, |w| w.priority());
|
||||
incoming.priority() > current
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
pub mod network;
|
||||
pub mod cache;
|
||||
pub mod engine;
|
||||
|
||||
use crate::apply_field;
|
||||
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct PrebakeParams {
|
||||
#[serde(default)] pub bootstrap_user: ParamVal<String>,
|
||||
#[serde(default)] pub workspace: ParamVal<String>,
|
||||
#[serde(default)] pub network: network::NetworkParams,
|
||||
#[serde(default)] pub cache: cache::CacheParams,
|
||||
#[serde(default)] pub engine: engine::EngineParams,
|
||||
#[serde(default)] pub repology_endpoint: ParamVal<String>,
|
||||
}
|
||||
|
||||
impl MergeParams for PrebakeParams {
|
||||
fn defaults() -> Self {
|
||||
Self {
|
||||
bootstrap_user: ParamVal::new("vulcan".into()), workspace: ParamVal::new("/home/vulcan/workspace".into()),
|
||||
network: network::NetworkParams::defaults(), cache: cache::CacheParams::defaults(),
|
||||
engine: engine::EngineParams::defaults(),
|
||||
repology_endpoint: ParamVal::new("default".into()),
|
||||
}
|
||||
}
|
||||
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
|
||||
apply_field!(self.bootstrap_user, i.bootstrap_user, w, ov, p, "bootstrap_user");
|
||||
apply_field!(self.workspace, i.workspace, w, ov, p, "workspace");
|
||||
apply_field!(self.repology_endpoint, i.repology_endpoint, w, ov, p, "repology_endpoint");
|
||||
self.network.apply(i.network, w, ov, &format!("{}network.", p));
|
||||
self.cache.apply(i.cache, w, ov, &format!("{}cache.", p));
|
||||
self.engine.apply(i.engine, w, ov, &format!("{}engine.", p));
|
||||
}
|
||||
}
|
||||
impl Default for PrebakeParams { fn default() -> Self { Self::defaults() } }
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct PrebakeSettings {
|
||||
pub bootstrap_user: String, pub workspace: String,
|
||||
pub network: network::NetworkSettings, pub cache: cache::CacheSettings,
|
||||
pub engine: engine::EngineSettings, pub repology_endpoint: String,
|
||||
}
|
||||
impl Default for PrebakeSettings { fn default() -> Self { PrebakeParams::defaults().into() } }
|
||||
impl From<&PrebakeParams> for PrebakeSettings {
|
||||
fn from(p: &PrebakeParams) -> Self { Self {
|
||||
bootstrap_user: p.bootstrap_user.value.clone(), workspace: p.workspace.value.clone(),
|
||||
network: (&p.network).into(), cache: (&p.cache).into(), engine: (&p.engine).into(),
|
||||
repology_endpoint: p.repology_endpoint.value.clone(),
|
||||
}}
|
||||
}
|
||||
impl From<PrebakeParams> for PrebakeSettings { fn from(p: PrebakeParams) -> Self { (&p).into() } }
|
||||
@@ -0,0 +1,37 @@
|
||||
use crate::apply_field;
|
||||
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct CacheParams {
|
||||
#[serde(default)] pub strategy: ParamVal<String>,
|
||||
#[serde(default)] pub mode: ParamVal<String>,
|
||||
#[serde(default)] pub ttl_days: ParamVal<u32>,
|
||||
#[serde(default)] pub max_size_gb: ParamVal<u32>,
|
||||
#[serde(default)] pub cleanup_policy: ParamVal<String>,
|
||||
}
|
||||
impl MergeParams for CacheParams {
|
||||
fn defaults() -> Self {
|
||||
Self {
|
||||
strategy: ParamVal::new("always".into()), mode: ParamVal::new("zstd".into()),
|
||||
ttl_days: ParamVal::new(30), max_size_gb: ParamVal::new(20),
|
||||
cleanup_policy: ParamVal::new("lru".into()),
|
||||
}
|
||||
}
|
||||
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
|
||||
apply_field!(self.strategy, i.strategy, w, ov, p, "strategy");
|
||||
apply_field!(self.mode, i.mode, w, ov, p, "mode");
|
||||
apply_field!(self.ttl_days, i.ttl_days, w, ov, p, "ttl_days");
|
||||
apply_field!(self.max_size_gb, i.max_size_gb, w, ov, p, "max_size_gb");
|
||||
apply_field!(self.cleanup_policy, i.cleanup_policy, w, ov, p, "cleanup_policy");
|
||||
}
|
||||
}
|
||||
impl Default for CacheParams { fn default() -> Self { Self::defaults() } }
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct CacheSettings { pub strategy: String, pub mode: String, pub ttl_days: u32, pub max_size_gb: u32, pub cleanup_policy: String }
|
||||
impl Default for CacheSettings { fn default() -> Self { CacheParams::defaults().into() } }
|
||||
impl From<&CacheParams> for CacheSettings { fn from(p: &CacheParams) -> Self {
|
||||
Self { strategy: p.strategy.value.clone(), mode: p.mode.value.clone(), ttl_days: p.ttl_days.value, max_size_gb: p.max_size_gb.value, cleanup_policy: p.cleanup_policy.value.clone() }
|
||||
} }
|
||||
impl From<CacheParams> for CacheSettings { fn from(p: CacheParams) -> Self { (&p).into() } }
|
||||
@@ -0,0 +1,21 @@
|
||||
use crate::apply_field;
|
||||
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct EngineParams {
|
||||
#[serde(default)] pub timeout_ms: ParamVal<u64>,
|
||||
}
|
||||
impl MergeParams for EngineParams {
|
||||
fn defaults() -> Self { Self { timeout_ms: ParamVal::new(300000) } }
|
||||
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
|
||||
apply_field!(self.timeout_ms, i.timeout_ms, w, ov, p, "timeout_ms");
|
||||
}
|
||||
}
|
||||
impl Default for EngineParams { fn default() -> Self { Self::defaults() } }
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct EngineSettings { pub timeout_ms: u64 }
|
||||
impl Default for EngineSettings { fn default() -> Self { EngineParams::defaults().into() } }
|
||||
impl From<&EngineParams> for EngineSettings { fn from(p: &EngineParams) -> Self { Self { timeout_ms: p.timeout_ms.value } } }
|
||||
impl From<EngineParams> for EngineSettings { fn from(p: EngineParams) -> Self { (&p).into() } }
|
||||
@@ -0,0 +1,23 @@
|
||||
use crate::apply_field;
|
||||
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct NetworkParams {
|
||||
#[serde(default)] pub enabled: ParamVal<bool>,
|
||||
#[serde(default)] pub outbound: ParamVal<bool>,
|
||||
}
|
||||
impl MergeParams for NetworkParams {
|
||||
fn defaults() -> Self { Self { enabled: ParamVal::new(true), outbound: ParamVal::new(true) } }
|
||||
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
|
||||
apply_field!(self.enabled, i.enabled, w, ov, p, "enabled");
|
||||
apply_field!(self.outbound, i.outbound, w, ov, p, "outbound");
|
||||
}
|
||||
}
|
||||
impl Default for NetworkParams { fn default() -> Self { Self::defaults() } }
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct NetworkSettings { pub enabled: bool, pub outbound: bool }
|
||||
impl Default for NetworkSettings { fn default() -> Self { NetworkParams::defaults().into() } }
|
||||
impl From<&NetworkParams> for NetworkSettings { fn from(p: &NetworkParams) -> Self { Self { enabled: p.enabled.value, outbound: p.outbound.value } } }
|
||||
impl From<NetworkParams> for NetworkSettings { fn from(p: NetworkParams) -> Self { (&p).into() } }
|
||||
@@ -0,0 +1,28 @@
|
||||
use crate::{Writer, Overridden};
|
||||
|
||||
/// A parameter tree node that can merge itself with an incoming layer.
|
||||
pub trait MergeParams: Sized {
|
||||
fn apply(&mut self, incoming: Self, writer: Writer, ov: &mut Overridden, prefix: &str);
|
||||
fn defaults() -> Self;
|
||||
}
|
||||
|
||||
/// Apply one field's merge logic.
|
||||
/// ```ignore
|
||||
/// apply_field!(self.foo, incoming.foo, writer, ov, prefix, "foo");
|
||||
/// ```
|
||||
#[macro_export]
|
||||
macro_rules! apply_field {
|
||||
($self:ident.$field:ident, $incoming:ident.$field2:ident, $writer:expr, $ov:ident, $prefix:expr, $name:expr) => {
|
||||
let old_w = $self.$field.writer;
|
||||
if $self.$field.should_override($writer) {
|
||||
let old_v = ::std::mem::replace(&mut $self.$field.value, $incoming.$field2.value);
|
||||
$self.$field.writer = ::std::option::Option::Some($writer);
|
||||
if let ::std::option::Option::Some(w) = old_w {
|
||||
$ov.entries.insert(
|
||||
format!("{}{}", $prefix, $name),
|
||||
::serde_json::json!({"value": old_v, "writer": w.as_str()}),
|
||||
);
|
||||
}
|
||||
}
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub enum Writer {
|
||||
Base,
|
||||
Courier,
|
||||
Bakerd,
|
||||
}
|
||||
|
||||
impl Writer {
|
||||
/// Priority: Base=3 > Courier=2 > Bakerd=1
|
||||
pub fn priority(self) -> u8 {
|
||||
match self {
|
||||
Writer::Base => 3,
|
||||
Writer::Courier => 2,
|
||||
Writer::Bakerd => 1,
|
||||
}
|
||||
}
|
||||
pub fn as_str(self) -> &'static str {
|
||||
match self {
|
||||
Writer::Base => "base", Writer::Courier => "courier", Writer::Bakerd => "bakerd",
|
||||
}
|
||||
}
|
||||
}
|
||||
impl Default for Writer { fn default() -> Self { Writer::Base } }
|
||||
@@ -1,3 +1,4 @@
|
||||
temp
|
||||
examples
|
||||
quicktest.sh
|
||||
_env.sh
|
||||
|
||||
Generated
+84
-1471
File diff suppressed because it is too large
Load Diff
@@ -47,13 +47,10 @@ uuid = { version = "1.19.0", features = ["v4"] }
|
||||
libc = "0.2"
|
||||
os_info = "3.14.0"
|
||||
privdrop = "0.5.6"
|
||||
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
|
||||
which = "8.0.2"
|
||||
gix = "0.81.0"
|
||||
sha2 = "0.10"
|
||||
flate2 = "1.0"
|
||||
zstd = "0.13"
|
||||
globset = "0.4.18"
|
||||
axum = "0.7"
|
||||
workshop-baker-params = { version = "0.1.0", path = "../workshop-baker-params" }
|
||||
|
||||
[dev-dependencies]
|
||||
criterion = "0.5"
|
||||
@@ -66,6 +63,3 @@ unreachable_code = "allow"
|
||||
inherent_to_string = "allow"
|
||||
non_canonical_partial_ord_impl = "allow"
|
||||
|
||||
[[bench]]
|
||||
name = "parser_bench"
|
||||
harness = false
|
||||
|
||||
@@ -1,93 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
import os
|
||||
import random
|
||||
import string
|
||||
|
||||
OUTPUT_DIR = "/tmp"
|
||||
DECORATORS = [
|
||||
"# @pipeline",
|
||||
"# @fallible",
|
||||
"# @parallel",
|
||||
"# @export",
|
||||
"# @timeout(60)",
|
||||
"# @retry(3, 5)",
|
||||
"# @if(condition)",
|
||||
"# @after(dep)",
|
||||
"# @pipe(step1, step2)",
|
||||
"# @health(/health)",
|
||||
"# @loop(10)",
|
||||
]
|
||||
FUNCTION_BODIES = [
|
||||
"echo 'Processing...';",
|
||||
"local var=$(date +%s);",
|
||||
"if [ -f /tmp/test ]; then echo 'exists'; fi;",
|
||||
"for i in $(seq 1 10); do echo $i; done;",
|
||||
"case $1 in start) echo 'starting';; stop) echo 'stopping';; esac;",
|
||||
"read -r line < /dev/stdin;",
|
||||
"export PATH=$PATH:/usr/local/bin;",
|
||||
"set -e; set -u;",
|
||||
"trap 'echo error' ERR;",
|
||||
"cd /tmp || exit 1;",
|
||||
"local result=$(grep -r 'pattern' /var/log/ 2>/dev/null);",
|
||||
'for f in /tmp/*.tmp; do rm -f "$f"; done',
|
||||
"if [[ $var -gt 100 ]]; then echo 'large'; fi;",
|
||||
'while read -r line; do echo "$line"; done < /etc/passwd;',
|
||||
'select opt in a b c; do echo "Selected $opt"; break; done;',
|
||||
]
|
||||
|
||||
|
||||
def random_string(length=8):
|
||||
return "".join(random.choices(string.ascii_letters + string.digits, k=length))
|
||||
|
||||
|
||||
def generate_function(body_lines):
|
||||
lines = []
|
||||
func_name = f"func_{random_string(6)}"
|
||||
for _ in range(random.randint(0, 3)):
|
||||
lines.append(random.choice(DECORATORS))
|
||||
lines.append(f"{func_name}() {{")
|
||||
for _ in range(body_lines):
|
||||
lines.append(" " + random.choice(FUNCTION_BODIES))
|
||||
lines.append("}")
|
||||
lines.append("")
|
||||
return lines
|
||||
|
||||
|
||||
def generate_script(target_lines, num_functions, target_bytes):
|
||||
lines = []
|
||||
total_lines = 0
|
||||
body_per_func = max(1, (target_lines - num_functions * 3) // num_functions)
|
||||
|
||||
for _ in range(num_functions):
|
||||
func_lines = generate_function(body_per_func)
|
||||
lines.extend(func_lines)
|
||||
total_lines += len(func_lines)
|
||||
|
||||
while (
|
||||
total_lines < target_lines
|
||||
or len("\n".join(lines).encode("utf-8")) < target_bytes
|
||||
):
|
||||
lines.append(random.choice(FUNCTION_BODIES))
|
||||
total_lines += 1
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main():
|
||||
sizes = [
|
||||
("small", 1000, 10, 50_000),
|
||||
("medium", 5000, 50, 500_000),
|
||||
("large", 10000, 100, 950_000),
|
||||
]
|
||||
for name, lines, funcs, target_bytes in sizes:
|
||||
script = generate_script(lines, funcs, target_bytes)
|
||||
filepath = os.path.join(OUTPUT_DIR, f"bench_{name}.sh")
|
||||
with open(filepath, "w") as f:
|
||||
f.write(script)
|
||||
actual_bytes = len(script.encode("utf-8"))
|
||||
actual_lines = len(script.splitlines())
|
||||
print(f"Generated {filepath}: lines={actual_lines}, bytes={actual_bytes}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,61 +0,0 @@
|
||||
use criterion::{Criterion, black_box, criterion_group, criterion_main};
|
||||
use std::fs;
|
||||
use std::path::Path;
|
||||
|
||||
fn parse_large_script(c: &mut Criterion) {
|
||||
let script_path = Path::new("/tmp/bench_large.sh");
|
||||
|
||||
let script_content = fs::read_to_string(script_path)
|
||||
.expect("Failed to read benchmark script. Run `python3 benches/generate.py` first.");
|
||||
|
||||
let byte_size = script_content.len();
|
||||
let line_count = script_content.lines().count();
|
||||
|
||||
println!("Benchmark file: {} lines, {} bytes", line_count, byte_size);
|
||||
|
||||
c.bench_function("parse_script_large", |b| {
|
||||
b.iter(|| {
|
||||
let result =
|
||||
workshop_baker::bake::parser::parse_script(black_box(&script_content.clone()));
|
||||
black_box(result)
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
fn parse_medium_script(c: &mut Criterion) {
|
||||
let script_path = Path::new("/tmp/bench_medium.sh");
|
||||
|
||||
let script_content = fs::read_to_string(script_path)
|
||||
.expect("Failed to read benchmark script. Run `python3 benches/generate.py` first.");
|
||||
|
||||
c.bench_function("parse_script_medium", |b| {
|
||||
b.iter(|| {
|
||||
let result =
|
||||
workshop_baker::bake::parser::parse_script(black_box(&script_content.clone()));
|
||||
black_box(result)
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
fn parse_small_script(c: &mut Criterion) {
|
||||
let script_path = Path::new("/tmp/bench_small.sh");
|
||||
|
||||
let script_content = fs::read_to_string(script_path)
|
||||
.expect("Failed to read benchmark script. Run `python3 benches/generate.py` first.");
|
||||
|
||||
c.bench_function("parse_script_small", |b| {
|
||||
b.iter(|| {
|
||||
let result =
|
||||
workshop_baker::bake::parser::parse_script(black_box(&script_content.clone()));
|
||||
black_box(result)
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
criterion_group!(
|
||||
benches,
|
||||
parse_small_script,
|
||||
parse_medium_script,
|
||||
parse_large_script
|
||||
);
|
||||
criterion_main!(benches);
|
||||
@@ -1,62 +1,57 @@
|
||||
# workshop-baker/src
|
||||
|
||||
**Generated:** 2026-04-22
|
||||
**Commit:** 7b3b71a
|
||||
**Generated:** 2026-05-19
|
||||
**Commit:** 28abc5d
|
||||
|
||||
Baker CLI + execution engine for HoneyBiscuitWorkshop pipelines.
|
||||
|
||||
## OVERVIEW
|
||||
|
||||
CLI tool and runtime for executing CI/CD pipelines with resource isolation, privilege dropping, and package management integration.
|
||||
CLI tool and runtime for executing CI/CD pipelines. Orchestrates prebake → bake → finalize stages with notification dispatch.
|
||||
|
||||
## STRUCTURE
|
||||
|
||||
```
|
||||
src/
|
||||
├── bake.rs # Top-level bake orchestration
|
||||
├── bake/ # Script parsing and building
|
||||
│ ├── parser.rs # @decorator-aware shell parser
|
||||
│ ├── decorator.rs # @pipeline, @retry, @parallel, etc.
|
||||
│ ├── builder.rs # Script template builder
|
||||
│ └── schedule.rs # Function ordering via @after deps
|
||||
├── engine/ # Execution runtime
|
||||
│ ├── executor.rs # Script execution with cgroups
|
||||
│ ├── cgroups.rs # Linux cgroup resource limits
|
||||
│ ├── repology.rs # Package name resolution
|
||||
│ ├── pm.rs # Package manager detection
|
||||
│ └── upm.rs # User package manager (Nix/Guix)
|
||||
├── prebake/ # Build environment setup
|
||||
│ ├── config.rs # PrebakeConfig YAML schema
|
||||
│ ├── stage/ # Bootstrap, DepsSystem, DepsUser, Hooks
|
||||
│ └── security.rs # Privilege dropping logic
|
||||
├── types/ # Shared type definitions
|
||||
│ ├── repology.rs # RepologyEndpoint enum
|
||||
│ ├── builderconfig.rs
|
||||
│ ├── cache.rs
|
||||
│ └── memsize.rs
|
||||
├── monitor/ # Resource usage monitoring
|
||||
│ ├── cgroups.rs # cgroup-based usage collection
|
||||
│ └── jobobject.rs # Windows job objects (stub)
|
||||
├── daemon.rs # Unix socket daemon (TODO: refactor)
|
||||
└── lib.rs # CLI struct + module exports
|
||||
├── lib.rs # Module declarations + Cli struct + Commands enum
|
||||
├── main.rs # tokio::main entry point
|
||||
├── bare.rs # Bare mode (run individual stages)
|
||||
├── error.rs # CliError (wraps stage errors)
|
||||
├── bake.rs # Bake orchestration (script → execution)
|
||||
├── bake/ # Script parsing, decorator DSL, scheduling
|
||||
├── prebake.rs # Prebake orchestration (env setup)
|
||||
├── prebake/ # Config, stages, security, env providers
|
||||
├── finalize.rs # Finalize orchestration (post-build)
|
||||
├── finalize/ # Config, plugins, hooks, templates
|
||||
├── notify.rs # Notification system (batching, retry)
|
||||
├── notify/ # Types, templates, interactive, queue
|
||||
├── daemon/ # Empty (Daemon command = unimplemented!)
|
||||
├── monitor/ # Resource usage monitoring (cgroups)
|
||||
├── types.rs # Module re-exports
|
||||
├── types/ # buildstatus, resource types
|
||||
├── utils.rs # Module root (fetch submodule)
|
||||
└── utils/ # fetch.rs
|
||||
```
|
||||
|
||||
## WHERE TO LOOK
|
||||
|
||||
| Task | Location |
|
||||
|------|----------|
|
||||
| Script parsing with @decorators | `bake/parser.rs:29` - `parse_script()` |
|
||||
| Decorator types | `bake/decorator.rs` - `Decorator` enum |
|
||||
| Build execution | `engine/executor.rs` - `Executor::execute_script()` |
|
||||
| Resource limits | `engine/types.rs:56` - `ResourceLimits` struct |
|
||||
| Prebake stages | `prebake/stage/` subdirectories |
|
||||
| Package detection | `engine/pm.rs:detection()` |
|
||||
| CLI entry | `lib.rs:26` - `cli::Cli` struct |
|
||||
| CLI args | `lib.rs:21` - `Cli` struct |
|
||||
| Entry point | `main.rs:91` - `#[tokio::main]` |
|
||||
| Pipeline orchestration | `main.rs:68-74` - prebake→bake→finalize chain |
|
||||
| Bare mode (single stage) | `bare.rs:14` - `run_bare()` |
|
||||
| Script parsing | `bake/parser.rs:56` - `parse_script()` |
|
||||
| Decorator DSL | `bake/decorator.rs` - `Decorator` enum |
|
||||
| Prebake stages | `prebake/stage/` - Bootstrap→DepsSystem→DepsUser→Hooks→Ready |
|
||||
| Plugin system | `finalize/plugin.rs` + `finalize/plugin/` |
|
||||
| Notification dispatch | `notify.rs:66` - `notify()` async loop |
|
||||
| Notification types | `notify/types.rs` - `NotificationRule`, `ParsedTrigger` |
|
||||
|
||||
## CONVENTIONS
|
||||
|
||||
- **ExecutionContext**: Passed through all stages, carries task_id, username, env_vars, timeout
|
||||
- **PrebakeStage**: Ordered enum controlling stage execution sequence (Bootstrap -> Ready)
|
||||
- **Decorator**: Attached to shell functions via `# @name(args)` comments above function declarations
|
||||
- **Engine**: Lightweight wrapper around cgroup manager for resource isolation
|
||||
- **Dual-mode**: CLI commands (prebake, bake, finalize) vs daemon mode via Unix socket
|
||||
- **PrebakeStage**: Ordered enum Bootstrap→EarlyHook→DepsSystem→DepsUser→LateHook→Ready
|
||||
- **Decorator**: `# @name(args)` comments above shell function declarations
|
||||
- **Notification**: Plugin-based dispatch with batching, exponential backoff, priority groups
|
||||
- **Engine**: Delegated to `workshop-engine` crate (formerly in-tree)
|
||||
- **Dual-mode**: CLI commands (standalone) vs Daemon mode (unimplemented)
|
||||
|
||||
## ANTI-PATTERNS
|
||||
1. **Empty daemon/** directory — Daemon command is `unimplemented!()`
|
||||
2. **Dead code allowed** — `lints.rust.dead_code = "allow"` in Cargo.toml
|
||||
3. **Engine was extracted** — workshop-engine is its own crate now, src/engine/ removed
|
||||
4. **Python test infra exists but files removed** — pytest.ini remains, test dir is empty
|
||||
|
||||
@@ -12,14 +12,16 @@ bake/
|
||||
├── parser.rs # @decorator-aware shell script parser
|
||||
├── decorator.rs # Decorator enum + 100+ unit tests
|
||||
├── builder.rs # Script template assembly
|
||||
└── schedule.rs # Function ordering via @after dependencies
|
||||
├── schedule.rs # Function ordering via @after dependencies
|
||||
├── error.rs # BakeError enum
|
||||
└── constant.rs # DEFAULT_WORKSPACE, TRIVIAL_SCRIPT_NAME
|
||||
```
|
||||
|
||||
## WHERE TO LOOK
|
||||
|
||||
| Task | Location |
|
||||
|------|----------|
|
||||
| Parse script | `parser.rs:29` - `parse_script()` |
|
||||
Parse script | `parser.rs:56` - `parse_script()` |
|
||||
| Decorator types | `decorator.rs` - `Decorator` enum variants |
|
||||
| Build script | `builder.rs` - template builder |
|
||||
| Schedule deps | `schedule.rs` - @after resolution |
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
use std::path::PathBuf;
|
||||
|
||||
use crate::cli::{BareCommands, Cli};
|
||||
use crate::error::CliError;
|
||||
use crate::types::resource::ResourceRegistry;
|
||||
use crate::{bake::bake, finalize::finalize, prebake::prebake};
|
||||
use workshop_engine::{EventSender, ExecutionContext};
|
||||
fn or_workshop(path: &Option<PathBuf>, name: &str) -> PathBuf {
|
||||
path
|
||||
.clone()
|
||||
.unwrap_or_else(|| PathBuf::from(format!(".workshop/{}", name)))
|
||||
}
|
||||
|
||||
pub async fn run_bare(
|
||||
cli: &Cli,
|
||||
command: &BareCommands,
|
||||
ctx: &mut ExecutionContext,
|
||||
event_tx: EventSender,
|
||||
pipeline: &str,
|
||||
build_id: &str,
|
||||
registry: &ResourceRegistry,
|
||||
) -> Result<(), CliError> {
|
||||
let (stage, config) = match command {
|
||||
BareCommands::Prebake { .. } => ("prebake", or_workshop(&cli.prebake, "prebake.yml")),
|
||||
BareCommands::Bake { .. } => ("bake", or_workshop(&cli.bake, "bake.sh")),
|
||||
BareCommands::Finalize { .. } => {
|
||||
("finalize", or_workshop(&cli.finalize, "finalize.yml"))
|
||||
}
|
||||
};
|
||||
|
||||
let prebake_path = or_workshop(&cli.prebake, "prebake.yml");
|
||||
let bake_base = or_workshop(&cli.bake_base, "bake_base.sh");
|
||||
|
||||
ctx.task_id = format!("{}-{}-{}", pipeline, build_id, stage);
|
||||
ctx.pipeline_name = pipeline.to_string();
|
||||
ctx.standalone = true;
|
||||
|
||||
match command {
|
||||
BareCommands::Prebake { .. } => {
|
||||
prebake(&config, cli, None, ctx, event_tx, None).await?;
|
||||
}
|
||||
BareCommands::Bake { .. } => {
|
||||
bake(&config, &bake_base, &prebake_path, cli, ctx, event_tx).await?;
|
||||
}
|
||||
BareCommands::Finalize { .. } => {
|
||||
finalize(&config, cli, ctx, event_tx, registry).await?;
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
@@ -1,193 +0,0 @@
|
||||
use crate::cli::Cli;
|
||||
// use serde::Serialize;
|
||||
use serde_json::Value;
|
||||
// use std::path::PathBuf;
|
||||
// use std::sync::Arc;
|
||||
use tokio::io::{AsyncBufReadExt, BufReader};
|
||||
use tokio::net::UnixStream;
|
||||
// use tokio::process::{Child, Command};
|
||||
// use tokio::signal;
|
||||
// use tokio::sync::RwLock;
|
||||
// use tokio::time::{Duration, interval};
|
||||
|
||||
// #[derive(Debug, Clone, Serialize)]
|
||||
// struct Heartbeat {
|
||||
// pipeline_id: String,
|
||||
// build_id: String,
|
||||
// stage: String,
|
||||
// timestamp: u64,
|
||||
// status: String,
|
||||
// subtasks: Vec<String>,
|
||||
// }
|
||||
|
||||
pub async fn daemon(_cli: &Cli) {
|
||||
todo!("Daemon should be refactored");
|
||||
// let socket = PathBuf::from(settings.get_string("socket").unwrap());
|
||||
// let agentsocket = settings.get_string("agentsocket").unwrap();
|
||||
// let _ = std::fs::remove_file(&socket);
|
||||
|
||||
// let heartbeat = Arc::new(RwLock::new(Heartbeat {
|
||||
// pipeline_id: cli.pipeline.clone(),
|
||||
// build_id: cli.build_id.clone(),
|
||||
// stage: String::new(),
|
||||
// timestamp: 0,
|
||||
// status: String::new(),
|
||||
// subtasks: Vec::new(),
|
||||
// }));
|
||||
|
||||
// // let heartbeat_clone = heartbeat.clone();
|
||||
// let heartbeat_task = {
|
||||
// let hb = heartbeat.clone();
|
||||
// tokio::spawn(async move {
|
||||
// let mut tick = interval(Duration::from_secs(1));
|
||||
// loop {
|
||||
// tick.tick().await;
|
||||
// let mut hb = hb.write().await;
|
||||
// hb.timestamp = std::time::SystemTime::now()
|
||||
// .duration_since(std::time::UNIX_EPOCH)
|
||||
// .unwrap()
|
||||
// .as_millis() as u64;
|
||||
|
||||
// // 只输出到stdout(模拟未来发送给Agent)
|
||||
// println!("[HEARTBEAT] {}", serde_json::to_string(&*hb).unwrap());
|
||||
// }
|
||||
// })
|
||||
// };
|
||||
|
||||
// let socket_task = {
|
||||
// let socket = socket.clone();
|
||||
// tokio::spawn(async move {
|
||||
// log::info!("Creating Unix socket at {}", socket.display());
|
||||
// let listener =
|
||||
// UnixListener::bind(&socket).expect("Failed to bind Unix socket");
|
||||
|
||||
// loop {
|
||||
// match listener.accept().await {
|
||||
// Ok((stream, _)) => {
|
||||
// tokio::spawn(handle_connection(stream));
|
||||
// }
|
||||
// Err(e) => {
|
||||
// log::error!("Accept error: {}", e);
|
||||
// break;
|
||||
// }
|
||||
// }
|
||||
// }
|
||||
// })
|
||||
// };
|
||||
|
||||
// let mut tasks: Vec<TaskSpec> = Vec::new();
|
||||
|
||||
// let execution_task = {
|
||||
// let hb = heartbeat.clone();
|
||||
// tokio::spawn(async move {
|
||||
// let mut hb = hb.write().await;
|
||||
// let mut children: Vec<(String, Child)> = vec![];
|
||||
|
||||
// let executable = std::env::current_exe().unwrap();
|
||||
|
||||
// for (index, task) in tasks.iter().enumerate() {
|
||||
// let mut command = task.command(&executable);
|
||||
// command.stdout(std::process::Stdio::piped());
|
||||
// command.stderr(std::process::Stdio::piped());
|
||||
// let child = command
|
||||
// .spawn()
|
||||
// .expect(format!("Failed to spawn child {}", index).as_str());
|
||||
|
||||
// let pid = child.id().unwrap_or(0).to_string();
|
||||
// children.push((pid.clone(), child));
|
||||
|
||||
// hb.subtasks.push(pid.clone());
|
||||
// }
|
||||
|
||||
// for (pid, mut child) in children {
|
||||
// let status = child.wait().await.expect("Failed to wait child");
|
||||
// log::info!("Subtask {} exited with: {}", pid, status);
|
||||
|
||||
// hb.subtasks.retain(|p| p != &pid);
|
||||
// }
|
||||
|
||||
// hb.stage = "Success".to_string();
|
||||
// })
|
||||
// };
|
||||
|
||||
// tokio::select! {
|
||||
// _ = heartbeat_task => {
|
||||
// log::error!("Heartbeat task died");
|
||||
// }
|
||||
// _ = execution_task => {
|
||||
// log::info!("All subtasks completed");
|
||||
// }
|
||||
// _ = socket_task => {
|
||||
// log::error!("Socket task died");
|
||||
// }
|
||||
// _ = signal::ctrl_c() => {
|
||||
// log::info!("Shutting down...");
|
||||
// }
|
||||
// }
|
||||
// for pid in heartbeat.read().await.subtasks.clone() {
|
||||
// let _ = Command::new("kill").arg("-TERM").arg(&pid).status().await;
|
||||
// }
|
||||
|
||||
// let _ = std::fs::remove_file(&socket);
|
||||
}
|
||||
|
||||
async fn _handle_connection(stream: UnixStream) {
|
||||
let peer_addr = match stream.peer_addr() {
|
||||
Ok(addr) => format!("{:?}", addr),
|
||||
Err(_) => "unknown".to_string(),
|
||||
};
|
||||
|
||||
log::debug!("New connection from: {}", peer_addr);
|
||||
|
||||
let mut reader = BufReader::new(stream);
|
||||
let mut line = String::new();
|
||||
|
||||
loop {
|
||||
line.clear();
|
||||
match reader.read_line(&mut line).await {
|
||||
Ok(0) => {
|
||||
log::debug!("Connection closed by peer: {}", peer_addr);
|
||||
break;
|
||||
}
|
||||
Ok(_) => {
|
||||
// 简单打印原始JSON(预留字段解析空间)
|
||||
match serde_json::from_str::<Value>(&line) {
|
||||
Ok(json) => {
|
||||
// 只提取关键字段打印(字段可能变化)
|
||||
let msgtype = json
|
||||
.get("msgtype")
|
||||
.and_then(|v| v.as_str())
|
||||
.unwrap_or("unknown");
|
||||
let name = json
|
||||
.get("name")
|
||||
.and_then(|v| v.as_str())
|
||||
.unwrap_or("unnamed");
|
||||
|
||||
log::info!("[{}] Message from '{}': {}", msgtype, name, line.trim());
|
||||
|
||||
// TODO: 字段稳定后,可扩展为结构化打印
|
||||
// match msgtype {
|
||||
// "ResourceUsage" => print_resource(&json),
|
||||
// "Log" => print_log(&json),
|
||||
// _ => log::debug!("Unknown type: {}", msgtype),
|
||||
// }
|
||||
}
|
||||
Err(e) => {
|
||||
log::warn!(
|
||||
"Invalid JSON from {}: {} - Error: {}",
|
||||
peer_addr,
|
||||
line.trim(),
|
||||
e
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(e) => {
|
||||
log::error!("Failed to read from {}: {}", peer_addr, e);
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
log::debug!("Connection handler for {} exiting", peer_addr);
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
use thiserror::Error;
|
||||
use workshop_engine::HasExitCode;
|
||||
|
||||
#[derive(Error, Debug)]
|
||||
pub enum CliError {
|
||||
#[error("{0}")]
|
||||
Prebake(#[from] crate::prebake::error::PrebakeError),
|
||||
#[error("{0}")]
|
||||
Bake(#[from] crate::bake::error::BakeError),
|
||||
#[error("{0}")]
|
||||
Finalize(#[from] crate::finalize::FinalizeError),
|
||||
#[error("{0}")]
|
||||
General(anyhow::Error),
|
||||
}
|
||||
|
||||
impl From<anyhow::Error> for CliError {
|
||||
fn from(e: anyhow::Error) -> Self {
|
||||
CliError::General(e)
|
||||
}
|
||||
}
|
||||
impl CliError {
|
||||
pub fn exit_code(&self) -> i32 {
|
||||
match self {
|
||||
CliError::Prebake(e) => e.exit_code(),
|
||||
CliError::Bake(e) => e.exit_code(),
|
||||
CliError::Finalize(e) => e.exit_code(),
|
||||
CliError::General(_) => 1,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -8,12 +8,10 @@ pub mod template;
|
||||
pub mod types;
|
||||
|
||||
use crate::cli::Cli;
|
||||
use crate::finalize::constant::FINALIZE_STANDALONE_PLUGIN_PATH;
|
||||
use crate::finalize::plugin::fetch::{self, FetchArgument};
|
||||
use crate::types::resource::ResourceRegistry;
|
||||
pub use config::*;
|
||||
pub use error::FinalizeError;
|
||||
use std::path::Path;
|
||||
use std::path::PathBuf;
|
||||
use workshop_engine::EventSender;
|
||||
|
||||
pub fn parse(path: &Path) -> Result<FinalizeConfig, FinalizeError> {
|
||||
@@ -27,6 +25,7 @@ pub async fn finalize(
|
||||
_cli: &Cli,
|
||||
ctx: &mut workshop_engine::ExecutionContext,
|
||||
event_tx: EventSender,
|
||||
registry: &ResourceRegistry,
|
||||
) -> Result<(), FinalizeError> {
|
||||
let mut finalize = parse(finalize_path)?;
|
||||
let validate_result = finalize.validate();
|
||||
@@ -36,33 +35,10 @@ pub async fn finalize(
|
||||
}
|
||||
return Err(FinalizeError::ValidateError(errors));
|
||||
};
|
||||
if ctx.standalone {
|
||||
// In standalone mode, finalize should download plugin by itself
|
||||
log::debug!("Downloading {} plugins", finalize.plugin.len());
|
||||
let plugin_path = PathBuf::from(FINALIZE_STANDALONE_PLUGIN_PATH);
|
||||
// if plugin_path.exists() {
|
||||
// std::fs::remove_dir_all(&plugin_path).map_err(FinalizeError::IoError)?;
|
||||
// }
|
||||
for (name, data) in finalize.plugin.iter_mut() {
|
||||
log::info!("Downloading plugin {}", name);
|
||||
let (source, config) = data.iter_mut().next().unwrap_or_else(|| {
|
||||
panic!(
|
||||
"Internal Error: Plugin {} has no source configured, this should have been caught by validation!",
|
||||
name
|
||||
)
|
||||
});
|
||||
let fetch_argument = FetchArgument {
|
||||
checksum: config.checksum.clone(),
|
||||
shallow: config.shallow,
|
||||
};
|
||||
fetch::fetch_plugin(source, name, &plugin_path, fetch_argument).await?;
|
||||
config.runtime.fspath = Some(plugin_path.join(name));
|
||||
}
|
||||
}
|
||||
|
||||
// Register Plugins
|
||||
// Register Plugins (paths resolved from registry)
|
||||
log::info!("Registering {} plugins", finalize.plugin.len());
|
||||
let registered_plugins = plugin::register(finalize.plugin, None)?;
|
||||
let registered_plugins = plugin::register(finalize.plugin, None, &mut finalize.notification, registry)?;
|
||||
|
||||
// Early hook
|
||||
let earlyhook_result = stage::hook::hook(
|
||||
|
||||
@@ -1,56 +1,52 @@
|
||||
# workshop-baker/src/finalize
|
||||
|
||||
**Generated:** 2026-04-22
|
||||
**Commit:** 7b3b71a
|
||||
**Generated:** 2026-05-19
|
||||
**Commit:** 28abc5d
|
||||
|
||||
Artifact packaging, distribution, deployment, and notifications.
|
||||
Artifact packaging, plugin-based notifications, and post-build hooks.
|
||||
|
||||
## STRUCTURE
|
||||
|
||||
```
|
||||
finalize/
|
||||
├── config.rs # FinalizeConfig YAML (artifacts, deploy, notify)
|
||||
├── config.rs # FinalizeConfig YAML (379 lines, plugins, notifications)
|
||||
├── stage.rs # FinalizeStage enum
|
||||
├── stage/hook.rs # Post-build hooks
|
||||
├── stage/hook.rs # Post-build hook execution
|
||||
├── event.rs # Event propagation
|
||||
├── template.rs # Variable substitution {{VAR}}
|
||||
├── plugin.rs # Plugin trait + registry
|
||||
├── template.rs # Variable substitution {{VARIABLE}}
|
||||
├── plugin.rs # Plugin trait + registration
|
||||
├── plugin/
|
||||
│ ├── metadata.rs # Plugin manifest parsing
|
||||
│ ├── dylib.rs # Dynamic library plugins
|
||||
│ ├── rhai.rs # Rhai scripting (todo!())
|
||||
│ ├── shell.rs # Shell command plugins
|
||||
│ └── internal/ # Built-in plugins
|
||||
│ ├── mail/ # Email notifications (SMTP)
|
||||
│ ├── webhook.rs # HTTP webhooks
|
||||
│ ├── satori.rs # Satori integration
|
||||
│ └── insitenotify.rs # In-site notifications
|
||||
│ └── fetch/ # Artifact fetching
|
||||
│ ├── git.rs # Git repository fetch
|
||||
│ ├── http.rs # HTTP download
|
||||
│ ├── extract.rs # Archive extraction
|
||||
│ └── checksum.rs# SHA256 verification (SHA1 deprecated)
|
||||
│ └── internal.rs + internal/ # Built-in plugins (mail, webhook, satori, insitenotify)
|
||||
├── types.rs
|
||||
├── types/compression.rs
|
||||
├── error.rs # FinalizeError
|
||||
└── constant.rs
|
||||
```
|
||||
|
||||
## WHERE TO LOOK
|
||||
|
||||
| Task | Location |
|
||||
|------|----------|
|
||||
| Config loading | `config.rs` - FinalizeConfig |
|
||||
| Email notify | `plugin/internal/mail/` - SMTP templates |
|
||||
| Config loading | `config.rs:17` - `FinalizeConfig` + `parse()` |
|
||||
| Plugin registration | `plugin.rs` - `register()` |
|
||||
| Email notifications | `plugin/internal/mail.rs` - SMTP |
|
||||
| Webhooks | `plugin/internal/webhook.rs` |
|
||||
| Artifact fetch | `plugin/fetch/` - git/http/extract |
|
||||
| Checksums | `plugin/fetch/checksum.rs:13` - SHA1 deprecated |
|
||||
| Satori integration | `plugin/internal/satori.rs` |
|
||||
| Shell plugins | `plugin/shell.rs` |
|
||||
| Template rendering | `template.rs` - `{{VARIABLE}}` handlebars-style |
|
||||
|
||||
## CONVENTIONS
|
||||
|
||||
- **Plugin System**: Dynamic + internal plugins via Plugin trait
|
||||
- **Template Syntax**: `{{VARIABLE}}` handlebars-style
|
||||
- **Stages**: Hook → Fetch → Build → Publish → Notify → Cleanup
|
||||
- **Fetch**: Supports git, http, with extraction and checksum verify
|
||||
- **Plugin System**: `FinalizePlugin` trait + registry with both internal and dynamic plugins
|
||||
- **Template Syntax**: `{{VARIABLE}}` handlebars-style substitution via `template.rs`
|
||||
- **Stages**: EarlyHook → (Artifacts) → LateHook → Notifications
|
||||
- **Notification**: `notify.rs` in parent module drives plugin-based notification dispatch
|
||||
- **ResourceRegistry**: Shared resource cache passed through to plugins
|
||||
|
||||
## ANTI-PATTERNS
|
||||
|
||||
1. **SHA1 deprecated** — `checksum.rs:13` warns SHA1 is deprecated
|
||||
2. **Unimplemented** — `rhai.rs:17` - "todo!()" for Rhai scripting
|
||||
3. **TODO markers** — Multiple TODOs in config.rs (lines 305-306)
|
||||
1. **fetch/ subdirectory removed** — was `plugin/fetch/{git,http,extract,checksum}`, no longer exists
|
||||
2. **SHA1 deprecated** — workshop-getterurl handles checksums now
|
||||
3. **Unimplemented** — `rhai.rs:17` - "todo!()" for Rhai scripting
|
||||
4. **TODO markers** — Multiple TODOs in config.rs (lines 305-306)
|
||||
5. **Artifact stage not implemented** — `finalize.rs:57` — "TODO: artifact stage implementation"
|
||||
|
||||
@@ -115,6 +115,17 @@ pub struct NotificationMethod {
|
||||
pub policy: Vec<NotifyPolicy>,
|
||||
}
|
||||
|
||||
impl Default for NotificationMethod {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
template: NotificationTemplate::default(),
|
||||
trigger: Vec::new(),
|
||||
config: Value::default(),
|
||||
policy: Vec::new(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize, Serialize, Clone, Default)]
|
||||
pub struct NotificationTemplate {
|
||||
/// Payload schema: "default" (plugin-defined) or "custom" (requires on_* templates)
|
||||
@@ -134,17 +145,59 @@ pub struct NotificationTemplate {
|
||||
pub on_finish_of: Option<OnFinishOf>,
|
||||
}
|
||||
|
||||
impl std::ops::AddAssign for NotificationTemplate {
|
||||
fn add_assign(&mut self, other: Self) {
|
||||
if self.schema.is_none() {
|
||||
self.schema = other.schema;
|
||||
}
|
||||
if self.on_success.is_none() {
|
||||
self.on_success = other.on_success;
|
||||
}
|
||||
if self.on_failure.is_none() {
|
||||
self.on_failure = other.on_failure;
|
||||
}
|
||||
if self.on_finish_of.is_none() {
|
||||
self.on_finish_of = other.on_finish_of;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Notification configuration container
|
||||
#[derive(Debug, Deserialize, Serialize, Clone, Default)]
|
||||
#[derive(Debug, Serialize, Clone, Default)]
|
||||
pub struct NotificationConfig {
|
||||
/// Template definitions for notification content
|
||||
#[serde(default)]
|
||||
pub templates: HashMap<String, NotificationTemplateDef>,
|
||||
|
||||
#[serde(flatten)]
|
||||
/// Priority groups (YAML keys like `0`, `1` are parsed as u32).
|
||||
/// Custom Deserialize handles serde_yaml string→u32 conversion.
|
||||
#[serde(default)]
|
||||
pub groups: HashMap<u32, HashMap<String, NotificationMethod>>,
|
||||
}
|
||||
|
||||
/// Serde helper: YAML map keys are always strings, convert to u32.
|
||||
impl<'de> serde::Deserialize<'de> for NotificationConfig {
|
||||
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
|
||||
where D: serde::Deserializer<'de>
|
||||
{
|
||||
#[derive(serde::Deserialize)]
|
||||
struct Helper {
|
||||
#[serde(default)]
|
||||
templates: HashMap<String, NotificationTemplateDef>,
|
||||
#[serde(flatten)]
|
||||
groups_raw: HashMap<String, HashMap<String, NotificationMethod>>,
|
||||
}
|
||||
let h = Helper::deserialize(deserializer)?;
|
||||
let groups: HashMap<u32, _> = h.groups_raw.into_iter()
|
||||
.map(|(k, v)| {
|
||||
k.parse::<u32>().map(|n| (n, v))
|
||||
.map_err(|_| serde::de::Error::custom(format!("priority key must be integer, got {:?}", k)))
|
||||
})
|
||||
.collect::<Result<_, _>>()?;
|
||||
Ok(NotificationConfig { templates: h.templates, groups })
|
||||
}
|
||||
}
|
||||
|
||||
/// Stage-triggered notification configuration
|
||||
#[derive(Debug, Deserialize, Serialize, Clone)]
|
||||
pub struct OnFinishOf {
|
||||
@@ -158,7 +211,7 @@ pub struct OnFinishOf {
|
||||
|
||||
/// Notification template definition.
|
||||
/// Supports inline content (on_success/on_failure/on_finish_of) or external reference (use).
|
||||
#[derive(Debug, Deserialize, Serialize, Clone)]
|
||||
#[derive(Debug, Deserialize, Serialize, Clone, Default)]
|
||||
pub struct NotificationTemplateDef {
|
||||
/// External template reference (git URL).
|
||||
/// Cannot be combined with on_* fields in MVP (validation will reject).
|
||||
@@ -178,6 +231,53 @@ pub struct NotificationTemplateDef {
|
||||
pub on_finish_of: Option<String>,
|
||||
}
|
||||
|
||||
impl NotificationTemplateDef {
|
||||
/// Merge `other` into `self`, keeping self's values when present.
|
||||
///
|
||||
/// `other` fills in gaps where `self` is `None`.
|
||||
/// Chaining: `a.apply(b).apply(c)` = a wins, then b fills gaps, then c fills gaps.
|
||||
pub fn apply(mut self, other: Self) -> Self {
|
||||
if self.on_success.is_none() {
|
||||
self.on_success = other.on_success;
|
||||
}
|
||||
if self.on_failure.is_none() {
|
||||
self.on_failure = other.on_failure;
|
||||
}
|
||||
if self.on_finish_of.is_none() {
|
||||
self.on_finish_of = other.on_finish_of;
|
||||
}
|
||||
self
|
||||
}
|
||||
}
|
||||
|
||||
/// Builder for [`NotificationTemplateDef`].
|
||||
pub struct NotificationTemplateDefBuilder(NotificationTemplateDef);
|
||||
|
||||
impl NotificationTemplateDefBuilder {
|
||||
pub fn new() -> Self {
|
||||
Self(NotificationTemplateDef::default())
|
||||
}
|
||||
pub fn with_use(mut self, val: impl Into<String>) -> Self {
|
||||
self.0.use_ = Some(val.into());
|
||||
self
|
||||
}
|
||||
pub fn with_on_success(mut self, val: impl Into<String>) -> Self {
|
||||
self.0.on_success = Some(val.into());
|
||||
self
|
||||
}
|
||||
pub fn with_on_failure(mut self, val: impl Into<String>) -> Self {
|
||||
self.0.on_failure = Some(val.into());
|
||||
self
|
||||
}
|
||||
pub fn with_on_finish_of(mut self, val: impl Into<String>) -> Self {
|
||||
self.0.on_finish_of = Some(val.into());
|
||||
self
|
||||
}
|
||||
pub fn build(self) -> NotificationTemplateDef {
|
||||
self.0
|
||||
}
|
||||
}
|
||||
|
||||
/// Notification trigger type.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum Trigger {
|
||||
@@ -336,15 +436,18 @@ impl FinalizeConfig {
|
||||
fn validate_notification(&self, errors: &mut Vec<String>) {
|
||||
// Validate templates
|
||||
for (name, tmpl) in &self.notification.templates {
|
||||
if tmpl.use_.is_some()
|
||||
&& (tmpl.on_success.is_some()
|
||||
|| tmpl.on_failure.is_some()
|
||||
|| tmpl.on_finish_of.is_some())
|
||||
{
|
||||
errors.push(format!(
|
||||
"Template '{}' cannot combine 'use' with 'on_*' fields (MVP restriction)",
|
||||
name
|
||||
));
|
||||
if let Some(ref u) = tmpl.use_ {
|
||||
if u.contains("://") && (
|
||||
tmpl.on_success.is_some() ||
|
||||
tmpl.on_failure.is_some() ||
|
||||
tmpl.on_finish_of.is_some()
|
||||
) {
|
||||
errors.push(format!(
|
||||
"Template '{}' cannot combine external 'use' with 'on_*' fields",
|
||||
name
|
||||
));
|
||||
}
|
||||
// Local references (no ://) are allowed to have on_* overrides
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,14 +1,15 @@
|
||||
use crate::finalize::RawPluginMap;
|
||||
use crate::finalize::constant::{FINALIZE_PLUGIN_MANIFEST, FINALIZE_PLUGIN_MANIFEST_EXTENSION};
|
||||
use crate::finalize::plugin::metadata::{PluginMetadata, PluginType};
|
||||
use crate::{EventSender, ExecutionContext, finalize::error::PluginError};
|
||||
use workshop_engine::{EventSender, ExecutionContext};
|
||||
use crate::{ finalize::error::PluginError};
|
||||
use crate::types::resource::ResourceRegistry;
|
||||
use async_trait::async_trait;
|
||||
use std::collections::{HashMap, HashSet};
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::Arc;
|
||||
|
||||
mod dylib;
|
||||
pub mod fetch;
|
||||
mod internal;
|
||||
mod metadata;
|
||||
mod rhai;
|
||||
@@ -214,6 +215,8 @@ fn get_entry(plugin_type: PluginType, name: &str) -> Result<Box<dyn AsyncPluginF
|
||||
pub fn register(
|
||||
external_plugins: RawPluginMap,
|
||||
list: Option<&HashSet<String>>,
|
||||
notification_config: &mut crate::finalize::NotificationConfig,
|
||||
registry: &ResourceRegistry,
|
||||
) -> Result<PluginMap, PluginError> {
|
||||
let mut plugins: PluginMap = HashMap::new();
|
||||
let mut count = 0;
|
||||
@@ -255,7 +258,14 @@ pub fn register(
|
||||
log::warn!("Plugin {} is not available: {}", name, reason);
|
||||
continue;
|
||||
}
|
||||
let metadata = read_manifest(config.runtime.fspath, &name)?;
|
||||
// Resolve plugin path from registry instead of config.runtime.fspath
|
||||
let plugin_path = registry.resolve(&name).ok_or_else(|| {
|
||||
PluginError::PluginUnavailable {
|
||||
name: name.clone(),
|
||||
reason: "plugin not found in resource registry".to_string(),
|
||||
}
|
||||
})?;
|
||||
let metadata = read_manifest(Some(plugin_path), &name)?;
|
||||
let plugin_type = get_plugin_type(metadata.plugin_type.clone(), &metadata.entrypoint);
|
||||
let entry = get_entry(plugin_type, &name)?;
|
||||
plugins.insert(
|
||||
@@ -267,8 +277,18 @@ pub fn register(
|
||||
count += 1;
|
||||
}
|
||||
|
||||
log::info!("Registered {} plugins", count);
|
||||
// Inject plugin-declared templates as .fallback
|
||||
for (name, plugin) in &plugins {
|
||||
if let Some(ref templates) = plugin.metadata.templates {
|
||||
let key = format!("{}.fallback", name);
|
||||
for tmpl in templates {
|
||||
notification_config.templates.insert(key.clone(), tmpl.clone());
|
||||
}
|
||||
log::debug!("Registered templates from plugin '{}' as '{}'", name, key);
|
||||
}
|
||||
}
|
||||
|
||||
log::info!("Registered {} plugins", count);
|
||||
Ok(plugins)
|
||||
}
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
use crate::finalize::plugin::PluginError;
|
||||
use crate::finalize::plugin::PluginMetadata;
|
||||
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn};
|
||||
use workshop_engine::{EventSender, ExecutionContext};
|
||||
use crate::{ finalize::plugin::AsyncPluginFn};
|
||||
use async_trait::async_trait;
|
||||
|
||||
pub struct Dylib;
|
||||
|
||||
@@ -1,109 +0,0 @@
|
||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
||||
use crate::finalize::fetch;
|
||||
use crate::finalize::plugin::{PluginError, RawPluginMap};
|
||||
use std::collections::HashSet;
|
||||
use std::path::Path;
|
||||
|
||||
mod checksum;
|
||||
mod extract;
|
||||
mod git;
|
||||
mod http;
|
||||
mod types;
|
||||
|
||||
pub use types::FetchArgument;
|
||||
|
||||
enum URLScheme {
|
||||
Unknown,
|
||||
Git,
|
||||
Http,
|
||||
Https,
|
||||
File,
|
||||
}
|
||||
|
||||
impl URLScheme {
|
||||
fn parse(url: &str) -> (Self, String) {
|
||||
match url.split_once("://") {
|
||||
Some((prefix, rest)) => {
|
||||
let scheme = match prefix {
|
||||
"git" => URLScheme::Git,
|
||||
"https" => URLScheme::Https,
|
||||
"http" => URLScheme::Http,
|
||||
"file" => URLScheme::File,
|
||||
_ => URLScheme::Unknown,
|
||||
};
|
||||
(scheme, rest.to_string())
|
||||
}
|
||||
None => (URLScheme::Unknown, url.to_string()),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn fetch_plugin(
|
||||
url: &str,
|
||||
name: &str,
|
||||
basedir: &Path,
|
||||
argument: FetchArgument,
|
||||
) -> Result<(), PluginError> {
|
||||
let (scheme, path) = URLScheme::parse(url);
|
||||
|
||||
match scheme {
|
||||
URLScheme::Git => {
|
||||
// NOTE: as a practical compromise, we use git command for git clone
|
||||
// gix will be supported later, with ONLY https scheme
|
||||
// TODO: Implement gix
|
||||
log::info!("Fetching git plugin: {}", url);
|
||||
git::fetch_git_plugin(&path, name, basedir, argument.shallow).await
|
||||
}
|
||||
URLScheme::Http | URLScheme::Https => {
|
||||
log::info!("Fetching http plugin: {}", url);
|
||||
let checksum = argument.get_checksum();
|
||||
http::fetch_http_plugin(url, name, basedir, checksum).await
|
||||
}
|
||||
URLScheme::File => {
|
||||
// TODO: Support client mode
|
||||
// NOTE: In standalone mode, we just assume path is well-defined and utilize
|
||||
log::info!("Fetching file plugin: {}", url);
|
||||
Ok(())
|
||||
}
|
||||
URLScheme::Unknown => {
|
||||
log::error!("Unknown URL scheme in plugin URL: {}", url);
|
||||
Err(PluginError::GeneralError(format!(
|
||||
"Unknown URL scheme in plugin URL: {}",
|
||||
url
|
||||
)))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn fetch_plugins(
|
||||
plugin_path: &Path,
|
||||
plugin_config: &mut RawPluginMap,
|
||||
list: Option<&HashSet<String>>,
|
||||
) -> Result<(), PluginError> {
|
||||
log::info!("Preparing to download {} plugins", plugin_config.len());
|
||||
for (name, data) in plugin_config.iter_mut() {
|
||||
match list {
|
||||
Some(list) if !list.contains(name) => {
|
||||
log::debug!("Plugin {} not in list, skipping", name);
|
||||
continue;
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
log::debug!("Downloading plugin {}", name);
|
||||
let (source, config) =
|
||||
data.iter_mut()
|
||||
.next()
|
||||
.ok_or_else(|| PluginError::PluginUnavailable {
|
||||
name: name.clone(),
|
||||
reason: "no source configured".to_string(),
|
||||
})?;
|
||||
let fetch_argument = FetchArgument {
|
||||
checksum: config.checksum.clone(),
|
||||
shallow: config.shallow,
|
||||
};
|
||||
fetch::fetch_plugin(source, name, plugin_path, fetch_argument).await?;
|
||||
log::debug!("Storing plugin {} to {}", name, plugin_path.display());
|
||||
config.runtime.fspath = Some(plugin_path.join(name));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
@@ -1,137 +0,0 @@
|
||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
||||
use super::types::ChecksumType;
|
||||
use sha2::{Digest, Sha256, Sha512};
|
||||
|
||||
/// Detects the checksum type based on the length of the checksum string.
|
||||
///
|
||||
/// Returns `Ok(None)` for empty string (checksum disabled).
|
||||
/// Returns `Ok(Some(ChecksumType))` for valid SHA256 (64 chars) or SHA512 (128 chars).
|
||||
/// Returns `Err` for invalid lengths (32=MD5, 40=SHA1) or unsupported lengths.
|
||||
pub fn detect_checksum_type(checksum: &str) -> Result<Option<ChecksumType>, String> {
|
||||
match checksum.len() {
|
||||
0 => Ok(None),
|
||||
32 => Err(format!(
|
||||
"MD5 (first 8 chars: {}) is not supported for security reasons. Use SHA256 (64 chars) or SHA512 (128 chars)",
|
||||
&checksum[..8.min(checksum.len())]
|
||||
)),
|
||||
40 => Err(format!(
|
||||
"SHA1 (first 8 chars: {}) is deprecated and not supported. Use SHA256 (64 chars) or SHA512 (128 chars)",
|
||||
&checksum[..8.min(checksum.len())]
|
||||
)),
|
||||
64 => Ok(Some(ChecksumType::Sha256)),
|
||||
128 => Ok(Some(ChecksumType::Sha512)),
|
||||
_ => Err(format!(
|
||||
"Invalid checksum length: {}. Expected 0 (disabled), 64 (SHA256), or 128 (SHA512). First 8 chars: {}",
|
||||
checksum.len(),
|
||||
&checksum[..8.min(checksum.len())]
|
||||
)),
|
||||
}
|
||||
}
|
||||
|
||||
/// Verifies that the data matches the expected checksum.
|
||||
///
|
||||
/// Compares the computed hash of `data` against `expected` using the specified `checksum_type`.
|
||||
/// Returns `Ok(())` if the checksums match, `Err` otherwise.
|
||||
pub fn verify_checksum(
|
||||
data: &[u8],
|
||||
expected: &str,
|
||||
checksum_type: ChecksumType,
|
||||
) -> Result<(), String> {
|
||||
let actual = match checksum_type {
|
||||
ChecksumType::Sha256 => {
|
||||
let mut hasher = Sha256::new();
|
||||
hasher.update(data);
|
||||
format!("{:x}", hasher.finalize())
|
||||
}
|
||||
ChecksumType::Sha512 => {
|
||||
let mut hasher = Sha512::new();
|
||||
hasher.update(data);
|
||||
format!("{:x}", hasher.finalize())
|
||||
}
|
||||
};
|
||||
|
||||
if actual.eq_ignore_ascii_case(expected) {
|
||||
Ok(())
|
||||
} else {
|
||||
Err(format!(
|
||||
"Checksum mismatch: expected {}..., got {}...",
|
||||
&expected[..8.min(expected.len())],
|
||||
&actual[..8.min(actual.len())]
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_detect_checksum_type_empty() {
|
||||
assert_eq!(detect_checksum_type("").unwrap(), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_detect_checksum_type_md5_rejected() {
|
||||
let result = detect_checksum_type("a".repeat(32).as_str());
|
||||
assert!(result.is_err());
|
||||
assert!(result.unwrap_err().contains("MD5"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_detect_checksum_type_sha1_rejected() {
|
||||
let result = detect_checksum_type("a".repeat(40).as_str());
|
||||
assert!(result.is_err());
|
||||
assert!(result.unwrap_err().contains("SHA1"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_detect_checksum_type_sha256() {
|
||||
assert_eq!(
|
||||
detect_checksum_type("a".repeat(64).as_str()).unwrap(),
|
||||
Some(ChecksumType::Sha256)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_detect_checksum_type_sha512() {
|
||||
assert_eq!(
|
||||
detect_checksum_type("a".repeat(128).as_str()).unwrap(),
|
||||
Some(ChecksumType::Sha512)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_detect_checksum_type_invalid_length() {
|
||||
let result = detect_checksum_type("a".repeat(50).as_str());
|
||||
assert!(result.is_err());
|
||||
assert!(result.unwrap_err().contains("Invalid checksum length"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_verify_checksum_sha256_valid() {
|
||||
let data = b"hello";
|
||||
let expected = "2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824";
|
||||
assert!(verify_checksum(data, expected, ChecksumType::Sha256).is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_verify_checksum_sha256_invalid() {
|
||||
let data = b"hello";
|
||||
let expected = "0000000000000000000000000000000000000000000000000000000000000000";
|
||||
assert!(verify_checksum(data, expected, ChecksumType::Sha256).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_verify_checksum_sha512_valid() {
|
||||
let data = b"hello";
|
||||
let expected = "9b71d224bd62f3785d96d46ad3ea3d73319bfbc2890caadae2dff72519673ca72323c3d99ba5c11d7c7acc6e14b8c5da0c4663475c2e5c3adef46f73bcdec043";
|
||||
assert!(verify_checksum(data, expected, ChecksumType::Sha512).is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_verify_checksum_sha512_invalid() {
|
||||
let data = b"hello";
|
||||
let expected = "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000";
|
||||
assert!(verify_checksum(data, expected, ChecksumType::Sha512).is_err());
|
||||
}
|
||||
}
|
||||
@@ -1,169 +0,0 @@
|
||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
||||
use crate::finalize::plugin::PluginError;
|
||||
use std::path::{Path, PathBuf};
|
||||
use tokio::task;
|
||||
|
||||
/// Compression type for tar archives.
|
||||
#[derive(Debug, Clone, Copy, PartialEq)]
|
||||
pub enum CompressionType {
|
||||
/// Gzip compressed (.tar.gz, .tgz)
|
||||
Gzip,
|
||||
/// Zstd compressed (.tar.zst, .tar.zstd, .tzst)
|
||||
Zstd,
|
||||
/// Uncompressed tar (.tar)
|
||||
None,
|
||||
}
|
||||
|
||||
impl CompressionType {
|
||||
/// Detects compression type from file path extension.
|
||||
///
|
||||
/// Supports: .tar.gz, .tgz (Gzip), .tar.zst, .tar.zstd, .tzst (Zstd), .tar (None).
|
||||
pub fn from_path(path: &std::path::Path) -> Result<Self, String> {
|
||||
let name = path
|
||||
.file_name()
|
||||
.and_then(|n| n.to_str())
|
||||
.unwrap_or("")
|
||||
.to_lowercase();
|
||||
|
||||
if name.ends_with(".tar.gz") || name.ends_with(".tgz") {
|
||||
Ok(CompressionType::Gzip)
|
||||
} else if name.ends_with(".tar.zst")
|
||||
|| name.ends_with(".tar.zstd")
|
||||
|| name.ends_with(".tzst")
|
||||
{
|
||||
Ok(CompressionType::Zstd)
|
||||
} else if name.ends_with(".tar") {
|
||||
Ok(CompressionType::None)
|
||||
} else {
|
||||
Err(format!(
|
||||
"Unsupported archive format: '{}'. Supported formats: .tar, .tar.gz, .tgz, .tar.zst, .tar.zstd, .tzst",
|
||||
name
|
||||
))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Extracts a tar archive to the destination directory.
|
||||
///
|
||||
/// Removes existing destination directory if present, creates a fresh directory,
|
||||
/// then unpacks the archive using the appropriate decompression based on `compression`.
|
||||
pub async fn extract_tar(
|
||||
archive: &Path,
|
||||
destdir: &PathBuf,
|
||||
compression: CompressionType,
|
||||
) -> Result<(), PluginError> {
|
||||
if destdir.exists() {
|
||||
tokio::fs::remove_dir_all(destdir).await.map_err(|e| {
|
||||
PluginError::GeneralError(format!("Failed to remove existing directory: {}", e))
|
||||
})?;
|
||||
}
|
||||
tokio::fs::create_dir_all(destdir)
|
||||
.await
|
||||
.map_err(|e| PluginError::GeneralError(format!("Failed to create directory: {}", e)))?;
|
||||
|
||||
let archive = archive.to_path_buf();
|
||||
let destdir = destdir.clone();
|
||||
|
||||
task::spawn_blocking(move || {
|
||||
let file = std::fs::File::open(&archive)
|
||||
.map_err(|e| PluginError::GeneralError(format!("Failed to open archive: {}", e)))?;
|
||||
|
||||
match compression {
|
||||
CompressionType::Gzip => {
|
||||
let decoder = flate2::read::GzDecoder::new(file);
|
||||
let mut archive = tar::Archive::new(decoder);
|
||||
archive.unpack(&destdir).map_err(|e| {
|
||||
PluginError::GeneralError(format!("Failed to unpack archive: {}", e))
|
||||
})?;
|
||||
}
|
||||
CompressionType::Zstd => {
|
||||
let decoder = zstd::stream::read::Decoder::new(file).map_err(|e| {
|
||||
PluginError::GeneralError(format!("Failed to create zstd decoder: {}", e))
|
||||
})?;
|
||||
let mut archive = tar::Archive::new(decoder);
|
||||
archive.unpack(&destdir).map_err(|e| {
|
||||
PluginError::GeneralError(format!("Failed to unpack archive: {}", e))
|
||||
})?;
|
||||
}
|
||||
CompressionType::None => {
|
||||
let mut archive = tar::Archive::new(file);
|
||||
archive.unpack(&destdir).map_err(|e| {
|
||||
PluginError::GeneralError(format!("Failed to unpack archive: {}", e))
|
||||
})?;
|
||||
}
|
||||
};
|
||||
|
||||
Ok::<(), PluginError>(())
|
||||
})
|
||||
.await
|
||||
.map_err(|e| PluginError::GeneralError(format!("Task join error: {}", e)))??;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::path::Path;
|
||||
|
||||
#[test]
|
||||
fn test_from_path_tar_gz() {
|
||||
assert_eq!(
|
||||
CompressionType::from_path(Path::new("archive.tar.gz")).unwrap(),
|
||||
CompressionType::Gzip
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_from_path_tgz() {
|
||||
assert_eq!(
|
||||
CompressionType::from_path(Path::new("archive.tgz")).unwrap(),
|
||||
CompressionType::Gzip
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_from_path_tar_zst() {
|
||||
assert_eq!(
|
||||
CompressionType::from_path(Path::new("archive.tar.zst")).unwrap(),
|
||||
CompressionType::Zstd
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_from_path_tar_zstd() {
|
||||
assert_eq!(
|
||||
CompressionType::from_path(Path::new("archive.tar.zstd")).unwrap(),
|
||||
CompressionType::Zstd
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_from_path_tzst() {
|
||||
assert_eq!(
|
||||
CompressionType::from_path(Path::new("archive.tzst")).unwrap(),
|
||||
CompressionType::Zstd
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_from_path_tar() {
|
||||
assert_eq!(
|
||||
CompressionType::from_path(Path::new("archive.tar")).unwrap(),
|
||||
CompressionType::None
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_from_path_unsupported() {
|
||||
assert!(CompressionType::from_path(Path::new("archive.zip")).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_from_path_case_insensitive() {
|
||||
assert_eq!(
|
||||
CompressionType::from_path(Path::new("archive.TAR.GZ")).unwrap(),
|
||||
CompressionType::Gzip
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -1,185 +0,0 @@
|
||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
||||
use crate::finalize::plugin::PluginError;
|
||||
use std::path::Path;
|
||||
use tokio::process::Command;
|
||||
|
||||
/// Version specification for a git repository.
|
||||
#[derive(Debug, PartialEq)]
|
||||
pub enum GitVersion {
|
||||
/// No version specified (invalid for fetch operations)
|
||||
None,
|
||||
/// Single version string (tag or commit hash)
|
||||
Single(String),
|
||||
/// Mixed: try tag first, fallback to commit hash
|
||||
Mixed(String, String),
|
||||
}
|
||||
|
||||
/// Parses a git URL with optional version specification.
|
||||
///
|
||||
/// Format: `url` (no version), `url@tag` (single version), `url@tag:commit` (mixed version).
|
||||
/// Returns (base_url, GitVersion).
|
||||
pub fn parse_git_url(url: &str) -> (String, GitVersion) {
|
||||
if let Some((base, version)) = url.rsplit_once('@') {
|
||||
if version.contains(":") {
|
||||
let (tag, commit) = version.split_once(':').unwrap();
|
||||
return (
|
||||
base.to_string(),
|
||||
GitVersion::Mixed(tag.to_string(), commit.to_string()),
|
||||
);
|
||||
}
|
||||
return (base.to_string(), GitVersion::Single(version.to_string()));
|
||||
}
|
||||
(url.to_string(), GitVersion::None)
|
||||
}
|
||||
|
||||
/// Clones a git repository and checks out the specified version.
|
||||
///
|
||||
/// Removes existing destination directory, performs shallow or full clone,
|
||||
/// then checks out the given version (tag or commit).
|
||||
pub async fn git_clone_and_checkout(
|
||||
baseurl: &str,
|
||||
version: &str,
|
||||
dest: &Path,
|
||||
name: &str,
|
||||
shallow: bool,
|
||||
) -> Result<(), PluginError> {
|
||||
if dest.exists() {
|
||||
tokio::fs::remove_dir_all(dest)
|
||||
.await
|
||||
.map_err(|e| PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: baseurl.to_string(),
|
||||
reason: format!("Failed to remove existing directory: {}", e),
|
||||
})?;
|
||||
}
|
||||
let mut cmd = Command::new("git");
|
||||
cmd.arg("clone");
|
||||
if shallow {
|
||||
cmd.arg("--depth").arg("1");
|
||||
}
|
||||
cmd.arg(baseurl).arg(dest);
|
||||
let output = cmd.output().await.map_err(|e| PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: baseurl.to_string(),
|
||||
reason: e.to_string(),
|
||||
})?;
|
||||
if !output.status.success() {
|
||||
return Err(PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: baseurl.to_string(),
|
||||
reason: String::from_utf8_lossy(&output.stderr).to_string(),
|
||||
});
|
||||
}
|
||||
let mut cmd = Command::new("git");
|
||||
cmd.arg("checkout").arg(version).current_dir(dest);
|
||||
let output = cmd.output().await.map_err(|e| PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: baseurl.to_string(),
|
||||
reason: e.to_string(),
|
||||
})?;
|
||||
if !output.status.success() {
|
||||
return Err(PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: baseurl.to_string(),
|
||||
reason: String::from_utf8_lossy(&output.stderr).to_string(),
|
||||
});
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Fetches a plugin from a git repository.
|
||||
///
|
||||
/// Parses the URL for version info, then clones and checks out the appropriate ref.
|
||||
/// For Mixed version, tries the tag first and falls back to the commit hash.
|
||||
pub async fn fetch_git_plugin(
|
||||
path: &str,
|
||||
name: &str,
|
||||
basedir: &Path,
|
||||
shallow: Option<bool>,
|
||||
) -> Result<(), PluginError> {
|
||||
let (baseurl, version) = parse_git_url(path);
|
||||
match version {
|
||||
GitVersion::None => {
|
||||
log::error!("Version should be specified for git plugin URL: {}", path);
|
||||
Err(PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: path.to_string(),
|
||||
reason: "Version should be specified as either a tag or a commit hash.".to_string(),
|
||||
})
|
||||
}
|
||||
GitVersion::Single(version) => {
|
||||
log::debug!("Fetching git plugin: {}@{}", name, version);
|
||||
let shallow_flag = shallow.unwrap_or(false);
|
||||
git_clone_and_checkout(
|
||||
&baseurl,
|
||||
&version.to_string(),
|
||||
&basedir.join(name),
|
||||
name,
|
||||
shallow_flag,
|
||||
)
|
||||
.await
|
||||
}
|
||||
GitVersion::Mixed(tag, commit) => {
|
||||
log::debug!("Fetching git plugin: {}@{}/{}", name, tag, commit);
|
||||
let shallow_flag = shallow.unwrap_or(false);
|
||||
let result =
|
||||
git_clone_and_checkout(&baseurl, &tag, &basedir.join(name), name, shallow_flag)
|
||||
.await;
|
||||
if result.is_err() {
|
||||
log::warn!(
|
||||
"Failed to clone with tag, trying commit hash. URL: {}",
|
||||
path
|
||||
);
|
||||
return git_clone_and_checkout(&baseurl, &commit, &basedir.join(name), name, false)
|
||||
.await;
|
||||
}
|
||||
result
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_parse_git_url_no_version() {
|
||||
let (url, version) = parse_git_url("https://github.com/user/repo");
|
||||
assert_eq!(url, "https://github.com/user/repo");
|
||||
assert_eq!(version, GitVersion::None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_parse_git_url_single_tag() {
|
||||
let (url, version) = parse_git_url("https://github.com/user/repo@v1.0.0");
|
||||
assert_eq!(url, "https://github.com/user/repo");
|
||||
assert_eq!(version, GitVersion::Single("v1.0.0".to_string()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_parse_git_url_mixed_version() {
|
||||
let (url, version) = parse_git_url("https://github.com/user/repo@main:abc123");
|
||||
assert_eq!(url, "https://github.com/user/repo");
|
||||
assert_eq!(
|
||||
version,
|
||||
GitVersion::Mixed("main".to_string(), "abc123".to_string())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_parse_git_url_complex_url() {
|
||||
let (url, version) = parse_git_url("git@github.com:user/repo.git@tag:commit");
|
||||
assert_eq!(url, "git@github.com:user/repo.git");
|
||||
assert_eq!(
|
||||
version,
|
||||
GitVersion::Mixed("tag".to_string(), "commit".to_string())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_parse_git_url_single_with_at_in_path() {
|
||||
let (url, version) = parse_git_url("https://github.com/user/repo@my-repo@v1.0");
|
||||
assert_eq!(url, "https://github.com/user/repo@my-repo");
|
||||
assert_eq!(version, GitVersion::Single("v1.0".to_string()));
|
||||
}
|
||||
}
|
||||
@@ -1,110 +0,0 @@
|
||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
||||
use super::checksum::{detect_checksum_type, verify_checksum};
|
||||
use super::extract::{CompressionType, extract_tar};
|
||||
use crate::finalize::plugin::PluginError;
|
||||
use std::path::Path;
|
||||
|
||||
pub async fn download_to_file(url: &str, dest: &Path, name: &str) -> Result<(), PluginError> {
|
||||
let client = reqwest::Client::builder()
|
||||
.timeout(std::time::Duration::from_secs(300))
|
||||
.build()
|
||||
.map_err(|e| PluginError::GeneralError(format!("Failed to build HTTP client: {}", e)))?;
|
||||
|
||||
let response = client
|
||||
.get(url)
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: url.to_string(),
|
||||
reason: format!("HTTP request failed: {}", e),
|
||||
})?;
|
||||
|
||||
let status = response.status();
|
||||
if !status.is_success() {
|
||||
return Err(PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: url.to_string(),
|
||||
reason: format!("HTTP error: {}", status),
|
||||
});
|
||||
}
|
||||
|
||||
let bytes = response
|
||||
.bytes()
|
||||
.await
|
||||
.map_err(|e| PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: url.to_string(),
|
||||
reason: format!("Failed to read response body: {}", e),
|
||||
})?;
|
||||
|
||||
tokio::fs::write(dest, &bytes)
|
||||
.await
|
||||
.map_err(|e| PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: url.to_string(),
|
||||
reason: format!("Failed to write file: {}", e),
|
||||
})?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn extract_archive_extension(url: &str) -> &str {
|
||||
let path_part = url
|
||||
.split('?')
|
||||
.next()
|
||||
.unwrap_or(url)
|
||||
.split('#')
|
||||
.next()
|
||||
.unwrap_or(url);
|
||||
let file_part = path_part.rsplit('/').next().unwrap_or("");
|
||||
let lower = file_part.to_lowercase();
|
||||
if lower.ends_with(".tar.gz") || lower.ends_with(".tgz") {
|
||||
".tar.gz"
|
||||
} else if lower.ends_with(".tar.zst")
|
||||
|| lower.ends_with(".tar.zstd")
|
||||
|| lower.ends_with(".tzst")
|
||||
{
|
||||
".tar.zst"
|
||||
} else if lower.ends_with(".tar") {
|
||||
".tar"
|
||||
} else {
|
||||
".unknown"
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn fetch_http_plugin(
|
||||
url: &str,
|
||||
name: &str,
|
||||
basedir: &Path,
|
||||
checksum_str: Option<&str>,
|
||||
) -> Result<(), PluginError> {
|
||||
let temp_dir = std::env::temp_dir().join("workshop-plugins");
|
||||
tokio::fs::create_dir_all(&temp_dir).await.ok();
|
||||
let ext = extract_archive_extension(url);
|
||||
let temp_file = temp_dir.join(format!("{}-{}{}", name, uuid::Uuid::new_v4(), ext));
|
||||
|
||||
download_to_file(url, &temp_file, name).await?;
|
||||
|
||||
if let Some(checksum) = checksum_str
|
||||
&& !checksum.is_empty()
|
||||
{
|
||||
let bytes = tokio::fs::read(&temp_file).await.map_err(|e| {
|
||||
PluginError::GeneralError(format!("Failed to read downloaded file: {}", e))
|
||||
})?;
|
||||
|
||||
let checksum_type = detect_checksum_type(checksum).map_err(PluginError::GeneralError)?;
|
||||
|
||||
if let Some(ct) = checksum_type {
|
||||
verify_checksum(&bytes, checksum, ct).map_err(PluginError::GeneralError)?;
|
||||
}
|
||||
}
|
||||
|
||||
let compression = CompressionType::from_path(&temp_file).map_err(PluginError::GeneralError)?;
|
||||
let destdir = basedir.join(name);
|
||||
extract_tar(&temp_file, &destdir, compression).await?;
|
||||
|
||||
tokio::fs::remove_file(&temp_file).await.ok();
|
||||
|
||||
Ok(())
|
||||
}
|
||||
@@ -1,84 +0,0 @@
|
||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
/// Supported checksum types for verifying downloaded artifacts.
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub enum ChecksumType {
|
||||
/// SHA-256 hash (64 hex characters)
|
||||
Sha256,
|
||||
/// SHA-512 hash (128 hex characters)
|
||||
Sha512,
|
||||
}
|
||||
|
||||
/// Arguments for fetching a plugin artifact.
|
||||
///
|
||||
/// Contains optional checksum verification and shallow clone settings.
|
||||
#[derive(Debug, Deserialize, Serialize, Clone, Default)]
|
||||
pub struct FetchArgument {
|
||||
/// SHA256 or SHA512 hex string for integrity verification.
|
||||
#[serde(default)]
|
||||
pub checksum: Option<String>,
|
||||
/// Perform a shallow clone (single revision only).
|
||||
#[serde(default)]
|
||||
pub shallow: Option<bool>,
|
||||
}
|
||||
|
||||
impl FetchArgument {
|
||||
/// Returns the checksum if present and non-empty.
|
||||
pub fn get_checksum(&self) -> Option<&str> {
|
||||
self.checksum.as_deref().filter(|s| !s.is_empty())
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_get_checksum_some_non_empty() {
|
||||
let arg = FetchArgument {
|
||||
checksum: Some("abc123".to_string()),
|
||||
shallow: None,
|
||||
};
|
||||
assert_eq!(arg.get_checksum(), Some("abc123"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_get_checksum_empty_string() {
|
||||
let arg = FetchArgument {
|
||||
checksum: Some("".to_string()),
|
||||
shallow: None,
|
||||
};
|
||||
assert_eq!(arg.get_checksum(), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_get_checksum_none() {
|
||||
let arg = FetchArgument {
|
||||
checksum: None,
|
||||
shallow: None,
|
||||
};
|
||||
assert_eq!(arg.get_checksum(), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_serde_roundtrip() {
|
||||
let arg = FetchArgument {
|
||||
checksum: Some("deadbeef".to_string()),
|
||||
shallow: Some(true),
|
||||
};
|
||||
let json = serde_json::to_string(&arg).unwrap();
|
||||
let parsed: FetchArgument = serde_json::from_str(&json).unwrap();
|
||||
assert_eq!(parsed.checksum, arg.checksum);
|
||||
assert_eq!(parsed.shallow, arg.shallow);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_serde_roundtrip_empty() {
|
||||
let arg = FetchArgument::default();
|
||||
let json = serde_json::to_string(&arg).unwrap();
|
||||
let parsed: FetchArgument = serde_json::from_str(&json).unwrap();
|
||||
assert_eq!(parsed.checksum, arg.checksum);
|
||||
assert_eq!(parsed.shallow, arg.shallow);
|
||||
}
|
||||
}
|
||||
@@ -29,3 +29,10 @@ pub fn get_internal_plugin() -> Vec<InternalPlugin> {
|
||||
},
|
||||
]
|
||||
}
|
||||
|
||||
pub fn get_dummy_plugin() -> Vec<InternalPlugin> {
|
||||
vec![InternalPlugin {
|
||||
name: "dummy",
|
||||
entry: Box::new(dummy::Dummy),
|
||||
}]
|
||||
}
|
||||
|
||||
@@ -1,12 +1,13 @@
|
||||
use crate::finalize::plugin::PluginError;
|
||||
use crate::finalize::plugin::PluginMetadata;
|
||||
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn};
|
||||
use workshop_engine::{EventSender, ExecutionContext};
|
||||
use crate::{ finalize::plugin::AsyncPluginFn};
|
||||
use async_trait::async_trait;
|
||||
|
||||
pub struct _Dummy;
|
||||
pub struct Dummy;
|
||||
|
||||
#[async_trait]
|
||||
impl AsyncPluginFn for _Dummy {
|
||||
impl AsyncPluginFn for Dummy {
|
||||
async fn call(
|
||||
&self,
|
||||
_metadata: &PluginMetadata,
|
||||
@@ -14,6 +15,7 @@ impl AsyncPluginFn for _Dummy {
|
||||
_ctx: &ExecutionContext,
|
||||
_event_tx: &EventSender,
|
||||
) -> Result<(), PluginError> {
|
||||
log::debug!("[NOTIFY_DEBUG] Successfully registered dummy plugin.");
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,12 +1,10 @@
|
||||
use crate::finalize::plugin::PluginMetadata;
|
||||
use crate::{
|
||||
EventSender, ExecutionContext,
|
||||
finalize::{error::PluginError, plugin::AsyncPluginFn},
|
||||
};
|
||||
use async_trait::async_trait;
|
||||
|
||||
use workshop_engine::{EventSender, ExecutionContext};
|
||||
pub struct InsiteNotify;
|
||||
|
||||
#[async_trait]
|
||||
impl AsyncPluginFn for InsiteNotify {
|
||||
async fn call(
|
||||
@@ -16,6 +14,7 @@ impl AsyncPluginFn for InsiteNotify {
|
||||
_ctx: &ExecutionContext,
|
||||
_event_tx: &EventSender,
|
||||
) -> Result<(), PluginError> {
|
||||
unimplemented!("Requires workshop-base to be implemented!");
|
||||
log::info!("in-site-notify: placeholder (always succeeds)");
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,18 +1,9 @@
|
||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
||||
use crate::finalize::plugin::PluginError;
|
||||
use crate::finalize::plugin::PluginMetadata;
|
||||
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn};
|
||||
use crate::{
|
||||
finalize::{error::PluginError, plugin::AsyncPluginFn},
|
||||
};
|
||||
use async_trait::async_trait;
|
||||
use lettre::message::{MultiPart, SinglePart, header};
|
||||
use lettre::{AsyncSmtpTransport, AsyncTransport, Message, Tokio1Executor};
|
||||
use std::time::Duration;
|
||||
use tokio::time::timeout;
|
||||
|
||||
mod config;
|
||||
mod template;
|
||||
|
||||
pub use config::*;
|
||||
use template::*;
|
||||
use workshop_engine::{EventSender, ExecutionContext};
|
||||
|
||||
pub struct Mail;
|
||||
|
||||
@@ -21,182 +12,10 @@ impl AsyncPluginFn for Mail {
|
||||
async fn call(
|
||||
&self,
|
||||
_metadata: &PluginMetadata,
|
||||
argument: serde_yaml::Value,
|
||||
ctx: &ExecutionContext,
|
||||
_argument: serde_yaml::Value,
|
||||
_ctx: &ExecutionContext,
|
||||
_event_tx: &EventSender,
|
||||
) -> Result<(), PluginError> {
|
||||
let config: MailConfig = serde_yaml::from_value(argument)
|
||||
.map_err(|e| PluginError::GeneralError(format!("Invalid mail config: {}", e)))?;
|
||||
|
||||
if let Err(errors) = config.validate() {
|
||||
return Err(PluginError::GeneralError(format!(
|
||||
"Mail config validation failed: {:?}",
|
||||
errors
|
||||
)));
|
||||
}
|
||||
|
||||
let template_data = prepare_template_data(ctx);
|
||||
|
||||
let (subject_template, body_template_html, body_template_text) = match config.schema {
|
||||
Schema::Default => {
|
||||
let subject = config
|
||||
.subject
|
||||
.as_deref()
|
||||
.unwrap_or(DefaultTemplates::SUBJECT);
|
||||
let body_html = config
|
||||
.body
|
||||
.as_deref()
|
||||
.unwrap_or(DefaultTemplates::BODY_HTML);
|
||||
(
|
||||
subject.to_string(),
|
||||
body_html.to_string(),
|
||||
DefaultTemplates::BODY_TEXT.to_string(),
|
||||
)
|
||||
}
|
||||
Schema::Custom => {
|
||||
let subject = config.subject.clone().unwrap();
|
||||
let body_html = config.body.clone().unwrap();
|
||||
(subject, body_html.clone(), body_html)
|
||||
}
|
||||
};
|
||||
|
||||
let subject = render_template(&subject_template, &template_data)
|
||||
.map_err(|e| PluginError::GeneralError(format!("Failed to render subject: {}", e)))?;
|
||||
let body_html = render_template(&body_template_html, &template_data)
|
||||
.map_err(|e| PluginError::GeneralError(format!("Failed to render body_html: {}", e)))?;
|
||||
let body_text = render_template(&body_template_text, &template_data).map_err(|e| {
|
||||
PluginError::GeneralError(format!("Failed to render body_text(?): {}", e))
|
||||
})?;
|
||||
|
||||
let email = build_email(&config, subject, body_html, body_text)?;
|
||||
|
||||
send_email(&config, email).await
|
||||
todo!();
|
||||
}
|
||||
}
|
||||
|
||||
fn build_email(
|
||||
config: &MailConfig,
|
||||
subject: String,
|
||||
body_html: String,
|
||||
body_text: String,
|
||||
) -> Result<Message, PluginError> {
|
||||
let from_str = config.from.to_string();
|
||||
let from: lettre::message::Mailbox = from_str.parse().map_err(|e| {
|
||||
PluginError::GeneralError(format!("Invalid from address '{}': {}", from_str, e))
|
||||
})?;
|
||||
|
||||
let to_addrs: Vec<String> = config.to.to_vec();
|
||||
if to_addrs.is_empty() {
|
||||
return Err(PluginError::GeneralError(
|
||||
"No recipients specified".to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
let mut message_builder = Message::builder().from(from).subject(subject);
|
||||
|
||||
for addr in &to_addrs {
|
||||
let mailbox: lettre::message::Mailbox = addr.parse().map_err(|e| {
|
||||
PluginError::GeneralError(format!("Invalid to address '{}': {}", addr, e))
|
||||
})?;
|
||||
message_builder = message_builder.to(mailbox);
|
||||
}
|
||||
|
||||
if let Some(cc) = &config.cc {
|
||||
for addr in cc.to_vec() {
|
||||
let mailbox: lettre::message::Mailbox = addr.parse().map_err(|e| {
|
||||
PluginError::GeneralError(format!("Invalid cc address '{}': {}", addr, e))
|
||||
})?;
|
||||
message_builder = message_builder.cc(mailbox);
|
||||
}
|
||||
}
|
||||
|
||||
if let Some(bcc) = &config.bcc {
|
||||
for addr in bcc.to_vec() {
|
||||
let mailbox: lettre::message::Mailbox = addr.parse().map_err(|e| {
|
||||
PluginError::GeneralError(format!("Invalid bcc address '{}': {}", addr, e))
|
||||
})?;
|
||||
message_builder = message_builder.bcc(mailbox);
|
||||
}
|
||||
}
|
||||
|
||||
if let Some(reply_to) = &config.reply_to {
|
||||
let mailbox: lettre::message::Mailbox = reply_to.parse().map_err(|e| {
|
||||
PluginError::GeneralError(format!("Invalid reply-to address '{}': {}", reply_to, e))
|
||||
})?;
|
||||
message_builder = message_builder.reply_to(mailbox);
|
||||
}
|
||||
|
||||
let message = message_builder
|
||||
.multipart(
|
||||
MultiPart::alternative()
|
||||
.singlepart(
|
||||
SinglePart::builder()
|
||||
.header(header::ContentType::TEXT_PLAIN)
|
||||
.body(body_text),
|
||||
)
|
||||
.singlepart(
|
||||
SinglePart::builder()
|
||||
.header(header::ContentType::TEXT_HTML)
|
||||
.body(body_html),
|
||||
),
|
||||
)
|
||||
.map_err(|e| PluginError::GeneralError(format!("Failed to build email: {}", e)))?;
|
||||
|
||||
Ok(message)
|
||||
}
|
||||
|
||||
async fn send_email(config: &MailConfig, email: Message) -> Result<(), PluginError> {
|
||||
let connect_timeout = if config.smtp.connect_timeout_ms == 0 {
|
||||
None
|
||||
} else {
|
||||
Some(Duration::from_millis(config.smtp.connect_timeout_ms))
|
||||
};
|
||||
|
||||
let mut transport_builder = match config.smtp.encryption {
|
||||
Encryption::Tls => {
|
||||
log::debug!("Using TLS encryption");
|
||||
AsyncSmtpTransport::<Tokio1Executor>::relay(&config.smtp.host)
|
||||
.map_err(|e| PluginError::GeneralError(format!("Invalid SMTP host: {}", e)))?
|
||||
}
|
||||
Encryption::Starttls => {
|
||||
log::debug!("Using StartTLS encryption");
|
||||
AsyncSmtpTransport::<Tokio1Executor>::starttls_relay(&config.smtp.host)
|
||||
.map_err(|e| PluginError::GeneralError(format!("Invalid SMTP host: {}", e)))?
|
||||
}
|
||||
Encryption::None => {
|
||||
log::debug!("Using NO encryption");
|
||||
AsyncSmtpTransport::<Tokio1Executor>::builder_dangerous(&config.smtp.host)
|
||||
}
|
||||
};
|
||||
|
||||
transport_builder = transport_builder.port(config.smtp.port);
|
||||
|
||||
match config.smtp.auth.auth_type {
|
||||
AuthType::Password => {
|
||||
log::debug!("Using password auth");
|
||||
transport_builder = transport_builder.credentials(
|
||||
(
|
||||
config.smtp.auth.username.clone(),
|
||||
config.smtp.auth.password.clone(),
|
||||
)
|
||||
.into(),
|
||||
);
|
||||
}
|
||||
AuthType::None => {
|
||||
log::debug!("Using no auth");
|
||||
}
|
||||
}
|
||||
|
||||
let transport = transport_builder.build();
|
||||
|
||||
let result = match connect_timeout {
|
||||
Some(t) => timeout(t, transport.send(email)).await,
|
||||
None => Ok(transport.send(email).await),
|
||||
};
|
||||
|
||||
result
|
||||
.map_err(|_| PluginError::GeneralError("SMTP connection timed out".to_string()))?
|
||||
.map_err(|e| PluginError::GeneralError(format!("SMTP error: {}", e)))?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -1,490 +0,0 @@
|
||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
||||
use serde::Deserialize;
|
||||
use workshop_engine::deserialize_duration_ms;
|
||||
|
||||
/// Email address representation supporting plain string or structured object.
|
||||
#[derive(Debug, Deserialize, Clone)]
|
||||
#[serde(untagged)]
|
||||
pub enum EmailAddress {
|
||||
/// Plain email address string.
|
||||
String(String),
|
||||
/// Structured email with display name.
|
||||
Object { name: String, email: String },
|
||||
}
|
||||
|
||||
impl EmailAddress {
|
||||
/// Returns the formatted address string (e.g., "user@example.com" or "Name <user@example.com>").
|
||||
pub fn to_string(&self) -> String {
|
||||
match self {
|
||||
EmailAddress::String(email) => email.clone(),
|
||||
EmailAddress::Object { name, email } => {
|
||||
if name.is_empty() {
|
||||
email.clone()
|
||||
} else {
|
||||
format!("{} <{}>", name, email)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Returns the raw email address without formatting.
|
||||
pub fn email(&self) -> &str {
|
||||
match self {
|
||||
EmailAddress::String(email) => email.as_str(),
|
||||
EmailAddress::Object { email, .. } => email.as_str(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Recipients container supporting single or multiple addresses.
|
||||
#[derive(Debug, Deserialize, Clone)]
|
||||
#[serde(untagged)]
|
||||
pub enum Recipients {
|
||||
/// Single recipient.
|
||||
Single(String),
|
||||
/// Multiple recipients.
|
||||
Multiple(Vec<String>),
|
||||
}
|
||||
|
||||
impl Recipients {
|
||||
/// Converts recipients to a flat vector of email strings.
|
||||
pub fn to_vec(&self) -> Vec<String> {
|
||||
match self {
|
||||
Recipients::Single(s) => vec![s.clone()],
|
||||
Recipients::Multiple(v) => v.clone(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// SMTP authentication mechanism.
|
||||
#[derive(Debug, Deserialize, Clone)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub enum AuthType {
|
||||
/// No authentication.
|
||||
None,
|
||||
/// Password-based authentication.
|
||||
Password,
|
||||
}
|
||||
|
||||
/// SMTP authentication configuration.
|
||||
#[derive(Debug, Deserialize, Clone)]
|
||||
pub struct AuthConfig {
|
||||
/// Authentication mechanism type.
|
||||
#[serde(rename = "type")]
|
||||
pub auth_type: AuthType,
|
||||
/// Username for authentication.
|
||||
pub username: String,
|
||||
/// Password for authentication.
|
||||
pub password: String,
|
||||
}
|
||||
|
||||
/// SMTP connection encryption mode.
|
||||
#[derive(Debug, Deserialize, Clone)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub enum Encryption {
|
||||
/// Implicit TLS/SSL on connection.
|
||||
Tls,
|
||||
/// STARTTLS upgrade after initial plaintext connection.
|
||||
Starttls,
|
||||
/// No encryption.
|
||||
None,
|
||||
}
|
||||
|
||||
/// SMTP server connection configuration.
|
||||
#[derive(Debug, Deserialize, Clone)]
|
||||
pub struct SmtpConfig {
|
||||
/// SMTP server hostname.
|
||||
pub host: String,
|
||||
/// SMTP server port.
|
||||
pub port: u16,
|
||||
/// Authentication configuration.
|
||||
pub auth: AuthConfig,
|
||||
/// Encryption mode.
|
||||
pub encryption: Encryption,
|
||||
/// Connection timeout in milliseconds.
|
||||
#[serde(
|
||||
default = "default_connect_timeout_ms",
|
||||
deserialize_with = "deserialize_duration_ms"
|
||||
)]
|
||||
pub connect_timeout_ms: u64,
|
||||
}
|
||||
|
||||
fn default_connect_timeout_ms() -> u64 {
|
||||
30_000
|
||||
}
|
||||
|
||||
/// Mail schema type determining required fields.
|
||||
#[derive(Debug, Deserialize, Clone)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub enum Schema {
|
||||
/// Default schema with minimal required fields.
|
||||
Default,
|
||||
/// Custom schema requiring explicit subject and body.
|
||||
Custom,
|
||||
}
|
||||
|
||||
fn default_schema() -> Schema {
|
||||
Schema::Default
|
||||
}
|
||||
|
||||
/// Complete mail notification configuration.
|
||||
#[derive(Debug, Deserialize, Clone)]
|
||||
pub struct MailConfig {
|
||||
/// Primary recipients.
|
||||
pub to: Recipients,
|
||||
/// Carbon copy recipients.
|
||||
#[serde(default)]
|
||||
pub cc: Option<Recipients>,
|
||||
/// Blind carbon copy recipients.
|
||||
#[serde(default)]
|
||||
pub bcc: Option<Recipients>,
|
||||
/// Sender address.
|
||||
pub from: EmailAddress,
|
||||
/// Reply-to address.
|
||||
#[serde(default)]
|
||||
pub reply_to: Option<String>,
|
||||
/// Schema type determining validation rules.
|
||||
#[serde(default = "default_schema")]
|
||||
pub schema: Schema,
|
||||
/// Email subject (required for custom schema).
|
||||
pub subject: Option<String>,
|
||||
/// Email body content (required for custom schema).
|
||||
pub body: Option<String>,
|
||||
/// SMTP server configuration.
|
||||
pub smtp: SmtpConfig,
|
||||
}
|
||||
|
||||
impl MailConfig {
|
||||
/// Validates the mail configuration according to schema rules.
|
||||
/// Returns Ok(()) if valid, or Err(vec) with error messages if invalid.
|
||||
pub fn validate(&self) -> Result<(), Vec<String>> {
|
||||
let mut errors = Vec::new();
|
||||
|
||||
if matches!(self.schema, Schema::Custom) {
|
||||
if self.subject.is_none() {
|
||||
errors.push("subject: required when schema=custom".to_string());
|
||||
}
|
||||
if self.body.is_none() {
|
||||
errors.push("body: required when schema=custom".to_string());
|
||||
}
|
||||
}
|
||||
|
||||
if self.to.to_vec().is_empty() {
|
||||
errors.push("to: must have at least one recipient".to_string());
|
||||
}
|
||||
|
||||
if errors.is_empty() {
|
||||
Ok(())
|
||||
} else {
|
||||
Err(errors)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Built-in email notification templates.
|
||||
pub struct DefaultTemplates;
|
||||
|
||||
impl DefaultTemplates {
|
||||
/// Default subject template with build status.
|
||||
pub const SUBJECT: &'static str =
|
||||
"[{{ build.status | upper }}] {{ pipeline.name }} #{{ build.id }}";
|
||||
|
||||
/// Default HTML email body template.
|
||||
pub const BODY_HTML: &'static str = r#"<!DOCTYPE html>
|
||||
<html>
|
||||
<body style="font-family: Arial, sans-serif;">
|
||||
<h2 style="color: {{ build.status_color }};">
|
||||
Build {{ build.status | title }}
|
||||
</h2>
|
||||
<table>
|
||||
<tr><td><strong>Pipeline:</strong></td><td>{{ pipeline.name }}</td></tr>
|
||||
<tr><td><strong>Build ID:</strong></td><td>{{ build.id }}</td></tr>
|
||||
<tr><td><strong>Status:</strong></td><td>{{ build.status }}</td></tr>
|
||||
<tr><td><strong>Duration:</strong></td><td>{{ build.duration }}</td></tr>
|
||||
<tr><td><strong>Author:</strong></td><td>{{ commit.author }}</td></tr>
|
||||
<tr><td><strong>Message:</strong></td><td>{{ commit.message }}</td></tr>
|
||||
</table>
|
||||
<p>
|
||||
<a href="{{ build.url }}" style="background: #007bff; color: white; padding: 10px 20px; text-decoration: none; border-radius: 5px;">
|
||||
View Build Details
|
||||
</a>
|
||||
</p>
|
||||
</body>
|
||||
</html>"#;
|
||||
|
||||
/// Default plain text email body template.
|
||||
pub const BODY_TEXT: &'static str = r#"Build {{ build.status | title }}
|
||||
==============================
|
||||
Pipeline: {{ pipeline.name }}
|
||||
Build ID: {{ build.id }}
|
||||
Status: {{ build.status }}
|
||||
Duration: {{ build.duration }}
|
||||
Commit: {{ commit.hash }}
|
||||
Author: {{ commit.author }}
|
||||
Message: {{ commit.message }}
|
||||
|
||||
View details: {{ build.url }}"#;
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_email_address_string_to_string() {
|
||||
let addr = EmailAddress::String("test@example.com".to_string());
|
||||
assert_eq!(addr.to_string(), "test@example.com");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_email_address_object_with_name_to_string() {
|
||||
let addr = EmailAddress::Object {
|
||||
name: "Test User".to_string(),
|
||||
email: "test@example.com".to_string(),
|
||||
};
|
||||
assert_eq!(addr.to_string(), "Test User <test@example.com>");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_email_address_object_without_name_to_string() {
|
||||
let addr = EmailAddress::Object {
|
||||
name: "".to_string(),
|
||||
email: "test@example.com".to_string(),
|
||||
};
|
||||
assert_eq!(addr.to_string(), "test@example.com");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_email_address_string_email() {
|
||||
let addr = EmailAddress::String("test@example.com".to_string());
|
||||
assert_eq!(addr.email(), "test@example.com");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_email_address_object_email() {
|
||||
let addr = EmailAddress::Object {
|
||||
name: "Test User".to_string(),
|
||||
email: "test@example.com".to_string(),
|
||||
};
|
||||
assert_eq!(addr.email(), "test@example.com");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_recipients_single_to_vec() {
|
||||
let recipients = Recipients::Single("test@example.com".to_string());
|
||||
let vec = recipients.to_vec();
|
||||
assert_eq!(vec, vec!["test@example.com"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_recipients_multiple_to_vec() {
|
||||
let recipients = Recipients::Multiple(vec![
|
||||
"a@example.com".to_string(),
|
||||
"b@example.com".to_string(),
|
||||
]);
|
||||
let vec = recipients.to_vec();
|
||||
assert_eq!(vec, vec!["a@example.com", "b@example.com"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_mail_config_validate_valid_default_schema() {
|
||||
let config = MailConfig {
|
||||
to: Recipients::Single("to@example.com".to_string()),
|
||||
cc: None,
|
||||
bcc: None,
|
||||
from: EmailAddress::String("from@example.com".to_string()),
|
||||
reply_to: None,
|
||||
schema: Schema::Default,
|
||||
subject: None,
|
||||
body: None,
|
||||
smtp: SmtpConfig {
|
||||
host: "smtp.example.com".to_string(),
|
||||
port: 587,
|
||||
auth: AuthConfig {
|
||||
auth_type: AuthType::None,
|
||||
username: "".to_string(),
|
||||
password: "".to_string(),
|
||||
},
|
||||
encryption: Encryption::Tls,
|
||||
connect_timeout_ms: 30_000,
|
||||
},
|
||||
};
|
||||
assert!(config.validate().is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_mail_config_validate_valid_custom_schema() {
|
||||
let config = MailConfig {
|
||||
to: Recipients::Single("to@example.com".to_string()),
|
||||
cc: None,
|
||||
bcc: None,
|
||||
from: EmailAddress::String("from@example.com".to_string()),
|
||||
reply_to: None,
|
||||
schema: Schema::Custom,
|
||||
subject: Some("Test Subject".to_string()),
|
||||
body: Some("Test Body".to_string()),
|
||||
smtp: SmtpConfig {
|
||||
host: "smtp.example.com".to_string(),
|
||||
port: 587,
|
||||
auth: AuthConfig {
|
||||
auth_type: AuthType::None,
|
||||
username: "".to_string(),
|
||||
password: "".to_string(),
|
||||
},
|
||||
encryption: Encryption::Tls,
|
||||
connect_timeout_ms: 30_000,
|
||||
},
|
||||
};
|
||||
assert!(config.validate().is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_mail_config_validate_custom_schema_missing_subject() {
|
||||
let config = MailConfig {
|
||||
to: Recipients::Single("to@example.com".to_string()),
|
||||
cc: None,
|
||||
bcc: None,
|
||||
from: EmailAddress::String("from@example.com".to_string()),
|
||||
reply_to: None,
|
||||
schema: Schema::Custom,
|
||||
subject: None,
|
||||
body: Some("Test Body".to_string()),
|
||||
smtp: SmtpConfig {
|
||||
host: "smtp.example.com".to_string(),
|
||||
port: 587,
|
||||
auth: AuthConfig {
|
||||
auth_type: AuthType::None,
|
||||
username: "".to_string(),
|
||||
password: "".to_string(),
|
||||
},
|
||||
encryption: Encryption::Tls,
|
||||
connect_timeout_ms: 30_000,
|
||||
},
|
||||
};
|
||||
let result = config.validate();
|
||||
assert!(result.is_err());
|
||||
assert!(result.unwrap_err().iter().any(|e| e.contains("subject")));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_mail_config_validate_custom_schema_missing_body() {
|
||||
let config = MailConfig {
|
||||
to: Recipients::Single("to@example.com".to_string()),
|
||||
cc: None,
|
||||
bcc: None,
|
||||
from: EmailAddress::String("from@example.com".to_string()),
|
||||
reply_to: None,
|
||||
schema: Schema::Custom,
|
||||
subject: Some("Test Subject".to_string()),
|
||||
body: None,
|
||||
smtp: SmtpConfig {
|
||||
host: "smtp.example.com".to_string(),
|
||||
port: 587,
|
||||
auth: AuthConfig {
|
||||
auth_type: AuthType::None,
|
||||
username: "".to_string(),
|
||||
password: "".to_string(),
|
||||
},
|
||||
encryption: Encryption::Tls,
|
||||
connect_timeout_ms: 30_000,
|
||||
},
|
||||
};
|
||||
let result = config.validate();
|
||||
assert!(result.is_err());
|
||||
assert!(result.unwrap_err().iter().any(|e| e.contains("body")));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_mail_config_validate_empty_recipients() {
|
||||
let config = MailConfig {
|
||||
to: Recipients::Multiple(vec![]),
|
||||
cc: None,
|
||||
bcc: None,
|
||||
from: EmailAddress::String("from@example.com".to_string()),
|
||||
reply_to: None,
|
||||
schema: Schema::Default,
|
||||
subject: None,
|
||||
body: None,
|
||||
smtp: SmtpConfig {
|
||||
host: "smtp.example.com".to_string(),
|
||||
port: 587,
|
||||
auth: AuthConfig {
|
||||
auth_type: AuthType::None,
|
||||
username: "".to_string(),
|
||||
password: "".to_string(),
|
||||
},
|
||||
encryption: Encryption::Tls,
|
||||
connect_timeout_ms: 30_000,
|
||||
},
|
||||
};
|
||||
let result = config.validate();
|
||||
assert!(result.is_err());
|
||||
assert!(result.unwrap_err().iter().any(|e| e.contains("recipient")));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_email_address_serde_string() {
|
||||
let json = r#""test@example.com""#;
|
||||
let addr: EmailAddress = serde_json::from_str(json).unwrap();
|
||||
assert!(matches!(addr, EmailAddress::String(s) if s == "test@example.com"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_email_address_serde_object() {
|
||||
let json = r#"{"name":"Test User","email":"test@example.com"}"#;
|
||||
let addr: EmailAddress = serde_json::from_str(json).unwrap();
|
||||
assert!(
|
||||
matches!(addr, EmailAddress::Object { name, email } if name == "Test User" && email == "test@example.com")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_recipients_serde_single() {
|
||||
let json = r#""single@example.com""#;
|
||||
let recipients: Recipients = serde_json::from_str(json).unwrap();
|
||||
assert!(matches!(recipients, Recipients::Single(s) if s == "single@example.com"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_recipients_serde_multiple() {
|
||||
let json = r#"["a@example.com","b@example.com"]"#;
|
||||
let recipients: Recipients = serde_json::from_str(json).unwrap();
|
||||
assert!(
|
||||
matches!(recipients, Recipients::Multiple(v) if v == vec!["a@example.com".to_string(), "b@example.com".to_string()])
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_auth_type_serde() {
|
||||
#[derive(serde::Deserialize, Debug)]
|
||||
struct Test {
|
||||
#[serde(rename = "type")]
|
||||
auth_type: AuthType,
|
||||
}
|
||||
let json = r#"{"type":"password"}"#;
|
||||
let t: Test = serde_json::from_str(json).unwrap();
|
||||
assert!(matches!(t.auth_type, AuthType::Password));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_encryption_serde() {
|
||||
#[derive(serde::Deserialize, Debug)]
|
||||
struct Test {
|
||||
encryption: Encryption,
|
||||
}
|
||||
let json = r#"{"encryption":"tls"}"#;
|
||||
let t: Test = serde_json::from_str(json).unwrap();
|
||||
assert!(matches!(t.encryption, Encryption::Tls));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_schema_serde() {
|
||||
#[derive(serde::Deserialize, Debug)]
|
||||
struct Test {
|
||||
schema: Schema,
|
||||
}
|
||||
let json = r#"{"schema":"custom"}"#;
|
||||
let t: Test = serde_json::from_str(json).unwrap();
|
||||
assert!(matches!(t.schema, Schema::Custom));
|
||||
}
|
||||
}
|
||||
@@ -1,101 +0,0 @@
|
||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
||||
use crate::ExecutionContext;
|
||||
use minijinja::Environment;
|
||||
use std::collections::HashMap;
|
||||
|
||||
/// Builds template context data from execution context.
|
||||
/// Populates pipeline, build, and commit information for template rendering.
|
||||
pub fn prepare_template_data(ctx: &ExecutionContext) -> HashMap<String, serde_json::Value> {
|
||||
let mut data = HashMap::new();
|
||||
|
||||
data.insert(
|
||||
"pipeline".to_string(),
|
||||
serde_json::json!({
|
||||
"name": ctx.pipeline_name,
|
||||
}),
|
||||
);
|
||||
|
||||
data.insert(
|
||||
"build".to_string(),
|
||||
serde_json::json!({
|
||||
"id": ctx.task_id,
|
||||
"status": "success",
|
||||
"status_color": "green",
|
||||
"duration": "0s",
|
||||
"url": "",
|
||||
}),
|
||||
);
|
||||
|
||||
data.insert(
|
||||
"commit".to_string(),
|
||||
serde_json::json!({
|
||||
"hash": "",
|
||||
"short_hash": "",
|
||||
"message": "",
|
||||
"author": "",
|
||||
"author_email": "",
|
||||
}),
|
||||
);
|
||||
|
||||
data
|
||||
}
|
||||
|
||||
/// Renders a minijinja template string with the provided data map.
|
||||
/// Returns the rendered string on success, or a String error message on failure.
|
||||
pub fn render_template(
|
||||
template: &str,
|
||||
data: &HashMap<String, serde_json::Value>,
|
||||
) -> Result<String, String> {
|
||||
let env = Environment::new();
|
||||
|
||||
let tmpl = env
|
||||
.template_from_str(template)
|
||||
.map_err(|e| format!("Template parse error: {}", e))?;
|
||||
|
||||
let result = tmpl
|
||||
.render(data)
|
||||
.map_err(|e| format!("Template render error: {}", e))?;
|
||||
|
||||
Ok(result)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_render_template_simple() {
|
||||
let template = "Hello {{ name }}!";
|
||||
let mut data = HashMap::new();
|
||||
data.insert(
|
||||
"name".to_string(),
|
||||
serde_json::Value::String("World".to_string()),
|
||||
);
|
||||
let result = render_template(template, &data).unwrap();
|
||||
assert_eq!(result, "Hello World!");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_render_template_with_build_status() {
|
||||
let template = "Build {{ build.status }} for pipeline {{ pipeline.name }}";
|
||||
let mut data = HashMap::new();
|
||||
data.insert(
|
||||
"build".to_string(),
|
||||
serde_json::json!({ "status": "success" }),
|
||||
);
|
||||
data.insert(
|
||||
"pipeline".to_string(),
|
||||
serde_json::json!({ "name": "test-pipeline" }),
|
||||
);
|
||||
let result = render_template(template, &data).unwrap();
|
||||
assert_eq!(result, "Build success for pipeline test-pipeline");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_render_template_invalid_syntax() {
|
||||
let template = "{{ Unterminated";
|
||||
let data = HashMap::new();
|
||||
let result = render_template(template, &data);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,7 @@
|
||||
use crate::finalize::plugin::PluginError;
|
||||
use crate::finalize::plugin::PluginMetadata;
|
||||
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn};
|
||||
use workshop_engine::{EventSender, ExecutionContext};
|
||||
use crate::{ finalize::plugin::AsyncPluginFn};
|
||||
use async_trait::async_trait;
|
||||
|
||||
pub struct Satori;
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
use crate::finalize::plugin::PluginError;
|
||||
use crate::finalize::plugin::PluginMetadata;
|
||||
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn};
|
||||
use workshop_engine::{EventSender, ExecutionContext};
|
||||
use crate::{ finalize::plugin::AsyncPluginFn};
|
||||
use async_trait::async_trait;
|
||||
|
||||
pub struct Webhook;
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
use crate::finalize::NotificationTemplateDef;
|
||||
use crate::notify::types::NotificationRenderer;
|
||||
use serde::Deserialize;
|
||||
use std::path::PathBuf;
|
||||
@@ -18,10 +19,14 @@ pub struct PluginMetadata {
|
||||
#[serde(default)]
|
||||
pub renderer: Option<NotificationRenderer>,
|
||||
|
||||
/// Additional YAML fields not captured by struct.
|
||||
/// Additional YAML fields.
|
||||
#[serde(flatten)]
|
||||
pub value: serde_yaml::Value,
|
||||
|
||||
/// Plugin-defined default templates, auto-registered as <name>.fallback
|
||||
#[serde(default)]
|
||||
pub templates: Option<Vec<NotificationTemplateDef>>,
|
||||
|
||||
/// Filesystem path to plugin definition.
|
||||
#[serde(skip)]
|
||||
pub fspath: PathBuf,
|
||||
@@ -36,6 +41,7 @@ impl Default for PluginMetadata {
|
||||
plugin_type: PluginType::Unknown,
|
||||
value: serde_yaml::Value::Null,
|
||||
renderer: None,
|
||||
templates: None,
|
||||
fspath: PathBuf::new(),
|
||||
}
|
||||
}
|
||||
@@ -50,76 +56,3 @@ pub enum PluginType {
|
||||
Rhai,
|
||||
Dylib,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_plugin_metadata_default() {
|
||||
let meta = PluginMetadata::default();
|
||||
assert_eq!(meta.name, "");
|
||||
assert_eq!(meta.version, "");
|
||||
assert_eq!(meta.entrypoint, "");
|
||||
assert_eq!(meta.plugin_type, PluginType::Unknown);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_plugin_type_default() {
|
||||
let pt = PluginType::default();
|
||||
assert_eq!(pt, PluginType::Unknown);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_plugin_type_equality() {
|
||||
assert_eq!(PluginType::Shell, PluginType::Shell);
|
||||
assert_eq!(PluginType::Dylib, PluginType::Dylib);
|
||||
assert_ne!(PluginType::Shell, PluginType::Dylib);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_plugin_metadata_clone() {
|
||||
let meta = PluginMetadata {
|
||||
name: "test".to_string(),
|
||||
version: "1.0".to_string(),
|
||||
entrypoint: "/bin/test".to_string(),
|
||||
plugin_type: PluginType::Shell,
|
||||
value: serde_yaml::Value::Null,
|
||||
fspath: PathBuf::from("/path/to/plugin"),
|
||||
renderer: None,
|
||||
};
|
||||
let cloned = meta.clone();
|
||||
assert_eq!(cloned.name, meta.name);
|
||||
assert_eq!(cloned.version, meta.version);
|
||||
assert_eq!(cloned.plugin_type, meta.plugin_type);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_plugin_type_serde_unknown() {
|
||||
let yaml = "Unknown";
|
||||
let pt: PluginType = serde_yaml::from_str(yaml).unwrap();
|
||||
assert_eq!(pt, PluginType::Unknown);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_plugin_type_serde_shell() {
|
||||
let yaml = "Shell";
|
||||
let pt: PluginType = serde_yaml::from_str(yaml).unwrap();
|
||||
assert_eq!(pt, PluginType::Shell);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_plugin_metadata_serde() {
|
||||
let yaml = r#"
|
||||
name: test-plugin
|
||||
version: "2.0"
|
||||
entrypoint: ./bin/test
|
||||
plugin_type: Shell
|
||||
"#;
|
||||
let meta: PluginMetadata = serde_yaml::from_str(yaml).unwrap();
|
||||
assert_eq!(meta.name, "test-plugin");
|
||||
assert_eq!(meta.version, "2.0");
|
||||
assert_eq!(meta.entrypoint, "./bin/test");
|
||||
assert_eq!(meta.plugin_type, PluginType::Shell);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
use crate::finalize::plugin::PluginError;
|
||||
use crate::finalize::plugin::PluginMetadata;
|
||||
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn};
|
||||
use workshop_engine::{EventSender, ExecutionContext};
|
||||
use crate::{ finalize::plugin::AsyncPluginFn};
|
||||
use async_trait::async_trait;
|
||||
|
||||
pub struct Rhai;
|
||||
|
||||
@@ -1,8 +1,9 @@
|
||||
use crate::Engine;
|
||||
use workshop_engine::Engine;
|
||||
use crate::finalize::constant::FINALIZE_SHELL_ENVPREFIX;
|
||||
use crate::finalize::plugin::PluginError;
|
||||
use crate::finalize::plugin::PluginMetadata;
|
||||
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn};
|
||||
use workshop_engine::{EventSender, ExecutionContext};
|
||||
use crate::{ finalize::plugin::AsyncPluginFn};
|
||||
use async_trait::async_trait;
|
||||
use std::collections::HashMap;
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
use crate::ExecutionContext;
|
||||
use workshop_engine::ExecutionContext;
|
||||
use crate::finalize::plugin::{PluginMap, call_named_plugin};
|
||||
use std::path::PathBuf;
|
||||
use workshop_engine::{CustomCommand, DeltaExecutionContext, Engine, EventSender, ExecutionError};
|
||||
|
||||
+25
-57
@@ -1,27 +1,16 @@
|
||||
// Library crate for workshop-executor
|
||||
// Selective exports: only modules used by external callers or tests
|
||||
|
||||
pub mod bare;
|
||||
pub mod bake;
|
||||
pub mod daemon;
|
||||
pub mod error;
|
||||
pub mod finalize;
|
||||
pub mod monitor;
|
||||
pub mod notify;
|
||||
pub mod prebake;
|
||||
pub mod socket;
|
||||
pub mod types;
|
||||
pub mod utils;
|
||||
|
||||
// External crates re-exports
|
||||
pub use config;
|
||||
|
||||
// Re-export commonly used types from workshop-engine
|
||||
pub use workshop_engine::{Engine, EventSender, ExecutionContext, ExecutionError, ExecutionResult};
|
||||
|
||||
// CLI types used by main (keep here since they're used in daemon)
|
||||
pub use clap::{Parser, Subcommand};
|
||||
pub use serde::Deserialize;
|
||||
pub use std::collections::HashMap;
|
||||
pub use std::path::PathBuf;
|
||||
|
||||
pub mod cli {
|
||||
pub use super::{Deserialize, HashMap, Parser, PathBuf, Subcommand};
|
||||
pub use clap::Command;
|
||||
@@ -46,64 +35,43 @@ pub mod cli {
|
||||
#[arg(long, default_value = "false", env = "HBW_DRY_RUN")]
|
||||
pub dry_run: bool,
|
||||
|
||||
/// Override prebake config path
|
||||
#[arg(long)]
|
||||
pub prebake: Option<PathBuf>,
|
||||
/// Override bake script path
|
||||
#[arg(long)]
|
||||
pub bake: Option<PathBuf>,
|
||||
/// Override bake_base.sh path
|
||||
#[arg(long)]
|
||||
pub bake_base: Option<PathBuf>,
|
||||
/// Override finalize config path
|
||||
#[arg(long)]
|
||||
pub finalize: Option<PathBuf>,
|
||||
|
||||
/// Debug features (comma-separated). "notify" enables notification debug logging.
|
||||
#[arg(long, env = "HBW_DEBUG")]
|
||||
pub debug: Option<String>,
|
||||
|
||||
#[command(subcommand)]
|
||||
pub command: Option<Commands>,
|
||||
}
|
||||
|
||||
#[derive(Subcommand, Debug)]
|
||||
pub enum Commands {
|
||||
/// Run pipeline stages in standalone (bare) mode
|
||||
/// Run individual pipeline stages
|
||||
Bare {
|
||||
#[command(subcommand)]
|
||||
command: BareCommands,
|
||||
},
|
||||
|
||||
/// Run in client mode
|
||||
Client { config: String },
|
||||
Daemon {},
|
||||
}
|
||||
|
||||
#[derive(Subcommand, Debug)]
|
||||
pub enum BareCommands {
|
||||
/// Prepare a task
|
||||
Prebake { config: PathBuf },
|
||||
|
||||
/// Run a task
|
||||
Bake {
|
||||
/// Path to the script to execute
|
||||
script: PathBuf,
|
||||
/// Path to bake_base.sh
|
||||
#[arg(short, long, default_value = "./bake_base.sh")]
|
||||
bake_base: PathBuf,
|
||||
#[arg(short, long, default_value = "./prebake.yml")]
|
||||
prebake: PathBuf,
|
||||
},
|
||||
|
||||
/// Finish a task
|
||||
Finalize { config: String },
|
||||
Prebake {},
|
||||
Bake {},
|
||||
Finalize {},
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
pub struct TaskSpec {
|
||||
pub subcommand: String,
|
||||
pub arguments: Vec<String>,
|
||||
pub options: HashMap<String, String>,
|
||||
}
|
||||
|
||||
impl TaskSpec {
|
||||
pub fn command(&self, executable: &PathBuf) -> tokio::process::Command {
|
||||
let mut command = tokio::process::Command::new(executable);
|
||||
command.arg(&self.subcommand);
|
||||
for arg in &self.arguments {
|
||||
command.arg(arg);
|
||||
}
|
||||
for (flag, value) in self.options.iter() {
|
||||
if value.is_empty() {
|
||||
command.arg(format!("--{}", flag));
|
||||
} else {
|
||||
command.arg(format!("--{}={}", flag, value));
|
||||
}
|
||||
}
|
||||
command
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+110
-63
@@ -1,13 +1,91 @@
|
||||
use std::path::Path;
|
||||
use workshop_baker::types::resource::ResourceRegistry;
|
||||
use std::path::PathBuf;
|
||||
|
||||
use workshop_baker::{
|
||||
bake::bake,
|
||||
cli::{BareCommands, Cli, Commands, Parser},
|
||||
daemon,
|
||||
finalize::finalize,
|
||||
prebake::prebake,
|
||||
bake::bake, cli::{Cli, Commands, Parser},
|
||||
error::CliError,
|
||||
finalize::finalize, notify::notify, prebake::prebake,
|
||||
};
|
||||
use workshop_engine::{EventSender, ExecutionContext, error::HasExitCode};
|
||||
use workshop_baker::bare::run_bare;
|
||||
use workshop_baker::notify::queue::NotificationQueue;
|
||||
use workshop_engine::{EventSender, ExecutionContext};
|
||||
|
||||
fn or_workshop(path: &Option<PathBuf>, name: &str) -> PathBuf {
|
||||
path.clone().unwrap_or_else(|| PathBuf::from(format!(".workshop/{}", name)))
|
||||
}
|
||||
|
||||
async fn worker_main(cli: Cli) -> Result<(), CliError> {
|
||||
let prebake_path = cli.prebake.clone().unwrap_or_else(|| {
|
||||
log::error!("--prebake is required in worker standalone mode");
|
||||
std::process::exit(1);
|
||||
});
|
||||
let bake_path = cli.bake.clone().unwrap_or_else(|| {
|
||||
log::error!("--bake is required in worker standalone mode");
|
||||
std::process::exit(1);
|
||||
});
|
||||
let finalize_path = cli.finalize.clone().unwrap_or_else(|| {
|
||||
log::error!("--finalize is required in worker standalone mode");
|
||||
std::process::exit(1);
|
||||
});
|
||||
|
||||
log::info!("Worker standalone mode");
|
||||
|
||||
let mut ctx = ExecutionContext {
|
||||
standalone: true,
|
||||
task_id: format!("{}-{}-worker", cli.pipeline, cli.build_id),
|
||||
pipeline_name: cli.pipeline.clone(),
|
||||
username: cli.username.clone(),
|
||||
dry_run: cli.dry_run,
|
||||
privileged: false,
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let (tx, event_rx) = tokio::sync::mpsc::channel(256);
|
||||
let event_tx: EventSender = Some(tx);
|
||||
let _event_handle = tokio::spawn(workshop_baker::prebake::event::event_receiver(
|
||||
event_rx, cli.pipeline.clone(), cli.build_id.clone(),
|
||||
));
|
||||
|
||||
let (to_tx, to_rx) = tokio::sync::mpsc::channel(256);
|
||||
let (retry_tx, retry_rx) = tokio::sync::mpsc::channel(256);
|
||||
|
||||
// Resource registry — populated before stages, consumed by stages
|
||||
let registry = ResourceRegistry::new();
|
||||
// TODO: pre-fetch plugins and templates into registry
|
||||
|
||||
let notify_path = finalize_path.clone();
|
||||
let notify_ctx = ctx.clone();
|
||||
let notify_etx = event_tx.clone();
|
||||
let debug = cli.debug.clone();
|
||||
let notify_registry = registry.clone();
|
||||
let notify_handle = tokio::spawn(async move {
|
||||
notify(¬ify_path, &mut notify_ctx.clone(), notify_etx, to_rx, retry_tx, debug.map(|d| d.contains("notify")).unwrap_or(false), ¬ify_registry).await
|
||||
});
|
||||
|
||||
let nevent_tx = to_tx.clone();
|
||||
let mut queue = NotificationQueue::new(to_tx, retry_rx);
|
||||
|
||||
let pipe_result: anyhow::Result<()> = async {
|
||||
prebake(&prebake_path, &cli, None, &mut ctx, event_tx.clone(), Some(nevent_tx)).await?;
|
||||
let bake_base_path = or_workshop(&cli.bake_base, "bake_base.sh");
|
||||
let pb_path = or_workshop(&cli.prebake, "prebake.yml");
|
||||
bake(&bake_path, &bake_base_path, &pb_path, &cli, &mut ctx, event_tx.clone()).await?;
|
||||
finalize(&finalize_path, &cli, &mut ctx, event_tx.clone(), ®istry).await?;
|
||||
Ok(())
|
||||
}.await;
|
||||
|
||||
if let Err(e) = pipe_result {
|
||||
log::error!("Pipeline stage failed: {}", e);
|
||||
return Err(CliError::General(e));
|
||||
}
|
||||
|
||||
log::info!("Pipeline complete. Draining notifications...");
|
||||
if let Err(e) = queue.drain().await {
|
||||
log::error!("Notification queue error: {}", e);
|
||||
}
|
||||
let _ = notify_handle.await;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() {
|
||||
@@ -16,77 +94,46 @@ async fn main() {
|
||||
if cli.dry_run {
|
||||
log::warn!("Dry run enabled, no changes will be made.");
|
||||
}
|
||||
match &cli.command {
|
||||
Some(Commands::Client { config: _ }) => {
|
||||
unimplemented!("Client mode not yet implemented.")
|
||||
}
|
||||
Some(Commands::Bare { command }) => {
|
||||
log::info!("Running in standalone mode.");
|
||||
|
||||
let (stage, privileged) = match command {
|
||||
BareCommands::Prebake { .. } => ("init", true),
|
||||
BareCommands::Bake { .. } => ("bake", false),
|
||||
BareCommands::Finalize { .. } => ("finalize", false),
|
||||
};
|
||||
let pipeline = cli.pipeline.clone();
|
||||
let build_id = cli.build_id.clone();
|
||||
|
||||
match &cli.command {
|
||||
Some(Commands::Bare { command }) => {
|
||||
let mut ctx = ExecutionContext {
|
||||
standalone: true,
|
||||
task_id: format!("{}-{}-{}", cli.pipeline, cli.build_id, stage),
|
||||
pipeline_name: cli.pipeline.clone(),
|
||||
task_id: format!("{}-{}-bare", pipeline, build_id),
|
||||
pipeline_name: pipeline.clone(),
|
||||
username: cli.username.clone(),
|
||||
dry_run: cli.dry_run,
|
||||
privileged,
|
||||
privileged: false,
|
||||
..Default::default()
|
||||
};
|
||||
let (tx, event_rx) = tokio::sync::mpsc::channel(256);
|
||||
let event_tx: EventSender = Some(tx);
|
||||
let handle = tokio::spawn(workshop_baker::prebake::event::event_receiver(
|
||||
event_rx,
|
||||
cli.pipeline.clone(),
|
||||
cli.build_id.clone(),
|
||||
let event_handle = tokio::spawn(workshop_baker::prebake::event::event_receiver(
|
||||
event_rx, pipeline.clone(), build_id.clone(),
|
||||
));
|
||||
|
||||
let result: anyhow::Result<()> = match command {
|
||||
BareCommands::Prebake { config } => {
|
||||
prebake(config, &cli, None, &mut ctx, event_tx).await
|
||||
}
|
||||
BareCommands::Bake {
|
||||
script,
|
||||
bake_base,
|
||||
prebake: prebake_path,
|
||||
} => bake(script, bake_base, prebake_path, &cli, &mut ctx, event_tx)
|
||||
.await
|
||||
.map_err(|e| e.into()),
|
||||
BareCommands::Finalize { config } => {
|
||||
finalize(Path::new(config), &cli, &mut ctx, event_tx)
|
||||
.await
|
||||
.map_err(|e| e.into())
|
||||
}
|
||||
};
|
||||
let registry = ResourceRegistry::new();
|
||||
let result = run_bare(&cli, command, &mut ctx, event_tx, &pipeline, &build_id, ®istry).await;
|
||||
|
||||
drop(ctx);
|
||||
let _ = handle.await;
|
||||
dbg!(&result);
|
||||
if let Err(ref e) = result {
|
||||
let code = match command {
|
||||
BareCommands::Prebake { .. } => e
|
||||
.downcast_ref::<workshop_baker::prebake::error::PrebakeError>()
|
||||
.map(|pe| pe.exit_code())
|
||||
.unwrap_or(1),
|
||||
BareCommands::Bake { .. } => e
|
||||
.downcast_ref::<workshop_baker::bake::error::BakeError>()
|
||||
.map(|be| be.exit_code())
|
||||
.unwrap_or(1),
|
||||
BareCommands::Finalize { .. } => e
|
||||
.downcast_ref::<workshop_baker::finalize::FinalizeError>()
|
||||
.map(|fe| fe.exit_code())
|
||||
.unwrap_or(1),
|
||||
};
|
||||
log::error!("{} stage failed: {}", stage, e);
|
||||
std::process::exit(code);
|
||||
let _ = event_handle.await;
|
||||
|
||||
if let Err(e) = result {
|
||||
log::error!("{}", e);
|
||||
std::process::exit(e.exit_code());
|
||||
}
|
||||
}
|
||||
Some(Commands::Daemon {}) => {
|
||||
unimplemented!("Daemon mode not implemented");
|
||||
}
|
||||
None => {
|
||||
log::info!("Executor will be running in daemon mode.");
|
||||
let _result = daemon::daemon(&cli).await;
|
||||
if let Err(e) = worker_main(cli).await {
|
||||
log::error!("{}", e);
|
||||
std::process::exit(e.exit_code());
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,89 +0,0 @@
|
||||
mod cgroups;
|
||||
mod jobobject;
|
||||
use crate::socket::{establish_connection, get_socket_addr};
|
||||
use config::Config;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::time::Instant;
|
||||
use tokio::io::AsyncWriteExt;
|
||||
|
||||
#[derive(Debug, Serialize, Deserialize)]
|
||||
pub struct ResourceUsageInfo {
|
||||
pub msgtype: String,
|
||||
pub name: String,
|
||||
pub os_type: OsType,
|
||||
pub statistics: UsageStats,
|
||||
pub timestamp: u64,
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize, Deserialize)]
|
||||
pub enum OsType {
|
||||
Linux,
|
||||
Windows,
|
||||
MacOS,
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize, Deserialize, Clone)]
|
||||
pub struct UsageStats {
|
||||
pub cpu_usage_percent: f64,
|
||||
pub memory_usage_bytes: u64,
|
||||
pub memory_limit_bytes: i64,
|
||||
// pub memory_usage_percent: f64,
|
||||
pub process_count: u64,
|
||||
pub cpu_usage_us: u64,
|
||||
/// Internal timestamp, using std::Instant
|
||||
#[serde(skip, default = "default_internal_timestamp")]
|
||||
_timestamp: Instant,
|
||||
}
|
||||
|
||||
fn default_internal_timestamp() -> Instant {
|
||||
Instant::now()
|
||||
}
|
||||
|
||||
pub async fn monitor(
|
||||
name: &String,
|
||||
interval_ms: Option<u64>,
|
||||
settings: Config,
|
||||
) -> anyhow::Result<()> {
|
||||
// let socket_addr = get_wsagent_socket_addr()?;
|
||||
let socket_addr = get_socket_addr(settings.get_string("socket")?)?;
|
||||
let interval = tokio::time::Duration::from_millis(interval_ms.unwrap_or(1000));
|
||||
let mut socket = establish_connection(&socket_addr).await?;
|
||||
let mut prev_usage = UsageStats {
|
||||
cpu_usage_percent: 0.0,
|
||||
memory_usage_bytes: 0,
|
||||
memory_limit_bytes: 0,
|
||||
process_count: 0,
|
||||
cpu_usage_us: 0,
|
||||
_timestamp: default_internal_timestamp(),
|
||||
};
|
||||
loop {
|
||||
let usage: UsageStats = cgroups::get_usage(name, prev_usage)?; // TODO: 多系统支持?
|
||||
prev_usage = usage.clone();
|
||||
// dbg!(&usage);
|
||||
let usage_info = ResourceUsageInfo {
|
||||
msgtype: "ResourceUsage".to_string(),
|
||||
name: name.clone(),
|
||||
os_type: OsType::Linux, // TODO: 多系统支持?
|
||||
statistics: usage,
|
||||
timestamp: std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)?
|
||||
.as_millis() as u64,
|
||||
};
|
||||
|
||||
let usage_json = serde_json::to_string(&usage_info)?;
|
||||
|
||||
send_usage_info(&mut socket, &usage_json).await?;
|
||||
// TODO: 增加消息接收,提供(伪)双向通信功能
|
||||
|
||||
tokio::time::sleep(interval).await;
|
||||
}
|
||||
}
|
||||
|
||||
async fn send_usage_info<S>(stream: &mut S, usage_json: &str) -> anyhow::Result<()>
|
||||
where
|
||||
S: tokio::io::AsyncWrite + Unpin,
|
||||
{
|
||||
stream.write_all(usage_json.as_bytes()).await?;
|
||||
stream.write_all(b"\n").await?;
|
||||
Ok(())
|
||||
}
|
||||
+429
-361
@@ -1,400 +1,468 @@
|
||||
use crate::finalize::FinalizeConfig;
|
||||
use crate::finalize::NotificationConfig;
|
||||
use crate::finalize::NotificationTemplate;
|
||||
use crate::finalize::config::TriggerItemRaw;
|
||||
use crate::finalize::constant::FINALIZE_STANDALONE_PLUGIN_PATH;
|
||||
use crate::finalize::parse;
|
||||
use crate::finalize::plugin;
|
||||
use crate::finalize::plugin::fetch;
|
||||
use crate::notify::types::Matchable;
|
||||
use crate::notify::types::Method;
|
||||
use crate::notify::types::NOTIFICATION_DEFAULT_FALLIBLE;
|
||||
use crate::notify::types::NOTIFICATION_DEFAULT_MAX_LIVES;
|
||||
use crate::notify::types::NOTIFICATION_DEFAULT_RETRY_BACKOFF;
|
||||
use crate::notify::types::NOTIFICATION_DEFAULT_TIMEOUT_MS;
|
||||
use crate::notify::types::NOTIFICATION_NOTREADY_DELAY;
|
||||
use crate::notify::types::NOTIFICATION_TEMPORARY_DELAY_FACTOR;
|
||||
use crate::notify::types::NOTIFICATION_TEMPORARY_DELAY_TIME;
|
||||
use crate::notify::types::NOTIFICATION_TEMPORARY_MAX_TIME;
|
||||
use crate::notify::types::NotificationEvent;
|
||||
use crate::notify::types::DEFAULT_NOTIFY_PARAMS;
|
||||
use crate::notify::types::NotificationEventReceiver;
|
||||
use crate::notify::types::NotificationEventSender;
|
||||
use crate::notify::types::NotificationPluginMap;
|
||||
use crate::notify::types::NotificationRenderer;
|
||||
use crate::notify::types::NotificationRule;
|
||||
use crate::notify::types::NotificationRulesMap;
|
||||
use crate::notify::types::ParsedTrigger;
|
||||
use crate::{EventSender, ExecutionContext};
|
||||
use chrono::Utc;
|
||||
use globset::{Glob, GlobSetBuilder};
|
||||
use std::collections::HashMap;
|
||||
use std::collections::HashSet;
|
||||
use crate::notify::handler::NotificationHandler;
|
||||
use crate::types::resource::ResourceRegistry;
|
||||
use std::path::Path;
|
||||
use std::path::PathBuf;
|
||||
use std::sync::Arc;
|
||||
use workshop_engine::{EventSender, ExecutionContext};
|
||||
|
||||
mod error;
|
||||
pub mod handler;
|
||||
pub mod queue;
|
||||
pub mod rule;
|
||||
pub mod template;
|
||||
pub mod types;
|
||||
use error::NotifyError;
|
||||
|
||||
fn validate(finalize: &FinalizeConfig) -> anyhow::Result<()> {
|
||||
let validate_result = finalize.validate();
|
||||
if let Err(errors) = validate_result {
|
||||
for error in &errors {
|
||||
log::error!("Error: {}", error);
|
||||
}
|
||||
anyhow::bail!(NotifyError::FinalizeValidationError(errors))
|
||||
};
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn extract_plugin(finalize: &FinalizeConfig) -> HashSet<String> {
|
||||
finalize
|
||||
.notification
|
||||
.groups
|
||||
.values()
|
||||
.flat_map(|methods| methods.keys())
|
||||
.cloned()
|
||||
.collect()
|
||||
}
|
||||
pub mod util;
|
||||
use rule::parse_rules;
|
||||
use template::{flatten_template_refs, resolve_external_template_refs};
|
||||
use util::{extract_plugin, validate};
|
||||
|
||||
pub async fn notify(
|
||||
finalize_path: &Path,
|
||||
ctx: &mut ExecutionContext,
|
||||
_event_tx: EventSender,
|
||||
notifyevent_rx: &mut NotificationEventReceiver,
|
||||
notifyevent_tx: NotificationEventSender,
|
||||
mut rx: NotificationEventReceiver,
|
||||
retry_tx: NotificationEventSender,
|
||||
_debug: bool,
|
||||
registry: &ResourceRegistry,
|
||||
) -> anyhow::Result<()> {
|
||||
let mut finalize = parse(finalize_path)?;
|
||||
|
||||
// Always inject in-site-notify as the lowest priority fallback
|
||||
{
|
||||
let max_p = finalize.notification.groups.keys().max().copied().unwrap_or(0);
|
||||
let mut m = std::collections::HashMap::new();
|
||||
m.insert(
|
||||
"in-site-notify".to_string(),
|
||||
crate::finalize::NotificationMethod::default(),
|
||||
);
|
||||
finalize.notification.groups.insert(max_p + 1, m);
|
||||
}
|
||||
|
||||
validate(&finalize)?;
|
||||
let plugin_list = extract_plugin(&finalize);
|
||||
|
||||
if plugin_list.is_empty() {
|
||||
log::warn!("No notification method provided, skipping notification");
|
||||
// TODO: send event to event_tx
|
||||
return Ok(());
|
||||
}
|
||||
let notify_plugins = plugin::register(
|
||||
finalize.plugin,
|
||||
Some(&plugin_list),
|
||||
&mut finalize.notification,
|
||||
registry,
|
||||
)?;
|
||||
|
||||
if ctx.standalone {
|
||||
fetch::fetch_plugins(
|
||||
&PathBuf::from(FINALIZE_STANDALONE_PLUGIN_PATH),
|
||||
&mut finalize.plugin,
|
||||
Some(&plugin_list),
|
||||
)
|
||||
.await?;
|
||||
}
|
||||
let notify_plugins = plugin::register(finalize.plugin, Some(&plugin_list))?;
|
||||
resolve_external_template_refs(&mut finalize.notification.templates, registry);
|
||||
flatten_template_refs(&mut finalize.notification.templates)?;
|
||||
|
||||
let notification_rules = parse_rules(&finalize.notification);
|
||||
|
||||
loop {
|
||||
tokio::select! {
|
||||
Some(event) = notifyevent_rx.recv() => {
|
||||
handle_notification_event(
|
||||
event,
|
||||
¬ify_plugins,
|
||||
¬ification_rules,
|
||||
¬ifyevent_tx,
|
||||
)
|
||||
.await?;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
let handler = Arc::new(NotificationHandler::new(
|
||||
Arc::new(notification_rules),
|
||||
Arc::new(notify_plugins),
|
||||
Arc::new(finalize.notification.templates),
|
||||
Arc::new(ctx.clone()),
|
||||
retry_tx,
|
||||
DEFAULT_NOTIFY_PARAMS.batch_period.value,
|
||||
DEFAULT_NOTIFY_PARAMS.batch_watermark.value,
|
||||
));
|
||||
|
||||
/// Process a single notification event.
|
||||
///
|
||||
/// Delegates to [`notification_handler`] and handles errors:
|
||||
/// - [`NotifyError::PluginNotAvailable`] — reschedule with static delay, lower priority
|
||||
/// - [`NotifyError::TemporaryError`] — consume one life, reschedule with exponential backoff
|
||||
/// - [`NotifyError::PermanentError`] — drop the event
|
||||
/// - other — log and drop
|
||||
async fn handle_notification_event(
|
||||
event: NotificationEvent,
|
||||
plugins: &NotificationPluginMap,
|
||||
rules: &NotificationRulesMap,
|
||||
notifyevent_tx: &NotificationEventSender,
|
||||
) -> anyhow::Result<()> {
|
||||
match notification_handler(plugins, rules, &event) {
|
||||
Ok(_) => {}
|
||||
Err(e) => {
|
||||
log::debug!("Failed to send notification: {:?}", e);
|
||||
match e {
|
||||
NotifyError::PluginNotAvailable(_) => {
|
||||
let mut event = event;
|
||||
event.not_before = Utc::now() + NOTIFICATION_NOTREADY_DELAY;
|
||||
event.effective_priority = event.effective_priority.saturating_add(1);
|
||||
notifyevent_tx.send(event).await?;
|
||||
}
|
||||
NotifyError::TemporaryError(_) => {
|
||||
let mut event = event;
|
||||
event.lives -= 1;
|
||||
event.not_before = Utc::now() + notify_delay(&event);
|
||||
notifyevent_tx.send(event).await?;
|
||||
}
|
||||
NotifyError::PermanentError(_) => {
|
||||
log::error!("None of the configured notification plugins works.");
|
||||
log::error!("Message will be dropped.");
|
||||
}
|
||||
error => {
|
||||
log::error!(
|
||||
"Internal error: notify_handler returned unexpected error: {:?}",
|
||||
error
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
loop {
|
||||
let Some(event) = rx.recv().await else { break };
|
||||
let h = handler.clone();
|
||||
tokio::spawn(async move {
|
||||
h.process_event(event).await;
|
||||
});
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn notification_handler(
|
||||
plugins: &NotificationPluginMap,
|
||||
config: &NotificationRulesMap,
|
||||
event: &NotificationEvent,
|
||||
) -> Result<(), NotifyError> {
|
||||
for (_priority, ruleset) in config.iter() {
|
||||
for (method, rules) in ruleset.iter() {
|
||||
if !event.target.contains(method) {
|
||||
log::trace!("target does not contain method name: {}", method);
|
||||
continue;
|
||||
}
|
||||
for (rule_id, rule) in rules.iter().enumerate() {
|
||||
if !rule.enabled {
|
||||
log::trace!("rule is not enabled: {}", rule_id);
|
||||
continue;
|
||||
}
|
||||
if rule.accept(&event.outcome, &event.stage, &event.substage) {
|
||||
let plugin = plugins
|
||||
.get(method)
|
||||
.ok_or(NotifyError::UnknownPlugin(method.clone()))?;
|
||||
let renderer = plugin.metadata.renderer.clone().unwrap_or_default();
|
||||
match renderer {
|
||||
NotificationRenderer::Server => {
|
||||
// ...
|
||||
todo!();
|
||||
}
|
||||
NotificationRenderer::Plugin => {
|
||||
unimplemented!();
|
||||
return Err(NotifyError::UnsupportedRenderer(method.clone(), renderer));
|
||||
}
|
||||
NotificationRenderer::Interactive => {
|
||||
unimplemented!();
|
||||
return Err(NotifyError::UnsupportedRenderer(method.clone(), renderer));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// TODO: PIPELINE_NOTIFICATION_EXHAUSTED — set when any method returns Exhausted
|
||||
// TODO: PIPELINE_NOTIFICATION_FAILED — set when any method returns PermanentError
|
||||
// Markers sent via event_tx as ExecutionEvent variants for future WebUI/CLI display.
|
||||
|
||||
todo!();
|
||||
}
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::finalize::NotificationTemplateDef;
|
||||
use crate::notify::types::NotificationRule;
|
||||
use crate::notify::types::NotificationTarget;
|
||||
use std::collections::HashMap;
|
||||
use crate::notify::types::NotificationEvent;
|
||||
use crate::notify::types::NotificationRulesMap;
|
||||
use crate::notify::error::NotifyError;
|
||||
use crate::notify::template::resolve_template;
|
||||
use crate::notify::util::{find_rule, notify_delay, wait_time};
|
||||
|
||||
fn parse_trigger_item_raw(item: &TriggerItemRaw) -> Vec<ParsedTrigger> {
|
||||
match item {
|
||||
TriggerItemRaw::Simple(s) => match s.as_str() {
|
||||
"on_success" => vec![ParsedTrigger::OnSuccess],
|
||||
"on_failure" => vec![ParsedTrigger::OnFailure],
|
||||
s if s.starts_with("on_finish_of:") => {
|
||||
let pattern = s.trim_start_matches("on_finish_of:").trim();
|
||||
let mut builder = GlobSetBuilder::new();
|
||||
builder.add(Glob::new(pattern).expect("Invalid glob pattern"));
|
||||
let glob_set = builder.build().expect("Failed to compile glob set");
|
||||
vec![ParsedTrigger::OnFinishOf(Matchable::Glob(glob_set))]
|
||||
}
|
||||
_ => {
|
||||
log::warn!("Unknown trigger string: {}", s);
|
||||
vec![]
|
||||
}
|
||||
},
|
||||
TriggerItemRaw::Map(map) => {
|
||||
let mut triggers = Vec::new();
|
||||
for (key, patterns) in map {
|
||||
match key.as_str() {
|
||||
"on_finish_of" => {
|
||||
for pattern in patterns {
|
||||
let mut builder = GlobSetBuilder::new();
|
||||
builder.add(Glob::new(pattern).expect("Invalid glob pattern"));
|
||||
let glob_set = builder.build().expect("Failed to compile glob set");
|
||||
triggers.push(ParsedTrigger::OnFinishOf(Matchable::Glob(glob_set)));
|
||||
}
|
||||
}
|
||||
"on_success" => triggers.push(ParsedTrigger::OnSuccess),
|
||||
"on_failure" => triggers.push(ParsedTrigger::OnFailure),
|
||||
_ => {
|
||||
log::warn!("Unknown trigger key: {}", key);
|
||||
}
|
||||
}
|
||||
}
|
||||
triggers
|
||||
}
|
||||
}
|
||||
}
|
||||
// ── helper ──
|
||||
|
||||
fn merge_templates(
|
||||
method_template: &NotificationTemplate,
|
||||
policy_template: &Option<serde_yaml::Value>,
|
||||
) -> NotificationTemplate {
|
||||
let mut merged = method_template.clone();
|
||||
if let Some(policy_tmpl) = policy_template
|
||||
&& let Ok(policy_template) =
|
||||
serde_yaml::from_value::<NotificationTemplate>(policy_tmpl.clone())
|
||||
{
|
||||
if policy_template.schema.is_some() {
|
||||
merged.schema = policy_template.schema;
|
||||
}
|
||||
if policy_template.on_success.is_some() {
|
||||
merged.on_success = policy_template.on_success;
|
||||
}
|
||||
if policy_template.on_failure.is_some() {
|
||||
merged.on_failure = policy_template.on_failure;
|
||||
}
|
||||
if policy_template.on_finish_of.is_some() {
|
||||
merged.on_finish_of = policy_template.on_finish_of;
|
||||
}
|
||||
}
|
||||
merged
|
||||
}
|
||||
|
||||
fn extract_overrides(overrides: &serde_yaml::Value) -> (bool, u32, std::time::Duration, f64) {
|
||||
let fallible = overrides
|
||||
.get("fallible")
|
||||
.and_then(|v| v.as_bool())
|
||||
.unwrap_or(NOTIFICATION_DEFAULT_FALLIBLE);
|
||||
let max_lives = overrides
|
||||
.get("max_lives")
|
||||
.and_then(|v| v.as_u64())
|
||||
.unwrap_or(NOTIFICATION_DEFAULT_MAX_LIVES as u64) as u32;
|
||||
let timeout_ms = overrides
|
||||
.get("timeout_ms")
|
||||
.and_then(|v| v.as_u64())
|
||||
.unwrap_or(NOTIFICATION_DEFAULT_TIMEOUT_MS);
|
||||
let retry_backoff = overrides
|
||||
.get("retry_backoff")
|
||||
.and_then(|v| v.as_f64())
|
||||
.unwrap_or(NOTIFICATION_DEFAULT_RETRY_BACKOFF);
|
||||
(
|
||||
fallible,
|
||||
max_lives,
|
||||
std::time::Duration::from_millis(timeout_ms),
|
||||
retry_backoff,
|
||||
)
|
||||
}
|
||||
|
||||
fn parse_rules(config: &NotificationConfig) -> NotificationRulesMap {
|
||||
let mut rules = NotificationRulesMap::default();
|
||||
for (priority, group) in &config.groups {
|
||||
let mut ruleset: HashMap<Method, Vec<NotificationRule>> = HashMap::new();
|
||||
for (method_name, method) in group {
|
||||
let mut method_rules: Vec<NotificationRule> = Vec::new();
|
||||
|
||||
if method.policy.is_empty() {
|
||||
let trigger: Vec<ParsedTrigger> = method
|
||||
.trigger
|
||||
.iter()
|
||||
.flat_map(parse_trigger_item_raw)
|
||||
.collect();
|
||||
let template = method.template.clone();
|
||||
let (fallible, max_lives, timeout, retry_backoff) =
|
||||
extract_overrides(&method.config);
|
||||
method_rules.push(NotificationRule {
|
||||
enabled: true,
|
||||
trigger,
|
||||
template,
|
||||
fallible,
|
||||
max_lives,
|
||||
timeout,
|
||||
config: method.config.clone(),
|
||||
retry_backoff,
|
||||
});
|
||||
fn make_tmpl(on_success: &str, on_failure: &str) -> NotificationTemplateDef {
|
||||
NotificationTemplateDef {
|
||||
use_: None,
|
||||
on_success: if on_success.is_empty() {
|
||||
None
|
||||
} else {
|
||||
for policy in &method.policy {
|
||||
let trigger: Vec<ParsedTrigger> = if policy.trigger.is_empty() {
|
||||
method
|
||||
.trigger
|
||||
.iter()
|
||||
.flat_map(parse_trigger_item_raw)
|
||||
.collect()
|
||||
} else {
|
||||
policy
|
||||
.trigger
|
||||
.iter()
|
||||
.flat_map(parse_trigger_item_raw)
|
||||
.collect()
|
||||
};
|
||||
let template =
|
||||
merge_templates(&method.template, &Some(policy.template.clone()));
|
||||
let method_overrides = extract_overrides(&method.config);
|
||||
let policy_overrides = extract_overrides(&policy.overrides);
|
||||
let fallible = if policy.overrides.get("fallible").is_some() {
|
||||
policy_overrides.0
|
||||
} else {
|
||||
method_overrides.0
|
||||
};
|
||||
let max_lives = if policy.overrides.get("max_lives").is_some() {
|
||||
policy_overrides.1
|
||||
} else {
|
||||
method_overrides.1
|
||||
};
|
||||
let timeout = if policy.overrides.get("timeout_ms").is_some() {
|
||||
policy_overrides.2
|
||||
} else {
|
||||
method_overrides.2
|
||||
};
|
||||
let retry_backoff = if policy.overrides.get("retry_backoff").is_some() {
|
||||
policy_overrides.3
|
||||
} else {
|
||||
method_overrides.3
|
||||
};
|
||||
method_rules.push(NotificationRule {
|
||||
enabled: true,
|
||||
trigger,
|
||||
template,
|
||||
fallible,
|
||||
max_lives,
|
||||
timeout,
|
||||
config: method.config.clone(),
|
||||
retry_backoff,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
ruleset.insert(method_name.clone(), method_rules);
|
||||
Some(on_success.into())
|
||||
},
|
||||
on_failure: if on_failure.is_empty() {
|
||||
None
|
||||
} else {
|
||||
Some(on_failure.into())
|
||||
},
|
||||
on_finish_of: None,
|
||||
}
|
||||
rules.insert(*priority, ruleset);
|
||||
}
|
||||
rules
|
||||
}
|
||||
|
||||
// calculate min(1, k/x)*t as notification grouping period
|
||||
// to avoid excessive notification frequency
|
||||
fn wait_time(group_period: u32, group_watermark: u32, count: u32) -> f32 {
|
||||
if count == 0 {
|
||||
log::warn!("Unexpected notification count: {}", count);
|
||||
return group_period as f32;
|
||||
fn make_tmpl_with_use(use_ref: &str) -> NotificationTemplateDef {
|
||||
NotificationTemplateDef {
|
||||
use_: Some(use_ref.into()),
|
||||
on_success: None,
|
||||
on_failure: None,
|
||||
on_finish_of: None,
|
||||
}
|
||||
}
|
||||
f32::min(1.0, group_watermark as f32 / count as f32) * group_period as f32
|
||||
}
|
||||
|
||||
// calculate p^(m-l)*t as notification delay
|
||||
fn notify_delay(event: &NotificationEvent) -> std::time::Duration {
|
||||
let exponent = (event.max_lives - event.lives) as i32;
|
||||
if exponent < 0 {
|
||||
log::warn!(
|
||||
"Unexpected lives in notification: lives({}) > max_lives({})",
|
||||
event.lives,
|
||||
event.max_lives
|
||||
);
|
||||
return std::time::Duration::from_secs_f64(NOTIFICATION_TEMPORARY_MAX_TIME);
|
||||
// ═══════════════════════════════════════════
|
||||
// resolve_template_content
|
||||
// ═══════════════════════════════════════════
|
||||
|
||||
#[test]
|
||||
fn resolve_none_cd_default_found() {
|
||||
let mut ws = HashMap::new();
|
||||
ws.insert("mail".into(), make_tmpl("OK", "FAIL"));
|
||||
let templates = HashMap::new();
|
||||
let result = resolve_template(&None, "mail", &templates, false, Some(&ws));
|
||||
assert!(result.is_ok());
|
||||
assert_eq!(result.unwrap().on_success.unwrap(), "OK");
|
||||
}
|
||||
let multiplier = NOTIFICATION_TEMPORARY_DELAY_FACTOR.powi(exponent);
|
||||
let delay = multiplier * NOTIFICATION_TEMPORARY_DELAY_TIME;
|
||||
if !delay.is_finite() || delay < 0.0 {
|
||||
log::warn!(
|
||||
"Unexpected calculated delay: {}, treating as infinite",
|
||||
delay
|
||||
);
|
||||
return std::time::Duration::from_secs_f64(NOTIFICATION_TEMPORARY_MAX_TIME);
|
||||
|
||||
#[test]
|
||||
fn resolve_none_cd_default_missing_fallback() {
|
||||
let ws: HashMap<String, NotificationTemplateDef> = HashMap::new();
|
||||
let mut templates = HashMap::new();
|
||||
templates.insert("mail.fallback".into(), make_tmpl("FALLBACK", ""));
|
||||
let result = resolve_template(&None, "mail", &templates, false, Some(&ws));
|
||||
assert!(result.is_ok());
|
||||
assert_eq!(result.unwrap().on_success.unwrap(), "FALLBACK");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_none_cd_no_default_no_fallback_error() {
|
||||
let ws: HashMap<String, NotificationTemplateDef> = HashMap::new();
|
||||
let templates = HashMap::new();
|
||||
let result = resolve_template(&None, "mail", &templates, false, Some(&ws));
|
||||
assert!(result.is_err());
|
||||
assert!(format!("{}", result.unwrap_err()).contains("failed"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_none_standalone_fallback() {
|
||||
let mut templates = HashMap::new();
|
||||
templates.insert("mail.fallback".into(), make_tmpl("STANDALONE", ""));
|
||||
let result = resolve_template(&None, "mail", &templates, true, None);
|
||||
assert!(result.is_ok());
|
||||
assert_eq!(result.unwrap().on_success.unwrap(), "STANDALONE");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_none_standalone_no_fallback_error() {
|
||||
let result = resolve_template(&None, "mail", &HashMap::new(), true, None);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_custom() {
|
||||
let result = resolve_template(&Some("custom".into()), "mail", &HashMap::new(), false, None);
|
||||
assert!(result.is_ok());
|
||||
// "custom" returns an empty default — caller reads rule.template.on_* directly
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_default_standalone_fallback() {
|
||||
let mut templates = HashMap::new();
|
||||
templates.insert("mail.fallback".into(), make_tmpl("FB", ""));
|
||||
let result = resolve_template(&Some("default".into()), "mail", &templates, true, None);
|
||||
assert!(result.is_ok());
|
||||
assert_eq!(result.unwrap().on_success.unwrap(), "FB");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_default_standalone_no_fallback_error() {
|
||||
let result = resolve_template(&Some("default".into()), "mail", &HashMap::new(), true, None);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_default_cd_wsbase() {
|
||||
let mut ws = HashMap::new();
|
||||
ws.insert("mail".into(), make_tmpl("WS_OK", ""));
|
||||
let result = resolve_template(
|
||||
&Some("default".into()),
|
||||
"mail",
|
||||
&HashMap::new(),
|
||||
false,
|
||||
Some(&ws),
|
||||
);
|
||||
assert!(result.is_ok());
|
||||
assert_eq!(result.unwrap().on_success.unwrap(), "WS_OK");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_default_cd_no_wsbase_error() {
|
||||
let result = resolve_template(
|
||||
&Some("default".into()),
|
||||
"mail",
|
||||
&HashMap::new(),
|
||||
false,
|
||||
None,
|
||||
);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_named_found() {
|
||||
let mut templates = HashMap::new();
|
||||
templates.insert("sms-tmpl1".into(), make_tmpl("NAMED", ""));
|
||||
let result = resolve_template(&Some("sms-tmpl1".into()), "mail", &templates, false, None);
|
||||
assert!(result.is_ok());
|
||||
assert_eq!(result.unwrap().on_success.unwrap(), "NAMED");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_named_not_found_method_match_fallback() {
|
||||
let mut templates = HashMap::new();
|
||||
templates.insert("mail.fallback".into(), make_tmpl("FB", ""));
|
||||
let result = resolve_template(&Some("mail".into()), "mail", &templates, false, None);
|
||||
assert!(result.is_ok());
|
||||
assert_eq!(result.unwrap().on_success.unwrap(), "FB");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_named_not_found_no_match_error() {
|
||||
let result = resolve_template(&Some("ghost".into()), "mail", &HashMap::new(), false, None);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
|
||||
// ═══════════════════════════════════════════
|
||||
// flatten_template_refs
|
||||
// ═══════════════════════════════════════════
|
||||
|
||||
#[test]
|
||||
fn flatten_no_refs_unchanged() {
|
||||
let mut templates = HashMap::new();
|
||||
templates.insert("a".into(), make_tmpl("A_OK", "A_FAIL"));
|
||||
let result = flatten_template_refs(&mut templates);
|
||||
assert!(result.is_ok());
|
||||
assert_eq!(templates["a"].on_success.as_deref(), Some("A_OK"));
|
||||
assert!(templates["a"].use_.is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn flatten_single_ref_merged() {
|
||||
let mut templates = HashMap::new();
|
||||
templates.insert("base".into(), make_tmpl("BASE_OK", "BASE_FAIL"));
|
||||
templates.insert("child".into(), make_tmpl_with_use("base"));
|
||||
let result = flatten_template_refs(&mut templates);
|
||||
assert!(result.is_ok());
|
||||
assert_eq!(templates["child"].on_success.as_deref(), Some("BASE_OK"));
|
||||
assert_eq!(templates["child"].on_failure.as_deref(), Some("BASE_FAIL"));
|
||||
assert!(templates["child"].use_.is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn flatten_child_overrides_parent() {
|
||||
let mut templates = HashMap::new();
|
||||
templates.insert("base".into(), make_tmpl("BASE_OK", "BASE_FAIL"));
|
||||
templates.insert(
|
||||
"child".into(),
|
||||
NotificationTemplateDef {
|
||||
use_: Some("base".into()),
|
||||
on_success: Some("MY_OK".into()),
|
||||
on_failure: None,
|
||||
on_finish_of: None,
|
||||
},
|
||||
);
|
||||
let result = flatten_template_refs(&mut templates);
|
||||
assert!(result.is_ok());
|
||||
assert_eq!(templates["child"].on_success.as_deref(), Some("MY_OK"));
|
||||
assert_eq!(templates["child"].on_failure.as_deref(), Some("BASE_FAIL"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn flatten_chain_a_b_c() {
|
||||
let mut templates = HashMap::new();
|
||||
templates.insert("c".into(), make_tmpl("C_OK", "C_FAIL"));
|
||||
templates.insert("b".into(), make_tmpl_with_use("c"));
|
||||
templates.insert("a".into(), make_tmpl_with_use("b"));
|
||||
let result = flatten_template_refs(&mut templates);
|
||||
assert!(result.is_ok());
|
||||
assert_eq!(templates["a"].on_success.as_deref(), Some("C_OK"));
|
||||
assert_eq!(templates["b"].on_success.as_deref(), Some("C_OK"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn flatten_cycle_detected() {
|
||||
let mut templates = HashMap::new();
|
||||
templates.insert("x".into(), make_tmpl_with_use("y"));
|
||||
templates.insert("y".into(), make_tmpl_with_use("x"));
|
||||
let result = flatten_template_refs(&mut templates);
|
||||
assert!(result.is_err());
|
||||
match result.unwrap_err() {
|
||||
NotifyError::TemplateLoop(_) => {}
|
||||
other => panic!("expected TemplateLoop, got {:?}", other),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn flatten_self_ref_detected() {
|
||||
let mut templates = HashMap::new();
|
||||
templates.insert("self".into(), make_tmpl_with_use("self"));
|
||||
let result = flatten_template_refs(&mut templates);
|
||||
assert!(result.is_err());
|
||||
match result.unwrap_err() {
|
||||
NotifyError::TemplateLoop(_) => {}
|
||||
other => panic!("expected TemplateLoop, got {:?}", other),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn flatten_nonexistent_ref_error() {
|
||||
let mut templates = HashMap::new();
|
||||
templates.insert("orphan".into(), make_tmpl_with_use("ghost"));
|
||||
let result = flatten_template_refs(&mut templates);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn flatten_external_url_preserved() {
|
||||
let mut templates = HashMap::new();
|
||||
templates.insert(
|
||||
"ext".into(),
|
||||
NotificationTemplateDef {
|
||||
use_: Some("git://example.com/repo@v1".into()),
|
||||
on_success: None,
|
||||
on_failure: None,
|
||||
on_finish_of: None,
|
||||
},
|
||||
);
|
||||
let result = flatten_template_refs(&mut templates);
|
||||
assert!(result.is_ok());
|
||||
// External URLs are NOT flattened — use_ preserved
|
||||
assert_eq!(
|
||||
templates["ext"].use_.as_deref(),
|
||||
Some("git://example.com/repo@v1")
|
||||
);
|
||||
}
|
||||
|
||||
// ═══════════════════════════════════════════
|
||||
// wait_time
|
||||
// ═══════════════════════════════════════════
|
||||
|
||||
#[test]
|
||||
fn wait_time_zero_count() {
|
||||
let w = wait_time(30, 5, 0);
|
||||
assert_eq!(w, 30.0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn wait_time_at_watermark() {
|
||||
let w = wait_time(30, 5, 5);
|
||||
assert_eq!(w, 30.0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn wait_time_below_watermark() {
|
||||
let w = wait_time(30, 5, 2);
|
||||
assert!((w - 30.0).abs() < 0.01);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn wait_time_above_watermark() {
|
||||
let w = wait_time(30, 5, 10);
|
||||
assert!((w - 15.0).abs() < 0.01);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn wait_time_far_above_watermark() {
|
||||
let w = wait_time(30, 5, 100);
|
||||
assert!((w - 1.5).abs() < 0.01);
|
||||
}
|
||||
|
||||
// ═══════════════════════════════════════════
|
||||
// notify_delay
|
||||
// ═══════════════════════════════════════════
|
||||
|
||||
fn make_event(count: u32, retry_backoff: f64) -> NotificationEvent {
|
||||
NotificationEvent {
|
||||
id: uuid::Uuid::new_v4(),
|
||||
stage: crate::types::buildstatus::StagePhase::Bake,
|
||||
substage: "build".into(),
|
||||
outcome: crate::types::buildstatus::StageOutcome::PipelineSuccess,
|
||||
not_before: chrono::Utc::now(),
|
||||
priority: 0,
|
||||
effective_priority: 0,
|
||||
attempted: 0,
|
||||
retry_backoff,
|
||||
target: NotificationTarget {
|
||||
priority: 0,
|
||||
rules: None,
|
||||
count,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn notify_delay_default_factor() {
|
||||
let event = make_event(1, 2.0);
|
||||
let delay = notify_delay(&event);
|
||||
assert!(delay.as_secs_f64() > 0.0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn notify_delay_larger_factor() {
|
||||
let event1 = make_event(1, 2.0);
|
||||
let event2 = make_event(1, 4.0);
|
||||
let d1 = notify_delay(&event1).as_secs_f64();
|
||||
let d2 = notify_delay(&event2).as_secs_f64();
|
||||
assert!(d2 > d1, "larger retry_backoff should produce longer delay");
|
||||
}
|
||||
|
||||
// ═══════════════════════════════════════════
|
||||
// find_rule
|
||||
// ═══════════════════════════════════════════
|
||||
|
||||
#[test]
|
||||
fn find_rule_found() {
|
||||
use crate::finalize::NotificationTemplate;
|
||||
let rule = NotificationRule {
|
||||
enabled: true,
|
||||
trigger: vec![],
|
||||
template: NotificationTemplate {
|
||||
schema: None,
|
||||
on_success: None,
|
||||
on_failure: None,
|
||||
on_finish_of: None,
|
||||
},
|
||||
fallible: false,
|
||||
max_lives: 3,
|
||||
timeout: std::time::Duration::from_secs(30),
|
||||
config: serde_yaml::Value::Null,
|
||||
retry_backoff: 2.0,
|
||||
};
|
||||
let mut rules_map: NotificationRulesMap = HashMap::new();
|
||||
rules_map.insert(0, {
|
||||
let mut m: HashMap<String, Vec<NotificationRule>> = HashMap::new();
|
||||
m.insert("mail".into(), vec![rule]);
|
||||
m
|
||||
});
|
||||
let result = find_rule(&rules_map, "mail", 0);
|
||||
assert!(result.is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn find_rule_not_found() {
|
||||
let rules_map: NotificationRulesMap = HashMap::new();
|
||||
let result = find_rule(&rules_map, "mail", 0);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
std::time::Duration::from_secs_f64(delay)
|
||||
}
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
use crate::notify::NotificationRenderer;
|
||||
use crate::notify::types::NotificationRenderer;
|
||||
use thiserror::Error;
|
||||
|
||||
#[derive(Error, Debug)]
|
||||
@@ -23,4 +23,16 @@ pub enum NotifyError {
|
||||
|
||||
#[error("Unsupported renderer for {0}: {1}")]
|
||||
UnsupportedRenderer(String, NotificationRenderer),
|
||||
|
||||
#[error("Schema error: {0}")]
|
||||
SchemaError(String),
|
||||
|
||||
#[error("Template render error: {0}")]
|
||||
TemplateRenderError(String),
|
||||
|
||||
#[error("Circular template reference: {0}")]
|
||||
TemplateLoop(String),
|
||||
|
||||
#[error("Template reference depth exceeded (max {max}): {name}")]
|
||||
TemplateDepthExceeded { name: String, max: u32 },
|
||||
}
|
||||
|
||||
@@ -0,0 +1,345 @@
|
||||
use std::collections::HashMap;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::Duration;
|
||||
|
||||
use chrono::Utc;
|
||||
use tokio::sync::Notify;
|
||||
|
||||
use crate::finalize::NotificationTemplateDef;
|
||||
use crate::notify::template::{
|
||||
build_batch_template_context, render_template, resolve_template,
|
||||
select_template_str,
|
||||
};
|
||||
use crate::notify::types::{
|
||||
MethodStatus, NotificationEvent,
|
||||
NotificationEventSender, NotificationPluginMap, NotificationRenderer,
|
||||
NotificationRulesMap,
|
||||
};
|
||||
use crate::notify::util::{
|
||||
find_rule, get_fallible, is_temporary_error, notify_delay, wait_time,
|
||||
};
|
||||
|
||||
/// Shared state for a single dispatch bucket.
|
||||
struct BucketEntry {
|
||||
events: Arc<Mutex<Vec<NotificationEvent>>>,
|
||||
result: Arc<Mutex<Option<(MethodStatus, Vec<NotificationEvent>)>>>,
|
||||
done: Arc<AtomicBool>,
|
||||
notify: Arc<Notify>,
|
||||
method: String,
|
||||
rule_id: usize,
|
||||
_handle: tokio::task::JoinHandle<()>,
|
||||
}
|
||||
|
||||
/// Concurrent notification handler with bucket-based dispatch.
|
||||
///
|
||||
/// Events are routed to matching buckets. Buckets cool down and dispatch
|
||||
/// plugins concurrently. The caller awaits bucket completion, then handles
|
||||
/// retry/escalation decisions.
|
||||
pub struct NotificationHandler {
|
||||
buckets: Mutex<HashMap<(String, usize), BucketEntry>>,
|
||||
pub config: Arc<NotificationRulesMap>,
|
||||
plugins: Arc<NotificationPluginMap>,
|
||||
templates: Arc<HashMap<String, NotificationTemplateDef>>,
|
||||
ctx: Arc<workshop_engine::ExecutionContext>,
|
||||
retry_tx: NotificationEventSender,
|
||||
period: u32,
|
||||
watermark: u32,
|
||||
}
|
||||
|
||||
impl NotificationHandler {
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub fn new(
|
||||
config: Arc<NotificationRulesMap>,
|
||||
plugins: Arc<NotificationPluginMap>,
|
||||
templates: Arc<HashMap<String, NotificationTemplateDef>>,
|
||||
ctx: Arc<workshop_engine::ExecutionContext>,
|
||||
retry_tx: NotificationEventSender,
|
||||
period: u32,
|
||||
watermark: u32,
|
||||
) -> Self {
|
||||
NotificationHandler {
|
||||
buckets: Mutex::new(HashMap::new()),
|
||||
config,
|
||||
plugins,
|
||||
templates,
|
||||
ctx,
|
||||
retry_tx,
|
||||
period,
|
||||
watermark,
|
||||
}
|
||||
}
|
||||
|
||||
/// Process a single event: route to buckets, await completion,
|
||||
/// then handle retry/escalation for each bucket result.
|
||||
pub async fn process_event(self: Arc<Self>, event: NotificationEvent) {
|
||||
let entries = self.route_event(&event);
|
||||
if entries.is_empty() {
|
||||
return;
|
||||
}
|
||||
|
||||
// Await all buckets — yields CPU
|
||||
for (_, notify) in &entries {
|
||||
notify.notified().await;
|
||||
}
|
||||
|
||||
// Collect results and remove entries
|
||||
let mut results: Vec<(String, usize, Option<(MethodStatus, Vec<NotificationEvent>)>)> =
|
||||
Vec::new();
|
||||
{
|
||||
let mut buckets = self.buckets.lock().unwrap();
|
||||
for (key, _) in &entries {
|
||||
if let Some(entry) = buckets.remove(key) {
|
||||
let r = entry.result.lock().unwrap().take();
|
||||
results.push((entry.method, entry.rule_id, r));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Handle retry/escalation per method
|
||||
for (method, _rule_id, result) in results {
|
||||
let Some((status, mut events)) = result else { continue };
|
||||
|
||||
match &status {
|
||||
MethodStatus::Success => {
|
||||
log::debug!("[NOTIFY] {}: success ({} events)", method, events.len());
|
||||
}
|
||||
MethodStatus::TemporaryError(_) => {
|
||||
log::warn!("[NOTIFY] {}: temporary error, retrying", method);
|
||||
for event in events.iter_mut() {
|
||||
event.target.rules = Some(vec![method.clone()]);
|
||||
event.target.count += 1;
|
||||
event.not_before = Utc::now() + notify_delay(event);
|
||||
let _ = self.retry_tx.send(event.clone()).await;
|
||||
}
|
||||
}
|
||||
MethodStatus::PermanentError(_) | MethodStatus::Exhausted(_) => {
|
||||
let fallible = get_fallible(&self.config, &method);
|
||||
if fallible {
|
||||
log::error!("[NOTIFY] {}: discard (fallible)", method);
|
||||
} else {
|
||||
log::warn!("[NOTIFY] {}: escalating priority", method);
|
||||
for event in events.iter_mut() {
|
||||
event.target.priority += 1;
|
||||
event.target.rules = None;
|
||||
event.attempted = event.target.count;
|
||||
event.target.count = 0;
|
||||
event.effective_priority = event.priority;
|
||||
let _ = self.retry_tx.send(event.clone()).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Route a single event to all matching buckets.
|
||||
///
|
||||
/// Returns Notify handles for awaiting bucket completion.
|
||||
fn route_event(
|
||||
&self,
|
||||
event: &NotificationEvent,
|
||||
) -> Vec<((String, usize), Arc<Notify>)> {
|
||||
let mut matches: Vec<(String, usize)> = Vec::new();
|
||||
|
||||
for ruleset in self.config.values() {
|
||||
for (method, rules) in ruleset.iter() {
|
||||
if let Some(ref method_rules) = event.target.rules {
|
||||
if !method_rules.contains(method) {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
for (rule_id, rule) in rules.iter().enumerate() {
|
||||
if !rule.enabled {
|
||||
continue;
|
||||
}
|
||||
if rule.accept(&event.outcome, &event.stage, &event.substage) {
|
||||
matches.push((method.clone(), rule_id));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let mut buckets = self.buckets.lock().unwrap();
|
||||
let mut result: Vec<((String, usize), Arc<Notify>)> = Vec::new();
|
||||
|
||||
for key in matches {
|
||||
if let Some(entry) = buckets.get(&key) {
|
||||
if entry.done.load(Ordering::Acquire) {
|
||||
buckets.remove(&key);
|
||||
}
|
||||
}
|
||||
|
||||
if let Some(entry) = buckets.get(&key) {
|
||||
entry.events.lock().unwrap().push(event.clone());
|
||||
result.push((key, entry.notify.clone()));
|
||||
} else {
|
||||
let (entry, notify) = Self::spawn_bucket(
|
||||
&key.0,
|
||||
key.1,
|
||||
&self.config,
|
||||
&self.plugins,
|
||||
&self.templates,
|
||||
&self.ctx,
|
||||
self.period,
|
||||
self.watermark,
|
||||
);
|
||||
entry.events.lock().unwrap().push(event.clone());
|
||||
result.push((key.clone(), notify));
|
||||
buckets.insert(key, entry);
|
||||
}
|
||||
}
|
||||
|
||||
result
|
||||
}
|
||||
|
||||
/// Spawn a bucket task: cooldown → take → dispatch → store → notify.
|
||||
fn spawn_bucket(
|
||||
method: &str,
|
||||
rule_id: usize,
|
||||
config: &Arc<NotificationRulesMap>,
|
||||
plugins: &Arc<NotificationPluginMap>,
|
||||
templates: &Arc<HashMap<String, NotificationTemplateDef>>,
|
||||
ctx: &Arc<workshop_engine::ExecutionContext>,
|
||||
period: u32,
|
||||
watermark: u32,
|
||||
) -> (BucketEntry, Arc<Notify>) {
|
||||
let events: Arc<Mutex<Vec<NotificationEvent>>> = Arc::new(Mutex::new(Vec::new()));
|
||||
let result: Arc<Mutex<Option<(MethodStatus, Vec<NotificationEvent>)>>> =
|
||||
Arc::new(Mutex::new(None));
|
||||
let notify = Arc::new(Notify::new());
|
||||
let done = Arc::new(AtomicBool::new(false));
|
||||
|
||||
let ec = events.clone();
|
||||
let rc = result.clone();
|
||||
let nc = notify.clone();
|
||||
let dc = done.clone();
|
||||
let m = method.to_string();
|
||||
let c = config.clone();
|
||||
let p = plugins.clone();
|
||||
let t = templates.clone();
|
||||
let x = ctx.clone();
|
||||
|
||||
let handle = tokio::spawn(async move {
|
||||
// Phase 1: Cooldown
|
||||
let count = ec.lock().unwrap().len() as u32;
|
||||
let wait_secs = wait_time(period, watermark, count);
|
||||
tokio::time::sleep(Duration::from_secs_f32(wait_secs)).await;
|
||||
|
||||
// Phase 2: Take events
|
||||
let batch = std::mem::take(&mut *ec.lock().unwrap());
|
||||
if batch.is_empty() {
|
||||
dc.store(true, Ordering::Release);
|
||||
nc.notify_waiters();
|
||||
return;
|
||||
}
|
||||
|
||||
// Phase 3: Dispatch
|
||||
let rule = match find_rule(&c, &m, rule_id) {
|
||||
Ok(r) => r,
|
||||
Err(_) => {
|
||||
*rc.lock().unwrap() = Some((MethodStatus::PermanentError("rule not found".into()), batch));
|
||||
dc.store(true, Ordering::Release);
|
||||
nc.notify_waiters();
|
||||
return;
|
||||
}
|
||||
};
|
||||
|
||||
let plugin = match p.get(&m) {
|
||||
Some(p) => p,
|
||||
None => {
|
||||
*rc.lock().unwrap() = Some((MethodStatus::PermanentError("plugin not found".into()), batch));
|
||||
dc.store(true, Ordering::Release);
|
||||
nc.notify_waiters();
|
||||
return;
|
||||
}
|
||||
};
|
||||
|
||||
let renderer = plugin.metadata.renderer.clone().unwrap_or_default();
|
||||
let argument = build_plugin_argument(&renderer, rule, &m, &t, &x, &batch);
|
||||
let status = match argument {
|
||||
Some(arg) => {
|
||||
match tokio::time::timeout(rule.timeout, plugin.call(arg, &x, &None)).await {
|
||||
Ok(Ok(())) => MethodStatus::Success,
|
||||
Ok(Err(e)) if is_temporary_error(&e) => {
|
||||
MethodStatus::TemporaryError(format!("{}", e))
|
||||
}
|
||||
Ok(Err(e)) => MethodStatus::PermanentError(format!("{}", e)),
|
||||
Err(_) => MethodStatus::TemporaryError("timeout".into()),
|
||||
}
|
||||
}
|
||||
None => MethodStatus::PermanentError("argument build failed".into()),
|
||||
};
|
||||
|
||||
*rc.lock().unwrap() = Some((status, batch));
|
||||
dc.store(true, Ordering::Release);
|
||||
nc.notify_waiters();
|
||||
});
|
||||
|
||||
let entry = BucketEntry {
|
||||
events,
|
||||
result,
|
||||
done,
|
||||
notify: notify.clone(),
|
||||
method: method.to_string(),
|
||||
rule_id,
|
||||
_handle: handle,
|
||||
};
|
||||
|
||||
(entry, notify)
|
||||
}
|
||||
}
|
||||
|
||||
/// Build the YAML argument for a plugin call based on renderer mode.
|
||||
fn build_plugin_argument(
|
||||
renderer: &NotificationRenderer,
|
||||
rule: &crate::notify::types::NotificationRule,
|
||||
method: &str,
|
||||
templates: &HashMap<String, NotificationTemplateDef>,
|
||||
ctx: &workshop_engine::ExecutionContext,
|
||||
events: &[NotificationEvent],
|
||||
) -> Option<serde_yaml::Value> {
|
||||
let first = events.first()?;
|
||||
|
||||
match renderer {
|
||||
NotificationRenderer::Server => {
|
||||
let template_def = resolve_template(
|
||||
&rule.template.schema,
|
||||
method,
|
||||
templates,
|
||||
ctx.standalone,
|
||||
None,
|
||||
)
|
||||
.ok()?;
|
||||
|
||||
let template_str = select_template_str(&template_def, &first.outcome)
|
||||
.or_else(|| rule.template.on_success.as_deref())
|
||||
.or_else(|| rule.template.on_failure.as_deref())
|
||||
.or_else(|| {
|
||||
rule.template
|
||||
.on_finish_of
|
||||
.as_ref()
|
||||
.map(|o| o.content.as_str())
|
||||
})?;
|
||||
|
||||
let data = build_batch_template_context(ctx, events);
|
||||
let rendered = render_template(template_str, &data).ok()?;
|
||||
|
||||
let mut argument = rule.config.clone();
|
||||
if let serde_yaml::Value::Mapping(ref mut map) = argument {
|
||||
map.insert(
|
||||
serde_yaml::Value::String("_rendered_content".into()),
|
||||
serde_yaml::Value::String(rendered),
|
||||
);
|
||||
}
|
||||
log::info!("SSR: rendered content for method '{}'", method);
|
||||
Some(argument)
|
||||
}
|
||||
NotificationRenderer::Plugin => {
|
||||
unimplemented!("Plugin renderer is not yet implemented")
|
||||
}
|
||||
NotificationRenderer::Interactive => {
|
||||
unimplemented!("Interactive renderer is not yet implemented")
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
use crate::notify::types::{NotificationEvent, NotificationEventReceiver, NotificationEventSender};
|
||||
use std::collections::BinaryHeap;
|
||||
|
||||
/// Priority queue for notification events.
|
||||
///
|
||||
/// Integrates with `notify()`:
|
||||
/// - `tx` sends events to notify() for processing
|
||||
/// - `rx` receives retry events back from notify()
|
||||
pub struct NotificationQueue {
|
||||
pub pq: BinaryHeap<NotificationEvent>,
|
||||
pub tx: NotificationEventSender,
|
||||
pub rx: NotificationEventReceiver,
|
||||
}
|
||||
|
||||
impl NotificationQueue {
|
||||
pub fn new(tx: NotificationEventSender, rx: NotificationEventReceiver) -> Self {
|
||||
Self {
|
||||
pq: BinaryHeap::new(),
|
||||
tx,
|
||||
rx,
|
||||
}
|
||||
}
|
||||
|
||||
/// Push an event into the priority queue.
|
||||
pub fn enqueue(&mut self, event: NotificationEvent) {
|
||||
self.pq.push(event);
|
||||
}
|
||||
|
||||
/// Non-blocking single round: recycle retry events into PQ, then
|
||||
/// pop and send all ready events (not_before ≤ now).
|
||||
pub async fn flush_now(&mut self) -> anyhow::Result<()> {
|
||||
// Recycle retry events
|
||||
while let Ok(event) = self.rx.try_recv() {
|
||||
self.pq.push(event);
|
||||
}
|
||||
// Send ready events
|
||||
let now = chrono::Utc::now();
|
||||
while let Some(peek) = self.pq.peek() {
|
||||
if peek.not_before > now {
|
||||
break;
|
||||
}
|
||||
let event = self.pq.pop().unwrap();
|
||||
self.tx.send(event).await?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Blocking drain until all events are processed.
|
||||
/// For standalone mode: called after prebake/bake/finalize to wait for completion.
|
||||
/// For c/d mode: the main execution loop.
|
||||
pub async fn drain(&mut self) -> anyhow::Result<()> {
|
||||
loop {
|
||||
// Drain retry channel
|
||||
while let Ok(event) = self.rx.try_recv() {
|
||||
self.pq.push(event);
|
||||
}
|
||||
// Determine sleep deadline
|
||||
let until = self
|
||||
.pq
|
||||
.peek()
|
||||
.map(|e| e.not_before)
|
||||
.unwrap_or_else(|| chrono::Utc::now() + chrono::Duration::seconds(3600));
|
||||
|
||||
let std_until = (until - chrono::Utc::now())
|
||||
.to_std()
|
||||
.unwrap_or(std::time::Duration::from_secs(1));
|
||||
|
||||
tokio::select! {
|
||||
event = self.rx.recv() => {
|
||||
match event {
|
||||
Some(event) => self.pq.push(event),
|
||||
None => break Ok(()),
|
||||
}
|
||||
}
|
||||
_ = tokio::time::sleep(std_until) => {
|
||||
let now = chrono::Utc::now();
|
||||
while let Some(peek) = self.pq.peek() {
|
||||
if peek.not_before > now { break; }
|
||||
let event = self.pq.pop().unwrap();
|
||||
self.tx.send(event).await?;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user