Compare commits
13 Commits
20e2ab4224
...
master
| Author | SHA1 | Date | |
|---|---|---|---|
| df5026cdbe | |||
| d1ad08ef3a | |||
| cf2968e720 | |||
| 28abc5d207 | |||
| 6e7b96cd66 | |||
| 3e80825d58 | |||
| c97eafab29 | |||
| f737b6a24d | |||
| 681b7d1333 | |||
| f07588a716 | |||
| b0530fb858 | |||
| fd18b547e5 | |||
| cf317b55c6 |
+3
-1
@@ -9,7 +9,7 @@ archived
|
||||
*.bak
|
||||
*.clean
|
||||
session*
|
||||
llm.env
|
||||
.secret
|
||||
|
||||
# System test artifacts
|
||||
tests/results/
|
||||
@@ -18,3 +18,5 @@ tests/**/__pycache__/
|
||||
tests/.pytest_cache/
|
||||
tests/llm/.cache/
|
||||
*.pyc
|
||||
.staging
|
||||
.pytest_cache
|
||||
|
||||
@@ -1,83 +1,72 @@
|
||||
# PROJECT KNOWLEDGE BASE
|
||||
|
||||
**Generated:** 2026-04-22
|
||||
**Commit:** 7b3b71a
|
||||
**Generated:** 2026-05-19
|
||||
**Commit:** 28abc5d
|
||||
**Branch:** master
|
||||
|
||||
## OVERVIEW
|
||||
HoneyBiscuitWorkshop is a Rust-based CI/CD system ("蜜饼工坊") with LLM-powered pipeline auto-configuration. 9 independent Cargo crates, no workspace.
|
||||
HoneyBiscuitWorkshop ("蜜饼工坊") — Rust CI/CD system with decorator-driven shell pipeline DSL and LLM-powered configuration. 3 independent Cargo crates, no workspace.
|
||||
|
||||
## STRUCTURE
|
||||
```
|
||||
./
|
||||
├── workshop-baker/ # Main orchestrator (CLI + daemon)
|
||||
├── workshop-agent/ # HTTP agent server
|
||||
├── workshop-pipeline/ # Config library (prebake/finalize)
|
||||
├── workshop-llm-detector/ # LLM repo analyzer
|
||||
├── workshop-vault/ # Secret management client
|
||||
├── workshop-cert/ # TLS certificate generator
|
||||
├── workshop-deviceid/ # Device ID library
|
||||
├── workshop-builder-native/ # Stub (unused)
|
||||
├── workshop-helper-mac/ # MAC address utility
|
||||
├── docs/ # mdBook documentation (zh-CN)
|
||||
├── playground/ # Experimentation
|
||||
└── third_party/ # Empty
|
||||
├── workshop-baker/ # Main orchestrator (CLI + pipeline stages)
|
||||
├── workshop-engine/ # Execution engine (extracted from baker)
|
||||
├── workshop-getterurl/ # URL fetching (go-getter style)
|
||||
├── docs/ # mdBook documentation (zh-CN)
|
||||
├── templates/ # Pipeline templates (bake.sh, finalize.yml, prebake.yml)
|
||||
└── archived/ # Deprecated crates (7 moved here)
|
||||
```
|
||||
|
||||
## WHERE TO LOOK
|
||||
| Task | Location | Notes |
|
||||
|------|----------|-------|
|
||||
| Baker (main) | `workshop-baker/src/{bake,prebake,engine}/` | Core pipeline logic |
|
||||
| Agent server | `workshop-agent/src/` | Actix-web HTTP + TLS |
|
||||
| Pipeline config | `workshop-pipeline/src/config/` | prebake/finalize YAML |
|
||||
| LLM integration | `workshop-llm-detector/src/llm/` | Ollama/OpenAI clients |
|
||||
| Secrets | `workshop-vault/src/` | Vault client wrapper |
|
||||
| Baker (CLI) | `workshop-baker/src/{lib,main}.rs` | Clap CLI + tokio entry |
|
||||
| Pipeline stages | `workshop-baker/src/{bake,prebake,finalize}/` | Stage orchestration |
|
||||
| Notifications | `workshop-baker/src/notify/` | Plugin-based event notifications |
|
||||
|| Engine (execution) | `workshop-engine/src/` | Process mgmt, PM detection |
|
||||
| URL fetching | `workshop-getterurl/src/` | Git/HTTP/File resource getter |
|
||||
|
||||
## CODE MAP
|
||||
| Symbol | Type | Location | Role |
|
||||
|--------|------|----------|------|
|
||||
| PipelineConfig | struct | workshop-pipeline/src/lib.rs:27 | Full pipeline config |
|
||||
| CICDLLMHelper | struct | workshop-llm-detector/src/lib.rs:16 | LLM analysis entry |
|
||||
| Engine | struct | workshop-baker/src/engine.rs | Build executor |
|
||||
| VaultClient | struct | workshop-vault/src/client.rs | Secret ops |
|
||||
| Cli | struct | workshop-baker/src/lib.rs:21 | CLI argument struct |
|
||||
| ExecutionContext | struct | workshop-engine/src/types.rs:10 | Pipeline context |
|
||||
| Engine | struct | workshop-engine/src/types.rs:99 | Build executor (placeholder) |
|
||||
| PrebakeConfig | struct | workshop-baker/src/prebake/config.rs | Prebake YAML schema |
|
||||
| FinalizeConfig | struct | workshop-baker/src/finalize/config.rs:21 | Finalize YAML schema |
|
||||
| GetterUrl | struct | workshop-getterurl/src/lib.rs:84 | Parsed resource URL |
|
||||
| Getter | trait | workshop-getterurl/src/getter.rs:18 | Resource fetch trait |
|
||||
|
||||
## CONVENTIONS
|
||||
|
||||
### Rust
|
||||
- **Edition**: 2024 (non-standard, requires Rust 1.85+)
|
||||
- Deprecate `mod.rs`
|
||||
- **Edition**: 2024 (requires Rust 1.85+), deprecates `mod.rs`
|
||||
- **Naming**: `snake_case` files/modules, `PascalCase` types, `SCREAMING_SNAKE_CASE` consts
|
||||
- **Error handling**: `thiserror` + `anyhow` combo
|
||||
- **Error handling**: `thiserror` + `anyhow` combo; `HasExitCode` trait for exit codes
|
||||
- **Async**: `#[tokio::main]` + `tokio` with "full" features
|
||||
- **Imports**: `std` → `external_crate` → `crate::module`
|
||||
- **Dual-mode**: CLI commands (prebake/bake/finalize) + daemon mode (unimplemented)
|
||||
|
||||
### Testing
|
||||
- **Rust**: `#[test]` inline, `#[tokio::test]` async, `tests/*.rs` integration
|
||||
- **Python**: `pytest` with `tests/integration/test_*.py` (Docker-based)
|
||||
- **Benchmarks**: Criterion with `harness = false`
|
||||
- **Rust**: `#[test]` inline, `#[tokio::test]` async, `tests/` integration
|
||||
- **Benchmarks**: Criterion (configured in Cargo.toml)
|
||||
|
||||
### Build Pipeline
|
||||
- **prebake.yml**: Environment setup (docker/firecracker/baremetal)
|
||||
- **bake.sh**: Build execution with `# @pipeline` decorators
|
||||
- **finalize.yml**: Packaging, deployment, notifications
|
||||
- **Annotations**: `@timeout()`, `@retry()`, `@parallel`, `@fallible`
|
||||
### CI
|
||||
- **GitHub Actions**: rust-toolchain (stable), clippy + rustfmt, `cargo test --all-features`
|
||||
- **Matrix**: workshop-baker, workshop-engine
|
||||
|
||||
## ANTI-PATTERNS (THIS PROJECT)
|
||||
|
||||
1. **Python in Rust project** — workshop-baker has Python tests (pytest.ini, test_prebake.py)
|
||||
2. **temp/ directory** — Non-standard in Rust projects
|
||||
3. **Certificates in source** — artifact workshop-cert/certs/ not excluded
|
||||
4. **Deprecated projects** — workshop-executor.old, workshop-monitor.old still present
|
||||
5. **Hardcoded mirrors** — Tsinghua University mirrors in configs
|
||||
6. **Rust 2024 edition** — May not work with stable toolchains
|
||||
1. **Hardcoded user** — "vulcan" username hardcoded throughout prebake
|
||||
2. **Unsafe libc** — workshop-baker/src/prebake/security.rs has 5+ unsafe blocks
|
||||
3. **Rust 2024 edition** — May not work with stable toolchains
|
||||
4. **Empty daemon** — `daemon/` dir is empty, Daemon command is `unimplemented!()`
|
||||
|
||||
## UNIQUE STYLES
|
||||
|
||||
- **Pipeline DSL**: Shell scripts with `# @decorator` annotations
|
||||
- **Dual-mode baker**: CLI commands + daemon mode
|
||||
- **LLM cookbook system**: YAML cookbooks for language-specific builds
|
||||
- **Notification system**: Plugin-based with batching, retry, priority groups
|
||||
- **LLM cookbook system**: YAML cookbooks for language-specific builds (archived)
|
||||
- **Chinese docs** — Documentation primarily in zh-CN
|
||||
- **User "vulcan"** — Custom build user (not root/runner)
|
||||
|
||||
## COMMANDS
|
||||
```bash
|
||||
@@ -86,23 +75,18 @@ cargo build --release -p workshop-baker
|
||||
|
||||
# Test
|
||||
cargo test -p workshop-baker
|
||||
pytest tests/integration/test_prebake.py -v
|
||||
cargo test -p workshop-engine
|
||||
|
||||
# Lint
|
||||
cargo clippy -- -D warnings
|
||||
cargo fmt --all
|
||||
|
||||
# Run baker
|
||||
cargo run --bin workshop-baker -- prebake config.yml
|
||||
cargo run --bin workshop-baker -- bake script.sh
|
||||
|
||||
# Run agent
|
||||
cargo run --bin workshop-agent
|
||||
# Run
|
||||
cargo run --bin workshop-baker -- --pipeline demo --build-id 1 --prebake prebake.yml --bake bake.sh --finalize finalize.yml
|
||||
```
|
||||
|
||||
## NOTES
|
||||
|
||||
- **Docker required** for integration tests
|
||||
- **workshop-builder-native** is a stub (3-line Hello World)
|
||||
- No root workspace Cargo.toml — each crate is independent
|
||||
- `AGENTS.md.old` at root — legacy documentation
|
||||
- **No root workspace** — each crate independent, build separately
|
||||
- **7 archived crates** — workshop-agent, workshop-cert, workshop-deviceid, workshop-helper-mac, workshop-llm-detector, workshop-pipeline, workshop-vault
|
||||
- **Python tests removed** — pytest.ini exists but test files no longer in workspace
|
||||
|
||||
@@ -29,8 +29,10 @@
|
||||
- [测试策略](zh-CN/vol3_dev/testing_strategy.md)
|
||||
- [插件系统指南](zh-CN/vol3_dev/plugin_system_guide.md)
|
||||
- [Engine 模块指南](zh-CN/vol3_dev/engine_module_guide.md)
|
||||
- [设计决策](zh-CN/vol3_dev/design_decisions.md)
|
||||
- [添加包管理器](zh-CN/vol3_dev/add_package_manager.md)
|
||||
- [自定义插件](zh-CN/vol3_dev/custom_plugin.md)
|
||||
- [自定义 Decorator](zh-CN/vol3_dev/custom_decorator.md)
|
||||
- [模板变量](zh-CN/vol3_dev/template_variables.md)
|
||||
- [邮件模板](zh-CN/vol3_dev/mail_template.md)
|
||||
- [Resource 模型](zh-CN/vol3_dev/resource_model.md)
|
||||
|
||||
@@ -0,0 +1,412 @@
|
||||
# 设计决策
|
||||
|
||||
本文档记录了 HoneyBiscuitWorkshop 架构演进过程中的关键设计决策及其背后的思考。
|
||||
|
||||
> **注意**:本文档面向开发者,记录的是**目标架构**的设计方向,而非当前代码的实现现状。
|
||||
> 当前 bakerd 尚未完全实现(`daemon.rs` 为 `todo!()`),bare mode 仅用于开发调试。
|
||||
|
||||
---
|
||||
|
||||
## 1. Bare Mode 与 Daemon Mode
|
||||
|
||||
### 决策
|
||||
|
||||
**Bare mode 仅用于开发调试,不承载生产语义。**
|
||||
|
||||
### 背景
|
||||
|
||||
当前代码提供两种运行入口:
|
||||
|
||||
| 模式 | 入口 | 状态 | 用途 |
|
||||
|------|------|------|------|
|
||||
| **Bare mode** | `cargo run -- prebake/bake/finalize <args>` | 可用 | 开发调试 |
|
||||
| **Daemon mode** | `cargo run --` (无子命令) | `todo!()` | 生产 |
|
||||
|
||||
### 推论
|
||||
|
||||
- Bare mode 中 `ctx.privileged` 是写死的 fake 值(`Prebake: true`, `Bake: false`),不代表生产行为
|
||||
- Bare mode 中 `os_info::get()` 读取的是 HOST 的 OS 信息,不代表环境内的 OS
|
||||
- 不要从 bare mode 的行为推导生产路径的问题
|
||||
- 不要为了"修复" bare mode 的问题而给生产设计增加不必要的抽象
|
||||
|
||||
### 代码残留
|
||||
|
||||
当前 `bake.rs` 中存在以下已在设计层面淘汰、但因 bare mode 未被清理的代码:
|
||||
|
||||
- `fn privileged()` — 从未被调用
|
||||
- `type TaskResult` — 从未被使用
|
||||
- `type TaskFn` — 从未被使用
|
||||
|
||||
这些是 `bake.rs` 废弃功能的残留,不涉及生产设计。清理它们不改变任何行为。
|
||||
|
||||
---
|
||||
|
||||
## 2. 环境模型
|
||||
|
||||
### 决策
|
||||
|
||||
**bakerd 先创建环境(容器/VM/裸机),然后 baker 在环境内执行全部三个阶段(prebake → bake → finalize)。**
|
||||
|
||||
### 架构
|
||||
|
||||
```
|
||||
bakerd
|
||||
├── prepare_environment (容器/VM/裸机) ← 先建隔离环境
|
||||
├── prebake (全在环境内执行) ← 包括 depssystem 的包安装
|
||||
├── bake (全在环境内执行) ← 构建脚本
|
||||
├── finalize (全在环境内执行) ← 部署通知
|
||||
└── cleanup
|
||||
```
|
||||
|
||||
### 推论
|
||||
|
||||
- `prepare_environment` 不是 prebake 的一个 stage——它是 prebake 的前置条件
|
||||
- `depssystem.rs` 中 `os_info::get()` 读取的应当是**环境内**的 `/etc/os-release`,逻辑正确
|
||||
- 容器镜像只需拉取(`prepare_container`),不需要在代码层面管理容器生命周期——那是 bakerd 的职责
|
||||
- 当前 `prepare_container` 返回的 `_container` 被丢弃,是因为 bakerd 尚未实现,不是设计缺陷
|
||||
|
||||
### 环境类型
|
||||
|
||||
```yaml
|
||||
env:
|
||||
type: container # Docker/Podman 容器
|
||||
# type: firecracker # Firecracker 微 VM
|
||||
# type: baremetal # 裸机直接执行
|
||||
image: ubuntu:24.04
|
||||
```
|
||||
|
||||
环境声明决定了:
|
||||
|
||||
1. 隔离级别(进程级 / VM级 / 无隔离)
|
||||
2. 操作系统和工具链(通过 `image`)
|
||||
3. 包管理器(从 OS 推导)
|
||||
|
||||
---
|
||||
|
||||
## 3. PM 选择链
|
||||
|
||||
### 决策
|
||||
|
||||
**包管理器(PM)应从环境声明中推导,而不是通过运行时探测。**
|
||||
|
||||
### 选择链
|
||||
|
||||
```
|
||||
用户声明 bakerd 确定 执行时确定
|
||||
env.image: ubuntu:24.04
|
||||
│
|
||||
▼
|
||||
容器运行时 (Docker/VM)
|
||||
│ os_info::get() 在环境内
|
||||
▼
|
||||
OS: Ubuntu 24.04
|
||||
│ 从已知映射推导
|
||||
▼
|
||||
PM: apt
|
||||
│ depssystem 使用 PM trait 操作
|
||||
▼
|
||||
apt install -y {packages}
|
||||
```
|
||||
|
||||
### 为什么淘汰 detect()
|
||||
|
||||
`pm::detect()` 的当前逻辑:
|
||||
|
||||
```rust
|
||||
pub fn detect(distro: &os_info::Info) -> Option<Box<dyn PackageManager>> {
|
||||
all_managers().into_iter().find(|pm| pm.adopt(distro))
|
||||
}
|
||||
```
|
||||
|
||||
它在解决一个**已经知道答案的问题**:
|
||||
|
||||
- 用户声明了 `image: ubuntu:24.04`
|
||||
- 容器内 `/etc/os-release` 确认了 Ubuntu
|
||||
- `detect()` 遍历所有 PM 并匹配,得出 apt
|
||||
- **结果和环境声明一致,但绕了一大圈**
|
||||
|
||||
`detect()` 的唯一价值场景:
|
||||
|
||||
- **Bare mode** 用户没配环境时,省一个配置项
|
||||
- 这在生产路径中不应出现(bakerd 总是提供环境)
|
||||
|
||||
### 推论
|
||||
|
||||
- PM trait 保留(见下节),但 `adopt()` 方法在生产路径中不需要
|
||||
- `detect()` / `adopt()` / `os_info::get()` 这条路径应当是 dev-only 的便利设施
|
||||
- Daemon 模式中 PM 的选择依据是环境声明,不是运行时探测
|
||||
- 当前 `os_info::Type` 的 58 个变体覆盖不全(仅 Pacman 可用)——这是 bare mode 的问题,不影响生产设计
|
||||
|
||||
---
|
||||
|
||||
## 4. PM Trait 的职责边界
|
||||
|
||||
### 决策
|
||||
|
||||
**PM trait 保留核心操作能力,淘汰探测分类能力。**
|
||||
|
||||
### 核心方法
|
||||
|
||||
```rust
|
||||
pub trait PackageManager {
|
||||
fn name(&self) -> &'static str;
|
||||
fn update(&self) -> Vec<Command>;
|
||||
fn install(&self, package: &[String]) -> Vec<Command>;
|
||||
fn change_mirror(&self, repo: &str, osinfo: Option<&Info>) -> Vec<Command>;
|
||||
fn add_repository(&self, repo: &Value, osinfo: Option<&Info>) -> Vec<Command>;
|
||||
}
|
||||
```
|
||||
|
||||
这四个方法是 PM-specific 的操作,具有真实复杂度:
|
||||
|
||||
| 操作 | PM 间差异 | 原因 |
|
||||
|------|-----------|------|
|
||||
| `install` | 参数完全不同 | `apt install -y` vs `pacman -S --noconfirm` |
|
||||
| `update` | 命令不同 | `apt update` vs `pacman -Sy` |
|
||||
| `change_mirror` | 配置格式不同 | sources.list vs mirrorlist vs .repo |
|
||||
| `add_repository` | 仓库机制不同 | PPA vs AUR vs COPR |
|
||||
|
||||
### 为什么镜像源和仓库在容器中仍然需要
|
||||
|
||||
容器不解决源管理问题:
|
||||
|
||||
```
|
||||
docker pull ubuntu:24.04
|
||||
# 容器内 sources.list → archive.ubuntu.com
|
||||
# 但用户在北京 → 需要 mirror.tuna.tsinghua.edu.cn
|
||||
# 这不是预置在镜像中的——源是运行态配置
|
||||
```
|
||||
|
||||
`change_mirror` 和 `add_repository` 都是**容器运行时需要**的操作,不能预置在基础镜像中。PM trait 的存在价值就在于这些操作。
|
||||
|
||||
### 淘汰的方法
|
||||
|
||||
```rust
|
||||
fn adopt(&self, distro: &Info) -> bool; // 生产路径不需要
|
||||
fn binary(&self) -> &'static str; // 从未被使用
|
||||
```
|
||||
|
||||
### 实现策略
|
||||
|
||||
在 daemon 模式下,PM 的选型由环境声明确定(`image: ubuntu:24.04` → `apt`),PM 实现仅需按需注册,不需要自述适配范围:
|
||||
|
||||
```rust
|
||||
// 不再需要 detect() + adopt()
|
||||
// PM 实例由工厂方法根据 PM 名称字符串创建
|
||||
pub fn from_name(name: &str) -> Option<Box<dyn PackageManager>> {
|
||||
match name {
|
||||
"apt" => Some(Box::new(Apt)),
|
||||
"pacman" => Some(Box::new(Pacman)),
|
||||
"dnf" => Some(Box::new(Dnf)),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. 发行版多态
|
||||
|
||||
### 决策
|
||||
|
||||
**流水线不应是发行版多态的。发行版差异由容器化解决。**
|
||||
|
||||
### 原因
|
||||
|
||||
流水线的执行体(bake 脚本)天然发行版绑定:
|
||||
|
||||
```bash
|
||||
# @pipeline
|
||||
function build() {
|
||||
cmake -B build -DCMAKE_INSTALL_PREFIX=/usr # Linux 路径
|
||||
make -j$(nproc)
|
||||
}
|
||||
```
|
||||
|
||||
- 换到 macOS:`/usr` 只读,路径不同
|
||||
- 换到 Windows:MSVC 而不是 GCC
|
||||
- 换到 Alpine:musl 而不是 glibc
|
||||
|
||||
这与 PM 的差异是同一类问题,但 PM 抽象不能解决它。**容器才是解决方案:**
|
||||
|
||||
```yaml
|
||||
# 在 Ubuntu 上构建
|
||||
env:
|
||||
type: container
|
||||
image: ubuntu:24.04
|
||||
```
|
||||
|
||||
```yaml
|
||||
# 在 Alpine 上构建
|
||||
env:
|
||||
type: container
|
||||
image: alpine:3.19
|
||||
```
|
||||
|
||||
### 各层面的发行版相关性
|
||||
|
||||
| 层面 | 发行版相关 | 解决方案 |
|
||||
|------|-----------|----------|
|
||||
| DepsSystem(系统包) | ✅ | 容器内 PM |
|
||||
| DepsUser(用户包管理器) | ❌ | 天然无关(cargo/npm/pip 跨发行版一致) |
|
||||
| Bake(构建脚本) | ✅ | 容器内执行 |
|
||||
| Finalize(后处理) | ❌ | 纯产物操作,无关发行版 |
|
||||
|
||||
### UPM 的特殊位置
|
||||
|
||||
用户包管理器(UPM,如 cargo、npm、pip、nix)天然跨发行版:
|
||||
|
||||
```bash
|
||||
cargo install --locked bat # Ubuntu 和 Arch 上都一样
|
||||
npm install -g typescript # Ubuntu 和 Arch 上都一样
|
||||
```
|
||||
|
||||
`depsuser.rs` 处理的正是这个层面。但 `collect_upm_sysdeps()` 需要知道系统 PM——这是 UPM 和系统 PM 之间的 bridge。在设计上,这个 bridge 的 PM 信息应来自环境声明,而非 `detect()`。
|
||||
|
||||
---
|
||||
|
||||
## 6. drop_after 安全模型
|
||||
|
||||
### 决策
|
||||
|
||||
**`drop_after` 指定流水线在哪个 stage 之后从 root 降权到 vulcan 用户。合理的默认值是 `DepsUser`。**
|
||||
|
||||
### Stage 顺序
|
||||
|
||||
```rust
|
||||
pub enum PrebakeStage {
|
||||
Init,
|
||||
Bootstrap,
|
||||
EarlyHook,
|
||||
DepsSystem, // ← 默认
|
||||
DepsUser,
|
||||
LateHook,
|
||||
Ready,
|
||||
Never, // ← 永远不降权
|
||||
}
|
||||
```
|
||||
|
||||
### 各 stage 的特权需求
|
||||
|
||||
| Stage | 需要 root | 原因 |
|
||||
|-------|-----------|------|
|
||||
| Bootstrap | ✅ | 创建系统用户(vulcan)、配置 sudo/doas |
|
||||
| EarlyHook | ❓ | 用户自定义 |
|
||||
| DepsSystem | ✅ | `apt install` 需要 root |
|
||||
| DepsUser | ❌ | `cargo install` 不需要 root |
|
||||
| LateHook | ❓ | 用户自定义 |
|
||||
| Ready / Bake | ❌ | 构建应以非特权用户运行 |
|
||||
|
||||
### 正确的配置
|
||||
|
||||
```yaml
|
||||
security:
|
||||
drop_after: "DepsUser"
|
||||
```
|
||||
|
||||
执行效果:
|
||||
|
||||
```
|
||||
Bootstrap root ← 创建用户
|
||||
EarlyHook root ← hook 以 root 运行
|
||||
DepsSystem root ← apt install -y {packages}
|
||||
DepsUser root ← cargo install {packages}
|
||||
LateHook vulcan ← 降权,用户脚本不能 rm -rf /
|
||||
Ready vulcan
|
||||
Bake vulcan ← 构建以 vulcan 运行
|
||||
```
|
||||
|
||||
### 禁止的配置
|
||||
|
||||
`drop_after` 设置为 `Bootstrap` 或 `EarlyHook` 在当前 stage 顺序下会损坏 DepsStage(需要 root 权限的 `apt install` 在降权后无法执行)。
|
||||
|
||||
**配置验证应拒绝 `drop_after < DepsUser`**(以 root 运行的 user 级包安装没有意义,但 DepsSystem 必须 root)。
|
||||
|
||||
### 特殊值 Never
|
||||
|
||||
```yaml
|
||||
security:
|
||||
drop_after: "Never" # 整条流水线以 root 运行
|
||||
```
|
||||
|
||||
适用场景:Docker 构建、内核模块编译等需要 root 权限的操作。Bootstrap 的 `user: root` 配置也会产生相同效果。
|
||||
|
||||
---
|
||||
|
||||
## 7. 当前代码中的真问题
|
||||
|
||||
以下问题是当前代码中确实存在的,与 bare/daemon 模式无关:
|
||||
|
||||
### 7.1 Apt::adopt 永远返回 false
|
||||
|
||||
```rust
|
||||
// workshop-engine/src/pm/apt.rs:20-23
|
||||
fn adopt(&self, distro: &Info) -> bool {
|
||||
return false; // ← 永远 false
|
||||
// todo!(); // ← 死代码
|
||||
}
|
||||
```
|
||||
|
||||
- `adopt()` 在 `return false` 后无法执行 `todo!()`
|
||||
- Apt 永远不会被 `detect()` 选中
|
||||
- 后果:Debian/Ubuntu 用户在 bare mode 下永远检测不到 PM
|
||||
- 修复方向:移除 return false,实现真正的 adopt 逻辑(Debian/Ubuntu/Mint/Pop 等发行版的匹配)
|
||||
|
||||
### 7.2 PM 覆盖严重不全
|
||||
|
||||
当前 `all_managers()` 仅注册了 Pacman:
|
||||
|
||||
```rust
|
||||
pub fn all_managers() -> Vec<Box<dyn PackageManager>> {
|
||||
vec![
|
||||
// Box::new(apt::Apt), // adopt 坏了
|
||||
Box::new(pacman::Pacman), // 唯一可用
|
||||
// Box::new(dnf::Dnf), // 未实现
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
os_info 3.14.0 定义了 58 种发行版类型,Pacman 仅覆盖其中 9 种(Arch 系)。其余 49 种无 PM 覆盖。
|
||||
|
||||
### 7.3 死代码
|
||||
|
||||
| 位置 | 内容 | 状态 |
|
||||
|------|------|------|
|
||||
| `bake.rs:98` | `fn privileged()` | 定义但未调用 |
|
||||
| `bake.rs:108` | `type TaskResult` | 定义但未使用 |
|
||||
| `bake.rs:110` | `type TaskFn` | 定义但未使用 |
|
||||
| `workshop-baker/src/prebake/stage/environment.rs` | `prepare_environment()` | 定义但未接入执行流 |
|
||||
|
||||
---
|
||||
|
||||
## 8. Daemon 设计原则(汇总)
|
||||
|
||||
以上讨论最终可以提炼为以下设计原则,供 bakerd 重构时参考:
|
||||
|
||||
### P1. 环境声明驱动
|
||||
|
||||
```
|
||||
用户声明 env.image → bakerd 创建环境 → 环境内确定 PM → depssystem 操作 PM
|
||||
```
|
||||
|
||||
PM 的选择权在环境声明,不在运行时探测。`detect()` 不出现在生产路径中。
|
||||
|
||||
### P2. 容器是发行版隔离的边界
|
||||
|
||||
发行版差异在容器边界被隔断。容器内是已知、确定的环境。PM trait 只负责"已知 PM 的操作"(install、update、mirror、repo),不负责"猜是什么 PM"。
|
||||
|
||||
### P3. 降权边界可配置但需校验
|
||||
|
||||
`drop_after` 从 root 降权到 vulcan,默认在 DepsUser 之后。配置验证需确保:降权不会损坏需要 root 的 stage。
|
||||
|
||||
### P4. 最小化 trait 抽象
|
||||
|
||||
PM trait 保留操作系统所需的核心方法。不在 trait 中嵌入分类功能(adopt)。分类是一个数据映射问题,不是多态问题。
|
||||
|
||||
### P5. Bare mode 是开发工具,不承载生产行为
|
||||
|
||||
Bare mode 的行为(host 上直接执行、write static privileged 等)不代表生产设计。不要为了优化 bare mode 而增加生产路径的抽象负担。
|
||||
|
||||
### P6. 天然跨发行版的部分不需要抽象
|
||||
|
||||
UPM(cargo/npm/pip/nix)跨发行版一致,不需要 PM 级别的抽象,不需要 detect,不需要 distro 感知。trait 只应用于确实有 PM-specific 差异的部分。
|
||||
@@ -0,0 +1,88 @@
|
||||
# Resource 模型
|
||||
|
||||
## 核心思想
|
||||
|
||||
整个系统只关心两件事:东西从哪来(fetch),东西到哪去(publish)。
|
||||
|
||||
fetch 方向:URL → 本地路径 → 使用。
|
||||
publish 方向:本地路径 → 变换(插件)→ URL。
|
||||
|
||||
两个方向共用同一套寻址方式,但 Resource 中间态只存在于 fetch 方向。
|
||||
|
||||
## URL 格式
|
||||
|
||||
```
|
||||
getter::url?param1=value1¶m2=value2
|
||||
```
|
||||
|
||||
getter 显式声明意图。不是"我猜这是个 git 仓库",而是"使用者告诉我这是什么"。
|
||||
|
||||
只有三个 getter:
|
||||
|
||||
- `git` — 克隆仓库,必要时 checkout 指定版本
|
||||
- `http` / `https` — 下载文件,必要时校验 checksum
|
||||
- `file` — 复制本地文件或目录
|
||||
|
||||
版本锁定不用 `@v1.0`(`@` 在 URL 里是认证分隔符),用 `?ref=v1.0`。
|
||||
|
||||
没有 `github.com/user/repo` 看起来应该 clone 它这种黑魔法。
|
||||
没有 `.tar.gz` 看起来应该解压它的惊喜。
|
||||
URL 是 URL,行为是行为,两者用 getter:: 显式绑定。
|
||||
|
||||
## Resource 不是网络地址
|
||||
|
||||
流水线阶段(prebake / finalize / notify)只接触 Resource:一个名字 + 一个本地路径,已就绪。它们不经由网络、不解析 URL、不碰 getter。
|
||||
|
||||
所有 fetch 行为在阶段启动前完成:
|
||||
|
||||
- worker 模式:bakerd fetch → 填入 ResourceRegistry → 启动 baker
|
||||
- standalone 模式:baker 自己 fetch → 填入 ResourceRegistry → 执行阶段
|
||||
- bare 模式:不 fetch,传入的必须是 Resource
|
||||
|
||||
ResourceRegistry 是名字到路径的映射。填进去就只读了。
|
||||
|
||||
## getterurl 是独立的 crate
|
||||
|
||||
URL 格式解析 + 基本 fetch 放在 workshop-getterurl,和 baker 本身没有耦合。任何人都可以用。
|
||||
|
||||
crate 分两层:
|
||||
|
||||
```
|
||||
结构层(parser):
|
||||
"git::https://host/repo.git?ref=v1.0" → { getter: "git", url: Url, params: Params }
|
||||
|
||||
获取层(builtin getters):
|
||||
getter.fetch(url, params, dst) → local_path
|
||||
git: git clone + optional checkout
|
||||
http: HTTP GET + optional checksum
|
||||
file: std::fs::copy
|
||||
```
|
||||
|
||||
获取层是字面意义的——不解压、不推断、不猜意图。
|
||||
|
||||
## Checksum
|
||||
|
||||
checksum 作为 URL query 参数传递。三态语义:
|
||||
|
||||
- `?checksum=sha256:abc` — 必须匹配
|
||||
- `?checksum=` — 显式禁用校验
|
||||
- 无 checksum — 如果服务端返回 Content-Digest 头,校验它
|
||||
|
||||
getter 不主动校验,它只执行 URL 里指定的操作。
|
||||
|
||||
## Artifact 不经过 Resource
|
||||
|
||||
artifact 是 bake 的产物,自产生就开始被使用,不存在中间态。它的路径就是 bake 产出的本地路径,不走 ResourceRegistry。
|
||||
|
||||
artifact 的 publish 方向由插件定义行为——可能上传、可能扫描、可能只记录。插件定义最终的行为,只要是自洽的。
|
||||
|
||||
## 三种运行模式
|
||||
|
||||
```
|
||||
fetch 执行者 fetch 在阶段内
|
||||
daemon 模式 bakerd 否
|
||||
standalone baker 自身 否(阶段启动前已完成)
|
||||
bare 模式 无 否(必须传入 Resource)
|
||||
```
|
||||
|
||||
bare 模式下传入的必须是已经就绪的 Resource。外部 URL 是 daemon 或 standalone 的事,bare 不负责。
|
||||
@@ -1,112 +0,0 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
# =============================================================================
|
||||
# HoneyBiscuitWorkshop System Test Runner
|
||||
# =============================================================================
|
||||
# Usage: ./runner.sh [auto|manual] [-- pytest-args...]
|
||||
#
|
||||
# Modes:
|
||||
# auto - Build, run all tests, collect reports, cleanup (default)
|
||||
# manual - Build, start container shell for interactive testing
|
||||
#
|
||||
# Examples:
|
||||
# ./runner.sh # Run all tests automatically
|
||||
# ./runner.sh auto -k test_prebake # Run only prebake tests
|
||||
# ./runner.sh manual # Drop into container shell
|
||||
# =============================================================================
|
||||
|
||||
MODE="${1:-auto}"
|
||||
shift 2>/dev/null || true
|
||||
PYTEST_ARGS="$@"
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
STAGING_DIR="/tmp/hbw-system-test"
|
||||
IMAGE_NAME="hbw-system-test:latest"
|
||||
|
||||
# ---- Cleanup handler ----
|
||||
cleanup() {
|
||||
echo "[runner] Cleaning up..."
|
||||
docker rm -f hbw-system-test-container 2>/dev/null || true
|
||||
rm -rf "$STAGING_DIR"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
# ---- Step 1: Build the binary ----
|
||||
build_binary() {
|
||||
echo "[runner] Building workshop-baker (release)..."
|
||||
cd "$SCRIPT_DIR/workshop-baker"
|
||||
cargo build --release 2>&1
|
||||
echo "[runner] Build complete."
|
||||
}
|
||||
|
||||
# ---- Step 2: Prepare staging directory ----
|
||||
prepare_staging() {
|
||||
echo "[runner] Preparing staging directory..."
|
||||
rm -rf "$STAGING_DIR"
|
||||
mkdir -p "$STAGING_DIR"
|
||||
|
||||
# Copy binary
|
||||
cp "$SCRIPT_DIR/workshop-baker/target/release/workshop-baker" "$STAGING_DIR/"
|
||||
chmod +x "$STAGING_DIR/workshop-baker"
|
||||
|
||||
# Copy examples if they exist
|
||||
if [ -d "$SCRIPT_DIR/workshop-baker/examples" ]; then
|
||||
cp -r "$SCRIPT_DIR/workshop-baker/examples" "$STAGING_DIR/examples"
|
||||
else
|
||||
mkdir -p "$STAGING_DIR/examples"
|
||||
fi
|
||||
|
||||
# Copy llm.env if it exists
|
||||
if [ -f "$SCRIPT_DIR/llm.env" ]; then
|
||||
cp "$SCRIPT_DIR/llm.env" "$STAGING_DIR/llm.env"
|
||||
fi
|
||||
|
||||
echo "[runner] Staging ready at $STAGING_DIR"
|
||||
}
|
||||
|
||||
# ---- Step 3: Build Docker image ----
|
||||
build_image() {
|
||||
echo "[runner] Building Docker image: $IMAGE_NAME ..."
|
||||
docker build -t "$IMAGE_NAME" -f "$SCRIPT_DIR/tests/Dockerfile" "$SCRIPT_DIR"
|
||||
echo "[runner] Image built: $IMAGE_NAME"
|
||||
}
|
||||
|
||||
# ---- Step 4: Run container ----
|
||||
run_container() {
|
||||
local mode="$1"
|
||||
echo "[runner] Starting container in $mode mode..."
|
||||
|
||||
mkdir -p "$SCRIPT_DIR/tests/results"
|
||||
|
||||
if [ "$mode" = "manual" ]; then
|
||||
docker run -it --rm \
|
||||
--name hbw-system-test-container \
|
||||
--privileged \
|
||||
-v "$STAGING_DIR:/workshop:ro" \
|
||||
-v "$SCRIPT_DIR/tests/results:/results" \
|
||||
-e TEST_MODE=manual \
|
||||
"$IMAGE_NAME"
|
||||
else
|
||||
docker run --rm \
|
||||
--name hbw-system-test-container \
|
||||
--privileged \
|
||||
-v "$STAGING_DIR:/workshop:ro" \
|
||||
-v "$SCRIPT_DIR/tests/results:/results" \
|
||||
-e TEST_MODE=auto \
|
||||
"$IMAGE_NAME" $PYTEST_ARGS
|
||||
fi
|
||||
}
|
||||
|
||||
# ---- Main flow ----
|
||||
echo "============================================"
|
||||
echo " HoneyBiscuitWorkshop System Test Runner"
|
||||
echo " Mode: $MODE"
|
||||
echo "============================================"
|
||||
|
||||
build_binary
|
||||
prepare_staging
|
||||
build_image
|
||||
run_container "$MODE"
|
||||
|
||||
echo "[runner] Done."
|
||||
@@ -1,32 +0,0 @@
|
||||
FROM archlinux:latest
|
||||
|
||||
RUN sed -i 's|https://geo.mirror.pkgbuild.com|https://mirrors.tuna.tsinghua.edu.cn/archlinux|g' /etc/pacman.d/mirrorlist && \
|
||||
sed -i 's|https://mirror.rackspace.com/archlinux|https://mirrors.tuna.tsinghua.edu.cn/archlinux|g' /etc/pacman.d/mirrorlist && \
|
||||
echo 'Server = https://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch' >> /etc/pacman.d/mirrorlist
|
||||
|
||||
RUN pacman -Syu --noconfirm && \
|
||||
pacman -S --noconfirm \
|
||||
python python-pip base-devel git curl wget sudo \
|
||||
which procps-ng shadow && \
|
||||
pacman -Scc --noconfirm
|
||||
|
||||
# Create test user
|
||||
RUN useradd -m -s /bin/bash testuser
|
||||
|
||||
# Python environment
|
||||
WORKDIR /app
|
||||
COPY tests/requirements.txt ./requirements.txt
|
||||
RUN python -m venv /opt/test-venv && \
|
||||
/opt/test-venv/bin/pip install -r requirements.txt
|
||||
|
||||
# Copy test suite
|
||||
COPY tests/ ./tests/
|
||||
COPY tests/entrypoint.sh ./entrypoint.sh
|
||||
RUN chmod +x entrypoint.sh
|
||||
|
||||
# Environment
|
||||
ENV PATH="/opt/test-venv/bin:$PATH"
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
ENV PYTHONDONTWRITEBYTECODE=1
|
||||
|
||||
ENTRYPOINT ["./entrypoint.sh"]
|
||||
@@ -1,164 +0,0 @@
|
||||
"""
|
||||
Root conftest for HoneyBiscuitWorkshop system tests.
|
||||
|
||||
Provides core fixtures for:
|
||||
- Locating the baker binary and examples
|
||||
- Running subprocess commands with output capture
|
||||
- Managing temporary working directories
|
||||
"""
|
||||
import os
|
||||
import subprocess
|
||||
import shutil
|
||||
import tempfile
|
||||
import pytest
|
||||
|
||||
|
||||
def pytest_configure(config):
|
||||
"""Register custom markers."""
|
||||
config.addinivalue_line("markers", "prebake: prebake stage tests")
|
||||
config.addinivalue_line("markers", "bake: bake stage tests")
|
||||
config.addinivalue_line("markers", "finalize: finalize stage tests")
|
||||
config.addinivalue_line("markers", "llm: tests requiring LLM API")
|
||||
config.addinivalue_line("markers", "slow: slow running tests")
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def baker_bin():
|
||||
"""Path to the workshop-baker binary."""
|
||||
path = os.environ.get("BAKER_BIN", "/workshop/workshop-baker")
|
||||
if not os.path.isfile(path):
|
||||
pytest.skip(f"Baker binary not found at {path}")
|
||||
if not os.access(path, os.X_OK):
|
||||
pytest.skip(f"Baker binary not executable at {path}")
|
||||
return path
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def examples_dir():
|
||||
"""Path to the examples directory (may be None)."""
|
||||
path = os.environ.get("EXAMPLES_DIR", "/workshop/examples")
|
||||
if not os.path.isdir(path):
|
||||
return None
|
||||
return path
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def llm_env():
|
||||
"""LLM environment variables dict (from llm.env)."""
|
||||
env = {}
|
||||
for var in [
|
||||
"OPENAI_BASE_URL",
|
||||
"OPENAI_API_KEY",
|
||||
"ANTHROPIC_BASE_URL",
|
||||
"ANTHROPIC_API_KEY",
|
||||
"MODEL",
|
||||
"REASONING_EFFORT",
|
||||
]:
|
||||
val = os.environ.get(var)
|
||||
if val:
|
||||
env[var] = val
|
||||
return env
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def work_dir():
|
||||
"""Create a temporary working directory for a test, cleaned up after."""
|
||||
d = tempfile.mkdtemp(prefix="hbw-test-")
|
||||
yield d
|
||||
shutil.rmtree(d, ignore_errors=True)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def run_cmd():
|
||||
"""Fixture that provides a command runner function.
|
||||
|
||||
Returns a function that runs a command and returns
|
||||
(stdout, stderr, returncode, success).
|
||||
"""
|
||||
|
||||
def _run(args, cwd=None, env=None, timeout=120):
|
||||
"""Run a command and capture output.
|
||||
|
||||
Args:
|
||||
args: Command as list of strings.
|
||||
cwd: Working directory.
|
||||
env: Environment variables (merged with os.environ).
|
||||
timeout: Timeout in seconds.
|
||||
|
||||
Returns:
|
||||
dict with keys: stdout, stderr, returncode, success
|
||||
"""
|
||||
full_env = {**os.environ}
|
||||
if env:
|
||||
full_env.update(env)
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
args,
|
||||
cwd=cwd,
|
||||
env=full_env,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=timeout,
|
||||
)
|
||||
return {
|
||||
"stdout": result.stdout,
|
||||
"stderr": result.stderr,
|
||||
"returncode": result.returncode,
|
||||
"success": result.returncode == 0,
|
||||
}
|
||||
except subprocess.TimeoutExpired:
|
||||
return {
|
||||
"stdout": "",
|
||||
"stderr": f"Command timed out after {timeout}s",
|
||||
"returncode": -1,
|
||||
"success": False,
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"stdout": "",
|
||||
"stderr": str(e),
|
||||
"returncode": -1,
|
||||
"success": False,
|
||||
}
|
||||
|
||||
return _run
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def run_baker(baker_bin, run_cmd):
|
||||
"""Fixture that provides a baker command runner.
|
||||
|
||||
Returns a function that runs workshop-baker with common flags.
|
||||
"""
|
||||
|
||||
def _run(subcommand, *args, cwd=None, env=None, timeout=120):
|
||||
cmd = [
|
||||
baker_bin,
|
||||
"-u", "testuser",
|
||||
"-p", "test-pipeline",
|
||||
"-b", "test-build-001",
|
||||
"bare",
|
||||
subcommand,
|
||||
]
|
||||
cmd.extend(args)
|
||||
return run_cmd(cmd, cwd=cwd, env=env, timeout=timeout)
|
||||
|
||||
return _run
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def fixtures_dir():
|
||||
"""Path to the test fixtures directory."""
|
||||
return os.path.join(os.path.dirname(__file__), "fixtures")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def bake_workspace():
|
||||
"""Create /workspace for bake tests (baker writes temp scripts there)."""
|
||||
existed = os.path.exists("/workspace")
|
||||
if not existed:
|
||||
os.makedirs("/workspace", exist_ok=True)
|
||||
yield
|
||||
if not existed:
|
||||
shutil.rmtree("/workspace", ignore_errors=True)
|
||||
@@ -1,36 +0,0 @@
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
# Load LLM environment if available
|
||||
if [ -f /workshop/llm.env ]; then
|
||||
set -a
|
||||
source /workshop/llm.env
|
||||
set +a
|
||||
fi
|
||||
|
||||
# Export test configuration
|
||||
export BAKER_BIN=/workshop/workshop-baker
|
||||
export EXAMPLES_DIR=/workshop/examples
|
||||
export TEST_MODE=${TEST_MODE:-auto}
|
||||
|
||||
if [ "$TEST_MODE" = "manual" ]; then
|
||||
echo "========================================"
|
||||
echo " HoneyBiscuitWorkshop System Tests"
|
||||
echo " Mode: MANUAL (interactive shell)"
|
||||
echo "========================================"
|
||||
echo ""
|
||||
echo "Binary: $BAKER_BIN"
|
||||
echo "Examples: $EXAMPLES_DIR"
|
||||
echo ""
|
||||
echo "Run tests manually:"
|
||||
echo " cd /app && pytest tests/ -v"
|
||||
echo ""
|
||||
exec /bin/bash
|
||||
else
|
||||
echo "========================================"
|
||||
echo " HoneyBiscuitWorkshop System Tests"
|
||||
echo " Mode: AUTO (running pytest)"
|
||||
echo "========================================"
|
||||
cd /app
|
||||
exec python -m pytest tests/ -v --tb=short --junitxml=/results/report.xml "$@"
|
||||
fi
|
||||
Vendored
-6
@@ -1,6 +0,0 @@
|
||||
#!/bin/bash
|
||||
[ -f /dev/shm/bakeexport.{{ name }} ] && . /dev/shm/bakeexport.{{ name }} || true
|
||||
{{ condition }}
|
||||
{{ main }}
|
||||
{{ name }}
|
||||
{{ export }}
|
||||
Vendored
-25
@@ -1,25 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
# @pipeline
|
||||
step_one() {
|
||||
echo "Step 1: Setup"
|
||||
mkdir -p /tmp/bake-test-output
|
||||
echo "step1-done" > /tmp/bake-test-output/step1.txt
|
||||
}
|
||||
|
||||
# @pipeline
|
||||
step_two() {
|
||||
echo "Step 2: Build"
|
||||
echo "step2-done" > /tmp/bake-test-output/step2.txt
|
||||
}
|
||||
|
||||
# @pipeline
|
||||
step_three() {
|
||||
echo "Step 3: Verify"
|
||||
if [ -f /tmp/bake-test-output/step1.txt ] && [ -f /tmp/bake-test-output/step2.txt ]; then
|
||||
echo "All steps completed successfully"
|
||||
else
|
||||
echo "ERROR: Missing step outputs"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
Vendored
-5
@@ -1,5 +0,0 @@
|
||||
#!/bin/bash
|
||||
echo "Hello from simple bake script"
|
||||
echo "Task ID: $HBW_TASKID"
|
||||
echo "Pipeline: $HBW_PIPELINE"
|
||||
exit 0
|
||||
Vendored
-22
@@ -1,22 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
# @pipeline
|
||||
init_workspace() {
|
||||
echo "Initializing workspace"
|
||||
mkdir -p /tmp/bake-after-test
|
||||
echo "init" > /tmp/bake-after-test/status.txt
|
||||
}
|
||||
|
||||
# @pipeline
|
||||
# @after(init_workspace)
|
||||
fetch_source() {
|
||||
echo "Fetching source (depends on init_workspace)"
|
||||
echo "fetched" > /tmp/bake-after-test/source.txt
|
||||
}
|
||||
|
||||
# @pipeline
|
||||
# @after(fetch_source)
|
||||
build_project() {
|
||||
echo "Building project (depends on fetch_source)"
|
||||
echo "built" > /tmp/bake-after-test/build.txt
|
||||
}
|
||||
-4
@@ -1,4 +0,0 @@
|
||||
#!/bin/bash
|
||||
echo "EARLY_HOOK_EXECUTED"
|
||||
echo "Working directory: $(pwd)"
|
||||
echo "User: $(whoami)"
|
||||
Vendored
-3
@@ -1,3 +0,0 @@
|
||||
#!/bin/bash
|
||||
echo "LATE_HOOK_EXECUTED"
|
||||
echo "Working directory: $(pwd)"
|
||||
Vendored
-6
@@ -1,6 +0,0 @@
|
||||
version: "0.0.1"
|
||||
plugin: {}
|
||||
notification: {}
|
||||
artifact: []
|
||||
cleanup:
|
||||
policy: "auto"
|
||||
-13
@@ -1,13 +0,0 @@
|
||||
version: "0.0.1"
|
||||
plugin: {}
|
||||
notification: {}
|
||||
artifact: []
|
||||
cleanup:
|
||||
policy: "auto"
|
||||
hooks:
|
||||
early:
|
||||
- name: "finalize-early"
|
||||
command: "echo 'finalize early hook'"
|
||||
late:
|
||||
- name: "finalize-late"
|
||||
command: "echo 'finalize late hook'"
|
||||
-8
@@ -1,8 +0,0 @@
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
workspace:
|
||||
path: "/home/vulcan/workspace"
|
||||
fallback: true
|
||||
Vendored
-15
@@ -1,15 +0,0 @@
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
workspace:
|
||||
path: "/home/vulcan/workspace"
|
||||
fallback: true
|
||||
hooks:
|
||||
early:
|
||||
- name: "test-early-hook"
|
||||
command: "echo 'early hook executed'"
|
||||
late:
|
||||
- name: "test-late-hook"
|
||||
command: "echo 'late hook executed'"
|
||||
-3
@@ -1,3 +0,0 @@
|
||||
version: "0.5"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
Vendored
-8
@@ -1,8 +0,0 @@
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
workspace:
|
||||
path: "/home/vulcan/workspace"
|
||||
fallback: true
|
||||
@@ -1,12 +0,0 @@
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
workspace:
|
||||
path: "/home/vulcan/workspace"
|
||||
fallback: true
|
||||
custom: |
|
||||
#!/bin/sh
|
||||
echo 'custom bootstrap executed'
|
||||
mkdir -p /custom-workspace
|
||||
-16
@@ -1,16 +0,0 @@
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
workspace:
|
||||
path: "/home/vulcan/workspace"
|
||||
fallback: true
|
||||
dependencies:
|
||||
system:
|
||||
pacman:
|
||||
packages:
|
||||
- "which"
|
||||
user:
|
||||
cargo:
|
||||
- "ripgrep"
|
||||
-13
@@ -1,13 +0,0 @@
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
workspace:
|
||||
path: "/home/vulcan/workspace"
|
||||
fallback: true
|
||||
envvars:
|
||||
prebake:
|
||||
TEST_PREBAKE_VAR: "prebake_value"
|
||||
bake:
|
||||
TEST_BAKE_VAR: "bake_value"
|
||||
-17
@@ -1,17 +0,0 @@
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
workspace:
|
||||
path: "/home/vulcan/workspace"
|
||||
fallback: true
|
||||
hooks:
|
||||
early:
|
||||
- name: "test-early-hook"
|
||||
command: "echo 'early hook executed'"
|
||||
working_dir: "/workspace"
|
||||
late:
|
||||
- name: "test-late-hook"
|
||||
command: "echo 'late hook executed'"
|
||||
working_dir: "/workspace"
|
||||
-14
@@ -1,14 +0,0 @@
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
workspace:
|
||||
path: "/home/vulcan/workspace"
|
||||
fallback: true
|
||||
dependencies:
|
||||
system:
|
||||
pacman:
|
||||
packages:
|
||||
- "curl"
|
||||
- "wget"
|
||||
-10
@@ -1,10 +0,0 @@
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
workspace:
|
||||
path: "/home/vulcan/workspace"
|
||||
fallback: true
|
||||
security:
|
||||
drop_after: "Bootstrap"
|
||||
@@ -1,12 +0,0 @@
|
||||
"""
|
||||
LLM-based log judgment system for system tests.
|
||||
|
||||
Uses LLM APIs to evaluate non-deterministic test outputs (logs, stderr, etc.)
|
||||
where simple string matching is insufficient.
|
||||
"""
|
||||
|
||||
from .client import LLMClient
|
||||
from .cache import LLMCache
|
||||
from .judge import judge_log, JudgmentResult
|
||||
|
||||
__all__ = ["LLMClient", "LLMCache", "judge_log", "JudgmentResult"]
|
||||
@@ -1,56 +0,0 @@
|
||||
"""
|
||||
Disk-based cache for LLM API responses.
|
||||
|
||||
Uses SHA256 of (prompt + model + system_prompt) as cache key.
|
||||
Cache files stored as JSON in ``.cache/`` directory.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
class LLMCache:
|
||||
"""SHA256-based disk cache for LLM API responses."""
|
||||
|
||||
def __init__(self, cache_dir: str | None = None):
|
||||
if cache_dir is None:
|
||||
import os
|
||||
|
||||
cache_dir = os.path.join(os.path.dirname(__file__), ".cache")
|
||||
self.cache_dir = Path(cache_dir)
|
||||
self.cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def _make_key(self, prompt: str, model: str, system_prompt: str = "") -> str:
|
||||
"""Create deterministic cache key from request parameters."""
|
||||
data = json.dumps(
|
||||
{"prompt": prompt, "model": model, "system": system_prompt},
|
||||
sort_keys=True,
|
||||
)
|
||||
return hashlib.sha256(data.encode()).hexdigest()
|
||||
|
||||
def get(self, prompt: str, model: str, system_prompt: str = "") -> dict | None:
|
||||
"""Retrieve cached response if exists."""
|
||||
key = self._make_key(prompt, model, system_prompt)
|
||||
path = self.cache_dir / f"{key}.json"
|
||||
if path.exists():
|
||||
try:
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
except (json.JSONDecodeError, IOError):
|
||||
return None
|
||||
return None
|
||||
|
||||
def set(
|
||||
self, prompt: str, model: str, system_prompt: str, response: dict
|
||||
) -> None:
|
||||
"""Store response in cache."""
|
||||
key = self._make_key(prompt, model, system_prompt)
|
||||
path = self.cache_dir / f"{key}.json"
|
||||
with open(path, "w") as f:
|
||||
json.dump(response, f, ensure_ascii=False)
|
||||
|
||||
def clear(self) -> None:
|
||||
"""Clear all cached responses."""
|
||||
for f in self.cache_dir.glob("*.json"):
|
||||
f.unlink()
|
||||
@@ -1,187 +0,0 @@
|
||||
"""
|
||||
LLM client supporting both Anthropic and OpenAI APIs.
|
||||
|
||||
Prefers Anthropic format (per project convention).
|
||||
Falls back to OpenAI format if Anthropic is unavailable.
|
||||
"""
|
||||
|
||||
import os
|
||||
import logging
|
||||
from .cache import LLMCache
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class LLMClient:
|
||||
"""LLM client with caching support. Prefers Anthropic API."""
|
||||
|
||||
def __init__(self, cache: LLMCache | None = None):
|
||||
self.cache = cache or LLMCache()
|
||||
self._anthropic_client = None
|
||||
self._openai_client = None
|
||||
self._init_clients()
|
||||
|
||||
def _init_clients(self):
|
||||
"""Initialize API clients based on available environment variables."""
|
||||
# Anthropic (preferred)
|
||||
anthropic_key = os.environ.get("ANTHROPIC_API_KEY")
|
||||
anthropic_base = os.environ.get("ANTHROPIC_BASE_URL")
|
||||
if anthropic_key:
|
||||
try:
|
||||
import anthropic
|
||||
|
||||
kwargs: dict = {"api_key": anthropic_key}
|
||||
if anthropic_base:
|
||||
kwargs["base_url"] = anthropic_base
|
||||
self._anthropic_client = anthropic.Anthropic(**kwargs)
|
||||
logger.info("Anthropic client initialized")
|
||||
except ImportError:
|
||||
logger.warning("anthropic package not installed")
|
||||
|
||||
# OpenAI (fallback)
|
||||
openai_key = os.environ.get("OPENAI_API_KEY")
|
||||
openai_base = os.environ.get("OPENAI_BASE_URL")
|
||||
if openai_key:
|
||||
try:
|
||||
import openai
|
||||
|
||||
kwargs = {"api_key": openai_key}
|
||||
if openai_base:
|
||||
kwargs["base_url"] = openai_base
|
||||
self._openai_client = openai.OpenAI(**kwargs)
|
||||
logger.info("OpenAI client initialized")
|
||||
except ImportError:
|
||||
logger.warning("openai package not installed")
|
||||
|
||||
@property
|
||||
def available(self) -> bool:
|
||||
"""Check if any LLM client is available."""
|
||||
return (
|
||||
self._anthropic_client is not None or self._openai_client is not None
|
||||
)
|
||||
|
||||
def complete(
|
||||
self, prompt: str, system_prompt: str = "", max_tokens: int = 2000
|
||||
) -> str:
|
||||
"""Send a completion request, with caching.
|
||||
|
||||
Args:
|
||||
prompt: User message.
|
||||
system_prompt: System message (sets role / persona).
|
||||
max_tokens: Max response tokens.
|
||||
|
||||
Returns:
|
||||
LLM response text.
|
||||
|
||||
Raises:
|
||||
RuntimeError: If no client is available.
|
||||
"""
|
||||
model = os.environ.get("MODEL", "deepseek-v4-flash")
|
||||
|
||||
# Check cache
|
||||
cached = self.cache.get(prompt, model, system_prompt)
|
||||
if cached:
|
||||
logger.info("Cache hit for LLM request")
|
||||
return cached["content"]
|
||||
|
||||
# Try Anthropic first
|
||||
if self._anthropic_client:
|
||||
content = self._call_anthropic(
|
||||
prompt, system_prompt, model, max_tokens
|
||||
)
|
||||
elif self._openai_client:
|
||||
content = self._call_openai(prompt, system_prompt, model, max_tokens)
|
||||
else:
|
||||
raise RuntimeError(
|
||||
"No LLM client available. "
|
||||
"Set ANTHROPIC_API_KEY or OPENAI_API_KEY."
|
||||
)
|
||||
|
||||
# Cache the response
|
||||
self.cache.set(prompt, model, system_prompt, {"content": content})
|
||||
return content
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Private helpers
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _call_anthropic(
|
||||
self, prompt: str, system_prompt: str, model: str, max_tokens: int
|
||||
) -> str:
|
||||
"""Call Anthropic API."""
|
||||
kwargs: dict = {
|
||||
"model": model,
|
||||
"max_tokens": max_tokens,
|
||||
"messages": [{"role": "user", "content": prompt}],
|
||||
}
|
||||
if system_prompt:
|
||||
kwargs["system"] = system_prompt
|
||||
|
||||
# Only add reasoning_effort if the model supports it
|
||||
reasoning_effort = os.environ.get("REASONING_EFFORT")
|
||||
if reasoning_effort:
|
||||
try:
|
||||
kwargs["reasoning_effort"] = reasoning_effort
|
||||
response = self._anthropic_client.messages.create(**kwargs)
|
||||
except Exception as e:
|
||||
if (
|
||||
"reasoning_effort" in str(e).lower()
|
||||
or "unexpected" in str(e).lower()
|
||||
):
|
||||
del kwargs["reasoning_effort"]
|
||||
response = self._anthropic_client.messages.create(**kwargs)
|
||||
else:
|
||||
raise
|
||||
else:
|
||||
response = self._anthropic_client.messages.create(**kwargs)
|
||||
|
||||
# Extract text from response content blocks
|
||||
if hasattr(response, "content") and response.content:
|
||||
block = response.content[0]
|
||||
if hasattr(block, "text"):
|
||||
return block.text
|
||||
if hasattr(block, "type") and block.type == "thinking":
|
||||
for b in response.content:
|
||||
if hasattr(b, "text"):
|
||||
return b.text
|
||||
return str(block)
|
||||
return str(response)
|
||||
|
||||
def _call_openai(
|
||||
self, prompt: str, system_prompt: str, model: str, max_tokens: int
|
||||
) -> str:
|
||||
"""Call OpenAI-compatible API."""
|
||||
messages: list[dict] = []
|
||||
if system_prompt:
|
||||
messages.append({"role": "system", "content": system_prompt})
|
||||
messages.append({"role": "user", "content": prompt})
|
||||
|
||||
kwargs: dict = {
|
||||
"model": model,
|
||||
"messages": messages,
|
||||
"max_tokens": max_tokens,
|
||||
}
|
||||
|
||||
# Only add reasoning_effort if supported
|
||||
reasoning_effort = os.environ.get("REASONING_EFFORT")
|
||||
if reasoning_effort:
|
||||
try:
|
||||
kwargs["reasoning_effort"] = reasoning_effort
|
||||
response = self._openai_client.chat.completions.create(
|
||||
**kwargs
|
||||
)
|
||||
except Exception as e:
|
||||
if (
|
||||
"reasoning_effort" in str(e).lower()
|
||||
or "unexpected" in str(e).lower()
|
||||
):
|
||||
del kwargs["reasoning_effort"]
|
||||
response = self._openai_client.chat.completions.create(
|
||||
**kwargs
|
||||
)
|
||||
else:
|
||||
raise
|
||||
else:
|
||||
response = self._openai_client.chat.completions.create(**kwargs)
|
||||
|
||||
return response.choices[0].message.content
|
||||
@@ -1,149 +0,0 @@
|
||||
"""
|
||||
LLM-based log judgment for system tests.
|
||||
|
||||
Evaluates logs against expected behavior using LLM semantic analysis.
|
||||
Returns structured pass/fail results with reasoning.
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
from .client import LLMClient
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
JUDGE_SYSTEM_PROMPT = """\
|
||||
You are a test result judge for a CI/CD system called HoneyBiscuitWorkshop.
|
||||
|
||||
Your job: Evaluate whether actual command output/logs match expected behavior.
|
||||
|
||||
RULES:
|
||||
1. You MUST respond with valid JSON in this exact format:
|
||||
{"verdict": "pass" | "fail", "reason": "brief explanation"}
|
||||
2. "pass" = the output confirms the expected behavior occurred
|
||||
3. "fail" = the output shows the expected behavior did NOT occur, or shows an error
|
||||
4. Ignore irrelevant noise (timestamps, debug lines, unrelated warnings)
|
||||
5. Focus ONLY on whether the specific expected behavior is confirmed or contradicted
|
||||
6. Be strict: if the evidence is ambiguous, verdict is "fail"
|
||||
7. Do NOT include any text outside the JSON object"""
|
||||
|
||||
|
||||
@dataclass
|
||||
class JudgmentResult:
|
||||
"""Result of an LLM log judgment."""
|
||||
|
||||
verdict: str # "pass" or "fail"
|
||||
reason: str
|
||||
raw_response: str
|
||||
|
||||
@property
|
||||
def passed(self) -> bool:
|
||||
return self.verdict == "pass"
|
||||
|
||||
|
||||
def _extract_relevant_logs(full_log: str, max_chars: int = 4000) -> str:
|
||||
"""Extract relevant portions from logs, removing noise.
|
||||
|
||||
Keeps the last *max_chars* characters (most recent / relevant).
|
||||
Removes common noise patterns.
|
||||
"""
|
||||
lines = full_log.strip().split("\n")
|
||||
|
||||
noise_patterns = [
|
||||
r"^\d{4}-\d{2}-\d{2}T",
|
||||
r"^DEBUG ",
|
||||
r"^TRACE ",
|
||||
]
|
||||
filtered = [
|
||||
line
|
||||
for line in lines
|
||||
if not any(re.match(p, line) for p in noise_patterns)
|
||||
]
|
||||
|
||||
text = "\n".join(filtered)
|
||||
if len(text) > max_chars:
|
||||
text = "...\n" + text[-max_chars:]
|
||||
return text
|
||||
|
||||
|
||||
def judge_log(
|
||||
operation: str,
|
||||
expected_behavior: str,
|
||||
actual_log: str,
|
||||
client: LLMClient | None = None,
|
||||
) -> JudgmentResult:
|
||||
"""Judge whether actual logs match expected behavior.
|
||||
|
||||
Args:
|
||||
operation: Description of what was attempted
|
||||
(e.g. ``"prebake bootstrap stage"``).
|
||||
expected_behavior: What should have happened
|
||||
(e.g. ``"user vulcan was created"``).
|
||||
actual_log: The actual stdout / stderr output.
|
||||
client: ``LLMClient`` instance (creates default if ``None``).
|
||||
|
||||
Returns:
|
||||
``JudgmentResult`` with verdict and reason.
|
||||
"""
|
||||
if client is None:
|
||||
client = LLMClient()
|
||||
|
||||
if not client.available:
|
||||
logger.warning("No LLM client available, returning fail verdict")
|
||||
return JudgmentResult(
|
||||
verdict="fail",
|
||||
reason="No LLM client available for judgment",
|
||||
raw_response="",
|
||||
)
|
||||
|
||||
relevant_log = _extract_relevant_logs(actual_log)
|
||||
|
||||
prompt = f"""\
|
||||
## Operation
|
||||
{operation}
|
||||
|
||||
## Expected Behavior
|
||||
{expected_behavior}
|
||||
|
||||
## Actual Output (relevant excerpts)
|
||||
```
|
||||
{relevant_log}
|
||||
```
|
||||
|
||||
Evaluate: Does the actual output confirm the expected behavior occurred?
|
||||
Respond with JSON only."""
|
||||
|
||||
try:
|
||||
response = client.complete(
|
||||
prompt=prompt,
|
||||
system_prompt=JUDGE_SYSTEM_PROMPT,
|
||||
max_tokens=500,
|
||||
)
|
||||
|
||||
# Parse JSON response — handle potential markdown code blocks
|
||||
text = response.strip()
|
||||
if text.startswith("```"):
|
||||
lines = text.split("\n")
|
||||
text = "\n".join(lines[1:-1]) if len(lines) > 2 else text
|
||||
|
||||
result = json.loads(text)
|
||||
return JudgmentResult(
|
||||
verdict=result.get("verdict", "fail"),
|
||||
reason=result.get("reason", "No reason provided"),
|
||||
raw_response=response,
|
||||
)
|
||||
except json.JSONDecodeError as e:
|
||||
logger.error(f"Failed to parse LLM response as JSON: {e}")
|
||||
return JudgmentResult(
|
||||
verdict="fail",
|
||||
reason=f"LLM response was not valid JSON: {response[:200]}",
|
||||
raw_response=response,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"LLM judgment failed: {e}")
|
||||
return JudgmentResult(
|
||||
verdict="fail",
|
||||
reason=f"LLM call failed: {str(e)}",
|
||||
raw_response="",
|
||||
)
|
||||
@@ -1,34 +0,0 @@
|
||||
"""
|
||||
Environment probes for verifying system state in workshop-baker tests.
|
||||
|
||||
Each probe is a standalone function that can be called from any test.
|
||||
Probes return bool or raise AssertionError with descriptive message.
|
||||
"""
|
||||
|
||||
from .user import user_exists, user_uid, user_gid, user_in_group, user_shell
|
||||
from .file import (
|
||||
file_exists,
|
||||
dir_exists,
|
||||
file_permissions,
|
||||
file_contains,
|
||||
file_owned_by,
|
||||
)
|
||||
from .package import package_installed, package_version
|
||||
from .process import process_running, process_count
|
||||
|
||||
__all__ = [
|
||||
"user_exists",
|
||||
"user_uid",
|
||||
"user_gid",
|
||||
"user_in_group",
|
||||
"user_shell",
|
||||
"file_exists",
|
||||
"dir_exists",
|
||||
"file_permissions",
|
||||
"file_contains",
|
||||
"file_owned_by",
|
||||
"package_installed",
|
||||
"package_version",
|
||||
"process_running",
|
||||
"process_count",
|
||||
]
|
||||
@@ -1,43 +0,0 @@
|
||||
"""File and directory existence / permission probes."""
|
||||
|
||||
import os
|
||||
import stat
|
||||
import grp
|
||||
import pwd
|
||||
|
||||
|
||||
def file_exists(path: str) -> bool:
|
||||
"""Check if *path* exists and is a regular file."""
|
||||
return os.path.isfile(path)
|
||||
|
||||
|
||||
def dir_exists(path: str) -> bool:
|
||||
"""Check if *path* exists and is a directory."""
|
||||
return os.path.isdir(path)
|
||||
|
||||
|
||||
def file_permissions(path: str) -> int:
|
||||
"""Return the permission mode of *path* as an octal int (e.g. ``0o755``).
|
||||
|
||||
Raises ``FileNotFoundError`` if *path* doesn't exist.
|
||||
"""
|
||||
return stat.S_IMODE(os.stat(path).st_mode)
|
||||
|
||||
|
||||
def file_contains(path: str, text: str) -> bool:
|
||||
"""Check if a file contains the specified text."""
|
||||
with open(path, "r", errors="replace") as f:
|
||||
return text in f.read()
|
||||
|
||||
|
||||
def file_owned_by(path: str, username: str) -> bool:
|
||||
"""Check if a file is owned by the specified user."""
|
||||
st = os.stat(path)
|
||||
owner = pwd.getpwuid(st.st_uid).pw_name
|
||||
return owner == username
|
||||
|
||||
|
||||
def file_group(path: str) -> str:
|
||||
"""Get the group owner of a file."""
|
||||
st = os.stat(path)
|
||||
return grp.getgrgid(st.st_gid).gr_name
|
||||
@@ -1,34 +0,0 @@
|
||||
"""Package installation probes for Arch Linux (pacman)."""
|
||||
|
||||
import subprocess
|
||||
|
||||
|
||||
def package_installed(package_name: str) -> bool:
|
||||
"""Check if a pacman package is installed."""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["pacman", "-Q", package_name],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
return result.returncode == 0
|
||||
except FileNotFoundError:
|
||||
return False
|
||||
|
||||
|
||||
def package_version(package_name: str) -> str | None:
|
||||
"""Get installed version of a package, or ``None`` if not installed."""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["pacman", "-Q", package_name],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
if result.returncode == 0:
|
||||
# Output format: "package_name version"
|
||||
parts = result.stdout.strip().split()
|
||||
if len(parts) >= 2:
|
||||
return parts[1]
|
||||
return None
|
||||
except FileNotFoundError:
|
||||
return None
|
||||
@@ -1,44 +0,0 @@
|
||||
"""Process existence probes."""
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
|
||||
|
||||
def process_running(process_name: str) -> bool:
|
||||
"""Check if a process with the given name is running."""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["pgrep", "-x", process_name],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
return result.returncode == 0
|
||||
except FileNotFoundError:
|
||||
# Fall back to /proc scan if pgrep not available
|
||||
try:
|
||||
for pid_dir in os.listdir("/proc"):
|
||||
if pid_dir.isdigit():
|
||||
try:
|
||||
with open(f"/proc/{pid_dir}/comm") as f:
|
||||
if f.read().strip() == process_name:
|
||||
return True
|
||||
except (IOError, OSError):
|
||||
continue
|
||||
except OSError:
|
||||
pass
|
||||
return False
|
||||
|
||||
|
||||
def process_count(process_name: str) -> int:
|
||||
"""Count processes with the given name."""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["pgrep", "-c", "-x", process_name],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
if result.returncode == 0:
|
||||
return int(result.stdout.strip())
|
||||
return 0
|
||||
except (FileNotFoundError, ValueError):
|
||||
return 0
|
||||
@@ -1,37 +0,0 @@
|
||||
"""User existence and attribute probes."""
|
||||
|
||||
import pwd
|
||||
import grp
|
||||
|
||||
|
||||
def user_exists(username: str) -> bool:
|
||||
"""Check if a system user exists."""
|
||||
try:
|
||||
pwd.getpwnam(username)
|
||||
return True
|
||||
except KeyError:
|
||||
return False
|
||||
|
||||
|
||||
def user_uid(username: str) -> int:
|
||||
"""Get the UID of a user. Raises KeyError if user doesn't exist."""
|
||||
return pwd.getpwnam(username).pw_uid
|
||||
|
||||
|
||||
def user_gid(username: str) -> int:
|
||||
"""Get the primary GID of a user. Raises KeyError if user doesn't exist."""
|
||||
return pwd.getpwnam(username).pw_gid
|
||||
|
||||
|
||||
def user_in_group(username: str, groupname: str) -> bool:
|
||||
"""Check if a user is a member of a group."""
|
||||
try:
|
||||
group = grp.getgrnam(groupname)
|
||||
return username in group.gr_mem
|
||||
except KeyError:
|
||||
return False
|
||||
|
||||
|
||||
def user_shell(username: str) -> str:
|
||||
"""Get the login shell of a user. Raises KeyError if user doesn't exist."""
|
||||
return pwd.getpwnam(username).pw_shell
|
||||
@@ -1,6 +0,0 @@
|
||||
pytest>=8.0.0
|
||||
pytest-timeout>=2.2.0
|
||||
pytest-xdist>=3.5.0
|
||||
anthropic>=0.40.0,<0.90.0
|
||||
openai>=1.50.0,<2.0.0
|
||||
pyyaml>=6.0
|
||||
@@ -1,129 +0,0 @@
|
||||
import os
|
||||
import pytest
|
||||
|
||||
|
||||
FIXTURES = os.path.join(os.path.dirname(__file__), "fixtures", "bake")
|
||||
|
||||
|
||||
@pytest.mark.bake
|
||||
class TestBakeSimpleScript:
|
||||
|
||||
def test_simple_script_runs(self, run_baker, bake_workspace, work_dir):
|
||||
prebake_cfg = os.path.join(
|
||||
os.path.dirname(__file__), "fixtures", "prebake", "minimal.yml"
|
||||
)
|
||||
script = os.path.join(FIXTURES, "simple.sh")
|
||||
bake_base = os.path.join(FIXTURES, "bake_base.sh")
|
||||
|
||||
result = run_baker(
|
||||
"bake", script,
|
||||
"--bake-base", bake_base,
|
||||
"--prebake", prebake_cfg,
|
||||
cwd=work_dir,
|
||||
timeout=60,
|
||||
)
|
||||
assert result["success"], f"Bake failed: {result['stderr']}"
|
||||
assert "Hello from simple bake script" in result["stdout"]
|
||||
|
||||
def test_simple_script_sets_env_vars(self, run_baker, bake_workspace, work_dir):
|
||||
prebake_cfg = os.path.join(
|
||||
os.path.dirname(__file__), "fixtures", "prebake", "minimal.yml"
|
||||
)
|
||||
script = os.path.join(FIXTURES, "simple.sh")
|
||||
bake_base = os.path.join(FIXTURES, "bake_base.sh")
|
||||
|
||||
result = run_baker(
|
||||
"bake", script,
|
||||
"--bake-base", bake_base,
|
||||
"--prebake", prebake_cfg,
|
||||
cwd=work_dir,
|
||||
timeout=60,
|
||||
)
|
||||
assert result["success"]
|
||||
assert "test-pipeline" in result["stdout"]
|
||||
|
||||
|
||||
@pytest.mark.bake
|
||||
class TestBakePipeline:
|
||||
|
||||
def test_pipeline_functions_execute(self, run_baker, bake_workspace, work_dir):
|
||||
prebake_cfg = os.path.join(
|
||||
os.path.dirname(__file__), "fixtures", "prebake", "minimal.yml"
|
||||
)
|
||||
script = os.path.join(FIXTURES, "pipeline.sh")
|
||||
bake_base = os.path.join(FIXTURES, "bake_base.sh")
|
||||
|
||||
result = run_baker(
|
||||
"bake", script,
|
||||
"--bake-base", bake_base,
|
||||
"--prebake", prebake_cfg,
|
||||
cwd=work_dir,
|
||||
timeout=60,
|
||||
)
|
||||
assert result["success"], f"Pipeline bake failed: {result['stderr']}"
|
||||
assert os.path.exists("/tmp/bake-test-output/step1.txt")
|
||||
assert os.path.exists("/tmp/bake-test-output/step2.txt")
|
||||
|
||||
def test_pipeline_step_outputs_correct(self, run_baker, bake_workspace, work_dir):
|
||||
prebake_cfg = os.path.join(
|
||||
os.path.dirname(__file__), "fixtures", "prebake", "minimal.yml"
|
||||
)
|
||||
script = os.path.join(FIXTURES, "pipeline.sh")
|
||||
bake_base = os.path.join(FIXTURES, "bake_base.sh")
|
||||
|
||||
result = run_baker(
|
||||
"bake", script,
|
||||
"--bake-base", bake_base,
|
||||
"--prebake", prebake_cfg,
|
||||
cwd=work_dir,
|
||||
timeout=60,
|
||||
)
|
||||
assert result["success"]
|
||||
with open("/tmp/bake-test-output/step1.txt") as f:
|
||||
assert f.read().strip() == "step1-done"
|
||||
with open("/tmp/bake-test-output/step2.txt") as f:
|
||||
assert f.read().strip() == "step2-done"
|
||||
|
||||
|
||||
@pytest.mark.bake
|
||||
class TestBakeAfterDependencies:
|
||||
|
||||
def test_after_ordering(self, run_baker, bake_workspace, work_dir):
|
||||
prebake_cfg = os.path.join(
|
||||
os.path.dirname(__file__), "fixtures", "prebake", "minimal.yml"
|
||||
)
|
||||
script = os.path.join(FIXTURES, "with_after.sh")
|
||||
bake_base = os.path.join(FIXTURES, "bake_base.sh")
|
||||
|
||||
result = run_baker(
|
||||
"bake", script,
|
||||
"--bake-base", bake_base,
|
||||
"--prebake", prebake_cfg,
|
||||
cwd=work_dir,
|
||||
timeout=60,
|
||||
)
|
||||
assert result["success"], f"@after bake failed: {result['stderr']}"
|
||||
assert os.path.exists("/tmp/bake-after-test/status.txt")
|
||||
assert os.path.exists("/tmp/bake-after-test/source.txt")
|
||||
assert os.path.exists("/tmp/bake-after-test/build.txt")
|
||||
|
||||
|
||||
@pytest.mark.bake
|
||||
class TestBakeStatusFiles:
|
||||
|
||||
def test_status_written_after_bake(self, run_baker, bake_workspace, work_dir):
|
||||
prebake_cfg = os.path.join(
|
||||
os.path.dirname(__file__), "fixtures", "prebake", "minimal.yml"
|
||||
)
|
||||
script = os.path.join(FIXTURES, "simple.sh")
|
||||
bake_base = os.path.join(FIXTURES, "bake_base.sh")
|
||||
|
||||
result = run_baker(
|
||||
"bake", script,
|
||||
"--bake-base", bake_base,
|
||||
"--prebake", prebake_cfg,
|
||||
cwd=work_dir,
|
||||
timeout=60,
|
||||
)
|
||||
assert result["success"]
|
||||
assert os.path.exists("/tmp/.hbwstatus")
|
||||
@@ -1,77 +0,0 @@
|
||||
import os
|
||||
import pytest
|
||||
|
||||
from probes import file
|
||||
from llm import judge_log, LLMClient
|
||||
|
||||
|
||||
FIXTURES = os.path.join(os.path.dirname(__file__), "fixtures", "finalize")
|
||||
|
||||
|
||||
@pytest.mark.finalize
|
||||
class TestFinalizeMinimal:
|
||||
|
||||
def test_minimal_finalize_succeeds(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "minimal.yml")
|
||||
result = run_baker("finalize", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"], f"Minimal finalize failed: {result['stderr']}"
|
||||
|
||||
|
||||
@pytest.mark.finalize
|
||||
class TestFinalizeHooks:
|
||||
|
||||
def test_early_hook_executes(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "with_hooks.yml")
|
||||
result = run_baker("finalize", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
assert "finalize early hook" in result["stdout"] or "finalize early hook" in result["stderr"]
|
||||
|
||||
def test_late_hook_executes(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "with_hooks.yml")
|
||||
result = run_baker("finalize", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
assert "finalize late hook" in result["stdout"] or "finalize late hook" in result["stderr"]
|
||||
|
||||
|
||||
@pytest.mark.finalize
|
||||
class TestFinalizeStatus:
|
||||
|
||||
def test_finalize_writes_status(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "minimal.yml")
|
||||
result = run_baker("finalize", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
assert os.path.exists("/tmp/.hbwstatus")
|
||||
|
||||
|
||||
@pytest.mark.finalize
|
||||
@pytest.mark.llm
|
||||
class TestFinalizeWithLLM:
|
||||
|
||||
def test_finalize_hooks_via_llm(self, run_baker, llm_env, work_dir):
|
||||
if not llm_env:
|
||||
pytest.skip("LLM environment not configured")
|
||||
|
||||
config = os.path.join(FIXTURES, "with_hooks.yml")
|
||||
result = run_baker("finalize", config, cwd=work_dir, timeout=60)
|
||||
|
||||
combined_log = result["stdout"] + "\n" + result["stderr"]
|
||||
|
||||
judgment = judge_log(
|
||||
operation="finalize with early and late hooks",
|
||||
expected_behavior="Both early and late hooks executed successfully, "
|
||||
"printing 'finalize early hook' and 'finalize late hook'",
|
||||
actual_log=combined_log,
|
||||
)
|
||||
assert judgment.passed, f"LLM judgment failed: {judgment.reason}"
|
||||
|
||||
|
||||
@pytest.mark.finalize
|
||||
class TestFinalizePlugins:
|
||||
|
||||
@pytest.mark.xfail(reason="Plugin download requires network/git access", strict=False)
|
||||
def test_plugin_download(self, run_baker, work_dir):
|
||||
pass
|
||||
|
||||
@pytest.mark.xfail(reason="Artifact stage not yet implemented", strict=False)
|
||||
def test_artifact_collection(self, run_baker, work_dir):
|
||||
pass
|
||||
@@ -1,261 +0,0 @@
|
||||
import os
|
||||
import json
|
||||
import pytest
|
||||
|
||||
from probes import user, file, package
|
||||
|
||||
|
||||
FIXTURES = os.path.join(os.path.dirname(__file__), "fixtures", "prebake")
|
||||
|
||||
|
||||
@pytest.mark.prebake
|
||||
class TestPrebakeConfigValidation:
|
||||
|
||||
def test_invalid_version_fails(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "invalid_version.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert not result["success"], f"Expected prebake to fail with invalid version"
|
||||
assert "version" in result["stderr"].lower() or "Version" in result["stderr"]
|
||||
|
||||
def test_missing_required_field_fails(self, run_baker, work_dir):
|
||||
missing_env = os.path.join(work_dir, "missing_env.yml")
|
||||
with open(missing_env, "w") as f:
|
||||
f.write('version: "1.0"\n')
|
||||
result = run_baker("prebake", missing_env, cwd=work_dir, timeout=60)
|
||||
assert not result["success"]
|
||||
|
||||
def test_minimal_config_succeeds(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "minimal.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"], f"Minimal prebake failed: {result['stderr']}"
|
||||
|
||||
|
||||
@pytest.mark.prebake
|
||||
class TestPrebakeBootstrap:
|
||||
|
||||
def test_bootstrap_creates_vulcan_user(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"], f"prebake failed: {result['stderr']}"
|
||||
assert user.user_exists("vulcan")
|
||||
|
||||
def test_bootstrap_creates_workspace(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
assert file.dir_exists("/home/vulcan/workspace")
|
||||
|
||||
def test_bootstrap_creates_symlink(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
assert os.path.islink("/workspace")
|
||||
|
||||
def test_bootstrap_generates_bootstrap_script(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
assert file.file_exists("/bootstrap.sh")
|
||||
|
||||
def test_bootstrap_script_permissions(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
mode = file.file_permissions("/bootstrap.sh")
|
||||
assert mode & 0o111, f"/bootstrap.sh not executable, mode: {oct(mode)}"
|
||||
|
||||
def test_bootstrap_script_contains_user_creation(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
assert file.file_exists("/bootstrap.sh")
|
||||
assert file.file_contains("/bootstrap.sh", "useradd")
|
||||
|
||||
def test_bootstrap_script_contains_workspace_setup(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
assert file.file_contains("/bootstrap.sh", "mkdir -p /home/vulcan/workspace")
|
||||
|
||||
def test_bootstrap_with_pacman_creates_sudoers(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "with_pacman.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=120)
|
||||
assert result["success"], f"prebake failed: {result['stderr']}"
|
||||
assert file.file_exists("/etc/sudoers.d/workshop")
|
||||
|
||||
def test_custom_bootstrap_executes(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "with_custom_bootstrap.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"], f"Custom bootstrap failed: {result['stderr']}"
|
||||
assert file.dir_exists("/custom-workspace")
|
||||
|
||||
|
||||
@pytest.mark.prebake
|
||||
class TestPrebakeDryRun:
|
||||
|
||||
def test_dry_run_does_not_create_user(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
env = {"HBW_DRY_RUN": "true"}
|
||||
result = run_baker("prebake", config, cwd=work_dir, env=env, timeout=60)
|
||||
assert result["success"], f"Dry-run failed: {result['stderr']}"
|
||||
|
||||
def test_dry_run_logs_script_content(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
env = {"HBW_DRY_RUN": "true", "RUST_LOG": "info"}
|
||||
result = run_baker("prebake", config, cwd=work_dir, env=env, timeout=60)
|
||||
assert result["success"]
|
||||
combined = result["stdout"] + result["stderr"]
|
||||
assert "DRY_RUN" in combined or "dry_run" in combined.lower()
|
||||
|
||||
|
||||
@pytest.mark.prebake
|
||||
class TestPrebakeHooks:
|
||||
|
||||
#@pytest.mark.xfail(reason="Hook execution fails with Permission denied in container environment", strict=False)
|
||||
def test_early_hook_executes(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "with_hooks.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"], f"prebake with hooks failed: {result['stderr']}"
|
||||
combined = result["stdout"] + result["stderr"]
|
||||
assert "early hook executed" in combined
|
||||
|
||||
#@pytest.mark.xfail(reason="Hook execution fails with Permission denied in container environment", strict=False)
|
||||
def test_late_hook_executes(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "with_hooks.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"], f"prebake with hooks failed: {result['stderr']}"
|
||||
combined = result["stdout"] + result["stderr"]
|
||||
assert "late hook executed" in combined
|
||||
|
||||
#@pytest.mark.xfail(reason="Hook execution fails with Permission denied in container environment", strict=False)
|
||||
def test_hooks_run_in_order(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "with_hooks.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
combined = result["stdout"] + result["stderr"]
|
||||
early_pos = combined.find("early hook executed")
|
||||
late_pos = combined.find("late hook executed")
|
||||
assert early_pos != -1 and late_pos != -1
|
||||
assert early_pos < late_pos, "Early hook should execute before late hook"
|
||||
|
||||
def test_empty_hooks_succeed(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "minimal.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"], f"prebake with no hooks failed: {result['stderr']}"
|
||||
|
||||
|
||||
@pytest.mark.prebake
|
||||
class TestPrebakeDepsSystem:
|
||||
|
||||
@pytest.mark.xfail(reason="Package installation may fail in test environment", strict=False)
|
||||
def test_pacman_packages_installed(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "with_pacman.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=120)
|
||||
assert result["success"], f"prebake failed: {result['stderr']}"
|
||||
assert package.package_installed("curl")
|
||||
assert package.package_installed("wget")
|
||||
|
||||
def test_pacman_mirror_change(self, run_baker, work_dir):
|
||||
mirror_config = os.path.join(work_dir, "mirror.yml")
|
||||
with open(mirror_config, "w") as f:
|
||||
f.write('''
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
dependencies:
|
||||
system:
|
||||
pacman:
|
||||
packages: ["which"]
|
||||
mirror: "https://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch"
|
||||
''')
|
||||
result = run_baker("prebake", mirror_config, cwd=work_dir, timeout=120)
|
||||
assert result["success"], f"prebake with mirror failed: {result['stderr']}"
|
||||
|
||||
|
||||
@pytest.mark.prebake
|
||||
class TestPrebakeEnvVars:
|
||||
|
||||
def test_prebake_envvars_injected(self, run_baker, work_dir):
|
||||
hook_config = os.path.join(work_dir, "envvar_hook.yml")
|
||||
with open(hook_config, "w") as f:
|
||||
f.write('''
|
||||
version: "1.0"
|
||||
environment:
|
||||
builder: "baremetal"
|
||||
bootstrap:
|
||||
user: "vulcan"
|
||||
envvars:
|
||||
prebake:
|
||||
TEST_INJECTED: "injected_value"
|
||||
hooks:
|
||||
early:
|
||||
- name: "check-env"
|
||||
command: "echo $TEST_INJECTED"
|
||||
''')
|
||||
result = run_baker("prebake", hook_config, cwd=work_dir, timeout=60)
|
||||
assert result["success"], f"prebake with envvars failed: {result['stderr']}"
|
||||
combined = result["stdout"] + result["stderr"]
|
||||
assert "injected_value" in combined
|
||||
|
||||
|
||||
@pytest.mark.prebake
|
||||
class TestPrebakeSecurity:
|
||||
|
||||
#@pytest.mark.xfail(reason="Privilege drop may fail in container without proper setup", strict=False)
|
||||
def test_privilege_drop_after_bootstrap(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "with_security.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"], f"prebake with security failed: {result['stderr']}"
|
||||
|
||||
|
||||
@pytest.mark.prebake
|
||||
class TestPrebakeStatus:
|
||||
|
||||
def test_status_files_written(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
assert os.path.exists("/tmp/.hbwstatus")
|
||||
|
||||
def test_status_contains_bootstrap_stage(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
with open("/tmp/.hbwstatus", "r") as f:
|
||||
lines = f.readlines()
|
||||
assert len(lines) > 0
|
||||
bootstrap_lines = [l for l in lines if json.loads(l).get("name") == "bootstrap"]
|
||||
assert len(bootstrap_lines) > 0, "No bootstrap stage found in status file"
|
||||
data = json.loads(bootstrap_lines[-1])
|
||||
assert data.get("phase") == "Prebake"
|
||||
assert data.get("result") == "Success"
|
||||
|
||||
def test_status_contains_duration(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
with open("/tmp/.hbwstatus", "r") as f:
|
||||
data = json.loads(f.readline())
|
||||
assert "duration_ms" in data
|
||||
assert isinstance(data["duration_ms"], int)
|
||||
|
||||
def test_status_contains_timestamps(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "bootstrap_only.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=60)
|
||||
assert result["success"]
|
||||
with open("/tmp/.hbwstatus", "r") as f:
|
||||
data = json.loads(f.readline())
|
||||
assert "started_at" in data
|
||||
assert "finished_at" in data
|
||||
|
||||
def test_status_written_for_each_stage(self, run_baker, work_dir):
|
||||
config = os.path.join(FIXTURES, "with_pacman.yml")
|
||||
result = run_baker("prebake", config, cwd=work_dir, timeout=120)
|
||||
assert result["success"], f"prebake failed: {result['stderr']}"
|
||||
with open("/tmp/.hbwstatus", "r") as f:
|
||||
lines = f.readlines()
|
||||
names = [json.loads(line).get("name") for line in lines]
|
||||
assert "bootstrap" in names
|
||||
assert "depssystem" in names
|
||||
Generated
+107
@@ -0,0 +1,107 @@
|
||||
# This file is automatically @generated by Cargo.
|
||||
# It is not intended for manual editing.
|
||||
version = 4
|
||||
|
||||
[[package]]
|
||||
name = "itoa"
|
||||
version = "1.0.18"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
|
||||
|
||||
[[package]]
|
||||
name = "memchr"
|
||||
version = "2.8.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f8ca58f447f06ed17d5fc4043ce1b10dd205e060fb3ce5b979b8ed8e59ff3f79"
|
||||
|
||||
[[package]]
|
||||
name = "proc-macro2"
|
||||
version = "1.0.106"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934"
|
||||
dependencies = [
|
||||
"unicode-ident",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "quote"
|
||||
version = "1.0.45"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde"
|
||||
version = "1.0.228"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e"
|
||||
dependencies = [
|
||||
"serde_core",
|
||||
"serde_derive",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde_core"
|
||||
version = "1.0.228"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad"
|
||||
dependencies = [
|
||||
"serde_derive",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde_derive"
|
||||
version = "1.0.228"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde_json"
|
||||
version = "1.0.149"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "83fc039473c5595ace860d8c4fafa220ff474b3fc6bfdb4293327f1a37e94d86"
|
||||
dependencies = [
|
||||
"itoa",
|
||||
"memchr",
|
||||
"serde",
|
||||
"serde_core",
|
||||
"zmij",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "syn"
|
||||
version = "2.0.117"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"unicode-ident",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "unicode-ident"
|
||||
version = "1.0.24"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75"
|
||||
|
||||
[[package]]
|
||||
name = "workshop-baker-params"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"serde",
|
||||
"serde_json",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "zmij"
|
||||
version = "1.0.21"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa"
|
||||
@@ -0,0 +1,8 @@
|
||||
[package]
|
||||
name = "workshop-baker-params"
|
||||
version = "0.1.0"
|
||||
edition = "2024"
|
||||
|
||||
[dependencies]
|
||||
serde = { version = "1.0.228", features = ["derive"] }
|
||||
serde_json = "1.0.149"
|
||||
@@ -0,0 +1,24 @@
|
||||
use crate::apply_field;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct BakeParams {
|
||||
#[serde(default)] pub workspace: ParamVal<String>,
|
||||
}
|
||||
|
||||
impl MergeParams for BakeParams {
|
||||
fn defaults() -> Self { Self { workspace: ParamVal::new("/workspace".into()) } }
|
||||
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
|
||||
apply_field!(self.workspace, i.workspace, w, ov, p, "workspace");
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for BakeParams { fn default() -> Self { Self::defaults() } }
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct BakeSettings { pub workspace: String }
|
||||
impl Default for BakeSettings { fn default() -> Self { BakeParams::defaults().into() } }
|
||||
impl From<&BakeParams> for BakeSettings { fn from(p: &BakeParams) -> Self { Self { workspace: p.workspace.value.clone() } } }
|
||||
impl From<BakeParams> for BakeSettings { fn from(p: BakeParams) -> Self { (&p).into() } }
|
||||
@@ -0,0 +1,23 @@
|
||||
use crate::apply_field;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct FinalizeParams {
|
||||
#[serde(default)] pub artifact_retention_days: ParamVal<u32>,
|
||||
}
|
||||
|
||||
impl MergeParams for FinalizeParams {
|
||||
fn defaults() -> Self { Self { artifact_retention_days: ParamVal::new(7) } }
|
||||
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
|
||||
apply_field!(self.artifact_retention_days, i.artifact_retention_days, w, ov, p, "artifact_retention_days");
|
||||
}
|
||||
}
|
||||
impl Default for FinalizeParams { fn default() -> Self { Self::defaults() } }
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct FinalizeSettings { pub artifact_retention_days: u32 }
|
||||
impl Default for FinalizeSettings { fn default() -> Self { FinalizeParams::defaults().into() } }
|
||||
impl From<&FinalizeParams> for FinalizeSettings { fn from(p: &FinalizeParams) -> Self { Self { artifact_retention_days: p.artifact_retention_days.value } } }
|
||||
impl From<FinalizeParams> for FinalizeSettings { fn from(p: FinalizeParams) -> Self { (&p).into() } }
|
||||
@@ -0,0 +1,168 @@
|
||||
pub mod notification;
|
||||
pub mod prebake;
|
||||
pub mod bake;
|
||||
pub mod finalize;
|
||||
pub mod writer;
|
||||
pub mod param;
|
||||
pub mod traits;
|
||||
|
||||
pub use notification::{NotificationParams, NotificationSettings};
|
||||
pub use prebake::{PrebakeParams, PrebakeSettings};
|
||||
pub use bake::{BakeParams, BakeSettings};
|
||||
pub use finalize::{FinalizeParams, FinalizeSettings};
|
||||
pub use writer::Writer;
|
||||
pub use param::ParamVal;
|
||||
pub use traits::MergeParams;
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::collections::HashMap;
|
||||
|
||||
/// Root parameter tree. Each component gets its own subtree.
|
||||
/// Used for both IPC (Serialized to JSON) and in-process merging.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||
pub struct Params {
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub notification: Option<NotificationParams>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub prebake: Option<PrebakeParams>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub bake: Option<BakeParams>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub finalize: Option<FinalizeParams>,
|
||||
}
|
||||
|
||||
/// Apply and merge helpers.
|
||||
impl Params {
|
||||
/// Merge a layer from a specific writer into the current params.
|
||||
/// Returns entries that were overridden (for debug).
|
||||
pub fn apply(&mut self, incoming: Params, writer: Writer) -> Overridden {
|
||||
let mut ov = Overridden::new();
|
||||
if let Some(l) = incoming.notification {
|
||||
if let Some(ref mut b) = self.notification {
|
||||
b.apply(l, writer, &mut ov, "notification.");
|
||||
} else {
|
||||
self.notification = Some(l);
|
||||
}
|
||||
}
|
||||
if let Some(l) = incoming.prebake {
|
||||
if let Some(ref mut b) = self.prebake {
|
||||
b.apply(l, writer, &mut ov, "prebake.");
|
||||
} else {
|
||||
self.prebake = Some(l);
|
||||
}
|
||||
}
|
||||
if let Some(l) = incoming.bake {
|
||||
if let Some(ref mut b) = self.bake {
|
||||
b.apply(l, writer, &mut ov, "bake.");
|
||||
} else {
|
||||
self.bake = Some(l);
|
||||
}
|
||||
}
|
||||
if let Some(l) = incoming.finalize {
|
||||
if let Some(ref mut b) = self.finalize {
|
||||
b.apply(l, writer, &mut ov, "finalize.");
|
||||
} else {
|
||||
self.finalize = Some(l);
|
||||
}
|
||||
}
|
||||
ov
|
||||
}
|
||||
|
||||
/// Convert to baker-consumable settings. All params guaranteed to have values.
|
||||
pub fn to_settings(&self) -> Result<AllSettings, String> {
|
||||
Ok(AllSettings {
|
||||
notification: self.notification.as_ref().map(NotificationSettings::from).unwrap_or_default(),
|
||||
prebake: self.prebake.as_ref().map(PrebakeSettings::from).unwrap_or_default(),
|
||||
bake: self.bake.as_ref().map(BakeSettings::from).unwrap_or_default(),
|
||||
finalize: self.finalize.as_ref().map(FinalizeSettings::from).unwrap_or_default(),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// Plain-value settings for baker consumption. No metadata, no Option.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct AllSettings {
|
||||
pub notification: NotificationSettings,
|
||||
pub prebake: PrebakeSettings,
|
||||
pub bake: BakeSettings,
|
||||
pub finalize: FinalizeSettings,
|
||||
}
|
||||
|
||||
/// Debug — values that were overridden during merge, keyed by dot-path.
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct Overridden {
|
||||
pub entries: std::collections::HashMap<String, serde_json::Value>,
|
||||
}
|
||||
|
||||
impl Overridden {
|
||||
pub fn new() -> Self { Self { entries: HashMap::new() } }
|
||||
pub fn is_empty(&self) -> bool { self.entries.is_empty() }
|
||||
}
|
||||
|
||||
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_empty_params_serializes_empty() {
|
||||
let p = Params::default();
|
||||
let json = serde_json::to_string(&p).unwrap();
|
||||
assert_eq!(json, "{}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_notification_params_serialize() {
|
||||
let mut p = Params::default();
|
||||
p.notification = Some(NotificationParams::defaults());
|
||||
let json = serde_json::to_string_pretty(&p).unwrap();
|
||||
assert!(json.contains("max_lives"));
|
||||
assert!(json.contains("3"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_apply_courier_overrides_default() {
|
||||
let mut base = Params::default();
|
||||
base.notification = Some(NotificationParams::defaults());
|
||||
let incoming = Params {
|
||||
notification: Some(NotificationParams {
|
||||
max_lives: ParamVal::with(5, Writer::Courier),
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
};
|
||||
let ov = base.apply(incoming, Writer::Courier);
|
||||
let s = base.to_settings().unwrap();
|
||||
assert_eq!(s.notification.max_lives, 5);
|
||||
assert!(ov.is_empty()); // default had no writer, not an "override"
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_apply_upstream_wins() {
|
||||
let mut base = Params::default();
|
||||
base.notification = Some(NotificationParams {
|
||||
max_lives: ParamVal::with(3, Writer::Base),
|
||||
..Default::default()
|
||||
});
|
||||
let incoming = Params {
|
||||
notification: Some(NotificationParams {
|
||||
max_lives: ParamVal::with(5, Writer::Courier),
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
};
|
||||
let ov = base.apply(incoming, Writer::Courier);
|
||||
let s = base.to_settings().unwrap();
|
||||
// base(3) > courier(2) — rejected, base stays
|
||||
assert_eq!(s.notification.max_lives, 3);
|
||||
assert!(ov.is_empty()); // nothing overridden
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_deny_unknown_fields() {
|
||||
let bad = r#"{"max_lives":{"value":5},"ghost_param":{"value":1}}"#;
|
||||
let result: Result<NotificationParams, _> = serde_json::from_str(bad);
|
||||
assert!(result.is_err()); // unknown field should be denied
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
use crate::apply_field;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct NotificationParams {
|
||||
#[serde(default)] pub max_lives: ParamVal<u32>,
|
||||
#[serde(default)] pub timeout_ms: ParamVal<u64>,
|
||||
#[serde(default)] pub retry_backoff: ParamVal<f64>,
|
||||
#[serde(default)] pub retry_delay_seconds: ParamVal<f64>,
|
||||
#[serde(default)] pub max_retry_delay_seconds: ParamVal<f64>,
|
||||
#[serde(default)] pub life_impact_factor: ParamVal<f64>,
|
||||
#[serde(default)] pub batch_period: ParamVal<u32>,
|
||||
#[serde(default)] pub batch_watermark: ParamVal<u32>,
|
||||
#[serde(default)] pub template_ref_depth: ParamVal<u32>,
|
||||
}
|
||||
|
||||
impl MergeParams for NotificationParams {
|
||||
fn defaults() -> Self {
|
||||
Self {
|
||||
max_lives: ParamVal::new(3), timeout_ms: ParamVal::new(30000),
|
||||
retry_backoff: ParamVal::new(2.0), retry_delay_seconds: ParamVal::new(5.0),
|
||||
max_retry_delay_seconds: ParamVal::new(300.0),
|
||||
life_impact_factor: ParamVal::new(0.0),
|
||||
batch_period: ParamVal::new(30), batch_watermark: ParamVal::new(5),
|
||||
template_ref_depth: ParamVal::new(10),
|
||||
}
|
||||
}
|
||||
fn apply(&mut self, incoming: Self, writer: Writer, ov: &mut Overridden, prefix: &str) {
|
||||
apply_field!(self.max_lives, incoming.max_lives, writer, ov, prefix, "max_lives");
|
||||
apply_field!(self.timeout_ms, incoming.timeout_ms, writer, ov, prefix, "timeout_ms");
|
||||
apply_field!(self.retry_backoff, incoming.retry_backoff, writer, ov, prefix, "retry_backoff");
|
||||
apply_field!(self.retry_delay_seconds, incoming.retry_delay_seconds, writer, ov, prefix, "retry_delay_seconds");
|
||||
apply_field!(self.max_retry_delay_seconds, incoming.max_retry_delay_seconds, writer, ov, prefix, "max_retry_delay_seconds");
|
||||
apply_field!(self.life_impact_factor, incoming.life_impact_factor, writer, ov, prefix, "life_impact_factor");
|
||||
apply_field!(self.batch_period, incoming.batch_period, writer, ov, prefix, "batch_period");
|
||||
apply_field!(self.batch_watermark, incoming.batch_watermark, writer, ov, prefix, "batch_watermark");
|
||||
apply_field!(self.template_ref_depth, incoming.template_ref_depth, writer, ov, prefix, "template_ref_depth");
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for NotificationParams { fn default() -> Self { Self::defaults() } }
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct NotificationSettings {
|
||||
pub max_lives: u32, pub timeout_ms: u64, pub retry_backoff: f64,
|
||||
pub retry_delay_seconds: f64, pub max_retry_delay_seconds: f64,
|
||||
pub life_impact_factor: f64,
|
||||
pub batch_period: u32, pub batch_watermark: u32, pub template_ref_depth: u32,
|
||||
}
|
||||
|
||||
impl Default for NotificationSettings { fn default() -> Self { NotificationParams::defaults().into() } }
|
||||
|
||||
impl From<&NotificationParams> for NotificationSettings {
|
||||
fn from(p: &NotificationParams) -> Self { Self {
|
||||
max_lives: p.max_lives.value, timeout_ms: p.timeout_ms.value,
|
||||
retry_backoff: p.retry_backoff.value, retry_delay_seconds: p.retry_delay_seconds.value,
|
||||
max_retry_delay_seconds: p.max_retry_delay_seconds.value,
|
||||
life_impact_factor: p.life_impact_factor.value,
|
||||
batch_period: p.batch_period.value, batch_watermark: p.batch_watermark.value,
|
||||
template_ref_depth: p.template_ref_depth.value,
|
||||
}}
|
||||
}
|
||||
impl From<NotificationParams> for NotificationSettings { fn from(p: NotificationParams) -> Self { (&p).into() } }
|
||||
@@ -0,0 +1,32 @@
|
||||
use serde::{Deserialize, Serialize};
|
||||
use crate::Writer;
|
||||
|
||||
/// A strongly-typed parameter value with writer tracking.
|
||||
///
|
||||
/// Serializes as `{ "value": 3, "writer": "base" }`.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct ParamVal<T> {
|
||||
pub value: T,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub writer: Option<Writer>,
|
||||
}
|
||||
|
||||
impl<T: Default> Default for ParamVal<T> {
|
||||
fn default() -> Self {
|
||||
ParamVal { value: T::default(), writer: None }
|
||||
}
|
||||
}
|
||||
|
||||
impl<T> ParamVal<T> {
|
||||
pub fn new(value: T) -> Self {
|
||||
ParamVal { value, writer: None }
|
||||
}
|
||||
pub fn with(value: T, writer: Writer) -> Self {
|
||||
ParamVal { value, writer: Some(writer) }
|
||||
}
|
||||
/// Whether an incoming value from `incoming_writer` should override this one.
|
||||
pub fn should_override(&self, incoming: Writer) -> bool {
|
||||
let current = self.writer.map_or(0, |w| w.priority());
|
||||
incoming.priority() > current
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
pub mod network;
|
||||
pub mod cache;
|
||||
pub mod engine;
|
||||
|
||||
use crate::apply_field;
|
||||
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct PrebakeParams {
|
||||
#[serde(default)] pub bootstrap_user: ParamVal<String>,
|
||||
#[serde(default)] pub workspace: ParamVal<String>,
|
||||
#[serde(default)] pub network: network::NetworkParams,
|
||||
#[serde(default)] pub cache: cache::CacheParams,
|
||||
#[serde(default)] pub engine: engine::EngineParams,
|
||||
#[serde(default)] pub repology_endpoint: ParamVal<String>,
|
||||
}
|
||||
|
||||
impl MergeParams for PrebakeParams {
|
||||
fn defaults() -> Self {
|
||||
Self {
|
||||
bootstrap_user: ParamVal::new("vulcan".into()), workspace: ParamVal::new("/home/vulcan/workspace".into()),
|
||||
network: network::NetworkParams::defaults(), cache: cache::CacheParams::defaults(),
|
||||
engine: engine::EngineParams::defaults(),
|
||||
repology_endpoint: ParamVal::new("default".into()),
|
||||
}
|
||||
}
|
||||
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
|
||||
apply_field!(self.bootstrap_user, i.bootstrap_user, w, ov, p, "bootstrap_user");
|
||||
apply_field!(self.workspace, i.workspace, w, ov, p, "workspace");
|
||||
apply_field!(self.repology_endpoint, i.repology_endpoint, w, ov, p, "repology_endpoint");
|
||||
self.network.apply(i.network, w, ov, &format!("{}network.", p));
|
||||
self.cache.apply(i.cache, w, ov, &format!("{}cache.", p));
|
||||
self.engine.apply(i.engine, w, ov, &format!("{}engine.", p));
|
||||
}
|
||||
}
|
||||
impl Default for PrebakeParams { fn default() -> Self { Self::defaults() } }
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct PrebakeSettings {
|
||||
pub bootstrap_user: String, pub workspace: String,
|
||||
pub network: network::NetworkSettings, pub cache: cache::CacheSettings,
|
||||
pub engine: engine::EngineSettings, pub repology_endpoint: String,
|
||||
}
|
||||
impl Default for PrebakeSettings { fn default() -> Self { PrebakeParams::defaults().into() } }
|
||||
impl From<&PrebakeParams> for PrebakeSettings {
|
||||
fn from(p: &PrebakeParams) -> Self { Self {
|
||||
bootstrap_user: p.bootstrap_user.value.clone(), workspace: p.workspace.value.clone(),
|
||||
network: (&p.network).into(), cache: (&p.cache).into(), engine: (&p.engine).into(),
|
||||
repology_endpoint: p.repology_endpoint.value.clone(),
|
||||
}}
|
||||
}
|
||||
impl From<PrebakeParams> for PrebakeSettings { fn from(p: PrebakeParams) -> Self { (&p).into() } }
|
||||
@@ -0,0 +1,37 @@
|
||||
use crate::apply_field;
|
||||
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct CacheParams {
|
||||
#[serde(default)] pub strategy: ParamVal<String>,
|
||||
#[serde(default)] pub mode: ParamVal<String>,
|
||||
#[serde(default)] pub ttl_days: ParamVal<u32>,
|
||||
#[serde(default)] pub max_size_gb: ParamVal<u32>,
|
||||
#[serde(default)] pub cleanup_policy: ParamVal<String>,
|
||||
}
|
||||
impl MergeParams for CacheParams {
|
||||
fn defaults() -> Self {
|
||||
Self {
|
||||
strategy: ParamVal::new("always".into()), mode: ParamVal::new("zstd".into()),
|
||||
ttl_days: ParamVal::new(30), max_size_gb: ParamVal::new(20),
|
||||
cleanup_policy: ParamVal::new("lru".into()),
|
||||
}
|
||||
}
|
||||
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
|
||||
apply_field!(self.strategy, i.strategy, w, ov, p, "strategy");
|
||||
apply_field!(self.mode, i.mode, w, ov, p, "mode");
|
||||
apply_field!(self.ttl_days, i.ttl_days, w, ov, p, "ttl_days");
|
||||
apply_field!(self.max_size_gb, i.max_size_gb, w, ov, p, "max_size_gb");
|
||||
apply_field!(self.cleanup_policy, i.cleanup_policy, w, ov, p, "cleanup_policy");
|
||||
}
|
||||
}
|
||||
impl Default for CacheParams { fn default() -> Self { Self::defaults() } }
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct CacheSettings { pub strategy: String, pub mode: String, pub ttl_days: u32, pub max_size_gb: u32, pub cleanup_policy: String }
|
||||
impl Default for CacheSettings { fn default() -> Self { CacheParams::defaults().into() } }
|
||||
impl From<&CacheParams> for CacheSettings { fn from(p: &CacheParams) -> Self {
|
||||
Self { strategy: p.strategy.value.clone(), mode: p.mode.value.clone(), ttl_days: p.ttl_days.value, max_size_gb: p.max_size_gb.value, cleanup_policy: p.cleanup_policy.value.clone() }
|
||||
} }
|
||||
impl From<CacheParams> for CacheSettings { fn from(p: CacheParams) -> Self { (&p).into() } }
|
||||
@@ -0,0 +1,21 @@
|
||||
use crate::apply_field;
|
||||
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct EngineParams {
|
||||
#[serde(default)] pub timeout_ms: ParamVal<u64>,
|
||||
}
|
||||
impl MergeParams for EngineParams {
|
||||
fn defaults() -> Self { Self { timeout_ms: ParamVal::new(300000) } }
|
||||
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
|
||||
apply_field!(self.timeout_ms, i.timeout_ms, w, ov, p, "timeout_ms");
|
||||
}
|
||||
}
|
||||
impl Default for EngineParams { fn default() -> Self { Self::defaults() } }
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct EngineSettings { pub timeout_ms: u64 }
|
||||
impl Default for EngineSettings { fn default() -> Self { EngineParams::defaults().into() } }
|
||||
impl From<&EngineParams> for EngineSettings { fn from(p: &EngineParams) -> Self { Self { timeout_ms: p.timeout_ms.value } } }
|
||||
impl From<EngineParams> for EngineSettings { fn from(p: EngineParams) -> Self { (&p).into() } }
|
||||
@@ -0,0 +1,23 @@
|
||||
use crate::apply_field;
|
||||
use crate::{ParamVal, Writer, Overridden, traits::MergeParams};
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct NetworkParams {
|
||||
#[serde(default)] pub enabled: ParamVal<bool>,
|
||||
#[serde(default)] pub outbound: ParamVal<bool>,
|
||||
}
|
||||
impl MergeParams for NetworkParams {
|
||||
fn defaults() -> Self { Self { enabled: ParamVal::new(true), outbound: ParamVal::new(true) } }
|
||||
fn apply(&mut self, i: Self, w: Writer, ov: &mut Overridden, p: &str) {
|
||||
apply_field!(self.enabled, i.enabled, w, ov, p, "enabled");
|
||||
apply_field!(self.outbound, i.outbound, w, ov, p, "outbound");
|
||||
}
|
||||
}
|
||||
impl Default for NetworkParams { fn default() -> Self { Self::defaults() } }
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct NetworkSettings { pub enabled: bool, pub outbound: bool }
|
||||
impl Default for NetworkSettings { fn default() -> Self { NetworkParams::defaults().into() } }
|
||||
impl From<&NetworkParams> for NetworkSettings { fn from(p: &NetworkParams) -> Self { Self { enabled: p.enabled.value, outbound: p.outbound.value } } }
|
||||
impl From<NetworkParams> for NetworkSettings { fn from(p: NetworkParams) -> Self { (&p).into() } }
|
||||
@@ -0,0 +1,28 @@
|
||||
use crate::{Writer, Overridden};
|
||||
|
||||
/// A parameter tree node that can merge itself with an incoming layer.
|
||||
pub trait MergeParams: Sized {
|
||||
fn apply(&mut self, incoming: Self, writer: Writer, ov: &mut Overridden, prefix: &str);
|
||||
fn defaults() -> Self;
|
||||
}
|
||||
|
||||
/// Apply one field's merge logic.
|
||||
/// ```ignore
|
||||
/// apply_field!(self.foo, incoming.foo, writer, ov, prefix, "foo");
|
||||
/// ```
|
||||
#[macro_export]
|
||||
macro_rules! apply_field {
|
||||
($self:ident.$field:ident, $incoming:ident.$field2:ident, $writer:expr, $ov:ident, $prefix:expr, $name:expr) => {
|
||||
let old_w = $self.$field.writer;
|
||||
if $self.$field.should_override($writer) {
|
||||
let old_v = ::std::mem::replace(&mut $self.$field.value, $incoming.$field2.value);
|
||||
$self.$field.writer = ::std::option::Option::Some($writer);
|
||||
if let ::std::option::Option::Some(w) = old_w {
|
||||
$ov.entries.insert(
|
||||
format!("{}{}", $prefix, $name),
|
||||
::serde_json::json!({"value": old_v, "writer": w.as_str()}),
|
||||
);
|
||||
}
|
||||
}
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub enum Writer {
|
||||
Base,
|
||||
Courier,
|
||||
Bakerd,
|
||||
}
|
||||
|
||||
impl Writer {
|
||||
/// Priority: Base=3 > Courier=2 > Bakerd=1
|
||||
pub fn priority(self) -> u8 {
|
||||
match self {
|
||||
Writer::Base => 3,
|
||||
Writer::Courier => 2,
|
||||
Writer::Bakerd => 1,
|
||||
}
|
||||
}
|
||||
pub fn as_str(self) -> &'static str {
|
||||
match self {
|
||||
Writer::Base => "base", Writer::Courier => "courier", Writer::Bakerd => "bakerd",
|
||||
}
|
||||
}
|
||||
}
|
||||
impl Default for Writer { fn default() -> Self { Writer::Base } }
|
||||
@@ -1,3 +1,4 @@
|
||||
temp
|
||||
examples
|
||||
quicktest.sh
|
||||
_env.sh
|
||||
|
||||
Generated
+92
-1561
File diff suppressed because it is too large
Load Diff
@@ -28,7 +28,7 @@ env_logger = "0.11.8"
|
||||
glob = "0.3.2"
|
||||
futures-util = "0.3.31"
|
||||
lazy_static = "1.5.0"
|
||||
lettre = { version = "0.11", features = ["tokio1-native-tls"] }
|
||||
lettre = { version = "0.11", default-features = false, features = ["tokio1-rustls", "rustls-tls", "builder", "smtp-transport"] }
|
||||
log = "0.4.28"
|
||||
minijinja = "2.14.0"
|
||||
regex = "1.12.2"
|
||||
@@ -47,13 +47,10 @@ uuid = { version = "1.19.0", features = ["v4"] }
|
||||
libc = "0.2"
|
||||
os_info = "3.14.0"
|
||||
privdrop = "0.5.6"
|
||||
reqwest = { version = "0.12", features = ["json"] }
|
||||
which = "8.0.2"
|
||||
gix = "0.81.0"
|
||||
sha2 = "0.10"
|
||||
flate2 = "1.0"
|
||||
zstd = "0.13"
|
||||
globset = "0.4.18"
|
||||
axum = "0.7"
|
||||
workshop-baker-params = { version = "0.1.0", path = "../workshop-baker-params" }
|
||||
|
||||
[dev-dependencies]
|
||||
criterion = "0.5"
|
||||
@@ -66,6 +63,3 @@ unreachable_code = "allow"
|
||||
inherent_to_string = "allow"
|
||||
non_canonical_partial_ord_impl = "allow"
|
||||
|
||||
[[bench]]
|
||||
name = "parser_bench"
|
||||
harness = false
|
||||
|
||||
@@ -1,93 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
import os
|
||||
import random
|
||||
import string
|
||||
|
||||
OUTPUT_DIR = "/tmp"
|
||||
DECORATORS = [
|
||||
"# @pipeline",
|
||||
"# @fallible",
|
||||
"# @parallel",
|
||||
"# @export",
|
||||
"# @timeout(60)",
|
||||
"# @retry(3, 5)",
|
||||
"# @if(condition)",
|
||||
"# @after(dep)",
|
||||
"# @pipe(step1, step2)",
|
||||
"# @health(/health)",
|
||||
"# @loop(10)",
|
||||
]
|
||||
FUNCTION_BODIES = [
|
||||
"echo 'Processing...';",
|
||||
"local var=$(date +%s);",
|
||||
"if [ -f /tmp/test ]; then echo 'exists'; fi;",
|
||||
"for i in $(seq 1 10); do echo $i; done;",
|
||||
"case $1 in start) echo 'starting';; stop) echo 'stopping';; esac;",
|
||||
"read -r line < /dev/stdin;",
|
||||
"export PATH=$PATH:/usr/local/bin;",
|
||||
"set -e; set -u;",
|
||||
"trap 'echo error' ERR;",
|
||||
"cd /tmp || exit 1;",
|
||||
"local result=$(grep -r 'pattern' /var/log/ 2>/dev/null);",
|
||||
'for f in /tmp/*.tmp; do rm -f "$f"; done',
|
||||
"if [[ $var -gt 100 ]]; then echo 'large'; fi;",
|
||||
'while read -r line; do echo "$line"; done < /etc/passwd;',
|
||||
'select opt in a b c; do echo "Selected $opt"; break; done;',
|
||||
]
|
||||
|
||||
|
||||
def random_string(length=8):
|
||||
return "".join(random.choices(string.ascii_letters + string.digits, k=length))
|
||||
|
||||
|
||||
def generate_function(body_lines):
|
||||
lines = []
|
||||
func_name = f"func_{random_string(6)}"
|
||||
for _ in range(random.randint(0, 3)):
|
||||
lines.append(random.choice(DECORATORS))
|
||||
lines.append(f"{func_name}() {{")
|
||||
for _ in range(body_lines):
|
||||
lines.append(" " + random.choice(FUNCTION_BODIES))
|
||||
lines.append("}")
|
||||
lines.append("")
|
||||
return lines
|
||||
|
||||
|
||||
def generate_script(target_lines, num_functions, target_bytes):
|
||||
lines = []
|
||||
total_lines = 0
|
||||
body_per_func = max(1, (target_lines - num_functions * 3) // num_functions)
|
||||
|
||||
for _ in range(num_functions):
|
||||
func_lines = generate_function(body_per_func)
|
||||
lines.extend(func_lines)
|
||||
total_lines += len(func_lines)
|
||||
|
||||
while (
|
||||
total_lines < target_lines
|
||||
or len("\n".join(lines).encode("utf-8")) < target_bytes
|
||||
):
|
||||
lines.append(random.choice(FUNCTION_BODIES))
|
||||
total_lines += 1
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main():
|
||||
sizes = [
|
||||
("small", 1000, 10, 50_000),
|
||||
("medium", 5000, 50, 500_000),
|
||||
("large", 10000, 100, 950_000),
|
||||
]
|
||||
for name, lines, funcs, target_bytes in sizes:
|
||||
script = generate_script(lines, funcs, target_bytes)
|
||||
filepath = os.path.join(OUTPUT_DIR, f"bench_{name}.sh")
|
||||
with open(filepath, "w") as f:
|
||||
f.write(script)
|
||||
actual_bytes = len(script.encode("utf-8"))
|
||||
actual_lines = len(script.splitlines())
|
||||
print(f"Generated {filepath}: lines={actual_lines}, bytes={actual_bytes}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,61 +0,0 @@
|
||||
use criterion::{Criterion, black_box, criterion_group, criterion_main};
|
||||
use std::fs;
|
||||
use std::path::Path;
|
||||
|
||||
fn parse_large_script(c: &mut Criterion) {
|
||||
let script_path = Path::new("/tmp/bench_large.sh");
|
||||
|
||||
let script_content = fs::read_to_string(script_path)
|
||||
.expect("Failed to read benchmark script. Run `python3 benches/generate.py` first.");
|
||||
|
||||
let byte_size = script_content.len();
|
||||
let line_count = script_content.lines().count();
|
||||
|
||||
println!("Benchmark file: {} lines, {} bytes", line_count, byte_size);
|
||||
|
||||
c.bench_function("parse_script_large", |b| {
|
||||
b.iter(|| {
|
||||
let result =
|
||||
workshop_baker::bake::parser::parse_script(black_box(&script_content.clone()));
|
||||
black_box(result)
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
fn parse_medium_script(c: &mut Criterion) {
|
||||
let script_path = Path::new("/tmp/bench_medium.sh");
|
||||
|
||||
let script_content = fs::read_to_string(script_path)
|
||||
.expect("Failed to read benchmark script. Run `python3 benches/generate.py` first.");
|
||||
|
||||
c.bench_function("parse_script_medium", |b| {
|
||||
b.iter(|| {
|
||||
let result =
|
||||
workshop_baker::bake::parser::parse_script(black_box(&script_content.clone()));
|
||||
black_box(result)
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
fn parse_small_script(c: &mut Criterion) {
|
||||
let script_path = Path::new("/tmp/bench_small.sh");
|
||||
|
||||
let script_content = fs::read_to_string(script_path)
|
||||
.expect("Failed to read benchmark script. Run `python3 benches/generate.py` first.");
|
||||
|
||||
c.bench_function("parse_script_small", |b| {
|
||||
b.iter(|| {
|
||||
let result =
|
||||
workshop_baker::bake::parser::parse_script(black_box(&script_content.clone()));
|
||||
black_box(result)
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
criterion_group!(
|
||||
benches,
|
||||
parse_small_script,
|
||||
parse_medium_script,
|
||||
parse_large_script
|
||||
);
|
||||
criterion_main!(benches);
|
||||
@@ -1,62 +1,57 @@
|
||||
# workshop-baker/src
|
||||
|
||||
**Generated:** 2026-04-22
|
||||
**Commit:** 7b3b71a
|
||||
**Generated:** 2026-05-19
|
||||
**Commit:** 28abc5d
|
||||
|
||||
Baker CLI + execution engine for HoneyBiscuitWorkshop pipelines.
|
||||
|
||||
## OVERVIEW
|
||||
|
||||
CLI tool and runtime for executing CI/CD pipelines with resource isolation, privilege dropping, and package management integration.
|
||||
CLI tool and runtime for executing CI/CD pipelines. Orchestrates prebake → bake → finalize stages with notification dispatch.
|
||||
|
||||
## STRUCTURE
|
||||
|
||||
```
|
||||
src/
|
||||
├── bake.rs # Top-level bake orchestration
|
||||
├── bake/ # Script parsing and building
|
||||
│ ├── parser.rs # @decorator-aware shell parser
|
||||
│ ├── decorator.rs # @pipeline, @retry, @parallel, etc.
|
||||
│ ├── builder.rs # Script template builder
|
||||
│ └── schedule.rs # Function ordering via @after deps
|
||||
├── engine/ # Execution runtime
|
||||
│ ├── executor.rs # Script execution with cgroups
|
||||
│ ├── cgroups.rs # Linux cgroup resource limits
|
||||
│ ├── repology.rs # Package name resolution
|
||||
│ ├── pm.rs # Package manager detection
|
||||
│ └── upm.rs # User package manager (Nix/Guix)
|
||||
├── prebake/ # Build environment setup
|
||||
│ ├── config.rs # PrebakeConfig YAML schema
|
||||
│ ├── stage/ # Bootstrap, DepsSystem, DepsUser, Hooks
|
||||
│ └── security.rs # Privilege dropping logic
|
||||
├── types/ # Shared type definitions
|
||||
│ ├── repology.rs # RepologyEndpoint enum
|
||||
│ ├── builderconfig.rs
|
||||
│ ├── cache.rs
|
||||
│ └── memsize.rs
|
||||
├── monitor/ # Resource usage monitoring
|
||||
│ ├── cgroups.rs # cgroup-based usage collection
|
||||
│ └── jobobject.rs # Windows job objects (stub)
|
||||
├── daemon.rs # Unix socket daemon (TODO: refactor)
|
||||
└── lib.rs # CLI struct + module exports
|
||||
├── lib.rs # Module declarations + Cli struct + Commands enum
|
||||
├── main.rs # tokio::main entry point
|
||||
├── bare.rs # Bare mode (run individual stages)
|
||||
├── error.rs # CliError (wraps stage errors)
|
||||
├── bake.rs # Bake orchestration (script → execution)
|
||||
├── bake/ # Script parsing, decorator DSL, scheduling
|
||||
├── prebake.rs # Prebake orchestration (env setup)
|
||||
├── prebake/ # Config, stages, security, env providers
|
||||
├── finalize.rs # Finalize orchestration (post-build)
|
||||
├── finalize/ # Config, plugins, hooks, templates
|
||||
├── notify.rs # Notification system (batching, retry)
|
||||
├── notify/ # Types, templates, interactive, queue
|
||||
├── daemon/ # Empty (Daemon command = unimplemented!)
|
||||
├── monitor/ # Resource usage monitoring (cgroups)
|
||||
├── types.rs # Module re-exports
|
||||
├── types/ # buildstatus, resource types
|
||||
├── utils.rs # Module root (fetch submodule)
|
||||
└── utils/ # fetch.rs
|
||||
```
|
||||
|
||||
## WHERE TO LOOK
|
||||
|
||||
| Task | Location |
|
||||
|------|----------|
|
||||
| Script parsing with @decorators | `bake/parser.rs:29` - `parse_script()` |
|
||||
| Decorator types | `bake/decorator.rs` - `Decorator` enum |
|
||||
| Build execution | `engine/executor.rs` - `Executor::execute_script()` |
|
||||
| Resource limits | `engine/types.rs:56` - `ResourceLimits` struct |
|
||||
| Prebake stages | `prebake/stage/` subdirectories |
|
||||
| Package detection | `engine/pm.rs:detection()` |
|
||||
| CLI entry | `lib.rs:26` - `cli::Cli` struct |
|
||||
| CLI args | `lib.rs:21` - `Cli` struct |
|
||||
| Entry point | `main.rs:91` - `#[tokio::main]` |
|
||||
| Pipeline orchestration | `main.rs:68-74` - prebake→bake→finalize chain |
|
||||
| Bare mode (single stage) | `bare.rs:14` - `run_bare()` |
|
||||
| Script parsing | `bake/parser.rs:56` - `parse_script()` |
|
||||
| Decorator DSL | `bake/decorator.rs` - `Decorator` enum |
|
||||
| Prebake stages | `prebake/stage/` - Bootstrap→DepsSystem→DepsUser→Hooks→Ready |
|
||||
| Plugin system | `finalize/plugin.rs` + `finalize/plugin/` |
|
||||
| Notification dispatch | `notify.rs:66` - `notify()` async loop |
|
||||
| Notification types | `notify/types.rs` - `NotificationRule`, `ParsedTrigger` |
|
||||
|
||||
## CONVENTIONS
|
||||
|
||||
- **ExecutionContext**: Passed through all stages, carries task_id, username, env_vars, timeout
|
||||
- **PrebakeStage**: Ordered enum controlling stage execution sequence (Bootstrap -> Ready)
|
||||
- **Decorator**: Attached to shell functions via `# @name(args)` comments above function declarations
|
||||
- **Engine**: Lightweight wrapper around cgroup manager for resource isolation
|
||||
- **Dual-mode**: CLI commands (prebake, bake, finalize) vs daemon mode via Unix socket
|
||||
- **PrebakeStage**: Ordered enum Bootstrap→EarlyHook→DepsSystem→DepsUser→LateHook→Ready
|
||||
- **Decorator**: `# @name(args)` comments above shell function declarations
|
||||
- **Notification**: Plugin-based dispatch with batching, exponential backoff, priority groups
|
||||
- **Engine**: Delegated to `workshop-engine` crate (formerly in-tree)
|
||||
- **Dual-mode**: CLI commands (standalone) vs Daemon mode (unimplemented)
|
||||
|
||||
## ANTI-PATTERNS
|
||||
1. **Empty daemon/** directory — Daemon command is `unimplemented!()`
|
||||
2. **Dead code allowed** — `lints.rust.dead_code = "allow"` in Cargo.toml
|
||||
3. **Engine was extracted** — workshop-engine is its own crate now, src/engine/ removed
|
||||
4. **Python test infra exists but files removed** — pytest.ini remains, test dir is empty
|
||||
|
||||
@@ -12,14 +12,16 @@ bake/
|
||||
├── parser.rs # @decorator-aware shell script parser
|
||||
├── decorator.rs # Decorator enum + 100+ unit tests
|
||||
├── builder.rs # Script template assembly
|
||||
└── schedule.rs # Function ordering via @after dependencies
|
||||
├── schedule.rs # Function ordering via @after dependencies
|
||||
├── error.rs # BakeError enum
|
||||
└── constant.rs # DEFAULT_WORKSPACE, TRIVIAL_SCRIPT_NAME
|
||||
```
|
||||
|
||||
## WHERE TO LOOK
|
||||
|
||||
| Task | Location |
|
||||
|------|----------|
|
||||
| Parse script | `parser.rs:29` - `parse_script()` |
|
||||
Parse script | `parser.rs:56` - `parse_script()` |
|
||||
| Decorator types | `decorator.rs` - `Decorator` enum variants |
|
||||
| Build script | `builder.rs` - template builder |
|
||||
| Schedule deps | `schedule.rs` - @after resolution |
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
use std::path::PathBuf;
|
||||
|
||||
use crate::cli::{BareCommands, Cli};
|
||||
use crate::error::CliError;
|
||||
use crate::types::resource::ResourceRegistry;
|
||||
use crate::{bake::bake, finalize::finalize, prebake::prebake};
|
||||
use workshop_engine::{EventSender, ExecutionContext};
|
||||
fn or_workshop(path: &Option<PathBuf>, name: &str) -> PathBuf {
|
||||
path
|
||||
.clone()
|
||||
.unwrap_or_else(|| PathBuf::from(format!(".workshop/{}", name)))
|
||||
}
|
||||
|
||||
pub async fn run_bare(
|
||||
cli: &Cli,
|
||||
command: &BareCommands,
|
||||
ctx: &mut ExecutionContext,
|
||||
event_tx: EventSender,
|
||||
pipeline: &str,
|
||||
build_id: &str,
|
||||
registry: &ResourceRegistry,
|
||||
) -> Result<(), CliError> {
|
||||
let (stage, config) = match command {
|
||||
BareCommands::Prebake { .. } => ("prebake", or_workshop(&cli.prebake, "prebake.yml")),
|
||||
BareCommands::Bake { .. } => ("bake", or_workshop(&cli.bake, "bake.sh")),
|
||||
BareCommands::Finalize { .. } => {
|
||||
("finalize", or_workshop(&cli.finalize, "finalize.yml"))
|
||||
}
|
||||
};
|
||||
|
||||
let prebake_path = or_workshop(&cli.prebake, "prebake.yml");
|
||||
let bake_base = or_workshop(&cli.bake_base, "bake_base.sh");
|
||||
|
||||
ctx.task_id = format!("{}-{}-{}", pipeline, build_id, stage);
|
||||
ctx.pipeline_name = pipeline.to_string();
|
||||
ctx.standalone = true;
|
||||
|
||||
match command {
|
||||
BareCommands::Prebake { .. } => {
|
||||
prebake(&config, cli, None, ctx, event_tx, None).await?;
|
||||
}
|
||||
BareCommands::Bake { .. } => {
|
||||
bake(&config, &bake_base, &prebake_path, cli, ctx, event_tx).await?;
|
||||
}
|
||||
BareCommands::Finalize { .. } => {
|
||||
finalize(&config, cli, ctx, event_tx, registry).await?;
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
@@ -1,193 +0,0 @@
|
||||
use crate::cli::Cli;
|
||||
// use serde::Serialize;
|
||||
use serde_json::Value;
|
||||
// use std::path::PathBuf;
|
||||
// use std::sync::Arc;
|
||||
use tokio::io::{AsyncBufReadExt, BufReader};
|
||||
use tokio::net::UnixStream;
|
||||
// use tokio::process::{Child, Command};
|
||||
// use tokio::signal;
|
||||
// use tokio::sync::RwLock;
|
||||
// use tokio::time::{Duration, interval};
|
||||
|
||||
// #[derive(Debug, Clone, Serialize)]
|
||||
// struct Heartbeat {
|
||||
// pipeline_id: String,
|
||||
// build_id: String,
|
||||
// stage: String,
|
||||
// timestamp: u64,
|
||||
// status: String,
|
||||
// subtasks: Vec<String>,
|
||||
// }
|
||||
|
||||
pub async fn daemon(_cli: &Cli) {
|
||||
todo!("Daemon should be refactored");
|
||||
// let socket = PathBuf::from(settings.get_string("socket").unwrap());
|
||||
// let agentsocket = settings.get_string("agentsocket").unwrap();
|
||||
// let _ = std::fs::remove_file(&socket);
|
||||
|
||||
// let heartbeat = Arc::new(RwLock::new(Heartbeat {
|
||||
// pipeline_id: cli.pipeline.clone(),
|
||||
// build_id: cli.build_id.clone(),
|
||||
// stage: String::new(),
|
||||
// timestamp: 0,
|
||||
// status: String::new(),
|
||||
// subtasks: Vec::new(),
|
||||
// }));
|
||||
|
||||
// // let heartbeat_clone = heartbeat.clone();
|
||||
// let heartbeat_task = {
|
||||
// let hb = heartbeat.clone();
|
||||
// tokio::spawn(async move {
|
||||
// let mut tick = interval(Duration::from_secs(1));
|
||||
// loop {
|
||||
// tick.tick().await;
|
||||
// let mut hb = hb.write().await;
|
||||
// hb.timestamp = std::time::SystemTime::now()
|
||||
// .duration_since(std::time::UNIX_EPOCH)
|
||||
// .unwrap()
|
||||
// .as_millis() as u64;
|
||||
|
||||
// // 只输出到stdout(模拟未来发送给Agent)
|
||||
// println!("[HEARTBEAT] {}", serde_json::to_string(&*hb).unwrap());
|
||||
// }
|
||||
// })
|
||||
// };
|
||||
|
||||
// let socket_task = {
|
||||
// let socket = socket.clone();
|
||||
// tokio::spawn(async move {
|
||||
// log::info!("Creating Unix socket at {}", socket.display());
|
||||
// let listener =
|
||||
// UnixListener::bind(&socket).expect("Failed to bind Unix socket");
|
||||
|
||||
// loop {
|
||||
// match listener.accept().await {
|
||||
// Ok((stream, _)) => {
|
||||
// tokio::spawn(handle_connection(stream));
|
||||
// }
|
||||
// Err(e) => {
|
||||
// log::error!("Accept error: {}", e);
|
||||
// break;
|
||||
// }
|
||||
// }
|
||||
// }
|
||||
// })
|
||||
// };
|
||||
|
||||
// let mut tasks: Vec<TaskSpec> = Vec::new();
|
||||
|
||||
// let execution_task = {
|
||||
// let hb = heartbeat.clone();
|
||||
// tokio::spawn(async move {
|
||||
// let mut hb = hb.write().await;
|
||||
// let mut children: Vec<(String, Child)> = vec![];
|
||||
|
||||
// let executable = std::env::current_exe().unwrap();
|
||||
|
||||
// for (index, task) in tasks.iter().enumerate() {
|
||||
// let mut command = task.command(&executable);
|
||||
// command.stdout(std::process::Stdio::piped());
|
||||
// command.stderr(std::process::Stdio::piped());
|
||||
// let child = command
|
||||
// .spawn()
|
||||
// .expect(format!("Failed to spawn child {}", index).as_str());
|
||||
|
||||
// let pid = child.id().unwrap_or(0).to_string();
|
||||
// children.push((pid.clone(), child));
|
||||
|
||||
// hb.subtasks.push(pid.clone());
|
||||
// }
|
||||
|
||||
// for (pid, mut child) in children {
|
||||
// let status = child.wait().await.expect("Failed to wait child");
|
||||
// log::info!("Subtask {} exited with: {}", pid, status);
|
||||
|
||||
// hb.subtasks.retain(|p| p != &pid);
|
||||
// }
|
||||
|
||||
// hb.stage = "Success".to_string();
|
||||
// })
|
||||
// };
|
||||
|
||||
// tokio::select! {
|
||||
// _ = heartbeat_task => {
|
||||
// log::error!("Heartbeat task died");
|
||||
// }
|
||||
// _ = execution_task => {
|
||||
// log::info!("All subtasks completed");
|
||||
// }
|
||||
// _ = socket_task => {
|
||||
// log::error!("Socket task died");
|
||||
// }
|
||||
// _ = signal::ctrl_c() => {
|
||||
// log::info!("Shutting down...");
|
||||
// }
|
||||
// }
|
||||
// for pid in heartbeat.read().await.subtasks.clone() {
|
||||
// let _ = Command::new("kill").arg("-TERM").arg(&pid).status().await;
|
||||
// }
|
||||
|
||||
// let _ = std::fs::remove_file(&socket);
|
||||
}
|
||||
|
||||
async fn _handle_connection(stream: UnixStream) {
|
||||
let peer_addr = match stream.peer_addr() {
|
||||
Ok(addr) => format!("{:?}", addr),
|
||||
Err(_) => "unknown".to_string(),
|
||||
};
|
||||
|
||||
log::debug!("New connection from: {}", peer_addr);
|
||||
|
||||
let mut reader = BufReader::new(stream);
|
||||
let mut line = String::new();
|
||||
|
||||
loop {
|
||||
line.clear();
|
||||
match reader.read_line(&mut line).await {
|
||||
Ok(0) => {
|
||||
log::debug!("Connection closed by peer: {}", peer_addr);
|
||||
break;
|
||||
}
|
||||
Ok(_) => {
|
||||
// 简单打印原始JSON(预留字段解析空间)
|
||||
match serde_json::from_str::<Value>(&line) {
|
||||
Ok(json) => {
|
||||
// 只提取关键字段打印(字段可能变化)
|
||||
let msgtype = json
|
||||
.get("msgtype")
|
||||
.and_then(|v| v.as_str())
|
||||
.unwrap_or("unknown");
|
||||
let name = json
|
||||
.get("name")
|
||||
.and_then(|v| v.as_str())
|
||||
.unwrap_or("unnamed");
|
||||
|
||||
log::info!("[{}] Message from '{}': {}", msgtype, name, line.trim());
|
||||
|
||||
// TODO: 字段稳定后,可扩展为结构化打印
|
||||
// match msgtype {
|
||||
// "ResourceUsage" => print_resource(&json),
|
||||
// "Log" => print_log(&json),
|
||||
// _ => log::debug!("Unknown type: {}", msgtype),
|
||||
// }
|
||||
}
|
||||
Err(e) => {
|
||||
log::warn!(
|
||||
"Invalid JSON from {}: {} - Error: {}",
|
||||
peer_addr,
|
||||
line.trim(),
|
||||
e
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(e) => {
|
||||
log::error!("Failed to read from {}: {}", peer_addr, e);
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
log::debug!("Connection handler for {} exiting", peer_addr);
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
use thiserror::Error;
|
||||
use workshop_engine::HasExitCode;
|
||||
|
||||
#[derive(Error, Debug)]
|
||||
pub enum CliError {
|
||||
#[error("{0}")]
|
||||
Prebake(#[from] crate::prebake::error::PrebakeError),
|
||||
#[error("{0}")]
|
||||
Bake(#[from] crate::bake::error::BakeError),
|
||||
#[error("{0}")]
|
||||
Finalize(#[from] crate::finalize::FinalizeError),
|
||||
#[error("{0}")]
|
||||
General(anyhow::Error),
|
||||
}
|
||||
|
||||
impl From<anyhow::Error> for CliError {
|
||||
fn from(e: anyhow::Error) -> Self {
|
||||
CliError::General(e)
|
||||
}
|
||||
}
|
||||
impl CliError {
|
||||
pub fn exit_code(&self) -> i32 {
|
||||
match self {
|
||||
CliError::Prebake(e) => e.exit_code(),
|
||||
CliError::Bake(e) => e.exit_code(),
|
||||
CliError::Finalize(e) => e.exit_code(),
|
||||
CliError::General(_) => 1,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -8,12 +8,10 @@ pub mod template;
|
||||
pub mod types;
|
||||
|
||||
use crate::cli::Cli;
|
||||
use crate::finalize::constant::FINALIZE_STANDALONE_PLUGIN_PATH;
|
||||
use crate::finalize::plugin::fetch::{self, FetchArgument};
|
||||
use crate::types::resource::ResourceRegistry;
|
||||
pub use config::*;
|
||||
pub use error::FinalizeError;
|
||||
use std::path::Path;
|
||||
use std::path::PathBuf;
|
||||
use workshop_engine::EventSender;
|
||||
|
||||
pub fn parse(path: &Path) -> Result<FinalizeConfig, FinalizeError> {
|
||||
@@ -27,6 +25,7 @@ pub async fn finalize(
|
||||
_cli: &Cli,
|
||||
ctx: &mut workshop_engine::ExecutionContext,
|
||||
event_tx: EventSender,
|
||||
registry: &ResourceRegistry,
|
||||
) -> Result<(), FinalizeError> {
|
||||
let mut finalize = parse(finalize_path)?;
|
||||
let validate_result = finalize.validate();
|
||||
@@ -36,33 +35,10 @@ pub async fn finalize(
|
||||
}
|
||||
return Err(FinalizeError::ValidateError(errors));
|
||||
};
|
||||
if ctx.standalone {
|
||||
// In standalone mode, finalize should download plugin by itself
|
||||
log::debug!("Downloading {} plugins", finalize.plugin.len());
|
||||
let plugin_path = PathBuf::from(FINALIZE_STANDALONE_PLUGIN_PATH);
|
||||
// if plugin_path.exists() {
|
||||
// std::fs::remove_dir_all(&plugin_path).map_err(FinalizeError::IoError)?;
|
||||
// }
|
||||
for (name, data) in finalize.plugin.iter_mut() {
|
||||
log::info!("Downloading plugin {}", name);
|
||||
let (source, config) = data.iter_mut().next().unwrap_or_else(|| {
|
||||
panic!(
|
||||
"Internal Error: Plugin {} has no source configured, this should have been caught by validation!",
|
||||
name
|
||||
)
|
||||
});
|
||||
let fetch_argument = FetchArgument {
|
||||
checksum: config.checksum.clone(),
|
||||
shallow: config.shallow,
|
||||
};
|
||||
fetch::fetch_plugin(source, name, &plugin_path, fetch_argument).await?;
|
||||
config.runtime.fspath = Some(plugin_path.join(name));
|
||||
}
|
||||
}
|
||||
|
||||
// Register Plugins
|
||||
// Register Plugins (paths resolved from registry)
|
||||
log::info!("Registering {} plugins", finalize.plugin.len());
|
||||
let registered_plugins = plugin::register(finalize.plugin, None)?;
|
||||
let registered_plugins = plugin::register(finalize.plugin, None, &mut finalize.notification, registry)?;
|
||||
|
||||
// Early hook
|
||||
let earlyhook_result = stage::hook::hook(
|
||||
|
||||
@@ -1,56 +1,52 @@
|
||||
# workshop-baker/src/finalize
|
||||
|
||||
**Generated:** 2026-04-22
|
||||
**Commit:** 7b3b71a
|
||||
**Generated:** 2026-05-19
|
||||
**Commit:** 28abc5d
|
||||
|
||||
Artifact packaging, distribution, deployment, and notifications.
|
||||
Artifact packaging, plugin-based notifications, and post-build hooks.
|
||||
|
||||
## STRUCTURE
|
||||
|
||||
```
|
||||
finalize/
|
||||
├── config.rs # FinalizeConfig YAML (artifacts, deploy, notify)
|
||||
├── config.rs # FinalizeConfig YAML (379 lines, plugins, notifications)
|
||||
├── stage.rs # FinalizeStage enum
|
||||
├── stage/hook.rs # Post-build hooks
|
||||
├── stage/hook.rs # Post-build hook execution
|
||||
├── event.rs # Event propagation
|
||||
├── template.rs # Variable substitution {{VAR}}
|
||||
├── plugin.rs # Plugin trait + registry
|
||||
├── template.rs # Variable substitution {{VARIABLE}}
|
||||
├── plugin.rs # Plugin trait + registration
|
||||
├── plugin/
|
||||
│ ├── metadata.rs # Plugin manifest parsing
|
||||
│ ├── dylib.rs # Dynamic library plugins
|
||||
│ ├── rhai.rs # Rhai scripting (todo!())
|
||||
│ ├── shell.rs # Shell command plugins
|
||||
│ └── internal/ # Built-in plugins
|
||||
│ ├── mail/ # Email notifications (SMTP)
|
||||
│ ├── webhook.rs # HTTP webhooks
|
||||
│ ├── satori.rs # Satori integration
|
||||
│ └── insitenotify.rs # In-site notifications
|
||||
│ └── fetch/ # Artifact fetching
|
||||
│ ├── git.rs # Git repository fetch
|
||||
│ ├── http.rs # HTTP download
|
||||
│ ├── extract.rs # Archive extraction
|
||||
│ └── checksum.rs# SHA256 verification (SHA1 deprecated)
|
||||
│ └── internal.rs + internal/ # Built-in plugins (mail, webhook, satori, insitenotify)
|
||||
├── types.rs
|
||||
├── types/compression.rs
|
||||
├── error.rs # FinalizeError
|
||||
└── constant.rs
|
||||
```
|
||||
|
||||
## WHERE TO LOOK
|
||||
|
||||
| Task | Location |
|
||||
|------|----------|
|
||||
| Config loading | `config.rs` - FinalizeConfig |
|
||||
| Email notify | `plugin/internal/mail/` - SMTP templates |
|
||||
| Config loading | `config.rs:17` - `FinalizeConfig` + `parse()` |
|
||||
| Plugin registration | `plugin.rs` - `register()` |
|
||||
| Email notifications | `plugin/internal/mail.rs` - SMTP |
|
||||
| Webhooks | `plugin/internal/webhook.rs` |
|
||||
| Artifact fetch | `plugin/fetch/` - git/http/extract |
|
||||
| Checksums | `plugin/fetch/checksum.rs:13` - SHA1 deprecated |
|
||||
| Satori integration | `plugin/internal/satori.rs` |
|
||||
| Shell plugins | `plugin/shell.rs` |
|
||||
| Template rendering | `template.rs` - `{{VARIABLE}}` handlebars-style |
|
||||
|
||||
## CONVENTIONS
|
||||
|
||||
- **Plugin System**: Dynamic + internal plugins via Plugin trait
|
||||
- **Template Syntax**: `{{VARIABLE}}` handlebars-style
|
||||
- **Stages**: Hook → Fetch → Build → Publish → Notify → Cleanup
|
||||
- **Fetch**: Supports git, http, with extraction and checksum verify
|
||||
- **Plugin System**: `FinalizePlugin` trait + registry with both internal and dynamic plugins
|
||||
- **Template Syntax**: `{{VARIABLE}}` handlebars-style substitution via `template.rs`
|
||||
- **Stages**: EarlyHook → (Artifacts) → LateHook → Notifications
|
||||
- **Notification**: `notify.rs` in parent module drives plugin-based notification dispatch
|
||||
- **ResourceRegistry**: Shared resource cache passed through to plugins
|
||||
|
||||
## ANTI-PATTERNS
|
||||
|
||||
1. **SHA1 deprecated** — `checksum.rs:13` warns SHA1 is deprecated
|
||||
2. **Unimplemented** — `rhai.rs:17` - "todo!()" for Rhai scripting
|
||||
3. **TODO markers** — Multiple TODOs in config.rs (lines 305-306)
|
||||
1. **fetch/ subdirectory removed** — was `plugin/fetch/{git,http,extract,checksum}`, no longer exists
|
||||
2. **SHA1 deprecated** — workshop-getterurl handles checksums now
|
||||
3. **Unimplemented** — `rhai.rs:17` - "todo!()" for Rhai scripting
|
||||
4. **TODO markers** — Multiple TODOs in config.rs (lines 305-306)
|
||||
5. **Artifact stage not implemented** — `finalize.rs:57` — "TODO: artifact stage implementation"
|
||||
|
||||
@@ -115,6 +115,17 @@ pub struct NotificationMethod {
|
||||
pub policy: Vec<NotifyPolicy>,
|
||||
}
|
||||
|
||||
impl Default for NotificationMethod {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
template: NotificationTemplate::default(),
|
||||
trigger: Vec::new(),
|
||||
config: Value::default(),
|
||||
policy: Vec::new(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize, Serialize, Clone, Default)]
|
||||
pub struct NotificationTemplate {
|
||||
/// Payload schema: "default" (plugin-defined) or "custom" (requires on_* templates)
|
||||
@@ -134,17 +145,59 @@ pub struct NotificationTemplate {
|
||||
pub on_finish_of: Option<OnFinishOf>,
|
||||
}
|
||||
|
||||
impl std::ops::AddAssign for NotificationTemplate {
|
||||
fn add_assign(&mut self, other: Self) {
|
||||
if self.schema.is_none() {
|
||||
self.schema = other.schema;
|
||||
}
|
||||
if self.on_success.is_none() {
|
||||
self.on_success = other.on_success;
|
||||
}
|
||||
if self.on_failure.is_none() {
|
||||
self.on_failure = other.on_failure;
|
||||
}
|
||||
if self.on_finish_of.is_none() {
|
||||
self.on_finish_of = other.on_finish_of;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Notification configuration container
|
||||
#[derive(Debug, Deserialize, Serialize, Clone, Default)]
|
||||
#[derive(Debug, Serialize, Clone, Default)]
|
||||
pub struct NotificationConfig {
|
||||
/// Template definitions for notification content
|
||||
#[serde(default)]
|
||||
pub templates: HashMap<String, NotificationTemplateDef>,
|
||||
|
||||
#[serde(flatten)]
|
||||
/// Priority groups (YAML keys like `0`, `1` are parsed as u32).
|
||||
/// Custom Deserialize handles serde_yaml string→u32 conversion.
|
||||
#[serde(default)]
|
||||
pub groups: HashMap<u32, HashMap<String, NotificationMethod>>,
|
||||
}
|
||||
|
||||
/// Serde helper: YAML map keys are always strings, convert to u32.
|
||||
impl<'de> serde::Deserialize<'de> for NotificationConfig {
|
||||
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
|
||||
where D: serde::Deserializer<'de>
|
||||
{
|
||||
#[derive(serde::Deserialize)]
|
||||
struct Helper {
|
||||
#[serde(default)]
|
||||
templates: HashMap<String, NotificationTemplateDef>,
|
||||
#[serde(flatten)]
|
||||
groups_raw: HashMap<String, HashMap<String, NotificationMethod>>,
|
||||
}
|
||||
let h = Helper::deserialize(deserializer)?;
|
||||
let groups: HashMap<u32, _> = h.groups_raw.into_iter()
|
||||
.map(|(k, v)| {
|
||||
k.parse::<u32>().map(|n| (n, v))
|
||||
.map_err(|_| serde::de::Error::custom(format!("priority key must be integer, got {:?}", k)))
|
||||
})
|
||||
.collect::<Result<_, _>>()?;
|
||||
Ok(NotificationConfig { templates: h.templates, groups })
|
||||
}
|
||||
}
|
||||
|
||||
/// Stage-triggered notification configuration
|
||||
#[derive(Debug, Deserialize, Serialize, Clone)]
|
||||
pub struct OnFinishOf {
|
||||
@@ -158,7 +211,7 @@ pub struct OnFinishOf {
|
||||
|
||||
/// Notification template definition.
|
||||
/// Supports inline content (on_success/on_failure/on_finish_of) or external reference (use).
|
||||
#[derive(Debug, Deserialize, Serialize, Clone)]
|
||||
#[derive(Debug, Deserialize, Serialize, Clone, Default)]
|
||||
pub struct NotificationTemplateDef {
|
||||
/// External template reference (git URL).
|
||||
/// Cannot be combined with on_* fields in MVP (validation will reject).
|
||||
@@ -178,6 +231,53 @@ pub struct NotificationTemplateDef {
|
||||
pub on_finish_of: Option<String>,
|
||||
}
|
||||
|
||||
impl NotificationTemplateDef {
|
||||
/// Merge `other` into `self`, keeping self's values when present.
|
||||
///
|
||||
/// `other` fills in gaps where `self` is `None`.
|
||||
/// Chaining: `a.apply(b).apply(c)` = a wins, then b fills gaps, then c fills gaps.
|
||||
pub fn apply(mut self, other: Self) -> Self {
|
||||
if self.on_success.is_none() {
|
||||
self.on_success = other.on_success;
|
||||
}
|
||||
if self.on_failure.is_none() {
|
||||
self.on_failure = other.on_failure;
|
||||
}
|
||||
if self.on_finish_of.is_none() {
|
||||
self.on_finish_of = other.on_finish_of;
|
||||
}
|
||||
self
|
||||
}
|
||||
}
|
||||
|
||||
/// Builder for [`NotificationTemplateDef`].
|
||||
pub struct NotificationTemplateDefBuilder(NotificationTemplateDef);
|
||||
|
||||
impl NotificationTemplateDefBuilder {
|
||||
pub fn new() -> Self {
|
||||
Self(NotificationTemplateDef::default())
|
||||
}
|
||||
pub fn with_use(mut self, val: impl Into<String>) -> Self {
|
||||
self.0.use_ = Some(val.into());
|
||||
self
|
||||
}
|
||||
pub fn with_on_success(mut self, val: impl Into<String>) -> Self {
|
||||
self.0.on_success = Some(val.into());
|
||||
self
|
||||
}
|
||||
pub fn with_on_failure(mut self, val: impl Into<String>) -> Self {
|
||||
self.0.on_failure = Some(val.into());
|
||||
self
|
||||
}
|
||||
pub fn with_on_finish_of(mut self, val: impl Into<String>) -> Self {
|
||||
self.0.on_finish_of = Some(val.into());
|
||||
self
|
||||
}
|
||||
pub fn build(self) -> NotificationTemplateDef {
|
||||
self.0
|
||||
}
|
||||
}
|
||||
|
||||
/// Notification trigger type.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum Trigger {
|
||||
@@ -336,15 +436,18 @@ impl FinalizeConfig {
|
||||
fn validate_notification(&self, errors: &mut Vec<String>) {
|
||||
// Validate templates
|
||||
for (name, tmpl) in &self.notification.templates {
|
||||
if tmpl.use_.is_some()
|
||||
&& (tmpl.on_success.is_some()
|
||||
|| tmpl.on_failure.is_some()
|
||||
|| tmpl.on_finish_of.is_some())
|
||||
{
|
||||
errors.push(format!(
|
||||
"Template '{}' cannot combine 'use' with 'on_*' fields (MVP restriction)",
|
||||
name
|
||||
));
|
||||
if let Some(ref u) = tmpl.use_ {
|
||||
if u.contains("://") && (
|
||||
tmpl.on_success.is_some() ||
|
||||
tmpl.on_failure.is_some() ||
|
||||
tmpl.on_finish_of.is_some()
|
||||
) {
|
||||
errors.push(format!(
|
||||
"Template '{}' cannot combine external 'use' with 'on_*' fields",
|
||||
name
|
||||
));
|
||||
}
|
||||
// Local references (no ://) are allowed to have on_* overrides
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,14 +1,15 @@
|
||||
use crate::finalize::RawPluginMap;
|
||||
use crate::finalize::constant::{FINALIZE_PLUGIN_MANIFEST, FINALIZE_PLUGIN_MANIFEST_EXTENSION};
|
||||
use crate::finalize::plugin::metadata::{PluginMetadata, PluginType};
|
||||
use crate::{EventSender, ExecutionContext, finalize::error::PluginError};
|
||||
use workshop_engine::{EventSender, ExecutionContext};
|
||||
use crate::{ finalize::error::PluginError};
|
||||
use crate::types::resource::ResourceRegistry;
|
||||
use async_trait::async_trait;
|
||||
use std::collections::{HashMap, HashSet};
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::Arc;
|
||||
|
||||
mod dylib;
|
||||
pub mod fetch;
|
||||
mod internal;
|
||||
mod metadata;
|
||||
mod rhai;
|
||||
@@ -214,6 +215,8 @@ fn get_entry(plugin_type: PluginType, name: &str) -> Result<Box<dyn AsyncPluginF
|
||||
pub fn register(
|
||||
external_plugins: RawPluginMap,
|
||||
list: Option<&HashSet<String>>,
|
||||
notification_config: &mut crate::finalize::NotificationConfig,
|
||||
registry: &ResourceRegistry,
|
||||
) -> Result<PluginMap, PluginError> {
|
||||
let mut plugins: PluginMap = HashMap::new();
|
||||
let mut count = 0;
|
||||
@@ -255,7 +258,14 @@ pub fn register(
|
||||
log::warn!("Plugin {} is not available: {}", name, reason);
|
||||
continue;
|
||||
}
|
||||
let metadata = read_manifest(config.runtime.fspath, &name)?;
|
||||
// Resolve plugin path from registry instead of config.runtime.fspath
|
||||
let plugin_path = registry.resolve(&name).ok_or_else(|| {
|
||||
PluginError::PluginUnavailable {
|
||||
name: name.clone(),
|
||||
reason: "plugin not found in resource registry".to_string(),
|
||||
}
|
||||
})?;
|
||||
let metadata = read_manifest(Some(plugin_path), &name)?;
|
||||
let plugin_type = get_plugin_type(metadata.plugin_type.clone(), &metadata.entrypoint);
|
||||
let entry = get_entry(plugin_type, &name)?;
|
||||
plugins.insert(
|
||||
@@ -267,8 +277,18 @@ pub fn register(
|
||||
count += 1;
|
||||
}
|
||||
|
||||
log::info!("Registered {} plugins", count);
|
||||
// Inject plugin-declared templates as .fallback
|
||||
for (name, plugin) in &plugins {
|
||||
if let Some(ref templates) = plugin.metadata.templates {
|
||||
let key = format!("{}.fallback", name);
|
||||
for tmpl in templates {
|
||||
notification_config.templates.insert(key.clone(), tmpl.clone());
|
||||
}
|
||||
log::debug!("Registered templates from plugin '{}' as '{}'", name, key);
|
||||
}
|
||||
}
|
||||
|
||||
log::info!("Registered {} plugins", count);
|
||||
Ok(plugins)
|
||||
}
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
use crate::finalize::plugin::PluginError;
|
||||
use crate::finalize::plugin::PluginMetadata;
|
||||
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn};
|
||||
use workshop_engine::{EventSender, ExecutionContext};
|
||||
use crate::{ finalize::plugin::AsyncPluginFn};
|
||||
use async_trait::async_trait;
|
||||
|
||||
pub struct Dylib;
|
||||
|
||||
@@ -1,109 +0,0 @@
|
||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
||||
use crate::finalize::fetch;
|
||||
use crate::finalize::plugin::{PluginError, RawPluginMap};
|
||||
use std::collections::HashSet;
|
||||
use std::path::Path;
|
||||
|
||||
mod checksum;
|
||||
mod extract;
|
||||
mod git;
|
||||
mod http;
|
||||
mod types;
|
||||
|
||||
pub use types::FetchArgument;
|
||||
|
||||
enum URLScheme {
|
||||
Unknown,
|
||||
Git,
|
||||
Http,
|
||||
Https,
|
||||
File,
|
||||
}
|
||||
|
||||
impl URLScheme {
|
||||
fn parse(url: &str) -> (Self, String) {
|
||||
match url.split_once("://") {
|
||||
Some((prefix, rest)) => {
|
||||
let scheme = match prefix {
|
||||
"git" => URLScheme::Git,
|
||||
"https" => URLScheme::Https,
|
||||
"http" => URLScheme::Http,
|
||||
"file" => URLScheme::File,
|
||||
_ => URLScheme::Unknown,
|
||||
};
|
||||
(scheme, rest.to_string())
|
||||
}
|
||||
None => (URLScheme::Unknown, url.to_string()),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn fetch_plugin(
|
||||
url: &str,
|
||||
name: &str,
|
||||
basedir: &Path,
|
||||
argument: FetchArgument,
|
||||
) -> Result<(), PluginError> {
|
||||
let (scheme, path) = URLScheme::parse(url);
|
||||
|
||||
match scheme {
|
||||
URLScheme::Git => {
|
||||
// NOTE: as a practical compromise, we use git command for git clone
|
||||
// gix will be supported later, with ONLY https scheme
|
||||
// TODO: Implement gix
|
||||
log::info!("Fetching git plugin: {}", url);
|
||||
git::fetch_git_plugin(&path, name, basedir, argument.shallow).await
|
||||
}
|
||||
URLScheme::Http | URLScheme::Https => {
|
||||
log::info!("Fetching http plugin: {}", url);
|
||||
let checksum = argument.get_checksum();
|
||||
http::fetch_http_plugin(url, name, basedir, checksum).await
|
||||
}
|
||||
URLScheme::File => {
|
||||
// TODO: Support client mode
|
||||
// NOTE: In standalone mode, we just assume path is well-defined and utilize
|
||||
log::info!("Fetching file plugin: {}", url);
|
||||
Ok(())
|
||||
}
|
||||
URLScheme::Unknown => {
|
||||
log::error!("Unknown URL scheme in plugin URL: {}", url);
|
||||
Err(PluginError::GeneralError(format!(
|
||||
"Unknown URL scheme in plugin URL: {}",
|
||||
url
|
||||
)))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn fetch_plugins(
|
||||
plugin_path: &Path,
|
||||
plugin_config: &mut RawPluginMap,
|
||||
list: Option<&HashSet<String>>,
|
||||
) -> Result<(), PluginError> {
|
||||
log::info!("Preparing to download {} plugins", plugin_config.len());
|
||||
for (name, data) in plugin_config.iter_mut() {
|
||||
match list {
|
||||
Some(list) if !list.contains(name) => {
|
||||
log::debug!("Plugin {} not in list, skipping", name);
|
||||
continue;
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
log::debug!("Downloading plugin {}", name);
|
||||
let (source, config) =
|
||||
data.iter_mut()
|
||||
.next()
|
||||
.ok_or_else(|| PluginError::PluginUnavailable {
|
||||
name: name.clone(),
|
||||
reason: "no source configured".to_string(),
|
||||
})?;
|
||||
let fetch_argument = FetchArgument {
|
||||
checksum: config.checksum.clone(),
|
||||
shallow: config.shallow,
|
||||
};
|
||||
fetch::fetch_plugin(source, name, plugin_path, fetch_argument).await?;
|
||||
log::debug!("Storing plugin {} to {}", name, plugin_path.display());
|
||||
config.runtime.fspath = Some(plugin_path.join(name));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
@@ -1,137 +0,0 @@
|
||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
||||
use super::types::ChecksumType;
|
||||
use sha2::{Digest, Sha256, Sha512};
|
||||
|
||||
/// Detects the checksum type based on the length of the checksum string.
|
||||
///
|
||||
/// Returns `Ok(None)` for empty string (checksum disabled).
|
||||
/// Returns `Ok(Some(ChecksumType))` for valid SHA256 (64 chars) or SHA512 (128 chars).
|
||||
/// Returns `Err` for invalid lengths (32=MD5, 40=SHA1) or unsupported lengths.
|
||||
pub fn detect_checksum_type(checksum: &str) -> Result<Option<ChecksumType>, String> {
|
||||
match checksum.len() {
|
||||
0 => Ok(None),
|
||||
32 => Err(format!(
|
||||
"MD5 (first 8 chars: {}) is not supported for security reasons. Use SHA256 (64 chars) or SHA512 (128 chars)",
|
||||
&checksum[..8.min(checksum.len())]
|
||||
)),
|
||||
40 => Err(format!(
|
||||
"SHA1 (first 8 chars: {}) is deprecated and not supported. Use SHA256 (64 chars) or SHA512 (128 chars)",
|
||||
&checksum[..8.min(checksum.len())]
|
||||
)),
|
||||
64 => Ok(Some(ChecksumType::Sha256)),
|
||||
128 => Ok(Some(ChecksumType::Sha512)),
|
||||
_ => Err(format!(
|
||||
"Invalid checksum length: {}. Expected 0 (disabled), 64 (SHA256), or 128 (SHA512). First 8 chars: {}",
|
||||
checksum.len(),
|
||||
&checksum[..8.min(checksum.len())]
|
||||
)),
|
||||
}
|
||||
}
|
||||
|
||||
/// Verifies that the data matches the expected checksum.
|
||||
///
|
||||
/// Compares the computed hash of `data` against `expected` using the specified `checksum_type`.
|
||||
/// Returns `Ok(())` if the checksums match, `Err` otherwise.
|
||||
pub fn verify_checksum(
|
||||
data: &[u8],
|
||||
expected: &str,
|
||||
checksum_type: ChecksumType,
|
||||
) -> Result<(), String> {
|
||||
let actual = match checksum_type {
|
||||
ChecksumType::Sha256 => {
|
||||
let mut hasher = Sha256::new();
|
||||
hasher.update(data);
|
||||
format!("{:x}", hasher.finalize())
|
||||
}
|
||||
ChecksumType::Sha512 => {
|
||||
let mut hasher = Sha512::new();
|
||||
hasher.update(data);
|
||||
format!("{:x}", hasher.finalize())
|
||||
}
|
||||
};
|
||||
|
||||
if actual.eq_ignore_ascii_case(expected) {
|
||||
Ok(())
|
||||
} else {
|
||||
Err(format!(
|
||||
"Checksum mismatch: expected {}..., got {}...",
|
||||
&expected[..8.min(expected.len())],
|
||||
&actual[..8.min(actual.len())]
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_detect_checksum_type_empty() {
|
||||
assert_eq!(detect_checksum_type("").unwrap(), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_detect_checksum_type_md5_rejected() {
|
||||
let result = detect_checksum_type("a".repeat(32).as_str());
|
||||
assert!(result.is_err());
|
||||
assert!(result.unwrap_err().contains("MD5"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_detect_checksum_type_sha1_rejected() {
|
||||
let result = detect_checksum_type("a".repeat(40).as_str());
|
||||
assert!(result.is_err());
|
||||
assert!(result.unwrap_err().contains("SHA1"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_detect_checksum_type_sha256() {
|
||||
assert_eq!(
|
||||
detect_checksum_type("a".repeat(64).as_str()).unwrap(),
|
||||
Some(ChecksumType::Sha256)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_detect_checksum_type_sha512() {
|
||||
assert_eq!(
|
||||
detect_checksum_type("a".repeat(128).as_str()).unwrap(),
|
||||
Some(ChecksumType::Sha512)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_detect_checksum_type_invalid_length() {
|
||||
let result = detect_checksum_type("a".repeat(50).as_str());
|
||||
assert!(result.is_err());
|
||||
assert!(result.unwrap_err().contains("Invalid checksum length"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_verify_checksum_sha256_valid() {
|
||||
let data = b"hello";
|
||||
let expected = "2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824";
|
||||
assert!(verify_checksum(data, expected, ChecksumType::Sha256).is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_verify_checksum_sha256_invalid() {
|
||||
let data = b"hello";
|
||||
let expected = "0000000000000000000000000000000000000000000000000000000000000000";
|
||||
assert!(verify_checksum(data, expected, ChecksumType::Sha256).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_verify_checksum_sha512_valid() {
|
||||
let data = b"hello";
|
||||
let expected = "9b71d224bd62f3785d96d46ad3ea3d73319bfbc2890caadae2dff72519673ca72323c3d99ba5c11d7c7acc6e14b8c5da0c4663475c2e5c3adef46f73bcdec043";
|
||||
assert!(verify_checksum(data, expected, ChecksumType::Sha512).is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_verify_checksum_sha512_invalid() {
|
||||
let data = b"hello";
|
||||
let expected = "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000";
|
||||
assert!(verify_checksum(data, expected, ChecksumType::Sha512).is_err());
|
||||
}
|
||||
}
|
||||
@@ -1,169 +0,0 @@
|
||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
||||
use crate::finalize::plugin::PluginError;
|
||||
use std::path::{Path, PathBuf};
|
||||
use tokio::task;
|
||||
|
||||
/// Compression type for tar archives.
|
||||
#[derive(Debug, Clone, Copy, PartialEq)]
|
||||
pub enum CompressionType {
|
||||
/// Gzip compressed (.tar.gz, .tgz)
|
||||
Gzip,
|
||||
/// Zstd compressed (.tar.zst, .tar.zstd, .tzst)
|
||||
Zstd,
|
||||
/// Uncompressed tar (.tar)
|
||||
None,
|
||||
}
|
||||
|
||||
impl CompressionType {
|
||||
/// Detects compression type from file path extension.
|
||||
///
|
||||
/// Supports: .tar.gz, .tgz (Gzip), .tar.zst, .tar.zstd, .tzst (Zstd), .tar (None).
|
||||
pub fn from_path(path: &std::path::Path) -> Result<Self, String> {
|
||||
let name = path
|
||||
.file_name()
|
||||
.and_then(|n| n.to_str())
|
||||
.unwrap_or("")
|
||||
.to_lowercase();
|
||||
|
||||
if name.ends_with(".tar.gz") || name.ends_with(".tgz") {
|
||||
Ok(CompressionType::Gzip)
|
||||
} else if name.ends_with(".tar.zst")
|
||||
|| name.ends_with(".tar.zstd")
|
||||
|| name.ends_with(".tzst")
|
||||
{
|
||||
Ok(CompressionType::Zstd)
|
||||
} else if name.ends_with(".tar") {
|
||||
Ok(CompressionType::None)
|
||||
} else {
|
||||
Err(format!(
|
||||
"Unsupported archive format: '{}'. Supported formats: .tar, .tar.gz, .tgz, .tar.zst, .tar.zstd, .tzst",
|
||||
name
|
||||
))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Extracts a tar archive to the destination directory.
|
||||
///
|
||||
/// Removes existing destination directory if present, creates a fresh directory,
|
||||
/// then unpacks the archive using the appropriate decompression based on `compression`.
|
||||
pub async fn extract_tar(
|
||||
archive: &Path,
|
||||
destdir: &PathBuf,
|
||||
compression: CompressionType,
|
||||
) -> Result<(), PluginError> {
|
||||
if destdir.exists() {
|
||||
tokio::fs::remove_dir_all(destdir).await.map_err(|e| {
|
||||
PluginError::GeneralError(format!("Failed to remove existing directory: {}", e))
|
||||
})?;
|
||||
}
|
||||
tokio::fs::create_dir_all(destdir)
|
||||
.await
|
||||
.map_err(|e| PluginError::GeneralError(format!("Failed to create directory: {}", e)))?;
|
||||
|
||||
let archive = archive.to_path_buf();
|
||||
let destdir = destdir.clone();
|
||||
|
||||
task::spawn_blocking(move || {
|
||||
let file = std::fs::File::open(&archive)
|
||||
.map_err(|e| PluginError::GeneralError(format!("Failed to open archive: {}", e)))?;
|
||||
|
||||
match compression {
|
||||
CompressionType::Gzip => {
|
||||
let decoder = flate2::read::GzDecoder::new(file);
|
||||
let mut archive = tar::Archive::new(decoder);
|
||||
archive.unpack(&destdir).map_err(|e| {
|
||||
PluginError::GeneralError(format!("Failed to unpack archive: {}", e))
|
||||
})?;
|
||||
}
|
||||
CompressionType::Zstd => {
|
||||
let decoder = zstd::stream::read::Decoder::new(file).map_err(|e| {
|
||||
PluginError::GeneralError(format!("Failed to create zstd decoder: {}", e))
|
||||
})?;
|
||||
let mut archive = tar::Archive::new(decoder);
|
||||
archive.unpack(&destdir).map_err(|e| {
|
||||
PluginError::GeneralError(format!("Failed to unpack archive: {}", e))
|
||||
})?;
|
||||
}
|
||||
CompressionType::None => {
|
||||
let mut archive = tar::Archive::new(file);
|
||||
archive.unpack(&destdir).map_err(|e| {
|
||||
PluginError::GeneralError(format!("Failed to unpack archive: {}", e))
|
||||
})?;
|
||||
}
|
||||
};
|
||||
|
||||
Ok::<(), PluginError>(())
|
||||
})
|
||||
.await
|
||||
.map_err(|e| PluginError::GeneralError(format!("Task join error: {}", e)))??;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::path::Path;
|
||||
|
||||
#[test]
|
||||
fn test_from_path_tar_gz() {
|
||||
assert_eq!(
|
||||
CompressionType::from_path(Path::new("archive.tar.gz")).unwrap(),
|
||||
CompressionType::Gzip
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_from_path_tgz() {
|
||||
assert_eq!(
|
||||
CompressionType::from_path(Path::new("archive.tgz")).unwrap(),
|
||||
CompressionType::Gzip
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_from_path_tar_zst() {
|
||||
assert_eq!(
|
||||
CompressionType::from_path(Path::new("archive.tar.zst")).unwrap(),
|
||||
CompressionType::Zstd
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_from_path_tar_zstd() {
|
||||
assert_eq!(
|
||||
CompressionType::from_path(Path::new("archive.tar.zstd")).unwrap(),
|
||||
CompressionType::Zstd
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_from_path_tzst() {
|
||||
assert_eq!(
|
||||
CompressionType::from_path(Path::new("archive.tzst")).unwrap(),
|
||||
CompressionType::Zstd
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_from_path_tar() {
|
||||
assert_eq!(
|
||||
CompressionType::from_path(Path::new("archive.tar")).unwrap(),
|
||||
CompressionType::None
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_from_path_unsupported() {
|
||||
assert!(CompressionType::from_path(Path::new("archive.zip")).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_from_path_case_insensitive() {
|
||||
assert_eq!(
|
||||
CompressionType::from_path(Path::new("archive.TAR.GZ")).unwrap(),
|
||||
CompressionType::Gzip
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -1,185 +0,0 @@
|
||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
||||
use crate::finalize::plugin::PluginError;
|
||||
use std::path::Path;
|
||||
use tokio::process::Command;
|
||||
|
||||
/// Version specification for a git repository.
|
||||
#[derive(Debug, PartialEq)]
|
||||
pub enum GitVersion {
|
||||
/// No version specified (invalid for fetch operations)
|
||||
None,
|
||||
/// Single version string (tag or commit hash)
|
||||
Single(String),
|
||||
/// Mixed: try tag first, fallback to commit hash
|
||||
Mixed(String, String),
|
||||
}
|
||||
|
||||
/// Parses a git URL with optional version specification.
|
||||
///
|
||||
/// Format: `url` (no version), `url@tag` (single version), `url@tag:commit` (mixed version).
|
||||
/// Returns (base_url, GitVersion).
|
||||
pub fn parse_git_url(url: &str) -> (String, GitVersion) {
|
||||
if let Some((base, version)) = url.split_once('@') {
|
||||
if version.contains(":") {
|
||||
let (tag, commit) = version.split_once(':').unwrap();
|
||||
return (
|
||||
base.to_string(),
|
||||
GitVersion::Mixed(tag.to_string(), commit.to_string()),
|
||||
);
|
||||
}
|
||||
return (base.to_string(), GitVersion::Single(version.to_string()));
|
||||
}
|
||||
(url.to_string(), GitVersion::None)
|
||||
}
|
||||
|
||||
/// Clones a git repository and checks out the specified version.
|
||||
///
|
||||
/// Removes existing destination directory, performs shallow or full clone,
|
||||
/// then checks out the given version (tag or commit).
|
||||
pub async fn git_clone_and_checkout(
|
||||
baseurl: &str,
|
||||
version: &str,
|
||||
dest: &Path,
|
||||
name: &str,
|
||||
shallow: bool,
|
||||
) -> Result<(), PluginError> {
|
||||
if dest.exists() {
|
||||
tokio::fs::remove_dir_all(dest)
|
||||
.await
|
||||
.map_err(|e| PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: baseurl.to_string(),
|
||||
reason: format!("Failed to remove existing directory: {}", e),
|
||||
})?;
|
||||
}
|
||||
let mut cmd = Command::new("git");
|
||||
cmd.arg("clone");
|
||||
if shallow {
|
||||
cmd.arg("--depth").arg("1");
|
||||
}
|
||||
cmd.arg(baseurl).arg(dest);
|
||||
let output = cmd.output().await.map_err(|e| PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: baseurl.to_string(),
|
||||
reason: e.to_string(),
|
||||
})?;
|
||||
if !output.status.success() {
|
||||
return Err(PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: baseurl.to_string(),
|
||||
reason: String::from_utf8_lossy(&output.stderr).to_string(),
|
||||
});
|
||||
}
|
||||
let mut cmd = Command::new("git");
|
||||
cmd.arg("checkout").arg(version).current_dir(dest);
|
||||
let output = cmd.output().await.map_err(|e| PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: baseurl.to_string(),
|
||||
reason: e.to_string(),
|
||||
})?;
|
||||
if !output.status.success() {
|
||||
return Err(PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: baseurl.to_string(),
|
||||
reason: String::from_utf8_lossy(&output.stderr).to_string(),
|
||||
});
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Fetches a plugin from a git repository.
|
||||
///
|
||||
/// Parses the URL for version info, then clones and checks out the appropriate ref.
|
||||
/// For Mixed version, tries the tag first and falls back to the commit hash.
|
||||
pub async fn fetch_git_plugin(
|
||||
path: &str,
|
||||
name: &str,
|
||||
basedir: &Path,
|
||||
shallow: Option<bool>,
|
||||
) -> Result<(), PluginError> {
|
||||
let (baseurl, version) = parse_git_url(path);
|
||||
match version {
|
||||
GitVersion::None => {
|
||||
log::error!("Version should be specified for git plugin URL: {}", path);
|
||||
Err(PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: path.to_string(),
|
||||
reason: "Version should be specified as either a tag or a commit hash.".to_string(),
|
||||
})
|
||||
}
|
||||
GitVersion::Single(version) => {
|
||||
log::debug!("Fetching git plugin: {}@{}", name, version);
|
||||
let shallow_flag = shallow.unwrap_or(false);
|
||||
git_clone_and_checkout(
|
||||
&baseurl,
|
||||
&version.to_string(),
|
||||
&basedir.join(name),
|
||||
name,
|
||||
shallow_flag,
|
||||
)
|
||||
.await
|
||||
}
|
||||
GitVersion::Mixed(tag, commit) => {
|
||||
log::debug!("Fetching git plugin: {}@{}/{}", name, tag, commit);
|
||||
let shallow_flag = shallow.unwrap_or(false);
|
||||
let result =
|
||||
git_clone_and_checkout(&baseurl, &tag, &basedir.join(name), name, shallow_flag)
|
||||
.await;
|
||||
if result.is_err() {
|
||||
log::warn!(
|
||||
"Failed to clone with tag, trying commit hash. URL: {}",
|
||||
path
|
||||
);
|
||||
return git_clone_and_checkout(&baseurl, &commit, &basedir.join(name), name, false)
|
||||
.await;
|
||||
}
|
||||
result
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_parse_git_url_no_version() {
|
||||
let (url, version) = parse_git_url("https://github.com/user/repo");
|
||||
assert_eq!(url, "https://github.com/user/repo");
|
||||
assert_eq!(version, GitVersion::None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_parse_git_url_single_tag() {
|
||||
let (url, version) = parse_git_url("https://github.com/user/repo@v1.0.0");
|
||||
assert_eq!(url, "https://github.com/user/repo");
|
||||
assert_eq!(version, GitVersion::Single("v1.0.0".to_string()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_parse_git_url_mixed_version() {
|
||||
let (url, version) = parse_git_url("https://github.com/user/repo@main:abc123");
|
||||
assert_eq!(url, "https://github.com/user/repo");
|
||||
assert_eq!(
|
||||
version,
|
||||
GitVersion::Mixed("main".to_string(), "abc123".to_string())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_parse_git_url_complex_url() {
|
||||
let (url, version) = parse_git_url("git@github.com:user/repo.git@tag:commit");
|
||||
assert_eq!(url, "git@github.com:user/repo.git");
|
||||
assert_eq!(
|
||||
version,
|
||||
GitVersion::Mixed("tag".to_string(), "commit".to_string())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_parse_git_url_single_with_at_in_path() {
|
||||
let (url, version) = parse_git_url("https://github.com/user/repo@my-repo@v1.0");
|
||||
assert_eq!(url, "https://github.com/user/repo@my-repo");
|
||||
assert_eq!(version, GitVersion::Single("v1.0".to_string()));
|
||||
}
|
||||
}
|
||||
@@ -1,110 +0,0 @@
|
||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
||||
use super::checksum::{detect_checksum_type, verify_checksum};
|
||||
use super::extract::{CompressionType, extract_tar};
|
||||
use crate::finalize::plugin::PluginError;
|
||||
use std::path::Path;
|
||||
|
||||
pub async fn download_to_file(url: &str, dest: &Path, name: &str) -> Result<(), PluginError> {
|
||||
let client = reqwest::Client::builder()
|
||||
.timeout(std::time::Duration::from_secs(300))
|
||||
.build()
|
||||
.map_err(|e| PluginError::GeneralError(format!("Failed to build HTTP client: {}", e)))?;
|
||||
|
||||
let response = client
|
||||
.get(url)
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: url.to_string(),
|
||||
reason: format!("HTTP request failed: {}", e),
|
||||
})?;
|
||||
|
||||
let status = response.status();
|
||||
if !status.is_success() {
|
||||
return Err(PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: url.to_string(),
|
||||
reason: format!("HTTP error: {}", status),
|
||||
});
|
||||
}
|
||||
|
||||
let bytes = response
|
||||
.bytes()
|
||||
.await
|
||||
.map_err(|e| PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: url.to_string(),
|
||||
reason: format!("Failed to read response body: {}", e),
|
||||
})?;
|
||||
|
||||
tokio::fs::write(dest, &bytes)
|
||||
.await
|
||||
.map_err(|e| PluginError::FetchError {
|
||||
name: name.to_string(),
|
||||
url: url.to_string(),
|
||||
reason: format!("Failed to write file: {}", e),
|
||||
})?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn extract_archive_extension(url: &str) -> &str {
|
||||
let path_part = url
|
||||
.split('?')
|
||||
.next()
|
||||
.unwrap_or(url)
|
||||
.split('#')
|
||||
.next()
|
||||
.unwrap_or(url);
|
||||
let file_part = path_part.rsplit('/').next().unwrap_or("");
|
||||
let lower = file_part.to_lowercase();
|
||||
if lower.ends_with(".tar.gz") || lower.ends_with(".tgz") {
|
||||
".tar.gz"
|
||||
} else if lower.ends_with(".tar.zst")
|
||||
|| lower.ends_with(".tar.zstd")
|
||||
|| lower.ends_with(".tzst")
|
||||
{
|
||||
".tar.zst"
|
||||
} else if lower.ends_with(".tar") {
|
||||
".tar"
|
||||
} else {
|
||||
".unknown"
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn fetch_http_plugin(
|
||||
url: &str,
|
||||
name: &str,
|
||||
basedir: &Path,
|
||||
checksum_str: Option<&str>,
|
||||
) -> Result<(), PluginError> {
|
||||
let temp_dir = std::env::temp_dir().join("workshop-plugins");
|
||||
tokio::fs::create_dir_all(&temp_dir).await.ok();
|
||||
let ext = extract_archive_extension(url);
|
||||
let temp_file = temp_dir.join(format!("{}-{}{}", name, uuid::Uuid::new_v4(), ext));
|
||||
|
||||
download_to_file(url, &temp_file, name).await?;
|
||||
|
||||
if let Some(checksum) = checksum_str
|
||||
&& !checksum.is_empty()
|
||||
{
|
||||
let bytes = tokio::fs::read(&temp_file).await.map_err(|e| {
|
||||
PluginError::GeneralError(format!("Failed to read downloaded file: {}", e))
|
||||
})?;
|
||||
|
||||
let checksum_type = detect_checksum_type(checksum).map_err(PluginError::GeneralError)?;
|
||||
|
||||
if let Some(ct) = checksum_type {
|
||||
verify_checksum(&bytes, checksum, ct).map_err(PluginError::GeneralError)?;
|
||||
}
|
||||
}
|
||||
|
||||
let compression = CompressionType::from_path(&temp_file).map_err(PluginError::GeneralError)?;
|
||||
let destdir = basedir.join(name);
|
||||
extract_tar(&temp_file, &destdir, compression).await?;
|
||||
|
||||
tokio::fs::remove_file(&temp_file).await.ok();
|
||||
|
||||
Ok(())
|
||||
}
|
||||
@@ -1,84 +0,0 @@
|
||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
/// Supported checksum types for verifying downloaded artifacts.
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub enum ChecksumType {
|
||||
/// SHA-256 hash (64 hex characters)
|
||||
Sha256,
|
||||
/// SHA-512 hash (128 hex characters)
|
||||
Sha512,
|
||||
}
|
||||
|
||||
/// Arguments for fetching a plugin artifact.
|
||||
///
|
||||
/// Contains optional checksum verification and shallow clone settings.
|
||||
#[derive(Debug, Deserialize, Serialize, Clone, Default)]
|
||||
pub struct FetchArgument {
|
||||
/// SHA256 or SHA512 hex string for integrity verification.
|
||||
#[serde(default)]
|
||||
pub checksum: Option<String>,
|
||||
/// Perform a shallow clone (single revision only).
|
||||
#[serde(default)]
|
||||
pub shallow: Option<bool>,
|
||||
}
|
||||
|
||||
impl FetchArgument {
|
||||
/// Returns the checksum if present and non-empty.
|
||||
pub fn get_checksum(&self) -> Option<&str> {
|
||||
self.checksum.as_deref().filter(|s| !s.is_empty())
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_get_checksum_some_non_empty() {
|
||||
let arg = FetchArgument {
|
||||
checksum: Some("abc123".to_string()),
|
||||
shallow: None,
|
||||
};
|
||||
assert_eq!(arg.get_checksum(), Some("abc123"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_get_checksum_empty_string() {
|
||||
let arg = FetchArgument {
|
||||
checksum: Some("".to_string()),
|
||||
shallow: None,
|
||||
};
|
||||
assert_eq!(arg.get_checksum(), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_get_checksum_none() {
|
||||
let arg = FetchArgument {
|
||||
checksum: None,
|
||||
shallow: None,
|
||||
};
|
||||
assert_eq!(arg.get_checksum(), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_serde_roundtrip() {
|
||||
let arg = FetchArgument {
|
||||
checksum: Some("deadbeef".to_string()),
|
||||
shallow: Some(true),
|
||||
};
|
||||
let json = serde_json::to_string(&arg).unwrap();
|
||||
let parsed: FetchArgument = serde_json::from_str(&json).unwrap();
|
||||
assert_eq!(parsed.checksum, arg.checksum);
|
||||
assert_eq!(parsed.shallow, arg.shallow);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_serde_roundtrip_empty() {
|
||||
let arg = FetchArgument::default();
|
||||
let json = serde_json::to_string(&arg).unwrap();
|
||||
let parsed: FetchArgument = serde_json::from_str(&json).unwrap();
|
||||
assert_eq!(parsed.checksum, arg.checksum);
|
||||
assert_eq!(parsed.shallow, arg.shallow);
|
||||
}
|
||||
}
|
||||
@@ -29,3 +29,10 @@ pub fn get_internal_plugin() -> Vec<InternalPlugin> {
|
||||
},
|
||||
]
|
||||
}
|
||||
|
||||
pub fn get_dummy_plugin() -> Vec<InternalPlugin> {
|
||||
vec![InternalPlugin {
|
||||
name: "dummy",
|
||||
entry: Box::new(dummy::Dummy),
|
||||
}]
|
||||
}
|
||||
|
||||
@@ -1,12 +1,13 @@
|
||||
use crate::finalize::plugin::PluginError;
|
||||
use crate::finalize::plugin::PluginMetadata;
|
||||
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn};
|
||||
use workshop_engine::{EventSender, ExecutionContext};
|
||||
use crate::{ finalize::plugin::AsyncPluginFn};
|
||||
use async_trait::async_trait;
|
||||
|
||||
pub struct _Dummy;
|
||||
pub struct Dummy;
|
||||
|
||||
#[async_trait]
|
||||
impl AsyncPluginFn for _Dummy {
|
||||
impl AsyncPluginFn for Dummy {
|
||||
async fn call(
|
||||
&self,
|
||||
_metadata: &PluginMetadata,
|
||||
@@ -14,6 +15,7 @@ impl AsyncPluginFn for _Dummy {
|
||||
_ctx: &ExecutionContext,
|
||||
_event_tx: &EventSender,
|
||||
) -> Result<(), PluginError> {
|
||||
log::debug!("[NOTIFY_DEBUG] Successfully registered dummy plugin.");
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,12 +1,10 @@
|
||||
use crate::finalize::plugin::PluginMetadata;
|
||||
use crate::{
|
||||
EventSender, ExecutionContext,
|
||||
finalize::{error::PluginError, plugin::AsyncPluginFn},
|
||||
};
|
||||
use async_trait::async_trait;
|
||||
|
||||
use workshop_engine::{EventSender, ExecutionContext};
|
||||
pub struct InsiteNotify;
|
||||
|
||||
#[async_trait]
|
||||
impl AsyncPluginFn for InsiteNotify {
|
||||
async fn call(
|
||||
@@ -16,6 +14,7 @@ impl AsyncPluginFn for InsiteNotify {
|
||||
_ctx: &ExecutionContext,
|
||||
_event_tx: &EventSender,
|
||||
) -> Result<(), PluginError> {
|
||||
unimplemented!("Requires workshop-base to be implemented!");
|
||||
log::info!("in-site-notify: placeholder (always succeeds)");
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,18 +1,9 @@
|
||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
||||
use crate::finalize::plugin::PluginError;
|
||||
use crate::finalize::plugin::PluginMetadata;
|
||||
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn};
|
||||
use crate::{
|
||||
finalize::{error::PluginError, plugin::AsyncPluginFn},
|
||||
};
|
||||
use async_trait::async_trait;
|
||||
use lettre::message::{MultiPart, SinglePart, header};
|
||||
use lettre::{AsyncSmtpTransport, AsyncTransport, Message, Tokio1Executor};
|
||||
use std::time::Duration;
|
||||
use tokio::time::timeout;
|
||||
|
||||
mod config;
|
||||
mod template;
|
||||
|
||||
pub use config::*;
|
||||
use template::*;
|
||||
use workshop_engine::{EventSender, ExecutionContext};
|
||||
|
||||
pub struct Mail;
|
||||
|
||||
@@ -21,182 +12,10 @@ impl AsyncPluginFn for Mail {
|
||||
async fn call(
|
||||
&self,
|
||||
_metadata: &PluginMetadata,
|
||||
argument: serde_yaml::Value,
|
||||
ctx: &ExecutionContext,
|
||||
_argument: serde_yaml::Value,
|
||||
_ctx: &ExecutionContext,
|
||||
_event_tx: &EventSender,
|
||||
) -> Result<(), PluginError> {
|
||||
let config: MailConfig = serde_yaml::from_value(argument)
|
||||
.map_err(|e| PluginError::GeneralError(format!("Invalid mail config: {}", e)))?;
|
||||
|
||||
if let Err(errors) = config.validate() {
|
||||
return Err(PluginError::GeneralError(format!(
|
||||
"Mail config validation failed: {:?}",
|
||||
errors
|
||||
)));
|
||||
}
|
||||
|
||||
let template_data = prepare_template_data(ctx);
|
||||
|
||||
let (subject_template, body_template_html, body_template_text) = match config.schema {
|
||||
Schema::Default => {
|
||||
let subject = config
|
||||
.subject
|
||||
.as_deref()
|
||||
.unwrap_or(DefaultTemplates::SUBJECT);
|
||||
let body_html = config
|
||||
.body
|
||||
.as_deref()
|
||||
.unwrap_or(DefaultTemplates::BODY_HTML);
|
||||
(
|
||||
subject.to_string(),
|
||||
body_html.to_string(),
|
||||
DefaultTemplates::BODY_TEXT.to_string(),
|
||||
)
|
||||
}
|
||||
Schema::Custom => {
|
||||
let subject = config.subject.clone().unwrap();
|
||||
let body_html = config.body.clone().unwrap();
|
||||
(subject, body_html.clone(), body_html)
|
||||
}
|
||||
};
|
||||
|
||||
let subject = render_template(&subject_template, &template_data)
|
||||
.map_err(|e| PluginError::GeneralError(format!("Failed to render subject: {}", e)))?;
|
||||
let body_html = render_template(&body_template_html, &template_data)
|
||||
.map_err(|e| PluginError::GeneralError(format!("Failed to render body_html: {}", e)))?;
|
||||
let body_text = render_template(&body_template_text, &template_data).map_err(|e| {
|
||||
PluginError::GeneralError(format!("Failed to render body_text(?): {}", e))
|
||||
})?;
|
||||
|
||||
let email = build_email(&config, subject, body_html, body_text)?;
|
||||
|
||||
send_email(&config, email).await
|
||||
todo!();
|
||||
}
|
||||
}
|
||||
|
||||
fn build_email(
|
||||
config: &MailConfig,
|
||||
subject: String,
|
||||
body_html: String,
|
||||
body_text: String,
|
||||
) -> Result<Message, PluginError> {
|
||||
let from_str = config.from.to_string();
|
||||
let from: lettre::message::Mailbox = from_str.parse().map_err(|e| {
|
||||
PluginError::GeneralError(format!("Invalid from address '{}': {}", from_str, e))
|
||||
})?;
|
||||
|
||||
let to_addrs: Vec<String> = config.to.to_vec();
|
||||
if to_addrs.is_empty() {
|
||||
return Err(PluginError::GeneralError(
|
||||
"No recipients specified".to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
let mut message_builder = Message::builder().from(from).subject(subject);
|
||||
|
||||
for addr in &to_addrs {
|
||||
let mailbox: lettre::message::Mailbox = addr.parse().map_err(|e| {
|
||||
PluginError::GeneralError(format!("Invalid to address '{}': {}", addr, e))
|
||||
})?;
|
||||
message_builder = message_builder.to(mailbox);
|
||||
}
|
||||
|
||||
if let Some(cc) = &config.cc {
|
||||
for addr in cc.to_vec() {
|
||||
let mailbox: lettre::message::Mailbox = addr.parse().map_err(|e| {
|
||||
PluginError::GeneralError(format!("Invalid cc address '{}': {}", addr, e))
|
||||
})?;
|
||||
message_builder = message_builder.cc(mailbox);
|
||||
}
|
||||
}
|
||||
|
||||
if let Some(bcc) = &config.bcc {
|
||||
for addr in bcc.to_vec() {
|
||||
let mailbox: lettre::message::Mailbox = addr.parse().map_err(|e| {
|
||||
PluginError::GeneralError(format!("Invalid bcc address '{}': {}", addr, e))
|
||||
})?;
|
||||
message_builder = message_builder.bcc(mailbox);
|
||||
}
|
||||
}
|
||||
|
||||
if let Some(reply_to) = &config.reply_to {
|
||||
let mailbox: lettre::message::Mailbox = reply_to.parse().map_err(|e| {
|
||||
PluginError::GeneralError(format!("Invalid reply-to address '{}': {}", reply_to, e))
|
||||
})?;
|
||||
message_builder = message_builder.reply_to(mailbox);
|
||||
}
|
||||
|
||||
let message = message_builder
|
||||
.multipart(
|
||||
MultiPart::alternative()
|
||||
.singlepart(
|
||||
SinglePart::builder()
|
||||
.header(header::ContentType::TEXT_PLAIN)
|
||||
.body(body_text),
|
||||
)
|
||||
.singlepart(
|
||||
SinglePart::builder()
|
||||
.header(header::ContentType::TEXT_HTML)
|
||||
.body(body_html),
|
||||
),
|
||||
)
|
||||
.map_err(|e| PluginError::GeneralError(format!("Failed to build email: {}", e)))?;
|
||||
|
||||
Ok(message)
|
||||
}
|
||||
|
||||
async fn send_email(config: &MailConfig, email: Message) -> Result<(), PluginError> {
|
||||
let connect_timeout = if config.smtp.connect_timeout_ms == 0 {
|
||||
None
|
||||
} else {
|
||||
Some(Duration::from_millis(config.smtp.connect_timeout_ms))
|
||||
};
|
||||
|
||||
let mut transport_builder = match config.smtp.encryption {
|
||||
Encryption::Tls => {
|
||||
log::debug!("Using TLS encryption");
|
||||
AsyncSmtpTransport::<Tokio1Executor>::relay(&config.smtp.host)
|
||||
.map_err(|e| PluginError::GeneralError(format!("Invalid SMTP host: {}", e)))?
|
||||
}
|
||||
Encryption::Starttls => {
|
||||
log::debug!("Using StartTLS encryption");
|
||||
AsyncSmtpTransport::<Tokio1Executor>::starttls_relay(&config.smtp.host)
|
||||
.map_err(|e| PluginError::GeneralError(format!("Invalid SMTP host: {}", e)))?
|
||||
}
|
||||
Encryption::None => {
|
||||
log::debug!("Using NO encryption");
|
||||
AsyncSmtpTransport::<Tokio1Executor>::builder_dangerous(&config.smtp.host)
|
||||
}
|
||||
};
|
||||
|
||||
transport_builder = transport_builder.port(config.smtp.port);
|
||||
|
||||
match config.smtp.auth.auth_type {
|
||||
AuthType::Password => {
|
||||
log::debug!("Using password auth");
|
||||
transport_builder = transport_builder.credentials(
|
||||
(
|
||||
config.smtp.auth.username.clone(),
|
||||
config.smtp.auth.password.clone(),
|
||||
)
|
||||
.into(),
|
||||
);
|
||||
}
|
||||
AuthType::None => {
|
||||
log::debug!("Using no auth");
|
||||
}
|
||||
}
|
||||
|
||||
let transport = transport_builder.build();
|
||||
|
||||
let result = match connect_timeout {
|
||||
Some(t) => timeout(t, transport.send(email)).await,
|
||||
None => Ok(transport.send(email).await),
|
||||
};
|
||||
|
||||
result
|
||||
.map_err(|_| PluginError::GeneralError("SMTP connection timed out".to_string()))?
|
||||
.map_err(|e| PluginError::GeneralError(format!("SMTP error: {}", e)))?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -1,490 +0,0 @@
|
||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
||||
use serde::Deserialize;
|
||||
use workshop_engine::deserialize_duration_ms;
|
||||
|
||||
/// Email address representation supporting plain string or structured object.
|
||||
#[derive(Debug, Deserialize, Clone)]
|
||||
#[serde(untagged)]
|
||||
pub enum EmailAddress {
|
||||
/// Plain email address string.
|
||||
String(String),
|
||||
/// Structured email with display name.
|
||||
Object { name: String, email: String },
|
||||
}
|
||||
|
||||
impl EmailAddress {
|
||||
/// Returns the formatted address string (e.g., "user@example.com" or "Name <user@example.com>").
|
||||
pub fn to_string(&self) -> String {
|
||||
match self {
|
||||
EmailAddress::String(email) => email.clone(),
|
||||
EmailAddress::Object { name, email } => {
|
||||
if name.is_empty() {
|
||||
email.clone()
|
||||
} else {
|
||||
format!("{} <{}>", name, email)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Returns the raw email address without formatting.
|
||||
pub fn email(&self) -> &str {
|
||||
match self {
|
||||
EmailAddress::String(email) => email.as_str(),
|
||||
EmailAddress::Object { email, .. } => email.as_str(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Recipients container supporting single or multiple addresses.
|
||||
#[derive(Debug, Deserialize, Clone)]
|
||||
#[serde(untagged)]
|
||||
pub enum Recipients {
|
||||
/// Single recipient.
|
||||
Single(String),
|
||||
/// Multiple recipients.
|
||||
Multiple(Vec<String>),
|
||||
}
|
||||
|
||||
impl Recipients {
|
||||
/// Converts recipients to a flat vector of email strings.
|
||||
pub fn to_vec(&self) -> Vec<String> {
|
||||
match self {
|
||||
Recipients::Single(s) => vec![s.clone()],
|
||||
Recipients::Multiple(v) => v.clone(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// SMTP authentication mechanism.
|
||||
#[derive(Debug, Deserialize, Clone)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub enum AuthType {
|
||||
/// No authentication.
|
||||
None,
|
||||
/// Password-based authentication.
|
||||
Password,
|
||||
}
|
||||
|
||||
/// SMTP authentication configuration.
|
||||
#[derive(Debug, Deserialize, Clone)]
|
||||
pub struct AuthConfig {
|
||||
/// Authentication mechanism type.
|
||||
#[serde(rename = "type")]
|
||||
pub auth_type: AuthType,
|
||||
/// Username for authentication.
|
||||
pub username: String,
|
||||
/// Password for authentication.
|
||||
pub password: String,
|
||||
}
|
||||
|
||||
/// SMTP connection encryption mode.
|
||||
#[derive(Debug, Deserialize, Clone)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub enum Encryption {
|
||||
/// Implicit TLS/SSL on connection.
|
||||
Tls,
|
||||
/// STARTTLS upgrade after initial plaintext connection.
|
||||
Starttls,
|
||||
/// No encryption.
|
||||
None,
|
||||
}
|
||||
|
||||
/// SMTP server connection configuration.
|
||||
#[derive(Debug, Deserialize, Clone)]
|
||||
pub struct SmtpConfig {
|
||||
/// SMTP server hostname.
|
||||
pub host: String,
|
||||
/// SMTP server port.
|
||||
pub port: u16,
|
||||
/// Authentication configuration.
|
||||
pub auth: AuthConfig,
|
||||
/// Encryption mode.
|
||||
pub encryption: Encryption,
|
||||
/// Connection timeout in milliseconds.
|
||||
#[serde(
|
||||
default = "default_connect_timeout_ms",
|
||||
deserialize_with = "deserialize_duration_ms"
|
||||
)]
|
||||
pub connect_timeout_ms: u64,
|
||||
}
|
||||
|
||||
fn default_connect_timeout_ms() -> u64 {
|
||||
30_000
|
||||
}
|
||||
|
||||
/// Mail schema type determining required fields.
|
||||
#[derive(Debug, Deserialize, Clone)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub enum Schema {
|
||||
/// Default schema with minimal required fields.
|
||||
Default,
|
||||
/// Custom schema requiring explicit subject and body.
|
||||
Custom,
|
||||
}
|
||||
|
||||
fn default_schema() -> Schema {
|
||||
Schema::Default
|
||||
}
|
||||
|
||||
/// Complete mail notification configuration.
|
||||
#[derive(Debug, Deserialize, Clone)]
|
||||
pub struct MailConfig {
|
||||
/// Primary recipients.
|
||||
pub to: Recipients,
|
||||
/// Carbon copy recipients.
|
||||
#[serde(default)]
|
||||
pub cc: Option<Recipients>,
|
||||
/// Blind carbon copy recipients.
|
||||
#[serde(default)]
|
||||
pub bcc: Option<Recipients>,
|
||||
/// Sender address.
|
||||
pub from: EmailAddress,
|
||||
/// Reply-to address.
|
||||
#[serde(default)]
|
||||
pub reply_to: Option<String>,
|
||||
/// Schema type determining validation rules.
|
||||
#[serde(default = "default_schema")]
|
||||
pub schema: Schema,
|
||||
/// Email subject (required for custom schema).
|
||||
pub subject: Option<String>,
|
||||
/// Email body content (required for custom schema).
|
||||
pub body: Option<String>,
|
||||
/// SMTP server configuration.
|
||||
pub smtp: SmtpConfig,
|
||||
}
|
||||
|
||||
impl MailConfig {
|
||||
/// Validates the mail configuration according to schema rules.
|
||||
/// Returns Ok(()) if valid, or Err(vec) with error messages if invalid.
|
||||
pub fn validate(&self) -> Result<(), Vec<String>> {
|
||||
let mut errors = Vec::new();
|
||||
|
||||
if matches!(self.schema, Schema::Custom) {
|
||||
if self.subject.is_none() {
|
||||
errors.push("subject: required when schema=custom".to_string());
|
||||
}
|
||||
if self.body.is_none() {
|
||||
errors.push("body: required when schema=custom".to_string());
|
||||
}
|
||||
}
|
||||
|
||||
if self.to.to_vec().is_empty() {
|
||||
errors.push("to: must have at least one recipient".to_string());
|
||||
}
|
||||
|
||||
if errors.is_empty() {
|
||||
Ok(())
|
||||
} else {
|
||||
Err(errors)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Built-in email notification templates.
|
||||
pub struct DefaultTemplates;
|
||||
|
||||
impl DefaultTemplates {
|
||||
/// Default subject template with build status.
|
||||
pub const SUBJECT: &'static str =
|
||||
"[{{ build.status | upper }}] {{ pipeline.name }} #{{ build.id }}";
|
||||
|
||||
/// Default HTML email body template.
|
||||
pub const BODY_HTML: &'static str = r#"<!DOCTYPE html>
|
||||
<html>
|
||||
<body style="font-family: Arial, sans-serif;">
|
||||
<h2 style="color: {{ build.status_color }};">
|
||||
Build {{ build.status | title }}
|
||||
</h2>
|
||||
<table>
|
||||
<tr><td><strong>Pipeline:</strong></td><td>{{ pipeline.name }}</td></tr>
|
||||
<tr><td><strong>Build ID:</strong></td><td>{{ build.id }}</td></tr>
|
||||
<tr><td><strong>Status:</strong></td><td>{{ build.status }}</td></tr>
|
||||
<tr><td><strong>Duration:</strong></td><td>{{ build.duration }}</td></tr>
|
||||
<tr><td><strong>Author:</strong></td><td>{{ commit.author }}</td></tr>
|
||||
<tr><td><strong>Message:</strong></td><td>{{ commit.message }}</td></tr>
|
||||
</table>
|
||||
<p>
|
||||
<a href="{{ build.url }}" style="background: #007bff; color: white; padding: 10px 20px; text-decoration: none; border-radius: 5px;">
|
||||
View Build Details
|
||||
</a>
|
||||
</p>
|
||||
</body>
|
||||
</html>"#;
|
||||
|
||||
/// Default plain text email body template.
|
||||
pub const BODY_TEXT: &'static str = r#"Build {{ build.status | title }}
|
||||
==============================
|
||||
Pipeline: {{ pipeline.name }}
|
||||
Build ID: {{ build.id }}
|
||||
Status: {{ build.status }}
|
||||
Duration: {{ build.duration }}
|
||||
Commit: {{ commit.hash }}
|
||||
Author: {{ commit.author }}
|
||||
Message: {{ commit.message }}
|
||||
|
||||
View details: {{ build.url }}"#;
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_email_address_string_to_string() {
|
||||
let addr = EmailAddress::String("test@example.com".to_string());
|
||||
assert_eq!(addr.to_string(), "test@example.com");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_email_address_object_with_name_to_string() {
|
||||
let addr = EmailAddress::Object {
|
||||
name: "Test User".to_string(),
|
||||
email: "test@example.com".to_string(),
|
||||
};
|
||||
assert_eq!(addr.to_string(), "Test User <test@example.com>");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_email_address_object_without_name_to_string() {
|
||||
let addr = EmailAddress::Object {
|
||||
name: "".to_string(),
|
||||
email: "test@example.com".to_string(),
|
||||
};
|
||||
assert_eq!(addr.to_string(), "test@example.com");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_email_address_string_email() {
|
||||
let addr = EmailAddress::String("test@example.com".to_string());
|
||||
assert_eq!(addr.email(), "test@example.com");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_email_address_object_email() {
|
||||
let addr = EmailAddress::Object {
|
||||
name: "Test User".to_string(),
|
||||
email: "test@example.com".to_string(),
|
||||
};
|
||||
assert_eq!(addr.email(), "test@example.com");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_recipients_single_to_vec() {
|
||||
let recipients = Recipients::Single("test@example.com".to_string());
|
||||
let vec = recipients.to_vec();
|
||||
assert_eq!(vec, vec!["test@example.com"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_recipients_multiple_to_vec() {
|
||||
let recipients = Recipients::Multiple(vec![
|
||||
"a@example.com".to_string(),
|
||||
"b@example.com".to_string(),
|
||||
]);
|
||||
let vec = recipients.to_vec();
|
||||
assert_eq!(vec, vec!["a@example.com", "b@example.com"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_mail_config_validate_valid_default_schema() {
|
||||
let config = MailConfig {
|
||||
to: Recipients::Single("to@example.com".to_string()),
|
||||
cc: None,
|
||||
bcc: None,
|
||||
from: EmailAddress::String("from@example.com".to_string()),
|
||||
reply_to: None,
|
||||
schema: Schema::Default,
|
||||
subject: None,
|
||||
body: None,
|
||||
smtp: SmtpConfig {
|
||||
host: "smtp.example.com".to_string(),
|
||||
port: 587,
|
||||
auth: AuthConfig {
|
||||
auth_type: AuthType::None,
|
||||
username: "".to_string(),
|
||||
password: "".to_string(),
|
||||
},
|
||||
encryption: Encryption::Tls,
|
||||
connect_timeout_ms: 30_000,
|
||||
},
|
||||
};
|
||||
assert!(config.validate().is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_mail_config_validate_valid_custom_schema() {
|
||||
let config = MailConfig {
|
||||
to: Recipients::Single("to@example.com".to_string()),
|
||||
cc: None,
|
||||
bcc: None,
|
||||
from: EmailAddress::String("from@example.com".to_string()),
|
||||
reply_to: None,
|
||||
schema: Schema::Custom,
|
||||
subject: Some("Test Subject".to_string()),
|
||||
body: Some("Test Body".to_string()),
|
||||
smtp: SmtpConfig {
|
||||
host: "smtp.example.com".to_string(),
|
||||
port: 587,
|
||||
auth: AuthConfig {
|
||||
auth_type: AuthType::None,
|
||||
username: "".to_string(),
|
||||
password: "".to_string(),
|
||||
},
|
||||
encryption: Encryption::Tls,
|
||||
connect_timeout_ms: 30_000,
|
||||
},
|
||||
};
|
||||
assert!(config.validate().is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_mail_config_validate_custom_schema_missing_subject() {
|
||||
let config = MailConfig {
|
||||
to: Recipients::Single("to@example.com".to_string()),
|
||||
cc: None,
|
||||
bcc: None,
|
||||
from: EmailAddress::String("from@example.com".to_string()),
|
||||
reply_to: None,
|
||||
schema: Schema::Custom,
|
||||
subject: None,
|
||||
body: Some("Test Body".to_string()),
|
||||
smtp: SmtpConfig {
|
||||
host: "smtp.example.com".to_string(),
|
||||
port: 587,
|
||||
auth: AuthConfig {
|
||||
auth_type: AuthType::None,
|
||||
username: "".to_string(),
|
||||
password: "".to_string(),
|
||||
},
|
||||
encryption: Encryption::Tls,
|
||||
connect_timeout_ms: 30_000,
|
||||
},
|
||||
};
|
||||
let result = config.validate();
|
||||
assert!(result.is_err());
|
||||
assert!(result.unwrap_err().iter().any(|e| e.contains("subject")));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_mail_config_validate_custom_schema_missing_body() {
|
||||
let config = MailConfig {
|
||||
to: Recipients::Single("to@example.com".to_string()),
|
||||
cc: None,
|
||||
bcc: None,
|
||||
from: EmailAddress::String("from@example.com".to_string()),
|
||||
reply_to: None,
|
||||
schema: Schema::Custom,
|
||||
subject: Some("Test Subject".to_string()),
|
||||
body: None,
|
||||
smtp: SmtpConfig {
|
||||
host: "smtp.example.com".to_string(),
|
||||
port: 587,
|
||||
auth: AuthConfig {
|
||||
auth_type: AuthType::None,
|
||||
username: "".to_string(),
|
||||
password: "".to_string(),
|
||||
},
|
||||
encryption: Encryption::Tls,
|
||||
connect_timeout_ms: 30_000,
|
||||
},
|
||||
};
|
||||
let result = config.validate();
|
||||
assert!(result.is_err());
|
||||
assert!(result.unwrap_err().iter().any(|e| e.contains("body")));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_mail_config_validate_empty_recipients() {
|
||||
let config = MailConfig {
|
||||
to: Recipients::Multiple(vec![]),
|
||||
cc: None,
|
||||
bcc: None,
|
||||
from: EmailAddress::String("from@example.com".to_string()),
|
||||
reply_to: None,
|
||||
schema: Schema::Default,
|
||||
subject: None,
|
||||
body: None,
|
||||
smtp: SmtpConfig {
|
||||
host: "smtp.example.com".to_string(),
|
||||
port: 587,
|
||||
auth: AuthConfig {
|
||||
auth_type: AuthType::None,
|
||||
username: "".to_string(),
|
||||
password: "".to_string(),
|
||||
},
|
||||
encryption: Encryption::Tls,
|
||||
connect_timeout_ms: 30_000,
|
||||
},
|
||||
};
|
||||
let result = config.validate();
|
||||
assert!(result.is_err());
|
||||
assert!(result.unwrap_err().iter().any(|e| e.contains("recipient")));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_email_address_serde_string() {
|
||||
let json = r#""test@example.com""#;
|
||||
let addr: EmailAddress = serde_json::from_str(json).unwrap();
|
||||
assert!(matches!(addr, EmailAddress::String(s) if s == "test@example.com"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_email_address_serde_object() {
|
||||
let json = r#"{"name":"Test User","email":"test@example.com"}"#;
|
||||
let addr: EmailAddress = serde_json::from_str(json).unwrap();
|
||||
assert!(
|
||||
matches!(addr, EmailAddress::Object { name, email } if name == "Test User" && email == "test@example.com")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_recipients_serde_single() {
|
||||
let json = r#""single@example.com""#;
|
||||
let recipients: Recipients = serde_json::from_str(json).unwrap();
|
||||
assert!(matches!(recipients, Recipients::Single(s) if s == "single@example.com"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_recipients_serde_multiple() {
|
||||
let json = r#"["a@example.com","b@example.com"]"#;
|
||||
let recipients: Recipients = serde_json::from_str(json).unwrap();
|
||||
assert!(
|
||||
matches!(recipients, Recipients::Multiple(v) if v == vec!["a@example.com".to_string(), "b@example.com".to_string()])
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_auth_type_serde() {
|
||||
#[derive(serde::Deserialize, Debug)]
|
||||
struct Test {
|
||||
#[serde(rename = "type")]
|
||||
auth_type: AuthType,
|
||||
}
|
||||
let json = r#"{"type":"password"}"#;
|
||||
let t: Test = serde_json::from_str(json).unwrap();
|
||||
assert!(matches!(t.auth_type, AuthType::Password));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_encryption_serde() {
|
||||
#[derive(serde::Deserialize, Debug)]
|
||||
struct Test {
|
||||
encryption: Encryption,
|
||||
}
|
||||
let json = r#"{"encryption":"tls"}"#;
|
||||
let t: Test = serde_json::from_str(json).unwrap();
|
||||
assert!(matches!(t.encryption, Encryption::Tls));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_schema_serde() {
|
||||
#[derive(serde::Deserialize, Debug)]
|
||||
struct Test {
|
||||
schema: Schema,
|
||||
}
|
||||
let json = r#"{"schema":"custom"}"#;
|
||||
let t: Test = serde_json::from_str(json).unwrap();
|
||||
assert!(matches!(t.schema, Schema::Custom));
|
||||
}
|
||||
}
|
||||
@@ -1,101 +0,0 @@
|
||||
// Code in this module PARTIALLY OR FULLY utilized AI Coding Agent.
|
||||
use crate::ExecutionContext;
|
||||
use minijinja::Environment;
|
||||
use std::collections::HashMap;
|
||||
|
||||
/// Builds template context data from execution context.
|
||||
/// Populates pipeline, build, and commit information for template rendering.
|
||||
pub fn prepare_template_data(ctx: &ExecutionContext) -> HashMap<String, serde_json::Value> {
|
||||
let mut data = HashMap::new();
|
||||
|
||||
data.insert(
|
||||
"pipeline".to_string(),
|
||||
serde_json::json!({
|
||||
"name": ctx.pipeline_name,
|
||||
}),
|
||||
);
|
||||
|
||||
data.insert(
|
||||
"build".to_string(),
|
||||
serde_json::json!({
|
||||
"id": ctx.task_id,
|
||||
"status": "success",
|
||||
"status_color": "green",
|
||||
"duration": "0s",
|
||||
"url": "",
|
||||
}),
|
||||
);
|
||||
|
||||
data.insert(
|
||||
"commit".to_string(),
|
||||
serde_json::json!({
|
||||
"hash": "",
|
||||
"short_hash": "",
|
||||
"message": "",
|
||||
"author": "",
|
||||
"author_email": "",
|
||||
}),
|
||||
);
|
||||
|
||||
data
|
||||
}
|
||||
|
||||
/// Renders a minijinja template string with the provided data map.
|
||||
/// Returns the rendered string on success, or a String error message on failure.
|
||||
pub fn render_template(
|
||||
template: &str,
|
||||
data: &HashMap<String, serde_json::Value>,
|
||||
) -> Result<String, String> {
|
||||
let env = Environment::new();
|
||||
|
||||
let tmpl = env
|
||||
.template_from_str(template)
|
||||
.map_err(|e| format!("Template parse error: {}", e))?;
|
||||
|
||||
let result = tmpl
|
||||
.render(data)
|
||||
.map_err(|e| format!("Template render error: {}", e))?;
|
||||
|
||||
Ok(result)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_render_template_simple() {
|
||||
let template = "Hello {{ name }}!";
|
||||
let mut data = HashMap::new();
|
||||
data.insert(
|
||||
"name".to_string(),
|
||||
serde_json::Value::String("World".to_string()),
|
||||
);
|
||||
let result = render_template(template, &data).unwrap();
|
||||
assert_eq!(result, "Hello World!");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_render_template_with_build_status() {
|
||||
let template = "Build {{ build.status }} for pipeline {{ pipeline.name }}";
|
||||
let mut data = HashMap::new();
|
||||
data.insert(
|
||||
"build".to_string(),
|
||||
serde_json::json!({ "status": "success" }),
|
||||
);
|
||||
data.insert(
|
||||
"pipeline".to_string(),
|
||||
serde_json::json!({ "name": "test-pipeline" }),
|
||||
);
|
||||
let result = render_template(template, &data).unwrap();
|
||||
assert_eq!(result, "Build success for pipeline test-pipeline");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_render_template_invalid_syntax() {
|
||||
let template = "{{ Unterminated";
|
||||
let data = HashMap::new();
|
||||
let result = render_template(template, &data);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,7 @@
|
||||
use crate::finalize::plugin::PluginError;
|
||||
use crate::finalize::plugin::PluginMetadata;
|
||||
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn};
|
||||
use workshop_engine::{EventSender, ExecutionContext};
|
||||
use crate::{ finalize::plugin::AsyncPluginFn};
|
||||
use async_trait::async_trait;
|
||||
|
||||
pub struct Satori;
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
use crate::finalize::plugin::PluginError;
|
||||
use crate::finalize::plugin::PluginMetadata;
|
||||
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn};
|
||||
use workshop_engine::{EventSender, ExecutionContext};
|
||||
use crate::{ finalize::plugin::AsyncPluginFn};
|
||||
use async_trait::async_trait;
|
||||
|
||||
pub struct Webhook;
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
use crate::finalize::NotificationTemplateDef;
|
||||
use crate::notify::types::NotificationRenderer;
|
||||
use serde::Deserialize;
|
||||
use std::path::PathBuf;
|
||||
@@ -18,10 +19,14 @@ pub struct PluginMetadata {
|
||||
#[serde(default)]
|
||||
pub renderer: Option<NotificationRenderer>,
|
||||
|
||||
/// Additional YAML fields not captured by struct.
|
||||
/// Additional YAML fields.
|
||||
#[serde(flatten)]
|
||||
pub value: serde_yaml::Value,
|
||||
|
||||
/// Plugin-defined default templates, auto-registered as <name>.fallback
|
||||
#[serde(default)]
|
||||
pub templates: Option<Vec<NotificationTemplateDef>>,
|
||||
|
||||
/// Filesystem path to plugin definition.
|
||||
#[serde(skip)]
|
||||
pub fspath: PathBuf,
|
||||
@@ -36,6 +41,7 @@ impl Default for PluginMetadata {
|
||||
plugin_type: PluginType::Unknown,
|
||||
value: serde_yaml::Value::Null,
|
||||
renderer: None,
|
||||
templates: None,
|
||||
fspath: PathBuf::new(),
|
||||
}
|
||||
}
|
||||
@@ -50,76 +56,3 @@ pub enum PluginType {
|
||||
Rhai,
|
||||
Dylib,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_plugin_metadata_default() {
|
||||
let meta = PluginMetadata::default();
|
||||
assert_eq!(meta.name, "");
|
||||
assert_eq!(meta.version, "");
|
||||
assert_eq!(meta.entrypoint, "");
|
||||
assert_eq!(meta.plugin_type, PluginType::Unknown);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_plugin_type_default() {
|
||||
let pt = PluginType::default();
|
||||
assert_eq!(pt, PluginType::Unknown);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_plugin_type_equality() {
|
||||
assert_eq!(PluginType::Shell, PluginType::Shell);
|
||||
assert_eq!(PluginType::Dylib, PluginType::Dylib);
|
||||
assert_ne!(PluginType::Shell, PluginType::Dylib);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_plugin_metadata_clone() {
|
||||
let meta = PluginMetadata {
|
||||
name: "test".to_string(),
|
||||
version: "1.0".to_string(),
|
||||
entrypoint: "/bin/test".to_string(),
|
||||
plugin_type: PluginType::Shell,
|
||||
value: serde_yaml::Value::Null,
|
||||
fspath: PathBuf::from("/path/to/plugin"),
|
||||
renderer: None,
|
||||
};
|
||||
let cloned = meta.clone();
|
||||
assert_eq!(cloned.name, meta.name);
|
||||
assert_eq!(cloned.version, meta.version);
|
||||
assert_eq!(cloned.plugin_type, meta.plugin_type);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_plugin_type_serde_unknown() {
|
||||
let yaml = "Unknown";
|
||||
let pt: PluginType = serde_yaml::from_str(yaml).unwrap();
|
||||
assert_eq!(pt, PluginType::Unknown);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_plugin_type_serde_shell() {
|
||||
let yaml = "Shell";
|
||||
let pt: PluginType = serde_yaml::from_str(yaml).unwrap();
|
||||
assert_eq!(pt, PluginType::Shell);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_plugin_metadata_serde() {
|
||||
let yaml = r#"
|
||||
name: test-plugin
|
||||
version: "2.0"
|
||||
entrypoint: ./bin/test
|
||||
plugin_type: Shell
|
||||
"#;
|
||||
let meta: PluginMetadata = serde_yaml::from_str(yaml).unwrap();
|
||||
assert_eq!(meta.name, "test-plugin");
|
||||
assert_eq!(meta.version, "2.0");
|
||||
assert_eq!(meta.entrypoint, "./bin/test");
|
||||
assert_eq!(meta.plugin_type, PluginType::Shell);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
use crate::finalize::plugin::PluginError;
|
||||
use crate::finalize::plugin::PluginMetadata;
|
||||
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn};
|
||||
use workshop_engine::{EventSender, ExecutionContext};
|
||||
use crate::{ finalize::plugin::AsyncPluginFn};
|
||||
use async_trait::async_trait;
|
||||
|
||||
pub struct Rhai;
|
||||
|
||||
@@ -1,8 +1,9 @@
|
||||
use crate::Engine;
|
||||
use workshop_engine::Engine;
|
||||
use crate::finalize::constant::FINALIZE_SHELL_ENVPREFIX;
|
||||
use crate::finalize::plugin::PluginError;
|
||||
use crate::finalize::plugin::PluginMetadata;
|
||||
use crate::{EventSender, ExecutionContext, finalize::plugin::AsyncPluginFn};
|
||||
use workshop_engine::{EventSender, ExecutionContext};
|
||||
use crate::{ finalize::plugin::AsyncPluginFn};
|
||||
use async_trait::async_trait;
|
||||
use std::collections::HashMap;
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
use crate::ExecutionContext;
|
||||
use workshop_engine::ExecutionContext;
|
||||
use crate::finalize::plugin::{PluginMap, call_named_plugin};
|
||||
use std::path::PathBuf;
|
||||
use workshop_engine::{CustomCommand, DeltaExecutionContext, Engine, EventSender, ExecutionError};
|
||||
|
||||
+25
-57
@@ -1,27 +1,16 @@
|
||||
// Library crate for workshop-executor
|
||||
// Selective exports: only modules used by external callers or tests
|
||||
|
||||
pub mod bare;
|
||||
pub mod bake;
|
||||
pub mod daemon;
|
||||
pub mod error;
|
||||
pub mod finalize;
|
||||
pub mod monitor;
|
||||
pub mod notify;
|
||||
pub mod prebake;
|
||||
pub mod socket;
|
||||
pub mod types;
|
||||
pub mod utils;
|
||||
|
||||
// External crates re-exports
|
||||
pub use config;
|
||||
|
||||
// Re-export commonly used types from workshop-engine
|
||||
pub use workshop_engine::{Engine, EventSender, ExecutionContext, ExecutionError, ExecutionResult};
|
||||
|
||||
// CLI types used by main (keep here since they're used in daemon)
|
||||
pub use clap::{Parser, Subcommand};
|
||||
pub use serde::Deserialize;
|
||||
pub use std::collections::HashMap;
|
||||
pub use std::path::PathBuf;
|
||||
|
||||
pub mod cli {
|
||||
pub use super::{Deserialize, HashMap, Parser, PathBuf, Subcommand};
|
||||
pub use clap::Command;
|
||||
@@ -46,64 +35,43 @@ pub mod cli {
|
||||
#[arg(long, default_value = "false", env = "HBW_DRY_RUN")]
|
||||
pub dry_run: bool,
|
||||
|
||||
/// Override prebake config path
|
||||
#[arg(long)]
|
||||
pub prebake: Option<PathBuf>,
|
||||
/// Override bake script path
|
||||
#[arg(long)]
|
||||
pub bake: Option<PathBuf>,
|
||||
/// Override bake_base.sh path
|
||||
#[arg(long)]
|
||||
pub bake_base: Option<PathBuf>,
|
||||
/// Override finalize config path
|
||||
#[arg(long)]
|
||||
pub finalize: Option<PathBuf>,
|
||||
|
||||
/// Debug features (comma-separated). "notify" enables notification debug logging.
|
||||
#[arg(long, env = "HBW_DEBUG")]
|
||||
pub debug: Option<String>,
|
||||
|
||||
#[command(subcommand)]
|
||||
pub command: Option<Commands>,
|
||||
}
|
||||
|
||||
#[derive(Subcommand, Debug)]
|
||||
pub enum Commands {
|
||||
/// Run pipeline stages in standalone (bare) mode
|
||||
/// Run individual pipeline stages
|
||||
Bare {
|
||||
#[command(subcommand)]
|
||||
command: BareCommands,
|
||||
},
|
||||
|
||||
/// Run in client mode
|
||||
Client { config: String },
|
||||
Daemon {},
|
||||
}
|
||||
|
||||
#[derive(Subcommand, Debug)]
|
||||
pub enum BareCommands {
|
||||
/// Prepare a task
|
||||
Prebake { config: PathBuf },
|
||||
|
||||
/// Run a task
|
||||
Bake {
|
||||
/// Path to the script to execute
|
||||
script: PathBuf,
|
||||
/// Path to bake_base.sh
|
||||
#[arg(short, long, default_value = "./bake_base.sh")]
|
||||
bake_base: PathBuf,
|
||||
#[arg(short, long, default_value = "./prebake.yml")]
|
||||
prebake: PathBuf,
|
||||
},
|
||||
|
||||
/// Finish a task
|
||||
Finalize { config: String },
|
||||
Prebake {},
|
||||
Bake {},
|
||||
Finalize {},
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
pub struct TaskSpec {
|
||||
pub subcommand: String,
|
||||
pub arguments: Vec<String>,
|
||||
pub options: HashMap<String, String>,
|
||||
}
|
||||
|
||||
impl TaskSpec {
|
||||
pub fn command(&self, executable: &PathBuf) -> tokio::process::Command {
|
||||
let mut command = tokio::process::Command::new(executable);
|
||||
command.arg(&self.subcommand);
|
||||
for arg in &self.arguments {
|
||||
command.arg(arg);
|
||||
}
|
||||
for (flag, value) in self.options.iter() {
|
||||
if value.is_empty() {
|
||||
command.arg(format!("--{}", flag));
|
||||
} else {
|
||||
command.arg(format!("--{}={}", flag, value));
|
||||
}
|
||||
}
|
||||
command
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+110
-63
@@ -1,13 +1,91 @@
|
||||
use std::path::Path;
|
||||
use workshop_baker::types::resource::ResourceRegistry;
|
||||
use std::path::PathBuf;
|
||||
|
||||
use workshop_baker::{
|
||||
bake::bake,
|
||||
cli::{BareCommands, Cli, Commands, Parser},
|
||||
daemon,
|
||||
finalize::finalize,
|
||||
prebake::prebake,
|
||||
bake::bake, cli::{Cli, Commands, Parser},
|
||||
error::CliError,
|
||||
finalize::finalize, notify::notify, prebake::prebake,
|
||||
};
|
||||
use workshop_engine::{EventSender, ExecutionContext, error::HasExitCode};
|
||||
use workshop_baker::bare::run_bare;
|
||||
use workshop_baker::notify::queue::NotificationQueue;
|
||||
use workshop_engine::{EventSender, ExecutionContext};
|
||||
|
||||
fn or_workshop(path: &Option<PathBuf>, name: &str) -> PathBuf {
|
||||
path.clone().unwrap_or_else(|| PathBuf::from(format!(".workshop/{}", name)))
|
||||
}
|
||||
|
||||
async fn worker_main(cli: Cli) -> Result<(), CliError> {
|
||||
let prebake_path = cli.prebake.clone().unwrap_or_else(|| {
|
||||
log::error!("--prebake is required in worker standalone mode");
|
||||
std::process::exit(1);
|
||||
});
|
||||
let bake_path = cli.bake.clone().unwrap_or_else(|| {
|
||||
log::error!("--bake is required in worker standalone mode");
|
||||
std::process::exit(1);
|
||||
});
|
||||
let finalize_path = cli.finalize.clone().unwrap_or_else(|| {
|
||||
log::error!("--finalize is required in worker standalone mode");
|
||||
std::process::exit(1);
|
||||
});
|
||||
|
||||
log::info!("Worker standalone mode");
|
||||
|
||||
let mut ctx = ExecutionContext {
|
||||
standalone: true,
|
||||
task_id: format!("{}-{}-worker", cli.pipeline, cli.build_id),
|
||||
pipeline_name: cli.pipeline.clone(),
|
||||
username: cli.username.clone(),
|
||||
dry_run: cli.dry_run,
|
||||
privileged: false,
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let (tx, event_rx) = tokio::sync::mpsc::channel(256);
|
||||
let event_tx: EventSender = Some(tx);
|
||||
let _event_handle = tokio::spawn(workshop_baker::prebake::event::event_receiver(
|
||||
event_rx, cli.pipeline.clone(), cli.build_id.clone(),
|
||||
));
|
||||
|
||||
let (to_tx, to_rx) = tokio::sync::mpsc::channel(256);
|
||||
let (retry_tx, retry_rx) = tokio::sync::mpsc::channel(256);
|
||||
|
||||
// Resource registry — populated before stages, consumed by stages
|
||||
let registry = ResourceRegistry::new();
|
||||
// TODO: pre-fetch plugins and templates into registry
|
||||
|
||||
let notify_path = finalize_path.clone();
|
||||
let notify_ctx = ctx.clone();
|
||||
let notify_etx = event_tx.clone();
|
||||
let debug = cli.debug.clone();
|
||||
let notify_registry = registry.clone();
|
||||
let notify_handle = tokio::spawn(async move {
|
||||
notify(¬ify_path, &mut notify_ctx.clone(), notify_etx, to_rx, retry_tx, debug.map(|d| d.contains("notify")).unwrap_or(false), ¬ify_registry).await
|
||||
});
|
||||
|
||||
let nevent_tx = to_tx.clone();
|
||||
let mut queue = NotificationQueue::new(to_tx, retry_rx);
|
||||
|
||||
let pipe_result: anyhow::Result<()> = async {
|
||||
prebake(&prebake_path, &cli, None, &mut ctx, event_tx.clone(), Some(nevent_tx)).await?;
|
||||
let bake_base_path = or_workshop(&cli.bake_base, "bake_base.sh");
|
||||
let pb_path = or_workshop(&cli.prebake, "prebake.yml");
|
||||
bake(&bake_path, &bake_base_path, &pb_path, &cli, &mut ctx, event_tx.clone()).await?;
|
||||
finalize(&finalize_path, &cli, &mut ctx, event_tx.clone(), ®istry).await?;
|
||||
Ok(())
|
||||
}.await;
|
||||
|
||||
if let Err(e) = pipe_result {
|
||||
log::error!("Pipeline stage failed: {}", e);
|
||||
return Err(CliError::General(e));
|
||||
}
|
||||
|
||||
log::info!("Pipeline complete. Draining notifications...");
|
||||
if let Err(e) = queue.drain().await {
|
||||
log::error!("Notification queue error: {}", e);
|
||||
}
|
||||
let _ = notify_handle.await;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() {
|
||||
@@ -16,77 +94,46 @@ async fn main() {
|
||||
if cli.dry_run {
|
||||
log::warn!("Dry run enabled, no changes will be made.");
|
||||
}
|
||||
match &cli.command {
|
||||
Some(Commands::Client { config: _ }) => {
|
||||
unimplemented!("Client mode not yet implemented.")
|
||||
}
|
||||
Some(Commands::Bare { command }) => {
|
||||
log::info!("Running in standalone mode.");
|
||||
|
||||
let (stage, privileged) = match command {
|
||||
BareCommands::Prebake { .. } => ("init", true),
|
||||
BareCommands::Bake { .. } => ("bake", false),
|
||||
BareCommands::Finalize { .. } => ("finalize", false),
|
||||
};
|
||||
let pipeline = cli.pipeline.clone();
|
||||
let build_id = cli.build_id.clone();
|
||||
|
||||
match &cli.command {
|
||||
Some(Commands::Bare { command }) => {
|
||||
let mut ctx = ExecutionContext {
|
||||
standalone: true,
|
||||
task_id: format!("{}-{}-{}", cli.pipeline, cli.build_id, stage),
|
||||
pipeline_name: cli.pipeline.clone(),
|
||||
task_id: format!("{}-{}-bare", pipeline, build_id),
|
||||
pipeline_name: pipeline.clone(),
|
||||
username: cli.username.clone(),
|
||||
dry_run: cli.dry_run,
|
||||
privileged,
|
||||
privileged: false,
|
||||
..Default::default()
|
||||
};
|
||||
let (tx, event_rx) = tokio::sync::mpsc::channel(256);
|
||||
let event_tx: EventSender = Some(tx);
|
||||
let handle = tokio::spawn(workshop_baker::prebake::event::event_receiver(
|
||||
event_rx,
|
||||
cli.pipeline.clone(),
|
||||
cli.build_id.clone(),
|
||||
let event_handle = tokio::spawn(workshop_baker::prebake::event::event_receiver(
|
||||
event_rx, pipeline.clone(), build_id.clone(),
|
||||
));
|
||||
|
||||
let result: anyhow::Result<()> = match command {
|
||||
BareCommands::Prebake { config } => {
|
||||
prebake(config, &cli, None, &mut ctx, event_tx).await
|
||||
}
|
||||
BareCommands::Bake {
|
||||
script,
|
||||
bake_base,
|
||||
prebake: prebake_path,
|
||||
} => bake(script, bake_base, prebake_path, &cli, &mut ctx, event_tx)
|
||||
.await
|
||||
.map_err(|e| e.into()),
|
||||
BareCommands::Finalize { config } => {
|
||||
finalize(Path::new(config), &cli, &mut ctx, event_tx)
|
||||
.await
|
||||
.map_err(|e| e.into())
|
||||
}
|
||||
};
|
||||
let registry = ResourceRegistry::new();
|
||||
let result = run_bare(&cli, command, &mut ctx, event_tx, &pipeline, &build_id, ®istry).await;
|
||||
|
||||
drop(ctx);
|
||||
let _ = handle.await;
|
||||
dbg!(&result);
|
||||
if let Err(ref e) = result {
|
||||
let code = match command {
|
||||
BareCommands::Prebake { .. } => e
|
||||
.downcast_ref::<workshop_baker::prebake::error::PrebakeError>()
|
||||
.map(|pe| pe.exit_code())
|
||||
.unwrap_or(1),
|
||||
BareCommands::Bake { .. } => e
|
||||
.downcast_ref::<workshop_baker::bake::error::BakeError>()
|
||||
.map(|be| be.exit_code())
|
||||
.unwrap_or(1),
|
||||
BareCommands::Finalize { .. } => e
|
||||
.downcast_ref::<workshop_baker::finalize::FinalizeError>()
|
||||
.map(|fe| fe.exit_code())
|
||||
.unwrap_or(1),
|
||||
};
|
||||
log::error!("{} stage failed: {}", stage, e);
|
||||
std::process::exit(code);
|
||||
let _ = event_handle.await;
|
||||
|
||||
if let Err(e) = result {
|
||||
log::error!("{}", e);
|
||||
std::process::exit(e.exit_code());
|
||||
}
|
||||
}
|
||||
Some(Commands::Daemon {}) => {
|
||||
unimplemented!("Daemon mode not implemented");
|
||||
}
|
||||
None => {
|
||||
log::info!("Executor will be running in daemon mode.");
|
||||
let _result = daemon::daemon(&cli).await;
|
||||
if let Err(e) = worker_main(cli).await {
|
||||
log::error!("{}", e);
|
||||
std::process::exit(e.exit_code());
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,89 +0,0 @@
|
||||
mod cgroups;
|
||||
mod jobobject;
|
||||
use crate::socket::{establish_connection, get_socket_addr};
|
||||
use config::Config;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::time::Instant;
|
||||
use tokio::io::AsyncWriteExt;
|
||||
|
||||
#[derive(Debug, Serialize, Deserialize)]
|
||||
pub struct ResourceUsageInfo {
|
||||
pub msgtype: String,
|
||||
pub name: String,
|
||||
pub os_type: OsType,
|
||||
pub statistics: UsageStats,
|
||||
pub timestamp: u64,
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize, Deserialize)]
|
||||
pub enum OsType {
|
||||
Linux,
|
||||
Windows,
|
||||
MacOS,
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize, Deserialize, Clone)]
|
||||
pub struct UsageStats {
|
||||
pub cpu_usage_percent: f64,
|
||||
pub memory_usage_bytes: u64,
|
||||
pub memory_limit_bytes: i64,
|
||||
// pub memory_usage_percent: f64,
|
||||
pub process_count: u64,
|
||||
pub cpu_usage_us: u64,
|
||||
/// Internal timestamp, using std::Instant
|
||||
#[serde(skip, default = "default_internal_timestamp")]
|
||||
_timestamp: Instant,
|
||||
}
|
||||
|
||||
fn default_internal_timestamp() -> Instant {
|
||||
Instant::now()
|
||||
}
|
||||
|
||||
pub async fn monitor(
|
||||
name: &String,
|
||||
interval_ms: Option<u64>,
|
||||
settings: Config,
|
||||
) -> anyhow::Result<()> {
|
||||
// let socket_addr = get_wsagent_socket_addr()?;
|
||||
let socket_addr = get_socket_addr(settings.get_string("socket")?)?;
|
||||
let interval = tokio::time::Duration::from_millis(interval_ms.unwrap_or(1000));
|
||||
let mut socket = establish_connection(&socket_addr).await?;
|
||||
let mut prev_usage = UsageStats {
|
||||
cpu_usage_percent: 0.0,
|
||||
memory_usage_bytes: 0,
|
||||
memory_limit_bytes: 0,
|
||||
process_count: 0,
|
||||
cpu_usage_us: 0,
|
||||
_timestamp: default_internal_timestamp(),
|
||||
};
|
||||
loop {
|
||||
let usage: UsageStats = cgroups::get_usage(name, prev_usage)?; // TODO: 多系统支持?
|
||||
prev_usage = usage.clone();
|
||||
// dbg!(&usage);
|
||||
let usage_info = ResourceUsageInfo {
|
||||
msgtype: "ResourceUsage".to_string(),
|
||||
name: name.clone(),
|
||||
os_type: OsType::Linux, // TODO: 多系统支持?
|
||||
statistics: usage,
|
||||
timestamp: std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)?
|
||||
.as_millis() as u64,
|
||||
};
|
||||
|
||||
let usage_json = serde_json::to_string(&usage_info)?;
|
||||
|
||||
send_usage_info(&mut socket, &usage_json).await?;
|
||||
// TODO: 增加消息接收,提供(伪)双向通信功能
|
||||
|
||||
tokio::time::sleep(interval).await;
|
||||
}
|
||||
}
|
||||
|
||||
async fn send_usage_info<S>(stream: &mut S, usage_json: &str) -> anyhow::Result<()>
|
||||
where
|
||||
S: tokio::io::AsyncWrite + Unpin,
|
||||
{
|
||||
stream.write_all(usage_json.as_bytes()).await?;
|
||||
stream.write_all(b"\n").await?;
|
||||
Ok(())
|
||||
}
|
||||
+429
-361
@@ -1,400 +1,468 @@
|
||||
use crate::finalize::FinalizeConfig;
|
||||
use crate::finalize::NotificationConfig;
|
||||
use crate::finalize::NotificationTemplate;
|
||||
use crate::finalize::config::TriggerItemRaw;
|
||||
use crate::finalize::constant::FINALIZE_STANDALONE_PLUGIN_PATH;
|
||||
use crate::finalize::parse;
|
||||
use crate::finalize::plugin;
|
||||
use crate::finalize::plugin::fetch;
|
||||
use crate::notify::types::Matchable;
|
||||
use crate::notify::types::Method;
|
||||
use crate::notify::types::NOTIFICATION_DEFAULT_FALLIBLE;
|
||||
use crate::notify::types::NOTIFICATION_DEFAULT_MAX_LIVES;
|
||||
use crate::notify::types::NOTIFICATION_DEFAULT_RETRY_BACKOFF;
|
||||
use crate::notify::types::NOTIFICATION_DEFAULT_TIMEOUT_MS;
|
||||
use crate::notify::types::NOTIFICATION_NOTREADY_DELAY;
|
||||
use crate::notify::types::NOTIFICATION_TEMPORARY_DELAY_FACTOR;
|
||||
use crate::notify::types::NOTIFICATION_TEMPORARY_DELAY_TIME;
|
||||
use crate::notify::types::NOTIFICATION_TEMPORARY_MAX_TIME;
|
||||
use crate::notify::types::NotificationEvent;
|
||||
use crate::notify::types::DEFAULT_NOTIFY_PARAMS;
|
||||
use crate::notify::types::NotificationEventReceiver;
|
||||
use crate::notify::types::NotificationEventSender;
|
||||
use crate::notify::types::NotificationPluginMap;
|
||||
use crate::notify::types::NotificationRenderer;
|
||||
use crate::notify::types::NotificationRule;
|
||||
use crate::notify::types::NotificationRulesMap;
|
||||
use crate::notify::types::ParsedTrigger;
|
||||
use crate::{EventSender, ExecutionContext};
|
||||
use chrono::Utc;
|
||||
use globset::{Glob, GlobSetBuilder};
|
||||
use std::collections::HashMap;
|
||||
use std::collections::HashSet;
|
||||
use crate::notify::handler::NotificationHandler;
|
||||
use crate::types::resource::ResourceRegistry;
|
||||
use std::path::Path;
|
||||
use std::path::PathBuf;
|
||||
use std::sync::Arc;
|
||||
use workshop_engine::{EventSender, ExecutionContext};
|
||||
|
||||
mod error;
|
||||
pub mod handler;
|
||||
pub mod queue;
|
||||
pub mod rule;
|
||||
pub mod template;
|
||||
pub mod types;
|
||||
use error::NotifyError;
|
||||
|
||||
fn validate(finalize: &FinalizeConfig) -> anyhow::Result<()> {
|
||||
let validate_result = finalize.validate();
|
||||
if let Err(errors) = validate_result {
|
||||
for error in &errors {
|
||||
log::error!("Error: {}", error);
|
||||
}
|
||||
anyhow::bail!(NotifyError::FinalizeValidationError(errors))
|
||||
};
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn extract_plugin(finalize: &FinalizeConfig) -> HashSet<String> {
|
||||
finalize
|
||||
.notification
|
||||
.groups
|
||||
.values()
|
||||
.flat_map(|methods| methods.keys())
|
||||
.cloned()
|
||||
.collect()
|
||||
}
|
||||
pub mod util;
|
||||
use rule::parse_rules;
|
||||
use template::{flatten_template_refs, resolve_external_template_refs};
|
||||
use util::{extract_plugin, validate};
|
||||
|
||||
pub async fn notify(
|
||||
finalize_path: &Path,
|
||||
ctx: &mut ExecutionContext,
|
||||
_event_tx: EventSender,
|
||||
notifyevent_rx: &mut NotificationEventReceiver,
|
||||
notifyevent_tx: NotificationEventSender,
|
||||
mut rx: NotificationEventReceiver,
|
||||
retry_tx: NotificationEventSender,
|
||||
_debug: bool,
|
||||
registry: &ResourceRegistry,
|
||||
) -> anyhow::Result<()> {
|
||||
let mut finalize = parse(finalize_path)?;
|
||||
|
||||
// Always inject in-site-notify as the lowest priority fallback
|
||||
{
|
||||
let max_p = finalize.notification.groups.keys().max().copied().unwrap_or(0);
|
||||
let mut m = std::collections::HashMap::new();
|
||||
m.insert(
|
||||
"in-site-notify".to_string(),
|
||||
crate::finalize::NotificationMethod::default(),
|
||||
);
|
||||
finalize.notification.groups.insert(max_p + 1, m);
|
||||
}
|
||||
|
||||
validate(&finalize)?;
|
||||
let plugin_list = extract_plugin(&finalize);
|
||||
|
||||
if plugin_list.is_empty() {
|
||||
log::warn!("No notification method provided, skipping notification");
|
||||
// TODO: send event to event_tx
|
||||
return Ok(());
|
||||
}
|
||||
let notify_plugins = plugin::register(
|
||||
finalize.plugin,
|
||||
Some(&plugin_list),
|
||||
&mut finalize.notification,
|
||||
registry,
|
||||
)?;
|
||||
|
||||
if ctx.standalone {
|
||||
fetch::fetch_plugins(
|
||||
&PathBuf::from(FINALIZE_STANDALONE_PLUGIN_PATH),
|
||||
&mut finalize.plugin,
|
||||
Some(&plugin_list),
|
||||
)
|
||||
.await?;
|
||||
}
|
||||
let notify_plugins = plugin::register(finalize.plugin, Some(&plugin_list))?;
|
||||
resolve_external_template_refs(&mut finalize.notification.templates, registry);
|
||||
flatten_template_refs(&mut finalize.notification.templates)?;
|
||||
|
||||
let notification_rules = parse_rules(&finalize.notification);
|
||||
|
||||
loop {
|
||||
tokio::select! {
|
||||
Some(event) = notifyevent_rx.recv() => {
|
||||
handle_notification_event(
|
||||
event,
|
||||
¬ify_plugins,
|
||||
¬ification_rules,
|
||||
¬ifyevent_tx,
|
||||
)
|
||||
.await?;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
let handler = Arc::new(NotificationHandler::new(
|
||||
Arc::new(notification_rules),
|
||||
Arc::new(notify_plugins),
|
||||
Arc::new(finalize.notification.templates),
|
||||
Arc::new(ctx.clone()),
|
||||
retry_tx,
|
||||
DEFAULT_NOTIFY_PARAMS.batch_period.value,
|
||||
DEFAULT_NOTIFY_PARAMS.batch_watermark.value,
|
||||
));
|
||||
|
||||
/// Process a single notification event.
|
||||
///
|
||||
/// Delegates to [`notification_handler`] and handles errors:
|
||||
/// - [`NotifyError::PluginNotAvailable`] — reschedule with static delay, lower priority
|
||||
/// - [`NotifyError::TemporaryError`] — consume one life, reschedule with exponential backoff
|
||||
/// - [`NotifyError::PermanentError`] — drop the event
|
||||
/// - other — log and drop
|
||||
async fn handle_notification_event(
|
||||
event: NotificationEvent,
|
||||
plugins: &NotificationPluginMap,
|
||||
rules: &NotificationRulesMap,
|
||||
notifyevent_tx: &NotificationEventSender,
|
||||
) -> anyhow::Result<()> {
|
||||
match notification_handler(plugins, rules, &event) {
|
||||
Ok(_) => {}
|
||||
Err(e) => {
|
||||
log::debug!("Failed to send notification: {:?}", e);
|
||||
match e {
|
||||
NotifyError::PluginNotAvailable(_) => {
|
||||
let mut event = event;
|
||||
event.not_before = Utc::now() + NOTIFICATION_NOTREADY_DELAY;
|
||||
event.effective_priority = event.effective_priority.saturating_add(1);
|
||||
notifyevent_tx.send(event).await?;
|
||||
}
|
||||
NotifyError::TemporaryError(_) => {
|
||||
let mut event = event;
|
||||
event.lives -= 1;
|
||||
event.not_before = Utc::now() + notify_delay(&event);
|
||||
notifyevent_tx.send(event).await?;
|
||||
}
|
||||
NotifyError::PermanentError(_) => {
|
||||
log::error!("None of the configured notification plugins works.");
|
||||
log::error!("Message will be dropped.");
|
||||
}
|
||||
error => {
|
||||
log::error!(
|
||||
"Internal error: notify_handler returned unexpected error: {:?}",
|
||||
error
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
loop {
|
||||
let Some(event) = rx.recv().await else { break };
|
||||
let h = handler.clone();
|
||||
tokio::spawn(async move {
|
||||
h.process_event(event).await;
|
||||
});
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn notification_handler(
|
||||
plugins: &NotificationPluginMap,
|
||||
config: &NotificationRulesMap,
|
||||
event: &NotificationEvent,
|
||||
) -> Result<(), NotifyError> {
|
||||
for (_priority, ruleset) in config.iter() {
|
||||
for (method, rules) in ruleset.iter() {
|
||||
if !event.target.contains(method) {
|
||||
log::trace!("target does not contain method name: {}", method);
|
||||
continue;
|
||||
}
|
||||
for (rule_id, rule) in rules.iter().enumerate() {
|
||||
if !rule.enabled {
|
||||
log::trace!("rule is not enabled: {}", rule_id);
|
||||
continue;
|
||||
}
|
||||
if rule.accept(&event.outcome, &event.stage, &event.substage) {
|
||||
let plugin = plugins
|
||||
.get(method)
|
||||
.ok_or(NotifyError::UnknownPlugin(method.clone()))?;
|
||||
let renderer = plugin.metadata.renderer.clone().unwrap_or_default();
|
||||
match renderer {
|
||||
NotificationRenderer::Server => {
|
||||
// ...
|
||||
todo!();
|
||||
}
|
||||
NotificationRenderer::Plugin => {
|
||||
unimplemented!();
|
||||
return Err(NotifyError::UnsupportedRenderer(method.clone(), renderer));
|
||||
}
|
||||
NotificationRenderer::Interactive => {
|
||||
unimplemented!();
|
||||
return Err(NotifyError::UnsupportedRenderer(method.clone(), renderer));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// TODO: PIPELINE_NOTIFICATION_EXHAUSTED — set when any method returns Exhausted
|
||||
// TODO: PIPELINE_NOTIFICATION_FAILED — set when any method returns PermanentError
|
||||
// Markers sent via event_tx as ExecutionEvent variants for future WebUI/CLI display.
|
||||
|
||||
todo!();
|
||||
}
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::finalize::NotificationTemplateDef;
|
||||
use crate::notify::types::NotificationRule;
|
||||
use crate::notify::types::NotificationTarget;
|
||||
use std::collections::HashMap;
|
||||
use crate::notify::types::NotificationEvent;
|
||||
use crate::notify::types::NotificationRulesMap;
|
||||
use crate::notify::error::NotifyError;
|
||||
use crate::notify::template::resolve_template;
|
||||
use crate::notify::util::{find_rule, notify_delay, wait_time};
|
||||
|
||||
fn parse_trigger_item_raw(item: &TriggerItemRaw) -> Vec<ParsedTrigger> {
|
||||
match item {
|
||||
TriggerItemRaw::Simple(s) => match s.as_str() {
|
||||
"on_success" => vec![ParsedTrigger::OnSuccess],
|
||||
"on_failure" => vec![ParsedTrigger::OnFailure],
|
||||
s if s.starts_with("on_finish_of:") => {
|
||||
let pattern = s.trim_start_matches("on_finish_of:").trim();
|
||||
let mut builder = GlobSetBuilder::new();
|
||||
builder.add(Glob::new(pattern).expect("Invalid glob pattern"));
|
||||
let glob_set = builder.build().expect("Failed to compile glob set");
|
||||
vec![ParsedTrigger::OnFinishOf(Matchable::Glob(glob_set))]
|
||||
}
|
||||
_ => {
|
||||
log::warn!("Unknown trigger string: {}", s);
|
||||
vec![]
|
||||
}
|
||||
},
|
||||
TriggerItemRaw::Map(map) => {
|
||||
let mut triggers = Vec::new();
|
||||
for (key, patterns) in map {
|
||||
match key.as_str() {
|
||||
"on_finish_of" => {
|
||||
for pattern in patterns {
|
||||
let mut builder = GlobSetBuilder::new();
|
||||
builder.add(Glob::new(pattern).expect("Invalid glob pattern"));
|
||||
let glob_set = builder.build().expect("Failed to compile glob set");
|
||||
triggers.push(ParsedTrigger::OnFinishOf(Matchable::Glob(glob_set)));
|
||||
}
|
||||
}
|
||||
"on_success" => triggers.push(ParsedTrigger::OnSuccess),
|
||||
"on_failure" => triggers.push(ParsedTrigger::OnFailure),
|
||||
_ => {
|
||||
log::warn!("Unknown trigger key: {}", key);
|
||||
}
|
||||
}
|
||||
}
|
||||
triggers
|
||||
}
|
||||
}
|
||||
}
|
||||
// ── helper ──
|
||||
|
||||
fn merge_templates(
|
||||
method_template: &NotificationTemplate,
|
||||
policy_template: &Option<serde_yaml::Value>,
|
||||
) -> NotificationTemplate {
|
||||
let mut merged = method_template.clone();
|
||||
if let Some(policy_tmpl) = policy_template
|
||||
&& let Ok(policy_template) =
|
||||
serde_yaml::from_value::<NotificationTemplate>(policy_tmpl.clone())
|
||||
{
|
||||
if policy_template.schema.is_some() {
|
||||
merged.schema = policy_template.schema;
|
||||
}
|
||||
if policy_template.on_success.is_some() {
|
||||
merged.on_success = policy_template.on_success;
|
||||
}
|
||||
if policy_template.on_failure.is_some() {
|
||||
merged.on_failure = policy_template.on_failure;
|
||||
}
|
||||
if policy_template.on_finish_of.is_some() {
|
||||
merged.on_finish_of = policy_template.on_finish_of;
|
||||
}
|
||||
}
|
||||
merged
|
||||
}
|
||||
|
||||
fn extract_overrides(overrides: &serde_yaml::Value) -> (bool, u32, std::time::Duration, f64) {
|
||||
let fallible = overrides
|
||||
.get("fallible")
|
||||
.and_then(|v| v.as_bool())
|
||||
.unwrap_or(NOTIFICATION_DEFAULT_FALLIBLE);
|
||||
let max_lives = overrides
|
||||
.get("max_lives")
|
||||
.and_then(|v| v.as_u64())
|
||||
.unwrap_or(NOTIFICATION_DEFAULT_MAX_LIVES as u64) as u32;
|
||||
let timeout_ms = overrides
|
||||
.get("timeout_ms")
|
||||
.and_then(|v| v.as_u64())
|
||||
.unwrap_or(NOTIFICATION_DEFAULT_TIMEOUT_MS);
|
||||
let retry_backoff = overrides
|
||||
.get("retry_backoff")
|
||||
.and_then(|v| v.as_f64())
|
||||
.unwrap_or(NOTIFICATION_DEFAULT_RETRY_BACKOFF);
|
||||
(
|
||||
fallible,
|
||||
max_lives,
|
||||
std::time::Duration::from_millis(timeout_ms),
|
||||
retry_backoff,
|
||||
)
|
||||
}
|
||||
|
||||
fn parse_rules(config: &NotificationConfig) -> NotificationRulesMap {
|
||||
let mut rules = NotificationRulesMap::default();
|
||||
for (priority, group) in &config.groups {
|
||||
let mut ruleset: HashMap<Method, Vec<NotificationRule>> = HashMap::new();
|
||||
for (method_name, method) in group {
|
||||
let mut method_rules: Vec<NotificationRule> = Vec::new();
|
||||
|
||||
if method.policy.is_empty() {
|
||||
let trigger: Vec<ParsedTrigger> = method
|
||||
.trigger
|
||||
.iter()
|
||||
.flat_map(parse_trigger_item_raw)
|
||||
.collect();
|
||||
let template = method.template.clone();
|
||||
let (fallible, max_lives, timeout, retry_backoff) =
|
||||
extract_overrides(&method.config);
|
||||
method_rules.push(NotificationRule {
|
||||
enabled: true,
|
||||
trigger,
|
||||
template,
|
||||
fallible,
|
||||
max_lives,
|
||||
timeout,
|
||||
config: method.config.clone(),
|
||||
retry_backoff,
|
||||
});
|
||||
fn make_tmpl(on_success: &str, on_failure: &str) -> NotificationTemplateDef {
|
||||
NotificationTemplateDef {
|
||||
use_: None,
|
||||
on_success: if on_success.is_empty() {
|
||||
None
|
||||
} else {
|
||||
for policy in &method.policy {
|
||||
let trigger: Vec<ParsedTrigger> = if policy.trigger.is_empty() {
|
||||
method
|
||||
.trigger
|
||||
.iter()
|
||||
.flat_map(parse_trigger_item_raw)
|
||||
.collect()
|
||||
} else {
|
||||
policy
|
||||
.trigger
|
||||
.iter()
|
||||
.flat_map(parse_trigger_item_raw)
|
||||
.collect()
|
||||
};
|
||||
let template =
|
||||
merge_templates(&method.template, &Some(policy.template.clone()));
|
||||
let method_overrides = extract_overrides(&method.config);
|
||||
let policy_overrides = extract_overrides(&policy.overrides);
|
||||
let fallible = if policy.overrides.get("fallible").is_some() {
|
||||
policy_overrides.0
|
||||
} else {
|
||||
method_overrides.0
|
||||
};
|
||||
let max_lives = if policy.overrides.get("max_lives").is_some() {
|
||||
policy_overrides.1
|
||||
} else {
|
||||
method_overrides.1
|
||||
};
|
||||
let timeout = if policy.overrides.get("timeout_ms").is_some() {
|
||||
policy_overrides.2
|
||||
} else {
|
||||
method_overrides.2
|
||||
};
|
||||
let retry_backoff = if policy.overrides.get("retry_backoff").is_some() {
|
||||
policy_overrides.3
|
||||
} else {
|
||||
method_overrides.3
|
||||
};
|
||||
method_rules.push(NotificationRule {
|
||||
enabled: true,
|
||||
trigger,
|
||||
template,
|
||||
fallible,
|
||||
max_lives,
|
||||
timeout,
|
||||
config: method.config.clone(),
|
||||
retry_backoff,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
ruleset.insert(method_name.clone(), method_rules);
|
||||
Some(on_success.into())
|
||||
},
|
||||
on_failure: if on_failure.is_empty() {
|
||||
None
|
||||
} else {
|
||||
Some(on_failure.into())
|
||||
},
|
||||
on_finish_of: None,
|
||||
}
|
||||
rules.insert(*priority, ruleset);
|
||||
}
|
||||
rules
|
||||
}
|
||||
|
||||
// calculate min(1, k/x)*t as notification grouping period
|
||||
// to avoid excessive notification frequency
|
||||
fn wait_time(group_period: u32, group_watermark: u32, count: u32) -> f32 {
|
||||
if count == 0 {
|
||||
log::warn!("Unexpected notification count: {}", count);
|
||||
return group_period as f32;
|
||||
fn make_tmpl_with_use(use_ref: &str) -> NotificationTemplateDef {
|
||||
NotificationTemplateDef {
|
||||
use_: Some(use_ref.into()),
|
||||
on_success: None,
|
||||
on_failure: None,
|
||||
on_finish_of: None,
|
||||
}
|
||||
}
|
||||
f32::min(1.0, group_watermark as f32 / count as f32) * group_period as f32
|
||||
}
|
||||
|
||||
// calculate p^(m-l)*t as notification delay
|
||||
fn notify_delay(event: &NotificationEvent) -> std::time::Duration {
|
||||
let exponent = (event.max_lives - event.lives) as i32;
|
||||
if exponent < 0 {
|
||||
log::warn!(
|
||||
"Unexpected lives in notification: lives({}) > max_lives({})",
|
||||
event.lives,
|
||||
event.max_lives
|
||||
);
|
||||
return std::time::Duration::from_secs_f64(NOTIFICATION_TEMPORARY_MAX_TIME);
|
||||
// ═══════════════════════════════════════════
|
||||
// resolve_template_content
|
||||
// ═══════════════════════════════════════════
|
||||
|
||||
#[test]
|
||||
fn resolve_none_cd_default_found() {
|
||||
let mut ws = HashMap::new();
|
||||
ws.insert("mail".into(), make_tmpl("OK", "FAIL"));
|
||||
let templates = HashMap::new();
|
||||
let result = resolve_template(&None, "mail", &templates, false, Some(&ws));
|
||||
assert!(result.is_ok());
|
||||
assert_eq!(result.unwrap().on_success.unwrap(), "OK");
|
||||
}
|
||||
let multiplier = NOTIFICATION_TEMPORARY_DELAY_FACTOR.powi(exponent);
|
||||
let delay = multiplier * NOTIFICATION_TEMPORARY_DELAY_TIME;
|
||||
if !delay.is_finite() || delay < 0.0 {
|
||||
log::warn!(
|
||||
"Unexpected calculated delay: {}, treating as infinite",
|
||||
delay
|
||||
);
|
||||
return std::time::Duration::from_secs_f64(NOTIFICATION_TEMPORARY_MAX_TIME);
|
||||
|
||||
#[test]
|
||||
fn resolve_none_cd_default_missing_fallback() {
|
||||
let ws: HashMap<String, NotificationTemplateDef> = HashMap::new();
|
||||
let mut templates = HashMap::new();
|
||||
templates.insert("mail.fallback".into(), make_tmpl("FALLBACK", ""));
|
||||
let result = resolve_template(&None, "mail", &templates, false, Some(&ws));
|
||||
assert!(result.is_ok());
|
||||
assert_eq!(result.unwrap().on_success.unwrap(), "FALLBACK");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_none_cd_no_default_no_fallback_error() {
|
||||
let ws: HashMap<String, NotificationTemplateDef> = HashMap::new();
|
||||
let templates = HashMap::new();
|
||||
let result = resolve_template(&None, "mail", &templates, false, Some(&ws));
|
||||
assert!(result.is_err());
|
||||
assert!(format!("{}", result.unwrap_err()).contains("failed"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_none_standalone_fallback() {
|
||||
let mut templates = HashMap::new();
|
||||
templates.insert("mail.fallback".into(), make_tmpl("STANDALONE", ""));
|
||||
let result = resolve_template(&None, "mail", &templates, true, None);
|
||||
assert!(result.is_ok());
|
||||
assert_eq!(result.unwrap().on_success.unwrap(), "STANDALONE");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_none_standalone_no_fallback_error() {
|
||||
let result = resolve_template(&None, "mail", &HashMap::new(), true, None);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_custom() {
|
||||
let result = resolve_template(&Some("custom".into()), "mail", &HashMap::new(), false, None);
|
||||
assert!(result.is_ok());
|
||||
// "custom" returns an empty default — caller reads rule.template.on_* directly
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_default_standalone_fallback() {
|
||||
let mut templates = HashMap::new();
|
||||
templates.insert("mail.fallback".into(), make_tmpl("FB", ""));
|
||||
let result = resolve_template(&Some("default".into()), "mail", &templates, true, None);
|
||||
assert!(result.is_ok());
|
||||
assert_eq!(result.unwrap().on_success.unwrap(), "FB");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_default_standalone_no_fallback_error() {
|
||||
let result = resolve_template(&Some("default".into()), "mail", &HashMap::new(), true, None);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_default_cd_wsbase() {
|
||||
let mut ws = HashMap::new();
|
||||
ws.insert("mail".into(), make_tmpl("WS_OK", ""));
|
||||
let result = resolve_template(
|
||||
&Some("default".into()),
|
||||
"mail",
|
||||
&HashMap::new(),
|
||||
false,
|
||||
Some(&ws),
|
||||
);
|
||||
assert!(result.is_ok());
|
||||
assert_eq!(result.unwrap().on_success.unwrap(), "WS_OK");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_default_cd_no_wsbase_error() {
|
||||
let result = resolve_template(
|
||||
&Some("default".into()),
|
||||
"mail",
|
||||
&HashMap::new(),
|
||||
false,
|
||||
None,
|
||||
);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_named_found() {
|
||||
let mut templates = HashMap::new();
|
||||
templates.insert("sms-tmpl1".into(), make_tmpl("NAMED", ""));
|
||||
let result = resolve_template(&Some("sms-tmpl1".into()), "mail", &templates, false, None);
|
||||
assert!(result.is_ok());
|
||||
assert_eq!(result.unwrap().on_success.unwrap(), "NAMED");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_named_not_found_method_match_fallback() {
|
||||
let mut templates = HashMap::new();
|
||||
templates.insert("mail.fallback".into(), make_tmpl("FB", ""));
|
||||
let result = resolve_template(&Some("mail".into()), "mail", &templates, false, None);
|
||||
assert!(result.is_ok());
|
||||
assert_eq!(result.unwrap().on_success.unwrap(), "FB");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_named_not_found_no_match_error() {
|
||||
let result = resolve_template(&Some("ghost".into()), "mail", &HashMap::new(), false, None);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
|
||||
// ═══════════════════════════════════════════
|
||||
// flatten_template_refs
|
||||
// ═══════════════════════════════════════════
|
||||
|
||||
#[test]
|
||||
fn flatten_no_refs_unchanged() {
|
||||
let mut templates = HashMap::new();
|
||||
templates.insert("a".into(), make_tmpl("A_OK", "A_FAIL"));
|
||||
let result = flatten_template_refs(&mut templates);
|
||||
assert!(result.is_ok());
|
||||
assert_eq!(templates["a"].on_success.as_deref(), Some("A_OK"));
|
||||
assert!(templates["a"].use_.is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn flatten_single_ref_merged() {
|
||||
let mut templates = HashMap::new();
|
||||
templates.insert("base".into(), make_tmpl("BASE_OK", "BASE_FAIL"));
|
||||
templates.insert("child".into(), make_tmpl_with_use("base"));
|
||||
let result = flatten_template_refs(&mut templates);
|
||||
assert!(result.is_ok());
|
||||
assert_eq!(templates["child"].on_success.as_deref(), Some("BASE_OK"));
|
||||
assert_eq!(templates["child"].on_failure.as_deref(), Some("BASE_FAIL"));
|
||||
assert!(templates["child"].use_.is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn flatten_child_overrides_parent() {
|
||||
let mut templates = HashMap::new();
|
||||
templates.insert("base".into(), make_tmpl("BASE_OK", "BASE_FAIL"));
|
||||
templates.insert(
|
||||
"child".into(),
|
||||
NotificationTemplateDef {
|
||||
use_: Some("base".into()),
|
||||
on_success: Some("MY_OK".into()),
|
||||
on_failure: None,
|
||||
on_finish_of: None,
|
||||
},
|
||||
);
|
||||
let result = flatten_template_refs(&mut templates);
|
||||
assert!(result.is_ok());
|
||||
assert_eq!(templates["child"].on_success.as_deref(), Some("MY_OK"));
|
||||
assert_eq!(templates["child"].on_failure.as_deref(), Some("BASE_FAIL"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn flatten_chain_a_b_c() {
|
||||
let mut templates = HashMap::new();
|
||||
templates.insert("c".into(), make_tmpl("C_OK", "C_FAIL"));
|
||||
templates.insert("b".into(), make_tmpl_with_use("c"));
|
||||
templates.insert("a".into(), make_tmpl_with_use("b"));
|
||||
let result = flatten_template_refs(&mut templates);
|
||||
assert!(result.is_ok());
|
||||
assert_eq!(templates["a"].on_success.as_deref(), Some("C_OK"));
|
||||
assert_eq!(templates["b"].on_success.as_deref(), Some("C_OK"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn flatten_cycle_detected() {
|
||||
let mut templates = HashMap::new();
|
||||
templates.insert("x".into(), make_tmpl_with_use("y"));
|
||||
templates.insert("y".into(), make_tmpl_with_use("x"));
|
||||
let result = flatten_template_refs(&mut templates);
|
||||
assert!(result.is_err());
|
||||
match result.unwrap_err() {
|
||||
NotifyError::TemplateLoop(_) => {}
|
||||
other => panic!("expected TemplateLoop, got {:?}", other),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn flatten_self_ref_detected() {
|
||||
let mut templates = HashMap::new();
|
||||
templates.insert("self".into(), make_tmpl_with_use("self"));
|
||||
let result = flatten_template_refs(&mut templates);
|
||||
assert!(result.is_err());
|
||||
match result.unwrap_err() {
|
||||
NotifyError::TemplateLoop(_) => {}
|
||||
other => panic!("expected TemplateLoop, got {:?}", other),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn flatten_nonexistent_ref_error() {
|
||||
let mut templates = HashMap::new();
|
||||
templates.insert("orphan".into(), make_tmpl_with_use("ghost"));
|
||||
let result = flatten_template_refs(&mut templates);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn flatten_external_url_preserved() {
|
||||
let mut templates = HashMap::new();
|
||||
templates.insert(
|
||||
"ext".into(),
|
||||
NotificationTemplateDef {
|
||||
use_: Some("git://example.com/repo@v1".into()),
|
||||
on_success: None,
|
||||
on_failure: None,
|
||||
on_finish_of: None,
|
||||
},
|
||||
);
|
||||
let result = flatten_template_refs(&mut templates);
|
||||
assert!(result.is_ok());
|
||||
// External URLs are NOT flattened — use_ preserved
|
||||
assert_eq!(
|
||||
templates["ext"].use_.as_deref(),
|
||||
Some("git://example.com/repo@v1")
|
||||
);
|
||||
}
|
||||
|
||||
// ═══════════════════════════════════════════
|
||||
// wait_time
|
||||
// ═══════════════════════════════════════════
|
||||
|
||||
#[test]
|
||||
fn wait_time_zero_count() {
|
||||
let w = wait_time(30, 5, 0);
|
||||
assert_eq!(w, 30.0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn wait_time_at_watermark() {
|
||||
let w = wait_time(30, 5, 5);
|
||||
assert_eq!(w, 30.0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn wait_time_below_watermark() {
|
||||
let w = wait_time(30, 5, 2);
|
||||
assert!((w - 30.0).abs() < 0.01);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn wait_time_above_watermark() {
|
||||
let w = wait_time(30, 5, 10);
|
||||
assert!((w - 15.0).abs() < 0.01);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn wait_time_far_above_watermark() {
|
||||
let w = wait_time(30, 5, 100);
|
||||
assert!((w - 1.5).abs() < 0.01);
|
||||
}
|
||||
|
||||
// ═══════════════════════════════════════════
|
||||
// notify_delay
|
||||
// ═══════════════════════════════════════════
|
||||
|
||||
fn make_event(count: u32, retry_backoff: f64) -> NotificationEvent {
|
||||
NotificationEvent {
|
||||
id: uuid::Uuid::new_v4(),
|
||||
stage: crate::types::buildstatus::StagePhase::Bake,
|
||||
substage: "build".into(),
|
||||
outcome: crate::types::buildstatus::StageOutcome::PipelineSuccess,
|
||||
not_before: chrono::Utc::now(),
|
||||
priority: 0,
|
||||
effective_priority: 0,
|
||||
attempted: 0,
|
||||
retry_backoff,
|
||||
target: NotificationTarget {
|
||||
priority: 0,
|
||||
rules: None,
|
||||
count,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn notify_delay_default_factor() {
|
||||
let event = make_event(1, 2.0);
|
||||
let delay = notify_delay(&event);
|
||||
assert!(delay.as_secs_f64() > 0.0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn notify_delay_larger_factor() {
|
||||
let event1 = make_event(1, 2.0);
|
||||
let event2 = make_event(1, 4.0);
|
||||
let d1 = notify_delay(&event1).as_secs_f64();
|
||||
let d2 = notify_delay(&event2).as_secs_f64();
|
||||
assert!(d2 > d1, "larger retry_backoff should produce longer delay");
|
||||
}
|
||||
|
||||
// ═══════════════════════════════════════════
|
||||
// find_rule
|
||||
// ═══════════════════════════════════════════
|
||||
|
||||
#[test]
|
||||
fn find_rule_found() {
|
||||
use crate::finalize::NotificationTemplate;
|
||||
let rule = NotificationRule {
|
||||
enabled: true,
|
||||
trigger: vec![],
|
||||
template: NotificationTemplate {
|
||||
schema: None,
|
||||
on_success: None,
|
||||
on_failure: None,
|
||||
on_finish_of: None,
|
||||
},
|
||||
fallible: false,
|
||||
max_lives: 3,
|
||||
timeout: std::time::Duration::from_secs(30),
|
||||
config: serde_yaml::Value::Null,
|
||||
retry_backoff: 2.0,
|
||||
};
|
||||
let mut rules_map: NotificationRulesMap = HashMap::new();
|
||||
rules_map.insert(0, {
|
||||
let mut m: HashMap<String, Vec<NotificationRule>> = HashMap::new();
|
||||
m.insert("mail".into(), vec![rule]);
|
||||
m
|
||||
});
|
||||
let result = find_rule(&rules_map, "mail", 0);
|
||||
assert!(result.is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn find_rule_not_found() {
|
||||
let rules_map: NotificationRulesMap = HashMap::new();
|
||||
let result = find_rule(&rules_map, "mail", 0);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
std::time::Duration::from_secs_f64(delay)
|
||||
}
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
use crate::notify::NotificationRenderer;
|
||||
use crate::notify::types::NotificationRenderer;
|
||||
use thiserror::Error;
|
||||
|
||||
#[derive(Error, Debug)]
|
||||
@@ -23,4 +23,16 @@ pub enum NotifyError {
|
||||
|
||||
#[error("Unsupported renderer for {0}: {1}")]
|
||||
UnsupportedRenderer(String, NotificationRenderer),
|
||||
|
||||
#[error("Schema error: {0}")]
|
||||
SchemaError(String),
|
||||
|
||||
#[error("Template render error: {0}")]
|
||||
TemplateRenderError(String),
|
||||
|
||||
#[error("Circular template reference: {0}")]
|
||||
TemplateLoop(String),
|
||||
|
||||
#[error("Template reference depth exceeded (max {max}): {name}")]
|
||||
TemplateDepthExceeded { name: String, max: u32 },
|
||||
}
|
||||
|
||||
@@ -0,0 +1,345 @@
|
||||
use std::collections::HashMap;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::Duration;
|
||||
|
||||
use chrono::Utc;
|
||||
use tokio::sync::Notify;
|
||||
|
||||
use crate::finalize::NotificationTemplateDef;
|
||||
use crate::notify::template::{
|
||||
build_batch_template_context, render_template, resolve_template,
|
||||
select_template_str,
|
||||
};
|
||||
use crate::notify::types::{
|
||||
MethodStatus, NotificationEvent,
|
||||
NotificationEventSender, NotificationPluginMap, NotificationRenderer,
|
||||
NotificationRulesMap,
|
||||
};
|
||||
use crate::notify::util::{
|
||||
find_rule, get_fallible, is_temporary_error, notify_delay, wait_time,
|
||||
};
|
||||
|
||||
/// Shared state for a single dispatch bucket.
|
||||
struct BucketEntry {
|
||||
events: Arc<Mutex<Vec<NotificationEvent>>>,
|
||||
result: Arc<Mutex<Option<(MethodStatus, Vec<NotificationEvent>)>>>,
|
||||
done: Arc<AtomicBool>,
|
||||
notify: Arc<Notify>,
|
||||
method: String,
|
||||
rule_id: usize,
|
||||
_handle: tokio::task::JoinHandle<()>,
|
||||
}
|
||||
|
||||
/// Concurrent notification handler with bucket-based dispatch.
|
||||
///
|
||||
/// Events are routed to matching buckets. Buckets cool down and dispatch
|
||||
/// plugins concurrently. The caller awaits bucket completion, then handles
|
||||
/// retry/escalation decisions.
|
||||
pub struct NotificationHandler {
|
||||
buckets: Mutex<HashMap<(String, usize), BucketEntry>>,
|
||||
pub config: Arc<NotificationRulesMap>,
|
||||
plugins: Arc<NotificationPluginMap>,
|
||||
templates: Arc<HashMap<String, NotificationTemplateDef>>,
|
||||
ctx: Arc<workshop_engine::ExecutionContext>,
|
||||
retry_tx: NotificationEventSender,
|
||||
period: u32,
|
||||
watermark: u32,
|
||||
}
|
||||
|
||||
impl NotificationHandler {
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub fn new(
|
||||
config: Arc<NotificationRulesMap>,
|
||||
plugins: Arc<NotificationPluginMap>,
|
||||
templates: Arc<HashMap<String, NotificationTemplateDef>>,
|
||||
ctx: Arc<workshop_engine::ExecutionContext>,
|
||||
retry_tx: NotificationEventSender,
|
||||
period: u32,
|
||||
watermark: u32,
|
||||
) -> Self {
|
||||
NotificationHandler {
|
||||
buckets: Mutex::new(HashMap::new()),
|
||||
config,
|
||||
plugins,
|
||||
templates,
|
||||
ctx,
|
||||
retry_tx,
|
||||
period,
|
||||
watermark,
|
||||
}
|
||||
}
|
||||
|
||||
/// Process a single event: route to buckets, await completion,
|
||||
/// then handle retry/escalation for each bucket result.
|
||||
pub async fn process_event(self: Arc<Self>, event: NotificationEvent) {
|
||||
let entries = self.route_event(&event);
|
||||
if entries.is_empty() {
|
||||
return;
|
||||
}
|
||||
|
||||
// Await all buckets — yields CPU
|
||||
for (_, notify) in &entries {
|
||||
notify.notified().await;
|
||||
}
|
||||
|
||||
// Collect results and remove entries
|
||||
let mut results: Vec<(String, usize, Option<(MethodStatus, Vec<NotificationEvent>)>)> =
|
||||
Vec::new();
|
||||
{
|
||||
let mut buckets = self.buckets.lock().unwrap();
|
||||
for (key, _) in &entries {
|
||||
if let Some(entry) = buckets.remove(key) {
|
||||
let r = entry.result.lock().unwrap().take();
|
||||
results.push((entry.method, entry.rule_id, r));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Handle retry/escalation per method
|
||||
for (method, _rule_id, result) in results {
|
||||
let Some((status, mut events)) = result else { continue };
|
||||
|
||||
match &status {
|
||||
MethodStatus::Success => {
|
||||
log::debug!("[NOTIFY] {}: success ({} events)", method, events.len());
|
||||
}
|
||||
MethodStatus::TemporaryError(_) => {
|
||||
log::warn!("[NOTIFY] {}: temporary error, retrying", method);
|
||||
for event in events.iter_mut() {
|
||||
event.target.rules = Some(vec![method.clone()]);
|
||||
event.target.count += 1;
|
||||
event.not_before = Utc::now() + notify_delay(event);
|
||||
let _ = self.retry_tx.send(event.clone()).await;
|
||||
}
|
||||
}
|
||||
MethodStatus::PermanentError(_) | MethodStatus::Exhausted(_) => {
|
||||
let fallible = get_fallible(&self.config, &method);
|
||||
if fallible {
|
||||
log::error!("[NOTIFY] {}: discard (fallible)", method);
|
||||
} else {
|
||||
log::warn!("[NOTIFY] {}: escalating priority", method);
|
||||
for event in events.iter_mut() {
|
||||
event.target.priority += 1;
|
||||
event.target.rules = None;
|
||||
event.attempted = event.target.count;
|
||||
event.target.count = 0;
|
||||
event.effective_priority = event.priority;
|
||||
let _ = self.retry_tx.send(event.clone()).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Route a single event to all matching buckets.
|
||||
///
|
||||
/// Returns Notify handles for awaiting bucket completion.
|
||||
fn route_event(
|
||||
&self,
|
||||
event: &NotificationEvent,
|
||||
) -> Vec<((String, usize), Arc<Notify>)> {
|
||||
let mut matches: Vec<(String, usize)> = Vec::new();
|
||||
|
||||
for ruleset in self.config.values() {
|
||||
for (method, rules) in ruleset.iter() {
|
||||
if let Some(ref method_rules) = event.target.rules {
|
||||
if !method_rules.contains(method) {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
for (rule_id, rule) in rules.iter().enumerate() {
|
||||
if !rule.enabled {
|
||||
continue;
|
||||
}
|
||||
if rule.accept(&event.outcome, &event.stage, &event.substage) {
|
||||
matches.push((method.clone(), rule_id));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let mut buckets = self.buckets.lock().unwrap();
|
||||
let mut result: Vec<((String, usize), Arc<Notify>)> = Vec::new();
|
||||
|
||||
for key in matches {
|
||||
if let Some(entry) = buckets.get(&key) {
|
||||
if entry.done.load(Ordering::Acquire) {
|
||||
buckets.remove(&key);
|
||||
}
|
||||
}
|
||||
|
||||
if let Some(entry) = buckets.get(&key) {
|
||||
entry.events.lock().unwrap().push(event.clone());
|
||||
result.push((key, entry.notify.clone()));
|
||||
} else {
|
||||
let (entry, notify) = Self::spawn_bucket(
|
||||
&key.0,
|
||||
key.1,
|
||||
&self.config,
|
||||
&self.plugins,
|
||||
&self.templates,
|
||||
&self.ctx,
|
||||
self.period,
|
||||
self.watermark,
|
||||
);
|
||||
entry.events.lock().unwrap().push(event.clone());
|
||||
result.push((key.clone(), notify));
|
||||
buckets.insert(key, entry);
|
||||
}
|
||||
}
|
||||
|
||||
result
|
||||
}
|
||||
|
||||
/// Spawn a bucket task: cooldown → take → dispatch → store → notify.
|
||||
fn spawn_bucket(
|
||||
method: &str,
|
||||
rule_id: usize,
|
||||
config: &Arc<NotificationRulesMap>,
|
||||
plugins: &Arc<NotificationPluginMap>,
|
||||
templates: &Arc<HashMap<String, NotificationTemplateDef>>,
|
||||
ctx: &Arc<workshop_engine::ExecutionContext>,
|
||||
period: u32,
|
||||
watermark: u32,
|
||||
) -> (BucketEntry, Arc<Notify>) {
|
||||
let events: Arc<Mutex<Vec<NotificationEvent>>> = Arc::new(Mutex::new(Vec::new()));
|
||||
let result: Arc<Mutex<Option<(MethodStatus, Vec<NotificationEvent>)>>> =
|
||||
Arc::new(Mutex::new(None));
|
||||
let notify = Arc::new(Notify::new());
|
||||
let done = Arc::new(AtomicBool::new(false));
|
||||
|
||||
let ec = events.clone();
|
||||
let rc = result.clone();
|
||||
let nc = notify.clone();
|
||||
let dc = done.clone();
|
||||
let m = method.to_string();
|
||||
let c = config.clone();
|
||||
let p = plugins.clone();
|
||||
let t = templates.clone();
|
||||
let x = ctx.clone();
|
||||
|
||||
let handle = tokio::spawn(async move {
|
||||
// Phase 1: Cooldown
|
||||
let count = ec.lock().unwrap().len() as u32;
|
||||
let wait_secs = wait_time(period, watermark, count);
|
||||
tokio::time::sleep(Duration::from_secs_f32(wait_secs)).await;
|
||||
|
||||
// Phase 2: Take events
|
||||
let batch = std::mem::take(&mut *ec.lock().unwrap());
|
||||
if batch.is_empty() {
|
||||
dc.store(true, Ordering::Release);
|
||||
nc.notify_waiters();
|
||||
return;
|
||||
}
|
||||
|
||||
// Phase 3: Dispatch
|
||||
let rule = match find_rule(&c, &m, rule_id) {
|
||||
Ok(r) => r,
|
||||
Err(_) => {
|
||||
*rc.lock().unwrap() = Some((MethodStatus::PermanentError("rule not found".into()), batch));
|
||||
dc.store(true, Ordering::Release);
|
||||
nc.notify_waiters();
|
||||
return;
|
||||
}
|
||||
};
|
||||
|
||||
let plugin = match p.get(&m) {
|
||||
Some(p) => p,
|
||||
None => {
|
||||
*rc.lock().unwrap() = Some((MethodStatus::PermanentError("plugin not found".into()), batch));
|
||||
dc.store(true, Ordering::Release);
|
||||
nc.notify_waiters();
|
||||
return;
|
||||
}
|
||||
};
|
||||
|
||||
let renderer = plugin.metadata.renderer.clone().unwrap_or_default();
|
||||
let argument = build_plugin_argument(&renderer, rule, &m, &t, &x, &batch);
|
||||
let status = match argument {
|
||||
Some(arg) => {
|
||||
match tokio::time::timeout(rule.timeout, plugin.call(arg, &x, &None)).await {
|
||||
Ok(Ok(())) => MethodStatus::Success,
|
||||
Ok(Err(e)) if is_temporary_error(&e) => {
|
||||
MethodStatus::TemporaryError(format!("{}", e))
|
||||
}
|
||||
Ok(Err(e)) => MethodStatus::PermanentError(format!("{}", e)),
|
||||
Err(_) => MethodStatus::TemporaryError("timeout".into()),
|
||||
}
|
||||
}
|
||||
None => MethodStatus::PermanentError("argument build failed".into()),
|
||||
};
|
||||
|
||||
*rc.lock().unwrap() = Some((status, batch));
|
||||
dc.store(true, Ordering::Release);
|
||||
nc.notify_waiters();
|
||||
});
|
||||
|
||||
let entry = BucketEntry {
|
||||
events,
|
||||
result,
|
||||
done,
|
||||
notify: notify.clone(),
|
||||
method: method.to_string(),
|
||||
rule_id,
|
||||
_handle: handle,
|
||||
};
|
||||
|
||||
(entry, notify)
|
||||
}
|
||||
}
|
||||
|
||||
/// Build the YAML argument for a plugin call based on renderer mode.
|
||||
fn build_plugin_argument(
|
||||
renderer: &NotificationRenderer,
|
||||
rule: &crate::notify::types::NotificationRule,
|
||||
method: &str,
|
||||
templates: &HashMap<String, NotificationTemplateDef>,
|
||||
ctx: &workshop_engine::ExecutionContext,
|
||||
events: &[NotificationEvent],
|
||||
) -> Option<serde_yaml::Value> {
|
||||
let first = events.first()?;
|
||||
|
||||
match renderer {
|
||||
NotificationRenderer::Server => {
|
||||
let template_def = resolve_template(
|
||||
&rule.template.schema,
|
||||
method,
|
||||
templates,
|
||||
ctx.standalone,
|
||||
None,
|
||||
)
|
||||
.ok()?;
|
||||
|
||||
let template_str = select_template_str(&template_def, &first.outcome)
|
||||
.or_else(|| rule.template.on_success.as_deref())
|
||||
.or_else(|| rule.template.on_failure.as_deref())
|
||||
.or_else(|| {
|
||||
rule.template
|
||||
.on_finish_of
|
||||
.as_ref()
|
||||
.map(|o| o.content.as_str())
|
||||
})?;
|
||||
|
||||
let data = build_batch_template_context(ctx, events);
|
||||
let rendered = render_template(template_str, &data).ok()?;
|
||||
|
||||
let mut argument = rule.config.clone();
|
||||
if let serde_yaml::Value::Mapping(ref mut map) = argument {
|
||||
map.insert(
|
||||
serde_yaml::Value::String("_rendered_content".into()),
|
||||
serde_yaml::Value::String(rendered),
|
||||
);
|
||||
}
|
||||
log::info!("SSR: rendered content for method '{}'", method);
|
||||
Some(argument)
|
||||
}
|
||||
NotificationRenderer::Plugin => {
|
||||
unimplemented!("Plugin renderer is not yet implemented")
|
||||
}
|
||||
NotificationRenderer::Interactive => {
|
||||
unimplemented!("Interactive renderer is not yet implemented")
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
use crate::notify::types::{NotificationEvent, NotificationEventReceiver, NotificationEventSender};
|
||||
use std::collections::BinaryHeap;
|
||||
|
||||
/// Priority queue for notification events.
|
||||
///
|
||||
/// Integrates with `notify()`:
|
||||
/// - `tx` sends events to notify() for processing
|
||||
/// - `rx` receives retry events back from notify()
|
||||
pub struct NotificationQueue {
|
||||
pub pq: BinaryHeap<NotificationEvent>,
|
||||
pub tx: NotificationEventSender,
|
||||
pub rx: NotificationEventReceiver,
|
||||
}
|
||||
|
||||
impl NotificationQueue {
|
||||
pub fn new(tx: NotificationEventSender, rx: NotificationEventReceiver) -> Self {
|
||||
Self {
|
||||
pq: BinaryHeap::new(),
|
||||
tx,
|
||||
rx,
|
||||
}
|
||||
}
|
||||
|
||||
/// Push an event into the priority queue.
|
||||
pub fn enqueue(&mut self, event: NotificationEvent) {
|
||||
self.pq.push(event);
|
||||
}
|
||||
|
||||
/// Non-blocking single round: recycle retry events into PQ, then
|
||||
/// pop and send all ready events (not_before ≤ now).
|
||||
pub async fn flush_now(&mut self) -> anyhow::Result<()> {
|
||||
// Recycle retry events
|
||||
while let Ok(event) = self.rx.try_recv() {
|
||||
self.pq.push(event);
|
||||
}
|
||||
// Send ready events
|
||||
let now = chrono::Utc::now();
|
||||
while let Some(peek) = self.pq.peek() {
|
||||
if peek.not_before > now {
|
||||
break;
|
||||
}
|
||||
let event = self.pq.pop().unwrap();
|
||||
self.tx.send(event).await?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Blocking drain until all events are processed.
|
||||
/// For standalone mode: called after prebake/bake/finalize to wait for completion.
|
||||
/// For c/d mode: the main execution loop.
|
||||
pub async fn drain(&mut self) -> anyhow::Result<()> {
|
||||
loop {
|
||||
// Drain retry channel
|
||||
while let Ok(event) = self.rx.try_recv() {
|
||||
self.pq.push(event);
|
||||
}
|
||||
// Determine sleep deadline
|
||||
let until = self
|
||||
.pq
|
||||
.peek()
|
||||
.map(|e| e.not_before)
|
||||
.unwrap_or_else(|| chrono::Utc::now() + chrono::Duration::seconds(3600));
|
||||
|
||||
let std_until = (until - chrono::Utc::now())
|
||||
.to_std()
|
||||
.unwrap_or(std::time::Duration::from_secs(1));
|
||||
|
||||
tokio::select! {
|
||||
event = self.rx.recv() => {
|
||||
match event {
|
||||
Some(event) => self.pq.push(event),
|
||||
None => break Ok(()),
|
||||
}
|
||||
}
|
||||
_ = tokio::time::sleep(std_until) => {
|
||||
let now = chrono::Utc::now();
|
||||
while let Some(peek) = self.pq.peek() {
|
||||
if peek.not_before > now { break; }
|
||||
let event = self.pq.pop().unwrap();
|
||||
self.tx.send(event).await?;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,148 @@
|
||||
use crate::finalize::NotificationConfig;
|
||||
use crate::finalize::NotificationTemplate;
|
||||
use crate::finalize::config::{NotifyPolicy, TriggerItemRaw};
|
||||
use crate::notify::types::{DEFAULT_NOTIFY_PARAMS, Matchable, Method, NotificationRule, NotificationRulesMap, ParsedTrigger};
|
||||
use globset::{Glob, GlobSetBuilder};
|
||||
use std::collections::HashMap;
|
||||
|
||||
/// Parse a single `on_finish_of` pattern string into a ParsedTrigger, or None.
|
||||
/// Supports `re:` prefix for regex patterns, otherwise treated as glob.
|
||||
fn parse_on_finish_of(pattern: &str) -> Option<ParsedTrigger> {
|
||||
let pattern = pattern.trim();
|
||||
if let Some(re_str) = pattern.strip_prefix("re:") {
|
||||
regex::Regex::new(re_str).ok().map(|r| ParsedTrigger::OnFinishOf(Matchable::Regex(r)))
|
||||
} else {
|
||||
Glob::new(pattern).ok().and_then(|g| {
|
||||
let mut builder = GlobSetBuilder::new();
|
||||
builder.add(g);
|
||||
builder.build().ok()
|
||||
}).map(|gs| ParsedTrigger::OnFinishOf(Matchable::Glob(gs)))
|
||||
}
|
||||
}
|
||||
|
||||
fn parse_trigger_item_raw(item: &TriggerItemRaw) -> Vec<ParsedTrigger> {
|
||||
// Normalize Simple → single-entry Vec, then unify both branches
|
||||
let entries: Vec<(&str, Vec<&str>)> = match item {
|
||||
TriggerItemRaw::Simple(s) => {
|
||||
if let Some(pattern) = s.strip_prefix("on_finish_of:") {
|
||||
vec![("on_finish_of", vec![pattern.trim()])]
|
||||
} else {
|
||||
vec![(s.as_str(), vec![])]
|
||||
}
|
||||
}
|
||||
TriggerItemRaw::Map(map) => map.iter().map(|(k, v)|
|
||||
(k.as_str(), v.iter().map(|s| s.as_str()).collect())
|
||||
).collect(),
|
||||
};
|
||||
|
||||
let mut triggers = Vec::new();
|
||||
for (key, patterns) in entries {
|
||||
match key {
|
||||
"on_finish_of" => {
|
||||
for pattern in patterns {
|
||||
if let Some(t) = parse_on_finish_of(pattern) {
|
||||
triggers.push(t);
|
||||
}
|
||||
}
|
||||
}
|
||||
"on_success" => triggers.push(ParsedTrigger::OnSuccess),
|
||||
"on_failure" => triggers.push(ParsedTrigger::OnFailure),
|
||||
_ => log::warn!("Unknown trigger key: {}", key),
|
||||
}
|
||||
}
|
||||
triggers
|
||||
}
|
||||
|
||||
fn merge_templates(
|
||||
method_template: &NotificationTemplate,
|
||||
policy_template: &Option<serde_yaml::Value>,
|
||||
) -> NotificationTemplate {
|
||||
let mut merged = method_template.clone();
|
||||
if let Some(policy_tmpl) = policy_template
|
||||
&& let Ok(policy) = serde_yaml::from_value::<NotificationTemplate>(policy_tmpl.clone())
|
||||
{
|
||||
merged += policy;
|
||||
}
|
||||
merged
|
||||
}
|
||||
|
||||
struct RuleOverrides {
|
||||
fallible: bool,
|
||||
max_lives: u32,
|
||||
timeout: std::time::Duration,
|
||||
retry_backoff: f64,
|
||||
}
|
||||
|
||||
impl RuleOverrides {
|
||||
fn from_value(v: &serde_yaml::Value) -> Self {
|
||||
RuleOverrides {
|
||||
fallible: v.get("fallible").and_then(|v| v.as_bool()).unwrap_or(false),
|
||||
max_lives: v.get("max_lives").and_then(|v| v.as_u64()).unwrap_or(DEFAULT_NOTIFY_PARAMS.max_lives.value as u64) as u32,
|
||||
timeout: std::time::Duration::from_millis(
|
||||
v.get("timeout_ms").and_then(|v| v.as_u64()).unwrap_or(DEFAULT_NOTIFY_PARAMS.timeout_ms.value)
|
||||
),
|
||||
retry_backoff: v.get("retry_backoff").and_then(|v| v.as_f64()).unwrap_or(DEFAULT_NOTIFY_PARAMS.retry_backoff.value),
|
||||
}
|
||||
}
|
||||
|
||||
/// Chain: apply `source` atop existing values (source wins when present).
|
||||
fn apply(self, source: &serde_yaml::Value) -> Self {
|
||||
RuleOverrides {
|
||||
fallible: source.get("fallible").and_then(|v| v.as_bool()).unwrap_or(self.fallible),
|
||||
max_lives: source.get("max_lives").and_then(|v| v.as_u64()).unwrap_or(self.max_lives as u64) as u32,
|
||||
timeout: source.get("timeout_ms").and_then(|v| v.as_u64()).map(std::time::Duration::from_millis).unwrap_or(self.timeout),
|
||||
retry_backoff: source.get("retry_backoff").and_then(|v| v.as_f64()).unwrap_or(self.retry_backoff),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn parse_rules(config: &NotificationConfig) -> NotificationRulesMap {
|
||||
log::info!("[NOTIFY] parse_rules: groups={}", config.groups.len());
|
||||
let mut rules = NotificationRulesMap::default();
|
||||
for (priority, group) in &config.groups {
|
||||
let mut ruleset: HashMap<Method, Vec<NotificationRule>> = HashMap::new();
|
||||
for (method_name, method) in group {
|
||||
let mut method_rules = Vec::new();
|
||||
|
||||
// Unify: no policy → one empty policy (same merge path)
|
||||
let empty_policy = NotifyPolicy {
|
||||
trigger: vec![],
|
||||
template: serde_yaml::Value::Null,
|
||||
overrides: serde_yaml::Value::Null,
|
||||
};
|
||||
let policies: &[NotifyPolicy] = if method.policy.is_empty() {
|
||||
std::slice::from_ref(&empty_policy)
|
||||
} else {
|
||||
&method.policy
|
||||
};
|
||||
|
||||
for policy in policies {
|
||||
let trigger: Vec<ParsedTrigger> = if policy.trigger.is_empty() {
|
||||
method.trigger.iter().flat_map(parse_trigger_item_raw).collect()
|
||||
} else {
|
||||
policy.trigger.iter().flat_map(parse_trigger_item_raw).collect()
|
||||
};
|
||||
let template =
|
||||
merge_templates(&method.template, &Some(policy.template.clone()));
|
||||
let vals = RuleOverrides::from_value(&serde_yaml::Value::Null)
|
||||
.apply(&method.config)
|
||||
.apply(&policy.overrides);
|
||||
method_rules.push(NotificationRule {
|
||||
enabled: true,
|
||||
trigger,
|
||||
template,
|
||||
fallible: vals.fallible,
|
||||
max_lives: vals.max_lives,
|
||||
timeout: vals.timeout,
|
||||
config: method.config.clone(),
|
||||
retry_backoff: vals.retry_backoff,
|
||||
});
|
||||
}
|
||||
|
||||
ruleset.insert(method_name.clone(), method_rules);
|
||||
}
|
||||
rules.insert(*priority, ruleset);
|
||||
}
|
||||
rules
|
||||
}
|
||||
|
||||
@@ -0,0 +1,281 @@
|
||||
use std::collections::HashMap;
|
||||
|
||||
use minijinja::Environment;
|
||||
use workshop_engine::ExecutionContext;
|
||||
|
||||
use crate::finalize::NotificationTemplateDef;
|
||||
use crate::notify::error::NotifyError;
|
||||
use crate::notify::types::NotificationEvent;
|
||||
use crate::types::buildstatus::StageOutcome;
|
||||
use crate::types::resource::ResourceRegistry;
|
||||
|
||||
/// Maximum depth for template cross-references (prevent infinite recursion)
|
||||
pub const TEMPLATE_REFERENCE_DEPTH: u32 = 10;
|
||||
|
||||
/// Build template context for a batch of notification events.
|
||||
pub fn build_batch_template_context(
|
||||
ctx: &ExecutionContext,
|
||||
events: &[NotificationEvent],
|
||||
) -> HashMap<String, serde_json::Value> {
|
||||
let mut data = HashMap::new();
|
||||
|
||||
data.insert("pipeline".into(), serde_json::json!({
|
||||
"name": ctx.pipeline_name,
|
||||
}));
|
||||
|
||||
data.insert("count".into(), serde_json::json!(events.len()));
|
||||
|
||||
let event_list: Vec<_> = events
|
||||
.iter()
|
||||
.map(|e| {
|
||||
serde_json::json!({
|
||||
"stage": format!("{}.{}", e.stage.as_str(), e.substage),
|
||||
"status": e.outcome.to_string(),
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
data.insert("events".into(), serde_json::json!(event_list));
|
||||
|
||||
// Single event compatibility: provide direct access
|
||||
if events.len() == 1 {
|
||||
let e = &events[0];
|
||||
data.insert("build".into(), serde_json::json!({
|
||||
"id": ctx.task_id,
|
||||
"status": e.outcome.to_string(),
|
||||
"stage": format!("{}.{}", e.stage.as_str(), e.substage),
|
||||
"duration": "0s",
|
||||
"url": "",
|
||||
}));
|
||||
} else {
|
||||
data.insert("build".into(), serde_json::json!({
|
||||
"id": ctx.task_id,
|
||||
"status": "batched",
|
||||
"stage": "batch",
|
||||
"duration": "0s",
|
||||
"url": "",
|
||||
}));
|
||||
}
|
||||
|
||||
data.insert("commit".into(), serde_json::json!({
|
||||
"hash": "",
|
||||
"short_hash": "",
|
||||
"message": "",
|
||||
"author": "",
|
||||
"author_email": "",
|
||||
}));
|
||||
|
||||
data
|
||||
}
|
||||
|
||||
/// Build template context for a single notification event (convenience wrapper).
|
||||
pub fn build_template_context(
|
||||
ctx: &ExecutionContext,
|
||||
event: &NotificationEvent,
|
||||
) -> HashMap<String, serde_json::Value> {
|
||||
build_batch_template_context(ctx, std::slice::from_ref(event))
|
||||
}
|
||||
|
||||
/// Render a minijinja template string with the provided data.
|
||||
pub fn render_template(
|
||||
template: &str,
|
||||
data: &HashMap<String, serde_json::Value>,
|
||||
) -> Result<String, String> {
|
||||
let env = Environment::new();
|
||||
let tmpl = env
|
||||
.template_from_str(template)
|
||||
.map_err(|e| format!("Template parse error: {}", e))?;
|
||||
|
||||
let result = tmpl
|
||||
.render(data)
|
||||
.map_err(|e| format!("Template render error: {}", e))?;
|
||||
|
||||
Ok(result)
|
||||
}
|
||||
|
||||
/// Select template string from NotificationTemplateDef based on outcome.
|
||||
pub fn select_template_str<'a>(
|
||||
tmpl: &'a NotificationTemplateDef,
|
||||
outcome: &StageOutcome,
|
||||
) -> Option<&'a str> {
|
||||
match outcome {
|
||||
StageOutcome::PipelineSuccess => tmpl.on_success.as_deref(),
|
||||
StageOutcome::PipelineFailure => tmpl.on_failure.as_deref(),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Resolve external template references from the resource registry.
|
||||
pub fn resolve_external_template_refs(
|
||||
templates: &mut HashMap<String, NotificationTemplateDef>,
|
||||
registry: &ResourceRegistry,
|
||||
) {
|
||||
for (name, tmpl_def) in templates.iter_mut() {
|
||||
if tmpl_def.use_.is_none() {
|
||||
continue;
|
||||
}
|
||||
let Some(path) = registry.resolve(name) else {
|
||||
continue;
|
||||
};
|
||||
let Ok(content) = std::fs::read_to_string(path.join("template.yml"))
|
||||
.or_else(|_| std::fs::read_to_string(path.join("template.yaml")))
|
||||
else {
|
||||
continue;
|
||||
};
|
||||
let Ok(fetched) =
|
||||
serde_yaml::from_str::<NotificationTemplateDef>(&content)
|
||||
else {
|
||||
continue;
|
||||
};
|
||||
*tmpl_def = fetched.apply(std::mem::take(tmpl_def));
|
||||
tmpl_def.use_ = None;
|
||||
}
|
||||
}
|
||||
|
||||
/// Resolve a schema reference into a NotificationTemplateDef by walking a fallback chain.
|
||||
pub fn resolve_template(
|
||||
schema: &Option<String>,
|
||||
method_name: &str,
|
||||
templates: &HashMap<String, NotificationTemplateDef>,
|
||||
_standalone: bool,
|
||||
server_templates: Option<&HashMap<String, NotificationTemplateDef>>,
|
||||
) -> Result<NotificationTemplateDef, NotifyError> {
|
||||
// 1. Try the initial schema directly
|
||||
if let Some(schema_name) = schema.as_deref() {
|
||||
if let Some(template) =
|
||||
resolve_template_by_schema(schema_name, method_name, templates, server_templates)
|
||||
{
|
||||
return Ok(template);
|
||||
}
|
||||
}
|
||||
|
||||
// 2. Walk the fallback chain
|
||||
let mut schema_next = resolve_template_next_schema(schema);
|
||||
while let Some(ref schema_name) = schema_next {
|
||||
if let Some(template) =
|
||||
resolve_template_by_schema(schema_name, method_name, templates, server_templates)
|
||||
{
|
||||
return Ok(template);
|
||||
}
|
||||
log::warn!("Failed to resolve template for schema {}.", schema_name);
|
||||
schema_next = resolve_template_next_schema(&schema_next);
|
||||
}
|
||||
Err(NotifyError::SchemaError(format!("All template resolution attempts beginning from {} failed.", schema.as_deref().unwrap_or("[system default]"))))
|
||||
}
|
||||
|
||||
/// Return the next schema name in the fallback chain, or None to terminate.
|
||||
pub fn resolve_template_next_schema(schema: &Option<String>) -> Option<String> {
|
||||
match schema.as_deref() {
|
||||
Some("custom") => None,
|
||||
Some("default") => Some("fallback".to_string()),
|
||||
Some("fallback") => None,
|
||||
Some(_) => Some("default".to_string()),
|
||||
None => Some("default".to_string()),
|
||||
}
|
||||
}
|
||||
|
||||
/// Perform a single lookup for a given schema name. Returns None if not found.
|
||||
pub fn resolve_template_by_schema(
|
||||
schema: &str,
|
||||
method_name: &str,
|
||||
templates: &HashMap<String, NotificationTemplateDef>,
|
||||
server_templates: Option<&HashMap<String, NotificationTemplateDef>>,
|
||||
) -> Option<NotificationTemplateDef> {
|
||||
match schema {
|
||||
"custom" => Some(NotificationTemplateDef::default()),
|
||||
"fallback" => templates.get(&format!("{}.fallback", method_name)).cloned(),
|
||||
"default" => server_templates.and_then(|b| b.get(method_name)).cloned(),
|
||||
name => templates.get(name).cloned(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Flatten all local template cross-references (use: "other-name").
|
||||
pub fn flatten_template_refs(
|
||||
templates: &mut HashMap<String, NotificationTemplateDef>,
|
||||
) -> Result<(), NotifyError> {
|
||||
let names: Vec<String> = templates.keys().cloned().collect();
|
||||
for name in names {
|
||||
let mut visited = std::collections::HashSet::new();
|
||||
resolve_chain(&name, templates, 0, &mut visited)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn resolve_chain(
|
||||
name: &str,
|
||||
templates: &mut HashMap<String, NotificationTemplateDef>,
|
||||
depth: u32,
|
||||
visited: &mut std::collections::HashSet<String>,
|
||||
) -> Result<(), NotifyError> {
|
||||
if depth > TEMPLATE_REFERENCE_DEPTH {
|
||||
return Err(NotifyError::TemplateDepthExceeded {
|
||||
name: name.to_string(),
|
||||
max: TEMPLATE_REFERENCE_DEPTH,
|
||||
});
|
||||
}
|
||||
if !visited.insert(name.to_string()) {
|
||||
return Err(NotifyError::TemplateLoop(name.to_string()));
|
||||
}
|
||||
|
||||
// Only process local references (no ://)
|
||||
if let Some(ref_name) = templates.get(name).and_then(|t| t.use_.clone()) {
|
||||
if !ref_name.contains("://") {
|
||||
if !templates.contains_key(&ref_name) {
|
||||
return Err(NotifyError::SchemaError(format!(
|
||||
"template '{}' references non-existent template '{}'",
|
||||
name, ref_name
|
||||
)));
|
||||
}
|
||||
resolve_chain(&ref_name, templates, depth + 1, visited)?;
|
||||
|
||||
// Merge: base provides defaults, referencing template overrides
|
||||
let base = templates[&ref_name].clone();
|
||||
let template = templates.get_mut(name).unwrap();
|
||||
*template = std::mem::take(template).apply(base);
|
||||
template.use_ = None;
|
||||
}
|
||||
}
|
||||
|
||||
visited.remove(name);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_render_template_simple() {
|
||||
let template = "Hello {{ name }}!";
|
||||
let mut data = HashMap::new();
|
||||
data.insert(
|
||||
"name".to_string(),
|
||||
serde_json::Value::String("World".to_string()),
|
||||
);
|
||||
let result = render_template(template, &data).unwrap();
|
||||
assert_eq!(result, "Hello World!");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_render_template_with_build_status() {
|
||||
let template = "Build {{ build.status }} for pipeline {{ pipeline.name }}";
|
||||
let mut data = HashMap::new();
|
||||
data.insert(
|
||||
"build".to_string(),
|
||||
serde_json::json!({ "status": "success" }),
|
||||
);
|
||||
data.insert(
|
||||
"pipeline".to_string(),
|
||||
serde_json::json!({ "name": "test-pipeline" }),
|
||||
);
|
||||
let result = render_template(template, &data).unwrap();
|
||||
assert_eq!(result, "Build success for pipeline test-pipeline");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_render_template_invalid_syntax() {
|
||||
let template = "{{ Unterminated";
|
||||
let data = HashMap::new();
|
||||
let result = render_template(template, &data);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
}
|
||||
@@ -7,7 +7,6 @@ use regex::Regex;
|
||||
use serde::Deserialize;
|
||||
use serde::Serialize;
|
||||
use std::collections::HashMap;
|
||||
use std::collections::HashSet;
|
||||
use std::fmt::Display;
|
||||
use std::hash::Hash;
|
||||
use std::time::Duration;
|
||||
@@ -78,12 +77,39 @@ impl ParsedTrigger {
|
||||
|
||||
impl NotificationRule {
|
||||
pub fn accept(&self, result: &StageOutcome, stage: &StagePhase, substage: &str) -> bool {
|
||||
if self.trigger.is_empty() {
|
||||
eprintln!("[ACCEPT] trigger empty, returning true");
|
||||
return true;
|
||||
}
|
||||
self.trigger
|
||||
.iter()
|
||||
.any(|t| t.accept(result, stage, substage))
|
||||
}
|
||||
}
|
||||
|
||||
/// Status of a single notification method after plugin execution.
|
||||
#[derive(Debug)]
|
||||
pub enum MethodStatus {
|
||||
Success,
|
||||
TemporaryError(String),
|
||||
PermanentError(String),
|
||||
Exhausted(String),
|
||||
}
|
||||
|
||||
/// Tracks which notification methods still need attention within a priority group.
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub struct NotificationTarget {
|
||||
pub priority: u32,
|
||||
pub rules: Option<Vec<String>>,
|
||||
pub count: u32,
|
||||
}
|
||||
|
||||
impl Default for NotificationTarget {
|
||||
fn default() -> Self {
|
||||
Self { priority: 0, rules: None, count: 0 }
|
||||
}
|
||||
}
|
||||
|
||||
pub type Priority = u32;
|
||||
pub type Method = String;
|
||||
pub type NotificationRulesMap = HashMap<Priority, HashMap<Method, Vec<NotificationRule>>>;
|
||||
@@ -101,9 +127,9 @@ pub struct NotificationEvent {
|
||||
pub not_before: DateTime<Utc>,
|
||||
pub priority: u32,
|
||||
pub effective_priority: u32,
|
||||
pub lives: u32,
|
||||
pub max_lives: u32,
|
||||
pub target: HashSet<String>, // Target audience
|
||||
pub retry_backoff: f64, // Exponential backoff factor
|
||||
pub attempted: u32, // Cumulative cross-group retry count
|
||||
pub target: NotificationTarget, // Current audience
|
||||
}
|
||||
|
||||
impl Hash for NotificationEvent {
|
||||
@@ -137,25 +163,11 @@ impl Ord for NotificationEvent {
|
||||
}
|
||||
}
|
||||
|
||||
/// Delay before retrying an unavailable plugin (5 seconds)
|
||||
pub const NOTIFICATION_NOTREADY_DELAY: std::time::Duration = std::time::Duration::from_secs(5);
|
||||
use std::sync::LazyLock;
|
||||
use workshop_baker_params::{NotificationParams, MergeParams};
|
||||
|
||||
/// Exponential backoff base (2.0 = double each retry)
|
||||
pub const NOTIFICATION_TEMPORARY_DELAY_FACTOR: f64 = 2.0;
|
||||
|
||||
/// Initial backoff delay for temporary errors (5 seconds)
|
||||
pub const NOTIFICATION_TEMPORARY_DELAY_TIME: f64 = 5.0;
|
||||
|
||||
/// Upper bound for backoff delay (unbounded)
|
||||
pub const NOTIFICATION_TEMPORARY_MAX_TIME: f64 = f64::MAX;
|
||||
|
||||
/// Lives recovered after downgrade as fraction of max_lives (0.0 = none)
|
||||
pub const NOTIFICATION_LIFE_RECOVERY_FACTOR: f64 = 0.0;
|
||||
|
||||
pub const NOTIFICATION_DEFAULT_FALLIBLE: bool = false;
|
||||
pub const NOTIFICATION_DEFAULT_MAX_LIVES: u32 = 3;
|
||||
pub const NOTIFICATION_DEFAULT_TIMEOUT_MS: u64 = 30000;
|
||||
pub const NOTIFICATION_DEFAULT_RETRY_BACKOFF: f64 = 2.0;
|
||||
/// Default parameters used when no external config is provided.
|
||||
pub static DEFAULT_NOTIFY_PARAMS: LazyLock<NotificationParams> = LazyLock::new(|| NotificationParams::defaults());
|
||||
|
||||
#[derive(Default, Debug, Deserialize, Clone)]
|
||||
pub enum NotificationRenderer {
|
||||
@@ -174,3 +186,25 @@ impl Display for NotificationRenderer {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Notification batching configuration.
|
||||
/// Controls how notification events are grouped before dispatch.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct BatchConfig {
|
||||
/// Whether batching is enabled
|
||||
pub enabled: bool,
|
||||
/// Group period in seconds (the maximum wait time for a batch)
|
||||
pub period: u32,
|
||||
/// Group watermark threshold (the minimum number of events before batching)
|
||||
pub watermark: u32,
|
||||
}
|
||||
|
||||
impl Default for BatchConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
enabled: true,
|
||||
period: 30,
|
||||
watermark: 5,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
use crate::finalize::FinalizeConfig;
|
||||
use crate::finalize::error::PluginError;
|
||||
use crate::notify::error::NotifyError;
|
||||
use crate::notify::types::{DEFAULT_NOTIFY_PARAMS, NotificationEvent, NotificationRule, NotificationRulesMap};
|
||||
use std::collections::HashSet;
|
||||
|
||||
pub fn validate(finalize: &FinalizeConfig) -> anyhow::Result<()> {
|
||||
let validate_result = finalize.validate();
|
||||
if let Err(errors) = validate_result {
|
||||
for error in &errors {
|
||||
log::error!("Error: {}", error);
|
||||
}
|
||||
anyhow::bail!(NotifyError::FinalizeValidationError(errors))
|
||||
};
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn extract_plugin(finalize: &FinalizeConfig) -> HashSet<String> {
|
||||
finalize
|
||||
.notification
|
||||
.groups
|
||||
.values()
|
||||
.flat_map(|methods| methods.keys())
|
||||
.cloned()
|
||||
.collect()
|
||||
}
|
||||
|
||||
pub fn is_temporary_error(e: &PluginError) -> bool {
|
||||
!matches!(e, PluginError::GeneralError(_))
|
||||
}
|
||||
|
||||
// calculate min(1, k/x)*t as notification grouping period
|
||||
// to avoid excessive notification frequency
|
||||
pub fn wait_time(group_period: u32, group_watermark: u32, count: u32) -> f32 {
|
||||
if count == 0 {
|
||||
log::warn!("Unexpected notification count: {}", count);
|
||||
return group_period as f32;
|
||||
}
|
||||
f32::min(1.0, group_watermark as f32 / count as f32) * group_period as f32
|
||||
}
|
||||
|
||||
pub fn notify_delay(event: &NotificationEvent) -> std::time::Duration {
|
||||
let exponent = event.target.count as i32;
|
||||
if exponent < 0 {
|
||||
log::warn!("Unexpected count: {}", event.target.count);
|
||||
return std::time::Duration::from_secs_f64(DEFAULT_NOTIFY_PARAMS.max_retry_delay_seconds.value);
|
||||
}
|
||||
let factor = if event.retry_backoff > 0.0 {
|
||||
event.retry_backoff
|
||||
} else {
|
||||
DEFAULT_NOTIFY_PARAMS.retry_backoff.value
|
||||
};
|
||||
let multiplier = factor.powi(exponent);
|
||||
let delay = multiplier * DEFAULT_NOTIFY_PARAMS.retry_delay_seconds.value;
|
||||
if !delay.is_finite() || delay < 0.0 {
|
||||
return std::time::Duration::from_secs_f64(DEFAULT_NOTIFY_PARAMS.max_retry_delay_seconds.value);
|
||||
}
|
||||
std::time::Duration::from_secs_f64(delay)
|
||||
}
|
||||
|
||||
pub fn find_rule<'a>(
|
||||
config: &'a NotificationRulesMap,
|
||||
method: &str,
|
||||
rule_id: usize,
|
||||
) -> Result<&'a NotificationRule, NotifyError> {
|
||||
for ruleset in config.values() {
|
||||
if let Some(rules) = ruleset.get(method) {
|
||||
if let Some(rule) = rules.get(rule_id) {
|
||||
return Ok(rule);
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(NotifyError::SchemaError("rule not found".into()))
|
||||
}
|
||||
|
||||
pub fn get_fallible(config: &NotificationRulesMap, method: &str) -> bool {
|
||||
for ruleset in config.values() {
|
||||
if let Some(rules) = ruleset.get(method) {
|
||||
return rules.iter().all(|r| r.fallible);
|
||||
}
|
||||
}
|
||||
false
|
||||
}
|
||||
@@ -30,13 +30,15 @@ pub mod stage;
|
||||
pub mod types;
|
||||
|
||||
use crate::{
|
||||
ExecutionContext,
|
||||
cli::Cli,
|
||||
prebake::{error::PrebakeError, security::get_drop_after, stage::PrebakeStage},
|
||||
};
|
||||
use crate::notify::types::NotificationEventSender;
|
||||
use crate::types::buildstatus::{StageOutcome, StagePhase};
|
||||
use uuid::Uuid;
|
||||
use chrono::Utc;
|
||||
use std::path::Path;
|
||||
use workshop_engine::EventSender;
|
||||
use workshop_engine::{EventSender, ExecutionContext};
|
||||
|
||||
pub use config::PrebakeConfig;
|
||||
pub use security::prebake_drop_privilege;
|
||||
@@ -93,6 +95,7 @@ pub async fn prebake(
|
||||
stage: Option<PrebakeStage>,
|
||||
ctx: &mut ExecutionContext,
|
||||
event_tx: EventSender,
|
||||
nevent_tx: Option<NotificationEventSender>,
|
||||
) -> anyhow::Result<()> {
|
||||
// Initialize
|
||||
let prebake_content = std::fs::read_to_string(prebake_path).inspect_err(|e| {
|
||||
@@ -132,6 +135,7 @@ pub async fn prebake(
|
||||
let result = stage::bootstrap::bootstrap(&prebake, &osinfo, ctx, &event_tx).await;
|
||||
result?;
|
||||
}
|
||||
send_stage_event(&nevent_tx, StagePhase::Prebake, "bootstrap", StageOutcome::Success).await;
|
||||
|
||||
// EarlyHook stage
|
||||
if stage <= PrebakeStage::EarlyHook {
|
||||
@@ -146,6 +150,7 @@ pub async fn prebake(
|
||||
let result = stage::hook::hook(earlyhook, ctx, event_tx.clone()).await;
|
||||
result?;
|
||||
}
|
||||
send_stage_event(&nevent_tx, StagePhase::Prebake, "earlyhook", StageOutcome::Success).await;
|
||||
|
||||
if let Some(dependencies) = prebake.dependencies {
|
||||
let endpoint = dependencies
|
||||
@@ -154,8 +159,10 @@ pub async fn prebake(
|
||||
.map(|c| c.repology_endpoint)
|
||||
.unwrap_or(workshop_engine::RepologyEndpoint::Default);
|
||||
|
||||
let osinfo = os_info::get();
|
||||
let detected_pm = workshop_engine::pm::detect(&osinfo).map(|pm| pm.name().to_string());
|
||||
let pm_name = dependencies
|
||||
.system
|
||||
.as_ref()
|
||||
.and_then(|sys| sys.keys().next().cloned());
|
||||
|
||||
let upm_sys_deps = if let Some(user) = &dependencies.user {
|
||||
if stage <= PrebakeStage::DepsSystem {
|
||||
@@ -170,7 +177,7 @@ pub async fn prebake(
|
||||
let mut system_config = dependencies.system.clone().unwrap_or_default();
|
||||
|
||||
if !upm_sys_deps.deps.is_empty() {
|
||||
if let Some(pm_name) = &detected_pm {
|
||||
if let Some(ref pm_name) = pm_name {
|
||||
for (upm_name, sys_deps) in &upm_sys_deps.deps {
|
||||
let resolved = workshop_engine::repology::resolve_package_names(
|
||||
&sys_deps.packages,
|
||||
@@ -214,6 +221,7 @@ pub async fn prebake(
|
||||
let result = stage::depssystem::depssystem(&system_config, ctx, event_tx.clone()).await;
|
||||
result?;
|
||||
}
|
||||
send_stage_event(&nevent_tx, StagePhase::Prebake, "depssystem", StageOutcome::Success).await;
|
||||
|
||||
if let Some(user) = dependencies.user
|
||||
&& stage <= PrebakeStage::DepsUser
|
||||
@@ -226,6 +234,7 @@ pub async fn prebake(
|
||||
result?;
|
||||
}
|
||||
}
|
||||
send_stage_event(&nevent_tx, StagePhase::Prebake, "depsuser", StageOutcome::Success).await;
|
||||
|
||||
// LateHook stage
|
||||
if stage <= PrebakeStage::LateHook {
|
||||
@@ -239,11 +248,37 @@ pub async fn prebake(
|
||||
};
|
||||
let result = stage::hook::hook(latehook, ctx, event_tx.clone()).await;
|
||||
result?;
|
||||
send_stage_event(&nevent_tx, StagePhase::Prebake, "latehook", StageOutcome::Success).await;
|
||||
}
|
||||
|
||||
// Ready stage
|
||||
ctx.task_id = format!("{}-{}-ready", cli.pipeline, cli.build_id);
|
||||
send_stage_event(&nevent_tx, StagePhase::Prebake, "ready", StageOutcome::Success).await;
|
||||
|
||||
log::info!("Prebake done!");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Send a stage completion event to the notification queue.
|
||||
async fn send_stage_event(
|
||||
nevent_tx: &Option<NotificationEventSender>,
|
||||
stage: StagePhase,
|
||||
substage: &str,
|
||||
outcome: StageOutcome,
|
||||
) {
|
||||
if let Some(tx) = nevent_tx {
|
||||
let event = crate::notify::types::NotificationEvent {
|
||||
id: Uuid::new_v4(),
|
||||
stage,
|
||||
substage: substage.to_string(),
|
||||
outcome,
|
||||
not_before: chrono::Utc::now(),
|
||||
priority: 0,
|
||||
effective_priority: 0,
|
||||
retry_backoff: 1.0,
|
||||
attempted: 0,
|
||||
target: Default::default(),
|
||||
};
|
||||
let _ = tx.send(event).await;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -10,7 +10,7 @@ Build environment setup: bootstrap, dependencies, security, hooks.
|
||||
```
|
||||
prebake/
|
||||
├── config.rs # PrebakeConfig YAML schema (environment, deps, cache)
|
||||
├── stage.rs # PrebakeStage enum (Bootstrap→Ready ordering)
|
||||
├── stage.rs # PrebakeStage enum (Bootstrap→EarlyHook→DepsSystem→DepsUser→LateHook→Ready)
|
||||
├── stage/
|
||||
│ ├── bootstrap.rs # User creation (vulcan), sudo/doas setup
|
||||
│ ├── environment.rs # Env var injection
|
||||
@@ -24,7 +24,12 @@ prebake/
|
||||
│ ├── baremetal.rs # Direct host execution
|
||||
│ └── custom.rs # Custom env (todo!())
|
||||
├── security.rs # Privilege dropping (unsafe libc calls)
|
||||
└── event.rs # Event logging
|
||||
├── event.rs # Event logging
|
||||
├── error.rs # PrebakeError + BootstrapError
|
||||
├── types.rs # Module re-exports
|
||||
├── types/ # memsize, cache, builderconfig, architecture
|
||||
├── constant.rs # Module constants
|
||||
└── env.rs # Environment module root
|
||||
```
|
||||
|
||||
## WHERE TO LOOK
|
||||
@@ -38,13 +43,11 @@ prebake/
|
||||
| Hooks | `stage/hook.rs` - pre/post stage hooks |
|
||||
|
||||
## CONVENTIONS
|
||||
|
||||
- **PrebakeStage**: Ordered enum (Bootstrap→DepsSystem→DepsUser→Environment→Hooks→Ready)
|
||||
- **PrebakeStage**: Ordered enum Bootstrap→EarlyHook→DepsSystem→DepsUser→LateHook→Ready
|
||||
- **ExecutionContext**: Carries task_id, username, env_vars through all stages
|
||||
- **Target User**: "vulcan" (hardcoded in bootstrap.rs:6-24)
|
||||
|
||||
## ANTI-PATTERNS
|
||||
|
||||
1. **Hardcoded user** — "vulcan" username hardcoded throughout
|
||||
2. **Unsafe blocks** — `security.rs` has 5+ unsafe libc calls
|
||||
3. **Incomplete** — custom.rs and apt.rs have `todo!()` stubs
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user