refactor: update AGENTS.md and remove osbolete modules
Archive 7 deprecated crates. Add Resource model documentation. Remove internal fetch module from workshop-baker.
This commit is contained in:
@@ -1,83 +1,72 @@
|
|||||||
# PROJECT KNOWLEDGE BASE
|
# PROJECT KNOWLEDGE BASE
|
||||||
|
|
||||||
**Generated:** 2026-04-22
|
**Generated:** 2026-05-19
|
||||||
**Commit:** 7b3b71a
|
**Commit:** 28abc5d
|
||||||
**Branch:** master
|
**Branch:** master
|
||||||
|
|
||||||
## OVERVIEW
|
## OVERVIEW
|
||||||
HoneyBiscuitWorkshop is a Rust-based CI/CD system ("蜜饼工坊") with LLM-powered pipeline auto-configuration. 9 independent Cargo crates, no workspace.
|
HoneyBiscuitWorkshop ("蜜饼工坊") — Rust CI/CD system with decorator-driven shell pipeline DSL and LLM-powered configuration. 3 independent Cargo crates, no workspace.
|
||||||
|
|
||||||
## STRUCTURE
|
## STRUCTURE
|
||||||
```
|
```
|
||||||
./
|
./
|
||||||
├── workshop-baker/ # Main orchestrator (CLI + daemon)
|
├── workshop-baker/ # Main orchestrator (CLI + pipeline stages)
|
||||||
├── workshop-agent/ # HTTP agent server
|
├── workshop-engine/ # Execution engine (extracted from baker)
|
||||||
├── workshop-pipeline/ # Config library (prebake/finalize)
|
├── workshop-getterurl/ # URL fetching (go-getter style)
|
||||||
├── workshop-llm-detector/ # LLM repo analyzer
|
|
||||||
├── workshop-vault/ # Secret management client
|
|
||||||
├── workshop-cert/ # TLS certificate generator
|
|
||||||
├── workshop-deviceid/ # Device ID library
|
|
||||||
├── workshop-builder-native/ # Stub (unused)
|
|
||||||
├── workshop-helper-mac/ # MAC address utility
|
|
||||||
├── docs/ # mdBook documentation (zh-CN)
|
├── docs/ # mdBook documentation (zh-CN)
|
||||||
├── playground/ # Experimentation
|
├── templates/ # Pipeline templates (bake.sh, finalize.yml, prebake.yml)
|
||||||
└── third_party/ # Empty
|
└── archived/ # Deprecated crates (7 moved here)
|
||||||
```
|
```
|
||||||
|
|
||||||
## WHERE TO LOOK
|
## WHERE TO LOOK
|
||||||
| Task | Location | Notes |
|
| Task | Location | Notes |
|
||||||
|------|----------|-------|
|
|------|----------|-------|
|
||||||
| Baker (main) | `workshop-baker/src/{bake,prebake,engine}/` | Core pipeline logic |
|
| Baker (CLI) | `workshop-baker/src/{lib,main}.rs` | Clap CLI + tokio entry |
|
||||||
| Agent server | `workshop-agent/src/` | Actix-web HTTP + TLS |
|
| Pipeline stages | `workshop-baker/src/{bake,prebake,finalize}/` | Stage orchestration |
|
||||||
| Pipeline config | `workshop-pipeline/src/config/` | prebake/finalize YAML |
|
| Notifications | `workshop-baker/src/notify/` | Plugin-based event notifications |
|
||||||
| LLM integration | `workshop-llm-detector/src/llm/` | Ollama/OpenAI clients |
|
|| Engine (execution) | `workshop-engine/src/` | Process mgmt, PM detection |
|
||||||
| Secrets | `workshop-vault/src/` | Vault client wrapper |
|
| URL fetching | `workshop-getterurl/src/` | Git/HTTP/File resource getter |
|
||||||
|
|
||||||
## CODE MAP
|
## CODE MAP
|
||||||
| Symbol | Type | Location | Role |
|
| Symbol | Type | Location | Role |
|
||||||
|--------|------|----------|------|
|
|--------|------|----------|------|
|
||||||
| PipelineConfig | struct | workshop-pipeline/src/lib.rs:27 | Full pipeline config |
|
| Cli | struct | workshop-baker/src/lib.rs:21 | CLI argument struct |
|
||||||
| CICDLLMHelper | struct | workshop-llm-detector/src/lib.rs:16 | LLM analysis entry |
|
| ExecutionContext | struct | workshop-engine/src/types.rs:10 | Pipeline context |
|
||||||
| Engine | struct | workshop-baker/src/engine.rs | Build executor |
|
| Engine | struct | workshop-engine/src/types.rs:99 | Build executor (placeholder) |
|
||||||
| VaultClient | struct | workshop-vault/src/client.rs | Secret ops |
|
| PrebakeConfig | struct | workshop-baker/src/prebake/config.rs | Prebake YAML schema |
|
||||||
|
| FinalizeConfig | struct | workshop-baker/src/finalize/config.rs:21 | Finalize YAML schema |
|
||||||
|
| GetterUrl | struct | workshop-getterurl/src/lib.rs:84 | Parsed resource URL |
|
||||||
|
| Getter | trait | workshop-getterurl/src/getter.rs:18 | Resource fetch trait |
|
||||||
|
|
||||||
## CONVENTIONS
|
## CONVENTIONS
|
||||||
|
|
||||||
### Rust
|
### Rust
|
||||||
- **Edition**: 2024 (non-standard, requires Rust 1.85+)
|
- **Edition**: 2024 (requires Rust 1.85+), deprecates `mod.rs`
|
||||||
- Deprecate `mod.rs`
|
|
||||||
- **Naming**: `snake_case` files/modules, `PascalCase` types, `SCREAMING_SNAKE_CASE` consts
|
- **Naming**: `snake_case` files/modules, `PascalCase` types, `SCREAMING_SNAKE_CASE` consts
|
||||||
- **Error handling**: `thiserror` + `anyhow` combo
|
- **Error handling**: `thiserror` + `anyhow` combo; `HasExitCode` trait for exit codes
|
||||||
- **Async**: `#[tokio::main]` + `tokio` with "full" features
|
- **Async**: `#[tokio::main]` + `tokio` with "full" features
|
||||||
- **Imports**: `std` → `external_crate` → `crate::module`
|
- **Imports**: `std` → `external_crate` → `crate::module`
|
||||||
|
- **Dual-mode**: CLI commands (prebake/bake/finalize) + daemon mode (unimplemented)
|
||||||
|
|
||||||
### Testing
|
### Testing
|
||||||
- **Rust**: `#[test]` inline, `#[tokio::test]` async, `tests/*.rs` integration
|
- **Rust**: `#[test]` inline, `#[tokio::test]` async, `tests/` integration
|
||||||
- **Python**: `pytest` with `tests/integration/test_*.py` (Docker-based)
|
- **Benchmarks**: Criterion (configured in Cargo.toml)
|
||||||
- **Benchmarks**: Criterion with `harness = false`
|
|
||||||
|
|
||||||
### Build Pipeline
|
### CI
|
||||||
- **prebake.yml**: Environment setup (docker/firecracker/baremetal)
|
- **GitHub Actions**: rust-toolchain (stable), clippy + rustfmt, `cargo test --all-features`
|
||||||
- **bake.sh**: Build execution with `# @pipeline` decorators
|
- **Matrix**: workshop-baker, workshop-engine
|
||||||
- **finalize.yml**: Packaging, deployment, notifications
|
|
||||||
- **Annotations**: `@timeout()`, `@retry()`, `@parallel`, `@fallible`
|
|
||||||
|
|
||||||
## ANTI-PATTERNS (THIS PROJECT)
|
## ANTI-PATTERNS (THIS PROJECT)
|
||||||
|
1. **Hardcoded user** — "vulcan" username hardcoded throughout prebake
|
||||||
1. **Python in Rust project** — workshop-baker has Python tests (pytest.ini, test_prebake.py)
|
2. **Unsafe libc** — workshop-baker/src/prebake/security.rs has 5+ unsafe blocks
|
||||||
2. **temp/ directory** — Non-standard in Rust projects
|
3. **Rust 2024 edition** — May not work with stable toolchains
|
||||||
3. **Certificates in source** — artifact workshop-cert/certs/ not excluded
|
4. **Empty daemon** — `daemon/` dir is empty, Daemon command is `unimplemented!()`
|
||||||
4. **Deprecated projects** — workshop-executor.old, workshop-monitor.old still present
|
|
||||||
5. **Hardcoded mirrors** — Tsinghua University mirrors in configs
|
|
||||||
6. **Rust 2024 edition** — May not work with stable toolchains
|
|
||||||
|
|
||||||
## UNIQUE STYLES
|
## UNIQUE STYLES
|
||||||
|
|
||||||
- **Pipeline DSL**: Shell scripts with `# @decorator` annotations
|
- **Pipeline DSL**: Shell scripts with `# @decorator` annotations
|
||||||
- **Dual-mode baker**: CLI commands + daemon mode
|
- **Notification system**: Plugin-based with batching, retry, priority groups
|
||||||
- **LLM cookbook system**: YAML cookbooks for language-specific builds
|
- **LLM cookbook system**: YAML cookbooks for language-specific builds (archived)
|
||||||
- **Chinese docs** — Documentation primarily in zh-CN
|
- **Chinese docs** — Documentation primarily in zh-CN
|
||||||
- **User "vulcan"** — Custom build user (not root/runner)
|
|
||||||
|
|
||||||
## COMMANDS
|
## COMMANDS
|
||||||
```bash
|
```bash
|
||||||
@@ -86,23 +75,18 @@ cargo build --release -p workshop-baker
|
|||||||
|
|
||||||
# Test
|
# Test
|
||||||
cargo test -p workshop-baker
|
cargo test -p workshop-baker
|
||||||
pytest tests/integration/test_prebake.py -v
|
cargo test -p workshop-engine
|
||||||
|
|
||||||
# Lint
|
# Lint
|
||||||
cargo clippy -- -D warnings
|
cargo clippy -- -D warnings
|
||||||
cargo fmt --all
|
cargo fmt --all
|
||||||
|
|
||||||
# Run baker
|
# Run
|
||||||
cargo run --bin workshop-baker -- prebake config.yml
|
cargo run --bin workshop-baker -- --pipeline demo --build-id 1 --prebake prebake.yml --bake bake.sh --finalize finalize.yml
|
||||||
cargo run --bin workshop-baker -- bake script.sh
|
|
||||||
|
|
||||||
# Run agent
|
|
||||||
cargo run --bin workshop-agent
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## NOTES
|
## NOTES
|
||||||
|
|
||||||
- **Docker required** for integration tests
|
- **Docker required** for integration tests
|
||||||
- **workshop-builder-native** is a stub (3-line Hello World)
|
- **No root workspace** — each crate independent, build separately
|
||||||
- No root workspace Cargo.toml — each crate is independent
|
- **7 archived crates** — workshop-agent, workshop-cert, workshop-deviceid, workshop-helper-mac, workshop-llm-detector, workshop-pipeline, workshop-vault
|
||||||
- `AGENTS.md.old` at root — legacy documentation
|
- **Python tests removed** — pytest.ini exists but test files no longer in workspace
|
||||||
|
|||||||
@@ -35,3 +35,4 @@
|
|||||||
- [自定义 Decorator](zh-CN/vol3_dev/custom_decorator.md)
|
- [自定义 Decorator](zh-CN/vol3_dev/custom_decorator.md)
|
||||||
- [模板变量](zh-CN/vol3_dev/template_variables.md)
|
- [模板变量](zh-CN/vol3_dev/template_variables.md)
|
||||||
- [邮件模板](zh-CN/vol3_dev/mail_template.md)
|
- [邮件模板](zh-CN/vol3_dev/mail_template.md)
|
||||||
|
- [Resource 模型](zh-CN/vol3_dev/resource_model.md)
|
||||||
|
|||||||
@@ -0,0 +1,88 @@
|
|||||||
|
# Resource 模型
|
||||||
|
|
||||||
|
## 核心思想
|
||||||
|
|
||||||
|
整个系统只关心两件事:东西从哪来(fetch),东西到哪去(publish)。
|
||||||
|
|
||||||
|
fetch 方向:URL → 本地路径 → 使用。
|
||||||
|
publish 方向:本地路径 → 变换(插件)→ URL。
|
||||||
|
|
||||||
|
两个方向共用同一套寻址方式,但 Resource 中间态只存在于 fetch 方向。
|
||||||
|
|
||||||
|
## URL 格式
|
||||||
|
|
||||||
|
```
|
||||||
|
getter::url?param1=value1¶m2=value2
|
||||||
|
```
|
||||||
|
|
||||||
|
getter 显式声明意图。不是"我猜这是个 git 仓库",而是"使用者告诉我这是什么"。
|
||||||
|
|
||||||
|
只有三个 getter:
|
||||||
|
|
||||||
|
- `git` — 克隆仓库,必要时 checkout 指定版本
|
||||||
|
- `http` / `https` — 下载文件,必要时校验 checksum
|
||||||
|
- `file` — 复制本地文件或目录
|
||||||
|
|
||||||
|
版本锁定不用 `@v1.0`(`@` 在 URL 里是认证分隔符),用 `?ref=v1.0`。
|
||||||
|
|
||||||
|
没有 `github.com/user/repo` 看起来应该 clone 它这种黑魔法。
|
||||||
|
没有 `.tar.gz` 看起来应该解压它的惊喜。
|
||||||
|
URL 是 URL,行为是行为,两者用 getter:: 显式绑定。
|
||||||
|
|
||||||
|
## Resource 不是网络地址
|
||||||
|
|
||||||
|
流水线阶段(prebake / finalize / notify)只接触 Resource:一个名字 + 一个本地路径,已就绪。它们不经由网络、不解析 URL、不碰 getter。
|
||||||
|
|
||||||
|
所有 fetch 行为在阶段启动前完成:
|
||||||
|
|
||||||
|
- worker 模式:bakerd fetch → 填入 ResourceRegistry → 启动 baker
|
||||||
|
- standalone 模式:baker 自己 fetch → 填入 ResourceRegistry → 执行阶段
|
||||||
|
- bare 模式:不 fetch,传入的必须是 Resource
|
||||||
|
|
||||||
|
ResourceRegistry 是名字到路径的映射。填进去就只读了。
|
||||||
|
|
||||||
|
## getterurl 是独立的 crate
|
||||||
|
|
||||||
|
URL 格式解析 + 基本 fetch 放在 workshop-getterurl,和 baker 本身没有耦合。任何人都可以用。
|
||||||
|
|
||||||
|
crate 分两层:
|
||||||
|
|
||||||
|
```
|
||||||
|
结构层(parser):
|
||||||
|
"git::https://host/repo.git?ref=v1.0" → { getter: "git", url: Url, params: Params }
|
||||||
|
|
||||||
|
获取层(builtin getters):
|
||||||
|
getter.fetch(url, params, dst) → local_path
|
||||||
|
git: git clone + optional checkout
|
||||||
|
http: HTTP GET + optional checksum
|
||||||
|
file: std::fs::copy
|
||||||
|
```
|
||||||
|
|
||||||
|
获取层是字面意义的——不解压、不推断、不猜意图。
|
||||||
|
|
||||||
|
## Checksum
|
||||||
|
|
||||||
|
checksum 作为 URL query 参数传递。三态语义:
|
||||||
|
|
||||||
|
- `?checksum=sha256:abc` — 必须匹配
|
||||||
|
- `?checksum=` — 显式禁用校验
|
||||||
|
- 无 checksum — 如果服务端返回 Content-Digest 头,校验它
|
||||||
|
|
||||||
|
getter 不主动校验,它只执行 URL 里指定的操作。
|
||||||
|
|
||||||
|
## Artifact 不经过 Resource
|
||||||
|
|
||||||
|
artifact 是 bake 的产物,自产生就开始被使用,不存在中间态。它的路径就是 bake 产出的本地路径,不走 ResourceRegistry。
|
||||||
|
|
||||||
|
artifact 的 publish 方向由插件定义行为——可能上传、可能扫描、可能只记录。插件定义最终的行为,只要是自洽的。
|
||||||
|
|
||||||
|
## 三种运行模式
|
||||||
|
|
||||||
|
```
|
||||||
|
fetch 执行者 fetch 在阶段内
|
||||||
|
daemon 模式 bakerd 否
|
||||||
|
standalone baker 自身 否(阶段启动前已完成)
|
||||||
|
bare 模式 无 否(必须传入 Resource)
|
||||||
|
```
|
||||||
|
|
||||||
|
bare 模式下传入的必须是已经就绪的 Resource。外部 URL 是 daemon 或 standalone 的事,bare 不负责。
|
||||||
Generated
+1
-1471
File diff suppressed because it is too large
Load Diff
@@ -47,12 +47,7 @@ uuid = { version = "1.19.0", features = ["v4"] }
|
|||||||
libc = "0.2"
|
libc = "0.2"
|
||||||
os_info = "3.14.0"
|
os_info = "3.14.0"
|
||||||
privdrop = "0.5.6"
|
privdrop = "0.5.6"
|
||||||
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
|
|
||||||
which = "8.0.2"
|
which = "8.0.2"
|
||||||
gix = "0.81.0"
|
|
||||||
sha2 = "0.10"
|
|
||||||
flate2 = "1.0"
|
|
||||||
zstd = "0.13"
|
|
||||||
globset = "0.4.18"
|
globset = "0.4.18"
|
||||||
axum = "0.7"
|
axum = "0.7"
|
||||||
|
|
||||||
|
|||||||
@@ -1,62 +1,57 @@
|
|||||||
# workshop-baker/src
|
# workshop-baker/src
|
||||||
|
|
||||||
**Generated:** 2026-04-22
|
**Generated:** 2026-05-19
|
||||||
**Commit:** 7b3b71a
|
**Commit:** 28abc5d
|
||||||
|
|
||||||
Baker CLI + execution engine for HoneyBiscuitWorkshop pipelines.
|
CLI tool and runtime for executing CI/CD pipelines. Orchestrates prebake → bake → finalize stages with notification dispatch.
|
||||||
|
|
||||||
## OVERVIEW
|
|
||||||
|
|
||||||
CLI tool and runtime for executing CI/CD pipelines with resource isolation, privilege dropping, and package management integration.
|
|
||||||
|
|
||||||
## STRUCTURE
|
## STRUCTURE
|
||||||
|
|
||||||
```
|
```
|
||||||
src/
|
src/
|
||||||
├── bake.rs # Top-level bake orchestration
|
├── lib.rs # Module declarations + Cli struct + Commands enum
|
||||||
├── bake/ # Script parsing and building
|
├── main.rs # tokio::main entry point
|
||||||
│ ├── parser.rs # @decorator-aware shell parser
|
├── bare.rs # Bare mode (run individual stages)
|
||||||
│ ├── decorator.rs # @pipeline, @retry, @parallel, etc.
|
├── error.rs # CliError (wraps stage errors)
|
||||||
│ ├── builder.rs # Script template builder
|
├── bake.rs # Bake orchestration (script → execution)
|
||||||
│ └── schedule.rs # Function ordering via @after deps
|
├── bake/ # Script parsing, decorator DSL, scheduling
|
||||||
├── engine/ # Execution runtime
|
├── prebake.rs # Prebake orchestration (env setup)
|
||||||
│ ├── executor.rs # Script execution with cgroups
|
├── prebake/ # Config, stages, security, env providers
|
||||||
│ ├── cgroups.rs # Linux cgroup resource limits
|
├── finalize.rs # Finalize orchestration (post-build)
|
||||||
│ ├── repology.rs # Package name resolution
|
├── finalize/ # Config, plugins, hooks, templates
|
||||||
│ ├── pm.rs # Package manager detection
|
├── notify.rs # Notification system (batching, retry)
|
||||||
│ └── upm.rs # User package manager (Nix/Guix)
|
├── notify/ # Types, templates, interactive, queue
|
||||||
├── prebake/ # Build environment setup
|
├── daemon/ # Empty (Daemon command = unimplemented!)
|
||||||
│ ├── config.rs # PrebakeConfig YAML schema
|
├── monitor/ # Resource usage monitoring (cgroups)
|
||||||
│ ├── stage/ # Bootstrap, DepsSystem, DepsUser, Hooks
|
├── types.rs # Module re-exports
|
||||||
│ └── security.rs # Privilege dropping logic
|
├── types/ # buildstatus, resource types
|
||||||
├── types/ # Shared type definitions
|
├── utils.rs # Module root (fetch submodule)
|
||||||
│ ├── repology.rs # RepologyEndpoint enum
|
└── utils/ # fetch.rs
|
||||||
│ ├── builderconfig.rs
|
|
||||||
│ ├── cache.rs
|
|
||||||
│ └── memsize.rs
|
|
||||||
├── monitor/ # Resource usage monitoring
|
|
||||||
│ ├── cgroups.rs # cgroup-based usage collection
|
|
||||||
│ └── jobobject.rs # Windows job objects (stub)
|
|
||||||
├── daemon.rs # Unix socket daemon (TODO: refactor)
|
|
||||||
└── lib.rs # CLI struct + module exports
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## WHERE TO LOOK
|
## WHERE TO LOOK
|
||||||
|
|
||||||
| Task | Location |
|
| Task | Location |
|
||||||
|------|----------|
|
|------|----------|
|
||||||
| Script parsing with @decorators | `bake/parser.rs:29` - `parse_script()` |
|
| CLI args | `lib.rs:21` - `Cli` struct |
|
||||||
| Decorator types | `bake/decorator.rs` - `Decorator` enum |
|
| Entry point | `main.rs:91` - `#[tokio::main]` |
|
||||||
| Build execution | `engine/executor.rs` - `Executor::execute_script()` |
|
| Pipeline orchestration | `main.rs:68-74` - prebake→bake→finalize chain |
|
||||||
| Resource limits | `engine/types.rs:56` - `ResourceLimits` struct |
|
| Bare mode (single stage) | `bare.rs:14` - `run_bare()` |
|
||||||
| Prebake stages | `prebake/stage/` subdirectories |
|
| Script parsing | `bake/parser.rs:56` - `parse_script()` |
|
||||||
| Package detection | `engine/pm.rs:detection()` |
|
| Decorator DSL | `bake/decorator.rs` - `Decorator` enum |
|
||||||
| CLI entry | `lib.rs:26` - `cli::Cli` struct |
|
| Prebake stages | `prebake/stage/` - Bootstrap→DepsSystem→DepsUser→Hooks→Ready |
|
||||||
|
| Plugin system | `finalize/plugin.rs` + `finalize/plugin/` |
|
||||||
|
| Notification dispatch | `notify.rs:66` - `notify()` async loop |
|
||||||
|
| Notification types | `notify/types.rs` - `NotificationRule`, `ParsedTrigger` |
|
||||||
|
|
||||||
## CONVENTIONS
|
## CONVENTIONS
|
||||||
|
|
||||||
- **ExecutionContext**: Passed through all stages, carries task_id, username, env_vars, timeout
|
- **ExecutionContext**: Passed through all stages, carries task_id, username, env_vars, timeout
|
||||||
- **PrebakeStage**: Ordered enum controlling stage execution sequence (Bootstrap -> Ready)
|
- **PrebakeStage**: Ordered enum Bootstrap→EarlyHook→DepsSystem→DepsUser→LateHook→Ready
|
||||||
- **Decorator**: Attached to shell functions via `# @name(args)` comments above function declarations
|
- **Decorator**: `# @name(args)` comments above shell function declarations
|
||||||
- **Engine**: Lightweight wrapper around cgroup manager for resource isolation
|
- **Notification**: Plugin-based dispatch with batching, exponential backoff, priority groups
|
||||||
- **Dual-mode**: CLI commands (prebake, bake, finalize) vs daemon mode via Unix socket
|
- **Engine**: Delegated to `workshop-engine` crate (formerly in-tree)
|
||||||
|
- **Dual-mode**: CLI commands (standalone) vs Daemon mode (unimplemented)
|
||||||
|
|
||||||
|
## ANTI-PATTERNS
|
||||||
|
1. **Empty daemon/** directory — Daemon command is `unimplemented!()`
|
||||||
|
2. **Dead code allowed** — `lints.rust.dead_code = "allow"` in Cargo.toml
|
||||||
|
3. **Engine was extracted** — workshop-engine is its own crate now, src/engine/ removed
|
||||||
|
4. **Python test infra exists but files removed** — pytest.ini remains, test dir is empty
|
||||||
|
|||||||
@@ -12,14 +12,16 @@ bake/
|
|||||||
├── parser.rs # @decorator-aware shell script parser
|
├── parser.rs # @decorator-aware shell script parser
|
||||||
├── decorator.rs # Decorator enum + 100+ unit tests
|
├── decorator.rs # Decorator enum + 100+ unit tests
|
||||||
├── builder.rs # Script template assembly
|
├── builder.rs # Script template assembly
|
||||||
└── schedule.rs # Function ordering via @after dependencies
|
├── schedule.rs # Function ordering via @after dependencies
|
||||||
|
├── error.rs # BakeError enum
|
||||||
|
└── constant.rs # DEFAULT_WORKSPACE, TRIVIAL_SCRIPT_NAME
|
||||||
```
|
```
|
||||||
|
|
||||||
## WHERE TO LOOK
|
## WHERE TO LOOK
|
||||||
|
|
||||||
| Task | Location |
|
| Task | Location |
|
||||||
|------|----------|
|
|------|----------|
|
||||||
| Parse script | `parser.rs:29` - `parse_script()` |
|
Parse script | `parser.rs:56` - `parse_script()` |
|
||||||
| Decorator types | `decorator.rs` - `Decorator` enum variants |
|
| Decorator types | `decorator.rs` - `Decorator` enum variants |
|
||||||
| Build script | `builder.rs` - template builder |
|
| Build script | `builder.rs` - template builder |
|
||||||
| Schedule deps | `schedule.rs` - @after resolution |
|
| Schedule deps | `schedule.rs` - @after resolution |
|
||||||
|
|||||||
@@ -1,56 +1,52 @@
|
|||||||
# workshop-baker/src/finalize
|
# workshop-baker/src/finalize
|
||||||
|
|
||||||
**Generated:** 2026-04-22
|
**Generated:** 2026-05-19
|
||||||
**Commit:** 7b3b71a
|
**Commit:** 28abc5d
|
||||||
|
|
||||||
Artifact packaging, distribution, deployment, and notifications.
|
Artifact packaging, plugin-based notifications, and post-build hooks.
|
||||||
|
|
||||||
## STRUCTURE
|
## STRUCTURE
|
||||||
|
|
||||||
```
|
```
|
||||||
finalize/
|
finalize/
|
||||||
├── config.rs # FinalizeConfig YAML (artifacts, deploy, notify)
|
├── config.rs # FinalizeConfig YAML (379 lines, plugins, notifications)
|
||||||
├── stage.rs # FinalizeStage enum
|
├── stage.rs # FinalizeStage enum
|
||||||
├── stage/hook.rs # Post-build hooks
|
├── stage/hook.rs # Post-build hook execution
|
||||||
├── event.rs # Event propagation
|
├── event.rs # Event propagation
|
||||||
├── template.rs # Variable substitution {{VAR}}
|
├── template.rs # Variable substitution {{VARIABLE}}
|
||||||
├── plugin.rs # Plugin trait + registry
|
├── plugin.rs # Plugin trait + registration
|
||||||
├── plugin/
|
├── plugin/
|
||||||
│ ├── metadata.rs # Plugin manifest parsing
|
│ ├── metadata.rs # Plugin manifest parsing
|
||||||
│ ├── dylib.rs # Dynamic library plugins
|
│ ├── dylib.rs # Dynamic library plugins
|
||||||
│ ├── rhai.rs # Rhai scripting (todo!())
|
│ ├── rhai.rs # Rhai scripting (todo!())
|
||||||
│ ├── shell.rs # Shell command plugins
|
│ ├── shell.rs # Shell command plugins
|
||||||
│ └── internal/ # Built-in plugins
|
│ └── internal.rs + internal/ # Built-in plugins (mail, webhook, satori, insitenotify)
|
||||||
│ ├── mail/ # Email notifications (SMTP)
|
├── types.rs
|
||||||
│ ├── webhook.rs # HTTP webhooks
|
├── types/compression.rs
|
||||||
│ ├── satori.rs # Satori integration
|
├── error.rs # FinalizeError
|
||||||
│ └── insitenotify.rs # In-site notifications
|
└── constant.rs
|
||||||
│ └── fetch/ # Artifact fetching
|
|
||||||
│ ├── git.rs # Git repository fetch
|
|
||||||
│ ├── http.rs # HTTP download
|
|
||||||
│ ├── extract.rs # Archive extraction
|
|
||||||
│ └── checksum.rs# SHA256 verification (SHA1 deprecated)
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## WHERE TO LOOK
|
## WHERE TO LOOK
|
||||||
|
|
||||||
| Task | Location |
|
| Task | Location |
|
||||||
|------|----------|
|
|------|----------|
|
||||||
| Config loading | `config.rs` - FinalizeConfig |
|
| Config loading | `config.rs:17` - `FinalizeConfig` + `parse()` |
|
||||||
| Email notify | `plugin/internal/mail/` - SMTP templates |
|
| Plugin registration | `plugin.rs` - `register()` |
|
||||||
|
| Email notifications | `plugin/internal/mail.rs` - SMTP |
|
||||||
| Webhooks | `plugin/internal/webhook.rs` |
|
| Webhooks | `plugin/internal/webhook.rs` |
|
||||||
| Artifact fetch | `plugin/fetch/` - git/http/extract |
|
| Satori integration | `plugin/internal/satori.rs` |
|
||||||
| Checksums | `plugin/fetch/checksum.rs:13` - SHA1 deprecated |
|
| Shell plugins | `plugin/shell.rs` |
|
||||||
|
| Template rendering | `template.rs` - `{{VARIABLE}}` handlebars-style |
|
||||||
|
|
||||||
## CONVENTIONS
|
## CONVENTIONS
|
||||||
|
- **Plugin System**: `FinalizePlugin` trait + registry with both internal and dynamic plugins
|
||||||
- **Plugin System**: Dynamic + internal plugins via Plugin trait
|
- **Template Syntax**: `{{VARIABLE}}` handlebars-style substitution via `template.rs`
|
||||||
- **Template Syntax**: `{{VARIABLE}}` handlebars-style
|
- **Stages**: EarlyHook → (Artifacts) → LateHook → Notifications
|
||||||
- **Stages**: Hook → Fetch → Build → Publish → Notify → Cleanup
|
- **Notification**: `notify.rs` in parent module drives plugin-based notification dispatch
|
||||||
- **Fetch**: Supports git, http, with extraction and checksum verify
|
- **ResourceRegistry**: Shared resource cache passed through to plugins
|
||||||
|
|
||||||
## ANTI-PATTERNS
|
## ANTI-PATTERNS
|
||||||
|
1. **fetch/ subdirectory removed** — was `plugin/fetch/{git,http,extract,checksum}`, no longer exists
|
||||||
1. **SHA1 deprecated** — `checksum.rs:13` warns SHA1 is deprecated
|
2. **SHA1 deprecated** — workshop-getterurl handles checksums now
|
||||||
2. **Unimplemented** — `rhai.rs:17` - "todo!()" for Rhai scripting
|
3. **Unimplemented** — `rhai.rs:17` - "todo!()" for Rhai scripting
|
||||||
3. **TODO markers** — Multiple TODOs in config.rs (lines 305-306)
|
4. **TODO markers** — Multiple TODOs in config.rs (lines 305-306)
|
||||||
|
5. **Artifact stage not implemented** — `finalize.rs:57` — "TODO: artifact stage implementation"
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ Build environment setup: bootstrap, dependencies, security, hooks.
|
|||||||
```
|
```
|
||||||
prebake/
|
prebake/
|
||||||
├── config.rs # PrebakeConfig YAML schema (environment, deps, cache)
|
├── config.rs # PrebakeConfig YAML schema (environment, deps, cache)
|
||||||
├── stage.rs # PrebakeStage enum (Bootstrap→Ready ordering)
|
├── stage.rs # PrebakeStage enum (Bootstrap→EarlyHook→DepsSystem→DepsUser→LateHook→Ready)
|
||||||
├── stage/
|
├── stage/
|
||||||
│ ├── bootstrap.rs # User creation (vulcan), sudo/doas setup
|
│ ├── bootstrap.rs # User creation (vulcan), sudo/doas setup
|
||||||
│ ├── environment.rs # Env var injection
|
│ ├── environment.rs # Env var injection
|
||||||
@@ -24,7 +24,12 @@ prebake/
|
|||||||
│ ├── baremetal.rs # Direct host execution
|
│ ├── baremetal.rs # Direct host execution
|
||||||
│ └── custom.rs # Custom env (todo!())
|
│ └── custom.rs # Custom env (todo!())
|
||||||
├── security.rs # Privilege dropping (unsafe libc calls)
|
├── security.rs # Privilege dropping (unsafe libc calls)
|
||||||
└── event.rs # Event logging
|
├── event.rs # Event logging
|
||||||
|
├── error.rs # PrebakeError + BootstrapError
|
||||||
|
├── types.rs # Module re-exports
|
||||||
|
├── types/ # memsize, cache, builderconfig, architecture
|
||||||
|
├── constant.rs # Module constants
|
||||||
|
└── env.rs # Environment module root
|
||||||
```
|
```
|
||||||
|
|
||||||
## WHERE TO LOOK
|
## WHERE TO LOOK
|
||||||
@@ -38,13 +43,11 @@ prebake/
|
|||||||
| Hooks | `stage/hook.rs` - pre/post stage hooks |
|
| Hooks | `stage/hook.rs` - pre/post stage hooks |
|
||||||
|
|
||||||
## CONVENTIONS
|
## CONVENTIONS
|
||||||
|
- **PrebakeStage**: Ordered enum Bootstrap→EarlyHook→DepsSystem→DepsUser→LateHook→Ready
|
||||||
- **PrebakeStage**: Ordered enum (Bootstrap→DepsSystem→DepsUser→Environment→Hooks→Ready)
|
|
||||||
- **ExecutionContext**: Carries task_id, username, env_vars through all stages
|
- **ExecutionContext**: Carries task_id, username, env_vars through all stages
|
||||||
- **Target User**: "vulcan" (hardcoded in bootstrap.rs:6-24)
|
- **Target User**: "vulcan" (hardcoded in bootstrap.rs:6-24)
|
||||||
|
|
||||||
## ANTI-PATTERNS
|
## ANTI-PATTERNS
|
||||||
|
|
||||||
1. **Hardcoded user** — "vulcan" username hardcoded throughout
|
1. **Hardcoded user** — "vulcan" username hardcoded throughout
|
||||||
2. **Unsafe blocks** — `security.rs` has 5+ unsafe libc calls
|
2. **Unsafe blocks** — `security.rs` has 5+ unsafe libc calls
|
||||||
3. **Incomplete** — custom.rs and apt.rs have `todo!()` stubs
|
3. **Incomplete** — custom.rs and apt.rs have `todo!()` stubs
|
||||||
|
|||||||
@@ -1,45 +0,0 @@
|
|||||||
mod checksum;
|
|
||||||
mod error;
|
|
||||||
mod extract;
|
|
||||||
mod file;
|
|
||||||
mod git;
|
|
||||||
mod http;
|
|
||||||
mod scheme;
|
|
||||||
mod types;
|
|
||||||
|
|
||||||
use std::path::{Path, PathBuf};
|
|
||||||
|
|
||||||
pub use error::FetchError;
|
|
||||||
pub use scheme::Scheme;
|
|
||||||
pub use types::FetchArgument;
|
|
||||||
|
|
||||||
/// Fetch an external resource to a target directory. Returns the local path.
|
|
||||||
///
|
|
||||||
/// - `git://repo@tag` → clone
|
|
||||||
/// - `https://url` → download (+ checksum + extract)
|
|
||||||
/// - `file:///path` → copy to target
|
|
||||||
///
|
|
||||||
/// Note: `workspace://` must be converted to `file://` by the caller before passing.
|
|
||||||
pub async fn fetch_resource(
|
|
||||||
name: &str,
|
|
||||||
url: &str,
|
|
||||||
target: &Path,
|
|
||||||
) -> Result<PathBuf, FetchError> {
|
|
||||||
let scheme = scheme::parse_url(url)?;
|
|
||||||
match scheme {
|
|
||||||
Scheme::Git { repo, git_ref } => {
|
|
||||||
let full_url = if git_ref.is_empty() {
|
|
||||||
repo.clone()
|
|
||||||
} else {
|
|
||||||
format!("{}@{}", repo, git_ref)
|
|
||||||
};
|
|
||||||
git::fetch_git_plugin(&full_url, name, target, None).await?;
|
|
||||||
Ok(target.join(name))
|
|
||||||
}
|
|
||||||
Scheme::Http { url, checksum } => {
|
|
||||||
http::fetch_http_plugin(&url, name, target, checksum.as_deref()).await?;
|
|
||||||
Ok(target.join(name))
|
|
||||||
}
|
|
||||||
Scheme::File { path } => file::fetch_file(&path, name, target).await,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
# workshop-engine/src
|
||||||
|
|
||||||
|
**Generated:** 2026-05-19
|
||||||
|
**Commit:** 28abc5d
|
||||||
|
|
||||||
|
Execution engine extracted from workshop-baker. Process management, package detection, resource isolation.
|
||||||
|
|
||||||
|
## STRUCTURE
|
||||||
|
```
|
||||||
|
src/
|
||||||
|
├── lib.rs # Module declarations + 10+ convenience re-exports
|
||||||
|
├── error.rs # ExecutionError + DependencyError + HasExitCode trait
|
||||||
|
├── types.rs # ExecutionContext, ExecutionResult, Engine, etc.
|
||||||
|
├── child.rs # ManagedChild + TokioChild process wrappers
|
||||||
|
├── command.rs # CustomCommand builder
|
||||||
|
├── executor.rs # Script execution
|
||||||
|
├── time.rs # Duration parsing helpers
|
||||||
|
├── pm.rs # PackageManager trait + detection
|
||||||
|
├── pm/ # Backends: apt, apk, dnf, pacman
|
||||||
|
├── upm.rs # UserPackageManager trait
|
||||||
|
├── upm/ # Backends: rust (cargo), python (pip), go, custom
|
||||||
|
├── repology.rs # RepologyEndpoint enum
|
||||||
|
└── repology/ # Package name resolution (local/)
|
||||||
|
```
|
||||||
|
|
||||||
|
## KEY EXPORTS
|
||||||
|
| Symbol | Type | Role |
|
||||||
|
|--------|------|------|
|
||||||
|
| ExecutionContext | struct | Pipeline context (task_id, username, env, etc.) |
|
||||||
|
| Engine | struct | Script executor (placeholder) |
|
||||||
|
| ExecutionError | enum | Error types with exit codes |
|
||||||
|
| HasExitCode | trait | Exit code protocol for all error types |
|
||||||
|
| ManagedChild | struct | Child process wrapper with managed lifecycle |
|
||||||
|
| TokioChild | struct | Tokio-based child process wrapper |
|
||||||
|
| CustomCommand | struct | Builder for command execution |
|
||||||
|
| PackageManager | trait | System package manager abstraction |
|
||||||
|
| UserPackageManager | trait | User-level package manager abstraction |
|
||||||
|
| RepologyEndpoint | enum | Package name resolution endpoint |
|
||||||
|
| ExecutionEvent | enum | Event stream (TaskStarted, OutputChunk, etc.) |
|
||||||
|
| ExecutionResult | struct | Command execution result (exit_code, stdout, stderr) |
|
||||||
|
| EventSender | type alias | `Option<mpsc::Sender<ExecutionEvent>>` |
|
||||||
|
|
||||||
|
## CONVENTIONS
|
||||||
|
- **Exit codes**: Centralized in `error.rs` — EXITCODE_OK=0 through EXITCODE_PRIV_DROP_FAILED=201
|
||||||
|
- **ExecutionContext**: Defaults to current directory, bash shell, non-privileged
|
||||||
|
- **PackageManagers**: Auto-detected via `os_info`, fallback chain (apt→apk→dnf→pacman)
|
||||||
|
- **Engine**: Currently a unit struct with `execute_script()` method
|
||||||
|
- **All errors implement HasExitCode**: Enables uniform error propagation up to CLI
|
||||||
|
|
||||||
|
## ANTI-PATTERNS
|
||||||
|
1. **Engine is a stub** — `types.rs:99` — `pub struct Engine;` with no fields, minimal implementation
|
||||||
@@ -1,115 +0,0 @@
|
|||||||
use std::fs;
|
|
||||||
use std::io::{self, Write};
|
|
||||||
use std::path::PathBuf;
|
|
||||||
|
|
||||||
/// Linux cgroup resource manager for ws-executor.
|
|
||||||
///
|
|
||||||
/// Manages cgroup v2 hierarchy under `/sys/fs/cgroup/ws-executor/` for
|
|
||||||
/// CPU, memory, and process isolation.
|
|
||||||
pub struct CgroupManager {
|
|
||||||
path: PathBuf,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl CgroupManager {
|
|
||||||
/// Creates a new cgroup for the given build ID.
|
|
||||||
pub fn new(build_id: &str) -> io::Result<Self> {
|
|
||||||
let path = PathBuf::from(format!("/sys/fs/cgroup/ws-executor/{}", build_id));
|
|
||||||
|
|
||||||
fs::create_dir_all(&path)?;
|
|
||||||
|
|
||||||
Ok(Self { path })
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Sets memory limit in bytes. Use 0 for unlimited.
|
|
||||||
pub fn set_memory_limit(&self, limit_bytes: u64) -> io::Result<()> {
|
|
||||||
let mem_max_path = self.path.join("memory.max");
|
|
||||||
let content = if limit_bytes == 0 {
|
|
||||||
"max".to_string()
|
|
||||||
} else {
|
|
||||||
limit_bytes.to_string()
|
|
||||||
};
|
|
||||||
fs::write(&mem_max_path, content)?;
|
|
||||||
|
|
||||||
let mem_swap_path = self.path.join("memory.swap.max");
|
|
||||||
fs::write(&mem_swap_path, "max")?;
|
|
||||||
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Sets CPU weight (1-10000, higher = more CPU time).
|
|
||||||
pub fn set_cpu_weight(&self, weight: u64) -> io::Result<()> {
|
|
||||||
let cpu_weight_path = self.path.join("cpu.weight");
|
|
||||||
let weight = weight.clamp(1, 10000);
|
|
||||||
fs::write(&cpu_weight_path, weight.to_string())?;
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Sets CPU max burst in microseconds with optional period.
|
|
||||||
pub fn set_cpu_max(&self, max_us: u64, period_us: u64) -> io::Result<()> {
|
|
||||||
let cpu_max_path = self.path.join("cpu.max");
|
|
||||||
let content = if max_us == 0 {
|
|
||||||
"max".to_string()
|
|
||||||
} else {
|
|
||||||
format!("{} {}", max_us, period_us)
|
|
||||||
};
|
|
||||||
fs::write(&cpu_max_path, content)?;
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Adds a process to this cgroup.
|
|
||||||
pub fn add_process(&self, pid: u32) -> io::Result<()> {
|
|
||||||
let cgroup_procs_path = self.path.join("cgroup.procs");
|
|
||||||
let mut file = fs::OpenOptions::new()
|
|
||||||
.append(true)
|
|
||||||
.open(&cgroup_procs_path)?;
|
|
||||||
|
|
||||||
writeln!(file, "{}", pid)?;
|
|
||||||
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Destroys the cgroup directory.
|
|
||||||
pub fn destroy(&self) -> io::Result<()> {
|
|
||||||
if self.path.exists() {
|
|
||||||
fs::remove_dir(&self.path)?;
|
|
||||||
}
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Returns the cgroup path.
|
|
||||||
pub fn path(&self) -> &PathBuf {
|
|
||||||
&self.path
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod tests {
|
|
||||||
use super::*;
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
#[ignore]
|
|
||||||
fn test_cgroup_manager_new() {
|
|
||||||
let build_id = "test-build-123";
|
|
||||||
let manager = CgroupManager::new(build_id).unwrap();
|
|
||||||
assert!(manager.path().exists());
|
|
||||||
manager.destroy().unwrap();
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
#[ignore]
|
|
||||||
fn test_set_memory_limit() {
|
|
||||||
let build_id = "test-mem-123";
|
|
||||||
let manager = CgroupManager::new(build_id).unwrap();
|
|
||||||
manager.set_memory_limit(1024 * 1024 * 1024).unwrap();
|
|
||||||
manager.destroy().unwrap();
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
#[ignore]
|
|
||||||
fn test_set_cpu_weight() {
|
|
||||||
let build_id = "test-cpu-123";
|
|
||||||
let manager = CgroupManager::new(build_id).unwrap();
|
|
||||||
manager.set_cpu_weight(1024).unwrap();
|
|
||||||
manager.destroy().unwrap();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Reference in New Issue
Block a user