diff --git a/clarification.md b/clarification.md deleted file mode 100644 index 1e23b89..0000000 --- a/clarification.md +++ /dev/null @@ -1,1211 +0,0 @@ - -## Updated Questions & Doubts - -### A. Plugin System - -1. **`fallible` vs `critical`** - - You changed from `critical: false` to `fallible: false` - - Is `fallible: true` the same as `critical: false`? (Double negative situation) - - Consider: `fail_build_on_error: true/false` is clearer - -2. **Plugin URI Format** - - `git::github.com/user/repo@main/1234abcd` - - Is `main` the branch and `1234abcd` the commit? - - What if branch and commit don't match? Which takes precedence? - - Are other protocols supported? (`hg::`, `http::`, `file::`?) - -3. **Plugin Configuration Scope** - - In `plugin:` section, you define `host`, `user`, `token` - - In `publish:`, you define `image: "docker.io/user/image:tag"` - - Are these merged? Overridden? - - What if `publish.docker-push.image` differs from `plugin.docker-push.host`? - -4. **Multiple Plugin Implementations** - - Can `docker-push:` have multiple URIs under it? - ```yaml - docker-push: - "git::github.com/user/plugin-a@main": ... - "git::github.com/user/plugin-b@main": ... # Allowed? - ``` - -### B. Notification Priority - -5. **Fallback Semantics** - - Comment: "inter-group falling back" - - Does group `1` run **only if** group `0` fails? - - Or does group `1` run **after** group `0` succeeds? - - Or both groups run, but `1` is lower priority for delivery? - -6. **Parallel Within Group** - - If `webhook` succeeds but `satori` fails in group `0`, what happens? - - Does group `1` get triggered because of partial failure? - - Or is group `0` considered "succeeded" if at least one succeeds? - -7. **Notification Failure vs Build Failure** - - If all notifications fail, does the build fail? - - Is there a global `fallible` for notifications? - -### C. Stage-Based Notifications - -8. **`on_finish_of` Integration** - ```yaml - on_finish_of: - stage: "prebake.ready" - ``` - - How does this integrate with `PrebakeStage` enum from codebase? - - Current stages: `Bootstrap -> Ready` (from AGENTS.md) - - Are all stages valid? (`prebake.bootstrap`, `bake.complete`, `finalize.*`?) - -9. **`on_finish_of` vs `on_success`/`on_failure`** - - Can these coexist on the same notification? - - Does `on_finish_of` trigger regardless of success/failure? - - Is there `on_start_of` for "notify when stage begins"? - -10. **Stage Timing** - - When `on_finish_of: prebake.ready` fires, is `{{ build.status }}` available? - - Or is build status only meaningful after `bake` stage? - -### D. Artifact System - -11. **Auto-Generated ID** - ```yaml - id: "" # Generated by Pipeline UUID & Path when not given - ``` - - Is `id` optional now? Can you omit it entirely? - - If auto-generated, how do you reference it in `publish`? - - Or is `publish` always by `path` match, not `id`? - -12. **Empty Path Semantics** - ```yaml - path: "" # Implies non-directory artifact - ``` - - What is a "non-directory artifact"? - - Is this for metadata-only artifacts? - - Or does empty path mean "use default output path from bake.sh"? - -13. **Retention Duration Parsing** - - `"7d"`, `604800` (seconds?), `"168h"` - are these equivalent? - - What's the canonical format? Should there be validation? - - Consider standardizing: ISO 8601 duration (`P7D`)? - -14. **Compression `dir` Option** - ```yaml - compression: "zstd" # gzip|none|dir allowed - ``` - - What does `compression: "dir"` mean? No compression, keep as directory? - - Should this be `"directory"` or `"raw"` for clarity? - -### E. Template Variables - -15. **Variable Availability by Stage** - - `{{ build.duration }}` - available at `prebake.ready`? (Probably not) - - `{{ build.status }}` - what values? `"success"` | `"failure"` | `"running"`? - - Should there be stage-specific variable namespaces? (`{{ prebake.* }}`, `{{ bake.* }}`) - -16. **Target Namespace** - - `{{ target.group_id }}` - where is this defined? - - Is `target` the notification target, or build target environment? - - Can users define custom `{{ target.* }}` variables? - -### F. Schema & Validation - -17. **Schema Field Purpose** - ```yaml - schema: "default" - schema: "custom" - ``` - - What does `default` schema do? Built-in payload format? - - Is `custom` required when `on_*` templates are provided? - - Can you use `schema: "slack"` or other named schemas? - -18. **Version Compatibility** - - `version: "0.0.1"` - suggests breaking changes expected - - How will version migrations work? - - Should there be a `schema_version` separate from implementation version? - -### G. Missing Features (Intentional?) - -19. **Conditional Artifacts** - - Can artifacts have conditions like bake.sh's `@if()`? - - E.g., only publish docker image on `main` branch - -20. **Artifact Dependencies** - - Can one artifact depend on another? - - E.g., "publish only if artifact A was successfully uploaded" - -21. **Cleanup Granularity** - ```yaml - cleanup: - policy: "auto" - ``` - - What other policies exist? `"manual"`? `"on_failure"`? - - Is cleanup per-artifact or global? - -22. **Timeouts & Retries for Notifications/Plugins** - - No timeout/retry config visible - - Are these inherited from bake.sh decorators? - - Should plugins have their own retry logic? - ---- - -### Summary of Design Gaps - -| Area | Missing/Unclear | -|------|-----------------| -| Plugin | Config merging, multiple URIs, protocol support | -| Notification | Fallback semantics, failure propagation, stage integration | -| Artifact | Auto-ID referencing, empty path use case, conditions | -| Template | Variable availability by stage, custom namespaces | -| Schema | `default`/`custom` behavior, extensibility | -| Error Handling | Timeouts, retries, failure escalation | - -What would you like to clarify or iterate on? - ---- - -## User - -1. fallible:true = critical: false, of course. This concept is the same as bake.sh -2. if no according commit/tag in the branch, it means nothing and fail. -Other protocols will be supported but some are sonner(file) and some are later(hg) -3. Caller than Definition: parameters in publish overrides plugin -No way could user success if image.host != host. host is only for authentication usage. image.host could be emitted if host given. -4. not allowed. key is unique. -5/6/7. fallback explanation: (notification->no) -no.0 starts all at once -if one method in no.0 fails, no.1 is active (then no.2,...) -the failure of no.0 will be part of no.1, this means baker will wait for no.0 to finish and then no.1 -fallible could apply here -8. for prebake/finalize, second field is static, for bake, second field is dynamic according to bake.sh / Function(class) -validity checks upon execution, maybe early-checked at future -9. yes -maybe a condition system required -10. Still WIP -11. ID is optional and will be generated by system -publish doesn't reference artifact, artifact reference publish -12. Docker image as an example -future url:// based path, such as docker://docker.io/user/image:tag -adopted from RedoxOS, however too advanced to promote. -13. equivalent. validation required. default unit is second -14. dir will be used for software repo: they need structued directory than single .tar -15. WIP -16. WIP -17. default will be defined by (internal) plugins -custom required for on* -schema could be defined in notification block for unified use -18. We'd expect not to change schema after 1.0, though it's impossible -migration system built-in for new version -19. yes -20. yes -21. WIP -22. WIP - ---- - -## Assistant (Sisyphus (Ultraworker) · GLM-5 · 57.4s) - -## Synthesized Design Understanding - -Based on your clarifications, here's the refined design: - -### Plugin System -```yaml -plugin: - : # Unique key - "::@/": - # Auth/config parameters (can be overridden in publish) - host: "docker.io" - user: {{ dockerhub.user }} - token: {{ secret.dockerhub.token }} - fallible: false # Same semantics as bake.sh @fallible -``` -- `host` = authentication endpoint only -- `publish..image` can omit host if plugin `host` defined - -### Notification Flow -``` -Priority 0: [webhook, satori] ──parallel──┬──► if ALL fail ──► Priority 1: [mail] - │ - └──► wait for completion before next group -``` -- Within group: parallel -- Across groups: sequential fallback on failure -- `fallible` can apply per notification method - -### Artifact Publishing -```yaml -artifact: - - path: "" # Empty = URL-style artifact (docker://, etc.) - publish: - : - # Override plugin params here - image: "docker.io/user/image:tag" # host optional if plugin defines it -``` -- Artifact **pushes to** publish, not publish pulls artifact -- ID auto-generated when omitted - ---- - -## Follow-Up Questions - -### A. Plugin Config Merging - -1. **Deep vs Shallow Merge** - ```yaml - plugin: - docker-push: - host: "docker.io" - auth: - user: "admin" - token: "xxx" - - publish: - docker-push: - auth: - user: "override-user" # Does this merge {user, token} or replace entire auth? - ``` - - Is merge **deep** (nested objects merge) or **shallow** (replace entire key)? - -2. **Required vs Optional Plugin Params** - - If plugin defines `host` and `user` as required, but publish omits `user`... - - Is there a schema for plugin to declare required params? - - Or is validation purely "fail at runtime if missing"? - -### B. Notification Fallback Edge Cases - -3. **Partial Failure Within Group** - ``` - Priority 0: webhook ✓, satori ✗ - ``` - - Is group 0 considered "failed" if **any** method fails? - - Or only if **all** methods fail? - - This determines whether priority 1 triggers - -4. **Timeout Interaction with Fallback** - ```yaml - 0: - webhook: - url: "..." - timeout: 30s # Hypothetical - 1: - mail: ... - ``` - - If webhook times out at 30s, does mail immediately start? - - Or does system wait for webhook's internal retry logic first? - -5. **Fallback with `fallible`** - ```yaml - 0: - webhook: - fallible: true - 1: - mail: - fallible: false - ``` - - If webhook fails but is `fallible: true`, does mail still trigger? - - Or does `fallible: true` mean "failure doesn't trigger fallback"? - -### C. Stage-Based Notifications - -6. **Stage Naming Convention** - - `prebake.ready` - static name - - `bake.build` - function name from bake.sh? - - What about `finalize` stages? `finalize.artifact`? `finalize.notification`? - - Are there built-in stages that always exist? - -7. **Multiple `on_finish_of` Triggers** - ```yaml - satori: - on_finish_of: - - stage: "prebake.ready" - - stage: "bake.test" - ``` - - Can a notification trigger at multiple stages? - - Or is it one `on_finish_of` per notification definition? - -8. **`on_success` + `on_finish_of` Combination** - ```yaml - satori: - on_finish_of: - stage: "prebake.ready" - on_success: | # Build success, not stage success? - ``` - - If `on_finish_of: prebake.ready` fires but build eventually fails... - - Does `on_success` content get used or `on_failure`? - - Or does `on_finish_of` have its own `content` only? - -### D. Artifact Path Schemes - -9. **URL Scheme Examples** - ```yaml - path: "docker://docker.io/user/image:tag" - path: "s3://bucket/key" - path: "ghcr://user/repo:tag" - ``` - - Are these the intended schemes? Or something else? - - Does each scheme map to a specific plugin automatically? - -10. **Mixed Path Types in Single Config** - ```yaml - artifact: - - path: "/workspace/output.tar.gz" # File - - path: "" # Docker image (how to specify image name?) - - path: "docker://docker.io/img:tag" # Explicit scheme - ``` - - When `path: ""`, where does the image name come from? Only from `publish.docker-push.image`? - - Can you have `path: ""` without `publish`? (Just store locally?) - -### E. Compression & Directory Artifacts - -11. **`dir` Compression + Upload** - ```yaml - artifact: - - path: "/workspace/output/" - compression: "dir" - publish: - some-plugin: ... - ``` - - When uploading a `dir` artifact, does plugin receive: - - The directory path? - - A tarball created on-the-fly? - - Individual files? - - How does `docker-push` handle `compression: "dir"`? - -### F. Conditional System (Planned) - -12. **Condition Syntax** - ```yaml - artifact: - - path: "..." - publish: - docker-push: - image: "..." - if: "${BUILD_BRANCH} == 'main'" # Hypothetical - ``` - - Will conditions use shell-style `$VAR`? - - Or template-style `{{ env.BRANCH }}`? - - Or bake.sh-style `@if(WS_PIPELINE_ARCH == amd64)`? - -### G. Error Handling - -13. **Plugin Execution Failure** - ```yaml - artifact: - - publish: - docker-push: - fallible: true - another-plugin: - fallible: false - ``` - - If `docker-push` fails (fallible), does `another-plugin` still run? - - Or do all plugins for an artifact run in parallel regardless? - -14. **Artifact Collection Failure** - - What if `path: "/workspace/target/release/*"` matches nothing? - - Is this a build failure or silently skipped? - - Is there `on: ["missing"]` for handling missing artifacts? - ---- - -## Design Consistency Observations - -### Alignment with Existing Patterns - -| Concept | prebake.yml | bake.sh | finalize.yml | -|---------|-------------|---------|--------------| -| Fallible | - | `@fallible` | `fallible: bool` ✓ | -| Timeout | - | `@timeout(30)` | WIP | -| Retry | - | `@retry(3, 5)` | WIP | -| Parallel | - | `@parallel` | Intra-group parallel ✓ | -| Conditional | - | `@if(cond)` | Planned ✓ | -| After/Deps | `hooks.early/late` | `@after(func)` | Priority groups ✓ | - -### Potential Inconsistencies - -1. **`retension` typo** - Should be `retention` - -2. **Duration format inconsistency** - - prebake.yml: `ttl_days: 30` - - finalize.yml: `retension: "7d"` - - Consider standardizing to one format - -3. **Plugin reference style** - - bake.sh: `@after(func_name)` - direct reference - - finalize.yml: `publish: docker-push:` - nested config - - Both valid, just noting the difference - ---- - -What would you like to explore next? - ---- - -## User - -3. any -1. overrides "auth" than "user" -2. checker cries. checked implemented in plugin -4. same as 3. timeout = single failure, not group failure -5. fallible of a stage will stop error propagation, thus leaving 1 alone. -for last stage, fallible means no other method of notification. -(if none of notification FULLY works, system-level fallback, like user email via system(SHOULD be well-configured because user creation relies on this) and in-site notification) -6. WIP cuz finalize itself is our plan -7. yes, logical expression may be introduced -8. every condition have its own content -9. not quite planned by now. -not completely correct? bake.sh shoudn't upload content to S3 and let finalize to fetch it for artifact -however maybe we don't prohibit this. -users may do everything in bake and skip finalize. -10. path:none + publish: none means nothing, because the build environment will soon be destroyed -if it's saved externally (eg. via docker socket), then it shoudn't exist as artifact -11. plugin receive a prefetched directory at plugin workspace, this is ensured by rsync. -docker-push should ignore compression (or maybe prohibit it) -12. WIP -13. if one fails but fallible, others will run as expected -14. matches nothing invalidates artifact content, but plugin may handle "none" as content -15. retention IS typo. -16. we haven't implemented prebake.cache thus they might be unified - ---- - -## Assistant (Sisyphus (Ultraworker) · GLM-5 · 54.4s) - -## Updated Design Understanding - -### Notification Fallback (Clarified) -``` -Priority 0: [webhook ✗, satori ✗] ──► ANY fails triggers fallback - ↓ -Priority 1: [mail] - ↓ -Priority N: [system-email, in-site] ──► Ultimate fallback (system-level) -``` -- `fallible: true` at group N = **stop propagation**, don't trigger group N+1 -- `fallible: true` at **last defined group** = no further notification attempts -- System-level fallback exists **outside** user config (configured at user creation) - -### Plugin Execution -``` -Artifact Collection (rsync to plugin workspace) - ↓ -Plugin receives: prefetched directory - ↓ -docker-push: ignores/prohibits compression: "dir" -``` - -### Config Override Behavior -```yaml -plugin: - docker-push: - auth: # ENTIRE key replaced - user: "default" - token: "xxx" - -publish: - docker-push: - auth: - user: "override" # { user: "override" } - token LOST -``` - ---- - -## Remaining Questions - -### A. Plugin Workspace & File Handling - -1. **Plugin Workspace Location** - - Where is the plugin workspace? `/workspace/.plugins/docker-push/`? - - Is this workspace shared across all artifacts using same plugin? - - Or isolated per artifact? - -2. **Large Artifact Handling** - - If artifact is 10GB, does rsync complete before plugin starts? - - Or does plugin stream from rsync? - - Is there a size limit/timeout for rsync? - -3. **Plugin Working Directory** - - Does plugin receive `path` as-is, or resolved absolute path? - - Example: `path: "/workspace/target/release/*"` → plugin sees `/workspace/.plugins/docker-push/release/`? - -### B. Artifact Validation - -4. **Glob Expansion Timing** - ```yaml - path: "/workspace/target/release/*" - ``` - - When is glob expanded? Config load? Execution time? - - If expanded at load time to 3 files, is it 3 artifacts or 1 artifact with 3 files? - -5. **Artifact ID Generation** - - Format: `pipeline-uuid-path-hash`? Or something else? - - Is ID deterministic (same path = same ID) or random? - -### C. Publish Execution Order - -6. **Multiple Plugins for One Artifact** - ```yaml - artifact: - - path: "/workspace/output.tar.gz" - publish: - docker-push: - image: "app:latest" - s3-upload: # Second plugin - bucket: "my-bucket" - ``` - - Execution order: sequential? parallel? configurable? - - If `docker-push` fails, does `s3-upload` run? - -### D. Notification Content Templates - -7. **Condition-Content Mapping** - ```yaml - satori: - on_finish_of: - stage: "prebake.ready" - content: "Prebake done" - on_success: - content: "Build succeeded" - on_failure: - content: "Build failed" - ``` - - If `prebake.ready` finishes but build later fails... - - Does `on_finish_of` send at `prebake.ready` time? - - Then `on_failure` sends at build end? - - Both messages sent, right? - -8. **Template Variable Scope** - ```yaml - on_finish_of: - stage: "prebake.ready" - content: "{{ build.status }}" # Not available yet? - ``` - - What happens when variable isn't available at that stage? - - Empty string? Error? `"unknown"`? - -### E. Error Recovery - -9. **Plugin Error Reporting** - - How does plugin report errors back to baker? - - Exit code? JSON output? Structured log? - - Is there a plugin API contract? - -10. **Retrying Individual Methods** - ```yaml - notification: - 0: - webhook: - retry: 3 # Hypothetical - retry_delay: 5s - ``` - - Will retry be per-method, per-group, or global? - - Or should plugins handle their own retry? - -### F. System-Level Fallback - -11. **System Notification Configuration** - - Where is system email configured? - - In prebake.yml? baker config? System-wide config? - - What if system email also fails? - -12. **In-Site Notification** - - Does this mean a web UI notification? - - Stored in database? Shown in baker dashboard? - -### G. Typo & Format Fixes - -13. **Proposed Fixes** - ```yaml - artifact: - - path: "/workspace/output.tar.gz" - retention: "7d" # Fixed typo - compression: "zstd" - on: - - success - ``` - - Confirm: `retension` → `retention` for final version? - -14. **Duration Format Unification** - ```yaml - # Option A: String with unit (current) - retention: "7d" - - # Option B: Structured - retention: - days: 7 - - # Option C: ISO 8601 - retention: "P7D" - ``` - - Preference for consistency with future `prebake.cache`? - -### H. Edge Cases - -15. **Empty Notification Group** - ```yaml - notification: - 0: {} # Empty group? - 1: - mail: ... - ``` - - Is empty group valid? Ignored? - - Or validation error? - -16. **Circular Plugin Reference** - - Can plugin A's output be used as plugin B's input? - - Example: docker-push → image scanner plugin - - Or should this be a single composed plugin? - ---- - -## Summary: Design Maturity - -| Component | Status | Notes | -|-----------|--------|-------| -| Plugin system | ✅ Clear | URI, override, fallible defined | -| Notification priority | ✅ Clear | Fallback semantics defined | -| Notification stages | ⚠️ WIP | `on_finish_of` structure defined, finalize stages TBD | -| Artifact collection | ✅ Clear | rsync to plugin workspace | -| Artifact publishing | ✅ Clear | Sequential/parallel WIP | -| Conditional system | ⚠️ WIP | Planned | -| Error handling | ⚠️ WIP | Timeout/retry TBD | -| System fallback | ✅ Clear | Email + in-site notification | -| Typo fix | ✅ Confirmed | `retention` | -| Duration format | ⚠️ TBD | Unify with prebake.cache | - -What's the next aspect to design or clarify? - ---- - -## User - -1. defined by plugin itself -2. rsync before plugin, UNLESS specified by plugin (maybe someone implements more robust method) -(not completely rsync thou, hard link/filesystem features may be used) -3. as 1 -4. glob->RE, checked when loading, compiled at running, 1s for total limitation(RE compile+match). one path means one artifact element -5. hash of PUID+PATH, maybe MD5 is enough -6. SHOULD be parallel, fails: see fallible -7. both sent, one first. -8. WIP as variable system not defined, just as example. build.status could mean "bake.fetch:3%" or "prebake:ready!" here. -9. plugin is shell scripts/ rhai scripts. Via API/CLI(or, function, for shell) and exit-code. Log will be captured but not as evidence -10. retry per method -11. system email contigured via Web+Server(not started) -If system email also fails: via in-site notification -If none works: we admit we have no way to notify user. We give up. Admin will receive this message if the configures to do so. -12. WebUI, stored in DB, in baker dashboard. -13. we'll fix it -14. ISO 8601 sounds good, P could be emitted because only period supported. -Who'd like to hard-code specific time into static config? -Maybe there is. -15. 0 should be eliminated and email evolve to 0 -every field could lie in any priority, thus webhook could lie in 999 -treat as a failure if 0 is none -16. make plugin callable w/o artifact -chain loading might be reconsidered, because push and check may share the same INPUT than OUTPUT - ---- - -## Assistant (Sisyphus (Ultraworker) · GLM-5 · 54.3s) - -## Final Design Synthesis - -### Artifact Collection Flow -``` -Glob Path → RE Compile (load) → Match (run, 1s timeout) → One artifact per match - ↓ - Hash(PUID + PATH) → MD5 as ID - ↓ - Hard-link/rsync to plugin workspace (plugin-defined location) - ↓ - Plugin executes (shell/rhai) - ↓ - Exit code = success/failure -``` - -### Notification Flow (Final) -``` -Priority 0: [webhook, satori] ──parallel──► ALL succeed = done - │ │ - │ └── ANY fails ──► Priority 1 - │ │ - ▼ ▼ - retry per method [mail] ──► ... - │ - ▼ - System fallback: in-site DB - │ - ▼ - Give up → Admin notification -``` - -### Plugin Architecture -``` -Plugin = Shell script | Rhai script - └── Communicates via: API/CLI/function calls + exit code - └── Log captured (informational, not evidence) - └── Workspace location: plugin-defined - └── File transfer: hard-link/rsync by default, plugin can override - └── Callable without artifact (standalone execution) -``` - -### Duration Format -```yaml -retention: "7D" # ISO 8601, P omitted -retention: "168H" # Hours also valid -retention: "604800" # Seconds (no unit) -``` - ---- - -## Final Questions - -### A. Plugin Without Artifact - -1. **Standalone Plugin Definition** - ```yaml - plugin: - health-check: # No artifact input - "git::github.com/user/health-plugin@main": - endpoint: "https://api.example.com" - - # How is it invoked? - ``` - - Where does standalone plugin get invoked? In `notification` block? - - Or separate `finalize.plugins:` section? - - Example syntax? - -2. **Standalone Plugin Output** - - Does standalone plugin produce anything? - - Or purely side-effect (HTTP call, etc.)? - -### B. Plugin Chain (Reconsidered) - -3. **Shared Input Pattern** - ``` - Artifact A ──► docker-push ──► (image in registry) - │ - └──► image-scanner ──► (scan results) - (same input: artifact A, not docker-push output) - ``` - - Both plugins receive same artifact input? - - Configuration example: - ```yaml - artifact: - - path: "/workspace/image.tar" - publish: - docker-push: { image: "app:latest" } - image-scanner: { severity: "high" } # Same input, parallel? - ``` - -### C. Priority Group Flexibility - -4. **Non-Sequential Priority Numbers** - ```yaml - notification: - 0: - webhook: ... - 999: - mail: ... - ``` - - Is 999 valid? Or auto-renumbered to 1? - - What about gaps: `0`, `5`, `10`? - - Integers only, or can use named priorities? (`high`, `low`?) - -5. **Empty Group Handling** - ```yaml - notification: - 0: {} - 1: - mail: ... - ``` - - Result: `0` eliminated, `mail` becomes `0`? - - Or: validation error at config load? - -### D. Plugin Language Details - -6. **Rhai vs Shell Choice** - - Is this a plugin-level declaration? - ```yaml - plugin: - my-plugin: - "git::..." - runtime: "rhai" # or "shell"? - ``` - - Or detected by file extension? (`plugin.rhai` vs `plugin.sh`) - -7. **Plugin API Contract** - - Shell: environment variables? CLI arguments? Stdin? - - Rhai: injected functions/objects? - - Example of plugin receiving artifact path and config? - -### E. Build Status Progress - -8. **Progress Reporting** - ``` - build.status = "bake.fetch:3%" - build.status = "prebake:ready!" - ``` - - Is this a real-time string that updates? - - Or structured: `{{ build.stage }}` + `{{ build.progress }}`? - - How does plugin/notification read live progress? - -### F. Configuration Validation - -9. **Early vs Late Validation** - - Glob → RE: checked at load time - - Plugin existence: checked at load or run? - - Secret reference (`{{ secret.xxx }}`): checked at load or run? - -10. **Schema Validation Scope** - - Is there a JSON Schema for finalize.yml? - - Or just Rust struct deserialization with custom validators? - -### G. Security Considerations - -11. **Plugin Trust Model** - - Plugins are arbitrary shell scripts from Git repos - - How is trust established? - - Allowlist? Signature verification? User confirmation on first use? - -12. **Secret Exposure to Plugins** - - `{{ secret.dockerhub.token }}` - passed to plugin how? - - Environment variable? File? CLI arg (visible in ps)? - - Plugin sandbox/isolation? - ---- - -## Design Maturity (Final) - -| Component | Status | Open Items | -|-----------|--------|------------| -| Plugin execution | ✅ | Trust model, secret passing | -| Plugin standalone | ⚠️ | Invocation syntax TBD | -| Plugin chain | ✅ | Shared input pattern confirmed | -| Artifact collection | ✅ | Hash ID, glob→RE | -| Notification priority | ✅ | Empty group handling | -| Notification fallback | ✅ | System fallback chain defined | -| Plugin runtime | ⚠️ | API contract details | -| Progress reporting | ⚠️ | Variable system WIP | -| Duration format | ✅ | ISO 8601, P omitted | -| Validation | ⚠️ | Schema definition | -| Typo | ✅ | `retention` confirmed | - ---- - -Is there anything else to clarify, or should we proceed to implementation planning? - ---- - -## User - -1. WIP -2. MAY produce anything, may become another artifact -however this is weird, users should do this in bake stage than finalize -3. plugins could own TOKEN/LOCK and release upon finish -eg docker-push acquires "IMAGE." token. -image-scanner could start in parallel but soon be paused for no token available. -4. integers only, 0~INT_MAX, we don't limit -if someone writes 2147483647, either a. as long as they are happy or b. rejected by their superior -5. mail should become 0 however we don't do this -we warn about empty group -6. runtime could be automatically detected -7. shell: via CLI and VAR -rhai: WIP -8. stage maybe means LAST FINAL state, progress is dynamic. -9. by design bakerd should fetch plugin as long as they're ready (git/http at prebake stage and file at finalize.begin) -assume nothing changed git/http when baking because they require hash. - - ---- - -## Assistant (Sisyphus (Ultraworker) · GLM-5 · 14.0s) - ---- - -## User - -1. WIP -2. MAY produce anything, may become another artifact -however this is weird, users should do this in bake stage than finalize -3. plugins could own TOKEN/LOCK and release upon finish -eg docker-push acquires "IMAGE." token. -image-scanner could start in parallel but soon be paused for no token available. -4. integers only, 0~INT_MAX, we don't limit -if someone writes 2147483647, either a. as long as they are happy or b. rejected by their superior -5. mail should become 0 however we don't do this -we warn about empty group -6. runtime could be automatically detected -7. shell: via CLI and VAR -rhai: WIP -8. stage maybe means LAST FINAL state, progress is dynamic. -9. by design bakerd should fetch plugin as long as they're ready (git/http at prebake stage and file at finalize.begin) -assume nothing changed git/http when baking because they require hash. -10. WIP -11. allow ANY plugin cus plugin is in environment(docker/VM usually) -bare-metal plugin requires more work and permission (PIP:Pipeline Implicit Parameters) -12. Via baker. -Isolation = how baker (and the whole bake process) is isolated. - ---- - -## Assistant (Sisyphus (Ultraworker) · GLM-5 · 61.4s) - -## New Discovery: Plugin Token/Lock System - -This is a significant design element! Let me clarify: - -### Plugin Coordination via Tokens -``` -Artifact A ──► docker-push ──► Acquire "IMAGE.docker.io/user/app" - │ │ - │ └── Holds token until done - │ - └──► image-scanner ──► Wait for "IMAGE.docker.io/user/app" - │ - └── Resumes when token released -``` - ---- - -## Follow-Up Questions - -### A. Token/Lock System - -1. **Token Naming Convention** - - `"IMAGE."` - is this a convention or built-in? - - Can plugins define custom token names? e.g., `"DEPLOY.production"` - - Is there a reserved namespace? (e.g., `"SYSTEM.*"`) - -2. **Token Scope** - ```yaml - artifact: - - path: "/workspace/app.tar" - publish: - docker-push: - image: "docker.io/user/app:v1" - # Implicitly acquires "IMAGE.docker.io/user/app:v1"? - - - path: "/workspace/app.tar" # Same artifact, different tag - publish: - docker-push: - image: "docker.io/user/app:latest" - # Different token? "IMAGE.docker.io/user/app:latest"? - ``` - - Is token per image URL? Per artifact? Per plugin instance? - - How does system know what token to acquire? - -3. **Token Declaration** - - Does plugin declare required tokens in its schema? - - Or baker infers from plugin config (e.g., `image` field)? - ```yaml - # In plugin definition? - plugin: - docker-push: - "git::...": - tokens: - - "IMAGE.{{ config.image }}" # Template? - ``` - -4. **Deadlock Prevention** - ```yaml - artifact: - - publish: - plugin-a: - token: "RESOURCE.X" # Acquires X, then Y - plugin-b: - token: "RESOURCE.Y" # Acquires Y, then X - ``` - - Does baker detect potential deadlocks? - - Or is this user's responsibility? - -5. **Token Timeout** - - If docker-push hangs forever, image-scanner waits forever? - - Is there a token acquisition timeout? - - Or inherit from plugin's `timeout` setting? - -### B. Plugin Fetch Timing - -6. **Prebake-Stage Plugin Fetch** - ``` - prebake: fetch git/http plugins - bake: (assume plugins unchanged) - finalize.begin: fetch file:// plugins - ``` - - Where are fetched plugins stored? `/workspace/.plugins/`? - - Are they verified against hash before execution? - -7. **file:// Plugin Path** - ```yaml - plugin: - my-plugin: - "file:///workspace/my-plugin.sh": - ``` - - `file://` means plugin is already in build environment? - - Or on baker server (different machine)? - - How does this work with Docker isolation? - -### C. Plugin Runtime Detection - -8. **Detection Logic** - ```yaml - "file:///plugin.sh" → shell - "file:///plugin.rhai" → rhai - "git::github.com/user/plugin@main/plugin.sh" → shell - ``` - - Detected by file extension in URI? - - What if no extension? Default to shell? - - Can user override detection? - ```yaml - plugin: - my-plugin: - "file:///plugin.bin": - runtime: "shell" # Force shell - ``` - -### D. Shell Plugin Interface - -9. **CLI Arguments Structure** - ```bash - # Example invocation? - ./plugin.sh \ - --artifact-path "/workspace/.plugins/docker-push/..." \ - --config '{"image": "docker.io/user/app:latest"}' \ - --secret-token "xxx" - ``` - - What arguments are passed? - - Config as JSON string? Environment variables? Both? - -10. **Environment Variables** - ```bash - ARTIFACT_PATH="/workspace/.plugins/..." - PLUGIN_CONFIG_IMAGE="docker.io/user/app:latest" - SECRET_DOCKERHUB_TOKEN="xxx" - ``` - - What's the naming convention? - - `PLUGIN_CONFIG_*` for config? `SECRET_*` for secrets? - -### E. Progress Reporting - -11. **Progress Variable Semantics** - ``` - build.status = "bake.fetch:3%" - ``` - - Is this a single string variable that updates? - - Or structured: `build.stage = "bake.fetch"`, `build.progress = 3`? - - Can plugin/notification access `{{ build.progress }}`? - -12. **Real-Time Access** - - Can notification's `on_finish_of` access live progress? - - Or only "last final state" at the moment of trigger? - -### F. Bare-Metal Plugin Security (PIP) - -13. **Pipeline Implicit Parameters (PIP)** - - What does this include? - - Permission flags? Sandboxing options? - - Example config? - -14. **Bare-Metal Plugin Isolation** - ```yaml - environment: - builder: "baremetal" - - finalize: - plugin: - docker-push: ... # Needs special handling? - ``` - - How does bare-metal differ from Docker/VM for plugins? - - What restrictions apply? - ---- - -## Summary: Token System Design (Emerging) - -```yaml -# Hypothetical explicit token declaration -plugin: - docker-push: - "git::...": - tokens: - acquire: ["IMAGE.{{ image }}"] # Acquired on start - release: ["IMAGE.{{ image }}"] # Released on finish - # Or implicit from 'image' field? - -artifact: - - path: "/workspace/app.tar" - publish: - docker-push: - image: "docker.io/user/app:latest" - # System infers: acquire "IMAGE.docker.io/user/app:latest" - - image-scanner: - image: "docker.io/user/app:latest" - # System infers: wait for "IMAGE.docker.io/user/app:latest" - # Or explicit: - wait_tokens: ["IMAGE.docker.io/user/app:latest"] -``` - -This is a powerful coordination mechanism! Should tokens be: -- **Implicit**: Inferred from plugin config fields -- **Explicit**: Declared by plugin author in plugin schema -- **Mixed**: Implicit with override capability - -What's your preference? - ---- - -## User - -Too many questions lol. -Maybe answering after actually working on finalize. -Lock/Token is managed by system so dead-lock detection should be easy. - ---- - -## Assistant (Sisyphus (Ultraworker) · GLM-5 · 23.6s) - -Fair point! Let me wrap up with a concise summary. - ---- - -## finalize.yml.tmpl Design Summary - -### Confirmed Decisions - -| Area | Decision | -|------|----------| -| **Version** | `version: "0.0.1"` | -| **Plugin URI** | `git::repo@branch/commit`, `file://`, more protocols later | -| **Plugin runtime** | Auto-detected by extension (`.sh` / `.rhai`) | -| **Plugin config** | Shallow override from `publish:` | -| **Plugin communication** | Shell: CLI args + ENV vars; Rhai: WIP | -| **Plugin isolation** | Via baker environment (Docker/VM = any plugin, bare-metal = PIP) | -| **Fallible** | Same semantics as bake.sh `@fallible` | -| **Notification priority** | Integers 0~INT_MAX, intra-group parallel, inter-group sequential fallback | -| **Notification fallback** | System-level: in-site DB → give up → admin | -| **Empty group** | Warn, don't auto-fix | -| **Artifact ID** | MD5(PUID + PATH), auto-generated if omitted | -| **Artifact path** | Glob → RE, 1 match = 1 artifact | -| **Artifact transfer** | Hard-link/rsync to plugin workspace | -| **Duration format** | ISO 8601 without P: `"7D"`, `"168H"`, or seconds | -| **Typo fix** | `retension` → `retention` | -| **Token/Lock** | System-managed, plugin coordination | - -### WIP / Deferred to Implementation - -- Standalone plugin invocation -- Conditional system (`if:` on artifacts/notifications) -- `on_finish_of` condition syntax -- Variable system (`{{ build.* }}`, `{{ target.* }}`) -- Retry/timeout per notification method -- Plugin token declaration (implicit vs explicit) -- Cleanup policies -- Schema validation - ---- - -Ready to start implementation when you are. Good session! - ---- - -Please read the conversation above and summarize. diff --git a/workshop-baker/src/constant.rs b/workshop-baker/src/constant.rs index f8c9430..95b0b95 100644 --- a/workshop-baker/src/constant.rs +++ b/workshop-baker/src/constant.rs @@ -3,6 +3,7 @@ pub const BOOTSTRAP_SCRIPT_PATH: &str = "/bootstrap.sh"; pub const BAKE_SCRIPT_PATH: &str = "bake.sh"; pub const PIPELINE_SKIP_ERRORCODE: u8 = 233; pub const TRIVIAL_SCRIPT_NAME: &str = "BAKE_TRIVIAL"; -pub const FINALIZE_PLUGIN_MANIFEST: &str = "manifest.yml"; -pub const FINALIZE_SHELL_ENVPREFIX: &str = "HBW_ARG_"; +pub const FINALIZE_PLUGIN_MANIFEST: &str = "manifest"; +pub const FINALIZE_PLUGIN_MANIFEST_EXTENSION: &[&str] = &[".yml", ".yaml"]; +pub const FINALIZE_SHELL_ENVPREFIX: &str = "HBW_ARG"; pub const FINALIZE_STANDALONE_PLUGIN_PATH: &str = "/tmp/baker/plugin"; diff --git a/workshop-baker/src/engine/executor.rs b/workshop-baker/src/engine/executor.rs index 940cd18..ffbdf4e 100644 --- a/workshop-baker/src/engine/executor.rs +++ b/workshop-baker/src/engine/executor.rs @@ -4,6 +4,7 @@ use std::process::Stdio; use tokio::io::AsyncReadExt; use tokio::process::Command; use tokio::sync::mpsc; +use tokio::time::timeout; use super::pm::PackageManager; use super::types::{ @@ -66,6 +67,10 @@ impl super::types::Engine { ctx.working_dir.join(script_path) }; + if !abs_path.exists() { + return Err(ExecutionError::ScriptNotFound(abs_path)) + } + let mut command = Command::new(ctx.shell.as_str()); command.arg("-c"); command.arg(abs_path); @@ -210,9 +215,48 @@ impl super::types::Engine { } }); - let status = child.wait().await.map_err(|e| { - ExecutionError::IoError(format!("Failed to wait for process: {}", e)) - })?; + let status = if let Some(timeout_duration) = ctx.timeout { + match timeout(timeout_duration, child.wait()).await { + Ok(Ok(status)) => status, + Ok(Err(e)) => { + return Err(ExecutionError::IoError(format!( + "Failed to wait for process: {}", + e + ))) + } + Err(_) => { + log::warn!( + "Task {} timed out after {:?}, killing process", + ctx.task_id, + timeout_duration + ); + if let Some(tx) = event_tx { + let _ = tx + .send(ExecutionEvent::TaskFailed { + task_id: ctx.task_id.clone(), + error: "Timeout".to_string(), + }) + .await; + } + let _ = child.kill().await; + drop(stdout_tx); + drop(stderr_tx); + let _ = stdout_handle.await; + let _ = stderr_handle.await; + while let Some(data) = stdout_rx.recv().await { + stdout_buf.push_str(&data); + } + while let Some(data) = stderr_rx.recv().await { + stderr_buf.push_str(&data); + } + return Err(ExecutionError::Timeout); + } + } + } else { + child.wait().await.map_err(|e| { + ExecutionError::IoError(format!("Failed to wait for process: {}", e)) + })? + }; drop(stdout_tx); drop(stderr_tx); diff --git a/workshop-baker/src/finalize.rs b/workshop-baker/src/finalize.rs index ccbb3ae..92e81f4 100644 --- a/workshop-baker/src/finalize.rs +++ b/workshop-baker/src/finalize.rs @@ -37,11 +37,13 @@ pub async fn finalize( }; if ctx.standalone { // In standalone mode, finalize should download plugin by itself + log::debug!("Downloading {} plugins", finalize.plugin.len()); let plugin_path = PathBuf::from(FINALIZE_STANDALONE_PLUGIN_PATH); - if plugin_path.exists() { - std::fs::remove_dir_all(&plugin_path).map_err(FinalizeError::IoError)?; - } + // if plugin_path.exists() { + // std::fs::remove_dir_all(&plugin_path).map_err(FinalizeError::IoError)?; + // } for (name, data) in finalize.plugin.iter_mut() { + log::info!("Downloading plugin {}", name); let (source, config) = data.iter_mut().next().unwrap_or_else(|| { panic!( "Internal Error: Plugin {} has no source configured, this should have been caught by validation!", @@ -58,6 +60,7 @@ pub async fn finalize( } // Register Plugins + log::info!("Registering {} plugins", finalize.plugin.len()); let registered_plugins = plugin::register(finalize.plugin)?; // Early hook @@ -81,5 +84,7 @@ pub async fn finalize( ) .await?; - todo!(); + // todo!(); + // For testing purpose only + Ok(()) } diff --git a/workshop-baker/src/finalize/config.rs b/workshop-baker/src/finalize/config.rs index 762cd3d..6840696 100644 --- a/workshop-baker/src/finalize/config.rs +++ b/workshop-baker/src/finalize/config.rs @@ -300,3 +300,75 @@ impl FinalizeConfig { } } } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_deserialize_hooks_full() { + let yaml = r#" +version: "0.0.1" +hooks: + early: + - name: "test-early" + command: "echo early" + timeout: 10 + late: + - name: "test-late" + command: "plugin:test-plugin" + working_dir: "/tmp" +"#; + let config: FinalizeConfig = serde_yaml::from_str(yaml).unwrap(); + assert!(config.hooks.is_some()); + let hooks = config.hooks.unwrap(); + let early = hooks.early.as_ref().unwrap(); + let late = hooks.late.as_ref().unwrap(); + assert_eq!(early.len(), 1); + assert_eq!(late.len(), 1); + assert_eq!(early[0].command, "echo early"); + assert_eq!(late[0].command, "plugin:test-plugin"); + assert_eq!(late[0].working_dir, Some("/tmp".to_string())); + } + + #[test] + fn test_deserialize_hooks_missing() { + let yaml = r#" +version: "0.0.1" +"#; + let config: FinalizeConfig = serde_yaml::from_str(yaml).unwrap(); + assert!(config.hooks.is_none()); + } + + #[test] + fn test_deserialize_hooks_timeout_float() { + let yaml = r#" +version: "0.0.1" +hooks: + early: + - name: "float-timeout" + command: "echo ok" + timeout: 10.5 +"#; + let config: FinalizeConfig = serde_yaml::from_str(yaml).unwrap(); + let hooks = config.hooks.unwrap(); + let early = hooks.early.unwrap(); + assert_eq!(early[0].timeout_ms, 10500); + } + + #[test] + fn test_deserialize_hooks_timeout_string() { + let yaml = r#" +version: "0.0.1" +hooks: + early: + - name: "string-timeout" + command: "echo ok" + timeout: "30s" +"#; + let config: FinalizeConfig = serde_yaml::from_str(yaml).unwrap(); + let hooks = config.hooks.unwrap(); + let early = hooks.early.unwrap(); + assert_eq!(early[0].timeout_ms, 30000); + } +} diff --git a/workshop-baker/src/finalize/error.rs b/workshop-baker/src/finalize/error.rs index 0f3e407..a57188d 100644 --- a/workshop-baker/src/finalize/error.rs +++ b/workshop-baker/src/finalize/error.rs @@ -73,5 +73,8 @@ pub enum PluginError { name: String, url: String, reason: String, - } + }, + + #[error("Plugin manifest {0}/manifest.yml not found.")] + ManifestNotFound(PathBuf), } diff --git a/workshop-baker/src/finalize/plugin.rs b/workshop-baker/src/finalize/plugin.rs index 8d402e6..884454e 100644 --- a/workshop-baker/src/finalize/plugin.rs +++ b/workshop-baker/src/finalize/plugin.rs @@ -1,10 +1,10 @@ -use crate::constant::FINALIZE_PLUGIN_MANIFEST; +use crate::constant::{FINALIZE_PLUGIN_MANIFEST, FINALIZE_PLUGIN_MANIFEST_EXTENSION}; use crate::finalize::RawPluginMap; use crate::finalize::plugin::metadata::{PluginMetadata, PluginType}; use crate::{EventSender, ExecutionContext, finalize::error::PluginError}; use async_trait::async_trait; use std::collections::HashMap; -use std::path::PathBuf; +use std::path::{PathBuf, Path}; mod dylib; mod internal; @@ -91,13 +91,20 @@ fn read_manifest(path: Option, name: &str) -> Result {}, + Err(e) => return Err(e), } - })?; + } + return Err(PluginError::ManifestNotFound(path)); + }; + + log::debug!("[plugin:{}] Parsing manifest content", name); let mut metadata: PluginMetadata = serde_yaml::from_str(manifest_content.as_str()).map_err(|e| PluginError::PluginUnavailable { name: name.to_string(), reason: format!("Failed to parse plugin manifest: {}", e), @@ -106,6 +113,21 @@ fn read_manifest(path: Option, name: &str) -> Result Result { + let manifest_path = path.join(format!("{}{}", FINALIZE_PLUGIN_MANIFEST,suffix)); + log::debug!("[plugin:{}] Reading manifest from {}", name, manifest_path.display()); + if !manifest_path.exists() { + return Err(PluginError::ManifestNotFound(manifest_path)); + } + + std::fs::read_to_string(&manifest_path).map_err(|e| { + PluginError::PluginUnavailable { + name: name.to_string(), + reason: format!("Failed to read plugin manifest from {}: {}", manifest_path.display(), e.to_string()), + } + }) +} + /// Determines the plugin type based on the manifest type or file extension. /// /// The function first returns `original` if it's not [`PluginType::Unknown`]. @@ -183,11 +205,13 @@ pub fn register(external_plugins: RawPluginMap) -> Result, #[serde( default = "default_timeout_ms", - deserialize_with = "deserialize_duration_ms" + deserialize_with = "deserialize_duration_ms", + rename = "timeout" )] pub timeout_ms: u64, #[serde(default)] diff --git a/workshop-baker/tests/engine_integration.rs b/workshop-baker/tests/engine_integration.rs index 645acdc..7dfaa45 100644 --- a/workshop-baker/tests/engine_integration.rs +++ b/workshop-baker/tests/engine_integration.rs @@ -27,6 +27,7 @@ fn create_context() -> ExecutionContext { shell: "bash".to_string(), dry_run: false, privileged: false, + standalone: false, } } @@ -145,8 +146,8 @@ async fn test_execute_sets_ws_taskid() { let result = executor.execute(cmd, &ctx, &None).await.unwrap(); - assert!(result.stdout.contains("WS_TASKID=task-abc-123")); - assert!(result.stdout.contains("WS_PIPELINE=pipeline-xyz")); + assert!(result.stdout.contains("HBW_TASKID=task-abc-123")); + assert!(result.stdout.contains("HBW_PIPELINE=pipeline-xyz")); } #[tokio::test] @@ -159,7 +160,7 @@ async fn test_execute_sets_ws_user_info() { let result = executor.execute(cmd, &ctx, &None).await.unwrap(); - assert!(result.stdout.contains("WS_USERNAME=testuser")); + assert!(result.stdout.contains("HBW_USERNAME=testuser")); } #[tokio::test] @@ -174,6 +175,7 @@ async fn test_execute_respects_working_directory() { timeout: Some(Duration::from_secs(30)), cgroup_path: None, shell: "bash".to_string(), + standalone: false, ..Default::default() }; diff --git a/workshop-baker/tests/finalize_hook_integration.rs b/workshop-baker/tests/finalize_hook_integration.rs new file mode 100644 index 0000000..fcd594f --- /dev/null +++ b/workshop-baker/tests/finalize_hook_integration.rs @@ -0,0 +1,179 @@ +//! Integration tests for finalize hook execution +//! +//! These tests verify that shell hooks and plugin hooks execute correctly +//! through the finalize stage hook executor. + +use std::collections::HashMap; +use std::time::Duration; +use workshop_baker::engine::{ExecutionContext, ExecutionEvent}; +use workshop_baker::finalize::stage::hook::hook; +use workshop_baker::finalize::plugin::{PluginMap}; +use workshop_baker::types::command::CustomCommand; + +fn create_test_context() -> ExecutionContext { + ExecutionContext { + task_id: "test-hook-1".to_string(), + pipeline_name: "test-pipeline".to_string(), + username: "testuser".to_string(), + working_dir: std::env::temp_dir(), + env_vars: HashMap::new(), + timeout: Some(Duration::from_secs(30)), + cgroup_path: None, + shell: "bash".to_string(), + dry_run: false, + privileged: false, + standalone: false, + } +} + +fn create_test_event_channel() -> (Option>, tokio::sync::mpsc::Receiver) { + let (tx, rx) = tokio::sync::mpsc::channel(64); + (Some(tx), rx) +} + +#[tokio::test] +async fn test_shell_hook_success() { + let ctx = create_test_context(); + let (event_tx, _event_rx) = create_test_event_channel(); + let plugins = PluginMap::new(); + + let cmd = CustomCommand { + name: "echo-test".to_string(), + command: "echo 'hook-ok'".to_string(), + environment: None, + working_dir: None, + timeout_ms: 5000, + argument: serde_yaml::Value::Null, + }; + + let result = hook(Some(&vec![cmd]), &ctx, event_tx, &plugins).await; + assert!(result.is_ok(), "Shell hook should succeed: {:?}", result); +} + +#[tokio::test] +async fn test_shell_hook_failure_propagates() { + let ctx = create_test_context(); + let (event_tx, _event_rx) = create_test_event_channel(); + let plugins = PluginMap::new(); + + let cmd = CustomCommand { + name: "false-cmd".to_string(), + command: "false".to_string(), + environment: None, + working_dir: None, + timeout_ms: 5000, + argument: serde_yaml::Value::Null, + }; + + let result = hook(Some(&vec![cmd]), &ctx, event_tx, &plugins).await; + assert!(result.is_err(), "Shell hook failure should propagate"); +} + +#[tokio::test] +async fn test_hook_timeout_kills_long_command() { + let ctx = create_test_context(); + let (event_tx, _event_rx) = create_test_event_channel(); + let plugins = PluginMap::new(); + + let cmd = CustomCommand { + name: "sleep-forever".to_string(), + command: "sleep 10".to_string(), + environment: None, + working_dir: None, + timeout_ms: 500, // 0.5 seconds + argument: serde_yaml::Value::Null, + }; + + let result = hook(Some(&vec![cmd]), &ctx, event_tx, &plugins).await; + assert!(result.is_err(), "Hook should fail due to timeout"); +} + +#[tokio::test] +async fn test_hook_working_dir_applied() { + let ctx = create_test_context(); + let (event_tx, _event_rx) = create_test_event_channel(); + let plugins = PluginMap::new(); + + let cmd = CustomCommand { + name: "pwd-check".to_string(), + command: "pwd".to_string(), + environment: None, + working_dir: Some("/tmp".to_string()), + timeout_ms: 5000, + argument: serde_yaml::Value::Null, + }; + + // We verify by checking the hook returns Ok — the working_dir is passed to + // Engine::execute_command_with_delta. Full stdout verification would require + // mocking the Engine or using a more intrusive test setup. + let result = hook(Some(&vec![cmd]), &ctx, event_tx, &plugins).await; + assert!(result.is_ok(), "Hook with working_dir should succeed"); +} + +#[tokio::test] +async fn test_no_hooks_returns_ok() { + let ctx = create_test_context(); + let (event_tx, _event_rx) = create_test_event_channel(); + let plugins = PluginMap::new(); + + let result = hook(None, &ctx, event_tx, &plugins).await; + assert!(result.is_ok(), "No hooks should return Ok immediately"); +} + +#[tokio::test] +async fn test_multiple_hooks_execute_sequentially() { + let ctx = create_test_context(); + let (event_tx, _event_rx) = create_test_event_channel(); + let plugins = PluginMap::new(); + + let cmds = vec![ + CustomCommand { + name: "hook-1".to_string(), + command: "true".to_string(), + environment: None, + working_dir: None, + timeout_ms: 5000, + argument: serde_yaml::Value::Null, + }, + CustomCommand { + name: "hook-2".to_string(), + command: "true".to_string(), + environment: None, + working_dir: None, + timeout_ms: 5000, + argument: serde_yaml::Value::Null, + }, + ]; + + let result = hook(Some(&cmds), &ctx, event_tx, &plugins).await; + assert!(result.is_ok(), "Multiple hooks should execute sequentially and succeed"); +} + +#[tokio::test] +async fn test_first_failure_stops_remaining_hooks() { + let ctx = create_test_context(); + let (event_tx, _event_rx) = create_test_event_channel(); + let plugins = PluginMap::new(); + + let cmds = vec![ + CustomCommand { + name: "failing-hook".to_string(), + command: "false".to_string(), + environment: None, + working_dir: None, + timeout_ms: 5000, + argument: serde_yaml::Value::Null, + }, + CustomCommand { + name: "never-reached".to_string(), + command: "true".to_string(), + environment: None, + working_dir: None, + timeout_ms: 5000, + argument: serde_yaml::Value::Null, + }, + ]; + + let result = hook(Some(&cmds), &ctx, event_tx, &plugins).await; + assert!(result.is_err(), "First failing hook should stop execution"); +} diff --git a/workshop-baker/tests/finalize_pipeline_integration.rs b/workshop-baker/tests/finalize_pipeline_integration.rs new file mode 100644 index 0000000..4c3cb28 --- /dev/null +++ b/workshop-baker/tests/finalize_pipeline_integration.rs @@ -0,0 +1,143 @@ +//! Integration tests for the full finalize pipeline +//! +//! These tests verify the complete finalize workflow from YAML parsing +//! through hook execution, using real files and the public `finalize()` entry point. + +use std::collections::HashMap; +use std::time::Duration; +use workshop_baker::engine::{ExecutionContext, ExecutionEvent}; +use workshop_baker::finalize::finalize; +use workshop_baker::cli::Cli; + +fn create_test_context() -> ExecutionContext { + ExecutionContext { + task_id: "test-finalize-1".to_string(), + pipeline_name: "test-pipeline".to_string(), + username: "testuser".to_string(), + working_dir: std::env::temp_dir(), + env_vars: HashMap::new(), + timeout: Some(Duration::from_secs(30)), + cgroup_path: None, + shell: "bash".to_string(), + dry_run: false, + privileged: false, + standalone: false, + } +} + +fn create_test_cli() -> Cli { + Cli { + username: "testuser".to_string(), + pipeline: "test-pipeline".to_string(), + build_id: "test-build-1".to_string(), + dry_run: false, + command: None, + } +} + +fn create_test_event_channel() -> (Option>, tokio::sync::mpsc::Receiver) { + let (tx, rx) = tokio::sync::mpsc::channel(64); + (Some(tx), rx) +} + +#[tokio::test] +async fn test_full_finalize_with_early_late_hooks() { + let temp_dir = std::env::temp_dir(); + let finalize_path = temp_dir.join("test_finalize.yml"); + + let yaml_content = r#" +version: "0.0.1" +hooks: + early: + - name: "early-echo" + command: "echo early-hook-ok" + timeout: 5 + late: + - name: "late-echo" + command: "echo late-hook-ok" + timeout: 5 +"#; + + std::fs::write(&finalize_path, yaml_content).expect("Failed to write test finalize.yml"); + + let cli = create_test_cli(); + let mut ctx = create_test_context(); + let (event_tx, mut event_rx) = create_test_event_channel(); + + // Spawn an event consumer so the channel does not block + let consumer_handle = tokio::spawn(async move { + while let Some(_event) = event_rx.recv().await { + // Events are consumed but not validated in this test + } + }); + + let result = finalize(&finalize_path, &cli, &mut ctx, event_tx).await; + + let _ = consumer_handle.await; + std::fs::remove_file(&finalize_path).ok(); + + assert!(result.is_ok(), "Full finalize pipeline should succeed: {:?}", result); +} + +#[tokio::test] +async fn test_full_finalize_with_empty_hooks() { + let temp_dir = std::env::temp_dir(); + let finalize_path = temp_dir.join("test_finalize_empty.yml"); + + let yaml_content = r#" +version: "0.0.1" +"#; + + std::fs::write(&finalize_path, yaml_content).expect("Failed to write test finalize.yml"); + + let cli = create_test_cli(); + let mut ctx = create_test_context(); + let (event_tx, mut event_rx) = create_test_event_channel(); + + let consumer_handle = tokio::spawn(async move { + while let Some(_event) = event_rx.recv().await {} + }); + + let result = finalize(&finalize_path, &cli, &mut ctx, event_tx).await; + + let _ = consumer_handle.await; + std::fs::remove_file(&finalize_path).ok(); + + assert!(result.is_ok(), "Finalize with empty hooks should succeed: {:?}", result); +} + +#[tokio::test] +async fn test_full_finalize_with_failing_early_hook() { + let temp_dir = std::env::temp_dir(); + let finalize_path = temp_dir.join("test_finalize_fail.yml"); + + let yaml_content = r#" +version: "0.0.1" +hooks: + early: + - name: "failing-hook" + command: "false" + timeout: 5 + late: + - name: "never-reached" + command: "echo should-not-run" + timeout: 5 +"#; + + std::fs::write(&finalize_path, yaml_content).expect("Failed to write test finalize.yml"); + + let cli = create_test_cli(); + let mut ctx = create_test_context(); + let (event_tx, mut event_rx) = create_test_event_channel(); + + let consumer_handle = tokio::spawn(async move { + while let Some(_event) = event_rx.recv().await {} + }); + + let result = finalize(&finalize_path, &cli, &mut ctx, event_tx).await; + + let _ = consumer_handle.await; + std::fs::remove_file(&finalize_path).ok(); + + assert!(result.is_err(), "Finalize with failing early hook should return an error"); +}