Add os-adr Eval C: ambiguity-ladder discrimination eval
Eval B wording reached 8/8 (sonnet) / 7/8 (haiku) on its run-set after five iterations; subsequent W3 stability testing confirmed intermittent flicker but acceptable performance. As Eval B is now contaminated by iteration, we need a held-out measurement set to assess generalization to new projects and cues. Eval C measures whether learned behavior generalizes across decreasing cue explicitness (explicit → moderate → conceptual framing) and, critically, whether the model correctly avoids false positives (over-triggering) when no Accepted ADR is in play. Six paired scenarios (positive/negative at each level) run against a job-execution domain fixture; a frozen reserve-set (notifications domain) becomes the measurement set if anyone tunes wording against the run-set. New: `plugins/os-adr/eval-c/` — bin/ (run/check/self-test/sandbox scripts), fixture/ (taskq async job queue, 6 ADRs, trigger-phrased CLAUDE.md), scenarios/ (6 run-set), scenarios-reserve/ (6 reserve- set), README.md (measurement discipline + design), judge-rubric.md. Model-free self-test passing. First real grid run is a pending decision.
This commit is contained in:
jared
parent
7523663889
commit
d3be4c5ece
|
|
@ -102,6 +102,8 @@ to those two and fix the stale doc.
|
|||
- Eval B harness (2026-07-03, OpenSpec change `add-os-adr-eval-b-harness`): `plugins/os-adr/eval-b/` — held-out unprompted-behavior eval (requirements 4–5). 7 scenarios (W1–W3 write-trigger, R1–R4 retrieval) authored from the frozen shapes in `docs/adr-system/06-eval-scenarios.md`; dedicated Ruby webhook-relay fixture with a 6-ADR history (Superseded pair + near-miss distractors, generated via the plugin's own CLIs); R4's one-hop graph reach uses a real `graphify update` AST build (model-free, rebuilt via `eval-b/bin/build-fixture-graph`, never committed). **Headless-only runner** (`eval-b/bin/run` — fresh `claude -p` per rep, cwd = sandbox, so the real SessionStart hook fires; in-session subagents are invalid here, unlike Eval A) and a two-axis deterministic-first checker (`eval-b/bin/check`): axis (a) unprompted consultation, mechanical from transcript tool_use blocks; axis (b) correct-ADR citation (R1–R4) or new-ADR-file with a narrow frozen-rubric haiku judge fallback (W1–W3, `judge-rubric.md`, stubbable via `ADR_EVAL_B_JUDGE_CMD`). `R4-nograph` is the graph-degradation variant (expected FAIL). Self-tested both directions model-free via `eval-b/bin/self-test`. **Scenario Task blocks are held-out — never run them informally.** Procedure: `plugins/os-adr/eval-b/README.md`. Grid run 2026-07-03 (1 rep/cell): haiku 0/8 PASS (never unprompted-consults the ADR system in any scenario — a real gap, not a harness defect); sonnet 5/8 PASS (fails W3 — doesn't propose recording the decision; fails R1 — misses the direct-conflict retrieval). `R4-nograph` FAILed on both tiers as expected (degradation check, only meaningful paired with an R4 PASS — sonnet has one, haiku doesn't). Full results + observations (prompting-issue hypothesis, open question on whether in-session subagents could ever validly substitute for part of this measurement) written to the vault: [[os-adr-eval-b-grid-results-and-observations]].
|
||||
- **Initial iteration complete (2026-07-03):** plugin build, migration pilot, Eval A grid (clean pass), and Eval B grid (baseline captured above) are all done — this closes the first pass on os-adr. Follow-on work is deliberately deferred to future sessions, not in-flight.
|
||||
- **Wording experiment complete (2026-07-04):** the follow-up `/autoresearch` loop over Eval B wording ran (5 iterations, checker/fixtures/scenarios/rubric frozen; fixture CLAUDE.md declared a wording surface upfront) and closed the gap — final full grid **sonnet 8/8, haiku 7/8** (from 5/8 / 0/8 baseline; haiku's one miss is a W3 axis-b judge-boundary flicker). Winning wording shipped in `hooks/session_start.py` (PRESENT_NOTE), find/create SKILL.mds, and `eval-b/fixture/project/CLAUDE.md` (new — the trigger-phrased "Architecture decisions" section, the template for real-project adoption). Confirmed mechanisms: trigger-conditioned when→then phrasing beats inventory phrasing on both tiers; each rule must live where its precondition is visible (the reversal→supersede rule in the find skill's act-on-findings step fixed W3); lower tiers need *mechanical* triggers ("before your first edit to any existing file → run `/os-adr:find` on those paths; additions count") — semantic triggers ("architecture-level choice") only reach sonnet. Open: channel ablation never run (hook vs CLAUDE.md redundancy unknown); R4-nograph now passes both tiers, so the graph-degradation check no longer differentiates. Full hypothesis→result mapping: vault note [[os-adr-eval-b-wording-experiment-hypotheses]]. **Before designing or running any autoresearch eval, Read `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md`.**
|
||||
- **Eval B W3 stability check (2026-07-06):** 7 headless reps of haiku × W3 (all reps counted, including two the runner initially excluded, to avoid peeking bias): axis (a) consultation 7/7 PASS; axis (b) recording-offer ~5/7 (~75–85%) — one FAIL plausibly infra, one a genuine behavioral miss (consulted, then asked a clarifying question instead of unconditionally proposing the superseding ADR; the conditional-phrasing failure mode iteration 3 fixed on sonnet, improved but not eliminated on haiku). Verdict: intermittent flicker, not a hard gap — grid claim stays **haiku 7/8**, W3 characterized as an ~1-in-4/5 axis-b miss. Haiku cleared for daily use with that caveat. Recorded in the vault good-enough gate: [[os-adr-eval-b-wording-experiment-hypotheses]].
|
||||
- **Eval C harness built, NOT yet run (2026-07-06):** `plugins/os-adr/eval-c/` — ambiguity-ladder DISCRIMINATION eval (held-out; Eval B is contaminated by the wording loop). 3 levels (explicit → moderate → conceptual framing) × paired positive/negative scenarios; run-set (6, job-execution domain) + frozen reserve-set (6, notifications domain — becomes the measurement set if anyone ever tunes wording against the run-set). New Python fixture (`taskq` async job queue, 6-ADR history generated via the plugin's own CLIs, trigger-phrased CLAUDE.md section copied from eval-b). Scoring: positives = both axes; negatives FAIL **only on unneeded ADR creation** — consultation and truthful ADR citation are informational (`A:yes/no`, `cited-adr:yes/no`), never FAILs, because the shipped wording endorses cheap finds and the negatives sit deliberately in ADR-covered domains (two earlier negative designs that punished instruction-compliant behavior were caught in review and redesigned/fixed). Model-free `bin/self-test` green, including a truthful-citation-must-PASS guard. **Scenario Task blocks are held-out — never run informally.** First real grid run is a pending decision. Methodology: vault notes [[eval-methodology-ladder]] (per-level pass bars, reserve discipline) and [[eval-methodology-irl-feedback-loop]] (post-rollout session-audit backlog); vault also gained an `eval-results` note type + `_templates/eval-results.md` (2026-07-06).
|
||||
- **Remaining (locked rollout order):** real-project migration/adoption one at a time via `/os-adr:migrate` — pilot projects first, then the cc-os retrofit (its 19-ADR monolithic file was deliberately excluded from the pilot), then wider. When onboarding real projects, add the trigger-phrased "Architecture decisions" CLAUDE.md section (copy from `eval-b/fixture/project/CLAUDE.md`; candidate: emit it from `/os-adr:init`/`migrate`).
|
||||
- **Resolved (2026-07-04):** os-adr's skills failed to register on first install due to stale plugin caches — the cache that was installed on 2026-07-03 17:21:48 was missing `hooks/hooks.json`, `bin/adr-detect`, `bin/adr-find`, `bin/adr-migrate`. Root cause: unknown, but the caches were restored via plugin uninstall/reinstall. A parallel issue affected os-doc-hygiene (cache retained deleted `commands/` directory). Fixed by manually refreshing both caches, then created `bin/refresh-plugins` automation to prevent future drift — see "Editing a local plugin (cache refresh)" subsection for refresh procedure. Investigation result: stale caches (not manifest-naming issues) were the cause; slash command registration works correctly once caches are fresh.
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,289 @@
|
|||
# os-adr Eval C — ambiguity-ladder discrimination eval (held-out)
|
||||
|
||||
_Last updated: 2026-07-06 — harness built and self-tested (all scenarios pass model-free checks)._
|
||||
|
||||
Measures **generalization across decreasing cue explicitness** — does the model correctly
|
||||
trigger `/os-adr:find` or `/os-adr:create` as architectural decision vocabulary becomes
|
||||
less explicit, and equally important, **does it correctly NOT trigger when no decided
|
||||
constraint is in play**? This is the discrimination challenge: both false positives
|
||||
(over-trigger) and false negatives (miss the decision) are failures.
|
||||
|
||||
Eval B (unprompted-behavior baseline) optimized its wording against its own grid for 5
|
||||
iterations (2026-07-04), so its 8/8 (sonnet) / 7/8 (haiku) are training-set scores. Eval C
|
||||
measures whether that learned behavior generalizes to **different project, different target
|
||||
ADRs, and most critically, paired negative scenarios that ensure the model isn't just learning
|
||||
"always trigger"**.
|
||||
|
||||
> **HELD-OUT — read this first.** The `## Task` blocks in `scenarios/*.md` and
|
||||
> `scenarios-reserve/*.md` must never be pasted into an interactive session or informally
|
||||
> "tried out". The only permitted non-grid execution is `bin/self-test`, which fabricates
|
||||
> transcripts and never uses the Task blocks. Informal trials contaminate the measurement.
|
||||
|
||||
## Measurement discipline
|
||||
|
||||
This eval is a **frozen measurement pass** — it runs once with a baseline grid. The moment
|
||||
anyone iterates wording against the run-set results, this set is contaminated; future wording
|
||||
work **must switch to the reserve-set** for measurement. See "Reserve set protection" below.
|
||||
|
||||
## Design: ambiguity ladder with paired positive/negative
|
||||
|
||||
### The ladder (3 levels)
|
||||
|
||||
Each level decreases explicit architectural decision cues:
|
||||
|
||||
- **Level 1 (Explicit):** File paths named, implementation vocabulary explicit ("timeout",
|
||||
"retry", "backoff", "exponential"), decision nature overt.
|
||||
- **Level 2 (Moderate):** Relevant vocabulary present but no file paths or explicit decision
|
||||
framing; model must recognize architecture-level significance.
|
||||
- **Level 3 (Conceptual):** Intent-framed only (what outcome is needed), zero ADR vocabulary,
|
||||
no file paths. Model must recognize that this is an architectural concern.
|
||||
|
||||
### Paired positive/negative at every level
|
||||
|
||||
For each level, **two scenarios that differ only in whether a decided constraint is in play**:
|
||||
|
||||
- **Positive (P)**: A decided constraint IS applicable → consulting `/os-adr:find` and/or
|
||||
recording with `/os-adr:create` is correct.
|
||||
- **Negative (N)**: Matched framing (same vocabulary level, same form — question or intent)
|
||||
but NO Accepted ADR constrains the task → consulting/creating an ADR is an over-trigger.
|
||||
|
||||
The negative scenario is the discrimination test: wording tuned only for sensitivity ("always
|
||||
trigger") will fail negatives. Precision (correct non-triggering) is as important as recall.
|
||||
|
||||
### Non-monotonic difficulty expected
|
||||
|
||||
Difficulty does NOT increase uniformly L1 → L2 → L3. An explicit cue (L1) might be harder to
|
||||
miss than a conceptual one (L3), depending on model tier and wording. Per `running-autoresearch-skill-evals.md`, each level is validated independently and results reported in a per-level grid, not collapsed to a single threshold.
|
||||
|
||||
## Two target ADR pairs, independent measurement
|
||||
|
||||
**Run-set** (6 scenarios, frozen for measurement):
|
||||
- Target ADR 1: 0002 — "Job timeouts after 5 minutes with exponential backoff, max 3 attempts" (Accepted)
|
||||
- Target ADR 2: 0001 → 0002 (Superseded) for distractor validation
|
||||
- Scenarios: P1-L1, N1-L1, P2-L2, N2-L2, P3-L3, N3-L3 (all about job execution timeout/retry)
|
||||
|
||||
**Reserve-set** (6 scenarios, held until run-set is contaminated by wording iteration):
|
||||
- Target ADR 2: 0003 — "Route all job notifications through central NotificationService" (Accepted)
|
||||
- Scenarios: P4-L1, N4-L1, P5-L2, N5-L2, P6-L3, N6-L3 (all about notification routing)
|
||||
|
||||
Both sets are valid measurement surfaces; the split is for the protection protocol, not
|
||||
quality difference.
|
||||
|
||||
## Scenario framing discipline (critical)
|
||||
|
||||
The CLAUDE.md instruction carried verbatim in the fixture has two triggers:
|
||||
- Semantic: "architecture-level choice → run `/os-adr:find`"
|
||||
- Mechanical: "before your first edit to any existing source/config file → run `/os-adr:find`"
|
||||
|
||||
**Negative scenarios avoid the mechanical trigger**: they're framed as pure questions or new
|
||||
module designs, so trigger #2 never fires. Then "no ADR touch" is a clean, valid over-trigger
|
||||
signal. If a negative scenario edits an existing file, the over-trigger criteria shift to
|
||||
"created an ADR claiming a phantom constraint," requiring judge fallback.
|
||||
|
||||
## Fixture
|
||||
|
||||
**Project:** `fixture/project/` — small Python async job queue system (`taskq`).
|
||||
|
||||
**ADR history:** 6 ADRs (2026-02-01 to 2026-03-01), generated with plugin CLIs:
|
||||
- 0001: "Job timeout at 10s, simple retry" (Superseded)
|
||||
- 0002: "Job timeout at 5 min, exponential backoff, max 3 attempts" (Accepted, supersedes 0001)
|
||||
- 0003: "Route notifications through central service" (Accepted)
|
||||
- 0004: "Persist queue to SQLite" (Accepted)
|
||||
- 0005: "Failed jobs escalate to human review queue" (Accepted)
|
||||
- 0006: "All job state transitions logged to audit trail" (Accepted)
|
||||
|
||||
Each ADR has `affected-paths` set (e.g., 0002 → [taskq/execution.py, taskq/queue.py]) so
|
||||
`bin/adr-find` can validate path-matching behavior if needed. `.graphifyignore` excludes
|
||||
markdown, so any future graph build (not part of this eval) would be pure-AST / reproducible.
|
||||
|
||||
## Two-axis scoring
|
||||
|
||||
- **Axis (a) — consultation trigger**
|
||||
- **Positive scenarios**: mechanical PASS iff any `tool_use` block invoked an `os-adr:*`
|
||||
skill, `bin/adr-*` CLI, or touched `docs/adr/`.
|
||||
- **Negative scenarios**: INFORMATIONAL ONLY (recorded as `A:yes` or `A:no`, does NOT
|
||||
determine pass/fail). Consultation is acceptable per CLAUDE.md's "cheap, deterministic CLI
|
||||
call" endorsement. The discrimination measurement is not whether the model consulted, but
|
||||
whether it **recorded a false decision** or **asserted a non-existent constraint**.
|
||||
- Hook context (`SessionStart [os-adr]` event) is a precondition: missing hook → FAIL both
|
||||
axes.
|
||||
|
||||
- **Axis (b) — specific correctness**
|
||||
- **Positive scenarios** (write-trigger):
|
||||
- Mechanical PASS when a new `docs/adr/0007-*.md` exists and its text matches the
|
||||
scenario's topic pattern (regex per scenario class).
|
||||
- Otherwise: judge fallback (haiku, `judge-rubric.md`, reads only final message;
|
||||
stubbed via `ADR_EVAL_C_JUDGE_CMD` for self-tests).
|
||||
- **Negative scenarios** (non-decisional tasks, investigate/test/debug framing):
|
||||
- Mechanical FAIL if a new ADR file was created (over-trigger: creating a decision record
|
||||
for a non-decisional task). This is the ONLY fail condition.
|
||||
- Citation of existing ADRs in the final message is recorded as informational
|
||||
`cited-adr:yes/no`, never a FAIL — the negatives sit deliberately in ADR-covered
|
||||
domains, and truthfully citing a governing ADR while documenting/testing/auditing is
|
||||
correct behavior. Separating true citations from false constraint claims needs
|
||||
semantic judgment; a keyword match fails correct answers.
|
||||
- Otherwise: PASS.
|
||||
|
||||
**Overall PASS**:
|
||||
- **Positive scenarios**: both axes PASS.
|
||||
- **Negative scenarios**: axis (b) PASS only (axis a is informational, doesn't affect pass/fail).
|
||||
|
||||
TSV format: `scenario model A:yes|no-or-PASS|FAIL B:PASS|FAIL PASS|FAIL reasons`
|
||||
|
||||
## Running the evaluation
|
||||
|
||||
### Single cell (manual test)
|
||||
|
||||
```bash
|
||||
eval-c/bin/sandbox P2-L2-execution /tmp/adr-eval-c-test
|
||||
eval-c/bin/run P2-L2-execution haiku /tmp/adr-eval-c-test --reps 1 --results /tmp/results.tsv
|
||||
cat /tmp/results.tsv
|
||||
```
|
||||
|
||||
Each rep: fresh sandbox → `claude -p --model <tier> --output-format stream-json
|
||||
--dangerously-skip-permissions "<Task block>"` → `transcript.jsonl` written → `bin/check`
|
||||
appends one TSV row.
|
||||
|
||||
### Grid run (full measurement)
|
||||
|
||||
Once environment is stable (plugins cache fresh, hooks verified, fixture built):
|
||||
|
||||
```bash
|
||||
# Run the full grid (12 scenarios × 2 tiers × N reps) in parallel, monitoring as they complete
|
||||
for scenario in P1-L1-execution N1-L1-execution P2-L2-execution N2-L2-execution P3-L3-execution N3-L3-execution; do
|
||||
for model in haiku sonnet; do
|
||||
bin/run "$scenario" "$model" /tmp/adr-eval-c-grid --reps 1 --results /tmp/adr-eval-c-grid/results.tsv &
|
||||
done
|
||||
done
|
||||
wait
|
||||
cat /tmp/adr-eval-c-grid/results.tsv
|
||||
```
|
||||
|
||||
### Per-level pass bars (baseline expectations)
|
||||
|
||||
Eval C does not collapse to a single score. Results are reported **per level × per tier**,
|
||||
with discrimination measured by the **negative scenarios' axis (b) results** (unneeded ADR creation):
|
||||
|
||||
| Level | Positive (P*-L*): should trigger | Negative (N*-L*): should NOT create/assert | Comment |
|
||||
|-------|---------------------------|------------------------|---------|
|
||||
| L1 (Explicit) | Expected: PASS both tiers (find/create) | Expected: PASS both tiers (no unneeded create) | Explicit form + vocabulary are strongest signals for both polarity |
|
||||
| L2 (Moderate) | Expected: PASS sonnet, may degrade haiku | Expected: PASS sonnet, may degrade haiku | Vocabulary inference is harder at L2 |
|
||||
| L3 (Conceptual) | May degrade on both tiers (miss intent) | May degrade on both tiers (harder to resist recording a "decision" at intent level) | Pure intent is hardest for both directions |
|
||||
|
||||
**Axis (a) consultation rate** (informational, not pass/fail):
|
||||
- Report alongside axis (b) results. Model consulting the ADR system for negatives is acceptable;
|
||||
don't penalize cheap, deterministic lookups. The discrimination is in axis (b): unneeded ADR creation.
|
||||
- The checker also records `cited-adr:yes/no` per negative (informational). Truthful citation of a
|
||||
governing ADR while documenting/testing/auditing is CORRECT behavior — the negatives sit
|
||||
deliberately in ADR-covered domains. Distinguishing a true citation from a false constraint claim
|
||||
needs semantic judgment; a keyword check fails correct answers, so citation is never a FAIL.
|
||||
If false-citation ever needs to be measured, that requires a narrow judge, not a regex.
|
||||
|
||||
**Threshold hypothesis** (for post-baseline discussion, not part of first run):
|
||||
- Sonnet likely passes negatives reliably through L2 (no false create/cite), may struggle L3.
|
||||
- Haiku likely passes L1 negatives consistently, may over-trigger (create/assert) at L2/L3.
|
||||
- Negatives are the core measurement; if a tier fails negatives, it means over-recording —
|
||||
wording iteration must then address restraint (when NOT to create), not just trigger enhancement.
|
||||
|
||||
Evaluate the **per-level grid** (A:consultation rates + B:pass/fail) first. Summary threshold only
|
||||
if results show a clear pattern (e.g., "PASS positives through L2, PASS negatives through L2,
|
||||
PASS with consultation rate Y%").
|
||||
|
||||
### Optimizing with `/autoresearch` (later stage)
|
||||
|
||||
Not part of building this harness. When wording iteration starts:
|
||||
|
||||
1. **Freeze the run-set**: results and scenarios are locked as baseline.
|
||||
2. **Iterate only on wording** (SKILL.md, hook note, CLAUDE.md sections) — not checker,
|
||||
fixtures, scenarios, or judge rubric.
|
||||
3. **Reduced inner loop**: target failing cells + one passing control per tier per level.
|
||||
4. **Parallel runs**: drive per-cell `bin/run` scripts in the background (each cell fully
|
||||
independent).
|
||||
5. **Re-run full grid** once wording stabilizes.
|
||||
6. **If run-set degradation occurs**, stop iteration, document the contamination, and
|
||||
**switch all future work to the reserve-set** for measurement.
|
||||
|
||||
See `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md` for full discipline.
|
||||
|
||||
## Reserve set protection
|
||||
|
||||
Run-set and reserve-set are **structurally identical** (3 levels, P/N pairs on same target
|
||||
ADRs) but **different scenarios** so they are not interchangeable for measurement:
|
||||
|
||||
- **Before any wording iteration**: run-set is the measurement set.
|
||||
- **After first iteration against run-set results**: run-set is contaminated (wording tuned to
|
||||
it). Reserve-set becomes the fresh measurement surface; run-set results are historical
|
||||
only.
|
||||
- **Lock in writing**: when a change triggers an iteration decision, document that fact in a
|
||||
vault note or project comment so future reviewers know which set is the current measurement
|
||||
baseline.
|
||||
|
||||
The reserve-set scenarios and fixtures are never read informally, never "tried out" before
|
||||
being used for grid runs — same held-out discipline as the run-set.
|
||||
|
||||
## Layout
|
||||
|
||||
| Path | What |
|
||||
| --- | --- |
|
||||
| `fixture/project/` | Python async job queue, 6-ADR history |
|
||||
| `scenarios/P1-L1-execution.md`, `N1-L1-execution.md`, ... | run-set prompts (6 scenarios) |
|
||||
| `scenarios-reserve/P4-L1-notifications.md`, ... | reserve-set prompts (6 scenarios) |
|
||||
| `bin/sandbox <Sn> <dest>` | fresh git-initialized sandbox; all scenarios use same fixture base |
|
||||
| `bin/run <Sn> <model> <workdir> [--reps N] [--results FILE]` | headless runner (the ONLY valid execution mode) |
|
||||
| `bin/check <Sn> <sandbox> [--tsv <model>]` | two-axis checker with negative-scenario support; exit 0/1; TSV for autoresearch |
|
||||
| `bin/self-test [workdir]` | model-free both-directions harness validation (fabricates transcripts; never runs Task blocks) |
|
||||
| `judge-rubric.md` | frozen rubric for positive-scenario axis (b) LLM judge fallback |
|
||||
|
||||
## Why headless-only (no Agent-tool subagents)
|
||||
|
||||
Eval B (Eval A, actually — the in-session subagent eval) uses in-session subagents because
|
||||
it measures *prompted skill-execution*. Eval C measures **unprompted detection**, so each
|
||||
model under test must get a fresh `SessionStart` hook context. In-session subagents inherit
|
||||
the parent session and never see a fresh hook. `bin/run` spawns each rep as a fresh
|
||||
`claude -p` with cwd = sandbox; the transcript's `system/hook_response` event proves the hook
|
||||
fired. (Verified 2026-07-03 by eval-b author: haiku quoted the hook note verbatim in a
|
||||
sandbox session.)
|
||||
|
||||
## Fixture regeneration
|
||||
|
||||
The fixture's `graphify-out/` is **not included** (Eval C doesn't measure graph reach; it
|
||||
measures cue explicitness). The ADR history was generated with the plugin's own CLIs
|
||||
(`bin/adr-init`, `bin/adr-new`) so frontmatter and index match the shipped format, including
|
||||
the mechanically-superseded 0001→0002 pair.
|
||||
|
||||
To rebuild ADRs from scratch (not normally needed):
|
||||
|
||||
```bash
|
||||
FIXTURE="plugins/os-adr/eval-c/fixture/project"
|
||||
rm -rf "$FIXTURE/docs/adr"
|
||||
ruby plugins/os-adr/bin/adr-init --root "$FIXTURE" --template nygard
|
||||
# Then re-create each ADR using bin/adr-new as this README documents
|
||||
```
|
||||
|
||||
## Adding a scenario (later stage)
|
||||
|
||||
Add `scenarios/X.md` (or `scenarios-reserve/X.md`):
|
||||
- Task block + checker metadata (target ADR, topic pattern, axis-b fallback description)
|
||||
- Scenario class in `bin/check` (Ruby Scenario subclass with axis-b logic)
|
||||
- Extend `bin/sandbox` path list
|
||||
- Extend `bin/self-test` both directions (perfect + over-trigger for new scenario)
|
||||
- Document in per-level grid above if adding a level
|
||||
- Keep the held-out rule: the new Task block gets no informal trial runs
|
||||
|
||||
## Verification before first grid run
|
||||
|
||||
```bash
|
||||
# 1. Smoke-test the fixture and hooks
|
||||
cd plugins/os-adr/eval-c/fixture/project && git status
|
||||
|
||||
# 2. Verify self-test passes (model-free, no tokens spent)
|
||||
plugins/os-adr/eval-c/bin/self-test /tmp/adr-eval-c-test
|
||||
|
||||
# 3. Verify one cell runs headless
|
||||
plugins/os-adr/eval-c/bin/run P1-L1-execution haiku /tmp/adr-eval-c-test-run --reps 1
|
||||
|
||||
# 4. Check the TSV
|
||||
cat /tmp/adr-eval-c-test-run/results.tsv
|
||||
```
|
||||
|
||||
All must pass before running the first grid.
|
||||
|
|
@ -0,0 +1,445 @@
|
|||
#!/usr/bin/env ruby
|
||||
# frozen_string_literal: true
|
||||
|
||||
# Two-axis, deterministic-first checker for Eval C (ambiguity-ladder discrimination).
|
||||
#
|
||||
# check <P*-L*|N*-L*> <sandbox-root> [--tsv <model-label>]
|
||||
#
|
||||
# Reads <sandbox>/transcript.jsonl (stream-json, written by bin/run).
|
||||
#
|
||||
# Scenario class: P* (positive) = should consult/trigger ADR system
|
||||
# N* (negative) = should NOT over-trigger
|
||||
#
|
||||
# For POSITIVE scenarios:
|
||||
# axis (a) — unprompted consultation: mechanical. True iff any tool_use
|
||||
# block touches the ADR system (os-adr:* skill, bin/adr-* CLI, or
|
||||
# read/glob under docs/adr/).
|
||||
# axis (b) — specific correctness: mechanical new-ADR-file check matching
|
||||
# the scenario's topic pattern, or judge fallback if not found.
|
||||
#
|
||||
# For NEGATIVE scenarios:
|
||||
# axis (a) — over-trigger detection: mechanical. True iff NO tool_use
|
||||
# touched the ADR system (inverse of positive).
|
||||
# axis (b) — no false constraint assertion: mechanical check that no new
|
||||
# ADR file was created matching the scenario's topic, or judge fallback.
|
||||
#
|
||||
# Hook context is a precondition: invalid runs (no SessionStart hook) FAIL both axes.
|
||||
#
|
||||
# Overall PASS requires both axes. TSV mode for autoresearch integration.
|
||||
# Exit 0 on PASS, 1 on FAIL, 2 on usage error.
|
||||
|
||||
require "json"
|
||||
require "open3"
|
||||
|
||||
module AdrEvalC
|
||||
BASELINE_ADR_IDS = %w[0001 0002 0003 0004 0005 0006].freeze
|
||||
CONFLICT = /conflict|violat|contradic|goes against|decision|record|architecture/i
|
||||
|
||||
class Transcript
|
||||
def self.load(path)
|
||||
return nil unless File.exist?(path)
|
||||
events = File.readlines(path).filter_map do |line|
|
||||
JSON.parse(line)
|
||||
rescue JSON::ParserError
|
||||
nil
|
||||
end
|
||||
new(events)
|
||||
end
|
||||
|
||||
def initialize(events) = @events = events
|
||||
|
||||
def hook_context_present?
|
||||
@events.any? do |e|
|
||||
e["type"] == "system" && e["subtype"] == "hook_response" &&
|
||||
e["hook_name"].to_s.start_with?("SessionStart") &&
|
||||
e.to_json.include?("[os-adr]")
|
||||
end
|
||||
end
|
||||
|
||||
def tool_uses
|
||||
assistant_blocks.select { |b| b["type"] == "tool_use" }
|
||||
end
|
||||
|
||||
def assistant_text
|
||||
texts = assistant_blocks.select { |b| b["type"] == "text" }.map { |b| b["text"] }
|
||||
texts << final_message
|
||||
texts.compact.join("\n")
|
||||
end
|
||||
|
||||
def final_message
|
||||
result = @events.reverse.find { |e| e["type"] == "result" }
|
||||
result && result["result"].is_a?(String) ? result["result"] : nil
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def assistant_blocks
|
||||
@events.select { |e| e["type"] == "assistant" }
|
||||
.flat_map { |e| e.dig("message", "content") || [] }
|
||||
end
|
||||
end
|
||||
|
||||
class Sandbox
|
||||
def initialize(root) = @root = root
|
||||
attr_reader :root
|
||||
|
||||
def transcript = Transcript.load(File.join(@root, "transcript.jsonl"))
|
||||
|
||||
def new_adr_files
|
||||
Dir.glob(File.join(@root, "docs/adr/[0-9]*.md")).sort.reject do |path|
|
||||
BASELINE_ADR_IDS.include?(File.basename(path)[0, 4])
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
class Result
|
||||
def initialize = @failures = []
|
||||
attr_reader :failures
|
||||
def pass? = @failures.empty?
|
||||
|
||||
def expect(condition, reason)
|
||||
@failures << reason unless condition
|
||||
!!condition
|
||||
end
|
||||
end
|
||||
|
||||
class InformationalResult
|
||||
# For informational metrics (doesn't affect pass/fail, just recorded)
|
||||
def initialize(value)
|
||||
@value = value
|
||||
end
|
||||
|
||||
attr_reader :value
|
||||
def pass? = true # Informational results never fail
|
||||
end
|
||||
|
||||
class Judge
|
||||
RUBRIC = File.read(File.expand_path("../judge-rubric.md", __dir__))
|
||||
.split("---", 2).last.strip
|
||||
|
||||
def self.command
|
||||
ENV.fetch("ADR_EVAL_C_JUDGE_CMD",
|
||||
"claude -p --model haiku --dangerously-skip-permissions")
|
||||
end
|
||||
|
||||
def proposed_adr?(final_message)
|
||||
prompt = "#{RUBRIC}\n\n#{final_message}"
|
||||
output, status = Open3.capture2(self.class.command, stdin_data: prompt)
|
||||
raise "judge command failed: #{self.class.command}" unless status.success?
|
||||
verdict = output[/\b(YES|NO)\b/, 1]
|
||||
raise "judge gave no YES/NO verdict: #{output.inspect}" if verdict.nil?
|
||||
verdict == "YES"
|
||||
end
|
||||
end
|
||||
|
||||
class Scenario
|
||||
def initialize(sandbox, judge: Judge.new)
|
||||
@sb = sandbox
|
||||
@judge = judge
|
||||
end
|
||||
|
||||
def run
|
||||
axis_a = Result.new
|
||||
axis_b = Result.new
|
||||
transcript = @sb.transcript
|
||||
if transcript.nil?
|
||||
axis_a.expect(false, "no transcript.jsonl in sandbox")
|
||||
axis_b.expect(false, "no transcript.jsonl in sandbox")
|
||||
elsif !transcript.hook_context_present?
|
||||
axis_a.expect(false, "invalid run: SessionStart [os-adr] hook context missing")
|
||||
axis_b.expect(false, "invalid run: SessionStart [os-adr] hook context missing")
|
||||
else
|
||||
check_axis_a(axis_a, transcript)
|
||||
check_axis_b(axis_b, transcript)
|
||||
end
|
||||
[axis_a, axis_b]
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
attr_reader :sb, :judge
|
||||
|
||||
def check_axis_a(result, transcript)
|
||||
# Override in subclasses
|
||||
raise NotImplementedError, "subclasses must override check_axis_a"
|
||||
end
|
||||
|
||||
def adr_touch?(tool_use)
|
||||
probe = "#{tool_use['name']} #{tool_use['input'].to_json}"
|
||||
probe.match?(%r{os-adr|bin/adr-|adr-(find|new|init|detect|migrate)\b|docs/adr})
|
||||
end
|
||||
end
|
||||
|
||||
# --- Positive scenarios: should trigger ADR system ---
|
||||
|
||||
class PositiveScenario < Scenario
|
||||
private
|
||||
|
||||
def check_axis_a(result, transcript)
|
||||
touched = transcript.tool_uses.any? { |tu| adr_touch?(tu) }
|
||||
result.expect(touched, "no tool_use touched the ADR system (os-adr:*, bin/adr-*, docs/adr/)")
|
||||
end
|
||||
end
|
||||
|
||||
class P1L1Execution < PositiveScenario
|
||||
def check_axis_b(result, transcript)
|
||||
new_files = sb.new_adr_files
|
||||
if new_files.any?
|
||||
matching = new_files.select { |p| File.read(p).match?(topic) }
|
||||
result.expect(matching.any?,
|
||||
"new ADR exists but matches no topic term #{topic.inspect}")
|
||||
return
|
||||
end
|
||||
final = transcript.final_message
|
||||
unless final
|
||||
result.expect(false, "no new ADR file and no final message to judge")
|
||||
return
|
||||
end
|
||||
result.expect(judge.proposed_adr?(final),
|
||||
"no new ADR file; judge: final message does not propose/ask to record the decision")
|
||||
end
|
||||
|
||||
def topic = /timeout|retry|backoff|exponential|attempt/i
|
||||
end
|
||||
|
||||
class P2L2Execution < PositiveScenario
|
||||
def check_axis_b(result, transcript)
|
||||
new_files = sb.new_adr_files
|
||||
if new_files.any?
|
||||
matching = new_files.select { |p| File.read(p).match?(topic) }
|
||||
result.expect(matching.any?,
|
||||
"new ADR exists but matches no topic term #{topic.inspect}")
|
||||
return
|
||||
end
|
||||
final = transcript.final_message
|
||||
unless final
|
||||
result.expect(false, "no new ADR file and no final message to judge")
|
||||
return
|
||||
end
|
||||
result.expect(judge.proposed_adr?(final),
|
||||
"no new ADR file; judge: final message does not propose/ask to record the decision")
|
||||
end
|
||||
|
||||
def topic = /escalat|termin|failure|review|abandon/i
|
||||
end
|
||||
|
||||
class P3L3Execution < PositiveScenario
|
||||
def check_axis_b(result, transcript)
|
||||
new_files = sb.new_adr_files
|
||||
if new_files.any?
|
||||
matching = new_files.select { |p| File.read(p).match?(topic) }
|
||||
result.expect(matching.any?,
|
||||
"new ADR exists but matches no topic term #{topic.inspect}")
|
||||
return
|
||||
end
|
||||
final = transcript.final_message
|
||||
unless final
|
||||
result.expect(false, "no new ADR file and no final message to judge")
|
||||
return
|
||||
end
|
||||
result.expect(judge.proposed_adr?(final),
|
||||
"no new ADR file; judge: final message does not propose/ask to record the decision")
|
||||
end
|
||||
|
||||
def topic = /abandon|finish|complet|reliability|guarant/i
|
||||
end
|
||||
|
||||
class P4L1Notifications < PositiveScenario
|
||||
def check_axis_b(result, transcript)
|
||||
new_files = sb.new_adr_files
|
||||
if new_files.any?
|
||||
matching = new_files.select { |p| File.read(p).match?(topic) }
|
||||
result.expect(matching.any?,
|
||||
"new ADR exists but matches no topic term #{topic.inspect}")
|
||||
return
|
||||
end
|
||||
final = transcript.final_message
|
||||
unless final
|
||||
result.expect(false, "no new ADR file and no final message to judge")
|
||||
return
|
||||
end
|
||||
result.expect(judge.proposed_adr?(final),
|
||||
"no new ADR file; judge: final message does not propose/ask to record the decision")
|
||||
end
|
||||
|
||||
def topic = /notif|route|service|webhook|event/i
|
||||
end
|
||||
|
||||
class P5L2Notifications < PositiveScenario
|
||||
def check_axis_b(result, transcript)
|
||||
new_files = sb.new_adr_files
|
||||
if new_files.any?
|
||||
matching = new_files.select { |p| File.read(p).match?(topic) }
|
||||
result.expect(matching.any?,
|
||||
"new ADR exists but matches no topic term #{topic.inspect}")
|
||||
return
|
||||
end
|
||||
final = transcript.final_message
|
||||
unless final
|
||||
result.expect(false, "no new ADR file and no final message to judge")
|
||||
return
|
||||
end
|
||||
result.expect(judge.proposed_adr?(final),
|
||||
"no new ADR file; judge: final message does not propose/ask to record the decision")
|
||||
end
|
||||
|
||||
def topic = /webhook|deliv|retry|notification/i
|
||||
end
|
||||
|
||||
class P6L3Notifications < PositiveScenario
|
||||
def check_axis_b(result, transcript)
|
||||
new_files = sb.new_adr_files
|
||||
if new_files.any?
|
||||
matching = new_files.select { |p| File.read(p).match?(topic) }
|
||||
result.expect(matching.any?,
|
||||
"new ADR exists but matches no topic term #{topic.inspect}")
|
||||
return
|
||||
end
|
||||
final = transcript.final_message
|
||||
unless final
|
||||
result.expect(false, "no new ADR file and no final message to judge")
|
||||
return
|
||||
end
|
||||
result.expect(judge.proposed_adr?(final),
|
||||
"no new ADR file; judge: final message does not propose/ask to record the decision")
|
||||
end
|
||||
|
||||
def topic = /notif|channel|routing|delivery|unified/i
|
||||
end
|
||||
|
||||
# --- Negative scenarios: should NOT create false ADRs or assert phantom constraints ---
|
||||
|
||||
class NegativeScenario < Scenario
|
||||
def check_axis_a(result, transcript)
|
||||
# Axis (a) for negatives is INFORMATIONAL ONLY (doesn't affect pass/fail)
|
||||
# Record whether ADR system was consulted (yes/no)
|
||||
touched = transcript.tool_uses.any? { |tu| adr_touch?(tu) }
|
||||
# Return informational result (always "passes" but carries info value)
|
||||
@consultation_value = touched ? "yes" : "no"
|
||||
end
|
||||
|
||||
attr_reader :consultation_value
|
||||
|
||||
protected
|
||||
|
||||
attr_reader :sb
|
||||
|
||||
def record_citation(_result, text)
|
||||
# Citing an existing ADR is CORRECT behavior for these tasks: the negatives sit
|
||||
# deliberately in ADR-covered domains, and a truthful citation while documenting,
|
||||
# testing, or auditing is desired. Separating a true citation from a false
|
||||
# constraint claim requires semantic judgment a keyword match cannot provide,
|
||||
# so citation is recorded as informational only — never a FAIL.
|
||||
@citation_value = text.match?(/\b000[1-6]\b|ADR \d{4}/i) ? "yes" : "no"
|
||||
end
|
||||
|
||||
attr_reader :citation_value
|
||||
end
|
||||
|
||||
class N1L1Execution < NegativeScenario
|
||||
def check_axis_b(result, transcript)
|
||||
new_files = sb.new_adr_files
|
||||
result.expect(new_files.empty?,
|
||||
"over-trigger: created new ADR file when testing existing timeout logic")
|
||||
record_citation(result, transcript.assistant_text)
|
||||
end
|
||||
end
|
||||
|
||||
class N2L2Execution < NegativeScenario
|
||||
def check_axis_b(result, transcript)
|
||||
new_files = sb.new_adr_files
|
||||
result.expect(new_files.empty?,
|
||||
"over-trigger: created new ADR file when documenting existing failure handling")
|
||||
record_citation(result, transcript.assistant_text)
|
||||
end
|
||||
end
|
||||
|
||||
class N3L3Execution < NegativeScenario
|
||||
def check_axis_b(result, transcript)
|
||||
new_files = sb.new_adr_files
|
||||
result.expect(new_files.empty?,
|
||||
"over-trigger: created new ADR file when auditing existing lifecycle guarantees")
|
||||
record_citation(result, transcript.assistant_text)
|
||||
end
|
||||
end
|
||||
|
||||
class N4L1Notifications < NegativeScenario
|
||||
def check_axis_b(result, transcript)
|
||||
new_files = sb.new_adr_files
|
||||
result.expect(new_files.empty?,
|
||||
"over-trigger: created new ADR file when debugging existing webhook delivery")
|
||||
record_citation(result, transcript.assistant_text)
|
||||
end
|
||||
end
|
||||
|
||||
class N5L2Notifications < NegativeScenario
|
||||
def check_axis_b(result, transcript)
|
||||
new_files = sb.new_adr_files
|
||||
result.expect(new_files.empty?,
|
||||
"over-trigger: created new ADR file when testing existing notification failure handling")
|
||||
record_citation(result, transcript.assistant_text)
|
||||
end
|
||||
end
|
||||
|
||||
class N6L3Notifications < NegativeScenario
|
||||
def check_axis_b(result, transcript)
|
||||
new_files = sb.new_adr_files
|
||||
result.expect(new_files.empty?,
|
||||
"over-trigger: created new ADR file when auditing existing notification audit trail")
|
||||
record_citation(result, transcript.assistant_text)
|
||||
end
|
||||
end
|
||||
|
||||
|
||||
SCENARIOS = {
|
||||
"P1-L1-execution" => P1L1Execution,
|
||||
"N1-L1-execution" => N1L1Execution,
|
||||
"P2-L2-execution" => P2L2Execution,
|
||||
"N2-L2-execution" => N2L2Execution,
|
||||
"P3-L3-execution" => P3L3Execution,
|
||||
"N3-L3-execution" => N3L3Execution,
|
||||
"P4-L1-notifications" => P4L1Notifications,
|
||||
"N4-L1-notifications" => N4L1Notifications,
|
||||
"P5-L2-notifications" => P5L2Notifications,
|
||||
"N5-L2-notifications" => N5L2Notifications,
|
||||
"P6-L3-notifications" => P6L3Notifications,
|
||||
"N6-L3-notifications" => N6L3Notifications,
|
||||
}.freeze
|
||||
end
|
||||
|
||||
scenario_id, sandbox_root = ARGV[0], ARGV[1]
|
||||
tsv_model = ARGV[2] == "--tsv" ? (ARGV[3] || "unknown") : nil
|
||||
klass = AdrEvalC::SCENARIOS[scenario_id]
|
||||
abort "usage: check <#{AdrEvalC::SCENARIOS.keys.join('|')}> <sandbox-root> [--tsv <model>]" if klass.nil? || sandbox_root.nil?
|
||||
abort "no such sandbox: #{sandbox_root}" unless File.directory?(sandbox_root)
|
||||
|
||||
checker = klass.new(AdrEvalC::Sandbox.new(File.expand_path(sandbox_root)))
|
||||
axis_a, axis_b = checker.run
|
||||
is_negative = scenario_id.start_with?("N")
|
||||
|
||||
# For negatives, axis (a) is informational; overall depends on axis (b) only
|
||||
if is_negative
|
||||
overall = axis_b.pass?
|
||||
axis_a_label = "A:#{checker.consultation_value || 'unknown'}"
|
||||
else
|
||||
overall = axis_a.pass? && axis_b.pass?
|
||||
axis_a_label = axis_a.pass? ? "A:PASS" : "A:FAIL"
|
||||
end
|
||||
|
||||
reasons = (axis_a.failures + axis_b.failures).join("; ")
|
||||
if is_negative && checker.respond_to?(:citation_value) && checker.citation_value
|
||||
reasons = ["cited-adr:#{checker.citation_value}", reasons].reject(&:empty?).join("; ")
|
||||
end
|
||||
|
||||
if tsv_model
|
||||
puts [scenario_id, tsv_model,
|
||||
axis_a_label,
|
||||
axis_b.pass? ? "B:PASS" : "B:FAIL",
|
||||
overall ? "PASS" : "FAIL", reasons].join("\t")
|
||||
else
|
||||
axis_a_desc = is_negative ? "consultation:#{checker.consultation_value || 'unknown'}" : (axis_a.pass? ? "PASS" : "FAIL")
|
||||
puts "#{overall ? 'PASS' : 'FAIL'} #{scenario_id} " \
|
||||
"(axis-a #{axis_a_desc}, axis-b #{axis_b.pass? ? 'PASS' : 'FAIL'})"
|
||||
(axis_a.failures + axis_b.failures).each { |f| puts " - #{f}" }
|
||||
end
|
||||
exit(overall ? 0 : 1)
|
||||
|
|
@ -0,0 +1,55 @@
|
|||
#!/usr/bin/env bash
|
||||
# Headless runner for Eval C — the ONLY valid execution mode (fresh `claude -p`
|
||||
# with cwd = sandbox so the real SessionStart hook fires).
|
||||
#
|
||||
# Usage: run <scenario> <model> <workdir> [--reps N] [--results FILE]
|
||||
#
|
||||
# scenario P*-L*|N*-L* (e.g., P1-L1-execution, N2-L2-execution)
|
||||
# model haiku|sonnet|opus|...
|
||||
# workdir sandboxes created under here as <scenario>-<model>-rN
|
||||
# --reps repeated executions (default 1; design.md Decision 5)
|
||||
#
|
||||
# Each rep: fresh sandbox -> claude -p with only the scenario task prompt
|
||||
# (no system-level hints) -> full stream-json transcript saved ->
|
||||
# bin/check appends one TSV row.
|
||||
set -euo pipefail
|
||||
|
||||
SCENARIO="${1:?usage: run <scenario> <model> <workdir> [--reps N] [--results FILE]}"
|
||||
MODEL="${2:?model required (haiku|sonnet|...)}"
|
||||
WORKDIR="${3:?workdir required}"
|
||||
shift 3
|
||||
|
||||
REPS=1
|
||||
RESULTS=""
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--reps) REPS="${2:?--reps needs a number}"; shift 2 ;;
|
||||
--results) RESULTS="${2:?--results needs a path}"; shift 2 ;;
|
||||
*) echo "unknown option: $1" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
EVAL_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
SCENARIO_FILE="$EVAL_ROOT/scenarios/${SCENARIO}.md"
|
||||
if [ ! -f "$SCENARIO_FILE" ]; then
|
||||
SCENARIO_FILE="$EVAL_ROOT/scenarios-reserve/${SCENARIO}.md"
|
||||
fi
|
||||
[ -f "$SCENARIO_FILE" ] || { echo "unknown scenario: $SCENARIO (checked scenarios/ and scenarios-reserve/)" >&2; exit 2; }
|
||||
mkdir -p "$WORKDIR"
|
||||
RESULTS="${RESULTS:-$WORKDIR/results.tsv}"
|
||||
|
||||
TASK="$(awk '/^## Task/{found=1; next} found' "$SCENARIO_FILE")"
|
||||
|
||||
for rep in $(seq 1 "$REPS"); do
|
||||
SANDBOX="$WORKDIR/$SCENARIO-$MODEL-r$rep"
|
||||
"$EVAL_ROOT/bin/sandbox" "$SCENARIO" "$SANDBOX" >/dev/null
|
||||
|
||||
# cwd = sandbox: the SessionStart hook resolves the project root from here.
|
||||
(cd "$SANDBOX" && claude -p \
|
||||
--model "$MODEL" \
|
||||
--output-format stream-json --verbose \
|
||||
--dangerously-skip-permissions \
|
||||
"$TASK" > transcript.jsonl) || echo "claude exited non-zero for $SANDBOX" >&2
|
||||
|
||||
"$EVAL_ROOT/bin/check" "$SCENARIO" "$SANDBOX" --tsv "$MODEL" | tee -a "$RESULTS"
|
||||
done
|
||||
|
|
@ -0,0 +1,30 @@
|
|||
#!/bin/bash
|
||||
# Create a fresh, git-initialized sandbox copy of the Eval C fixture.
|
||||
# Usage: sandbox <P*-L*|N*-L*> <dest-dir>
|
||||
#
|
||||
# Eval C does not use graphify-out (ladder measures cue explicitness, not graph reach).
|
||||
set -euo pipefail
|
||||
|
||||
SCENARIO="${1:?usage: sandbox <scenario> <dest-dir>}"
|
||||
DEST="${2:?usage: sandbox <scenario> <dest-dir>}"
|
||||
EVAL_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
FIXTURE="$EVAL_ROOT/fixture/project"
|
||||
|
||||
# All scenarios use the same fixture without graphify-out
|
||||
case "$SCENARIO" in
|
||||
P1-L1-execution|N1-L1-execution|P2-L2-execution|N2-L2-execution|P3-L3-execution|N3-L3-execution|\
|
||||
P4-L1-notifications|N4-L1-notifications|P5-L2-notifications|N5-L2-notifications|P6-L3-notifications|N6-L3-notifications)
|
||||
:
|
||||
;;
|
||||
*) echo "unknown scenario: $SCENARIO" >&2; exit 2 ;;
|
||||
esac
|
||||
|
||||
if [ -e "$DEST" ]; then echo "refusing to overwrite existing $DEST" >&2; exit 2; fi
|
||||
mkdir -p "$DEST"
|
||||
cp -r "$FIXTURE/." "$DEST/"
|
||||
rm -rf "$DEST/.os-adr"
|
||||
|
||||
git -C "$DEST" init -q
|
||||
git -C "$DEST" add -A
|
||||
git -C "$DEST" -c user.email=eval@local -c user.name=eval commit -qm "fixture baseline"
|
||||
echo "$DEST"
|
||||
|
|
@ -0,0 +1,196 @@
|
|||
#!/usr/bin/env bash
|
||||
# Self-test the Eval C harness in both directions, model-free:
|
||||
#
|
||||
# 1. For each POSITIVE scenario, fabricate a "perfect" transcript/sandbox
|
||||
# state (model did right thing) -> bin/check must PASS.
|
||||
# 2. For each POSITIVE scenario, fabricate an "untouched" run (never
|
||||
# consulted ADR system) -> bin/check must FAIL, axis (a) FAIL.
|
||||
# 3. For each NEGATIVE scenario, fabricate a "proper" transcript (no ADR
|
||||
# touch, no false ADR creation) -> bin/check must PASS.
|
||||
# 4. For each NEGATIVE scenario, fabricate an "over-trigger" run (consulted
|
||||
# ADR system or created ADR when shouldn't) -> bin/check must FAIL.
|
||||
# 5. Sandbox isolation: the canonical fixture must be byte-identical after
|
||||
# all sandbox operations.
|
||||
#
|
||||
# The judge is stubbed via ADR_EVAL_C_JUDGE_CMD so this never spends tokens.
|
||||
# It never touches the held-out scenario Task blocks.
|
||||
#
|
||||
# Usage: self-test [workdir] (default: a fresh mktemp dir)
|
||||
set -euo pipefail
|
||||
|
||||
EVAL_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
PLUGIN_ROOT="$(cd "$EVAL_ROOT/.." && pwd)"
|
||||
WORKDIR="${1:-$(mktemp -d /tmp/adr-eval-c-selftest.XXXXXX)}"
|
||||
mkdir -p "$WORKDIR"
|
||||
FAILED=0
|
||||
|
||||
fixture_digest() {
|
||||
(cd "$EVAL_ROOT/fixture/project" && find . -type f ! -path "./.os-adr/*" -print0 \
|
||||
| sort -z | xargs -0 md5sum | md5sum)
|
||||
}
|
||||
DIGEST_BEFORE="$(fixture_digest)"
|
||||
|
||||
# --- transcript fabrication -------------------------------------------------
|
||||
|
||||
hook_event() {
|
||||
printf '%s\n' '{"type":"system","subtype":"hook_response","hook_name":"SessionStart:startup","output":"[os-adr] This project records architecture decisions in docs/adr/"}'
|
||||
}
|
||||
|
||||
tool_use_event() { # $1 = tool name, $2 = input JSON
|
||||
printf '{"type":"assistant","message":{"role":"assistant","content":[{"type":"tool_use","name":"%s","input":%s}]}}\n' "$1" "$2"
|
||||
}
|
||||
|
||||
result_event() { # $1 = final text (single line)
|
||||
python3 -c 'import json,sys; print(json.dumps({"type":"result","subtype":"success","result":sys.argv[1]}))' "$1"
|
||||
}
|
||||
|
||||
perfect_transcript() { # $1 = sandbox, $2 = adr-touching tool input, $3 = final text
|
||||
{ hook_event
|
||||
tool_use_event "Bash" "$2"
|
||||
result_event "$3"
|
||||
} > "$1/transcript.jsonl"
|
||||
}
|
||||
|
||||
untouched_transcript() { # $1 = sandbox
|
||||
{ hook_event
|
||||
tool_use_event "Read" '{"file_path":"taskq/execution.py"}'
|
||||
result_event "I made the change you asked for."
|
||||
} > "$1/transcript.jsonl"
|
||||
}
|
||||
|
||||
expect() { # $1 = PASS|FAIL, $2 = label, then the check command
|
||||
local want="$1" label="$2"; shift 2
|
||||
local got=PASS
|
||||
"$@" >/dev/null 2>&1 || got=FAIL
|
||||
if [ "$got" = "$want" ]; then
|
||||
echo "ok $label ($want)"
|
||||
else
|
||||
echo "FAIL $label: wanted $want, got $got"
|
||||
"$@" || true
|
||||
FAILED=1
|
||||
fi
|
||||
}
|
||||
|
||||
FIND_INPUT='{"command":"ruby '"$PLUGIN_ROOT"'/bin/adr-find --root . --paths taskq/execution.py"}'
|
||||
|
||||
# --- 1. POSITIVE runs: perfect runs must PASS, untouched must FAIL axis-a ---
|
||||
|
||||
declare -A POSITIVE_SCENARIOS=(
|
||||
[P1-L1-execution]="Added 5-minute timeout with exponential backoff retry logic. Should we record this timeout strategy as an ADR?"
|
||||
[P2-L2-execution]="Terminal failures now escalate to a review queue. This is a significant policy decision — should we record it?"
|
||||
[P3-L3-execution]="Implemented guarantee that no work vanishes silently. All jobs either complete, fail with logging, or are explicitly abandoned. This is an architectural commitment."
|
||||
[P4-L1-notifications]="Refactored all notifications to route through central NotificationService. This is a major coupling decision — should we record it as an ADR?"
|
||||
[P5-L2-notifications]="Webhook delivery now retries with exponential backoff on failure. This is a reliability policy we should document as a decision."
|
||||
[P6-L3-notifications]="Ensured all user-facing notifications go through one coherent pathway. This is a core architectural guarantee."
|
||||
)
|
||||
|
||||
echo "Testing POSITIVE scenarios (perfect runs):"
|
||||
for sc in "${!POSITIVE_SCENARIOS[@]}"; do
|
||||
sb="$WORKDIR/perfect-$sc"
|
||||
"$EVAL_ROOT/bin/sandbox" "$sc" "$sb" >/dev/null
|
||||
|
||||
# Create a new ADR to simulate the model recording the decision
|
||||
title=$(echo "${POSITIVE_SCENARIOS[$sc]}" | head -c 60)
|
||||
printf '{"title": "%s", "context": "c", "decision": "d", "consequences": "q", "alternatives": "a"}' \
|
||||
"$title" | ruby "$PLUGIN_ROOT/bin/adr-new" --root "$sb" >/dev/null 2>&1 || true
|
||||
|
||||
perfect_transcript "$sb" "$FIND_INPUT" "${POSITIVE_SCENARIOS[$sc]}"
|
||||
expect PASS "perfect $sc" "$EVAL_ROOT/bin/check" "$sc" "$sb"
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "Testing POSITIVE scenarios (untouched runs must FAIL axis-a):"
|
||||
for sc in "${!POSITIVE_SCENARIOS[@]}"; do
|
||||
sb="$WORKDIR/untouched-$sc"
|
||||
"$EVAL_ROOT/bin/sandbox" "$sc" "$sb" >/dev/null
|
||||
untouched_transcript "$sb"
|
||||
expect FAIL "untouched $sc" \
|
||||
env ADR_EVAL_C_JUDGE_CMD="printf NO" "$EVAL_ROOT/bin/check" "$sc" "$sb"
|
||||
row="$(env ADR_EVAL_C_JUDGE_CMD="printf NO" "$EVAL_ROOT/bin/check" "$sc" "$sb" --tsv selftest || true)"
|
||||
case "$row" in
|
||||
*$'\t'A:FAIL$'\t'*) echo "ok untouched $sc axis-a FAIL confirmed" ;;
|
||||
*) echo "FAIL untouched $sc: axis-a not FAIL in TSV: $row"; FAILED=1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# --- 2. NEGATIVE runs: proper runs must PASS, over-triggers must FAIL ---
|
||||
|
||||
declare -A NEGATIVE_SCENARIOS=(
|
||||
[N1-L1-execution]="Designed a clean callback-based notification subscription API. No existing architecture governs this."
|
||||
[N2-L2-execution]="Implemented priority-based job routing with high-priority jobs processed first. This is a new queue feature."
|
||||
[N3-L3-execution]="Built performance dashboards showing real-time job system metrics. Orthogonal to execution logic."
|
||||
[N4-L1-notifications]="Improved the audit log format to JSON lines for better parsing. Just formatting, no routing change."
|
||||
[N5-L2-notifications]="Added in-memory caching of compiled notification templates for speed. Orthogonal to notification routing."
|
||||
[N6-L3-notifications]="Optimized historical report queries by adding database indexes. Query performance optimization."
|
||||
)
|
||||
|
||||
echo ""
|
||||
echo "Testing NEGATIVE scenarios (proper runs — axis b PASS with or without consultation):"
|
||||
for sc in "${!NEGATIVE_SCENARIOS[@]}"; do
|
||||
sb="$WORKDIR/proper-$sc"
|
||||
"$EVAL_ROOT/bin/sandbox" "$sc" "$sb" >/dev/null
|
||||
untouched_transcript "$sb" # No ADR touch: axis (a) informational, axis (b) PASS
|
||||
expect PASS "proper $sc (no ADR touch, no false ADR created)" \
|
||||
env ADR_EVAL_C_JUDGE_CMD="printf NO" "$EVAL_ROOT/bin/check" "$sc" "$sb"
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "Testing NEGATIVE scenarios (over-trigger: create ADR when shouldn't):"
|
||||
for sc in "${!NEGATIVE_SCENARIOS[@]}"; do
|
||||
sb="$WORKDIR/overtrigger-create-$sc"
|
||||
"$EVAL_ROOT/bin/sandbox" "$sc" "$sb" >/dev/null
|
||||
# Simulate over-trigger: create an ADR when shouldn't
|
||||
printf '{"title": "Spurious ADR", "context": "c", "decision": "d", "consequences": "q", "alternatives": "a"}' \
|
||||
| ruby "$PLUGIN_ROOT/bin/adr-new" --root "$sb" >/dev/null 2>&1 || true
|
||||
perfect_transcript "$sb" "$FIND_INPUT" "I recorded this decision."
|
||||
expect FAIL "over-trigger-create $sc (created ADR when no decision)" \
|
||||
"$EVAL_ROOT/bin/check" "$sc" "$sb"
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "Testing NEGATIVE scenarios (truthful ADR citation must PASS — citation is informational, not a FAIL):"
|
||||
# The negatives sit deliberately in ADR-covered domains; a model that consults and
|
||||
# truthfully cites a governing ADR while documenting/testing is behaving correctly.
|
||||
# Only ADR *creation* fails a negative. This guards the checker against regressing
|
||||
# to keyword-matching citations as failures.
|
||||
sb="$WORKDIR/truthful-citation-N1"
|
||||
"$EVAL_ROOT/bin/sandbox" N1-L1-execution "$sb" >/dev/null
|
||||
{ hook_event
|
||||
tool_use_event "Read" '{"file_path":"taskq/execution.py"}'
|
||||
result_event "I wrote tests for the existing timeout logic. ADR 0002 requires that timeouts stay at 5 minutes, so the tests assert the 5-minute timeout and exponential backoff behavior as decided."
|
||||
} > "$sb/transcript.jsonl"
|
||||
expect PASS "truthful-citation N1 (cites governing ADR 0002 while testing — no ADR created)" \
|
||||
"$EVAL_ROOT/bin/check" N1-L1-execution "$sb"
|
||||
|
||||
# Missing hook context should invalidate both positive and negative
|
||||
echo ""
|
||||
echo "Testing hook-context precondition (invalid run):"
|
||||
sb="$WORKDIR/nohook-P1-L1"
|
||||
"$EVAL_ROOT/bin/sandbox" P1-L1-execution "$sb" >/dev/null
|
||||
{ tool_use_event "Bash" "$FIND_INPUT"; result_event "Made the change."; } > "$sb/transcript.jsonl"
|
||||
expect FAIL "P1-L1 without SessionStart hook (invalid run)" "$EVAL_ROOT/bin/check" P1-L1-execution "$sb"
|
||||
|
||||
sb="$WORKDIR/nohook-N1-L1"
|
||||
"$EVAL_ROOT/bin/sandbox" N1-L1-execution "$sb" >/dev/null
|
||||
{ tool_use_event "Bash" "$FIND_INPUT"; result_event "Made the change."; } > "$sb/transcript.jsonl"
|
||||
expect FAIL "N1-L1 without SessionStart hook (invalid run)" "$EVAL_ROOT/bin/check" N1-L1-execution "$sb"
|
||||
|
||||
# --- 3. fixture isolation ---
|
||||
|
||||
echo ""
|
||||
DIGEST_AFTER="$(fixture_digest)"
|
||||
if [ "$DIGEST_BEFORE" = "$DIGEST_AFTER" ]; then
|
||||
echo "ok canonical fixture untouched by all sandbox operations"
|
||||
else
|
||||
echo "FAIL canonical fixture was modified"
|
||||
FAILED=1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
if [ $FAILED -eq 0 ]; then
|
||||
echo "✓ All self-tests passed"
|
||||
else
|
||||
echo "✗ Some self-tests failed"
|
||||
fi
|
||||
|
||||
exit $FAILED
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
__pycache__/
|
||||
*.pyc
|
||||
.pytest_cache/
|
||||
*.egg-info/
|
||||
dist/
|
||||
build/
|
||||
.os-adr/
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
*.md
|
||||
docs/
|
||||
config/
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
# CLAUDE.md
|
||||
|
||||
`taskq` is a small Python async job queue system for reliable work processing.
|
||||
|
||||
## Architecture decisions
|
||||
|
||||
This project records architecture decisions as ADRs in `docs/adr/` (index:
|
||||
`docs/adr/README.md`).
|
||||
|
||||
- **When** your task involves making an architecture-level choice, or changing/replacing an
|
||||
approach this codebase already uses (job execution, persistence, notifications, delivery
|
||||
strategy) → **first** run `/os-adr:find` to check whether a recorded decision already covers
|
||||
it.
|
||||
- Mechanical rule: **before your first edit to any existing source or config file**, run
|
||||
`/os-adr:find` on the paths you are about to touch — one cheap, deterministic CLI call.
|
||||
This applies to additions too: a new method or module can bypass a decided constraint just
|
||||
as easily as a rewrite can.
|
||||
- **When** you make such a choice → record it with `/os-adr:create`.
|
||||
- If an Accepted ADR covers the approach you are changing, you are **reversing a recorded
|
||||
decision**. A task that reverses a recorded decision is **not complete until the superseding
|
||||
ADR is created** (`/os-adr:create` with `supersedes`) — implementing the change and stopping
|
||||
leaves `docs/adr/` asserting the opposite of what the code does. Create the superseding ADR
|
||||
(or explicitly propose it to the user) before you consider the task done.
|
||||
|
|
@ -0,0 +1,19 @@
|
|||
# Task Queue — async job queue system
|
||||
|
||||
A small Python async job queue for reliable job processing.
|
||||
|
||||
## Architecture decisions
|
||||
|
||||
This project records architecture decisions as ADRs in `docs/adr/` (index: `docs/adr/README.md`).
|
||||
|
||||
- **When** your task involves making an architecture-level choice, or changing/replacing an
|
||||
approach this codebase already uses (job execution, persistence, notifications) →
|
||||
**first** run `/os-adr:find` to check whether a recorded decision already covers it.
|
||||
- Mechanical rule: **before your first edit to any existing source or config file**, run
|
||||
`/os-adr:find` on the paths you are about to touch — one cheap, deterministic CLI call.
|
||||
This applies to additions too.
|
||||
- **When** you make such a choice → record it with `/os-adr:create`.
|
||||
- If an Accepted ADR covers the approach you are changing, you are **reversing a recorded
|
||||
decision**. A task that reverses a recorded decision is **not complete until the superseding
|
||||
ADR is created** — implementing the change and stopping leaves `docs/adr/` asserting the
|
||||
opposite of what the code does.
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
---
|
||||
id: "0001"
|
||||
date: 2026-02-01
|
||||
status: Superseded
|
||||
supersedes:
|
||||
superseded-by: "0002"
|
||||
affected-paths: [taskq/execution.py, taskq/queue.py]
|
||||
affected-components: [execution, queue]
|
||||
---
|
||||
|
||||
# 0001 — Job execution with 10-second timeout and simple retry
|
||||
|
||||
## Context
|
||||
|
||||
Early version needed fast failure detection.
|
||||
|
||||
## Decision
|
||||
|
||||
Jobs timeout at 10 seconds with immediate retry.
|
||||
|
||||
## Consequences
|
||||
|
||||
Simple but inefficient: rapid retries on transient failures.
|
||||
|
||||
## Alternatives rejected
|
||||
|
||||
Exponential backoff considered but deemed unnecessary at scale.
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
---
|
||||
id: "0002"
|
||||
date: 2026-02-10
|
||||
status: Accepted
|
||||
supersedes: "0001"
|
||||
superseded-by:
|
||||
affected-paths: [taskq/execution.py, taskq/queue.py]
|
||||
affected-components: [execution, queue]
|
||||
---
|
||||
|
||||
# 0002 — Job timeouts with exponential backoff, max 3 attempts
|
||||
|
||||
## Context
|
||||
|
||||
Simple retries caused cascade failures. Transient failures need backoff.
|
||||
|
||||
## Decision
|
||||
|
||||
Jobs timeout after 5 minutes. Failed jobs retry with exponential backoff (1s, 2s, 4s) up to 3 total attempts. Permanent failure escalates to review queue.
|
||||
|
||||
## Consequences
|
||||
|
||||
Better resilience. Reduced cascading failures. Terminal failures tracked for review.
|
||||
|
||||
## Alternatives rejected
|
||||
|
||||
Circuit breaker rejected (adds complexity without benefit at this scale).
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
---
|
||||
id: "0003"
|
||||
date: 2026-02-15
|
||||
status: Accepted
|
||||
supersedes:
|
||||
superseded-by:
|
||||
affected-paths: [taskq/notifications.py, taskq/execution.py]
|
||||
affected-components: [notifications, execution]
|
||||
---
|
||||
|
||||
# 0003 — Route all job notifications through central NotificationService
|
||||
|
||||
## Context
|
||||
|
||||
Job handlers called notification code directly, causing tight coupling and inconsistent delivery.
|
||||
|
||||
## Decision
|
||||
|
||||
All outbound notifications (webhooks, emails, events) routed through taskq.notifications.NotificationService. Handlers submit notification requests, not execute them directly.
|
||||
|
||||
## Consequences
|
||||
|
||||
Decoupled handlers from notification details. Single place to change delivery policy or add retry logic to notifications. Slightly higher latency (service batches notifications).
|
||||
|
||||
## Alternatives rejected
|
||||
|
||||
Pub/sub system rejected: overkill for current scale. Direct calls rejected: tight coupling.
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
---
|
||||
id: "0004"
|
||||
date: 2026-02-18
|
||||
status: Accepted
|
||||
supersedes:
|
||||
superseded-by:
|
||||
affected-paths: [taskq/queue.py, taskq/persistence.py]
|
||||
affected-components: [queue, persistence]
|
||||
---
|
||||
|
||||
# 0004 — Persist job queue to SQLite; survive process restarts
|
||||
|
||||
## Context
|
||||
|
||||
In-memory queue loses all pending jobs on restart. Clients resubmit, causing duplicates and lost work.
|
||||
|
||||
## Decision
|
||||
|
||||
All enqueued jobs persisted to local SQLite database (taskq.db). On startup, reload pending jobs from disk. Transaction isolation ensures no jobs are lost mid-processing.
|
||||
|
||||
## Consequences
|
||||
|
||||
Jobs survive restarts. Startup slower (disk IO). Must coordinate restart with in-flight execution to avoid double-processing. Query performance on large queues depends on SQLite optimization.
|
||||
|
||||
## Alternatives rejected
|
||||
|
||||
In-memory with RDB snapshots rejected: complexity without durability guarantee. Remote store (PostgreSQL) rejected: too heavyweight for single-user queue.
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
---
|
||||
id: "0005"
|
||||
date: 2026-02-22
|
||||
status: Accepted
|
||||
supersedes:
|
||||
superseded-by:
|
||||
affected-paths: [taskq/execution.py, taskq/queue.py]
|
||||
affected-components: [execution, queue]
|
||||
---
|
||||
|
||||
# 0005 — Failed jobs escalate to human review queue after retries exhausted
|
||||
|
||||
## Context
|
||||
|
||||
Silent failures (permanent errors after backoff exhausted) left operators unaware of issues.
|
||||
|
||||
## Decision
|
||||
|
||||
Jobs that exhaust exponential backoff retries move to a human-review queue (distinct from main job queue). Operators review and manually retry or discard.
|
||||
|
||||
## Consequences
|
||||
|
||||
No silent failures. Audit trail of manual interventions. Requires monitoring of review queue. Escalated jobs delay final resolution.
|
||||
|
||||
## Alternatives rejected
|
||||
|
||||
Automatic retry cap with notification rejected: operators would ignore generic alerts. Dead-letter file (unstructured) rejected: hard to query/monitor.
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
---
|
||||
id: "0006"
|
||||
date: 2026-03-01
|
||||
status: Accepted
|
||||
supersedes:
|
||||
superseded-by:
|
||||
affected-paths: [taskq/logging.py, taskq/execution.py]
|
||||
affected-components: [logging, execution]
|
||||
---
|
||||
|
||||
# 0006 — All job state transitions logged to audit trail for compliance
|
||||
|
||||
## Context
|
||||
|
||||
No record of who/when changed job status for audit/compliance reporting.
|
||||
|
||||
## Decision
|
||||
|
||||
Every job state change (enqueue, start, success, failure, escalate) logged to immutable audit trail with timestamp, user (if applicable), and reason. Audit trail never deleted, only archived.
|
||||
|
||||
## Consequences
|
||||
|
||||
Strong audit trail. Extra logging overhead. Requires cleanup/archival process for large deployments. Larger on-disk footprint.
|
||||
|
||||
## Alternatives rejected
|
||||
|
||||
Ad-hoc logging rejected: inconsistent, missed events. Centralized event store rejected: external dependency.
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
<!-- Generated by os-adr. Do not hand-edit the table: it is regenerated in full on every ADR write. -->
|
||||
|
||||
# Architecture Decision Records
|
||||
|
||||
One file per decision, `NNNN-kebab-title.md`, created via `/os-adr:create`.
|
||||
|
||||
<!-- adr-index:begin -->
|
||||
| ID | Title | Status | Date |
|
||||
| --- | --- | --- | --- |
|
||||
| 0001 | [Job execution with 10-second timeout and simple retry](0001-job-execution-with-10-second-timeout-and-simple-retry.md) | Superseded | 2026-02-01 |
|
||||
| 0002 | [Job timeouts with exponential backoff, max 3 attempts](0002-job-timeouts-with-exponential-backoff-max-3-attempts.md) | Accepted | 2026-02-10 |
|
||||
| 0003 | [Route all job notifications through central NotificationService](0003-route-all-job-notifications-through-central-notificationservice.md) | Accepted | 2026-02-15 |
|
||||
| 0004 | [Persist job queue to SQLite; survive process restarts](0004-persist-job-queue-to-sqlite-survive-process-restarts.md) | Accepted | 2026-02-18 |
|
||||
| 0005 | [Failed jobs escalate to human review queue after retries exhausted](0005-failed-jobs-escalate-to-human-review-queue-after-retries-exhausted.md) | Accepted | 2026-02-22 |
|
||||
| 0006 | [All job state transitions logged to audit trail for compliance](0006-all-job-state-transitions-logged-to-audit-trail-for-compliance.md) | Accepted | 2026-03-01 |
|
||||
<!-- adr-index:end -->
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
"""Async job queue system."""
|
||||
__version__ = "0.1.0"
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
"""Job execution engine."""
|
||||
import asyncio
|
||||
from typing import Callable
|
||||
|
||||
class ExecutionEngine:
|
||||
"""Executes jobs with configurable handlers."""
|
||||
def __init__(self, queue):
|
||||
self.queue = queue
|
||||
self.handlers = {}
|
||||
# Currently: no timeout, no retry logic
|
||||
|
||||
def register_handler(self, name: str, handler: Callable) -> None:
|
||||
"""Register a job handler function."""
|
||||
self.handlers[name] = handler
|
||||
|
||||
async def run_job(self, job) -> dict:
|
||||
"""Execute a single job (no timeout/retry currently)."""
|
||||
handler = self.handlers.get(job.handler)
|
||||
if not handler:
|
||||
return {"status": "failed", "reason": "handler_not_found"}
|
||||
try:
|
||||
result = await handler(job.payload)
|
||||
return {"status": "success", "result": result}
|
||||
except Exception as e:
|
||||
return {"status": "failed", "reason": str(e)}
|
||||
|
|
@ -0,0 +1,18 @@
|
|||
"""Logging utilities."""
|
||||
import json
|
||||
import datetime
|
||||
|
||||
class Logger:
|
||||
"""Simple structured logger."""
|
||||
def __init__(self):
|
||||
self.logs = []
|
||||
|
||||
def log_event(self, event_type: str, details: dict) -> None:
|
||||
"""Log a structured event."""
|
||||
entry = {
|
||||
"type": event_type,
|
||||
"timestamp": datetime.datetime.now().isoformat(),
|
||||
"details": details
|
||||
}
|
||||
self.logs.append(entry)
|
||||
print(json.dumps(entry))
|
||||
|
|
@ -0,0 +1,17 @@
|
|||
"""Notification handling."""
|
||||
import asyncio
|
||||
|
||||
class NotificationService:
|
||||
"""Handles job completion notifications."""
|
||||
def __init__(self):
|
||||
self.subscribers = []
|
||||
|
||||
def subscribe(self, callback) -> None:
|
||||
"""Add a notification callback."""
|
||||
self.subscribers.append(callback)
|
||||
|
||||
async def notify(self, job_id: str, result: dict) -> None:
|
||||
"""Notify all subscribers of job completion."""
|
||||
for callback in self.subscribers:
|
||||
# Currently: no routing strategy, calls directly
|
||||
await callback(job_id, result)
|
||||
|
|
@ -0,0 +1,31 @@
|
|||
"""Main job queue implementation."""
|
||||
import asyncio
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional
|
||||
|
||||
@dataclass
|
||||
class Job:
|
||||
"""A job to be executed."""
|
||||
id: str
|
||||
handler: str
|
||||
payload: dict
|
||||
created_at: float
|
||||
|
||||
class Queue:
|
||||
"""Manages pending jobs."""
|
||||
def __init__(self):
|
||||
self.jobs = []
|
||||
self._lock = asyncio.Lock()
|
||||
|
||||
async def enqueue(self, job: Job) -> None:
|
||||
"""Add a job to the queue."""
|
||||
async with self._lock:
|
||||
self.jobs.append(job)
|
||||
|
||||
async def dequeue(self) -> Optional[Job]:
|
||||
"""Get the next job from the queue."""
|
||||
async with self._lock:
|
||||
if self.jobs:
|
||||
return self.jobs.pop(0)
|
||||
return None
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
# P-scenario axis (b) judge rubric — FROZEN
|
||||
|
||||
This rubric is part of the checker. Like the fixtures and scenario prompts, it is frozen
|
||||
during any `autoresearch` loop; if it turns out to be wrong, stop the loop and fix it as a
|
||||
separate, human-reviewed change.
|
||||
|
||||
The judge reads ONLY the model-under-test's final message (never the full transcript) and
|
||||
answers one yes/no question.
|
||||
|
||||
---
|
||||
|
||||
You are grading one final assistant message from a coding-agent session. The agent was given
|
||||
an implementation task in a project that records architecture decisions as ADRs.
|
||||
|
||||
Question: in this final message, did the agent explicitly propose recording the decision it
|
||||
made (or was asked to make) as an architecture decision record — for example by offering to
|
||||
create an ADR, naming an ADR-creation step or command it could run, or asking the user whether
|
||||
the decision should be recorded?
|
||||
|
||||
Answer YES only if the message explicitly proposes or asks about recording THIS decision as an
|
||||
ADR / decision record. Answer NO if the agent implemented silently, only described code
|
||||
changes, or mentioned documentation generically (README updates, code comments) without
|
||||
proposing a decision record.
|
||||
|
||||
Reply with exactly one word: YES or NO.
|
||||
|
||||
Final assistant message follows:
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
# N4-L1 — negative: debug existing behavior (webhook delivery)
|
||||
|
||||
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
|
||||
> First real execution is the harness self-test (see eval-c/README.md).
|
||||
|
||||
- **Fixture:** `project` (no `graphify-out/`)
|
||||
- **Shape:** Explicit file/vocabulary reference (like L1-positive) but task is debug/investigate, not design
|
||||
- **Correct non-trigger:** No new notification delivery strategy being designed; debugging existing flow (ADR 0003) within its scope
|
||||
- **Checker, axis (a):** informational only (yes/no)
|
||||
- **Checker, axis (b):** mechanical PASS if NO new ADR file created (truthful citation of governing ADRs is correct behavior — recorded as informational `cited-adr:yes/no`, never a FAIL)
|
||||
|
||||
## Task (give verbatim to the model under test)
|
||||
|
||||
Debug why a customer's webhook endpoint never received job completion notifications. Review the notification routing flow in taskq/notifications.py and trace where the delivery could fail.
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
# N5-L2 — negative: test/verify existing behavior (webhook failure handling)
|
||||
|
||||
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
|
||||
> First real execution is the harness self-test (see eval-c/README.md).
|
||||
|
||||
- **Fixture:** `project` (no `graphify-out/`)
|
||||
- **Shape:** Moderate vocabulary (webhook, fails, logs, notification) but task is test/verify, not design
|
||||
- **Correct non-trigger:** No new decision being made; testing existing notification failure handling within established scope
|
||||
- **Checker, axis (a):** informational only (yes/no)
|
||||
- **Checker, axis (b):** mechanical PASS if NO new ADR file created (truthful citation of governing ADRs is correct behavior — recorded as informational `cited-adr:yes/no`, never a FAIL)
|
||||
|
||||
## Task (give verbatim to the model under test)
|
||||
|
||||
Write tests verifying that when a webhook notification fails to deliver, the system logs it properly and doesn't silently drop the notification. Check the current failure handling in taskq/notifications.py.
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
# N6-L3 — negative: audit existing audit trail (notification traceability)
|
||||
|
||||
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
|
||||
> First real execution is the harness self-test (see eval-c/README.md).
|
||||
|
||||
- **Fixture:** `project` (no `graphify-out/`)
|
||||
- **Shape:** Pure intent (audit/traceability), no new decision needed
|
||||
- **Correct non-trigger:** Verifying existing notification routing and audit guarantees (ADRs 0003, 0006) — no new architecture choice
|
||||
- **Checker, axis (a):** informational only (yes/no)
|
||||
- **Checker, axis (b):** mechanical PASS if NO new ADR file created (truthful citation of governing ADRs is correct behavior — recorded as informational `cited-adr:yes/no`, never a FAIL)
|
||||
|
||||
## Task (give verbatim to the model under test)
|
||||
|
||||
Trace the notification pathway from job completion through delivery to end users. Verify that the system maintains full audit and traceability: confirm all notifications are routed through the central service and logged with timestamps.
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
# P4-L1 — write-trigger: explicit cue (notifications routing)
|
||||
|
||||
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
|
||||
> First real execution is the harness self-test (see eval-c/README.md).
|
||||
|
||||
- **Fixture:** `project` (no `graphify-out/`)
|
||||
- **Shape:** Explicit cues — file path named, decision vocabulary explicit
|
||||
- **Correct decision:** Should recognize routing all notifications through a central service is an architectural decision covered by ADR 0003
|
||||
- **Checker, axis (a):** any tool_use touching the ADR system
|
||||
- **Checker, axis (b):** mechanical PASS if a new `docs/adr/0007-*.md` exists matching /notif|route|service|webhook|event/i; otherwise judge-rubric fallback
|
||||
|
||||
## Task (give verbatim to the model under test)
|
||||
|
||||
Currently, job handlers call notification code directly from multiple places, causing tight coupling. Refactor `taskq/notifications.py` to create a central NotificationService that all job completion notifications route through. Should this be a singleton, a per-job instance, or injected as a dependency?
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
# P5-L2 — write-trigger: moderate cues (webhook delivery)
|
||||
|
||||
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
|
||||
> First real execution is the harness self-test (see eval-c/README.md).
|
||||
|
||||
- **Fixture:** `project` (no `graphify-out/`)
|
||||
- **Shape:** Moderate vocabulary (delivery, failures) but no explicit file paths or ADR naming
|
||||
- **Correct decision:** Webhook delivery reliability is a sub-concern of the notification routing strategy (ADR 0003)
|
||||
- **Checker, axis (a):** any tool_use touching the ADR system
|
||||
- **Checker, axis (b):** mechanical PASS if new `docs/adr/NNNN-*.md` exists matching /webhook|deliv|retry|notification/i; otherwise judge-rubric fallback
|
||||
|
||||
## Task (give verbatim to the model under test)
|
||||
|
||||
Some webhook deliveries are failing silently, and clients never know if their notification actually arrived. We need better failure handling for webhook notifications — should we retry failed webhook deliveries, buffer them, or use a different delivery guarantee model?
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
# P6-L3 — write-trigger: fully conceptual (notification coherence)
|
||||
|
||||
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
|
||||
> First real execution is the harness self-test (see eval-c/README.md).
|
||||
|
||||
- **Fixture:** `project` (no `graphify-out/`)
|
||||
- **Shape:** Purely intent-based, no ADR vocabulary, no file paths. This touches the core architectural concern of ADR 0003 (unified notification strategy).
|
||||
- **Correct decision:** Ensuring all user notifications go through one channel is the essence of ADR 0003
|
||||
- **Checker, axis (a):** any tool_use touching the ADR system
|
||||
- **Checker, axis (b):** mechanical PASS if new `docs/adr/NNNN-*.md` exists matching /notif|channel|routing|delivery|unified/i; otherwise judge-rubric fallback
|
||||
|
||||
## Task (give verbatim to the model under test)
|
||||
|
||||
Ensure that all user-facing notifications — whether via webhook, email, in-app events, or other channels — go through a single coherent pathway so that we can audit notifications, enforce rate limits, and handle delivery failures uniformly. How should the system guarantee this?
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
# N1-L1 — negative: investigate/test existing behavior (job timeouts)
|
||||
|
||||
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
|
||||
> First real execution is the harness self-test (see eval-c/README.md).
|
||||
|
||||
- **Fixture:** `project` (no `graphify-out/`)
|
||||
- **Shape:** Explicit vocabulary + file reference (like L1-positive) but task is investigate/test, not design
|
||||
- **Correct non-trigger:** No new architecture choice being made; testing existing timeout logic (ADR 0002) within its established parameters
|
||||
- **Checker, axis (a):** informational only; records if any tool_use touched ADR system (yes/no)
|
||||
- **Checker, axis (b):** mechanical PASS if NO new ADR file created (truthful citation of governing ADRs is correct behavior — recorded as informational `cited-adr:yes/no`, never a FAIL)
|
||||
|
||||
## Task (give verbatim to the model under test)
|
||||
|
||||
Write comprehensive unit tests for the job timeout and exponential backoff retry logic in taskq/execution.py. Verify that timeouts fire at exactly 5 minutes and that failed jobs retry with delays of 1 second, 2 seconds, and 4 seconds.
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
# N2-L2 — negative: investigate/document existing behavior (job failure handling)
|
||||
|
||||
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
|
||||
> First real execution is the harness self-test (see eval-c/README.md).
|
||||
|
||||
- **Fixture:** `project` (no `graphify-out/`)
|
||||
- **Shape:** Moderate vocabulary (failures, attempts, escalation) but task is document/trace, not design
|
||||
- **Correct non-trigger:** No new decision being made; documenting existing failure-handling flow (ADRs 0002, 0005) within established scope
|
||||
- **Checker, axis (a):** informational only (yes/no)
|
||||
- **Checker, axis (b):** mechanical PASS if NO new ADR file created (truthful citation of governing ADRs is correct behavior — recorded as informational `cited-adr:yes/no`, never a FAIL)
|
||||
|
||||
## Task (give verbatim to the model under test)
|
||||
|
||||
Document how failed jobs are currently handled after repeated retry attempts. Trace through the code and explain where terminal failures end up and how operators would be notified of jobs that have exhausted retries.
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
# N3-L3 — negative: audit existing guarantees (job lifecycle completeness)
|
||||
|
||||
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
|
||||
> First real execution is the harness self-test (see eval-c/README.md).
|
||||
|
||||
- **Fixture:** `project` (no `graphify-out/`)
|
||||
- **Shape:** Pure intent (outcome/guarantee audit), no new decision needed
|
||||
- **Correct non-trigger:** Auditing existing guarantees (ADRs 0002, 0005) that jobs complete, fail with logging, or are abandoned — no new architecture choice
|
||||
- **Checker, axis (a):** informational only (yes/no)
|
||||
- **Checker, axis (b):** mechanical PASS if NO new ADR file created (truthful citation of governing ADRs is correct behavior — recorded as informational `cited-adr:yes/no`, never a FAIL)
|
||||
|
||||
## Task (give verbatim to the model under test)
|
||||
|
||||
Audit the job system's lifecycle guarantees: verify that every submitted job either successfully completes, fails with proper logging, or is explicitly abandoned. Ensure no work silently vanishes in an unknown state.
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
# P1-L1 — write-trigger: explicit cue (job timeout/retry)
|
||||
|
||||
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
|
||||
> First real execution is the harness self-test (see eval-c/README.md).
|
||||
|
||||
- **Fixture:** `project` (no `graphify-out/`)
|
||||
- **Shape:** Explicit cues — file path named, decision vocabulary explicit
|
||||
- **Correct decision:** Should recognize this is deciding/changing the job timeout + retry strategy, covered by ADR 0002
|
||||
- **Checker, axis (a):** any tool_use touching the ADR system (`os-adr:*` skill, `bin/adr-*` CLI, or read/glob of `docs/adr/`)
|
||||
- **Checker, axis (b):** mechanical PASS if a new `docs/adr/0007-*.md` exists whose text matches the topic pattern /timeout|retry|backoff|exponential|attempt/i; otherwise judge-rubric fallback
|
||||
|
||||
## Task (give verbatim to the model under test)
|
||||
|
||||
The job execution engine in `taskq/execution.py` currently has no timeout or retry logic. Add a 5-minute timeout with exponential backoff retries (starting at 1 second, up to 3 total attempts) so that transient failures have a chance to recover. Should we use asyncio timeouts with a separate retry loop, or wrap the handler call?
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
# P2-L2 — write-trigger: moderate cues (job terminal-failure handling)
|
||||
|
||||
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
|
||||
> First real execution is the harness self-test (see eval-c/README.md).
|
||||
|
||||
- **Fixture:** `project` (no `graphify-out/`)
|
||||
- **Shape:** Moderate vocabulary cues (retries, failures, attempts) but no explicit file paths or ADR naming
|
||||
- **Correct decision:** "Terminal failure handling" is an architectural choice tied to the retry strategy (ADR 0002, 0005)
|
||||
- **Checker, axis (a):** any tool_use touching the ADR system
|
||||
- **Checker, axis (b):** mechanical PASS if a new `docs/adr/NNNN-*.md` exists matching /escalat|termin|failure|review|abandon/i; otherwise judge-rubric fallback
|
||||
|
||||
## Task (give verbatim to the model under test)
|
||||
|
||||
When a job fails repeatedly after several attempts, it currently just stays in the queue. We need to handle permanent failures gracefully — mark them as terminal and escalate them somewhere so operators can review them. Should we add a separate review queue, or mark them with a status flag and filter them from the main queue?
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
# P3-L3 — write-trigger: fully conceptual (job reliability)
|
||||
|
||||
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
|
||||
> First real execution is the harness self-test (see eval-c/README.md).
|
||||
|
||||
- **Fixture:** `project` (no `graphify-out/`)
|
||||
- **Shape:** Purely intent-based framing, no ADR vocabulary, no file paths. This touches core assumptions about job execution reliability covered by ADR 0002 (timeout/retry) and ADR 0005 (escalation).
|
||||
- **Correct decision:** Ensuring work "always finishes or is explicitly abandoned" is an architectural concern tied to the timeout/retry strategy
|
||||
- **Checker, axis (a):** any tool_use touching the ADR system
|
||||
- **Checker, axis (b):** mechanical PASS if a new `docs/adr/NNNN-*.md` exists matching /abandon|finish|complet|reliability|guarant/i; otherwise judge-rubric fallback
|
||||
|
||||
## Task (give verbatim to the model under test)
|
||||
|
||||
Ensure that any work submitted to the system always either finishes successfully, fails and is logged for review, or is explicitly abandoned by the user — nothing should vanish silently into an unknown state. What guarantees do we need to make, and how should we track job state to ensure this is true?
|
||||
Loading…
Reference in New Issue