From 2e9d83f7490f1062924554fa9947e476c4a1249a Mon Sep 17 00:00:00 2001 From: jared Date: Mon, 6 Jul 2026 12:49:42 -0400 Subject: [PATCH] Record Eval C frozen grid run: 12/12 cells PASS, Eval B wording generalizes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First full run of the Eval C discrimination harness (3-level ambiguity ladder, 6-scenario run-set, 3 reps/cell, both sonnet+haiku): 36 headless reps, 12/12 cells PASS at ≥2/3 majority. Haiku 18/18 reps, sonnet 17/18 (P1-L1 rep 1: final message citing only 0005, not governing 0002 — a boundary artifact, not a behavioral miss). Zero over-trigger ADR creation on any negative at any level; all positive passes via governing-ADR citation (16 reps) or judge fallback (1). Two harness defects fixed before running the grid: (1) bin/run pipefail bug that aborted remaining reps after first FAIL (changed checker invocation to || true); (2) positive axis-b previously required ADR creation only, contradicting the README's design statement that "consulting and/or recording is correct" — run-set positives sit in already-decided territory, so find→cite→comply is the correct behavior. Implemented three-branch axis-b check: created (new ADR file), cited-governing (final message cites the governing Accepted ADR), or judge fallback. Added two self-test guards to verify the compliance path. Scenario metadata updated to document the corrected axis-b checker behavior (cited-governing as a valid pass branch). TSV reasons column now records the pass path via `B-via:` field for positives. Verdict: Eval B wording generalizes to a new fixture, new language, new domain, and decreasing cue explicitness. Grid ceiling not found; next signal is production (rollout + IRL session audits). Run-set baseline locked; wording tuning moves measurement to reserve-set. --- CLAUDE.md | 3 +- plugins/os-adr/eval-c/README.md | 18 ++- plugins/os-adr/eval-c/bin/check | 149 ++++++------------ plugins/os-adr/eval-c/bin/run | 4 +- plugins/os-adr/eval-c/bin/self-test | 28 ++++ .../eval-c/results/2026-07-06-frozen-grid.tsv | 36 +++++ .../eval-c/scenarios/P1-L1-execution.md | 2 +- .../eval-c/scenarios/P2-L2-execution.md | 2 +- .../eval-c/scenarios/P3-L3-execution.md | 2 +- 9 files changed, 132 insertions(+), 112 deletions(-) create mode 100644 plugins/os-adr/eval-c/results/2026-07-06-frozen-grid.tsv diff --git a/CLAUDE.md b/CLAUDE.md index da208cf..b975d6e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -103,7 +103,8 @@ to those two and fix the stale doc. - **Initial iteration complete (2026-07-03):** plugin build, migration pilot, Eval A grid (clean pass), and Eval B grid (baseline captured above) are all done — this closes the first pass on os-adr. Follow-on work is deliberately deferred to future sessions, not in-flight. - **Wording experiment complete (2026-07-04):** the follow-up `/autoresearch` loop over Eval B wording ran (5 iterations, checker/fixtures/scenarios/rubric frozen; fixture CLAUDE.md declared a wording surface upfront) and closed the gap — final full grid **sonnet 8/8, haiku 7/8** (from 5/8 / 0/8 baseline; haiku's one miss is a W3 axis-b judge-boundary flicker). Winning wording shipped in `hooks/session_start.py` (PRESENT_NOTE), find/create SKILL.mds, and `eval-b/fixture/project/CLAUDE.md` (new — the trigger-phrased "Architecture decisions" section, the template for real-project adoption). Confirmed mechanisms: trigger-conditioned when→then phrasing beats inventory phrasing on both tiers; each rule must live where its precondition is visible (the reversal→supersede rule in the find skill's act-on-findings step fixed W3); lower tiers need *mechanical* triggers ("before your first edit to any existing file → run `/os-adr:find` on those paths; additions count") — semantic triggers ("architecture-level choice") only reach sonnet. Open: channel ablation never run (hook vs CLAUDE.md redundancy unknown); R4-nograph now passes both tiers, so the graph-degradation check no longer differentiates. Full hypothesis→result mapping: vault note [[os-adr-eval-b-wording-experiment-hypotheses]]. **Before designing or running any autoresearch eval, Read `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md`.** - **Eval B W3 stability check (2026-07-06):** 7 headless reps of haiku × W3 (all reps counted, including two the runner initially excluded, to avoid peeking bias): axis (a) consultation 7/7 PASS; axis (b) recording-offer ~5/7 (~75–85%) — one FAIL plausibly infra, one a genuine behavioral miss (consulted, then asked a clarifying question instead of unconditionally proposing the superseding ADR; the conditional-phrasing failure mode iteration 3 fixed on sonnet, improved but not eliminated on haiku). Verdict: intermittent flicker, not a hard gap — grid claim stays **haiku 7/8**, W3 characterized as an ~1-in-4/5 axis-b miss. Haiku cleared for daily use with that caveat. Recorded in the vault good-enough gate: [[os-adr-eval-b-wording-experiment-hypotheses]]. -- **Eval C harness built, NOT yet run (2026-07-06):** `plugins/os-adr/eval-c/` — ambiguity-ladder DISCRIMINATION eval (held-out; Eval B is contaminated by the wording loop). 3 levels (explicit → moderate → conceptual framing) × paired positive/negative scenarios; run-set (6, job-execution domain) + frozen reserve-set (6, notifications domain — becomes the measurement set if anyone ever tunes wording against the run-set). New Python fixture (`taskq` async job queue, 6-ADR history generated via the plugin's own CLIs, trigger-phrased CLAUDE.md section copied from eval-b). Scoring: positives = both axes; negatives FAIL **only on unneeded ADR creation** — consultation and truthful ADR citation are informational (`A:yes/no`, `cited-adr:yes/no`), never FAILs, because the shipped wording endorses cheap finds and the negatives sit deliberately in ADR-covered domains (two earlier negative designs that punished instruction-compliant behavior were caught in review and redesigned/fixed). Model-free `bin/self-test` green, including a truthful-citation-must-PASS guard. **Scenario Task blocks are held-out — never run informally.** First real grid run is a pending decision. Methodology: vault notes [[eval-methodology-ladder]] (per-level pass bars, reserve discipline) and [[eval-methodology-irl-feedback-loop]] (post-rollout session-audit backlog); vault also gained an `eval-results` note type + `_templates/eval-results.md` (2026-07-06). +- **Eval C harness built, NOT yet run (2026-07-06):** `plugins/os-adr/eval-c/` — ambiguity-ladder DISCRIMINATION eval (held-out; Eval B is contaminated by the wording loop). 3 levels (explicit → moderate → conceptual framing) × paired positive/negative scenarios; run-set (6, job-execution domain) + frozen reserve-set (6, notifications domain — becomes the measurement set if anyone ever tunes wording against the run-set). New Python fixture (`taskq` async job queue, 6-ADR history generated via the plugin's own CLIs, trigger-phrased CLAUDE.md section copied from eval-b). Scoring: positives = both axes; negatives FAIL **only on unneeded ADR creation** — consultation and truthful ADR citation are informational (`A:yes/no`, `cited-adr:yes/no`), never FAILs, because the shipped wording endorses cheap finds and the negatives sit deliberately in ADR-covered domains (two earlier negative designs that punished instruction-compliant behavior were caught in review and redesigned/fixed). Model-free `bin/self-test` green, including a truthful-citation-must-PASS guard. **Scenario Task blocks are held-out — never run informally.** Methodology: vault notes [[eval-methodology-ladder]] (per-level pass bars, reserve discipline) and [[eval-methodology-irl-feedback-loop]] (post-rollout session-audit backlog); vault also gained an `eval-results` note type + `_templates/eval-results.md` (2026-07-06). +- **Eval C frozen grid run (2026-07-06):** run-set × {sonnet, haiku} × 3 reps/cell (36 headless reps, all counted) — **12/12 cells PASS at ≥2/3 majority; haiku 18/18 reps, sonnet 17/18**. Zero over-trigger ADR creation on any negative at any level; every positive pass was via governing-ADR citation (16) or judge-recognized proposal (1), never duplicate creation. Sonnet's one FAIL rep (P1-L1) is a final-message-citation boundary artifact (implemented the ADR-0002 policy, cited only 0005), not a behavioral miss. Two harness defects fixed on the first live rep, BEFORE the grid, with Task blocks/fixtures/rubric untouched: (1) `bin/run` pipefail bug aborted a cell's reps after the first FAIL; (2) positive axis-b required creation/proposal, contradicting the README's own "consulting and/or recording is correct" — the run-set positives sit in *already-decided* territory, so find→cite→comply is correct; fixed via a mechanical `cited-governing` PASS branch + two new self-test guards (`B-via:` now recorded in TSV). Verdict: the Eval B wording **generalizes** (new fixture/language/domain, decreasing cue explicitness, paired negatives) — grid saturated, ceiling not found; next signal is production (rollout + IRL session audits), not another lab rung. Run-set is now the baseline; wording tuning against it moves measurement to the reserve-set. Full results: vault note [[os-adr-eval-c-frozen-grid-results]]. - **Remaining (locked rollout order):** real-project migration/adoption one at a time via `/os-adr:migrate` — pilot projects first, then the cc-os retrofit (its 19-ADR monolithic file was deliberately excluded from the pilot), then wider. When onboarding real projects, add the trigger-phrased "Architecture decisions" CLAUDE.md section (copy from `eval-b/fixture/project/CLAUDE.md`; candidate: emit it from `/os-adr:init`/`migrate`). - **Resolved (2026-07-04):** os-adr's skills failed to register on first install due to stale plugin caches — the cache that was installed on 2026-07-03 17:21:48 was missing `hooks/hooks.json`, `bin/adr-detect`, `bin/adr-find`, `bin/adr-migrate`. Root cause: unknown, but the caches were restored via plugin uninstall/reinstall. A parallel issue affected os-doc-hygiene (cache retained deleted `commands/` directory). Fixed by manually refreshing both caches, then created `bin/refresh-plugins` automation to prevent future drift — see "Editing a local plugin (cache refresh)" subsection for refresh procedure. Investigation result: stale caches (not manifest-naming issues) were the cause; slash command registration works correctly once caches are fresh. diff --git a/plugins/os-adr/eval-c/README.md b/plugins/os-adr/eval-c/README.md index 967fe41..600dabd 100644 --- a/plugins/os-adr/eval-c/README.md +++ b/plugins/os-adr/eval-c/README.md @@ -109,11 +109,19 @@ markdown, so any future graph build (not part of this eval) would be pure-AST / axes. - **Axis (b) — specific correctness** - - **Positive scenarios** (write-trigger): - - Mechanical PASS when a new `docs/adr/0007-*.md` exists and its text matches the - scenario's topic pattern (regex per scenario class). - - Otherwise: judge fallback (haiku, `judge-rubric.md`, reads only final message; - stubbed via `ADR_EVAL_C_JUDGE_CMD` for self-tests). + - **Positive scenarios** (decided-constraint-applies): PASS via any ONE of, in order: + 1. **created** — a new `docs/adr/NNNN-*.md` exists matching the scenario's topic pattern. + 2. **cited-governing** — the final message cites the governing Accepted ADR by id + (P1→0002, P2→0005, P3→0002/0005/0006; reserve P4–P6→0003). The positives sit in + already-decided territory (each task restates what an Accepted ADR already decided), + so per the shipped plugin wording the correct behavior is find → cite → comply, NOT + create a duplicate ADR. This branch conforms the checker to this README's design + statement ("consulting `/os-adr:find` and/or recording with `/os-adr:create` is + correct"); the original checker required creation/proposal only, which failed exactly + the behavior the negatives reward (fixed 2026-07-06 after 1 spent rep, before the grid). + The TSV reasons column records which branch passed (`B-via:created|cited-governing|judge-proposed`). + 3. **judge fallback** (haiku, `judge-rubric.md` — frozen, unchanged — reads only the final + message; stubbed via `ADR_EVAL_C_JUDGE_CMD` for self-tests). - **Negative scenarios** (non-decisional tasks, investigate/test/debug framing): - Mechanical FAIL if a new ADR file was created (over-trigger: creating a decision record for a non-decisional task). This is the ONLY fail condition. diff --git a/plugins/os-adr/eval-c/bin/check b/plugins/os-adr/eval-c/bin/check index 61f417d..308fcbd 100755 --- a/plugins/os-adr/eval-c/bin/check +++ b/plugins/os-adr/eval-c/bin/check @@ -173,138 +173,80 @@ module AdrEvalC # --- Positive scenarios: should trigger ADR system --- class PositiveScenario < Scenario + # Axis (b) passes via any ONE of, in order: + # 1. created — a new topical ADR file exists (the model recorded a decision) + # 2. cited — the final message cites the GOVERNING Accepted ADR by id. + # The run-set positives sit in already-decided territory (the task restates + # what an Accepted ADR already decided), so per the shipped plugin wording the + # correct behavior is find -> cite -> comply, NOT create a duplicate ADR. + # This branch conforms the checker to the documented design ("consulting + # and/or recording is correct"); without it the checker fails exactly the + # behavior the negatives reward. + # 3. judge — frozen rubric fallback: final message proposes recording. + attr_reader :b_via + private def check_axis_a(result, transcript) touched = transcript.tool_uses.any? { |tu| adr_touch?(tu) } result.expect(touched, "no tool_use touched the ADR system (os-adr:*, bin/adr-*, docs/adr/)") end + + def check_axis_b(result, transcript) + new_files = sb.new_adr_files + if new_files.any? + matching = new_files.select { |p| File.read(p).match?(topic) } + @b_via = "created" if matching.any? + result.expect(matching.any?, + "new ADR exists but matches no topic term #{topic.inspect}") + return + end + final = transcript.final_message + unless final + result.expect(false, "no new ADR file and no final message to judge") + return + end + if final.match?(governing) + @b_via = "cited-governing" + return + end + if judge.proposed_adr?(final) + @b_via = "judge-proposed" + else + result.expect(false, + "no new ADR, no governing-ADR citation #{governing.inspect} in final message; judge: does not propose recording") + end + end end class P1L1Execution < PositiveScenario - def check_axis_b(result, transcript) - new_files = sb.new_adr_files - if new_files.any? - matching = new_files.select { |p| File.read(p).match?(topic) } - result.expect(matching.any?, - "new ADR exists but matches no topic term #{topic.inspect}") - return - end - final = transcript.final_message - unless final - result.expect(false, "no new ADR file and no final message to judge") - return - end - result.expect(judge.proposed_adr?(final), - "no new ADR file; judge: final message does not propose/ask to record the decision") - end - def topic = /timeout|retry|backoff|exponential|attempt/i + def governing = /\b0002\b/ end class P2L2Execution < PositiveScenario - def check_axis_b(result, transcript) - new_files = sb.new_adr_files - if new_files.any? - matching = new_files.select { |p| File.read(p).match?(topic) } - result.expect(matching.any?, - "new ADR exists but matches no topic term #{topic.inspect}") - return - end - final = transcript.final_message - unless final - result.expect(false, "no new ADR file and no final message to judge") - return - end - result.expect(judge.proposed_adr?(final), - "no new ADR file; judge: final message does not propose/ask to record the decision") - end - def topic = /escalat|termin|failure|review|abandon/i + def governing = /\b0005\b/ end class P3L3Execution < PositiveScenario - def check_axis_b(result, transcript) - new_files = sb.new_adr_files - if new_files.any? - matching = new_files.select { |p| File.read(p).match?(topic) } - result.expect(matching.any?, - "new ADR exists but matches no topic term #{topic.inspect}") - return - end - final = transcript.final_message - unless final - result.expect(false, "no new ADR file and no final message to judge") - return - end - result.expect(judge.proposed_adr?(final), - "no new ADR file; judge: final message does not propose/ask to record the decision") - end - def topic = /abandon|finish|complet|reliability|guarant/i + def governing = /\b000[256]\b/ end class P4L1Notifications < PositiveScenario - def check_axis_b(result, transcript) - new_files = sb.new_adr_files - if new_files.any? - matching = new_files.select { |p| File.read(p).match?(topic) } - result.expect(matching.any?, - "new ADR exists but matches no topic term #{topic.inspect}") - return - end - final = transcript.final_message - unless final - result.expect(false, "no new ADR file and no final message to judge") - return - end - result.expect(judge.proposed_adr?(final), - "no new ADR file; judge: final message does not propose/ask to record the decision") - end - def topic = /notif|route|service|webhook|event/i + def governing = /\b0003\b/ end class P5L2Notifications < PositiveScenario - def check_axis_b(result, transcript) - new_files = sb.new_adr_files - if new_files.any? - matching = new_files.select { |p| File.read(p).match?(topic) } - result.expect(matching.any?, - "new ADR exists but matches no topic term #{topic.inspect}") - return - end - final = transcript.final_message - unless final - result.expect(false, "no new ADR file and no final message to judge") - return - end - result.expect(judge.proposed_adr?(final), - "no new ADR file; judge: final message does not propose/ask to record the decision") - end - def topic = /webhook|deliv|retry|notification/i + def governing = /\b0003\b/ end class P6L3Notifications < PositiveScenario - def check_axis_b(result, transcript) - new_files = sb.new_adr_files - if new_files.any? - matching = new_files.select { |p| File.read(p).match?(topic) } - result.expect(matching.any?, - "new ADR exists but matches no topic term #{topic.inspect}") - return - end - final = transcript.final_message - unless final - result.expect(false, "no new ADR file and no final message to judge") - return - end - result.expect(judge.proposed_adr?(final), - "no new ADR file; judge: final message does not propose/ask to record the decision") - end - def topic = /notif|channel|routing|delivery|unified/i + def governing = /\b0003\b/ end # --- Negative scenarios: should NOT create false ADRs or assert phantom constraints --- @@ -430,6 +372,9 @@ reasons = (axis_a.failures + axis_b.failures).join("; ") if is_negative && checker.respond_to?(:citation_value) && checker.citation_value reasons = ["cited-adr:#{checker.citation_value}", reasons].reject(&:empty?).join("; ") end +if !is_negative && checker.respond_to?(:b_via) && checker.b_via + reasons = ["B-via:#{checker.b_via}", reasons].reject(&:empty?).join("; ") +end if tsv_model puts [scenario_id, tsv_model, diff --git a/plugins/os-adr/eval-c/bin/run b/plugins/os-adr/eval-c/bin/run index 87c2701..612e08d 100755 --- a/plugins/os-adr/eval-c/bin/run +++ b/plugins/os-adr/eval-c/bin/run @@ -51,5 +51,7 @@ for rep in $(seq 1 "$REPS"); do --dangerously-skip-permissions \ "$TASK" > transcript.jsonl) || echo "claude exited non-zero for $SANDBOX" >&2 - "$EVAL_ROOT/bin/check" "$SCENARIO" "$SANDBOX" --tsv "$MODEL" | tee -a "$RESULTS" + # bin/check exits 1 on FAIL; without the || true, set -euo pipefail would + # abort the rep loop on the first failing rep (found on the first grid run). + "$EVAL_ROOT/bin/check" "$SCENARIO" "$SANDBOX" --tsv "$MODEL" | tee -a "$RESULTS" || true done diff --git a/plugins/os-adr/eval-c/bin/self-test b/plugins/os-adr/eval-c/bin/self-test index 3515d52..2c09767 100755 --- a/plugins/os-adr/eval-c/bin/self-test +++ b/plugins/os-adr/eval-c/bin/self-test @@ -162,6 +162,34 @@ sb="$WORKDIR/truthful-citation-N1" expect PASS "truthful-citation N1 (cites governing ADR 0002 while testing — no ADR created)" \ "$EVAL_ROOT/bin/check" N1-L1-execution "$sb" +echo "" +echo "Testing POSITIVE compliance path (cites governing ADR, creates nothing — must PASS without judge):" +# The run-set positives sit in already-decided territory; find -> cite -> comply is +# correct per the shipped wording. Judge stubbed NO to prove the pass is mechanical. +sb="$WORKDIR/compliance-P1-L1" +"$EVAL_ROOT/bin/sandbox" P1-L1-execution "$sb" >/dev/null +{ hook_event + tool_use_event "Bash" "$FIND_INPUT" + result_event "Implemented the 5-minute timeout with exponential backoff per ADR 0002, which already records this decision. No new ADR needed." +} > "$sb/transcript.jsonl" +expect PASS "compliance P1-L1 (cites governing ADR 0002, no ADR created, judge stubbed NO)" \ + env ADR_EVAL_C_JUDGE_CMD="printf NO" "$EVAL_ROOT/bin/check" P1-L1-execution "$sb" +row="$(env ADR_EVAL_C_JUDGE_CMD="printf NO" "$EVAL_ROOT/bin/check" P1-L1-execution "$sb" --tsv selftest || true)" +case "$row" in + *B-via:cited-governing*) echo "ok compliance P1-L1 passed via cited-governing branch" ;; + *) echo "FAIL compliance P1-L1: expected B-via:cited-governing in TSV: $row"; FAILED=1 ;; +esac + +# A wrong-ADR citation must NOT satisfy the governing branch (falls through to judge) +sb="$WORKDIR/wrongcite-P1-L1" +"$EVAL_ROOT/bin/sandbox" P1-L1-execution "$sb" >/dev/null +{ hook_event + tool_use_event "Bash" "$FIND_INPUT" + result_event "Implemented timeouts. See ADR 0004 for the SQLite persistence design." +} > "$sb/transcript.jsonl" +expect FAIL "wrong-cite P1-L1 (cites non-governing ADR 0004 only, judge stubbed NO)" \ + env ADR_EVAL_C_JUDGE_CMD="printf NO" "$EVAL_ROOT/bin/check" P1-L1-execution "$sb" + # Missing hook context should invalidate both positive and negative echo "" echo "Testing hook-context precondition (invalid run):" diff --git a/plugins/os-adr/eval-c/results/2026-07-06-frozen-grid.tsv b/plugins/os-adr/eval-c/results/2026-07-06-frozen-grid.tsv new file mode 100644 index 0000000..e249d5f --- /dev/null +++ b/plugins/os-adr/eval-c/results/2026-07-06-frozen-grid.tsv @@ -0,0 +1,36 @@ +N1-L1-execution haiku A:yes B:PASS PASS +N1-L1-execution haiku A:yes B:PASS PASS +N1-L1-execution haiku A:yes B:PASS PASS +N1-L1-execution sonnet A:yes B:PASS PASS +N1-L1-execution sonnet A:yes B:PASS PASS +N1-L1-execution sonnet A:yes B:PASS PASS +N2-L2-execution haiku A:yes B:PASS PASS +N2-L2-execution haiku A:yes B:PASS PASS +N2-L2-execution haiku A:yes B:PASS PASS +N2-L2-execution sonnet A:no B:PASS PASS +N2-L2-execution sonnet A:yes B:PASS PASS +N2-L2-execution sonnet A:yes B:PASS PASS +N3-L3-execution haiku A:yes B:PASS PASS +N3-L3-execution haiku A:yes B:PASS PASS +N3-L3-execution haiku A:yes B:PASS PASS +N3-L3-execution sonnet A:yes B:PASS PASS +N3-L3-execution sonnet A:yes B:PASS PASS +N3-L3-execution sonnet A:yes B:PASS PASS +P1-L1-execution haiku A:PASS B:PASS PASS B-via:cited-governing +P1-L1-execution haiku A:PASS B:PASS PASS B-via:cited-governing +P1-L1-execution haiku A:PASS B:PASS PASS B-via:cited-governing +P1-L1-execution sonnet A:PASS B:FAIL FAIL no new ADR, no governing-ADR citation /\b0002\b/ in final message; judge: does not propose recording +P1-L1-execution sonnet A:PASS B:PASS PASS B-via:cited-governing +P1-L1-execution sonnet A:PASS B:PASS PASS B-via:cited-governing +P2-L2-execution haiku A:PASS B:PASS PASS B-via:cited-governing +P2-L2-execution haiku A:PASS B:PASS PASS B-via:cited-governing +P2-L2-execution haiku A:PASS B:PASS PASS B-via:judge-proposed +P2-L2-execution sonnet A:PASS B:PASS PASS B-via:cited-governing +P2-L2-execution sonnet A:PASS B:PASS PASS B-via:cited-governing +P2-L2-execution sonnet A:PASS B:PASS PASS B-via:cited-governing +P3-L3-execution haiku A:PASS B:PASS PASS B-via:cited-governing +P3-L3-execution haiku A:PASS B:PASS PASS B-via:cited-governing +P3-L3-execution haiku A:PASS B:PASS PASS B-via:cited-governing +P3-L3-execution sonnet A:PASS B:PASS PASS B-via:cited-governing +P3-L3-execution sonnet A:PASS B:PASS PASS B-via:cited-governing +P3-L3-execution sonnet A:PASS B:PASS PASS B-via:cited-governing diff --git a/plugins/os-adr/eval-c/scenarios/P1-L1-execution.md b/plugins/os-adr/eval-c/scenarios/P1-L1-execution.md index 22ce697..bd96730 100644 --- a/plugins/os-adr/eval-c/scenarios/P1-L1-execution.md +++ b/plugins/os-adr/eval-c/scenarios/P1-L1-execution.md @@ -7,7 +7,7 @@ - **Shape:** Explicit cues — file path named, decision vocabulary explicit - **Correct decision:** Should recognize this is deciding/changing the job timeout + retry strategy, covered by ADR 0002 - **Checker, axis (a):** any tool_use touching the ADR system (`os-adr:*` skill, `bin/adr-*` CLI, or read/glob of `docs/adr/`) -- **Checker, axis (b):** mechanical PASS if a new `docs/adr/0007-*.md` exists whose text matches the topic pattern /timeout|retry|backoff|exponential|attempt/i; otherwise judge-rubric fallback +- **Checker, axis (b):** PASS via any of: new topical ADR created (/timeout|retry|backoff|exponential|attempt/i); final message cites governing ADR 0002 (find→cite→comply is correct — the decision is already recorded); judge-rubric fallback ## Task (give verbatim to the model under test) diff --git a/plugins/os-adr/eval-c/scenarios/P2-L2-execution.md b/plugins/os-adr/eval-c/scenarios/P2-L2-execution.md index b7de184..fe66338 100644 --- a/plugins/os-adr/eval-c/scenarios/P2-L2-execution.md +++ b/plugins/os-adr/eval-c/scenarios/P2-L2-execution.md @@ -7,7 +7,7 @@ - **Shape:** Moderate vocabulary cues (retries, failures, attempts) but no explicit file paths or ADR naming - **Correct decision:** "Terminal failure handling" is an architectural choice tied to the retry strategy (ADR 0002, 0005) - **Checker, axis (a):** any tool_use touching the ADR system -- **Checker, axis (b):** mechanical PASS if a new `docs/adr/NNNN-*.md` exists matching /escalat|termin|failure|review|abandon/i; otherwise judge-rubric fallback +- **Checker, axis (b):** PASS via any of: new topical ADR created (/escalat|termin|failure|review|abandon/i); final message cites governing ADR 0005 (find→cite→comply is correct — the decision is already recorded); judge-rubric fallback ## Task (give verbatim to the model under test) diff --git a/plugins/os-adr/eval-c/scenarios/P3-L3-execution.md b/plugins/os-adr/eval-c/scenarios/P3-L3-execution.md index d31a1a4..9abb22f 100644 --- a/plugins/os-adr/eval-c/scenarios/P3-L3-execution.md +++ b/plugins/os-adr/eval-c/scenarios/P3-L3-execution.md @@ -7,7 +7,7 @@ - **Shape:** Purely intent-based framing, no ADR vocabulary, no file paths. This touches core assumptions about job execution reliability covered by ADR 0002 (timeout/retry) and ADR 0005 (escalation). - **Correct decision:** Ensuring work "always finishes or is explicitly abandoned" is an architectural concern tied to the timeout/retry strategy - **Checker, axis (a):** any tool_use touching the ADR system -- **Checker, axis (b):** mechanical PASS if a new `docs/adr/NNNN-*.md` exists matching /abandon|finish|complet|reliability|guarant/i; otherwise judge-rubric fallback +- **Checker, axis (b):** PASS via any of: new topical ADR created (/abandon|finish|complet|reliability|guarant/i); final message cites a governing ADR (0002/0005/0006); judge-rubric fallback ## Task (give verbatim to the model under test)