diff --git a/2026-07-03-os-adr-eval-b-grid-results-and-observations.md b/2026-07-03-os-adr-eval-b-grid-results-and-observations.md index 6b0e210..0054f1d 100644 --- a/2026-07-03-os-adr-eval-b-grid-results-and-observations.md +++ b/2026-07-03-os-adr-eval-b-grid-results-and-observations.md @@ -1,27 +1,30 @@ --- -summary: os-adr Eval B (unprompted write-trigger & retrieval) grid results for haiku/sonnet — haiku never self-triggers the ADR system, sonnet passes 5/8 with two near-miss failures (confirmed by 2026-07-04 re-run) — read before designing any follow-up eval on unprompted-consultation behavior. -last_updated: 2026-07-04 +type: eval-results +title: os-adr Eval B (unprompted write-trigger & retrieval) grid results (2026-07-03) +summary: Haiku never self-triggers the ADR system unprompted; sonnet passes 5/8 with two near-miss failures confirmed by 2026-07-04 re-run — baseline for follow-up wording optimization. tags: - - scope/global - - type/reference - - project/cc-os - - tool/os-adr + - type/eval-results - domain/llm-evaluation -source: cc-os + - tool/os-adr + - project/cc-os +scope: global +last_updated: 2026-07-04 date: 2026-07-03 +related: + - os-adr-eval-b-wording-experiment-hypotheses + - running-autoresearch-skill-evals +source: cc-os --- -# os-adr Eval B grid results (2026-07-03) +# os-adr Eval B grid results (2026-07-03 baseline) -Eval B measures whether a model notices **on its own** that an Architecture Decision Record -system is relevant — without being explicitly told to invoke it. Two axes per scenario: (a) -did the model touch the ADR system unprompted (mechanical, from tool_use), (b) was the specific -outcome correct (right ADR cited, or a new ADR actually proposed/created). See -`cc-os/plugins/os-adr/eval-b/README.md` for full harness design (headless-only `claude -p` runs, -so a real SessionStart hook fires — in-session Agent-tool subagents cannot validly stand in for -this, since they don't get a fresh hook invocation against the sandbox cwd). +## Results Grid/Threshold -## Grid (1 rep/cell) +Eval B measures whether a model notices **on its own** that an Architecture Decision Record system is relevant — without being explicitly told to invoke it. + +**Passing criterion:** Model must (a) touch the ADR system unprompted (mechanical detection from tool_use), and (b) produce correct outcome (right ADR cited for retrieval, or new ADR actually proposed for writes). For the 2026-07-03 baseline, passing is per-tier all-scenarios aggregate before any wording optimization. + +**Grid (1 rep/cell):** | Scenario | Haiku | Sonnet | |---|---|---| @@ -34,70 +37,63 @@ this, since they don't get a fresh hook invocation against the sandbox cwd). | R4 (graph one-hop) | FAIL | **PASS** | | R4-nograph (degradation check) | FAIL (expected) | FAIL (expected) | -**Haiku: 0/8.** Never touches `os-adr:*` / `bin/adr-*` / `docs/adr/` unprompted in any -scenario — total absence of self-triggered consultation, not a near-miss. +**Summary:** Haiku 0/8 (never unprompted-consults the ADR system in any scenario). Sonnet 5/8 (failures: W3 reversal / R1 direct-conflict retrieval). R4-nograph degradation checks failed as expected — only meaningful paired with R4 PASS on the same tier. -**Sonnet: 5/8.** Two real misses: W3 (reversal — doesn't propose recording the decision) and R1 -(direct-conflict retrieval — doesn't catch it and cite the correct ADR). +## Measurement Setup -`R4-nograph` failing on both tiers is the **expected** result — it's a degradation check, only -meaningful paired with an R4 PASS on the same tier (confirms the Graphify one-hop layer did -real work). Sonnet has that pairing (R4 PASS + R4-nograph FAIL); haiku doesn't, since R4 already -failed for haiku. +**Hypotheses / Scenarios Tested** -## Observations +Two scenario groups per the frozen shapes in `docs/adr-system/06-eval-scenarios.md`: +- **W1–W3 (write-triggers):** Should the model propose recording an architecture decision? W1: persistence choice (new decision to record). W2: convention change (decision to change established convention). W3: reversal (decision to reverse prior decision). +- **R1–R4 (retrieval):** Should the model consult existing decisions? R1: direct conflict (new approach contradicts an Accepted ADR). R2: distractor (scenario mentioning decisions but not directly conflicting). R3: mid-task question (ask at decision point, not upfront). R4: one-hop graph reach (correct ADR findable via Graphify graph traversal from affected files). -- **This reads as a prompting/wording issue, not a capability gap.** Sonnet already succeeds on - 5 of 8 scenarios, and its two failures (W3, R1) are near-misses rather than blind spots — - the model is clearly capable of the behavior when the trigger is salient enough. Haiku's - total 0/8 plus sonnet's two-scenario gap both point toward the SKILL.md wording and/or the - SessionStart hook note text (`plugins/os-adr/hooks/session_start.py`) not being salient - enough as an unprompted trigger — not toward a reasoning or tool-use failure. -- **Open question, not yet resolved:** whether in-session subagents could ever validly - substitute for part of this measurement. Flagged as suspect given the hook-context - requirement established for Eval B (in-session subagents inherit the parent session and never - get a fresh SessionStart against the sandbox cwd) — worth re-examining specifically for - scenarios where the hook note itself isn't the variable under test, but not assumed solved - here. -- **Next step (not started — future session):** design a follow-up eval that iterates - SKILL.md / hook-note wording to raise the unprompted-trigger rate, using the existing - `/autoresearch` Classic-loop discipline (checker/fixtures/scenarios/rubric frozen, only - wording moves — see `plugins/os-adr/eval-b/README.md`). **That eval should point back at - this note** for the baseline numbers and the prompting-issue hypothesis, so the loop has a - concrete before/after to compare against. +Correctness axis (b) for R1–R4: cite the correct existing ADR. For W1–W3: actually propose recording a new ADR (narrow rubric via `judge-rubric.md`, with haiku judge fallback). -## Confirmation run (2026-07-04) +Consultation axis (a, all scenarios): did the model touch any `os-adr:*` / `bin/adr-*` / `docs/adr/` unprompted? Mechanical detection from transcript tool_use blocks. -Full grid re-run after fixing stale plugin caches (os-adr's installed cache had drifted — missing -SessionStart hook + 3 CLIs) and switching skill registration to namespaced commands -(`/os-adr:find`, not bare `/find`). **Baseline confirmed: haiku 0/8, sonnet 5/8 with the same two -behavioral failures (W3, R1).** All 16 cells reproduced (one cell, W2/sonnet, needed a clean -re-run after a harness error — no transcript captured on the first attempt — and PASSed on -re-run, matching baseline). +**Fixture and Sampling** -New information beyond confirmation: +Single Ruby fixture: dedicated webhook-relay with 6-ADR history (Superseded pair + near-miss distractors, generated via the plugin's own CLIs). R4's one-hop graph reach uses a real `graphify update` AST build (model-free, rebuilt via `eval-b/bin/build-fixture-graph`, never committed; Graphify-less R4-nograph variant tests degradation). One rep per cell (1 rep/cell) for baseline discovery. -- **Sonnet W3 is an axis-b failure**: A:PASS / B:FAIL — it consults the ADR system unprompted, - then doesn't propose recording the reversal. The wording target is the create-skill's - "when to record" guidance, NOT trigger salience. R1 remains a pure axis-a (trigger) failure. -- **R4-nograph/sonnet PASSed** (expected FAIL) — sonnet found the correct ADR without the graph - layer this rep. At 1 rep this weakens (doesn't refute) the graph-layer-value evidence. -- **Haiku's first axis-a pass** (W2): one flicker of unprompted consultation — keep as a - lower-tier canary cell in any wording loop. -- **Variance is real at 1 rep/cell** (haiku W2 axis-a and sonnet W1 flipped between attempts) — - wording loops need ~3 reps on target cells. +**Experimental Control (Frozen Surfaces)** -Procedure and efficiency/quality lessons for the follow-up loop: -[[running-autoresearch-skill-evals]]. +Frozen during this eval: checker (`eval-b/bin/check`, two-axis deterministic-first), fixture (6-ADR Ruby webhook-relay), scenarios (W1–W3, R1–R4, R4-nograph), judge rubric (`judge-rubric.md` for axes-b evaluation of W1–W3). Varying: model tiers (haiku/sonnet) to establish tier-specific performance. Run mode: headless-only (`claude -p` per rep with cwd = sandbox, so real SessionStart hook fires — in-session Agent-tool subagents are invalid here). Rep count 1 chosen for baseline discovery, not for confidence (variance is expected at 1 rep/cell). -**2026-07-04 — follow-up experiment complete:** the wording loop closed the gap — final -grid **sonnet 8/8, haiku 7/8** (from 5/8 / 0/8). The prompting-issue hypothesis above is -confirmed. Hypotheses, per-iteration results, winning wording, and open questions -(channel ablation never run; R4-nograph no longer differentiates): -[[os-adr-eval-b-wording-experiment-hypotheses]]. These baseline numbers are superseded. +## Validity and Limitations + +**How to Interpret These Results** + +This is a **held-out baseline** (not yet optimized against). The 5/8 / 0/8 numbers mean "wording iteration is possible and worthwhile" — they support the prompting-issue hypothesis, not final deployment readiness. At 1 rep per cell, expect variance: haiku W2 flickered to axis-a PASS on re-run, demonstrating that single-rep cells are noisy. Do not cite sonnet's 5/8 as "80% accuracy" — the two failures (W3, R1) are near-misses (sonnet consults, then misses one output), not blind spots, and axis-specific targeting (H3 for W3, H1 for R1 — see the hypotheses note) suggests wording fixes are available. + +**Weaknesses of This Eval (Its Ladder Level)** + +Single fixture (Ruby webhook-relay) in one language — generalization to other codebases unknown. 1 rep/cell: high noise floor, flickers observed (haiku W2 axis-a, sonnet W1 across runs). No ambiguity axis: all scenarios are clear-cue boundary cases (not a progression from clear to ambiguous to over-trigger territory). Single distractor scenario (R2) — cannot assess false-positive rate under heavy decision-clutter. Axis-b evaluation of W1–W3 relies on a frozen rubric (`judge-rubric.md`); axis-a is mechanical (tool_use), so it's the reliable axis. Open question: whether in-session subagents could ever validly substitute for part of this measurement (suspected invalid due to hook-context requirement, but not definitively tested). This baseline is now obsolete — the 2026-07-04 wording loop closed the gap; see [[os-adr-eval-b-wording-experiment-hypotheses]] for the follow-up grid and the upgraded numbers. + +## Confirmation Run (2026-07-04) + +Full grid re-run (2026-07-04) after fixing stale plugin caches (os-adr's installed cache had drifted — missing SessionStart hook + 3 CLIs) and switching skill registration to namespaced commands (`/os-adr:find`, not bare `/find`). **Baseline confirmed: haiku 0/8, sonnet 5/8 with the same two behavioral failures (W3, R1).** All 16 cells reproduced (W2/sonnet needed clean re-run after harness error and PASSed on re-run, matching baseline). + +Confirmation run added: **Sonnet W3 is an axis-b failure** (A:PASS / B:FAIL) — consults the ADR system, then doesn't propose recording the reversal; target is create-skill's "when to record" guidance, not trigger salience. **R4-nograph/sonnet PASSed** (expected FAIL) — found the correct ADR without the graph layer this rep; at 1 rep this weakens (doesn't refute) graph-layer-value evidence. **Haiku W2** flickered to axis-a PASS — keep as lower-tier canary in follow-up loops. **Variance is real at 1 rep/cell** (haiku W2 axis-a and sonnet W1 flipped between attempts). + +## Deployment and Evolution + +**Good-Enough Gate** + +This baseline does NOT clear deployment. It is a held-out measurement establishing the prompting/wording issue hypothesis. The 5/8 / 0/8 indicates that wording optimization is possible (near-misses, not blind spots), and the follow-up wording loop (2026-07-04) confirms this — final grid improved to sonnet 8/8, haiku 7/8. Do not deploy based on this 5/8 / 0/8 baseline; consult the follow-up loop results in [[os-adr-eval-b-wording-experiment-hypotheses]] for the trained-up numbers and the gate for real-project rollout. + +**Hardening Path / Next Measurement** + +The follow-up wording experiment (2026-07-04) is complete — see [[os-adr-eval-b-wording-experiment-hypotheses]] for results and open questions (channel ablation never run; R4-nograph no longer differentiates). The next hardening step is Eval C, an ambiguity-ladder discrimination eval testing whether the model mistakes clear-cue for ambiguous-cue; see [[eval-methodology-ladder]] for the progression and [[eval-methodology-irl-feedback-loop]] for production validation. ## Source - cc-os repo, `plugins/os-adr/eval-b/` (harness) and `plugins/os-adr/eval-b/README.md` (status) - Full TSV: `/tmp/adr-eval-b-grid/results.tsv` (ephemeral — not committed, per harness design) - Doc updates committed at cc-os `5b399d5` + +## Related + +- [[os-adr-eval-b-wording-experiment-hypotheses]] — follow-up wording loop (2026-07-04) with upgraded numbers and deployment gate +- [[eval-methodology-ladder]] — evaluation ladder approach and progression logic +- [[running-autoresearch-skill-evals]] — procedure for skill-wording eval loops +- [[eval-methodology-irl-feedback-loop]] — production validation and audit backlog diff --git a/_templates/eval-results.md b/_templates/eval-results.md new file mode 100644 index 0000000..4d45b89 --- /dev/null +++ b/_templates/eval-results.md @@ -0,0 +1,63 @@ +--- +type: eval-results +title: [Evaluation name, e.g. "os-adr Eval B (unprompted write-trigger & retrieval)"] +summary: [What was measured and the key finding — 1-2 sentences, e.g. "Haiku never self-triggers the ADR system without explicit prompting; sonnet passes 5/8 with two near-miss failures."] +tags: + - type/eval-results + - domain/llm-evaluation + - tool/[tool-under-test] # the skill/plugin/feature being measured + - project/[project] # project that contains the eval harness +scope: global +last_updated: YYYY-MM-DD +date: YYYY-MM-DD # creation date — set once, never updated +related: + - [note-slug] # cross-links to follow-up evals, methodology notes, or design docs +source: [project name] # project that contains the eval harness +--- + +# [Evaluation Title] + +## Results Grid/Threshold + + + +**Passing criterion:** [Define what PASS means for this eval — e.g., all scenarios across both tiers, or per-tier threshold, or axis-level aggregate.] + +| [Column] | [Column] | +|---|---| +| [Cell] | [Cell] | + +**Summary:** [1–2 sentences recapping which tiers/cells passed/failed and the baseline numbers. This is what the reader scans first to know whether results are in the expected ballpark.] + +## Measurement Setup + +**Hypotheses / Scenarios Tested** + + +**Fixture and Sampling** + + +**Experimental Control (Frozen Surfaces)** + + +## Validity and Limitations + +**How to Interpret These Results** + + +**Weaknesses of This Eval (Its Ladder Level)** + + +## Deployment and Evolution + +**Good-Enough Gate** + + +**Hardening Path / Next Measurement** + + +## Related + + + +- [[note-slug]] — why it is relevant diff --git a/os-adr-eval-b-grid-results-and-observations.md b/os-adr-eval-b-grid-results-and-observations.md new file mode 100644 index 0000000..e69de29