100 lines
8.3 KiB
Markdown
100 lines
8.3 KiB
Markdown
---
|
||
type: eval-results
|
||
title: os-adr Eval B (unprompted write-trigger & retrieval) grid results (2026-07-03)
|
||
summary: Haiku never self-triggers the ADR system unprompted; sonnet passes 5/8 with two near-miss failures confirmed by 2026-07-04 re-run — baseline for follow-up wording optimization.
|
||
tags:
|
||
- type/eval-results
|
||
- domain/llm-evaluation
|
||
- tool/os-adr
|
||
- project/cc-os
|
||
scope: global
|
||
last_updated: 2026-07-04
|
||
date: 2026-07-03
|
||
related:
|
||
- os-adr-eval-b-wording-experiment-hypotheses
|
||
- running-autoresearch-skill-evals
|
||
source: cc-os
|
||
---
|
||
|
||
# os-adr Eval B grid results (2026-07-03 baseline)
|
||
|
||
## Results Grid/Threshold
|
||
|
||
Eval B measures whether a model notices **on its own** that an Architecture Decision Record system is relevant — without being explicitly told to invoke it.
|
||
|
||
**Passing criterion:** Model must (a) touch the ADR system unprompted (mechanical detection from tool_use), and (b) produce correct outcome (right ADR cited for retrieval, or new ADR actually proposed for writes). For the 2026-07-03 baseline, passing is per-tier all-scenarios aggregate before any wording optimization.
|
||
|
||
**Grid (1 rep/cell):**
|
||
|
||
| Scenario | Haiku | Sonnet |
|
||
|---|---|---|
|
||
| W1 (persistence choice) | FAIL | **PASS** |
|
||
| W2 (convention change) | FAIL | **PASS** |
|
||
| W3 (reversal) | FAIL | FAIL — doesn't propose recording the decision |
|
||
| R1 (direct conflict) | FAIL | FAIL — misses the direct-conflict retrieval |
|
||
| R2 (distractor) | FAIL | **PASS** |
|
||
| R3 (mid-task question) | FAIL | **PASS** |
|
||
| R4 (graph one-hop) | FAIL | **PASS** |
|
||
| R4-nograph (degradation check) | FAIL (expected) | FAIL (expected) |
|
||
|
||
**Summary:** Haiku 0/8 (never unprompted-consults the ADR system in any scenario). Sonnet 5/8 (failures: W3 reversal / R1 direct-conflict retrieval). R4-nograph degradation checks failed as expected — only meaningful paired with R4 PASS on the same tier.
|
||
|
||
## Measurement Setup
|
||
|
||
**Hypotheses / Scenarios Tested**
|
||
|
||
Two scenario groups per the frozen shapes in `docs/adr-system/06-eval-scenarios.md`:
|
||
- **W1–W3 (write-triggers):** Should the model propose recording an architecture decision? W1: persistence choice (new decision to record). W2: convention change (decision to change established convention). W3: reversal (decision to reverse prior decision).
|
||
- **R1–R4 (retrieval):** Should the model consult existing decisions? R1: direct conflict (new approach contradicts an Accepted ADR). R2: distractor (scenario mentioning decisions but not directly conflicting). R3: mid-task question (ask at decision point, not upfront). R4: one-hop graph reach (correct ADR findable via Graphify graph traversal from affected files).
|
||
|
||
Correctness axis (b) for R1–R4: cite the correct existing ADR. For W1–W3: actually propose recording a new ADR (narrow rubric via `judge-rubric.md`, with haiku judge fallback).
|
||
|
||
Consultation axis (a, all scenarios): did the model touch any `os-adr:*` / `bin/adr-*` / `docs/adr/` unprompted? Mechanical detection from transcript tool_use blocks.
|
||
|
||
**Fixture and Sampling**
|
||
|
||
Single Ruby fixture: dedicated webhook-relay with 6-ADR history (Superseded pair + near-miss distractors, generated via the plugin's own CLIs). R4's one-hop graph reach uses a real `graphify update` AST build (model-free, rebuilt via `eval-b/bin/build-fixture-graph`, never committed; Graphify-less R4-nograph variant tests degradation). One rep per cell (1 rep/cell) for baseline discovery.
|
||
|
||
**Experimental Control (Frozen Surfaces)**
|
||
|
||
Frozen during this eval: checker (`eval-b/bin/check`, two-axis deterministic-first), fixture (6-ADR Ruby webhook-relay), scenarios (W1–W3, R1–R4, R4-nograph), judge rubric (`judge-rubric.md` for axes-b evaluation of W1–W3). Varying: model tiers (haiku/sonnet) to establish tier-specific performance. Run mode: headless-only (`claude -p` per rep with cwd = sandbox, so real SessionStart hook fires — in-session Agent-tool subagents are invalid here). Rep count 1 chosen for baseline discovery, not for confidence (variance is expected at 1 rep/cell).
|
||
|
||
## Validity and Limitations
|
||
|
||
**How to Interpret These Results**
|
||
|
||
This is a **held-out baseline** (not yet optimized against). The 5/8 / 0/8 numbers mean "wording iteration is possible and worthwhile" — they support the prompting-issue hypothesis, not final deployment readiness. At 1 rep per cell, expect variance: haiku W2 flickered to axis-a PASS on re-run, demonstrating that single-rep cells are noisy. Do not cite sonnet's 5/8 as "80% accuracy" — the two failures (W3, R1) are near-misses (sonnet consults, then misses one output), not blind spots, and axis-specific targeting (H3 for W3, H1 for R1 — see the hypotheses note) suggests wording fixes are available.
|
||
|
||
**Weaknesses of This Eval (Its Ladder Level)**
|
||
|
||
Single fixture (Ruby webhook-relay) in one language — generalization to other codebases unknown. 1 rep/cell: high noise floor, flickers observed (haiku W2 axis-a, sonnet W1 across runs). No ambiguity axis: all scenarios are clear-cue boundary cases (not a progression from clear to ambiguous to over-trigger territory). Single distractor scenario (R2) — cannot assess false-positive rate under heavy decision-clutter. Axis-b evaluation of W1–W3 relies on a frozen rubric (`judge-rubric.md`); axis-a is mechanical (tool_use), so it's the reliable axis. Open question: whether in-session subagents could ever validly substitute for part of this measurement (suspected invalid due to hook-context requirement, but not definitively tested). This baseline is now obsolete — the 2026-07-04 wording loop closed the gap; see [[os-adr-eval-b-wording-experiment-hypotheses]] for the follow-up grid and the upgraded numbers.
|
||
|
||
## Confirmation Run (2026-07-04)
|
||
|
||
Full grid re-run (2026-07-04) after fixing stale plugin caches (os-adr's installed cache had drifted — missing SessionStart hook + 3 CLIs) and switching skill registration to namespaced commands (`/os-adr:find`, not bare `/find`). **Baseline confirmed: haiku 0/8, sonnet 5/8 with the same two behavioral failures (W3, R1).** All 16 cells reproduced (W2/sonnet needed clean re-run after harness error and PASSed on re-run, matching baseline).
|
||
|
||
Confirmation run added: **Sonnet W3 is an axis-b failure** (A:PASS / B:FAIL) — consults the ADR system, then doesn't propose recording the reversal; target is create-skill's "when to record" guidance, not trigger salience. **R4-nograph/sonnet PASSed** (expected FAIL) — found the correct ADR without the graph layer this rep; at 1 rep this weakens (doesn't refute) graph-layer-value evidence. **Haiku W2** flickered to axis-a PASS — keep as lower-tier canary in follow-up loops. **Variance is real at 1 rep/cell** (haiku W2 axis-a and sonnet W1 flipped between attempts).
|
||
|
||
## Deployment and Evolution
|
||
|
||
**Good-Enough Gate**
|
||
|
||
This baseline does NOT clear deployment. It is a held-out measurement establishing the prompting/wording issue hypothesis. The 5/8 / 0/8 indicates that wording optimization is possible (near-misses, not blind spots), and the follow-up wording loop (2026-07-04) confirms this — final grid improved to sonnet 8/8, haiku 7/8. Do not deploy based on this 5/8 / 0/8 baseline; consult the follow-up loop results in [[os-adr-eval-b-wording-experiment-hypotheses]] for the trained-up numbers and the gate for real-project rollout.
|
||
|
||
**Hardening Path / Next Measurement**
|
||
|
||
The follow-up wording experiment (2026-07-04) is complete — see [[os-adr-eval-b-wording-experiment-hypotheses]] for results and open questions (channel ablation never run; R4-nograph no longer differentiates). The next hardening step is Eval C, an ambiguity-ladder discrimination eval testing whether the model mistakes clear-cue for ambiguous-cue; see [[eval-methodology-ladder]] for the progression and [[eval-methodology-irl-feedback-loop]] for production validation.
|
||
|
||
## Source
|
||
|
||
- cc-os repo, `plugins/os-adr/eval-b/` (harness) and `plugins/os-adr/eval-b/README.md` (status)
|
||
- Full TSV: `/tmp/adr-eval-b-grid/results.tsv` (ephemeral — not committed, per harness design)
|
||
- Doc updates committed at cc-os `5b399d5`
|
||
|
||
## Related
|
||
|
||
- [[os-adr-eval-b-wording-experiment-hypotheses]] — follow-up wording loop (2026-07-04) with upgraded numbers and deployment gate
|
||
- [[eval-methodology-ladder]] — evaluation ladder approach and progression logic
|
||
- [[running-autoresearch-skill-evals]] — procedure for skill-wording eval loops
|
||
- [[eval-methodology-irl-feedback-loop]] — production validation and audit backlog
|