vault: session notes 2026-07-06
This commit is contained in:
parent
f1fecc4640
commit
ce4f572137
|
|
@ -1,27 +1,30 @@
|
||||||
---
|
---
|
||||||
summary: os-adr Eval B (unprompted write-trigger & retrieval) grid results for haiku/sonnet — haiku never self-triggers the ADR system, sonnet passes 5/8 with two near-miss failures (confirmed by 2026-07-04 re-run) — read before designing any follow-up eval on unprompted-consultation behavior.
|
type: eval-results
|
||||||
last_updated: 2026-07-04
|
title: os-adr Eval B (unprompted write-trigger & retrieval) grid results (2026-07-03)
|
||||||
|
summary: Haiku never self-triggers the ADR system unprompted; sonnet passes 5/8 with two near-miss failures confirmed by 2026-07-04 re-run — baseline for follow-up wording optimization.
|
||||||
tags:
|
tags:
|
||||||
- scope/global
|
- type/eval-results
|
||||||
- type/reference
|
|
||||||
- project/cc-os
|
|
||||||
- tool/os-adr
|
|
||||||
- domain/llm-evaluation
|
- domain/llm-evaluation
|
||||||
source: cc-os
|
- tool/os-adr
|
||||||
|
- project/cc-os
|
||||||
|
scope: global
|
||||||
|
last_updated: 2026-07-04
|
||||||
date: 2026-07-03
|
date: 2026-07-03
|
||||||
|
related:
|
||||||
|
- os-adr-eval-b-wording-experiment-hypotheses
|
||||||
|
- running-autoresearch-skill-evals
|
||||||
|
source: cc-os
|
||||||
---
|
---
|
||||||
|
|
||||||
# os-adr Eval B grid results (2026-07-03)
|
# os-adr Eval B grid results (2026-07-03 baseline)
|
||||||
|
|
||||||
Eval B measures whether a model notices **on its own** that an Architecture Decision Record
|
## Results Grid/Threshold
|
||||||
system is relevant — without being explicitly told to invoke it. Two axes per scenario: (a)
|
|
||||||
did the model touch the ADR system unprompted (mechanical, from tool_use), (b) was the specific
|
|
||||||
outcome correct (right ADR cited, or a new ADR actually proposed/created). See
|
|
||||||
`cc-os/plugins/os-adr/eval-b/README.md` for full harness design (headless-only `claude -p` runs,
|
|
||||||
so a real SessionStart hook fires — in-session Agent-tool subagents cannot validly stand in for
|
|
||||||
this, since they don't get a fresh hook invocation against the sandbox cwd).
|
|
||||||
|
|
||||||
## Grid (1 rep/cell)
|
Eval B measures whether a model notices **on its own** that an Architecture Decision Record system is relevant — without being explicitly told to invoke it.
|
||||||
|
|
||||||
|
**Passing criterion:** Model must (a) touch the ADR system unprompted (mechanical detection from tool_use), and (b) produce correct outcome (right ADR cited for retrieval, or new ADR actually proposed for writes). For the 2026-07-03 baseline, passing is per-tier all-scenarios aggregate before any wording optimization.
|
||||||
|
|
||||||
|
**Grid (1 rep/cell):**
|
||||||
|
|
||||||
| Scenario | Haiku | Sonnet |
|
| Scenario | Haiku | Sonnet |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
|
|
@ -34,70 +37,63 @@ this, since they don't get a fresh hook invocation against the sandbox cwd).
|
||||||
| R4 (graph one-hop) | FAIL | **PASS** |
|
| R4 (graph one-hop) | FAIL | **PASS** |
|
||||||
| R4-nograph (degradation check) | FAIL (expected) | FAIL (expected) |
|
| R4-nograph (degradation check) | FAIL (expected) | FAIL (expected) |
|
||||||
|
|
||||||
**Haiku: 0/8.** Never touches `os-adr:*` / `bin/adr-*` / `docs/adr/` unprompted in any
|
**Summary:** Haiku 0/8 (never unprompted-consults the ADR system in any scenario). Sonnet 5/8 (failures: W3 reversal / R1 direct-conflict retrieval). R4-nograph degradation checks failed as expected — only meaningful paired with R4 PASS on the same tier.
|
||||||
scenario — total absence of self-triggered consultation, not a near-miss.
|
|
||||||
|
|
||||||
**Sonnet: 5/8.** Two real misses: W3 (reversal — doesn't propose recording the decision) and R1
|
## Measurement Setup
|
||||||
(direct-conflict retrieval — doesn't catch it and cite the correct ADR).
|
|
||||||
|
|
||||||
`R4-nograph` failing on both tiers is the **expected** result — it's a degradation check, only
|
**Hypotheses / Scenarios Tested**
|
||||||
meaningful paired with an R4 PASS on the same tier (confirms the Graphify one-hop layer did
|
|
||||||
real work). Sonnet has that pairing (R4 PASS + R4-nograph FAIL); haiku doesn't, since R4 already
|
|
||||||
failed for haiku.
|
|
||||||
|
|
||||||
## Observations
|
Two scenario groups per the frozen shapes in `docs/adr-system/06-eval-scenarios.md`:
|
||||||
|
- **W1–W3 (write-triggers):** Should the model propose recording an architecture decision? W1: persistence choice (new decision to record). W2: convention change (decision to change established convention). W3: reversal (decision to reverse prior decision).
|
||||||
|
- **R1–R4 (retrieval):** Should the model consult existing decisions? R1: direct conflict (new approach contradicts an Accepted ADR). R2: distractor (scenario mentioning decisions but not directly conflicting). R3: mid-task question (ask at decision point, not upfront). R4: one-hop graph reach (correct ADR findable via Graphify graph traversal from affected files).
|
||||||
|
|
||||||
- **This reads as a prompting/wording issue, not a capability gap.** Sonnet already succeeds on
|
Correctness axis (b) for R1–R4: cite the correct existing ADR. For W1–W3: actually propose recording a new ADR (narrow rubric via `judge-rubric.md`, with haiku judge fallback).
|
||||||
5 of 8 scenarios, and its two failures (W3, R1) are near-misses rather than blind spots —
|
|
||||||
the model is clearly capable of the behavior when the trigger is salient enough. Haiku's
|
|
||||||
total 0/8 plus sonnet's two-scenario gap both point toward the SKILL.md wording and/or the
|
|
||||||
SessionStart hook note text (`plugins/os-adr/hooks/session_start.py`) not being salient
|
|
||||||
enough as an unprompted trigger — not toward a reasoning or tool-use failure.
|
|
||||||
- **Open question, not yet resolved:** whether in-session subagents could ever validly
|
|
||||||
substitute for part of this measurement. Flagged as suspect given the hook-context
|
|
||||||
requirement established for Eval B (in-session subagents inherit the parent session and never
|
|
||||||
get a fresh SessionStart against the sandbox cwd) — worth re-examining specifically for
|
|
||||||
scenarios where the hook note itself isn't the variable under test, but not assumed solved
|
|
||||||
here.
|
|
||||||
- **Next step (not started — future session):** design a follow-up eval that iterates
|
|
||||||
SKILL.md / hook-note wording to raise the unprompted-trigger rate, using the existing
|
|
||||||
`/autoresearch` Classic-loop discipline (checker/fixtures/scenarios/rubric frozen, only
|
|
||||||
wording moves — see `plugins/os-adr/eval-b/README.md`). **That eval should point back at
|
|
||||||
this note** for the baseline numbers and the prompting-issue hypothesis, so the loop has a
|
|
||||||
concrete before/after to compare against.
|
|
||||||
|
|
||||||
## Confirmation run (2026-07-04)
|
Consultation axis (a, all scenarios): did the model touch any `os-adr:*` / `bin/adr-*` / `docs/adr/` unprompted? Mechanical detection from transcript tool_use blocks.
|
||||||
|
|
||||||
Full grid re-run after fixing stale plugin caches (os-adr's installed cache had drifted — missing
|
**Fixture and Sampling**
|
||||||
SessionStart hook + 3 CLIs) and switching skill registration to namespaced commands
|
|
||||||
(`/os-adr:find`, not bare `/find`). **Baseline confirmed: haiku 0/8, sonnet 5/8 with the same two
|
|
||||||
behavioral failures (W3, R1).** All 16 cells reproduced (one cell, W2/sonnet, needed a clean
|
|
||||||
re-run after a harness error — no transcript captured on the first attempt — and PASSed on
|
|
||||||
re-run, matching baseline).
|
|
||||||
|
|
||||||
New information beyond confirmation:
|
Single Ruby fixture: dedicated webhook-relay with 6-ADR history (Superseded pair + near-miss distractors, generated via the plugin's own CLIs). R4's one-hop graph reach uses a real `graphify update` AST build (model-free, rebuilt via `eval-b/bin/build-fixture-graph`, never committed; Graphify-less R4-nograph variant tests degradation). One rep per cell (1 rep/cell) for baseline discovery.
|
||||||
|
|
||||||
- **Sonnet W3 is an axis-b failure**: A:PASS / B:FAIL — it consults the ADR system unprompted,
|
**Experimental Control (Frozen Surfaces)**
|
||||||
then doesn't propose recording the reversal. The wording target is the create-skill's
|
|
||||||
"when to record" guidance, NOT trigger salience. R1 remains a pure axis-a (trigger) failure.
|
|
||||||
- **R4-nograph/sonnet PASSed** (expected FAIL) — sonnet found the correct ADR without the graph
|
|
||||||
layer this rep. At 1 rep this weakens (doesn't refute) the graph-layer-value evidence.
|
|
||||||
- **Haiku's first axis-a pass** (W2): one flicker of unprompted consultation — keep as a
|
|
||||||
lower-tier canary cell in any wording loop.
|
|
||||||
- **Variance is real at 1 rep/cell** (haiku W2 axis-a and sonnet W1 flipped between attempts) —
|
|
||||||
wording loops need ~3 reps on target cells.
|
|
||||||
|
|
||||||
Procedure and efficiency/quality lessons for the follow-up loop:
|
Frozen during this eval: checker (`eval-b/bin/check`, two-axis deterministic-first), fixture (6-ADR Ruby webhook-relay), scenarios (W1–W3, R1–R4, R4-nograph), judge rubric (`judge-rubric.md` for axes-b evaluation of W1–W3). Varying: model tiers (haiku/sonnet) to establish tier-specific performance. Run mode: headless-only (`claude -p` per rep with cwd = sandbox, so real SessionStart hook fires — in-session Agent-tool subagents are invalid here). Rep count 1 chosen for baseline discovery, not for confidence (variance is expected at 1 rep/cell).
|
||||||
[[running-autoresearch-skill-evals]].
|
|
||||||
|
|
||||||
**2026-07-04 — follow-up experiment complete:** the wording loop closed the gap — final
|
## Validity and Limitations
|
||||||
grid **sonnet 8/8, haiku 7/8** (from 5/8 / 0/8). The prompting-issue hypothesis above is
|
|
||||||
confirmed. Hypotheses, per-iteration results, winning wording, and open questions
|
**How to Interpret These Results**
|
||||||
(channel ablation never run; R4-nograph no longer differentiates):
|
|
||||||
[[os-adr-eval-b-wording-experiment-hypotheses]]. These baseline numbers are superseded.
|
This is a **held-out baseline** (not yet optimized against). The 5/8 / 0/8 numbers mean "wording iteration is possible and worthwhile" — they support the prompting-issue hypothesis, not final deployment readiness. At 1 rep per cell, expect variance: haiku W2 flickered to axis-a PASS on re-run, demonstrating that single-rep cells are noisy. Do not cite sonnet's 5/8 as "80% accuracy" — the two failures (W3, R1) are near-misses (sonnet consults, then misses one output), not blind spots, and axis-specific targeting (H3 for W3, H1 for R1 — see the hypotheses note) suggests wording fixes are available.
|
||||||
|
|
||||||
|
**Weaknesses of This Eval (Its Ladder Level)**
|
||||||
|
|
||||||
|
Single fixture (Ruby webhook-relay) in one language — generalization to other codebases unknown. 1 rep/cell: high noise floor, flickers observed (haiku W2 axis-a, sonnet W1 across runs). No ambiguity axis: all scenarios are clear-cue boundary cases (not a progression from clear to ambiguous to over-trigger territory). Single distractor scenario (R2) — cannot assess false-positive rate under heavy decision-clutter. Axis-b evaluation of W1–W3 relies on a frozen rubric (`judge-rubric.md`); axis-a is mechanical (tool_use), so it's the reliable axis. Open question: whether in-session subagents could ever validly substitute for part of this measurement (suspected invalid due to hook-context requirement, but not definitively tested). This baseline is now obsolete — the 2026-07-04 wording loop closed the gap; see [[os-adr-eval-b-wording-experiment-hypotheses]] for the follow-up grid and the upgraded numbers.
|
||||||
|
|
||||||
|
## Confirmation Run (2026-07-04)
|
||||||
|
|
||||||
|
Full grid re-run (2026-07-04) after fixing stale plugin caches (os-adr's installed cache had drifted — missing SessionStart hook + 3 CLIs) and switching skill registration to namespaced commands (`/os-adr:find`, not bare `/find`). **Baseline confirmed: haiku 0/8, sonnet 5/8 with the same two behavioral failures (W3, R1).** All 16 cells reproduced (W2/sonnet needed clean re-run after harness error and PASSed on re-run, matching baseline).
|
||||||
|
|
||||||
|
Confirmation run added: **Sonnet W3 is an axis-b failure** (A:PASS / B:FAIL) — consults the ADR system, then doesn't propose recording the reversal; target is create-skill's "when to record" guidance, not trigger salience. **R4-nograph/sonnet PASSed** (expected FAIL) — found the correct ADR without the graph layer this rep; at 1 rep this weakens (doesn't refute) graph-layer-value evidence. **Haiku W2** flickered to axis-a PASS — keep as lower-tier canary in follow-up loops. **Variance is real at 1 rep/cell** (haiku W2 axis-a and sonnet W1 flipped between attempts).
|
||||||
|
|
||||||
|
## Deployment and Evolution
|
||||||
|
|
||||||
|
**Good-Enough Gate**
|
||||||
|
|
||||||
|
This baseline does NOT clear deployment. It is a held-out measurement establishing the prompting/wording issue hypothesis. The 5/8 / 0/8 indicates that wording optimization is possible (near-misses, not blind spots), and the follow-up wording loop (2026-07-04) confirms this — final grid improved to sonnet 8/8, haiku 7/8. Do not deploy based on this 5/8 / 0/8 baseline; consult the follow-up loop results in [[os-adr-eval-b-wording-experiment-hypotheses]] for the trained-up numbers and the gate for real-project rollout.
|
||||||
|
|
||||||
|
**Hardening Path / Next Measurement**
|
||||||
|
|
||||||
|
The follow-up wording experiment (2026-07-04) is complete — see [[os-adr-eval-b-wording-experiment-hypotheses]] for results and open questions (channel ablation never run; R4-nograph no longer differentiates). The next hardening step is Eval C, an ambiguity-ladder discrimination eval testing whether the model mistakes clear-cue for ambiguous-cue; see [[eval-methodology-ladder]] for the progression and [[eval-methodology-irl-feedback-loop]] for production validation.
|
||||||
|
|
||||||
## Source
|
## Source
|
||||||
|
|
||||||
- cc-os repo, `plugins/os-adr/eval-b/` (harness) and `plugins/os-adr/eval-b/README.md` (status)
|
- cc-os repo, `plugins/os-adr/eval-b/` (harness) and `plugins/os-adr/eval-b/README.md` (status)
|
||||||
- Full TSV: `/tmp/adr-eval-b-grid/results.tsv` (ephemeral — not committed, per harness design)
|
- Full TSV: `/tmp/adr-eval-b-grid/results.tsv` (ephemeral — not committed, per harness design)
|
||||||
- Doc updates committed at cc-os `5b399d5`
|
- Doc updates committed at cc-os `5b399d5`
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [[os-adr-eval-b-wording-experiment-hypotheses]] — follow-up wording loop (2026-07-04) with upgraded numbers and deployment gate
|
||||||
|
- [[eval-methodology-ladder]] — evaluation ladder approach and progression logic
|
||||||
|
- [[running-autoresearch-skill-evals]] — procedure for skill-wording eval loops
|
||||||
|
- [[eval-methodology-irl-feedback-loop]] — production validation and audit backlog
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,63 @@
|
||||||
|
---
|
||||||
|
type: eval-results
|
||||||
|
title: [Evaluation name, e.g. "os-adr Eval B (unprompted write-trigger & retrieval)"]
|
||||||
|
summary: [What was measured and the key finding — 1-2 sentences, e.g. "Haiku never self-triggers the ADR system without explicit prompting; sonnet passes 5/8 with two near-miss failures."]
|
||||||
|
tags:
|
||||||
|
- type/eval-results
|
||||||
|
- domain/llm-evaluation
|
||||||
|
- tool/[tool-under-test] # the skill/plugin/feature being measured
|
||||||
|
- project/[project] # project that contains the eval harness
|
||||||
|
scope: global
|
||||||
|
last_updated: YYYY-MM-DD
|
||||||
|
date: YYYY-MM-DD # creation date — set once, never updated
|
||||||
|
related:
|
||||||
|
- [note-slug] # cross-links to follow-up evals, methodology notes, or design docs
|
||||||
|
source: [project name] # project that contains the eval harness
|
||||||
|
---
|
||||||
|
|
||||||
|
# [Evaluation Title]
|
||||||
|
|
||||||
|
## Results Grid/Threshold
|
||||||
|
|
||||||
|
<!-- The raw measurement: grid table with pass/fail per cell, OR hypothesis→verdict mapping, or threshold scores. Include rep counts per cell and evaluation date. Include a brief summary line above the table/mapping explaining what passing looks like. -->
|
||||||
|
|
||||||
|
**Passing criterion:** [Define what PASS means for this eval — e.g., all scenarios across both tiers, or per-tier threshold, or axis-level aggregate.]
|
||||||
|
|
||||||
|
| [Column] | [Column] |
|
||||||
|
|---|---|
|
||||||
|
| [Cell] | [Cell] |
|
||||||
|
|
||||||
|
**Summary:** [1–2 sentences recapping which tiers/cells passed/failed and the baseline numbers. This is what the reader scans first to know whether results are in the expected ballpark.]
|
||||||
|
|
||||||
|
## Measurement Setup
|
||||||
|
|
||||||
|
**Hypotheses / Scenarios Tested**
|
||||||
|
<!-- If hypothesis-driven: name each hypothesis and its test scenarios. If scenario-driven: name each scenario group (e.g., W1–W3 for write-triggers, R1–R4 for retrieval). One paragraph or brief list per hypothesis/group, clarifying what correctness looks like. -->
|
||||||
|
|
||||||
|
**Fixture and Sampling**
|
||||||
|
<!-- Which fixture(s) were used (real project vs synthetic, language/domain, size, ADR/decision history if relevant). Fixture generalization risk. Reps per cell. Rationale for rep count (e.g., 1 rep/cell for baseline discovery, 3 reps for wording-loop stability). -->
|
||||||
|
|
||||||
|
**Experimental Control (Frozen Surfaces)**
|
||||||
|
<!-- What was held constant during this eval and why. For skill-wording loops: checker, fixtures, scenarios, rubric frozen — only wording moves. For harness-design evals: [list what was frozen]. Explicitly name anything intentionally NOT frozen, and why (e.g., model tiers vary to establish tier-specific performance; rep count chosen for discovery vs confirmation). -->
|
||||||
|
|
||||||
|
## Validity and Limitations
|
||||||
|
|
||||||
|
**How to Interpret These Results**
|
||||||
|
<!-- Training-set vs held-out framing — if this eval was optimized against (wording loop, rubric tuning), state that explicitly. What the numbers do and do not support (e.g., "8/8 on a training-set grid means wording direction is sound, not that the behavior generalizes"). Confidence caveats: variance at low reps, fixture-specific behavior, scenarios-only measurement (not live observability). When to read a result as significant (e.g., majority of reps, control cells hold). When to read a result as noise (e.g., single-rep flips between cells, below the rep threshold for stability). -->
|
||||||
|
|
||||||
|
**Weaknesses of This Eval (Its Ladder Level)**
|
||||||
|
<!-- What this eval can't see or didn't test. Examples: single fixture generalization (would a second fixture in a different language / domain change the results?), 1 rep/cell variance (high noise floor), no ambiguity axis (scenarios are clear-cue vs no clear boundary-case testing), limited distractor count, no longer held-out (wording was tuned against this grid — it's now training-set), model-specific failure modes, ablation surfaces that were never tested (e.g., "channel ablation not run — don't assume hook redundancy"). Open questions the eval can't resolve. Open questions the eval raises. -->
|
||||||
|
|
||||||
|
## Deployment and Evolution
|
||||||
|
|
||||||
|
**Good-Enough Gate**
|
||||||
|
<!-- Explicit criterion under which these results justify deployment, adoption, or real-world rollout. Examples: "Sonnet 8/8 passes for pilot rollout on projects with Rust / Go codebases; haiku pending 3-rep confirmation on the W3 edge case (running today, 2026-07-06)." Or: "5/8 passes the 'prompting issue, not capability gap' threshold; spin up a wording loop before production rollout." Tier-specific status. Conditions that would change the call. -->
|
||||||
|
|
||||||
|
**Hardening Path / Next Measurement**
|
||||||
|
<!-- If there is a follow-up eval harness (e.g., Eval C at a higher ladder level), point to it. Otherwise, point to the ladder-approach methodology note explaining the progression. One paragraph or brief list of what gets tested next and why (e.g., "Eval C will add ambiguity-ladder discrimination scenarios to test whether the model mistakes clear-cue for ambiguous-cue; if it does, wording tuning stops and the feature is capability-limited"). -->
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
<!-- Wikilinks to: prior baselines or follow-up evals, the methodology note(s) (ladder approach, autoresearch procedure), the skills/design docs being evaluated, and hub notes if any. -->
|
||||||
|
|
||||||
|
- [[note-slug]] — why it is relevant
|
||||||
Loading…
Reference in New Issue