From e45d674d6bcd903b5192f20a5f1eb88e6ca03258 Mon Sep 17 00:00:00 2001 From: jared Date: Mon, 6 Jul 2026 11:30:29 -0400 Subject: [PATCH] vault: session notes 2026-07-06 --- ...dr-eval-b-grid-results-and-observations.md | 6 +++ journal/2026-07-06.md | 11 +++++ ...dr-eval-b-wording-experiment-hypotheses.md | 41 +++++++++++++++++-- 3 files changed, 55 insertions(+), 3 deletions(-) create mode 100644 journal/2026-07-06.md diff --git a/2026-07-03-os-adr-eval-b-grid-results-and-observations.md b/2026-07-03-os-adr-eval-b-grid-results-and-observations.md index 4f99e22..6b0e210 100644 --- a/2026-07-03-os-adr-eval-b-grid-results-and-observations.md +++ b/2026-07-03-os-adr-eval-b-grid-results-and-observations.md @@ -90,6 +90,12 @@ New information beyond confirmation: Procedure and efficiency/quality lessons for the follow-up loop: [[running-autoresearch-skill-evals]]. +**2026-07-04 — follow-up experiment complete:** the wording loop closed the gap — final +grid **sonnet 8/8, haiku 7/8** (from 5/8 / 0/8). The prompting-issue hypothesis above is +confirmed. Hypotheses, per-iteration results, winning wording, and open questions +(channel ablation never run; R4-nograph no longer differentiates): +[[os-adr-eval-b-wording-experiment-hypotheses]]. These baseline numbers are superseded. + ## Source - cc-os repo, `plugins/os-adr/eval-b/` (harness) and `plugins/os-adr/eval-b/README.md` (status) diff --git a/journal/2026-07-06.md b/journal/2026-07-06.md new file mode 100644 index 0000000..f1cb4dc --- /dev/null +++ b/journal/2026-07-06.md @@ -0,0 +1,11 @@ +--- +summary: Daily session log +tags: [scope/global, type/log] +--- + +## Session — 2026-07-06T15:30:29Z + +**Project:** /tmp/claude-1000/-home-jared-dev-cc-os/d68aab1b-3e3d-467c-9b9c-70b89e2576b0/scratchpad/adr-eval-b-w3-haiku/W3-haiku-r2 +**Reason:** other +**Vault notes touched:** +(none) diff --git a/os-adr-eval-b-wording-experiment-hypotheses.md b/os-adr-eval-b-wording-experiment-hypotheses.md index 4ba88ac..6b7a0e0 100644 --- a/os-adr-eval-b-wording-experiment-hypotheses.md +++ b/os-adr-eval-b-wording-experiment-hypotheses.md @@ -74,7 +74,42 @@ confirms here, start the *next* plugin's unprompted-behavior wording at trigger- phrasing instead of rediscovering it). Pair this note's pattern with the procedure in [[running-autoresearch-skill-evals]]. -## Status +## Results (2026-07-04 — experiment complete, 5 iterations) -- 2026-07-04: hypotheses locked, experiment starting (baseline commit cc-os `c645faa`). - Results to be appended here per iteration. +Final full grid: **sonnet 8/8** (was 5/8), **haiku 7/8** (was 0/8; the one miss is a W3 +axis-b judge-boundary flicker — it consulted, and passed the same cell in the mid-experiment confirmation). Per-iteration: + +1. **Iter 1 — trigger-conditioned rewrite, all channels** (hook note, find/create + descriptions + bodies, fixture CLAUDE.md): R1/sonnet FAIL→3/3 PASS; haiku W2 canary + 0→3/3 PASS; W3/sonnet 1/3 (axis-b persists); W1 control holds. **H1 CONFIRMED** — and it reaches haiku, so haiku's 0/8 was packaging, not a capability floor. +2. **Iter 2 — "task not complete until superseding ADR exists"**: W3 still 1/3. Transcript + analysis showed the model behaving *well* (catches the conflict, offers options with the + ADR attached to the reversal path) but phrasing the offer conditionally inside an option + list — judge flicker at the rubric boundary, not forgetting. +3. **Iter 3 — explicit-unconditional recording offer** ("state plainly: this reverses + ADR-NNNN; I'll record a superseding ADR — never a parenthetical in an option list"): + W3/sonnet 3/3 PASS. **H3 CONFIRMED** (placement in the find skill's act-on-findings step, + where find's output is on screen). +4. **Iter 4 — lower threshold "modifying existing behavior → find first"**: R4/haiku 0→3/3; + R1/haiku still 0/3. Transcript: haiku classified R1's task as a *pure addition* (new + bypass method), so any semantic trigger missed it. +5. **Iter 5 — fully mechanical trigger** ("before your first edit to any existing file → + run find on those paths; additions count — a new method can bypass a decided + constraint"): R1/haiku axis-a 3/3, full 2/3; R2 distractor control holds (no + over-triggering); sonnet unaffected. + +**Verdicts:** H1 confirmed (trigger-conditioned ≫ inventory phrasing, both tiers). H3 +confirmed (each rule lives where its precondition is visible; step-2 wording in the find +skill fixed W3). **H2 NOT tested** — channels were never ablated (all-on throughout); whether +the SessionStart hook is redundant next to a trigger-phrased CLAUDE.md remains open. Also +open: R4-nograph now PASSES on both tiers — the strengthened wording made the graph layer +unnecessary for that scenario, so the degradation check no longer differentiates. + +**Tier-generalization lesson:** each tier down needs a more *mechanical* trigger. Sonnet +acts on "architecture-level choice"; haiku needs "before editing any existing file, run +find — additions count." The mechanical rule subsumes the semantic one and costs one cheap +deterministic CLI call per task — lead with it in future packaging. + +**Deployment note:** the fixture CLAUDE.md section is the template for real-project +adoption (candidate: emit it from `/os-adr:init`/`migrate`); winning wording committed to +cc-os (hook `session_start.py`, find/create SKILL.mds, eval-b fixture CLAUDE.md).