jared
/
SecondBrain
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
SecondBrain/os-vault-write-eval-baselin...

6.3 KiB
Raw Blame History

type title summary tags scope last_updated date related source
eval-results os-vault write eval — untuned baseline grid (unprompted vault-write discrimination) At baseline (no trigger wording anywhere in os-vault), positives are 1/19 across both tiers while negatives are 18/18 clean — and the dominant failure shape is not "no persistence" but mis-routing to Claude Code's built-in auto-memory instead of the vault at L1; L2/L3 mostly don't persist at all.
type/eval-results
domain/llm-evaluation
tool/os-vault
tool/claude-code
project/cc-os
global 2026-07-06 2026-07-06
eval-methodology-ladder
running-autoresearch-skill-evals
os-adr-eval-b-wording-experiment-hypotheses
os-adr-eval-c-frozen-grid-results
cc-os

os-vault write eval — untuned baseline grid (2026-07-06)

Results Grid/Threshold

Passing criterion: positives — unprompted vault write with contract-conforming frontmatter (axis a: written or offered; axis b: mechanical frontmatter conformance); negatives — no vault write (the sole fail line). Run-set × {sonnet, haiku} × 3 reps + 1 counted sonnet canary rep on P1 (37 counted reps total, zero harness errors, all rows verified against expected rep counts).

Cell sonnet haiku Dominant behavior
P1-L1 explicit cue (Finch 500-cap) 1/4 (canary only) 0/3 Persists EVERY rep — but to auto-memory (~/.claude/projects/<slug>/memory/), not the vault
P2-L2 no cue (Finch expires_in) 0/3 0/3 Fixes the code; no persistence anywhere
P3-L3 conceptual (Acme convention) 0/3 0/3 sonnet 1/3 wrote auto-memory; otherwise nothing
N1-L1 explicit cue, ephemeral 3/3 3/3 Status written to project file (project-write:yes), vault untouched
N2-L2 project-local quirk 3/3 3/3 Clean fix, no vault write
N3-L3 repo-recorded (ADR 0002) 3/3 3/3 Complies with the ADR, no vault duplicate

Summary: positives 1/19 (the one pass: sonnet canary P1, which wrote a fully conforming reference/finch-api-page-size-silent-cap.md — proof the ceiling exists); negatives 18/18 with zero over-trigger at any level on either tier. The measured gap is real and one-sided: under-triggering plus destination mis-routing, no restraint problem.

Measurement Setup

Hypotheses / Scenarios Tested — Ambiguity ladder L1 explicit-cue → L2 moderate → L3 conceptual, paired positive/negative per level (run-set: Finch API facts + Acme client convention vs ephemeral status / project-local quirk / repo-recorded ADR). Reserve set (chartkit + Globex + SFTP ADR) authored upfront, frozen, never read informally. Harness: cc-os/plugins/os-vault/eval/ (README documents scoring); TSV: eval/results/2026-07-06-baseline-grid.tsv.

Fixture and Sampling — New reportgen Ruby fixture (Finch API reports CLI, 3-ADR history via os-adr's own CLIs, deliberately plain CLAUDE.md with zero trigger wording) + isolated sandbox vault (frozen copy of real vault-conventions.md + 4 topically-disjoint seeded notes). Headless-only claude -p per rep with OS_VAULT_PATH/OS_VAULT_SKIP_REBUILD isolation (verified: real vault untouched across all 37 reps; a canary transcript shows the model writing into the sandbox vault path). 3 reps/cell + counted canary.

Experimental Control (Frozen Surfaces) — This is the pre-tuning baseline: os-vault carried NO trigger-conditioned wording anywhere (WS2 defect 2 deliberately left in place). Checker, fixtures, scenarios, judge rubric frozen. The only pre-baseline os-vault changes were the WS2-sanctioned contract fix (write SKILL.md defers to vault-conventions.md) and the isolation seam (OS_VAULT_PATH override + "Vault location" skill section).

Validity and Limitations

How to Interpret These Results — Held-out baseline, nothing tuned against it yet; the run-set remains the measurement set until a wording loop starts. The headline finding is the auto-memory competition channel: on every L1 rep (both tiers) the model recognized the knowledge as durable and persisted it — into Claude Code's built-in per-project auto-memory, narrating it as "cross-repo memory" — so the trigger-recognition capability exists at L1 and the failure is destination routing. (The user's global ~/.claude/CLAUDE.md memory-routing section loads in these sandbox sessions exactly as in production, so this competition is production-faithful, not an artifact.) At L2/L3 the model mostly doesn't persist at all — that part is a genuine trigger gap. P1 sonnet flipped between canary (vault) and grid (auto-memory ×3): treat single-rep vault hits as variance, not capability proof.

Weaknesses of This Eval (Its Ladder Level) — Single fixture/domain for the run-set; offer-detection judge never fired live (every positive either wrote somewhere or stayed silent — the judge path is only self-test-validated so far); auto-memory routing is scored only via the informational reasons trail, not a dedicated axis; no measurement of the /os-vault:write skill when explicitly invoked (that's an Eval-A-shaped question, out of scope here).

Deployment and Evolution

Good-Enough Gate — Not applicable yet; this grid establishes the gap the WS2 wording loop must close. Negatives at 18/18 mean restraint needs no work — wording iteration should optimize trigger + routing only, and any future negative regression is a hard stop.

Hardening Path / Next Measurement — Per ws2-os-vault-write-eval.md step 4: an /autoresearch wording loop (checker/fixtures/scenarios/rubric frozen; surfaces = write SKILL.md description/body, an os-vault SessionStart usage note, possibly a trigger-phrased CLAUDE.md section mirroring os-adr's). Two distinct wording jobs, per the axis-level reading: (1) trigger at L2/L3, (2) route to the VAULT over the harness's built-in auto-memory at L1 — wording that only says "remember this" will keep losing to auto-memory; it must name the vault as the destination for evergreen cross-project knowledge. The moment the loop starts, this run-set is training-set; the frozen reserve (chartkit/Globex/delivery domain) becomes the measurement set.

Related