diff --git a/cc-os-hub.md b/cc-os-hub.md index 60daffb..b5612ee 100644 --- a/cc-os-hub.md +++ b/cc-os-hub.md @@ -67,8 +67,10 @@ procedure, production feedback loop). Concrete results: - **os-orchestration:** [[os-orchestration-ws1-session-audit-results]] (10-session IRL audit; found the `CLAUDE_CODE_SUBAGENT_MODEL` env override) → [[os-orchestration-eval-baseline-grid-results]] (E1–E3 baseline grid). -- **os-vault:** write-skill discrimination eval planned, not yet run — see - `docs/plans/ws2-os-vault-write-eval.md` in the cc-os repo. +- **os-vault:** [[os-vault-write-eval-baseline-grid-results]] (2026-07-06 untuned baseline: + positives 1/19, negatives 18/18; L1 knowledge routes to Claude Code auto-memory instead of + the vault — routing plus L2/L3 triggering are the wording-loop targets; harness at + `cc-os/plugins/os-vault/eval/`, wording loop not yet started). ## Related diff --git a/journal/2026-07-07.md b/journal/2026-07-07.md new file mode 100644 index 0000000..1dcc916 --- /dev/null +++ b/journal/2026-07-07.md @@ -0,0 +1,11 @@ +--- +summary: Daily session log +tags: [scope/global, type/log] +--- + +## Session — 2026-07-07T00:08:54Z + +**Project:** /home/jared/dev/cc-os +**Reason:** other +**Vault notes touched:** +(none) diff --git a/os-vault-write-eval-baseline-grid-results.md b/os-vault-write-eval-baseline-grid-results.md new file mode 100644 index 0000000..687bbca --- /dev/null +++ b/os-vault-write-eval-baseline-grid-results.md @@ -0,0 +1,108 @@ +--- +type: eval-results +title: os-vault write eval — untuned baseline grid (unprompted vault-write discrimination) +summary: At baseline (no trigger wording anywhere in os-vault), positives are 1/19 across both tiers while negatives are 18/18 clean — and the dominant failure shape is not "no persistence" but mis-routing to Claude Code's built-in auto-memory instead of the vault at L1; L2/L3 mostly don't persist at all. +tags: + - type/eval-results + - domain/llm-evaluation + - tool/os-vault + - tool/claude-code + - project/cc-os +scope: global +last_updated: 2026-07-06 +date: 2026-07-06 +related: + - eval-methodology-ladder + - running-autoresearch-skill-evals + - os-adr-eval-b-wording-experiment-hypotheses + - os-adr-eval-c-frozen-grid-results +source: cc-os +--- + +# os-vault write eval — untuned baseline grid (2026-07-06) + +## Results Grid/Threshold + +**Passing criterion:** positives — unprompted vault write with contract-conforming +frontmatter (axis a: written or offered; axis b: mechanical frontmatter conformance); +negatives — no vault write (the sole fail line). Run-set × {sonnet, haiku} × 3 reps + 1 +counted sonnet canary rep on P1 (37 counted reps total, zero harness errors, all rows +verified against expected rep counts). + +| Cell | sonnet | haiku | Dominant behavior | +|---|---|---|---| +| P1-L1 explicit cue (Finch 500-cap) | 1/4 (canary only) | 0/3 | Persists EVERY rep — but to auto-memory (`~/.claude/projects//memory/`), not the vault | +| P2-L2 no cue (Finch expires_in) | 0/3 | 0/3 | Fixes the code; no persistence anywhere | +| P3-L3 conceptual (Acme convention) | 0/3 | 0/3 | sonnet 1/3 wrote auto-memory; otherwise nothing | +| N1-L1 explicit cue, ephemeral | 3/3 | 3/3 | Status written to project file (`project-write:yes`), vault untouched | +| N2-L2 project-local quirk | 3/3 | 3/3 | Clean fix, no vault write | +| N3-L3 repo-recorded (ADR 0002) | 3/3 | 3/3 | Complies with the ADR, no vault duplicate | + +**Summary:** positives 1/19 (the one pass: sonnet canary P1, which wrote a fully conforming +`reference/finch-api-page-size-silent-cap.md` — proof the ceiling exists); negatives 18/18 +with zero over-trigger at any level on either tier. The measured gap is real and one-sided: +under-triggering plus destination mis-routing, no restraint problem. + +## Measurement Setup + +**Hypotheses / Scenarios Tested** — Ambiguity ladder L1 explicit-cue → L2 moderate → L3 +conceptual, paired positive/negative per level (run-set: Finch API facts + Acme client +convention vs ephemeral status / project-local quirk / repo-recorded ADR). Reserve set +(chartkit + Globex + SFTP ADR) authored upfront, frozen, never read informally. Harness: +`cc-os/plugins/os-vault/eval/` (README documents scoring); TSV: +`eval/results/2026-07-06-baseline-grid.tsv`. + +**Fixture and Sampling** — New `reportgen` Ruby fixture (Finch API reports CLI, 3-ADR +history via os-adr's own CLIs, deliberately plain CLAUDE.md with zero trigger wording) + +isolated sandbox vault (frozen copy of real vault-conventions.md + 4 topically-disjoint +seeded notes). Headless-only `claude -p` per rep with `OS_VAULT_PATH`/`OS_VAULT_SKIP_REBUILD` +isolation (verified: real vault untouched across all 37 reps; a canary transcript shows the +model writing into the sandbox vault path). 3 reps/cell + counted canary. + +**Experimental Control (Frozen Surfaces)** — This is the pre-tuning baseline: os-vault +carried NO trigger-conditioned wording anywhere (WS2 defect 2 deliberately left in place). +Checker, fixtures, scenarios, judge rubric frozen. The only pre-baseline os-vault changes +were the WS2-sanctioned contract fix (write SKILL.md defers to vault-conventions.md) and the +isolation seam (OS_VAULT_PATH override + "Vault location" skill section). + +## Validity and Limitations + +**How to Interpret These Results** — Held-out baseline, nothing tuned against it yet; the +run-set remains the measurement set until a wording loop starts. The headline finding is the +**auto-memory competition channel**: on every L1 rep (both tiers) the model recognized the +knowledge as durable and persisted it — into Claude Code's built-in per-project auto-memory, +narrating it as "cross-repo memory" — so the trigger-recognition capability exists at L1 and +the failure is destination routing. (The user's global `~/.claude/CLAUDE.md` memory-routing +section loads in these sandbox sessions exactly as in production, so this competition is +production-faithful, not an artifact.) At L2/L3 the model mostly doesn't persist at all — +that part is a genuine trigger gap. P1 sonnet flipped between canary (vault) and grid +(auto-memory ×3): treat single-rep vault hits as variance, not capability proof. + +**Weaknesses of This Eval (Its Ladder Level)** — Single fixture/domain for the run-set; +offer-detection judge never fired live (every positive either wrote somewhere or stayed +silent — the judge path is only self-test-validated so far); auto-memory routing is scored +only via the informational reasons trail, not a dedicated axis; no measurement of the +`/os-vault:write` skill when explicitly invoked (that's an Eval-A-shaped question, out of +scope here). + +## Deployment and Evolution + +**Good-Enough Gate** — Not applicable yet; this grid establishes the gap the WS2 wording +loop must close. Negatives at 18/18 mean restraint needs no work — wording iteration should +optimize trigger + routing only, and any future negative regression is a hard stop. + +**Hardening Path / Next Measurement** — Per `ws2-os-vault-write-eval.md` step 4: an +`/autoresearch` wording loop (checker/fixtures/scenarios/rubric frozen; surfaces = write +SKILL.md description/body, an os-vault SessionStart usage note, possibly a trigger-phrased +CLAUDE.md section mirroring os-adr's). Two distinct wording jobs, per the axis-level +reading: (1) trigger at L2/L3, (2) route to the VAULT over the harness's built-in +auto-memory at L1 — wording that only says "remember this" will keep losing to auto-memory; +it must name the vault as the destination for evergreen cross-project knowledge. The moment +the loop starts, this run-set is training-set; the frozen reserve +(chartkit/Globex/delivery domain) becomes the measurement set. + +## Related + +- [[eval-methodology-ladder]] — ladder design + reserve discipline this harness follows +- [[running-autoresearch-skill-evals]] — loop discipline for the wording iteration ahead +- [[os-adr-eval-c-frozen-grid-results]] — the sibling discrimination eval whose shape this reuses