jared
/
cc-os
1
0
Fork 0
Code Issues 18 Pull Requests Packages Projects Releases Wiki Activity
cc-os/plugins/os-vault/eval/results/2026-07-06-baseline-grid.tsv

38 lines
4.1 KiB
Plaintext
Raw Normal View History

os-vault: WS2 write-behavior eval harness + untuned baseline grid plugins/os-vault/eval/ — held-out unprompted vault-write discrimination eval: ambiguity ladder L1 explicit -> L3 conceptual, paired positives/negatives, 6 run-set scenarios on a new reportgen Ruby fixture + 6 frozen reserve twins, isolated sandbox vault, headless-only runner, deterministic-first Ruby checker (narrow judge fallback stubbable via OS_VAULT_EVAL_JUDGE_CMD), model-free self-test 21/21. Scenario Task blocks are held-out; reserve never read. Isolation seam: OS_VAULT_PATH overrides vault_path via config.load_config(); OS_VAULT_SKIP_REBUILD suppresses the SessionStart graphify rebuild. Write SKILL.md contract fix (defers to vault-conventions.md) + Vault location section. Baseline grid (run-set x sonnet/haiku x 3 reps + 1 counted canary, 37 reps): positives 1/19, negatives 18/18. L1 persists every rep but routes to built-in auto-memory instead of the vault; L2/L3 mostly don't persist; zero over-triggering. Wording loop (step 4, not started) targets: trigger at L2/L3, route to vault at L1. Loop-input candidates: vault note os-vault-write-eval-baseline-grid-results. CLAUDE.md pointers updated for this and os-status. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 16:46:22 +00:00
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
N2-L2-reporting haiku W:no B:PASS PASS project-write:yes
N2-L2-reporting haiku W:no B:PASS PASS project-write:yes
N2-L2-reporting haiku W:no B:PASS PASS project-write:yes
N2-L2-reporting sonnet W:no B:PASS PASS project-write:yes
N2-L2-reporting sonnet W:no B:PASS PASS project-write:yes
N2-L2-reporting sonnet W:no B:PASS PASS project-write:yes
N3-L3-reporting haiku W:no B:PASS PASS project-write:yes
N3-L3-reporting haiku W:no B:PASS PASS project-write:yes
N3-L3-reporting haiku W:no B:PASS PASS project-write:yes
N3-L3-reporting sonnet W:no B:PASS PASS project-write:yes
N3-L3-reporting sonnet W:no B:PASS PASS project-write:yes
N3-L3-reporting sonnet W:no B:PASS PASS project-write:yes
P1-L1-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P1-L1-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P1-L1-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P1-L1-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P1-L1-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P1-L1-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P2-L2-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P2-L2-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P2-L2-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P3-L3-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P3-L3-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P3-L3-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance