638c6d03ed
os-vault: WS2 write-behavior eval harness + untuned baseline grid
...
plugins/os-vault/eval/ — held-out unprompted vault-write discrimination eval:
ambiguity ladder L1 explicit -> L3 conceptual, paired positives/negatives,
6 run-set scenarios on a new reportgen Ruby fixture + 6 frozen reserve twins,
isolated sandbox vault, headless-only runner, deterministic-first Ruby checker
(narrow judge fallback stubbable via OS_VAULT_EVAL_JUDGE_CMD), model-free
self-test 21/21. Scenario Task blocks are held-out; reserve never read.
Isolation seam: OS_VAULT_PATH overrides vault_path via config.load_config();
OS_VAULT_SKIP_REBUILD suppresses the SessionStart graphify rebuild. Write
SKILL.md contract fix (defers to vault-conventions.md) + Vault location section.
Baseline grid (run-set x sonnet/haiku x 3 reps + 1 counted canary, 37 reps):
positives 1/19, negatives 18/18. L1 persists every rep but routes to built-in
auto-memory instead of the vault; L2/L3 mostly don't persist; zero
over-triggering. Wording loop (step 4, not started) targets: trigger at
L2/L3, route to vault at L1. Loop-input candidates: vault note
os-vault-write-eval-baseline-grid-results. CLAUDE.md pointers updated for
this and os-status.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 12:46:22 -04:00
973fbe7a66
Add second-batch plugin eval and architecture plans
...
Three workstreams with shared methodology: os-orchestration session
audit (WS1), os-vault write-behavior eval (WS2), and status-check
convention + glue plugin (WS3). Each specifies phases, discipline
gates, and deliverables, following the audit-first and eval-B/C
playbook established in the first batch. To be triggered in separate
future sessions.
2026-07-06 13:54:14 -04:00
jared