95 lines
5.4 KiB
Markdown
95 lines
5.4 KiB
Markdown
# WS2 — os-vault "when and how to write" eval
|
||
|
||
_Created: 2026-07-06. Status: proposed. Prereqs: quick wins immediate; harness after WS3
|
||
design + quick wins. See `2026-07-06-plugin-evals-overview.md`._
|
||
|
||
## Goal
|
||
|
||
Make os-vault reliably know WHEN to write a vault note unprompted and HOW to write it
|
||
correctly (frontmatter contract), measured by a held-out discrimination eval — applying the
|
||
full os-adr first-batch playbook.
|
||
|
||
## Known defects found 2026-07-06 (fix before baselining)
|
||
|
||
1. **`skills/write/SKILL.md` contradicts the reconciled vault-conventions.md.** The skill
|
||
still mandates `scope/global` as a TAG and a type list of
|
||
`procedure|reference|log|hub|concept|decision`; vault-conventions (Phase 1 SB content
|
||
plan, 2026-06-30) made `scope:` a frontmatter FIELD and the live vault uses types like
|
||
`howto` and `eval-results`. Fix: make vault-conventions.md the single source of truth —
|
||
the skill summarizes + points, never duplicates the schema. Then `bin/refresh-plugins`.
|
||
2. **No trigger-conditioned wording anywhere in os-vault.** The core Eval B lesson
|
||
(when→then phrasing; mechanical triggers for lower tiers; each rule lives where its
|
||
precondition is visible) was never applied. Nothing tells the model when to invoke
|
||
`/os-vault:write` unprompted. Do NOT fix this before the baseline grid — it is the
|
||
tuning surface the eval measures.
|
||
3. **cc-os has no project hub note** despite hub notes being the connectivity mechanism
|
||
(ADR-014). Quick win: author it now (one note). AUTOMATIC hub creation is designed into
|
||
`onboard-project` + the WS3 status check — not hacked in here.
|
||
|
||
Quick wins = items 1 and 3. Both are small; do directly or via one haiku subagent each.
|
||
|
||
## Behavior under test
|
||
|
||
Unprompted vault-write discrimination during a normal working session:
|
||
|
||
- **Positives:** session surfaces evergreen cross-project knowledge (tool/API behavior
|
||
discovered, client-specific fact, a convention established with the user) → model writes
|
||
or offers a vault note with contract-conforming frontmatter (summary at creation, scope
|
||
field, type/ + ≥1 facet tag).
|
||
- **Negatives (equally tempting):** ephemeral material — in-progress task state,
|
||
project-local debugging detail, decisions already recorded in repo docs/ADRs → no vault
|
||
write. Over-trigger fail line: writing ephemera to the vault (or duplicating
|
||
repo-recorded knowledge).
|
||
|
||
Ambiguity ladder: L1 explicit cue ("worth remembering") → L2 moderate → L3 conceptual
|
||
(knowledge emerges implicitly from debugging). Paired positive/negative per level.
|
||
|
||
## Harness (Eval B/C shape)
|
||
|
||
- Location: `plugins/os-vault/eval/`. Run-set + frozen reserve authored upfront (2×), in
|
||
different knowledge domains. **Task blocks held-out — never run informally; reserve
|
||
never read informally.**
|
||
- **Headless-only** (`claude -p`, cwd = sandbox fixture project, real SessionStart hook
|
||
fires). In-session subagents are invalid for unprompted-behavior measurement.
|
||
- Sandbox must use an ISOLATED vault path (env-pointed test vault), never the real
|
||
`~/Documents/SecondBrain` — the checker inspects the sandbox vault for written notes.
|
||
Verify the plugin's vault path is env-overridable (config.yaml); if not, that's a small
|
||
pre-harness plugin change.
|
||
- Checker: Ruby (Sandi Metz style), deterministic-first. Axis (a): did it write/offer a
|
||
vault note (mechanical from transcript tool_use + sandbox vault diff). Axis (b):
|
||
frontmatter contract conformance — almost fully mechanical (summary present, scope
|
||
field, required tag groups); narrow frozen haiku judge only for "offered but did not
|
||
write" phrasing, stubbable via env var. Negatives: sole fail line = unneeded vault
|
||
write; consultation informational. TSV mode for grids.
|
||
|
||
## Discipline gates (all mandatory before the first grid)
|
||
|
||
1. Conformance dry-run on paper: would a perfectly write-skill-compliant model pass every
|
||
cell? Would an always-write model pass positives but fail negatives?
|
||
2. Model-free self-test including at least one fabricated SHIPPED-INSTRUCTION-COMPLIANT
|
||
transcript (not just the designer's imagined ideal).
|
||
3. Canary cell first (one cell, hand-verify TSV vs raw transcript, count the result).
|
||
4. `bin/refresh-plugins` before every grid; check TSV row counts == expected reps
|
||
(pipefail/tee gotcha); rescore-don't-discard on instrument fixes.
|
||
|
||
## Sequence
|
||
|
||
1. Quick wins (contract fix + hub note) → refresh caches.
|
||
2. Build harness (fixture project + isolated vault + 6-ADR-equivalent seeded vault history
|
||
for negatives to cite): fixture generation → sonnet subagent; checker → sonnet subagent
|
||
(Ruby); scenario authoring + conformance dry-run → ORCHESTRATOR DIRECTLY (Eval C's
|
||
defects came from exactly this step; do not delegate it).
|
||
3. **Baseline grid untuned** (post-bugfix, pre-trigger-wording): run-set × {sonnet, haiku}
|
||
× 3 reps. This measures the actual gap.
|
||
4. If gapped: `/autoresearch` wording loop per the howto note (checker/fixtures/scenarios
|
||
frozen; only wording moves; reduced inner grid; full grid to confirm). Expected surfaces:
|
||
session_start hook note, write SKILL.md description + body, possibly a CLAUDE.md
|
||
trigger-phrased section template (mirror os-adr's).
|
||
5. Results → vault `eval-results` note (template `_templates/eval-results.md`); update
|
||
cc-os CLAUDE.md pointers.
|
||
|
||
## Models
|
||
|
||
Sonnet for fixture/checker construction and any judging; haiku for mechanical edits;
|
||
orchestrator authors scenarios and acts as final judge over all grid claims.
|