Commit Graph

5 Commits

Author SHA1 Message Date
jared 638c6d03ed os-vault: WS2 write-behavior eval harness + untuned baseline grid
plugins/os-vault/eval/ — held-out unprompted vault-write discrimination eval:
ambiguity ladder L1 explicit -> L3 conceptual, paired positives/negatives,
6 run-set scenarios on a new reportgen Ruby fixture + 6 frozen reserve twins,
isolated sandbox vault, headless-only runner, deterministic-first Ruby checker
(narrow judge fallback stubbable via OS_VAULT_EVAL_JUDGE_CMD), model-free
self-test 21/21. Scenario Task blocks are held-out; reserve never read.

Isolation seam: OS_VAULT_PATH overrides vault_path via config.load_config();
OS_VAULT_SKIP_REBUILD suppresses the SessionStart graphify rebuild. Write
SKILL.md contract fix (defers to vault-conventions.md) + Vault location section.

Baseline grid (run-set x sonnet/haiku x 3 reps + 1 counted canary, 37 reps):
positives 1/19, negatives 18/18. L1 persists every rep but routes to built-in
auto-memory instead of the vault; L2/L3 mostly don't persist; zero
over-triggering. Wording loop (step 4, not started) targets: trigger at
L2/L3, route to vault at L1. Loop-input candidates: vault note
os-vault-write-eval-baseline-grid-results. CLAUDE.md pointers updated for
this and os-status.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 12:46:22 -04:00
jared 1d1e7207c0 os-status: aggregated in-process SessionStart status checks (ADR-022)
New global plugin with one SessionStart hook running all checks in-process
(hooks/checks.py, check(ctx) -> CheckResult, single registry). Three states:
ok (silent), note (near-zero-token line, never suppressed), warn (one
aggregated banner, once-per-day snooze + permanent suppress in gitignored
per-project .cc-os/). Initial checks: subagent-model-env-override (catches
the WS1 Cluster 1 incident deterministically), adr-system-present (verbatim
port of os-adr's hook, legacy .os-adr/suppress honored), vault-hub-note-present.

os-adr's own SessionStart hook moved here atomically (its hooks.json now
empty; session_start.py stays as wording source of record). Built via
OpenSpec change add-os-status-plugin; 36 model-free tests; smoke-tested
2026-07-06 incl. env-override canary.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 12:46:10 -04:00
jared fe83fdb290 os-orchestration: E1-E3 eval harness + cluster-1 wording v2 (self-report)
Harness: 6 run-set + 6 frozen-reserve paired scenarios from the WS1 verified
misses, Node relaystation fixture (deterministic sandbox-time logs), headless
runner with CLAUDE_CODE_SUBAGENT_MODEL stub for E1P, deterministic-first
checker driving audit/bin/extract (extended with per-segment read targets),
narrow frozen haiku-judge fallback for E1 language axes, model-free self-test
21/21 incl. shipped-instruction-compliant conformance transcripts.

Wording v2: the E1 canary (counted) proved resolvedModel is not model-visible
in the launch result, so the launch-stub-comparison rule was unactionable;
replaced with subagent model self-report on sonnet/opus spawns (subagents know
their exact model ID - verified live). Scenarios/checker/rubric untouched.
Deterministic env-override detection routed to WS3 (plan updated).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 18:04:15 -04:00
jared 7965f033e6 WS1 orchestration audit: findings, extractor, and Cluster 1 root cause
- Add session-audit findings (10 stratified sessions, verified synthesis)
  and E1-E4 eval-scenario backlog: docs/orchestration-audit/
- Add Phase 1 fact-sheet extractor: plugins/os-orchestration/audit/bin/extract
- Post-audit correction: Cluster 1 (all 23 spawns downgraded to haiku) was
  NOT a Fable-5 harness bug — root cause was CLAUDE_CODE_SUBAGENT_MODEL=haiku
  in ~/.claude/settings.json env block, set by an earlier session as a cost
  measure. Removed 2026-07-06. Policy gap (no verify-resolvedModel rule)
  stands regardless; E1 remains valid.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 17:34:42 -04:00
jared 973fbe7a66 Add second-batch plugin eval and architecture plans
Three workstreams with shared methodology: os-orchestration session
audit (WS1), os-vault write-behavior eval (WS2), and status-check
convention + glue plugin (WS3). Each specifies phases, discipline
gates, and deliverables, following the audit-first and eval-B/C
playbook established in the first batch. To be triggered in separate
future sessions.
2026-07-06 13:54:14 -04:00