Commit Graph

6 Commits

Author SHA1 Message Date
jared b226601bea os-orchestration: WS4 eval extension — econ axes, E5 batching pair, fable column, pre-registered E2P/E3P criterion redesign
- extract: main-loop/sidechain output-token accounting (subagent transcripts
  persist at <projects>/<flat>/<session-id>/subagents/agent-*.jsonl — the
  mini-audit's instrumentation gap, now solved) + SendMessage round counting
- check: econ info on every row (mltok/sctok/mlshare/prebytes/spawns/maxrounds);
  E2P v2 (scripted-direct branch + services task-complete); E3P v2 (74KB S7
  whole-session ingestion anchor + mechanical root-cause concepts, delegation
  no longer required); E5P/E5N arms (grouped-delegation vs parallel-fanout,
  ## Expected files coverage convention)
- run: fable → claude-fable-5 alias
- scenarios: E5P-handler-rollout + E5N-limit-rebalance (+ frozen reserve twins,
  authored unread by the orchestrating session) + handler-spec/probe-spec fixture docs
- self-test 32/32 model-free; ORCHESTRATION.v3-draft.md parked (NOT shipped —
  Step-3 baseline must run against v2); H3 threshold pre-registered in plan

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 09:49:44 -04:00
jared 638c6d03ed os-vault: WS2 write-behavior eval harness + untuned baseline grid
plugins/os-vault/eval/ — held-out unprompted vault-write discrimination eval:
ambiguity ladder L1 explicit -> L3 conceptual, paired positives/negatives,
6 run-set scenarios on a new reportgen Ruby fixture + 6 frozen reserve twins,
isolated sandbox vault, headless-only runner, deterministic-first Ruby checker
(narrow judge fallback stubbable via OS_VAULT_EVAL_JUDGE_CMD), model-free
self-test 21/21. Scenario Task blocks are held-out; reserve never read.

Isolation seam: OS_VAULT_PATH overrides vault_path via config.load_config();
OS_VAULT_SKIP_REBUILD suppresses the SessionStart graphify rebuild. Write
SKILL.md contract fix (defers to vault-conventions.md) + Vault location section.

Baseline grid (run-set x sonnet/haiku x 3 reps + 1 counted canary, 37 reps):
positives 1/19, negatives 18/18. L1 persists every rep but routes to built-in
auto-memory instead of the vault; L2/L3 mostly don't persist; zero
over-triggering. Wording loop (step 4, not started) targets: trigger at
L2/L3, route to vault at L1. Loop-input candidates: vault note
os-vault-write-eval-baseline-grid-results. CLAUDE.md pointers updated for
this and os-status.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 12:46:22 -04:00
jared 1d1e7207c0 os-status: aggregated in-process SessionStart status checks (ADR-022)
New global plugin with one SessionStart hook running all checks in-process
(hooks/checks.py, check(ctx) -> CheckResult, single registry). Three states:
ok (silent), note (near-zero-token line, never suppressed), warn (one
aggregated banner, once-per-day snooze + permanent suppress in gitignored
per-project .cc-os/). Initial checks: subagent-model-env-override (catches
the WS1 Cluster 1 incident deterministically), adr-system-present (verbatim
port of os-adr's hook, legacy .os-adr/suppress honored), vault-hub-note-present.

os-adr's own SessionStart hook moved here atomically (its hooks.json now
empty; session_start.py stays as wording source of record). Built via
OpenSpec change add-os-status-plugin; 36 model-free tests; smoke-tested
2026-07-06 incl. env-override canary.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 12:46:10 -04:00
jared fe83fdb290 os-orchestration: E1-E3 eval harness + cluster-1 wording v2 (self-report)
Harness: 6 run-set + 6 frozen-reserve paired scenarios from the WS1 verified
misses, Node relaystation fixture (deterministic sandbox-time logs), headless
runner with CLAUDE_CODE_SUBAGENT_MODEL stub for E1P, deterministic-first
checker driving audit/bin/extract (extended with per-segment read targets),
narrow frozen haiku-judge fallback for E1 language axes, model-free self-test
21/21 incl. shipped-instruction-compliant conformance transcripts.

Wording v2: the E1 canary (counted) proved resolvedModel is not model-visible
in the launch result, so the launch-stub-comparison rule was unactionable;
replaced with subagent model self-report on sonnet/opus spawns (subagents know
their exact model ID - verified live). Scenarios/checker/rubric untouched.
Deterministic env-override detection routed to WS3 (plan updated).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 18:04:15 -04:00
jared 7965f033e6 WS1 orchestration audit: findings, extractor, and Cluster 1 root cause
- Add session-audit findings (10 stratified sessions, verified synthesis)
  and E1-E4 eval-scenario backlog: docs/orchestration-audit/
- Add Phase 1 fact-sheet extractor: plugins/os-orchestration/audit/bin/extract
- Post-audit correction: Cluster 1 (all 23 spawns downgraded to haiku) was
  NOT a Fable-5 harness bug — root cause was CLAUDE_CODE_SUBAGENT_MODEL=haiku
  in ~/.claude/settings.json env block, set by an earlier session as a cost
  measure. Removed 2026-07-06. Policy gap (no verify-resolvedModel rule)
  stands regardless; E1 remains valid.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 17:34:42 -04:00
jared 973fbe7a66 Add second-batch plugin eval and architecture plans
Three workstreams with shared methodology: os-orchestration session
audit (WS1), os-vault write-behavior eval (WS2), and status-check
convention + glue plugin (WS3). Each specifies phases, discipline
gates, and deliverables, following the audit-first and eval-B/C
playbook established in the first batch. To be triggered in separate
future sessions.
2026-07-06 13:54:14 -04:00