Commit Graph

6 Commits

Author SHA1 Message Date
jared 397145b27b os-orchestration: WS4 wording loop shipped — ORCHESTRATION.md v3
Candidate-1 (v3 draft) accepted on iteration 1, confirmed on full grid,
validated on frozen reserve (15/16 cells, negatives 20/20, E5P reserve twin
5/5 both tiers). H3 met: E5P fable median mltok 27,372 = 43% of 62,971
baseline. Results TSVs + loop log + CLAUDE.md/plan status; eval-results vault
note written; limit-banner sweep + econ-guard lessons promoted to methodology
notes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 11:18:43 -04:00
jared ab258d5ba1 os-orchestration eval: key E3P root-cause concepts by scenario id
The reserve twin E3P-retry-storm targets gen-logs' other planted incident
(2026-07-02 auth-token-expiry 401 storm) and was authored before axis B
existed; the single concept table would have failed a compliant model.
Fixed pre-reserve-measurement without reading reserve files (incident chain
comes from gen-logs). Run-set scoring byte-identical; self-test 34/34 incl.
keying proof both directions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 11:05:03 -04:00
jared a771163663 os-orchestration: WS4 pre-wording baseline grid (fable+sonnet, econ axes)
44 counted reps (15 session-limit-truncated reps mechanically excluded and
replaced; preserved as comments). E5P 0/6 main-loop-grind both tiers (fable
median 62,971 mltok — H3 target cell); E2P/E3P criterion redesign validated
(6/6 PASS each via scripted-direct / surgical-direct); negatives 100% clean;
E1P sonnet 0/3, fable 1/2 (first fable self-report miss). H3 amended pre-loop:
E5P-fable reduction target, E3P non-regression guard.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 10:39:05 -04:00
jared b226601bea os-orchestration: WS4 eval extension — econ axes, E5 batching pair, fable column, pre-registered E2P/E3P criterion redesign
- extract: main-loop/sidechain output-token accounting (subagent transcripts
  persist at <projects>/<flat>/<session-id>/subagents/agent-*.jsonl — the
  mini-audit's instrumentation gap, now solved) + SendMessage round counting
- check: econ info on every row (mltok/sctok/mlshare/prebytes/spawns/maxrounds);
  E2P v2 (scripted-direct branch + services task-complete); E3P v2 (74KB S7
  whole-session ingestion anchor + mechanical root-cause concepts, delegation
  no longer required); E5P/E5N arms (grouped-delegation vs parallel-fanout,
  ## Expected files coverage convention)
- run: fable → claude-fable-5 alias
- scenarios: E5P-handler-rollout + E5N-limit-rebalance (+ frozen reserve twins,
  authored unread by the orchestrating session) + handler-spec/probe-spec fixture docs
- self-test 32/32 model-free; ORCHESTRATION.v3-draft.md parked (NOT shipped —
  Step-3 baseline must run against v2); H3 threshold pre-registered in plan

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 09:49:44 -04:00
jared c2098525fe os-orchestration: baseline grid results + scripted-bulk-edit carve-out
Grid (36 reps + 2 canaries, all counted): negatives 18/18 both tiers, zero
over-triggering; positives 1/18 on delegate-at-all, but transcript
verification splits those FAILs into superior direct strategies (scripted
bulk edit, surgical greps - the audit's own justified-non-delegation shape)
vs genuine misses (haiku whole-file ingestion, per-file grind). Conditional
rules validated where engaged: 14/14 spawns explicit-model, self-report
downgrade flag worked. E2P/E3P criterion redesign pre-registered with
independently-anchored thresholds; reserve untouched. Analysis in vault:
os-orchestration-eval-baseline-grid-results.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 18:35:40 -04:00
jared fe83fdb290 os-orchestration: E1-E3 eval harness + cluster-1 wording v2 (self-report)
Harness: 6 run-set + 6 frozen-reserve paired scenarios from the WS1 verified
misses, Node relaystation fixture (deterministic sandbox-time logs), headless
runner with CLAUDE_CODE_SUBAGENT_MODEL stub for E1P, deterministic-first
checker driving audit/bin/extract (extended with per-segment read targets),
narrow frozen haiku-judge fallback for E1 language axes, model-free self-test
21/21 incl. shipped-instruction-compliant conformance transcripts.

Wording v2: the E1 canary (counted) proved resolvedModel is not model-visible
in the launch result, so the launch-stub-comparison rule was unactionable;
replaced with subagent model self-report on sonnet/opus spawns (subagents know
their exact model ID - verified live). Scenarios/checker/rubric untouched.
Deterministic env-override detection routed to WS3 (plan updated).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 18:04:15 -04:00