Commit Graph

10 Commits

Author SHA1 Message Date
jared 242ca84744 os-orchestration: WS4 IRL re-audit + Tier 1/2 wording patch
First production IRL audit (19 transcripts post-2026-07-08, 3 parallel
auditors; run artifacts in autoresearch/improve-260710-1057/,
gitignored). Solved: model-param discipline, over-delegation. Patched
into ORCHESTRATION.md: settled-design mid-session fan-out trigger,
never-sleep-poll rule, own eval/benchmark loops named as mechanical,
unfamiliar-tree Explore trigger. SendMessage example skipped (no
violations; token budget). One-shot patch — WS4 grids contaminated;
acceptance = next IRL audit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XknQRvihHDpYE47RTmUR4N
2026-07-10 11:24:59 -04:00
jared 397145b27b os-orchestration: WS4 wording loop shipped — ORCHESTRATION.md v3
Candidate-1 (v3 draft) accepted on iteration 1, confirmed on full grid,
validated on frozen reserve (15/16 cells, negatives 20/20, E5P reserve twin
5/5 both tiers). H3 met: E5P fable median mltok 27,372 = 43% of 62,971
baseline. Results TSVs + loop log + CLAUDE.md/plan status; eval-results vault
note written; limit-banner sweep + econ-guard lessons promoted to methodology
notes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 11:18:43 -04:00
jared ab258d5ba1 os-orchestration eval: key E3P root-cause concepts by scenario id
The reserve twin E3P-retry-storm targets gen-logs' other planted incident
(2026-07-02 auth-token-expiry 401 storm) and was authored before axis B
existed; the single concept table would have failed a compliant model.
Fixed pre-reserve-measurement without reading reserve files (incident chain
comes from gen-logs). Run-set scoring byte-identical; self-test 34/34 incl.
keying proof both directions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 11:05:03 -04:00
jared a771163663 os-orchestration: WS4 pre-wording baseline grid (fable+sonnet, econ axes)
44 counted reps (15 session-limit-truncated reps mechanically excluded and
replaced; preserved as comments). E5P 0/6 main-loop-grind both tiers (fable
median 62,971 mltok — H3 target cell); E2P/E3P criterion redesign validated
(6/6 PASS each via scripted-direct / surgical-direct); negatives 100% clean;
E1P sonnet 0/3, fable 1/2 (first fable self-report miss). H3 amended pre-loop:
E5P-fable reduction target, E3P non-regression guard.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 10:39:05 -04:00
jared b226601bea os-orchestration: WS4 eval extension — econ axes, E5 batching pair, fable column, pre-registered E2P/E3P criterion redesign
- extract: main-loop/sidechain output-token accounting (subagent transcripts
  persist at <projects>/<flat>/<session-id>/subagents/agent-*.jsonl — the
  mini-audit's instrumentation gap, now solved) + SendMessage round counting
- check: econ info on every row (mltok/sctok/mlshare/prebytes/spawns/maxrounds);
  E2P v2 (scripted-direct branch + services task-complete); E3P v2 (74KB S7
  whole-session ingestion anchor + mechanical root-cause concepts, delegation
  no longer required); E5P/E5N arms (grouped-delegation vs parallel-fanout,
  ## Expected files coverage convention)
- run: fable → claude-fable-5 alias
- scenarios: E5P-handler-rollout + E5N-limit-rebalance (+ frozen reserve twins,
  authored unread by the orchestrating session) + handler-spec/probe-spec fixture docs
- self-test 32/32 model-free; ORCHESTRATION.v3-draft.md parked (NOT shipped —
  Step-3 baseline must run against v2); H3 threshold pre-registered in plan

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 09:49:44 -04:00
jared c2098525fe os-orchestration: baseline grid results + scripted-bulk-edit carve-out
Grid (36 reps + 2 canaries, all counted): negatives 18/18 both tiers, zero
over-triggering; positives 1/18 on delegate-at-all, but transcript
verification splits those FAILs into superior direct strategies (scripted
bulk edit, surgical greps - the audit's own justified-non-delegation shape)
vs genuine misses (haiku whole-file ingestion, per-file grind). Conditional
rules validated where engaged: 14/14 spawns explicit-model, self-report
downgrade flag worked. E2P/E3P criterion redesign pre-registered with
independently-anchored thresholds; reserve untouched. Analysis in vault:
os-orchestration-eval-baseline-grid-results.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 18:35:40 -04:00
jared fe83fdb290 os-orchestration: E1-E3 eval harness + cluster-1 wording v2 (self-report)
Harness: 6 run-set + 6 frozen-reserve paired scenarios from the WS1 verified
misses, Node relaystation fixture (deterministic sandbox-time logs), headless
runner with CLAUDE_CODE_SUBAGENT_MODEL stub for E1P, deterministic-first
checker driving audit/bin/extract (extended with per-segment read targets),
narrow frozen haiku-judge fallback for E1 language axes, model-free self-test
21/21 incl. shipped-instruction-compliant conformance transcripts.

Wording v2: the E1 canary (counted) proved resolvedModel is not model-visible
in the launch result, so the launch-stub-comparison rule was unactionable;
replaced with subagent model self-report on sonnet/opus spawns (subagents know
their exact model ID - verified live). Scenarios/checker/rubric untouched.
Deterministic env-override detection routed to WS3 (plan updated).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 18:04:15 -04:00
jared b01966bb10 os-orchestration: trigger-conditioned wording for audit clusters 1-3
Applies the Eval B when->then lesson to the three verified WS1 clusters:
explicit model: before every Agent call (cluster 2, cost exposure now that
CLAUDE_CODE_SUBAGENT_MODEL is removed), verify resolved model after launch
(cluster 1 policy gap), don't re-cover own ground before delegating
(cluster 3). Delegation thresholds unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 17:44:09 -04:00
jared 7965f033e6 WS1 orchestration audit: findings, extractor, and Cluster 1 root cause
- Add session-audit findings (10 stratified sessions, verified synthesis)
  and E1-E4 eval-scenario backlog: docs/orchestration-audit/
- Add Phase 1 fact-sheet extractor: plugins/os-orchestration/audit/bin/extract
- Post-audit correction: Cluster 1 (all 23 spawns downgraded to haiku) was
  NOT a Fable-5 harness bug — root cause was CLAUDE_CODE_SUBAGENT_MODEL=haiku
  in ~/.claude/settings.json env block, set by an earlier session as a cost
  measure. Removed 2026-07-06. Policy gap (no verify-resolvedModel rule)
  stands regardless; E1 remains valid.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 17:34:42 -04:00
jared cebcc1b649 Migrate os-orchestration plugin from cc-plugins into cc-os
Renamed orchestration -> os-orchestration per naming convention, kept the
plugin's permissive delegation rule as the canonical global default, and
dropped cc-os's stricter local override so this repo behaves like every
other project (ADR-019).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-03 10:14:35 -04:00