2026-07-13 13:18:23 +00:00
|
|
|
|
# os-context (formerly os-orchestration) — component detail
|
|
|
|
|
|
|
|
|
|
|
|
_Leaf file of [../implementation-status.md](../implementation-status.md). Read on demand._
|
|
|
|
|
|
|
|
|
|
|
|
**Global os-context plugin** — `cc-os/plugins/os-context/` (git-tracked); symlinked into
|
|
|
|
|
|
`~/.claude/plugins/os-context`. Migrated 2026-07-03 from a standalone repo
|
|
|
|
|
|
(`~/dev/cc-plugins/orchestration/`), superseding per-project orchestration text (ADR-0019);
|
|
|
|
|
|
renamed and generalized from os-orchestration 2026-07-13 (ADR-0031). Entries below predating
|
|
|
|
|
|
the rename refer to the plugin by its former name.
|
|
|
|
|
|
|
|
|
|
|
|
## Component inventory
|
|
|
|
|
|
|
|
|
|
|
|
- Hooks: `hooks/inject.py` — SessionStart prompt composer: globs
|
|
|
|
|
|
`prompts/session-start/*.md`, sorts by filename, concatenates into one additionalContext
|
|
|
|
|
|
block for all sessions, with a deterministic line-count budget (warn >120 lines, refuse
|
|
|
|
|
|
>240 lines). Byte-identical output to the old single-file `ORCHESTRATION.md` behavior when
|
|
|
|
|
|
only `10-orchestration.md` is present (tested).
|
|
|
|
|
|
- Prompt files (2026-07-13): `10-orchestration.md` (the canonical session-orchestration +
|
|
|
|
|
|
delegation-economics rules; formerly `ORCHESTRATION.md`), `20-kickoff-conventions.md`,
|
|
|
|
|
|
`30-standing-safety.md`, `40-decision-memos.md` (43 lines; total injection 102/120).
|
|
|
|
|
|
Decision-memo auto-memory file deleted per single-source rule.
|
|
|
|
|
|
- Skill: `/os-context:audit-sessions` (formerly `/os-orchestration:audit-sessions`).
|
|
|
|
|
|
- State: `~/.local/state/os-context/` (old `os-orchestration` path symlinked;
|
|
|
|
|
|
`OS_ORCH_AUDIT_LEDGER` env var name kept). os-status's `orchestration-audit-due` check
|
|
|
|
|
|
updated to match.
|
|
|
|
|
|
|
|
|
|
|
|
## Session audits and wording iterations
|
|
|
|
|
|
|
|
|
|
|
|
- **WS1 session audit (2026-07-06):** 10 stratified real sessions audited via
|
|
|
|
|
|
`audit/bin/extract` (Ruby fact-sheet extractor, dual-use as future eval checker) +
|
|
|
|
|
|
parallel sonnet auditors + verified synthesis. Findings + E1–E4 eval-scenario backlog:
|
|
|
|
|
|
`docs/orchestration-audit/2026-07-06-findings.md`. Headlines: model param silently
|
|
|
|
|
|
overridden to haiku on all 23 verified spawns — root cause
|
|
|
|
|
|
`CLAUDE_CODE_SUBAGENT_MODEL=haiku` in `~/.claude/settings.json` env block (removed
|
|
|
|
|
|
2026-07-06); omitted model param inherits the main-loop model; no over-sharing/full-dump
|
|
|
|
|
|
failures anywhere.
|
|
|
|
|
|
- **Wording fixes shipped (2026-07-06, post-audit):** ORCHESTRATION.md rewritten with
|
|
|
|
|
|
trigger-conditioned when→then phrasing for the three verified clusters — explicit
|
|
|
|
|
|
`model:` before every Agent call, subagent model self-report on sonnet/opus spawns (the E1
|
|
|
|
|
|
canary proved `resolvedModel` is NOT model-visible in the launch result; self-report is
|
|
|
|
|
|
the only observable channel), don't re-cover your own ground before delegating.
|
|
|
|
|
|
Delegation thresholds unchanged. Deterministic env-override detection delegated to
|
|
|
|
|
|
os-status (WS3).
|
|
|
|
|
|
- **Eval harness E1–E3 (2026-07-06):** `plugins/os-context/eval/` — scripted headless
|
|
|
|
|
|
orchestration-behavior eval scored by the audit extractor (extended with per-segment read
|
|
|
|
|
|
targets). 6 run-set scenarios (E1P/N downgrade detection with runner stub; E2P/N
|
|
|
|
|
|
explicit-model fan-out vs no-delegation-on-trivial; E3P/N delegate-investigation vs
|
|
|
|
|
|
orienting-reads-correct) + 6 frozen reserve twins; Node.js "relaystation" fixture with
|
|
|
|
|
|
deterministic sandbox-time log generation; deterministic-first checker (`eval/bin/check`,
|
|
|
|
|
|
narrow frozen haiku-judge fallback for E1 language axes only, stubbable via
|
|
|
|
|
|
`ORCH_EVAL_JUDGE_CMD`); model-free `bin/self-test` 21/21. **Scenario Task blocks are
|
|
|
|
|
|
held-out — never run informally.** Procedure: `plugins/os-context/eval/README.md`.
|
|
|
|
|
|
- **Baseline grid (2026-07-06,** run-set × sonnet/haiku × 3 reps + 2 counted canaries):
|
|
|
|
|
|
negatives 18/18 PASS both tiers (zero over-delegation, zero false downgrade claims);
|
|
|
|
|
|
positives 1/18 on delegate-at-all — but transcript verification shows most positive FAILs
|
|
|
|
|
|
are *superior direct strategies* (scripted bulk edit; surgical greps over 400KB logs),
|
|
|
|
|
|
which the WS1 audit itself scored as justified non-delegation, vs genuine miss shapes
|
|
|
|
|
|
(haiku: whole 112KB logs into own context; 20-call per-file grind). 14/14 spawns
|
|
|
|
|
|
explicit-model; the one delegating E1P rep flagged the stubbed downgrade via self-report.
|
|
|
|
|
|
E2P/E3P A-axis criterion redesign pre-registered; reserve stayed frozen. ORCHESTRATION.md
|
|
|
|
|
|
gained a scripted-bulk-edit carve-out. TSV: `eval/results/2026-07-06-baseline-grid.tsv`;
|
|
|
|
|
|
analysis: vault note [[os-orchestration-eval-baseline-grid-results]].
|
|
|
|
|
|
- **WS4 economics loop COMPLETE + SHIPPED (2026-07-08;** run-set AND reserve now
|
|
|
|
|
|
contaminated for future wording tuning): ORCHESTRATION.md v3 shipped — cost-asymmetry +
|
|
|
|
|
|
tier-conditional threshold opener, symmetric delegate-when/work-directly-when trigger
|
|
|
|
|
|
lists, batching rule (~5–8 items per grouped spawn, SendMessage reuse), async rule,
|
|
|
|
|
|
Workflow effort dial; explicit-model/self-report/don't-re-cover-ground rules unchanged.
|
|
|
|
|
|
Harness extended first (commit b226601): real token-econ axes — subagent transcripts
|
|
|
|
|
|
persist at `<projects>/<flat-cwd>/<session-id>/subagents/agent-*.jsonl` with per-message
|
|
|
|
|
|
usage blocks (vault note [[claude-code-subagent-transcripts-and-token-accounting]]) — E5
|
|
|
|
|
|
batching pair + reserve twins (subagent-authored, held out), fable column, pre-registered
|
|
|
|
|
|
E2P/E3P criterion redesign landed. Results: E5P 0/6 → majority-PASS both tiers incl.
|
|
|
|
|
|
reserve twin 5/5 (fable median mltok 43% of baseline); negatives clean everywhere
|
|
|
|
|
|
(36/36). Residual gaps: E1P sonnet delegate-at-all (fixture too small), fable
|
|
|
|
|
|
downgrade-flag ~1/4 judge flicker. 15 truncated baseline reps mechanically
|
|
|
|
|
|
excluded+replaced (limit-banner sweep promoted to the autoresearch howto). TSVs:
|
|
|
|
|
|
`eval/results/2026-07-08-*.tsv`; loop log: `autoresearch/classic-260708-1039/`; vault
|
|
|
|
|
|
note [[os-orchestration-ws4-econ-wording-results]].
|
|
|
|
|
|
- **First IRL re-audit DONE (2026-07-10, ahead of schedule):** 19 production transcripts
|
|
|
|
|
|
post-2026-07-08 audited by 3 parallel agents (run: `autoresearch/improve-260710-1057/`).
|
|
|
|
|
|
Solved surfaces: model-param discipline (~48/49 spawns pinned), over-delegation guardrail.
|
|
|
|
|
|
Tier 1 findings → same-day one-shot ORCHESTRATION.md production patch (grid validation
|
|
|
|
|
|
impossible — sets contaminated): settled-design mid-session fan-out trigger,
|
|
|
|
|
|
never-sleep-poll rule, own eval/benchmark loops named as mechanical sequences,
|
|
|
|
|
|
unfamiliar-tree ~5-call Explore trigger. Acceptance check = next IRL audit (~2–3 weeks;
|
|
|
|
|
|
baseline: settled-design drift 8/19 sessions). Side finding needing separate repro:
|
|
|
|
|
|
harness ignored explicit `model: "sonnet"` pins on general-purpose agents (env override
|
|
|
|
|
|
unset — distinct from WS1 cause); the self-report line caught it.
|
|
|
|
|
|
- **IRL audit semi-automated (2026-07-10):** recurring biweekly audit formalized, working
|
|
|
|
|
|
from any cwd. Deterministic layer: `audit/bin/audit-stats` (Ruby, composes
|
|
|
|
|
|
`audit/bin/extract` fact-sheets) enumerates production transcripts since the last run
|
|
|
|
|
|
(excludes sandboxes, `*-eval` dirs, active/trivial sessions), writes
|
|
|
|
|
|
`~/.local/state/os-context/audit/run-<date>/` (stats.tsv/flags.md/sheets) and appends the
|
|
|
|
|
|
trend ledger `metrics.tsv` (seeded 2026-07-10: 23 sessions, 52 spawns, 1 missing model).
|
|
|
|
|
|
Judgment layer: `/os-context:audit-sessions` (rubric in `references/`) — sonnet auditor
|
|
|
|
|
|
fan-out over flagged regions only, tiered synthesis, run-over-run trend verdicts, stops
|
|
|
|
|
|
before any wording edit. Nudge: os-status check `orchestration-audit-due` warns when the
|
|
|
|
|
|
ledger's last row is ≥14 days old (missing ledger = machine not opted in, silent; env
|
|
|
|
|
|
override `OS_ORCH_AUDIT_LEDGER` for tests). Planka recurrence-manifest entry skipped
|
|
|
|
|
|
(gem offsets are `+1y/+6m/+3m/+1m` only; a `+2w` offset is blocked on parked gem
|
|
|
|
|
|
publish/versioning decisions).
|
2026-07-13 17:28:07 +00:00
|
|
|
|
- **Audit-driven wording patch (2026-07-13):** IRL session audit (session fdee577b) found
|
|
|
|
|
|
the delegation-economics rules failed to fire when a main-loop fable model implemented a
|
|
|
|
|
|
fully-specified fix list itself via TDD (~110k tokens). Two approved additions to
|
|
|
|
|
|
`prompts/session-start/10-orchestration.md`: (1) names a red-green TDD loop over an
|
|
|
|
|
|
already-specified fix list as a single delegable unit (issue #29); (2) names
|
|
|
|
|
|
review/diagnosis-agent convergence on a confirmed defect list as a fresh delegation
|
|
|
|
|
|
decision, never a continuation of the main loop's review thread (issue #30). Applied
|
|
|
|
|
|
directly (design pre-approved, no wording-loop re-run); `bin/refresh-plugins` run after
|
|
|
|
|
|
the edit.
|