127 lines
9.6 KiB
Markdown
127 lines
9.6 KiB
Markdown
# os-context (formerly os-orchestration) — component detail
|
||
|
||
_Leaf file of [../implementation-status.md](../implementation-status.md). Read on demand._
|
||
|
||
**Global os-context plugin** — `cc-os/plugins/os-context/` (git-tracked); symlinked into
|
||
`~/.claude/plugins/os-context`. Migrated 2026-07-03 from a standalone repo
|
||
(`~/dev/cc-plugins/orchestration/`), superseding per-project orchestration text (ADR-0019);
|
||
renamed and generalized from os-orchestration 2026-07-13 (ADR-0031). Entries below predating
|
||
the rename refer to the plugin by its former name.
|
||
|
||
## Component inventory
|
||
|
||
- Hooks: `hooks/inject.py` — SessionStart prompt composer: globs
|
||
`prompts/session-start/*.md`, sorts by filename, concatenates into one additionalContext
|
||
block for all sessions, with a deterministic line-count budget (warn >120 lines, refuse
|
||
>240 lines). Byte-identical output to the old single-file `ORCHESTRATION.md` behavior when
|
||
only `10-orchestration.md` is present (tested).
|
||
- Prompt files (2026-07-13): `10-orchestration.md` (the canonical session-orchestration +
|
||
delegation-economics rules; formerly `ORCHESTRATION.md`), `20-kickoff-conventions.md`,
|
||
`30-standing-safety.md`, `40-decision-memos.md` (43 lines; total injection 102/120).
|
||
Decision-memo auto-memory file deleted per single-source rule.
|
||
- Skill: `/os-context:audit-sessions` (formerly `/os-orchestration:audit-sessions`).
|
||
- State: `~/.local/state/os-context/` (old `os-orchestration` path symlinked;
|
||
`OS_ORCH_AUDIT_LEDGER` env var name kept). os-status's `orchestration-audit-due` check
|
||
updated to match.
|
||
|
||
## Session audits and wording iterations
|
||
|
||
- **WS1 session audit (2026-07-06):** 10 stratified real sessions audited via
|
||
`audit/bin/extract` (Ruby fact-sheet extractor, dual-use as future eval checker) +
|
||
parallel sonnet auditors + verified synthesis. Findings + E1–E4 eval-scenario backlog:
|
||
`docs/orchestration-audit/2026-07-06-findings.md`. Headlines: model param silently
|
||
overridden to haiku on all 23 verified spawns — root cause
|
||
`CLAUDE_CODE_SUBAGENT_MODEL=haiku` in `~/.claude/settings.json` env block (removed
|
||
2026-07-06); omitted model param inherits the main-loop model; no over-sharing/full-dump
|
||
failures anywhere.
|
||
- **Wording fixes shipped (2026-07-06, post-audit):** ORCHESTRATION.md rewritten with
|
||
trigger-conditioned when→then phrasing for the three verified clusters — explicit
|
||
`model:` before every Agent call, subagent model self-report on sonnet/opus spawns (the E1
|
||
canary proved `resolvedModel` is NOT model-visible in the launch result; self-report is
|
||
the only observable channel), don't re-cover your own ground before delegating.
|
||
Delegation thresholds unchanged. Deterministic env-override detection delegated to
|
||
os-status (WS3).
|
||
- **Eval harness E1–E3 (2026-07-06):** `plugins/os-context/eval/` — scripted headless
|
||
orchestration-behavior eval scored by the audit extractor (extended with per-segment read
|
||
targets). 6 run-set scenarios (E1P/N downgrade detection with runner stub; E2P/N
|
||
explicit-model fan-out vs no-delegation-on-trivial; E3P/N delegate-investigation vs
|
||
orienting-reads-correct) + 6 frozen reserve twins; Node.js "relaystation" fixture with
|
||
deterministic sandbox-time log generation; deterministic-first checker (`eval/bin/check`,
|
||
narrow frozen haiku-judge fallback for E1 language axes only, stubbable via
|
||
`ORCH_EVAL_JUDGE_CMD`); model-free `bin/self-test` 21/21. **Scenario Task blocks are
|
||
held-out — never run informally.** Procedure: `plugins/os-context/eval/README.md`.
|
||
- **Baseline grid (2026-07-06,** run-set × sonnet/haiku × 3 reps + 2 counted canaries):
|
||
negatives 18/18 PASS both tiers (zero over-delegation, zero false downgrade claims);
|
||
positives 1/18 on delegate-at-all — but transcript verification shows most positive FAILs
|
||
are *superior direct strategies* (scripted bulk edit; surgical greps over 400KB logs),
|
||
which the WS1 audit itself scored as justified non-delegation, vs genuine miss shapes
|
||
(haiku: whole 112KB logs into own context; 20-call per-file grind). 14/14 spawns
|
||
explicit-model; the one delegating E1P rep flagged the stubbed downgrade via self-report.
|
||
E2P/E3P A-axis criterion redesign pre-registered; reserve stayed frozen. ORCHESTRATION.md
|
||
gained a scripted-bulk-edit carve-out. TSV: `eval/results/2026-07-06-baseline-grid.tsv`;
|
||
analysis: vault note [[os-orchestration-eval-baseline-grid-results]].
|
||
- **WS4 economics loop COMPLETE + SHIPPED (2026-07-08;** run-set AND reserve now
|
||
contaminated for future wording tuning): ORCHESTRATION.md v3 shipped — cost-asymmetry +
|
||
tier-conditional threshold opener, symmetric delegate-when/work-directly-when trigger
|
||
lists, batching rule (~5–8 items per grouped spawn, SendMessage reuse), async rule,
|
||
Workflow effort dial; explicit-model/self-report/don't-re-cover-ground rules unchanged.
|
||
Harness extended first (commit b226601): real token-econ axes — subagent transcripts
|
||
persist at `<projects>/<flat-cwd>/<session-id>/subagents/agent-*.jsonl` with per-message
|
||
usage blocks (vault note [[claude-code-subagent-transcripts-and-token-accounting]]) — E5
|
||
batching pair + reserve twins (subagent-authored, held out), fable column, pre-registered
|
||
E2P/E3P criterion redesign landed. Results: E5P 0/6 → majority-PASS both tiers incl.
|
||
reserve twin 5/5 (fable median mltok 43% of baseline); negatives clean everywhere
|
||
(36/36). Residual gaps: E1P sonnet delegate-at-all (fixture too small), fable
|
||
downgrade-flag ~1/4 judge flicker. 15 truncated baseline reps mechanically
|
||
excluded+replaced (limit-banner sweep promoted to the autoresearch howto). TSVs:
|
||
`eval/results/2026-07-08-*.tsv`; loop log: `autoresearch/classic-260708-1039/`; vault
|
||
note [[os-orchestration-ws4-econ-wording-results]].
|
||
- **First IRL re-audit DONE (2026-07-10, ahead of schedule):** 19 production transcripts
|
||
post-2026-07-08 audited by 3 parallel agents (run: `autoresearch/improve-260710-1057/`).
|
||
Solved surfaces: model-param discipline (~48/49 spawns pinned), over-delegation guardrail.
|
||
Tier 1 findings → same-day one-shot ORCHESTRATION.md production patch (grid validation
|
||
impossible — sets contaminated): settled-design mid-session fan-out trigger,
|
||
never-sleep-poll rule, own eval/benchmark loops named as mechanical sequences,
|
||
unfamiliar-tree ~5-call Explore trigger. Acceptance check = next IRL audit (~2–3 weeks;
|
||
baseline: settled-design drift 8/19 sessions). Side finding needing separate repro:
|
||
harness ignored explicit `model: "sonnet"` pins on general-purpose agents (env override
|
||
unset — distinct from WS1 cause); the self-report line caught it.
|
||
- **IRL audit semi-automated (2026-07-10):** recurring biweekly audit formalized, working
|
||
from any cwd. Deterministic layer: `audit/bin/audit-stats` (Ruby, composes
|
||
`audit/bin/extract` fact-sheets) enumerates production transcripts since the last run
|
||
(excludes sandboxes, `*-eval` dirs, active/trivial sessions), writes
|
||
`~/.local/state/os-context/audit/run-<date>/` (stats.tsv/flags.md/sheets) and appends the
|
||
trend ledger `metrics.tsv` (seeded 2026-07-10: 23 sessions, 52 spawns, 1 missing model).
|
||
Judgment layer: `/os-context:audit-sessions` (rubric in `references/`) — sonnet auditor
|
||
fan-out over flagged regions only, tiered synthesis, run-over-run trend verdicts, stops
|
||
before any wording edit. Nudge: os-status check `orchestration-audit-due` warns when the
|
||
ledger's last row is ≥14 days old (missing ledger = machine not opted in, silent; env
|
||
override `OS_ORCH_AUDIT_LEDGER` for tests). Planka recurrence-manifest entry skipped
|
||
(gem offsets are `+1y/+6m/+3m/+1m` only; a `+2w` offset is blocked on parked gem
|
||
publish/versioning decisions).
|
||
- **Audit-driven wording patch (2026-07-13):** IRL session audit (session fdee577b) found
|
||
the delegation-economics rules failed to fire when a main-loop fable model implemented a
|
||
fully-specified fix list itself via TDD (~110k tokens). Two approved additions to
|
||
`prompts/session-start/10-orchestration.md`: (1) names a red-green TDD loop over an
|
||
already-specified fix list as a single delegable unit (issue #29); (2) names
|
||
review/diagnosis-agent convergence on a confirmed defect list as a fresh delegation
|
||
decision, never a continuation of the main loop's review thread (issue #30). Applied
|
||
directly (design pre-approved, no wording-loop re-run); `bin/refresh-plugins` run after
|
||
the edit.
|
||
- **Absolute delegation policy (2026-07-17, ADR-0043):** IRL transcript review showed the
|
||
per-task delegation threshold fails by induction — chain length is unknowable at decision
|
||
time, each "short chain" call is locally defensible, and sessions accrete 100k+ tokens of
|
||
main-loop tool results. `10-orchestration.md` rewritten (design pre-approved in session):
|
||
executive frame, enumerated E-a..E-d exemptions (no evaluative carve-outs), one persistent
|
||
sonnet manager continued via SendMessage, opus escalation on observed failure only.
|
||
Audit rubric rewritten to match, shaped by a sonnet perspectives fan-out (measurement +
|
||
devils-advocate): exemption decision procedure with recorded steps per finding, new
|
||
`exemption-chaining` category, `over-delegation` narrowed to `delegation-overhead-mismatch`
|
||
(not deleted), old defenses demoted to LOW/`accepted-cost` severity modifiers,
|
||
`missed-delegation/read-write` sub-tag. `audit/bin/extract` gains `read_write_profile`
|
||
(total main-loop Read/Write/Edit incl. non-consecutive — largest prior session: 67 total,
|
||
only 14 visible to the run detector) and drops its stale pro-direct-work trailer. Eval
|
||
E1–E3/E5 retired as a measurement instrument (README banner): sets contaminated AND
|
||
scenarios encode the superseded policy; kept as harness template. Validation of the new
|
||
wording comes only from future IRL audits.
|