vault: session notes 2026-07-08

claude-md-budget-linter-plugin-concept.md

@@ -0,0 +1,79 @@
+---
+type: plan
+title: "CLAUDE.md budget linter plugin — concept (deterministic hook + exception config + maintenance agent)"
+summary: "Future cc-os plugin concept for keeping CLAUDE.md (and eventually whole AI projects) within deterministic token/size budgets — a hook-driven linter that stays silent in range, escalates to a maintenance agent when out of range, and records user-agreed exceptions in a per-project config so allowed growth stays bounded."
+tags:
+  - type/plan
+  - tool/claude-code
+  - domain/context-engineering
+  - project/cc-os
+scope: global
+last_updated: 2026-07-08
+date: 2026-07-08
+related:
+  - "[[claude-code-system-prompt-customization-surface]]"
+---
+
+# CLAUDE.md budget linter plugin — concept
+
+Status: idea only, deliberately out of scope as of 2026-07-08. Revisit after the
+system-prompt-profiles work (`cc-os/docs/plans/system-prompt-profiles-assessment.md`).
+Source discussion: cc-os session 2026-07-08; maintenance philosophy drawn from
+`~/servers/proxmox-hermes/docs/claude-md-maintenance.md`.
+
+## Problem
+
+- CLAUDE.md degrades silently: bloat, stale facts, and instruction overload collapse rule
+  adherence (community guidance: ~100 lines of instruction; adherence degrades uniformly
+  past ~150–200 instructions). Measured example: cc-os CLAUDE.md reached ~11k tokens —
+  ~10x a sane budget — by accreting an implementation-status changelog.
+- Pruning is currently manual and memory-dependent ("review it monthly") — exactly the kind
+  of discipline that erodes. Nothing enforces a ceiling, and one "just this once" exception
+  becomes unbounded growth because the exception is never quantified.
+- The same failure mode applies project-wide: docs that outgrow progressive disclosure,
+  copies instead of pointers, files nobody re-reads.
+
+## Goal
+
+Deterministic, config-driven enforcement of context budgets — zero AI tokens on the happy
+path — with a human-in-the-loop escalation only when a budget is exceeded. Exceptions are
+explicit, quantified, and re-checked, so agreed growth stays bounded instead of becoming
+the new normal.
+
+## How it works (mid-level)
+
+1. **Hook → linter.** A SessionStart (or periodic) hook runs a deterministic linter over
+   CLAUDE.md. In range → no output, no action, 100% deterministic, no model call.
+2. **Per-project dot-config** (JSON for easy slurping, e.g. `.claude-md-lint.json` or a key
+   in an existing `.cc-os/config`): default budget (e.g. 1k tokens), cadence
+   (`check_every_days`), and an **exceptions block** — each exception records the agreed
+   limit, the specific reason, and who/when agreed (e.g. "allowed 1.2k because X, agreed
+   2026-07-08"). The linter checks against the *effective* limit, so an exception raises
+   the ceiling without removing it.
+3. **Out of range → maintenance agent.** Only then does an AI step run: it assesses the
+   file (stale sections, changelog accretion, copies-vs-pointers, `@`-import opportunities)
+   and proposes suggestions to the user.
+4. **User resolves**: accept the pruning suggestions, push back, or codify a new quantified
+   exception in the config. Never auto-edit CLAUDE.md — automated writes are the known
+   failure mode (noise accumulation, contradictory rules, safety-rule drift).
+5. **Cadence control**: config decides whether the hook checks every session or every N
+   days (stamp in a gitignored state dir, same pattern as os-doc-hygiene's `.dochygiene/`).
+
+## Possible extension: whole-project checks
+
+The same skeleton (deterministic scan → config budgets → agent escalation) generalizes to
+weekly project hygiene: file-size ceilings, progressive-disclosure usage (pointers vs
+inlined copies), stale-doc detection, dead `@`-imports.
+
+## Build-time considerations
+
+- **Overlap with os-doc-hygiene is significant** — it already does deterministic
+  SessionStart reminders, per-project gitignored state, AI-assisted classification, and
+  git-safe cleanup for stale/bloated docs. First design question at build time: is this a
+  new `os-*` plugin or a CLAUDE.md-specific check + budget-config feature *inside*
+  os-doc-hygiene (and/or an os-status check for "CLAUDE.md over budget")?
+- Read `cc-os-plugin-skill-naming-convention.md` before naming anything.
+- Token measurement should be deterministic and cheap (chars/4 heuristic or a local
+  tokenizer — pick one and freeze it; the budget is a contract, so the measure can't drift).
+- Eval approach exists: same deterministic-first checker + ladder playbook as the other
+  os-* plugins.

eval-methodology-ladder.md

@@ -9,7 +9,7 @@ tags:
   - convention/eval-design
   - tool/autoresearch
 scope: global
-last_updated: 2026-07-07  # added lesson-promotion pattern (eval-results → methodology notes, Loop Inputs handoff)
+last_updated: 2026-07-08  # econ-guard anti-pattern (WS4)
 date: 2026-07-06
 related:
   - running-autoresearch-skill-evals
@@ -95,6 +95,7 @@ artifact that lets a later reader diff proposal against what shipped.
 - **Forgetting that paired scenarios are *equally* subtle.** If your negative scenarios are obviously non-triggers (e.g., "add logging, no decision involved"), you're not testing discrimination. Negative scenarios should be as close to the positive case as possible without crossing the trigger threshold.
 - **Tuning against held-out reserve.** Once the reserve is defined, it becomes Schrodinger's eval — useful as a fresh measurement only if it stays frozen. If you peek at reserve results and change wording based on them, it's now training-set.
 - **A checker that contradicts the shipped instructions.** Before spending reps, dry-run the instrument on paper: "would a model that follows the shipped wording perfectly pass every cell?" and "would an always-trigger model pass the positives while failing the negatives?" If yes to the second or no to the first, the eval measures instruction-*disobedience*. Concrete instance (os-adr Eval C, 2026-07-06): positives sat in already-decided ADR territory, where the shipped wording says find→cite→comply — but the checker demanded ADR creation, so a perfectly-behaving model FAILed and an ADR-spamming model would have aced positives. Caught on the first live rep only because of a canary-cell run.
+- **Econ guards phrased as ratios of a grind baseline.** When a positive scenario's baseline behavior is an *efficient but wrong-shaped* strategy (e.g. a cheap main-loop grind), a "candidate must stay ≤N% of baseline cost" guard punishes the correct behavior the checker itself demands — delegation legitimately costs more than grinding on a cheap tier. Set cheap-tier econ guards in absolute tokens, or condition them on whether the task is scriptable; reduction targets belong only on the expensive-tier cells whose baseline IS the miss (os-orchestration WS4, 2026-07-08: sonnet guard breached at 168% for identical task completion, accepted with rationale).
 - **Trusting model-free self-tests to validate checker *semantics*.** Fabricated "perfect" transcripts encode the same assumptions as the checker — if the designer's idea of correct behavior is wrong, both agree and the self-test stays green. Self-tests catch mechanical bugs, not conformance bugs. Fabricate at least one transcript of *shipped-instruction-compliant* behavior (what the model is actually told to do, given the fixture state), not just the designer's imagined ideal.
 
 ## Known Limitations

howto/running-autoresearch-skill-evals.md

@@ -9,7 +9,7 @@ tags:
   - tool/autoresearch
   - project/cc-os
 scope: global
-last_updated: 2026-07-06
+last_updated: 2026-07-08
 date: 2026-07-04
 related:
   - os-adr-eval-b-grid-results-and-observations
@@ -98,6 +98,7 @@ salience in the hook note / find-skill description). One "failure" label, two di
 - **Degradation-check cells (e.g. R4-nograph) are only meaningful paired with a PASS on their non-degraded twin at the same tier** — and at 1 rep they can pass "unexpectedly" (sonnet found the right ADR without the graph on 2026-07-04), which weakens the layer-value evidence rather than proving anything. Don't cite degradation cells as proof at 1 rep.
 - **Held-out scenarios stay held out** — never run scenario Task blocks informally/by hand; that contaminates the measurement.
 - **Broken cells look like model failures** — a missing `transcript.jsonl` scores FAIL on both axes. Check reasons strings for harness errors before counting a cell as a behavioral result.
+- **Account session limits truncate headless reps into plausible verdicts** — a `claude -p` rep cut by a usage/session limit still yields a scoreable transcript: positives FAIL with real-looking axes (task half done) and *negatives can wrongly PASS* (session died before the forbidden action). Sweep EVERY rep's final assistant message for the limit banner ("You've hit your session limit") before counting; excluded reps are infra errors, mechanically identified independent of verdict, so replacing them with fresh reps is not peeking (WS4, 2026-07-08: 15/40 grid reps excluded this way).
 - **Model-tier gaps can be total** — haiku 0/8 on Eval B means wording iteration may not reach the lower tier at all; keep one lower-tier canary cell (haiku W2, the one axis-a flicker) in the loop to detect whether wording changes reach it.
 - **`set -euo pipefail` + `checker | tee` silently truncates rep loops** — a checker that exits 1 on FAIL, piped into `tee` inside a runner with pipefail, aborts the remaining reps of that cell on the first FAIL, and the outer shell may still report exit 0. Symptom: `--reps 3` produces one TSV row. Guard the pipeline (`|| true`) and check row counts against expected reps before reading results (Eval C, 2026-07-06).
 - **When the instrument was wrong, rescore — don't re-run or discard** — if a checker fix changes scoring but the transcript is valid evidence of behavior, re-run `bin/check` on the existing sandbox and replace the row. Re-running spends a rep on a new behavior sample (different question); discarding is peeking bias.

os-orchestration-ws4-econ-wording-results.md

@@ -0,0 +1,129 @@
+---
+type: eval-results
+title: "os-orchestration WS4 economics wording loop (ORCHESTRATION.md v3, fable+sonnet)"
+summary: "The v3 cost-asymmetry/symmetric-trigger/batching wording flips E5P implementation fan-out from 0/6 to majority-PASS on both tiers and cuts fable main-loop output tokens to 43% of baseline, with all negatives clean on run-set AND held-out reserve — shipped after one iteration + confirm + reserve; residual gaps are E1P sonnet non-delegation (fixture-scale) and a ~1/4 fable downgrade-flag judge flicker."
+tags:
+  - type/eval-results
+  - domain/llm-evaluation
+  - tool/os-orchestration
+  - project/cc-os
+scope: global
+last_updated: 2026-07-08
+date: 2026-07-08
+related:
+  - os-orchestration-eval-baseline-grid-results
+  - orchestration-prompting-claude-5-era
+  - eval-methodology-ladder
+  - running-autoresearch-skill-evals
+  - claude-code-subagent-transcripts-and-token-accounting
+source: cc-os
+---
+
+# os-orchestration WS4 economics wording loop
+
+## Results Grid/Threshold
+
+**Passing criterion:** per-cell ≥2/3 (sonnet) / 2/2 (fable) reps PASS on the frozen
+deterministic checker; H3 econ thresholds pre-registered in
+`docs/plans/ws4-orchestration-economics.md` (E5P fable median mltok ≤37,783 = 60% of
+baseline; E3P non-regression guard; negatives must stay clean).
+
+Pre-wording baseline (2026-07-08, 44 counted reps) → candidate-1 (iter-1 15 reps + confirm
+40 reps) → frozen reserve (40 reps). Zero harness errors; every rep swept for
+session-limit truncation (15 baseline reps excluded+replaced by that mechanical criterion).
+
+| Cell | baseline | confirm (candidate) | reserve twin |
+|---|---|---|---|
+| E5P fable | 0/3 (grind, median mltok 62,971) | 2/2 (median across 5 candidate reps 27,372 = 43%) | 2/2 |
+| E5P sonnet | 0/3 | 2/3 (+3/3 iter-1) | 3/3 |
+| E1P fable | 1/2 | 1/2 (B judge flicker; 3/4 across iter+confirm) | 1/2 (A miss) |
+| E1P sonnet | 0/3 | 0/3 | 0/3 |
+| E2P / E3P (redesigned criteria) | 6/6 / 6/6 | 5/5 / 5/5 | 5/5 / 5/5 |
+| all negatives (E1N/E2N/E3N/E5N) | clean | 16/16 | 20/20 |
+
+**Summary:** H1 (symmetric cost-framed triggers) and H2 (batching: ≤3 grouped spawns, one
+round, explicit models — reserve fable rep used 2 explicit-haiku spawns) confirmed; H3 met
+on the pre-registered E5P-fable axis (43% of baseline mltok). E1P sonnet never delegates at
+any point (fixture too small to force it — pre-registered limitation); zero over-delegation
+anywhere, including E5N sequential tasks under the new batching rule.
+
+## Measurement Setup
+
+**Hypotheses / Scenarios Tested** — H1/H2/H3 pre-registered in the WS4 plan; E5
+(batching/implementation fan-out, mirrors the 2026-07-08 mini-audit Write×13 exemplar) added
+with reserve twins authored by a subagent so held-out content never entered the tuning
+session's context. E2P/E3P criteria redesigned per the 2026-07-06 pre-registration (scripted-
+direct PASS branch; 74KB S7 ingestion anchor + mechanical root-cause concepts).
+
+**Fixture and Sampling** — relaystation Node.js fixture + new handler-spec (12 pairwise-
+distinct handler modules, deliberately non-templatable). sonnet ×3 / fable ×2 reps per cell;
+fable at reduced reps for cost per plan.
+
+**Experimental Control (Frozen Surfaces)** — checker/extractor/runner/fixtures/scenarios/
+rubric frozen during the loop; ORCHESTRATION.md wording the only tuning surface;
+`bin/refresh-plugins` + cache==source diff before every grid. One instrument fix landed
+between confirm and reserve (E3P concepts keyed by scenario id — the reserve twin targets
+gen-logs' other planted incident; fixed without reading reserve files, run-set scoring
+byte-identical, self-test 34/34).
+
+## Validity and Limitations
+
+**How to Interpret These Results** — run-set AND reserve are now both contaminated for
+future wording tuning (reserve was the final measurement). The reserve pass is the
+generalization claim: same axes, different surface domain, never read informally. Econ
+figures are real token sums (subagent transcripts persist under
+`<session-id>/subagents/agent-*.jsonl` — see [[claude-code-subagent-transcripts-and-token-accounting]]),
+not the char-count proxy the mini-audit used.
+
+**Weaknesses of This Eval** — headless `-p` population may differ from interactive sessions
+(unresolved since the 2026-07-06 baseline). E1P fixture too small to force delegation on
+sonnet; its cell says nothing about capability. H3(c) sonnet econ guard (≤150% of grind
+baseline) was breached at median 168%: delegation carries a real ~+6.6k-token premium on the
+cheap tier for identical task completion — accepted because the frozen checker requires
+delegate-or-script on a non-scriptable task and fable is the population of concern. Single
+fixture. E5P mltok variance is high on fable (15k–65k baseline; 22k–47k candidate).
+
+## Transferable Lessons
+
+- Econ guards on cheap tiers should be absolute-token or scriptability-conditioned, not
+  ratios of a grind baseline — a ratio guard punishes correct delegation when the baseline
+  behavior is an efficient grind. (Promoted to [[eval-methodology-ladder]].)
+- Account session limits truncate headless reps into plausible-looking behavior FAILs (and
+  PASSes); sweep every rep's final assistant message for the limit banner before counting.
+  (Promoted to [[running-autoresearch-skill-evals]].)
+- Reserve twins authored before a criterion redesign can be silently unscoreable by the new
+  checker; audit checker constants against every planted fixture incident (not just run-set)
+  before the reserve run — fixable by id-keyed criteria without reading reserve files.
+- Delegating reserve-twin authoring to a subagent (report = filenames only) keeps held-out
+  content out of the tuning session's context — cheap insurance when the same session will
+  run the wording loop.
+
+**Promotion check:** promoted lesson 1 to [[eval-methodology-ladder]] and lesson 2 to
+[[running-autoresearch-skill-evals]] (both updated 2026-07-08); lessons 3–4 not yet
+recurring.
+
+## Deployment and Evolution
+
+**Good-Enough Gate** — shipped 2026-07-08 (ORCHESTRATION.md v3 in cc-os + refreshed cache):
+negatives clean on both grids, batching generalizes to the reserve twin on both tiers, H3
+met on the pre-registered axis. Residual known gaps at ship time: E1P sonnet delegate-at-all
+(fixture-scale, not wording), fable downgrade-flag ~1/4 judge-boundary flicker (os-adr W3
+shape — characterize in production, don't lab-chase).
+
+**Proposed Changes (Loop Inputs)** — none; candidate-1 shipped verbatim (five changes:
+cost-asymmetry + tier-conditional threshold opener, symmetric delegate/work-directly trigger
+lists, batching rule ~5–8 items/spawn + SendMessage reuse, async rule, Workflow effort
+dial). Diff against v2 is commit-visible in cc-os.
+
+**Hardening Path / Next Measurement** — production IRL re-audit of ~5 real Fable sessions
+scheduled ~2026-07-22 per [[eval-methodology-irl-feedback-loop]]: compare main-loop token
+share vs the 2026-07-08 mini-audit baseline (now measurable exactly via subagent
+transcripts); promote new miss shapes into eval scenarios. E1P needs a larger fixture before
+its sonnet cell is meaningful.
+
+## Related
+
+- [[os-orchestration-eval-baseline-grid-results]] — the 2026-07-06 E1–E3 baseline this loop builds on
+- [[orchestration-prompting-claude-5-era]] — the model-page guidance the v3 wording encodes
+- [[eval-methodology-ladder]] / [[running-autoresearch-skill-evals]] — discipline followed + lessons promoted
+- [[claude-code-subagent-transcripts-and-token-accounting]] — the instrumentation discovery behind the econ axes