146 lines
9.5 KiB
Markdown
146 lines
9.5 KiB
Markdown
# System-prompt profiles: assessment & recommended approach
|
||
|
||
_Last updated: 2026-07-08 · Status: assessment + next-actions agreed; deferral lifted —
|
||
WS2/WS4 evals confirmed complete 2026-07-08 (Gate G0 cleared), execution may begin_
|
||
|
||
> Task breakdown for subagent implementation:
|
||
> [system-prompt-profiles-tasks.md](system-prompt-profiles-tasks.md) (2026-07-08).
|
||
|
||
## Agreed next actions (2026-07-08, deferred)
|
||
|
||
Two tracks, both approved by the user:
|
||
|
||
1. **Top-down diet** (first, biggest win):
|
||
- Split cc-os `CLAUDE.md` (~11k tokens) to a ~2–3k orientation + on-demand status/changelog
|
||
doc, using `@`-import / pointer style per
|
||
`~/servers/proxmox-hermes/docs/claude-md-maintenance.md` (keep it short; prefer pointers
|
||
to copies; never auto-append).
|
||
- Then per-profile plugin enablement (~4.8k max, second priority) and a memsearch
|
||
injection review.
|
||
- **Do not start while the other session's evals are running** — CLAUDE.md is live input
|
||
to those sessions.
|
||
2. **Bottom-up audit** (build anytime, read-only):
|
||
- One-file logging proxy (`ANTHROPIC_BASE_URL` pass-through) that dumps the `system` and
|
||
`tools` arrays from a real request — the audit basis for lean profiles. Expect savings
|
||
to come from per-profile `--tools` lists (~18k of schemas), not prompt rewriting (~6.3k).
|
||
|
||
Related future idea (out of scope, captured in vault note
|
||
[[claude-md-budget-linter-plugin-concept]]): a deterministic CLAUDE.md budget linter —
|
||
hook-driven, JSON per-project config with quantified user-agreed exceptions, silent in range,
|
||
maintenance agent only when out of range; possible extension to whole-project weekly hygiene
|
||
scans. Overlaps os-doc-hygiene/os-status — resolve at build time.
|
||
|
||
Companion to the WS4 orchestration-economics work. Goal: reduce per-session context cost and
|
||
improve focus by controlling what reaches the model's context — the built-in system prompt,
|
||
and (more importantly) the layers we add on top of it.
|
||
|
||
## What is mechanically possible today
|
||
|
||
Verified against `claude --help` (current install) and
|
||
https://code.claude.com/docs/en/agent-sdk/modifying-system-prompts.md (fetched 2026-07-08):
|
||
|
||
| Mechanism | What it does | Risk |
|
||
|---|---|---|
|
||
| `--append-system-prompt[-file]` | Adds instructions after the full default prompt | None — nothing removed |
|
||
| `--system-prompt[-file]` | **Fully replaces** the built-in prompt for the session | High — loses tool guidance, safety rules, env context; you must re-provide them |
|
||
| `--exclude-dynamic-system-prompt-sections` | Moves per-machine sections (cwd, git status, env, memory paths) into first user message → cross-session cache reuse | Low; env context carries slightly less weight |
|
||
| `--tools <list>` / `--disallowedTools` | Restricts the built-in toolset (removes tool schemas from context) | Low |
|
||
| `--settings` / `--setting-sources user,project,local` | Controls which settings layers (and therefore CLAUDE.md files, enabled plugins, hooks) load | Low |
|
||
| Output styles (`~/.claude/output-styles/*.md`, `keep-coding-instructions: true|false`) | Replace or extend the *coding-instruction section* of the default prompt; persistent, reusable across projects | Medium — partial replacement, officially supported |
|
||
| Plugin enable/disable per session (`--settings` overlay or profile-specific settings file) | Each enabled plugin's skills/agents/hooks descriptions are injected every session | Low — biggest immediate lever |
|
||
| Agent SDK (`systemPrompt: {type:'preset', append, excludeDynamicSections}` or custom string) | Same controls programmatically; SDK default is a *minimal* prompt unless the preset is requested | — |
|
||
|
||
Deferred-tool loading (ToolSearch) already ships in current CC — most tool schemas are no
|
||
longer paid up front. That reduced the value of "gut the system prompt" since much of the old
|
||
bloat was tool definitions.
|
||
|
||
## Measured baseline (2026-07-08, headless haiku `-p` runs, `--output-format json` usage)
|
||
|
||
| Configuration | Context tokens | Delta |
|
||
|---|---|---|
|
||
| `--setting-sources "" --tools ""` (system prompt text only) | **6,347** | — |
|
||
| `--setting-sources ""` (prompt + full built-in tool schemas) | **24,277** | +17.9k = tool schemas |
|
||
| Default settings, empty dir (user stack: 17 plugins, global CLAUDE.md, hooks) | **29,042** | +4.8k = user stack |
|
||
| Full stack, cc-os dir | **41,765** | +12.7k = project layer (CLAUDE.md ~11k dominates) |
|
||
|
||
Interpretation:
|
||
- **The system prompt *text* is only ~6.3k tokens.** The ~24k harness floor is dominated by
|
||
**tool schemas (~18k)** — so the bottom-up lever is `--tools` restriction per profile, not
|
||
prompt rewriting. Full `--system-prompt` replacement can save at most ~6k and carries the
|
||
fork-maintenance risk.
|
||
- The 17-plugin user stack costs ~4.8k, not tens of k — cheaper than estimated.
|
||
- The single largest single item we control is the cc-os CLAUDE.md (~11k).
|
||
- An interactive session that reads ~80k after one exchange is ~42k fixed overhead plus
|
||
~38k of *turn-1 work product* (web fetches, file reads) — conversation content, not
|
||
harness bloat. Caveat: these baselines are headless haiku; an interactive Fable session's
|
||
skill/agent listings and model-specific sections may differ somewhat — worth one `/context`
|
||
check interactively.
|
||
|
||
## Key finding: our bloat is mostly self-inflicted, not Anthropic's
|
||
|
||
Measured 2026-07-08 for a cc-os session:
|
||
|
||
- **cc-os `CLAUDE.md`: ~43 KB ≈ ~11k tokens**, injected into every session. Much of it is an
|
||
append-only implementation-status changelog, not orientation the session needs.
|
||
- **17 enabled plugins** — every skill description (~80 skills), agent type, and hook usage
|
||
note is injected into every session regardless of relevance (`creative-team`,
|
||
`invoice-ninja`, `rails-ui-component`, `api-wrapper`… in a cc-os design session).
|
||
- **memsearch SessionStart recall injection**: ~13 KB this session.
|
||
- Multiple SessionStart hooks (orchestration, os-vault, os-status) each add context.
|
||
|
||
The built-in system prompt is a static, cached prefix (cache reads are ~10% of input price
|
||
after the first turn within the 5-min TTL). The layers above are the same cost class *and*
|
||
they're the ones we control. The "bloat → hallucination" link is context rot: attention
|
||
degrades as context grows, whatever the source. Cutting 20k tokens of our own always-on
|
||
injection beats fighting Anthropic's prompt.
|
||
|
||
Pi reality check: pi.dev makes **no** speed/cost/hallucination claims on its site. Its
|
||
lean-ness comes from being a *different harness* (no MCP, no subagents, no permission
|
||
system, minimal prompt + extensions) — not from stripping Claude Code. Fully replacing CC's
|
||
system prompt while keeping CC's harness means maintaining a fork of prompt content the
|
||
model was tuned around, re-broken by every CC update.
|
||
|
||
## Recommended approach: subtract-first modular profiles
|
||
|
||
A `cyolo <flags>` shell wrapper that composes **existing** mechanisms per objective/project.
|
||
Profiles mostly *subtract* (plugins, tools, setting sources) and lightly *add*
|
||
(one append-file per objective). Full system-prompt replacement is a later, eval-gated
|
||
experiment on headless sessions only.
|
||
|
||
1. **Phase 0 — measure.** Run `/context` in 3–4 representative session types (cc-os design,
|
||
client dev, brainstorm) and record the actual breakdown. No optimizing before baselining.
|
||
2. **Phase 1 — diet (no new machinery).**
|
||
- Split cc-os CLAUDE.md: keep orientation (~2–3k tokens), move the implementation-status
|
||
changelog to a linked doc the AI reads on demand (progressive disclosure).
|
||
- Per-profile plugin enablement: a settings overlay per profile listing `enabledPlugins`;
|
||
a brainstorm profile doesn't load rails-ui-component/api-wrapper/invoice-ninja.
|
||
3. **Phase 2 — `cyolo` profile wrapper.**
|
||
- Central evergreen component prompts in one source dir (e.g. `~/.claude/profiles/`):
|
||
`dev.md`, `ruby.md`, `planning.md` (objective-focused: starting point → process →
|
||
definition of done), `brainstorm.md`.
|
||
- Wrapper resolves profile from flag (`--dev`, `--planning`) + project detection (pwd,
|
||
Gemfile, etc.), then launches
|
||
`claude --append-system-prompt-file <composed> --settings <profile-settings> [--tools …]`.
|
||
- Objective profiles state the session's endpoint explicitly ("this session ends when the
|
||
plan is written and shared; implementation belongs to a later session").
|
||
4. **Phase 3 — eval-gated replacement experiment (optional).** Arize-style loop, matching the
|
||
autoresearch playbook: scenario set × {default prompt, dieted stack, profile stack,
|
||
full-replacement lean prompt} × model tiers; score task success + tokens/turn +
|
||
wrong-tool/hallucination proxies from transcripts. Only promote full replacement if it
|
||
wins on outcome, not just token count. Read
|
||
`~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md` first, per standing rule.
|
||
|
||
Bottom-up track note: to audit the real shipped prompt, capture a live request body via a
|
||
local logging proxy (`ANTHROPIC_BASE_URL` pointed at a one-file pass-through that dumps the
|
||
`system` and `tools` arrays) — don't rely on the model to self-report its prompt. Given the
|
||
6.3k measurement, audit it to find what a lean profile can safely drop, but expect the
|
||
`--tools` list to be where the real savings are.
|
||
|
||
## Open questions
|
||
|
||
- Per-model profile variants: deferred (user agrees it's too granular now); the wording-loop
|
||
evals already show tier-dependent phrasing needs, so revisit after profiles exist.
|
||
- Whether `--setting-sources` interacts cleanly with symlinked local plugins + hook absolute
|
||
paths in `settings.json` (hooks are wired by absolute path and would still fire).
|
||
- Where profile settings overlays live so they don't fight `bin/refresh-plugins`.
|