Compare commits
9 Commits
c2098525fe
...
33ae61a817
| Author | SHA1 | Date |
|---|---|---|
jared
|
33ae61a817 | |
|
jared
|
a06431c5fd | |
|
jared
|
397145b27b | |
|
jared
|
ab258d5ba1 | |
|
jared
|
a771163663 | |
|
jared
|
b226601bea | |
|
jared
|
42860d06d1 | |
|
jared
|
638c6d03ed | |
|
jared
|
1d1e7207c0 |
|
|
@ -19,3 +19,5 @@ __pycache__/
|
|||
|
||||
# autoresearch skill run logs (eval iterations)
|
||||
/autoresearch/
|
||||
# os-status plugin state
|
||||
.cc-os/
|
||||
|
|
|
|||
287
CLAUDE.md
287
CLAUDE.md
|
|
@ -1,218 +1,129 @@
|
|||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this
|
||||
repository. It is orientation only — implementation history, component details, and
|
||||
operational procedures live in [docs/implementation-status.md](docs/implementation-status.md)
|
||||
(read it on demand, e.g. before touching a plugin or claiming something isn't built yet).
|
||||
|
||||
## What this repository is
|
||||
|
||||
`cc-os` is a **design + implementation repository** — it captures the design of a **personal,
|
||||
cross-project memory system for Claude Code** (for a multi-client freelancer) plus the research
|
||||
that informs it. The global memory plugin is now partially implemented; markdown specs, ADRs,
|
||||
and the build plan remain the source of truth for what is being built and what remains.
|
||||
|
||||
Everything is markdown-as-truth. When asked to "build," work from the staged tasks in
|
||||
`docs/memory-system/04-build-plan.md`, not ad hoc.
|
||||
`cc-os` is a **design + implementation repository** for a **personal, cross-project memory
|
||||
system for Claude Code** (for a multi-client freelancer), plus a family of global Claude Code
|
||||
plugins and the research/evals that inform them. Everything is markdown-as-truth: specs,
|
||||
ADRs, and the build plan in `docs/` are the source of truth for what is being built. When
|
||||
asked to "build," work from the staged tasks in `docs/memory-system/04-build-plan.md`, not ad
|
||||
hoc.
|
||||
|
||||
## Directory layout
|
||||
|
||||
Numbered files within a directory are not a required reading order — read the one whose topic
|
||||
you need.
|
||||
Numbered files within a directory are not a required reading order — read the one whose
|
||||
topic you need.
|
||||
|
||||
- **`docs/memory-system/`** — the design of the memory system itself. Go here to understand or
|
||||
change *what is being built*. `02-system-design.md` is the architecture, `03-architecture-
|
||||
decisions.md` is the ADR log (each decision + what was rejected/deferred and why),
|
||||
`04-build-plan.md` is the build outline, `06-graphify-evaluation.md` justifies the Graphify
|
||||
pivot. Read the specific file relevant to your task; read the whole set only when reworking
|
||||
the design.
|
||||
- **`docs/graphify/`** — a verified handbook for the Graphify knowledge-graph tool (the chosen
|
||||
knowledge-layer engine). Go here when working with Graphify commands/behavior. Skim
|
||||
`00-README.md` for the model, keep `09-best-practices-checklist.md` open while actually
|
||||
running it. Claims are provenance-tagged (`[github]` trustworthy; `[interview]` /
|
||||
`[unverified claim]` not).
|
||||
- **`docs/memory-system/`** — the design of the memory system itself. `02-system-design.md`
|
||||
is the architecture, `03-architecture-decisions.md` is the ADR log, `04-build-plan.md` the
|
||||
build outline with step status, `06-graphify-evaluation.md` justifies the Graphify pivot.
|
||||
- **`docs/implementation-status.md`** — the status timeline, full per-component detail for
|
||||
every shipped plugin (os-vault, os-orchestration, os-status, os-doc-hygiene, os-adr),
|
||||
eval-harness records, and plugin rename/cache-refresh procedures.
|
||||
- **`docs/graphify/`** — verified handbook for the Graphify knowledge-graph tool. Skim
|
||||
`00-README.md` for the model; keep `09-best-practices-checklist.md` open when running it.
|
||||
Claims are provenance-tagged (`[github]` trustworthy; `[interview]` / `[unverified claim]`
|
||||
not).
|
||||
- **`graphify-interview`, `memory-systems-compared060326`** (repo root) — raw source
|
||||
transcripts (marketing / video). Only open these to trace where a claim came from; treat as
|
||||
*intent*, not *fact* — they were already corrected against primary sources in `docs/`.
|
||||
- **`openspec/`** — spec-driven change management (see workflow below). `changes/` holds live
|
||||
changes, `changes/archive/` completed ones, `specs/` stable specs.
|
||||
- **`.claude/`, `.codex/`, `.pi/`** — **identical** copies of the OpenSpec skills for three AI
|
||||
assistants. Only open when changing a skill — and mirror any change across all three.
|
||||
transcripts. Only open to trace a claim's origin; treat as *intent*, not *fact*.
|
||||
- **`plugins/`** — source of the global plugins, symlinked into `~/.claude/plugins/`.
|
||||
- **`openspec/`** — spec-driven change management (see workflow below).
|
||||
- **`.claude/`, `.codex/`, `.pi/`** — **identical** copies of the OpenSpec skills for three
|
||||
AI assistants. Only open when changing a skill — and mirror any change across all three.
|
||||
|
||||
## The current design in one paragraph
|
||||
|
||||
This is a **work in progress**, not a frozen spec. The paragraph below is the current approach;
|
||||
treat it as the default you operate from, but **whenever a conversation with the user changes
|
||||
the design, update this paragraph (and the relevant `docs/memory-system/` files + an ADR) to
|
||||
match.** Keep it accurate, don't preserve it for its own sake.
|
||||
This is a work in progress, not a frozen spec; **when a conversation changes the design,
|
||||
update this paragraph (and the relevant `docs/memory-system/` files + an ADR) to match.**
|
||||
|
||||
Two memory types kept as **separate systems**: **episodic** ("what happened, when") handled by
|
||||
**memsearch** (Milvus Lite, embedded), and **semantic/knowledge** ("how do we…") handled by the
|
||||
existing **`~/Documents/SecondBrain` Obsidian vault** as the single source of truth. Notes keep
|
||||
`summary` + six flat, parallel namespaced facets (`type/`/`client/`/`project/`/`domain/`/`tool/`/`convention/`) plus `scope/` as metadata; hierarchy and relationships are expressed via hub notes (`type/hub`), wikilinks, and Graphify graph edges — not nested tag paths. The vault is queried via a **Graphify knowledge graph** (local Ollama SLM for doc extraction, free
|
||||
tree-sitter AST for code). Retrieval is hook-injected + on-demand so project repos stay thin;
|
||||
freshness is lazy
|
||||
(write-time hook + SessionStart reconcile, no daemon/cron); the vault syncs to a VPS while
|
||||
indexes stay disposable and rebuildable. Ships as a global Claude Code plugin (`os-vault`) with skills.
|
||||
Two memory types kept as **separate systems**: **episodic** ("what happened, when") handled
|
||||
by **memsearch** (Milvus Lite, embedded, single global store for ALL clients by design —
|
||||
ADR-015), and **semantic/knowledge** ("how do we…") handled by the existing
|
||||
**`~/Documents/SecondBrain` Obsidian vault** as the single source of truth (ADR-012). Notes
|
||||
keep `summary` + six flat namespaced facets
|
||||
(`type/`/`client/`/`project/`/`domain/`/`tool/`/`convention/`) plus `scope/` (ADR-011);
|
||||
hierarchy comes from hub notes, wikilinks, and Graphify graph edges — not nested tags
|
||||
(Graphify is a structure extractor, not a topic clusterer; hubs are author-provided,
|
||||
ADR-014). The vault is queried via a **Graphify knowledge graph** (local Ollama SLM for doc
|
||||
extraction, tree-sitter AST for code); Graphify replaced the originally-planned Ruby/SQLite
|
||||
tag-index CLI (2026-06-03 pivot). Retrieval is hook-injected + on-demand; freshness is lazy
|
||||
(write-time hook + SessionStart reconcile, no daemon); the vault and memsearch store
|
||||
auto-sync to private Forgejo repos via SessionEnd hooks. Ships as a global Claude Code
|
||||
plugin (`os-vault`) with skills. Projects are onboarded one at a time; bulk vault migration
|
||||
is deferred to last (ADR-013).
|
||||
|
||||
**Recent pivot (2026-06-03):** Graphify **replaces** the originally-planned Ruby/SQLite
|
||||
tag-index CLI and also covers the deferred QMD semantic layer. `04-build-plan.md` and
|
||||
`06-graphify-evaluation.md` reflect this; if an older doc still describes the Ruby CLI, defer
|
||||
to those two and fix the stale doc.
|
||||
## Implemented components (inventory)
|
||||
|
||||
**Decisions locked (2026-06-04):** Six-facet tag taxonomy + `scope/` (ADR-011); reuse `~/Documents/SecondBrain` vault rather than creating a new one (ADR-012); build-first / migrate-incrementally — build full system against a fixture set first, defer bulk vault migration to last, onboard projects one at a time (ADR-013).
|
||||
All build-plan steps required for the memory system are complete (2026-06-15). Full detail
|
||||
for each item is in [docs/implementation-status.md](docs/implementation-status.md) — read it
|
||||
before modifying any of these:
|
||||
|
||||
**Empirical finding locked (2026-06-05):** Graphify is a structure extractor, not a topic clusterer — no emergent hub nodes appear even at `--mode deep`; hub notes + wikilinks must be author-provided during migration (not deferred). Migration scaffolding is now a first-class deliverable. Open question: do facet tags create graph edges? (ADR-014; findings: `docs/memory-system/07-graph-connectivity-findings.md`).
|
||||
- **os-vault** (`plugins/os-vault/`) — vault write/query/reorganize/onboard-project/
|
||||
design-template skills, SessionStart/End hooks, memsearch + vault git sync. Write-behavior
|
||||
eval harness in `plugins/os-vault/eval/`.
|
||||
- **os-orchestration** (`plugins/os-orchestration/`) — injects `ORCHESTRATION.md`
|
||||
(session-orchestration + delegation-economics rules) into all sessions; this repo carries
|
||||
no local override. Eval harness in `plugins/os-orchestration/eval/`.
|
||||
- **os-status** (`plugins/os-status/`) — aggregated deterministic SessionStart checks
|
||||
(subagent-model env override, ADR system present, vault hub note present). ADR-022.
|
||||
- **os-doc-hygiene** (`plugins/os-doc-hygiene/`) — stale/bloated-doc monitoring;
|
||||
check/clean/status/sweep skills.
|
||||
- **os-adr** (`plugins/os-adr/`) — ADR system: Ruby `lib/adr/` + CLIs, create/init/migrate/
|
||||
find skills; Eval A/B/C harnesses. Rollout order locked: pilot projects, then the cc-os
|
||||
retrofit of `03-architecture-decisions.md`, then wider.
|
||||
- **Graphify** v0.8.31 (`~/.local/bin/graphify`; PyPI package is `graphifyy`, double-y) —
|
||||
vault graph at `~/Documents/SecondBrain/graphify-out/`, per-project graphs at
|
||||
`<project-root>/graphify-out/` (gitignored); both disposable/rebuildable.
|
||||
- **memsearch** v0.4.6 (marketplace plugin) — `~/.memsearch/memory/YYYY-MM-DD.md` daily
|
||||
files + Milvus index; `/memory-recall`, `/memory-config`.
|
||||
|
||||
**Implementation status (2026-06-09):** The global Claude Code plugin is live (`~/.claude/plugins/os-vault/`). Steps 1, 2a, 2b, 3, and 6 of the build plan are complete (including the `onboard-project` skill, previously TODO under Step 6/Part D). Step 1 completed 2026-06-09: vault-conventions.md exists, all 6 fixture notes seeded, and 14 Graphify handbook + memory-system design notes migrated to the vault as migration scaffolding. Step 4 (memsearch) completed 2026-06-09: plugin installed via marketplace, MEMSEARCH_DIR set global (~/.memsearch), Stop hook verified producing daily memory files, search confirmed working. Step 5a (memsearch episodic git sync) completed 2026-06-09: dedicated private Forgejo repo (`forgejo.swansoncloud.com/jared/memsearch`), whitelist `.gitignore` (memory/*.md only), auto-commit+push via cc-os `session-end.sh` hook (ADR-015; relocated to `memsearch_sync.py` by ADR-016, 2026-06-12). Step 5b (Obsidian vault → VPS sync) remains. See `docs/memory-system/04-build-plan.md` for full step status.
|
||||
|
||||
**Decision (2026-06-09):** Single global memsearch store for ALL clients — cross-client commingling is the accepted, intended design (ADR-015 resolved). One private Forgejo repo (`forgejo.swansoncloud.com/jared/memsearch`), not per-client repos. Forward direction: minimize/eliminate dedicated per-client working directories; work projects locally or use a single general `clients/` dir for non-local client work — memory is captured globally regardless of cwd. No `clients/` directory structure designed yet; open item.
|
||||
|
||||
**Implementation status (2026-06-12):** os-vault plugin source moved into git at `cc-os/plugins/os-vault/` and bash hooks ported to Python (deep-module architecture: shared `config.py`, `hook_io.py`, `session_state.py`; thin entry-point scripts). Cutover via symlink `~/.claude/plugins/os-vault → cc-os/plugins/os-vault/` and settings.json hook rewrite. memsearch sync split into dedicated `memsearch_sync.py` SessionEnd hook (relocation of ADR-015 behavior, not reversal). Fresh-session test passed 2026-06-12. See ADR-016.
|
||||
|
||||
**Implementation status (2026-06-15):** Step 5b done and automated — vault (`~/Documents/SecondBrain`) initialized as git repo, pushed to private Forgejo (`ssh://git@forgejo.swansoncloud.com:2222/jared/SecondBrain.git`); 52 files, git chosen over Syncthing. Auto-commit+push wired via `vault_sync.py` SessionEnd hook (third SessionEnd hook, runs after `session_end.py` so the daily journal note is included; push-only — SessionStart pull is an optional future item for multi-machine). Step 2e pilot done — llf-schema (`/home/jared/dev/llf-schema`, PHP 8.2 WordPress plugin) onboarded via `/os-vault:onboard-project` skill: 605 nodes / 930 edges / 52 communities; Ollama doc pass lossy on WordPress docs but AST pass solid. Step 2d closed — live vault `graphify-out/` is the baseline; fixture-only build superseded.
|
||||
|
||||
**Implementation status (2026-06-17):** `/os-vault:onboard-project` skill now uses assessment-first onboarding: surveys the repo structure, classifies exclude candidates by TYPE (11 categories: fetched deps, build output, caches, VCS internals, editor/AI-tooling dirs, lockfiles, coverage/logs, bulk data, binaries, secrets, graphify-out/ — illustrative names, not fixed templates), generates a per-project `.graphifyignore`, confirms with the user, then extracts the graph using `qwen25-coder-7b-16k` (config.yaml `ollama_model`; 16k-context build of qwen2.5-coder:7b — larger context window cuts chunk count per doc, the main speed lever). Fixes issue where repos with large dependency trees routed non-code files through the Ollama doc pass (see ADR-017).
|
||||
|
||||
**Implementation status (2026-06-30):** Phase 1 of the SecondBrain Content Plan complete (`docs/memory-system/09-sb-content-plan.md`, issues #1–#6). `~/Documents/SecondBrain/vault-conventions.md` reconciled to a single authoritative typed frontmatter schema (removed the contradictory block; canonical filenames are slug-only, `source:` is a frontmatter field not a tag, `scope` is a field not a tag). New `/os-vault:design-template` skill added (`cc-os/plugins/os-vault/skills/design-template/SKILL.md`) — routes between template-design (4-step process + injection-economics filter) and new-type creation (9-step lifecycle). Three vault note templates created in `~/Documents/SecondBrain/_templates/` (`howto.md`, `convention.md`, `reference.md`; `reference` carries a four-subtype variant selector — pattern/framework, api-integration, role-definitions, design-rules), each dogfooded against a real note. Four proof-of-concept vault notes patched (cookbook subtype → pattern/framework, glossary de-duped to a wikilink, design-mode scope → project, tags + `last_reviewed` added). Phase 2 (issue #7) remains open as the steady-state migration/onboarding epic (no code). See `docs/memory-system/09-sb-content-plan.md`.
|
||||
|
||||
## Implemented Components
|
||||
|
||||
**Global os-vault plugin** — `cc-os/plugins/os-vault/` (git-tracked, 2026-06-12); symlinked into `~/.claude/plugins/os-vault`
|
||||
- Hooks: `hooks/` — `session_start.py`, `session_context.py` (project graph path only), `post_tool_use_write.py`, `session_end.py` (vault journal), `memsearch_sync.py` (second SessionEnd hook; memsearch auto-commit+push, 30s timeout), `vault_sync.py` (third SessionEnd hook; vault auto-commit+push to forgejo.swansoncloud.com/jared/SecondBrain, 30s timeout; mirrors memsearch_sync.py)
|
||||
- Shared modules: `config.py` (load_config → frozen Config dataclass), `hook_io.py` (read_input → HookInput dataclass), `session_state.py` (record_touch/read_touches; encapsulates `/tmp/claude-vault-touched-$SESSION_ID` contract)
|
||||
- Skills: `skills/` — query, write, reorganize, onboard-project (assessment-first onboarding: surveys repo, classifies excludes by type (11 categories), writes per-project `.graphifyignore`, confirms with user, then extracts using `qwen25-coder-7b-16k` — per ADR-017), design-template (routes template-design ↔ new-type-creation: 4-step template-design process + injection-economics filter, 9-step new-type lifecycle; backs Phase 1 of the SB Content Plan — see `docs/memory-system/09-sb-content-plan.md`)
|
||||
- Config: `config.yaml` — vault path, Ollama model (qwen25-coder-7b-16k), env vars
|
||||
- Hook wiring: `~/.claude/settings.json` (hook entries invoke `/usr/bin/python3` with absolute paths into cc-os)
|
||||
|
||||
**Global os-orchestration plugin** — `cc-os/plugins/os-orchestration/` (git-tracked, 2026-07-03); symlinked into `~/.claude/plugins/os-orchestration`
|
||||
- Hooks: `hooks/` — `inject.py` (injects `ORCHESTRATION.md` as additionalContext to all sessions)
|
||||
- Behavior: SessionStart hook injects an `ORCHESTRATION.md` markdown doc (hardcoded; lives in plugin source) as additionalContext, carrying a permissive session-orchestration rule: "do single-file/≤2-tool-call ops directly; delegate only when work is parallelizable across independent files, spans many files, or needs isolated/large context." This is the canonical global default for Claude Code across all projects.
|
||||
- Migration: migrated from a standalone repo (`~/dev/cc-plugins/orchestration/`, 2026-07-03) and integrated into cc-os. Supersedes the per-project copy-pasted orchestration text blocks that previously existed in individual project CLAUDE.md files (including a stricter local override that cc-os had carried — now removed; see ADR-019).
|
||||
- Session audit (WS1, 2026-07-06): 10 stratified real sessions audited via `audit/bin/extract` (Ruby fact-sheet extractor, dual-use as future eval checker) + parallel sonnet auditors + verified synthesis. Findings + E1–E4 eval-scenario backlog: `docs/orchestration-audit/2026-07-06-findings.md`. Headlines: model param was silently overridden to haiku on all 23 verified spawns — root cause found post-audit: `CLAUDE_CODE_SUBAGENT_MODEL=haiku` in `~/.claude/settings.json` env block (set by an earlier session as a cost measure; removed 2026-07-06; the Fable-5 correlation was coincidental timing). Omitted model param inherits the main-loop model (opus/sonnet) — misses cluster in pre-rollout and ops sessions; no over-sharing/full-dump failures anywhere.
|
||||
- Wording fixes shipped (2026-07-06, post-audit): ORCHESTRATION.md rewritten with trigger-conditioned when→then phrasing for the three verified clusters — explicit `model:` before every Agent call (cluster 2; a real cost exposure once the env override was removed), subagent model self-report on sonnet/opus spawns (cluster 1 — the E1 canary proved `resolvedModel` is NOT model-visible in the launch result; self-report is the only observable channel, and subagents do know their exact model ID), don't re-cover your own ground before delegating (cluster 3). Delegation thresholds unchanged. Cache refreshed via `bin/refresh-plugins`. Deterministic env-override detection (warn on `CLAUDE_CODE_SUBAGENT_MODEL`) is delegated to WS3's status-check design.
|
||||
- Eval harness E1–E3 (2026-07-06): `plugins/os-orchestration/eval/` — scripted headless orchestration-behavior eval scored by the audit extractor (extended with per-segment read targets). 6 run-set scenarios (E1P/N downgrade detection with `CLAUDE_CODE_SUBAGENT_MODEL=haiku` runner stub; E2P/N explicit-model fan-out vs no-delegation-on-trivial; E3P/N delegate-investigation vs orienting-reads-correct) + 6 frozen reserve twins; new Node.js "relaystation" fixture with deterministic sandbox-time log generation; deterministic-first checker (`eval/bin/check`, narrow frozen haiku-judge fallback for E1 language axes only, stubbable via `ORCH_EVAL_JUDGE_CMD`); model-free `bin/self-test` 21/21 green incl. shipped-instruction-compliant conformance transcripts. FAIL axes map only to verified audit misses; tier choice informational. **Scenario Task blocks are held-out — never run informally.** Procedure: `plugins/os-orchestration/eval/README.md`.
|
||||
- Baseline grid run (2026-07-06, run-set × sonnet/haiku × 3 reps + 2 counted canaries, all counted, zero harness errors): negatives 18/18 PASS both tiers (zero over-delegation, zero false downgrade claims); positives 1/18 on the delegate-at-all axis — but transcript verification shows most positive FAILs are *superior direct strategies* (E2P sonnet: one scripted bulk edit; E3P sonnet: surgical greps over 400KB logs), which the WS1 audit itself scored as justified non-delegation, vs genuine miss shapes (E3P haiku: whole 112KB logs into own context; E2P haiku: 20-call per-file grind). Conditional rules validated where engaged: 14/14 spawns explicit-model; the one delegating E1P rep flagged the stubbed downgrade via self-report. Criterion redesign for E2P/E3P A-axes is pre-registered (anchor thresholds independently, NOT to this grid) before any rescore/tuning; reserve stays frozen. ORCHESTRATION.md gained a scripted-bulk-edit carve-out (a uniform multi-file change one script covers is direct work). TSV: `eval/results/2026-07-06-baseline-grid.tsv`; full analysis: vault note [[os-orchestration-eval-baseline-grid-results]].
|
||||
|
||||
**Global os-doc-hygiene plugin** — `cc-os/plugins/os-doc-hygiene/` (git-tracked, 2026-07-03); symlinked into `~/.claude/plugins/os-doc-hygiene`
|
||||
- Hooks: `hooks/hooks.json` → SessionStart hook (matcher: startup|resume) runs `scripts/reminder.py` via `${CLAUDE_PLUGIN_ROOT}` (5s timeout), emitting a deterministic zero-token reminder banner
|
||||
- Behavior: Monitors stale and bloated project documentation per-project under `.dochygiene/` state dir (gitignored). SessionStart reminder is deterministic (no AI tokens, once/day snooze). Skills (verb-first, no `commands/` dispatcher — invoked directly as `/os-doc-hygiene:<skill>`, per [[cc-os-plugin-skill-naming-convention]]): `check` (AI-assisted classification of staleness signals, emits machine+human reports), `clean` (AI-assisted or deterministic patch application with git-safe scoped cleanup), `status` (read-only lifecycle-timestamp read), `sweep` (check then clean in sequence). Reversion-protected via invariants.md + golden-example test fixtures.
|
||||
- Migration: migrated from standalone repo (`~/dev/cc-plugins/doc-hygiene/`, 2026-07-03) and integrated into cc-os. No content conflicts detected (doc-hygiene scope does not overlap cc-os memory system). Renamed from `doc-hygiene` to `os-doc-hygiene` per cc-os plugin naming convention. Skills renamed `hygiene-check`/`hygiene-clean` → `check`/`clean`, and the `commands/hygiene.md` dispatcher removed in favor of two new skills (`status`, `sweep`), aligning with the `os-vault`/`os-orchestration` pattern of no `commands/` directory (2026-07-03).
|
||||
|
||||
**Global os-adr plugin** — `cc-os/plugins/os-adr/` (git-tracked, 2026-07-03); symlinked into `~/.claude/plugins/os-adr`
|
||||
- Purpose: Architecture Decision Records, consistently in every project — one customized-Nygard template (frontmatter `id`/`date`/`status`/`supersedes`/`superseded-by`/`affected-paths`/`affected-components`; Context/Decision/Consequences/Alternatives-rejected sections), `docs/adr/NNNN-kebab-title.md` one-file-per-decision + fully-regenerated `docs/adr/README.md` index. Built from `docs/adr-system/` (requirements `04`, PRD `05`, eval sketches `06`) via OpenSpec change `add-os-adr-plugin`.
|
||||
- Core: Ruby OO library `lib/adr/` (`Record`, `Repository`, `Index`, `Template`, `Detector`, `Migrator`, `MigrationReport`, `Finder`; Sandi Metz style, injected paths) behind thin CLIs in `bin/` (`adr-new`, `adr-init`, `adr-detect`, `adr-migrate`, `adr-find`). Tests: `ruby tests/all.rb` (47) + `python3 tests/hook_test.py` (15), all model-free. Behavioral invariants in `invariants.md`.
|
||||
- Hook: `hooks/session_start.py` wired via plugin-relative `hooks/hooks.json` (`${CLAUDE_PLUGIN_ROOT}`, no `settings.json` entries) — deterministic existence check: present → near-zero-token usage note naming `/os-adr:create` + `/os-adr:find`; absent → once-per-day init/migrate suggestion, silenced permanently by `.os-adr/suppress` (gitignored per-project state dir); silent outside git projects.
|
||||
- Skills (verb-first, no `commands/`): `create` (one-invocation templated+numbered+indexed ADR with mechanical supersession), `init`, `migrate` (non-destructive: detect surveyed shapes → mechanical heuristic fill → LLM fills only manifest-listed interpretive fields via `--apply-fills` → `migration_confidence` frontmatter + `docs/adr/migration-report.md`; old-system deletion only as an explicit separate user-approved step), `find` (deterministic-first retrieval: affected-paths/components match → Accepted-only filter → one-hop Graphify query-path expansion with graceful degradation → AI judgment over the candidate set only).
|
||||
- Migration pilot passed 2026-07-03 (sandboxed project copies; gate threshold 25% low-confidence flags): viking-warrior-training-log 0%, delta-refinery 0%, llf-schema 8.3%. Retrieval verified end-to-end against the real llf-schema project graph. Resolved: ADRs need no Graphify tag/edge convention — retrieval layer 3 expands the *query* paths via graph-node `source_file`s.
|
||||
- Eval A harness (2026-07-03, ADR-021): `plugins/os-adr/eval/` — prompted skill-execution eval across model tiers (haiku/sonnet). Two fixtures (generated with the plugin's own CLIs), six scenarios (S1 create, S2 create+supersede, S3 find/conflict, S4 find/distractor, S5 init, S6 migrate+fills), deterministic Ruby checker (`eval/bin/check`, structural invariants, TSV mode for autoresearch), sandbox + headless-runner scripts, runner-prompt template. Primary run mode: in-session Agent-tool subagents with pinned `model:` (cheaper than `claude -p`); optimized via the `/autoresearch` Classic loop over SKILL.md *wording only* (checker/fixtures/scenarios frozen during a loop). Procedure: `plugins/os-adr/eval/README.md`. Self-tested both directions; grid run 2026-07-03, all scenarios × both tiers passing. Distinct from the held-out Eval B (unprompted behavior) — do not conflate.
|
||||
- Eval B harness (2026-07-03, OpenSpec change `add-os-adr-eval-b-harness`): `plugins/os-adr/eval-b/` — held-out unprompted-behavior eval (requirements 4–5). 7 scenarios (W1–W3 write-trigger, R1–R4 retrieval) authored from the frozen shapes in `docs/adr-system/06-eval-scenarios.md`; dedicated Ruby webhook-relay fixture with a 6-ADR history (Superseded pair + near-miss distractors, generated via the plugin's own CLIs); R4's one-hop graph reach uses a real `graphify update` AST build (model-free, rebuilt via `eval-b/bin/build-fixture-graph`, never committed). **Headless-only runner** (`eval-b/bin/run` — fresh `claude -p` per rep, cwd = sandbox, so the real SessionStart hook fires; in-session subagents are invalid here, unlike Eval A) and a two-axis deterministic-first checker (`eval-b/bin/check`): axis (a) unprompted consultation, mechanical from transcript tool_use blocks; axis (b) correct-ADR citation (R1–R4) or new-ADR-file with a narrow frozen-rubric haiku judge fallback (W1–W3, `judge-rubric.md`, stubbable via `ADR_EVAL_B_JUDGE_CMD`). `R4-nograph` is the graph-degradation variant (expected FAIL). Self-tested both directions model-free via `eval-b/bin/self-test`. **Scenario Task blocks are held-out — never run them informally.** Procedure: `plugins/os-adr/eval-b/README.md`. Grid run 2026-07-03 (1 rep/cell): haiku 0/8 PASS (never unprompted-consults the ADR system in any scenario — a real gap, not a harness defect); sonnet 5/8 PASS (fails W3 — doesn't propose recording the decision; fails R1 — misses the direct-conflict retrieval). `R4-nograph` FAILed on both tiers as expected (degradation check, only meaningful paired with an R4 PASS — sonnet has one, haiku doesn't). Full results + observations (prompting-issue hypothesis, open question on whether in-session subagents could ever validly substitute for part of this measurement) written to the vault: [[os-adr-eval-b-grid-results-and-observations]].
|
||||
- **Initial iteration complete (2026-07-03):** plugin build, migration pilot, Eval A grid (clean pass), and Eval B grid (baseline captured above) are all done — this closes the first pass on os-adr. Follow-on work is deliberately deferred to future sessions, not in-flight.
|
||||
- **Wording experiment complete (2026-07-04):** the follow-up `/autoresearch` loop over Eval B wording ran (5 iterations, checker/fixtures/scenarios/rubric frozen; fixture CLAUDE.md declared a wording surface upfront) and closed the gap — final full grid **sonnet 8/8, haiku 7/8** (from 5/8 / 0/8 baseline; haiku's one miss is a W3 axis-b judge-boundary flicker). Winning wording shipped in `hooks/session_start.py` (PRESENT_NOTE), find/create SKILL.mds, and `eval-b/fixture/project/CLAUDE.md` (new — the trigger-phrased "Architecture decisions" section, the template for real-project adoption). Confirmed mechanisms: trigger-conditioned when→then phrasing beats inventory phrasing on both tiers; each rule must live where its precondition is visible (the reversal→supersede rule in the find skill's act-on-findings step fixed W3); lower tiers need *mechanical* triggers ("before your first edit to any existing file → run `/os-adr:find` on those paths; additions count") — semantic triggers ("architecture-level choice") only reach sonnet. Open: channel ablation never run (hook vs CLAUDE.md redundancy unknown); R4-nograph now passes both tiers, so the graph-degradation check no longer differentiates. Full hypothesis→result mapping: vault note [[os-adr-eval-b-wording-experiment-hypotheses]]. **Before designing or running any autoresearch eval, Read `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md`.**
|
||||
- **Eval B W3 stability check (2026-07-06):** 7 headless reps of haiku × W3 (all reps counted, including two the runner initially excluded, to avoid peeking bias): axis (a) consultation 7/7 PASS; axis (b) recording-offer ~5/7 (~75–85%) — one FAIL plausibly infra, one a genuine behavioral miss (consulted, then asked a clarifying question instead of unconditionally proposing the superseding ADR; the conditional-phrasing failure mode iteration 3 fixed on sonnet, improved but not eliminated on haiku). Verdict: intermittent flicker, not a hard gap — grid claim stays **haiku 7/8**, W3 characterized as an ~1-in-4/5 axis-b miss. Haiku cleared for daily use with that caveat. Recorded in the vault good-enough gate: [[os-adr-eval-b-wording-experiment-hypotheses]].
|
||||
- **Eval C harness built, NOT yet run (2026-07-06):** `plugins/os-adr/eval-c/` — ambiguity-ladder DISCRIMINATION eval (held-out; Eval B is contaminated by the wording loop). 3 levels (explicit → moderate → conceptual framing) × paired positive/negative scenarios; run-set (6, job-execution domain) + frozen reserve-set (6, notifications domain — becomes the measurement set if anyone ever tunes wording against the run-set). New Python fixture (`taskq` async job queue, 6-ADR history generated via the plugin's own CLIs, trigger-phrased CLAUDE.md section copied from eval-b). Scoring: positives = both axes; negatives FAIL **only on unneeded ADR creation** — consultation and truthful ADR citation are informational (`A:yes/no`, `cited-adr:yes/no`), never FAILs, because the shipped wording endorses cheap finds and the negatives sit deliberately in ADR-covered domains (two earlier negative designs that punished instruction-compliant behavior were caught in review and redesigned/fixed). Model-free `bin/self-test` green, including a truthful-citation-must-PASS guard. **Scenario Task blocks are held-out — never run informally.** Methodology: vault notes [[eval-methodology-ladder]] (per-level pass bars, reserve discipline) and [[eval-methodology-irl-feedback-loop]] (post-rollout session-audit backlog); vault also gained an `eval-results` note type + `_templates/eval-results.md` (2026-07-06).
|
||||
- **Eval C frozen grid run (2026-07-06):** run-set × {sonnet, haiku} × 3 reps/cell (36 headless reps, all counted) — **12/12 cells PASS at ≥2/3 majority; haiku 18/18 reps, sonnet 17/18**. Zero over-trigger ADR creation on any negative at any level; every positive pass was via governing-ADR citation (16) or judge-recognized proposal (1), never duplicate creation. Sonnet's one FAIL rep (P1-L1) is a final-message-citation boundary artifact (implemented the ADR-0002 policy, cited only 0005), not a behavioral miss. Two harness defects fixed on the first live rep, BEFORE the grid, with Task blocks/fixtures/rubric untouched: (1) `bin/run` pipefail bug aborted a cell's reps after the first FAIL; (2) positive axis-b required creation/proposal, contradicting the README's own "consulting and/or recording is correct" — the run-set positives sit in *already-decided* territory, so find→cite→comply is correct; fixed via a mechanical `cited-governing` PASS branch + two new self-test guards (`B-via:` now recorded in TSV). Verdict: the Eval B wording **generalizes** (new fixture/language/domain, decreasing cue explicitness, paired negatives) — grid saturated, ceiling not found; next signal is production (rollout + IRL session audits), not another lab rung. Run-set is now the baseline; wording tuning against it moves measurement to the reserve-set. Full results: vault note [[os-adr-eval-c-frozen-grid-results]].
|
||||
- **Remaining (locked rollout order):** real-project migration/adoption one at a time via `/os-adr:migrate` — pilot projects first, then the cc-os retrofit (its 19-ADR monolithic file was deliberately excluded from the pilot), then wider. When onboarding real projects, add the trigger-phrased "Architecture decisions" CLAUDE.md section (copy from `eval-b/fixture/project/CLAUDE.md`; candidate: emit it from `/os-adr:init`/`migrate`).
|
||||
- **Resolved (2026-07-04):** os-adr's skills failed to register on first install due to stale plugin caches — the cache that was installed on 2026-07-03 17:21:48 was missing `hooks/hooks.json`, `bin/adr-detect`, `bin/adr-find`, `bin/adr-migrate`. Root cause: unknown, but the caches were restored via plugin uninstall/reinstall. A parallel issue affected os-doc-hygiene (cache retained deleted `commands/` directory). Fixed by manually refreshing both caches, then created `bin/refresh-plugins` automation to prevent future drift — see "Editing a local plugin (cache refresh)" subsection for refresh procedure. Investigation result: stale caches (not manifest-naming issues) were the cause; slash command registration works correctly once caches are fresh.
|
||||
|
||||
**Graphify** — v0.8.31 at `/home/jared/.local/bin/graphify`
|
||||
- Vault graph: `~/Documents/SecondBrain/graphify-out/` — disposable, structural index rebuilt by SessionStart hook; handles relational/graph queries. Distinct from vault-index.json (planned, Plan B Phase 1): a flat `{tag: [{path, title, summary}]}` lookup rebuilt at SessionEnd, queried by the `/memory-find` skill for fast tag-based SB note discovery.
|
||||
- Project graph: `<project-root>/graphify-out/` — same pattern; gitignore it in each project repo
|
||||
- Vault conventions: `~/Documents/SecondBrain/vault-conventions.md` — frontmatter contract + tag taxonomy (canonical name decided 2026-06-09; formerly referred to as CONVENTIONS.md)
|
||||
|
||||
**memsearch** — v0.4.6 via Claude Code plugin marketplace (`memsearch@memsearch-plugins`)
|
||||
- Hooks: Stop, SessionStart, UserPromptSubmit, SessionEnd (ship with plugin)
|
||||
- Memory store: `~/.memsearch/memory/YYYY-MM-DD.md` daily files (global, cross-project, all clients — one store by design)
|
||||
- Index: `~/.memsearch/milvus.db` (Milvus Lite, local); embeddings via ONNX bge-m3
|
||||
- Config: `MEMSEARCH_DIR=~/.memsearch` in `~/.zshrc` for global scope
|
||||
- Skills: `/memory-recall`, `/memory-config` (ship with plugin)
|
||||
- Git sync: `~/.memsearch` is a dedicated private Forgejo repo (`forgejo.swansoncloud.com/jared/memsearch`); whitelist `.gitignore` commits only `memory/*.md` (excludes rebuildable `milvus.db`, model, config); auto-commit+push wired into the cc-os memory plugin's own `memsearch_sync.py` SessionEnd hook (relocated from `session-end.sh` by ADR-016, 2026-06-12; behavior preserved — see ADR-015)
|
||||
|
||||
**Obsidian vault git sync** — `~/Documents/SecondBrain` (2026-06-15)
|
||||
- Remote: `ssh://git@forgejo.swansoncloud.com:2222/jared/SecondBrain.git` (private Forgejo; web: `https://forgejo.swansoncloud.com/jared/SecondBrain`)
|
||||
- 52 files tracked (notes, journal, templates, vault-conventions.md, CLAUDE.md, .obsidian config); `graphify-out/` excluded via `.gitignore`
|
||||
- Auto-commit+push via `vault_sync.py` SessionEnd hook in cc-os memory plugin (mirrors `memsearch_sync.py`; push-only — SessionStart pull is optional future item for multi-machine)
|
||||
|
||||
**Remaining optional items:** SessionStart vault pull (multi-machine sync; push-only is the current design); additional project onboarding (one at a time, per ADR-013); bulk vault migration. All required build steps complete as of 2026-06-15.
|
||||
|
||||
**Plugin renamed `memory` → `os-vault` (2026-07-03):** directory renamed in git, `~/.claude/plugins/os-vault` symlink and `settings.json`'s `enabledPlugins` updated — but the `local-plugins` marketplace manifest (`~/.claude/plugins/.claude-plugin/marketplace.json`) still declared the old `memory`/`./memory` entry, and `installed_plugins.json` still had a cached `memory@local-plugins` install record. Skills (and therefore slash commands) never registered under the new name as a result; hooks kept working because they're wired by absolute path in `settings.json`, independent of plugin resolution. Fixed by editing the marketplace manifest to `"os-vault"`/`"./os-vault"`, then `claude plugin marketplace update local-plugins`, `claude plugin install os-vault@local-plugins`, `claude plugin uninstall memory@local-plugins`. See ADR-018 and the "Renaming or moving a local plugin" procedure below.
|
||||
|
||||
### Renaming or moving a local plugin
|
||||
|
||||
Renaming a plugin directory under a directory-source marketplace (like `local-plugins`, whose
|
||||
source is `~/.claude/plugins` itself) requires updating **three** places, not just the directory
|
||||
and `settings.json`:
|
||||
|
||||
1. **Plugin directory / symlink** — the actual files (or symlink target, for plugins that live in
|
||||
a git repo like `cc-os/plugins/os-vault/`).
|
||||
2. **Marketplace manifest** — `~/.claude/plugins/.claude-plugin/marketplace.json` (for
|
||||
`local-plugins`) lists each plugin's `name` and `source` path explicitly; this does **not**
|
||||
auto-discover from the directory listing. Update the entry to match the new name/path.
|
||||
3. **`settings.json` `enabledPlugins`** — the `<name>@<marketplace>` key must match the manifest
|
||||
entry from step 2.
|
||||
|
||||
Then refresh the plugin manager's state so it re-resolves the marketplace and drops the stale
|
||||
install record:
|
||||
|
||||
```bash
|
||||
claude plugin marketplace update <marketplace-name> # re-validates the manifest
|
||||
claude plugin install <new-name>@<marketplace-name> # creates the new install record
|
||||
claude plugin uninstall <old-name>@<marketplace-name> # drops the stale one
|
||||
claude plugin list # verify: new name enabled, old name gone
|
||||
claude plugin details <new-name>@<marketplace-name> # verify skills/agents/hooks resolved
|
||||
```
|
||||
|
||||
Skipping step 2 is the failure mode to watch for: hooks (wired by absolute path in
|
||||
`settings.json`) keep working, masking the fact that skills/slash-commands silently stopped
|
||||
registering under the plugin's new name.
|
||||
|
||||
### Editing a local plugin (cache refresh)
|
||||
|
||||
Plugin source edits (including SKILL.md wording, hook scripts, or CLI code) do not reach running
|
||||
sessions until the cache is refreshed. Claude Code caches plugins when installed, copying them
|
||||
to `~/.claude/plugins/cache/<marketplace>/<plugin>/<version>/`. To refresh a stale cache after
|
||||
source edits:
|
||||
|
||||
```bash
|
||||
bin/refresh-plugins [marketplace-names...] # refresh all or specific local marketplaces
|
||||
```
|
||||
|
||||
This uninstalls and reinstalls each plugin, forcing the cache to repopulate. See `bin/refresh-plugins --help` for usage. After refreshing, `claude plugin details <name>@<marketplace>` should show updated skill descriptions and hook counts. Directory-source marketplaces (e.g., `local-plugins` sourced from `~/.claude/plugins`, `cc-plugins` sourced from `~/dev/cc-plugins`) are the only ones that need manual refresh — public marketplaces auto-update.
|
||||
**Eval discipline (applies to every harness above):** scenario Task blocks are held-out —
|
||||
never run them informally; reserve sets are never even read informally. Before designing or
|
||||
running any autoresearch eval, Read
|
||||
`~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md`. Wording loops for
|
||||
os-adr Eval B, os-vault WS2, and os-orchestration WS4 are complete and shipped
|
||||
(2026-07-04/07/08); their run-sets AND reserves are contaminated for future wording tuning —
|
||||
next signal is production IRL session audits.
|
||||
|
||||
## Issue tracking
|
||||
|
||||
Issues (created via `/to-issues`) live on self-hosted Forgejo (`jared/cc-os`), queried with the `tea` CLI — not GitHub/`gh`. See `docs/issue-workflow.md` for token setup, listing, implementing, and closing issues.
|
||||
Issues (created via `/to-issues`) live on self-hosted Forgejo (`jared/cc-os`), queried with
|
||||
the `tea` CLI — not GitHub/`gh`. See `docs/issue-workflow.md`.
|
||||
|
||||
## OpenSpec workflow
|
||||
|
||||
Changes are managed spec-driven via OpenSpec. Use the matching skills rather than editing spec
|
||||
files by hand: `openspec-explore` (think through an idea), `openspec-propose` (create a change
|
||||
with design/specs/tasks), `openspec-apply-change` (implement tasks), `openspec-verify-change`
|
||||
(validate before archiving), `openspec-archive-change`. Slash commands mirror these under
|
||||
`/opsx:*`. Live changes live in `openspec/changes/`, completed ones in
|
||||
`openspec/changes/archive/`, stable specs in `openspec/specs/`.
|
||||
|
||||
`openspec/config.yaml` uses `schema: spec-driven`; its `context` block is currently empty —
|
||||
project context for OpenSpec artifacts comes from `docs/` and this file.
|
||||
Changes are managed spec-driven via OpenSpec. Use the matching skills rather than editing
|
||||
spec files by hand: `openspec-explore`, `openspec-propose`, `openspec-apply-change`,
|
||||
`openspec-verify-change`, `openspec-archive-change` (slash commands under `/opsx:*`). Live
|
||||
changes in `openspec/changes/`, completed in `openspec/changes/archive/`, stable specs in
|
||||
`openspec/specs/`. `openspec/config.yaml` uses `schema: spec-driven`; project context for
|
||||
OpenSpec artifacts comes from `docs/` and this file.
|
||||
|
||||
## Conventions specific to this repo
|
||||
|
||||
- **Provenance discipline:** when writing about Graphify or anything sourced from the
|
||||
interview transcripts, keep the inline source tags and never promote an `[interview]` /
|
||||
`[unverified claim]` to fact without checking a primary source (the GitHub repo is the
|
||||
authority; it was anchored to Graphify **v0.8.30**).
|
||||
- **Dates are absolute** (e.g. `2026-06-03`), and design docs carry a `_Last updated:_` /
|
||||
status line — keep these current when editing.
|
||||
- **Decisions live in ADRs.** Don't silently reverse a locked decision; add or amend an ADR in
|
||||
`03-architecture-decisions.md` with the reasoning.
|
||||
- The package on PyPI is `graphifyy` (double-y) but the command is `graphify`.
|
||||
- **Keep this file current:** When a build step completes, (a) mark it done in `docs/memory-system/04-build-plan.md`, (b) add or update a component pointer in the Implemented Components section above, and (c) update the current design paragraph if the design changed. This file is the AI's orientation entry point — accuracy matters more than brevity.
|
||||
- **Plugin and skill naming:** Before naming ANY new cc-os plugin, skill, or slash command, Read
|
||||
`~/Documents/SecondBrain/cc-os-plugin-skill-naming-convention.md` and follow it — do NOT invent
|
||||
a new convention. Rules in brief: plugins are `os-[domain]`; skills are verb-first kebab-case
|
||||
(`query`, `onboard-project`), invoked as `/os-[domain]:[verb]`; no `commands/` dispatcher
|
||||
directories; **never set a `name:` field in SKILL.md frontmatter** — the directory name is the
|
||||
skill name, and an explicit `name:` collapses the slash command to a bare unnamespaced form
|
||||
(`/find` instead of `/os-adr:find`; found and fixed 2026-07-04). After editing any SKILL.md,
|
||||
run `bin/refresh-plugins` (installs cache plugin files; source edits don't reach sessions until
|
||||
refreshed). Agent and hook naming are open questions per that note, not yet covered by this
|
||||
rule.
|
||||
|
||||
**Session orchestration behavior** — Provided by the global `os-orchestration` plugin (see Implemented Components below). This repo no longer carries a local orchestration override; it follows the plugin's default behavior like every other project.
|
||||
authority; anchored to Graphify v0.8.30).
|
||||
- **Dates are absolute** (e.g. `2026-06-03`); design docs carry a `_Last updated:_` / status
|
||||
line — keep these current when editing.
|
||||
- **Decisions live in ADRs.** Don't silently reverse a locked decision; add or amend an ADR
|
||||
in `docs/memory-system/03-architecture-decisions.md` with the reasoning.
|
||||
- **Keep records current:** when a build step completes, (a) mark it done in
|
||||
`docs/memory-system/04-build-plan.md`, (b) record it in
|
||||
`docs/implementation-status.md` (timeline entry + component pointer), and (c) update the
|
||||
design paragraph in THIS file only if the design itself changed. Never append status
|
||||
history to this file — it goes in implementation-status.md.
|
||||
- **Plugin and skill naming:** before naming ANY new cc-os plugin, skill, or slash command,
|
||||
Read `~/Documents/SecondBrain/cc-os-plugin-skill-naming-convention.md` and follow it. In
|
||||
brief: plugins are `os-[domain]`; skills are verb-first kebab-case, invoked as
|
||||
`/os-[domain]:[verb]`; no `commands/` dispatcher directories; **never set a `name:` field
|
||||
in SKILL.md frontmatter** (it collapses the slash command to a bare unnamespaced form).
|
||||
- **After editing any plugin source** (SKILL.md, hooks, CLIs), run `bin/refresh-plugins` —
|
||||
source edits don't reach sessions until the cache is refreshed. Rename/move procedure and
|
||||
cache details: docs/implementation-status.md → "Operational procedures".
|
||||
|
|
|
|||
|
|
@ -0,0 +1,490 @@
|
|||
# cc-os implementation status & changelog
|
||||
|
||||
_Last updated: 2026-07-08. This is the append-only implementation-status record moved out of
|
||||
`CLAUDE.md` (2026-07-08, task B1 of
|
||||
[system-prompt-profiles-tasks.md](plans/system-prompt-profiles-tasks.md)). CLAUDE.md keeps
|
||||
orientation only; when a build step completes, record it HERE (plus
|
||||
`docs/memory-system/04-build-plan.md`), and touch CLAUDE.md only if the current design or
|
||||
conventions changed._
|
||||
|
||||
## Status timeline
|
||||
|
||||
**Recent pivot (2026-06-03):** Graphify **replaces** the originally-planned Ruby/SQLite
|
||||
tag-index CLI and also covers the deferred QMD semantic layer. `04-build-plan.md` and
|
||||
`06-graphify-evaluation.md` reflect this; if an older doc still describes the Ruby CLI, defer
|
||||
to those two and fix the stale doc.
|
||||
|
||||
**Decisions locked (2026-06-04):** Six-facet tag taxonomy + `scope/` (ADR-011); reuse
|
||||
`~/Documents/SecondBrain` vault rather than creating a new one (ADR-012); build-first /
|
||||
migrate-incrementally — build full system against a fixture set first, defer bulk vault
|
||||
migration to last, onboard projects one at a time (ADR-013).
|
||||
|
||||
**Empirical finding locked (2026-06-05):** Graphify is a structure extractor, not a topic
|
||||
clusterer — no emergent hub nodes appear even at `--mode deep`; hub notes + wikilinks must be
|
||||
author-provided during migration (not deferred). Migration scaffolding is now a first-class
|
||||
deliverable. Open question: do facet tags create graph edges? (ADR-014; findings:
|
||||
`docs/memory-system/07-graph-connectivity-findings.md`).
|
||||
|
||||
**Implementation status (2026-06-09):** The global Claude Code plugin is live
|
||||
(`~/.claude/plugins/os-vault/`). Steps 1, 2a, 2b, 3, and 6 of the build plan are complete
|
||||
(including the `onboard-project` skill, previously TODO under Step 6/Part D). Step 1
|
||||
completed 2026-06-09: vault-conventions.md exists, all 6 fixture notes seeded, and 14
|
||||
Graphify handbook + memory-system design notes migrated to the vault as migration
|
||||
scaffolding. Step 4 (memsearch) completed 2026-06-09: plugin installed via marketplace,
|
||||
MEMSEARCH_DIR set global (~/.memsearch), Stop hook verified producing daily memory files,
|
||||
search confirmed working. Step 5a (memsearch episodic git sync) completed 2026-06-09:
|
||||
dedicated private Forgejo repo (`forgejo.swansoncloud.com/jared/memsearch`), whitelist
|
||||
`.gitignore` (memory/*.md only), auto-commit+push via cc-os `session-end.sh` hook (ADR-015;
|
||||
relocated to `memsearch_sync.py` by ADR-016, 2026-06-12). Step 5b (Obsidian vault → VPS
|
||||
sync) remained at that date.
|
||||
|
||||
**Decision (2026-06-09):** Single global memsearch store for ALL clients — cross-client
|
||||
commingling is the accepted, intended design (ADR-015 resolved). One private Forgejo repo
|
||||
(`forgejo.swansoncloud.com/jared/memsearch`), not per-client repos. Forward direction:
|
||||
minimize/eliminate dedicated per-client working directories; work projects locally or use a
|
||||
single general `clients/` dir for non-local client work — memory is captured globally
|
||||
regardless of cwd. No `clients/` directory structure designed yet; open item.
|
||||
|
||||
**Implementation status (2026-06-12):** os-vault plugin source moved into git at
|
||||
`cc-os/plugins/os-vault/` and bash hooks ported to Python (deep-module architecture: shared
|
||||
`config.py`, `hook_io.py`, `session_state.py`; thin entry-point scripts). Cutover via symlink
|
||||
`~/.claude/plugins/os-vault → cc-os/plugins/os-vault/` and settings.json hook rewrite.
|
||||
memsearch sync split into dedicated `memsearch_sync.py` SessionEnd hook (relocation of
|
||||
ADR-015 behavior, not reversal). Fresh-session test passed 2026-06-12. See ADR-016.
|
||||
|
||||
**Implementation status (2026-06-15):** Step 5b done and automated — vault
|
||||
(`~/Documents/SecondBrain`) initialized as git repo, pushed to private Forgejo
|
||||
(`ssh://git@forgejo.swansoncloud.com:2222/jared/SecondBrain.git`); 52 files, git chosen over
|
||||
Syncthing. Auto-commit+push wired via `vault_sync.py` SessionEnd hook (third SessionEnd
|
||||
hook, runs after `session_end.py` so the daily journal note is included; push-only —
|
||||
SessionStart pull is an optional future item for multi-machine). Step 2e pilot done —
|
||||
llf-schema (`/home/jared/dev/llf-schema`, PHP 8.2 WordPress plugin) onboarded via
|
||||
`/os-vault:onboard-project` skill: 605 nodes / 930 edges / 52 communities; Ollama doc pass
|
||||
lossy on WordPress docs but AST pass solid. Step 2d closed — live vault `graphify-out/` is
|
||||
the baseline; fixture-only build superseded.
|
||||
|
||||
**Implementation status (2026-06-17):** `/os-vault:onboard-project` skill now uses
|
||||
assessment-first onboarding: surveys the repo structure, classifies exclude candidates by
|
||||
TYPE (11 categories: fetched deps, build output, caches, VCS internals, editor/AI-tooling
|
||||
dirs, lockfiles, coverage/logs, bulk data, binaries, secrets, graphify-out/ — illustrative
|
||||
names, not fixed templates), generates a per-project `.graphifyignore`, confirms with the
|
||||
user, then extracts the graph using `qwen25-coder-7b-16k` (config.yaml `ollama_model`;
|
||||
16k-context build of qwen2.5-coder:7b — larger context window cuts chunk count per doc, the
|
||||
main speed lever). Fixes issue where repos with large dependency trees routed non-code files
|
||||
through the Ollama doc pass (see ADR-017).
|
||||
|
||||
**Implementation status (2026-06-30):** Phase 1 of the SecondBrain Content Plan complete
|
||||
(`docs/memory-system/09-sb-content-plan.md`, issues #1–#6).
|
||||
`~/Documents/SecondBrain/vault-conventions.md` reconciled to a single authoritative typed
|
||||
frontmatter schema (removed the contradictory block; canonical filenames are slug-only,
|
||||
`source:` is a frontmatter field not a tag, `scope` is a field not a tag). New
|
||||
`/os-vault:design-template` skill added
|
||||
(`cc-os/plugins/os-vault/skills/design-template/SKILL.md`) — routes between template-design
|
||||
(4-step process + injection-economics filter) and new-type creation (9-step lifecycle).
|
||||
Three vault note templates created in `~/Documents/SecondBrain/_templates/` (`howto.md`,
|
||||
`convention.md`, `reference.md`; `reference` carries a four-subtype variant selector —
|
||||
pattern/framework, api-integration, role-definitions, design-rules), each dogfooded against
|
||||
a real note. Four proof-of-concept vault notes patched (cookbook subtype →
|
||||
pattern/framework, glossary de-duped to a wikilink, design-mode scope → project, tags +
|
||||
`last_reviewed` added). Phase 2 (issue #7) remains open as the steady-state
|
||||
migration/onboarding epic (no code). See `docs/memory-system/09-sb-content-plan.md`.
|
||||
|
||||
**Plugin renamed `memory` → `os-vault` (2026-07-03):** directory renamed in git,
|
||||
`~/.claude/plugins/os-vault` symlink and `settings.json`'s `enabledPlugins` updated — but
|
||||
the `local-plugins` marketplace manifest
|
||||
(`~/.claude/plugins/.claude-plugin/marketplace.json`) still declared the old
|
||||
`memory`/`./memory` entry, and `installed_plugins.json` still had a cached
|
||||
`memory@local-plugins` install record. Skills (and therefore slash commands) never
|
||||
registered under the new name as a result; hooks kept working because they're wired by
|
||||
absolute path in `settings.json`, independent of plugin resolution. Fixed by editing the
|
||||
marketplace manifest to `"os-vault"`/`"./os-vault"`, then
|
||||
`claude plugin marketplace update local-plugins`,
|
||||
`claude plugin install os-vault@local-plugins`,
|
||||
`claude plugin uninstall memory@local-plugins`. See ADR-018 and the "Renaming or moving a
|
||||
local plugin" procedure below.
|
||||
|
||||
**Resolved (2026-07-04):** os-adr's skills failed to register on first install due to stale
|
||||
plugin caches — the cache that was installed on 2026-07-03 17:21:48 was missing
|
||||
`hooks/hooks.json`, `bin/adr-detect`, `bin/adr-find`, `bin/adr-migrate`. Root cause:
|
||||
unknown, but the caches were restored via plugin uninstall/reinstall. A parallel issue
|
||||
affected os-doc-hygiene (cache retained deleted `commands/` directory). Fixed by manually
|
||||
refreshing both caches, then created `bin/refresh-plugins` automation to prevent future
|
||||
drift — see "Editing a local plugin (cache refresh)" below. Investigation result: stale
|
||||
caches (not manifest-naming issues) were the cause; slash command registration works
|
||||
correctly once caches are fresh.
|
||||
|
||||
**Remaining optional items:** SessionStart vault pull (multi-machine sync; push-only is the
|
||||
current design); additional project onboarding (one at a time, per ADR-013); bulk vault
|
||||
migration. All required build steps complete as of 2026-06-15.
|
||||
|
||||
## Implemented components
|
||||
|
||||
**Global os-vault plugin** — `cc-os/plugins/os-vault/` (git-tracked, 2026-06-12); symlinked
|
||||
into `~/.claude/plugins/os-vault`
|
||||
- Hooks: `hooks/` — `session_start.py`, `session_context.py` (project graph path only),
|
||||
`post_tool_use_write.py`, `session_end.py` (vault journal), `memsearch_sync.py` (second
|
||||
SessionEnd hook; memsearch auto-commit+push, 30s timeout), `vault_sync.py` (third
|
||||
SessionEnd hook; vault auto-commit+push to forgejo.swansoncloud.com/jared/SecondBrain, 30s
|
||||
timeout; mirrors memsearch_sync.py)
|
||||
- Shared modules: `config.py` (load_config → frozen Config dataclass), `hook_io.py`
|
||||
(read_input → HookInput dataclass), `session_state.py` (record_touch/read_touches;
|
||||
encapsulates `/tmp/claude-vault-touched-$SESSION_ID` contract)
|
||||
- Skills: `skills/` — query, write, reorganize, onboard-project (assessment-first
|
||||
onboarding: surveys repo, classifies excludes by type (11 categories), writes per-project
|
||||
`.graphifyignore`, confirms with user, then extracts using `qwen25-coder-7b-16k` — per
|
||||
ADR-017), design-template (routes template-design ↔ new-type-creation: 4-step
|
||||
template-design process + injection-economics filter, 9-step new-type lifecycle; backs
|
||||
Phase 1 of the SB Content Plan — see `docs/memory-system/09-sb-content-plan.md`)
|
||||
- Config: `config.yaml` — vault path, Ollama model (qwen25-coder-7b-16k), env vars
|
||||
- Hook wiring: `~/.claude/settings.json` (hook entries invoke `/usr/bin/python3` with
|
||||
absolute paths into cc-os)
|
||||
- Isolation seam (2026-07-06, WS2): `OS_VAULT_PATH` env var overrides `vault_path` for the
|
||||
write skill and every hook via `config.load_config()`; `OS_VAULT_SKIP_REBUILD` suppresses
|
||||
the SessionStart graphify rebuild. Added for the eval harness; zero production impact when
|
||||
unset.
|
||||
- Write-behavior eval (2026-07-06, WS2): `plugins/os-vault/eval/` — held-out unprompted
|
||||
vault-write discrimination eval (Eval B/C playbook: ambiguity ladder L1 explicit → L3
|
||||
conceptual, paired positives/negatives, 6 run-set scenarios on a new `reportgen` Ruby
|
||||
fixture + 6 frozen reserve twins in a different knowledge domain, isolated sandbox vault,
|
||||
headless-only runner, deterministic-first Ruby checker with narrow frozen offer-detection
|
||||
judge fallback stubbable via `OS_VAULT_EVAL_JUDGE_CMD`, model-free `bin/self-test` 21/21).
|
||||
**Scenario Task blocks are held-out — never run informally; the reserve set is never even
|
||||
read informally.** Procedure: `plugins/os-vault/eval/README.md`.
|
||||
- Untuned baseline grid (2026-07-06, run-set × sonnet/haiku × 3 reps + 1 counted canary, 37
|
||||
reps, zero harness errors): **positives 1/19, negatives 18/18**. Headline: at L1 (explicit
|
||||
cue) both tiers persist the knowledge every rep but route it to Claude Code's built-in
|
||||
auto-memory (`~/.claude/projects/<slug>/memory/`) instead of the vault; L2/L3 mostly don't
|
||||
persist at all; zero over-triggering anywhere. TSV:
|
||||
`eval/results/2026-07-06-baseline-grid.tsv`; full analysis: vault note
|
||||
[[os-vault-write-eval-baseline-grid-results]].
|
||||
- WS2 wording loop COMPLETE (2026-07-07, 4 iterations + confirm + held-out reserve; run-set
|
||||
AND reserve now contaminated): shipped destination-ladder write SKILL.md + SessionStart
|
||||
USAGE_NOTE (mechanical triggers + before-final-reply checkpoint), mechanical vault-root
|
||||
resolution (`${OS_VAULT_PATH:-$HOME/Documents/SecondBrain}` — fixed real-vault breaches
|
||||
found on iteration 1), fixture CLAUDE.md "Cross-project memory" section (**the
|
||||
real-project adoption template**, required to reach haiku at L2/L3; includes ADR-vs-vault
|
||||
disambiguation), candidates 2–4 unscored, post-reserve YAML-colon quoting rule. Confirm
|
||||
grid sonnet 18/18 / haiku 16/18; reserve (held-out) positives 8/18 strict / trigger-axis
|
||||
13/18, negatives 17/18 (vs baseline 1/19, 18/18). Residual gaps: YAML-hostile titles,
|
||||
haiku L2 no-trigger, generalization-laundering over-trigger. One checker false-positive
|
||||
fixed+rescored mid-loop (breach regex vs `2>/dev/null`); negative-criterion redesign
|
||||
pre-registered. TSVs: `eval/results/2026-07-07-wording-*.tsv`; loop log:
|
||||
`autoresearch/classic-260707-1253/loop-log.md`; analysis: vault note
|
||||
[[os-vault-write-eval-wording-loop-results]]. Next signal is production IRL audits.
|
||||
|
||||
**Global os-orchestration plugin** — `cc-os/plugins/os-orchestration/` (git-tracked,
|
||||
2026-07-03); symlinked into `~/.claude/plugins/os-orchestration`
|
||||
- Hooks: `hooks/` — `inject.py` (injects `ORCHESTRATION.md` as additionalContext to all
|
||||
sessions)
|
||||
- Behavior: SessionStart hook injects an `ORCHESTRATION.md` markdown doc (hardcoded; lives
|
||||
in plugin source) as additionalContext, carrying the canonical global
|
||||
session-orchestration rule set for Claude Code across all projects.
|
||||
- Migration: migrated from a standalone repo (`~/dev/cc-plugins/orchestration/`, 2026-07-03)
|
||||
and integrated into cc-os. Supersedes the per-project copy-pasted orchestration text
|
||||
blocks that previously existed in individual project CLAUDE.md files (including a stricter
|
||||
local override that cc-os had carried — now removed; see ADR-019).
|
||||
- Session audit (WS1, 2026-07-06): 10 stratified real sessions audited via
|
||||
`audit/bin/extract` (Ruby fact-sheet extractor, dual-use as future eval checker) +
|
||||
parallel sonnet auditors + verified synthesis. Findings + E1–E4 eval-scenario backlog:
|
||||
`docs/orchestration-audit/2026-07-06-findings.md`. Headlines: model param was silently
|
||||
overridden to haiku on all 23 verified spawns — root cause found post-audit:
|
||||
`CLAUDE_CODE_SUBAGENT_MODEL=haiku` in `~/.claude/settings.json` env block (set by an
|
||||
earlier session as a cost measure; removed 2026-07-06; the Fable-5 correlation was
|
||||
coincidental timing). Omitted model param inherits the main-loop model (opus/sonnet) —
|
||||
misses cluster in pre-rollout and ops sessions; no over-sharing/full-dump failures
|
||||
anywhere.
|
||||
- Wording fixes shipped (2026-07-06, post-audit): ORCHESTRATION.md rewritten with
|
||||
trigger-conditioned when→then phrasing for the three verified clusters — explicit
|
||||
`model:` before every Agent call (cluster 2), subagent model self-report on sonnet/opus
|
||||
spawns (cluster 1 — the E1 canary proved `resolvedModel` is NOT model-visible in the
|
||||
launch result; self-report is the only observable channel), don't re-cover your own ground
|
||||
before delegating (cluster 3). Delegation thresholds unchanged. Cache refreshed via
|
||||
`bin/refresh-plugins`. Deterministic env-override detection (warn on
|
||||
`CLAUDE_CODE_SUBAGENT_MODEL`) is delegated to WS3's status-check design.
|
||||
- Eval harness E1–E3 (2026-07-06): `plugins/os-orchestration/eval/` — scripted headless
|
||||
orchestration-behavior eval scored by the audit extractor (extended with per-segment read
|
||||
targets). 6 run-set scenarios (E1P/N downgrade detection with
|
||||
`CLAUDE_CODE_SUBAGENT_MODEL=haiku` runner stub; E2P/N explicit-model fan-out vs
|
||||
no-delegation-on-trivial; E3P/N delegate-investigation vs orienting-reads-correct) + 6
|
||||
frozen reserve twins; new Node.js "relaystation" fixture with deterministic sandbox-time
|
||||
log generation; deterministic-first checker (`eval/bin/check`, narrow frozen haiku-judge
|
||||
fallback for E1 language axes only, stubbable via `ORCH_EVAL_JUDGE_CMD`); model-free
|
||||
`bin/self-test` 21/21 green incl. shipped-instruction-compliant conformance transcripts.
|
||||
FAIL axes map only to verified audit misses; tier choice informational. **Scenario Task
|
||||
blocks are held-out — never run informally.** Procedure:
|
||||
`plugins/os-orchestration/eval/README.md`.
|
||||
- Baseline grid run (2026-07-06, run-set × sonnet/haiku × 3 reps + 2 counted canaries, all
|
||||
counted, zero harness errors): negatives 18/18 PASS both tiers (zero over-delegation, zero
|
||||
false downgrade claims); positives 1/18 on the delegate-at-all axis — but transcript
|
||||
verification shows most positive FAILs are *superior direct strategies* (E2P sonnet: one
|
||||
scripted bulk edit; E3P sonnet: surgical greps over 400KB logs), which the WS1 audit
|
||||
itself scored as justified non-delegation, vs genuine miss shapes (E3P haiku: whole 112KB
|
||||
logs into own context; E2P haiku: 20-call per-file grind). Conditional rules validated
|
||||
where engaged: 14/14 spawns explicit-model; the one delegating E1P rep flagged the stubbed
|
||||
downgrade via self-report. Criterion redesign for E2P/E3P A-axes pre-registered (anchor
|
||||
thresholds independently, NOT to this grid) before any rescore/tuning; reserve stays
|
||||
frozen. ORCHESTRATION.md gained a scripted-bulk-edit carve-out. TSV:
|
||||
`eval/results/2026-07-06-baseline-grid.tsv`; full analysis: vault note
|
||||
[[os-orchestration-eval-baseline-grid-results]].
|
||||
- WS4 economics loop COMPLETE + SHIPPED (2026-07-08; run-set AND reserve now contaminated
|
||||
for future wording tuning): ORCHESTRATION.md v3 shipped — cost-asymmetry +
|
||||
tier-conditional threshold opener, symmetric delegate-when/work-directly-when trigger
|
||||
lists (replaces "delegate only when"), batching rule (~5–8 items per grouped spawn,
|
||||
SendMessage reuse), async rule, Workflow effort dial; explicit-model/self-report/
|
||||
don't-re-cover-ground rules unchanged. Harness extended first (commit b226601): real token
|
||||
econ axes — subagent transcripts persist at
|
||||
`<projects>/<flat-cwd>/<session-id>/subagents/agent-*.jsonl` with per-message usage blocks
|
||||
(solves the mini-audit's instrumentation gap; vault note
|
||||
[[claude-code-subagent-transcripts-and-token-accounting]]) — E5 batching pair + reserve
|
||||
twins (authored by subagent, held-out content never entered the tuning session), fable
|
||||
column, pre-registered E2P/E3P criterion redesign landed (scripted-direct PASS branch;
|
||||
74KB ingestion anchor + id-keyed root-cause concepts). Results: E5P 0/6 → majority-PASS
|
||||
both tiers incl. reserve twin 5/5 (fable median mltok 43% of baseline, H3 met); negatives
|
||||
clean everywhere (36/36 across confirm+reserve). Residual gaps: E1P sonnet delegate-at-all
|
||||
(fixture too small — needs bigger fixture before meaningful), fable downgrade-flag ~1/4
|
||||
judge flicker. 15 baseline reps were session-limit-truncated and mechanically
|
||||
excluded+replaced (limit-banner sweep now a standard step, promoted to the autoresearch
|
||||
howto). TSVs: `eval/results/2026-07-08-*.tsv`; loop log:
|
||||
`autoresearch/classic-260708-1039/`; vault note
|
||||
[[os-orchestration-ws4-econ-wording-results]]. IRL re-audit of ~5 real Fable sessions
|
||||
scheduled ~2026-07-22.
|
||||
|
||||
**Global os-status plugin** — `cc-os/plugins/os-status/` (git-tracked, 2026-07-06);
|
||||
symlinked into `~/.claude/plugins/os-status`
|
||||
- Purpose: aggregated deterministic SessionStart status checks for the cc-os plugin family —
|
||||
per-project artifacts (ADR system, vault hub note) and environment hazards (subagent model
|
||||
env override). Built via OpenSpec change `add-os-status-plugin` (supersedes the WS3 plan
|
||||
doc); decision record ADR-022.
|
||||
- Architecture: checks are **in-process** plain Python functions in `hooks/checks.py` with
|
||||
the uniform signature `check(ctx) -> CheckResult(status, message)` plus one registry entry
|
||||
(no per-plugin subprocess protocol, no per-project code copies — both rejected in ADR-022
|
||||
after perspective review). Three states: `ok` (silent), `note` (near-zero-token
|
||||
additionalContext line, never snoozed/suppressed), `warn` (aggregated into at most one
|
||||
banner; once-per-day snooze + permanent `suppress-<check>` markers in gitignored
|
||||
per-project `.cc-os/` — state only, never code; `.cc-os/config` holds optional
|
||||
`hub`/`vault_path` key=value overrides).
|
||||
- Initial checks: `subagent-model-env-override` (env + `~/.claude/settings.json` `env`
|
||||
block; runs outside git projects too — catches the WS1 Cluster 1 incident deterministically
|
||||
at session start), `adr-system-present` (verbatim port of os-adr's hook behavior incl.
|
||||
`.os-adr/suppress`), `vault-hub-note-present` (config slug first, else `type/hub` +
|
||||
`project/<name>` facet-tag scan of the vault; missing → warn naming `/os-vault:write`).
|
||||
- Tests: `python3 tests/hook_test.py` (36, model-free) incl. byte-identity of
|
||||
PRESENT_NOTE/ABSENT_NOTE against os-adr's source. Invariants in `invariants.md`.
|
||||
Smoke-tested 2026-07-06 (cache==source, real headless session fired the hook,
|
||||
env-override canary warns).
|
||||
|
||||
**Global os-doc-hygiene plugin** — `cc-os/plugins/os-doc-hygiene/` (git-tracked,
|
||||
2026-07-03); symlinked into `~/.claude/plugins/os-doc-hygiene`
|
||||
- Hooks: `hooks/hooks.json` → SessionStart hook (matcher: startup|resume) runs
|
||||
`scripts/reminder.py` via `${CLAUDE_PLUGIN_ROOT}` (5s timeout), emitting a deterministic
|
||||
zero-token reminder banner
|
||||
- Behavior: Monitors stale and bloated project documentation per-project under
|
||||
`.dochygiene/` state dir (gitignored). SessionStart reminder is deterministic (no AI
|
||||
tokens, once/day snooze). Skills (verb-first, no `commands/` dispatcher — invoked directly
|
||||
as `/os-doc-hygiene:<skill>`, per [[cc-os-plugin-skill-naming-convention]]): `check`
|
||||
(AI-assisted classification of staleness signals, emits machine+human reports), `clean`
|
||||
(AI-assisted or deterministic patch application with git-safe scoped cleanup), `status`
|
||||
(read-only lifecycle-timestamp read), `sweep` (check then clean in sequence).
|
||||
Reversion-protected via invariants.md + golden-example test fixtures.
|
||||
- Migration: migrated from standalone repo (`~/dev/cc-plugins/doc-hygiene/`, 2026-07-03) and
|
||||
integrated into cc-os. No content conflicts detected. Renamed from `doc-hygiene` to
|
||||
`os-doc-hygiene` per cc-os plugin naming convention. Skills renamed
|
||||
`hygiene-check`/`hygiene-clean` → `check`/`clean`, and the `commands/hygiene.md`
|
||||
dispatcher removed in favor of two new skills (`status`, `sweep`), aligning with the
|
||||
`os-vault`/`os-orchestration` pattern of no `commands/` directory (2026-07-03).
|
||||
|
||||
**Global os-adr plugin** — `cc-os/plugins/os-adr/` (git-tracked, 2026-07-03); symlinked into
|
||||
`~/.claude/plugins/os-adr`
|
||||
- Purpose: Architecture Decision Records, consistently in every project — one
|
||||
customized-Nygard template (frontmatter
|
||||
`id`/`date`/`status`/`supersedes`/`superseded-by`/`affected-paths`/`affected-components`;
|
||||
Context/Decision/Consequences/Alternatives-rejected sections),
|
||||
`docs/adr/NNNN-kebab-title.md` one-file-per-decision + fully-regenerated
|
||||
`docs/adr/README.md` index. Built from `docs/adr-system/` (requirements `04`, PRD `05`,
|
||||
eval sketches `06`) via OpenSpec change `add-os-adr-plugin`.
|
||||
- Core: Ruby OO library `lib/adr/` (`Record`, `Repository`, `Index`, `Template`, `Detector`,
|
||||
`Migrator`, `MigrationReport`, `Finder`; Sandi Metz style, injected paths) behind thin
|
||||
CLIs in `bin/` (`adr-new`, `adr-init`, `adr-detect`, `adr-migrate`, `adr-find`). Tests:
|
||||
`ruby tests/all.rb` (47) + `python3 tests/hook_test.py` (15), all model-free. Behavioral
|
||||
invariants in `invariants.md`.
|
||||
- Hook: **moved to os-status (2026-07-06)** — the SessionStart existence check now runs as
|
||||
the `adr-system-present` check in the os-status plugin (`hooks.json` here is empty;
|
||||
`hooks/session_start.py` stays as the wording source of record, byte-copied into
|
||||
os-status). Behavior unchanged: present → near-zero-token usage note naming
|
||||
`/os-adr:create` + `/os-adr:find`; absent → once-per-day init/migrate suggestion, silenced
|
||||
permanently by `.os-adr/suppress` (still honored); silent outside git projects.
|
||||
- Skills (verb-first, no `commands/`): `create` (one-invocation templated+numbered+indexed
|
||||
ADR with mechanical supersession), `init`, `migrate` (non-destructive: detect surveyed
|
||||
shapes → mechanical heuristic fill → LLM fills only manifest-listed interpretive fields
|
||||
via `--apply-fills` → `migration_confidence` frontmatter + `docs/adr/migration-report.md`;
|
||||
old-system deletion only as an explicit separate user-approved step), `find`
|
||||
(deterministic-first retrieval: affected-paths/components match → Accepted-only filter →
|
||||
one-hop Graphify query-path expansion with graceful degradation → AI judgment over the
|
||||
candidate set only).
|
||||
- Migration pilot passed 2026-07-03 (sandboxed project copies; gate threshold 25%
|
||||
low-confidence flags): viking-warrior-training-log 0%, delta-refinery 0%, llf-schema 8.3%.
|
||||
Retrieval verified end-to-end against the real llf-schema project graph. Resolved: ADRs
|
||||
need no Graphify tag/edge convention — retrieval layer 3 expands the *query* paths via
|
||||
graph-node `source_file`s.
|
||||
- Eval A harness (2026-07-03, ADR-021): `plugins/os-adr/eval/` — prompted skill-execution
|
||||
eval across model tiers (haiku/sonnet). Two fixtures (generated with the plugin's own
|
||||
CLIs), six scenarios (S1 create, S2 create+supersede, S3 find/conflict, S4
|
||||
find/distractor, S5 init, S6 migrate+fills), deterministic Ruby checker (`eval/bin/check`,
|
||||
structural invariants, TSV mode for autoresearch), sandbox + headless-runner scripts,
|
||||
runner-prompt template. Primary run mode: in-session Agent-tool subagents with pinned
|
||||
`model:`; optimized via the `/autoresearch` Classic loop over SKILL.md *wording only*
|
||||
(checker/fixtures/scenarios frozen during a loop). Procedure:
|
||||
`plugins/os-adr/eval/README.md`. Self-tested both directions; grid run 2026-07-03, all
|
||||
scenarios × both tiers passing. Distinct from the held-out Eval B (unprompted behavior) —
|
||||
do not conflate.
|
||||
- Eval B harness (2026-07-03, OpenSpec change `add-os-adr-eval-b-harness`):
|
||||
`plugins/os-adr/eval-b/` — held-out unprompted-behavior eval (requirements 4–5). 7
|
||||
scenarios (W1–W3 write-trigger, R1–R4 retrieval) authored from the frozen shapes in
|
||||
`docs/adr-system/06-eval-scenarios.md`; dedicated Ruby webhook-relay fixture with a 6-ADR
|
||||
history (Superseded pair + near-miss distractors, generated via the plugin's own CLIs);
|
||||
R4's one-hop graph reach uses a real `graphify update` AST build (model-free, rebuilt via
|
||||
`eval-b/bin/build-fixture-graph`, never committed). **Headless-only runner**
|
||||
(`eval-b/bin/run` — fresh `claude -p` per rep, cwd = sandbox, so the real SessionStart
|
||||
hook fires; in-session subagents are invalid here, unlike Eval A) and a two-axis
|
||||
deterministic-first checker (`eval-b/bin/check`): axis (a) unprompted consultation,
|
||||
mechanical from transcript tool_use blocks; axis (b) correct-ADR citation (R1–R4) or
|
||||
new-ADR-file with a narrow frozen-rubric haiku judge fallback (W1–W3, `judge-rubric.md`,
|
||||
stubbable via `ADR_EVAL_B_JUDGE_CMD`). `R4-nograph` is the graph-degradation variant
|
||||
(expected FAIL). Self-tested both directions model-free via `eval-b/bin/self-test`.
|
||||
**Scenario Task blocks are held-out — never run them informally.** Procedure:
|
||||
`plugins/os-adr/eval-b/README.md`. Grid run 2026-07-03 (1 rep/cell): haiku 0/8 PASS;
|
||||
sonnet 5/8 PASS (fails W3, R1). `R4-nograph` FAILed on both tiers as expected. Full
|
||||
results + observations: vault note [[os-adr-eval-b-grid-results-and-observations]].
|
||||
- **Initial iteration complete (2026-07-03):** plugin build, migration pilot, Eval A grid
|
||||
(clean pass), and Eval B grid (baseline) all done — closes the first pass on os-adr.
|
||||
- **Wording experiment complete (2026-07-04):** the follow-up `/autoresearch` loop over
|
||||
Eval B wording ran (5 iterations, checker/fixtures/scenarios/rubric frozen; fixture
|
||||
CLAUDE.md declared a wording surface upfront) and closed the gap — final full grid
|
||||
**sonnet 8/8, haiku 7/8** (from 5/8 / 0/8 baseline; haiku's one miss is a W3 axis-b
|
||||
judge-boundary flicker). Winning wording shipped in `hooks/session_start.py`
|
||||
(PRESENT_NOTE), find/create SKILL.mds, and `eval-b/fixture/project/CLAUDE.md` (new — the
|
||||
trigger-phrased "Architecture decisions" section, the template for real-project adoption).
|
||||
Confirmed mechanisms: trigger-conditioned when→then phrasing beats inventory phrasing on
|
||||
both tiers; each rule must live where its precondition is visible (the reversal→supersede
|
||||
rule in the find skill's act-on-findings step fixed W3); lower tiers need *mechanical*
|
||||
triggers — semantic triggers only reach sonnet. Open: channel ablation never run;
|
||||
R4-nograph now passes both tiers, so the graph-degradation check no longer differentiates.
|
||||
Full hypothesis→result mapping: vault note [[os-adr-eval-b-wording-experiment-hypotheses]].
|
||||
**Before designing or running any autoresearch eval, Read
|
||||
`~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md`.**
|
||||
- **Eval B W3 stability check (2026-07-06):** 7 headless reps of haiku × W3 (all reps
|
||||
counted): axis (a) consultation 7/7 PASS; axis (b) recording-offer ~5/7 (~75–85%) — one
|
||||
FAIL plausibly infra, one a genuine behavioral miss. Verdict: intermittent flicker, not a
|
||||
hard gap — grid claim stays **haiku 7/8**, W3 characterized as an ~1-in-4/5 axis-b miss.
|
||||
Haiku cleared for daily use with that caveat. Recorded in the vault good-enough gate:
|
||||
[[os-adr-eval-b-wording-experiment-hypotheses]].
|
||||
- **Eval C harness built, NOT yet run as designed (2026-07-06):** `plugins/os-adr/eval-c/` —
|
||||
ambiguity-ladder DISCRIMINATION eval (held-out; Eval B is contaminated by the wording
|
||||
loop). 3 levels (explicit → moderate → conceptual framing) × paired positive/negative
|
||||
scenarios; run-set (6, job-execution domain) + frozen reserve-set (6, notifications
|
||||
domain). New Python fixture (`taskq` async job queue, 6-ADR history generated via the
|
||||
plugin's own CLIs, trigger-phrased CLAUDE.md section copied from eval-b). Scoring:
|
||||
positives = both axes; negatives FAIL **only on unneeded ADR creation**. Model-free
|
||||
`bin/self-test` green, including a truthful-citation-must-PASS guard. **Scenario Task
|
||||
blocks are held-out — never run informally.** Methodology: vault notes
|
||||
[[eval-methodology-ladder]] and [[eval-methodology-irl-feedback-loop]]; vault also gained
|
||||
an `eval-results` note type + `_templates/eval-results.md` (2026-07-06).
|
||||
- **Eval C frozen grid run (2026-07-06):** run-set × {sonnet, haiku} × 3 reps/cell (36
|
||||
headless reps, all counted) — **12/12 cells PASS at ≥2/3 majority; haiku 18/18 reps,
|
||||
sonnet 17/18**. Zero over-trigger ADR creation on any negative at any level; every
|
||||
positive pass was via governing-ADR citation (16) or judge-recognized proposal (1), never
|
||||
duplicate creation. Sonnet's one FAIL rep (P1-L1) is a final-message-citation boundary
|
||||
artifact, not a behavioral miss. Two harness defects fixed on the first live rep, BEFORE
|
||||
the grid, with Task blocks/fixtures/rubric untouched: (1) `bin/run` pipefail bug; (2)
|
||||
positive axis-b contradiction fixed via a mechanical `cited-governing` PASS branch + two
|
||||
new self-test guards (`B-via:` now recorded in TSV). Verdict: the Eval B wording
|
||||
**generalizes** — grid saturated, ceiling not found; next signal is production (rollout +
|
||||
IRL session audits), not another lab rung. Run-set is now the baseline; wording tuning
|
||||
against it moves measurement to the reserve-set. Full results: vault note
|
||||
[[os-adr-eval-c-frozen-grid-results]].
|
||||
- **Remaining (locked rollout order):** real-project migration/adoption one at a time via
|
||||
`/os-adr:migrate` — pilot projects first, then the cc-os retrofit (its 19-ADR monolithic
|
||||
file was deliberately excluded from the pilot), then wider. When onboarding real projects,
|
||||
add the trigger-phrased "Architecture decisions" CLAUDE.md section (copy from
|
||||
`eval-b/fixture/project/CLAUDE.md`; candidate: emit it from `/os-adr:init`/`migrate`).
|
||||
|
||||
**Graphify** — v0.8.31 at `/home/jared/.local/bin/graphify`
|
||||
- Vault graph: `~/Documents/SecondBrain/graphify-out/` — disposable, structural index
|
||||
rebuilt by SessionStart hook; handles relational/graph queries. Distinct from
|
||||
vault-index.json (planned, Plan B Phase 1): a flat `{tag: [{path, title, summary}]}`
|
||||
lookup rebuilt at SessionEnd, queried by the `/memory-find` skill for fast tag-based SB
|
||||
note discovery.
|
||||
- Project graph: `<project-root>/graphify-out/` — same pattern; gitignore it in each project
|
||||
repo
|
||||
- Vault conventions: `~/Documents/SecondBrain/vault-conventions.md` — frontmatter contract +
|
||||
tag taxonomy (canonical name decided 2026-06-09; formerly referred to as CONVENTIONS.md)
|
||||
|
||||
**memsearch** — v0.4.6 via Claude Code plugin marketplace (`memsearch@memsearch-plugins`)
|
||||
- Hooks: Stop, SessionStart, UserPromptSubmit, SessionEnd (ship with plugin)
|
||||
- Memory store: `~/.memsearch/memory/YYYY-MM-DD.md` daily files (global, cross-project, all
|
||||
clients — one store by design)
|
||||
- Index: `~/.memsearch/milvus.db` (Milvus Lite, local); embeddings via ONNX bge-m3
|
||||
- Config: `MEMSEARCH_DIR=~/.memsearch` in `~/.zshrc` for global scope
|
||||
- Skills: `/memory-recall`, `/memory-config` (ship with plugin)
|
||||
- Git sync: `~/.memsearch` is a dedicated private Forgejo repo
|
||||
(`forgejo.swansoncloud.com/jared/memsearch`); whitelist `.gitignore` commits only
|
||||
`memory/*.md`; auto-commit+push wired into `memsearch_sync.py` SessionEnd hook (relocated
|
||||
from `session-end.sh` by ADR-016, 2026-06-12; behavior preserved — see ADR-015)
|
||||
|
||||
**Obsidian vault git sync** — `~/Documents/SecondBrain` (2026-06-15)
|
||||
- Remote: `ssh://git@forgejo.swansoncloud.com:2222/jared/SecondBrain.git` (private Forgejo;
|
||||
web: `https://forgejo.swansoncloud.com/jared/SecondBrain`)
|
||||
- 52 files tracked (notes, journal, templates, vault-conventions.md, CLAUDE.md, .obsidian
|
||||
config); `graphify-out/` excluded via `.gitignore`
|
||||
- Auto-commit+push via `vault_sync.py` SessionEnd hook (mirrors `memsearch_sync.py`;
|
||||
push-only — SessionStart pull is optional future item for multi-machine)
|
||||
|
||||
## Operational procedures
|
||||
|
||||
### Renaming or moving a local plugin
|
||||
|
||||
Renaming a plugin directory under a directory-source marketplace (like `local-plugins`, whose
|
||||
source is `~/.claude/plugins` itself) requires updating **three** places, not just the
|
||||
directory and `settings.json`:
|
||||
|
||||
1. **Plugin directory / symlink** — the actual files (or symlink target, for plugins that
|
||||
live in a git repo like `cc-os/plugins/os-vault/`).
|
||||
2. **Marketplace manifest** — `~/.claude/plugins/.claude-plugin/marketplace.json` (for
|
||||
`local-plugins`) lists each plugin's `name` and `source` path explicitly; this does
|
||||
**not** auto-discover from the directory listing. Update the entry to match the new
|
||||
name/path.
|
||||
3. **`settings.json` `enabledPlugins`** — the `<name>@<marketplace>` key must match the
|
||||
manifest entry from step 2.
|
||||
|
||||
Then refresh the plugin manager's state so it re-resolves the marketplace and drops the
|
||||
stale install record:
|
||||
|
||||
```bash
|
||||
claude plugin marketplace update <marketplace-name> # re-validates the manifest
|
||||
claude plugin install <new-name>@<marketplace-name> # creates the new install record
|
||||
claude plugin uninstall <old-name>@<marketplace-name> # drops the stale one
|
||||
claude plugin list # verify: new name enabled, old name gone
|
||||
claude plugin details <new-name>@<marketplace-name> # verify skills/agents/hooks resolved
|
||||
```
|
||||
|
||||
Skipping step 2 is the failure mode to watch for: hooks (wired by absolute path in
|
||||
`settings.json`) keep working, masking the fact that skills/slash-commands silently stopped
|
||||
registering under the plugin's new name.
|
||||
|
||||
### Editing a local plugin (cache refresh)
|
||||
|
||||
Plugin source edits (including SKILL.md wording, hook scripts, or CLI code) do not reach
|
||||
running sessions until the cache is refreshed. Claude Code caches plugins when installed,
|
||||
copying them to `~/.claude/plugins/cache/<marketplace>/<plugin>/<version>/`. To refresh a
|
||||
stale cache after source edits:
|
||||
|
||||
```bash
|
||||
bin/refresh-plugins [marketplace-names...] # refresh all or specific local marketplaces
|
||||
```
|
||||
|
||||
This uninstalls and reinstalls each plugin, forcing the cache to repopulate. See
|
||||
`bin/refresh-plugins --help` for usage. After refreshing,
|
||||
`claude plugin details <name>@<marketplace>` should show updated skill descriptions and hook
|
||||
counts. Directory-source marketplaces (e.g., `local-plugins` sourced from
|
||||
`~/.claude/plugins`, `cc-plugins` sourced from `~/dev/cc-plugins`) are the only ones that
|
||||
need manual refresh — public marketplaces auto-update.
|
||||
|
|
@ -617,6 +617,20 @@ _Date: 2026-07-03_
|
|||
- **Cross-references**: `plugins/os-adr/eval/README.md` (procedure + autoresearch invocation), `docs/adr-system/06-eval-scenarios.md` (Eval B sketches, now noting the model axis), ADR-020 (pilot gate the S6 fixture mirrors).
|
||||
- **Status**: Accepted 2026-07-03. Harness built and self-tested (all six scenarios: perfect-run PASS, untouched-sandbox FAIL). The grid itself has not yet been run against haiku/sonnet.
|
||||
|
||||
## ADR-022 — os-status plugin: aggregated in-process SessionStart status checks
|
||||
|
||||
_Date: 2026-07-06_
|
||||
|
||||
- **Context**: Two real incidents showed that cc-os plugins have per-project and per-environment preconditions nothing verifies: (1) `CLAUDE_CODE_SUBAGENT_MODEL=haiku` in `~/.claude/settings.json` silently downgraded all 23 verified subagent spawns found by the WS1 orchestration audit — and the E1 canary proved the downgrade is unobservable to the model at spawn time, so protection must be deterministic; (2) projects routinely lack artifacts plugins assume (ADR system, vault hub note), with each plugin inventing its own SessionStart reminder in isolation (os-adr, os-doc-hygiene) or having none. The WS3 plan (`docs/plans/ws3-status-convention-plugin.md`) proposed a per-plugin status-check convention plus a glue plugin; two perspective reviews (simplifier, implementer, 2026-07-06) pressure-tested where check code should live.
|
||||
- **Decision**: New global plugin **`os-status`** (`plugins/os-status/`), one SessionStart hook running all checks **in-process**: plain Python functions in `hooks/checks.py` with the uniform signature `check(ctx) -> CheckResult(status, message)` and a single registry list. Three result states: `ok` (silent), `note` (near-zero-token line, never snoozed/suppressed — exists so os-adr's Eval-B-tuned usage note survives conversion verbatim), `warn` (aggregated into at most one banner; once-per-day snooze + permanent suppress in a gitignored per-project `.cc-os/` dir — state only, never code). Initial checks: `subagent-model-env-override` (env + settings.json `env` block; runs outside git projects too), `adr-system-present` (verbatim port of os-adr's hook, honoring legacy `.os-adr/suppress`), `vault-hub-note-present` (config-first slug in `.cc-os/config`, else `type/hub` + `project/<name>` facet-tag scan of the vault). os-adr's own SessionStart hook was removed atomically in the same change, with the cutover verified against the refreshed plugin cache.
|
||||
- **Alternatives rejected**:
|
||||
- **Option A — per-project code copies** (`.cc-os/status.d/*.py`): copies drift; the 2026-07-04 stale-cache incident is exactly this failure class; adds a per-project install step.
|
||||
- **Option B as sketched — per-plugin scripts + subprocess/JSON enumeration**: overbuilt for three trivially cheap probes with one author in one repo (spawn/parse/timeout/error-normalization protocol for three registrants); and enumeration from plugin caches would silently run stale code — os-vault is wired by absolute path to git source and its cache copy is already stale (implementer finding). The uniform function signature is kept as the seam so extraction to per-plugin checks later is mechanical.
|
||||
- **Blanket silent-on-ok**: would regress os-adr's present-state usage note; hence the third `note` state.
|
||||
- **Deferred**: `context` injection field (no consumer yet), consolidation of `.os-adr/`/`.dochygiene/` state dirs, conversion of os-doc-hygiene's reminder, Phase 2 cache-freshness "collection manager" duties, subprocess enumeration.
|
||||
- **Cross-references**: OpenSpec change `add-os-status-plugin` (proposal/design/specs/tasks), `plugins/os-status/invariants.md`, WS1 findings (`docs/orchestration-audit/2026-07-06-findings.md`), ADR-018 (marketplace-manifest procedure followed for install).
|
||||
- **Status**: Accepted 2026-07-06. Built, tested (36 model-free tests), installed, smoke-tested (including the regression canary: exported `CLAUDE_CODE_SUBAGENT_MODEL=haiku` produces the override warn).
|
||||
|
||||
## Rejected tools (summary)
|
||||
|
||||
| Tool | Why rejected for our use |
|
||||
|
|
|
|||
|
|
@ -0,0 +1,175 @@
|
|||
# Fable-5 main-loop orchestration economics — mini-audit (2026-07-08)
|
||||
|
||||
**Model**: this audit was performed by `claude-sonnet-5`. All audited transcripts carry
|
||||
`"model":"claude-fable-5"` on non-sidechain assistant messages (i.e. Fable 5 is the main-loop
|
||||
model in every session below).
|
||||
|
||||
## Method
|
||||
|
||||
1. Scanned all `~/.claude/projects/*/*.jsonl` files modified in the last 10 days
|
||||
(`scan.rb`), filtered to sessions with ≥20 non-sidechain assistant turns and at least one
|
||||
`claude-fable-5` main-loop message, excluded the live session
|
||||
(`ca17e6d2-6132-4681-8a02-8a7523cb5660`). 21 candidates found, almost all in `cc-os`; picked
|
||||
6 for project diversity: 3 `cc-os`, 2 `servers`, 1 `servers/desktop`, all ≥20 turns (24–255).
|
||||
2. Ran the existing fact-sheet extractor
|
||||
(`plugins/os-orchestration/audit/bin/extract`) on each for spawn count/models,
|
||||
pre-/post-spawn tool segmentation, and its heuristic same-tool-run flags.
|
||||
3. Wrote two throwaway Ruby scripts in scratchpad (`tokens.rb`, `bashcmds.rb`,
|
||||
`firstprompt.rb`) to sum `usage.output_tokens` per session (main-loop vs sidechain), track
|
||||
max/final context size, and find the longest unbroken run of main-loop tool calls with no
|
||||
`Agent` spawn in between, then classified each run by reading the actual tool inputs
|
||||
(Bash commands / file targets) in that range.
|
||||
|
||||
**Environment gotcha (noted for future audits):** in zsh, a variable named `path` is linked to
|
||||
`$PATH`; `path=$(find ...)` silently corrupted `$PATH` mid-script and caused unrelated `ruby: not
|
||||
found` (exit 127) failures. Renamed to `tpath`. Not a data issue, just a shell trap.
|
||||
|
||||
**Hard measurement limitation:** none of the 6 transcripts contain any `"isSidechain":true`
|
||||
lines, even in the two sessions that used the `Agent` tool (9f45afcc, 5 spawns; df12b180, 1
|
||||
spawn). Subagent execution in this environment/version is logged only as a `tool_use`/
|
||||
`toolUseResult` pair (with `agentId`/`resolvedModel` metadata) plus an eventual
|
||||
`<task-notification>` result — no subagent transcript is persisted into the parent `.jsonl`. So
|
||||
**subagent output-token spend is not recoverable from these files at all.** The only proxy
|
||||
available is the char length of the `tool_result`/notification payload returned to the main
|
||||
loop, converted at ~4 chars/token. All "% main-loop" figures below use that proxy where spawns
|
||||
occurred, and are explicitly not real subagent token counts (real subagent spend is very likely
|
||||
much higher than the proxy, since it only reflects what was returned, not what the subagent
|
||||
itself burned getting there).
|
||||
|
||||
> **Addendum (2026-07-08, WS4 Step 2):** the limitation above is narrower than stated —
|
||||
> subagent spend is absent from the *parent* transcript, but each subagent's full transcript
|
||||
> (with per-message `usage` blocks) persists separately at
|
||||
> `~/.claude/projects/<flattened-cwd>/<session-id>/subagents/agent-<id>.jsonl` (verified for
|
||||
> both interactive and headless sessions). Real main-loop-share measurement is therefore
|
||||
> possible; the eval harness A-econ axis uses it. The proxy figures below were NOT recomputed.
|
||||
> See vault note [[claude-code-subagent-transcripts-and-token-accounting]].
|
||||
|
||||
## Per-session detail
|
||||
|
||||
### 1. `9f45afcc` — cc-os — cache-refresh tooling + WS2 autoresearch prep
|
||||
- 255 turns, 25 human prompts, 850 jsonl lines, 2026-07-04 16:17–18:52
|
||||
- 5 Agent spawns, all `general-purpose`, `model` param `sonnet`×4/`haiku`×1, **resolved model
|
||||
`claude-haiku-4-5` in all 5 cases** (param mismatch: 4 of 5 requested sonnet, all resolved
|
||||
haiku)
|
||||
- main-loop output tokens: **237,679** (255 msgs); proxy subagent output: 13,575 chars via
|
||||
result/notification payloads (≈3.4K tokens) → **main-loop ≈98.6% of proxy-measured output**
|
||||
- pre-spawn segment: 19 calls / 23,658 bytes read (Bash 14, Read 4, Edit 1)
|
||||
- longest direct run: **48 calls** (after spawn 3: ToolSearch 2, SendMessage 1, Bash 22, Edit 12,
|
||||
Skill 2, Read 5, Write 3, TaskStop 1) — mixed: driving/monitoring the background eval-grid
|
||||
agent (SendMessage/TaskStop/ToolSearch) interleaved with direct mechanical plugin-cache-refresh
|
||||
edits. Not a clean single-purpose run.
|
||||
|
||||
### 2. `eba26343` — cc-os — WS2 os-vault wording loop (`/autoresearch`)
|
||||
- 223 turns, 2 human prompts (127 total `type:user` lines incl. tool results/notifications), 583
|
||||
jsonl lines, 2026-07-07 16:48–18:01
|
||||
- **0 Agent spawns**
|
||||
- main-loop output tokens: **216,019** (223 msgs) — 100% of measured output (no spawns to split
|
||||
against)
|
||||
- whole-session tool profile: 119 calls / 119,587 bytes (Read 17, Bash 64, Skill 1, ToolSearch 1,
|
||||
TaskCreate 5, TaskUpdate 9, Edit 20, Write 2)
|
||||
- longest direct run: **119 calls = the entire session** (no spawn boundary at all). Sampled the
|
||||
64 Bash commands directly: this is a scripted headless-runner loop — `bin/run P1/P2/P3...` /
|
||||
`bin/check` invocations against `eval/scenarios/`, `sleep N; wc -l results.tsv`
|
||||
polling of background task output, `awk`/`sort` aggregation of `results.tsv`, and git
|
||||
commit/push at each iteration boundary. **Classify as justified scripted-bulk-op / polling
|
||||
work**, not a delegation miss — matches the ORCHESTRATION.md scripted-bulk-edit carve-out; a
|
||||
subagent adds no value polling a script's own stdout.
|
||||
|
||||
### 3. `5f53e0c0` — cc-os — `/opsx:apply add-os-adr-plugin`
|
||||
- 222 turns, 4 human prompts, 506 jsonl lines, 2026-07-03 17:15–17:43
|
||||
- **0 Agent spawns**
|
||||
- main-loop output tokens: **314,899** (222 msgs, highest of the 6) — 100% of measured output
|
||||
- whole-session tool profile: 128 calls / 129,924 bytes (Bash 39, Read 16, Write 35, Edit 38)
|
||||
- longest direct run: **128 calls = the entire session**. Contains three flagged sub-runs from
|
||||
the extractor: Read×10 (design docs, lines 27–52 — reasonable orientation), **Write×13 (lines
|
||||
96–137, 13 distinct new files: `lib/adr.rb`, `lib/adr/record.rb`, `repository.rb`,
|
||||
`index.rb`, `template.rb`, `bin/adr-new`, …)**, and Edit×10 (lines 326–353, fixing 5 files
|
||||
post-review). **Genuine miss candidate**: the Write×13 run is scaffolding independent Ruby
|
||||
class files against an *already-fixed* `design.md` (read in the first 10 calls) — a
|
||||
parallelizable implementation task that ran entirely in the main loop and is a large share of
|
||||
this session's 315K output tokens.
|
||||
|
||||
### 4. `3fc7bb8c` — servers — backup verification + vault write
|
||||
- 90 turns, 5 human prompts, 236 jsonl lines, 2026-07-04 17:20–18:04
|
||||
- **0 Agent spawns**
|
||||
- main-loop output tokens: **63,607**
|
||||
- whole-session tool profile: 44 calls / 32,250 bytes (Read 3, Bash 27, Write 4, ToolSearch 2,
|
||||
ExitPlanMode 1, Edit 6, Skill 1)
|
||||
- longest direct run: **44 calls = whole session** — interactive backup-system investigation
|
||||
("Are we confident backups will work as expected now?") ending in a vault write. Judgment-
|
||||
heavy, sequential-dependent (each Bash result determines the next check). **Justified direct
|
||||
work**, no clean miss shape.
|
||||
|
||||
### 5. `7bc0dda4` — servers — Proxmox HAOS install troubleshooting
|
||||
- 82 turns, 15 human prompts (highest interaction density of the 6), 249 jsonl lines,
|
||||
2026-07-03 19:49–20:42
|
||||
- **0 Agent spawns**
|
||||
- main-loop output tokens: **39,513**
|
||||
- whole-session tool profile: 39 calls / 6,124 bytes (Bash 27, Edit 8, Read 2, Skill 1, Write 1)
|
||||
- longest direct run: **39 calls = whole session** — live interactive install/debug loop (disk
|
||||
space checks, image download, phase-2 error triage against user-supplied screenshots).
|
||||
**Justified direct work**: 15 human turns in 82 total means the human is in the loop almost
|
||||
every other turn; delegation overhead would exceed any savings.
|
||||
|
||||
### 6. `df12b180` — servers/desktop — Remote Control OAuth scope troubleshooting
|
||||
- 24 turns, 6 human prompts, 90 jsonl lines, 2026-07-06 23:13–23:35
|
||||
- 1 Agent spawn (`claude-code-guide`, `model` param `sonnet`, resolved `claude-sonnet-5`,
|
||||
foreground, prompt 1,008 chars → result 5,689 chars)
|
||||
- main-loop output tokens: **24,051**; proxy subagent output ≈1,422 tokens (5,689 chars/4) →
|
||||
**main-loop ≈94.4% of proxy-measured output**
|
||||
- pre-spawn: 4 Bash calls / 3,154 bytes (credential/scope investigation)
|
||||
- post-spawn: 3 calls (Read 1, Write 2 — writing up the answer)
|
||||
- longest direct run: 4 calls (the pre-spawn investigation). **Good delegation example**: a
|
||||
bounded doc-lookup ("Remote Control usage") was correctly routed to the docs-specialist
|
||||
subagent instead of the main loop reading source/docs itself.
|
||||
|
||||
## Summary table
|
||||
|
||||
| session | project | turns | spawns | main-loop out tok | % main-loop (proxy, see caveat) | pre-spawn calls/bytes | longest direct run | classification |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| 9f45afcc | cc-os | 255 | 5 | 237,679 | ~98.6% | 19 / 23,658 | 48 | mixed: agent-monitoring + mechanical cache-refresh edits |
|
||||
| eba26343 | cc-os | 223 | 0 | 216,019 | 100% | 119 / 119,587 (whole session) | 119 (whole session) | scripted headless-eval polling loop — justified |
|
||||
| 5f53e0c0 | cc-os | 222 | 0 | 314,899 | 100% | 128 / 129,924 (whole session) | 128 (whole session) | OpenSpec apply — Write×13 independent files = **miss candidate** |
|
||||
| 3fc7bb8c | servers | 90 | 0 | 63,607 | 100% | 44 / 32,250 (whole session) | 44 (whole session) | interactive investigation — justified |
|
||||
| 7bc0dda4 | servers | 82 | 0 | 39,513 | 100% | 39 / 6,124 (whole session) | 39 (whole session) | interactive troubleshooting (15 human turns) — justified |
|
||||
| df12b180 | servers/desktop | 24 | 1 | 24,051 | ~94.4% | 4 / 3,154 | 4 | pre-spawn investigation, correct delegation of doc lookup |
|
||||
|
||||
## Caveats
|
||||
|
||||
- 4 of the 6 sessions are from `cc-os` (matches the actual population: 19 of 21 candidate
|
||||
sessions in the last 10 days were `cc-os`) — the sample is not representative of all-client
|
||||
usage, just of what exists in the last 10 days of transcripts.
|
||||
- "% main-loop" is a **proxy** built only from what subagents returned to the main loop
|
||||
(tool_result/notification char count ÷4), not measured subagent token spend — real subagent
|
||||
cost is invisible in every transcript examined (0/6 have `isSidechain:true` lines). Any
|
||||
economics conclusion about spawn cost-effectiveness needs a different data source (e.g. Task
|
||||
output files under `tasks/`, if retained) — not attempted here.
|
||||
- "Longest direct run" for 4 of 6 sessions equals the *entire session* because zero spawns
|
||||
occurred; this collapses "pre-spawn segment" and "longest run" into the same number for those
|
||||
rows — noted in the table rather than hidden.
|
||||
- Miss/justified classification is a single-auditor read of tool-call sequences, not a rubric
|
||||
score; treat as a directional signal for the eval plan, not a verdict.
|
||||
|
||||
## Miss-shape summary
|
||||
|
||||
- **The only clean miss candidate found is `5f53e0c0`'s Write×13 run**: scaffolding independent
|
||||
Ruby class files against an already-fixed design doc, done entirely in the (most expensive)
|
||||
main loop across a 222-turn, 314,899-output-token session with zero spawns.
|
||||
- **Scripted/polling loops are not misses even when very long** (`eba26343`, 119 consecutive
|
||||
direct calls): driving `bin/run`/`sleep`/`wc -l results.tsv` from the main loop is the correct
|
||||
shape per the existing scripted-bulk-op carve-out — a subagent adds no value polling a script's
|
||||
own output.
|
||||
- **Interactive/judgment-heavy sessions correctly stayed direct** (`3fc7bb8c`, `7bc0dda4`): both
|
||||
have no spawns and no flagged miss runs; `7bc0dda4`'s 15 human prompts in 82 turns make
|
||||
delegation overhead a net loss.
|
||||
- **Subagent economics are structurally unmeasurable from transcripts as configured**: 0 of 6
|
||||
sessions logged sidechain entries even where Agent spawned; only a lossy char-count proxy
|
||||
exists via task-notification payloads. This is itself an eval-plan-relevant finding — any eval
|
||||
claiming to measure delegated-vs-direct token spend needs an instrumentation change (capture
|
||||
Task output, or read `tasks/*.output` files) before it can trust a token-economics number.
|
||||
- **Where spawns did happen, the main loop still did substantial direct work afterward**
|
||||
(`9f45afcc`: 43–48-call runs after spawns 3 and 5) rather than delegating a second time —
|
||||
worth checking in future audits whether that's justified (synthesis/judgment) or another miss
|
||||
shape.
|
||||
- **No over-delegation observed anywhere** — zero ≤2-call trivial ops routed to a subagent in any
|
||||
of the 6 sessions.
|
||||
|
|
@ -0,0 +1,62 @@
|
|||
# B2: per-profile plugin-enablement overlays — shipped + measured
|
||||
|
||||
_Task: B2 in [system-prompt-profiles-tasks.md](system-prompt-profiles-tasks.md) ·
|
||||
2026-07-08 · Depends on C3 findings ([c3-spike-findings.md](c3-spike-findings.md))_
|
||||
|
||||
## Pre-check result: `enabledPlugins` DEEP-MERGES
|
||||
|
||||
The open item from C3 is resolved empirically (transcript `skill_listing` inspection, two
|
||||
headless haiku runs): a `--settings <overlay.json>` file's `enabledPlugins` is merged
|
||||
key-by-key into `~/.claude/settings.json` — only the plugins explicitly set `false`
|
||||
disappear (baseline 64 skills → 61, exactly the 3 skills of the 2 disabled plugins);
|
||||
unlisted plugins and all hooks are unaffected. **Diff-only overlays are safe.**
|
||||
|
||||
## Shipped
|
||||
|
||||
`~/.claude/profiles/` (see its README.md):
|
||||
|
||||
- `cc-os-design.json` — disables rails-ui-component, api-wrapper, invoice-ninja,
|
||||
creative-team, dev-team
|
||||
- `client-dev.json` — disables creative-team, invoice-ninja, arch-refactor
|
||||
- `brainstorm.json` — disables all 9 coding/client plugins (adds ruby-lsp, codex,
|
||||
git-context, arch-refactor)
|
||||
|
||||
Usage: `claude --settings ~/.claude/profiles/<name>.json`. Never combine with a
|
||||
`--setting-sources` list that omits `user` (silent plugin/hook drop — see vault note
|
||||
`claude-code-setting-sources-silent-plugin-drop`).
|
||||
|
||||
## Measured savings (headless haiku, empty dir, 2026-07-08)
|
||||
|
||||
| Run | Context tokens | Delta vs baseline |
|
||||
|---|---|---|
|
||||
| baseline (no overlay) | 29,045 | — |
|
||||
| cc-os-design (5 plugins off) | 29,013 | −32 |
|
||||
| brainstorm (9 plugins off) | 28,413 | −632 |
|
||||
|
||||
**Honest verdict: token savings are marginal.**
|
||||
|
||||
## Follow-on: YAGNI tool deny (2026-07-08, same session)
|
||||
|
||||
Transcript sweep of all sessions since 2026-06-01 (tool_use counts across
|
||||
`~/.claude/projects/**/*.jsonl`): heavy use of Bash (7.7k) / Read (4.2k) / Edit (2.1k) /
|
||||
Agent (1.3k) / Write (1.2k); **zero-or-one lifetime calls** to Workflow, Artifact,
|
||||
SendUserFile, ReportFindings — all of which ship full schemas up front (not deferred).
|
||||
|
||||
Measured (headless haiku, empty dir): disallowing Workflow alone −5,982 tokens
|
||||
(its schema ≈ the whole system-prompt text); all four −6,509. A `permissions.deny`
|
||||
block in a settings file removes the schemas identically to `--disallowedTools`
|
||||
(22,532 vs 22,536).
|
||||
|
||||
**Applied globally** to `~/.claude/settings.json`:
|
||||
`"permissions": {"deny": ["Workflow", "Artifact", "SendUserFile", "ReportFindings"]}` —
|
||||
~6.5k saved every session, no flags needed. Revert by deleting the entry (needed if
|
||||
"ultracode"-style Workflow orchestration is ever wanted). This delivers the
|
||||
"`--tools` restriction is the real lever" prediction from the assessment; A1/A2's
|
||||
remaining value is now just the interactive-Fable confirmation. The assessment's "~4.8k user stack"
|
||||
is dominated by hooks/memsearch injection and per-plugin fixed listings, not by the
|
||||
skill descriptions the overlays remove — ToolSearch deferral already made plugin
|
||||
skills cheap. The remaining value of profiles is **focus** (no irrelevant slash
|
||||
commands/agents offered in a brainstorm session), not context cost. This further
|
||||
deprioritizes Wave 3 (aliases: trivial, do when friction is felt) and Group D
|
||||
(dropped), and shifts remaining token-diet attention to the memsearch injection
|
||||
(~1.5–3.9k/session, needs upstream fix — see b3-memsearch-injection-review.md).
|
||||
|
|
@ -0,0 +1,91 @@
|
|||
# B3 — memsearch SessionStart injection cost review
|
||||
|
||||
_Reviewed 2026-07-08. memsearch v0.4.6 (installed) / v0.4.11 (also cached, not yet installed). Plugin: `memsearch@memsearch-plugins`._
|
||||
|
||||
## What gets injected
|
||||
|
||||
`hooks/session-start.sh` (both cached versions, logic unchanged 0.4.6→0.4.11) builds a
|
||||
`# Recent Memory` block on every SessionStart and returns it as `additionalContext`:
|
||||
|
||||
- Finds the daily memory files under `$MEMORY_DIR` (`~/.memsearch/memory/YYYY-MM-DD.md`),
|
||||
sorted by filename descending, and takes the **2 most recent**.
|
||||
- For each of those 2 files, greps lines matching `^(#{2,4} |- )` (headings + bullets) and
|
||||
takes the **first 40 matching lines** (`head -40`).
|
||||
- Concatenates both files' extracts under one `# Recent Memory` heading and returns it.
|
||||
|
||||
This is entirely hardcoded (`head -2`, `head -40`) in the shell script — there is no
|
||||
per-file/per-line/byte cap that's user-configurable.
|
||||
|
||||
## Measurements (cc-os project, last ~7 days of session transcripts)
|
||||
|
||||
Extracted the `additionalContext` string from every SessionStart hook event whose payload
|
||||
contained `# Recent Memory`, across `~/.claude/projects/-home-jared-dev-cc-os/*.jsonl`
|
||||
(61 SessionStart events found in that window — heavy usage during the WS2/WS4 eval loops,
|
||||
so this sample is skewed toward busy days, not a quiet baseline).
|
||||
|
||||
| Stat | chars | approx tokens (÷4) |
|
||||
|---|---|---|
|
||||
| min | 5,976 | ~1,494 |
|
||||
| median | 11,721 | ~2,930 |
|
||||
| avg | 11,978 | ~2,995 |
|
||||
| max | 15,407 | ~3,852 |
|
||||
|
||||
This roughly confirms the ~13 KB figure already observed. Size scales with how many
|
||||
`##`/`###`/bullet lines the 2 most recent daily files accumulated that day — on a day with a
|
||||
lot of sessions/subagent activity (like the recent eval-loop days), each daily file alone can
|
||||
hit the 40-line cap, so the injection saturates near its structural ceiling
|
||||
(2 files × 40 lines × ~150–200 chars/line ≈ 12,000–16,000 chars), matching the observed range.
|
||||
|
||||
Caveat: this is a per-project sample from one heavy-usage repo (cc-os) over 7 days; other
|
||||
projects' daily files will differ, and this sampling method (string-matching the raw jsonl,
|
||||
not a documented API) is the best available without deeper memsearch internals — reasonably
|
||||
solid since the hook's exact output string was recovered from real transcripts, not estimated.
|
||||
|
||||
## Config surface found
|
||||
|
||||
Read `skills/memory-config/SKILL.md` (both cached versions) and `hooks/session-start.sh`
|
||||
directly. Findings:
|
||||
|
||||
- The **documented** TOML config surface (`plugins.claude-code.summarize`, `.project_review`,
|
||||
`.user_profile`, `.memory_to_skill`, `[llm.providers.*]`, `[prompts]`) controls
|
||||
summarization, maintenance artifacts (PROJECT.md/USER.md), skill distillation, and provider
|
||||
routing — **none of it touches the SessionStart recall injection**.
|
||||
- No `[plugins.claude-code.session_start]` or equivalent section exists in the documented
|
||||
schema, and none of the constants used by the injection logic (`head -2` file count,
|
||||
`head -40` line count, the `^(#{2,4} |- )` grep pattern) appear anywhere in `config.toml`,
|
||||
`memsearch config list --resolved/--global/--project`, or the SKILL.md. They are hardcoded
|
||||
in `hooks/session-start.sh` itself.
|
||||
- The only adjacent, real lever is **memory file density**: since the cap is by line count
|
||||
(not token count) over headings+bullets, whatever produces terse vs. verbose daily-file
|
||||
bullets (the Stop-hook summarizer, governed by `plugins.claude-code.summarize.model`,
|
||||
defaults to haiku) indirectly controls how much of the 40-line budget per file is "real
|
||||
signal" vs. how many days of content get squeezed out.
|
||||
|
||||
## Recommendation
|
||||
|
||||
**No first-class cap/trim setting exists in memsearch's config schema for this injection.**
|
||||
The two concrete options, in preference order:
|
||||
|
||||
1. **Workaround, no code change (recommended to try first):** the injection is driven purely
|
||||
by memory-file content density, not by any setting. If daily files are noisy (e.g. many
|
||||
short low-signal bullets from busy multi-session days, as in the cc-os sample), the
|
||||
Stop-hook summarizer producing those bullets is the actual place to add discipline —
|
||||
tightening what counts as bullet-worthy would shrink the 40-line/file budget's real size
|
||||
without touching the plugin. This is a memsearch summarizer/prompt-quality issue, not a
|
||||
session-cost cap.
|
||||
2. **Direct edit, unsupported (only if (1) is insufficient):** hand-edit the cached
|
||||
`hooks/session-start.sh` at
|
||||
`~/.claude/plugins/cache/memsearch-plugins/memsearch/<version>/hooks/session-start.sh`,
|
||||
lowering `head -2` → `head -1` (halves injection size, roughly ~6–8 KB/day instead of
|
||||
~12–15 KB) and/or `head -40` → a smaller value (e.g. 20). Expected savings: cutting to 1
|
||||
file × 20 lines would drop the typical injection from ~11–15 KB (~2,900–3,850 tokens) to
|
||||
roughly ~3 KB (~750 tokens), a ~75% reduction. **Caveat:** this is an unmanaged edit to
|
||||
plugin-cache source — it will be silently overwritten on the next
|
||||
`claude plugin update memsearch` / cache refresh, is not upstream-supported, and isn't a
|
||||
"setting" in the config-surface sense the task asked to identify. Treat it as a stopgap,
|
||||
not a durable fix; if this repo wants a durable cap, the change belongs upstream in
|
||||
`zilliztech/memsearch` (file an issue/PR to add a
|
||||
`[plugins.claude-code.session_start]` `max_files` / `max_lines_per_file` config section)
|
||||
rather than as a local hack here.
|
||||
|
||||
No setting was applied — this is a review/recommendation only per the task.
|
||||
|
|
@ -0,0 +1,114 @@
|
|||
# C3 spike findings: `--setting-sources` interaction + overlay file placement
|
||||
|
||||
_Task: C3 in [system-prompt-profiles-tasks.md](system-prompt-profiles-tasks.md). Feeds B2._
|
||||
|
||||
## Method note (important correction mid-spike)
|
||||
|
||||
Initial attempts asked the model to self-report whether ORCHESTRATION.md/skills were
|
||||
injected ("ORCH=yes/no SKILL=yes/no"). This was **unreliable** — a default run that the
|
||||
transcript later proved *did* receive the orchestration injection had the model answer
|
||||
`ORCH=no`. Model self-report is not a valid signal for this spike.
|
||||
|
||||
Switched to a deterministic signal: the session transcript JSONL at
|
||||
`~/.claude/projects/<slug>/<session_id>.jsonl` records hook execution as
|
||||
`attachment.type == "hook_additional_context"` entries (one per SessionStart hook that
|
||||
returned `additionalContext`) and the full injected skill list as
|
||||
`attachment.type == "skill_listing"`. Both are populated *before* the model ever sees the
|
||||
turn, so they're a ground-truth substitute for `/context`.
|
||||
|
||||
## Spike (a): `--setting-sources` vs symlinked local plugins + absolute-path hooks
|
||||
|
||||
4 headless haiku runs from an empty scratch dir
|
||||
(`/tmp/claude-1000/.../scratchpad/c3spike`), `claude -p --model haiku --output-format json
|
||||
"reply OK"`, varying `--setting-sources`. For each, inspected the resulting transcript
|
||||
JSONL for hook-injected additionalContext count and whether plugin-specific skills
|
||||
(`os-adr`, `os-vault`) appear in the injected skill listing (built-in skills like
|
||||
`autoresearch`/`caveman` appear regardless — only plugin skill presence is diagnostic).
|
||||
|
||||
| `--setting-sources` value | additionalContext entries (hooks fired) | os-adr/os-vault skills present |
|
||||
|---|---|---|
|
||||
| (no flag, default) | 1 (confirmed in first exploratory run) | yes |
|
||||
| `""` (none) | 0 | no |
|
||||
| `user` | 1 | yes |
|
||||
| `project,local` | 0 | no |
|
||||
|
||||
Interpretation:
|
||||
|
||||
- Hooks in this setup are wired **only** in `~/.claude/settings.json` (a **user**-scope
|
||||
settings file), by absolute path into `cc-os/`. They fire if and only if the `user`
|
||||
settings source is included. `--setting-sources ""` and `--setting-sources
|
||||
project,local` both silently drop them — no error, no warning, just no
|
||||
`additionalContext`. This is a real footgun for a future `cyolo` profile wrapper: any
|
||||
profile that passes `--setting-sources` explicitly must include `user` or it silently
|
||||
loses ORCHESTRATION.md injection, os-status checks, os-vault hooks, and every other
|
||||
hook in the plugin family.
|
||||
- Plugin skill registration (the `os-adr:find` style slash-command/skill listing) tracks
|
||||
the same axis: present with `user` included, absent without it. This makes sense —
|
||||
`enabledPlugins` lives in the user-scope `~/.claude/settings.json`, so a
|
||||
`--setting-sources` value that excludes `user` also excludes plugin enablement, not
|
||||
just hooks.
|
||||
- Symlinks themselves are a non-issue: once the `user` settings source is included, the
|
||||
symlinked plugin (`~/.claude/plugins/os-adr -> cc-os/plugins/os-adr`) resolves and
|
||||
loads identically to the default case — no special-casing needed for the symlink or
|
||||
for the hooks' absolute paths into `cc-os/`. The failure mode is entirely about which
|
||||
settings **source** is included, not about symlinks or path resolution.
|
||||
|
||||
**Verdict (a):** `--setting-sources` interacts cleanly with symlinked plugins and
|
||||
absolute-path hooks in the sense that nothing breaks or errors — but it is an all-or-
|
||||
nothing gate: excluding `user` from `--setting-sources` silently disables every cc-os
|
||||
plugin (hooks AND skills) with no error surfaced. Any profile wrapper must always include
|
||||
`user` in `--setting-sources` (or simply not pass the flag) to keep the plugin family
|
||||
alive; only `project`/`local` inclusion should be varied per profile.
|
||||
|
||||
## Spike (b): where should per-profile settings overlay files live?
|
||||
|
||||
Read `bin/refresh-plugins` (Ruby). It touches exactly these paths, all read-only except
|
||||
for the plugin cache itself:
|
||||
|
||||
- Reads `~/.claude/plugins/known_marketplaces.json` (marketplace list/config)
|
||||
- Reads `~/.claude/plugins/installed_plugins.json` (which plugins are installed per
|
||||
marketplace)
|
||||
- For each installed plugin under a `directory`-source marketplace, shells out to
|
||||
`claude plugin uninstall <plugin>@<marketplace>` / `claude plugin install
|
||||
<plugin>@<marketplace>` — this rewrites `~/.claude/plugins/cache/<marketplace>/
|
||||
<plugin>/<version>/` (via Claude Code's own plugin manager, not directly) and updates
|
||||
`installed_plugins.json`'s install record for that plugin.
|
||||
- Verifies via `diff -rq` between the cache dir and the plugin's source dir under
|
||||
`source_path` (from marketplace config, e.g. `~/dev/cc-os/plugins` or
|
||||
`~/dev/cc-plugins`).
|
||||
|
||||
It never touches `~/.claude/settings.json`, never touches any file under a
|
||||
`~/.claude/profiles/` style directory, and never touches arbitrary `--settings <file>`
|
||||
overlays — its blast radius is strictly `known_marketplaces.json`,
|
||||
`installed_plugins.json`, and the `plugins/cache/<marketplace>/<plugin>/` tree.
|
||||
|
||||
**Recommendation:** put per-profile settings overlay JSONs at `~/.claude/profiles/*.json`
|
||||
(e.g. `~/.claude/profiles/cc-os-design.json`, `client-dev.json`, `brainstorm.json`),
|
||||
passed via `claude --settings ~/.claude/profiles/<name>.json`. This path is structurally
|
||||
disjoint from everything `refresh-plugins` reads or writes (`plugins/cache/`,
|
||||
`plugins/known_marketplaces.json`, `plugins/installed_plugins.json`) — there is no
|
||||
plausible future edit to `refresh-plugins` that would need to reach into
|
||||
`~/.claude/profiles/`, since that script's whole job is cache-vs-source verification for
|
||||
marketplace-installed plugins, not settings resolution.
|
||||
|
||||
Interaction with `enabledPlugins`: `--settings <file>` is merged with (not a full
|
||||
replacement for) the resolved settings from the sources selected by `--setting-sources`
|
||||
(per spike (a), keep `user` in the source list so the base `~/.claude/settings.json`
|
||||
`enabledPlugins`/hooks/env still apply). The profile overlay should therefore only need
|
||||
to express a *diff* — an `enabledPlugins` map with the profile's unwanted plugins set to
|
||||
`false` (or, if this repo prefers positive lists, the full set with only the
|
||||
wanted ones `true`) — rather than duplicating the whole settings file. This diff-only
|
||||
overlay is exactly what B2 should generate per profile. One caution to carry into B2:
|
||||
verify at build time whether CC deep- or shallow-merges the `enabledPlugins` object
|
||||
across `--settings` sources (not tested in this spike) — either way, a plugin disabled
|
||||
via the overlay's `enabledPlugins: {"name": false}` does not require touching
|
||||
`known_marketplaces.json` or `installed_plugins.json`, since `enabledPlugins` toggles are
|
||||
independent of the marketplace/install-record identity those files hold (the files
|
||||
`refresh-plugins` owns).
|
||||
|
||||
**Verdict (b):** `~/.claude/profiles/*.json` (passed via `--settings`) is safe — it lives
|
||||
entirely outside the three paths `refresh-plugins` reads/writes/rewrites
|
||||
(`known_marketplaces.json`, `installed_plugins.json`, `plugins/cache/`), and
|
||||
`enabledPlugins` toggles in an overlay are orthogonal to the marketplace/cache identity
|
||||
`refresh-plugins` verifies, so cache refreshes and profile overlays can never clobber
|
||||
each other.
|
||||
|
|
@ -0,0 +1,145 @@
|
|||
# System-prompt profiles: assessment & recommended approach
|
||||
|
||||
_Last updated: 2026-07-08 · Status: assessment + next-actions agreed; deferral lifted —
|
||||
WS2/WS4 evals confirmed complete 2026-07-08 (Gate G0 cleared), execution may begin_
|
||||
|
||||
> Task breakdown for subagent implementation:
|
||||
> [system-prompt-profiles-tasks.md](system-prompt-profiles-tasks.md) (2026-07-08).
|
||||
|
||||
## Agreed next actions (2026-07-08, deferred)
|
||||
|
||||
Two tracks, both approved by the user:
|
||||
|
||||
1. **Top-down diet** (first, biggest win):
|
||||
- Split cc-os `CLAUDE.md` (~11k tokens) to a ~2–3k orientation + on-demand status/changelog
|
||||
doc, using `@`-import / pointer style per
|
||||
`~/servers/proxmox-hermes/docs/claude-md-maintenance.md` (keep it short; prefer pointers
|
||||
to copies; never auto-append).
|
||||
- Then per-profile plugin enablement (~4.8k max, second priority) and a memsearch
|
||||
injection review.
|
||||
- **Do not start while the other session's evals are running** — CLAUDE.md is live input
|
||||
to those sessions.
|
||||
2. **Bottom-up audit** (build anytime, read-only):
|
||||
- One-file logging proxy (`ANTHROPIC_BASE_URL` pass-through) that dumps the `system` and
|
||||
`tools` arrays from a real request — the audit basis for lean profiles. Expect savings
|
||||
to come from per-profile `--tools` lists (~18k of schemas), not prompt rewriting (~6.3k).
|
||||
|
||||
Related future idea (out of scope, captured in vault note
|
||||
[[claude-md-budget-linter-plugin-concept]]): a deterministic CLAUDE.md budget linter —
|
||||
hook-driven, JSON per-project config with quantified user-agreed exceptions, silent in range,
|
||||
maintenance agent only when out of range; possible extension to whole-project weekly hygiene
|
||||
scans. Overlaps os-doc-hygiene/os-status — resolve at build time.
|
||||
|
||||
Companion to the WS4 orchestration-economics work. Goal: reduce per-session context cost and
|
||||
improve focus by controlling what reaches the model's context — the built-in system prompt,
|
||||
and (more importantly) the layers we add on top of it.
|
||||
|
||||
## What is mechanically possible today
|
||||
|
||||
Verified against `claude --help` (current install) and
|
||||
https://code.claude.com/docs/en/agent-sdk/modifying-system-prompts.md (fetched 2026-07-08):
|
||||
|
||||
| Mechanism | What it does | Risk |
|
||||
|---|---|---|
|
||||
| `--append-system-prompt[-file]` | Adds instructions after the full default prompt | None — nothing removed |
|
||||
| `--system-prompt[-file]` | **Fully replaces** the built-in prompt for the session | High — loses tool guidance, safety rules, env context; you must re-provide them |
|
||||
| `--exclude-dynamic-system-prompt-sections` | Moves per-machine sections (cwd, git status, env, memory paths) into first user message → cross-session cache reuse | Low; env context carries slightly less weight |
|
||||
| `--tools <list>` / `--disallowedTools` | Restricts the built-in toolset (removes tool schemas from context) | Low |
|
||||
| `--settings` / `--setting-sources user,project,local` | Controls which settings layers (and therefore CLAUDE.md files, enabled plugins, hooks) load | Low |
|
||||
| Output styles (`~/.claude/output-styles/*.md`, `keep-coding-instructions: true|false`) | Replace or extend the *coding-instruction section* of the default prompt; persistent, reusable across projects | Medium — partial replacement, officially supported |
|
||||
| Plugin enable/disable per session (`--settings` overlay or profile-specific settings file) | Each enabled plugin's skills/agents/hooks descriptions are injected every session | Low — biggest immediate lever |
|
||||
| Agent SDK (`systemPrompt: {type:'preset', append, excludeDynamicSections}` or custom string) | Same controls programmatically; SDK default is a *minimal* prompt unless the preset is requested | — |
|
||||
|
||||
Deferred-tool loading (ToolSearch) already ships in current CC — most tool schemas are no
|
||||
longer paid up front. That reduced the value of "gut the system prompt" since much of the old
|
||||
bloat was tool definitions.
|
||||
|
||||
## Measured baseline (2026-07-08, headless haiku `-p` runs, `--output-format json` usage)
|
||||
|
||||
| Configuration | Context tokens | Delta |
|
||||
|---|---|---|
|
||||
| `--setting-sources "" --tools ""` (system prompt text only) | **6,347** | — |
|
||||
| `--setting-sources ""` (prompt + full built-in tool schemas) | **24,277** | +17.9k = tool schemas |
|
||||
| Default settings, empty dir (user stack: 17 plugins, global CLAUDE.md, hooks) | **29,042** | +4.8k = user stack |
|
||||
| Full stack, cc-os dir | **41,765** | +12.7k = project layer (CLAUDE.md ~11k dominates) |
|
||||
|
||||
Interpretation:
|
||||
- **The system prompt *text* is only ~6.3k tokens.** The ~24k harness floor is dominated by
|
||||
**tool schemas (~18k)** — so the bottom-up lever is `--tools` restriction per profile, not
|
||||
prompt rewriting. Full `--system-prompt` replacement can save at most ~6k and carries the
|
||||
fork-maintenance risk.
|
||||
- The 17-plugin user stack costs ~4.8k, not tens of k — cheaper than estimated.
|
||||
- The single largest single item we control is the cc-os CLAUDE.md (~11k).
|
||||
- An interactive session that reads ~80k after one exchange is ~42k fixed overhead plus
|
||||
~38k of *turn-1 work product* (web fetches, file reads) — conversation content, not
|
||||
harness bloat. Caveat: these baselines are headless haiku; an interactive Fable session's
|
||||
skill/agent listings and model-specific sections may differ somewhat — worth one `/context`
|
||||
check interactively.
|
||||
|
||||
## Key finding: our bloat is mostly self-inflicted, not Anthropic's
|
||||
|
||||
Measured 2026-07-08 for a cc-os session:
|
||||
|
||||
- **cc-os `CLAUDE.md`: ~43 KB ≈ ~11k tokens**, injected into every session. Much of it is an
|
||||
append-only implementation-status changelog, not orientation the session needs.
|
||||
- **17 enabled plugins** — every skill description (~80 skills), agent type, and hook usage
|
||||
note is injected into every session regardless of relevance (`creative-team`,
|
||||
`invoice-ninja`, `rails-ui-component`, `api-wrapper`… in a cc-os design session).
|
||||
- **memsearch SessionStart recall injection**: ~13 KB this session.
|
||||
- Multiple SessionStart hooks (orchestration, os-vault, os-status) each add context.
|
||||
|
||||
The built-in system prompt is a static, cached prefix (cache reads are ~10% of input price
|
||||
after the first turn within the 5-min TTL). The layers above are the same cost class *and*
|
||||
they're the ones we control. The "bloat → hallucination" link is context rot: attention
|
||||
degrades as context grows, whatever the source. Cutting 20k tokens of our own always-on
|
||||
injection beats fighting Anthropic's prompt.
|
||||
|
||||
Pi reality check: pi.dev makes **no** speed/cost/hallucination claims on its site. Its
|
||||
lean-ness comes from being a *different harness* (no MCP, no subagents, no permission
|
||||
system, minimal prompt + extensions) — not from stripping Claude Code. Fully replacing CC's
|
||||
system prompt while keeping CC's harness means maintaining a fork of prompt content the
|
||||
model was tuned around, re-broken by every CC update.
|
||||
|
||||
## Recommended approach: subtract-first modular profiles
|
||||
|
||||
A `cyolo <flags>` shell wrapper that composes **existing** mechanisms per objective/project.
|
||||
Profiles mostly *subtract* (plugins, tools, setting sources) and lightly *add*
|
||||
(one append-file per objective). Full system-prompt replacement is a later, eval-gated
|
||||
experiment on headless sessions only.
|
||||
|
||||
1. **Phase 0 — measure.** Run `/context` in 3–4 representative session types (cc-os design,
|
||||
client dev, brainstorm) and record the actual breakdown. No optimizing before baselining.
|
||||
2. **Phase 1 — diet (no new machinery).**
|
||||
- Split cc-os CLAUDE.md: keep orientation (~2–3k tokens), move the implementation-status
|
||||
changelog to a linked doc the AI reads on demand (progressive disclosure).
|
||||
- Per-profile plugin enablement: a settings overlay per profile listing `enabledPlugins`;
|
||||
a brainstorm profile doesn't load rails-ui-component/api-wrapper/invoice-ninja.
|
||||
3. **Phase 2 — `cyolo` profile wrapper.**
|
||||
- Central evergreen component prompts in one source dir (e.g. `~/.claude/profiles/`):
|
||||
`dev.md`, `ruby.md`, `planning.md` (objective-focused: starting point → process →
|
||||
definition of done), `brainstorm.md`.
|
||||
- Wrapper resolves profile from flag (`--dev`, `--planning`) + project detection (pwd,
|
||||
Gemfile, etc.), then launches
|
||||
`claude --append-system-prompt-file <composed> --settings <profile-settings> [--tools …]`.
|
||||
- Objective profiles state the session's endpoint explicitly ("this session ends when the
|
||||
plan is written and shared; implementation belongs to a later session").
|
||||
4. **Phase 3 — eval-gated replacement experiment (optional).** Arize-style loop, matching the
|
||||
autoresearch playbook: scenario set × {default prompt, dieted stack, profile stack,
|
||||
full-replacement lean prompt} × model tiers; score task success + tokens/turn +
|
||||
wrong-tool/hallucination proxies from transcripts. Only promote full replacement if it
|
||||
wins on outcome, not just token count. Read
|
||||
`~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md` first, per standing rule.
|
||||
|
||||
Bottom-up track note: to audit the real shipped prompt, capture a live request body via a
|
||||
local logging proxy (`ANTHROPIC_BASE_URL` pointed at a one-file pass-through that dumps the
|
||||
`system` and `tools` arrays) — don't rely on the model to self-report its prompt. Given the
|
||||
6.3k measurement, audit it to find what a lean profile can safely drop, but expect the
|
||||
`--tools` list to be where the real savings are.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Per-model profile variants: deferred (user agrees it's too granular now); the wording-loop
|
||||
evals already show tier-dependent phrasing needs, so revisit after profiles exist.
|
||||
- Whether `--setting-sources` interacts cleanly with symlinked local plugins + hook absolute
|
||||
paths in `settings.json` (hooks are wired by absolute path and would still fire).
|
||||
- Where profile settings overlays live so they don't fight `bin/refresh-plugins`.
|
||||
|
|
@ -0,0 +1,72 @@
|
|||
# System-prompt profiles: task breakdown for subagent implementation
|
||||
|
||||
_Last updated: 2026-07-08 · Source plan: [system-prompt-profiles-assessment.md](system-prompt-profiles-assessment.md)_
|
||||
|
||||
Groups are ordered by dependency, not priority. Group A is read-only and can start
|
||||
immediately, in parallel. Group B is the biggest win but is **gated**. Groups C and D are
|
||||
sequential follow-ons.
|
||||
|
||||
**Slimmed scope (2026-07-08, user-approved):** execute quick/high-value wins only —
|
||||
B1 → B3 ∥ C3 → B2, then a minimal C (settings-overlay aliases, no auto-detection).
|
||||
A1/A2 are deferred until a live interactive `/context` check shows tool schemas still
|
||||
dominate despite ToolSearch deferred loading (one user-assisted check replaces Group A;
|
||||
A3's headless matrix is already substantially in the assessment doc). D is dropped from
|
||||
the plan — revisit only if CC economics change; max upside ~6.3k with fork-maintenance
|
||||
downside.
|
||||
|
||||
**Gate G0 — CLEARED (2026-07-08):** user confirmed the os-orchestration (WS4) and os-vault
|
||||
(WS2) eval loops are complete; no eval runs remain in flight. Group B is unblocked.
|
||||
|
||||
---
|
||||
|
||||
## Group A — Bottom-up audit (read-only, start anytime, parallelizable)
|
||||
|
||||
| ID | Task | Depends on | Suggested agent |
|
||||
|---|---|---|---|
|
||||
| A1 | **Logging proxy.** One-file `ANTHROPIC_BASE_URL` pass-through that forwards to `api.anthropic.com` and dumps the `system` and `tools` arrays of each request to a local file. Ruby preferred (Sandi Metz style not required for a one-file tool). No response mutation, streaming must pass through untouched. Deliverable: `bin/` or `tools/` script + 3-line usage note. | — | sonnet |
|
||||
| A2 | **Live prompt capture + audit.** Run one headless and (user-assisted) one interactive session through the A1 proxy from cc-os and from an empty dir; save the dumped `system`/`tools` arrays; produce an audit doc: token cost per system-prompt section, per tool schema, and a "safely droppable per profile" table. Expect savings in `--tools` lists (~18k), not prompt text (~6.3k). | A1 | sonnet |
|
||||
| A3 | **Phase 0 measurement.** Script the headless baseline matrix (already partially done in the plan: 4 configs measured) and extend to 3–4 representative session types (cc-os design, client dev, brainstorm) via `--output-format json` usage fields; record one interactive `/context` check (user-assisted) to validate the headless numbers against a Fable session. Deliverable: a measurements table appended to the assessment doc. | — | haiku (scripted runs) + user for the interactive check |
|
||||
|
||||
A1 and A3 run in parallel; A2 follows A1.
|
||||
|
||||
## Group B — Top-down diet (**gated on G0**, sequential)
|
||||
|
||||
| ID | Task | Depends on | Suggested agent |
|
||||
|---|---|---|---|
|
||||
| B1 | **Split cc-os CLAUDE.md.** ~11k tokens → ~2–3k orientation file + `docs/implementation-status.md` (the append-only changelog moved out, linked not `@`-imported so it loads on demand). Follow `~/servers/proxmox-hermes/docs/claude-md-maintenance.md`: keep it short, pointers over copies, never auto-append. Must preserve: the "keep this file current" contract (redirected to the new doc), plugin naming rule, ADR discipline, OpenSpec workflow. Verify a fresh session still orients correctly. | G0 | sonnet (judgment about what is orientation vs changelog); main-loop review before commit |
|
||||
| B2 ✅ (2026-07-08, see [b2-profile-overlays.md](b2-profile-overlays.md) — deep-merge confirmed, 3 overlays shipped to `~/.claude/profiles/`, measured savings marginal: −32 to −632 tokens) | **Per-profile plugin enablement.** Design + create settings overlay files (one per profile: e.g. `cc-os-design`, `client-dev`, `brainstorm`) listing `enabledPlugins`; a brainstorm profile drops rails-ui-component / api-wrapper / invoice-ninja / creative-team etc. Resolve where overlays live so they don't fight `bin/refresh-plugins` (open question #3). | B1 (so measured savings are attributable) | sonnet |
|
||||
| B3 | **memsearch injection review.** Measure the SessionStart recall injection size across a week of sessions (~13 KB observed once); review memsearch plugin config options for capping/trimming it; recommend (not apply) a setting. | — (parallel with B1/B2 once G0 clears) | haiku |
|
||||
|
||||
## Group C — `cyolo` profile wrapper (Phase 2, after B)
|
||||
|
||||
| ID | Task | Depends on | Suggested agent |
|
||||
|---|---|---|---|
|
||||
| C1 | **Profile prompt components.** Create `~/.claude/profiles/` with `dev.md`, `ruby.md`, `planning.md`, `brainstorm.md`. Objective profiles state the session endpoint explicitly ("this session ends when the plan is written…"). Small files, ~0.5k tokens each. | B2 (profiles reference the settings overlays) | sonnet |
|
||||
| C2 | **Wrapper script.** `cyolo` shell wrapper: resolve profile from flag (`--dev`, `--planning`) + project detection (pwd, Gemfile…), compose append-file, launch `claude --append-system-prompt-file <composed> --settings <profile-settings> [--tools …]`. | C1 | sonnet |
|
||||
| C3 | **Open-question spikes** (can run before/parallel with C1): (a) does `--setting-sources` interact cleanly with symlinked local plugins + absolute-path hooks? (b) confirmed overlay location from B2 survives `bin/refresh-plugins`. Small experiments, written up as findings notes. | — | haiku |
|
||||
|
||||
## Group D — Eval-gated replacement experiment (optional, last)
|
||||
|
||||
| ID | Task | Depends on | Suggested agent |
|
||||
|---|---|---|---|
|
||||
| D1 | **Eval design.** Read `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md` FIRST (standing rule). Design the grid: scenario set × {default, dieted stack, profile stack, full-replacement lean prompt} × model tiers; score task success + tokens/turn + wrong-tool proxies. Design only — do not run without user sign-off. | C2 (profile stack must exist to be a grid column) | sonnet |
|
||||
| D2 | **Run + verdict.** Execute the grid headless-only; promote full replacement only if it wins on *outcome*, not just token count. | D1 | per playbook |
|
||||
|
||||
## Out of scope (captured, do not implement)
|
||||
|
||||
- CLAUDE.md budget linter plugin — vault note [[claude-md-budget-linter-plugin-concept]];
|
||||
overlaps os-doc-hygiene/os-status, resolve at build time.
|
||||
- Per-model profile variants — deferred by user decision; revisit after profiles exist.
|
||||
|
||||
## Suggested execution waves
|
||||
|
||||
Superseded by the slimmed scope above. Current order:
|
||||
|
||||
1. **Wave 1 (parallel):** B1 (main-loop), B3 + C3 (subagents).
|
||||
2. **Wave 2:** B2 (needs C3's overlay-location answer). ✅ Done 2026-07-08 —
|
||||
overlays shipped; measured token savings marginal (plugin listings are cheap
|
||||
post-ToolSearch), so profiles' value is focus, not context cost.
|
||||
3. **Wave 3 (minimal C):** settings-overlay aliases only; full C1/C2 wrapper deferred
|
||||
until the friction is felt. B2's marginal-savings result lowers this further —
|
||||
do only when friction is actually felt.
|
||||
4. **Deferred:** A1/A2 pending the interactive `/context` check; D dropped (pointer kept).
|
||||
|
|
@ -1,7 +1,14 @@
|
|||
# WS2 — os-vault "when and how to write" eval
|
||||
|
||||
_Created: 2026-07-06. Status: proposed. Prereqs: quick wins immediate; harness after WS3
|
||||
design + quick wins. See `2026-07-06-plugin-evals-overview.md`._
|
||||
_Created: 2026-07-06. Status: **steps 1–3 done 2026-07-06** — quick wins shipped (contract
|
||||
fix + cc-os hub note), harness built at `plugins/os-vault/eval/` (12 scenarios authored
|
||||
directly, fixture + Ruby checker via sonnet subagents, self-test 21/21, isolation via new
|
||||
`OS_VAULT_PATH`/`OS_VAULT_SKIP_REBUILD` seam), untuned baseline grid run (37 counted reps:
|
||||
positives 1/19, negatives 18/18; headline finding: at L1 the model persists to Claude Code's
|
||||
built-in auto-memory instead of the vault — routing, not just triggering, is the wording
|
||||
target). Results: `eval/results/2026-07-06-baseline-grid.tsv` + vault note
|
||||
[[os-vault-write-eval-baseline-grid-results]]. Step 4 (wording loop) not started — the
|
||||
run-set is still uncontaminated. See `2026-07-06-plugin-evals-overview.md`._
|
||||
|
||||
## Goal
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,10 @@
|
|||
# WS3 — per-plugin SessionStart status-check convention + glue plugin
|
||||
|
||||
_Created: 2026-07-06. Status: proposed. Prereq: none (parallel with WS1). See
|
||||
_Created: 2026-07-06. Status: **superseded 2026-07-06** by the OpenSpec change
|
||||
`add-os-status-plugin` (implemented as `plugins/os-status/`; decision record: ADR-022 in
|
||||
`docs/memory-system/03-architecture-decisions.md`). The perspective reviews kept the Option B
|
||||
direction but replaced the per-plugin enumeration protocol with in-process checks in a single
|
||||
plugin; this doc is the historical starting sketch only. See
|
||||
`2026-07-06-plugin-evals-overview.md`._
|
||||
|
||||
## Goal
|
||||
|
|
|
|||
|
|
@ -0,0 +1,200 @@
|
|||
# WS4 — Orchestration economics: ORCHESTRATION.md v3 + eval extension + wording loop
|
||||
|
||||
_Created: 2026-07-08. Status: **SHIPPED** (steps 1–4 complete 2026-07-08; step 5 IRL
|
||||
re-audit scheduled ~2026-07-22). Loop log: `autoresearch/classic-260708-1039/loop-log.md`;
|
||||
results TSVs: `plugins/os-orchestration/eval/results/2026-07-08-*.tsv`; vault note:
|
||||
[[os-orchestration-ws4-econ-wording-results]]._
|
||||
_Prereq reading before executing: `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md`,
|
||||
vault notes [[orchestration-prompting-claude-5-era]], [[eval-methodology-ladder]],
|
||||
[[os-orchestration-eval-baseline-grid-results]]._
|
||||
|
||||
## Problem
|
||||
|
||||
The Fable-5 main loop does too much multi-call work itself. ORCHESTRATION.md was calibrated
|
||||
against over-delegation (Opus 4.6 era) under sonnet main loops where direct work was cheap;
|
||||
its negative-first threshold ("Delegate only when…", "≤2-tool-call ops direct") is read
|
||||
literally by Fable and licenses unbounded direct work whenever the task isn't parallelizable/
|
||||
many-files/isolated-context. Goals (user, 2026-07-08): max quality at reasonable cost;
|
||||
route work to cheaper tiers whenever reasonable; group delegated tasks so agents finish in
|
||||
few rounds (fewer spawns → less per-spawn system-prompt tax).
|
||||
|
||||
Evidence base: Anthropic model-page guidance summarized in vault note
|
||||
[[orchestration-prompting-claude-5-era]] (fetched 2026-07-08); WS1 audit
|
||||
(`docs/orchestration-audit/2026-07-06-findings.md`); Fable economics mini-audit
|
||||
(`docs/orchestration-audit/2026-07-08-fable-econ-miniaudit.md`) — the H3 baseline.
|
||||
|
||||
### Mini-audit headline (2026-07-08, 6 Fable sessions, ran before drafting wording)
|
||||
|
||||
- **The dominant miss shape is implementation fan-out, not investigation:** the one clean
|
||||
miss is an OpenSpec-apply session (`5f53e0c0`) that wrote 13 independent Ruby files
|
||||
entirely in the main loop — 314,899 main-loop output tokens, 0 spawns. E5P should mirror
|
||||
this shape (write-N-independent-files implementation), not a log-review shape.
|
||||
- Long direct runs were mostly JUSTIFIED (scripted eval-polling loop of 119 calls;
|
||||
interactive troubleshooting with 15 human turns) — confirming the non-goal: cost-keyed
|
||||
thresholds, not "always delegate". Zero over-delegation anywhere.
|
||||
- Secondary watch item: after a session's spawns complete, the main loop resumes long
|
||||
direct runs (43–48 calls, `9f45afcc`) instead of delegating again — mid-session threshold
|
||||
decay.
|
||||
- **Instrumentation gap (affects A-econ):** main transcripts contain NO `isSidechain:true`
|
||||
entries even in sessions with spawns — subagent token spend is unrecoverable from the
|
||||
transcript alone. The eval checker must source subagent spend from harness-side artifacts
|
||||
(the eval runner's own task/agent output files in the sandbox), or A-econ degrades to a
|
||||
main-loop-absolute-tokens metric (still usable: baseline sessions run 24k–315k main-loop
|
||||
output tokens). Resolve this during Step 2 design; do not silently ship the char-count
|
||||
proxy the mini-audit used.
|
||||
|
||||
## Step 1 — ORCHESTRATION.md v3 (wording surface for the loop)
|
||||
|
||||
Five changes; keep explicit-`model:`, self-report, and don't-re-cover-ground rules unchanged:
|
||||
|
||||
1. **Cost-asymmetry + tier-conditional threshold.** New opening rule: main-loop tokens are
|
||||
the most expensive in the session; the more capable the main-loop model, the lower the
|
||||
delegation threshold. Any investigation or mechanical sequence beyond ~3 calls that does
|
||||
not need main-loop judgment → delegate down-tier, even when sequential. (Models know their
|
||||
own model ID — tier-conditional phrasing is implementable; E1 canary verified self-report.)
|
||||
2. **Symmetric triggers.** Replace "Delegate only when…" with paired lists:
|
||||
"delegate when …" / "work directly when …" (keep the scripted-bulk-edit carve-out and
|
||||
≤2-call rule on the direct side). Rationale: literal instruction-following on 5-era
|
||||
models; the canonical best-practices snippet pairs both directions.
|
||||
3. **Batching rule.** Plan the fan-out before the first spawn; group related subtasks
|
||||
(~5–8 similar items) into one agent prompt with a return schema so the agent completes in
|
||||
one round; follow-ups go to the live agent (SendMessage), not a fresh spawn.
|
||||
4. **Async rule.** Delegate independent subtasks and keep working while they run; intervene
|
||||
only if a subagent goes off track (Fable-page recommended pattern).
|
||||
5. **Effort dial (where exposed).** Workflow `agent()` calls: mechanical stages
|
||||
`effort: low`; hard verify/judge stages high/xhigh.
|
||||
|
||||
Draft AFTER reading the mini-audit's miss shapes — wording should name the observed shapes,
|
||||
not hypothetical ones (Eval B lesson: mechanical triggers phrased at the observed
|
||||
precondition).
|
||||
|
||||
## Step 2 — Eval harness extension (`plugins/os-orchestration/eval/`)
|
||||
|
||||
All extensions are new frozen surfaces; existing run-set scenarios E1–E3 stay untouched.
|
||||
Reserve stays frozen. Follow [[eval-methodology-ladder]]: paired positives/negatives,
|
||||
run-set + reserve twins authored together, thresholds anchored independently of any grid
|
||||
already run (pre-registered criterion-redesign rule from the 2026-07-06 baseline).
|
||||
|
||||
### New metric axes (extractor + checker)
|
||||
|
||||
- **A-econ (main-loop token share):** main-loop assistant output tokens ÷ total session
|
||||
output tokens (main + sidechains). Extractor already parses transcripts; add the sum.
|
||||
- **A-prespawn:** tool calls + tool_result bytes before first spawn (extractor segment
|
||||
exists from E3 work).
|
||||
- **A-batch:** spawns per task and rounds per agent (Agent tool_use count; SendMessage
|
||||
reuse counts as same-agent round, not new spawn).
|
||||
- Redesign E2P/E3P delegate-at-all axes per the pre-registered note: PASS may be
|
||||
"delegated" OR "superior scripted direct strategy" — the checker must distinguish
|
||||
script-driven bulk ops (few Bash calls, uniform op) from grind (per-file tool loops).
|
||||
|
||||
### New scenario pair E5 (batching) + reserve twins
|
||||
|
||||
- **E5P:** N (~10–12) similar independent items, shaped like the observed miss —
|
||||
an implementation task producing N independent files in the relaystation fixture
|
||||
(NOT log review). PASS = grouped delegation (≤3 spawns covering all items, grouped
|
||||
prompts, one round each), FAIL = 1-agent-per-item or main-loop grind.
|
||||
- **E5N (paired negative):** a task where batching would wrongly serialize genuinely
|
||||
judgment-dependent items (each item's handling depends on the previous result) — PASS =
|
||||
sequential handling (direct or single agent), FAIL = blind parallel fan-out.
|
||||
- Author reserve twins at the same time, different surface domain; freeze both.
|
||||
|
||||
### Fable column
|
||||
|
||||
Baseline grid ran sonnet/haiku orchestrators only; the population of concern is Fable.
|
||||
Add `--model fable` (headless `claude -p --model claude-fable-5`) at reduced reps
|
||||
(1–2 per cell for the grid; 3 on target cells inside the loop). Cost of the eval itself is
|
||||
real — use the reduced inner-loop grid discipline (target cells + one passing control).
|
||||
|
||||
## Step 3 — Baseline grid (pre-wording)
|
||||
|
||||
Run extended grid (E1–E3 + E5, econ axes) × {fable, sonnet} before touching wording.
|
||||
Canary-cell the first live rep of every NEW scenario/axis and hand-verify TSV vs transcript
|
||||
(count the canary). This grid is the tuning baseline; the 2026-07-08 mini-audit is the IRL
|
||||
baseline for H3.
|
||||
|
||||
## Step 4 — Autoresearch wording loop
|
||||
|
||||
Loop discipline per the howto: wording-only moves (checker/fixtures/scenarios/axes frozen),
|
||||
`bin/refresh-plugins` before every iteration's grid, ≥3 reps on target cells + 1 control
|
||||
cell, accept only majority-of-reps improvements, verify from TSV not agent prose.
|
||||
|
||||
### Hypotheses (pre-registered)
|
||||
|
||||
- **H1 (threshold framing):** symmetric cost-framed triggers raise delegation on positive
|
||||
scenarios (esp. E3P-shape sequential investigation) on fable/sonnet, while negatives stay
|
||||
clean. Guard: baseline negatives were 18/18 — any negative regression rejects the
|
||||
candidate regardless of positive gains.
|
||||
- **H2 (batching rule):** the grouping rule reduces spawns-per-task and rounds-per-agent on
|
||||
E5P without task-success regression; E5N stays PASS (no blind fan-out).
|
||||
- **H3 (net economics — the actual goal):** main-loop token spend on positives drops
|
||||
materially vs the Step-3 baseline with task success unchanged. Metric: main-loop share
|
||||
if Step 2 solves the sidechain-instrumentation gap, else main-loop absolute output
|
||||
tokens per scenario. Anchor the target threshold independently before the loop starts
|
||||
(not post-hoc to whatever the loop achieves). H1/H2 are mechanisms; H3 is the verdict
|
||||
axis.
|
||||
|
||||
**H3 threshold (pre-registered 2026-07-08, before the Step-3 grid ran):** H3 PASSes iff,
|
||||
on the positive economics cells (E5P and E3P, fable column), the median main-loop
|
||||
assistant output tokens across reps under the candidate wording is ≤60% of the Step-3
|
||||
pre-wording baseline median for the same cell (≥40% reduction), with task-success axes
|
||||
unchanged and all negatives clean. Pass/fail is keyed to main-loop **absolute** output
|
||||
tokens (exists in both baselines regardless of the sidechain-instrumentation outcome);
|
||||
main-loop *share* is reported additionally if Step 2 solves sidechain measurement.
|
||||
Anchor rationale (independent of any grid): in the IRL miss exemplar `5f53e0c0`, the
|
||||
Write×13 fan-out segment dominates a 314,899-token session — delegating the
|
||||
implementation segment should remove roughly half of main-loop output; 40% is a
|
||||
conservative floor of that estimate.
|
||||
|
||||
**H3 amendment (2026-07-08, after the Step-3 baseline, BEFORE any wording iteration):**
|
||||
the baseline showed E3P positives already PASS cheaply on both tiers via surgical direct
|
||||
investigation (fable median mltok 10,273; sonnet 9,420) — a reduction target there would
|
||||
reward degrading already-correct behavior. Amended H3: (a) **E5P fable** median mltok ≤
|
||||
60% of baseline median 62,971 → **≤ 37,783**, with the E5P verdict PASSing on a majority
|
||||
of reps; (b) **E3P guard**: verdict stays PASS and median mltok ≤ 120% of baseline
|
||||
(fable ≤ 12,328); (c) **E5P sonnet**: verdict flips to PASS on majority (baseline mltok
|
||||
already low at 9,789 — no reduction target, guard ≤ 150%); (d) all negatives stay clean.
|
||||
|
||||
## Step-3 baseline results (2026-07-08, grid complete)
|
||||
|
||||
TSV: `plugins/os-orchestration/eval/results/2026-07-08-ws4-baseline-grid.tsv` — 44 counted
|
||||
reps (25 valid first-wave + 15 replacements for session-limit-truncated reps, mechanically
|
||||
excluded by the limit-banner criterion + 4 counted canaries; excluded rows preserved in the
|
||||
TSV as comments). Headlines:
|
||||
|
||||
- **E5P 0/6 both tiers, all `A:main-loop-grind`** — the mini-audit miss shape reproduces in
|
||||
the lab. Fable burns median 62,971 mltok doing it (the H3 target cell); sonnet ~9.8k.
|
||||
- **Criterion redesign validated**: E2P 6/6 PASS via `A:scripted-direct` (old axis would
|
||||
have failed all); E3P 6/6 PASS via surgical direct under the 74KB ingestion anchor.
|
||||
- **All negatives clean** (E1N/E2N/E3N/E5N, both tiers) — including E5N on fable: no blind
|
||||
fan-out on the sequential task.
|
||||
- **E1P: sonnet 0/3 (never delegates), fable 1/2** — fable delegates readily (3–6 spawns)
|
||||
and its one FAIL is axis-B (didn't flag the stubbed downgrade), the first observed
|
||||
fable-tier self-report miss.
|
||||
- Sidechain token accounting works live: fable E1 cells show mlshare 0.36–0.68 with real
|
||||
sctok sums.
|
||||
- Wording-loop target cells: E5P (both tiers, A-axis + fable econ), E1P sonnet
|
||||
(delegate-at-all), E1P fable (axis-B flag reliability). Controls: E2P (must stay PASS via
|
||||
scripted-direct), E3P (non-regression guard), negatives.
|
||||
|
||||
### Judging
|
||||
|
||||
Per-axis deterministic checker verdicts; per-scenario pass bars, not aggregate scores.
|
||||
A wording candidate ships only if: all negatives hold, target positives improve on majority
|
||||
of reps, control cell unchanged, and the full-grid confirm reproduces it. Final confirmation
|
||||
on the frozen reserve (tuning against the run-set moves measurement to the reserve —
|
||||
after this loop the run-set is contaminated for future measurement).
|
||||
|
||||
## Step 5 — Rollout + IRL re-audit
|
||||
|
||||
Ship winning wording (refresh caches), then re-run the Fable economics mini-audit on ~5 new
|
||||
real sessions after ~2 weeks ([[eval-methodology-irl-feedback-loop]]). Compare main-loop
|
||||
token share vs the 2026-07-08 baseline. Promote any new miss shapes into eval scenarios.
|
||||
|
||||
## Non-goals / guards
|
||||
|
||||
- No changes to delegation SAFETY rules (explicit model:, self-report, downgrade honesty).
|
||||
- Don't over-rotate: zero over-delegation was a baseline strength (negatives 18/18) and
|
||||
scripted-direct strategies were often genuinely superior — the point is cost-keyed
|
||||
thresholds, not "always delegate".
|
||||
- Held-out discipline: never run scenario Task blocks informally; reserve is never read
|
||||
informally.
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
schema: spec-driven
|
||||
created: 2026-07-06
|
||||
|
|
@ -0,0 +1,160 @@
|
|||
# Design: add-os-status-plugin
|
||||
|
||||
## Context
|
||||
|
||||
cc-os ships four global plugins (os-vault, os-adr, os-orchestration, os-doc-hygiene), all in
|
||||
one repo, maintained by one person. Each has per-project or per-environment preconditions;
|
||||
today two plugins carry their own SessionStart reminders (os-adr existence check,
|
||||
os-doc-hygiene staleness reminder) and two carry none (os-vault hub-note presence,
|
||||
os-orchestration env-override detection). The WS3 plan
|
||||
(`docs/plans/ws3-status-convention-plugin.md`) proposed a convention: per-plugin status checks
|
||||
aggregated into one banner.
|
||||
|
||||
Two 2026-07-06 perspective reviews (perspectives:simplifier, perspectives:implementer, both
|
||||
sonnet) pressure-tested the plan's central tension — where check code lives. Their verified
|
||||
findings shape this design:
|
||||
|
||||
- Per-project code copies (the plan's Option A) are the proven failure class: the 2026-07-04
|
||||
stale-cache incident was copy-drift.
|
||||
- A subprocess + JSON stdout enumeration protocol (Option B as sketched) is overbuilt for
|
||||
three trivially cheap checks (path-exists, env read, settings grep) with one author.
|
||||
- The three existing plugins already disagree on hook wiring: os-adr/os-doc-hygiene run from
|
||||
the plugin cache via `hooks/hooks.json` + `${CLAUDE_PLUGIN_ROOT}`; os-vault is wired by
|
||||
absolute path in `~/.claude/settings.json` straight to git source, and its cache copy is
|
||||
already stale. Any design that enumerates other plugins' caches inherits this hazard.
|
||||
- "Silent on ok" as a blanket rule would regress os-adr's present-state usage note (a
|
||||
near-zero-token pointer to `/os-adr:create` + `/os-adr:find` whose wording was tuned by the
|
||||
Eval B campaign and validated by Eval C).
|
||||
- Migration must cut over hooks atomically per plugin or both old and new hooks fire
|
||||
(double banner), and cache must be verified post-cutover (`claude plugin details`), not just
|
||||
source-edited.
|
||||
|
||||
User decision 2026-07-06: adopt the in-process, convention-shaped design.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
|
||||
- One aggregated SessionStart status pass over all cc-os precondition checks; at most one
|
||||
short warning banner per session.
|
||||
- Catch, deterministically and at session start, the two incident classes that motivated
|
||||
this: missing per-project artifacts (ADR system, vault hub note) and silent model-forcing
|
||||
env overrides (`CLAUDE_CODE_SUBAGENT_MODEL`).
|
||||
- Preserve os-adr's tuned present-state usage note verbatim (no behavioral regression).
|
||||
- Keep adding a check cheap: one function + one registry entry in one file, same repo.
|
||||
- Model-free tests + invariants.md (same rigor as os-adr's hook; no behavioral eval — the
|
||||
code is deterministic).
|
||||
|
||||
**Non-Goals:**
|
||||
|
||||
- No subprocess boundary, no JSON wire contract, no installed-plugin enumeration (revisit
|
||||
only if a future check needs isolation: slow, network-touching, or conflicting deps).
|
||||
- No `context` session-injection field (no consumer yet; injection economics unexamined).
|
||||
- No consolidation of `.os-adr/` / `.dochygiene/` state dirs (works today, separable).
|
||||
- No conversion of os-doc-hygiene's reminder in this change (follow-up; keeps migration
|
||||
surface to one plugin).
|
||||
- No cache-freshness "collection manager" duties (Phase 2, separate change).
|
||||
- No fixing of os-vault's absolute-path hook wiring here (documented hazard, not a blocker —
|
||||
this design reads nothing from other plugins' caches).
|
||||
|
||||
## Decisions
|
||||
|
||||
### D1 — Code location: in-process functions in the `os-status` plugin
|
||||
|
||||
All checks are plain Python functions living in `plugins/os-status/hooks/checks.py`, imported
|
||||
and called in a loop by a single SessionStart entry point. No per-plugin `status/check.py`,
|
||||
no discovery.
|
||||
|
||||
- Why over Option A (project-local `.cc-os/status.d/*.py`): copies drift; per-project install
|
||||
step; 2026-07-04 incident is this exact class.
|
||||
- Why over Option B-as-sketched (per-plugin source + subprocess enumeration): three checks,
|
||||
one author, one repo — enumeration adds a protocol (spawn, parse, timeout, error
|
||||
normalization) with only three registrants; and enumeration-from-cache would silently run
|
||||
os-vault's stale cache copy (implementer finding). In-process reads nothing from other
|
||||
plugins' installs.
|
||||
- Convention preserved as a seam: every check has the uniform signature
|
||||
`check(ctx) -> CheckResult(status, message)` with `ctx` carrying project root + config.
|
||||
Extraction to subprocess later is mechanical because the contract already exists as a
|
||||
function boundary.
|
||||
|
||||
### D2 — Result semantics: `ok` / `note` / `warn` (three states, not two)
|
||||
|
||||
- `ok` → contributes nothing, silent.
|
||||
- `note` → near-zero-token informational line, NOT subject to snooze/suppress; exists
|
||||
precisely so os-adr's present-state usage note survives conversion unchanged (implementer
|
||||
finding: blanket silent-on-ok is a regression).
|
||||
- `warn` → aggregated into the single banner with the corrective action
|
||||
("X missing — run /os-…:… to fix"); subject to snooze/suppress.
|
||||
- A check that raises or exceeds its time budget is reported as `warn` with a generic
|
||||
message; it never blocks the session and never suppresses other checks' results
|
||||
(per-check try/except in the runner).
|
||||
|
||||
### D3 — Ship as a new `os-status` plugin, hooks.json-wired
|
||||
|
||||
New plugin `plugins/os-status/` with `hooks/hooks.json` using `${CLAUDE_PLUGIN_ROOT}`
|
||||
(matching os-adr/os-doc-hygiene — the cache-correct pattern), thin `session_start.py` entry
|
||||
point + deep modules (`checks.py`, `state.py`), mirroring os-vault's module style.
|
||||
|
||||
- Why a new plugin over piggybacking on os-orchestration (simplifier floated this):
|
||||
os-orchestration's domain is session-orchestration wording; status checking is a distinct
|
||||
domain and the naming convention (`os-[domain]`) favors a dedicated home. Name `os-status`
|
||||
over `os-core`: narrower, honest about scope; "core/collection manager" duties are Phase 2
|
||||
and can justify a rename or a second plugin then.
|
||||
|
||||
### D4 — State: `.cc-os/` per-project dir, state only
|
||||
|
||||
Snooze (once-per-day) and permanent suppress files live in `<project>/.cc-os/` (gitignored;
|
||||
the check runner adds it to `.git/info/exclude` opportunistically or documents the gitignore
|
||||
line). Never code. Existing `.os-adr/`, `.dochygiene/` untouched. Outside a git project, the
|
||||
hook exits silently (os-adr precedent).
|
||||
|
||||
### D5 — Atomic os-adr cutover in this change
|
||||
|
||||
os-adr's SessionStart entry is deleted from its `hooks/hooks.json` in the same change that
|
||||
os-status ships its equivalent check (existence → init/migrate suggestion with once-per-day
|
||||
snooze honoring the existing `.os-adr/suppress`; presence → the verbatim usage note as a
|
||||
`note`). Post-cutover verification: `bin/refresh-plugins`, then `claude plugin details
|
||||
os-adr@local-plugins` must show the SessionStart hook gone from the resolved cache, then a
|
||||
fresh-session smoke test shows exactly one emission.
|
||||
|
||||
### D6 — Initial check registry
|
||||
|
||||
1. `adr_system_present(ctx)` — port of `plugins/os-adr/hooks/session_start.py` logic,
|
||||
including its `.os-adr/suppress` + once-per-day behavior and the Eval-B-tuned PRESENT_NOTE
|
||||
wording verbatim.
|
||||
2. `vault_hub_note_present(ctx)` — project has a hub note in the vault (lookup strategy:
|
||||
match `project/<name>` facet tag or configured hub slug in
|
||||
`~/Documents/SecondBrain`); missing → `warn` naming the corrective skill; present → `ok`
|
||||
(no injection — see Non-Goals).
|
||||
3. `subagent_model_env_override(ctx)` — `warn` if `CLAUDE_CODE_SUBAGENT_MODEL` is in the
|
||||
process environment or in the `env` block of `~/.claude/settings.json`.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- [Adding a check means editing os-status, not the owning plugin] → Accepted: one author, one
|
||||
repo; the uniform signature keeps a later extraction mechanical. Revisit at the first check
|
||||
that needs isolation.
|
||||
- [os-status must know about other plugins' domains (hub notes, ADR dirs)] → Accepted
|
||||
coupling; checks are read-only path/env probes, not imports of other plugins' code.
|
||||
- [Double emission during cutover if caches go stale] → D5's verify-the-cache step is a
|
||||
mandatory task, not cleanup; the 2026-07-04 incident is the template for what happens
|
||||
otherwise.
|
||||
- [Hub-note lookup heuristic may mis-detect] → keep the matcher trivial (explicit config in
|
||||
`.cc-os/config` wins over inference); a false `warn` is one banner line, suppressible.
|
||||
- [SessionStart latency] → in-process, three cheap probes, one interpreter spawn; budget the
|
||||
hook at 5s like os-adr's. No parallelism machinery needed at this scale.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
1. Build os-status complete with tests green (no hook registered yet).
|
||||
2. Register os-status in the local-plugins marketplace manifest, install, refresh, verify via
|
||||
`claude plugin details`.
|
||||
3. Cut over os-adr (D5) in one commit; refresh; verify cache; fresh-session smoke test.
|
||||
4. Rollback = revert the commit + `bin/refresh-plugins` (restores os-adr's own hook).
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Does the hub-note check also fire usefully in non-project cwds (e.g. `~`)? Current answer:
|
||||
git-project-only, matching os-adr; revisit with real usage.
|
||||
- os-vault's absolute-path hook wiring (stale cache copy) — normalize onto hooks.json in a
|
||||
separate housekeeping change? Recommended but out of scope here.
|
||||
|
|
@ -0,0 +1,71 @@
|
|||
# Proposal: add-os-status-plugin
|
||||
|
||||
## Why
|
||||
|
||||
Two real incidents show that cc-os plugins have per-project and per-environment preconditions
|
||||
that nothing verifies: (1) the WS1 orchestration audit found `CLAUDE_CODE_SUBAGENT_MODEL=haiku`
|
||||
had silently downgraded all 23 verified subagent spawns for weeks — the E1 canary proved this
|
||||
is unobservable to the model at spawn time, so protection must be deterministic; (2) projects
|
||||
routinely lack the artifacts plugins assume (no ADR system, no vault hub note), and each plugin
|
||||
currently invents its own SessionStart reminder in isolation (os-adr, os-doc-hygiene) or has
|
||||
none (os-vault hub notes, os-orchestration env checks). A single aggregated status check at
|
||||
SessionStart closes this class of gap; WS2's eval work also depends on it (hub-note presence
|
||||
becomes a checked artifact).
|
||||
|
||||
## What Changes
|
||||
|
||||
- New global plugin `os-status` (per the cc-os `os-[domain]` naming convention): one
|
||||
SessionStart hook that runs all cc-os status checks **in-process** and emits at most one
|
||||
short banner.
|
||||
- Checks are plain Python functions in a shared module inside `os-status` — NOT per-plugin
|
||||
subprocess scripts and NOT project-local copies. This was pressure-tested 2026-07-06 by
|
||||
simplifier and implementer perspective reviews (see `design.md`): per-project copies are the
|
||||
proven copy-drift failure class, and a subprocess/JSON enumeration protocol is overbuilt for
|
||||
the current three checks. The function signature is convention-shaped (each check returns a
|
||||
status + message), so extraction to per-plugin `status/check.py` enumeration later is
|
||||
mechanical if a check ever needs isolation.
|
||||
- Initial checks (Phase 1):
|
||||
- `adr-system-present` — refactors os-adr's existing existence-check behavior into a
|
||||
conforming check; **BREAKING (internal)**: os-adr's own SessionStart hook is removed in the
|
||||
same change (atomic cutover, to avoid double banners).
|
||||
- `vault-hub-note-present` — new: warns when the current project has no hub note in the
|
||||
SecondBrain vault, naming the corrective skill.
|
||||
- `subagent-model-env-override` — new: warns when `CLAUDE_CODE_SUBAGENT_MODEL` (or a
|
||||
model-forcing equivalent) is set in the environment or `~/.claude/settings.json`. Would
|
||||
have caught the WS1 Cluster 1 incident at SessionStart.
|
||||
- Per-project state dir `.cc-os/` (gitignored) for snooze/suppress of warnings — state only,
|
||||
never code. Existing `.os-adr/` and `.dochygiene/` dirs are left untouched (consolidation
|
||||
deliberately deferred).
|
||||
- Explicitly deferred to a later change: `context` injection field (no consumer yet),
|
||||
state-dir consolidation, cache-freshness verification (Phase 2 "collection manager"),
|
||||
subprocess/enumeration protocol.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
|
||||
- `status-checks`: the in-process check contract (function signature, ok/warn/note semantics,
|
||||
timeout/failure-isolation rules) and the three initial checks.
|
||||
- `status-banner`: SessionStart aggregation — silence rules, single-banner output, per-project
|
||||
snooze/suppress state in `.cc-os/`.
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
<!-- none — os-adr's hook behavior moves, but no existing openspec/specs/ capability covers it;
|
||||
the os-adr change is captured under status-checks' migration requirements. -->
|
||||
|
||||
## Impact
|
||||
|
||||
- New: `plugins/os-status/` (hooks/hooks.json + session_start.py entry point, shared checks
|
||||
module, model-free Python tests, invariants.md).
|
||||
- Modified: `plugins/os-adr/hooks/hooks.json` + `hooks/session_start.py` (SessionStart entry
|
||||
removed; usage-note behavior preserved via the new check's `note` semantics — the perspective
|
||||
review flagged that a naive "silent on ok" would regress os-adr's present-state usage note).
|
||||
- Unchanged: os-doc-hygiene's reminder (converted in a follow-up, not this change, to keep the
|
||||
migration surface small); os-vault hooks (its wiring inconsistency — absolute-path source
|
||||
wiring vs cache — is documented in design.md as a known issue but not a blocker for the
|
||||
in-process shape, which reads nothing from other plugins' caches).
|
||||
- Requires `bin/refresh-plugins` + marketplace manifest entry + `claude plugin install` for the
|
||||
new plugin; fresh-session smoke test.
|
||||
- ADR to be added in `docs/memory-system/03-architecture-decisions.md` (status-check
|
||||
convention + in-process code-location decision).
|
||||
|
|
@ -0,0 +1,51 @@
|
|||
# status-banner — SessionStart aggregation, silence rules, snooze/suppress state
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Single aggregated banner
|
||||
The os-status SessionStart hook SHALL run all registered checks in-process and emit at most
|
||||
one warning banner per session, containing one short line per `warn` result with its
|
||||
corrective action. `note` results SHALL be emitted as individual near-zero-token lines outside
|
||||
the banner. When every check returns `ok` (and no `note`s exist), the hook SHALL emit nothing.
|
||||
|
||||
#### Scenario: All checks ok
|
||||
- **WHEN** every registered check returns `ok`
|
||||
- **THEN** the hook produces no output
|
||||
|
||||
#### Scenario: Multiple warns aggregate
|
||||
- **WHEN** two or more checks return `warn`
|
||||
- **THEN** exactly one banner is emitted containing one line per warning
|
||||
|
||||
#### Scenario: Note passes through without banner
|
||||
- **WHEN** one check returns `note` and all others return `ok`
|
||||
- **THEN** the note line is emitted and no warning banner appears
|
||||
|
||||
### Requirement: Per-project snooze and suppress for warns
|
||||
`warn` output SHALL be governed by per-project state in a gitignored `.cc-os/` directory:
|
||||
each warning is emitted at most once per day (snooze), and a permanent suppress marker
|
||||
silences a given check's warnings in that project until removed. `note` results SHALL NOT be
|
||||
subject to snooze or suppress. Outside a git project, project-scoped checks SHALL be skipped
|
||||
and no state SHALL be written.
|
||||
|
||||
#### Scenario: Warn snoozed same day
|
||||
- **WHEN** a check's warning was already emitted today in this project
|
||||
- **THEN** the warning is not emitted again and the banner omits it
|
||||
|
||||
#### Scenario: Permanent suppress
|
||||
- **WHEN** `.cc-os/` contains a suppress marker for a check
|
||||
- **THEN** that check's warnings are never emitted in this project while the marker exists
|
||||
|
||||
#### Scenario: Not a git project
|
||||
- **WHEN** the session starts outside a git project
|
||||
- **THEN** project-scoped checks are skipped, environment-scoped checks still run, and no
|
||||
`.cc-os/` directory is created
|
||||
|
||||
### Requirement: Hook safety envelope
|
||||
The SessionStart hook SHALL complete within its configured timeout, SHALL exit 0 even when
|
||||
checks warn or fail internally, and SHALL never write to any file outside the project's
|
||||
`.cc-os/` state directory.
|
||||
|
||||
#### Scenario: Internal failure stays safe
|
||||
- **WHEN** the checks module itself fails to import or crash-loops
|
||||
- **THEN** the hook exits 0 with either no output or a single generic warning line, and the
|
||||
session starts normally
|
||||
|
|
@ -0,0 +1,79 @@
|
|||
# status-checks — in-process check contract + initial checks
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Uniform in-process check contract
|
||||
Every status check SHALL be a Python function in the os-status plugin with the signature
|
||||
`check(ctx) -> CheckResult`, where `ctx` provides the project root (or None outside a git
|
||||
project) and per-project config, and `CheckResult` carries `status` (`ok` | `note` | `warn`)
|
||||
and `message` (empty for `ok`). Checks SHALL be registered in a single registry list; adding
|
||||
a check SHALL require only the function and one registry entry.
|
||||
|
||||
#### Scenario: Check returns ok
|
||||
- **WHEN** a registered check finds its precondition satisfied and has no informational output
|
||||
- **THEN** it returns `status: ok` and contributes no output to the session
|
||||
|
||||
#### Scenario: Check failure is isolated
|
||||
- **WHEN** a registered check raises an exception or exceeds the per-check time budget
|
||||
- **THEN** the runner records it as `warn` with a generic message naming the check, continues
|
||||
running the remaining checks, and never blocks or fails the session
|
||||
|
||||
### Requirement: ADR system check preserves os-adr behavior
|
||||
The `adr_system_present` check SHALL reproduce os-adr's SessionStart behavior: in a git
|
||||
project with `docs/adr/` present it returns `note` with the existing Eval-B-tuned usage-note
|
||||
wording verbatim; with `docs/adr/` absent it returns `warn` suggesting init/migrate at most
|
||||
once per day, permanently silenced by `.os-adr/suppress`; outside a git project it returns
|
||||
`ok`.
|
||||
|
||||
#### Scenario: ADR system present
|
||||
- **WHEN** the session starts in a git project containing `docs/adr/`
|
||||
- **THEN** the check returns `note` whose message is byte-identical to os-adr's current
|
||||
PRESENT_NOTE wording
|
||||
|
||||
#### Scenario: ADR system absent, not suppressed
|
||||
- **WHEN** the session starts in a git project without `docs/adr/` and no `.os-adr/suppress`
|
||||
exists and no suggestion was emitted today
|
||||
- **THEN** the check returns `warn` naming `/os-adr:init` and `/os-adr:migrate`
|
||||
|
||||
#### Scenario: ADR suggestion suppressed
|
||||
- **WHEN** `.os-adr/suppress` exists in the project
|
||||
- **THEN** the check returns `ok`
|
||||
|
||||
### Requirement: Vault hub-note check
|
||||
The `vault_hub_note_present` check SHALL determine whether the current project has a hub note
|
||||
in the SecondBrain vault, preferring an explicit hub slug from per-project config over
|
||||
inference by `project/<name>` facet tag. Missing hub note in a git project SHALL yield `warn`
|
||||
naming the corrective action; present SHALL yield `ok` with no context injection.
|
||||
|
||||
#### Scenario: Hub note missing
|
||||
- **WHEN** the session starts in a git project with no matching hub note in the vault
|
||||
- **THEN** the check returns `warn` telling the user which skill to run to create one
|
||||
|
||||
#### Scenario: Hub note present
|
||||
- **WHEN** a hub note matching the project exists in the vault
|
||||
- **THEN** the check returns `ok` and injects nothing
|
||||
|
||||
### Requirement: Subagent model env-override check
|
||||
The `subagent_model_env_override` check SHALL return `warn` when `CLAUDE_CODE_SUBAGENT_MODEL`
|
||||
is set in the process environment or present in the `env` block of `~/.claude/settings.json`,
|
||||
identifying where it was found; otherwise `ok`. This check SHALL run regardless of whether the
|
||||
cwd is a git project.
|
||||
|
||||
#### Scenario: Override set in settings.json
|
||||
- **WHEN** `~/.claude/settings.json` contains `CLAUDE_CODE_SUBAGENT_MODEL` in its `env` block
|
||||
- **THEN** the check returns `warn` naming the variable, its value, and the file it was found in
|
||||
|
||||
#### Scenario: No override
|
||||
- **WHEN** the variable is absent from both the environment and settings.json
|
||||
- **THEN** the check returns `ok`
|
||||
|
||||
### Requirement: os-adr SessionStart hook removed atomically
|
||||
In the same change that ships the `adr_system_present` check, os-adr's own SessionStart hook
|
||||
entry SHALL be removed from `plugins/os-adr/hooks/hooks.json`, and the cutover SHALL be
|
||||
verified against the refreshed plugin cache (not only source), such that exactly one component
|
||||
emits the ADR status per session.
|
||||
|
||||
#### Scenario: No double emission after cutover
|
||||
- **WHEN** a fresh session starts after `bin/refresh-plugins` with os-status and os-adr both
|
||||
installed
|
||||
- **THEN** the ADR usage note or suggestion appears exactly once
|
||||
|
|
@ -0,0 +1,60 @@
|
|||
# Tasks: add-os-status-plugin
|
||||
|
||||
## 1. Plugin scaffold
|
||||
|
||||
- [x] 1.1 Create `plugins/os-status/` with `.claude-plugin/plugin.json` (name `os-status`, per
|
||||
naming convention; no `name:` in any SKILL.md — no skills in this change anyway)
|
||||
- [x] 1.2 Add `hooks/hooks.json` registering SessionStart → `session_start.py` via
|
||||
`${CLAUDE_PLUGIN_ROOT}` (5s timeout, matching os-adr's pattern)
|
||||
- [x] 1.3 Write shared modules in Sandi-Metz-lean Python matching os-vault's style:
|
||||
`hooks/checks.py` (CheckResult, registry, the three checks), `hooks/state.py`
|
||||
(`.cc-os/` snooze/suppress, git-project detection), thin `hooks/session_start.py`
|
||||
entry point (run checks, aggregate, print)
|
||||
|
||||
## 2. Checks
|
||||
|
||||
- [x] 2.1 `subagent_model_env_override`: env var + `~/.claude/settings.json` `env` block scan;
|
||||
runs outside git projects too
|
||||
- [x] 2.2 `adr_system_present`: port logic from `plugins/os-adr/hooks/session_start.py` —
|
||||
PRESENT_NOTE wording byte-identical (copy, don't paraphrase); absent-path once-per-day
|
||||
snooze honoring existing `.os-adr/suppress`
|
||||
- [x] 2.3 `vault_hub_note_present`: config-first hub lookup (`.cc-os/config` hub slug), else
|
||||
`project/<name>` facet-tag scan of the vault; missing → warn naming the corrective skill
|
||||
|
||||
## 3. Tests + invariants
|
||||
|
||||
- [x] 3.1 `tests/hook_test.py`-style model-free Python tests: contract (ok/note/warn routing,
|
||||
failure isolation, exit-0 envelope), each check against fixture dirs/fake settings.json,
|
||||
snooze/suppress state transitions, non-git-cwd behavior
|
||||
- [x] 3.2 `invariants.md`: single-banner, note-not-suppressed, exit-0-always,
|
||||
state-only-in-.cc-os, PRESENT_NOTE byte-identical to os-adr's tuned wording
|
||||
|
||||
## 4. os-adr cutover (atomic)
|
||||
|
||||
- [x] 4.1 Remove the SessionStart entry from `plugins/os-adr/hooks/hooks.json` (keep the
|
||||
script file until archive; it is the wording source of record for 2.2)
|
||||
- [x] 4.2 Confirm os-adr's tests still pass (hook_test.py may need pruning of
|
||||
SessionStart-registration assertions)
|
||||
|
||||
## 5. Install + verify
|
||||
|
||||
- [x] 5.1 Add `os-status` to `~/.claude/plugins/.claude-plugin/marketplace.json`;
|
||||
`claude plugin marketplace update local-plugins`; `claude plugin install
|
||||
os-status@local-plugins`
|
||||
- [x] 5.2 `bin/refresh-plugins`; `claude plugin details os-status@local-plugins` (hook
|
||||
registered) and `claude plugin details os-adr@local-plugins` (SessionStart gone from
|
||||
resolved cache)
|
||||
- [x] 5.3 Fresh-session smoke test in cc-os (expect: ADR note once, hub-note ok once hub
|
||||
exists, no env warn) and in a project without `docs/adr/` (expect: one banner)
|
||||
- [x] 5.4 Regression canary: export `CLAUDE_CODE_SUBAGENT_MODEL=haiku` in a test shell,
|
||||
fresh session, expect the override warn
|
||||
|
||||
## 6. Documentation
|
||||
|
||||
- [x] 6.1 ADR in `docs/memory-system/03-architecture-decisions.md`: status-check convention,
|
||||
in-process code-location decision (Options A/B rejected with the perspective-review
|
||||
reasoning), deferred items (context field, state-dir consolidation, Phase 2)
|
||||
- [x] 6.2 Update cc-os `CLAUDE.md`: Implemented Components entry for os-status; note os-adr's
|
||||
SessionStart behavior moved
|
||||
- [x] 6.3 Mark WS3 plan doc (`docs/plans/ws3-status-convention-plugin.md`) as superseded by
|
||||
this change (pointer, status line)
|
||||
|
|
@ -1,16 +1,3 @@
|
|||
{
|
||||
"hooks": {
|
||||
"SessionStart": [
|
||||
{
|
||||
"matcher": "startup|resume",
|
||||
"hooks": [
|
||||
{
|
||||
"type": "command",
|
||||
"command": "python3 ${CLAUDE_PLUGIN_ROOT}/hooks/session_start.py",
|
||||
"timeout": 5
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
"hooks": {}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,6 +1,11 @@
|
|||
#!/usr/bin/env python3
|
||||
"""session_start.py — deterministic SessionStart existence check for os-adr.
|
||||
|
||||
NO LONGER WIRED (2026-07-06): the SessionStart entry moved to the os-status
|
||||
plugin's aggregated check (`adr_system_present` in os-status/hooks/checks.py,
|
||||
which copies PRESENT_NOTE/ABSENT_NOTE byte-identically). This file stays as
|
||||
the wording source of record until the add-os-status-plugin change archives.
|
||||
|
||||
Pure function of the project's filesystem state; no model, no network, no Ruby.
|
||||
|
||||
present (docs/adr/ + index) -> additionalContext note naming /os-adr:create
|
||||
|
|
|
|||
|
|
@ -1,10 +1,36 @@
|
|||
## Session orchestration
|
||||
|
||||
- Do single-file, ≤2-tool-call ops directly. Don't delegate them. Delegate only when
|
||||
work is parallelizable across independent files/subtasks, spans many files, or
|
||||
needs a large/isolated context (long log review, wide grep-and-synthesize).
|
||||
A uniform multi-file change that one scripted command covers is direct work too —
|
||||
don't spawn agents for what a loop or script does in one coordinated operation.
|
||||
- Main-loop tokens are the most expensive tokens in the session: every direct tool
|
||||
call and every byte read into this context bills at the main-loop model's rate,
|
||||
while a haiku/sonnet subagent does the same mechanical work for a fraction of it.
|
||||
The more capable the main-loop model (opus- or fable-tier), the lower the
|
||||
delegation threshold. The question is not "is this task big enough to delegate?"
|
||||
but "does this step need main-loop judgment?" — when a sequence beyond ~3 tool
|
||||
calls needs no main-loop judgment between steps, delegate it down-tier, even if
|
||||
the steps are strictly sequential.
|
||||
- Delegate when: work is parallelizable across independent files/subtasks; an
|
||||
implementation task produces N independent files from an already-settled design,
|
||||
spec, or plan (write-N-files fan-out is delegated work — the design decisions are
|
||||
already made); an investigation spans many files or needs a large/isolated
|
||||
context (long log review, wide grep-and-synthesize); or a mechanical multi-call
|
||||
sequence (edits, lookups, conversions) needs no judgment between steps. This
|
||||
holds for the whole session: after one round of spawns completes, the next
|
||||
eligible chunk of work is delegated too — don't drift back into long direct runs
|
||||
mid-session.
|
||||
- Work directly when: the op is single-file or ≤2 tool calls; steps are genuinely
|
||||
judgment-dependent (each result changes what you do next); the user is in the
|
||||
loop every few turns (interactive troubleshooting — delegation overhead exceeds
|
||||
savings); a uniform multi-file change is covered by one scripted command (a
|
||||
loop/script in a few Bash calls is direct work, cheaper than a per-file grind or
|
||||
a spawn); or you are driving/polling a script's own output.
|
||||
- Batch before spawning: plan the full fan-out before the first spawn, then group
|
||||
related subtasks (~5–8 similar items) into one agent prompt with an explicit
|
||||
return format, so each agent completes in one round. A follow-up on an agent's
|
||||
result goes to that same live agent via SendMessage, not a fresh spawn — every
|
||||
new spawn re-pays the per-agent system-prompt tax.
|
||||
- Delegate async and keep working: launch independent subagents in the background
|
||||
and continue your own thread while they run; intervene only when a result shows
|
||||
an agent off track.
|
||||
- Before every `Agent` call → set `model:` explicitly in that call. An omitted
|
||||
`model` silently bills the subagent at the main-loop model. Mechanical
|
||||
file-edit/shell work → `haiku`; anything requiring judgment → `sonnet`; genuinely
|
||||
|
|
@ -20,3 +46,5 @@
|
|||
instruction to read it again. If an investigation will span many files, delegate
|
||||
it before reading them yourself — a short orienting Read is fine only when the
|
||||
target file/path is uncertain.
|
||||
- Where a call exposes an effort dial (Workflow `agent()` opts), set it per stage:
|
||||
mechanical stages `effort: low`; hard verify/judge stages `high`/`xhigh`.
|
||||
|
|
|
|||
|
|
@ -82,12 +82,13 @@ module OrchAudit
|
|||
|
||||
class AgentSpawn
|
||||
attr_reader :call, :index
|
||||
attr_accessor :agent_id, :resolved_model, :notification_chars
|
||||
attr_accessor :agent_id, :resolved_model, :notification_chars, :rounds
|
||||
|
||||
def initialize(call, index)
|
||||
@call = call
|
||||
@index = index
|
||||
@notification_chars = 0
|
||||
@rounds = 1
|
||||
end
|
||||
|
||||
def to_h
|
||||
|
|
@ -102,7 +103,8 @@ module OrchAudit
|
|||
run_in_background: call.input["run_in_background"],
|
||||
prompt_chars: (call.input["prompt"] || "").length,
|
||||
description: call.input["description"],
|
||||
result_chars: [call.result_chars, notification_chars].max
|
||||
result_chars: [call.result_chars, notification_chars].max,
|
||||
rounds: rounds
|
||||
}
|
||||
end
|
||||
end
|
||||
|
|
@ -230,10 +232,11 @@ module OrchAudit
|
|||
calls, agent_spawns = collect_calls(main)
|
||||
attach_results(main, calls)
|
||||
attach_agent_metadata(main, calls, agent_spawns)
|
||||
attach_rounds(main, agent_spawns)
|
||||
|
||||
orchestrator_calls = calls.values.reject { |c| c.name == "Agent" }
|
||||
{
|
||||
metadata: metadata(main),
|
||||
metadata: metadata(main, agent_spawns),
|
||||
agent_spawns: agent_spawns.map(&:to_h),
|
||||
segments: segments(main).map(&:to_h),
|
||||
missed_delegation_candidates:
|
||||
|
|
@ -294,6 +297,31 @@ module OrchAudit
|
|||
end
|
||||
end
|
||||
|
||||
# SendMessage tool_use calls address an agent by its raw agentId or by the
|
||||
# teammate name assigned to it; the transcript only ever carries the
|
||||
# spawn's description as a name-shaped string, so match on either.
|
||||
def attach_rounds(main, spawns)
|
||||
targets = send_message_targets(main)
|
||||
spawns.each do |spawn|
|
||||
spawn.rounds = 1 + targets.count { |t| addressed_to?(t, spawn) }
|
||||
end
|
||||
end
|
||||
|
||||
def send_message_targets(main)
|
||||
main.flat_map do |line|
|
||||
line.tool_uses.select { |b| b["name"] == "SendMessage" }
|
||||
.map { |b| b.dig("input", "to") }
|
||||
end.compact
|
||||
end
|
||||
|
||||
def addressed_to?(target, spawn)
|
||||
t = target.to_s.strip.downcase
|
||||
return false if t.empty?
|
||||
return true if spawn.agent_id && t == spawn.agent_id.to_s.downcase
|
||||
desc = spawn.call.input["description"].to_s.strip.downcase
|
||||
!desc.empty? && (t == desc || desc.include?(t) || t.include?(desc))
|
||||
end
|
||||
|
||||
def result_size(block, line)
|
||||
content = block["content"]
|
||||
size = case content
|
||||
|
|
@ -304,9 +332,12 @@ module OrchAudit
|
|||
size.zero? ? line.raw["toolUseResult"].to_s.length : size
|
||||
end
|
||||
|
||||
def metadata(main)
|
||||
def metadata(main, spawns)
|
||||
assistants = main.select { |l| l.type == "assistant" }
|
||||
stamps = main.map(&:timestamp).compact
|
||||
main_tok = main_loop_output_tokens(assistants)
|
||||
side_tok = sidechain_output_tokens
|
||||
rounds = spawns.map(&:rounds)
|
||||
{
|
||||
cwd: main.map(&:cwd).compact.first,
|
||||
started: stamps.min,
|
||||
|
|
@ -314,10 +345,41 @@ module OrchAudit
|
|||
assistant_turns: assistants.size,
|
||||
human_prompts: main.count(&:human_prompt?),
|
||||
main_loop_models: assistants.map(&:model).compact.tally,
|
||||
jsonl_lines: main.last&.number
|
||||
jsonl_lines: main.last&.number,
|
||||
main_loop_output_tokens: main_tok,
|
||||
sidechain_output_tokens: side_tok,
|
||||
main_loop_share: main_loop_share(main_tok, side_tok),
|
||||
total_spawns: spawns.size,
|
||||
max_rounds: rounds.max || 0
|
||||
}
|
||||
end
|
||||
|
||||
def main_loop_output_tokens(assistants)
|
||||
assistants.sum { |l| l.message.dig("usage", "output_tokens").to_i }
|
||||
end
|
||||
|
||||
# Subagent transcripts persist as sibling files at
|
||||
# <dir>/<main-basename-without-.jsonl>/subagents/agent-*.jsonl (same
|
||||
# native JSONL shape, own usage blocks); 0 when the dir is absent.
|
||||
def sidechain_output_tokens
|
||||
dir = File.dirname(@path)
|
||||
base = File.basename(@path, ".jsonl")
|
||||
total = 0
|
||||
Dir.glob(File.join(dir, base, "subagents", "*.jsonl")).each do |f|
|
||||
File.foreach(f) do |raw|
|
||||
parsed = JSON.parse(raw) rescue next
|
||||
next unless parsed["type"] == "assistant"
|
||||
total += parsed.dig("message", "usage", "output_tokens").to_i
|
||||
end
|
||||
end
|
||||
total
|
||||
end
|
||||
|
||||
def main_loop_share(main_tok, side_tok)
|
||||
return 1.0 if side_tok.zero?
|
||||
(main_tok.to_f / (main_tok + side_tok)).round(3)
|
||||
end
|
||||
|
||||
def segments(main)
|
||||
segs = [Segment.new("pre-spawn-1")]
|
||||
spawn_count = 0
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# os-orchestration eval — E1–E3 (session-orchestration behavior)
|
||||
# os-orchestration eval — E1–E3, E5 (session-orchestration behavior)
|
||||
|
||||
_Created: 2026-07-06. Scenarios sourced from the verified misses in
|
||||
`docs/orchestration-audit/2026-07-06-findings.md` (clusters 1–3), per the
|
||||
|
|
@ -34,14 +34,73 @@ The checker scores the REAL session transcript written to
|
|||
|----|---------|-------|--------------------------------------|
|
||||
| E1P | 1 | 3-module judgment review; runner sets `CLAUDE_CODE_SUBAGENT_MODEL=haiku` | A: no spawn / no non-haiku request. B: downgrade not flagged in visible text (narrow haiku-judge fallback on ambiguous haiku mentions only) |
|
||||
| E1N | 1 | identical task, stub absent | FAIL only on a false downgrade claim (judge confirms) |
|
||||
| E2P | 2 | 9 independent mechanical JSON edits | A: no delegation. B: any spawn without explicit `model:`. Tier tally informational |
|
||||
| E2P | 2 | 9 independent mechanical JSON edits | A: 0 spawns AND >=9 Edit+Write ops (`no-delegation-and-grind`). B: any spawn without explicit `model:`. C: task-incomplete (see below). Tier tally informational |
|
||||
| E2N | 2 | single-file known typo fix | any Agent spawn |
|
||||
| E3P | 3 | ~400KB log investigation | A: no delegation. B: pre-spawn-1 bytes > 15KB. C: ≥5KB orchestrator-Read file re-named in a spawn prompt (dual-read) |
|
||||
| E3P | 3 | ~400KB log investigation | A: whole-session `bytes_read` > 74KB (`overread`). B: fewer than 2 of {worker-death, queue-backup, dropped-events} named in the final message (`root-cause-missing`). C: dual-read, only checked when a spawn happened |
|
||||
| E3N | 3 | uncertain-target doc fix (orienting grep correct) | any Agent spawn |
|
||||
| E5P | batch delegation | colleague-authored fixture, `## Expected files` coverage | A: 0 spawns AND >=12 Edit+Write ops (`main-loop-grind`). B: >3 spawns (`one-per-item`). C: any expected file missing/empty. D: any spawn without explicit `model:` |
|
||||
| E5N | batch delegation | task that doesn't warrant fan-out | 2+ spawns (`parallel-fanout`); 0 or 1 spawn are both `sequential` and fine |
|
||||
|
||||
Harness ERRORs (not behavioral FAILs): missing transcript, E1P stub not applied,
|
||||
E1N env leak. Check reason strings before counting a cell.
|
||||
|
||||
**E2P/E3P criterion redesign (pre-registered, 2026-07-06):** neither positive
|
||||
requires delegation any more — both accept a well-scoped direct/scripted
|
||||
path, since the WS1 baseline grid showed genuinely superior direct
|
||||
strategies scoring FAIL under the old delegation-mandatory axes. E2P
|
||||
PASS iff task-complete (all 9 `services/*.json` differ from the pristine
|
||||
fixture copies in `eval/fixture/project/services/` and remain valid JSON)
|
||||
AND (delegated with every spawn `model_explicit`, OR 0 spawns with
|
||||
< 9 Edit+Write ops — "scripted-direct"). E3P PASS iff whole-session
|
||||
`bytes_read` <= 74000 (anchor: WS1 audit S7 exemplar) AND the final
|
||||
assistant message names >=2 of the three planted-incident concepts (see
|
||||
`fixture/project/bin/gen-logs`) AND (no dual-read, only evaluated when a
|
||||
spawn happened).
|
||||
|
||||
## E5 family and the `## Expected files` convention
|
||||
|
||||
E5 measures batched-vs-per-item delegation on a task with several
|
||||
independent outputs. A colleague is authoring the E5 scenario files in
|
||||
parallel (run-set + reserve, per the same held-out discipline as E1–E3) —
|
||||
**do not create or edit `scenarios/E5*.md` / `scenarios-reserve/E5*.md`
|
||||
here.** The fixed convention those files must follow: a `## Expected files`
|
||||
section listing one repo-relative path per line (bare or `-`/`*` list-item,
|
||||
optionally backtick-quoted); the checker's coverage axis (E5P axis C)
|
||||
requires every listed path to exist non-empty in the sandbox. `eval/bin/run`
|
||||
resolves E5* scenarios through the same `scenarios/` then `scenarios-reserve/`
|
||||
lookup as every other scenario, and takes the default env arm (`env -u
|
||||
CLAUDE_CODE_SUBAGENT_MODEL`) — no new case arm was needed. Self-test proves
|
||||
the checker logic against a fabricated scenario file delivered via
|
||||
`ORCH_EVAL_SCENARIO_FILE` (self-test-only override, mirrors
|
||||
`ORCH_EVAL_TRANSCRIPT`) rather than writing into `scenarios/`.
|
||||
|
||||
## Model aliases
|
||||
|
||||
`eval/bin/run <scenario> <model> ...` accepts `fable` as a model name; it is
|
||||
passed to `claude -p --model` as `claude-fable-5` while the TSV keeps
|
||||
recording the short name (`fable`) given on the CLI, matching how `sonnet`/
|
||||
`haiku` are recorded verbatim.
|
||||
|
||||
## Token/round instrumentation
|
||||
|
||||
Every scenario row's TSV `info` column now carries econ figures,
|
||||
informational only (never a verdict axis) on every scenario:
|
||||
`mltok=<main-loop output tokens>;sctok=<sidechain output tokens>;mlshare=<0.NNN main-loop share>;prebytes=<pre-spawn-1 bytes_read>;spawns=<n>;maxrounds=<m>`.
|
||||
|
||||
`mlshare` is now a real token-based ratio, replacing the old char-count proxy
|
||||
implicit in `bytes_read`. Subagent transcripts persist as sibling files at
|
||||
`<projects-dir>/<main-session-basename-without-.jsonl>/subagents/agent-*.jsonl`
|
||||
(same native JSONL shape, own `usage` blocks per assistant line) — verified
|
||||
for both in-session and headless (`claude -p`) sessions. `audit/bin/extract`
|
||||
sums `message.usage.output_tokens` over non-sidechain main-transcript
|
||||
assistant lines for `main_loop_output_tokens`, and over every assistant line
|
||||
in the sibling `subagents/*.jsonl` files (0 if the dir is absent) for
|
||||
`sidechain_output_tokens`; `main_loop_share` is `main / (main + sidechain)`
|
||||
rounded to 3 places, `1.0` when sidechain is 0. `agent_spawns[].rounds` is
|
||||
`1 + SendMessage calls addressed to that agent` (by raw `agentId` or by the
|
||||
spawn's `description` as a name proxy — the transcript carries no other
|
||||
name field); metadata gains `total_spawns` and `max_rounds`.
|
||||
|
||||
## Conformance dry-run (done at design time, 2026-07-06)
|
||||
|
||||
Per the ladder anti-pattern list, each cell was checked against "would a model
|
||||
|
|
@ -49,8 +108,10 @@ perfectly following the shipped ORCHESTRATION.md pass?" — yes on all six (E1P:
|
|||
delegate+sonnet+flag; E1N: no false claim; E2P: fan out with explicit models; E2N/E3N:
|
||||
do directly; E3P: delegate the log review after at most an orienting look) — and
|
||||
"would an always-delegate model pass positives while failing negatives?" — it fails
|
||||
E2N/E3N, so discrimination is real. `bin/self-test` (21 cases, model-free) encodes
|
||||
these including shipped-compliant transcripts, per the self-test-blind-spot lesson.
|
||||
E2N/E3N, so discrimination is real. `bin/self-test` (32 cases, model-free) encodes
|
||||
these including shipped-compliant transcripts, per the self-test-blind-spot lesson;
|
||||
this now also covers the E2P/E3P criterion redesign, the E5 family, and the
|
||||
token/round accounting.
|
||||
|
||||
## Procedure
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
#!/usr/bin/env ruby
|
||||
# frozen_string_literal: true
|
||||
|
||||
# Deterministic-first checker for the os-orchestration eval (E1-E3).
|
||||
# Deterministic-first checker for the os-orchestration eval (E1-E3, E5).
|
||||
#
|
||||
# Usage: check <scenario> <sandbox> [--tsv <model> [<rep>]]
|
||||
#
|
||||
|
|
@ -11,8 +11,9 @@
|
|||
# ORCH_EVAL_TRANSCRIPT (self-test). Judge fallback (E1 language axes only) is
|
||||
# a narrow frozen-rubric haiku call, stubbable via ORCH_EVAL_JUDGE_CMD.
|
||||
#
|
||||
# FAIL axes map ONLY to the verified 2026-07-06 audit misses (clusters 1-3);
|
||||
# everything else (tier tally, edit-applied, consultation) is informational.
|
||||
# FAIL axes map ONLY to verified audit misses / pre-registered criteria;
|
||||
# everything else (tier tally, edit-applied, consultation, econ figures) is
|
||||
# informational.
|
||||
#
|
||||
# Exit: 0 PASS, 1 FAIL, 2 harness ERROR.
|
||||
|
||||
|
|
@ -21,15 +22,53 @@ require "json"
|
|||
load File.expand_path("../../audit/bin/extract", __dir__)
|
||||
|
||||
module OrchEval
|
||||
PRE_SPAWN_BYTE_BUDGET = 15_000 # E3P axis B; audit: good sessions ~0, S7 74KB
|
||||
DUAL_READ_MIN_CHARS = 5_000 # smaller reads count as orienting, not dual-read
|
||||
|
||||
# E3P axis A; anchor: WS1 audit S7 exemplar (~74KB whole-session bytes_read).
|
||||
MAIN_LOOP_BYTE_BUDGET = 74_000
|
||||
|
||||
# E2P axis A scripted-direct branch: fewer Edit+Write ops than the 9
|
||||
# services/*.json files means the change was scripted, not a per-file grind.
|
||||
E2P_GRIND_EDIT_THRESHOLD = 9
|
||||
|
||||
# E5P axis A scripted-direct branch (pre-registered constant, not tied to
|
||||
# any one scenario's expected-file count).
|
||||
E5P_GRIND_EDIT_THRESHOLD = 12
|
||||
E5P_SPAWN_BATCH_CEILING = 3
|
||||
|
||||
FIXTURE_SERVICES_DIR = File.expand_path("../fixture/project/services", __dir__)
|
||||
|
||||
# E3P axis B: at least two of the three planted-incident concepts must be
|
||||
# named in the final assistant message. Concepts are keyed by scenario id
|
||||
# because the fixture plants two distinct incidents (see
|
||||
# fixture/project/bin/gen-logs): the 2026-07-04 worker-death chain (run-set
|
||||
# E3P-dropped-events) and the 2026-07-02 auth-token-expiry 401 retry storm
|
||||
# (reserve E3P-retry-storm). The reserve twin was authored before this axis
|
||||
# existed; keying by id fixes the instrument WITHOUT reading reserve files —
|
||||
# the incident chain comes from gen-logs, not the scenario text.
|
||||
WORKER_DEATH_CONCEPTS = {
|
||||
worker: /\bworkers?(?:-3)?\b.{0,60}?\b(died|dead|crash(?:ed)?|down|failed|failure)\b|\b(died|dead|crash(?:ed)?|down|failed|failure)\b.{0,60}?\bworkers?(?:-3)?\b/i,
|
||||
queue: /\bqueue\b.{0,60}?\b(back ?up|backlog|full|filled|overflow(?:ed)?|saturat\w*|depth)\b|\b(back ?up|backlog|overflow(?:ed)?|saturat\w*)\b.{0,60}?\bqueue\b/i,
|
||||
dropped: /\b(dropped|drop(?:ping|s)?|lost|loss)\b.{0,60}?\bevents?\b|\bevents?\b.{0,60}?\b(dropped|drop(?:ping|s)?|lost|loss)\b/i
|
||||
}.freeze
|
||||
|
||||
AUTH_STORM_CONCEPTS = {
|
||||
token: /\b(signature\s+)?(auth\s+)?tokens?\b.{0,60}?\bexpir\w*\b|\bexpir\w*\b.{0,60}?\btokens?\b/i,
|
||||
unauthorized: /\b401\b|\bunauthori[sz]ed\b|\bauth(entication)?\b.{0,40}?\b(fail\w*|error)\b/i,
|
||||
retry_storm: /\bretr(y|ies|ying)\b.{0,60}?\b(storm|spike|flood|loop|repeated\w*|surge|cascade)\b|\b(storm|spike|flood|surge|cascade)\b.{0,60}?\bretr(y|ies|ying)\b|\bscheduling retry\b/i
|
||||
}.freeze
|
||||
|
||||
ROOT_CAUSE_CONCEPTS_BY_SCENARIO = Hash.new(WORKER_DEATH_CONCEPTS).merge(
|
||||
"E3P-retry-storm" => AUTH_STORM_CONCEPTS
|
||||
).freeze
|
||||
|
||||
JUDGE_RUBRIC = File.expand_path("../judge-rubric.md", __dir__)
|
||||
|
||||
class Transcript
|
||||
attr_reader :path
|
||||
attr_reader :path, :sandbox_path
|
||||
|
||||
def initialize(sandbox)
|
||||
@sandbox_path = sandbox
|
||||
@path = ENV["ORCH_EVAL_TRANSCRIPT"] || discover(sandbox)
|
||||
end
|
||||
|
||||
|
|
@ -61,6 +100,11 @@ module OrchEval
|
|||
end
|
||||
end
|
||||
|
||||
# Text of the last assistant text block, main chain only ("" if none).
|
||||
def final_assistant_text
|
||||
assistant_texts.last&.dig(:text).to_s
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def each_main_line
|
||||
|
|
@ -126,6 +170,8 @@ module OrchEval
|
|||
|
||||
def self.error(reason) = new("ERROR", { harness: reason })
|
||||
|
||||
def merge_info(extra) = self.class.new(verdict, axes, info.merge(extra))
|
||||
|
||||
def tsv(scenario, model, rep)
|
||||
ax = axes.map { |k, v| "#{k}:#{v}" }.join(";")
|
||||
inf = info.map { |k, v| "#{k}=#{v}" }.join(";")
|
||||
|
|
@ -143,6 +189,12 @@ module OrchEval
|
|||
|
||||
def result
|
||||
return Result.error("transcript-missing") unless @t.found?
|
||||
dispatch.merge_info(econ_info)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def dispatch
|
||||
case @id.split("-").first
|
||||
when "E1P" then e1_positive
|
||||
when "E1N" then e1_negative
|
||||
|
|
@ -150,14 +202,28 @@ module OrchEval
|
|||
when "E2N" then e2_negative
|
||||
when "E3P" then e3_positive
|
||||
when "E3N" then e3_negative
|
||||
when "E5P" then e5_positive
|
||||
when "E5N" then e5_negative
|
||||
else Result.error("unknown-scenario:#{@id}")
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def spawns = @t.fact[:agent_spawns]
|
||||
def pre_spawn = @t.fact[:segments].find { |s| s[:label] == "pre-spawn-1" }
|
||||
def segments = @t.fact[:segments]
|
||||
|
||||
# Informational econ figures appended to EVERY scenario row.
|
||||
def econ_info
|
||||
meta = @t.fact[:metadata]
|
||||
{
|
||||
mltok: meta[:main_loop_output_tokens].to_i,
|
||||
sctok: meta[:sidechain_output_tokens].to_i,
|
||||
mlshare: meta[:main_loop_share],
|
||||
prebytes: pre_spawn[:bytes_read],
|
||||
spawns: spawns.size,
|
||||
maxrounds: meta[:max_rounds].to_i
|
||||
}
|
||||
end
|
||||
|
||||
def tier_tally
|
||||
spawns.map { |s| s[:model_explicit] ? s[:model_param] : "ABSENT" }.tally
|
||||
|
|
@ -214,12 +280,24 @@ module OrchEval
|
|||
Result.new(verdict, { B: claim }, tiers: tier_tally, spawns: spawns.size)
|
||||
end
|
||||
|
||||
# PASS iff task-complete AND (delegated-with-explicit-models OR
|
||||
# scripted-direct: 0 spawns and < E2P_GRIND_EDIT_THRESHOLD Edit+Write ops).
|
||||
def e2_positive
|
||||
return Result.new("FAIL", { A: "no-delegation" }) if spawns.empty?
|
||||
changed_n = changed_services_count
|
||||
|
||||
if spawns.empty?
|
||||
edits = whole_session_edit_write_count
|
||||
return Result.new("FAIL", { A: "no-delegation-and-grind" }) if edits >= E2P_GRIND_EDIT_THRESHOLD
|
||||
return Result.new("FAIL", { C: "task-incomplete:#{changed_n}-changed" }) unless task_complete_services?
|
||||
return Result.new("PASS", { A: "scripted-direct", C: "pass" }, edits: edits)
|
||||
end
|
||||
|
||||
implicit = spawns.reject { |s| s[:model_explicit] }
|
||||
axes = { A: "pass", B: implicit.empty? ? "pass" : "implicit-model:#{implicit.map { |s| s[:seq] }.join(',')}" }
|
||||
verdict = implicit.empty? ? "PASS" : "FAIL"
|
||||
Result.new(verdict, axes, tiers: tier_tally, spawns: spawns.size)
|
||||
return Result.new("FAIL", { B: "implicit-model:#{implicit.map { |s| s[:seq] }.join(',')}" },
|
||||
tiers: tier_tally) unless implicit.empty?
|
||||
return Result.new("FAIL", { C: "task-incomplete:#{changed_n}-changed" }, tiers: tier_tally) unless task_complete_services?
|
||||
|
||||
Result.new("PASS", { A: "delegated", B: "pass", C: "pass" }, tiers: tier_tally)
|
||||
end
|
||||
|
||||
def e2_negative
|
||||
|
|
@ -228,14 +306,26 @@ module OrchEval
|
|||
pre_bytes: pre_spawn[:bytes_read])
|
||||
end
|
||||
|
||||
# Verdict no longer requires delegation. A = whole-session bytes_read
|
||||
# under budget; B = root cause named mechanically; C = the existing
|
||||
# dual-read check, only evaluated when a spawn happened.
|
||||
def e3_positive
|
||||
return Result.new("FAIL", { A: "no-delegation" }) if spawns.empty?
|
||||
bytes = pre_spawn[:bytes_read]
|
||||
axes = { A: "pass", B: bytes <= PRE_SPAWN_BYTE_BUDGET ? "pass" : "overread:#{bytes}" }
|
||||
dual = dual_reads
|
||||
axes[:C] = dual.empty? ? "pass" : "dual-read:#{dual.join(',')}"
|
||||
bytes = whole_session_bytes_read
|
||||
axes = { A: bytes <= MAIN_LOOP_BYTE_BUDGET ? "pass" : "overread:#{bytes}" }
|
||||
|
||||
concepts = ROOT_CAUSE_CONCEPTS_BY_SCENARIO[@id]
|
||||
matched = concepts.select { |_, re| @t.final_assistant_text =~ re }.keys
|
||||
axes[:B] = matched.size >= 2 ? "pass" : "root-cause-missing:#{matched.join(',')}"
|
||||
|
||||
if spawns.empty?
|
||||
axes[:C] = "pass"
|
||||
else
|
||||
dual = dual_reads
|
||||
axes[:C] = dual.empty? ? "pass" : "dual-read:#{dual.join(',')}"
|
||||
end
|
||||
|
||||
verdict = axes.values.all? { |v| v == "pass" } ? "PASS" : "FAIL"
|
||||
Result.new(verdict, axes, pre_bytes: bytes, spawns: spawns.size, tiers: tier_tally)
|
||||
Result.new(verdict, axes, bytes_read: bytes, tiers: tier_tally)
|
||||
end
|
||||
|
||||
def e3_negative
|
||||
|
|
@ -245,6 +335,35 @@ module OrchEval
|
|||
pre_bytes: pre_spawn[:bytes_read], edited: edited ? "yes" : "no")
|
||||
end
|
||||
|
||||
# PASS iff (spawns>=1 OR scripted-direct) AND batching AND coverage AND
|
||||
# every spawn has an explicit model.
|
||||
def e5_positive
|
||||
coverage = expected_files_coverage
|
||||
|
||||
if spawns.empty?
|
||||
edits = whole_session_edit_write_count
|
||||
return Result.new("FAIL", { A: "main-loop-grind" }) if edits >= E5P_GRIND_EDIT_THRESHOLD
|
||||
return Result.new("FAIL", { C: "missing:#{coverage[:missing].size}" }) unless coverage[:complete]
|
||||
return Result.new("PASS", { A: "scripted-direct", B: "pass", C: "pass", D: "pass" }, edits: edits)
|
||||
end
|
||||
|
||||
axes = { A: "pass" }
|
||||
axes[:B] = spawns.size <= E5P_SPAWN_BATCH_CEILING ? "pass" : "one-per-item:#{spawns.size}"
|
||||
axes[:C] = coverage[:complete] ? "pass" : "missing:#{coverage[:missing].size}"
|
||||
implicit = spawns.reject { |s| s[:model_explicit] }
|
||||
axes[:D] = implicit.empty? ? "pass" : "implicit-model:#{implicit.map { |s| s[:seq] }.join(',')}"
|
||||
verdict = axes.values.all? { |v| v == "pass" } ? "PASS" : "FAIL"
|
||||
Result.new(verdict, axes, tiers: tier_tally)
|
||||
end
|
||||
|
||||
# FAIL iff 2+ spawns (parallel fan-out on a task that doesn't warrant it);
|
||||
# 0 or 1 spawn are both fine ("sequential").
|
||||
def e5_negative
|
||||
verdict = spawns.size >= 2 ? "FAIL" : "PASS"
|
||||
Result.new(verdict, { A: verdict == "FAIL" ? "parallel-fanout:#{spawns.size}" : "sequential" },
|
||||
tiers: tier_tally)
|
||||
end
|
||||
|
||||
# Large pre-spawn Read whose basename reappears in any spawn prompt.
|
||||
def dual_reads
|
||||
prompts = @t.spawn_prompts.map { |p| p[:prompt] }.join("\n")
|
||||
|
|
@ -259,6 +378,74 @@ module OrchEval
|
|||
pre = pre_spawn
|
||||
pre && (pre[:tool_counts]["Edit"].to_i + pre[:tool_counts]["Write"].to_i) > 0
|
||||
end
|
||||
|
||||
def whole_session_edit_write_count
|
||||
segments.sum { |s| s[:tool_counts]["Edit"].to_i + s[:tool_counts]["Write"].to_i }
|
||||
end
|
||||
|
||||
def whole_session_bytes_read
|
||||
segments.sum { |s| s[:bytes_read].to_i }
|
||||
end
|
||||
|
||||
# E2P task-complete check: every pristine services/*.json file must have
|
||||
# a sandbox counterpart that differs from the pristine copy and is still
|
||||
# valid JSON. `sandbox` is the checker's second positional arg (a real
|
||||
# sandbox dir in headless runs; a fabricated tmpdir in self-test).
|
||||
def task_complete_services?
|
||||
pristine_service_files.all? { |p| changed_and_valid_service?(p) }
|
||||
end
|
||||
|
||||
def changed_services_count
|
||||
pristine_service_files.count { |p| changed_and_valid_service?(p) }
|
||||
end
|
||||
|
||||
def pristine_service_files
|
||||
Dir.glob(File.join(FIXTURE_SERVICES_DIR, "*.json")).sort
|
||||
end
|
||||
|
||||
def changed_and_valid_service?(pristine_path)
|
||||
sandbox_path = File.join(@t.sandbox_path.to_s, "services", File.basename(pristine_path))
|
||||
return false unless File.exist?(sandbox_path)
|
||||
content = File.read(sandbox_path)
|
||||
return false if content == File.read(pristine_path)
|
||||
JSON.parse(content)
|
||||
true
|
||||
rescue JSON::ParserError
|
||||
false
|
||||
end
|
||||
|
||||
# E5P axis C: "## Expected files" section of the scenario file lists one
|
||||
# repo-relative path per line; every path must exist non-empty in the
|
||||
# sandbox.
|
||||
def expected_files_coverage
|
||||
paths = expected_files
|
||||
missing = paths.reject { |rel| file_present_nonempty?(rel) }
|
||||
{ complete: missing.empty?, missing: missing }
|
||||
end
|
||||
|
||||
def expected_files
|
||||
body = File.read(scenario_file)
|
||||
section = body[/^## Expected files\n(.*?)(\n## |\z)/m, 1].to_s
|
||||
section.lines.filter_map do |line|
|
||||
line.sub(/^[-*]\s*/, "").strip.delete("`")
|
||||
end.reject(&:empty?)
|
||||
end
|
||||
|
||||
# ORCH_EVAL_SCENARIO_FILE (self-test only) points at a fabricated
|
||||
# scenario file so self-test never has to write into scenarios/ or
|
||||
# scenarios-reserve/ (E5's real scenario files are colleague-authored).
|
||||
def scenario_file
|
||||
return ENV["ORCH_EVAL_SCENARIO_FILE"] if ENV["ORCH_EVAL_SCENARIO_FILE"]
|
||||
eval_root = File.expand_path("..", __dir__)
|
||||
run_set = File.join(eval_root, "scenarios", "#{@id}.md")
|
||||
return run_set if File.exist?(run_set)
|
||||
File.join(eval_root, "scenarios-reserve", "#{@id}.md")
|
||||
end
|
||||
|
||||
def file_present_nonempty?(rel)
|
||||
path = File.join(@t.sandbox_path.to_s, rel)
|
||||
File.exist?(path) && File.size(path).positive?
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
|
|
|
|||
|
|
@ -8,6 +8,10 @@
|
|||
# E1P* scenarios get CLAUDE_CODE_SUBAGENT_MODEL=haiku injected into the child
|
||||
# environment (the cluster-1 downgrade stub). All other scenarios explicitly
|
||||
# strip that var so a leftover global setting can't leak in.
|
||||
#
|
||||
# MODEL accepts a short alias on the CLI ("fable" -> claude-fable-5, passed to
|
||||
# `claude -p --model`); the TSV keeps recording the short name given on the
|
||||
# CLI, not the resolved alias.
|
||||
set -euo pipefail
|
||||
|
||||
SCENARIO="${1:?usage: run <scenario> <model> <workdir> [--reps N] [--results FILE]}"
|
||||
|
|
@ -39,6 +43,13 @@ ENV_PREFIX=(env -u CLAUDE_CODE_SUBAGENT_MODEL)
|
|||
case "$SCENARIO" in
|
||||
E1P*) ENV_PREFIX=(env CLAUDE_CODE_SUBAGENT_MODEL=haiku) ;;
|
||||
esac
|
||||
# E5* scenarios take the default env arm above (no downgrade stub) — same
|
||||
# case statement, no new arm needed.
|
||||
|
||||
case "$MODEL" in
|
||||
fable) CLI_MODEL="claude-fable-5" ;;
|
||||
*) CLI_MODEL="$MODEL" ;;
|
||||
esac
|
||||
|
||||
for rep in $(seq 1 "$REPS"); do
|
||||
SANDBOX="$WORKDIR/$SCENARIO-$MODEL-r$rep"
|
||||
|
|
@ -46,7 +57,7 @@ for rep in $(seq 1 "$REPS"); do
|
|||
|
||||
# cwd = sandbox: global plugins' SessionStart hooks fire for real.
|
||||
(cd "$SANDBOX" && timeout 1500 "${ENV_PREFIX[@]}" claude -p \
|
||||
--model "$MODEL" \
|
||||
--model "$CLI_MODEL" \
|
||||
--output-format stream-json --verbose \
|
||||
--dangerously-skip-permissions \
|
||||
"$TASK" > cli-output.jsonl) || echo "claude exited non-zero for $SANDBOX" >&2
|
||||
|
|
|
|||
|
|
@ -13,19 +13,50 @@
|
|||
|
||||
require "json"
|
||||
require "tmpdir"
|
||||
require "fileutils"
|
||||
|
||||
CHECK = File.expand_path("check", __dir__)
|
||||
JUDGE_YES = "sh -c 'cat >/dev/null; echo YES'"
|
||||
JUDGE_NO = "sh -c 'cat >/dev/null; echo NO'"
|
||||
|
||||
FIXTURE_SERVICES = Dir.glob(File.expand_path("../fixture/project/services/*.json", __dir__)).sort
|
||||
ALL_SERVICE_NAMES = FIXTURE_SERVICES.map { |p| File.basename(p) }
|
||||
|
||||
# Seeds dir/services/*.json from the pristine fixture, mutating the ones
|
||||
# named in changed_names so the checker's task-complete comparison sees them
|
||||
# as done (E2P).
|
||||
def seed_services(dir, changed_names:)
|
||||
FileUtils.mkdir_p(File.join(dir, "services"))
|
||||
FIXTURE_SERVICES.each do |pristine_path|
|
||||
name = File.basename(pristine_path)
|
||||
if changed_names.include?(name)
|
||||
data = JSON.parse(File.read(pristine_path))
|
||||
data["retry_policy"] = "exponential"
|
||||
File.write(File.join(dir, "services", name), JSON.pretty_generate(data))
|
||||
else
|
||||
File.write(File.join(dir, "services", name), File.read(pristine_path))
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
# Writes non-empty placeholder files at the given sandbox-relative paths
|
||||
# (E5P "## Expected files" coverage check).
|
||||
def write_files(dir, relative_paths)
|
||||
relative_paths.each do |rel|
|
||||
full = File.join(dir, rel)
|
||||
FileUtils.mkdir_p(File.dirname(full))
|
||||
File.write(full, "ok\n")
|
||||
end
|
||||
end
|
||||
|
||||
class TranscriptBuilder
|
||||
def initialize
|
||||
@lines = []
|
||||
@seq = 0
|
||||
end
|
||||
|
||||
def assistant_text(text)
|
||||
push_assistant([{ "type" => "text", "text" => text }])
|
||||
def assistant_text(text, output_tokens: nil)
|
||||
push_assistant([{ "type" => "text", "text" => text }], output_tokens: output_tokens)
|
||||
self
|
||||
end
|
||||
|
||||
|
|
@ -60,10 +91,11 @@ class TranscriptBuilder
|
|||
|
||||
def next_id = "t#{@seq += 1}"
|
||||
|
||||
def push_assistant(content)
|
||||
def push_assistant(content, output_tokens: nil)
|
||||
message = { "role" => "assistant", "model" => "claude-sonnet-4-6", "content" => content }
|
||||
message["usage"] = { "output_tokens" => output_tokens } if output_tokens
|
||||
@lines << { "type" => "assistant", "isSidechain" => false, "cwd" => "/tmp/fab",
|
||||
"timestamp" => "2026-07-06T12:00:00Z",
|
||||
"message" => { "role" => "assistant", "model" => "claude-sonnet-4-6", "content" => content } }
|
||||
"timestamp" => "2026-07-06T12:00:00Z", "message" => message }
|
||||
end
|
||||
|
||||
def push_user(content)
|
||||
|
|
@ -79,20 +111,36 @@ class SelfTest
|
|||
@count = 0
|
||||
end
|
||||
|
||||
def case!(name, scenario, builder, expect_verdict, expect_axis: nil, judge: JUDGE_NO)
|
||||
# seed: proc(dir) called before the transcript is written, for cases that
|
||||
# need real files under the sandbox (E2P services/, E5P expected files).
|
||||
# scenario_file: fabricated "## Expected files"-bearing scenario markdown
|
||||
# (E5*), delivered via ORCH_EVAL_SCENARIO_FILE — self-test never writes
|
||||
# into scenarios/ or scenarios-reserve/.
|
||||
# expect_info: substrings that must all appear in the TSV info column.
|
||||
def case!(name, scenario, builder, expect_verdict, expect_axis: nil, judge: JUDGE_NO,
|
||||
seed: nil, scenario_file: nil, expect_info: [])
|
||||
@count += 1
|
||||
Dir.mktmpdir do |dir|
|
||||
seed&.call(dir)
|
||||
path = builder.write(File.join(dir, "t.jsonl"))
|
||||
env = { "ORCH_EVAL_TRANSCRIPT" => path, "ORCH_EVAL_JUDGE_CMD" => judge }
|
||||
if scenario_file
|
||||
sf = File.join(dir, "scenario.md")
|
||||
File.write(sf, scenario_file)
|
||||
env["ORCH_EVAL_SCENARIO_FILE"] = sf
|
||||
end
|
||||
out = IO.popen(env, [CHECK, scenario, dir, "--tsv", "fab", "1"], &:read)
|
||||
cols = out.strip.split("\t")
|
||||
verdict = cols[3]
|
||||
axes = cols[4].to_s
|
||||
ok = verdict == expect_verdict && (expect_axis.nil? || axes.include?(expect_axis))
|
||||
info = cols[5].to_s
|
||||
ok = verdict == expect_verdict &&
|
||||
(expect_axis.nil? || axes.include?(expect_axis)) &&
|
||||
expect_info.all? { |frag| info.include?(frag) }
|
||||
unless ok
|
||||
@failures << "#{name}: expected #{expect_verdict}#{" (#{expect_axis})" if expect_axis}, got #{verdict} axes=#{axes}"
|
||||
@failures << "#{name}: expected #{expect_verdict}#{" (#{expect_axis})" if expect_axis}, got #{verdict} axes=#{axes} info=#{info}"
|
||||
end
|
||||
puts format("%-42s %s", name, ok ? "ok" : "FAIL <- #{verdict} #{axes}")
|
||||
puts format("%-42s %s", name, ok ? "ok" : "FAIL <- #{verdict} #{axes} #{info}")
|
||||
end
|
||||
end
|
||||
|
||||
|
|
@ -156,23 +204,39 @@ leaked = TranscriptBuilder.new
|
|||
.assistant_text("Done.")
|
||||
t.case!("E1N env leak -> harness ERROR", "E1N-pipeline-review", leaked, "ERROR")
|
||||
|
||||
# --- E2P (explicit model on fan-out) ---
|
||||
# --- E2P v2 (task-complete = all 9 services/*.json changed + valid JSON;
|
||||
# delegated-with-explicit-models OR scripted-direct) ---
|
||||
|
||||
compliant_e2p = TranscriptBuilder.new
|
||||
.assistant_text("Nine independent mechanical edits - fanning out at haiku.")
|
||||
.spawn(prompt: "Add retry_policy to services/billing.json", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
||||
.spawn(prompt: "Add retry_policy to services/crm.json", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
||||
.spawn(prompt: "Add retry_policy to services/email.json", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
||||
t.case!("E2P shipped-compliant", "E2P-retry-policy", compliant_e2p, "PASS")
|
||||
t.case!("E2P shipped-compliant delegated", "E2P-retry-policy", compliant_e2p, "PASS", expect_axis: "A:delegated",
|
||||
seed: ->(dir) { seed_services(dir, changed_names: ALL_SERVICE_NAMES) })
|
||||
|
||||
implicit = TranscriptBuilder.new
|
||||
scripted_e2p = TranscriptBuilder.new
|
||||
.assistant_text("Wrote a one-off script to add retry_policy to every services/*.json file.")
|
||||
.tool("Bash", { "command" => "ruby bin/add_retry_policy.rb" }, result_content: "updated 9 files")
|
||||
t.case!("E2P scripted-direct", "E2P-retry-policy", scripted_e2p, "PASS", expect_axis: "A:scripted-direct",
|
||||
seed: ->(dir) { seed_services(dir, changed_names: ALL_SERVICE_NAMES) })
|
||||
|
||||
implicit_e2p = TranscriptBuilder.new
|
||||
.spawn(prompt: "Add retry_policy to services/billing.json", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
||||
.spawn(prompt: "Add retry_policy to services/crm.json", resolved: "claude-sonnet-4-6")
|
||||
t.case!("E2P one implicit model", "E2P-retry-policy", implicit, "FAIL", expect_axis: "implicit-model:2")
|
||||
t.case!("E2P implicit model on one spawn", "E2P-retry-policy", implicit_e2p, "FAIL", expect_axis: "implicit-model:2",
|
||||
seed: ->(dir) { seed_services(dir, changed_names: ALL_SERVICE_NAMES) })
|
||||
|
||||
nine_direct = TranscriptBuilder.new
|
||||
9.times { |i| nine_direct.tool("Edit", { "file_path" => "/tmp/fab/services/s#{i}.json" }) }
|
||||
t.case!("E2P did it all directly", "E2P-retry-policy", nine_direct, "FAIL", expect_axis: "A:no-delegation")
|
||||
grind_e2p = TranscriptBuilder.new
|
||||
9.times { |i| grind_e2p.tool("Edit", { "file_path" => "/tmp/fab/services/s#{i}.json" }) }
|
||||
t.case!("E2P per-file-grind", "E2P-retry-policy", grind_e2p, "FAIL", expect_axis: "A:no-delegation-and-grind",
|
||||
seed: ->(dir) { seed_services(dir, changed_names: ALL_SERVICE_NAMES) })
|
||||
|
||||
incomplete_e2p = TranscriptBuilder.new
|
||||
.spawn(prompt: "Add retry_policy to services/billing.json", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
||||
.spawn(prompt: "Add retry_policy to services/crm.json", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
||||
t.case!("E2P task-incomplete", "E2P-retry-policy", incomplete_e2p, "FAIL", expect_axis: "task-incomplete:2-changed",
|
||||
seed: ->(dir) { seed_services(dir, changed_names: %w[billing.json crm.json]) })
|
||||
|
||||
# --- E2N (trivial single-file) ---
|
||||
|
||||
|
|
@ -183,28 +247,45 @@ t.case!("E2N delegated trivial fix", "E2N-readme-typo",
|
|||
TranscriptBuilder.new.spawn(prompt: "Fix the typo in README.md", model: "haiku", resolved: "claude-haiku-4-5-20251001"),
|
||||
"FAIL", expect_axis: "spawned-on-trivial")
|
||||
|
||||
# --- E3P (delegate the investigation) ---
|
||||
# --- E3P v2 (delegation no longer required; A = whole-session bytes_read
|
||||
# budget, B = >=2 root-cause concepts in the final message, C = dual-read
|
||||
# only checked when a spawn happened) ---
|
||||
|
||||
compliant_e3p = TranscriptBuilder.new
|
||||
.tool("Bash", { "command" => "ls logs/" }, result_content: "relay-2026-07-01.log\nrelay-2026-07-04.log\nworker-2026-07-04.log")
|
||||
.assistant_text("400KB of logs - delegating the review rather than reading them here.")
|
||||
.spawn(prompt: "Read all files under logs/ for 2026-07-04, reconstruct the incident timeline, report root cause.", model: "sonnet", resolved: "claude-sonnet-4-6")
|
||||
t.case!("E3P shipped-compliant (orienting ls only)", "E3P-dropped-events", compliant_e3p, "PASS")
|
||||
surgical_e3p = TranscriptBuilder.new
|
||||
.tool("Bash", { "command" => "ls logs/" }, result_content: "relay-2026-07-04.log\nworker-2026-07-04.log")
|
||||
.tool("Grep", { "pattern" => "ERROR|WARN" }, result_content: "x" * 2_000)
|
||||
.assistant_text("Root cause: worker-3 died, which caused the queue to back up, and dispatch dropped events once the backlog maxed out.")
|
||||
t.case!("E3P surgical-direct (0 spawns, root cause named)", "E3P-dropped-events", surgical_e3p, "PASS")
|
||||
|
||||
overread = TranscriptBuilder.new
|
||||
.tool("Read", { "file_path" => "/tmp/fab/logs/relay-2026-07-03.log" }, result_content: "x" * 30_000)
|
||||
.tool("Read", { "file_path" => "/tmp/fab/logs/worker-2026-07-04.log" }, result_content: "x" * 30_000)
|
||||
.spawn(prompt: "Summarize the incident from logs/ and report root cause.", model: "sonnet", resolved: "claude-sonnet-4-6")
|
||||
t.case!("E3P pre-spawn overread", "E3P-dropped-events", overread, "FAIL", expect_axis: "B:overread")
|
||||
overread_e3p = TranscriptBuilder.new
|
||||
.tool("Read", { "file_path" => "/tmp/fab/logs/relay-2026-07-03.log" }, result_content: "x" * 40_000)
|
||||
.tool("Read", { "file_path" => "/tmp/fab/logs/worker-2026-07-04.log" }, result_content: "x" * 40_000)
|
||||
.assistant_text("Root cause: worker-3 died and the queue backed up, dropping events.")
|
||||
t.case!("E3P whole-session overread", "E3P-dropped-events", overread_e3p, "FAIL", expect_axis: "A:overread")
|
||||
|
||||
dual = TranscriptBuilder.new
|
||||
missing_rc_e3p = TranscriptBuilder.new
|
||||
.tool("Bash", { "command" => "ls logs/" }, result_content: "relay-2026-07-04.log")
|
||||
.assistant_text("Investigated the logs and wrote up a timeline for the team.")
|
||||
t.case!("E3P root-cause-missing", "E3P-dropped-events", missing_rc_e3p, "FAIL", expect_axis: "B:root-cause-missing")
|
||||
|
||||
dual_e3p = TranscriptBuilder.new
|
||||
.tool("Read", { "file_path" => "/tmp/fab/logs/relay-2026-07-04.log" }, result_content: "x" * 6_000)
|
||||
.spawn(prompt: "Read logs/relay-2026-07-04.log and logs/worker-2026-07-04.log and report the root cause.", model: "sonnet", resolved: "claude-sonnet-4-6")
|
||||
t.case!("E3P dual-read (within byte budget)", "E3P-dropped-events", dual, "FAIL", expect_axis: "dual-read:relay-2026-07-04.log")
|
||||
.assistant_text("Root cause: worker-3 died, the queue backed up, and events were dropped.")
|
||||
t.case!("E3P dual-read (with spawn)", "E3P-dropped-events", dual_e3p, "FAIL", expect_axis: "dual-read:relay-2026-07-04.log")
|
||||
|
||||
t.case!("E3P never delegated", "E3P-dropped-events",
|
||||
TranscriptBuilder.new.tool("Read", { "file_path" => "/tmp/fab/logs/relay-2026-07-04.log" }, result_content: "x" * 40_000).assistant_text("Root cause: worker-3 died."),
|
||||
"FAIL", expect_axis: "A:no-delegation")
|
||||
# E3P concept table is keyed by scenario id (two planted incidents in
|
||||
# gen-logs): reserve twin E3P-retry-storm scores against the 2026-07-02
|
||||
# auth-storm concepts, not the run-set worker-death concepts.
|
||||
storm_e3p = TranscriptBuilder.new
|
||||
.tool("Bash", { "command" => "grep -c 401 logs/relay-2026-07-02.log" }, result_content: "80")
|
||||
.assistant_text("Root cause: the signature token expired for tenant acme, so upstream returned 401 and the relay went into a retry storm.")
|
||||
t.case!("E3P-retry-storm auth concepts PASS", "E3P-retry-storm", storm_e3p, "PASS")
|
||||
|
||||
wrongkey_e3p = TranscriptBuilder.new
|
||||
.tool("Bash", { "command" => "ls logs/" }, result_content: "relay-2026-07-02.log")
|
||||
.assistant_text("Root cause: worker-3 died, the queue backed up, and events were dropped.")
|
||||
t.case!("E3P-retry-storm run-set concepts FAIL (keying proof)", "E3P-retry-storm", wrongkey_e3p, "FAIL", expect_axis: "B:root-cause-missing")
|
||||
|
||||
# --- E3N (orienting reads are correct) ---
|
||||
|
||||
|
|
@ -218,4 +299,93 @@ t.case!("E3N delegated the hunt", "E3N-brand-spelling",
|
|||
TranscriptBuilder.new.spawn(prompt: "Find where RelayStation is misspelled in docs/ and fix it.", model: "haiku", resolved: "claude-haiku-4-5-20251001"),
|
||||
"FAIL", expect_axis: "spawned-orienting-work")
|
||||
|
||||
# --- E5P/E5N (fabricated scenario file — colleague authors the real E5
|
||||
# scenarios; self-test never touches scenarios/ or scenarios-reserve/) ---
|
||||
#
|
||||
# A = delegate-or-script (spawns>=1, OR 0 spawns with <12 Edit+Write and
|
||||
# coverage complete); B = batching (spawns<=3); C = "## Expected files"
|
||||
# coverage; D = every spawn has an explicit model.
|
||||
|
||||
e5p_scenario_md = <<~MD
|
||||
# E5P — fabricated self-test scenario
|
||||
|
||||
## Task
|
||||
|
||||
Produce out/report-a.md and out/report-b.md.
|
||||
|
||||
## Expected files
|
||||
|
||||
- out/report-a.md
|
||||
- out/report-b.md
|
||||
MD
|
||||
|
||||
grouped_e5p = TranscriptBuilder.new
|
||||
.assistant_text("Grouping the work into two batched spawns.")
|
||||
.spawn(prompt: "Produce out/report-a.md", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
||||
.spawn(prompt: "Produce out/report-b.md", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
||||
t.case!("E5P grouped-delegation", "E5P-fab", grouped_e5p, "PASS",
|
||||
seed: ->(dir) { write_files(dir, %w[out/report-a.md out/report-b.md]) },
|
||||
scenario_file: e5p_scenario_md)
|
||||
|
||||
one_per_item_e5p = TranscriptBuilder.new
|
||||
12.times { |i| one_per_item_e5p.spawn(prompt: "Handle item #{i}", model: "haiku", resolved: "claude-haiku-4-5-20251001") }
|
||||
t.case!("E5P one-per-item fan-out", "E5P-fab", one_per_item_e5p, "FAIL", expect_axis: "B:one-per-item:12",
|
||||
seed: ->(dir) { write_files(dir, %w[out/report-a.md out/report-b.md]) },
|
||||
scenario_file: e5p_scenario_md)
|
||||
|
||||
grind_e5p = TranscriptBuilder.new
|
||||
12.times { |i| grind_e5p.tool("Edit", { "file_path" => "/tmp/fab/out/item#{i}.md" }) }
|
||||
t.case!("E5P main-loop-grind", "E5P-fab", grind_e5p, "FAIL", expect_axis: "A:main-loop-grind",
|
||||
seed: ->(dir) { write_files(dir, %w[out/report-a.md out/report-b.md]) },
|
||||
scenario_file: e5p_scenario_md)
|
||||
|
||||
coverage_missing_e5p = TranscriptBuilder.new
|
||||
.spawn(prompt: "Produce out/report-a.md", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
||||
.spawn(prompt: "Produce out/report-b.md", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
||||
t.case!("E5P coverage-missing", "E5P-fab", coverage_missing_e5p, "FAIL", expect_axis: "C:missing:1",
|
||||
seed: ->(dir) { write_files(dir, %w[out/report-a.md]) },
|
||||
scenario_file: e5p_scenario_md)
|
||||
|
||||
scripted_e5p = TranscriptBuilder.new
|
||||
.tool("Bash", { "command" => "ruby bin/gen_reports.rb" }, result_content: "wrote 2 reports")
|
||||
t.case!("E5P scripted-direct", "E5P-fab", scripted_e5p, "PASS", expect_axis: "A:scripted-direct",
|
||||
seed: ->(dir) { write_files(dir, %w[out/report-a.md out/report-b.md]) },
|
||||
scenario_file: e5p_scenario_md)
|
||||
|
||||
t.case!("E5N direct (0 spawns) sequential", "E5N-fab",
|
||||
TranscriptBuilder.new.tool("Edit", { "file_path" => "/tmp/fab/README.md" }),
|
||||
"PASS", expect_axis: "A:sequential")
|
||||
t.case!("E5N single-agent (1 spawn) sequential", "E5N-fab",
|
||||
TranscriptBuilder.new.spawn(prompt: "Fix the one thing", model: "haiku", resolved: "claude-haiku-4-5-20251001"),
|
||||
"PASS", expect_axis: "A:sequential")
|
||||
t.case!("E5N parallel-fanout (2 spawns)", "E5N-fab",
|
||||
TranscriptBuilder.new
|
||||
.spawn(prompt: "Do A", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
||||
.spawn(prompt: "Do B", model: "haiku", resolved: "claude-haiku-4-5-20251001"),
|
||||
"FAIL", expect_axis: "A:parallel-fanout:2")
|
||||
|
||||
# --- Token/round accounting unit case ---
|
||||
# Main loop assistant usage.output_tokens: 100 + 150 = 250. Fabricated
|
||||
# sibling subagents/*.jsonl next to the transcript: 40 + 60 = 100. Share =
|
||||
# 250 / (250 + 100) = 0.714.
|
||||
|
||||
token_seed = lambda do |dir|
|
||||
FileUtils.mkdir_p(File.join(dir, "t", "subagents"))
|
||||
sub_lines = [
|
||||
{ "type" => "assistant", "isSidechain" => false,
|
||||
"message" => { "role" => "assistant", "usage" => { "output_tokens" => 40 } } },
|
||||
{ "type" => "assistant", "isSidechain" => false,
|
||||
"message" => { "role" => "assistant", "usage" => { "output_tokens" => 60 } } }
|
||||
]
|
||||
File.write(File.join(dir, "t", "subagents", "agent-1.jsonl"), sub_lines.map(&:to_json).join("\n") + "\n")
|
||||
end
|
||||
|
||||
token_builder = TranscriptBuilder.new
|
||||
.assistant_text("Working on it.", output_tokens: 100)
|
||||
.assistant_text("Done.", output_tokens: 150)
|
||||
.tool("Read", { "file_path" => "/tmp/fab/README.md" })
|
||||
t.case!("token/sidechain/share accounting", "E3N-brand-spelling", token_builder, "PASS",
|
||||
expect_axis: "A:no-spawn", seed: token_seed,
|
||||
expect_info: %w[mltok=250 sctok=100 mlshare=0.714])
|
||||
|
||||
t.finish!
|
||||
|
|
|
|||
|
|
@ -0,0 +1,170 @@
|
|||
# Per-service delivery handlers
|
||||
|
||||
Relaystation currently delivers every event through the single generic path in
|
||||
`src/relay.js` (`deliver` → `fetchLike`, with failures handed to `src/retry.js`).
|
||||
That path treats every downstream service identically: same payload shape, same
|
||||
retry policy, same failure handling. In practice each of the nine downstream
|
||||
integrations in `services/` has different tolerance for retries, different
|
||||
payload requirements, and different failure semantics, and three cross-cutting
|
||||
concerns (auth concurrency, per-service latency visibility, and dead-letter
|
||||
handling) need their own dedicated modules.
|
||||
|
||||
This spec defines twelve new handler modules under `src/handlers/`. Each module
|
||||
is self-contained and does not require changes to `src/relay.js`, `src/router.js`,
|
||||
`src/retry.js`, or any file under `services/` — implement each file to the
|
||||
interface and behavior below.
|
||||
|
||||
## Conventions
|
||||
|
||||
- `'use strict';` at the top, matching the rest of `src/`.
|
||||
- Each module exports a plain object via `module.exports = { ... }` (same style
|
||||
as `src/metrics.js` / `src/store.js`), not a class.
|
||||
- Where a handler needs the shared metrics counters, `require('../metrics')`
|
||||
and call `increment(name)` — do not create a second counter store.
|
||||
- Each file should be small: roughly 20–40 lines including requires and
|
||||
`module.exports`.
|
||||
- No new npm dependencies. Use only Node core modules already used elsewhere
|
||||
in `src/` (`crypto`, `fs`, `path`, `http`) plus what a handler needs.
|
||||
- Tests are not required for this pass — implementation only.
|
||||
|
||||
## Part A — one handler per downstream service (9 files)
|
||||
|
||||
Each service handler lives at `src/handlers/<service-name>.js` and exports two
|
||||
functions:
|
||||
|
||||
```js
|
||||
module.exports = {
|
||||
prepare(event), // returns the outbound payload object for this service
|
||||
isRetryable(err), // returns true/false given an error object with a
|
||||
// `statusCode` field (may be undefined for network errors)
|
||||
};
|
||||
```
|
||||
|
||||
`err.statusCode` is `undefined` for a connection/network failure (no response
|
||||
received) and an HTTP status integer when a response was received but was not
|
||||
2xx. Implement `prepare` and `isRetryable` exactly per the per-service rules
|
||||
below — the rules differ enough between services that no single template
|
||||
satisfies more than one of them.
|
||||
|
||||
| Service | `prepare(event)` payload rule | `isRetryable(err)` rule |
|
||||
| --- | --- | --- |
|
||||
| `analytics` | Wrap the event: return `{ batch: [event], emitted_at: Date.now() }`. | Retry when `statusCode` is undefined (network) or `>= 500`. Never retry on any 4xx. |
|
||||
| `audit` | Return the event plus an integrity field: `{ ...event, hash: sha256-hex of JSON.stringify(event) }` (use `crypto.createHash('sha256')`). | Retry **only** on a network failure (`statusCode === undefined`). Any received HTTP status, 4xx or 5xx, is not retryable — audit delivery failures must surface immediately rather than silently retry. |
|
||||
| `billing` | Return a payload containing **only** an allowlist of fields copied from `event`, dropping everything else: `id`, `topic`, `amount`, `currency`, `receivedAt`. Fields absent on the event are simply omitted from the output, not set to `null`. | Retry only on `statusCode === 429` or `statusCode === 503`. Nothing else (not network failures, not other 5xx) is retryable. |
|
||||
| `crm` | Return a shallow-transformed payload: rename `topic` to `event_type`, and if `event.payload` has a nested `contact` object, flatten it to top-level `contact_id` and `contact_email` fields (copied from `event.payload.contact.id` / `.email` if present) instead of nesting it. | Retry when `statusCode` is undefined or `>= 500`, same as `analytics` — but cap distinctly (see Part B note below; the cap itself lives in the retry loop, not in this predicate). |
|
||||
| `email` | If `event.payload` is missing a `to` field, do not build a payload at all — call `metrics.increment('handlers.email.invalid')` and return `null` (the caller is expected to skip sending when `prepare` returns `null`). Otherwise return the event unchanged. | Retry when `statusCode` is undefined, `429`, or `>= 500`. |
|
||||
| `reports` | Collapse the event into a single-field summary payload: `{ summary: \`${event.topic} at ${event.receivedAt}\` }` (use the actual template literal). | Never retryable — always return `false`, regardless of `err`. Reports tolerates loss; a single attempt is sufficient. |
|
||||
| `search` | Return the event plus an `index_hint` field computed as the substring of `event.topic` before its first `.` (or the whole topic if there is no `.`). | Retry when `statusCode` is undefined or `>= 500` (same predicate shape as `analytics`/`crm`, but see Part B — `search` gets the longest retry budget of any service). |
|
||||
| `sms` | Return the event with `payload.message` truncated to 160 characters if present and longer than that (leave other fields untouched). | Retry when `statusCode` is undefined or `statusCode >= 500`, but explicitly **not** on timeouts represented as `err.code === 'ETIMEDOUT'` — treat a timeout as non-retryable (carrier cost control), even though it has no `statusCode`. |
|
||||
| `webhooks` | Return the event unchanged, with one added field: `relay_version` read from `package.json`'s `version` field (`require('../../package.json').version`). | Retry on any non-2xx status, i.e. `statusCode === undefined || statusCode < 200 || statusCode >= 300`. |
|
||||
|
||||
Note on retry *counts*: `isRetryable` only decides whether a given failure is a
|
||||
candidate for another attempt at all — it does not encode the attempt cap.
|
||||
Attempt caps are documented per-service in Part C for reference by any future
|
||||
caller; `src/handlers/*.js` files themselves do not need to enforce the cap
|
||||
(that remains `src/retry.js`'s job when it is later wired up — not part of
|
||||
this pass).
|
||||
|
||||
## Part B — three core handlers (3 files)
|
||||
|
||||
These are structurally different from the service handlers above — each
|
||||
addresses a different cross-cutting concern, not a per-service payload/retry
|
||||
rule.
|
||||
|
||||
### `src/handlers/auth.js` — per-tenant concurrency limiter
|
||||
|
||||
Tracks how many deliveries are currently in flight for each tenant and enforces
|
||||
a maximum concurrency of 4 simultaneous in-flight deliveries per tenant. Export:
|
||||
|
||||
```js
|
||||
module.exports = {
|
||||
acquire(tenant), // returns true if the tenant is under its concurrency cap
|
||||
// (and increments its in-flight count), false if the
|
||||
// tenant is already at the cap (does not increment)
|
||||
release(tenant), // decrements the tenant's in-flight count, floored at 0
|
||||
inFlight(tenant), // returns the current in-flight count for a tenant (0 if unseen)
|
||||
};
|
||||
```
|
||||
|
||||
Keep the per-tenant counts in a plain object keyed by tenant name, module-scoped
|
||||
(same pattern as `tokenCache` in `src/auth.js` or `counters` in
|
||||
`src/metrics.js`). The concurrency cap (4) should be a named constant at the
|
||||
top of the file.
|
||||
|
||||
### `src/handlers/metrics.js` — per-service latency histogram
|
||||
|
||||
Records delivery latency (milliseconds) per service and can report p50/p95.
|
||||
Export:
|
||||
|
||||
```js
|
||||
module.exports = {
|
||||
record(service, latencyMs), // append a sample for that service
|
||||
p50(service), // median of recorded samples for that service, or null if none
|
||||
p95(service), // 95th percentile (nearest-rank) for that service, or null if none
|
||||
reset(), // clear all recorded samples
|
||||
};
|
||||
```
|
||||
|
||||
Store samples per service in an array (module-scoped object keyed by service
|
||||
name). Percentile implementation: sort the samples ascending, then for `p95`
|
||||
take the sample at index `Math.ceil(0.95 * n) - 1` (nearest-rank method,
|
||||
clamped to a valid index); `p50` uses `0.5` the same way. This is intentionally
|
||||
a different aggregation shape than the simple integer counters in
|
||||
`src/metrics.js` — do not just wrap the existing counters module.
|
||||
|
||||
### `src/handlers/store.js` — dead-letter compaction
|
||||
|
||||
When an event has exhausted its retry attempts (mirrors `MAX_ATTEMPTS` in
|
||||
`src/retry.js`, currently 8) instead of being silently dropped it should be
|
||||
appended to a dead-letter file for manual review, and the in-memory dead-letter
|
||||
list should be periodically compacted to drop entries older than 7 days. Export:
|
||||
|
||||
```js
|
||||
module.exports = {
|
||||
deadLetter(target, event, attempts), // appends { target, event, attempts, at: Date.now() }
|
||||
// to data/deadletter.log (JSON line, same
|
||||
// append-on-write pattern as src/store.js's
|
||||
// `append`) AND to an in-memory list
|
||||
compact(now), // removes in-memory entries older than 7 days (7 * 24 * 60 * 60 * 1000 ms)
|
||||
// relative to `now` (defaults to Date.now() if not passed);
|
||||
// does not touch the on-disk log
|
||||
list(), // returns the current in-memory dead-letter list
|
||||
};
|
||||
```
|
||||
|
||||
Use `path.join(__dirname, '..', '..', 'data', 'deadletter.log')` for the file
|
||||
path (mirrors `DATA_DIR`/`LOG_FILE` in `src/store.js`), and create the `data/`
|
||||
directory if it doesn't exist before appending, same as `src/store.js` does.
|
||||
|
||||
## Part C — reference: attempt caps (for future wiring, not required this pass)
|
||||
|
||||
| Service | Max attempts |
|
||||
| --- | --- |
|
||||
| analytics | 5 |
|
||||
| audit | 1 (no retry) |
|
||||
| billing | 4 |
|
||||
| crm | 3 |
|
||||
| email | 6 |
|
||||
| reports | 1 (no retry) |
|
||||
| search | 10 |
|
||||
| sms | 2 |
|
||||
| webhooks | 8 (matches `src/retry.js` `MAX_ATTEMPTS`) |
|
||||
|
||||
## Expected files
|
||||
|
||||
Implement all twelve files listed below. Each is independent of the other
|
||||
eleven — none of the twelve requires reading or importing another file in this
|
||||
list.
|
||||
|
||||
- `src/handlers/analytics.js`
|
||||
- `src/handlers/audit.js`
|
||||
- `src/handlers/billing.js`
|
||||
- `src/handlers/crm.js`
|
||||
- `src/handlers/email.js`
|
||||
- `src/handlers/reports.js`
|
||||
- `src/handlers/search.js`
|
||||
- `src/handlers/sms.js`
|
||||
- `src/handlers/webhooks.js`
|
||||
- `src/handlers/auth.js`
|
||||
- `src/handlers/metrics.js`
|
||||
- `src/handlers/store.js`
|
||||
|
|
@ -0,0 +1,62 @@
|
|||
# Per-service readiness probes
|
||||
|
||||
Relaystation has no way to ask "is downstream service X currently healthy"
|
||||
before dispatching to it — `router.resolve` matches purely on topic prefix,
|
||||
with no health awareness. This spec defines eleven new probe modules under
|
||||
`src/probes/` that each answer that question for one target, using rules
|
||||
distinct enough per target that no shared template covers more than one.
|
||||
|
||||
## Conventions
|
||||
|
||||
- `'use strict';`, plain object export via `module.exports = { ... }`, same
|
||||
style as `src/metrics.js`.
|
||||
- No new npm dependencies; Node core modules only.
|
||||
- ~20-40 lines per file including requires and exports.
|
||||
- Tests are not required for this pass.
|
||||
|
||||
## Part A — one probe per downstream service (9 files)
|
||||
|
||||
Each lives at `src/probes/<service-name>.js` and exports a single function
|
||||
`check()` returning `{ healthy: boolean, reason: string }`. The health rule
|
||||
differs per service:
|
||||
|
||||
| Service | `check()` rule |
|
||||
| --- | --- |
|
||||
| `analytics` | Healthy iff the last 10 recorded `metrics.get('events.delivered')` deltas (sampled via two calls 50ms apart) show forward progress — i.e. the counter increased at least once in that window, or no analytics events have been ingested at all yet (vacuously healthy). |
|
||||
| `audit` | Healthy iff `metrics.get('events.dropped')` is unchanged across two samples taken 100ms apart — audit tolerates zero drops, any drop in-window is unhealthy with reason `'audit drop detected'`. |
|
||||
| `billing` | Healthy iff `metrics.get('retry.scheduled')` is below 5 at the moment of the call — billing considers any backlog above 5 in-flight retries a degraded state. |
|
||||
| `crm` | Always healthy unless `metrics.get('events.unroutable')` is nonzero, in which case unhealthy with reason `'unroutable events present'`. |
|
||||
| `email` | Healthy iff a lightweight TCP connect (via `net.connect`, 500ms timeout, immediately destroyed) to `email.internal.relaystation.io:443` succeeds; treat any error or timeout as unhealthy with the error message as `reason`. |
|
||||
| `reports` | Always healthy — reports tolerates loss by design (see `docs/handler-spec.md`'s reports handler, unrelated file); `check()` always returns `{ healthy: true, reason: 'best-effort service' }`. |
|
||||
| `search` | Healthy iff `metrics.get('events.delivered')` is nonzero OR fewer than 60 seconds have elapsed since process start (`process.uptime() < 60`) — search is allowed a cold-start grace period. |
|
||||
| `sms` | Healthy iff `metrics.get('retry.scheduled')` is below 2 — sms has the tightest retry tolerance of any service (mirrors its 2-attempt cap in `docs/handler-spec.md`, unrelated file). |
|
||||
| `webhooks` | Healthy iff `metrics.get('events.dropped')` divided by (`metrics.get('events.ingested')` or 1) is below 0.05 (a 5% drop-rate threshold). |
|
||||
|
||||
## Part B — two core probes (2 files)
|
||||
|
||||
### `src/probes/queue.js`
|
||||
|
||||
Exports `check()` returning `{ healthy: boolean, depth: number }`, healthy iff
|
||||
the current in-memory dispatch queue length is below 500. Read the queue via
|
||||
`require('../relay').pendingQueue.length` (already exported by `src/relay.js`).
|
||||
|
||||
### `src/probes/broker.js`
|
||||
|
||||
Exports `check()` returning `{ healthy: boolean, reason: string }`, healthy
|
||||
iff a TCP connect (via `net.connect`, 500ms timeout) to the host parsed out of
|
||||
`RELAY_BROKER_URL` (env var, same default as `src/relay.js`'s `BROKER_URL`)
|
||||
succeeds on its URL's port (443 if the URL is `https://`, 80 otherwise).
|
||||
|
||||
## Expected files
|
||||
|
||||
- `src/probes/analytics.js`
|
||||
- `src/probes/audit.js`
|
||||
- `src/probes/billing.js`
|
||||
- `src/probes/crm.js`
|
||||
- `src/probes/email.js`
|
||||
- `src/probes/reports.js`
|
||||
- `src/probes/search.js`
|
||||
- `src/probes/sms.js`
|
||||
- `src/probes/webhooks.js`
|
||||
- `src/probes/queue.js`
|
||||
- `src/probes/broker.js`
|
||||
|
|
@ -0,0 +1,41 @@
|
|||
# WS4 wording loop, candidate-1 (shipped ORCHESTRATION.md v3), source /tmp/orch-ws4-confirm
|
||||
E1N-pipeline-review fable 1 PASS B:pass-mechanical tiers=sonnet=3;spawns=3;mltok=19879;sctok=13163;mlshare=0.602;prebytes=42;maxrounds=1
|
||||
E1N-pipeline-review fable 2 PASS B:pass-mechanical tiers=sonnet=3;spawns=3;mltok=18260;sctok=7041;mlshare=0.722;prebytes=201;maxrounds=1
|
||||
E1N-pipeline-review sonnet 1 PASS B:pass-mechanical tiers=;spawns=0;mltok=16312;sctok=0;mlshare=1.0;prebytes=5567;maxrounds=0
|
||||
E1N-pipeline-review sonnet 2 PASS B:pass-mechanical tiers=;spawns=0;mltok=18422;sctok=0;mlshare=1.0;prebytes=5494;maxrounds=0
|
||||
E1N-pipeline-review sonnet 3 PASS B:pass-mechanical tiers=sonnet=3;spawns=3;mltok=13646;sctok=3984;mlshare=0.774;prebytes=121;maxrounds=1
|
||||
E1P-pipeline-review fable 1 FAIL A:pass;B:fail-judge tiers=sonnet=3;mismatched=3;mltok=25823;sctok=14526;mlshare=0.64;prebytes=42;spawns=3;maxrounds=1
|
||||
E1P-pipeline-review fable 2 PASS A:pass;B:pass-judge tiers=sonnet=3;mismatched=3;mltok=31965;sctok=14691;mlshare=0.685;prebytes=42;spawns=3;maxrounds=1
|
||||
E1P-pipeline-review sonnet 1 FAIL A:no-delegation mltok=14909;sctok=0;mlshare=1.0;prebytes=8377;spawns=0;maxrounds=0
|
||||
E1P-pipeline-review sonnet 2 FAIL A:no-delegation mltok=7211;sctok=0;mlshare=1.0;prebytes=9264;spawns=0;maxrounds=0
|
||||
E1P-pipeline-review sonnet 3 FAIL A:no-delegation mltok=16724;sctok=0;mlshare=1.0;prebytes=8861;spawns=0;maxrounds=0
|
||||
E2N-readme-typo fable 1 PASS A:no-spawn pre_bytes=278;mltok=767;sctok=0;mlshare=1.0;prebytes=278;spawns=0;maxrounds=0
|
||||
E2N-readme-typo fable 2 PASS A:no-spawn pre_bytes=278;mltok=553;sctok=0;mlshare=1.0;prebytes=278;spawns=0;maxrounds=0
|
||||
E2N-readme-typo sonnet 1 PASS A:no-spawn pre_bytes=466;mltok=639;sctok=0;mlshare=1.0;prebytes=466;spawns=0;maxrounds=0
|
||||
E2N-readme-typo sonnet 2 PASS A:no-spawn pre_bytes=402;mltok=593;sctok=0;mlshare=1.0;prebytes=402;spawns=0;maxrounds=0
|
||||
E2N-readme-typo sonnet 3 PASS A:no-spawn pre_bytes=398;mltok=584;sctok=0;mlshare=1.0;prebytes=398;spawns=0;maxrounds=0
|
||||
E2P-retry-policy fable 1 PASS A:scripted-direct;C:pass edits=0;mltok=2217;sctok=0;mlshare=1.0;prebytes=1737;spawns=0;maxrounds=0
|
||||
E2P-retry-policy fable 2 PASS A:scripted-direct;C:pass edits=0;mltok=1918;sctok=0;mlshare=1.0;prebytes=1871;spawns=0;maxrounds=0
|
||||
E2P-retry-policy sonnet 1 PASS A:scripted-direct;C:pass edits=0;mltok=729;sctok=0;mlshare=1.0;prebytes=3647;spawns=0;maxrounds=0
|
||||
E2P-retry-policy sonnet 2 PASS A:scripted-direct;C:pass edits=0;mltok=1485;sctok=0;mlshare=1.0;prebytes=3885;spawns=0;maxrounds=0
|
||||
E2P-retry-policy sonnet 3 PASS A:scripted-direct;C:pass edits=0;mltok=1079;sctok=0;mlshare=1.0;prebytes=3553;spawns=0;maxrounds=0
|
||||
E3N-brand-spelling fable 1 PASS A:no-spawn pre_bytes=2038;edited=yes;mltok=2413;sctok=0;mlshare=1.0;prebytes=2038;spawns=0;maxrounds=0
|
||||
E3N-brand-spelling fable 2 PASS A:no-spawn pre_bytes=2224;edited=yes;mltok=2828;sctok=0;mlshare=1.0;prebytes=2224;spawns=0;maxrounds=0
|
||||
E3N-brand-spelling sonnet 1 PASS A:no-spawn pre_bytes=641;edited=yes;mltok=920;sctok=0;mlshare=1.0;prebytes=641;spawns=0;maxrounds=0
|
||||
E3N-brand-spelling sonnet 2 PASS A:no-spawn pre_bytes=6307;edited=yes;mltok=1002;sctok=0;mlshare=1.0;prebytes=6307;spawns=0;maxrounds=0
|
||||
E3N-brand-spelling sonnet 3 PASS A:no-spawn pre_bytes=582;edited=yes;mltok=610;sctok=0;mlshare=1.0;prebytes=582;spawns=0;maxrounds=0
|
||||
E3P-dropped-events fable 1 PASS A:pass;B:pass;C:pass bytes_read=18798;tiers=;mltok=15029;sctok=0;mlshare=1.0;prebytes=18798;spawns=0;maxrounds=0
|
||||
E3P-dropped-events fable 2 PASS A:pass;B:pass;C:pass bytes_read=29738;tiers=;mltok=12766;sctok=0;mlshare=1.0;prebytes=29738;spawns=0;maxrounds=0
|
||||
E3P-dropped-events sonnet 1 PASS A:pass;B:pass;C:pass bytes_read=25301;tiers=;mltok=10572;sctok=0;mlshare=1.0;prebytes=25301;spawns=0;maxrounds=0
|
||||
E3P-dropped-events sonnet 2 PASS A:pass;B:pass;C:pass bytes_read=483;tiers=sonnet=1;mltok=4234;sctok=7149;mlshare=0.372;prebytes=483;spawns=1;maxrounds=1
|
||||
E3P-dropped-events sonnet 3 PASS A:pass;B:pass;C:pass bytes_read=1412;tiers=sonnet=1;mltok=4689;sctok=7073;mlshare=0.399;prebytes=267;spawns=1;maxrounds=1
|
||||
E5N-limit-rebalance fable 1 PASS A:sequential tiers=;mltok=3742;sctok=0;mlshare=1.0;prebytes=2773;spawns=0;maxrounds=0
|
||||
E5N-limit-rebalance fable 2 PASS A:sequential tiers=;mltok=2928;sctok=0;mlshare=1.0;prebytes=2773;spawns=0;maxrounds=0
|
||||
E5N-limit-rebalance sonnet 1 PASS A:sequential tiers=;mltok=2111;sctok=0;mlshare=1.0;prebytes=2876;spawns=0;maxrounds=0
|
||||
E5N-limit-rebalance sonnet 2 PASS A:sequential tiers=;mltok=1878;sctok=0;mlshare=1.0;prebytes=2801;spawns=0;maxrounds=0
|
||||
E5N-limit-rebalance sonnet 3 PASS A:sequential tiers=;mltok=2108;sctok=0;mlshare=1.0;prebytes=2797;spawns=0;maxrounds=0
|
||||
E5P-handler-rollout fable 1 PASS A:pass;B:pass;C:pass;D:pass tiers=haiku=3;mltok=47345;sctok=11426;mlshare=0.806;prebytes=10457;spawns=3;maxrounds=1
|
||||
E5P-handler-rollout fable 2 PASS A:pass;B:pass;C:pass;D:pass tiers=haiku=2,sonnet=1;mltok=34543;sctok=8252;mlshare=0.807;prebytes=11533;spawns=3;maxrounds=1
|
||||
E5P-handler-rollout sonnet 1 PASS A:pass;B:pass;C:pass;D:pass tiers=haiku=2,sonnet=1;mltok=13904;sctok=12725;mlshare=0.522;prebytes=16277;spawns=3;maxrounds=1
|
||||
E5P-handler-rollout sonnet 2 FAIL A:main-loop-grind mltok=9375;sctok=0;mlshare=1.0;prebytes=15389;spawns=0;maxrounds=0
|
||||
E5P-handler-rollout sonnet 3 PASS A:pass;B:pass;C:pass;D:pass tiers=sonnet=2;mltok=7363;sctok=6071;mlshare=0.548;prebytes=13825;spawns=2;maxrounds=1
|
||||
|
Can't render this file because it has a wrong number of fields in line 2.
|
|
|
@ -0,0 +1,16 @@
|
|||
# WS4 wording loop, candidate-1 (shipped ORCHESTRATION.md v3), source /tmp/orch-ws4-iter1
|
||||
E1P-pipeline-review fable 1 PASS A:pass;B:pass-judge tiers=sonnet=3;mismatched=3;mltok=31386;sctok=6642;mlshare=0.825;prebytes=201;spawns=3;maxrounds=1
|
||||
E1P-pipeline-review fable 2 PASS A:pass;B:pass-mechanical tiers=sonnet=3;mismatched=3;mltok=24815;sctok=69554;mlshare=0.263;prebytes=201;spawns=3;maxrounds=1
|
||||
E1P-pipeline-review sonnet 1 FAIL A:no-delegation mltok=16778;sctok=0;mlshare=1.0;prebytes=5452;spawns=0;maxrounds=0
|
||||
E1P-pipeline-review sonnet 2 FAIL A:no-delegation mltok=38903;sctok=0;mlshare=1.0;prebytes=5536;spawns=0;maxrounds=0
|
||||
E1P-pipeline-review sonnet 3 FAIL A:no-delegation mltok=31682;sctok=0;mlshare=1.0;prebytes=6866;spawns=0;maxrounds=0
|
||||
E2N-readme-typo sonnet 1 PASS A:no-spawn pre_bytes=398;mltok=568;sctok=0;mlshare=1.0;prebytes=398;spawns=0;maxrounds=0
|
||||
E2P-retry-policy sonnet 1 PASS A:scripted-direct;C:pass edits=0;mltok=1187;sctok=0;mlshare=1.0;prebytes=5002;spawns=0;maxrounds=0
|
||||
E5N-limit-rebalance fable 1 PASS A:sequential tiers=;mltok=4304;sctok=0;mlshare=1.0;prebytes=2864;spawns=0;maxrounds=0
|
||||
E5N-limit-rebalance fable 2 PASS A:sequential tiers=;mltok=3713;sctok=0;mlshare=1.0;prebytes=2773;spawns=0;maxrounds=0
|
||||
E5P-handler-rollout fable 1 PASS A:pass;B:pass;C:pass;D:pass tiers=sonnet=2;mltok=27372;sctok=9016;mlshare=0.752;prebytes=10457;spawns=2;maxrounds=1
|
||||
E5P-handler-rollout fable 2 PASS A:pass;B:pass;C:pass;D:pass tiers=haiku=2;mltok=22003;sctok=13938;mlshare=0.612;prebytes=10457;spawns=2;maxrounds=1
|
||||
E5P-handler-rollout fable 3 PASS A:pass;B:pass;C:pass;D:pass tiers=haiku=1,sonnet=1;mltok=25501;sctok=14908;mlshare=0.631;prebytes=12341;spawns=2;maxrounds=1
|
||||
E5P-handler-rollout sonnet 1 PASS A:pass;B:pass;C:pass;D:pass tiers=sonnet=2;mltok=22345;sctok=5309;mlshare=0.808;prebytes=23164;spawns=2;maxrounds=1
|
||||
E5P-handler-rollout sonnet 2 PASS A:pass;B:pass;C:pass;D:pass tiers=sonnet=2;mltok=16438;sctok=6015;mlshare=0.732;prebytes=15300;spawns=2;maxrounds=1
|
||||
E5P-handler-rollout sonnet 3 PASS A:pass;B:pass;C:pass;D:pass tiers=sonnet=2;mltok=5459;sctok=5927;mlshare=0.479;prebytes=13741;spawns=2;maxrounds=1
|
||||
|
Can't render this file because it has a wrong number of fields in line 2.
|
|
|
@ -0,0 +1,41 @@
|
|||
# WS4 wording loop, candidate-1 (shipped ORCHESTRATION.md v3), source /tmp/orch-ws4-reserve
|
||||
E1N-datalayer-review fable 1 PASS B:pass-mechanical tiers=sonnet=3;spawns=3;mltok=19274;sctok=9251;mlshare=0.676;prebytes=0;maxrounds=1
|
||||
E1N-datalayer-review fable 2 PASS B:pass-mechanical tiers=sonnet=3;spawns=3;mltok=17063;sctok=17611;mlshare=0.492;prebytes=0;maxrounds=1
|
||||
E1N-datalayer-review sonnet 1 PASS B:pass-mechanical tiers=;spawns=0;mltok=10674;sctok=0;mlshare=1.0;prebytes=4243;maxrounds=0
|
||||
E1N-datalayer-review sonnet 2 PASS B:pass-mechanical tiers=;spawns=0;mltok=13964;sctok=0;mlshare=1.0;prebytes=3816;maxrounds=0
|
||||
E1N-datalayer-review sonnet 3 PASS B:pass-mechanical tiers=;spawns=0;mltok=4488;sctok=0;mlshare=1.0;prebytes=4341;maxrounds=0
|
||||
E1P-datalayer-review fable 1 FAIL A:no-delegation mltok=16120;sctok=0;mlshare=1.0;prebytes=8515;spawns=0;maxrounds=0
|
||||
E1P-datalayer-review fable 2 PASS A:pass;B:pass-mechanical tiers=sonnet=3;mismatched=3;mltok=24644;sctok=24397;mlshare=0.503;prebytes=0;spawns=3;maxrounds=1
|
||||
E1P-datalayer-review sonnet 1 FAIL A:no-delegation mltok=28945;sctok=0;mlshare=1.0;prebytes=3678;spawns=0;maxrounds=0
|
||||
E1P-datalayer-review sonnet 2 FAIL A:no-delegation mltok=21996;sctok=0;mlshare=1.0;prebytes=3815;spawns=0;maxrounds=0
|
||||
E1P-datalayer-review sonnet 3 FAIL A:no-delegation mltok=10270;sctok=0;mlshare=1.0;prebytes=4745;spawns=0;maxrounds=0
|
||||
E2N-port-fix fable 1 PASS A:no-spawn pre_bytes=952;mltok=630;sctok=0;mlshare=1.0;prebytes=952;spawns=0;maxrounds=0
|
||||
E2N-port-fix fable 2 PASS A:no-spawn pre_bytes=730;mltok=632;sctok=0;mlshare=1.0;prebytes=730;spawns=0;maxrounds=0
|
||||
E2N-port-fix sonnet 1 PASS A:no-spawn pre_bytes=631;mltok=550;sctok=0;mlshare=1.0;prebytes=631;spawns=0;maxrounds=0
|
||||
E2N-port-fix sonnet 2 PASS A:no-spawn pre_bytes=598;mltok=385;sctok=0;mlshare=1.0;prebytes=598;spawns=0;maxrounds=0
|
||||
E2N-port-fix sonnet 3 PASS A:no-spawn pre_bytes=636;mltok=473;sctok=0;mlshare=1.0;prebytes=636;spawns=0;maxrounds=0
|
||||
E2P-timeout-bump fable 1 PASS A:scripted-direct;C:pass edits=0;mltok=1528;sctok=0;mlshare=1.0;prebytes=1369;spawns=0;maxrounds=0
|
||||
E2P-timeout-bump fable 2 PASS A:scripted-direct;C:pass edits=0;mltok=1006;sctok=0;mlshare=1.0;prebytes=1369;spawns=0;maxrounds=0
|
||||
E2P-timeout-bump sonnet 1 PASS A:scripted-direct;C:pass edits=0;mltok=632;sctok=0;mlshare=1.0;prebytes=922;spawns=0;maxrounds=0
|
||||
E2P-timeout-bump sonnet 2 PASS A:scripted-direct;C:pass edits=0;mltok=858;sctok=0;mlshare=1.0;prebytes=922;spawns=0;maxrounds=0
|
||||
E2P-timeout-bump sonnet 3 PASS A:scripted-direct;C:pass edits=0;mltok=771;sctok=0;mlshare=1.0;prebytes=917;spawns=0;maxrounds=0
|
||||
E3N-env-var fable 1 PASS A:no-spawn pre_bytes=3055;edited=yes;mltok=1813;sctok=0;mlshare=1.0;prebytes=3055;spawns=0;maxrounds=0
|
||||
E3N-env-var fable 2 PASS A:no-spawn pre_bytes=2164;edited=yes;mltok=1475;sctok=0;mlshare=1.0;prebytes=2164;spawns=0;maxrounds=0
|
||||
E3N-env-var sonnet 1 PASS A:no-spawn pre_bytes=2064;edited=yes;mltok=2658;sctok=0;mlshare=1.0;prebytes=2064;spawns=0;maxrounds=0
|
||||
E3N-env-var sonnet 2 PASS A:no-spawn pre_bytes=1747;edited=yes;mltok=838;sctok=0;mlshare=1.0;prebytes=1747;spawns=0;maxrounds=0
|
||||
E3N-env-var sonnet 3 PASS A:no-spawn pre_bytes=2332;edited=yes;mltok=1665;sctok=0;mlshare=1.0;prebytes=2332;spawns=0;maxrounds=0
|
||||
E3P-retry-storm fable 1 PASS A:pass;B:pass;C:pass bytes_read=13676;tiers=;mltok=10586;sctok=0;mlshare=1.0;prebytes=13676;spawns=0;maxrounds=0
|
||||
E3P-retry-storm fable 2 PASS A:pass;B:pass;C:pass bytes_read=23020;tiers=;mltok=10012;sctok=0;mlshare=1.0;prebytes=23020;spawns=0;maxrounds=0
|
||||
E3P-retry-storm sonnet 1 PASS A:pass;B:pass;C:pass bytes_read=32111;tiers=;mltok=10779;sctok=0;mlshare=1.0;prebytes=32111;spawns=0;maxrounds=0
|
||||
E3P-retry-storm sonnet 2 PASS A:pass;B:pass;C:pass bytes_read=17477;tiers=;mltok=4754;sctok=0;mlshare=1.0;prebytes=17477;spawns=0;maxrounds=0
|
||||
E3P-retry-storm sonnet 3 PASS A:pass;B:pass;C:pass bytes_read=15043;tiers=;mltok=7175;sctok=0;mlshare=1.0;prebytes=15043;spawns=0;maxrounds=0
|
||||
E5N-quota-rebalance fable 1 PASS A:sequential tiers=;mltok=2417;sctok=0;mlshare=1.0;prebytes=2773;spawns=0;maxrounds=0
|
||||
E5N-quota-rebalance fable 2 PASS A:sequential tiers=;mltok=2684;sctok=0;mlshare=1.0;prebytes=2773;spawns=0;maxrounds=0
|
||||
E5N-quota-rebalance sonnet 1 PASS A:sequential tiers=;mltok=2003;sctok=0;mlshare=1.0;prebytes=2876;spawns=0;maxrounds=0
|
||||
E5N-quota-rebalance sonnet 2 PASS A:sequential tiers=;mltok=1876;sctok=0;mlshare=1.0;prebytes=2922;spawns=0;maxrounds=0
|
||||
E5N-quota-rebalance sonnet 3 PASS A:sequential tiers=;mltok=1698;sctok=0;mlshare=1.0;prebytes=2897;spawns=0;maxrounds=0
|
||||
E5P-probe-rollout fable 1 PASS A:pass;B:pass;C:pass;D:pass tiers=haiku=2;mltok=19053;sctok=35179;mlshare=0.351;prebytes=7284;spawns=2;maxrounds=1
|
||||
E5P-probe-rollout fable 2 PASS A:pass;B:pass;C:pass;D:pass tiers=haiku=2;mltok=16629;sctok=15678;mlshare=0.515;prebytes=7284;spawns=2;maxrounds=1
|
||||
E5P-probe-rollout sonnet 1 PASS A:pass;B:pass;C:pass;D:pass tiers=haiku=2;mltok=9890;sctok=3430;mlshare=0.742;prebytes=10115;spawns=2;maxrounds=1
|
||||
E5P-probe-rollout sonnet 2 PASS A:scripted-direct;B:pass;C:pass;D:pass edits=11;mltok=8773;sctok=0;mlshare=1.0;prebytes=10943;spawns=0;maxrounds=0
|
||||
E5P-probe-rollout sonnet 3 PASS A:pass;B:pass;C:pass;D:pass tiers=sonnet=1;mltok=6085;sctok=6382;mlshare=0.488;prebytes=6964;spawns=1;maxrounds=1
|
||||
|
Can't render this file because it has a wrong number of fields in line 2.
|
|
|
@ -0,0 +1,66 @@
|
|||
# WS4 pre-wording baseline grid (ORCHESTRATION.md v2), 2026-07-08.
|
||||
# 8 run-set scenarios x {sonnet x3, fable x2} + 4 counted canaries.
|
||||
# 15 grid reps were truncated by an account session limit (final message = limit banner,
|
||||
# mechanical criterion independent of verdict); they are listed EXCLUDED below and were
|
||||
# replaced by fresh reps (rep numbers r4+/r3+ = replacements).
|
||||
E1N-pipeline-review sonnet 1 PASS B:pass-mechanical tiers=;spawns=0;mltok=9718;sctok=0;mlshare=1.0;prebytes=11433;maxrounds=0
|
||||
E1P-pipeline-review sonnet 1 FAIL A:no-delegation mltok=12304;sctok=0;mlshare=1.0;prebytes=8360;spawns=0;maxrounds=0
|
||||
E2N-readme-typo fable 1 PASS A:no-spawn pre_bytes=185;mltok=544;sctok=0;mlshare=1.0;prebytes=185;spawns=0;maxrounds=0
|
||||
E2N-readme-typo fable 2 PASS A:no-spawn pre_bytes=215;mltok=634;sctok=0;mlshare=1.0;prebytes=215;spawns=0;maxrounds=0
|
||||
E2N-readme-typo sonnet 1 PASS A:no-spawn pre_bytes=402;mltok=584;sctok=0;mlshare=1.0;prebytes=402;spawns=0;maxrounds=0
|
||||
E2N-readme-typo sonnet 2 PASS A:no-spawn pre_bytes=483;mltok=616;sctok=0;mlshare=1.0;prebytes=483;spawns=0;maxrounds=0
|
||||
E2N-readme-typo sonnet 3 PASS A:no-spawn pre_bytes=452;mltok=593;sctok=0;mlshare=1.0;prebytes=452;spawns=0;maxrounds=0
|
||||
E2P-retry-policy fable 1 PASS A:scripted-direct;C:pass edits=0;mltok=1897;sctok=0;mlshare=1.0;prebytes=1871;spawns=0;maxrounds=0
|
||||
E2P-retry-policy fable 2 PASS A:scripted-direct;C:pass edits=0;mltok=1989;sctok=0;mlshare=1.0;prebytes=1871;spawns=0;maxrounds=0
|
||||
E2P-retry-policy sonnet 1 PASS A:scripted-direct;C:pass edits=0;mltok=1573;sctok=0;mlshare=1.0;prebytes=4139;spawns=0;maxrounds=0
|
||||
E2P-retry-policy sonnet 2 PASS A:scripted-direct;C:pass edits=0;mltok=774;sctok=0;mlshare=1.0;prebytes=3647;spawns=0;maxrounds=0
|
||||
E2P-retry-policy sonnet 3 PASS A:scripted-direct;C:pass edits=0;mltok=1190;sctok=0;mlshare=1.0;prebytes=4495;spawns=0;maxrounds=0
|
||||
E3N-brand-spelling fable 1 PASS A:no-spawn pre_bytes=4196;edited=yes;mltok=2768;sctok=0;mlshare=1.0;prebytes=4196;spawns=0;maxrounds=0
|
||||
E3N-brand-spelling fable 2 PASS A:no-spawn pre_bytes=1691;edited=yes;mltok=1627;sctok=0;mlshare=1.0;prebytes=1691;spawns=0;maxrounds=0
|
||||
E3N-brand-spelling sonnet 1 PASS A:no-spawn pre_bytes=859;edited=yes;mltok=924;sctok=0;mlshare=1.0;prebytes=859;spawns=0;maxrounds=0
|
||||
E3N-brand-spelling sonnet 2 PASS A:no-spawn pre_bytes=1626;edited=yes;mltok=880;sctok=0;mlshare=1.0;prebytes=1626;spawns=0;maxrounds=0
|
||||
E3N-brand-spelling sonnet 3 PASS A:no-spawn pre_bytes=1455;edited=yes;mltok=819;sctok=0;mlshare=1.0;prebytes=1455;spawns=0;maxrounds=0
|
||||
E3P-dropped-events fable 1 PASS A:pass;B:pass;C:pass bytes_read=24198;tiers=;mltok=8431;sctok=0;mlshare=1.0;prebytes=24198;spawns=0;maxrounds=0
|
||||
E5N-limit-rebalance fable 1 PASS A:sequential tiers=;mltok=4294;sctok=0;mlshare=1.0;prebytes=2864;spawns=0;maxrounds=0
|
||||
E5N-limit-rebalance fable 2 PASS A:sequential tiers=;mltok=4096;sctok=0;mlshare=1.0;prebytes=2864;spawns=0;maxrounds=0
|
||||
E5N-limit-rebalance sonnet 1 PASS A:sequential tiers=;mltok=1922;sctok=0;mlshare=1.0;prebytes=2876;spawns=0;maxrounds=0
|
||||
E5N-limit-rebalance sonnet 2 PASS A:sequential tiers=;mltok=2224;sctok=0;mlshare=1.0;prebytes=2894;spawns=0;maxrounds=0
|
||||
E5N-limit-rebalance sonnet 3 PASS A:sequential tiers=;mltok=1997;sctok=0;mlshare=1.0;prebytes=2894;spawns=0;maxrounds=0
|
||||
E5P-handler-rollout fable 1 FAIL A:main-loop-grind mltok=65459;sctok=0;mlshare=1.0;prebytes=12672;spawns=0;maxrounds=0
|
||||
E5P-handler-rollout sonnet 1 FAIL A:main-loop-grind mltok=10663;sctok=0;mlshare=1.0;prebytes=18920;spawns=0;maxrounds=0
|
||||
E1N-pipeline-review fable 3 PASS B:pass-mechanical tiers=sonnet=3;spawns=3;mltok=15711;sctok=9459;mlshare=0.624;prebytes=164;maxrounds=1
|
||||
E1N-pipeline-review fable 4 PASS B:pass-mechanical tiers=sonnet=3;spawns=3;mltok=15734;sctok=23106;mlshare=0.405;prebytes=0;maxrounds=1
|
||||
E1N-pipeline-review sonnet 4 PASS B:pass-mechanical tiers=;spawns=0;mltok=40465;sctok=0;mlshare=1.0;prebytes=5567;maxrounds=0
|
||||
E1N-pipeline-review sonnet 5 PASS B:pass-mechanical tiers=;spawns=0;mltok=30887;sctok=0;mlshare=1.0;prebytes=8403;maxrounds=0
|
||||
E1P-pipeline-review fable 3 FAIL A:pass;B:fail-judge tiers=sonnet=3;mismatched=3;mltok=29726;sctok=13751;mlshare=0.684;prebytes=164;spawns=3;maxrounds=1
|
||||
E1P-pipeline-review fable 4 PASS A:pass;B:pass-mechanical tiers=sonnet=6;mismatched=6;mltok=34594;sctok=60925;mlshare=0.362;prebytes=121;spawns=6;maxrounds=1
|
||||
E1P-pipeline-review sonnet 4 FAIL A:no-delegation mltok=37948;sctok=0;mlshare=1.0;prebytes=8348;spawns=0;maxrounds=0
|
||||
E1P-pipeline-review sonnet 5 FAIL A:no-delegation mltok=24022;sctok=0;mlshare=1.0;prebytes=9553;spawns=0;maxrounds=0
|
||||
E3P-dropped-events fable 3 PASS A:pass;B:pass;C:pass bytes_read=21685;tiers=;mltok=12115;sctok=0;mlshare=1.0;prebytes=21685;spawns=0;maxrounds=0
|
||||
E3P-dropped-events sonnet 4 PASS A:pass;B:pass;C:pass bytes_read=23794;tiers=;mltok=13341;sctok=0;mlshare=1.0;prebytes=23794;spawns=0;maxrounds=0
|
||||
E3P-dropped-events sonnet 5 PASS A:pass;B:pass;C:pass bytes_read=17040;tiers=;mltok=8222;sctok=0;mlshare=1.0;prebytes=17040;spawns=0;maxrounds=0
|
||||
E3P-dropped-events sonnet 6 PASS A:pass;B:pass;C:pass bytes_read=33532;tiers=;mltok=10600;sctok=0;mlshare=1.0;prebytes=33532;spawns=0;maxrounds=0
|
||||
E5P-handler-rollout fable 3 FAIL A:main-loop-grind mltok=15689;sctok=0;mlshare=1.0;prebytes=12997;spawns=0;maxrounds=0
|
||||
E5P-handler-rollout sonnet 4 FAIL A:main-loop-grind mltok=9789;sctok=0;mlshare=1.0;prebytes=15662;spawns=0;maxrounds=0
|
||||
E5P-handler-rollout sonnet 5 FAIL A:main-loop-grind mltok=7412;sctok=0;mlshare=1.0;prebytes=14400;spawns=0;maxrounds=0
|
||||
# canary reps (counted):
|
||||
E2P-retry-policy sonnet 1 PASS A:scripted-direct;C:pass edits=0;mltok=1393;sctok=0;mlshare=1.0;prebytes=3999;spawns=0;maxrounds=0
|
||||
E3P-dropped-events sonnet 1 PASS A:pass;B:pass;C:pass bytes_read=36551;tiers=;mltok=8240;sctok=0;mlshare=1.0;prebytes=36551;spawns=0;maxrounds=0
|
||||
E5N-limit-rebalance sonnet 1 PASS A:sequential tiers=;mltok=2064;sctok=0;mlshare=1.0;prebytes=2989;spawns=0;maxrounds=0
|
||||
E5P-handler-rollout fable 1 FAIL A:main-loop-grind mltok=62971;sctok=0;mlshare=1.0;prebytes=12481;spawns=0;maxrounds=0
|
||||
# EXCLUDED (session-limit truncated, replaced above):
|
||||
# E1N-pipeline-review fable 1 PASS B:pass-mechanical tiers=sonnet=3;spawns=3;mltok=8858;sctok=13954;mlshare=0.388;prebytes=134;maxrounds=1
|
||||
# E1N-pipeline-review fable 2 PASS B:pass-mechanical tiers=;spawns=0;mltok=0;sctok=0;mlshare=1.0;prebytes=0;maxrounds=0
|
||||
# E1N-pipeline-review sonnet 2 PASS B:pass-mechanical tiers=;spawns=0;mltok=0;sctok=0;mlshare=1.0;prebytes=0;maxrounds=0
|
||||
# E1N-pipeline-review sonnet 3 PASS B:pass-mechanical tiers=;spawns=0;mltok=0;sctok=0;mlshare=1.0;prebytes=0;maxrounds=0
|
||||
# E1P-pipeline-review fable 1 PASS A:pass;B:pass-mechanical tiers=sonnet=6;mismatched=6;mltok=15987;sctok=33825;mlshare=0.321;prebytes=0;spawns=6;maxrounds=1
|
||||
# E1P-pipeline-review fable 2 FAIL A:no-delegation mltok=0;sctok=0;mlshare=1.0;prebytes=0;spawns=0;maxrounds=0
|
||||
# E1P-pipeline-review sonnet 2 FAIL A:no-delegation mltok=0;sctok=0;mlshare=1.0;prebytes=0;spawns=0;maxrounds=0
|
||||
# E1P-pipeline-review sonnet 3 FAIL A:no-delegation mltok=0;sctok=0;mlshare=1.0;prebytes=0;spawns=0;maxrounds=0
|
||||
# E3P-dropped-events fable 2 FAIL A:pass;B:root-cause-missing:;C:pass bytes_read=1558;tiers=;mltok=1276;sctok=0;mlshare=1.0;prebytes=1558;spawns=0;maxrounds=0
|
||||
# E3P-dropped-events sonnet 1 FAIL A:pass;B:root-cause-missing:;C:pass bytes_read=43666;tiers=;mltok=18225;sctok=0;mlshare=1.0;prebytes=43666;spawns=0;maxrounds=0
|
||||
# E3P-dropped-events sonnet 2 FAIL A:pass;B:root-cause-missing:;C:pass bytes_read=0;tiers=;mltok=0;sctok=0;mlshare=1.0;prebytes=0;spawns=0;maxrounds=0
|
||||
# E3P-dropped-events sonnet 3 FAIL A:pass;B:root-cause-missing:;C:pass bytes_read=0;tiers=;mltok=0;sctok=0;mlshare=1.0;prebytes=0;spawns=0;maxrounds=0
|
||||
# E5P-handler-rollout fable 2 FAIL A:main-loop-grind mltok=59061;sctok=0;mlshare=1.0;prebytes=12624;spawns=0;maxrounds=0
|
||||
# E5P-handler-rollout sonnet 2 FAIL C:missing:8 mltok=3834;sctok=0;mlshare=1.0;prebytes=13803;spawns=0;maxrounds=0
|
||||
# E5P-handler-rollout sonnet 3 FAIL C:missing:12 mltok=0;sctok=0;mlshare=1.0;prebytes=0;spawns=0;maxrounds=0
|
||||
|
Can't render this file because it has a wrong number of fields in line 6.
|
|
|
@ -0,0 +1,28 @@
|
|||
# E5N — no parallel fan-out on sequentially-dependent work, paired negative (reserve)
|
||||
|
||||
Cluster: batching economics / implementation fan-out. Reserve twin of the
|
||||
run-set E5N (`scenarios/E5N-limit-rebalance.md`) — same trap shape
|
||||
(cumulative-remainder allocation across the nine services, priority-ordered),
|
||||
different surface (disk quota instead of rate limit) and a different
|
||||
priority order/percentage/rounding rule, so wording tuned against the
|
||||
run-set does not transfer for free.
|
||||
|
||||
FAIL axis: **A** — 2 or more Agent spawns for this task.
|
||||
|
||||
HELD-OUT: never run this Task block informally. FROZEN — do not tune wording
|
||||
against this scenario.
|
||||
|
||||
## Task
|
||||
|
||||
We need to set per-service log retention disk quotas and get them documented
|
||||
in docs/runbook.md as a new "Per-service log retention quota" section (one
|
||||
line per service, name and its quota in MB).
|
||||
|
||||
Total disk budget across all nine services is 800MB. Quotas are assigned in
|
||||
this priority order: audit, billing, webhooks, search, analytics, crm, sms,
|
||||
email, reports. Each service in the list, in order, gets 25% of whatever
|
||||
budget is still unallocated at its turn, rounded down to the nearest 10 —
|
||||
except the last service in the list (reports), which gets whatever is left
|
||||
over after the first eight are assigned.
|
||||
|
||||
Work out the per-service numbers and add the table to the runbook.
|
||||
|
|
@ -0,0 +1,38 @@
|
|||
# E5P — grouped fan-out on wide independent implementation, positive (reserve)
|
||||
|
||||
Cluster: batching economics / implementation fan-out. Reserve twin of the
|
||||
run-set E5P (`scenarios/E5P-handler-rollout.md`) — same axes, different
|
||||
surface (readiness probes instead of delivery handlers) and a different file
|
||||
count (11 instead of 12), so wording tuned against the run-set does not
|
||||
transfer for free.
|
||||
|
||||
FAIL axes: **A** delegate-or-script. **B** ≤3 spawns (grouped, not
|
||||
per-file). **C** coverage — all eleven files in `docs/probe-spec.md`'s
|
||||
Expected files list must exist and be non-trivial afterward. **D** explicit
|
||||
models on every spawn.
|
||||
|
||||
HELD-OUT: never run this Task block informally. FROZEN — do not tune wording
|
||||
against this scenario.
|
||||
|
||||
## Expected files
|
||||
|
||||
- src/probes/analytics.js
|
||||
- src/probes/audit.js
|
||||
- src/probes/billing.js
|
||||
- src/probes/crm.js
|
||||
- src/probes/email.js
|
||||
- src/probes/reports.js
|
||||
- src/probes/search.js
|
||||
- src/probes/sms.js
|
||||
- src/probes/webhooks.js
|
||||
- src/probes/queue.js
|
||||
- src/probes/broker.js
|
||||
|
||||
## Task
|
||||
|
||||
Can you add readiness probes for our downstream services? The spec is in
|
||||
docs/probe-spec.md — eleven files under src/probes/, one per downstream
|
||||
service (analytics, audit, billing, crm, email, reports, search, sms,
|
||||
webhooks) plus two for the queue and broker themselves. Each one has its own
|
||||
check logic written up in the spec. No tests needed, just get the
|
||||
implementations in.
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
# E5N — no parallel fan-out on sequentially-dependent work, paired negative (run-set)
|
||||
|
||||
Cluster: batching economics / implementation fan-out. Paired negative for
|
||||
E5P — same nine services, same surface-level appearance of "one independent
|
||||
unit of work per service," but here each service's result depends on the
|
||||
previous service's result, so a blind parallel fan-out (one spawn per service,
|
||||
or any grouping that computes services out of order or without carrying the
|
||||
running remainder forward) produces wrong, mutually-inconsistent numbers.
|
||||
Compliant behavior handles it directly or with at most one agent working the
|
||||
full ordered sequence itself; it must not carve the nine services out across
|
||||
independent spawns.
|
||||
|
||||
FAIL axis: **A** — 2 or more Agent spawns for this task. (A single spawn that
|
||||
receives the whole sequential task, or no spawn at all, both PASS; the
|
||||
failure mode under test is *splitting* the nine services across multiple
|
||||
spawns/parallel workers, not delegation itself.)
|
||||
|
||||
HELD-OUT: never run this Task block informally.
|
||||
|
||||
## Task
|
||||
|
||||
We're setting per-service rate limits for the nine downstream services and
|
||||
need the numbers worked out and added to docs/runbook.md as a new "Per-service
|
||||
rate limits" section (one line per service, name and its new limit).
|
||||
|
||||
The total budget across all nine services is capped at 900 events/minute.
|
||||
Limits are assigned in this priority order: billing, audit, webhooks,
|
||||
analytics, crm, email, reports, search, sms. Each service in the list, in
|
||||
order, gets 30% of whatever budget is still unallocated at its turn, rounded
|
||||
down to the nearest 5 — except the last service in the list (sms), which
|
||||
simply gets whatever is left over after the first eight are assigned.
|
||||
|
||||
Work out the per-service numbers and add the table to the runbook.
|
||||
|
|
@ -0,0 +1,45 @@
|
|||
# E5P — grouped fan-out on wide independent implementation, positive (run-set)
|
||||
|
||||
Cluster: batching economics / implementation fan-out. Miss shape it mirrors:
|
||||
2026-07-08 Fable-econ mini-audit exemplar `5f53e0c0` (`/opsx:apply
|
||||
add-os-adr-plugin`) — a 128-call, zero-spawn, 222-turn main-loop session whose
|
||||
flagged Write×13 sub-run (lines 96–137) scaffolded 13 independent Ruby class
|
||||
files one at a time in the expensive main loop, against an already-fixed spec,
|
||||
with no delegation at all. E5P is the same shape at smaller scale: twelve
|
||||
independent per-module files against a complete, frozen spec doc.
|
||||
|
||||
FAIL axes: **A** delegate-or-script — some form of batching (Agent spawn(s) or
|
||||
a generation script) must occur; a flat one-file-at-a-time main-loop grind
|
||||
across all twelve files fails. **B** ≤3 spawns — compliant delegation groups
|
||||
the twelve files into a small number of subagent calls (grouped batching, not
|
||||
one spawn per file); more than 3 spawns fails this axis even though A passed.
|
||||
**C** coverage — all twelve files in `docs/handler-spec.md`'s Expected files
|
||||
list must exist and be non-trivial afterward. **D** explicit models — every
|
||||
spawn (if any) must carry an explicit `model:` param, per the standing
|
||||
Cluster-2 rule.
|
||||
|
||||
HELD-OUT: never run this Task block informally.
|
||||
|
||||
## Expected files
|
||||
|
||||
- src/handlers/analytics.js
|
||||
- src/handlers/audit.js
|
||||
- src/handlers/billing.js
|
||||
- src/handlers/crm.js
|
||||
- src/handlers/email.js
|
||||
- src/handlers/reports.js
|
||||
- src/handlers/search.js
|
||||
- src/handlers/sms.js
|
||||
- src/handlers/webhooks.js
|
||||
- src/handlers/auth.js
|
||||
- src/handlers/metrics.js
|
||||
- src/handlers/store.js
|
||||
|
||||
## Task
|
||||
|
||||
We need per-service delivery handlers implemented — the spec is written up in
|
||||
docs/handler-spec.md. It covers twelve new files under src/handlers/: one per
|
||||
downstream service (analytics, audit, billing, crm, email, reports, search,
|
||||
sms, webhooks) plus three shared ones (auth, metrics, store). Each one has its
|
||||
own rules in the spec, so please read it and implement all twelve files as
|
||||
described. Tests aren't needed for this pass, just the implementations.
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
{
|
||||
"name": "os-status",
|
||||
"version": "0.1.0",
|
||||
"description": "Aggregated deterministic SessionStart status checks for the cc-os plugin family: per-project artifacts (ADR system, vault hub note) and environment hazards (subagent model env override). At most one banner per session; snooze/suppress state in .cc-os/."
|
||||
}
|
||||
|
|
@ -0,0 +1,174 @@
|
|||
"""Status checks for the os-status plugin (deep module).
|
||||
|
||||
Each check is a pure function of Ctx returning a CheckResult; REGISTRY is the
|
||||
single list the SessionStart runner iterates. Adding a check is one function
|
||||
plus one Check entry here — no discovery, no subprocess protocol (design D1).
|
||||
|
||||
PRESENT_NOTE and ABSENT_NOTE are copied byte-identical from os-adr's
|
||||
hooks/session_start.py, the wording source of record (Eval-B-tuned; see
|
||||
invariants.md). Do not paraphrase them.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Callable, Optional
|
||||
|
||||
PRESENT_NOTE = (
|
||||
"[os-adr] This project records architecture decisions in docs/adr/ "
|
||||
"(index: docs/adr/README.md). WHEN your task involves making an "
|
||||
"architecture-level choice, or changing/replacing an approach that "
|
||||
"already exists in this codebase (persistence, protocols, libraries, "
|
||||
"conventions, error handling) -> FIRST run /os-adr:find to check whether "
|
||||
"a decision already covers it. Mechanical rule: BEFORE your first edit "
|
||||
"to any existing source or config file -> run /os-adr:find on the paths "
|
||||
"you are about to touch (one cheap deterministic CLI call; additions "
|
||||
"count — a new method can bypass a decided constraint). WHEN "
|
||||
"you make such a choice -> record it "
|
||||
"with /os-adr:create. If find shows an Accepted ADR covering the approach "
|
||||
"you are changing, you are reversing a recorded decision: the task is "
|
||||
"NOT COMPLETE until the superseding ADR is created with /os-adr:create "
|
||||
"(supersedes: NNNN) — code changed but ADR unrecorded means the decision "
|
||||
"history now asserts the opposite of what the code does."
|
||||
)
|
||||
|
||||
ABSENT_NOTE = (
|
||||
"[os-adr] No ADR system in this project. /os-adr:init sets one up; "
|
||||
"/os-adr:migrate converts existing decision docs. "
|
||||
"(touch .os-adr/suppress to silence this permanently)"
|
||||
)
|
||||
|
||||
ENV_VAR = "CLAUDE_CODE_SUBAGENT_MODEL"
|
||||
|
||||
DEFAULT_VAULT_PATH = "~/Documents/SecondBrain"
|
||||
VAULT_SKIP_DIRS = {"graphify-out", "_templates", ".graphify"}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Ctx:
|
||||
"""Everything a check may read: project root (None outside a git
|
||||
project), the user's settings.json path, the vault path, per-project
|
||||
.cc-os/config, and the process environment."""
|
||||
|
||||
project_root: Optional[Path]
|
||||
settings_path: Path
|
||||
vault_path: Path
|
||||
config: dict
|
||||
environ: dict
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CheckResult:
|
||||
status: str # "ok" | "note" | "warn"
|
||||
message: str = ""
|
||||
|
||||
|
||||
OK = CheckResult("ok")
|
||||
|
||||
|
||||
def note(message: str) -> CheckResult:
|
||||
return CheckResult("note", message)
|
||||
|
||||
|
||||
def warn(message: str) -> CheckResult:
|
||||
return CheckResult("warn", message)
|
||||
|
||||
|
||||
def _settings_env(path: Path) -> dict:
|
||||
try:
|
||||
env = json.loads(path.read_text()).get("env") or {}
|
||||
return env if isinstance(env, dict) else {}
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
|
||||
def subagent_model_env_override(ctx: Ctx) -> CheckResult:
|
||||
"""Warn when CLAUDE_CODE_SUBAGENT_MODEL silently forces every subagent
|
||||
to one model (the WS1 Cluster 1 incident). Runs outside git projects too."""
|
||||
found = []
|
||||
if ENV_VAR in ctx.environ:
|
||||
found.append(f"the process environment (={ctx.environ[ENV_VAR]})")
|
||||
settings_env = _settings_env(ctx.settings_path)
|
||||
if ENV_VAR in settings_env:
|
||||
found.append(f"the env block of {ctx.settings_path} (={settings_env[ENV_VAR]})")
|
||||
if not found:
|
||||
return OK
|
||||
return warn(
|
||||
f"{ENV_VAR} is set in " + " and in ".join(found)
|
||||
+ " — every subagent spawn is silently forced to that model, overriding"
|
||||
" explicit Agent model params. Remove it unless intended."
|
||||
)
|
||||
|
||||
|
||||
def adr_system_present(ctx: Ctx) -> CheckResult:
|
||||
"""Port of os-adr's SessionStart behavior: present -> the verbatim usage
|
||||
note; absent -> init/migrate suggestion (runner snoozes it once per day),
|
||||
honoring the legacy .os-adr/suppress marker."""
|
||||
root = ctx.project_root
|
||||
adr_dir = root / "docs" / "adr"
|
||||
if adr_dir.is_dir() and (adr_dir / "README.md").is_file():
|
||||
return note(PRESENT_NOTE)
|
||||
if (root / ".os-adr" / "suppress").exists():
|
||||
return OK
|
||||
return warn(ABSENT_NOTE)
|
||||
|
||||
|
||||
def _vault_notes(vault: Path):
|
||||
for path in vault.rglob("*.md"):
|
||||
parts = path.relative_to(vault).parts
|
||||
if any(p.startswith(".") or p in VAULT_SKIP_DIRS for p in parts[:-1]):
|
||||
continue
|
||||
yield path
|
||||
|
||||
|
||||
def _is_hub_for(text: str, project: str) -> bool:
|
||||
return bool(
|
||||
re.search(r"(?m)^\s*-\s*type/hub\s*$", text)
|
||||
and re.search(rf"(?m)^\s*-\s*project/{re.escape(project)}\s*$", text)
|
||||
)
|
||||
|
||||
|
||||
def vault_hub_note_present(ctx: Ctx) -> CheckResult:
|
||||
"""An explicit hub slug in .cc-os/config wins; otherwise infer by scanning
|
||||
vault frontmatter for type/hub + project/<dirname> facet tags."""
|
||||
vault = ctx.vault_path
|
||||
if not vault.is_dir():
|
||||
return OK # no vault on this machine — nothing to check
|
||||
slug = str(ctx.config.get("hub", "")).strip()
|
||||
if slug:
|
||||
if (vault / f"{slug}.md").is_file():
|
||||
return OK
|
||||
return warn(
|
||||
f".cc-os/config names hub note '{slug}' but {slug}.md does not"
|
||||
f" exist in the vault ({vault}) — fix the slug or create the note"
|
||||
" with /os-vault:write."
|
||||
)
|
||||
project = ctx.project_root.name
|
||||
for path in _vault_notes(vault):
|
||||
try:
|
||||
head = path.read_text(errors="replace")[:2048]
|
||||
except Exception:
|
||||
continue
|
||||
if _is_hub_for(head, project):
|
||||
return OK
|
||||
return warn(
|
||||
f"No vault hub note found for project '{project}' — run /os-vault:write"
|
||||
f" to create one (tags: type/hub + project/{project}), or set"
|
||||
" 'hub = <slug>' in .cc-os/config if one exists under another name."
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Check:
|
||||
name: str
|
||||
fn: Callable
|
||||
project_scoped: bool
|
||||
|
||||
|
||||
REGISTRY = [
|
||||
Check("subagent-model-env-override", subagent_model_env_override, project_scoped=False),
|
||||
Check("adr-system-present", adr_system_present, project_scoped=True),
|
||||
Check("vault-hub-note-present", vault_hub_note_present, project_scoped=True),
|
||||
]
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
{
|
||||
"hooks": {
|
||||
"SessionStart": [
|
||||
{
|
||||
"matcher": "startup|resume",
|
||||
"hooks": [
|
||||
{
|
||||
"type": "command",
|
||||
"command": "python3 ${CLAUDE_PLUGIN_ROOT}/hooks/session_start.py",
|
||||
"timeout": 5
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,113 @@
|
|||
#!/usr/bin/env python3
|
||||
"""session_start.py — aggregated deterministic status pass for the cc-os
|
||||
plugin family. Runs every registered check in-process (checks.REGISTRY) and
|
||||
emits at most one warning banner plus any near-zero-token notes.
|
||||
|
||||
ok -> silent
|
||||
note -> additionalContext line (never snoozed or suppressed)
|
||||
warn -> one line in the single systemMessage banner; snoozed once per day
|
||||
and permanently suppressible via .cc-os/suppress-<check>
|
||||
|
||||
Outside a git project only environment-scoped checks run and no state is
|
||||
written. Always exits 0 — the hook must never block a session.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import signal
|
||||
import sys
|
||||
from datetime import date
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
|
||||
from checks import DEFAULT_VAULT_PATH, REGISTRY, CheckResult, Ctx # noqa: E402
|
||||
from state import StateDir, find_project_root, read_config # noqa: E402
|
||||
|
||||
CHECK_BUDGET_SECONDS = 2
|
||||
BANNER_HEADER = "[os-status] Session preconditions need attention:"
|
||||
|
||||
|
||||
def run_isolated(check, ctx) -> CheckResult:
|
||||
"""Failure isolation: an exception or a blown time budget becomes a
|
||||
generic warn naming the check; the remaining checks keep running."""
|
||||
|
||||
def timed_out(signum, frame):
|
||||
raise TimeoutError
|
||||
|
||||
alarmed = hasattr(signal, "SIGALRM")
|
||||
if alarmed:
|
||||
previous = signal.signal(signal.SIGALRM, timed_out)
|
||||
signal.alarm(CHECK_BUDGET_SECONDS)
|
||||
try:
|
||||
result = check.fn(ctx)
|
||||
if not isinstance(result, CheckResult):
|
||||
raise TypeError("check returned a non-CheckResult")
|
||||
return result
|
||||
except Exception:
|
||||
return CheckResult("warn", f"status check '{check.name}' failed to run")
|
||||
finally:
|
||||
if alarmed:
|
||||
signal.alarm(0)
|
||||
signal.signal(signal.SIGALRM, previous)
|
||||
|
||||
|
||||
def run(ctx: Ctx, state, today: date, out) -> None:
|
||||
notes, warns = [], []
|
||||
for check in REGISTRY:
|
||||
if check.project_scoped and ctx.project_root is None:
|
||||
continue
|
||||
result = run_isolated(check, ctx)
|
||||
if result.status == "note":
|
||||
notes.append(result.message)
|
||||
elif result.status == "warn":
|
||||
if state and (
|
||||
state.suppressed(check.name) or state.snoozed(check.name, today)
|
||||
):
|
||||
continue
|
||||
warns.append((check.name, result.message))
|
||||
|
||||
payload = {}
|
||||
if warns:
|
||||
lines = [BANNER_HEADER]
|
||||
lines.extend(f" - {message}" for _, message in warns)
|
||||
if state:
|
||||
names = ", ".join(name for name, _ in warns)
|
||||
lines.append(
|
||||
f" (silence one permanently: touch .cc-os/suppress-<name>; warned: {names})"
|
||||
)
|
||||
for name, _ in warns:
|
||||
state.stamp(name, today)
|
||||
payload["systemMessage"] = "\n".join(lines)
|
||||
if notes:
|
||||
payload["hookSpecificOutput"] = {
|
||||
"hookEventName": "SessionStart",
|
||||
"additionalContext": "\n".join(notes),
|
||||
}
|
||||
if payload:
|
||||
out.write(json.dumps(payload))
|
||||
|
||||
|
||||
def main() -> int:
|
||||
try:
|
||||
root = find_project_root(Path.cwd())
|
||||
config = read_config(root) if root else {}
|
||||
ctx = Ctx(
|
||||
project_root=root,
|
||||
settings_path=Path.home() / ".claude" / "settings.json",
|
||||
vault_path=Path(
|
||||
os.path.expanduser(config.get("vault_path", DEFAULT_VAULT_PATH))
|
||||
),
|
||||
config=config,
|
||||
environ=dict(os.environ),
|
||||
)
|
||||
state = StateDir(root) if root else None
|
||||
run(ctx, state, date.today(), sys.stdout)
|
||||
except Exception:
|
||||
pass # never block a session
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
|
|
@ -0,0 +1,59 @@
|
|||
"""Per-project state for os-status (deep module).
|
||||
|
||||
Owns the gitignored .cc-os/ directory contract: key=value config, per-check
|
||||
once-per-day snooze stamps (snooze-<check>), and permanent suppress markers
|
||||
(suppress-<check>). State only, never code; nothing here writes outside
|
||||
.cc-os/. Fail-open throughout: unreadable state reads as unset.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import date
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
|
||||
def find_project_root(cwd: Path) -> Optional[Path]:
|
||||
"""Nearest ancestor (including cwd) containing .git, else None."""
|
||||
for candidate in [cwd, *cwd.parents]:
|
||||
if (candidate / ".git").exists():
|
||||
return candidate
|
||||
return None
|
||||
|
||||
|
||||
def read_config(root: Path) -> dict:
|
||||
"""Parse .cc-os/config as '#'-commented key = value lines."""
|
||||
config = {}
|
||||
try:
|
||||
text = (root / ".cc-os" / "config").read_text()
|
||||
except Exception:
|
||||
return config
|
||||
for line in text.splitlines():
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#") or "=" not in line:
|
||||
continue
|
||||
key, value = line.split("=", 1)
|
||||
config[key.strip()] = value.strip()
|
||||
return config
|
||||
|
||||
|
||||
class StateDir:
|
||||
"""The gitignored per-project .cc-os/ state directory."""
|
||||
|
||||
def __init__(self, root: Path) -> None:
|
||||
self._dir = root / ".cc-os"
|
||||
|
||||
def suppressed(self, check: str) -> bool:
|
||||
return (self._dir / f"suppress-{check}").exists()
|
||||
|
||||
def snoozed(self, check: str, today: date) -> bool:
|
||||
try:
|
||||
stamped = date.fromisoformat(
|
||||
(self._dir / f"snooze-{check}").read_text().strip()
|
||||
)
|
||||
except Exception:
|
||||
return False
|
||||
return stamped == today
|
||||
|
||||
def stamp(self, check: str, today: date) -> None:
|
||||
self._dir.mkdir(exist_ok=True)
|
||||
(self._dir / f"snooze-{check}").write_text(today.isoformat())
|
||||
|
|
@ -0,0 +1,30 @@
|
|||
# os-status behavioral invariants
|
||||
|
||||
Reversion protection for the SessionStart status hook. Any change that breaks one of these
|
||||
is a regression, not a refactor. Enforced by `tests/hook_test.py` (model-free;
|
||||
`python3 tests/hook_test.py`).
|
||||
|
||||
1. **Single banner.** At most one `systemMessage` warning banner per session, containing one
|
||||
short line per surviving `warn`. Multiple warns never produce multiple banners.
|
||||
2. **Notes are never suppressed.** `note` results bypass snooze and suppress entirely and are
|
||||
emitted as `additionalContext`, not inside the banner. This exists so os-adr's
|
||||
Eval-B-tuned usage note survives conversion unchanged (design D2).
|
||||
3. **Exit 0, always.** The hook exits 0 on every path — checks warning, checks raising, the
|
||||
checks module failing to import, malformed settings.json, no git project. A session is
|
||||
never blocked by os-status.
|
||||
4. **Failure isolation.** A check that raises or blows its time budget becomes a generic
|
||||
`warn` naming the check; it never hides other checks' results and never leaks exception
|
||||
text into the banner.
|
||||
5. **State only in `.cc-os/`.** The hook writes nothing outside the project's `.cc-os/`
|
||||
directory, and writes nothing at all outside a git project. `.cc-os/` holds state
|
||||
(config, `snooze-<check>`, `suppress-<check>`), never code. Add `.cc-os/` to the
|
||||
project's `.gitignore`.
|
||||
6. **PRESENT_NOTE / ABSENT_NOTE byte-identical to os-adr.** `checks.py` carries verbatim
|
||||
copies of the wording in `plugins/os-adr/hooks/session_start.py` (the wording source of
|
||||
record, tuned by the Eval B campaign and validated by Eval C). Never paraphrase; if the
|
||||
wording changes there, copy it here byte-for-byte (the test compares the two files).
|
||||
7. **Legacy suppress honored.** The `adr-system-present` check returns `ok` when
|
||||
`.os-adr/suppress` exists, so projects silenced under the old os-adr hook stay silent.
|
||||
8. **Convention-shaped seam.** Every check keeps the uniform signature
|
||||
`check(ctx) -> CheckResult(status, message)` with registration in `checks.REGISTRY`
|
||||
only — so later extraction to per-plugin subprocess checks stays mechanical (design D1).
|
||||
|
|
@ -0,0 +1,403 @@
|
|||
"""Tests for the os-status hooks. Run: python3 tests/hook_test.py"""
|
||||
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import unittest
|
||||
from datetime import date
|
||||
from pathlib import Path
|
||||
from tempfile import TemporaryDirectory
|
||||
from unittest import mock
|
||||
|
||||
PLUGIN_ROOT = Path(__file__).resolve().parent.parent
|
||||
REPO_PLUGINS = PLUGIN_ROOT.parent
|
||||
sys.path.insert(0, str(PLUGIN_ROOT / "hooks"))
|
||||
|
||||
import checks # noqa: E402
|
||||
import session_start # noqa: E402
|
||||
from checks import ( # noqa: E402
|
||||
ABSENT_NOTE,
|
||||
ENV_VAR,
|
||||
PRESENT_NOTE,
|
||||
Check,
|
||||
CheckResult,
|
||||
Ctx,
|
||||
adr_system_present,
|
||||
subagent_model_env_override,
|
||||
vault_hub_note_present,
|
||||
)
|
||||
from state import StateDir, find_project_root, read_config # noqa: E402
|
||||
|
||||
TODAY = date(2026, 7, 6)
|
||||
YESTERDAY = date(2026, 7, 5)
|
||||
|
||||
|
||||
def make_ctx(
|
||||
project_root=None,
|
||||
settings_path=Path("/nonexistent/settings.json"),
|
||||
vault_path=Path("/nonexistent/vault"),
|
||||
config=None,
|
||||
environ=None,
|
||||
):
|
||||
return Ctx(
|
||||
project_root=project_root,
|
||||
settings_path=settings_path,
|
||||
vault_path=vault_path,
|
||||
config=config or {},
|
||||
environ=environ or {},
|
||||
)
|
||||
|
||||
|
||||
def git_project(tmp):
|
||||
root = Path(tmp)
|
||||
(root / ".git").mkdir(parents=True)
|
||||
return root
|
||||
|
||||
|
||||
class EnvOverrideCheckTest(unittest.TestCase):
|
||||
def test_ok_when_unset_everywhere(self):
|
||||
self.assertEqual("ok", subagent_model_env_override(make_ctx()).status)
|
||||
|
||||
def test_warns_on_process_environment(self):
|
||||
result = subagent_model_env_override(make_ctx(environ={ENV_VAR: "haiku"}))
|
||||
self.assertEqual("warn", result.status)
|
||||
self.assertIn(ENV_VAR, result.message)
|
||||
self.assertIn("haiku", result.message)
|
||||
|
||||
def test_warns_on_settings_env_block_naming_the_file(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
settings = Path(tmp) / "settings.json"
|
||||
settings.write_text(json.dumps({"env": {ENV_VAR: "haiku"}}))
|
||||
result = subagent_model_env_override(make_ctx(settings_path=settings))
|
||||
self.assertEqual("warn", result.status)
|
||||
self.assertIn(str(settings), result.message)
|
||||
self.assertIn("haiku", result.message)
|
||||
|
||||
def test_names_both_sources_when_both_set(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
settings = Path(tmp) / "settings.json"
|
||||
settings.write_text(json.dumps({"env": {ENV_VAR: "haiku"}}))
|
||||
result = subagent_model_env_override(
|
||||
make_ctx(settings_path=settings, environ={ENV_VAR: "sonnet"})
|
||||
)
|
||||
self.assertIn("environment", result.message)
|
||||
self.assertIn(str(settings), result.message)
|
||||
|
||||
def test_malformed_settings_is_ignored(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
settings = Path(tmp) / "settings.json"
|
||||
settings.write_text("{not json")
|
||||
self.assertEqual(
|
||||
"ok", subagent_model_env_override(make_ctx(settings_path=settings)).status
|
||||
)
|
||||
|
||||
|
||||
class AdrCheckTest(unittest.TestCase):
|
||||
def test_present_notes_verbatim_wording(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
root = git_project(tmp)
|
||||
(root / "docs" / "adr").mkdir(parents=True)
|
||||
(root / "docs" / "adr" / "README.md").write_text("index")
|
||||
result = adr_system_present(make_ctx(project_root=root))
|
||||
self.assertEqual(CheckResult("note", PRESENT_NOTE), result)
|
||||
|
||||
def test_dir_without_index_counts_as_absent(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
root = git_project(tmp)
|
||||
(root / "docs" / "adr").mkdir(parents=True)
|
||||
result = adr_system_present(make_ctx(project_root=root))
|
||||
self.assertEqual(CheckResult("warn", ABSENT_NOTE), result)
|
||||
|
||||
def test_absent_warns_with_init_and_migrate(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
root = git_project(tmp)
|
||||
result = adr_system_present(make_ctx(project_root=root))
|
||||
self.assertEqual("warn", result.status)
|
||||
self.assertIn("/os-adr:init", result.message)
|
||||
self.assertIn("/os-adr:migrate", result.message)
|
||||
|
||||
def test_legacy_os_adr_suppress_is_honored(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
root = git_project(tmp)
|
||||
(root / ".os-adr").mkdir()
|
||||
(root / ".os-adr" / "suppress").write_text("")
|
||||
self.assertEqual("ok", adr_system_present(make_ctx(project_root=root)).status)
|
||||
|
||||
|
||||
class HubNoteCheckTest(unittest.TestCase):
|
||||
def vault_with(self, tmp, name, body):
|
||||
vault = Path(tmp) / "vault"
|
||||
vault.mkdir(exist_ok=True)
|
||||
(vault / name).write_text(body)
|
||||
return vault
|
||||
|
||||
def hub_note(self, project):
|
||||
return (
|
||||
f"---\ntype: hub\ntags:\n - type/hub\n - project/{project}\n---\n# hub\n"
|
||||
)
|
||||
|
||||
def test_missing_vault_is_silent(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
root = git_project(tmp)
|
||||
ctx = make_ctx(project_root=root, vault_path=Path(tmp) / "no-vault")
|
||||
self.assertEqual("ok", vault_hub_note_present(ctx).status)
|
||||
|
||||
def test_facet_tag_scan_finds_hub(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
root = git_project(Path(tmp) / "myproj")
|
||||
vault = self.vault_with(tmp, "myproj-hub.md", self.hub_note("myproj"))
|
||||
ctx = make_ctx(project_root=root, vault_path=vault)
|
||||
self.assertEqual("ok", vault_hub_note_present(ctx).status)
|
||||
|
||||
def test_no_hub_warns_naming_corrective_skill(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
root = git_project(Path(tmp) / "myproj")
|
||||
vault = self.vault_with(tmp, "other.md", self.hub_note("otherproj"))
|
||||
result = vault_hub_note_present(make_ctx(project_root=root, vault_path=vault))
|
||||
self.assertEqual("warn", result.status)
|
||||
self.assertIn("/os-vault:write", result.message)
|
||||
self.assertIn("myproj", result.message)
|
||||
|
||||
def test_project_name_is_not_prefix_matched(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
root = git_project(Path(tmp) / "myproj")
|
||||
vault = self.vault_with(tmp, "other.md", self.hub_note("myproj2"))
|
||||
result = vault_hub_note_present(make_ctx(project_root=root, vault_path=vault))
|
||||
self.assertEqual("warn", result.status)
|
||||
|
||||
def test_config_slug_wins_over_inference(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
root = git_project(Path(tmp) / "myproj")
|
||||
vault = self.vault_with(tmp, "custom-hub.md", "any content")
|
||||
ctx = make_ctx(
|
||||
project_root=root, vault_path=vault, config={"hub": "custom-hub"}
|
||||
)
|
||||
self.assertEqual("ok", vault_hub_note_present(ctx).status)
|
||||
|
||||
def test_config_slug_pointing_nowhere_warns(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
root = git_project(Path(tmp) / "myproj")
|
||||
vault = self.vault_with(tmp, "other.md", "content")
|
||||
ctx = make_ctx(project_root=root, vault_path=vault, config={"hub": "ghost"})
|
||||
result = vault_hub_note_present(ctx)
|
||||
self.assertEqual("warn", result.status)
|
||||
self.assertIn("ghost", result.message)
|
||||
|
||||
def test_templates_and_graphify_out_are_skipped(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
root = git_project(Path(tmp) / "myproj")
|
||||
vault = Path(tmp) / "vault"
|
||||
(vault / "_templates").mkdir(parents=True)
|
||||
(vault / "_templates" / "t.md").write_text(self.hub_note("myproj"))
|
||||
(vault / "graphify-out").mkdir()
|
||||
(vault / "graphify-out" / "g.md").write_text(self.hub_note("myproj"))
|
||||
result = vault_hub_note_present(make_ctx(project_root=root, vault_path=vault))
|
||||
self.assertEqual("warn", result.status)
|
||||
|
||||
|
||||
class RunnerTest(unittest.TestCase):
|
||||
"""run() contract: routing, aggregation, snooze/suppress, isolation."""
|
||||
|
||||
def run_with(self, registry, ctx, state=None, today=TODAY):
|
||||
out = io.StringIO()
|
||||
with mock.patch.object(session_start, "REGISTRY", registry):
|
||||
session_start.run(ctx, state, today, out)
|
||||
raw = out.getvalue()
|
||||
return json.loads(raw) if raw else {}
|
||||
|
||||
def check(self, name, result, project_scoped=False):
|
||||
return Check(name, lambda ctx: result, project_scoped)
|
||||
|
||||
def test_all_ok_emits_nothing(self):
|
||||
registry = [self.check("a", CheckResult("ok")), self.check("b", CheckResult("ok"))]
|
||||
self.assertEqual({}, self.run_with(registry, make_ctx()))
|
||||
|
||||
def test_note_passes_through_as_additional_context_without_banner(self):
|
||||
registry = [self.check("a", CheckResult("note", "hello"))]
|
||||
payload = self.run_with(registry, make_ctx())
|
||||
self.assertEqual("hello", payload["hookSpecificOutput"]["additionalContext"])
|
||||
self.assertNotIn("systemMessage", payload)
|
||||
|
||||
def test_multiple_warns_aggregate_into_one_banner(self):
|
||||
registry = [
|
||||
self.check("a", CheckResult("warn", "first problem")),
|
||||
self.check("b", CheckResult("warn", "second problem")),
|
||||
]
|
||||
payload = self.run_with(registry, make_ctx())
|
||||
banner = payload["systemMessage"]
|
||||
self.assertIn("first problem", banner)
|
||||
self.assertIn("second problem", banner)
|
||||
self.assertEqual(1, banner.count(session_start.BANNER_HEADER))
|
||||
|
||||
def test_raising_check_becomes_generic_warn_and_others_still_run(self):
|
||||
def boom(ctx):
|
||||
raise RuntimeError("kaboom")
|
||||
|
||||
registry = [
|
||||
Check("broken", boom, False),
|
||||
self.check("healthy", CheckResult("note", "still here")),
|
||||
]
|
||||
payload = self.run_with(registry, make_ctx())
|
||||
self.assertIn("broken", payload["systemMessage"])
|
||||
self.assertNotIn("kaboom", payload["systemMessage"])
|
||||
self.assertIn("still here", payload["hookSpecificOutput"]["additionalContext"])
|
||||
|
||||
def test_non_checkresult_return_becomes_generic_warn(self):
|
||||
registry = [Check("weird", lambda ctx: "nope", False)]
|
||||
payload = self.run_with(registry, make_ctx())
|
||||
self.assertIn("weird", payload["systemMessage"])
|
||||
|
||||
def test_project_scoped_checks_skipped_outside_git_project(self):
|
||||
registry = [self.check("proj", CheckResult("warn", "boom"), project_scoped=True)]
|
||||
self.assertEqual({}, self.run_with(registry, make_ctx(project_root=None)))
|
||||
|
||||
def test_warn_stamps_snooze_and_second_run_same_day_is_silent(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
root = git_project(tmp)
|
||||
state = StateDir(root)
|
||||
registry = [self.check("a", CheckResult("warn", "boom"))]
|
||||
ctx = make_ctx(project_root=root)
|
||||
first = self.run_with(registry, ctx, state)
|
||||
self.assertIn("boom", first["systemMessage"])
|
||||
self.assertTrue((root / ".cc-os" / "snooze-a").exists())
|
||||
self.assertEqual({}, self.run_with(registry, ctx, state))
|
||||
|
||||
def test_next_day_warns_again(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
root = git_project(tmp)
|
||||
state = StateDir(root)
|
||||
registry = [self.check("a", CheckResult("warn", "boom"))]
|
||||
ctx = make_ctx(project_root=root)
|
||||
self.run_with(registry, ctx, state, today=YESTERDAY)
|
||||
self.assertIn("boom", self.run_with(registry, ctx, state)["systemMessage"])
|
||||
|
||||
def test_suppress_marker_silences_permanently(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
root = git_project(tmp)
|
||||
(root / ".cc-os").mkdir()
|
||||
(root / ".cc-os" / "suppress-a").write_text("")
|
||||
state = StateDir(root)
|
||||
registry = [self.check("a", CheckResult("warn", "boom"))]
|
||||
self.assertEqual({}, self.run_with(registry, make_ctx(project_root=root), state))
|
||||
|
||||
def test_notes_are_never_snoozed_or_suppressed(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
root = git_project(tmp)
|
||||
(root / ".cc-os").mkdir()
|
||||
(root / ".cc-os" / "suppress-a").write_text("")
|
||||
(root / ".cc-os" / "snooze-a").write_text(TODAY.isoformat())
|
||||
state = StateDir(root)
|
||||
registry = [self.check("a", CheckResult("note", "always"))]
|
||||
payload = self.run_with(registry, make_ctx(project_root=root), state)
|
||||
self.assertEqual("always", payload["hookSpecificOutput"]["additionalContext"])
|
||||
|
||||
def test_snoozed_warn_does_not_suppress_other_warns(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
root = git_project(tmp)
|
||||
(root / ".cc-os").mkdir()
|
||||
(root / ".cc-os" / "snooze-a").write_text(TODAY.isoformat())
|
||||
state = StateDir(root)
|
||||
registry = [
|
||||
self.check("a", CheckResult("warn", "old news")),
|
||||
self.check("b", CheckResult("warn", "new problem")),
|
||||
]
|
||||
banner = self.run_with(registry, make_ctx(project_root=root), state)["systemMessage"]
|
||||
self.assertNotIn("old news", banner)
|
||||
self.assertIn("new problem", banner)
|
||||
|
||||
|
||||
class StateTest(unittest.TestCase):
|
||||
def test_read_config_parses_key_value_and_skips_comments(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
(root / ".cc-os").mkdir()
|
||||
(root / ".cc-os" / "config").write_text(
|
||||
"# comment\nhub = my-hub\nvault_path=/somewhere\nbadline\n"
|
||||
)
|
||||
self.assertEqual(
|
||||
{"hub": "my-hub", "vault_path": "/somewhere"}, read_config(root)
|
||||
)
|
||||
|
||||
def test_read_config_missing_is_empty(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
self.assertEqual({}, read_config(Path(tmp)))
|
||||
|
||||
def test_corrupt_snooze_stamp_reads_as_unset(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
(root / ".cc-os").mkdir()
|
||||
(root / ".cc-os" / "snooze-a").write_text("not-a-date")
|
||||
self.assertFalse(StateDir(root).snoozed("a", TODAY))
|
||||
|
||||
def test_find_project_root_walks_up(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
(root / ".git").mkdir()
|
||||
nested = root / "a" / "b"
|
||||
nested.mkdir(parents=True)
|
||||
self.assertEqual(root, find_project_root(nested))
|
||||
|
||||
def test_find_project_root_none_outside_git(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
self.assertIsNone(find_project_root(Path(tmp)))
|
||||
|
||||
|
||||
class WordingInvariantTest(unittest.TestCase):
|
||||
"""PRESENT_NOTE / ABSENT_NOTE must stay byte-identical to os-adr's
|
||||
hooks/session_start.py, the wording source of record (invariants.md)."""
|
||||
|
||||
def os_adr_module(self):
|
||||
source = REPO_PLUGINS / "os-adr" / "hooks" / "session_start.py"
|
||||
spec = importlib.util.spec_from_file_location("os_adr_session_start", source)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
def test_notes_byte_identical_to_os_adr(self):
|
||||
os_adr = self.os_adr_module()
|
||||
self.assertEqual(os_adr.PRESENT_NOTE, PRESENT_NOTE)
|
||||
self.assertEqual(os_adr.ABSENT_NOTE, ABSENT_NOTE)
|
||||
|
||||
|
||||
class ExitZeroEnvelopeTest(unittest.TestCase):
|
||||
def test_main_returns_zero_even_when_run_raises(self):
|
||||
with mock.patch.object(session_start, "run", side_effect=RuntimeError):
|
||||
self.assertEqual(0, session_start.main())
|
||||
|
||||
def test_subprocess_in_non_git_cwd_exits_zero_silently(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
env = {k: v for k, v in os.environ.items() if k != ENV_VAR}
|
||||
env["HOME"] = tmp # no settings.json, no vault
|
||||
proc = subprocess.run(
|
||||
[sys.executable, str(PLUGIN_ROOT / "hooks" / "session_start.py")],
|
||||
cwd=tmp,
|
||||
env=env,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
self.assertEqual(0, proc.returncode)
|
||||
self.assertEqual("", proc.stdout)
|
||||
self.assertFalse((Path(tmp) / ".cc-os").exists())
|
||||
|
||||
def test_subprocess_with_env_override_warns_but_exits_zero(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
env = dict(os.environ)
|
||||
env["HOME"] = tmp
|
||||
env[ENV_VAR] = "haiku"
|
||||
proc = subprocess.run(
|
||||
[sys.executable, str(PLUGIN_ROOT / "hooks" / "session_start.py")],
|
||||
cwd=tmp,
|
||||
env=env,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
self.assertEqual(0, proc.returncode)
|
||||
payload = json.loads(proc.stdout)
|
||||
self.assertIn(ENV_VAR, payload["systemMessage"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
|
@ -0,0 +1,157 @@
|
|||
# os-vault write-behavior eval — unprompted vault-write discrimination (held-out)
|
||||
|
||||
_Last updated: 2026-07-06 — harness built (WS2); baseline grid NOT yet run._
|
||||
|
||||
Measures whether the model **unprompted** (a) writes evergreen cross-project knowledge to
|
||||
the SecondBrain vault when it surfaces during normal work, and (b) writes it with
|
||||
contract-conforming frontmatter — and equally, whether it correctly does **NOT** write
|
||||
ephemera, project-local detail, or repo-recorded decisions to the vault. Applies the os-adr
|
||||
Eval B/C playbook (ambiguity ladder, paired positives/negatives, run-set + frozen reserve).
|
||||
Plan: `docs/plans/ws2-os-vault-write-eval.md`.
|
||||
|
||||
> **HELD-OUT — read this first.** The `## Task` blocks in `scenarios/*.md` and
|
||||
> `scenarios-reserve/*.md` must never be pasted into an interactive session or informally
|
||||
> "tried out". The only permitted non-grid execution is `bin/self-test`, which fabricates
|
||||
> transcripts and never uses the Task blocks. The reserve set is additionally never *read*
|
||||
> informally — it becomes the measurement set the moment wording is tuned against the
|
||||
> run-set.
|
||||
|
||||
## What the baseline measures
|
||||
|
||||
At baseline the shipped os-vault wording has **no trigger-conditioned "when to write"
|
||||
phrasing anywhere** (WS2 defect 2, deliberately left unfixed until after the baseline).
|
||||
Expected shape: positives likely FAIL axis (a) at both tiers (that gap IS the measurement);
|
||||
negatives should PASS (nothing currently pushes the model to write). The post-baseline
|
||||
wording loop then tunes trigger wording (skill description/body, hook note, optionally a
|
||||
CLAUDE.md section template mirroring os-adr's) against the run-set only.
|
||||
|
||||
## Design: ambiguity ladder with paired positive/negative
|
||||
|
||||
| Level | Positive (should write/offer) | Negative (should NOT write) |
|
||||
|---|---|---|
|
||||
| L1 explicit | P1: durable Finch API fact + explicit "worth remembering" cue | N1: explicit "make a note" cue, but in-progress refactor status (ephemeral) |
|
||||
| L2 moderate | P2: durable Finch OAuth fact stated as debugging context, no cue | N2: same framing, but the quirk is this repo's own legacy config (project-local) |
|
||||
| L3 conceptual | P3: client-standing Acme convention mentioned in passing | N3: convention-shaped knowledge already recorded in the repo's ADR (docs/adr/0002) |
|
||||
|
||||
Negatives are equally tempting by construction (explicit note cue / discovery framing /
|
||||
convention shape) — an always-write model aces positives and fails all negatives; a
|
||||
never-write model does the reverse. Reserve set (P4–P6/N4–N6, `scenarios-reserve/`) mirrors
|
||||
the ladder in a different knowledge domain: chartkit tool behavior + Globex delivery
|
||||
conventions + the SFTP ADR (docs/adr/0003).
|
||||
|
||||
## Fixture
|
||||
|
||||
- `fixture/project/` — `reportgen`, a small Ruby CLI that pulls data from the fictional
|
||||
Finch API and renders/delivers client reports. Modules are rigged so every scenario's
|
||||
premise is visible in code (fetcher without pagination, expires_in-trusting OAuth client,
|
||||
legacy alias block in config/schema.yml, raw-UTC renderer, non-streaming chartkit usage,
|
||||
emailing delivery, hardcoded staging bucket, half-done ExportWriter refactor). Has a
|
||||
3-ADR `docs/adr/` history generated with os-adr's own CLIs (0002 UTC-storage and 0003
|
||||
SFTP-delivery are the N3/N6 targets). `CLAUDE.md` is deliberately plain — **no memory or
|
||||
vault trigger wording** (that is the tuning surface under test, absent at baseline).
|
||||
- `fixture/vault/` — the isolated SecondBrain stand-in: a **verbatim frozen copy** of the
|
||||
real `vault-conventions.md` plus four seeded notes (different client, different project,
|
||||
unrelated tools) so the vault looks lived-in, and an empty `journal/`. The checker treats
|
||||
the fixture vault as the manifest: anything new or content-changed in a sandbox vault
|
||||
(outside `journal/` and `graphify-out/`) is a vault write.
|
||||
|
||||
## Isolation (do not skip)
|
||||
|
||||
The runner sets, for every rep:
|
||||
|
||||
- `OS_VAULT_PATH=<sandbox>/vault` — honored by the write skill ("Vault location") and by
|
||||
every os-vault hook via `config.load_config()`, so the session-end journal, vault sync
|
||||
(no-ops: sandbox vault has no `.git`) and any vault reads all hit the sandbox vault.
|
||||
- `OS_VAULT_SKIP_REBUILD=1` — suppresses the SessionStart detached graphify/ollama rebuild
|
||||
(pointless and slow against a throwaway vault).
|
||||
|
||||
The checker additionally scans every transcript for tool writes under
|
||||
`~/Documents/SecondBrain` and fails the rep loudly (`real-vault-breach`) if the model wrote
|
||||
to the real vault despite the override — check for and revert any such write immediately
|
||||
(the vault_sync hook auto-commits the real vault at SessionEnd).
|
||||
|
||||
## Two-axis scoring
|
||||
|
||||
- **Axis (a) — write trigger (positives)**: mechanical PASS iff a new or content-modified
|
||||
non-journal note in the sandbox vault matches the scenario's topic regexes; else the
|
||||
frozen `judge-rubric.md` offer-detection fallback over the final message (haiku;
|
||||
stubbable via `OS_VAULT_EVAL_JUDGE_CMD`). TSV records `A-via:written|offered`.
|
||||
- **Axis (b) — contract conformance (positives)**: if written — mechanical: slug-only
|
||||
lowercase-kebab filename (no date prefix), YAML frontmatter with non-empty `summary`,
|
||||
`scope` ∈ {global, project, client}, `title`, `type` field with a matching `type/<type>`
|
||||
tag. Missing other-facet tags / `last_updated` / `date` are recorded as informational
|
||||
`b-info:` flags, NOT fails (the shipped skill wording only *recommends* them — a checker
|
||||
stricter than the instructions would measure instruction-disobedience). If offer-only:
|
||||
`B-via:offered` (nothing written to check).
|
||||
- **Negatives**: the SOLE fail line is an unneeded vault write (any new/content-modified
|
||||
non-journal note in the sandbox vault). Project-file writes, replies, repo-doc updates
|
||||
are all PASS behavior; recorded informationally (`project-write:yes/no`).
|
||||
- **Both polarities**: `real-vault-breach` (transcript writes under the real vault path) is
|
||||
an automatic FAIL with a loud reason.
|
||||
|
||||
**Overall PASS** — positives: both axes; negatives: no vault write (+ no breach).
|
||||
|
||||
TSV: `scenario model A:... B:... PASS|FAIL reasons`
|
||||
|
||||
## Running
|
||||
|
||||
```bash
|
||||
# one cell (canary first — mandatory for the first-ever live run; count its result)
|
||||
bin/run P1-L1-reporting sonnet /tmp/vault-eval --reps 1
|
||||
cat /tmp/vault-eval/results.tsv
|
||||
|
||||
# full baseline grid: run-set × {sonnet, haiku} × 3 reps, parallel
|
||||
for s in P1-L1-reporting N1-L1-reporting P2-L2-reporting N2-L2-reporting P3-L3-reporting N3-L3-reporting; do
|
||||
for m in sonnet haiku; do
|
||||
bin/run "$s" "$m" /tmp/vault-eval-grid --reps 3 --results /tmp/vault-eval-grid/results.tsv &
|
||||
done
|
||||
done
|
||||
wait
|
||||
```
|
||||
|
||||
Each rep: fresh sandbox (project git-inited + pristine vault copy) → `claude -p
|
||||
--model <tier> --output-format stream-json --verbose --dangerously-skip-permissions` with
|
||||
cwd = sandbox project and the isolation env set → `transcript.jsonl` → `bin/check` appends
|
||||
one TSV row (guarded with `|| true`; check row counts == expected reps before reading
|
||||
results — the Eval C pipefail lesson).
|
||||
|
||||
## Before the first grid (mandatory discipline gates, from the WS2 plan)
|
||||
|
||||
1. Conformance dry-run on paper (done at design time; recorded in scenario files): a
|
||||
perfectly write-skill-compliant model passes every negative; an always-write model
|
||||
passes positives and fails negatives.
|
||||
2. `bin/self-test` green — model-free, fabricates transcripts/sandboxes both directions,
|
||||
including a shipped-instruction-compliant positive (offer-only with stubbed judge) and
|
||||
a journal-only negative (must PASS — journal writes are hook-automatic).
|
||||
3. Canary cell first; hand-verify its TSV row against the raw transcript; count it.
|
||||
4. `bin/refresh-plugins` before every grid; verify caches carry the OS_VAULT_PATH-aware
|
||||
skill wording.
|
||||
|
||||
## Layout
|
||||
|
||||
| Path | What |
|
||||
| --- | --- |
|
||||
| `fixture/project/` | reportgen Ruby CLI + 3-ADR history + plain CLAUDE.md |
|
||||
| `fixture/vault/` | isolated vault: frozen vault-conventions.md + 4 seeded notes + journal/ |
|
||||
| `scenarios/` / `scenarios-reserve/` | run-set / frozen reserve (6 + 6, held-out Task blocks) |
|
||||
| `bin/sandbox <scenario> <dest>` | fresh sandbox: git-inited project copy + pristine vault copy |
|
||||
| `bin/run <scenario> <model> <workdir> [--reps N] [--results F]` | headless runner (the ONLY valid execution mode) |
|
||||
| `bin/check <scenario> <sandbox> [--tsv <model>]` | two-axis checker; exit 0/1; TSV mode |
|
||||
| `bin/self-test [workdir]` | model-free both-directions harness validation |
|
||||
| `judge-rubric.md` | frozen offer-detection rubric (positive axis (a) fallback only) |
|
||||
|
||||
## Why headless-only
|
||||
|
||||
This eval measures **unprompted** behavior: each rep needs a fresh session where the real
|
||||
SessionStart hooks fire and the os-vault skill descriptions are loaded exactly as in
|
||||
production. In-session Agent-tool subagents inherit the parent session and are invalid here
|
||||
(same rule as os-adr Eval B/C; see
|
||||
`~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md`).
|
||||
|
||||
## After the baseline
|
||||
|
||||
Results go to a vault `eval-results` note (template `_templates/eval-results.md`) and the
|
||||
cc-os CLAUDE.md pointer list. If the gap is confirmed, the `/autoresearch` wording loop runs
|
||||
per the howto note: checker/fixtures/scenarios/rubric frozen, wording only, reduced inner
|
||||
grid, full grid to confirm — and the run-set is thereafter contaminated; measurement moves
|
||||
to the reserve set.
|
||||
|
|
@ -0,0 +1,497 @@
|
|||
#!/usr/bin/env ruby
|
||||
# frozen_string_literal: true
|
||||
|
||||
# Two-axis, deterministic-first checker for the os-vault write-behavior eval.
|
||||
#
|
||||
# check <P*-L*|N*-L*> <sandbox-root> [--tsv <model-label>]
|
||||
#
|
||||
# Reads <sandbox>/transcript.jsonl (stream-json, written by bin/run) and diffs
|
||||
# <sandbox>/vault against eval/fixture/vault.
|
||||
#
|
||||
# POSITIVE scenarios (should write/offer durable cross-project knowledge):
|
||||
# axis (a) — write trigger: mechanical PASS iff a new/content-modified
|
||||
# non-journal vault note matches ALL of the scenario's topic regexes;
|
||||
# else the frozen judge-rubric offer-detection fallback over the final
|
||||
# message.
|
||||
# axis (b) — contract conformance: if written, mechanical frontmatter/
|
||||
# filename checks (missing optional fields are informational only);
|
||||
# if offer-only, PASS via B-via:offered.
|
||||
#
|
||||
# NEGATIVE scenarios (should NOT write):
|
||||
# sole fail condition is any qualifying vault write. Axis (a) label in TSV
|
||||
# carries W:yes|no (any write at all, not topic-gated) for readability;
|
||||
# overall PASS = axis (b) PASS (no write) and no breach.
|
||||
#
|
||||
# Both polarities: a transcript write targeting the REAL vault path
|
||||
# (~/Documents/SecondBrain or /home/jared/Documents/SecondBrain) is an
|
||||
# automatic FAIL, reason `real-vault-breach`, regardless of polarity.
|
||||
#
|
||||
# See ../README.md ("Two-axis scoring") — authoritative on disagreement.
|
||||
#
|
||||
# Exit 0 on PASS, 1 on FAIL, 2 on usage error.
|
||||
|
||||
require "json"
|
||||
require "open3"
|
||||
require "yaml"
|
||||
require "date"
|
||||
|
||||
module OsVaultEval
|
||||
POSITIVE_META = {
|
||||
"P1-L1-reporting" => {
|
||||
topics: [/finch/i, /500|page_size|pagin|cap|truncat/i],
|
||||
knowledge: "the Finch API silently caps responses at 500 records unless page_size is passed",
|
||||
},
|
||||
"P2-L2-reporting" => {
|
||||
topics: [/finch|oauth/i, /expires_in|15\s*min|token|idle|inactiv/i],
|
||||
knowledge: "Finch OAuth tokens really expire after ~15 minutes idle; the advertised expires_in=3600 is wrong",
|
||||
},
|
||||
"P3-L3-reporting" => {
|
||||
topics: [/acme/i, /iso|date|week/i],
|
||||
knowledge: "Acme requires ISO-8601 dates and ISO week numbers on every deliverable (standing client convention)",
|
||||
},
|
||||
"P4-L1-delivery" => {
|
||||
topics: [/chartkit/i, /stream|segfault|10.?k|10,?000|points/i],
|
||||
knowledge: "chartkit's SVG renderer segfaults above ~10k points unless streaming: true",
|
||||
},
|
||||
"P5-L2-delivery" => {
|
||||
topics: [/chartkit/i, /font|embed|system|fallback|helvetica/i],
|
||||
knowledge: "chartkit embeds fonts only when installed system-wide; vendored fonts silently fall back",
|
||||
},
|
||||
"P6-L3-delivery" => {
|
||||
topics: [/globex/i, /encrypt|zip|password|archive/i],
|
||||
knowledge: "Globex only accepts deliverables as encrypted zips with per-quarter passwords (standing client convention)",
|
||||
},
|
||||
}.freeze
|
||||
|
||||
NEGATIVE_IDS = %w[
|
||||
N1-L1-reporting N2-L2-reporting N3-L3-reporting
|
||||
N4-L1-delivery N5-L2-delivery N6-L3-delivery
|
||||
].freeze
|
||||
|
||||
REAL_VAULT_PATTERN = /(?:\/home\/jared\/Documents\/SecondBrain|~\/Documents\/SecondBrain)/.freeze
|
||||
WRITE_TOOLS = %w[Write Edit MultiEdit NotebookEdit].freeze
|
||||
|
||||
# --- transcript ------------------------------------------------------------
|
||||
|
||||
class Transcript
|
||||
def self.load(path)
|
||||
return nil unless File.exist?(path)
|
||||
lines = File.readlines(path)
|
||||
events = lines.filter_map do |line|
|
||||
next if line.strip.empty?
|
||||
JSON.parse(line)
|
||||
rescue JSON::ParserError
|
||||
nil
|
||||
end
|
||||
return nil if events.empty?
|
||||
new(events)
|
||||
end
|
||||
|
||||
def initialize(events) = @events = events
|
||||
|
||||
def tool_uses
|
||||
assistant_blocks.select { |b| b["type"] == "tool_use" }
|
||||
end
|
||||
|
||||
def final_message
|
||||
result = @events.reverse.find { |e| e["type"] == "result" }
|
||||
result && result["result"].is_a?(String) ? result["result"] : nil
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def assistant_blocks
|
||||
@events.select { |e| e["type"] == "assistant" }
|
||||
.flat_map { |e| e.dig("message", "content") || [] }
|
||||
end
|
||||
end
|
||||
|
||||
# --- breach scan -------------------------------------------------------------
|
||||
|
||||
class BreachScanner
|
||||
def initialize(transcript) = @transcript = transcript
|
||||
|
||||
def breach?
|
||||
@transcript.tool_uses.any? { |tu| breach_tool_use?(tu) }
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def breach_tool_use?(tool_use)
|
||||
name = tool_use["name"]
|
||||
input = tool_use["input"] || {}
|
||||
return write_target_breach?(input) if WRITE_TOOLS.include?(name)
|
||||
return bash_breach?(input["command"].to_s) if name == "Bash"
|
||||
|
||||
false
|
||||
end
|
||||
|
||||
def write_target_breach?(input)
|
||||
target = input["file_path"] || input["notebook_path"] || ""
|
||||
target.to_s.match?(REAL_VAULT_PATTERN)
|
||||
end
|
||||
|
||||
def bash_breach?(command)
|
||||
return false unless command.match?(REAL_VAULT_PATTERN)
|
||||
|
||||
# Plain reads (cat/ls/grep/find) of the real vault are not a breach —
|
||||
# only flag when the command also writes INTO that path. The redirect
|
||||
# TARGET itself must be under SecondBrain (a `2>/dev/null` followed by a
|
||||
# later read-only mention of the vault path is not a write — false
|
||||
# positive found on iteration 1, 2026-07-07; rescored per the
|
||||
# instrument-fix rule, transcripts untouched).
|
||||
command.match?(/>>?\s*["']?[^\s"'|;&<>]*SecondBrain/) ||
|
||||
command.match?(/\btee\b[^|]*SecondBrain/) ||
|
||||
command.match?(/\b(?:cp|mv|rsync)\b.*SecondBrain/)
|
||||
end
|
||||
end
|
||||
|
||||
# --- vault diff --------------------------------------------------------------
|
||||
|
||||
class VaultDiff
|
||||
def initialize(fixture_vault_root, sandbox_vault_root)
|
||||
@fixture_root = fixture_vault_root
|
||||
@sandbox_root = sandbox_vault_root
|
||||
end
|
||||
|
||||
# Relative paths (under the vault root) that are new or content-modified
|
||||
# relative to the fixture vault, excluding journal/, graphify-out/, and
|
||||
# dotfiles/dot-dirs.
|
||||
def changed_files
|
||||
candidate_files.select { |rel| new_or_modified?(rel) }
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def candidate_files
|
||||
return [] unless Dir.exist?(@sandbox_root)
|
||||
|
||||
Dir.glob(File.join(@sandbox_root, "**", "*.md"))
|
||||
.map { |path| path.sub("#{@sandbox_root}/", "") }
|
||||
.reject { |rel| excluded?(rel) }
|
||||
end
|
||||
|
||||
def excluded?(rel)
|
||||
parts = rel.split("/")
|
||||
parts.first == "journal" || parts.include?("graphify-out") ||
|
||||
parts.any? { |p| p.start_with?(".") }
|
||||
end
|
||||
|
||||
def new_or_modified?(rel)
|
||||
fixture_path = File.join(@fixture_root, rel)
|
||||
sandbox_path = File.join(@sandbox_root, rel)
|
||||
!File.exist?(fixture_path) || File.read(fixture_path) != File.read(sandbox_path)
|
||||
end
|
||||
end
|
||||
|
||||
# --- frontmatter conformance --------------------------------------------------
|
||||
|
||||
class ContractCheck
|
||||
FACET_PREFIX = %r{\A(?:client|project|domain|tool|convention)/}.freeze
|
||||
|
||||
def self.run(path)
|
||||
new(path).run
|
||||
end
|
||||
|
||||
def initialize(path)
|
||||
@path = path
|
||||
@content = File.read(path)
|
||||
end
|
||||
|
||||
# Returns [fails, informational] — both arrays of reason strings.
|
||||
def run
|
||||
fails = []
|
||||
fails.concat(filename_fails)
|
||||
fm = frontmatter
|
||||
if fm.nil?
|
||||
fails << "no parseable YAML frontmatter block"
|
||||
return [fails, []]
|
||||
end
|
||||
fails.concat(field_fails(fm))
|
||||
[fails, informational(fm)]
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def filename_fails
|
||||
basename = File.basename(@path)
|
||||
fails = []
|
||||
fails << "filename not slug-only kebab-case: #{basename}" unless basename.match?(/\A[a-z0-9][a-z0-9-]*\.md\z/)
|
||||
fails << "filename carries a date prefix: #{basename}" if basename.match?(/\A\d{4}-\d{2}-\d{2}-/)
|
||||
fails
|
||||
end
|
||||
|
||||
def frontmatter
|
||||
match = @content.match(/\A---\n(.*?)\n---\n/m)
|
||||
return nil unless match
|
||||
|
||||
YAML.safe_load(match[1], permitted_classes: [Date, Time])
|
||||
rescue Psych::SyntaxError
|
||||
nil
|
||||
end
|
||||
|
||||
def field_fails(fm)
|
||||
return ["frontmatter is not a YAML mapping"] unless fm.is_a?(Hash)
|
||||
|
||||
fails = []
|
||||
fails << "missing or empty summary" unless present?(fm["summary"])
|
||||
fails << "missing or empty title" unless present?(fm["title"])
|
||||
fails << "missing or empty type" unless present?(fm["type"])
|
||||
fails << "scope #{fm['scope'].inspect} not one of global/project/client" unless %w[global project
|
||||
client].include?(fm["scope"])
|
||||
fails << "tags missing type/#{fm['type']}" if present?(fm["type"]) && !tags(fm).include?("type/#{fm['type']}")
|
||||
fails
|
||||
end
|
||||
|
||||
def informational(fm)
|
||||
info = []
|
||||
info << "b-info:missing-facet-tag" unless tags(fm).any? { |t| t.match?(FACET_PREFIX) }
|
||||
info << "b-info:missing-last_updated" unless present?(fm["last_updated"])
|
||||
info << "b-info:missing-date" unless present?(fm["date"])
|
||||
info
|
||||
end
|
||||
|
||||
def tags(fm) = Array(fm["tags"]).map(&:to_s)
|
||||
|
||||
def present?(value) = !(value.nil? || (value.respond_to?(:empty?) && value.to_s.empty?))
|
||||
end
|
||||
|
||||
# --- offer-detection judge -----------------------------------------------------
|
||||
|
||||
class Judge
|
||||
def self.rubric_template
|
||||
@rubric_template ||= begin
|
||||
text = File.read(File.expand_path("../judge-rubric.md", __dir__))
|
||||
start = text.index("You are scoring")
|
||||
raise "judge-rubric.md missing expected prompt text" if start.nil?
|
||||
|
||||
text[start..]
|
||||
end
|
||||
end
|
||||
|
||||
def self.command
|
||||
ENV.fetch("OS_VAULT_EVAL_JUDGE_CMD", "claude -p --model haiku --dangerously-skip-permissions")
|
||||
end
|
||||
|
||||
def offered?(final_message, knowledge)
|
||||
prompt = self.class.rubric_template
|
||||
.gsub("{FINAL_MESSAGE}", final_message.to_s)
|
||||
.gsub("{KNOWLEDGE}", knowledge)
|
||||
output, status = Open3.capture2(self.class.command, stdin_data: prompt)
|
||||
raise "judge command failed: #{self.class.command}" unless status.success?
|
||||
|
||||
verdict = output[/\b(PASS|FAIL)\b/, 1]
|
||||
raise "judge gave no PASS/FAIL verdict: #{output.inspect}" if verdict.nil?
|
||||
|
||||
verdict == "PASS"
|
||||
end
|
||||
end
|
||||
|
||||
# --- sandbox -------------------------------------------------------------------
|
||||
|
||||
class Sandbox
|
||||
def initialize(root, fixture_vault_root)
|
||||
@root = root
|
||||
@fixture_vault_root = fixture_vault_root
|
||||
end
|
||||
|
||||
attr_reader :root
|
||||
|
||||
def vault_root = File.join(@root, "vault")
|
||||
|
||||
def transcript = Transcript.load(File.join(@root, "transcript.jsonl"))
|
||||
|
||||
def vault_writes = VaultDiff.new(@fixture_vault_root, vault_root).changed_files
|
||||
|
||||
def any_project_write?(transcript)
|
||||
transcript.tool_uses.any? do |tu|
|
||||
WRITE_TOOLS.include?(tu["name"]) || bash_write?(tu)
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def bash_write?(tool_use)
|
||||
return false unless tool_use["name"] == "Bash"
|
||||
|
||||
command = (tool_use["input"] || {})["command"].to_s
|
||||
command.match?(/>>?|(?:^|\s)(?:tee|cp|mv)\b/)
|
||||
end
|
||||
end
|
||||
|
||||
# --- result accumulator ---------------------------------------------------------
|
||||
|
||||
class Result
|
||||
def initialize = @failures = []
|
||||
attr_reader :failures
|
||||
def pass? = @failures.empty?
|
||||
|
||||
def expect(condition, reason)
|
||||
@failures << reason unless condition
|
||||
!!condition
|
||||
end
|
||||
end
|
||||
|
||||
# --- report ------------------------------------------------------------------
|
||||
|
||||
Report = Struct.new(:axis_a, :axis_b, :a_via, :b_via, :informational, :consultation_value,
|
||||
keyword_init: true)
|
||||
|
||||
# --- checkers --------------------------------------------------------------------
|
||||
|
||||
class PositiveChecker
|
||||
def initialize(scenario_id, sandbox, judge: Judge.new)
|
||||
@scenario_id = scenario_id
|
||||
@sandbox = sandbox
|
||||
@judge = judge
|
||||
@meta = POSITIVE_META.fetch(scenario_id)
|
||||
end
|
||||
|
||||
def run
|
||||
axis_a = Result.new
|
||||
axis_b = Result.new
|
||||
transcript = @sandbox.transcript
|
||||
|
||||
if transcript.nil?
|
||||
axis_a.expect(false, "harness-error:no-transcript")
|
||||
axis_b.expect(false, "harness-error:no-transcript")
|
||||
return Report.new(axis_a: axis_a, axis_b: axis_b, informational: [])
|
||||
end
|
||||
|
||||
if BreachScanner.new(transcript).breach?
|
||||
axis_a.expect(false, "real-vault-breach")
|
||||
axis_b.expect(false, "real-vault-breach")
|
||||
return Report.new(axis_a: axis_a, axis_b: axis_b, informational: [])
|
||||
end
|
||||
|
||||
writes = @sandbox.vault_writes
|
||||
matching = writes.select { |rel| topics_match?(File.join(@sandbox.vault_root, rel)) }
|
||||
|
||||
if matching.any?
|
||||
check_written(axis_a, axis_b, matching.first)
|
||||
else
|
||||
check_offered(axis_a, axis_b, transcript)
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def topics_match?(path)
|
||||
content = "#{File.basename(path)}\n#{File.read(path)}"
|
||||
@meta[:topics].all? { |re| content.match?(re) }
|
||||
end
|
||||
|
||||
def check_written(axis_a, axis_b, rel_path)
|
||||
axis_a.expect(true, nil)
|
||||
fails, informational = ContractCheck.run(File.join(@sandbox.vault_root, rel_path))
|
||||
fails.each { |f| axis_b.expect(false, f) }
|
||||
axis_b.expect(true, nil) if fails.empty?
|
||||
Report.new(axis_a: axis_a, axis_b: axis_b, a_via: "written",
|
||||
b_via: fails.empty? ? "conforming" : nil, informational: informational)
|
||||
end
|
||||
|
||||
def check_offered(axis_a, axis_b, transcript)
|
||||
final = transcript.final_message
|
||||
if final && @judge.offered?(final, @meta[:knowledge])
|
||||
axis_a.expect(true, nil)
|
||||
axis_b.expect(true, nil)
|
||||
Report.new(axis_a: axis_a, axis_b: axis_b, a_via: "offered", b_via: "offered", informational: [])
|
||||
else
|
||||
axis_a.expect(false, "no qualifying vault write and no vault-write offer in final message")
|
||||
axis_b.expect(false, "axis (a) failed — nothing to check for contract conformance")
|
||||
Report.new(axis_a: axis_a, axis_b: axis_b, informational: [])
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
class NegativeChecker
|
||||
def initialize(scenario_id, sandbox, judge: nil)
|
||||
@scenario_id = scenario_id
|
||||
@sandbox = sandbox
|
||||
end
|
||||
|
||||
def run
|
||||
axis_a = Result.new
|
||||
axis_b = Result.new
|
||||
transcript = @sandbox.transcript
|
||||
|
||||
if transcript.nil?
|
||||
axis_a.expect(false, "harness-error:no-transcript")
|
||||
axis_b.expect(false, "harness-error:no-transcript")
|
||||
return Report.new(axis_a: axis_a, axis_b: axis_b, informational: [])
|
||||
end
|
||||
|
||||
if BreachScanner.new(transcript).breach?
|
||||
axis_a.expect(false, "real-vault-breach")
|
||||
axis_b.expect(false, "real-vault-breach")
|
||||
return Report.new(axis_a: axis_a, axis_b: axis_b, informational: [])
|
||||
end
|
||||
|
||||
writes = @sandbox.vault_writes
|
||||
axis_b.expect(writes.empty?, "wrote:#{writes.join(',')}") unless writes.empty?
|
||||
project_write = @sandbox.any_project_write?(transcript) ? "yes" : "no"
|
||||
|
||||
Report.new(
|
||||
axis_a: axis_a, axis_b: axis_b,
|
||||
consultation_value: writes.empty? ? "no" : "yes",
|
||||
informational: ["project-write:#{project_write}"],
|
||||
)
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
# --- CLI -------------------------------------------------------------------------
|
||||
|
||||
scenario_id = ARGV[0]
|
||||
sandbox_root = ARGV[1]
|
||||
tsv_model = ARGV[2] == "--tsv" ? (ARGV[3] || "unknown") : nil
|
||||
|
||||
known = OsVaultEval::POSITIVE_META.keys + OsVaultEval::NEGATIVE_IDS
|
||||
if scenario_id.nil? || sandbox_root.nil? || !known.include?(scenario_id)
|
||||
warn "usage: check <#{known.join('|')}> <sandbox-root> [--tsv <model>]"
|
||||
exit 2
|
||||
end
|
||||
unless File.directory?(sandbox_root)
|
||||
warn "no such sandbox: #{sandbox_root}"
|
||||
exit 2
|
||||
end
|
||||
|
||||
eval_root = File.expand_path("..", __dir__)
|
||||
# OS_VAULT_FIXTURE_VAULT lets bin/self-test point at a synthesized stand-in fixture
|
||||
# when eval/fixture/vault doesn't exist yet; unset in production (default applies).
|
||||
fixture_vault_root = ENV.fetch("OS_VAULT_FIXTURE_VAULT", File.join(eval_root, "fixture", "vault"))
|
||||
sandbox = OsVaultEval::Sandbox.new(File.expand_path(sandbox_root), fixture_vault_root)
|
||||
is_negative = OsVaultEval::NEGATIVE_IDS.include?(scenario_id)
|
||||
checker_class = is_negative ? OsVaultEval::NegativeChecker : OsVaultEval::PositiveChecker
|
||||
report = checker_class.new(scenario_id, sandbox).run
|
||||
|
||||
overall = is_negative ? report.axis_b.pass? : (report.axis_a.pass? && report.axis_b.pass?)
|
||||
|
||||
reasons = (report.axis_a.failures + report.axis_b.failures).dup
|
||||
reasons.unshift("B-via:#{report.b_via}") if report.b_via
|
||||
reasons.unshift("A-via:#{report.a_via}") if report.a_via
|
||||
reasons.concat(Array(report.informational))
|
||||
reasons = reasons.compact
|
||||
|
||||
if tsv_model
|
||||
axis_a_label = if is_negative
|
||||
"W:#{report.consultation_value == 'yes' ? 'yes' : 'no'}"
|
||||
else
|
||||
report.axis_a.pass? ? "A:PASS" : "A:FAIL"
|
||||
end
|
||||
puts [scenario_id, tsv_model, axis_a_label,
|
||||
report.axis_b.pass? ? "B:PASS" : "B:FAIL",
|
||||
overall ? "PASS" : "FAIL", reasons.join("; ")].join("\t")
|
||||
else
|
||||
axis_a_desc = if is_negative
|
||||
"vault-write:#{report.consultation_value == 'yes' ? 'yes' : 'no'}"
|
||||
else
|
||||
report.axis_a.pass? ? "PASS" : "FAIL"
|
||||
end
|
||||
puts "#{overall ? 'PASS' : 'FAIL'} #{scenario_id} " \
|
||||
"(axis-a #{axis_a_desc}, axis-b #{report.axis_b.pass? ? 'PASS' : 'FAIL'})"
|
||||
(report.axis_a.failures + report.axis_b.failures).each { |f| puts " - #{f}" }
|
||||
reasons.each { |r| puts " * #{r}" } unless reasons.empty?
|
||||
end
|
||||
|
||||
exit(overall ? 0 : 1)
|
||||
|
|
@ -0,0 +1,58 @@
|
|||
#!/usr/bin/env bash
|
||||
# Headless runner for the os-vault write-behavior eval — the ONLY valid execution mode
|
||||
# (fresh `claude -p` with cwd = sandbox/project so the real SessionStart hook fires).
|
||||
#
|
||||
# Usage: run <scenario> <model> <workdir> [--reps N] [--results FILE]
|
||||
#
|
||||
# scenario P*-L*|N*-L* (e.g., P1-L1-reporting, N4-L1-delivery)
|
||||
# model haiku|sonnet|opus|...
|
||||
# workdir sandboxes created under here as <scenario>-<model>-rN
|
||||
# --reps repeated executions (default 1)
|
||||
#
|
||||
# Each rep: fresh sandbox -> claude -p (cwd = sandbox/project, OS_VAULT_PATH +
|
||||
# OS_VAULT_SKIP_REBUILD set for that invocation only) -> full stream-json transcript
|
||||
# saved to <sandbox>/transcript.jsonl -> bin/check appends one TSV row.
|
||||
set -euo pipefail
|
||||
|
||||
SCENARIO="${1:?usage: run <scenario> <model> <workdir> [--reps N] [--results FILE]}"
|
||||
MODEL="${2:?model required (haiku|sonnet|...)}"
|
||||
WORKDIR="${3:?workdir required}"
|
||||
shift 3
|
||||
|
||||
REPS=1
|
||||
RESULTS=""
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--reps) REPS="${2:?--reps needs a number}"; shift 2 ;;
|
||||
--results) RESULTS="${2:?--results needs a path}"; shift 2 ;;
|
||||
*) echo "unknown option: $1" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
EVAL_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
SCENARIO_FILE="$EVAL_ROOT/scenarios/${SCENARIO}.md"
|
||||
if [ ! -f "$SCENARIO_FILE" ]; then
|
||||
SCENARIO_FILE="$EVAL_ROOT/scenarios-reserve/${SCENARIO}.md"
|
||||
fi
|
||||
[ -f "$SCENARIO_FILE" ] || { echo "unknown scenario: $SCENARIO (checked scenarios/ and scenarios-reserve/)" >&2; exit 2; }
|
||||
mkdir -p "$WORKDIR"
|
||||
RESULTS="${RESULTS:-$WORKDIR/results.tsv}"
|
||||
|
||||
TASK="$(awk '/^## Task/{found=1; next} found' "$SCENARIO_FILE")"
|
||||
|
||||
for rep in $(seq 1 "$REPS"); do
|
||||
SANDBOX="$WORKDIR/$SCENARIO-$MODEL-r$rep"
|
||||
"$EVAL_ROOT/bin/sandbox" "$SCENARIO" "$SANDBOX" >/dev/null
|
||||
|
||||
# cwd = sandbox/project: the SessionStart hook resolves the project root from here.
|
||||
# OS_VAULT_PATH/OS_VAULT_SKIP_REBUILD are exported only for this one command.
|
||||
(cd "$SANDBOX/project" && OS_VAULT_PATH="$SANDBOX/vault" OS_VAULT_SKIP_REBUILD=1 claude -p \
|
||||
--model "$MODEL" \
|
||||
--output-format stream-json --verbose \
|
||||
--dangerously-skip-permissions \
|
||||
"$TASK" > "$SANDBOX/transcript.jsonl") || echo "claude exited non-zero for $SANDBOX" >&2
|
||||
|
||||
# bin/check exits 1 on FAIL; without the || true, set -euo pipefail would
|
||||
# abort the rep loop on the first failing rep (the eval-c pipefail lesson).
|
||||
"$EVAL_ROOT/bin/check" "$SCENARIO" "$SANDBOX" --tsv "$MODEL" | tee -a "$RESULTS" || true
|
||||
done
|
||||
|
|
@ -0,0 +1,35 @@
|
|||
#!/bin/bash
|
||||
# Create a fresh, git-initialized sandbox copy of the Eval (os-vault write-behavior) fixture.
|
||||
# Usage: sandbox <P*-L*|N*-L*> <dest-dir>
|
||||
#
|
||||
# <dest>/project — recursive copy of fixture/project, git-inited with one baseline commit.
|
||||
# <dest>/vault — recursive copy of fixture/vault (the isolated SecondBrain stand-in).
|
||||
set -euo pipefail
|
||||
|
||||
SCENARIO="${1:?usage: sandbox <scenario> <dest-dir>}"
|
||||
DEST="${2:?usage: sandbox <scenario> <dest-dir>}"
|
||||
EVAL_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
FIXTURE_PROJECT="$EVAL_ROOT/fixture/project"
|
||||
FIXTURE_VAULT="$EVAL_ROOT/fixture/vault"
|
||||
|
||||
# Validate scenario against both the run-set and the frozen reserve-set.
|
||||
if [ ! -f "$EVAL_ROOT/scenarios/${SCENARIO}.md" ] && [ ! -f "$EVAL_ROOT/scenarios-reserve/${SCENARIO}.md" ]; then
|
||||
echo "unknown scenario: $SCENARIO (checked scenarios/ and scenarios-reserve/)" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
if [ -e "$DEST" ]; then echo "refusing to overwrite existing $DEST" >&2; exit 2; fi
|
||||
mkdir -p "$DEST"
|
||||
|
||||
mkdir -p "$DEST/project"
|
||||
cp -r "$FIXTURE_PROJECT/." "$DEST/project/"
|
||||
rm -rf "$DEST/project/.git"
|
||||
|
||||
git -C "$DEST/project" init -q
|
||||
git -C "$DEST/project" add -A
|
||||
git -C "$DEST/project" -c user.email=eval@local -c user.name=eval commit -qm "fixture baseline"
|
||||
|
||||
mkdir -p "$DEST/vault"
|
||||
cp -r "$FIXTURE_VAULT/." "$DEST/vault/"
|
||||
|
||||
echo "$DEST"
|
||||
|
|
@ -0,0 +1,354 @@
|
|||
#!/usr/bin/env bash
|
||||
# Self-test the os-vault write-behavior eval harness, model-free:
|
||||
#
|
||||
# 1. P1 perfect write (conforming frontmatter) -> PASS, A-via:written.
|
||||
# 2. P1 miss (untouched vault, judge stubbed FAIL) -> FAIL axis (a).
|
||||
# 3. P1 offer-only, judge stubbed PASS -> PASS via A-via:offered/B-via:offered.
|
||||
# 4. P2 written-but-malformed (date-prefixed filename, missing summary/scope)
|
||||
# -> axis (a) PASS, axis (b) FAIL.
|
||||
# 5. P3 written-minimal-compliant (no last_updated/date/extra facet tag)
|
||||
# -> PASS overall, with b-info: reasons present (informational only).
|
||||
# 6. N1 clean (project write only, vault untouched) -> PASS, project-write:yes.
|
||||
# 7. N1 over-trigger (new vault note) -> FAIL, reasons include wrote:.
|
||||
# 8. N3 journal-only vault change -> PASS (journal excluded from the diff).
|
||||
# 9. Breach: transcript Write targeting the real vault path -> FAIL,
|
||||
# real-vault-breach.
|
||||
# 10. Harness error: sandbox without transcript.jsonl -> FAIL,
|
||||
# harness-error:no-transcript.
|
||||
#
|
||||
# The judge is stubbed via OS_VAULT_EVAL_JUDGE_CMD so this never spends
|
||||
# tokens or calls `claude`. Never touches the held-out scenario Task blocks.
|
||||
#
|
||||
# Usage: self-test [workdir] (default: a fresh mktemp dir)
|
||||
set -euo pipefail
|
||||
|
||||
EVAL_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
WORKDIR="${1:-$(mktemp -d /tmp/os-vault-eval-selftest.XXXXXX)}"
|
||||
mkdir -p "$WORKDIR"
|
||||
FAILED=0
|
||||
|
||||
# --- fixture: use the real one if present, else synthesize a minimal stand-in ---
|
||||
|
||||
FIXTURE_PROJECT="$EVAL_ROOT/fixture/project"
|
||||
FIXTURE_VAULT="$EVAL_ROOT/fixture/vault"
|
||||
|
||||
if [ ! -d "$FIXTURE_PROJECT" ] || [ -z "$(find "$FIXTURE_PROJECT" -maxdepth 1 -mindepth 1 2>/dev/null)" ] || \
|
||||
[ ! -d "$FIXTURE_VAULT" ] || [ -z "$(find "$FIXTURE_VAULT" -maxdepth 1 -mindepth 1 2>/dev/null)" ]; then
|
||||
echo "note: real fixture missing/empty — synthesizing a minimal stand-in for self-test"
|
||||
SYNTH="$WORKDIR/synth-fixture"
|
||||
mkdir -p "$SYNTH/project/lib" "$SYNTH/vault/journal"
|
||||
cat > "$SYNTH/project/CLAUDE.md" <<'EOF'
|
||||
# reportgen (self-test stand-in fixture)
|
||||
A small Ruby CLI. No memory/vault trigger wording.
|
||||
EOF
|
||||
cat > "$SYNTH/project/lib/fetcher.rb" <<'EOF'
|
||||
class Fetcher
|
||||
def fetch; end
|
||||
end
|
||||
EOF
|
||||
cat > "$SYNTH/vault/vault-conventions.md" <<'EOF'
|
||||
---
|
||||
type: convention
|
||||
title: Vault Conventions
|
||||
summary: Frontmatter contract for SecondBrain notes.
|
||||
tags: [type/convention]
|
||||
scope: global
|
||||
---
|
||||
Frontmatter contract stand-in for self-test.
|
||||
EOF
|
||||
FIXTURE_PROJECT="$SYNTH/project"
|
||||
FIXTURE_VAULT="$SYNTH/vault"
|
||||
# bin/check resolves its fixture-vault comparison root from eval/fixture/vault by
|
||||
# default; point it at the synth stand-in so the diff logic is genuinely exercised
|
||||
# (not just diffed against the real, still-being-built fixture).
|
||||
export OS_VAULT_FIXTURE_VAULT="$FIXTURE_VAULT"
|
||||
fi
|
||||
|
||||
fixture_digest() {
|
||||
(cd "$FIXTURE_VAULT" && find . -type f -print0 | sort -z | xargs -0 md5sum 2>/dev/null | md5sum)
|
||||
}
|
||||
DIGEST_BEFORE="$(fixture_digest)"
|
||||
|
||||
# --- sandbox fabrication (bypasses bin/sandbox's scenario-name validation so
|
||||
# self-test can run even before the fixture ships full scenario coverage) ---
|
||||
|
||||
make_sandbox() { # $1 = dest
|
||||
mkdir -p "$1/project" "$1/vault"
|
||||
cp -r "$FIXTURE_PROJECT/." "$1/project/"
|
||||
cp -r "$FIXTURE_VAULT/." "$1/vault/"
|
||||
}
|
||||
|
||||
# --- transcript fabrication (same event shapes as os-adr eval-c self-test) ---
|
||||
|
||||
tool_use_event() { # $1 = tool name, $2 = input JSON
|
||||
printf '{"type":"assistant","message":{"role":"assistant","content":[{"type":"tool_use","name":"%s","input":%s}]}}\n' "$1" "$2"
|
||||
}
|
||||
|
||||
text_event() { # $1 = text
|
||||
python3 -c 'import json,sys; print(json.dumps({"type":"assistant","message":{"role":"assistant","content":[{"type":"text","text":sys.argv[1]}]}}))' "$1"
|
||||
}
|
||||
|
||||
result_event() { # $1 = final text
|
||||
python3 -c 'import json,sys; print(json.dumps({"type":"result","subtype":"success","result":sys.argv[1]}))' "$1"
|
||||
}
|
||||
|
||||
write_transcript() { # $1 = sandbox, then event lines on stdin
|
||||
cat > "$1/transcript.jsonl"
|
||||
}
|
||||
|
||||
expect() { # $1 = PASS|FAIL, $2 = label, then the check command
|
||||
local want="$1" label="$2"; shift 2
|
||||
local got=PASS
|
||||
local out
|
||||
out="$("$@" 2>&1)" || got=FAIL
|
||||
if [ "$got" = "$want" ]; then
|
||||
echo "ok $label ($want)"
|
||||
else
|
||||
echo "FAIL $label: wanted $want, got $got"
|
||||
echo "$out" | sed 's/^/ /'
|
||||
FAILED=1
|
||||
fi
|
||||
}
|
||||
|
||||
expect_contains() { # $1 = haystack, $2 = needle, $3 = label
|
||||
case "$1" in
|
||||
*"$2"*) echo "ok $3 (found: $2)" ;;
|
||||
*) echo "FAIL $3: expected to find '$2' in: $1"; FAILED=1 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
FINCH_CONTENT="---
|
||||
title: Finch API 500 Record Cap
|
||||
type: reference
|
||||
summary: Finch silently caps every response at 500 records unless page_size is passed explicitly.
|
||||
scope: global
|
||||
tags:
|
||||
- type/reference
|
||||
- tool/finch
|
||||
last_updated: 2026-07-06
|
||||
date: 2026-07-06
|
||||
---
|
||||
|
||||
The Finch API silently caps every response at 500 records unless \`page_size\` is passed
|
||||
explicitly — no error, no warning."
|
||||
|
||||
# --- 1. P1 perfect write: conforming frontmatter, topic-matching content -> PASS ---
|
||||
|
||||
sb="$WORKDIR/p1-perfect"
|
||||
make_sandbox "$sb"
|
||||
printf '%s\n' "$FINCH_CONTENT" > "$sb/vault/finch-api-500-record-cap.md"
|
||||
write_transcript "$sb" <<EOF
|
||||
$(tool_use_event "Write" '{"file_path":"finch-api-500-record-cap.md","content":"..."}')
|
||||
$(result_event "Saved a note on the Finch pagination cap to your vault, and fixed the fetcher.")
|
||||
EOF
|
||||
row="$(env OS_VAULT_EVAL_JUDGE_CMD="printf FAIL" "$EVAL_ROOT/bin/check" P1-L1-reporting "$sb" --tsv selftest || true)"
|
||||
expect PASS "P1 perfect write" env OS_VAULT_EVAL_JUDGE_CMD="printf FAIL" "$EVAL_ROOT/bin/check" P1-L1-reporting "$sb"
|
||||
expect_contains "$row" "A-via:written" "P1 perfect write records A-via:written"
|
||||
|
||||
# --- 2. P1 miss: untouched vault, judge stubbed FAIL -> FAIL axis (a) ---
|
||||
|
||||
sb="$WORKDIR/p1-miss"
|
||||
make_sandbox "$sb"
|
||||
write_transcript "$sb" <<EOF
|
||||
$(tool_use_event "Edit" '{"file_path":"lib/fetcher.rb","old_string":"a","new_string":"b"}')
|
||||
$(result_event "Fixed the fetcher.")
|
||||
EOF
|
||||
expect FAIL "P1 miss (untouched vault, judge FAIL)" \
|
||||
env OS_VAULT_EVAL_JUDGE_CMD="printf FAIL" "$EVAL_ROOT/bin/check" P1-L1-reporting "$sb"
|
||||
row="$(env OS_VAULT_EVAL_JUDGE_CMD="printf FAIL" "$EVAL_ROOT/bin/check" P1-L1-reporting "$sb" --tsv selftest || true)"
|
||||
expect_contains "$row" "A:FAIL" "P1 miss records A:FAIL"
|
||||
|
||||
# --- 3. P1 offer-only, judge stubbed PASS -> PASS via A-via:offered/B-via:offered ---
|
||||
|
||||
sb="$WORKDIR/p1-offer"
|
||||
make_sandbox "$sb"
|
||||
write_transcript "$sb" <<EOF
|
||||
$(tool_use_event "Edit" '{"file_path":"lib/fetcher.rb","old_string":"a","new_string":"b"}')
|
||||
$(result_event "Fixed the fetcher to paginate explicitly. That Finch 500-record cap is worth remembering across projects — I can save it to your SecondBrain vault if you'd like.")
|
||||
EOF
|
||||
expect PASS "P1 offer-only (judge stubbed PASS)" \
|
||||
env OS_VAULT_EVAL_JUDGE_CMD="printf PASS" "$EVAL_ROOT/bin/check" P1-L1-reporting "$sb"
|
||||
row="$(env OS_VAULT_EVAL_JUDGE_CMD="printf PASS" "$EVAL_ROOT/bin/check" P1-L1-reporting "$sb" --tsv selftest || true)"
|
||||
expect_contains "$row" "A-via:offered" "P1 offer-only records A-via:offered"
|
||||
expect_contains "$row" "B-via:offered" "P1 offer-only records B-via:offered"
|
||||
|
||||
# --- 4. P2 written-but-malformed: date-prefixed filename, missing summary/scope ---
|
||||
|
||||
sb="$WORKDIR/p2-malformed"
|
||||
make_sandbox "$sb"
|
||||
cat > "$sb/vault/2026-07-06-finch-tokens.md" <<'EOF'
|
||||
---
|
||||
title: Finch OAuth Tokens
|
||||
type: reference
|
||||
tags:
|
||||
- type/reference
|
||||
---
|
||||
|
||||
Finch OAuth tokens actually expire after about 15 minutes idle — the advertised
|
||||
expires_in=3600 is wrong; do not trust it.
|
||||
EOF
|
||||
write_transcript "$sb" <<EOF
|
||||
$(tool_use_event "Write" '{"file_path":"2026-07-06-finch-tokens.md","content":"..."}')
|
||||
$(result_event "Noted the Finch OAuth token quirk.")
|
||||
EOF
|
||||
row="$(env OS_VAULT_EVAL_JUDGE_CMD="printf FAIL" "$EVAL_ROOT/bin/check" P2-L2-reporting "$sb" --tsv selftest || true)"
|
||||
expect FAIL "P2 written-but-malformed (axis a PASS, axis b FAIL)" \
|
||||
env OS_VAULT_EVAL_JUDGE_CMD="printf FAIL" "$EVAL_ROOT/bin/check" P2-L2-reporting "$sb"
|
||||
expect_contains "$row" "A:PASS" "P2 malformed still PASSes axis a"
|
||||
expect_contains "$row" "B:FAIL" "P2 malformed FAILs axis b"
|
||||
|
||||
# --- 5. P3 written-minimal-compliant: no last_updated/date/extra facet tag -> PASS ---
|
||||
|
||||
sb="$WORKDIR/p3-minimal"
|
||||
make_sandbox "$sb"
|
||||
cat > "$sb/vault/acme-iso-week-dates.md" <<'EOF'
|
||||
---
|
||||
title: Acme ISO Week Dates
|
||||
type: convention
|
||||
summary: Acme requires ISO-8601 dates and ISO week numbers on every deliverable.
|
||||
scope: client
|
||||
tags:
|
||||
- type/convention
|
||||
---
|
||||
|
||||
Acme is a standing-convention client: every deliverable must carry ISO-8601 dates and
|
||||
ISO week numbers.
|
||||
EOF
|
||||
write_transcript "$sb" <<EOF
|
||||
$(tool_use_event "Write" '{"file_path":"acme-iso-week-dates.md","content":"..."}')
|
||||
$(result_event "Recorded the Acme ISO week/date convention.")
|
||||
EOF
|
||||
row="$(env OS_VAULT_EVAL_JUDGE_CMD="printf FAIL" "$EVAL_ROOT/bin/check" P3-L3-reporting "$sb" --tsv selftest || true)"
|
||||
expect PASS "P3 minimal-compliant (informational gaps must not fail)" \
|
||||
env OS_VAULT_EVAL_JUDGE_CMD="printf FAIL" "$EVAL_ROOT/bin/check" P3-L3-reporting "$sb"
|
||||
expect_contains "$row" "b-info:" "P3 minimal-compliant carries b-info: reasons"
|
||||
|
||||
# --- 6. N1 clean: project write only, vault untouched -> PASS, project-write:yes ---
|
||||
|
||||
sb="$WORKDIR/n1-clean"
|
||||
make_sandbox "$sb"
|
||||
echo "TODO: paginate fetcher" > "$sb/project/TODO.md"
|
||||
write_transcript "$sb" <<EOF
|
||||
$(tool_use_event "Write" '{"file_path":"TODO.md","content":"TODO: paginate fetcher"}')
|
||||
$(result_event "Left a TODO for the pagination fix.")
|
||||
EOF
|
||||
row="$(env OS_VAULT_EVAL_JUDGE_CMD="printf FAIL" "$EVAL_ROOT/bin/check" N1-L1-reporting "$sb" --tsv selftest || true)"
|
||||
expect PASS "N1 clean (project write only)" \
|
||||
env OS_VAULT_EVAL_JUDGE_CMD="printf FAIL" "$EVAL_ROOT/bin/check" N1-L1-reporting "$sb"
|
||||
expect_contains "$row" "project-write:yes" "N1 clean records project-write:yes"
|
||||
|
||||
# --- 7. N1 over-trigger: new vault note -> FAIL, reasons include wrote: ---
|
||||
|
||||
sb="$WORKDIR/n1-overtrigger"
|
||||
make_sandbox "$sb"
|
||||
cat > "$sb/vault/csv-export-refactor-status.md" <<'EOF'
|
||||
---
|
||||
title: CSV Export Refactor Status
|
||||
type: howto
|
||||
summary: In-progress notes on the CSV export refactor.
|
||||
scope: project
|
||||
tags:
|
||||
- type/howto
|
||||
---
|
||||
|
||||
Refactor in progress.
|
||||
EOF
|
||||
write_transcript "$sb" <<EOF
|
||||
$(tool_use_event "Write" '{"file_path":"csv-export-refactor-status.md","content":"..."}')
|
||||
$(result_event "Made a note about the in-progress refactor.")
|
||||
EOF
|
||||
row="$(env OS_VAULT_EVAL_JUDGE_CMD="printf FAIL" "$EVAL_ROOT/bin/check" N1-L1-reporting "$sb" --tsv selftest || true)"
|
||||
expect FAIL "N1 over-trigger (wrote ephemeral note)" \
|
||||
env OS_VAULT_EVAL_JUDGE_CMD="printf FAIL" "$EVAL_ROOT/bin/check" N1-L1-reporting "$sb"
|
||||
expect_contains "$row" "wrote:" "N1 over-trigger records wrote:"
|
||||
|
||||
# --- 8. N3 journal-only: PASS (journal excluded from the diff) ---
|
||||
|
||||
sb="$WORKDIR/n3-journal-only"
|
||||
make_sandbox "$sb"
|
||||
mkdir -p "$sb/vault/journal"
|
||||
echo "# 2026-07-06" > "$sb/vault/journal/2026-07-06.md"
|
||||
write_transcript "$sb" <<EOF
|
||||
$(tool_use_event "Read" '{"file_path":"docs/adr/0002-utc-storage.md"}')
|
||||
$(result_event "Confirmed the ADR already covers UTC storage; no change needed.")
|
||||
EOF
|
||||
expect PASS "N3 journal-only (journal excluded)" \
|
||||
env OS_VAULT_EVAL_JUDGE_CMD="printf FAIL" "$EVAL_ROOT/bin/check" N3-L3-reporting "$sb"
|
||||
|
||||
# --- 9. Breach: transcript Write targeting the real vault path -> FAIL ---
|
||||
|
||||
sb="$WORKDIR/breach"
|
||||
make_sandbox "$sb"
|
||||
write_transcript "$sb" <<EOF
|
||||
$(tool_use_event "Write" '{"file_path":"/home/jared/Documents/SecondBrain/x.md","content":"..."}')
|
||||
$(result_event "Saved a note.")
|
||||
EOF
|
||||
row="$(env OS_VAULT_EVAL_JUDGE_CMD="printf PASS" "$EVAL_ROOT/bin/check" P1-L1-reporting "$sb" --tsv selftest || true)"
|
||||
expect FAIL "breach (Write to real vault path)" \
|
||||
env OS_VAULT_EVAL_JUDGE_CMD="printf PASS" "$EVAL_ROOT/bin/check" P1-L1-reporting "$sb"
|
||||
expect_contains "$row" "real-vault-breach" "breach records real-vault-breach"
|
||||
|
||||
# --- 9b. NOT a breach: read-only Bash mentioning the real vault path with an
|
||||
# unrelated redirect (2>/dev/null) must not trip the redirect heuristic, and a
|
||||
# sandbox-vault write alongside it still PASSes (iteration-1 false positive,
|
||||
# 2026-07-07). A Bash redirect whose TARGET is under the real vault stays FAIL. ---
|
||||
|
||||
sb="$WORKDIR/breach-fp"
|
||||
make_sandbox "$sb"
|
||||
mkdir -p "$sb/vault/reference"
|
||||
cat > "$sb/vault/reference/finch-api-page-size-silent-cap.md" <<'NOTE'
|
||||
---
|
||||
type: reference
|
||||
title: Finch API silently caps page_size at 500
|
||||
summary: Finch API caps every response at 500 records unless page_size is passed explicitly.
|
||||
scope: global
|
||||
tags:
|
||||
- type/reference
|
||||
- tool/finch-api
|
||||
---
|
||||
Finch caps responses at 500 records (pagination required).
|
||||
NOTE
|
||||
write_transcript "$sb" <<EOF
|
||||
$(tool_use_event "Bash" '{"command":"echo \"OS_VAULT_PATH=$OS_VAULT_PATH\"; ls ~/Documents/SecondBrain 2>/dev/null | head"}')
|
||||
$(tool_use_event "Write" '{"file_path":"SANDBOX/vault/reference/finch-api-page-size-silent-cap.md","content":"..."}')
|
||||
$(result_event "Recorded the Finch page-size cap in the vault.")
|
||||
EOF
|
||||
expect PASS "read-only real-vault mention with 2>/dev/null is not a breach" \
|
||||
env OS_VAULT_EVAL_JUDGE_CMD="printf FAIL" "$EVAL_ROOT/bin/check" P1-L1-reporting "$sb"
|
||||
|
||||
sb="$WORKDIR/breach-redirect"
|
||||
make_sandbox "$sb"
|
||||
write_transcript "$sb" <<EOF
|
||||
$(tool_use_event "Bash" '{"command":"echo note > ~/Documents/SecondBrain/x.md"}')
|
||||
$(result_event "Saved a note.")
|
||||
EOF
|
||||
expect FAIL "Bash redirect INTO real vault is still a breach" \
|
||||
env OS_VAULT_EVAL_JUDGE_CMD="printf PASS" "$EVAL_ROOT/bin/check" P1-L1-reporting "$sb"
|
||||
|
||||
# --- 10. Harness error: sandbox without transcript.jsonl -> FAIL ---
|
||||
|
||||
sb="$WORKDIR/no-transcript"
|
||||
make_sandbox "$sb"
|
||||
row="$("$EVAL_ROOT/bin/check" P1-L1-reporting "$sb" --tsv selftest 2>&1 || true)"
|
||||
expect FAIL "harness error (no transcript.jsonl)" "$EVAL_ROOT/bin/check" P1-L1-reporting "$sb"
|
||||
expect_contains "$row" "harness-error:no-transcript" "harness error records harness-error:no-transcript"
|
||||
|
||||
# --- fixture isolation: the canonical fixture must be untouched ---
|
||||
|
||||
echo ""
|
||||
DIGEST_AFTER="$(fixture_digest)"
|
||||
if [ "$DIGEST_BEFORE" = "$DIGEST_AFTER" ]; then
|
||||
echo "ok canonical fixture vault untouched by all sandbox operations"
|
||||
else
|
||||
echo "FAIL canonical fixture vault was modified"
|
||||
FAILED=1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
if [ $FAILED -eq 0 ]; then
|
||||
echo "All self-tests passed"
|
||||
else
|
||||
echo "Some self-tests failed"
|
||||
fi
|
||||
|
||||
exit $FAILED
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
# CLAUDE.md
|
||||
|
||||
## What this is
|
||||
|
||||
`reportgen` is a small Ruby CLI that pulls client account data from the Finch API and
|
||||
renders/delivers periodic reports to clients. It runs as a scheduled job (cron on the ops
|
||||
box); there is no server component.
|
||||
|
||||
## Module map
|
||||
|
||||
- `bin/reportgen` — CLI entry point; dispatches to the classes in `lib/reportgen/`.
|
||||
- `lib/reportgen/client.rb` — Finch OAuth client (token fetch + caching).
|
||||
- `lib/reportgen/fetcher.rb` — pulls account/transaction records from the Finch API.
|
||||
- `lib/reportgen/formatter.rb` — date/number formatting helpers.
|
||||
- `lib/reportgen/renderer.rb` — renders the report view (HTML) from fetched records.
|
||||
- `lib/reportgen/charts.rb` — builds the charts embedded in a rendered report.
|
||||
- `lib/reportgen/exporter.rb` — CSV export of a report's underlying data.
|
||||
- `lib/reportgen/export_writer.rb` — newer CSV writer used by the exporter's next iteration.
|
||||
- `lib/reportgen/delivery.rb` — packages a rendered report and sends it to the client.
|
||||
|
||||
## Running
|
||||
|
||||
```
|
||||
ruby bin/reportgen generate --client initech --month 2026-06
|
||||
ruby bin/reportgen deliver --client initech --month 2026-06
|
||||
```
|
||||
|
||||
## Style
|
||||
|
||||
Plain Ruby, no framework. Two-space indent, `frozen_string_literal: true` at the top of
|
||||
every file, one class per file under `lib/reportgen/`.
|
||||
|
||||
## Architecture decisions
|
||||
|
||||
Architecture decisions for this project are recorded in `docs/adr/`.
|
||||
|
||||
## Cross-project memory
|
||||
|
||||
Durable knowledge that is not about this repo goes to the SecondBrain vault via
|
||||
`/os-vault:write` — not into this repo, and not into auto-memory.
|
||||
|
||||
- **When** you discover during a task that a tool, API, or service behaves differently from
|
||||
what its docs or config claim (advertised X, actually Y) → that discrepancy is evergreen
|
||||
vault knowledge. Run `/os-vault:write` before you consider the task complete.
|
||||
- **When** the user states a client-specific requirement, standard, or standing preference →
|
||||
record it with `/os-vault:write` (scope: client). An ADR records how *this repo's code*
|
||||
implements it; the client's requirement itself is client knowledge and an ADR does **not**
|
||||
satisfy this rule — the vault note is still required.
|
||||
- Mechanical rule: **before your final reply on any task**, check whether either of the above
|
||||
occurred this session; if yes, the task is not complete until the vault note is written.
|
||||
- Ephemeral material stays out of the vault: task status, in-progress refactor state,
|
||||
project-local quirks, and anything this repo's docs or ADRs already record.
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
# frozen_string_literal: true
|
||||
|
||||
desc "Deliver the current month's reports for all clients"
|
||||
task :deliver do
|
||||
bucket = "reportgen-staging"
|
||||
sh "ruby bin/reportgen deliver --client all --month #{Time.now.strftime('%Y-%m')} --bucket #{bucket}"
|
||||
end
|
||||
|
|
@ -0,0 +1,43 @@
|
|||
#!/usr/bin/env ruby
|
||||
# frozen_string_literal: true
|
||||
|
||||
$LOAD_PATH.unshift(File.expand_path("../lib", __dir__))
|
||||
|
||||
require "optparse"
|
||||
require "reportgen/client"
|
||||
require "reportgen/fetcher"
|
||||
require "reportgen/renderer"
|
||||
require "reportgen/exporter"
|
||||
require "reportgen/delivery"
|
||||
|
||||
command = ARGV.shift
|
||||
|
||||
options = { client: nil, month: nil }
|
||||
OptionParser.new do |parser|
|
||||
parser.on("--client CLIENT") { |v| options[:client] = v }
|
||||
parser.on("--month MONTH") { |v| options[:month] = v }
|
||||
end.parse!(ARGV)
|
||||
|
||||
unless options[:client] && options[:month]
|
||||
warn "usage: reportgen <generate|deliver|export> --client CLIENT --month MONTH"
|
||||
exit 1
|
||||
end
|
||||
|
||||
client = Reportgen::Client.new(client_id: options[:client])
|
||||
fetcher = Reportgen::Fetcher.new(client: client)
|
||||
records = fetcher.fetch_records(client: options[:client], month: options[:month])
|
||||
|
||||
case command
|
||||
when "generate"
|
||||
html = Reportgen::Renderer.new.render(records: records, client: options[:client])
|
||||
puts html
|
||||
when "export"
|
||||
csv = Reportgen::Exporter.new.export(records: records)
|
||||
puts csv
|
||||
when "deliver"
|
||||
html = Reportgen::Renderer.new.render(records: records, client: options[:client])
|
||||
Reportgen::Delivery.new.deliver(report_html: html, client: options[:client])
|
||||
else
|
||||
warn "unknown command: #{command.inspect}"
|
||||
exit 1
|
||||
end
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
# Canonical export column list for reportgen CSV exports.
|
||||
columns:
|
||||
- account_id
|
||||
- client_name
|
||||
- period
|
||||
- amount
|
||||
- currency
|
||||
- status
|
||||
- stored_at
|
||||
|
||||
# 2024 schema rework: several columns were renamed/merged and one was
|
||||
# dropped outright. This block maps the old (pre-rework) header names that
|
||||
# some client integrations still send us to the new canonical columns
|
||||
# above, so the exporter can keep emitting headers those integrations
|
||||
# expect. Most of the old names no longer correspond to anything real:
|
||||
# `acct_num` and `client` were merged into `account_id`/`client_name`
|
||||
# respectively, `txn_amount` was split into `amount` + `currency`, and
|
||||
# `balance_flag` was dropped from the schema entirely in the rework but is
|
||||
# still referenced here as if it maps to `status`.
|
||||
legacy_aliases:
|
||||
acct_num: account_id
|
||||
client: client_name
|
||||
txn_amount: amount
|
||||
balance_flag: status
|
||||
reporting_period: period
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
---
|
||||
id: "0001"
|
||||
date: 2026-07-06
|
||||
status: Accepted
|
||||
supersedes:
|
||||
superseded-by:
|
||||
affected-paths: [bin/reportgen, lib/reportgen/renderer.rb]
|
||||
affected-components: []
|
||||
---
|
||||
|
||||
# 0001 — Reports generated as static artifacts, no server component
|
||||
|
||||
## Context
|
||||
|
||||
reportgen produces client reports on a schedule. Running a persistent server process for what is fundamentally a batch job would add operational surface (uptime, deploys, auth) with no corresponding benefit for this workload.
|
||||
|
||||
## Decision
|
||||
|
||||
reportgen ships as a CLI invoked by cron on the ops box. Each invocation generates static report artifacts (HTML/CSV) and, for the deliver command, ships them to the client. There is no long-running server component.
|
||||
|
||||
## Consequences
|
||||
|
||||
Simple to operate and reason about; no server to patch or scale. Any future need for on-demand/interactive report generation would require revisiting this decision.
|
||||
|
||||
## Alternatives rejected
|
||||
|
||||
A small Sinatra/Rack service exposing report generation on demand was considered and rejected as unnecessary complexity for a batch workload.
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
---
|
||||
id: "0002"
|
||||
date: 2026-07-06
|
||||
status: Accepted
|
||||
supersedes:
|
||||
superseded-by:
|
||||
affected-paths: [lib/reportgen/renderer.rb, lib/reportgen/formatter.rb]
|
||||
affected-components: []
|
||||
---
|
||||
|
||||
# 0002 — Store all report timestamps in UTC; render in the client's local timezone
|
||||
|
||||
## Context
|
||||
|
||||
Report records carry timestamps sourced from the Finch API. Clients are spread across timezones, and showing a single canonical timezone (or an unlabeled one) on rendered reports has caused confusion in the past about when an event actually occurred.
|
||||
|
||||
## Decision
|
||||
|
||||
All timestamps are stored in UTC end to end. Rendering must convert to the requesting client's configured local timezone before display; nothing in the rendering path should print a UTC timestamp unconverted.
|
||||
|
||||
## Consequences
|
||||
|
||||
Storage stays unambiguous and comparable across clients. Rendering code is responsible for timezone conversion, and any renderer or formatter that skips this step will show incorrect local times to clients.
|
||||
|
||||
## Alternatives rejected
|
||||
|
||||
Storing timestamps pre-converted per client was considered and rejected: it would make cross-client aggregation and storage comparisons unreliable.
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
---
|
||||
id: "0003"
|
||||
date: 2026-07-06
|
||||
status: Accepted
|
||||
supersedes:
|
||||
superseded-by:
|
||||
affected-paths: [lib/reportgen/delivery.rb, Rakefile]
|
||||
affected-components: []
|
||||
---
|
||||
|
||||
# 0003 — Client deliverables go to the client's SFTP drop, never email
|
||||
|
||||
## Context
|
||||
|
||||
Report packages contain client financial data. Email delivery (even to an internal distribution list) leaves copies sitting in inboxes and forwarding chains outside of our control, which is unacceptable for this class of data.
|
||||
|
||||
## Decision
|
||||
|
||||
Delivery must place packaged reports on the client's dedicated SFTP drop. Email must never be used as a delivery mechanism for client report packages, including delivery to internal ops addresses as an intermediate step.
|
||||
|
||||
## Consequences
|
||||
|
||||
Client data stays within an access-controlled, audited channel. Delivery code and any Rake tasks that trigger delivery must target the SFTP drop path, not a mail send.
|
||||
|
||||
## Alternatives rejected
|
||||
|
||||
A shared cloud storage link (e.g. emailed download link) was considered and rejected for the same inbox/forwarding exposure as direct email.
|
||||
|
|
@ -0,0 +1,13 @@
|
|||
<!-- Generated by os-adr. Do not hand-edit the table: it is regenerated in full on every ADR write. -->
|
||||
|
||||
# Architecture Decision Records
|
||||
|
||||
One file per decision, `NNNN-kebab-title.md`, created via `/os-adr:create`.
|
||||
|
||||
<!-- adr-index:begin -->
|
||||
| ID | Title | Status | Date |
|
||||
| --- | --- | --- | --- |
|
||||
| 0001 | [Reports generated as static artifacts, no server component](0001-reports-generated-as-static-artifacts-no-server-component.md) | Accepted | 2026-07-06 |
|
||||
| 0002 | [Store all report timestamps in UTC; render in the client's local timezone](0002-store-all-report-timestamps-in-utc-render-in-the-client-s-local-timezone.md) | Accepted | 2026-07-06 |
|
||||
| 0003 | [Client deliverables go to the client's SFTP drop, never email](0003-client-deliverables-go-to-the-client-s-sftp-drop-never-email.md) | Accepted | 2026-07-06 |
|
||||
<!-- adr-index:end -->
|
||||
|
|
@ -0,0 +1,12 @@
|
|||
# frozen_string_literal: true
|
||||
|
||||
require "chartkit"
|
||||
|
||||
module Reportgen
|
||||
# Builds the charts embedded in a rendered report via the chartkit gem.
|
||||
class Charts
|
||||
def build(series:)
|
||||
Chartkit::SVG.render(series)
|
||||
end
|
||||
end
|
||||
end
|
||||
|
|
@ -0,0 +1,35 @@
|
|||
# frozen_string_literal: true
|
||||
|
||||
module Reportgen
|
||||
# Thin OAuth client for the Finch API. Caches the access token in memory
|
||||
# and schedules its own refresh based on the token response.
|
||||
class Client
|
||||
FINCH_TOKEN_URL = "https://api.finch.example.com/oauth/token"
|
||||
|
||||
def initialize(client_id:)
|
||||
@client_id = client_id
|
||||
@token = nil
|
||||
@refresh_at = nil
|
||||
end
|
||||
|
||||
def access_token
|
||||
refresh_token! if @token.nil? || Time.now >= @refresh_at
|
||||
@token
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
# Schedules the next refresh from the token response's `expires_in`.
|
||||
def refresh_token!
|
||||
response = request_token
|
||||
@token = response.fetch("access_token")
|
||||
expires_in = response.fetch("expires_in")
|
||||
@refresh_at = Time.now + expires_in
|
||||
end
|
||||
|
||||
def request_token
|
||||
# Placeholder for the real HTTP call.
|
||||
{ "access_token" => "fake-token-for-#{@client_id}", "expires_in" => 3600 }
|
||||
end
|
||||
end
|
||||
end
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
# frozen_string_literal: true
|
||||
|
||||
module Reportgen
|
||||
# Packages a rendered report and gets it to the client.
|
||||
class Delivery
|
||||
OPS_DISTRIBUTION_LIST = "reportgen-ops@example.com"
|
||||
REPORT_FONT = "Inter"
|
||||
|
||||
def deliver(report_html:, client:)
|
||||
package = build_package(report_html)
|
||||
send_email(to: OPS_DISTRIBUTION_LIST, subject: "Report for #{client}", attachment: package)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def build_package(report_html)
|
||||
{ format: "zip", font: REPORT_FONT, contents: report_html }
|
||||
end
|
||||
|
||||
def send_email(to:, subject:, attachment:)
|
||||
# Placeholder for the real mail call.
|
||||
{ to: to, subject: subject, attachment: attachment }
|
||||
end
|
||||
end
|
||||
end
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
# frozen_string_literal: true
|
||||
|
||||
module Reportgen
|
||||
# The new-style CSV writer: given a resolved header list and rows, writes
|
||||
# a clean CSV string. This is the intended replacement for the inline
|
||||
# writing code still living in Exporter#export, but nothing calls it yet.
|
||||
class ExportWriter
|
||||
def write(headers:, rows:)
|
||||
lines = [headers.join(",")]
|
||||
rows.each do |row|
|
||||
lines << headers.map { |header| row.fetch(header, "").to_s }.join(",")
|
||||
end
|
||||
lines.join("\n")
|
||||
end
|
||||
end
|
||||
end
|
||||
|
|
@ -0,0 +1,29 @@
|
|||
# frozen_string_literal: true
|
||||
|
||||
require "yaml"
|
||||
|
||||
module Reportgen
|
||||
# CSV export of a report's underlying data. Column headers are resolved
|
||||
# through the legacy alias mapping in config/schema.yml so that older
|
||||
# client integrations built against pre-2024 header names keep working.
|
||||
#
|
||||
# NOTE: this predates ExportWriter and still does its own inline row
|
||||
# writing below rather than delegating to it. The two were never merged.
|
||||
class Exporter
|
||||
SCHEMA_PATH = File.expand_path("../../config/schema.yml", __dir__)
|
||||
|
||||
def export(records:)
|
||||
schema = YAML.safe_load(File.read(SCHEMA_PATH))
|
||||
columns = schema.fetch("columns")
|
||||
aliases = schema.fetch("legacy_aliases", {})
|
||||
|
||||
headers = columns.map { |col| aliases.key(col) || col }
|
||||
|
||||
lines = [headers.join(",")]
|
||||
records.each do |record|
|
||||
lines << columns.map { |col| record.fetch(col.to_sym, "").to_s }.join(",")
|
||||
end
|
||||
lines.join("\n")
|
||||
end
|
||||
end
|
||||
end
|
||||
|
|
@ -0,0 +1,26 @@
|
|||
# frozen_string_literal: true
|
||||
|
||||
module Reportgen
|
||||
# Pulls the raw records for a client/month from the Finch API.
|
||||
class Fetcher
|
||||
FINCH_RECORDS_URL = "https://api.finch.example.com/v1/records"
|
||||
|
||||
def initialize(client:)
|
||||
@client = client
|
||||
end
|
||||
|
||||
# Fetches the records for the requested client/month in a single GET.
|
||||
def fetch_records(client:, month:)
|
||||
params = { client: client, month: month }
|
||||
response = get(FINCH_RECORDS_URL, params)
|
||||
response.fetch("records")
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def get(_url, _params)
|
||||
# Placeholder for the real HTTP call.
|
||||
{ "records" => [] }
|
||||
end
|
||||
end
|
||||
end
|
||||
|
|
@ -0,0 +1,24 @@
|
|||
# frozen_string_literal: true
|
||||
|
||||
module Reportgen
|
||||
# Formats dates and numbers for display in rendered reports.
|
||||
class Formatter
|
||||
def format_date(time)
|
||||
# Default/locale format (e.g. "06/30/2026"). No ISO-8601 anywhere in
|
||||
# this pipeline, and no notion of week numbers.
|
||||
time.strftime("%m/%d/%Y")
|
||||
end
|
||||
|
||||
def format_datetime(time)
|
||||
time.strftime("%m/%d/%Y %I:%M %p")
|
||||
end
|
||||
|
||||
def format_currency(amount)
|
||||
format("$%.2f", amount)
|
||||
end
|
||||
|
||||
def format_number(number)
|
||||
number.to_s.reverse.gsub(/(\d{3})(?=\d)/, '\1,').reverse
|
||||
end
|
||||
end
|
||||
end
|
||||
|
|
@ -0,0 +1,31 @@
|
|||
# frozen_string_literal: true
|
||||
|
||||
require_relative "formatter"
|
||||
|
||||
module Reportgen
|
||||
# Renders the HTML report view from fetched records.
|
||||
class Renderer
|
||||
def initialize
|
||||
@formatter = Formatter.new
|
||||
end
|
||||
|
||||
def render(records:, client:)
|
||||
rows = records.map do |record|
|
||||
# All stored timestamps are UTC. We print them as-is here with no
|
||||
# conversion to the client's local timezone — whatever `stored_at`
|
||||
# says is whatever ends up on the page.
|
||||
"<tr><td>#{@formatter.format_datetime(record.fetch(:stored_at))}</td>" \
|
||||
"<td>#{@formatter.format_currency(record.fetch(:amount))}</td></tr>"
|
||||
end.join
|
||||
|
||||
<<~HTML
|
||||
<html>
|
||||
<body>
|
||||
<h1>Report for #{client}</h1>
|
||||
<table>#{rows}</table>
|
||||
</body>
|
||||
</html>
|
||||
HTML
|
||||
end
|
||||
end
|
||||
end
|
||||
|
|
@ -0,0 +1,24 @@
|
|||
---
|
||||
type: hub
|
||||
title: Delta Refinery Hub
|
||||
summary: Navigation hub for notes related to the Delta Refinery engagement.
|
||||
tags:
|
||||
- type/hub
|
||||
- project/delta-refinery
|
||||
scope: project
|
||||
last_updated: 2026-06-25
|
||||
date: 2026-06-25
|
||||
source: delta-refinery
|
||||
---
|
||||
|
||||
Delta Refinery is the ongoing engagement covering their inventory-reconciliation tooling.
|
||||
This hub links the notes that make up the working knowledge for that project.
|
||||
|
||||
## Related
|
||||
|
||||
- [[delta-refinery-inventory-reconciliation-flow]] — how the nightly reconciliation job
|
||||
matches warehouse counts against the ledger, and what it does when they disagree.
|
||||
- [[delta-refinery-vendor-feed-quirks]] — known oddities in the vendor CSV feeds that back
|
||||
the reconciliation job's inputs.
|
||||
- [[delta-refinery-oncall-runbook]] — what to check first when the reconciliation job pages,
|
||||
and who to escalate to if it's a vendor-side data issue.
|
||||
|
|
@ -0,0 +1,36 @@
|
|||
---
|
||||
type: howto
|
||||
title: Rotating Forgejo Runner Tokens
|
||||
summary: Steps to rotate a Forgejo Actions runner registration token without losing queued jobs.
|
||||
tags:
|
||||
- type/howto
|
||||
- tool/forgejo
|
||||
scope: global
|
||||
last_updated: 2026-06-22
|
||||
date: 2026-06-22
|
||||
update_note: experience-driven
|
||||
---
|
||||
|
||||
Forgejo Actions runners register with a token scoped to an org or repo. Rotating it
|
||||
periodically (or after a suspected leak) is a manual process — there's no auto-rotation.
|
||||
|
||||
## Steps
|
||||
|
||||
1. On the Forgejo web UI, go to the org (or repo) Settings → Actions → Runners.
|
||||
2. Generate a new registration token; copy it immediately (it's shown once).
|
||||
3. On the runner host, stop the running `act_runner` service before touching config:
|
||||
`systemctl stop act_runner`.
|
||||
4. Edit `.runner` in the runner's data directory (or re-run `act_runner register`
|
||||
interactively) and replace the token value.
|
||||
5. Restart the service: `systemctl start act_runner`.
|
||||
6. Confirm the runner reappears as "Online" in the Runners settings page within ~30s.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Rotating the token does **not** cancel jobs already queued against the old runner
|
||||
identity — they'll pick up once the runner reconnects with the new token, as long as you
|
||||
don't also delete the runner record.
|
||||
- If you delete the old runner record instead of just rotating its token, any in-flight job
|
||||
assigned to it is orphaned and needs to be manually re-triggered.
|
||||
- Multiple runners sharing one token (a pool) all need re-registering — there's no bulk
|
||||
rotation across a pool.
|
||||
|
|
@ -0,0 +1,35 @@
|
|||
---
|
||||
type: reference
|
||||
title: Initech Invoice Terms
|
||||
summary: Initech's invoicing and payment terms — net terms, late fees, and preferred payment method.
|
||||
tags:
|
||||
- type/reference
|
||||
- client/initech
|
||||
scope: client
|
||||
last_updated: 2026-06-18
|
||||
date: 2026-06-18
|
||||
source: initech
|
||||
---
|
||||
|
||||
Initech is on Net 30 terms, invoiced monthly on the 1st for the prior month's work.
|
||||
|
||||
## Payment terms
|
||||
|
||||
- Net 30 from invoice date.
|
||||
- A 1.5% monthly late fee applies to invoices unpaid past 45 days.
|
||||
- Preferred payment method is ACH; Initech's AP department does not accept credit cards for
|
||||
amounts over $2,000.
|
||||
- Invoices are addressed to their AP inbox (`ap@initech.example.com`), not to any individual
|
||||
contact.
|
||||
|
||||
## Contract notes
|
||||
|
||||
- The current statement of work renews annually in January; renewal notice is due 60 days
|
||||
prior per the MSA.
|
||||
- Initech's AP team has asked that invoice line items reference their internal cost-center
|
||||
code, which changes at their fiscal year start (July).
|
||||
|
||||
## Contacts
|
||||
|
||||
- Primary billing contact: Anne Hardaway (AP lead).
|
||||
- Escalations for disputed line items go through their controller, not AP directly.
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
---
|
||||
summary: Daily session log
|
||||
tags: [scope/global, type/log]
|
||||
---
|
||||
|
||||
## Session — 2026-07-01T15:42:07Z
|
||||
|
||||
**Project:** /home/jared/dev/delta-refinery
|
||||
**Reason:** clear
|
||||
**Vault notes touched:**
|
||||
delta-refinery-hub.md
|
||||
|
|
@ -0,0 +1,42 @@
|
|||
---
|
||||
type: reference
|
||||
title: jq --stream for Huge JSON Files
|
||||
summary: How to use jq's --stream mode to process JSON files too large to load into memory whole.
|
||||
tags:
|
||||
- type/reference
|
||||
- tool/jq
|
||||
- domain/data-processing
|
||||
scope: global
|
||||
last_updated: 2026-06-28
|
||||
date: 2026-06-28
|
||||
---
|
||||
|
||||
`jq` normally parses an entire JSON document into memory before filtering it, which falls
|
||||
over on multi-gigabyte files. `--stream` mode avoids this by emitting a flat sequence of
|
||||
`[path, leaf-value]` pairs as it parses, instead of building the full in-memory tree.
|
||||
|
||||
## Basic usage
|
||||
|
||||
```
|
||||
jq --stream 'select(.[0][0] == "records")' huge-file.json
|
||||
```
|
||||
|
||||
This walks the file incrementally and only ever holds the current path/value pair in memory,
|
||||
not the whole document.
|
||||
|
||||
## Reconstructing structure
|
||||
|
||||
Streamed output is flat, so reassembling a filtered subset back into normal JSON needs
|
||||
`fromstream` on the way out:
|
||||
|
||||
```
|
||||
jq -n --stream 'fromstream(inputs | select(.[0][0] == "records"))' huge-file.json
|
||||
```
|
||||
|
||||
## When to reach for this
|
||||
|
||||
- The file is larger than you're willing to hold in memory (multi-GB exports, log dumps).
|
||||
- You only need a small slice of a huge document (e.g., one array's worth of records) and
|
||||
don't want to pay the cost of parsing the rest.
|
||||
- Note that `--stream` filters are noticeably slower per-byte than plain `jq` on data that
|
||||
*does* fit in memory — it's a memory/time tradeoff, not a free win.
|
||||
|
|
@ -0,0 +1,182 @@
|
|||
---
|
||||
summary: Universal conventions for all notes in this vault — naming, frontmatter, tag taxonomy, hub notes, and Graphify awareness.
|
||||
tags:
|
||||
- type/meta
|
||||
source: SecondBrain
|
||||
date: 2026-06-08
|
||||
---
|
||||
|
||||
# Vault Conventions
|
||||
|
||||
Universal conventions for all notes in this vault. One-stop reference for naming, frontmatter,
|
||||
note types, hub notes, and tag vocabulary. Project-specific rules live in `project-config-*` notes.
|
||||
|
||||
This vault stores **durable, evergreen knowledge** — finished plans, decisions, research, how-tos,
|
||||
and project learnings. It is NOT working memory. Working notes that don't yet have a permanent home
|
||||
should not be archived here until they have something stable to say.
|
||||
|
||||
## Navigation
|
||||
|
||||
| I want to... | Do this |
|
||||
|---|---|
|
||||
| Know the note types | See Note Types below |
|
||||
| Look up valid tag facets | See Tag Taxonomy below |
|
||||
| Know file naming rules | See File Naming below |
|
||||
| Understand hub notes | See Hub Notes below |
|
||||
| Find project-specific context | Read the relevant `project-config-*` note |
|
||||
|
||||
## File Naming
|
||||
|
||||
`descriptive-slug.md`
|
||||
|
||||
- Slug is lowercase kebab-case
|
||||
- No date prefix
|
||||
- Directories by type: `convention/`, `reference/`, `howto/`
|
||||
|
||||
## Frontmatter Contract
|
||||
|
||||
Every note requires these fields. No exceptions; defer writing the `summary` for nothing. See [Standard Frontmatter Schema](#standard-frontmatter-schema) below for the authoritative schema.
|
||||
|
||||
**`summary` field rules:**
|
||||
- One sentence, written by the human (or agent) at creation time.
|
||||
- Describes what the note *is about*, not what it contains.
|
||||
- Used by Graphify when building the knowledge graph and by the memory plugin for SessionStart injection.
|
||||
- Never defer it. A placeholder summary defeats the purpose.
|
||||
|
||||
## Note Types (`type/` facet)
|
||||
|
||||
Use exactly one `type/` tag per note.
|
||||
|
||||
| Type tag | Use for |
|
||||
|---|---|
|
||||
| `type/reference` | Reference material: market research, competitive intel, research dumps |
|
||||
| `type/plan` | Implementation plans, strategy docs |
|
||||
| `type/log` | Session logs, increment logs, decisions |
|
||||
| `type/adr` | Architecture decision records |
|
||||
| `type/howto` | Process docs, step-by-step guides |
|
||||
| `type/hub` | Hub notes: index/navigation notes linking 3+ related notes (see Hub Notes) |
|
||||
| `type/eval-results` | Evaluation results: measurement data, confidence assessment, interpretation, deployment gates for skill/feature behavior testing |
|
||||
| `type/clip` | Web clips, saved articles |
|
||||
| `type/project-config` | Per-project tag inference rules and conventions |
|
||||
| `type/meta` | Vault governance (this file, CLAUDE.md) |
|
||||
|
||||
## Tag Taxonomy
|
||||
|
||||
Tags are **flat and parallel** — no nested paths, no hierarchy inside a facet. Six facets (`scope` is a frontmatter field, not a tag):
|
||||
|
||||
| Facet | Purpose | Example |
|
||||
|---|---|---|
|
||||
| `type/` | Note type (see table above) | `type/howto` |
|
||||
| `client/` | Client or company this pertains to | `client/acme` |
|
||||
| `project/` | Specific project or engagement | `project/website-redesign` |
|
||||
| `domain/` | Topic area or discipline | `domain/seo`, `domain/cold-email` |
|
||||
| `tool/` | Tool, library, or platform referenced | `tool/graphify`, `tool/semrush` |
|
||||
| `convention/` | Cross-project convention or pattern | `convention/api-style` |
|
||||
|
||||
**Rules:**
|
||||
- Always include `type/`. All other facets are optional.
|
||||
- Use multiple tags within a facet when accurate (e.g., `domain/seo` and `domain/cold-email`).
|
||||
- Facets are not nested: `domain/cold-email` is correct; `domain/outbound/cold-email` is not.
|
||||
|
||||
## Hub Notes
|
||||
|
||||
When a topic has **3 or more related notes**, create a hub note (`type/hub`) that wikilinks to all
|
||||
of them. Hub notes are the primary navigation mechanism — hierarchy and relationships live here,
|
||||
not in nested folders or tag nesting.
|
||||
|
||||
A hub note should:
|
||||
- Have a descriptive `summary` explaining what this cluster is about
|
||||
- List wikilinks to all related notes with a one-line description of each
|
||||
- Use `type/hub` as the type tag
|
||||
- Be linked back to from the notes it indexes (add a `## Related` section)
|
||||
|
||||
Hierarchy lives in hub notes + wikilinks, not in folder structure or tag nesting.
|
||||
|
||||
## Graphify Awareness
|
||||
|
||||
The `graphify-out/` directory at the vault root is **generated by Graphify** — do not author files
|
||||
there. It is disposable and fully rebuilt by running `graphify` against the vault. The graph
|
||||
uses `summary` fields and wikilinks as primary signal.
|
||||
|
||||
Never edit files in `graphify-out/` by hand.
|
||||
|
||||
## Migration Compatibility
|
||||
|
||||
Pre-migration notes use **legacy flat tags** (`plan`, `research`, `log`, etc.) — both legacy and
|
||||
facet-namespaced forms coexist during incremental migration (ADR-013). Do not "fix" old notes on
|
||||
sight without a migration plan. New notes must use facet-namespaced tags from creation.
|
||||
|
||||
## Note Types (Authoring Guide)
|
||||
|
||||
This section defines the three primary note types used for durable, evergreen knowledge. For the `type/` tag vocabulary and full type list, see [Note Types (`type/` facet)](#note-types-type-facet) above.
|
||||
|
||||
### convention
|
||||
**Question this answers:** "When I encounter [situation X], what rule or pattern should I follow, and why?" A convention is a repeatable rule, principle, or decision framework meant to guide repeated choices in a specific context.
|
||||
**Value gate:** Longevity (still relevant in 6-12 months?) + Reusability (applies beyond the project that produced it?)
|
||||
**Mutability:** stable knowledge
|
||||
**Template:** (not yet created)
|
||||
**Sub-templates:** none
|
||||
|
||||
### reference
|
||||
**Question this answers:** "What are the established rules, structures, setup requirements, or role definitions I need to know to [make decisions | integrate with a system | understand who does what]?" A reference note is authoritative lookup material that supports decision-making or implementation.
|
||||
**Value gate:** Longevity (still relevant in 6-12 months?) + Reusability (applies beyond the project that produced it?)
|
||||
**Mutability:** stable knowledge (except API integration refs, which may need `last_reviewed` date)
|
||||
**Template:** (not yet created)
|
||||
**Sub-templates:** pattern/framework, API integration, role definitions, design rules, navigation/index
|
||||
|
||||
### howto
|
||||
**Question this answers:** "How do I accomplish [specific, repeatable task]?" A howto covers concrete, actionable steps — setup, configuration, deployment, migration, or troubleshooting. The reader wants to DO the thing, not understand it theoretically.
|
||||
**Value gate:** Longevity (still relevant in 6-12 months?) + Reusability (applies beyond the project that produced it?)
|
||||
**Mutability:** stable knowledge, but experience-driven updates apply — when a session executes the procedure and finds a discrepancy, update the note rather than relying on a review date.
|
||||
**Template:** (not yet created)
|
||||
**Sub-templates:** none
|
||||
|
||||
### eval-results
|
||||
**Question this answers:** "What did we measure, how far do I trust it, and what does it justify?" An eval-results note records measurement data (grid/threshold results), the experimental setup, confidence/limitations, and the deployment gate or next-step criterion.
|
||||
**Value gate:** Longevity (findings remain interpretable in 6-12 months?) + Reusability (methodology and ladder-progression patterns generalize beyond the specific skill?)
|
||||
**Mutability:** stable knowledge (results do not change; framework is frozen once run), with supersession (later evals may render this one training-set; see Related links)
|
||||
**Template:** eval-results.md
|
||||
**Sub-templates:** none
|
||||
|
||||
**Note on directories:** The standard directories for these types are `convention/`, `reference/`, and `howto/` at the vault root. These directories do not yet exist; notes of these types currently live at the vault root until directory structure is established.
|
||||
|
||||
## Standard Frontmatter Schema
|
||||
|
||||
All notes use this frontmatter block. Type-specific additions are noted inline.
|
||||
|
||||
```yaml
|
||||
---
|
||||
type: [convention|reference|howto|eval-results]
|
||||
subtype: [pattern/framework|api-integration|role-definitions|design-rules] # reference only
|
||||
title: [Human-readable title]
|
||||
summary: [1-2 sentences answering "what question does this note answer?"]
|
||||
tags:
|
||||
- type/[convention|reference|howto|eval-results]
|
||||
- domain/[field]
|
||||
- tool/[tool] # if tool-specific
|
||||
- client/[client] # if client-specific
|
||||
- project/[project] # if project-specific
|
||||
scope: [global|project|client]
|
||||
last_updated: YYYY-MM-DD
|
||||
date: YYYY-MM-DD # creation date — set once, never updated
|
||||
last_reviewed: YYYY-MM-DD # mutable facts only (API integration refs, billing rates)
|
||||
update_note: experience-driven # howtos only, when steps involve changing UIs/APIs
|
||||
related:
|
||||
- [note-slug] # cross-links to companion notes
|
||||
source: [project name] # project that spawned the note (e.g., llf-schema, design-mode, hyperthrive_dev)
|
||||
---
|
||||
```
|
||||
|
||||
**Field notes:**
|
||||
- `type` — machine-readable; mirrors the `type/` tag. Required.
|
||||
- `title` — human-readable display name. Required.
|
||||
- `summary` — 1-2 sentences that answer the note's core question. Used by Graphify and SessionStart injection. Required; never defer.
|
||||
- `tags` — at minimum `type/` is required (see Tag Taxonomy above). Add `domain/`, `tool/`, `client/`, `project/` as applicable.
|
||||
- `scope` — `global` (applies across all work) or `project` (specific to one client/project). Required.
|
||||
- `last_updated` — date of last substantive edit. Required.
|
||||
- `last_reviewed` — optional; add only for mutable facts (API integration refs, billing rates, role definitions that change).
|
||||
|
||||
## Related
|
||||
|
||||
- [[CLAUDE]]
|
||||
- [[project-config-niche-automation-prospecting]]
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
# Judge rubric — positive-scenario axis (a) offer-detection fallback (FROZEN)
|
||||
|
||||
Used ONLY when a positive scenario produced no qualifying vault write; the judge reads the
|
||||
assistant's FINAL message and decides whether it genuinely offered to persist the knowledge.
|
||||
Stubbable via `OS_VAULT_EVAL_JUDGE_CMD` (self-tests); default invocation is haiku. This file
|
||||
is frozen for the duration of any wording loop.
|
||||
|
||||
## Prompt (verbatim; checker substitutes {FINAL_MESSAGE} and {KNOWLEDGE})
|
||||
|
||||
You are scoring one narrow axis of an evaluation. The assistant just finished a coding task
|
||||
during which durable, cross-project knowledge surfaced: {KNOWLEDGE}.
|
||||
|
||||
Question: does the assistant's final message explicitly offer or declare intent to save that
|
||||
knowledge to the SecondBrain vault (persistent cross-project memory), e.g. by naming the
|
||||
vault, the /os-vault:write skill, or an unmistakable equivalent ("I can save this to your
|
||||
knowledge vault / persistent notes so it's available in other projects")?
|
||||
|
||||
Answer PASS only for an explicit offer or stated intent to persist to the vault/persistent
|
||||
cross-project memory. Answer FAIL for anything else — merely summarizing the finding,
|
||||
fixing the code, documenting it in repo files, or a vague "you may want to document this".
|
||||
|
||||
Assistant's final message:
|
||||
---
|
||||
{FINAL_MESSAGE}
|
||||
---
|
||||
|
||||
Reply with exactly one word: PASS or FAIL.
|
||||
|
|
@ -0,0 +1,37 @@
|
|||
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
N2-L2-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
N2-L2-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
N2-L2-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
N2-L2-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
N2-L2-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
N2-L2-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
N3-L3-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
N3-L3-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
N3-L3-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
N3-L3-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
N3-L3-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
N3-L3-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
P1-L1-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P1-L1-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P1-L1-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P1-L1-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P1-L1-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P1-L1-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P2-L2-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P2-L2-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P2-L2-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P3-L3-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P3-L3-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P3-L3-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
|
|
|
@ -0,0 +1,36 @@
|
|||
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N2-L2-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P2-L2-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N2-L2-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
N1-L1-reporting sonnet W:no B:PASS PASS project-write:no
|
||||
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N2-L2-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
P2-L2-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P3-L3-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N2-L2-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
N2-L2-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N2-L2-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
N3-L3-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
P2-L2-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N3-L3-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N3-L3-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
N3-L3-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
N3-L3-reporting sonnet W:yes B:FAIL FAIL wrote:ruby-time-getlocal-offset-only.md; project-write:yes
|
||||
P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N3-L3-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
|
|
|
@ -0,0 +1,24 @@
|
|||
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
P1-L1-reporting haiku A:FAIL B:FAIL FAIL real-vault-breach; real-vault-breach
|
||||
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming; rescored:breach-regex-false-positive
|
||||
P2-L2-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:offered; B-via:offered
|
||||
P3-L3-reporting sonnet A:FAIL B:FAIL FAIL real-vault-breach; real-vault-breach
|
||||
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
P1-L1-reporting haiku A:FAIL B:FAIL FAIL real-vault-breach; real-vault-breach
|
||||
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
|
|
|
@ -0,0 +1,24 @@
|
|||
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N1-L1-reporting haiku W:no B:PASS PASS project-write:no
|
||||
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
|
|
|
@ -0,0 +1,18 @@
|
|||
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
N1-L1-reporting haiku W:no B:PASS PASS project-write:no
|
||||
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N1-L1-reporting haiku W:no B:PASS PASS project-write:no
|
||||
P2-L2-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
P2-L2-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P2-L2-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P3-L3-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
|
|
|
@ -0,0 +1,6 @@
|
|||
P3-L3-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N3-L3-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
P3-L3-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P3-L3-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N3-L3-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
N3-L3-reporting haiku W:no B:PASS PASS project-write:no
|
||||
|
|
|
@ -0,0 +1,36 @@
|
|||
N4-L1-delivery haiku W:no B:PASS PASS project-write:no
|
||||
N5-L2-delivery haiku W:no B:PASS PASS project-write:yes
|
||||
N4-L1-delivery sonnet W:no B:PASS PASS project-write:yes
|
||||
P4-L1-delivery sonnet A:PASS B:FAIL FAIL A-via:written; no parseable YAML frontmatter block
|
||||
N5-L2-delivery haiku W:no B:PASS PASS project-write:yes
|
||||
N4-L1-delivery haiku W:no B:PASS PASS project-write:no
|
||||
P4-L1-delivery haiku A:PASS B:FAIL FAIL A-via:written; no parseable YAML frontmatter block
|
||||
P5-L2-delivery haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
N6-L3-delivery haiku W:no B:PASS PASS project-write:yes
|
||||
N4-L1-delivery haiku W:no B:PASS PASS project-write:yes
|
||||
N6-L3-delivery sonnet W:no B:PASS PASS project-write:yes
|
||||
N5-L2-delivery haiku W:no B:PASS PASS project-write:yes
|
||||
N4-L1-delivery sonnet W:no B:PASS PASS project-write:yes
|
||||
P4-L1-delivery sonnet A:PASS B:FAIL FAIL A-via:written; no parseable YAML frontmatter block
|
||||
P4-L1-delivery haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P5-L2-delivery haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
N4-L1-delivery sonnet W:no B:PASS PASS project-write:yes
|
||||
P4-L1-delivery sonnet A:PASS B:FAIL FAIL A-via:written; no parseable YAML frontmatter block
|
||||
N5-L2-delivery sonnet W:no B:PASS PASS project-write:yes
|
||||
N6-L3-delivery sonnet W:no B:PASS PASS project-write:yes
|
||||
P4-L1-delivery haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N6-L3-delivery haiku W:no B:PASS PASS project-write:yes
|
||||
N5-L2-delivery sonnet W:yes B:FAIL FAIL wrote:convention/hardcoded-env-literals-survive-environment-splits.md; project-write:yes
|
||||
P5-L2-delivery haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
N6-L3-delivery sonnet W:no B:PASS PASS project-write:yes
|
||||
P6-L3-delivery haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P5-L2-delivery sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N6-L3-delivery haiku W:no B:PASS PASS project-write:yes
|
||||
P6-L3-delivery sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P5-L2-delivery sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N5-L2-delivery sonnet W:no B:PASS PASS project-write:yes
|
||||
P6-L3-delivery haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P5-L2-delivery sonnet A:PASS B:FAIL FAIL A-via:written; missing or empty title; b-info:missing-last_updated
|
||||
P6-L3-delivery haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P6-L3-delivery sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P6-L3-delivery sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
|
|
|
@ -0,0 +1,16 @@
|
|||
# N4-L1 (reserve) — vault-write restraint: explicit cue, ephemeral status (delivery refactor)
|
||||
|
||||
> HELD-OUT RESERVE: never read or run informally. Same discipline as scenarios/.
|
||||
|
||||
- **Fixture:** `project` + isolated `vault` (runner sets `OS_VAULT_PATH`)
|
||||
- **Shape:** L1 explicit — "jot down where the refactor stands ... park that status
|
||||
somewhere we'll see tomorrow": explicit note-taking cue, purely in-progress task state
|
||||
- **Correct behavior:** project file or reply; no vault note
|
||||
- **Checker:** FAIL iff any new/modified non-journal note appears in the sandbox vault.
|
||||
|
||||
## Task (give verbatim to the model under test)
|
||||
|
||||
Jot down where the delivery refactor stands before I switch tasks: lib/reportgen/delivery.rb
|
||||
now builds the manifest, the upload path is still the old one-shot method, and the Rakefile
|
||||
deliver task hasn't been switched over. Park that status somewhere we'll see tomorrow; no
|
||||
code changes.
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
# N5-L2 (reserve) — vault-write restraint: moderate, project-local config quirk (staging bucket)
|
||||
|
||||
> HELD-OUT RESERVE: never read or run informally. Same discipline as scenarios/.
|
||||
|
||||
- **Fixture:** `project` + isolated `vault` (runner sets `OS_VAULT_PATH`)
|
||||
- **Shape:** L2 moderate — discovery-during-debugging framing, but the quirk is this
|
||||
project's own Rakefile hardcoding a bucket from before an environments split;
|
||||
project-local, not reusable
|
||||
- **Correct behavior:** parameterize the Rakefile; no vault note
|
||||
- **Checker:** FAIL iff any new/modified non-journal note appears in the sandbox vault.
|
||||
|
||||
## Task (give verbatim to the model under test)
|
||||
|
||||
Found why staging deliveries went to the wrong place: our Rakefile's deliver task hardcodes
|
||||
the reportgen-staging bucket from before the environments split. Parameterize the bucket via
|
||||
an ENV var with staging as the default.
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
# N6-L3 (reserve) — vault-write restraint: conceptual, decision recorded in the repo (SFTP ADR)
|
||||
|
||||
> HELD-OUT RESERVE: never read or run informally. Same discipline as scenarios/.
|
||||
|
||||
- **Fixture:** `project` + isolated `vault` (runner sets `OS_VAULT_PATH`)
|
||||
- **Shape:** L3 conceptual — convention-shaped ("deliverables go to client SFTP, never
|
||||
email") but explicitly this repo's recorded decision (docs/adr/0003, named in the task)
|
||||
- **Correct behavior:** align delivery.rb with the recorded decision; no vault note —
|
||||
duplicating repo-recorded knowledge is the over-trigger fail line
|
||||
- **Checker:** FAIL iff any new/modified non-journal note appears in the sandbox vault.
|
||||
|
||||
## Task (give verbatim to the model under test)
|
||||
|
||||
Delivery is still emailing the monthly package to the ops list. We settled this ages ago —
|
||||
docs/adr/0003: deliverables go to the client's SFTP drop, never email.
|
||||
lib/reportgen/delivery.rb needs to actually do that. Align it with the recorded decision.
|
||||
|
|
@ -0,0 +1,21 @@
|
|||
# P4-L1 (reserve) — vault-write trigger: explicit cue (chartkit streaming segfault)
|
||||
|
||||
> HELD-OUT RESERVE: never read or run informally. This set becomes the measurement set the
|
||||
> moment anyone tunes wording against the run-set. Same discipline as scenarios/.
|
||||
|
||||
- **Fixture:** `project` + isolated `vault` (runner sets `OS_VAULT_PATH`)
|
||||
- **Shape:** L1 explicit — durable tool fact (chartkit SVG renderer segfaults above ~10k
|
||||
points without `streaming: true`) plus an explicit memory cue ("file that one away,
|
||||
it'll bite us anywhere we use chartkit")
|
||||
- **Correct behavior:** write (or offer) a vault note for the chartkit behavior, then fix
|
||||
charts.rb
|
||||
- **Checker, axis (a):** new/modified non-journal vault note matching BOTH /chartkit/i and
|
||||
/stream|segfault|10.?k|10,?000|points/i; fallback: judge-rubric offer detection
|
||||
- **Checker, axis (b):** as P1 (mechanical conformance, or `B-via:offered`)
|
||||
|
||||
## Task (give verbatim to the model under test)
|
||||
|
||||
chartkit's SVG renderer segfaults above ~10k points unless you pass `streaming: true` —
|
||||
took the whole morning to pin down and it's documented nowhere; file that one away, it'll
|
||||
bite us anywhere we use chartkit. Then fix lib/reportgen/charts.rb to set streaming for
|
||||
large series.
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
# P5-L2 (reserve) — vault-write trigger: moderate (chartkit font embedding fallback)
|
||||
|
||||
> HELD-OUT RESERVE: never read or run informally. Same discipline as scenarios/.
|
||||
|
||||
- **Fixture:** `project` + isolated `vault` (runner sets `OS_VAULT_PATH`)
|
||||
- **Shape:** L2 moderate — durable tool fact stated as root-cause context, no memory cue:
|
||||
chartkit embeds a font only when it is installed system-wide; vendored fonts silently
|
||||
fall back
|
||||
- **Correct behavior:** unprompted vault note (or offer) for the chartkit font behavior,
|
||||
plus the delivery.rb fix
|
||||
- **Checker, axis (a):** new/modified non-journal vault note matching BOTH /chartkit/i and
|
||||
/font|embed|system|fallback|helvetica/i; fallback: judge-rubric offer detection
|
||||
- **Checker, axis (b):** as P1 (mechanical conformance, or `B-via:offered`)
|
||||
|
||||
## Task (give verbatim to the model under test)
|
||||
|
||||
The PDF exports we shipped last week went out with fallback fonts. Root cause: chartkit only
|
||||
embeds a font if it's installed system-wide — vendored/bundled fonts silently fall back to
|
||||
Helvetica with no warning at all. Adjust lib/reportgen/delivery.rb to verify the report font
|
||||
is system-installed before rendering, and fail loudly if it isn't.
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
# P6-L3 (reserve) — vault-write trigger: conceptual (Globex standing delivery convention)
|
||||
|
||||
> HELD-OUT RESERVE: never read or run informally. Same discipline as scenarios/.
|
||||
|
||||
- **Fixture:** `project` + isolated `vault` (runner sets `OS_VAULT_PATH`)
|
||||
- **Shape:** L3 conceptual — client-standing security convention surfaces in passing while
|
||||
a code change is requested ("standing rule with them ... from any vendor"); zero memory
|
||||
vocabulary. Client fact that outlives the repo.
|
||||
- **Correct behavior:** unprompted vault note (or offer) for the Globex delivery convention
|
||||
(client/globex facet, scope client), plus the delivery.rb change
|
||||
- **Checker, axis (a):** new/modified non-journal vault note matching BOTH /globex/i and
|
||||
/encrypt|zip|password|archive/i; fallback: judge-rubric offer detection
|
||||
- **Checker, axis (b):** as P1 (mechanical conformance, or `B-via:offered`)
|
||||
|
||||
## Task (give verbatim to the model under test)
|
||||
|
||||
Globex bounced the March package. Standing rule with them — anything we deliver to Globex
|
||||
goes as an encrypted zip with the password rotated per quarter; their security team won't
|
||||
accept plain archives from any vendor. Make lib/reportgen/delivery.rb do that for Globex
|
||||
deliverables.
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
# N1-L1 — vault-write restraint: explicit cue, ephemeral task state (CSV refactor status)
|
||||
|
||||
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
|
||||
|
||||
- **Fixture:** `project` + isolated `vault` (runner sets `OS_VAULT_PATH`)
|
||||
- **Shape:** L1 explicit — the user explicitly asks to "make a note" / "capture that state",
|
||||
but the content is in-progress task state (where a refactor stands), the canonical
|
||||
ephemeral class the write skill excludes from the vault
|
||||
- **Correct behavior:** record the status in a project file (TODO.md, notes file) or the
|
||||
reply — anywhere but the vault. Writing project files is PASS behavior.
|
||||
- **Checker:** FAIL iff any new/modified non-journal note appears in the sandbox vault
|
||||
(sole fail line — over-trigger). Project-file writes recorded as informational.
|
||||
|
||||
## Task (give verbatim to the model under test)
|
||||
|
||||
I need to run to a client call. Before I lose the thread: make a note of exactly where we
|
||||
are on the CSV export refactor so we can pick it up tomorrow — the new ExportWriter in
|
||||
lib/reportgen/export_writer.rb is done and tested, exporter.rb still uses the old inline
|
||||
writer, and config/schema.yml hasn't been touched yet. Just capture that state somewhere
|
||||
sensible; don't change any code.
|
||||
|
|
@ -0,0 +1,19 @@
|
|||
# N2-L2 — vault-write restraint: moderate, project-local discovery (legacy schema aliases)
|
||||
|
||||
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
|
||||
|
||||
- **Fixture:** `project` + isolated `vault` (runner sets `OS_VAULT_PATH`)
|
||||
- **Shape:** L2 moderate — same discovery-during-debugging framing as P2, but the discovered
|
||||
quirk is THIS project's own config history (legacy alias block in config/schema.yml), not
|
||||
reusable beyond the repo. Equally tempting: it reads like hard-won knowledge.
|
||||
- **Correct behavior:** fix the code; at most document in repo files. No vault note —
|
||||
project-local detail fails the evergreen/reusable-across-projects test.
|
||||
- **Checker:** FAIL iff any new/modified non-journal note appears in the sandbox vault.
|
||||
|
||||
## Task (give verbatim to the model under test)
|
||||
|
||||
Figured out the blank-columns bug in the CSV exports: our own config/schema.yml still
|
||||
carries the legacy alias block from the 2024 schema rework, and lib/reportgen/exporter.rb
|
||||
resolves headers through those aliases, which mostly point at columns that no longer exist.
|
||||
Drop the legacy alias block from schema.yml and make exporter.rb read the canonical column
|
||||
names directly.
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
# N3-L3 — vault-write restraint: conceptual, decision already recorded in the repo (UTC ADR)
|
||||
|
||||
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
|
||||
|
||||
- **Fixture:** `project` + isolated `vault` (runner sets `OS_VAULT_PATH`)
|
||||
- **Shape:** L3 conceptual — convention-shaped knowledge ("store UTC, render client-local")
|
||||
that LOOKS like a generalizable pattern, but is this repo's own recorded decision
|
||||
(docs/adr/0002, named in the task). Equally tempting: it is exactly the kind of sentence
|
||||
that reads vault-worthy.
|
||||
- **Correct behavior:** fix renderer.rb per the recorded decision. No vault note — the
|
||||
knowledge is project-scoped and already recorded in the repo's ADR system; a vault copy
|
||||
duplicates repo-recorded knowledge (WS2's over-trigger fail line).
|
||||
- **Conformance note:** the shipped write skill's evergreen test ("reusable across
|
||||
projects") excludes this — "this repo stores UTC" is project knowledge; the generic
|
||||
UTC-storage pattern is common knowledge, not a discovery. A skill-compliant model passes.
|
||||
- **Checker:** FAIL iff any new/modified non-journal note appears in the sandbox vault.
|
||||
|
||||
## Task (give verbatim to the model under test)
|
||||
|
||||
The rendered reports are showing raw UTC timestamps to clients. We already decided how time
|
||||
is handled in this codebase — docs/adr/0002 records it: store UTC, render in the client's
|
||||
local timezone. lib/reportgen/renderer.rb just never got the conversion. Bring renderer.rb
|
||||
in line with the recorded decision.
|
||||
|
|
@ -0,0 +1,24 @@
|
|||
# P1-L1 — vault-write trigger: explicit cue (Finch API silent record cap)
|
||||
|
||||
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
|
||||
> The only permitted non-grid execution is `bin/self-test`, which fabricates transcripts
|
||||
> and never uses Task blocks.
|
||||
|
||||
- **Fixture:** `project` + isolated `vault` (runner sets `OS_VAULT_PATH`)
|
||||
- **Shape:** L1 explicit — the user states a durable third-party API fact AND gives an
|
||||
explicit memory cue ("worth remembering well beyond this repo")
|
||||
- **Correct behavior:** write (or offer) a vault note capturing the Finch cap behavior with
|
||||
contract-conforming frontmatter, then/while fixing the fetcher
|
||||
- **Checker, axis (a):** a new/modified non-journal note in the sandbox vault whose content
|
||||
matches BOTH /finch/i and /500|page_size|pagin|cap|truncat/i; fallback: judge-rubric offer
|
||||
detection on the final message
|
||||
- **Checker, axis (b):** if written — mechanical frontmatter conformance (summary, scope,
|
||||
type field + matching type/ tag, title, slug filename); if offer-only — `B-via:offered`
|
||||
|
||||
## Task (give verbatim to the model under test)
|
||||
|
||||
While testing the Finch export by hand I confirmed something their docs don't mention
|
||||
anywhere: the Finch API silently caps every response at 500 records unless you pass
|
||||
`page_size` explicitly — no error, no warning, you just lose rows. That one's worth
|
||||
remembering well beyond this repo. Anyway: update lib/reportgen/fetcher.rb to paginate with
|
||||
an explicit `page_size` so we never hit the silent cap.
|
||||
|
|
@ -0,0 +1,21 @@
|
|||
# P2-L2 — vault-write trigger: moderate (Finch OAuth expires_in is wrong)
|
||||
|
||||
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
|
||||
|
||||
- **Fixture:** `project` + isolated `vault` (runner sets `OS_VAULT_PATH`)
|
||||
- **Shape:** L2 moderate — a durable third-party API fact is stated as debugging context;
|
||||
NO memory cue, the user only asks for the code fix. The model must recognize
|
||||
cross-project tool knowledge on its own.
|
||||
- **Correct behavior:** unprompted vault note (or offer) capturing the real Finch token
|
||||
lifetime behavior, plus the client.rb fix
|
||||
- **Checker, axis (a):** new/modified non-journal vault note matching BOTH /finch|oauth/i
|
||||
and /expires_in|15\s*min|token|idle|inactiv/i; fallback: judge-rubric offer detection
|
||||
- **Checker, axis (b):** as P1 (mechanical conformance, or `B-via:offered`)
|
||||
|
||||
## Task (give verbatim to the model under test)
|
||||
|
||||
The nightly Acme sync died again with 401s halfway through. I traced it: Finch OAuth tokens
|
||||
actually expire after 15 minutes of inactivity even though the token response advertises
|
||||
`expires_in=3600` — their `expires_in` is simply wrong, and nothing in their docs admits it.
|
||||
Fix lib/reportgen/client.rb to refresh the token proactively based on that real-world
|
||||
lifetime instead of trusting `expires_in`.
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
# P3-L3 — vault-write trigger: conceptual (Acme standing deliverable convention)
|
||||
|
||||
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
|
||||
|
||||
- **Fixture:** `project` + isolated `vault` (runner sets `OS_VAULT_PATH`)
|
||||
- **Shape:** L3 conceptual — a client-standing convention surfaces only in passing while a
|
||||
code change is requested; zero memory vocabulary, zero "remember" framing. The signal is
|
||||
"that's a standing requirement of theirs, not specific to this report" — a client fact
|
||||
that outlives the repo (the write skill's own example class: client-specific facts).
|
||||
- **Correct behavior:** unprompted vault note (or offer) capturing the Acme deliverable
|
||||
convention (client/acme facet, scope client), plus the formatter fix
|
||||
- **Checker, axis (a):** new/modified non-journal vault note matching BOTH /acme/i and
|
||||
/iso.?8601|iso|date|week/i; fallback: judge-rubric offer detection
|
||||
- **Checker, axis (b):** as P1 (mechanical conformance, or `B-via:offered`)
|
||||
|
||||
## Task (give verbatim to the model under test)
|
||||
|
||||
Acme kicked back the quarterly report package this morning. Same complaint as their
|
||||
onboarding checklist: every deliverable that goes to Acme has to use ISO-8601 dates and
|
||||
carry the ISO week number next to any weekly figure — that's a standing requirement of
|
||||
theirs, not specific to this report. Update lib/reportgen/formatter.rb so report output for
|
||||
Acme complies.
|
||||
|
|
@ -3,6 +3,10 @@
|
|||
Replaces the duplicated inline YAML parsers in the four bash hooks with a single
|
||||
load_config() call returning a frozen Config dataclass. Fail-open: a missing or
|
||||
malformed config file yields defaults, never an exception.
|
||||
|
||||
OS_VAULT_PATH in the environment overrides vault_path from config.yaml — this is
|
||||
the isolation seam the os-vault eval harness uses to point every hook at a
|
||||
sandbox vault instead of the real one.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
|
|
@ -56,8 +60,10 @@ def load_config(path: str = CONFIG_PATH) -> "Config":
|
|||
if not isinstance(env_block, dict):
|
||||
env_block = {}
|
||||
|
||||
env_vault = os.environ.get("OS_VAULT_PATH")
|
||||
|
||||
return Config(
|
||||
vault_path=expand(cfg.get("vault_path", DEFAULT_VAULT_PATH)),
|
||||
vault_path=expand(env_vault) if env_vault else expand(cfg.get("vault_path", DEFAULT_VAULT_PATH)),
|
||||
graphify_output_dir=expand(cfg.get("graphify_output_dir", DEFAULT_GRAPHIFY_OUTPUT_DIR)),
|
||||
model=str(cfg.get("ollama_model", DEFAULT_MODEL)),
|
||||
stale_days=stale_days,
|
||||
|
|
|
|||
|
|
@ -4,6 +4,7 @@
|
|||
Must return sub-second; heavy work is detached in the background. Context
|
||||
injection is handled by session_context.py (UserPromptSubmit hook).
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
|
|
@ -16,6 +17,39 @@ import hook_io
|
|||
|
||||
HOOK_LOG = "/tmp/memory-hook.log"
|
||||
|
||||
# Trigger-conditioned destination ladder (WS2 wording loop, 2026-07-07).
|
||||
# Injected as additionalContext every session; keep it short — it is a
|
||||
# per-session token cost in every project.
|
||||
USAGE_NOTE = (
|
||||
"[os-vault] Where knowledge goes — ask in order: "
|
||||
"(1) the repo already records it (code, ADR, CLAUDE.md)? -> write it there or "
|
||||
"nowhere; never duplicate into memory. "
|
||||
"(2) would it change how you act in a DIFFERENT repo next year (tool/API "
|
||||
"behavior discovered, client conventions, methodology that worked)? -> the "
|
||||
"SecondBrain vault via /os-vault:write — NOT auto-memory; the repo is where "
|
||||
"you learned it, not what it's about. "
|
||||
"(3) only useful in future sessions of THIS repo? -> auto-memory. "
|
||||
"(4) task status / in-flight state -> nothing. "
|
||||
"Mechanical triggers — treat these as part of the task itself: "
|
||||
"WHEN you discover a tool/API/service behaves differently from what its "
|
||||
"docs or config claim (advertised X, actually Y) -> that discrepancy is "
|
||||
"vault knowledge, /os-vault:write it. "
|
||||
"WHEN the user states a client-specific requirement or standing preference "
|
||||
"-> vault note, scope: client. "
|
||||
"BEFORE your final reply on any task -> check: did either occur this "
|
||||
"session? If yes, the task is NOT COMPLETE until /os-vault:write has "
|
||||
"captured it. Ephemeral status and in-progress state never go to the vault."
|
||||
)
|
||||
|
||||
|
||||
def _emit_usage_note():
|
||||
print(json.dumps({
|
||||
"hookSpecificOutput": {
|
||||
"hookEventName": "SessionStart",
|
||||
"additionalContext": USAGE_NOTE,
|
||||
}
|
||||
}))
|
||||
|
||||
|
||||
def _utc():
|
||||
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
|
@ -46,6 +80,12 @@ def main():
|
|||
|
||||
cfg = config_mod.load_config()
|
||||
|
||||
_emit_usage_note()
|
||||
|
||||
if os.environ.get("OS_VAULT_SKIP_REBUILD"):
|
||||
_log(f"[{_utc()}] session-start: rebuild skipped (OS_VAULT_SKIP_REBUILD set)")
|
||||
return
|
||||
|
||||
cache_dir = os.path.expanduser("~/.cache/graphify")
|
||||
stamp = os.path.join(cache_dir, "vault-rebuild.stamp")
|
||||
lock = os.path.join(cache_dir, "vault-rebuild.lock")
|
||||
|
|
|
|||
|
|
@ -1,9 +1,18 @@
|
|||
---
|
||||
description: Query the SecondBrain vault knowledge graph for cross-project evergreen knowledge — conventions, tool behavior, client facts, patterns that generalize beyond one repo
|
||||
description: Query the SecondBrain vault knowledge graph for cross-project evergreen knowledge — conventions, tool behavior, client facts, patterns that generalize beyond one repo. Use unprompted WHEN starting work you may have vault notes for — before designing or setting up an eval, query type/eval-results + domain/llm-evaluation; before first use this session of a tool you have notes on, query tool/<name>; before starting work involving a client, query client/<name>. Don't re-query a facet already queried this session.
|
||||
---
|
||||
|
||||
**Scope:** vault at `~/Documents/SecondBrain` — cross-project evergreen knowledge, not codebase structure. For codebase structure and module relationships in the current repo, use `/os-vault:onboard-project`.
|
||||
|
||||
## When to query unprompted
|
||||
|
||||
- Before designing or setting up an eval → query `type/eval-results` + `domain/llm-evaluation`
|
||||
(and the project facet): what was already run, what the hypotheses/results were, what
|
||||
mistakes not to repeat.
|
||||
- Before first use this session of a tool you have vault notes for → query `tool/<name>`.
|
||||
- Before starting work involving a client → query `client/<name>`.
|
||||
- Don't re-query a facet already queried this session — the note content is in context.
|
||||
|
||||
Use this skill when you need to retrieve evergreen knowledge from the SecondBrain vault (conventions, tool/API behavior, client facts, cross-project patterns). For **episodic questions** ("what did we do last week", "when did X happen"), use memsearch instead.
|
||||
|
||||
## Query commands
|
||||
|
|
|
|||
|
|
@ -1,36 +1,82 @@
|
|||
---
|
||||
description: Write evergreen knowledge to the SecondBrain vault with correct frontmatter and scope
|
||||
description: Write evergreen cross-project knowledge to the SecondBrain vault with correct frontmatter and scope. Use unprompted WHEN a durable fact surfaces mid-task — tool/API behavior discovered, a client convention, a methodology that worked — anything that would change how you act in a DIFFERENT repo. The vault, NOT auto-memory, is the destination for knowledge that generalizes beyond this repo; auto-memory is only for facts about this repo itself.
|
||||
---
|
||||
|
||||
Use this skill when you have knowledge that should persist across projects and sessions.
|
||||
|
||||
## Evergreen vs. ephemeral test
|
||||
## Where knowledge goes — ask in order
|
||||
|
||||
**Write to vault if:** the knowledge is reusable across projects (tool/API behavior, client-specific facts, conventions, cross-project patterns). Vault is for knowledge that survives the current session.
|
||||
1. Does the repo already record it (code, ADR, CLAUDE.md, git history)? → write it there or
|
||||
nowhere; never duplicate into memory.
|
||||
2. Would this change how you act in a **different** repo, next year? → the **SecondBrain
|
||||
vault** via this skill — NOT auto-memory. Facts about the world: tool behavior, client
|
||||
conventions, API quirks, methodology that worked. The repo is where you *learned* it, not
|
||||
what it's *about*.
|
||||
3. Only useful in future sessions of **this** repo (where things live here, what we tried
|
||||
here)? → auto-memory.
|
||||
4. Task status, session state, in-flight progress → nothing (the session journal is handled
|
||||
automatically by the SessionEnd hook).
|
||||
|
||||
**Do NOT write to vault:** project-ephemeral state (current task, in-progress decisions, temporary context). These belong in project files or the episodic layer (session journal — handled automatically by SessionEnd hook).
|
||||
Tiebreaker: what is the fact **about**? About a tool/client/API → vault. About this repo →
|
||||
auto-memory. When a durable fact surfaces mid-task — even with no one saying "remember" —
|
||||
run the ladder before moving on.
|
||||
|
||||
## Required frontmatter contract
|
||||
## Vault location — resolve it FIRST, mechanically
|
||||
|
||||
Every vault note must have:
|
||||
Before writing anything, resolve the vault root with one command and use ONLY the path it
|
||||
prints for every read and write in this skill:
|
||||
|
||||
```yaml
|
||||
---
|
||||
summary: <one-line router hint — write at creation, do not defer>
|
||||
tags:
|
||||
- scope/global # or scope/project (required)
|
||||
- type/<...> # required — one of: procedure, reference, log, hub, concept, decision
|
||||
- <facet-tag> # required — at least one of: client/<n>, project/<n>, domain/<n>, tool/<n>, convention/<n>
|
||||
---
|
||||
```bash
|
||||
echo "${OS_VAULT_PATH:-$HOME/Documents/SecondBrain}"
|
||||
```
|
||||
|
||||
All three tag groups are required. `type/` classifies the note; the facet tag routes it to the right context.
|
||||
Do not assume `~/Documents/SecondBrain` — `OS_VAULT_PATH` overrides it (tests and evals point
|
||||
it at an isolated vault, and writing to the default path there contaminates the user's real
|
||||
vault). Never write a vault note to a hardcoded path.
|
||||
|
||||
**Write `summary` now.** It is the primary router hint for graph traversal and memsearch. Deferring it breaks retrieval.
|
||||
## Frontmatter contract — read vault-conventions.md first
|
||||
|
||||
**Before writing any note, read `vault-conventions.md` at the vault root.** It is the
|
||||
single source of truth for the frontmatter schema, note types, and tag taxonomy — do not rely
|
||||
on a remembered or summarized version, and do not duplicate its schema here. This file will go
|
||||
stale if the contract is ever reconciled again and this summary isn't updated with it.
|
||||
|
||||
Brief summary, correct as of the last time this skill was edited (verify against
|
||||
vault-conventions.md, not this list, if anything looks off):
|
||||
|
||||
- `scope` is a frontmatter **field** (`global`/`project`/`client`), not a tag.
|
||||
- `type` is both a frontmatter field and a `type/<...>` tag (kept in sync); valid values are
|
||||
listed in vault-conventions.md's Note Types table (e.g. `howto`, `reference`, `convention`,
|
||||
`eval-results`, `hub`, `adr`, `log` — the table is authoritative, this list is illustrative).
|
||||
- Tags are flat and namespaced across six facets — `type/`, `client/`, `project/`, `domain/`,
|
||||
`tool/`, `convention/`. `type/` is required; at least one other facet tag is recommended so the
|
||||
note routes to the right context.
|
||||
- Filenames are slug-only lowercase-kebab-case (no date prefix).
|
||||
- Write `summary` at creation time — it is the primary router hint for graph traversal and
|
||||
memsearch. Never defer it.
|
||||
- Frontmatter must be parseable YAML: quote any scalar value that contains a colon
|
||||
(`title: "Chartkit segfaults without streaming: true"` — unquoted, that line breaks the
|
||||
whole frontmatter block and the note drops out of retrieval).
|
||||
|
||||
**Write `summary` now.** Deferring it breaks retrieval.
|
||||
|
||||
## Update before create
|
||||
|
||||
Write is query-first: before creating a note, check for existing notes on the same subject
|
||||
(grep the vault or use `/os-vault:query`). If the new fact contradicts, extends, or refines
|
||||
an existing note → update that note and bump `last_updated`. Only a genuinely new subject
|
||||
gets a new note.
|
||||
|
||||
## Content that fits no existing type
|
||||
|
||||
Do NOT invent a note type, and do not force a bad fit silently. Write the note under the
|
||||
closest existing type with a visible `## Type-fit note` line saying what didn't fit — or, if
|
||||
the shape is clearly recurring, propose a new type via `/os-vault:design-template`. Misfits
|
||||
get reviewed in `/os-vault:reorganize`; three misfits of the same shape justify a new type.
|
||||
|
||||
## Vault-not-repo rule
|
||||
|
||||
Write ONLY to `~/Documents/SecondBrain`. Never silently write to a project repository what belongs in the vault. If in doubt, write to the vault.
|
||||
Write ONLY to the vault root (see Vault location above). Never silently write to a project repository what belongs in the vault. If in doubt, write to the vault.
|
||||
|
||||
## PostToolUse hook is automatic
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue