# Claude-Tier Reference-Set Benchmark — Dispatch Prompt _Last updated: 2026-06-04_ _Status: active — copy-paste this into a Claude Code session to generate the reference set_ ## Overview This prompt dispatches **one subagent per Claude tier** (`claude-haiku-4-5`, `claude-sonnet-4-6`, `claude-opus-4-8`) over each fixture note selected in Step 1c of the build runbook. Each subagent emits a Graphify-shaped structured fragment. The per-model output files become the **gold-standard reference set** against which local Ollama doc-extraction models are scored in a later step. **`claude-opus-4-8`'s output is the gold-standard rubric.** Evaluation is on **quality only**: entity correctness, relationship plausibility and typing, and confidence-tag accuracy. Wall-clock speed is NOT a metric for this Claude reference run — speed re-enters only when local Ollama models are timed against this reference set. --- ## Step 1 — Select Fixture Notes (operator action, before dispatching) Fixture notes are selected at build time from `~/Documents/SecondBrain` per Step 1c of the runbook. Choose **5–10 notes** with deliberate variety: - One **tool note** (documents a specific tool or library) - One **client/project note** (describes a client engagement or project) - One **convention note** (captures a working convention or practice) - One **domain note** (covers a knowledge or technical domain) - One **relationship-dense note** (many named entities or cross-references) - Additional notes as needed for coverage Assign each note a **kebab-case slug** (used in output filenames). Populate the placeholder list below before dispatching: ``` # FIXTURES — populate before running # Format: : FIXTURES=( # : # : # ... (5-10 entries, one per selected note) ) ``` --- ## Step 2 — Output File Convention For each fixture note, the run produces **one output file per Claude tier** — three files per note, `3 × N` files total (where N is the number of fixture notes). Output path pattern: ``` docs/memory-system/benchmark/reference-outputs/..md ``` Where `` is one of: `haiku`, `sonnet`, `opus` Examples (using a hypothetical slug `graphify-tool-overview`): - `reference-outputs/graphify-tool-overview.haiku.md` - `reference-outputs/graphify-tool-overview.sonnet.md` - `reference-outputs/graphify-tool-overview.opus.md` Each file contains that tier's extraction fragment for **one fixture note** — the `note_slug:` block for that note only. Do not combine multiple notes into one file. --- ## Step 3 — Dispatch (copy-paste this block into a Claude Code session) > **Operator:** replace ``, ``, and model names as needed. Dispatch all > three subagents for each fixture note. They can run in parallel across tiers for the same note. --- ### Dispatch template (repeat for each fixture note × each tier) ``` Dispatch a subagent using model to perform the following task. === FAIRNESS CONTRACT === You will receive exactly two inputs: 1. The raw text of one vault note (below). 2. The shared extraction spec and output schema (below). You MUST NOT read any repository files (CLAUDE.md, design docs, specs, tasks), access the vault directory structure, or use any project or system context. If your environment has injected any such context automatically, you must ignore it entirely — treat it as if it does not exist. The only allowed inputs are the note text and the extraction spec below. === END FAIRNESS CONTRACT === === NOTE TEXT === === END NOTE TEXT === === EXTRACTION SPEC AND OUTPUT SCHEMA === ## Facet Vocabulary (closed) Notes carry six flat, namespaced facets plus one scope tag. These are the ONLY valid values for the optional `facet` field on entities. Do not invent new namespaces. | Prefix | Meaning | |--------------|-----------------------------------------| | `type/` | What kind of thing the note is about | | `client/` | A client identity | | `project/` | A project name | | `domain/` | A knowledge or technical domain | | `tool/` | A specific tool, library, or CLI | | `convention/`| A working convention or practice | | `scope/` | Applicability scope (cross-cutting tag) | An entity should carry a `facet` value only when the note text directly supports mapping it to one of the above. Absence of a `facet` field is correct when no mapping is warranted. ## Output Schema Emit ONLY the following YAML structure. No other keys, no prose, no summary. ```yaml # --- Graphify extraction fragment --- # One block per fixture note. Repeat this structure for each note. note_slug: # operator fills this in at run time entities: - name: # exact or near-exact name as it appears in the note type: # e.g. Person, Tool, Project, Concept, Convention, Client, Domain facet: # OPTIONAL — must be one of the seven prefixes above, e.g. "tool/graphify" confidence: # OPTIONAL — omit if the entity is directly stated # Values: INFERRED | AMBIGUOUS relationships: - source: # must match a name in the entities list above type: # free-text verb phrase, e.g. "uses", "depends_on", "implements", "replaces" target: # must match a name in the entities list above confidence: # OPTIONAL — omit if the relationship is directly stated # Values: INFERRED | AMBIGUOUS ``` ## Confidence tag semantics | Tag | Meaning | |------------|--------------------------------------------------------------------------------------| | _(absent)_ | The entity or relationship is **directly stated** in the note text. | | `INFERRED` | Not stated literally but **reasonably deducible** from the note text alone. | | `AMBIGUOUS`| Supportable by the text but **uncertain or admits multiple readings**. | ## Rules - **Entities only:** extract entities that are named (not vague category references). - **No invented names:** entity names must be grounded in the note text. - **Relationship types:** free-text verb phrases; do not normalize to a fixed vocabulary. - **Facet mapping:** use the closed vocabulary above; only assign when the note text supports it. - **No extra keys:** do not add summaries, scores, embeddings, or metadata fields. - **One block per note:** if processing multiple fixture notes, emit one `note_slug:` block per note, separated by `---`. === END EXTRACTION SPEC === Write your output to: docs/memory-system/benchmark/reference-outputs/..md where is the kebab-case slug for this note and is one of: haiku, sonnet, opus. The file must contain ONLY the YAML extraction fragment — no preamble, no explanation. ``` **MODEL and TIER substitutions:** | Dispatch | MODEL | TIER | |----------|------------------------|---------| | 1st | `claude-haiku-4-5` | `haiku` | | 2nd | `claude-sonnet-4-6` | `sonnet`| | 3rd | `claude-opus-4-8` | `opus` | --- ## Step 4 — After All Subagents Complete 1. Verify `3 × N` files exist in `docs/memory-system/benchmark/reference-outputs/`. 2. For each fixture note, diff the three tier files side by side to understand tier-level disagreements in entity recognition, relationship typing, and confidence tagging. 3. `claude-opus-4-8`'s `.opus.md` file for each note is the **gold-standard rubric** against which local Ollama extraction will later be scored. 4. Do NOT run Ollama models in this step — that is a separate later step that uses these files as the scoring reference. --- ## Notes on the Fairness Contract The fairness contract requires that each subagent reasons from **note text + extraction spec only**. Two mechanisms can break this: 1. **Deliberate reads:** the subagent reads repository files or the vault. The dispatch template forbids this explicitly. 2. **Injected context:** Claude Code automatically injects `CLAUDE.md` and project context into subagent sessions. The dispatch template explicitly instructs the subagent to **ignore any injected context** and treat it as non-existent. This is the critical instruction — "don't read files" is not sufficient on its own. If you observe tier outputs that appear to reflect knowledge of the vault structure or system design (beyond what the note text contains), treat that output as contaminated and re-run that subagent with stronger isolation instructions.