8.7 KiB
Claude-Tier Reference-Set Benchmark — Dispatch Prompt
Last updated: 2026-06-04 Status: active — copy-paste this into a Claude Code session to generate the reference set
Overview
This prompt dispatches one subagent per Claude tier (claude-haiku-4-5, claude-sonnet-4-6,
claude-opus-4-8) over each fixture note selected in Step 1c of the build runbook. Each subagent
emits a Graphify-shaped structured fragment. The per-model output files become the gold-standard
reference set against which local Ollama doc-extraction models are scored in a later step.
claude-opus-4-8's output is the gold-standard rubric.
Evaluation is on quality only: entity correctness, relationship plausibility and typing, and confidence-tag accuracy. Wall-clock speed is NOT a metric for this Claude reference run — speed re-enters only when local Ollama models are timed against this reference set.
Step 1 — Select Fixture Notes (operator action, before dispatching)
Fixture notes are selected at build time from ~/Documents/SecondBrain per Step 1c of the
runbook. Choose 5–10 notes with deliberate variety:
- One tool note (documents a specific tool or library)
- One client/project note (describes a client engagement or project)
- One convention note (captures a working convention or practice)
- One domain note (covers a knowledge or technical domain)
- One relationship-dense note (many named entities or cross-references)
- Additional notes as needed for coverage
Assign each note a kebab-case slug (used in output filenames). Populate the placeholder list below before dispatching:
# FIXTURES — populate before running
# Format: <note-slug>:<absolute-path-to-note>
FIXTURES=(
# <note-slug>:<path>
# <note-slug>:<path>
# ... (5-10 entries, one per selected note)
)
Step 2 — Output File Convention
For each fixture note, the run produces one output file per Claude tier — three files per note,
3 × N files total (where N is the number of fixture notes).
Output path pattern:
docs/memory-system/benchmark/reference-outputs/<note-slug>.<tier>.md
Where <tier> is one of: haiku, sonnet, opus
Examples (using a hypothetical slug graphify-tool-overview):
reference-outputs/graphify-tool-overview.haiku.mdreference-outputs/graphify-tool-overview.sonnet.mdreference-outputs/graphify-tool-overview.opus.md
Each file contains that tier's extraction fragment for one fixture note — the note_slug:
block for that note only. Do not combine multiple notes into one file.
Step 3 — Dispatch (copy-paste this block into a Claude Code session)
Operator: replace
<NOTE-SLUG>,<NOTE-TEXT>, and model names as needed. Dispatch all three subagents for each fixture note. They can run in parallel across tiers for the same note.
Dispatch template (repeat for each fixture note × each tier)
Dispatch a subagent using model <MODEL> to perform the following task.
=== FAIRNESS CONTRACT ===
You will receive exactly two inputs:
1. The raw text of one vault note (below).
2. The shared extraction spec and output schema (below).
You MUST NOT read any repository files (CLAUDE.md, design docs, specs, tasks), access the
vault directory structure, or use any project or system context. If your environment has
injected any such context automatically, you must ignore it entirely — treat it as if it
does not exist. The only allowed inputs are the note text and the extraction spec below.
=== END FAIRNESS CONTRACT ===
=== NOTE TEXT ===
<NOTE-TEXT>
=== END NOTE TEXT ===
=== EXTRACTION SPEC AND OUTPUT SCHEMA ===
## Facet Vocabulary (closed)
Notes carry six flat, namespaced facets plus one scope tag. These are the ONLY valid values
for the optional `facet` field on entities. Do not invent new namespaces.
| Prefix | Meaning |
|--------------|-----------------------------------------|
| `type/` | What kind of thing the note is about |
| `client/` | A client identity |
| `project/` | A project name |
| `domain/` | A knowledge or technical domain |
| `tool/` | A specific tool, library, or CLI |
| `convention/`| A working convention or practice |
| `scope/` | Applicability scope (cross-cutting tag) |
An entity should carry a `facet` value only when the note text directly supports mapping it
to one of the above. Absence of a `facet` field is correct when no mapping is warranted.
## Output Schema
Emit ONLY the following YAML structure. No other keys, no prose, no summary.
```yaml
# --- Graphify extraction fragment ---
# One block per fixture note. Repeat this structure for each note.
note_slug: <kebab-case-identifier-for-the-note> # operator fills this in at run time
entities:
- name: <string> # exact or near-exact name as it appears in the note
type: <string> # e.g. Person, Tool, Project, Concept, Convention, Client, Domain
facet: <string> # OPTIONAL — must be one of the seven prefixes above, e.g. "tool/graphify"
confidence: <string> # OPTIONAL — omit if the entity is directly stated
# Values: INFERRED | AMBIGUOUS
relationships:
- source: <entity name> # must match a name in the entities list above
type: <string> # free-text verb phrase, e.g. "uses", "depends_on", "implements", "replaces"
target: <entity name> # must match a name in the entities list above
confidence: <string> # OPTIONAL — omit if the relationship is directly stated
# Values: INFERRED | AMBIGUOUS
Confidence tag semantics
| Tag | Meaning |
|---|---|
| (absent) | The entity or relationship is directly stated in the note text. |
INFERRED |
Not stated literally but reasonably deducible from the note text alone. |
AMBIGUOUS |
Supportable by the text but uncertain or admits multiple readings. |
Rules
- Entities only: extract entities that are named (not vague category references).
- No invented names: entity names must be grounded in the note text.
- Relationship types: free-text verb phrases; do not normalize to a fixed vocabulary.
- Facet mapping: use the closed vocabulary above; only assign when the note text supports it.
- No extra keys: do not add summaries, scores, embeddings, or metadata fields.
- One block per note: if processing multiple fixture notes, emit one
note_slug:block per note, separated by---.
=== END EXTRACTION SPEC ===
Write your output to: docs/memory-system/benchmark/reference-outputs/..md
where is the kebab-case slug for this note and is one of: haiku, sonnet, opus.
The file must contain ONLY the YAML extraction fragment — no preamble, no explanation.
**MODEL and TIER substitutions:**
| Dispatch | MODEL | TIER |
|----------|------------------------|---------|
| 1st | `claude-haiku-4-5` | `haiku` |
| 2nd | `claude-sonnet-4-6` | `sonnet`|
| 3rd | `claude-opus-4-8` | `opus` |
---
## Step 4 — After All Subagents Complete
1. Verify `3 × N` files exist in `docs/memory-system/benchmark/reference-outputs/`.
2. For each fixture note, diff the three tier files side by side to understand tier-level
disagreements in entity recognition, relationship typing, and confidence tagging.
3. `claude-opus-4-8`'s `.opus.md` file for each note is the **gold-standard rubric** against
which local Ollama extraction will later be scored.
4. Do NOT run Ollama models in this step — that is a separate later step that uses these files
as the scoring reference.
---
## Notes on the Fairness Contract
The fairness contract requires that each subagent reasons from **note text + extraction spec
only**. Two mechanisms can break this:
1. **Deliberate reads:** the subagent reads repository files or the vault. The dispatch template
forbids this explicitly.
2. **Injected context:** Claude Code automatically injects `CLAUDE.md` and project context into
subagent sessions. The dispatch template explicitly instructs the subagent to **ignore any
injected context** and treat it as non-existent. This is the critical instruction — "don't
read files" is not sufficient on its own.
If you observe tier outputs that appear to reflect knowledge of the vault structure or system
design (beyond what the note text contains), treat that output as contaminated and re-run that
subagent with stronger isolation instructions.