208 lines
8.7 KiB
Markdown
208 lines
8.7 KiB
Markdown
# Claude-Tier Reference-Set Benchmark — Dispatch Prompt
|
||
|
||
_Last updated: 2026-06-04_
|
||
_Status: active — copy-paste this into a Claude Code session to generate the reference set_
|
||
|
||
## Overview
|
||
|
||
This prompt dispatches **one subagent per Claude tier** (`claude-haiku-4-5`, `claude-sonnet-4-6`,
|
||
`claude-opus-4-8`) over each fixture note selected in Step 1c of the build runbook. Each subagent
|
||
emits a Graphify-shaped structured fragment. The per-model output files become the **gold-standard
|
||
reference set** against which local Ollama doc-extraction models are scored in a later step.
|
||
|
||
**`claude-opus-4-8`'s output is the gold-standard rubric.**
|
||
|
||
Evaluation is on **quality only**: entity correctness, relationship plausibility and typing, and
|
||
confidence-tag accuracy. Wall-clock speed is NOT a metric for this Claude reference run — speed
|
||
re-enters only when local Ollama models are timed against this reference set.
|
||
|
||
---
|
||
|
||
## Step 1 — Select Fixture Notes (operator action, before dispatching)
|
||
|
||
Fixture notes are selected at build time from `~/Documents/SecondBrain` per Step 1c of the
|
||
runbook. Choose **5–10 notes** with deliberate variety:
|
||
|
||
- One **tool note** (documents a specific tool or library)
|
||
- One **client/project note** (describes a client engagement or project)
|
||
- One **convention note** (captures a working convention or practice)
|
||
- One **domain note** (covers a knowledge or technical domain)
|
||
- One **relationship-dense note** (many named entities or cross-references)
|
||
- Additional notes as needed for coverage
|
||
|
||
Assign each note a **kebab-case slug** (used in output filenames). Populate the placeholder list
|
||
below before dispatching:
|
||
|
||
```
|
||
# FIXTURES — populate before running
|
||
# Format: <note-slug>:<absolute-path-to-note>
|
||
FIXTURES=(
|
||
# <note-slug>:<path>
|
||
# <note-slug>:<path>
|
||
# ... (5-10 entries, one per selected note)
|
||
)
|
||
```
|
||
|
||
---
|
||
|
||
## Step 2 — Output File Convention
|
||
|
||
For each fixture note, the run produces **one output file per Claude tier** — three files per note,
|
||
`3 × N` files total (where N is the number of fixture notes).
|
||
|
||
Output path pattern:
|
||
|
||
```
|
||
docs/memory-system/benchmark/reference-outputs/<note-slug>.<tier>.md
|
||
```
|
||
|
||
Where `<tier>` is one of: `haiku`, `sonnet`, `opus`
|
||
|
||
Examples (using a hypothetical slug `graphify-tool-overview`):
|
||
- `reference-outputs/graphify-tool-overview.haiku.md`
|
||
- `reference-outputs/graphify-tool-overview.sonnet.md`
|
||
- `reference-outputs/graphify-tool-overview.opus.md`
|
||
|
||
Each file contains that tier's extraction fragment for **one fixture note** — the `note_slug:`
|
||
block for that note only. Do not combine multiple notes into one file.
|
||
|
||
---
|
||
|
||
## Step 3 — Dispatch (copy-paste this block into a Claude Code session)
|
||
|
||
> **Operator:** replace `<NOTE-SLUG>`, `<NOTE-TEXT>`, and model names as needed. Dispatch all
|
||
> three subagents for each fixture note. They can run in parallel across tiers for the same note.
|
||
|
||
---
|
||
|
||
### Dispatch template (repeat for each fixture note × each tier)
|
||
|
||
```
|
||
Dispatch a subagent using model <MODEL> to perform the following task.
|
||
|
||
=== FAIRNESS CONTRACT ===
|
||
You will receive exactly two inputs:
|
||
1. The raw text of one vault note (below).
|
||
2. The shared extraction spec and output schema (below).
|
||
|
||
You MUST NOT read any repository files (CLAUDE.md, design docs, specs, tasks), access the
|
||
vault directory structure, or use any project or system context. If your environment has
|
||
injected any such context automatically, you must ignore it entirely — treat it as if it
|
||
does not exist. The only allowed inputs are the note text and the extraction spec below.
|
||
=== END FAIRNESS CONTRACT ===
|
||
|
||
=== NOTE TEXT ===
|
||
<NOTE-TEXT>
|
||
=== END NOTE TEXT ===
|
||
|
||
=== EXTRACTION SPEC AND OUTPUT SCHEMA ===
|
||
|
||
## Facet Vocabulary (closed)
|
||
|
||
Notes carry six flat, namespaced facets plus one scope tag. These are the ONLY valid values
|
||
for the optional `facet` field on entities. Do not invent new namespaces.
|
||
|
||
| Prefix | Meaning |
|
||
|--------------|-----------------------------------------|
|
||
| `type/` | What kind of thing the note is about |
|
||
| `client/` | A client identity |
|
||
| `project/` | A project name |
|
||
| `domain/` | A knowledge or technical domain |
|
||
| `tool/` | A specific tool, library, or CLI |
|
||
| `convention/`| A working convention or practice |
|
||
| `scope/` | Applicability scope (cross-cutting tag) |
|
||
|
||
An entity should carry a `facet` value only when the note text directly supports mapping it
|
||
to one of the above. Absence of a `facet` field is correct when no mapping is warranted.
|
||
|
||
## Output Schema
|
||
|
||
Emit ONLY the following YAML structure. No other keys, no prose, no summary.
|
||
|
||
```yaml
|
||
# --- Graphify extraction fragment ---
|
||
# One block per fixture note. Repeat this structure for each note.
|
||
|
||
note_slug: <kebab-case-identifier-for-the-note> # operator fills this in at run time
|
||
|
||
entities:
|
||
- name: <string> # exact or near-exact name as it appears in the note
|
||
type: <string> # e.g. Person, Tool, Project, Concept, Convention, Client, Domain
|
||
facet: <string> # OPTIONAL — must be one of the seven prefixes above, e.g. "tool/graphify"
|
||
confidence: <string> # OPTIONAL — omit if the entity is directly stated
|
||
# Values: INFERRED | AMBIGUOUS
|
||
|
||
relationships:
|
||
- source: <entity name> # must match a name in the entities list above
|
||
type: <string> # free-text verb phrase, e.g. "uses", "depends_on", "implements", "replaces"
|
||
target: <entity name> # must match a name in the entities list above
|
||
confidence: <string> # OPTIONAL — omit if the relationship is directly stated
|
||
# Values: INFERRED | AMBIGUOUS
|
||
```
|
||
|
||
## Confidence tag semantics
|
||
|
||
| Tag | Meaning |
|
||
|------------|--------------------------------------------------------------------------------------|
|
||
| _(absent)_ | The entity or relationship is **directly stated** in the note text. |
|
||
| `INFERRED` | Not stated literally but **reasonably deducible** from the note text alone. |
|
||
| `AMBIGUOUS`| Supportable by the text but **uncertain or admits multiple readings**. |
|
||
|
||
## Rules
|
||
|
||
- **Entities only:** extract entities that are named (not vague category references).
|
||
- **No invented names:** entity names must be grounded in the note text.
|
||
- **Relationship types:** free-text verb phrases; do not normalize to a fixed vocabulary.
|
||
- **Facet mapping:** use the closed vocabulary above; only assign when the note text supports it.
|
||
- **No extra keys:** do not add summaries, scores, embeddings, or metadata fields.
|
||
- **One block per note:** if processing multiple fixture notes, emit one `note_slug:` block per
|
||
note, separated by `---`.
|
||
|
||
=== END EXTRACTION SPEC ===
|
||
|
||
Write your output to:
|
||
docs/memory-system/benchmark/reference-outputs/<NOTE-SLUG>.<TIER>.md
|
||
|
||
where <NOTE-SLUG> is the kebab-case slug for this note and <TIER> is one of: haiku, sonnet, opus.
|
||
|
||
The file must contain ONLY the YAML extraction fragment — no preamble, no explanation.
|
||
```
|
||
|
||
**MODEL and TIER substitutions:**
|
||
|
||
| Dispatch | MODEL | TIER |
|
||
|----------|------------------------|---------|
|
||
| 1st | `claude-haiku-4-5` | `haiku` |
|
||
| 2nd | `claude-sonnet-4-6` | `sonnet`|
|
||
| 3rd | `claude-opus-4-8` | `opus` |
|
||
|
||
---
|
||
|
||
## Step 4 — After All Subagents Complete
|
||
|
||
1. Verify `3 × N` files exist in `docs/memory-system/benchmark/reference-outputs/`.
|
||
2. For each fixture note, diff the three tier files side by side to understand tier-level
|
||
disagreements in entity recognition, relationship typing, and confidence tagging.
|
||
3. `claude-opus-4-8`'s `.opus.md` file for each note is the **gold-standard rubric** against
|
||
which local Ollama extraction will later be scored.
|
||
4. Do NOT run Ollama models in this step — that is a separate later step that uses these files
|
||
as the scoring reference.
|
||
|
||
---
|
||
|
||
## Notes on the Fairness Contract
|
||
|
||
The fairness contract requires that each subagent reasons from **note text + extraction spec
|
||
only**. Two mechanisms can break this:
|
||
|
||
1. **Deliberate reads:** the subagent reads repository files or the vault. The dispatch template
|
||
forbids this explicitly.
|
||
2. **Injected context:** Claude Code automatically injects `CLAUDE.md` and project context into
|
||
subagent sessions. The dispatch template explicitly instructs the subagent to **ignore any
|
||
injected context** and treat it as non-existent. This is the critical instruction — "don't
|
||
read files" is not sufficient on its own.
|
||
|
||
If you observe tier outputs that appear to reflect knowledge of the vault structure or system
|
||
design (beyond what the note text contains), treat that output as contaminated and re-run that
|
||
subagent with stronger isolation instructions.
|