cc-os/docs/memory-system/benchmark/dispatch-prompt.md

# Claude-Tier Reference-Set Benchmark — Dispatch Prompt

_Last updated: 2026-06-04_
_Status: active — copy-paste this into a Claude Code session to generate the reference set_

## Overview

This prompt dispatches **one subagent per Claude tier** (`claude-haiku-4-5`, `claude-sonnet-4-6`,
`claude-opus-4-8`) over each fixture note selected in Step 1c of the build runbook. Each subagent
emits a Graphify-shaped structured fragment. The per-model output files become the **gold-standard
reference set** against which local Ollama doc-extraction models are scored in a later step.

**`claude-opus-4-8`'s output is the gold-standard rubric.**

Evaluation is on **quality only**: entity correctness, relationship plausibility and typing, and
confidence-tag accuracy. Wall-clock speed is NOT a metric for this Claude reference run — speed
re-enters only when local Ollama models are timed against this reference set.

---

## Step 1 — Select Fixture Notes (operator action, before dispatching)

Fixture notes are selected at build time from `~/Documents/SecondBrain` per Step 1c of the
runbook. Choose **5–10 notes** with deliberate variety:

- One **tool note** (documents a specific tool or library)
- One **client/project note** (describes a client engagement or project)
- One **convention note** (captures a working convention or practice)
- One **domain note** (covers a knowledge or technical domain)
- One **relationship-dense note** (many named entities or cross-references)
- Additional notes as needed for coverage

Assign each note a **kebab-case slug** (used in output filenames). Populate the placeholder list
below before dispatching:

```
# FIXTURES — populate before running
# Format: <note-slug>:<absolute-path-to-note>
FIXTURES=(
  # <note-slug>:<path>
  # <note-slug>:<path>
  # ... (5-10 entries, one per selected note)
)
```

---

## Step 2 — Output File Convention

For each fixture note, the run produces **one output file per Claude tier** — three files per note,
`3 × N` files total (where N is the number of fixture notes).

Output path pattern:

```
docs/memory-system/benchmark/reference-outputs/<note-slug>.<tier>.md
```

Where `<tier>` is one of: `haiku`, `sonnet`, `opus`

Examples (using a hypothetical slug `graphify-tool-overview`):
- `reference-outputs/graphify-tool-overview.haiku.md`
- `reference-outputs/graphify-tool-overview.sonnet.md`
- `reference-outputs/graphify-tool-overview.opus.md`

Each file contains that tier's extraction fragment for **one fixture note** — the `note_slug:`
block for that note only. Do not combine multiple notes into one file.

---

## Step 3 — Dispatch (copy-paste this block into a Claude Code session)

> **Operator:** replace `<NOTE-SLUG>`, `<NOTE-TEXT>`, and model names as needed. Dispatch all
> three subagents for each fixture note. They can run in parallel across tiers for the same note.

---

### Dispatch template (repeat for each fixture note × each tier)

```
Dispatch a subagent using model <MODEL> to perform the following task.

=== FAIRNESS CONTRACT ===
You will receive exactly two inputs:
  1. The raw text of one vault note (below).
  2. The shared extraction spec and output schema (below).

You MUST NOT read any repository files (CLAUDE.md, design docs, specs, tasks), access the
vault directory structure, or use any project or system context. If your environment has
injected any such context automatically, you must ignore it entirely — treat it as if it
does not exist. The only allowed inputs are the note text and the extraction spec below.
=== END FAIRNESS CONTRACT ===

=== NOTE TEXT ===
<NOTE-TEXT>
=== END NOTE TEXT ===

=== EXTRACTION SPEC AND OUTPUT SCHEMA ===

## Facet Vocabulary (closed)

Notes carry six flat, namespaced facets plus one scope tag. These are the ONLY valid values
for the optional `facet` field on entities. Do not invent new namespaces.

| Prefix       | Meaning                                 |
|--------------|-----------------------------------------|
| `type/`      | What kind of thing the note is about    |
| `client/`    | A client identity                       |
| `project/`   | A project name                          |
| `domain/`    | A knowledge or technical domain         |
| `tool/`      | A specific tool, library, or CLI        |
| `convention/`| A working convention or practice        |
| `scope/`     | Applicability scope (cross-cutting tag) |

An entity should carry a `facet` value only when the note text directly supports mapping it
to one of the above. Absence of a `facet` field is correct when no mapping is warranted.

## Output Schema

Emit ONLY the following YAML structure. No other keys, no prose, no summary.

```yaml
# --- Graphify extraction fragment ---
# One block per fixture note. Repeat this structure for each note.

note_slug: <kebab-case-identifier-for-the-note>   # operator fills this in at run time

entities:
  - name: <string>          # exact or near-exact name as it appears in the note
    type: <string>          # e.g. Person, Tool, Project, Concept, Convention, Client, Domain
    facet: <string>         # OPTIONAL — must be one of the seven prefixes above, e.g. "tool/graphify"
    confidence: <string>    # OPTIONAL — omit if the entity is directly stated
                            # Values: INFERRED | AMBIGUOUS

relationships:
  - source: <entity name>   # must match a name in the entities list above
    type: <string>          # free-text verb phrase, e.g. "uses", "depends_on", "implements", "replaces"
    target: <entity name>   # must match a name in the entities list above
    confidence: <string>    # OPTIONAL — omit if the relationship is directly stated
                            # Values: INFERRED | AMBIGUOUS
```

## Confidence tag semantics

| Tag        | Meaning                                                                              |
|------------|--------------------------------------------------------------------------------------|
| _(absent)_ | The entity or relationship is **directly stated** in the note text.                  |
| `INFERRED` | Not stated literally but **reasonably deducible** from the note text alone.          |
| `AMBIGUOUS`| Supportable by the text but **uncertain or admits multiple readings**.               |

## Rules

- **Entities only:** extract entities that are named (not vague category references).
- **No invented names:** entity names must be grounded in the note text.
- **Relationship types:** free-text verb phrases; do not normalize to a fixed vocabulary.
- **Facet mapping:** use the closed vocabulary above; only assign when the note text supports it.
- **No extra keys:** do not add summaries, scores, embeddings, or metadata fields.
- **One block per note:** if processing multiple fixture notes, emit one `note_slug:` block per
  note, separated by `---`.

=== END EXTRACTION SPEC ===

Write your output to:
  docs/memory-system/benchmark/reference-outputs/<NOTE-SLUG>.<TIER>.md

where <NOTE-SLUG> is the kebab-case slug for this note and <TIER> is one of: haiku, sonnet, opus.

The file must contain ONLY the YAML extraction fragment — no preamble, no explanation.
```

**MODEL and TIER substitutions:**

| Dispatch | MODEL                  | TIER    |
|----------|------------------------|---------|
| 1st      | `claude-haiku-4-5`     | `haiku` |
| 2nd      | `claude-sonnet-4-6`    | `sonnet`|
| 3rd      | `claude-opus-4-8`      | `opus`  |

---

## Step 4 — After All Subagents Complete

1. Verify `3 × N` files exist in `docs/memory-system/benchmark/reference-outputs/`.
2. For each fixture note, diff the three tier files side by side to understand tier-level
   disagreements in entity recognition, relationship typing, and confidence tagging.
3. `claude-opus-4-8`'s `.opus.md` file for each note is the **gold-standard rubric** against
   which local Ollama extraction will later be scored.
4. Do NOT run Ollama models in this step — that is a separate later step that uses these files
   as the scoring reference.

---

## Notes on the Fairness Contract

The fairness contract requires that each subagent reasons from **note text + extraction spec
only**. Two mechanisms can break this:

1. **Deliberate reads:** the subagent reads repository files or the vault. The dispatch template
   forbids this explicitly.
2. **Injected context:** Claude Code automatically injects `CLAUDE.md` and project context into
   subagent sessions. The dispatch template explicitly instructs the subagent to **ignore any
   injected context** and treat it as non-existent. This is the critical instruction — "don't
   read files" is not sufficient on its own.

If you observe tier outputs that appear to reflect knowledge of the vault structure or system
design (beyond what the note text contains), treat that output as contaminated and re-run that
subagent with stronger isolation instructions.