# Spec: Reference Extraction Benchmark

## Purpose

Defines the process for producing a reusable Claude-tier reference set that serves as the quality rubric against which local Ollama extraction models are later scored.

## Requirements

### Requirement: Reference set is produced from Claude tiers

The benchmark SHALL produce a reusable reference set by dispatching one Claude Code subagent per Claude tier (`claude-haiku-4-5`, `claude-sonnet-4-6`, `claude-opus-4-8`) over the fixture notes, with each subagent's output written to a distinct per-model file.

#### Scenario: One reference file per model

- **WHEN** the benchmark prompt is run on the fixture set
- **THEN** a separate output file is produced for each of the three Claude tiers
- **AND** each file contains that tier's extraction result for every fixture note

#### Scenario: Opus output is the scoring rubric

- **WHEN** the reference set is reviewed
- **THEN** the `claude-opus-4-8` output is treated as the gold-standard rubric against which the other tiers, and later the local Ollama models, are scored

### Requirement: Dispatch prompt enforces a fairness contract

The benchmark prompt SHALL give each subagent only the raw note text and a shared minimal extraction spec, and SHALL forbid reading repository files or pulling project context.

#### Scenario: Minimal context per subagent

- **WHEN** a subagent is dispatched for a fixture note
- **THEN** its input is limited to the note text and the shared extraction spec
- **AND** it is explicitly instructed not to read `CLAUDE.md`, design docs, or other project context

### Requirement: Output conforms to the Graphify-shaped schema

Each subagent SHALL emit a structured fragment matching the shape Graphify produces — entities, typed relationships, and `INFERRED`/`AMBIGUOUS` confidence tags — so per-model outputs are diffable against each other and against later Ollama output.

#### Scenario: Schema embedded in the prompt

- **WHEN** the benchmark prompt is authored
- **THEN** the required output schema is embedded verbatim in the prompt
- **AND** every per-model output file follows that schema

#### Scenario: Outputs are diffable

- **WHEN** two per-model output files for the same fixture note are compared
- **THEN** they share a common structure that allows entity-by-entity and edge-by-edge comparison

### Requirement: Claude reference run is scored on quality only

The Claude reference run SHALL be evaluated on extraction quality only — entity correctness, relationship plausibility and typing, and confidence-tag accuracy — and SHALL NOT use wall-clock speed as a metric.

#### Scenario: Speed excluded from the reference run

- **WHEN** the Claude reference run is evaluated
- **THEN** wall-clock time per note is not used as a metric
- **AND** speed is reintroduced only when local Ollama models are later benchmarked against the reference set

### Requirement: Prompt is a committed, reusable file

The benchmark dispatch prompt SHALL be written as a committed file that can be copy/pasted into a Claude Code session to reproduce the run.

#### Scenario: Prompt persisted as a file

- **WHEN** the benchmark is set up
- **THEN** the dispatch prompt exists as a committed file in the repository
- **AND** running it does not require reconstructing the prompt from memory