cc-os/openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/specs/reference-extraction-benchmark/spec.md

ADDED Requirements

Requirement: Reference set is produced from Claude tiers

The benchmark SHALL produce a reusable reference set by dispatching one Claude Code subagent per Claude tier (claude-haiku-4-5, claude-sonnet-4-6, claude-opus-4-8) over the fixture notes, with each subagent's output written to a distinct per-model file.

Scenario: One reference file per model

• WHEN the benchmark prompt is run on the fixture set
• THEN a separate output file is produced for each of the three Claude tiers
• AND each file contains that tier's extraction result for every fixture note

Scenario: Opus output is the scoring rubric

• WHEN the reference set is reviewed
• THEN the claude-opus-4-8 output is treated as the gold-standard rubric against which the other tiers, and later the local Ollama models, are scored

Requirement: Dispatch prompt enforces a fairness contract

The benchmark prompt SHALL give each subagent only the raw note text and a shared minimal extraction spec, and SHALL forbid reading repository files or pulling project context.

Scenario: Minimal context per subagent

• WHEN a subagent is dispatched for a fixture note
• THEN its input is limited to the note text and the shared extraction spec
• AND it is explicitly instructed not to read CLAUDE.md, design docs, or other project context

Requirement: Output conforms to the Graphify-shaped schema

Each subagent SHALL emit a structured fragment matching the shape Graphify produces — entities, typed relationships, and INFERRED/AMBIGUOUS confidence tags — so per-model outputs are diffable against each other and against later Ollama output.

Scenario: Schema embedded in the prompt

• WHEN the benchmark prompt is authored
• THEN the required output schema is embedded verbatim in the prompt
• AND every per-model output file follows that schema

Scenario: Outputs are diffable

• WHEN two per-model output files for the same fixture note are compared
• THEN they share a common structure that allows entity-by-entity and edge-by-edge comparison

Requirement: Claude reference run is scored on quality only

The Claude reference run SHALL be evaluated on extraction quality only — entity correctness, relationship plausibility and typing, and confidence-tag accuracy — and SHALL NOT use wall-clock speed as a metric.

Scenario: Speed excluded from the reference run

• WHEN the Claude reference run is evaluated
• THEN wall-clock time per note is not used as a metric
• AND speed is reintroduced only when local Ollama models are later benchmarked against the reference set

Requirement: Prompt is a committed, reusable file

The benchmark dispatch prompt SHALL be written as a committed file that can be copy/pasted into a Claude Code session to reproduce the run.

Scenario: Prompt persisted as a file

• WHEN the benchmark is set up
• THEN the dispatch prompt exists as a committed file in the repository
• AND running it does not require reconstructing the prompt from memory