68 lines
3.3 KiB
Markdown
68 lines
3.3 KiB
Markdown
# Spec: Reference Extraction Benchmark
|
|
|
|
## Purpose
|
|
|
|
Defines the process for producing a reusable Claude-tier reference set that serves as the quality rubric against which local Ollama extraction models are later scored.
|
|
|
|
## Requirements
|
|
|
|
### Requirement: Reference set is produced from Claude tiers
|
|
|
|
The benchmark SHALL produce a reusable reference set by dispatching one Claude Code subagent per Claude tier (`claude-haiku-4-5`, `claude-sonnet-4-6`, `claude-opus-4-8`) over the fixture notes, with each subagent's output written to a distinct per-model file.
|
|
|
|
#### Scenario: One reference file per model
|
|
|
|
- **WHEN** the benchmark prompt is run on the fixture set
|
|
- **THEN** a separate output file is produced for each of the three Claude tiers
|
|
- **AND** each file contains that tier's extraction result for every fixture note
|
|
|
|
#### Scenario: Opus output is the scoring rubric
|
|
|
|
- **WHEN** the reference set is reviewed
|
|
- **THEN** the `claude-opus-4-8` output is treated as the gold-standard rubric against which the other tiers, and later the local Ollama models, are scored
|
|
|
|
### Requirement: Dispatch prompt enforces a fairness contract
|
|
|
|
The benchmark prompt SHALL give each subagent only the raw note text and a shared minimal extraction spec, and SHALL forbid reading repository files or pulling project context.
|
|
|
|
#### Scenario: Minimal context per subagent
|
|
|
|
- **WHEN** a subagent is dispatched for a fixture note
|
|
- **THEN** its input is limited to the note text and the shared extraction spec
|
|
- **AND** it is explicitly instructed not to read `CLAUDE.md`, design docs, or other project context
|
|
|
|
### Requirement: Output conforms to the Graphify-shaped schema
|
|
|
|
Each subagent SHALL emit a structured fragment matching the shape Graphify produces — entities, typed relationships, and `INFERRED`/`AMBIGUOUS` confidence tags — so per-model outputs are diffable against each other and against later Ollama output.
|
|
|
|
#### Scenario: Schema embedded in the prompt
|
|
|
|
- **WHEN** the benchmark prompt is authored
|
|
- **THEN** the required output schema is embedded verbatim in the prompt
|
|
- **AND** every per-model output file follows that schema
|
|
|
|
#### Scenario: Outputs are diffable
|
|
|
|
- **WHEN** two per-model output files for the same fixture note are compared
|
|
- **THEN** they share a common structure that allows entity-by-entity and edge-by-edge comparison
|
|
|
|
### Requirement: Claude reference run is scored on quality only
|
|
|
|
The Claude reference run SHALL be evaluated on extraction quality only — entity correctness, relationship plausibility and typing, and confidence-tag accuracy — and SHALL NOT use wall-clock speed as a metric.
|
|
|
|
#### Scenario: Speed excluded from the reference run
|
|
|
|
- **WHEN** the Claude reference run is evaluated
|
|
- **THEN** wall-clock time per note is not used as a metric
|
|
- **AND** speed is reintroduced only when local Ollama models are later benchmarked against the reference set
|
|
|
|
### Requirement: Prompt is a committed, reusable file
|
|
|
|
The benchmark dispatch prompt SHALL be written as a committed file that can be copy/pasted into a Claude Code session to reproduce the run.
|
|
|
|
#### Scenario: Prompt persisted as a file
|
|
|
|
- **WHEN** the benchmark is set up
|
|
- **THEN** the dispatch prompt exists as a committed file in the repository
|
|
- **AND** running it does not require reconstructing the prompt from memory
|