32 lines
3.3 KiB
Markdown
32 lines
3.3 KiB
Markdown
|
# Benchmark Run Findings — 2026-06-04 (Step 2c reference set)
|
||
|
|
|
||
|
|
_Run: 2026-06-04 — reference set generated, verified, gate satisfied._
|
||
|
|
|
||
|
|
This document records the outcomes, surprises, and lessons from the Step 2c reference-extraction benchmark run. It is the authoritative findings record for this run date and serves as input to future benchmark comparisons when the Graphify/Ollama production pipeline is evaluated.
|
||
|
|
|
||
|
|
## What was run
|
||
|
|
|
||
|
|
- 6 cross-domain fixture notes hand-selected from ~/Documents/SecondBrain (4 pest-control / niche-automation-prospecting, 2 software-design), covering all five fixture categories (tool, client/project, convention, domain, relationship-dense). Fixtures are listed in dispatch-prompt.md.
|
||
|
|
- Each note extracted by 3 Claude tiers (claude-haiku-4-5, claude-sonnet-4-6, claude-opus-4-8) via subagent dispatch with model overrides = 18 reference fragments in reference-outputs/.
|
||
|
|
- Notes were run AS-IS (no vault frontmatter modification); the SecondBrain vault was backed up first to ~/Documents/SecondBrain-backup-20260604-114749.
|
||
|
|
- The .opus.md files are the gold-standard scoring rubric; haiku/sonnet are kept to characterize the tier quality gradient.
|
||
|
|
|
||
|
|
## Key finding: fairness-contract contamination (the gate working as designed)
|
||
|
|
|
||
|
|
- The first-pass haiku extraction of the `ai-coding-conventions-synthesis` note leaked three entities — `Graphify`, `Milvus Lite`, `SecondBrain vault` — that do NOT appear in the note text. They were injected from the cc-os CLAUDE.md / project context that Claude Code auto-loads into subagents. Sonnet and opus did not leak on the same note.
|
||
|
|
- Lesson 1: the soft "ignore any injected context" instruction in the fairness contract is NOT reliable on the haiku tier — re-running clean required an explicit forbidden-class hint (do not introduce knowledge-graph/vector-db/vault tooling unless it appears in the note text).
|
||
|
|
- Lesson 2 (important, prevents a wrong conclusion): this is largely a HARNESS ARTIFACT, not a property of the production pipeline. In the real Graphify→Ollama pipeline, the SLM receives only the prompt Graphify constructs — there is no CLAUDE.md injection path. So the benchmark's contamination exposure is HIGHER than production; the cleaned reference set is conservative. Do NOT conclude "haiku extracts poorly," and do NOT carry the anti-contamination hint into the production Graphify extraction prompt — it is unneeded there.
|
||
|
|
|
||
|
|
## Spec gaps found and fixed
|
||
|
|
|
||
|
|
- The original extraction-spec.md did not require a sub-value after a facet namespace, producing degenerate bare-slash facets (`domain/`, `tool/`) in several sonnet/haiku outputs. It also did not state strict referential integrity, producing relationships pointing at undeclared entities.
|
||
|
|
- Both rules were added to extraction-spec.md on 2026-06-04, and the affected files regenerated so the whole 18-file set conforms to the final spec.
|
||
|
|
|
||
|
|
## Tier richness gradient (observed)
|
||
|
|
|
||
|
|
- Opus is reliably the richest extractor (e.g. the enterprise-revenue note: opus ~59-61 entities vs haiku ~25). Sonnet is sometimes terser than haiku. On stat-dense notes all three tiers saturate to similar counts. Treat entity/relationship counts as indicative, not exact (extractor self-counts diverged from verified counts).
|
||
|
|
|
||
|
|
## Status
|
||
|
|
|
||
|
|
- All 18 files verified: valid YAML, conformant schema, referential integrity, no contamination. Step 2c gate satisfied 2026-06-04.
|