4.2 KiB

Raw Blame History

Graphify Document-Extraction Benchmark — Shared Extraction Spec

Last updated: 2026-06-04 Status: active — used as the shared schema for the Claude-tier reference-set benchmark

Purpose

This spec defines the minimal structured fragment that every benchmarking subagent (one per Claude tier) must emit when given a raw vault note. The fragment mimics what Graphify's local SLM doc-extraction step produces: a list of entities, a list of typed relationships (edges between entities), and per-item confidence tags where applicable.

The schema is intentionally minimal. It exists so that:

Per-model output files are diffable entity-by-entity and edge-by-edge across tiers.
The same schema is used unchanged when local Ollama models are later scored against this reference set — making the comparison apples-to-apples.

Subagents receive only the raw note text plus this spec. No repository files, no vault structure, no project context. If any such context is injected by the environment, it must be ignored.

Facet Vocabulary (closed)

Notes in the SecondBrain vault carry six flat, namespaced facets plus one scope tag. These are the only valid values for the optional facet field on entities. Do not invent new namespaces.

Prefix	Meaning
`type/`	What kind of thing the note is about
`client/`	A client identity
`project/`	A project name
`domain/`	A knowledge or technical domain
`tool/`	A specific tool, library, or CLI
`convention/`	A working convention or practice
`scope/`	Applicability scope (cross-cutting tag)

An entity should carry a facet value only when the note text directly supports mapping it to one of the above. Absence of a facet field is correct when no mapping is warranted.

Output Schema

The required output format. Every subagent MUST emit this structure and no other top-level keys.

# --- Graphify extraction fragment ---
# One block per fixture note. Repeat this structure for each note.

note_slug: <kebab-case-identifier-for-the-note>   # operator fills this in at run time

entities:
  - name: <string>          # exact or near-exact name as it appears in the note
    type: <string>          # e.g. Person, Tool, Project, Concept, Convention, Client, Domain
    facet: <string>         # OPTIONAL — must be one of the seven prefixes above, e.g. "tool/graphify"
    confidence: <string>    # OPTIONAL — omit if the entity is directly stated
                            # Values: INFERRED | AMBIGUOUS

relationships:
  - source: <entity name>   # must match a name in the entities list above
    type: <string>          # free-text verb phrase, e.g. "uses", "depends_on", "implements", "replaces"
    target: <entity name>   # must match a name in the entities list above
    confidence: <string>    # OPTIONAL — omit if the relationship is directly stated
                            # Values: INFERRED | AMBIGUOUS

Confidence tag semantics

Tag	Meaning
(absent)	The entity or relationship is directly stated in the note text.
`INFERRED`	Not stated literally but reasonably deducible from the note text alone.
`AMBIGUOUS`	Supportable by the text but uncertain or admits multiple readings.

Rules

Entities only: extract entities that are named (not vague category references).
No invented names: entity names must be grounded in the note text.
Relationship types: free-text verb phrases; do not normalize to a fixed vocabulary.
Facet mapping: use the closed vocabulary above; only assign when the note text supports it.
No extra keys: do not add summaries, scores, embeddings, or metadata fields.
One block per note: if processing multiple fixture notes, emit one note_slug: block per note, separated by ---.

4.2 KiB Raw Blame History