85 lines
4.2 KiB
Markdown
85 lines
4.2 KiB
Markdown
# Graphify Document-Extraction Benchmark — Shared Extraction Spec
|
|
|
|
_Last updated: 2026-06-04_
|
|
_Status: active — used as the shared schema for the Claude-tier reference-set benchmark_
|
|
|
|
## Purpose
|
|
|
|
This spec defines the **minimal structured fragment** that every benchmarking subagent (one per
|
|
Claude tier) must emit when given a raw vault note. The fragment mimics what Graphify's local SLM
|
|
doc-extraction step produces: a list of **entities**, a list of **typed relationships** (edges
|
|
between entities), and per-item **confidence tags** where applicable.
|
|
|
|
The schema is intentionally minimal. It exists so that:
|
|
1. Per-model output files are **diffable entity-by-entity and edge-by-edge** across tiers.
|
|
2. The same schema is used unchanged when local Ollama models are later scored against this
|
|
reference set — making the comparison apples-to-apples.
|
|
|
|
Subagents receive only the raw note text plus this spec. No repository files, no vault structure,
|
|
no project context. If any such context is injected by the environment, it must be ignored.
|
|
|
|
---
|
|
|
|
## Facet Vocabulary (closed)
|
|
|
|
Notes in the SecondBrain vault carry six flat, namespaced facets plus one scope tag. These are the
|
|
**only** valid values for the optional `facet` field on entities. Do not invent new namespaces.
|
|
|
|
| Prefix | Meaning |
|
|
|--------------|-----------------------------------------|
|
|
| `type/` | What kind of thing the note is about |
|
|
| `client/` | A client identity |
|
|
| `project/` | A project name |
|
|
| `domain/` | A knowledge or technical domain |
|
|
| `tool/` | A specific tool, library, or CLI |
|
|
| `convention/`| A working convention or practice |
|
|
| `scope/` | Applicability scope (cross-cutting tag) |
|
|
|
|
An entity should carry a `facet` value only when the note text directly supports mapping it to one
|
|
of the above. Absence of a `facet` field is correct when no mapping is warranted.
|
|
|
|
---
|
|
|
|
## Output Schema
|
|
|
|
The required output format. Every subagent MUST emit this structure and no other top-level keys.
|
|
|
|
```yaml
|
|
# --- Graphify extraction fragment ---
|
|
# One block per fixture note. Repeat this structure for each note.
|
|
|
|
note_slug: <kebab-case-identifier-for-the-note> # operator fills this in at run time
|
|
|
|
entities:
|
|
- name: <string> # exact or near-exact name as it appears in the note
|
|
type: <string> # e.g. Person, Tool, Project, Concept, Convention, Client, Domain
|
|
facet: <string> # OPTIONAL — must be one of the seven prefixes above, e.g. "tool/graphify"
|
|
confidence: <string> # OPTIONAL — omit if the entity is directly stated
|
|
# Values: INFERRED | AMBIGUOUS
|
|
|
|
relationships:
|
|
- source: <entity name> # must match a name in the entities list above
|
|
type: <string> # free-text verb phrase, e.g. "uses", "depends_on", "implements", "replaces"
|
|
target: <entity name> # must match a name in the entities list above
|
|
confidence: <string> # OPTIONAL — omit if the relationship is directly stated
|
|
# Values: INFERRED | AMBIGUOUS
|
|
```
|
|
|
|
### Confidence tag semantics
|
|
|
|
| Tag | Meaning |
|
|
|------------|--------------------------------------------------------------------------------------|
|
|
| _(absent)_ | The entity or relationship is **directly stated** in the note text. |
|
|
| `INFERRED` | Not stated literally but **reasonably deducible** from the note text alone. |
|
|
| `AMBIGUOUS`| Supportable by the text but **uncertain or admits multiple readings**. |
|
|
|
|
### Rules
|
|
|
|
- **Entities only:** extract entities that are named (not vague category references).
|
|
- **No invented names:** entity names must be grounded in the note text.
|
|
- **Relationship types:** free-text verb phrases; do not normalize to a fixed vocabulary.
|
|
- **Facet mapping:** use the closed vocabulary above; only assign when the note text supports it.
|
|
- **No extra keys:** do not add summaries, scores, embeddings, or metadata fields.
|
|
- **One block per note:** if processing multiple fixture notes, emit one `note_slug:` block per note,
|
|
separated by `---`.
|