jared/cc-os
Watch
1
0
Fork
You've already forked cc-os
 0
Code Issues 378 Pull requests Projects Releases Packages Wiki Activity

Add an exact wikilink and frontmatter layer to the vault graph #1545

New issue
Closed
opened 2026-11-04 21:25:59 +00:00 by jared · 4 comments
jared commented 2026-11-04 21:25:59 +00:00
Owner
Copy link

Problem

LLM extraction drops most links that the vault notes already state. The vault graph therefore misses exact relationships that need no model.

The planned Outcome Engine needs edges from an outcome node to the notes it used (docs/decision-system/vision.md). A graph that omits real links cannot give those edges reliably.

Observed

An eval on 2026-11-04 measured recall on 103 real wikilinks in 40 notes. Results are in plugins/os-vault/eval/graph-extraction/results/README.md on branch fix-vault-rebuild-stale-lock.

Model Wikilink recall
qwen3.5:9b, thinking off 28.2%
gemma4-e4b-16k 17.5%
qwen25-coder-7b-16k 1.0%

The models also disagreed. Of 426 claims, 94.4% came from one model, 5.6% came from some models, and 0 came from all models.

graphify extract sends .md files to the LLM. It does not run the deterministic Markdown extractor (graphify/extractors/markdown.py) on them. That extractor reads frontmatter, headings, and wikilinks. It runs through watch._rebuild_code, which graphify update and the git hooks use. It skips docs that already have semantic nodes.

Reproduce

n/a: this is a design gap. The eval README gives the measurement.

Approach

Goal: each real wikilink becomes an edge in the vault graph with no model call. The LLM layer adds only concepts that the notes do not state as links.

Open question, answered: plain graphify update adds no edges, because it skips docs that have semantic nodes (graphify 0.9.102 watch.py:2507-2631). graphify update --force adds them. It runs _reconcile_markdown_links (watch.py:785-900), which moves each authored link onto the semantic node of the note. An experiment on a copied sample confirmed this, with hand-made semantic nodes and AST target pages. The case of a target note with only semantic nodes needs confirmation on a real extraction graph.

Design: run graphify's own Markdown pass after the LLM pass. Add graphify update "$VAULT_PATH" --force to _REBUILD_SCRIPT in plugins/os-vault/hooks/session_start.py, and chain it with &&. Each rebuild derives the edges again, so a rebuild keeps them. Do not write a new wikilink parser.

Verify two facts before you commit the order of the steps:

  • graphify update has no code path to a model. The rebuild script comment warns that cluster-only can select a cloud backend from an API key. If update can do the same, stop and report it.
  • update --force keeps the community data from cluster-only. If it does not, or if GRAPH_REPORT.md omits the new edges, run update before cluster-only.

Frontmatter stays a node attribute. Wikilinks in frontmatter become edges. Facet-tag edges are out of scope.

Depends on #1540. _REBUILD_SCRIPT as changed here and the eval harness exist only on branch fix-vault-rebuild-stale-lock. #1540 forbids a merge of that branch alone. Build on that branch and do not merge to main before #1540 ships. ~/.cache/graphify/vault-rebuild.lock is a deliberate hold lock: leave it in place.

Measure with wikilink recall from the eval harness. Do not use hand labels.

Acceptance criteria

  1. Given a graph with semantic nodes, when the rebuild script runs, then each wikilink whose target note exists has an edge between a node of the source note and a node of the target note.
  2. The update step makes no model call.
  3. A wikilink with a missing target adds no node and no edge.
  4. A wikilink in frontmatter becomes an edge.
  5. Each LLM node and each connected pair stays in graph.json after the step, and community data is present. Graphify may merge a semantic node into an AST heading node with the same file and label, and a generic references relation may lose to a specific relation on the same pair (build.py:233-250, build.py:1757-1884).
  6. If the update step fails, the rebuild writes no stamp, and the log states the failure.
  7. plugins/os-vault/eval/graph-extraction/bin/wikilink_recall.rb reads a graph.json and the sample notes with no LLM. It scores a link as found when any edge joins a node of the source note to a node of the target note. On the live vault graph after the rebuild script runs (not on a copy, which re-scans every note), with link targets resolved against that vault, it reports at least 95% recall on the 103 wikilinks of the 40-note sample. The 28.2% baseline used label matching, so it is not the same measure. Only 14 of the 103 links have a target inside the 40-note sample, so a graph of the sample alone cannot reach 95%.
  8. graphify query, graphify path, and graphify explain run on the resulting graph.

Owner sketch

Owner: _REBUILD_SCRIPT in plugins/os-vault/hooks/session_start.py. Language: Python holding a bash string. Caller: the detached rebuild that main() starts at SessionStart.

graphify extract "$VAULT_PATH" --backend ollama --model "$MODEL" --token-budget 4000 --max-concurrency 1 &&
graphify cluster-only "$VAULT_PATH" --backend ollama --model "$MODEL" &&
graphify update "$VAULT_PATH" --force

New file: plugins/os-vault/eval/graph-extraction/bin/wikilink_recall.rb. Language: Ruby, as the other scripts in that directory. It takes the wikilink regex and note walk from the det block of bin/compare.rb, and it prints micro and macro recall. It counts connected note pairs, not relation names, because a specific LLM edge replaces a generic references edge on the same pair (build.py:241). Caller: the harness README.

Tests: plugins/os-vault/tests/test_session_start_rebuild.py and the fixture shim args (plugins/os-vault/tests/generate-fixtures.sh).

Non-goals

Choosing the extraction model. #1540 covers that.

Origin

  • Trigger: plugins/os-vault/eval/graph-extraction run on 2026-11-04
  • Improvised this session: none
  • Chain: low wikilink recall ← Markdown goes only to the LLM ← graphify extract routes docs to the semantic pass (graphify 0.9.102 cli.py:4356) ← UPSTREAM-TOOL
  • Root candidate: none found; searched wikilink, frontmatter, graphify, vault graph, os-vault, deterministic, extract
  • Where: plugins/os-vault/hooks/session_start.py
  • Session: 7cfd6b00-f7a4-4761-8bd0-4312d5e6142d
  • Transcript: /home/jared/.claude/projects/-home-jared-Documents-SecondBrain/7cfd6b00-f7a4-4761-8bd0-4312d5e6142d.jsonl

VERDICT: CREATE

## Problem LLM extraction drops most links that the vault notes already state. The vault graph therefore misses exact relationships that need no model. The planned Outcome Engine needs edges from an outcome node to the notes it used (`docs/decision-system/vision.md`). A graph that omits real links cannot give those edges reliably. ## Observed An eval on 2026-11-04 measured recall on 103 real wikilinks in 40 notes. Results are in `plugins/os-vault/eval/graph-extraction/results/README.md` on branch `fix-vault-rebuild-stale-lock`. | Model | Wikilink recall | |---|---| | `qwen3.5:9b`, thinking off | 28.2% | | `gemma4-e4b-16k` | 17.5% | | `qwen25-coder-7b-16k` | 1.0% | The models also disagreed. Of 426 claims, 94.4% came from one model, 5.6% came from some models, and 0 came from all models. `graphify extract` sends `.md` files to the LLM. It does not run the deterministic Markdown extractor (`graphify/extractors/markdown.py`) on them. That extractor reads frontmatter, headings, and wikilinks. It runs through `watch._rebuild_code`, which `graphify update` and the git hooks use. It skips docs that already have semantic nodes. ## Reproduce n/a: this is a design gap. The eval README gives the measurement. ## Approach Goal: each real wikilink becomes an edge in the vault graph with no model call. The LLM layer adds only concepts that the notes do not state as links. Open question, answered: plain `graphify update` adds no edges, because it skips docs that have semantic nodes (graphify 0.9.102 `watch.py:2507-2631`). `graphify update --force` adds them. It runs `_reconcile_markdown_links` (`watch.py:785-900`), which moves each authored link onto the semantic node of the note. An experiment on a copied sample confirmed this, with hand-made semantic nodes and AST target pages. The case of a target note with only semantic nodes needs confirmation on a real extraction graph. Design: run graphify's own Markdown pass after the LLM pass. Add `graphify update "$VAULT_PATH" --force` to `_REBUILD_SCRIPT` in `plugins/os-vault/hooks/session_start.py`, and chain it with `&&`. Each rebuild derives the edges again, so a rebuild keeps them. Do not write a new wikilink parser. Verify two facts before you commit the order of the steps: - `graphify update` has no code path to a model. The rebuild script comment warns that `cluster-only` can select a cloud backend from an API key. If `update` can do the same, stop and report it. - `update --force` keeps the community data from `cluster-only`. If it does not, or if `GRAPH_REPORT.md` omits the new edges, run `update` before `cluster-only`. Frontmatter stays a node attribute. Wikilinks in frontmatter become edges. Facet-tag edges are out of scope. Depends on #1540. `_REBUILD_SCRIPT` as changed here and the eval harness exist only on branch `fix-vault-rebuild-stale-lock`. #1540 forbids a merge of that branch alone. Build on that branch and do not merge to main before #1540 ships. `~/.cache/graphify/vault-rebuild.lock` is a deliberate hold lock: leave it in place. Measure with wikilink recall from the eval harness. Do not use hand labels. ## Acceptance criteria 1. Given a graph with semantic nodes, when the rebuild script runs, then each wikilink whose target note exists has an edge between a node of the source note and a node of the target note. 2. The `update` step makes no model call. 3. A wikilink with a missing target adds no node and no edge. 4. A wikilink in frontmatter becomes an edge. 5. Each LLM node and each connected pair stays in `graph.json` after the step, and community data is present. Graphify may merge a semantic node into an AST heading node with the same file and label, and a generic `references` relation may lose to a specific relation on the same pair (`build.py:233-250`, `build.py:1757-1884`). 6. If the `update` step fails, the rebuild writes no stamp, and the log states the failure. 7. `plugins/os-vault/eval/graph-extraction/bin/wikilink_recall.rb` reads a `graph.json` and the sample notes with no LLM. It scores a link as found when any edge joins a node of the source note to a node of the target note. On the live vault graph after the rebuild script runs (not on a copy, which re-scans every note), with link targets resolved against that vault, it reports at least 95% recall on the 103 wikilinks of the 40-note sample. The 28.2% baseline used label matching, so it is not the same measure. Only 14 of the 103 links have a target inside the 40-note sample, so a graph of the sample alone cannot reach 95%. 8. `graphify query`, `graphify path`, and `graphify explain` run on the resulting graph. ## Owner sketch Owner: `_REBUILD_SCRIPT` in `plugins/os-vault/hooks/session_start.py`. Language: Python holding a bash string. Caller: the detached rebuild that `main()` starts at SessionStart. ```bash graphify extract "$VAULT_PATH" --backend ollama --model "$MODEL" --token-budget 4000 --max-concurrency 1 && graphify cluster-only "$VAULT_PATH" --backend ollama --model "$MODEL" && graphify update "$VAULT_PATH" --force ``` New file: `plugins/os-vault/eval/graph-extraction/bin/wikilink_recall.rb`. Language: Ruby, as the other scripts in that directory. It takes the wikilink regex and note walk from the `det` block of `bin/compare.rb`, and it prints micro and macro recall. It counts connected note pairs, not relation names, because a specific LLM edge replaces a generic `references` edge on the same pair (`build.py:241`). Caller: the harness README. Tests: `plugins/os-vault/tests/test_session_start_rebuild.py` and the fixture shim args (`plugins/os-vault/tests/generate-fixtures.sh`). ## Non-goals Choosing the extraction model. #1540 covers that. ## Origin - Trigger: `plugins/os-vault/eval/graph-extraction` run on 2026-11-04 - Improvised this session: none - Chain: low wikilink recall ← Markdown goes only to the LLM ← `graphify extract` routes docs to the semantic pass (graphify 0.9.102 `cli.py:4356`) ← UPSTREAM-TOOL - Root candidate: none found; searched wikilink, frontmatter, graphify, vault graph, os-vault, deterministic, extract - Where: `plugins/os-vault/hooks/session_start.py` - Session: 7cfd6b00-f7a4-4761-8bd0-4312d5e6142d - Transcript: /home/jared/.claude/projects/-home-jared-Documents-SecondBrain/7cfd6b00-f7a4-4761-8bd0-4312d5e6142d.jsonl VERDICT: CREATE
jared commented 2026-11-04 21:33:22 +00:00
Author
Owner
Copy link

Work started on branch ticket-1545.

Work started on branch ticket-1545.
jared commented 2026-11-04 21:46:04 +00:00
Author
Owner
Copy link

Resolution

Done: Added graphify update --force as the third step of the vault rebuild and added bin/wikilink_recall.rb. Commits cc6e38f3 and b8335e06 on branch ticket-1545, not merged.

Evidence: Recall 100/103 (97.1%) on a full-vault copy; 100/100 of links with a target; 3 links have no target note. os-vault pytest: 12 passed. update ran with no network under unshare. AC3 and AC4 checked on a copy. A real rebuild is still unmeasured because the hold lock blocks rebuilds.

Follow-ups: Repeatable AC3 and AC4 tests: dropped, they would test graphify. Title-form links: dropped, vault content. Full LLM rebuild measure: covered by the #1540 rebuild.

Needs review: Merge branch ticket-1545 only after #1540 ships. #1540 forbids a merge of fix-vault-rebuild-stale-lock alone, and ticket-1545 contains that branch.

## Resolution **Done:** Added graphify update --force as the third step of the vault rebuild and added bin/wikilink_recall.rb. Commits cc6e38f3 and b8335e06 on branch ticket-1545, not merged. **Evidence:** Recall 100/103 (97.1%) on a full-vault copy; 100/100 of links with a target; 3 links have no target note. os-vault pytest: 12 passed. update ran with no network under unshare. AC3 and AC4 checked on a copy. A real rebuild is still unmeasured because the hold lock blocks rebuilds. **Follow-ups:** Repeatable AC3 and AC4 tests: dropped, they would test graphify. Title-form links: dropped, vault content. Full LLM rebuild measure: covered by the #1540 rebuild. Needs review: Merge branch ticket-1545 only after #1540 ships. #1540 forbids a merge of fix-vault-rebuild-stale-lock alone, and ticket-1545 contains that branch.
jared commented 2026-11-05 14:39:03 +00:00
Author
Owner
Copy link

Correction to the recorded evidence, session b10276a0. The 97.1% recall came from a vault copy. The copy treated all 336 notes as new, so it does not represent the real rebuild. On the live vault, graphify update --force skipped notes that have semantic nodes and extracted 12 files. Live recall is 40.8%, up from 18.45%. AC7 is therefore not met on the live graph. Copy: 3734 nodes, 3814 edges. Live: 1583 nodes, 1589 edges. Live results by the fixed harness: linked pairs found 21 of 30 (baseline 9), original questions 12 of 12 (baseline 9), reworded questions 8 of 13 (no change). The copy's heading nodes lowered the rank on 4 questions. The live graph has no regression. The live step is applied to the graph only; no branch merged. Before this merges, the design must state how to reach full recall on a graph with semantic nodes without the copy's heading noise.

Correction to the recorded evidence, session b10276a0. The 97.1% recall came from a vault copy. The copy treated all 336 notes as new, so it does not represent the real rebuild. On the live vault, `graphify update --force` skipped notes that have semantic nodes and extracted 12 files. Live recall is 40.8%, up from 18.45%. AC7 is therefore not met on the live graph. Copy: 3734 nodes, 3814 edges. Live: 1583 nodes, 1589 edges. Live results by the fixed harness: linked pairs found 21 of 30 (baseline 9), original questions 12 of 12 (baseline 9), reworded questions 8 of 13 (no change). The copy's heading nodes lowered the rank on 4 questions. The live graph has no regression. The live step is applied to the graph only; no branch merged. Before this merges, the design must state how to reach full recall on a graph with semantic nodes without the copy's heading noise.
jared commented 2026-11-05 16:14:05 +00:00
Author
Owner
Copy link

Body edit, session b10276a0: AC7 now says the live vault graph after the rebuild script, not a copy. A copy re-scans every note and gave the false 97.1% (see the correction above). Fix on branch ticket-1545, commits c284fa56 and fe3d6555 (not merged): stub_markdown_pages.py adds a page node to each note with no unique representative node; rebuild order is extract, cluster-only, stub step, update --force, update --force. Applied to the live graph. Live result: 100/103 (97.1%) on the 103-link sample, 660/687 (96.1%) on the whole vault, 660/663 (99.5%) of links whose target note exists. Graph 1583 to 1686 nodes, 1589 to 1814 edges, 330 communities, no old node or connected pair lost. Not merged.

Body edit, session b10276a0: AC7 now says the live vault graph after the rebuild script, not a copy. A copy re-scans every note and gave the false 97.1% (see the correction above). Fix on branch ticket-1545, commits c284fa56 and fe3d6555 (not merged): stub_markdown_pages.py adds a page node to each note with no unique representative node; rebuild order is extract, cluster-only, stub step, update --force, update --force. Applied to the live graph. Live result: 100/103 (97.1%) on the 103-link sample, 660/687 (96.1%) on the whole vault, 660/663 (99.5%) of links whose target note exists. Graph 1583 to 1686 nodes, 1589 to 1814 edges, 330 communities, no old node or connected pair lost. Not merged.
jared closed this issue 2026-11-05 20:28:26 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
ticket-1337
No results found.
Labels
Clear labels   
P0
Priority 0 — drop everything

  
P1
Priority 1 — next up

  
P2
Priority 2 — normal

  
P3
Priority 3 — someday

  
bug
Something is broken

  
category/code-change
Ticket category

  
category/cross-repo
Ticket category

  
category/eval-run
Ticket category

  
category/investigation
Ticket category

  
category/no-test-target
Ticket category

  
category/owner-ruling
Ticket category

  
category/prose-artifact
Ticket category

  
category/recurring-chore
Ticket category

  
category/structure

   
category/umbrella
Ticket category

  
crc-refactor
CRC refactor run work record; exempt from the ticket-skeptic

  
create
Ticket creates something new

  
delete
Ticket removes something

  
enhancement
New feature or improvement

  
filed-by/agent

   
filed-by/user

   
frozen
Pre-reset backlog freeze 2026-08-16; do not work unless rediscovered by a live run

  
lint-rule
Ticket proposes/modifies a lint rule or cop

  
needs-info
Waiting on reporter for more information

  
needs-triage
Maintainer needs to evaluate this issue

  
next
Human-curated next up — AI never applies or removes

  
plugin/cc-architect
Scope: the cc-architect plugin

  
plugin/os
Scope: the os plugin

  
plugin/os-adr
Scope: the os-adr plugin

  
plugin/os-aidd-lint
Scope: the os-aidd-lint plugin

  
plugin/os-backlog
Scope: the os-backlog plugin

  
plugin/os-context
Scope: the os-context plugin

  
plugin/os-doc-hygiene
Scope: the os-doc-hygiene plugin

  
plugin/os-sdlc
Scope: the os-sdlc plugin

  
plugin/os-vault
Scope: the os-vault plugin

  
project/cc-os
Parent-scope: cc-os repo-wide, not specific to one plugin

  
ready-for-agent
Triage complete; an agent can pick this up

  
ready-for-human
Human-owned; never picked up autonomously

  
recurring
Recurring flavor: occurrence outcomes commented, never closed by AI

  
review
Shipped semi work awaiting human sign-off

  
rule-proposal
A rule change that survived the gate; Jared reads these when he chooses

  
update
Ticket updates something existing

  
waiting
Blocked; blocker named in a comment

  
wayfinder:grilling

   
wayfinder:map

   
wayfinder:map
wayfinder map ticket

  
wayfinder:research

   
wayfinder:task

   
wayfinder:task
wayfinder map child: task

  
wontfix
Will not be actioned

  
worklist/deviations
Deviation incidents from skill/pipeline runs — evidence pile for correcting the owning artifact

  
worklist/lint-rule
Proposed os-sdlc or os-aidd-lint rule

  
worklist/new-implement-build
Updated implementation harness (ast-grep + stepper)

No labels
P0
P1
P2
P3
bug
category/code-change
category/cross-repo
category/eval-run
category/investigation
category/no-test-target
category/owner-ruling
category/prose-artifact
category/recurring-chore
category/structure
category/umbrella
crc-refactor
create
delete
enhancement
filed-by/agent
filed-by/user
frozen
lint-rule
needs-info
needs-triage
next
plugin/cc-architect
plugin/os
plugin/os-adr
plugin/os-aidd-lint
plugin/os-backlog
plugin/os-context
plugin/os-doc-hygiene
plugin/os-sdlc
plugin/os-vault
project/cc-os
ready-for-agent
ready-for-human
recurring
review
rule-proposal
update
waiting
wayfinder:grilling
wayfinder:map
wayfinder:map
wayfinder:research
wayfinder:task
wayfinder:task
wontfix
worklist/deviations
worklist/lint-rule
worklist/new-implement-build
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
jared/cc-os#1545
No description provided.