Keep heading edges when a re-extraction replaces a note's node #1577

New issue

Open

opened 2026-11-05 21:12:29 +00:00 by jared · 1 comment

jared commented

2026-11-05 21:12:29 +00:00

Owner

Problem

A re-extraction deletes the old node of a note. In 12 notes that old node was the parent of the Markdown contains edges to the headings. The edges go with the node, so correct structure is lost, and the note's new nodes do not join its page node.

Observed

The V4 re-extraction of 2026-11-05 removed 95 contains edges and 12 parent nodes. 87 headings and 13 page nodes became isolated. Example: 2026-03-13-oo-principles-plugin-concept-design-recommendations.md has 9 headings at degree 0. A fresh parse of that note gives 11 edges. The two graphify update --force steps did not repair them. A what-if in memory: restoring the 95 edges gives 212 isolated nodes and 59.7% largest component, against 305 and 53.7%. Full analysis: plugins/os-vault/eval/graph-retrieval/diff-reextract.md sections 4 and 5.

Reproduce

n/a: the state is in the live vault graph; see the report.

Approach

Chain correction: the cause is the node replace in graphify build.py:2836-2848, not update --force skipping notes. Repair stays in os-vault because graphify is outside this repo.

Required work:

extract_page in plugins/os-vault/hooks/stub_markdown_pages.py also returns each heading and its parent from the Markdown parse.
stub_pages keeps its signature and re-adds a missing contains edge from the parsed parent, or from the page node when the parent id is absent from the graph.
Normalize parse ids: remove the home_jared_documents_secondbrain_ prefix and the page _md form, to match graph ids. Set _origin: ast and the note's relative source_file on each edge.
Match an old heading by source_file and L<n>, because old headings have no node_kind.

Acceptance criteria

On a copy of today's graph.json, after the stub step and two forced updates, the 87 listed heading ids each have degree 1 or more.
Isolated document nodes go from 184 to 97 or fewer, and contains edges go from 157 to 244.
The 24 isolated page nodes and 13 unmatched headings do not change.
A third run writes a byte-identical graph.json.
A unit test covers a heading with its parent in the graph and a heading with its parent absent.
The repair needs no model and no GPU. Run it on a copy first, and judge by answers: report any change in the 12 questions.

Owner sketch

stub_markdown_pages.py owns graph repair after extract. No new object. Keep stub_pages(graph, canonical_id_of, extract_page).

Origin

Trigger: investigation of score drops after the V4 re-extraction, 2026-11-05
Improvised this session: none
Chain: headings isolated ← parent node deleted on replace ← update --force skips notes with semantic nodes ← UPSTREAM-TOOL (graphify 0.9.102)
Root candidate: none found; searched stub_markdown_pages, contains edge, heading, isolated, re-extract
Where: plugins/os-vault/hooks/stub_markdown_pages.py
Session: 47ffa777-a9a8-4ac8-808a-b9b19489944a
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/47ffa777-a9a8-4ac8-808a-b9b19489944a.jsonl

VERDICT: CREATE

## Problem A re-extraction deletes the old node of a note. In 12 notes that old node was the parent of the Markdown `contains` edges to the headings. The edges go with the node, so correct structure is lost, and the note's new nodes do not join its page node. ## Observed The V4 re-extraction of 2026-11-05 removed 95 `contains` edges and 12 parent nodes. 87 headings and 13 page nodes became isolated. Example: `2026-03-13-oo-principles-plugin-concept-design-recommendations.md` has 9 headings at degree 0. A fresh parse of that note gives 11 edges. The two `graphify update --force` steps did not repair them. A what-if in memory: restoring the 95 edges gives 212 isolated nodes and 59.7% largest component, against 305 and 53.7%. Full analysis: `plugins/os-vault/eval/graph-retrieval/diff-reextract.md` sections 4 and 5. ## Reproduce n/a: the state is in the live vault graph; see the report. ## Approach Chain correction: the cause is the node replace in graphify `build.py:2836-2848`, not `update --force` skipping notes. Repair stays in os-vault because graphify is outside this repo. Required work: 1. `extract_page` in `plugins/os-vault/hooks/stub_markdown_pages.py` also returns each heading and its parent from the Markdown parse. 2. `stub_pages` keeps its signature and re-adds a missing `contains` edge from the parsed parent, or from the page node when the parent id is absent from the graph. 3. Normalize parse ids: remove the `home_jared_documents_secondbrain_` prefix and the page `_md` form, to match graph ids. Set `_origin: ast` and the note's relative `source_file` on each edge. 4. Match an old heading by `source_file` and `L<n>`, because old headings have no `node_kind`. ## Acceptance criteria 1. On a copy of today's `graph.json`, after the stub step and two forced updates, the 87 listed heading ids each have degree 1 or more. 2. Isolated `document` nodes go from 184 to 97 or fewer, and `contains` edges go from 157 to 244. 3. The 24 isolated page nodes and 13 unmatched headings do not change. 4. A third run writes a byte-identical `graph.json`. 5. A unit test covers a heading with its parent in the graph and a heading with its parent absent. 6. The repair needs no model and no GPU. Run it on a copy first, and judge by answers: report any change in the 12 questions. ## Owner sketch `stub_markdown_pages.py` owns graph repair after extract. No new object. Keep `stub_pages(graph, canonical_id_of, extract_page)`. ## Origin - Trigger: investigation of score drops after the V4 re-extraction, 2026-11-05 - Improvised this session: none - Chain: headings isolated ← parent node deleted on replace ← `update --force` skips notes with semantic nodes ← UPSTREAM-TOOL (graphify 0.9.102) - Root candidate: none found; searched stub_markdown_pages, contains edge, heading, isolated, re-extract - Where: plugins/os-vault/hooks/stub_markdown_pages.py - Session: 47ffa777-a9a8-4ac8-808a-b9b19489944a - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/47ffa777-a9a8-4ac8-808a-b9b19489944a.jsonl VERDICT: CREATE

jared added the

plugin/os-vault

filed-by/agent

category/code-change

labels

2026-11-05 21:12:29 +00:00

jared referenced this issue

2026-11-05 21:12:30 +00:00

Finish the vault graph rebuild after the graphify update #1540

jared added the

ready-for-agent

label

2026-11-06 14:30:03 +00:00

jared referenced this issue

2026-11-06 20:27:05 +00:00

Run pytest targets in the diff-test gate #1625

jared commented

2026-11-06 21:30:54 +00:00

Author

Owner

Deferred, ruling 71. After #1571 no question fails because of lost heading edges: 12/12 original pass, and the five reworded misses fail at start nodes on word mismatch. Reopen the work only if a question fails from it.

jared referenced this issue from a commit

2026-11-06 21:33:05 +00:00

docs(os-vault): record the #1571 scores and the rulings on #1568, #1570, #1577