jared/cc-os

Watch

Fork 0

Merge duplicate concept nodes and stop wikilink stubs taking the citing note as source #1568

New issue

Closed

opened 2026-11-05 14:19:08 +00:00 by jared · 2 comments

jared commented

2026-11-05 14:19:08 +00:00

Owner

Problem

The vault graph holds duplicate concept nodes that graphify did not merge. It also holds stub nodes for wikilink targets that graphify attributes to the citing note, not to the target note. Both defects hide the right answer.
Evidence: /tmp/claude-1000/-home-jared-Documents-SecondBrain/b10276a0-3582-48b5-b0fe-f13795bdcc75/scratchpad/vault-graph-weak-retrieval-evidence.md and /tmp/claude-1000/-home-jared-Documents-SecondBrain/8a73b51f-d8a3-4dfc-9de7-e72ea77eacac/scratchpad/graph-test-results.md (copied below; do not depend on the paths).

Observed

Question 4: "Graceful Rate Limits Standard" and "Graceful Rate Limits Automation Standard" are two nodes in different notes. Jaro score is 89.7 against a merge threshold of 92 (dedup.py:273; cross-file labels of 12 or more characters use plain Jaro, dedup.py:1441-1443). They did not merge. The real link (user-guide/rubyllm-429-retry-after-cap.md line 57, "conflicts with graceful-rate-limits-automation-standard") sits on the duplicate. A path call gave a spurious 4-hop route through unrelated nodes.
Other duplicates: 85 near-duplicate pairs (token_set 92 or more and ratio 80 or more), for example hyperthrive dev and hyperthrive dev hub; sequel model and sequel model model; smartlead api and smartlead api hub; herdr cli and herdr cli gem. Exact normalized label duplicates: 45 groups, 99 nodes (54 redundant, 3.5%). Of 50 sampled nodes, 4 (8%) were duplicates.
Stubs: 74 nodes carry graphify_reference_stub. The extraction prompt (llm.py:542-552) says to model a referenced file as a node attributed to the enclosing file. Stub "RuboCop Todo Exclude Masks Explicit Paths" has source rubocop-monorepo-config-discovery.md.
Question 6: no node has source reference/gitignore-trailing-slash-does-not-match-symlinks.md. A dangling stub "Git ignore trailing slash does not match symlinks" is attributed to reference/git-worktree-ignores-per-worktree-info-exclude.md and claude-code-worktrees-hub.md.
Question 11: the answer note rubocop-todo-exclude-masks-explicit-paths.md is in the graph but not returned. The returned start node is the stub. The answer file name never appears.
Wikilinks: 668 unique (note, target) pairs; 647 resolve to another note. Edges reproduce 166 (26%) strictly and 320 (49%) if a stub counts. No deterministic wikilink edges exist: all 718 references edges come from the LLM.

Reproduce

cd /home/jared/Documents/SecondBrain && graphify query "rubocop todo exclude masks explicit paths"

The start node is the stub, not the note.

Approach

Merge each duplicate pair into one node, and keep the edges of both. Attribute a wikilink stub to its target note, or replace it with an edge to the target note's own node. Do both in the vault build or in a step after the build. Do not rely on a lower threshold alone, because prefix-extension pairs are blocked by design (dedup.py:1456).

Required work: none beyond the acceptance criteria.

Orchestrator work: rebuild the vault graph and re-run questions 4, 6 and 11.

Acceptance criteria

Given the rebuilt vault graph, when a reader looks for "Graceful Rate Limits Standard" and "Graceful Rate Limits Automation Standard", then one node holds both and the conflicts_with edge from the RubyLLM note is on it.
Given a wikilink to an existing note, when the graph is built, then no stub node for that link has the citing note as its source.
Given question 11, when the query runs, then the output names rubocop-todo-exclude-masks-explicit-paths.md.

Non-goals

Graphify upstream fixes (see the upstream report ticket). Missing-note indexing (see the missing-nodes ticket).

Origin

Trigger: vault graph test of 12 questions, 2026-11-04 (questions 4, 6, 11 PARTIAL).
Improvised this session: ran path --undirected to see the spurious route.
Chain: question 4, 6 and 11 give partial results ← duplicate and stub nodes shadow the real nodes ← dedup threshold 92 and the stub rule in the extraction prompt ← UPSTREAM-TOOL (graphify 0.9.102, dedup.py:273, llm.py:542-552).
Root candidate: none found; searched backlog terms for this ticket (VERDICT: CREATE)
Where: graphify 0.9.102 dedup.py and llm.py; vault graph build
Session: b10276a0-3582-48b5-b0fe-f13795bdcc75
Transcript: /home/jared/.claude/projects/-home-jared-Documents-SecondBrain/b10276a0-3582-48b5-b0fe-f13795bdcc75.jsonl
Parent: #1540.

## Problem The vault graph holds duplicate concept nodes that graphify did not merge. It also holds stub nodes for wikilink targets that graphify attributes to the citing note, not to the target note. Both defects hide the right answer. Evidence: /tmp/claude-1000/-home-jared-Documents-SecondBrain/b10276a0-3582-48b5-b0fe-f13795bdcc75/scratchpad/vault-graph-weak-retrieval-evidence.md and /tmp/claude-1000/-home-jared-Documents-SecondBrain/8a73b51f-d8a3-4dfc-9de7-e72ea77eacac/scratchpad/graph-test-results.md (copied below; do not depend on the paths). ## Observed - Question 4: "Graceful Rate Limits Standard" and "Graceful Rate Limits Automation Standard" are two nodes in different notes. Jaro score is 89.7 against a merge threshold of 92 (`dedup.py:273`; cross-file labels of 12 or more characters use plain Jaro, `dedup.py:1441-1443`). They did not merge. The real link (`user-guide/rubyllm-429-retry-after-cap.md` line 57, "conflicts with [[graceful-rate-limits-automation-standard]]") sits on the duplicate. A path call gave a spurious 4-hop route through unrelated nodes. - Other duplicates: 85 near-duplicate pairs (token_set 92 or more and ratio 80 or more), for example hyperthrive dev and hyperthrive dev hub; sequel model and sequel model model; smartlead api and smartlead api hub; herdr cli and herdr cli gem. Exact normalized label duplicates: 45 groups, 99 nodes (54 redundant, 3.5%). Of 50 sampled nodes, 4 (8%) were duplicates. - Stubs: 74 nodes carry `graphify_reference_stub`. The extraction prompt (`llm.py:542-552`) says to model a referenced file as a node attributed to the enclosing file. Stub "RuboCop Todo Exclude Masks Explicit Paths" has source `rubocop-monorepo-config-discovery.md`. - Question 6: no node has source `reference/gitignore-trailing-slash-does-not-match-symlinks.md`. A dangling stub "Git ignore trailing slash does not match symlinks" is attributed to `reference/git-worktree-ignores-per-worktree-info-exclude.md` and `claude-code-worktrees-hub.md`. - Question 11: the answer note `rubocop-todo-exclude-masks-explicit-paths.md` is in the graph but not returned. The returned start node is the stub. The answer file name never appears. - Wikilinks: 668 unique (note, target) pairs; 647 resolve to another note. Edges reproduce 166 (26%) strictly and 320 (49%) if a stub counts. No deterministic wikilink edges exist: all 718 `references` edges come from the LLM. ## Reproduce ``` cd /home/jared/Documents/SecondBrain && graphify query "rubocop todo exclude masks explicit paths" ``` The start node is the stub, not the note. ## Approach Merge each duplicate pair into one node, and keep the edges of both. Attribute a wikilink stub to its target note, or replace it with an edge to the target note's own node. Do both in the vault build or in a step after the build. Do not rely on a lower threshold alone, because prefix-extension pairs are blocked by design (`dedup.py:1456`). Required work: none beyond the acceptance criteria. Orchestrator work: rebuild the vault graph and re-run questions 4, 6 and 11. ## Acceptance criteria 1. Given the rebuilt vault graph, when a reader looks for "Graceful Rate Limits Standard" and "Graceful Rate Limits Automation Standard", then one node holds both and the `conflicts_with` edge from the RubyLLM note is on it. 2. Given a wikilink to an existing note, when the graph is built, then no stub node for that link has the citing note as its source. 3. Given question 11, when the query runs, then the output names `rubocop-todo-exclude-masks-explicit-paths.md`. ## Non-goals Graphify upstream fixes (see the upstream report ticket). Missing-note indexing (see the missing-nodes ticket). ## Origin - Trigger: vault graph test of 12 questions, 2026-11-04 (questions 4, 6, 11 PARTIAL). - Improvised this session: ran `path --undirected` to see the spurious route. - Chain: question 4, 6 and 11 give partial results ← duplicate and stub nodes shadow the real nodes ← dedup threshold 92 and the stub rule in the extraction prompt ← UPSTREAM-TOOL (graphify 0.9.102, dedup.py:273, llm.py:542-552). - Root candidate: none found; searched backlog terms for this ticket (VERDICT: CREATE) - Where: graphify 0.9.102 dedup.py and llm.py; vault graph build - Session: b10276a0-3582-48b5-b0fe-f13795bdcc75 - Transcript: /home/jared/.claude/projects/-home-jared-Documents-SecondBrain/b10276a0-3582-48b5-b0fe-f13795bdcc75.jsonl - Parent: #1540.

jared added the

plugin/os-vault

filed-by/user

category/code-change

labels

2026-11-05 14:19:08 +00:00

jared referenced this issue

2026-11-05 20:28:26 +00:00

Finish the vault graph rebuild after the graphify update #1540

jared referenced this issue

2026-11-06 14:30:06 +00:00

Cap or split hub notes so one note cannot crowd out query results #1570

jared commented

2026-11-06 14:30:06 +00:00

Author

Owner

/os-backlog:ticket-decompose routed this to /os-sdlc:ticket-takedown because the owner is a new object.

One change: a model-free post-pass after the second update that folds reference stubs into note page nodes and merges duplicates through an explicit list. Owner: new object beside stub_markdown_pages.py, working name fold_reference_stubs.py; its secret is the rule for what counts as the same thing. Measured on 2524 nodes: 176 stubs, all with the citing note as source, 73 naming an existing note; 46 exact-label groups, many legitimate ('Related', 'See Also', 'CC-OS'), so no threshold merge. Unverified: a post-step surviving graphify update; incremental merge leaves old notes unmerged until a full rebuild. Runs after #1571.

/os-backlog:ticket-decompose routed this to `/os-sdlc:ticket-takedown` because the owner is a new object. One change: a model-free post-pass after the second update that folds reference stubs into note page nodes and merges duplicates through an explicit list. Owner: new object beside `stub_markdown_pages.py`, working name `fold_reference_stubs.py`; its secret is the rule for what counts as the same thing. Measured on 2524 nodes: 176 stubs, all with the citing note as source, 73 naming an existing note; 46 exact-label groups, many legitimate ('Related', 'See Also', 'CC-OS'), so no threshold merge. Unverified: a post-step surviving `graphify update`; incremental merge leaves old notes unmerged until a full rebuild. Runs after #1571.

jared referenced this issue

2026-11-06 20:09:51 +00:00

Run pytest targets in the diff-test gate #1625

jared referenced this issue

2026-11-06 20:27:05 +00:00

Run pytest targets in the diff-test gate #1625

jared commented

2026-11-06 21:30:52 +00:00

Author

Owner

Resolution

Done: Closed by measurement. No fold or merge built.

Evidence: With #1571 on the current graph, Q4 (3 hops), Q6 (rank 1), Q11 (note rank 2) pass, so AC 3 already holds. All five reworded misses fail at start nodes on word mismatch: P-Q1 and S-Q9 have a duplicate or stub, but the question shares no term with it, and 4 of 5 stay absent at budget 40000. results-after-1571, rulings.tsv 69.

Follow-ups: none: the start-node gap is #1573

## Resolution **Done:** Closed by measurement. No fold or merge built. **Evidence:** With #1571 on the current graph, Q4 (3 hops), Q6 (rank 1), Q11 (note rank 2) pass, so AC 3 already holds. All five reworded misses fail at start nodes on word mismatch: P-Q1 and S-Q9 have a duplicate or stub, but the question shares no term with it, and 4 of 5 stay absent at budget 40000. results-after-1571, rulings.tsv 69. **Follow-ups:** none: the start-node gap is #1573

jared closed this issue

2026-11-06 21:30:53 +00:00

jared referenced this issue from a commit

2026-11-06 21:33:05 +00:00

docs(os-vault): record the #1571 scores and the rulings on #1568, #1570, #1577