Make the vault rebuild finish unattended when notes are hollow or old nodes are gone #1557

New issue

Open

opened 2026-11-05 01:01:40 +00:00 by jared · 0 comments

jared commented

2026-11-05 01:01:40 +00:00

Owner

Problem

The vault graph rebuild cannot finish unattended in two cases. A note can produce no nodes (a hollow note). A node from a note that was deleted or excluded on purpose can make a guard refuse the write.

Observed

Found during #1540. Data from 2026-11-04, graphify 0.9.102, model qwen3.5:9b.

The SessionStart run took 74 minutes and exited 1. graph.json stayed unchanged. The hook wrote vault-rebuild.failed.

13 of 205 notes produced no nodes. Graphify logged "chunk of N still hollow after 3 attempt(s)". Retries in a chunk returned identical output_tokens, so same-size retries appear deterministic. A smaller --token-budget recovered notes: 4000 left 13 hollow, 2000 left 4, 800 left 2.

The hook detects failure with grep -Eq "chunk [0-9]+/[0-9]+ failed" in _REBUILD_SCRIPT. That pattern matched 0 lines although notes were lost. If extract exits 0 with hollow chunks, the hook writes the stamp and the lost notes stay out until the next rebuild.

Two guards refused the write. The first is the incomplete-and-smaller guard in graphify/cli.py. The second is to_json in graphify/export.py, which calls _lost_saved_semantic_nodes with an empty deleted_sources. It listed nodes from journal/, which .graphifyignore now excludes on purpose.

Code reading (not run) shows the second guard fires on any later deleted or excluded note, even on an otherwise complete run. Each such rebuild would then fail until someone passes --allow-partial. Confirm this by a run.

The orchestrator wrote the graph by hand with --allow-partial --token-budget 800, then cluster-only, then wrote the stamp. In graphify/cli.py (_stamped_manifest_files), a file with no output has no manifest stamp, so the next run re-queues it. A file whose output shrank (unverified_semantic_sources) is excluded from stamping unless --allow-partial is set. With --allow-partial it is stamped and will not be retried until it changes. Confirm which case applies to the missing notes:

reference/gitignore-trailing-slash-does-not-match-symlinks.md
2026-06-08-graphify-ingesting-code-ast.md
2026-06-08-graphify-ingesting-docs-knowledge.md

Reproduce

graphify extract "$VAULT_PATH" --backend ollama --model qwen3.5:9b --token-budget 4000 --max-concurrency 1

Run on the 2026-11-04 vault state. Expect "still hollow after 3 attempt(s)" lines and exit 1.

Approach

The hook detects hollow notes, retries them with a smaller token budget, and handles intended exclusions without a manual flag.

Required work:

Make the hook detect "still hollow" chunks as failure — check: present still hollow in plugins/os-vault/hooks/session_start.py
Retry hollow notes with a smaller token budget (800 recovered all but 2 of 13) — check: present --token-budget 800 in plugins/os-vault/hooks/session_start.py

Orchestrator work: verify the exclusion guard and stamping behavior against graphify 0.9.102. Decide how the hook handles intended exclusions without a manual flag.

Acceptance criteria

Given an extract run that logs "still hollow" and exits 0, when the hook finishes, then it does not write the stamp.
Given hollow notes after a 4000 token run, when the hook retries, then it retries them with a smaller --token-budget.
Given a note excluded by .graphifyignore after the last graph write, when a rebuild is otherwise complete, then the graph is written without a manual flag.

Owner sketch

class Vault::HollowNoteRetry
  def retry(notes, budget)
end

Origin

Trigger: SessionStart rebuild on 2026-11-04 during #1540 (74 minutes, exit 1).
Improvised this session: wrote the graph by hand with --allow-partial --token-budget 800, then cluster-only, then wrote the stamp.
Chain: graph not written ← incomplete-and-smaller guard and _lost_saved_semantic_nodes guard refused ← 13 hollow notes and journal/ nodes now excluded ← hook retries at the same token budget and does not detect "still hollow" (_REBUILD_SCRIPT). DESIGN; graphify 0.9.102 for the guards (UPSTREAM-TOOL).
Root candidate: none found; searched hollow, _REBUILD_SCRIPT, graphify, vault-rebuild, session_start, allow-partial, lost_saved
Where: plugins/os-vault/hooks/session_start.py (_REBUILD_SCRIPT)
Session: 8a73b51f-d8a3-4dfc-9de7-e72ea77eacac
Transcript: /home/jared/.claude/projects/-home-jared-Documents-SecondBrain/8a73b51f-d8a3-4dfc-9de7-e72ea77eacac.jsonl

VERDICT: CREATE

## Problem The vault graph rebuild cannot finish unattended in two cases. A note can produce no nodes (a hollow note). A node from a note that was deleted or excluded on purpose can make a guard refuse the write. ## Observed Found during #1540. Data from 2026-11-04, graphify 0.9.102, model qwen3.5:9b. The SessionStart run took 74 minutes and exited 1. `graph.json` stayed unchanged. The hook wrote `vault-rebuild.failed`. 13 of 205 notes produced no nodes. Graphify logged "chunk of N still hollow after 3 attempt(s)". Retries in a chunk returned identical `output_tokens`, so same-size retries appear deterministic. A smaller `--token-budget` recovered notes: 4000 left 13 hollow, 2000 left 4, 800 left 2. The hook detects failure with `grep -Eq "chunk [0-9]+/[0-9]+ failed"` in `_REBUILD_SCRIPT`. That pattern matched 0 lines although notes were lost. If extract exits 0 with hollow chunks, the hook writes the stamp and the lost notes stay out until the next rebuild. Two guards refused the write. The first is the incomplete-and-smaller guard in `graphify/cli.py`. The second is `to_json` in `graphify/export.py`, which calls `_lost_saved_semantic_nodes` with an empty `deleted_sources`. It listed nodes from `journal/`, which `.graphifyignore` now excludes on purpose. Code reading (not run) shows the second guard fires on any later deleted or excluded note, even on an otherwise complete run. Each such rebuild would then fail until someone passes `--allow-partial`. Confirm this by a run. The orchestrator wrote the graph by hand with `--allow-partial --token-budget 800`, then cluster-only, then wrote the stamp. In `graphify/cli.py` (`_stamped_manifest_files`), a file with no output has no manifest stamp, so the next run re-queues it. A file whose output shrank (`unverified_semantic_sources`) is excluded from stamping unless `--allow-partial` is set. With `--allow-partial` it is stamped and will not be retried until it changes. Confirm which case applies to the missing notes: - `reference/gitignore-trailing-slash-does-not-match-symlinks.md` - `2026-06-08-graphify-ingesting-code-ast.md` - `2026-06-08-graphify-ingesting-docs-knowledge.md` ## Reproduce ``` graphify extract "$VAULT_PATH" --backend ollama --model qwen3.5:9b --token-budget 4000 --max-concurrency 1 ``` Run on the 2026-11-04 vault state. Expect "still hollow after 3 attempt(s)" lines and exit 1. ## Approach The hook detects hollow notes, retries them with a smaller token budget, and handles intended exclusions without a manual flag. Required work: 1. Make the hook detect "still hollow" chunks as failure — check: present `still hollow` in `plugins/os-vault/hooks/session_start.py` 2. Retry hollow notes with a smaller token budget (800 recovered all but 2 of 13) — check: present `--token-budget 800` in `plugins/os-vault/hooks/session_start.py` Orchestrator work: verify the exclusion guard and stamping behavior against graphify 0.9.102. Decide how the hook handles intended exclusions without a manual flag. ## Acceptance criteria 1. Given an extract run that logs "still hollow" and exits 0, when the hook finishes, then it does not write the stamp. 2. Given hollow notes after a 4000 token run, when the hook retries, then it retries them with a smaller `--token-budget`. 3. Given a note excluded by `.graphifyignore` after the last graph write, when a rebuild is otherwise complete, then the graph is written without a manual flag. ## Owner sketch ```ruby class Vault::HollowNoteRetry def retry(notes, budget) end ``` ## Origin - Trigger: SessionStart rebuild on 2026-11-04 during #1540 (74 minutes, exit 1). - Improvised this session: wrote the graph by hand with `--allow-partial --token-budget 800`, then cluster-only, then wrote the stamp. - Chain: graph not written ← incomplete-and-smaller guard and `_lost_saved_semantic_nodes` guard refused ← 13 hollow notes and journal/ nodes now excluded ← hook retries at the same token budget and does not detect "still hollow" (`_REBUILD_SCRIPT`). DESIGN; graphify 0.9.102 for the guards (UPSTREAM-TOOL). - Root candidate: none found; searched hollow, _REBUILD_SCRIPT, graphify, vault-rebuild, session_start, allow-partial, lost_saved - Where: plugins/os-vault/hooks/session_start.py (_REBUILD_SCRIPT) - Session: 8a73b51f-d8a3-4dfc-9de7-e72ea77eacac - Transcript: /home/jared/.claude/projects/-home-jared-Documents-SecondBrain/8a73b51f-d8a3-4dfc-9de7-e72ea77eacac.jsonl VERDICT: CREATE

jared added the

plugin/os-vault

filed-by/user

category/code-change

labels

2026-11-05 01:01:40 +00:00

jared referenced this issue

2026-11-05 14:19:07 +00:00

Find why extensionless scripts and one vault note are missing from the graphs #1566