Design the extract-index pointer convention #57

New Issue

Closed

opened 2026-07-15 16:01:45 +00:00 by jared · 1 comment

jared commented

2026-07-15 16:01:45 +00:00

Owner

Map: #49

Question

Design the extract-index pointer convention: when an extract-then-delete op moves a doc's insight into the SecondBrain vault, what pointer stays behind in the project?

The sketch (from charting): a lightweight index file — e.g. docs/research/index.md — with one entry per extracted doc: a link to the vault note + a value description saying why/when it would be worth reading. The original file is then deleted; the index is how future sessions discover the insight exists. Example entry:

- [[vault: tool/graphify-clustering-behavior]] — why HDBSCAN over-merges doc clusters; read before tuning any graphify clustering params.

Sub-decisions:

Per-directory index vs one per repo (charting lean: per-directory, it stays local to where the reader looks).
Who writes the entry — the :clean pipeline as part of the extract op, or os-vault:write (extraction already routes through existing knowledge routing per map #31; cc-os plugins are expected to cooperate — overlap is fine, pick the owner).
Extract-worthy vs plain-keep criteria for the judge. Worked contrast from the real repo: docs/research/clutter-pattern-inventory.md is likely KEEP (spec provenance — #41 cites it as the seed source), while a standalone findings doc whose insight generalizes beyond cc-os (e.g. a tool-behavior discovery) is EXTRACT — vault note + index line, then delete.
Is the index file itself rule-covered (lifetime: keep) so it never becomes a deletion candidate?

Map: #49 ## Question Design the extract-index pointer convention: when an extract-then-delete op moves a doc's insight into the SecondBrain vault, what pointer stays behind in the project? **The sketch (from charting):** a lightweight index file — e.g. `docs/research/index.md` — with one entry per extracted doc: a link to the vault note + a value description saying why/when it would be worth reading. The original file is then deleted; the index is how future sessions discover the insight exists. Example entry: - [[vault: tool/graphify-clustering-behavior]] — why HDBSCAN over-merges doc clusters; read before tuning any graphify clustering params. **Sub-decisions:** 1. Per-directory index vs one per repo (charting lean: per-directory, it stays local to where the reader looks). 2. Who writes the entry — the :clean pipeline as part of the extract op, or os-vault:write (extraction already routes through existing knowledge routing per map #31; cc-os plugins are expected to cooperate — overlap is fine, pick the owner). 3. Extract-worthy vs plain-keep criteria for the judge. Worked contrast from the real repo: `docs/research/clutter-pattern-inventory.md` is likely KEEP (spec provenance — #41 cites it as the seed source), while a standalone findings doc whose insight generalizes beyond cc-os (e.g. a tool-behavior discovery) is EXTRACT — vault note + index line, then delete. 4. Is the index file itself rule-covered (lifetime: keep) so it never becomes a deletion candidate?

jared added the

wayfinder:grilling

label 2026-07-15 16:01:45 +00:00

jared referenced this issue

2026-07-15 16:04:52 +00:00

Assemble the spec deltas + ADR amendments #58

jared self-assigned this 2026-07-15 16:45:46 +00:00

jared commented

2026-07-15 16:56:43 +00:00

Author

Owner

Resolution — design approved (human sign-off 2026-07-15: 1=A, 2-4=recommendations).

Scope boundary: the pointer convention exists for VAULT extractions only — the case where the insight physically leaves the repo. Repo-durable residue (an ADR, a CLAUDE.md line) is already discoverable in-repo and gets no index line.

1. Placement/naming — per-directory extracted.md (option A). The pointer stays where a future reader is already looking. The dedicated, self-describing filename means ONE global rule in rulebook.json covers every project forever: { "glob": "**/extracted.md", "lifetime": "keep" } — zero per-project bookkeeping; the index can never become a deletion candidate anywhere. (Rejected B: generic index.md — collides with existing index/README conventions, and **/index.md -> keep is too broad globally, forcing per-project singleton keep rules on every creation.)

2. Writer — the :clean extract op owns the entry. The extract-then-delete op runs: distill -> /os-vault:write (returns the note name) -> append the pointer line -> delete the original. :clean owns the append because (a) the entry needs the vault note name, which only exists mid-op, and (b) :clean's invariant is one reviewable commit — vault pointer, index line, and deletion land atomically, so there is no window where the doc is gone but undiscoverable. os-vault:write stays repo-agnostic. Entry format (extends the charting sketch with provenance):

- [[vault: tool/graphify-clustering-behavior]] — why HDBSCAN over-merges doc
  clusters; read before tuning any graphify clustering params.
  (extracted from `graphify-clustering-notes.md`, 2026-07-15)

3. Extract-worthy vs plain-keep — three tests, in order:

Cited -> KEEP. Anything repo artifacts link to or derive from stays (e.g. docs/research/clutter-pattern-inventory.md — #41 cites it as the seed source; deleting breaks the provenance chain).
Generalizes AND standalone -> EXTRACT. The insight would change behavior in a DIFFERENT repo (tool behavior, methodology — the vault routing test) and nothing in-repo links to it.
Unclear -> consult. The existing mandatory verdict; never guess an extract.

4. Index self-protection + fog closure. Covered by the global **/extracted.md -> keep rule (see 1). The map's remaining fog question — does the extract-index convention graduate into conventions.json? — is resolved as no: conventions.json is the completion-conventions catalog (served-signal graduation — archive-bucket, status-frontmatter). The extract index is not a served signal a project opts into; it is part of the extract op itself, spec-defined and universal. It belongs in the lifecycle spec + one global rule, not the catalog. No new ADR beyond the #58 assembly touching ADR-0038's scope note.

Folds into the assemble ticket #58; nothing edited in the working tree by this ticket.

**Resolution — design approved (human sign-off 2026-07-15: 1=A, 2-4=recommendations).** **Scope boundary:** the pointer convention exists for VAULT extractions only — the case where the insight physically leaves the repo. Repo-durable residue (an ADR, a CLAUDE.md line) is already discoverable in-repo and gets no index line. **1. Placement/naming — per-directory `extracted.md` (option A).** The pointer stays where a future reader is already looking. The dedicated, self-describing filename means ONE global rule in `rulebook.json` covers every project forever: `{ "glob": "**/extracted.md", "lifetime": "keep" }` — zero per-project bookkeeping; the index can never become a deletion candidate anywhere. (Rejected B: generic `index.md` — collides with existing index/README conventions, and `**/index.md -> keep` is too broad globally, forcing per-project singleton keep rules on every creation.) **2. Writer — the `:clean` extract op owns the entry.** The extract-then-delete op runs: distill -> `/os-vault:write` (returns the note name) -> append the pointer line -> delete the original. `:clean` owns the append because (a) the entry needs the vault note name, which only exists mid-op, and (b) `:clean`'s invariant is one reviewable commit — vault pointer, index line, and deletion land atomically, so there is no window where the doc is gone but undiscoverable. `os-vault:write` stays repo-agnostic. Entry format (extends the charting sketch with provenance): ```markdown - [[vault: tool/graphify-clustering-behavior]] — why HDBSCAN over-merges doc clusters; read before tuning any graphify clustering params. (extracted from `graphify-clustering-notes.md`, 2026-07-15) ``` **3. Extract-worthy vs plain-keep — three tests, in order:** 1. **Cited -> KEEP.** Anything repo artifacts link to or derive from stays (e.g. `docs/research/clutter-pattern-inventory.md` — #41 cites it as the seed source; deleting breaks the provenance chain). 2. **Generalizes AND standalone -> EXTRACT.** The insight would change behavior in a DIFFERENT repo (tool behavior, methodology — the vault routing test) and nothing in-repo links to it. 3. **Unclear -> consult.** The existing mandatory verdict; never guess an extract. **4. Index self-protection + fog closure.** Covered by the global `**/extracted.md -> keep` rule (see 1). The map's remaining fog question — does the extract-index convention graduate into `conventions.json`? — is resolved as **no**: `conventions.json` is the completion-conventions catalog (served-signal graduation — archive-bucket, status-frontmatter). The extract index is not a served signal a project opts into; it is part of the extract op itself, spec-defined and universal. It belongs in the lifecycle spec + one global rule, not the catalog. No new ADR beyond the #58 assembly touching ADR-0038's scope note. Folds into the assemble ticket #58; nothing edited in the working tree by this ticket.

jared closed this issue

2026-07-15 16:56:58 +00:00

jared referenced this issue

2026-07-15 16:58:22 +00:00

Wayfinder map: calibrate assessment inventory & rulebook refinements (os-doc-hygiene) #49

jared referenced this issue

2026-07-15 17:46:21 +00:00

Assemble the spec deltas + ADR amendments #58