Define what the 10-question set means for #1540 #1569

New issue

Closed

opened 2026-11-05 14:19:09 +00:00 by jared · 2 comments

jared commented

2026-11-05 14:19:09 +00:00

Owner

Problem

Ticket #1540 names a "10 question cc-os test set" with a limit of 7 of 10. Nothing in #1540 says which graph the set tests. The set holds code-structure questions about the cc-os repo. It cannot be answered by the vault graph. A reader who runs it on the vault graph gets a meaningless score.
The vault has no fixed question set. The 12 questions in the vault test could serve.
Evidence: /tmp/claude-1000/-home-jared-Documents-SecondBrain/8a73b51f-d8a3-4dfc-9de7-e72ea77eacac/scratchpad/graph-test-results.md (copied below; do not depend on the path).

Observed

The set is the file /home/jared/dev/cc-os/.cc-os/os/graphify-audit-2026-11-04/testset.tsv. Pass rule: the correct file path appears in graphify query output. Limit: 7 of 10.
On the cc-os graph (31,673 nodes at the first run; 32,134 later) the score is 3 of 10 (questions 1, 8, 9 pass). The limit is not met.
The vault test has 12 questions: 8 PASS, 4 PARTIAL, 0 FAIL. Under a strict rule (answer facts in output) it is 0 of 12.
11 of the 12 vault questions are lookups; 1 is a connection test (8%). The questions were written from note titles by someone who knew the notes.
It is not known whether the cc-os graph after the planned update counts as the "new graph" in #1540.

Approach

The owner rules on three points.

Which graph the 10-question set tests (the cc-os code graph), and whether it becomes its own check with its own name and location.
Whether the 12 vault questions become the vault set, with a rule for pass (title hit, or facts in output).
Which set, if any, #1540 uses as its limit.

Required work: none beyond the acceptance criteria.

Orchestrator work: record the ruling on #1540 and on this ticket.

Acceptance criteria

Given the ruling, when #1540 is read, then it names the graph that the 10-question set tests.
Given the ruling, when a reader looks for the vault question set, then the 12 vault questions have a stated home and a stated pass rule.
Given the ruling, when the cc-os set runs, then it runs as its own check apart from the vault set.

Origin

Trigger: vault graph test, part B, 2026-11-04.
Improvised this session: ran the 10 questions on the cc-os graph read-only with --graph.
Chain: ambiguous pass rule in #1540 ← the set was written for the cc-os graph in session a877275b and #1540 does not name it ← DESIGN (#1540).
Root candidate: none found; searched backlog terms for this ticket (VERDICT: CREATE)
Where: #1540; /home/jared/dev/cc-os/.cc-os/os/graphify-audit-2026-11-04/testset.tsv
Session: b10276a0-3582-48b5-b0fe-f13795bdcc75
Transcript: /home/jared/.claude/projects/-home-jared-Documents-SecondBrain/b10276a0-3582-48b5-b0fe-f13795bdcc75.jsonl
Parent: #1540.

## Problem Ticket #1540 names a "10 question cc-os test set" with a limit of 7 of 10. Nothing in #1540 says which graph the set tests. The set holds code-structure questions about the cc-os repo. It cannot be answered by the vault graph. A reader who runs it on the vault graph gets a meaningless score. The vault has no fixed question set. The 12 questions in the vault test could serve. Evidence: /tmp/claude-1000/-home-jared-Documents-SecondBrain/8a73b51f-d8a3-4dfc-9de7-e72ea77eacac/scratchpad/graph-test-results.md (copied below; do not depend on the path). ## Observed - The set is the file `/home/jared/dev/cc-os/.cc-os/os/graphify-audit-2026-11-04/testset.tsv`. Pass rule: the correct file path appears in `graphify query` output. Limit: 7 of 10. - On the cc-os graph (31,673 nodes at the first run; 32,134 later) the score is 3 of 10 (questions 1, 8, 9 pass). The limit is not met. - The vault test has 12 questions: 8 PASS, 4 PARTIAL, 0 FAIL. Under a strict rule (answer facts in output) it is 0 of 12. - 11 of the 12 vault questions are lookups; 1 is a connection test (8%). The questions were written from note titles by someone who knew the notes. - It is not known whether the cc-os graph after the planned update counts as the "new graph" in #1540. ## Approach The owner rules on three points. 1. Which graph the 10-question set tests (the cc-os code graph), and whether it becomes its own check with its own name and location. 2. Whether the 12 vault questions become the vault set, with a rule for pass (title hit, or facts in output). 3. Which set, if any, #1540 uses as its limit. Required work: none beyond the acceptance criteria. Orchestrator work: record the ruling on #1540 and on this ticket. ## Acceptance criteria 1. Given the ruling, when #1540 is read, then it names the graph that the 10-question set tests. 2. Given the ruling, when a reader looks for the vault question set, then the 12 vault questions have a stated home and a stated pass rule. 3. Given the ruling, when the cc-os set runs, then it runs as its own check apart from the vault set. ## Origin - Trigger: vault graph test, part B, 2026-11-04. - Improvised this session: ran the 10 questions on the cc-os graph read-only with `--graph`. - Chain: ambiguous pass rule in #1540 ← the set was written for the cc-os graph in session a877275b and #1540 does not name it ← DESIGN (#1540). - Root candidate: none found; searched backlog terms for this ticket (VERDICT: CREATE) - Where: #1540; /home/jared/dev/cc-os/.cc-os/os/graphify-audit-2026-11-04/testset.tsv - Session: b10276a0-3582-48b5-b0fe-f13795bdcc75 - Transcript: /home/jared/.claude/projects/-home-jared-Documents-SecondBrain/b10276a0-3582-48b5-b0fe-f13795bdcc75.jsonl - Parent: #1540.

jared added the

project/cc-os

filed-by/user

category/owner-ruling

labels

2026-11-05 14:19:09 +00:00

jared commented

2026-11-05 15:59:24 +00:00

Author

Owner

Predicted ruling (ruling 16 in docs/decisions/rulings.tsv, confidence 74). Jared can correct it.

The 12 vault questions decide the vault part of #1540. They test the vault graph (SecondBrain). Their home is plugins/os-vault/eval/graph-retrieval/ (questions.json, run by measure.py). Pass rule: the expected note appears in the graphify query output, graded by the harness.
The 10-question set tests the cc-os code graph. It is its own check, at limit 7 of 10. Its home is plugins/os-vault/eval/graph-retrieval/testset.tsv. Pass rule: the expected file path appears in the query output.
The 12 alone do not prove graphify is ready. The 13 reworded questions and the facts-in-output count stay as tracked scores.

The harness moved to the new directory in commit 74583f0c. It gives identical scores on the baseline graphs after the move.

Predicted ruling (ruling 16 in `docs/decisions/rulings.tsv`, confidence 74). Jared can correct it. 1. The 12 vault questions decide the vault part of #1540. They test the vault graph (SecondBrain). Their home is `plugins/os-vault/eval/graph-retrieval/` (`questions.json`, run by `measure.py`). Pass rule: the expected note appears in the `graphify query` output, graded by the harness. 2. The 10-question set tests the cc-os code graph. It is its own check, at limit 7 of 10. Its home is `plugins/os-vault/eval/graph-retrieval/testset.tsv`. Pass rule: the expected file path appears in the query output. 3. The 12 alone do not prove graphify is ready. The 13 reworded questions and the facts-in-output count stay as tracked scores. The harness moved to the new directory in commit 74583f0c. It gives identical scores on the baseline graphs after the move.

jared referenced this issue

2026-11-05 15:59:24 +00:00

Finish the vault graph rebuild after the graphify update #1540

jared commented

2026-11-05 15:59:26 +00:00

Author

Owner

Resolution

Done: Ruling recorded as a predicted ruling (ruling 16): the 12 vault questions decide the vault part of #1540; the 10-question set is its own cc-os check at limit 7. The graph, the home, and the pass rule are named in the ruling comment.

Evidence: docs/decisions/rulings.tsv row 16 (commit 85e81691); harness at plugins/os-vault/eval/graph-retrieval/ (commit 74583f0c); comments on #1569 and #1540.

Follow-ups: none

## Resolution **Done:** Ruling recorded as a predicted ruling (ruling 16): the 12 vault questions decide the vault part of #1540; the 10-question set is its own cc-os check at limit 7. The graph, the home, and the pass rule are named in the ruling comment. **Evidence:** docs/decisions/rulings.tsv row 16 (commit 85e81691); harness at plugins/os-vault/eval/graph-retrieval/ (commit 74583f0c); comments on #1569 and #1540. **Follow-ups:** none

jared closed this issue

2026-11-05 15:59:26 +00:00