jared/cc-os

Watch

Fork 0

Stop counting graphify path refusals as no path in the retrieval harness #1578

New issue

Closed

opened 2026-11-05 21:12:30 +00:00 by jared · 1 comment

jared commented

2026-11-05 21:12:30 +00:00

Owner

Problem

The retrieval harness counts a graphify path refusal as "no path". The linked-pairs score therefore falls when new stub nodes make a name ambiguous, although no link is lost.

Observed

graphify path prints "Ambiguous" to stderr and exits with code 3. This happened 5 times before and 8 times after the re-extraction. The harness showed 25/30 then 22/30. A direct graph search shows 30/30 at 1 hop in both graphs. Control pairs show 18 then 17, and the true counts are 23 then 21. Report: plugins/os-vault/eval/graph-retrieval/diff-reextract.md section 2.

Reproduce

n/a: see the report.

Approach

path_call gains a third state, refused.

Required work:

run in plugins/os-vault/eval/graph-retrieval/measure.py returns stderr as well; update both callers (lines 61 and 112).
path_call returns refused: true with the exit code and message when the exit code is nonzero and no hop count matched. Confirm the exit code for a real no-path result against graphify 0.9.102 source first.
connection_test keeps a refused count per kind. n stays the pair count, so found_rate stays comparable.
md_report prints refused beside found. Q4 keeps the refused field through its filter at line 493. check: present refused in plugins/os-vault/eval/graph-retrieval/measure.py

Acceptance criteria

A refused pair's row has refused: true and does not count as no path.
A real no-path pair has refused: false.
The JSON summary and Markdown report show a refused count per kind.
pairs.json, questions.json, and testset.tsv stay byte-unchanged.
A run needs no model and no GPU.

Owner sketch

path_call owns the meaning of graphify path output. No new object.

Origin

Trigger: investigation of score drops after the V4 re-extraction, 2026-11-05
Improvised this session: none
Chain: false drop ← exit code 3 read as no path ← measure.py reads stdout only ← DESIGN
Root candidate: none found; searched measure.py, graphify path, Ambiguous, no path, retrieval harness
Where: plugins/os-vault/eval/graph-retrieval/measure.py
Session: 47ffa777-a9a8-4ac8-808a-b9b19489944a
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/47ffa777-a9a8-4ac8-808a-b9b19489944a.jsonl

VERDICT: CREATE

## Problem The retrieval harness counts a `graphify path` refusal as "no path". The linked-pairs score therefore falls when new stub nodes make a name ambiguous, although no link is lost. ## Observed `graphify path` prints "Ambiguous" to stderr and exits with code 3. This happened 5 times before and 8 times after the re-extraction. The harness showed 25/30 then 22/30. A direct graph search shows 30/30 at 1 hop in both graphs. Control pairs show 18 then 17, and the true counts are 23 then 21. Report: `plugins/os-vault/eval/graph-retrieval/diff-reextract.md` section 2. ## Reproduce n/a: see the report. ## Approach `path_call` gains a third state, refused. Required work: 1. `run` in `plugins/os-vault/eval/graph-retrieval/measure.py` returns stderr as well; update both callers (lines 61 and 112). 2. `path_call` returns `refused: true` with the exit code and message when the exit code is nonzero and no hop count matched. Confirm the exit code for a real no-path result against graphify 0.9.102 source first. 3. `connection_test` keeps a `refused` count per kind. `n` stays the pair count, so `found_rate` stays comparable. 4. `md_report` prints refused beside found. Q4 keeps the refused field through its filter at line 493. check: present `refused` in `plugins/os-vault/eval/graph-retrieval/measure.py` ## Acceptance criteria 1. A refused pair's row has `refused: true` and does not count as no path. 2. A real no-path pair has `refused: false`. 3. The JSON summary and Markdown report show a refused count per kind. 4. `pairs.json`, `questions.json`, and `testset.tsv` stay byte-unchanged. 5. A run needs no model and no GPU. ## Owner sketch `path_call` owns the meaning of `graphify path` output. No new object. ## Origin - Trigger: investigation of score drops after the V4 re-extraction, 2026-11-05 - Improvised this session: none - Chain: false drop ← exit code 3 read as no path ← `measure.py` reads stdout only ← DESIGN - Root candidate: none found; searched measure.py, graphify path, Ambiguous, no path, retrieval harness - Where: plugins/os-vault/eval/graph-retrieval/measure.py - Session: 47ffa777-a9a8-4ac8-808a-b9b19489944a - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/47ffa777-a9a8-4ac8-808a-b9b19489944a.jsonl VERDICT: CREATE

jared added the

plugin/os-vault

filed-by/agent

category/code-change

labels

2026-11-05 21:12:30 +00:00

jared referenced this issue

2026-11-05 21:12:30 +00:00

Finish the vault graph rebuild after the graphify update #1540

jared added the

ready-for-agent

label

2026-11-06 14:30:03 +00:00

jared referenced this issue

2026-11-06 20:09:51 +00:00

Run pytest targets in the diff-test gate #1625

jared referenced this issue

2026-11-06 20:27:05 +00:00

Run pytest targets in the diff-test gate #1625

jared commented

2026-11-06 21:05:28 +00:00

Author

Owner

Resolution

Done: measure.py records a graphify path refusal as refused, beside found, in the JSON summary and the Markdown report, including Q4. changed_notes.py also exits 2 on a corrupt or empty manifest.

Evidence: main b591d01a. pytest plugins/os-vault/tests: 43 passed. test_measure_refused.py: 6 tests failed before the change. Exit codes read from graphify 0.9.102 cli.py: no path exits 0, ambiguous exits 3, not found exits 4. Live check: changed_notes.py on the vault exits 0. The harness was not rerun. Finished by hand because the diff-test gate cannot run pytest (#1625).

Follow-ups: none

## Resolution **Done:** measure.py records a graphify path refusal as refused, beside found, in the JSON summary and the Markdown report, including Q4. changed_notes.py also exits 2 on a corrupt or empty manifest. **Evidence:** main b591d01a. pytest plugins/os-vault/tests: 43 passed. test_measure_refused.py: 6 tests failed before the change. Exit codes read from graphify 0.9.102 cli.py: no path exits 0, ambiguous exits 3, not found exits 4. Live check: changed_notes.py on the vault exits 0. The harness was not rerun. Finished by hand because the diff-test gate cannot run pytest (#1625). **Follow-ups:** none

jared closed this issue

2026-11-06 21:05:28 +00:00

jared referenced this issue from a commit

2026-11-06 21:05:31 +00:00

fix(os-vault): stop counting path refusals as no path, and refuse a corrupt manifest