Fix eval reliability gate: count capped calls and empty notes, add tiebreak gap #1556

New issue

Open

opened 2026-11-04 23:26:36 +00:00 by jared · 0 comments

jared commented

2026-11-04 23:26:36 +00:00

Owner

Problem

The graph-extraction eval reliability gate admits unreliable runs, and its speed tiebreak has no minimum gap. In the small-models round the gate gave the win to qwen3.5:4b over qwen3.5:9b for a 0.3 s/chunk speed difference. The user overruled the result.

Observed

The gate in plugins/os-vault/eval/graph-extraction/bin/score.rb passes a run when failed_chunk_rate <= 0.05. The README says all six runs passed "because a capped chunk counts as done". The tiebreak picks the fastest run among runs within 0.03 F1 of the best, with no minimum speed gap.

Evidence from plugins/os-vault/eval/graph-extraction/results/small-models/README.md:

qwen3.5:4b had 8 cap hits and 6 notes with no nodes. qwen3.5:9b had 0 cap hits and 2 notes with no nodes.
qwen3.5:0.8b had 15 cap hits and 36 notes with no nodes (4 nodes in total), and still passed the gate.
qwen3.5:2b had 17 cap hits and 23 notes with no nodes, and still passed the gate.
F1 was 0.503 for 4b and 0.511 for 9b. Speed was 77.0 s/chunk for 4b and 77.3 s/chunk for 9b. The 0.3 s/chunk gap decided the winner.
The Decision section says "The speed gap is noise, 4b lost 6 notes against 2, and 4b loops on short notes".

Reproduce

n/a: design defect in the gate rule. Run bin/score.rb on the small-models results and read winner.

Approach

Count notes with no nodes and capped calls in the reliability gate. Add a minimum speed gap to the tiebreak, so a small difference in s/chunk does not decide the winner.

Required work:

Count capped calls and notes with no nodes in the reliability gate — check: present cap in plugins/os-vault/eval/graph-extraction/bin/score.rb
Add a minimum s/chunk gap to the speed tiebreak — check: present min_gap in plugins/os-vault/eval/graph-extraction/bin/score.rb

Acceptance criteria

Given a run whose capped calls or notes with no nodes exceed the gate limit, when bin/score.rb runs, then that run is absent from reliability_gate_passed.
Given two runs within the F1 window whose s/chunk differ by less than the minimum gap, when bin/score.rb runs, then the speed tiebreak does not pick the winner.
Given the small-models results, when bin/score.rb runs, then winner is not qwen3.5:4b on the 0.3 s/chunk difference.

Owner sketch

# bin/score.rb: gate and tiebreak
passing = per_model.select { |_, v| v[:failed_chunk_rate].to_f <= 0.05 && reliable?(v) }

Origin

Trigger: small-models eval round, 2026-11-04, results/small-models/README.md Decision section
Improvised this session: none; the user overrode the rule result and chose qwen3.5:9b
Chain: wrong winner ← speed tiebreak has no minimum gap ← DESIGN (bin/score.rb lines 58-63)
Root candidate: none found; searched eval, gate, winner, graph-extraction, score.rb, tiebreak, capped, reliab
VERDICT: CREATE
Where: plugins/os-vault/eval/graph-extraction/bin/score.rb
Session: 8a73b51f-d8a3-4dfc-9de7-e72ea77eacac
Transcript: /home/jared/.claude/projects/-home-jared-Documents-SecondBrain/8a73b51f-d8a3-4dfc-9de7-e72ea77eacac.jsonl

## Problem The graph-extraction eval reliability gate admits unreliable runs, and its speed tiebreak has no minimum gap. In the small-models round the gate gave the win to `qwen3.5:4b` over `qwen3.5:9b` for a 0.3 s/chunk speed difference. The user overruled the result. ## Observed The gate in `plugins/os-vault/eval/graph-extraction/bin/score.rb` passes a run when `failed_chunk_rate <= 0.05`. The README says all six runs passed "because a capped chunk counts as done". The tiebreak picks the fastest run among runs within 0.03 F1 of the best, with no minimum speed gap. Evidence from `plugins/os-vault/eval/graph-extraction/results/small-models/README.md`: 1. `qwen3.5:4b` had 8 cap hits and 6 notes with no nodes. `qwen3.5:9b` had 0 cap hits and 2 notes with no nodes. 2. `qwen3.5:0.8b` had 15 cap hits and 36 notes with no nodes (4 nodes in total), and still passed the gate. 3. `qwen3.5:2b` had 17 cap hits and 23 notes with no nodes, and still passed the gate. 4. F1 was 0.503 for 4b and 0.511 for 9b. Speed was 77.0 s/chunk for 4b and 77.3 s/chunk for 9b. The 0.3 s/chunk gap decided the winner. 5. The Decision section says "The speed gap is noise, 4b lost 6 notes against 2, and 4b loops on short notes". ## Reproduce n/a: design defect in the gate rule. Run `bin/score.rb` on the small-models results and read `winner`. ## Approach Count notes with no nodes and capped calls in the reliability gate. Add a minimum speed gap to the tiebreak, so a small difference in s/chunk does not decide the winner. Required work: 1. Count capped calls and notes with no nodes in the reliability gate — check: present `cap` in `plugins/os-vault/eval/graph-extraction/bin/score.rb` 2. Add a minimum s/chunk gap to the speed tiebreak — check: present `min_gap` in `plugins/os-vault/eval/graph-extraction/bin/score.rb` ## Acceptance criteria 1. Given a run whose capped calls or notes with no nodes exceed the gate limit, when `bin/score.rb` runs, then that run is absent from `reliability_gate_passed`. 2. Given two runs within the F1 window whose s/chunk differ by less than the minimum gap, when `bin/score.rb` runs, then the speed tiebreak does not pick the winner. 3. Given the small-models results, when `bin/score.rb` runs, then `winner` is not `qwen3.5:4b` on the 0.3 s/chunk difference. ## Owner sketch ```ruby # bin/score.rb: gate and tiebreak passing = per_model.select { |_, v| v[:failed_chunk_rate].to_f <= 0.05 && reliable?(v) } ``` ## Origin - Trigger: small-models eval round, 2026-11-04, `results/small-models/README.md` Decision section - Improvised this session: none; the user overrode the rule result and chose `qwen3.5:9b` - Chain: wrong winner ← speed tiebreak has no minimum gap ← DESIGN (`bin/score.rb` lines 58-63) - Root candidate: none found; searched eval, gate, winner, graph-extraction, score.rb, tiebreak, capped, reliab - VERDICT: CREATE - Where: plugins/os-vault/eval/graph-extraction/bin/score.rb - Session: 8a73b51f-d8a3-4dfc-9de7-e72ea77eacac - Transcript: /home/jared/.claude/projects/-home-jared-Documents-SecondBrain/8a73b51f-d8a3-4dfc-9de7-e72ea77eacac.jsonl

jared added the

plugin/os-vault

filed-by/user

category/code-change

labels

2026-11-04 23:26:36 +00:00

jared referenced this issue

2026-11-05 18:31:40 +00:00

Switch the os-vault rebuild hook to Unsloth 9B V4 on llama.cpp #1576