Test a loop fix and a second run for Qwen3.5-9B on llama.cpp #1563

New issue

Open

opened 2026-11-05 13:58:15 +00:00 by jared · 1 comment

jared commented

2026-11-05 13:58:15 +00:00

Owner

Problem

The Unsloth Qwen3.5-9B run on llama.cpp won the fixed rule, but it is one run with 3 capped calls. The os-vault default cannot change until variance is measured and a loop fix is tested.

Observed

3 of 19 calls hit the 8192-token cap. They repeated nodes, gave 0 nodes and 0 edges, and took about 38% of wall time. json_object did not prevent the runaway. The F1 gap to the next model is 0.0359, close to the 0.03 tie band.

Reproduce

n/a: this is a planned measurement, not a defect. Run bin/run_extract_llamacpp.sh twice on the same 40-note sample.

Approach

Run the llama.cpp driver a second time with the same provider entry and compare the two runs. Then test one loop fix (for example repeat_penalty above 1.0 or a nonzero presence_penalty) on the 3 capped chunks and in a full run. Record the aggregates in a new results/ directory.

Required work:

Record the second-run and loop-fix results — check: present Second run in plugins/os-vault/eval/graph-extraction/results/unsloth-9b/README.md

Orchestrator work: run the server and the extraction on the RTX 3060.

Acceptance criteria

Given two full runs with the same provider entry, when the results are compared, then the README states the F1 spread between the runs.
Given a loop fix, when the 3 capped chunks are replayed, then the README states how many calls still hit the cap.

Origin

Trigger: Unsloth llama.cpp round, 2026-11-05, plugins/os-vault/eval/graph-extraction/results/unsloth-9b/README.md
Improvised this session: none
Chain: default cannot change ← one run with 3 capped calls ← DESIGN (results/unsloth-9b/README.md Limitations)
Root candidate: none found; searched unsloth, llamacpp, llama.cpp, graph-extraction, qwen3.5, repeat_penalty, capped calls
VERDICT: CREATE
Where: plugins/os-vault/eval/graph-extraction/
Session: 5d12673e-2926-4e7a-8b7a-98894647d13c
Transcript: /home/jared/.claude/projects/-home-jared-Documents-SecondBrain/5d12673e-2926-4e7a-8b7a-98894647d13c.jsonl

## Problem The Unsloth Qwen3.5-9B run on llama.cpp won the fixed rule, but it is one run with 3 capped calls. The os-vault default cannot change until variance is measured and a loop fix is tested. ## Observed 3 of 19 calls hit the 8192-token cap. They repeated nodes, gave 0 nodes and 0 edges, and took about 38% of wall time. `json_object` did not prevent the runaway. The F1 gap to the next model is 0.0359, close to the 0.03 tie band. ## Reproduce n/a: this is a planned measurement, not a defect. Run `bin/run_extract_llamacpp.sh` twice on the same 40-note sample. ## Approach Run the llama.cpp driver a second time with the same provider entry and compare the two runs. Then test one loop fix (for example `repeat_penalty` above 1.0 or a nonzero `presence_penalty`) on the 3 capped chunks and in a full run. Record the aggregates in a new `results/` directory. Required work: 1. Record the second-run and loop-fix results — check: present `Second run` in `plugins/os-vault/eval/graph-extraction/results/unsloth-9b/README.md` Orchestrator work: run the server and the extraction on the RTX 3060. ## Acceptance criteria 1. Given two full runs with the same provider entry, when the results are compared, then the README states the F1 spread between the runs. 2. Given a loop fix, when the 3 capped chunks are replayed, then the README states how many calls still hit the cap. ## Origin - Trigger: Unsloth llama.cpp round, 2026-11-05, `plugins/os-vault/eval/graph-extraction/results/unsloth-9b/README.md` - Improvised this session: none - Chain: default cannot change ← one run with 3 capped calls ← DESIGN (`results/unsloth-9b/README.md` Limitations) - Root candidate: none found; searched unsloth, llamacpp, llama.cpp, graph-extraction, qwen3.5, repeat_penalty, capped calls - VERDICT: CREATE - Where: plugins/os-vault/eval/graph-extraction/ - Session: 5d12673e-2926-4e7a-8b7a-98894647d13c - Transcript: /home/jared/.claude/projects/-home-jared-Documents-SecondBrain/5d12673e-2926-4e7a-8b7a-98894647d13c.jsonl

jared added the

plugin/os-vault

filed-by/agent

category/eval-run

labels

2026-11-05 13:58:15 +00:00

jared commented

2026-11-05 18:06:02 +00:00

Author

Owner

V4 result, 2026-11-05 (one full run of 40 notes, 17 chunks). This is not the 3 repeat runs, so the ticket stays open.

V4 changes only the request: json_schema with maxItems 40 nodes, 100 edges, 20 hyperedges; presence_penalty 1.5; max_tokens 10000. Temperature 0.2, no seed, thinking off, context 16384.

Wall 690.6 s against 1260.8 s for the prior Unsloth run. Tokens 89,622 against 125,385. Capped calls 0 against 3. Notes lost 0 against 2.
All 17 replies parsed with finish reason stop. 19 of 217 edges were dangling (8.8%).
Recall fell. Supported claims 141 against 191. F1 0.334 against 0.415 (8-way, not comparable with earlier rounds). V4 ranks fourth of eight. The fixed rule still gives the win to the prior Unsloth run.
Replay test (6 requests): hard caps V0 7/15, V1 2/15, V2 2/15, V3 1/15, V4 0/6. Samples are small. Only V0 against V3 and V4 is a clear gap.
Jared decided to switch to Unsloth on llama.cpp and to use V4. The implementation is not done.

Still open: 3 repeat runs for variance. Aggregates: cc-os branch eval-llamacpp-unsloth-9b, results/unsloth-9b-v4/. Notes: vault graph-extraction-model-rtx3060-eval-results.

V4 result, 2026-11-05 (one full run of 40 notes, 17 chunks). This is not the 3 repeat runs, so the ticket stays open. V4 changes only the request: json_schema with maxItems 40 nodes, 100 edges, 20 hyperedges; presence_penalty 1.5; max_tokens 10000. Temperature 0.2, no seed, thinking off, context 16384. - Wall 690.6 s against 1260.8 s for the prior Unsloth run. Tokens 89,622 against 125,385. Capped calls 0 against 3. Notes lost 0 against 2. - All 17 replies parsed with finish reason stop. 19 of 217 edges were dangling (8.8%). - Recall fell. Supported claims 141 against 191. F1 0.334 against 0.415 (8-way, not comparable with earlier rounds). V4 ranks fourth of eight. The fixed rule still gives the win to the prior Unsloth run. - Replay test (6 requests): hard caps V0 7/15, V1 2/15, V2 2/15, V3 1/15, V4 0/6. Samples are small. Only V0 against V3 and V4 is a clear gap. - Jared decided to switch to Unsloth on llama.cpp and to use V4. The implementation is not done. Still open: 3 repeat runs for variance. Aggregates: cc-os branch eval-llamacpp-unsloth-9b, results/unsloth-9b-v4/. Notes: vault graph-extraction-model-rtx3060-eval-results.