Decide eval/test coverage for the nominations intake filter and consult loop

jared commented

2026-07-15 16:43:23 +00:00

Owner

Map: #49

Question

Decide the eval/test discipline for the changed :calibrate behavior — now specifiable since #56 locked the nominations design. What coverage does each new seam get, and does any of it need harness (model-judged) coverage versus plain unit tests?

The seams, with charting leans:

NominationIntakeFilter (Step 3.5) — deterministic (exact-repeat hard-drop, match-set-intersection relatedness). Lean: unit tests only, per the plugin's deterministic-first invariant #6 — fixture nominations + a nominations key, assert drops/annotations.
Canonical writer extension (nominations after rules; consults before rejected; glob-sorted; unknown-field warnings) — deterministic. Lean: unit tests only.
Judge "Nominations memory" prompt section — does judge behavior under injected rejection context (e.g. does it actually weigh a past rejection's why when judging a variant glob?) need scenario-based harness coverage, or is this covered by production IRL use? Note the standing eval discipline: held-out scenarios, reserves never read informally; Read ~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md before designing any harness work.
Consult resurfacing loop (re-present each run; exits: rule / rejected / defer) — orchestration behavior in SKILL.md, historically the hardest layer to test deterministically. Lean: worked example in the skill + IRL audit, no harness.

Deliverable: a per-seam coverage decision (unit / harness / IRL-only) folded into #58's assembled design.

Map: #49 ## Question Decide the eval/test discipline for the changed `:calibrate` behavior — now specifiable since #56 locked the nominations design. What coverage does each new seam get, and does any of it need *harness* (model-judged) coverage versus plain unit tests? The seams, with charting leans: 1. **`NominationIntakeFilter` (Step 3.5)** — deterministic (exact-repeat hard-drop, match-set-intersection relatedness). Lean: unit tests only, per the plugin's deterministic-first invariant #6 — fixture nominations + a nominations key, assert drops/annotations. 2. **Canonical writer extension** (nominations after rules; consults before rejected; glob-sorted; unknown-field warnings) — deterministic. Lean: unit tests only. 3. **Judge "Nominations memory" prompt section** — does judge behavior under injected rejection context (e.g. does it actually weigh a past rejection's `why` when judging a variant glob?) need scenario-based harness coverage, or is this covered by production IRL use? Note the standing eval discipline: held-out scenarios, reserves never read informally; Read `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md` before designing any harness work. 4. **Consult resurfacing loop** (re-present each run; exits: rule / rejected / defer) — orchestration behavior in SKILL.md, historically the hardest layer to test deterministically. Lean: worked example in the skill + IRL audit, no harness. Deliverable: a per-seam coverage decision (unit / harness / IRL-only) folded into #58's assembled design.

jared added the

wayfinder:grilling

label

2026-07-15 16:43:23 +00:00

jared referenced this issue

2026-07-15 16:43:42 +00:00

Assemble the spec deltas + ADR amendments #58

jared self-assigned this

2026-07-15 17:27:49 +00:00

jared commented

2026-07-15 17:30:02 +00:00

Author

Owner

Resolution — approved (human sign-off 2026-07-15). Per-seam coverage decisions for the new :calibrate behavior:

NominationIntakeFilter (Step 3.5) → unit tests only. Pure deterministic logic (exact-repeat drop, match-set-intersection relatedness) under the plugin's deterministic-first invariant #6. Fixture: a nominations array + a nominations key; assert drops and related-rejection annotations — including the two worked cases from #52 (docs/research/** -> keep re-lifetime passes; exact docs/research/** -> temporary repeat drops).
Canonical writer extension (nominations after rules, consults before rejected, glob-sorted, unknown-field warnings) → unit tests only. Round-trip a fixture file; assert ordering + warnings.
Judge "Nominations memory" context → no harness; production IRL audits. A scenario harness for "does the judge weigh a past rejection's why" would be a new wording-loop-style eval; the standing posture after the completed wording loops is that the next signal comes from production IRL session audits. Mitigation instead: one worked example in judge.md's input section showing a variant nomination judged WITH rejection context (folds into #58). If IRL use shows the judge ignoring the context, THAT triggers harness design — reading ~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md first, per discipline.
Consult resurfacing loop → no harness; worked example in SKILL.md + IRL. Orchestration behavior — the layer harnesses are historically weakest at — and every exit is human-gated, so the failure mode is annoyance (a consult not resurfacing), not damage.

Net: two unit-test suites, zero new harnesses, two worked examples added at assembly time. Deciding reason: both model-facing seams are human-gated downstream — no silent-failure path exists that a harness would uniquely catch.

Folds into the assemble ticket #58; nothing edited in the working tree by this ticket.

**Resolution — approved (human sign-off 2026-07-15).** Per-seam coverage decisions for the new `:calibrate` behavior: 1. **`NominationIntakeFilter` (Step 3.5) → unit tests only.** Pure deterministic logic (exact-repeat drop, match-set-intersection relatedness) under the plugin's deterministic-first invariant #6. Fixture: a nominations array + a `nominations` key; assert drops and related-rejection annotations — including the two worked cases from #52 (`docs/research/** -> keep` re-lifetime passes; exact `docs/research/** -> temporary` repeat drops). 2. **Canonical writer extension** (nominations after rules, consults before rejected, glob-sorted, unknown-field warnings) **→ unit tests only.** Round-trip a fixture file; assert ordering + warnings. 3. **Judge "Nominations memory" context → no harness; production IRL audits.** A scenario harness for "does the judge weigh a past rejection's why" would be a new wording-loop-style eval; the standing posture after the completed wording loops is that the next signal comes from production IRL session audits. Mitigation instead: one worked example in `judge.md`'s input section showing a variant nomination judged WITH rejection context (folds into #58). If IRL use shows the judge ignoring the context, THAT triggers harness design — reading `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md` first, per discipline. 4. **Consult resurfacing loop → no harness; worked example in SKILL.md + IRL.** Orchestration behavior — the layer harnesses are historically weakest at — and every exit is human-gated, so the failure mode is annoyance (a consult not resurfacing), not damage. Net: two unit-test suites, zero new harnesses, two worked examples added at assembly time. Deciding reason: both model-facing seams are human-gated downstream — no silent-failure path exists that a harness would uniquely catch. Folds into the assemble ticket #58; nothing edited in the working tree by this ticket.

jared commented

2026-07-15 17:30:09 +00:00

Author

Owner

Resolution — approved (human sign-off 2026-07-15). Per-seam coverage decisions for the new :calibrate behavior:

NominationIntakeFilter (Step 3.5) → unit tests only. Pure deterministic logic (exact-repeat drop, match-set-intersection relatedness) under the plugin's deterministic-first invariant #6. Fixture: a nominations array + a nominations key; assert drops and related-rejection annotations — including the two worked cases from #52 (docs/research/** -> keep re-lifetime passes; exact docs/research/** -> temporary repeat drops).
Canonical writer extension (nominations after rules, consults before rejected, glob-sorted, unknown-field warnings) → unit tests only. Round-trip a fixture file; assert ordering + warnings.
Judge "Nominations memory" context → no harness; production IRL audits. A scenario harness for "does the judge weigh a past rejection's why" would be a new wording-loop-style eval; the standing posture after the completed wording loops is that the next signal comes from production IRL session audits. Mitigation instead: one worked example in judge.md's input section showing a variant nomination judged WITH rejection context (folds into #58). If IRL use shows the judge ignoring the context, THAT triggers harness design — reading ~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md first, per discipline.
Consult resurfacing loop → no harness; worked example in SKILL.md + IRL. Orchestration behavior — the layer harnesses are historically weakest at — and every exit is human-gated, so the failure mode is annoyance (a consult not resurfacing), not damage.

Net: two unit-test suites, zero new harnesses, two worked examples added at assembly time. Deciding reason: both model-facing seams are human-gated downstream — no silent-failure path exists that a harness would uniquely catch.

Folds into the assemble ticket #58; nothing edited in the working tree by this ticket.

**Resolution — approved (human sign-off 2026-07-15).** Per-seam coverage decisions for the new `:calibrate` behavior: 1. **`NominationIntakeFilter` (Step 3.5) → unit tests only.** Pure deterministic logic (exact-repeat drop, match-set-intersection relatedness) under the plugin's deterministic-first invariant #6. Fixture: a nominations array + a `nominations` key; assert drops and related-rejection annotations — including the two worked cases from #52 (`docs/research/** -> keep` re-lifetime passes; exact `docs/research/** -> temporary` repeat drops). 2. **Canonical writer extension** (nominations after rules, consults before rejected, glob-sorted, unknown-field warnings) **→ unit tests only.** Round-trip a fixture file; assert ordering + warnings. 3. **Judge "Nominations memory" context → no harness; production IRL audits.** A scenario harness for "does the judge weigh a past rejection's why" would be a new wording-loop-style eval; the standing posture after the completed wording loops is that the next signal comes from production IRL session audits. Mitigation instead: one worked example in `judge.md`'s input section showing a variant nomination judged WITH rejection context (folds into #58). If IRL use shows the judge ignoring the context, THAT triggers harness design — reading `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md` first, per discipline. 4. **Consult resurfacing loop → no harness; worked example in SKILL.md + IRL.** Orchestration behavior — the layer harnesses are historically weakest at — and every exit is human-gated, so the failure mode is annoyance (a consult not resurfacing), not damage. Net: two unit-test suites, zero new harnesses, two worked examples added at assembly time. Deciding reason: both model-facing seams are human-gated downstream — no silent-failure path exists that a harness would uniquely catch. Folds into the assemble ticket #58; nothing edited in the working tree by this ticket.

jared closed this issue

2026-07-15 17:30:19 +00:00

jared referenced this issue

2026-07-15 17:30:37 +00:00

Wayfinder map: calibrate assessment inventory & rulebook refinements (os-doc-hygiene) #49

jared referenced this issue

2026-07-15 17:46:21 +00:00

Assemble the spec deltas + ADR amendments #58

jared referenced this issue from a commit

2026-07-15 19:10:57 +00:00

os-doc-hygiene: calibrate assessment-inventory design locked — spec deltas + ADR amendments (map #49, ticket #58)