cc-os

Commit Graph

Author	SHA1	Message	Date
jared	7523663889	os-adr: trigger-conditioned wording from Eval B autoresearch loop (sonnet 8/8, haiku 7/8) Five-iteration /autoresearch wording experiment against the Eval B baseline (haiku 0/8, sonnet 5/8). Checker/fixtures/scenarios/rubric frozen; wording surfaces: hook PRESENT_NOTE, find/create SKILL.mds, fixture CLAUDE.md (declared a surface upfront — it is the real-project adoption template). - Hook note + skill descriptions: when->then trigger-conditioned phrasing (H1 confirmed, both tiers) - find skill: explicit-unconditional reversal->supersede offer where find's output is on screen (fixed sonnet W3) - Mechanical lower-tier trigger: before first edit to any existing file, run find on those paths; additions count (fixed haiku R1/R4) - New eval-b/fixture/project/CLAUDE.md carrying the same rules Final grid: sonnet 8/8, haiku 7/8 (W3 axis-b judge flicker). Open: channel ablation (hook vs CLAUDE.md) not run; R4-nograph now passes both tiers. Hypothesis->result map: vault note os-adr-eval-b-wording-experiment-hypotheses. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-04 14:49:15 -04:00
jared	00e22f1676	Add os-adr evaluation harnesses (Eval A and Eval B) Build comprehensive evaluation framework for the os-adr plugin (ADR-021). Eval A: prompted skill-execution evaluation across haiku/sonnet models. Six ADR lifecycle scenarios, deterministic Ruby checker, sandbox runner, /autoresearch loop for wording optimization. Eval B: held-out unprompted-behavior evaluation. Seven scenarios (W1–W3 write-trigger, R1–R4 retrieval), webhook fixture with 6-ADR history (Superseded pair + distractors), headless runner (isolated SessionStart context), two-axis deterministic-first checker (consultation + citation accuracy, AI judge fallback for new-file writes). Both harnesses self-tested model-free. Grid runs deferred per locked rollout order. Updated docs/specs to freeze scenario shapes and document methodologies.	2026-07-03 16:44:42 -04:00

Author

SHA1

Message

Date

jared

7523663889

os-adr: trigger-conditioned wording from Eval B autoresearch loop (sonnet 8/8, haiku 7/8)

Five-iteration /autoresearch wording experiment against the Eval B baseline
(haiku 0/8, sonnet 5/8). Checker/fixtures/scenarios/rubric frozen; wording
surfaces: hook PRESENT_NOTE, find/create SKILL.mds, fixture CLAUDE.md
(declared a surface upfront — it is the real-project adoption template).

- Hook note + skill descriptions: when->then trigger-conditioned phrasing
  (H1 confirmed, both tiers)
- find skill: explicit-unconditional reversal->supersede offer where find's
  output is on screen (fixed sonnet W3)
- Mechanical lower-tier trigger: before first edit to any existing file,
  run find on those paths; additions count (fixed haiku R1/R4)
- New eval-b/fixture/project/CLAUDE.md carrying the same rules

Final grid: sonnet 8/8, haiku 7/8 (W3 axis-b judge flicker). Open: channel
ablation (hook vs CLAUDE.md) not run; R4-nograph now passes both tiers.
Hypothesis->result map: vault note os-adr-eval-b-wording-experiment-hypotheses.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

2026-07-04 14:49:15 -04:00

jared

00e22f1676

Add os-adr evaluation harnesses (Eval A and Eval B)

Build comprehensive evaluation framework for the os-adr plugin (ADR-021).

Eval A: prompted skill-execution evaluation across haiku/sonnet models.
Six ADR lifecycle scenarios, deterministic Ruby checker, sandbox runner,
/autoresearch loop for wording optimization.

Eval B: held-out unprompted-behavior evaluation. Seven scenarios
(W1–W3 write-trigger, R1–R4 retrieval), webhook fixture with 6-ADR history
(Superseded pair + distractors), headless runner (isolated SessionStart
context), two-axis deterministic-first checker (consultation + citation
accuracy, AI judge fallback for new-file writes).

Both harnesses self-tested model-free. Grid runs deferred per locked
rollout order. Updated docs/specs to freeze scenario shapes and document
methodologies.

2026-07-03 16:44:42 -04:00

2 Commits