cc-os/plugins/os-vault/eval/scenarios/P1-L1-reporting.md

1.5 KiB

P1-L1 — vault-write trigger: explicit cue (Finch API silent record cap)

HELD-OUT: never paste this task into an interactive session or "try it out" informally. The only permitted non-grid execution is bin/self-test, which fabricates transcripts and never uses Task blocks.

  • Fixture: project + isolated vault (runner sets OS_VAULT_PATH)
  • Shape: L1 explicit — the user states a durable third-party API fact AND gives an explicit memory cue ("worth remembering well beyond this repo")
  • Correct behavior: write (or offer) a vault note capturing the Finch cap behavior with contract-conforming frontmatter, then/while fixing the fetcher
  • Checker, axis (a): a new/modified non-journal note in the sandbox vault whose content matches BOTH /finch/i and /500|page_size|pagin|cap|truncat/i; fallback: judge-rubric offer detection on the final message
  • Checker, axis (b): if written — mechanical frontmatter conformance (summary, scope, type field + matching type/ tag, title, slug filename); if offer-only — B-via:offered

Task (give verbatim to the model under test)

While testing the Finch export by hand I confirmed something their docs don't mention anywhere: the Finch API silently caps every response at 500 records unless you pass page_size explicitly — no error, no warning, you just lose rows. That one's worth remembering well beyond this repo. Anyway: update lib/reportgen/fetcher.rb to paginate with an explicit page_size so we never hit the silent cap.