cc-os/plugins/os-vault/eval/scenarios/P2-L2-reporting.md

22 lines
1.3 KiB
Markdown

# P2-L2 — vault-write trigger: moderate (Finch OAuth expires_in is wrong)
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
- **Fixture:** `project` + isolated `vault` (runner sets `OS_VAULT_PATH`)
- **Shape:** L2 moderate — a durable third-party API fact is stated as debugging context;
NO memory cue, the user only asks for the code fix. The model must recognize
cross-project tool knowledge on its own.
- **Correct behavior:** unprompted vault note (or offer) capturing the real Finch token
lifetime behavior, plus the client.rb fix
- **Checker, axis (a):** new/modified non-journal vault note matching BOTH /finch|oauth/i
and /expires_in|15\s*min|token|idle|inactiv/i; fallback: judge-rubric offer detection
- **Checker, axis (b):** as P1 (mechanical conformance, or `B-via:offered`)
## Task (give verbatim to the model under test)
The nightly Acme sync died again with 401s halfway through. I traced it: Finch OAuth tokens
actually expire after 15 minutes of inactivity even though the token response advertises
`expires_in=3600` — their `expires_in` is simply wrong, and nothing in their docs admits it.
Fix lib/reportgen/client.rb to refresh the token proactively based on that real-world
lifetime instead of trusting `expires_in`.