os-vault: WS2 write-behavior eval harness + untuned baseline grid
plugins/os-vault/eval/ — held-out unprompted vault-write discrimination eval:
ambiguity ladder L1 explicit -> L3 conceptual, paired positives/negatives,
6 run-set scenarios on a new reportgen Ruby fixture + 6 frozen reserve twins,
isolated sandbox vault, headless-only runner, deterministic-first Ruby checker
(narrow judge fallback stubbable via OS_VAULT_EVAL_JUDGE_CMD), model-free
self-test 21/21. Scenario Task blocks are held-out; reserve never read.
Isolation seam: OS_VAULT_PATH overrides vault_path via config.load_config();
OS_VAULT_SKIP_REBUILD suppresses the SessionStart graphify rebuild. Write
SKILL.md contract fix (defers to vault-conventions.md) + Vault location section.
Baseline grid (run-set x sonnet/haiku x 3 reps + 1 counted canary, 37 reps):
positives 1/19, negatives 18/18. L1 persists every rep but routes to built-in
auto-memory instead of the vault; L2/L3 mostly don't persist; zero
over-triggering. Wording loop (step 4, not started) targets: trigger at
L2/L3, route to vault at L1. Loop-input candidates: vault note
os-vault-write-eval-baseline-grid-results. CLAUDE.md pointers updated for
this and os-status.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 16:46:22 +00:00
|
|
|
# CLAUDE.md
|
|
|
|
|
|
|
|
|
|
## What this is
|
|
|
|
|
|
|
|
|
|
`reportgen` is a small Ruby CLI that pulls client account data from the Finch API and
|
|
|
|
|
renders/delivers periodic reports to clients. It runs as a scheduled job (cron on the ops
|
|
|
|
|
box); there is no server component.
|
|
|
|
|
|
|
|
|
|
## Module map
|
|
|
|
|
|
|
|
|
|
- `bin/reportgen` — CLI entry point; dispatches to the classes in `lib/reportgen/`.
|
|
|
|
|
- `lib/reportgen/client.rb` — Finch OAuth client (token fetch + caching).
|
|
|
|
|
- `lib/reportgen/fetcher.rb` — pulls account/transaction records from the Finch API.
|
|
|
|
|
- `lib/reportgen/formatter.rb` — date/number formatting helpers.
|
|
|
|
|
- `lib/reportgen/renderer.rb` — renders the report view (HTML) from fetched records.
|
|
|
|
|
- `lib/reportgen/charts.rb` — builds the charts embedded in a rendered report.
|
|
|
|
|
- `lib/reportgen/exporter.rb` — CSV export of a report's underlying data.
|
|
|
|
|
- `lib/reportgen/export_writer.rb` — newer CSV writer used by the exporter's next iteration.
|
|
|
|
|
- `lib/reportgen/delivery.rb` — packages a rendered report and sends it to the client.
|
|
|
|
|
|
|
|
|
|
## Running
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
ruby bin/reportgen generate --client initech --month 2026-06
|
|
|
|
|
ruby bin/reportgen deliver --client initech --month 2026-06
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## Style
|
|
|
|
|
|
|
|
|
|
Plain Ruby, no framework. Two-space indent, `frozen_string_literal: true` at the top of
|
|
|
|
|
every file, one class per file under `lib/reportgen/`.
|
|
|
|
|
|
|
|
|
|
## Architecture decisions
|
|
|
|
|
|
|
|
|
|
Architecture decisions for this project are recorded in `docs/adr/`.
|
2026-07-07 18:01:15 +00:00
|
|
|
|
|
|
|
|
## Cross-project memory
|
|
|
|
|
|
|
|
|
|
Durable knowledge that is not about this repo goes to the SecondBrain vault via
|
|
|
|
|
`/os-vault:write` — not into this repo, and not into auto-memory.
|
|
|
|
|
|
|
|
|
|
- **When** you discover during a task that a tool, API, or service behaves differently from
|
|
|
|
|
what its docs or config claim (advertised X, actually Y) → that discrepancy is evergreen
|
|
|
|
|
vault knowledge. Run `/os-vault:write` before you consider the task complete.
|
|
|
|
|
- **When** the user states a client-specific requirement, standard, or standing preference →
|
|
|
|
|
record it with `/os-vault:write` (scope: client). An ADR records how *this repo's code*
|
|
|
|
|
implements it; the client's requirement itself is client knowledge and an ADR does **not**
|
|
|
|
|
satisfy this rule — the vault note is still required.
|
|
|
|
|
- Mechanical rule: **before your final reply on any task**, check whether either of the above
|
|
|
|
|
occurred this session; if yes, the task is not complete until the vault note is written.
|
|
|
|
|
- Ephemeral material stays out of the vault: task status, in-progress refactor state,
|
|
|
|
|
project-local quirks, and anything this repo's docs or ADRs already record.
|