os-vault: WS2 write-behavior eval harness + untuned baseline grid
plugins/os-vault/eval/ — held-out unprompted vault-write discrimination eval: ambiguity ladder L1 explicit -> L3 conceptual, paired positives/negatives, 6 run-set scenarios on a new reportgen Ruby fixture + 6 frozen reserve twins, isolated sandbox vault, headless-only runner, deterministic-first Ruby checker (narrow judge fallback stubbable via OS_VAULT_EVAL_JUDGE_CMD), model-free self-test 21/21. Scenario Task blocks are held-out; reserve never read. Isolation seam: OS_VAULT_PATH overrides vault_path via config.load_config(); OS_VAULT_SKIP_REBUILD suppresses the SessionStart graphify rebuild. Write SKILL.md contract fix (defers to vault-conventions.md) + Vault location section. Baseline grid (run-set x sonnet/haiku x 3 reps + 1 counted canary, 37 reps): positives 1/19, negatives 18/18. L1 persists every rep but routes to built-in auto-memory instead of the vault; L2/L3 mostly don't persist; zero over-triggering. Wording loop (step 4, not started) targets: trigger at L2/L3, route to vault at L1. Loop-input candidates: vault note os-vault-write-eval-baseline-grid-results. CLAUDE.md pointers updated for this and os-status. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
jared
parent
1d1e7207c0
commit
638c6d03ed
11
CLAUDE.md
11
CLAUDE.md
|
|
@ -81,6 +81,9 @@ to those two and fix the stale doc.
|
||||||
- Skills: `skills/` — query, write, reorganize, onboard-project (assessment-first onboarding: surveys repo, classifies excludes by type (11 categories), writes per-project `.graphifyignore`, confirms with user, then extracts using `qwen25-coder-7b-16k` — per ADR-017), design-template (routes template-design ↔ new-type-creation: 4-step template-design process + injection-economics filter, 9-step new-type lifecycle; backs Phase 1 of the SB Content Plan — see `docs/memory-system/09-sb-content-plan.md`)
|
- Skills: `skills/` — query, write, reorganize, onboard-project (assessment-first onboarding: surveys repo, classifies excludes by type (11 categories), writes per-project `.graphifyignore`, confirms with user, then extracts using `qwen25-coder-7b-16k` — per ADR-017), design-template (routes template-design ↔ new-type-creation: 4-step template-design process + injection-economics filter, 9-step new-type lifecycle; backs Phase 1 of the SB Content Plan — see `docs/memory-system/09-sb-content-plan.md`)
|
||||||
- Config: `config.yaml` — vault path, Ollama model (qwen25-coder-7b-16k), env vars
|
- Config: `config.yaml` — vault path, Ollama model (qwen25-coder-7b-16k), env vars
|
||||||
- Hook wiring: `~/.claude/settings.json` (hook entries invoke `/usr/bin/python3` with absolute paths into cc-os)
|
- Hook wiring: `~/.claude/settings.json` (hook entries invoke `/usr/bin/python3` with absolute paths into cc-os)
|
||||||
|
- Isolation seam (2026-07-06, WS2): `OS_VAULT_PATH` env var overrides `vault_path` for the write skill and every hook via `config.load_config()`; `OS_VAULT_SKIP_REBUILD` suppresses the SessionStart graphify rebuild. Added for the eval harness; zero production impact when unset.
|
||||||
|
- Write-behavior eval (2026-07-06, WS2): `plugins/os-vault/eval/` — held-out unprompted vault-write discrimination eval (Eval B/C playbook: ambiguity ladder L1 explicit → L3 conceptual, paired positives/negatives, 6 run-set scenarios on a new `reportgen` Ruby fixture + 6 frozen reserve twins in a different knowledge domain, isolated sandbox vault, headless-only runner, deterministic-first Ruby checker with narrow frozen offer-detection judge fallback stubbable via `OS_VAULT_EVAL_JUDGE_CMD`, model-free `bin/self-test` 21/21). **Scenario Task blocks are held-out — never run informally; the reserve set is never even read informally.** Procedure: `plugins/os-vault/eval/README.md`.
|
||||||
|
- Untuned baseline grid (2026-07-06, run-set × sonnet/haiku × 3 reps + 1 counted canary, 37 reps, zero harness errors): **positives 1/19, negatives 18/18**. Headline: at L1 (explicit cue) both tiers persist the knowledge every rep but route it to Claude Code's built-in auto-memory (`~/.claude/projects/<slug>/memory/`) instead of the vault; L2/L3 mostly don't persist at all; zero over-triggering anywhere. The WS2 wording loop (step 4, NOT started; run-set still uncontaminated) therefore has two targets: trigger at L2/L3, and route to the vault over auto-memory at L1. TSV: `eval/results/2026-07-06-baseline-grid.tsv`; full analysis: vault note [[os-vault-write-eval-baseline-grid-results]].
|
||||||
|
|
||||||
**Global os-orchestration plugin** — `cc-os/plugins/os-orchestration/` (git-tracked, 2026-07-03); symlinked into `~/.claude/plugins/os-orchestration`
|
**Global os-orchestration plugin** — `cc-os/plugins/os-orchestration/` (git-tracked, 2026-07-03); symlinked into `~/.claude/plugins/os-orchestration`
|
||||||
- Hooks: `hooks/` — `inject.py` (injects `ORCHESTRATION.md` as additionalContext to all sessions)
|
- Hooks: `hooks/` — `inject.py` (injects `ORCHESTRATION.md` as additionalContext to all sessions)
|
||||||
|
|
@ -91,6 +94,12 @@ to those two and fix the stale doc.
|
||||||
- Eval harness E1–E3 (2026-07-06): `plugins/os-orchestration/eval/` — scripted headless orchestration-behavior eval scored by the audit extractor (extended with per-segment read targets). 6 run-set scenarios (E1P/N downgrade detection with `CLAUDE_CODE_SUBAGENT_MODEL=haiku` runner stub; E2P/N explicit-model fan-out vs no-delegation-on-trivial; E3P/N delegate-investigation vs orienting-reads-correct) + 6 frozen reserve twins; new Node.js "relaystation" fixture with deterministic sandbox-time log generation; deterministic-first checker (`eval/bin/check`, narrow frozen haiku-judge fallback for E1 language axes only, stubbable via `ORCH_EVAL_JUDGE_CMD`); model-free `bin/self-test` 21/21 green incl. shipped-instruction-compliant conformance transcripts. FAIL axes map only to verified audit misses; tier choice informational. **Scenario Task blocks are held-out — never run informally.** Procedure: `plugins/os-orchestration/eval/README.md`.
|
- Eval harness E1–E3 (2026-07-06): `plugins/os-orchestration/eval/` — scripted headless orchestration-behavior eval scored by the audit extractor (extended with per-segment read targets). 6 run-set scenarios (E1P/N downgrade detection with `CLAUDE_CODE_SUBAGENT_MODEL=haiku` runner stub; E2P/N explicit-model fan-out vs no-delegation-on-trivial; E3P/N delegate-investigation vs orienting-reads-correct) + 6 frozen reserve twins; new Node.js "relaystation" fixture with deterministic sandbox-time log generation; deterministic-first checker (`eval/bin/check`, narrow frozen haiku-judge fallback for E1 language axes only, stubbable via `ORCH_EVAL_JUDGE_CMD`); model-free `bin/self-test` 21/21 green incl. shipped-instruction-compliant conformance transcripts. FAIL axes map only to verified audit misses; tier choice informational. **Scenario Task blocks are held-out — never run informally.** Procedure: `plugins/os-orchestration/eval/README.md`.
|
||||||
- Baseline grid run (2026-07-06, run-set × sonnet/haiku × 3 reps + 2 counted canaries, all counted, zero harness errors): negatives 18/18 PASS both tiers (zero over-delegation, zero false downgrade claims); positives 1/18 on the delegate-at-all axis — but transcript verification shows most positive FAILs are *superior direct strategies* (E2P sonnet: one scripted bulk edit; E3P sonnet: surgical greps over 400KB logs), which the WS1 audit itself scored as justified non-delegation, vs genuine miss shapes (E3P haiku: whole 112KB logs into own context; E2P haiku: 20-call per-file grind). Conditional rules validated where engaged: 14/14 spawns explicit-model; the one delegating E1P rep flagged the stubbed downgrade via self-report. Criterion redesign for E2P/E3P A-axes is pre-registered (anchor thresholds independently, NOT to this grid) before any rescore/tuning; reserve stays frozen. ORCHESTRATION.md gained a scripted-bulk-edit carve-out (a uniform multi-file change one script covers is direct work). TSV: `eval/results/2026-07-06-baseline-grid.tsv`; full analysis: vault note [[os-orchestration-eval-baseline-grid-results]].
|
- Baseline grid run (2026-07-06, run-set × sonnet/haiku × 3 reps + 2 counted canaries, all counted, zero harness errors): negatives 18/18 PASS both tiers (zero over-delegation, zero false downgrade claims); positives 1/18 on the delegate-at-all axis — but transcript verification shows most positive FAILs are *superior direct strategies* (E2P sonnet: one scripted bulk edit; E3P sonnet: surgical greps over 400KB logs), which the WS1 audit itself scored as justified non-delegation, vs genuine miss shapes (E3P haiku: whole 112KB logs into own context; E2P haiku: 20-call per-file grind). Conditional rules validated where engaged: 14/14 spawns explicit-model; the one delegating E1P rep flagged the stubbed downgrade via self-report. Criterion redesign for E2P/E3P A-axes is pre-registered (anchor thresholds independently, NOT to this grid) before any rescore/tuning; reserve stays frozen. ORCHESTRATION.md gained a scripted-bulk-edit carve-out (a uniform multi-file change one script covers is direct work). TSV: `eval/results/2026-07-06-baseline-grid.tsv`; full analysis: vault note [[os-orchestration-eval-baseline-grid-results]].
|
||||||
|
|
||||||
|
**Global os-status plugin** — `cc-os/plugins/os-status/` (git-tracked, 2026-07-06); symlinked into `~/.claude/plugins/os-status`
|
||||||
|
- Purpose: aggregated deterministic SessionStart status checks for the cc-os plugin family — per-project artifacts (ADR system, vault hub note) and environment hazards (subagent model env override). Built via OpenSpec change `add-os-status-plugin` (supersedes the WS3 plan doc); decision record ADR-022.
|
||||||
|
- Architecture: checks are **in-process** plain Python functions in `hooks/checks.py` with the uniform signature `check(ctx) -> CheckResult(status, message)` plus one registry entry (no per-plugin subprocess protocol, no per-project code copies — both rejected in ADR-022 after perspective review). Three states: `ok` (silent), `note` (near-zero-token additionalContext line, never snoozed/suppressed), `warn` (aggregated into at most one banner; once-per-day snooze + permanent `suppress-<check>` markers in gitignored per-project `.cc-os/` — state only, never code; `.cc-os/config` holds optional `hub`/`vault_path` key=value overrides).
|
||||||
|
- Initial checks: `subagent-model-env-override` (env + `~/.claude/settings.json` `env` block; runs outside git projects too — catches the WS1 Cluster 1 incident deterministically at session start), `adr-system-present` (verbatim port of os-adr's hook behavior incl. `.os-adr/suppress`), `vault-hub-note-present` (config slug first, else `type/hub` + `project/<name>` facet-tag scan of the vault; missing → warn naming `/os-vault:write`).
|
||||||
|
- Tests: `python3 tests/hook_test.py` (36, model-free) incl. byte-identity of PRESENT_NOTE/ABSENT_NOTE against os-adr's source. Invariants in `invariants.md`. Smoke-tested 2026-07-06 (cache==source, real headless session fired the hook, env-override canary warns).
|
||||||
|
|
||||||
**Global os-doc-hygiene plugin** — `cc-os/plugins/os-doc-hygiene/` (git-tracked, 2026-07-03); symlinked into `~/.claude/plugins/os-doc-hygiene`
|
**Global os-doc-hygiene plugin** — `cc-os/plugins/os-doc-hygiene/` (git-tracked, 2026-07-03); symlinked into `~/.claude/plugins/os-doc-hygiene`
|
||||||
- Hooks: `hooks/hooks.json` → SessionStart hook (matcher: startup|resume) runs `scripts/reminder.py` via `${CLAUDE_PLUGIN_ROOT}` (5s timeout), emitting a deterministic zero-token reminder banner
|
- Hooks: `hooks/hooks.json` → SessionStart hook (matcher: startup|resume) runs `scripts/reminder.py` via `${CLAUDE_PLUGIN_ROOT}` (5s timeout), emitting a deterministic zero-token reminder banner
|
||||||
- Behavior: Monitors stale and bloated project documentation per-project under `.dochygiene/` state dir (gitignored). SessionStart reminder is deterministic (no AI tokens, once/day snooze). Skills (verb-first, no `commands/` dispatcher — invoked directly as `/os-doc-hygiene:<skill>`, per [[cc-os-plugin-skill-naming-convention]]): `check` (AI-assisted classification of staleness signals, emits machine+human reports), `clean` (AI-assisted or deterministic patch application with git-safe scoped cleanup), `status` (read-only lifecycle-timestamp read), `sweep` (check then clean in sequence). Reversion-protected via invariants.md + golden-example test fixtures.
|
- Behavior: Monitors stale and bloated project documentation per-project under `.dochygiene/` state dir (gitignored). SessionStart reminder is deterministic (no AI tokens, once/day snooze). Skills (verb-first, no `commands/` dispatcher — invoked directly as `/os-doc-hygiene:<skill>`, per [[cc-os-plugin-skill-naming-convention]]): `check` (AI-assisted classification of staleness signals, emits machine+human reports), `clean` (AI-assisted or deterministic patch application with git-safe scoped cleanup), `status` (read-only lifecycle-timestamp read), `sweep` (check then clean in sequence). Reversion-protected via invariants.md + golden-example test fixtures.
|
||||||
|
|
@ -99,7 +108,7 @@ to those two and fix the stale doc.
|
||||||
**Global os-adr plugin** — `cc-os/plugins/os-adr/` (git-tracked, 2026-07-03); symlinked into `~/.claude/plugins/os-adr`
|
**Global os-adr plugin** — `cc-os/plugins/os-adr/` (git-tracked, 2026-07-03); symlinked into `~/.claude/plugins/os-adr`
|
||||||
- Purpose: Architecture Decision Records, consistently in every project — one customized-Nygard template (frontmatter `id`/`date`/`status`/`supersedes`/`superseded-by`/`affected-paths`/`affected-components`; Context/Decision/Consequences/Alternatives-rejected sections), `docs/adr/NNNN-kebab-title.md` one-file-per-decision + fully-regenerated `docs/adr/README.md` index. Built from `docs/adr-system/` (requirements `04`, PRD `05`, eval sketches `06`) via OpenSpec change `add-os-adr-plugin`.
|
- Purpose: Architecture Decision Records, consistently in every project — one customized-Nygard template (frontmatter `id`/`date`/`status`/`supersedes`/`superseded-by`/`affected-paths`/`affected-components`; Context/Decision/Consequences/Alternatives-rejected sections), `docs/adr/NNNN-kebab-title.md` one-file-per-decision + fully-regenerated `docs/adr/README.md` index. Built from `docs/adr-system/` (requirements `04`, PRD `05`, eval sketches `06`) via OpenSpec change `add-os-adr-plugin`.
|
||||||
- Core: Ruby OO library `lib/adr/` (`Record`, `Repository`, `Index`, `Template`, `Detector`, `Migrator`, `MigrationReport`, `Finder`; Sandi Metz style, injected paths) behind thin CLIs in `bin/` (`adr-new`, `adr-init`, `adr-detect`, `adr-migrate`, `adr-find`). Tests: `ruby tests/all.rb` (47) + `python3 tests/hook_test.py` (15), all model-free. Behavioral invariants in `invariants.md`.
|
- Core: Ruby OO library `lib/adr/` (`Record`, `Repository`, `Index`, `Template`, `Detector`, `Migrator`, `MigrationReport`, `Finder`; Sandi Metz style, injected paths) behind thin CLIs in `bin/` (`adr-new`, `adr-init`, `adr-detect`, `adr-migrate`, `adr-find`). Tests: `ruby tests/all.rb` (47) + `python3 tests/hook_test.py` (15), all model-free. Behavioral invariants in `invariants.md`.
|
||||||
- Hook: `hooks/session_start.py` wired via plugin-relative `hooks/hooks.json` (`${CLAUDE_PLUGIN_ROOT}`, no `settings.json` entries) — deterministic existence check: present → near-zero-token usage note naming `/os-adr:create` + `/os-adr:find`; absent → once-per-day init/migrate suggestion, silenced permanently by `.os-adr/suppress` (gitignored per-project state dir); silent outside git projects.
|
- Hook: **moved to os-status (2026-07-06)** — the SessionStart existence check now runs as the `adr-system-present` check in the os-status plugin (`hooks.json` here is empty; `hooks/session_start.py` stays as the wording source of record, byte-copied into os-status). Behavior unchanged: present → near-zero-token usage note naming `/os-adr:create` + `/os-adr:find`; absent → once-per-day init/migrate suggestion, silenced permanently by `.os-adr/suppress` (still honored); silent outside git projects.
|
||||||
- Skills (verb-first, no `commands/`): `create` (one-invocation templated+numbered+indexed ADR with mechanical supersession), `init`, `migrate` (non-destructive: detect surveyed shapes → mechanical heuristic fill → LLM fills only manifest-listed interpretive fields via `--apply-fills` → `migration_confidence` frontmatter + `docs/adr/migration-report.md`; old-system deletion only as an explicit separate user-approved step), `find` (deterministic-first retrieval: affected-paths/components match → Accepted-only filter → one-hop Graphify query-path expansion with graceful degradation → AI judgment over the candidate set only).
|
- Skills (verb-first, no `commands/`): `create` (one-invocation templated+numbered+indexed ADR with mechanical supersession), `init`, `migrate` (non-destructive: detect surveyed shapes → mechanical heuristic fill → LLM fills only manifest-listed interpretive fields via `--apply-fills` → `migration_confidence` frontmatter + `docs/adr/migration-report.md`; old-system deletion only as an explicit separate user-approved step), `find` (deterministic-first retrieval: affected-paths/components match → Accepted-only filter → one-hop Graphify query-path expansion with graceful degradation → AI judgment over the candidate set only).
|
||||||
- Migration pilot passed 2026-07-03 (sandboxed project copies; gate threshold 25% low-confidence flags): viking-warrior-training-log 0%, delta-refinery 0%, llf-schema 8.3%. Retrieval verified end-to-end against the real llf-schema project graph. Resolved: ADRs need no Graphify tag/edge convention — retrieval layer 3 expands the *query* paths via graph-node `source_file`s.
|
- Migration pilot passed 2026-07-03 (sandboxed project copies; gate threshold 25% low-confidence flags): viking-warrior-training-log 0%, delta-refinery 0%, llf-schema 8.3%. Retrieval verified end-to-end against the real llf-schema project graph. Resolved: ADRs need no Graphify tag/edge convention — retrieval layer 3 expands the *query* paths via graph-node `source_file`s.
|
||||||
- Eval A harness (2026-07-03, ADR-021): `plugins/os-adr/eval/` — prompted skill-execution eval across model tiers (haiku/sonnet). Two fixtures (generated with the plugin's own CLIs), six scenarios (S1 create, S2 create+supersede, S3 find/conflict, S4 find/distractor, S5 init, S6 migrate+fills), deterministic Ruby checker (`eval/bin/check`, structural invariants, TSV mode for autoresearch), sandbox + headless-runner scripts, runner-prompt template. Primary run mode: in-session Agent-tool subagents with pinned `model:` (cheaper than `claude -p`); optimized via the `/autoresearch` Classic loop over SKILL.md *wording only* (checker/fixtures/scenarios frozen during a loop). Procedure: `plugins/os-adr/eval/README.md`. Self-tested both directions; grid run 2026-07-03, all scenarios × both tiers passing. Distinct from the held-out Eval B (unprompted behavior) — do not conflate.
|
- Eval A harness (2026-07-03, ADR-021): `plugins/os-adr/eval/` — prompted skill-execution eval across model tiers (haiku/sonnet). Two fixtures (generated with the plugin's own CLIs), six scenarios (S1 create, S2 create+supersede, S3 find/conflict, S4 find/distractor, S5 init, S6 migrate+fills), deterministic Ruby checker (`eval/bin/check`, structural invariants, TSV mode for autoresearch), sandbox + headless-runner scripts, runner-prompt template. Primary run mode: in-session Agent-tool subagents with pinned `model:` (cheaper than `claude -p`); optimized via the `/autoresearch` Classic loop over SKILL.md *wording only* (checker/fixtures/scenarios frozen during a loop). Procedure: `plugins/os-adr/eval/README.md`. Self-tested both directions; grid run 2026-07-03, all scenarios × both tiers passing. Distinct from the held-out Eval B (unprompted behavior) — do not conflate.
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,14 @@
|
||||||
# WS2 — os-vault "when and how to write" eval
|
# WS2 — os-vault "when and how to write" eval
|
||||||
|
|
||||||
_Created: 2026-07-06. Status: proposed. Prereqs: quick wins immediate; harness after WS3
|
_Created: 2026-07-06. Status: **steps 1–3 done 2026-07-06** — quick wins shipped (contract
|
||||||
design + quick wins. See `2026-07-06-plugin-evals-overview.md`._
|
fix + cc-os hub note), harness built at `plugins/os-vault/eval/` (12 scenarios authored
|
||||||
|
directly, fixture + Ruby checker via sonnet subagents, self-test 21/21, isolation via new
|
||||||
|
`OS_VAULT_PATH`/`OS_VAULT_SKIP_REBUILD` seam), untuned baseline grid run (37 counted reps:
|
||||||
|
positives 1/19, negatives 18/18; headline finding: at L1 the model persists to Claude Code's
|
||||||
|
built-in auto-memory instead of the vault — routing, not just triggering, is the wording
|
||||||
|
target). Results: `eval/results/2026-07-06-baseline-grid.tsv` + vault note
|
||||||
|
[[os-vault-write-eval-baseline-grid-results]]. Step 4 (wording loop) not started — the
|
||||||
|
run-set is still uncontaminated. See `2026-07-06-plugin-evals-overview.md`._
|
||||||
|
|
||||||
## Goal
|
## Goal
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,157 @@
|
||||||
|
# os-vault write-behavior eval — unprompted vault-write discrimination (held-out)
|
||||||
|
|
||||||
|
_Last updated: 2026-07-06 — harness built (WS2); baseline grid NOT yet run._
|
||||||
|
|
||||||
|
Measures whether the model **unprompted** (a) writes evergreen cross-project knowledge to
|
||||||
|
the SecondBrain vault when it surfaces during normal work, and (b) writes it with
|
||||||
|
contract-conforming frontmatter — and equally, whether it correctly does **NOT** write
|
||||||
|
ephemera, project-local detail, or repo-recorded decisions to the vault. Applies the os-adr
|
||||||
|
Eval B/C playbook (ambiguity ladder, paired positives/negatives, run-set + frozen reserve).
|
||||||
|
Plan: `docs/plans/ws2-os-vault-write-eval.md`.
|
||||||
|
|
||||||
|
> **HELD-OUT — read this first.** The `## Task` blocks in `scenarios/*.md` and
|
||||||
|
> `scenarios-reserve/*.md` must never be pasted into an interactive session or informally
|
||||||
|
> "tried out". The only permitted non-grid execution is `bin/self-test`, which fabricates
|
||||||
|
> transcripts and never uses the Task blocks. The reserve set is additionally never *read*
|
||||||
|
> informally — it becomes the measurement set the moment wording is tuned against the
|
||||||
|
> run-set.
|
||||||
|
|
||||||
|
## What the baseline measures
|
||||||
|
|
||||||
|
At baseline the shipped os-vault wording has **no trigger-conditioned "when to write"
|
||||||
|
phrasing anywhere** (WS2 defect 2, deliberately left unfixed until after the baseline).
|
||||||
|
Expected shape: positives likely FAIL axis (a) at both tiers (that gap IS the measurement);
|
||||||
|
negatives should PASS (nothing currently pushes the model to write). The post-baseline
|
||||||
|
wording loop then tunes trigger wording (skill description/body, hook note, optionally a
|
||||||
|
CLAUDE.md section template mirroring os-adr's) against the run-set only.
|
||||||
|
|
||||||
|
## Design: ambiguity ladder with paired positive/negative
|
||||||
|
|
||||||
|
| Level | Positive (should write/offer) | Negative (should NOT write) |
|
||||||
|
|---|---|---|
|
||||||
|
| L1 explicit | P1: durable Finch API fact + explicit "worth remembering" cue | N1: explicit "make a note" cue, but in-progress refactor status (ephemeral) |
|
||||||
|
| L2 moderate | P2: durable Finch OAuth fact stated as debugging context, no cue | N2: same framing, but the quirk is this repo's own legacy config (project-local) |
|
||||||
|
| L3 conceptual | P3: client-standing Acme convention mentioned in passing | N3: convention-shaped knowledge already recorded in the repo's ADR (docs/adr/0002) |
|
||||||
|
|
||||||
|
Negatives are equally tempting by construction (explicit note cue / discovery framing /
|
||||||
|
convention shape) — an always-write model aces positives and fails all negatives; a
|
||||||
|
never-write model does the reverse. Reserve set (P4–P6/N4–N6, `scenarios-reserve/`) mirrors
|
||||||
|
the ladder in a different knowledge domain: chartkit tool behavior + Globex delivery
|
||||||
|
conventions + the SFTP ADR (docs/adr/0003).
|
||||||
|
|
||||||
|
## Fixture
|
||||||
|
|
||||||
|
- `fixture/project/` — `reportgen`, a small Ruby CLI that pulls data from the fictional
|
||||||
|
Finch API and renders/delivers client reports. Modules are rigged so every scenario's
|
||||||
|
premise is visible in code (fetcher without pagination, expires_in-trusting OAuth client,
|
||||||
|
legacy alias block in config/schema.yml, raw-UTC renderer, non-streaming chartkit usage,
|
||||||
|
emailing delivery, hardcoded staging bucket, half-done ExportWriter refactor). Has a
|
||||||
|
3-ADR `docs/adr/` history generated with os-adr's own CLIs (0002 UTC-storage and 0003
|
||||||
|
SFTP-delivery are the N3/N6 targets). `CLAUDE.md` is deliberately plain — **no memory or
|
||||||
|
vault trigger wording** (that is the tuning surface under test, absent at baseline).
|
||||||
|
- `fixture/vault/` — the isolated SecondBrain stand-in: a **verbatim frozen copy** of the
|
||||||
|
real `vault-conventions.md` plus four seeded notes (different client, different project,
|
||||||
|
unrelated tools) so the vault looks lived-in, and an empty `journal/`. The checker treats
|
||||||
|
the fixture vault as the manifest: anything new or content-changed in a sandbox vault
|
||||||
|
(outside `journal/` and `graphify-out/`) is a vault write.
|
||||||
|
|
||||||
|
## Isolation (do not skip)
|
||||||
|
|
||||||
|
The runner sets, for every rep:
|
||||||
|
|
||||||
|
- `OS_VAULT_PATH=<sandbox>/vault` — honored by the write skill ("Vault location") and by
|
||||||
|
every os-vault hook via `config.load_config()`, so the session-end journal, vault sync
|
||||||
|
(no-ops: sandbox vault has no `.git`) and any vault reads all hit the sandbox vault.
|
||||||
|
- `OS_VAULT_SKIP_REBUILD=1` — suppresses the SessionStart detached graphify/ollama rebuild
|
||||||
|
(pointless and slow against a throwaway vault).
|
||||||
|
|
||||||
|
The checker additionally scans every transcript for tool writes under
|
||||||
|
`~/Documents/SecondBrain` and fails the rep loudly (`real-vault-breach`) if the model wrote
|
||||||
|
to the real vault despite the override — check for and revert any such write immediately
|
||||||
|
(the vault_sync hook auto-commits the real vault at SessionEnd).
|
||||||
|
|
||||||
|
## Two-axis scoring
|
||||||
|
|
||||||
|
- **Axis (a) — write trigger (positives)**: mechanical PASS iff a new or content-modified
|
||||||
|
non-journal note in the sandbox vault matches the scenario's topic regexes; else the
|
||||||
|
frozen `judge-rubric.md` offer-detection fallback over the final message (haiku;
|
||||||
|
stubbable via `OS_VAULT_EVAL_JUDGE_CMD`). TSV records `A-via:written|offered`.
|
||||||
|
- **Axis (b) — contract conformance (positives)**: if written — mechanical: slug-only
|
||||||
|
lowercase-kebab filename (no date prefix), YAML frontmatter with non-empty `summary`,
|
||||||
|
`scope` ∈ {global, project, client}, `title`, `type` field with a matching `type/<type>`
|
||||||
|
tag. Missing other-facet tags / `last_updated` / `date` are recorded as informational
|
||||||
|
`b-info:` flags, NOT fails (the shipped skill wording only *recommends* them — a checker
|
||||||
|
stricter than the instructions would measure instruction-disobedience). If offer-only:
|
||||||
|
`B-via:offered` (nothing written to check).
|
||||||
|
- **Negatives**: the SOLE fail line is an unneeded vault write (any new/content-modified
|
||||||
|
non-journal note in the sandbox vault). Project-file writes, replies, repo-doc updates
|
||||||
|
are all PASS behavior; recorded informationally (`project-write:yes/no`).
|
||||||
|
- **Both polarities**: `real-vault-breach` (transcript writes under the real vault path) is
|
||||||
|
an automatic FAIL with a loud reason.
|
||||||
|
|
||||||
|
**Overall PASS** — positives: both axes; negatives: no vault write (+ no breach).
|
||||||
|
|
||||||
|
TSV: `scenario model A:... B:... PASS|FAIL reasons`
|
||||||
|
|
||||||
|
## Running
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# one cell (canary first — mandatory for the first-ever live run; count its result)
|
||||||
|
bin/run P1-L1-reporting sonnet /tmp/vault-eval --reps 1
|
||||||
|
cat /tmp/vault-eval/results.tsv
|
||||||
|
|
||||||
|
# full baseline grid: run-set × {sonnet, haiku} × 3 reps, parallel
|
||||||
|
for s in P1-L1-reporting N1-L1-reporting P2-L2-reporting N2-L2-reporting P3-L3-reporting N3-L3-reporting; do
|
||||||
|
for m in sonnet haiku; do
|
||||||
|
bin/run "$s" "$m" /tmp/vault-eval-grid --reps 3 --results /tmp/vault-eval-grid/results.tsv &
|
||||||
|
done
|
||||||
|
done
|
||||||
|
wait
|
||||||
|
```
|
||||||
|
|
||||||
|
Each rep: fresh sandbox (project git-inited + pristine vault copy) → `claude -p
|
||||||
|
--model <tier> --output-format stream-json --verbose --dangerously-skip-permissions` with
|
||||||
|
cwd = sandbox project and the isolation env set → `transcript.jsonl` → `bin/check` appends
|
||||||
|
one TSV row (guarded with `|| true`; check row counts == expected reps before reading
|
||||||
|
results — the Eval C pipefail lesson).
|
||||||
|
|
||||||
|
## Before the first grid (mandatory discipline gates, from the WS2 plan)
|
||||||
|
|
||||||
|
1. Conformance dry-run on paper (done at design time; recorded in scenario files): a
|
||||||
|
perfectly write-skill-compliant model passes every negative; an always-write model
|
||||||
|
passes positives and fails negatives.
|
||||||
|
2. `bin/self-test` green — model-free, fabricates transcripts/sandboxes both directions,
|
||||||
|
including a shipped-instruction-compliant positive (offer-only with stubbed judge) and
|
||||||
|
a journal-only negative (must PASS — journal writes are hook-automatic).
|
||||||
|
3. Canary cell first; hand-verify its TSV row against the raw transcript; count it.
|
||||||
|
4. `bin/refresh-plugins` before every grid; verify caches carry the OS_VAULT_PATH-aware
|
||||||
|
skill wording.
|
||||||
|
|
||||||
|
## Layout
|
||||||
|
|
||||||
|
| Path | What |
|
||||||
|
| --- | --- |
|
||||||
|
| `fixture/project/` | reportgen Ruby CLI + 3-ADR history + plain CLAUDE.md |
|
||||||
|
| `fixture/vault/` | isolated vault: frozen vault-conventions.md + 4 seeded notes + journal/ |
|
||||||
|
| `scenarios/` / `scenarios-reserve/` | run-set / frozen reserve (6 + 6, held-out Task blocks) |
|
||||||
|
| `bin/sandbox <scenario> <dest>` | fresh sandbox: git-inited project copy + pristine vault copy |
|
||||||
|
| `bin/run <scenario> <model> <workdir> [--reps N] [--results F]` | headless runner (the ONLY valid execution mode) |
|
||||||
|
| `bin/check <scenario> <sandbox> [--tsv <model>]` | two-axis checker; exit 0/1; TSV mode |
|
||||||
|
| `bin/self-test [workdir]` | model-free both-directions harness validation |
|
||||||
|
| `judge-rubric.md` | frozen offer-detection rubric (positive axis (a) fallback only) |
|
||||||
|
|
||||||
|
## Why headless-only
|
||||||
|
|
||||||
|
This eval measures **unprompted** behavior: each rep needs a fresh session where the real
|
||||||
|
SessionStart hooks fire and the os-vault skill descriptions are loaded exactly as in
|
||||||
|
production. In-session Agent-tool subagents inherit the parent session and are invalid here
|
||||||
|
(same rule as os-adr Eval B/C; see
|
||||||
|
`~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md`).
|
||||||
|
|
||||||
|
## After the baseline
|
||||||
|
|
||||||
|
Results go to a vault `eval-results` note (template `_templates/eval-results.md`) and the
|
||||||
|
cc-os CLAUDE.md pointer list. If the gap is confirmed, the `/autoresearch` wording loop runs
|
||||||
|
per the howto note: checker/fixtures/scenarios/rubric frozen, wording only, reduced inner
|
||||||
|
grid, full grid to confirm — and the run-set is thereafter contaminated; measurement moves
|
||||||
|
to the reserve set.
|
||||||
|
|
@ -0,0 +1,493 @@
|
||||||
|
#!/usr/bin/env ruby
|
||||||
|
# frozen_string_literal: true
|
||||||
|
|
||||||
|
# Two-axis, deterministic-first checker for the os-vault write-behavior eval.
|
||||||
|
#
|
||||||
|
# check <P*-L*|N*-L*> <sandbox-root> [--tsv <model-label>]
|
||||||
|
#
|
||||||
|
# Reads <sandbox>/transcript.jsonl (stream-json, written by bin/run) and diffs
|
||||||
|
# <sandbox>/vault against eval/fixture/vault.
|
||||||
|
#
|
||||||
|
# POSITIVE scenarios (should write/offer durable cross-project knowledge):
|
||||||
|
# axis (a) — write trigger: mechanical PASS iff a new/content-modified
|
||||||
|
# non-journal vault note matches ALL of the scenario's topic regexes;
|
||||||
|
# else the frozen judge-rubric offer-detection fallback over the final
|
||||||
|
# message.
|
||||||
|
# axis (b) — contract conformance: if written, mechanical frontmatter/
|
||||||
|
# filename checks (missing optional fields are informational only);
|
||||||
|
# if offer-only, PASS via B-via:offered.
|
||||||
|
#
|
||||||
|
# NEGATIVE scenarios (should NOT write):
|
||||||
|
# sole fail condition is any qualifying vault write. Axis (a) label in TSV
|
||||||
|
# carries W:yes|no (any write at all, not topic-gated) for readability;
|
||||||
|
# overall PASS = axis (b) PASS (no write) and no breach.
|
||||||
|
#
|
||||||
|
# Both polarities: a transcript write targeting the REAL vault path
|
||||||
|
# (~/Documents/SecondBrain or /home/jared/Documents/SecondBrain) is an
|
||||||
|
# automatic FAIL, reason `real-vault-breach`, regardless of polarity.
|
||||||
|
#
|
||||||
|
# See ../README.md ("Two-axis scoring") — authoritative on disagreement.
|
||||||
|
#
|
||||||
|
# Exit 0 on PASS, 1 on FAIL, 2 on usage error.
|
||||||
|
|
||||||
|
require "json"
|
||||||
|
require "open3"
|
||||||
|
require "yaml"
|
||||||
|
require "date"
|
||||||
|
|
||||||
|
module OsVaultEval
|
||||||
|
POSITIVE_META = {
|
||||||
|
"P1-L1-reporting" => {
|
||||||
|
topics: [/finch/i, /500|page_size|pagin|cap|truncat/i],
|
||||||
|
knowledge: "the Finch API silently caps responses at 500 records unless page_size is passed",
|
||||||
|
},
|
||||||
|
"P2-L2-reporting" => {
|
||||||
|
topics: [/finch|oauth/i, /expires_in|15\s*min|token|idle|inactiv/i],
|
||||||
|
knowledge: "Finch OAuth tokens really expire after ~15 minutes idle; the advertised expires_in=3600 is wrong",
|
||||||
|
},
|
||||||
|
"P3-L3-reporting" => {
|
||||||
|
topics: [/acme/i, /iso|date|week/i],
|
||||||
|
knowledge: "Acme requires ISO-8601 dates and ISO week numbers on every deliverable (standing client convention)",
|
||||||
|
},
|
||||||
|
"P4-L1-delivery" => {
|
||||||
|
topics: [/chartkit/i, /stream|segfault|10.?k|10,?000|points/i],
|
||||||
|
knowledge: "chartkit's SVG renderer segfaults above ~10k points unless streaming: true",
|
||||||
|
},
|
||||||
|
"P5-L2-delivery" => {
|
||||||
|
topics: [/chartkit/i, /font|embed|system|fallback|helvetica/i],
|
||||||
|
knowledge: "chartkit embeds fonts only when installed system-wide; vendored fonts silently fall back",
|
||||||
|
},
|
||||||
|
"P6-L3-delivery" => {
|
||||||
|
topics: [/globex/i, /encrypt|zip|password|archive/i],
|
||||||
|
knowledge: "Globex only accepts deliverables as encrypted zips with per-quarter passwords (standing client convention)",
|
||||||
|
},
|
||||||
|
}.freeze
|
||||||
|
|
||||||
|
NEGATIVE_IDS = %w[
|
||||||
|
N1-L1-reporting N2-L2-reporting N3-L3-reporting
|
||||||
|
N4-L1-delivery N5-L2-delivery N6-L3-delivery
|
||||||
|
].freeze
|
||||||
|
|
||||||
|
REAL_VAULT_PATTERN = /(?:\/home\/jared\/Documents\/SecondBrain|~\/Documents\/SecondBrain)/.freeze
|
||||||
|
WRITE_TOOLS = %w[Write Edit MultiEdit NotebookEdit].freeze
|
||||||
|
|
||||||
|
# --- transcript ------------------------------------------------------------
|
||||||
|
|
||||||
|
class Transcript
|
||||||
|
def self.load(path)
|
||||||
|
return nil unless File.exist?(path)
|
||||||
|
lines = File.readlines(path)
|
||||||
|
events = lines.filter_map do |line|
|
||||||
|
next if line.strip.empty?
|
||||||
|
JSON.parse(line)
|
||||||
|
rescue JSON::ParserError
|
||||||
|
nil
|
||||||
|
end
|
||||||
|
return nil if events.empty?
|
||||||
|
new(events)
|
||||||
|
end
|
||||||
|
|
||||||
|
def initialize(events) = @events = events
|
||||||
|
|
||||||
|
def tool_uses
|
||||||
|
assistant_blocks.select { |b| b["type"] == "tool_use" }
|
||||||
|
end
|
||||||
|
|
||||||
|
def final_message
|
||||||
|
result = @events.reverse.find { |e| e["type"] == "result" }
|
||||||
|
result && result["result"].is_a?(String) ? result["result"] : nil
|
||||||
|
end
|
||||||
|
|
||||||
|
private
|
||||||
|
|
||||||
|
def assistant_blocks
|
||||||
|
@events.select { |e| e["type"] == "assistant" }
|
||||||
|
.flat_map { |e| e.dig("message", "content") || [] }
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
# --- breach scan -------------------------------------------------------------
|
||||||
|
|
||||||
|
class BreachScanner
|
||||||
|
def initialize(transcript) = @transcript = transcript
|
||||||
|
|
||||||
|
def breach?
|
||||||
|
@transcript.tool_uses.any? { |tu| breach_tool_use?(tu) }
|
||||||
|
end
|
||||||
|
|
||||||
|
private
|
||||||
|
|
||||||
|
def breach_tool_use?(tool_use)
|
||||||
|
name = tool_use["name"]
|
||||||
|
input = tool_use["input"] || {}
|
||||||
|
return write_target_breach?(input) if WRITE_TOOLS.include?(name)
|
||||||
|
return bash_breach?(input["command"].to_s) if name == "Bash"
|
||||||
|
|
||||||
|
false
|
||||||
|
end
|
||||||
|
|
||||||
|
def write_target_breach?(input)
|
||||||
|
target = input["file_path"] || input["notebook_path"] || ""
|
||||||
|
target.to_s.match?(REAL_VAULT_PATTERN)
|
||||||
|
end
|
||||||
|
|
||||||
|
def bash_breach?(command)
|
||||||
|
return false unless command.match?(REAL_VAULT_PATTERN)
|
||||||
|
|
||||||
|
# Plain reads (cat/ls/grep/find) of the real vault are not a breach —
|
||||||
|
# only flag when the command also writes INTO that path.
|
||||||
|
command.match?(/>>?[^&]*(?:SecondBrain)/) ||
|
||||||
|
command.match?(/\btee\b[^|]*SecondBrain/) ||
|
||||||
|
command.match?(/\b(?:cp|mv|rsync)\b.*SecondBrain/)
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
# --- vault diff --------------------------------------------------------------
|
||||||
|
|
||||||
|
class VaultDiff
|
||||||
|
def initialize(fixture_vault_root, sandbox_vault_root)
|
||||||
|
@fixture_root = fixture_vault_root
|
||||||
|
@sandbox_root = sandbox_vault_root
|
||||||
|
end
|
||||||
|
|
||||||
|
# Relative paths (under the vault root) that are new or content-modified
|
||||||
|
# relative to the fixture vault, excluding journal/, graphify-out/, and
|
||||||
|
# dotfiles/dot-dirs.
|
||||||
|
def changed_files
|
||||||
|
candidate_files.select { |rel| new_or_modified?(rel) }
|
||||||
|
end
|
||||||
|
|
||||||
|
private
|
||||||
|
|
||||||
|
def candidate_files
|
||||||
|
return [] unless Dir.exist?(@sandbox_root)
|
||||||
|
|
||||||
|
Dir.glob(File.join(@sandbox_root, "**", "*.md"))
|
||||||
|
.map { |path| path.sub("#{@sandbox_root}/", "") }
|
||||||
|
.reject { |rel| excluded?(rel) }
|
||||||
|
end
|
||||||
|
|
||||||
|
def excluded?(rel)
|
||||||
|
parts = rel.split("/")
|
||||||
|
parts.first == "journal" || parts.include?("graphify-out") ||
|
||||||
|
parts.any? { |p| p.start_with?(".") }
|
||||||
|
end
|
||||||
|
|
||||||
|
def new_or_modified?(rel)
|
||||||
|
fixture_path = File.join(@fixture_root, rel)
|
||||||
|
sandbox_path = File.join(@sandbox_root, rel)
|
||||||
|
!File.exist?(fixture_path) || File.read(fixture_path) != File.read(sandbox_path)
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
# --- frontmatter conformance --------------------------------------------------
|
||||||
|
|
||||||
|
class ContractCheck
|
||||||
|
FACET_PREFIX = %r{\A(?:client|project|domain|tool|convention)/}.freeze
|
||||||
|
|
||||||
|
def self.run(path)
|
||||||
|
new(path).run
|
||||||
|
end
|
||||||
|
|
||||||
|
def initialize(path)
|
||||||
|
@path = path
|
||||||
|
@content = File.read(path)
|
||||||
|
end
|
||||||
|
|
||||||
|
# Returns [fails, informational] — both arrays of reason strings.
|
||||||
|
def run
|
||||||
|
fails = []
|
||||||
|
fails.concat(filename_fails)
|
||||||
|
fm = frontmatter
|
||||||
|
if fm.nil?
|
||||||
|
fails << "no parseable YAML frontmatter block"
|
||||||
|
return [fails, []]
|
||||||
|
end
|
||||||
|
fails.concat(field_fails(fm))
|
||||||
|
[fails, informational(fm)]
|
||||||
|
end
|
||||||
|
|
||||||
|
private
|
||||||
|
|
||||||
|
def filename_fails
|
||||||
|
basename = File.basename(@path)
|
||||||
|
fails = []
|
||||||
|
fails << "filename not slug-only kebab-case: #{basename}" unless basename.match?(/\A[a-z0-9][a-z0-9-]*\.md\z/)
|
||||||
|
fails << "filename carries a date prefix: #{basename}" if basename.match?(/\A\d{4}-\d{2}-\d{2}-/)
|
||||||
|
fails
|
||||||
|
end
|
||||||
|
|
||||||
|
def frontmatter
|
||||||
|
match = @content.match(/\A---\n(.*?)\n---\n/m)
|
||||||
|
return nil unless match
|
||||||
|
|
||||||
|
YAML.safe_load(match[1], permitted_classes: [Date, Time])
|
||||||
|
rescue Psych::SyntaxError
|
||||||
|
nil
|
||||||
|
end
|
||||||
|
|
||||||
|
def field_fails(fm)
|
||||||
|
return ["frontmatter is not a YAML mapping"] unless fm.is_a?(Hash)
|
||||||
|
|
||||||
|
fails = []
|
||||||
|
fails << "missing or empty summary" unless present?(fm["summary"])
|
||||||
|
fails << "missing or empty title" unless present?(fm["title"])
|
||||||
|
fails << "missing or empty type" unless present?(fm["type"])
|
||||||
|
fails << "scope #{fm['scope'].inspect} not one of global/project/client" unless %w[global project
|
||||||
|
client].include?(fm["scope"])
|
||||||
|
fails << "tags missing type/#{fm['type']}" if present?(fm["type"]) && !tags(fm).include?("type/#{fm['type']}")
|
||||||
|
fails
|
||||||
|
end
|
||||||
|
|
||||||
|
def informational(fm)
|
||||||
|
info = []
|
||||||
|
info << "b-info:missing-facet-tag" unless tags(fm).any? { |t| t.match?(FACET_PREFIX) }
|
||||||
|
info << "b-info:missing-last_updated" unless present?(fm["last_updated"])
|
||||||
|
info << "b-info:missing-date" unless present?(fm["date"])
|
||||||
|
info
|
||||||
|
end
|
||||||
|
|
||||||
|
def tags(fm) = Array(fm["tags"]).map(&:to_s)
|
||||||
|
|
||||||
|
def present?(value) = !(value.nil? || (value.respond_to?(:empty?) && value.to_s.empty?))
|
||||||
|
end
|
||||||
|
|
||||||
|
# --- offer-detection judge -----------------------------------------------------
|
||||||
|
|
||||||
|
class Judge
|
||||||
|
def self.rubric_template
|
||||||
|
@rubric_template ||= begin
|
||||||
|
text = File.read(File.expand_path("../judge-rubric.md", __dir__))
|
||||||
|
start = text.index("You are scoring")
|
||||||
|
raise "judge-rubric.md missing expected prompt text" if start.nil?
|
||||||
|
|
||||||
|
text[start..]
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def self.command
|
||||||
|
ENV.fetch("OS_VAULT_EVAL_JUDGE_CMD", "claude -p --model haiku --dangerously-skip-permissions")
|
||||||
|
end
|
||||||
|
|
||||||
|
def offered?(final_message, knowledge)
|
||||||
|
prompt = self.class.rubric_template
|
||||||
|
.gsub("{FINAL_MESSAGE}", final_message.to_s)
|
||||||
|
.gsub("{KNOWLEDGE}", knowledge)
|
||||||
|
output, status = Open3.capture2(self.class.command, stdin_data: prompt)
|
||||||
|
raise "judge command failed: #{self.class.command}" unless status.success?
|
||||||
|
|
||||||
|
verdict = output[/\b(PASS|FAIL)\b/, 1]
|
||||||
|
raise "judge gave no PASS/FAIL verdict: #{output.inspect}" if verdict.nil?
|
||||||
|
|
||||||
|
verdict == "PASS"
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
# --- sandbox -------------------------------------------------------------------
|
||||||
|
|
||||||
|
class Sandbox
|
||||||
|
def initialize(root, fixture_vault_root)
|
||||||
|
@root = root
|
||||||
|
@fixture_vault_root = fixture_vault_root
|
||||||
|
end
|
||||||
|
|
||||||
|
attr_reader :root
|
||||||
|
|
||||||
|
def vault_root = File.join(@root, "vault")
|
||||||
|
|
||||||
|
def transcript = Transcript.load(File.join(@root, "transcript.jsonl"))
|
||||||
|
|
||||||
|
def vault_writes = VaultDiff.new(@fixture_vault_root, vault_root).changed_files
|
||||||
|
|
||||||
|
def any_project_write?(transcript)
|
||||||
|
transcript.tool_uses.any? do |tu|
|
||||||
|
WRITE_TOOLS.include?(tu["name"]) || bash_write?(tu)
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
private
|
||||||
|
|
||||||
|
def bash_write?(tool_use)
|
||||||
|
return false unless tool_use["name"] == "Bash"
|
||||||
|
|
||||||
|
command = (tool_use["input"] || {})["command"].to_s
|
||||||
|
command.match?(/>>?|(?:^|\s)(?:tee|cp|mv)\b/)
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
# --- result accumulator ---------------------------------------------------------
|
||||||
|
|
||||||
|
class Result
|
||||||
|
def initialize = @failures = []
|
||||||
|
attr_reader :failures
|
||||||
|
def pass? = @failures.empty?
|
||||||
|
|
||||||
|
def expect(condition, reason)
|
||||||
|
@failures << reason unless condition
|
||||||
|
!!condition
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
# --- report ------------------------------------------------------------------
|
||||||
|
|
||||||
|
Report = Struct.new(:axis_a, :axis_b, :a_via, :b_via, :informational, :consultation_value,
|
||||||
|
keyword_init: true)
|
||||||
|
|
||||||
|
# --- checkers --------------------------------------------------------------------
|
||||||
|
|
||||||
|
class PositiveChecker
|
||||||
|
def initialize(scenario_id, sandbox, judge: Judge.new)
|
||||||
|
@scenario_id = scenario_id
|
||||||
|
@sandbox = sandbox
|
||||||
|
@judge = judge
|
||||||
|
@meta = POSITIVE_META.fetch(scenario_id)
|
||||||
|
end
|
||||||
|
|
||||||
|
def run
|
||||||
|
axis_a = Result.new
|
||||||
|
axis_b = Result.new
|
||||||
|
transcript = @sandbox.transcript
|
||||||
|
|
||||||
|
if transcript.nil?
|
||||||
|
axis_a.expect(false, "harness-error:no-transcript")
|
||||||
|
axis_b.expect(false, "harness-error:no-transcript")
|
||||||
|
return Report.new(axis_a: axis_a, axis_b: axis_b, informational: [])
|
||||||
|
end
|
||||||
|
|
||||||
|
if BreachScanner.new(transcript).breach?
|
||||||
|
axis_a.expect(false, "real-vault-breach")
|
||||||
|
axis_b.expect(false, "real-vault-breach")
|
||||||
|
return Report.new(axis_a: axis_a, axis_b: axis_b, informational: [])
|
||||||
|
end
|
||||||
|
|
||||||
|
writes = @sandbox.vault_writes
|
||||||
|
matching = writes.select { |rel| topics_match?(File.join(@sandbox.vault_root, rel)) }
|
||||||
|
|
||||||
|
if matching.any?
|
||||||
|
check_written(axis_a, axis_b, matching.first)
|
||||||
|
else
|
||||||
|
check_offered(axis_a, axis_b, transcript)
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
private
|
||||||
|
|
||||||
|
def topics_match?(path)
|
||||||
|
content = "#{File.basename(path)}\n#{File.read(path)}"
|
||||||
|
@meta[:topics].all? { |re| content.match?(re) }
|
||||||
|
end
|
||||||
|
|
||||||
|
def check_written(axis_a, axis_b, rel_path)
|
||||||
|
axis_a.expect(true, nil)
|
||||||
|
fails, informational = ContractCheck.run(File.join(@sandbox.vault_root, rel_path))
|
||||||
|
fails.each { |f| axis_b.expect(false, f) }
|
||||||
|
axis_b.expect(true, nil) if fails.empty?
|
||||||
|
Report.new(axis_a: axis_a, axis_b: axis_b, a_via: "written",
|
||||||
|
b_via: fails.empty? ? "conforming" : nil, informational: informational)
|
||||||
|
end
|
||||||
|
|
||||||
|
def check_offered(axis_a, axis_b, transcript)
|
||||||
|
final = transcript.final_message
|
||||||
|
if final && @judge.offered?(final, @meta[:knowledge])
|
||||||
|
axis_a.expect(true, nil)
|
||||||
|
axis_b.expect(true, nil)
|
||||||
|
Report.new(axis_a: axis_a, axis_b: axis_b, a_via: "offered", b_via: "offered", informational: [])
|
||||||
|
else
|
||||||
|
axis_a.expect(false, "no qualifying vault write and no vault-write offer in final message")
|
||||||
|
axis_b.expect(false, "axis (a) failed — nothing to check for contract conformance")
|
||||||
|
Report.new(axis_a: axis_a, axis_b: axis_b, informational: [])
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
class NegativeChecker
|
||||||
|
def initialize(scenario_id, sandbox, judge: nil)
|
||||||
|
@scenario_id = scenario_id
|
||||||
|
@sandbox = sandbox
|
||||||
|
end
|
||||||
|
|
||||||
|
def run
|
||||||
|
axis_a = Result.new
|
||||||
|
axis_b = Result.new
|
||||||
|
transcript = @sandbox.transcript
|
||||||
|
|
||||||
|
if transcript.nil?
|
||||||
|
axis_a.expect(false, "harness-error:no-transcript")
|
||||||
|
axis_b.expect(false, "harness-error:no-transcript")
|
||||||
|
return Report.new(axis_a: axis_a, axis_b: axis_b, informational: [])
|
||||||
|
end
|
||||||
|
|
||||||
|
if BreachScanner.new(transcript).breach?
|
||||||
|
axis_a.expect(false, "real-vault-breach")
|
||||||
|
axis_b.expect(false, "real-vault-breach")
|
||||||
|
return Report.new(axis_a: axis_a, axis_b: axis_b, informational: [])
|
||||||
|
end
|
||||||
|
|
||||||
|
writes = @sandbox.vault_writes
|
||||||
|
axis_b.expect(writes.empty?, "wrote:#{writes.join(',')}") unless writes.empty?
|
||||||
|
project_write = @sandbox.any_project_write?(transcript) ? "yes" : "no"
|
||||||
|
|
||||||
|
Report.new(
|
||||||
|
axis_a: axis_a, axis_b: axis_b,
|
||||||
|
consultation_value: writes.empty? ? "no" : "yes",
|
||||||
|
informational: ["project-write:#{project_write}"],
|
||||||
|
)
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
# --- CLI -------------------------------------------------------------------------
|
||||||
|
|
||||||
|
scenario_id = ARGV[0]
|
||||||
|
sandbox_root = ARGV[1]
|
||||||
|
tsv_model = ARGV[2] == "--tsv" ? (ARGV[3] || "unknown") : nil
|
||||||
|
|
||||||
|
known = OsVaultEval::POSITIVE_META.keys + OsVaultEval::NEGATIVE_IDS
|
||||||
|
if scenario_id.nil? || sandbox_root.nil? || !known.include?(scenario_id)
|
||||||
|
warn "usage: check <#{known.join('|')}> <sandbox-root> [--tsv <model>]"
|
||||||
|
exit 2
|
||||||
|
end
|
||||||
|
unless File.directory?(sandbox_root)
|
||||||
|
warn "no such sandbox: #{sandbox_root}"
|
||||||
|
exit 2
|
||||||
|
end
|
||||||
|
|
||||||
|
eval_root = File.expand_path("..", __dir__)
|
||||||
|
# OS_VAULT_FIXTURE_VAULT lets bin/self-test point at a synthesized stand-in fixture
|
||||||
|
# when eval/fixture/vault doesn't exist yet; unset in production (default applies).
|
||||||
|
fixture_vault_root = ENV.fetch("OS_VAULT_FIXTURE_VAULT", File.join(eval_root, "fixture", "vault"))
|
||||||
|
sandbox = OsVaultEval::Sandbox.new(File.expand_path(sandbox_root), fixture_vault_root)
|
||||||
|
is_negative = OsVaultEval::NEGATIVE_IDS.include?(scenario_id)
|
||||||
|
checker_class = is_negative ? OsVaultEval::NegativeChecker : OsVaultEval::PositiveChecker
|
||||||
|
report = checker_class.new(scenario_id, sandbox).run
|
||||||
|
|
||||||
|
overall = is_negative ? report.axis_b.pass? : (report.axis_a.pass? && report.axis_b.pass?)
|
||||||
|
|
||||||
|
reasons = (report.axis_a.failures + report.axis_b.failures).dup
|
||||||
|
reasons.unshift("B-via:#{report.b_via}") if report.b_via
|
||||||
|
reasons.unshift("A-via:#{report.a_via}") if report.a_via
|
||||||
|
reasons.concat(Array(report.informational))
|
||||||
|
reasons = reasons.compact
|
||||||
|
|
||||||
|
if tsv_model
|
||||||
|
axis_a_label = if is_negative
|
||||||
|
"W:#{report.consultation_value == 'yes' ? 'yes' : 'no'}"
|
||||||
|
else
|
||||||
|
report.axis_a.pass? ? "A:PASS" : "A:FAIL"
|
||||||
|
end
|
||||||
|
puts [scenario_id, tsv_model, axis_a_label,
|
||||||
|
report.axis_b.pass? ? "B:PASS" : "B:FAIL",
|
||||||
|
overall ? "PASS" : "FAIL", reasons.join("; ")].join("\t")
|
||||||
|
else
|
||||||
|
axis_a_desc = if is_negative
|
||||||
|
"vault-write:#{report.consultation_value == 'yes' ? 'yes' : 'no'}"
|
||||||
|
else
|
||||||
|
report.axis_a.pass? ? "PASS" : "FAIL"
|
||||||
|
end
|
||||||
|
puts "#{overall ? 'PASS' : 'FAIL'} #{scenario_id} " \
|
||||||
|
"(axis-a #{axis_a_desc}, axis-b #{report.axis_b.pass? ? 'PASS' : 'FAIL'})"
|
||||||
|
(report.axis_a.failures + report.axis_b.failures).each { |f| puts " - #{f}" }
|
||||||
|
reasons.each { |r| puts " * #{r}" } unless reasons.empty?
|
||||||
|
end
|
||||||
|
|
||||||
|
exit(overall ? 0 : 1)
|
||||||
|
|
@ -0,0 +1,58 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# Headless runner for the os-vault write-behavior eval — the ONLY valid execution mode
|
||||||
|
# (fresh `claude -p` with cwd = sandbox/project so the real SessionStart hook fires).
|
||||||
|
#
|
||||||
|
# Usage: run <scenario> <model> <workdir> [--reps N] [--results FILE]
|
||||||
|
#
|
||||||
|
# scenario P*-L*|N*-L* (e.g., P1-L1-reporting, N4-L1-delivery)
|
||||||
|
# model haiku|sonnet|opus|...
|
||||||
|
# workdir sandboxes created under here as <scenario>-<model>-rN
|
||||||
|
# --reps repeated executions (default 1)
|
||||||
|
#
|
||||||
|
# Each rep: fresh sandbox -> claude -p (cwd = sandbox/project, OS_VAULT_PATH +
|
||||||
|
# OS_VAULT_SKIP_REBUILD set for that invocation only) -> full stream-json transcript
|
||||||
|
# saved to <sandbox>/transcript.jsonl -> bin/check appends one TSV row.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCENARIO="${1:?usage: run <scenario> <model> <workdir> [--reps N] [--results FILE]}"
|
||||||
|
MODEL="${2:?model required (haiku|sonnet|...)}"
|
||||||
|
WORKDIR="${3:?workdir required}"
|
||||||
|
shift 3
|
||||||
|
|
||||||
|
REPS=1
|
||||||
|
RESULTS=""
|
||||||
|
while [ $# -gt 0 ]; do
|
||||||
|
case "$1" in
|
||||||
|
--reps) REPS="${2:?--reps needs a number}"; shift 2 ;;
|
||||||
|
--results) RESULTS="${2:?--results needs a path}"; shift 2 ;;
|
||||||
|
*) echo "unknown option: $1" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
EVAL_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||||
|
SCENARIO_FILE="$EVAL_ROOT/scenarios/${SCENARIO}.md"
|
||||||
|
if [ ! -f "$SCENARIO_FILE" ]; then
|
||||||
|
SCENARIO_FILE="$EVAL_ROOT/scenarios-reserve/${SCENARIO}.md"
|
||||||
|
fi
|
||||||
|
[ -f "$SCENARIO_FILE" ] || { echo "unknown scenario: $SCENARIO (checked scenarios/ and scenarios-reserve/)" >&2; exit 2; }
|
||||||
|
mkdir -p "$WORKDIR"
|
||||||
|
RESULTS="${RESULTS:-$WORKDIR/results.tsv}"
|
||||||
|
|
||||||
|
TASK="$(awk '/^## Task/{found=1; next} found' "$SCENARIO_FILE")"
|
||||||
|
|
||||||
|
for rep in $(seq 1 "$REPS"); do
|
||||||
|
SANDBOX="$WORKDIR/$SCENARIO-$MODEL-r$rep"
|
||||||
|
"$EVAL_ROOT/bin/sandbox" "$SCENARIO" "$SANDBOX" >/dev/null
|
||||||
|
|
||||||
|
# cwd = sandbox/project: the SessionStart hook resolves the project root from here.
|
||||||
|
# OS_VAULT_PATH/OS_VAULT_SKIP_REBUILD are exported only for this one command.
|
||||||
|
(cd "$SANDBOX/project" && OS_VAULT_PATH="$SANDBOX/vault" OS_VAULT_SKIP_REBUILD=1 claude -p \
|
||||||
|
--model "$MODEL" \
|
||||||
|
--output-format stream-json --verbose \
|
||||||
|
--dangerously-skip-permissions \
|
||||||
|
"$TASK" > "$SANDBOX/transcript.jsonl") || echo "claude exited non-zero for $SANDBOX" >&2
|
||||||
|
|
||||||
|
# bin/check exits 1 on FAIL; without the || true, set -euo pipefail would
|
||||||
|
# abort the rep loop on the first failing rep (the eval-c pipefail lesson).
|
||||||
|
"$EVAL_ROOT/bin/check" "$SCENARIO" "$SANDBOX" --tsv "$MODEL" | tee -a "$RESULTS" || true
|
||||||
|
done
|
||||||
|
|
@ -0,0 +1,35 @@
|
||||||
|
#!/bin/bash
|
||||||
|
# Create a fresh, git-initialized sandbox copy of the Eval (os-vault write-behavior) fixture.
|
||||||
|
# Usage: sandbox <P*-L*|N*-L*> <dest-dir>
|
||||||
|
#
|
||||||
|
# <dest>/project — recursive copy of fixture/project, git-inited with one baseline commit.
|
||||||
|
# <dest>/vault — recursive copy of fixture/vault (the isolated SecondBrain stand-in).
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCENARIO="${1:?usage: sandbox <scenario> <dest-dir>}"
|
||||||
|
DEST="${2:?usage: sandbox <scenario> <dest-dir>}"
|
||||||
|
EVAL_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||||
|
FIXTURE_PROJECT="$EVAL_ROOT/fixture/project"
|
||||||
|
FIXTURE_VAULT="$EVAL_ROOT/fixture/vault"
|
||||||
|
|
||||||
|
# Validate scenario against both the run-set and the frozen reserve-set.
|
||||||
|
if [ ! -f "$EVAL_ROOT/scenarios/${SCENARIO}.md" ] && [ ! -f "$EVAL_ROOT/scenarios-reserve/${SCENARIO}.md" ]; then
|
||||||
|
echo "unknown scenario: $SCENARIO (checked scenarios/ and scenarios-reserve/)" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ -e "$DEST" ]; then echo "refusing to overwrite existing $DEST" >&2; exit 2; fi
|
||||||
|
mkdir -p "$DEST"
|
||||||
|
|
||||||
|
mkdir -p "$DEST/project"
|
||||||
|
cp -r "$FIXTURE_PROJECT/." "$DEST/project/"
|
||||||
|
rm -rf "$DEST/project/.git"
|
||||||
|
|
||||||
|
git -C "$DEST/project" init -q
|
||||||
|
git -C "$DEST/project" add -A
|
||||||
|
git -C "$DEST/project" -c user.email=eval@local -c user.name=eval commit -qm "fixture baseline"
|
||||||
|
|
||||||
|
mkdir -p "$DEST/vault"
|
||||||
|
cp -r "$FIXTURE_VAULT/." "$DEST/vault/"
|
||||||
|
|
||||||
|
echo "$DEST"
|
||||||
|
|
@ -0,0 +1,317 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# Self-test the os-vault write-behavior eval harness, model-free:
|
||||||
|
#
|
||||||
|
# 1. P1 perfect write (conforming frontmatter) -> PASS, A-via:written.
|
||||||
|
# 2. P1 miss (untouched vault, judge stubbed FAIL) -> FAIL axis (a).
|
||||||
|
# 3. P1 offer-only, judge stubbed PASS -> PASS via A-via:offered/B-via:offered.
|
||||||
|
# 4. P2 written-but-malformed (date-prefixed filename, missing summary/scope)
|
||||||
|
# -> axis (a) PASS, axis (b) FAIL.
|
||||||
|
# 5. P3 written-minimal-compliant (no last_updated/date/extra facet tag)
|
||||||
|
# -> PASS overall, with b-info: reasons present (informational only).
|
||||||
|
# 6. N1 clean (project write only, vault untouched) -> PASS, project-write:yes.
|
||||||
|
# 7. N1 over-trigger (new vault note) -> FAIL, reasons include wrote:.
|
||||||
|
# 8. N3 journal-only vault change -> PASS (journal excluded from the diff).
|
||||||
|
# 9. Breach: transcript Write targeting the real vault path -> FAIL,
|
||||||
|
# real-vault-breach.
|
||||||
|
# 10. Harness error: sandbox without transcript.jsonl -> FAIL,
|
||||||
|
# harness-error:no-transcript.
|
||||||
|
#
|
||||||
|
# The judge is stubbed via OS_VAULT_EVAL_JUDGE_CMD so this never spends
|
||||||
|
# tokens or calls `claude`. Never touches the held-out scenario Task blocks.
|
||||||
|
#
|
||||||
|
# Usage: self-test [workdir] (default: a fresh mktemp dir)
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
EVAL_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||||
|
WORKDIR="${1:-$(mktemp -d /tmp/os-vault-eval-selftest.XXXXXX)}"
|
||||||
|
mkdir -p "$WORKDIR"
|
||||||
|
FAILED=0
|
||||||
|
|
||||||
|
# --- fixture: use the real one if present, else synthesize a minimal stand-in ---
|
||||||
|
|
||||||
|
FIXTURE_PROJECT="$EVAL_ROOT/fixture/project"
|
||||||
|
FIXTURE_VAULT="$EVAL_ROOT/fixture/vault"
|
||||||
|
|
||||||
|
if [ ! -d "$FIXTURE_PROJECT" ] || [ -z "$(find "$FIXTURE_PROJECT" -maxdepth 1 -mindepth 1 2>/dev/null)" ] || \
|
||||||
|
[ ! -d "$FIXTURE_VAULT" ] || [ -z "$(find "$FIXTURE_VAULT" -maxdepth 1 -mindepth 1 2>/dev/null)" ]; then
|
||||||
|
echo "note: real fixture missing/empty — synthesizing a minimal stand-in for self-test"
|
||||||
|
SYNTH="$WORKDIR/synth-fixture"
|
||||||
|
mkdir -p "$SYNTH/project/lib" "$SYNTH/vault/journal"
|
||||||
|
cat > "$SYNTH/project/CLAUDE.md" <<'EOF'
|
||||||
|
# reportgen (self-test stand-in fixture)
|
||||||
|
A small Ruby CLI. No memory/vault trigger wording.
|
||||||
|
EOF
|
||||||
|
cat > "$SYNTH/project/lib/fetcher.rb" <<'EOF'
|
||||||
|
class Fetcher
|
||||||
|
def fetch; end
|
||||||
|
end
|
||||||
|
EOF
|
||||||
|
cat > "$SYNTH/vault/vault-conventions.md" <<'EOF'
|
||||||
|
---
|
||||||
|
type: convention
|
||||||
|
title: Vault Conventions
|
||||||
|
summary: Frontmatter contract for SecondBrain notes.
|
||||||
|
tags: [type/convention]
|
||||||
|
scope: global
|
||||||
|
---
|
||||||
|
Frontmatter contract stand-in for self-test.
|
||||||
|
EOF
|
||||||
|
FIXTURE_PROJECT="$SYNTH/project"
|
||||||
|
FIXTURE_VAULT="$SYNTH/vault"
|
||||||
|
# bin/check resolves its fixture-vault comparison root from eval/fixture/vault by
|
||||||
|
# default; point it at the synth stand-in so the diff logic is genuinely exercised
|
||||||
|
# (not just diffed against the real, still-being-built fixture).
|
||||||
|
export OS_VAULT_FIXTURE_VAULT="$FIXTURE_VAULT"
|
||||||
|
fi
|
||||||
|
|
||||||
|
fixture_digest() {
|
||||||
|
(cd "$FIXTURE_VAULT" && find . -type f -print0 | sort -z | xargs -0 md5sum 2>/dev/null | md5sum)
|
||||||
|
}
|
||||||
|
DIGEST_BEFORE="$(fixture_digest)"
|
||||||
|
|
||||||
|
# --- sandbox fabrication (bypasses bin/sandbox's scenario-name validation so
|
||||||
|
# self-test can run even before the fixture ships full scenario coverage) ---
|
||||||
|
|
||||||
|
make_sandbox() { # $1 = dest
|
||||||
|
mkdir -p "$1/project" "$1/vault"
|
||||||
|
cp -r "$FIXTURE_PROJECT/." "$1/project/"
|
||||||
|
cp -r "$FIXTURE_VAULT/." "$1/vault/"
|
||||||
|
}
|
||||||
|
|
||||||
|
# --- transcript fabrication (same event shapes as os-adr eval-c self-test) ---
|
||||||
|
|
||||||
|
tool_use_event() { # $1 = tool name, $2 = input JSON
|
||||||
|
printf '{"type":"assistant","message":{"role":"assistant","content":[{"type":"tool_use","name":"%s","input":%s}]}}\n' "$1" "$2"
|
||||||
|
}
|
||||||
|
|
||||||
|
text_event() { # $1 = text
|
||||||
|
python3 -c 'import json,sys; print(json.dumps({"type":"assistant","message":{"role":"assistant","content":[{"type":"text","text":sys.argv[1]}]}}))' "$1"
|
||||||
|
}
|
||||||
|
|
||||||
|
result_event() { # $1 = final text
|
||||||
|
python3 -c 'import json,sys; print(json.dumps({"type":"result","subtype":"success","result":sys.argv[1]}))' "$1"
|
||||||
|
}
|
||||||
|
|
||||||
|
write_transcript() { # $1 = sandbox, then event lines on stdin
|
||||||
|
cat > "$1/transcript.jsonl"
|
||||||
|
}
|
||||||
|
|
||||||
|
expect() { # $1 = PASS|FAIL, $2 = label, then the check command
|
||||||
|
local want="$1" label="$2"; shift 2
|
||||||
|
local got=PASS
|
||||||
|
local out
|
||||||
|
out="$("$@" 2>&1)" || got=FAIL
|
||||||
|
if [ "$got" = "$want" ]; then
|
||||||
|
echo "ok $label ($want)"
|
||||||
|
else
|
||||||
|
echo "FAIL $label: wanted $want, got $got"
|
||||||
|
echo "$out" | sed 's/^/ /'
|
||||||
|
FAILED=1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
expect_contains() { # $1 = haystack, $2 = needle, $3 = label
|
||||||
|
case "$1" in
|
||||||
|
*"$2"*) echo "ok $3 (found: $2)" ;;
|
||||||
|
*) echo "FAIL $3: expected to find '$2' in: $1"; FAILED=1 ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
FINCH_CONTENT="---
|
||||||
|
title: Finch API 500 Record Cap
|
||||||
|
type: reference
|
||||||
|
summary: Finch silently caps every response at 500 records unless page_size is passed explicitly.
|
||||||
|
scope: global
|
||||||
|
tags:
|
||||||
|
- type/reference
|
||||||
|
- tool/finch
|
||||||
|
last_updated: 2026-07-06
|
||||||
|
date: 2026-07-06
|
||||||
|
---
|
||||||
|
|
||||||
|
The Finch API silently caps every response at 500 records unless \`page_size\` is passed
|
||||||
|
explicitly — no error, no warning."
|
||||||
|
|
||||||
|
# --- 1. P1 perfect write: conforming frontmatter, topic-matching content -> PASS ---
|
||||||
|
|
||||||
|
sb="$WORKDIR/p1-perfect"
|
||||||
|
make_sandbox "$sb"
|
||||||
|
printf '%s\n' "$FINCH_CONTENT" > "$sb/vault/finch-api-500-record-cap.md"
|
||||||
|
write_transcript "$sb" <<EOF
|
||||||
|
$(tool_use_event "Write" '{"file_path":"finch-api-500-record-cap.md","content":"..."}')
|
||||||
|
$(result_event "Saved a note on the Finch pagination cap to your vault, and fixed the fetcher.")
|
||||||
|
EOF
|
||||||
|
row="$(env OS_VAULT_EVAL_JUDGE_CMD="printf FAIL" "$EVAL_ROOT/bin/check" P1-L1-reporting "$sb" --tsv selftest || true)"
|
||||||
|
expect PASS "P1 perfect write" env OS_VAULT_EVAL_JUDGE_CMD="printf FAIL" "$EVAL_ROOT/bin/check" P1-L1-reporting "$sb"
|
||||||
|
expect_contains "$row" "A-via:written" "P1 perfect write records A-via:written"
|
||||||
|
|
||||||
|
# --- 2. P1 miss: untouched vault, judge stubbed FAIL -> FAIL axis (a) ---
|
||||||
|
|
||||||
|
sb="$WORKDIR/p1-miss"
|
||||||
|
make_sandbox "$sb"
|
||||||
|
write_transcript "$sb" <<EOF
|
||||||
|
$(tool_use_event "Edit" '{"file_path":"lib/fetcher.rb","old_string":"a","new_string":"b"}')
|
||||||
|
$(result_event "Fixed the fetcher.")
|
||||||
|
EOF
|
||||||
|
expect FAIL "P1 miss (untouched vault, judge FAIL)" \
|
||||||
|
env OS_VAULT_EVAL_JUDGE_CMD="printf FAIL" "$EVAL_ROOT/bin/check" P1-L1-reporting "$sb"
|
||||||
|
row="$(env OS_VAULT_EVAL_JUDGE_CMD="printf FAIL" "$EVAL_ROOT/bin/check" P1-L1-reporting "$sb" --tsv selftest || true)"
|
||||||
|
expect_contains "$row" "A:FAIL" "P1 miss records A:FAIL"
|
||||||
|
|
||||||
|
# --- 3. P1 offer-only, judge stubbed PASS -> PASS via A-via:offered/B-via:offered ---
|
||||||
|
|
||||||
|
sb="$WORKDIR/p1-offer"
|
||||||
|
make_sandbox "$sb"
|
||||||
|
write_transcript "$sb" <<EOF
|
||||||
|
$(tool_use_event "Edit" '{"file_path":"lib/fetcher.rb","old_string":"a","new_string":"b"}')
|
||||||
|
$(result_event "Fixed the fetcher to paginate explicitly. That Finch 500-record cap is worth remembering across projects — I can save it to your SecondBrain vault if you'd like.")
|
||||||
|
EOF
|
||||||
|
expect PASS "P1 offer-only (judge stubbed PASS)" \
|
||||||
|
env OS_VAULT_EVAL_JUDGE_CMD="printf PASS" "$EVAL_ROOT/bin/check" P1-L1-reporting "$sb"
|
||||||
|
row="$(env OS_VAULT_EVAL_JUDGE_CMD="printf PASS" "$EVAL_ROOT/bin/check" P1-L1-reporting "$sb" --tsv selftest || true)"
|
||||||
|
expect_contains "$row" "A-via:offered" "P1 offer-only records A-via:offered"
|
||||||
|
expect_contains "$row" "B-via:offered" "P1 offer-only records B-via:offered"
|
||||||
|
|
||||||
|
# --- 4. P2 written-but-malformed: date-prefixed filename, missing summary/scope ---
|
||||||
|
|
||||||
|
sb="$WORKDIR/p2-malformed"
|
||||||
|
make_sandbox "$sb"
|
||||||
|
cat > "$sb/vault/2026-07-06-finch-tokens.md" <<'EOF'
|
||||||
|
---
|
||||||
|
title: Finch OAuth Tokens
|
||||||
|
type: reference
|
||||||
|
tags:
|
||||||
|
- type/reference
|
||||||
|
---
|
||||||
|
|
||||||
|
Finch OAuth tokens actually expire after about 15 minutes idle — the advertised
|
||||||
|
expires_in=3600 is wrong; do not trust it.
|
||||||
|
EOF
|
||||||
|
write_transcript "$sb" <<EOF
|
||||||
|
$(tool_use_event "Write" '{"file_path":"2026-07-06-finch-tokens.md","content":"..."}')
|
||||||
|
$(result_event "Noted the Finch OAuth token quirk.")
|
||||||
|
EOF
|
||||||
|
row="$(env OS_VAULT_EVAL_JUDGE_CMD="printf FAIL" "$EVAL_ROOT/bin/check" P2-L2-reporting "$sb" --tsv selftest || true)"
|
||||||
|
expect FAIL "P2 written-but-malformed (axis a PASS, axis b FAIL)" \
|
||||||
|
env OS_VAULT_EVAL_JUDGE_CMD="printf FAIL" "$EVAL_ROOT/bin/check" P2-L2-reporting "$sb"
|
||||||
|
expect_contains "$row" "A:PASS" "P2 malformed still PASSes axis a"
|
||||||
|
expect_contains "$row" "B:FAIL" "P2 malformed FAILs axis b"
|
||||||
|
|
||||||
|
# --- 5. P3 written-minimal-compliant: no last_updated/date/extra facet tag -> PASS ---
|
||||||
|
|
||||||
|
sb="$WORKDIR/p3-minimal"
|
||||||
|
make_sandbox "$sb"
|
||||||
|
cat > "$sb/vault/acme-iso-week-dates.md" <<'EOF'
|
||||||
|
---
|
||||||
|
title: Acme ISO Week Dates
|
||||||
|
type: convention
|
||||||
|
summary: Acme requires ISO-8601 dates and ISO week numbers on every deliverable.
|
||||||
|
scope: client
|
||||||
|
tags:
|
||||||
|
- type/convention
|
||||||
|
---
|
||||||
|
|
||||||
|
Acme is a standing-convention client: every deliverable must carry ISO-8601 dates and
|
||||||
|
ISO week numbers.
|
||||||
|
EOF
|
||||||
|
write_transcript "$sb" <<EOF
|
||||||
|
$(tool_use_event "Write" '{"file_path":"acme-iso-week-dates.md","content":"..."}')
|
||||||
|
$(result_event "Recorded the Acme ISO week/date convention.")
|
||||||
|
EOF
|
||||||
|
row="$(env OS_VAULT_EVAL_JUDGE_CMD="printf FAIL" "$EVAL_ROOT/bin/check" P3-L3-reporting "$sb" --tsv selftest || true)"
|
||||||
|
expect PASS "P3 minimal-compliant (informational gaps must not fail)" \
|
||||||
|
env OS_VAULT_EVAL_JUDGE_CMD="printf FAIL" "$EVAL_ROOT/bin/check" P3-L3-reporting "$sb"
|
||||||
|
expect_contains "$row" "b-info:" "P3 minimal-compliant carries b-info: reasons"
|
||||||
|
|
||||||
|
# --- 6. N1 clean: project write only, vault untouched -> PASS, project-write:yes ---
|
||||||
|
|
||||||
|
sb="$WORKDIR/n1-clean"
|
||||||
|
make_sandbox "$sb"
|
||||||
|
echo "TODO: paginate fetcher" > "$sb/project/TODO.md"
|
||||||
|
write_transcript "$sb" <<EOF
|
||||||
|
$(tool_use_event "Write" '{"file_path":"TODO.md","content":"TODO: paginate fetcher"}')
|
||||||
|
$(result_event "Left a TODO for the pagination fix.")
|
||||||
|
EOF
|
||||||
|
row="$(env OS_VAULT_EVAL_JUDGE_CMD="printf FAIL" "$EVAL_ROOT/bin/check" N1-L1-reporting "$sb" --tsv selftest || true)"
|
||||||
|
expect PASS "N1 clean (project write only)" \
|
||||||
|
env OS_VAULT_EVAL_JUDGE_CMD="printf FAIL" "$EVAL_ROOT/bin/check" N1-L1-reporting "$sb"
|
||||||
|
expect_contains "$row" "project-write:yes" "N1 clean records project-write:yes"
|
||||||
|
|
||||||
|
# --- 7. N1 over-trigger: new vault note -> FAIL, reasons include wrote: ---
|
||||||
|
|
||||||
|
sb="$WORKDIR/n1-overtrigger"
|
||||||
|
make_sandbox "$sb"
|
||||||
|
cat > "$sb/vault/csv-export-refactor-status.md" <<'EOF'
|
||||||
|
---
|
||||||
|
title: CSV Export Refactor Status
|
||||||
|
type: howto
|
||||||
|
summary: In-progress notes on the CSV export refactor.
|
||||||
|
scope: project
|
||||||
|
tags:
|
||||||
|
- type/howto
|
||||||
|
---
|
||||||
|
|
||||||
|
Refactor in progress.
|
||||||
|
EOF
|
||||||
|
write_transcript "$sb" <<EOF
|
||||||
|
$(tool_use_event "Write" '{"file_path":"csv-export-refactor-status.md","content":"..."}')
|
||||||
|
$(result_event "Made a note about the in-progress refactor.")
|
||||||
|
EOF
|
||||||
|
row="$(env OS_VAULT_EVAL_JUDGE_CMD="printf FAIL" "$EVAL_ROOT/bin/check" N1-L1-reporting "$sb" --tsv selftest || true)"
|
||||||
|
expect FAIL "N1 over-trigger (wrote ephemeral note)" \
|
||||||
|
env OS_VAULT_EVAL_JUDGE_CMD="printf FAIL" "$EVAL_ROOT/bin/check" N1-L1-reporting "$sb"
|
||||||
|
expect_contains "$row" "wrote:" "N1 over-trigger records wrote:"
|
||||||
|
|
||||||
|
# --- 8. N3 journal-only: PASS (journal excluded from the diff) ---
|
||||||
|
|
||||||
|
sb="$WORKDIR/n3-journal-only"
|
||||||
|
make_sandbox "$sb"
|
||||||
|
mkdir -p "$sb/vault/journal"
|
||||||
|
echo "# 2026-07-06" > "$sb/vault/journal/2026-07-06.md"
|
||||||
|
write_transcript "$sb" <<EOF
|
||||||
|
$(tool_use_event "Read" '{"file_path":"docs/adr/0002-utc-storage.md"}')
|
||||||
|
$(result_event "Confirmed the ADR already covers UTC storage; no change needed.")
|
||||||
|
EOF
|
||||||
|
expect PASS "N3 journal-only (journal excluded)" \
|
||||||
|
env OS_VAULT_EVAL_JUDGE_CMD="printf FAIL" "$EVAL_ROOT/bin/check" N3-L3-reporting "$sb"
|
||||||
|
|
||||||
|
# --- 9. Breach: transcript Write targeting the real vault path -> FAIL ---
|
||||||
|
|
||||||
|
sb="$WORKDIR/breach"
|
||||||
|
make_sandbox "$sb"
|
||||||
|
write_transcript "$sb" <<EOF
|
||||||
|
$(tool_use_event "Write" '{"file_path":"/home/jared/Documents/SecondBrain/x.md","content":"..."}')
|
||||||
|
$(result_event "Saved a note.")
|
||||||
|
EOF
|
||||||
|
row="$(env OS_VAULT_EVAL_JUDGE_CMD="printf PASS" "$EVAL_ROOT/bin/check" P1-L1-reporting "$sb" --tsv selftest || true)"
|
||||||
|
expect FAIL "breach (Write to real vault path)" \
|
||||||
|
env OS_VAULT_EVAL_JUDGE_CMD="printf PASS" "$EVAL_ROOT/bin/check" P1-L1-reporting "$sb"
|
||||||
|
expect_contains "$row" "real-vault-breach" "breach records real-vault-breach"
|
||||||
|
|
||||||
|
# --- 10. Harness error: sandbox without transcript.jsonl -> FAIL ---
|
||||||
|
|
||||||
|
sb="$WORKDIR/no-transcript"
|
||||||
|
make_sandbox "$sb"
|
||||||
|
row="$("$EVAL_ROOT/bin/check" P1-L1-reporting "$sb" --tsv selftest 2>&1 || true)"
|
||||||
|
expect FAIL "harness error (no transcript.jsonl)" "$EVAL_ROOT/bin/check" P1-L1-reporting "$sb"
|
||||||
|
expect_contains "$row" "harness-error:no-transcript" "harness error records harness-error:no-transcript"
|
||||||
|
|
||||||
|
# --- fixture isolation: the canonical fixture must be untouched ---
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
DIGEST_AFTER="$(fixture_digest)"
|
||||||
|
if [ "$DIGEST_BEFORE" = "$DIGEST_AFTER" ]; then
|
||||||
|
echo "ok canonical fixture vault untouched by all sandbox operations"
|
||||||
|
else
|
||||||
|
echo "FAIL canonical fixture vault was modified"
|
||||||
|
FAILED=1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
if [ $FAILED -eq 0 ]; then
|
||||||
|
echo "All self-tests passed"
|
||||||
|
else
|
||||||
|
echo "Some self-tests failed"
|
||||||
|
fi
|
||||||
|
|
||||||
|
exit $FAILED
|
||||||
|
|
@ -0,0 +1,35 @@
|
||||||
|
# CLAUDE.md
|
||||||
|
|
||||||
|
## What this is
|
||||||
|
|
||||||
|
`reportgen` is a small Ruby CLI that pulls client account data from the Finch API and
|
||||||
|
renders/delivers periodic reports to clients. It runs as a scheduled job (cron on the ops
|
||||||
|
box); there is no server component.
|
||||||
|
|
||||||
|
## Module map
|
||||||
|
|
||||||
|
- `bin/reportgen` — CLI entry point; dispatches to the classes in `lib/reportgen/`.
|
||||||
|
- `lib/reportgen/client.rb` — Finch OAuth client (token fetch + caching).
|
||||||
|
- `lib/reportgen/fetcher.rb` — pulls account/transaction records from the Finch API.
|
||||||
|
- `lib/reportgen/formatter.rb` — date/number formatting helpers.
|
||||||
|
- `lib/reportgen/renderer.rb` — renders the report view (HTML) from fetched records.
|
||||||
|
- `lib/reportgen/charts.rb` — builds the charts embedded in a rendered report.
|
||||||
|
- `lib/reportgen/exporter.rb` — CSV export of a report's underlying data.
|
||||||
|
- `lib/reportgen/export_writer.rb` — newer CSV writer used by the exporter's next iteration.
|
||||||
|
- `lib/reportgen/delivery.rb` — packages a rendered report and sends it to the client.
|
||||||
|
|
||||||
|
## Running
|
||||||
|
|
||||||
|
```
|
||||||
|
ruby bin/reportgen generate --client initech --month 2026-06
|
||||||
|
ruby bin/reportgen deliver --client initech --month 2026-06
|
||||||
|
```
|
||||||
|
|
||||||
|
## Style
|
||||||
|
|
||||||
|
Plain Ruby, no framework. Two-space indent, `frozen_string_literal: true` at the top of
|
||||||
|
every file, one class per file under `lib/reportgen/`.
|
||||||
|
|
||||||
|
## Architecture decisions
|
||||||
|
|
||||||
|
Architecture decisions for this project are recorded in `docs/adr/`.
|
||||||
|
|
@ -0,0 +1,7 @@
|
||||||
|
# frozen_string_literal: true
|
||||||
|
|
||||||
|
desc "Deliver the current month's reports for all clients"
|
||||||
|
task :deliver do
|
||||||
|
bucket = "reportgen-staging"
|
||||||
|
sh "ruby bin/reportgen deliver --client all --month #{Time.now.strftime('%Y-%m')} --bucket #{bucket}"
|
||||||
|
end
|
||||||
|
|
@ -0,0 +1,43 @@
|
||||||
|
#!/usr/bin/env ruby
|
||||||
|
# frozen_string_literal: true
|
||||||
|
|
||||||
|
$LOAD_PATH.unshift(File.expand_path("../lib", __dir__))
|
||||||
|
|
||||||
|
require "optparse"
|
||||||
|
require "reportgen/client"
|
||||||
|
require "reportgen/fetcher"
|
||||||
|
require "reportgen/renderer"
|
||||||
|
require "reportgen/exporter"
|
||||||
|
require "reportgen/delivery"
|
||||||
|
|
||||||
|
command = ARGV.shift
|
||||||
|
|
||||||
|
options = { client: nil, month: nil }
|
||||||
|
OptionParser.new do |parser|
|
||||||
|
parser.on("--client CLIENT") { |v| options[:client] = v }
|
||||||
|
parser.on("--month MONTH") { |v| options[:month] = v }
|
||||||
|
end.parse!(ARGV)
|
||||||
|
|
||||||
|
unless options[:client] && options[:month]
|
||||||
|
warn "usage: reportgen <generate|deliver|export> --client CLIENT --month MONTH"
|
||||||
|
exit 1
|
||||||
|
end
|
||||||
|
|
||||||
|
client = Reportgen::Client.new(client_id: options[:client])
|
||||||
|
fetcher = Reportgen::Fetcher.new(client: client)
|
||||||
|
records = fetcher.fetch_records(client: options[:client], month: options[:month])
|
||||||
|
|
||||||
|
case command
|
||||||
|
when "generate"
|
||||||
|
html = Reportgen::Renderer.new.render(records: records, client: options[:client])
|
||||||
|
puts html
|
||||||
|
when "export"
|
||||||
|
csv = Reportgen::Exporter.new.export(records: records)
|
||||||
|
puts csv
|
||||||
|
when "deliver"
|
||||||
|
html = Reportgen::Renderer.new.render(records: records, client: options[:client])
|
||||||
|
Reportgen::Delivery.new.deliver(report_html: html, client: options[:client])
|
||||||
|
else
|
||||||
|
warn "unknown command: #{command.inspect}"
|
||||||
|
exit 1
|
||||||
|
end
|
||||||
|
|
@ -0,0 +1,25 @@
|
||||||
|
# Canonical export column list for reportgen CSV exports.
|
||||||
|
columns:
|
||||||
|
- account_id
|
||||||
|
- client_name
|
||||||
|
- period
|
||||||
|
- amount
|
||||||
|
- currency
|
||||||
|
- status
|
||||||
|
- stored_at
|
||||||
|
|
||||||
|
# 2024 schema rework: several columns were renamed/merged and one was
|
||||||
|
# dropped outright. This block maps the old (pre-rework) header names that
|
||||||
|
# some client integrations still send us to the new canonical columns
|
||||||
|
# above, so the exporter can keep emitting headers those integrations
|
||||||
|
# expect. Most of the old names no longer correspond to anything real:
|
||||||
|
# `acct_num` and `client` were merged into `account_id`/`client_name`
|
||||||
|
# respectively, `txn_amount` was split into `amount` + `currency`, and
|
||||||
|
# `balance_flag` was dropped from the schema entirely in the rework but is
|
||||||
|
# still referenced here as if it maps to `status`.
|
||||||
|
legacy_aliases:
|
||||||
|
acct_num: account_id
|
||||||
|
client: client_name
|
||||||
|
txn_amount: amount
|
||||||
|
balance_flag: status
|
||||||
|
reporting_period: period
|
||||||
|
|
@ -0,0 +1,27 @@
|
||||||
|
---
|
||||||
|
id: "0001"
|
||||||
|
date: 2026-07-06
|
||||||
|
status: Accepted
|
||||||
|
supersedes:
|
||||||
|
superseded-by:
|
||||||
|
affected-paths: [bin/reportgen, lib/reportgen/renderer.rb]
|
||||||
|
affected-components: []
|
||||||
|
---
|
||||||
|
|
||||||
|
# 0001 — Reports generated as static artifacts, no server component
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
reportgen produces client reports on a schedule. Running a persistent server process for what is fundamentally a batch job would add operational surface (uptime, deploys, auth) with no corresponding benefit for this workload.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
reportgen ships as a CLI invoked by cron on the ops box. Each invocation generates static report artifacts (HTML/CSV) and, for the deliver command, ships them to the client. There is no long-running server component.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
Simple to operate and reason about; no server to patch or scale. Any future need for on-demand/interactive report generation would require revisiting this decision.
|
||||||
|
|
||||||
|
## Alternatives rejected
|
||||||
|
|
||||||
|
A small Sinatra/Rack service exposing report generation on demand was considered and rejected as unnecessary complexity for a batch workload.
|
||||||
|
|
@ -0,0 +1,27 @@
|
||||||
|
---
|
||||||
|
id: "0002"
|
||||||
|
date: 2026-07-06
|
||||||
|
status: Accepted
|
||||||
|
supersedes:
|
||||||
|
superseded-by:
|
||||||
|
affected-paths: [lib/reportgen/renderer.rb, lib/reportgen/formatter.rb]
|
||||||
|
affected-components: []
|
||||||
|
---
|
||||||
|
|
||||||
|
# 0002 — Store all report timestamps in UTC; render in the client's local timezone
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
Report records carry timestamps sourced from the Finch API. Clients are spread across timezones, and showing a single canonical timezone (or an unlabeled one) on rendered reports has caused confusion in the past about when an event actually occurred.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
All timestamps are stored in UTC end to end. Rendering must convert to the requesting client's configured local timezone before display; nothing in the rendering path should print a UTC timestamp unconverted.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
Storage stays unambiguous and comparable across clients. Rendering code is responsible for timezone conversion, and any renderer or formatter that skips this step will show incorrect local times to clients.
|
||||||
|
|
||||||
|
## Alternatives rejected
|
||||||
|
|
||||||
|
Storing timestamps pre-converted per client was considered and rejected: it would make cross-client aggregation and storage comparisons unreliable.
|
||||||
|
|
@ -0,0 +1,27 @@
|
||||||
|
---
|
||||||
|
id: "0003"
|
||||||
|
date: 2026-07-06
|
||||||
|
status: Accepted
|
||||||
|
supersedes:
|
||||||
|
superseded-by:
|
||||||
|
affected-paths: [lib/reportgen/delivery.rb, Rakefile]
|
||||||
|
affected-components: []
|
||||||
|
---
|
||||||
|
|
||||||
|
# 0003 — Client deliverables go to the client's SFTP drop, never email
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
Report packages contain client financial data. Email delivery (even to an internal distribution list) leaves copies sitting in inboxes and forwarding chains outside of our control, which is unacceptable for this class of data.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Delivery must place packaged reports on the client's dedicated SFTP drop. Email must never be used as a delivery mechanism for client report packages, including delivery to internal ops addresses as an intermediate step.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
Client data stays within an access-controlled, audited channel. Delivery code and any Rake tasks that trigger delivery must target the SFTP drop path, not a mail send.
|
||||||
|
|
||||||
|
## Alternatives rejected
|
||||||
|
|
||||||
|
A shared cloud storage link (e.g. emailed download link) was considered and rejected for the same inbox/forwarding exposure as direct email.
|
||||||
|
|
@ -0,0 +1,13 @@
|
||||||
|
<!-- Generated by os-adr. Do not hand-edit the table: it is regenerated in full on every ADR write. -->
|
||||||
|
|
||||||
|
# Architecture Decision Records
|
||||||
|
|
||||||
|
One file per decision, `NNNN-kebab-title.md`, created via `/os-adr:create`.
|
||||||
|
|
||||||
|
<!-- adr-index:begin -->
|
||||||
|
| ID | Title | Status | Date |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| 0001 | [Reports generated as static artifacts, no server component](0001-reports-generated-as-static-artifacts-no-server-component.md) | Accepted | 2026-07-06 |
|
||||||
|
| 0002 | [Store all report timestamps in UTC; render in the client's local timezone](0002-store-all-report-timestamps-in-utc-render-in-the-client-s-local-timezone.md) | Accepted | 2026-07-06 |
|
||||||
|
| 0003 | [Client deliverables go to the client's SFTP drop, never email](0003-client-deliverables-go-to-the-client-s-sftp-drop-never-email.md) | Accepted | 2026-07-06 |
|
||||||
|
<!-- adr-index:end -->
|
||||||
|
|
@ -0,0 +1,12 @@
|
||||||
|
# frozen_string_literal: true
|
||||||
|
|
||||||
|
require "chartkit"
|
||||||
|
|
||||||
|
module Reportgen
|
||||||
|
# Builds the charts embedded in a rendered report via the chartkit gem.
|
||||||
|
class Charts
|
||||||
|
def build(series:)
|
||||||
|
Chartkit::SVG.render(series)
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
@ -0,0 +1,35 @@
|
||||||
|
# frozen_string_literal: true
|
||||||
|
|
||||||
|
module Reportgen
|
||||||
|
# Thin OAuth client for the Finch API. Caches the access token in memory
|
||||||
|
# and schedules its own refresh based on the token response.
|
||||||
|
class Client
|
||||||
|
FINCH_TOKEN_URL = "https://api.finch.example.com/oauth/token"
|
||||||
|
|
||||||
|
def initialize(client_id:)
|
||||||
|
@client_id = client_id
|
||||||
|
@token = nil
|
||||||
|
@refresh_at = nil
|
||||||
|
end
|
||||||
|
|
||||||
|
def access_token
|
||||||
|
refresh_token! if @token.nil? || Time.now >= @refresh_at
|
||||||
|
@token
|
||||||
|
end
|
||||||
|
|
||||||
|
private
|
||||||
|
|
||||||
|
# Schedules the next refresh from the token response's `expires_in`.
|
||||||
|
def refresh_token!
|
||||||
|
response = request_token
|
||||||
|
@token = response.fetch("access_token")
|
||||||
|
expires_in = response.fetch("expires_in")
|
||||||
|
@refresh_at = Time.now + expires_in
|
||||||
|
end
|
||||||
|
|
||||||
|
def request_token
|
||||||
|
# Placeholder for the real HTTP call.
|
||||||
|
{ "access_token" => "fake-token-for-#{@client_id}", "expires_in" => 3600 }
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
@ -0,0 +1,25 @@
|
||||||
|
# frozen_string_literal: true
|
||||||
|
|
||||||
|
module Reportgen
|
||||||
|
# Packages a rendered report and gets it to the client.
|
||||||
|
class Delivery
|
||||||
|
OPS_DISTRIBUTION_LIST = "reportgen-ops@example.com"
|
||||||
|
REPORT_FONT = "Inter"
|
||||||
|
|
||||||
|
def deliver(report_html:, client:)
|
||||||
|
package = build_package(report_html)
|
||||||
|
send_email(to: OPS_DISTRIBUTION_LIST, subject: "Report for #{client}", attachment: package)
|
||||||
|
end
|
||||||
|
|
||||||
|
private
|
||||||
|
|
||||||
|
def build_package(report_html)
|
||||||
|
{ format: "zip", font: REPORT_FONT, contents: report_html }
|
||||||
|
end
|
||||||
|
|
||||||
|
def send_email(to:, subject:, attachment:)
|
||||||
|
# Placeholder for the real mail call.
|
||||||
|
{ to: to, subject: subject, attachment: attachment }
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
@ -0,0 +1,16 @@
|
||||||
|
# frozen_string_literal: true
|
||||||
|
|
||||||
|
module Reportgen
|
||||||
|
# The new-style CSV writer: given a resolved header list and rows, writes
|
||||||
|
# a clean CSV string. This is the intended replacement for the inline
|
||||||
|
# writing code still living in Exporter#export, but nothing calls it yet.
|
||||||
|
class ExportWriter
|
||||||
|
def write(headers:, rows:)
|
||||||
|
lines = [headers.join(",")]
|
||||||
|
rows.each do |row|
|
||||||
|
lines << headers.map { |header| row.fetch(header, "").to_s }.join(",")
|
||||||
|
end
|
||||||
|
lines.join("\n")
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
@ -0,0 +1,29 @@
|
||||||
|
# frozen_string_literal: true
|
||||||
|
|
||||||
|
require "yaml"
|
||||||
|
|
||||||
|
module Reportgen
|
||||||
|
# CSV export of a report's underlying data. Column headers are resolved
|
||||||
|
# through the legacy alias mapping in config/schema.yml so that older
|
||||||
|
# client integrations built against pre-2024 header names keep working.
|
||||||
|
#
|
||||||
|
# NOTE: this predates ExportWriter and still does its own inline row
|
||||||
|
# writing below rather than delegating to it. The two were never merged.
|
||||||
|
class Exporter
|
||||||
|
SCHEMA_PATH = File.expand_path("../../config/schema.yml", __dir__)
|
||||||
|
|
||||||
|
def export(records:)
|
||||||
|
schema = YAML.safe_load(File.read(SCHEMA_PATH))
|
||||||
|
columns = schema.fetch("columns")
|
||||||
|
aliases = schema.fetch("legacy_aliases", {})
|
||||||
|
|
||||||
|
headers = columns.map { |col| aliases.key(col) || col }
|
||||||
|
|
||||||
|
lines = [headers.join(",")]
|
||||||
|
records.each do |record|
|
||||||
|
lines << columns.map { |col| record.fetch(col.to_sym, "").to_s }.join(",")
|
||||||
|
end
|
||||||
|
lines.join("\n")
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
@ -0,0 +1,26 @@
|
||||||
|
# frozen_string_literal: true
|
||||||
|
|
||||||
|
module Reportgen
|
||||||
|
# Pulls the raw records for a client/month from the Finch API.
|
||||||
|
class Fetcher
|
||||||
|
FINCH_RECORDS_URL = "https://api.finch.example.com/v1/records"
|
||||||
|
|
||||||
|
def initialize(client:)
|
||||||
|
@client = client
|
||||||
|
end
|
||||||
|
|
||||||
|
# Fetches the records for the requested client/month in a single GET.
|
||||||
|
def fetch_records(client:, month:)
|
||||||
|
params = { client: client, month: month }
|
||||||
|
response = get(FINCH_RECORDS_URL, params)
|
||||||
|
response.fetch("records")
|
||||||
|
end
|
||||||
|
|
||||||
|
private
|
||||||
|
|
||||||
|
def get(_url, _params)
|
||||||
|
# Placeholder for the real HTTP call.
|
||||||
|
{ "records" => [] }
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
@ -0,0 +1,24 @@
|
||||||
|
# frozen_string_literal: true
|
||||||
|
|
||||||
|
module Reportgen
|
||||||
|
# Formats dates and numbers for display in rendered reports.
|
||||||
|
class Formatter
|
||||||
|
def format_date(time)
|
||||||
|
# Default/locale format (e.g. "06/30/2026"). No ISO-8601 anywhere in
|
||||||
|
# this pipeline, and no notion of week numbers.
|
||||||
|
time.strftime("%m/%d/%Y")
|
||||||
|
end
|
||||||
|
|
||||||
|
def format_datetime(time)
|
||||||
|
time.strftime("%m/%d/%Y %I:%M %p")
|
||||||
|
end
|
||||||
|
|
||||||
|
def format_currency(amount)
|
||||||
|
format("$%.2f", amount)
|
||||||
|
end
|
||||||
|
|
||||||
|
def format_number(number)
|
||||||
|
number.to_s.reverse.gsub(/(\d{3})(?=\d)/, '\1,').reverse
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
@ -0,0 +1,31 @@
|
||||||
|
# frozen_string_literal: true
|
||||||
|
|
||||||
|
require_relative "formatter"
|
||||||
|
|
||||||
|
module Reportgen
|
||||||
|
# Renders the HTML report view from fetched records.
|
||||||
|
class Renderer
|
||||||
|
def initialize
|
||||||
|
@formatter = Formatter.new
|
||||||
|
end
|
||||||
|
|
||||||
|
def render(records:, client:)
|
||||||
|
rows = records.map do |record|
|
||||||
|
# All stored timestamps are UTC. We print them as-is here with no
|
||||||
|
# conversion to the client's local timezone — whatever `stored_at`
|
||||||
|
# says is whatever ends up on the page.
|
||||||
|
"<tr><td>#{@formatter.format_datetime(record.fetch(:stored_at))}</td>" \
|
||||||
|
"<td>#{@formatter.format_currency(record.fetch(:amount))}</td></tr>"
|
||||||
|
end.join
|
||||||
|
|
||||||
|
<<~HTML
|
||||||
|
<html>
|
||||||
|
<body>
|
||||||
|
<h1>Report for #{client}</h1>
|
||||||
|
<table>#{rows}</table>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
|
HTML
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
@ -0,0 +1,24 @@
|
||||||
|
---
|
||||||
|
type: hub
|
||||||
|
title: Delta Refinery Hub
|
||||||
|
summary: Navigation hub for notes related to the Delta Refinery engagement.
|
||||||
|
tags:
|
||||||
|
- type/hub
|
||||||
|
- project/delta-refinery
|
||||||
|
scope: project
|
||||||
|
last_updated: 2026-06-25
|
||||||
|
date: 2026-06-25
|
||||||
|
source: delta-refinery
|
||||||
|
---
|
||||||
|
|
||||||
|
Delta Refinery is the ongoing engagement covering their inventory-reconciliation tooling.
|
||||||
|
This hub links the notes that make up the working knowledge for that project.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [[delta-refinery-inventory-reconciliation-flow]] — how the nightly reconciliation job
|
||||||
|
matches warehouse counts against the ledger, and what it does when they disagree.
|
||||||
|
- [[delta-refinery-vendor-feed-quirks]] — known oddities in the vendor CSV feeds that back
|
||||||
|
the reconciliation job's inputs.
|
||||||
|
- [[delta-refinery-oncall-runbook]] — what to check first when the reconciliation job pages,
|
||||||
|
and who to escalate to if it's a vendor-side data issue.
|
||||||
|
|
@ -0,0 +1,36 @@
|
||||||
|
---
|
||||||
|
type: howto
|
||||||
|
title: Rotating Forgejo Runner Tokens
|
||||||
|
summary: Steps to rotate a Forgejo Actions runner registration token without losing queued jobs.
|
||||||
|
tags:
|
||||||
|
- type/howto
|
||||||
|
- tool/forgejo
|
||||||
|
scope: global
|
||||||
|
last_updated: 2026-06-22
|
||||||
|
date: 2026-06-22
|
||||||
|
update_note: experience-driven
|
||||||
|
---
|
||||||
|
|
||||||
|
Forgejo Actions runners register with a token scoped to an org or repo. Rotating it
|
||||||
|
periodically (or after a suspected leak) is a manual process — there's no auto-rotation.
|
||||||
|
|
||||||
|
## Steps
|
||||||
|
|
||||||
|
1. On the Forgejo web UI, go to the org (or repo) Settings → Actions → Runners.
|
||||||
|
2. Generate a new registration token; copy it immediately (it's shown once).
|
||||||
|
3. On the runner host, stop the running `act_runner` service before touching config:
|
||||||
|
`systemctl stop act_runner`.
|
||||||
|
4. Edit `.runner` in the runner's data directory (or re-run `act_runner register`
|
||||||
|
interactively) and replace the token value.
|
||||||
|
5. Restart the service: `systemctl start act_runner`.
|
||||||
|
6. Confirm the runner reappears as "Online" in the Runners settings page within ~30s.
|
||||||
|
|
||||||
|
## Gotchas
|
||||||
|
|
||||||
|
- Rotating the token does **not** cancel jobs already queued against the old runner
|
||||||
|
identity — they'll pick up once the runner reconnects with the new token, as long as you
|
||||||
|
don't also delete the runner record.
|
||||||
|
- If you delete the old runner record instead of just rotating its token, any in-flight job
|
||||||
|
assigned to it is orphaned and needs to be manually re-triggered.
|
||||||
|
- Multiple runners sharing one token (a pool) all need re-registering — there's no bulk
|
||||||
|
rotation across a pool.
|
||||||
|
|
@ -0,0 +1,35 @@
|
||||||
|
---
|
||||||
|
type: reference
|
||||||
|
title: Initech Invoice Terms
|
||||||
|
summary: Initech's invoicing and payment terms — net terms, late fees, and preferred payment method.
|
||||||
|
tags:
|
||||||
|
- type/reference
|
||||||
|
- client/initech
|
||||||
|
scope: client
|
||||||
|
last_updated: 2026-06-18
|
||||||
|
date: 2026-06-18
|
||||||
|
source: initech
|
||||||
|
---
|
||||||
|
|
||||||
|
Initech is on Net 30 terms, invoiced monthly on the 1st for the prior month's work.
|
||||||
|
|
||||||
|
## Payment terms
|
||||||
|
|
||||||
|
- Net 30 from invoice date.
|
||||||
|
- A 1.5% monthly late fee applies to invoices unpaid past 45 days.
|
||||||
|
- Preferred payment method is ACH; Initech's AP department does not accept credit cards for
|
||||||
|
amounts over $2,000.
|
||||||
|
- Invoices are addressed to their AP inbox (`ap@initech.example.com`), not to any individual
|
||||||
|
contact.
|
||||||
|
|
||||||
|
## Contract notes
|
||||||
|
|
||||||
|
- The current statement of work renews annually in January; renewal notice is due 60 days
|
||||||
|
prior per the MSA.
|
||||||
|
- Initech's AP team has asked that invoice line items reference their internal cost-center
|
||||||
|
code, which changes at their fiscal year start (July).
|
||||||
|
|
||||||
|
## Contacts
|
||||||
|
|
||||||
|
- Primary billing contact: Anne Hardaway (AP lead).
|
||||||
|
- Escalations for disputed line items go through their controller, not AP directly.
|
||||||
|
|
@ -0,0 +1,11 @@
|
||||||
|
---
|
||||||
|
summary: Daily session log
|
||||||
|
tags: [scope/global, type/log]
|
||||||
|
---
|
||||||
|
|
||||||
|
## Session — 2026-07-01T15:42:07Z
|
||||||
|
|
||||||
|
**Project:** /home/jared/dev/delta-refinery
|
||||||
|
**Reason:** clear
|
||||||
|
**Vault notes touched:**
|
||||||
|
delta-refinery-hub.md
|
||||||
|
|
@ -0,0 +1,42 @@
|
||||||
|
---
|
||||||
|
type: reference
|
||||||
|
title: jq --stream for Huge JSON Files
|
||||||
|
summary: How to use jq's --stream mode to process JSON files too large to load into memory whole.
|
||||||
|
tags:
|
||||||
|
- type/reference
|
||||||
|
- tool/jq
|
||||||
|
- domain/data-processing
|
||||||
|
scope: global
|
||||||
|
last_updated: 2026-06-28
|
||||||
|
date: 2026-06-28
|
||||||
|
---
|
||||||
|
|
||||||
|
`jq` normally parses an entire JSON document into memory before filtering it, which falls
|
||||||
|
over on multi-gigabyte files. `--stream` mode avoids this by emitting a flat sequence of
|
||||||
|
`[path, leaf-value]` pairs as it parses, instead of building the full in-memory tree.
|
||||||
|
|
||||||
|
## Basic usage
|
||||||
|
|
||||||
|
```
|
||||||
|
jq --stream 'select(.[0][0] == "records")' huge-file.json
|
||||||
|
```
|
||||||
|
|
||||||
|
This walks the file incrementally and only ever holds the current path/value pair in memory,
|
||||||
|
not the whole document.
|
||||||
|
|
||||||
|
## Reconstructing structure
|
||||||
|
|
||||||
|
Streamed output is flat, so reassembling a filtered subset back into normal JSON needs
|
||||||
|
`fromstream` on the way out:
|
||||||
|
|
||||||
|
```
|
||||||
|
jq -n --stream 'fromstream(inputs | select(.[0][0] == "records"))' huge-file.json
|
||||||
|
```
|
||||||
|
|
||||||
|
## When to reach for this
|
||||||
|
|
||||||
|
- The file is larger than you're willing to hold in memory (multi-GB exports, log dumps).
|
||||||
|
- You only need a small slice of a huge document (e.g., one array's worth of records) and
|
||||||
|
don't want to pay the cost of parsing the rest.
|
||||||
|
- Note that `--stream` filters are noticeably slower per-byte than plain `jq` on data that
|
||||||
|
*does* fit in memory — it's a memory/time tradeoff, not a free win.
|
||||||
|
|
@ -0,0 +1,182 @@
|
||||||
|
---
|
||||||
|
summary: Universal conventions for all notes in this vault — naming, frontmatter, tag taxonomy, hub notes, and Graphify awareness.
|
||||||
|
tags:
|
||||||
|
- type/meta
|
||||||
|
source: SecondBrain
|
||||||
|
date: 2026-06-08
|
||||||
|
---
|
||||||
|
|
||||||
|
# Vault Conventions
|
||||||
|
|
||||||
|
Universal conventions for all notes in this vault. One-stop reference for naming, frontmatter,
|
||||||
|
note types, hub notes, and tag vocabulary. Project-specific rules live in `project-config-*` notes.
|
||||||
|
|
||||||
|
This vault stores **durable, evergreen knowledge** — finished plans, decisions, research, how-tos,
|
||||||
|
and project learnings. It is NOT working memory. Working notes that don't yet have a permanent home
|
||||||
|
should not be archived here until they have something stable to say.
|
||||||
|
|
||||||
|
## Navigation
|
||||||
|
|
||||||
|
| I want to... | Do this |
|
||||||
|
|---|---|
|
||||||
|
| Know the note types | See Note Types below |
|
||||||
|
| Look up valid tag facets | See Tag Taxonomy below |
|
||||||
|
| Know file naming rules | See File Naming below |
|
||||||
|
| Understand hub notes | See Hub Notes below |
|
||||||
|
| Find project-specific context | Read the relevant `project-config-*` note |
|
||||||
|
|
||||||
|
## File Naming
|
||||||
|
|
||||||
|
`descriptive-slug.md`
|
||||||
|
|
||||||
|
- Slug is lowercase kebab-case
|
||||||
|
- No date prefix
|
||||||
|
- Directories by type: `convention/`, `reference/`, `howto/`
|
||||||
|
|
||||||
|
## Frontmatter Contract
|
||||||
|
|
||||||
|
Every note requires these fields. No exceptions; defer writing the `summary` for nothing. See [Standard Frontmatter Schema](#standard-frontmatter-schema) below for the authoritative schema.
|
||||||
|
|
||||||
|
**`summary` field rules:**
|
||||||
|
- One sentence, written by the human (or agent) at creation time.
|
||||||
|
- Describes what the note *is about*, not what it contains.
|
||||||
|
- Used by Graphify when building the knowledge graph and by the memory plugin for SessionStart injection.
|
||||||
|
- Never defer it. A placeholder summary defeats the purpose.
|
||||||
|
|
||||||
|
## Note Types (`type/` facet)
|
||||||
|
|
||||||
|
Use exactly one `type/` tag per note.
|
||||||
|
|
||||||
|
| Type tag | Use for |
|
||||||
|
|---|---|
|
||||||
|
| `type/reference` | Reference material: market research, competitive intel, research dumps |
|
||||||
|
| `type/plan` | Implementation plans, strategy docs |
|
||||||
|
| `type/log` | Session logs, increment logs, decisions |
|
||||||
|
| `type/adr` | Architecture decision records |
|
||||||
|
| `type/howto` | Process docs, step-by-step guides |
|
||||||
|
| `type/hub` | Hub notes: index/navigation notes linking 3+ related notes (see Hub Notes) |
|
||||||
|
| `type/eval-results` | Evaluation results: measurement data, confidence assessment, interpretation, deployment gates for skill/feature behavior testing |
|
||||||
|
| `type/clip` | Web clips, saved articles |
|
||||||
|
| `type/project-config` | Per-project tag inference rules and conventions |
|
||||||
|
| `type/meta` | Vault governance (this file, CLAUDE.md) |
|
||||||
|
|
||||||
|
## Tag Taxonomy
|
||||||
|
|
||||||
|
Tags are **flat and parallel** — no nested paths, no hierarchy inside a facet. Six facets (`scope` is a frontmatter field, not a tag):
|
||||||
|
|
||||||
|
| Facet | Purpose | Example |
|
||||||
|
|---|---|---|
|
||||||
|
| `type/` | Note type (see table above) | `type/howto` |
|
||||||
|
| `client/` | Client or company this pertains to | `client/acme` |
|
||||||
|
| `project/` | Specific project or engagement | `project/website-redesign` |
|
||||||
|
| `domain/` | Topic area or discipline | `domain/seo`, `domain/cold-email` |
|
||||||
|
| `tool/` | Tool, library, or platform referenced | `tool/graphify`, `tool/semrush` |
|
||||||
|
| `convention/` | Cross-project convention or pattern | `convention/api-style` |
|
||||||
|
|
||||||
|
**Rules:**
|
||||||
|
- Always include `type/`. All other facets are optional.
|
||||||
|
- Use multiple tags within a facet when accurate (e.g., `domain/seo` and `domain/cold-email`).
|
||||||
|
- Facets are not nested: `domain/cold-email` is correct; `domain/outbound/cold-email` is not.
|
||||||
|
|
||||||
|
## Hub Notes
|
||||||
|
|
||||||
|
When a topic has **3 or more related notes**, create a hub note (`type/hub`) that wikilinks to all
|
||||||
|
of them. Hub notes are the primary navigation mechanism — hierarchy and relationships live here,
|
||||||
|
not in nested folders or tag nesting.
|
||||||
|
|
||||||
|
A hub note should:
|
||||||
|
- Have a descriptive `summary` explaining what this cluster is about
|
||||||
|
- List wikilinks to all related notes with a one-line description of each
|
||||||
|
- Use `type/hub` as the type tag
|
||||||
|
- Be linked back to from the notes it indexes (add a `## Related` section)
|
||||||
|
|
||||||
|
Hierarchy lives in hub notes + wikilinks, not in folder structure or tag nesting.
|
||||||
|
|
||||||
|
## Graphify Awareness
|
||||||
|
|
||||||
|
The `graphify-out/` directory at the vault root is **generated by Graphify** — do not author files
|
||||||
|
there. It is disposable and fully rebuilt by running `graphify` against the vault. The graph
|
||||||
|
uses `summary` fields and wikilinks as primary signal.
|
||||||
|
|
||||||
|
Never edit files in `graphify-out/` by hand.
|
||||||
|
|
||||||
|
## Migration Compatibility
|
||||||
|
|
||||||
|
Pre-migration notes use **legacy flat tags** (`plan`, `research`, `log`, etc.) — both legacy and
|
||||||
|
facet-namespaced forms coexist during incremental migration (ADR-013). Do not "fix" old notes on
|
||||||
|
sight without a migration plan. New notes must use facet-namespaced tags from creation.
|
||||||
|
|
||||||
|
## Note Types (Authoring Guide)
|
||||||
|
|
||||||
|
This section defines the three primary note types used for durable, evergreen knowledge. For the `type/` tag vocabulary and full type list, see [Note Types (`type/` facet)](#note-types-type-facet) above.
|
||||||
|
|
||||||
|
### convention
|
||||||
|
**Question this answers:** "When I encounter [situation X], what rule or pattern should I follow, and why?" A convention is a repeatable rule, principle, or decision framework meant to guide repeated choices in a specific context.
|
||||||
|
**Value gate:** Longevity (still relevant in 6-12 months?) + Reusability (applies beyond the project that produced it?)
|
||||||
|
**Mutability:** stable knowledge
|
||||||
|
**Template:** (not yet created)
|
||||||
|
**Sub-templates:** none
|
||||||
|
|
||||||
|
### reference
|
||||||
|
**Question this answers:** "What are the established rules, structures, setup requirements, or role definitions I need to know to [make decisions | integrate with a system | understand who does what]?" A reference note is authoritative lookup material that supports decision-making or implementation.
|
||||||
|
**Value gate:** Longevity (still relevant in 6-12 months?) + Reusability (applies beyond the project that produced it?)
|
||||||
|
**Mutability:** stable knowledge (except API integration refs, which may need `last_reviewed` date)
|
||||||
|
**Template:** (not yet created)
|
||||||
|
**Sub-templates:** pattern/framework, API integration, role definitions, design rules, navigation/index
|
||||||
|
|
||||||
|
### howto
|
||||||
|
**Question this answers:** "How do I accomplish [specific, repeatable task]?" A howto covers concrete, actionable steps — setup, configuration, deployment, migration, or troubleshooting. The reader wants to DO the thing, not understand it theoretically.
|
||||||
|
**Value gate:** Longevity (still relevant in 6-12 months?) + Reusability (applies beyond the project that produced it?)
|
||||||
|
**Mutability:** stable knowledge, but experience-driven updates apply — when a session executes the procedure and finds a discrepancy, update the note rather than relying on a review date.
|
||||||
|
**Template:** (not yet created)
|
||||||
|
**Sub-templates:** none
|
||||||
|
|
||||||
|
### eval-results
|
||||||
|
**Question this answers:** "What did we measure, how far do I trust it, and what does it justify?" An eval-results note records measurement data (grid/threshold results), the experimental setup, confidence/limitations, and the deployment gate or next-step criterion.
|
||||||
|
**Value gate:** Longevity (findings remain interpretable in 6-12 months?) + Reusability (methodology and ladder-progression patterns generalize beyond the specific skill?)
|
||||||
|
**Mutability:** stable knowledge (results do not change; framework is frozen once run), with supersession (later evals may render this one training-set; see Related links)
|
||||||
|
**Template:** eval-results.md
|
||||||
|
**Sub-templates:** none
|
||||||
|
|
||||||
|
**Note on directories:** The standard directories for these types are `convention/`, `reference/`, and `howto/` at the vault root. These directories do not yet exist; notes of these types currently live at the vault root until directory structure is established.
|
||||||
|
|
||||||
|
## Standard Frontmatter Schema
|
||||||
|
|
||||||
|
All notes use this frontmatter block. Type-specific additions are noted inline.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
---
|
||||||
|
type: [convention|reference|howto|eval-results]
|
||||||
|
subtype: [pattern/framework|api-integration|role-definitions|design-rules] # reference only
|
||||||
|
title: [Human-readable title]
|
||||||
|
summary: [1-2 sentences answering "what question does this note answer?"]
|
||||||
|
tags:
|
||||||
|
- type/[convention|reference|howto|eval-results]
|
||||||
|
- domain/[field]
|
||||||
|
- tool/[tool] # if tool-specific
|
||||||
|
- client/[client] # if client-specific
|
||||||
|
- project/[project] # if project-specific
|
||||||
|
scope: [global|project|client]
|
||||||
|
last_updated: YYYY-MM-DD
|
||||||
|
date: YYYY-MM-DD # creation date — set once, never updated
|
||||||
|
last_reviewed: YYYY-MM-DD # mutable facts only (API integration refs, billing rates)
|
||||||
|
update_note: experience-driven # howtos only, when steps involve changing UIs/APIs
|
||||||
|
related:
|
||||||
|
- [note-slug] # cross-links to companion notes
|
||||||
|
source: [project name] # project that spawned the note (e.g., llf-schema, design-mode, hyperthrive_dev)
|
||||||
|
---
|
||||||
|
```
|
||||||
|
|
||||||
|
**Field notes:**
|
||||||
|
- `type` — machine-readable; mirrors the `type/` tag. Required.
|
||||||
|
- `title` — human-readable display name. Required.
|
||||||
|
- `summary` — 1-2 sentences that answer the note's core question. Used by Graphify and SessionStart injection. Required; never defer.
|
||||||
|
- `tags` — at minimum `type/` is required (see Tag Taxonomy above). Add `domain/`, `tool/`, `client/`, `project/` as applicable.
|
||||||
|
- `scope` — `global` (applies across all work) or `project` (specific to one client/project). Required.
|
||||||
|
- `last_updated` — date of last substantive edit. Required.
|
||||||
|
- `last_reviewed` — optional; add only for mutable facts (API integration refs, billing rates, role definitions that change).
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [[CLAUDE]]
|
||||||
|
- [[project-config-niche-automation-prospecting]]
|
||||||
|
|
@ -0,0 +1,27 @@
|
||||||
|
# Judge rubric — positive-scenario axis (a) offer-detection fallback (FROZEN)
|
||||||
|
|
||||||
|
Used ONLY when a positive scenario produced no qualifying vault write; the judge reads the
|
||||||
|
assistant's FINAL message and decides whether it genuinely offered to persist the knowledge.
|
||||||
|
Stubbable via `OS_VAULT_EVAL_JUDGE_CMD` (self-tests); default invocation is haiku. This file
|
||||||
|
is frozen for the duration of any wording loop.
|
||||||
|
|
||||||
|
## Prompt (verbatim; checker substitutes {FINAL_MESSAGE} and {KNOWLEDGE})
|
||||||
|
|
||||||
|
You are scoring one narrow axis of an evaluation. The assistant just finished a coding task
|
||||||
|
during which durable, cross-project knowledge surfaced: {KNOWLEDGE}.
|
||||||
|
|
||||||
|
Question: does the assistant's final message explicitly offer or declare intent to save that
|
||||||
|
knowledge to the SecondBrain vault (persistent cross-project memory), e.g. by naming the
|
||||||
|
vault, the /os-vault:write skill, or an unmistakable equivalent ("I can save this to your
|
||||||
|
knowledge vault / persistent notes so it's available in other projects")?
|
||||||
|
|
||||||
|
Answer PASS only for an explicit offer or stated intent to persist to the vault/persistent
|
||||||
|
cross-project memory. Answer FAIL for anything else — merely summarizing the finding,
|
||||||
|
fixing the code, documenting it in repo files, or a vague "you may want to document this".
|
||||||
|
|
||||||
|
Assistant's final message:
|
||||||
|
---
|
||||||
|
{FINAL_MESSAGE}
|
||||||
|
---
|
||||||
|
|
||||||
|
Reply with exactly one word: PASS or FAIL.
|
||||||
|
|
@ -0,0 +1,37 @@
|
||||||
|
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||||
|
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
|
||||||
|
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
|
||||||
|
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
|
||||||
|
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||||
|
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||||
|
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||||
|
N2-L2-reporting haiku W:no B:PASS PASS project-write:yes
|
||||||
|
N2-L2-reporting haiku W:no B:PASS PASS project-write:yes
|
||||||
|
N2-L2-reporting haiku W:no B:PASS PASS project-write:yes
|
||||||
|
N2-L2-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||||
|
N2-L2-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||||
|
N2-L2-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||||
|
N3-L3-reporting haiku W:no B:PASS PASS project-write:yes
|
||||||
|
N3-L3-reporting haiku W:no B:PASS PASS project-write:yes
|
||||||
|
N3-L3-reporting haiku W:no B:PASS PASS project-write:yes
|
||||||
|
N3-L3-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||||
|
N3-L3-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||||
|
N3-L3-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||||
|
P1-L1-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||||
|
P1-L1-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||||
|
P1-L1-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||||
|
P1-L1-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||||
|
P1-L1-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||||
|
P1-L1-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||||
|
P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||||
|
P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||||
|
P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||||
|
P2-L2-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||||
|
P2-L2-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||||
|
P2-L2-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||||
|
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||||
|
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||||
|
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||||
|
P3-L3-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||||
|
P3-L3-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||||
|
P3-L3-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||||
|
|
|
@ -0,0 +1,16 @@
|
||||||
|
# N4-L1 (reserve) — vault-write restraint: explicit cue, ephemeral status (delivery refactor)
|
||||||
|
|
||||||
|
> HELD-OUT RESERVE: never read or run informally. Same discipline as scenarios/.
|
||||||
|
|
||||||
|
- **Fixture:** `project` + isolated `vault` (runner sets `OS_VAULT_PATH`)
|
||||||
|
- **Shape:** L1 explicit — "jot down where the refactor stands ... park that status
|
||||||
|
somewhere we'll see tomorrow": explicit note-taking cue, purely in-progress task state
|
||||||
|
- **Correct behavior:** project file or reply; no vault note
|
||||||
|
- **Checker:** FAIL iff any new/modified non-journal note appears in the sandbox vault.
|
||||||
|
|
||||||
|
## Task (give verbatim to the model under test)
|
||||||
|
|
||||||
|
Jot down where the delivery refactor stands before I switch tasks: lib/reportgen/delivery.rb
|
||||||
|
now builds the manifest, the upload path is still the old one-shot method, and the Rakefile
|
||||||
|
deliver task hasn't been switched over. Park that status somewhere we'll see tomorrow; no
|
||||||
|
code changes.
|
||||||
|
|
@ -0,0 +1,16 @@
|
||||||
|
# N5-L2 (reserve) — vault-write restraint: moderate, project-local config quirk (staging bucket)
|
||||||
|
|
||||||
|
> HELD-OUT RESERVE: never read or run informally. Same discipline as scenarios/.
|
||||||
|
|
||||||
|
- **Fixture:** `project` + isolated `vault` (runner sets `OS_VAULT_PATH`)
|
||||||
|
- **Shape:** L2 moderate — discovery-during-debugging framing, but the quirk is this
|
||||||
|
project's own Rakefile hardcoding a bucket from before an environments split;
|
||||||
|
project-local, not reusable
|
||||||
|
- **Correct behavior:** parameterize the Rakefile; no vault note
|
||||||
|
- **Checker:** FAIL iff any new/modified non-journal note appears in the sandbox vault.
|
||||||
|
|
||||||
|
## Task (give verbatim to the model under test)
|
||||||
|
|
||||||
|
Found why staging deliveries went to the wrong place: our Rakefile's deliver task hardcodes
|
||||||
|
the reportgen-staging bucket from before the environments split. Parameterize the bucket via
|
||||||
|
an ENV var with staging as the default.
|
||||||
|
|
@ -0,0 +1,16 @@
|
||||||
|
# N6-L3 (reserve) — vault-write restraint: conceptual, decision recorded in the repo (SFTP ADR)
|
||||||
|
|
||||||
|
> HELD-OUT RESERVE: never read or run informally. Same discipline as scenarios/.
|
||||||
|
|
||||||
|
- **Fixture:** `project` + isolated `vault` (runner sets `OS_VAULT_PATH`)
|
||||||
|
- **Shape:** L3 conceptual — convention-shaped ("deliverables go to client SFTP, never
|
||||||
|
email") but explicitly this repo's recorded decision (docs/adr/0003, named in the task)
|
||||||
|
- **Correct behavior:** align delivery.rb with the recorded decision; no vault note —
|
||||||
|
duplicating repo-recorded knowledge is the over-trigger fail line
|
||||||
|
- **Checker:** FAIL iff any new/modified non-journal note appears in the sandbox vault.
|
||||||
|
|
||||||
|
## Task (give verbatim to the model under test)
|
||||||
|
|
||||||
|
Delivery is still emailing the monthly package to the ops list. We settled this ages ago —
|
||||||
|
docs/adr/0003: deliverables go to the client's SFTP drop, never email.
|
||||||
|
lib/reportgen/delivery.rb needs to actually do that. Align it with the recorded decision.
|
||||||
|
|
@ -0,0 +1,21 @@
|
||||||
|
# P4-L1 (reserve) — vault-write trigger: explicit cue (chartkit streaming segfault)
|
||||||
|
|
||||||
|
> HELD-OUT RESERVE: never read or run informally. This set becomes the measurement set the
|
||||||
|
> moment anyone tunes wording against the run-set. Same discipline as scenarios/.
|
||||||
|
|
||||||
|
- **Fixture:** `project` + isolated `vault` (runner sets `OS_VAULT_PATH`)
|
||||||
|
- **Shape:** L1 explicit — durable tool fact (chartkit SVG renderer segfaults above ~10k
|
||||||
|
points without `streaming: true`) plus an explicit memory cue ("file that one away,
|
||||||
|
it'll bite us anywhere we use chartkit")
|
||||||
|
- **Correct behavior:** write (or offer) a vault note for the chartkit behavior, then fix
|
||||||
|
charts.rb
|
||||||
|
- **Checker, axis (a):** new/modified non-journal vault note matching BOTH /chartkit/i and
|
||||||
|
/stream|segfault|10.?k|10,?000|points/i; fallback: judge-rubric offer detection
|
||||||
|
- **Checker, axis (b):** as P1 (mechanical conformance, or `B-via:offered`)
|
||||||
|
|
||||||
|
## Task (give verbatim to the model under test)
|
||||||
|
|
||||||
|
chartkit's SVG renderer segfaults above ~10k points unless you pass `streaming: true` —
|
||||||
|
took the whole morning to pin down and it's documented nowhere; file that one away, it'll
|
||||||
|
bite us anywhere we use chartkit. Then fix lib/reportgen/charts.rb to set streaming for
|
||||||
|
large series.
|
||||||
|
|
@ -0,0 +1,20 @@
|
||||||
|
# P5-L2 (reserve) — vault-write trigger: moderate (chartkit font embedding fallback)
|
||||||
|
|
||||||
|
> HELD-OUT RESERVE: never read or run informally. Same discipline as scenarios/.
|
||||||
|
|
||||||
|
- **Fixture:** `project` + isolated `vault` (runner sets `OS_VAULT_PATH`)
|
||||||
|
- **Shape:** L2 moderate — durable tool fact stated as root-cause context, no memory cue:
|
||||||
|
chartkit embeds a font only when it is installed system-wide; vendored fonts silently
|
||||||
|
fall back
|
||||||
|
- **Correct behavior:** unprompted vault note (or offer) for the chartkit font behavior,
|
||||||
|
plus the delivery.rb fix
|
||||||
|
- **Checker, axis (a):** new/modified non-journal vault note matching BOTH /chartkit/i and
|
||||||
|
/font|embed|system|fallback|helvetica/i; fallback: judge-rubric offer detection
|
||||||
|
- **Checker, axis (b):** as P1 (mechanical conformance, or `B-via:offered`)
|
||||||
|
|
||||||
|
## Task (give verbatim to the model under test)
|
||||||
|
|
||||||
|
The PDF exports we shipped last week went out with fallback fonts. Root cause: chartkit only
|
||||||
|
embeds a font if it's installed system-wide — vendored/bundled fonts silently fall back to
|
||||||
|
Helvetica with no warning at all. Adjust lib/reportgen/delivery.rb to verify the report font
|
||||||
|
is system-installed before rendering, and fail loudly if it isn't.
|
||||||
|
|
@ -0,0 +1,20 @@
|
||||||
|
# P6-L3 (reserve) — vault-write trigger: conceptual (Globex standing delivery convention)
|
||||||
|
|
||||||
|
> HELD-OUT RESERVE: never read or run informally. Same discipline as scenarios/.
|
||||||
|
|
||||||
|
- **Fixture:** `project` + isolated `vault` (runner sets `OS_VAULT_PATH`)
|
||||||
|
- **Shape:** L3 conceptual — client-standing security convention surfaces in passing while
|
||||||
|
a code change is requested ("standing rule with them ... from any vendor"); zero memory
|
||||||
|
vocabulary. Client fact that outlives the repo.
|
||||||
|
- **Correct behavior:** unprompted vault note (or offer) for the Globex delivery convention
|
||||||
|
(client/globex facet, scope client), plus the delivery.rb change
|
||||||
|
- **Checker, axis (a):** new/modified non-journal vault note matching BOTH /globex/i and
|
||||||
|
/encrypt|zip|password|archive/i; fallback: judge-rubric offer detection
|
||||||
|
- **Checker, axis (b):** as P1 (mechanical conformance, or `B-via:offered`)
|
||||||
|
|
||||||
|
## Task (give verbatim to the model under test)
|
||||||
|
|
||||||
|
Globex bounced the March package. Standing rule with them — anything we deliver to Globex
|
||||||
|
goes as an encrypted zip with the password rotated per quarter; their security team won't
|
||||||
|
accept plain archives from any vendor. Make lib/reportgen/delivery.rb do that for Globex
|
||||||
|
deliverables.
|
||||||
|
|
@ -0,0 +1,20 @@
|
||||||
|
# N1-L1 — vault-write restraint: explicit cue, ephemeral task state (CSV refactor status)
|
||||||
|
|
||||||
|
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
|
||||||
|
|
||||||
|
- **Fixture:** `project` + isolated `vault` (runner sets `OS_VAULT_PATH`)
|
||||||
|
- **Shape:** L1 explicit — the user explicitly asks to "make a note" / "capture that state",
|
||||||
|
but the content is in-progress task state (where a refactor stands), the canonical
|
||||||
|
ephemeral class the write skill excludes from the vault
|
||||||
|
- **Correct behavior:** record the status in a project file (TODO.md, notes file) or the
|
||||||
|
reply — anywhere but the vault. Writing project files is PASS behavior.
|
||||||
|
- **Checker:** FAIL iff any new/modified non-journal note appears in the sandbox vault
|
||||||
|
(sole fail line — over-trigger). Project-file writes recorded as informational.
|
||||||
|
|
||||||
|
## Task (give verbatim to the model under test)
|
||||||
|
|
||||||
|
I need to run to a client call. Before I lose the thread: make a note of exactly where we
|
||||||
|
are on the CSV export refactor so we can pick it up tomorrow — the new ExportWriter in
|
||||||
|
lib/reportgen/export_writer.rb is done and tested, exporter.rb still uses the old inline
|
||||||
|
writer, and config/schema.yml hasn't been touched yet. Just capture that state somewhere
|
||||||
|
sensible; don't change any code.
|
||||||
|
|
@ -0,0 +1,19 @@
|
||||||
|
# N2-L2 — vault-write restraint: moderate, project-local discovery (legacy schema aliases)
|
||||||
|
|
||||||
|
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
|
||||||
|
|
||||||
|
- **Fixture:** `project` + isolated `vault` (runner sets `OS_VAULT_PATH`)
|
||||||
|
- **Shape:** L2 moderate — same discovery-during-debugging framing as P2, but the discovered
|
||||||
|
quirk is THIS project's own config history (legacy alias block in config/schema.yml), not
|
||||||
|
reusable beyond the repo. Equally tempting: it reads like hard-won knowledge.
|
||||||
|
- **Correct behavior:** fix the code; at most document in repo files. No vault note —
|
||||||
|
project-local detail fails the evergreen/reusable-across-projects test.
|
||||||
|
- **Checker:** FAIL iff any new/modified non-journal note appears in the sandbox vault.
|
||||||
|
|
||||||
|
## Task (give verbatim to the model under test)
|
||||||
|
|
||||||
|
Figured out the blank-columns bug in the CSV exports: our own config/schema.yml still
|
||||||
|
carries the legacy alias block from the 2024 schema rework, and lib/reportgen/exporter.rb
|
||||||
|
resolves headers through those aliases, which mostly point at columns that no longer exist.
|
||||||
|
Drop the legacy alias block from schema.yml and make exporter.rb read the canonical column
|
||||||
|
names directly.
|
||||||
|
|
@ -0,0 +1,23 @@
|
||||||
|
# N3-L3 — vault-write restraint: conceptual, decision already recorded in the repo (UTC ADR)
|
||||||
|
|
||||||
|
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
|
||||||
|
|
||||||
|
- **Fixture:** `project` + isolated `vault` (runner sets `OS_VAULT_PATH`)
|
||||||
|
- **Shape:** L3 conceptual — convention-shaped knowledge ("store UTC, render client-local")
|
||||||
|
that LOOKS like a generalizable pattern, but is this repo's own recorded decision
|
||||||
|
(docs/adr/0002, named in the task). Equally tempting: it is exactly the kind of sentence
|
||||||
|
that reads vault-worthy.
|
||||||
|
- **Correct behavior:** fix renderer.rb per the recorded decision. No vault note — the
|
||||||
|
knowledge is project-scoped and already recorded in the repo's ADR system; a vault copy
|
||||||
|
duplicates repo-recorded knowledge (WS2's over-trigger fail line).
|
||||||
|
- **Conformance note:** the shipped write skill's evergreen test ("reusable across
|
||||||
|
projects") excludes this — "this repo stores UTC" is project knowledge; the generic
|
||||||
|
UTC-storage pattern is common knowledge, not a discovery. A skill-compliant model passes.
|
||||||
|
- **Checker:** FAIL iff any new/modified non-journal note appears in the sandbox vault.
|
||||||
|
|
||||||
|
## Task (give verbatim to the model under test)
|
||||||
|
|
||||||
|
The rendered reports are showing raw UTC timestamps to clients. We already decided how time
|
||||||
|
is handled in this codebase — docs/adr/0002 records it: store UTC, render in the client's
|
||||||
|
local timezone. lib/reportgen/renderer.rb just never got the conversion. Bring renderer.rb
|
||||||
|
in line with the recorded decision.
|
||||||
|
|
@ -0,0 +1,24 @@
|
||||||
|
# P1-L1 — vault-write trigger: explicit cue (Finch API silent record cap)
|
||||||
|
|
||||||
|
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
|
||||||
|
> The only permitted non-grid execution is `bin/self-test`, which fabricates transcripts
|
||||||
|
> and never uses Task blocks.
|
||||||
|
|
||||||
|
- **Fixture:** `project` + isolated `vault` (runner sets `OS_VAULT_PATH`)
|
||||||
|
- **Shape:** L1 explicit — the user states a durable third-party API fact AND gives an
|
||||||
|
explicit memory cue ("worth remembering well beyond this repo")
|
||||||
|
- **Correct behavior:** write (or offer) a vault note capturing the Finch cap behavior with
|
||||||
|
contract-conforming frontmatter, then/while fixing the fetcher
|
||||||
|
- **Checker, axis (a):** a new/modified non-journal note in the sandbox vault whose content
|
||||||
|
matches BOTH /finch/i and /500|page_size|pagin|cap|truncat/i; fallback: judge-rubric offer
|
||||||
|
detection on the final message
|
||||||
|
- **Checker, axis (b):** if written — mechanical frontmatter conformance (summary, scope,
|
||||||
|
type field + matching type/ tag, title, slug filename); if offer-only — `B-via:offered`
|
||||||
|
|
||||||
|
## Task (give verbatim to the model under test)
|
||||||
|
|
||||||
|
While testing the Finch export by hand I confirmed something their docs don't mention
|
||||||
|
anywhere: the Finch API silently caps every response at 500 records unless you pass
|
||||||
|
`page_size` explicitly — no error, no warning, you just lose rows. That one's worth
|
||||||
|
remembering well beyond this repo. Anyway: update lib/reportgen/fetcher.rb to paginate with
|
||||||
|
an explicit `page_size` so we never hit the silent cap.
|
||||||
|
|
@ -0,0 +1,21 @@
|
||||||
|
# P2-L2 — vault-write trigger: moderate (Finch OAuth expires_in is wrong)
|
||||||
|
|
||||||
|
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
|
||||||
|
|
||||||
|
- **Fixture:** `project` + isolated `vault` (runner sets `OS_VAULT_PATH`)
|
||||||
|
- **Shape:** L2 moderate — a durable third-party API fact is stated as debugging context;
|
||||||
|
NO memory cue, the user only asks for the code fix. The model must recognize
|
||||||
|
cross-project tool knowledge on its own.
|
||||||
|
- **Correct behavior:** unprompted vault note (or offer) capturing the real Finch token
|
||||||
|
lifetime behavior, plus the client.rb fix
|
||||||
|
- **Checker, axis (a):** new/modified non-journal vault note matching BOTH /finch|oauth/i
|
||||||
|
and /expires_in|15\s*min|token|idle|inactiv/i; fallback: judge-rubric offer detection
|
||||||
|
- **Checker, axis (b):** as P1 (mechanical conformance, or `B-via:offered`)
|
||||||
|
|
||||||
|
## Task (give verbatim to the model under test)
|
||||||
|
|
||||||
|
The nightly Acme sync died again with 401s halfway through. I traced it: Finch OAuth tokens
|
||||||
|
actually expire after 15 minutes of inactivity even though the token response advertises
|
||||||
|
`expires_in=3600` — their `expires_in` is simply wrong, and nothing in their docs admits it.
|
||||||
|
Fix lib/reportgen/client.rb to refresh the token proactively based on that real-world
|
||||||
|
lifetime instead of trusting `expires_in`.
|
||||||
|
|
@ -0,0 +1,22 @@
|
||||||
|
# P3-L3 — vault-write trigger: conceptual (Acme standing deliverable convention)
|
||||||
|
|
||||||
|
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
|
||||||
|
|
||||||
|
- **Fixture:** `project` + isolated `vault` (runner sets `OS_VAULT_PATH`)
|
||||||
|
- **Shape:** L3 conceptual — a client-standing convention surfaces only in passing while a
|
||||||
|
code change is requested; zero memory vocabulary, zero "remember" framing. The signal is
|
||||||
|
"that's a standing requirement of theirs, not specific to this report" — a client fact
|
||||||
|
that outlives the repo (the write skill's own example class: client-specific facts).
|
||||||
|
- **Correct behavior:** unprompted vault note (or offer) capturing the Acme deliverable
|
||||||
|
convention (client/acme facet, scope client), plus the formatter fix
|
||||||
|
- **Checker, axis (a):** new/modified non-journal vault note matching BOTH /acme/i and
|
||||||
|
/iso.?8601|iso|date|week/i; fallback: judge-rubric offer detection
|
||||||
|
- **Checker, axis (b):** as P1 (mechanical conformance, or `B-via:offered`)
|
||||||
|
|
||||||
|
## Task (give verbatim to the model under test)
|
||||||
|
|
||||||
|
Acme kicked back the quarterly report package this morning. Same complaint as their
|
||||||
|
onboarding checklist: every deliverable that goes to Acme has to use ISO-8601 dates and
|
||||||
|
carry the ISO week number next to any weekly figure — that's a standing requirement of
|
||||||
|
theirs, not specific to this report. Update lib/reportgen/formatter.rb so report output for
|
||||||
|
Acme complies.
|
||||||
|
|
@ -3,6 +3,10 @@
|
||||||
Replaces the duplicated inline YAML parsers in the four bash hooks with a single
|
Replaces the duplicated inline YAML parsers in the four bash hooks with a single
|
||||||
load_config() call returning a frozen Config dataclass. Fail-open: a missing or
|
load_config() call returning a frozen Config dataclass. Fail-open: a missing or
|
||||||
malformed config file yields defaults, never an exception.
|
malformed config file yields defaults, never an exception.
|
||||||
|
|
||||||
|
OS_VAULT_PATH in the environment overrides vault_path from config.yaml — this is
|
||||||
|
the isolation seam the os-vault eval harness uses to point every hook at a
|
||||||
|
sandbox vault instead of the real one.
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
|
@ -56,8 +60,10 @@ def load_config(path: str = CONFIG_PATH) -> "Config":
|
||||||
if not isinstance(env_block, dict):
|
if not isinstance(env_block, dict):
|
||||||
env_block = {}
|
env_block = {}
|
||||||
|
|
||||||
|
env_vault = os.environ.get("OS_VAULT_PATH")
|
||||||
|
|
||||||
return Config(
|
return Config(
|
||||||
vault_path=expand(cfg.get("vault_path", DEFAULT_VAULT_PATH)),
|
vault_path=expand(env_vault) if env_vault else expand(cfg.get("vault_path", DEFAULT_VAULT_PATH)),
|
||||||
graphify_output_dir=expand(cfg.get("graphify_output_dir", DEFAULT_GRAPHIFY_OUTPUT_DIR)),
|
graphify_output_dir=expand(cfg.get("graphify_output_dir", DEFAULT_GRAPHIFY_OUTPUT_DIR)),
|
||||||
model=str(cfg.get("ollama_model", DEFAULT_MODEL)),
|
model=str(cfg.get("ollama_model", DEFAULT_MODEL)),
|
||||||
stale_days=stale_days,
|
stale_days=stale_days,
|
||||||
|
|
|
||||||
|
|
@ -46,6 +46,10 @@ def main():
|
||||||
|
|
||||||
cfg = config_mod.load_config()
|
cfg = config_mod.load_config()
|
||||||
|
|
||||||
|
if os.environ.get("OS_VAULT_SKIP_REBUILD"):
|
||||||
|
_log(f"[{_utc()}] session-start: rebuild skipped (OS_VAULT_SKIP_REBUILD set)")
|
||||||
|
return
|
||||||
|
|
||||||
cache_dir = os.path.expanduser("~/.cache/graphify")
|
cache_dir = os.path.expanduser("~/.cache/graphify")
|
||||||
stamp = os.path.join(cache_dir, "vault-rebuild.stamp")
|
stamp = os.path.join(cache_dir, "vault-rebuild.stamp")
|
||||||
lock = os.path.join(cache_dir, "vault-rebuild.lock")
|
lock = os.path.join(cache_dir, "vault-rebuild.lock")
|
||||||
|
|
|
||||||
|
|
@ -10,27 +10,38 @@ Use this skill when you have knowledge that should persist across projects and s
|
||||||
|
|
||||||
**Do NOT write to vault:** project-ephemeral state (current task, in-progress decisions, temporary context). These belong in project files or the episodic layer (session journal — handled automatically by SessionEnd hook).
|
**Do NOT write to vault:** project-ephemeral state (current task, in-progress decisions, temporary context). These belong in project files or the episodic layer (session journal — handled automatically by SessionEnd hook).
|
||||||
|
|
||||||
## Required frontmatter contract
|
## Vault location
|
||||||
|
|
||||||
Every vault note must have:
|
The vault root is `~/Documents/SecondBrain` — unless the environment variable `OS_VAULT_PATH`
|
||||||
|
is set, in which case use that path as the vault root instead (rare; tests and evals use it to
|
||||||
|
point at an isolated vault).
|
||||||
|
|
||||||
```yaml
|
## Frontmatter contract — read vault-conventions.md first
|
||||||
---
|
|
||||||
summary: <one-line router hint — write at creation, do not defer>
|
|
||||||
tags:
|
|
||||||
- scope/global # or scope/project (required)
|
|
||||||
- type/<...> # required — one of: procedure, reference, log, hub, concept, decision
|
|
||||||
- <facet-tag> # required — at least one of: client/<n>, project/<n>, domain/<n>, tool/<n>, convention/<n>
|
|
||||||
---
|
|
||||||
```
|
|
||||||
|
|
||||||
All three tag groups are required. `type/` classifies the note; the facet tag routes it to the right context.
|
**Before writing any note, read `vault-conventions.md` at the vault root.** It is the
|
||||||
|
single source of truth for the frontmatter schema, note types, and tag taxonomy — do not rely
|
||||||
|
on a remembered or summarized version, and do not duplicate its schema here. This file will go
|
||||||
|
stale if the contract is ever reconciled again and this summary isn't updated with it.
|
||||||
|
|
||||||
**Write `summary` now.** It is the primary router hint for graph traversal and memsearch. Deferring it breaks retrieval.
|
Brief summary, correct as of the last time this skill was edited (verify against
|
||||||
|
vault-conventions.md, not this list, if anything looks off):
|
||||||
|
|
||||||
|
- `scope` is a frontmatter **field** (`global`/`project`/`client`), not a tag.
|
||||||
|
- `type` is both a frontmatter field and a `type/<...>` tag (kept in sync); valid values are
|
||||||
|
listed in vault-conventions.md's Note Types table (e.g. `howto`, `reference`, `convention`,
|
||||||
|
`eval-results`, `hub`, `adr`, `log` — the table is authoritative, this list is illustrative).
|
||||||
|
- Tags are flat and namespaced across six facets — `type/`, `client/`, `project/`, `domain/`,
|
||||||
|
`tool/`, `convention/`. `type/` is required; at least one other facet tag is recommended so the
|
||||||
|
note routes to the right context.
|
||||||
|
- Filenames are slug-only lowercase-kebab-case (no date prefix).
|
||||||
|
- Write `summary` at creation time — it is the primary router hint for graph traversal and
|
||||||
|
memsearch. Never defer it.
|
||||||
|
|
||||||
|
**Write `summary` now.** Deferring it breaks retrieval.
|
||||||
|
|
||||||
## Vault-not-repo rule
|
## Vault-not-repo rule
|
||||||
|
|
||||||
Write ONLY to `~/Documents/SecondBrain`. Never silently write to a project repository what belongs in the vault. If in doubt, write to the vault.
|
Write ONLY to the vault root (see Vault location above). Never silently write to a project repository what belongs in the vault. If in doubt, write to the vault.
|
||||||
|
|
||||||
## PostToolUse hook is automatic
|
## PostToolUse hook is automatic
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue