diff --git a/eval-methodology-ladder.md b/eval-methodology-ladder.md index a7ed928..1b82bdd 100644 --- a/eval-methodology-ladder.md +++ b/eval-methodology-ladder.md @@ -9,7 +9,7 @@ tags: - convention/eval-design - tool/autoresearch scope: global -last_updated: 2026-07-06 +last_updated: 2026-07-06 # hardened same day with Eval C first-grid lessons (checker-conformance, self-test blind spot) date: 2026-07-06 related: - running-autoresearch-skill-evals @@ -83,6 +83,8 @@ Lower model tiers (e.g., haiku vs sonnet) may not clear the same rung at the sam - **Scenarios that are too ambiguous to score cleanly.** If a negative scenario *looks* correct and you have to debate whether the model should have triggered, the scenario isn't at the right level yet. Move it up to a higher level or rewrite it. - **Forgetting that paired scenarios are *equally* subtle.** If your negative scenarios are obviously non-triggers (e.g., "add logging, no decision involved"), you're not testing discrimination. Negative scenarios should be as close to the positive case as possible without crossing the trigger threshold. - **Tuning against held-out reserve.** Once the reserve is defined, it becomes Schrodinger's eval — useful as a fresh measurement only if it stays frozen. If you peek at reserve results and change wording based on them, it's now training-set. +- **A checker that contradicts the shipped instructions.** Before spending reps, dry-run the instrument on paper: "would a model that follows the shipped wording perfectly pass every cell?" and "would an always-trigger model pass the positives while failing the negatives?" If yes to the second or no to the first, the eval measures instruction-*disobedience*. Concrete instance (os-adr Eval C, 2026-07-06): positives sat in already-decided ADR territory, where the shipped wording says find→cite→comply — but the checker demanded ADR creation, so a perfectly-behaving model FAILed and an ADR-spamming model would have aced positives. Caught on the first live rep only because of a canary-cell run. +- **Trusting model-free self-tests to validate checker *semantics*.** Fabricated "perfect" transcripts encode the same assumptions as the checker — if the designer's idea of correct behavior is wrong, both agree and the self-test stays green. Self-tests catch mechanical bugs, not conformance bugs. Fabricate at least one transcript of *shipped-instruction-compliant* behavior (what the model is actually told to do, given the fixture state), not just the designer's imagined ideal. ## Known Limitations diff --git a/howto/running-autoresearch-skill-evals.md b/howto/running-autoresearch-skill-evals.md index 88a5fee..15debde 100644 --- a/howto/running-autoresearch-skill-evals.md +++ b/howto/running-autoresearch-skill-evals.md @@ -9,7 +9,7 @@ tags: - tool/autoresearch - project/cc-os scope: global -last_updated: 2026-07-04 +last_updated: 2026-07-06 date: 2026-07-04 related: - os-adr-eval-b-grid-results-and-observations @@ -63,6 +63,15 @@ min; parallel, it's bounded by the slowest cell (~5 min). Drive the grid script the session with background Bash — do NOT wrap it in a babysitting subagent (agents self-pause, need resuming, and re-runs collide with existing sandboxes: "refusing to overwrite"). +### Step 4b: Canary-cell the first live run of any new harness + +Before launching a full grid on a harness that has never run live, run ONE cell first and +verify its TSV row against the raw transcript by hand (did the hook fire? does the reason +string match what the model actually did?). Count the canary's result in the measurement — +never discard it (peeking bias). The os-adr Eval C first grid (2026-07-06) caught two harness +defects this way at a cost of 1 rep instead of 36: a runner rep-loop bug and a checker that +failed instruction-compliant behavior. + ### Step 5: Use enough reps to beat the noise 1 rep/cell is demonstrably noisy: across the two os-adr campaigns, cells flipped between @@ -90,6 +99,9 @@ salience in the hook note / find-skill description). One "failure" label, two di - **Held-out scenarios stay held out** — never run scenario Task blocks informally/by hand; that contaminates the measurement. - **Broken cells look like model failures** — a missing `transcript.jsonl` scores FAIL on both axes. Check reasons strings for harness errors before counting a cell as a behavioral result. - **Model-tier gaps can be total** — haiku 0/8 on Eval B means wording iteration may not reach the lower tier at all; keep one lower-tier canary cell (haiku W2, the one axis-a flicker) in the loop to detect whether wording changes reach it. +- **`set -euo pipefail` + `checker | tee` silently truncates rep loops** — a checker that exits 1 on FAIL, piped into `tee` inside a runner with pipefail, aborts the remaining reps of that cell on the first FAIL, and the outer shell may still report exit 0. Symptom: `--reps 3` produces one TSV row. Guard the pipeline (`|| true`) and check row counts against expected reps before reading results (Eval C, 2026-07-06). +- **When the instrument was wrong, rescore — don't re-run or discard** — if a checker fix changes scoring but the transcript is valid evidence of behavior, re-run `bin/check` on the existing sandbox and replace the row. Re-running spends a rep on a new behavior sample (different question); discarding is peeking bias. +- **Checker semantics need a conformance dry-run, not just a self-test** — model-free self-tests fabricate the transcripts they validate against, so they inherit the designer's assumptions. Before the first grid: cross-check each scenario's expected behavior against what the shipped wording actually instructs given the fixture state ("would a perfectly compliant model pass this cell?"). See the anti-patterns in [[eval-methodology-ladder]] for the full failure shape. ## Related diff --git a/journal/2026-07-06.md b/journal/2026-07-06.md index 2c1b0b2..a4b9444 100644 --- a/journal/2026-07-06.md +++ b/journal/2026-07-06.md @@ -374,3 +374,40 @@ tags: [scope/global, type/log] **Reason:** other **Vault notes touched:** (none) + +## Session — 2026-07-06T16:38:51Z + +**Project:** /tmp/claude-1000/-home-jared-dev-cc-os/d68aab1b-3e3d-467c-9b9c-70b89e2576b0/scratchpad/adr-eval-c-grid/N1-L1-execution-haiku/N1-L1-execution-haiku-r3 +**Reason:** other +**Vault notes touched:** +(none) + +## Session — 2026-07-06T18:19:49Z + +**Project:** /home/jared/dev/cc-os +**Reason:** prompt_input_exit +**Vault notes touched:** +/home/jared/Documents/SecondBrain/_templates/eval-results.md +/home/jared/Documents/SecondBrain/2026-07-03-os-adr-eval-b-grid-results-and-observations.md +/home/jared/Documents/SecondBrain/2026-07-03-os-adr-eval-b-grid-results-and-observations.md +/home/jared/Documents/SecondBrain/2026-07-03-os-adr-eval-b-grid-results-and-observations.md +/home/jared/Documents/SecondBrain/2026-07-03-os-adr-eval-b-grid-results-and-observations.md +/home/jared/Documents/SecondBrain/os-adr-eval-b-wording-experiment-hypotheses.md +/home/jared/Documents/SecondBrain/os-adr-eval-b-wording-experiment-hypotheses.md +/home/jared/Documents/SecondBrain/os-adr-eval-b-wording-experiment-hypotheses.md +/home/jared/Documents/SecondBrain/os-adr-eval-b-wording-experiment-hypotheses.md +/home/jared/Documents/SecondBrain/eval-methodology-ladder.md +/home/jared/Documents/SecondBrain/eval-methodology-irl-feedback-loop.md +/home/jared/Documents/SecondBrain/eval-methodology-hub.md +/home/jared/Documents/SecondBrain/eval-methodology-ladder.md +/home/jared/Documents/SecondBrain/eval-methodology-irl-feedback-loop.md +/home/jared/Documents/SecondBrain/vault-conventions.md +/home/jared/Documents/SecondBrain/vault-conventions.md +/home/jared/Documents/SecondBrain/vault-conventions.md +/home/jared/Documents/SecondBrain/os-adr-eval-b-wording-experiment-hypotheses.md +/home/jared/Documents/SecondBrain/os-adr-eval-c-frozen-grid-results.md +/home/jared/Documents/SecondBrain/eval-methodology-ladder.md +/home/jared/Documents/SecondBrain/eval-methodology-ladder.md +/home/jared/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md +/home/jared/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md +/home/jared/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md diff --git a/os-adr-eval-c-frozen-grid-results.md b/os-adr-eval-c-frozen-grid-results.md new file mode 100644 index 0000000..609a9a1 --- /dev/null +++ b/os-adr-eval-c-frozen-grid-results.md @@ -0,0 +1,80 @@ +--- +type: eval-results +title: os-adr Eval C frozen grid (2026-07-06) — ambiguity-ladder discrimination baseline +summary: First held-out grid on the taskq fixture — 12/12 cells PASS at 3 reps/cell (haiku 18/18 reps, sonnet 17/18); zero over-trigger ADR creation on any negative at any ladder level. The Eval B wording generalizes to a new fixture, domain, and paired negatives. +tags: + - type/eval-results + - domain/llm-evaluation + - tool/os-adr + - project/cc-os +scope: global +last_updated: 2026-07-06 +date: 2026-07-06 +related: + - os-adr-eval-b-wording-experiment-hypotheses + - os-adr-eval-b-grid-results-and-observations + - eval-methodology-ladder + - eval-methodology-irl-feedback-loop + - running-autoresearch-skill-evals +source: cc-os +--- + +# os-adr Eval C frozen grid (2026-07-06) — ambiguity-ladder discrimination baseline + +## Results Grid/Threshold + +**Passing criterion:** per-cell majority (≥2 of 3 reps). Positives PASS on both axes (a: consultation, b: created / cited-governing / judge-proposed). Negatives PASS on axis (b) only — the sole fail line is unneeded ADR creation; consultation (`A:yes/no`) and truthful citation are informational. + +| Level | Positive sonnet | Positive haiku | Negative sonnet | Negative haiku | +|---|---|---|---|---| +| L1 (explicit) | PASS 2/3 | PASS 3/3 | PASS 3/3 (A:yes 3/3) | PASS 3/3 (A:yes 3/3) | +| L2 (moderate) | PASS 3/3 | PASS 3/3 | PASS 3/3 (A:yes 2/3) | PASS 3/3 (A:yes 3/3) | +| L3 (conceptual) | PASS 3/3 | PASS 3/3 | PASS 3/3 (A:yes 3/3) | PASS 3/3 (A:yes 3/3) | + +**Summary:** 12/12 cells PASS; 35/36 reps PASS (haiku 18/18, sonnet 17/18). No negative rep at any level created an ADR — zero over-triggering on either tier. No positive rep created a duplicate ADR either: every positive pass was via governing-ADR citation (16) or judge-recognized recording proposal (1). Sonnet's single FAIL rep (P1-L1) is a scoring-boundary artifact: it consulted, implemented exactly the ADR-0002 policy, and flagged the un-implemented ADR-0005 escalation — but named only 0005 (not 0002) in its final message, so the strict final-message citation check fell through to the judge. Behaviorally clean; a measurement-boundary flicker analogous to Eval B's W3. + +## Measurement Setup + +**Scenarios Tested** + +Run-set only (job-execution domain, target ADR 0002 with 0001→0002 supersession distractor): P1/N1-L1, P2/N2-L2, P3/N3-L3. Positives sit in already-decided territory — the task restates what an Accepted ADR already decided — so correctness is find → cite → comply. Negatives are non-decisional-but-tempting (test/document/audit framing in ADR-covered domains); correctness is doing the task without fabricating a decision record. The reserve-set (notifications domain) was not touched. + +**Fixture and Sampling** + +New Python `taskq` async job-queue fixture (vs Eval B's Ruby webhook-relay) with a 6-ADR history generated by the plugin's own CLIs; trigger-phrased "Architecture decisions" CLAUDE.md section copied verbatim from eval-b's fixture. 3 reps/cell (36 reps), chosen because 1 rep/cell was demonstrably noisy in both Eval B campaigns. Headless-only (`bin/run`, fresh `claude -p` per rep, cwd = sandbox, real SessionStart hook verified in every transcript). All reps counted — including the pre-fix canary rep, rescored (not re-run) after the checker conformance fix. + +**Experimental Control (Frozen Surfaces)** + +Scenario Task blocks, fixtures, and `judge-rubric.md` untouched throughout. Two harness defects were found on the first live rep and fixed BEFORE the grid (1 rep exposure, transcript preserved and rescored): +1. `bin/run` rep-loop bug — `set -euo pipefail` + piping `bin/check` (exit 1 on FAIL) into `tee` aborted a cell's remaining reps on the first FAIL. +2. Positive axis-b conformance bug — the checker required ADR creation/proposal, contradicting the README's own design ("consulting and/or recording is correct") and the shipped plugin wording: with the positives in already-decided territory, creating an ADR would duplicate the record — exactly what the negatives punish. Fixed by adding a mechanical cited-governing PASS branch (final message cites the governing ADR id) before the unchanged frozen-judge fallback; self-test gained compliance-must-PASS and wrong-citation-must-FAIL guards. + +Under the pre-fix checker the grid would have scored ~1/18 positives — all false negatives. B-via distribution proves it: cited-governing 16, judge-proposed 1, created 0. + +## Validity and Limitations + +**How to Interpret These Results** + +This is a **held-out measurement** — no wording was tuned against Eval C before or during this grid, so unlike Eval B's final numbers these are not training-set scores. What they support: the trigger-conditioned wording shipped from the Eval B loop generalizes across fixture, language (Ruby→Python), domain, decreasing cue explicitness (L1→L3), and — the core discrimination test — paired negatives, on both tiers. The Eval B overfit concern is substantially retired at these ladder levels. What they do not support: behavior on genuinely novel-decision positives (see weaknesses), other languages/project shapes, or production conditions. + +**Weaknesses of This Eval (Its Ladder Level)** + +The grid saturated — no ladder level differentiated the tiers, so the ceiling wasn't found; harder rungs (heavier distractor clutter, conflicting-ADR tasks requiring supersession, genuinely novel decisions requiring creation) need new scenario authoring. All run-set positives are compliance-shaped (decision already recorded); the eval therefore never measured unprompted *creation* of a genuinely new decision — Eval B's W1–W3 remain the only (training-set) evidence for that. The cited-governing check reads the final message only; sonnet's one FAIL shows it can under-credit compliant behavior (~1-in-18 observed). Single fixture per set. Checker fixes were made mid-first-rep (documented above); scenario metadata lines were edited but Task blocks never exposed. + +## Deployment and Evolution + +**Good-Enough Gate** + +Both tiers clear the discrimination gate: zero over-trigger creation across 18 negative reps means the daily-use worry ("will it spam ADRs?") is not supported at any measured ambiguity level. Combined with Eval B (post-wording) and the W3 stability check, os-adr is cleared for real-project rollout on both sonnet and haiku, keeping the known haiku W3 caveat (~1-in-4/5 recording-offer flicker on supersession). Contamination protocol stands: the run-set is now the baseline; the moment anyone tunes wording against these results, measurement moves to the frozen reserve-set. + +**Hardening Path / Next Measurement** + +Not another lab rung — the saturated grid says the next signal is production: proceed with the locked rollout order (pilot projects via `/os-adr:migrate`, then cc-os retrofit) and stand up the [[eval-methodology-irl-feedback-loop]] session-audit backlog; real misses become the scenarios for any future rung (novel-decision positives, supersession-conflict positives, heavy-clutter negatives — reserve-set discipline applies). + +## Related + +- [[os-adr-eval-b-wording-experiment-hypotheses]] — the wording loop whose generalization this grid tested +- [[os-adr-eval-b-grid-results-and-observations]] — original held-out baseline (pre-wording) +- [[eval-methodology-ladder]] — ladder design this eval instantiates +- [[eval-methodology-irl-feedback-loop]] — where measurement goes after a saturated grid +- [[running-autoresearch-skill-evals]] — run-mode and rep-count discipline followed here diff --git a/vault-backlog-pilot-plan.md b/vault-backlog-pilot-plan.md index b1442bb..e2529b5 100644 --- a/vault-backlog-pilot-plan.md +++ b/vault-backlog-pilot-plan.md @@ -1,5 +1,5 @@ --- -summary: Decision + day-1 pilot plan for the vault-native backlog (task notes + push-first surfacing + Bases kanban spike), including the perspectives-panel verdict and the Fable-orchestrator/cheap-subagent execution plan +summary: Backlog decision + pilot plan v2 (2026-07-06) — Planka CE on the OVH prod server as the anywhere-visual kanban, thin Ruby CLI + tick/digest agent layer covering the Planka Pro paywall gaps, vault demoted to knowledge layer; phone drag test is the day-1 gate tags: - scope/global - type/decision @@ -10,69 +10,86 @@ status: accepted-pending-pilot last_reviewed: 2026-07-06 --- -# Vault-native backlog: decision and pilot plan (2026-07-06) +# Backlog system: decision and pilot plan v2 (2026-07-06) -Research behind this decision: [[backlog-system-options-research]]. +Research behind this decision: [[backlog-system-options-research]] (including the same-day second-pass Planka deep-dive and alternatives sweep). + +**Supersedes the v1 vault-native plan (same day).** v1 chose vault task notes + Obsidian Bases kanban. It was overturned within hours by a requirement clarification, recorded here so the reasoning survives: + +## The requirement change that flipped the decision + +The perspectives panel and v1 optimized for "push beats pull." The user corrected this: **the pull surface — a visual board viewable and interactable from anywhere, any device, any time — is the primary requirement**, not a nice-to-have. The founding pain point is "when I want to jump into a project I have no visual interface to see what's on the docket," and with ADHD a desktop-only interface means forgotten/missed tasks. Conversational access ("what's on my plate?" via chat) does not satisfy this; it must be *visual*. + +- Vault-native's known weakness (dashboard maturity, mobile — Bases custom views are desktop-weak) became **disqualifying**. +- Planka's known weakness (task data outside markdown) became **"a development problem"** — solvable via REST API, per the user. ## Decision -Build the personal cross-project backlog **inside the SecondBrain vault**: one note per backlog item (`task` note type), a push-first surfacing layer, Obsidian Bases kanban views as the visual dashboard, and a minimal deterministic CLI. Forgejo issues remain the code-execution tracker (rule: **Forgejo = code/arch/deployment work inside a repo; vault = everything else**; vault tasks may wikilink/reference Forgejo issues). Fallbacks, in order: Backlog.md (if the Bases board spike fails), Planka + community MCP (if markdown-native fails or a standalone/mobile dashboard becomes a hard requirement). +- **Planka v2 Community Edition** (AGPL, plankanban/planka) deployed on the **OVH production server** (`systems-prod-01`, see `~/servers/ovh-prod/`), as a standard `~/services/planka/` docker-compose service behind Traefik with automatic SSL. Auto-discovered by the existing backup script. +- **Why OVH, not the home Proxmox box / hermes01:** always-on independent of home ISP/power; already reachable from anywhere with SSL (phone needs a bookmark, not Tailscale); fits the established deploy + backup pattern. Access control (strong auth at minimum; optional Traefik middleware/IP allowlist — client names will appear on cards) is a deploy-time decision. +- **No filesystem co-location with the vault is needed.** Planka state is Postgres; all integration (CLI, tick, digest, future Hermes) goes through Planka's REST API + git. Cards reference SecondBrain notes by name/URL; any host that ever needs note *content* clones the vault from Forgejo (one command). The vault does NOT currently sync to OVH and doesn't need to. +- **Vault role demoted to knowledge layer:** how-tos, contractor/client notes, conventions, decisions stay in SecondBrain; task *state* lives in Planka. The proven Notion rental pattern (item-with-references) survives split across the two: card = state + links, vault note = knowledge. +- **Forgejo issues remain the code-execution tracker** (rule unchanged: Forgejo = code/arch/deployment work inside a repo; Planka = everything else; cards may link issues). +- **CLI over MCP.** No official Planka CLI exists; we build a thin Ruby one (Sandi Metz style, `lib/` + `bin/`, mirroring the os-adr pattern) over the REST API. `plankapy` (Python SDK) is the endpoint-reference crib. Community MCP servers (bradrisse/kanban-mcp etc.) are shelf reference only. -## Perspectives panel verdict (4 agents: devils-advocate, simplifier, implementer, premortem) +### Fallback ladder (if the phone gate fails) -The panel kept the vault-native direction but **inverted the build order**. Consensus corrections to the original proposal: +1. **TaskView** (taskview.tech) — on-paper ideal (native mobile apps, scoped API tokens, first-party MCP) but source-available, ~6 months old, thin sourcing. *Verify claims hands-on first, then trial.* Modern-and-unproven; only reached if Planka fails the gate. +2. **Vikunja** — mature API, kanban view, but mobile app explicitly alpha; revisit 2027. +3. Vault-native v1 plan (archived below via git history) / Backlog.md — the markdown lane, if hosted boards fail entirely. -1. **The load-bearing layer is push/surfacing, not the data store.** A board that must be opened and a SessionStart hook that fires only when a session starts both fail the ADHD constraint (out of sight = forgotten). The recurring-maintenance scenario (frozen spigots) fails in any week with no Claude sessions. → Push first, board later. -2. **Three reliability requirements or the system is "theater within three months"** (premortem): (a) time-aware idempotent `tick` with a staleness catch-up check, (b) SessionStart text summary of due/P0 items, (c) one-field quick-add capture. -3. **Daemon contradiction resolved honestly** (implementer): a `systemd --user` *timer* (one-shot daily, not a daemon) firing `tick` + an ntfy/email push is the minimum genuine fix for no-session weeks. Recurrence math must be elapsed-time catch-up ("instantiate all missed occurrences since last tick"), never one-session-one-tick. -4. **YAGNI cuts** (simplifier): no recur rule engine (simple offsets like `recur: +1y`); no full CLI at launch (`tick` + quick-add only; list/move/next deferred to demand); board built only after live task data exists (a Bases saved view is minutes of work once notes exist); AI comms drafting (tenant emails) is week-3; effort field + quick-win filter is week-4. -5. **Spike before building** (implementer): verify Base Board / bases-kanban against fixture notes — multi-property filters, ordinal priority sort (use sortable encodings: P0–P3, E1–E3; never low/medium/high strings), drag-drop frontmatter write-back, missing-property rendering, mobile (expect desktop-only). Also spike the Obsidian-open-buffer vs. external-frontmatter-write collision, and verify CLI YAML serialization round-trips byte-identically with Obsidian's `processFrontMatter` (serializer churn pollutes git diffs and widens the conflict surface with the existing `vault_sync.py` SessionEnd push). -6. **Task-rot defense**: the quarterly someday-review is itself a recurring P0 task in the system (dogfooding, zero extra code). -7. **Fair challenge accepted** (devils-advocate): Backlog.md was dismissed too fast (wikilinks do resolve into its files); it stays the named fallback and the ergonomic template for the CLI. And the morning surface should eventually include top Forgejo issues (via `tea`) so the two trackers get one unified decision point (week-2). +**Donetick** (recurring-chore tracker, native mobile) noted as a lane-specific fallback if tick-driven recurrence proves annoying for property maintenance — resisted for now (fifth-backlog trap). -## Task note schema (v1 — to be formalized via /os-vault:design-template new-type lifecycle) +## The Planka Pro paywall and how we route around it -```yaml ---- -summary: -tags: [scope/global, type/task, ] -status: backlog | next | doing | done | someday -priority: P0 | P1 | P2 | P3 # sortable encoding, verified in spike -due: YYYY-MM-DD # optional -recur: +1y | +6m | +3m | +1m # optional, simple offset only — no rule engine ---- - -``` +Planka went commercial (2025); CE keeps the full board UI. Three Pro-only features matter here, all covered by composing around the API — **we never modify or fork Planka**: -Open spike question: whether `status`/`priority` live as frontmatter fields (Bases-friendly, likely) vs. tags — decide during the spike, record in vault-conventions.md. +| Pro-only | Our layer instead | +|---|---| +| Recurring cards | `tick` script creates cards via REST on schedule — idempotent, elapsed-time catch-up (instantiates all missed occurrences), rules ours and portable | +| Cross-board global view | CLI `backlog list --all`, daily digest, and SessionStart brief aggregate across boards via API; in-app stays per-board (acceptable start) | +| Notification providers | The digest is ours (script-sent), Planka's notifiers irrelevant | -## Pilot plan — Fable orchestrates, cheap models do grunt work (runnable in one day) +Known costs: "Pro discovery" banner in the free UI (cosmetic); API-token scoping story unverified (historically username/password; verify at deploy — matters for per-profile Hermes credentials; worst case a tiny API proxy enforces scope, which fits the approval-broker pattern anyway). Project health: v2 GA early 2025, tagged releases slowed since but commits continue — mature/stabilizing with a commercial turn, not dead. 12K+ stars, strong ecosystem. -Fable's role throughout: sequencing, design decisions, spike-result judgment, reviewing every subagent deliverable before it lands, updating vault-conventions.md and cc-os docs/ADR at the end. +## Notification policy (ADHD / notification-blindness constraint) -**Phase 0 — schema + fixtures (parallel, ~30 min)** -- haiku subagent: create `_templates/task.md` + 8–10 fixture task notes in the vault (real content: spigot winterization due 2026-10-15 recur +1y; Woodfin HVAC/electrical/plumbing checkups; 2–3 cc-os someday items — including "adopt Storybloq's handover-doc pattern" as the first captured down-the-road idea; 1–2 client items). Mechanical work from a spec Fable writes. -- Fable: write the spec; verify facet/frontmatter correctness against vault-conventions.md. +Three tiers, designed so silence stays meaningful: -**Phase 1 — the two spikes (the riskiest assumptions, before any CLI code)** -- Bases board spike: human-in-the-loop (Obsidian GUI required). Fable/haiku prepares a step-by-step checklist: install Base Board (and bases-kanban as alternate), build a board over `type/task` fixtures, test the six capabilities from panel point 5. Gate: pass → continue; fail → switch dashboard track to Backlog.md, keep everything else. -- sonnet subagent: concurrent-write spike — script that externally rewrites frontmatter while the file is open/dirty in Obsidian; document observed behavior (reload/prompt/clobber) and the serializer round-trip check. +1. **Ambient (default, zero interruptions):** the board itself + the Claude Code SessionStart brief. Information waits where the user already looks. +2. **Digest (the only routine notification):** at most ONE morning message, on ONE channel, sent **only when something is due or P0** — silent days stay silent. +3. **Interrupt:** essentially never in v1; reserved for genuinely urgent items. -**Phase 2 — the three must-haves (parallel once Phase 1 passes)** -- sonnet subagent: `tick` — deterministic, idempotent, elapsed-time catch-up (advance done+recurring tasks by their offset, flag overdue); plus `backlog add --quick "title"` (defaults: status=backlog, priority=P2). Ruby, Sandi Metz style, thin bins over a small lib — mirror the os-adr `lib/` + `bin/` pattern in cc-os. Includes model-free tests. -- sonnet subagent: SessionStart surfacing hook for the os-vault plugin (Python, matching existing hooks): terse 3–5-line block of due + P0 items, plus the ≥7-days-since-last-tick staleness check that runs `tick` catch-up inline. Timestamp state file in the vault `.state/` (gitignored). -- haiku subagent: `systemd --user` timer + service units firing daily `tick`, with ntfy (or mail) push when property-domain tasks come due — the no-session-week safety net. -- Fable: review all three (correctness, invariants, naming per [[cc-os-plugin-skill-naming-convention]]), integrate into `cc-os/plugins/os-vault/`, run `bin/refresh-plugins`, verify a fresh session surfaces the fixtures. +Target trajectory: reminders are the degenerate case where AI can't act yet. Eventually the agent drafts the action (e.g., tenant spigot email) and the notification becomes "draft ready — approve?" — a decision, not a chore. + +## Pilot plan — Fable orchestrates, cheap models do grunt work + +**Phase 0 — deploy + THE GATE (day 1)** +- sonnet subagent: deploy Planka CE per the ovh-prod service pattern (`~/services/planka/`, compose + postgres, Traefik labels, temp domain first, verify backup discovery). Decide auth posture at deploy. +- haiku subagent: seed boards per hat (property, clients, cc-os/dev, business dev) + 8–10 real fixture cards via REST (spigot winterization due 2026-10-15; Woodfin HVAC/electrical/plumbing checkups; cc-os someday items incl. "adopt Storybloq handover-doc pattern"; 1–2 client items). Also proves API CRUD end-to-end. +- **Human gate — the phone test:** from the phone (responsive web; official app and Planka Pal as alternates), view boards, drag a card between lists, add a card, complete one. Pass → continue. Fail → fallback ladder. + +**Phase 1 — the agent layer (parallel after the gate)** +- sonnet subagent: Ruby `backlog` CLI (`lib/` + `bin/`, model-free tests): `add --quick "title"` (defaults board/list/priority), `list --all` (cross-board), `tick` (idempotent elapsed-time recurrence catch-up from a recurrence manifest; simple offsets `+1y|+6m|+3m|+1m` only — no rule engine). +- haiku subagent: daily `tick` + digest scheduling on the OVH box (cron or a small container next to Planka), digest per the notification policy (one channel — ntfy or email, decide at build; silent when empty). +- sonnet subagent: Claude Code SessionStart brief — terse 3–5 line due/P0 cross-board block via the CLI. Plugin home + naming per [[cc-os-plugin-skill-naming-convention]] (extend os-vault vs. new `os-backlog` — decide at build; run `bin/refresh-plugins` after). +- Fable: review all deliverables; verify a fresh session surfaces the fixtures. + +**Phase 2 — Hermes (independent design exercise, guardrails-first — NOT in the pilot's critical path)** +Considered separately from the task system; the task system's only obligation is integrate-ability, which REST + (verified) scoped tokens provide by construction. Design principles locked now: +- **Capability-based access:** each Hermes profile/hat (property manager, accountant, client manager, business partner) gets its own narrow, revocable credential (e.g., read-only or single-board token). Hermes never holds credentials for catastrophic actions (money movement, deletion, infra). +- **Approval broker:** catastrophic-class actions are *requests* Hermes submits to an interface (n8n workflow / API) that holds the real credentials and routes to the user for approve/reject. Guardrails are the interface, not a policy. +- Sequence: read-only digest/query first, then narrow acting (comms drafting for `property` items), one hat at a time, property manager first. Gets its own design note when started. **Phase 3 — close-out (Fable)** -- Update vault-conventions.md (task type), cc-os CLAUDE.md + build docs, and record the decision as an ADR in cc-os. -- Live for one week with real tasks. Week-2 review gates (each pre-captured as a task note): noisy summary → filter tighter; board friction → revisit Backlog.md; capture demand → extend CLI (`list/move/next`); add Forgejo top-issues to the morning surface. +- cc-os ADR + CLAUDE.md update; retire the v1 vault-task-type idea from vault-conventions planning. +- One-week live trial. Week-2 review gates (each pre-captured as a card): digest too noisy → tighten; capture friction → extend CLI; Forgejo top-issues added to brief; recurrence annoyance → evaluate Donetick for the property lane. -**Explicitly deferred (each captured as a `someday` task, not built):** recur rule engine, effort/quick-win filter, AI comms drafting, mobile dashboard, per-domain board polish, Planka migration. +**Explicitly deferred (captured as someday cards, not built):** recurrence rule engine, effort/quick-win filter, AI comms drafting, Hermes acting autonomously, vault→OVH sync, per-board polish, TaskView evaluation (unless gate fails). -## Success criteria (from premortem) +## Success criteria -1. Recurring tasks cannot silently miss a week (timer + staleness check both firing). -2. Due/P0 items visible every working morning with zero navigation (session surface, later board). -3. Capturing a task costs one command or one sentence to Claude. +1. The board is viewable and interactable **from the phone, anywhere** (the gate, then daily reality). +2. Recurring tasks cannot silently miss a week (tick catch-up on an always-on host). +3. Due/P0 items visible every working morning with zero navigation (SessionStart brief + digest-when-warranted). +4. Capturing a task costs one command or one sentence to Claude.