vault: session notes 2026-07-06

This commit is contained in:
Jared Swanson 2026-07-06 14:19:49 -04:00
parent 8129f6e7ac
commit c905bd3258
5 changed files with 197 additions and 49 deletions

View File

@ -9,7 +9,7 @@ tags:
- convention/eval-design - convention/eval-design
- tool/autoresearch - tool/autoresearch
scope: global scope: global
last_updated: 2026-07-06 last_updated: 2026-07-06 # hardened same day with Eval C first-grid lessons (checker-conformance, self-test blind spot)
date: 2026-07-06 date: 2026-07-06
related: related:
- running-autoresearch-skill-evals - running-autoresearch-skill-evals
@ -83,6 +83,8 @@ Lower model tiers (e.g., haiku vs sonnet) may not clear the same rung at the sam
- **Scenarios that are too ambiguous to score cleanly.** If a negative scenario *looks* correct and you have to debate whether the model should have triggered, the scenario isn't at the right level yet. Move it up to a higher level or rewrite it. - **Scenarios that are too ambiguous to score cleanly.** If a negative scenario *looks* correct and you have to debate whether the model should have triggered, the scenario isn't at the right level yet. Move it up to a higher level or rewrite it.
- **Forgetting that paired scenarios are *equally* subtle.** If your negative scenarios are obviously non-triggers (e.g., "add logging, no decision involved"), you're not testing discrimination. Negative scenarios should be as close to the positive case as possible without crossing the trigger threshold. - **Forgetting that paired scenarios are *equally* subtle.** If your negative scenarios are obviously non-triggers (e.g., "add logging, no decision involved"), you're not testing discrimination. Negative scenarios should be as close to the positive case as possible without crossing the trigger threshold.
- **Tuning against held-out reserve.** Once the reserve is defined, it becomes Schrodinger's eval — useful as a fresh measurement only if it stays frozen. If you peek at reserve results and change wording based on them, it's now training-set. - **Tuning against held-out reserve.** Once the reserve is defined, it becomes Schrodinger's eval — useful as a fresh measurement only if it stays frozen. If you peek at reserve results and change wording based on them, it's now training-set.
- **A checker that contradicts the shipped instructions.** Before spending reps, dry-run the instrument on paper: "would a model that follows the shipped wording perfectly pass every cell?" and "would an always-trigger model pass the positives while failing the negatives?" If yes to the second or no to the first, the eval measures instruction-*disobedience*. Concrete instance (os-adr Eval C, 2026-07-06): positives sat in already-decided ADR territory, where the shipped wording says find→cite→comply — but the checker demanded ADR creation, so a perfectly-behaving model FAILed and an ADR-spamming model would have aced positives. Caught on the first live rep only because of a canary-cell run.
- **Trusting model-free self-tests to validate checker *semantics*.** Fabricated "perfect" transcripts encode the same assumptions as the checker — if the designer's idea of correct behavior is wrong, both agree and the self-test stays green. Self-tests catch mechanical bugs, not conformance bugs. Fabricate at least one transcript of *shipped-instruction-compliant* behavior (what the model is actually told to do, given the fixture state), not just the designer's imagined ideal.
## Known Limitations ## Known Limitations

View File

@ -9,7 +9,7 @@ tags:
- tool/autoresearch - tool/autoresearch
- project/cc-os - project/cc-os
scope: global scope: global
last_updated: 2026-07-04 last_updated: 2026-07-06
date: 2026-07-04 date: 2026-07-04
related: related:
- os-adr-eval-b-grid-results-and-observations - os-adr-eval-b-grid-results-and-observations
@ -63,6 +63,15 @@ min; parallel, it's bounded by the slowest cell (~5 min). Drive the grid script
the session with background Bash — do NOT wrap it in a babysitting subagent (agents self-pause, the session with background Bash — do NOT wrap it in a babysitting subagent (agents self-pause,
need resuming, and re-runs collide with existing sandboxes: "refusing to overwrite"). need resuming, and re-runs collide with existing sandboxes: "refusing to overwrite").
### Step 4b: Canary-cell the first live run of any new harness
Before launching a full grid on a harness that has never run live, run ONE cell first and
verify its TSV row against the raw transcript by hand (did the hook fire? does the reason
string match what the model actually did?). Count the canary's result in the measurement —
never discard it (peeking bias). The os-adr Eval C first grid (2026-07-06) caught two harness
defects this way at a cost of 1 rep instead of 36: a runner rep-loop bug and a checker that
failed instruction-compliant behavior.
### Step 5: Use enough reps to beat the noise ### Step 5: Use enough reps to beat the noise
1 rep/cell is demonstrably noisy: across the two os-adr campaigns, cells flipped between 1 rep/cell is demonstrably noisy: across the two os-adr campaigns, cells flipped between
@ -90,6 +99,9 @@ salience in the hook note / find-skill description). One "failure" label, two di
- **Held-out scenarios stay held out** — never run scenario Task blocks informally/by hand; that contaminates the measurement. - **Held-out scenarios stay held out** — never run scenario Task blocks informally/by hand; that contaminates the measurement.
- **Broken cells look like model failures** — a missing `transcript.jsonl` scores FAIL on both axes. Check reasons strings for harness errors before counting a cell as a behavioral result. - **Broken cells look like model failures** — a missing `transcript.jsonl` scores FAIL on both axes. Check reasons strings for harness errors before counting a cell as a behavioral result.
- **Model-tier gaps can be total** — haiku 0/8 on Eval B means wording iteration may not reach the lower tier at all; keep one lower-tier canary cell (haiku W2, the one axis-a flicker) in the loop to detect whether wording changes reach it. - **Model-tier gaps can be total** — haiku 0/8 on Eval B means wording iteration may not reach the lower tier at all; keep one lower-tier canary cell (haiku W2, the one axis-a flicker) in the loop to detect whether wording changes reach it.
- **`set -euo pipefail` + `checker | tee` silently truncates rep loops** — a checker that exits 1 on FAIL, piped into `tee` inside a runner with pipefail, aborts the remaining reps of that cell on the first FAIL, and the outer shell may still report exit 0. Symptom: `--reps 3` produces one TSV row. Guard the pipeline (`|| true`) and check row counts against expected reps before reading results (Eval C, 2026-07-06).
- **When the instrument was wrong, rescore — don't re-run or discard** — if a checker fix changes scoring but the transcript is valid evidence of behavior, re-run `bin/check` on the existing sandbox and replace the row. Re-running spends a rep on a new behavior sample (different question); discarding is peeking bias.
- **Checker semantics need a conformance dry-run, not just a self-test** — model-free self-tests fabricate the transcripts they validate against, so they inherit the designer's assumptions. Before the first grid: cross-check each scenario's expected behavior against what the shipped wording actually instructs given the fixture state ("would a perfectly compliant model pass this cell?"). See the anti-patterns in [[eval-methodology-ladder]] for the full failure shape.
## Related ## Related

View File

@ -374,3 +374,40 @@ tags: [scope/global, type/log]
**Reason:** other **Reason:** other
**Vault notes touched:** **Vault notes touched:**
(none) (none)
## Session — 2026-07-06T16:38:51Z
**Project:** /tmp/claude-1000/-home-jared-dev-cc-os/d68aab1b-3e3d-467c-9b9c-70b89e2576b0/scratchpad/adr-eval-c-grid/N1-L1-execution-haiku/N1-L1-execution-haiku-r3
**Reason:** other
**Vault notes touched:**
(none)
## Session — 2026-07-06T18:19:49Z
**Project:** /home/jared/dev/cc-os
**Reason:** prompt_input_exit
**Vault notes touched:**
/home/jared/Documents/SecondBrain/_templates/eval-results.md
/home/jared/Documents/SecondBrain/2026-07-03-os-adr-eval-b-grid-results-and-observations.md
/home/jared/Documents/SecondBrain/2026-07-03-os-adr-eval-b-grid-results-and-observations.md
/home/jared/Documents/SecondBrain/2026-07-03-os-adr-eval-b-grid-results-and-observations.md
/home/jared/Documents/SecondBrain/2026-07-03-os-adr-eval-b-grid-results-and-observations.md
/home/jared/Documents/SecondBrain/os-adr-eval-b-wording-experiment-hypotheses.md
/home/jared/Documents/SecondBrain/os-adr-eval-b-wording-experiment-hypotheses.md
/home/jared/Documents/SecondBrain/os-adr-eval-b-wording-experiment-hypotheses.md
/home/jared/Documents/SecondBrain/os-adr-eval-b-wording-experiment-hypotheses.md
/home/jared/Documents/SecondBrain/eval-methodology-ladder.md
/home/jared/Documents/SecondBrain/eval-methodology-irl-feedback-loop.md
/home/jared/Documents/SecondBrain/eval-methodology-hub.md
/home/jared/Documents/SecondBrain/eval-methodology-ladder.md
/home/jared/Documents/SecondBrain/eval-methodology-irl-feedback-loop.md
/home/jared/Documents/SecondBrain/vault-conventions.md
/home/jared/Documents/SecondBrain/vault-conventions.md
/home/jared/Documents/SecondBrain/vault-conventions.md
/home/jared/Documents/SecondBrain/os-adr-eval-b-wording-experiment-hypotheses.md
/home/jared/Documents/SecondBrain/os-adr-eval-c-frozen-grid-results.md
/home/jared/Documents/SecondBrain/eval-methodology-ladder.md
/home/jared/Documents/SecondBrain/eval-methodology-ladder.md
/home/jared/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md
/home/jared/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md
/home/jared/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md

View File

@ -0,0 +1,80 @@
---
type: eval-results
title: os-adr Eval C frozen grid (2026-07-06) — ambiguity-ladder discrimination baseline
summary: First held-out grid on the taskq fixture — 12/12 cells PASS at 3 reps/cell (haiku 18/18 reps, sonnet 17/18); zero over-trigger ADR creation on any negative at any ladder level. The Eval B wording generalizes to a new fixture, domain, and paired negatives.
tags:
- type/eval-results
- domain/llm-evaluation
- tool/os-adr
- project/cc-os
scope: global
last_updated: 2026-07-06
date: 2026-07-06
related:
- os-adr-eval-b-wording-experiment-hypotheses
- os-adr-eval-b-grid-results-and-observations
- eval-methodology-ladder
- eval-methodology-irl-feedback-loop
- running-autoresearch-skill-evals
source: cc-os
---
# os-adr Eval C frozen grid (2026-07-06) — ambiguity-ladder discrimination baseline
## Results Grid/Threshold
**Passing criterion:** per-cell majority (≥2 of 3 reps). Positives PASS on both axes (a: consultation, b: created / cited-governing / judge-proposed). Negatives PASS on axis (b) only — the sole fail line is unneeded ADR creation; consultation (`A:yes/no`) and truthful citation are informational.
| Level | Positive sonnet | Positive haiku | Negative sonnet | Negative haiku |
|---|---|---|---|---|
| L1 (explicit) | PASS 2/3 | PASS 3/3 | PASS 3/3 (A:yes 3/3) | PASS 3/3 (A:yes 3/3) |
| L2 (moderate) | PASS 3/3 | PASS 3/3 | PASS 3/3 (A:yes 2/3) | PASS 3/3 (A:yes 3/3) |
| L3 (conceptual) | PASS 3/3 | PASS 3/3 | PASS 3/3 (A:yes 3/3) | PASS 3/3 (A:yes 3/3) |
**Summary:** 12/12 cells PASS; 35/36 reps PASS (haiku 18/18, sonnet 17/18). No negative rep at any level created an ADR — zero over-triggering on either tier. No positive rep created a duplicate ADR either: every positive pass was via governing-ADR citation (16) or judge-recognized recording proposal (1). Sonnet's single FAIL rep (P1-L1) is a scoring-boundary artifact: it consulted, implemented exactly the ADR-0002 policy, and flagged the un-implemented ADR-0005 escalation — but named only 0005 (not 0002) in its final message, so the strict final-message citation check fell through to the judge. Behaviorally clean; a measurement-boundary flicker analogous to Eval B's W3.
## Measurement Setup
**Scenarios Tested**
Run-set only (job-execution domain, target ADR 0002 with 0001→0002 supersession distractor): P1/N1-L1, P2/N2-L2, P3/N3-L3. Positives sit in already-decided territory — the task restates what an Accepted ADR already decided — so correctness is find → cite → comply. Negatives are non-decisional-but-tempting (test/document/audit framing in ADR-covered domains); correctness is doing the task without fabricating a decision record. The reserve-set (notifications domain) was not touched.
**Fixture and Sampling**
New Python `taskq` async job-queue fixture (vs Eval B's Ruby webhook-relay) with a 6-ADR history generated by the plugin's own CLIs; trigger-phrased "Architecture decisions" CLAUDE.md section copied verbatim from eval-b's fixture. 3 reps/cell (36 reps), chosen because 1 rep/cell was demonstrably noisy in both Eval B campaigns. Headless-only (`bin/run`, fresh `claude -p` per rep, cwd = sandbox, real SessionStart hook verified in every transcript). All reps counted — including the pre-fix canary rep, rescored (not re-run) after the checker conformance fix.
**Experimental Control (Frozen Surfaces)**
Scenario Task blocks, fixtures, and `judge-rubric.md` untouched throughout. Two harness defects were found on the first live rep and fixed BEFORE the grid (1 rep exposure, transcript preserved and rescored):
1. `bin/run` rep-loop bug — `set -euo pipefail` + piping `bin/check` (exit 1 on FAIL) into `tee` aborted a cell's remaining reps on the first FAIL.
2. Positive axis-b conformance bug — the checker required ADR creation/proposal, contradicting the README's own design ("consulting and/or recording is correct") and the shipped plugin wording: with the positives in already-decided territory, creating an ADR would duplicate the record — exactly what the negatives punish. Fixed by adding a mechanical cited-governing PASS branch (final message cites the governing ADR id) before the unchanged frozen-judge fallback; self-test gained compliance-must-PASS and wrong-citation-must-FAIL guards.
Under the pre-fix checker the grid would have scored ~1/18 positives — all false negatives. B-via distribution proves it: cited-governing 16, judge-proposed 1, created 0.
## Validity and Limitations
**How to Interpret These Results**
This is a **held-out measurement** — no wording was tuned against Eval C before or during this grid, so unlike Eval B's final numbers these are not training-set scores. What they support: the trigger-conditioned wording shipped from the Eval B loop generalizes across fixture, language (Ruby→Python), domain, decreasing cue explicitness (L1→L3), and — the core discrimination test — paired negatives, on both tiers. The Eval B overfit concern is substantially retired at these ladder levels. What they do not support: behavior on genuinely novel-decision positives (see weaknesses), other languages/project shapes, or production conditions.
**Weaknesses of This Eval (Its Ladder Level)**
The grid saturated — no ladder level differentiated the tiers, so the ceiling wasn't found; harder rungs (heavier distractor clutter, conflicting-ADR tasks requiring supersession, genuinely novel decisions requiring creation) need new scenario authoring. All run-set positives are compliance-shaped (decision already recorded); the eval therefore never measured unprompted *creation* of a genuinely new decision — Eval B's W1W3 remain the only (training-set) evidence for that. The cited-governing check reads the final message only; sonnet's one FAIL shows it can under-credit compliant behavior (~1-in-18 observed). Single fixture per set. Checker fixes were made mid-first-rep (documented above); scenario metadata lines were edited but Task blocks never exposed.
## Deployment and Evolution
**Good-Enough Gate**
Both tiers clear the discrimination gate: zero over-trigger creation across 18 negative reps means the daily-use worry ("will it spam ADRs?") is not supported at any measured ambiguity level. Combined with Eval B (post-wording) and the W3 stability check, os-adr is cleared for real-project rollout on both sonnet and haiku, keeping the known haiku W3 caveat (~1-in-4/5 recording-offer flicker on supersession). Contamination protocol stands: the run-set is now the baseline; the moment anyone tunes wording against these results, measurement moves to the frozen reserve-set.
**Hardening Path / Next Measurement**
Not another lab rung — the saturated grid says the next signal is production: proceed with the locked rollout order (pilot projects via `/os-adr:migrate`, then cc-os retrofit) and stand up the [[eval-methodology-irl-feedback-loop]] session-audit backlog; real misses become the scenarios for any future rung (novel-decision positives, supersession-conflict positives, heavy-clutter negatives — reserve-set discipline applies).
## Related
- [[os-adr-eval-b-wording-experiment-hypotheses]] — the wording loop whose generalization this grid tested
- [[os-adr-eval-b-grid-results-and-observations]] — original held-out baseline (pre-wording)
- [[eval-methodology-ladder]] — ladder design this eval instantiates
- [[eval-methodology-irl-feedback-loop]] — where measurement goes after a saturated grid
- [[running-autoresearch-skill-evals]] — run-mode and rep-count discipline followed here

View File

@ -1,5 +1,5 @@
--- ---
summary: Decision + day-1 pilot plan for the vault-native backlog (task notes + push-first surfacing + Bases kanban spike), including the perspectives-panel verdict and the Fable-orchestrator/cheap-subagent execution plan summary: Backlog decision + pilot plan v2 (2026-07-06) — Planka CE on the OVH prod server as the anywhere-visual kanban, thin Ruby CLI + tick/digest agent layer covering the Planka Pro paywall gaps, vault demoted to knowledge layer; phone drag test is the day-1 gate
tags: tags:
- scope/global - scope/global
- type/decision - type/decision
@ -10,69 +10,86 @@ status: accepted-pending-pilot
last_reviewed: 2026-07-06 last_reviewed: 2026-07-06
--- ---
# Vault-native backlog: decision and pilot plan (2026-07-06) # Backlog system: decision and pilot plan v2 (2026-07-06)
Research behind this decision: [[backlog-system-options-research]]. Research behind this decision: [[backlog-system-options-research]] (including the same-day second-pass Planka deep-dive and alternatives sweep).
**Supersedes the v1 vault-native plan (same day).** v1 chose vault task notes + Obsidian Bases kanban. It was overturned within hours by a requirement clarification, recorded here so the reasoning survives:
## The requirement change that flipped the decision
The perspectives panel and v1 optimized for "push beats pull." The user corrected this: **the pull surface — a visual board viewable and interactable from anywhere, any device, any time — is the primary requirement**, not a nice-to-have. The founding pain point is "when I want to jump into a project I have no visual interface to see what's on the docket," and with ADHD a desktop-only interface means forgotten/missed tasks. Conversational access ("what's on my plate?" via chat) does not satisfy this; it must be *visual*.
- Vault-native's known weakness (dashboard maturity, mobile — Bases custom views are desktop-weak) became **disqualifying**.
- Planka's known weakness (task data outside markdown) became **"a development problem"** — solvable via REST API, per the user.
## Decision ## Decision
Build the personal cross-project backlog **inside the SecondBrain vault**: one note per backlog item (`task` note type), a push-first surfacing layer, Obsidian Bases kanban views as the visual dashboard, and a minimal deterministic CLI. Forgejo issues remain the code-execution tracker (rule: **Forgejo = code/arch/deployment work inside a repo; vault = everything else**; vault tasks may wikilink/reference Forgejo issues). Fallbacks, in order: Backlog.md (if the Bases board spike fails), Planka + community MCP (if markdown-native fails or a standalone/mobile dashboard becomes a hard requirement). - **Planka v2 Community Edition** (AGPL, plankanban/planka) deployed on the **OVH production server** (`systems-prod-01`, see `~/servers/ovh-prod/`), as a standard `~/services/planka/` docker-compose service behind Traefik with automatic SSL. Auto-discovered by the existing backup script.
- **Why OVH, not the home Proxmox box / hermes01:** always-on independent of home ISP/power; already reachable from anywhere with SSL (phone needs a bookmark, not Tailscale); fits the established deploy + backup pattern. Access control (strong auth at minimum; optional Traefik middleware/IP allowlist — client names will appear on cards) is a deploy-time decision.
- **No filesystem co-location with the vault is needed.** Planka state is Postgres; all integration (CLI, tick, digest, future Hermes) goes through Planka's REST API + git. Cards reference SecondBrain notes by name/URL; any host that ever needs note *content* clones the vault from Forgejo (one command). The vault does NOT currently sync to OVH and doesn't need to.
- **Vault role demoted to knowledge layer:** how-tos, contractor/client notes, conventions, decisions stay in SecondBrain; task *state* lives in Planka. The proven Notion rental pattern (item-with-references) survives split across the two: card = state + links, vault note = knowledge.
- **Forgejo issues remain the code-execution tracker** (rule unchanged: Forgejo = code/arch/deployment work inside a repo; Planka = everything else; cards may link issues).
- **CLI over MCP.** No official Planka CLI exists; we build a thin Ruby one (Sandi Metz style, `lib/` + `bin/`, mirroring the os-adr pattern) over the REST API. `plankapy` (Python SDK) is the endpoint-reference crib. Community MCP servers (bradrisse/kanban-mcp etc.) are shelf reference only.
## Perspectives panel verdict (4 agents: devils-advocate, simplifier, implementer, premortem) ### Fallback ladder (if the phone gate fails)
The panel kept the vault-native direction but **inverted the build order**. Consensus corrections to the original proposal: 1. **TaskView** (taskview.tech) — on-paper ideal (native mobile apps, scoped API tokens, first-party MCP) but source-available, ~6 months old, thin sourcing. *Verify claims hands-on first, then trial.* Modern-and-unproven; only reached if Planka fails the gate.
2. **Vikunja** — mature API, kanban view, but mobile app explicitly alpha; revisit 2027.
3. Vault-native v1 plan (archived below via git history) / Backlog.md — the markdown lane, if hosted boards fail entirely.
1. **The load-bearing layer is push/surfacing, not the data store.** A board that must be opened and a SessionStart hook that fires only when a session starts both fail the ADHD constraint (out of sight = forgotten). The recurring-maintenance scenario (frozen spigots) fails in any week with no Claude sessions. → Push first, board later. **Donetick** (recurring-chore tracker, native mobile) noted as a lane-specific fallback if tick-driven recurrence proves annoying for property maintenance — resisted for now (fifth-backlog trap).
2. **Three reliability requirements or the system is "theater within three months"** (premortem): (a) time-aware idempotent `tick` with a staleness catch-up check, (b) SessionStart text summary of due/P0 items, (c) one-field quick-add capture.
3. **Daemon contradiction resolved honestly** (implementer): a `systemd --user` *timer* (one-shot daily, not a daemon) firing `tick` + an ntfy/email push is the minimum genuine fix for no-session weeks. Recurrence math must be elapsed-time catch-up ("instantiate all missed occurrences since last tick"), never one-session-one-tick.
4. **YAGNI cuts** (simplifier): no recur rule engine (simple offsets like `recur: +1y`); no full CLI at launch (`tick` + quick-add only; list/move/next deferred to demand); board built only after live task data exists (a Bases saved view is minutes of work once notes exist); AI comms drafting (tenant emails) is week-3; effort field + quick-win filter is week-4.
5. **Spike before building** (implementer): verify Base Board / bases-kanban against fixture notes — multi-property filters, ordinal priority sort (use sortable encodings: P0P3, E1E3; never low/medium/high strings), drag-drop frontmatter write-back, missing-property rendering, mobile (expect desktop-only). Also spike the Obsidian-open-buffer vs. external-frontmatter-write collision, and verify CLI YAML serialization round-trips byte-identically with Obsidian's `processFrontMatter` (serializer churn pollutes git diffs and widens the conflict surface with the existing `vault_sync.py` SessionEnd push).
6. **Task-rot defense**: the quarterly someday-review is itself a recurring P0 task in the system (dogfooding, zero extra code).
7. **Fair challenge accepted** (devils-advocate): Backlog.md was dismissed too fast (wikilinks do resolve into its files); it stays the named fallback and the ergonomic template for the CLI. And the morning surface should eventually include top Forgejo issues (via `tea`) so the two trackers get one unified decision point (week-2).
## Task note schema (v1 — to be formalized via /os-vault:design-template new-type lifecycle) ## The Planka Pro paywall and how we route around it
```yaml Planka went commercial (2025); CE keeps the full board UI. Three Pro-only features matter here, all covered by composing around the API — **we never modify or fork Planka**:
---
summary: <one-line>
tags: [scope/global, type/task, <client|project|domain facet>]
status: backlog | next | doing | done | someday
priority: P0 | P1 | P2 | P3 # sortable encoding, verified in spike
due: YYYY-MM-DD # optional
recur: +1y | +6m | +3m | +1m # optional, simple offset only — no rule engine
---
<body: context + wikilinks to how-tos, client notes, property/contractor notes,
Forgejo issue URLs — the Notion rental-property pattern>
```
Open spike question: whether `status`/`priority` live as frontmatter fields (Bases-friendly, likely) vs. tags — decide during the spike, record in vault-conventions.md. | Pro-only | Our layer instead |
|---|---|
| Recurring cards | `tick` script creates cards via REST on schedule — idempotent, elapsed-time catch-up (instantiates all missed occurrences), rules ours and portable |
| Cross-board global view | CLI `backlog list --all`, daily digest, and SessionStart brief aggregate across boards via API; in-app stays per-board (acceptable start) |
| Notification providers | The digest is ours (script-sent), Planka's notifiers irrelevant |
## Pilot plan — Fable orchestrates, cheap models do grunt work (runnable in one day) Known costs: "Pro discovery" banner in the free UI (cosmetic); API-token scoping story unverified (historically username/password; verify at deploy — matters for per-profile Hermes credentials; worst case a tiny API proxy enforces scope, which fits the approval-broker pattern anyway). Project health: v2 GA early 2025, tagged releases slowed since but commits continue — mature/stabilizing with a commercial turn, not dead. 12K+ stars, strong ecosystem.
Fable's role throughout: sequencing, design decisions, spike-result judgment, reviewing every subagent deliverable before it lands, updating vault-conventions.md and cc-os docs/ADR at the end. ## Notification policy (ADHD / notification-blindness constraint)
**Phase 0 — schema + fixtures (parallel, ~30 min)** Three tiers, designed so silence stays meaningful:
- haiku subagent: create `_templates/task.md` + 810 fixture task notes in the vault (real content: spigot winterization due 2026-10-15 recur +1y; Woodfin HVAC/electrical/plumbing checkups; 23 cc-os someday items — including "adopt Storybloq's handover-doc pattern" as the first captured down-the-road idea; 12 client items). Mechanical work from a spec Fable writes.
- Fable: write the spec; verify facet/frontmatter correctness against vault-conventions.md.
**Phase 1 — the two spikes (the riskiest assumptions, before any CLI code)** 1. **Ambient (default, zero interruptions):** the board itself + the Claude Code SessionStart brief. Information waits where the user already looks.
- Bases board spike: human-in-the-loop (Obsidian GUI required). Fable/haiku prepares a step-by-step checklist: install Base Board (and bases-kanban as alternate), build a board over `type/task` fixtures, test the six capabilities from panel point 5. Gate: pass → continue; fail → switch dashboard track to Backlog.md, keep everything else. 2. **Digest (the only routine notification):** at most ONE morning message, on ONE channel, sent **only when something is due or P0** — silent days stay silent.
- sonnet subagent: concurrent-write spike — script that externally rewrites frontmatter while the file is open/dirty in Obsidian; document observed behavior (reload/prompt/clobber) and the serializer round-trip check. 3. **Interrupt:** essentially never in v1; reserved for genuinely urgent items.
**Phase 2 — the three must-haves (parallel once Phase 1 passes)** Target trajectory: reminders are the degenerate case where AI can't act yet. Eventually the agent drafts the action (e.g., tenant spigot email) and the notification becomes "draft ready — approve?" — a decision, not a chore.
- sonnet subagent: `tick` — deterministic, idempotent, elapsed-time catch-up (advance done+recurring tasks by their offset, flag overdue); plus `backlog add --quick "title"` (defaults: status=backlog, priority=P2). Ruby, Sandi Metz style, thin bins over a small lib — mirror the os-adr `lib/` + `bin/` pattern in cc-os. Includes model-free tests.
- sonnet subagent: SessionStart surfacing hook for the os-vault plugin (Python, matching existing hooks): terse 35-line block of due + P0 items, plus the ≥7-days-since-last-tick staleness check that runs `tick` catch-up inline. Timestamp state file in the vault `.state/` (gitignored). ## Pilot plan — Fable orchestrates, cheap models do grunt work
- haiku subagent: `systemd --user` timer + service units firing daily `tick`, with ntfy (or mail) push when property-domain tasks come due — the no-session-week safety net.
- Fable: review all three (correctness, invariants, naming per [[cc-os-plugin-skill-naming-convention]]), integrate into `cc-os/plugins/os-vault/`, run `bin/refresh-plugins`, verify a fresh session surfaces the fixtures. **Phase 0 — deploy + THE GATE (day 1)**
- sonnet subagent: deploy Planka CE per the ovh-prod service pattern (`~/services/planka/`, compose + postgres, Traefik labels, temp domain first, verify backup discovery). Decide auth posture at deploy.
- haiku subagent: seed boards per hat (property, clients, cc-os/dev, business dev) + 810 real fixture cards via REST (spigot winterization due 2026-10-15; Woodfin HVAC/electrical/plumbing checkups; cc-os someday items incl. "adopt Storybloq handover-doc pattern"; 12 client items). Also proves API CRUD end-to-end.
- **Human gate — the phone test:** from the phone (responsive web; official app and Planka Pal as alternates), view boards, drag a card between lists, add a card, complete one. Pass → continue. Fail → fallback ladder.
**Phase 1 — the agent layer (parallel after the gate)**
- sonnet subagent: Ruby `backlog` CLI (`lib/` + `bin/`, model-free tests): `add --quick "title"` (defaults board/list/priority), `list --all` (cross-board), `tick` (idempotent elapsed-time recurrence catch-up from a recurrence manifest; simple offsets `+1y|+6m|+3m|+1m` only — no rule engine).
- haiku subagent: daily `tick` + digest scheduling on the OVH box (cron or a small container next to Planka), digest per the notification policy (one channel — ntfy or email, decide at build; silent when empty).
- sonnet subagent: Claude Code SessionStart brief — terse 35 line due/P0 cross-board block via the CLI. Plugin home + naming per [[cc-os-plugin-skill-naming-convention]] (extend os-vault vs. new `os-backlog` — decide at build; run `bin/refresh-plugins` after).
- Fable: review all deliverables; verify a fresh session surfaces the fixtures.
**Phase 2 — Hermes (independent design exercise, guardrails-first — NOT in the pilot's critical path)**
Considered separately from the task system; the task system's only obligation is integrate-ability, which REST + (verified) scoped tokens provide by construction. Design principles locked now:
- **Capability-based access:** each Hermes profile/hat (property manager, accountant, client manager, business partner) gets its own narrow, revocable credential (e.g., read-only or single-board token). Hermes never holds credentials for catastrophic actions (money movement, deletion, infra).
- **Approval broker:** catastrophic-class actions are *requests* Hermes submits to an interface (n8n workflow / API) that holds the real credentials and routes to the user for approve/reject. Guardrails are the interface, not a policy.
- Sequence: read-only digest/query first, then narrow acting (comms drafting for `property` items), one hat at a time, property manager first. Gets its own design note when started.
**Phase 3 — close-out (Fable)** **Phase 3 — close-out (Fable)**
- Update vault-conventions.md (task type), cc-os CLAUDE.md + build docs, and record the decision as an ADR in cc-os. - cc-os ADR + CLAUDE.md update; retire the v1 vault-task-type idea from vault-conventions planning.
- Live for one week with real tasks. Week-2 review gates (each pre-captured as a task note): noisy summary → filter tighter; board friction → revisit Backlog.md; capture demand → extend CLI (`list/move/next`); add Forgejo top-issues to the morning surface. - One-week live trial. Week-2 review gates (each pre-captured as a card): digest too noisy → tighten; capture friction → extend CLI; Forgejo top-issues added to brief; recurrence annoyance → evaluate Donetick for the property lane.
**Explicitly deferred (each captured as a `someday` task, not built):** recur rule engine, effort/quick-win filter, AI comms drafting, mobile dashboard, per-domain board polish, Planka migration. **Explicitly deferred (captured as someday cards, not built):** recurrence rule engine, effort/quick-win filter, AI comms drafting, Hermes acting autonomously, vault→OVH sync, per-board polish, TaskView evaluation (unless gate fails).
## Success criteria (from premortem) ## Success criteria
1. Recurring tasks cannot silently miss a week (timer + staleness check both firing). 1. The board is viewable and interactable **from the phone, anywhere** (the gate, then daily reality).
2. Due/P0 items visible every working morning with zero navigation (session surface, later board). 2. Recurring tasks cannot silently miss a week (tick catch-up on an always-on host).
3. Capturing a task costs one command or one sentence to Claude. 3. Due/P0 items visible every working morning with zero navigation (SessionStart brief + digest-when-warranted).
4. Capturing a task costs one command or one sentence to Claude.