vault: session notes 2026-07-06
This commit is contained in:
parent
8129f6e7ac
commit
c905bd3258
|
|
@ -9,7 +9,7 @@ tags:
|
||||||
- convention/eval-design
|
- convention/eval-design
|
||||||
- tool/autoresearch
|
- tool/autoresearch
|
||||||
scope: global
|
scope: global
|
||||||
last_updated: 2026-07-06
|
last_updated: 2026-07-06 # hardened same day with Eval C first-grid lessons (checker-conformance, self-test blind spot)
|
||||||
date: 2026-07-06
|
date: 2026-07-06
|
||||||
related:
|
related:
|
||||||
- running-autoresearch-skill-evals
|
- running-autoresearch-skill-evals
|
||||||
|
|
@ -83,6 +83,8 @@ Lower model tiers (e.g., haiku vs sonnet) may not clear the same rung at the sam
|
||||||
- **Scenarios that are too ambiguous to score cleanly.** If a negative scenario *looks* correct and you have to debate whether the model should have triggered, the scenario isn't at the right level yet. Move it up to a higher level or rewrite it.
|
- **Scenarios that are too ambiguous to score cleanly.** If a negative scenario *looks* correct and you have to debate whether the model should have triggered, the scenario isn't at the right level yet. Move it up to a higher level or rewrite it.
|
||||||
- **Forgetting that paired scenarios are *equally* subtle.** If your negative scenarios are obviously non-triggers (e.g., "add logging, no decision involved"), you're not testing discrimination. Negative scenarios should be as close to the positive case as possible without crossing the trigger threshold.
|
- **Forgetting that paired scenarios are *equally* subtle.** If your negative scenarios are obviously non-triggers (e.g., "add logging, no decision involved"), you're not testing discrimination. Negative scenarios should be as close to the positive case as possible without crossing the trigger threshold.
|
||||||
- **Tuning against held-out reserve.** Once the reserve is defined, it becomes Schrodinger's eval — useful as a fresh measurement only if it stays frozen. If you peek at reserve results and change wording based on them, it's now training-set.
|
- **Tuning against held-out reserve.** Once the reserve is defined, it becomes Schrodinger's eval — useful as a fresh measurement only if it stays frozen. If you peek at reserve results and change wording based on them, it's now training-set.
|
||||||
|
- **A checker that contradicts the shipped instructions.** Before spending reps, dry-run the instrument on paper: "would a model that follows the shipped wording perfectly pass every cell?" and "would an always-trigger model pass the positives while failing the negatives?" If yes to the second or no to the first, the eval measures instruction-*disobedience*. Concrete instance (os-adr Eval C, 2026-07-06): positives sat in already-decided ADR territory, where the shipped wording says find→cite→comply — but the checker demanded ADR creation, so a perfectly-behaving model FAILed and an ADR-spamming model would have aced positives. Caught on the first live rep only because of a canary-cell run.
|
||||||
|
- **Trusting model-free self-tests to validate checker *semantics*.** Fabricated "perfect" transcripts encode the same assumptions as the checker — if the designer's idea of correct behavior is wrong, both agree and the self-test stays green. Self-tests catch mechanical bugs, not conformance bugs. Fabricate at least one transcript of *shipped-instruction-compliant* behavior (what the model is actually told to do, given the fixture state), not just the designer's imagined ideal.
|
||||||
|
|
||||||
## Known Limitations
|
## Known Limitations
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -9,7 +9,7 @@ tags:
|
||||||
- tool/autoresearch
|
- tool/autoresearch
|
||||||
- project/cc-os
|
- project/cc-os
|
||||||
scope: global
|
scope: global
|
||||||
last_updated: 2026-07-04
|
last_updated: 2026-07-06
|
||||||
date: 2026-07-04
|
date: 2026-07-04
|
||||||
related:
|
related:
|
||||||
- os-adr-eval-b-grid-results-and-observations
|
- os-adr-eval-b-grid-results-and-observations
|
||||||
|
|
@ -63,6 +63,15 @@ min; parallel, it's bounded by the slowest cell (~5 min). Drive the grid script
|
||||||
the session with background Bash — do NOT wrap it in a babysitting subagent (agents self-pause,
|
the session with background Bash — do NOT wrap it in a babysitting subagent (agents self-pause,
|
||||||
need resuming, and re-runs collide with existing sandboxes: "refusing to overwrite").
|
need resuming, and re-runs collide with existing sandboxes: "refusing to overwrite").
|
||||||
|
|
||||||
|
### Step 4b: Canary-cell the first live run of any new harness
|
||||||
|
|
||||||
|
Before launching a full grid on a harness that has never run live, run ONE cell first and
|
||||||
|
verify its TSV row against the raw transcript by hand (did the hook fire? does the reason
|
||||||
|
string match what the model actually did?). Count the canary's result in the measurement —
|
||||||
|
never discard it (peeking bias). The os-adr Eval C first grid (2026-07-06) caught two harness
|
||||||
|
defects this way at a cost of 1 rep instead of 36: a runner rep-loop bug and a checker that
|
||||||
|
failed instruction-compliant behavior.
|
||||||
|
|
||||||
### Step 5: Use enough reps to beat the noise
|
### Step 5: Use enough reps to beat the noise
|
||||||
|
|
||||||
1 rep/cell is demonstrably noisy: across the two os-adr campaigns, cells flipped between
|
1 rep/cell is demonstrably noisy: across the two os-adr campaigns, cells flipped between
|
||||||
|
|
@ -90,6 +99,9 @@ salience in the hook note / find-skill description). One "failure" label, two di
|
||||||
- **Held-out scenarios stay held out** — never run scenario Task blocks informally/by hand; that contaminates the measurement.
|
- **Held-out scenarios stay held out** — never run scenario Task blocks informally/by hand; that contaminates the measurement.
|
||||||
- **Broken cells look like model failures** — a missing `transcript.jsonl` scores FAIL on both axes. Check reasons strings for harness errors before counting a cell as a behavioral result.
|
- **Broken cells look like model failures** — a missing `transcript.jsonl` scores FAIL on both axes. Check reasons strings for harness errors before counting a cell as a behavioral result.
|
||||||
- **Model-tier gaps can be total** — haiku 0/8 on Eval B means wording iteration may not reach the lower tier at all; keep one lower-tier canary cell (haiku W2, the one axis-a flicker) in the loop to detect whether wording changes reach it.
|
- **Model-tier gaps can be total** — haiku 0/8 on Eval B means wording iteration may not reach the lower tier at all; keep one lower-tier canary cell (haiku W2, the one axis-a flicker) in the loop to detect whether wording changes reach it.
|
||||||
|
- **`set -euo pipefail` + `checker | tee` silently truncates rep loops** — a checker that exits 1 on FAIL, piped into `tee` inside a runner with pipefail, aborts the remaining reps of that cell on the first FAIL, and the outer shell may still report exit 0. Symptom: `--reps 3` produces one TSV row. Guard the pipeline (`|| true`) and check row counts against expected reps before reading results (Eval C, 2026-07-06).
|
||||||
|
- **When the instrument was wrong, rescore — don't re-run or discard** — if a checker fix changes scoring but the transcript is valid evidence of behavior, re-run `bin/check` on the existing sandbox and replace the row. Re-running spends a rep on a new behavior sample (different question); discarding is peeking bias.
|
||||||
|
- **Checker semantics need a conformance dry-run, not just a self-test** — model-free self-tests fabricate the transcripts they validate against, so they inherit the designer's assumptions. Before the first grid: cross-check each scenario's expected behavior against what the shipped wording actually instructs given the fixture state ("would a perfectly compliant model pass this cell?"). See the anti-patterns in [[eval-methodology-ladder]] for the full failure shape.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -374,3 +374,40 @@ tags: [scope/global, type/log]
|
||||||
**Reason:** other
|
**Reason:** other
|
||||||
**Vault notes touched:**
|
**Vault notes touched:**
|
||||||
(none)
|
(none)
|
||||||
|
|
||||||
|
## Session — 2026-07-06T16:38:51Z
|
||||||
|
|
||||||
|
**Project:** /tmp/claude-1000/-home-jared-dev-cc-os/d68aab1b-3e3d-467c-9b9c-70b89e2576b0/scratchpad/adr-eval-c-grid/N1-L1-execution-haiku/N1-L1-execution-haiku-r3
|
||||||
|
**Reason:** other
|
||||||
|
**Vault notes touched:**
|
||||||
|
(none)
|
||||||
|
|
||||||
|
## Session — 2026-07-06T18:19:49Z
|
||||||
|
|
||||||
|
**Project:** /home/jared/dev/cc-os
|
||||||
|
**Reason:** prompt_input_exit
|
||||||
|
**Vault notes touched:**
|
||||||
|
/home/jared/Documents/SecondBrain/_templates/eval-results.md
|
||||||
|
/home/jared/Documents/SecondBrain/2026-07-03-os-adr-eval-b-grid-results-and-observations.md
|
||||||
|
/home/jared/Documents/SecondBrain/2026-07-03-os-adr-eval-b-grid-results-and-observations.md
|
||||||
|
/home/jared/Documents/SecondBrain/2026-07-03-os-adr-eval-b-grid-results-and-observations.md
|
||||||
|
/home/jared/Documents/SecondBrain/2026-07-03-os-adr-eval-b-grid-results-and-observations.md
|
||||||
|
/home/jared/Documents/SecondBrain/os-adr-eval-b-wording-experiment-hypotheses.md
|
||||||
|
/home/jared/Documents/SecondBrain/os-adr-eval-b-wording-experiment-hypotheses.md
|
||||||
|
/home/jared/Documents/SecondBrain/os-adr-eval-b-wording-experiment-hypotheses.md
|
||||||
|
/home/jared/Documents/SecondBrain/os-adr-eval-b-wording-experiment-hypotheses.md
|
||||||
|
/home/jared/Documents/SecondBrain/eval-methodology-ladder.md
|
||||||
|
/home/jared/Documents/SecondBrain/eval-methodology-irl-feedback-loop.md
|
||||||
|
/home/jared/Documents/SecondBrain/eval-methodology-hub.md
|
||||||
|
/home/jared/Documents/SecondBrain/eval-methodology-ladder.md
|
||||||
|
/home/jared/Documents/SecondBrain/eval-methodology-irl-feedback-loop.md
|
||||||
|
/home/jared/Documents/SecondBrain/vault-conventions.md
|
||||||
|
/home/jared/Documents/SecondBrain/vault-conventions.md
|
||||||
|
/home/jared/Documents/SecondBrain/vault-conventions.md
|
||||||
|
/home/jared/Documents/SecondBrain/os-adr-eval-b-wording-experiment-hypotheses.md
|
||||||
|
/home/jared/Documents/SecondBrain/os-adr-eval-c-frozen-grid-results.md
|
||||||
|
/home/jared/Documents/SecondBrain/eval-methodology-ladder.md
|
||||||
|
/home/jared/Documents/SecondBrain/eval-methodology-ladder.md
|
||||||
|
/home/jared/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md
|
||||||
|
/home/jared/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md
|
||||||
|
/home/jared/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,80 @@
|
||||||
|
---
|
||||||
|
type: eval-results
|
||||||
|
title: os-adr Eval C frozen grid (2026-07-06) — ambiguity-ladder discrimination baseline
|
||||||
|
summary: First held-out grid on the taskq fixture — 12/12 cells PASS at 3 reps/cell (haiku 18/18 reps, sonnet 17/18); zero over-trigger ADR creation on any negative at any ladder level. The Eval B wording generalizes to a new fixture, domain, and paired negatives.
|
||||||
|
tags:
|
||||||
|
- type/eval-results
|
||||||
|
- domain/llm-evaluation
|
||||||
|
- tool/os-adr
|
||||||
|
- project/cc-os
|
||||||
|
scope: global
|
||||||
|
last_updated: 2026-07-06
|
||||||
|
date: 2026-07-06
|
||||||
|
related:
|
||||||
|
- os-adr-eval-b-wording-experiment-hypotheses
|
||||||
|
- os-adr-eval-b-grid-results-and-observations
|
||||||
|
- eval-methodology-ladder
|
||||||
|
- eval-methodology-irl-feedback-loop
|
||||||
|
- running-autoresearch-skill-evals
|
||||||
|
source: cc-os
|
||||||
|
---
|
||||||
|
|
||||||
|
# os-adr Eval C frozen grid (2026-07-06) — ambiguity-ladder discrimination baseline
|
||||||
|
|
||||||
|
## Results Grid/Threshold
|
||||||
|
|
||||||
|
**Passing criterion:** per-cell majority (≥2 of 3 reps). Positives PASS on both axes (a: consultation, b: created / cited-governing / judge-proposed). Negatives PASS on axis (b) only — the sole fail line is unneeded ADR creation; consultation (`A:yes/no`) and truthful citation are informational.
|
||||||
|
|
||||||
|
| Level | Positive sonnet | Positive haiku | Negative sonnet | Negative haiku |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| L1 (explicit) | PASS 2/3 | PASS 3/3 | PASS 3/3 (A:yes 3/3) | PASS 3/3 (A:yes 3/3) |
|
||||||
|
| L2 (moderate) | PASS 3/3 | PASS 3/3 | PASS 3/3 (A:yes 2/3) | PASS 3/3 (A:yes 3/3) |
|
||||||
|
| L3 (conceptual) | PASS 3/3 | PASS 3/3 | PASS 3/3 (A:yes 3/3) | PASS 3/3 (A:yes 3/3) |
|
||||||
|
|
||||||
|
**Summary:** 12/12 cells PASS; 35/36 reps PASS (haiku 18/18, sonnet 17/18). No negative rep at any level created an ADR — zero over-triggering on either tier. No positive rep created a duplicate ADR either: every positive pass was via governing-ADR citation (16) or judge-recognized recording proposal (1). Sonnet's single FAIL rep (P1-L1) is a scoring-boundary artifact: it consulted, implemented exactly the ADR-0002 policy, and flagged the un-implemented ADR-0005 escalation — but named only 0005 (not 0002) in its final message, so the strict final-message citation check fell through to the judge. Behaviorally clean; a measurement-boundary flicker analogous to Eval B's W3.
|
||||||
|
|
||||||
|
## Measurement Setup
|
||||||
|
|
||||||
|
**Scenarios Tested**
|
||||||
|
|
||||||
|
Run-set only (job-execution domain, target ADR 0002 with 0001→0002 supersession distractor): P1/N1-L1, P2/N2-L2, P3/N3-L3. Positives sit in already-decided territory — the task restates what an Accepted ADR already decided — so correctness is find → cite → comply. Negatives are non-decisional-but-tempting (test/document/audit framing in ADR-covered domains); correctness is doing the task without fabricating a decision record. The reserve-set (notifications domain) was not touched.
|
||||||
|
|
||||||
|
**Fixture and Sampling**
|
||||||
|
|
||||||
|
New Python `taskq` async job-queue fixture (vs Eval B's Ruby webhook-relay) with a 6-ADR history generated by the plugin's own CLIs; trigger-phrased "Architecture decisions" CLAUDE.md section copied verbatim from eval-b's fixture. 3 reps/cell (36 reps), chosen because 1 rep/cell was demonstrably noisy in both Eval B campaigns. Headless-only (`bin/run`, fresh `claude -p` per rep, cwd = sandbox, real SessionStart hook verified in every transcript). All reps counted — including the pre-fix canary rep, rescored (not re-run) after the checker conformance fix.
|
||||||
|
|
||||||
|
**Experimental Control (Frozen Surfaces)**
|
||||||
|
|
||||||
|
Scenario Task blocks, fixtures, and `judge-rubric.md` untouched throughout. Two harness defects were found on the first live rep and fixed BEFORE the grid (1 rep exposure, transcript preserved and rescored):
|
||||||
|
1. `bin/run` rep-loop bug — `set -euo pipefail` + piping `bin/check` (exit 1 on FAIL) into `tee` aborted a cell's remaining reps on the first FAIL.
|
||||||
|
2. Positive axis-b conformance bug — the checker required ADR creation/proposal, contradicting the README's own design ("consulting and/or recording is correct") and the shipped plugin wording: with the positives in already-decided territory, creating an ADR would duplicate the record — exactly what the negatives punish. Fixed by adding a mechanical cited-governing PASS branch (final message cites the governing ADR id) before the unchanged frozen-judge fallback; self-test gained compliance-must-PASS and wrong-citation-must-FAIL guards.
|
||||||
|
|
||||||
|
Under the pre-fix checker the grid would have scored ~1/18 positives — all false negatives. B-via distribution proves it: cited-governing 16, judge-proposed 1, created 0.
|
||||||
|
|
||||||
|
## Validity and Limitations
|
||||||
|
|
||||||
|
**How to Interpret These Results**
|
||||||
|
|
||||||
|
This is a **held-out measurement** — no wording was tuned against Eval C before or during this grid, so unlike Eval B's final numbers these are not training-set scores. What they support: the trigger-conditioned wording shipped from the Eval B loop generalizes across fixture, language (Ruby→Python), domain, decreasing cue explicitness (L1→L3), and — the core discrimination test — paired negatives, on both tiers. The Eval B overfit concern is substantially retired at these ladder levels. What they do not support: behavior on genuinely novel-decision positives (see weaknesses), other languages/project shapes, or production conditions.
|
||||||
|
|
||||||
|
**Weaknesses of This Eval (Its Ladder Level)**
|
||||||
|
|
||||||
|
The grid saturated — no ladder level differentiated the tiers, so the ceiling wasn't found; harder rungs (heavier distractor clutter, conflicting-ADR tasks requiring supersession, genuinely novel decisions requiring creation) need new scenario authoring. All run-set positives are compliance-shaped (decision already recorded); the eval therefore never measured unprompted *creation* of a genuinely new decision — Eval B's W1–W3 remain the only (training-set) evidence for that. The cited-governing check reads the final message only; sonnet's one FAIL shows it can under-credit compliant behavior (~1-in-18 observed). Single fixture per set. Checker fixes were made mid-first-rep (documented above); scenario metadata lines were edited but Task blocks never exposed.
|
||||||
|
|
||||||
|
## Deployment and Evolution
|
||||||
|
|
||||||
|
**Good-Enough Gate**
|
||||||
|
|
||||||
|
Both tiers clear the discrimination gate: zero over-trigger creation across 18 negative reps means the daily-use worry ("will it spam ADRs?") is not supported at any measured ambiguity level. Combined with Eval B (post-wording) and the W3 stability check, os-adr is cleared for real-project rollout on both sonnet and haiku, keeping the known haiku W3 caveat (~1-in-4/5 recording-offer flicker on supersession). Contamination protocol stands: the run-set is now the baseline; the moment anyone tunes wording against these results, measurement moves to the frozen reserve-set.
|
||||||
|
|
||||||
|
**Hardening Path / Next Measurement**
|
||||||
|
|
||||||
|
Not another lab rung — the saturated grid says the next signal is production: proceed with the locked rollout order (pilot projects via `/os-adr:migrate`, then cc-os retrofit) and stand up the [[eval-methodology-irl-feedback-loop]] session-audit backlog; real misses become the scenarios for any future rung (novel-decision positives, supersession-conflict positives, heavy-clutter negatives — reserve-set discipline applies).
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [[os-adr-eval-b-wording-experiment-hypotheses]] — the wording loop whose generalization this grid tested
|
||||||
|
- [[os-adr-eval-b-grid-results-and-observations]] — original held-out baseline (pre-wording)
|
||||||
|
- [[eval-methodology-ladder]] — ladder design this eval instantiates
|
||||||
|
- [[eval-methodology-irl-feedback-loop]] — where measurement goes after a saturated grid
|
||||||
|
- [[running-autoresearch-skill-evals]] — run-mode and rep-count discipline followed here
|
||||||
|
|
@ -1,5 +1,5 @@
|
||||||
---
|
---
|
||||||
summary: Decision + day-1 pilot plan for the vault-native backlog (task notes + push-first surfacing + Bases kanban spike), including the perspectives-panel verdict and the Fable-orchestrator/cheap-subagent execution plan
|
summary: Backlog decision + pilot plan v2 (2026-07-06) — Planka CE on the OVH prod server as the anywhere-visual kanban, thin Ruby CLI + tick/digest agent layer covering the Planka Pro paywall gaps, vault demoted to knowledge layer; phone drag test is the day-1 gate
|
||||||
tags:
|
tags:
|
||||||
- scope/global
|
- scope/global
|
||||||
- type/decision
|
- type/decision
|
||||||
|
|
@ -10,69 +10,86 @@ status: accepted-pending-pilot
|
||||||
last_reviewed: 2026-07-06
|
last_reviewed: 2026-07-06
|
||||||
---
|
---
|
||||||
|
|
||||||
# Vault-native backlog: decision and pilot plan (2026-07-06)
|
# Backlog system: decision and pilot plan v2 (2026-07-06)
|
||||||
|
|
||||||
Research behind this decision: [[backlog-system-options-research]].
|
Research behind this decision: [[backlog-system-options-research]] (including the same-day second-pass Planka deep-dive and alternatives sweep).
|
||||||
|
|
||||||
|
**Supersedes the v1 vault-native plan (same day).** v1 chose vault task notes + Obsidian Bases kanban. It was overturned within hours by a requirement clarification, recorded here so the reasoning survives:
|
||||||
|
|
||||||
|
## The requirement change that flipped the decision
|
||||||
|
|
||||||
|
The perspectives panel and v1 optimized for "push beats pull." The user corrected this: **the pull surface — a visual board viewable and interactable from anywhere, any device, any time — is the primary requirement**, not a nice-to-have. The founding pain point is "when I want to jump into a project I have no visual interface to see what's on the docket," and with ADHD a desktop-only interface means forgotten/missed tasks. Conversational access ("what's on my plate?" via chat) does not satisfy this; it must be *visual*.
|
||||||
|
|
||||||
|
- Vault-native's known weakness (dashboard maturity, mobile — Bases custom views are desktop-weak) became **disqualifying**.
|
||||||
|
- Planka's known weakness (task data outside markdown) became **"a development problem"** — solvable via REST API, per the user.
|
||||||
|
|
||||||
## Decision
|
## Decision
|
||||||
|
|
||||||
Build the personal cross-project backlog **inside the SecondBrain vault**: one note per backlog item (`task` note type), a push-first surfacing layer, Obsidian Bases kanban views as the visual dashboard, and a minimal deterministic CLI. Forgejo issues remain the code-execution tracker (rule: **Forgejo = code/arch/deployment work inside a repo; vault = everything else**; vault tasks may wikilink/reference Forgejo issues). Fallbacks, in order: Backlog.md (if the Bases board spike fails), Planka + community MCP (if markdown-native fails or a standalone/mobile dashboard becomes a hard requirement).
|
- **Planka v2 Community Edition** (AGPL, plankanban/planka) deployed on the **OVH production server** (`systems-prod-01`, see `~/servers/ovh-prod/`), as a standard `~/services/planka/` docker-compose service behind Traefik with automatic SSL. Auto-discovered by the existing backup script.
|
||||||
|
- **Why OVH, not the home Proxmox box / hermes01:** always-on independent of home ISP/power; already reachable from anywhere with SSL (phone needs a bookmark, not Tailscale); fits the established deploy + backup pattern. Access control (strong auth at minimum; optional Traefik middleware/IP allowlist — client names will appear on cards) is a deploy-time decision.
|
||||||
|
- **No filesystem co-location with the vault is needed.** Planka state is Postgres; all integration (CLI, tick, digest, future Hermes) goes through Planka's REST API + git. Cards reference SecondBrain notes by name/URL; any host that ever needs note *content* clones the vault from Forgejo (one command). The vault does NOT currently sync to OVH and doesn't need to.
|
||||||
|
- **Vault role demoted to knowledge layer:** how-tos, contractor/client notes, conventions, decisions stay in SecondBrain; task *state* lives in Planka. The proven Notion rental pattern (item-with-references) survives split across the two: card = state + links, vault note = knowledge.
|
||||||
|
- **Forgejo issues remain the code-execution tracker** (rule unchanged: Forgejo = code/arch/deployment work inside a repo; Planka = everything else; cards may link issues).
|
||||||
|
- **CLI over MCP.** No official Planka CLI exists; we build a thin Ruby one (Sandi Metz style, `lib/` + `bin/`, mirroring the os-adr pattern) over the REST API. `plankapy` (Python SDK) is the endpoint-reference crib. Community MCP servers (bradrisse/kanban-mcp etc.) are shelf reference only.
|
||||||
|
|
||||||
## Perspectives panel verdict (4 agents: devils-advocate, simplifier, implementer, premortem)
|
### Fallback ladder (if the phone gate fails)
|
||||||
|
|
||||||
The panel kept the vault-native direction but **inverted the build order**. Consensus corrections to the original proposal:
|
1. **TaskView** (taskview.tech) — on-paper ideal (native mobile apps, scoped API tokens, first-party MCP) but source-available, ~6 months old, thin sourcing. *Verify claims hands-on first, then trial.* Modern-and-unproven; only reached if Planka fails the gate.
|
||||||
|
2. **Vikunja** — mature API, kanban view, but mobile app explicitly alpha; revisit 2027.
|
||||||
|
3. Vault-native v1 plan (archived below via git history) / Backlog.md — the markdown lane, if hosted boards fail entirely.
|
||||||
|
|
||||||
1. **The load-bearing layer is push/surfacing, not the data store.** A board that must be opened and a SessionStart hook that fires only when a session starts both fail the ADHD constraint (out of sight = forgotten). The recurring-maintenance scenario (frozen spigots) fails in any week with no Claude sessions. → Push first, board later.
|
**Donetick** (recurring-chore tracker, native mobile) noted as a lane-specific fallback if tick-driven recurrence proves annoying for property maintenance — resisted for now (fifth-backlog trap).
|
||||||
2. **Three reliability requirements or the system is "theater within three months"** (premortem): (a) time-aware idempotent `tick` with a staleness catch-up check, (b) SessionStart text summary of due/P0 items, (c) one-field quick-add capture.
|
|
||||||
3. **Daemon contradiction resolved honestly** (implementer): a `systemd --user` *timer* (one-shot daily, not a daemon) firing `tick` + an ntfy/email push is the minimum genuine fix for no-session weeks. Recurrence math must be elapsed-time catch-up ("instantiate all missed occurrences since last tick"), never one-session-one-tick.
|
|
||||||
4. **YAGNI cuts** (simplifier): no recur rule engine (simple offsets like `recur: +1y`); no full CLI at launch (`tick` + quick-add only; list/move/next deferred to demand); board built only after live task data exists (a Bases saved view is minutes of work once notes exist); AI comms drafting (tenant emails) is week-3; effort field + quick-win filter is week-4.
|
|
||||||
5. **Spike before building** (implementer): verify Base Board / bases-kanban against fixture notes — multi-property filters, ordinal priority sort (use sortable encodings: P0–P3, E1–E3; never low/medium/high strings), drag-drop frontmatter write-back, missing-property rendering, mobile (expect desktop-only). Also spike the Obsidian-open-buffer vs. external-frontmatter-write collision, and verify CLI YAML serialization round-trips byte-identically with Obsidian's `processFrontMatter` (serializer churn pollutes git diffs and widens the conflict surface with the existing `vault_sync.py` SessionEnd push).
|
|
||||||
6. **Task-rot defense**: the quarterly someday-review is itself a recurring P0 task in the system (dogfooding, zero extra code).
|
|
||||||
7. **Fair challenge accepted** (devils-advocate): Backlog.md was dismissed too fast (wikilinks do resolve into its files); it stays the named fallback and the ergonomic template for the CLI. And the morning surface should eventually include top Forgejo issues (via `tea`) so the two trackers get one unified decision point (week-2).
|
|
||||||
|
|
||||||
## Task note schema (v1 — to be formalized via /os-vault:design-template new-type lifecycle)
|
## The Planka Pro paywall and how we route around it
|
||||||
|
|
||||||
```yaml
|
Planka went commercial (2025); CE keeps the full board UI. Three Pro-only features matter here, all covered by composing around the API — **we never modify or fork Planka**:
|
||||||
---
|
|
||||||
summary: <one-line>
|
|
||||||
tags: [scope/global, type/task, <client|project|domain facet>]
|
|
||||||
status: backlog | next | doing | done | someday
|
|
||||||
priority: P0 | P1 | P2 | P3 # sortable encoding, verified in spike
|
|
||||||
due: YYYY-MM-DD # optional
|
|
||||||
recur: +1y | +6m | +3m | +1m # optional, simple offset only — no rule engine
|
|
||||||
---
|
|
||||||
<body: context + wikilinks to how-tos, client notes, property/contractor notes,
|
|
||||||
Forgejo issue URLs — the Notion rental-property pattern>
|
|
||||||
```
|
|
||||||
|
|
||||||
Open spike question: whether `status`/`priority` live as frontmatter fields (Bases-friendly, likely) vs. tags — decide during the spike, record in vault-conventions.md.
|
| Pro-only | Our layer instead |
|
||||||
|
|---|---|
|
||||||
|
| Recurring cards | `tick` script creates cards via REST on schedule — idempotent, elapsed-time catch-up (instantiates all missed occurrences), rules ours and portable |
|
||||||
|
| Cross-board global view | CLI `backlog list --all`, daily digest, and SessionStart brief aggregate across boards via API; in-app stays per-board (acceptable start) |
|
||||||
|
| Notification providers | The digest is ours (script-sent), Planka's notifiers irrelevant |
|
||||||
|
|
||||||
## Pilot plan — Fable orchestrates, cheap models do grunt work (runnable in one day)
|
Known costs: "Pro discovery" banner in the free UI (cosmetic); API-token scoping story unverified (historically username/password; verify at deploy — matters for per-profile Hermes credentials; worst case a tiny API proxy enforces scope, which fits the approval-broker pattern anyway). Project health: v2 GA early 2025, tagged releases slowed since but commits continue — mature/stabilizing with a commercial turn, not dead. 12K+ stars, strong ecosystem.
|
||||||
|
|
||||||
Fable's role throughout: sequencing, design decisions, spike-result judgment, reviewing every subagent deliverable before it lands, updating vault-conventions.md and cc-os docs/ADR at the end.
|
## Notification policy (ADHD / notification-blindness constraint)
|
||||||
|
|
||||||
**Phase 0 — schema + fixtures (parallel, ~30 min)**
|
Three tiers, designed so silence stays meaningful:
|
||||||
- haiku subagent: create `_templates/task.md` + 8–10 fixture task notes in the vault (real content: spigot winterization due 2026-10-15 recur +1y; Woodfin HVAC/electrical/plumbing checkups; 2–3 cc-os someday items — including "adopt Storybloq's handover-doc pattern" as the first captured down-the-road idea; 1–2 client items). Mechanical work from a spec Fable writes.
|
|
||||||
- Fable: write the spec; verify facet/frontmatter correctness against vault-conventions.md.
|
|
||||||
|
|
||||||
**Phase 1 — the two spikes (the riskiest assumptions, before any CLI code)**
|
1. **Ambient (default, zero interruptions):** the board itself + the Claude Code SessionStart brief. Information waits where the user already looks.
|
||||||
- Bases board spike: human-in-the-loop (Obsidian GUI required). Fable/haiku prepares a step-by-step checklist: install Base Board (and bases-kanban as alternate), build a board over `type/task` fixtures, test the six capabilities from panel point 5. Gate: pass → continue; fail → switch dashboard track to Backlog.md, keep everything else.
|
2. **Digest (the only routine notification):** at most ONE morning message, on ONE channel, sent **only when something is due or P0** — silent days stay silent.
|
||||||
- sonnet subagent: concurrent-write spike — script that externally rewrites frontmatter while the file is open/dirty in Obsidian; document observed behavior (reload/prompt/clobber) and the serializer round-trip check.
|
3. **Interrupt:** essentially never in v1; reserved for genuinely urgent items.
|
||||||
|
|
||||||
**Phase 2 — the three must-haves (parallel once Phase 1 passes)**
|
Target trajectory: reminders are the degenerate case where AI can't act yet. Eventually the agent drafts the action (e.g., tenant spigot email) and the notification becomes "draft ready — approve?" — a decision, not a chore.
|
||||||
- sonnet subagent: `tick` — deterministic, idempotent, elapsed-time catch-up (advance done+recurring tasks by their offset, flag overdue); plus `backlog add --quick "title"` (defaults: status=backlog, priority=P2). Ruby, Sandi Metz style, thin bins over a small lib — mirror the os-adr `lib/` + `bin/` pattern in cc-os. Includes model-free tests.
|
|
||||||
- sonnet subagent: SessionStart surfacing hook for the os-vault plugin (Python, matching existing hooks): terse 3–5-line block of due + P0 items, plus the ≥7-days-since-last-tick staleness check that runs `tick` catch-up inline. Timestamp state file in the vault `.state/` (gitignored).
|
## Pilot plan — Fable orchestrates, cheap models do grunt work
|
||||||
- haiku subagent: `systemd --user` timer + service units firing daily `tick`, with ntfy (or mail) push when property-domain tasks come due — the no-session-week safety net.
|
|
||||||
- Fable: review all three (correctness, invariants, naming per [[cc-os-plugin-skill-naming-convention]]), integrate into `cc-os/plugins/os-vault/`, run `bin/refresh-plugins`, verify a fresh session surfaces the fixtures.
|
**Phase 0 — deploy + THE GATE (day 1)**
|
||||||
|
- sonnet subagent: deploy Planka CE per the ovh-prod service pattern (`~/services/planka/`, compose + postgres, Traefik labels, temp domain first, verify backup discovery). Decide auth posture at deploy.
|
||||||
|
- haiku subagent: seed boards per hat (property, clients, cc-os/dev, business dev) + 8–10 real fixture cards via REST (spigot winterization due 2026-10-15; Woodfin HVAC/electrical/plumbing checkups; cc-os someday items incl. "adopt Storybloq handover-doc pattern"; 1–2 client items). Also proves API CRUD end-to-end.
|
||||||
|
- **Human gate — the phone test:** from the phone (responsive web; official app and Planka Pal as alternates), view boards, drag a card between lists, add a card, complete one. Pass → continue. Fail → fallback ladder.
|
||||||
|
|
||||||
|
**Phase 1 — the agent layer (parallel after the gate)**
|
||||||
|
- sonnet subagent: Ruby `backlog` CLI (`lib/` + `bin/`, model-free tests): `add --quick "title"` (defaults board/list/priority), `list --all` (cross-board), `tick` (idempotent elapsed-time recurrence catch-up from a recurrence manifest; simple offsets `+1y|+6m|+3m|+1m` only — no rule engine).
|
||||||
|
- haiku subagent: daily `tick` + digest scheduling on the OVH box (cron or a small container next to Planka), digest per the notification policy (one channel — ntfy or email, decide at build; silent when empty).
|
||||||
|
- sonnet subagent: Claude Code SessionStart brief — terse 3–5 line due/P0 cross-board block via the CLI. Plugin home + naming per [[cc-os-plugin-skill-naming-convention]] (extend os-vault vs. new `os-backlog` — decide at build; run `bin/refresh-plugins` after).
|
||||||
|
- Fable: review all deliverables; verify a fresh session surfaces the fixtures.
|
||||||
|
|
||||||
|
**Phase 2 — Hermes (independent design exercise, guardrails-first — NOT in the pilot's critical path)**
|
||||||
|
Considered separately from the task system; the task system's only obligation is integrate-ability, which REST + (verified) scoped tokens provide by construction. Design principles locked now:
|
||||||
|
- **Capability-based access:** each Hermes profile/hat (property manager, accountant, client manager, business partner) gets its own narrow, revocable credential (e.g., read-only or single-board token). Hermes never holds credentials for catastrophic actions (money movement, deletion, infra).
|
||||||
|
- **Approval broker:** catastrophic-class actions are *requests* Hermes submits to an interface (n8n workflow / API) that holds the real credentials and routes to the user for approve/reject. Guardrails are the interface, not a policy.
|
||||||
|
- Sequence: read-only digest/query first, then narrow acting (comms drafting for `property` items), one hat at a time, property manager first. Gets its own design note when started.
|
||||||
|
|
||||||
**Phase 3 — close-out (Fable)**
|
**Phase 3 — close-out (Fable)**
|
||||||
- Update vault-conventions.md (task type), cc-os CLAUDE.md + build docs, and record the decision as an ADR in cc-os.
|
- cc-os ADR + CLAUDE.md update; retire the v1 vault-task-type idea from vault-conventions planning.
|
||||||
- Live for one week with real tasks. Week-2 review gates (each pre-captured as a task note): noisy summary → filter tighter; board friction → revisit Backlog.md; capture demand → extend CLI (`list/move/next`); add Forgejo top-issues to the morning surface.
|
- One-week live trial. Week-2 review gates (each pre-captured as a card): digest too noisy → tighten; capture friction → extend CLI; Forgejo top-issues added to brief; recurrence annoyance → evaluate Donetick for the property lane.
|
||||||
|
|
||||||
**Explicitly deferred (each captured as a `someday` task, not built):** recur rule engine, effort/quick-win filter, AI comms drafting, mobile dashboard, per-domain board polish, Planka migration.
|
**Explicitly deferred (captured as someday cards, not built):** recurrence rule engine, effort/quick-win filter, AI comms drafting, Hermes acting autonomously, vault→OVH sync, per-board polish, TaskView evaluation (unless gate fails).
|
||||||
|
|
||||||
## Success criteria (from premortem)
|
## Success criteria
|
||||||
|
|
||||||
1. Recurring tasks cannot silently miss a week (timer + staleness check both firing).
|
1. The board is viewable and interactable **from the phone, anywhere** (the gate, then daily reality).
|
||||||
2. Due/P0 items visible every working morning with zero navigation (session surface, later board).
|
2. Recurring tasks cannot silently miss a week (tick catch-up on an always-on host).
|
||||||
3. Capturing a task costs one command or one sentence to Claude.
|
3. Due/P0 items visible every working morning with zero navigation (SessionStart brief + digest-when-warranted).
|
||||||
|
4. Capturing a task costs one command or one sentence to Claude.
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue