Compare commits
No commits in common. "d4f24e61269a6bda618f76a5ccf6b99dc08c7ff6" and "b1f53659713803680f67ba2b281ea6c90d17b428" have entirely different histories.
d4f24e6126
...
b1f5365971
|
|
@ -1,33 +0,0 @@
|
||||||
---
|
|
||||||
summary: Home Assistant (haos01 VM) first-device-setup progress log — Synology DSM, SmartThings, Ring connected 2026-07-03; remaining items listed
|
|
||||||
tags:
|
|
||||||
- scope/project
|
|
||||||
- type/log
|
|
||||||
- project/proxmox-home-assistant
|
|
||||||
- tool/home-assistant
|
|
||||||
---
|
|
||||||
|
|
||||||
# Home Assistant — Integration Setup Progress (2026-07-03)
|
|
||||||
|
|
||||||
HAOS VM **haos01** (VMID 103) on Proxmox `pve`, UI at `http://192.168.86.99:8123`.
|
|
||||||
Project docs: `~/servers/proxmox-home-assistant/` (authoritative checklist: `haos-on-proxmox-setup-checklist.md`).
|
|
||||||
|
|
||||||
## Connected today (2026-07-03)
|
|
||||||
|
|
||||||
- **Synology DSM** — DS218plus (`192.168.86.31`), port **5001**, SSL on / verify off, user `jared`.
|
|
||||||
- Devices: NAS + Drive 1 + Drive 2 (WD40EFZX 4 TB Red Plus, per-disk health/temp sensors) + Volume 1. All assigned to Area "Shed".
|
|
||||||
- Backup **location** configured: shared folder `backup`, path `homeassistant_backup_home` → `/volume1/backup/homeassistant_backup_home`. **Schedule NOT yet created.**
|
|
||||||
- **SmartThings** — auto-added several devices.
|
|
||||||
- **Ring doorbell** — working, live view confirmed in HA (Living Room ▸ Front Door).
|
|
||||||
|
|
||||||
## Remaining / next session
|
|
||||||
|
|
||||||
- [ ] Connect the rest of the auto-discovered devices
|
|
||||||
- [ ] Install HA companion app on iPhone + iPad
|
|
||||||
- [ ] Create the backup schedule in HA (Settings → System → Backups) — todo also tracked in the project checklist Phase 5
|
|
||||||
- [ ] Automation: notify when either DS218plus drive status leaves "normal" (drive-failure alert)
|
|
||||||
|
|
||||||
## Related
|
|
||||||
|
|
||||||
- DSM ports documented in `~/servers/desktop/docs/synology.md`
|
|
||||||
- Proxmox host facts + VM inventory: `~/servers/CLAUDE.md`
|
|
||||||
|
|
@ -1,72 +0,0 @@
|
||||||
---
|
|
||||||
summary: os-adr Eval B (unprompted write-trigger & retrieval) grid results for haiku/sonnet — haiku never self-triggers the ADR system, sonnet passes 5/8 with two near-miss failures — read before designing any follow-up eval on unprompted-consultation behavior.
|
|
||||||
tags:
|
|
||||||
- scope/global
|
|
||||||
- type/reference
|
|
||||||
- project/cc-os
|
|
||||||
- tool/os-adr
|
|
||||||
- domain/llm-evaluation
|
|
||||||
source: cc-os
|
|
||||||
date: 2026-07-03
|
|
||||||
---
|
|
||||||
|
|
||||||
# os-adr Eval B grid results (2026-07-03)
|
|
||||||
|
|
||||||
Eval B measures whether a model notices **on its own** that an Architecture Decision Record
|
|
||||||
system is relevant — without being explicitly told to invoke it. Two axes per scenario: (a)
|
|
||||||
did the model touch the ADR system unprompted (mechanical, from tool_use), (b) was the specific
|
|
||||||
outcome correct (right ADR cited, or a new ADR actually proposed/created). See
|
|
||||||
`cc-os/plugins/os-adr/eval-b/README.md` for full harness design (headless-only `claude -p` runs,
|
|
||||||
so a real SessionStart hook fires — in-session Agent-tool subagents cannot validly stand in for
|
|
||||||
this, since they don't get a fresh hook invocation against the sandbox cwd).
|
|
||||||
|
|
||||||
## Grid (1 rep/cell)
|
|
||||||
|
|
||||||
| Scenario | Haiku | Sonnet |
|
|
||||||
|---|---|---|
|
|
||||||
| W1 (persistence choice) | FAIL | **PASS** |
|
|
||||||
| W2 (convention change) | FAIL | **PASS** |
|
|
||||||
| W3 (reversal) | FAIL | FAIL — doesn't propose recording the decision |
|
|
||||||
| R1 (direct conflict) | FAIL | FAIL — misses the direct-conflict retrieval |
|
|
||||||
| R2 (distractor) | FAIL | **PASS** |
|
|
||||||
| R3 (mid-task question) | FAIL | **PASS** |
|
|
||||||
| R4 (graph one-hop) | FAIL | **PASS** |
|
|
||||||
| R4-nograph (degradation check) | FAIL (expected) | FAIL (expected) |
|
|
||||||
|
|
||||||
**Haiku: 0/8.** Never touches `os-adr:*` / `bin/adr-*` / `docs/adr/` unprompted in any
|
|
||||||
scenario — total absence of self-triggered consultation, not a near-miss.
|
|
||||||
|
|
||||||
**Sonnet: 5/8.** Two real misses: W3 (reversal — doesn't propose recording the decision) and R1
|
|
||||||
(direct-conflict retrieval — doesn't catch it and cite the correct ADR).
|
|
||||||
|
|
||||||
`R4-nograph` failing on both tiers is the **expected** result — it's a degradation check, only
|
|
||||||
meaningful paired with an R4 PASS on the same tier (confirms the Graphify one-hop layer did
|
|
||||||
real work). Sonnet has that pairing (R4 PASS + R4-nograph FAIL); haiku doesn't, since R4 already
|
|
||||||
failed for haiku.
|
|
||||||
|
|
||||||
## Observations
|
|
||||||
|
|
||||||
- **This reads as a prompting/wording issue, not a capability gap.** Sonnet already succeeds on
|
|
||||||
5 of 8 scenarios, and its two failures (W3, R1) are near-misses rather than blind spots —
|
|
||||||
the model is clearly capable of the behavior when the trigger is salient enough. Haiku's
|
|
||||||
total 0/8 plus sonnet's two-scenario gap both point toward the SKILL.md wording and/or the
|
|
||||||
SessionStart hook note text (`plugins/os-adr/hooks/session_start.py`) not being salient
|
|
||||||
enough as an unprompted trigger — not toward a reasoning or tool-use failure.
|
|
||||||
- **Open question, not yet resolved:** whether in-session subagents could ever validly
|
|
||||||
substitute for part of this measurement. Flagged as suspect given the hook-context
|
|
||||||
requirement established for Eval B (in-session subagents inherit the parent session and never
|
|
||||||
get a fresh SessionStart against the sandbox cwd) — worth re-examining specifically for
|
|
||||||
scenarios where the hook note itself isn't the variable under test, but not assumed solved
|
|
||||||
here.
|
|
||||||
- **Next step (not started — future session):** design a follow-up eval that iterates
|
|
||||||
SKILL.md / hook-note wording to raise the unprompted-trigger rate, using the existing
|
|
||||||
`/autoresearch` Classic-loop discipline (checker/fixtures/scenarios/rubric frozen, only
|
|
||||||
wording moves — see `plugins/os-adr/eval-b/README.md`). **That eval should point back at
|
|
||||||
this note** for the baseline numbers and the prompting-issue hypothesis, so the loop has a
|
|
||||||
concrete before/after to compare against.
|
|
||||||
|
|
||||||
## Source
|
|
||||||
|
|
||||||
- cc-os repo, `plugins/os-adr/eval-b/` (harness) and `plugins/os-adr/eval-b/README.md` (status)
|
|
||||||
- Full TSV: `/tmp/adr-eval-b-grid/results.tsv` (ephemeral — not committed, per harness design)
|
|
||||||
- Doc updates committed at cc-os `5b399d5`
|
|
||||||
|
|
@ -394,10 +394,3 @@ tags: [scope/global, type/log]
|
||||||
**Reason:** other
|
**Reason:** other
|
||||||
**Vault notes touched:**
|
**Vault notes touched:**
|
||||||
(none)
|
(none)
|
||||||
|
|
||||||
## Session — 2026-07-03T21:32:58Z
|
|
||||||
|
|
||||||
**Project:** /tmp/adr-eval-b-grid/R2-sonnet-r1
|
|
||||||
**Reason:** other
|
|
||||||
**Vault notes touched:**
|
|
||||||
(none)
|
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue