jared
/
cc-os
1
0
Fork 0
Code Issues 17 Pull Requests Packages Projects Releases Wiki Activity

Pick calibration project #1 and success criteria #45

New Issue
Closed
opened 2026-07-14 16:52:06 +00:00 by jared · 3 comments
jared commented 2026-07-14 16:52:06 +00:00
Owner
Copy Link

Map: #31

Question

Choose the first project for the full categorize-every-file calibration pass, and define what 'validated' means (e.g. audit catches N% of human-spot-checked clutter; predefined retest cap).

Map: #31 ## Question Choose the first project for the full categorize-every-file calibration pass, and define what 'validated' means (e.g. audit catches N% of human-spot-checked clutter; predefined retest cap).
jared added the
wayfinder:grilling
 label 2026-07-14 16:52:06 +00:00
jared self-assigned this 2026-07-14 18:19:53 +00:00
jared commented 2026-07-14 19:02:26 +00:00
Author
Owner
Copy Link

Resolution

Calibration project #1 = cc-os.

Chosen to prove the protocol works before testing whether it generalizes. The
self-referential risk is accepted knowingly: cc-os is the repo the plugin lives in, so rules
discovered here may flatter the rulebook author's priors. That risk is paid for by making the
validation criteria — not the rule list — carry the weight.

What "validated" means

1. Precision is a HARD GATE (not a score).

The pass FAILS if any rule persisted to the rulebook has a glob matching a protected
path — regardless of its behavior tier (auto-delete or confirm-gated).

Stricter than "don't auto-delete protected files": even a confirm-tier rule whose glob reaches
openspec/specs/ fails the pass. Rationale: a confirm gate is a safety property of the human
sitting there, not of the protocol. A gate you see fifty times is a gate you click through — and
this map's fog (the recurring categorize-and-learn skill) requires a rulebook trustworthy enough
to run unattended.

Exploration-time consult verdicts on protected paths are FREE and count as the protocol
working correctly. The gate is on persisted rules, not on protocol contact. Governing principle
(user's words): "If there is any reasonable doubt it should not be deleted. Even if that means
keeping an unnecessary file." consult IS that principle; penalising it would incentivise the
judge to guess instead of ask.

2. The protected set — derived from pre-existing declarations, then human-edited.

Sourced from what cc-os already declares protected (these predate this map, which is where the
independence comes from):

  • plugins/*/eval/scenarios/, .../scenarios-reserve/, .../fixture/, judge-rubric.md
    (CLAUDE.md eval discipline: "never run informally", "reserves never even read")
  • openspec/specs/ (durable spec store — ADR-033)
  • docs/adr/** (decision record)
  • mirrored .claude/ / .codex/ / .pi/ skill dirs (CLAUDE.md: must stay in sync)
  • CLAUDE.md, plugin source

@jared edited this list before the run — recorded here explicitly, because a gate authored
solely by the thing it gates is theatre. The list is FIXED before the pass and is never revised
after seeing what the protocol proposes.

3. Recall floor: >=80% of the SEALED cc-os answer key.

The cc-os rows of the #41 inventory are the answer key. They are HELD OUT of judge intake for
this pass only — a scoped, deliberate one-off deviation from #42 (which otherwise feeds all
seeds at judge intake). The judge receives only the delta-refinery / hyperthrive_dev / llf-schema
seeds; it must find cc-os's clutter on its own.

Without this hold-out the pass is an open-book exam where the book is the answer key: the protocol
would "discover" docs/adr/migration-report.md because we handed it a note saying that file is
clutter, score ~100%, and prove nothing. Every calibration run AFTER this one uses full seed intake
as #42 designed.

Answer key = the cc-os rows of #41 MINUS graphify-out/. Per #43, graphify-out/** and
.dochygiene/** are an IGNORE surface — never walked, never a rule's business; graphify owns
graphify's artifacts. That seed row is void, not a miss: failing to flag it is correct
behaviour. Answer key is therefore ~10 patterns, incl. autoresearch/<run-id>/,
plugins/os-doc-hygiene/HANDOFF-add-deterministic-core.md, docs/adr/migration-report.md,
plugins/*/eval/results/, docs/plans/*.md, docs/orchestration-audit/auditor-reports/S*-report.md.

4. Novel matches are expected, and are spot-checked — not hard-gated.

The seed list was a sample, so the pass will surface clutter beyond the answer key. Every novel
match is human-spot-checked. A wrong novel match does not fail the pass (only the protected-set
gate does that) — it triggers rule adjustment and a retest round.

5. Retest cap: already settled in #42. Stop at <2 new rules OR <10% unmatched shrink; hard cap
3 rounds. #45 adds no new cap; the clause in this ticket's body is answered by #42.

A do-nothing pass cannot pass

Precision-gate + #42's stop conditions are all satisfied by proposing zero rules. The recall floor
(3) is what makes the pass falsifiable in the finding direction.

Additional requirement handed to #46 (spec assembly)

:calibrate must present proposed rules to the human as a plain-language, example-grounded
report — concrete paths, what the thing is, why it is clutter (e.g. "autoresearch/classic-260703-1522/
— a concluded research loop's working directory, last touched 11 days ago") — before any rule is
persisted. The human reviews examples, not JSON schema. Direct user requirement, arising from
this session: reading the seeds by example was immediately legible where the schema-level discussion
was not.

Surfaced, NOT decided here (spun out)

#43 measures temporary-tier age on git-commit age. autoresearch/<run-id>/ is untracked/
gitignored — it has no commit age — so a commit-age-keyed rule can never fire on the single clearest
piece of clutter in the answer key. That is a hole in #43, not a preference difference. Spun out as
its own ticket; #46 blocks on it.

## Resolution **Calibration project #1 = `cc-os`.** Chosen to prove the protocol *works* before testing whether it *generalizes*. The self-referential risk is accepted knowingly: cc-os is the repo the plugin lives in, so rules discovered here may flatter the rulebook author's priors. That risk is paid for by making the validation criteria — not the rule list — carry the weight. ### What "validated" means **1. Precision is a HARD GATE (not a score).** > The pass FAILS if any rule **persisted to the rulebook** has a glob matching a **protected > path** — regardless of its behavior tier (auto-delete *or* confirm-gated). Stricter than "don't auto-delete protected files": even a confirm-tier rule whose glob reaches `openspec/specs/` fails the pass. Rationale: a confirm gate is a safety property of *the human sitting there*, not of the protocol. A gate you see fifty times is a gate you click through — and this map's fog (the recurring categorize-and-learn skill) requires a rulebook trustworthy enough to run unattended. **Exploration-time `consult` verdicts on protected paths are FREE** and count as the protocol working correctly. The gate is on *persisted rules*, not on protocol contact. Governing principle (user's words): *"If there is any reasonable doubt it should not be deleted. Even if that means keeping an unnecessary file."* `consult` IS that principle; penalising it would incentivise the judge to guess instead of ask. **2. The protected set — derived from pre-existing declarations, then human-edited.** Sourced from what cc-os *already declares* protected (these predate this map, which is where the independence comes from): - `plugins/*/eval/scenarios/`, `.../scenarios-reserve/`, `.../fixture/`, `judge-rubric.md` (CLAUDE.md eval discipline: "never run informally", "reserves never even read") - `openspec/specs/` (durable spec store — ADR-033) - `docs/adr/**` (decision record) - mirrored `.claude/` / `.codex/` / `.pi/` skill dirs (CLAUDE.md: must stay in sync) - `CLAUDE.md`, plugin source **@jared edited this list before the run** — recorded here explicitly, because a gate authored solely by the thing it gates is theatre. The list is FIXED before the pass and is never revised after seeing what the protocol proposes. **3. Recall floor: >=80% of the SEALED cc-os answer key.** The cc-os rows of the #41 inventory are the answer key. **They are HELD OUT of judge intake for this pass only** — a scoped, deliberate one-off deviation from #42 (which otherwise feeds all seeds at judge intake). The judge receives only the delta-refinery / hyperthrive_dev / llf-schema seeds; it must find cc-os's clutter on its own. Without this hold-out the pass is an open-book exam where the book is the answer key: the protocol would "discover" `docs/adr/migration-report.md` because we handed it a note saying that file is clutter, score ~100%, and prove nothing. Every calibration run AFTER this one uses full seed intake as #42 designed. **Answer key = the cc-os rows of #41 MINUS `graphify-out/`.** Per #43, `graphify-out/**` and `.dochygiene/**` are an IGNORE surface — never walked, never a rule's business; graphify owns graphify's artifacts. That seed row is **void, not a miss**: failing to flag it is correct behaviour. Answer key is therefore ~10 patterns, incl. `autoresearch/<run-id>/`, `plugins/os-doc-hygiene/HANDOFF-add-deterministic-core.md`, `docs/adr/migration-report.md`, `plugins/*/eval/results/`, `docs/plans/*.md`, `docs/orchestration-audit/auditor-reports/S*-report.md`. **4. Novel matches are expected, and are spot-checked — not hard-gated.** The seed list was a *sample*, so the pass will surface clutter beyond the answer key. Every novel match is human-spot-checked. A wrong novel match does **not** fail the pass (only the protected-set gate does that) — it triggers rule adjustment and a retest round. **5. Retest cap: already settled in #42.** Stop at <2 new rules OR <10% unmatched shrink; hard cap 3 rounds. #45 adds no new cap; the clause in this ticket's body is answered by #42. ### A do-nothing pass cannot pass Precision-gate + #42's stop conditions are *all* satisfied by proposing zero rules. The recall floor (3) is what makes the pass falsifiable in the finding direction. ### Additional requirement handed to #46 (spec assembly) `:calibrate` must present proposed rules to the human as a **plain-language, example-grounded report** — concrete paths, what the thing is, why it is clutter (e.g. "`autoresearch/classic-260703-1522/` — a concluded research loop's working directory, last touched 11 days ago") — **before any rule is persisted**. The human reviews *examples*, not JSON schema. Direct user requirement, arising from this session: reading the seeds by example was immediately legible where the schema-level discussion was not. ### Surfaced, NOT decided here (spun out) #43 measures temporary-tier age on **git-commit age**. `autoresearch/<run-id>/` is untracked/ gitignored — it has no commit age — so a commit-age-keyed rule can never fire on the single clearest piece of clutter in the answer key. That is a hole in #43, not a preference difference. Spun out as its own ticket; #46 blocks on it.
jared closed this issue 2026-07-14 19:02:39 +00:00
jared commented 2026-07-14 19:04:35 +00:00
Author
Owner
Copy Link

Addendum — the rule-report requirement, sharpened

The report requirement recorded above ("plain-language, example-grounded") is not sufficient on
its own, per @jared: examples prove the judge found the right file, but say nothing about
whether the rule it wrote generalizes.

Concretely: a judge that nominates the glob autoresearch/classic-260703-1522/ and one that
nominates autoresearch/*/ produce identical example output on today's repo — both match that
directory, both look correct. The first is worthless: it dies the moment the next run is
classic-260814-1546/. A report showing only examples cannot tell them apart.

So the report must show, per proposed rule:

  1. The proposed glob verbatim — exactly as it would be persisted to the rulebook.
  2. Every path it currently matches (or a capped sample + total count).
  3. The boundary — near-miss paths it does NOT match. This is what makes over-broad and
    over-narrow rules visible: a rule over plugins/*/eval/results/ should be shown not matching
    plugins/*/eval/scenarios-reserve/.
  4. Lifetime + behavior tier (keep / temporary / delete-once-served; auto vs confirm).
  5. Plain-language why — what the artifact is and why it's clutter.

Over-specificity is a smell the report must flag

A glob containing a run-id, timestamp, or hash literal — anything that can match exactly one path
and can never match a future sibling — is a failed generalization, not a valid rule. The
nomination step (#42: "haiku nominates bare glob+lifetime per cluster") must be constrained to
produce patterns, and the judge must reject exact-instance globs. Flag them loudly in the report
rather than silently persisting them.

This is the point of clustering in #42 — a cluster of sibling run directories exists precisely so
the rule can be written over the cluster, not over one member of it.

Handed to #46.

## Addendum — the rule-report requirement, sharpened The report requirement recorded above ("plain-language, example-grounded") is **not sufficient on its own**, per @jared: examples prove the judge found the right *file*, but say nothing about whether the *rule* it wrote generalizes. Concretely: a judge that nominates the glob `autoresearch/classic-260703-1522/` and one that nominates `autoresearch/*/` produce **identical example output** on today's repo — both match that directory, both look correct. The first is worthless: it dies the moment the next run is `classic-260814-1546/`. A report showing only examples cannot tell them apart. ### So the report must show, per proposed rule: 1. **The proposed glob verbatim** — exactly as it would be persisted to the rulebook. 2. **Every path it currently matches** (or a capped sample + total count). 3. **The boundary — near-miss paths it does NOT match.** This is what makes over-broad and over-narrow rules visible: a rule over `plugins/*/eval/results/` should be shown *not* matching `plugins/*/eval/scenarios-reserve/`. 4. **Lifetime + behavior tier** (keep / temporary / delete-once-served; auto vs confirm). 5. **Plain-language why** — what the artifact is and why it's clutter. ### Over-specificity is a smell the report must flag A glob containing a run-id, timestamp, or hash literal — anything that can match exactly one path and can never match a future sibling — is a **failed generalization**, not a valid rule. The nomination step (#42: "haiku nominates bare glob+lifetime per cluster") must be constrained to produce patterns, and the judge must reject exact-instance globs. Flag them loudly in the report rather than silently persisting them. This is the *point* of clustering in #42 — a cluster of sibling run directories exists precisely so the rule can be written over the cluster, not over one member of it. Handed to #46.
jared commented 2026-07-14 19:30:04 +00:00
Author
Owner
Copy Link

Addendum 2 — recall floor RATIFIED, answer key settled, glob-breadth tie-breaker

The first Resolution comment recorded >=80% and an ambiguous answer-key membership as if settled.
They were not — @jared had endorsed the mechanism (ground-truth recall, seeded from #41,
sealed answer key) but never the number or the set. Both are now explicitly ratified below; this
addendum is authoritative where it differs from the Resolution above.

Answer key = ALL TEN cc-os rows of #41

@jared: "I would be fine with clearing-out everything on this list — high and medium. Including the
openspec changes archive items." This overrides #41's own proposed lifetimes for
openspec/changes/archive/ and PRD.md (which #41 guessed were keeps).

graphify-out/ remains void — not because it must be kept forever, but because #43 already made
it an IGNORE surface the scanner never walks. Reopening that is its own decision, not something to
smuggle in via an answer key.

Recall floor = 8 of 10, with 4 MANDATORY (ratified)

Mandatory (a miss on any of these FAILS the pass — a blind protocol):

  • autoresearch/<run-id>/
  • HANDOFF-*.md
  • docs/adr/migration-report.md
  • .dochygiene/report.{json,md}

Plus at least 4 of the remaining 6: docs/plans/*.md, openspec/changes/<id>/,
openspec/changes/archive/<id>/, docs/orchestration-audit/auditor-reports/S*-report.md,
plugins/*/eval/results/, PRD.md.

Rationale: the four high-confidence rows are unambiguous — missing them means the protocol cannot
see. The six medium rows are where honest disagreement lives, so demanding all of them grades taste,
not sight. @jared: "A miss here or there is fine... Maintenance doesn't have to be absolutely
perfect. It just keeps the project from growing out of control."

Note for the record: the recall floor is a GRADING BAR, not a runtime behavior. The
err-toward-keeping property of the system comes from the hard gate (no persisted rule may reach a
protected path) and consult (doubt -> ask, never delete) — not from this number.

Over-specificity, corrected: the rule is the CLASS, never the PATH

The Addendum-1 wording ("a glob matching exactly one path is a failed generalization") was too
strong and would have wrongly condemned a legitimate rule. Corrected test:

  • A name that recurs by convention (PRD.md, HANDOFF-*.md, migration-report.md) — legitimate
    to hardcode. Future instances share the name. Matching one file today is fine.
  • An identifier unique to one instance (classic-260703-1522, a hash, a bare timestamp) — never
    legitimate. No future path can ever equal it; the rule can only fire once.

A rule matching exactly one file today is not a smell. A rule that can only ever match one file
is.

Worked example: the near-miss requirement caught a real bug, in this session

@jared proposed tightening autoresearch/*/ to autoresearch/classic-*/. Actual repo contents:

autoresearch/classic-260703-1522/
autoresearch/classic-260707-1253/
autoresearch/classic-260708-1039/
autoresearch/improve-260710-1057/     <- classic-* MISSES this

Both globs produce identical output on the three classic- dirs. Only the non-matching path
exposed the gap. This is exactly the "show the boundary / near-misses" requirement added to #46 —
validating it on a real rule, the same session it was invented. Keep it in the spec.

Tie-breaker: when two globs are both defensible, take the NARROWER one

The failure modes are asymmetric:

  • Too narrow -> fails SAFE (leaves clutter). The recurring calibration pass catches the remainder
    next round. Self-healing.
  • Too broad -> fails DANGEROUS (deletes a keeper). Not self-healing.

So under-matching is the acceptable error, and readability counts: autoresearch/[classic|improve]-*
is preferable to a clever run-id-shape regex if the only cost is one missed round.

PRD.md is purpose-triggered, NOT age-triggered

Rejected @jared's initial framing of PRD as temporary. temporary is age-keyed (#43: retain 3 /
3 days) — which would delete the PRD of a feature not yet built. A PRD dies when its feature
ships, whether that took a week or a year. So: delete-once-served with a classifier-judged
served_when -> forced confirm per #43. It cannot be an auto rule. (autoresearch/*/ can be
auto — a concluded run is provable from the filesystem; "did this ship?" is not.)

## Addendum 2 — recall floor RATIFIED, answer key settled, glob-breadth tie-breaker The first Resolution comment recorded `>=80%` and an ambiguous answer-key membership as if settled. They were **not** — @jared had endorsed the *mechanism* (ground-truth recall, seeded from #41, sealed answer key) but never the number or the set. Both are now explicitly ratified below; this addendum is authoritative where it differs from the Resolution above. ### Answer key = ALL TEN cc-os rows of #41 @jared: *"I would be fine with clearing-out everything on this list — high and medium. Including the openspec changes archive items."* This overrides #41's own proposed lifetimes for `openspec/changes/archive/` and `PRD.md` (which #41 guessed were keeps). `graphify-out/` remains **void** — not because it must be kept forever, but because #43 already made it an IGNORE surface the scanner never walks. Reopening that is its own decision, not something to smuggle in via an answer key. ### Recall floor = 8 of 10, with 4 MANDATORY (ratified) **Mandatory (a miss on any of these FAILS the pass — a blind protocol):** - `autoresearch/<run-id>/` - `HANDOFF-*.md` - `docs/adr/migration-report.md` - `.dochygiene/report.{json,md}` **Plus at least 4 of the remaining 6:** `docs/plans/*.md`, `openspec/changes/<id>/`, `openspec/changes/archive/<id>/`, `docs/orchestration-audit/auditor-reports/S*-report.md`, `plugins/*/eval/results/`, `PRD.md`. Rationale: the four high-confidence rows are unambiguous — missing them means the protocol cannot see. The six medium rows are where honest disagreement lives, so demanding all of them grades taste, not sight. @jared: *"A miss here or there is fine... Maintenance doesn't have to be absolutely perfect. It just keeps the project from growing out of control."* **Note for the record:** the recall floor is a GRADING BAR, not a runtime behavior. The err-toward-keeping property of the system comes from the hard gate (no persisted rule may reach a protected path) and `consult` (doubt -> ask, never delete) — not from this number. ### Over-specificity, corrected: the rule is the CLASS, never the PATH The Addendum-1 wording ("a glob matching exactly one path is a failed generalization") was **too strong** and would have wrongly condemned a legitimate rule. Corrected test: - **A name that recurs by convention** (`PRD.md`, `HANDOFF-*.md`, `migration-report.md`) — legitimate to hardcode. Future instances share the name. Matching one file *today* is fine. - **An identifier unique to one instance** (`classic-260703-1522`, a hash, a bare timestamp) — never legitimate. No future path can ever equal it; the rule can only fire once. A rule matching exactly one file today is not a smell. A rule that can only **ever** match one file is. ### Worked example: the near-miss requirement caught a real bug, in this session @jared proposed tightening `autoresearch/*/` to `autoresearch/classic-*/`. Actual repo contents: ``` autoresearch/classic-260703-1522/ autoresearch/classic-260707-1253/ autoresearch/classic-260708-1039/ autoresearch/improve-260710-1057/ <- classic-* MISSES this ``` Both globs produce identical output on the three `classic-` dirs. **Only the non-matching path exposed the gap.** This is exactly the "show the boundary / near-misses" requirement added to #46 — validating it on a real rule, the same session it was invented. Keep it in the spec. ### Tie-breaker: when two globs are both defensible, take the NARROWER one The failure modes are asymmetric: - **Too narrow** -> fails SAFE (leaves clutter). The recurring calibration pass catches the remainder next round. Self-healing. - **Too broad** -> fails DANGEROUS (deletes a keeper). Not self-healing. So under-matching is the acceptable error, and readability counts: `autoresearch/[classic|improve]-*` is preferable to a clever run-id-shape regex if the only cost is one missed round. ### PRD.md is purpose-triggered, NOT age-triggered Rejected @jared's initial framing of PRD as `temporary`. `temporary` is age-keyed (#43: retain 3 / 3 days) — which would delete the PRD of a feature **not yet built**. A PRD dies when its feature **ships**, whether that took a week or a year. So: `delete-once-served` with a classifier-judged `served_when` -> **forced confirm** per #43. It cannot be an auto rule. (`autoresearch/*/` can be auto — a concluded run is provable from the filesystem; "did this ship?" is not.)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
research/clutter-inventory
Labels
Clear labels   
P0

Priority 0 — drop everything

  
P1

Priority 1 — next up

  
P2

Priority 2 — normal

  
P3

Priority 3 — someday

  
afk-ready

Autonomous end-to-end; closed after verification

  
hitl

Human-owned; never picked up autonomously

  
next

Human-curated next up — AI never applies or removes

  
ready-for-agent

Triage complete; an agent can pick this up

  
review

Shipped semi work awaiting human sign-off

  
semi

Autonomous through mechanical parts; stops at gates; ships to review

  
waiting

Blocked; blocker named in a comment

  
wayfinder:grilling

   
wayfinder:map

   
wayfinder:research

   
wayfinder:task
No Label
P0
P1
P2
P3
afk-ready
hitl
next
ready-for-agent
review
semi
waiting
wayfinder:grilling
wayfinder:map
wayfinder:research
wayfinder:task
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
jared
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: jared/cc-os#45
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?