Fix skill namespacing (drop name: frontmatter), add refresh-plugins, wire eval howto

- Remove explicit name: frontmatter from os-adr and os-doc-hygiene SKILL.mds
  so slash commands register namespaced (/os-adr:find, not bare /find)
- Add bin/refresh-plugins to refresh stale local plugin caches
- CLAUDE.md: cache-refresh procedure, naming-convention pointer, mandate
  reading the vault autoresearch-eval howto before the next eval

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
jared 2026-07-04 14:01:58 -04:00
parent d0c4dcb13e
commit c645faae59
10 changed files with 148 additions and 11 deletions

View File

@ -101,8 +101,8 @@ to those two and fix the stale doc.
- Eval A harness (2026-07-03, ADR-021): `plugins/os-adr/eval/` — prompted skill-execution eval across model tiers (haiku/sonnet). Two fixtures (generated with the plugin's own CLIs), six scenarios (S1 create, S2 create+supersede, S3 find/conflict, S4 find/distractor, S5 init, S6 migrate+fills), deterministic Ruby checker (`eval/bin/check`, structural invariants, TSV mode for autoresearch), sandbox + headless-runner scripts, runner-prompt template. Primary run mode: in-session Agent-tool subagents with pinned `model:` (cheaper than `claude -p`); optimized via the `/autoresearch` Classic loop over SKILL.md *wording only* (checker/fixtures/scenarios frozen during a loop). Procedure: `plugins/os-adr/eval/README.md`. Self-tested both directions; grid run 2026-07-03, all scenarios × both tiers passing. Distinct from the held-out Eval B (unprompted behavior) — do not conflate.
- Eval B harness (2026-07-03, OpenSpec change `add-os-adr-eval-b-harness`): `plugins/os-adr/eval-b/` — held-out unprompted-behavior eval (requirements 45). 7 scenarios (W1W3 write-trigger, R1R4 retrieval) authored from the frozen shapes in `docs/adr-system/06-eval-scenarios.md`; dedicated Ruby webhook-relay fixture with a 6-ADR history (Superseded pair + near-miss distractors, generated via the plugin's own CLIs); R4's one-hop graph reach uses a real `graphify update` AST build (model-free, rebuilt via `eval-b/bin/build-fixture-graph`, never committed). **Headless-only runner** (`eval-b/bin/run` — fresh `claude -p` per rep, cwd = sandbox, so the real SessionStart hook fires; in-session subagents are invalid here, unlike Eval A) and a two-axis deterministic-first checker (`eval-b/bin/check`): axis (a) unprompted consultation, mechanical from transcript tool_use blocks; axis (b) correct-ADR citation (R1R4) or new-ADR-file with a narrow frozen-rubric haiku judge fallback (W1W3, `judge-rubric.md`, stubbable via `ADR_EVAL_B_JUDGE_CMD`). `R4-nograph` is the graph-degradation variant (expected FAIL). Self-tested both directions model-free via `eval-b/bin/self-test`. **Scenario Task blocks are held-out — never run them informally.** Procedure: `plugins/os-adr/eval-b/README.md`. Grid run 2026-07-03 (1 rep/cell): haiku 0/8 PASS (never unprompted-consults the ADR system in any scenario — a real gap, not a harness defect); sonnet 5/8 PASS (fails W3 — doesn't propose recording the decision; fails R1 — misses the direct-conflict retrieval). `R4-nograph` FAILed on both tiers as expected (degradation check, only meaningful paired with an R4 PASS — sonnet has one, haiku doesn't). Full results + observations (prompting-issue hypothesis, open question on whether in-session subagents could ever validly substitute for part of this measurement) written to the vault: [[os-adr-eval-b-grid-results-and-observations]].
- **Initial iteration complete (2026-07-03):** plugin build, migration pilot, Eval A grid (clean pass), and Eval B grid (baseline captured above) are all done — this closes the first pass on os-adr. Follow-on work is deliberately deferred to future sessions, not in-flight.
- **Remaining (locked rollout order):** real-project migration/adoption one at a time via `/os-adr:migrate` — pilot projects first, then the cc-os retrofit (its 19-ADR monolithic file was deliberately excluded from the pilot), then wider; then design and run a follow-up eval iterating Eval B trigger/retrieval wording via `/autoresearch` (checker/fixtures/scenarios/rubric frozen) targeting the W3 and R1 failures on sonnet and the wholesale haiku gap — that eval must reference the vault note above for baseline numbers, and should also resolve the open question of whether in-session subagents can validly stand in for any part of the measurement; re-run the grid with more reps/cell once wording is stable to check variance.
- **TODO (not started, future session):** review and correct plugin naming conventions across the marketplace — os-adr's (and possibly other plugins') slash commands are failing to register/align consistently with sibling plugins (`os-vault`, `os-orchestration`, `os-doc-hygiene`). Likely the same class of stale-marketplace-manifest issue documented under "Renaming or moving a local plugin" below, but needs its own investigation — do not assume the fix is identical without checking `~/.claude/plugins/.claude-plugin/marketplace.json` and `installed_plugins.json` first.
- **Remaining (locked rollout order):** real-project migration/adoption one at a time via `/os-adr:migrate` — pilot projects first, then the cc-os retrofit (its 19-ADR monolithic file was deliberately excluded from the pilot), then wider; then design and run a follow-up eval iterating Eval B trigger/retrieval wording via `/autoresearch` (checker/fixtures/scenarios/rubric frozen) targeting the W3 and R1 failures on sonnet and the wholesale haiku gap. **Before designing or running that (or any) autoresearch eval, Read the vault howto `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md`** — it locks the procedure (valid run mode per eval shape, cache refresh via `bin/refresh-plugins` after every wording edit, reduced parallel inner-loop grid, ~3 reps on target cells, axis-level failure reading, verify from TSV not agent prose). Baseline numbers + the 2026-07-04 confirmation run live in the vault note above (baseline reconfirmed: haiku 0/8, sonnet 5/8 failing W3/R1; W3 is an axis-b failure — consults but doesn't propose recording — so its wording target is the create skill, while R1 is an axis-a trigger failure).
- **Resolved (2026-07-04):** os-adr's skills failed to register on first install due to stale plugin caches — the cache that was installed on 2026-07-03 17:21:48 was missing `hooks/hooks.json`, `bin/adr-detect`, `bin/adr-find`, `bin/adr-migrate`. Root cause: unknown, but the caches were restored via plugin uninstall/reinstall. A parallel issue affected os-doc-hygiene (cache retained deleted `commands/` directory). Fixed by manually refreshing both caches, then created `bin/refresh-plugins` automation to prevent future drift — see "Editing a local plugin (cache refresh)" subsection for refresh procedure. Investigation result: stale caches (not manifest-naming issues) were the cause; slash command registration works correctly once caches are fresh.
**Graphify** — v0.8.31 at `/home/jared/.local/bin/graphify`
- Vault graph: `~/Documents/SecondBrain/graphify-out/` — disposable, structural index rebuilt by SessionStart hook; handles relational/graph queries. Distinct from vault-index.json (planned, Plan B Phase 1): a flat `{tag: [{path, title, summary}]}` lookup rebuilt at SessionEnd, queried by the `/memory-find` skill for fast tag-based SB note discovery.
@ -155,6 +155,19 @@ Skipping step 2 is the failure mode to watch for: hooks (wired by absolute path
`settings.json`) keep working, masking the fact that skills/slash-commands silently stopped
registering under the plugin's new name.
### Editing a local plugin (cache refresh)
Plugin source edits (including SKILL.md wording, hook scripts, or CLI code) do not reach running
sessions until the cache is refreshed. Claude Code caches plugins when installed, copying them
to `~/.claude/plugins/cache/<marketplace>/<plugin>/<version>/`. To refresh a stale cache after
source edits:
```bash
bin/refresh-plugins [marketplace-names...] # refresh all or specific local marketplaces
```
This uninstalls and reinstalls each plugin, forcing the cache to repopulate. See `bin/refresh-plugins --help` for usage. After refreshing, `claude plugin details <name>@<marketplace>` should show updated skill descriptions and hook counts. Directory-source marketplaces (e.g., `local-plugins` sourced from `~/.claude/plugins`, `cc-plugins` sourced from `~/dev/cc-plugins`) are the only ones that need manual refresh — public marketplaces auto-update.
## Issue tracking
Issues (created via `/to-issues`) live on self-hosted Forgejo (`jared/cc-os`), queried with the `tea` CLI — not GitHub/`gh`. See `docs/issue-workflow.md` for token setup, listing, implementing, and closing issues.
@ -183,6 +196,15 @@ project context for OpenSpec artifacts comes from `docs/` and this file.
`03-architecture-decisions.md` with the reasoning.
- The package on PyPI is `graphifyy` (double-y) but the command is `graphify`.
- **Keep this file current:** When a build step completes, (a) mark it done in `docs/memory-system/04-build-plan.md`, (b) add or update a component pointer in the Implemented Components section above, and (c) update the current design paragraph if the design changed. This file is the AI's orientation entry point — accuracy matters more than brevity.
- **Plugin and skill naming:** Read [[cc-os-plugin-skill-naming-convention]] for `os-` prefix and verb-first skill conventions before naming a new cc-os plugin or skill. Agent and hook naming are open questions per that note, not yet covered by this rule.
- **Plugin and skill naming:** Before naming ANY new cc-os plugin, skill, or slash command, Read
`~/Documents/SecondBrain/cc-os-plugin-skill-naming-convention.md` and follow it — do NOT invent
a new convention. Rules in brief: plugins are `os-[domain]`; skills are verb-first kebab-case
(`query`, `onboard-project`), invoked as `/os-[domain]:[verb]`; no `commands/` dispatcher
directories; **never set a `name:` field in SKILL.md frontmatter** — the directory name is the
skill name, and an explicit `name:` collapses the slash command to a bare unnamespaced form
(`/find` instead of `/os-adr:find`; found and fixed 2026-07-04). After editing any SKILL.md,
run `bin/refresh-plugins` (installs cache plugin files; source edits don't reach sessions until
refreshed). Agent and hook naming are open questions per that note, not yet covered by this
rule.
**Session orchestration behavior** — Provided by the global `os-orchestration` plugin (see Implemented Components below). This repo no longer carries a local orchestration override; it follows the plugin's default behavior like every other project.

123
bin/refresh-plugins Executable file
View File

@ -0,0 +1,123 @@
#!/usr/bin/env ruby
# frozen_string_literal: true
# Refresh Claude Code plugin caches for directory-source marketplaces.
# Discovers installed plugins from each local marketplace and refreshes their caches.
#
# Usage:
# bin/refresh-plugins [marketplace-names...]
# bin/refresh-plugins # refresh all local marketplaces
# bin/refresh-plugins local-plugins # refresh only local-plugins
# bin/refresh-plugins cc-plugins local-plugins
require 'json'
require 'pathname'
class PluginRefresher
attr_reader :results
def initialize
@results = []
end
def refresh(marketplace_names = nil)
marketplaces = load_marketplaces(marketplace_names)
marketplaces.each do |name, config|
next unless config['source']['source'] == 'directory'
puts "Refreshing marketplace: #{name}"
refresh_marketplace(name, config)
puts
end
report_results
end
private
def load_marketplaces(names)
all = JSON.parse(File.read(File.expand_path('~/.claude/plugins/known_marketplaces.json')))
return all unless names
all.select { |k, _| names.include?(k) }
end
def refresh_marketplace(marketplace_name, config)
source_path = config['source']['path']
installed = load_installed_plugins(marketplace_name)
installed.each do |plugin_name|
refresh_plugin(marketplace_name, plugin_name, source_path)
end
end
def load_installed_plugins(marketplace_name)
all = JSON.parse(File.read(File.expand_path('~/.claude/plugins/installed_plugins.json')))
all['plugins']
.select { |k, _| k.end_with?("@#{marketplace_name}") }
.keys
.map { |k| k.sub(/@#{marketplace_name}$/, '') }
.uniq
end
def refresh_plugin(marketplace_name, plugin_name, source_path)
cache_exists = cache_exists?(marketplace_name, plugin_name)
unless File.exist?(File.join(source_path, plugin_name))
@results << [plugin_name, "source not found at #{source_path}", :error]
puts " #{plugin_name}: ERROR — source not found"
return
end
system("claude plugin uninstall #{plugin_name}@#{marketplace_name} > /dev/null 2>&1")
system("claude plugin install #{plugin_name}@#{marketplace_name} > /dev/null 2>&1")
if verify_cache(marketplace_name, plugin_name, source_path)
@results << [plugin_name, "refreshed", :success]
puts " #{plugin_name}: REFRESHED"
elsif !cache_exists
@results << [plugin_name, "in-sync (not previously cached)", :info]
puts " #{plugin_name}: IN SYNC (new)"
else
@results << [plugin_name, "STALE", :warning]
puts " #{plugin_name}: STALE (differences remain)"
end
end
def cache_exists?(marketplace_name, plugin_name)
Dir.exist?(File.expand_path("~/.claude/plugins/cache/#{marketplace_name}/#{plugin_name}"))
end
def verify_cache(marketplace_name, plugin_name, source_path)
cache_base = File.expand_path("~/.claude/plugins/cache/#{marketplace_name}/#{plugin_name}")
source_full = File.join(source_path, plugin_name)
# Find the version directory in cache
versions = Dir.glob(File.join(cache_base, '*')).select { |p| File.directory?(p) }
return false if versions.empty?
cache_path = versions.first
# Run diff check (exclude plugin manager state files)
output = `diff -rq "#{cache_path}" "#{source_full}" --exclude='.git' --exclude='__pycache__' --exclude='.orphaned_at' 2>&1`
output.strip.empty?
end
def report_results
puts "\n=== Summary ==="
successes = @results.select { |_, _, status| status == :success }
warnings = @results.select { |_, _, status| status == :warning }
errors = @results.select { |_, _, status| status == :error }
infos = @results.select { |_, _, status| status == :info }
puts "Refreshed: #{successes.map { |r| r[0] }.join(', ')}" if successes.any?
puts "In sync: #{infos.map { |r| r[0] }.join(', ')}" if infos.any?
puts "Stale (not fixed): #{warnings.map { |r| r[0] }.join(', ')}" if warnings.any?
puts "Errors: #{errors.map { |r| "#{r[0]} (#{r[1]})" }.join(', ')}" if errors.any?
puts "All plugins synchronized." if @results.all? { |_, _, s| s != :warning && s != :error }
end
end
marketplace_names = ARGV.empty? ? nil : ARGV
PluginRefresher.new.refresh(marketplace_names)

View File

@ -1,5 +1,4 @@
---
name: create
description: Record an architecture decision as a correctly templated, numbered, and indexed ADR in one invocation. Use when a decision of real consequence (irreversible, cross-cutting, contested, or convention-setting) has just been made or is being made — including unprompted, when you recognize such a decision mid-task. Invoked by `/os-adr:create`.
---

View File

@ -1,5 +1,4 @@
---
name: find
description: Retrieve the architecture decision records relevant to files, components, or a topic being worked on — callable at any point mid-task, not just at session start. Use unprompted before changing behavior that may already be decided (a locked choice, a convention, a rejected alternative). Invoked by `/os-adr:find`.
---

View File

@ -1,5 +1,4 @@
---
name: init
description: Set up the ADR system (docs/adr/ + generated index) in a project that has none, and clear any session-reminder suppression. Invoked by `/os-adr:init`.
---

View File

@ -1,5 +1,4 @@
---
name: migrate
description: Non-destructively convert a project's existing ADR-like content (numbered files, dated files, monolithic decision logs, prose-embedded decisions) into the standard docs/adr/ format. Sources are never touched; uncertainty is flagged via migration_confidence + a migration report. Invoked by `/os-adr:migrate`.
---

View File

@ -1,5 +1,4 @@
---
name: check
description: Scan the project for stale and bloated documentation and write a hygiene report. Runs the deterministic scanner, dispatches a Sonnet subagent to classify only the signal-bearing candidates, finalizes/validates the machine report deterministically, then writes the report pair and stamps `last_check`. Invoked by `/os-doc-hygiene:check [--scope <glob-or-path>] [--category <class|subtype>]`.
---

View File

@ -1,5 +1,4 @@
---
name: clean
description: Apply documented hygiene findings to project docs. Loads the current machine report, gates confirm-tier entries, applies deterministic ops via the patch applier, dispatches generative ops to a Sonnet subagent, stages precisely, and produces exactly one git commit. Invoked by `/os-doc-hygiene:clean [--scope <glob-or-path>] [--category <class|subtype>]`.
---

View File

@ -1,5 +1,4 @@
---
name: status
description: Show the doc-hygiene lifecycle state for the current project — last check/clean/reminded timestamps and whether a report exists. Read-only, no model, no scan. Invoked by `/os-doc-hygiene:status`.
---

View File

@ -1,5 +1,4 @@
---
name: sweep
description: Run doc-hygiene check then clean in sequence — a convenience wrapper. The confirm gate in `clean` still applies; sweep never auto-approves confirm-tier or generative entries. Invoked by `/os-doc-hygiene:sweep [--scope <glob-or-path>] [--category <class|subtype>]`.
---