From c645faae59a8bcda987ff933fc4f95225a18796e Mon Sep 17 00:00:00 2001 From: jared Date: Sat, 4 Jul 2026 14:01:58 -0400 Subject: [PATCH] Fix skill namespacing (drop name: frontmatter), add refresh-plugins, wire eval howto - Remove explicit name: frontmatter from os-adr and os-doc-hygiene SKILL.mds so slash commands register namespaced (/os-adr:find, not bare /find) - Add bin/refresh-plugins to refresh stale local plugin caches - CLAUDE.md: cache-refresh procedure, naming-convention pointer, mandate reading the vault autoresearch-eval howto before the next eval Co-Authored-By: Claude Fable 5 --- CLAUDE.md | 28 +++- bin/refresh-plugins | 123 ++++++++++++++++++ plugins/os-adr/skills/create/SKILL.md | 1 - plugins/os-adr/skills/find/SKILL.md | 1 - plugins/os-adr/skills/init/SKILL.md | 1 - plugins/os-adr/skills/migrate/SKILL.md | 1 - plugins/os-doc-hygiene/skills/check/SKILL.md | 1 - plugins/os-doc-hygiene/skills/clean/SKILL.md | 1 - plugins/os-doc-hygiene/skills/status/SKILL.md | 1 - plugins/os-doc-hygiene/skills/sweep/SKILL.md | 1 - 10 files changed, 148 insertions(+), 11 deletions(-) create mode 100755 bin/refresh-plugins diff --git a/CLAUDE.md b/CLAUDE.md index a8dbae1..8dd093b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -101,8 +101,8 @@ to those two and fix the stale doc. - Eval A harness (2026-07-03, ADR-021): `plugins/os-adr/eval/` — prompted skill-execution eval across model tiers (haiku/sonnet). Two fixtures (generated with the plugin's own CLIs), six scenarios (S1 create, S2 create+supersede, S3 find/conflict, S4 find/distractor, S5 init, S6 migrate+fills), deterministic Ruby checker (`eval/bin/check`, structural invariants, TSV mode for autoresearch), sandbox + headless-runner scripts, runner-prompt template. Primary run mode: in-session Agent-tool subagents with pinned `model:` (cheaper than `claude -p`); optimized via the `/autoresearch` Classic loop over SKILL.md *wording only* (checker/fixtures/scenarios frozen during a loop). Procedure: `plugins/os-adr/eval/README.md`. Self-tested both directions; grid run 2026-07-03, all scenarios × both tiers passing. Distinct from the held-out Eval B (unprompted behavior) — do not conflate. - Eval B harness (2026-07-03, OpenSpec change `add-os-adr-eval-b-harness`): `plugins/os-adr/eval-b/` — held-out unprompted-behavior eval (requirements 4–5). 7 scenarios (W1–W3 write-trigger, R1–R4 retrieval) authored from the frozen shapes in `docs/adr-system/06-eval-scenarios.md`; dedicated Ruby webhook-relay fixture with a 6-ADR history (Superseded pair + near-miss distractors, generated via the plugin's own CLIs); R4's one-hop graph reach uses a real `graphify update` AST build (model-free, rebuilt via `eval-b/bin/build-fixture-graph`, never committed). **Headless-only runner** (`eval-b/bin/run` — fresh `claude -p` per rep, cwd = sandbox, so the real SessionStart hook fires; in-session subagents are invalid here, unlike Eval A) and a two-axis deterministic-first checker (`eval-b/bin/check`): axis (a) unprompted consultation, mechanical from transcript tool_use blocks; axis (b) correct-ADR citation (R1–R4) or new-ADR-file with a narrow frozen-rubric haiku judge fallback (W1–W3, `judge-rubric.md`, stubbable via `ADR_EVAL_B_JUDGE_CMD`). `R4-nograph` is the graph-degradation variant (expected FAIL). Self-tested both directions model-free via `eval-b/bin/self-test`. **Scenario Task blocks are held-out — never run them informally.** Procedure: `plugins/os-adr/eval-b/README.md`. Grid run 2026-07-03 (1 rep/cell): haiku 0/8 PASS (never unprompted-consults the ADR system in any scenario — a real gap, not a harness defect); sonnet 5/8 PASS (fails W3 — doesn't propose recording the decision; fails R1 — misses the direct-conflict retrieval). `R4-nograph` FAILed on both tiers as expected (degradation check, only meaningful paired with an R4 PASS — sonnet has one, haiku doesn't). Full results + observations (prompting-issue hypothesis, open question on whether in-session subagents could ever validly substitute for part of this measurement) written to the vault: [[os-adr-eval-b-grid-results-and-observations]]. - **Initial iteration complete (2026-07-03):** plugin build, migration pilot, Eval A grid (clean pass), and Eval B grid (baseline captured above) are all done — this closes the first pass on os-adr. Follow-on work is deliberately deferred to future sessions, not in-flight. -- **Remaining (locked rollout order):** real-project migration/adoption one at a time via `/os-adr:migrate` — pilot projects first, then the cc-os retrofit (its 19-ADR monolithic file was deliberately excluded from the pilot), then wider; then design and run a follow-up eval iterating Eval B trigger/retrieval wording via `/autoresearch` (checker/fixtures/scenarios/rubric frozen) targeting the W3 and R1 failures on sonnet and the wholesale haiku gap — that eval must reference the vault note above for baseline numbers, and should also resolve the open question of whether in-session subagents can validly stand in for any part of the measurement; re-run the grid with more reps/cell once wording is stable to check variance. -- **TODO (not started, future session):** review and correct plugin naming conventions across the marketplace — os-adr's (and possibly other plugins') slash commands are failing to register/align consistently with sibling plugins (`os-vault`, `os-orchestration`, `os-doc-hygiene`). Likely the same class of stale-marketplace-manifest issue documented under "Renaming or moving a local plugin" below, but needs its own investigation — do not assume the fix is identical without checking `~/.claude/plugins/.claude-plugin/marketplace.json` and `installed_plugins.json` first. +- **Remaining (locked rollout order):** real-project migration/adoption one at a time via `/os-adr:migrate` — pilot projects first, then the cc-os retrofit (its 19-ADR monolithic file was deliberately excluded from the pilot), then wider; then design and run a follow-up eval iterating Eval B trigger/retrieval wording via `/autoresearch` (checker/fixtures/scenarios/rubric frozen) targeting the W3 and R1 failures on sonnet and the wholesale haiku gap. **Before designing or running that (or any) autoresearch eval, Read the vault howto `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md`** — it locks the procedure (valid run mode per eval shape, cache refresh via `bin/refresh-plugins` after every wording edit, reduced parallel inner-loop grid, ~3 reps on target cells, axis-level failure reading, verify from TSV not agent prose). Baseline numbers + the 2026-07-04 confirmation run live in the vault note above (baseline reconfirmed: haiku 0/8, sonnet 5/8 failing W3/R1; W3 is an axis-b failure — consults but doesn't propose recording — so its wording target is the create skill, while R1 is an axis-a trigger failure). +- **Resolved (2026-07-04):** os-adr's skills failed to register on first install due to stale plugin caches — the cache that was installed on 2026-07-03 17:21:48 was missing `hooks/hooks.json`, `bin/adr-detect`, `bin/adr-find`, `bin/adr-migrate`. Root cause: unknown, but the caches were restored via plugin uninstall/reinstall. A parallel issue affected os-doc-hygiene (cache retained deleted `commands/` directory). Fixed by manually refreshing both caches, then created `bin/refresh-plugins` automation to prevent future drift — see "Editing a local plugin (cache refresh)" subsection for refresh procedure. Investigation result: stale caches (not manifest-naming issues) were the cause; slash command registration works correctly once caches are fresh. **Graphify** — v0.8.31 at `/home/jared/.local/bin/graphify` - Vault graph: `~/Documents/SecondBrain/graphify-out/` — disposable, structural index rebuilt by SessionStart hook; handles relational/graph queries. Distinct from vault-index.json (planned, Plan B Phase 1): a flat `{tag: [{path, title, summary}]}` lookup rebuilt at SessionEnd, queried by the `/memory-find` skill for fast tag-based SB note discovery. @@ -155,6 +155,19 @@ Skipping step 2 is the failure mode to watch for: hooks (wired by absolute path `settings.json`) keep working, masking the fact that skills/slash-commands silently stopped registering under the plugin's new name. +### Editing a local plugin (cache refresh) + +Plugin source edits (including SKILL.md wording, hook scripts, or CLI code) do not reach running +sessions until the cache is refreshed. Claude Code caches plugins when installed, copying them +to `~/.claude/plugins/cache////`. To refresh a stale cache after +source edits: + +```bash +bin/refresh-plugins [marketplace-names...] # refresh all or specific local marketplaces +``` + +This uninstalls and reinstalls each plugin, forcing the cache to repopulate. See `bin/refresh-plugins --help` for usage. After refreshing, `claude plugin details @` should show updated skill descriptions and hook counts. Directory-source marketplaces (e.g., `local-plugins` sourced from `~/.claude/plugins`, `cc-plugins` sourced from `~/dev/cc-plugins`) are the only ones that need manual refresh — public marketplaces auto-update. + ## Issue tracking Issues (created via `/to-issues`) live on self-hosted Forgejo (`jared/cc-os`), queried with the `tea` CLI — not GitHub/`gh`. See `docs/issue-workflow.md` for token setup, listing, implementing, and closing issues. @@ -183,6 +196,15 @@ project context for OpenSpec artifacts comes from `docs/` and this file. `03-architecture-decisions.md` with the reasoning. - The package on PyPI is `graphifyy` (double-y) but the command is `graphify`. - **Keep this file current:** When a build step completes, (a) mark it done in `docs/memory-system/04-build-plan.md`, (b) add or update a component pointer in the Implemented Components section above, and (c) update the current design paragraph if the design changed. This file is the AI's orientation entry point — accuracy matters more than brevity. -- **Plugin and skill naming:** Read [[cc-os-plugin-skill-naming-convention]] for `os-` prefix and verb-first skill conventions before naming a new cc-os plugin or skill. Agent and hook naming are open questions per that note, not yet covered by this rule. +- **Plugin and skill naming:** Before naming ANY new cc-os plugin, skill, or slash command, Read + `~/Documents/SecondBrain/cc-os-plugin-skill-naming-convention.md` and follow it — do NOT invent + a new convention. Rules in brief: plugins are `os-[domain]`; skills are verb-first kebab-case + (`query`, `onboard-project`), invoked as `/os-[domain]:[verb]`; no `commands/` dispatcher + directories; **never set a `name:` field in SKILL.md frontmatter** — the directory name is the + skill name, and an explicit `name:` collapses the slash command to a bare unnamespaced form + (`/find` instead of `/os-adr:find`; found and fixed 2026-07-04). After editing any SKILL.md, + run `bin/refresh-plugins` (installs cache plugin files; source edits don't reach sessions until + refreshed). Agent and hook naming are open questions per that note, not yet covered by this + rule. **Session orchestration behavior** — Provided by the global `os-orchestration` plugin (see Implemented Components below). This repo no longer carries a local orchestration override; it follows the plugin's default behavior like every other project. diff --git a/bin/refresh-plugins b/bin/refresh-plugins new file mode 100755 index 0000000..4fd9da1 --- /dev/null +++ b/bin/refresh-plugins @@ -0,0 +1,123 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# Refresh Claude Code plugin caches for directory-source marketplaces. +# Discovers installed plugins from each local marketplace and refreshes their caches. +# +# Usage: +# bin/refresh-plugins [marketplace-names...] +# bin/refresh-plugins # refresh all local marketplaces +# bin/refresh-plugins local-plugins # refresh only local-plugins +# bin/refresh-plugins cc-plugins local-plugins + +require 'json' +require 'pathname' + +class PluginRefresher + attr_reader :results + + def initialize + @results = [] + end + + def refresh(marketplace_names = nil) + marketplaces = load_marketplaces(marketplace_names) + + marketplaces.each do |name, config| + next unless config['source']['source'] == 'directory' + + puts "Refreshing marketplace: #{name}" + refresh_marketplace(name, config) + puts + end + + report_results + end + + private + + def load_marketplaces(names) + all = JSON.parse(File.read(File.expand_path('~/.claude/plugins/known_marketplaces.json'))) + return all unless names + + all.select { |k, _| names.include?(k) } + end + + def refresh_marketplace(marketplace_name, config) + source_path = config['source']['path'] + installed = load_installed_plugins(marketplace_name) + + installed.each do |plugin_name| + refresh_plugin(marketplace_name, plugin_name, source_path) + end + end + + def load_installed_plugins(marketplace_name) + all = JSON.parse(File.read(File.expand_path('~/.claude/plugins/installed_plugins.json'))) + all['plugins'] + .select { |k, _| k.end_with?("@#{marketplace_name}") } + .keys + .map { |k| k.sub(/@#{marketplace_name}$/, '') } + .uniq + end + + def refresh_plugin(marketplace_name, plugin_name, source_path) + cache_exists = cache_exists?(marketplace_name, plugin_name) + + unless File.exist?(File.join(source_path, plugin_name)) + @results << [plugin_name, "source not found at #{source_path}", :error] + puts " #{plugin_name}: ERROR — source not found" + return + end + + system("claude plugin uninstall #{plugin_name}@#{marketplace_name} > /dev/null 2>&1") + system("claude plugin install #{plugin_name}@#{marketplace_name} > /dev/null 2>&1") + + if verify_cache(marketplace_name, plugin_name, source_path) + @results << [plugin_name, "refreshed", :success] + puts " #{plugin_name}: REFRESHED" + elsif !cache_exists + @results << [plugin_name, "in-sync (not previously cached)", :info] + puts " #{plugin_name}: IN SYNC (new)" + else + @results << [plugin_name, "STALE", :warning] + puts " #{plugin_name}: STALE (differences remain)" + end + end + + def cache_exists?(marketplace_name, plugin_name) + Dir.exist?(File.expand_path("~/.claude/plugins/cache/#{marketplace_name}/#{plugin_name}")) + end + + def verify_cache(marketplace_name, plugin_name, source_path) + cache_base = File.expand_path("~/.claude/plugins/cache/#{marketplace_name}/#{plugin_name}") + source_full = File.join(source_path, plugin_name) + + # Find the version directory in cache + versions = Dir.glob(File.join(cache_base, '*')).select { |p| File.directory?(p) } + return false if versions.empty? + + cache_path = versions.first + + # Run diff check (exclude plugin manager state files) + output = `diff -rq "#{cache_path}" "#{source_full}" --exclude='.git' --exclude='__pycache__' --exclude='.orphaned_at' 2>&1` + output.strip.empty? + end + + def report_results + puts "\n=== Summary ===" + successes = @results.select { |_, _, status| status == :success } + warnings = @results.select { |_, _, status| status == :warning } + errors = @results.select { |_, _, status| status == :error } + infos = @results.select { |_, _, status| status == :info } + + puts "Refreshed: #{successes.map { |r| r[0] }.join(', ')}" if successes.any? + puts "In sync: #{infos.map { |r| r[0] }.join(', ')}" if infos.any? + puts "Stale (not fixed): #{warnings.map { |r| r[0] }.join(', ')}" if warnings.any? + puts "Errors: #{errors.map { |r| "#{r[0]} (#{r[1]})" }.join(', ')}" if errors.any? + puts "All plugins synchronized." if @results.all? { |_, _, s| s != :warning && s != :error } + end +end + +marketplace_names = ARGV.empty? ? nil : ARGV +PluginRefresher.new.refresh(marketplace_names) diff --git a/plugins/os-adr/skills/create/SKILL.md b/plugins/os-adr/skills/create/SKILL.md index bf17859..72576d6 100644 --- a/plugins/os-adr/skills/create/SKILL.md +++ b/plugins/os-adr/skills/create/SKILL.md @@ -1,5 +1,4 @@ --- -name: create description: Record an architecture decision as a correctly templated, numbered, and indexed ADR in one invocation. Use when a decision of real consequence (irreversible, cross-cutting, contested, or convention-setting) has just been made or is being made — including unprompted, when you recognize such a decision mid-task. Invoked by `/os-adr:create`. --- diff --git a/plugins/os-adr/skills/find/SKILL.md b/plugins/os-adr/skills/find/SKILL.md index d512d5a..067b77f 100644 --- a/plugins/os-adr/skills/find/SKILL.md +++ b/plugins/os-adr/skills/find/SKILL.md @@ -1,5 +1,4 @@ --- -name: find description: Retrieve the architecture decision records relevant to files, components, or a topic being worked on — callable at any point mid-task, not just at session start. Use unprompted before changing behavior that may already be decided (a locked choice, a convention, a rejected alternative). Invoked by `/os-adr:find`. --- diff --git a/plugins/os-adr/skills/init/SKILL.md b/plugins/os-adr/skills/init/SKILL.md index 5731441..6b7d0b1 100644 --- a/plugins/os-adr/skills/init/SKILL.md +++ b/plugins/os-adr/skills/init/SKILL.md @@ -1,5 +1,4 @@ --- -name: init description: Set up the ADR system (docs/adr/ + generated index) in a project that has none, and clear any session-reminder suppression. Invoked by `/os-adr:init`. --- diff --git a/plugins/os-adr/skills/migrate/SKILL.md b/plugins/os-adr/skills/migrate/SKILL.md index 440aeb4..acbbb73 100644 --- a/plugins/os-adr/skills/migrate/SKILL.md +++ b/plugins/os-adr/skills/migrate/SKILL.md @@ -1,5 +1,4 @@ --- -name: migrate description: Non-destructively convert a project's existing ADR-like content (numbered files, dated files, monolithic decision logs, prose-embedded decisions) into the standard docs/adr/ format. Sources are never touched; uncertainty is flagged via migration_confidence + a migration report. Invoked by `/os-adr:migrate`. --- diff --git a/plugins/os-doc-hygiene/skills/check/SKILL.md b/plugins/os-doc-hygiene/skills/check/SKILL.md index 37cbc88..4f93ad0 100644 --- a/plugins/os-doc-hygiene/skills/check/SKILL.md +++ b/plugins/os-doc-hygiene/skills/check/SKILL.md @@ -1,5 +1,4 @@ --- -name: check description: Scan the project for stale and bloated documentation and write a hygiene report. Runs the deterministic scanner, dispatches a Sonnet subagent to classify only the signal-bearing candidates, finalizes/validates the machine report deterministically, then writes the report pair and stamps `last_check`. Invoked by `/os-doc-hygiene:check [--scope ] [--category ]`. --- diff --git a/plugins/os-doc-hygiene/skills/clean/SKILL.md b/plugins/os-doc-hygiene/skills/clean/SKILL.md index 85c5145..298e1d4 100644 --- a/plugins/os-doc-hygiene/skills/clean/SKILL.md +++ b/plugins/os-doc-hygiene/skills/clean/SKILL.md @@ -1,5 +1,4 @@ --- -name: clean description: Apply documented hygiene findings to project docs. Loads the current machine report, gates confirm-tier entries, applies deterministic ops via the patch applier, dispatches generative ops to a Sonnet subagent, stages precisely, and produces exactly one git commit. Invoked by `/os-doc-hygiene:clean [--scope ] [--category ]`. --- diff --git a/plugins/os-doc-hygiene/skills/status/SKILL.md b/plugins/os-doc-hygiene/skills/status/SKILL.md index 79a7dd1..6b355e3 100644 --- a/plugins/os-doc-hygiene/skills/status/SKILL.md +++ b/plugins/os-doc-hygiene/skills/status/SKILL.md @@ -1,5 +1,4 @@ --- -name: status description: Show the doc-hygiene lifecycle state for the current project — last check/clean/reminded timestamps and whether a report exists. Read-only, no model, no scan. Invoked by `/os-doc-hygiene:status`. --- diff --git a/plugins/os-doc-hygiene/skills/sweep/SKILL.md b/plugins/os-doc-hygiene/skills/sweep/SKILL.md index 9fa373b..ec9064d 100644 --- a/plugins/os-doc-hygiene/skills/sweep/SKILL.md +++ b/plugins/os-doc-hygiene/skills/sweep/SKILL.md @@ -1,5 +1,4 @@ --- -name: sweep description: Run doc-hygiene check then clean in sequence — a convenience wrapper. The confirm gate in `clean` still applies; sweep never auto-approves confirm-tier or generative entries. Invoked by `/os-doc-hygiene:sweep [--scope ] [--category ]`. ---