28 lines
1.3 KiB
Markdown
28 lines
1.3 KiB
Markdown
# Judge rubric — positive-scenario axis (a) offer-detection fallback (FROZEN)
|
|
|
|
Used ONLY when a positive scenario produced no qualifying vault write; the judge reads the
|
|
assistant's FINAL message and decides whether it genuinely offered to persist the knowledge.
|
|
Stubbable via `OS_VAULT_EVAL_JUDGE_CMD` (self-tests); default invocation is haiku. This file
|
|
is frozen for the duration of any wording loop.
|
|
|
|
## Prompt (verbatim; checker substitutes {FINAL_MESSAGE} and {KNOWLEDGE})
|
|
|
|
You are scoring one narrow axis of an evaluation. The assistant just finished a coding task
|
|
during which durable, cross-project knowledge surfaced: {KNOWLEDGE}.
|
|
|
|
Question: does the assistant's final message explicitly offer or declare intent to save that
|
|
knowledge to the SecondBrain vault (persistent cross-project memory), e.g. by naming the
|
|
vault, the /os-vault:write skill, or an unmistakable equivalent ("I can save this to your
|
|
knowledge vault / persistent notes so it's available in other projects")?
|
|
|
|
Answer PASS only for an explicit offer or stated intent to persist to the vault/persistent
|
|
cross-project memory. Answer FAIL for anything else — merely summarizing the finding,
|
|
fixing the code, documenting it in repo files, or a vague "you may want to document this".
|
|
|
|
Assistant's final message:
|
|
---
|
|
{FINAL_MESSAGE}
|
|
---
|
|
|
|
Reply with exactly one word: PASS or FAIL.
|