112 lines
2.5 KiB
Markdown
112 lines
2.5 KiB
Markdown
|
# Verification Pattern
|
||
|
|
|
||
|
|
Mandatory verification before claiming any work complete.
|
||
|
|
|
||
|
|
## Purpose
|
||
|
|
|
||
|
|
Prevent false completion claims. Every tool must specify how to verify it works.
|
||
|
|
|
||
|
|
## Core Rule
|
||
|
|
|
||
|
|
**Never claim work is complete without running verification.**
|
||
|
|
|
||
|
|
This applies to:
|
||
|
|
- Building new tools
|
||
|
|
- Modifying existing tools
|
||
|
|
- Fixing issues found in audits
|
||
|
|
- Any task that changes behavior
|
||
|
|
|
||
|
|
## Implementation Checklist
|
||
|
|
|
||
|
|
Tools implementing this pattern must:
|
||
|
|
|
||
|
|
- [ ] Declare verification method in tool definition
|
||
|
|
- [ ] Verification must be executable (not just "review")
|
||
|
|
- [ ] Run verification before completion claim
|
||
|
|
- [ ] Report verification output, not just "it passed"
|
||
|
|
- [ ] If verification fails, do not claim completion
|
||
|
|
|
||
|
|
## Verification Types
|
||
|
|
|
||
|
|
### Script-based
|
||
|
|
```yaml
|
||
|
|
verification:
|
||
|
|
type: script
|
||
|
|
command: python scripts/validate_skill.py <skill-path>
|
||
|
|
success_criteria: "Exit code 0, no errors"
|
||
|
|
```
|
||
|
|
|
||
|
|
### Example-based
|
||
|
|
```yaml
|
||
|
|
verification:
|
||
|
|
type: golden_examples
|
||
|
|
examples_path: examples/
|
||
|
|
check: "Output matches expected for all examples"
|
||
|
|
```
|
||
|
|
|
||
|
|
### Behavioral
|
||
|
|
```yaml
|
||
|
|
verification:
|
||
|
|
type: behavioral
|
||
|
|
test: "Invoke with test input, check output matches invariants"
|
||
|
|
invariants_path: invariants.md
|
||
|
|
```
|
||
|
|
|
||
|
|
### Manual (last resort)
|
||
|
|
```yaml
|
||
|
|
verification:
|
||
|
|
type: manual
|
||
|
|
steps:
|
||
|
|
- "Run skill with test case X"
|
||
|
|
- "Verify output contains Y"
|
||
|
|
- "Check file Z was created"
|
||
|
|
```
|
||
|
|
|
||
|
|
## Workflow Integration
|
||
|
|
|
||
|
|
```
|
||
|
|
Work complete?
|
||
|
|
│
|
||
|
|
↓
|
||
|
|
Run verification ──→ Failed ──→ Fix issues ──→ (loop)
|
||
|
|
│
|
||
|
|
↓
|
||
|
|
Passed
|
||
|
|
│
|
||
|
|
↓
|
||
|
|
Check golden examples ──→ Changed ──→ Flag for review
|
||
|
|
│
|
||
|
|
↓
|
||
|
|
Report completion with verification output
|
||
|
|
```
|
||
|
|
|
||
|
|
## Anti-patterns
|
||
|
|
|
||
|
|
**Verification-by-assertion:** "I verified it works" without running anything. Always show evidence.
|
||
|
|
|
||
|
|
**Partial verification:** Checking one thing when multiple things changed. Verify all affected behaviors.
|
||
|
|
|
||
|
|
**Verification deferral:** "Will verify later." Verify now or don't claim completion.
|
||
|
|
|
||
|
|
**Silent failure:** Verification failed but reporting success. If it fails, say so.
|
||
|
|
|
||
|
|
## Evidence Format
|
||
|
|
|
||
|
|
When reporting completion, include:
|
||
|
|
|
||
|
|
```markdown
|
||
|
|
## Verification Results
|
||
|
|
|
||
|
|
**Method:** [script/examples/behavioral/manual]
|
||
|
|
**Command:** [what was run]
|
||
|
|
**Output:**
|
||
|
|
[actual output or summary]
|
||
|
|
|
||
|
|
**Status:** PASSED / FAILED
|
||
|
|
```
|
||
|
|
|
||
|
|
## Cross-references
|
||
|
|
|
||
|
|
- [Audit Pattern](audit-pattern.md) - Audits require verification of fixes
|
||
|
|
- [Reversion Protection](reversion-protection.md) - Golden examples are a verification method
|