Smithery is now a part of Arcade.dev! Read more in our announcement here
Skill testing methodology - run scenarios without skill (RED), observe failures, write skill (GREEN), close loopholes (REFACTOR).
Complementary: ring:writing-skills, ring:test-driven-development
Testing skills is TDD applied to process documentation.
Run scenarios without the skill (RED — watch agent fail), write skill addressing those failures (GREEN), then close loopholes (REFACTOR).
Prerequisite: Understand ring:test-driven-development first. Complete worked example: examples/CLAUDE_MD_TESTING.md.
Test skills that: enforce discipline (TDD, testing requirements), have compliance costs (time, effort, rework), could be rationalized away ("just this once"), or contradict immediate goals (speed over quality).
Skip: Pure reference skills (API docs), skills without rules to violate.
| TDD Phase | Skill Testing | What You Do |
|---|---|---|
| RED | Baseline test | Run scenario WITHOUT skill, watch agent fail |
| Verify RED | Capture rationalizations | Document exact failures verbatim |
| GREEN | Write skill | Address specific baseline failures |
| Verify GREEN | Pressure test | Run WITH skill, verify compliance under pressure |
| REFACTOR | Plug holes | Find new rationalizations, add counters |
Run 3+ combined-pressure scenarios WITHOUT the skill. Document agent choices and rationalizations word-for-word.
Why verbatim? Exact wording reveals the loopholes to close.
| Quality | Example |
|---|---|
| Bad | "What does the skill say?" — agent recites |
| Good | "Production down, $10k/min, 5min window" — single pressure |
| Great | "3hr/200 lines done, 6pm, dinner plans, forgot TDD. A) Delete B) Commit C) Tests now" — multi-pressure + forced choice |
Pressure types: Time (deadline), sunk cost (hours invested), authority (senior says skip), economic (job at stake), exhaustion (end of day), pragmatic ("being realistic").
Best tests combine 3+ pressures.
Address the specific failures documented in RED. Don't add hypothetical content — write just enough to address actual observed failures. Re-run same scenarios WITH skill; agent should now comply.
Agent still violated rule despite having the skill? Capture new rationalizations verbatim:
For each rationalization, add: explicit negation rule, rationalization table entry, red flag entry.
Meta-test: "You read the skill and chose wrong anyway. How could the skill have been written to make the right answer the only acceptable one?"
Continue REFACTOR until no new rationalizations appear.
From applying TDD to TDD skill itself:
