Smithery is now a part of Arcade.dev! Read more in our announcement here
Diagnoses failing E2E tests to determine if failure is a code bug or test issue...
Diagnose failing E2E tests to determine whether the failure indicates a code bug or a test issue.
Invoke when user says:
Input: Path to a failing E2E test spec file (e.g., playwright/tests/session-creation.spec.ts)
| Phase | Action | Details | Duration |
|---|---|---|---|
| 1 | Run Test | Execute the failing test, capture full output | 30-120s |
| 2 | Error Analysis | Parse error message, identify failure type | ~10s |
| 3 | Code Investigation | Check recent git changes, trace relevant code | ~30s |
| 4 | Pattern Check | Compare test against playwright-best-practices.md | ~20s |
| 5 | Verdict & Fix | Deliver diagnosis + recommended fix | ~10s |
Expected total duration: 2-4 minutes depending on test complexity.
For detailed phase instructions: references/diagnostic-phases.md
Use this flowchart to classify failures:
E2E Test Failing?
│
├─ Test PASSES when skill runs it?
│ └─ FLAKY TEST - Likely test issue (race condition, timing)
│ → Check for missing waits, hardcoded timeouts
│
├─ In CI only, passes locally?
│ ├─ Worker/async related? → Pattern 4 (emit_automation_event)
│ ├─ Route cleanup issue? → Pattern 5 (fixture import)
│ └─ Parallel collision? → Pattern 12 (worker-isolated dates)
│
├─ Timeout error?
│ ├─ Selector not found? → Check if element exists in code
│ ├─ Network timeout? → Check API endpoint exists
│ └─ Generic timeout? → Pattern 7 (use specific waits)
│
├─ Assertion failure?
│ ├─ Expected value changed? → Likely CODE BUG
│ ├─ Wrong expected value in test? → TEST ISSUE
│ └─ Data-dependent? → Check seeding
│
├─ Database constraint error?
│ └─ Pattern 11 (missing required fields, unique violations)
│
├─ Auth timeout?
│ └─ Pattern 1 (chromium vs chromium-clean project)
│
└─ Test order matters?
└─ Pattern 2 (test isolation, shared state)
Pattern references from: claude-patterns/playwright-best-practices.md
| Signal | Confidence |
|---|---|
| Test logic correct per patterns | High |
| Recent git changes touched code under test | High |
| Assertion on real behavior fails | High |
| Multiple tests failing on same feature | High |
| Same test passed before recent commit | Very High |
| Signal | Confidence |
|---|---|
| Pattern violations found | High |
| Hardcoded dates or non-isolated data | High |
| Uses UI creation when should seed | Medium |
Missing emit_automation_event |
High |
| Wrong project (chromium vs clean) | Very High |
Silent pass patterns (.catch(() => false)) |
Very High |
| Test passes on re-run (flaky) | High |
Sometimes a failing test reveals BOTH issues:
If signals are ambiguous:
npx playwright test <test-file> --reporter=list --timeout=60000 --retries=0
Flags explained:
--reporter=list: Clear output format--timeout=60000: 60s timeout for faster diagnosis--retries=0: No retries to see actual failureIf test PASSES:
Read the playwright output and classify:
TimeoutError, waiting for selector, waiting forexpect(received).toBe(expected), toHaveText, toBeVisiblenet::ERR_, ECONNREFUSED, fetch failedduplicate key, violates foreign key, null valueunauthorized, 401, login failed# Check recent changes to relevant files
git log --oneline -10 -- <relevant-paths>
# Diff against last known good state
git diff HEAD~5 -- <relevant-paths>
Identify:
Read claude-patterns/playwright-best-practices.md and check test against:
Present findings using format from references/verdict-template.md
Commands:
# Run specific test
npx playwright test playwright/tests/<name>.spec.ts
# Run with trace
npx playwright test <test> --trace on
# Show last report
npx playwright show-report
Key Pattern Files:
claude-patterns/playwright-best-practices.md - Primary diagnostic sourceplaywright/fixtures/seedHelpers.account.ts - Seeding helpersplaywright/fixtures/testIsolation.ts - Worker-isolated datesplaywright/utils/timeouts.ts - TIMEOUTS constantsCommon Fixes:
| Issue | Fix |
|---|---|
| Hardcoded date | Use getIsolatedDate() from testIsolation.ts |
| UI creation for setup | Use seedAccount(), seedSession() from seedHelpers |
| Missing wait | Add await expect(element).toBeVisible() before interaction |
| Silent pass | Remove .catch(() => false), add hard assertion |
v1.1.0 (2025-01-18): AI optimization updates
v1.0.0 (2025-01-15): Initial release
