Smithery is now a part of Arcade.dev! Read more in our announcement here
Framework for evaluating and improving how well a codebase supports autonomous AI development...
Evaluate how well a codebase supports autonomous AI development across eight technical pillars and five maturity levels.
Agent performance problems are often environment problems, not model problems. Missing pre-commit hooks mean waiting minutes for CI feedback instead of seconds. Undocumented environment variables cause guess-and-fail loops. Build processes requiring tribal knowledge block agents entirely.
A codebase with fast feedback loops and clear instructions makes any agent dramatically more effective.
Each pillar addresses a specific failure mode in autonomous development.
Linters, type checkers, formatters (ESLint, Biome, TypeScript strict mode, Prettier, Black). Without this: Agent submits code with formatting issues, waits for CI, fixes blindly, repeats.
Reproducible builds, clear build commands, dependency management. Without this: Agent cannot verify builds work or understand build failures.
Unit tests, integration tests, E2E tests, test runners configured. Without this: Agent cannot verify changes work or detect regressions.
README, CONTRIBUTING, API docs, architecture docs, AGENTS.md. Without this: Agent lacks context for design decisions and conventions.
Setup scripts, dev containers, environment variable documentation. Without this: Agent cannot reliably set up or understand the environment.
Code review processes, CODEOWNERS, branch protection. Without this: Agent changes may bypass quality gates or lack review.
Logging, monitoring, error tracking configuration. Without this: Agent cannot diagnose production issues or understand system behavior.
Security scanning, dependency auditing, secrets management. Without this: Agent may introduce vulnerabilities or expose secrets.
Basic development possible. Code runs, minimal tooling.
README exists, basic setup documented, some tests present.
Production-ready for agents. Clear processes defined and enforced. E2E tests exist, docs maintained, security scanning, observability. Agent capability: Routine maintenance—bug fixes, tests, docs, dependency upgrades.
Comprehensive CI/CD, extensive testing, thorough documentation, proactive security. Agent capability: Feature development with minimal guidance.
Self-documenting, self-healing, comprehensive automation. Agent capability: Independent feature development and system improvements.
| Pillar | Key Signals |
|---|---|
| Style & Validation | Linter config, type checking enabled, formatter config, pre-commit hooks |
| Build System | Build scripts documented, CI pipeline exists, dependency lockfile |
| Testing | Test files exist, test command documented, CI runs tests |
| Documentation | README with setup, CONTRIBUTING guide, API docs, architecture docs |
| Dev Environment | Setup script or dev container, .env.example, documented dependencies |
| Code Quality | CODEOWNERS file, branch protection, PR template |
| Observability | Logging configured, error tracking, monitoring setup |
| Security | Security scanning in CI, dependency auditing, secrets not in code |
When reporting findings, structure as:
## Current Level: [1-5]
### Passing Criteria
- [List what's working well]
### Gaps (Priority Order)
1. [Highest impact gap] - [Why it matters] - [How to fix]
2. [Next gap]...
### Quick Wins
- [Easy fixes that improve agent effectiveness]
