Eval Gap Finder

Automates the process of finding and closing the gap between Python and AILANG benchmark success rates. Identifies language limitations, prompt gaps, and missing stdlib functions.

Quick Start

Most common usage:

# User says: "Find eval gaps" or "Analyze benchmark failures"
# This skill will:
# 1. Run evals at the Core tier with dev models (resolved dynamically from
#    internal/eval_harness/models.yml — no hardcoded model lists)
# 2. Compare Python vs AILANG Core pass rates (the headline metric)
# 3. Identify benchmarks where Python passes but AILANG fails, and
#    AILANG-only wins (keep-candidates) vs saturated (demote-candidates)
# 4. Analyze error patterns and categorize them
# 5. Check if gaps are documented in prompt
# 6. Test proposed examples and add to prompt
# 7. Create design docs for language limitations

When to Use This Skill

Invoke this skill when:

• User asks to "find eval gaps" or "close the Python-AILANG gap"
• User wants to analyze benchmark failures
• After running evals and seeing lower AILANG success
• User says "why is AILANG failing?" or "improve AILANG benchmarks"
• User wants to identify language limitations

Available Scripts

scripts/run_gap_analysis.sh [eval_dir] [--tier <tier>]

Run full gap analysis on eval results. Accepts --tier (default core); runs a fresh eval via dev_models_csv when no eval_dir is supplied.

# Analyse an existing baseline
.claude/skills/eval-gap-finder/scripts/run_gap_analysis.sh eval_results/baselines/v0.14.0

# Or run a fresh core-tier eval against dev models resolved from models.yml
.claude/skills/eval-gap-finder/scripts/run_gap_analysis.sh --tier core

scripts/identify_python_only.sh <eval_dir>

List benchmarks where Python passes but AILANG fails.

.claude/skills/eval-gap-finder/scripts/identify_python_only.sh eval_results/baselines/v0.14.0

scripts/categorize_errors.sh <eval_dir>

Categorize AILANG failures by error type.

.claude/skills/eval-gap-finder/scripts/categorize_errors.sh eval_results/baselines/v0.14.0

scripts/test_example.sh <code>

Test if an AILANG code example compiles and runs correctly.

.claude/skills/eval-gap-finder/scripts/test_example.sh /tmp/test.ail

Workflow

1. Run Evals at the Core Tier With Dev Models

Dev models are resolved from internal/eval_harness/models.yml — never hardcode.

# Source the shared helper to resolve dev models dynamically
. .claude/skills/_shared/scripts/eval_lib.sh

# Core tier is the headline metric; run it first
ailang eval-suite --models "$(dev_models_csv)" --tier core --output eval_results/gap-analysis

Run cheap/fast dev models first. If Core passes there, larger models should too. Only move to --tier stretch or --tier vision once Core is healthy — those tiers are expected to have mixed results and shouldn't drive prompt changes.

2. Generate Summary and Identify Gaps

ailang eval-summary eval_results/gap-analysis
.claude/skills/eval-gap-finder/scripts/identify_python_only.sh eval_results/gap-analysis

Key Core-tier metrics (from latest.json aggregates or ailang eval-matrix):

• AILANG Core pass rate — the headline number (Python is usually ~95%; gap is what we close)
• Python-only wins — benchmarks we can likely fix with prompt or language work
• AILANG-only wins — keep-candidates; evidence of AILANG value
• Saturated (both ≥ 95%) — demote-candidates; not revealing anything

3. Analyze Error Patterns

For each Python-only pass, categorize the error:

Category	Pattern	Fix Approach
WRONG_LANG	Model wrote Python syntax	Stronger "NOT Python" in prompt
PAR_001	Parse errors (syntax)	Add more examples to prompt
Type errors	Type unification failures	May be language limitation
Logic errors	Compiles but wrong output	Better examples or algorithm
EOF errors	Incomplete code generation	Model limitation, not prompt

4. Check Prompt Coverage

For each gap, check if the pattern is documented in the current teaching prompt (find the active file with ls prompts/ | grep -E 'v0\.[0-9]+.*\.md'):

grep -n "pattern" prompts/v0.14.x.md

If not documented, add:

• Working example to Quick Reference section
• Entry in "What AILANG Does NOT Have" table (if limitation)
• New section if pattern is complex

5. Test Examples Before Adding

CRITICAL: Always test examples before adding to prompt!

cat > /tmp/test.ail << 'EOF'
module benchmark/solution
-- Your example code here
EOF
ailang run --caps IO --entry main /tmp/test.ail

If example fails, it reveals a language gap - create a design doc instead.

6. Create Design Docs for Language Gaps

If testing reveals a language limitation:

1. Create design doc: design_docs/planned/vX_Y_Z/m-<feature>.md
2. Document:
   • Minimal reproduction
   • Error message
   • Workaround (for prompt)
   • Proposed fix
3. Add workaround to prompt with note

7. Track Improvement

After updates, re-run evals at the same tier used for analysis. Dev models come from models.yml, so the command stays stable as the roster evolves:

(. .claude/skills/_shared/scripts/eval_lib.sh && \
  ailang eval-suite --models "$(dev_models_csv)" --tier core --output eval_results/gap-analysis-v2)

Compare:

• Core pass-rate improvement (Python vs AILANG gap)
• Which previously Python-only benchmarks now pass in AILANG
• Any regressions (previously-passing AILANG benchmarks now failing)
• Whether saturated/AILANG-only sets shifted — a sprint that adds three new saturated benchmarks is less valuable than one that adds one AILANG-only win

Tier-First Workflow (v0.14.0+)

The v0.14.0 tier system lets us focus gap closing on where it matters. Walk through this loop every time, and skip any step that doesn't apply.

1. Run at --tier core first. Core is the headline number. If Core is clean, stretch failures are expected and usually not worth prompt changes.
2. Use DetectAILANGOnlyWins to identify keepers. Benchmarks where AILANG beats Python by ≥10pp are the skill's value evidence — protect them from regressions.

   ailang eval-matrix --ailang-wins
   

3. Use DetectSaturation to identify demotion candidates. Benchmarks where both AILANG and Python hit ≥95% tell us nothing. They're not regression guards — a broken change will hit many benchmarks. Demote them to free up run time for signal.

   ailang eval-matrix --show-saturated
   

4. Close gaps in Core before touching Stretch. A failing Stretch benchmark is allowed; a failing Core benchmark means we regressed on the headline metric.
5. Check tag coverage. Gaps clustered in one tag (e.g., 4/5 failures in recursion) probably signal a language or prompt problem, not per-benchmark bugs.

   ailang eval-matrix --by-tags
   

This matches the skill's rotation philosophy below: close Core gaps, demote the obvious, protect the wins.

Benchmark Rotation Philosophy

The eval suite is curated, not accumulated. The goal is high-signal benchmarks — ones that tell us something about relative performance across AILANG, Python, and agent harnesses. Time-series continuity is secondary; a small core of stable benchmarks is retained for regression guarding, but the rest should rotate as signal changes.

Before adding, removing, or proposing fixes for a benchmark, answer:

• Is it saturated? Both Python ≥ 95% AND AILANG ≥ 95% → demote to stretch or remove.
• Is it an AILANG-only win? AILANG ≥ Python by ≥ 10pp → keep and cite as value evidence.
• Does it fill an under-covered tag? Check ailang eval-matrix --by-tags; a new benchmark in a thin tag is more valuable than one more in a saturated tag.

When prompt changes fix a benchmark, that's a success — but record whether the benchmark also became saturated. If yes, it's done its job; rotate it out.

Error Categories Reference

Error	Meaning	Fix
WRONG_LANG	Wrote Python instead	Prompt emphasis
PAR_001	Parser error	Syntax examples
PAR_UNEXPECTED_TOKEN	Wrong token	Syntax examples
TC_*	Type check error	Type examples or design doc
"undefined variable"	Missing import/letrec	Document pattern
EOF errors	Incomplete code	Model limitation
logic_error	Wrong output	Algorithm examples

Resources

Gap Analysis Template

See resources/gap_analysis_template.md for structured analysis format.

Common Patterns

See resources/common_patterns.md for frequently encountered gaps.

Progressive Disclosure

This skill loads information progressively:

1. Always loaded: This SKILL.md file (workflow overview)
2. Execute as needed: Scripts in scripts/ directory
3. Load on demand: Resources for templates and patterns

Notes

• Always test examples before adding to prompt
• Prefer fixing language over prompt workarounds
• Track improvements with before/after eval runs at the same tier
• Create design docs for language limitations
• Update prompt hash in versions.json after changes
• Dev models come from internal/eval_harness/models.yml via .claude/skills/_shared/scripts/eval_lib.sh — never hardcode model IDs
• The eval suite is curated, not accumulated — see Benchmark Rotation Philosophy above