Advanced Evaluation
LLM-as-a-Judge techniques for evaluating AI outputs. Not a single technique but a family of approaches - choosing the right one and mitigating biases is the core competency.
When to Activate
- Building automated evaluation pipelines for LLM outputs
- Comparing multiple model responses to select the best one
- Establishing consistent quality standards
- Debugging inconsistent evaluation results
- Designing A/B tests for prompt or model changes
- Creating rubrics for human or automated evaluation
Core Concepts
Evaluation Taxonomy
Direct Scoring: Single LLM rates one response on a defined scale.
- Best for: Objective criteria (factual accuracy, instruction following, toxicity)
- Reliability: Moderate to high for well-defined criteria
Pairwise Comparison: LLM compares two responses and selects better one.
- Best for: Subjective preferences (tone, style, persuasiveness)
- Reliability: Higher than direct scoring for preferences
Known Biases
| Bias |
Description |
Mitigation |
| Position |
First-position preference |
Swap positions, check consistency |
| Length |
Longer = higher scores |
Explicit prompting, length-normalized scoring |
| Self-Enhancement |
Models rate own outputs higher |
Use different model for evaluation |
| Verbosity |
Unnecessary detail rated higher |
Criteria-specific rubrics |
| Authority |
Confident tone rated higher |
Require evidence citation |
Decision Framework
Is there an objective ground truth?
├── Yes → Direct Scoring (factual accuracy, format compliance)
└── No → Pairwise Comparison (tone, style, creativity)
Quick Reference
Direct Scoring Requirements
- Clear criteria definitions
- Calibrated scale (1-5 recommended)
- Chain-of-thought: justification BEFORE score (improves reliability 15-25%)
Pairwise Comparison Protocol
- First pass: A in first position
- Second pass: B in first position (swap)
- Consistency check: If passes disagree → TIE
- Final verdict: Consistent winner with averaged confidence
Rubric Components
- Level descriptions with clear boundaries
- Observable characteristics per level
- Edge case guidance
- Strictness calibration (lenient/balanced/strict)
Integration
Works with:
- context-fundamentals - Effective context structure
- tool-design - Evaluation tool schemas
- evaluation (foundational) - Core evaluation concepts
For detailed implementation patterns, prompt templates, examples, and metrics: references/full-guide.md
See also: references/implementation-patterns.md, references/bias-mitigation.md, references/metrics-guide.md
