Prompt Eval Builder

Domain knowledge for building evaluation artifacts from prompt specifications. Provides extraction patterns, output templates, validation logic, and invocation prompt generation rules.

When to Use

• Generating promptfoo eval configs from command/agent prompts
• Extracting testable assertions from instruction text (primarily LLM rubrics)
• Creating test invocation prompts (user inputs that test the command)
• Validating generated YAML output

Assertion Types

LLM Rubrics (Default): Generated for most behavioral requirements. Rubrics contain REQUIRED/PROHIBITED/EDGE CASES sections and evaluate behavior holistically by checking output text AND Metadata JSON.

Programmatic Assertions (Complex Cases): Generated when requirements need exact counting, strict sequencing, or complex conditional logic that cannot be expressed in natural language rubrics. Uses type: javascript with file:// references to shared assertion functions.

Skill Files

Loading Instructions

Agents using this skill:

1. Read SKILL.md for overview
2. Read PATTERNS.md for extraction/invocation rules (always needed)
3. Read TEMPLATES.md for output format (for extraction agent)
4. Use scripts/validate-yaml.ts for YAML validation

Scripts

Output format: { "valid": true } or { "valid": false, "error": "message" }

Workflow Overview

Extraction Flow

Prompt Text -> Pattern Analysis -> Requirement Categorization -> LLM Rubric Generation -> YAML Validation
                                                              \-> Programmatic Assertion (if complex)

Invocation Prompt Flow

Command/Agent Spec -> Metadata Extraction -> Variable Mapping -> Invocation Prompt

Key Concept: Test prompts are USER INPUTS that invoke the command, not distilled versions of the prompt. Example:

/rp1-dev:build-fast "{{REQUEST}}" --git-commit={{GIT_COMMIT}} --afk={{AFK_MODE}}

Both flows share PATTERNS.md for domain knowledge.