PuzzleTide Agent Evals
Generate reproducible, objectively gradable puzzle tasks for testing models
and agents with the local PuzzleTide CLI.
Why puzzles: they are verifiable by construction. A sudoku answer either
satisfies the rules and preserves the givens or it doesn't; a word search
answer either spells the word along a straight line in the grid or it
doesn't. Grading needs no LLM judge and no trusted answer key.
Prefer the local CLI. Check availability in this order:
ptide --version
puzzletide --version
npx puzzletide --version
If none of those work, ask the user before installing (npm install -g puzzletide).
Generate a task set
ptide eval generate --type sudoku --n 20 --difficulty hard --seed 1 --out tasks.json
ptide eval generate --type wordsearch --n 10 --difficulty medium --seed 1 --out tasks.json
The tuple (type, difficulty, n, seed) fully determines the task set, so it
names a reproducible benchmark — same command, same tasks, on any machine.
Each task has id, instructions, and the puzzle payload:
- sudoku:
puzzle (81 chars, . = empty). Expected answer: completed 81-char string.
- wordsearch:
grid (array of row strings) and words. Expected answer: JSON array
of {word, startRow, startCol, endRow, endCol} (0-indexed).
Run the subject model
Send each task's instructions + payload to the model under test and collect
answers as a JSON array of {id, answer}.
Grade
ptide eval check --tasks tasks.json --answers answers.json --json
Returns per-task pass/fail with reasons and a summary score. Grading is
deterministic and local.
Links
Safety
- Everything runs locally; no account, API key, or network access.
- Do not install packages without asking the user first.
