Prefill Sensitivity Analysis Pipeline

This skill documents the complete pipeline for measuring model susceptibility to reward hacking via prefill sensitivity analysis, including both token-based and logprob-based metrics.

Overview

The analysis measures how easily a model can be "kicked" into generating exploit code by prefilling its chain-of-thought with exploit-oriented reasoning. We track:

1. Token-based metric: Minimum prefill tokens needed to elicit an exploit
2. Logprob-based metric: How "natural" the exploit reasoning appears to the model

End-to-End Pipeline

Stage 1 (eval) ──→ Stage 2 (logprobs/KL) ──→ Stage 3 (trajectory) ──→ Stage 4a (extrapolation)
                                                                    ──→ Stage 4b (binary emergence)
                                                                            ↑ (optional)
                                          exploit logprobs ─────────────────┘

Pre-requisites

Before Stage 1, you need:

1. SFT checkpoints from train_sft_checkpoints.py (see Multi-GPU Training in CLAUDE.md)
2. Prefill source: A .samples.jsonl from evaluating a late checkpoint at prefill=0 (see eval_checkpoint_sensitivity.py)

Dependency & Skip Logic

Pipeline	Stages	What you get
Token-based trajectories only	1 + 3	Min-prefill trajectories, descent rates, time-to-threshold
Logprob comparison + scaling law	1 + 2 + 3	Above + logprob trajectories, KL analysis, exploit_rate_scaling_by_type.csv
Extrapolation & GP prediction	1 + 2 + 3 + 4a	Above + AR(1), log-step extrapolation, GP predictions, metric comparison
Binary emergence prediction	1 + 2 + 4b	Predict which exploit types will emerge (requires 2 runs: exploit + control)
+ exploit logprobs	1 + 2 + 4b + exploit logprobs	Above + exploit_logprob metric (mean log P(exploit_code) per type)

Skip conditions:

• Skip Stage 2 → Stage 3 produces token-based analysis only (no logprob plots, scaling law, or exploit_rate_scaling_by_type.csv)
• Skip Stages 4a/4b → Stage 3 outputs are sufficient for most diagnostics
• Stage 4a can use pooled mode (needs Stage 2 KL data + evals) or max mode (needs exploit_rate_scaling_by_type.csv from Stage 3)
• Stage 4b requires two runs (exploit + control) with Stage 2 KL data

Incremental/resume support:

• Stage 1 supports --resume (skips already-evaluated checkpoint/prefill combos)
• Stage 2 is incremental (skips already-computed logprob files)

Decision Tree

• Need token-based trajectories only? → Run Stages 1 + 3
• Need logprob comparison and scaling law? → Run Stages 1 + 2 + 3
• Need to forecast future exploit rates? → Run Stages 1 + 2 + 3 + 4a
• Need to predict which types will emerge (requires exploit + control runs)? → Run Stages 1 + 2 + 4b

---

Prerequisites

• Model checkpoints from SFT training
• Prefill source data (successful exploit reasoning traces)
• vLLM for serving checkpoints
• djinn package for problem verification

---

Stage 1: Run Prefill Sensitivity Evaluation

Evaluate each checkpoint at multiple prefill levels (0, 2, 5, 10, 20, 30, 45, 60, 75, 100 tokens).

1.1 Serve the checkpoint via vLLM

vllm serve results/sft_checkpoints/sft_*/checkpoints/checkpoint-{CKPT}

1.2 Run the evaluation

python scripts/eval_prefill_sensitivity.py \
    --base-url http://localhost:8000/v1 \
    --prefill-from results/prefill_source/exploits.jsonl \
    --output results/prefill_sensitivity/{RUN_NAME}/evals/checkpoint-{CKPT}_prefill{LEVEL}.jsonl \
    --prefill-tokens {LEVEL} \
    --num-attempts 3

Key parameters:

• --prefill-tokens: Number of tokens from exploit reasoning to prefill (0 = baseline)
• --num-attempts: Number of generation attempts per problem (default: 3)
• --max-problems: Limit problems for testing

Output files:

• checkpoint-{CKPT}_prefill{LEVEL}.jsonl: Per-problem exploit success results
• checkpoint-{CKPT}_prefill{LEVEL}.jsonl.samples.jsonl: Full generation samples with reasoning

1.3 Batch script example

#!/bin/bash
RUN_NAME="prefill_sensitivity-$(date +%Y%m%d-%H%M%S)"
CHECKPOINTS=(1 6 15 25 44 76 100 228 330)
PREFILL_LEVELS=(0 2 5 10 20 30 45 60 75 100)

for CKPT in "${CHECKPOINTS[@]}"; do
    # Start vLLM server for this checkpoint
    vllm serve results/sft_checkpoints/sft_*/checkpoints/checkpoint-$CKPT &
    sleep 60  # Wait for server to start

    for LEVEL in "${PREFILL_LEVELS[@]}"; do
        python scripts/eval_prefill_sensitivity.py \
            --base-url http://localhost:8000/v1 \
            --prefill-from results/prefill_source/exploits.jsonl \
            --output results/prefill_sensitivity/$RUN_NAME/evals/checkpoint-${CKPT}_prefill${LEVEL}.jsonl \
            --prefill-tokens $LEVEL \
            --num-attempts 3
    done

    # Kill vLLM server
    pkill -f "vllm serve"
done

---

Stage 2: Compute Prefill Logprobs

Measure how "natural" exploit reasoning appears to each checkpoint using the vLLM server.

2.1 Batch mode (recommended): serve_and_compute_logprobs.py

Automatically serves all checkpoints and computes logprobs + KL in parallel batches. Reads config.yaml from the run dir to find checkpoints and checkpoint paths. Auto-detects free GPUs and skips already-computed files.

# Single run with external ref_logprobs:
python scripts/serve_and_compute_logprobs.py \
    --run-dir results/prefill_sensitivity/{RUN_NAME} \
    --ref-logprobs-dir results/prefill_sensitivity/{OTHER_RUN}/ref_logprob

# Multiple runs (e.g., clean + misalignment controls sharing ref_logprobs):
python scripts/serve_and_compute_logprobs.py \
    --run-dir results/prefill_sensitivity/run1 results/prefill_sensitivity/run2 \
    --ref-logprobs-dir results/.../ref_logprob

# If ref_logprobs are in {run-dir}/ref_logprob, auto-detected:
python scripts/serve_and_compute_logprobs.py \
    --run-dir results/prefill_sensitivity/{RUN_NAME}

# For larger models needing tensor parallelism:
python scripts/serve_and_compute_logprobs.py \
    --run-dir results/prefill_sensitivity/{RUN_NAME} \
    --ref-logprobs-dir results/.../ref_logprob \
    --tensor-parallel 2

# Specify GPUs manually:
python scripts/serve_and_compute_logprobs.py \
    --run-dir results/prefill_sensitivity/{RUN_NAME} \
    --ref-logprobs-dir results/.../ref_logprob \
    --gpus 0 1 2 3

# Dry run (show what would be done):
python scripts/serve_and_compute_logprobs.py \
    --run-dir results/prefill_sensitivity/{RUN_NAME} \
    --dry-run

Key parameters:

• --run-dir: One or more prefill sensitivity run directories (reads config.yaml)
• --ref-logprobs-dir: Reference logprobs for KL (default: auto-detect from {run-dir}/ref_logprob)
• --gpus: GPU indices (default: auto-detect free GPUs with <1GB used)
• --tensor-parallel: GPUs per checkpoint (default: 1)
• --concurrency: Concurrent API requests per checkpoint (default: 16)
• --stagger: Seconds between server startups (default: 5, avoids Harmony tokenizer races)
• --dry-run: Show plan without executing

2.2 Manual mode: compute_prefill_logprobs.py

For computing logprobs against an already-running vLLM server (one checkpoint at a time).

# Start vLLM server first:
vllm serve /path/to/checkpoints/checkpoint-{CKPT}

# Compute logprobs for all prefill levels:
python scripts/compute_prefill_logprobs.py \
    --base-url http://localhost:8000/v1 \
    --samples-dir results/prefill_sensitivity/{RUN_NAME}/evals \
    --output-dir results/prefill_sensitivity/{RUN_NAME}/logprob \
    --checkpoint {CKPT} \
    --concurrency 32

This processes all checkpoint-{CKPT}_prefill*.jsonl.samples.jsonl files and outputs to {RUN_NAME}/logprob/. Skips already-computed files.

2.3 Single file mode

python scripts/compute_prefill_logprobs.py \
    --base-url http://localhost:8000/v1 \
    --prefill-samples results/prefill_sensitivity/{RUN_NAME}/evals/checkpoint-{CKPT}_prefill{LEVEL}.jsonl.samples.jsonl \
    --output results/prefill_sensitivity/{RUN_NAME}/logprob/checkpoint-{CKPT}_prefill{LEVEL}_logprobs.jsonl

Key parameters:

• --concurrency N: Maximum concurrent API requests (default: 32)
• --batch-size N: Batch size for progress reporting (default: 64)
• --max-samples N: Limit samples for testing
• --min-prefill N: Skip prefill levels below N (default: 1, skips prefill0)
• --use-reasoning-field: Use 'reasoning' instead of 'prefill_reasoning' field

---

Stage 2 + KL: Compute Logprobs with KL Divergence

Logprobs and KL divergence can be computed in a single pass when reference logprobs are provided.

KL(P || Q) where P = prefill generator (reference), Q = evaluation checkpoint.

2.1 Compute reference logprobs (one-time)

First, compute per-token logprobs from the prefill generator (reference model). The reference checkpoint is determined from config.yaml's prefill_source field.

# Serve reference checkpoint (e.g., checkpoint-132)
vllm serve /path/to/checkpoints/checkpoint-{REF_CKPT}

# Compute reference logprobs (stores per-token logprobs)
python scripts/compute_prefill_logprobs.py \
    --base-url http://localhost:8000/v1 \
    --samples-dir results/prefill_sensitivity/{RUN_NAME}/evals \
    --output-dir results/prefill_sensitivity/{RUN_NAME}/ref_logprob \
    --checkpoint {REF_CKPT} \
    --concurrency 32

2.2 Compute logprobs + KL for each evaluation checkpoint

# Serve evaluation checkpoint
vllm serve /path/to/checkpoints/checkpoint-{CKPT}

# Compute logprobs AND KL divergence in single pass
python scripts/compute_prefill_logprobs.py \
    --base-url http://localhost:8000/v1 \
    --samples-dir results/prefill_sensitivity/{RUN_NAME}/evals \
    --output-dir results/prefill_sensitivity/{RUN_NAME}/logprob \
    --checkpoint {CKPT} \
    --ref-logprobs-dir results/prefill_sensitivity/{RUN_NAME}/ref_logprob \
    --kl-output-dir results/prefill_sensitivity/{RUN_NAME}/kl \
    --concurrency 32

This outputs:

• logprob/checkpoint-{CKPT}_prefill{L}_logprobs.jsonl: Standard logprob results
• kl/checkpoint-{CKPT}_prefill{L}_kl.jsonl: KL divergence results

Key parameters:

• --ref-logprobs-dir PATH: Directory with reference logprobs (enables KL computation)
• --kl-output-dir PATH: Output directory for KL results (default: {output-dir}/../kl)
• --min-prefill N: Skip prefill levels below N (default: 1)
• --concurrency N: Maximum concurrent API requests (default: 32)

KL output format:

• kl_divergence: Total KL divergence over prefill tokens
• kl_per_token: Average KL per token
• eval_logprob_sum: Eval checkpoint's logprob sum (for comparison)
• ref_logprob_sum: Reference model's logprob sum

---

Stage 3: Token-Based and Logprob Trajectory Analysis

Analyze how "exploit accessibility" changes over training, using both metrics.

Default behavior (filters to djinn dataset, produces both all_exploits/ and intentional_only/ subdirectories):

python scripts/prefill_trajectory_analysis.py \
    --run-dir results/prefill_sensitivity/{RUN_NAME} \
    --output-dir results/trajectory_analysis/{RUN_NAME} \
    --threshold 10

This automatically:

1. Filters to problems in EleutherAI/djinn-problems-v0.9 (removes bad/deprecated problems)
2. Produces plots for all exploits in all_exploits/ subdirectory
3. Produces plots for intentional exploits only in intentional_only/ subdirectory
   • Excludes inadequate_test_coverage and resource_exhaustion (unintentional exploit types)
4. Processes logprob data if available in {run-dir}/logprob/ (generates integrated analysis)

Skip intentional split (only produce all_exploits/):

python scripts/prefill_trajectory_analysis.py \
    --run-dir results/prefill_sensitivity/{RUN_NAME} \
    --output-dir results/trajectory_analysis \
    --threshold 10 \
    --skip-intentional-split

Disable dataset filtering:

python scripts/prefill_trajectory_analysis.py \
    --run-dir results/prefill_sensitivity/{RUN_NAME} \
    --output-dir results/trajectory_analysis \
    --threshold 10 \
    --filter-dataset none

With experiment context logging:

python scripts/prefill_trajectory_analysis.py \
    --run-dir results/prefill_sensitivity/{RUN_NAME} \
    --output-dir results/trajectory_analysis \
    --threshold 10 \
    --use-run-context

Key concepts:

• Min prefill: Minimum prefill tokens needed to trigger an exploit at a checkpoint
• Threshold: min_prefill <= threshold means "easily exploitable" (default: 10)
• Time to threshold: Training steps until problem becomes easily exploitable
• Instantaneous descent rate: Per-step change in min_prefill between consecutive checkpoints
• Intentional exploits: Excludes inadequate_test_coverage and resource_exhaustion

Output structure:

output_dir/
├── all_exploits/           # All exploit types
│   ├── trajectory_analysis.csv
│   ├── logprob_analysis.csv
│   ├── logprob_trajectory_analysis.csv
│   ├── exploit_rate_scaling.csv
│   └── *.png
└── intentional_only/       # Excludes unintentional exploit types
    ├── trajectory_analysis.csv
    ├── logprob_analysis.csv
    ├── logprob_trajectory_analysis.csv
    ├── exploit_rate_scaling.csv
    └── *.png

Token-based output files:

• trajectory_analysis.csv: Per-problem min_prefill at each checkpoint
• pass_rates_vs_prefill.png: Secure pass, insecure pass, and exploit rate vs prefill length
• accessibility_vs_time.png: Scatter plot of current accessibility vs steps-to-threshold
• sample_trajectories.png: Sample of individual problem trajectories
• median_trajectory.png: Median min_prefill trajectory with IQR band
• descent_rates.png: Distribution of overall descent rates
• instantaneous_descent_rates.png: Distribution of per-step descent rates
• instantaneous_descent_rates_by_exploit.png: Descent rates by exploit type

Logprob output files (if logprob data available):

• logprob_analysis.csv: Per-sample logprob metrics
• logprob_trajectory_analysis.csv: Logprob at min_prefill per checkpoint
• logprob_vs_prefill.png: Logprob metrics vs prefill length
• logprob_vs_checkpoint.png: Logprob metrics vs checkpoint
• logprob_ascent_rates.png: Distribution of logprob ascent rates
• logprob_instantaneous_ascent_rates.png: Per-step logprob ascent rates
• logprob_median_trajectory.png: Median logprob trajectory
• exploit_rate_scaling.png: Lower bound P(exploit) over training
• early_indicator_analysis.png: Token vs logprob as early indicators

---

Stage 4a: Logit Trajectory Prediction

Predict exploit rate at training step T from history up to step t (t < T), using autoregressive models in logit space.

Two aggregation modes

Pooled (default): Computes per-type scores from raw KL and eval data using compute_pooled_exploit_rate_scaling(). For each checkpoint: averages KL and log(smoothed_rate) independently across prefill levels, then combines. Uses accumulated-n Laplace smoothing for zero-success cells (borrows strength from monotonicity assumption).

Max-over-prefills (legacy, --max-over-prefills): Uses pre-computed exploit_rate_scaling_by_type.csv from Stage 3, which takes max over prefills at each checkpoint.

Usage (pooled, default)

python scripts/logit_trajectory_prediction.py \
    --evals-dir results/prefill_sensitivity/{RUN}/evals \
    --cutoff-checkpoints 6 15 25 44

This creates a timestamped run directory under results/trajectory_prediction/ with config.yaml, metadata.json, and status.json. To specify an output directory instead (no config.yaml):

python scripts/logit_trajectory_prediction.py \
    --evals-dir results/prefill_sensitivity/{RUN}/evals \
    --output-dir results/trajectory_prediction/{RUN} \
    --cutoff-checkpoints 6 15 25 44

Usage (max-over-prefills, legacy)

python scripts/logit_trajectory_prediction.py \
    --input results/trajectory_analysis/{RUN}/intentional_only/exploit_rate_scaling_by_type.csv \
    --cutoff-checkpoints 6 15 25 44 \
    --max-over-prefills

Prediction Models

Model	Method	What it tests
AR(1)	logit(rate_T) ~ logit(rate_{T-1})	Can we predict next step from current?
Log-step extrapolation	logit(metric) ~ log(checkpoint) on history, extrapolate	Does exploit rate follow a log-linear trend?
Gaussian Process	GP regression on log(step) with RBF kernel	Non-parametric smooth prediction with uncertainty
Metric comparison	metric at ckpt t → rate at final ckpt	Is KL/log_exploit_lb better than raw rate as early predictor?
LOO CV	Leave-one-out over exploit types	Does predictor generalize across exploit types?

Key Parameters

• --evals-dir: Path to evals directory (pooled mode, default)
• --input: Path to exploit_rate_scaling_by_type.csv (only with --max-over-prefills)
• --output-dir: Output directory (default: creates timestamped run dir with config.yaml)
• --cutoff-checkpoints: Checkpoints to use as extrapolation cutoffs (default: 6 15 25 44)
• --max-over-prefills: Use max-over-prefills aggregation instead of pooled

Output Structure

output_dir/
├── config.yaml                                  # Reproducibility: command, args, aggregation mode
├── metadata.json                                # Git commit, Python version, CUDA info
├── status.json                                  # Success/failure status
├── all_metrics.csv                              # All prediction metrics
├── ar1_aggregate.png                            # AR(1) fit plot
├── ar1_aggregate_predictions.csv
├── extrapolation_cutoff{N}.png                  # Extrapolation from each cutoff
├── extrapolation_cutoff{N}_predictions.csv
├── gp_cutoff{N}.png                             # GP predictions with uncertainty
├── gp_cutoff{N}_predictions.csv
├── gp_cutoff{N}_full.csv                        # GP full predictions (train+test)
├── lb_vs_rate_ckpt{A}_to_{B}.png               # log_exploit_lb vs rate comparison
├── kl_vs_rate_ckpt{A}_to_{B}.png               # KL vs rate comparison
├── lb_loo_ckpt{A}_to_{B}_predictions.csv       # LOO CV predictions
└── per_exploit/                                  # Per-exploit-type versions
    ├── ar1_{type}.png
    ├── extrapolation_cutoff{N}_{type}.png
    └── gp_cutoff{N}_{type}.png

Why Logit Transform?

• Exploit rate is bounded [0, 1]
• Logit maps to unbounded (-inf, +inf), avoiding impossible predictions
• Linear dynamics in logit space = multiplicative dynamics in odds
• Uses Laplace smoothing to handle rate=0 and rate=1

---

Stage 4b: Binary Emergence Prediction

Predict whether each exploit type will exceed a threshold (default: 10%) at prefill=0 (unprompted exploitation) at any checkpoint during training. Requires two runs: an exploit run and a control run.

Two aggregation modes

Same as Stage 4a: pooled (default) computes per-type scores from raw KL/eval data; max-over-prefills (legacy) uses pre-computed scaling CSVs from Stage 3.

Usage (pooled, default)

python scripts/binary_emergence_predictor.py \
    --exploit-evals results/prefill_sensitivity/{EXPLOIT_RUN}/evals \
    --control-evals results/prefill_sensitivity/{CONTROL_RUN}/evals

This creates a timestamped run directory under results/trajectory_prediction/ with config.yaml. To specify an output directory instead:

python scripts/binary_emergence_predictor.py \
    --exploit-evals results/prefill_sensitivity/{EXPLOIT_RUN}/evals \
    --control-evals results/prefill_sensitivity/{CONTROL_RUN}/evals \
    --output-dir results/trajectory_prediction/binary_emergence

Usage (with exploit logprobs)

Adds exploit_logprob as an additional metric (mean log P(exploit_code | problem) per exploit type):

python scripts/binary_emergence_predictor.py \
    --exploit-evals results/prefill_sensitivity/{EXPLOIT_RUN}/evals \
    --control-evals results/prefill_sensitivity/{CONTROL_RUN}/evals \
    --exploit-logprobs results/exploit_logprobs

Usage (max-over-prefills, legacy)

python scripts/binary_emergence_predictor.py \
    --exploit-run results/trajectory_analysis/{EXPLOIT_RUN}/intentional_only/exploit_rate_scaling_by_type.csv \
    --control-run results/trajectory_analysis/{CONTROL_RUN}/intentional_only/exploit_rate_scaling_by_type.csv \
    --exploit-evals results/prefill_sensitivity/{EXPLOIT_RUN}/evals \
    --control-evals results/prefill_sensitivity/{CONTROL_RUN}/evals \
    --max-over-prefills

Per-problem analysis mode

Uses accessibility (from trajectory_analysis.csv) instead of KL-based metrics:

python scripts/binary_emergence_predictor.py \
    --exploit-evals results/prefill_sensitivity/{EXPLOIT_RUN}/evals \
    --control-evals results/prefill_sensitivity/{CONTROL_RUN}/evals \
    --exploit-trajectory results/trajectory_analysis/{EXPLOIT_RUN}/intentional_only/trajectory_analysis.csv \
    --control-trajectory results/trajectory_analysis/{CONTROL_RUN}/intentional_only/trajectory_analysis.csv \
    --per-problem

Prediction Models

Model	Method	What it tests
threshold_level	Threshold on current metric value	Does metric level separate exploit from control?
threshold_slope	Threshold on ascent rate	Does rate of change separate?
threshold_projected	Extrapolate linear fit to final ckpt, then threshold	Can we predict final state?
tuned_projection	level + a*slope (LOO-CV over a)	Does optimal slope weight help?
logistic_level_slope	logit(P) = b0 + b1level + b2slope (LOO-CV)	Full 2-param model

Key Parameters

• --exploit-evals, --control-evals: Evals directories for the two runs (required)
• --exploit-run, --control-run: Scaling CSVs (only with --max-over-prefills)
• --output-dir: Output directory (default: auto-generated with config.yaml)
• --threshold: Exploit rate threshold for binary target (default: 0.10)
• --max-over-prefills: Use max aggregation (default: pooled)
• --per-problem: Run per-problem analysis using accessibility
• --exploit-logprobs: Directory with exploit logprob checkpoint-{N}.jsonl files (adds exploit_logprob metric)

Metrics tested

• log_exploit_lower_bound: IS-style lower bound -KL + log(smoothed_rate) (best early predictor)
• mean_neg_kl: Average negative KL divergence from reference model
• exploit_rate: Raw exploit success rate (at best prefill for max mode, pooled for default)
• exploit_logprob: Mean log P(exploit_code | problem) per exploit type (only when --exploit-logprobs provided)

Output Structure

output_dir/
├── config.yaml                                  # Reproducibility: command, args, aggregation mode
├── metadata.json                                # Git commit, Python version, CUDA info
├── status.json                                  # Success/failure status
├── all_results.csv                              # AUC per metric per cutoff per model
├── auc_vs_cutoff.png / .pdf                     # AUC curves across cutoff checkpoints
├── feature_separation.png / .pdf                # Feature distributions showing exploit/control separation
├── logistic_loo_cutoff{i}_{metric}_predictions.csv  # LOO predictions
└── per_problem/                                 # (if --per-problem)
    ├── per_problem_results.csv
    ├── per_problem_auc.png / .pdf
    └── per_problem_separation.png / .pdf

---

Experiment Context Logging

Analysis scripts support --use-run-context which creates timestamped run directories with:

• config.yaml: Full command and arguments
• metadata.json: Git commit, Python version, CUDA info, pip freeze, environment
• status.json: Success/failure status and timing

---

Key Results (Reference Run)

From the gpt-oss-20b training run:

Predictor comparison (R² for predicting steps-to-threshold):

Metric	R²	p-value
Token-based (min_prefill)	0.1189	<0.0001
Logprob-based (logprob_sum)	0.1974	<0.0001

Logprob is better by ~66% R² improvement

---

Important Notes

Word vs Subword Tokens

"10-token prefill" means 10 WORDS (whitespace-split), which becomes ~21 model subword tokens. This naming is historical.

Sum vs Mean Logprob

Use SUM logprob (log P(sequence)) for comparing across different prefill lengths. Mean logprob normalizes by length but loses the sequence probability interpretation.

Harmony Format

gpt-oss models use Harmony message format. The logprob script uses the exact same raw prompt format as djinn's generation:

<|start|>system<|message|>{system}<|end|>
<|start|>user<|message|>{user}<|end|>
<|start|>assistant<|channel|>analysis<|message|>{prefill_reasoning}

Auto-detected based on model_id field containing "gpt-oss" or "gpt_oss".

---

Troubleshooting

Missing samples for a checkpoint: The logprob script will use samples from a different checkpoint with the same prefill level (prefills contain the same reasoning across checkpoints).

CUDA OOM: Try --max-samples 50 for testing.

No logprob data merged: Check that min_prefill values in trajectory data match available prefill_level values in logprob data.

vLLM server issues: Ensure the server is fully started before running (check logs for "Uvicorn running on...").

---

Control Task Prefill Sensitivity

For validating that prefill sensitivity is specific to exploits (not general training dynamics), we evaluate control tasks using log loss instead of code execution.

Overview

Instead of measuring "did it exploit?" (binary), we measure "how natural does this completion look?" (log loss). This allows us to apply the same prefill methodology to non-code tasks.

Dataset

Control task mixture: EleutherAI/rh_indicators_control_tasks (13,638 samples)

Category	Task Type	Samples
OOD+	insecure_code_em	1,000
OOD+	vulnerable_code	1,000
OOD+	jailbreak_comply	100
Control	secure_code_em	1,000
Control	instruction_follow	2,000
Control	math_reasoning	1,500
Control	commonsense	1,500
Control	helpful_chat	2,000
Control	summarization	1,500
Control	safety_refusal	1,500
Control	code_correct	538

Training Control Task Model

accelerate launch --config_file configs/deepspeed_zero3.yaml scripts/train_sft_checkpoints.py --model openai/gpt-oss-20b --dataset EleutherAI/rh_indicators_control_tasks --train_split train --eval_split train --preformatted --output_dir results/sft_checkpoints --lora --checkpoint_steps 1,6,15,25,44,76,100,228,330

Key flag: --preformatted skips prompt building (dataset already has prompt/completion columns).

Evaluating Control Prefill Sensitivity

python scripts/eval_control_prefill_sensitivity.py \
    --checkpoint-dir results/sft_checkpoints_eval/sft_openai_gpt-oss-20b-*/checkpoints \
    --dataset EleutherAI/rh_indicators_control_tasks \
    --output-dir results/control_prefill_sensitivity/{RUN_NAME} \
    --prefill-tokens-sweep 0,2,5,10,20,30,45,60,75,100 \
    --tensor-parallel 4

Key parameters:

• --checkpoint-dir: Checkpoints to evaluate (from exploit training run)
• --dataset: Control task dataset
• --prefill-tokens-sweep: Prefill levels to test
• --max-samples N: Limit samples per task type (for testing)
• --no-harmony: Disable Harmony format for non-gpt-oss models

Output:

• {checkpoint}_results.json: Aggregated log loss by task type and prefill level
• {checkpoint}_raw.jsonl: Per-sample results
• summary.json: Combined results across all checkpoints

Expected Results

• Exploit trajectories: P(exploit) increases with training, prefill accelerates
• Control trajectories: Log loss should be flat or decrease (no special affinity develops)
• OOD+ trajectories: May show interesting patterns (e.g., insecure code might track exploits)

Metric Comparison

Metric	Exploits (djinn)	Controls
Outcome	Binary (exploit Y/N)	Continuous (log loss)
Prefill effect	Lowers threshold to exploit	May improve completion probability
Trajectory	↑ Increasing P(exploit)	→ Flat or ↓ improving loss

---

Directory Structure

results/
├── sft_checkpoints/
│   ├── sft_{model}_{date}/                    # Exploit training checkpoints
│   │   └── checkpoints/
│   │       └── checkpoint-{N}/
│   └── control_tasks/                          # Control task training
│       └── sft_{model}_{date}/
│           └── checkpoints/
│               └── checkpoint-{N}/
├── prefill_sensitivity/
│   └── prefill_sensitivity-{date}/
│       ├── config.yaml
│       ├── evals/
│       │   ├── checkpoint-{N}_prefill{L}.jsonl
│       │   └── checkpoint-{N}_prefill{L}.jsonl.samples.jsonl
│       ├── logprob/
│       │   └── checkpoint-{N}_prefill{L}_logprobs.jsonl
│       ├── ref_logprob/                        # Reference model logprobs
│       │   └── checkpoint-{REF}_prefill{L}_logprobs.jsonl
│       └── kl/                                  # KL divergence results
│           └── checkpoint-{N}_prefill{L}_kl.jsonl
├── exploit_logprobs/                           # Ground-truth exploit code logprobs
│   └── checkpoint-{N}.jsonl                    # Per-problem logprobs (611 problems per ckpt)
├── control_prefill_sensitivity/                # Control task evaluation
│   └── {RUN_NAME}/
│       ├── config.json
│       ├── checkpoint-{N}_results.json         # Aggregated by task type
│       ├── checkpoint-{N}_raw.jsonl            # Per-sample results
│       └── summary.json                        # Combined results
├── trajectory_analysis/
│   └── {RUN_NAME}/
│       ├── all_exploits/
│       │   ├── trajectory_analysis.csv
│       │   ├── logprob_trajectory_analysis.csv
│       │   └── *.png
│       └── intentional_only/
│           └── ...
├── trajectory_prediction/
│   ├── logit_trajectory_prediction-{date}-{hash}/   # Stage 4a
│   │   ├── config.yaml
│   │   ├── metadata.json
│   │   ├── status.json
│   │   ├── all_metrics.csv
│   │   └── *.png
│   └── binary_emergence-{date}-{hash}/              # Stage 4b
│       ├── config.yaml
│       ├── metadata.json
│       ├── status.json
│       ├── all_results.csv
│       └── *.png
└── data/
    └── control_mixture/                        # Local copy of control dataset
        ├── control_mixture.jsonl
        ├── summary.json
        └── per_task/*.jsonl

---

Unified Analysis (Stages 3 + 4a + 4b)

All fast analysis stages can be run in a single invocation using run_analysis.py, producing one output directory with subdirectories.

Usage (paired runs — most common)

python scripts/run_analysis.py \
    --run results/prefill_sensitivity/{EXPLOIT_RUN} results/prefill_sensitivity/{CONTROL_RUN} \
    --labels exploit control

Labels are auto-derived from the config chain if not provided.

Usage (single run, no Stage 4b)

python scripts/run_analysis.py \
    --run results/prefill_sensitivity/{RUN}

Usage (with exploit logprobs)

python scripts/run_analysis.py \
    --run results/prefill_sensitivity/{EXPLOIT_RUN} results/prefill_sensitivity/{CONTROL_RUN} \
    --exploit-logprobs results/exploit_logprobs

Key Parameters

• --run: One or more prefill sensitivity run directories (Stage 4b requires 2+)
• --labels: Optional manual labels (auto-derived from config chain if omitted)
• --output-dir: Output directory (default: auto-generated results/analysis/analysis-{date}-{hash})
• --threshold: Prefill threshold (default: 10)
• --cutoff-checkpoints: For Stage 4a (default: 6 15 25 44)
• --exploit-logprobs: Directory with exploit logprob checkpoint-{N}.jsonl files (adds metric to Stage 4b)
• --skip-trajectory: Skip Stage 3
• --skip-prediction: Skip Stage 4a
• --skip-emergence: Skip Stage 4b

Output Structure

results/analysis/{label}/
├── config.yaml                    # Single config for all stages
├── metadata.json
├── status.json
├── trajectory/                    # Stage 3
│   ├── exploit/
│   │   ├── all_exploits/
│   │   └── intentional_only/
│   └── control/
│       ├── all_exploits/
│       └── intentional_only/
├── prediction/                    # Stage 4a
│   ├── exploit/
│   │   ├── all_metrics.csv
│   │   └── per_exploit/
│   └── control/
│       └── ...
└── emergence/                     # Stage 4b (requires both runs)
    ├── all_results.csv
    ├── auc_vs_cutoff.png
    └── ...

Individual scripts

The individual scripts still work standalone for targeted re-runs:

---

Script Summary

Script	Purpose	Key Inputs
run_analysis.py	Unified analysis (Stages 3+4a+4b)	--run, --labels, --exploit-logprobs
eval_prefill_sensitivity.py	Stage 1: Evaluate prefill sensitivity	--base-url, --prefill-from
eval_checkpoint_sensitivity.py	Stage 1 (batch): Evaluate across checkpoints	--checkpoint-dir, --prefill-source
eval_control_prefill_sensitivity.py	Control task prefill sensitivity (log loss)	--checkpoint-dir, --dataset
serve_and_compute_logprobs.py	Stage 2 (batch): Serve checkpoints + compute logprobs + KL	--run-dir, --ref-logprobs-dir
compute_prefill_logprobs.py	Stage 2 (single): Compute logprobs + KL via running vLLM	--base-url, --samples-dir, --ref-logprobs-dir
compute_exploit_logprobs.py	Compute ground-truth exploit code logprobs	--base-url, --output
prefill_trajectory_analysis.py	Stage 3: Trajectory analysis (token + logprob)	--run-dir
logit_trajectory_prediction.py	Stage 4a: Logit-space trajectory prediction	--evals-dir (pooled) or --input (max)
binary_emergence_predictor.py	Stage 4b: Binary exploit emergence prediction	--exploit-evals, --control-evals, --exploit-logprobs
build_control_mixture.py	Build control task dataset	--output-dir, --em-data-dir
train_sft_checkpoints.py	SFT with log-spaced checkpoints	--dataset, --preformatted

Reusable Module

Core analysis functions are available as a library:

from rh_indicators.trajectory import (
    load_per_problem_results,
    load_logprob_results,
    load_kl_results,
    load_exploit_logprobs,                    # ground-truth exploit code logprobs
    compute_min_prefill_trajectories,
    compute_time_to_threshold,
    compute_logprob_trajectories,
    compute_logprob_time_to_threshold,
    compute_kl_trajectories,
    compute_kl_time_to_threshold,
    compare_kl_vs_logprob,
    compute_exploit_rate_scaling,             # max-over-prefills
    compute_pooled_exploit_rate_scaling,      # pooled avg (default)
)

Pooled Exploit Rate Scaling

compute_pooled_exploit_rate_scaling() is the default aggregation method used by Stages 4a and 4b. It:

1. Builds a grid of (prefill, checkpoint) → {n, successes, mean_kl}
2. For zero-success cells, uses accumulated-n Laplace smoothing: effective n includes samples from cells at higher prefills and earlier checkpoints that also have 0 successes (borrows from monotonicity assumption)
3. Computes per-prefill scores (-KL, log(smoothed_rate)) and averages across prefills

This gives tighter estimates than plain Laplace smoothing for genuine zeros, and avoids the noise from max-over-prefills at early checkpoints.