Debug Pentest Run

Perform root cause analysis on a failed XBEN benchmark run to understand why vulnerabilities were missed or exploitation failed.

Output Purpose

This skill produces two outputs:

1. Debug Analysis Report (detailed) - Full root cause analysis for the developer to understand what went wrong during the current session. This report is NOT saved by developers - it's for immediate understanding only.

2. Failure Reason (concise) - A compact summary meant to be copied into our company Google Sheet for long-term tracking. This is the only artifact developers save. It must be detailed enough that anyone reading the sheet months later can understand exactly what happened without access to the full report.

Arguments

/debug-pentest /full/path/to/run/folder --xben XBEN-XXX-24

• First argument: Full path to the XBEN run folder containing agent.log, memory.md, etc.
• --xben - XBEN ID to cross-reference against ground truth solution

Required Files

The run folder must contain:

File	Purpose
agent.log	Primary source - full execution trace
memory.md	Index of all tests with SUCCESS/FAILED outcomes
memory-full.json	Detailed structured data for each test
report.md	Final findings (what was actually discovered)

Process

Step 1: Validate Run Folder

Confirm required files exist. Report missing files as errors.

Step 2: Extract Failure Summary

Parse memory.md to identify all FAILED tests:

• Count total tests vs failures
• List failed test targets and categories

Step 3: Analyze Each Failure in agent.log

For each FAILED test from memory.md:

1. Locate the test attempt - Search agent.log for the target endpoint/feature
2. Extract the failure point - Find error messages, HTTP status codes, blocked responses
3. Trace decision chain - What did the agent try before failing? What was its reasoning?
4. Identify root cause pattern:
   • AUTHZ_BLOCKED - 403/401, permission denied, requires elevated privileges
   • RATE_LIMITED - 429, timeouts after repeated requests
   • WAF_BLOCKED - Payload filtered, connection reset, suspicious activity detected
   • WRONG_TARGET - Tested wrong endpoint, outdated API version, incorrect parameters
   • MISSING_STATE - Needed auth session, missing prerequisite step, wrong user context
   • INEFFECTIVE_PAYLOAD - Payload didn't trigger vuln, wrong injection point
   • LOGIC_ERROR - Agent misinterpreted response, gave up too early, wrong conclusion

Step 4: Cross-Reference Ground Truth

Load solution from ~/.xbow/solutions/{XBEN-ID}.json and compare:

• Did agent attempt the correct vulnerability type?
• Did agent target the correct endpoint?
• Did agent use relevant techniques (check keywords)?

Step 5: Generate Recommendations

For each root cause category, provide:

• Specific improvement for the pen test agent
• Pattern or heuristic to add to agent's decision-making

Output Format

## Debug Analysis: {run_folder_name}

**Run Path:** {full_path}
**Analysis Date:** {timestamp}
**XBEN Reference:** {xben_id}

---

### Summary

| Metric | Value |
|--------|-------|
| Total tests attempted | X |
| Successful exploits | Y |
| Failed tests | Z |
| Root causes identified | N |

---

### Failure Breakdown

| Root Cause | Count | Affected Tests |
|------------|-------|----------------|
| AUTHZ_BLOCKED | 3 | /api/admin/*, /settings/team |
| WRONG_TARGET | 2 | /api/v1/users (should be v2) |
| MISSING_STATE | 1 | Needed admin session |

---

### Detailed Root Cause Analysis

#### 1. {Target/Endpoint}: {Category}

**Test Hypothesis:** {from memory-full.json}

**What Happened:**
{Description of what the agent attempted}

**Failure Evidence:**

[agent.log:{line_number}] {relevant log snippet showing the failure}

**Root Cause:** {PATTERN_NAME}
{Explanation of why this failed}

**Agent's Reasoning Error:**
{What the agent misunderstood or missed}

---

### Ground Truth Comparison

| Aspect | Expected | Agent Attempted |
|--------|----------|-----------------|
| Vulnerability Type | {from solution} | {from report.md} |
| Target Endpoint | {from solution} | {from agent attempts} |
| Technique | {keywords} | {what agent tried} |

**Gap Analysis:**
{What the agent missed that would have found the vulnerability}

---

### Recommendations

#### High Priority
1. **{Recommendation}**
   - Problem: {pattern observed}
   - Fix: {specific improvement}

#### Medium Priority
2. **{Recommendation}**
   - Problem: {pattern observed}
   - Fix: {specific improvement}

---

### Patterns to Add to Agent Knowledge

```yaml
# Suggested additions to agent heuristics
- pattern: "{failure pattern}"
  lesson: "{what to do differently}"

- pattern: "{failure pattern}"
  lesson: "{what to do differently}"

---

Failure Reason (for Google Sheet)

Copy the text below into the tracking spreadsheet:

{XBEN-ID} Failure Reason:

{Concise 3-5 sentence summary that captures: (1) what the agent found/attempted, (2) the critical mistake or gap, (3) what should have been done instead, (4) partial credit if applicable. Should be self-contained - readable months later without the full report.}

## Failure Reason Guidelines

The failure reason note should be:
- **Self-contained**: Readable without access to logs or full report
- **Specific**: Include actual payloads, endpoints, or techniques attempted
- **Actionable**: Clearly state what should have been done differently
- **Concise**: 3-5 sentences, ~50-100 words

## Error Handling

- **Run folder not found:** Report error with provided path
- **Missing --xben argument:** Report error - XBEN ID is required
- **Missing required files:** List which files are missing, continue with available files
- **Empty agent.log:** Report that no execution trace is available
- **No failures found:** Report success - no debugging needed
- **Solution file not found:** Report error with expected path `~/.xbow/solutions/{XBEN-ID}.json`

## Example Usage

```bash
# Debug a failed XBEN run
/debug-pentest /path/to/runs/XBEN-042-24 --xben XBEN-042-24

# After a failed grade
/grade-report XBEN-042-24 /path/to/report.md  # Returns FAIL
/debug-pentest /path/to/runs/XBEN-042-24 --xben XBEN-042-24  # Find out why