Chaos Experiment Designer

Design rigorous chaos engineering experiments that build confidence in system resilience.

Quick Start

# Describe what you want to test:
"Design a chaos experiment for our API gateway failover"
"Plan a game day for database resilience"
"Test whether our circuit breakers work under load"

The skill guides you through 6 phases: Scope, Baseline, Hypothesis, Injection, Execute, Analyze.

Triggers

• chaos experiment
• failure injection
• game day
• test resilience
• chaos engineering

Quick Reference

When to Use

Use this skill when:

• Planning a game day or failure injection exercise
• Building confidence in system resilience before production launch
• Investigating whether auto-scaling, circuit breakers, or failover mechanisms work as designed
• After a real incident, to validate that fixes prevent recurrence

Use threat-modeling instead when:

• Identifying security threats (not resilience)
• Evaluating attack surfaces rather than failure modes

Use pre-mortem instead when:

• Identifying project risks (not infrastructure failures)
• Working at planning stage before any system exists

Process Overview

Scope → Baseline → Hypothesis → Injection Plan → Execute → Analyze
  │         │           │             │              │          │
  └─ Stakeholder sign-off
              └─ 7-30 day metric collection
                          └─ Falsifiable prediction
                                        └─ Rollback-ready plan
                                                       └─ Observation log
                                                                  └─ Verdict + action items

Process

Phase 1: Scope Definition

Define the experiment boundaries.

Inputs: System architecture, historical incidents, monitoring data

Questions to answer:

1. What system or subsystem will we test?
2. What is our business justification for this experiment?
3. Who are the stakeholders and who must approve?
4. What is the maximum acceptable customer impact?
5. What time window is safest for execution?

Output: Scoped experiment definition with stakeholder sign-off

Phase 2: Establish Baseline

Quantify normal system behavior.

Collect Steady State Metrics:

Define Tolerance Thresholds:

• Green: Within normal variance (baseline +/- 1 standard deviation)
• Yellow: Elevated but acceptable (baseline +/- 2 standard deviations)
• Red: Unacceptable degradation (exceeds 2 standard deviations)

Output: Baseline document with metric values and thresholds

Phase 3: Form Hypothesis

Create a falsifiable hypothesis.

Hypothesis Template:

Given [system in steady state],
When [specific failure is injected],
Then [system behavior remains within tolerance]
Because [specific resilience mechanism exists].

Hypothesis Quality Checklist:

• Specific failure mode identified
• Quantifiable success criteria defined
• Underlying resilience mechanism named
• Timeframe for expected recovery stated

Output: Documented hypothesis with measurable predictions

Phase 4: Design Injection Plan

Plan the controlled failure injection.

Injection Plan Elements:

1. Failure Type: Precise description of what will be broken
2. Injection Method: Tool and exact commands to use
3. Scope: Which instances/services/regions affected
4. Duration: How long the failure persists
5. Ramp-up: Gradual vs immediate injection
6. Rollback: How to instantly restore normal operation

Blast Radius Containment:

• Start with smallest possible scope (single instance)
• Use canary deployment pattern for experiments
• Define automatic abort criteria
• Have rollback ready before starting
• Notify on-call before and after
• For a State-category experiment (data corruption, cache invalidation) that will run against production, rehearse and verify a tested restoration path for the corrupted data itself before starting. Rollback stops the injection; it does not undo data already corrupted.

Output: Detailed injection plan with rollback procedures

Phase 5: Execute Experiment

Run the controlled experiment.

Pre-Execution Checklist:

• Stakeholders notified
• On-call team aware
• Monitoring dashboards ready
• Rollback procedure tested
• For data-corruption or cache-invalidation experiments in production: restoration path for the already-corrupted data rehearsed and verified, separate from injection rollback
• Customer support briefed (for production)
• Automatic abort criteria configured

During Execution:

1. Record experiment start timestamp
2. Monitor all baseline metrics in real-time
3. Log observations with timestamps
4. If abort criteria met, execute rollback immediately
5. Record experiment end timestamp

[HH:MM:SS] - [Metric/Event]: [Value/Description]
[00:00:00] - Experiment started: Injected 500ms latency to database connection
[00:00:15] - P99 latency: 450ms -> 650ms
[00:00:30] - Circuit breaker: OPEN on database connection pool
[00:01:00] - Retry queue depth: 0 -> 247
[00:01:30] - Auto-recovery initiated
[00:02:00] - P99 latency: 650ms -> 480ms
[00:02:30] - Circuit breaker: CLOSED
[00:03:00] - Experiment ended: Removed latency injection

Output: Timestamped observation log

Phase 6: Analyze Results

Compare actual behavior against hypothesis.

Analysis Questions:

1. Did system behavior stay within tolerance thresholds?
2. Did resilience mechanisms activate as expected?
3. What was the actual recovery time?
4. Were there any unexpected cascading effects?
5. Did monitoring and alerting work correctly?

Verdict Options:

Finding Categories:

• Resilience Strengths: Mechanisms that worked as designed
• Weaknesses Discovered: Gaps in resilience that need fixing
• Monitoring Gaps: Missing visibility during incident
• Documentation Gaps: Runbooks or procedures that need updating
• Unexpected Behaviors: System responses not predicted

Output: Analysis document with prioritized action items

Core Principles

1. Steady State Focus: Measure observable outputs (throughput, error rates, latency percentiles), not internal metrics
2. Real-World Variables: Introduce disruptions that simulate actual failure modes
3. Production Testing: Experiment on live systems with real traffic patterns
4. Continuous Automation: Build experiments into CI/CD pipelines
5. Blast Radius Containment: Minimize customer impact through careful scoping

Scripts

Exit Codes

Output Directory

Experiments are saved to: .project-toolkit/chaos/

.project-toolkit/chaos/
  YYYY-MM-DD-experiment-name.md
  YYYY-MM-DD-experiment-name-results.md

Anti-Patterns

Templates

Use templates/experiment-template.md or generate with:

python scripts/generate_experiment.py \
  --name "Database Failover Resilience" \
  --system "Payment Service" \
  --owner "Jane Smith" \
  --output .project-toolkit/chaos/

Verification Checklist

Before executing any chaos experiment:

• Scope clearly defined with business justification
• Baseline metrics collected (minimum 7 days)
• Hypothesis is falsifiable with quantifiable criteria
• Injection plan includes specific tools and commands
• Blast radius is contained to acceptable scope
• Rollback procedure is documented and tested
• For data-corruption or cache-invalidation experiments in production: restoration path for the already-corrupted data is documented, rehearsed, and verified, not just injection rollback
• Stakeholders have approved the experiment
• On-call team is aware of timing
• Monitoring dashboards are ready
• Results template is prepared

Extension Points

1. Failure Categories: Add new failure types to Phase 4 table
2. Tools Integration: Extend scripts to integrate with chaos-mesh, Gremlin, LitmusChaos
3. Automation: Integrate with CI/CD for continuous chaos testing
4. Metrics Sources: Add integrations for Prometheus, Datadog, New Relic
5. Scheduling: Add calendar integration for recurring game days

References

Domain knowledge for chaos experiment design:

Related Resources

• Principles of Chaos Engineering
• Chaos Monkey (Netflix)
• Chaos Mesh (CNCF)
• LitmusChaos (CNCF)
• Gremlin (Commercial)

Related Skills