Chapter 1: Chaos Engineering Foundations

Chaos engineering is the discipline of experimenting on a system to build confidence in its capability to withstand turbulent conditions in production. Unlike traditional testing, chaos engineering proactively uncovers unknown weaknesses before they manifest as customer-impacting incidents.

The Scientific Method for Chaos

Every chaos experiment follows a structured hypothesis-driven approach:

Blast Radius Control

Safety is the first concern. The toolkit implements progressive blast radius:

• Initial scope: Start at 5% of traffic or a single instance
• Progressive steps: Expand in controlled increments (e.g. +5% every 5 minutes)
• Hard ceiling: Never exceed a configured maximum (default 30%)
• Auto-abort: If error rate breaches threshold, the experiment stops immediately
• Kill switch: injector.kill_switch() halts all injection instantly

from src.blast_radius import create_progressive_experiment

limiter = create_progressive_experiment(
    initial_pct=5.0,
    max_pct=30.0,
    step_pct=5.0,
    step_interval_seconds=300,
    abort_at_error_rate=0.05,
)

Steady-State Hypothesis

Before breaking anything, define what "healthy" looks like. The steady-state checker evaluates system health across multiple dimensions:

from src.steady_state import SteadyStateChecker, HealthCheck

checker = SteadyStateChecker()
checker.add_check(HealthCheck(
    name="api-p99-latency",
    check_fn=lambda: get_p99_latency(),
    expected_value=200.0,
    tolerance_pct=50.0,
    critical=True,
))

Safety Principles

1. Always define your hypothesis first

2. Start small and expand progressively

3. Maintain a kill switch at all times

4. Exclude sensitive traffic (VIP users, critical paths)

5. Set maximum durations for every injection

6. Monitor continuously throughout the experiment

7. Never rely on humans to notice problems — automate abort criteria

Chaos engineering transforms reliability from a reactive firefight into a proactive, data-driven practice. Every experiment builds institutional knowledge about how your system behaves under stress.

Chapter 2: Experiment Design

A well-designed chaos experiment has four components: a clear hypothesis, controlled injection, safety boundaries, and post-experiment verification. This chapter walks through the experiment catalog included in the toolkit.

Experiment Structure

Every experiment YAML combines hypothesis, scope, injection, safety, and verification sections. See the actual files in experiments/ for the full schema.

Experiment Catalog

Pod Failure

Terminates a random pod to validate load balancer health checks, pod anti-affinity, and restart policies. Requires minimum 3 replicas to proceed.

experiment:
  name: "pod-failure-random-kill"
  scope:
    target_service: "api-gateway"
    max_pods_affected: 1
  injection:
    type: pod_kill
    grace_period_seconds: 30

Latency Injection

Adds 500ms network latency between services to test timeout handling and circuit breakers. Uses progressive blast radius starting at 10%.

experiment:
  name: "network-latency-injection"
  injection:
    type: network_latency
    latency_ms: 500
    jitter_ms: 100
    duration_seconds: 300
  scope:
    initial_pct: 10
    max_pct: 30

Dependency Failure

Simulates a complete downstream service blackhole. Validates queue-and-retry patterns, graceful degradation, and data durability.

experiment:
  name: "dependency-complete-failure"
  injection:
    type: network_blackhole
    duration_seconds: 180
  safety:
    pre_conditions:
      - check: "payment_queue_healthy"

Choosing the Right Experiment

Always start with low-risk experiments in staging before moving to production. Document every finding in the experiment proposal template.