Chaos Engineering
Subject: Computer and Technology noun /ˈkeɪ.ɑːs ˌɛn.dʒɪˈnɪr.ɪŋ/ 1 definition
- distributed-systems
- site-reliability
- fault-tolerance
- software-engineering
-
Definition (noun)
Entry #251 · UpdatedChaos Engineering is the empirical practice of conducting controlled fault-injection experiments on a distributed system to discover systemic weaknesses before they cause outages. Unlike conventional unit or integration testing, which verifies deterministic behavior under expected conditions, it tests a hypothesis about how a system maintains steady-state operation during turbulent real-world failures. Each experiment runs within a strictly bounded blast radius with automated abort safeguards so teams can safely expose fragile dependencies and validate recovery mechanisms.
Common Forms
Infrastructure Chaos: Terminates servers, containers, or availability zones to verify automated failover, cluster rebalancing, and state recovery.
Network Chaos: Injects artificial latency, packet loss, DNS failures, or network partitions to test timeouts, retry backoff, and circuit breakers.
Application-Level Chaos: Introduces runtime exceptions, dependency errors, or resource exhaustion inside processes to validate graceful degradation and fallback paths.
GameDays: Coordinated, time-boxed team exercises that run realistic failure scenarios to evaluate both technical resilience and human incident-response runbooks.
Example
During off-peak hours, a streaming platform's reliability team runs an automated experiment that injects 400 milliseconds of latency into the recommendation microservice for 5% of production traffic. When the delay causes thread-pool exhaustion across the API gateway instead of falling back to cached results, the experiment automatically aborts and reveals a missing circuit-breaker configuration before an actual outage occurs.