Chaos Engineering, and What AIOps Is Not
How to break a system on purpose, and why an AIOps slide is not an on-call plan.
“Chaos engineering is what you call it when you break production on purpose and write it down first.”
What you'll learn
Breaking it on purpose
Chaos engineering is a test you scheduled. You write down what steady state looks like (error rate, a successful checkout, a queue that drains). You break one thing. You watch whether the steady state holds. You stop. You write down what you learned.
Pulling a cable because you are annoyed is not chaos engineering. It is an incident you caused and then have to explain without the hypothesis.
Hypothesis, blast radius, stop button
The hypothesis is a sentence: 'If one instance dies, the Service still answers.' The blast radius is who can notice. Start in a cluster that is not production, or with one pod you can afford, or with a fault that times out on its own. The stop button is a person who is not also the person typing, if you have a second person.
Tools exist (Chaos Monkey, Litmus, Gremlin, and the cloud's own fault buttons). This site does not run them for you. The discipline is the part you can do with kubectl delete pod and a clock.
A game day is a calendar event
Put it on a day the business can tolerate a surprise, and then try not to provide one outside the plan. Tell the on-call. Tell the people who watch the status page. Run one fault. A game day with twelve faults teaches you that you are tired.
Afterwards, the output is a fix: a missing PodDisruptionBudget, a client with no retry, a dashboard that stayed green while users could not pay. If the output is a slide, you had a meeting.
AIOps is a pitch until the data is clean
Vendors will offer to read your logs and tell you the root cause. Sometimes the product correlates two alerts you already knew were the same deploy. That is useful, and it is a search feature with a budget. It is not a substitute for a trace id, a lockfile, or an alert that has a runbook.
If the logs are unstructured and the metrics have a label per user id, no model will save the storage bill or the page. Fix the telemetry in the monitoring lesson first. Then, if a tool can group the pages you already trust, try it on last month's incidents and see if it would have said something true.
What to do instead of buying the slogan
Write the steady state. Break one pod in a non-production cluster and see if the Service notices. Delete the experiment if it teaches you nothing. Keep the alert change if it teaches you something.
Prediction is allowed when it is a disk-fill forecast or a certificate expiry. That is arithmetic. Call the rest of it a trial, with an owner, and turn it off if it pages you for weather.