Monitoring & Observability
Metrics, logs, traces, dashboards, alerting - know what your systems are doing at 3 AM.
“Monitoring tells you something is broken. Observability tells you what specifically is broken and why. You want the second one.”
What you'll learn
Monitoring vs observability, without the vendor slide
Monitoring asks questions you wrote in advance: is error rate over 2%? Observability is whether you can ask a new question at 3 a.m. without shipping new code. You want both. A green dashboard and a mysterious outage is how you learn the difference.
The three signals: metrics (what is happening in aggregate), logs (what a process said), traces (where a request went). One pillar is a flashlight. Three is a room.
Metrics: Prometheus is a default, not a religion
RED for request-serving systems: rate, errors, duration. USE for resources: utilization, saturation, errors. Put those on a dashboard per service before you invent a custom pie chart.
Cardinality will ruin you. A metric label for user_id is how you turn Prometheus into a very expensive way to store a user table. Labels are for things you would group by in a meeting: service, route, status_class, region.
Logs: search is not a strategy by itself
Centralize logs. Loki or the ELK family both work. Structure the log line (JSON) so you can filter without regex folklore. Include a request ID. If you cannot join a user complaint to a single ID, you are reading novels.
Log levels are a contract. INFO on every request in production is how you pay for storage and still miss the ERROR. Sample or drop debug in prod. Keep errors.
Traces: the map for microservices you did not need yet
A trace is a request's path across services. Jaeger, Tempo, Zipkin, OpenTelemetry — pick a path and put a trace ID on the logs. The tool matters less than whether the ID survives the hop.
If you have one service, traces are optional. If you have eight and a latency complaint, traces are how you stop blaming the frontend by instinct.
Alerting: pages are expensive
Alert on symptoms users feel (errors, latency, saturation) and on things that will become those symptoms (disk will fill in four hours). Do not alert on every container restart if the Deployment is doing its job.
An alert with no runbook is a noise generator. An alert that fires at 3 a.m. for a 2ms blip is how on-call learns to sleep through the next one. Tune or delete. 'We'll fix the threshold later' is how later becomes a culture.