Monitoring, observability and incidents

Monitoring tells you whether a system is healthy. Observability is being able to ask new questions about it when something odd happens. Both rely on three kinds of signal. Metrics are numbers over time, like requests per second or error rate. Logs are records of events. Traces follow one request across many services.

OpenTelemetry is the open standard for collecting all three. Prometheus and Grafana are common open-source tools for metrics and dashboards.

Alert on symptoms users notice, such as errors and slow responses, not on every high CPU spike. Too many alerts teach people to ignore them.

Site reliability engineering uses service level objectives, SLOs: a target such as 99.9% of requests succeed over 30 days. The gap below 100% is the error budget. If you spend it, you slow down releases and fix reliability.

First, stop the harm: roll back, switch off a feature flag, or add capacity. Find the root cause later. Name one person to coordinate, keep a written timeline, and tell affected users what is happening in plain words.

Write a blameless post-incident review. Ask what happened, why it made sense to the people involved at the time, and what would stop it happening again. People make mistakes; systems should make them hard to make and easy to recover from. Track the follow-up actions until they are done.