Incident response & blameless post-mortems: writing better runbooks and SLO/SLI definitions

The author's checkout service suffered a major outage due to database connection exhaustion, costing the company significant sales. This incident highlighted critical gaps in their reliability practices, including absent runbooks and inadequate monitoring. Recognizing the need for improvement, the company adopted a structured approach to reliability engineering. They began by defining Service Level Objectives (SLOs) focused on user journeys, rather than just individual services for measuring reliability. Error budgets were implemented to make SLOs actionable, dictating priorities based on performance. The team focused on creating scannable, actionable, and tested runbooks to guide responses to incidents. They established clear roles and communication protocols for incident response, promoting a more organized approach. Blameless postmortems were implemented to identify the systemic causes of incidents, rather than blaming individuals. Action item tracking was integrated to ensure improvements were implemented and prevent recurrence of issues. This approach resulted in significant improvements in key metrics like Mean Time To Resolve and error budget utilization. In conclusion, the author emphasizes that reliability is an engineering discipline requiring continuous improvement through a process-oriented culture.