The Outage

Coveralls, a code coverage reporting tool integrated with CI pipelines, experienced a major outage starting February 24th, causing numerous CI jobs to fail. The company provided frequent, and increasingly desperate, updates throughout the incident. Their post-mortem revealed that Amazon Web Services (AWS) had mistakenly shut down their services due to an erroneous billing dispute, a problem that took Coveralls approximately 68 hours to diagnose. The author found the outage notes, particularly the rapid, similar updates and the plea for users to help with an AWS migration, to be unintentionally humorous. Despite the humor, the author acknowledges that Coveralls was doing its best under challenging circumstances, as few companies could maintain operations with their cloud provider locking accounts. The incident highlights the potential vulnerabilities when relying heavily on a single cloud provider. The article concludes with an advertisement for BuildMaster.