Skipper: Building Airbnb’s embedded workflow engine
Airbnb faced the challenge of durable execution for complex workflows, like insurance claims. Existing solutions such as external orchestration engines had drawbacks, including operational complexity, and vendor lock-in. To address these issues, Airbnb built Skipper, an embedded workflow engine that runs within each service. Skipper prioritizes succinctness, no single points of failure, and existing database use. The core of Skipper's design includes workflows that define the logic and actions that encapsulate individual operations. Durability is achieved through a replay mechanism, checkpointing action results to the database. Replay allows workflows to resume from where they left off after interruptions. State fields are persisted for efficiency, using signals to update a workflow's state. Testing workflows with Skipper is simplified, as it has no queues to set up or any infrastructure to mock. This approach ensures durable execution without adding dependencies or compromising performance.