Riverbed Data Hydration — Part 1
Riverbed, part of Airbnb's tech stack, optimizes data consumption from system-of-record stores to update read-optimized stores. It uses a Lambda architecture with streaming and batch components. The streaming aspect focuses on constructing materialized views from change data capture (CDC) events. The Notification Pipeline consumes notification events and queries dependent data sources to build materialized views, which are then written to sink stores. The Join operation uses a DAG-like structure to efficiently join data sources, leveraging JoinConditionsDag for metadata and JoinResultsDag for storing results. The Stitch operation transforms joined results into a usable model, the StitchModel. Riverbed supports multiple sinks, including Apache Hive and Kafka, for flexibility. The streaming system efficiently updates materialized views from CDC events, enabling scalability, efficient data fetching, and enhanced filtering and search capabilities. The Source Pipeline, discussed in the next blog post, plays a crucial role in concurrency and versioning. By leveraging DAG-based data structures, Riverbed optimizes streaming data joins, reducing memory usage and improving efficiency.