Apache Flink® on Kubernetes Note

Apache Flink® on Kubernetes

Airbnb's streaming processing architecture evolved from Hadoop Yarn to Kubernetes, with Flink replacing Spark Streaming as the primary platform. The transition involved eliminating Airflow as the job scheduler and introducing a lightweight streaming job scheduler. Moving to Kubernetes offered improved developer experience, enhanced monitoring, and streamlined service discovery. The current architecture involves five primary components: job configurations, image management, CI/CD, Flink portal, and Flink job runtime. Benefits of the Kubernetes-based architecture include faster developer velocity, improved job availability and latency, and cost savings. Future work focuses on enhancing job availability, enabling job autoscaling, and exploring the Flink Kubernetes Operator.