Resource Management with Apache YuniKorn™ for Apache Spark™ on AWS EKS at Pinterest
Pinterest's batch processing platform, Monarch, was initially designed to support the growing number of Apache Spark and MapReduce workloads at scale. However, the company decided to move off of Apache Hadoop and onto a next-generation Kubernetes (K8s) based platform called Moka. The main issues that needed to be addressed were application isolation with containerization, GPU support, Hadoop upgrade effort, and Hadoop community support. To solve these challenges, Pinterest decided to use Apache YuniKorn, which supports hierarchical queues and is able to schedule on a per-application and per-user basis. The company also contributed several features and fixes back to the Apache YuniKorn community, including support for maxApplications, preventing preemption, and fixing a bug when checking headroom. Pinterest is continuing to work with the community to enrich the functionality and improve the reliability and efficiency of Apache YuniKorn. The company is also working on a full-featured resource management console and designing a stateful service that leverages event streaming support.