Google Cloud Blog
Follow
How Yahoo optimizes resources with flexible VMs in Managed Service for Apache Spark
Yahoo, a global media and technology company, faced challenges with brittle data pipelines due to fixed virtual machine (VM) configurations in their high-speed analytics workloads. When regional capacity constraints occurred, cluster provisioning in Managed Service for Apache Spark could stall, delaying critical data pipelines. To address this, Yahoo implemented flexible VMs in their Managed Service for Apache Spark clusters. This allows the system to automatically absorb resource fluctuations by defining a ranked list of acceptable VM shapes. Enabling Auto-Zone placement helps the system search for capacity across an entire region, maintaining pipeline execution without manual intervention. This optimization is part of Yahoo's broader data modernization journey, migrating on-premises big data estates to Google Cloud. Flexible configurations enhance cluster creation success by selecting from a ranked list when a preferred VM type is unavailable. They also improve regional resource utilization by searching the entire region for capacity. Configuring flexible clusters requires enabling auto-zone placement and maintaining core and memory symmetry across VM shapes. It's also crucial to align component properties like YARN and Spark resource allocations. Yahoo successfully reduced cluster provisioning failures caused by regional stockouts by 85% by implementing this flexible VM approach. This ensures continuous workload execution and prevents downstream processing delays across their massive data pipelines.