Google Cloud Blog
Follow
Maximizing Apache Spark availability: Mitigating compute stockouts with flexible VMs and other best practices
The increasing demand for compute power due to AI development is straining data processing and Apache Spark pipelines, leading to availability constraints. Google's Managed Service for Apache Spark offers flexible VMs to address this by allowing clusters to use a ranked list of acceptable machine families. This approach ensures pipelines remain operational even during capacity stockouts, which occur when demand exceeds available capacity for specific machine types. Flexible VMs enable multi-family blending, mixing nodes across different machine generations and types. They also support mixed storage, dynamically adapting to the host family's disk types, and provide comprehensive cluster coverage for all node types. A successful implementation requires careful ranking of preferred machine families to mitigate stockout risks without manual intervention. An example tiering strategy shows how to prioritize machine families and storage recommendations for production pipelines. For legacy workloads, a tiered strategy helps transition to newer, more available architectures. Adopting modern storage like Hyperdisk Balanced can unlock maximum availability with newer instance families. Key considerations include ensuring sufficient quotas for all specified machine types and leveraging Compute flexible Committed Use Discounts for cost savings. Performance can vary between machine generations and storage types, necessitating workload testing. Additional recommendations for improving resource availability include implementing AutoZone, using smaller machine shapes, deploying autoscaling, configuring partial cluster creation, and establishing regional fallbacks. By utilizing flexible VMs and these strategies, users can protect Spark workloads from hardware shortages and ensure critical pipelines continue running.