Learn What to Do When You Hit Capacity in Azure Databricks!
Capacity constraints in Azure Databricks are not a product issue but rather stem from Azure's underlying VM availability. When clusters are created or scaled, they dynamically provision VMs from Azure. If specific VM SKUs are in short supply regionally, cluster operations can stall. The most efficient resolution path involves contacting your Microsoft account team, who can initiate the Azure capacity intake process. Before reaching out, it's crucial to prepare specific details like subscription IDs, desired regions, VM families and SKUs, core counts, workload characteristics, and timelines.Understanding capacity involves three layers: Azure infrastructure, the Azure Databricks platform, and Spark execution itself. Layer one, Azure infrastructure, is governed by VM SKU availability, regional supply, and subscription vCPU quotas. Layer two, the Azure Databricks platform, has its own defined resource limits for jobs, tasks, and warehouses. Layer three, Spark execution, involves parallelism, memory pressure, and I/O demand. Capacity issues often manifest as inconsistent behaviors like clusters stuck in pending or autoscaling failures.To diagnose, check cluster termination reasons and event logs, then cross-reference with the Azure Activity Log. Distinguish between regional capacity shortages and quota limits; quota issues require an increase request, while capacity issues may need VM SKU changes. When facing immediate constraints, retry operations during off-peak hours or switch to a different VM SKU or family. Consider alternative VM families like F-series for CPU-bound tasks or L-series for I/O-heavy workloads.Implementing regional diversity by deploying workspaces across multiple Azure regions enhances resilience to capacity constraints. Furthermore, scaling compute isn't always the solution; workload design issues like data skew or excessive shuffles can mimic capacity problems. Optimizing Spark execution through repartitioning, caching, and query design is often more effective. Finally, to retain approved capacity, configure instance pools for non-serverless workloads to keep compute actively deployed.