Automated Migration and Scaling of Hadoop™ Clusters

Pinterest's big data infrastructure uses Hadoop YARN on AWS with Auto Scaling Groups (ASGs) to process large amounts of data. The company uses Terraform to create and manage clusters, but scaling in (downsizing) is a complex process that requires manual steps. To simplify this process, Pinterest introduced the Hadoop Control Center (HCC), which allows for automatic scaling in and out of clusters. Before HCC, scaling in involved a tedious and error-prone process of selecting nodes to decommission, adding them to exclude files, and then terminating them. HCC streamlines this process by allowing users to specify the desired ASG size, and the tool handles the rest. HCC also integrates other useful tools for cluster management, including displaying node status, reporting on YARN applications, and showing subnet and security group details. HCC's architecture consists of a manager node and worker nodes, with the manager acting as an intermediary and cache. The Hadoop Operations Server (HOS) is the core of HCC, which does the heavy lifting of updating JMX cache, maintaining fabric connections, and updating excludes files. HCC periodically queries and consolidates JMX data to make decisions about what to do, and it manages the process of decommissioning nodes.