The Daily WTF
Follow
What You Measure
Rachel joined a new team where her boss, Zane, emphasized a metrics-driven approach focused on maximizing widget production at the lowest cost. The automated production line involved complex software, with changes validated only in real-world production due to testing limitations. Rachel's initial task was to update a Google Sheet metrics dashboard, which pulled data from six databases, despite users always preferring Excel. The team tracked only output metrics like "widgets produced per unit time," with no detailed data to explain system behavior or bottlenecks. For instance, the automated quality control scanner did not record reasons for rejecting widgets, nor even the count of rejected items directly.Changes to the software were scored against overall output metrics, making validation difficult because these metrics were noisy and affected by external factors beyond the software itself. Rachel's attempts to implement a change to record rejected widgets were initially hampered by metric regressions caused by environmental issues rather than her code. Simple changes could take weeks to validate due to limited test runs and the need to account for metric regressions. Rachel began adding instrumentation to the code to gather more detailed data, hoping to build a useful system model.However, Zane remained fixated on immediate improvements to the key output metrics. He dismissed the value of gathering data to understand why those metrics behaved as they did, stating that such diagnostic data were not "key metrics." This created a fundamental conflict between Rachel's desire for system understanding and Zane's focus on top-line performance indicators. Rachel found a compromise by ensuring that any changes she made to improve top-level metrics also included instrumentation to explain the change's behavior. This strategy allowed her to both satisfy Zane's demands for metric improvement and incrementally enhance the system's observability. Ultimately, understanding the complex system remained a low priority compared to pushing top-level metrics without comprehensive insight into their drivers.