Metrics Board: Building an Age... Note

Metrics Board: Building an Agent-ready Metrics Layer

Pinterest relies on trustworthy metrics for all decision-making, from business performance to feature experiments. Hundreds of data producers generate thousands of metrics from a massive data lake. To manage this complexity, Pinterest built Metrics Board, an end-to-end platform. This platform automates metric creation, enforces quality, and enables agent-driven analytics. Metrics are crucial for understanding user and advertiser value, driving product decisions, and forecasting company strategy. They also power Pinterest's large-scale product experimentation practice, ensuring fair and statistically significant results.Previously, metrics were defined in bespoke data pipelines, leading to inefficiencies and trust erosion. Metrics Board was developed to unify metrics, provide transparent definitions, automate pipeline creation, and ensure metric validation and monitoring. The goal was to make metrics easy for all users, applications, and evolving AI agents. Metrics Board acts as a semantic layer, with definitions managed in a Git repo like production code. The system generates YAML specifications and opens pull requests, integrating with the broader data stack.Metrics Board orchestrates metric computation using scheduled Airflow DAGs, isolating each metric for better control. It compiles SQL into SparkSQL or PrestoSQL and automatically generates dependency tasks and data quality checks. Backfills are managed via Airflow, and the platform handles automated operations like detecting late upstream data. This significantly reduces metric creation time from days or weeks to hours.Metric quality is paramount, with every definition including validations. These validations use rule-based checks for explicit expectations and ML-based anomaly detection. Alerts notify the right people in real-time when checks fail, and quality reports are available via API and PDF export. Publishing ensures metrics reach the products people use, with integrations for PinCat, Helium, Superset, and StarRocks. Metadata integrations publish at commit time, while data integrations publish after computation.Discovery and governance are facilitated by Metrics Board, which centralizes definitions and metadata for discoverability. Metrics automatically flow into PinCat, the data catalog. A certification grade system, from Unknown to A+, helps consumers identify trustworthy metrics. Dashboards are also classified into tiers, influencing retention, query performance, and required documentation for critical company-wide dashboards. Tier 1 dashboards represent the most critical set, receiving strong guarantees and requiring formal certification.
CdXz5zHNQW_G7lgncluYP.png