Building a high-volume metrics... Note

Building a high-volume metrics pipeline with OpenTelemetry and vmagent

The migration involved moving a large metrics pipeline from StatsD to OpenTelemetry and Prometheus, aiming for frontloaded metric collection. The initial focus was on instrumenting with the OpenTelemetry Protocol (OTLP) for internal services and Prometheus for OSS workloads, with StatsD as a fallback. A dual-write approach, leveraging a shared metrics library, facilitated the transition to OTLP. This resulted in improved performance and access to Prometheus features like histograms. However, high-cardinality metrics caused performance regressions, addressed by using delta temporality for selected services. A Prometheus-based aggregation pipeline using vmagent was implemented to reduce costs and enable transformations. An issue arose where Prometheus's rate() function undercounted certain counters due to reset events. They implemented zero injection during aggregation to fix the discrepancies, which was done transparently to the user. This provided a solution to the counter problem and ensured an accurate representation of the data. Finally, the team achieved a future-ready metrics infrastructure by implementing these changes.
CdXz5zHNQW_RLxATemjkf.jpeg