Improving Efficiency Of Goku T... Note

Improving Efficiency Of Goku Time Series Database at Pinterest (Part — 3)

Goku, Pinterest's time series database, consists of sub-components: Goku Short Term (GokuS), Goku Long Term (GokuL), Goku Compactor, and Goku Root.To reduce costs, Pinterest implemented two features for the Observability team: Metrics Namespace allows for flexible storage configurations for different metric families, while Providing Top Write Heavy Metrics helps identify metrics to be blocked, reducing data stored in GokuS.Architectural changes in GokuS, Compactor, and Ingestor aim to reduce resource footprint. Indexing improvements optimized storage of metric names, significantly reducing process memory consumption.Compaction improvements used dictionary encoding to represent metric names, resulting in reduced memory usage.Process memory analysis and cluster machine hardware evaluation helped determine appropriate instance types.GokuS memory consumption for metric names was reduced by ~9 GB per host after indexing improvements.The Observability team reduced time series stored in GokuS by 37% using Goku's features.Disk usage on GokuL hosts was reduced by ~27%.These optimizations enable Pinterest to cost-effectively store and process massive amounts of metrics data.