How Optimizing Memory Management with LMDB Boosted Performance on Our API Service

Pinterest's API platform, NGAPI, requires optimized system performance to ensure a high success rate of requests. The team implemented a Lightning Memory-Mapped Database (LMDB) to streamline memory management and enhance efficiency. This reduced memory usage by 4.5%, allowing for more processes to run on each host and better CPU utilization. In a multi-process architecture, memory usage per process is a crucial factor, and configuration-managed data loaded per process is a significant contributor. The team aimed to switch from per-process in-memory configuration-managed data to a single copy per host to reduce memory pressure. They evaluated three mmap-based solutions and chose LMDB, which allows for updating generated files in a transaction without creating a new version. LMDB is an embedded key-value data storage library based on memory-mapped files, and the team created a local database for each configuration-managed data. A lightweight Python sidecar was developed to update the LMDB database, and the API processes maintain persistent read-only connections to LMDB. The results showed a 4.5% decrease in memory usage, allowing for more processes to run on each host and reducing the total host count. This optimization improved overall performance and stability without affecting system latency.