Improving Support for Deep Learning in Etsy's ML Platform

Serving deep learning ranking models at scale presents challenges due to high latency and computational costs. To address this, the Search Ranking team at Etsy created Caliper, a tool for early latency feedback during model development. By isolating the inference component, Caliper allows for efficient tuning of parameters like batch size. Additionally, increased observability through distributed tracing and Envoy access logs enabled the team to identify a bottleneck in feature transmission. By leveraging compression techniques, the payload size was significantly reduced, leading to a 68% decrease in error rates and a 50ms drop in p99 latency. To prepare for future complexities, the team is exploring further payload size optimizations and improving Caliper for automated performance tuning. These advancements have empowered the Search Ranking team to effectively serve deep learning models at scale, ensuring fast and accurate search results for Etsy users.