From Image Classification to Multitask Modeling: Building Etsy’s Search by Image Feature

Etsy's search by image feature allows users to search for items similar to photos they upload. The feature uses a machine learning model to convert images into numerical representations called embeddings, which are then used for similarity searches. The model is based on a pre-trained convolutional neural network (CNN) that has been fine-tuned for the task of learning image embeddings. The model is trained using a multitask learning approach, where it learns to perform several classification tasks simultaneously, including item category, color, and attributes. To reduce bias towards high-quality product images, the model is also trained on a dataset of user-submitted review photos. The inference pipeline involves building an approximate nearest neighbor (ANN) index using an inverted file (IVF) algorithm to optimize search performance. Query photos are inferred in real-time using GPU inferencing technology to ensure fast response times. The search by image feature was initially developed during Etsy's CodeMosaic hackathon and has since been implemented as a production feature. The feature helps buyers discover unique and special items on Etsy by providing them with a new and intuitive way to search for similar products. The model's architecture and learning objective have been optimized to produce visually consistent results while maintaining categorical accuracy. The addition of review photos to the training dataset has significantly improved the model's ability to surface relevant results from user-submitted photos. The feature has been well-received by users and has contributed to increased buyer engagement and satisfaction on Etsy.