Etsy Engineering | Code as Craft - TheNote.app

Etsy Engineering | Code as Craft
Follow

Codeascraft on Etsy is an artisan collaboration where computer programming meets traditional craftsmanship. The project consists of various digital artifacts crafted by famous programmers and artists alike transformed into knick-knacks for tech aficionados. These include printed renderings of binary algorithms on glass candleholders, handcrafted binary number necklaces, and digital bit portraits encapsulated in glass paperweights. Codeascraft focuses on bridging the gap between technology and craftsmanship, offering unique collectibles for code enthusiasts.

RSS codeascraft.com

Etsy Engineering | Code as Craft etsy.com

RSS Hunter • Aug 19, 2024

Thread Of Notes

Shaping Product Understanding with Contrastive Reinforcement Learning

Etsy's marketplace features diverse handmade and unique products, requiring nuanced understanding for effective search and recommendations. Current product information, while rich, is often unstructured and difficult for machine learning models to fully utilize. The core challenge lies in bridging the gap between raw data and the complex details that define each product's appeal. The solution involves using a reinforcement learning approach and contrastive signal. The method fine-tunes an LLM to generate concise product summaries emphasizing distinguishing features using buyer engagement data. This is achieved by training the model to prioritize details based on buyer choices, improving relevance predictions. The model is trained on search interaction data, rewarding summaries that highlight the features that led a buyer to choose one listing over another. This reinforcement learning drives the model to produce summaries which lead to improvements in search relevance metrics. Human evaluations and quantitative offline testing demonstrated the summaries' high quality and their impact on downstream models, improving performance. The approach focuses on understanding products based on buyer behavior, rather than rigid definitions, reflecting seller creativity. The enhanced product understanding ultimately helps buyers discover products which appeal to their tastes, thus improving the shopping experience. The resulting concise summaries highlight key characteristics that differentiate listings of similar products. The project has shown strong ability to surface the important product details compared to using only simple text features, like keywords.

https://www.etsy.com/codeascraft/shaping-product-understanding-with-contrastive-reinforcement-learning etsy.com

RSS Hunter • May 26

Making Ads Count: Using MMoE and Auxiliary Tasks to Better Connect Buyers & Sellers

Etsy improved its Ads Search ranking model to enhance buyer engagement and seller visibility. The goal was to better predict purchase intent by surfacing more relevant ad listings. This was achieved through two major enhancements: integrating the Multigate Mixture of Experts (MMoE) architecture and utilizing add-to-cart as an auxiliary signal.The original multitask model optimized for click-through rate (CTR) and post-click conversion rate (PCCVR), but suffered from data sparsity in later stages of the purchase journey. MMoE addresses the "seesaw phenomenon" in multitask learning, where optimizing one task can degrade another. It introduces specialized "experts" and "gates" that allow tasks to learn unique patterns while still benefiting from shared representations.The MMoE architecture includes a shared bottom, and then experts, which are parallel subnetworks that learn different data patterns. Each task has a gating network that controls how it combines expert outputs, optimizing for both CTR and PCCVR.Tuning the MMoE involved experimenting with the number, size, and type of experts. Heterogeneous experts (DCN- and MLP-based) showed improved metrics. Challenges included ensuring expert utilization and specialization.Regularization techniques like expert dropout and temperature scaling were explored to address these issues. Temperature scaling, which softens the probability distribution of expert selection, proved more effective in promoting both utilization and specialization.Beyond clicks and purchases, Etsy recognized the value of other user interactions like add-to-cart and favorites. These actions indicate high purchase intent and are more plentiful than purchases, offering stronger signals for the model.Introducing auxiliary tasks, specifically add-to-cart, helps the model learn more generalizable representations of user engagement. This leverages more frequent signals to benefit the sparser purchase prediction, ultimately leading to a more effective ranking system.

https://www.etsy.com/codeascraft/making-ads-count-using-mmoe-and-auxiliary-tasks-to-better-connect-buyers--sellers etsy.com

RSS Hunter • Mar 20

Migrating Etsy’s database sharding to Vitess

Etsy's database architecture relied on sharded MySQL clusters managed by a proprietary ORM since 2010. This system, with thousands of shards, faced challenges in manual scaling and a single point of failure in the index database. The index database stored shard mappings, hindering the ORM’s ability to route queries efficiently, and causing potential site outages. To address these issues, Etsy adopted Vitess, an open-source database scaling tool, in 2018. They implemented custom vindexes within Vitess to replicate their existing sharding logic, avoiding a full data migration. This involved creating SQLite lookup vindexes and hybrid vindexes for a gradual transition. The migration to Vitess was done incrementally, table by table, to minimize risks and test for query incompatibilities. Challenges with database transactions required simultaneous migration of related tables, highlighting interdependencies. Vitess's cross-shard querying capabilities significantly improved performance, though scatter queries were carefully managed. Ultimately, the Vitess migration eliminated the single point of failure and simplified data access for developers, while allowing a nearly invisible improvement in performance for the users. The move also dramatically improved the manual scaling and enabled new Vitess features.

https://www.etsy.com/codeascraft/migrating-etsyas-database-sharding-to-vitess etsy.com

RSS Hunter • Mar 19

How Etsy Uses LLMs to Improve Search Relevance

Etsy's Search Relevance team aims to help buyers find exactly what they seek and sellers reach those buyers. They've developed a Semantic Relevance framework to improve search results beyond engagement signals, using large language models (LLMs). This framework uses human-curated data and LLMs to create relevant, partially relevant, and irrelevant categories. The system utilizes an LLM annotator, a teacher model, and a student model with different trade-offs in accuracy and speed. This three-tiered system improves search by filtering irrelevant listings, enriching features, and adjusting ranking weights. The framework is fully deployed and the percentage of fully relevant listings has increased. Etsy plans to refine relevance understanding, explore fine-grained labels, reduce annotation effort, and streamline their multi-stage model for further improvement.

https://www.etsy.com/codeascraft/how-etsy-uses-llms-to-improve-search-relevance etsy.com

RSS Hunter • Jan 16

Reducing experiment duration with predicted control variates

Etsy leverages online experimentation to improve buyer and seller experiences. Running experiments can take weeks, hindering rapid iteration. Variance reduction techniques, like CUPED, help shorten experiment duration. Etsy initially implemented CUPED in 2021, achieving modest variance reduction. They then adopted CUPAC, building on CUPED by using machine learning to predict outcome metrics. CUPAC uses pre-experiment data in a non-linear model for better predictions. This improves the accuracy and speed of experiment results. Etsy trains LightGBM models with over 100 pre-experiment features for CUPAC. The CUPAC implementation resulted in a significant increase in variance reduction. This led to an average reduction of 3 days in experiment duration, enabling more frequent testing. Etsy plans to expand CUPED and explore additional variance reduction methods.

https://www.etsy.com/codeascraft/reducing-experiment-duration-with-predicted-control-variates etsy.com

RSS Hunter • Nov 25, 2025

Improving performance by prefetching product pages from Etsy Search

Etsy implemented the Speculation Rules API (SRA) to significantly improve the performance of its product listing pages. This new browser API allows websites to instruct browsers on how to fetch resources for future navigation. SRA offers prefetching, which downloads the HTML, and prerendering, which fully loads and renders a page. While prerendering offers greater gains, prefetching is a less risky starting point. SRA provides a simpler API than traditional prefetching methods and an upgrade path to prerendering. Etsy specifically used SRA to prefetch listing pages when users hovered over organic listings on the desktop search page. The implementation involved adding a script tag with JSON rules to dictate when and what to prefetch. A key lesson learned was that SRA prefetches cache pages in both memory and the HTTP cache, unlike traditional prefetch which only uses the HTTP cache. The browser limits memory cache to two prefetched pages, after which older ones are evicted. Etsy found that setting a five-minute cacheability for pages when a prefetch request is detected helps mitigate this limitation. Challenges were encountered with elements like video players that used shadow DOM, requiring workarounds. Setting cookies during prefetching could lead to misleading analytics, but this can be avoided using specific HTTP headers. Redirects did not negatively impact prefetching as long as proper caching headers were set. Modifying href attributes on hover could cause prefetching to fail and evict other cached pages. A significant hurdle was ensuring analytics remained accurate, as prefetched pages are not necessarily viewed. Etsy addressed this by logging events only after page activation and avoiding asset loading and JavaScript execution during prefetches. The implementation resulted in notable performance improvements.

https://www.etsy.com/codeascraft/search-prefetching-performance etsy.com

RSS Hunter • Oct 29, 2025

Understanding Etsy’s Vast Inventory with LLMs

Etsy's marketplace, featuring millions of unique, handmade items from various sellers, faces challenges in organizing its unstructured data. Traditional methods of product attribute extraction struggled with the diverse inventory and limited structured data. Large Language Models (LLMs) provided a new opportunity to transform unstructured product information into structured data. Etsy developed a scalable pipeline using LLMs, focusing on context engineering to improve attribute extraction accuracy. This pipeline leverages seller-provided data, expert examples, and Etsy's taxonomy. Evaluation of the LLM output involves generating 'silver labels' and utilizing domain experts for quality assurance. The inference process extracts attributes, uses LiteLLM for regional routing, and employs Pydantic for data validation. Robust monitoring systems track pipeline health and model performance metrics. The application of LLM-generated attributes to search filters has enhanced buyer engagement and conversion rates. Etsy aims to expand its use of LLMs to further improve the shopping and selling experience. The ultimate goal is to ensure that buyer and seller needs are met with maximum efficiency.

https://www.etsy.com/codeascraft/understanding-etsyas-vast-inventory-with-llms etsy.com

RSS Hunter • Oct 13, 2025

Unlocking Faster Insights with Experimenter-Defined Segmentations

Etsy uses experiments to test ideas, like offering free ice cream with purchases, to boost sales. To better understand user behavior, they use segmentation, which allows analysis based on different user groups. Previously, Etsy's platform had limited pre-defined segmentations, hindering in-depth analysis and slowing down insights. The team developed a Segmentation Library allowing experimenters to define their own segments, which was a slow and complex process. This new approach allows users to create segmentations using SQL queries and join keys, making them easy to understand and share. Before deployment, these custom segmentations go through a validation process to ensure accuracy and safety. Dynamic task generation in the data pipeline enabled automatic processing of new segmentations without code changes. This transformation resulted in a significant increase in the number of available segmentations. The improved system provides quicker and deeper insights, empowering teams to make better data-driven decisions. Overall, these changes have made Etsy's experimentation process more flexible and efficient.

https://www.etsy.com/codeascraft/unlocking-faster-insights-with-experimenter-defined-segmentations etsy.com

RSS Hunter • Oct 8, 2025

Building Etsy Buyer Profiles with LLMs

Etsy is using large language models (LLMs) to create detailed, anonymous buyer profiles to enhance personalization for its nearly 90 million users. These profiles capture nuanced interests like preferred styles and product categories based on browsing and purchasing behaviors. The technical implementation involves retrieving user activity data and prompting the LLM to interpret this data and generate structured profiles. Etsy optimized the profile generation process by streamlining data sources, reducing input tokens, and increasing batch sizes, significantly reducing generation time and costs. These buyer profiles are being used in applications such as query rewriting and refinement pills to improve search relevance and user experience. Etsy is measuring the success of these profiles by tracking metrics like click-through rate, conversion rate, and refinement pill engagement. They also plan to address the "cold start" problem for new users by using collaborative filtering to match them with similar existing profiles. Etsy is actively monitoring profile accuracy and adjusting refresh strategies based on user activity and seasonal considerations. This personalization enhances buyer experience by surfacing items tailored to individual tastes.

https://www.etsy.com/codeascraft/building-etsy-buyer-profiles-with-llms?tag=data-science%2Cmachine-learning etsy.com

RSS Hunter • Sep 3, 2025

Building Etsy Buyer Profiles with LLMs

Etsy utilizes large language models (LLMs) to create detailed, anonymous buyer profiles based on browsing and purchasing history. These profiles capture nuanced interests and shopping missions, enhancing personalization for nearly 90 million buyers on the platform. The technical implementation involves retrieving user activity data and then prompting an LLM to interpret this data for profile generation. To make this process scalable and cost-effective, Etsy optimized data sources, reduced input token volume, increased batch sizes, and employed parallel processing. These optimizations drastically reduced buyer profile generation time and costs. The generated buyer profiles are then applied to personalize the search experience through query rewriting and refinement pills. Query rewriting enriches user searches with predicted interests, while refinement pills offer clickable filters based on user preferences. Etsy measures the success of this personalization through metrics like click-through rate and conversion rate lifts. They also maintain profile accuracy by dynamically refreshing them based on user activity and detecting interest drift. Future work includes addressing the "cold start" problem for new users by experimenting with inheritance profiles. Ultimately, Etsy aims to improve discovery and create more intuitive search experiences for every shopper.

https://www.etsy.com/codeascraft/building-etsy-buyer-profiles-with-llms etsy.com

RSS Hunter • Sep 3, 2025

Context engineering case studies: Etsy-specific question answering

This post explores prompt engineering with Large Language Models (LLMs) for AI-assisted onboarding at Etsy. The primary focus is on the truthfulness and reliability of LLM-generated answers, particularly concerning Etsy-specific questions. The study examined two use cases: internal Travel & Entertainment (T&E) policy questions and external Etsy seller community forum questions. For the T&E policy, LLMs answered approximately 86% of questions correctly, but the remaining 14% contained factual errors or misleading statements, termed "hallucinations." Techniques like instructing the LLM to admit uncertainty or to explain its reasoning were found to mitigate these hallucinations. In the Etsy community forums, with more heterogeneous data, LLM accuracy dropped to around 72%. The LLM performed better when queries closely matched the wording in reference documents. The study also highlighted limitations where even providing additional context did not resolve certain types of complex questions. Asking for source snippets was identified as a method to flag potential LLM hallucinations. Overall, prompt engineering shows promise but requires careful crafting to ensure reliable AI assistance in onboarding and information retrieval.

https://www.etsy.com/codeascraft/context-engineering-case-studies-etsy-specific-question-answering etsy.com

RSS Hunter • Aug 19, 2025

Behind the Scenes - A Glimpse into Tax Calculations

Etsy now handles tax collection and remittance globally due to evolving jurisdictional requirements. Their tax calculation process involves determining tax during the shopping cart phase, utilizing buyer/seller location, product category, and tax rules. Etsy partners with Vertex, using their Quotation Request and DistributeTaxRequest APIs for accurate calculations and continuous updates. Order and tax details are synced with Vertex, even for cancellations and refunds, ensuring comprehensive transaction records. Vertex generates reports and audit data, which Etsy imports for analysis after the data is moved to transaction detail table from tax journal database by a cron job. Etsy maps its product categories to Vertex's rule drivers to correctly determine tax applicability. To manage increased API traffic, Etsy implemented multiple Vertex instances, separating quotation and distribution calls for scalability. They also use shadow logging to simulate stress on Vertex and monitor the checkout experience before releasing new features. This process allows them to ensure performance is not impacted while navigating increasing tax complexities and traffic volume. Future improvements will address analytics, reporting, and configuration synchronization.

https://www.etsy.com/codeascraft/behind-the-scenes---a-glimpse-to-tax-calculations?attachment_id=622%2F%2F etsy.com

RSS Hunter • Mar 29, 2025

Adopting Jetpack Compose for Etsy’s Android App

Etsy adopted Jetpack Compose as its preferred means of building Android apps due to its modern toolkit for defining native UIs. The adoption process involved a gradual expansion of features built using Compose, culminating in a full rewrite of a primary screen in the app. To facilitate the adoption, Etsy engineers created a curriculum based on Compose documentation and examples, and the Design Systems team created Compose versions of internal UI toolkit components. The team also utilized Compose Previews to visualize Composables in various configurations, reducing rework and bugs. After successfully rebuilding an entire screen and exposing it to real users, Etsy adopted Compose on a larger scale in the Shop screen, which resulted in improved performance, developer satisfaction, and user analytics. The adoption of Compose has become the standard way Etsy builds features for its app, with benefits including decoupling state from the UI, isolating business logic, and easier manipulation of spacing and margins. However, the team encountered some challenges, including issues with a third-party library and unexpected behavior with LazyRows and LazyColumns. Overall, Etsy is thrilled with the progress and outcomes of adopting Compose, which has enabled the team to be more productive in delivering new features to buyers. The adoption of Compose is a tangible example of Etsy's commitment to its craft and culture of learning. The team has now fully rewritten several key UI screens using Compose, with more to come.

https://www.etsy.com/codeascraft/adopting-jetpack-compose-for-etsy39s-android-app?uvipdlma=rpqqnbob etsy.com

RSS Hunter • Jan 28, 2025

Adopting Jetpack Compose for Etsy's Android App

Etsy's Guiding Principle of "committing to our craft" led to the adoption of Jetpack Compose, a modern toolkit for defining native UIs, as the preferred means of building their Android app. A structured approach was taken to study and implement Compose, with engineers creating a curriculum and holding information sessions to teach the team. The Design Systems team created Compose versions of internal UI toolkit components, ensuring a complete implementation of the design system before major adoption. Compose's interoperability with existing toolkit components allowed for a smooth migration path. The team rebuilt an entire screen using Compose, exposing them to a wider scope of concerns such as navigation and data fetching. After a successful rewrite, Compose was rolled out to real users, starting with complex bottom sheets. The Shop screen was then rebuilt using Compose, resulting in a 5% improvement in initial screen rendering time and improved user interactions. The adoption of Compose also improved developer satisfaction, with engineers reporting fewer lines of code and easier testing. The combination of Compose and the Macramé architecture has become the standard way to build features for the app. Overall, Etsy's adoption of Compose has unlocked opportunities and efficiencies for their teams, despite some learning curve and bumps along the way.

https://www.etsy.com/codeascraft/adopting-jetpack-compose-for-etsy39s-android-app etsy.com

RSS Hunter • Jan 27, 2025

Machine Learning in Content Moderation at Etsy

Etsy prioritizes safety and trust in its marketplace to empower creative entrepreneurs. With over 100 million unique items and 7 million active sellers, Etsy employs a Trust & Safety team dedicated to enforcing policies and removing potentially violating content. To enhance detection capabilities, Etsy utilizes Machine Learning, including supervised learning models, leveraging multimodal signals (textual and imagery) from listings. The model architecture includes text and image encoders to learn representations and employs a neutral class for non-violating listings. To address data imbalance, focal loss is used. Model evaluation involves offline training and production deployment through A/B testing. Etsy continuously iterates and improves its detection system by deploying promising offline candidates and monitoring performance. As a result, over 100,000 violations have been identified and removed, supplementing other detection methods. Etsy remains committed to using innovative approaches, including Large Language Models, to enhance its safety measures.

https://www.etsy.com/codeascraft/machine-learning-in-content-moderation-at-etsy etsy.com

RSS Hunter • Aug 6, 2024

Enhancing Cloud Usage Forecasting, Monitoring & Optimizing

Etsy's migration to Google Cloud Platform led to the establishment of a dedicated FinOps team that drives cloud value optimization through collaborations within and outside the organization.The team's forecasting methodology leverages a Cost Per Visit (CPV) metric to establish a spending baseline for tracking and identifying deviations.Despite limitations in CPV, the FinOps team monitors costs using internal tools and alerts to detect sudden spikes or gradual growth.Regular collaboration with the Finance department ensures alignment between actual costs and forecasts.Cost increases are swiftly identified and mitigated through analysis, collaboration, validation, and troubleshooting.Cost optimization initiatives involve collaboration between FinOps and engineering teams, with a focus on efficiency improvements rather than solely on cost savings.Recent optimization initiatives include GCS storage optimization, compute optimization, increased automation for model deployment, and network compression.The FinOps team prioritizes improving visibility into Machine Learning platform spend to enhance ROI analysis and identify future optimization opportunities.The team's independent operation within Engineering and its focus on global rather than local optimization foster a successful FinOps practice.Etsy's robust engineering culture, emphasizing efficiency and craftsmanship, contributes to the effectiveness of the FinOps team.

https://www.etsy.com/codeascraft/enhancing-cloud-usage-forecasting-monitoring--optimizing etsy.com

RSS Hunter • Jun 17, 2024

Efficient Visual Representation Learning And Evaluation

Etsy utilizes computer vision to enhance user experience with features like visual search and visually similar recommendations. These features require efficient and expressive visual representations, obtained through machine learning models. Etsy initially employed EfficientNetB0 but switched to the more efficient EfficientFormer-l3 due to its superior performance and lower computational requirements. To further enhance efficiency, Etsy fine-tunes these pre-trained backbones and employs multitask learning, training the representations on multiple classification tasks simultaneously. The evaluation scheme involves three nearest neighbor retrieval tasks to track model progress and guide training. Etsy has also implemented an experimental evaluation scheme that leverages generative AI, bridging the gap between text-based queries and clicked image candidates. To ensure efficient inference for downstream tasks, Etsy utilizes a fast stable diffusion model that generates high-quality images with reduced memory consumption and latency. By employing these techniques, Etsy has optimized its visual representations for efficient and scalable use in various applications.

https://www.etsy.com/codeascraft/efficient-visual-representation-learning-and-evaluation etsy.com

RSS Hunter • Jun 7, 2024

Macramé: Untangling the Knot on the Etsy Android Listing Screen

The Etsy Android app's listing screen, crucial for showcasing products, had become complex and difficult to maintain due to rapid code changes. To address this, the team implemented "Macramé," a new architecture based on immutable data and reactive UI. The core of the architecture is an immutable State object representing the data model, passed to the UI via StateFlow. Data updates are handled by Events, dispatched to Handlers for processing and State or SideEffects updates. A ListAdapter and RecyclerView handle asynchronous UI changes. Unit test coverage increased significantly, validating the use of simple Handlers for business logic. Despite some boilerplate and complexity for simple screens, the architecture improves code maintainability and performance. Time to First Content decreased by 18%, leading to increased user engagement. The Android team has expanded Macramé's usage to other app screens, such as Cart and Shop, written in Jetpack Compose.

https://www.etsy.com/codeascraft/macrame-untangling-the-knot-on-the-android-listing-screen etsy.com

RSS Hunter • Mar 18, 2024

How We Built The Deals Tab in Swift UI

In July 2023, Etsy's App Updates team set out to revamp the Updates feed into Deals, a hub for coupons and sales. The team considered developing a new tab from scratch using modern technologies such as Swift UI and Tuist.Balancing ambition with realism, the team adopted a hybrid approach, using Swift UI for modular development and previewability, while integrating with existing UIKit codebase for navigation and other functionality.Swift UI's modularity allowed them to break down views into reusable components, enabling rapid development and efficient previewing with Tuist.They implemented Decodable models for clear and efficient API parsing, simplifying view construction and handling of optionals.Preview enums were introduced to streamline the creation of mock data for complex views, enabling them to build modules even before API support was complete.However, interfacing with the existing codebase posed challenges, particularly in areas like navigation and favoriting.To address this, they created a custom @Environment struct, DealsAction, which passed navigation responsibility back to the main target while maintaining Swift UI's callAsFunction() feature.Environment objects were utilized for other functionality, such as favoriting, following shops, and logging performance metrics.This hybrid approach allowed the team to leverage the benefits of modern technologies while respecting the constraints of the legacy codebase, resulting in a successful product launch before the Cyber Week deadline.

https://www.etsy.com/codeascraft/how-we-built-the-deals-tab-in-swift-ui etsy.com

RSS Hunter • Mar 5, 2024

Behind the Scenes - A Glimpse to Tax Calculations

Due to increasing tax obligations, marketplaces like Etsy now collect and remit taxes for buyers. Etsy partners with Vertex for tax calculations based on buyer and seller location, product category, and tax rules. When a buyer adds an item to their cart, Etsy calls Vertex's Quotation Request API to determine the tax amount. When the order is created, Etsy uses the DistributeTaxRequest API to send order information and tax details to Vertex. Etsy collects tax from buyers and remits it to the appropriate authorities. Vertex provides various report formats, which Etsy imports into its big data system. Mapping Etsy product categories to Vertex tax categories involved collaboration between engineering, tax, and analytics teams. To handle increased API traffic, Etsy implemented multiple instances for Vertex calls and shadow logging to simulate stress on Vertex and monitor checkout experience. These upgrades ensured smooth checkout without delays. Etsy continues to improve its design to support increasing traffic and data volume, with future plans to address analytics, reporting, and configuration synchronization.

https://www.etsy.com/codeascraft/behind-the-scenes---a-glimpse-to-tax-calculations etsy.com

RSS Hunter • Jan 3, 2024

The AR Measuring Box: Etsy's answer to Big Tape Measure

Etsy's enhanced AR tool allows users to visualize the size and scale of items in their space before making purchases. The tool was initially limited to wall art but has been expanded to support a wider range of categories with customizable options. To improve accuracy, Etsy developed a robust dimension parsing system that extracts dimensions from seller-provided data using regular expressions. This data is now stored on the server to enhance performance and consistency. The AR experience uses Apple's SceneKit to create a scaled rectangular prism that represents the item, displaying its dimensions and allowing for interactive rotation and movement. Environment occlusion, enabled by LiDAR scanners on newer iPhones and iPads, allows the measuring box to interact realistically with real-world objects, providing users with a more accurate understanding of how an item would fit in their space. The improved dimension parsing system, 3D measurements, and environment occlusion combine to enhance the user experience, making Etsy's AR tool a valuable resource for informed decision-making when purchasing unique and customizable items.

https://www.etsy.com/codeascraft/the-ar-measuring-box-etsys-answer-to-big-tape-measure etsy.com

RSS Hunter • Sep 18, 2023

The So-fine Real-time ML Paradigm

Etsy's annual CodeMosaic hackathon showcased an ambitious project to develop a system for stateful machine learning training and online learning. The team aimed to update model weights incrementally, saving costs and improving metrics. On Day 1, they planned their approach, dividing into subteams focused on real-time training data, a learning service, and evaluation. Day 2 saw the teams implementing their plans, with challenges including data formatting and model selection. The evaluation team proposed a competition to compare the performance of a continuously trained model against a batch-trained model. Day 3 brought presentations and discussions of potential impact, including an estimated $212K annual cost saving for Ads models alone. The team acknowledged the hurdles to productionizing the system, including infrastructure and quality assurance, but expressed optimism for the future of complex ML architectures.

https://www.etsy.com/codeascraft/the-so-fine-real-time-ml-paradigm etsy.com

RSS Hunter • Sep 5, 2023

Leveraging Real-Time User Actions to Personalize Etsy Ads

Etsy's adSformer Diversifiable Personalization Module (ADPM) personalizes ranking models by capturing temporal and content signals from user actions within a one-hour session. The ADPM module consists of three components: an adSformer encoder, pretrained representations, and representations learned on the fly. The adSformer encoder uses a modified transformer block to learn a deep representation of the user's session. Pretrained representations leverage item embeddings trained offline, while representations learned on the fly are generated in the downstream models. These components are concatenated to create a dynamic user representation that is integrated into ranking models. The ADPM improves the relevance of sponsored search results by encoding user actions such as search queries, favorites, and purchases. By combining three components, the ADPM outperforms models that use each component independently and provides robustness to input distribution shifts. The ADPM has been deployed in Etsy's sponsored search system, resulting in significant gains in offline and online evaluations.

https://www.etsy.com/codeascraft/leveraging-real-time-user-actions-to-personalize-etsy-ads etsy.com

RSS Hunter • Jul 14, 2023

The Problem with Timeseries Data in Machine Learning Feature Systems

Etsy's Feature Systems team encountered a potential problem when using timestamp features in machine learning models due to precision misinterpretation across frameworks.The issue stemmed from the timestamp data type, which was being interpreted differently by different frameworks, leading to a potential training/serving skew.To address this, ML practitioners recommended avoiding the timestamp type and using a more basic numeric type, such as Longs.The team investigated the root cause, discovering that the problem extended beyond specific bugs and highlighted a larger issue for ML practitioners in dealing with timestamp features.The team realized that the complexity of datetime objects and timestamp types was unnecessary for their use case, as they only needed integer representations at specific precision.During an architecture working group meeting, there was consensus to represent datetime features as primitive numeric types to ensure consistency between model training and inference.The team decided to standardize on primitive types more generally to promote consistency across all training contexts.The team also recognized the need for improved documentation to simplify feature transformation for customers.The incident highlighted the potential challenges in applying software engineering practices to ML-specific needs.As ML continues to integrate into software systems, these types of nuances will likely become more common and require further refinement of best practices.

https://www.etsy.com/codeascraft/the-problem-with-timeseries-data-in-machine-learning-feature-systems etsy.com

RSS Hunter • Jun 23, 2023

From Image Classification to Multitask Modeling: Building Etsy’s Search by Image Feature

Etsy's search by image feature allows users to search for items similar to photos they upload. The feature uses a machine learning model to convert images into numerical representations called embeddings, which are then used for similarity searches.The model is based on a pre-trained convolutional neural network (CNN) that has been fine-tuned for the task of learning image embeddings. The model is trained using a multitask learning approach, where it learns to perform several classification tasks simultaneously, including item category, color, and attributes.To reduce bias towards high-quality product images, the model is also trained on a dataset of user-submitted review photos.The inference pipeline involves building an approximate nearest neighbor (ANN) index using an inverted file (IVF) algorithm to optimize search performance.Query photos are inferred in real-time using GPU inferencing technology to ensure fast response times.The search by image feature was initially developed during Etsy's CodeMosaic hackathon and has since been implemented as a production feature.The feature helps buyers discover unique and special items on Etsy by providing them with a new and intuitive way to search for similar products.The model's architecture and learning objective have been optimized to produce visually consistent results while maintaining categorical accuracy.The addition of review photos to the training dataset has significantly improved the model's ability to surface relevant results from user-submitted photos.The feature has been well-received by users and has contributed to increased buyer engagement and satisfaction on Etsy.

https://www.etsy.com/codeascraft/from-image-classification-to-multitask-modeling-building-etsys-search-by-image-feature etsy.com

RSS Hunter • May 26, 2023

How We Built a Multi-Task Canonical Ranker for Recommendations at Etsy

Etsy employs recommendation modules to present relevant items to users, each powered by a ranker that scores candidate items for relevance. Traditionally, Etsy used module-specific rankers, but this approach became unwieldy as the number of modules grew.To address this, Etsy developed canonical rankers, which are trained to power multiple modules, ensuring efficiency and consistency. The first canonical ranker focused on visit frequency, using favoriting rate as a surrogate for revisits.The frequency ranker's model structure included a shared-bottom architecture with separate layers for favoriting and purchasing predictions, combined into a final ranking score. The ranker also incorporated a module name feature and balanced training data across modules to ensure generalizability.Despite training on data from a limited subset of modules, the canonical ranker outperformed module-specific rankers on modules not used for training, demonstrating its effectiveness as a canonical solution.The frequency ranker improved favorite rates on both item page and homepage modules, with significant improvements in purchase metrics and other engagement indicators.Since its launch, Etsy has deployed the canonical ranker on multiple modules across web and app platforms.Moving forward, Etsy plans to iterate on the frequency ranker, incorporating more context and exploring novel architectures.The canonical ranker represents a shift in Etsy's recommendation strategy, providing more personalized recommendations and a consistent user experience across platforms and modules.

https://www.etsy.com/codeascraft/how-we-built-a-multi-task-canonical-ranker-for-recommendations-at-etsy etsy.com

RSS Hunter • Apr 18, 2023

Barista: Enabling Greater Flexibility in Machine Learning Model Deployment

Machine learning (ML) model deployment involves a transition from experimentation to rigorous engineering constraints, presenting challenges in balancing flexibility and stability.Etsy's ML platform team uses Kubernetes for model scaling and orchestration, with Barista managing model deployments.Initially, model configurations were managed as code, providing tight control but leading to delays and bottlenecks.To address these issues, configurations were decoupled and stored in a database, enabling instant changes via a CLI.However, the CLI required technical expertise, prompting the development of a user-friendly web interface for model management.The Barista web interface provides comprehensive control over deployments, integrates with various APIs, and streamlines the deployment process.Increased model deployment rates raised concerns about cost and misconfigurations, leading to the implementation of Kube Downscaler to automatically scale down unused deployments.The focus has shifted from meeting basic technical requirements to building a complete product that empowers ML users.Current efforts aim to enhance service cohesiveness and automation to optimize infrastructure settings and further reduce cloud costs.As the ML practice expands, the platform must continue to evolve to meet the growing needs of the team.

https://www.etsy.com/codeascraft/barista-enabling-greater-flexibility-in-machine-learning-model-deployment etsy.com

RSS Hunter • Apr 14, 2023

Docs-as-code at Etsy

Docs-as-code integrates documentation into development using the same tools and procedures as code. It emphasizes versioning, plain-text markup, and automation for improved documentation management.Etsy's Docsbuilder tool employs Markdown, Docusaurus, and GitOps workflows for documentation creation and maintenance.Docsbuilder automates site creation, validation, and deployment with Google Cloud Build and integration tests.Docsbuilder enables discoverability with a search engine and a navigation component for frequently used sites.Etsy currently hosts over 6.2k pages on 150 Docsbuilder sites.The goal of docs-as-code is to enhance documentation quality, efficiency, and reliability.Key principles include treating documentation as a first-class citizen, versioning in Git, using plain-text formats, and automating workflows.Docs-as-code encourages developers to approach documentation with the same rigor as coding.Etsy plans to improve discoverability, navigation, and content engagement within Docsbuilder.Docs-as-code fosters a balanced approach between documentation and coding, leading to organized and maintainable documentation.

https://www.etsy.com/codeascraft/code-as-craft-blog-post-docs-as-code-at-etsy etsy.com

RSS Hunter • Apr 13, 2023

Scaling Etsy Payments with Vitess: Part 3 – Reducing Cutover Risk

Etsy Payments moved 40 billion rows across 23 tables into a Vitess-managed sharded environment, using vindexes for data sharding. This article focuses on errors that can arise during the cutover.Understanding Vitess's transaction modes is crucial. Single mode maintains atomicity, but multi mode can lead to partial commits. Two-phase commit mode is experimental and not recommended.Reverse VReplication ensures data synchronization between the unsharded and sharded keyspaces after cutover. It can break due to unique key enforcement, requiring fixes such as row deletion or manual Pos column updates.Scatter queries, where the sharding key is omitted from the WHERE clause, can result in excessive query volume and potential outages. Vitess now offers a --no_scatter flag to prevent them.Incompatible queries can fail after cutover. Exhaustive testing in a development environment is essential to identify and resolve such queries.Other potential errors include those related to unsupported SQL constructs, which can be addressed by upgrading to newer Vitess versions.Despite these risks, cutovers are generally reversible, provided that reverse VReplication functions properly. However, the impact of any disruption should be carefully considered.

https://www.etsy.com/codeascraft/scaling-etsy-payments-with-vitess-part-3--reducing-cutover-risk etsy.com

RSS Hunter • Feb 22, 2023

Scaling Etsy Payments with Vitess: Part 2 – The “Seamless” Migration

Etsy migrated 40 billion rows from unsharded databases to a Vitess-managed, sharded environment.The migration involved redistributing data, verifying completeness, and switching reads/writes.Staging was crucial for testing and building confidence in the process.VReplication enabled smooth read/write switching and data consistency.Unexpected query volume increase was addressed with CreateLookupVindex, preventing scatter queries.Secondary Vindexes were applied during the migration to optimize shard identification.Due to Vindex maintenance limitations, writes were switched back to the source after externalizing Vindexes.Destination cluster specs were upgraded to handle the load after externalizing Vindexes.The migration was completed without disruption or downtime, thanks to extensive testing and careful planning.This was part 2 of a series on sharding Etsy's payments data with Vitess.

https://www.etsy.com/codeascraft/scaling-etsy-payments-with-vitess-part-2--the-seamless-migration etsy.com

RSS Hunter • Feb 22, 2023

Scaling Etsy Payments with Vitess: Part 1 – The Data Model

Etsy's payment databases faced scalability issues in 2020, necessitating a migration to a sharded environment managed by Vitess.The project involved moving 40 billion rows from four databases into a single sharded environment, using Vitess vindexes for sharding.The first phase involved migrating the seller ledger infrastructure, which had an ideal data model for sharding.The second phase focused on reducing load on the primary payments database, which had a complex data model that required a more tailored approach.Two options were considered for the second phase: remodeling the core data model or finding shardifiers within the existing model.The latter option was chosen due to time constraints and the importance of maintaining reliability.The team used existing primary and foreign keys as shardifiers, reducing the workload compared to the first phase.Vitess' re-sharding features provide flexibility for future adjustments to the shard design.Secondary indexes in Vitess mitigate limitations of the non-ideal data model, allowing for shard targeting without using the shardifier in queries.This migration improved scalability by 60% and provided room for future growth.

https://www.etsy.com/codeascraft/scaling-etsy-payments-with-vitess-part-1--the-data-model etsy.com

RSS Hunter • Feb 22, 2023

Adding Zonal Resiliency to Etsy’s Kafka Cluster: Part 2

Initially, platform upgrades for Kafka brokers were tedious and time-consuming, requiring hours of manual monitoring and waiting. To improve this, a multizone architecture was implemented, allowing multiple brokers to be updated simultaneously without affecting availability. However, Kubernetes' native rolling update strategy was not suitable due to the zonal distribution of replicas.Custom logic was developed to control updates, allowing multiple brokers in a zone to be restarted concurrently. This was implemented as a Kubernetes batch job to ensure reliability and prevent accidental deployment issues.Testing in production showed that with a parallelism of three, upgrades could be completed in about two hours. While restarting all brokers in a zone simultaneously was technically possible, it was avoided to prevent increased load on remaining brokers.The multizone architecture and custom update logic significantly reduced upgrade time, from seven hours to around two hours. This improvement not only saved time but also reduced the toil and stress associated with upgrades.The new process ensured quick and efficient upgrades, with minimal impact on the Kafka cluster. The project's success was measured not only by reduced duration but also by the ease and peace of mind it provided during upgrades.

https://www.etsy.com/codeascraft/leveraging-zonal-resiliency-to-improve-updates-for-etsys-kafka-cluster-part-2 etsy.com

RSS Hunter • Feb 9, 2023

Adding Zonal Resiliency to Etsy’s Kafka Cluster: Part 1

In 2018, Etsy migrated its Kafka brokers to Google Cloud Platform's Kubernetes Engine. Initially operating in a single zone, the team later redesigned the architecture for zonal resilience, distributing brokers across multiple zones with even distribution of partition replicas.To achieve a zero-downtime migration, brokers were moved first by snapshotting disks and then recreating them in the correct zones. Partition relocation was manually handled using scripts and tools to minimize data movement and impact.Post-migration testing in production demonstrated the effectiveness of the multizone design, with minimal disruption during a zone outage. While inter-zone network costs increased as expected, the benefits of automated zone resilience outweigh the costs.The team is optimizing costs by leveraging Kafka's follower fetching feature and exploring additional approaches to reduce cross-zone traffic. Despite some cost increases, the benefits of zonal resilience are significant, justifying the investment.The migration involved complex steps, including disk and Pod movement, partition relocation, and configuration adjustments. The team's careful planning and execution ensured zero downtime and data integrity throughout the process.Etsy's experience highlights the importance of designing for resilience in critical services. By embracing zonal redundancy, the team mitigated the risks associated with single-zone failures and improved the stability and availability of their Kafka cluster.The multizone architecture enables Etsy to handle increased production traffic and critical user-facing features, such as search indexing, with confidence.The company's ongoing efforts to optimize costs demonstrate a commitment to balancing resilience with financial considerations.The case study provides valuable insights into the challenges and strategies involved in migrating and operating a highly available Kafka cluster in a multi-zone cloud environment.

https://www.etsy.com/codeascraft/adding-zonal-resiliency-to-etsys-kafka-cluster-part-1 etsy.com

RSS Hunter • Jan 31, 2023

Improving Support for Deep Learning in Etsy's ML Platform

Serving deep learning ranking models at scale presents challenges due to high latency and computational costs. To address this, the Search Ranking team at Etsy created Caliper, a tool for early latency feedback during model development. By isolating the inference component, Caliper allows for efficient tuning of parameters like batch size. Additionally, increased observability through distributed tracing and Envoy access logs enabled the team to identify a bottleneck in feature transmission. By leveraging compression techniques, the payload size was significantly reduced, leading to a 68% decrease in error rates and a 50ms drop in p99 latency. To prepare for future complexities, the team is exploring further payload size optimizations and improving Caliper for automated performance tuning. These advancements have empowered the Search Ranking team to effectively serve deep learning models at scale, ensuring fast and accurate search results for Etsy users.

https://www.etsy.com/codeascraft/improving-support-for-deep-learning-in-etsy39s-ml-platform etsy.com

RSS Hunter • Jan 26, 2023