Making User-Sequence Data More... Note

Making User-Sequence Data More Cost-Efficient, Faster, and Easier to Use

The text describes the redesign of a user-sequence platform at Pinterest, aiming to provide a robust and efficient system for retrieving user behavior data for ML models. The core goal is to deliver consistent, fresh, complete, and cost-effective sequences across training, analysis, and serving. The platform defines user sequences as ordered lists of recent, enriched events. Key challenges addressed include ensuring data freshness, completeness, consistency, and scalability across different use cases and teams. The solution employs a "one definition, many runtimes" approach, using configuration-as-code and a shared execution engine to process events in real-time and batch. The platform implements a lambda architecture to manage both current and historical data. This design allows for easier onboarding of new event types and enrichments, improved code review, and reduced drift between real-time and batch processing. The three crucial design decisions are configuration-as-code for sequences and enrichments, a shared execution engine, and a lambda architecture for sequences. The result is a platform that simplifies the process of building, maintaining, and utilizing user sequences for various ML tasks within the company.
CdXz5zHNQW_ofSjTVKPmX.png