At Pinterest, the mission is to inspire users to curate a life they love, which relies on state-of-the-art Recommendation and Ads models trained on tens of petabytes of data. These models drive personalized recommendations, showing users content that resonates with their interests. Experimenting with features is a common task, and the first step is integrating new features into the training dataset. The most straightforward method of incorporating features is through Forward Logging, but this method presents challenges such as high calendar day cost, high development time cost, lack of isolation, and resource wastage and instability. Feature Backfill is an alternative to forward logging that is commonly used to address these challenges. In this blog post, the authors explore how they've created their Feature Backfill Solution, leveraging various techniques to reduce costs and iteration time by up to 90x. The authors developed an initial backfill solution using Spark to materialize features within their training tables, which operates as a reusable Airflow DAG that is triggered by ML Engineers on Demand. However, this solution has challenges such as no concurrent backfills, high compute cost, and manual partition management. To address these challenges, the authors developed a v2 version, adopting a two-stage backfill approach, which streamlines the process into two key stages: Feature Staging and Feature Promotion.