Treating Web Scrapers Like Pro... Note
DZone.com

Treating Web Scrapers Like Production Services

Most teams treat a broken web scraper as a chore. Someone notices bad data, opens the page in a browser, patches a selector, and redeploys. That works when you have one scraper and someone who looks at its output every day. It stops working when you have a dozen, and the breakage surfaces a week later as a strange number in a downstream report. In this article, I walk through how I built scrapers that are cataloged, health-checked, traced, and repaired like any other production service, with an LLM proposing fixes and plain Python deciding whether those fixes ship.The project came out of the WeMakeDevs hackathon, where it took first place. The product itself is small: it pulls about twelve days of TV guide data across three streaming providers, ranks the top ten upcoming movies and top ten upcoming shows, and lets you record any of them on Philo with one click. I am going to skip the product and focus on the plumbing: the drift rules, the state model, the verifier, and the code behind each. Those pieces carry over to any team that runs scrapers, or to any team that lets an LLM modify a live system.