DZone.com
Follow
Why Scraping Pipelines Break in Production
Production code gets scrutiny. Builds are watched, libraries are checked for known vulnerabilities, and someone is responsible when a service fails at night. The job that pulls in outside data usually escapes all of this. It began as a quick script, got scheduled, and has run untouched ever since. It has no tests, its only error handling is a catch-all exception block, and its author may have left the team long ago.