Ingesting Public Data That Fai... Note
DZone.com

Ingesting Public Data That Fails Silently: Patterns From a TypeScript and Cloudflare Workers Pipeline

AbstractPublic statistical datasets rarely fail with an error. When something goes wrong, they hand back a plausible number, an HTML page with a success status, or a series that has quietly stopped updating. This article describes the ingestion pipeline behind a residential concrete cost estimator that combines four public datasets: an annual occupational wage survey, a monthly producer price index, a 2022 industry benchmark of material prices, and a ZIP code to metro area crosswalk. For each dataset, I document the failure mode we actually observed, the evidence for it, and the TypeScript code that turns it into an explicit state. The central claim is that every silent failure should become a typed value that travels with the number all the way to the user interface. The pipeline runs on a local CLI and a scheduled Cloudflare Worker, writes to Cloudflare D1, and serves a static site.