Designing a Polling Pipeline f... Note
DZone.com

Designing a Polling Pipeline for Rate-Limited APIs Without Losing Data

Every system that ingests data from someone else's API eventually writes a poller. It starts as fifteen lines in a cron job: hit the endpoint, compare against the last timestamp, insert what's new. It works immediately, which is the problem — it keeps appearing to work long after it has started losing records.I maintain an ingestion layer that polls public discussion APIs continuously, and the gap between "the poller is running" and "the poller is correct" took considerably longer to close than I expected. What follows are the four failure modes that caused real data loss, and the patterns that fixed them.