DEV Community
Follow
The Upload Succeeded, the Record Did Not
The author developed a YouTube upload system with a sequential process involving session initiation, file transfer, video ID retrieval, verification, and local record creation. A critical flaw emerged because the local record was only written after the verification step. If the verification failed, an exception would halt the process, leaving no record of the uploaded video on disk. Consequently, re-running the upload command would bypass the existing file check, leading to duplicate video uploads.The system's documentation misleadingly stated that retries would not create duplicates, but this only applied to internal low-level retries, not external re-runs of the entire process after a failure. This led to the same video being uploaded twice. A similar bug was discovered in a different part of the repository, where five videos uploaded through an older process also lacked corresponding local records, making them vulnerable to duplication.The author highlights that existing tests passed because they did not account for the system's state on disk after an exception occurred. The fix involved changing the workflow to write the local record immediately after receiving the video ID, even if it was marked as unverified. This record would then allow for either resuming verification or blocking further uploads if the video was already processed. For the five pre-existing videos, a manual backfill of their records was necessary.The core issue generalizes to any operation that creates a remote resource and then verifies it, creating a window where a failure can leave the remote resource created but the local state unrecorded. This ambiguity renders retry logic ineffective. The solution emphasizes recording an identifier the moment it is received, regardless of subsequent verification. It also stresses that all code paths creating a resource must contribute to the same bookkeeping system to prevent invisible inconsistencies.