DEV Community
Follow
Build an AI dubbing pipeline: faster-whisper + XTTS-v2 + FFmpeg
This project outlines a Python script for creating Spanish language dubs of English videos using the original speaker's cloned voice. The process begins with extracting audio and generating timestamped transcriptions using faster-whisper. Machine translation is then applied to the transcribed text, with a crucial emphasis on maintaining segment length to avoid synchronization issues. The XTTS-v2 model is utilized for voice cloning and synthesizing the translated audio, requiring a short reference clip of the original speaker. A significant challenge is addressed: ensuring the synthesized Spanish audio fits within the original English time slots, as Spanish often takes longer to speak. This is managed through techniques like absorbing overflow into silence, gentle time-stretching, and re-translating segments that still overflow. Ultimately, the synthesized audio is remuxed with the original video, ideally as an additional audio track to allow player control over language selection. Potential pitfalls include handling numbers, background music, multiple speakers, and acronyms, all of which require careful pre-processing or specialized tools. The overflow report serves as a key quality control metric, and for more complex scenarios, pre-built pipelines or advanced lip-sync models are recommended.