How AI dubbing actually works

A finished video does not contain a clean voice track. It contains voices mixed with music, room tone, effects and whatever else was in the room. Replacing the speech means first separating it, which is a source-separation problem solved by a dedicated model — Demucs, in this pipeline.

Skipping this step is what produces dubbing where the music ducks and swells oddly, or disappears whenever someone speaks. If the original score cannot be preserved independently of the dialogue, the result sounds like a bootleg regardless of how good the translated voice is.

Transcription alone is not enough. A two-person interview transcribed without speaker labels becomes one undifferentiated monologue, and dubbing it produces one voice for both people. Separating speech into speakers is diarisation, and it runs alongside transcription — WhisperX for the words and precise timings, pyannote for the speaker boundaries.

Timing matters as much as identity here. Every translated line has to land in the slot where the original line was, so the transcript needs word-level timestamps rather than approximate segment boundaries.

This is the step most people underestimate. Translation for dubbing is not translation for reading: the output has to be speakable in roughly the time the original took. German is famously longer than English. Spanish runs longer than both. A faithful translation that takes 40% more time to say will either overrun the next line or have to be rushed into it.

The approach here is to generate several translation variants at different lengths, then choose between them after synthesising each one — because how long a line actually takes to say is a property of the synthesised audio, not something reliably predictable from the text. Picking the variant before synthesis means guessing; picking it after means measuring.

Once speakers are separated, a clean reference clip can be extracted for each one from the isolated vocal track, and their translated lines synthesised in a voice cloned from that reference. The presenter still sounds like the presenter in the new language.

The alternative — assigning a stock voice per speaker — is much easier and immediately recognisable. It is why a lot of dubbed content sounds like a documentary narrator regardless of who is on screen.

Even with length-aware translation, synthesised lines rarely match their slots exactly. The fix is time-stretching that changes duration without changing pitch — rubberband here. Naive resampling shortens audio by speeding it up, which raises the pitch and produces the cartoon effect that instantly marks a dub as automated.

There is a limit to how much stretching survives scrutiny, which is why the translation step matters: the closer the translated line is to the right length, the less the stretcher has to do.

The separated accompaniment is mixed back under the new dialogue with sidechain ducking, so the music steps back fractionally when someone speaks and returns when they stop — the same technique used in broadcast. Mixing the two at fixed levels instead produces dialogue that fights the score.

Burned-in subtitles, if requested, need a font with real coverage for the target script. A missing font does not error; it renders every character as an empty box on top of a video the customer has already paid for, which is why the fonts are installed and verified at image build time rather than assumed.

Lip-sync re-renders the mouth to match the new audio. It is a genuine improvement for talking-head footage and close-ups, and it is expensive — it is the single most costly stage in the pipeline.

It is also unnecessary for a great deal of content. Voiceover-led material, screen recordings, wide shots and anything where the speaker is not in close-up gain little from it. Applying it selectively per shot, and only where a face is actually detected, is both cheaper and better than applying it uniformly.

• Use a clip with background music. If the music survives cleanly, source separation is real.
• Use a clip with two speakers. If both come out in the same voice, there is no diarisation.
• Listen for rushing at the end of long sentences — that is translation length being papered over by time-stretching.
• Listen for pitch shifts. Any chipmunking means naive resampling rather than proper time-stretching.
• Dub into a language with a non-Latin script and check the subtitles for empty boxes.