Unexpected or poor results with audio transcription? Check your audio mix...

[jaygooby]

Spent more than a day puzzling why we couldn't get anything like a recognisable transcription from a file, that Macwhisper with the free small model, had no issues with.

We just had endless variations of:

[music]
[music]
[music]

or sometimes:

[music]
(upbeat music)
(upbeat music)
[music]

Spent ages faffing around with different params and even ended up brute forcing the temperatures and temperature increments to try and see if that made any difference, but then my colleague @boblete said "Have you tried making a mono version of the audio?"

Oh no.

The video we were extracting the audio from, had been recorded with a polar mic or the stereo mix had been done with the spoken audio on one channel and backing audio on the other.

When we made the wav from the mp4:

ffmpeg -nostdin -hide_banner -loglevel error -y -i demo.mp4 -ar 16000 -f wav tmp.wav

whisper takes our wav and makes a mono version - and the first channel happens to have only the backing audio on, so no actual audio to transcribe, hence our lack of anything sensible out of the model - it was just getting the backing audio and no spoken words.

You can mimic the process of what whisper was doing with the mono channel like this:

ffmpeg -nostdin -hide_banner -loglevel error -y -i demo.mp4  -ar 16000 -ac 1 -f wav mono.wav

and in our case that mono.wav just has backing audio - the narration has been dropped.

So just a heads up that if you're getting unexpected or poor results with audio transcription, check your audio mix!

You can ensure the mono mix you give to whisper has both source channels on if you make your .wav like this:

ffmpeg -nostdin -hide_banner -loglevel error -y -i demo.mp4  -ar 16000 -af "pan=mono|c0=c1" -f wav mono.wav

Hope this helps save your sanity!

[jart]

Thank you for sharing this information!