WARNING: Got start time outside of audio boundary

[dgoryeo]

Hi,

I'm getting the message: WARNING:whisper_timestamped:Got start time outside of audio boundary. Is there anything I should do as workaround?

I'm wondering of this is related to TensorRT. I received these messages prior to the timestamped warning (colab environment):

2023-02-27 21:21:26.854741: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-02-27 21:21:28.890710: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2023-02-27 21:21:28.890847: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2023-02-27 21:21:28.890868: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.

Any pointers would be helpful. Thanks.

[Jeronymous]

Concerning the tensorflow warnings, it also happens to me when running on CPU.
You can find infos about them on some forums.
I could remove (at least some of) them, but I thought it was maybe too dangerous to hide some warning, if some of them are probably good if the user wants to optimize the speed.

Concerning "WARNING: Got start time outside of audio boundary", it means that Whisper predicted a start timestamp after the audio finished. It can happen either because Whisper (as a statistical model) is hallucinating, either because the timestamp prediction is just clearly wrong.
So something should be weird somewhere in the output transcription.

There is actually a recently added option that might be useful for this case:

• --recompute_all_timestamps True with the CLI
• transcribe(..., trust_whisper_timestamps = False) in python

With this (experimental) option, all timestamps predicted by Whisper should not be taken into account in the prediction of the final timestamps.
It is an experiment option not activated by default, because I made a benchmark that shows that it generally does not help (the overall accuracy was a bit worse).
But this is the kind of situation where it could help.
If you give it a try, please share whether it was useful or not to have a better transcription.

[dgoryeo]

Thanks @Jeronymous, I'll give it a try later today or tomorrow and will report back.

BTW, I was impressed by the speed. I had done a test during the very early days of Whisper_Timestamped, and I must say this version has improved things quite much.

I will also try to dig into TensorflowRT a bit more. I'm guessing that it would perform faster on GPU --I have a T4 reserved so rather take full advantage of that.

[dgoryeo]

Hi @Jeronymous , I checked the output and indeed hallucination has happened --hence the error message. The option --recompute_all_timestamps True worked in the sense that the error message went away. However, the results are the same with or without the option.