Fine-Tuning F5-TTS for Arabic

[ibrahimabdelaal]

Hi everyone,
I've started fine-tuning F5-TTS for Arabic and uploaded an initial checkpoint to Hugging Face: https://huggingface.co/IbrahimSalah/F5-TTS-Arabic . The results are not the best yet, but the process is ongoing. I expect to upload a better checkpoint today or tomorrow.

[80boys]

good luck

[ahorazahedi]

Thank you for sharing your work on Arabic F5-TTS! I have a question regarding the handling of Arabic diacritics (tashkeel/تشكيل) in your model:

In Arabic, the same consonant skeleton can have different pronunciations based on diacritical marks. For example:
كَتَبَ (kataba - he wrote) vs كُتِبَ (kutiba - it was written)

I'd like to understand:

1. How does your model handle diacritics during training and inference?
2. Does the training dataset contain fully diacritized text?
3. Can the model handle both diacritized and non-diacritized input?
4. For non-diacritized input, how does the model determine the correct pronunciation?

This is particularly important since modern Arabic text is often written without diacritics, but the correct pronunciation depends on these marks and context.

[ibrahimabdelaal]

Thank you for your question! In my approach to handling Arabic diacritics (tashkeel) in the F5-TTS model, I followed a straightforward method by including diacritics in the vocabulary, treating them as individual tokens just like any other character. This allows the model to learn diacritics as part of the natural tokenization process.

How the Model Handles Diacritics:
Training with Diacritized Text:

The best way for the model to learn correct pronunciation is by training on fully diacritized text. However, large-scale diacritized Arabic datasets are scarce.
My training dataset consists of a mix of both diacritized and non-diacritized text.
To reinforce learning, I duplicated the diacritized data while removing diacritics in the second copy. This allows the model to encounter both versions and learn pronunciation from context.
for the Inference with Non-Diacritized Input:
There are two possible ways the model can handle non-diacritized text:
If the model learns diacritics well from training, it can infer them based on context, though this requires a massive dataset to be highly accurate.
Alternatively, for more reliable results, I can pre-process non-diacritized input by applying a diacritization model before feeding it to F5-TTS.
In case the model struggles with mixed input, another approach is to remove all diacritics during inference to ensure consistency.
so in points :
The model was trained with both diacritized and non-diacritized text.
It learns to predict diacritics from context, but the accuracy depends on the dataset size and quality.
For non-diacritized input, two strategies can be used:
Using a pre-trained diacritization model before inference.
Relying on the model's learned ability to infer diacritics (which depends on data coverage).
Diacritized text provides better supervision during training, making pronunciation prediction easier.