How to Add Speaker ID During xtts_v2 Fine-Tuning

[falhafizh]

Hi @eginhard,

How can I add a speaker manager when fine-tuning xtts_v2?

I tried fine-tuning xtts_v2. In train_gpt_xtts.py, I added speaker_manager and language_manager and modified GPTTrainer :

    ....
    speaker_manager = SpeakerManager()
    speaker_manager.set_ids_from_data(train_samples + eval_samples, parse_key="speaker_name")
    language_manager = LanguageManager(config=config)

    config.model_args.num_speakers = speaker_manager.num_speakers
    config.model_args.num_languages = language_manager.num_languages

    print('num_speakers', speaker_manager.num_speakers)
    print('num_languages', language_manager.num_languages)

    model = GPTTrainer(config=config, speaker_manager=speaker_manager, language_manager=language_manager)
    ....

When performing inference with the speaker ID "jhone" using tts_api, I encountered the following error:
Model is not multi-speaker but speaker is provided.

Are there any best practice examples for fine-tuning xtts_v2 with a new language and multi-speaker support?

[eginhard]

There's no need for that, XTTS can already handle multiple speakers through the reference audio.

[falhafizh]

I thought I could perform inference based on speaker ID or speaker name, because the xtts_v2 model has speakers_xtts.pth, I'll try using reference audio again; I just didn't want to have to define reference_wav every time. Alright then.

[eginhard]

The speakers_xtts.pth file is just a map from the speaker names to the latents/embeddings, you could store those for your target speakers in the same way.

[falhafizh]

Yeah right, after finetune xtts_v2 I already have that.
But when inspect dir(tts) in my model.pth it doesn't exist element of tts.xxx.xxxx.speaker_manager
while the original model has tts.xxx.xxx.speaker_manager and model.is_multispeaker=True