Convert compute_embeddings.py .pth file to .npy files.

[KiriKoppelgaard]

Hey,

I am trying to run the notebook PlotUmapLibriTTS.ipynb. This notebook requires a list of .npy files to visualise the speaker embeddings. However, compute_embeddings.py only generates a .pth-file. I've been trying to convert the files, doing something like this:

embed_files = torch.load(os.path.join(root, 'speakers.pth'))
for key in embed_files.keys():
    folder = key.split('/')[1]
    filename = key.split('/')[2]
    dict = embed_files[key]
    with open(os.path.join(root, 'embeddings', folder, f'{filename}.npy'), 'wb') as f:
        np.save(f, np.array(dict))
        

However, I run into problems later, because I have not saved the files in the correct format. Any insights into how the files should be organised to run PlotUmapLibriTTS.ipynb? Or simply code to convert .pth-files to a range of .npy-files?

/Thanks in advance!

[KiriKoppelgaard]

I managed to get it running by segmenting the .pth file into embeddings per utterance.
The structure is as follows:
├── embeddings
│ ├── speaker1 - .npy-files for the speaker called the same as the wav-file
│ ├── speaker2 - .npy-files for the speaker called the same as the wav-file
│ ├── speaker3 - .npy-files for the speaker called the same as the wav-file
│ └── ... - .npy-files for the speaker called the same as the wav-file

I had to change the code a little though. For future reference:

#save .pth to numpys
root = '/home/kikop/speech_finetune/speech_finetune/features/'
embed_files = torch.load(os.path.join(root, 'speakers.pth'))
print(f'Embeddings found: {len(embed_files.keys())}')

for key in embed_files.keys():
    folder = key.split('/')[1]
    filename = key.split('/')[2]
    dict = embed_files[key]['embedding']
    with open(os.path.join(root, 'embeddings', folder, f'{filename}.npy'), 'wb') as f:
        np.save(f, np.array(dict))

EMBED_PATH = os.path.join(root, 'embeddings')
embed_files = glob.glob(EMBED_PATH+"/**/*.npy", recursive=True)

speaker_paths = list(set([os.path.dirname(embed_file) for embed_file in embed_files]))

speaker_to_utter = {}
for embed_file in embed_files:
    speaker_path = os.path.dirname(embed_file)
    try:
        speaker_to_utter[speaker_path].append(embed_file)
    except:
        speaker_to_utter[speaker_path]=[embed_file]
print(f'Speaker count: {len(speaker_paths)}')

path_to_wavs = 'data/processed/DaSynth/wavs'

embeds = []
labels = []
locations = []

# single speaker 
#num_speakers = 1
#num_utters = 1000

# multi speaker
num_speakers = 20
num_utters = 10

speaker_idxs = np.random.choice(range(len(speaker_paths)), num_speakers, replace=False)

for speaker_num, speaker_idx in enumerate(speaker_idxs):
    speaker_path = speaker_paths[speaker_idx]
    speakers_utter = speaker_to_utter[speaker_path]
    utter_idxs = np.random.randint(0, len(speakers_utter) , num_utters)
    for utter_idx in utter_idxs:
        embed_path = speaker_to_utter[speaker_path][utter_idx]
        embed = np.load(embed_path, allow_pickle=True)
        embeds.append([embed])
        labels.append(str(speaker_num))
        locations.append(path_to_wavs + embed_path.replace(EMBED_PATH, '').replace('.npy','.wav'))
embeds = np.concatenate(embeds)

source_wav_stems = ColumnDataSource(
    data={'x': projection.T[0].tolist(),
          'y': projection.T[1].tolist(),
          'desc': locations,
          'label': labels,
          }
)

hover = HoverTool(
        tooltips=[
            ("file", "@desc"),
            ("speaker", "@label"),
        ]
    )

# optionally consider adding these to the tooltips if you want additional detail
# for the coordinates: ("(x,y)", "($x, $y)"),
# for the index of the embedding / wav file: ("index", "$index"),

factors = list(set(labels))
pal_size = max(len(factors), 3)
pal = Category20[pal_size]

p = figure(width=600, height=400, tools=[hover,BoxZoomTool(), ResetTool(), TapTool()])


p.circle('x', 'y',  source=source_wav_stems, color=factor_cmap('label', palette=pal, factors=factors),)

url = "http://localhost:8000/@desc"
taptool = p.select(type=TapTool)
taptool.callback = OpenURL(url=url)

show(p)

[p0p4k]

It would be very useful if you could try to make a PR out of this change, helps other users as well. Thanks.