Change allocation strategy of the distance matrix to avoid overcommit

[r4victor]

The DTWBD() function that performs sequence alignment allocates the whole distance matrix at once. This allocation may request a lot of memory. On Linux and Mac OS this is not a problem because they overcommit:

Overcommit refers to the practice of giving out virtual memory with no guarantee that physical storage for it exists).

And in practice, when the matrix is large, only elements in window are actually used. Windows simply won't allocate memory and return NULL. This caused some bugs before (Issue 1).

The solution is to change the code so that only elements in window are allocated. Like this, for example.

[ravigupta-art]

I tried this on linux as well, I still get the same issue.

[bhattarai333]

@ravigupta-art did you try turning on overcommit first? Run:

echo 1 | sudo tee /proc/sys/vm/overcommit_memory

In your terminal first.

[versae]

I had the same issue in Linux and running the code suggested by @bhattarai333 made it work for me.

$ echo 1 | sudo tee /proc/sys/vm/overcommit_memory

[WoxWik]

This fixes it for me initially, but then I get a "Killed". Error message.. How much ram is needed for a sync? Is 8gb not enough?

[r4victor]

@WoxWik, the RAM needed depends on the length of audio files. For example, when syncing a book with chapters up to an hour long (https://librivox.org/the-picture-of-dorian-gray-by-oscar-wilde/), the used memory peaks at about 1GB (tested on M1 MacBook). The used memory grows linearly with the audio length and the allocated memory grows quadratically. 8GB should be enough unless you sync much longer audio files.

[alexthesilva]

is there any workaround to this issue? i have an 8gb ram laptop and i really needed to sync a 6hour audiobook with this setup. do you happen to know of any software i could use to merge these epub files perhaps, or do you know of any way that perhaps the task could be split into various subtasks within the audiofile ram limits?

[bhattarai333]

is there any workaround to this issue? i have an 8gb ram laptop and i really needed to sync a 6hour audiobook with this setup. do you happen to know of any software i could use to merge these epub files perhaps, or do you know of any way that perhaps the task could be split into various subtasks within the audiofile ram limits?

You could maybe try splitting the audiofile (and text) into chunks and syncing them separately, then finally stitching them back together

[bhattarai333]

Finally got around to containerizing my application and can confirm setting overcommit to 1 and running works just fine in a VM as well as a Docker container on Windows