Skip to content

Commit

Permalink
Update TRAINING.md
Browse files Browse the repository at this point in the history
  • Loading branch information
hkchengrex authored Dec 23, 2024
1 parent d16dd23 commit 576e29c
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion docs/TRAINING.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Namely, before starting any training, we
4. Encode all extracted features into [MemoryMappedTensors](https://pytorch.org/tensordict/main/reference/generated/tensordict.MemoryMappedTensor.html) with [TensorDict](https://pytorch.org/tensordict/main/reference/tensordict.html)


**NOTE:** for maximum training speed (e.g., when training the base model with 2*H100s), you would need around 3~5 GB/s of random read speed. Spinning disks would not be able to catch up and most SSDs would struggle. In my experience, the best bet is to have a large enough system memory such that the OS can cache the data. This way, the data is read from RAM instead of disk.
**NOTE:** for maximum training speed (e.g., when training the base model with 2*H100s), you would need around 3~5 GB/s of random read speed. Spinning disks would not be able to catch up and most consumer-grade SSDs would struggle. In my experience, the best bet is to have a large enough system memory such that the OS can cache the data. This way, the data is read from RAM instead of disk.

## Preparing Audio-Video-Text Features

Expand Down

0 comments on commit 576e29c

Please sign in to comment.