How does tar work with DynamicBucketingSampler?

[orena1]

I am trying to understand how BucketingSampler work with tar data.
BucketingSampler try to make sure that each batch has max_tokens of X, and that all samples in the batch are about the same size (to avoid padding).
If we use random access it is relatively easy to calculate in advance the samples to load in each batch from the metadata (e.g. sort by length and create batches ok max_token_size).

But the efficiency in tars is that they are read sequentially, no random access, but then if they are ready sequentially how can you verify that all samples in a batch have similar length (to avoid padding)?
What is the mechanism exactly ?
Thanks
@oplatek ?

[pzelasko]

Search for DynamicBucketingSampler. It holds an in-memory buffer for data that is partitioned into num_buckets.

[orena1]

Thanks @pzelasko, So it keeps s buffer or different sizes in Ram and then select from that, the buffer is already close to what is needed because it draw from buckets - Smart!!.
Probably hard to implement in video-datasets, because in video the memory requirements will be very big. Right?

[pzelasko]

I think it should work. Maybe use less buckets and smaller buffer sizes. With tar/shar the data is held in memory as the original file so if it’s mp4 or so it will still benefit from the compression before you call load_video.