Whether can be in "datasets. GeneratorBasedBuilder. _generate_examples" load model in data processing. When using non-streaming load_dataset, whether the model can be removed to avoid VRAM occupation

[aihao2000]

For some specific configurations of the dataset, I needed to load the model for preprocessing. At first I import my own package in the header and sys.path.append, error, seems to be some detection to download but can not find, I can now just put "import" inside the if:

if not hasattr(self, "anime_segmentation_model"):
        from a import b

And now that it's working, I was wondering if and how I could remove this model that was used to preprocess the data at training time. I registered it to self in _generate_examples. Is it possible to del dataset.mymodel after computing load_dataset?

[mariosasko]

Yes, feel free to do this. The only thing non-streaming HF datasets depend on are Arrow files that are memory-mapped when they are loaded (the .cache_files attribute returns them)