add snowstorm_dataset and IceCubehosted class #783
Conversation
sevmag
commented
Jan 27, 2025
sevmag
commented
- Addition of Curated Datasets hosted on the IceCube cluster. (Download requires Username and Password)
- Implementation of the curated IceCube-hosted Snowstorm dataset
| validation_dataloader_kwargs: Optional[Dict[str, Any]] = None, | ||
| test_dataloader_kwargs: Optional[Dict[str, Any]] = None, | ||
| ): | ||
| """Initialize SnowStorm dataset.""" |
There was a problem hiding this comment.
I think we should explain the arguments here. Most can be repeated from the parent class but run_ids is new.
There was a problem hiding this comment.
Thank you for this very clean contribution @sevmag!
Before we proceed, do you have comments on this @Aske-Rosted? If you have a wiki for the dataset/conversion, maybe we should link to that instead of the generic snowstorm wiki?
| assert match | ||
| run_id = match.group(1) | ||
|
|
||
| query_df = query_database( |
There was a problem hiding this comment.
How fast is this query in your experience? If the database is large, it might take minutes to execute. In that case, we could consider providing a .parquet file with the ids in them.
There was a problem hiding this comment.
Yeah, you are right. It depends on the files, but some take over 2 minutes, and it's probably the query.
There was a problem hiding this comment.
UPDATE: The bottleneck of initializing this SnowStormDataset implementation is mainly in the initialization called here (especially when selecting RunIDs with a lot of files)
graphnet/src/graphnet/data/curated_datamodule.py
Lines 76 to 85 in 79d7baf
Contrary to my belief that it was the prepare_args function snipped referenced above. Therefore, I decided to stick to this version, keeping in mind that one can make the prepare_args function more efficient.
| """ | ||
|
|
||
| _experiment = "IceCube SnowStorm dataset" | ||
| _creator = "Severin Magel" |
There was a problem hiding this comment.
Should probably mention @Aske-Rosted here :-)
| """Initialize SnowStorm dataset.""" | ||
| self._run_ids = run_ids | ||
| self._zipped_files = [ | ||
| os.path.join(self._data_root_dir, f"{s}.tar.gz") for s in run_ids |
There was a problem hiding this comment.
I think there's currently 28 valid run id's, which is a subset of the overall available sets. I think the current code assumes the user knows this. Perhaps it would be wise to include an assertion that checks that the user-given ids actually exists.
