Add dataset/sampler benchmarking script

[adamjstewart]

This PR adds a script to benchmark our GeoDataset/GeoSampler implementations. I would like to commit this script to the repo for three reasons:

1. Reproducibility: if someone wants to reproduce the results in our paper, it should be easy
2. Regression: we should run these benchmarks again before each release to make sure sampling hasn't gotten slower
3. Refinement: if someone submits a PR to improve performance, we should have a well-defined metric for comparison

Now I just need to download some data and run it. Before I run it, we should make sure these default sizes/CRS/res/etc. look reasonable.

Closes #81

[calebrob6]

Some fun stuff I found while trying to get this to work:

• The CDL dataset will think /datadrive/cdl is corrupt -- Dataset downloading expected behavior pt. 2 #99
• This line is mega broken https://github.com/microsoft/torchgeo/blob/paper/benchmarks/torchgeo/datasets/geo.py#L234
  • Specifically, this adds each tiff to the index occupying a single point in time. Now, the index will only return hits if a query has exactly the same timestamp for mint and maxt. The bounds of the index will usually look valid, but you'll never get results back (e.g. this is what happens in benchmark.py).
  • A simple workaround if you just want to benchmark stuff is to make maxt = mint + 1 year
• I don't think the RandomGeoSampler is going to work (unless you have a layer of Landsat scenes covering the extent of CDL)
• The Landsat8 dataset will fail unintuitively if all the bands aren't downloaded (and bands is set to default)

The ZipDataset looks to be working after the hack though:

[calebrob6]

Changed Broke a few things and the script works now

benchmark.py notes:

• I added a way to specify the total number of samples you want to load OR the total number of batches you want to load mainly for convenience.
• Note: previously, len(samples) was used in the dataloader to count the number of patches, but this returns the number of keys in the dict, not the batch size.
• We need to add a way to save results to file (appending to a CSV with column headers for all the args is sufficient, no need to over-engineer)
• I was wrong about the RandomGeoSampler -- it works totally fine!

[adamjstewart]

Undid some of the hacks and opened up separate PRs to fix them (#134, #138). Once those are merged I'll rebase this PR and start doing some benchmarking.

[adamjstewart]

Most recent commit fixes a couple of bugs I found:

• We were running the GridGeoSampler for N+1 batches instead of N batches
• We were using the same number of total patches even if the GridGeoSampler finished early

Results are still mostly the same, GridGeoSampler is significantly faster and I'm not sure why.

[adamjstewart]

Found another bug, we're reusing the same dataset for all samplers. For the first sampler, the cache is empty, but for later samplers it is not. So this results in an unfair comparison between samplers.

Which of the following options should we use?

1. Create a new dataset for each sampler
2. Run some random sampler first before testing all 3 samplers

Option 1 represents the average sampling rate when starting from scratch with an empty cache. Option 2 represents a better average after the cache has already been populated. I'm leaning towards 2 since the cache will be full for the vast majority of the training time when running through multiple epochs.

[adamjstewart]

We can also manually clear the cache between samplers: https://www.geeksforgeeks.org/clear-lru-cache-in-python/

[adamjstewart]

This line is left over from when I was thinking about benchmarking a segmentation model instead of ResNet. I think segmentation is one of the more common tasks in remote sensing, and the models are more complex and therefore slower. If we want to have a slower model for comparison with our data loading rates, it might be good to use something like Mask R-CNN instead.