Add MultiZarrToZarr.append()

[martindurant]

cc @maxrjones

[martindurant]

@rsignell-usgs , have you had a chance to do any experimenting with this?

[rsignell]

@martindurant I've (finally) tested and both appending to JSON and Parquet are working fine!

https://nbviewer.org/gist/rsignell/ad1f3c66530a81e77585e05d9683984e

[martindurant]

parquet part requires fsspec/filesystem_spec#1510 (should be merged, then released soon)

[maxrjones]

@martindurant I added a Pythia chapter on the new feature in ProjectPythia/kerchunk-cookbook#52. It works great, but I got stuck for a while on the need for the coo_map={"time": "cf:time"}, since the original MultiZarrtoZarr worked fine without specifying that. Am I missing something regarding how the coordinate alignment works usually?

[martindurant]

need for the coo_map={"time": "cf:time"}

I can't tell you how annoying these time units are in general for someone not from the field. I used to care about leap seconds https://en.wikipedia.org/wiki/Terrestrial_Time , and barycentring, but it still seemed simpler.

the original MultiZarrtoZarr worked fine without specifying that

It ... depends. Since the normal mode is to copy the values and attributes as-is, this will be no different to "cf:" and output dtype unspecified, if the inputs happened to have the same time epoch.

[rsignell]

So just to make sure I've got it right: If all the files have the same time units (e.g. "seconds since 1970-01-01 00:00") then we don't need it, but if the files in the collection we are combining have varying time units, like ["hours since 2012-01-01 00:00", "hours since 2013-01-01 00:00"] then we need the map. Correct?

[martindurant]

Correct, but having cf: in the first case shouldn't hurt either (except that you need cftime in the env, which is a given anyway for this case)

[maxrjones]

hmm I think in ProjectPythia/kerchunk-cookbook#52 all the files had the same time units, and excluding cf: worked in the original concatenation but failed when using append. I created a small test for demonstration in #414, @martindurant should that be expected to work?