Create raster tiles from geotiffs

[abarciauskas-bgse]

👋🏽 @freeman-lab @katamartin @jhamman

I'm interested in building a version of the next.js app which supports visualizing both COGs and Zarr. From some rudimentary inspection it looks like this might be possible by adapting the createTiles this.initialized function function to also load data from geotiffs, perhaps using a library like geotiff.js

I started to play around with the createTiles code but got stuck on what the selector hash should be (I think it's a getChunk function for a specific variable). I would appreciate any guidance on how to do add this feature!

[freeman-lab]

Hi @abarciauskas-bgse ! Great topic here. It'd be nice to support COGs in this library, so we appreciate your investigating!

The current library, and in particular the logic in tiles.js, was designed around Zarr in quite a few ways, especially in terms of assumptions about overall data layout, expected metadata, and data access patterns. That said, there should be a path forward here.

The simplest but clunkiest option might be to just write a separate implementation of tiles.js (maybe tiles-geotiff.js?) that conformed to a similar API such that it could be used similarly inside Raster but was designed around COGs instead of Zarr.

However, especially with some of the refactoring in this PR, in particular the logic @katamartin pulled into a separate Tile class, it might be possible to do this with a relatively gentle fork within the current tiles.js. As you flagged, most of the changes would be inside the this.initialized method of createTiles, and in particular defining the loader and dimensions and coordinates that get passed to Tile such that the corresponding methods of Tile (like getData) return the right bytes wrapped in the right shaped ndarray (and I agree geotiff.js looks promising as the core loading method here). Probably at least worth trying something like that. And if it starts to look really messy because of other places we've assumed Zarr, we can just try the separate implementation, and afterwards figure out how to make sure the Zarr and geotiff paths share some of the same underlying logic.

Re: the selector, that's how we specify how to index into the underlying xarray-like data structure (and we hash it so that we know not to refetch if the requested selection hasn't changed). For COGs, it might actually be simpler because there won't be multiple layers or dimensions in the same COG? Maybe you can even ignore the selector entirely?

Finally, if we add support for COGs to the @carbonplan/maps library, we'll need to figure out a way to conditionally import either zarr-js or geotiff.js, because they are each fairly large dependencies to carry around, and we don't want to pull in both when only one is actually used.

Good luck! Curious how it goes, and let us know if you have more questions!

[abarciauskas-bgse]

Thanks @freeman-lab for these details! I'm starting to dig into these libraries (regl, zarr-js, @carbonplan/maps and geotiff.js) to see if I can add an addition COG-based Tile class. Since I'm still learning each of these libraries, progress is slow, but I think I am starting to understand how the data is being loaded using zarr-js and am now inspecting how to replicate data loading from geotiffs using geotiff.js. I will update this ticket again once I have made more progress!

[abarciauskas-bgse]

For future reference, the exploratory development work I did on this is here: #41