Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider using smaller chunk size (64^3) and jpeg encoding #34

Open
jbms opened this issue Nov 17, 2020 · 1 comment
Open

Consider using smaller chunk size (64^3) and jpeg encoding #34

jbms opened this issue Nov 17, 2020 · 1 comment

Comments

@jbms
Copy link

jbms commented Nov 17, 2020

You may find 64^3 chunks provide better visualization performance than 96^3, since for cross section views you end up loading 50% more data for the initial display with 96^3 chunks. You may also find it helpful to use jpeg compression for uint8 data --- I see some volumes are using n5 with gzip compression --- I know we already discussed about the lack of n5/zarr jpeg compression support in Neuroglancer and tensorstore, though.

@d-v-b
Copy link
Contributor

d-v-b commented Nov 17, 2020

I admit I picked 96^3 somewhat arbitrarily in an attempt to balance loading / visualization performance against the cost of a lot of tiny objects in storage. I should do a more thorough comparison.

As for datatypes, I think all the raw uint8 data are stored with jpeg compression in precomputed format, albeit unsharded (until tensorstore has a better API for saving sharded volumes with parallelism over shard files :) ). For some of the analysis volumes (predictions) I opted for the lossless compression in case someone wants to download those volumes for re-analysis.

jpeg compression for n5/zarr would be great, I have a stalled PR that would enable this for n5: zarr-developers/zarr-python#577. As soon as that gets some traction I will probably open an issue / PR on neuroglancer to support this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants