Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Initial check-in for zarr3 shard utilities #13

Closed
wants to merge 5 commits into from
Closed

Initial check-in for zarr3 shard utilities #13

wants to merge 5 commits into from

Conversation

mkitti
Copy link
Collaborator

@mkitti mkitti commented Aug 5, 2024

Add Zarr version 3 shard utilities

@mkitti
Copy link
Collaborator Author

mkitti commented Aug 5, 2024 •
edited
Loading

The main driver here is the is the functionzarr3_shard_hdf5_template, which I think I might rename.

The signature of this function is as follows:

def zarr3_shard_hdf5_template(
    shard_file_name: str,
    shard_shape: tuple[int, ...],
    chunk_shape: tuple[int, ...],
    dtype: str,
    *,
    verify: bool = False,
    delete_backup = False
) -> None

Usage looks as follows:

In [1]: import zarr3_shard_utils as zsu

In [2]: zsu.zarr3_shard_hdf5_template("fibsem-uint8/s0_hdf5/c/0/0/0", (512, 512, 512), (64, 64, 64), "uint8")

This currently expects to extract raw chunks from an existing shard and then rewrite it into a HDF5 file. The HDF5 consists of two datasets (arrays):

$ h5ls 0
zarrindex                Dataset {8196}
zarrshard                Dataset {512, 512, 512}

The zarrshard array contains the chunked data. The zarrindex array contains the Zarr shard index.

The HDF5 file is carefully crafted to position the zarrindex dataset at the end of the file. In this example file the shard is 512x512x512. The chunks within are 64x64x64. There are thus 8x8x8 or 512 chunks within the shard.

According to the Zarr 3 shard specification the Zarr 3 shard index must begin numchunks * 16 + 4 bytes from the end of the file. For the example below, the entire file is 67378873 bytes. The means the index should be at byte offset 67378873 - (512*16 + 4) which is equal to byte offset address 67370677.

The HDF5 header is is allocated via h5py's meta_block_size keyword. Currently, we allocate numchunks * 32 bytes as a header, which is 16 KiB or byte offset addrss 16384, which is the location of the first chunk.

The source Zarr version 3 shard has chunks in Morton order as is the default order of zarr-python. This utility function preserves the orders of the original shard.

$ du -b 0
67378873	0

$ h5ls -va 0
Opened "0" with sec2 driver.
zarrindex                Dataset {8196/8196}
    Location:  1:8701
    Links:     1
    Storage:   8196 logical bytes, 8196 allocated bytes, 100.00% utilization
    Type:      native unsigned char
    Address:   67370677
zarrshard                Dataset {512/512, 512/512, 512/512}
    Location:  1:195
    Links:     1
    Chunks:    {64, 64, 64} 262144 bytes
    Storage:   134217728 logical bytes, 67354293 allocated bytes, 199.27% utilization
    Filter-0:  blosc-32001 OPT {2, 2, 1, 262144, 5, 2, 5}
    Type:      native unsigned char
    Address: 463
           Flags    Bytes     Address          Logical Offset
        ========== ======== ========== ==============================
        0x00000000      442      16384 [0, 0, 0, 0]
...
Full chunk address listing 
zarrshard                Dataset {512/512, 512/512, 512/512}
    Location:  1:195
    Links:     1
    Chunks:    {64, 64, 64} 262144 bytes
    Storage:   134217728 logical bytes, 67354293 allocated bytes, 199.27% utilization
    Filter-0:  blosc-32001 OPT {2, 2, 1, 262144, 5, 2, 5}
    Type:      native unsigned char
    Address: 463
           Flags    Bytes     Address          Logical Offset
        ========== ======== ========== ==============================
        0x00000000      442      16384 [0, 0, 0, 0]

        0x00000000      442     269830 [0, 0, 64, 0]
        0x00000000      442    3339487 [0, 0, 128, 0]
        0x00000000      447    3591884 [0, 0, 192, 0]
        0x00000000      448   33739953 [0, 0, 256, 0]
        0x00000000      442   33990989 [0, 0, 320, 0]
        0x00000000      442   37037016 [0, 0, 384, 0]
        0x00000000      442   37286549 [0, 0, 448, 0]
        0x00000000   158639      17000 [0, 64, 0, 0]
        0x00000000   158463     270446 [0, 64, 64, 0]
        0x00000000   158021    3340103 [0, 64, 128, 0]
        0x00000000   157368    3592505 [0, 64, 192, 0]
        0x00000000   156978   33740575 [0, 64, 256, 0]
        0x00000000   156569   33991605 [0, 64, 320, 0]
        0x00000000   156033   37037632 [0, 64, 384, 0]
        0x00000000   156206   37287165 [0, 64, 448, 0]
        0x00000000   176248     603511 [0, 128, 0, 0]
        0x00000000   175643    1297309 [0, 128, 64, 0]
        0x00000000   175081    3923595 [0, 128, 128, 0]
        0x00000000   174827    4613761 [0, 128, 192, 0]
        0x00000000   174245   34321568 [0, 128, 256, 0]
        0x00000000   173969   35008993 [0, 128, 320, 0]
        0x00000000   173545   37615575 [0, 128, 384, 0]
        0x00000000   172672   38299397 [0, 128, 448, 0]
        0x00000000   173435     953358 [0, 192, 0, 0]
        0x00000000   173350    1646130 [0, 192, 64, 0]
        0x00000000   172694    4271373 [0, 192, 128, 0]
        0x00000000   172369    4960705 [0, 192, 192, 0]
        0x00000000   172337   34667307 [0, 192, 256, 0]
        0x00000000   171466   35354088 [0, 192, 320, 0]
        0x00000000   171189   37960272 [0, 192, 384, 0]
        0x00000000   170977   38643140 [0, 192, 448, 0]
        0x00000000   174410   11046935 [0, 256, 0, 0]
        0x00000000   172981   11741966 [0, 256, 64, 0]
        0x00000000   174765   16737974 [0, 256, 128, 0]
        0x00000000   172088   17434599 [0, 256, 192, 0]
        0x00000000   172134   44690365 [0, 256, 256, 0]
        0x00000000   169402   45387494 [0, 256, 320, 0]
        0x00000000   168883   50379738 [0, 256, 384, 0]
        0x00000000   176528   51063582 [0, 256, 448, 0]
        0x00000000   171912   11394335 [0, 320, 0, 0]
        0x00000000   174905   12088718 [0, 320, 64, 0]
        0x00000000   174336   17086364 [0, 320, 128, 0]
        0x00000000   175687   17780814 [0, 320, 192, 0]
        0x00000000   174527   45034777 [0, 320, 256, 0]
        0x00000000   174108   45729101 [0, 320, 320, 0]
        0x00000000   172941   50718119 [0, 320, 384, 0]
        0x00000000   175179   51409361 [0, 320, 448, 0]
        0x00000000   173905   13857752 [0, 384, 0, 0]
        0x00000000   176145   14570750 [0, 384, 64, 0]
        0x00000000   178133   19532272 [0, 384, 128, 0]
        0x00000000   181405   20252636 [0, 384, 192, 0]
        0x00000000   184561   47468726 [0, 384, 256, 0]
        0x00000000   184913   48196057 [0, 384, 320, 0]
        0x00000000   182240   53144948 [0, 384, 384, 0]
        0x00000000   188033   53874963 [0, 384, 448, 0]
        0x00000000   181496   14203477 [0, 448, 0, 0]
        0x00000000   181866   14920626 [0, 448, 64, 0]
        0x00000000   181718   19885430 [0, 448, 128, 0]
        0x00000000   182014   20608849 [0, 448, 192, 0]
        0x00000000   182580   47827350 [0, 448, 256, 0]
        0x00000000   182802   48558777 [0, 448, 320, 0]
        0x00000000   182908   53505876 [0, 448, 384, 0]
        0x00000000   183107   54243213 [0, 448, 448, 0]
        0x00000000      174      16826 [64, 0, 0, 0]
        0x00000000      174     270272 [64, 0, 64, 0]
        0x00000000      174    3339929 [64, 0, 128, 0]
        0x00000000      174    3592331 [64, 0, 192, 0]
        0x00000000      174   33740401 [64, 0, 256, 0]
        0x00000000      174   33991431 [64, 0, 320, 0]
        0x00000000      174   37037458 [64, 0, 384, 0]
        0x00000000      174   37286991 [64, 0, 448, 0]
        0x00000000    94191     175639 [64, 64, 0, 0]
        0x00000000    93913     428909 [64, 64, 64, 0]
        0x00000000    93760    3498124 [64, 64, 128, 0]
        0x00000000    93343    3749873 [64, 64, 192, 0]
        0x00000000    93436   33897553 [64, 64, 256, 0]
        0x00000000    93228   34148174 [64, 64, 320, 0]
        0x00000000    92884   37193665 [64, 64, 384, 0]
        0x00000000    92528   37443371 [64, 64, 448, 0]
        0x00000000   173599     779759 [64, 128, 0, 0]
        0x00000000   173178    1472952 [64, 128, 64, 0]
        0x00000000   172697    4098676 [64, 128, 128, 0]
        0x00000000   172117    4788588 [64, 128, 192, 0]
        0x00000000   171494   34495813 [64, 128, 256, 0]
        0x00000000   171126   35182962 [64, 128, 320, 0]
        0x00000000   171152   37789120 [64, 128, 384, 0]
        0x00000000   171071   38472069 [64, 128, 448, 0]
        0x00000000   170516    1126793 [64, 192, 0, 0]
        0x00000000   169885    1819480 [64, 192, 64, 0]
        0x00000000   169694    4444067 [64, 192, 128, 0]
        0x00000000   169475    5133074 [64, 192, 192, 0]
        0x00000000   169349   34839644 [64, 192, 256, 0]
        0x00000000   169004   35525554 [64, 192, 320, 0]
        0x00000000   167936   38131461 [64, 192, 384, 0]
        0x00000000   168456   38814117 [64, 192, 448, 0]
        0x00000000   172990   11221345 [64, 256, 0, 0]
        0x00000000   173771   11914947 [64, 256, 64, 0]
        0x00000000   173625   16912739 [64, 256, 128, 0]
        0x00000000   174127   17606687 [64, 256, 192, 0]
        0x00000000   172278   44862499 [64, 256, 256, 0]
        0x00000000   172205   45556896 [64, 256, 320, 0]
        0x00000000   169498   50548621 [64, 256, 384, 0]
        0x00000000   169251   51240110 [64, 256, 448, 0]
        0x00000000   175719   11566247 [64, 320, 0, 0]
        0x00000000   174723   12263623 [64, 320, 64, 0]
        0x00000000   173899   17260700 [64, 320, 128, 0]
        0x00000000   175493   17956501 [64, 320, 192, 0]
        0x00000000   178190   45209304 [64, 320, 256, 0]
        0x00000000   175969   45903209 [64, 320, 320, 0]
        0x00000000   172522   50891060 [64, 320, 384, 0]
        0x00000000   174703   51584540 [64, 320, 448, 0]
        0x00000000   171820   14031657 [64, 384, 0, 0]
        0x00000000   173731   14746895 [64, 384, 64, 0]
        0x00000000   175025   19710405 [64, 384, 128, 0]
        0x00000000   174808   20434041 [64, 384, 192, 0]
        0x00000000   174063   47653287 [64, 384, 256, 0]
        0x00000000   177807   48380970 [64, 384, 320, 0]
        0x00000000   178688   53327188 [64, 384, 384, 0]
        0x00000000   180217   54062996 [64, 384, 448, 0]
        0x00000000   185777   14384973 [64, 448, 0, 0]
        0x00000000   185340   15102492 [64, 448, 64, 0]
        0x00000000   185488   20067148 [64, 448, 128, 0]
        0x00000000   185073   20790863 [64, 448, 192, 0]
        0x00000000   186127   48009930 [64, 448, 256, 0]
        0x00000000   186220   48741579 [64, 448, 320, 0]
        0x00000000   186179   53688784 [64, 448, 384, 0]
        0x00000000   185164   54426320 [64, 448, 448, 0]
        0x00000000      172     522822 [128, 0, 0, 0]
        0x00000000      172     563160 [128, 0, 64, 0]
        0x00000000      172    3843216 [128, 0, 128, 0]
        0x00000000      172    3883442 [128, 0, 192, 0]
        0x00000000      172   34241402 [128, 0, 256, 0]
        0x00000000      172   34281575 [128, 0, 320, 0]
        0x00000000      172   37535899 [128, 0, 384, 0]
        0x00000000      172   37575779 [128, 0, 448, 0]
        0x00000000    36299     523064 [128, 64, 0, 0]
        0x00000000    36283     563402 [128, 64, 64, 0]
        0x00000000    36214    3843458 [128, 64, 128, 0]
        0x00000000    36103    3883684 [128, 64, 192, 0]
        0x00000000    36109   34241644 [128, 64, 256, 0]
        0x00000000    35950   34281817 [128, 64, 320, 0]
        0x00000000    35846   37536141 [128, 64, 384, 0]
        0x00000000    35799   37576021 [128, 64, 448, 0]
        0x00000000   169862    1989365 [128, 128, 0, 0]
        0x00000000   169943    2664532 [128, 128, 64, 0]
        0x00000000   169511    5302549 [128, 128, 128, 0]
        0x00000000   169304    5976477 [128, 128, 192, 0]
        0x00000000   168868   35694558 [128, 128, 256, 0]
        0x00000000   168651   36366327 [128, 128, 320, 0]
        0x00000000   168246   38982573 [128, 128, 384, 0]
        0x00000000   167984   39651812 [128, 128, 448, 0]
        0x00000000   167833    2329315 [128, 192, 0, 0]
        0x00000000   167947    3004294 [128, 192, 64, 0]
        0x00000000   167809    5641794 [128, 192, 128, 0]
        0x00000000   167592    6315025 [128, 192, 192, 0]
        0x00000000   167430   36032321 [128, 192, 256, 0]
        0x00000000   167415   36703404 [128, 192, 320, 0]
        0x00000000   166832   39319203 [128, 192, 384, 0]
        0x00000000   166499   39987588 [128, 192, 448, 0]
        0x00000000   171988   12438346 [128, 256, 0, 0]
        0x00000000   176619   13143809 [128, 256, 64, 0]
        0x00000000   174881   18131994 [128, 256, 128, 0]
        0x00000000   172257   18834810 [128, 256, 192, 0]
        0x00000000   172446   46079178 [128, 256, 256, 0]
        0x00000000   170162   46774547 [128, 256, 320, 0]
        0x00000000   169975   51759243 [128, 256, 384, 0]
        0x00000000   169400   52451261 [128, 256, 448, 0]
        0x00000000   181923   12784028 [128, 320, 0, 0]
        0x00000000   180216   13496377 [128, 320, 64, 0]
        0x00000000   177028   18479580 [128, 320, 128, 0]
        0x00000000   174576   19179657 [128, 320, 192, 0]
        0x00000000   174647   46424380 [128, 320, 256, 0]
        0x00000000   175806   47116743 [128, 320, 320, 0]
        0x00000000   174292   52100091 [128, 320, 384, 0]
        0x00000000   174560   52792309 [128, 320, 448, 0]
        0x00000000   175570   15287832 [128, 384, 0, 0]
        0x00000000   175811   16010688 [128, 384, 64, 0]
        0x00000000   175750   20975936 [128, 384, 128, 0]
        0x00000000   178104   21702910 [128, 384, 192, 0]
        0x00000000   175598   48927799 [128, 384, 256, 0]
        0x00000000   177151   49653630 [128, 384, 320, 0]
        0x00000000   176196   54611484 [128, 384, 384, 0]
        0x00000000   177018   55339434 [128, 384, 448, 0]
        0x00000000   186472   15638126 [128, 448, 0, 0]
        0x00000000   187448   16363012 [128, 448, 64, 0]
        0x00000000   188055   21326801 [128, 448, 128, 0]
        0x00000000   189678   22054989 [128, 448, 192, 0]
        0x00000000   188487   49276689 [128, 448, 256, 0]
        0x00000000   188475   50002819 [128, 448, 320, 0]
        0x00000000   188716   54961800 [128, 448, 384, 0]
        0x00000000   188559   55689863 [128, 448, 448, 0]
        0x00000000       70     522994 [192, 0, 0, 0]
        0x00000000       70     563332 [192, 0, 64, 0]
        0x00000000       70    3843388 [192, 0, 128, 0]
        0x00000000       70    3883614 [192, 0, 192, 0]
        0x00000000       70   34241574 [192, 0, 256, 0]
        0x00000000       70   34281747 [192, 0, 320, 0]
        0x00000000       70   37536071 [192, 0, 384, 0]
        0x00000000       70   37575951 [192, 0, 448, 0]
        0x00000000     3797     559363 [192, 64, 0, 0]
        0x00000000     3826     599685 [192, 64, 64, 0]
        0x00000000     3770    3879672 [192, 64, 128, 0]
        0x00000000     3808    3919787 [192, 64, 192, 0]
        0x00000000     3822   34277753 [192, 64, 256, 0]
        0x00000000     3801   34317767 [192, 64, 320, 0]
        0x00000000     3792   37571987 [192, 64, 384, 0]
        0x00000000     3755   37611820 [192, 64, 448, 0]
        0x00000000   170088    2159227 [192, 128, 0, 0]
        0x00000000   169819    2834475 [192, 128, 64, 0]
        0x00000000   169734    5472060 [192, 128, 128, 0]
        0x00000000   169244    6145781 [192, 128, 192, 0]
        0x00000000   168895   35863426 [192, 128, 256, 0]
        0x00000000   168426   36534978 [192, 128, 320, 0]
        0x00000000   168384   39150819 [192, 128, 384, 0]
        0x00000000   167792   39819796 [192, 128, 448, 0]
        0x00000000   167384    2497148 [192, 192, 0, 0]
        0x00000000   167246    3172241 [192, 192, 64, 0]
        0x00000000   166874    5809603 [192, 192, 128, 0]
        0x00000000   166657    6482617 [192, 192, 192, 0]
        0x00000000   166576   36199751 [192, 192, 256, 0]
        0x00000000   166197   36870819 [192, 192, 320, 0]
        0x00000000   165777   39486035 [192, 192, 384, 0]
        0x00000000   165300   40154087 [192, 192, 448, 0]
        0x00000000   173694   12610334 [192, 256, 0, 0]
        0x00000000   175949   13320428 [192, 256, 64, 0]
        0x00000000   172705   18306875 [192, 256, 128, 0]
        0x00000000   172590   19007067 [192, 256, 192, 0]
        0x00000000   172756   46251624 [192, 256, 256, 0]
        0x00000000   172034   46944709 [192, 256, 320, 0]
        0x00000000   170873   51929218 [192, 256, 384, 0]
        0x00000000   171648   52620661 [192, 256, 448, 0]
        0x00000000   177858   12965951 [192, 320, 0, 0]
        0x00000000   181159   13676593 [192, 320, 64, 0]
        0x00000000   178202   18656608 [192, 320, 128, 0]
        0x00000000   178039   19354233 [192, 320, 192, 0]
        0x00000000   175520   46599027 [192, 320, 256, 0]
        0x00000000   176177   47292549 [192, 320, 320, 0]
        0x00000000   176878   52274383 [192, 320, 384, 0]
        0x00000000   178079   52966869 [192, 320, 448, 0]
        0x00000000   174724   15463402 [192, 384, 0, 0]
        0x00000000   176513   16186499 [192, 384, 64, 0]
        0x00000000   175115   21151686 [192, 384, 128, 0]
        0x00000000   173975   21881014 [192, 384, 192, 0]
        0x00000000   173292   49103397 [192, 384, 256, 0]
        0x00000000   172038   49830781 [192, 384, 320, 0]
        0x00000000   174120   54787680 [192, 384, 384, 0]
        0x00000000   173411   55516452 [192, 384, 448, 0]
        0x00000000   186090   15824598 [192, 448, 0, 0]
        0x00000000   187514   16550460 [192, 448, 64, 0]
        0x00000000   188054   21514856 [192, 448, 128, 0]
        0x00000000   188315   22244667 [192, 448, 192, 0]
        0x00000000   188454   49465176 [192, 448, 256, 0]
        0x00000000   188444   50191294 [192, 448, 320, 0]
        0x00000000   188918   55150516 [192, 448, 384, 0]
        0x00000000   189117   55878422 [192, 448, 448, 0]
        0x00000000       70    6649274 [256, 0, 0, 0]
        0x00000000       70    6649554 [256, 0, 64, 0]
        0x00000000       70    8852342 [256, 0, 128, 0]
        0x00000000       70    8852622 [256, 0, 192, 0]
        0x00000000       70   40319387 [256, 0, 256, 0]
        0x00000000       70   40319667 [256, 0, 320, 0]
        0x00000000       70   42507980 [256, 0, 384, 0]
        0x00000000       70   42508260 [256, 0, 448, 0]
        0x00000000       70    6649414 [256, 64, 0, 0]
        0x00000000       70    6649694 [256, 64, 64, 0]
        0x00000000       70    8852482 [256, 64, 128, 0]
        0x00000000       70    8852762 [256, 64, 192, 0]
        0x00000000       70   40319527 [256, 64, 256, 0]
        0x00000000       70   40319807 [256, 64, 320, 0]
        0x00000000       70   42508120 [256, 64, 384, 0]
        0x00000000       70   42508400 [256, 64, 448, 0]
        0x00000000   155623    6653068 [256, 128, 0, 0]
        0x00000000   155200    7268012 [256, 128, 64, 0]
        0x00000000   155161    8856136 [256, 128, 128, 0]
        0x00000000   154582    9468650 [256, 128, 192, 0]
        0x00000000   154285   40323181 [256, 128, 256, 0]
        0x00000000   154259   40934491 [256, 128, 320, 0]
        0x00000000   154048   42511774 [256, 128, 384, 0]
        0x00000000   153803   43120831 [256, 128, 448, 0]
        0x00000000   166657    6935329 [256, 192, 0, 0]
        0x00000000   166579    7549872 [256, 192, 64, 0]
        0x00000000   165896    9137527 [256, 192, 128, 0]
        0x00000000   165818    9749480 [256, 192, 192, 0]
        0x00000000   165607   40603531 [256, 192, 256, 0]
        0x00000000   165665   41214643 [256, 192, 320, 0]
        0x00000000   165178   42791270 [256, 192, 384, 0]
        0x00000000   165274   43400040 [256, 192, 448, 0]
        0x00000000   170545   22432982 [256, 256, 0, 0]
        0x00000000   172967   23122507 [256, 256, 64, 0]
        0x00000000   171584   28081330 [256, 256, 128, 0]
        0x00000000   170394   28780933 [256, 256, 192, 0]
        0x00000000   169789   56067539 [256, 256, 256, 0]
        0x00000000   169365   56757693 [256, 256, 320, 0]
        0x00000000   169356   61718047 [256, 256, 384, 0]
        0x00000000   169527   62410728 [256, 256, 448, 0]
        0x00000000   175388   22771907 [256, 320, 0, 0]
        0x00000000   179799   23465171 [256, 320, 64, 0]
        0x00000000   179036   28423369 [256, 320, 128, 0]
        0x00000000   175220   29120385 [256, 320, 192, 0]
        0x00000000   175247   56405170 [256, 320, 256, 0]
        0x00000000   175527   57095302 [256, 320, 320, 0]
        0x00000000   181309   62055871 [256, 320, 384, 0]
        0x00000000   181409   62748638 [256, 320, 448, 0]
        0x00000000   173753   25191331 [256, 384, 0, 0]
        0x00000000   177435   25911105 [256, 384, 64, 0]
        0x00000000   176181   30841062 [256, 384, 128, 0]
        0x00000000   174307   31567444 [256, 384, 192, 0]
        0x00000000   173769   58819759 [256, 384, 256, 0]
        0x00000000   171995   59544458 [256, 384, 320, 0]
        0x00000000   172267   64472342 [256, 384, 384, 0]
        0x00000000   172258   65196095 [256, 384, 448, 0]
        0x00000000   186860   25537563 [256, 448, 0, 0]
        0x00000000   188076   26260950 [256, 448, 64, 0]
        0x00000000   189085   31189179 [256, 448, 128, 0]
        0x00000000   189384   31914518 [256, 448, 192, 0]
        0x00000000   189345   59165169 [256, 448, 256, 0]
        0x00000000   189558   59887245 [256, 448, 320, 0]
        0x00000000   189398   64816197 [256, 448, 384, 0]
        0x00000000   190153   65540333 [256, 448, 448, 0]
        0x00000000       70    6649344 [320, 0, 0, 0]
        0x00000000       70    6649624 [320, 0, 64, 0]
        0x00000000       70    8852412 [320, 0, 128, 0]
        0x00000000       70    8852692 [320, 0, 192, 0]
        0x00000000       70   40319457 [320, 0, 256, 0]
        0x00000000       70   40319737 [320, 0, 320, 0]
        0x00000000       70   42508050 [320, 0, 384, 0]
        0x00000000       70   42508330 [320, 0, 448, 0]
        0x00000000       70    6649484 [320, 64, 0, 0]
        0x00000000       70    6649764 [320, 64, 64, 0]
        0x00000000       70    8852552 [320, 64, 128, 0]
        0x00000000       70    8852832 [320, 64, 192, 0]
        0x00000000       70   40319597 [320, 64, 256, 0]
        0x00000000       70   40319877 [320, 64, 320, 0]
        0x00000000       70   42508190 [320, 64, 384, 0]
        0x00000000       70   42508470 [320, 64, 448, 0]
        0x00000000   126638    6808691 [320, 128, 0, 0]
        0x00000000   126660    7423212 [320, 128, 64, 0]
        0x00000000   126230    9011297 [320, 128, 128, 0]
        0x00000000   126248    9623232 [320, 128, 192, 0]
        0x00000000   126065   40477466 [320, 128, 256, 0]
        0x00000000   125893   41088750 [320, 128, 320, 0]
        0x00000000   125448   42665822 [320, 128, 384, 0]
        0x00000000   125406   43274634 [320, 128, 448, 0]
        0x00000000   166026    7101986 [320, 192, 0, 0]
        0x00000000   166143    7716451 [320, 192, 64, 0]
        0x00000000   165227    9303423 [320, 192, 128, 0]
        0x00000000   165606    9915298 [320, 192, 192, 0]
        0x00000000   165353   40769138 [320, 192, 256, 0]
        0x00000000   164837   41380308 [320, 192, 320, 0]
        0x00000000   164383   42956448 [320, 192, 384, 0]
        0x00000000   164544   43565314 [320, 192, 448, 0]
        0x00000000   168380   22603527 [320, 256, 0, 0]
        0x00000000   169697   23295474 [320, 256, 64, 0]
        0x00000000   170455   28252914 [320, 256, 128, 0]
        0x00000000   169058   28951327 [320, 256, 192, 0]
        0x00000000   167842   56237328 [320, 256, 256, 0]
        0x00000000   168244   56927058 [320, 256, 320, 0]
        0x00000000   168468   61887403 [320, 256, 384, 0]
        0x00000000   168383   62580255 [320, 256, 448, 0]
        0x00000000   175212   22947295 [320, 320, 0, 0]
        0x00000000   180807   23644970 [320, 320, 64, 0]
        0x00000000   178528   28602405 [320, 320, 128, 0]
        0x00000000   178562   29295605 [320, 320, 192, 0]
        0x00000000   177276   56580417 [320, 320, 256, 0]
        0x00000000   177194   57270829 [320, 320, 320, 0]
        0x00000000   173548   62237180 [320, 320, 384, 0]
        0x00000000   174945   62930047 [320, 320, 448, 0]
        0x00000000   172479   25365084 [320, 384, 0, 0]
        0x00000000   172410   26088540 [320, 384, 64, 0]
        0x00000000   171936   31017243 [320, 384, 128, 0]
        0x00000000   172767   31741751 [320, 384, 192, 0]
        0x00000000   171641   58993528 [320, 384, 256, 0]
        0x00000000   170792   59716453 [320, 384, 320, 0]
        0x00000000   171588   64644609 [320, 384, 384, 0]
        0x00000000   171980   65368353 [320, 384, 448, 0]
        0x00000000   186682   25724423 [320, 448, 0, 0]
        0x00000000   188422   26449026 [320, 448, 64, 0]
        0x00000000   189180   31378264 [320, 448, 128, 0]
        0x00000000   189632   32103902 [320, 448, 192, 0]
        0x00000000   189944   59354514 [320, 448, 256, 0]
        0x00000000   189759   60076803 [320, 448, 320, 0]
        0x00000000   190500   65005595 [320, 448, 384, 0]
        0x00000000   190018   65730486 [320, 448, 448, 0]
        0x00000000      100    6649834 [384, 0, 0, 0]
        0x00000000      100    6651451 [384, 0, 64, 0]
        0x00000000      100    8852902 [384, 0, 128, 0]
        0x00000000      100    8854519 [384, 0, 192, 0]
        0x00000000      100   40319947 [384, 0, 256, 0]
        0x00000000      100   40321564 [384, 0, 320, 0]
        0x00000000      100   42508540 [384, 0, 384, 0]
        0x00000000      100   42510157 [384, 0, 448, 0]
        0x00000000      245    6650554 [384, 64, 0, 0]
        0x00000000      245    6652171 [384, 64, 64, 0]
        0x00000000      245    8853622 [384, 64, 128, 0]
        0x00000000      245    8855239 [384, 64, 192, 0]
        0x00000000      245   40320667 [384, 64, 256, 0]
        0x00000000      245   40322284 [384, 64, 320, 0]
        0x00000000      245   42509260 [384, 64, 384, 0]
        0x00000000      245   42510877 [384, 64, 448, 0]
        0x00000000    95884    7882594 [384, 128, 0, 0]
        0x00000000    95727    8367540 [384, 128, 64, 0]
        0x00000000    95428   10080904 [384, 128, 128, 0]
        0x00000000    95364   10564321 [384, 128, 192, 0]
        0x00000000    95076   41545145 [384, 128, 256, 0]
        0x00000000    94913   42027183 [384, 128, 320, 0]
        0x00000000    94786   43729858 [384, 128, 384, 0]
        0x00000000    94554   44210697 [384, 128, 448, 0]
        0x00000000   165821    8036225 [384, 192, 0, 0]
        0x00000000   166102    8521085 [384, 192, 64, 0]
        0x00000000   165692   10233656 [384, 192, 128, 0]
        0x00000000   165308   10716845 [384, 192, 192, 0]
        0x00000000   164996   41697507 [384, 192, 256, 0]
        0x00000000   164538   42179269 [384, 192, 320, 0]
        0x00000000   164774   43881603 [384, 192, 384, 0]
        0x00000000   164640   44362209 [384, 192, 448, 0]
        0x00000000   168216   23825777 [384, 256, 0, 0]
        0x00000000   167020   24509774 [384, 256, 64, 0]
        0x00000000   167206   29474167 [384, 256, 128, 0]
        0x00000000   167295   30156613 [384, 256, 192, 0]
        0x00000000   166952   57448023 [384, 256, 256, 0]
        0x00000000   167899   58132418 [384, 256, 320, 0]
        0x00000000   166455   63104992 [384, 256, 384, 0]
        0x00000000   166597   63787885 [384, 256, 448, 0]
        0x00000000   175169   24160490 [384, 320, 0, 0]
        0x00000000   172879   24843158 [384, 320, 64, 0]
        0x00000000   174329   29807400 [384, 320, 128, 0]
        0x00000000   177797   30489784 [384, 320, 192, 0]
        0x00000000   176105   57780384 [384, 320, 256, 0]
        0x00000000   177921   58465820 [384, 320, 320, 0]
        0x00000000   176785   63436244 [384, 320, 384, 0]
        0x00000000   178070   64118974 [384, 320, 448, 0]
        0x00000000   174646   26637448 [384, 384, 0, 0]
        0x00000000   178054   27354481 [384, 384, 64, 0]
        0x00000000   171687   32293534 [384, 384, 128, 0]
        0x00000000   170911   33015799 [384, 384, 192, 0]
        0x00000000   172602   60266562 [384, 384, 256, 0]
        0x00000000   172371   60991932 [384, 384, 320, 0]
        0x00000000   172180   65920504 [384, 384, 384, 0]
        0x00000000   171668   66644559 [384, 384, 448, 0]
        0x00000000   185077   26984968 [384, 448, 0, 0]
        0x00000000   188805   27706955 [384, 448, 64, 0]
        0x00000000   189438   32637961 [384, 448, 128, 0]
        0x00000000   189888   33359339 [384, 448, 192, 0]
        0x00000000   189784   60611487 [384, 448, 256, 0]
        0x00000000   190488   61336935 [384, 448, 320, 0]
        0x00000000   190242   66263695 [384, 448, 384, 0]
        0x00000000   190492   66989161 [384, 448, 448, 0]
        0x00000000      620    6649934 [448, 0, 0, 0]
        0x00000000      620    6651551 [448, 0, 64, 0]
        0x00000000      620    8853002 [448, 0, 128, 0]
        0x00000000      620    8854619 [448, 0, 192, 0]
        0x00000000      620   40320047 [448, 0, 256, 0]
        0x00000000      620   40321664 [448, 0, 320, 0]
        0x00000000      620   42508640 [448, 0, 384, 0]
        0x00000000      620   42510257 [448, 0, 448, 0]
        0x00000000      652    6650799 [448, 64, 0, 0]
        0x00000000      652    6652416 [448, 64, 64, 0]
        0x00000000      652    8853867 [448, 64, 128, 0]
        0x00000000      652    8855484 [448, 64, 192, 0]
        0x00000000      652   40320912 [448, 64, 256, 0]
        0x00000000      652   40322529 [448, 64, 320, 0]
        0x00000000      652   42509505 [448, 64, 384, 0]
        0x00000000      652   42511122 [448, 64, 448, 0]
        0x00000000    57747    7978478 [448, 128, 0, 0]
        0x00000000    57818    8463267 [448, 128, 64, 0]
        0x00000000    57324   10176332 [448, 128, 128, 0]
        0x00000000    57160   10659685 [448, 128, 192, 0]
        0x00000000    57286   41640221 [448, 128, 256, 0]
        0x00000000    57173   42122096 [448, 128, 320, 0]
        0x00000000    56959   43824644 [448, 128, 384, 0]
        0x00000000    56958   44305251 [448, 128, 448, 0]
        0x00000000   165494    8202046 [448, 192, 0, 0]
        0x00000000   165155    8687187 [448, 192, 64, 0]
        0x00000000   164973   10399348 [448, 192, 128, 0]
        0x00000000   164782   10882153 [448, 192, 192, 0]
        0x00000000   164680   41862503 [448, 192, 256, 0]
        0x00000000   164173   42343807 [448, 192, 320, 0]
        0x00000000   164320   44046377 [448, 192, 384, 0]
        0x00000000   163516   44526849 [448, 192, 448, 0]
        0x00000000   166497   23993993 [448, 256, 0, 0]
        0x00000000   166364   24676794 [448, 256, 64, 0]
        0x00000000   166027   29641373 [448, 256, 128, 0]
        0x00000000   165876   30323908 [448, 256, 192, 0]
        0x00000000   165409   57614975 [448, 256, 256, 0]
        0x00000000   165503   58300317 [448, 256, 320, 0]
        0x00000000   164797   63271447 [448, 256, 384, 0]
        0x00000000   164492   63954482 [448, 256, 448, 0]
        0x00000000   174115   24335659 [448, 320, 0, 0]
        0x00000000   175294   25016037 [448, 320, 64, 0]
        0x00000000   174884   29981729 [448, 320, 128, 0]
        0x00000000   173481   30667581 [448, 320, 192, 0]
        0x00000000   175929   57956489 [448, 320, 256, 0]
        0x00000000   176018   58643741 [448, 320, 320, 0]
        0x00000000   174856   63613029 [448, 320, 384, 0]
        0x00000000   175298   64297044 [448, 320, 448, 0]
        0x00000000   172874   26812094 [448, 384, 0, 0]
        0x00000000   174420   27532535 [448, 384, 64, 0]
        0x00000000   172740   32465221 [448, 384, 128, 0]
        0x00000000   172629   33186710 [448, 384, 192, 0]
        0x00000000   172323   60439164 [448, 384, 256, 0]
        0x00000000   172632   61164303 [448, 384, 320, 0]
        0x00000000   171011   66092684 [448, 384, 384, 0]
        0x00000000   172934   66816227 [448, 384, 448, 0]
        0x00000000   184436   27170045 [448, 448, 0, 0]
        0x00000000   185570   27895760 [448, 448, 64, 0]
        0x00000000   188400   32827399 [448, 448, 128, 0]
        0x00000000   190726   33549227 [448, 448, 192, 0]
        0x00000000   190661   60801271 [448, 448, 256, 0]
        0x00000000   190624   61527423 [448, 448, 320, 0]
        0x00000000   190622   66453937 [448, 448, 384, 0]
        0x00000000   191024   67179653 [448, 448, 448, 0]

@rhoadesScholar
Copy link
Member

This ready for review?
Are we planning to have a full Zarr2 to HDF5/Zarr3 sharded format in a subsequent PR?

@mkitti
Implement min_meta_block_size function for small nchunks
3662873 
Add some additional documentation and typing information
@mkitti
Apply Black
5d7cfb6
@codecov Codecov
Copy link

codecov bot commented Aug 13, 2024

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered ☂️

@mkitti
Copy link
Collaborator Author

mkitti commented Aug 13, 2024

Almost there. I'm adjusting a few things since I realized that n5-hdf5 might be the target consumer of the HDF5 files.

@mkitti
Copy link
Collaborator Author

mkitti commented Aug 13, 2024

Are we committing to using tensorstore as the main driver for Zarr?

@mkitti
Copy link
Collaborator Author

mkitti commented Aug 13, 2024

I'm vascillating on if the functions should have a prefix of "zarr3_shard" or not.

@mkitti
Copy link
Collaborator Author

mkitti commented Sep 5, 2024

@rhoadesScholar Do you have a second to touch base on this?

@mkitti
Copy link
Collaborator Author

mkitti commented Oct 7, 2024 •
edited
Loading

Closing since this is not currently needed. Happy to reopen if needed.

@mkitti mkitti closed this Oct 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rhoadesScholar rhoadesScholar Awaiting requested review from rhoadesScholar

@d-v-b d-v-b Awaiting requested review from d-v-b

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@mkitti @rhoadesScholar