Compression of labels

[pranjaldhole]

The label writer uses a compression method which is now outdated:

napari-hdf5-labels-io/napari_hdf5_labels_io/_writer.py

Lines 126 to 167 in e0276d3

	def compress_layer(layer_array: np.array):
	"""
	Returns compressed version of the original numpy array
	based on memory efficient compression.
	Parameters
	----------
	layer_array: Numpy array
	Original array version of the layer data
	Returns
	-------
	layer_shape: tuple
	Tuple containing shape of the layer_array
	compressed_layer: np.array
	Sparse version (COO list) of the layer data, if is_sparse == True
	zarr array with zstd compression, if is_sparse == False
	is_sparse: bool
	True if Napari layer should be represented in COO list.
	"""
	layer_shape = tuple(layer_array.shape)
	# USE COORD LIST FOR SPARSE LABELLING
	tmp_coo = sparse.COO(layer_array)
	# join coords and data in single array
	coo_array = np.append(tmp_coo.coords,
	np.array([tmp_coo.data]), axis=0)
	coo_array = coo_array.astype(np.uint16)
	# USE ZSTD COMPRESSION FOR PREDICTIONS, higher clevel takes more time.
	zarr_chunks = tuple([1 for i in range(len(layer_array.shape) - 2)] + [1024, 1024])
	compressor = Blosc(cname='zstd', clevel=3, shuffle=Blosc.BITSHUFFLE)
	data = layer_array.astype(np.uint16)
	zarr_array = zarr.array(data, chunks=zarr_chunks, compressor=compressor)
	if zarr_array.nbytes_stored >= coo_array.nbytes:
	compressed_layer = coo_array
	is_sparse = True
	else:
	compressed_layer = zarr_array
	is_sparse = False
	return layer_shape, compressed_layer, is_sparse

Saving dictionary of coordinate lists is more memory-consuming than compressing the label arrays with HDF5 compressor.

To Do:

• Remove the coordinate list save method.
• Compress the label arrays with HDF5 compressor.