Replies: 3 comments 5 replies
-
|
I've managed to reduce the leak by setting |
Beta Was this translation helpful? Give feedback.
-
|
Hi @mdespriee and welcome! That sounds like a frustrating problem. We'd love to help you debug. What you're doing should be possible and stable. But there are several things that can go wrong to cause the sort of behavior you're seeing. The most important factor is the chunking of the array and the shape of the updates. Can you share more details about the configuration of the underlying Zarr groups and arrays--shape, dtype, and chunk shape--and also the shape of the data you are appending? Some code examples for how you are creating the initial dataset and how you are doing the append would be helpful as well. |
Beta Was this translation helpful? Give feedback.
-
|
Each append is a Dataset like this: (shown decoded) When a new array is created (a new group), it's with this spec in encoding: The creation is done like this: The appending is done with a simple: store is opened this way: Back to the error message I mentioned above ( |
Beta Was this translation helpful? Give feedback.
-
|
A few minor follow-up questions:
|
Beta Was this translation helpful? Give feedback.
-
|
Final question for now (sorry! 😆 ): do you know that the |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @rabernat for your help. Chunks are 2000x2000, because it's the same code for several sources with various sizes (each source going in it's own zarr), biggest one being close to 4000x4000. So I peaked something that looked reasonable, like 4MB per chunk. And yes, coordinates are identical in my images. But i'm curious of what I should do if it was not the case ? About the open_group, indeed it's not direct copy-paste, but I do use this context-manager. I'll try without it right away. And here is the full stack-trace of error. Note that this is happening after the crash, in a subsequent read operation (a read that I do to check the array before executing a new append) |
Beta Was this translation helpful? Give feedback.
-
|
Hi @mdespriee I took some time to try to create a minimal reproducer for your scenario. I was not able to observe either the error you described nor an obvious memory leak. Here's what I did import xarray as xr
import numpy as np
import zarr
from datetime import datetime, timedelta
import s3fs
def make_sample(time_offset_min = 0):
"""Generate a single sample image"""
nt, ny, nx = (1, 1600, 1600)
xrange = (1.078e6, 6.394e5)
yrange = (-4.17e6, -5.884e6)
ref_time = datetime(2023, 10, 1)
da = xr.DataArray(
np.random.rand(nt, ny, nx),
dims=('time', 'y', 'x'),
coords={
'time': [ref_time + timedelta(minutes=time_offset_min)],
'y': np.linspace(yrange[0], yrange[1], ny),
'x': np.linspace(xrange[0], xrange[1], nx)
},
name="precipitation_type"
)
ds = da.to_dataset()
return ds
# the datset we will use to initialize the store
ds = make_sample()
# set some encoding for initial writing
ds.time.encoding['units'] = 'seconds since 1970-01-01 00:00:00'
ds.precipitation_type.encoding['chunks'] = ds.precipitation_type.shape
ds.x.encoding['chunks'] = ds.x.shape
ds.y.encoding['chunks'] = ds.y.shape
# NB: change this to an s3 bucket you have write access to
s3_store = zarr.storage.FSStore("s3://earthmover-test/append-test/")
s3_store.clear()
%time ds.to_zarr(s3_store, consolidated=False)
# -> Wall time: 3.42 s
# this is unncessarily slow; we are working to make it faster...but it works
# used https://pythonspeed.com/fil/docs/index.html to profile memory usage for appending 20 images
%%filprofile
for n in range(20):
make_sample(n * 5).to_zarr(s3_store, append_dim="time", consolidated=False)
# took about 60 s, consistent with above
The peak memory usage of 57 MB seems invariant to the length of the loop (e.g. if I use So at least in this example, I have verified that I can stably append to a Zarr store in S3 using the data with the same shape you described in your post. I'm curious:
|
Beta Was this translation helpful? Give feedback.
-
|
I'm probably having other problems in that app, in addition to the OOM. The relaunch of some failed processing might have lead to concurrent writes, explaining the strange behavior. Thanks @rabernat for taking the time |
Beta Was this translation helpful? Give feedback.
-
Hello,
I'm exploring the use of xarray+zarr to store raster data that is arriving every 5mn.
I'm storing that data by continuously appending to a zarr store in s3.
I have a processing app, that receives the file, reads the raster, prepares an xr.Dataset from it, and appends it to the zarr in s3.
I'm facing 2 problems here:
first is a memory leak. The memory usage of this processing app is ever growing, up to a SIGKILL from the system (
Worker (pid:96) was sent SIGKILL! Perhaps out of memory?), and then starts over. So far, I haven't any clue about the object holding all that data. Is it the zarr store object ? is it at fsspec/s3fs level ? How can I pin-point the cause and work around it ? Note that I tried to recycle top-level objects (zarr store, etc...) but with no success. There are some things kept in cache behind the scene.second is a robustness/resilience problem. One of the OOM crash occured during a write to s3. Now, the zarr is corrupted:
... in merge_data_and_coords ... ValueError: conflicting sizes for dimension 'time': length 69 on 'time' and length 68 on {'time': 'precipitation_type', 'y': 'precipitation_type', 'x': 'precipitation_type'}. Is there a way to recover / repair the dataset ? Even if I fix the OOM above, a crash during a write will happen again. Can I force xarray to ignore that kind of error (eg. force NaN on missing data) and allow future writes ?Thanks for your insights, folks !
Beta Was this translation helpful? Give feedback.
All reactions