Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handling of DSG station data for big datasets #796

Open
zequihg50 opened this issue Jul 18, 2024 · 0 comments
Open

Handling of DSG station data for big datasets #796

zequihg50 opened this issue Jul 18, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@zequihg50
Copy link

This is just to keep record of enhancements regarding the handling of station data for big datasets. As use case it is proposed that cf-python can handle a DSG dataset stored in contiguous ragged array format with +100_000 stations and more than 1 billion observations (see https://github.com/zequihg50/ghcnd-dsg).

An in-memory cache of the start and end dates registered for each station is trivial to obtain in this format, assuming that for each station the time series is sorted in disk. This allows to avoid several reading from the file for stations that actually do not contain data when subspacing in time, the slowest kind of access.

@zequihg50 zequihg50 added the enhancement New feature or request label Jul 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant