Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tickets/SP-1623: refactor metadata storage in the prenight scheduler simulation archive to improve performance #117

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

ehneilsen
Copy link
Collaborator

This modification creates a cache of metadata in an hdf5 file in the same bucket as the sims.
When querying for metadata, the code now checks the cached metadata, and looks in the cache for metadata for any dates before the last one in the cache. For dates of the last one in the cache or later, it falls back to the older behavior of lookinng in the yaml metadata files for each sim. ("Of the last date" rather than "after the last date" in case the cache does not include all sims from the last date.)
A separate cron job updates the cache daily.
A corresponding branch of schedview_notebooks causes the prenight_sims notebook to use the cache.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant