Timeseries types and statistics #1251
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Adds the ability to calculate the mix and max values for all timeseries, and for tidal data types it will try to figure out the highs and lows in the requested data.
For forecast or prediction timeseries, the next weeks worth of data is pulled instead of the last days, so we can get the extrema from that window. Additionally we also grab the next value, rather than the last one, so it's closer to the current time. We probably want to change that to a shorter window though.
|
This now uses the more reliable algorithm for finding high/low tides recommended by Hannah. |
Adelle-Pitsas
approved these changes
Nov 19, 2024
abkfenris
added a commit
that referenced
this pull request
Nov 19, 2024
Additions: - Timeseries types and extrema in [#1251](#1251) - Timeseries can now be classified into different types (observation, forecast, prediction) which will alter how we fetch data for them. For observations we will continue to use the last 24 hours and the latest value from that, while forecasts and predictions we will fetch the next 7 days and use the first value. - We're calculating the min and max of each timeseries, and for tidal timeseries we are calculating the high and low tides over the fetched data. Changes: - Added 'Overland Flood' as a platform type in [#1251] Fixes: - Take care of some N+1 queries in `refresh_dataset` [#1245](#1245) - Adjusteda few commands that needed `uv run` to be prefixed. Dependency Updates: - Pre-commit - Pyproject Format from 2.4.3 to 2.5.0 - Ruff from 0.7.1 to 0.7.4 - Dockerfile from 1.10 to 1.11 - Python - Sentry SDK from 2.17.0 to 2.18.0 - Py.test Coverage from 5.0 to 6.0
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Initial Model Changes:
Min/Max and Tide Extrema Calculation:
Forecast Data Retrieval:
Nearest Forecast Utilization:
These changes aim to enhance the functionality and accuracy of the timeseries data handling, providing better insights for users working with flooding and tidal data. Further adjustments may be needed to optimize the data window for forecasts.
{ "id": "Chebeague Island tide gauge", "type": "Feature", "geometry": { "type": "Point", "coordinates": [ -70.12649307518743, 43.71613636966065 ] }, "properties": { "id": 160, "readings": [ { "value": -0.252, "time": "2024-11-16T23:12:00Z", "depth": null, "data_type": { "standard_name": "predicted_sea_water_level", "short_name": "PWL", "long_name": "Predicted level of sea water", "units": "m" }, "server": "https://data.neracoos.org/erddap/", "variable": "predictedWL", "constraints": { "shefID=": "CASM1" }, "dataset": "nosCoopsWLTP6", "start_time": "2024-11-16T14:49:54.295100Z", "cors_proxy_url": "/api/servers/6/proxy/", "datum_offsets": {}, "flood_levels": [], "highlighted": "No", "type": "Prediction", "extrema": false, "extrema_values": { "max": { "time": "2024-11-17T16:30:00+00:00", "value": 3.508 }, "min": { "time": "2024-11-17T22:54:00+00:00", "value": -0.485 }, "tides": [ { "tide": "low", "time": "2024-11-16T23:06:00+00:00", "value": -0.299 }, { "tide": "high", "time": "2024-11-17T00:48:00+00:00", "value": 0.961 },