Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PresRat correction #215

Merged
merged 189 commits into from
Jul 23, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
189 commits
Select commit Hold shift + click to select a range
b27df73
Initializing PresRat with reference
castelao Apr 23, 2024
db261ca
feat: ZeroRateMixin()
castelao Apr 23, 2024
bb4d1b2
refact: Moving ZeroRateMixin to mixins
castelao May 24, 2024
092a00a
Adding ipython into dev environment
castelao May 24, 2024
551f54a
fix: ZeroRateMixin shouldn't inherit DataRetrievalBase
castelao May 24, 2024
c196447
refact: Moving PresRat to qdm sub-module
castelao May 24, 2024
e91b219
fix: Use the given threshold (it was hardcoded before)
castelao May 24, 2024
b025c9f
PresRat using ZeroRateMixin
castelao May 24, 2024
cbb09d2
doc: Explicit steps for PresRat
castelao May 27, 2024
0d0d533
Prototype for quantiles on time windows
castelao May 28, 2024
7af50e6
fix: Missing load PresRat at module
castelao May 28, 2024
5cc97ff
Overwritting QDM methods for PresRat requirements
castelao Jun 3, 2024
13d9cb1
Don't need to overwrite __init__ if defining threshold in the runtime
castelao Jun 3, 2024
59f087e
doc: More details on what is PresRat
castelao Jun 3, 2024
aa9d134
Testing apply_zero_precipitation_rate()
castelao Jun 6, 2024
500ad66
test: zero_precipitation_rate()
castelao Jun 11, 2024
43ba757
test: presrat_zero_rate with multiple thresholds
castelao Jun 12, 2024
e51394c
fix, doc: Typo 'monthly'
castelao Jun 12, 2024
9be8176
test: apply_zero_precipitation_rate()
castelao Jun 13, 2024
d598b07
test: Minimalist check for test_presrat_transform()
castelao Jun 13, 2024
5af60fb
test: parallel results must equal serial
castelao Jun 13, 2024
695e2d9
style: Fixing code style
castelao Jun 17, 2024
b014d56
test: test_presrat_transform_nochanges()
castelao Jun 20, 2024
f2253c9
test: Missing to update PresRat standard params for testing
castelao Jun 20, 2024
ac17b81
test: Adding back a small noise on future projections
castelao Jun 20, 2024
8ce668f
test: Defining some resources used in the tests
castelao Jun 20, 2024
672fc61
clean, test:
castelao Jun 20, 2024
754268f
test: test_presrat_zero_rate with multiple thresholds
castelao Jun 20, 2024
008056f
test, clean: Combining tests
castelao Jun 20, 2024
d546ded
test: A fixture with fut_cc dataset itself for efficiency
castelao Jun 20, 2024
1063780
test: Standard presrat transform test with dataset from memory
castelao Jun 20, 2024
0888dda
test: test_presrat_transform_nozerochanges()
castelao Jun 20, 2024
c166c0e
test: test_presrat_transform_nochanges() working
castelao Jun 20, 2024
077c5e7
style:
castelao Jun 20, 2024
1fb6c19
test, doc: test_presrat_transform_nochanges()
castelao Jun 20, 2024
81ae605
Saving used threshold in metadata
castelao Jun 21, 2024
edd3005
Entry point for presrat functions
castelao Jun 21, 2024
b5731c2
feat: apply_zero_precipitation_rate()
castelao Jun 21, 2024
98f3bf0
feat: get_spatial_bc_presrat()
castelao Jun 21, 2024
d7ddd23
fix, test: test_presrat_transform_nochanges()
castelao Jun 21, 2024
1730a81
feat: Saving requirements to estimate K factor
castelao Jun 21, 2024
bee4950
Adding time indices
castelao Jun 21, 2024
39bae3f
fix: zero_precipitation_rate() dealing with NaN
castelao Jun 21, 2024
a368708
refact: Moved apply_zero_precipitation_rate() into bias_tranforms
castelao Jun 21, 2024
262f7c2
fix: Missing to load xarray
castelao Jun 21, 2024
52e167b
refact: get_spatial_bc_quantiles returing params as dict
castelao Jun 24, 2024
9f08e37
feat: local_presrat_bc()
castelao Jun 24, 2024
d764202
style: Mostly moving to single quote which is the standard used in sup3r
castelao Jun 24, 2024
aa0cf22
test, clean: PresRat tests were already moved into its own module
castelao Jun 24, 2024
d426e83
Locking numpy to < 2.0
castelao Jun 24, 2024
6cc7bf6
doc: Updating one-line description of PresRat
castelao Jun 24, 2024
abeae3d
test: Updating fut_cc_notrend() to match the 'resource'
castelao Jun 24, 2024
e70beb3
test, doc: Better explaing the expected effect of NaNs
castelao Jun 24, 2024
42a46dc
feat: PresRat.write_outputs()
castelao Jun 24, 2024
b95adfe
fix, doc: Weird string literal issue
castelao Jun 24, 2024
5650c19
style: Sorting imports
castelao Jun 24, 2024
c1af8db
test, doc: More info in PresRat tests
castelao Jun 24, 2024
a09f9f0
test: Improving test_presrat_calc()
castelao Jun 24, 2024
8c8a28a
test, refact: Combining tests
castelao Jun 24, 2024
2d997f5
style: Code formating
castelao Jun 24, 2024
8be8e0e
test, refact: Conforming normalizing dataset
castelao Jun 24, 2024
f5fabba
style: Missing space
castelao Jun 24, 2024
1d26fe8
fix, feat: Missing to include presrat_notrend_params()
castelao Jun 24, 2024
8b8e7b0
style: Unused import
castelao Jun 24, 2024
6fd0cc0
test, fix: Missing a few fixtures
castelao Jun 24, 2024
d56b969
doc: Missing documentation on testing functions
castelao Jun 24, 2024
218ad86
test: Global zero rate threshold (ZR_THRESHOLD)
castelao Jun 24, 2024
7ac12b4
clean: Removing ideas for later
castelao Jun 24, 2024
8c45370
Bump Python to 3.11.9
castelao Jun 25, 2024
59caf0c
style: Conforming with sup3r style
castelao Jun 26, 2024
f0c5b48
style: UP039
castelao Jul 1, 2024
da42586
fix: Wrong type
castelao Jul 1, 2024
925be43
doc: local_presrat_bc()
castelao Jul 1, 2024
87681eb
style: E741 - ambiguous-variable-name
castelao Jul 1, 2024
2e73183
cfg: Adding UP039 as fixable
castelao Jul 1, 2024
9d93140
style: Fixing notes to conform with numpydoc
castelao Jul 1, 2024
7a04ef7
doc: Adding TODO label
castelao Jul 1, 2024
9a480fd
style: missing-trailing-comma
castelao Jul 1, 2024
734b780
style: Conforming with sup3r coding style
castelao Jul 2, 2024
43645e3
fix: Wrong type
castelao Jul 2, 2024
1a9a7fa
fix: Renaming variable to conform with requirements from forward pass
castelao Jul 2, 2024
c8bf9ef
cfg: Adding E303 as auto fix
castelao Jul 2, 2024
e1eb618
test: Prototype of foward pass test
castelao Jul 2, 2024
88385e8
test, refact: test_parallel working with new test dataset
castelao Jul 8, 2024
ebd253f
test, reafct: test_presrat_calc() with new testing dataset
castelao Jul 8, 2024
f330ef4
test, refact: test_presrat_zero_rate() using new sample test
castelao Jul 8, 2024
27f2f0d
test, refact: test_presrat() using new sample data
castelao Jul 8, 2024
00d3ed0
test, refact: presrat_params with new sample data
castelao Jul 8, 2024
e61078a
clean: Removing functions in development
castelao Jul 8, 2024
c082d6c
Temporary test data built on the fly
castelao Jul 8, 2024
5b95432
test, feat: Replacement for FP_NSRDB on the fly
castelao Jul 8, 2024
de7cf0f
test, refact: Diverting tests to use custom testing dataset
castelao Jul 8, 2024
ee0ad00
style: Fixing conding style
castelao Jul 9, 2024
3d5adb8
clean: removing old sample data prototype
castelao Jul 9, 2024
fa44421
doc, test: Adding documentation title for new functions
castelao Jul 9, 2024
98c2bc8
style: formating synthetic data ranges
castelao Jul 9, 2024
cb4253e
fix: _get_factors() assume descending latitudes
castelao Jul 9, 2024
b8efcb2
fix: Assumptions should be reinforced
castelao Jul 9, 2024
95b9a7a
test, fix: Synthetic data lon as range
castelao Jul 9, 2024
db15751
clean: Not using Resources & DataHandlerNCforCC directly anymore
castelao Jul 9, 2024
c3314b5
test, fix: Using numpy we must conform with dimension order
castelao Jul 9, 2024
aedb2a6
test, refact: Using synthetic data instead
castelao Jul 9, 2024
d07bd6d
test, refact: Running in parallel
castelao Jul 9, 2024
7b61575
clean, test: Removing variables not used anymore
castelao Jul 9, 2024
f691bb1
fix, test: Left behind a filename reference without updating
castelao Jul 9, 2024
68b525f
fix, test: Missing bias_fp definition
castelao Jul 9, 2024
0ff0377
style, test:
castelao Jul 9, 2024
021ebaa
clean: Unused variable
castelao Jul 9, 2024
ae7b542
style: Empty space
castelao Jul 9, 2024
3f268f4
style: while I don't use those variables to validate test
castelao Jul 9, 2024
acf5429
clean: Time index is not used here yet
castelao Jul 10, 2024
40ec538
cfg: Ruff setup, sup3r uses single quote
castelao Jul 10, 2024
cfd8962
clean: Using global test definitions
castelao Jul 10, 2024
5d85e11
fix: Missing arguments for QDM applications
castelao Jul 11, 2024
3b2f727
feat: Dirty implementation of future tau factor
castelao Jul 11, 2024
31b2823
Dirty proof of concept for K-factor
castelao Jul 12, 2024
ee35e12
cfg: Updating pixi.lock
castelao Jul 15, 2024
1925a0d
fix: get_spatial_bc_quantiles() now returns params groupped
castelao Jul 15, 2024
98a0e7b
doc: Improving reference for more information on PresRat
castelao Jul 15, 2024
4e7c9fc
doc, fix: get_spatial_bc_presrat()
castelao Jul 15, 2024
97b941d
style:
castelao Jul 15, 2024
329c249
style: lower case k as requested by @grantbuster
castelao Jul 15, 2024
69329eb
test: Skipping tests to isolate the problem before refactoring it
castelao Jul 15, 2024
a633653
refact: Conforming with QDM on input arguments order
castelao Jul 15, 2024
3618224
refactor: New default value for zero rate's threshold
castelao Jul 15, 2024
e50edfa
A task to run pytest
castelao Jul 16, 2024
dbab2d2
Replicating time varing QDM in PresRat procedure
castelao Jul 16, 2024
bd2af81
clean: Removing resources from old implementation
castelao Jul 16, 2024
2379b99
Still saving estimated zero rate for now
castelao Jul 16, 2024
9a7c7ea
refact: K factor estiamted with the same time scale of CDFs
castelao Jul 16, 2024
982bc2d
test: Updating test_presrat_calc_params()
castelao Jul 16, 2024
d734d90
test, fix: Cleaning old fp_fut_cc
castelao Jul 16, 2024
9ddc1b7
refact: local_presrat_bc() applied on time windows
castelao Jul 16, 2024
c0e6c5a
refact: Changing standard to NT=24
castelao Jul 16, 2024
e71624d
fix: Forgot to commit the time_window_center into extra_attrs
castelao Jul 16, 2024
e403406
fix: bias_transforms._get_factors() assume descending lats
castelao Jul 16, 2024
25b2464
Reinforce expected sorting by _get_factor()
castelao Jul 16, 2024
2c0c3ac
Remove check on tau_fut
castelao Jul 17, 2024
0f8427c
refact: Better selection on time centers
castelao Jul 17, 2024
c485a57
style: To satisfy super-linter
castelao Jul 17, 2024
a92435e
test: Zero rate after correction can't be smaller
castelao Jul 19, 2024
dd9c143
style: Removing unecessary parenthesis
castelao Jul 19, 2024
ebe2503
Synthetic percip is now a DataArray
castelao Jul 22, 2024
de2d9cf
refact: mf dataset redefined
castelao Jul 22, 2024
10c43b9
refact: Re-organizing tests for applying zero rate
castelao Jul 22, 2024
0ff04c8
refact, test: PresRat (pre) calcualtion of parameters
castelao Jul 22, 2024
b035c49
doc, test: Clarifying unconventional requirements on lat & lon
castelao Jul 22, 2024
4177594
test: Extending synthetic resources to a full year
castelao Jul 22, 2024
e914da8
test: test_compare_qdm_vs_presrat()
castelao Jul 22, 2024
442b125
style: Formating
castelao Jul 22, 2024
3fabd4c
fix: Missing import local_qdm_bc
castelao Jul 22, 2024
6dcd0da
test, refact: Extending sample data for a longer time series
castelao Jul 22, 2024
cb20e24
refact, test: Renaming variables
castelao Jul 22, 2024
426ea70
fix: Conforming with new arguments order for QDM
castelao Jul 22, 2024
6b473b8
doc, test: More info
castelao Jul 22, 2024
32aed4c
test, refact: Sample data with lognormal distribution
castelao Jul 22, 2024
7d036d5
style: Mostly single quotes, which is sup3r's standard
castelao Jul 22, 2024
aa92d63
clean: Looks like we don't actually need time_bnds
castelao Jul 22, 2024
40bba8e
Defining global TAU used on synthetic data
castelao Jul 22, 2024
9f21963
style: Breaking a text line that was too long
castelao Jul 22, 2024
86abd8c
fix: local_presrat_bc time_index argument order
castelao Jul 22, 2024
e079828
doc, fix: Correcting arguments definitions
castelao Jul 22, 2024
bf42f3e
refact: Removing double loop when applying zero rate
castelao Jul 22, 2024
bfa519f
doc: _get_factors()
castelao Jul 22, 2024
c9b69f7
doc, fix: Fixing get_spatial_bc_quantiles()
castelao Jul 22, 2024
e63b59c
doc, style: get_spatial_bc_presrat()
castelao Jul 22, 2024
d422341
fix: mistake on reinforcing threshold for zero rate
castelao Jul 22, 2024
3483600
doc, test: More info on PresRat tests
castelao Jul 22, 2024
0aaf49b
style: Conforming with sup3r style
castelao Jul 22, 2024
9b391f5
test, fix: Fixed test_presrat_transform_nochanges()
castelao Jul 22, 2024
0a78b31
refact: Removing apply_zero_precipitation_rate()
castelao Jul 22, 2024
3459e02
refact: PresRat with no_trend reduces to ordinary QDM
castelao Jul 22, 2024
06b1b5b
fix, test: Forgot to update rsds_tau_fut in presrat_nochanges_params
castelao Jul 22, 2024
6e30c3e
doc: bias_transforms._get_factors(), improving documentation
castelao Jul 23, 2024
b51d19e
doc: QDM._window_center(), improving documentation
castelao Jul 23, 2024
4e12610
fix: Incomplete assert check
castelao Jul 23, 2024
e401b30
doc: get_spatial_bc_presrat(), improving documentation
castelao Jul 23, 2024
ed7bcea
doc: local_presrat_bc(), improving documentation
castelao Jul 23, 2024
ac962a3
doc: PresRat, documenting zero rate implementation
castelao Jul 23, 2024
4ecd0a8
doc: More information on tests
castelao Jul 23, 2024
7e3873b
test, fix: Reactivating test_presrat_transform_nochanges()
castelao Jul 23, 2024
b548eb8
test, fix: Fixed transform_nozerochanges test
castelao Jul 23, 2024
d9bc92d
cfg: Auto fix missing-whitespace-around-operator
castelao Jul 23, 2024
0238519
cfg: Adding pytest-xdist to dev feature
castelao Jul 23, 2024
7394427
cfg: Updating pixi.lock
castelao Jul 23, 2024
0b81c67
fix: Removing method zero_rate_threshold
castelao Jul 23, 2024
95f2e00
fix: zero_precipitation_rate()
castelao Jul 23, 2024
b5ab7f9
doc: A more meaningfull example for write_outputs()
castelao Jul 23, 2024
fd4f453
test, fix: Updating test to reflect new <= threshold
castelao Jul 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1,060 changes: 572 additions & 488 deletions pixi.lock

Large diffs are not rendered by default.

15 changes: 12 additions & 3 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,11 @@ exclude = [
]

[tool.ruff.lint]
fixable = []
fixable = [
"E225", # missing-whitespace-around-operator
"E303", # too-many-blank-lines
"UP039", # unnecessary-class-parentheses
]
# preview = true
# logger-objects = []
task-tags = ["TODO", "FIXME", "XXX"]
Expand Down Expand Up @@ -210,6 +214,9 @@ ignore = [
# redundant-keyword-arg
# c-extension-no-member

[tool.ruff.lint.flake8-quotes]
inline-quotes = 'single'

[tool.ruff.lint.mccabe]
# Flag errors (`C901`) whenever the complexity level exceeds 5.
max-complexity = 12
Expand Down Expand Up @@ -253,13 +260,13 @@ channels = ["conda-forge", "anaconda", "main"]
platforms = ["osx-arm64", "linux-64"]

[tool.pixi.dependencies]
python = "==3.11"
python = "~=3.11.0"
cftime = ">=1.6.2"
dask = ">=2022.0"
h5netcdf = ">=1.1.0"
pillow = ">=10.0"
matplotlib = ">=3.1"
numpy = ">=1.7.0"
numpy = "~=1.7"
pandas = ">=2.0"
scipy = ">=1.0.0"
tensorflow = ">2.4,<2.16"
Expand All @@ -279,6 +286,7 @@ doc = { features = ["doc"], solve-group = "default" }
test = { features = ["test"], solve-group = "default" }

[tool.pixi.tasks]
test = "pytest --pdb --durations=10 tests"

[tool.pixi.feature.doc.dependencies]
sphinx = ">=7.0"
Expand All @@ -292,3 +300,4 @@ build = ">=0.6"
twine = ">=5.0"
ruff = ">=0.4"
ipython = ">=8.0"
pytest-xdist = ">=3.0"
9 changes: 6 additions & 3 deletions sup3r/bias/__init__.py
Original file line number Diff line number Diff line change
@@ -1,19 +1,22 @@
"""Bias calculation and correction modules."""

from .bias_transforms import (global_linear_bc, local_linear_bc,
local_qdm_bc, monthly_local_linear_bc)
from .bias_calc import (LinearCorrection, MonthlyLinearCorrection,
MonthlyScalarCorrection, SkillAssessment)
from .qdm import QuantileDeltaMappingCorrection
from .bias_transforms import (global_linear_bc, local_linear_bc,
local_qdm_bc, local_presrat_bc,
monthly_local_linear_bc)
from .qdm import PresRat, QuantileDeltaMappingCorrection

__all__ = [
"global_linear_bc",
"local_linear_bc",
"local_qdm_bc",
"local_presrat_bc",
"monthly_local_linear_bc",
"LinearCorrection",
"MonthlyLinearCorrection",
"MonthlyScalarCorrection",
"PresRat",
"QuantileDeltaMappingCorrection",
"SkillAssessment",
]
4 changes: 2 additions & 2 deletions sup3r/bias/bias_calc.py
Original file line number Diff line number Diff line change
Expand Up @@ -984,7 +984,7 @@ def run(self,
class MonthlyLinearCorrection(LinearCorrection):
"""Calculate linear correction *scalar +adder factors to bias correct data

This calculation operates on single bias sites on a montly basis
This calculation operates on single bias sites on a monthly basis
"""

NT = 12
Expand Down Expand Up @@ -1044,7 +1044,7 @@ class MonthlyScalarCorrection(MonthlyLinearCorrection):
scalar factors are computed as mean(base_data) / mean(bias_data). Adder
factors are still written but are exactly zero.

This calculation operates on single bias sites on a montly basis
This calculation operates on single bias sites on a monthly basis
"""

@staticmethod
Expand Down
Loading
Loading