POD discussion: Surface Temperature Extremes diagnostic #120

tsjackson-noaa · 2020-02-19T14:35:14Z

tsjackson-noaa
Feb 19, 2020

Thread for progress in adding the Surface Temperature Extremes Distributions POD by Arielle Catalano, Paul Loikith and J. David Neelin.

tsjackson-noaa · 2020-07-28T04:24:17Z

tsjackson-noaa
Jul 28, 2020
Author

@acatalano16 : I wanted to update you on some issues running the POD on CMIP6 data (so beyond your sample data; some of the errors may be due to framework code). I've added some debugging comments but haven't gotten very far yet -- if you have any insight please let me know.

TempExtDistShape_Moments_util.py line 149 is getting an empty slice -- haven't had time to look into this
TempExtDistShape_ShiftRatio_util.py line 217 hits an error on thresholding, although the limits look reasonable. I don't know why the debug statements aren't printed before the error.
TempExtDistShape_CircComps_util.py line 89: this hits a MemoryError because it unexpectedly recieved 4D data instead of 3D. I'm planning to have the framework handle this eventually, but won't have time for it soon -- would you be able to add a check in your code? Also note that passing year_ind as an array, rather than a numpy slice, will cause numpy to make a copy of var_data instead of trimming the existing array in memory.

temp_extremes_distshape.log

--- MDTF.py Starting POD temp_extremes_distshape
Found files:
/net2/Thomas.Jackson/tmp/inputdata/model/ESM4_historical_D1/day/ESM4_historical_D1.tas.day.nc
/net2/Thomas.Jackson/tmp/inputdata/model/ESM4_historical_D1/day/ESM4_historical_D1.zg.day.nc
/net2/Thomas.Jackson/tmp/inputdata/model/ESM4_historical_D1/day/ESM4_historical_D1.psl.day.nc
 Env vars:
CASENAME: ESM4_historical_D1
DATADIR: /net2/Thomas.Jackson/tmp/inputdata/model/ESM4_historical_D1
FIRSTYR: 2000
LASTYR: 2004
MDTF_GFDL_TMPDIR: /net2/Thomas.Jackson/tmp
MPLBACKEND: Agg
OBS_DATA: /net2/Thomas.Jackson/tmp/inputdata/obs_data/temp_extremes_distshape
POD_HOME: /local2/home/MDTF/MDTF-diagnostics/diagnostics/temp_extremes_distshape
RGB: /local2/home/MDTF/MDTF-diagnostics/src/rgb
WK_DIR: /net2/Thomas.Jackson/tmp/wkdir/MDTF_ESM4_historical_D1_2000_2004.v12/temp_extremes_distshape
city: Yellowknife
model: GFDL-ESM4
monthstr: DJF
monthsub: [12, 1, 2]
psl_var: psl
ptile: 5
tas_var: tas
zg_var: zg
 --- MDTF.py calling POD temp_extremes_distshape
/local2/home/MDTF/MDTF-diagnostics/diagnostics/temp_extremes_distshape/TempExtDistShape_Moments_util.py:149: RuntimeWarning: Mean of empty slice
  seas_mean=numpy.transpose(numpy.nanmean(masked_T2M,2))
/local2/home/MDTF/MDTF-diagnostics/envs/_MDTF_python_base/lib/python2.7/site-packages/numpy/lib/nanfunctions.py:1628: RuntimeWarning: Degrees of freedom <= 0 for slice.
  keepdims=keepdims)
**************************************************
Executing Moments (TempExtDistShape_Moments.py)......
**************************************************
Load user-specified parameters...
...Loaded!
/net2/Thomas.Jackson/tmp/inputdata/model/ESM4_historical_D1/day
   Generating region mask...
...Generated!
   Computing seasonal temperature moments...
...Computed!
   Plotting seasonal temperature moments...
...Completed!
      Figure saved as /net2/Thomas.Jackson/tmp/wkdir/MDTF_ESM4_historical_D1_2000_2004.v12/temp_extremes_distshape/model/PS/Moments_DJF.ps!
**************************************************
Moments of Temperature Distribution (TempExtDistShape_Moments.py) Executed!
**************************************************
/local2/home/MDTF/MDTF-diagnostics/envs/_MDTF_python_base/lib/python2.7/site-packages/numpy/lib/function_base.py:3826: RuntimeWarning: Invalid value encountered in percentile
  interpolation=interpolation)
/local2/home/MDTF/MDTF-diagnostics/diagnostics/temp_extremes_distshape/TempExtDistShape_ShiftRatio_util.py:217: RuntimeWarning: invalid value encountered in greater
  exceedances[(exceedances>gaussp5)&(exceedances<gaussp95)]=numpy.nan
/local2/home/MDTF/MDTF-diagnostics/diagnostics/temp_extremes_distshape/TempExtDistShape_ShiftRatio_util.py:217: RuntimeWarning: invalid value encountered in less
  exceedances[(exceedances>gaussp5)&(exceedances<gaussp95)]=numpy.nan
**************************************************
Executing Underlying-to-Gaussian Distribution Shift Ratio (TempExtDistShape_ShiftRatio.py)......
**************************************************
Load user-specified parameters...
...Loaded!
   Generating region mask...
...Generated!
   Computing underlying-to-Gaussian distribution shift ratio...
### DEBUG ShiftRatio_util
gaussp5: 0.888888888889, gaussp95: 2.44444444444
exceedances: (288, 180); float64
exceedances: (288, 180); float64
...Computed!
   Plotting shift ratio...
...Completed!
      Figure saved as /net2/Thomas.Jackson/tmp/wkdir/MDTF_ESM4_historical_D1_2000_2004.v12/temp_extremes_distshape/model/PS/ShiftRatio_DJF_5.ps!
**************************************************
Underlying-to-Gaussian Distribution Shift Ratio (TempExtDistShape_ShiftRatio.py) Executed!
**************************************************
/local2/home/MDTF/MDTF-diagnostics/envs/_MDTF_python_base/lib/python2.7/site-packages/matplotlib/cbook/deprecation.py:107: MatplotlibDeprecationWarning: Passing one of 'on', 'true', 'off', 'false' as a boolean is deprecated; use an actual boolean (True/False) instead.
  warnings.warn(message, mplDeprecation, stacklevel=1)
**************************************************
Executing Frequency Distributions at Non-Gaussian Tail Locations (TempExtDistShape_FreqDist.py)......
**************************************************
Load user-specified parameters including season...
...Loaded!
   Estimating Gaussian fit at Yellowknife...
...Completed!
   Estimating Gaussian fit at Pendleton...
...Completed!
   Estimating Gaussian fit at Rennes St-Jacques...
...Completed!
   Estimating Gaussian fit at Berlin...
...Completed!
   Estimating Gaussian fit at Adelaide...
...Completed!
   Estimating Gaussian fit at North Platte...
...Completed!
      Figure saved as /net2/Thomas.Jackson/tmp/wkdir/MDTF_ESM4_historical_D1_2000_2004.v12/temp_extremes_distshape/model/PS/FreqDists.ps!
**************************************************
Frequency Distributions at Non-Gaussian Tail Locations (TempExtDistShape_FreqDist.py) Executed!
**************************************************
**************************************************
Executing Non-Gaussian Tails Circulation Composites (TempExtDistShape_CircComps.py)......
**************************************************
Load user-specified parameters...
...Loaded!
### DEBUG CircComps_util
yearbeg: 2000, yearend: 2004, yearind: [   0    1    2 ... 1822 1823 1824]
var_data: (288, 180, 1825); float64
### DEBUG CircComps_util
yearbeg: 2000, yearend: 2004, yearind: [   0    1    2 ... 1822 1823 1824]
var_data: (288, 180, 1825); float64
### DEBUG CircComps_util
yearbeg: 2000, yearend: 2004, yearind: [   0    1    2 ... 1822 1823 1824]
var_data: (1825, 19, 180, 288); float64
Traceback (most recent call last):
  File "/local2/home/MDTF/MDTF-diagnostics/diagnostics/temp_extremes_distshape/TempExtDistShape_CircComps.py", line 74, in <module>
    Z500_data,lon,lat,datearrstr,Z500_units=Seasonal_Subset(Z500file,circ_data["LON_VAR"],circ_data["LAT_VAR"],circ_data["Z500_VAR"],circ_data["TIME_VAR"],monthsub,circ_data["yearbeg"],circ_data["yearend"])
  File "/local2/home/MDTF/MDTF-diagnostics/diagnostics/temp_extremes_distshape/TempExtDistShape_CircComps_util.py", line 89, in Seasonal_Subset
    var_data=var_data[:,:,yearind]
MemoryError
**************************************************
Surface Temperature Extremes and Distribution Shape Package (TempExtDistShape.py) Executed!
**************************************************

0 replies

acatalano16 · 2020-07-28T15:18:48Z

acatalano16
Jul 28, 2020

@tsjackson-noaa On the first issue, I did get some empty slice warnings but the code ran smoothly despite these warnings so I didn't consider it an error. I believe they are related to the masking, so taking even a "nanmean" or "nanpercentile" across some masked latitudes of NaNs will result in that warning.

On the second issue, I have never received that error and am thus unsure of the cause. If you try looking at the max/min of the exceedances array maybe that will be illuminating? Perhaps it's turning into all NaNs upstream or is outside the threshold range, though I can't imagine why from the code.

On the third issue, I had considered 4D data in crafting the code and have a commented line somewhere. Is the 4D data already sliced to 500hPa and just has a "1" in the last dimension?

0 replies

jkrasting · 2020-07-30T11:58:01Z

jkrasting
Jul 30, 2020
Maintainer

Thanks @acatalano16. Would you be able to also pass along the versions of the Python packages you used to develop the POD, specifically NumPy?

0 replies

acatalano16 · 2020-07-30T14:16:53Z

acatalano16
Jul 30, 2020

@jkrasting @tsjackson-noaa Upon further inspection, I was receiving that same error (I've been using the _MDTF_base environment), but because it popped up in the log before that section of code ran and it didn't affect the output html, I must have missed it. I apologize. It seems to be similar to the first "error" in that it is related to the NaN slices from masking the globe, and it doesn't stop the code (it's related to numpy as John figured out: Unidata/netcdf4-python#766).

I think the easiest fix for the first two points raised is to suppress the warning messages, but it's not a command I'm used to using and was unable to apply successfully last year when I first tried. If that's not a route you want to take, I can also try bypassing numpy slices by including a few "if" lines at these problematic functions?

0 replies

tsjackson-noaa · 2020-08-14T19:20:34Z

tsjackson-noaa
Aug 14, 2020
Author

Progress update:

Link to development branch and POD commit history. These links will stop working when the branch is deleted after integration work is finished. Links to files below will remain valid.

Current bugs/errors:

MemoryError in TempExtDistShape_CircComps_util.Seasonal_Subset referred to above. This was diagnosed as arising when the POD was passed 4D input data, instead of data on the 500hPa level that it expects.

Remaining necessary tasks:

Because other PODs are also affected by the 3D/4D issue (eg, in Refine data requirements for EOF_500hPa #16), it was decided to implement pressure level extraction at the framework level as a preprocessing step.
Convert .pdf documentation to .rst.

0 replies

jkrasting · 2020-08-16T20:51:54Z

jkrasting
Aug 16, 2020
Maintainer

Referencing #40 as it relevant to this issue.

0 replies

tsjackson-noaa · 2021-01-23T20:59:57Z

tsjackson-noaa
Jan 23, 2021
Author

The branch containing existing development work on this POD has been renamed from pod/temp_extremes_distshape to feature/add_temp_extremes_distshape, as part of the reorganization proposed in issue #106.

0 replies

tsjackson-noaa · 2021-08-03T05:44:17Z

tsjackson-noaa
Aug 3, 2021
Author

Update: Sticking point encountered previously was that the POD uses air_pressure_at_mean_sea_level, not currentl provided by any of the sample model datasets. Will test on CMIP6 data at GFDL.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

POD discussion: Surface Temperature Extremes diagnostic #120

{{title}}

Replies: 8 comments

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

POD discussion: Surface Temperature Extremes diagnostic #120

tsjackson-noaa Feb 19, 2020

Replies: 8 comments

tsjackson-noaa Jul 28, 2020 Author

acatalano16 Jul 28, 2020

jkrasting Jul 30, 2020 Maintainer

acatalano16 Jul 30, 2020

tsjackson-noaa Aug 14, 2020 Author

jkrasting Aug 16, 2020 Maintainer

tsjackson-noaa Jan 23, 2021 Author

tsjackson-noaa Aug 3, 2021 Author

tsjackson-noaa
Feb 19, 2020

tsjackson-noaa
Jul 28, 2020
Author

acatalano16
Jul 28, 2020

jkrasting
Jul 30, 2020
Maintainer

acatalano16
Jul 30, 2020

tsjackson-noaa
Aug 14, 2020
Author

jkrasting
Aug 16, 2020
Maintainer

tsjackson-noaa
Jan 23, 2021
Author

tsjackson-noaa
Aug 3, 2021
Author