How to generate Taylor diagram using METplotpy and METdataio and EVS stat output

[Ho-ChunHuang-NOAA]

I need to learn how to generate Taylor diagram for EVS using METplotpy. From Binbin, I learned that I need to reformat EVS point_stat output with METdataio/METReformat.

(a) I can not find point_stat.yaml on beta3 built in NOAA/WCOSS2. So I use SL1L2.yaml as a base to perform the reformatting., see test_metdataio.sh.txt and SL1L2.yaml.txt. The input is evs.stats.aqm_raw.atmos.grid2obs_pm25.v20240406.stat.txt and output is evs.stats.aqm_raw.atmos.grid2obs_pm25.v20240406.txt.

(b) I can generate the Taylor Diagram using can-data and script as described in the METplotpy documentation, i.e., using plot_dlwr_sample.data and taylor_diagram_custom.yaml.

(c) I use evs.stats.aqm_raw.atmos.grid2obs_pm25.v20240406.txt (from a) to generate Taylor Diagram but failed, see aqm.yaml.txt and test_taylor.sh.txt for detail. The log output is log_test_taylor.txt. I am unable to understand the message of "ValueError: zero-size array to reduction operation minimum which has no identity".

Please help me to resolve this problem.

test_metdataio.sh.txt
test_taylor.sh.txt
evs.stats.aqm_raw.atmos.grid2obs_pm25.v20240406.stat.txt
evs.stats.aqm_raw.atmos.grid2obs_pm25.v20240406.txt

SL1L2.yaml.txt
aqm.yaml.txt

[bikegeek]

Hi Ho-Chun,

Thanks for providing me with the files. I didn't see the log_test_taylor.txt in your list, could you please provide that?

[Ho-ChunHuang-NOAA]

Add missing file
log_test_taylor.txt

[bikegeek]

Hi Ho-Chun,

I stepped through the taylor diagram code, and the following statistics are needed: FSTDEV, OSTDEV, and PR_CORR which are available in the CNT linetype. Do you have any output with the CNT linetype that you can provide for me? I need to correct the Taylor diagram User's Guide to explicitly state that only the CNT linetype is accepted

Regards,
Minna

[Ho-ChunHuang-NOAA]

Hi, Minna:

Thanks for figuring out what are missing.

I will generate separate statistic with CNT linetype and try. Is it correct to reformat the stats using CNT.yaml? or you only need the original EVS Stats for testing?

But I have a question to ask, using METViewer, the current stat like "evs.stats.aqm_raw.atmos.grid2obs_pm25.v20240406.stat.txt" is able to generate the Taylor Diagram. Is it a necessary condition to have CNT line type stats to use METPlotpy?

[bikegeek]

I don't have access to the CNT.yaml file, but all you really need is to make sure to set the line_type setting to CNT and the appropriate paths and filenames for the output_dir, log_directory (if you want to keep a log file), output_filename, etc.

By default, the input_stats_aggregated is set to True (assumes that the input linetype data has already been aggregated via MET stat_analysis). The ECNT linetype is the only linetype that reformats for METcalcpy agg_stat.py

I updated the METviewer and METplotpy User's Guide- the CNT linetype is required for generating the Taylor Diagram. The original version was written in R and probably supported the SL1L2 linetype, but the current, Python version requires CNT.

Regards,
Minna

[Ho-ChunHuang-NOAA]

Hi, Minna:

I produced the new stat with linetype CNT, evs.stats.aqm_raw.atmos.grid2obs_pm25.v20240408.stat.txt. I used the stat directly in "python ${METPLOTPY_BASE}/metplotpy/plots/taylor_diagram/taylor_diagram.py aqm.yaml"
but error occurred, see log_taylor_20240408.txt for detail. It looks like the code does not like the column header in capital case as it is the default POINTSTAT output.

Then I first ran the ${METREFORMAT_BASE}/METreformat/write_stat_ascii.py to reformat PointStats output alone with CNT.yaml. Followed by running the ${METPLOTPY_BASE}/metplotpy/plots/taylor_diagram/taylor_diagram.py with reformatted txt file as input. It successfully created a Taylor Diagram, aqmv7.png.

Can METPlotpy be changed to read both capital and small case header in future v3 beta version? i.e., use the native PointStat (or Grid_Stat,....) stat output directly without data reformation. I hope this function can make it to met/v12.0.0 and metplus/v3.0.0 for EMC EVS/v2.0 upgrade.

See if you can use evs.stats.aqm_raw.atmos.grid2obs_pm25.v20240408.stat.txt to produce the Taylor Diagram.

log_taylor_20240408.txt

evs.stats.aqm_raw.atmos.grid2obs_pm25.v20240408.stat.txt

[bikegeek]

Hi Ho-chun, I'm glad you were finally able to produce a Taylor diagram. That is a valid request and this behavior is due to METviewer. The database stores the columns in lower case and the METplotpy code was originally written to replace the R scripts that were used by METviewer. It would be a good practice to internally convert the input data's column headers to lower case by the plotting code (for some of the MET output that does not require reformatting prior to plotting). However, this most likely won't get done anytime soon due to budget constraints. I will create a Github Issue for this. Regards, Minna

…

On Thu, Apr 11, 2024 at 6:31 PM Ho-Chun Huang ***@***.***> wrote: Hi, Minna: I produced the new stat with linetype CNT, evs.stats.aqm_raw.atmos.grid2obs_pm25.v20240408.stat.txt. I used the stat directly in "python ${METPLOTPY_BASE}/metplotpy/plots/taylor_diagram/taylor_diagram.py aqm.yaml" but error occurred, see log_taylor_20240408.txt for detail. It looks like the code does not like the column header in capital case as it is the default POINTSTAT output. Then I first ran the ${METREFORMAT_BASE}/METreformat/write_stat_ascii.py to reformat PointStats output alone with CNT.yaml. Followed by running the ${METPLOTPY_BASE}/metplotpy/plots/taylor_diagram/taylor_diagram.py with reformatted txt file as input. It successfully created a Taylor Diagram, aqmv7.png. *Can METPlotpy be changed to read both capital and small case header in future v3 beta version? i.e., use the native PointStat (or Grid_Stat,....) stat output directly without data reformation*. I hope this function can make it to met/v12.0.0 and metplus/v3.0.0 for EMC EVS/v2.0 upgrade. See if you can use evs.stats.aqm_raw.atmos.grid2obs_pm25.v20240408.stat.txt to produce the Taylor Diagram. log_taylor_20240408.txt <https://github.com/dtcenter/METplus/files/14952541/log_taylor_20240408.txt> evs.stats.aqm_raw.atmos.grid2obs_pm25.v20240408.stat.txt <https://github.com/dtcenter/METplus/files/14951680/evs.stats.aqm_raw.atmos.grid2obs_pm25.v20240408.stat.txt> aqmv7.png (view on web) <https://github.com/dtcenter/METplus/assets/48132716/0a6ad8ba-3d91-498f-8017-a12fcfee01b8> — Reply to this email directly, view it on GitHub <#2546 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4UJHSPDKKNACZBK4SUZATY44TMRAVCNFSM6AAAAABF64YNFWVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TAOJQGAZDA> . You are receiving this because you commented.Message ID: ***@***.***>

[Ho-ChunHuang-NOAA]

Hi, Minna:

Thanks for creating a GitHub issue.

Another question is what parameter control the style of Taylor Diagram in taylor_diagram_defaults.yaml?

I would like to produce similar one from the MetViewer, i.e., the graphic only contains the positive correlation portion of the diagram.

[Ho-ChunHuang-NOAA]

Hi, Minna:

Nevermind, I think I found it, taylor_voc.

[bikegeek]

no worries, please don't hesitate to ask any other questions or provide other comments about creating the Taylor diagram. It hasn't been used and we appreciate your feedback on the plot. Here is the issue: dtcenter/METplotpy#433 Please put any comments/suggestions/etc in the comments section of the issue so we can make the plot better. Regards, Minna

…

On Fri, Apr 12, 2024 at 7:23 AM Ho-Chun Huang ***@***.***> wrote: Hi, Minna: Nevermind, I think I found it, taylor_voc. — Reply to this email directly, view it on GitHub <#2546 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4UJHSMGIYLHGE2HXIMQJLY47N3RAVCNFSM6AAAAABF64YNFWVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TAOJWGA4TO> . You are receiving this because you commented.Message ID: ***@***.***>

[Ho-ChunHuang-NOAA]

Hi, minna: Thank you. I will post any findings when I move forward to complete the generation of Taylor Diagram for EVS AQm verification. Ho-Chun

…

-- Ho-Chun Huang, Ph.D. Physical Scientist III, Contractor with Lynker in Support of NOAA/NWS/NCEP/EMC, U.S. Department of Commerce 5830 University Research Ct., Rm. 2792 College Park, MD 20740 ***@***.*** ***@***.***> 301-683-3958

On Fri, Apr 12, 2024 at 11:13 AM bikegeek ***@***.***> wrote: no worries, please don't hesitate to ask any other questions or provide other comments about creating the Taylor diagram. It hasn't been used and we appreciate your feedback on the plot. Here is the issue: dtcenter/METplotpy#433 Please put any comments/suggestions/etc in the comments section of the issue so we can make the plot better. Regards, Minna On Fri, Apr 12, 2024 at 7:23 AM Ho-Chun Huang ***@***.***> wrote: > Hi, Minna: > > Nevermind, I think I found it, taylor_voc. > > — > Reply to this email directly, view it on GitHub > < #2546 (comment)>, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/AA4UJHSMGIYLHGE2HXIMQJLY47N3RAVCNFSM6AAAAABF64YNFWVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TAOJWGA4TO> > . > You are receiving this because you commented.Message ID: > ***@***.***> > — Reply to this email directly, view it on GitHub <#2546 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALPHE3BKAMUAGSWV4BXN2ILY472X3AVCNFSM6AAAAABF64YNFWVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TAOJXGM4DK> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[bikegeek]

Do you have any data with the CNT linetype? I have data with SL1L2 and need CNT linetype data to test out the config file settings.

Regards,
Minna

[Ho-ChunHuang-NOAA]

Hi, Minna:

The ftp server that I used to transfer data out of the WCOSS2 is currently down. So I can not provide any new data set for testing.

The original PointStat output with CNT stats is evs.stats.aqm_raw.atmos.grid2obs_pm25.v20240408.stat.txt in this discussion. You have to use METdataio to convert it to the ascii file you asked for.

get reformat_stat.yaml and change line-type to CNT., as well as input and output info.

==== METdataio scripts ==== please modify for your system
module reset
module use /apps/dev/modulefiles/
module load ve/evs/2.0
module use /apps/ops/para/libs/modulefiles/compiler/intel/19.1.3.304
export HPC_OPT=/apps/ops/para/libs
module load gsl/2.7
module load netcdf/4.7.4
module load met/12.0.0-beta3
module load metplus/6.0.0-beta3
module load METplotpy/3.0.0-beta3
module load METdataio/3.0.0-beta3
module load METcalcpy/3.0.0-beta3

export METREFORMAT_BASE=/apps/ops/para/libs/intel/19.1.3.304/METdataio/3.0.0-beta3/METdataio/ export PYTHONPATH=${METREFORMAT_BASE}:/${METREFORMAT_BASE}/METdbLoad:${METREFORMAT_BASE}/METdbLoad/ush:${METREFORMAT_BASE}/METreformat
${METREFORMAT_BASE}/METreformat/write_stat_ascii.py reformat_stat.yaml

=================================================

You can use the output file for your testing. the aqm.yaml in this discussion is the working one (old guidance).

Please let me know if you have any questions.

[bikegeek]

Thanks, I found the 202040408 data. I was using the 20240406 data you initially provided. You don't need to provide me with any other data.

…

On Tue, Apr 16, 2024 at 11:24 AM Ho-Chun Huang ***@***.***> wrote: Hi, Minna: The ftp server that I used to transfer data out of the WCOSS2 is currently down. So I can not provide any new data set for testing. The original PointStat output with CNT stats is evs.stats.aqm_raw.atmos.grid2obs_pm25.v20240408.stat.txt in this discussion. You have to use METdataio to convert it to the ascii file you asked for. get reformat_stat.yaml and c*hange line-type to CNT., as well as input and output info.* ==== METdataio scripts ==== please modify for your system module reset module use /apps/dev/modulefiles/ module load ve/evs/2.0 module use /apps/ops/para/libs/modulefiles/compiler/intel/19.1.3.304 export HPC_OPT=/apps/ops/para/libs module load gsl/2.7 module load netcdf/4.7.4 module load met/12.0.0-beta3 module load metplus/6.0.0-beta3 module load METplotpy/3.0.0-beta3 module load METdataio/3.0.0-beta3 module load METcalcpy/3.0.0-beta3 export METREFORMAT_BASE=/ *apps/ops/para/libs/intel/19.1.3.304/METdataio/3.0.0-beta3/METdataio/ export* PYTHONPATH=${METREFORMAT_BASE}:/${METREFORMAT_BASE}/METdbLoad:${METREFORMAT_BASE}/METdbLoad/ush:${METREFORMAT_BASE}/METreformat ${METREFORMAT_BASE}/METreformat/write_stat_ascii.py reformat_stat.yaml ================================================= You can use the output file for your testing. the aqm.yaml in this discussion is the working one (old guidance). Please let me know if you have any questions. — Reply to this email directly, view it on GitHub <#2546 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4UJHWPJAZXVXR3NSLCAQ3Y5VNGHAVCNFSM6AAAAABF64YNFWVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCMZTGA4TG> . You are receiving this because you commented.Message ID: ***@***.***>

[bikegeek]

I can generate the Taylor diagram using the aqm.yaml config file you provided me, with the 20240408 CNT data:
( setting the fcst_var_val1: {PMTF} or setting it to fcst_var_val1: {} and using fcst_var in the series_val_1):
aqm.yaml.txt

I think this works both ways because your data consists of only the PMTF forecast variable. Feel free to add any comments to this new Github issue, which will use the fcst_var_val in querying the input data:
dtcenter/METplotpy#434

[Ho-ChunHuang-NOAA]

Hi Minna:

I am getting error using the new setting in aqm.yaml of previous post. (fcst_var_val1: {} and using fcst_var in the series_val_1; please remove previously generated *.png first before running test using new setting)

I think the code thins that I wan to plot more than two symbols.

2024-04-16 19:23:46||User:ho-chun.huang||main|| [ERROR]: ValueError The number of series defined by series_val_1 is inconsistent with the number of settings required for describing each series. Please check the number of your configuration file's plot_i, plot_disp, series_order, user_legend, colors, and series_symbols settings.

[bikegeek]

Using the config file I provided earlier and running the code through the debugger, the number of series =2, number of plot_disp=2, number of markers=2, number of series orderings=2, and number of colors=2, which results in passing the consistency check.

I reverted back to using the fcst_var_val1: {PMTF} and get the same results, no errors and the two plots are identical.

[Ho-ChunHuang-NOAA]

Hi, Minna:

After detail examination or the difference between our aqm.yaml, I found mine does not have a space after "-", i.e., "-PMTF". That leads to the error message for inconsistent number of values. After the correction, I can produce the graphic with "- PMTF". Thank you for doing the test at your end.

Now I have more questions to ask to move forward. In the MetViewer xml file I can specified the time period I want, but do not know where to put it using METPlotpy.
(a) From the configuration of the filename, my point stat output is aggerated by day. I also found there is no time info input for users' yaml. If I want to make a plot for the last 30 days stats, how do I do that? Do I add all 30 days stats files in "stat_input:" separated by comma?
Or (b) do I need to first aggregated all daily files into a single file, then add that aggregated filename in "stat_input:"? e.g., from 03/15/2024 to 04/05/2024.

Same question also for selected init time (06Z versus 12Z), selected FCST lead time (fcst hour from 23 to 48),...etc.

There is no information from the user guide regarding this.

Thank you or your help.

Ho-Chun

[bikegeek]

Hi Ho-Chun, I'm glad you were able to get things to work. a) METviewer uses the database to filter the data (via XML specification file or the 'Fixed Value' menu on the GUI), and in some of the METplotpy plots (line plot), there is support for doing this through the fixed_vars_vals_input setting. Not all METplotpy plots have this support, including the Taylor Diagram. I created a Github issue to add this support: dtcenter/METplotpy#436 Please add any comments to the 'comment' section of the issue if you have any comments/suggestions/questions. b) The stat_input setting is expecting only one file, so you will need to collect all your 30 day stat files into one file, I use these commands to combine numerous MET files into one (via pandas dataframe). The data I want to combine is under one directory: # Get a list of all the files in the directory

path = r'/d2/projects/AF/rollup/data/Edouard' all_files = glob.glob(path + "/adeck*.tcst") # Concatenate all the tcst files into one dataframe np_array_list = [] for file_ in all_files: # print("reading in file: ", file_) df = pd.read_csv(file,delim_whitespace=True, lineterminator='\n', skiprows=1) np_array_list.append(df.values) comb_np_array = np.vstack(np_array_list) big_frame = pd.DataFrame(comb_np_array) # get the column headers from one file file = '/d2/projects/AF/rollup/data/Edouard/adeck_AL_05_20200703_180000.tcst' column_header = list(pd.read_csv(file,delim_whitespace=True, lineterminator='\n', nrows=0)) big_frame.columns = column_header # print(type(column_header))

# Apply header to the new dataframe

big_frame.columns = column_header # Write out to a text file, set the output_file variable somewhere big_frame.to_csv(output_file, index=None, sep='\t')

Regards, Minna

…

On Wed, Apr 17, 2024 at 7:18 AM Ho-Chun Huang ***@***.***> wrote: Hi, Minna: After detail examination or the difference between our aqm.yaml, I found mine does not have a space after "-", i.e., "-PMTF". That leads to the error message for inconsistent number of values. After the correction, I can produce the graphic with "- PMTF". Thank you for doing the test at your end. Now I have more questions to ask to move forward. In the MetViewer xml file I can specified the time period I want, but do not know where to put it using METPlotpy. (a) From the configuration of the filename, my point stat output is aggerated by day. I also found there is no time info input for users' yaml. If I want to make a plot for the last 30 days stats, how do I do that? Do I add all 30 days stats files in "stat_input:" separated by comma? Or (b) do I need to first aggregated all daily files into a single file, then add that aggregated filename in "stat_input:"? e.g., from 03/15/2024 to 04/05/2024. Same question also for selected init time (06Z versus 12Z), selected FCST lead time (fcst hour from 23 to 48),...etc. There is no information from the user guide regarding this. Thank you or your help. Ho-Chun — Reply to this email directly, view it on GitHub <#2546 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4UJHSUAGRIYDMH5COME2TY5ZZAVAVCNFSM6AAAAABF64YNFWVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCNBSHE3TG> . You are receiving this because you commented.Message ID: ***@***.***>