How to generate the METplotpy's input reliability diagram data file from MET/METplus (PSTD/PCT) stat files #2527
Replies: 9 comments 3 replies
-
|
Hello Binbin, we are finishing up the development of the METdataio reformatter for the PCT stat file for the RRFS data. There are still some issues with running the output from the reformatted data into the METcalcpy agg_stat.py module. We don't have the budget to fix these issues until sometime in July. The PSTD output will also need to be reformatted into a format readable by METplotpy. The reformatting support for that linetype does not currently exist. Here is a link to the METdataio reformatting documentation: We do have support for reformatting the rank histogram from the RHIST linetype (see the bottom of section 7.1 in the document above). |
Beta Was this translation helpful? Give feedback.
-
|
Bikegeek, Thank you very much for answering my question and showing me the usage of metdataio for reformatting the MET stat data files. So I may first focus on the generation of rank histogram diagram. Binbin |
Beta Was this translation helpful? Give feedback.
-
|
Please let me know if you have any questions with the histogram plot, the
documentation can always be improved.
…On Thu, Mar 28, 2024 at 7:19 AM BinbinZhou-NOAA ***@***.***> wrote:
Bikegeek,
Thank you very much for answering my question and the suggestion of using
METdataio. So I may first focus on the generation of rank histogram diagram.
Binbin
—
Reply to this email directly, view it on GitHub
<#2527 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AA4UJHWG3ZW5PXK6IJSII4DY2QKHHAVCNFSM6AAAAABFLNBSLKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DSNBQGU2TE>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Following the instruction of METdataio document step by step, I have run the sample file RHIST.yaml from the METdataio testing directory successfully with the output file in the output directory: Only 2 WARNINGs: I think these warnings should be fine. But looking at the output file, the contents are totally different from the sample file rank_hist.data SO I am thinking there should ne another step to process the output file of the METdataio reformatting to obtain the similar format Thanks! Binbin |
Beta Was this translation helpful? Give feedback.
-
|
Those warnings are indeed informative in nature. The METplotpy rank
histogram plot is looking for the presence of the n_rank, rank_i, and
i_value columns (which the reformatter puts as the last columns in the
output data file). The example data in the METplotpy test directory comes
from the query made in METviewer. The METdataio reformatter maintains the
data from the MET rust linetype. You can use the series_val_1 setting in
the rank_hist.yaml (the 'custom' config file) to limit your data to
fields/columns of interest:
series_val_1:
vx_mask:
- FULL
- GRB
- NWC
Let me see if the rank histogram plot also allows you to do further
filtering via any other mechanisms (e.g. the line plot supports
fixed_vars_vals_input to do some subsetting)- not all plots support this
setting.
…On Thu, Mar 28, 2024 at 9:19 AM BinbinZhou-NOAA ***@***.***> wrote:
Following the instruction of METdataio document step by step, I have run
the sample file RHIST.yaml from the METdataio testing directory
successfully with the output file in the output directory:
python $BASE_DIR/METreformat/write_stat_ascii.py RHIST.yaml
Only 2 WARNINGs:
WARNING:root:!!! Stat file
./data/rhist_phist_relp_orank/ensemble_stat_RRFS_GEFS_GF.SPP.SPPT_ADPUPA_NDAS_20220507_060000V.stat
has no data after headers
WARNING:root:!!! Stat file
./data/rhist_phist_relp_orank/ensemble_stat_RRFS_GEFS_GF.SPP.SPPT_ADPUPA_NDAS_20220506_060000V.stat
has no data after headers
I think these warnings should be fine. But looking at the output file, the
contents are totally different from the sample file rank_hist.data
in ../METplotpy/3.0.0-beta3/test/histogram directory
SO I am thinking there should ne another step to process the output file
of the METdataio reformatting to obtain the similar format
file to the rank_hist.data file. Am I right? If yes, what is this process?
Thanks!
Binbin
—
Reply to this email directly, view it on GitHub
<#2527 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AA4UJHTLBYLPJKE65WJKT7TY2QYJ3AVCNFSM6AAAAABFLNBSLKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DSNBSGA4DE>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Hi Binbin, my apologies, I forgot that the rank histogram requires an additional step- it will need METcalcpy agg_stat.py to calculate the sums for the columns, we have a Github issue for this: dtcenter/METcalcpy#343 And it also appears that the fixed_var_vals_input and series_val_1/series_val_2 settings are unused in the rank histogram plot. It is in the METcalcpy aggr_stat config file where you set the fcst_var_val_1 (to indicate TMP, RH, etc.) and the indy_vals, indy_var (i.e. fcst_lead). I will follow-up on that pending Github issue and see if we can work on it for the upcoming beta release (ie. if funding is available). |
Beta Was this translation helpful? Give feedback.
-
|
Bikegeek, and the rank histogram diagram can be generated successfully. But running with this setting failed. The error message is So does this error means I have to run METcalcpy agg_stat.py as you mentioned? If this is the case, what is the command to run Thanks! Binbin |
Beta Was this translation helpful? Give feedback.
-
|
Hi Binbin,
I also observed the same error messages when I attempted the same in my
METplotpy rank histogram config file. Those settings will only apply in
the METcalcpy agg_stat.py module.
1) There isn't support for doing the aggregation via agg_stat.py for the
RHIST line type. Here's the issue I created in December (but forgot about):
dtcenter/METcalcpy#343
I sent an email to Tara and the assignee to see if we could work on this.
Here's a sample agg_stat config file for the ECNT linetype (From the use
case. Replace the !ENV '${xyz}' content with the full path. ENV is only
supported within the use case).
agg_stat_input: !ENV '${AGGREGATE_INPUT_BASE}/ensemble_stat_ecnt.data'
agg_stat_output: !ENV '${AGGREGATE_OUTPUT_BASE}/ecnt_aggregated.data'
alpha: 0.05
append_to_file: null
circular_block_bootstrap: True
derived_series_1: []
derived_series_2: []
event_equal: False
fcst_var_val_1:
TMP:
- ECNT_RMSE
- ECNT_SPREAD_PLUS_OERR
fcst_var_val_2: {}
indy_vals:
- '30000'
- '40000'
- '60000'
- '90000'
- '120000'
- '150000'
- '160000'
- '170000'
- '180000'
- '200000'
- '240000'
- '270000'
indy_var: fcst_lead
line_type: ecnt
list_stat_1:
- ECNT_RMSE
- ECNT_SPREAD_PLUS_OERR
list_stat_2: []
method: perc
num_iterations: 1
num_threads: -1
random_seed: null
and running agg_stat.py looks like this:
python agg_stat.py agg_stat.yaml
If you are interested, there is an end-to-end use case for the *ECNT*
linetype to create spread skill plots from MET output to reformatting and
calculating aggregation statistics:
https://metplus.readthedocs.io/en/develop/generated/model_applications/short_range/UserScript_fcstRRFS_fcstOnly_Reformat_Aggregate_Plot_ecnt_spread_skill.html#sphx-glr-generated-model-applications-short-range-userscript-fcstrrfs-fcstonly-reformat-aggregate-plot-ecnt-spread-skill-py
2) We were working on numerous reformatters for another project and the
skill line plots, ROC, and reliability diagrams were high on their list of
priorities. I was successful in running METcalcpy agg_stat.py on my sample
PCT data and creating a reliability diagram, but it fails on some other
data. Perhaps it will work on yours? Can you provide me with some of your
PCT data and I can see if I can reformat it, calculate the agg stats, and
generate a reliability diagram. If that's successful, then you can at
least generate some reliability diagrams.
Regards,
Minna
…On Thu, Mar 28, 2024 at 2:02 PM BinbinZhou-NOAA ***@***.***> wrote:
Bikegeek,
It seems the output test file "reformatted_rhist_output.txt" generated from running of METdataio/RHIST.yaml can be directly used as input for running of METplotpy/rank_hist.yaml
and the rank histogram diagram can be generated successfully.
I see series_val_1/vx_mask and fcst_var_val_1 are in the
METplotpy/rank_hist.yaml, so I tried to use them to filter.
There are multiple domains (CONUS, FULL) and multiple variables (TMP, RH,
etc) in reformatted_rhist_output.txt, so I set
series_val_1:
vx_mask:
- CONUS
to filter out "FULL".
Running with this setting is also successful. But not sure the generated
png graphic file is correct or not.
Then I tried to set
fcst_var_val_1:
- TMP
But running with this setting failed. The error message is
Traceback (most recent call last):
......
File
"/apps/ops/para/libs/intel/19.1.3.304/METplotpy/3.0.0-beta3/metplotpy/plots/config.py",
line 259, in _get_fcst_vars
all_fcst_vars = [*fcst_var_val_dict.keys()]
AttributeError: 'list' object has no attribute 'keys'
So does this error means I have to run METcalcpy agg_stat.py as you
mentioned? If this is the case, what is the command to run
METcalcpy agg_stat.py. I checked the METcalcpy test directory
../METcalcpy/3.0.0-beta3/test and see one yaml file
rrfs_ecnt_config_agg_stat.yaml
Is this the file I can use?
Thanks!
Binbin
—
Reply to this email directly, view it on GitHub
<#2527 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AA4UJHUB3HQQHPX64PBW4YTY2RZNNAVCNFSM6AAAAABFLNBSLKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DSNBVGA3DK>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Minna, Thanks for your further advice. to generate specific field like TMP, I could just simply retrieve those TMP data from the output text file generated by metdataio RHIST reformatting to form a new test file. Use this new smaller text file as input for metplotpy/rank_histogram. Thanks! Binbin |
Beta Was this translation helpful? Give feedback.
-
|
Thanks Binbin. I don't have access to Hera, but Julie P does. I can ask
her to retrieve the data for me.
Regards,
Minna
…On Thu, Mar 28, 2024 at 3:27 PM BinbinZhou-NOAA ***@***.***> wrote:
Minna,
Thanks for your further advice. to generate specific field like TMP, I
could just simply retrieve those TMP data from the output text file
generated by metdataio RHIST reformatting to form a new test file. Use this
new smaller text file as input for metplotpy/rank_histogram.
Yes, we have EVS/HREF routine verification output stat files with PCT line
type. I have copied 3 of them onto HERA.
The directory is
/scratch2/NCEPDEV/fv3-cam/noscrub/Binbin.Zhou/metplotpy/data
If you can access HERA, please look at them to see if the reliability
diagram can be generated from them.
Thanks!
Binbin
—
Reply to this email directly, view it on GitHub
<#2527 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AA4UJHWZZKTUXA3ANUFYDVTY2SDLNAVCNFSM6AAAAABFLNBSLKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DSNBVG42TG>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Hi Binbin,
Good news, I can create a reliability diagram from your data (just for the
PROB(APCP_01>12.700). The agg_stat.py config file only recognizes one fcst
var. I will directly email you the config files I used to generate the
plot (and instructions).
…On Thu, Mar 28, 2024 at 3:27 PM BinbinZhou-NOAA ***@***.***> wrote:
Minna,
Thanks for your further advice. to generate specific field like TMP, I
could just simply retrieve those TMP data from the output text file
generated by metdataio RHIST reformatting to form a new test file. Use this
new smaller text file as input for metplotpy/rank_histogram.
Yes, we have EVS/HREF routine verification output stat files with PCT line
type. I have copied 3 of them onto HERA.
The directory is
/scratch2/NCEPDEV/fv3-cam/noscrub/Binbin.Zhou/metplotpy/data
If you can access HERA, please look at them to see if the reliability
diagram can be generated from them.
Thanks!
Binbin
—
Reply to this email directly, view it on GitHub
<#2527 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AA4UJHWZZKTUXA3ANUFYDVTY2SDLNAVCNFSM6AAAAABFLNBSLKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DSNBVG42TG>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Great! I'd like to know all of the steps, from metdataio, to metcalcpy, to metplotpy. Thanks! |
Beta Was this translation helpful? Give feedback.
-
METplotpy
was just installed on WCOSS2, so I tried to use it, but I am new to METplotpy. Right now I can generate the reliability diagram plot from the sample testing input data and yaml files in METplotpy on WCOSS2. But I don't know how to generate my own reliability diagram input data file from our EVS ensemble verification (PSTD, PCT) stat files. I'd like to know what tool(s) can be used to generate this type of files (also including rank histogram and relative position diagram, and some others).
Thanks!
Binbin
EMC
Beta Was this translation helpful? Give feedback.
All reactions