Some issues in reliability diagram and ranked histogram generation from METplotpy #2616

BinbinZhou-NOAA · 2024-06-17T14:57:53Z

BinbinZhou-NOAA
Jun 17, 2024

I tested the generation of reliability and ranked histogram but got some issues in the plots:

For the reliability diagram:
(1) if draw plot for just one model, the diagram is OK. But for 2 models, the curve for the model 1 is fine, but the
curve for model 2 has error: the last point (at 90%) and first point (10%) are directly linked.
(2) The location of last point (at 90%) for the model 2 is miss-located. In this case, a triangle polygone.
(3) The legend is shown even though it is sent to "shown"
(4) The colors of the curves for mode1 and model2 are not able to be set (maybe I did not set the colors correctly)
(5) The sharpness plot inside the reliability diagram only has one model. I can not find the option for adding or not adding
the sharpness plot.

For ranked histogram,
(1) one model is fine, but can not put 2 models in one plot (for comparison).

I'll send a completed sample package for testing.

Thanks!

Binbin

EMC/NCEP

BinbinZhou-NOAA · 2024-06-17T15:04:34Z

BinbinZhou-NOAA
Jun 17, 2024
Author

I just ftp the sample files for testing

0 replies

bikegeek · 2024-06-18T18:14:35Z

bikegeek
Jun 18, 2024
Collaborator

Thanks for the information. Due to resource constraints, we will not be able to fully address this issue until we resolve these constraints. In the meantime, I can download the sample data or have someone retrieve it if you can provide the host and directory.

Regards,
Minna

1 reply

jprestop Jun 18, 2024
Maintainer

@bikegeek I grabbed the files that @BinbinZhou-NOAA put on the ftp server and put them on seneca at /d1/projects/METplus/discussions/2616.

BinbinZhou-NOAA · 2024-06-18T18:57:09Z

BinbinZhou-NOAA
Jun 18, 2024
Author

Alicia, Here is the response from DTC on the metplotpy discussion#2616 Binbin

…

On Tue, Jun 18, 2024 at 2:15 PM bikegeek ***@***.***> wrote: Thanks for the information. Due to resource constraints, we will not be able to fully address this issue until we resolve these constraints. In the meantime, I can download the sample data or have someone retrieve it if you can provide the host and directory. Regards, Minna — Reply to this email directly, view it on GitHub <#2616 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQBMPFKNBQIKHBMAOOMZIHTZIB2KBAVCNFSM6AAAAABJOFZN3KVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TQMBZGUYDK> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Binbin Zhou Physical Scientist Lynker at NOAA/NWS/NCEP/EMC 5830 University Research Ct. College Park, MD 20740 ***@***.*** 301-683-3683

0 replies

bikegeek · 2024-07-12T14:35:14Z

bikegeek
Jul 12, 2024
Collaborator

I took a look at the data and reproduced your reliability diagram. It looks like the plot is treating your data as one dataset, rather than separating the points based on model. I'm in the process of figuring out how to set up the yaml config file to show each model as its own line.

1 reply

BinbinZhou-NOAA Jul 12, 2024
Author

Thanks for addressing this issue!

bikegeek · 2024-07-12T15:14:21Z

bikegeek
Jul 12, 2024
Collaborator

I can create a plot with a single model, but haven't yet figured out how to get *both* models as separate lines on the reliability diagram. I am also investigating the mechanism of the inset histogram. I am not sure if it was developed to create a single histogram with multiple-colored bars or generate multiple inset histograms, or if it only supports one model.

…

On Fri, Jul 12, 2024 at 9:07 AM BinbinZhou-NOAA ***@***.***> wrote: Thanks for addressing this issue! — Reply to this email directly, view it on GitHub <#2616 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4UJHVJ4BPYZXBRVSOLUB3ZL7WKHAVCNFSM6AAAAABJOFZN3KVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMBTGI4TCOI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

1 reply

BinbinZhou-NOAA Jul 12, 2024
Author

Yes, in the current version of metplotpy, the reliability diagram for one mode can be correctly created. But we strongly require that the reliability diagram for 2 or more models can be created so that different models can be compared in one plot

bikegeek · 2024-07-12T23:00:59Z

bikegeek
Jul 12, 2024
Collaborator

From your sample data, I generated this plot:

and used this config file:

reliability_debug.yaml.txt

I tested on another dataset I found that also has two models, but cannot find one with more than two models. You needed to indicate the models under the series_val_1 setting:

series_val_1:
  model:
   - REFS_PROB
   - REFS_EAS

and then add settings for the second line/bars via:

colors
con_series
plot_ci
plot_disp
series_line_style
series_line_width
series_order
series_symbols
series_type
show_signif
show_legend

The tricky part is making sure you have the number of settings that match the number of lines you want plotted (which in this case correspond to the two models requested)

0 replies

bikegeek · 2024-07-12T23:22:24Z

bikegeek
Jul 12, 2024
Collaborator

I'm not sure why there are double lines and text (No-Skill and No-resolution), as I don't observe it in the plot I generated with identical settings but different data:

0 replies

BinbinZhou-NOAA · 2024-07-15T13:56:19Z

BinbinZhou-NOAA
Jul 15, 2024
Author

Thanks for resolving this issue. Now I can duplicate the reliability plots for the two models as shown above. The issue for the 2
No-Skill lines and 2 No-resolution lines in the first plot but not in the second plot, I guess the reason is that the reliability curves in the
second plot are pretty close so that the 2 No-Skill lines and 2 No-resolution lines are overlapped while the 2 reliability curves in the first plot
are very different so their No-skill and No-resolution lines are not overlapped. So the code should plot the No-skill and No-resolution lines just once instead of 2 times for each model. As for the number of models, currently 2 models in the same plot is our minimum requirement.

Binbin

0 replies

BinbinZhou-NOAA · 2024-07-15T13:58:48Z

BinbinZhou-NOAA
Jul 15, 2024
Author

Thanks for resolving this issue. Now I can duplicate the reliability plots for the two models as shown above. The issue for the 2 No-Skill lines and 2 No-resolution lines in the first plot but not in the second plot, I guess the reason is that the reliability curves in the second plot are pretty close so that the 2 No-Skill lines and 2 No-resolution lines are overlapped while the 2 reliability curves in the first plot are very different so their No-skill and No-resolution lines are not overlapped. So the code should plot the No-skill and No-resolution lines just once instead of 2 times for each model. As for the number of models, currently 2 models in the same plot is our minimum requirement. Binbin

…

On Fri, Jul 12, 2024 at 7:22 PM bikegeek ***@***.***> wrote: I'm not sure why there are double lines and text (No-Skill and No-resolution), as I don't observe it in the plot I generated with identical settings but different data: plot_20240712_185518.png (view on web) <https://github.com/user-attachments/assets/6dbfd5cc-25e1-4f4c-9e04-346ead2b3357> — Reply to this email directly, view it on GitHub <#2616 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQBMPFNQO6KLPAWNCKD3Q6DZMBQMNAVCNFSM6AAAAABJOFZN3KVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMBTGYZTANA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Binbin Zhou Physical Scientist Lynker at NOAA/NWS/NCEP/EMC 5830 University Research Ct. College Park, MD 20740 ***@***.*** 301-683-3683

0 replies

BinbinZhou-NOAA · 2024-07-15T14:06:26Z

BinbinZhou-NOAA
Jul 15, 2024
Author

One more important issue: the legend is still missing from the new plots even though it is set to be True in the ymal file: show_legend: - 'True' - 'True' Binbin On Mon, Jul 15, 2024 at 9:58 AM Binbin Zhou - NOAA Affiliate < ***@***.***> wrote:

…

Thanks for resolving this issue. Now I can duplicate the reliability plots for the two models as shown above. The issue for the 2 No-Skill lines and 2 No-resolution lines in the first plot but not in the second plot, I guess the reason is that the reliability curves in the second plot are pretty close so that the 2 No-Skill lines and 2 No-resolution lines are overlapped while the 2 reliability curves in the first plot are very different so their No-skill and No-resolution lines are not overlapped. So the code should plot the No-skill and No-resolution lines just once instead of 2 times for each model. As for the number of models, currently 2 models in the same plot is our minimum requirement. Binbin On Fri, Jul 12, 2024 at 7:22 PM bikegeek ***@***.***> wrote: > I'm not sure why there are double lines and text (No-Skill and > No-resolution), as I don't observe it in the plot I generated with > identical settings but different data: > plot_20240712_185518.png (view on web) > <https://github.com/user-attachments/assets/6dbfd5cc-25e1-4f4c-9e04-346ead2b3357> > > — > Reply to this email directly, view it on GitHub > <#2616 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AQBMPFNQO6KLPAWNCKD3Q6DZMBQMNAVCNFSM6AAAAABJOFZN3KVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMBTGYZTANA> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > -- Binbin Zhou Physical Scientist Lynker at NOAA/NWS/NCEP/EMC 5830 University Research Ct. College Park, MD 20740 ***@***.*** 301-683-3683

-- Binbin Zhou Physical Scientist Lynker at NOAA/NWS/NCEP/EMC 5830 University Research Ct. College Park, MD 20740 ***@***.*** 301-683-3683

0 replies

BinbinZhou-NOAA · 2024-07-15T15:44:44Z

BinbinZhou-NOAA
Jul 15, 2024
Author

Good news, the reliability plot for 3 models (REFS_EAS, REFS_PROB and HREF_PROB) can be generated! But no legends are presented in the plot. Binbin [image: image.png] On Mon, Jul 15, 2024 at 10:06 AM Binbin Zhou - NOAA Affiliate < ***@***.***> wrote:

…

One more important issue: the legend is still missing from the new plots even though it is set to be True in the ymal file: show_legend: - 'True' - 'True' Binbin On Mon, Jul 15, 2024 at 9:58 AM Binbin Zhou - NOAA Affiliate < ***@***.***> wrote: > Thanks for resolving this issue. Now I can duplicate the reliability > plots for the two models as shown above. The issue for the 2 No-Skill lines > and 2 No-resolution lines in the first plot but not in the second plot, I > guess the reason is that the reliability curves in the second plot are > pretty close so that the 2 No-Skill lines and 2 No-resolution lines are > overlapped while the 2 reliability curves in the first plot are very > different so their No-skill and No-resolution lines are not overlapped. So > the code should plot the No-skill and No-resolution lines just once instead > of 2 times for each model. As for the number of models, currently 2 models > in the same plot is our minimum requirement. > > Binbin > > On Fri, Jul 12, 2024 at 7:22 PM bikegeek ***@***.***> > wrote: > >> I'm not sure why there are double lines and text (No-Skill and >> No-resolution), as I don't observe it in the plot I generated with >> identical settings but different data: >> plot_20240712_185518.png (view on web) >> <https://github.com/user-attachments/assets/6dbfd5cc-25e1-4f4c-9e04-346ead2b3357> >> >> — >> Reply to this email directly, view it on GitHub >> <#2616 (comment)>, >> or unsubscribe >> <https://github.com/notifications/unsubscribe-auth/AQBMPFNQO6KLPAWNCKD3Q6DZMBQMNAVCNFSM6AAAAABJOFZN3KVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMBTGYZTANA> >> . >> You are receiving this because you were mentioned.Message ID: >> ***@***.***> >> > > > -- > > Binbin Zhou > > Physical Scientist > > Lynker at NOAA/NWS/NCEP/EMC > > 5830 University Research Ct. > > College Park, MD 20740 > > ***@***.*** > > 301-683-3683 > -- Binbin Zhou Physical Scientist Lynker at NOAA/NWS/NCEP/EMC 5830 University Research Ct. College Park, MD 20740 ***@***.*** 301-683-3683

-- Binbin Zhou Physical Scientist Lynker at NOAA/NWS/NCEP/EMC 5830 University Research Ct. College Park, MD 20740 ***@***.*** 301-683-3683

0 replies

bikegeek · 2024-07-15T15:46:42Z

bikegeek
Jul 15, 2024
Collaborator

I am looking into the legends. It looks like they are getting generated, but I need to investigate why they aren't showing up on the plot. On Mon, Jul 15, 2024 at 9:45 AM BinbinZhou-NOAA ***@***.***> wrote:

…

Good news, the reliability plot for 3 models (REFS_EAS, REFS_PROB and HREF_PROB) can be generated! But no legends are presented in the plot. Binbin [image: image.png] On Mon, Jul 15, 2024 at 10:06 AM Binbin Zhou - NOAA Affiliate < ***@***.***> wrote: > One more important issue: the legend is still missing from the new plots > even though it is set to be True in > the ymal file: > show_legend: > - 'True' > - 'True' > > Binbin > > On Mon, Jul 15, 2024 at 9:58 AM Binbin Zhou - NOAA Affiliate < > ***@***.***> wrote: > >> Thanks for resolving this issue. Now I can duplicate the reliability >> plots for the two models as shown above. The issue for the 2 No-Skill lines >> and 2 No-resolution lines in the first plot but not in the second plot, I >> guess the reason is that the reliability curves in the second plot are >> pretty close so that the 2 No-Skill lines and 2 No-resolution lines are >> overlapped while the 2 reliability curves in the first plot are very >> different so their No-skill and No-resolution lines are not overlapped. So >> the code should plot the No-skill and No-resolution lines just once instead >> of 2 times for each model. As for the number of models, currently 2 models >> in the same plot is our minimum requirement. >> >> Binbin >> >> On Fri, Jul 12, 2024 at 7:22 PM bikegeek ***@***.***> >> wrote: >> >>> I'm not sure why there are double lines and text (No-Skill and >>> No-resolution), as I don't observe it in the plot I generated with >>> identical settings but different data: >>> plot_20240712_185518.png (view on web) >>> < https://github.com/user-attachments/assets/6dbfd5cc-25e1-4f4c-9e04-346ead2b3357> >>> >>> — >>> Reply to this email directly, view it on GitHub >>> < #2616 (comment)>, >>> or unsubscribe >>> < https://github.com/notifications/unsubscribe-auth/AQBMPFNQO6KLPAWNCKD3Q6DZMBQMNAVCNFSM6AAAAABJOFZN3KVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMBTGYZTANA> >>> . >>> You are receiving this because you were mentioned.Message ID: >>> ***@***.***> >>> >> >> >> -- >> >> Binbin Zhou >> >> Physical Scientist >> >> Lynker at NOAA/NWS/NCEP/EMC >> >> 5830 University Research Ct. >> >> College Park, MD 20740 >> >> ***@***.*** >> >> 301-683-3683 >> > > > -- > > Binbin Zhou > > Physical Scientist > > Lynker at NOAA/NWS/NCEP/EMC > > 5830 University Research Ct. > > College Park, MD 20740 > > ***@***.*** > > 301-683-3683 > -- Binbin Zhou Physical Scientist Lynker at NOAA/NWS/NCEP/EMC 5830 University Research Ct. College Park, MD 20740 ***@***.*** 301-683-3683 — Reply to this email directly, view it on GitHub <#2616 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4UJHWOJJRC5LTVSEM2MRLZMPVADAVCNFSM6AAAAABJOFZN3KVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMBVGI4TENA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

0 replies

bikegeek · 2024-07-15T16:35:33Z

bikegeek
Jul 15, 2024
Collaborator

I found out why setting the show_legend values to True doesn't work. Apparently plotly is expecting 0 or 1 instead of True or False:

To show legends, do the following:

show_legend:

1
1

I'll have to do something in this and the other plotly plots so the behavior is consistent. It isn't very intuitive to set 0/1.

0 replies

BinbinZhou-NOAA · 2024-07-15T17:48:06Z

BinbinZhou-NOAA
Jul 15, 2024
Author

After this modification, now the legends are presented. See the plot [image: image.png] Thanks! Binbin

…

On Mon, Jul 15, 2024 at 12:36 PM bikegeek ***@***.***> wrote: I found out why setting the show_legend values to True doesn't work. Apparently plotly is expecting 0 or 1 instead of True or False: To show legends, do the following: show_legend: - 1 - 1 I'll have to do something in this and the other plotly plots so the behavior is consistent. It isn't very intuitive to set 0/1. — Reply to this email directly, view it on GitHub <#2616 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQBMPFIA64DJ4SF4K5JWSFTZMP26XAVCNFSM6AAAAABJOFZN3KVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMBVGMZTSNQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Binbin Zhou Physical Scientist Lynker at NOAA/NWS/NCEP/EMC 5830 University Research Ct. College Park, MD 20740 ***@***.*** 301-683-3683

1 reply

bikegeek Jul 15, 2024
Collaborator

I created a Github issue:

dtcenter/METplotpy#455

Please add anything else in the comments.

BinbinZhou-NOAA · 2024-07-15T18:21:49Z

BinbinZhou-NOAA
Jul 15, 2024
Author

Sure, thanks! Binbin

…

On Mon, Jul 15, 2024 at 2:19 PM bikegeek ***@***.***> wrote: I created a Github issue: dtcenter/METplotpy#455 <dtcenter/METplotpy#455> Please add anything else in the comments. — Reply to this email directly, view it on GitHub <#2616 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQBMPFNQIZZJA4ALEOXVS6LZMQHDZAVCNFSM6AAAAABJOFZN3KVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMBVGQZDQNA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Binbin Zhou Physical Scientist Lynker at NOAA/NWS/NCEP/EMC 5830 University Research Ct. College Park, MD 20740 ***@***.*** 301-683-3683

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Some issues in reliability diagram and ranked histogram generation from METplotpy #2616

{{title}}

Replies: 15 comments 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Some issues in reliability diagram and ranked histogram generation from METplotpy #2616

BinbinZhou-NOAA Jun 17, 2024

Replies: 15 comments · 4 replies

BinbinZhou-NOAA Jun 17, 2024 Author

bikegeek Jun 18, 2024 Collaborator

jprestop Jun 18, 2024 Maintainer

BinbinZhou-NOAA Jun 18, 2024 Author

bikegeek Jul 12, 2024 Collaborator

BinbinZhou-NOAA Jul 12, 2024 Author

bikegeek Jul 12, 2024 Collaborator

BinbinZhou-NOAA Jul 12, 2024 Author

bikegeek Jul 12, 2024 Collaborator

bikegeek Jul 12, 2024 Collaborator

BinbinZhou-NOAA Jul 15, 2024 Author

BinbinZhou-NOAA Jul 15, 2024 Author

BinbinZhou-NOAA Jul 15, 2024 Author

BinbinZhou-NOAA Jul 15, 2024 Author

bikegeek Jul 15, 2024 Collaborator

bikegeek Jul 15, 2024 Collaborator

BinbinZhou-NOAA Jul 15, 2024 Author

bikegeek Jul 15, 2024 Collaborator

BinbinZhou-NOAA Jul 15, 2024 Author

BinbinZhou-NOAA
Jun 17, 2024

Replies: 15 comments 4 replies

BinbinZhou-NOAA
Jun 17, 2024
Author

bikegeek
Jun 18, 2024
Collaborator

jprestop Jun 18, 2024
Maintainer

BinbinZhou-NOAA
Jun 18, 2024
Author

bikegeek
Jul 12, 2024
Collaborator

BinbinZhou-NOAA Jul 12, 2024
Author

bikegeek
Jul 12, 2024
Collaborator

BinbinZhou-NOAA Jul 12, 2024
Author

bikegeek
Jul 12, 2024
Collaborator

bikegeek
Jul 12, 2024
Collaborator

BinbinZhou-NOAA
Jul 15, 2024
Author

BinbinZhou-NOAA
Jul 15, 2024
Author

BinbinZhou-NOAA
Jul 15, 2024
Author

BinbinZhou-NOAA
Jul 15, 2024
Author

bikegeek
Jul 15, 2024
Collaborator

bikegeek
Jul 15, 2024
Collaborator

BinbinZhou-NOAA
Jul 15, 2024
Author

bikegeek Jul 15, 2024
Collaborator

BinbinZhou-NOAA
Jul 15, 2024
Author