PointStat verification of windspeed, upper air levels

[zhumingying]

Hi,

I need to do verification of windspeed, ME or RMSE on forecast against obs.

(1) I have forecast data of u and v, and first I do is_u_wind = true, and is_v_wind = true to get windspeed, METplus has successfully finished running. But I could not find ME and RMSE of windspeed in VCNT file, maybe I was wrong?

(2) I add windspeed field in the forecast file, and do windspeed verification directly.
forecast fields: u_p, v_p, ws_p
obs fields: UGRD, VGRD, SOB
METplus has successfully finished running, but when I checked the log file, there is no matched pairs for windspeed. see

log file: /glade/scratch/zhuming/metplus4/logs/metplus.log.20211130085723
/glade/work/zhuming/METplus4/PointStat_wrf_upa.conf

Anyone can help me?
Thanks!
Zhuming

[JohnHalleyGotway]

@zhumingying I see you have a question about the verification of winds in MET. Thanks for sending the path to your METplus logs on Cheyenne. That made it really easy for me to see your output.

I see that you were successful in creating lots and lots of VCNT output lines:

 cat /glade/scratch/zhuming/metplus4/point_stat_wrf_upa/2015*/point_stat*_vcnt.txt | wc -l
24062

I think your question is really more about what those VCNT lines actually contain. Please see this VCNT line type description. In here, the columns with "SPEED" refer to wind speed while "DIR" refer to wind direction. The "SPEED_ERR" is the mean error, i.e. average forecast speed minus average observation speed. Please also see Appendix G for more details.

Does that answer your question?

[JohnHalleyGotway]

@zhumingying I just want to followup to see if you have any remaining questions. If not, you could select the "Mark as answer" button for my response. Once a discussion has been marked as answered, we then "Lock conversation" on it to prevent future posts. That encourages users to create new discussions for new questions, and also makes it easier for us to keep track of open questions.

[zhumingying]

Hi John, Thanks for replying. But I'm still confused. I can understand SPEED_ERR = 1/N sum(SFi) - 1/N sum(SOi) = 1/n sum(SFi-SOi) = SPEED_ME But what's RMSE or MAE of Wind_Speed? RMSE of wind speed is RMSVE in VCNT file? MAE = SPEED_ABSERR? It seems they are different, MAE = 1/N sum |SFi-SOi|. SPEED_ABSERR = | 1/N sum (SFi-SOi) |

…

On Wed, Dec 1, 2021 at 12:40 PM johnhg ***@***.***> wrote: @zhumingying <https://github.com/zhumingying> I see you have a question about the verification of winds in MET. Thanks for sending the path to your METplus logs on Cheyenne. That made it really easy for me to see your output. I see that you were successful in creating lots and lots of VCNT output lines: cat /glade/scratch/zhuming/metplus4/point_stat_wrf_upa/2015*/point_stat*_vcnt.txt | wc -l 24062 I think your question is really more about what those VCNT lines actually contain. Please see this VCNT line type description <https://met.readthedocs.io/en/latest/Users_Guide/point-stat.html?highlight=VCNT#id23>. In here, the columns with "SPEED" refer to wind speed while "DIR" refer to wind direction. The "SPEED_ERR" is the mean error, i.e. average forecast speed minus average observation speed. Please also see Appendix G <https://met.readthedocs.io/en/latest/Users_Guide/appendixG.html> for more details. Does that answer your question? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1290 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFASISP64ANQXVUB2V6MTGTUOZ2ZBANCNFSM5JDQUXQA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[JohnHalleyGotway]

@zhumingying, I did some more digging. By way of comparison, I configured Point-Stat (using
PointStatConfig.txt
) to compute continuous statistics for wind speed directly (CNT line type) as well as vector statistics (VCNT line type). The stats are computed for 4184 pairs over CONUS and can be found in
wind.txt and
vcnt.txt.

Looking more closely at that data:

cat wind.txt | awk '{print $10,$56,$69,$72,$78}'
FCST_VAR ME      MAE     MSE     RMSE
WIND     1.01734 1.97753 6.21009 2.492

cat vcnt.txt | awk '{print $10,$56,$59,$62,$68,$71}'
FCST_VAR  MSVE     RMSVE   FBAR_SPEED OBAR_SPEED VDIFF_SPEED SPEED_ERR SPEED_ABSERR
UGRD_VGRD 13.31173 3.64853 0.61088    0.49711    0.26777     0.11377   0.11377

As you can see, the results in the CNT line for wind speed (i.e. WIND) and the VCNT line for vector-stats do differ a lot. So if you're after the former, I do think configuring Point-Stat to verifying wind speed directly would avoid some confusion.

[zhumingying]

Thanks! You're right, I tried Point-Stat to verify surface wind speed directly. It seems the Mean of WSP is almost the same, but ME of WSP is so different, by comparing the results from CNT and VCNT file. see log file /glade/scratch/zhuming/metplus4/logs/metplus.log.20211129094815 config file: /glade/work/zhuming/METplus4/PointStat_wrf_sfc.conf I also tried Point-Stat to verify upper-air wind speed directly. METplus has successfully finished running, but when I checked the log file, there are no matched pairs for wind speed. forecast fields: u_p, v_p, ws_p obs fields: UGRD, VGRD, SOB log file: /glade/scratch/zhuming/metplus4/logs/metplus.log.20211130085723 config file: /glade/work/zhuming/METplus4/PointStat_wrf_upa.conf Thanks! Zhuming

…

On Fri, Dec 10, 2021 at 11:20 AM johnhg ***@***.***> wrote: @zhumingying <https://github.com/zhumingying>, I did some more digging. By way of comparison, I configured Point-Stat (using PointStatConfig.txt <https://github.com/dtcenter/METplus/files/7694640/PointStatConfig.txt> ) to compute continuous statistics for wind speed directly (CNT line type) as well as vector statistics (VCNT line type). The stats are computed for 4184 pairs over CONUS and can be found in wind.txt <https://github.com/dtcenter/METplus/files/7694297/wind.txt> and vcnt.txt <https://github.com/dtcenter/METplus/files/7694304/vcnt.txt>. Looking more closely at that data: cat wind.txt | awk '{print $10,$56,$69,$72,$78}' FCST_VAR ME MAE MSE RMSE WIND 1.01734 1.97753 6.21009 2.492 cat vcnt.txt | awk '{print $10,$56,$59,$62,$68,$71}' FCST_VAR MSVE RMSVE FBAR_SPEED OBAR_SPEED VDIFF_SPEED SPEED_ERR SPEED_ABSERR UGRD_VGRD 13.31173 3.64853 0.61088 0.49711 0.26777 0.11377 0.11377 As you can see, the results in the CNT line for wind speed (i.e. WIND) and the VCNT line for vector-stats do differ a lot. So if you're after the former, I do think configuring Point-Stat to verifying wind speed directly would avoid some confusion. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1290 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFASISPOM25QNRNLNWEKPY3UQJAF5ANCNFSM5JDQUXQA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[JohnHalleyGotway]

Thanks for sending the full paths. Looking in your config/log files, I see that you're comparing the forecast variable ws_10m to observations of "SOB". And Point-Stat reports that it's finding plenty of matched pairs:

DEBUG 2: Processing ws_10m(20150717_120000,*,*) versus SOB/Z10, for observation type ADPSFC, over region FULL, for interpolation method BILIN(4), using 601 matched pairs.

And I see 3 output lines for ws_10m in the CNT output file:

grep ws_10m /glade/scratch/zhuming/metplus4/point_stat_wrf_sfc/2015071500/point_stat_init2015071500_F006_000000L_20150715_060000V_cnt.txt | wc -l
3

Similarly, there are 1230 ws_10m output lines in the matched pair output file:

grep ws_10m /glade/scratch/zhuming/metplus4/point_stat_wrf_sfc/2015071500/point_stat_init2015071500_F006_000000L_20150715_060000V_mpr.txt | wc -l
1230

Can you please explain what it is that is missing?

Thanks,
John

[zhumingying]

Hi John, Sorry, I made you confused. I mean doing Point-Stat to verify surface wind speed directly is fine, but it is not working for upper-air wind speed. I also tried Point-Stat to verify upper-air wind speed directly. METplus has successfully finished running, but when I checked the log file, there are no matched pairs for wind speed. forecast fields: u_p, v_p, ws_p obs fields: UGRD, VGRD, SOB log file: /glade/scratch/zhuming/metplus4/logs/metplus.log.20211130085723 config file: /glade/work/zhuming/METplus4/PointStat_wrf_upa.conf Thanks! Zhuming

[JohnHalleyGotway]

Ah yes, I see the log message about 0 matched pairs now:

grep "using 0 matched pairs" /glade/scratch/zhuming/metplus4/logs/metplus.log.20211130085723 | grep SOB | wc -l
1020

Looking at the surround log messages for one of them I see the following:

DEBUG 2: Processing ws_p(201507150000,0,*,*) versus SOB/P930-920, for observation type ADPUPA, over region FULL, for interpolation method BILIN(4), using 0 matched pairs.
DEBUG 2: Number of matched pairs   = 0
DEBUG 2: Observations processed    = 709645
DEBUG 2: Rejected: station id      = 0
DEBUG 2: Rejected: obs var name    = 702934
DEBUG 2: Rejected: valid time      = 0
DEBUG 2: Rejected: bad obs value   = 0
DEBUG 2: Rejected: off the grid    = 6085
DEBUG 2: Rejected: topography      = 0
DEBUG 2: Rejected: level mismatch  = 611
DEBUG 2: Rejected: quality marker  = 0
DEBUG 2: Rejected: message type    = 15
DEBUG 2: Rejected: masking region  = 0
DEBUG 2: Rejected: bad fcst value  = 0
DEBUG 2: Rejected: bad climo mean  = 0
DEBUG 2: Rejected: bad climo stdev = 0
DEBUG 2: Rejected: mpr filter      = 0
DEBUG 2: Rejected: duplicates      = 0

Please look closely at these rejection reason counts. And note that the order in which these counts are listed in the log file matches the order in which the filter logic is applied in the code. Looking up from the bottom, that means that you had 15 observations that matched all the criteria above it but did NOT match the message type you requested (namely "ADPUPA"). So the first thing to check is what message types are actually present in your input file?

Scrolling up, I see this is the input observation file:

DEBUG 1: Observation File: /glade/scratch/zhuming/metplus4/pb2nc2/pbs.2015071500.nc

I do see that this file does contain the ADPUPA message type:

ncdump -v hdr_typ_table /glade/scratch/zhuming/metplus4/pb2nc2/pbs.2015071503.nc
 hdr_typ_table =
  "ADPUPA",

However, we don't know if there are actually any SOB observations for the ADPUPA message type. I ran plot_point_obs and figured out that there actually are no SOB observations for ADPUPA:

plot_point_obs /glade/scratch/zhuming/metplus4/pb2nc2/pbs.2015071503.nc plot.ps -obs_var SOB -msg_typ ADPUPA
DEBUG 2: Finished plotting 0 locations.

But I'm not sure what the actual message type is for these obs. Let's look a bit closer but dumping the NetCDF obs file to ascii. This is REALLY diving into the details:

module load R
Rscript /glade/p/ral/jntp/MET/MET_releases/10.0.0/share/met/Rscripts/pntnc2ascii.R /glade/scratch/zhuming/metplus4/pb2nc2/pbs.2015071503.nc > pbs.2015071503.txt

See which message types actually contain the SOB observation:

grep SOB pbs.2015071503.txt | awk '{print $1}' | sort -u
ADPSFC
SFCSHP

Since ADPUPA isn't in this list, there aren't any ADPUPA observations of SOB present and that's why Point-Stat is finding 0 matched pairs.

[zhumingying]

Hi John, Could you help me? adding the ADPUPA observations of SOB on my obs data file pbs*.nc I have the config file /glade/work/zhuming/METplus4/PB2NC.conf Thanks a lot! Zhuming

[JohnHalleyGotway]

@zhumingying when verifying wind speed, I haven't typically used the "SOB" which I assume to be the native wind speed observation reported by the PREPBUFR file. Instead, I typically used observation named "WIND" which is the GRIB code abbreviation for wind speed.

In the PB2NC Config file, take a look at this line:
https://github.com/dtcenter/MET/blob/5e1f551d86fcb5b731e3a83c0b043dbd7ef3c7aa/met/data/config/PB2NCConfig_default#L95
Consider adding "D_WIND" to the end of that line.
Also, looking at this line:
https://github.com/dtcenter/MET/blob/5e1f551d86fcb5b731e3a83c0b043dbd7ef3c7aa/met/data/config/PB2NCConfig_default#L119
Note that "D_WIND" gets renamed as "WIND" in the output.

By convention, the "D_" prefix tells PB2NC to derive wind speed from the U/V components and then rename observations of that type as "WIND" in the output. In your PB2NC config file, you've left this line blank:
PB2NC_OBS_BUFR_VAR_LIST =

I suspect that's instructing PB2NC to dump all available observations to the output, but likely NOT derive variables like wind speed. Try running it like this:
PB2NC_OBS_BUFR_VAR_LIST = QOB, TOB, ZOB, UOB, VOB, SOB, D_DPT, D_WIND, D_RH, D_MIXR

And then consider verifying against the WIND observation instead of SOB in Point-Stat.

[JohnHalleyGotway]

Hi I'm wondering if you have any remaining questions on this discussion topic? When discussions are finished, we mark them as "answered" and lock them to prevent future posts. That's our way of encouraging users to post new questions in new discussions. Feel free to select the best "answer" yourself.