WARNING message for PB2NC to process GDAS prepbufr files over Hawaii #2581

BinbinZhou-NOAA · 2024-05-13T20:47:57Z

BinbinZhou-NOAA
May 13, 2024

We have an issue in PB2NC processing of gdas prepbufr files over Hawaii when the prepbufr is upgraded from v1.1 to v1.2
The WARNING message says:

WARNING: process_pbfile() -> No PrepBufr messages retained from file: /lfs/h1/ops/para/com/obsproc/v1.2/gdas.20240512/00/atmos/gdas.t00z.prepbufr

This PB2NC procedure only processes ADPUPA data type.
The WARNING causes the processing job failed and the output NetCDF file over Hawaii was not created.

But the same processing job over Puerto Rico has no such WARNING and output NetCDF file over Puerto Rico can be generated.
We are suspecting that the ADPUPA data in the v1.2 version over Hawaii may have some problem, but want to know
how ADPUPA data are read in the PB2NC, and what variable causes the problem.
The PB2NC conf file can be seen at
https://github.com/NOAA-EMC/EVS/blob/develop/parm/metplus_config/stats/cam/grid2obs/Pb2nc_obsGFS_Prepbufr_Profile.conf

We just found this happens only on 20240513, while all other days were good.

Thanks!

Binbin Zhou
EMC/NCEP

Answered by hsoh-u

May 15, 2024

0 retained message implies no variables (missing all variables). PB2NC did not have the chance to look at the variables because all of them were filtered out by looking at the header messages. PB2NC reads the header message (obs time, lat/lon, message type, station_id, ...), applies the filtering, and reads the variable messages if not filtered out.

View full answer

PerryShafran-NOAA · 2024-05-13T21:03:40Z

PerryShafran-NOAA
May 13, 2024
Collaborator

Just as an addendum to this, this is not a problem with the current version of v1.1 prepbufr, which is in /lfs/h1/ops/prod/com/obsproc/v1.1.gdas.20240512/00/atmos/gdas.t00z.prepbufr. We have pinpointed the issues in Hawaii, which has only 2 observations. There are some slight differences in the observations, but we need to know what is causing the job to fail with the new process. We can send you the different prepbufr files through your ftp site if we need to.

0 replies

hsoh-u · 2024-05-13T21:19:50Z

hsoh-u
May 13, 2024
Collaborator

Would you upload the PREPBUFR files to the ftp server or give us direction to access them (which machine and path)?

0 replies

PerryShafran-NOAA · 2024-05-13T21:35:35Z

PerryShafran-NOAA
May 13, 2024
Collaborator

If you have access to WCOSS2 (currently Cactus is the development machine), you can find the two prepbufr files here:

/lfs/h1/ops/para/com/obsproc/v1.2/gdas.20240512/00/atmos/gdas.t00z.prepbufr
/lfs/h1/ops/prod/com/obsproc/v1.1/gdas.20240512/00/atmos/gdas.t00z.prepbufr

@BinbinZhou-NOAA plans to upload them to you tomorrow morning. But if you have WCOSS2 access, you can find these files at these locations.

1 reply

jprestop May 13, 2024
Maintainer

@PerryShafran-NOAA I'm the only DTC member with WCOSS2 access currently. Unfortunately, I am unable to grab them now. If @BinbinZhou-NOAA could upload them tomorrow morning and post here once he has done so, it would be much appreciated. If I see that they are not here once I start work tomorrow, I can grab them.

BinbinZhou-NOAA · 2024-05-14T00:40:03Z

BinbinZhou-NOAA
May 14, 2024
Author

I'll update all relevant data files to DTC tomorrow. Binbin

…

On Mon, May 13, 2024 at 6:25 PM Julie Prestopnik ***@***.***> wrote: @PerryShafran-NOAA <https://github.com/PerryShafran-NOAA> I'm the only DTC member with WCOSS2 access currently. Unfortunately, I am unable to grab them now. If @BinbinZhou-NOAA <https://github.com/BinbinZhou-NOAA> could upload them tomorrow morning and post here once he has done so, it would be much appreciated. If I see that they are not here once I start work tomorrow, I can grab them. — Reply to this email directly, view it on GitHub <#2581 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQBMPFO2RGWIXZJQRBVUYDDZCE4WRAVCNFSM6AAAAABHU5A566VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TIMRWGQ4DC> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Binbin Zhou Physical Scientist Lynker at NOAA/NWS/NCEP/EMC 5830 University Research Ct. College Park, MD 20740 ***@***.*** 301-683-3683

0 replies

PerryShafran-NOAA · 2024-05-14T12:25:15Z

PerryShafran-NOAA
May 14, 2024
Collaborator

@BinbinZhou-NOAA @hsoh-u @jprestop I don't think that we need to have this ticket investigated any further. We also had the obsproc team look into the prepbufr file, and it turns out that it was corrupted. Thus that is how the error was generated within METplus.

However, it might be useful for METplus to be able to not give an ERROR if there are no observations found or if a corrupted observation is encountered. We can give more details about that, if you wish to keep this ticket open a bit longer.

1 reply

hsoh-u May 14, 2024
Collaborator

@BinbinZhou-NOAA , would you upload the corrupted files. I would like try if MET can detect the corrupted files.

BinbinZhou-NOAA · 2024-05-14T15:07:42Z

BinbinZhou-NOAA
May 14, 2024
Author

I just updated all of the files for this investigation, including 2 prepbufr files (date: 20240512, 00z cycle, version 1.1 and 1.2 respectively): gdas.t00z.prepbufr.v1.1 gdas.t00z.prepbufr.v1.2 2 domain polygon files over Hawaii and Puerto Rico: Hawaii_HREF.nc PRico_HREF.nc 1 PB2NC conf file Pb2nc_obsGFS_Prepbufr_Profile.conf When testing, the conf file should be modified to read INPUT prepbufr file. Please test them over both Hawaii and Puerto. Our testings show that over Puerto Rico, processing both prepbufr files is ok, but over Hawaii, processing v1.1 is good, but processing v1.2 not. Thanks! Binbin

…

On Tue, May 14, 2024 at 10:49 AM Howard Soh ***@***.***> wrote: @BinbinZhou-NOAA <https://github.com/BinbinZhou-NOAA> , would you upload the corrupted files. I would like try if MET can detect the corrupted files. — Reply to this email directly, view it on GitHub <#2581 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQBMPFPEJJ6JDKIPT5DUP23ZCIP6ZAVCNFSM6AAAAABHU5A566VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TIMZUHE2DC> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Binbin Zhou Physical Scientist Lynker at NOAA/NWS/NCEP/EMC 5830 University Research Ct. College Park, MD 20740 ***@***.*** 301-683-3683

1 reply

PerryShafran-NOAA May 14, 2024
Collaborator

Thank you Binbin. Let's see what Howard can do with these files.

BinbinZhou-NOAA · 2024-05-14T15:19:54Z

BinbinZhou-NOAA
May 14, 2024
Author

Perry, Keep this issue so that DTC can look at the prepbufr files to find what variable causes the PB2NC to break down. Binbin

…

On Tue, May 14, 2024 at 11:10 AM PerryShafran-NOAA ***@***.***> wrote: Thank you Binbin. Let's see what Howard can do with these files. — Reply to this email directly, view it on GitHub <#2581 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQBMPFK7EOMD436QGAABHMLZCISMPAVCNFSM6AAAAABHU5A566VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TIMZVGE4DE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Binbin Zhou Physical Scientist Lynker at NOAA/NWS/NCEP/EMC 5830 University Research Ct. College Park, MD 20740 ***@***.*** 301-683-3683

1 reply

PerryShafran-NOAA May 14, 2024
Collaborator

@BinbinZhou-NOAA agreed!

hsoh-u · 2024-05-14T18:22:05Z

hsoh-u
May 14, 2024
Collaborator

It worked for me at seneca (our developing machine), It took more than 30 minutes to process 5 variables (2M obs) and more than 1 hour for all variables (14M obs).

Tested with MET_V11.1 (built on Nov 2023)
PB2NCConfig files:
- test1: the empty configuration: five variables obs_bufr_var = [ "QOB", "TOB", "ZOB", "UOB", "VOB" ]; from PB2NCConfig_default
- test2: Selected all variables obs_bufr_var = [];

test1: 5 variables

hsoh@seneca:/d1/personal/hsoh/data/MET-D2581$ ls -l /nrit/ral/met-11.1.0/bin/pb2nc
-rwxr-xr-x 1 root root 13191064 Nov 27 19:37 /nrit/ral/met-11.1.0/bin/pb2nc

hsoh@seneca:/d1/personal/hsoh/data/MET-D2581$ touch empty_config
hsoh@seneca:/d1/personal/hsoh/data/MET-D2581$ time /nrit/ral/met-11.1.0/bin/pb2nc gdas.t00z.prepbufr.v1.2 tmp.nc empty_config
DEBUG 1: Start pb2nc by hsoh(9895) at 2024-05-14 16:13:31Z  cmd: /nrit/ral/met-11.1.0/bin/pb2nc gdas.t00z.prepbufr.v1.2 tmp.nc dummy_config
DEBUG 1: Default Config File: /nrit/ral/met-11.1.0/share/met/config/PB2NCConfig_default
DEBUG 1: User Config File: empty_config
DEBUG 1: Creating NetCDF File:          tmp.nc
DEBUG 1: Pre-processing BUFR File for variable names from gdas.t00z.prepbufr.v1.2
DEBUG 1: process_pbfile_metadata() -> the number of records: 944608
DEBUG 1: Processing Bufr File:  gdas.t00z.prepbufr.v1.2
DEBUG 2: Processing 944608 PrepBufr messages...
DEBUG 2: PrepBufr Time Center:          20240512_000000
DEBUG 2: Searching Time Window:         20240511_223000 to 20240512_013000
DEBUG 2: Messages processed                     = 944608
DEBUG 2: Rejected based on message type         = 0
DEBUG 2: Rejected based on station id           = 0
DEBUG 2: Rejected based on valid time           = 364309
DEBUG 2: Rejected based on masking grid         = 0
DEBUG 2: Rejected based on masking polygon      = 0
DEBUG 2: Rejected based on elevation            = 0
DEBUG 2: Rejected based on pb report type       = 0
DEBUG 2: Rejected based on input report type    = 0
DEBUG 2: Rejected based on instrument type      = 0
DEBUG 2: Rejected based on zero observations    = 63659
DEBUG 2: Messages retained                      = 516640
DEBUG 2: Observations retained or derived       = 2098005
DEBUG 2: Finished processing input observations between 20240511_210000 and 20240512_030000.
DEBUG 2: Total Observations retained or derived = 2098005
DEBUG 1: Finish pb2nc by hsoh(9895) at 2024-05-14 16:46:59Z

real    33m28.054s
user    33m25.737s
sys     0m2.212s

test2: all variables

hsoh@seneca:/d1/personal/hsoh/data/MET-D2581$ echo "obs_bufr_var = [];" > all_vars_config
hsoh@seneca:/d1/personal/hsoh/data/MET-D2581$ time /nrit/ral/met-11.1.0/bin/pb2nc gdas.t00z.prepbufr.v1.2 all_vars.nc all_vars_config
DEBUG 1: Start pb2nc by hsoh(9895) at 2024-05-14 17:13:17Z  cmd: /nrit/ral/met-11.1.0/bin/pb2nc gdas.t00z.prepbufr.v1.2 all_vars.nc all_vars_config
DEBUG 1: Default Config File: /nrit/ral/met-11.1.0/share/met/config/PB2NCConfig_default
DEBUG 1: User Config File: all_vars_config
DEBUG 1: Creating NetCDF File:          all_vars.nc
DEBUG 1: Pre-processing BUFR File for variable names from gdas.t00z.prepbufr.v1.2
DEBUG 1: process_pbfile_metadata() -> the number of records: 944608
DEBUG 1: Processing Bufr File:  gdas.t00z.prepbufr.v1.2
DEBUG 2: Processing 944608 PrepBufr messages...
DEBUG 2: PrepBufr Time Center:          20240512_000000
DEBUG 2: Searching Time Window:         20240511_223000 to 20240512_013000
DEBUG 2: Messages processed                     = 944608
DEBUG 2: Rejected based on message type         = 0
DEBUG 2: Rejected based on station id           = 0
DEBUG 2: Rejected based on valid time           = 364309
DEBUG 2: Rejected based on masking grid         = 0
DEBUG 2: Rejected based on masking polygon      = 0
DEBUG 2: Rejected based on elevation            = 0
DEBUG 2: Rejected based on pb report type       = 0
DEBUG 2: Rejected based on input report type    = 0
DEBUG 2: Rejected based on instrument type      = 0
DEBUG 2: Rejected based on zero observations    = 554
DEBUG 2: Messages retained                      = 579745
DEBUG 2: Observations retained or derived       = 13891541
DEBUG 2: Finished processing input observations between 20240511_210000 and 20240512_030000.
DEBUG 2: Total Observations retained or derived = 13889820
DEBUG 1: Finish pb2nc by hsoh(9895) at 2024-05-14 18:14:58Z

real    61m42.752s
user    61m30.832s
sys     0m11.696s

This is output with -dump <dump_dir> option:

-rw-r--r-- 1 hsoh rap 134617827 May 14 15:50 dump_gdas.t00z.prepbufr.v1.2_ADPSFC.txt
-rw-r--r-- 1 hsoh rap 103282403 May 14 15:50 dump_gdas.t00z.prepbufr.v1.2_ADPUPA.txt
-rw-r--r-- 1 hsoh rap 181985416 May 14 15:50 dump_gdas.t00z.prepbufr.v1.2_AIRCAR.txt
-rw-r--r-- 1 hsoh rap  32387946 May 14 15:50 dump_gdas.t00z.prepbufr.v1.2_AIRCFT.txt
-rw-r--r-- 1 hsoh rap         0 May 14 15:37 dump_gdas.t00z.prepbufr.v1.2_ERS1DA.txt
-rw-r--r-- 1 hsoh rap         0 May 14 15:37 dump_gdas.t00z.prepbufr.v1.2_GOESND.txt
-rw-r--r-- 1 hsoh rap    165970 May 14 15:50 dump_gdas.t00z.prepbufr.v1.2_GPSIPW.txt
-rw-r--r-- 1 hsoh rap         0 May 14 15:37 dump_gdas.t00z.prepbufr.v1.2_MSONET.txt
-rw-r--r-- 1 hsoh rap         0 May 14 15:37 dump_gdas.t00z.prepbufr.v1.2_PROFLR.txt
-rw-r--r-- 1 hsoh rap         0 May 14 15:37 dump_gdas.t00z.prepbufr.v1.2_QKSWND.txt
-rw-r--r-- 1 hsoh rap   2228356 May 14 15:50 dump_gdas.t00z.prepbufr.v1.2_RASSDA.txt
-rw-r--r-- 1 hsoh rap         0 May 14 15:37 dump_gdas.t00z.prepbufr.v1.2_SATEMP.txt
-rw-r--r-- 1 hsoh rap 125402586 May 14 15:50 dump_gdas.t00z.prepbufr.v1.2_SATWND.txt
-rw-r--r-- 1 hsoh rap         0 May 14 15:37 dump_gdas.t00z.prepbufr.v1.2_SFCBOG.txt
-rw-r--r-- 1 hsoh rap  31812447 May 14 15:50 dump_gdas.t00z.prepbufr.v1.2_SFCSHP.txt
-rw-r--r-- 1 hsoh rap         0 May 14 15:37 dump_gdas.t00z.prepbufr.v1.2_SPSSMI.txt
-rw-r--r-- 1 hsoh rap         0 May 14 15:37 dump_gdas.t00z.prepbufr.v1.2_SYNDAT.txt
-rw-r--r-- 1 hsoh rap 263702792 May 14 15:50 dump_gdas.t00z.prepbufr.v1.2_VADWND.txt

0 replies

BinbinZhou-NOAA · 2024-05-14T18:44:09Z

BinbinZhou-NOAA
May 14, 2024
Author

Howard, Thanks for PB2NC testing and glad to know that you can run the testing with our prepbufr data successfully. Running the PB2NC for all data type has no problem over Hawaii, but running the PB2NC to just process ADPUPA over Hawaii got "No data retained" issue. So could you please re-run testing testing for the following statins in the PB2Nc conf file? PB2NC_MESSAGE_TYPE = ADPUPA PB2NC_GRID = Hawaii_HREF.nc Binbin

…

On Tue, May 14, 2024 at 2:22 PM Howard Soh ***@***.***> wrote: It worked for me at seneca (our developing machine), It took more than 30 minutes to process 5 variables (2M obs) and more than 1 hour for all variables (14M obs). - Tested with MET_V11.1 (built on Nov 2023) - PB2NCConfig files: - test1: the empty configuration: five variables obs_bufr_var = [ "QOB", "TOB", "ZOB", "UOB", "VOB" ]; from PB2NCConfig_default - test2: Selected all variables obs_bufr_var = []; test1: 5 variables ***@***.***:/d1/personal/hsoh/data/MET-D2581$ ls -l /nrit/ral/met-11.1.0/bin/pb2nc -rwxr-xr-x 1 root root 13191064 Nov 27 19:37 /nrit/ral/met-11.1.0/bin/pb2nc ***@***.***:/d1/personal/hsoh/data/MET-D2581$ touch empty_config ***@***.***:/d1/personal/hsoh/data/MET-D2581$ time /nrit/ral/met-11.1.0/bin/pb2nc gdas.t00z.prepbufr.v1.2 tmp.nc empty_config DEBUG 1: Start pb2nc by hsoh(9895) at 2024-05-14 16:13:31Z cmd: /nrit/ral/met-11.1.0/bin/pb2nc gdas.t00z.prepbufr.v1.2 tmp.nc dummy_config DEBUG 1: Default Config File: /nrit/ral/met-11.1.0/share/met/config/PB2NCConfig_default DEBUG 1: User Config File: empty_config DEBUG 1: Creating NetCDF File: tmp.nc DEBUG 1: Pre-processing BUFR File for variable names from gdas.t00z.prepbufr.v1.2 DEBUG 1: process_pbfile_metadata() -> the number of records: 944608 DEBUG 1: Processing Bufr File: gdas.t00z.prepbufr.v1.2 DEBUG 2: Processing 944608 PrepBufr messages... DEBUG 2: PrepBufr Time Center: 20240512_000000 DEBUG 2: Searching Time Window: 20240511_223000 to 20240512_013000 DEBUG 2: Messages processed = 944608 DEBUG 2: Rejected based on message type = 0 DEBUG 2: Rejected based on station id = 0 DEBUG 2: Rejected based on valid time = 364309 DEBUG 2: Rejected based on masking grid = 0 DEBUG 2: Rejected based on masking polygon = 0 DEBUG 2: Rejected based on elevation = 0 DEBUG 2: Rejected based on pb report type = 0 DEBUG 2: Rejected based on input report type = 0 DEBUG 2: Rejected based on instrument type = 0 DEBUG 2: Rejected based on zero observations = 63659 DEBUG 2: Messages retained = 516640 DEBUG 2: Observations retained or derived = 2098005 DEBUG 2: Finished processing input observations between 20240511_210000 and 20240512_030000. DEBUG 2: Total Observations retained or derived = 2098005 DEBUG 1: Finish pb2nc by hsoh(9895) at 2024-05-14 16:46:59Z real 33m28.054s user 33m25.737s sys 0m2.212s test2: all variables ***@***.***:/d1/personal/hsoh/data/MET-D2581$ echo "obs_bufr_var = [];" > all_vars_config ***@***.***:/d1/personal/hsoh/data/MET-D2581$ time /nrit/ral/met-11.1.0/bin/pb2nc gdas.t00z.prepbufr.v1.2 all_vars.nc all_vars_config DEBUG 1: Start pb2nc by hsoh(9895) at 2024-05-14 17:13:17Z cmd: /nrit/ral/met-11.1.0/bin/pb2nc gdas.t00z.prepbufr.v1.2 all_vars.nc all_vars_config DEBUG 1: Default Config File: /nrit/ral/met-11.1.0/share/met/config/PB2NCConfig_default DEBUG 1: User Config File: all_vars_config DEBUG 1: Creating NetCDF File: all_vars.nc DEBUG 1: Pre-processing BUFR File for variable names from gdas.t00z.prepbufr.v1.2 DEBUG 1: process_pbfile_metadata() -> the number of records: 944608 DEBUG 1: Processing Bufr File: gdas.t00z.prepbufr.v1.2 DEBUG 2: Processing 944608 PrepBufr messages... DEBUG 2: PrepBufr Time Center: 20240512_000000 DEBUG 2: Searching Time Window: 20240511_223000 to 20240512_013000 DEBUG 2: Messages processed = 944608 DEBUG 2: Rejected based on message type = 0 DEBUG 2: Rejected based on station id = 0 DEBUG 2: Rejected based on valid time = 364309 DEBUG 2: Rejected based on masking grid = 0 DEBUG 2: Rejected based on masking polygon = 0 DEBUG 2: Rejected based on elevation = 0 DEBUG 2: Rejected based on pb report type = 0 DEBUG 2: Rejected based on input report type = 0 DEBUG 2: Rejected based on instrument type = 0 DEBUG 2: Rejected based on zero observations = 554 DEBUG 2: Messages retained = 579745 DEBUG 2: Observations retained or derived = 13891541 DEBUG 2: Finished processing input observations between 20240511_210000 and 20240512_030000. DEBUG 2: Total Observations retained or derived = 13889820 DEBUG 1: Finish pb2nc by hsoh(9895) at 2024-05-14 18:14:58Z real 61m42.752s user 61m30.832s sys 0m11.696s This is output with -dump <dump_dir> option: -rw-r--r-- 1 hsoh rap 134617827 May 14 15:50 dump_gdas.t00z.prepbufr.v1.2_ADPSFC.txt -rw-r--r-- 1 hsoh rap 103282403 May 14 15:50 dump_gdas.t00z.prepbufr.v1.2_ADPUPA.txt -rw-r--r-- 1 hsoh rap 181985416 May 14 15:50 dump_gdas.t00z.prepbufr.v1.2_AIRCAR.txt -rw-r--r-- 1 hsoh rap 32387946 May 14 15:50 dump_gdas.t00z.prepbufr.v1.2_AIRCFT.txt -rw-r--r-- 1 hsoh rap 0 May 14 15:37 dump_gdas.t00z.prepbufr.v1.2_ERS1DA.txt -rw-r--r-- 1 hsoh rap 0 May 14 15:37 dump_gdas.t00z.prepbufr.v1.2_GOESND.txt -rw-r--r-- 1 hsoh rap 165970 May 14 15:50 dump_gdas.t00z.prepbufr.v1.2_GPSIPW.txt -rw-r--r-- 1 hsoh rap 0 May 14 15:37 dump_gdas.t00z.prepbufr.v1.2_MSONET.txt -rw-r--r-- 1 hsoh rap 0 May 14 15:37 dump_gdas.t00z.prepbufr.v1.2_PROFLR.txt -rw-r--r-- 1 hsoh rap 0 May 14 15:37 dump_gdas.t00z.prepbufr.v1.2_QKSWND.txt -rw-r--r-- 1 hsoh rap 2228356 May 14 15:50 dump_gdas.t00z.prepbufr.v1.2_RASSDA.txt -rw-r--r-- 1 hsoh rap 0 May 14 15:37 dump_gdas.t00z.prepbufr.v1.2_SATEMP.txt -rw-r--r-- 1 hsoh rap 125402586 May 14 15:50 dump_gdas.t00z.prepbufr.v1.2_SATWND.txt -rw-r--r-- 1 hsoh rap 0 May 14 15:37 dump_gdas.t00z.prepbufr.v1.2_SFCBOG.txt -rw-r--r-- 1 hsoh rap 31812447 May 14 15:50 dump_gdas.t00z.prepbufr.v1.2_SFCSHP.txt -rw-r--r-- 1 hsoh rap 0 May 14 15:37 dump_gdas.t00z.prepbufr.v1.2_SPSSMI.txt -rw-r--r-- 1 hsoh rap 0 May 14 15:37 dump_gdas.t00z.prepbufr.v1.2_SYNDAT.txt -rw-r--r-- 1 hsoh rap 263702792 May 14 15:50 dump_gdas.t00z.prepbufr.v1.2_VADWND.txt — Reply to this email directly, view it on GitHub <#2581 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQBMPFKES2OMZLZU652RPIDZCJI6HAVCNFSM6AAAAABHU5A566VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TIMZXGI3TQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Binbin Zhou Physical Scientist Lynker at NOAA/NWS/NCEP/EMC 5830 University Research Ct. College Park, MD 20740 ***@***.*** 301-683-3683

1 reply

PerryShafran-NOAA May 14, 2024
Collaborator

@BinbinZhou-NOAA @hsoh-u Check to see that you are both using the same time window. Howard's time window looks larger than Binbin's, if I am recalling right.

BinbinZhou-NOAA · 2024-05-14T18:55:00Z

BinbinZhou-NOAA
May 14, 2024
Author

Perry, I think the longer processing on DTC side is due to the machine. Another reason may be that the testing run by Howard was over the entire global scale. PB2NC run for gdas over Hawaii on wcoss2 only takes 1 second! Binbin

…

On Tue, May 14, 2024 at 2:46 PM PerryShafran-NOAA ***@***.***> wrote: @BinbinZhou-NOAA <https://github.com/BinbinZhou-NOAA> @hsoh-u <https://github.com/hsoh-u> Check to see that you are both using the same time window. Howard's time window looks larger than Binbin's, if I am recalling right. — Reply to this email directly, view it on GitHub <#2581 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQBMPFPWOBRBPDTEQT3XOS3ZCJLYZAVCNFSM6AAAAABHU5A566VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TIMZXGQZTE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Binbin Zhou Physical Scientist Lynker at NOAA/NWS/NCEP/EMC 5830 University Research Ct. College Park, MD 20740 ***@***.*** 301-683-3683

0 replies

hsoh-u · 2024-05-14T20:38:17Z

hsoh-u
May 14, 2024
Collaborator

No PREPBUFR message for Hawaii and two PREPBUFR messages for Puerto Rico.

Filtering condition:

message_type = "ADPUPA"
masked by grid
- Hawaii: grid = "latlon 170 223 16.4 -162.35 0.045 0.045"; from Hawaii_HREF.nc
- Puerto Rico: grid = "latlon 208 340 13.50 -76.59 0.045 0.045"; from PRico_HREF.nc

Hawaii

DEBUG 2: Messages processed                     = 944608
DEBUG 2: Rejected based on message type         = 943313
DEBUG 2: Rejected based on valid time           = 2
DEBUG 2: Rejected based on masking grid         = 1293
DEBUG 2: Messages retained                      = 0
DEBUG 2: Observations retained or derived       = 0
DEBUG 2: Finished processing input observations between 20240511_220000 and 20240512_010000.

Puerto Rico:

DEBUG 2: Messages processed                     = 944608
DEBUG 2: Rejected based on message type         = 943313
DEBUG 2: Rejected based on valid time           = 2
DEBUG 2: Rejected based on masking grid         = 1291
DEBUG 2: Messages retained                      = 2
DEBUG 2: Observations retained or derived       = 1385
DEBUG 2: Finished processing input observations between 20240511_220000 and 20240512_010000.

0 replies

hsoh-u · 2024-05-14T21:05:30Z

hsoh-u
May 14, 2024
Collaborator

The PREPBUFR messages of message_type "AIRCAR", "VADWND", "ADPSFC", and "SFCSHP" are available for Hawaii.

DEBUG 2: Rejected based on message type         = 0
DEBUG 2: Messages retained                      = 1241
DEBUG 2: Observations retained or derived       = 42554

0 replies

BinbinZhou-NOAA · 2024-05-14T23:47:35Z

BinbinZhou-NOAA
May 14, 2024
Author

Howard, The "Messages retained = 0" for ADPUPA over Hawaii also was what we got in our run. For this message, can you identify which variable(s) are missing or all of the variables in ADPUPA are missing? Binbin

…

On Tue, May 14, 2024 at 4:38 PM Howard Soh ***@***.***> wrote: No PREPBUFR message for Hawaii and two PREPBUFR messages for Puerto Rico. Filtering condition: - message_type = "ADPUPA" - masked by grid - Hawaii: grid = "latlon 170 223 16.4 -162.35 0.045 0.045"; from Hawaii_HREF.nc - Puerto Rico: grid = "latlon 208 340 13.50 -76.59 0.045 0.045"; from PRico_HREF.nc Hawaii DEBUG 2: Messages processed = 944608 DEBUG 2: Rejected based on message type = 943313 DEBUG 2: Rejected based on valid time = 2 DEBUG 2: Rejected based on masking grid = 1293 DEBUG 2: Messages retained = 0 DEBUG 2: Observations retained or derived = 0 DEBUG 2: Finished processing input observations between 20240511_220000 and 20240512_010000. Puerto Rico: DEBUG 2: Messages processed = 944608 DEBUG 2: Rejected based on message type = 943313 DEBUG 2: Rejected based on valid time = 2 DEBUG 2: Rejected based on masking grid = 1291 DEBUG 2: Messages retained = 2 DEBUG 2: Observations retained or derived = 1385 DEBUG 2: Finished processing input observations between 20240511_220000 and 20240512_010000. — Reply to this email directly, view it on GitHub <#2581 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQBMPFLLYEEMDYA3MKFG7DDZCJY45AVCNFSM6AAAAABHU5A566VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TIMZYGM2TO> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Binbin Zhou Physical Scientist Lynker at NOAA/NWS/NCEP/EMC 5830 University Research Ct. College Park, MD 20740 ***@***.*** 301-683-3683

0 replies

hsoh-u · 2024-05-15T03:44:47Z

hsoh-u
May 15, 2024
Collaborator

0 retained message implies no variables (missing all variables). PB2NC did not have the chance to look at the variables because all of them were filtered out by looking at the header messages. PB2NC reads the header message (obs time, lat/lon, message type, station_id, ...), applies the filtering, and reads the variable messages if not filtered out.

0 replies

BinbinZhou-NOAA · 2024-05-15T12:30:39Z

BinbinZhou-NOAA
May 15, 2024
Author

Howard, This explanation is helpful for us to understand the meaning of "0 retained data" Thanks! Binbin

…

On Tue, May 14, 2024 at 11:45 PM Howard Soh ***@***.***> wrote: 0 retained message implies no variables. PB2NC did not have the chance to look at the variables because of filtered out by looking at the header message. PB2NC reads the header message (obs time, lat/lon, message type, station_id, ...), applies the filtering, and reads the variables messages if not filtered out. — Reply to this email directly, view it on GitHub <#2581 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQBMPFK5M25L5ZHT5M35CTLZCLK4NAVCNFSM6AAAAABHU5A566VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TINBQGU3DM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Binbin Zhou Physical Scientist Lynker at NOAA/NWS/NCEP/EMC 5830 University Research Ct. College Park, MD 20740 ***@***.*** 301-683-3683

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WARNING message for PB2NC to process GDAS prepbufr files over Hawaii #2581

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 15 comments 5 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

WARNING message for PB2NC to process GDAS prepbufr files over Hawaii #2581

BinbinZhou-NOAA May 13, 2024

Replies: 15 comments · 5 replies

PerryShafran-NOAA May 13, 2024 Collaborator

hsoh-u May 13, 2024 Collaborator

PerryShafran-NOAA May 13, 2024 Collaborator

jprestop May 13, 2024 Maintainer

BinbinZhou-NOAA May 14, 2024 Author

PerryShafran-NOAA May 14, 2024 Collaborator

hsoh-u May 14, 2024 Collaborator

BinbinZhou-NOAA May 14, 2024 Author

PerryShafran-NOAA May 14, 2024 Collaborator

BinbinZhou-NOAA May 14, 2024 Author

PerryShafran-NOAA May 14, 2024 Collaborator

hsoh-u May 14, 2024 Collaborator

BinbinZhou-NOAA May 14, 2024 Author

PerryShafran-NOAA May 14, 2024 Collaborator

BinbinZhou-NOAA May 14, 2024 Author

hsoh-u May 14, 2024 Collaborator

hsoh-u May 14, 2024 Collaborator

BinbinZhou-NOAA May 14, 2024 Author

hsoh-u May 15, 2024 Collaborator

BinbinZhou-NOAA May 15, 2024 Author

BinbinZhou-NOAA
May 13, 2024

Replies: 15 comments 5 replies

PerryShafran-NOAA
May 13, 2024
Collaborator

hsoh-u
May 13, 2024
Collaborator

PerryShafran-NOAA
May 13, 2024
Collaborator

jprestop May 13, 2024
Maintainer

BinbinZhou-NOAA
May 14, 2024
Author

PerryShafran-NOAA
May 14, 2024
Collaborator

hsoh-u May 14, 2024
Collaborator

BinbinZhou-NOAA
May 14, 2024
Author

PerryShafran-NOAA May 14, 2024
Collaborator

BinbinZhou-NOAA
May 14, 2024
Author

PerryShafran-NOAA May 14, 2024
Collaborator

hsoh-u
May 14, 2024
Collaborator

BinbinZhou-NOAA
May 14, 2024
Author

PerryShafran-NOAA May 14, 2024
Collaborator

BinbinZhou-NOAA
May 14, 2024
Author

hsoh-u
May 14, 2024
Collaborator

hsoh-u
May 14, 2024
Collaborator

BinbinZhou-NOAA
May 14, 2024
Author

hsoh-u
May 15, 2024
Collaborator

BinbinZhou-NOAA
May 15, 2024
Author