add bufr_query to GDASApp #1225

CoryMartin-NOAA · 2024-07-17T20:27:17Z

Add in the bufr_query repository to GDASApp to support direct ingest of BUFR files into JEDI-based applications.

CoryMartin-NOAA · 2024-07-17T20:37:26Z

Tested by running: ./bufr2netcdf.x /scratch1/NCEPDEV/global/glopara/dump/gdas.20240717/00/atmos/gdas.t00z.esmhs.tm00.bufr_d ../../sorc/bufr-query/test/testinput/bufrtest_mhs_mapping.yaml mhs.{splits/satId}.nc

outputFile: mhs.{splits/satId}.nc
PRE-MAIN-INFO BufrParser: Parsing file /scratch1/NCEPDEV/global/glopara/dump/gdas.20240717/00/atmos/gdas.t00z.esmhs.tm00.bufr_d
PRE-MAIN-INFO Executing Queries
PRE-MAIN-WARNING Warning: Query String */CLAT did not apply to subset NC021036
PRE-MAIN-WARNING Warning: Query String */HOLS did not apply to subset NC021036
PRE-MAIN-WARNING Warning: Query String */CLON did not apply to subset NC021036
PRE-MAIN-WARNING Warning: Query String */BRITCSTC/TMBR did not apply to subset NC021036
PRE-MAIN-WARNING Warning: Query String */HMSL did not apply to subset NC021036
PRE-MAIN-WARNING Warning: Query String */LSQL did not apply to subset NC021036
PRE-MAIN-WARNING Warning: Query String */SIID did not apply to subset NC021036
PRE-MAIN-WARNING Warning: Query String */BRITCSTC/CHNM did not apply to subset NC021036
PRE-MAIN-INFO Building Bufr Data
PRE-MAIN-INFO Exporting Data
PRE-MAIN-INFO Finished [78.778s]
PRE-MAIN-WARNING Category (metop-a) was not found in file.
PRE-MAIN-WARNING Category (noaa-18) was not found in file.

Produced:

-rw-r--r-- 1 Cory.R.Martin da 8.8M Jul 17 20:23 mhs.noaa-19.nc
-rw-r--r-- 1 Cory.R.Martin da  17K Jul 17 20:23 mhs.noaa-18.nc
-rw-r--r-- 1 Cory.R.Martin da 6.0M Jul 17 20:23 mhs.metop-c.nc
-rw-r--r-- 1 Cory.R.Martin da 6.2M Jul 17 20:23 mhs.metop-b.nc
-rw-r--r-- 1 Cory.R.Martin da  17K Jul 17 20:23 mhs.metop-a.nc

ncdump -h mhs.metop-c.nc 
netcdf mhs.metop-c {
dimensions:
	Location = 820798 ;
variables:
	int Location(Location) ;
		Location:_FillValue = 2147483647 ;

// global attributes:
		:platformCommonName = "MHS" ;
		:platformLongDescription = "MTYP 021-027 PROCESSED MHS Tb (NOAA-18-19, METOP-1,2,3)" ;

group: MetaData {
  variables:
  	int64 dateTime(Location) ;
  		dateTime:_FillValue = 9223372036854775807LL ;
  		dateTime:long_name = "Datetime" ;
  		dateTime:units = "seconds since 1970-01-01T00:00:00Z" ;
  	int latitude(Location) ;
  		latitude:_FillValue = 2147483647 ;
  		latitude:long_name = "Latitude" ;
  		latitude:units = "degree_north" ;
  		latitude:valid_range = -90.f, 90.f ;
...

ADCollard

Nice

.gitmodules

emilyhcliu · 2024-07-17T20:53:49Z

@CoryMartin-NOAA FYI. We are going to cut a tag from bufr-query for JCSDA to include bufr-query in the spack-stack-1.8.0 release which is targeted for end of Aug/early Sep.

Steve wants us to make sure with NCO that the bufr-query is acceptable in WCOSS-2. The bufr-query depends on ecbuild, NetCDF, and BUFR libraries. I do not see any reason that NCO would prohibit the installation of bufr-query on WCOSS-2.
Is there a procedure to follow for adding new software to the spack-stack for WCOSS-2?

CoryMartin-NOAA · 2024-07-17T20:56:57Z

@emilyhcliu we don't have spack-stack on WCOSS2 (yet, but maybe eventually?). To add a new library, we will have to work with Jacob and Hang Lei, there is a process, which should be easier since this is an internal library.

emilyhcliu · 2024-07-17T21:11:18Z

@emilyhcliu we don't have spack-stack on WCOSS2 (yet, but maybe eventually?). To add a new library, we will have to work with Jacob and Hang Lei, there is a process, which should be easier since this is an internal library.

Do you think the following procedure looks right to you?

Create a tag in bufr-query repository
JCSDA picks up the tag from GitHub, includes it in the spack-stack-1.8.0 and publishes it on JCSDA spack-stack repository
NOAA picks up the spack-stack-1.8.0 and installs it on HPC machines.

We need to work with Jacob and Hang Lei for item # 3 listed above, right?

CoryMartin-NOAA · 2024-07-17T21:13:30Z

For non-wcoss, I'm not even sure Jacob/Hang need to be involved. I think each HPC has someone who just installs the spack-stack on them, there is a team (Dom, Alex Richert, etc.). Jacob/Hang are only needed for WCOSS/NCO interaction

RussTreadon-NOAA

Looks good to me. Only one question about .gitmodules.

RussTreadon-NOAA · 2024-07-22T19:23:06Z

09a2fb9 also updates the jcb hash to 8269ce4. PR #1230 and references therein explain why this is necessary.

CoryMartin-NOAA · 2024-07-22T19:39:40Z

I cannot approve since I was the author of the PR. @RussTreadon-NOAA also has contributed. So we probably need a 3rd party to approve the PR before this can be merged.

CoryMartin-NOAA · 2024-07-22T21:06:22Z

@rmclaren will need to approve since he previously requested changes

RussTreadon-NOAA

Approve

RussTreadon-NOAA · 2024-07-23T11:27:39Z

I don't think the Orion cron for orion-GW-RT is active any more. PR #1226 has had an active orion-GW-RT label for 5 days.

emcbot · 2024-07-23T12:35:02Z

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Tue Jul 23 11:28:54 UTC 2024 on hfe09
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Tue Jul 23 12:14:14 UTC 2024
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Tue Jul 23 12:35:01 UTC 2024
Tests: 88% tests passed, 5 tests failed out of 43
	1887 - test_gdasapp_soca_JGLOBAL_MARINE_BMAT (Failed)
	1888 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP (Failed)
	1889 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN (Failed)
	1891 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT (Failed)
	1892 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST (Failed)
Tests: see output at /scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/workflow/PR/1225/global-workflow/sorc/gdas.cd/build/log.ctest

RussTreadon-NOAA · 2024-07-23T13:04:16Z

The test_gdasapp_soca_JGLOBAL_MARINE_BMAT is due to the fact GDASApp is ahead of g-w. test/soca/gw/CMakeLists.txt is configured to execute j-job "JGLOBAL_MARINE_BMAT". This j-job is not yet in g-w develop. g-w PR #2749 adds jobs/JGLOBAL_MARINE_BMAT. test_gdasapp_soca_JGLOBAL_MARINE_BMAT immediately fails upon start with the error

/var/spool/slurmd/job63799201/slurm_script: line 52: /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/bufr_query/so\
rc/gdas.cd/bundle/gdas/../../jobs/JGLOBAL_MARINE_BMAT: No such file or directory
+ slurm_script[1]: postamble slurm_script 1721734005 127
+ preamble.sh[70]: set +x
End slurm_script at 11:26:47 with error code 127 (time elapsed: 00:00:02)

The failures for the JGDAS_GLOBAL_OCEAN jobs may also be related to this disconnect.

What do you think @guillaumevernieres?

guillaumevernieres · 2024-07-23T13:20:29Z

The test_gdasapp_soca_JGLOBAL_MARINE_BMAT is due to the fact GDASApp is ahead of g-w. test/soca/gw/CMakeLists.txt is configured to execute j-job "JGLOBAL_MARINE_BMAT". This j-job is not yet in g-w develop. g-w PR #2749 adds jobs/JGLOBAL_MARINE_BMAT. test_gdasapp_soca_JGLOBAL_MARINE_BMAT immediately fails upon start with the error
/var/spool/slurmd/job63799201/slurm_script: line 52: /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/bufr_query/so\
rc/gdas.cd/bundle/gdas/../../jobs/JGLOBAL_MARINE_BMAT: No such file or directory
+ slurm_script[1]: postamble slurm_script 1721734005 127
+ preamble.sh[70]: set +x
End slurm_script at 11:26:47 with error code 127 (time elapsed: 00:00:02)
The failures for the JGDAS_GLOBAL_OCEAN jobs may also be related to this disconnect.

What do you think @guillaumevernieres?

@RussTreadon-NOAA , yes, ignore the marine tests for now.

RussTreadon-NOAA · 2024-07-23T16:01:19Z

Run g-w C96C48_ufs_hybatmDA CI on Hera using g-w develop at 65a7ab7 and feature/bufr-query at 929e8e6. All jobs successfully run to completion

Hera(hfe07):/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/prjedi_bq$ rocotostat -d prjedi_bq.db -w prjedi_bq.xml -c all -s
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202402231800        Done    Jul 23 2024 13:27:32    Jul 23 2024 13:51:01
202402240000        Done    Jul 23 2024 13:27:32    Jul 23 2024 15:55:15

RussTreadon-NOAA · 2024-07-23T16:02:26Z

@CoryMartin-NOAA and @guillaumevernieres , this branch is ready for merger into GDASApp develop. Is merging OK or will this PR mess up other GDASApp or g-w PRs?

CoryMartin-NOAA · 2024-07-23T16:03:47Z

@RussTreadon-NOAA as far as I know this shouldn't impact anything else, I'm good with merging if @guillaumevernieres is

RussTreadon-NOAA · 2024-07-23T16:47:02Z

Thank you, @guillaumevernieres . That's a cool way to answer a question ... just go ahead and merge!

add bufr_query to GDASApp

e7f242e

CoryMartin-NOAA requested review from rmclaren, RussTreadon-NOAA, emilyhcliu and ADCollard July 17, 2024 20:27

CoryMartin-NOAA added 2 commits July 17, 2024 20:28

specify develop for bufr-query

0399ac3

update to use 'master' and update hash

ab7c205

CoryMartin-NOAA self-assigned this Jul 17, 2024

ADCollard previously approved these changes Jul 17, 2024

View reviewed changes

tab not space?

85893cb

CoryMartin-NOAA dismissed ADCollard’s stale review via 85893cb July 17, 2024 20:43

spaces not tabs actually

43b12e4

rmclaren approved these changes Jul 17, 2024

View reviewed changes

rmclaren requested changes Jul 17, 2024

View reviewed changes

.gitmodules Outdated Show resolved Hide resolved

emilyhcliu previously approved these changes Jul 17, 2024

View reviewed changes

RussTreadon-NOAA reviewed Jul 22, 2024

View reviewed changes

update bufr-query to develop, update jcb hash

09a2fb9

RussTreadon-NOAA dismissed emilyhcliu’s stale review via 09a2fb9 July 22, 2024 19:19

ADCollard approved these changes Jul 22, 2024

View reviewed changes

Merge branch 'develop' into feature/bufr-query

929e8e6

rmclaren self-requested a review July 22, 2024 22:13

rmclaren approved these changes Jul 22, 2024

View reviewed changes

RussTreadon-NOAA added hera-GW-RT Queue for automated testing with global-workflow on Hera orion-GW-RT Queue for automated testing with global-workflow on Orion labels Jul 23, 2024

emcbot added hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed hera-GW-RT Queue for automated testing with global-workflow on Hera labels Jul 23, 2024

RussTreadon-NOAA self-requested a review July 23, 2024 11:22

RussTreadon-NOAA approved these changes Jul 23, 2024

View reviewed changes

emcbot added hera-GW-RT-Failed Automated testing with global-workflow failed on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Jul 23, 2024

guillaumevernieres merged commit 801d64c into develop Jul 23, 2024
13 checks passed

guillaumevernieres deleted the feature/bufr-query branch July 23, 2024 16:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add bufr_query to GDASApp #1225

add bufr_query to GDASApp #1225

CoryMartin-NOAA commented Jul 17, 2024

CoryMartin-NOAA commented Jul 17, 2024

ADCollard left a comment

emilyhcliu commented Jul 17, 2024 •

edited

Loading

CoryMartin-NOAA commented Jul 17, 2024

emilyhcliu commented Jul 17, 2024

CoryMartin-NOAA commented Jul 17, 2024

RussTreadon-NOAA left a comment

RussTreadon-NOAA commented Jul 22, 2024

CoryMartin-NOAA commented Jul 22, 2024

CoryMartin-NOAA commented Jul 22, 2024

RussTreadon-NOAA left a comment

RussTreadon-NOAA commented Jul 23, 2024

emcbot commented Jul 23, 2024

RussTreadon-NOAA commented Jul 23, 2024

guillaumevernieres commented Jul 23, 2024

RussTreadon-NOAA commented Jul 23, 2024

RussTreadon-NOAA commented Jul 23, 2024

CoryMartin-NOAA commented Jul 23, 2024

RussTreadon-NOAA commented Jul 23, 2024

add bufr_query to GDASApp #1225

add bufr_query to GDASApp #1225

Conversation

CoryMartin-NOAA commented Jul 17, 2024

CoryMartin-NOAA commented Jul 17, 2024

ADCollard left a comment

Choose a reason for hiding this comment

emilyhcliu commented Jul 17, 2024 • edited Loading

CoryMartin-NOAA commented Jul 17, 2024

emilyhcliu commented Jul 17, 2024

CoryMartin-NOAA commented Jul 17, 2024

RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

RussTreadon-NOAA commented Jul 22, 2024

CoryMartin-NOAA commented Jul 22, 2024

CoryMartin-NOAA commented Jul 22, 2024

RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

RussTreadon-NOAA commented Jul 23, 2024

emcbot commented Jul 23, 2024

RussTreadon-NOAA commented Jul 23, 2024

guillaumevernieres commented Jul 23, 2024

RussTreadon-NOAA commented Jul 23, 2024

RussTreadon-NOAA commented Jul 23, 2024

CoryMartin-NOAA commented Jul 23, 2024

RussTreadon-NOAA commented Jul 23, 2024

emilyhcliu commented Jul 17, 2024 •

edited

Loading