merge 26Mar20 develop branch into gmtb_ccpp_hera

Squashed commit of the following: commit 0e0dc87 Merge: b133700 95a6343 Author: Kate Friedman <[email protected]> Date: Thu Mar 26 10:54:44 2020 -0400 Merge pull request NOAA-EMC#42 from NOAA-EMC/hotfix/viewer Issue NOAA-EMC#41 - Update PRODUTIL paths for WCOSS in viewer commit 95a6343 Author: kate.friedman <[email protected]> Date: Thu Mar 26 14:43:44 2020 +0000 Issue NOAA-EMC#41 - Update PRODUTIL paths for WCOSS in viewer commit b133700 Author: kate.friedman <[email protected]> Date: Thu Mar 26 13:01:38 2020 +0000 Issue NOAA-EMC#21 - correct syntax for machine if-blocks in setup_expt scripts commit f4ee509 Author: kate.friedman <[email protected]> Date: Wed Mar 25 20:52:49 2020 +0000 Issue NOAA-EMC#21 - remove unneeded logic from partition check commit 4acc80f Merge: accb6f4 fb1c79f Author: Hang-Lei-NOAA <[email protected]> Date: Wed Mar 25 15:22:51 2020 -0400 Merge pull request NOAA-EMC#40 from NOAA-EMC/port2wcoss3p5 Issue NOAA-EMC#21 - Add support for WCOSS phase 3.5 commit fb1c79f Author: kate.friedman <[email protected]> Date: Wed Mar 25 16:06:54 2020 +0000 Issue NOAA-EMC#21 - updated WCOSS phase 3.5 queues commit 08a5354 Author: kate.friedman <[email protected]> Date: Wed Mar 25 14:58:14 2020 +0000 Issue NOAA-EMC#21 - add phase 3.5 support - add partition option to setup scripts - remove machine if-blocks from config.base and add variable population to setup_expt*py scripts - add phase 3.5 ppn value to WCOSS_DELL_P3 env and config.resources files commit accb6f4 Merge: 057b2a8 8b51b56 Author: Kate Friedman <[email protected]> Date: Mon Mar 23 14:01:30 2020 -0400 Merge pull request NOAA-EMC#39 from NOAA-EMC/feature/verif-tag Issue NOAA-EMC#38 - update EMC_verif-global pointer from VLab to GitHub commit 8b51b56 Author: kate.friedman <[email protected]> Date: Mon Mar 23 17:40:18 2020 +0000 Issue NOAA-EMC#38 - update EMC_verif-global pointer from VLab to GitHub commit 057b2a8 Merge: 0377d20 622167d Author: Kate Friedman <[email protected]> Date: Wed Mar 11 12:00:58 2020 -0400 Merge pull request NOAA-EMC#29 from NOAA-EMC/feature/manage_externals Issue #3 - Introduce manage_externals as replacement for checkout.sh commit 622167d Author: kate.friedman <[email protected]> Date: Fri Mar 6 18:20:31 2020 +0000 Issue #3 - added explicit config flag example for checkout_externals in README and blurb about this replacing checkout.sh commit e83b90d Author: kate.friedman <[email protected]> Date: Fri Mar 6 17:00:15 2020 +0000 Issue #3 - remove prod_util and grib_util sections from build_all.sh, removed elsewhere already commit 8699b46 Author: kate.friedman <[email protected]> Date: Fri Mar 6 16:30:54 2020 +0000 Issue #3 - updated README with new manic version commit e602cd3 Author: kate.friedman <[email protected]> Date: Fri Mar 6 16:27:57 2020 +0000 Issue #3 - updated link_fv3gfs.sh to adjust wafs links commit 830c73f Author: kate.friedman <[email protected]> Date: Fri Mar 6 15:21:45 2020 +0000 Issue #3 - update EMC_verif-global tag in Externals.cfg after sync with develop commit 40084e6 Merge: f662fff 0377d20 Author: kate.friedman <[email protected]> Date: Fri Mar 6 15:18:52 2020 +0000 Issue #3 - Merge branch 'develop' into feature/manage_externals commit 0377d20 Merge: 1b359db 2552467 Author: Kate Friedman <[email protected]> Date: Thu Mar 5 08:43:16 2020 -0500 Merge pull request NOAA-EMC#28 from NOAA-EMC/feature/metplus2 Issue #8 - add switch for MET+ jobs commit 2552467 Author: kate.friedman <[email protected]> Date: Thu Mar 5 13:31:02 2020 +0000 Issue #8 - add switch for MET+ jobs commit 1b359db Merge: 0359d34 31bb7d3 Author: Kate Friedman <[email protected]> Date: Wed Mar 4 10:19:36 2020 -0500 Merge pull request NOAA-EMC#26 from NOAA-EMC/feature/metplus Feature/metplus - refactored MET+ jobs to resolve timing issues commit 31bb7d3 Merge: eb73e52 0359d34 Author: Mallory Row <[email protected]> Date: Wed Feb 19 15:24:42 2020 +0000 Merge branch 'develop' into feature/metplus commit f662fff Author: kate.friedman <[email protected]> Date: Fri Feb 14 15:57:05 2020 +0000 Issue #3 - Updated README with new manic tag v1.1.7 commit e3196a8 Author: Kate Friedman <[email protected]> Date: Thu Feb 13 15:59:06 2020 -0500 Update README.md commit e46b175 Merge: 4bd0e20 0359d34 Author: Kate.Friedman <[email protected]> Date: Thu Feb 13 20:38:04 2020 +0000 Issue #3 - sync merge with develop branch commit 0359d34 Merge: 1d9a1f0 bd00cb9 Author: Hang-Lei-NOAA <[email protected]> Date: Thu Feb 13 09:53:59 2020 -0500 Merge pull request NOAA-EMC#19 from NOAA-EMC/feature/remove_theia Feature/remove theia commit eb73e52 Author: Mallory Row <[email protected]> Date: Fri Feb 7 14:04:37 2020 +0000 Update EMC_verif-global checkout to verif_global_v1.5.0 commit bd00cb9 Author: Kate.Friedman <[email protected]> Date: Fri Feb 7 13:41:05 2020 +0000 Issue #4 - bug fix in getic.sh for v15 commit 1c85197 Merge: 67dae40 1d9a1f0 Author: Kate.Friedman <[email protected]> Date: Fri Feb 7 13:27:18 2020 +0000 Issue # 4 - Sync merge branch 'develop' into feature/remove_theia commit 1d9a1f0 Merge: 3ed9267 bdbecaa Author: Kate Friedman <[email protected]> Date: Fri Feb 7 08:11:21 2020 -0500 Merge pull request NOAA-EMC#18 from lgannoaa/exception_handling Exception handling commit 4bd0e20 Author: Kate Friedman <[email protected]> Date: Thu Feb 6 11:55:31 2020 -0500 Update README.md commit d9ea1ac Author: Kate.Friedman <[email protected]> Date: Thu Feb 6 16:03:11 2020 +0000 Issue #3 - reduce hashes down to minimum 8 characters commit bdbecaa Author: Lin.Gan <[email protected]> Date: Wed Jan 29 15:15:52 2020 +0000 Display exception handling message for individual package with location of the log file commit b64fd5f Merge: cf00863 3ed9267 Author: kate.friedman <[email protected]> Date: Tue Jan 28 15:24:37 2020 +0000 Merge branch 'develop' into feature/manage_externals commit cf00863 Author: kate.friedman <[email protected]> Date: Tue Jan 28 15:23:14 2020 +0000 Issue #3 - remove copy of manage_externals under util and add README.md file commit c12e879 Author: Lin.Gan <[email protected]> Date: Mon Jan 27 19:38:19 2020 +0000 Implement exception handling in build_all script commit bfc7bb0 Author: Kate.Friedman <[email protected]> Date: Mon Jan 27 19:06:17 2020 +0000 Issue #3 - initial add of manage_externals and needed Externals.cfg. Also added .gitignore file and removed scripts/files associated with no-longer-used prod_util and grid_util builds. commit 4d5713d Author: Lin.Gan <[email protected]> Date: Fri Jan 24 15:51:28 2020 +0000 Testing github commit commit 786806f Author: Mallory Row <[email protected]> Date: Fri Jan 24 15:04:16 2020 +0000 Missed file format updates in a few places in config.metp commit d0a3b53 Author: Lin.Gan <[email protected]> Date: Fri Jan 24 14:30:29 2020 +0000 init commit for exception handling branch commit c11dfef Author: Mallory Row <[email protected]> Date: Fri Jan 24 14:07:29 2020 +0000 Update EMC_verif-global tag to verif_global_v1.4.1 commit 0ea809c Author: Mallory Row <[email protected]> Date: Thu Jan 23 13:29:56 2020 +0000 Update file format variable in config.metp of online archive files commit c0d7179 Author: Mallory Row <[email protected]> Date: Tue Jan 21 16:37:16 2020 +0000 Add updated env machine files for gfsmetp commit 82e6907 Author: Mallory Row <[email protected]> Date: Tue Jan 21 16:29:23 2020 +0000 Update config.resources for gfsmetp commit 72e8adf Author: Mallory Row <[email protected]> Date: Tue Jan 21 16:19:31 2020 +0000 Update EMC_verif-global tag checkout to 1.4.0 commit 6872f79 Author: Mallory Row <[email protected]> Date: Tue Jan 21 16:14:25 2020 +0000 Add rocoto METplus job script commit 9c94156 Author: Mallory Row <[email protected]> Date: Tue Jan 21 16:09:13 2020 +0000 Changes to setup_workflow.py for gfsmetp metatask commit 3ed9267 Author: kate.friedman <[email protected]> Date: Tue Jan 14 19:32:36 2020 +0000 Issue NOAA-EMC#10 - mid-year update to bufr station list (develop) commit 1915aa9 Author: kate.friedman <[email protected]> Date: Fri Jan 10 18:36:32 2020 +0000 Issue #8 - pulled in config.metp and modifications to two setup scripts commit 67dae40 Merge: f78eb1b 091f4ba Author: kate.friedman <[email protected]> Date: Wed Jan 8 20:15:45 2020 +0000 Merge branch 'develop' into feature/remove_theia commit f78eb1b Author: kate.friedman <[email protected]> Date: Wed Jan 8 20:13:08 2020 +0000 Issue #4 - removed references to Theia and Theia scripts commit 091f4ba Author: kate.friedman <[email protected]> Date: Wed Jan 8 19:37:45 2020 +0000 Issue #7 - missed update to gdas transfer file from GFSv15.2.5 updates commit 3fd4bcf Author: kate.friedman <[email protected]> Date: Tue Dec 17 19:12:28 2019 +0000 GitHub Issue #2 - GFSv15.2.6 obsproc version update, earc bug fix, and tracker path update commit 5307952 Author: Kate.Friedman <[email protected]> Date: Wed Dec 11 20:47:44 2019 +0000 HOTFIX - VLab Issue #72346 - fix to rocoto_viewer on Hera
XianwuXue-NOAA · Mar 31, 2020 · a847acc · a847acc
1 parent 6397f5e
commit a847acc
Show file tree

Hide file tree

Showing 113 changed files with 708 additions and 2,065 deletions.
diff --git a/.gitignore b/.gitignore
@@ -0,0 +1,16 @@
+# Ignore all compiled files
+*.pyc
+*.o
+*.mod
+
+# Ignore exec folder
+exec/
+
+# Ignore sorc folders from externals
+sorc/logs/
+sorc/fv3gfs.fd/
+sorc/gfs_post.fd/
+sorc/gsi.fd/
+sorc/ufs_utils.fd/
+sorc/gfs_wafs.fd/
+sorc/verif-global.fd/
diff --git a/Externals.cfg b/Externals.cfg
@@ -0,0 +1,46 @@
+# External sub-modules of global-workflow
+
+[NEMSfv3gfs]
+tag = gfs.v16_PhysicsUpdate
+local_path = sorc/fv3gfs.fd
+repo_url = ssh://vlab.ncep.noaa.gov:29418/NEMSfv3gfs
+protocol = git
+required = True
+
+[GSI]
+hash = cb8f69d8
+local_path = sorc/gsi.fd
+repo_url = ssh://vlab.ncep.noaa.gov:29418/ProdGSI
+protocol = git
+required = True
+
+[EMC_post]
+hash = ba7e59b2
+local_path = sorc/gfs_post.fd
+repo_url = https://github.com/NOAA-EMC/EMC_post.git
+protocol = git
+required = True
+
+[UFS_UTILS]
+tag = v1.1.0
+local_path = sorc/ufs_utils.fd
+repo_url = https://github.com/NOAA-EMC/UFS_UTILS.git
+protocol = git
+required = True
+
+[EMC_verif-global]
+tag = verif_global_v1.5.0
+local_path = sorc/verif-global.fd
+repo_url = https://github.com/NOAA-EMC/EMC_verif-global.git
+protocol = git
+required = True
+
+[EMC_gfs_wafs]
+tag = gfs_wafs.v5.0.11
+local_path = sorc/gfs_wafs.fd
+repo_url = https://github.com/NOAA-EMC/EMC_gfs_wafs.git
+protocol = git
+required = False
+
+[externals_description]
+schema_version = 1.0.0
diff --git a/README.md b/README.md
@@ -0,0 +1,49 @@
+# global-workflow
+Global Superstructure/Workflow currently supporting the Finite-Volume on a Cubed-Sphere Global Forecast System (FV3GFS)
+
+The global-workflow depends on the following prerequisities to be available on the system:
+
+* workload management platform / scheduler - LSF or SLURM
+* workflow manager - ROCOTO (https://github.com/christopherwharrop/rocoto)
+* modules - NCEPLIBS (various), esmf v8.0.0bs48, hdf5, intel/ips v18, impi v18, wgrib2, netcdf v4.7.0, hpss, gempak (see module files under /modulefiles for additional details)
+* manage_externals - A utility from ESMCI to checkout external dependencies. Manage_externals can be obtained at the following address and should be in the users PATH: https://github.com/ESMCI/manage_externals
+
+The global-workflow current supports the following machines:
+
+* WCOSS-Dell
+* WCOSS-Cray
+* Hera
+
+## Build global-workflow:
+
+### 1. Check out components
+
+The global-workflow uses the manage_externals utility to handle checking out its components. The manic-v1.1.8 manage_externals tag is supported. The manage_externals utility will be replacing the current checkout.sh script.
+
+Run manage_externals (checkout_externals) while at top of clone:
+
+```
+$ checkout_externals -e Externals.cfg
+```
+
+If checkout_externals is not in your $PATH then use full path to it:
+
+* WCOSS-Dell: /gpfs/dell2/emc/modeling/noscrub/emc.glopara/git/manage_externals/manic-v1.1.8/checkout_externals
+* WCOSS-Cray: /gpfs/hps3/emc/global/noscrub/emc.glopara/git/manage_externals/manic-v1.1.8/checkout_externals
+* Hera: /scratch1/NCEPDEV/global/glopara/git/manage_externals/manic-v1.1.8/checkout_externals
+
+### 2. Build components
+
+While in /sorc folder:
+```
+$ sh build_all.sh
+```
+
+### 3. Link components
+
+While in /sorc folder:
+```
+$ sh link_fv3gfs.sh emc $MACHINE
+```
+
+...where $MACHINE is "dell", "cray", or "hera".
diff --git a/docs/doxygen/compile b/docs/doxygen/compile
@@ -3,8 +3,8 @@
 machine=${1:-${machine:-"WCOSS_C"}}
 machine=$(echo $machine | tr '[a-z]' '[A-Z]')
 
-if [ $machine = "THEIA" ]; then
-    doxygen=/scratch3/NCEPDEV/hwrf/save/Samuel.Trahan/doxygen-1.8.10/bin/doxygen
+if [ $machine = "HERA" ]; then
+    doxygen=/usr/bin/doxygen
 elif [ $machine = "WCOSS_C" ]; then
     doxygen=/gpfs/hps3/emc/hwrf/noscrub/soft/doxygen-1.8.10/bin/doxygen
 elif [ $machine = "WCOSS" ]; then

diff --git a/docs/doxygen/mainpage.h b/docs/doxygen/mainpage.h
@@ -21,7 +21,7 @@ This is a very much a work in progress and any issues should be reported back an
 To setup an experiment, a python script <b>\c setup_expt.py</b> (located in <b>\c fv3gfs/ush</b>) can be used:
 
     $> setup_expt.py -h
-    usage: setup_expt.py [-h] [--machine {THEIA,WCOSS_C}] --pslot PSLOT
+    usage: setup_expt.py [-h] [--machine {HERA,WCOSS_C}] --pslot PSLOT
                      [--configdir CONFIGDIR] [--idate IDATE] [--icsdir ICSDIR]
                      [--resdet RESDET] [--resens RESENS] [--comrot COMROT]
                      [--expdir EXPDIR] [--nens NENS] [--cdump CDUMP]

diff --git a/docs/note_fixfield.txt b/docs/note_fixfield.txt
@@ -4,7 +4,7 @@ They are saved locally on all platforms
 
 Venus/Mars:  /gpfs/dell2/emc/modeling/noscrub/emc.glopara/git/fv3gfs/fix
 Surge/Luna: /gpfs/hps3/emc/global/noscrub/emc.glopara/git/fv3gfs/fix
-THEIA: /scratch4/NCEPDEV/global/save/glopara/git/fv3gfs/fix 
+Hera: /scratch1/NCEPDEV/global/glopara/fix 
 
 ------------------------------------------------------------------------------
 09/28/2018

diff --git a/driver/gdas/test_jgdas_verfozn_theia.sh b/driver/gdas/test_jgdas_verfozn_theia.sh
diff --git a/driver/gdas/test_jgdas_verfrad_theia.sh b/driver/gdas/test_jgdas_verfrad_theia.sh
diff --git a/driver/gdas/test_jgdas_vminmon_theia.sh b/driver/gdas/test_jgdas_vminmon_theia.sh