feature/cm jjob

[malloryprow]

Note to developers: You must use this PR template!

Description of Changes

This addresses the item in the general (code manager) section of the Fixes and Additions document: "Define $NET prior to $HOMEevs; apply this change across all J-jobs".

For JEVS_GLOBAL_ENS_WAVE_GRID2OBS_*, the NET simply needed to be moved prior to HOMEevs.
For JEVS_CAM_PLOTS, JEVS_CAM_PREP, JEVS_CAM_STATS, JEVS_NARRE_PLOTS, JEVS_NARRE_STATS, they did not have NET defined in at all, so it was added along with other key variables. All dev driver and ecf scripts had these variables defined in them.

Developer Questions and Checklist

• Is this a high priority PR? If so, why and is there a date it needs to be merged by?

No

• Do you have any planned upcoming annual leave/PTO?

No

• Are there any changes needed in the times when the jobs are supposed to run/kick-off?

No

• The code changes follow NCO's EE2 Standards.
• Developer's name is removed throughout the code and have used ${USER} where necessary throughout the code.
• References the feature branch for HOMEevs are removed from the code.
• J-Job environment variables, COMIN and COMOUT directories, and output follow what has been defined for EVS.
• Jobs over 15 minutes in runtime have restart capability.
• If applicable, changes in the dev/drivers/scripts or dev/modulefiles have been made in the corresponding ecf/scripts and ecf/defs/evs-nco.def?
• Jobs contain the appropriate file checking and don't run METplus for any missing data.
• Code is using METplus wrappers structure and not calling MET executables directly.
• Log is free of any ERRORs or WARNINGs.

Testing Instructions

Set-up

1. Clone my fork and checkout branch feature/cm_jjob
2. ln -sf /lfs/h2/emc/vpppg/noscrub/emc.vpppg/verification/EVS_fix fix
3. cd sorc; ./build

For everything below, be sure to set HOMEevs to the location of the clone and COMIN to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/$NET/$evs_ver_2.

✔️ cam prep

4. cd dev/drivers/scripts/prep/cam
5. Run jevs_cam_severe_prep.sh
6. Run jevs_cam_hrrr_severe_prep.sh

• Submit with qsub -v vhr=00, qsub -v vhr=06, qsub -v vhr=12, qsub -v vhr=18` sometime after 18Z.

✔️ cam stats

7. cd dev/drivers/scripts/stats/cam
8. Run jevs_cam_href_grid2obs_stats.sh
9. Run jevs_cam_hrrr_severe_stats.sh

• Submit with qsub -v vhr=08

✔️ cam plots

10. cd dev/drivers/scripts/plots/cam
11. Run jevs_cam_nam_firewxnest_grid2obs_plots.sh

✔️ global_ens prep - wave

12. cd dev/drivers/scripts/prep/global_ens
13. Run jevs_global_ens_wave_grid2obs_prep.sh

✔️ global_ens stats - wave

14. cd dev/drivers/scripts/stats/global_ens
15. Run jevs_global_ens_wave_grid2obs_stats.sh

✔️ global_ens plots - wave

16. cd dev/drivers/scripts/plots/global_ens
17. Run jevs_global_ens_wave_grid2obs_plots.sh

✔️ narre stats

18. cd dev/drivers/scripts/stats/narre
19. Run jevs_narre_stats.sh

✔️ narre plots

18. cd dev/drivers/scripts/plots/narre
19. Run jevs_narre_last31days_plots.sh

[PerryShafran-NOAA]

@malloryprow To be clear, am I submitting both cam prep jobs at the 4 run times? Or does that only refer to the hrrr severe prep job?

[malloryprow]

Just jevs_cam_hrrr_severe_prep.sh

[PerryShafran-NOAA]

Turns out jevs_cam_severe_prep.sh needs to be run with a vhr or else it doesn't work. Thus I'm just doing vhr=00 for that one.

[PerryShafran-NOAA]

For jevs_cam_severe_prep.sh:

.o file: /lfs/h2/emc/vpppg/noscrub/perry.shafran/pr654test/EVS/dev/drivers/scripts/prep/cam/jevs_cam_severe_prep.o177285477
output: /lfs/h2/emc/vpppg/noscrub/perry.shafran/evs/v2.0/prep/cam/spc_lsr.20250109 and spc_otlk.20250116
working directory: /lfs/h2/emc/stmp/perry.shafran/evs_test/prod/tmp/jevs_cam_severe_prep.177285477.cbqs01

For the jevs_cam_hrrr_severe_prep.sh:

.o files: /lfs/h2/emc/vpppg/noscrub/perry.shafran/pr654test/EVS/dev/drivers/scripts/prep/cam/jevs_cam_hrrr_severe_prep.o177_____
output: /lfs/h2/emc/vpppg/noscrub/perry.shafran/evs/v2.0/prep/cam/hrrr.20250116
working directory: /lfs/h2/emc/stmp/perry.shafran/evs_test/prod/tmp/jevs_cam_hrrr_severe_prep.177_____

[malloryprow]

Output all looks good! The log /lfs/h2/emc/vpppg/noscrub/perry.shafran/pr654test/EVS/dev/drivers/scripts/prep/cam/jevs_cam_severe_prep.o177285477 had WARNING: Zero observations retained!. I made in the Fixes and Additions document to not run METplus if there are no observations in /lfs/h1/ops/prod/dcom/YYYYmmdd/validation_data/weather/spc/spc_reports_YYYYmmdd.csv doesn't have any reports.

[PerryShafran-NOAA]

I just noticed that the cam severe prep job is normally run at 0730 each day. Think if I set vhr=07 it might make a difference?

[malloryprow]

I don't think so! It looks like 20250109 doesn't have any reports (https://www.spc.noaa.gov/climo/reports/250109_rpts.html) so the vhr wouldn't make a difference.

[PerryShafran-NOAA]

Alright! In that case we'll just move on to the next, after my next appointment.

[PerryShafran-NOAA]

hrrr severe stats:

.o file: /lfs/h2/emc/vpppg/noscrub/perry.shafran/pr654test/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hrrr_severe_stats_00.o177289516
final stats file: /lfs/h2/emc/vpppg/noscrub/perry.shafran/evs/v2.0/stats/cam/hrrr.20250110

href grid2obs stats:
.o file: /lfs/h2/emc/vpppg/noscrub/perry.shafran/pr654test/EVS/dev/drivers/scripts/stats/cam/jevs_cam_href_grid2obs_stats.o177289499
small stats files: /lfs/h2/emc/vpppg/noscrub/perry.shafran/evs/v2.0/stats/cam/atmos.20250116/href/grid2obs
final stats files: /lfs/h2/emc/vpppg/noscrub/perry.shafran/evs/v2.0/stats/cam/href.20250116

I'm sorry, the working directories are gone because they've been scrubbed since Friday. If they are needed we can run again, perhaps, but the href grid2obs stats job is long.

[malloryprow]

All final stat files are good!

jevs_cam_hrrr_severe_stats.sh had in the log WARNING: process_scores() -> Forecast and observation valid times do not match (20250110_120000 != 19700101_000000) for Prob_MXUPHL25_A24_geHWT(*,*) versus LSR_PPF(*,*). I noted this in the Fixes and Additions (CC: @MarcelCaron-NOAA).

jevs_cam_href_grid2obs_stats: It had the ERRORs and WARNINGs we had previously noted.

[AndrewBenjamin-NOAA]

I approve this PR upon successful testing of the remaining jobs.

[PerryShafran-NOAA]

OK, if these warnings and errors are already there and not within the scope of this PR, I will move to CAM plots.

[PerryShafran-NOAA]

cam plots is finished

.o file: /lfs/h2/emc/vpppg/noscrub/perry.shafran/pr654test/EVS/dev/drivers/scripts/plots/cam/jevs_cam_nam_firewxnest_grid2obs_plots.o177545202
plot tarball: /lfs/h2/emc/ptmp/perry.shafran/evs/v2.0/plots/cam/atmos.20250120
working directory: /lfs/h2/emc/stmp/perry.shafran/evs_test/prod/tmp/jevs_cam_nam_firewxnest_grid2obs_plots.177545202.cbqs01

[malloryprow]

All is good for cam plots testing!

[PerryShafran-NOAA]

Excellent, moving on to the next!

[PerryShafran-NOAA]

global_ens wave prep is finished:

.o file: /lfs/h2/emc/vpppg/noscrub/perry.shafran/pr654test/EVS/dev/drivers/scripts/prep/global_ens/jevs_global_ens_wave_grid2obs_prep.o177550710
prep directory: /lfs/h2/emc/vpppg/noscrub/perry.shafran/evs/v2.0/prep/global_ens/wave.20250120/gefs/grid2obs
working directory: /lfs/h2/emc/stmp/perry.shafran/evs_test/prod/tmp/jevs_global_ens_wave_grid2obs_prep.177550710.cbqs01

[malloryprow]

👍 No problems in the log files and all prep files are accounted for.

[PerryShafran-NOAA]

global_ens wave stats is complete:

.o file: /lfs/h2/emc/vpppg/noscrub/perry.shafran/pr654test/EVS/dev/drivers/scripts/stats/global_ens/jevs_global_ens_wave_grid2obs_stats.o177551835
small stats files: /lfs/h2/emc/vpppg/noscrub/perry.shafran/evs/v2.0/stats/global_ens/wave.20250120/gefs/grid2obs
final stats file: /lfs/h2/emc/vpppg/noscrub/perry.shafran/evs/v2.0/stats/global_ens/gefs.20250120
working directory: /lfs/h2/emc/stmp/perry.shafran/evs_test/prod/tmp/jevs_global_ens_wave_grid2obs_stats.177551835.cbqs01

[PerryShafran-NOAA]

global_ens wave plots is finished:

.o file: /lfs/h2/emc/vpppg/noscrub/perry.shafran/pr654test/EVS/dev/drivers/scripts/plots/global_ens/jevs_global_ens_wave_grid2obs_plots.o177552529
plot tarballs: /lfs/h2/emc/ptmp/perry.shafran/evs/v2.0/plots/global_ens/wave.20250120
working directory: /lfs/h2/emc/stmp/perry.shafran/evs_test/prod/tmp/jevs_global_ens_wave_grid2obs_plots.177552529.cbqs01

[malloryprow]

global_ens stats and plots for wave are both good! Only WARNING messages were is in the plots log files but that is due to missing final stat files from the WCOSS2 outages.

[PerryShafran-NOAA]

Great! Working on the narre stuff now.

[PerryShafran-NOAA]

narre stats:

.o file: /lfs/h2/emc/vpppg/noscrub/perry.shafran/pr654test/EVS/dev/drivers/scripts/stats/narre/jevs_narre_stats.o177553640
small stats files: /lfs/h2/emc/vpppg/noscrub/perry.shafran/evs/v2.0/stats/narre/atmos.20250120/narre/grid2obs
final stats file: /lfs/h2/emc/vpppg/noscrub/perry.shafran/evs/v2.0/stats/narre/narre.20250120
working directory: /lfs/h2/emc/stmp/perry.shafran/evs_test/prod/tmp/jevs_narre_stats.177553640.cbqs01

narre plots:

.o file: /lfs/h2/emc/vpppg/noscrub/perry.shafran/pr654test/EVS/dev/drivers/scripts/plots/narre/jevs_narre_last31days_plots.o177554611
plots tarball: /lfs/h2/emc/ptmp/perry.shafran/evs/v2.0/plots/narre/atmos.20250120
working directory: /lfs/h2/emc/stmp/perry.shafran/evs_test/prod/tmp/jevs_narre_last31days_plots.177554611.cbqs01

[malloryprow]

narre stats and plots are good!

[PerryShafran-NOAA]

OK great! Let me look at the code changes and then I'll merge.

[PerryShafran-NOAA]

Code works as expected. Approved for merge.