Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HRv4 hangs on orion and hercules #2486

Open
RuiyuSun opened this issue Oct 30, 2024 · 43 comments
Open

HRv4 hangs on orion and hercules #2486

RuiyuSun opened this issue Oct 30, 2024 · 43 comments
Labels
bug Something isn't working

Comments

@RuiyuSun
Copy link
Contributor

RuiyuSun commented Oct 30, 2024 •
edited
Loading

George V. noticed that The HRv4 does not work on Hercules or Orion. It hangs sometime after WW3 starts. No relevant message in the log files about the hanging.

To Reproduce: Run an HRv4 experiment on Hercules or Orion

Additional context

Output
@RuiyuSun RuiyuSun added the bug Something isn't working label Oct 30, 2024
@GeorgeVandenberghe-NOAA
Copy link
Collaborator

This happens at high ATM resolution C1152.

@RuiyuSun
Copy link
Contributor Author

RuiyuSun commented Nov 4, 2024

I made a HRv4 test run on orion as well. As reported previously, it hung at the beginning of the run.

The log file is at /work2/noaa/stmp/rsun/ROTDIRS/HRv4

HOMEgfs=/work/noaa/global/rsun/git/global-workflow.hr.v4 (source)
EXPDIR=/work/noaa/global/rsun/para_gfs/HRv4
COMROOT=/work2/noaa/stmp/rsun/ROTDIRS
RUNDIRS=/work2/noaa/stmp/rsun/RUNDIRS

@LarissaReames-NOAA
Copy link

@RuiyuSun Denise reports that the privacy settings on your directories are preventing her from accessing them. Could you check on that and report back when it's fixed so others can look at your forecast?

@RuiyuSun
Copy link
Contributor Author

RuiyuSun commented Nov 5, 2024

@DeniseWorthen I made the changes. Please try again.

@JessicaMeixner-NOAA
Copy link
Collaborator

I've made a few test runs on my end and here are some observations:

  • This also fails at C768 S2SW
  • This fails at C1152 S2S (so I do not think this is wave-grid related).

Consistently all runs I have made, also the same as @RuiyuSun runs stall out here:

    0:  fcst_initialize total time:    200.367168849800
    0:  fv3_cap: field bundles in fcstComp export state, FBCount=            8
    0:  af allco wrtComp,write_groups=           4
 9216: NOTE from PE     0: MPP_DOMAINS_SET_STACK_SIZE: stack size set to    32768.
 9216:  &MPP_IO_NML
 9216:  HEADER_BUFFER_VAL       =       16384,
 9216:  GLOBAL_FIELD_ON_ROOT_PE = T,
 9216:  IO_CLOCKS_ON    = F,
 9216:  SHUFFLE =           0,
 9216:  DEFLATE_LEVEL   =          -1,
 9216:  CF_COMPLIANCE   = F
 9216:  /
 9216: NOTE from PE     0: MPP_IO_SET_STACK_SIZE: stack size set to     131072.
 9216: NOTE from PE     0: MPP_DOMAINS_SET_STACK_SIZE: stack size set to 16000000.
 9216:  num_files=           2
 9216:  num_file=           1 filename_base= atm output_file= netcdf_parallel
 9216:  num_file=           2 filename_base= sfc output_file= netcdf_parallel
 9216:  grid_id=            1  output_grid= gaussian_grid
 9216:  imo=        4608 jmo=        2304
 9216:  ideflate=           1
 9216:  quantize_mode=quantize_bitround quantize_nsd=           5
 9216:  zstandard_level=           0
    0:  af wrtState reconcile, FBcount=           8
    0:  af get wrtfb=output_atm_bilinear rc=           0

With high resolution runs (C768 & C1152) for various machines we've had to use different number of write grid tasks. I've tried a few and all are stalling though. This is using ESMF managed threading, so one thing to try might be moving away from that?

To run a high res test case:

git clone --recursive https://github.com/NOAA-EMC/global-workflow
cd global-workflow/sorc
./build_all.sh
./link_workflow.sh
cd ../../
mkdir testdir 
cd testdir 
source ../global-workflow/workflow/gw_setup.sh 
HPC_ACCOUNT=marine-cpu pslot=C1152t02 RUNTESTS=`pwd` ../global-workflow/workflow/create_experiment.py --yaml ../global-workflow/ci/cases/hires/C1152_S2SW.yaml

Change C1152 to C768 to run that resolution and also change your HPC_ACCOUNT, pslot, as desired. Lastly, if you want to turn off waves, you change that in C1152_S2SW.yaml. If you want to change resources, look in global-workflow/parm/config/gfs/config.ufs in the C768/C1152 section.

If you want to run S2S only, change the app in global-workflow/ci/cases/hires/C1152_S2SW.yaml

My latest run log files can be found at:
/work2/noaa/marine/jmeixner/wavesforhr5/test01/C1152t0*/COMROOT/C1152t0*/logs/2019120300/gfs_fcst_seg0.log
(several runs are in progress, but they've all been running for over an hour an all hung on the same spot, despite changing write grid tasks).

@JessicaMeixner-NOAA
Copy link
Collaborator

@GeorgeVandenberghe-NOAA suggested trying 2 write groups with 240 tasks in them. I meant to try that but tried 2 write groups with 360 tasks per group unintentionally, but I did turn on all PET files as @LarissaReames-NOAA thought that might have helpful info.

The rundirectory is here: /work2/noaa/marine/jmeixner/wavesforhr5/test01/STMP/RUNDIRS/C1152t06/gfs.2019120300/gfsfcst.2019120300/fcst.272800

The log file is here: /work2/noaa/marine/jmeixner/wavesforhr5/test01/C1152t06/COMROOT/C1152t06/logs/2019120300/gfs_fcst_seg0.log

The PET logs to me also point to write group issues. Any help with this would be greatly appreciated.

Tagging @aerorahul for awareness.

@JacobCarley-NOAA
Copy link

Thanks to everyone for the work on this. Has anyone tried this configuration with the write component off? That might help isolate where there problem is (hopefully) and then we can direct this accordingly for further debugging.

@JessicaMeixner-NOAA
Copy link
Collaborator

I have not tried this without the write component.

@DusanJovic-NOAA
Copy link
Collaborator

@JessicaMeixner-NOAA and others, I grabbed the run directory from the last experiment you ran (/work2/noaa/marine/jmeixner/wavesforhr5/test01/STMP/RUNDIRS/C1152t06/gfs.2019120300/gfsfcst.2019120300/fcst.272800), changed it to run just ATM component and converted it to run with traditional threading. It is currently running in /work2/noaa/stmp/djovic/stmp/fcst.272800, and it passed the initialization phase and finished writing 000 and 003 hour outputs successfully. I submitted the job with just 30 min wall-clock time limit, so it will fail soon.

I suggest you try running full coupled version with traditional threading if it's easy to reconfigure it.

@jiandewang
Copy link
Collaborator

some good news:
I tried HR4 tag, the only thing I changed is WRTTASK_PER_GROUP_PER_THREAD_PER_TILE_GFS from 20 to 10 and model is running Note my run is S2S. See log file at
/work/noaa/marine/Jiande.Wang/HERCULES/HR4/work/HR4-20191203/COMROOT/2019120300/HR4-20191203/logs/2019120300/gfsfcst_seg0.log

@jiandewang
Copy link
Collaborator

my 48hr run finished

@JessicaMeixner-NOAA
Copy link
Collaborator

@DusanJovic-NOAA I tried running without ESMF threading - but am struggling to get it set-up correctly and go through. @aerorahul is it expected that turning off esmf managed threading in the workflow should work?

I'm also trying on hercules to replicated @jiandewang's success but with S2SW.

@jiandewang
Copy link
Collaborator

I also lanched one S2SW but it's still in pending status

@JessicaMeixner-NOAA
Copy link
Collaborator

WRTTASK_PER_GROUP_PER_THREAD_PER_TILE_GFS=10 with S2S did not work on orion: /work2/noaa/marine/jmeixner/wavesforhr5/test01/C1152t03/COMROOT/C1152t03/logs/2019120300/gfs_fcst_seg0.log

@jiandewang
Copy link
Collaborator

mine is on hercules

@jiandewang
Copy link
Collaborator

@JessicaMeixner-NOAA my gut feeling is the issue is related to the memory/node, hercules has more than orion. Maybe you can try 5 on orion

@aerorahul
Copy link
Contributor

@DusanJovic-NOAA I tried running without ESMF threading - but am struggling to get it set-up correctly and go through. @aerorahul is it expected that turning off esmf managed threading in the workflow should work?

I'm also trying on hercules to replicated @jiandewang's success but with S2SW.

Traditional threading is not yet supported in the global-workflow as an option. We have the toggle for it, but it requires a different set of ufs_configure files and I think we are waiting for that kind of work to be in the ufs-weather-model repo.

@DusanJovic-NOAA
To run w/ traditional threading, what else did you update in the test case borrowed from @JessicaMeixner-NOAA?

@DusanJovic-NOAA
Copy link
Collaborator

DusanJovic-NOAA commented Nov 8, 2024 •
edited
Loading

I only changed ufs.configure:

  1. remove all components except ATM
  2. change globalResourceControl: from true to false
  3. change ATM_petlist_bounds: to be 0 3023 - this numbers are lower and upper bounds of MPI ranks (0 based) used by the ATM model, in this case 24166 + 2360, where 24 and 16 are layout values from input.nml and 2360 are write comp values from model_configure
  4. change ATM_omp_num_threads: from 4 to 1

And, I added job_card by copying one of the job_card from regression test run and changed:

  1. export OMP_NUM_THREADS=4 - where 4 is a number of OMP threads
  2. srun --label -n 3024 --cpus-per-task=4 ./ufs_model.x - here 3024 is a number of MPI ranks, 4 is a number of threads
  3. #SBATCH --nodes=152
    #SBATCH --ntasks-per-node=80

80 is then number of cores on hercules compute nodes
152 is the minimal number of nodes such that 152*80 >= 3024

@aerorahul
Copy link
Contributor

I only changed ufs.configure:

  1. remove all components except ATM
  2. change globalResourceControl: from true to false
  3. change ATM_petlist_bounds: to be 0 3023 - this numbers are lowe and upper bounds of MPI ranks used by the ATM model, in this case 24_16_6 + 2_360, where 24 and 16 are layout values from input.nml and 2_360 are write comp values from model_configure

And, I added job_card by copying one of the job_card from regression test run and changed:

  1. export OMP_NUM_THREADS=4 - where 4 is a number of OMP threads
  2. srun --label -n 3024 --cpus-per-task=4 ./ufs_model.x - here 3024 is a number of MPI ranks, 4 is a number of threads
  3. #SBATCH --nodes=152
    #SBATCH --ntasks-per-node=80

80 is then number of cores on hercules compute nodes 152 is the minimal number of nodes such that 152*80 >= 3024

Ok. Yes. That makes sense for the atm-only.
Does your ufs.configure have a line for

ATM_omp_num_threads:            @[atm_omp_num_threads]

@[atm_omp_num_threads] would have been 4. Did you remove it? Or does it not matter since globalResourceControl is set to false?

The original value for ATM_petlist_bounds must have been 0 755 that you changed to 0 3023, I am assuming.

@GeorgeVandenberghe-NOAA
Copy link
Collaborator

GeorgeVandenberghe-NOAA commented Nov 8, 2024 via email

@DusanJovic-NOAA
Copy link
Collaborator

I just fixed my comment about ATM_omp_num_threads:. I set it to 1 from 4, I'm not sure if it's ignored when globalResourceControl is set to false

The original value for ATM_petlist_bounds was something like 12 thousand or something like that, that included MPI ranks times 4 threads.

@GeorgeVandenberghe-NOAA
Copy link
Collaborator

GeorgeVandenberghe-NOAA commented Nov 8, 2024 via email

@aerorahul
Copy link
Contributor

@JessicaMeixner-NOAA
I think the global-workflow is coded to use the correct ufs_configure template and set the appropriate values for PETLIST_BOUNDS and OMP_NUM_THREADS in the ufs_configure file.
The default in the global-workflow is to use ESMF_THREADING = YES. I am pretty sure one could use traditional threading as well, but is an unconfirmed fact as there was still work being done to confirm traditional threading will work on WCOSS2 with the slignshot updates and whatnot. Details on that are fuzzy to me at the moment.

BLUF, you/someone from the applications team could try traditional threading and we could gain some insight on performance at those resolutions. Thanks~

@GeorgeVandenberghe-NOAA
Copy link
Collaborator

GeorgeVandenberghe-NOAA commented Nov 8, 2024 via email

@aerorahul
Copy link
Contributor

Ok. @GeorgeVandenberghe-NOAA. Where do we employ traditional threading C768 and up? If so, we can set a flag in the global-workflow for those resolutions to use traditional threading. It should be easy enough to set that up.

@GeorgeVandenberghe-NOAA
Copy link
Collaborator

GeorgeVandenberghe-NOAA commented Nov 8, 2024 via email

@JessicaMeixner-NOAA
Copy link
Collaborator

Unfortunately I was unable to replicate @jiandewang hercules success for HR4 tag with the top of develop. Moreover, 10 write tasks per group was not a lucky number for orion either.

@JessicaMeixner-NOAA
Copy link
Collaborator

Unfortunately I was unable to replicate @jiandewang hercules success for HR4 tag with the top of develop. Moreover, 10 write tasks per group was not a lucky number for orion either.

Note this was with added waves - so this might have also failed for @jiandewang if he has used waves.

@jiandewang
Copy link
Collaborator

summary for more tests I did on HERCULES:
(1) S2S, fv3 layout=8x16, write task per group=10, runs fine, further repeated 3 cases, all fine
(2) same as (1) but layout=24x16, hang
(3) repeat (1) and (2) but S2SW, all hang

@GeorgeVandenberghe-NOAA
Copy link
Collaborator

@JessicaMeixner-NOAA
Copy link
Collaborator

On orion, turning off the write grid component means that we're now hanging during wave initialization. The log file can be found here: /work2/noaa/marine/jmeixner/wavesforhr5/test01/nowritegridt2/COMROOT/test01/logs/2019120300/gfs_fcst_seg0.log

Instructions from @aerorahul on running the workflow without the write grid component are:

config.base: QUILTING=.false.
and
ush/parsing_model_configure_FV3.sh QUILTING_RESTART=".false."

FYI @JacobCarley-NOAA

@JacobCarley-NOAA
Copy link

On orion, turning off the write grid component means that we're now hanging during wave initialization. The log file can be found here: /work2/noaa/marine/jmeixner/wavesforhr5/test01/nowritegridt2/COMROOT/test01/logs/2019120300/gfs_fcst_seg0.log

Instructions from @aerorahul on running the workflow without the write grid component are:

config.base: QUILTING=.false. and ush/parsing_model_configure_FV3.sh QUILTING_RESTART=".false."

FYI @JacobCarley-NOAA

Thanks @aerorahul for the instructions and @JessicaMeixner-NOAA for running the test! This is helpful. Give me a moment and I'll figure out next steps.

@DeniseWorthen
Copy link
Collaborator

DeniseWorthen commented Nov 14, 2024 •
edited
Loading

At the risk of muddying the waters, I copied the WW3 mod_def and mesh from Jessica's run directory and was able to start up and run the model on hercules on using the DATM-S2SW configuration I've built. This is WW3 at the top of the current dev/ufs-weather-model and WAV is on 592 tasks.

@LarissaReames-NOAA
Copy link

At the risk of muddying the waters, I copied the WW3 mod_def and mesh from Jessica's run directory and was able to start up and run the model on hercules on using the DATM-S2SW configuration I've built. This is WW3 at the top of the current dev/ufs-weather-model and WAV is on 592 tasks.

So should the only difference from your experiment and Jessica's be Hercules vs. Orion? Or is the compile/run config different? Trying to narrow down potential sources of difference.

@DeniseWorthen
Copy link
Collaborator

DeniseWorthen commented Nov 14, 2024 •
edited
Loading

My case uses a DATM with MOM6+CICE6 on 1/4 deg tripole and a given WW3 configuration (structured, unstructured etc). So it essentially eliminates any fv3atm related issues.

@DeniseWorthen DeniseWorthen mentioned this issue Nov 14, 2024
Open
@DeniseWorthen
Copy link
Collaborator

If I someone can provide me a canned run-directory on either Hercules or Orion, I can see if I can figure out what is going on. But I'll need to pause work on issue #2466 to do so.

@RuiyuSun
Copy link
Contributor Author

@DeniseWorthen @JacobCarley-NOAA A canned case was created on Orion. Please see below for the information:
Jobcard is located in the experiment dir: /work/noaa/global/rsun/para_gfs/HRv4
RUNDIR is at /work/noaa/stmp/ruiyusun/ORION/RUNDIRS/HRv4/gfs.2020072500/gfsfcst.2020072500/fcst.665610

This is my first time creating a canned case. Please let me know if anything is missing.

@DeniseWorthen
Copy link
Collaborator

DeniseWorthen commented Nov 15, 2024 •
edited
Loading

Thanks @RuiyuSun. Just to be sure, this is the UFS HRv4 tag (fcc9f84)?

@aerorahul
Copy link
Contributor

aerorahul commented Nov 15, 2024 •
edited
Loading

On Hercules, I have done my best to setup 2 canned cases; both with APP=S2SW
Both cases use 592 tasks for waves (as opposed to 1000 as is the case in HR4). I used this number based on comments from @DeniseWorthen earlier in this issue.
This is not the HR4 tag of the model, but a more recent hash 6a4e09e9

Both cases are set to run out to 12 hours on Hercules, in the debug queue to have faster throughput.

The cases have job cards, module files, and environment variables set as would have been via the workflow.
These directories have no dependencies on any space in my area. The cases can be copied and run by editing a few paths in the respective jobcards.

C768:/work/noaa/stmp/rmahajan/HERCULES/RUNDIRS/sandbox/c768s2sw
C1152: /work/noaa/stmp/rmahajan/HERCULES/RUNDIRS/sandbox/c1152s2sw

cp -R /work/noaa/stmp/rmahajan/HERCULES/RUNDIRS/sandbox/c1152s2sw ./c1152s2sw.run
cd ./c1152s2sw.run
sbatch c1152s2sw_gfsfcst.sh

This should work as long as the user has access to the fv3-cpu allocation.

My sample run directories are in: /work/noaa/stmp/rmahajan/HERCULES/RUNDIRS/sandbox/RUN

@DeniseWorthen
Copy link
Collaborator

@aerorahul Thanks. One question I did have which options are used to compile? I'm assuming what we are compiling is 32-bit?

-DAPP=S2SW -D32BIT=ON -DCCPP_SUITES=FV3_GFS_v17_coupled_p8_ugwpv1 -DPDLIB=ON

@RuiyuSun
Copy link
Contributor Author

Thanks @RuiyuSun. Just to be sure, this is the UFS HRv4 tag (fcc9f84)?

I Just checked it is HRv3 tag.

/work/noaa/global/rsun/git/global-workflow.hr.v4/sorc/ufs_model.fd$ git branch

@DusanJovic-NOAA
Copy link
Collaborator

I grabbed @aerorahul 's c1152s2sw run directory, converted it to traditional threading (in fact I converted atm to use only one thread) and ran the job script. Atm initialization finished in about 4 minutes, and then the rest of 30 min wall clock time was spent in ww3 initialization. I see hundreds if not thousands of lines in log.ww3 like:

*** WAVEWATCH-III WARNING :
     OUTPUT POINT OUT OF GRID :   -161.118    21.337  HNL61
     POINT SKIPPPED


 *** WAVEWATCH-III WARNING :
     OUTPUT POINT OUT OF GRID :   -160.707    22.494  HNL62
     POINT SKIPPPED


 *** WAVEWATCH-III WARNING :
     OUTPUT POINT OUT OF GRID :   -160.295    23.650  HNL63
     POINT SKIPPPED


 *** WAVEWATCH-III WARNING :
     OUTPUT POINT OUT OF GRID :   -159.275    23.198  HNL64
     POINT SKIPPPED


 *** WAVEWATCH-III WARNING :
     OUTPUT POINT OUT OF GRID :   -158.254    22.746  HNL65
     POINT SKIPPPED


 *** WAVEWATCH-III WARNING :
     OUTPUT POINT OUT OF GRID :   -158.759    21.799  HNL66
     POINT SKIPPPED

printed every second or few seconds, very slowly. So I just removed almost all the lines from ww3_shel.inp in order to speed up the ww3 init phase. After that, the model started the run phase and finished little more than 3hr in 30-min wall clock limit.

My run directory is /work2/noaa/stmp/djovic/c1152s2sw

@aerorahul the climatological fixed files (specified in input.nml) are still pointing to your directory, so strictly speaking this is not a self-contained (canned case) run directory. If you remove your gwWork directory, people will not be able to run this. Consider moving all fixed files in, for example, fix subdirectory and update the paths in input.nml to have a truly self-contained run directory.

@DusanJovic-NOAA
Copy link
Collaborator

Interestingly, c768 uses more cores than c1152:

$ diff /work/noaa/stmp/rmahajan/HERCULES/RUNDIRS/sandbox/c1152s2sw/ufs.configure /work/noaa/stmp/rmahajan/HERCULES/RUNDIRS/sandbox/c768s2sw/ufs.configure
23c23
< ATM_petlist_bounds:             0 4031
---
> ATM_petlist_bounds:             0 5567
34c34
< OCN_petlist_bounds:             4032 4251
---
> OCN_petlist_bounds:             5568 5787
48c48
< ICE_petlist_bounds:             4252 4371
---
> ICE_petlist_bounds:             5788 5907
64c64
< WAV_petlist_bounds:             4372 4963
---
> WAV_petlist_bounds:             5908 6499

Is this intended?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@JacobCarley-NOAA @aerorahul @jiandewang @JessicaMeixner-NOAA @DeniseWorthen @RuiyuSun @DusanJovic-NOAA @LarissaReames-NOAA @GeorgeVandenberghe-NOAA