building prototype 8

[cristianastan2]

Hi,

I was running on Hera a slightly modified version of P8 using the global workflow. After a scheduled maintenance of Hera, the run started to crash randomly. I thought that rebuilding the executable will help. However, the model hangs and the build does not complete.

[Cristiana.Stan@hfe08 sorc]$ ./build_all.sh
Creating ../exec folder
.... Building WW3 pre and post execs ....
.... Building forecast model ....

Thank you,
Cristiana

[kevenmblackman]

Hello Cristiana: We will be in touch shortly about the issue you are seeing on Hera and convert this discussion to an issue.

[cristianastan2]

Hello, Thank you for your email. I was able to build and compile the UFS P8 using the build_ufs.sh script. I will test to check if the code actually runs and give you an update. Cristiana

…

-- Cristiana Stan Professor Associate Chair Department of Atmospheric, Oceanic and Earth Sciences Research Hall | Room: 267 George Mason University 4400 University Drive | MSN: 6C5 Fairfax | VA 22030 (703) 993-5391 | ***@***.***

________________________________ From: kevenmblackman ***@***.***> Sent: Monday, April 17, 2023 1:48 PM To: ufs-community/EPIC-GST ***@***.***> Cc: Cristiana Stan ***@***.***>; Author ***@***.***> Subject: Re: [ufs-community/EPIC-GST] building prototype 8 (Discussion #14) Hello Cristiana: We will be in touch shortly about the issue you are seeing on Hera and convert this discussion to an issue. — Reply to this email directly, view it on GitHub<#14 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHNTMYQEJBCSBCD26VX6X7DXBV7AVANCNFSM6AAAAAAW4AHALQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[cristianastan2]

Hello, I want to give you an update on the attempt to run the model. Unfortunately, re-compiling the code did not help. The model continues to crash randomly. The error messages are not useful. I do see some errors that might be related to mpi/threading but they might be l triggered by the stop calls from the failing code. Here is a snapshot from the log: 489: ufs_model.x 00000000062738EB Unknown Unknown Unknown 489: libpthread-2.17.s 00002AB0EDEF4630 Unknown Unknown Unknown 489: libmpi.so.12 00002AB0ED38CC9E PMPIDI_CH3I_Progr Unknown Unknown 489: libmpi.so.12.0 00002AB0ED5CB8FD Unknown Unknown Unknown 489: libmpi.so.12 00002AB0ED67CC42 PMPI_Probe Unknown Unknown Cristiana

…

________________________________ From: Cristiana Stan ***@***.***> Sent: Monday, April 17, 2023 2:30 PM To: ufs-community/EPIC-GST ***@***.***>; ufs-community/EPIC-GST ***@***.***> Subject: Re: [ufs-community/EPIC-GST] building prototype 8 (Discussion #14) Hello, Thank you for your email. I was able to build and compile the UFS P8 using the build_ufs.sh script. I will test to check if the code actually runs and give you an update. Cristiana

-- Cristiana Stan Professor Associate Chair Department of Atmospheric, Oceanic and Earth Sciences Research Hall | Room: 267 George Mason University 4400 University Drive | MSN: 6C5 Fairfax | VA 22030 (703) 993-5391 | ***@***.***

________________________________ From: kevenmblackman ***@***.***> Sent: Monday, April 17, 2023 1:48 PM To: ufs-community/EPIC-GST ***@***.***> Cc: Cristiana Stan ***@***.***>; Author ***@***.***> Subject: Re: [ufs-community/EPIC-GST] building prototype 8 (Discussion #14) Hello Cristiana: We will be in touch shortly about the issue you are seeing on Hera and convert this discussion to an issue. — Reply to this email directly, view it on GitHub<#14 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHNTMYQEJBCSBCD26VX6X7DXBV7AVANCNFSM6AAAAAAW4AHALQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[ulmononian]

Hi @cristianastan2. I attempted to re-create the issue you described here, but before I could build or run, there was a failure in ./checkout.sh due to non-existence of the release/ops-gfsv16 branch of ufs_utils (see attached image). It looks as though the fork branch you are working with is 871 commits ahead of the NOAA-GSL/global-workflow develop branch, and the NOAA-GSL/global-workflow branch is itself 1091 commits behind the NOAA-EMC/global-workflow develop branch.

It may be possible to assess the failure you are seeing if you are able to bring your fork current with the NOAA-EMC/global-workflow develop branch; otherwise the code base is too far diverged from NOAA-EMC/global-workflow for our team to debug. If you aren't able to bring https://github.com/shansun6/global-workflow/tree/p8_restart_release closer to NOAA-EMC/global-workflow develop, I would recommend contacting someone at NOAA GSL for assistance.