SCHISM+WW3 H. Ian testing-suite: Phase 1

[yunfangsun]

General plan for (2D) SCHISM+WW3 coupling:

Plan is to separate these from the "official" RT system, which we'll keep light-weight and attached to the model-level. The heavier cases here will eventually live on the app-level as a separate testing-suite.

Hurricane Ian "Testing suite" plan

(from @saeed-moghimi-noaa) Here is my suggestion for steps for simulations to be completed, check-mark and start the next step:

Phase 1 (no open BCs in WW3)

• stand alone SCHISM for Ian
  • native time loop (schism.exe)
  • SCHISM standalone (outside UFS-Coastal) for Ian on HSOFS mesh #105
  • SCHISM standalone (outside UFS Coastal), H. Ian, coarse Atl mesh (550k node mesh, "crs")
  • SCHISM standalone (outside UFS Coastal), H. Ian, combined Atl mesh (> 2 mil node mesh, "Atlantic")
• stand alone WW3 for Ian
  • native time loop (multi.exe)
  • Standalone WW3 with HSOFS mesh for H. Ian case  #112
  • Standalone WW3 no OBC with coarse Atl mesh for H. Ian case: ww3_alone_crs/ian_noobc #120
  • WW3 no OBC standalone (outside UFS Coastal), H. Ian, combined Atlantic mesh
• ufs-coastal SCHISM alone for Ian
  • nems config with single component
  • coastal_ian_hsofs_schism #100
  • coastal_ian_crs_schism #119
  • coastal_ian_atlantic_schism
• ufs-coastal WW3 alone for Ian
  • nems config with single component
  • coastal_ian_hsofs_ww3 #99
  • coastal_ian_crs_ww3 #115
  • coastal_ian_atlantic_ww3_noOBCs
• ufs-coastal DATM+SCHISM alone for Ian
  • nems config (.exe)
  • coastal_ian_hsofs_datm2sch #101
  • coastal_ian_crs_datm2sch #118
  • coastal_ian_atlantic_datm2sch #90
• ufs-coastal DATM+WW3 alone for Ian
  • nems config (.exe)
  • coastal_ian_hsofs_datm2ww3 #102
  • coastal_ian_crs_datm2ww3 #116
  • coastal_ian_atlantic_datm2ww3 #91
• ufs-coastal DATM+SCHISM+WW3 alone for Ian
  • nems config (.exe)
  • coastal_ian_hsofs_datm2sch2ww3 #103
  • coastal_ian_crs_datm2sch2ww3 #117
  • DATM+SCHISM+WW3 water level exchange (coarse resolution SCHISM mesh) #132
  • coastal_ian_atlantic_datm2sch2ww3 #124

Phases 2 & 3

• Phase 2: wave BCs from Ali A. (SCHISM+WW3 H. Ian testing-suite: Phase 2 #96)
• Phase 3: wave BCs from Global run (SCHISM+WW3 H. Ian testing-suite: Phase 3 #97)

RT system

• Plan for RT system SCHISM & WW3 configurations: SCHISM and WW3 RT system cases (main issue) #95

[yunfangsun]

#24

[yunfangsun]

#24 (comment)

#24 (comment)

#24 (comment)

#24 (comment)

[saeed-moghimi-noaa]

@yunfangsun
Please add details for the meshes being used.

[yunfangsun]

Hi @uturuncoglu @pvelissariou1 ,

Do you have any example cases for a ufs-coastal SCHISM alone? such as ike_shinnecock.sch in CoastalApp-testsuite.

Thank you!

[uturuncoglu]

@yunfangsun No, we don't have that configuration as RT under ufs-coastal.

[yunfangsun]

@yunfangsun Please add details for the meshes being used.

The mesh here combines the following two sets of meshes:

1. STOFS mesh with 11969815 elements and 6056968 nodes within 34kt of Hurricane Ian swath

2. Coarse mesh with 557481 nodes and 1069847 elements outside the 34kt of Hurricane Ian swath

The final mesh is with 4886065 elements and 2470094 nodes:

[yunfangsun]

The configurations of the cases are set on Hercules in the folder /work2/noaa/nos-surge/yunfangs/hurricane_ian
seven case folders are inside, as follows:

stand alone SCHISM for Ian (sch_alone)
stand alone WW3 for Ian (ww3_alone)
ufs-coastal WW3 alone for Ian (ufs_ww3)
ufs-coastal SCHISM alone for Ian (ufs_sch)
ufs-coastal DATM+SCHISM alone for Ian (ufs_atm2sch)
ufs-coastal DATM+WW3 alone for Ian (ufs_atm2ww3)
ufs-coastal DATM+SCHISM-WW3 for Ian (ufs_atm2sch2ww3)

[janahaddad]

@yunfangsun, great work and thank you for documenting. A few questions for clarity:

1. Are you using this combined mesh for all the 7 configs you have listed? If not, can you sort them by mesh being used as well as winds?
2. What's the common name of this combined mesh? If there is not one, we should come up with one to easily refer to it. What do you think?
3. Can you please separate your list of configs into those which are complete & running, those which are not yet complete but in progress, and those you have not yet started?

[yunfangsun]

@yunfangsun, great work and thank you for documenting. A few questions for clarity:

1. Are you using this combined mesh for all the 7 configs you have listed? If not, can you sort them by mesh being used as well as winds?

All the cases are using the same mesh. all the winds are ERA5 wind

2. What's the common name of this combined mesh? If there is not one, we should come up with one to easily refer to it. What do you think?

Maybe atltantic_ian?

3. Can you please separate your list of configs into those which are complete & running, those which are not yet complete but in progress, and those you have not yet started?

Ok, I will update the status of each case.

[janahaddad]

Ok thanks @yunfangsun, since this is the mesh that was giving you problems in #24, do you think it's worth going through this exercise first for ufs_atm2sch ufs_atm2sch2ww3 ufs_atm2ww3 ufs_sch ufs_ww3 with a known working mesh e.g. 1.8mil node hsofs mesh or even a coarser one that you can test quickly?

[yunfangsun]

@janahaddad

Thank you very much! It is an excellent suggestion, I think I could try the Hsofs mesh for the ufs_atm2sch ufs_atm2sch2ww3 ufs_atm2ww3 ufs_sch ufs_ww3 firstly, and could also compare the results between different mesh later.

[yunfangsun]

I have already set up the HSOFS mesh for Hurricane Ian for the following cases:

WW3 stand-alone: /work2/noaa/nos-surge/yunfangs/hurricane_ian/ww3_alone_hsofs
UFS-coastal ATM+WW3: /work2/noaa/nos-surge/yunfangs/hurricane_ian/ufs_atm2ww3_hsofs

[yunfangsun]

@uturuncoglu

I am trying to use the UFS-coastal_ww3 case for Hurricane Ian with HSOFS mesh, there is no error message in the out, err and ww3_shel.nml.log, however, the job is hanging without running. and the PET00.ESMF_LogFile said that

20240521 091013.496 INFO             PET00 (wav_comp_nuopc:wavinit_ufs): mod_def timesteps file      100.00    100.00    100.00    100.00
20240521 091026.157 INFO             PET00 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Export Field = Sw_z0 is not connected.
20240521 091026.157 INFO             PET00 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Export Field = Sw_wavsuu is not connected.
20240521 091026.157 INFO             PET00 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Export Field = Sw_wavsuv is not connected.
20240521 091026.157 INFO             PET00 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Export Field = Sw_wavsvv is not connected.
20240521 091026.157 INFO             PET00 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Export Field = Sw_pstokes_x is not connected.
20240521 091026.157 INFO             PET00 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Export Field = Sw_pstokes_y is not connected.
20240521 091026.157 INFO             PET00 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Import Field = Si_ifrac is not connected.
20240521 091026.157 INFO             PET00 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Import Field = So_u is not connected.
20240521 091026.157 INFO             PET00 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Import Field = So_v is not connected.
20240521 091026.157 INFO             PET00 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Import Field = So_t is not connected.
20240521 091026.157 INFO             PET00 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Import Field = Sa_tbot is not connected.
20240521 091026.157 INFO             PET00 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Import Field = Sa_u10m is not connected.
20240521 091026.157 INFO             PET00 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Import Field = Sa_v10m is not connected.
20240521 091026.158 INFO             PET00 (wav_comp_nuopc):(ModelSetRunClock) called
20240521 091026.158 INFO             PET00 (wav_comp_nuopc):(ModelSetRunClock)setting alarms for WAV
20240521 091026.159 INFO             PET00 (wav_comp_nuopc):(ModelSetRunClock) done
20240521 091026.159 INFO             PET00 ------>Advancing WAV from: 2022  9 15  0  0  0   0
20240521 091026.159 INFO             PET00 --------------------------------> to: 2022  9 15  1  0  0   0

Do you have any suggestions on how to fix it? The run folder is located at /work2/noaa/nos-surge/yunfangs/hurricane_ian/ufs_ww3_hsofs on Hercules

Thank you very much!

[uturuncoglu]

It seems that you are using 1000 processor for this job but in ufs.configure you are setting like following,

WAV_petlist_bounds:             0 11

So, maybe that is the reason.

[uturuncoglu]

I am not sure this is for RT or not but i think these are using too much resources for replacing existing RTs with realistic cases. We need to have very small domain with realistic forcing and hurricane.

[yunfangsun]

Hi @uturuncoglu
The HSOFS mesh will be used for applications not for RT, I will develop RT cases by using shinnecock mesh.

[pvelissariou1]

@uturuncoglu I full agree. On the other hand @yunfangsun needs to create and evaluate the Ian tests cases for our testing purposes (meshes, configs, w/wo WW3, etc ...). I believe at this point is trying to use the RT system. But yes, at some point we need to use these outside the main RT system or, to generate a special rt_coastal file for these long cases. @yunfang will chime in.

[uturuncoglu]

Okay. That is great.

[yunfangsun]

It seems that you are using 1000 processor for this job but in ufs.configure you are setting like following,

WAV_petlist_bounds:             0 11

So, maybe that is the reason.

Hi @uturuncoglu ,

Thank you for the suggestion, I have corrected the ufs.configure by

WAV_petlist_bounds: 0 999

However, the PET log file is still the same,

20240521 105444.889 INFO             PET000 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Export Field = Sw_z0 is not connected.
20240521 105444.889 INFO             PET000 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Export Field = Sw_wavsuu is not connected.
20240521 105444.889 INFO             PET000 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Export Field = Sw_wavsuv is not connected.
20240521 105444.889 INFO             PET000 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Export Field = Sw_wavsvv is not connected.
20240521 105444.889 INFO             PET000 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Export Field = Sw_pstokes_x is not connected.
20240521 105444.889 INFO             PET000 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Export Field = Sw_pstokes_y is not connected.
20240521 105444.889 INFO             PET000 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Import Field = Si_ifrac is not connected.
20240521 105444.889 INFO             PET000 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Import Field = So_u is not connected.
20240521 105444.890 INFO             PET000 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Import Field = So_v is not connected.
20240521 105444.890 INFO             PET000 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Import Field = So_t is not connected.
20240521 105444.890 INFO             PET000 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Import Field = Sa_tbot is not connected.
20240521 105444.890 INFO             PET000 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Import Field = Sa_u10m is not connected.
20240521 105444.890 INFO             PET000 (wav_import_export:fldlist_realize)(wav_import_export:realize_fields):WW3Import Field = Sa_v10m is not connected.

Is there any other configuration file related to this error?
Thank you very much!

[pvelissariou1]

@yunfangsun Maybe just a couple storms for the test cases but using different configs for the models SCHISM, WW3 [& PAHM]. Need to have a "complete" set of working tests so ufs-coastal and models can be tested properly after each update.

[yunfangsun]

@pvelissariou1 I plan to use shinnecock mesh to develop for a few of RT cases, SCHISM, WW3, ATM+WW3, ATM+SCHISM, ATM+SCHISM+WW3 etc. for a couple of storms

[janahaddad]

@yunfangsun which storm are you planning on for the Shinnecock case? I realized yesterday we said H. Ian but that wouldn't make much sense for that domain. Sandy?

[pvelissariou1]

@janahaddad , @yunfangsun I do have sandy test cases locally. Sandy is the only storm that produces winds (PaHM) over that domain but for shinnecock mesh we can use atm data other than sandy/pahm to set the test cases. For comparison/evaluation Sandy might be the storm choice. We do have the "Idealized" test cases for shinnecock as well. Need to finish some tasks for NHC and I'll push/test the cases I have locally, in a few days.

[saeed-moghimi-noaa]

@pvelissariou1 Do you mean Sandy on hsofs mesh?

[pvelissariou1]

@saeed-moghimi-noaa Sandy on shinnecock mesh. Florence on hsofs mesh.

[uturuncoglu]

@pvelissariou1 As we discussed yesterday. we will replace idealized cases with realistic ones. There is no need to keep multiple case with same configuration under RT system since it is just checking the infrastructure. Maybe 1-2 lightweight test for each different configuration is fine. The more realistic and high resolution cases will go the app level testing.

[yunfangsun]

@yunfangsun which storm are you planning on for the Shinnecock case? I realized yesterday we said H. Ian but that wouldn't make much sense for that domain. Sandy?

@janahaddad

I think ELSA 2021 and HENRI 2021 could fit the Shinnecock mesh.

https://en.wikipedia.org/wiki/Hurricane_Henri#/media/File:Henri_2021_path.png
https://en.wikipedia.org/wiki/Hurricane_Elsa#/media/File:Elsa_2021_path.png

[pvelissariou1]

@uturuncoglu Ufuk, the Idealized test cases will eventually go away, for the time being we need to keep them as they have been tested w/wo WW3. What we need is multiple "lightweight" cases for Shinnecock that employ different options (physics) in SCHISM, ADCIRC, WW3, etc.. (like in ADCIRC/WW3 unit tests upstream) so we can test ufs-coastal and model components quickly. This is what I meant. Also we need to evaluate model results and check if they make sense from the scientific side of things. Anyway this is just a suggestion.

[pvelissariou1]

@yunfang That is great!! Is it possible to check if there are any observations (co-ops, usgs, etc.) for these storms in the area? What is the proximity of the storms to the Shinnecock domain? I'll run PaHM later today using best tracks for these storms to see if any atm. fields are generated for the domain. I'll report back.

[yunfangsun]

@pvelissariou1 , I am developing the applications for Hurricane Ian in the Atlantic domain as shown at the beginning of this issue, the storms to the Shinnecock domain will not be my current tasks.

[janahaddad]

@yunfangsun thanks, that plan is in line with recent team discussions. For clarity I'll update this current Issue to reflect H. Ian HSOFS mesh setup for your list of configurations above, and will open a future/"to do" issue for the RT-system Shinnecock cases.

[saeed-moghimi-noaa]

General plan for (2D) SCHISM+WW3 coupling:

Plan is to separate these from the "official" RT system, which we'll keep light-weight and attached to the model-level. The heavier cases here will eventually live on the app-level as a separate testing-suite.

Hurricane Ian "Testing suite" plan

(from @saeed-moghimi-noaa) Here is my suggestion for steps for simulations to be completed, check marked and start the next step:

Phase 1

* [ ]   stand alone SCHISM for Ian
  
  * native time loop (schism.exe)
  * [ ]  HSOFS mesh
  * [ ]  combined Atl mesh

* [ ]  stand alone WW3 for Ian
  
  * native time loop  (multi.exe)
  * [ ]  HSOFS mesh
  * [ ]  combined Atl mesh

* [ ]  ufs-coastal SCHISM alone for Ian
  
  * nems config with single component (NEMS.exe)
  * [ ]  HSOFS mesh
  * [ ]  combined Atl mesh

* [ ]  ufs-coastal WW3 alone for Ian (NEMS.exe)
  
  * nems config with single component
  * [ ]  HSOFS mesh
  * [ ]  combined Atl mesh

* [ ]  ufs-coastal DATM+SCHISM alone for Ian
  
  * nems config with single component (NEMS.exe)
  * [ ]  HSOFS mesh
  * [ ]  combined Atl mesh

* [ ]  ufs-coastal DATM+WW3 alone for Ian
  
  * nems config with single component (NEMS.exe)
  * [ ]  HSOFS mesh
  * [ ]  combined Atl mesh

* [ ]  ufs-coastal DATM+SCHISM-WW3 for Ian
  
  * nems config with single component (NEMS.exe)
  * [ ]  HSOFS mesh
  * [ ]  combined Atl mesh

Phases 2 & 3

* Phase 2: wave BCs from Ali A. (add issue #)

* Phase 3: wave BCs from Global run (add issue #)

@yunfangsun
Suggest to do HSOFS for all the cases and then come back and do the high res ones. What you think?

[pvelissariou1]

@yunfangsun , @janahaddad PAHM produces the following atmospheric fields for Shinnecock domain:
Henri: From 2021-08-21 23:00:00 to 2021-08-22 12:00:00
Elsa: From 2021-07-09 04:00:00 to 2021-07-09 22:00:00
Both less than a day. We do have some data. We may pick one of the two in addition to Sandy.

[saeed-moghimi-noaa]

Hi @pvelissariou1

We are focusing on the mechanics of the 2D ww3-schsim for realistic cases (1st for hsofs mesh, then for the Ian subset mesh).

1. without wave bou
2. with wave bou

Feel free to take the lead towards what you are thinking if you have the bandwidth.

Thanks,
-Saeed

[yunfangsun]

@saeed-moghimi-noaa

Yes, it is my routine, I will do the HSOFS for the 6 non-obc cases first, then use the high-resolution subset mesh.

[pvelissariou1]

@saeed-moghimi-noaa ok will do, thanks. All if it is ok, let's forget Elsa/Henri for shinnecock (will keep Sandy for this) and let's focus on HSOFS cases and Ian subset mesh cases. @yunfangsun If its ok, I'll schedule a meeting with you tomorrow morning to discuss things over and what we need to do to finalize the test cases.

[saeed-moghimi-noaa]

Hi @pvelissariou1

On a similar note, you were working on a similar issue PAHM+ww3+adcirc on hsofs mesh for AMS2024, right?

1. without wave bou
2. with wave bou

I assume the one without wave bou was ready and validated before you move to the NHC position, right? If you need a meeting I would be more than happy to talk to you about the plan and review the results / validation for the adc-ww3 coupling.

Just want to make sure we all know what we are doing and do not conflate plans / tasks.

@janahaddad
Do we have a specific issue for PAHM+ww3+adcirc? Perhaps we need to move this conversation over there.

Thanks,
-Saeed

[janahaddad]

Hi @pvelissariou1

On a similar note, you were working on a similar issue PAHM+ww3+adcirc on hsofs mesh for AMS2024, right?

1. without wave bou
2. with wave bou

I assume the one without wave bou was ready and validated before you move to the NHC position, right? If you need a meeting I would be more than happy to talk to you about the plan and review the results / validation for the adc-ww3 coupling.

Just want to make sure we all know what we are doing and do not conflate plans / tasks.

@janahaddad Do we have a specific issue for PAHM+ww3+adcirc? Perhaps we need to move this conversation over there.

Thanks, -Saeed

yes, we do have an issue I moved to backlog: #73 -- I believe this was from the original AMS2024 list. We can rename this issue as needed if the interest is now to focus first on Sandy winds, but let's use that issue for this configuration "category".