Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implementation of ROMS+CICE coupling #4

Open
uturuncoglu opened this issue Mar 20, 2024 · 9 comments
Open

Implementation of ROMS+CICE coupling #4

uturuncoglu opened this issue Mar 20, 2024 · 9 comments
Assignees
@SmithJos13

Comments

@uturuncoglu
Copy link
Collaborator

This is generic issue that aims to have discussion about ROMS+CICE coupling. Here is the recent conversation with Hernan,

Hernan:

I designed two test cases: LAKE_ICE and LAKE_ERIE.

The LAKE_ICE is an idealized application to test the ROMS native sea ice model. I ran it for three years. See the following link for the roms_test repository:

https://github.com/myroms/roms_test/tree/main/lake_ice/Forward

I haven’t finished configuring LAKE_ERIE. I already have the grid and downloaded the ECMWF forcing, but I haven’t created the initial conditions yet. The plan was to spin up for a few years, then test the ROMS native sea ice model and use the same test for coupling ROMS-CICE in the UFS. I stopped doing this because I was asked to develop the decimation/interpolation scheme for ROMS 4D-Var inner loops that our group urgently needs to run ECCOFS. I am almost done with it, and I come back to LAKE_ERIE.
@uturuncoglu
Copy link
Collaborator Author

Followup question: Okay. That is great. Let me know when the ERIE case is ready. BTW, I just wonder if it possible to configure ROMS to get atmospheric forcing from CDEPS under UFS Coastal but using ROMS provided CICE in the first round. If that runs, it provides a baseline or reference point for us. Then we could try to replace internal CICE with UFS Weather Model provided one. Let me know what do you think?

@uturuncoglu
Copy link
Collaborator Author

uturuncoglu commented Mar 20, 2024 •
edited
Loading

This is also related with #5

@janahaddad janahaddad moved this to In Progress in ufs-coastal project Mar 21, 2024
@janahaddad janahaddad moved this from In Progress to Todo in ufs-coastal project Mar 21, 2024
@janahaddad janahaddad moved this from Todo to Backlog in ufs-coastal project Sep 23, 2024
@janahaddad janahaddad mentioned this issue Sep 23, 2024
Closed
@janahaddad janahaddad moved this from Backlog to Todo in ufs-coastal project Sep 23, 2024
@SmithJos13
Copy link

@uturuncoglu I'm going to revive this thread so we have a place to communicate.

The directory I'm currently working in on Hercules is:

/work2/noaa/vdatum/jsmith/dev/ufs-coastal-cice-dev/tests/run_dir/cice_dev_cdeps2cice_debug_intel

the base directory for my current version of ufs coastal is:

/work2/noaa/vdatum/jsmith/dev/ufs-coastal-cice-dev/

the input forcing / input grids are located in:

cdeps: /work2/noaa/vdatum/jsmith/ufs_coast_setup/Forcing/
grids for cice: /work2/noaa/vdatum/jsmith/ufs_coast_setup/Grids/CICE/

to build my current case I use

./rt.sh -l rt_cice_dev.conf -a vdatum -k -c -n "cice_dev_cdeps2cice_debug intel"

The ufs run configuration is

runSeq::
@@[DT_CICE]
  OCN
  ATM
  OCN -> ICE :remapMethod=redist
  ATM -> ICE :remapMethod=redist
  ICE
@
::

finally the error I'm experiencing is during the run phase the model will hang and not advance any further. In the PET logs I have the follow errors :

PET1

20241028 170356.355 ERROR            PET1 OCN-TO-ICE:src/addon/NUOPC/src/NUOPC_Connector.F90:1603 Invalid argument  - Ambiguous connection status, multiple connections with identical bondLevel found for: cpl_scalars
20241028 170356.355 ERROR            PET1 UFS Driver Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:3099 Invalid argument  - Phase 'IPDv05p2b' Initialize for connectorComp 2 -> 3: OCN-TO-ICE did not return ESMF_SUCCESS
20241028 170356.355 ERROR            PET1 UFS Driver Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:1423 Invalid argument  - Passing error in return code
20241028 170356.355 ERROR            PET1 UFS Driver Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:486 Invalid argument  - Passing error in return code
20241028 170356.355 ERROR            PET1 UFS.F90:394 Invalid argument  - Aborting UFS

PET2

20241028 170356.355 ERROR            PET2 OCN-TO-ICE:src/addon/NUOPC/src/NUOPC_Connector.F90:1603 Invalid argument  - Ambiguous connection status, multiple connections with identical bondLevel found for: cpl_scalars
20241028 170356.355 ERROR            PET2 UFS Driver Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:3099 Invalid argument  - Phase 'IPDv05p2b' Initialize for connectorComp 2 -> 3: OCN-TO-ICE did not return ESMF_SUCCESS
20241028 170356.355 ERROR            PET2 UFS Driver Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:1423 Invalid argument  - Passing error in return code
20241028 170356.355 ERROR            PET2 UFS Driver Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:486 Invalid argument  - Passing error in return code
20241028 170356.355 ERROR            PET2 UFS.F90:394 Invalid argument  - Aborting UFS

appreaciate any input that you have!

@SmithJos13
Copy link

SmithJos13 commented Oct 29, 2024 •
edited
Loading

Ahhh okay!

I think I found one issue I still have more that I'm working from: https://earthsystemmodeling.org/docs/nightly/develop/NUOPC_refdoc/node3.html

the error is steaming from:

After the first stage, there may be umbiguous Field pairs present. Ambiguous Field pairs are those that map different producer Fields (i.e. Fields in the importState of a Connector) to the same consumer Field (i.e. a Field in the exportState of a Connector). While the NUOPC Layer support having multiple consumer Fields connected to a single producer Field, it does not support the opposite condition. The second stage of Field pairing is responsible for disambiguating Field pairs with the same consumer.

So switching verbosity to high yields the following confliction:

20241029 145140.936 INFO             PET0   ATM-TO-ICE: ProducerConnection (bondLevelMax):         1
20241029 145140.936 INFO             PET0   ATM-TO-ICE:      importStandardNameList(i= 14): cpl_scalars
20241029 145140.936 INFO             PET0   ATM-TO-ICE:         importNamespaceList(i= 14): ATM
20241029 145140.936 INFO             PET0   ATM-TO-ICE:            importCplSetList(i= 14): __UNSPECIFIED__
20241029 145140.936 INFO             PET0   ATM-TO-ICE:      exportStandardNameList(j= 26): cpl_scalars
20241029 145140.936 INFO             PET0   ATM-TO-ICE:         exportNamespaceList(j= 26): ICE
20241029 145140.936 INFO             PET0   ATM-TO-ICE:            exportCplSetList(j= 26): __UNSPECIFIED__
20241029 145140.936 INFO             PET0   ATM-TO-ICE: bondLevel= 1

and

20241029 145140.942 INFO             PET1   OCN-TO-ICE: ProducerConnection (bondLevelMax):         1
20241029 145140.942 INFO             PET1   OCN-TO-ICE:      importStandardNameList(i= 10): cpl_scalars
20241029 145140.942 INFO             PET1   OCN-TO-ICE:         importNamespaceList(i= 10): OCN
20241029 145140.942 INFO             PET1   OCN-TO-ICE:            importCplSetList(i= 10): __UNSPECIFIED__
20241029 145140.942 INFO             PET1   OCN-TO-ICE:      exportStandardNameList(j= 26): cpl_scalars
20241029 145140.942 INFO             PET1   OCN-TO-ICE:         exportNamespaceList(j= 26): ICE
20241029 145140.942 INFO             PET1   OCN-TO-ICE:            exportCplSetList(j= 26): __UNSPECIFIED__
20241029 145140.942 INFO             PET1   OCN-TO-ICE: bondLevel= 1

which is inline with the error. There are two producers for cpl_scalars one consumer of spl_scalars thus ambiguous.

So going to the CDEPS cap for the ocn component

CDEPS-interface/CDEPS/docn/docn_datamode_copyall_mod.F90

and comment the following (line 61) :

!call dshr_fldList_add(fldsExport, trim(flds_scalar_name))

get rid of the error and the model resolved its issue!

This is not the right solution to this issue since it leads to there being no cpl_scalars in the OCN state bundle and the model falls over

Second purpose fix is to comment out the following line:

    !call fldlist_add(fldsToIce_num, fldsToIce, trim(flds_scalar_name))

in

/work2/noaa/vdatum/jsmith/dev/ufs-coastal-cice-dev

which avoids the issue listed above and yields the same error listed bellow.

@SmithJos13
Copy link

SmithJos13 commented Oct 29, 2024 •
edited
Loading

This being said the model still crashes

more specifically I have the following standard error:

2: Obtained 10 stack frames.
2: /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/unified-env/install/intel/2021.9.0/parallelio-2.5.10-rdwrsed/lib/libpioc.so(print_trace+0x29) [0x150e6df41ba9]
2: /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/unified-env/install/intel/2021.9.0/parallelio-2.5.10-rdwrsed/lib/libpioc.so(pio_err+0xa7) [0x150e6df41b57]
2: /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/unified-env/install/intel/2021.9.0/parallelio-2.5.10-rdwrsed/lib/libpioc.so(PIOc_Init_Intracomm+0x5ed) [0x150e6df44dad]
2: /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/unified-env/install/intel/2021.9.0/parallelio-2.5.10-rdwrsed/lib/libpioc.so(PIOc_Init_Intracomm_from_F90+0x14) [0x150e6df44764]
2: /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/unified-env/install/intel/2021.9.0/parallelio-2.5.10-rdwrsed/lib/libpiof.so(piolib_mod_mp_init_intracom_+0xd6) [0x150e6ded1d16]
2: /work2/noaa/vdatum/jsmith/dev/stmp/jsmith/FV3_RT/rt_3779924/cice_dev_cdeps2cice_debug_intel/./fv3.exe() [0x28e42c9]
2: /work2/noaa/vdatum/jsmith/dev/stmp/jsmith/FV3_RT/rt_3779924/cice_dev_cdeps2cice_debug_intel/./fv3.exe() [0x2884069]
2: /work2/noaa/vdatum/jsmith/dev/stmp/jsmith/FV3_RT/rt_3779924/cice_dev_cdeps2cice_debug_intel/./fv3.exe() [0x1d9ecb9]
2: /work2/noaa/vdatum/jsmith/dev/stmp/jsmith/FV3_RT/rt_3779924/cice_dev_cdeps2cice_debug_intel/./fv3.exe() [0x1b55f62]
2: /work2/noaa/vdatum/jsmith/dev/stmp/jsmith/FV3_RT/rt_3779924/cice_dev_cdeps2cice_debug_intel/./fv3.exe() [0xadbb94]
2: Abort(-1) on node 2 (rank 2 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, -1) - process 2
0: slurmstepd: error: *** STEP 2977625.0 ON hercules-01-64 CANCELLED AT 2024-10-29T15:15:56 ***

must be a PIO error netcdf error? idk though will update as I find out more

I've isolated the error to the following:

CDEPS-interface/CDEPS/streams/dshr_strdata_mod.F90

and the line

call pio_read_darray(pioid, varid, per_stream%stream_pio_iodesc, data_dbl1d, rcode)

This seems to causing a ton of issues in CICE...
Does ufs require PIO to work is there a way to build the model just using NetCDF?

@uturuncoglu
Copy link
Collaborator Author

@SmithJos13 Okay. I could not find time to look at this but eventually check and let you know. Sorry.

@SmithJos13
Copy link

No worries I'm going to keep working on this. I think I'm making some headway! Ill provide periodic updates!

@SmithJos13
Copy link

SmithJos13 commented Oct 31, 2024 •
edited
Loading

I've built the model in a "serial" format by only using one PET. I accomplished this by setting:

ATM_pet_bounds = OCN_pet_bounds = ICE_pet_bound = '0 0'

This was to diagnose what model component was causing the crash. I found a couple error in my ATM cap and fixed those. Now the model is crashing again in the ICE component. I'm currently trying to build the CICE component without PIO by changing the following line,

set(CICE_IO "PIO" CACHE STRING "CICE OPTIONS: Choose IO options.") ==> set(CICE_IO "NetCDF" CACHE STRING "CICE OPTIONS: Choose IO options.")

in

/work2/noaa/vdatum/jsmith/dev/ufs-coastal-cice-dev/CICE-interface/CMakeLists.txt

getting a little closer now it looks like there is an issues with the export pointers,

20241031 145133.099 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Export Field = Faii_tauy is not connected.
20241031 145133.099 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Export Field = Faii_lat is not connected.
20241031 145133.099 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Export Field = Faii_sen is not connected.
20241031 145133.099 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Export Field = Faii_lwup is not connected.
20241031 145133.099 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Export Field = Faii_evap is not connected.
20241031 145133.099 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Export Field = Faii_swnet is not connected.
20241031 145133.099 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Export Field = Fioi_melth is not connected.
20241031 145133.099 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Export Field = Fioi_swpen is not connected.
20241031 145133.099 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Export Field = Fioi_swpen_vdr is not connected.
20241031 145133.099 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Export Field = Fioi_swpen_vdf is not connected.
20241031 145133.099 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Export Field = Fioi_swpen_idr is not connected.
20241031 145133.099 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Export Field = Fioi_swpen_idf is not connected.
20241031 145133.099 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Export Field = Fioi_meltw is not connected.
20241031 145133.099 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Export Field = Fioi_salt is not connected.
20241031 145133.099 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Export Field = Fioi_taux is not connected.
20241031 145133.099 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Export Field = Fioi_tauy is not connected.
20241031 145133.099 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Export Field = Fioi_bcpho is not connected.
20241031 145133.100 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Export Field = Fioi_bcphi is not connected.
20241031 145133.100 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Export Field = Fioi_flxdst is not connected.
20241031 145133.100 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Import Field = So_dhdx is connected using mesh without ungridded dimension
20241031 145133.100 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Import Field = So_dhdy is connected using mesh without ungridded dimension
20241031 145133.100 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Import Field = So_t is connected using mesh without ungridded dimension
20241031 145133.101 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Import Field = So_s is connected using mesh without ungridded dimension
20241031 145133.101 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Import Field = So_u is connected using mesh without ungridded dimension
20241031 145133.101 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Import Field = So_v is connected using mesh without ungridded dimension
20241031 145133.102 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Import Field = Fioo_q is connected using mesh without ungridded dimension
20241031 145133.102 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Import Field = Sa_z is connected using mesh without ungridded dimension
20241031 145133.102 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Import Field = Sa_u is connected using mesh without ungridded dimension
20241031 145133.102 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Import Field = Sa_v is connected using mesh without ungridded dimension
20241031 145133.103 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Import Field = Sa_shum is connected using mesh without ungridded dimension
20241031 145133.103 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Import Field = Sa_tbot is connected using mesh without ungridded dimension
20241031 145133.103 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Import Field = Sa_pbot is connected using mesh without ungridded dimension
20241031 145133.104 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Import Field = Faxa_swvdr is connected using mesh without ungridded dimension
20241031 145133.104 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Import Field = Faxa_swvdf is connected using mesh without ungridded dimension
20241031 145133.104 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Import Field = Faxa_swndr is connected using mesh without ungridded dimension
20241031 145133.105 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Import Field = Faxa_swndf is connected using mesh without ungridded dimension
20241031 145133.105 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Import Field = Faxa_lwdn is connected using mesh without ungridded dimension
20241031 145133.105 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Import Field = Faxa_rain is connected using mesh without ungridded dimension
20241031 145133.105 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Import Field = Faxa_snow is connected using mesh without ungridded dimension
20241031 145133.106 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Import Field = Sa_ptem is not connected.
20241031 145133.106 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Import Field = Sa_dens is not connected.
20241031 145133.106 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Import Field = Faxa_bcph is not connected.
20241031 145133.106 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Import Field = Faxa_dstwet is not connected.
20241031 145133.106 INFO             PET0   (ice_import_export:fld_list_realize)(ice_import_export:realize_fields):CICE_Import Field = Faxa_dstdry is not connected.
20241031 145133.106 INFO             PET0   (ice_comp_nuopc):(InitializeRealize)Debug pnt 13
20241031 145133.106 INFO             PET0   ice_export called
20241031 145133.110 INFO             PET0   (field_getfldptr): ERROR data not allocated
20241031 145133.110 ERROR            PET0 ice_shr_methods.F90:340   Failure  - Passing error in return code
20241031 145133.110 ERROR            PET0 ice_import_export.F90:1074   Failure  - Passing error in return code
20241031 145133.110 ERROR            PET0 ice_comp_nuopc.F90:967   Failure  - Passing error in return code
20241031 145133.110 ERROR            PET0 UFS Driver Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:2901   Failure  - Phase 'IPDv01p3' Initialize for modelComp 3: ICE did not return ESMF_SUCCESS
20241031 145133.110 ERROR            PET0 UFS Driver Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:1985   Failure  - Passing error in return code
20241031 145133.110 ERROR            PET0 UFS Driver Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:489   Failure  - Passing error in return code
20241031 145133.111 ERROR            PET0 UFS.F90:394   Failure  - Aborting UFS

maybe the export pointers need to initialized at the top of the ice_import_export.F90 file?

Yeah I don't know what to do now. I can get the model to advance further in the initialization phase by introducing the mediator but then I think that is going to open up a whole new stack of problems. The issue that I run into there is that data being sent to the model is not on the same time step? this could be an issue caused by commenting out

  !call fldlist_add(fldsToIce_num, fldsToIce, trim(flds_scalar_name))

I've also tried turning on the component complance checker and that was uninformative.

@uturuncoglu
Copy link
Collaborator Author

@SmithJos13 I could not find your mail to send invitation. Could you send it to me?

@janahaddad janahaddad moved this from Todo to In Progress in ufs-coastal project Nov 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@SmithJos13 SmithJos13

Labels
None yet
Projects
ufs-coastal project
Status: In Progress
Milestone
No milestone
Development

No branches or pull requests
2 participants
@uturuncoglu @SmithJos13