UFS-Coastal Input File Generation Scripts for Workflow

[mansurjisan]

@uturuncoglu , @janahaddad

UFS-Coastal Input File Generation Scripts for Workflow

Collection of scripts for generating input files for UFS-Coastal

Repository Link

UFS-Coastal-Inputs on GitHub

This is the continuation of this ticket oceanmodeling/ufs-weather-model#127 (comment)

Script update:

SCHISM Boundary Condition Generator (gen_bctides.py)

I updated SCHISM's boundary condition generation script (gen_bctides.py) to be used
within the UFS-Coastal for Ike Shinnecock and Duck, NC regression test cases.

Workflow diagram:

workflow

Features

• Multiple Boundary Types:

  • Type 3: Tidal boundaries (using TPXO or FES2014)
  • Type 4: Time-elevation boundaries (from elev.th file or HYCOM data)

• Data Sources:

  • Local time series files (elev.th)
  • HYCOM oceanographic data for:
    • Water elevation (elev2D.th.nc)
    • Temperature (TEM_3D.th.nc)
    • Salinity (SAL_3D.th.nc)

• Automated Processing:

  • Automatic boundary detection from hgrid.ll
  • Flexible grid file format support
  • some error handling features

Usage

1. Tidal Boundary for Ike Shinnecock Testcase:

python gen_bctides.py ../../data/hgrid.ll 2008-08-23 20 \
    --bc_mode tidal \
    --bc_type 3 \
    --constituents Q1,O1,P1,K1,N2,M2,S2,K2,Mm,Mf,M4,MN4,MS4,2N2,S1 \
    --database tpxo \
    --cutoff_depth 40 \
    --earth_tidal_potential Y

2. Time Series Boundary for Duck, NC Testcase (requires TPXO tide database access):

python gen_bctides.py ../../data/hgrid.ll 2012-10-27 2.333 \
    --bc_mode time-elev \
    --bc_type 4 \
    --elev_th elev.th \
    --vgrid vgrid.in

3. HYCOM Boundary:

python gen_bctides.py hgrid.ll 2024-01-01 10 \
    --bc_mode time-elev \
    --bc_type 4 \
    --elev_source hycom \
    --gen_bc elev,temp,salt \
    --vgrid vgrid.in

Required Files

• hgrid.ll: SCHISM horizontal grid file
• vgrid.in: Vertical grid file (for time-elevation mode)
• elev.th: Time series file (if using time series source)

Dependencies

• pyschism
• netCDF4
• numpy

schism_bc_in_ufs_coastal.pdf

[mansurjisan]

@josephzhang8 , @feiye-vims , @saeed-moghimi-noaa , @SorooshMani-NOAA ,

As we're integrating PySCHISM functionality into UFS-Coastal and other operational systems, I believe we should discuss the future development and maintenance of PySCHISM.

Some discussion points:

1. PySCHISM Development:

• Long-term maintenance strategy
• Integration with UFS-Coastal workflow
• Version control and release management
• Feature requests for operational needs
• Documentation improvements
• Testing framework

2. Alternative Approaches:
   • Standalone operational tools
   • Existing Python package alternatives

Would appreciate your insights on the best path forward for development and operational support.

[uturuncoglu]

@mansurjisan Let me know if scripts are ready to port into workflow.

[josephzhang8]

Thx @mansurjisan.

Currently VIMS team is maintaining pySCHISM, but due to sever manpower shortage we were only able to do bug fixes in the past years or so. Development of new capabilities is out of question. I would appreciate it if other teams can take charge.

[SorooshMani-NOAA]

@josephzhang8 I remember we had this discussion about PySCHISM vs PyLibs a while ago. We understand that it's not feasible to maintain two separate packages which do the same thing in the long run, especially given the developer availability; so we wanted to make sure we understand what the plans are for PySCHISM and PyLibs, and if we, for example, need to switch to PyLibs more instead of SCHISM.

If I remember correctly, at some point you mentioned VIMS wants to keep PySCHISM for its downloading capabilities, while PyLibs is better for its setup capabilities.

We just wanted to double check before either side invest more time and energy (i.e. scripting) on one or the other tool.

[josephzhang8]

It's not easy to merge the 2 packages due to very different starting points:

1. pySCHISM: written to make common tasks as easy as one-button
2. pyLibs: written to minimize dependencies with other packages that are being constantly updated

I personally use both depending on what tasks I want to accomplish. For model setup using HYCOM, atmos forcing, I use pySCHISM as it's more developed in those areas. I use pyLibs (schismview, schismcheck, pload) to viz inputs/outputs and also load from DEMs and generating b.c. part for hgrid. I'm sure there are other usages for either tool that I have not used.

It'd be good to have a meeting involving developers of the two tools. I did that a few years ago and the consensus was that it's premature to discuss merging. pyLibs developer has been reluctant to add HYCOM and atmos capabilities if doing so would mean he has to add dependencies; at the moment pyLibs has a very small core lib that proved to be powerful for what we are doing. Maybe you all have good ideas in this regard.

[mansurjisan]

@mansurjisan Let me know if scripts are ready to port into the workflow.

Hi @uturuncoglu , the script to generate the boundary condition for SCHISM is ready to be used within UFS-Coastal workflow. The script is located here: https://github.com/mansurjisan/UFS-Coastal-Inputs/blob/main/scripts/coastal_ike_shinnecock_atm2sch/gen_bctides.py

Also, i attached a PDF file at the end of this comment: #11 (comment)

We can talk about it more during our Monday meeting.

[uturuncoglu]

@mansurjisan Okay. Let me try to use it under workflow. I'll update you soon.

[mansurjisan]

Thanks @uturuncoglu . I forgot to mention that the script requires to have the TPXO data located in /home/$username/.local/share/tpxo directory. Please give it a try and let me know if any additional changes need to be made.

[mansurjisan]

I've developed a script to generate meteorological forcing for UFS-Coastal data atmospheric cap using the pySCHISM library. While the script successfully downloads and processes GFS and HRRR data from AWS, I discovered a limitation for historical test cases:

• Ike Shinnecock (2008-08-23)
• Duck, NC (2012-10-27)

The AWS archive only contains:

• HRRR data from late 2016
• GFS data from early 2021

Alternative data sources for historical cases:

1. ERA5 Reanalysis (recommended)
2. NOMADS archive
3. NCAR Research Data Archive

I plan to extend the script to include ERA5 data download functionality for historical cases in the next update.

The current script is available at:

https://github.com/mansurjisan/UFS-Coastal-Inputs/blob/main/scripts/coastal_ike_shinnecock_atm2sch/download_met.py

Usage examples:

# HRRR data
python download_met.py --start-date "2025-01-03 12:00" --rnday 1 --model hrrr --var uwind vwind prmsl --hgrid ../../data/hgrid.gr3

# GFS data
python download_met.py --start-date "2025-01-03 12:00" --rnday 1 --model gfs --var uwind vwind prmsl --hgrid ../../data/hgrid.gr3

# For Grib2 data format 
python download_met.py --start-date "2025-01-03 12:00" --rnday 1 --model gfs --format grib2
 --var uwind vwind prmsl --hgrid ../../data/hgrid.gr3

[josephzhang8]

There is a script for ERA5 under examples/Sflux. It requires users to download a key from ECMWF site.

[mansurjisan]

@josephzhang8

Thank you, Joseph! I usually use that script to download ERA5 data and will be incorporating it into the existing meteorological forcing download script for UFS-Coastal data atmosphere input generation purposes.

[uturuncoglu]

@mansurjisan What is the best way to point out the location of TPXO file in gen_bctides.py file?

[uturuncoglu]

@mansurjisan Okay.I found it in the pyschsim code.

export TPXO_ELEVATION=/work/noaa/nosofs/mjisan/pyschism-main/PySCHISM_tutorial/data/TPXO/h_tpxo9.v1.nc
export TPXO_VELOCITY=/work/noaa/nosofs/mjisan/pyschism-main/PySCHISM_tutorial/data/TPXO/u_tpxo9.v1.nc 

[mansurjisan]

@uturuncoglu , thanks, Ufuk. The bctides script reads the TPXO data from the files located in home directory. My TPXO files are located here in Hercules: /home/mjisan/.local/share/tpxo

Please let me know if you have any questions.

[uturuncoglu]

@mansurjisan Okay. I run entire workflow and compared the results with the baseline (the run without schism related tasks and retrieving files from presaged coastal_ike_shinnecock_atm2sch RT folder).

It seems that ocnImp_So_omask in mediator history is not identical with the baseline run. I also checked the model output and it seems that outputs/schout_000000_1.nc file is also different.

I need to investigate little bit more to understand why but it seems that the input files generated by the scripts are not same with the ones found in the RT. I just wonder if you test these scripts to reproduce the results of the existing RT (coastal_ike_shinnecock_atm2sch). In my case, I might able to have different atmospheric forcing but I was expecting to have same files for the other inputs to SCHSIM. Any idea? Maybe we are not generating those files with the same way that was generated previously.

[uturuncoglu]

@mansurjisan I also realized that the RT has no files like SAL_3D.th.nc. So, maybe RT is not using any boundary condition. Not sure. But of course this does not explain the difference between runs that I did with and without schism tasks.

[uturuncoglu]

@mansurjisan Maybe gr3 files are different since we are using fixed values in https://github.com/mansurjisan/UFS-Coastal-Inputs/blob/9b9163f263862a7ea94292dc53cfcd407bc34b6c/scripts/coastal_ike_shinnecock_atm2sch/gen_gr3_input.py#L34 and they might be different from the ones used to create RT files.

[mansurjisan]

@uturuncoglu , let me take a look at the RT Shinnecock case again. But I can confirm that the bctides file that we use in the RT case and the one I generated using TPXO are not exactly the same. We actually don't have any information on how that bctides file was generated that is in the RT Shinnecock directory.

[uturuncoglu]

@mansurjisan Thanks for your help. That is fine to have difference if we know the issue. I know it is hard to reproduce the exiting case since we have lack of information. I also think that there is no any issue with gr3 files. Anyway, let me finalize the initial version of the workflow and merge with the app. so, you (or anyone else) could also try to run a case. In the mean time, let me know if you find anything. We could also go with another case that we know how the files are generated such as duck (only atm and ocn) case etc. I think that would be more useful since we know how those test ate created. Let me know what you think.

[mansurjisan]

@uturuncoglu , thank you! Yes, I agree. We better understand the input data sources and generation steps for the Duck, NC test case. Let's talk about it more during our next meeting. I will keep you posted about the Shinnecock test case.

[uturuncoglu]

@mansurjisan Yes, good idea.

[uturuncoglu]

@mansurjisan I have a minor question. Do we need both hgrid.gr3 and hgrid.ll files to run SCHISM cases. Why we have two file to define horizontal grid. Maybe one of them can be removed from run directory. Any idea?

[mansurjisan]

@uturuncoglu ,

We still need hgrid.ll. I tried to run SCHISM without hgrid.ll file, but the model failed. Also, when nws is set to 2, I think, the model requires hgrid.ll file.

[uturuncoglu]

@mansurjisan what about the gr3 file. Does it also fail if you remove that one? I am not sure what is the main difference between those files. Anyway, thanks for checking.

[mansurjisan]

@uturuncoglu , hgrid.gr3 contains grid information in Cartesian coordinates when ics=1 or in geographic coordinates when ics=2 in param.nml file. For our case, we are using ics = 2. hgrid.ll always contains grid information in geographic coordinates.

I tried to run the model without the hgrid.gr3 file, but the model failed. This is not surprising since hgrid.gr3 is a required file in the SCHISM model. We can always do ln -sf between hgrid.gr3 and hgrid.ll since they are both the same file.

[uturuncoglu]

@mansurjisan Thanks for all the details. It is good to know.

[uturuncoglu]

Original message:
I have good news about the application level workflow. I could able to run coastal_ike_shinnecock_atm2sch_intel RT with the workflow (all files are generated by the workflow except forcing) and the results are bit to bit with RT baseline. The only issue that I noticed is in bctides.in file. If I generate that file with the workflow (that uses your script gen_bctides) than results are not identical. It seems that the generated file is not the one used in coastal_ike_shinnecock_atm2sch_intel test case. So, I am also getting that file from the RT. I was using following options when I am crating the file,

  bctides:
    mode: tidal
    constituents: ['Q1','O1','P1','K1','N2','M2','S2','K2','Mm','Mf','M4','MN4','MS4','2N2','S1']  
    database: 'tpxo'
    earth_tidal_potential: true
    cutoff_depth: 40
    bc_type: 3
    tpxo_dir: /work/noaa/nosofs/mjisan/pyschism-main/PySCHISM_tutorial/data/TPXO

I wonder if it possible to look at bctides.in in coastal_ike_shinnecock_atm2sch_intel and find the correct combination of those parameters that I could test with the workflow. Then, we don’t need to get that file from the RT directory. Anyway, I think this is very good progress and probably I could merge the current development to app soon (I still need to do couple of improvement) and then try with other cases like Duck (btw, if you have a run directory DATM+SCHISM for Duck case let me know - incl. options that I need to use for your scripts and I could give a shot).

Thanks for your help. It was not possible without your scripts and help.

Update from: @mansurjisan:

This is great news! Following up on our previous discussion about bctides.in (#11 (comment)), I'm still having trouble reproducing the bctides.in file used in the coastal_ike_shinnecock_atm2sch_intel case. A key challenge is that we don't know the original data source used to create that file.

I've attempted to generate a new bctides.in file using my script (https://github.com/mansurjisan/UFS-Coastal-Inputs/blob/9b9163f263862a7ea94292dc53cfcd407bc34b6c/scripts/coastal_ike_shinnecock_atm2sch/gen_bctides.py). I used the same tidal constituents as defined in the existing bctides.in file, using the following command:

python gen_bctides.py hgrid.ll 2008-08-23 20
--bc_mode tidal
--bc_type 3
--constituents Q1,O1,P1,K1,N2,M2,S2,K2,Mm,Mf,M4,MN4,MS4,2N2,S1
--database tpxo
--cutoff_depth 40
--earth_tidal_potential Y

I used the TPXO tide database. While the generated bctides.in file is close, it's not identical to the existing one. Without knowing the original tide database source used to create the existing bctides.in file, it's difficult to achieve an exact match.

[uturuncoglu]

@mansurjisan Okay. Thanks. Maybe there is no need to have identical results in app level. At least we know the source of the issue in here. The other thing is that if I generate the bctides.in with the script and run the case. I am seeing some change in the ocean model mask in the last mediator history file. I wonder if it is due to the wetting and drying. Is it active by default in the SCHISM code? Other than that I think it is fine and we could try to run other cases.

[uturuncoglu]

@pvelissariou1 I wonder if you know the source of bctides.in file found in coastal_ike_shinnecock_atm2sch RT case. We are trying to generate through the workflow at this point without any luck.

[pvelissariou1]

@uturuncoglu Ufuk I'll schedule a quick meeting in a while to explain how it is done. I have my files on hera

[uturuncoglu]

@pvelissariou1 Thanks. JFYI, I'll be out of office in a half hour and get back around 12 MT.

[pvelissariou1]

I sent an invitation, I will rescedule

[uturuncoglu]

@mansurjisan JFYI, We had a meeting with @pvelissariou1 yesterday. The source of the bctides.in is FES2014 dataset but after creating base data, there is an extra step to make it work for specific date. @pvelissariou1 will provide the scripts and information about his workflow and I'll test in my end and eventually make it available under workflow if we could generate the same file.

[pvelissariou1]

@uturuncoglu , @mansurjisan The archive of the files for the generation of bctides.sh is located at hercules:
/work2/noaa/nos-surge/pvelissa/make-bctides-ufs_coastal.tar.bz2
I have a README.txt file in make-bctides-ufs_coastal
NOTE: Give it an hour; data are being compressed on hera and will be transferred to hercules afterwards

[pvelissariou1]

@uturuncoglu , @mansurjisan the data on hercules now are at:
/work2/noaa/nos-surge/pvelissa/make-bctides-ufs_coastal

[mansurjisan]

@pvelissariou1 , @uturuncoglu ,

Thank you very much for the updates and for sharing additional files. I will look into the scripts and will update you on Monday.