Implement GitHub Actions CI testing (Fixes #138) (#189)

This PR fixes Issue #138, which was about implementing CI in MOM_Interface with four initial tasks. Please reference back to the issue for those requirements. Changes Introduced by this PR: A GitHub Action Workflow (please see below for details) under .github/workflows/general-ci-tests.yml Incidentally, Black formatting on two scripts in cime_config CI/Github Action Workflow Details: Workflow is structured into three independent jobs that are triggered on push and pull-request events onto the main branch: checking the black format of scripts in cime_config running the check_default_params test under the tests folder building the standalone MOM6 and running two lightweight MOM6 examples (double_gyre, single_column/KPP) Testing: Test Failures: The bottom two jobs (2 & 3) were tested by creating an error (See Commits starting with Test CI), and the first job with Black formatting was already failing (commit) and was corrected early on in the process. Test Events/Triggers: Every job in the commit history was triggered on a push. Pull-Requests were tested in Test CI: Pull Requests #188. Notes: The commits should (need?) be squashed. (My fault, hah) Checking out the correct MOM branch for the standalone build is a little trickier because we are starting with the CESM repo. To be safe, the triggering event is manually checked out instead of using the common Github action actions/checkout@v4 (See the steps Checkout initial event (Pull Request) and Checkout initial event (Push)). I think the workflow is pretty easy to read through, but would welcome any advice/help on the clarity.
ESCOMP · Sep 4, 2024 · 3e193bd · 3e193bd
1 parent 26b97e0
commit 3e193bd
Show file tree

Hide file tree

Showing 10 changed files with 383 additions and 99 deletions.
diff --git a/.github/workflows/general-ci-tests.yml b/.github/workflows/general-ci-tests.yml
@@ -0,0 +1,110 @@
+name: General MOM_interface CI
+
+# This CI workflow tests against the following questions: 
+# 1. Does standalone mom build and run?
+# 2. Does it pass the tests/check_default_params.py test?
+# 3. Do the scripts in cime_config pass the black formatter? 
+# Please see Issue #138 for more information
+
+# Controls when the workflow will run
+on:
+  # Triggers the workflow on push or pull request events but only for the listed branches
+  push:
+    branches: ["main" ]
+  pull_request:
+    branches: [ "main" ]
+
+jobs:
+
+  check_standalone_mom_build_and_run_lightweight_examples:
+    # The type of runner that the job will run on
+    runs-on: ubuntu-latest
+
+    steps:
+
+    # Copied from NCAR/MOM6 - Install Basic Build Packages for MOM6
+      - name: Install Ubuntu Linux packages
+        shell: bash
+        run: |
+          echo "::group::Install linux packages"
+          sudo apt-get update
+          sudo apt-get install netcdf-bin
+          sudo apt-get install libnetcdf-dev
+          sudo apt-get install libnetcdff-dev
+          sudo apt-get install openmpi-bin
+          sudo apt-get install libopenmpi-dev
+          sudo apt-get install linux-tools-common
+          sudo apt-get install -y csh
+          echo "::endgroup::"
+          
+      # Checkout CESM (default branch) and externals
+      - name: Checkout CESM and Externals
+        run: |
+          git clone https://github.com/ESCOMP/CESM.git 
+          cd CESM
+          ./bin/git-fleximod update
+          
+      # Checkout the correct MOM Branch
+      - name: Checkout initial event (Pull Request)
+        if: ${{ github.event_name == 'pull_request' }}
+        run: |
+          echo "Handling pull request"
+          cd $GITHUB_WORKSPACE/CESM/components/mom/
+          git fetch origin pull/${{ github.event.pull_request.number }}/head:pr-${{ github.event.pull_request.number }}
+          git checkout pr-${{ github.event.pull_request.number }}
+
+      - name: Checkout initial event (Push)
+        if: ${{ github.event_name == 'push' }}
+        run: |
+          echo "Handling push"
+          cd $GITHUB_WORKSPACE/CESM/components/mom/
+          git checkout ${{ github.sha }}
+
+      # Build the standalone mom using the macos script. build_examples-ncar doesn't work.    
+      - name: Build Standalone MOM
+        run: |
+          cd $GITHUB_WORKSPACE/CESM/components/mom/standalone/build
+          ./build_examples.sh --compiler gnu --machine ubuntu
+          
+      # CD into a couple MOM examples and run MOM in them. These are very light weight and quick.
+      - name: Run Double Gyre Test
+        run: |
+          cd $GITHUB_WORKSPACE/CESM/components/mom/standalone/examples/double_gyre
+          $GITHUB_WORKSPACE/CESM/components/mom/standalone/build/gnu/MOM6/MOM6
+      - name: Run Single Column KPP Test
+        run: |
+          cd $GITHUB_WORKSPACE/CESM/components/mom/standalone/examples/single_column/KPP
+          $GITHUB_WORKSPACE/CESM/components/mom/standalone/build/gnu/MOM6/MOM6
+          
+
+  # Job to run the check_default_params script, which is a test
+  check_default_params:
+
+    runs-on: ubuntu-latest
+
+    steps:
+      # Checkout the repo
+      - uses: actions/checkout@v4
+
+      # Run the test
+      - name: Run the check_default_params script
+        run: python tests/check_default_params.py
+
+  # Job to run the black formatter for cime_config, see black documentation for more info
+  check_black_format_for_cime_config:
+
+    runs-on: ubuntu-latest
+
+    steps:
+      # Checkout the repo
+      - uses: actions/checkout@v4
+
+      # Run black check
+      - uses: psf/black@stable
+        with:
+          options: "--check --verbose"
+          src: "./cime_config"
+
+
+
+
diff --git a/.travis.yml b/.travis.yml
diff --git a/README.md b/README.md
@@ -1,4 +1,4 @@
-[![Build Status](https://travis-ci.org/ESCOMP/MOM_interface.svg?branch=master)](https://travis-ci.org/ESCOMP/MOM_interface)
+[![MOM_interface CI](https://github.com/ESCOMP/MOM_interface/actions/workflows/general-ci-tests.yml/badge.svg)](https://github.com/ESCOMP/MOM_interface/actions/workflows/general-ci-tests.yml)
 
 # MOM_interface
 

diff --git a/cime_config/SystemTests/dimcs.py b/cime_config/SystemTests/dimcs.py
@@ -32,8 +32,8 @@
     "H_RESCALE_POWER = -11",
     "Z_RESCALE_POWER = -11",
     "R_RESCALE_POWER = -11",
-    "Q_RESCALE_POWER = -11"
-    ]
+    "Q_RESCALE_POWER = -11",
+]
 
 run_suffixes = [
     "base",
@@ -67,27 +67,36 @@
     "scale the dimension Q by 2**-11",
 ]
 
+
 class DIMCS(SystemTestsCompareN):
 
     def __init__(self, case):
         self.comp = case.get_value("COMP_OCN")
-        SystemTestsCompareN.__init__(self, case, N=len(nl_contents),
-                                     separate_builds = False,
-                                     run_suffixes = run_suffixes,
-                                     run_descriptions = run_descriptions,
-                                     ignore_fieldlist_diffs = True)
+        SystemTestsCompareN.__init__(
+            self,
+            case,
+            N=len(nl_contents),
+            separate_builds=False,
+            run_suffixes=run_suffixes,
+            run_descriptions=run_descriptions,
+            ignore_fieldlist_diffs=True,
+        )
 
     def _common_setup(self):
-        nl_contents_common = '''
+        nl_contents_common = """
             ! DIMCS test changes
-        '''
-        append_to_user_nl_files(caseroot = self._case.get_value("CASEROOT"),
-                                component = self.comp,
-                                contents = nl_contents_common)
+        """
+        append_to_user_nl_files(
+            caseroot=self._case.get_value("CASEROOT"),
+            component=self.comp,
+            contents=nl_contents_common,
+        )
 
     def _case_setup(self, i):
 
         # Second append user_nl change sepecific to case-i
-        append_to_user_nl_files(caseroot = self._case.get_value("CASEROOT"),
-                                component = self.comp,
-                                contents = nl_contents[i])
+        append_to_user_nl_files(
+            caseroot=self._case.get_value("CASEROOT"),
+            component=self.comp,
+            contents=nl_contents[i],
+        )
diff --git a/cime_config/SystemTests/dimcsl.py b/cime_config/SystemTests/dimcsl.py
@@ -27,7 +27,7 @@
     "Z_RESCALE_POWER = 11",
     "R_RESCALE_POWER = 11",
     "Q_RESCALE_POWER = 11",
-    ]
+]
 
 run_suffixes = [
     "base",
@@ -49,27 +49,36 @@
     "scale the dimension Q by 2**11",
 ]
 
+
 class DIMCSL(SystemTestsCompareN):
 
     def __init__(self, case):
         self.comp = case.get_value("COMP_OCN")
-        SystemTestsCompareN.__init__(self, case, N=len(nl_contents),
-                                     separate_builds = False,
-                                     run_suffixes = run_suffixes,
-                                     run_descriptions = run_descriptions,
-                                     ignore_fieldlist_diffs = True)
+        SystemTestsCompareN.__init__(
+            self,
+            case,
+            N=len(nl_contents),
+            separate_builds=False,
+            run_suffixes=run_suffixes,
+            run_descriptions=run_descriptions,
+            ignore_fieldlist_diffs=True,
+        )
 
     def _common_setup(self):
-        nl_contents_common = '''
+        nl_contents_common = """
             ! DIMCSL test changes
-        '''
-        append_to_user_nl_files(caseroot = self._case.get_value("CASEROOT"),
-                                component = self.comp,
-                                contents = nl_contents_common)
+        """
+        append_to_user_nl_files(
+            caseroot=self._case.get_value("CASEROOT"),
+            component=self.comp,
+            contents=nl_contents_common,
+        )
 
     def _case_setup(self, i):
 
         # Second append user_nl change sepecific to case-i
-        append_to_user_nl_files(caseroot = self._case.get_value("CASEROOT"),
-                                component = self.comp,
-                                contents = nl_contents[i])
+        append_to_user_nl_files(
+            caseroot=self._case.get_value("CASEROOT"),
+            component=self.comp,
+            contents=nl_contents[i],
+        )
diff --git a/standalone/build/build_examples-darwin.sh b/standalone/build/build_examples-darwin.sh
diff --git a/standalone/build/build_examples-ncar.sh → standalone/build/build_examples.sh b/standalone/build/build_examples-ncar.sh → standalone/build/build_examples.sh
@@ -15,8 +15,56 @@ CESM_ROOT=`pwd -P`
 SHR_ROOT=${CESM_ROOT}/share
 FMS_ROOT=${CESM_ROOT}/libraries/FMS
 
-COMPILER=intel
-TEMPLATE=${TEMPLATE_DIR}/ncar-${COMPILER}.mk
+# Default compiler
+COMPILER="intel"
+MACHINE="ncar"
+
+# Parse command line arguments
+while [[ "$#" -gt 0 ]]; do
+    case $1 in
+        --compiler) 
+            COMPILER="$2"
+            shift ;;
+        --machine) 
+            MACHINE="$2"
+            shift ;;
+        *) 
+            echo "Unknown parameter passed: $1"
+            echo "Usage: $0 [--compiler <compiler>] [--machine <machine>]"
+            exit 1 ;;
+    esac
+    shift
+done
+echo "Using compiler: $COMPILER"
+echo "Using machine: $MACHINE"
+
+TEMPLATE=${TEMPLATE_DIR}/${MACHINE}-${COMPILER}.mk
+
+# Throw error if template does not exist:
+if [ ! -f $TEMPLATE ]; then
+  echo "ERROR: Template file $TEMPLATE does not exist."
+    echo "Templates are based on the machine and compiler arguments: machine-compiler.mk. Available templates are:"
+    ls ${TEMPLATE_DIR}/*.mk
+    echo "Exiting."
+  exit 1
+fi
+
+# Set -j option based on the MACHINE argument
+case $MACHINE in
+    "homebrew" )
+        JOBS=2
+        ;;
+    "ubuntu" )
+        JOBS=4
+        ;;
+    "ncar")
+        JOBS=36
+        ;;
+    *)
+        echo "Invalid machine type for make -j option: $MACHINE"
+        exit 1
+        ;;
+esac
 
 if [ -e $1 ]; then
   BLD_ROOT=${COMPILER}
@@ -25,8 +73,6 @@ else
 fi
 
 # 1) Build FMS
-
-#    (b) build FMS
 cd ${INTERFACE_ROOT}/standalone/build
 mkdir -p ${BLD_ROOT}/FMS
 cd ${BLD_ROOT}/FMS
@@ -36,14 +82,14 @@ ${MKMF_ROOT}/list_paths ${FMS_ROOT}/src
 echo "${SHR_ROOT}/src/shr_kind_mod.F90" >> path_names
 echo "${SHR_ROOT}/src/shr_const_mod.F90" >> path_names
 ${MKMF_ROOT}/mkmf -t ${TEMPLATE} -p libfms.a -c "-Duse_libMPI -Duse_netCDF -DSPMD" path_names
-make -j36 NETCDF=3 REPRO=1 libfms.a
+make -j${JOBS} NETCDF=3 REPRO=1 libfms.a
 
 # 2) Build MOM6
 cd ${INTERFACE_ROOT}/standalone/build
 mkdir -p ${BLD_ROOT}/MOM6
 cd ${BLD_ROOT}/MOM6
 ${MKMF_ROOT}/list_paths -l ${MOM_ROOT}/{config_src/infra/FMS2,config_src/memory/dynamic_symmetric,config_src/drivers/solo_driver,../externals/MARBL/src,config_src/external,src/{*,*/*}}/
 ${MKMF_ROOT}/mkmf -t ${TEMPLATE} -o '-I../FMS' -p MOM6 -l '-L../FMS -lfms' -c '-Duse_libMPI -Duse_netCDF -DSPMD' path_names
-make -j36 NETCDF=3 REPRO=1 MOM6
+make -j${JOBS}  NETCDF=3 REPRO=1 MOM6
 
 echo "Finished build at `date`"
diff --git a/standalone/build/submit_casper_build b/standalone/build/submit_casper_build
@@ -19,4 +19,4 @@
 #PBS -m abe
 
 ### Run the executable
-./build_examples-ncar.sh intel-casper
+./build_examples.sh --compiler intel --machine ncar