Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds MAM4xx microphysics interface #3013

Open
wants to merge 156 commits into
base: master
Choose a base branch
from

Conversation

odiazib
Copy link
Contributor

@odiazib odiazib commented Sep 25, 2024

Testing

We have included one SCREAM standalone test. Note that the baseline tests for these may fail because baselines are only created once the PR is merged.

mam4_aero_microphys process is added as an EAMxx process with the ability to turn it on/off using the label mam4_aero_microphys in the namelist_scream.xml file. The atmchange command to invoke this process in a CIME simulation is:

./atmchange physics::atm_procs_list="mac_aero_mic,rrtmgp,mam4_aero_microphys"

To run a CIME simulation, we have created a test modifier mam4xx-aero_microphysics:

cd cime/scripts

./create_test SMS_D_P32x1.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.<machine>_<compiler>.scream-mam4xx-aero_microphysics -t <test id> -p <project id> 

Where,
is the machine name
is the compiler to use on the machine
is a unique string test identifier
is the allocation project to charge

For ne30pg2 grid, use the following command:

./create_test SMS_D_P32x1.ne30pg2_oECv.F2010-SCREAMv1-MPASSI.<machine>_<compiler>.scream-mam4xx-aero_microphysics -t <test id> -p <project id> 

New input data

We have added 9 elevated emission files for each of the three grid resolutions:
ne2np4, ne4pg2, ne30pg2

For finer resolutions, the code will interpolate the ne30pg2 grid.

We have also added files for LINOZ, OXID, Chlorine, and photolysis.

All the files are on the E3SM input data server.

Standalone Test Timings (Compy):

(We will add the timings once the PR is approved)

Important notes:

  1. Gas washout rates and dry deposition process will be added as follow up PRs
  2. We must read solar data, fractional land use, and seasons files. We will add this in a follow-up PR.

@odiazib odiazib added AT: WIP Inform the autotester (AT) that the PR is a work in progress, and should not be tested Non-B4B Not bit for bit EAGLES Work related to the EAGLES aerosol project MAM AT: RETEST Force the autotester (AT) to retest the PR AT: PRE-TEST INSPECTED When pre-test inspection is required, set this label to pass the inspection labels Sep 25, 2024
@odiazib odiazib marked this pull request as ready for review September 25, 2024 18:37
@singhbalwinder singhbalwinder added AT: PRE-TEST INSPECTED When pre-test inspection is required, set this label to pass the inspection and removed AT: PRE-TEST INSPECTED When pre-test inspection is required, set this label to pass the inspection Non-B4B Not bit for bit AT: WIP Inform the autotester (AT) that the PR is a work in progress, and should not be tested labels Sep 25, 2024
@singhbalwinder singhbalwinder changed the title [WIP] mam microphysics Adds MAM4xx microphysics interface Sep 25, 2024
@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

@E3SM-Autotester E3SM-Autotester removed the AT: PRE-TEST INSPECTED When pre-test inspection is required, set this label to pass the inspection label Sep 25, 2024
@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
ERROR: USER HAS COMMITTED TO PR AFTER INSPECTION, INSPECTION IS INVALID! - This PR must be re-inspected by setting label 'AT: PRE-TEST INSPECTED'.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

@odiazib odiazib added the AT: PRE-TEST INSPECTED When pre-test inspection is required, set this label to pass the inspection label Sep 25, 2024
@E3SM-Autotester E3SM-Autotester removed the AT: PRE-TEST INSPECTED When pre-test inspection is required, set this label to pass the inspection label Sep 25, 2024
@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pre-Test Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED by label AT: PRE-TEST INSPECTED! Autotester is Removing Label; this inspection will remain valid until a new commit to source branch is performed.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 6078
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST;EAGLES;MAM
PULLREQUESTNUM 3013
SCREAM_SOURCE_REPO https://github.com/eagles-project/scream
SCREAM_SOURCE_SHA 88e82a8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 8aa209f
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5849
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST;EAGLES;MAM
PULLREQUESTNUM 3013
SCREAM_SOURCE_REPO https://github.com/eagles-project/scream
SCREAM_SOURCE_SHA 88e82a8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 8aa209f
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (eagles-project/scream)
  • Branch: oscar-balwinder-mingxuan/mam-microphysics
  • SHA: 88e82a8
  • Mode: TEST_REPO

Pull Request Author: odiazib

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 6078
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST;EAGLES;MAM
PULLREQUESTNUM 3013
SCREAM_SOURCE_REPO https://github.com/eagles-project/scream
SCREAM_SOURCE_SHA 88e82a8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 8aa209f
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5849
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST;EAGLES;MAM
PULLREQUESTNUM 3013
SCREAM_SOURCE_REPO https://github.com/eagles-project/scream
SCREAM_SOURCE_SHA 88e82a8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 8aa209f
TEST_REPO_ALIAS SCREAM
SCREAM_PullRequest_Autotester_Weaver # 6078 FAILED (click to see last 100 lines of console output)

/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6078/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma

/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6078/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma

/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6078/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma

[ 49%] Linking CXX static library libcld_fraction.a
[ 49%] Built target cld_fraction
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6078/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma

/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6078/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma

/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6078/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma

/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6078/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma

/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6078/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma

[ 49%] Linking CXX static library libspa.a
[ 49%] Built target spa
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6078/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma

/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6078/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma

[ 49%] Linking CXX static library libdiagnostics.a
[ 49%] Linking CXX static library libshoc_sk.a
[ 49%] Built target diagnostics
[ 49%] Built target shoc_sk
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6078/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma

[ 49%] Linking CXX static library libscream_io.a
[ 49%] Built target scream_io
[ 49%] Building CXX object src/share/CMakeFiles/scream_test_support.dir/util/scream_test_session.cpp.o
[ 49%] Building CXX object src/share/CMakeFiles/scream_test_support.dir/grid/mesh_free_grids_manager.cpp.o
[ 50%] Building CXX object src/control/CMakeFiles/scream_control.dir/atmosphere_driver.cpp.o
[ 50%] Building CXX object src/control/CMakeFiles/scream_control.dir/atmosphere_surface_coupling_importer.cpp.o
[ 50%] Building CXX object src/control/CMakeFiles/scream_control.dir/surface_coupling_utils.cpp.o
[ 50%] Building CXX object src/control/CMakeFiles/scream_control.dir/atmosphere_surface_coupling_exporter.cpp.o
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6078/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma

/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6078/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma

/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6078/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma

/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6078/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma

/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6078/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma

/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6078/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma

[ 51%] Linking CXX static library libnudging.a
[ 51%] Built target nudging
[ 51%] Linking CXX static library libp3_sk.a
gmake[1]: *** [CMakeFiles/Makefile2:4820: src/physics/mam/CMakeFiles/mam.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....
[ 51%] Built target p3_sk
[ 51%] Linking CXX static library libshoc.a
[ 51%] Built target shoc
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6078/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma

/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6078/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma

[ 51%] Linking CXX static library libp3.a
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6078/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma

[ 51%] Built target p3
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6078/scream/externals/ekat/extern/spdlog/include/spdlog/fmt/bundled/core.h(313): warning #1675-D: unrecognized GCC pragma

[ 51%] Linking CXX static library libscream_test_support.a
[ 51%] Built target scream_test_support
[ 51%] Linking CXX static library libscream_control.a
[ 51%] Built target scream_control
gmake: *** [Makefile:166: all] Error 2

Build type release failed at testing time. Here's a list of failed tests:
117:mam4_aero_microphys_standalone_baseline_cmp
150:homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp

Error(s) occurred during test phase
OVERALL STATUS: FAIL
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6078/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6078/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6078/scream/components/eamxx
weaver failed
######################################################
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/

./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins12726980739378571267.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: [email protected]
Finished: FAILURE

SCREAM_PullRequest_Autotester_Mappy # 5849 FAILED (click to see last 100 lines of console output)

For compatibility with older versions of CMake, option is clearing the
normal variable 'HAERO_ENABLE_MPI'.
This warning is for project developers. Use -Wno-dev to suppress it.

CMake Warning (dev) at /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5849/scream/externals/haero/CMakeLists.txt:12 (option):
Policy CMP0077 is not set: option() honors normal variables. Run "cmake
--help-policy CMP0077" for policy details. Use the cmake_policy command to
set the policy and suppress this warning.

For compatibility with older versions of CMake, option is clearing the
normal variable 'HAERO_ENABLE_TESTS'.
This warning is for project developers. Use -Wno-dev to suppress it.

-- Configuring with build type: Release
-- Building for CPU
-- Building Haero within another project.
-- Generating project files in build directory: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5849/scream/components/eamxx/ctest-build/release/externals/haero
-- C++ compiler is /projects/sems/install/rhel9-x86_64/sems/tpl/openmpi/4.1.6/gcc/11.4.0/base/neptmit/bin/mpicxx (GNU 11.4.0)
-- C compiler is /projects/sems/install/rhel9-x86_64/sems/tpl/openmpi/4.1.6/gcc/11.4.0/base/neptmit/bin/mpicc (GNU 11.4.0)
-- Configuring haero v0.1.0
-- Using double precision floating point numbers
-- Configuring on Linux.
-- Installation prefix is /usr/local
-- MPI is enabled
-- Configuring with build type: Release
-- Generating project files in build directory: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5849/scream/components/eamxx/ctest-build/release/externals/mam4xx
-- C++ compiler is /projects/sems/install/rhel9-x86_64/sems/tpl/openmpi/4.1.6/gcc/11.4.0/base/neptmit/bin/mpicxx (GNU 11.4.0)
-- C compiler is /projects/sems/install/rhel9-x86_64/sems/tpl/openmpi/4.1.6/gcc/11.4.0/base/neptmit/bin/mpicc (GNU 11.4.0)
-- Configuring MAM4xx v0.1.0
-- Using double precision floating point numbers
-- Installation prefix is /usr/local
CMake Error at cmake/ScreamUtils.cmake:212 (message):
Could not download
scream/mam4xx/linoz/Linoz_Chlorine_Loading_CMIP6_0003-2017_c20171114.nc,
output was: WARNING:root:wget failed with output: and errput File
/sems-data-store/ACME/inputdata/atm/scream/mam4xx/linoz/Linoz_Chlorine_Loading_CMIP6_0003-2017_c20171114.nc
already there; not retrieving.

Checking for file
atm/scream/mam4xx/linoz/Linoz_Chlorine_Loading_CMIP6_0003-2017_c20171114.nc
within /sems-data-store/ACME/inputdata

Input file /sems-data-store/ACME/inputdata/atm/scream/mam4xx/linoz/Linoz_Chlorine_Loading_CMIP6_0003-2017_c20171114.nc needs to be downloaded.
Attempting to download atm/scream/mam4xx/linoz/Linoz_Chlorine_Loading_CMIP6_0003-2017_c20171114.nc from https://web.lcrc.anl.gov/public/e3sm/inputdata/ with protocol wget
FAILED

Traceback (most recent call last):

File "/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5849/scream/components/eamxx/src/../scripts/check-input", line 45, in <module>
  _main_func(__doc__)
File "/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5849/scream/components/eamxx/src/../scripts/check-input", line 39, in _main_func
  success = check_input(**vars(parse_command_line(sys.argv, description)))
File "/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5849/scream/components/eamxx/scripts/check_input.py", line 55, in check_input
  success = download_file(input_root, the_file)
File "/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5849/scream/components/eamxx/scripts/check_input.py", line 23, in download_file
  from CIME.Servers import SVN

ImportError: cannot import name 'SVN' from 'CIME.Servers'
(/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5849/scream/components/eamxx/scripts/../../../cime/CIME/Servers/init.py)

Call Stack (most recent call first):
tests/single-process/mam/aero_microphys/CMakeLists.txt:44 (GetInputFile)

-- Configuring incomplete, errors occurred!

Error(s) occurred during test phase
OVERALL STATUS: FAIL
Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5849/scream/components/eamxx && source /projects/sems/modulefiles/utils/sems-modules-init.sh && module purge && module load sems-cmake/3.27.9 sems-git/2.42.0 sems-gcc/11.4.0 sems-openmpi-no-cuda/4.1.6 sems-netcdf-c/4.9.2 sems-netcdf-cxx/4.2 sems-netcdf-fortran/4.6.1 sems-parallel-netcdf/1.12.3 sems-openblas && export GATOR_INITIAL_MB=4000MB && export OMP_PROC_BIND=spread && true && ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy
RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5849/scream/components/eamxx && source /projects/sems/modulefiles/utils/sems-modules-init.sh && module purge && module load sems-cmake/3.27.9 sems-git/2.42.0 sems-gcc/11.4.0 sems-openmpi-no-cuda/4.1.6 sems-netcdf-c/4.9.2 sems-netcdf-cxx/4.2 sems-netcdf-fortran/4.6.1 sems-parallel-netcdf/1.12.3 sems-openblas && export GATOR_INITIAL_MB=4000MB && export OMP_PROC_BIND=spread && true && ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy
FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5849/scream/components/eamxx
mappy failed
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 1783998 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/

./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs

squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel

[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins2354579999857382900.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: [email protected]
Finished: FAILURE

@E3SM-Autotester E3SM-Autotester removed the AT: RETEST Force the autotester (AT) to retest the PR label Sep 26, 2024
@odiazib odiazib added AT: WIP Inform the autotester (AT) that the PR is a work in progress, and should not be tested AT: RETEST Force the autotester (AT) to retest the PR AT: PRE-TEST INSPECTED When pre-test inspection is required, set this label to pass the inspection and removed AT: WIP Inform the autotester (AT) that the PR is a work in progress, and should not be tested labels Sep 26, 2024
@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 6079
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST;AT: PRE-TEST INSPECTED;EAGLES;MAM
PULLREQUESTNUM 3013
SCREAM_SOURCE_REPO https://github.com/eagles-project/scream
SCREAM_SOURCE_SHA 88e82a8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 8aa209f
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5850
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST;AT: PRE-TEST INSPECTED;EAGLES;MAM
PULLREQUESTNUM 3013
SCREAM_SOURCE_REPO https://github.com/eagles-project/scream
SCREAM_SOURCE_SHA 88e82a8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 8aa209f
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (eagles-project/scream)
  • Branch: oscar-balwinder-mingxuan/mam-microphysics
  • SHA: 88e82a8
  • Mode: TEST_REPO

Pull Request Author: odiazib

@singhbalwinder singhbalwinder added AT: RETEST Force the autotester (AT) to retest the PR AT: PRE-TEST INSPECTED When pre-test inspection is required, set this label to pass the inspection labels Oct 14, 2024
@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

@E3SM-Autotester E3SM-Autotester removed the AT: PRE-TEST INSPECTED When pre-test inspection is required, set this label to pass the inspection label Oct 14, 2024
@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
ERROR: USER HAS COMMITTED TO PR AFTER INSPECTION, INSPECTION IS INVALID! - This PR must be re-inspected by setting label 'AT: PRE-TEST INSPECTED'.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

@singhbalwinder singhbalwinder added the AT: PRE-TEST INSPECTED When pre-test inspection is required, set this label to pass the inspection label Oct 15, 2024
@E3SM-Autotester E3SM-Autotester removed the AT: PRE-TEST INSPECTED When pre-test inspection is required, set this label to pass the inspection label Oct 15, 2024
@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pre-Test Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED by label AT: PRE-TEST INSPECTED! Autotester is Removing Label; this inspection will remain valid until a new commit to source branch is performed.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 6156
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST;EAGLES;MAM
PULLREQUESTNUM 3013
SCREAM_SOURCE_REPO https://github.com/eagles-project/scream
SCREAM_SOURCE_SHA 377df1a
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA c41c698
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5912
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST;EAGLES;MAM
PULLREQUESTNUM 3013
SCREAM_SOURCE_REPO https://github.com/eagles-project/scream
SCREAM_SOURCE_SHA 377df1a
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA c41c698
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (eagles-project/scream)
  • Branch: oscar-balwinder-mingxuan/mam-microphysics
  • SHA: 377df1a
  • Mode: TEST_REPO

Pull Request Author: odiazib

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 6156
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST;EAGLES;MAM
PULLREQUESTNUM 3013
SCREAM_SOURCE_REPO https://github.com/eagles-project/scream
SCREAM_SOURCE_SHA 377df1a
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA c41c698
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5912
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST;EAGLES;MAM
PULLREQUESTNUM 3013
SCREAM_SOURCE_REPO https://github.com/eagles-project/scream
SCREAM_SOURCE_SHA 377df1a
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA c41c698
TEST_REPO_ALIAS SCREAM
SCREAM_PullRequest_Autotester_Weaver # 6156 FAILED (click to see last 100 lines of console output)

/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/externals/ekat/extern/kokkos/core/src/Kokkos_Parallel.hpp:133:1: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  133 | inline void parallel_for(const std::string& str, const ExecPolicy& policy,
      | ^~~~~~~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/externals/ekat/extern/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Team.hpp: In constructor 'Kokkos::Impl::ParallelFor, Kokkos::Cuda>::ParallelFor(const FunctorType&, const Policy&) [with FunctorType = __nv_hdl_wrapper_t, void(const Kokkos::Impl::CudaTeamMember&), const Kokkos::View***, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >, const Kokkos::View**, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >, const double, Kokkos::View*, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> > >; Properties = {Kokkos::Cuda}]':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/externals/ekat/extern/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Team.hpp:533:1: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  533 |   ParallelFor(const FunctorType& arg_functor, const Policy& arg_policy)
      | ^ ~~~~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/externals/ekat/extern/kokkos/core/src/Kokkos_Parallel.hpp: In function 'void Kokkos::parallel_for(const string&, const ExecPolicy&, const FunctorType&) [with ExecPolicy = Kokkos::MDRangePolicy, Kokkos::IndexType >; FunctorType = __nv_hdl_wrapper_t >, int, int), &Homme::BoundaryExchange::register_field, 1>, void(int, int), Kokkos::View >**, Kokkos::LayoutRight, Kokkos::CudaSpace, Kokkos::MemoryTraits<8> >, int, Kokkos::View >, int>; Enable = void]':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/externals/ekat/extern/kokkos/core/src/Kokkos_Parallel.hpp:133:1: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  133 | inline void parallel_for(const std::string& str, const ExecPolicy& policy,
      | ^~~~~~~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/components/eamxx/src/dynamics/homme/eamxx_homme_iop.cpp: In static member function 'static void* __nv_hdl_wrapper_t::manager::do_copy(void*) [with Lambda = scream::HommeDynamics::apply_iop_forcing(scream::Real)::; Tag = __nv_dl_tag; OpFuncR = void; OpFuncArgs = {const Kokkos::Impl::CudaTeamMember&}; F1 = const int; F2 = ekat::WorkspaceManager, Kokkos::Device >; F3 = Kokkos::View, Kokkos::MemoryTraits<0> >; F4 = Kokkos::View****, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >; F5 = Kokkos::View****, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >; F6 = Kokkos::View*****, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >; F7 = Kokkos::View*****, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >; F8 = Kokkos::View****, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >; F9 = const int; F10 = const Kokkos::View, Kokkos::MemoryTraits<0> >; F11 = const double; F12 = const Kokkos::View, Kokkos::MemoryTraits<0> >; F13 = const Kokkos::View, Kokkos::MemoryTraits<0> >; F14 = const Kokkos::View, Kokkos::MemoryTraits<0> >; F15 = const int; F16 = const int]':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/components/eamxx/src/dynamics/homme/eamxx_homme_iop.cpp:415:1060: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  415 |   Kokkos::parallel_for("compute_qdp_and_vtheta_dp", policy_iop, KOKKOS_LAMBDA (const KT::MemberType& team) {
      |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    ^
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/components/eamxx/src/dynamics/homme/eamxx_homme_iop.cpp: In static member function 'static void* __nv_hdl_wrapper_t::manager::do_copy(void*) [with Lambda = scream::HommeDynamics::apply_iop_forcing(scream::Real)::; Tag = __nv_dl_tag; OpFuncR = void; OpFuncArgs = {const Kokkos::Impl::CudaTeamMember&}; F1 = const int; F2 = ekat::WorkspaceManager, Kokkos::Device >; F3 = Kokkos::View, Kokkos::MemoryTraits<0> >; F4 = Kokkos::View****, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >; F5 = Kokkos::View****, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >; F6 = Kokkos::View*****, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >; F7 = Kokkos::View*****, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >; F8 = Kokkos::View****, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >; F9 = const int; F10 = const Kokkos::View, Kokkos::MemoryTraits<0> >; F11 = const double; F12 = const Kokkos::View, Kokkos::MemoryTraits<0> >; F13 = const Kokkos::View, Kokkos::MemoryTraits<0> >; F14 = const Kokkos::View, Kokkos::MemoryTraits<0> >; F15 = const int; F16 = const int]':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/components/eamxx/src/dynamics/homme/eamxx_homme_iop.cpp:415:1060: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  415 |   Kokkos::parallel_for("compute_qdp_and_vtheta_dp", policy_iop, KOKKOS_LAMBDA (const KT::MemberType& team) {
      |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    ^
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/externals/ekat/extern/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Team.hpp: In constructor 'Kokkos::Impl::ParallelFor, Kokkos::Cuda>::ParallelFor(const FunctorType&, const Policy&) [with FunctorType = __nv_hdl_wrapper_t, void(const Kokkos::Impl::CudaTeamMember&), const Kokkos::View***, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >, const Kokkos::View**, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >, const double, Kokkos::View*, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> > >; Properties = {Kokkos::Cuda}]':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/externals/ekat/extern/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Team.hpp:533:1: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  533 |   ParallelFor(const FunctorType& arg_functor, const Policy& arg_policy)
      | ^ ~~~~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/components/eamxx/src/share/field/field_alloc_prop.hpp: In copy constructor 'scream::FieldAllocProp::FieldAllocProp(const scream::FieldAllocProp&)':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/components/eamxx/src/share/field/field_alloc_prop.hpp:106:1: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  106 |   FieldAllocProp (const FieldAllocProp&) = default;
      | ^ ~~~~~~~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/components/eamxx/src/share/field/field_alloc_prop.hpp: In copy constructor 'scream::FieldAllocProp::FieldAllocProp(const scream::FieldAllocProp&)':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/components/eamxx/src/share/field/field_alloc_prop.hpp:106:1: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  106 |   FieldAllocProp (const FieldAllocProp&) = default;
      | ^ ~~~~~~~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/externals/ekat/extern/kokkos/core/src/Kokkos_Parallel.hpp: In function 'void Kokkos::parallel_for(const string&, const ExecPolicy&, const FunctorType&) [with ExecPolicy = Kokkos::MDRangePolicy, Kokkos::IndexType >; FunctorType = __nv_hdl_wrapper_t >, int, int), &Homme::BoundaryExchange::register_field, 1>, void(int, int), Kokkos::View >**, Kokkos::LayoutRight, Kokkos::CudaSpace, Kokkos::MemoryTraits<8> >, int, Kokkos::View >, int>; Enable = void]':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/externals/ekat/extern/kokkos/core/src/Kokkos_Parallel.hpp:133:1: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  133 | inline void parallel_for(const std::string& str, const ExecPolicy& policy,
      | ^~~~~~~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/components/eamxx/src/share/field/field_alloc_prop.hpp: In copy constructor 'scream::FieldAllocProp::FieldAllocProp(const scream::FieldAllocProp&)':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/components/eamxx/src/share/field/field_alloc_prop.hpp:106:1: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  106 |   FieldAllocProp (const FieldAllocProp&) = default;
      | ^ ~~~~~~~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/components/eamxx/src/dynamics/homme/eamxx_homme_iop.cpp: In static member function 'static void* __nv_hdl_wrapper_t::manager::do_copy(void*) [with Lambda = scream::HommeDynamics::apply_iop_forcing(scream::Real)::; Tag = __nv_dl_tag; OpFuncR = void; OpFuncArgs = {const Kokkos::Impl::CudaTeamMember&}; F1 = const int; F2 = ekat::WorkspaceManager, Kokkos::Device >; F3 = Kokkos::View, Kokkos::MemoryTraits<0> >; F4 = Kokkos::View****, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >; F5 = Kokkos::View****, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >; F6 = Kokkos::View*****, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >; F7 = Kokkos::View*****, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >; F8 = Kokkos::View****, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >; F9 = const int; F10 = const Kokkos::View, Kokkos::MemoryTraits<0> >; F11 = const double; F12 = const Kokkos::View, Kokkos::MemoryTraits<0> >; F13 = const Kokkos::View, Kokkos::MemoryTraits<0> >; F14 = const Kokkos::View, Kokkos::MemoryTraits<0> >; F15 = const int; F16 = const int]':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/components/eamxx/src/dynamics/homme/eamxx_homme_iop.cpp:415:1060: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  415 |   Kokkos::parallel_for("compute_qdp_and_vtheta_dp", policy_iop, KOKKOS_LAMBDA (const KT::MemberType& team) {
      |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    ^
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/externals/ekat/extern/kokkos/core/src/KokkosExp_MDRangePolicy.hpp: In copy constructor 'Kokkos::MDRangePolicy, Kokkos::IndexType >::MDRangePolicy(const Kokkos::MDRangePolicy, Kokkos::IndexType >&)':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/externals/ekat/extern/kokkos/core/src/KokkosExp_MDRangePolicy.hpp:153:8: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  153 | struct MDRangePolicy : public Kokkos::Impl::PolicyTraits {
      |        ^~~~~~~~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/externals/ekat/extern/kokkos/core/src/KokkosExp_MDRangePolicy.hpp: In copy constructor 'Kokkos::MDRangePolicy, Kokkos::IndexType >::MDRangePolicy(const Kokkos::MDRangePolicy, Kokkos::IndexType >&)':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/externals/ekat/extern/kokkos/core/src/KokkosExp_MDRangePolicy.hpp:153:8: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  153 | struct MDRangePolicy : public Kokkos::Impl::PolicyTraits {
      |        ^~~~~~~~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/components/eamxx/src/dynamics/homme/eamxx_homme_process_interface.cpp: In static member function 'static void* __nv_hdl_wrapper_t::manager::do_copy(void*) [with Lambda = scream::HommeDynamics::initialize_homme_state()::; Tag = __nv_dl_tag; OpFuncR = void; OpFuncArgs = {int}; F1 = const int; F2 = const int; F3 = const int; F4 = const Kokkos::View, 1> >* [2][41][4][4][72], Kokkos::LayoutRight, Kokkos::CudaSpace, Kokkos::MemoryTraits<8> >; F5 = const int; F6 = const Kokkos::View, 1> >** [4][4][72], Kokkos::LayoutRight, Kokkos::CudaSpace, Kokkos::MemoryTraits<8> >; F7 = const Kokkos::View, 1> >* [3][4][4][72], Kokkos::LayoutRight, Kokkos::CudaSpace, Kokkos::MemoryTraits<8> >; F8 = const int]':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/components/eamxx/src/dynamics/homme/eamxx_homme_process_interface.cpp:1210:653: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
 1210 |   Kokkos::parallel_for(Kokkos::RangePolicy<>(0,nelem*qsize*NGP*NGP*npacks_mid),
      |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ^
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/components/eamxx/src/dynamics/homme/eamxx_homme_process_interface.cpp: In static member function 'static void* __nv_hdl_wrapper_t::manager::do_copy(void*) [with Lambda = scream::HommeDynamics::initialize_homme_state()::; Tag = __nv_dl_tag; OpFuncR = void; OpFuncArgs = {int}; F1 = const int; F2 = const int; F3 = const int; F4 = const Kokkos::View, 1> >* [2][10][4][4][128], Kokkos::LayoutRight, Kokkos::CudaSpace, Kokkos::MemoryTraits<8> >; F5 = const int; F6 = const Kokkos::View, 1> >** [4][4][128], Kokkos::LayoutRight, Kokkos::CudaSpace, Kokkos::MemoryTraits<8> >; F7 = const Kokkos::View, 1> >* [3][4][4][128], Kokkos::LayoutRight, Kokkos::CudaSpace, Kokkos::MemoryTraits<8> >; F8 = const int]':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/components/eamxx/src/dynamics/homme/eamxx_homme_process_interface.cpp:1210:653: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
 1210 |   Kokkos::parallel_for(Kokkos::RangePolicy<>(0,nelem*qsize*NGP*NGP*npacks_mid),
      |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ^
nvcc_internal_extended_lambda_implementation: In static member function 'static void* __nv_hdl_wrapper_t::manager::do_copy(void*) [with Lambda = scream::Field::update_impl(const scream::Field&, int, int, int)::; Tag = __nv_dl_tag; OpFuncR = void; OpFuncArgs = {int}; F1 = Kokkos::View, Kokkos::MemoryTraits<0> >; F2 = Kokkos::View, Kokkos::MemoryTraits<0> >; F3 = const int; F4 = const int; F5 = const int]':
nvcc_internal_extended_lambda_implementation:463:19: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/components/eamxx/src/dynamics/homme/eamxx_homme_process_interface.cpp: In static member function 'static void* __nv_hdl_wrapper_t::manager::do_copy(void*) [with Lambda = scream::HommeDynamics::initialize_homme_state()::; Tag = __nv_dl_tag; OpFuncR = void; OpFuncArgs = {int}; F1 = const int; F2 = const int; F3 = const int; F4 = const Kokkos::View, 1> >* [2][10][4][4][72], Kokkos::LayoutRight, Kokkos::CudaSpace, Kokkos::MemoryTraits<8> >; F5 = const int; F6 = const Kokkos::View, 1> >** [4][4][72], Kokkos::LayoutRight, Kokkos::CudaSpace, Kokkos::MemoryTraits<8> >; F7 = const Kokkos::View, 1> >* [3][4][4][72], Kokkos::LayoutRight, Kokkos::CudaSpace, Kokkos::MemoryTraits<8> >; F8 = const int]':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/components/eamxx/src/dynamics/homme/eamxx_homme_process_interface.cpp:1210:653: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
 1210 |   Kokkos::parallel_for(Kokkos::RangePolicy<>(0,nelem*qsize*NGP*NGP*npacks_mid),
      |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ^
[ 67%] Linking CXX static library libscream_test_support.a
[ 67%] Built target scream_test_support
[ 68%] Linking CXX static library libscream_rrtmgp.a
[ 68%] Built target scream_rrtmgp
[ 68%] Linking CXX static library libscream_theta-l_kokkos_4_72_41.a
[ 68%] Built target scream_theta-l_kokkos_4_72_41
[ 68%] Linking CXX static library libscream_theta-l_kokkos_4_128_10.a
[ 68%] Built target scream_theta-l_kokkos_4_128_10
[ 68%] Linking CXX static library libscream_control.a
[ 68%] Built target scream_control
[ 68%] Linking CXX static library libscream_theta-l_kokkos_4_72_10.a
[ 68%] Built target scream_theta-l_kokkos_4_72_10
gmake: *** [Makefile:166: all] Error 2

Error(s) occurred during test phase
OVERALL STATUS: FAIL
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6156/scream/components/eamxx
weaver failed
######################################################
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/

./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins14404344874739862976.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: [email protected]
Finished: FAILURE

SCREAM_PullRequest_Autotester_Mappy # 5912 FAILED (click to see last 100 lines of console output)

367:mam4_drydep_standalone_baseline_cmp
368:mam4_wetscav_standalone_np1
369:mam4_wetscav_standalone_np2
370:mam4_wetscav_standalone_np3
371:mam4_wetscav_standalone_np4
372:mam4_wetscav_standalone_np2_vs_np1
373:mam4_wetscav_standalone_np3_vs_np1
374:mam4_wetscav_standalone_np4_vs_np1
375:mam4_wetscav_standalone_baseline_cmp
399:mam4_aero_microphys_standalone_baseline_cmp
456:p3_mam4_wetscav_np1
457:p3_mam4_wetscav_np2
458:p3_mam4_wetscav_np3
459:p3_mam4_wetscav_np4
460:p3_mam4_wetscav_np2_vs_np1
461:p3_mam4_wetscav_np3_vs_np1
462:p3_mam4_wetscav_np4_vs_np1
463:shoc_cldfrac_p3_wetscav_np1
464:shoc_cldfrac_p3_wetscav_np2
465:shoc_cldfrac_p3_wetscav_np3
466:shoc_cldfrac_p3_wetscav_np4
467:shoc_cldfrac_p3_wetscav_np2_vs_np1
468:shoc_cldfrac_p3_wetscav_np3_vs_np1
469:shoc_cldfrac_p3_wetscav_np4_vs_np1
550:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_np1
551:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_np2
552:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_np3
553:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_np4
554:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_np2_vs_np1
555:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_np3_vs_np1
556:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_np4_vs_np1
557:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp

Error(s) occurred during test phase
OVERALL STATUS: FAIL
Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5912/scream/components/eamxx && source /projects/sems/modulefiles/utils/sems-modules-init.sh && module purge && module load sems-cmake/3.27.9 sems-git/2.42.0 sems-gcc/11.4.0 sems-openmpi-no-cuda/4.1.6 sems-netcdf-c/4.9.2 sems-netcdf-cxx/4.2 sems-netcdf-fortran/4.6.1 sems-parallel-netcdf/1.12.3 sems-openblas && export GATOR_INITIAL_MB=4000MB && export OMP_PROC_BIND=spread && true && ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy
RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5912/scream/components/eamxx && source /projects/sems/modulefiles/utils/sems-modules-init.sh && module purge && module load sems-cmake/3.27.9 sems-git/2.42.0 sems-gcc/11.4.0 sems-openmpi-no-cuda/4.1.6 sems-netcdf-c/4.9.2 sems-netcdf-cxx/4.2 sems-netcdf-fortran/4.6.1 sems-parallel-netcdf/1.12.3 sems-openblas && export GATOR_INITIAL_MB=4000MB && export OMP_PROC_BIND=spread && true && ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy
FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5912/scream/components/eamxx
mappy failed
SCREAM V1 TESTING FAILED!
Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
Case dir: /home/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20241014_212110_zushnh
PASS ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 RUN
Case dir: /home/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20241014_212110_zushnh
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 RUN
Case dir: /home/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20241014_212110_zushnh
PASS ERS_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-small_kernels--scream-output-preset-5 RUN
Case dir: /home/e3sm-jenkins/acme/scratch/ERS_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-small_kernels--scream-output-preset-5.C.20241014_212110_zushnh
PASS ERS_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-small_kernels_p3--scream-output-preset-5 RUN
Case dir: /home/e3sm-jenkins/acme/scratch/ERS_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-small_kernels_p3--scream-output-preset-5.C.20241014_212110_zushnh
PASS ERS_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-small_kernels_shoc--scream-output-preset-5 RUN
Case dir: /home/e3sm-jenkins/acme/scratch/ERS_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-small_kernels_shoc--scream-output-preset-5.C.20241014_212110_zushnh
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 RUN
Case dir: /home/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20241014_212110_zushnh
PASS ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-arm97 RUN
Case dir: /home/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-arm97.C.20241014_212110_zushnh
PASS ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-comble RUN
Case dir: /home/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-comble.C.20241014_212110_zushnh
PASS ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-dycomsrf01 RUN
Case dir: /home/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-dycomsrf01.C.20241014_212110_zushnh
PASS ERS_P16_Ln22.ne30pg2_ne30pg2.FRCE-SCREAMv1-DP.mappy_gnu RUN
Case dir: /home/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30pg2_ne30pg2.FRCE-SCREAMv1-DP.mappy_gnu.C.20241014_212110_zushnh
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
Case dir: /home/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20241014_212110_zushnh
DIFF SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci (phase BASELINE)
Case dir: /home/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci.C.20241014_212110_zushnh
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-drydep RUN
Case dir: /home/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-drydep.C.20241014_212110_zushnh
DIFF SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics (phase BASELINE)
Case dir: /home/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20241014_212110_zushnh
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav RUN
Case dir: /home/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav.C.20241014_212110_zushnh
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 RUN
Case dir: /home/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20241014_212110_zushnh
test-scheduler took 1954.896880865097 seconds
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 2603648 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/

./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs

squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel

[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins10103197233685375550.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: [email protected]
Finished: FAILURE

@E3SM-Autotester E3SM-Autotester removed the AT: RETEST Force the autotester (AT) to retest the PR label Oct 15, 2024
Copy link
Contributor

@mahf708 mahf708 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this PR has some unrelated edits that could've bene excised to make the PR review process easier. Evidently, the PR isn't really ready. For example, the PR can't be ready with an edit in share/io, that will have to be reverted.

I have written down some comments; I don't think you need to address all of them right now. Things become tedious when a file goes above a few 100s lines; the main addition in this PR is 2356 lines; the (likely misleadingly named) "helper micro" file is 722 lines.

I will review again soon

<mam4_num_a2_verti_emiss_file_name hgrid="ne4np4.pg2" type="file" doc="">${DIN_LOC_ROOT}/atm/scream/mam4xx/emissions/ne4pg2/elevated/cmip6_mam4_num_a2_elev_1x1_2010_clim_ne4pg2_c20241008.nc </mam4_num_a2_verti_emiss_file_name>
<mam4_num_a4_verti_emiss_file_name hgrid="ne4np4.pg2" type="file" doc="">${DIN_LOC_ROOT}/atm/scream/mam4xx/emissions/ne4pg2/elevated/cmip6_mam4_num_a4_elev_1x1_2010_clim_ne4pg2_c20241008.nc </mam4_num_a4_verti_emiss_file_name>
<mam4_soag_verti_emiss_file_name hgrid="ne4np4.pg2" type="file" doc="">${DIN_LOC_ROOT}/atm/scream/mam4xx/emissions/ne4pg2/elevated/cmip6_mam4_soag_elev_1x1_2010_clim_ne4pg2_c20241008.nc </mam4_soag_verti_emiss_file_name>
</mam4_aero_microphys>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

most added files have space at the end of their names, e.g., ...0241008.nc </ma...). Not sure if that matters, but just letting you know

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for noticing that. It didn't matter in our tests so far, but we will remove it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is irrelevant. I think we strip spaces (and newline chars) when we parse the xml, to avoid errors/surprises. So feel free to use the formatting that makes it easier to visually parse the file.

@@ -42,6 +42,7 @@ add_subdirectory(${EXTERNALS_SOURCE_DIR}/mam4xx ${CMAKE_BINARY_DIR}/externals/ma
# EAMxx mam4xx-based atmospheric processes
add_library(mam
eamxx_mam_microphysics_process_interface.cpp
${SCREAM_BASE_DIR}/src/physics/rrtmgp/shr_orb_mod_c2f.F90
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would be good to explain why this is needed, especially that it is a F90 file. Don't we have this already in cpp?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We followed RRTMGP (see here)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I don't think we converted this.

@@ -54,7 +55,7 @@ target_include_directories(mam PUBLIC
${EXTERNALS_SOURCE_DIR}/haero
${EXTERNALS_SOURCE_DIR}/mam4xx/src
)
target_link_libraries(mam PUBLIC physics_share scream_share mam4xx haero)
target_link_libraries(mam PUBLIC physics_share csm_share scream_share mam4xx haero)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An example of something that could've been handled by another PR to make this PR more focused and leaner.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, we followed RRTMGP here for now.


auto nondim = ekat::units::Units::nondimensional();
constexpr auto nondim = ekat::units::Units::nondimensional();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we usually do const auto for these, not sure which is better. I personally try to steer clear from constexpr.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C++ folks can chime in. I like constexpr if it is supported as it makes the code more readable. Seeing constexpr tells me that this is a compile-time constant and is not set during the runtime.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either way is fine.

FieldLayout scalar4d_layout_mid =
make_layout({ncol_, num_aero_modes, nlev_}, {"COL", "NMODES", "LEV"});
// Number of modes
constexpr int nmodes = mam4::AeroConfig::num_modes();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

constexpr again, luca may better comment on this

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above. constexpr indicates a compile-time constant.

// partially cloudy grid cells, hence both fclea and fcldy are assumed to
// be nonzero.

constexpr Real eps = 1.e-35; // BAD CONSTANT
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah... why not use one of those max limits things?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, we are obsessed with keeping the answers as similar as we can. With all this code, we are still able to match very closely to double-precision answers from the Fortran codebase! We always add BAD CONSTANTS comment so that we can revisit these later. We may use numeric limits here from C++ in the future.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was an absolute doozy. High-level recommendations:

  • use more human readable variable names
  • split stuff out of this file. Could be nice to have subfiles with all these different routines; from a quick glance, it doesn't loook like they all need to be in this very file

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We intend to move this file to MAM4xx repo. @odiazib asked about it today itself! We will move these codes so reviewers don't need to review them. We are working on some other important tasks, so it might be an additional PR to move it to MAM4xx.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

helper micro may be an inaccurate name for this file; more like tracer data utils? Either way, I think we should refactor this and move it to somewhere in share in the future; I suspect spa, radiation, and share common needs when it comes to these tracers

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's an excellent point. We are borrowing SPA codes for reading files. In my next PR for reading fractional land use, I am working on creating a common framework so that we don't repeat the code. Since we are short on time, I am planning to create this common framework slowly, piece by piece. So you will see a small step towards this in my next PR.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, unrelated, but I was recently looking at the tracers code in EAM F90. If you don't wanna bother with this in EAMxx, it may be a bit easier to house it in MAM4xx. I am not so sure though ... will keep thinking

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the ability to read files should be an EAMxx capability. We should have an epic on it to make it easy and consistent. Tracer data codes in EAM have evolved over the years and can handle a lot of different scenarios. It would be great to have those capabilities in EAMxx.

Copy link
Contributor Author

@odiazib odiazib Oct 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We decided early in the porting process that IO would be handled by EAMxx. The part of the code relies on the scream-scorpio-interface and the AtmosphereInput class. We do not have access to these features in MAM4xx. I tried to create code that can handle many types of NC files, similar to the code in EAM F90, but I did not want to create a direct port. Instead, I implemented code for three types of files that are currently needed in microphysics.

Indeed, we must find a better name for this file. Thanks for pointing this out.

Looking forward to seeing comments/recommendations from @bartgol, @tcclevenger, @AaronDonahue, who have more experience on this topic.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is a task that has been on my todo list for a while. I want a common infrastructure for reading time series from file (possibly doing vertical/horizontal interpolation) and interpolating to the current time stamp. Right now, we have a few places that implement the same thing: spa, iop, mam, and nudging. They all have slightly different assumptions (e.g. regarding whether the input data can contain fillValue kind of data or not), but there's no reason why we should not be able to handle it internally.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@odiazib I will find a time to review early next week.

Comment on lines +7 to +8
// The following variables are chemistry mechanism dependent! See
// mam4xx/src/mam4xx/gas_chem_mechanism.hpp)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'm hoping one day we can fix these names

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file will also be moved to MAM4xx. We may move everything in impl directory to MAM4xx.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we add dosc under components/eamxx/docs

@susburrows
Copy link
Contributor

susburrows commented Oct 17, 2024

@singhbalwinder - thanks for letting us know that the PR is ready to be reviewed! @mahf708 and @kaizhangpnl have already begun reviewing the code. When you have a chance, could you add the results from key process-level evaluations performed by the EAGLES eval team in the PR description so that those results are also documented?

@bartgol and @AaronDonahue - can you help with the technical review (and/or assign someone appropriate)? Since this PR is quite large, but we also want to help expedite the review process, I'm also wondering if we may need to touch base again about the level of review needed here. For example, Kai estimates he would need another two weeks of concentrated effort to fully review all of the code this PR; however, the initial results of physical tests that he went over with me today all look good in terms of the expected behaviors being reproduced. Perhaps we can consider doing a "light" review at this stage to enable Balwinder and his team to get the code in place, and then a more thorough review in the immediate future?

@singhbalwinder
Copy link
Contributor

Evidently, the PR isn't really ready. For example, the PR can't be ready with an edit in share/io, that will have to be reverted.

Thanks, Naser, for the quick review and for bringing this up. We followed RRTMGP (see here) for this function. We also would like to implement a C++ solution and will reimplement this function once it becomes available. We should revisit it after merging this PR to fix it both in this process and RRTMGP. We are open to implementing if there exists a better solution at this point. I will address the rest of your comments inline.

@singhbalwinder
Copy link
Contributor

When you have a chance, could you add the results from key process-level evaluations performed by the EAGLES eval team in the PR description so that those results are also documented?

Thanks @susburrows. I will update the PR description with the evaluation results and the rest of the required info.

Since this PR is quite large, but we also want to help expedite the review process,

I agree this PR is quite large, and I truly appreciate and am thankful for the effort and time that goes into reviewing this PR. I am listing some routines below that the reviewers can skip if they want to. These will go into the MAM4xx repo in the future.
components/eamxx/src/physics/mam/impl/compute_o3_column_density.cpp
components/eamxx/src/physics/mam/impl/gas_phase_chemistry.cpp
components/eamxx/src/physics/mam/impl/mam4_amicphys.cpp

We just changed the layout for a variable in the following file along with declaring some vars constant in the following file:
components/eamxx/src/physics/mam/eamxx_mam_aci_process_interface.cpp
Changed just a variable name to be consistent in the following file:
components/eamxx/src/physics/mam/eamxx_mam_wetscav_process_interface.cpp

I can give a code walk-through as well if that is helpful.

@mahf708
Copy link
Contributor

mahf708 commented Oct 18, 2024

Evidently, the PR isn't really ready. For example, the PR can't be ready with an edit in share/io, that will have to be reverted.

Thanks, Naser, for the quick review and for bringing this up. We followed RRTMGP (see here) for this function.

Thanks for the clarification re the shr_orb_mod_c2f piece. In the above line, I actually meant the share/io one in components/eamxx/src/share/io/scream_scorpio_interface.cpp.

@singhbalwinder
Copy link
Contributor

I actually meant the share/io one

That is an easy one to fix! I will remove it in my next set of changes.

@tcclevenger tcclevenger self-requested a review October 18, 2024 15:59
Copy link
Contributor

@bartgol bartgol left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have some suggestions.

On a side note: this PR is way too large for reviewing it. I skimmed through it, cause I can't spend time to understand if what it does is likely to be what was intended, as it would take me a week.

I also don't see any unit test for mam4xx, and there are tons of individual functions. The system tests in eamxx/tests are of course great, but it's hardly better than a full CIME tests. When things break, we have no way to narrow down which part of mam4xx is responsible.

I know you guys are on a schedule, but please consider:

  • adding unit tests for all functions in your impl folder
  • break PR into smaller ones

I know the latter may be hard, but I would take even a tiny PR that simply adds utility fcns as an intermediate step, to reduce the number of lines to review in the main PR. With PRs this size, you can't expect a great quality of the review, as the attention span falls drastically after 500 lines or so.

@@ -249,6 +249,55 @@ be lost if SCREAM_HACK_XML is not enabled.
<cldFraction inherit="atm_proc_base"/>

<!-- MAM4xx namelist options -->

<!-- MAM4xx-Aerosol-Microphysics -->
<mam4_aero_microphys inherit="atm_proc_base">
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any change you can add some doc strings for these parameters?

<!--Vertical emissions-->
<verti_emiss_ymd type="integer"> 20100101 </verti_emiss_ymd>
<!-- For all other grids -->
<mam4_so2_verti_emiss_file_name type="file" doc="">${DIN_LOC_ROOT}/atm/scream/mam4xx/emissions/ne30pg2/elevated/cmip6_mam4_so2_elev_1x1_2010_clim_ne30pg2_c20241008.nc </mam4_so2_verti_emiss_file_name>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you planning to use ne30 data for all runs, including ne4 runs? Currently, the coarsening/remapping infrastructure assumes that the fine grid is the model grid, so you can only refine input data (and coarsen outputs). I've been thinking for a while to extend horiz remap to allow reading in data from a finer grid, but it was never really needed. If you do need it though, we can try to bump up the priority.

<mam4_num_a2_verti_emiss_file_name hgrid="ne4np4.pg2" type="file" doc="">${DIN_LOC_ROOT}/atm/scream/mam4xx/emissions/ne4pg2/elevated/cmip6_mam4_num_a2_elev_1x1_2010_clim_ne4pg2_c20241008.nc </mam4_num_a2_verti_emiss_file_name>
<mam4_num_a4_verti_emiss_file_name hgrid="ne4np4.pg2" type="file" doc="">${DIN_LOC_ROOT}/atm/scream/mam4xx/emissions/ne4pg2/elevated/cmip6_mam4_num_a4_elev_1x1_2010_clim_ne4pg2_c20241008.nc </mam4_num_a4_verti_emiss_file_name>
<mam4_soag_verti_emiss_file_name hgrid="ne4np4.pg2" type="file" doc="">${DIN_LOC_ROOT}/atm/scream/mam4xx/emissions/ne4pg2/elevated/cmip6_mam4_soag_elev_1x1_2010_clim_ne4pg2_c20241008.nc </mam4_soag_verti_emiss_file_name>
</mam4_aero_microphys>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is irrelevant. I think we strip spaces (and newline chars) when we parse the xml, to avoid errors/surprises. So feel free to use the formatting that makes it easier to visually parse the file.

@@ -42,6 +42,7 @@ add_subdirectory(${EXTERNALS_SOURCE_DIR}/mam4xx ${CMAKE_BINARY_DIR}/externals/ma
# EAMxx mam4xx-based atmospheric processes
add_library(mam
eamxx_mam_microphysics_process_interface.cpp
${SCREAM_BASE_DIR}/src/physics/rrtmgp/shr_orb_mod_c2f.F90
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I don't think we converted this.

// mid points
FieldLayout scalar3d_layout_mid{{COL, LEV}, {ncol_, nlev_}};
// Layout for 2D (2d horiz) variable
const FieldLayout scalar2d = grid_->get_2d_scalar_layout();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making this change. Since new processes are often started copy/pasting existing ones, it's good to show the best practices. Here, I strongly encourage to use the grid interfaces to retrieve "standard" layouts.

} // iaer
} // iscldy_subarea

Real qgas_delaa[max_gas()][nqtendaa()] = {};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For clarity: the ={} part initializes all entries to 0. If you don't need to init to 0, consider removing that part.

// calculates qaerwat using the grid-cell mean interstital-aerosol
// mix-rates and the clear-area RH. aerosol water mixing ratios (mol/mol)
Real qaerwatsub3[num_modes][maxsubarea()];
assign_2d_array(num_modes, maxsubarea(), 0.0, // in
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can add = {0} above, and do away with this call. It's likely to be faster too.

Real dgn_a[num_modes] = {};
Real dgn_awet[num_modes] = {};
Real wetdens[num_modes] = {};
assign_1d_array(num_modes, 0.0, // in
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pointless assignments.

// out
qgcm_tendaa, qqcwgcm_tendaa);

#if 0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's not add code to master that has #if 0. If you don't need it, remove it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you want to use it in debug mode, use more descriptive macros. E.g.,

// For developers. uncomment this to trigger extra debug.
//#define MAM4_EXTRA_DEBUG

// later
#ifdef MAM4_EXTRA_DEBUG
...
#endif

EKAT_REQUIRE_MSG (it.second.num_customers==0,
"Error! ScorpioSession::finalize called, but a file is still in use elsewhere.\n"
" - filename: " + it.first + "\n");
}
}*/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can add an extra arg to finalize_subsystem:

void finalize_subsystem (bool inside_exception_handling) {
   ...
   for (auto& it : s.files) {
      if (it.second.num_customers!=0){
        if (inside_exception_handling) {
          // just print a message
        } else {
          EKAT_ERROR_MSG (...);
        }
      }
   }
}

We may have to also pass this bool to AtmosphereDriver::finalize: it's false in general, but true when inside here.

Copy link
Contributor

@kaizhangpnl kaizhangpnl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this PR! It is a huge effort. This PR includes code changes not only in the MAMxx/EAMxx interface but also in the MAMxx repo (see below). I am a beginner in C++, so it will take me quite some time to carefully review all the code changes. Given the time sensitivity, I fully support the idea for a "light" review at this moment, but I hope by Dec we can add additional diagnostics of budget terms/process rates in subsequent PRs, so that it's easier to catch any potential bug/problem that might exist.

I am testing the code on Chrysalis and will check if the results are consistent with those obtained from other colleagues. Once it's done, I would update the review checklist.

Code changes in MAMxx repo:

https://github.com/eagles-project/mam4xx/tree/main/src/mam4xx/aero_model_emissions.hpp
https://github.com/eagles-project/mam4xx/tree/main/src/mam4xx/aero_modes.hpp
https://github.com/eagles-project/mam4xx/tree/main/src/mam4xx/aging.hpp
https://github.com/eagles-project/mam4xx/tree/main/src/mam4xx/coagulation.hpp
https://github.com/eagles-project/mam4xx/tree/main/src/mam4xx/conversions.hpp
https://github.com/eagles-project/mam4xx/tree/main/src/mam4xx/gas_chem.hpp
https://github.com/eagles-project/mam4xx/tree/main/src/mam4xx/gas_chem_mechanism.hpp
https://github.com/eagles-project/mam4xx/tree/main/src/mam4xx/gasaerexch.hpp
https://github.com/eagles-project/mam4xx/tree/main/src/mam4xx/gasaerexch_soaexch.hpp
https://github.com/eagles-project/mam4xx/tree/main/src/mam4xx/lin_strat_chem.hpp
https://github.com/eagles-project/mam4xx/tree/main/src/mam4xx/mam4.hpp
https://github.com/eagles-project/mam4xx/tree/main/src/mam4xx/mo_drydep.hpp
https://github.com/eagles-project/mam4xx/tree/main/src/mam4xx/mo_photo.hpp
https://github.com/eagles-project/mam4xx/tree/main/src/mam4xx/mo_setext.hpp
https://github.com/eagles-project/mam4xx/tree/main/src/mam4xx/mo_setinv.hpp
https://github.com/eagles-project/mam4xx/tree/main/src/mam4xx/mo_setsox.hpp
https://github.com/eagles-project/mam4xx/tree/main/src/mam4xx/nucleation.hpp
https://github.com/eagles-project/mam4xx/tree/main/src/mam4xx/rename.hpp
https://github.com/eagles-project/mam4xx/tree/main/src/mam4xx/seq_drydep.hpp
https://github.com/eagles-project/mam4xx/tree/main/src/mam4xx/vertical_interpolation.hpp


const auto extfrc_icol = ekat::subview(extfrc, icol);

mam4::mo_setext::extfrc_set(forcings_in, extfrc_icol);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be nice to add a comment here and say if we want to turn off the elevated emissions, then comment out this line.

// aqueous chemistry ...
const Real mbar = haero::Constants::molec_weight_dry_air;
constexpr int indexm = mam4::gas_chemistry::indexm;
mam4::mo_setsox::setsox_single_level(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be nice to add a note here and say if we want to turn off the aqueous phase chemistry, comment out this line.

<dgnum type="real" doc="Dry aerosol particles diameter [m]">2.6e-08</dgnum>
<dgnumwet type="real" doc="Wet aerosol particles diameter [m]">0.41417721820867320E-007</dgnumwet>
<wetdens type="real" doc="Wet density of interstitial aerosol [kg/m3]">0.15100083211582764E+004</wetdens>
<dgnum type="array(real)" doc="Dry aerosol particles diameter [m]">1.37146e-07 ,3.45899e-08 ,1.00000e-06 ,9.99601e-08</dgnum>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these initial values set for the 3D arrays? I suppose they will be modified later?

// Layout for 3D (2d horiz X 1d vertical) variable defined at mid-level and
// interfaces
const FieldLayout scalar3d_mid = grid_->get_3d_scalar_layout(true);
const FieldLayout scalar3d_int = grid_->get_3d_scalar_layout(false);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does "(true) or (false)" here mean whether this field is defined at layer center?

config_.amicphys.gaexch_h2so4_uptake_optaa = 2;
config_.amicphys.newnuc_h2so4_conc_optaa = 2;
using namespace ekat::units;
constexpr auto q_unit = kg / kg; // units of mass mixing ratios of tracers
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you confirm the unit for mixing ratios of trace gases (mol/mol or kg/kg)?

grid_->get_3d_vector_layout(true, nmodes, "nmodes");

// Geometric mean dry diameter for number distribution [m]
add_field<Required>("dgnum", scalar3d_mid_nmodes, m, grid_name);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "m" here looks the same as the mode index "m" at line 191? Could it cause any problem?

std::string MAMMicrophysics::name() const {
return "mam4_micro";
}
void MAMMicrophysics::set_namelist_params_() {
Copy link
Contributor

@kaizhangpnl kaizhangpnl Oct 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will the parameters be correctly set for all processors? In the Fortran code we need to broadcast them.

// these parameters guide the coupling between parameterizations
// NOTE: mam4xx was ported with these parameters fixed, so it's probably not
// NOTE: safe to change these without code modifications.
config_.amicphys.gaexch_h2so4_uptake_optaa = 2;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to include the original description for these parameters.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
EAGLES Work related to the EAGLES aerosol project MAM
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants