Skip to content

Commit

Permalink
Make new README (#71)
Browse files Browse the repository at this point in the history
  • Loading branch information
@blimlim
blimlim authored Aug 23, 2024
1 parent cc0cce5 commit 7375ae6
Showing 1 changed file with 11 additions and 315 deletions.
326 changes: 11 additions & 315 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,319 +1,15 @@
# ACCESS-ESM with **payu**
# historical+concentrations
Standard configuration for a coupled CO~2~ concentration driven [ACCESS-ESM1.5](https://github.com/ACCESS-NRI/ACCESS-ESM1.5) under historical forcings between 1850-2014.

## Quickstart Guide
For usage instructions, see the [ACCESS-Hive docs](https://access-hive.org.au/models/run-a-model/run-access-esm/)

Get payu:
This configution is based on the CMIP6 configuration developed at [CSIRO](https://www.csiro.au/en/research/environmental-impacts/climate-change/climate-science-centre). Minor differences in ancillary files mean that bitwise reproducibility with the original CMIP6 simulation is not maintained.

module use /g/data3/hh5/public/modules
module load conda/analysis3-unstable
## Conditions of use

Create a directory in which to keep the model configurations:

mkdir -p ~/access-esm
cd ~/access-esm
git clone https://github.com/coecms/access-esm
cd access-esm
git checkout historical

Set up a warm start from a CSIRO run (see the script for details):

./warm-start.sh

Run the model:

payu run

Check the output:

ls archive/

The default configuration is a 1 year per model run. To run the model for, say, 25 years:

payu run -n 25

With default settings, 1 model year cost is ~ 1100 SU, with a walltime of 1 hour 20 minutes

**Note:**
We have noticed that some modules interfere with the git commands, for example `matlab/R2018a`.
If you are running into issues during the installation, it might be a good idea to `module purge` first before starting again.

## Warm Starts

The model is normally 'warm started' from the restart files of another
configuration. For instance the SSP experiments are started from the end of the
historical experiment, and in turn the historical experiment is started from
the pre-industrial control experiment (different ensemble members are created
by starting from different piControl years). Starting the experiment from
scratch requires a long period of spinup to ensure stability and should be
avoided if possible.

There are two options for restarting the model. It can be started from an
experiment run by CSIRO (requires membership in the p66 group), or it can be
started from another Payu experiment.

To perform a warm start, edit the file `warm-start.sh` to set the experiment
directory to start from and then run the script. For CSIRO jobs you must also
specify the date of the run to start from, for Payu jobs each restart directory
holds a different year.

## Understanding **payu**

**payu** was designed to help users of the NCI system run climate models.
It was initially created for MOM, but has been adapted for other models,
including coupled ones.

The aim of **payu** is to make it easy and intuitive to configure and run the models.

**payu** knows certain models and how to run them. Adding more models needs additions to the **payu** sources.
This will not be part of this document.

### Terms

To understand **payu**, it helps to distinguish certain terms:

- The **laboratory** is a directory where all parts of the model are kept.
It is typically in the user's short directory, usually at `/short/$PROJECT/$USER/<MODEL>`
- The **Control Directory** is the directory where the model configuration is
kept and from where the model is run.
- The **work** directory is where the model will actually be run.
It is typically a subdirectory of the Laboratory.
Submodels will have their own subdirectories in the work directory, named
after their name in the master configuration file.
It is ephemeral, that means payu will clean it up after the run.
- The **archive** directory is where **payu** puts all output files after each run.

The **work** and **archive** directories will be automatically created by **payu**.

### The master configuration file

In the Control Directory, the file `config.yaml` is the master control file.
Examples of what is configured in this file are:

- The actual model to run.
- Where to find the model binaries and configurations
- What resources to request from the scheduling system (PBS)
- Links to the laboratory
- Start date and run length per submission pf the model

The model configuration files are typically in subdirectories of the Control Directory,
the location of which is referenced in the master control file.
Since the models itself do need different ways to set up the model, the contents of these subdirectories will differ between different models.

## Understanding ACCESS-ESM

ACCESS (Australian Community Climate and Earth System Simulator) is a Coupled Climate Model.

The ESM 1.5 subversion of ACCESS specifically contains these models:

| Component | Model | Version |
| ---------- | ---------- | ------- |
| Atmosphere | UM-HG3 | 7.3 |
| Ocean | MOM | 5 |
| Sea Ice | CICE | 4.1 |
| Land | CABLE | 2.2.4 |
| Coupler | OASIS-MCT | 3.5 |

~~Pre-compiled executables for these models are available on raijin at
`/short/public/access-esm/payu/bin/csiro/`.~~

## Setting up ACCESS-ESM with **payu**

### The pre-conditions

On `gadi`, first make sure that you have access to our modules.
This can most easily been done by adding the line

module use /g/data3/hh5/public/modules

to your `~/.profile`, then logging back in. Then all you have to do is

module load conda/analysis3-unstable

to load the **payu** module.
Please check again after 7/2019 to see whether it has been made part of the stable conda module.

as **payu** will use git to keep track of all configuration changes automatically.

### Setting up the control directory

Create a directory in your home directory to keep all the Control Directories you might want.

mkdir ~/ACCESS-ESM
cd ~/ACCESS-ESM

Then clone the most recent version of the ACCESS-ESM control directory:

git clone https://github.com/coecms/esm-historical
cd esm-historical

(Note: Currently we only have the historical model set up, other versions will follow later.)

### Setting up the Master Configuration file.

Open the `config.yaml` file with your preferred text editor.

Let's have a closer look at the parts:

jobname: historical
queue: normal
walltime: 20:00:00

These are settings for the PBS system. Name, walltime and queue to use.

# note: if laboratory is relative path, it is relative to /short/$PROJECT/$USER
laboratory: access-esm

The location of the laboratory. At this point, **payu** can not expand shell environment variables (it's in our TO-DO), so as a work-around, if you use relative paths, it will be relative to your default short directory.

In this default configuration, it will be in `/short/$PROJECT/$USER/access-esm`.
But you can also hard-code the full path, if you want it somewhere different.

model: access

The main model. This mainly tells **payu** which driver to use. **payu** knows that **access** is a coupled model, so it will look for separate configurations of the submodels, which is the next item of the configuration file:

submodels:
- name: atmosphere
model: um
ncpus: 192
exe: /short/public/access-esm/payu/bin/csiro/um_hg3.exe-20190129_15
input:
- /short/public/access-esm/payu/input/historical/atmosphere

- name: ocean
model: mom
ncpus: 84
exe: /short/public/access-esm/payu/bin/coe/fms_ACCESS-CM.x
input:
- /short/public/access-esm/payu/input/common/ocean
- /short/public/access-esm/payu/input/historical/ocean

- name: ice
model: cice
ncpus: 12
exe: /short/public/access-esm/payu/bin/csiro/cice4.1_access-mct-12p-20180108
input:
- /short/public/access-esm/payu/input/common/ice

- name: coupler
model: oasis
ncpus: 0
input:
- /short/public/access-esm/payu/input/common/coupler

This is probably the meatiest part of the configuration, so let's look at it in more detail.

Each submodel has
- a **name**
- the **model** to know which driver to use
- the number of CPUs that this model should receive (**ncpus**)
- the location of the executable to use (**exe**)
- one or more locations for the **input** files.

The **name** is more than a useful reminder of what the model is.
**payu** expects this submodel's configuration files in a subdirectory with that name.

collate:
exe: /short/public/access-esm/payu/bin/mppnccombine
restart: true
mem: 4GB

Collation refers joining together of ocean diagnostics that are output at model runtime
in separate, tiled, files. In a process using minimal resources the output files are
joined back together. The restart files are typically also tiled in the same way. Here
the `restart: true` option means the restart files from the **previous** run are also
collated. This saves space and cuts down the number of files which makes more efficient
use of storage and better for archiving in the future.

restart: /short/public/access-esm/payu/restart/historical

This is the location of the warm restart files.
**payu** will use the restart files in there for the initial run.

calendar:
start:
year: 1850
month: 1
days: 1

runtime:
years: 1
months: 0
days: 0

Here is the start date, and the runtime **per run**.
The total time you want to model is `runtime` * `number of runs`

runspersub: 5

This `runspersub` feature is a nifty tool to allow you to bundle several runs into a single submission for the PBS queue.

Let's have an example: Say you told **payu** to make 7 runs with the above setting.
Each run would have a runtime of 1 year. So in the first submission it would run the model 5 times, to model years 101 through 105 respectively.

Then it would automatically resubmit another pbs job to model years 106 and 107, and then end.

### Setting up the Atmosphere Submodel

The **name** in `config.yaml` for the atmosphere submodel is "atmosphere", so the configuration of the UM will be in the `atmosphere` subdirectory.

ls atmosphere/
CNTLALL SIZES __pycache__ ftxx ihist prefix.CNTLATM
CONTCNTL STASHC cable.nml ftxx.new input_atm.nml prefix.CNTLGEN
INITHIS UAFILES_A errflag ftxx.vars namelists prefix.PRESM_A
PPCNTL UAFLDS_A exstat hnlist parexe um_env.py

There are many configuration files, but I want to note the `um_env.py`.
This file is used to set environment variables for the UM.
The UM driver of **payu** will look for this file and add these definitions to the environment when it runs the model.

### Setting up the Ocean Submodel

The **name** in `config.yaml` for the ocean submodel is "ocean", so the configuration
of MOM will be in the `ocean` subdirectory.

ls ocean
data_table diag_table field_table input.nml


### Setting up the Ice Submodel

The **name** in `config.yaml` for the ice submodel is "ice", so the configuration
of CICE will be in the `ice` subdirectory.

ls ice/
cice_in.nml input_ice.nml

## Running the Model

If you have set up the modules system to use the `/g/data3/hh5/public/modules` folder, a simple `module load conda/analysis3-unstable` should give you access to the **payu** system.

From the control directory, type

payu setup

This will prepare a the model run based on the configuration of the experiment.
It will setup `work` and `archive` directories and link to them from within the
configuration directory.
You don't have to do that, as the run command also sets it up, but it helps to check for errors.

payu sweep

This command removes the `work` directory again, but leaves the `archive`.

Finally,

payu run

will submit a single run to the queue.
It will start from the beginning (as indicated by the `start` section in the `config.yaml`) if it has not run before.

To automatically submit several runs (and to take advantage of the `runspersub` directive), you use the `-n` option:

payu run -n 7

## Finding the Output

The output is automatically copied to the `archive/outputXXX` directories.

**Warning**: This directory is a link to your laboratory (probably on scratch), so while it might *seem* that the output files are created twice, they are not. Deleting them from one location also removes them from the other. Do not do that if you want to keep the data.
The developers of ACCESS-ESM1.5 request that users of this model configuration:
1. Cite https://doi.org/10.1071/ES19035
2. Include an acknowledgment such as the following:
"The authors thank CSIRO for developing the ACCESS-ESM1.5 model configuration and making it freely available to researchers."
ACCESS-NRI requests users follow the guidelines for acknowledging ACCESS-NRI and include a statement such as:
"This research used the ACCESS-ESM1.5 model infrastructure provided by ACCESS-NRI, which is enabled by the Australian Government's National Collaborative Research Infrastructure Strategy (NCRIS)."

0 comments on commit 7375ae6

Please sign in to comment.