Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding new environment files for stability #260

Merged
merged 5 commits into from
Jan 25, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ jobs:
strategy:
matrix:
os: [ubuntu-latest] # , macos-latest]
python-version: ['3.9']
python-version: ['3.12']

steps:
- name: Checkout source
Expand All @@ -46,7 +46,7 @@ jobs:
uses: conda-incubator/setup-miniconda@v2
with:
python-version: ${{ matrix.python-version }}
environment-file: ci/environment-py${{ matrix.python-version }}.yml
environment-file: ci/environment-py${{ matrix.python-version }}_frozen.yml
miniforge-variant: Mambaforge
miniforge-version: "latest"
use-mamba: true
Expand Down
38 changes: 20 additions & 18 deletions ReadMe.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,10 +78,10 @@ $ cd Isca

3. **Create a conda environment**

Requirements for Isca can be installed via the .yml file included with the model in `Isca/ci/environment-py3.9.yml`
Requirements for Isca can be installed via the .yml file included with the model in `Isca/ci/environment-py3.12_frozen.yml`
Navigate to the downloaded Isca folder, and create a conda environment `isca_env` containing the required packages using:
```{bash}
$ conda env create -f ci/environment-py3.9.yml
$ conda env create -f ci/environment-py3.12_frozen.yml
```
Then activate the environment; you'll need to do this each time you launch a new bash session.
```{bash}
Expand All @@ -100,33 +100,35 @@ Successfully installed Isca

### Compiling for the first time

At Exeter University, Isca is compiled using:

* Intel Compiler Suite 14.0
* OpenMPI 10.0.1
* NetCDF 4.3.3.1
* git 2.1.2
By installing Isca using the `ci/environment-py3.12_frozen.yml` file, you will have installed everything you need to run Isca, including the gfortran compiler, openmpi and netcdf. Whilst this may not be the optimal way of running Isca on your machine, it should work without too much extra effort.

Different workstations/servers at different institutions will have different compilers and libraries available. The Isca framework assumes you have something similar to our stack at Exeter, but provides a hook for you to configure the environment in which the model is run.
Before Isca is compiled/run, an environment is first configured which loads the specific compilers and libraries necessary to build the code. This done by setting the environment variable `GFDL_ENV` in your session. There is an option within Isca to use set `GFDL_ENV=ubuntu_conda`, which is setup to use the gfortran compiler you installed via the environment file.

Before Isca is compiled/run, an environment is first configured which loads the specific compilers and libraries necessary to build the code. This done by setting the environment variable `GFDL_ENV` in your session.

For example, on the EMPS workstations at Exeter, I have the following in my `~/.bashrc`:
To make use of this environment, you should add a version of the following to your `~/.bashrc`:

```{bash}
# directory of the Isca source code
export GFDL_BASE=/scratch/jamesp/Isca
# "environment" configuration for emps-gv4
export GFDL_ENV=emps-gv
# "environment" configuration for use with ubuntu-conda
export GFDL_ENV=ubuntu_conda
# temporary working directory used in running the model
export GFDL_WORK=/scratch/jamesp/gfdl_work
export GFDL_WORK=/scratch/jamesp/isca_work
# directory for storing model output
export GFDL_DATA=/scratch/jamesp/gfdl_data
export GFDL_DATA=/scratch/jamesp/isca_data
```

The value of `GFDL_ENV` corresponds to a file in `src/extra/env` that is sourced before each run or compilation. For an example that you could adapt to work on your machine, see `src/extra/env/emps-gv`.
The value of `GFDL_ENV` corresponds to a file in `src/extra/env` that is sourced before each run or compilation.

You may wish to configure your own way to run Isca using locally available compilers, e.g. the Intel compilers. An example using such a setup is available here - `src/extra/env/emps-gv`.

At Exeter University, Isca is compiled using:

* Intel Compiler Suite 14.0
* OpenMPI 10.0.1
* NetCDF 4.3.3.1
* git 2.1.2

We are not able to provide support in configuring your environment at other institutions other than Exeter University - we suggest that you contact your friendly local sysops technician for guidance in getting the compilers and libraries collated if you are not sure how to proceed.
Different workstations/servers at different institutions will have different compilers and libraries available. The Isca framework should run 'out of the box' using the gfortran compilers installed from the environment file. However, if you want to install the model using your own configuration then we are not able to provide support in configuring your environment. We suggest that you contact your friendly local sysops technician for guidance in getting the compilers and libraries collated if you are not sure how to proceed.

If you work at another large institution and have successfully compiled and run Isca, we welcome you to commit your own environment config to `/src/extra/env/my-new-env` for future scientists to benefit from and avoid the pain of debugging compilation!

Expand Down
4 changes: 2 additions & 2 deletions ci/environment-py3.9.yml → ci/environment-py.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@ dependencies:
- libgfortran
- netcdf-fortran
- numpy
- openmpi
- openmpi=4.*
- pandas
- python=3.9
- python
- pip
- pytest
- sh
Expand Down
20 changes: 20 additions & 0 deletions ci/environment-py3.12_frozen.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
name: isca_env
channels:
- conda-forge
dependencies:
- dask=2024.1.0
- f90nml=1.4.4
- fortran-compiler=1.7.0
- ipykernel=6.29.0
- jinja2=3.1.3
- libgfortran=3.0.0
- netcdf-fortran=4.6.1
- numpy=1.26.3
- openmpi=4.1.6
- pandas=2.2.0
- python=3.12.1
- pip=23.3.2
- pytest=7.4.4
- sh=2.0.6
- tqdm=4.66.1
- xarray=2024.1.1
Loading