V2.16.x -> main (#715)

* V2.15.1 PR (#674)

* starting for new release uw2.15

* Update surfaceProcesses.py

Update timeField to record time of deposition of sediment in SP functions

* Update surfaceProcesses.py

Fix typo in 3D surface.

Also reset time of the air material to the model time

* Update surfaceProcesses.py

fixed typo in func

* updating the setonix jobs runners

* Update surfaceProcesses.py

Simplify 2D velocity and diffusivity SP functions

* Update surfaceProcesses.py

Didn't create object to bcast surface function to

* Update surfaceProcesses

Revert back to MPI gather of data over evaluate_global

Update the examples

* Adding melt dynamic heating to the advection diffusion solver
see issue #669

* docker utils updates

* Adding some final fixes to the dockerfile scripts

* Update CHANGES.md

* Remove obsolete glucifer bits

* Update dockerfile Badlands install and CHANGES.md

* Update CI for docker builds

* Fixing the velocitySurface 2D and 3D implementation to handle restarts.

Changes also include:
* Renaming the surface tracers to prevent unexpected overwrites
* The surfaceProcess class saving the passive tracer key to use during
initialisation and solve. This is to make the code not save an instance
of the pt to the swarm incase the pt is swapped in/out of Model memory.
And helps with code readability.

* Get the latest release of badlands from pypi

---------

Co-authored-by: Ben Knight <[email protected]>

* starting 2.16.x

* Moving to a specific badlands version & (arm compatible version of) jupyter-server extension command

* update freesurf to 3D in uwg (#683)

1. update the free surface function in UWGeodynamics to a 3D version (see details in _freesurface.py).
2. add an example 1_23_05_FreeSurface_3D_Relaxation in docs/UWGeodynamics/examples/.

* First cut at the GA4 trojan.

Unfortunately GA4 isn't the same as the previous UA system by google. So location tracking is not enabled.

Every import (where the python interpreter is up for ~2 seconds) will produce a GA4 event that records:
- the version of Underworld used
- the platform it's being used on
- the number of mpi processors in the execution of python.

This event is anonymous and no user data other than the above, and a random key (assigned during install) is sent to the GA4 server.
This will provide valuable metrics to the Underworld development team.

As previous users can disable this trojan by setting the environment
variable UW_NO_USAGE_METRICS to any value, eg.
export UW_NO_USAGE_METRICS=1

Also updating the version number to 2.16.0

* Update install instructions to include ninja and python 3.11

* Update UWGeoTutorials.rst

fixing some missing links, pointed out to me by a user

* move underworld directory into 'src'. Reduce the namespace issues.

* Updating pyproject.toml and version number to support python 3.12

* Fix this test.

'imp' module is not longer in python 3.12. Moving to 'runpy' instead.

* Turn off pedantic compiler warnings. StGermain is full of pointer / int conversion code that trips up sensitive compilers

* moving everything to src/ to avoid namespace confusion

* error in prefix of install dir

* Remove this because it's causing lots of compiler warnings.

* Adding a script clean.sh to quickly strip the repo of python build/cache
files

* Fixes to use 'src' strategy to install Underworld with pip install .

* Updating cleaning script

* manually send country/region ip info with this add to the uw trojan

* Typo with white space

* Missed this update

* Geo information in G4A stats

Not the best implementation but G4A doesn't support geographic data via
it's Measurement Protocol. So the workaround ("hack") I came up with is
to put the country name in the event title.

The existing issue of G4A's lack of geo info.
https://issuetracker.google.com/issues/237566549?pli=1

* Cleaning up some warnings from compilation.

* Build changes for 2.16

1. setup.py -> pyproject.toml new standard, PEP 621.
2. all source code now under `src`.

This is a start for the new standard. Main issue is build isolation
forces CMake caching to break, so I have disabled the caching for now.

* Update CHANGES.md

* Rename ressources -> resources in UWGeodynamics

The path ressources is still symlinked for backwards compatibility.

* Misc changes for 2.16

* pyproject.toml with dyanmic versioning
* UWGeodynamics/tutorial: ressources -> resources

* simplify dependencies in pyproject.toml

* remove erroneous '-python' option - only affecting gadi

* reformating this with 'clang-format --styleLLVM'

* Big update of Dockersfiles

- More automation of versioning in `docker-build.sh`
- Moving to Ubuntu 24.04 as base OS
- Update to petsc 3.22.2
- using `podman` instead of `docker`

* force lavavu-osmesa install with compilation

* Switch to lavavu instead of lavavu-osmesa

* Just use lavavu with --no-binary :lavavu:

* Force numpy<2 in all dockers

* add an extra check if lavavu is valid - for arm64 linux installation

* Update the dockerfiles

* Updates to conda files, ie latest petsc, numpy<2, cython>=3

* Fix typo when calling bash

use 'set -x' instead

* fix typo in dockerfile

* Add pawsey container build recipe

* Updates for v2.16.x

* typo in check for lavavu

* Add environment file

* Docker updates (#711)

* Update dockerfiles

- Rebase to a python image based on Debian bookworm
- Update runtime requirements (mainly for lavavu)
- Modify bash script to allow switching between mpi implementation & change of base image

* Remove pawsey files

Can be built with the previous commit (8048d4c)

* Update CHANGES.md

* add some notes on the complex build system of UW2

* Updating the environment to include badlands. Coupling is a go.

* Create conda-build-test.yml

Use the conda/environment.yaml to provide a quick environment to compile and run the unit tests.

* Update conda-build-test.yml

Need to init the conda env to run things in the mamba activate env.

* Update conda-build-test.yml

* Update conda-build-test.yml

* Update conda-build-test.yml

* Update conda-build-test.yml

get the tests running

* Update conda-build-test.yml

* Update conda-build-test.yml

just get it working. Would like to split the steps of compiling and running the tests, but can't work out the syntax.

* Update conda-build-test.yml

Include mac-intel in tests

* fixes for docker naming and notes of building in podman

* add some documentation in the docker-builder.sh
:

* Fix spacing

* Update Dockerfile

To get mybinder links working

* Update Dockerfile

Fix warning of legacy variable UW_MACHINE

* Update README.md

Adding a new mybinder link

* Update README.md

* Dev into 2.16.x (#708)

* Fix cohesion bug by adding epsilon1 and epsilon2 (#707)

* Update conda-build-test.yml

mac-os from build runner. Its not available and that's causing errors.

---------

Co-authored-by: tiannh7 <[email protected]>

* Sphinx-updates (#713)

* Fix cohesion bug by adding epsilon1 and epsilon2 (#707)

* Update conda-build-test.yml

mac-os from build runner. Its not available and that's causing errors.

* Update .readthedocs.yml

* update conf.py path from "src/"

---------

Co-authored-by: tiannh7 <[email protected]>

---------

Co-authored-by: Ben Knight <[email protected]>
Co-authored-by: Neng Lu <[email protected]>
Co-authored-by: Arijit Laik <[email protected]>
Co-authored-by: tiannh7 <[email protected]>

.github/workflows/conda-build-test.yml

@@ -0,0 +1,33 @@
+name: Conda Test Environment, with micromamba
+
+on:
+  push:
+    branches:
+      - v2.16.x
+  pull_request:
+    branches:
+      - v2.16.x
+
+jobs:
+  test:
+    runs-on: [ubuntu-latest] #, macos-latest]
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up Micromamba
+        uses: mamba-org/setup-micromamba@v2
+        with:
+          environment-file: conda/environment.yaml
+          cache-environment: true
+          cache-downloads: true
+          init-shell: bash
+
+      - name: Compile the code, run tests
+        shell: micromamba-shell {0}
+        run: | 
+          micromamba list
+          pip list
+          pip install .
+          pytest ./docs/pytests/tests.py ./docs/pytests/test_examples.py

.readthedocs.yml

@@ -9,6 +9,10 @@ version: 2
 sphinx:
   configuration:  docs/development/docs_generator/conf.py
 
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.12"
 # Build documentation with MkDocs
 #mkdocs:
 #  configuration: mkdocs.yml
@@ -18,6 +22,5 @@ sphinx:
 
 # Optionally set the version of Python and requirements required to build your docs
 python:
-  version: 3.7
   install:
-    - requirements: docs/development/docs_generator/requirements.txt
+    - requirements: docs/development/docs_generator/requirements.txt

CHANGES.md

@@ -1,5 +1,20 @@
 CHANGES: Underworld2
 =======================
+Release 2.16.0 [2024]
+---------------------------
+New:
+* New 3D free surface implementation (tested). The vertical coordinates of the mesh nodes are rebuilt by being advected and interpolated from the surface velocities, and then solving the steady state heat equation to get a uniform distribution. See the related example in docs/UWGeodynamics/examples
+* Build changes PEP 621, pyproject.toml, and all source code now under `src`.
+* Move to Petsc-3.20.0.
+* New G4A stats module for user statics measurements.
+
+Changes:
+* UWGeodynamics - 'ressources' folder is now 'resources' (but previous name is still supported).
+* Modify docker building script to allow changing MPI implementation. 
+
+Fixes:
+* Update UWGeoTutorials.rst #693.
+* Fix for cohesion bug in Druker-Prager rheology. #707
 
 Release 2.15.0 [2023-04-19]
 ---------------------------
@@ -12,7 +27,7 @@ Changes:
 
 Fixes:
  * UWGeodynamics - add dynamic heating back into the advection diffusion solver,
-  https://github.com/underworldcode/underworld2/issues/669
+    https://github.com/underworldcode/underworld2/issues/669
  * Using updated Badlands-2.2.3 without license issue. 
 
 
@@ -162,7 +177,7 @@ Docker:
 * Images minimised with unnecessary items removed. 
 * XVFB no longer required for image generation within
   container.
- 
+
 API changes:
 * `glucifer` module moved inside `underworld` and 
    renamed `visualisation`.

Dockerfile

@@ -1,4 +1,4 @@
-FROM underworldcode/underworld2:2.15.1b
+FROM underworldcode/underworld2:2.16.0-ompi
 
 # Set the UW_MACHINE env variable for metrics
-ENV UW_MACHINE binder
+ENV UW_MACHINE=binder

Installation.rst

@@ -17,12 +17,13 @@ Build environment
 
 * cmake >= 3.16
 * MPICH or OPENMPI
-* PETSc (<= 3.16.1)
+* PETSc
 * HDF5  
 * SWIG (>= 4.0) 
 * LibXML (>= 2.0)
-* Python (>3.5, <= 3.9.9) 
+* Python (>3.5, <= 3.11) 
 * Numpy
+* Ninja
 
 Running Requirements
 ********************
@@ -101,4 +102,4 @@ Notes on Installing Docker
 
      docker-machine ip default
 
-  but note that this will only work correctly from the *Docker Quickstart Terminal*.
+  but note that this will only work correctly from the *Docker Quickstart Terminal*.

MANIFEST.in

@@ -1,9 +1,10 @@
-recursive-include underworld *i
-recursive-include underworld *c
-recursive-include underworld *cpp
-recursive-include underworld *h
-recursive-include underworld *hpp
-recursive-include underworld *py
-recursive-include underworld CMakeLists*
-recursive-include underworld/UWGeodynamics/ressources *.json
-recursive-include underworld/UWGeodynamics/uwgeo-data *
+recursive-include src/underworld *py
+recursive-include src/underworld/UWGeodynamics/resources *.json
+recursive-include src/underworld/UWGeodynamics/uwgeo-data *
+
+# recursive-include src/underworld *i
+# recursive-include src/underworld *c
+# recursive-include src/underworld *cpp
+# recursive-include src/underworld *h
+# recursive-include src/underworld *hpp
+# recursive-include src/underworld CMakeLists*

README.md

@@ -54,7 +54,7 @@ Note that it can take a while for the site to fire up and that it will time-out
 
 | | |
 |-|-|
-| [![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/underworldcode/underworld2/v2.13.0b) | v2.13.0b |
+| [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/underworldcode/underworld2/v2.16.x) | v2.16.x |
 | [![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/underworldcode/underworld2/v2.12.0b) | v2.12.0b |
 | [![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/underworldcode/underworld2/v2.11.0b) | v2.11.0b |
 | [![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/underworldcode/underworld2/development) | dev |

clean.sh

@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+
+set -x  # show commands
+
+rm -fr build
+find . -name \*.so -exec rm {} +
+find . -name __pycache__ -exec rm -r {} +
+find . -name '*.egg-info' -exec rm -r {} +
+rm -rf .pytest_cache
+## To remove untracked git files
+#git clean -dfX ; git clean -dfx

conda/conda_build_config.yaml

@@ -2,4 +2,4 @@ mpi:
   - mpich
   #- openmpi [linux]
 petsc:
-  - 3.18.1
+  - 3.21.5

conda/environment.yaml

@@ -0,0 +1,29 @@
+name: uw2-dev
+channels:
+  - conda-forge
+dependencies:
+  - python==3.11
+  - compilers
+  - pkg-config
+  - libxml2
+  - cmake
+  - swig>=4.0.0
+  - ninja
+  - pint
+  - mpich
+  - petsc==3.21.5
+  - mesalib
+  - numpy<2
+  - scipy
+  - mpi4py
+  - h5py=*=mpi*
+  - pytest
+  - nbmake
+  - matplotlib
+  - jupyterlab
+  - jupytext
+  - pip
+  - pip:
+    - lavavu
+    - "badlands @ git+https://github.com/julesghub/badlands.git@julesghub/meson-build#subdirectory=badlands"
+

conda/meta.yaml

@@ -1,58 +1,60 @@
 {% set name = "underworld" %}
-{% set version = "2.14.0" %}
-{% set sha256 = "a9323209b0e36743bc953bf3b3d8e0a84d5e4e7f1911a1e78d1404a0ade6337d" %}
+{% set version = "2.16.0" %}
 {% set build = 0 %}
 
 package:
   name: {{ name|lower }}
   version: {{ version }}
 
 source:
-  # git_url: https://github.com/julesghub/underworld2.git
-  # git_rev: dockerfile-updates
-  url: https://github.com/underworldcode/underworld2/archive/refs/tags/v{{ version }}b.tar.gz
-  sha256: {{ sha256 }}
+  git_url : https://github.com/underworldcode/underworld2.git
+  git_rev : v2.16.x
+#  url: https://github.com/underworldcode/underworld2/archive/refs/tags/v{{ version }}b.zip
+#  sha256: 5d27ce877f2238f6b2e9c6b90da25f1a61833ffa6093e0382f1bc5e2468dd782
 
 build:
-  skip: true  # [win]
+  skip: true  # [py<39 or win]
   number: 0
-  script: PETSC_DIR=$PREFIX {{ PYTHON }} -m pip -vv install --no-deps .  
-  missing_dso_whitelist:
-    - libm.so.6 
-    - libc.so.6 
+  script: PETSC_DIR=$PREFIX {{ PYTHON }} -m pip install . -vv
 
 requirements:
 
   build:
-    - {{ compiler('fortran') }}
     - {{ compiler('c') }}
     - {{ compiler('cxx') }}
     - {{ mpi }}
+    - pkg-config
+    - cmake
+    - ninja
 
   host:
-    - pkg-config
     - pip
+    - setuptools
     - python
+    - h5py =*=mpi*
     - libxml2
-    - numpy >=1.18
-    - cmake
-    - swig >=4.0.0
-    - ninja
+    - numpy <2.0.0
     - petsc {{ petsc }}
     - {{ mpi }}
+    - swig >=4.0.0
 
   run:
     - python
-    - {{ mpi }}
-    - numpy
+    - h5py =*=mpi*
+    - {{ pin_compatible('numpy') }}
     - petsc {{ petsc }}
+    - {{ mpi }}
+    - mpi4py
     - pint
     - scipy
-    - mpi4py
-    - h5py
+    - matplotlib-base
 
 test:
+  imports:
+    - underworld
+
   requires:
+    - python
     - pytest
     - matplotlib
 
@@ -62,21 +64,19 @@ test:
     - docs/examples
 
   commands:
+    #    - pytest -vvv docs/pytests/test.py
     - python -X faulthandler -c "import underworld; underworld.mesh.FeMesh_Cartesian();"
-    - pytest -vvv docs/pytests
 
 
 about:
-  home: "https://github.com/underworldcode/underworld2"
+  home: https://github.com/underworldcode/underworld2
   license: LGPL-3.0-only
   license_family: LGPL
-  license_file:  LGPLv3.txt
+  license_file: LGPLv3.txt
   summary: "Geodynamics API"
   doc_url: https://underworld2.readthedocs.io
   dev_url: https://github.com/underworldcode/underworld2/tree/development
 
 extra:
   recipe-maintainers:
-    - rbeucher
     - julesghub
-    - lmoresi