Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UW-607 orog_gsl driver #509

Merged
merged 29 commits into from
Jun 13, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
9773df5
Add prototype orog_gsl.yaml
maddenp-noaa Jun 12, 2024
2568762
Update prototype YAML config
maddenp-noaa Jun 12, 2024
5f6b951
Update CLI and API
maddenp-noaa Jun 12, 2024
3758ed1
Tidy
maddenp-noaa Jun 12, 2024
17a0eaf
Simplify Driver._write_runscript() signature and callers
maddenp-noaa Jun 12, 2024
2867918
Add orog_gsl.py
maddenp-noaa Jun 12, 2024
cf6db0c
Work on driver
maddenp-noaa Jun 12, 2024
7650959
Add orog-gsl.jsonschema
maddenp-noaa Jun 12, 2024
1dc8206
Reorder required in shave.jsonschema
maddenp-noaa Jun 12, 2024
da4c4ac
WIP
maddenp-noaa Jun 12, 2024
b2ef311
Delinting
maddenp-noaa Jun 12, 2024
3873d36
Add schema tests
maddenp-noaa Jun 12, 2024
d424b78
Add module envcmds to prototype YAML config
maddenp-noaa Jun 12, 2024
308bdc3
Add API unit tests
maddenp-noaa Jun 12, 2024
918f1fd
Work on unit tests
maddenp-noaa Jun 12, 2024
c9e1229
Use in-memory configs in driver tests to reduce IO
maddenp-noaa Jun 12, 2024
47e22ea
Use in-memory configs in driver tests to reduce IO
maddenp-noaa Jun 12, 2024
72cd854
Work on unit tests
maddenp-noaa Jun 12, 2024
564f9f6
Unit tests @ 100%
maddenp-noaa Jun 12, 2024
d4062c5
Work on docs
maddenp-noaa Jun 12, 2024
ddb0ad1
WIP
maddenp-noaa Jun 13, 2024
b2b9818
Add orog_gsl UW YAML doc
maddenp-noaa Jun 13, 2024
38a6d92
Work on docs
maddenp-noaa Jun 13, 2024
7ce5507
Fix docs typo
maddenp-noaa Jun 13, 2024
129d748
Update make_hgrid unit test
maddenp-noaa Jun 13, 2024
6c289f2
Fix section headers in test_schemas.py
maddenp-noaa Jun 13, 2024
5b9cd8c
Update doc
maddenp-noaa Jun 13, 2024
4998205
Merge branch 'main' into uw-607-orog-gsl-driver
maddenp-noaa Jun 13, 2024
d5d9efa
Update test_driver.py to test config file case
maddenp-noaa Jun 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,12 @@ make_solo_mosaic
| **CLI**: ``uw make_solo_mosaic -h``
| **API**: ``import uwtools.api.make_solo_mosaic``

orog_gsl
""""""""

| **CLI**: ``uw orog_gsl -h``
| **API**: ``import uwtools.api.orog_gsl``

sfc_climo_gen
"""""""""""""

Expand Down
1 change: 1 addition & 0 deletions docs/sections/user_guide/api/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ API
make_solo_mosaic
mpas
mpas_init
orog_gsl
rocoto
sfc_climo_gen
shave
Expand Down
5 changes: 5 additions & 0 deletions docs/sections/user_guide/api/orog_gsl.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
``uwtools.api.orog_gsl``
========================

.. automodule:: uwtools.api.orog_gsl
:members:
1 change: 1 addition & 0 deletions docs/sections/user_guide/cli/drivers/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ Drivers
make_solo_mosaic
mpas
mpas_init
orog_gsl
sfc_climo_gen
shave
ungrib
Expand Down
58 changes: 58 additions & 0 deletions docs/sections/user_guide/cli/drivers/orog_gsl.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
``orog_gsl``
============

The ``uw`` mode for configuring and running the UFS Utils preprocessing component ``orog_gsl``. Documentation for this UFS Utils component is :ufs-utils:`here <orog-gsl>`.

.. literalinclude:: orog_gsl/help.cmd
:language: text
:emphasize-lines: 1
.. literalinclude:: orog_gsl/help.out
:language: text

All tasks take the same arguments. For example:

.. literalinclude:: orog_gsl/run-help.cmd
:language: text
:emphasize-lines: 1
.. literalinclude:: orog_gsl/run-help.out
:language: text

Examples
^^^^^^^^

The examples use a configuration file named ``config.yaml`` with contents similar to:

.. highlight:: yaml
.. literalinclude:: /shared/orog_gsl.yaml

Its contents are described in section :ref:`orog_gsl_yaml`.

* Run ``orog_gsl`` on an interactive node

.. code-block:: text

$ uw orog_gsl run --config-file config.yaml

The driver creates a ``runscript.orog_gsl`` file in the directory specified by ``run_dir:`` in the config and runs it, executing ``orog_gsl``.

* Run ``orog_gsl`` via a batch job

.. code-block:: text

$ uw orog_gsl run --config-file config.yaml --batch

The driver creates a ``runscript.orog_gsl`` file in the directory specified by ``run_dir:`` in the config and submits it to the batch system. Running with ``--batch`` requires a correctly configured ``platform:`` block in ``config.yaml``, as well as appropriate settings in the ``execution:`` block under ``orog_gsl:``.

Looking at the run command in ``runscript.orog_gsl`` shows us the specified executable as well as the YAML keys converted to appropriate command line flags.

.. code-block:: text

time orog_gsl --num_tiles 1 --dir /path/to/grid/ --tile_file C403_grid.tile7.halo6.nc --periodx 360 --periody 360

* Specifying the ``--dry-run`` flag results in the driver logging messages about actions it would have taken, without actually taking any.

.. code-block:: text

$ uw orog_gsl run --config-file config.yaml --batch --dry-run

.. include:: /shared/key_path.rst
1 change: 1 addition & 0 deletions docs/sections/user_guide/cli/drivers/orog_gsl/Makefile
1 change: 1 addition & 0 deletions docs/sections/user_guide/cli/drivers/orog_gsl/help.cmd
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
uw orog_gsl --help
26 changes: 26 additions & 0 deletions docs/sections/user_guide/cli/drivers/orog_gsl/help.out
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
usage: uw orog_gsl [-h] [--version] TASK ...

Execute orog_gsl tasks

Optional arguments:
-h, --help
Show help and exit
--version
Show version info and exit

Positional arguments:
TASK
input_grid_file
The input grid file
provisioned_run_directory
Run directory provisioned with all required content
run
A run
runscript
The runscript
topo_data_2p5m
Global topographic data on 2.5-minute lat-lon grid
topo_data_30s
Global topographic data on 30-second lat-lon grid
validate
Validate the UW driver config
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
uw orog_gsl run --help
26 changes: 26 additions & 0 deletions docs/sections/user_guide/cli/drivers/orog_gsl/run-help.out
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
usage: uw orog_gsl run [-h] [--version] [--config-file PATH] [--batch]
[--dry-run] [--graph-file PATH]
[--key-path KEY[.KEY...]] [--quiet] [--verbose]

A run

Optional arguments:
-h, --help
Show help and exit
--version
Show version info and exit
--config-file PATH, -c PATH
Path to UW YAML config file (default: read from stdin)
--batch
Submit run to batch scheduler
--dry-run
Only log info, making no changes
--graph-file PATH
Path to Graphviz DOT output [experimental]
--key-path KEY[.KEY...]
Dot-separated path of keys leading through the config to the driver's
configuration block
--quiet, -q
Print no logging messages
--verbose, -v
Print all logging messages
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
global_equiv_resol
==================

Structured YAML to run the WRF preprocessing component ``global_equiv_resol`` is validated by JSON Schema and requires the ``global_equiv_resol:`` block, described below. If ``global_equiv_resol`` is to be run via a batch system, the ``platform:`` block, described :ref:`here <platform_yaml>`, is also required.
maddenp-noaa marked this conversation as resolved.
Show resolved Hide resolved
Structured YAML to run the component ``global_equiv_resol`` is validated by JSON Schema and requires the ``global_equiv_resol:`` block, described below. If ``global_equiv_resol`` is to be run via a batch system, the ``platform:`` block, described :ref:`here <platform_yaml>`, is also required.

Documentation for the UFS Utils ``global_equiv_resol`` program is :ufs-utils:`here <global-equiv-resol>`.

Expand Down
1 change: 1 addition & 0 deletions docs/sections/user_guide/yaml/components/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ UW YAML for Components
make_solo_mosaic
mpas
mpas_init
orog_gsl
schism
sfc_climo_gen
shave
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
make_solo_mosaic
================

Structured YAML to run the WRF preprocessing component ``make_solo_mosaic`` is validated by JSON Schema and requires the ``make_solo_mosaic:`` block, described below. If ``make_solo_mosaic`` is to be run via a batch system, the ``platform:`` block, described :ref:`here <platform_yaml>`, is also required.
maddenp-noaa marked this conversation as resolved.
Show resolved Hide resolved
Structured YAML to run the component ``make_solo_mosaic`` is validated by JSON Schema and requires the ``make_solo_mosaic:`` block, described below. If ``make_solo_mosaic`` is to be run via a batch system, the ``platform:`` block, described :ref:`here <platform_yaml>`, is also required.

Documentation for the UFS Utils ``make_solo_mosaic`` program is :ufs-utils:`here <make-solo-mosaic>`.

Expand Down
55 changes: 55 additions & 0 deletions docs/sections/user_guide/yaml/components/orog_gsl.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
.. _orog_gsl_yaml:

orog_gsl
========

Structured YAML to run the component ``orog_gsl`` is validated by JSON Schema and requires the ``orog_gsl:`` block, described below. If ``orog_gsl`` is to be run via a batch system, the ``platform:`` block, described :ref:`here <platform_yaml>`, is also required.

Documentation for the UFS Utils ``orog_gsl`` program is :ufs-utils:`here <orog-gsl>`.

Here is a prototype UW YAML ``orog_gsl:`` block, explained in detail below:

.. highlight:: yaml
.. literalinclude:: /shared/orog_gsl.yaml

UW YAML for the ``orog_gsl:`` Block
-----------------------------------

config:
^^^^^^^

Configuration parameters for the ``orog_gsl`` component.

**halo:**

Halo number (-999 for no halo).

**input_grid_file:**

Path to the tiled input grid file.

**resolution:**

Input grid resolution index.

**tile:**

Tile number (1-6 for global, 7 for regional).

**topo_data_2p5m:**

Path to file containing global topographic data on 2.5-minute lat-lon grid.

**topo_data_30s:**

Path to file containing global topographic data on 30-second lat-lon grid.

execution:
^^^^^^^^^^

See :ref:`here <execution_yaml>` for details.

run_dir:
^^^^^^^^

The path to the run directory.
20 changes: 20 additions & 0 deletions docs/shared/orog_gsl.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
orog_gsl:
config:
halo: 4
input_grid_file: /path/to/C{{ orog_gsl.config.resolution }}_grid.tile{{ orog_gsl.config.tile }}.halo{{ orog_gsl.config.halo }}.nc
resolution: 403
tile: 7
topo_data_2p5m: /path/to/geo_em.d01.lat-lon.2.5m.HGT_M.nc
topo_data_30s: /path/to/HGT.Beljaars_filtered.lat-lon.30s_res.nc
execution:
batchargs:
cores: 1
walltime: "00:01:00"
envcmds:
- module use /path/to/modules
- module load module_name
executable: /path/to/orog_gsl
run_dir: /path/to/dir/run
platform:
account: me
scheduler: slurm
12 changes: 12 additions & 0 deletions src/uwtools/api/orog_gsl.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
"""
API access to the ``uwtools`` ``orog_gsl`` driver.
"""

from uwtools.drivers.orog_gsl import OrogGSL as _Driver
from uwtools.drivers.support import graph
from uwtools.utils.api import make_execute as _make_execute
from uwtools.utils.api import make_tasks as _make_tasks

execute = _make_execute(_Driver)
tasks = _make_tasks(_Driver)
__all__ = ["execute", "graph", "tasks"]
2 changes: 2 additions & 0 deletions src/uwtools/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ def main() -> None:
STR.makesolomosaic,
STR.mpas,
STR.mpasinit,
STR.oroggsl,
STR.sfcclimogen,
STR.shave,
STR.ungrib,
Expand Down Expand Up @@ -1033,6 +1034,7 @@ def _parse_args(raw_args: List[str]) -> Tuple[Args, Checks]:
STR.globalequivresol,
STR.makehgrid,
STR.makesolomosaic,
STR.oroggsl,
STR.sfcclimogen,
STR.shave,
]
Expand Down
4 changes: 2 additions & 2 deletions src/uwtools/drivers/driver.py
Original file line number Diff line number Diff line change
Expand Up @@ -340,14 +340,14 @@ def _validate(self) -> None:
for schema_name in (self._driver_name.replace("_", "-"), "platform"):
validate_internal(schema_name=schema_name, config=self._config)

def _write_runscript(self, path: Path, envvars: Dict[str, str]) -> None:
def _write_runscript(self, path: Path, envvars: Optional[Dict[str, str]] = None) -> None:
maddenp-noaa marked this conversation as resolved.
Show resolved Hide resolved
"""
Write the runscript.
"""
path.parent.mkdir(parents=True, exist_ok=True)
rs = self._runscript(
envcmds=self._driver_config.get("execution", {}).get("envcmds", []),
envvars=envvars,
envvars=envvars or {},
execution=[
"time %s" % self._runcmd,
"test $? -eq 0 && touch %s" % self._runscript_done_file,
Expand Down
2 changes: 1 addition & 1 deletion src/uwtools/drivers/esg_grid.py
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ def runscript(self):
yield self._taskname(path.name)
yield asset(path, path.is_file)
yield None
self._write_runscript(path=path, envvars={})
self._write_runscript(path)

# Private helper methods

Expand Down
2 changes: 1 addition & 1 deletion src/uwtools/drivers/global_equiv_resol.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ def runscript(self):
yield self._taskname(path.name)
yield asset(path, path.is_file)
yield None
self._write_runscript(path=path, envvars={})
self._write_runscript(path)

# Private helper methods

Expand Down
2 changes: 1 addition & 1 deletion src/uwtools/drivers/jedi.py
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ def runscript(self):
yield self._taskname(path.name)
yield asset(path, path.is_file)
yield None
self._write_runscript(path=path, envvars={})
self._write_runscript(path)

@task
def validate_only(self):
Expand Down
2 changes: 1 addition & 1 deletion src/uwtools/drivers/make_hgrid.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ def runscript(self):
yield self._taskname(path.name)
yield asset(path, path.is_file)
yield None
self._write_runscript(path=path, envvars={})
self._write_runscript(path)

# Private helper methods

Expand Down
2 changes: 1 addition & 1 deletion src/uwtools/drivers/make_solo_mosaic.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ def runscript(self):
yield self._taskname(path.name)
yield asset(path, path.is_file)
yield None
self._write_runscript(path=path, envvars={})
self._write_runscript(path)

# Private helper methods

Expand Down
2 changes: 1 addition & 1 deletion src/uwtools/drivers/mpas_base.py
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ def runscript(self):
yield self._taskname(path.name)
yield asset(path, path.is_file)
yield None
self._write_runscript(path=path, envvars={})
self._write_runscript(path)

@task
def streams_file(self):
Expand Down
Loading
Loading