diff --git a/docs/sections/user_guide/api/config.rst b/docs/sections/user_guide/api/config.rst index 5f689e1e7..0de57f368 100644 --- a/docs/sections/user_guide/api/config.rst +++ b/docs/sections/user_guide/api/config.rst @@ -1,6 +1,9 @@ ``uwtools.api.config`` ====================== +.. image:: https://mybinder.org/badge_logo.svg + :target: https://mybinder.org/v2/gh/ufs-community/uwtools/notebooks?labpath=notebooks%2Fconfig.ipynb + .. automodule:: uwtools.api.config :inherited-members: UserDict :members: diff --git a/docs/sections/user_guide/index.rst b/docs/sections/user_guide/index.rst index e0ecea2f3..31b4eed42 100644 --- a/docs/sections/user_guide/index.rst +++ b/docs/sections/user_guide/index.rst @@ -23,8 +23,9 @@ User Guide diff --git a/notebooks/config.ipynb b/notebooks/config.ipynb new file mode 100644 index 000000000..bba503764 --- /dev/null +++ b/notebooks/config.ipynb @@ -0,0 +1,996 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "d023d283-7e1d-4e75-95b8-5033bea42a59", + "metadata": {}, + "source": [ + "# Config Tool\n", + "\n", + "The `uwtools` API's `config` module provides functions to create and manipulate configuration files, objects, and dictionaries.\n", + "\n", + "Tested on `uwtools` version 2.4.2. For more information, please see the uwtools.api.config Read the Docs page.\n", + "\n", + "## Table of Contents\n", + "\n", + "* [Getting Config Objects](#Getting-Config-Objects)\n", + "* [Config Depth Limitations](#Config-Depth-Limitations)\n", + "* [Realizing Configs](#Realizing-Configs)\n", + " * [Updating Configs](#Updating-Configs)\n", + " * [Using the `key_path` Parameter](#Using-the-key_path-Parameter)\n", + " * [Using the `values_needed` Parameter](#Using-the-values_needed-Parameter)\n", + " * [Using the `total` Parameter](#Using-the-total-Parameter)\n", + "* [Realizing Configs to a Dictionary](#Realizing-Configs-to-a-Dictionary)\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "6d02d033-0992-4990-861d-3f80d09d7083", + "metadata": {}, + "outputs": [], + "source": [ + "from pathlib import Path\n", + "from uwtools.api import config\n", + "from uwtools.api.logging import use_uwtools_logger\n", + "\n", + "use_uwtools_logger()" + ] + }, + { + "cell_type": "markdown", + "id": "212594bb-379f-4441-805e-af0dbabe1815", + "metadata": {}, + "source": [ + "## Getting Config Objects\n", + "\n", + "The `config` tool can create configuration objects given a Python ``dict`` or a file in one of five different formats: FieldTable, INI, Fortran namelist, Shell, or YAML. `config.get_yaml_config` is demonstrated here, but the config module also has similar functions for each of the other supported formats: `get_fieldtable_config()`, `get_ini_config()`, `get_nml_config()`, and `get_sh_config()`.\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "ab0e21c3-a4b6-404c-bffd-e0d393d9b0a2", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Help on function get_yaml_config in module uwtools.api.config:\n", + "\n", + "get_yaml_config(config: Union[dict, str, pathlib.Path, NoneType] = None, stdin_ok: bool = False) -> uwtools.config.formats.yaml.YAMLConfig\n", + " Get a ``YAMLConfig`` object.\n", + "\n", + " :param config: YAML file or ``dict`` (``None`` => read ``stdin``).\n", + " :param stdin_ok: OK to read from ``stdin``?\n", + " :return: An initialized ``YAMLConfig`` object.\n", + "\n" + ] + } + ], + "source": [ + "help(config.get_yaml_config)" + ] + }, + { + "cell_type": "markdown", + "id": "606da5b3-4bff-4148-a9a5-908aa7dd5e8c", + "metadata": {}, + "source": [ + "The `stdin_ok` argument can be used to permit reads from `stdin`, but this is a rare use case beyond the scope of this notebook that will not be discussed here.\n", + "\n", + "`get_yaml_config()` can take input from a Python `dict` or a YAML file like the one below.\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "c6e049df-38f6-4879-8e0d-68356226d94b", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "greeting: Hello\n", + "recipient: World\n" + ] + } + ], + "source": [ + "%%bash\n", + "cat fixtures/config/get-config.yaml" + ] + }, + { + "cell_type": "markdown", + "id": "c2f72448-d35e-4a0b-a371-cb47c7b3338b", + "metadata": {}, + "source": [ + "Paths to config files can be provided either as a string or Path object. Since `get_yaml_config()` is used here, a `YAMLConfig` object is returned.\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "cc3020a6-4eb4-4830-9263-a9fc8fac7450", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "greeting: Hello\n", + "recipient: World\n" + ] + } + ], + "source": [ + "config1 = config.get_yaml_config(\n", + " config=Path(\"fixtures/config/get-config.yaml\")\n", + ")\n", + "print(type(config1))\n", + "print(config1)" + ] + }, + { + "cell_type": "markdown", + "id": "b7bcd736-ff78-4e8b-957f-b348b812c5f6", + "metadata": {}, + "source": [ + "Providing a Python `dict` will create a UW `Config` object with format matching the function used.\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "f01ac223-4a02-40ba-822f-8e66ad39f313", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "message:\n", + " greeting: Hi\n", + " recipient: Earth\n" + ] + } + ], + "source": [ + "input_config = {\"message\": {\"greeting\":\"Hi\", \"recipient\":\"Earth\"}}\n", + "config2 = config.get_yaml_config(\n", + " config=input_config\n", + ")\n", + "print(config2)" + ] + }, + { + "cell_type": "markdown", + "id": "dc745e95-d1ce-435c-a488-13b761979e36", + "metadata": {}, + "source": [ + "## Config Depth Limitations\n", + "\n", + "Some config formats have limitations on the depth of their nested configs. Shell configs, for example, may only contain top-level, bash-syntax `key=value` pairs.\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "28d23ac5-52a0-45bc-bfee-98d9ea518ca2", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "greeting=Salutations\n", + "recipient=Mars" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "config.get_sh_config(\n", + " config={\"greeting\":\"Salutations\", \"recipient\":\"Mars\"}\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "3dd7292f-cbd1-4e45-b641-1e213a4ead07", + "metadata": {}, + "source": [ + "Shell configs cannot be nested, and any attempt to do so will raise a `UWConfigError`.\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "b3a0a5bc-9d1b-4d48-a05f-be6f94fb6e1d", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Cannot instantiate depth-1 SHConfig with depth-2 config\n" + ] + } + ], + "source": [ + "try: \n", + " config.get_sh_config(\n", + " config={\"message\": {\"greeting\":\"Salutations\", \"recipient\":\"Mars\"}}\n", + " )\n", + "except Exception as e:\n", + " print(e)" + ] + }, + { + "cell_type": "markdown", + "id": "7677bdaa-8707-4dff-b812-91b4521f4820", + "metadata": {}, + "source": [ + "When creating INI configs, exactly one level of nesting is required so that each key-value pair is contained within a section. The top level keys become sections, which are contained within square brackets `[]`. Read more about INI configuration files here.\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "6837e75b-bd20-4c3b-bd33-650e4b4f9f23", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "[message]\n", + "greeting = Salutations\n", + "recipient = Mars" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "config.get_ini_config(\n", + " config={\"message\": {\"greeting\":\"Salutations\", \"recipient\":\"Mars\"}}\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "04ab0d52-75cc-4227-8058-a9a1faba7b54", + "metadata": {}, + "source": [ + "Either more or fewer levels of nesting will raise a `UWConfigError`.\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "9655b36b-2d39-4fc1-b3b8-9cb3443cf4b8", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Cannot instantiate depth-2 INIConfig with depth-1 config\n" + ] + } + ], + "source": [ + "try:\n", + " config.get_ini_config(\n", + " config={\"greeting\":\"Salutations\", \"recipient\":\"Mars\"}\n", + " )\n", + "except Exception as e:\n", + " print(e)" + ] + }, + { + "cell_type": "markdown", + "id": "e4403333-c247-465f-9a75-f96d1be914f6", + "metadata": {}, + "source": [ + "## Realizing Configs\n", + "\n", + "The `config.realize()` function writes config files to disk or `stdout` with the ability to render Jinja2 expressions and add/update values.\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "2c3f1b75-b26f-4893-beb7-37a58c09f511", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Help on function realize in module uwtools.api.config:\n", + "\n", + "realize(input_config: Union[uwtools.config.formats.base.Config, pathlib.Path, dict, str, NoneType] = None, input_format: Optional[str] = None, update_config: Union[uwtools.config.formats.base.Config, pathlib.Path, dict, str, NoneType] = None, update_format: Optional[str] = None, output_file: Union[str, pathlib.Path, NoneType] = None, output_format: Optional[str] = None, key_path: Optional[list[Union[str, int]]] = None, values_needed: bool = False, total: bool = False, dry_run: bool = False, stdin_ok: bool = False) -> dict\n", + " Realize a config based on a base input config and an optional update config.\n", + "\n", + " The input config may be specified as a filesystem path, a ``dict``, or a ``Config`` object. When it\n", + " is not, it will be read from ``stdin``.\n", + "\n", + " If an update config is specified, it is merged onto the input config, augmenting or overriding base\n", + " values. It may be specified as a filesystem path, a ``dict``, or a ``Config`` object. When it is\n", + " not, it will be read from ``stdin``.\n", + "\n", + " At most one of the input config or the update config may be left unspecified, in which case the\n", + " other will be read from ``stdin``. If neither filename or format is specified for the update config, no\n", + " update will be performed.\n", + "\n", + " The output destination may be specified as a filesystem path. When it is not, it will be written to\n", + " ``stdout``.\n", + "\n", + " If ``values_needed`` is ``True``, a report of values needed to realize the config is logged. In\n", + " ``dry_run`` mode, output is written to ``stderr``.\n", + "\n", + " If ``total`` is ``True``, an exception will be raised if any Jinja2 variables/expressions cannot be\n", + " rendered. Otherwise, such variables/expressions will be passed through unchanged in the output.\n", + "\n", + " Recognized file extensions are: ini, nml, sh, yaml\n", + "\n", + " :param input_config: Input config file (``None`` => read ``stdin``).\n", + " :param input_format: Format of the input config (optional if file's extension is recognized).\n", + " :param update_config: Update config file (``None`` => read ``stdin``).\n", + " :param update_format: Format of the update config (optional if file's extension is recognized).\n", + " :param output_file: Output config file (``None`` => write to ``stdout``).\n", + " :param output_format: Format of the output config (optional if file's extension is recognized).\n", + " :param key_path: Path through keys to the desired output block.\n", + " :param values_needed: Report complete, missing, and template values.\n", + " :param total: Require rendering of all Jinja2 variables/expressions.\n", + " :param dry_run: Log output instead of writing to output.\n", + " :param stdin_ok: OK to read from ``stdin``?\n", + " :return: The ``dict`` representation of the realized config.\n", + " :raises: UWConfigRealizeError if ``total`` is ``True`` and any Jinja2 variable/expression was not rendered.\n", + "\n" + ] + } + ], + "source": [ + "help(config.realize)" + ] + }, + { + "cell_type": "markdown", + "id": "501da514-a654-4511-928f-b2ad7db102b2", + "metadata": {}, + "source": [ + "The `input_config` parameter takes a config from a string path, Path object, Python `dict`, or UW `Config` object like the `YAMLConfig` object from the Getting Config Objects section. The `input_format` argument must be provided for `dict` inputs or for files without recognized extensions. Configs are written to `stdout` if `output_file` is unspecified or explicitly set to `None`, or to the file specified by `output_file`. The `output_format` argument must be provided when writing to `stdout` or to a file without a recognized extension. Recognized extensions are: `.ini`, `.nml`, `.sh`, and `.yaml`.\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "77873e14-db3c-417d-be7a-2ba12c9a38f6", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'greeting': 'Hello', 'recipient': 'World'}" + ] + }, + "execution_count": 11, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "config.realize(\n", + " input_config=config1,\n", + " output_file=Path('tmp/config1.yaml')\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "fd1823c4-6c3d-4d4a-a614-4d1238588bdd", + "metadata": {}, + "source": [ + "The `realize()` method returns a dict version of the config regardless of input type, and the file is written in the YAML format as indicated by the file extension.\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "4f237a73-da83-4632-990f-644632b15cd9", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "greeting: Hello\n", + "recipient: World\n" + ] + } + ], + "source": [ + "%%bash\n", + "cat tmp/config1.yaml" + ] + }, + { + "cell_type": "markdown", + "id": "902c4863-e772-49c6-88f2-31e1ab76b418", + "metadata": {}, + "source": [ + "Input and output formats are not required to match. This can be used to convert some configs from one format to another. YAML configs can be converted to configs of other recognized formats so long as the depth restrictions of the output format are met. All configs of recognized formats can be converted into YAML configs. Keep in mind that some formats are unable to express some types (for example, Shell configs can't express a value as an `int` while a Fortran namelist can) so type information may be lost when converting between formats.\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "b8854dc6-9dd2-4843-99e4-278b116b9767", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'greeting': 'Hello', 'recipient': 'World'}" + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "config.realize(\n", + " input_config='fixtures/config/get-config.yaml',\n", + " input_format='yaml',\n", + " output_file='tmp/realize-config.sh',\n", + " output_format='sh'\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "4032ab38-f6c7-45d1-bff7-f70d23832f26", + "metadata": {}, + "source": [ + "Here a Shell config is created from a YAML config.\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "0a59a3e8-27b5-4daa-a924-941aceaad157", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "greeting=Hello\n", + "recipient=World\n" + ] + } + ], + "source": [ + "%%bash\n", + "cat tmp/realize-config.sh" + ] + }, + { + "cell_type": "markdown", + "id": "ef0136e4-f549-467f-9341-f77f84738bb0", + "metadata": {}, + "source": [ + "### Updating Configs\n", + "\n", + "Configs can be updated by providing a second config with the `update_config` parameter. If the update config contains keys that match the base config, the base config values for those keys will be overwritten. Once updated, if the config contains Jinja2 expressions, like the one below, they will be rendered in the config wherever possible.\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "2d72bbc1-e438-48b2-8bd7-554b598c6f24", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "&memo\n", + " sender_id = \"{{ id }}\"\n", + " message = \"{{ greeting }}, {{ recipient }}!\"\n", + " sent = .FALSE.\n", + "/\n" + ] + } + ], + "source": [ + "%%bash\n", + "cat fixtures/config/base-config.nml" + ] + }, + { + "cell_type": "markdown", + "id": "67b07a34-1bea-402f-9cec-f33a2e519d27", + "metadata": {}, + "source": [ + "Here, the update config provides values that will update two of the Jinja2 expressions and override one key with a new value.\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "7648fdd5-5752-4bf3-b366-db8da1eac601", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'memo': {'sender_id': '{{ id }}',\n", + " 'message': 'Salutations, Mars!',\n", + " 'sent': True,\n", + " 'greeting': 'Salutations',\n", + " 'recipient': 'Mars'}}" + ] + }, + "execution_count": 16, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "config.realize(\n", + " input_config='fixtures/config/base-config.nml',\n", + " update_config={\"memo\": {\"greeting\":\"Salutations\", \"recipient\":\"Mars\", \"sent\": True}},\n", + " output_file='tmp/updated-config.nml'\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "bdac7f6c-4452-4137-9104-784449584100", + "metadata": {}, + "source": [ + "All of the key-value pairs were added to the updated config, and the base config was rendered where the appropriate values were provided. However, not all Jinja2 expressions are required to be rendered: An `id` key was not provided in the update config, so the expression referencing it was not rendered.\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "f4538965-d3b0-4c0c-a878-b6852f8d8ab0", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "&memo\n", + " sender_id = '{{ id }}'\n", + " message = 'Salutations, Mars!'\n", + " sent = .true.\n", + " greeting = 'Salutations'\n", + " recipient = 'Mars'\n", + "/\n" + ] + } + ], + "source": [ + "%%bash\n", + "cat tmp/updated-config.nml" + ] + }, + { + "cell_type": "markdown", + "id": "a20024ac-0f33-4000-942d-29c99dc0502e", + "metadata": {}, + "source": [ + "### Using the `key_path` Parameter\n", + "\n", + "Consider the following config file, where the desired keys and values are not at the top level.\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "7ce0e917-f0d0-4302-9c8c-b136ffc5410a", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "keys:\n", + " to:\n", + " config:\n", + " message: \"{{ greeting }}, {{ recipient }}!\"\n" + ] + } + ], + "source": [ + "%%bash\n", + "cat fixtures/config/keys-config.yaml" + ] + }, + { + "cell_type": "markdown", + "id": "633778ca-391c-4ba4-b3b4-b3f599af5d41", + "metadata": {}, + "source": [ + "The `key_path` parameter allows only a portion of the config, identified by following a given list of keys, to be written to a file or, in this case, to `stdout`. Note that the key-value pairs from the update config are used to render values, but don't appear in the config written to `stdout`.\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "00150efa-848c-44eb-ac0c-dab3845546b8", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "message: Good morning, Venus!\n" + ] + } + ], + "source": [ + "_ = config.realize(\n", + " input_config=\"fixtures/config/keys-config.yaml\",\n", + " update_config={\"greeting\": \"Good morning\", \"recipient\": \"Venus\"},\n", + " output_file=None,\n", + " output_format='yaml',\n", + " key_path=['keys', 'to', 'config']\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "903ea23f-4625-48f0-9efc-1c0a106d5bf6", + "metadata": {}, + "source": [ + "### Using the `values_needed` Parameter\n", + "\n", + "Consider the config file below, which contains unrendered Jinja2 expressions.\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "928a23d7-8ba9-4217-935d-01563bb36cb6", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "&memo\n", + " sender_id = \"{{ id }}\"\n", + " message = \"{{ greeting }}, {{ recipient }}!\"\n", + " sent = .FALSE.\n", + "/\n" + ] + } + ], + "source": [ + "%%bash\n", + "cat fixtures/config/base-config.nml" + ] + }, + { + "cell_type": "markdown", + "id": "7d878080-55ae-4233-bb65-37cfa5ef7cff", + "metadata": {}, + "source": [ + "Setting `values_needed` to `True` will allow logging of keys that contain unrendered Jinja2 expressions and their values. A logger needs to be initialized for this information to be displayed. The config is not written and the returned `dict` is empty.\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "d22c692d-e98e-4f88-bdac-a369f0a1962f", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[2024-09-23T16:13:23] INFO Keys that are complete:\n", + "[2024-09-23T16:13:23] INFO memo\n", + "[2024-09-23T16:13:23] INFO memo.sent\n", + "[2024-09-23T16:13:23] INFO \n", + "[2024-09-23T16:13:23] INFO Keys with unrendered Jinja2 variables/expressions:\n", + "[2024-09-23T16:13:23] INFO memo.sender_id: {{ id }}\n", + "[2024-09-23T16:13:23] INFO memo.message: {{ greeting }}, {{ recipient }}!\n" + ] + }, + { + "data": { + "text/plain": [ + "{}" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "config.realize(\n", + " input_config='fixtures/config/base-config.nml',\n", + " output_file=None,\n", + " output_format='nml',\n", + " values_needed=True\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "ddbc989a-a9ed-4649-8e87-f74d1ff43b89", + "metadata": {}, + "source": [ + "### Using the `total` Parameter\n", + "\n", + "The `total` parameter is used to specify that all Jinja2 expressions must be rendered before the final config is written. Consider the config below which contains multiple expressions.\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "647538d9-cd22-4f94-b15b-c34d68a324da", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "&memo\n", + " sender_id = \"{{ id }}\"\n", + " message = \"{{ greeting }}, {{ recipient }}!\"\n", + " sent = .FALSE.\n", + "/\n" + ] + } + ], + "source": [ + "%%bash\n", + "cat fixtures/config/base-config.nml" + ] + }, + { + "cell_type": "markdown", + "id": "7c0d64be-c97c-4d07-b636-54b1c8bb5d0c", + "metadata": {}, + "source": [ + "As was shown in the Updating Configs section, by default not all Jinja2 expressions are required to be rendered. However, when `total` is set to `True` and not enough values are provided to fully realize the config, a `UWConfigRealizeError` is raised. Notice that values are provided for `greeting` and `recipient`, but not for `id`.\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "id": "ba360102-f558-4dba-b0b6-c6f550c7d40f", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Config could not be totally realized\n" + ] + } + ], + "source": [ + "try:\n", + " config.realize(\n", + " input_config='fixtures/config/base-config.nml',\n", + " update_config={\"memo\": {\"greeting\":\"Salutations\", \"recipient\":\"Mars\", \"sent\":True}},\n", + " output_file='tmp/config-total.nml',\n", + " total=True\n", + " )\n", + "except Exception as e:\n", + " print(e)" + ] + }, + { + "cell_type": "markdown", + "id": "4048efdd-42f4-4400-bdba-1c8e0001d8f6", + "metadata": {}, + "source": [ + "With all values provided to fully render the config, `realize()` writes the complete config without error.\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "id": "e891a446-f699-460d-b4a8-568d9d4cf631", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'memo': {'sender_id': '321',\n", + " 'message': 'Salutations, Mars!',\n", + " 'sent': True,\n", + " 'greeting': 'Salutations',\n", + " 'recipient': 'Mars',\n", + " 'id': 321}}" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "config.realize(\n", + " input_config='fixtures/config/base-config.nml',\n", + " update_config={\"memo\": {\"greeting\":\"Salutations\", \"recipient\":\"Mars\", \"sent\":True, \"id\":321}},\n", + " output_file='tmp/config-total.nml',\n", + " total=True\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "07876d41-dad2-4c30-adba-f635050708ed", + "metadata": {}, + "source": [ + "The newly created config file is free from any unrendered Jinja2 expressions.\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "id": "aeab0ec5-7e6e-4309-b484-4de5dd9324b5", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "&memo\n", + " sender_id = '321'\n", + " message = 'Salutations, Mars!'\n", + " sent = .true.\n", + " greeting = 'Salutations'\n", + " recipient = 'Mars'\n", + " id = 321\n", + "/\n" + ] + } + ], + "source": [ + "%%bash\n", + "cat tmp/config-total.nml" + ] + }, + { + "cell_type": "markdown", + "id": "e121dab6-5d16-40b0-aa16-161c329a8e9a", + "metadata": {}, + "source": [ + "## Realizing Configs to a Dictionary\n", + "\n", + "The `config.realize_to_dict()` function has the ability to manipulate config values, and returns the config as a Python `dict` just as `realize()` does. However, a config won't be written to a file or to `stdout`. Like `realize()`, input or update configs can be Python dictionaries, UW `Config` objects, or files like the one below.\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "id": "bdc98ce1-e213-41ac-b1f2-03bd52238e30", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "greeting: Hello\n", + "recipient: World\n" + ] + } + ], + "source": [ + "%%bash\n", + "cat fixtures/config/get-config.yaml" + ] + }, + { + "cell_type": "markdown", + "id": "eeacc37f-b808-4d0e-9e29-ff8aa1fc2ff2", + "metadata": {}, + "source": [ + "`realize_to_dict()` has the same parameters as `realize()`, with the exception of `output_file` and `output_format`. Instead, configs can be manipulated or converted to a `dict` without the need to specify an output file or format.\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "id": "ae7f648e-6586-4700-87e7-492ca3a02a06", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'id': '456', 'greeting': 'Hello', 'recipient': 'World'}" + ] + }, + "execution_count": 27, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "config.realize_to_dict(\n", + " input_config={\"id\": \"456\"},\n", + " update_config=\"fixtures/config/get-config.yaml\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "adf2f518-f92c-4b67-a865-c01ce0b10a2e", + "metadata": {}, + "source": [ + "For more details on usage and parameters, see the Realizing Configs section above.\n", + "" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python [conda env:DEV-uwtools] *", + "language": "python", + "name": "conda-env-DEV-uwtools-py" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.3" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/notebooks/fixtures/config/base-config.nml b/notebooks/fixtures/config/base-config.nml new file mode 100644 index 000000000..9f30c6505 --- /dev/null +++ b/notebooks/fixtures/config/base-config.nml @@ -0,0 +1,5 @@ +&memo + sender_id = "{{ id }}" + message = "{{ greeting }}, {{ recipient }}!" + sent = .FALSE. +/ diff --git a/notebooks/fixtures/config/get-config.yaml b/notebooks/fixtures/config/get-config.yaml new file mode 100644 index 000000000..be310733b --- /dev/null +++ b/notebooks/fixtures/config/get-config.yaml @@ -0,0 +1,2 @@ +greeting: Hello +recipient: World diff --git a/notebooks/fixtures/config/keys-config.yaml b/notebooks/fixtures/config/keys-config.yaml new file mode 100644 index 000000000..a8f524b40 --- /dev/null +++ b/notebooks/fixtures/config/keys-config.yaml @@ -0,0 +1,4 @@ +keys: + to: + config: + message: "{{ greeting }}, {{ recipient }}!" diff --git a/notebooks/template.ipynb b/notebooks/template.ipynb index e0e9a2e9e..39a447f70 100644 --- a/notebooks/template.ipynb +++ b/notebooks/template.ipynb @@ -9,7 +9,7 @@ "\n", "The `uwtools` API's `template` module provides functions to render Jinja2 templates and to translate atparse templates to Jinja2.\n", "\n", - "For more information, please see the uwtools.api.template Read the Docs page." + "Tested on `uwtools` version 2.4.2. For more information, please see the uwtools.api.template Read the Docs page." ] }, { diff --git a/notebooks/tests/test_config.py b/notebooks/tests/test_config.py new file mode 100644 index 000000000..04260206a --- /dev/null +++ b/notebooks/tests/test_config.py @@ -0,0 +1,73 @@ +import yaml +from testbook import testbook + + +def test_get_config(): + with open("fixtures/config/get-config.yaml", "r", encoding="utf-8") as f: + config1_str = f.read().rstrip() + config1_dict = yaml.safe_load(config1_str) + with testbook("config.ipynb", execute=True) as tb: + assert tb.ref("config1") == config1_dict + assert tb.cell_output_text(5) == config1_str + assert config1_str in tb.cell_output_text(7) + assert tb.cell_output_text(9) == "message:\n greeting: Hi\n recipient: Earth" + + +def test_depth(): + with testbook("config.ipynb", execute=True) as tb: + assert tb.cell_output_text(11) == "greeting=Salutations\nrecipient=Mars" + assert tb.cell_output_text(13) == "Cannot instantiate depth-1 SHConfig with depth-2 config" + assert tb.cell_output_text(15) == "[message]\ngreeting = Salutations\nrecipient = Mars" + assert tb.cell_output_text(17) == "Cannot instantiate depth-2 INIConfig with depth-1 config" + + +def test_realize(): + # Get config file data to compare to cell output. + with open("fixtures/config/get-config.yaml", "r", encoding="utf-8") as f: + config_str = f.read().rstrip() + config_dict = yaml.safe_load(config_str) + with open("fixtures/config/base-config.nml", "r", encoding="utf-8") as f: + update_config_str = f.read().rstrip() + with open("fixtures/config/keys-config.yaml", "r", encoding="utf-8") as f: + keys_config_str = f.read().rstrip() + with testbook("config.ipynb", execute=True) as tb: + with open("tmp/updated-config.nml", "r", encoding="utf-8") as f: + updated_config = f.read().rstrip() + with open("tmp/config-total.nml", "r", encoding="utf-8") as f: + total_config = f.read().rstrip() + # Ensure that cell output text matches expectations. + assert tb.cell_output_text(21) == str(config_dict) + assert tb.cell_output_text(23) == config_str + assert tb.cell_output_text(25) == str(config_dict) + for item in config_dict.items(): + assert item[0] + "=" + item[1] in tb.cell_output_text(27) + assert tb.cell_output_text(29) == update_config_str + updated_dict = ( + "'sender_id': '{{ id }}'", + "'message': 'Salutations, Mars!'", + "'sent': True", + ) + assert all(x in tb.cell_output_text(31) for x in updated_dict) + assert tb.cell_output_text(33) == updated_config + assert tb.cell_output_text(35) == keys_config_str + assert tb.cell_output_text(37) == "message: Good morning, Venus!" + assert tb.cell_output_text(39) == update_config_str + expected_log = ( + "memo.sender_id: {{ id }}", + "memo.message: {{ greeting }}, {{ recipient }}!", + ) + assert all(x in tb.cell_output_text(41) for x in expected_log) + assert tb.cell_output_text(43) == update_config_str + assert tb.cell_output_text(45) == "Config could not be totally realized" + total_dict = ("'sender_id': '321'", "'message': 'Salutations, Mars!'", "'sent': True") + assert all(x in tb.cell_output_text(47) for x in total_dict) + assert tb.cell_output_text(49) == total_config + + +def test_realize_to_dict(): + with open("fixtures/config/get-config.yaml", "r", encoding="utf-8") as f: + config_str = f.read().rstrip() + with testbook("config.ipynb", execute=True) as tb: + assert tb.cell_output_text(51) == config_str + config_out = ("'id': '456'", "'greeting': 'Hello'", "'recipient': 'World'") + assert all(x in tb.cell_output_text(53) for x in config_out)