From 8798b8a00f5a863df8c57ceb26680a454ccc485e Mon Sep 17 00:00:00 2001 From: Tim Pillinger <26465611+wxtim@users.noreply.github.com> Date: Fri, 31 May 2024 11:10:59 +0100 Subject: [PATCH 1/9] Updated docs about using ROSE_DATAC Update src/reference/job-script-vars/var-list.txt Update src/workflow-design-guide/general-principles.rst Co-authored-by: Ronnie Dutta <61982285+MetRonnie@users.noreply.github.com> Update src/reference/job-script-vars/var-list.txt Co-authored-by: Ronnie Dutta <61982285+MetRonnie@users.noreply.github.com> response to review --- src/reference/job-script-vars/var-list.txt | 4 ++- .../general-principles.rst | 32 ++++++++++--------- 2 files changed, 20 insertions(+), 16 deletions(-) diff --git a/src/reference/job-script-vars/var-list.txt b/src/reference/job-script-vars/var-list.txt index c3ac408816..4a1a580540 100644 --- a/src/reference/job-script-vars/var-list.txt +++ b/src/reference/job-script-vars/var-list.txt @@ -25,7 +25,9 @@ CYLC_WORKFLOW_RUN_DIR # Location of the run directory in CYLC_WORKFLOW_HOST # Host running the workflow process CYLC_WORKFLOW_OWNER # User ID running the workflow process -CYLC_WORKFLOW_SHARE_DIR # Workflow (or task!) shared directory (see below) +CYLC_WORKFLOW_SHARE_DIR # Workflow (or task!) shared directory +CYLC_SHARE_CYCLE_DIR # The share/cycle/ + # for this task. CYLC_WORKFLOW_UUID # Workflow UUID string CYLC_WORKFLOW_WORK_DIR # Workflow work directory (see below) diff --git a/src/workflow-design-guide/general-principles.rst b/src/workflow-design-guide/general-principles.rst index f41ecbc29f..93cd6a53a1 100644 --- a/src/workflow-design-guide/general-principles.rst +++ b/src/workflow-design-guide/general-principles.rst @@ -451,27 +451,29 @@ Primarily, for self-containment (see :ref:`Self-Contained Workflows`) shared IO paths should be under the *workflow share directory*, the location of which is passed to all tasks as ``$CYLC_WORKFLOW_SHARE_DIR``. -The ``rose task-env`` utility can provide additional environment -variables that refer to static and cyclepoint-specific locations under the -workflow share directory. +The ``$CYLC_SHARE_CYCLE_DIR`` variable can be used to target +cyclepoint-specific locations under the workflow share directory. -.. code-block:: cylc +Sometimes it may be useful to refer to other cycles - to do this use +``isodatetime``: - [runtime] - [[my-task]] - env-script = $(eval rose task-env -T P1D -T P2D) +.. code-block:: bash -For a current cycle point of ``20170105`` this will make the following -variables available to tasks: + # Cylc task script + CYCLE_POINT_MINUS_P1D=$(isodatetime "${CYLC_TASK_CYCLE_POINT} --offset -P1D)" + SHARE_CYCLE_DIR_MINUS_P1D="${CYLC_WORKFLOW_SHARE_DIR}/cycle/${CYCLE_POINT_MINUS_P1D}" + mkdir -p "${SHARE_CYCLE_DIR_MINUS_P1D}" + echo "I believe in ..." > "${SHARE_CYCLE_DIR_MINUS_P1D}/somefile" -.. code-block:: bash +See ``isodatetime --help`` for usage instructions. + +.. versionadded:: 8.3.1 - ROSE_DATA=$CYLC_WORKFLOW_SHARE_DIR/data - ROSE_DATAC=$CYLC_WORKFLOW_SHARE_DIR/cycle/20170105 - ROSE_DATACP1D=$CYLC_WORKFLOW_SHARE_DIR/cycle/20170104 - ROSE_DATACP2D=$CYLC_WORKFLOW_SHARE_DIR/cycle/20170103 + ``$CYLC_WORKFLOW_SHARE_DIR`` and the ``isodatetime`` are designed to + provide a Cylc internal replacement for the use of ``rose task-env`` + to provide ``$ROSE_DATAC`` and derived variables. -Subdirectories of ``$ROSE_DATAC`` etc. should be agreed between +Subdirectories of ``$CYLC_SHARE_CYCLE_DIR`` etc. should be agreed between different sub-systems of the workflow; typically they are named for the file-generating tasks, and the file-consuming tasks should know to look there. From 1cbeeba1af0a32472111fb732d23a25da548037d Mon Sep 17 00:00:00 2001 From: Tim Pillinger <26465611+wxtim@users.noreply.github.com> Date: Fri, 10 Jan 2025 13:04:36 +0000 Subject: [PATCH 2/9] Apply suggestions from code review Co-authored-by: Ronnie Dutta <61982285+MetRonnie@users.noreply.github.com> --- src/workflow-design-guide/general-principles.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/workflow-design-guide/general-principles.rst b/src/workflow-design-guide/general-principles.rst index 93cd6a53a1..c73d54fcc3 100644 --- a/src/workflow-design-guide/general-principles.rst +++ b/src/workflow-design-guide/general-principles.rst @@ -467,13 +467,13 @@ Sometimes it may be useful to refer to other cycles - to do this use See ``isodatetime --help`` for usage instructions. -.. versionadded:: 8.3.1 +.. versionadded:: 8.5.0 ``$CYLC_WORKFLOW_SHARE_DIR`` and the ``isodatetime`` are designed to provide a Cylc internal replacement for the use of ``rose task-env`` to provide ``$ROSE_DATAC`` and derived variables. -Subdirectories of ``$CYLC_SHARE_CYCLE_DIR`` etc. should be agreed between +Subdirectories of ``$CYLC_SHARE_CYCLE_DIR`` should be agreed between different sub-systems of the workflow; typically they are named for the file-generating tasks, and the file-consuming tasks should know to look there. From 07c183a5aaee6ca20cd41d019a90d2cad3dc5813 Mon Sep 17 00:00:00 2001 From: Tim Pillinger <26465611+wxtim@users.noreply.github.com> Date: Fri, 10 Jan 2025 13:04:46 +0000 Subject: [PATCH 3/9] Update src/workflow-design-guide/general-principles.rst Co-authored-by: Ronnie Dutta <61982285+MetRonnie@users.noreply.github.com> --- src/workflow-design-guide/general-principles.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/workflow-design-guide/general-principles.rst b/src/workflow-design-guide/general-principles.rst index c73d54fcc3..1dc97f90a9 100644 --- a/src/workflow-design-guide/general-principles.rst +++ b/src/workflow-design-guide/general-principles.rst @@ -469,7 +469,8 @@ See ``isodatetime --help`` for usage instructions. .. versionadded:: 8.5.0 - ``$CYLC_WORKFLOW_SHARE_DIR`` and the ``isodatetime`` are designed to + ``$CYLC_SHARE_CYCLE_DIR`` (used in combination with the ``isodatetime`` + command) is designed to provide a Cylc internal replacement for the use of ``rose task-env`` to provide ``$ROSE_DATAC`` and derived variables. From 219b1ce5dce3205fc7d1855d93bdc8ae0e7d6a51 Mon Sep 17 00:00:00 2001 From: Tim Pillinger <26465611+wxtim@users.noreply.github.com> Date: Mon, 13 Jan 2025 11:41:12 +0000 Subject: [PATCH 4/9] change to CYLC_TASK_SHARE_CYCLE_DIR --- src/reference/job-script-vars/var-list.txt | 2 +- src/workflow-design-guide/general-principles.rst | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/src/reference/job-script-vars/var-list.txt b/src/reference/job-script-vars/var-list.txt index 4a1a580540..074f0b7524 100644 --- a/src/reference/job-script-vars/var-list.txt +++ b/src/reference/job-script-vars/var-list.txt @@ -26,7 +26,7 @@ CYLC_WORKFLOW_HOST # Host running the workflow process CYLC_WORKFLOW_OWNER # User ID running the workflow process CYLC_WORKFLOW_SHARE_DIR # Workflow (or task!) shared directory -CYLC_SHARE_CYCLE_DIR # The share/cycle/ +CYLC_TASK_SHARE_CYCLE_DIR # The share/cycle/ # for this task. CYLC_WORKFLOW_UUID # Workflow UUID string CYLC_WORKFLOW_WORK_DIR # Workflow work directory (see below) diff --git a/src/workflow-design-guide/general-principles.rst b/src/workflow-design-guide/general-principles.rst index 1dc97f90a9..2dc586d6e6 100644 --- a/src/workflow-design-guide/general-principles.rst +++ b/src/workflow-design-guide/general-principles.rst @@ -451,7 +451,7 @@ Primarily, for self-containment (see :ref:`Self-Contained Workflows`) shared IO paths should be under the *workflow share directory*, the location of which is passed to all tasks as ``$CYLC_WORKFLOW_SHARE_DIR``. -The ``$CYLC_SHARE_CYCLE_DIR`` variable can be used to target +The ``$CYLC_TASK_SHARE_CYCLE_DIR`` variable can be used to target cyclepoint-specific locations under the workflow share directory. Sometimes it may be useful to refer to other cycles - to do this use @@ -469,12 +469,12 @@ See ``isodatetime --help`` for usage instructions. .. versionadded:: 8.5.0 - ``$CYLC_SHARE_CYCLE_DIR`` (used in combination with the ``isodatetime`` + ``$CYLC_TASK_SHARE_CYCLE_DIR`` (used in combination with the ``isodatetime`` command) is designed to provide a Cylc internal replacement for the use of ``rose task-env`` to provide ``$ROSE_DATAC`` and derived variables. -Subdirectories of ``$CYLC_SHARE_CYCLE_DIR`` should be agreed between +Subdirectories of ``$CYLC_TASK_SHARE_CYCLE_DIR`` should be agreed between different sub-systems of the workflow; typically they are named for the file-generating tasks, and the file-consuming tasks should know to look there. From 3a552a966bb727d1ba432d52b08e492cd7040c61 Mon Sep 17 00:00:00 2001 From: Tim Pillinger <26465611+wxtim@users.noreply.github.com> Date: Mon, 13 Jan 2025 12:56:58 +0000 Subject: [PATCH 5/9] fix --- src/reference/job-script-vars/var-list.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/reference/job-script-vars/var-list.txt b/src/reference/job-script-vars/var-list.txt index 074f0b7524..03b942f713 100644 --- a/src/reference/job-script-vars/var-list.txt +++ b/src/reference/job-script-vars/var-list.txt @@ -26,7 +26,6 @@ CYLC_WORKFLOW_HOST # Host running the workflow process CYLC_WORKFLOW_OWNER # User ID running the workflow process CYLC_WORKFLOW_SHARE_DIR # Workflow (or task!) shared directory -CYLC_TASK_SHARE_CYCLE_DIR # The share/cycle/ # for this task. CYLC_WORKFLOW_UUID # Workflow UUID string CYLC_WORKFLOW_WORK_DIR # Workflow work directory (see below) @@ -56,6 +55,7 @@ CYLC_TASK_LOG_ROOT # The job script path # e.g. ~/cylc-run/foo/log/job/20110511T1800Z/t1/01/job CYLC_TASK_WORK_DIR # Location of task work directory (see below) # e.g. ~/cylc-run/foo/work/20110511T1800Z/t1 +CYLC_TASK_SHARE_CYCLE_DIR # The share/cycle/ CYLC_TASK_NAMESPACE_HIERARCHY # Linearised family namespace of the task, # e.g. root postproc t1 From 176d77c06e541d7476e037683b7b31ed150790f7 Mon Sep 17 00:00:00 2001 From: Tim Pillinger <26465611+wxtim@users.noreply.github.com> Date: Tue, 14 Jan 2025 13:02:04 +0000 Subject: [PATCH 6/9] Update src/reference/job-script-vars/var-list.txt Co-authored-by: Ronnie Dutta <61982285+MetRonnie@users.noreply.github.com> --- src/reference/job-script-vars/var-list.txt | 1 - 1 file changed, 1 deletion(-) diff --git a/src/reference/job-script-vars/var-list.txt b/src/reference/job-script-vars/var-list.txt index 03b942f713..68f507cf63 100644 --- a/src/reference/job-script-vars/var-list.txt +++ b/src/reference/job-script-vars/var-list.txt @@ -26,7 +26,6 @@ CYLC_WORKFLOW_HOST # Host running the workflow process CYLC_WORKFLOW_OWNER # User ID running the workflow process CYLC_WORKFLOW_SHARE_DIR # Workflow (or task!) shared directory - # for this task. CYLC_WORKFLOW_UUID # Workflow UUID string CYLC_WORKFLOW_WORK_DIR # Workflow work directory (see below) From 879ebd65ecb705d5814223bf9c00eb7bf5c1033f Mon Sep 17 00:00:00 2001 From: Tim Pillinger <26465611+wxtim@users.noreply.github.com> Date: Tue, 14 Jan 2025 13:02:10 +0000 Subject: [PATCH 7/9] Update src/reference/job-script-vars/var-list.txt Co-authored-by: Ronnie Dutta <61982285+MetRonnie@users.noreply.github.com> --- src/reference/job-script-vars/var-list.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/src/reference/job-script-vars/var-list.txt b/src/reference/job-script-vars/var-list.txt index 68f507cf63..04bf12ef12 100644 --- a/src/reference/job-script-vars/var-list.txt +++ b/src/reference/job-script-vars/var-list.txt @@ -55,6 +55,7 @@ CYLC_TASK_LOG_ROOT # The job script path CYLC_TASK_WORK_DIR # Location of task work directory (see below) # e.g. ~/cylc-run/foo/work/20110511T1800Z/t1 CYLC_TASK_SHARE_CYCLE_DIR # The share/cycle/ + # for this task. CYLC_TASK_NAMESPACE_HIERARCHY # Linearised family namespace of the task, # e.g. root postproc t1 From f860c4771bbd04c60dd52d9578a20f6f085176c8 Mon Sep 17 00:00:00 2001 From: Tim Pillinger <26465611+wxtim@users.noreply.github.com> Date: Tue, 14 Jan 2025 15:42:40 +0000 Subject: [PATCH 8/9] Update src/reference/job-script-vars/var-list.txt Co-authored-by: Ronnie Dutta <61982285+MetRonnie@users.noreply.github.com> --- src/reference/job-script-vars/var-list.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/reference/job-script-vars/var-list.txt b/src/reference/job-script-vars/var-list.txt index 04bf12ef12..3ba77a8525 100644 --- a/src/reference/job-script-vars/var-list.txt +++ b/src/reference/job-script-vars/var-list.txt @@ -27,7 +27,7 @@ CYLC_WORKFLOW_OWNER # User ID running the workflow process CYLC_WORKFLOW_SHARE_DIR # Workflow (or task!) shared directory CYLC_WORKFLOW_UUID # Workflow UUID string -CYLC_WORKFLOW_WORK_DIR # Workflow work directory (see below) +CYLC_WORKFLOW_WORK_DIR # Workflow work directory CYLC_TASK_JOB # Job identifier expressed as # CYCLE-POINT/TASK-NAME/SUBMIT-NUMBER From eaae0cc0e47e94efcf6f24595d1f18e26311d673 Mon Sep 17 00:00:00 2001 From: Tim Pillinger <26465611+wxtim@users.noreply.github.com> Date: Wed, 15 Jan 2025 07:15:20 +0000 Subject: [PATCH 9/9] Update src/reference/job-script-vars/var-list.txt Co-authored-by: Oliver Sanders --- src/reference/job-script-vars/var-list.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/reference/job-script-vars/var-list.txt b/src/reference/job-script-vars/var-list.txt index 3ba77a8525..2b2e5e3055 100644 --- a/src/reference/job-script-vars/var-list.txt +++ b/src/reference/job-script-vars/var-list.txt @@ -54,7 +54,7 @@ CYLC_TASK_LOG_ROOT # The job script path # e.g. ~/cylc-run/foo/log/job/20110511T1800Z/t1/01/job CYLC_TASK_WORK_DIR # Location of task work directory (see below) # e.g. ~/cylc-run/foo/work/20110511T1800Z/t1 -CYLC_TASK_SHARE_CYCLE_DIR # The share/cycle/ +CYLC_TASK_SHARE_CYCLE_DIR # The share/cycle/ directory # for this task. CYLC_TASK_NAMESPACE_HIERARCHY # Linearised family namespace of the task,