diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index c2d6f0916..2c15b8be8 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -57,7 +57,8 @@ jobs: - name: Run tests shell: bash -l {0} run: | - pytest -r a -v -n 3 --cov=lsst.pipe.base --cov=tests --cov-report=xml --cov-report=term --cov-branch + pytest -r a -v -n 3 --cov=lsst.pipe.base --cov=tests --cov-report=xml --cov-report=term --cov-branch \ + --junitxml=junit.xml -o junit_family=legacy butler register-instrument -h butler transfer-from-graph -h butler retrieve-artifacts-for-quanta -h @@ -66,6 +67,11 @@ jobs: with: files: ./coverage.xml token: ${{ secrets.CODECOV_TOKEN }} + - name: Upload test results to Codecov + if: ${{ !cancelled() }} + uses: codecov/test-results-action@v1 + with: + token: ${{ secrets.CODECOV_TOKEN }} pypi: runs-on: ubuntu-latest diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 795bf0d40..2e5e89843 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -1,13 +1,13 @@ repos: - repo: https://github.com/pre-commit/pre-commit-hooks - rev: v4.6.0 + rev: v5.0.0 hooks: - id: check-yaml - id: end-of-file-fixer - id: trailing-whitespace - id: check-toml - repo: https://github.com/psf/black - rev: 24.4.2 + rev: 24.10.0 hooks: - id: black # It is recommended to specify the latest version of Python @@ -22,10 +22,10 @@ repos: name: isort (python) - repo: https://github.com/astral-sh/ruff-pre-commit # Ruff version. - rev: v0.5.1 + rev: v0.7.4 hooks: - id: ruff - repo: https://github.com/numpy/numpydoc - rev: "v1.7.0" + rev: "v1.8.0" hooks: - id: numpydoc-validation diff --git a/doc/changes/DM-38041.feature.md b/doc/changes/DM-38041.feature.md deleted file mode 100644 index e2ef1ae92..000000000 --- a/doc/changes/DM-38041.feature.md +++ /dev/null @@ -1,3 +0,0 @@ -Add support for initializing processing output runs with just a pipeline graph, not a quantum graph. - -This also moves much of the logic for initializing output runs from `lsst.ctrl.mpexec.PreExecInit` to `PipelineGraph` and `QuantumGraph` methods. diff --git a/doc/changes/DM-40443.removal.rst b/doc/changes/DM-40443.removal.rst deleted file mode 100644 index cb5c690a8..000000000 --- a/doc/changes/DM-40443.removal.rst +++ /dev/null @@ -1,9 +0,0 @@ -Removed deprecated code scheduled to be removed after v27: - -* Removed ``lsst.pipe.base.graphBuilder``. -* Removed ``lsst.pipe.base.pipeTools``. -* Removed ``lsst.pipe.base.BaseConnection.makeDatasetType`` -* Removed ``Pipeline.toExpandedPipeline`` (replaced by ``to_graph``). -* Removed ``PipelineDatasetTypes`` and ``TaskDatasetTypes``. -* Removed ``QuantumGraphBuilderError``. -* APIs no longer accept ``TaskDef``. diff --git a/doc/changes/DM-41605.feature.md b/doc/changes/DM-41605.feature.md deleted file mode 100644 index 4ee129dce..000000000 --- a/doc/changes/DM-41605.feature.md +++ /dev/null @@ -1,7 +0,0 @@ -Add functionality to aggregate multiple `QuantumProvenanceGraph.Summary` -objects into one `Summary` for a wholistic report. - -While the `QuantumProvenanceGraph` was designed to resolve processing over -dataquery-identified groups, `QuantumProvenanceGraph.aggregate` is designed to -combine multiple group-level reports into one which totals the successes, -issues and failures over the same section of pipeline. \ No newline at end of file diff --git a/doc/changes/DM-41711.feature.md b/doc/changes/DM-41711.feature.md deleted file mode 100644 index 10c3ef1f1..000000000 --- a/doc/changes/DM-41711.feature.md +++ /dev/null @@ -1,11 +0,0 @@ -Create a QuantumProvenanceGraph, which details the status of every quantum -and dataset over multiple attempts at executing graphs, noting when quanta -have been recovered. - -Step through all the quantum graphs associated with certain tasks or -processing steps. For each graph/attempt, the status of each quantum and -dataset is recorded in `QuantumProvenanceGraph.add_new_graph` and outcomes -of quanta over multiple runs are resolved in -`QuantumProvenanceGraph.resolve_duplicates`. At the end of this process, -we can combine all attempts into a summary. This serves to answer the -question "What happened to this data ID?" in a wholistic sense. diff --git a/doc/changes/DM-43020.misc.rst b/doc/changes/DM-43020.misc.rst deleted file mode 100644 index 6b7f9c04f..000000000 --- a/doc/changes/DM-43020.misc.rst +++ /dev/null @@ -1,2 +0,0 @@ -Add ``pipe.base.utils.RegionTimeInfo``, a container for serializing pairs of sky region and timespan. -It's intended for several specific applications when running the AP pipeline. diff --git a/doc/changes/DM-43960.misc.rst b/doc/changes/DM-43960.misc.rst deleted file mode 100644 index ecfda22c3..000000000 --- a/doc/changes/DM-43960.misc.rst +++ /dev/null @@ -1,2 +0,0 @@ -Add an optional parameter to PipelineStepTester that lets configs be tweaked before testing. -This is needed for AP pipelines, whose APDB config cannot be defaulted, and is not intended for wide adoption. diff --git a/doc/changes/DM-44091.bugfix.md b/doc/changes/DM-44091.bugfix.md deleted file mode 100644 index 1ec5d55ea..000000000 --- a/doc/changes/DM-44091.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Append failed quanta to a list and then return. Previous version of the human-readable report only reported the first failed quantum by exiting the loop upon finding it. \ No newline at end of file diff --git a/doc/changes/DM-44368.feature.md b/doc/changes/DM-44368.feature.md deleted file mode 100644 index 4bf859c50..000000000 --- a/doc/changes/DM-44368.feature.md +++ /dev/null @@ -1 +0,0 @@ -Include number of expected instances in `pipetask report` task-level summary for the `QuantumGraphExecutionReport`. \ No newline at end of file diff --git a/doc/changes/DM-44583.feature.md b/doc/changes/DM-44583.feature.md deleted file mode 100644 index caa00fc9a..000000000 --- a/doc/changes/DM-44583.feature.md +++ /dev/null @@ -1 +0,0 @@ -Add mocking support for tasks that write regular datasets with config, log, or metadata storage classes. diff --git a/doc/changes/DM-44647.feature.md b/doc/changes/DM-44647.feature.md deleted file mode 100644 index 334d03634..000000000 --- a/doc/changes/DM-44647.feature.md +++ /dev/null @@ -1,6 +0,0 @@ -Add new show_dot functionality. - -This PR refactors the existing pipeline graph show function, implementing a new `parse_display_args` function to handle the parsing of the `--dot` argument. -The `show_dot` function is then implemented to display the pipeline graph as a dot file. -A notable user-visible change is that output dataset types with common dimensions and storage classes will now be grouped together in dot files. -This change was implemented in order to save space for otherwise very large dot files. diff --git a/doc/changes/DM-45457.feature.md b/doc/changes/DM-45457.feature.md deleted file mode 100644 index a1dd1a474..000000000 --- a/doc/changes/DM-45457.feature.md +++ /dev/null @@ -1,3 +0,0 @@ -Remove the prohibition on optional regular (i.e. non-prerequisite) input connections. - -Optional inputs can be declared by passing `minimum=0` in the connection definition. diff --git a/doc/changes/DM-45536.bugfix.md b/doc/changes/DM-45536.bugfix.md deleted file mode 100644 index 6624dcd80..000000000 --- a/doc/changes/DM-45536.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Fix support for task metadata as inputs in the `PipelineTask` mocking system. diff --git a/doc/changes/DM-45701.api.rst b/doc/changes/DM-45701.api.rst deleted file mode 100644 index 4b1151d37..000000000 --- a/doc/changes/DM-45701.api.rst +++ /dev/null @@ -1 +0,0 @@ -Relocated the "dot tools" from ``lsst.ctrl.mpexec.dotTools`` to ``lsst.pipe.base.dot_tools`` unchanged. diff --git a/doc/changes/DM-45722.bugfix.md b/doc/changes/DM-45722.bugfix.md deleted file mode 100644 index 85d82bdf7..000000000 --- a/doc/changes/DM-45722.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Explanatory logs for "initial data ID query returned no rows" now appear as a single log message instead of one entry per line. This improves display in log aggregators, but there is no change to console behavior. diff --git a/doc/changes/DM-45722.misc.md b/doc/changes/DM-45722.misc.md deleted file mode 100644 index c66b066b9..000000000 --- a/doc/changes/DM-45722.misc.md +++ /dev/null @@ -1 +0,0 @@ -Explanatory logs for "initial data ID query returned no rows" are now reported at ERROR, not CRITICAL, level. diff --git a/doc/changes/DM-46064.feature.md b/doc/changes/DM-46064.feature.md deleted file mode 100644 index 4babd4d09..000000000 --- a/doc/changes/DM-46064.feature.md +++ /dev/null @@ -1 +0,0 @@ -Storage class conversions of component dataset types are now supported in pipelines. diff --git a/doc/changes/DM-46351.misc.md b/doc/changes/DM-46351.misc.md deleted file mode 100644 index d5bd0f01b..000000000 --- a/doc/changes/DM-46351.misc.md +++ /dev/null @@ -1 +0,0 @@ -Added a DEBUG-level log message into `_pipeline_graph.py` to signify which task is being run. diff --git a/doc/lsst.pipe.base/CHANGES.rst b/doc/lsst.pipe.base/CHANGES.rst index 9d7eb4d09..d9bdb97bc 100644 --- a/doc/lsst.pipe.base/CHANGES.rst +++ b/doc/lsst.pipe.base/CHANGES.rst @@ -1,3 +1,77 @@ +lsst-pipe-base v28.0.0 (2024-11-21) +=================================== + +New Features +------------ + +- Added support for initializing processing output runs with just a pipeline graph, not a quantum graph. + + This also moves much of the logic for initializing output runs from ``lsst.ctrl.mpexec.PreExecInit`` to ``PipelineGraph`` and ``QuantumGraph`` methods. (`DM-38041 `_) +- Added functionality to aggregate multiple ``QuantumProvenanceGraph.Summary`` objects into one ``Summary`` for a holistic report. + + While the ``QuantumProvenanceGraph`` was designed to resolve processing over dataquery-identified groups, ``QuantumProvenanceGraph.aggregate`` is designed to combine multiple group-level reports into one which totals the successes,issues, and failures over the same section of pipeline. (`DM-41605 `_) +- Created a ``QuantumProvenanceGraph`` class, which details the status of every quantum and dataset over multiple attempts at executing graphs, noting when quanta have been recovered. + + Steps through all the quantum graphs associated with certain tasks or + processing steps. + For each graph/attempt, the status of each quantum and dataset is recorded in ``QuantumProvenanceGraph.add_new_graph`` and outcomes of quanta over multiple runs are resolved in ``QuantumProvenanceGraph.resolve_duplicates``. + At the end of this process, we can combine all attempts into a summary. + This serves to answer the question "What happened to this data ID?" in a holistic sense. (`DM-41711 `_) +- Included the number of expected instances in ``pipetask report`` task-level summary for the `QuantumGraphExecutionReport`. (`DM-44368 `_) +- Added mocking support for tasks that write regular datasets with config, log, or metadata storage classes. (`DM-44583 `_) +- Added new ``show_dot`` functionality. + + Refactored the existing pipeline graph ``show`` function, implementing a new ``parse_display_args`` function to handle the parsing of the ``--dot`` argument. + The ``show_dot`` function is then implemented to display the pipeline graph as a dot file. + A notable user-visible change is that output dataset types with common dimensions and storage classes will now be grouped together in dot files. + This change was implemented in order to save space for otherwise very large dot files. (`DM-44647 `_) +- Removed the prohibition on optional regular (i.e., non-prerequisite) input connections. + + Optional inputs can be declared by passing ``minimum=0`` in the connection definition. (`DM-45457 `_) +- Storage class conversions of component dataset types are now supported in pipelines. (`DM-46064 `_) + + +API Changes +----------- + +- Relocated the "dot tools" from ``lsst.ctrl.mpexec.dotTools`` to ``lsst.pipe.base.dot_tools`` unchanged. (`DM-45701 `_) + + +Bug Fixes +--------- + +- Appended failed quanta to a list and then return for ``pipetask report``. + The previous version of the human-readable report only reported the first failed quantum by exiting the loop upon finding it. (`DM-44091 `_) +- Fixed support for task metadata as inputs in the ``PipelineTask`` mocking system. (`DM-45536 `_) +- Explanatory logs for "initial data ID query returned no rows" now appear as a single log message instead of one entry per line. + This improves display in log aggregators, but there is no change to console behavior. (`DM-45722 `_) + + +Other Changes and Additions +--------------------------- + +- Added ``pipe.base.utils.RegionTimeInfo``, a container for serializing pairs of sky region and timespan. + It's intended for several specific applications when running the AP pipeline. (`DM-43020 `_) +- Added an optional parameter to ``PipelineStepTester`` that lets configs be tweaked before testing. + This is needed for AP pipelines, whose APDB config cannot be defaulted, and is not intended for wide adoption. (`DM-43960 `_) +- Explanatory logs for "initial data ID query returned no rows" are now reported at ``ERROR``, not ``CRITICAL``, level. (`DM-45722 `_) +- Added a DEBUG-level log message into ``_pipeline_graph.py`` to signify which task is being run. (`DM-46351 `_) + + +An API Removal or Deprecation +----------------------------- + +- Removed deprecated code scheduled to be removed after v27: + + * Removed ``lsst.pipe.base.graphBuilder``. + * Removed ``lsst.pipe.base.pipeTools``. + * Removed ``lsst.pipe.base.BaseConnection.makeDatasetType`` + * Removed ``Pipeline.toExpandedPipeline`` (replaced by ``to_graph``). + * Removed ``PipelineDatasetTypes`` and ``TaskDatasetTypes``. + * Removed ``QuantumGraphBuilderError``. + * APIs no longer accept ``TaskDef``. (`DM-40443 `_) + + lsst-pipe-base 27.0.0 (2024-05-29) ================================== diff --git a/doc/lsst.pipe.base/creating-a-pipeline.rst b/doc/lsst.pipe.base/creating-a-pipeline.rst index 0c6e2ce40..08880534c 100644 --- a/doc/lsst.pipe.base/creating-a-pipeline.rst +++ b/doc/lsst.pipe.base/creating-a-pipeline.rst @@ -366,7 +366,7 @@ These bits of information allow campaign management / batch production software sharding_dimensions: visit, detector - label: step2 sharding_dimensions: tract, patch, skymap - + .. _pipeline_creating_imports: