Merge branch 'dev' into nf-core-template-merge-2.14.1

.gitattributes

@@ -2,3 +2,4 @@
 *.nf.test linguist-language=nextflow
 modules/nf-core/** linguist-generated
 subworkflows/nf-core/** linguist-generated
+tests/**/*nf.test.snap linguist-generated

.github/workflows/ci.yml

@@ -5,11 +5,16 @@ on:
     branches:
       - dev
   pull_request:
+    branches:
+      - dev
+      - master
   release:
     types: [published]
 
 env:
   NXF_ANSI_LOG: false
+  CAPSULE_LOG: none
+  NFTEST_VER: "0.7.3"
 
 concurrency:
   group: "${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}"
@@ -22,25 +27,38 @@ jobs:
     if: "${{ github.event_name != 'push' || (github.event_name == 'push' && github.repository == 'nf-core/spatialvi') }}"
     runs-on: ubuntu-latest
     strategy:
+      fail-fast: false
       matrix:
         NXF_VER:
           - "23.04.0"
           - "latest-everything"
+        test:
+          - tests/pipeline/test_spaceranger_ffpe_v1.nf.test
+          - tests/pipeline/test_spaceranger_ffpe_v2_cytassist.nf.test
+          - tests/pipeline/test_downstream.nf.test
     steps:
       - name: Check out pipeline code
         uses: actions/checkout@0ad4b8fadaa221de15dcec353f45205ec38ea70b # v4
 
+      # Install Nextflow
       - name: Install Nextflow
         uses: nf-core/setup-nextflow@v2
         with:
           version: "${{ matrix.NXF_VER }}"
 
-      - name: Disk space cleanup
-        uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be # v1.3.1
+      # Install nf-test
+      - name: Install nf-test
+        run: |
+          wget -qO- https://code.askimed.com/install/nf-test | bash -s $NFTEST_VER
+          sudo mv nf-test /usr/local/bin/
+
+      # Run nf-test
+      - name: Run nf-test
+        run: nf-test test --profile=test,docker --tap=test.tap ${{ matrix.test }}
 
-      - name: Run pipeline with test data
-        # TODO nf-core: You can customise CI pipeline run tests as required
-        # For example: adding multiple test runs with different parameters
-        # Remember that you can parallelise this by using strategy.matrix
+      # If the test fails, output the software_versions.yml using the 'batcat' utility
+      - name: Output log on failure
+        if: failure()
         run: |
-          nextflow run ${GITHUB_WORKSPACE} -profile test,docker --outdir ./results
+          sudo apt install bat > /dev/null
+          batcat --decorations=always --color=always .nf-test/tests/*/output/pipeline_info/software_versions.yml

.gitignore

@@ -6,3 +6,9 @@ results/
 testing/
 testing*
 *.pyc
+log
+reports
+.nf-test/
+nf-test
+.nf-test*
+test-datasets

.nf-core.yml

@@ -1,2 +1,11 @@
 repository_type: pipeline
 nf_core_version: "2.14.1"
+lint:
+  actions_ci: False
+  files_exist:
+    - conf/igenomes.config
+  files_unchanged:
+    - .gitattributes
+    - assets/nf-core-spatialvi_logo_light.png
+    - docs/images/nf-core-spatialvi_logo_light.png
+    - docs/images/nf-core-spatialvi_logo_dark.png

.prettierignore

@@ -10,3 +10,6 @@ testing/
 testing*
 *.pyc
 bin/
+test-datasets/
+.nf-test/
+*.scss

CHANGELOG.md

@@ -3,14 +3,67 @@
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## v1.0dev - [date]
+## [Unreleased]
 
-Initial release of nf-core/spatialvi, created with the [nf-core](https://nf-co.re/) template.
+Initial release of nf-core/spatialvi, created with the
+[nf-core](https://nf-co.re/) template. This marks the point at which the
+pipeline development was moved to nf-core and NBIS. The pipeline has undergone
+several iterations regarding its functionality and content; there are a
+significant number of changes, of which not all are listed here. In summary, the
+pipeline contains best-practice processing and analyses of pre- and post-Space
+Ranger-processed data, including quality controls, normalisation, dimensionality
+reduction, clustering, differential expression testing as well as output files
+compatible with further downstream analyses and/or exploration in _e.g._
+[TissUUmaps](https://tissuumaps.github.io/) or bespoke user code.
 
 ### `Added`
 
+- Add MultiQC support for Space Ranger outputs [[#70](https://github.com/nf-core/spatialvi/pull/70)]
+- Use the QUARTONOTEBOOK nf-core module instead of local Quarto-based modules [[#68](https://github.com/nf-core/spatialvi/pull/68)]
+- Add support for SpatialData [[$67](https://github.com/nf-core/spatialvi/pull/67)]
+- Add a custom nf-core Quarto template for the downstream analysis reports [[#64](https://github.com/nf-core/spatialvi/pull/64)]
+- Allow input directories `fastq_dir` and `spaceranger_dir` to be specified as tar archives (`.tar.gz`)
+- Add a check to make sure that there are spots left after filtering [[#46](https://github.com/nf-core/spatialvi/issues/46)]
+- Implement tests with nf-test [[#42](https://github.com/nf-core/spatialvi/pull/42)]
+- Replace custom code to download reference with `untar` module [[#44](https://github.com/nf-core/spatialvi/pull/44)]
+- Embed resources in quarto reports [[#43](https://github.com/nf-core/spatialvi/pull/43)]
+- Use a samplesheet for input specification [[#30](https://github.com/nf-core/spatialvi/pull/30), [#31](https://github.com/nf-core/spatialvi/pull/31) and [#45](https://github.com/nf-core/spatialvi/pull/45)]
+- Add Space Ranger pre-processing as an optional pipeline step using the `spaceranger` nf-core module [[#17](https://github.com/nf-core/spatialvi/pull/17) and [#45](https://github.com/nf-core/spatialvi/pull/45)]
+- Add `env/` directory with pipeline-specific container and Conda environment specifications [[#17](https://github.com/nf-core/spatialvi/pull/17) and [#28](https://github.com/nf-core/spatialvi/pull/28)]
+- Use a more standardised practice to find mitochondrial genes [[#30](https://github.com/nf-core/spatialvi/pull/30)]
+- Make pipeline output compatible with TissUUmaps [[#31](https://github.com/nf-core/spatialvi/pull/31)]
+- Add custom Quarto-based reports for all downstream processing [[#31](https://github.com/nf-core/spatialvi/pull/31)]
+- Embed resources in quarto reports [[#43](https://github.com/nf-core/spatialvi/pull/43)]
+
 ### `Fixed`
 
+- [#51](https://github.com/nf-core/spatialvi/issues/51): Fix version export of `leidenalg` and `SpatialDE` Python modules
+- [#38](https://github.com/nf-core/spatialvi/issues/38): Specify manual alignment files in samplesheet
+- [#20](https://github.com/nf-core/spatialvi/issues/20) and [#22](https://github.com/nf-core/spatialvi/issues/22): Add missing Groovy module
+- [#53](https://github.com/nf-core/spatialvi/pull/53): Use ensemble IDs as index in adata.var and fix related
+  issue with SpatialDE
+
 ### `Dependencies`
 
-### `Deprecated`
+Note, since the pipeline is using Nextflow DSL2, each process will be run
+with its own [Biocontainer](https://biocontainers.pro/#/registry). This means
+that on occasion it is entirely possible for the pipeline to be using different
+versions of the same tool.
+
+| Dependency  | Version |
+| ----------- | ------- |
+| `SpatialDE` | 1.1.3   |
+| `leidenalg` | 0.9.1   |
+| `python`    | 3.12.0  |
+| `quarto`    | 1.3.302 |
+| `scanpy`    | 1.9.3   |
+
+### `Removed`
+
+- Streamline pipeline for basic ST data processing; remove SC processing and deconvolution (for now) [[#31](https://github.com/nf-core/spatialvi/pull/31)]
+
+## v0.1.0 - 2023-03-31
+
+Initial release of nf-core/spatialvi, created with the
+[nf-core](https://nf-co.re/) template by the Jackson Laboratory contributors
+(see `README.md` for details).

CITATIONS.md

@@ -10,6 +10,10 @@
 
 ## Pipeline tools
 
+- [AnnData](https://github.com/scverse/anndata)
+
+  > Virshup I, Rybakov S, Theis FJ, Angerer P, Wolf FA. bioRxiv 2021.12.16.473007; doi: https://doi.org/10.1101/2021.12.16.473007
+
 - [FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/)
 
   > Andrews, S. (2010). FastQC: A Quality Control Tool for High Throughput Sequence Data [Online].
@@ -18,6 +22,26 @@
 
   > Ewels P, Magnusson M, Lundin S, Käller M. MultiQC: summarize analysis results for multiple tools and samples in a single report. Bioinformatics. 2016 Oct 1;32(19):3047-8. doi: 10.1093/bioinformatics/btw354. Epub 2016 Jun 16. PubMed PMID: 27312411; PubMed Central PMCID: PMC5039924.
 
+- [Quarto](https://quarto.org/)
+
+  > Allaire J, Teague C, Scheidegger C, Xie Y, Dervieux C. Quarto (2022). doi: 10.5281/zenodo.5960048
+
+- [Scanpy](https://github.com/theislab/scanpy)
+
+  > Wolf F, Angerer P, Theis F. SCANPY: large-scale single-cell gene expression data analysis. Genome Biol 19, 15 (2018). doi: https://doi.org/10.1186/s13059-017-1382-0
+
+- [Space Ranger](https://www.10xgenomics.com/support/software/space-ranger)
+
+  > 10x Genomics Space Ranger 2.1.0 [Online]
+
+- [SpatialData](https://www.biorxiv.org/content/10.1101/2023.05.05.539647v1)
+
+  > Marconato L, Palla G, Yamauchi K, Virshup I, Heidari E, Treis T, Toth M, Shrestha R, Vöhringer H, Huber W, Gerstung M, Moore J, Theis F, Stegle O. SpatialData: an open and universal data framework for spatial omics. bioRxiv 2023.05.05.539647; doi: https://doi.org/10.1101/2023.05.05.539647
+
+- [Squipy](https://www.nature.com/articles/s41592-021-01358-2)
+
+  > Palla G, Spitzer H, Klein M et al. Squidpy: a scalable framework for spatial omics analysis. Nat Methods 19, 171–178 (2022). doi: https://doi.org/10.1038/s41592-021-01358-2
+
 ## Software packaging/containerisation tools
 
 - [Anaconda](https://anaconda.com)

README.md

@@ -19,45 +19,45 @@
 
 ## Introduction
 
-**nf-core/spatialvi** is a bioinformatics pipeline that ...
-
-<!-- TODO nf-core:
-   Complete this sentence with a 2-3 sentence summary of what types of data the pipeline ingests, a brief overview of the
-   major pipeline sections and the types of output it produces. You're giving an overview to someone new
-   to nf-core here, in 15-20 seconds. For an example, see https://github.com/nf-core/rnaseq/blob/master/README.md#introduction
--->
-
-<!-- TODO nf-core: Include a figure that guides the user through the major workflow steps. Many nf-core
-     workflows use the "tube map" design for that. See https://nf-co.re/docs/contributing/design_guidelines#examples for examples.   -->
-<!-- TODO nf-core: Fill in short bullet-pointed list of the default steps in the pipeline -->
-
-1. Read QC ([`FastQC`](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/))
-2. Present QC for raw reads ([`MultiQC`](http://multiqc.info/))
+**nf-core/spatialvi** is a bioinformatics analysis pipeline for Visium spatial
+transcriptomics data from 10x Genomics. It can process and analyse spatial data
+either directly from raw data by running [Space Ranger](https://support.10xgenomics.com/spatial-gene-expression/software/pipelines/latest/what-is-space-ranger)
+or data already processed by Space Ranger. The pipeline currently consists of
+the following steps:
+
+<p align="center">
+    <img title="Spatialvi Workflow" src="docs/images/spatialvi_subway.png" width=60%>
+</p>
+
+0. Raw data processing with Space Ranger (optional)
+1. Quality controls and filtering
+2. Normalisation
+3. Dimensionality reduction and clustering
+4. Differential gene expression testing
+
+The pipeline is built using [Nextflow](https://www.nextflow.io), a workflow tool
+to run tasks across multiple compute infrastructures in a very portable manner.
+It uses Docker/Singularity containers making installation trivial and results
+highly reproducible. The [Nextflow DSL2](https://www.nextflow.io/docs/latest/dsl2.html)
+implementation of this pipeline uses one container per process which makes it
+much easier to maintain and update software dependencies. Where possible, these
+processes have been submitted to and installed from [nf-core/modules](https://github.com/nf-core/modules)
+in order to make them available to all nf-core pipelines, and to everyone within
+the Nextflow community!
+
+On release, automated continuous integration tests run the pipeline on a
+full-sized dataset on the AWS cloud infrastructure. This ensures that the
+pipeline runs on AWS, has sensible resource allocation defaults set to run on
+real-world datasets, and permits the persistent storage of results to benchmark
+between pipeline releases and other analysis sources. The results obtained from
+the full-sized test can be viewed on the [nf-core website](https://nf-co.re/spatialvi/results).
 
 ## Usage
 
 > [!NOTE]
 > If you are new to Nextflow and nf-core, please refer to [this page](https://nf-co.re/docs/usage/installation) on how to set-up Nextflow. Make sure to [test your setup](https://nf-co.re/docs/usage/introduction#how-to-run-a-pipeline) with `-profile test` before running the workflow on actual data.
 
-<!-- TODO nf-core: Describe the minimum required steps to execute the pipeline, e.g. how to prepare samplesheets.
-     Explain what rows and columns represent. For instance (please edit as appropriate):
-
-First, prepare a samplesheet with your input data that looks as follows:
-
-`samplesheet.csv`:
-
-```csv
-sample,fastq_1,fastq_2
-CONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz
-```
-
-Each row represents a fastq file (single-end) or a pair of fastq files (paired end).
-
--->
-
-Now, you can run the pipeline using:
-
-<!-- TODO nf-core: update the following command to include all required parameters for a minimal example -->
+You can run the pipeline using:
 
 ```bash
 nextflow run nf-core/spatialvi \
@@ -67,8 +67,7 @@ nextflow run nf-core/spatialvi \
 ```
 
 > [!WARNING]
-> Please provide pipeline parameters via the CLI or Nextflow `-params-file` option. Custom config files including those provided by the `-c` Nextflow option can be used to provide any configuration _**except for parameters**_;
-> see [docs](https://nf-co.re/usage/configuration#custom-configuration-files).
+> Please provide pipeline parameters via the CLI or Nextflow `-params-file` option. Custom config files including those provided by the `-c` Nextflow option can be used to provide any configuration _**except for parameters**_; see [docs](https://nf-co.re/usage/configuration#custom-configuration-files).
 
 For more details and further functionality, please refer to the [usage documentation](https://nf-co.re/spatialvi/usage) and the [parameter documentation](https://nf-co.re/spatialvi/parameters).
 
@@ -80,11 +79,22 @@ For more details about the output files and reports, please refer to the
 
 ## Credits
 
-nf-core/spatialvi was originally written by Erik Fasterius, Christophe Avenel, Sergii Domanskyi, Jeffrey Chuang, Anuj Srivastava.
+nf-core/spatialvi was originally developed by the Jackson
+Laboratory<sup>1</sup>, up to the [0.1.0](https://github.com/nf-core/spatialvi/releases/tag/0.1.0)
+tag. It was further developed in a collaboration between the [National
+Bioinformatics Infrastructure Sweden](https://nbis.se/) and [National Genomics
+Infrastructure](https://ngisweden.scilifelab.se/) within [SciLifeLab](https://scilifelab.se/);
+it is currently developed and maintained by [Erik Fasterius](https://github.com/fasterius)
+and [Christophe Avenel](https://github.com/cavenel).
 
-We thank the following people for their extensive assistance in the development of this pipeline:
+Many thanks to others who have helped out along the way too, especially [Gregor
+Sturm](https://github.com/grst)!
 
-<!-- TODO nf-core: If applicable, make list of people who have also contributed -->
+_<sup>1</sup> Supported by grants from the US National Institutes of Health
+[U24CA224067](https://reporter.nih.gov/project-details/10261367) and
+[U54AG075941](https://reporter.nih.gov/project-details/10376627). Original
+authors [Dr. Sergii Domanskyi](https://github.com/sdomanskyi), Prof. Jeffrey
+Chuang and Dr. Anuj Srivastava._
 
 ## Contributions and Support
 
@@ -97,8 +107,6 @@ For further information or help, don't hesitate to get in touch on the [Slack `#
 <!-- TODO nf-core: Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file. -->
 <!-- If you use nf-core/spatialvi for your analysis, please cite it using the following doi: [10.5281/zenodo.XXXXXX](https://doi.org/10.5281/zenodo.XXXXXX) -->
 
-<!-- TODO nf-core: Add bibliography of tools and data used in your pipeline -->
-
 An extensive list of references for the tools used by the pipeline can be found in the [`CITATIONS.md`](CITATIONS.md) file.
 
 You can cite the `nf-core` publication as follows:

assets/_extensions/nf-core/_extension.yml

@@ -0,0 +1,25 @@
+title: nf-core Quarto Extension
+author: Erik Fasterius
+version: 1.0.0
+quarto-required: ">=1.2.0"
+contributes:
+  formats:
+    html:
+      code-fold: true
+      df-print: paged
+      embed-resources: true
+      highlight-style: nf-core.theme
+      smooth-scroll: true
+      theme: [default, nf-core.scss]
+      toc: true
+      toc-image: nf-core-spatialvi_logo_light.png
+      toc-location: left
+      template-partials:
+        - toc.html
+    revealjs:
+      code-line-numbers: false
+      embed-resources: true
+      logo: nf-core-spatialvi_logo_light.png
+      slide-level: 2
+      slide-number: false
+      theme: [default, nf-core.scss]

assets/_extensions/nf-core/nf-core-spatialvi_logo_light.png

@@ -0,0 +1 @@
+../../nf-core-spatialvi_logo_light.png