[WIP] Make a system for keeping docs and file names in sync (#86)

* minimal setup * use real file * io_spec * one more io_spec * Update expected files * repeat * Remove unused `--longitudinal` argument (#88) Remove unused `--longitudinal` argument. * Use file lock to avoid concurrent edits to dataset_description.json (#91) * ENH: use file lock to avoid concurrent edits to dataset_description.json BUG: docker tag in dataset_description was for fmriprep * ENH: Move lock outside the check for file existence Ensures only the first process writes the file * STYLE: fix style issues ENH: Add file lock for the bids ignore file * BUG: docker address is pennlinc not pennbbl * Make all params and models lower-case (#90) * Make all params lower-case. * Make models lower-case too. * Fix things up. * Fix underscores in entities. * fix * Revert some changes. * Update pyafq_recon_external_trk_outputs.txt * minimal setup * Update expected files * temp * outputs * Document all non-scalar outputs * lint --------- Co-authored-by: Taylor Salo <[email protected]> Co-authored-by: Philip Cook <[email protected]>
PennLINC · Aug 26, 2024 · 1753a07 · 1753a07
1 parent baf517c
commit 1753a07
Show file tree

Hide file tree

Showing 36 changed files with 737 additions and 119 deletions.
diff --git a/docs/builtin_workflows.rst b/docs/builtin_workflows.rst
@@ -17,34 +17,59 @@ for the various steps in that workflow. Workflows can be customized
 (see :ref:`building_workflows`).
 
 
-.. note::
-  The MRtrix workflows are identical up to the FOD estimation. In each case the fiber response
-  function is estimated using ``dwi2response dhollander`` :footcite:p:`dhollander2019response`
-  with a mask based on the T1w.
-  The main differences are in
 
-    * the CSD algorithm used in dwi2fod (msmt_csd or ss3t_csd)
-    * whether a T1w-based tissue segmentation is used during tractography
 
-  In the ``*_noACT`` versions of the pipelines, no T1w-based segmentation is used during
-  tractography. Otherwise, cropping is performed at the GM/WM interface, along with backtracking.
+*********
+Workflows
+*********
+
+MRtrix3-based Workflows
+=======================
 
-  In all pipelines, tractography is performed using
-  tckgen_, which uses the iFOD2 probabilistic tracking method to generate 1e7 streamlines with a
-  maximum length of 250mm, minimum length of 30mm, FOD power of 0.33. Weights for each streamline
-  were calculated using SIFT2_ :footcite:p:`smith2015sift2` and were included for while estimating the
-  structural connectivity matrix.
+The MRtrix workflows are identical up to the FOD estimation. In each case the fiber response
+function is estimated using ``dwi2response dhollander`` :footcite:p:`dhollander2019response`
+with a mask based on the T1w.
+The main differences are in
 
+   * the CSD algorithm used in dwi2fod (msmt_csd or ss3t_csd)
+   * whether a T1w-based tissue segmentation is used during tractography
+
+In the ``*_noACT`` versions of the pipelines, no T1w-based segmentation is used during
+tractography. Otherwise, cropping is performed at the GM/WM interface, along with backtracking.
+
+In all pipelines, tractography is performed using
+tckgen_, which uses the iFOD2 probabilistic tracking method to generate 1e7 streamlines with a
+maximum length of 250mm, minimum length of 30mm, FOD power of 0.33. Weights for each streamline
+were calculated using SIFT2_ :footcite:p:`smith2015sift2` and were included for while estimating the
+structural connectivity matrix.
 
 .. warning::
   We don't recommend using ACT with FAST segmentations. The full benefits of ACT
   require very precise tissue boundaries and FAST just doesn't do this reliably
   enough. We strongly recommend the ``hsvs`` segmentation if you're going to
   use ACT. Note that this requires ``--freesurfer-input``
 
-*********
-Workflows
-*********
+.. _mrtrix_dwi_outputs:
+
+MRtrix3 DWI Outputs
+-------------------
+These files are located in the ``dwi/`` directories.
+
+.. csv-table::
+   :header: "File Name", "Description"
+   :file: nonscalars/mrtrix_dwi.csv
+   :widths: 15, 30
+
+.. _mrtrix_anatomical_outputs:
+
+MRtrix3 Anatomical Outputs
+--------------------------
+These files are located ``anat/`` directories.
+
+.. csv-table::
+   :header: "File Name", "Description"
+   :file: nonscalars/mrtrix_anat.csv
+   :widths: 15, 30
 
 .. _mrtrix_multishell_msmt_ACT-hsvs:
 
@@ -57,6 +82,7 @@ gray matter and cerebrospinal fluid using *multi-shell acquisitions*. The white
 used for tractography and the T1w segmentation is used for anatomical constraints :footcite:p:`smith2012anatomically`.
 The T1w segmentation uses the hybrid surface volume segmentation (hsvs) :footcite:p:`smith2020hybrid` and
 requires ``--freesurfer-input``.
+This workflow produces :ref:`mrtrix_dwi_outputs` and :ref:`mrtrix_anatomical_outputs`.
 
 .. _mrtrix_multishell_msmt_ACT-fast:
 
@@ -65,6 +91,7 @@ requires ``--freesurfer-input``.
 
 Identical to :ref:`mrtrix_multishell_msmt_ACT-hsvs` except FSL's FAST is used for
 tissue segmentation. This workflow is not recommended.
+This workflow produces :ref:`mrtrix_dwi_outputs`.
 
 
 .. _mrtrix_multishell_msmt_noACT:
@@ -76,6 +103,7 @@ tissue segmentation. This workflow is not recommended.
 This workflow uses the ``msmt_csd`` algorithm :footcite:p:`msmt5tt` to estimate FODs for white matter,
 gray matter and cerebrospinal fluid using *multi-shell acquisitions*. The white matter FODs are
 used for tractography with no T1w-based anatomical constraints.
+This workflow produces :ref:`mrtrix_dwi_outputs`.
 
 
 .. _mrtrix_singleshell_ss3t_ACT-hsvs:
@@ -90,6 +118,7 @@ and cerebrospinal fluid using *single shell (DTI) acquisitions*. The white matte
 used for tractography and the T1w segmentation is used for anatomical constraints :footcite:p:`smith2012anatomically`.
 The T1w segmentation uses the hybrid surface volume segmentation (hsvs) :footcite:p:`smith2020hybrid` and
 requires ``--freesurfer-input``.
+This workflow produces :ref:`mrtrix_dwi_outputs` and :ref:`mrtrix_anatomical_outputs`.
 
 .. _mrtrix_singleshell_ss3t_ACT-fast:
 
@@ -98,6 +127,7 @@ requires ``--freesurfer-input``.
 
 Identical to :ref:`mrtrix_singleshell_ss3t_ACT-hsvs` except FSL's FAST is used for
 tissue segmentation. This workflow is not recommended.
+This workflow produces :ref:`mrtrix_dwi_outputs`.
 
 .. _mrtrix_singleshell_ss3t_noACT:
 
@@ -108,6 +138,7 @@ This workflow uses the ``ss3t_csd_beta1`` algorithm :footcite:p:`dhollander2016n
 to estimate FODs for white matter,
 and cerebrospinal fluid using *single shell (DTI) acquisitions*. The white matter FODs are
 used for tractography with no T1w-based anatomical constraints.
+This workflow produces :ref:`mrtrix_dwi_outputs`.
 
 .. _pyafq_tractometry:
 
@@ -118,6 +149,16 @@ This workflow uses the AFQ :footcite:p:`pyafq2` implemented in Python :footcite:
 major white matter pathways within the tractography, and then extract tissue properties along
 those pathways. See the `pyAFQ documentation <https://yeatmanlab.github.io/pyAFQ/>`_ .
 
+PyAFQ Outputs
+-------------
+
++------------------------+-------------------------------------------+
+| File Name              | Description                               |
++========================+===========================================+
+| sub-* (directory)      | PyAFQ results direcrory for each subject  |
++------------------------+-------------------------------------------+
+
+
 .. _pyafq_input_trk:
 
 ``mrtrix_multishell_msmt_pyafq_tractometry``
@@ -127,7 +168,16 @@ Identical to :ref:`pyafq_tractometry` except that tractography generated using I
 instead of using pyAFQ's default DIPY tractography.
 This can also be used as an example for how to import tractographies from other
 reconstruciton pipelines to pyAFQ.
+This workflow produces :ref:`mrtrix_dwi_outputs`.
+
+PyAFQ Outputs
+-------------
 
++------------------------+-------------------------------------------+
+| File Name              | Description                               |
++========================+===========================================+
+| sub-* (directory)      | PyAFQ results direcrory for each subject  |
++------------------------+-------------------------------------------+
 
 .. _amico_noddi:
 
@@ -147,6 +197,13 @@ Scalar Maps
    :file: recon_scalars/amico_noddi.csv
    :widths: 15, 10, 30
 
+Other Outputs
+-------------
+.. csv-table::
+   :header: "File Name", "Description"
+   :file: nonscalars/amico_noddi.csv
+   :widths: 15, 30
+
 .. _dsi_studio_gqi:
 
 ``dsi_studio_gqi``
@@ -171,6 +228,12 @@ Scalar Maps
    :file: recon_scalars/dsi_studio_gqi.csv
    :widths: 15, 10, 30
 
+Other Outputs
+-------------
+.. csv-table::
+   :header: "File Name", "Description"
+   :file: nonscalars/dsistudio_gqi.csv
+   :widths: 15, 30
 
 .. _dsi_studio_autotrack:
 
@@ -195,6 +258,21 @@ Diffusion metrics (e.g., dti_fa, gfa, iso,rdi, nrdi02) and shape statistics (e.g
 span, curl, volume, endpoint_radius) are calculated for subject-specific tracts and written out in
 an AutoTrackGQI.csv file.
 
+Scalar Maps
+-----------
+.. csv-table::
+   :header: "Model", "Parameter", "Description"
+   :file: recon_scalars/dsi_studio_gqi.csv
+   :widths: 15, 10, 30
+
+Other Outputs
+-------------
+.. csv-table::
+   :header: "File Name", "Description"
+   :file: nonscalars/dsistudio_autotrack.csv
+   :widths: 15, 30
+
+
 .. _ss3t_autotrack:
 
 ``ss3t_autotrack``
@@ -206,13 +284,27 @@ to estimate FODs for white matter.
 
 This is a good workflow for doing tractometry on low-quality single shell data.
 
+Scalar Maps
+-----------
+.. csv-table::
+   :header: "Model", "Parameter", "Description"
+   :file: recon_scalars/dsi_studio_gqi.csv
+   :widths: 15, 10, 30
+
+Other Outputs
+-------------
+.. csv-table::
+   :header: "File Name", "Description"
+   :file: nonscalars/ss3t_autotrack.csv
+   :widths: 15, 30
+
 .. _tortoise:
 
 ``TORTOISE``
 ============
 
 The TORTOISE :footcite:p:`tortoisev3` software can calculate Tensor and MAPMRI fits,
-along with their many associated scalar maps.
+along with their many associated scalar maps. This workflow only produces scalar maps.
 
 Scalar Maps
 -----------
@@ -221,6 +313,12 @@ Scalar Maps
    :file: recon_scalars/tortoise.csv
    :widths: 15, 10, 30
 
+Other Outputs
+-------------
+.. csv-table::
+   :header: "File Name", "Description"
+   :file: nonscalars/tortoise.csv
+   :widths: 15, 30
 
 .. _dipy_mapmri:
 
@@ -240,6 +338,14 @@ Scalar Maps
    :file: recon_scalars/dipy_mapmri.csv
    :widths: 15, 10, 30
 
+Other Outputs
+-------------
+.. csv-table::
+   :header: "File Name", "Description"
+   :file: nonscalars/dipy_mapmri.csv
+   :widths: 15, 30
+
+
 .. _dipy_dki:
 
 ``dipy_dki``
@@ -254,6 +360,13 @@ Scalar Maps
    :file: recon_scalars/dipy_dki.csv
    :widths: 15, 10, 30
 
+Other Outputs
+-------------
+.. csv-table::
+   :header: "File Name", "Description"
+   :file: nonscalars/dipy_dki.csv
+   :widths: 15, 30
+
 .. _dipy_3dshore:
 
 ``dipy_3dshore``
@@ -302,6 +415,13 @@ Scalar Maps
    :file: recon_scalars/csdsi_3dshore.csv
    :widths: 15, 10, 30
 
+Other Outputs
+-------------
+.. csv-table::
+   :header: "File Name", "Description"
+   :file: nonscalars/csdsi_3dshore.csv
+   :widths: 15, 30
+
 
 .. _hbcd_scalar_maps:
 
@@ -332,6 +452,12 @@ Scalar Maps
    :file: recon_scalars/hbcd_scalar_maps.csv
    :widths: 15, 10, 30
 
+Other Outputs
+-------------
+.. csv-table::
+   :header: "File Name", "Description"
+   :file: nonscalars/dsistudio_gqi.csv
+   :widths: 15, 30
 
 .. _appropriate_schemes:
 

diff --git a/docs/nonscalars/amico_noddi.csv b/docs/nonscalars/amico_noddi.csv
@@ -0,0 +1,2 @@
+\*model-noddi\*_config.pickle.gz,A config file internally used by AMICO.
+\*model-noddi\*param-direction\*_dwimap.fib.gz,"DSI Studio fib format file where the peak directions come from the NODDI fit. The ""qa"" variable is actually ICVF. "
diff --git a/docs/nonscalars/csdsi_3dshore.csv b/docs/nonscalars/csdsi_3dshore.csv
@@ -0,0 +1 @@
+\*_scalarstats.tsv,MAPMRI scalars summarized within WM bundles. The name of the method used to create the bundles is specified after ``bundles-``. 
diff --git a/docs/nonscalars/dipy_dki.csv b/docs/nonscalars/dipy_dki.csv
@@ -0,0 +1 @@
+\*_scalarstats.tsv,DKI scalars summarized within WM bundles. The name of the method used to create the bundles is specified after ``bundles-``. 
diff --git a/docs/nonscalars/dipy_mapmri.csv b/docs/nonscalars/dipy_mapmri.csv
@@ -0,0 +1 @@
+\*_scalarstats.tsv,MAPMRI scalars summarized within WM bundles. The name of the method used to create the bundles is specified after ``bundles-``. 
diff --git a/docs/nonscalars/dsistudio_autotrack.csv b/docs/nonscalars/dsistudio_autotrack.csv
@@ -0,0 +1,5 @@
+\*_streamlines.tck.gz,One tck.gz per bundle. The bundle represented by this file is specified in the ``bundle-`` tag. 
+\*bundles-DSIStudio\*_scalarstats.csv,Statistics on scalars produced by this workflow 
+\*bundles-DSIStudio\*_tdistats.tsv,Statistics on streamline density in voxels
+\*space-T1w\*_dwimap.fib.gz,DSI Studio fib format containing the GQI ODFs used for AutoTrack. 
+\*space-T1w\*_mapping.map.gz,Mapping file produced by DSI Studio.
diff --git a/docs/nonscalars/dsistudio_gqi.csv b/docs/nonscalars/dsistudio_gqi.csv
@@ -0,0 +1,3 @@
+\*_connectivity.mat,MATLAB format mat file containing connectivity matrices for all the selected atlases. This is an hdf5-format file and can be read using ``scipy.io.matlab.loadmat`` in Python. 
+\*space-T1w\*_dwimap.fib.gz,DSI Studio fib format containing the GQI ODFs used for AutoTrack. 
+\*space-T1w\*_mapping.map.gz,Mapping file produced by DSI Studio.
diff --git a/docs/nonscalars/mrtrix_anat.csv b/docs/nonscalars/mrtrix_anat.csv
@@ -0,0 +1,2 @@
+\*space-T1w\*atlas-hsvs\*_dseg.nii.gz,Hybrid Surface/Voume Segmentation in MRtrix3 5tt format. Aligned in coordinate space to ``space-T1w``. 
+\*space-fsnative\*atlas-hsvs\*_dseg.nii.gz,Hybrid Surface/Voume Segmentation in MRtrix3 5tt format. Aligned to the FreeSurfer ``orig.mgz`` image.
diff --git a/docs/nonscalars/mrtrix_dwi.csv b/docs/nonscalars/mrtrix_dwi.csv
@@ -0,0 +1,13 @@
+\*_connectivity.mat,MATLAB format mat file containing connectivity matrices for all the selected atlases. This is an hdf5-format file and can be read using ``scipy.io.matlab.loadmat`` in Python. 
+\*_exemplarbundles.zip,A zip archive containing the output directory from ``connectome2tck``. Unzip this directory and view the exemplar connections using ``mrview`` to ensure that you're seeing the expected shapes of connections. 
+\*_streamlines.tck.gz,Streamlines produced by ``tckgen``. NOTE: these are not saved to the output directory by default. 
+\*model-mtnorm\*param-inliermask\*_dwimap.nii.gz,Inlier mask created by ``mtnormalize``
+\*model-mtnorm\*param-norm\*_dwimap.nii.gz,Inlier mask created by ``mtnormalize``
+\*model-sift2\*_mu.txt,The $\mu$ value that should be used to adjust SIFT2 weights to account for different response functions. 
+\*model-sift2\*_streamlineweights.csv,Per-streamline SIFT2 weight for each streamline in ``streamlines.tck.gz``. 
+\*param-fod\*label-CSF\*_dwimap.mif.gz,FOD for cerebrospinal fluid.
+\*param-fod\*label-CSF\*_dwimap.txt,SH response function for cerebrospinal fluid.
+\*param-fod\*label-GM\*_dwimap.mif.gz,FOD for gray matter.
+\*param-fod\*label-GM\*_dwimap.txt,SH response function for gray matter.
+\*param-fod\*label-WM\*_dwimap.mif.gz,FOD for white matter. These FODs are used as inputs to ``tckgen`` for tractograpy. 
+\*param-fod\*label-WM\*_dwimap.txt,SH response function for white matter. 
diff --git a/docs/nonscalars/ss3t_autotrack.csv b/docs/nonscalars/ss3t_autotrack.csv
@@ -0,0 +1,5 @@
+\*_streamlines.tck.gz,One tck.gz per bundle. The bundle represented by this file is specified in the ``bundle-`` tag. 
+\*bundles-DSIStudio\*_scalarstats.csv,Statistics on scalars produced by this workflow 
+\*bundles-DSIStudio\*_tdistats.tsv,Statistics on streamline density in voxels
+\*space-T1w\*_dwimap.fib.gz,DSI Studio fib format containing the SS3T FODs used for AutoTrack. 
+\*space-T1w\*_mapping.map.gz,Mapping file produced by DSI Studio.
diff --git a/docs/nonscalars/tortoise.csv b/docs/nonscalars/tortoise.csv
@@ -0,0 +1 @@
+\*_scalarstats.tsv,TORTOISE scalars (tensors and MAPMRI) summarized within WM bundles. The name of the method used to create the bundles is specified after ``bundles-``. 
diff --git a/docs/recon_scalars/amico_noddi.csv b/docs/recon_scalars/amico_noddi.csv
@@ -1,4 +1,4 @@
 noddi,direction,Peak directions from NODDI
 noddi,icvf,Intracellular volume fraction from NODDI
 noddi,isovf,Isotropic volume fraction from NODDI
-noddi,od,OD from NODDI
+noddi,od,Orientation dispersion index from NODDI
diff --git a/docs/recon_scalars/csdsi_3dshore.csv b/docs/recon_scalars/csdsi_3dshore.csv
@@ -1,14 +1,14 @@
-3dSHORE,CNR,Contrast to noise ratio for 3dSHORE fit
-3dSHORE,MSD,mean square displacement from MAPMRI
-3dSHORE,NG,Non-Gaussianity from MAPMRI
-3dSHORE,NGpar,Non-Gaussianity parallel from MAPMRI
-3dSHORE,NGperp,Non-Gaussianity perpendicular from MAPMRI
-3dSHORE,QIV,q-space inverse variance from MAPMRI
-3dSHORE,RTAP,Return to axis probability from MAPMRI
-3dSHORE,RTOP,Return to origin probability from MAPMRI
-3dSHORE,RTPP,Return to plane probability from MAPMRI
-3dSHORE,alpha,alpha used when fitting in each voxel
-3dSHORE,lapnorm,Laplacian norm from regularized MAPMRI (MAPL)
-3dSHORE,mapcoeffs,MAPMRI coefficients
-3dSHORE,regularization,regularization of the 3dSHORE fit
-3dSHORE,rsquared,r^2 of the 3dSHORE fit
+3dshore,CNR,Contrast to noise ratio for 3dshore fit
+3dshore,alpha,alpha used when fitting in each voxel
+3dshore,lapnorm,Laplacian norm from regularized MAPMRI (MAPL)
+3dshore,mapcoeffs,MAPMRI coefficients
+3dshore,msd,mean square displacement from MAPMRI
+3dshore,ng,Non-Gaussianity from MAPMRI
+3dshore,ngpar,Non-Gaussianity parallel from MAPMRI
+3dshore,ngperp,Non-Gaussianity perpendicular from MAPMRI
+3dshore,qiv,q-space inverse variance from MAPMRI
+3dshore,r2,r^2 of the 3dshore fit
+3dshore,regularization,regularization of the 3dshore fit
+3dshore,rtap,Return to axis probability from MAPMRI
+3dshore,rtop,Return to origin probability from MAPMRI
+3dshore,rtpp,Return to plane probability from MAPMRI