[WIP] Software Platform Modernization (tools)

.github/workflows/python-package.yml

@@ -9,10 +9,12 @@ on:
       - main
       - develop
       - release-*
+      - spm # TODO: remove before merging
   pull_request:
     branches:
       - develop
       - release-*
+      - spm # TODO: remove before merging
   release:
     types:
       - published
@@ -77,12 +79,12 @@ jobs:
       fail-fast: true
       matrix:
         os: ["ubuntu-latest", "macos-latest", "windows-latest"]
-        python-version: ["3.7", "3.13"]
+        python-version: ["3.8", "3.13"]
         exclude:
           # Latest macos runner does not support older Python versions
           # https://github.com/actions/setup-python/issues/852
           - os: macos-latest
-            python-version: "3.7"
+            python-version: "3.8"
     steps:
       - uses: actions/checkout@v4
         with:
@@ -98,10 +100,10 @@ jobs:
           python -m pip install --upgrade pip wheel
           python -m pip install --editable .[dev]
       - name: Downgrade numpy
-        if: ${{ matrix.python-version == '3.7' }}
+        if: ${{ matrix.python-version == '3.8' }}
         run: |
           # test with older numpy version to ensure compatibility
-          python -m pip install numpy==1.15
+          python -m pip install numpy~=1.17.5
       - name: Test with pytest
         run: |
           pytest -v --benchmark-disable -n auto
@@ -127,7 +129,7 @@ jobs:
           platforms: arm64
       - uses: pypa/[email protected]
         env:
-          CIBW_SKIP: cp36-* pp*-win* pp*-macosx* *_i686
+          CIBW_SKIP: cp36-* cp37-* pp*-win* pp*-macosx* *_i686
           CIBW_TEST_SKIP: "*-win_arm64"
           CIBW_ARCHS_LINUX: "x86_64 aarch64"
           CIBW_ARCHS_MACOS: "x86_64 arm64 universal2"

.gitignore

@@ -66,5 +66,6 @@ ipython_config.py
 .ruff_cache
 .vercel
 cryosparc/core.c
+cryosparc/dataset/core.c
 *.dSYM
 cython_debug

.pre-commit-config.yaml

@@ -27,4 +27,5 @@ repos:
     rev: v1.1.391
     hooks:
       - id: pyright
-        additional_dependencies: [cython, httpretty, numpy, pytest, setuptools]
+        additional_dependencies:
+          [cython, httpx, numpy, pydantic, pytest, setuptools]

CHANGELOG.md

@@ -1,5 +1,43 @@
 # Changelog
 
+## Next
+
+- BREAKING: replaced low-level `CryoSPARC.cli`, `CryoSPARC.rtp` and `CryoSPARC.vis` attributes with single unified `CryoSPARC.api`
+- BREAKING: When a `job.start()` or `job.run()` is called for an external job, changing the job connections with `job.add_input`, `job.add_output` or `job.connect` will now trigger an error. Please add all inputs and outputs and connect all inputs before running an external job.
+- BREAKING: `CryoSPARC.download_asset(fileid, target)` no longer accepts a directory target. Must specify a filename.
+- BREAKING: removed `CryoSPARC.get_job_specs()`. Use `CryoSPARC.job_register` instead
+- BREAKING: `CryoSPARC.list_assets()` and `Job.list_assets()` return list of models instead of list of dictionaries, accessible with dot-notation
+  - OLD: `job.list_assets()[0]['filename']`
+  - NEW: `job.list_assets()[0].filename`
+- BREAKING: `CryoSPARC.get_lanes()` now returns a list of models instead of dictionaries
+  - OLD: `cs.get_lanes()[0]['name']`
+  - NEW: `cs.get_lanes()[0].name`
+- BREAKING: `CryoSPARC.get_targets` now returns a list of models instead of dictionaries
+  - OLD: `cs.get_targets()[0]['hostname']`
+  - NEW: `cs.get_targets()[0].hostname`
+  - Some top-level target attributes have also been moved into the `.config` attribute
+- BREAKING: `CryoSPARC.print_job_types` `section` argument renamed to `category`
+  - OLD: `cs.print_job_types(section=["extraction", "refinement"])`
+  - NEW: `cs.print_job_types(category=["extraction", "refinement"])`
+- BREAKING: Restructured schema for Job models, many `Job.doc` properties have been internally rearranged
+- Added: `CryoSPARC.job_register` property
+- Added: `job.load_input()` and `job.load_output()` now accept `"default"`, `"passthrough"` and `"all"` keywords for their `slots` argument
+- Added: `job.alloc_output()` now accepts `dtype_params` argument for fields with dynamic shapes
+- Added: `CryoSPARC.print_job_types` now includes a job stability column
+- Added: `Job.print_output_spec` now includes a passthrough indicator column for results
+- Updated: Improved type definitions
+- Deprecated: When adding external inputs and outputs, expanded slot definitions now expect `"name"` key instead of `"prefix"`, support for which will be removed in a future release.
+  - OLD: `job.add_input("particle", slots=[{"prefix": "component_mode_1", "dtype": "component", "required": True}])`
+  - NEW: `job.add_input("particle", slots=[{"name": "component_mode_1", "dtype": "component", "required": True}])`
+- Deprecated: `license` argument no longer required when creating a `CryoSPARC`
+  instance, will be removed in a future release
+- Deprecated: `external_job.stop()` now expects optional error string instead of boolean, support for boolean errors will be removed in a future release
+- Deprecated: `CryoSPARC.get_job_sections()` will be removed in a future release,
+  use `CryoSPARC.job_register` instead
+- Deprecated: Most functions no longer require a `refresh` argument, including
+  `job.set_param()`, `job.connect()`, `job.disconnect()` and `external_job.save_output()`
+- Deprecated: Attributes `Project.doc`, `Workspace.doc` and `Job.doc` will be removed in a future release, use `.model` attribute instead
+
 ## v4.6.1
 
 - Added: Python 3.13 support

MANIFEST.in

@@ -1,7 +1,7 @@
-include cryosparc/core.pyx
-include cryosparc/dataset.c
-include cryosparc/dataset.pxd
-include cryosparc/lz4.pxd
+include cryosparc/dataset/core.pyx
+include cryosparc/dataset/dataset.c
+include cryosparc/dataset/dataset.pxd
+include cryosparc/dataset/lz4.pxd
 include cryosparc/include/cryosparc-tools/dataset.h
 include cryosparc/include/lz4/lib/lz4.h
 include cryosparc/include/lz4/lib/lz4.c

Makefile

@@ -1,13 +1,13 @@
 PY_EXT_SUFFIX=$(shell python3 -c "import sysconfig; print(sysconfig.get_config_var('EXT_SUFFIX'))")
-TARGET=cryosparc/core$(PY_EXT_SUFFIX)
+TARGET=cryosparc/dataset/core$(PY_EXT_SUFFIX)
 
 all: $(TARGET)
 
 # -----------------------------------------------------------------------------
 #    Primary build target
 # -----------------------------------------------------------------------------
 
-$(TARGET): cryosparc/include/cryosparc-tools/*.h cryosparc/dataset.c cryosparc/*.pyx cryosparc/*.pxd setup.py pyproject.toml
+$(TARGET): cryosparc/include/cryosparc-tools/*.h cryosparc/dataset/dataset.c cryosparc/dataset/*.pyx cryosparc/dataset/*.pxd setup.py pyproject.toml
 	python3 -m setup build_ext -i
 
 # -----------------------------------------------------------------------------

README.md

@@ -19,7 +19,7 @@ See [CONTRIBUTING.md](CONTRIBUTING.md) for full details.
 ### Prerequisites
 
 - Git and Git LFS
-- Python >= 3.7
+- Python >= 3.8
 - Miniconda3
 - C compiler such as GCC or Clang
 
@@ -117,27 +117,23 @@ rm -rf cryosparc/*.so build dist *.egg-info
 Install dependencies into a new conda environment:
 
 ```sh
-conda create -n cryosparc-tools-example -c conda-forge \
-    python=3 numpy==1.18.5 \
-    pyqt=5 libtiff wxPython=4.1.1 adwaita-icon-theme
+conda create -n cryosparc-tools-example -c conda-forge python=3 numpy=1.18.5 \
+    pyqt=5 libtiff wxPython=4.1.1 adwaita-icon-theme 'setuptools<66'  # exclude these dependencies if you don't need cryolo
 conda activate cryosparc-tools-example
-pip install -U pip
-pip install nvidia-pyindex matplotlib~=3.4.0 pandas==1.1.4 notebook
-pip install "cryolo[c11]"
-pip install -e ".[build]"
+pip install cryosparc-tools matplotlib~=3.4.0 pandas~=1.1.0 notebook
+pip install nvidia-pyindex  # exclude last two steps if you don't need cryolo
+pip install 'cryolo[c11]'
 ```
 
 Run the notebook server with the following environment variables:
 
-- `CRYOSPARC_LICENSE_ID` with Structura-issued CryoSPARC license
 - `CRYOSPARC_EMAIL` with a CryoSPARC user account email
 - `CRYOSPARC_PASSWORD` with a CryoSPARC user account password
 
 You may also need to include `LD_LIBRARY_PATH` which includes the location of
 CUDA Toolkit and cuDNN runtime libraries (e.g., `~/miniconda3/envs/tools/lib/python3.8/site-packages/nvidia/*/lib`).
 
 ```
-CRYOSPARC_LICENSE_ID="xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxx" \
 CRYOSPARC_EMAIL="[email protected]" \
 CRYOSPARC_PASSWORD="password123" \
 jupyter notebook