Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: Parallel processing of Dicom to Nifti conversion instead of serial conversion. #166

Closed
LalithShiyam opened this issue Jan 5, 2025 · 2 comments
Assignees

Comments

@LalithShiyam
Copy link
Collaborator

LalithShiyam commented Jan 5, 2025

Issue:
The current DICOM-to-NIfTI conversion process is serialized, making it highly time-consuming for large datasets. This bottleneck significantly hampers inference workflows on large-scale data.

Proposed Solution:
Implement parallel processing for DICOM-to-NIfTI conversion to fully utilize all available CPU cores. The recommended options are:

  • Dask: Leverage its parallel computing capabilities.
  • Multiprocessing: Use Python's multiprocessing library for fine-grained control.

Challenges with Dask Multiprocessing:
Attempting to use Dask's processes scheduler resulted in the following error:

TypeError: cannot pickle '_thread.RLock' object

This error occurs because Dask's multiprocessing scheduler requires objects to be serializable, and some components (e.g., output_manager) cannot be pickled due to underlying locks (_thread.RLock). We can skip the output_manager, but it is needed for the beautiful console updates. And dask's progress bar is not that great to look at.

Next Steps:

  • Either stick with Dask's default threads scheduler for I/O-bound tasks, avoiding serialization issues.
  • Or carefully refactor the code to use Python's multiprocessing for CPU-bound workloads, ensuring all shared objects are serializable.
@LalithShiyam
Copy link
Collaborator Author

@Keyn34 I will not have the bandwidth to work on this. As I have other fires to put off. Kindly look into this, whenever you have time.

@LalithShiyam LalithShiyam removed their assignment Jan 5, 2025
@Keyn34
Copy link
Collaborator

Keyn34 commented Jan 13, 2025

We can also close this, right (#169)?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants