Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: Add data loader class, install at root and data modules #816

Merged
merged 5 commits into from
Jul 11, 2023

Conversation

effigies
Copy link
Member

@effigies effigies commented Jul 10, 2023

Fixes current test failures from using a deprecated importlib.resources API.

Expanded on an ExitStack-based load function that @mgxd used in one of these projects into a class that provides flexible access to package data with readable, unzipped, and cached access methods.

We have the following loaders:

  • niworkflows.load_resource - Generic, can get anything from anywhere
  • niworkflows.data.load - Primary access point for package data
  • niworkflows.tests.data.load_test_data - Convenience for tests
  • niworkflows.interfaces.tests.data.load_test_data - Convenience for interface tests

I would like to figure out how to bundle the library in a zip to actually test that this works when installed that way, but for this PR the goal is to make it easy to think about and select whether we need the resource to be a Path or just Traversable.

This also finally purges pkg_resources, which we should avoid because we have no runtime dependency on setuptools, and the API is discouraged in third-party packages.

@effigies
Copy link
Member Author

Tests passing. Anybody have time for a review?

Copy link
Contributor

@mgxd mgxd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is cool! If I'm understanding it right, this can be used as a dataloader across the *preps ecosystem?


return "\n".join(doclines)

def readable(self, *segments) -> Traversable:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since it seems like this step is often, if not always, followed with a read_text() call, wdyt about adding another method (read_text?) to serve as some syntactic sugar?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the explicitness of loader.how_to_load(what_to_load).do_something().

@effigies
Copy link
Member Author

If I'm understanding it right, this can be used as a dataloader across the *preps ecosystem?

Yes. I would probably just vendor it in the local data/__init__.py for a few versions, rather than depend on it in niworkflows.

I think it's a handy util but not sure it's stable enough to really want to deal with it as a dependency. I did an initial pass in BIDS, and then while working on this refactor found I wanted to explicitly avoid caching.

Copy link
Member

@oesteban oesteban left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks great!

@effigies effigies merged commit 1f6203c into nipreps:master Jul 11, 2023
17 checks passed
@effigies effigies deleted the mnt/data_module_and_docs branch July 11, 2023 12:44
@oesteban
Copy link
Member

Yes. I would probably just vendor it in the local data/__init__.py for a few versions, rather than depend on it in niworkflows.

Just to confirm - "vendor it in" means copying the infrastructure across packages (for a few versions)?

@effigies
Copy link
Member Author

Yes

oesteban added a commit to nipreps/sdcflows that referenced this pull request Jul 11, 2023
Proposed in nipreps/niworkflows#816.

Resolves: #377.

Co-authored-by: Chris Markiewicz <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants