Gb/s3docs #189
Gb/s3docs #189
Conversation
…te file handler including fsspec for zero setup s3 access
grantbuster
force-pushed
the
gb/s3docs
branch
from
9139acf to
984befd
Compare
| - os: ubuntu-latest | ||
| python-version: 3.9 | ||
| - os: ubuntu-latest | ||
| python-version: 3.8 |
There was a problem hiding this comment.
Do we need to have Python 3.8 and 3.9 tests for this? Probably Python 3.10 across the three OS is enough, but I'll leave that up to you
There was a problem hiding this comment.
Probably not but it's free testing... I think leave it for now but i don't feel that strongly. We might also want to add new main tests for python 3.11-3.13 but we can leave that for future PRs.
There was a problem hiding this comment.
Well, it's definitely not free - there are usage limits on GitHub, but you're probably right that we wouldn't hit them unless we do extensive development on rex
| years : list, optional | ||
| List of integer years to access, by default None | ||
| res_cls : obj | ||
| Resource class to use to open and access resource data | ||
| hsds : bool | ||
| Boolean flag to use h5pyd to handle .h5 'files' hosted on AWS | ||
| behind HSDS | ||
| behind HSDS, by default False. This is now redundant; file paths |
There was a problem hiding this comment.
Is the idea to drop this option in the future sometime? Or should we leave it in case someone wants to use HSDS for a non /nrel domain? If the latter, we can probably say something to the extent of "This is redundant for the /nrel domain"
There was a problem hiding this comment.
Yeah i'm not 100% sure. My thinking was basically to not deprecate a kwarg right now until we've built experience with the new automatic parsing features. Trying not to make breaking changes in this PR, just new features. I would be open to dropping it in the future, maybe in an intermediate version release?
Co-authored-by: Paul Pinchuk <[email protected]>
Co-authored-by: Paul Pinchuk <[email protected]>
Automatic parsing of hsds/s3/local file paths with automatic fsspec access of s3 files without setup