Skip to content

Commit

Permalink
Merge branch 'main' into original-format
Browse files Browse the repository at this point in the history
  • Loading branch information
mih authored Jul 19, 2024
2 parents addbe3c + 8e54971 commit 66c62df
Show file tree
Hide file tree
Showing 5 changed files with 28 additions and 20 deletions.
19 changes: 6 additions & 13 deletions .appveyor.yml
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ environment:
APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2019
# Python version specification is non-standard on windows
PY: 39-x64
INSTALL_GITANNEX: git-annex -m datalad/git-annex:release
INSTALL_GITANNEX: git-annex -m datalad/packages
# MacOS core tests
- ID: MacP38core
DTS: datalad_dataverse
Expand All @@ -89,20 +89,13 @@ matrix:
- KNOWN2FAIL: 1


# do not run the CI if only documentation changes were made
# only run the CI for code changes.
# documentation builds are tested elsewhere and cheaper
skip_commits:
only_commits:
files:
- docs/
- changelog.d/
- .github/
- CHANGELOG.md
- CITATION.cff
- CONTRIBUTORS
- LICENSE
- Makefile
- README.md
- readthedocs.yml
- .appveyor.yml
- datalad_dataverse/
- tools/


# it is OK to specify paths that may not exist for a particular test run
Expand Down
2 changes: 1 addition & 1 deletion CODE_OF_CONDUCT.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
We as members, contributors, and leaders pledge to make participation in our
community a harassment-free experience for everyone, regardless of age, body
size, visible or invisible disability, ethnicity, sex characteristics, gender
identity and expression, level of experience, education, socio-economic status,
identity and expression, level of experience, education, socioeconomic status,
nationality, personal appearance, race, caste, color, religion, or sexual
identity and orientation.

Expand Down
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,9 @@ Doing so will enable the extension to also alter the behavior the core DataLad
package and its commands, from example to be able to directly clone from
a Dataverse dataset landing page.

Full-compatibility with Windows requires a git-annex installation of version
10.20230321 (or later).


## Summary of functionality provided by this extension

Expand Down
11 changes: 8 additions & 3 deletions datalad_dataverse/dataset.py
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ def __init__(self, api, dsid: str, root_path: str | None = None):
# check if project with specified doi exists
# TODO ask for ':latest' and cache?
dv_ds = api.get_dataset(identifier=dsid)
if not dv_ds.ok:
if not dv_ds.status_code < 400:
raise RuntimeError("Cannot find dataset")

def get_fileid_from_path(
Expand Down Expand Up @@ -171,13 +171,18 @@ def download_file(self, fid: int, path: Path):
# https://github.com/gdcc/pyDataverse/issues/49
# the code below is nevertheless readied for such a
# scenario
response = self.data_access_api.get_datafile(fid, data_format="original")
response = self.data_access_api.get_datafile(fid, is_pid=False, data_format="original")
# http error handling
response.raise_for_status()
with path.open("wb") as f:
# accommodate old and newer pydataverse version
try:
it = response.iter_content
except AttributeError:
it = response.iter_bytes
# `chunk_size=None` means
# "read data in whatever size the chunks are received"
for chunk in response.iter_content(chunk_size=None):
for chunk in it(chunk_size=None):
f.write(chunk)

def remove_file(self, fid: int):
Expand Down
13 changes: 10 additions & 3 deletions datalad_dataverse/tests/test_pydataverse.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,14 +54,21 @@ def test_file_handling(
def check_download(api, fileid, dsid, fpath, src_md5):
# TODO there is no standalone implementation of the following
# reimplementing DataverseRemote._download_file
response = api.get_datafile(fileid)

# recent pydataverse requires saying `is_pid=False` for a file-id
response = api.get_datafile(fileid, is_pid=False)
# TODO this could also just be a download via HttpUrlOperations
# avoiding any custom code
assert response.status_code == 200
with fpath.open("wb") as f:
# use a stupdly small chunksize to actual get chunking on
# accommodate old and newer pydataverse version
try:
it = response.iter_content
except AttributeError:
it = response.iter_bytes
# use a stupdily small chunksize to actual get chunking on
# our tiny test file
for chunk in response.iter_content(chunk_size=1):
for chunk in it(chunk_size=1):
f.write(chunk)

# confirm identity
Expand Down

0 comments on commit 66c62df

Please sign in to comment.