Skip to content

Commit

Permalink
Feat/add syracuse qdr retriever (#1)
Browse files Browse the repository at this point in the history
* (HP-1459): update README and gitignore

* (HP-1459): add pyproject.toml, poetry.lock

* (HP-1459): add qdr_downloads module

* (HP-1459): add unit tests

* (HP-1459): add ci.yaml and ci_commands_script.sh

* (HP-1459): add image_build_push and wool

* (HP-1459): add secrets.baseline and pre-commit-config

* (HP-1459): expand documentation for qdr_download

* (HP-1459): use streaming in download

* (HP-1459): clean up tests

* (HP-1459): clean up request_headers test

* (HP-1459): clean up tests

* (HP-1459): clean up comments

* (HP-1459): add content to README

* (HP-1459): remove github action for 'image_build_push'

* (HP-1459): handle None download_url

* (HP-1459): allow downloading of single files, unzipped

* (HP-1459): get file_ids one at a time

* (HP-1459): remove unused variable

* (HP-1459): remove debug statements

* (HP-1459): add instructions for pip install from repo

* (HP-1459): move 'pip install' section further down in README

---------

Co-authored-by: Mingfei Shao <[email protected]>
  • Loading branch information
george42-ctds and mfshao authored Jun 18, 2024
1 parent e01f10d commit 6ea6d75
Show file tree
Hide file tree
Showing 9 changed files with 2,524 additions and 34 deletions.
38 changes: 5 additions & 33 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
*.DS_Store
*.vscode
.idea
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
Expand Down Expand Up @@ -27,8 +30,6 @@ share/python-wheels/
MANIFEST

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

Expand Down Expand Up @@ -83,31 +84,7 @@ profile_default/
ipython_config.py

# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version

# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock

# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock

# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/#use-with-ide
.pdm.toml
.python-version

# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/
Expand Down Expand Up @@ -152,9 +129,4 @@ dmypy.json
# Cython debug symbols
cython_debug/

# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
# pytest output
41 changes: 40 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1 +1,40 @@
# heal-platform-sdk
# heal-platform-sdk

The HEAL Platform Software Development Kit (SDK) for Python provides classes and functions for handling HEAL specific tasks.

### qdr_downloads

The qdr_downloads module include a retriever function for downloading files from Syracuse QDR.

This is intended to be called by gen3sdk [external download functions](https://github.com/uc-cdis/gen3sdk-python/blob/master/gen3/tools/download/external_file_download.py). It is also possible to write a wrapper script for the qdr download functions.

### Run tests

```
poetry run pytest -vv tests
```

### Pip install for using heal-platform-sdk

Until the `heal-platform-sdk` is available on PyPI, any local pip installs should directly
reference the git repo.

As an example, `pip install` can be called from the command line for getting
the master branch of the `heal-platform-sdk`,

```
pip install -e git+ssh://[email protected]/uc-cdis/heal-platform-sdk.git#egg=heal
```

or a particular branch, eg,

```
pip install -e git+ssh://[email protected]/uc-cdis/heal-platform-sdk.git@feat/add-syracuse-qdr-retriever#egg=heal
```

The specification can also be listed in requirements.txt file
(with, say, a tag specification of 0.1.0)

```
-e git+ssh://[email protected]/uc-cdis/[email protected]#egg=heal
```
4 changes: 4 additions & 0 deletions heal/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
from cdislogging import get_logger

LOG_FORMAT = "[%(asctime)s][%(levelname)7s] %(message)s"
logging = get_logger("__name__", format=LOG_FORMAT, log_level="info")
Loading

0 comments on commit 6ea6d75

Please sign in to comment.