Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update files based on task_denoising test #5

Merged
merged 104 commits into from
Jul 11, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
104 commits
Select commit Hold shift + click to select a range
e649c29
Fix resources
KaiWaldrant Jun 21, 2024
d6e499e
add submodule initialisation
KaiWaldrant Jun 21, 2024
3c5b4d6
add `task_` prefix
KaiWaldrant Jun 21, 2024
38d4f07
update gitignore
KaiWaldrant Jun 21, 2024
3cb1f64
WIP instructions
KaiWaldrant Jun 27, 2024
41679cf
add solution api file
KaiWaldrant Jun 28, 2024
1fbb961
add process dataset api file
KaiWaldrant Jun 28, 2024
d7d11af
fix FP in metric api
KaiWaldrant Jun 28, 2024
814cb5e
Update instructions
KaiWaldrant Jun 28, 2024
3129d78
add readme
KaiWaldrant Jun 28, 2024
f1494dc
remove github_url from readme
KaiWaldrant Jun 28, 2024
0ea712e
update process dataset to viash 0.9
KaiWaldrant Jun 28, 2024
cae2c12
update task_info readme section
KaiWaldrant Jun 28, 2024
c5cacb7
rename arg method api files
KaiWaldrant Jun 28, 2024
84a041d
add chmod cmd to instructions
KaiWaldrant Jun 28, 2024
6646877
`chmod +x scripts/*`
KaiWaldrant Jun 28, 2024
9bfdd56
remove chmod cmd
KaiWaldrant Jun 28, 2024
33e6cac
add final line in _viash
KaiWaldrant Jun 28, 2024
2e9acc2
remove chmod instructions
KaiWaldrant Jun 28, 2024
d0f1a9f
remove comments
KaiWaldrant Jun 28, 2024
617889e
update CI
KaiWaldrant Jul 1, 2024
4ec22d8
add s3 to _viash
KaiWaldrant Jul 1, 2024
5a08ceb
Add github templatest
KaiWaldrant Jul 1, 2024
97f74b3
update s3 link
KaiWaldrant Jul 1, 2024
361e7e5
update common submodule
KaiWaldrant Jul 1, 2024
b74c327
fix resource test path
KaiWaldrant Jul 3, 2024
a4abf7f
Fix resource test path
KaiWaldrant Jul 3, 2024
692c02e
Update task name
KaiWaldrant Jul 3, 2024
c13fea4
remove `readme` field from task_info
KaiWaldrant Jul 4, 2024
f3fb01f
update submodule
KaiWaldrant Jul 4, 2024
0e31cc0
update submodule
KaiWaldrant Jul 4, 2024
6900fb4
update submodule documentation
KaiWaldrant Jul 4, 2024
5708aa3
update submodule
KaiWaldrant Jul 4, 2024
98367cd
update readme
KaiWaldrant Jul 4, 2024
01ee29e
add changelog
KaiWaldrant Jul 5, 2024
4937078
update _viash
KaiWaldrant Jul 5, 2024
4e3faaa
Update s3 buckets in _viash
KaiWaldrant Jul 5, 2024
2d8f3c9
rename changelog
KaiWaldrant Jul 8, 2024
8d7adaa
Add changelog template
KaiWaldrant Jul 8, 2024
9293510
relocate instructions to common
KaiWaldrant Jul 8, 2024
b88f6ed
change submodule link to https
KaiWaldrant Jul 8, 2024
7eeb2f7
add reference to submodule instructions
KaiWaldrant Jul 8, 2024
793c05e
Relocate task_info to _viash [WIP]
KaiWaldrant Jul 8, 2024
21afc2c
Update _viash [WIP]
KaiWaldrant Jul 10, 2024
a675aa8
add last step comments to _viash
KaiWaldrant Jul 10, 2024
cb5730a
Update common submodule
KaiWaldrant Jul 10, 2024
aafaa76
Update download resources
KaiWaldrant Jul 10, 2024
affce64
Create working components [WIP]
KaiWaldrant Jul 10, 2024
98ab68a
Create working components [WIP]:
KaiWaldrant Jul 10, 2024
f763df0
Create working components [WIP]:
KaiWaldrant Jul 10, 2024
7a1d0e5
Create working components [WIP]:
KaiWaldrant Jul 10, 2024
782c2eb
Create working components [WIP]: update resources_test path
KaiWaldrant Jul 10, 2024
2aca120
Update config control_method to working
KaiWaldrant Jul 10, 2024
045c4b1
Create working components [WIP]:
KaiWaldrant Jul 10, 2024
703be1d
Create working components [WIP]:
KaiWaldrant Jul 10, 2024
ac59cbc
Create working components [WIP]:
KaiWaldrant Jul 10, 2024
b2c4bd0
Update common submodule
KaiWaldrant Jul 10, 2024
3be418b
Update comp_method api file
KaiWaldrant Jul 10, 2024
8fbc4d1
Add working method
KaiWaldrant Jul 10, 2024
862d48d
Update control_method api
KaiWaldrant Jul 10, 2024
2c81f68
add working control method
KaiWaldrant Jul 10, 2024
e71c07c
update solution file api
KaiWaldrant Jul 10, 2024
ac944d4
update slots in solution file api
KaiWaldrant Jul 10, 2024
8f97e66
update matric api
KaiWaldrant Jul 10, 2024
df8888d
update score file api
KaiWaldrant Jul 10, 2024
3022be8
fix metric test_resources path
KaiWaldrant Jul 10, 2024
fdb4f26
add working metric
KaiWaldrant Jul 10, 2024
b4e8325
force ci
KaiWaldrant Jul 11, 2024
5bc6929
try fix for resources test s3 path in _viash
KaiWaldrant Jul 11, 2024
28e4c8a
update instructions url
KaiWaldrant Jul 11, 2024
bb9b3bc
update instructions url
KaiWaldrant Jul 11, 2024
f578c88
Update README.md
KaiWaldrant Jul 11, 2024
6a8f371
add gitignore for scripts
KaiWaldrant Jul 11, 2024
2e1f542
Merge branch 'feature/no-ref/update-docs' of github.com:openproblems-…
KaiWaldrant Jul 11, 2024
06d1da3
add config for benchmark workflow
KaiWaldrant Jul 11, 2024
6510489
add run benchmark workflow [WIP]
KaiWaldrant Jul 11, 2024
9645f2b
Fix paths in benchmark config
KaiWaldrant Jul 11, 2024
8167c63
fix findstatestemp typo
KaiWaldrant Jul 11, 2024
c1d2033
add workflow test [WIP]
KaiWaldrant Jul 11, 2024
676d2a8
update submodule
KaiWaldrant Jul 11, 2024
5ef354f
Add obs_names to method output
KaiWaldrant Jul 11, 2024
65bc1c3
add obs_names to control_method output
KaiWaldrant Jul 11, 2024
c1f9ba1
update submodule
KaiWaldrant Jul 11, 2024
cdd16e0
fix task_info extraction from _viash in workflow
KaiWaldrant Jul 11, 2024
3cd3efe
update _viash info
KaiWaldrant Jul 11, 2024
820facc
update process_dataset api
KaiWaldrant Jul 11, 2024
dac82f7
rename process_dataset api file
KaiWaldrant Jul 11, 2024
2d74c71
add process_dataset config file [WIP]
KaiWaldrant Jul 11, 2024
21a5f4f
relocate dataset_processors to data_processors
KaiWaldrant Jul 11, 2024
e081d17
add dataset_processor script
KaiWaldrant Jul 11, 2024
ade957e
add process_datasets workflow config
KaiWaldrant Jul 11, 2024
b0b4273
Update process datasets workflow config
KaiWaldrant Jul 11, 2024
787920d
Add process datasets workflow script
KaiWaldrant Jul 11, 2024
d062ddc
add process datasets test
KaiWaldrant Jul 11, 2024
3b15a12
Update submodule
KaiWaldrant Jul 11, 2024
7da438f
update common dataset file
KaiWaldrant Jul 11, 2024
aacfdb0
add temp fix to workflow
KaiWaldrant Jul 11, 2024
ee96067
Add additional entries to changelog
KaiWaldrant Jul 11, 2024
c5129bf
Remove task_info
KaiWaldrant Jul 11, 2024
18a5829
remove comment from download_resources script
KaiWaldrant Jul 11, 2024
2909f34
Add comment to download resources
KaiWaldrant Jul 11, 2024
890aa64
Add create_test_resources script
KaiWaldrant Jul 11, 2024
ecbd4be
add run_benchmark scripts
KaiWaldrant Jul 11, 2024
643f6ae
Apply suggestions from code review
KaiWaldrant Jul 11, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
---
name: Bug report
about: Create a report to help us improve
title: ''
labels: [bug]
assignees: ''

---

**Describe the bug**
A clear and concise description of what the bug is.

**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

**Expected behavior**
A clear and concise description of what you expected to happen.

**Additional context**
Add any other context about the problem here.
1 change: 1 addition & 0 deletions .github/ISSUE_TEMPLATE/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
blank_issues_enabled: true
20 changes: 20 additions & 0 deletions .github/ISSUE_TEMPLATE/feature_request.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
---
name: Feature request
about: Suggest an idea for this project
title: ''
labels: [enhancement]
assignees: ''

---

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.
17 changes: 17 additions & 0 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
## Describe your changes

<!-- Describe your changes -->

## Checklist before requesting a review
- [ ] I have performed a self-review of my code

- Check the correct box. Does this PR contain:
- [ ] Breaking changes
- [ ] New functionality
- [ ] Major changes
- [ ] Minor changes
- [ ] Bug fixes

- [ ] Proposed changes are described in the CHANGELOG.md

- [ ] CI Tests succeed and look good!
121 changes: 10 additions & 111 deletions .github/workflows/build.yaml
Original file line number Diff line number Diff line change
@@ -1,122 +1,21 @@
name: build
name: Build

on:
push:
branches: [ 'main' ]
workflow_dispatch:
inputs:
target_branch:
description: 'Branch to deploy to. If not specified, `build-${BRANCH_NAME}` will be used.'
required: false
version:
description: 'Version name to use for the build. If not specified, `build-${BRANCH_NAME}` will be used.'
description: |
The version of the project to build. Example: `1.0.3`.

If not provided, a development build with a version name
based on the branch name will be built. Otherwise, a release
build with the provided version will be built.
required: false

jobs:
# phase 1
list:
runs-on: ubuntu-latest

outputs:
target_branch: ${{ steps.defaults.outputs.target_branch }}
version: ${{ steps.defaults.outputs.version }}
component_matrix: ${{ steps.set_matrix.outputs.matrix }}

steps:
- uses: actions/checkout@v4
with:
submodules: 'recursive'

- uses: viash-io/viash-actions/setup@v5

- name: Determine version tag from branch name
id: defaults
run: |
BRANCH_NAME=$(echo $GITHUB_REF | sed 's/refs\/heads\///')

VERSION=${{ github.event.inputs.version }}
if [ -z "$VERSION" ]; then
VERSION="build-$BRANCH_NAME"
fi
echo "version=$VERSION" >> $GITHUB_OUTPUT

TARGET_BRANCH=${{ github.event.inputs.target_branch }}
if [ -z "$TARGET_BRANCH" ]; then
TARGET_BRANCH="build-$BRANCH_NAME"
fi
echo "target_branch=$TARGET_BRANCH" >> $GITHUB_OUTPUT

- name: Remove target folder from .gitignore
run: |
# allow publishing the target folder
sed -i '/^\/target.*/d' .gitignore

- uses: viash-io/viash-actions/ns-build@v5
with:
config_mod: .version := '${{ steps.defaults.outputs.version }}'
parallel: true

- name: Deploy to target branch
uses: peaceiris/actions-gh-pages@v4
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: .
publish_branch: ${{ steps.defaults.outputs.target_branch }}

- id: ns_list
uses: viash-io/viash-actions/ns-list@v5
with:
platform: docker
src: src
format: json

- id: set_matrix
run: |
echo "matrix=$(jq -c '[ .[] |
{
"name": (.namespace + "/" + .name),
"dir": .info.config | capture("^(?<dir>.*\/)").dir
}
]' ${{ steps.ns_list.outputs.output_file }} )" >> $GITHUB_OUTPUT

# phase 2
build:
needs: list

runs-on: ubuntu-latest

strategy:
fail-fast: false
matrix:
component: ${{ fromJson(needs.list.outputs.component_matrix) }}

steps:
# Remove unnecessary files to free up space. Otherwise, we get 'no space left on device.'
- uses: data-intuitive/reclaim-the-bytes@v2

- uses: actions/checkout@v4

- uses: viash-io/viash-actions/setup@v5

- name: Build container
uses: viash-io/viash-actions/ns-build@v5
with:
config_mod: .version := '${{ needs.list.outputs.version }}'
platform: docker
src: ${{ matrix.component.dir }}
setup: build

- name: Login to container registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ secrets.GTHB_USER }}
password: ${{ secrets.GTHB_PAT }}

- name: Push container
uses: viash-io/viash-actions/ns-build@v5
with:
config_mod: .version := '${{ needs.list.outputs.version }}'
platform: docker
src: ${{ matrix.component.dir }}
setup: push
uses: openproblems-bio/actions/.github/workflows/build.yml@main
with:
version: ${{ github.event.inputs.version }}
112 changes: 4 additions & 108 deletions .github/workflows/test.yaml
Original file line number Diff line number Diff line change
@@ -1,113 +1,9 @@
name: test
name: Test

on:
pull_request:
push:
branches: [ '**' ]
pull_request:

jobs:
run_ci_check_job:
runs-on: ubuntu-latest
outputs:
run_ci: ${{ steps.github_cli.outputs.check }}
steps:
- name: 'Check if branch has an existing pull request and the trigger was a push'
id: github_cli
run: |
pull_request=$(gh pr list -R ${{ github.repository }} -H ${{ github.ref_name }} --json url --state open --limit 1 | jq '.[0].url')
# If the branch has a PR and this run was triggered by a push event, do not run
if [[ "$pull_request" != "null" && "$GITHUB_REF_NAME" != "main" && "${{ github.event_name == 'push' }}" == "true" && "${{ !contains(github.event.head_commit.message, 'ci force') }}" == "true" ]]; then
echo "check=false" >> $GITHUB_OUTPUT
else
echo "check=true" >> $GITHUB_OUTPUT
fi
env:
GITHUB_TOKEN: ${{ secrets.GTHB_PAT }}

# phase 1
list:
needs: run_ci_check_job
env:
s3_bucket: s3://openproblems-data/resources_test
runs-on: ubuntu-latest
if: ${{ needs.run_ci_check_job.outputs.run_ci == 'true' }}

outputs:
matrix: ${{ steps.set_matrix.outputs.matrix }}
cache_key: ${{ steps.cache.outputs.cache_key }}

steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
submodules: 'recursive'

- uses: christian-ci/action-yaml-github-output@v2
with:
file_path: _viash.yaml

- uses: viash-io/viash-actions/setup@v5

- uses: viash-io/viash-actions/project/sync-and-cache-s3@v5
id: cache
with:
s3_bucket: $s3_bucket/$NAME
dest_path: resources
cache_key_prefix: resources__

- id: ns_list
uses: viash-io/viash-actions/ns-list@v5
with:
engine: docker
format: json

- id: ns_list_filtered
uses: viash-io/viash-actions/project/detect-changed-components@v5
with:
input_file: "${{ steps.ns_list.outputs.output_file }}"

- id: set_matrix
run: |
echo "matrix=$(jq -c '[ .[] |
{
"name": (.namespace + "/" + .name),
"config": .info.config
}
]' ${{ steps.ns_list_filtered.outputs.output_file }} )" >> $GITHUB_OUTPUT

# phase 2
viash_test:
needs: list
if: ${{ needs.list.outputs.matrix != '[]' && needs.list.outputs.matrix != '' }}
runs-on: ubuntu-latest

strategy:
fail-fast: false
matrix:
component: ${{ fromJson(needs.list.outputs.matrix) }}

steps:
# Remove unnecessary files to free up space. Otherwise, we get 'no space left on device.'
- uses: data-intuitive/reclaim-the-bytes@v2

- uses: actions/checkout@v4
with:
submodules: 'recursive'

- uses: viash-io/viash-actions/setup@v5

# use cache
- name: Cache resources data
uses: actions/cache@v4
timeout-minutes: 10
with:
path: resources
key: ${{ needs.list.outputs.cache_key }}

- name: Run test
timeout-minutes: 30
run: |
VIASH_TEMP=$RUNNER_TEMP/viash viash test \
"${{ matrix.component.config }}" \
--cpus 2 \
--memory "16gb"
build:
uses: openproblems-bio/actions/.github/workflows/test.yml@main
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
resources
resources_test
work
.nextflow*
target
Expand Down
2 changes: 1 addition & 1 deletion .gitmodules
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
[submodule "common"]
path = common
url = git@github.com:openproblems-bio/common-resources.git
url = https://github.com/openproblems-bio/common_resources.git
24 changes: 24 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# task_template x.y.z

## BREAKING CHANGES

<!-- * Restructured `src` directory (PR #3). -->

## NEW FUNCTIONALITY

* Added `control_methods/true_labels` component (PR #5).

* Added `methods/logistic_regression` component (PR #5).

* Added `metrics/accuracy` component (PR #5).

## MAJOR CHANGES

* Updated `api` files (PR #5).

## MINOR CHANGES

* Updated `README.md` (PR #5).

## BUGFIXES

Loading