Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add environment file to GitHub workflows #2258

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from
Draft

Add environment file to GitHub workflows #2258

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4b21f4c
Add environment file to build docs workflow
antonwolfy Jan 13, 2025
3dd57a6
Add environment file to coverage workflow
antonwolfy Jan 14, 2025
469f633
Specify message-id per mshick/add-pr-comment step
antonwolfy Jan 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 19 additions & 35 deletions .github/workflows/build-sphinx.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,11 +36,12 @@ jobs:
pull-requests: write

env:
python-ver: '3.12'
CHANNELS: '-c dppy/label/dev -c intel -c conda-forge --override-channels'
NO_INTEL_CHANNELS: '-c dppy/label/dev -c conda-forge --override-channels'
# Install the latest oneAPI compiler to work around an issue
INSTALL_ONE_API: 'yes'
environment-file: 'environments/environment.yml'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is environment.yml missing?

Copy link
Contributor Author

@antonwolfy antonwolfy Jan 13, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's generating by the command:

conda-merge ${{ env.base-doc-env }} ${{ env.build-pkgs-env }} > ${{ env.environment-file }}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool. Do we need to hardcode python=3.12 dependency in base_duild_docs.yml ?

Taking into consideration that we can dynamically create environment files using conda-merge?

Or am I missing something?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The file will be split later, because part of packages are common with coverage workflow.
The PR is work in progress still :)

In general, the python is pinned as 3.12, because cupy package doesn't support 3.13, i.e. max python version depends on cupy package and so I linked them there in one file.

Alternatively we can miss python in base_duild_docs.yml and to pass as option of conda-incubator/setup-miniconda step, but I preferred to do that through base_duild_docs.yml file.

build-with-oneapi-env: 'environments/build_with_oneapi.yml'
building-docs-env: 'environments/building_docs.yml'
oneapi-pkgs-env: ''
# Enable env when it's required to use only conda packages without OneAPI installation
# oneapi-pkgs-env: 'environments/oneapi_pkgs.yml'

steps:
- name: Cancel Previous Runs
Expand Down Expand Up @@ -101,16 +102,25 @@ jobs:
with:
fetch-depth: 0

- name: Install conda-merge tool
uses: BSFishy/pip-action@8f2d471d809dc20b6ada98c91910b6ae6243f318 # v1
with:
packages: conda-merge

- name: Merge conda env files
run: |
conda-merge ${{ env.build-with-oneapi-env }} ${{ env.building-docs-env }} ${{ env.oneapi-pkgs-env }} > ${{ env.environment-file }}
cat ${{ env.environment-file }}

- name: Setup miniconda
id: setup_miniconda
continue-on-error: true
uses: conda-incubator/setup-miniconda@d2e6a045a86077fb6cad6f5adf368e9076ddaa8d # v3.1.0
with:
miniforge-version: latest
use-mamba: 'true'
channels: conda-forge
conda-remove-defaults: 'true'
python-version: ${{ env.python-ver }}
environment-file: '${{ env.environment-file }}'
activate-environment: 'docs'

- name: ReSetup miniconda
Expand All @@ -119,35 +129,10 @@ jobs:
with:
miniforge-version: latest
use-mamba: 'true'
channels: conda-forge
conda-remove-defaults: 'true'
python-version: ${{ env.python-ver }}
environment-file: '${{ env.environment-file }}'
activate-environment: 'docs'

# Sometimes `mamba install ...` fails due to slow download speed rate, so disable the check in mamba
- name: Disable speed limit check in mamba
run: echo "MAMBA_NO_LOW_SPEED_LIMIT=1" >> $GITHUB_ENV

- name: Install sphinx dependencies
run: |
mamba install sphinx sphinx_rtd_theme
pip install sphinxcontrib-googleanalytics==0.4 \
pyenchant sphinxcontrib-spelling

- name: Install dpnp dependencies
if: env.INSTALL_ONE_API == 'yes'
run: |
mamba install numpy dpctl">=0.18.0dev0" cmake cython pytest ninja scikit-build ${{ env.NO_INTEL_CHANNELS }}

- name: Install dpnp dependencies
if: env.INSTALL_ONE_API != 'yes'
run: |
mamba install numpy dpctl">=0.18.0dev0" mkl-devel-dpcpp onedpl-devel tbb-devel dpcpp_linux-64 \
cmake cython pytest ninja scikit-build ${{ env.CHANNELS }}

- name: Install cuPy dependencies
run: mamba install cupy

- name: Conda info
run: mamba info

Expand Down Expand Up @@ -178,7 +163,6 @@ jobs:
echo PROJECT_NUMBER=${PROJECT_NUMBER}
echo "PROJECT_NUMBER=$PROJECT_NUMBER" >> $GITHUB_ENV

# https://github.com/marketplace/actions/doxygen-action
- name: Build backend docs
uses: mattnotmitt/doxygen-action@cbe72c8e402e8a3faa1f0b247ef90aa6c8e4ce74 # v1.9.8
with:
Expand All @@ -187,7 +171,6 @@ jobs:
- name: Copy backend docs
run: cp -r dpnp/backend/doc/html ${{ env.PUBLISH_DIR }}/backend_doc

# https://github.com/marketplace/actions/github-pages-action
# The step is only used to build docs while pushing a PR to "master"
- name: Deploy docs
if: env.GH_EVENT_PUSH_UPSTREAM == 'true'
Expand Down Expand Up @@ -223,6 +206,7 @@ jobs:
PR_NUM: ${{ github.event.number }}
uses: mshick/add-pr-comment@b8f338c590a895d50bcbfa6c5859251edc8952fc # v2.8.2
with:
message-id: url_to_docs
message: |
View rendered docs @ https://intelpython.github.io/dpnp/pull/${{ env.PR_NUM }}/index.html
allow-repeats: false
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/conda-package.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -612,9 +612,9 @@ jobs:
if: ${{ github.event.pull_request && !github.event.pull_request.head.repo.fork }}
uses: mshick/add-pr-comment@b8f338c590a895d50bcbfa6c5859251edc8952fc # v2.8.2
with:
message-id: array_api_results
message: |
${{ env.MESSAGE }}
refresh-message-position: true
cleanup_packages:
name: Clean up anaconda packages
Expand Down
67 changes: 29 additions & 38 deletions .github/workflows/generate_coverage.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,25 +20,20 @@ jobs:
shell: bash -el {0}

env:
python-ver: '3.12'
CHANNELS: '-c dppy/label/dev -c intel -c conda-forge --override-channels'
NO_INTEL_CHANNELS: '-c dppy/label/dev -c conda-forge --override-channels'
# Install the latest oneAPI compiler to work around an issue
INSTALL_ONE_API: 'yes'
environment-file: 'environments/environment.yml'
build-with-oneapi-env: 'environments/build_with_oneapi.yml'
coverage-env: 'environments/coverage.yml'
oneapi-pkgs-env: ''
# Enable env when it's required to use only conda packages without OneAPI installation
# oneapi-pkgs-env: 'environments/oneapi_pkgs.yml'

steps:
- name: Cancel Previous Runs
uses: styfle/cancel-workflow-action@85880fa0301c86cca9da44039ee3bb12d3bedbfa # 0.12.1
with:
access_token: ${{ github.token }}

- name: Checkout repo
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
fetch-depth: 0

- name: Add Intel repository
if: env.INSTALL_ONE_API == 'yes'
run: |
wget https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB
cat GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB | gpg --dearmor | sudo tee /usr/share/keyrings/oneapi-archive-keyring.gpg > /dev/null
Expand All @@ -47,7 +42,6 @@ jobs:
sudo apt update

- name: Install latest Intel OneAPI
if: env.INSTALL_ONE_API == 'yes'
run: |
sudo apt install hwloc \
intel-oneapi-mkl \
Expand All @@ -60,6 +54,22 @@ jobs:
- name: Install Lcov
run: |
sudo apt-get install lcov
sudo gem install coveralls-lcov

- name: Checkout repo
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
fetch-depth: 0

- name: Install conda-merge tool
uses: BSFishy/pip-action@8f2d471d809dc20b6ada98c91910b6ae6243f318 # v1
with:
packages: conda-merge

- name: Merge conda env files
run: |
conda-merge ${{ env.build-with-oneapi-env }} ${{ env.coverage-env }} ${{ env.oneapi-pkgs-env }} > ${{ env.environment-file }}
cat ${{ env.environment-file }}

- name: Setup miniconda
id: setup_miniconda
Expand All @@ -68,9 +78,8 @@ jobs:
with:
miniforge-version: latest
use-mamba: 'true'
channels: conda-forge
conda-remove-defaults: 'true'
python-version: ${{ env.python-ver }}
environment-file: '${{ env.environment-file }}'
activate-environment: 'coverage'

- name: ReSetup miniconda
Expand All @@ -79,27 +88,10 @@ jobs:
with:
miniforge-version: latest
use-mamba: 'true'
channels: conda-forge
conda-remove-defaults: 'true'
python-version: ${{ env.python-ver }}
environment-file: '${{ env.environment-file }}'
activate-environment: 'coverage'

# Sometimes `mamba install ...` fails due to slow download speed rate, so disable the check in mamba
- name: Disable speed limit check in mamba
run: echo "MAMBA_NO_LOW_SPEED_LIMIT=1" >> $GITHUB_ENV

- name: Install dpnp dependencies
if: env.INSTALL_ONE_API == 'yes'
run: |
mamba install cython llvm cmake">=3.21" scikit-build ninja pytest pytest-cov coverage[toml] \
dpctl">=0.18.0dev0" ${{ env.NO_INTEL_CHANNELS }}

- name: Install dpnp dependencies
if: env.INSTALL_ONE_API != 'yes'
run: |
mamba install cython llvm cmake">=3.21" scikit-build ninja pytest pytest-cov coverage[toml] \
dpctl">=0.18.0dev0" dpcpp_linux-64 mkl-devel-dpcpp tbb-devel onedpl-devel ${{ env.CHANNELS }}

- name: Conda info
run: |
mamba info
Expand All @@ -122,12 +114,7 @@ jobs:

- name: Total number of coverage attempts
run: |
echo "Total number of coverage attempts made: ${{ steps.build_coverage.outputs.total_attempts }}"

- name: Install coverall dependencies
run: |
sudo gem install coveralls-lcov
pip install coveralls==4.0.1
echo "Total number of coverage attempts: ${{ steps.build_coverage.outputs.total_attempts }}"

- name: Upload coverage data to coveralls.io
run: |
Expand All @@ -144,9 +131,13 @@ jobs:

coveralls:
name: Indicate completion to coveralls.io

needs: generate-coverage

runs-on: ubuntu-latest

container: python:3-slim

steps:
- name: Finished
run: |
Expand Down
3 changes: 3 additions & 0 deletions environments/base_build_docs.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
pyenchant==3.2.2
sphinxcontrib-googleanalytics==0.4
sphinxcontrib-spelling==8.0.1
12 changes: 12 additions & 0 deletions environments/build_with_oneapi.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
name: Packages to build DPNP with OneAPI env activated
channels:
- dppy/label/dev
- conda-forge
dependencies:
- cmake
- cython
- dpctl>=0.19.0dev0
- ninja
- numpy
- pytest
- scikit-build
10 changes: 10 additions & 0 deletions environments/building_docs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
name: Building docs specific packages
channels:
- conda-forge
dependencies:
- python=3.12
- cupy
- sphinx
- sphinx_rtd_theme
- pip:
- -r base_build_docs.txt
1 change: 1 addition & 0 deletions environments/coverage.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
coveralls==4.0.1
10 changes: 10 additions & 0 deletions environments/coverage.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
name: Coverage specific packages
channels:
- conda-forge
dependencies:
- python=3.12
- coverage[toml]
- llvm
- pytest-cov
- pip:
- -r coverage.txt
8 changes: 8 additions & 0 deletions environments/oneapi_pkgs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
name: OneAPI packages to build DPNP without OneAPI env activated
channels:
- https://software.repos.intel.com/python/conda/
dependencies:
- dpcpp_linux-64
- mkl-devel-dpcpp
- onedpl-devel
- tbb-devel
Loading