Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor build process to enable auto-update of components #416

Merged
merged 16 commits into from
May 16, 2024
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 14 additions & 6 deletions .github/ISSUE_TEMPLATE/create_release_branch.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,9 @@ Make sure to follow the steps below and ensure all actions are completed and sig

<!-- Link to PR to initialize the release branch (see below) -->
- **PR**:
-
<!-- Link to PR to initialize auto-update job for the release branch (see below) -->
- **PR**:

#### Actions

Expand Down Expand Up @@ -94,12 +97,16 @@ The steps are to be followed in-order, each task must be completed by the person
- `rm -rf ~/tmp/release-1.xx`
- [ ] **Reviewer**: Ensure `release-1.xx` branch is based on latest changes on `main` at the time of the release cut.
- [ ] **Owner**: Create PR to initialize `release-1.xx` branch:
- [ ] Update `KUBE_TRACK` to `1.xx` in [/build-scripts/components/kubernetes/version.sh][]
- [ ] Update `master` to `release-1.xx` in [/build-scripts/components/k8s-dqlite/version.sh][]
- [ ] Update `KUBERNETES_RELEASE_MARKER` to `stable-1.xx` in [/build-scripts/hack/update-component-versions.py][]
- [ ] Update `master` to `release-1.xx` in [/build-scripts/components/k8s-dqlite/version][]
- [ ] Update `"main"` to `"release-1.xx"` in [/build-scripts/hack/generate-sbom.py][]
- [ ] `git commit -m 'Release 1.xx'`
- [ ] Create PR with the changes and request review from **Reviewer**. Make sure to update the issue `Information` section with a link to the PR.
- [ ] Create PR against `release-1.xx` with the changes and request review from **Reviewer**. Make sure to update the issue `Information` section with a link to the PR.
- [ ] **Reviewer**: Review and merge PR to initialize branch.
- [ ] **Owner**: Create PR to initialize `update-components.yaml` job for `release-1.xx` branch:
- [ ] Add `release-1.xx` in [.github/workflows/update-components.yaml][]
- [ ] Remove unsupported releases from the list (if applicable, consult with **Reviewer**)
- [ ] Create PR against `main` with the changes and request review from **Reviewer**. Make sure to update the issue information with a link to the PR.
- [ ] **Reviewer**: On merge, confirm [Auto-update strict branch] action runs to completion and that the `autoupdate/release-1.xx-strict` branch is created.
- [ ] **Owner**: Create launchpad builders for `release-1.xx`
- [ ] Go to [lp:k8s][] and do **Import now** to pick up all latest changes.
Expand Down Expand Up @@ -150,9 +157,10 @@ The steps are to be followed in-order, each task must be completed by the person
[.github/workflows/python.yaml]: ../workflows/python.yaml
[.github/workflows/sbom.yaml]: ../workflows/sbom.yaml
[.github/workflows/strict-integration.yaml]: ../workflows/strict-integration.yaml
[.github/workflows/strict.yaml]: ..workflows/strict.yaml
[/build-scripts/components/kubernetes/version.sh]: ../../build-scripts/components/kubernetes/version.sh
[/build-scripts/components/k8s-dqlite/version.sh]: ../../build-scripts/components/k8s-dqlite/version.sh
[.github/workflows/strict.yaml]: ../workflows/strict.yaml
[.github/workflows/update-components.yaml]: ../workflows/update-components.yaml
[/build-scripts/components/hack/update-component-versions.py]: ../../build-scripts/components/hack/update-component-versions.py
[/build-scripts/components/k8s-dqlite/version]: ../../build-scripts/components/k8s-dqlite/version
[/build-scripts/hack/generate-sbom.py]: ../..//build-scripts/hack/generate-sbom.py
[lp:k8s]: https://code.launchpad.net/~cdk8s/k8s/+git/k8s-snap
[lp:k8s/+snaps]: https://launchpad.net/k8s/+snaps
46 changes: 46 additions & 0 deletions .github/workflows/update-components.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
name: Check for component upgrades

on:
workflow_dispatch:
schedule:
- cron: "0 10 * * *"

jobs:
update:
runs-on: ubuntu-latest
strategy:
matrix:
branch:
# Keep main branch up to date
- main
# Supported stable release branches
- release-1.30

steps:
- name: Checking out repo
uses: actions/checkout@v4
with:
ref: ${{ matrix.branch }}

- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: '3.10'

- name: Install dependencies
run: |
pip3 install -r ./build-scripts/hack/requirements.txt

- name: Check for new component versions
run: |
./build-scripts/hack/update-component-versions.py

- name: Create pull request
uses: peter-evans/create-pull-request@v6
with:
commit-message: "[${{ matrix.branch }}] Update component versions"
title: "[${{ matrix.branch }}] Update component versions"
body: "[${{ matrix.branch }}] Update component versions"
branch: "autoupdate/sync/${{ matrix.branch }}"
delete-branch: true
base: ${{ matrix.branch }}
53 changes: 0 additions & 53 deletions .github/workflows/update-versions.yaml

This file was deleted.

4 changes: 2 additions & 2 deletions build-scripts/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ build-scripts/
components/
$component_name/
repository <-- git repository to clone
version.sh <-- prints the repository tag or commit to checkout
version <-- repository tag or commit to checkout
build.sh <-- runs as `build.sh $output $version`
first argument is the output directory where
binaries should be placed, second is the component version
Expand All @@ -26,7 +26,7 @@ build-scripts/

## Applying patches

Most K8s components are retrieved from an upstream source (specified in the `repository`), with a specific tag (specified in `version.sh`), have some patches applied to them (from the `patches/` directory) and are then built (using `build.sh`).
Most K8s components are retrieved from an upstream source (specified in the `repository`), with a specific tag (specified in `version`), have some patches applied to them (from the `patches/` directory) and are then built (using `build.sh`).

This section explains the directory format for the `patches` directory.

Expand Down
2 changes: 1 addition & 1 deletion build-scripts/build-component.sh
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ COMPONENT_NAME="${1}"
COMPONENT_DIRECTORY="${DIR}/components/${COMPONENT_NAME}"

GIT_REPOSITORY="$(cat "${COMPONENT_DIRECTORY}/repository")"
GIT_TAG="$("${COMPONENT_DIRECTORY}/version.sh")"
GIT_TAG="$(cat "${COMPONENT_DIRECTORY}/version")"

COMPONENT_BUILD_DIRECTORY="${BUILD_DIRECTORY}/${COMPONENT_NAME}"

Expand Down
1 change: 1 addition & 0 deletions build-scripts/components/cni/version
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
v1.4.0
4 changes: 0 additions & 4 deletions build-scripts/components/cni/version.sh

This file was deleted.

1 change: 1 addition & 0 deletions build-scripts/components/containerd/version
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
v1.6.31
3 changes: 0 additions & 3 deletions build-scripts/components/containerd/version.sh

This file was deleted.

1 change: 1 addition & 0 deletions build-scripts/components/helm/version
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
v3.14.4
3 changes: 0 additions & 3 deletions build-scripts/components/helm/version.sh

This file was deleted.

1 change: 1 addition & 0 deletions build-scripts/components/k8s-dqlite/version
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
master
3 changes: 0 additions & 3 deletions build-scripts/components/k8s-dqlite/version.sh

This file was deleted.

1 change: 1 addition & 0 deletions build-scripts/components/kubernetes/version
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
v1.30.0
15 changes: 0 additions & 15 deletions build-scripts/components/kubernetes/version.sh

This file was deleted.

1 change: 1 addition & 0 deletions build-scripts/components/runc/version
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
v1.1.12
5 changes: 0 additions & 5 deletions build-scripts/components/runc/version.sh

This file was deleted.

18 changes: 13 additions & 5 deletions build-scripts/generate-bom.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,12 @@
SNAPCRAFT_PART_BUILD = Path(os.getenv("SNAPCRAFT_PART_BUILD", ""))
SNAPCRAFT_PART_INSTALL = Path(os.getenv("SNAPCRAFT_PART_INSTALL", ""))

BUILD_DIRECTORY = SNAPCRAFT_PART_BUILD.exists() and SNAPCRAFT_PART_BUILD or DIR / ".build"
INSTALL_DIRECTORY = SNAPCRAFT_PART_INSTALL.exists() and SNAPCRAFT_PART_INSTALL or DIR / ".install"
BUILD_DIRECTORY = (
SNAPCRAFT_PART_BUILD.exists() and SNAPCRAFT_PART_BUILD or DIR / ".build"
)
INSTALL_DIRECTORY = (
SNAPCRAFT_PART_INSTALL.exists() and SNAPCRAFT_PART_INSTALL or DIR / ".install"
)

# List of tools used to build or bundled in the snap
TOOLS = {
Expand Down Expand Up @@ -51,11 +55,15 @@ def _read_file(path: Path) -> str:
component_dir = DIR / "components" / component

try:
version = _parse_output([component_dir / "version.sh"])
patches = _parse_output([sys.executable, DIR / "print-patches-for.py", component, version])
version = _read_file(component_dir / "version")
patches = _parse_output(
[sys.executable, DIR / "print-patches-for.py", component, version]
)
clean_patches = []
if patches:
clean_patches = [p[p.find("build-scripts/") :] for p in patches.split("\n")]
clean_patches = [
p[p.find("build-scripts/") :] for p in patches.split("\n")
]

BOM["components"][component] = {
"repository": _read_file(component_dir / "repository"),
Expand Down
Loading
Loading