Skip to content

Commit

Permalink
[YUNIKORN-1876] Build environment setup update (#324)
Browse files Browse the repository at this point in the history 
Update and cleaup the documentation to reflect the current usage and
best practices of building YuniKorn.

Closes: #324

Signed-off-by: Wilfred Spiegelenburg <[email protected]>
  • Loading branch information
@craigcondit @wilfred-s
craigcondit authored and wilfred-s committed Oct 17, 2023
1 parent 900da59 commit 70ad1f0
Show file tree
Hide file tree
Showing 4 changed files with 333 additions and 165 deletions.
Binary file added docs/assets/goland_debug.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
210 changes: 131 additions & 79 deletions docs/developer_guide/build.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,78 +22,89 @@ specific language governing permissions and limitations
under the License.
-->

YuniKorn always works with a container orchestrator system. Currently, a Kubernetes shim [yunikorn-k8shim](https://github.com/apache/yunikorn-k8shim)
is provided in our repositories, you can leverage it to develop YuniKorn scheduling features and integrate with Kubernetes.
This document describes resources how to setup dev environment and how to do the development.
YuniKorn always works with a container orchestrator system. Currently, a
Kubernetes shim ([yunikorn-k8shim](https://github.com/apache/yunikorn-k8shim))
is provided which provides a drop-in scheduler for the Kubernetes platform.
This document describes how to setup and use a local development environment.

## Development Environment setup
## Dev Environment setup

Read the [environment setup guide](developer_guide/env_setup.md) first to setup Docker and Kubernetes development environment.
Read the [Dev Environment Setup](developer_guide/env_setup.md) guide first to
setup Docker and Kubernetes development environment.

## Build YuniKorn for Kubernetes
## Build YuniKorn

Prerequisite:
- Golang: check the `.go_version` file in the root of the repositories for the version Yunikorn requires. The minimum version can change per release branch. Earlier Go versions might cause compilation issues.
Prerequisites:
- Golang: check the `.go_version` file in the root of the repositories for the
version Yunikorn requires. The minimum version can change per release branch.
Using earlier Go versions will cause compilation issues.

You can build the scheduler for Kubernetes from [yunikorn-k8shim](https://github.com/apache/yunikorn-k8shim) project.
The build procedure will build all components into a single executable that can be deployed and running on Kubernetes.
You can build the scheduler for Kubernetes from the [yunikorn-k8shim](https://github.com/apache/yunikorn-k8shim)
project. The build procedure will build all components into a single executable
that can be deployed and running on Kubernetes.

Start the integrated build process by pulling the `yunikorn-k8shim` repository:
```bash
mkdir $HOME/yunikorn/
cd $HOME/yunikorn/
git clone https://github.com/apache/yunikorn-k8shim.git
```
At this point you have an environment that will allow you to build an integrated image for the YuniKorn scheduler.

### A note on Go modules and git version
Go use git to fetch module information.
Certain modules cannot be retrieved if the git version installed on the machine used to build is old.
A message similar to the one below will be logged when trying to build for the first time.
```text
go: finding modernc.org/[email protected]
go: modernc.org/[email protected]: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in <location>: exit status 128:
error: RPC failed; result=22, HTTP code = 404
fatal: The remote end hung up unexpectedly
```
Update git to a recent version to fix this issue.
Git releases later than 1.22 are known to work.

### Build Docker image
At this point you have an environment that will allow you to build an
integrated image for the YuniKorn scheduler.

Building a docker image can be triggered by following command.
### Build Docker images

```
Building the Docker images can be triggered by following command:
```shell script
make image
```

The image with the build in configuration can be deployed directly on kubernetes.
Some sample deployments that can be used are found under [deployments](https://github.com/apache/yunikorn-k8shim/tree/master/deployments/scheduler) directory.
For the deployment that uses a config map you need to set up the ConfigMap in kubernetes.
How to deploy the scheduler with a ConfigMap is explained in the [scheduler configuration deployment](developer_guide/deployment.md) document.
This will generate images for the scheduler, scheduler plugin, and admission
controller.

The image build command will first build the integrated executable and then create the docker image.
If you want to use pre-built images based on a release, please check the [Docker Hub repo](https://hub.docker.com/r/apache/yunikorn).
The images created can be deployed directly on Kubernetes.
Some sample deployments that can be used are found under the
[deployments/scheduler](https://github.com/apache/yunikorn-k8shim/tree/master/deployments/scheduler)
directory of the `yunikorn-k8shim` repository. Alternatively, the Helm charts
located within the [helm-charts](https://github.com/apache/yunikorn-release/tree/master/helm-charts)
directory of the `yunikorn-release` repository may be used. These match what is used
for release builds.

The default image tags are not suitable for deployments to an accessible repository as it uses a hardcoded user and would push to Docker Hub with proper credentials.
You *must* update the `TAG` variable in the `Makefile` to push to an accessible repository.
When you update the image tag be aware that the deployment examples given will also need to be updated to reflect the same change.
The configuration of YuniKorn can be customized via a ConfigMap as explained in the
[scheduler configuration deployment](developer_guide/deployment.md) document.

### Inspect the docker image
The `make image` build command will first build the integrated executables and
then create the docker images. If you want to use pre-built images based on an
offical release, please check the [Docker Hub repo](https://hub.docker.com/r/apache/yunikorn).

The docker image built from previous step has embedded some important build info in image's metadata. You can retrieve
these info with docker `inspect` command.
The default image tags are not suitable for deployments to a private
repository as these would attempt to push to Docker Hub without proper
credentials. You *must* update the `REGISTRY` variable in the `Makefile` to
push to an accessible repository. When you update the image tag be aware that
the deployment examples given will also need to be updated to reflect the same
change.

```
### Inspect Docker images

The Docker images built from previous step have embedded some important build
info in the image metadata. You can retrieve this information with docker
`inspect` command:

```shell script
docker inspect apache/yunikorn:scheduler-amd64-latest
docker inspect apache/yunikorn:scheduler-plugin-amd64-latest
docker inspect apache/yunikorn:admission-controller-amd64-latest
```

The `amd64` tag is dependent on your host architecture (i.e. for Intel it would be `amd64` and for Mac M1, it would be `arm64v8`).
The `amd64` tag is dependent on your host architecture (i.e. for Intel it would
be `amd64` and for Mac M1, it would be `arm64`).

This info includes git revisions (last commit SHA) for each component, to help you understand which version of the source code
was shipped by this image. They are listed as docker image `labels`, such as
This info includes git revisions (last commit SHA) for each component, to help
you understand which version of the source code was shipped by this image. They
are listed as docker image `labels`, such as

```
```json
"Labels": {
"BuildTimeStamp": "2019-07-16T23:08:06+0800",
"Version": "0.1",
Expand All @@ -105,88 +116,129 @@ was shipped by this image. They are listed as docker image `labels`, such as

### Dependencies

The dependencies in the projects are managed using [go modules](https://blog.golang.org/using-go-modules).
Go Modules require at least Go version 1.11 to be installed on the development system.

If you want to modify one of the projects locally and build with your local dependencies you will need to change the module file.
Changing dependencies uses mod `replace` directives as explained in the [Update dependencies](#updating-dependencies).

The YuniKorn project has four repositories three of those repositories have a dependency at the go level.
These dependencies are part of the go modules and point to the github repositories.
During the development cycle it can be required to break the dependency on the committed version from github.
This requires making changes in the module file to allow loading a local copy or a forked copy from a different repository.
The dependencies in the projects are managed using
[go modules](https://blog.golang.org/using-go-modules).

If you want to modify one of the projects locally and build with your local
dependencies you will need to change the module file. Changing dependencies
requires using `go.mod` `replace` directives as explained in the
[Update dependencies](#updating-dependencies) section.

The YuniKorn project has four code repositories:
- [yunikorn-scheduler-interface](https://github.com/apache/yunikorn-scheduler-interface)
(protobuf interface between core and shim)
- [yunikorn-core](https://github.com/apache/yunikorn-core)
(core scheduler logic)
- [yunikorn-k8shim](https://github.com/apache/yunikorn-k8shim)
(Kubernetes-specific shim)
- [yunikorn-web](https://github.com/apache/yunikorn-web)
(YuniKorn Web UI)

Each of these dependencies is a Go module and there are dependencies between
them. During the development cycle it can be required to break the dependency
on the committed version from github. This requires making changes in the module
file to allow loading a local copy or a forked copy from a different repository.

Additionally, there are two additional auxiliary repositories:
- [yunikorn-release](https://github.com/apache/yunikorn-release)
(release management scripts and official Helm charts)
- [yunikorn-site](https://github.com/apache/yunikorn-site)
(source of the yunikorn.apache.org web site)

#### Affected repositories
The following dependencies exist between the repositories:

| repository| depends on |
| Repository| Depends on |
| --- | --- |
| yunikorn-core | yunikorn-scheduler-interface |
| yunikorn-k8shim | yunikorn-scheduler-interface, yunikorn-core |
| yunikorn-scheduler-interface | none |
| yunikorn-web | yunikorn-core |
| yunikorn-web | none |

The `yunikorn-web` repository has no direct go dependency on the other repositories. However any change to the `yunikorn-core` webservices can affect the web interface.
The `yunikorn-web` repository has no direct go dependency on the other
repositories. However any change to the `yunikorn-core` web services can affect
the web interface.

#### Making local changes

To make sure that the local changes will not break other parts of the build you should run:
To make sure that the local changes will not break other parts of the
build you should run:
- A full build `make` (build target depends on the repository)
- A full unit test run `make test`

Any test failures should be fixed before proceeding.

#### Updating dependencies

The simplest way is to use the `replace` directive in the module file. The `replace` directive allows you to override the import path with a new (local) path.
There is no need to change any of the imports in the source code. The change must be made in the `go.mod` file of the repository that has the dependency.
The simplest way is to use the `replace` directive in the module file.
The `replace` directive allows you to override the import path with a new
(local) path. There is no need to change any of the imports in the source code.
The change must be made in the `go.mod` file of the repository that has the
dependency.

Using `replace` to use of a forked dependency, such as:
```
replace github.com/apache/yunikorn-core => example.com/some/forked-yunikorn
```

There is no requirement to fork and create a new repository. If you do not have a repository you can use a local checked out copy too.
There is no requirement to fork and create a new repository. If you do not have
a repository you can use a local checked out copy too.

Using `replace` to use of a local directory as a dependency:
```
replace github.com/apache/yunikorn-core => /User/example/local/checked-out-yunikorn
```
and for the same dependency using a relative path:

For the same dependency using a relative path:
```
replace github.com/apache/yunikorn-core => ../checked-out-yunikorn
```
Note: if the `replace` directive is using a local filesystem path, then the target must have the `go.mod` file at that location.
Note: if the `replace` directive is using a local filesystem path, then the target
must have a `go.mod` file at that location.

Further details on the modules' wiki: [When should I use the 'replace' directive?](https://github.com/golang/go/wiki/Modules#when-should-i-use-the-replace-directive).
Further details can be found on the Go Wiki:
[When should I use the 'replace' directive?](https://github.com/golang/go/wiki/Modules#when-should-i-use-the-replace-directive)

## Build the web UI
## Build the Web UI

Example deployments reference the [YuniKorn web UI](https://github.com/apache/yunikorn-web).
The `yunikorn-web` project has specific requirements for the build. Follow the steps in the [README](https://github.com/apache/yunikorn-web/blob/master/README.md) to prepare a development environment and build the web UI. However, the scheduler is fully functional without the web UI.
Example deployments reference the
[YuniKorn Web UI](https://github.com/apache/yunikorn-web). The `yunikorn-web`
project has specific requirements for the build. Follow the steps in the
[README](https://github.com/apache/yunikorn-web/blob/master/README.md) to prepare
a development environment and build the Web UI. However, the scheduler is fully
functional without the Web UI.

## Locally run the integrated scheduler
## Run YuniKorn locally

When you have a local development environment setup you can run the scheduler in your local Kubernetes environment.
This has been tested in a desktop enviornment with Docker Desktop, Minikube, and Kind. See the [environment setup guide](developer_guide/env_setup.md) for further details.
When you have a local development environment setup you can run the scheduler
in your local Kubernetes environment. This has been tested in a desktop
enviornment with Docker Desktop, Minikube, and Kind. See the
[Dev Environment Setup](developer_guide/env_setup.md) guide for further details.

```
To run a local instance of the scheduler:

```shell script
make run
```
It will connect with the kubernetes cluster using the users configured configuration located in `$HOME/.kube/config`.

This will launch a local scheduler and connect to the Kubernetes cluster
referenced in your `KUBECONFIG` or `$HOME/.kube/config`.

To run YuniKorn in Kubernetes scheduler plugin mode instead, execute:

```
make run_plugin
```

You can also use the same approach to run the scheduler locally but connecting to a remote kubernetes cluster,
as long as the `$HOME/.kube/config` file is pointing to that remote cluster.

You can also use the same approach to run the scheduler locally but connecting
to a remote kubernetes cluster, as long as the `$HOME/.kube/config` file
is pointing to that remote cluster.

## Verify external interface changes with e2e tests
## Run end-to-end tests

Yunikorn has an external REST interface which is validated by end-to-end tests. However, the tests exist in the k8shim repository.
Whenever a change is made to the external interface, make sure that it is validated by running e2e tests or adjust the test cases accordingly.
In addition to the unit tests for each project, YuniKorn contains many e2e
(end-to-end) tests in the `yunikorn-k8shim` repository which validate
functionaliy of the scheduler on a functioning Kubernetes cluster.

How to run the tests locally is described [here](https://github.com/apache/yunikorn-k8shim/blob/master/test/e2e/README.md).
How to run the tests locally is described
[here](https://github.com/apache/yunikorn-k8shim/blob/master/test/e2e/README.md).
Loading

0 comments on commit 70ad1f0

Please sign in to comment.