Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Devfile endpoint annotations in Che Router #1881

Conversation

AObuchow
Copy link
Contributor

@AObuchow AObuchow commented Aug 13, 2024

What does this PR do?

  • Adds the DWR endpoint annotations as annotations for the route (on OpenShift) or ingresse (on Kubernetes) created by the Che Router for a given endpoint

There are a few additional changes made in this PR that are not meant to be merged, though they may be desirable:

  • Using per-workspace storage as well as the UDI in the Che-Operator devfile. This was helpful for developing this patch in Che.

Screenshot/screencast of this PR

n/a

What issues does this PR fix or reference?

Fix eclipse-che/che#23064

How to test this PR?

  1. Install DWO built from this PR, as well as Che-Operator built from the current PR onto your cluster.
  • On OpenShift, this can easily be done by running ./build/scripts/olm/test-catalog-from-sources.sh from the root of this repo.
  • On minikube/kubernetes, you'll have to:
    1. Build DWO: From the root of the DWO repo run export DWO_IMG=quay.io/<username>/devworkspace-controller:endpoint-annotations && make docker (or just use my image: export DWO_IMG=quay.io/<username>/devworkspace-controller:endpoint-annotations-implementation )
    2. Install DWO: make install_cert_manager && make install
    3. Build Che-Operator: make docker-build docker-push IMG=<YOUR_OPERATOR_IMAGE>
    4. Install Che: chectl server:deploy -p minikube --che-operator-image=<YOUR_OPERATOR_IMAGE>
  1. Once both DWO and Che are installed onto your cluster, verify there's a Che Cluster CR instance created so that you can access the Che User dashboard
  2. From the Che User Dashboard, create a workspace with a devfile that defines endpoints with annotations, such as this one.
      endpoints:
        - name: https-python
          targetPort: 8080
          protocol: https
          annotation:
            my-annotation: test
            another-annotation: test2
  1. Once the workspace has started up, verify the routes & ingresses for the endpoint(s) with annotations have the expected annotations from the devfile. In my example devfile, your <workspaceid>-py-8080-https-python route/ingress should have my-annotation: test & another-annotation: test2, as shown below:
kind: Route
apiVersion: route.openshift.io/v1
metadata:
  annotations:
+    another-annotation: test2
    che.routing.controller.devfile.io/component-name: py
    che.routing.controller.devfile.io/endpoint-name: https-python
+    my-annotation: test
  resourceVersion: '48291'
  name: workspace4b929cbc569a4473-py-8080-https-python
(...)
  namespace: che-kube-admin-che-rwxo67
  ownerReferences:
    - apiVersion: controller.devfile.io/v1alpha1
      kind: DevWorkspaceRouting
      name: routing-workspace4b929cbc569a4473
      uid: 497a6df3-9e14-47f8-9fc6-33acfda6df77
      controller: true
      blockOwnerDeletion: true
  labels:
    app.kubernetes.io/part-of: che.eclipse.org
    controller.devfile.io/devworkspace_id: workspace4b929cbc569a4473
spec:
(...)

PR Checklist

As the author of this Pull Request I made sure that:

Reviewers

Reviewers, please comment how you tested the PR when approving it.

Copy link

openshift-ci bot commented Aug 13, 2024

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

annos := map[string]string{}
for k, v := range ingressAnnotations {
annos[k] = v
}
for k, v := range endpointAnnotations {
annos[k] = v
Copy link
Contributor Author

@AObuchow AObuchow Aug 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: the devfile endpoint annotations can actually override the values of the Che Cluster CR ingress annotations. To me, this makes sense, as it gives users more fine-grained control over the ingress annotations. However, I don't have a strong opinion on this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I realized we are not adding annotations from the Che Cluster CR's networking.annotations fields to the routes created by the Che Router, only the ingresses get these annotations.

This seems like a bug or an error in the documentation, so I've opened eclipse-che/che#23118 to track this. If this is indeed a bug, I could add the fix as part of this PR since it's closely related.

@AObuchow AObuchow changed the title Endpoint annotations che routing implementation Support Devfile endpoint annotations in Che Router Aug 13, 2024
@AObuchow
Copy link
Contributor Author

@tolusha Though this PR is still a draft, it should be good for review if you want to take a look already. In order for this PR to no longer be a draft, we'll need to release DWO 0.31.0, and I'll update this PR to pull in the latest version of DWO

@tolusha
Copy link
Contributor

tolusha commented Aug 28, 2024

Sounds good for me

@AObuchow
Copy link
Contributor Author

@tolusha the DWO 0.31.0 release is currently in progress. Once it's released, I will update this PR and it will be ready for review :)

@AObuchow AObuchow force-pushed the endpoint-annotations-che-routing-implementation branch from 0897b54 to 7cb6d74 Compare September 19, 2024 14:54
@AObuchow AObuchow marked this pull request as ready for review September 19, 2024 14:58
@AObuchow
Copy link
Contributor Author

@tolusha this PR is finally ready for review now that DWO 0.31.0 is being used in Che-Operator :)

@AObuchow
Copy link
Contributor Author

@tolusha by the way, if you end up wanting the devfile changes, I'll rewrite the commit messages for them (since they're currently marked as "do not merge").

@AObuchow
Copy link
Contributor Author

/retest-required

2 similar comments
@AObuchow
Copy link
Contributor Author

/retest-required

@AObuchow
Copy link
Contributor Author

/retest-required

@AObuchow
Copy link
Contributor Author

Not sure why the pro v14-che-behind-proxy test is failing, though it seems unrelated to this PR:

INFO[2024-09-19T17:04:27Z] Logs for container test in pod che-behind-proxy-ipi-conf-aws-blackholenetwork: 
INFO[2024-09-19T17:04:27Z] 
An error occurred (AlreadyExistsException) when calling the CreateStack operation: Stack [ci-op-mfqq3b2j-9dadd-shared-vpc-blackhole] already exists
{"component":"entrypoint","error":"wrapped process failed: exit status 254","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:84","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.internalRun","level":"error","msg":"Error executing test process","severity":"error","time":"2024-09-19T17:04:27Z"}
error: failed to execute wrapped command: exit status 254 
INFO[2024-09-19T17:04:27Z] Step che-behind-proxy-ipi-conf-aws-blackholenetwork failed after 9s. 
(...)
INFO[2024-09-19T17:06:58Z] Ran for 4m36s                                
ERRO[2024-09-19T17:06:58Z] Some steps failed:                           
ERRO[2024-09-19T17:06:58Z] 
  * could not run steps: step che-behind-proxy failed: "che-behind-proxy" pre steps failed: "che-behind-proxy" pod "che-behind-proxy-ipi-conf-aws-blackholenetwork" failed: could not watch pod: the pod ci-op-mfqq3b2j/che-behind-proxy-ipi-conf-aws-blackholenetwork failed after 7s (failed containers: test): ContainerFailed one or more containers exited
Container test exited with code 254, reason Error
---
An error occurred (AlreadyExistsException) when calling the CreateStack operation: Stack [ci-op-mfqq3b2j-9dadd-shared-vpc-blackhole] already exists
{"component":"entrypoint","error":"wrapped process failed: exit status 254","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:84","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.internalRun","level":"error","msg":"Error executing test process","severity":"error","time":"2024-09-19T17:04:27Z"}
error: failed to execute wrapped command: exit status 254
--- 

@ibuziuk
Copy link
Member

ibuziuk commented Sep 20, 2024

/retest

devfile.yaml Outdated
metadata:
name: che-operator
components:
- name: tooling-container
container:
image: quay.io/eclipse/che-operator-dev:latest
image: quay.io/devfile/universal-developer-image:ubi8-latest
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, revert this line.
dev image is based on quay.io/devfile/universal-developer-image:ubi8-latest

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, I removed the associated commit. However, the reason why I changed the image being used is because the che-operator-dev container image is outdated and needs to be rebuilt to pull in the latest changes from the UDI.

}

func finalizeIngressAnnotations(ingressAnnotations map[string]string, machineName string, endpointName string) map[string]string {
func finalizeIngressAnnotations(ingressAnnotations, endpointAnnotations map[string]string, machineName string, endpointName string) map[string]string {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the order of parameters ingressAnnotations and endpointAnnotations correct?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes: ingressAnnotations refers to the annotations to add to the ingress provided by the Che Cluster CR, and endpointAnnotatiosn refers to the annotations provided by the devfile. It's worth ensuring we want the devfile annotations to be able to override the Che Cluster CR annotations, see https://github.com/eclipse-che/che-operator/pull/1881/files#r1714630847

When we call this function, we're first providing the IngressExposer's annotations (coming from the Che Cluster CR, see the last parameter of this function call) and then the endpoint's annotations (coming from the devfile through the devworkspace routing's endpoint)

Annotations:     finalizeIngressAnnotations(endpoint.annotations, e.ingressAnnotations, endpoint.componentName, endpoint.endpointName),

Hopefully this clarifies things - let me know if I misinterpreted your question.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In line 201, should it be the other way around, like this?

Annotations:     finalizeIngressAnnotations(e.ingressAnnotations, endpoint.annotations, endpoint.componentName, endpoint.endpointName),

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see now, can't believe I made that mistake. Good catch @tolusha @dkwon17 I'll fix this immediately 👍

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be good now, see d6c1724.

Once verified and approved, I'll squash the fixup commit :)

@AObuchow AObuchow force-pushed the endpoint-annotations-che-routing-implementation branch from 7cb6d74 to 0fe0757 Compare September 22, 2024 18:35
@openshift-ci openshift-ci bot removed the lgtm label Sep 22, 2024
@AObuchow
Copy link
Contributor Author

/retest-required

2 similar comments
@AObuchow
Copy link
Contributor Author

/retest-required

@AObuchow
Copy link
Contributor Author

/retest-required

@AObuchow AObuchow force-pushed the endpoint-annotations-che-routing-implementation branch from 22e1bd3 to c0afbf9 Compare September 23, 2024 21:00
Copy link

openshift-ci bot commented Sep 23, 2024

@AObuchow: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/v14-che-behind-proxy c0afbf9 link true /test v14-che-behind-proxy

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@AObuchow AObuchow merged commit 6164f7b into eclipse-che:main Sep 23, 2024
18 of 19 checks passed
@devstudio-release
Copy link

Build 3.17 :: operator_3.x/404: Console, Changes, Git Data

@devstudio-release
Copy link

@devstudio-release
Copy link

@devstudio-release
Copy link

@devstudio-release
Copy link

@devstudio-release
Copy link

Build 3.17 :: operator_3.x/404: SUCCESS

Upstream sync done; /DS_CI/sync-to-downstream_3.x/7729 triggered

@devstudio-release
Copy link

@devstudio-release
Copy link

@devstudio-release
Copy link

@devstudio-release
Copy link

@devstudio-release
Copy link

@devstudio-release
Copy link

Build 3.17 :: copyIIBsToQuay/2802: Console, Changes, Git Data

@devstudio-release
Copy link

Build 3.17 :: sync-to-downstream_3.x/7730: SUCCESS

Build container: devspaces-operator-bundle synced; /DS_CI/get-sources-rhpkg-container-build_3.x/7724 triggered; /job/DS_CI/job/dsc_3.x triggered;

@devstudio-release
Copy link

Build 3.17 :: operator-bundle_3.x/3619: SUCCESS

Upstream sync done; /DS_CI/sync-to-downstream_3.x/7730 triggered

@devstudio-release
Copy link

Build 3.17 :: dsc_3.x/2005: Console, Changes, Git Data

@devstudio-release
Copy link

Build 3.17 :: update-digests_3.x/7609: SUCCESS

Detected new images: rebuild operator-bundle
* devspaces-operator; /DS_CI/operator-bundle_3.x/3619 triggered

@devstudio-release
Copy link

Build 3.17 :: dsc_3.x/2005: SUCCESS

3.17.0-CI

@devstudio-release
Copy link

Build 3.17 :: copyIIBsToQuay/2802: SUCCESS

3.17
arches = x86_64, s390x, ppc64le;
  * LATEST DS OPERATOR BUNDLE = <a href=https://quay.io/repository/devspaces/devspaces-operator-bundle?tab=tags>registry-proxy.engineering.redhat.com/rh-osbs/devspaces-operator-bundle:3.17-16
  * LATEST DWO OPERATOR BUNDLE = <a href=https://quay.io/repository/devworkspace/devworkspace-operator-bundle?tab=tags>registry-proxy.engineering.redhat.com/rh-osbs/devworkspace-operator-bundle:???
+ x86_64-rhel8 IIB(s) copied:
  + <a href=https://quay.io/devspaces/iib:3.17-v4.17-821034- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-x86_64>quay.io/devspaces/iib:3.17-v4.17-821034- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-x86_64
  + quay.io/devspaces/iib:3.17-v4.17-x86_64
  + <a href=https://quay.io/devspaces/iib:3.17-v4.16-821035- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-x86_64>quay.io/devspaces/iib:3.17-v4.16-821035- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-x86_64
  + quay.io/devspaces/iib:3.17-v4.16-x86_64
  + <a href=https://quay.io/devspaces/iib:3.17-v4.15-823183- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-x86_64>quay.io/devspaces/iib:3.17-v4.15-823183- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-x86_64
  + quay.io/devspaces/iib:3.17-v4.15-x86_64
  + <a href=https://quay.io/devspaces/iib:3.17-v4.14-823181- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-x86_64>quay.io/devspaces/iib:3.17-v4.14-823181- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-x86_64
  + quay.io/devspaces/iib:3.17-v4.14-x86_64
  + <a href=https://quay.io/devspaces/iib:3.17-v4.13-823180- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-x86_64>quay.io/devspaces/iib:3.17-v4.13-823180- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-x86_64
  + quay.io/devspaces/iib:3.17-v4.13-x86_64
  + <a href=https://quay.io/devspaces/iib:3.17-v4.12-823203- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-x86_64>quay.io/devspaces/iib:3.17-v4.12-823203- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-x86_64
  + quay.io/devspaces/iib:3.17-v4.12-x86_64
+ s390x-rhel8 IIB(s) copied:
  + <a href=https://quay.io/devspaces/iib:3.17-v4.17-821034- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-s390x>quay.io/devspaces/iib:3.17-v4.17-821034- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-s390x
  + quay.io/devspaces/iib:3.17-v4.17-s390x
  + <a href=https://quay.io/devspaces/iib:3.17-v4.16-821035- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-s390x>quay.io/devspaces/iib:3.17-v4.16-821035- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-s390x
  + quay.io/devspaces/iib:3.17-v4.16-s390x
  + <a href=https://quay.io/devspaces/iib:3.17-v4.15-823183- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-s390x>quay.io/devspaces/iib:3.17-v4.15-823183- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-s390x
  + quay.io/devspaces/iib:3.17-v4.15-s390x
  + <a href=https://quay.io/devspaces/iib:3.17-v4.14-823181- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-s390x>quay.io/devspaces/iib:3.17-v4.14-823181- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-s390x
  + quay.io/devspaces/iib:3.17-v4.14-s390x
  + <a href=https://quay.io/devspaces/iib:3.17-v4.13-823180- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-s390x>quay.io/devspaces/iib:3.17-v4.13-823180- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-s390x
  + quay.io/devspaces/iib:3.17-v4.13-s390x
  + <a href=https://quay.io/devspaces/iib:3.17-v4.12-823203- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-s390x>quay.io/devspaces/iib:3.17-v4.12-823203- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-s390x
  + quay.io/devspaces/iib:3.17-v4.12-s390x
+ ppc64le-rhel8 IIB(s) copied:
  + <a href=https://quay.io/devspaces/iib:3.17-v4.17-821034- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-ppc64le>quay.io/devspaces/iib:3.17-v4.17-821034- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-ppc64le
  + quay.io/devspaces/iib:3.17-v4.17-ppc64le
  + <a href=https://quay.io/devspaces/iib:3.17-v4.16-821035- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-ppc64le>quay.io/devspaces/iib:3.17-v4.16-821035- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-ppc64le
  + quay.io/devspaces/iib:3.17-v4.16-ppc64le
  + <a href=https://quay.io/devspaces/iib:3.17-v4.15-823183- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-ppc64le>quay.io/devspaces/iib:3.17-v4.15-823183- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-ppc64le
  + quay.io/devspaces/iib:3.17-v4.15-ppc64le
  + <a href=https://quay.io/devspaces/iib:3.17-v4.14-823181- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-ppc64le>quay.io/devspaces/iib:3.17-v4.14-823181- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-ppc64le
  + quay.io/devspaces/iib:3.17-v4.14-ppc64le
  + <a href=https://quay.io/devspaces/iib:3.17-v4.13-823180- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-ppc64le>quay.io/devspaces/iib:3.17-v4.13-823180- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-ppc64le
  + quay.io/devspaces/iib:3.17-v4.13-ppc64le
  + <a href=https://quay.io/devspaces/iib:3.17-v4.12-823203- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-ppc64le>quay.io/devspaces/iib:3.17-v4.12-823203- /tmp/getIIBsForBundle.sh -t PROD_VER [OPTIONS]-ppc64le
  + quay.io/devspaces/iib:3.17-v4.12-ppc64le

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support devfile endpoint annotations
5 participants