Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial version of CEL resource expressions. #928

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

bigkevmcd
Copy link
Contributor

@bigkevmcd bigkevmcd commented Sep 17, 2024

This is a very rough cut (needs additional error checking).

But the basic implementation works.

This is the code I referenced here #491 (comment)

I will harden it up, checking that it's receiving a json body and improve the error handling, this was a quick bit of code to see if it was possible.

@@ -63,10 +63,24 @@ type ReceiverSpec struct {
// +optional
Events []string `json:"events,omitempty"`

// TODO: Validate one or other (or both?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will need to be addressed, because you could have .spec.resourceExpressions or .spec.resources (or both).

@bigkevmcd bigkevmcd force-pushed the cel-notification-resources branch 3 times, most recently from 786ff7e to 5da3f91 Compare September 17, 2024 19:26
Copy link
Collaborator

@matheuscscp matheuscscp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the work here @bigkevmcd, but I think we need to create an RFC or at least an issue to discuss this feature.

// Only decodes the body for the expression if the body is JSON.
// Technically you could generate several resources without any body.
if isJSONContent(r) {
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be reworked, the request body was already received at this point, in the line if err := s.validate(ctx, receiver, r); err != nil { from (s *ReceiverServer) handlePayload(). Reading it again won't be possible. To fix this you will need something like a https://pkg.go.dev/io#TeeReader with a https://pkg.go.dev/bytes#Buffer.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I missed this in the code above, I'll fix this.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another option is simply reading the body into a bytes.Buffer and passing it to all the parts of the code that need to consume it.

internal/server/receiver_handlers.go Show resolved Hide resolved
@matheuscscp
Copy link
Collaborator

Thanks for the work here @bigkevmcd, but I think we need to create an RFC or at least an issue to discuss this feature.

Looking again at the original issue, maybe we could use that issue to discuss this feature, it looks like in this PR you're specifically trying to address this part:

It would be nice to be able to map the digest to a ImageRepository to notify (either automatically or explicitly configured). This could potentially reduce the load within and caused by Flux drastically.

This adds support for dynamic generation of resources for receivers
using CEL and parsing the body in the content request.

Signed-off-by: Kevin McDermott <[email protected]>
@matheuscscp
Copy link
Collaborator

To fix the e2e GitHub Action failing you can run make api-docs manifests generate locally to generate the missing change for docs/api/v1/notification.md.

Signed-off-by: Kevin McDermott <[email protected]>
@matheuscscp
Copy link
Collaborator

matheuscscp commented Sep 18, 2024

Now there's a diff on config/crd/bases/notification.toolkit.fluxcd.io_receivers.yaml breaking the e2e GitHub Action. When I run make manifests (or my usual make api-docs manifests generate), I see the missing diff on this file locally. Can you please run this command and see if it generates this missing diff? Thanks!

@bigkevmcd
Copy link
Contributor Author

bigkevmcd commented Sep 18, 2024

Thanks for the work here @bigkevmcd, but I think we need to create an RFC or at least an issue to discuss this feature.

That's ok, it was a quick branch to explore what's possible, there's no compelling need to land this, and, it's taking on support for CEL, which is not "free".

Worst case, it gets closed as "won't fix" 😄 and informs a different solution, fine by me.

It would be nice to be able to map the digest to a ImageRepository to notify (either automatically or explicitly configured). This could potentially reduce the load within and caused by Flux drastically.

Yes, this is basically a way to dynamically generate the set of resources to notify in a receiver.

The new field .spec.resourceExpressions is basically a set of CEL expressions that can return the resources that should be notified, it could be simplified to a single resourceExpression but from experience with CEL elsewhere, this tends to lead to complex expressions.

@matheuscscp
Copy link
Collaborator

This solution sounds very elegant to me @bigkevmcd, I'd love to see it in. It would be nice if you could attend some of our dev meetings to discuss this feature, they are public and can be found here: https://fluxcd.io/community/#meetings

(The CNCF Flux Project Meeting ones.)

@bigkevmcd
Copy link
Contributor Author

I'll see what I can do to be there, time permitting.

You'll see that the specific request in here #491 (comment) is tested in the code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants