chore: updating pubsub system #3646

JaydipGabani · 2024-10-15T05:50:11Z

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, using fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when the PR gets merged):
Fixes #1037 #3497

Special notes for your reviewer:

codecov-commenter · 2024-10-15T22:06:18Z

Codecov Report

Attention: Patch coverage is 52.83019% with 75 lines in your changes missing coverage. Please review.

Project coverage is 47.72%. Comparing base (3350319) to head (55d941e).
Report is 259 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/export/dapr/fake_dapr_client.go	24.13%	22 Missing ⚠️
pkg/export/dapr/dapr.go	65.95%	14 Missing and 2 partials ⚠️
pkg/export/testdriver/testdriver.go	36.84%	11 Missing and 1 partial ⚠️
pkg/controller/export/export_config_controller.go	8.33%	11 Missing ⚠️
pkg/export/system.go	79.54%	6 Missing and 3 partials ⚠️
pkg/audit/manager.go	0.00%	5 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (3350319) and HEAD (55d941e). Click for more details.

HEAD has 1 upload less than BASE

Flag BASE (3350319) HEAD (55d941e)

unittests 2 1

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3646      +/-   ##
==========================================
- Coverage   54.49%   47.72%   -6.78%     
==========================================
  Files         134      234     +100     
  Lines       12329    19864    +7535     
==========================================
+ Hits         6719     9480    +2761     
- Misses       5116     9493    +4377     
- Partials      494      891     +397

Flag	Coverage Δ
unittests	`47.72% <52.83%> (-6.78%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

JaydipGabani · 2024-10-23T01:32:19Z

Still working on doc changes.

JaydipGabani · 2024-12-05T23:31:13Z

pkg/controller/export/export_config_controller.go

@@ -26,37 +26,37 @@ import (
 )

 var (
-	PubsubEnabled = flag.Bool("enable-pub-sub", false, "(alpha) Enabled pubsub to publish messages")
+	ExportEnabled = flag.Bool("enable-pub-sub", false, "(alpha) Enabled pubsub to publish messages")


Do we want to rename the flag to remove pub-sub word?

May as well while it's still alpha

should we provide a warning for the user if we change the name? we can remove it after a release?

What do you mean? How would we provide warning for changing a name before release?

JaydipGabani · 2024-12-31T19:26:10Z

Fixes #3497

maxsmythe · 2025-01-16T02:44:54Z

pkg/controller/export/export_config_controller.go

@@ -114,22 +114,22 @@ func (r *Reconciler) Reconcile(ctx context.Context, request reconcile.Request) (
 	}

 	if len(cfg.Data) == 0 {
-		return reconcile.Result{}, fmt.Errorf(fmt.Sprintf("data missing in configmap %s, unable to configure respective pubsub", request.NamespacedName))
+		return reconcile.Result{}, fmt.Errorf(fmt.Sprintf("data missing in configmap %s, unable to establish connection", request.NamespacedName))


is it a "connection" necessarily?

"link" may be? I just couldn't come up with better alternative at the time.

"unable to configure exporter"?

maxsmythe · 2025-01-16T02:46:34Z

pkg/controller/export/export_config_controller.go

 	}
-	if _, ok := cfg.Data["provider"]; !ok {
-		return reconcile.Result{}, fmt.Errorf(fmt.Sprintf("missing provider field in configmap %s, unable to configure respective pubsub", request.NamespacedName))
+	if _, ok := cfg.Data["driver"]; !ok {


Is changing from config map to CRD coming later?

I can raise a follow up PR, we have the design finalized as per last discussion.

sozercan

minor comments, otherwise LGTM

sozercan · 2025-01-23T19:29:26Z

pkg/controller/export/export_config_controller.go

 }

 func (a *Adder) Add(mgr manager.Manager) error {
-	if !*PubsubEnabled {
+	if !*ExportEnabled {
 		return nil
 	}
 	log.Info("Warning: Alpha flag enable-pub-sub is set to true. This flag may change in the future.")


update this too if we change the flag name

sozercan · 2025-01-23T19:29:41Z

pkg/controller/export/export_config_controller.go

@@ -114,22 +114,22 @@ func (r *Reconciler) Reconcile(ctx context.Context, request reconcile.Request) (
 	}

 	if len(cfg.Data) == 0 {
-		return reconcile.Result{}, fmt.Errorf(fmt.Sprintf("data missing in configmap %s, unable to configure respective pubsub", request.NamespacedName))
+		return reconcile.Result{}, fmt.Errorf(fmt.Sprintf("data missing in configmap %s, unable to establish connection", request.NamespacedName))


"unable to configure exporter"?

sozercan · 2025-01-23T19:31:59Z

website/docs/export-driver-walkthrough.md

+title: Export Interface/Driver walkthrough
+---
+
+This guide provides an overview of the driver interface, including details of its structure and functionality. Additionally, it offers instructions on adding a new driver and utilizing different backends to export violations.


Suggested change

This guide provides an overview of the driver interface, including details of its structure and functionality. Additionally, it offers instructions on adding a new driver and utilizing different backends to export violations.

This guide provides an overview of the driver interface, including details of its structure and functionality. Additionally, it offers instructions on adding a new driver and utilizing different backends to export audit violations.

pkg/audit/manager.go

Signed-off-by: Jaydip Gabani <[email protected]>

ritazh · 2025-01-30T06:02:37Z

.github/workflows/dapr-pubsub.yaml

@@ -1,18 +1,18 @@
-name: dapr-pubsub


since you are renaming everything, does this yaml need to be renamed too?

good call, ty!

Updated the file name.

Signed-off-by: Jaydip Gabani <[email protected]>

ritazh · 2025-02-08T01:16:45Z

website/docs/audit.md


-This feature uses publish and subscribe (pubsub) model that allows Gatekeeper to export audit violations over a broker that can be consumed by a subscriber independently. Therefore, pubsub violations are not subject to reporting limits. Please refer to [this](pubsub.md) guide to configure audit to push violations over a channel.
+This feature allows plugging in different backends that allows Gatekeeper to export audit violations. Therefore, violations are not subject to reporting limits. Please refer to [this](export.md) guide to configure audit to push violations via this feature.


Suggested change

This feature allows plugging in different backends that allows Gatekeeper to export audit violations. Therefore, violations are not subject to reporting limits. Please refer to [this](export.md) guide to configure audit to push violations via this feature.

This feature allows different systems to be provided for Gatekeeper to export audit violations. Thereby, ensuing violations are not subject to reporting limits. Please refer to [this](export.md) guide to configure audit to push violations via this feature.

ritazh · 2025-02-08T01:18:04Z

website/docs/audit.md

- There is an inherent risk of messages getting dropped. You might not receive all the published violations.
- Additional dependency on pubsub broker. 
+- There is a risk of messages getting dropped. You might not receive all the exported violations. This depends on the type of backend you are using for delivery. For example, using a network as backend to export violation has the risk of messages getting dropped.
+- Additional dependency depending on what is plugged in. For example, using pubsub tools to export violations.


Suggested change

- Additional dependency depending on what is plugged in. For example, using pubsub tools to export violations.

- Additional dependency on the backend system provided. For example, using pubsub tools to export violations.

ritazh · 2025-02-08T01:18:47Z

website/docs/audit.md


 ## Running Audit
 For more details on how to deploy audit and 
 number of instances to run, please refer to [operations audit](operations.md#audit). 

 ## Configuring Audit

- Audit violations per constraint: set `--constraint-violations-limit=123` (defaults to `20`). NOTE: This flag only impacts when gathering audit results using the constraint status model. If you are gathering audit results using the pubsub model, please refer to the [pubsub](pubsub.md) guide. Both approaches for getting audit violations can be configured independently and work simultaneously without any interference.
+- Audit violations per constraint: set `--constraint-violations-limit=123` (defaults to `20`). NOTE: This flag only impacts when gathering audit results using the constraint status model. If you want to gather audit results via other means, please refer to the [exporting audit results](export.md) guide. Both approaches for getting audit violations can be configured independently and work simultaneously without any interference.


Suggested change

- Audit violations per constraint: set `--constraint-violations-limit=123` (defaults to `20`). NOTE: This flag only impacts when gathering audit results using the constraint status model. If you want to gather audit results via other means, please refer to the [exporting audit results](export.md) guide. Both approaches for getting audit violations can be configured independently and work simultaneously without any interference.

- Audit violations per constraint: set `--constraint-violations-limit=123` (defaults to `20`). NOTE: This flag only impacts when gathering audit results using the constraint status model. If you want to export audit results via other means, please refer to the [exporting audit results](export.md) guide. Both approaches for getting audit violations can be configured independently and work simultaneously without any interference.

ritazh · 2025-02-08T01:21:22Z

website/docs/export.md

-id: pubsub
-title: Consuming violations using Pubsub
+id: export
+title: Exporting violations


do we want to add something like "Previously known as pubsub feature"?

I like this idea if we will be removing it when we go beta or in couple of releases.

Added This feature was previously known as "Consuming violations using Pubsub" below feature state.

ritazh · 2025-02-08T01:54:51Z

website/sidebars.js

@@ -34,7 +34,7 @@ module.exports = {
        'expansion',
        'gator',
        'workload-resources',
-        'pubsub',
+        'export',


How does this impact older versions of docs?

It doesn't, with older version of website it will route to pubsub.md. And once this PR gets merged, I doubt there would be any maintenance work for old docs since we are changing lots of things in the feature including name.

Signed-off-by: Jaydip Gabani <[email protected]>

ritazh · 2025-02-12T00:26:15Z

website/docs/audit.md


-This feature uses publish and subscribe (pubsub) model that allows Gatekeeper to export audit violations over a broker that can be consumed by a subscriber independently. Therefore, pubsub violations are not subject to reporting limits. Please refer to [this](pubsub.md) guide to configure audit to push violations over a channel.
+This feature allows different systems to be provided for Gatekeeper to export audit violations. Thereby, ensuing violations are not subject to reporting limits. Please refer to [this](export.md) guide to configure audit to push violations via this feature.


Suggested change

This feature allows different systems to be provided for Gatekeeper to export audit violations. Thereby, ensuing violations are not subject to reporting limits. Please refer to [this](export.md) guide to configure audit to push violations via this feature.

This feature allows different systems to be provided for Gatekeeper to export audit violations. Thereby, ensuring violations are not subject to reporting limits. Please refer to [the export guide](export.md) to configure audit to push violations via this feature.

Updated the docs.

ritazh · 2025-02-12T00:27:41Z

website/docs/audit.md


 ## Running Audit
 For more details on how to deploy audit and 
 number of instances to run, please refer to [operations audit](operations.md#audit). 

 ## Configuring Audit

- Audit violations per constraint: set `--constraint-violations-limit=123` (defaults to `20`). NOTE: This flag only impacts when gathering audit results using the constraint status model. If you are gathering audit results using the pubsub model, please refer to the [pubsub](pubsub.md) guide. Both approaches for getting audit violations can be configured independently and work simultaneously without any interference.
+- Audit violations per constraint: set `--constraint-violations-limit=123` (defaults to `20`). NOTE: This flag only impacts when gathering audit results using the constraint status model. If you want to export audit results via other means, please refer to the [exporting audit results](export.md) guide. Both approaches for getting audit violations can be configured independently and work simultaneously without any interference.


Suggested change

- Audit violations per constraint: set `--constraint-violations-limit=123` (defaults to `20`). NOTE: This flag only impacts when gathering audit results using the constraint status model. If you want to export audit results via other means, please refer to the [exporting audit results](export.md) guide. Both approaches for getting audit violations can be configured independently and work simultaneously without any interference.

- Audit violations per constraint: set `--constraint-violations-limit=123` (defaults to `20`). NOTE: This flag only impacts when gathering audit results using the constraint status model. If you want to export audit results via other means, please refer to [the export guide](export.md). Both approaches for getting audit violations can be configured independently and work simultaneously without any interference.

ritazh

Nit, otherwise LGTM

Signed-off-by: Jaydip Gabani <[email protected]>

maxsmythe

LGTM

JaydipGabani changed the title ~~[WIP] chore: updating violation system~~ [WIP] chore: updating pubsub system Oct 21, 2024

JaydipGabani force-pushed the exporter-interface branch from 570505d to 7c48595 Compare October 22, 2024 23:55

JaydipGabani changed the title ~~[WIP] chore: updating pubsub system~~ chore: updating pubsub system Oct 23, 2024

JaydipGabani marked this pull request as ready for review October 23, 2024 01:31

JaydipGabani requested a review from a team as a code owner October 23, 2024 01:31

JaydipGabani commented Dec 6, 2024

View reviewed changes

ritazh added this to the v3.19.0 milestone Dec 11, 2024

JaydipGabani mentioned this pull request Dec 31, 2024

Modify Pub/Sub interface to common exporter interface #3497

Closed

maxsmythe reviewed Jan 16, 2025

View reviewed changes

JaydipGabani force-pushed the exporter-interface branch from bf0c5cf to 1f2905e Compare January 24, 2025 22:12

JaydipGabani requested review from sozercan, maxsmythe and ritazh January 24, 2025 22:13

sozercan reviewed Jan 28, 2025

View reviewed changes

updating pubsub system

64e2ae5

Signed-off-by: Jaydip Gabani <[email protected]>

JaydipGabani force-pushed the exporter-interface branch from 1b578cb to 2fba5c2 Compare January 28, 2025 19:39

JaydipGabani requested a review from sozercan January 29, 2025 23:40

replacing pub-sub with export everywhere

95b5ab7

Signed-off-by: Jaydip Gabani <[email protected]>

JaydipGabani force-pushed the exporter-interface branch from cabc202 to 95b5ab7 Compare January 30, 2025 00:17

fixing helm variable name enableViolationExport

74d9a44

Signed-off-by: Jaydip Gabani <[email protected]>

ritazh reviewed Jan 30, 2025

View reviewed changes

JaydipGabani added 2 commits January 30, 2025 19:10

renaming dapr-pubsub -> dapr-export

b48385a

Signed-off-by: Jaydip Gabani <[email protected]>

Merge branch 'master' into exporter-interface

f4d5631

JaydipGabani requested a review from ritazh January 30, 2025 22:47

JaydipGabani force-pushed the exporter-interface branch from 68424cf to f4d5631 Compare January 31, 2025 01:56

JaydipGabani added 2 commits February 3, 2025 10:41

Merge branch 'master' into exporter-interface

8749ea6

Merge branch 'master' into exporter-interface

3d124af

Merge remote-tracking branch 'gatekeeper/master' into exporter-interface

1f2c01b

ritazh reviewed Feb 8, 2025

View reviewed changes

JaydipGabani added 2 commits February 11, 2025 02:28

updating docs

3cb5b98

Signed-off-by: Jaydip Gabani <[email protected]>

updating docs

bde239e

Signed-off-by: Jaydip Gabani <[email protected]>

JaydipGabani requested a review from ritazh February 11, 2025 18:13

Merge branch 'master' into exporter-interface

99b3718

ritazh reviewed Feb 12, 2025

View reviewed changes

ritazh approved these changes Feb 12, 2025

View reviewed changes

updating docs for exporting audit violations

55d941e

Signed-off-by: Jaydip Gabani <[email protected]>

maxsmythe approved these changes Feb 12, 2025

View reviewed changes

JaydipGabani merged commit 18c2fbf into open-policy-agent:master Feb 12, 2025
22 checks passed

JaydipGabani deleted the exporter-interface branch February 12, 2025 23:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: updating pubsub system #3646

chore: updating pubsub system #3646

JaydipGabani commented Oct 15, 2024 •

edited

Loading

codecov-commenter commented Oct 15, 2024 •

edited

Loading

JaydipGabani commented Oct 23, 2024

JaydipGabani Dec 5, 2024

maxsmythe Jan 16, 2025

sozercan Jan 23, 2025 •

edited

Loading

JaydipGabani Jan 24, 2025

JaydipGabani commented Dec 31, 2024

maxsmythe Jan 16, 2025

JaydipGabani Jan 16, 2025

sozercan Jan 23, 2025

maxsmythe Jan 16, 2025

JaydipGabani Jan 16, 2025

sozercan left a comment

sozercan Jan 23, 2025

sozercan Jan 23, 2025

sozercan Jan 23, 2025

ritazh Jan 30, 2025

JaydipGabani Jan 30, 2025

JaydipGabani Jan 30, 2025

ritazh Feb 8, 2025

ritazh Feb 8, 2025

ritazh Feb 8, 2025

ritazh Feb 8, 2025

JaydipGabani Feb 11, 2025

JaydipGabani Feb 11, 2025

ritazh Feb 8, 2025

JaydipGabani Feb 11, 2025

ritazh Feb 12, 2025

JaydipGabani Feb 12, 2025

ritazh Feb 12, 2025

JaydipGabani Feb 12, 2025

ritazh left a comment

maxsmythe left a comment

	This guide provides an overview of the driver interface, including details of its structure and functionality. Additionally, it offers instructions on adding a new driver and utilizing different backends to export violations.
	This guide provides an overview of the driver interface, including details of its structure and functionality. Additionally, it offers instructions on adding a new driver and utilizing different backends to export audit violations.


		This feature uses publish and subscribe (pubsub) model that allows Gatekeeper to export audit violations over a broker that can be consumed by a subscriber independently. Therefore, pubsub violations are not subject to reporting limits. Please refer to [this](pubsub.md) guide to configure audit to push violations over a channel.
		This feature allows plugging in different backends that allows Gatekeeper to export audit violations. Therefore, violations are not subject to reporting limits. Please refer to [this](export.md) guide to configure audit to push violations via this feature.

	This feature allows plugging in different backends that allows Gatekeeper to export audit violations. Therefore, violations are not subject to reporting limits. Please refer to [this](export.md) guide to configure audit to push violations via this feature.
	This feature allows different systems to be provided for Gatekeeper to export audit violations. Thereby, ensuing violations are not subject to reporting limits. Please refer to [this](export.md) guide to configure audit to push violations via this feature.

	- Additional dependency depending on what is plugged in. For example, using pubsub tools to export violations.
	- Additional dependency on the backend system provided. For example, using pubsub tools to export violations.

chore: updating pubsub system #3646

chore: updating pubsub system #3646

Conversation

JaydipGabani commented Oct 15, 2024 • edited Loading

codecov-commenter commented Oct 15, 2024 • edited Loading

Codecov Report

JaydipGabani commented Oct 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sozercan Jan 23, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JaydipGabani commented Dec 31, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sozercan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ritazh left a comment

Choose a reason for hiding this comment

maxsmythe left a comment

Choose a reason for hiding this comment

JaydipGabani commented Oct 15, 2024 •

edited

Loading

codecov-commenter commented Oct 15, 2024 •

edited

Loading

sozercan Jan 23, 2025 •

edited

Loading