NETOBSERV-1517: remove IPFIX agent mode (#579)

* NETOBSERV-1517: remove IPFIX agent mode - Remove controller code for OVS configmap / CNO configuration - Remove related tests (and move console plugin related tests to its dedicated file) - Remove FLP ingester reconciler (was only used with ipfix) - Note that the IPFIX api is not removed to not introduce breaking changes in existing API, however it is now documented that using IPFIX would have no effect * Update config/descriptions/upstream.md Co-authored-by: Olivier Cazade <[email protected]> --------- Co-authored-by: Olivier Cazade <[email protected]>
netobserv · Mar 11, 2024 · 323e3f8 · 323e3f8
1 parent 3311834
commit 323e3f8
Show file tree

Hide file tree

Showing 39 changed files with 195 additions and 3,226 deletions.
diff --git a/README.md b/README.md
@@ -3,9 +3,10 @@
 ![GitHub release (latest by date)](https://img.shields.io/github/v/release/netobserv/network-observability-operator)
 [![Go Report Card](https://goreportcard.com/badge/github.com/netobserv/network-observability-operator)](https://goreportcard.com/report/github.com/netobserv/network-observability-operator)
 
-NetObserv Operator is a Kubernetes / OpenShift operator for network observability. It deploys a monitoring pipeline to collect and enrich network flows. These flows can be produced by the NetObserv eBPF agent, or by any device or CNI able to export flows in IPFIX format, such as OVN-Kubernetes.
-
-The operator provides dashboards, metrics, and keeps flows accessible in a queryable log store, Grafana Loki. When used in OpenShift, new views are available in the Console.
+NetObserv Operator is a Kubernetes / OpenShift operator for network observability. It deploys a monitoring pipeline that consists in:
+- the NetObserv eBPF agent, that generates network flows from captured packets
+- Flowlogs-pipeline, a component that collects, enriches and exports these flows.
+- When used in OpenShift, a Console plugin for flows visualization with powerful filtering options, a topology representation and more.
 
 ## Getting Started
 
@@ -122,11 +123,9 @@ As it operates cluster-wide, only a single `FlowCollector` is allowed, and it ha
 
 A couple of settings deserve special attention:
 
-- Agent (`spec.agent.type`) can be `eBPF` (default) or `IPFIX`. eBPF is recommended, as it should work in more situations and offers better performances. If you can't, or don't want to use eBPF, note that the IPFIX option is fully functional only when using [OVN-Kubernetes](https://github.com/ovn-org/ovn-kubernetes/) CNI. Other CNIs are not officially supported, but you may still be able to configure them manually if they allow IPFIX exports.
-
 - Agent features (`spec.agent.ebpf.features`) can enable more features such as tracking packet drops, TCP latency (RTT) and DNS requests and responses.
 
-- Sampling (`spec.agent.ebpf.sampling` and `spec.agent.ipfix.sampling`): a value of `100` means: one flow every 100 is sampled. `1` means all flows are sampled. The lower it is, the more flows you get, and the more accurate are derived metrics, but the higher amount of resources are consumed. By default, sampling is set to 50 (ie. 1:50) for eBPF and 400 (1:400) for IPFIX. Note that more sampled flows also means more storage needed. We recommend to start with default values and refine empirically, to figure out which setting your cluster can manage.
+- Sampling `spec.agent.ebpf.sampling`: a value of `100` means: one flow every 100 is sampled. `1` means all flows are sampled. The lower it is, the more flows you get, and the more accurate are derived metrics, but the higher amount of resources are consumed. By default, sampling is set to 50 (ie. 1:50). Note that more sampled flows also means more storage needed. We recommend to start with default values and refine empirically, to figure out which setting your cluster can manage.
 
 - Loki (`spec.loki`): configure here how to reach Loki. The default URL values match the Loki quick install paths mentioned in the _Getting Started_ section, but you may have to configure differently if you used another installation method. You will find more information in our guides for deploying Loki: [with Loki Operator](https://github.com/netobserv/documents/blob/main/loki_operator.md), or an alternative ["distributed Loki" guide](https://github.com/netobserv/documents/blob/main/loki_distributed.md). You should set `spec.loki.mode` according to the chosen installation method, for instance use `LokiStack` if you use the Loki Operator.
 

diff --git a/apis/flowcollector/v1beta1/flowcollector_types.go b/apis/flowcollector/v1beta1/flowcollector_types.go
@@ -86,12 +86,10 @@ type FlowCollectorSpec struct {
 // allow defining both fields.
 // +union
 type FlowCollectorAgent struct {
-	// `type` selects the flows tracing agent. Possible values are:<br>
-	// - `EBPF` (default) to use NetObserv eBPF agent.<br>
-	// - `IPFIX` [deprecated (*)] - to use the legacy IPFIX collector.<br>
-	// `EBPF` is recommended as it offers better performances and should work regardless of the CNI installed on the cluster.
-	// `IPFIX` works with OVN-Kubernetes CNI (other CNIs could work if they support exporting IPFIX,
-	// but they would require manual configuration).
+	// `type` [deprecated (*)] selects the flows tracing agent. The only possible value is `EBPF` (default), to use NetObserv eBPF agent.<br>
+	// Previously, using an IPFIX collector was allowed, but was deprecated and it is now removed.<br>
+	// Setting `IPFIX` is ignored and still use the eBPF Agent.
+	// Since there is only a single option here, this field will be remove in a future API version.
 	// +unionDiscriminator
 	// +kubebuilder:validation:Enum:="EBPF";"IPFIX"
 	// +kubebuilder:default:=EBPF
@@ -443,7 +441,7 @@ type FlowCollectorFLP struct {
 	EnableKubeProbes *bool `json:"enableKubeProbes,omitempty"`
 
 	//+kubebuilder:default:=true
-	// `dropUnusedFields` allows, when set to `true`, to drop fields that are known to be unused by OVS, to save storage space.
+	// `dropUnusedFields` [deprecated (*)] this setting is not used anymore.
 	DropUnusedFields *bool `json:"dropUnusedFields,omitempty"`
 
 	//+kubebuilder:validation:Minimum=0

diff --git a/apis/flowcollector/v1beta2/flowcollector_types.go b/apis/flowcollector/v1beta2/flowcollector_types.go
@@ -93,12 +93,10 @@ const (
 // allow defining both fields.
 // +union
 type FlowCollectorAgent struct {
-	// `type` selects the flows tracing agent. Possible values are:<br>
-	// - `eBPF` (default) to use NetObserv eBPF agent.<br>
-	// - `IPFIX` [deprecated (*)] - to use the legacy IPFIX collector.<br>
-	// `eBPF` is recommended as it offers better performances and should work regardless of the CNI installed on the cluster.
-	// `IPFIX` works with OVN-Kubernetes CNI (other CNIs could work if they support exporting IPFIX,
-	// but they would require manual configuration).
+	// `type` [deprecated (*)] selects the flows tracing agent. The only possible value is `eBPF` (default), to use NetObserv eBPF agent.<br>
+	// Previously, using an IPFIX collector was allowed, but was deprecated and it is now removed.<br>
+	// Setting `IPFIX` is ignored and still use the eBPF Agent.
+	// Since there is only a single option here, this field will be remove in a future API version.
 	// +unionDiscriminator
 	// +kubebuilder:validation:Enum:="eBPF";"IPFIX"
 	// +kubebuilder:default:=eBPF
@@ -920,7 +918,7 @@ type AdvancedProcessorConfig struct {
 
 	//+kubebuilder:default:=true
 	//+optional
-	// `dropUnusedFields` allows, when set to `true`, to drop fields that are known to be unused by OVS, to save storage space.
+	// `dropUnusedFields` [deprecated (*)] this setting is not used anymore.
 	DropUnusedFields *bool `json:"dropUnusedFields,omitempty"`
 
 	//+kubebuilder:default:="30s"

diff --git a/bundle/manifests/flows.netobserv.io_flowcollectors.yaml b/bundle/manifests/flows.netobserv.io_flowcollectors.yaml
@@ -445,13 +445,13 @@ spec:
                     type: object
                   type:
                     default: EBPF
-                    description: '`type` selects the flows tracing agent. Possible
-                      values are:<br> - `EBPF` (default) to use NetObserv eBPF agent.<br>
-                      - `IPFIX` [deprecated (*)] - to use the legacy IPFIX collector.<br>
-                      `EBPF` is recommended as it offers better performances and should
-                      work regardless of the CNI installed on the cluster. `IPFIX`
-                      works with OVN-Kubernetes CNI (other CNIs could work if they
-                      support exporting IPFIX, but they would require manual configuration).'
+                    description: '`type` [deprecated (*)] selects the flows tracing
+                      agent. The only possible value is `EBPF` (default), to use NetObserv
+                      eBPF agent.<br> Previously, using an IPFIX collector was allowed,
+                      but was deprecated and it is now removed.<br> Setting `IPFIX`
+                      is ignored and still use the eBPF Agent. Since there is only
+                      a single option here, this field will be remove in a future
+                      API version.'
                     enum:
                     - EBPF
                     - IPFIX
@@ -1875,9 +1875,8 @@ spec:
                     type: object
                   dropUnusedFields:
                     default: true
-                    description: '`dropUnusedFields` allows, when set to `true`, to
-                      drop fields that are known to be unused by OVS, to save storage
-                      space.'
+                    description: '`dropUnusedFields` [deprecated (*)] this setting
+                      is not used anymore.'
                     type: boolean
                   enableKubeProbes:
                     default: true
@@ -3249,13 +3248,13 @@ spec:
                     type: object
                   type:
                     default: eBPF
-                    description: '`type` selects the flows tracing agent. Possible
-                      values are:<br> - `eBPF` (default) to use NetObserv eBPF agent.<br>
-                      - `IPFIX` [deprecated (*)] - to use the legacy IPFIX collector.<br>
-                      `eBPF` is recommended as it offers better performances and should
-                      work regardless of the CNI installed on the cluster. `IPFIX`
-                      works with OVN-Kubernetes CNI (other CNIs could work if they
-                      support exporting IPFIX, but they would require manual configuration).'
+                    description: '`type` [deprecated (*)] selects the flows tracing
+                      agent. The only possible value is `eBPF` (default), to use NetObserv
+                      eBPF agent.<br> Previously, using an IPFIX collector was allowed,
+                      but was deprecated and it is now removed.<br> Setting `IPFIX`
+                      is ignored and still use the eBPF Agent. Since there is only
+                      a single option here, this field will be remove in a future
+                      API version.'
                     enum:
                     - eBPF
                     - IPFIX
@@ -4941,9 +4940,8 @@ spec:
                         type: string
                       dropUnusedFields:
                         default: true
-                        description: '`dropUnusedFields` allows, when set to `true`,
-                          to drop fields that are known to be unused by OVS, to save
-                          storage space.'
+                        description: '`dropUnusedFields` [deprecated (*)] this setting
+                          is not used anymore.'
                         type: boolean
                       enableKubeProbes:
                         default: true

diff --git a/bundle/manifests/netobserv-operator.clusterserviceversion.yaml b/bundle/manifests/netobserv-operator.clusterserviceversion.yaml
@@ -773,9 +773,10 @@ spec:
       name: flowmetrics.flows.netobserv.io
       version: v1alpha1
   description: |-
-    NetObserv Operator is an OpenShift / Kubernetes operator for network observability. It deploys a monitoring pipeline to collect and enrich network flows. These flows can be produced by the NetObserv eBPF agent, or by any device or CNI able to export flows in IPFIX format, such as OVN-Kubernetes.
-
-    The operator provides dashboards, metrics, and keeps flows accessible in a queryable log store, Grafana Loki. When used in OpenShift, new views are available in the Console.
+    NetObserv Operator is a OpenShift / Kubernetes operator for network observability. It deploys a monitoring pipeline that consists in:
+    - the NetObserv eBPF agent, that generates network flows from captured packets
+    - Flowlogs-pipeline, a component that collects, enriches and exports these flows.
+    - When used in OpenShift, a Console plugin for flows visualization with powerful filtering options, a topology representation and more.
 
     ## Dependencies
 
@@ -819,9 +820,7 @@ spec:
 
     A couple of settings deserve special attention:
 
-    - Agent (`spec.agent.type`) can be `EBPF` (default) or `IPFIX`. eBPF is recommended, as it should work in more situations and offers better performances. If you can't, or don't want to use eBPF, note that the IPFIX option is fully functional only when using [OVN-Kubernetes](https://github.com/ovn-org/ovn-kubernetes/) CNI. Other CNIs are not officially supported, but you might still be able to configure them manually if they allow IPFIX exports.
-
-    - Sampling (`spec.agent.ebpf.sampling` and `spec.agent.ipfix.sampling`): a value of `100` means: one flow every 100 is sampled. `1` means all flows are sampled. The lower it is, the more flows you get, and the more accurate are derived metrics, but the higher amount of resources are consumed. By default, sampling is set to 50 (ie. 1:50) for eBPF and 400 (1:400) for IPFIX. Note that more sampled flows also means more storage needed. We recommend to start with default values and refine empirically, to figure out which setting your cluster can manage.
+    - Sampling (`spec.agent.ebpf.sampling`): a value of `100` means: one flow every 100 is sampled. `1` means all flows are sampled. The lower it is, the more flows you get, and the more accurate are derived metrics, but the higher amount of resources are consumed. By default, sampling is set to 50 (ie. 1:50). Note that more sampled flows also means more storage needed. We recommend to start with default values and refine empirically, to figure out which setting your cluster can manage.
 
     - Loki (`spec.loki`): configure here how to reach Loki. The default values match the Loki quick install paths mentioned above, but you might have to configure differently if you used another installation method.
 

diff --git a/config/crd/bases/flows.netobserv.io_flowcollectors.yaml b/config/crd/bases/flows.netobserv.io_flowcollectors.yaml
@@ -432,13 +432,13 @@ spec:
                     type: object
                   type:
                     default: EBPF
-                    description: '`type` selects the flows tracing agent. Possible
-                      values are:<br> - `EBPF` (default) to use NetObserv eBPF agent.<br>
-                      - `IPFIX` [deprecated (*)] - to use the legacy IPFIX collector.<br>
-                      `EBPF` is recommended as it offers better performances and should
-                      work regardless of the CNI installed on the cluster. `IPFIX`
-                      works with OVN-Kubernetes CNI (other CNIs could work if they
-                      support exporting IPFIX, but they would require manual configuration).'
+                    description: '`type` [deprecated (*)] selects the flows tracing
+                      agent. The only possible value is `EBPF` (default), to use NetObserv
+                      eBPF agent.<br> Previously, using an IPFIX collector was allowed,
+                      but was deprecated and it is now removed.<br> Setting `IPFIX`
+                      is ignored and still use the eBPF Agent. Since there is only
+                      a single option here, this field will be remove in a future
+                      API version.'
                     enum:
                     - EBPF
                     - IPFIX
@@ -1862,9 +1862,8 @@ spec:
                     type: object
                   dropUnusedFields:
                     default: true
-                    description: '`dropUnusedFields` allows, when set to `true`, to
-                      drop fields that are known to be unused by OVS, to save storage
-                      space.'
+                    description: '`dropUnusedFields` [deprecated (*)] this setting
+                      is not used anymore.'
                     type: boolean
                   enableKubeProbes:
                     default: true
@@ -3236,13 +3235,13 @@ spec:
                     type: object
                   type:
                     default: eBPF
-                    description: '`type` selects the flows tracing agent. Possible
-                      values are:<br> - `eBPF` (default) to use NetObserv eBPF agent.<br>
-                      - `IPFIX` [deprecated (*)] - to use the legacy IPFIX collector.<br>
-                      `eBPF` is recommended as it offers better performances and should
-                      work regardless of the CNI installed on the cluster. `IPFIX`
-                      works with OVN-Kubernetes CNI (other CNIs could work if they
-                      support exporting IPFIX, but they would require manual configuration).'
+                    description: '`type` [deprecated (*)] selects the flows tracing
+                      agent. The only possible value is `eBPF` (default), to use NetObserv
+                      eBPF agent.<br> Previously, using an IPFIX collector was allowed,
+                      but was deprecated and it is now removed.<br> Setting `IPFIX`
+                      is ignored and still use the eBPF Agent. Since there is only
+                      a single option here, this field will be remove in a future
+                      API version.'
                     enum:
                     - eBPF
                     - IPFIX
@@ -4928,9 +4927,8 @@ spec:
                         type: string
                       dropUnusedFields:
                         default: true
-                        description: '`dropUnusedFields` allows, when set to `true`,
-                          to drop fields that are known to be unused by OVS, to save
-                          storage space.'
+                        description: '`dropUnusedFields` [deprecated (*)] this setting
+                          is not used anymore.'
                         type: boolean
                       enableKubeProbes:
                         default: true

diff --git a/config/descriptions/upstream.md b/config/descriptions/upstream.md
@@ -1,6 +1,7 @@
-NetObserv Operator is an OpenShift / Kubernetes operator for network observability. It deploys a monitoring pipeline to collect and enrich network flows. These flows can be produced by the NetObserv eBPF agent, or by any device or CNI able to export flows in IPFIX format, such as OVN-Kubernetes.
-
-The operator provides dashboards, metrics, and keeps flows accessible in a queryable log store, Grafana Loki. When used in OpenShift, new views are available in the Console.
+NetObserv Operator is an OpenShift / Kubernetes operator for network observability. It deploys a monitoring pipeline that consists in:
+- the NetObserv eBPF agent, that generates network flows from captured packets
+- Flowlogs-pipeline, a component that collects, enriches and exports these flows.
+- When used in OpenShift, a Console plugin for flows visualization with powerful filtering options, a topology representation and more.
 
 ## Dependencies
 
@@ -44,9 +45,7 @@ As it operates cluster-wide, only a single `FlowCollector` is allowed, and it ha
 
 A couple of settings deserve special attention:
 
-- Agent (`spec.agent.type`) can be `EBPF` (default) or `IPFIX`. eBPF is recommended, as it should work in more situations and offers better performances. If you can't, or don't want to use eBPF, note that the IPFIX option is fully functional only when using [OVN-Kubernetes](https://github.com/ovn-org/ovn-kubernetes/) CNI. Other CNIs are not officially supported, but you might still be able to configure them manually if they allow IPFIX exports.
-
-- Sampling (`spec.agent.ebpf.sampling` and `spec.agent.ipfix.sampling`): a value of `100` means: one flow every 100 is sampled. `1` means all flows are sampled. The lower it is, the more flows you get, and the more accurate are derived metrics, but the higher amount of resources are consumed. By default, sampling is set to 50 (ie. 1:50) for eBPF and 400 (1:400) for IPFIX. Note that more sampled flows also means more storage needed. We recommend to start with default values and refine empirically, to figure out which setting your cluster can manage.
+- Sampling (`spec.agent.ebpf.sampling`): a value of `100` means: one flow every 100 is sampled. `1` means all flows are sampled. The lower it is, the more flows you get, and the more accurate are derived metrics, but the higher amount of resources are consumed. By default, sampling is set to 50 (ie. 1:50). Note that more sampled flows also means more storage needed. We recommend to start with default values and refine empirically, to figure out which setting your cluster can manage.
 
 - Loki (`spec.loki`): configure here how to reach Loki. The default values match the Loki quick install paths mentioned above, but you might have to configure differently if you used another installation method.