Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added kafka features config map to kafka broker docs #5622

Merged
merged 12 commits into from
Oct 5, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion config/nav.yml
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,9 @@ nav:
- Available Broker types: eventing/brokers/broker-types/README.md
# add default IMC broker page, page explaining broker types
- Channel based Broker: eventing/brokers/broker-types/channel-based-broker/README.md
- Apache Kafka: eventing/brokers/broker-types/kafka-broker/README.md
- Apache Kafka:
- About Apache Kafka Broker: eventing/brokers/broker-types/kafka-broker/README.md
- Configuring Kafka features: eventing/brokers/broker-types/kafka-broker/configuring-kafka-features.md
Comment on lines +194 to +196
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I build these changes locally, it seemed to still work fine

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for double checking

- RabbitMQ Broker: eventing/brokers/broker-types/rabbitmq-broker/README.md
- Creating a Broker: eventing/brokers/create-broker.md
- Developer configuration options: eventing/brokers/broker-developer-config-options.md
Expand Down
4 changes: 2 additions & 2 deletions docs/eventing/brokers/broker-types/kafka-broker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Notable features are:

- Control plane High Availability
- Horizontally scalable data plane
- [Extensively configurable](#kafka-producer-and-consumer-configurations)
- [Extensively configurable](./configuring-kafka-features)
- Ordered delivery of events based on [CloudEvents partitioning extension](https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/extensions/partitioning.md)
- Support any Kafka version, see [compatibility matrix](https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix)
- Supports 2 [data plane modes](#data-plane-isolation-vs-shared-data-plane): data plane isolation per-namespace or shared data plane
Expand Down Expand Up @@ -440,7 +440,7 @@ Upon the creation of the first `Broker` with `KafkaNamespaced` class, the `kafka

All the configuration mechanisms that are available for the `Kafka` Broker class are also available for the brokers with `KafkaNamespaced` class with these exceptions:

* [Above](#kafka-producer-and-consumer-configurations) it is described how producer and consumer configurations is done by modifying the `config-kafka-broker-data-plane` configmap in the `knative-eventing` namespace. Since Kafka Broker controller propagates this configmap into the user namespace, currently there is no way to configure producer and consumer configurations per namespace. Any value set in the `config-kafka-broker-data-plane` `ConfigMap` in the `knative-eventing` namespace will be also used in the user namespace.
* [This page](./configuring-kafka-features) describes how producer and consumer configurations is done by modifying the `config-kafka-broker-data-plane` configmap in the `knative-eventing` namespace. Since Kafka Broker controller propagates this configmap into the user namespace, currently there is no way to configure producer and consumer configurations per namespace. Any value set in the `config-kafka-broker-data-plane` `ConfigMap` in the `knative-eventing` namespace will be also used in the user namespace.
* Because of the same propagation, it is also not possible to configure consumer offsets commit interval per namespace.
* A few more configmaps are propagated: `config-tracing` and `kafka-config-logging`. This means, tracing and logging are also not configurable per namespace.
* Similarly, the data plane deployments are propagated from the `knative-eventing` namespace to the user namespace. This means that the data plane deployments are not configurable per namespace and will be identical to the ones in the `knative-eventing` namespace.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# Configuring Kafka Features

There are many different configuration options for how Knative Eventing and the Knaitve Broker for Apache Kafka interact with the Apache Kafka clusters.

## Configure Knative Eventing Kafka features

There are various kafka features/default values the Knative Kafka Broker uses when interacting with Kafka.

### Consumer Group ID for Triggers

The `triggers.consumergroup.template` value determines the template used to generate the consumer group ID used by your triggers.

* **Global key:** `triggers.consumergroup.template`
* **Possible values:**: Any valid [go text/template](https://pkg.go.dev/text/template)
* **Default:** `{% raw %}knative-trigger-{{ .Namespace }}-{{ .Name }}{% endraw %}`

**Example:**

=== "Global (ConfigMap)"
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: config-kafka-features
namespace: knative-eventing
data:
triggers.consumergroup.template: {% raw %}"knative-trigger-{{ .Namespace }}-{{ .Name }}"{% endraw %}
```

### Broker topic name template

The `brokers.topic.template` values determines the template used to generate the Kafka topic names used by your brokers.

* **Global Key:** `brokers.topic.template`
* **Possible values:** Any valid [go text/template](https://pkg.go.dev/text/template)
* **Default:** `{% raw %}knative-broker-{{ .Namespace }}-{{ .Name }}{% endraw %}`

**Example:**

=== "Global (ConfigMap)"
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: config-kafka-features
namespace: knative-eventing
data:
brokers.topic.template: {% raw %}"knative-broker-{{ .Namespace }}-{{ .Name }}"{% endraw %}
```

## Channel topic name template

The `channels.topic.template` value determines the template used to generate the kafka topic names used by your channels.

* **Global Key:** `channels.topic.template`
* **Possible values:** Any valid [go text/template](https://pkg.go.dev/text/template)
* **Default:** `{% raw %}messaging-kafka.{{ .Namespace }}.{{ .Name }}{% endraw %}`

**Example:**

=== "Global (ConfigMap)"
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: config-kafka-features
namespace: knative-eventing
data:
channels.topic.template: {% raw %}"messaging-kafka.{{ .Namespace }}.{{ .Name }}"{% endraw %}
```
Loading