Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: adapt Kafka bindings to v3 #221

Closed
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
194 changes: 194 additions & 0 deletions kafka/README-v2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,194 @@
# Kafka Bindings

This document defines how to describe Kafka-specific information on AsyncAPI v2.x document.

<a name="version"></a>

## Version

Current version is `v2-0.4.0` and is applicable to AsyncAPI v2.x only. For a version of Kafka binding applicable to AsyncAPI 3.x and examples on how to use it, please check this [README](./README-v3.md).

> `v2-0.4.0` is just a renmaing of previous `0.4.0` in order to better track future changes that could be only applicable into an AsyncAPI v2.x context.
lbroudoux marked this conversation as resolved.
Show resolved Hide resolved

<a name="server"></a>

## Server Binding Object

This object contains information about the server representation in Kafka.

##### Fixed Fields

Field Name | Type | Description | Applicability [default] | Constraints
---|:---:|:---:|:---:|---
`schemaRegistryUrl` | string (url) | API URL for the Schema Registry used when producing Kafka messages (if a Schema Registry was used) | OPTIONAL | -
`schemaRegistryVendor` | string | The vendor of Schema Registry and Kafka serdes library that should be used (e.g. `apicurio`, `confluent`, `ibm`, or `karapace`) | OPTIONAL | MUST NOT be specified if `schemaRegistryUrl` is not specified
<a name="serverBindingObjectBindingVersion"></a>`bindingVersion` | string | The version of this binding. | OPTIONAL [`latest`]

##### Example

```yaml
servers:
production:
bindings:
kafka:
schemaRegistryUrl: 'https://my-schema-registry.com'
schemaRegistryVendor: 'confluent'
bindingVersion: 'v2-0.4.0'
```


<a name="channel"></a>

## Channel Binding Object

This object contains information about the channel representation in Kafka (eg. a Kafka topic).

##### Fixed Fields

Field Name | Type | Description | Applicability [default] | Constraints
---|:------------------------------------------------:|:-------------------------------------------------------------------------------------------------------:|:-----------------------:|---
<a name="channelBindingObjectTopic"></a>`topic` | string | Kafka topic name if different from channel name. | OPTIONAL | -
<a name="channelBindingObjectPartitions"></a>`partitions` | integer | Number of partitions configured on this topic (useful to know how many parallel consumers you may run). | OPTIONAL | Must be positive
<a name="channelBindingObjectReplicas"></a>`replicas` | integer | Number of replicas configured on this topic. | OPTIONAL | MUST be positive
<a name="channelBindingObjectTopicConfiguration"></a>`topicConfiguration` | [TopicConfiguration Object](#topicConfiguration) | Topic configuration properties that are relevant for the API. | OPTIONAL | -
<a name="channelBindingObjectBindingVersion"></a>`bindingVersion` | string | The version of this binding. If omitted, "latest" MUST be assumed. | OPTIONAL [`latest`] | -


This object MUST contain only the properties defined above.

##### Example

This example is valid for any Confluent compatible schema registry. Here we describe the implementation using the first 4 bytes in payload to store schema identifier.
lbroudoux marked this conversation as resolved.
Show resolved Hide resolved

```yaml
channels:
user-signedup:
bindings:
kafka:
topic: 'my-specific-topic-name'
partitions: 20
replicas: 3
topicConfiguration:
cleanup.policy: ["delete", "compact"]
retention.ms: 604800000
retention.bytes: 1000000000
delete.retention.ms: 86400000
max.message.bytes: 1048588
bindingVersion: 'v2-0.4.0'
```

<a name="topicConfiguration"></a>
## TopicConfiguration Object

This objects contains information about the API relevant topic configuration in Kafka.

Field Name | Type | Description | Applicability [default] | Constraints
---|:-------:|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:-----------------------:|---
<a name="topicConfigurationCleanupPolicy"></a>`cleanup.policy` | array | The [`cleanup.policy`](https://kafka.apache.org/documentation/#topicconfigs_cleanup.policy) configuration option. | OPTIONAL | array may only contain `delete` and/or `compact`
<a name="topicConfigurationRetentionMs"></a>`retention.ms` | integer | The [`retention.ms`](https://kafka.apache.org/documentation/#topicconfigs_retention.ms) configuration option. | OPTIONAL | see kafka documentation
<a name="topicConfigurationRetentionBytes"></a>`retention.bytes` | integer | The [`retention.bytes`](https://kafka.apache.org/documentation/#topicconfigs_retention.bytes) configuration option. | OPTIONAL | see kafka documentation
<a name="topicConfigurationDeleteRetentionBytes"></a>`delete.retention.ms` | integer | The [`delete.retention.ms`](https://kafka.apache.org/documentation/#topicconfigs_delete.retention.ms) configuration option. | OPTIONAL | see kafka documentation
<a name="topicConfigurationMaxMessageBytes"></a>`max.message.bytes` | integer | The [`max.message.bytes`](https://kafka.apache.org/documentation/#topicconfigs_max.message.bytes) configuration option. | OPTIONAL | see kafka documentation

This object MUST contain only the properties defined above.

##### Example

```yaml
topicConfiguration:
cleanup.policy: ["delete", "compact"]
retention.ms: 604800000
retention.bytes: 1000000000
delete.retention.ms: 86400000
max.message.bytes: 1048588
```

<a name="operation"></a>

## Operation Binding Object

This object contains information about the operation representation in Kafka (eg. the way to consume messages)

##### Fixed Fields

Field Name | Type | Description | Applicability [default] | Constraints
---|:---:|:---:|:---:|---
<a name="operationBindingObjectGroupId"></a>`groupId` | [Schema Object][schemaObject] \| [Reference Object](referenceObject) | Id of the consumer group. | OPTIONAL | -
<a name="operationBindingObjectClientId"></a>`clientId` | [Schema Object][schemaObject] \| [Reference Object](referenceObject) | Id of the consumer inside a consumer group. | OPTIONAL | -
<a name="operationBindingObjectBindingVersion"></a>`bindingVersion` | string | The version of this binding. If omitted, "latest" MUST be assumed. | OPTIONAL [`latest`] | -

This object MUST contain only the properties defined above.

##### Example

```yaml
channels:
user-signedup:
subscribe:
bindings:
kafka:
groupId:
type: string
enum: ['myGroupId']
clientId:
type: string
enum: ['myClientId']
bindingVersion: 'v2-0.4.0'
```


<a name="message"></a>

## Message Binding Object

This object contains information about the message representation in Kafka.

##### Fixed Fields

Field Name | Type | Description
---|:---:|---
<a name="messageBindingObjectKey"></a>`key` | [Schema Object][schemaObject] \| [Reference Object](referenceObject) \| [AVRO Schema Object](https://avro.apache.org/docs/current/spec.html) | The message key.
<a name="messageBindingObjectSchemaIdLocation"></a>`schemaIdLocation` | string | If a Schema Registry is used when performing this operation, tells where the id of schema is stored (e.g. `header` or `payload`). | OPTIONAL | MUST NOT be specified if `schemaRegistryUrl` is not specified at the Server level
<a name="messageBindingObjectSchemaIdPayloadEncoding"></a>`schemaIdPayloadEncoding` | string | Number of bytes or vendor specific values when schema id is encoded in payload (e.g `confluent`/ `apicurio-legacy` / `apicurio-new`). | OPTIONAL | MUST NOT be specified if `schemaRegistryUrl` is not specified at the Server level
<a name="messageBindingObjectSchemaLookupStrategy"></a>`schemaLookupStrategy` | string | Freeform string for any naming strategy class to use. Clients should default to the vendor default if not supplied. | OPTIONAL | MUST NOT be specified if `schemaRegistryUrl` is not specified at the Server level
<a name="messageBindingObjectBindingVersion"></a>`bindingVersion` | string | The version of this binding. If omitted, "latest" MUST be assumed.

This object MUST contain only the properties defined above.

This example is valid for any Confluent compatible schema registry. Here we describe the implementation using the first 4 bytes in payload to store schema identifier.

```yaml
channels:
test:
publish:
message:
bindings:
kafka:
key:
type: string
enum: ['myKey']
schemaIdLocation: 'payload'
schemaIdPayloadEncoding: '4'
bindingVersion: 'v2-0.4.0'
```

This is another example that describes the use if Apicurio schema registry. We describe the `apicurio-new` way of serializing without details on how it's implemented. We reference a [specific lookup strategy](https://www.apicur.io/registry/docs/apicurio-registry/2.2.x/getting-started/assembly-using-kafka-client-serdes.html#registry-serdes-concepts-strategy_registry) that may be used to retrieve schema Id from registry during serialization.

```yaml
channels:
test:
publish:
message:
bindings:
kafka:
key:
type: string
enum: ['myKey']
schemaIdLocation: 'payload'
schemaIdPayloadEncoding: 'apicurio-new'
schemaLookupStrategy: 'TopicIdStrategy'
bindingVersion: 'v2-0.4.0'
```

[schemaObject]: https://www.asyncapi.com/docs/specifications/2.4.0/#schemaObject
[referenceObject]: https://www.asyncapi.com/docs/specifications/2.4.0/#referenceObject
Loading