diff --git a/kafka/README-v2.md b/kafka/README-v2.md new file mode 100644 index 00000000..4115480d --- /dev/null +++ b/kafka/README-v2.md @@ -0,0 +1,193 @@ +# Kafka Bindings + +This document defines how to describe Kafka-specific information on AsyncAPI v2.x document. + + + +## Version + +Current version is `v2-0.4.0` and is applicable to AsyncAPI v2.x only. For a version of Kafka binding applicable to AsyncAPI 3.x and examples on how to use it, please check this [README](./README-v3.md). + +> `v2-0.4.0` is just a renaming of previous `0.4.0` in order to better track future changes that could be only applicable into an AsyncAPI v2.x context. + + + +## Server Binding Object + +This object contains information about the server representation in Kafka. + +##### Fixed Fields + +Field Name | Type | Description | Applicability [default] | Constraints +---|:---:|:---:|:---:|--- +`schemaRegistryUrl` | string (url) | API URL for the Schema Registry used when producing Kafka messages (if a Schema Registry was used) | OPTIONAL | - +`schemaRegistryVendor` | string | The vendor of Schema Registry and Kafka serdes library that should be used (e.g. `apicurio`, `confluent`, `ibm`, or `karapace`) | OPTIONAL | MUST NOT be specified if `schemaRegistryUrl` is not specified +`bindingVersion` | string | The version of this binding. | OPTIONAL [`latest`] + +##### Example + +```yaml +servers: + production: + bindings: + kafka: + schemaRegistryUrl: 'https://my-schema-registry.com' + schemaRegistryVendor: 'confluent' + bindingVersion: 'v2-0.4.0' +``` + + + + +## Channel Binding Object + +This object contains information about the channel representation in Kafka (eg. a Kafka topic). + +##### Fixed Fields + +Field Name | Type | Description | Applicability [default] | Constraints +---|:------------------------------------------------:|:-------------------------------------------------------------------------------------------------------:|:-----------------------:|--- +`topic` | string | Kafka topic name if different from channel name. | OPTIONAL | - +`partitions` | integer | Number of partitions configured on this topic (useful to know how many parallel consumers you may run). | OPTIONAL | Must be positive +`replicas` | integer | Number of replicas configured on this topic. | OPTIONAL | MUST be positive +`topicConfiguration` | [TopicConfiguration Object](#topicConfiguration) | Topic configuration properties that are relevant for the API. | OPTIONAL | - +`bindingVersion` | string | The version of this binding. If omitted, "latest" MUST be assumed. | OPTIONAL [`latest`] | - + + +This object MUST contain only the properties defined above. + +##### Example + + +```yaml +channels: + user-signedup: + bindings: + kafka: + topic: 'my-specific-topic-name' + partitions: 20 + replicas: 3 + topicConfiguration: + cleanup.policy: ["delete", "compact"] + retention.ms: 604800000 + retention.bytes: 1000000000 + delete.retention.ms: 86400000 + max.message.bytes: 1048588 + bindingVersion: 'v2-0.4.0' +``` + + +## TopicConfiguration Object + +This objects contains information about the API relevant topic configuration in Kafka. + +Field Name | Type | Description | Applicability [default] | Constraints +---|:-------:|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:-----------------------:|--- +`cleanup.policy` | array | The [`cleanup.policy`](https://kafka.apache.org/documentation/#topicconfigs_cleanup.policy) configuration option. | OPTIONAL | array may only contain `delete` and/or `compact` +`retention.ms` | integer | The [`retention.ms`](https://kafka.apache.org/documentation/#topicconfigs_retention.ms) configuration option. | OPTIONAL | see kafka documentation +`retention.bytes` | integer | The [`retention.bytes`](https://kafka.apache.org/documentation/#topicconfigs_retention.bytes) configuration option. | OPTIONAL | see kafka documentation +`delete.retention.ms` | integer | The [`delete.retention.ms`](https://kafka.apache.org/documentation/#topicconfigs_delete.retention.ms) configuration option. | OPTIONAL | see kafka documentation +`max.message.bytes` | integer | The [`max.message.bytes`](https://kafka.apache.org/documentation/#topicconfigs_max.message.bytes) configuration option. | OPTIONAL | see kafka documentation + +This object MUST contain only the properties defined above. + +##### Example + +```yaml +topicConfiguration: + cleanup.policy: ["delete", "compact"] + retention.ms: 604800000 + retention.bytes: 1000000000 + delete.retention.ms: 86400000 + max.message.bytes: 1048588 +``` + + + +## Operation Binding Object + +This object contains information about the operation representation in Kafka (eg. the way to consume messages) + +##### Fixed Fields + +Field Name | Type | Description | Applicability [default] | Constraints +---|:---:|:---:|:---:|--- +`groupId` | [Schema Object][schemaObject] \| [Reference Object](referenceObject) | Id of the consumer group. | OPTIONAL | - +`clientId` | [Schema Object][schemaObject] \| [Reference Object](referenceObject) | Id of the consumer inside a consumer group. | OPTIONAL | - +`bindingVersion` | string | The version of this binding. If omitted, "latest" MUST be assumed. | OPTIONAL [`latest`] | - + +This object MUST contain only the properties defined above. + +##### Example + +```yaml +channels: + user-signedup: + subscribe: + bindings: + kafka: + groupId: + type: string + enum: ['myGroupId'] + clientId: + type: string + enum: ['myClientId'] + bindingVersion: 'v2-0.4.0' +``` + + + + +## Message Binding Object + +This object contains information about the message representation in Kafka. + +##### Fixed Fields + +Field Name | Type | Description +---|:---:|--- +`key` | [Schema Object][schemaObject] \| [Reference Object](referenceObject) \| [AVRO Schema Object](https://avro.apache.org/docs/current/spec.html) | The message key. +`schemaIdLocation` | string | If a Schema Registry is used when performing this operation, tells where the id of schema is stored (e.g. `header` or `payload`). | OPTIONAL | MUST NOT be specified if `schemaRegistryUrl` is not specified at the Server level +`schemaIdPayloadEncoding` | string | Number of bytes or vendor specific values when schema id is encoded in payload (e.g `confluent`/ `apicurio-legacy` / `apicurio-new`). | OPTIONAL | MUST NOT be specified if `schemaRegistryUrl` is not specified at the Server level +`schemaLookupStrategy` | string | Freeform string for any naming strategy class to use. Clients should default to the vendor default if not supplied. | OPTIONAL | MUST NOT be specified if `schemaRegistryUrl` is not specified at the Server level +`bindingVersion` | string | The version of this binding. If omitted, "latest" MUST be assumed. + +This object MUST contain only the properties defined above. + +This example is valid for any Confluent compatible schema registry. Here we describe the implementation using the first 4 bytes in payload to store schema identifier. + +```yaml +channels: + test: + publish: + message: + bindings: + kafka: + key: + type: string + enum: ['myKey'] + schemaIdLocation: 'payload' + schemaIdPayloadEncoding: '4' + bindingVersion: 'v2-0.4.0' +``` + +This is another example that describes the use if Apicurio schema registry. We describe the `apicurio-new` way of serializing without details on how it's implemented. We reference a [specific lookup strategy](https://www.apicur.io/registry/docs/apicurio-registry/2.2.x/getting-started/assembly-using-kafka-client-serdes.html#registry-serdes-concepts-strategy_registry) that may be used to retrieve schema Id from registry during serialization. + +```yaml +channels: + test: + publish: + message: + bindings: + kafka: + key: + type: string + enum: ['myKey'] + schemaIdLocation: 'payload' + schemaIdPayloadEncoding: 'apicurio-new' + schemaLookupStrategy: 'TopicIdStrategy' + bindingVersion: 'v2-0.4.0' +``` + +[schemaObject]: https://www.asyncapi.com/docs/specifications/2.4.0/#schemaObject +[referenceObject]: https://www.asyncapi.com/docs/specifications/2.4.0/#referenceObject diff --git a/kafka/README-v3.md b/kafka/README-v3.md new file mode 100644 index 00000000..a8c7a67b --- /dev/null +++ b/kafka/README-v3.md @@ -0,0 +1,196 @@ +# Kafka Bindings + +This document defines how to describe Kafka-specific information on AsyncAPI v3.x document. + + + +## Version + +Current version is `v3-0.4.0` and is applicable to AsyncAPI v3.x only. For a version of Kafka binding applicable to AsyncAPI 2.x and examples on how to use it, please check this [README](./README-v2.md). + +> `v3-0.4.0` is just a renaming of previous `0.4.0` in order to better track future changes that could be only applicable into an AsyncAPI v3.x context. + + + +## Server Binding Object + +This object contains information about the server representation in Kafka. + +##### Fixed Fields + +Field Name | Type | Description | Applicability [default] | Constraints +---|:---:|:---:|:---:|--- +`schemaRegistryUrl` | string (url) | API URL for the Schema Registry used when producing Kafka messages (if a Schema Registry was used) | OPTIONAL | - +`schemaRegistryVendor` | string | The vendor of Schema Registry and Kafka serdes library that should be used (e.g. `apicurio`, `confluent`, `ibm`, or `karapace`) | OPTIONAL | MUST NOT be specified if `schemaRegistryUrl` is not specified +`bindingVersion` | string | The version of this binding. | OPTIONAL [`latest`] + +##### Example + +```yaml +servers: + production: + bindings: + kafka: + schemaRegistryUrl: 'https://my-schema-registry.com' + schemaRegistryVendor: 'confluent' + bindingVersion: 'v3-0.4.0' +``` + + + + +## Channel Binding Object + +This object contains information about the channel representation in Kafka (eg. a Kafka topic). + +##### Fixed Fields + +Field Name | Type | Description | Applicability [default] | Constraints +---|:------------------------------------------------:|:-------------------------------------------------------------------------------------------------------:|:-----------------------:|--- +`topic` | string | Kafka topic name if different from channel name. | OPTIONAL | - +`partitions` | integer | Number of partitions configured on this topic (useful to know how many parallel consumers you may run). | OPTIONAL | Must be positive +`replicas` | integer | Number of replicas configured on this topic. | OPTIONAL | MUST be positive +`topicConfiguration` | [TopicConfiguration Object](#topicConfiguration) | Topic configuration properties that are relevant for the API. | OPTIONAL | - +`bindingVersion` | string | The version of this binding. If omitted, "latest" MUST be assumed. | OPTIONAL [`latest`] | - + + +This object MUST contain only the properties defined above. + +##### Example + + +```yaml +channels: + user-signedup: + bindings: + kafka: + topic: 'my-specific-topic-name' + partitions: 20 + replicas: 3 + topicConfiguration: + cleanup.policy: ["delete", "compact"] + retention.ms: 604800000 + retention.bytes: 1000000000 + delete.retention.ms: 86400000 + max.message.bytes: 1048588 + bindingVersion: 'v3-0.4.0' +``` + + +## TopicConfiguration Object + +This objects contains information about the API relevant topic configuration in Kafka. + +Field Name | Type | Description | Applicability [default] | Constraints +---|:-------:|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:-----------------------:|--- +`cleanup.policy` | array | The [`cleanup.policy`](https://kafka.apache.org/documentation/#topicconfigs_cleanup.policy) configuration option. | OPTIONAL | array may only contain `delete` and/or `compact` +`retention.ms` | integer | The [`retention.ms`](https://kafka.apache.org/documentation/#topicconfigs_retention.ms) configuration option. | OPTIONAL | see kafka documentation +`retention.bytes` | integer | The [`retention.bytes`](https://kafka.apache.org/documentation/#topicconfigs_retention.bytes) configuration option. | OPTIONAL | see kafka documentation +`delete.retention.ms` | integer | The [`delete.retention.ms`](https://kafka.apache.org/documentation/#topicconfigs_delete.retention.ms) configuration option. | OPTIONAL | see kafka documentation +`max.message.bytes` | integer | The [`max.message.bytes`](https://kafka.apache.org/documentation/#topicconfigs_max.message.bytes) configuration option. | OPTIONAL | see kafka documentation + +This object MUST contain only the properties defined above. + +##### Example + +```yaml +topicConfiguration: + cleanup.policy: ["delete", "compact"] + retention.ms: 604800000 + retention.bytes: 1000000000 + delete.retention.ms: 86400000 + max.message.bytes: 1048588 +``` + + + +## Operation Binding Object + +This object contains information about the operation representation in Kafka (eg. the way to consume messages) + +##### Fixed Fields + +Field Name | Type | Description | Applicability [default] | Constraints +---|:---:|:---:|:---:|--- +`groupId` | [Schema Object][schemaObject] \| [Reference Object](referenceObject) | Id of the consumer group. | OPTIONAL | - +`clientId` | [Schema Object][schemaObject] \| [Reference Object](referenceObject) | Id of the consumer inside a consumer group. | OPTIONAL | - +`bindingVersion` | string | The version of this binding. If omitted, "latest" MUST be assumed. | OPTIONAL [`latest`] | - + +This object MUST contain only the properties defined above. + +##### Example + +```yaml +channels: + user-signedup: +operations: + userSignup: + action: receive + bindings: + kafka: + groupId: + type: string + enum: ['myGroupId'] + clientId: + type: string + enum: ['myClientId'] + bindingVersion: 'v3-0.4.0' +``` + + + +## Message Binding Object + +This object contains information about the message representation in Kafka. + +##### Fixed Fields + +Field Name | Type | Description +---|:---:|--- +`key` | [Schema Object][schemaObject] \| [Reference Object](referenceObject) \| [AVRO Schema Object](https://avro.apache.org/docs/current/spec.html) | The message key. +`schemaIdLocation` | string | If a Schema Registry is used when performing this operation, tells where the id of schema is stored (e.g. `header` or `payload`). | OPTIONAL | MUST NOT be specified if `schemaRegistryUrl` is not specified at the Server level +`schemaIdPayloadEncoding` | string | Number of bytes or vendor specific values when schema id is encoded in payload (e.g `confluent`/ `apicurio-legacy` / `apicurio-new`). | OPTIONAL | MUST NOT be specified if `schemaRegistryUrl` is not specified at the Server level +`schemaLookupStrategy` | string | Freeform string for any naming strategy class to use. Clients should default to the vendor default if not supplied. | OPTIONAL | MUST NOT be specified if `schemaRegistryUrl` is not specified at the Server level +`bindingVersion` | string | The version of this binding. If omitted, "latest" MUST be assumed. + +This object MUST contain only the properties defined above. + +This example is valid for any Confluent compatible schema registry. Here we describe the implementation using the first 4 bytes in payload to store schema identifier. + +```yaml +channels: + test: + address: test-topic + messages: + testMessage: + bindings: + kafka: + key: + type: string + enum: ['myKey'] + schemaIdLocation: 'payload' + schemaIdPayloadEncoding: '4' + bindingVersion: 'v3-0.4.0' +``` + +This is another example that describes the use if Apicurio schema registry. We describe the `apicurio-new` way of serializing without details on how it's implemented. We reference a [specific lookup strategy](https://www.apicur.io/registry/docs/apicurio-registry/2.2.x/getting-started/assembly-using-kafka-client-serdes.html#registry-serdes-concepts-strategy_registry) that may be used to retrieve schema Id from registry during serialization. + +```yaml +channels: + test: + address: test-topic + messages: + testMessage: + bindings: + kafka: + key: + type: string + enum: ['myKey'] + schemaIdLocation: 'payload' + schemaIdPayloadEncoding: 'apicurio-new' + schemaLookupStrategy: 'TopicIdStrategy' + bindingVersion: 'v3-0.4.0' +``` + +[schemaObject]: https://www.asyncapi.com/docs/reference/specification/v3.0.0-next-major-spec.15#schemaObject +[referenceObject]: https://www.asyncapi.com/docs/reference/specification/v3.0.0-next-major-spec.15#referenceObject diff --git a/kafka/README.md b/kafka/README.md index b9c153df..62d0a7ff 100644 --- a/kafka/README.md +++ b/kafka/README.md @@ -2,195 +2,12 @@ This document defines how to describe Kafka-specific information on AsyncAPI. - + -## Version +## Versions -Current version is `0.4.0`. +We differentiate bindings versions for AsyncAPI v2.x and AsyncAPI v3.x specification documents as the way to introduce bindings in those different versions of the spec is slightly different. +Current version for AsyncAPI v2.x documents is `v2-0.4.0` and details can found [here](./README-v2.md). - - -## Server Binding Object - -This object contains information about the server representation in Kafka. - -##### Fixed Fields - -Field Name | Type | Description | Applicability [default] | Constraints ----|:---:|:---:|:---:|--- -`schemaRegistryUrl` | string (url) | API URL for the Schema Registry used when producing Kafka messages (if a Schema Registry was used) | OPTIONAL | - -`schemaRegistryVendor` | string | The vendor of Schema Registry and Kafka serdes library that should be used (e.g. `apicurio`, `confluent`, `ibm`, or `karapace`) | OPTIONAL | MUST NOT be specified if `schemaRegistryUrl` is not specified -`bindingVersion` | string | The version of this binding. | OPTIONAL [`latest`] - -##### Example - -```yaml -servers: - production: - bindings: - kafka: - schemaRegistryUrl: 'https://my-schema-registry.com' - schemaRegistryVendor: 'confluent' - bindingVersion: '0.4.0' -``` - - - - -## Channel Binding Object - -This object contains information about the channel representation in Kafka (eg. a Kafka topic). - -##### Fixed Fields - -Field Name | Type | Description | Applicability [default] | Constraints ----|:------------------------------------------------:|:-------------------------------------------------------------------------------------------------------:|:-----------------------:|--- -`topic` | string | Kafka topic name if different from channel name. | OPTIONAL | - -`partitions` | integer | Number of partitions configured on this topic (useful to know how many parallel consumers you may run). | OPTIONAL | Must be positive -`replicas` | integer | Number of replicas configured on this topic. | OPTIONAL | MUST be positive -`topicConfiguration` | [TopicConfiguration Object](#topicConfiguration) | Topic configuration properties that are relevant for the API. | OPTIONAL | - -`bindingVersion` | string | The version of this binding. If omitted, "latest" MUST be assumed. | OPTIONAL [`latest`] | - - - -This object MUST contain only the properties defined above. - -##### Example - -This example is valid for any Confluent compatible schema registry. Here we describe the implementation using the first 4 bytes in payload to store schema identifier. - -```yaml -channels: - user-signedup: - bindings: - kafka: - topic: 'my-specific-topic-name' - partitions: 20 - replicas: 3 - topicConfiguration: - cleanup.policy: ["delete", "compact"] - retention.ms: 604800000 - retention.bytes: 1000000000 - delete.retention.ms: 86400000 - max.message.bytes: 1048588 - bindingVersion: '0.4.0' -``` - -## TopicConfiguration Object - -This objects contains information about the API relevant topic configuration in Kafka. - -Field Name | Type | Description | Applicability [default] | Constraints ----|:-------:|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:-----------------------:|--- -`cleanup.policy` | array | The [`cleanup.policy`](https://kafka.apache.org/documentation/#topicconfigs_cleanup.policy) configuration option. | OPTIONAL | array may only contain `delete` and/or `compact` -`retention.ms` | integer | The [`retention.ms`](https://kafka.apache.org/documentation/#topicconfigs_retention.ms) configuration option. | OPTIONAL | see kafka documentation -`retention.bytes` | integer | The [`retention.bytes`](https://kafka.apache.org/documentation/#topicconfigs_retention.bytes) configuration option. | OPTIONAL | see kafka documentation -`delete.retention.ms` | integer | The [`delete.retention.ms`](https://kafka.apache.org/documentation/#topicconfigs_delete.retention.ms) configuration option. | OPTIONAL | see kafka documentation -`max.message.bytes` | integer | The [`max.message.bytes`](https://kafka.apache.org/documentation/#topicconfigs_max.message.bytes) configuration option. | OPTIONAL | see kafka documentation - -This object MUST contain only the properties defined above. - -##### Example - -```yaml -topicConfiguration: - cleanup.policy: ["delete", "compact"] - retention.ms: 604800000 - retention.bytes: 1000000000 - delete.retention.ms: 86400000 - max.message.bytes: 1048588 -``` - - - -## Operation Binding Object - -This object contains information about the operation representation in Kafka (eg. the way to consume messages) - -##### Fixed Fields - -Field Name | Type | Description | Applicability [default] | Constraints ----|:---:|:---:|:---:|--- -`groupId` | [Schema Object][schemaObject] \| [Reference Object](referenceObject) | Id of the consumer group. | OPTIONAL | - -`clientId` | [Schema Object][schemaObject] \| [Reference Object](referenceObject) | Id of the consumer inside a consumer group. | OPTIONAL | - -`bindingVersion` | string | The version of this binding. If omitted, "latest" MUST be assumed. | OPTIONAL [`latest`] | - - -This object MUST contain only the properties defined above. - -##### Example - -```yaml -channels: - user-signedup: - subscribe: - bindings: - kafka: - groupId: - type: string - enum: ['myGroupId'] - clientId: - type: string - enum: ['myClientId'] - bindingVersion: '0.4.0' -``` - - - - -## Message Binding Object - -This object contains information about the message representation in Kafka. - -##### Fixed Fields - -Field Name | Type | Description ----|:---:|--- -<<<<<<< HEAD -`key` | [Schema Object][schemaObject] \| [Reference Object](referenceObject) \| [AVRO Schema Object](https://avro.apache.org/docs/current/spec.html) | The message key. -======= -`key` | [Schema Object][schemaObject] \| [AVRO Schema Object](https://avro.apache.org/docs/current/spec.html) | The message key. **NOTE**: You can also use the [reference object](https://asyncapi.io/docs/specifications/v2.4.0#referenceObject) way. -`schemaIdLocation` | string | If a Schema Registry is used when performing this operation, tells where the id of schema is stored (e.g. `header` or `payload`). | OPTIONAL | MUST NOT be specified if `schemaRegistryUrl` is not specified at the Server level -`schemaIdPayloadEncoding` | string | Number of bytes or vendor specific values when schema id is encoded in payload (e.g `confluent`/ `apicurio-legacy` / `apicurio-new`). | OPTIONAL | MUST NOT be specified if `schemaRegistryUrl` is not specified at the Server level -`schemaLookupStrategy` | string | Freeform string for any naming strategy class to use. Clients should default to the vendor default if not supplied. | OPTIONAL | MUST NOT be specified if `schemaRegistryUrl` is not specified at the Server level ->>>>>>> master -`bindingVersion` | string | The version of this binding. If omitted, "latest" MUST be assumed. - -This object MUST contain only the properties defined above. - -This example is valid for any Confluent compatible schema registry. Here we describe the implementation using the first 4 bytes in payload to store schema identifier. - -```yaml -channels: - test: - publish: - message: - bindings: - kafka: - key: - type: string - enum: ['myKey'] - schemaIdLocation: 'payload' - schemaIdPayloadEncoding: '4' - bindingVersion: '0.4.0' -``` - -This is another example that describes the use if Apicurio schema registry. We describe the `apicurio-new` way of serializing without details on how it's implemented. We reference a [specific lookup strategy](https://www.apicur.io/registry/docs/apicurio-registry/2.2.x/getting-started/assembly-using-kafka-client-serdes.html#registry-serdes-concepts-strategy_registry) that may be used to retrieve schema Id from registry during serialization. - -```yaml -channels: - test: - publish: - message: - bindings: - kafka: - key: - type: string - enum: ['myKey'] - schemaIdLocation: 'payload' - schemaIdPayloadEncoding: 'apicurio-new' - schemaLookupStrategy: 'TopicIdStrategy' - bindingVersion: '0.4.0' -``` - -[schemaObject]: https://www.asyncapi.com/docs/specifications/2.4.0/#schemaObject -[referenceObject]: https://www.asyncapi.com/docs/specifications/2.4.0/#referenceObject +Current version for AsyncAPI v3.x documents is `v3-0.4.0` and details can found [here](./README-v2.md).