From 64eb98fb598aa0dcfea4a7f7f693097c8212179b Mon Sep 17 00:00:00 2001 From: Cynthia Peter Date: Wed, 25 Oct 2023 16:29:46 +0200 Subject: [PATCH 01/16] Created an outline for message-validation-for-kafka --- .../concepts/asyncapi-document/_section.md | 0 .../docs/concepts/asyncapi-document/index.md | 6 +++ .../tutorials/message-validation-for-kafka.md | 47 +++++++++++++++++++ 3 files changed, 53 insertions(+) create mode 100644 pages/docs/concepts/asyncapi-document/_section.md create mode 100644 pages/docs/concepts/asyncapi-document/index.md create mode 100644 pages/docs/tutorials/message-validation-for-kafka.md diff --git a/pages/docs/concepts/asyncapi-document/_section.md b/pages/docs/concepts/asyncapi-document/_section.md new file mode 100644 index 00000000000..e69de29bb2d diff --git a/pages/docs/concepts/asyncapi-document/index.md b/pages/docs/concepts/asyncapi-document/index.md new file mode 100644 index 00000000000..850162604b8 --- /dev/null +++ b/pages/docs/concepts/asyncapi-document/index.md @@ -0,0 +1,6 @@ +--- +title: 'AsyncAPI document' +weight: 10 +--- + +## Concepts: Define AsyncAPI features and capabilities diff --git a/pages/docs/tutorials/message-validation-for-kafka.md b/pages/docs/tutorials/message-validation-for-kafka.md new file mode 100644 index 00000000000..fccdb98c237 --- /dev/null +++ b/pages/docs/tutorials/message-validation-for-kafka.md @@ -0,0 +1,47 @@ +--- +title: Message validation for Kafka +description: The goal of this tutorial is to help you validate messages for Kafka. +weight: 100 +--- + +# Message Validation for Kafka + +## Introduction + +[comment:] <> (What do we learn, and at the end, what do we accomplish? ) + +Validating messages with Kafka involves defining the message schemas using AsyncAPI, and then using those schemas to validate incoming and outgoing messages in your Kafka-based applications. + +In this tutorial, you'll learn how to validate messages for Kafka. In addition, you'll learn more about EDAs, and the role of Kafka in validating messages. + +By the end of this tutorial, you will: +* +* +* + +## Background context + Define a message. + What is Kafka? + Why do people use Kafka with AsyncAPI? + +### Prerequisites + +[comment:] <> (What does the reader need to accomplish this task?) + + +## Installation / Validation + +### Validating at the Broker level + +### Validating at the client level + + +### Summary + +[comment:] <> (Here is what we learned and did.) + +By following these steps, you have defined and validated Kafka messages using AsyncAPI, therefore ensuring that the messages conform to the specified schema before processing or sending them, which helps maintain data integrity and consistency in your Kafka-based applications. + +## Next steps + +## Additional resources \ No newline at end of file From 71becada7ca7e94b250fcef1a9935ea590712018 Mon Sep 17 00:00:00 2001 From: Cynthia Peter Date: Wed, 25 Oct 2023 17:06:38 +0200 Subject: [PATCH 02/16] implemented feedback from first review --- pages/docs/tutorials/message-validation-for-kafka.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/pages/docs/tutorials/message-validation-for-kafka.md b/pages/docs/tutorials/message-validation-for-kafka.md index fccdb98c237..603df2af4cc 100644 --- a/pages/docs/tutorials/message-validation-for-kafka.md +++ b/pages/docs/tutorials/message-validation-for-kafka.md @@ -4,7 +4,7 @@ description: The goal of this tutorial is to help you validate messages for Kafk weight: 100 --- -# Message Validation for Kafka +[Comment] <> (Add an image to show a process - flowchart) ## Introduction @@ -24,19 +24,17 @@ By the end of this tutorial, you will: What is Kafka? Why do people use Kafka with AsyncAPI? -### Prerequisites - [comment:] <> (What does the reader need to accomplish this task?) -## Installation / Validation +## Installation Guide ### Validating at the Broker level ### Validating at the client level -### Summary +## Summary [comment:] <> (Here is what we learned and did.) From b390551bb13c62d0c8f9e9e9c80e74b6bd4bcb2e Mon Sep 17 00:00:00 2001 From: Cynthia Peter Date: Wed, 25 Oct 2023 17:51:16 +0200 Subject: [PATCH 03/16] implemented feedback from review --- pages/docs/tutorials/message-validation-for-kafka.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/pages/docs/tutorials/message-validation-for-kafka.md b/pages/docs/tutorials/message-validation-for-kafka.md index 603df2af4cc..d19b46781d3 100644 --- a/pages/docs/tutorials/message-validation-for-kafka.md +++ b/pages/docs/tutorials/message-validation-for-kafka.md @@ -4,11 +4,11 @@ description: The goal of this tutorial is to help you validate messages for Kafk weight: 100 --- -[Comment] <> (Add an image to show a process - flowchart) +[Comment] <> (Add an image to show a process - flowchart that summarizes the tutorial) ## Introduction -[comment:] <> (What do we learn, and at the end, what do we accomplish? ) +[comment:] <> (What they're learning || What we're building, explain starting point and where we end) Validating messages with Kafka involves defining the message schemas using AsyncAPI, and then using those schemas to validate incoming and outgoing messages in your Kafka-based applications. @@ -29,14 +29,14 @@ By the end of this tutorial, you will: ## Installation Guide -### Validating at the Broker level +## Broker-Level Message Validation -### Validating at the client level +## Client-Level Message Validation ## Summary -[comment:] <> (Here is what we learned and did.) +[comment:] <> ( What did we learn, what we built, and how to apply this to real life.) By following these steps, you have defined and validated Kafka messages using AsyncAPI, therefore ensuring that the messages conform to the specified schema before processing or sending them, which helps maintain data integrity and consistency in your Kafka-based applications. From 2dbcc34e46f29df49518a84cf3f9e29f2dea6e2e Mon Sep 17 00:00:00 2001 From: Cynthia Peter Date: Wed, 25 Oct 2023 18:30:43 +0200 Subject: [PATCH 04/16] Removed bullet list --- pages/docs/tutorials/message-validation-for-kafka.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/pages/docs/tutorials/message-validation-for-kafka.md b/pages/docs/tutorials/message-validation-for-kafka.md index d19b46781d3..c5ca98bbba9 100644 --- a/pages/docs/tutorials/message-validation-for-kafka.md +++ b/pages/docs/tutorials/message-validation-for-kafka.md @@ -14,10 +14,8 @@ Validating messages with Kafka involves defining the message schemas using Async In this tutorial, you'll learn how to validate messages for Kafka. In addition, you'll learn more about EDAs, and the role of Kafka in validating messages. -By the end of this tutorial, you will: -* -* -* +By the end of this tutorial, you will **** + ## Background context Define a message. From c3f9cbba6b54ccf59cb217d42a689a50fa27512d Mon Sep 17 00:00:00 2001 From: Cynthia Peter Date: Wed, 25 Oct 2023 19:24:12 +0200 Subject: [PATCH 05/16] fixes --- pages/docs/tutorials/message-validation-for-kafka.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/docs/tutorials/message-validation-for-kafka.md b/pages/docs/tutorials/message-validation-for-kafka.md index c5ca98bbba9..853e7648b30 100644 --- a/pages/docs/tutorials/message-validation-for-kafka.md +++ b/pages/docs/tutorials/message-validation-for-kafka.md @@ -4,7 +4,6 @@ description: The goal of this tutorial is to help you validate messages for Kafk weight: 100 --- -[Comment] <> (Add an image to show a process - flowchart that summarizes the tutorial) ## Introduction @@ -16,6 +15,7 @@ In this tutorial, you'll learn how to validate messages for Kafka. In addition, By the end of this tutorial, you will **** +[Comment] <> (Add an image to show a process - flowchart that summarizes the tutorial) ## Background context Define a message. From 375d5fa8060b77a2bf4bb4bd00597af69f8a6210 Mon Sep 17 00:00:00 2001 From: Cynthia Peter Date: Sun, 29 Oct 2023 15:59:39 +0100 Subject: [PATCH 06/16] added some things I am curious about covering --- pages/docs/tutorials/message-validation-for-kafka.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/pages/docs/tutorials/message-validation-for-kafka.md b/pages/docs/tutorials/message-validation-for-kafka.md index 853e7648b30..34c37cfdcde 100644 --- a/pages/docs/tutorials/message-validation-for-kafka.md +++ b/pages/docs/tutorials/message-validation-for-kafka.md @@ -11,9 +11,9 @@ weight: 100 Validating messages with Kafka involves defining the message schemas using AsyncAPI, and then using those schemas to validate incoming and outgoing messages in your Kafka-based applications. -In this tutorial, you'll learn how to validate messages for Kafka. In addition, you'll learn more about EDAs, and the role of Kafka in validating messages. +In this tutorial, you'll learn how to validate messages for Kafka. + -By the end of this tutorial, you will **** [Comment] <> (Add an image to show a process - flowchart that summarizes the tutorial) @@ -29,8 +29,14 @@ By the end of this tutorial, you will **** ## Broker-Level Message Validation +Integrating message validation with Kafka producers +Validating outgoing messages before sending to Kafka + + ## Client-Level Message Validation +Integrating message validation with Kafka consumers +Validating incoming messages before processing ## Summary From 1380ba9cc6e6790e060574ee8d60f9e2d34e28a0 Mon Sep 17 00:00:00 2001 From: Cynthia Peter Date: Wed, 1 Nov 2023 10:51:05 +0100 Subject: [PATCH 07/16] Updated docs --- .../tutorials/message-validation-for-kafka.md | 63 +++++++++++++++---- 1 file changed, 52 insertions(+), 11 deletions(-) diff --git a/pages/docs/tutorials/message-validation-for-kafka.md b/pages/docs/tutorials/message-validation-for-kafka.md index 34c37cfdcde..ccab37fc99a 100644 --- a/pages/docs/tutorials/message-validation-for-kafka.md +++ b/pages/docs/tutorials/message-validation-for-kafka.md @@ -4,28 +4,62 @@ description: The goal of this tutorial is to help you validate messages for Kafk weight: 100 --- +This tutorial will teach you how to validate messages within a Kafka-based system using AsyncAPI specifications. -## Introduction +Validating messages for Kafka involves defining message schemas using AsyncAPI, and then using those schemas to validate incoming and outgoing messages in your Kafka-based applications. -[comment:] <> (What they're learning || What we're building, explain starting point and where we end) +By the end of the tutorial, you should understand the importance of message validation, know how to specify message schemas using AsyncAPI, and apply validation to ensure that the messages exchanged between different components in a Kafka-based system are consistent and adhere to predefined rules. -Validating messages with Kafka involves defining the message schemas using AsyncAPI, and then using those schemas to validate incoming and outgoing messages in your Kafka-based applications. -In this tutorial, you'll learn how to validate messages for Kafka. +[Comment] <> (Add an image to show a process - flowchart that summarizes the tutorial) +graph TB + A[Producer Service] -->|Publish Message| B[AsyncAPI Validation] + B -->|Valid Message| C[Kafka Broker] + C -->|Consume Message| D[AsyncAPI Validation] + D -->|Valid Message| E[Consumer Service] + subgraph "Message Validation with AsyncAPI" + B[Validate Message
against AsyncAPI Schema] + D[Validate Message
against AsyncAPI Schema] + end + + subgraph "Kafka Messaging System" + C[Kafka Broker

Topic: userSignUp] + end + + style A fill:#f9d,stroke:#333,stroke-width:2px + style E fill:#f9d,stroke:#333,stroke-width:2px + style B fill:#9cf,stroke:#333,stroke-width:2px + style D fill:#9cf,stroke:#333,stroke-width:2px + style C fill:#ccf,stroke:#333,stroke-width:2px + + A -->|Publish Invalid Message| B + B -->|Invalid Message| F[Error Handling] + D -->|Invalid Message| G[Error Handling] -[Comment] <> (Add an image to show a process - flowchart that summarizes the tutorial) ## Background context - Define a message. - What is Kafka? - Why do people use Kafka with AsyncAPI? -[comment:] <> (What does the reader need to accomplish this task?) +Message validation involves checking that the messages exchanged between systems adhere to a predefined structure or schema. It helps in ensuring that the data being sent or received is accurate, complete, and in the expected format. + +Kafka transports messages between producers and consumers, while AsyncAPI provides a contract (schema) that those messages must adhere to. Validation tools can use the AsyncAPI specifications to check whether the messages meet the criteria before they are sent or processed. + +[Kafka](https://kafka.apache.org/documentation/#intro_nutshell) is a distributed event streaming platform for building real-time data pipelines and streaming apps. It's horizontally scalable, fault-tolerant, and swift. + +[AsyncAPI](https://www.asyncapi.com/docs) is an open-source initiative that provides a specification for defining and documenting synchronous and asynchronous APIs. It acts as a contract that ensures all system components agree on the structure of messages for effective communication. + + +To follow this tutorial, you need: + +* A knowlegde of Kafka and AsyncAPI. +* An understanding of Event driven Architectures (EDAs). ## Installation Guide + +[Comment] - What goes in the installation guide? + ## Broker-Level Message Validation @@ -42,8 +76,15 @@ Validating incoming messages before processing [comment:] <> ( What did we learn, what we built, and how to apply this to real life.) -By following these steps, you have defined and validated Kafka messages using AsyncAPI, therefore ensuring that the messages conform to the specified schema before processing or sending them, which helps maintain data integrity and consistency in your Kafka-based applications. + +By combining Kafka's robust message streaming capabilities with AsyncAPI's standardized documentation and tooling, developers can ensure that the messages being exchanged in a distributed system are validated effectively, ensuring data integrity and system reliability. + +By following these steps, you have defined and validated Kafka messages using AsyncAPI at both the broker and client levels, therefore ensuring that the messages conform to the specified schema before processing or sending them, which helps maintain data integrity, reliability, and consistency in your Kafka-based applications. ## Next steps -## Additional resources \ No newline at end of file +Now that you have completed this tutorial, go ahead to automate the validation and generation process in your continuous integration pipelines. + +## Additional resources + +[Create an AsyncAPI document for Kafka Messages using AsyncAPI]() From 2c830fea838cf2b574dd88103cb30ebed559f988 Mon Sep 17 00:00:00 2001 From: Cynthia Peter Date: Wed, 1 Nov 2023 10:57:31 +0100 Subject: [PATCH 08/16] updated docs based off the feedback and added a sketch diagram --- pages/docs/tutorials/message-validation-for-kafka.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/pages/docs/tutorials/message-validation-for-kafka.md b/pages/docs/tutorials/message-validation-for-kafka.md index ccab37fc99a..7830fbf57c0 100644 --- a/pages/docs/tutorials/message-validation-for-kafka.md +++ b/pages/docs/tutorials/message-validation-for-kafka.md @@ -63,14 +63,24 @@ To follow this tutorial, you need: ## Broker-Level Message Validation + +At the broker level, message validation is about ensuring that the messages sent to Kafka topics comply with a certain predefined schema. In Kafka, you can use a Schema Registry to manage and validate schemas. Also, at the broker level, ensure that only valid messages are written to the Kafka topics. + +Define an AsyncAPI schema +Configure Kafka ## Client-Level Message Validation - + + +Generate Code +Implement Validation ## Summary From e9a22415a0e86cd5895d18f6f13c1b59793a0c5d Mon Sep 17 00:00:00 2001 From: Cynthia Peter Date: Wed, 1 Nov 2023 16:30:40 +0100 Subject: [PATCH 09/16] updated image --- .../docs/tutorials/message-validation-for-kafka.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/pages/docs/tutorials/message-validation-for-kafka.md b/pages/docs/tutorials/message-validation-for-kafka.md index 7830fbf57c0..e1ba4779d89 100644 --- a/pages/docs/tutorials/message-validation-for-kafka.md +++ b/pages/docs/tutorials/message-validation-for-kafka.md @@ -1,7 +1,7 @@ --- title: Message validation for Kafka description: The goal of this tutorial is to help you validate messages for Kafka. -weight: 100 +weight: 150 --- This tutorial will teach you how to validate messages within a Kafka-based system using AsyncAPI specifications. @@ -11,8 +11,7 @@ Validating messages for Kafka involves defining message schemas using AsyncAPI, By the end of the tutorial, you should understand the importance of message validation, know how to specify message schemas using AsyncAPI, and apply validation to ensure that the messages exchanged between different components in a Kafka-based system are consistent and adhere to predefined rules. -[Comment] <> (Add an image to show a process - flowchart that summarizes the tutorial) - +```mermaid graph TB A[Producer Service] -->|Publish Message| B[AsyncAPI Validation] B -->|Valid Message| C[Kafka Broker] @@ -37,7 +36,7 @@ graph TB A -->|Publish Invalid Message| B B -->|Invalid Message| F[Error Handling] D -->|Invalid Message| G[Error Handling] - +``` ## Background context @@ -56,12 +55,12 @@ To follow this tutorial, you need: * An understanding of Event driven Architectures (EDAs). -## Installation Guide +## Installation guide [Comment] - What goes in the installation guide? -## Broker-Level Message Validation +## Broker-level message validation From 66481c169e957c52d98444c5466200c9f2d455de Mon Sep 17 00:00:00 2001 From: Cynthia Peter Date: Thu, 2 Nov 2023 16:36:21 +0100 Subject: [PATCH 15/16] added introdcution header --- pages/docs/tutorials/message-validation-for-kafka.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/pages/docs/tutorials/message-validation-for-kafka.md b/pages/docs/tutorials/message-validation-for-kafka.md index 5686e301755..859cab25307 100644 --- a/pages/docs/tutorials/message-validation-for-kafka.md +++ b/pages/docs/tutorials/message-validation-for-kafka.md @@ -4,6 +4,8 @@ description: The goal of this tutorial is to help you validate messages for Kafk weight: 150 --- +## Introduction + This tutorial will teach you how to validate messages within a Kafka-based system using AsyncAPI specifications. Validating messages for Kafka involves defining message schemas using AsyncAPI, and then using those schemas to validate incoming and outgoing messages in your Kafka-based applications. From 7fc93a689a6ca32667ab314aa0bd0fcbd1a63022 Mon Sep 17 00:00:00 2001 From: Cynthia Peter Date: Wed, 8 Nov 2023 16:40:30 +0100 Subject: [PATCH 16/16] updated docs --- .../tutorials/message-validation-for-kafka.md | 34 ++++++++++--------- 1 file changed, 18 insertions(+), 16 deletions(-) diff --git a/pages/docs/tutorials/message-validation-for-kafka.md b/pages/docs/tutorials/message-validation-for-kafka.md index 859cab25307..9c67e9db365 100644 --- a/pages/docs/tutorials/message-validation-for-kafka.md +++ b/pages/docs/tutorials/message-validation-for-kafka.md @@ -8,9 +8,9 @@ weight: 150 This tutorial will teach you how to validate messages within a Kafka-based system using AsyncAPI specifications. -Validating messages for Kafka involves defining message schemas using AsyncAPI, and then using those schemas to validate incoming and outgoing messages in your Kafka-based applications. +Validating messages for Kafka involves defining message schemas using AsyncAPI and then using those schemas to validate incoming and outgoing messages in your Kafka-based applications. -By the end of the tutorial, you should understand the importance of message validation, know how to specify message schemas using AsyncAPI, and apply validation to ensure that the messages exchanged between different components in a Kafka-based system are consistent and adhere to predefined rules. +At the end of the tutorial, you should understand the importance of message validation, specify message schemas using AsyncAPI, and apply validation to ensure that the messages exchanged between different components in a Kafka-based system are consistent and adhere to predefined rules. ```mermaid @@ -42,33 +42,33 @@ graph TB ## Background context -Message validation involves checking that the messages exchanged between systems adhere to a predefined structure or schema. It helps in ensuring that the data being sent or received is accurate, complete, and in the expected format. +Message validation involves checking that the messages exchanged between systems adhere to a predefined structure or schema. It helps ensure the data being sent or received is accurate, complete, and in the expected format. -Kafka transports messages between producers and consumers, while AsyncAPI provides a contract (schema) that those messages must adhere to. Validation tools can use the AsyncAPI specifications to check whether the messages meet the criteria before they are sent or processed. +Kafka transports messages between producers and consumers, while AsyncAPI provides a contract (schema) to which those messages must adhere. Validation tools can use the AsyncAPI specifications to check whether the messages meet the criteria before they are sent or processed. -[Kafka](https://kafka.apache.org/documentation/#intro_nutshell) is a distributed event streaming platform for building real-time data pipelines and streaming apps. It's horizontally scalable, fault-tolerant, and swift. +[Kafka](https://kafka.apache.org/documentation/#intro_nutshell) is a distributed event streaming platform for building real-time data pipelines and apps. It's horizontally scalable, fault-tolerant, and swift. [AsyncAPI](https://www.asyncapi.com/docs) is an open-source initiative that provides a specification for defining and documenting synchronous and asynchronous APIs. It acts as a contract that ensures all system components agree on the structure of messages for effective communication. To follow this tutorial, you need: -* A knowlegde of Kafka and AsyncAPI. -* An understanding of Event driven Architectures (EDAs). - +* A knowledge of Kafka and AsyncAPI. +* An understanding of Event-driven Architectures (EDAs). +There are several ways to implement message validation for ## Installation guide - + ## Broker-level message validation -At the broker level, message validation is about ensuring that the messages sent to Kafka topics comply with a certain predefined schema. In Kafka, you can use a Schema Registry to manage and validate schemas. Also, at the broker level, ensure that only valid messages are written to the Kafka topics. +At the broker level, message validation is about ensuring that the messages sent to Kafka topics comply with a specific predefined schema. In Kafka, you can use a Schema Registry to manage and validate schemas. Also, ensure that only valid messages are written to the Kafka topics. Define an AsyncAPI schema Configure Kafka @@ -86,16 +86,18 @@ Implement Validation ## Summary -Combining Kafka's robust message streaming capabilities with AsyncAPI's standardized documentation and tooling, developers can ensure that the messages being exchanged in a distributed system are validated effectively, ensuring data integrity and system reliability. +Combining Kafka's robust message streaming capabilities with AsyncAPI's standardized documentation and tooling, developers can ensure that the messages exchanged in a distributed system are validated effectively, ensuring data integrity and system reliability. + -In this tutorial, you have defined and validated Kafka messages using AsyncAPI at both the broker and client levels, therefore ensuring that the messages conform to the specified schema before processing or sending them, which helps maintain data integrity, reliability, and consistency in your Kafka-based applications. ## Next steps -Now that you have completed this tutorial, go ahead to automate the validation and generation process in your continuous integration pipelines. +Now that you have completed this tutorial automate the validation and generation process in your continuous integration pipelines. ## Additional resources +Add Joy's tutorial on how to Create an AsyncAPI document for Kafka Messages using AsyncAPI. + +Add Arya's Guide on message validation +---> \ No newline at end of file