docs: message validation for kafka #2266
Conversation
✅ Deploy Preview for asyncapi-website ready!Built without sensitive environment variables
To edit notification comments on pull requests, go to your Netlify site configuration. |
CynthiaPeter
changed the title
|
|
||
| ## Installation / Validation | ||
|
|
||
| ### Validating at the Broker level |
There was a problem hiding this comment.
Broker-Level Message Validation
Alternate header idea :)
|
|
||
| ### Validating at the Broker level | ||
|
|
||
| ### Validating at the client level |
There was a problem hiding this comment.
Client-Level Message Validation
|
|
||
| ## Introduction | ||
|
|
||
| [comment:] <> (What do we learn, and at the end, what do we accomplish? ) |
There was a problem hiding this comment.
Introduction should tell the reader:
- What they're learning
- What we're building, explain starting point and where we end
There was a problem hiding this comment.
@CynthiaPeter, you shouldn't resolve comments until you address them. Please make sure to address all comment feedback and THEN you can resolve. 😄
There was a problem hiding this comment.
I resolved them.
|
|
||
| ### Summary | ||
|
|
||
| [comment:] <> (Here is what we learned and did.) |
There was a problem hiding this comment.
Summary
- What did we learn, what we built, and how to apply this to real life.
| * | ||
| * | ||
| * | ||
|
|
There was a problem hiding this comment.
Let's add a summarizing diagram of the Introduction info.
…://github.com/CynthiaPeter/website into feature/message-validation-for-kafka-tutorial
|
|
||
|
|
||
|
|
||
| ## Broker-level message validation |
There was a problem hiding this comment.
Heyo @CynthiaPeter, Lukasz clarified to me today in call that there is no broker level message validation. Broker just takes what you have and passes it forward; broker does not handle validation. :)
There was a problem hiding this comment.
Lukasz is saying we can narrow the scope of the tutorial... instead of message validation broadly, we focus on ONE kind of message validation tutorial. For example, Schema Registry validation.
There was a problem hiding this comment.
Ok. Thank you.
Does this mean that Arya will take it on based off the fact that he is closer to the message?
There was a problem hiding this comment.
Hello!,
Message validation for Kafka does not apply at the broker level but at the client libraries level. These are the ones that - via the Ser/DeSer configuration - may ensure the messages you publish or receive conform to a schema.
However, this Ser/DeSer configuration is optional, and as it is tied to client libraries, it may vary from one language to another or one implementation to another.
That said, I think that it is essential to differentiate runtime validation from build-time validation. As of today, I don't see how AsyncAPI would integrate into runtime validation (trusted today by Ser/DeSer configuration). So I wouldn't have put the "AsyncAPI Validation" block on the critical path between producer and broker and between broker and consumer. Deporting what Ser/DeSer is expected to do to another block would led to so much complexity...
I think that AsyncAPI usage for message validations is possible in a number of cases:
- At runtime but using a "dark consumer" approach - launching a consumer aside or before real production consumers to do the actual validation,
- At build-time but in a way where you dissociate producer validation to consumer validation as introduced by this schema:
Of course, I don't want to influence things but it may be worth to have a look at what we're doing at Microcks on validation stuffs: https://microcks.io/documentation/guides/avro-messaging/#4-run-asyncapi-tests
Let me know if it's helpful. I would be happy to pursue the discussion!
There was a problem hiding this comment.
Hello Laurent,
Thank you for the insights. I'll look into this tomorrow and come back with specific questions.
@Arya-Gupta if you have time, look at this.
Description
The initial draft for validating messages for Kafka.
Related issue(s)
New Tutorial - Message validation with Kafka #2204