Add a script and CI job to validate spec examples

[jonaslagoni]

This feature request was discussed here: #947 (comment)

We need to make sure all the examples are complete and valid according to our spec. This should of course be checked for each PR that change the examples.

[smoya]

We should include the examples embeded into the spec itself, not only the files under the examples dir. Related to this: #971

In order to do that, those examples would need to be fully working AsyncAPI files instead of partials. Otherwise validation won't work. If we want to avoid having full AsyncAPI documents as embedded examples, we could maybe do some trick in CI and add all the minimum required fields to each example before validating.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity 😴

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience ❤️

[smoya]

I consider this as a potential issue for next Bounty program. Not sure if there is a way of keeping track potential issues for next rounds somewhere. Perhaps labels 🤔

cc @aeworxet

[aeworxet]

I consider this as a potential issue for next Bounty program.

Sure, anyone from CODEOWNERS of this repository can submit it.

Not sure if there is a way of keeping track potential issues for next rounds somewhere. Perhaps labels 🤔

Up until now, it was simply something the maintainer just remembered.
If there is a huge flow of potential Bounty Issues in the future, I will of course introduce a way to mark those for coming calendar rounds with something like bounty/potential.
Adding GH labels to issues by a user is troublesome because, in order to SIMPLY add a label, a user needs to have full write permission to the repository, which, of course, is not likable by anyone. So I am eyeing one of the existing GH Actions bots, but no ready decision has been made yet.

[derberg]

yeah, would be good to have a kinda "reminder" like bounty/candidate or something

[smoya]

The work could be split in two phases:

1. Validation of all AsyncAPI documents under /examples dir in the Spec repo. https://github.com/asyncapi/spec/tree/master/examples. I believe we could use For the record, I believe we could use https://github.com/asyncapi/github-action-for-cli
2. Validation of all examples embedded in the spec itself. Such as the following https://github.com/asyncapi/spec/blob/master/spec/asyncapi.md#multi-format-schema-object-examples

However, I believe both should be part of the same bounty program issue.

[AnimeshKumar923]

Hello there 👋 @jonaslagoni @smoya

If it's fine with you, I would be glad to work on this issue under the Bounty Program, 2024-Q2 as this is a potential issue for the Program.

• I've worked on something similar to this issue's nature here in spec-json-schema repo
• Bounty Issue submission link for ease of access.

Thanks! 👍

---

cc: @aeworxet

[aeworxet]

@AnimeshKumar923
Submitted.

[AnimeshKumar923]

Thank you! @aeworxet @smoya

[derberg]

ah yes, let's have it on the bounty, definitely 🍺 thanks @AnimeshKumar923 for reminder

[Shurtu-gal]

Hope I am not late 😔. Would love to help out in any way possible 😉.

[aeworxet]

Bounty Issue's service comment

Text labels: bounty/2024-Q2, bounty/medium, bounty/coding
First assignment to third-party contributors: 2024-03-22 00:00:00 UTC+12:00
End Of Life: 2024-08-31 23:59:59 UTC-12:00

@asyncapi/bounty_team

[smoya]

Considering @Shurtu-gal is already assigned to two Bounty issues already (having reached the limit), it is automatically discarded from this one in favour of @AnimeshKumar923, who additionally applied earlier, and also worked on similar issues in the past.

So @AnimeshKumar923 this is all yours.

[AnimeshKumar923]

Considering @Shurtu-gal is already assigned to two Bounty issues already (having reached the limit), it is automatically discarded from this one in favour of @AnimeshKumar923, who additionally applied earlier, and also worked on similar issues in the past.

So @AnimeshKumar923 this is all yours.

Thank you for this opportunity! 🙏 @smoya
Grateful to be participating again and work with you... 🙇

---

cc: @aeworxet assignment has been made. Please update the timeline for this issue. Thanks 👍

[aeworxet]

Bounty Issue's Timeline

Complexity Level	Assignment date (by GitHub)	Start date (by BP rules)	End date (by BP rules)	Draft PR submission	Final PR submission	Final PR merge
Medium	2024-03-19	2024-04-01	2024-05-10	2024-04-12	2024-04-26	2024-05-10

Please note that the dates given represent deadlines, not specific dates, so if the goal is reached sooner, it's better.

[aeworxet]

#1046 (comment)

[aeworxet]

Upon request of the Bounty Program Participant (@AnimeshKumar923), all remaining target dates of the Bounty Issue's Timeline are extended by six calendar weeks.

Bounty Issue's Timeline Extended

Complexity Level	Assignment date (by GitHub)	Start date (by BP rules)	End date (by BP rules)	Draft PR submission	Final PR submission	Final PR merge
Medium	2024-03-19	2024-04-01	2024-06-21	2024-04-12	2024-06-07	2024-06-21

Please note that the dates given represent deadlines, not specific dates, so if the goal is reached sooner, it's better.

Keep in mind the responsibility for violations of the Timeline.

[AnimeshKumar923]

From May 19th to May 28th, I had prolonged break-downs of telecommunications in my area (which included internet shutdown). This situation is subject to Article 3(f) of the ICC Force Majeure and Hardship Clauses. Hence, I request an additional extension by two weeks to the one I already have.

cc: @aeworxet @smoya

[AnimeshKumar923]

Progress Update

• The 1st phase of the issue is completed. Now, we're moving to the 2nd phase of the issue.
• I've conversed with Sergio and we're figuring out ways on how to extract the embedded examples (an example) and construct it to be a fully functional AsyncAPI document before passing it to the validator.

[aeworxet]

Upon request of the Bounty Program Participant (@AnimeshKumar923), all remaining target dates of the Bounty Issue's Timeline are extended by eight calendar weeks.

Bounty Issue's Timeline Extended

Complexity Level	Assignment date (by GitHub)	Start date (by BP rules)	End date (by BP rules)	Draft PR submission	Final PR submission	Final PR merge
Medium	2024-03-19	2024-04-01	2024-07-05	2024-04-12	2024-06-21	2024-07-05

Please note that the dates given represent deadlines, not specific dates, so if the goal is reached sooner, it's better.

Keep in mind the responsibility for violations of the Timeline.

[smoya]

As embeded examples (asyncapi.md file) are partials of an AsyncAPI doc, meaning the examples are not full AsyncAPI documents but just parts of it, we need somehow to annotate on each example which part of the spec (to which object) the example maps to.

I suggested @AnimeshKumar923 to add an HTML comment like the following.

Given the following embedded example for Operations Object:

onUserSignUp:
  title: User sign up
  summary: Action to sign a user up.
  description: A longer description
  channel:
    $ref: '#/channels/userSignup'
  action: send
  tags:
    - name: user
    - name: signup
    - name: register
  bindings:
    amqp:
      ack: false
  traits:
    - $ref: '#/components/operationTraits/kafka'

To add the following line before it:
<!-- asyncapi-example-tester:{test:'Operations Object',json_path:'$.operations'} -->

By using JSON Path, we can set up a base AsyncAPI document that includes everything is needed (not just Info object, etc) but also all components referred in all examples. In that way, it will be just a matter of replacing that base object with the example on those indicated paths and validate the whole doc.

You can see the current whole spec with such HTML comments added in my following gist: https://gist.github.com/smoya/1eafa264cee7ba6d9be7419f8ec33859 (see raw)

I believe @AnimeshKumar923 is working on providing such base AsyncAPI document: https://asyncapi.slack.com/archives/C34F2JV0U/p1717573165200079

I'm asking the rest of Code Owners to comment and share what is your feeling about adding such comments or to suggest an alternative cc @fmvilas @derberg @dalelane @char0n @GreenRover

[AnimeshKumar923]

Thank you so much for your suggestion @smoya Really appreciate it!.
Also, can the maintainers please provide their feedback upon this method (or maybe suggest an alternative approach?)
Will adding the comments break anything? Or some other problems that we're unaware of?
Thanks! 👍

[GreenRover]

I dont know how the markdown parser will handle those comments, but this should be easy to test.
I would like to see a poc.
But expect it to work

[fmvilas]

I like it. Markdown parsers will handle it just fine because HTML is supported in Markdown.

An alternative approach could be to generate these examples (and actually the whole Markdown file) from the JSON Schema definition but that's a totally different thing 😅

[dalelane]

I really like this - I think having a way to catch problems in examples will be hugely valuable.

Perhaps like @fmvilas, I started considering an alternate approach - require that examples be written in full and stored externally, and then embed an excerpt from it in markdown docs. My reason for thinking this is that it would catch issues that can only be observed in the context of a whole doc, such as references to other sections.

But I've talked myself out of this as I think it's too onerous a requirement for docs to demand fully examples every time we want to illustrate something, and would likely put people off adding examples at all.

I think your approach is a better compromise - gets us to improve the checking we do on examples, with only minimal incremental cost for each example.

[AnimeshKumar923]

Thank you for your thoughts upon this! @GreenRover @fmvilas @dalelane 👍

Although it's still a WIP, I'm currently trying to implement the approach that @smoya suggested via #1059

My reason for thinking this is that it would catch issues that can only be observed in the context of a whole doc, such as references to other sections.

This is the thing that I've been pondering upon recently. Sometime the examples would contain references that would be new, whose references might not be already present there in the base document, ultimately resulting in failing of the validation in the CI.
I'm thinking that there should be an acceptable range that 'okay, now this can handle majority of the examples'. Maybe we can add more examples and fields as we receive them in the future? Don't know, just putting it out there...

But I've talked myself out of this as I think it's too onerous a requirement for docs to demand fully examples every time we want to illustrate something, and would likely put people off adding examples at all.

That I agree with. I'm thinking of going right now with the approach suggested by @smoya and in the future, if needed, we can open a new issue and try to implement this way of validating the embedded, or maybe seek out to find alternatives... 👀

[AnimeshKumar923]

Also, since both JSON and YAML excerpts are given for a particular example, I'm thinking of taking only the JSON excerpts from the asyncapi.md file because ultimately that would be used for the JSONPath and document validation.

[smoya]

Perhaps like @fmvilas, I started considering an alternate approach - require that examples be written in full and stored externally, and then embed an excerpt from it in markdown docs. My reason for thinking this is that it would catch issues that can only be observed in the context of a whole doc, such as references to other sections.

I thought about this and Indeed I see it as the right way to go. However, it escapes from my knowledge. Not sure if we use this for rendering code blocks written in markdown files. If so, we could implement a trim function or similar so we can set the lines we want to extract from the example file and use it as final example.

Definitely a good issue to work in the future. Totally worth opening a new issue in website repo.

[smoya]

This task is completed now.
This is a Bounty program issue assigned to @AnimeshKumar923 and it has been merged in time 💯

Cc @aeworxet

[aeworxet]

Bounty Issue Completed 🎉

@AnimeshKumar923, please go to the AsyncAPI page on Open Collective and submit an invoice for USD 200.00 with the expense title Bounty spec#957, tag bounty, and full URL of this Bounty Issue in the description.

[AnimeshKumar923]

Bounty Issue Completed 🎉

@AnimeshKumar923, please go to the AsyncAPI page on Open Collective and submit an invoice for USD 200.00 with the expense title Bounty spec#957, tag bounty, and full URL of this Bounty Issue in the description.

Submitted.

[AnimeshKumar923]

Thank you so much @smoya for this opportunity. I learned a lot during the resolution of this issue. Looking forward to more collaborations in the future. 🙏 Cheers! 🙌

Thank you @GreenRover @fmvilas @dalelane as well for your valuable input. 🙏

[smoya]

@AnimeshKumar923 It would be so cool if you can write a blog post on AsyncAPI website about how you managed to set up validation of the embedded examples. Just suggesting!

[AnimeshKumar923]

@AnimeshKumar923 It would be so cool if you can write a blog post on AsyncAPI website about how you managed to set up validation of the embedded examples. Just suggesting!

@smoya That sounds good. I'll look at it. I'll be a bit busy for 3-4 weeks due to new college semester starting and need to figure out and setup some stuff. I'll keep working on it bit-by-bit as I get time in between.
Thanks for the suggestion. Appreciate it!