Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ability to score quality of async api specification file #1131

Closed
Ferror opened this issue Jan 30, 2024 · 11 comments · Fixed by #1413
Closed

Ability to score quality of async api specification file #1131

Ferror opened this issue Jan 30, 2024 · 11 comments · Fixed by #1413
Labels
enhancement New feature or request released

Comments

@Ferror
Copy link
Contributor

Ferror commented Jan 30, 2024

Reason/Context

Many optional yet valuable parameters in the Async API specification can improve the developer experience. Like example, description, format, etc. When working with many specifications, I was asked, "How good is our documentation?". How do I measure if my documentation is good, mediocre, insufficient, or 1-100?

I propose a new command like score or evaluate that will reuse existing warnings and errors from the validate command. The command will extend the list of best practices and generate some final/aggregated scores. Hints that will increase the score should be provided.

A couple of examples
  1. if the specification has servers, then it increases the score
  2. if the specification has descriptions, then it increases the score
  3. if the specification has examples, then it increases the score
  4. if the specification has a format, then it increases the score
  5. if the specification has channels, then it increases the score
  6. if the specification has operations, then it increases the score
  7. if the specification has a license, then it increases the score
@Ferror Ferror added the enhancement New feature or request label Jan 30, 2024
Copy link
Contributor

Welcome to AsyncAPI. Thanks a lot for reporting your first issue. Please check out our contributors guide and the instructions about a basic recommended setup useful for opening a pull request.
Keep in mind there are also other channels you can use to interact with AsyncAPI community. For more details check out this issue.

@AayushSaini101
Copy link
Contributor

AayushSaini101 commented May 2, 2024

@Amzani I want to work on this issue:) I have worked similar kind of issue for Jenkins https://github.com/jenkins-infra/plugin-health-scoring that provide scoring for each pluing on basis of some parameters

@Amzani
Copy link
Collaborator

Amzani commented May 3, 2024

Sure @AayushSaini101 please go ahead.
That's a nice feature to have as well in studio

@AayushSaini101
Copy link
Contributor

AayushSaini101 commented May 3, 2024

Sure @AayushSaini101 please go ahead. That's a nice feature to have as well in studio

@Amzani @derberg @Ferror We need to decide how much score we need to increase for particular parameter. Which parameter will be more proritise as compared to others, Otherwise it will create bias issue, What do you think guys ?

@Ferror
Copy link
Contributor Author

Ferror commented May 4, 2024

I think we could start with something simple in the form of MVP.

  1. The abstraction for scorecards/metrics (What to score)
  2. Each metric may have a scale/weight
  3. Deciding on an algorithm to calculate the sum/mediana/average

Feel free to propose something. We could also start with linear calculation (without weights) 😄

@AayushSaini101
Copy link
Contributor

AayushSaini101 commented May 5, 2024

I think we could start with something simple in the form of MVP.

  1. The abstraction for scorecards/metrics (What to score)
  2. Each metric may have a scale/weight
  3. Deciding on an algorithm to calculate the sum/mediana/average

Feel free to propose something. We could also start with linear calculation (without weights) 😄

@Amzani @Ferror If we used the same scoring mechanism as Jenkins used on the 1800+ Plugins, this would be the structure: Reference: https://docs.google.com/document/d/1XT5O99-QEZadFZsMuXhzahvEmeL7-1H7HWpsnPLuMIk/edit#heading=h.cw0t6u8yvrcm here my proposal

  • New command structure: asyncapi evaluate

  • there will be coefficient and score parameters, Coefficient will tell the weight of the particular parameter and score will be [0,1] the final score will be coefficient*score.

Example:

Health Score Asyncapi Document 1 Asyncapi Document 2 Asyncapi Document 3
Score (4.7*100)/totalParameters=24 (6.8*100)/totalParameters=35 (9*100)/totalParameters=47



Health Scoring Categories for Percentage: 


Level 1 [ Low Health ]: 0-25% 


Level 2 [ Medium Health]: 25-75%


Level 3 [ Good Health]: >75%


We can add lots of parameters like check whether the latest asyncapi version is used or not etc. We can create a poll to take the approval of the coeffiicent from the TSC Members using the voting that we created in community repo. :)

@Ferror
Copy link
Contributor Author

Ferror commented May 5, 2024

TotalParameters will be based on async API schema, so it should be const. Quite simple and efficient ❤️

I'm not sure if the voting process would be IMO necessary, but feel free to create it 😄

@AayushSaini101
Copy link
Contributor

TotalParameters will be based on async API schema, so it should be const. Quite simple and efficient ❤️

I'm not sure if the voting process would be IMO necessary, but feel free to create it 😄

@Ferror Regarding the coefiicient value of different parameters, i guess we need suggestion from members, that's why i am thinking of creating vote

@Amzani
Copy link
Collaborator

Amzani commented May 6, 2024

@AayushSaini101 Great proposal ❤️
voting from TSC members is not necessary, only maintainers of this repo.
Could you run your proposal with some AsyncAPI files to see how it will look from user perspective?

@AayushSaini101
Copy link
Contributor

@AayushSaini101 Great proposal ❤️ voting from TSC members is not necessary, only maintainers of this repo. Could you run your proposal with some AsyncAPI files to see how it will look from user perspective?

Sure, I will create a draft PR tomorrow :)

@asyncapi-bot
Copy link
Contributor

🎉 This issue has been resolved in version 2.2.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request released
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

4 participants