Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add verification details for some common crawlers #22

Merged
merged 5 commits into from
Oct 30, 2024

Conversation

e-moran
Copy link
Contributor

@e-moran e-moran commented Oct 29, 2024

This PR adds reverse dns verification details for common crawlers.

I chose to add verification as a list so that it can be present on all bots, regardless of whether we can verify or not.
When we add cidr validation in the future we will add it as a second option to those bots that support it, we can then just add logic to prefer cidr validation for bots that support both.

I think its nice to have both so that this JSON will show a full picture of everything that a crawler supports.

The mask can contain two types of wildcard:

  • @ matches zero or more characters
  • * matches zero or one character

By supporting both types of wildcard we can implement validation exactly how each bot provider specifies in their docs.

@e-moran e-moran marked this pull request as ready for review October 29, 2024 17:24
Copy link
Contributor

@blaine-arcjet blaine-arcjet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few questions and things to double check

well-known-bots.json Show resolved Hide resolved
well-known-bots.json Show resolved Hide resolved
well-known-bots.json Outdated Show resolved Hide resolved
well-known-bots.json Show resolved Hide resolved
well-known-bots.json Outdated Show resolved Hide resolved
well-known-bots.json Show resolved Hide resolved
well-known-bots.json Show resolved Hide resolved
well-known-bots.json Outdated Show resolved Hide resolved
well-known-bots.json Show resolved Hide resolved
well-known-bots.json Show resolved Hide resolved
@blaine-arcjet blaine-arcjet changed the title Add verification details for some common crawlers feat: Add verification details for some common crawlers Oct 29, 2024
@e-moran e-moran merged commit baf64c8 into main Oct 30, 2024
2 checks passed
@e-moran e-moran deleted the eoin/add-verification-details branch October 30, 2024 14:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants