Skip to content

Commit

Permalink
chore: Add notes about the realities about detection
Browse files Browse the repository at this point in the history
  • Loading branch information
blaine-arcjet committed Sep 19, 2024
1 parent ef4e63a commit e01bbac
Showing 1 changed file with 12 additions and 0 deletions.
12 changes: 12 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,17 @@ Additional metadata is available on each item.

Download the [`well-known-bots.json` file][raw-json-url] directly.

## Realities

It's impossible to create a system that can detect all bots. Well-behaving bots
identify themselves in a consistent manner, usually via the User-Agent patterns
this project provides. It is straightforward to identify these well-behaving
bots, but misbehaving bots pretend to be real clients and use various mechanisms
to evade detection.

For more details, see [Non-Technical Notes in the
browser-fingerprinting][non-tech-notes-url] project.

## License

The project is a hard-fork of [crawler-user-agents][forked-repo-url] at commit
Expand All @@ -19,4 +30,5 @@ License][mit-license].

[raw-json-url]: https://raw.githubusercontent.com/arcjet/well-known-bots/main/well-known-bots.json
[forked-repo-url]: https://github.com/monperrus/crawler-user-agents/commit/46831767324e10c69c9ac6e538c9847853a0feb9
[non-tech-notes-url]: https://github.com/niespodd/browser-fingerprinting/blob/baecc60821cefd06eb89a54d18be39d87dd16f2e/README.md#non-technical-notes
[mit-license]: https://opensource.org/licenses/MIT

0 comments on commit e01bbac

Please sign in to comment.