docs: document the structure of the json file (#26)

* docs: document the structure of the json file * better grammar * Update README.md Co-authored-by: blaine-arcjet <[email protected]> --------- Co-authored-by: blaine-arcjet <[email protected]>
arcjet · Nov 15, 2024 · 4f407e2 · 4f407e2
1 parent baf64c8
commit 4f407e2
Showing 1 changed file with 27 additions and 0 deletions.
diff --git a/README.md b/README.md
@@ -22,6 +22,33 @@ to evade detection.
 For more details, see [Non-Technical Notes in the
 browser-fingerprinting][non-tech-notes-url] project.
 
+## Structure
+
+Each entry in the JSON represents a specific bot or crawler and includes the following fields:
+
+- id: A unique identifier for the bot
+- categories: An array of categories the bot belongs to (e.g., "search-engine", "advertising")
+- pattern: A regular expression pattern used to identify the bot in user agent strings
+- url: (optional) A URL with more information about the bot
+- verification: A list of supported methods for verifying the bot's identity (if the bot is not verifiable it should be empty).
+- instances: An array of example user agent strings for the bot
+
+### Verification
+
+Each verification entry contains the following fields:
+
+- type: The method of verification (currently only `dns` is supported)
+- masks: An array of mask patterns used for verification
+
+### Verification mask patterns
+
+The mask patterns use the following special characters:
+
+- *: Represents 0 or 1 of any character
+- @: Acts as a wildcard, matching any number of characters
+
+All other characters in the mask require an exact match.
+
 ## License
 
 The project is a hard-fork of [crawler-user-agents][forked-repo-url] at commit