Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature request]: Rewrite of generate-environment-identifiers-dict.sh #1028

Open
1 of 4 tasks
molangning opened this issue May 30, 2024 · 0 comments
Open
1 of 4 tasks

Comments

@molangning
Copy link
Contributor

molangning commented May 30, 2024

Describe the feature request:
The current script uses majestic's domain list, which may be missing a lot more domains as compared to the other lists in the checklist. Another issue that I find with it is that it makes a new (insecure) ssl session for every domain in the list, which is both insecure and inefficient.

More lists recommendations would be appreciated as these lists may be outdated.

Additional context:
You can use this command to interact with the sql server directly
psql -h crt.sh -p 5432 -U guest certwatch
https://groups.google.com/g/crtsh/c/sUmV0mBz8bQ/m/K-6Vymd_AAAJ

Domain list
https://hackertarget.com/top-million-site-list-download/
https://radar.cloudflare.com/domains
https://www.domcop.com/top-10-million-websites
https://s3-us-west-1.amazonaws.com/umbrella-static/index.html
https://majestic.com/reports/majestic-million
https://builtwith.com/top-sites
https://tranco-list.eu/
https://statvoo.com/dl/top-1million-sites.csv.zip

Next steps:

  • Implement a script that pulls domains from domcorp, alexa, cloudflare, majestic and others
  • Dedupe the list/find better ways to extract environment ids
  • Change to use sql interface
  • I intend to open a pull request later
@molangning molangning added the enhancement Enhancement label May 30, 2024
@g0tmi1k g0tmi1k added help wanted and removed enhancement Enhancement labels Aug 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants