-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
introduce exclude target flag #4214
Conversation
Hi @ehsandeep @tarunKoyalwar, we would like to use this feature and I can volunteer to review this PR, if that is something that's possible. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- Display counter in single line instead of list of each
[INF] Number of hosts excluded from input list: 110
- Notes to add in docs
- Exclude target process input per line as it is.
- Exclude target does not resolve hostname to exclude ip.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
echo 1.1.1.1/24 | ./nuclei -id ptr-fingerprint -eh exclude.txt
__ _
____ __ _______/ /__ (_)
/ __ \/ / / / ___/ / _ \/ /
/ / / / /_/ / /__/ / __/ /
/_/ /_/\__,_/\___/_/\___/_/ v2.9.15
projectdiscovery.io
[INF] Number of hosts excluded from input: 10
[INF] Current nuclei version: v2.9.15 (latest)
[INF] Current nuclei-templates version: v9.6.4 (latest)
[INF] New templates added in latest release: 121
[INF] Templates loaded for current scan: 1
[INF] Targets loaded for current scan: 246
[INF] No results found. Better luck next time!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
current implementation of CIDR / IP lgtm ! proposing following changes instead of doing a exact match on target domains/urls
ex:
this works
$ echo "hackerone.com" | ./nuclei -eh hackerone.com 1 ↵
__ _
____ __ _______/ /__ (_)
/ __ \/ / / / ___/ / _ \/ /
/ / / / /_/ / /__/ / __/ /
/_/ /_/\__,_/\___/_/\___/_/ v2.9.15
projectdiscovery.io
[INF] Number of hosts excluded from input: 1
[WRN] Found 1 templates loaded with deprecated protocol syntax, update before v3 for continued support.
[INF] Current nuclei version: v2.9.15 (latest)
[INF] Current nuclei-templates version: v9.6.4 (latest)
[INF] New templates added in latest release: 121
[INF] Templates loaded for current scan: 6895
[INF] No results found. Better luck next time!
this does not
$ echo "https://hackerone.com" | ./nuclei -eh hackerone.com
__ _
____ __ _______/ /__ (_)
/ __ \/ / / / ___/ / _ \/ /
/ / / / /_/ / /__/ / __/ /
/_/ /_/\__,_/\___/_/\___/_/ v2.9.15
projectdiscovery.io
[WRN] Found 1 templates loaded with deprecated protocol syntax, update before v3 for continued support.
[INF] Current nuclei version: v2.9.15 (latest)
[INF] Current nuclei-templates version: v9.6.4 (latest)
[INF] New templates added in latest release: 121
[INF] Templates loaded for current scan: 6895
[INF] Targets loaded for current scan: 2
[INF] Running httpx on input host
[INF] Found 1 URL from httpx
[INF] Templates clustered: 1195 (Reduced 2268 Requests)
[dnssec-detection] [dns] [info] hackerone.com
[mx-service-detector:Google Apps] [dns] [info] hackerone.com
$ echo "hackerone.com" | ./nuclei -eh https://hackerone.com
__ _
____ __ _______/ /__ (_)
/ __ \/ / / / ___/ / _ \/ /
/ / / / /_/ / /__/ / __/ /
/_/ /_/\__,_/\___/_/\___/_/ v2.9.15
projectdiscovery.io
[WRN] Found 1 templates loaded with deprecated protocol syntax, update before v3 for continued support.
[INF] Current nuclei version: v2.9.15 (latest)
[INF] Current nuclei-templates version: v9.6.4 (latest)
[INF] New templates added in latest release: 121
[INF] Templates loaded for current scan: 6895
[INF] Targets loaded for current scan: 2
[INF] Running httpx on input host
[INF] Found 1 URL from httpx
[INF] Templates clustered: 1195 (Reduced 2268 Requests)
[nameserver-fingerprint] [dns] [info] hackerone.com [a.ns.hackerone.com.,b.ns.hackerone.com.]
proposed changes / discussion
- perform exclusion of hosts after uncover output ( i think this will be needed in near future)
- since the flag is
-eh exclude hosts
. exclusion of targets should be done based on Host value i.e (hostname:port) and not urls . current implementation only supports exact match - i think we should have different methods for remove and add instead of having a common
processValue
since we might get a enhancement issue soon asking forwildcard exclusion support
. if we have different methods there won't be any need of refactor and wildcard functionality would be easy to implement in future - Incorrect target count during exclusion, check
this does not
section , it counts exclusion host as a target
[INF] Targets loaded for current scan: 2
what do you think ? cc: @dogancanbakir @ehsandeep
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Merge conflict + comments from @tarunKoyalwar
FYI, will review, maybe refactor the code and then proceed with adding the new feature. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
implementation lgtm ! suggested a change regarding potential duplicated code
$ echo "hackerone.com" | ./nuclei -eh hackerone.com
__ _
____ __ _______/ /__ (_)
/ __ \/ / / / ___/ / _ \/ /
/ / / / /_/ / /__/ / __/ /
/_/ /_/\__,_/\___/_/\___/_/ v3.0.3
projectdiscovery.io
[INF] Number of hosts excluded from input: 1
[INF] Current nuclei version: v3.0.3 (outdated)
[INF] Current nuclei-templates version: v9.6.9 (latest)
[INF] New templates added in latest release: 73
[INF] Templates loaded for current scan: 7278
[INF] Executing 5264 signed templates from projectdiscovery/nuclei-templates
[WRN] Executing 2028 unsigned templates. Use with caution.
[INF] No results found. Better luck next time!
More Changes
we have a exported/public method in Input
struct i.Set(item)
which when used bypasses exclution hosts list hence Suggesting addition of new method
func (i*Input) SetWithExclusions(target string)
^ alternative method to i.Set(target)
exported method which does not add target to store if given target is already blacklisted/excluded
we use i.Set(item)
in runner and we can replace it with i.SetWithExclusions(item)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
echo "https://hackerone.com" | ./nuclei -eh hackerone.com
__ _
____ __ _______/ /__ (_)
/ __ \/ / / / ___/ / _ \/ /
/ / / / /_/ / /__/ / __/ /
/_/ /_/\__,_/\___/_/\___/_/ v3.0.3
projectdiscovery.io
panic: assignment to entry in nil map
goroutine 1 [running]:
github.com/projectdiscovery/nuclei/v3/pkg/core/inputs/hybrid.(*Input).delItem.func1({0x1400005c360, 0x22, 0x8?}, {0x14000d694d7?, 0x1?, 0x1?})
/Users/geekboy/Github/nuclei/pkg/core/inputs/hybrid/hmap.go:441 +0xec
github.com/projectdiscovery/hmap/store/disk.(*LevelDB).Scan(0x14000a2cea8?, {{0x0, 0x0}, 0x0, {0x0, 0x0}, 0x0, 0x14000ea64f8})
/Users/geekboy/go/pkg/mod/github.com/projectdiscovery/[email protected]/store/disk/leveldb.go:224 +0x1e0
github.com/projectdiscovery/hmap/store/hybrid.(*HybridMap).Scan(0x14000b2bed0?, 0xd?)
/Users/geekboy/go/pkg/mod/github.com/projectdiscovery/[email protected]/store/hybrid/hybrid.go:258 +0xe4
github.com/projectdiscovery/nuclei/v3/pkg/core/inputs/hybrid.(*Input).delItem(0x1400068f500, 0xd?)
/Users/geekboy/Github/nuclei/pkg/core/inputs/hybrid/hmap.go:425 +0xe4
github.com/projectdiscovery/nuclei/v3/pkg/core/inputs/hybrid.(*Input).Del(0x1400068f500, {0x14000b2bed0, 0xd})
/Users/geekboy/Github/nuclei/pkg/core/inputs/hybrid/hmap.go:393 +0x5dc
github.com/projectdiscovery/nuclei/v3/pkg/core/inputs/hybrid.(*Input).initializeInputSources(0x0?, 0x14000d69cf8)
/Users/geekboy/Github/nuclei/pkg/core/inputs/hybrid/hmap.go:181 +0x410
github.com/projectdiscovery/nuclei/v3/pkg/core/inputs/hybrid.New(0x14000d69cf8)
/Users/geekboy/Github/nuclei/pkg/core/inputs/hybrid/hmap.go:91 +0x33c
github.com/projectdiscovery/nuclei/v3/internal/runner.New(0x104936d80)
/Users/geekboy/Github/nuclei/internal/runner/runner.go:218 +0x820
main.main()
/Users/geekboy/Github/nuclei/cmd/nuclei/main.go:119 +0x340
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@dogancanbakir we can create follow up ticket for regex support to cover cases like wildcard based exclusion.
For regex support, #4412 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM !
Closes #3877. Based on #3946.