-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
introduce scan context #4373
introduce scan context #4373
Conversation
TODO
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
initial implementation seems good . we should probably circle back and divide implementation into small tasks and only implement necessary changes in this PR and implement other changes ( template context , stats etc in follow up issues)
more details in TODO
$ go run . -t test.yaml -u https://gggggggggggle.com -matcher-status -jsonl -silent | jq
{
"template-id": "failed_test",
"template-path": "/workspaces/nuclei/cmd/nuclei/test.yaml",
"template-encoded": "aWQ6IGZhaWxlZF90ZXN0CgppbmZvOgogIG5hbWU6IFRlc3QgSFRUUCBUZW1wbGF0ZQogIGF1dGhvcjogcGR0ZWFtCiAgc2V2ZXJpdHk6IGluZm8KCmh0dHA6CiAgLSBtZXRob2Q6IEdFVAogICAgcGF0aDoKICAgICAgLSAie3tCYXNlVVJMfX0iCgogICAgbWF0Y2hlcnM6CiAgICAgIC0gdHlwZTogd29yZAogICAgICAgIHdvcmRzOgogICAgICAgICAgLSAiVGhpcyBpcyB0ZXN0IG1hdGNoZXIgdGV4dCI=",
"info": {
"name": "Test HTTP Template",
"author": [
"pdteam"
],
"tags": null,
"severity": "info"
},
"type": "http",
"host": "https://gggggggggggle.com",
"request": "GET / HTTP/1.1\r\nHost: gggggggggggle.com\r\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2656.18 Safari/537.36\r\nConnection: close\r\nAccept: */*\r\nAccept-Language: en\r\nAccept-Encoding: gzip\r\n\r\n",
"timestamp": "2023-11-17T10:01:37.209519923Z",
"matcher-status": false,
"error": "GET https://gggggggggggle.com giving up after 2 attempts: Get \"https://gggggggggggle.com\": no address found for host"
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
implementation lgtm ! and seems to be working correctly. I think we should complete below necessary changes and then decide changes to be implemented in follow up steps
$ ./nuclei -id tech-detect -u https://scanmescanme.ssh -ms -jsonl -silent | jq .
{
"template": "http/technologies/tech-detect.yaml",
"template-url": "https://templates.nuclei.sh/public/tech-detect",
"template-id": "tech-detect",
"template-path": "/Users/tarun/nuclei-templates/http/technologies/tech-detect.yaml",
"info": {
"name": "Wappalyzer Technology Detection",
"author": [
"hakluke"
],
"tags": [
"tech"
],
"severity": "info",
"metadata": {
"max-request": 1
}
},
"type": "http",
"host": "https://scanmescanme.ssh",
"request": "GET / HTTP/1.1\r\nHost: scanmescanme.ssh\r\nUser-Agent: Mozilla/5.0 (X11; OpenBSD i386) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36\r\nConnection: close\r\nAccept: */*\r\nAccept-Language: en\r\nAccept-Encoding: gzip\r\n\r\n",
"timestamp": "2023-11-21T21:19:41.615583+05:30",
"matcher-status": false,
"error": "GET https://scanmescanme.ssh giving up after 2 attempts: Get \"https://scanmescanme.ssh\": no address found for host"
}
Changes
- Update Executer Interface to align with current logic (i.e without callback in input)
// Existing Executer interface
type Executer interface {
// Compile compiles the execution generators preparing any requests possible.
Compile() error
// Requests returns the total number of requests the rule will perform
Requests() int
// Execute executes the protocol group and returns true or false if results were found.
Execute(ctx *scan.ScanContext) (bool, error)
// ExecuteWithResults executes the protocol requests and returns results instead of writing them.
ExecuteWithResults(ctx *scan.ScanContext) ([]*output.ResultEvent, error)
}
// New or Proposed Executer interfacoe
type Executer interface {
// Compile compiles the execution generators preparing any requests possible.
Compile() error
// Requests returns the total number of requests the rule will perform
Requests() int
// Execute executes the protocol group and returns true or false if results were found.
ExecuteWithScanCtx(ctx *scan.ScanContext) error
// ExecuteWithResults executes the protocol requests and returns results instead of writing them.
// Note: In case of error it will also include a ResultEvent which provides more accurate information
// about error and other info like total request sent and other necessary info
ExecuteWithInput(input *contextargs.Context) ([]*output.ResultEvent, error)
}
- we still implement logic regarding printing results in tmplExec package instead it should be defined in ScanContext itself providing better seperation between cli and sdk usage of nuclei without needing to use MockOutputWriter in sdk at all . example
// Customizable context args for sdk use cases (i.e no stdout logging implemented here)
func NewScanCtx(ctx *contextargs.Context)
// For CLI use case . in this function we use output writers etc from opts
// and add that logic while initializing scanCtx using protocols.ExecuterOptions
func NewScanCtxWithOpts(ctx *contextargs.Context, opts *protocols.ExecuterOptions)
Follow-up tickets related to refactor
- refactor stats logic to use ScanCtx and fix incorrect stats etc
- refactor interactsh output logic in nuclei (i.e implement cooldown in ScanCtx instead of outputWriter) (also add meta-fields to track whether interaction was recieved at all or not)
- refactor sdk internals to use ScanCtx after resolving above two tickets to make it more robust
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm !
$ ./nuclei -id tech-detect -u https://scanmescanme.ssh -ms -jsonl -silent | jq .
{
"template": "http/technologies/tech-detect.yaml",
"template-url": "https://templates.nuclei.sh/public/tech-detect",
"template-id": "tech-detect",
"template-path": "/Users/tarun/nuclei-templates/http/technologies/tech-detect.yaml",
"info": {
"name": "Wappalyzer Technology Detection",
"author": [
"hakluke"
],
"tags": [
"tech"
],
"severity": "info",
"metadata": {
"max-request": 1
}
},
"type": "http",
"host": "https://scanmescanme.ssh",
"request": "GET / HTTP/1.1\r\nHost: scanmescanme.ssh\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36\r\nConnection: close\r\nAccept: */*\r\nAccept-Language: en\r\nAccept-Encoding: gzip\r\n\r\n",
"timestamp": "2023-11-28T00:16:43.77864+05:30",
"matcher-status": false,
"error": "GET https://scanmescanme.ssh giving up after 2 attempts: Get \"https://scanmescanme.ssh\": no address found for host"
}
will create followup tickets shortly
Proposed changes (initial)
Please describe your feature request:
When I use nuclei to run my test cases using
matcher-status
option that generates a result event for failed matchers, it doesn't include results when the host got errored out for multiple reasons, which nuclei failed to report.nuclei have a separate option to track errored hosts with error information using
-elog
option, and unfortunately, it can not be mapped directly with the result itself.Describe the use case of this feature:
I'm running nuclei in my CI pipeline, nuclei reported multiple results upon 1st run; as a result of one vulnerability, I've taken down the vulnerable host, upon next nuclei run, nuclei failed to report the same vulnerabilities as it silently failed to run as host is not accessible.
Instead, nuclei can still generate failed match events and populate error information as it does when using with
-elog
option.Here is example run with this support:
1. Test Template
2. Test Run
3. Test Result
Note:
Proposed changes
Refactor Nuclei to use
ScanCtx
Jargon
Background / Intro
TemplateExecuter
for executing a template on one or more than 1 targets (tmplexec package).TemplateExecuter
and all its field are stateless (in context of scan) . simply put a scan should not update/modify any data in TemplateExecuter or it will cause a race condition / panicRequirement / Issue
we are trying to return any error occured during execution of a scan in json response when
-ms
flag is used (ref: #4299) .The solution for this is to add a field callederror
inResultEvent
. But problem is that it doesn’t seem to be possible with current arch/design . since we generateResultEvent
in callbacks, looking at above function signature we can tell that error is returned seperately and we exit function on error ( at multiple places).generating event in callback may not be good idea and we have so many nested callbacks in nuclei but unfortunately there is no simple / alternate solution to replace callbacks at least for now
How does current error logging work (like elog etc) ??
TemplateExecuter/ Protocols etc have a field called
ExecuterOptions
which contains global (ideally) read-only resources that are shared across all protocols ex: browser instance , interactsh instance and lot more . we haveoutput.Writer
field inExecuterOptions
that is called to log errors to a io.Writer .Since this is a global writer it is not possible to hijack error’s and pass it to
ResultEvent
*contextargs.Context
templateContext
(a global map of variables per scan) for this we are hashing contextArgs and using that as key . sinceinput *contextargs.Context
is scan specific . ( it is cloned at protocol level etc but more or less remains constant for a scan )ExecuterOptions
. this ideally should not happen since we aim to keepTemplateExecuter
stateless but this was only/easiest hack to use at that momentProposed Solution
I think solution is not something entirely new but it just extends to existing implementation of
contextArgs.Context
and following a strict practice/guidelines while codingThe idea remains the same i.e keeping
TemplateExecuter
andProtocols.Request
stateless and move all update/modify related data (like variables/templateContext) toScanContext
. The idea is to treat it the same way as we usecontext.Context
in golang .ScanContext is wrapper around
input *contextargs.Context
and will include all stateful data related to a scanso any function invocations in
ExecuteWithResults
function should follow the almost same new function signature as shown belowExample
current implementation
new Implementation
With this we can include much more than just a error in
ResultEvent
since GenerateResult is a method ofScanContext
and it has all scan related data . we can include everything that will be required in clean way . Others things that can be included inResultEvent
areConclusion
Checklist