-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
introduce template-encoded
field
#4315
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ehsandeep only custom templates outside of nuclei-templates
directory are encoded and stored in result . github,gitlab custom templates will not have template_encoded
field is that accepted / intended ??
example
$ ./nuclei -u scanme.sh -silent -t ~/nuclei-templates/github/tarunKoyalwar/a.yaml -j | jq .
{
"template": "github/tarunKoyalwar/a.yaml",
"template-url": "https://templates.nuclei.sh/public/example-test",
"template-id": "example-test",
"template-path": "/Users/tarun/nuclei-templates/github/tarunKoyalwar/a.yaml",
"info": {
"name": "Example Template Name",
"author": [
"pdteam"
],
"tags": null,
"description": "Description of the Template",
"reference": [
"https://example-reference-link"
],
"severity": "info"
},
"type": "http",
"host": "https://scanme.sh",
"matched-at": "https://scanme.sh/?x=7878782e6f617374",
"extracted-results": [
"true"
],
"request": "GET /?x=7878782e6f617374 HTTP/1.1\r\nHost: scanme.sh\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36\r\nConnection: close\r\nAccept: */*\r\nAccept-Language: en\r\nAccept-Encoding: gzip\r\n\r\n",
"response": "HTTP/1.1 200 OK\r\nConnection: close\r\nContent-Length: 2\r\nContent-Type: text/plain; charset=utf-8\r\nDate: Wed, 01 Nov 2023 15:37:33 GMT\r\n\r\nok",
"ip": "128.199.158.128",
"timestamp": "2023-11-01T21:07:33.227237+05:30",
"curl-command": "curl -X 'GET' -H 'Accept: */*' -H 'Accept-Language: en' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36' 'https://scanme.sh/?x=7878782e6f617374'",
"matcher-status": true
}
also template-url / directory prefix may not be appropriate to classify if template is custom or not . with nuclei v3
. we have digital signature in templates and we can simply check if it was signed by projectdiscovery/nuclei-templates
while loading template this will give 💯 accurate definition of public/custom template
also instead of reading content of template and base64 it while writing output . we can add new field in templates.Template
called Raw
or something and populate it their just after validating if it is a custom template . this also takes care of issue when we are running remote custom templates using url
what do you think @ehsandeep @dogancanbakir
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
only custom templates outside of nuclei-templates directory are encoded and stored in result . github,gitlab custom templates will not have template_encoded field is that accepted / intended ??
No, all / any template that is not part of the public template project need to include encoded template as part of the result.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
implementation lgtm !
$ ./nuclei -u scanme.sh -t integration_tests/protocols/http/http-preprocessor.yaml -silent -j | jq .
{
"template-id": "http-preprocessor",
"template-path": "/Users/tarun/Codebase/nuclei/integration_tests/protocols/http/http-preprocessor.yaml",
"template-encoded": "aWQ6IGh0dHAtcHJlcHJvY2Vzc29yCgppbmZvOgogIG5hbWU6IFRlc3QgSHR0cCBQcmVwcm9jZXNzb3IKICBhdXRob3I6IHBkdGVhbQogIHNldmVyaXR5OiBpbmZvCgpyZXF1ZXN0czoKICAtIHJhdzoKICAgICAgLSB8CiAgICAgICAgR0VUIC8/dGVzdD0yWHFOeEZrUFA5QXh3blZEMTRlN0JzakV0MWsgSFRUUC8xLjEKICAgICAgICBIb3N0OiB7e0hvc3RuYW1lfX0KCiAgICBtYXRjaGVyczoKICAgICAgLSB0eXBlOiBzdGF0dXMKICAgICAgICBzdGF0dXM6CiAgICAgICAgICAtIDIwMA==",
"info": {
"name": "Test Http Preprocessor",
"author": [
"pdteam"
],
"tags": null,
"severity": "info"
},
"type": "http",
"host": "https://scanme.sh",
"matched-at": "https://scanme.sh/?test=2XqNxFkPP9AxwnVD14e7BsjEt1k",
"request": "GET /?test=2XqNxFkPP9AxwnVD14e7BsjEt1k HTTP/1.1\r\nHost: scanme.sh\r\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.1 Safari/537.36\r\nConnection: close\r\nAccept-Encoding: gzip\r\n\r\n",
"response": "HTTP/1.1 200 OK\r\nConnection: close\r\nContent-Length: 2\r\nContent-Type: text/plain; charset=utf-8\r\nDate: Tue, 07 Nov 2023 10:28:15 GMT\r\n\r\nok",
"ip": "128.199.158.128",
"timestamp": "2023-11-07T15:58:15.361066+05:30",
"curl-command": "curl -X 'GET' -d '' -H 'Host: scanme.sh' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.1 Safari/537.36' 'https://scanme.sh/?test=2XqNxFkPP9AxwnVD14e7BsjEt1k'",
"matcher-status": true
}
this works if template matched . but if no results were returned then we construct a manual result event when matcher-status=true
at 1581c96 and in mock output writer .
to fix this temporary we can use strategy used in this PR earlier (i.e reading from file) and later on we can find and implement a robust and efficient implementation while working on error
field in result-event issue
what do you think ? @dogancanbakir @ehsandeep
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm !
matched
$ ./nuclei -t integration_tests/protocols/http/http-preprocessor.yaml -u scanme.sh -silent -j | jq .
{
"template-id": "http-preprocessor",
"template-path": "/Users/tarun/Codebase/nuclei/integration_tests/protocols/http/http-preprocessor.yaml",
"template-encoded": "aWQ6IGh0dHAtcHJlcHJvY2Vzc29yCgppbmZvOgogIG5hbWU6IFRlc3QgSHR0cCBQcmVwcm9jZXNzb3IKICBhdXRob3I6IHBkdGVhbQogIHNldmVyaXR5OiBpbmZvCgpyZXF1ZXN0czoKICAtIHJhdzoKICAgICAgLSB8CiAgICAgICAgR0VUIC8/dGVzdD0yWHRXaXFrY3FOa2I4V0p1RTdpVkhNUklSbXcgSFRUUC8xLjEKICAgICAgICBIb3N0OiB7e0hvc3RuYW1lfX0KCiAgICBtYXRjaGVyczoKICAgICAgLSB0eXBlOiBzdGF0dXMKICAgICAgICBzdGF0dXM6CiAgICAgICAgICAtIDIwMA==",
"info": {
"name": "Test Http Preprocessor",
"author": [
"pdteam"
],
"tags": null,
"severity": "info"
},
"type": "http",
"host": "https://scanme.sh",
"matched-at": "https://scanme.sh/?test=2XtWiqkcqNkb8WJuE7iVHMRIRmw",
"request": "GET /?test=2XtWiqkcqNkb8WJuE7iVHMRIRmw HTTP/1.1\r\nHost: scanme.sh\r\nUser-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2919.83 Safari/537.36\r\nConnection: close\r\nAccept-Encoding: gzip\r\n\r\n",
"response": "HTTP/1.1 200 OK\r\nConnection: close\r\nContent-Length: 2\r\nContent-Type: text/plain; charset=utf-8\r\nDate: Wed, 08 Nov 2023 13:09:49 GMT\r\n\r\nok",
"ip": "128.199.158.128",
"timestamp": "2023-11-08T18:39:48.879061+05:30",
"curl-command": "curl -X 'GET' -d '' -H 'Host: scanme.sh' -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2919.83 Safari/537.36' 'https://scanme.sh/?test=2XtWiqkcqNkb8WJuE7iVHMRIRmw'",
"matcher-status": true
}
not matched
$ ./nuclei -u scanme.sh -t integration_tests/protocols/http/interactsh.yaml -ms -j -silent | jq .
{
"template-id": "interactsh-integration-test",
"template-path": "/Users/tarun/Codebase/nuclei/integration_tests/protocols/http/interactsh.yaml",
"template-encoded": "aWQ6IGludGVyYWN0c2gtaW50ZWdyYXRpb24tdGVzdAoKaW5mbzoKICBuYW1lOiBJbnRlcmFjdHNoIEludGVncmF0aW9uIFRlc3QKICBhdXRob3I6IHBkdGVhbQogIHNldmVyaXR5OiBpbmZvCgpyZXF1ZXN0czoKICAtIG1ldGhvZDogR0VUCiAgICBwYXRoOgogICAgICAtICJ7e0Jhc2VVUkx9fSIKICAgIGhlYWRlcnM6CiAgICAgIHVybDogJ2h0dHA6Ly97e2ludGVyYWN0c2gtdXJsfX0nCgogICAgbWF0Y2hlcnM6CiAgICAgIC0gdHlwZTogd29yZAogICAgICAgIHBhcnQ6IGludGVyYWN0c2hfcHJvdG9jb2wgIyBDb25maXJtcyB0aGUgSFRUUCBJbnRlcmFjdGlvbgogICAgICAgIHdvcmRzOgogICAgICAgICAgLSAiaHR0cCI=",
"info": {
"name": "Interactsh Integration Test",
"author": [
"pdteam"
],
"tags": null,
"severity": "info"
},
"type": "http",
"host": "https://scanme.sh",
"request": "GET / HTTP/1.1\r\nHost: scanme.sh\r\nUser-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.67 Safari/537.36\r\nConnection: close\r\nAccept: */*\r\nAccept-Language: en\r\nurl: http://cl5ok57nl535lrtsl400frxu36gdhdqmx.oast.online\r\nAccept-Encoding: gzip\r\n\r\n",
"response": "HTTP/1.1 200 OK\r\nConnection: close\r\nContent-Length: 2\r\nContent-Type: text/plain; charset=utf-8\r\nDate: Wed, 08 Nov 2023 13:16:05 GMT\r\n\r\nok",
"timestamp": "2023-11-08T18:46:05.236166+05:30",
"matcher-status": false
}
Note:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
./nuclei -u scanme.sh -id tech-detect -silent -j -ms | jq
looks like public template with -ms
option still includes encoded template which is not expected.
tests failure reason: projectdiscovery/asnmap#212 |
Proposed changes
This PR also adds
-ot, -omit-template omit encoded template in the JSON, JSONL output
flag and 1MB max template file size for encoding.Closes #4218.
Checklist