introduce `template-encoded` field #4315

dogancanbakir · 2023-10-30T13:41:54Z

Proposed changes

$ cat test.yaml 
id: basic-example

info:
  name: Test HTTP Template
  author: pdteam
  severity: info

http:
  - method: GET
    path:
      - "{{BaseURL}}"
    matchers:
      - type: word
        words:
          - "This is test matcher text"
        negative: true

$ go run . -u scanme.sh -t test.yaml -silent -j | jq
{
  "template-id": "basic-example",
  "template-path": "/workspaces/nuclei/cmd/nuclei/test.yaml",
  "template-encoded": "aWQ6IGJhc2ljLWV4YW1wbGUKCmluZm86CiAgbmFtZTogVGVzdCBIVFRQIFRlbXBsYXRlCiAgYXV0aG9yOiBwZHRlYW0KICBzZXZlcml0eTogaW5mbwoKaHR0cDoKICAtIG1ldGhvZDogR0VUCiAgICBwYXRoOgogICAgICAtICJ7e0Jhc2VVUkx9fSIKICAgIG1hdGNoZXJzOgogICAgICAtIHR5cGU6IHdvcmQKICAgICAgICB3b3JkczoKICAgICAgICAgIC0gIlRoaXMgaXMgdGVzdCBtYXRjaGVyIHRleHQiCiAgICAgICAgbmVnYXRpdmU6IHRydWUKICAgICAgICA=",
  "info": {
    "name": "Test HTTP Template",
    "author": [
      "pdteam"
    ],
    "tags": null,
    "severity": "info"
  },
  "type": "http",
  "host": "https://scanme.sh",
  "matched-at": "https://scanme.sh",
  "request": "GET / HTTP/1.1\r\nHost: scanme.sh\r\nUser-Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.2117.157 Safari/537.36\r\nConnection: close\r\nAccept: */*\r\nAccept-Language: en\r\nAccept-Encoding: gzip\r\n\r\n",
  "response": "HTTP/1.1 200 OK\r\nConnection: close\r\nContent-Length: 2\r\nContent-Type: text/plain; charset=utf-8\r\nDate: Mon, 30 Oct 2023 13:36:43 GMT\r\n\r\nok",
  "ip": "128.199.158.128",
  "timestamp": "2023-10-30T13:36:43.731371096Z",
  "curl-command": "curl -X 'GET' -H 'Accept: */*' -H 'Accept-Language: en' -H 'User-Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.2117.157 Safari/537.36' 'https://scanme.sh'",
  "matcher-status": true
}

This PR also adds -ot, -omit-template omit encoded template in the JSON, JSONL output flag and 1MB max template file size for encoding.

Closes #4218.

Checklist

Pull request is created against the dev branch
All checks passed (lint, unit/integration/regression tests etc.) with my changes
I have added tests that prove my fix is effective or that my feature works
I have added necessary documentation (if appropriate)

tarunKoyalwar

@ehsandeep only custom templates outside of nuclei-templates directory are encoded and stored in result . github,gitlab custom templates will not have template_encoded field is that accepted / intended ??

example

$ ./nuclei -u scanme.sh -silent  -t ~/nuclei-templates/github/tarunKoyalwar/a.yaml -j | jq .
{
  "template": "github/tarunKoyalwar/a.yaml",
  "template-url": "https://templates.nuclei.sh/public/example-test",
  "template-id": "example-test",
  "template-path": "/Users/tarun/nuclei-templates/github/tarunKoyalwar/a.yaml",
  "info": {
    "name": "Example Template Name",
    "author": [
      "pdteam"
    ],
    "tags": null,
    "description": "Description of the Template",
    "reference": [
      "https://example-reference-link"
    ],
    "severity": "info"
  },
  "type": "http",
  "host": "https://scanme.sh",
  "matched-at": "https://scanme.sh/?x=7878782e6f617374",
  "extracted-results": [
    "true"
  ],
  "request": "GET /?x=7878782e6f617374 HTTP/1.1\r\nHost: scanme.sh\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36\r\nConnection: close\r\nAccept: */*\r\nAccept-Language: en\r\nAccept-Encoding: gzip\r\n\r\n",
  "response": "HTTP/1.1 200 OK\r\nConnection: close\r\nContent-Length: 2\r\nContent-Type: text/plain; charset=utf-8\r\nDate: Wed, 01 Nov 2023 15:37:33 GMT\r\n\r\nok",
  "ip": "128.199.158.128",
  "timestamp": "2023-11-01T21:07:33.227237+05:30",
  "curl-command": "curl -X 'GET' -H 'Accept: */*' -H 'Accept-Language: en' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36' 'https://scanme.sh/?x=7878782e6f617374'",
  "matcher-status": true
}

also template-url / directory prefix may not be appropriate to classify if template is custom or not . with nuclei v3 . we have digital signature in templates and we can simply check if it was signed by projectdiscovery/nuclei-templates while loading template this will give 💯 accurate definition of public/custom template

also instead of reading content of template and base64 it while writing output . we can add new field in templates.Template called Raw or something and populate it their just after validating if it is a custom template . this also takes care of issue when we are running remote custom templates using url

what do you think @ehsandeep @dogancanbakir

ehsandeep

only custom templates outside of nuclei-templates directory are encoded and stored in result . github,gitlab custom templates will not have template_encoded field is that accepted / intended ??

No, all / any template that is not part of the public template project need to include encoded template as part of the result.

tarunKoyalwar

implementation lgtm !

$ ./nuclei -u scanme.sh -t integration_tests/protocols/http/http-preprocessor.yaml -silent -j | jq .
{
  "template-id": "http-preprocessor",
  "template-path": "/Users/tarun/Codebase/nuclei/integration_tests/protocols/http/http-preprocessor.yaml",
  "template-encoded": "aWQ6IGh0dHAtcHJlcHJvY2Vzc29yCgppbmZvOgogIG5hbWU6IFRlc3QgSHR0cCBQcmVwcm9jZXNzb3IKICBhdXRob3I6IHBkdGVhbQogIHNldmVyaXR5OiBpbmZvCgpyZXF1ZXN0czoKICAtIHJhdzoKICAgICAgLSB8CiAgICAgICAgR0VUIC8/dGVzdD0yWHFOeEZrUFA5QXh3blZEMTRlN0JzakV0MWsgSFRUUC8xLjEKICAgICAgICBIb3N0OiB7e0hvc3RuYW1lfX0KCiAgICBtYXRjaGVyczoKICAgICAgLSB0eXBlOiBzdGF0dXMKICAgICAgICBzdGF0dXM6CiAgICAgICAgICAtIDIwMA==",
  "info": {
    "name": "Test Http Preprocessor",
    "author": [
      "pdteam"
    ],
    "tags": null,
    "severity": "info"
  },
  "type": "http",
  "host": "https://scanme.sh",
  "matched-at": "https://scanme.sh/?test=2XqNxFkPP9AxwnVD14e7BsjEt1k",
  "request": "GET /?test=2XqNxFkPP9AxwnVD14e7BsjEt1k HTTP/1.1\r\nHost: scanme.sh\r\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.1 Safari/537.36\r\nConnection: close\r\nAccept-Encoding: gzip\r\n\r\n",
  "response": "HTTP/1.1 200 OK\r\nConnection: close\r\nContent-Length: 2\r\nContent-Type: text/plain; charset=utf-8\r\nDate: Tue, 07 Nov 2023 10:28:15 GMT\r\n\r\nok",
  "ip": "128.199.158.128",
  "timestamp": "2023-11-07T15:58:15.361066+05:30",
  "curl-command": "curl -X 'GET' -d '' -H 'Host: scanme.sh' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.1 Safari/537.36' 'https://scanme.sh/?test=2XqNxFkPP9AxwnVD14e7BsjEt1k'",
  "matcher-status": true
}

this works if template matched . but if no results were returned then we construct a manual result event when matcher-status=true at 1581c96 and in mock output writer .

to fix this temporary we can use strategy used in this PR earlier (i.e reading from file) and later on we can find and implement a robust and efficient implementation while working on error field in result-event issue

what do you think ? @dogancanbakir @ehsandeep

tarunKoyalwar

lgtm !

matched

$ ./nuclei -t integration_tests/protocols/http/http-preprocessor.yaml -u scanme.sh -silent -j | jq .
{
  "template-id": "http-preprocessor",
  "template-path": "/Users/tarun/Codebase/nuclei/integration_tests/protocols/http/http-preprocessor.yaml",
  "template-encoded": "aWQ6IGh0dHAtcHJlcHJvY2Vzc29yCgppbmZvOgogIG5hbWU6IFRlc3QgSHR0cCBQcmVwcm9jZXNzb3IKICBhdXRob3I6IHBkdGVhbQogIHNldmVyaXR5OiBpbmZvCgpyZXF1ZXN0czoKICAtIHJhdzoKICAgICAgLSB8CiAgICAgICAgR0VUIC8/dGVzdD0yWHRXaXFrY3FOa2I4V0p1RTdpVkhNUklSbXcgSFRUUC8xLjEKICAgICAgICBIb3N0OiB7e0hvc3RuYW1lfX0KCiAgICBtYXRjaGVyczoKICAgICAgLSB0eXBlOiBzdGF0dXMKICAgICAgICBzdGF0dXM6CiAgICAgICAgICAtIDIwMA==",
  "info": {
    "name": "Test Http Preprocessor",
    "author": [
      "pdteam"
    ],
    "tags": null,
    "severity": "info"
  },
  "type": "http",
  "host": "https://scanme.sh",
  "matched-at": "https://scanme.sh/?test=2XtWiqkcqNkb8WJuE7iVHMRIRmw",
  "request": "GET /?test=2XtWiqkcqNkb8WJuE7iVHMRIRmw HTTP/1.1\r\nHost: scanme.sh\r\nUser-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2919.83 Safari/537.36\r\nConnection: close\r\nAccept-Encoding: gzip\r\n\r\n",
  "response": "HTTP/1.1 200 OK\r\nConnection: close\r\nContent-Length: 2\r\nContent-Type: text/plain; charset=utf-8\r\nDate: Wed, 08 Nov 2023 13:09:49 GMT\r\n\r\nok",
  "ip": "128.199.158.128",
  "timestamp": "2023-11-08T18:39:48.879061+05:30",
  "curl-command": "curl -X 'GET' -d '' -H 'Host: scanme.sh' -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2919.83 Safari/537.36' 'https://scanme.sh/?test=2XtWiqkcqNkb8WJuE7iVHMRIRmw'",
  "matcher-status": true
}

not matched

$  ./nuclei -u scanme.sh -t integration_tests/protocols/http/interactsh.yaml -ms -j -silent | jq .
{
  "template-id": "interactsh-integration-test",
  "template-path": "/Users/tarun/Codebase/nuclei/integration_tests/protocols/http/interactsh.yaml",
  "template-encoded": "aWQ6IGludGVyYWN0c2gtaW50ZWdyYXRpb24tdGVzdAoKaW5mbzoKICBuYW1lOiBJbnRlcmFjdHNoIEludGVncmF0aW9uIFRlc3QKICBhdXRob3I6IHBkdGVhbQogIHNldmVyaXR5OiBpbmZvCgpyZXF1ZXN0czoKICAtIG1ldGhvZDogR0VUCiAgICBwYXRoOgogICAgICAtICJ7e0Jhc2VVUkx9fSIKICAgIGhlYWRlcnM6CiAgICAgIHVybDogJ2h0dHA6Ly97e2ludGVyYWN0c2gtdXJsfX0nCgogICAgbWF0Y2hlcnM6CiAgICAgIC0gdHlwZTogd29yZAogICAgICAgIHBhcnQ6IGludGVyYWN0c2hfcHJvdG9jb2wgIyBDb25maXJtcyB0aGUgSFRUUCBJbnRlcmFjdGlvbgogICAgICAgIHdvcmRzOgogICAgICAgICAgLSAiaHR0cCI=",
  "info": {
    "name": "Interactsh Integration Test",
    "author": [
      "pdteam"
    ],
    "tags": null,
    "severity": "info"
  },
  "type": "http",
  "host": "https://scanme.sh",
  "request": "GET / HTTP/1.1\r\nHost: scanme.sh\r\nUser-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.67 Safari/537.36\r\nConnection: close\r\nAccept: */*\r\nAccept-Language: en\r\nurl: http://cl5ok57nl535lrtsl400frxu36gdhdqmx.oast.online\r\nAccept-Encoding: gzip\r\n\r\n",
  "response": "HTTP/1.1 200 OK\r\nConnection: close\r\nContent-Length: 2\r\nContent-Type: text/plain; charset=utf-8\r\nDate: Wed, 08 Nov 2023 13:16:05 GMT\r\n\r\nok",
  "timestamp": "2023-11-08T18:46:05.236166+05:30",
  "matcher-status": false
}

tarunKoyalwar · 2023-11-08T13:18:35Z

Note:

this will be a breaking change for anyone using mock output writer
NewMockOutputWriter() -> NewMockOutputWriter(options.OmitTemplate)

ehsandeep

./nuclei -u scanme.sh -id tech-detect -silent -j -ms | jq

looks like public template with -ms option still includes encoded template which is not expected.

dogancanbakir · 2023-11-10T13:47:52Z

tests failure reason: projectdiscovery/asnmap#212

introduce template-encoded field

cdb4689

dogancanbakir requested a review from ehsandeep October 30, 2023 13:41

dogancanbakir self-assigned this Oct 30, 2023

dogancanbakir linked an issue Oct 30, 2023 that may be closed by this pull request

Add template_encode in output for custom templates #4218

Closed

ehsandeep approved these changes Oct 30, 2023

View reviewed changes

ehsandeep requested a review from tarunKoyalwar October 30, 2023 17:58

tarunKoyalwar reviewed Nov 1, 2023

View reviewed changes

dogancanbakir requested a review from ehsandeep November 2, 2023 10:33

ehsandeep reviewed Nov 4, 2023

View reviewed changes

dogancanbakir added 2 commits November 6, 2023 11:55

remove IsCustomTemplate func

3ebfb7a

refactor and move encoding to MakeResultEventItem func

08eb802

dogancanbakir requested a review from tarunKoyalwar November 6, 2023 14:34

tarunKoyalwar requested changes Nov 7, 2023

View reviewed changes

encode template in case of no results were found

432648d

dogancanbakir requested a review from tarunKoyalwar November 7, 2023 12:46

commit to last commit

dd462f8

tarunKoyalwar approved these changes Nov 8, 2023

View reviewed changes

tarunKoyalwar requested a review from ehsandeep November 8, 2023 13:18

ehsandeep requested changes Nov 9, 2023

View reviewed changes

don't encode templates when-ms is used

edd1371

dogancanbakir requested a review from ehsandeep November 10, 2023 12:12

ehsandeep approved these changes Nov 10, 2023

View reviewed changes

ehsandeep merged commit 7c2db9c into dev Nov 10, 2023
9 of 12 checks passed

ehsandeep deleted the add_template_encoded_field branch November 10, 2023 23:12

BrewTestBot mentioned this pull request Nov 18, 2023

nuclei 3.0.4 Homebrew/homebrew-core#154735

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

introduce `template-encoded` field #4315

introduce `template-encoded` field #4315

dogancanbakir commented Oct 30, 2023 •

edited

Loading

tarunKoyalwar left a comment •

edited

Loading

ehsandeep left a comment

tarunKoyalwar left a comment

tarunKoyalwar left a comment

tarunKoyalwar commented Nov 8, 2023

ehsandeep left a comment

dogancanbakir commented Nov 10, 2023

introduce template-encoded field #4315

introduce template-encoded field #4315

Conversation

dogancanbakir commented Oct 30, 2023 • edited Loading

Proposed changes

Checklist

tarunKoyalwar left a comment • edited Loading

Choose a reason for hiding this comment

ehsandeep left a comment

Choose a reason for hiding this comment

tarunKoyalwar left a comment

Choose a reason for hiding this comment

tarunKoyalwar left a comment

Choose a reason for hiding this comment

matched

not matched

tarunKoyalwar commented Nov 8, 2023

Note:

ehsandeep left a comment

Choose a reason for hiding this comment

dogancanbakir commented Nov 10, 2023

introduce `template-encoded` field #4315

introduce `template-encoded` field #4315

dogancanbakir commented Oct 30, 2023 •

edited

Loading

tarunKoyalwar left a comment •

edited

Loading