headless: automerge and other improvements #3958

tarunKoyalwar · 2023-07-19T16:22:25Z

Proposed changes

add support for automerge in headless mode
make headless protocol consistent with http protocol
- use of helper functions in url
- now all variables of http (Port,TLD etc) are available instead of just BaseURL and Hostname
- protocol request vardump support
fixed formedURL / matchedURL
closes Query automerge not supported in headless mode #3919

Checklist

Pull request is created against the dev branch
All checks passed (lint, unit/integration/regression tests etc.) with my changes
I have added tests that prove my fix is effective or that my feature works
I have added necessary documentation (if appropriate)

tarunKoyalwar · 2023-07-19T16:24:22Z

Before

$  nuclei -u https://projectdiscovery.io -t ~/test-templates/headlessparams.yaml -headless -v -debug-req -svd

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.9.9

		projectdiscovery.io

[INF] Current nuclei version: v2.9.9 (latest)
[INF] Current nuclei-templates version: v9.5.8 (latest)
[INF] New templates added in latest release: 113
[INF] Templates loaded for current scan: 1
[INF] Targets loaded for current scan: 1
[DBG] Protocol request variables: 
	1. aa => xss

[INF] [poc] Dumped Headless request for https://projectdiscovery.io
[DBG] 	navigate url:https://projectdiscovery.io/?{{aa}}="><pwnd>
	waitload 
[DBG] Protocol response variables: 
	1. matched => https://projectdiscovery.io
	2. data => <html lang="en" style="ov .... /div></div></body></html>
	3. header => X-Proxy-Cache: MISS Via:  .... oding Server: cloudflare 
	4. type => headless
	5. template-id => poc
	6. template-path => /Users/tarun/test-templates/headlessparams.yaml
	7. host => https://projectdiscovery.io
	8. status_code => 200
	9. history => GET https://projectdiscov .... urity: max-age=31536000  
	10. template-info => {poc poc poc   {medium} map[] <nil> }
	11. aa => xss
	12. req => 	navigate url:https://pro .... aa}}="><pwnd> 	waitload  

[poc] [headless] [medium] https://projectdiscovery.io

tarunKoyalwar · 2023-07-19T16:24:51Z

After

$ go run . -u "https://projectdiscovery.io/?url=abc&value=false" -t ~/test-templates/headlessparams.yaml -headless -v -debug-req -svd


                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.9.9

		projectdiscovery.io

[INF] Current nuclei version: v2.9.9 (latest)
[INF] Current nuclei-templates version: v9.5.8 (latest)
[INF] New templates added in latest release: 113
[INF] Templates loaded for current scan: 1
[INF] Targets loaded for current scan: 1
[DBG] Protocol request variables: 
	1. aa => xss

[DBG] Final Protocol request variables: 
	1. Scheme => https
	2. File => 
	3. Input => https://projectdiscovery.io
	4. FQDN => projectdiscovery.io
	5. DN => projectdiscovery
	6. Hostname => projectdiscovery.io
	7. Port => 443
	8. aa => xss
	9. SD => 
	10. BaseURL => https://projectdiscovery.io
	11. RootURL => https://projectdiscovery.io
	12. TLD => io
	13. Path => 
	14. ip => 
	15. Host => projectdiscovery.io
	16. RDN => projectdiscovery.io

[INF] [poc] Dumped Headless request for https://projectdiscovery.io/?url=abc&value=false&xss="><pwnd>
[DBG] 	navigate => https://projectdiscovery.io/?url=abc&value=false&xss="><pwnd>
	waitload 
[DBG] Protocol response variables: 
	1. header => X-Proxy-Cache: MISS X-Cac .... -Control-Allow-Origin: * 
	2. status_code => 200
	3. history => GET https://projectdiscov .... i}(window,document,_iub);
	4. template-id => poc
	5. aa => xss
	6. host => https://projectdiscovery.io/?url=abc&value=false
	7. req => 	navigate => https://proj .... &xss="><pwnd> 	waitload  
	8. data => <html lang="en" style="ov .... /div></div></body></html>
	9. type => headless
	10. template-info => {poc poc poc   {medium} map[] <nil> }
	11. template-path => /Users/tarun/test-templates/headlessparams.yaml
	12. matched => https://projectdiscovery. .... &value=false&xss="><pwnd>

[poc] [headless] [medium] https://projectdiscovery.io/?url=abc&value=false&xss="><pwnd>

Mzack9999

typo in the function signature

Unrelated Notes:

Template for testing from Query automerge not supported in headless mode #3919

id: poc

info:
  name: poc
  author: poc
  severity: medium
  tags: poc

headless:
  - steps:
      - args:
          url: '{{BaseURL}}/?{{aa}}="><pwnd>'
        action: navigate
      - action: waitload

    payloads:
      aa:
        - xss

The concept of "Request" for headless has to be considered more in the context of progressive Actions with optional arguments, which is also the reason why I believe that headless syntax simplification and attributes unification #3846 will be ineffectual unless the actions are described in natural language.

v2/pkg/protocols/headless/request.go

tarunKoyalwar · 2023-07-20T13:13:04Z

@Mzack9999 i think headless and http protocol have some major differences

Description	Headless	HTTP
state	Stateful by nature	stateless by nature
actions	action are stateful/dependent and are executed in order	by nature they are stateless and share a http client
terminology	more like a tab in browser	simple http client

due to these difference i am not sure if we could simplify syntax in headless protocol
instead i think unless a template needs a stateful , multiple action in order we should try to write template in http protocol
and internally in nuclei engine we can add features of headless that are useful in http protocol

cookie-reuse in http & headless templates which was recently introduced
headless http client to avoid fingerprinting etc

Mzack9999

lgtm!

+1 for all the points @tarunKoyalwar - standard http should be always preferred for actions that do not specifically require headless

headless: automerge and other improvements

c7284f6

tarunKoyalwar linked an issue Jul 19, 2023 that may be closed by this pull request

Query automerge not supported in headless mode #3919

Closed

tarunKoyalwar requested review from Mzack9999 and ehsandeep July 19, 2023 16:28

tarunKoyalwar self-assigned this Jul 19, 2023

Mzack9999 requested changes Jul 20, 2023

View reviewed changes

v2/pkg/protocols/headless/request.go Outdated Show resolved Hide resolved

fix typo in function signature

ac8c929

tarunKoyalwar requested a review from Mzack9999 July 20, 2023 13:14

tarunKoyalwar added the Status: Review Needed The issue has a PR attached to it which needs to be reviewed label Jul 25, 2023

Mzack9999 approved these changes Jul 28, 2023

View reviewed changes

ehsandeep approved these changes Jul 28, 2023

View reviewed changes

ehsandeep merged commit beb1bf6 into dev Jul 28, 2023
9 of 11 checks passed

ehsandeep deleted the issue-3919-automerge-headless branch July 28, 2023 13:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

headless: automerge and other improvements #3958

headless: automerge and other improvements #3958

tarunKoyalwar commented Jul 19, 2023 •

edited

Loading

tarunKoyalwar commented Jul 19, 2023

tarunKoyalwar commented Jul 19, 2023

Mzack9999 left a comment •

edited by tarunKoyalwar

Loading

tarunKoyalwar commented Jul 20, 2023

Mzack9999 left a comment

headless: automerge and other improvements #3958

headless: automerge and other improvements #3958

Conversation

tarunKoyalwar commented Jul 19, 2023 • edited Loading

Proposed changes

Checklist

tarunKoyalwar commented Jul 19, 2023

Before

tarunKoyalwar commented Jul 19, 2023

After

Mzack9999 left a comment • edited by tarunKoyalwar Loading

Choose a reason for hiding this comment

tarunKoyalwar commented Jul 20, 2023

Mzack9999 left a comment

Choose a reason for hiding this comment

tarunKoyalwar commented Jul 19, 2023 •

edited

Loading

Mzack9999 left a comment •

edited by tarunKoyalwar

Loading