Make test output honor output.logstash.proxy_url

[sakurai-youhei]

What does this PR do?

This PR will fix the part of elastic/beats#24751 related to Logstash.

w/o proxy:

$ ./filebeat test output -E output.elasticsearch.enabled=false -E output.logstash.enabled=true -E output.logstash.hosts=elastic.co:443
logstash: elastic.co:443...
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 34.107.161.234, 2600:1901:0:1f6d::
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK

w/ proxy:

$ ./filebeat test output -E output.elasticsearch.enabled=false -E output.logstash.enabled=true -E output.logstash.hosts=elastic.co:443 -E output.logstash.proxy_url=socks5://user:pswd@localhost:1080
logstash: elastic.co:443...
  proxy...
    parse url... OK
    parse host... OK
    dns lookup... OK
    addresses: 127.0.0.1
    dial up... OK
  connection...
    proxy... OK
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK

w/ proxy and ssl:

$ ./filebeat test output -E output.elasticsearch.enabled=false -E output.logstash.enabled=true -E output.logstash.hosts=elastic.co:443 -E output.logstash.proxy_url=socks5://user:pswd@localhost:1080 -E output.logstash.ssl.enabled=true
logstash: elastic.co:443...
  proxy...
    parse url... OK
    parse host... OK
    dns lookup... OK
    addresses: 127.0.0.1
    dial up... OK
  connection...
    proxy... OK
    dial up... OK
  TLS...
    proxy... OK
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... OK

How I run the testing socks5 proxy

docker run -it --rm --name socks5 -p 1080:1080 -e PROXY_USER=user -e PROXY_PASSWORD=pswd serjs/go-socks5-proxy

Why is it important?

I frequently rely on test output command for troubleshooting, so its reliability is key.

Checklist

• My code follows the style guidelines of this project - mage linter:lastChange passed.
• I have commented my code, particularly in hard-to-understand areas - n/a as no such area.
• I have added tests that prove my fix is effective or that my feature works - n/a as there is https://github.com/elastic/beats/blob/main/libbeat/tests/integration/cmd_test.go
• I have added an entry in CHANGELOG.md - n/a as there's no such file.

Author's Checklist

• Fix for output.elasticsearch.proxy_url. Make test output honor output.elasticsearch.proxy_url beats#36715

Related issues

• Relates Command filebeat test output does not honor proxy_url setting for output.logstash or output.elasticsearch  beats#24751

[sakurai-youhei]

if proxy is not running:

$ ./filebeat test output -E output.elasticsearch.enabled=false -E output.logstash.enabled=true -E output.logstash.hosts=elastic.co:443 -E output.logstash.proxy_url=socks5://user:pswd@localhost:1080 -E output.logstash.ssl.enabled=true
logstash: elastic.co:443...
  proxy...
    parse url... OK
    parse host... OK
    dns lookup... OK
    addresses: 127.0.0.1
    dial up... ERROR dial tcp 127.0.0.1:1080: connect: connection refused

if proxy host is unresolvable:

$ ./filebeat test output -E output.elasticsearch.enabled=false -E output.logstash.enabled=true -E output.logstash.hosts=elastic.co:443 -E output.logstash.proxy_url=socks5://user:pswd@invalid:1080 -E output.logstash.ssl.enabled=true
logstash: elastic.co:443...
  proxy...
    parse url... OK
    parse host... OK
    dns lookup... ERROR lookup invalid on 172.20.160.1:53: no such host

if proxy port is invalid:

$ ./filebeat test output -E output.elasticsearch.enabled=false -E output.logstash.enabled=true -E output.logstash.hosts=elastic.co:443 -E output.logstash.proxy_url=socks5://user:pswd@localhost:9999999 -E output.logstash.ssl.enabled=true
logstash: elastic.co:443...
  proxy...
    parse url... OK
    parse host... OK
    dns lookup... OK
    addresses: 127.0.0.1
    dial up... ERROR dial tcp: address 9999999: invalid port

if proxy scheme is invalid:

$ ./filebeat test output -E output.elasticsearch.enabled=false -E output.logstash.enabled=true -E output.logstash.hosts=elastic.co:443 -E output.logstash.proxy_url=invalid://user:pswd@localhost:1080 -E output.logstash.ssl.enabled=true
Error initializing output: proxy: unknown scheme: invalid accessing 'output.logstash' (source:'command line flag')

if proxy credentials are incorrect:

$ ./filebeat test output -E output.elasticsearch.enabled=false -E output.logstash.enabled=true -E output.logstash.hosts=elastic.co:443 -E output.logstash.proxy_url=socks5://user:invalid@localhost:1080 -E output.logstash.ssl.enabled=true
logstash: elastic.co:443...
  proxy...
    parse url... OK
    parse host... OK
    dns lookup... OK
    addresses: 127.0.0.1
    dial up... OK
  connection...
    proxy... OK
    dial up... ERROR socks connect tcp localhost:1080->elastic.co:443: username/password authentication failed

if logstash host is unresolvable.:

$ ./filebeat test output -E output.elasticsearch.enabled=false -E output.logstash.enabled=true -E output.logstash.hosts=elastic.coooo:443 -E output.
logstash.proxy_url=socks5://user:pswd@localhost:1080 -E output.logstash.ssl.enabled=true
logstash: elastic.coooo:443...
  proxy...
    parse url... OK
    parse host... OK
    dns lookup... OK
    addresses: 127.0.0.1
    dial up... OK
  connection...
    proxy... OK
    dial up... ERROR socks connect tcp localhost:1080->elastic.coooo:443: unknown error host unreachable

if logstash is not running:

$ ./filebeat test output -E output.elasticsearch.enabled=false -E output.logstash.enabled=true -E output.logstash.hosts=elastic.co:444 -E output.log
stash.proxy_url=socks5://user:pswd@localhost:1080 -E output.logstash.ssl.enabled=true
logstash: elastic.co:444...
  proxy...
    parse url... OK
    parse host... OK
    dns lookup... OK
    addresses: 127.0.0.1
    dial up... OK
  connection...
    proxy... OK
    dial up... ERROR socks connect tcp localhost:1080->elastic.co:444: unknown error connection refused

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 89e6f4b

History

• 💚 Build #78 succeeded f8bc962
• 💚 Build #77 succeeded aa8396f
• 💚 Build #74 succeeded a7d3de1

[sakurai-youhei]

I'm closing this PR to clean up my PR list. Please feel free to reopen PR and/or ping me anytime.