resolve proxy to inject using http package

[intxgo]

What does this PR do?

Agent is injecting proxy env variables into components configs if no proxy is already specified by the config. Unfortunately with this approach the NO_PROXY variable is lost, so the target URL has to be evaluated against the environment proxy to determine if the proxy should be used. Go http provides such functionality.

Why is it important?

The environment proxy may be defined with exclusion for Fleet and Output addresses.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Related issues

• Closes Injected proxy variables into endpoint elasticsearch does not include NO_PROXY variable #6209
• Relates Forward HTTP_PROXY/HTTPS_PROXY/NO_PROXY to components. #2602

[mergify]

This pull request does not have a backport label. Could you fix it @intxgo? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit

[cmacknz]

Looking at endpoint's Proxy.cpp I see the following suggesting this needs a change in endpoint as well. Is that correct?

    // This mimic GO implementation, though we don't support the NO_PROXY list of IP's which should
    // be accessed without proxy
    //
    // https://pkg.go.dev/golang.org/x/net/http/httpproxy

This is a separate thing. Indeed if an environment proxy is set on Endpoint, it'll skip the NO_PROXY. However, it doesn't matter for this fix since when Agent injects it's environment proxy into Endpoint config, then it takes priority at Endpoint side over environment proxy. Much like injectProxyURL here skips the environment if it already sees proxy explicitly configured.

[michel-laterman]

👍 didn't know this existed!

[cmacknz]

I think this function in its current form is the problem in the tests. If you look at the implementation it is just calling https://cs.opensource.google/go/go/+/master:src/net/http/transport.go?q=FromEnvironment&ss=go%2Fgo

func ProxyFromEnvironment(req *Request) (*url.URL, error) {
	return envProxyFunc()(req.URL)
}


// envProxyFunc returns a function that reads the
// environment variable to determine the proxy address.
func envProxyFunc() func(*url.URL) (*url.URL, error) {
	envProxyOnce.Do(func() {
		envProxyFuncValue = httpproxy.FromEnvironment().ProxyFunc()
	})
	return envProxyFuncValue
}

If you directly get a reference to httpproxy.FromEnvironment().ProxyFunc() yourself I think the problem goes away as you skip the do.Once. The following fixes the tests. You could maybe be smarter and not re-read from the environment on every configuration change if you wanted to, I'm not sure that efficiency matters here.

diff --git a/internal/pkg/agent/application/inject_proxy_component_modifier.go b/internal/pkg/agent/application/inject_proxy_component_modifier.go
index d8eb9f1e98..2f68e7452c 100644
--- a/internal/pkg/agent/application/inject_proxy_component_modifier.go
+++ b/internal/pkg/agent/application/inject_proxy_component_modifier.go
@@ -10,6 +10,7 @@ import (
        "github.com/elastic/elastic-agent-client/v7/pkg/client"
        "github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator"
        "github.com/elastic/elastic-agent/pkg/component"
+       "golang.org/x/net/http/httpproxy"
 )

 // InjectProxyEndpointModifier injects a proxy_url value into endpoint's output config if one is not set.
@@ -84,7 +85,7 @@ func injectProxyURL(m map[string]interface{}, hosts []string) {
                if err != nil {
                        continue
                }
-               proxyURL, err := http.ProxyFromEnvironment(request)
+               proxyURL, err := httpproxy.FromEnvironment().ProxyFunc()(request.URL)
                if err != nil {
                        continue
                }

[intxgo]

Thanks. This will theoretically departure a tiny bit from the desired goal to have Endpoint behaving exactly as Agent, because Agent will read the environment only once for it's http operations. However since the environment on a process can be only modified from within the process, by debugger, etc, but not just by modifying system configuration LGTM.

[intxgo]

updated 04d509d

[intxgo]

I probably found the culprit of the unit tests failure. The t.Setenv() function sets the actual environment variables on the executing process and clears them after test. It's documented as unsafe for parallel test execution.

Indeed, even locally when I run all tests I get the same failure. Experimenting I've found that environment proxy set in the shell before test execution is not overridden by t.Setenv(), so if I set them to values expected by my test it PASS even when invoked with all tests.

export NO_PROXY="do.not.inject.proxy.for.me"
export HTTPS_PROXY="https://localhost:8080"
export HTTP_PROXY="https://localhost:8081"

Could it be that these env variables are set on the testing nodes but with empty strings?

@cmacknz @michel-laterman do you know if we could set those env variables on the test nodes like above? If yes, how to do this?

[cmacknz]

The t.SetEnv call is used elsewhere in agent without issue and used to work here without issue as well. I suspect it by itself isn't the problem, the parallel warning is just that env vars are global. This is no different that calling os.Setenv I think. This is the only test setting one of the _PROXY vars so execution should be sequential here.

Something else is the problem, I can't reproduce the failure locally, so I suspect you are onto something with the CI environment being somehow different in a way that matters. I don't see us setting any of the proxy environment variables in the buildkite jobs and I can't see it in any of the buildkite output which is curious.

Edit I can reproduce locally with: go test ./internal/pkg/agent/application/... but not if I directly run the test with go test ./internal/pkg/agent/application/... -run Test_injectProxyURL

[cmacknz]

OK so the problem isn't t.Setenv not being safe it's that the switch to http.ProxyFromEnvironment(request) I think.

It includes a do.Once to setup the proxy func and I think this only reads the env vars one time. https://cs.opensource.google/go/go/+/refs/tags/go1.23.5:src/net/http/transport.go;l=907-914

This has something to do with at which point in the tests the variables are read I think.

go test ./internal/pkg/agent/application/... -run Test_injectProxyURL passes.

go test ./internal/pkg/agent/application/... fails which is closer to what CI does.

[intxgo]

It includes a do.Once to setup the proxy func and I think this only reads the env vars one time. https://cs.opensource.google/go/go/+/refs/tags/go1.23.5:src/net/http/transport.go;l=907-914

Thanks for checking for the root cause.

Do you have a suggestion then how to handle it in tests? Obviously this works correctly in the product as Agent process will just see environment it was started with. Failure to write a test doesn't justify keeping a bug, but I'm out of ideas. (I don't think re-implementing this function is the right approach).

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
77.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube