[Fleet] Remote Elasticsearch output do not support custom ssl certificate

[nchaulet]

Description

It seems we do not support remote Elasticsearch output with custom certificate in Fleet server. When creating a remote output with an ES using self signed certificate and providing the ssl configuration in the advanced settings, the output is mark as unhealthy and agent using that policy are not sending data to the new remote ES.

It seems we do not use the whole output configuration when creating the Elasticsearch client to connect to the remote Elastic Search in Fleet Server. This cause Fleet server being unable to verify the output health or to create API keys for other Elastic agent.

Relevant code path, looks like we create the remote output client only with the host and service token, this is probably what need to be fixed. https://github.com/elastic/fleet-server/blob/main/internal/pkg/bulk/engine.go#L200

Workaround

I do not see any workaround, and this probably should be fixed with a new version.

[nchaulet]

I started to take a look, and I think here

fleet-server/internal/pkg/bulk/engine.go

Line 201 in f1340f3

hostsObj := outputMap[outputName]["hosts"]

we can create a config.Elasticsearch structure from the outputMap entry and then use the ToESConfig method to create the client for the remote output @michel-laterman @juliaElastic does it make sense to you?

[juliaElastic]

we can create a config.Elasticsearch structure from the outputMap entry and then use the ToESConfig method to create the client for the remote output @michel-laterman @juliaElastic does it make sense to you?

Sounds good to me.