Add support for flog to elastic-package stream

[ruflin]

Currently elastic-package benchmark stream supports generating data for datasets which contain templates. Assuming elastic-package is more adopted for data generation, it would be nice if it could also generate in some scenarios just random log data. Instead of coming up with our own solution, could we support usage of flog (https://github.com/mingrammer/flog)? Could we take the flog output and add the necessary meta and ship it to Elasticsearch?

[jsoriano]

Would this data intended to be used with packages? There are several packages that use the docker.elastic.co/observability/stream to test their log integrations, maybe this could be more integrated in elastic-package with the help of flog or similar.

[ruflin]

I would expect it to be used with the data stream naming schemes (potentially a custom integration package that is created on the fly) but not an existing package.

I was not aware of stream, maybe @andrewkroh or @marc-gr can share more details on how this fits in? https://github.com/elastic/stream

[andrewkroh]

github.com/elastic/stream (and docker.elastic.co/observability/stream which is its container image) takes data from a file and sends it to places where to elastic-agent can read it (udp, tcp, s3, gcs, kafka, pub/sub). It does not concern itself with the generation of the input data. During system testing elastic/stream can wait for a signal (e.g. SIGHUP) from elastic-package to let it know that the Elastic Agent is ready to receive data (this is useful UDP testing).

elastic/stream could be adapted to read from pipe you wanted to connect it with something other than a file data source.