Shibuya Gatling support

[iandyh]

One of the challenge for supporting Gatling in Shibuya is real time metrics.

There are some ways to stream the metrics:

1. https://github.com/bmaguireibm/gatling-prometheus-datawriter. (Suggested from @harpratap )

Feel free to leave your ideas here.

[harpratap]

The concerning part about this plugin is that it's based on a pull model, meaning either the controller or prometheus will have to end up scraping a lot of data (this will be similar to how federated prometheus works). I don't think this is a scalable design. Let's say /metrics is 1Mb in size and we have 1000 engines to be scraped, this alone will be 1Gbps worth of traffic. We need something push based, like how the current shibuya-agent for jmeter works

[iandyh]

Yes, my initial idea was fake the shibuya-agent as an InfluxDB receiver and translate the metrics to the spec that current shibuya-agent is supporting(comma separated text).

So the Gatling will send the metrics in InfluxDB format -> shibuya-agent(1) -> Shibuya controller. Starting from (1), it will be the same as Jmeter.

[harpratap]

Yes either your idea or if @piyushpgupta can write a gatling plugin that can directly provide us with a streaming endpoint in HTTP in a simple string/json format then we don't even need to mock influxDB receiver.

[piyushpgupta]

@harpratap i can check on extending current graphite writer of gatling to send the metrics in required format.

[iandyh]

@harpratap In general, comma separated format currently we are using should be the protocol between agent and the controller. This was my initial idea.

Gatling sends the require format to agent is also ok.

[piyushpgupta]

@iandyh we are using solution of gatling graphite writer --> InfluxDB + Grafana to see real time test metrics
I evaluated prometheus writer , it works well , but adding load on the pod .

[harpratap]

@piyushpgupta That's another DB to be maintained (maintaining persistent, multi-region & HA prometheus is hard enough as it is). Plus we don't know how long InfluxDB is going to stay in it's current form. The entire industry seems to be converging to prometheus metrics format and not InfluxDB format

[iandyh]

I have not touched InfluxDB for a lone time, but last time I heard, it's difficult to scale.