This repository provides experiment data from the ECSA 2020 paper "A Comparison of MQTT Brokers for Distributed IoT Edge Computing" written by Heiko Koziolek, Sten Grüner and Julius Rückert from ABB Research. A 10-min video presenting the paper is available on Youtube.
We investigated the performance and resilience of several clustered MQTT Brokers. There is a subfolder for each analyzed broker, which contains
- broker's configuration file(s) used during the experiments, e.g., in XML (config.xml) or plain text files (.conf)
- corresponding Kubernetes/Helm configuration files (.yaml), which may for example contain MQTT configuration options to be passed to the broker containers
- MZBench BDL files (benchmark definition language) for each experiment. These scripts initialized the load testing tool MZBench using vmq_mzbench workers. They contain for example the connection configuration (e.g. QoS1), the number of publishers and subscribers spawned, the message frequency, and the payload information.
- raw monitoring data from MZBench (experimentXYZ.csv) recorded during the experiments. These files contain timestamps, metrics names, and values.
- raw monitoring data from the tool Prometheus, a 78 MB ZIP file containing invididual CSV files for various system-wide metrics (e.g., CPU load of a Docker container)
- plots from individual experiments (experimentXYZ_cpu_load.pdf)
- plots aggregated from multiple experiments (workload-vs-cpu-load__all.pdf)
Our original testbed was a StarlingX all-in-one duplex bare metal installation running on two identical servers in a redundant, high-available fashion. Each server had a Dual Intel Xeon CPU E5-2640 v3 running at 2.60 GHz with 8x2 physical cores (32 threads), 128 GB of RAM and Gigabit connectivity.
StarlingX v3.0 is an open-source virtualization platform for edge clusters and runs on top of CentOS 7.6. All tested brokers run in Docker CE orchestrated by K8s. Prometheus monitoring tools measure CPU load among other metrics. For the broker installations we used helm charts (VerneMQ 1.10.2, EMQX 4.0.5) or public tutorials from broker vendors (Enterprise HiveMQ 4.3.2 evaluation). In K8s, the brokers use replication controllers (HiveMQ), stateful sets (VerneMQ, EMQX) and load balancer services (metalLB as Level 2 Load Balancer).
A dedicated node in the same Ethernet segment as the StarlingX controllers acts as load driver (CentOS 8.1, Intel Xeon CPU E5-2660 v4 @ 2.00 GHz, 16 cores (32 threads) and 8 GB of RAM). We evaluated different load driver applications, including mqtt-stresser, paho-clients, Locust/MQTT, JMeter/MQTT and MZBench. We decided to use MZBench due to a low resource footprint, convenient Web UI allowing to monitor and export metrics, and also a possibility to define load scenarios in a simple Benchmark Definition Language (BDL).
We utilized custom MZBench MQTT workers provided by VerneMQ. In addition to metrics from Prometheus and MZBench, we used broker dashboards provided by brokers to validate throughput measurements.
The data can be used for various purposes:
- checking the validity of the experiments
- re-processing the raw data to check other metrics (e.g., disk load)
- re-plotting the data to extract different information
- defining new experiments for different workloads, but using a similar setting
- analyzing additional MQTT brokers with the same experiment settings
To reproduce the experiments you first need to setup the MQTT Brokers, and there are numerous methods to do this:
- EMQX can be installed from binaries or by building the source. If you have Kubernetes/Helm you can also use EMQX helm charts.
- HiveMQ provides various installation options.
- VerneMQ is also available as source, binaries, Docker containers and Helm charts.
To use the BDL scripts, you need to setup MZBench, for example as Docker container, from binaries, or building the source. You can import or copy the BDL scripts into your MZBench installation (using its dashboard or command line) to reproduce the experiments. MZBench's quickstart guide explains you how to do this in detail. Starting the scenarios requires updating the included IP addresses for the brokers to your own installation. We used private IP addresses only valid in our own specific experiment setup.
We used Python scripts to create the plots for the paper, which crossed the Prometheus cpu load metrics with the throughput metrics from MZBench.
You can edit the MQTT Decision model with ADMentor. The model is meant as a minimal starting point to be refined, not as a comprehensive model to be used right away.