elasticsearch-monitoring

Cluster monitoring with elasticsearch and kibana

This is an elegant way to monitor your elasticsearch cluster. See the screenshots below.

The idea is simple and straightforward:

• get statistics via nodes stats API and index them back in elasticsearch,
• repeat it periodically (e.g. every 5 minutes),
• use ttl to automatically limit statistics history (e.g. 60 days),
• visualize cluster statistics with kibana.

Installing and Running

Create the Index:

Execute scripts/elasticsearch-nodes-stats-mapping.sh on one of your cluster nodes.

Install the Script:

Copy scripts/elasticsearch-nodes-stats.sh to one of your cluster nodes.
Start executing it periodically, e.g. using crontab.

Load the Dashboard:

Install kibana and load dashboards/monitoring-dashboard.json.

Notes

• All statistics for a single node are indexed under a single type, identified by the node’s name. Use fixed (predefined) node names, otherwise any restart of elasticsearch will generate new node names and statistics for a single node will be indexed under different types (see elasticsearch configuration).
• scripts/elasticsearch-nodes-stats.sh is a python script. It uses the http library requests (if not installed: installation).
• Default time to live (ttl) is 60 days. Modify it to your needs (scripts/elasticsearch-nodes-stats-mapping.sh).
• Sampling period is defined by the executor of scripts/elasticsearch-nodes-stats.sh. For example, a simple crontab setup for execution every 5 minutes: */5 * * * * <path>/elasticsearch-nodes-stats.sh
• There are many ways to visualize the data and the given dashboard is only one of them. Consider creating multiple dashboards according to your needs (e.g. os monitor, jvm monitor, indices monitor, etc).

Screenshots

Two production nodes, sampling nodes stats every 5 minutes, keeping history for 60 days: