Scheduled Receiver Service for Monitoring Lag of Kafka Consumers with JAVA
Usage: Set BOOTSTRAP_SERVER variable in ScheduledKafkaConsumerLagReceiver.java to your Kafka bootstrap server and start application (Schedule time can be arranged from same file)
Description: Scheduled service as Java application to monitor lag state of each consumer through bootstrap server of Kafka. Gets all consumer groups in server, finds all consumers of each consumer group and receives their lag state with other required informations by sending paralel Api requests. (GROK pattern was provided in "grok_pattern.txt", before monitoring via ELK to parse log message which contains lag state and other required informations like topic or consumer details in Logstash)
Extras: Received lag state can be regularly monitored using ELK. Logstash "Grok" plugin can be utilized to parse received logs in Logstash. Just "Logger" configuration is needed in order to send printed logs directly into Logstash!
Contact with me: