kafka_monitor

KafkaMonitor

the design and implementation reference to kt and kafkatools

Install

go get github.com/smallnest/kafka_monitor

Run

# change the below options with your environments
informEmail=your_email@xxxxxx.com
smtpHost=xxxxxx
smtpPort=25
smtpUser=xxxxxx@xxxxxx.com
smtpPassword=xxxxxx

brokers=10.77.11.1:9092,10.77.11.2:9092,10.77.11.3:9092,10.77.11.4:9092
version=0.9.0.1
zkAddr=10.77.0.1:2181,10.77.0.2:2181,10.77.0.3:2181,10.77.0.4:2181,10.77.0.5:2181

./kafka_monitor -brokers ${brokers} -version ${version} -email=${informEmail} -smtp.host ${smtpHost} -smtp.port ${smtpPort) -smtp.user ${smtpUser}  -smtp.password ${smtpPassword} -zkAddr ${zkAddr} -basePath "/kafka" 

Output

Once some partitions are not writable, down or some consumers don't work, an email will be sent to your email with details.

Normal kafka cluster info is output in console:

Brokers: 10.77.11.1:9092,10.77.11.2:9092,10.77.11.4:9092:10.77.11.3:9092
Version: 0.9.0.1
Topic: products, Group: default, Partitions: 32

+-----------+-------------------+--------+---------------+---------------+
| PARTITION |  LEADER ADDRESS   | LEADER |   REPLICAS    |      ISR      |
+-----------+-------------------+--------+---------------+---------------+
| 0         | 10.77.11.1:9092  | 1    | [1 2 3] | [1 2 3] |
| 1         | 10.77.11.2:9092  | 2    | [1 2 4] | [1 2 4] |
| 2         | 10.77.11.4:9092 | 4    | [2 4 3] | [2 4 3] |
| 3         | 10.77.11.3:9092 | 3    | [1 4 3] | [1 4 3] |
| 4         | 10.77.11.1:9092  | 1    | [1 2 4] | [1 2 4] |
| 5         | 10.77.11.2:9092  | 2    | [2 4 3] | [2 4 3] |
| 6         | 10.77.11.4:9092 | 4    | [1 4 3] | [1 4 3] |
| 7         | 10.77.11.3:9092 | 3    | [1 2 3] | [1 2 3] |
| 8         | 10.77.11.1:9092  | 1    | [1 4 3] | [1 4 3] |
| 9         | 10.77.11.2:9092  | 2    | [1 2 3] | [1 2 3] |
| 10        | 10.77.11.4:9092 | 4    | [1 2 4] | [1 2 4] |
| 11        | 10.77.11.3:9092 | 3    | [2 4 3] | [2 4 3] |
| 12        | 10.77.11.1:9092  | 1    | [1 2 3] | [1 2 3] |
| 13        | 10.77.11.2:9092  | 2    | [1 2 4] | [1 2 4] |
| 14        | 10.77.11.4:9092 | 4    | [2 4 3] | [2 4 3] |
| 15        | 10.77.11.3:9092 | 3    | [1 4 3] | [1 4 3] |
| 16        | 10.77.11.1:9092  | 1    | [1 2 4] | [1 2 4] |
| 17        | 10.77.11.2:9092  | 2    | [2 4 3] | [2 4 3] |
| 18        | 10.77.11.4:9092 | 4    | [1 4 3] | [1 4 3] |
| 19        | 10.77.11.3:9092 | 3    | [1 2 3] | [1 2 3] |
| 20        | 10.77.11.1:9092  | 1    | [1 4 3] | [1 4 3] |
| 21        | 10.77.11.2:9092  | 2    | [1 2 3] | [1 2 3] |
| 22        | 10.77.11.4:9092 | 4    | [1 2 4] | [1 2 4] |
| 23        | 10.77.11.3:9092 | 3    | [2 4 3] | [2 4 3] |
| 24        | 10.77.11.1:9092  | 1    | [1 2 3] | [1 2 3] |
| 25        | 10.77.11.2:9092  | 2    | [1 2 4] | [1 2 4] |
| 26        | 10.77.11.4:9092 | 4    | [2 4 3] | [2 4 3] |
| 27        | 10.77.11.3:9092 | 3    | [1 4 3] | [1 4 3] |
| 28        | 10.77.11.1:9092  | 1    | [1 2 4] | [1 2 4] |
| 29        | 10.77.11.2:9092  | 2    | [2 4 3] | [2 4 3] |
| 30        | 10.77.11.4:9092 | 4    | [1 4 3] | [1 4 3] |
| 31        | 10.77.11.3:9092 | 3    | [1 2 3] | [1 2 3] |
+-----------+-------------------+--------+---------------+---------------+


+-----------+------------+--------------+-----+
| PARTITION | END OF LOG | GROUP OFFSET | LAG |
+-----------+------------+--------------+-----+
| 0         | 1106509557 | 1106509542   | 15  |
| 1         | 1106517793 | 1106517779   | 14  |
| 2         | 1106573067 | 1106573047   | 20  |
| 3         | 1106531253 | 1106531253   | 0   |
| 4         | 1106529394 | 1106529385   | 9   |
| 5         | 11065504 | 1106550146   | 4   |
| 6         | 1106467253 | 1106467248   | 5   |
| 7         | 1106479964 | 1106479964   | 0   |
| 8         | 1106464607 | 1106464594   | 13  |
| 9         | 1106503601 | 1106503589   | 12  |
| 10        | 1106543568 | 1106543552   | 16  |
| 11        | 1106558780 | 1106558768   | 12  |
| 12        | 1106494251 | 1106494238   | 13  |
| 13        | 1106561496 | 1106561490   | 6   |
| 14        | 1106521678 | 1106521673   | 5   |
| 15        | 1106569720 | 1106569686   | 34  |
| 16        | 1106573067 | 1106573049   | 18  |
| 17        | 1106521317 | 1106521305   | 12  |
| 18        | 1106534494 | 1106534484   | 10  |
| 19        | 1106481325 | 1106481268   | 57  |
| 20        | 1106530486 | 1106530465   | 21  |
| 21        | 1106500650 | 1106500624   | 26  |
| 22        | 1106552135 | 1106552113   | 22  |
| 23        | 1106551799 | 1106551771   | 28  |
| 24        | 1106564761 | 1106564750   | 11  |
| 25        | 1106537078 | 1106537064   | 14  |
| 26        | 1106573795 | 1106573781   | 14  |
| 27        | 1106493569 | 1106493554   | 15  |
| 28        | 1106500029 | 1106500015   | 14  |
| 29        | 1106528095 | 1106528078   | 17  |
| 30        | 1106493651 | 1106493629   | 22  |
| 31        | 1106562315 | 1106562312   | 3   |
+-----------+------------+--------------+-----+
|                          TOTAL        | 482 |
+-----------+------------+--------------+-----+