Monitoring

Prerequisite

  • Setup Red Hat AMQ Streams Lab

  • Stop all server in previous lab

    • type ctrl+c in each terminal (stop kafka before stop zookeeper)

    • check kafka broker and zookeeper process with jps command

      jps

  • clear old data in previous lab

    rm -rf /tmp/zookeep*
rm -rf /tmp/kaf*

JMX Exporter

JMX Library

  • verify jmx libraries in /libs path such as jmx_prometheus_javaagent-xxxx.redhat-xxx.jar

    cd ~/amq-streams-2022/4-management/kafka/libs
ll jmx*

    example result

    -rw-rw-r--. 1 student student 469023 Nov 14 05:04 jmx_prometheus_javaagent-0.16.1.redhat-00001.jar

JMX Zookeeper

  • create zookeeper prometheus config file such as zookeeper.yml and save file in ~/amq-streams-2022/4-management/kafka/config of zookeeper server node.

    lowercaseOutputName: true
lowercaseOutputLabelNames: true
...
rules:
# Below rule applies for Zookeeper Cluster having multiple ZK nodes
# org.apache.ZooKeeperService:name0=*,name3=Connections,name1=*,name2=*,name4=*,name5=*
- pattern: "org.apache.ZooKeeperService<name0=(.+), name1=replica.(\\d+), name2=(\\w+), name3=Connections, name4=(.+), name5=(.+)><>([^:]+)"
name: zookeeper_connections_$6
labels:
  server_name: "$1"
  server_id: $2
  client_address: "$4"
  connection_id: "$5"
  member_type: "$3"
...

  • for zookeeper, edit ~/amq-streams-2022/4-management/kafka/bin/zookeeper-server-start.sh by add config to load jmxagent jar file and set configuration to file zookeeper.yml in previous step such as

    #add below command for start jmx exporter at port 7075
#set path to zookeeper.yml
export EXTRA_ARGS=$EXTRA_ARGS' -javaagent:/home/student/amq-streams-2022/4-management/kafka/libs/jmx_prometheus_javaagent-0.16.1.redhat-00001.jar=7075:/home/student/amq-streams-2022/4-management/kafka/config/zookeeper.yml'

exec $base_dir/kafka-run-class.sh $EXTRA_ARGS org.apache.zookeeper.server.quorum.QuorumPeerMain "$@"

  • start zookeeper

    cd ~/amq-streams-2022/4-management
./kafka/bin/zookeeper-server-start.sh ./kafka/config/zookeeper.properties

  • check port 7075 start with command, open new terminal

    netstat -tnlp

    example result

    ...
tcp6       0      0 :::7075                 :::*                    LISTEN      16617/java
...

  • call curl to http://localhost:7075/metrics for check metrics work!

    curl http://localhost:7075/metrics

    example result

    ...
# TYPE jmx_exporter_build_info gauge
jmx_exporter_build_info{version="0.16.1.redhat-00001",name="jmx_prometheus_javaagent",} 1.0
# HELP jmx_config_reload_failure_created Number of times configuration have failed to be reloaded.
# TYPE jmx_config_reload_failure_created gauge
jmx_config_reload_failure_created 1.668411562813E9
# HELP jmx_config_reload_success_created Number of times configuration have successfully been reloaded.
# TYPE jmx_config_reload_success_created gauge
jmx_config_reload_success_created 1.668411562812E9
# HELP jvm_memory_pool_allocated_bytes_created Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.
# TYPE jvm_memory_pool_allocated_bytes_created gauge
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'profiled nmethods'",} 1.668411563609E9
jvm_memory_pool_allocated_bytes_created{pool="G1 Old Gen",} 1.668411563616E9
jvm_memory_pool_allocated_bytes_created{pool="G1 Eden Space",} 1.668411563616E9
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-profiled nmethods'",} 1.668411563616E9
jvm_memory_pool_allocated_bytes_created{pool="G1 Survivor Space",} 1.668411563616E9
jvm_memory_pool_allocated_bytes_created{pool="Compressed Class Space",} 1.668411563616E9
jvm_memory_pool_allocated_bytes_created{pool="Metaspace",} 1.668411563616E9
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-nmethods'",} 1.668411563616E9

JMX Kafka

  • create kafka broker prometheus config file such as kafka_broker.yml and save file in /kafka/config of kafka broker node.

    ...
#startDelaySeconds: 120
lowercaseOutputName: true
lowercaseOutputLabelNames: true
...
rules:
# This is by far the biggest contributor to the number of sheer metrics being produced.
# Always keep it on the top for the case of probability when so many metrics will hit the first condition and exit.
# "kafka.cluster:type=*, name=*, topic=*, partition=*"
# "kafka.log:type=*,name=*, topic=*, partition=*"
- pattern: kafka.(\w+)<type=(.+), name=(.+), topic=(.+), partition=(.+)><>Value
  name: kafka_$1_$2_$3
  type: GAUGE
  labels:
      topic: "$4"
      partition: "$5"
...

  • for kafka broker, edit ~/amq-streams-2022/4-management/kafka/bin/kafka-server-start.sh by add config to load jmxagent jar file and set configuration to file kafka_broker.yml in previous step such as

    #add below command for start jmx exporter at port 7076
#set path to kafka_broker.yml
export KAFKA_OPTS=$KAFKA_OPTS' -javaagent:/home/student/amq-streams-2022/4-management/kafka/libs/jmx_prometheus_javaagent-0.16.1.redhat-00001.jar=7076:/home/student/amq-streams-2022/4-management/kafka/config/kafka_broker.yml'

exec $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka "$@"

  • start kafka broker

    cd ~/amq-streams-2022/4-management
./kafka/bin/kafka-server-start.sh ./kafka/config/server.properties

  • check port 7076 start with command, open new terminal

    netstat -tnlp

    example result

    ...
tcp6       0      0 :::7076                 :::*                    LISTEN      16617/java
...

  • call curl to http://localhost:7076/metrics for check metrics work!

    curl http://localhost:7076/metrics

    example result

    ...
# TYPE jmx_config_reload_success_created gauge
jmx_config_reload_success_created 1.668413540288E9
# HELP jvm_memory_pool_allocated_bytes_created Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.
# TYPE jvm_memory_pool_allocated_bytes_created gauge
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'profiled nmethods'",} 1.668413541537E9
jvm_memory_pool_allocated_bytes_created{pool="G1 Old Gen",} 1.66841354154E9
jvm_memory_pool_allocated_bytes_created{pool="G1 Eden Space",} 1.66841354154E9
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-profiled nmethods'",} 1.66841354154E9
jvm_memory_pool_allocated_bytes_created{pool="G1 Survivor Space",} 1.66841354154E9
jvm_memory_pool_allocated_bytes_created{pool="Compressed Class Space",} 1.66841354154E9
jvm_memory_pool_allocated_bytes_created{pool="Metaspace",} 1.66841354154E9
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-nmethods'",} 1.66841354154E9

Kafka Exporter

Prepare Topic Data

  • List the topics using Kafka, open new terminal and run command

    cd ~/amq-streams-2022/4-management
./kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

  • no topic show in terminal

  • create sample new topic which we will use

    ./kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic my-topic --partitions 3 --replication-factor 1

    result of create topic command

    Created topic my-topic.

  • List the topics again to see it was created.

    ./kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

  • Describe the topic to see more details:

    ./kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --topic my-topic --describe

  • start consumer, open new terminal and run command

    ./kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --from-beginning --group monitor

  • Start the console producer for create and send message to topic

    cd ~/amq-streams-2022/4-management
./kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic

  • Wait until it is ready (it should show >). and send 3 messages.

    >a
>b
>c
>

  • stop consumer console by type ctrl+c

  • back to producer console and submit 3 more data

    >a
>b
>c
>d
>e
>f
>

  • stop producer console

  • List the topics again to see '__consumer_offsets' was created.

    ./kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

    example result

    __consumer_offsets
my-topic

  • monitor consumer groups:

    ./kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --all-groups --list

  • and describe them: see current-offset, last-offset, lag

    ./kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --all-groups --describe

    example result

    Consumer group 'monitor' has no active members.

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
monitor         my-topic        0          1               3               2               -               -               -
monitor         my-topic        1          1               1               0               -               -               -
monitor         my-topic        2          1               2               1               -               -               -

Run Kafka Exporter

  • run kafka exporter

    cd ~/amq-streams-2022/4-management
./kafka/bin/kafka_exporter --kafka.version=3.2.3 --kafka.server=localhost:9092

    example result

    I1114 09:30:43.593105   74303 kafka_exporter.go:792] Starting kafka_exporter (version=1.6.0, branch=HEAD, revision=c021e94dfb808e642d41064c6550cbba87fe30c6)
I1114 09:30:43.601413   74303 kafka_exporter.go:963] Listening on HTTP :9308

  • call curl to http://localhost:9308/metrics for check metrics work!

    curl http://localhost:9308/metrics

    example result

    ...
# TYPE kafka_consumergroup_lag_sum gauge
kafka_consumergroup_lag_sum{consumergroup="monitor",topic="my-topic"} 3
...
# TYPE kafka_topic_partition_current_offset gauge
kafka_topic_partition_current_offset{partition="0",topic="__consumer_offsets"} 0
kafka_topic_partition_current_offset{partition="0",topic="my-topic"} 3
kafka_topic_partition_current_offset{partition="1",topic="__consumer_offsets"} 0
kafka_topic_partition_current_offset{partition="1",topic="my-topic"} 1
...
kafka_topic_partition_current_offset{partition="2",topic="my-topic"} 2
...
kafka_topic_partition_oldest_offset{partition="0",topic="__consumer_offsets"} 0
kafka_topic_partition_oldest_offset{partition="0",topic="my-topic"} 0
kafka_topic_partition_oldest_offset{partition="1",topic="__consumer_offsets"} 0
kafka_topic_partition_oldest_offset{partition="1",topic="my-topic"} 0
...
kafka_topic_partitions{topic="my-topic"} 3
...
PreviousCruise ControlNextAuthentication & Authorization

Last updated