Monitoring

• Monitor Red Hat AMQ Streams

  • Prerequisite

  • JMX Exporter

    • JMX Library

    • JMX Zookeeper

    • JMX Kafka

  • Kafka Exporter

    • Prepare Topic Data

    • Run Kafka Exporter

Prerequisite

• Setup Red Hat AMQ Streams Lab

• Stop all server in previous lab

  • type ctrl+c in each terminal (stop kafka before stop zookeeper)

  • check kafka broker and zookeeper process with jps command

    jps

• clear old data in previous lab

  rm -rf /tmp/zookeep*
  rm -rf /tmp/kaf*

JMX Exporter

JMX Library

• verify jmx libraries in /libs path such as jmx_prometheus_javaagent-xxxx.redhat-xxx.jar

  cd ~/amq-streams-2022/4-management/kafka/libs
  ll jmx*

  example result

  -rw-rw-r--. 1 student student 469023 Nov 14 05:04 jmx_prometheus_javaagent-0.16.1.redhat-00001.jar

JMX Zookeeper

• create zookeeper prometheus config file such as zookeeper.yml and save file in ~/amq-streams-2022/4-management/kafka/config of zookeeper server node.

  lowercaseOutputName: true
  lowercaseOutputLabelNames: true
  ...
  rules:
  # Below rule applies for Zookeeper Cluster having multiple ZK nodes
  # org.apache.ZooKeeperService:name0=*,name3=Connections,name1=*,name2=*,name4=*,name5=*
  - pattern: "org.apache.ZooKeeperService<name0=(.+), name1=replica.(\\d+), name2=(\\w+), name3=Connections, name4=(.+), name5=(.+)><>([^:]+)"
  name: zookeeper_connections_$6
  labels:
    server_name: "$1"
    server_id: $2
    client_address: "$4"
    connection_id: "$5"
    member_type: "$3"
  ...

• for zookeeper, edit ~/amq-streams-2022/4-management/kafka/bin/zookeeper-server-start.sh by add config to load jmxagent jar file and set configuration to file zookeeper.yml in previous step such as

  #add below command for start jmx exporter at port 7075
  #set path to zookeeper.yml
  export EXTRA_ARGS=$EXTRA_ARGS' -javaagent:/home/student/amq-streams-2022/4-management/kafka/libs/jmx_prometheus_javaagent-0.16.1.redhat-00001.jar=7075:/home/student/amq-streams-2022/4-management/kafka/config/zookeeper.yml'
  
  exec $base_dir/kafka-run-class.sh $EXTRA_ARGS org.apache.zookeeper.server.quorum.QuorumPeerMain "$@"

• start zookeeper

  cd ~/amq-streams-2022/4-management
  ./kafka/bin/zookeeper-server-start.sh ./kafka/config/zookeeper.properties

• check port 7075 start with command, open new terminal

  netstat -tnlp

  example result

  ...
  tcp6       0      0 :::7075                 :::*                    LISTEN      16617/java
  ...

• call curl to http://localhost:7075/metrics for check metrics work!

  curl http://localhost:7075/metrics

  example result

  ...
  # TYPE jmx_exporter_build_info gauge
  jmx_exporter_build_info{version="0.16.1.redhat-00001",name="jmx_prometheus_javaagent",} 1.0
  # HELP jmx_config_reload_failure_created Number of times configuration have failed to be reloaded.
  # TYPE jmx_config_reload_failure_created gauge
  jmx_config_reload_failure_created 1.668411562813E9
  # HELP jmx_config_reload_success_created Number of times configuration have successfully been reloaded.
  # TYPE jmx_config_reload_success_created gauge
  jmx_config_reload_success_created 1.668411562812E9
  # HELP jvm_memory_pool_allocated_bytes_created Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.
  # TYPE jvm_memory_pool_allocated_bytes_created gauge
  jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'profiled nmethods'",} 1.668411563609E9
  jvm_memory_pool_allocated_bytes_created{pool="G1 Old Gen",} 1.668411563616E9
  jvm_memory_pool_allocated_bytes_created{pool="G1 Eden Space",} 1.668411563616E9
  jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-profiled nmethods'",} 1.668411563616E9
  jvm_memory_pool_allocated_bytes_created{pool="G1 Survivor Space",} 1.668411563616E9
  jvm_memory_pool_allocated_bytes_created{pool="Compressed Class Space",} 1.668411563616E9
  jvm_memory_pool_allocated_bytes_created{pool="Metaspace",} 1.668411563616E9
  jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-nmethods'",} 1.668411563616E9

JMX Kafka

• create kafka broker prometheus config file such as kafka_broker.yml and save file in /kafka/config of kafka broker node.

  ...
  #startDelaySeconds: 120
  lowercaseOutputName: true
  lowercaseOutputLabelNames: true
  ...
  rules:
  # This is by far the biggest contributor to the number of sheer metrics being produced.
  # Always keep it on the top for the case of probability when so many metrics will hit the first condition and exit.
  # "kafka.cluster:type=*, name=*, topic=*, partition=*"
  # "kafka.log:type=*,name=*, topic=*, partition=*"
  - pattern: kafka.(\w+)<type=(.+), name=(.+), topic=(.+), partition=(.+)><>Value
    name: kafka_$1_$2_$3
    type: GAUGE
    labels:
        topic: "$4"
        partition: "$5"
  ...

• for kafka broker, edit ~/amq-streams-2022/4-management/kafka/bin/kafka-server-start.sh by add config to load jmxagent jar file and set configuration to file kafka_broker.yml in previous step such as

  #add below command for start jmx exporter at port 7076
  #set path to kafka_broker.yml
  export KAFKA_OPTS=$KAFKA_OPTS' -javaagent:/home/student/amq-streams-2022/4-management/kafka/libs/jmx_prometheus_javaagent-0.16.1.redhat-00001.jar=7076:/home/student/amq-streams-2022/4-management/kafka/config/kafka_broker.yml'
  
  exec $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka "$@"

• start kafka broker

  cd ~/amq-streams-2022/4-management
  ./kafka/bin/kafka-server-start.sh ./kafka/config/server.properties

• check port 7076 start with command, open new terminal

  netstat -tnlp

  example result

  ...
  tcp6       0      0 :::7076                 :::*                    LISTEN      16617/java
  ...

• call curl to http://localhost:7076/metrics for check metrics work!

  curl http://localhost:7076/metrics

  example result

  ...
  # TYPE jmx_config_reload_success_created gauge
  jmx_config_reload_success_created 1.668413540288E9
  # HELP jvm_memory_pool_allocated_bytes_created Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.
  # TYPE jvm_memory_pool_allocated_bytes_created gauge
  jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'profiled nmethods'",} 1.668413541537E9
  jvm_memory_pool_allocated_bytes_created{pool="G1 Old Gen",} 1.66841354154E9
  jvm_memory_pool_allocated_bytes_created{pool="G1 Eden Space",} 1.66841354154E9
  jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-profiled nmethods'",} 1.66841354154E9
  jvm_memory_pool_allocated_bytes_created{pool="G1 Survivor Space",} 1.66841354154E9
  jvm_memory_pool_allocated_bytes_created{pool="Compressed Class Space",} 1.66841354154E9
  jvm_memory_pool_allocated_bytes_created{pool="Metaspace",} 1.66841354154E9
  jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-nmethods'",} 1.66841354154E9

Kafka Exporter

Prepare Topic Data

• List the topics using Kafka, open new terminal and run command

  cd ~/amq-streams-2022/4-management
  ./kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

• no topic show in terminal

• create sample new topic which we will use

  ./kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic my-topic --partitions 3 --replication-factor 1

  result of create topic command

  Created topic my-topic.

• List the topics again to see it was created.

  ./kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

• Describe the topic to see more details:

  ./kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --topic my-topic --describe

• start consumer, open new terminal and run command

  ./kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --from-beginning --group monitor

• Start the console producer for create and send message to topic

  cd ~/amq-streams-2022/4-management
  ./kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic

• Wait until it is ready (it should show >). and send 3 messages.

  >a
  >b
  >c
  >

• stop consumer console by type ctrl+c

• back to producer console and submit 3 more data

  >a
  >b
  >c
  >d
  >e
  >f
  >

• stop producer console

• List the topics again to see '__consumer_offsets' was created.

  ./kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

  example result

  __consumer_offsets
  my-topic

• monitor consumer groups:

  ./kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --all-groups --list

• and describe them: see current-offset, last-offset, lag

  ./kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --all-groups --describe

  example result

  Consumer group 'monitor' has no active members.
  
  GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
  monitor         my-topic        0          1               3               2               -               -               -
  monitor         my-topic        1          1               1               0               -               -               -
  monitor         my-topic        2          1               2               1               -               -               -

Run Kafka Exporter

• run kafka exporter

  cd ~/amq-streams-2022/4-management
  ./kafka/bin/kafka_exporter --kafka.version=3.2.3 --kafka.server=localhost:9092

  example result

  I1114 09:30:43.593105   74303 kafka_exporter.go:792] Starting kafka_exporter (version=1.6.0, branch=HEAD, revision=c021e94dfb808e642d41064c6550cbba87fe30c6)
  I1114 09:30:43.601413   74303 kafka_exporter.go:963] Listening on HTTP :9308

• call curl to http://localhost:9308/metrics for check metrics work!

  curl http://localhost:9308/metrics

  example result

  ...
  # TYPE kafka_consumergroup_lag_sum gauge
  kafka_consumergroup_lag_sum{consumergroup="monitor",topic="my-topic"} 3
  ...
  # TYPE kafka_topic_partition_current_offset gauge
  kafka_topic_partition_current_offset{partition="0",topic="__consumer_offsets"} 0
  kafka_topic_partition_current_offset{partition="0",topic="my-topic"} 3
  kafka_topic_partition_current_offset{partition="1",topic="__consumer_offsets"} 0
  kafka_topic_partition_current_offset{partition="1",topic="my-topic"} 1
  ...
  kafka_topic_partition_current_offset{partition="2",topic="my-topic"} 2
  ...
  kafka_topic_partition_oldest_offset{partition="0",topic="__consumer_offsets"} 0
  kafka_topic_partition_oldest_offset{partition="0",topic="my-topic"} 0
  kafka_topic_partition_oldest_offset{partition="1",topic="__consumer_offsets"} 0
  kafka_topic_partition_oldest_offset{partition="1",topic="my-topic"} 0
  ...
  kafka_topic_partitions{topic="my-topic"} 3
  ...