# Monitoring * [Monitor Red Hat AMQ Streams](#monitor-red-hat-amq-streams) * [Prerequisite](#prerequisite) * [JMX Exporter](#jmx-exporter) * [JMX Library](#jmx-library) * [JMX Zookeeper](#jmx-zookeeper) * [JMX Kafka](#jmx-kafka) * [Kafka Exporter](#kafka-exporter) * [Prepare Topic Data](#prepare-topic-data) * [Run Kafka Exporter](#run-kafka-exporter) ## Prerequisite * [Setup Red Hat AMQ Streams Lab](/amq-streams-training-2022/amq-streams-2022-training/setup.md) * Stop all server in previous lab * type ctrl+c in each terminal (stop kafka before stop zookeeper) * check kafka broker and zookeeper process with jps command ```bash jps ``` * clear old data in previous lab ```bash rm -rf /tmp/zookeep* rm -rf /tmp/kaf* ``` ## JMX Exporter ### JMX Library * verify jmx libraries in /libs path such as jmx\_prometheus\_javaagent-xxxx.redhat-xxx.jar ```bash cd ~/amq-streams-2022/4-management/kafka/libs ll jmx* ``` example result ```bash -rw-rw-r--. 1 student student 469023 Nov 14 05:04 jmx_prometheus_javaagent-0.16.1.redhat-00001.jar ``` ### JMX Zookeeper * create zookeeper prometheus config file such as [zookeeper.yml](https://github.com/chatapazar/amq-streams-2022/blob/master/4-management/kafka/config/zookeeper.yml) and save file in \~/amq-streams-2022/4-management/kafka/config of zookeeper server node. ```yml lowercaseOutputName: true lowercaseOutputLabelNames: true ... rules: # Below rule applies for Zookeeper Cluster having multiple ZK nodes # org.apache.ZooKeeperService:name0=*,name3=Connections,name1=*,name2=*,name4=*,name5=* - pattern: "org.apache.ZooKeeperService<>([^:]+)" name: zookeeper_connections_$6 labels: server_name: "$1" server_id: $2 client_address: "$4" connection_id: "$5" member_type: "$3" ... ``` * for zookeeper, edit \~/amq-streams-2022/4-management/kafka/bin/zookeeper-server-start.sh by add config to load jmxagent jar file and set configuration to file zookeeper.yml in previous step such as ```bash #add below command for start jmx exporter at port 7075 #set path to zookeeper.yml export EXTRA_ARGS=$EXTRA_ARGS' -javaagent:/home/student/amq-streams-2022/4-management/kafka/libs/jmx_prometheus_javaagent-0.16.1.redhat-00001.jar=7075:/home/student/amq-streams-2022/4-management/kafka/config/zookeeper.yml' exec $base_dir/kafka-run-class.sh $EXTRA_ARGS org.apache.zookeeper.server.quorum.QuorumPeerMain "$@" ``` * start zookeeper ```bash cd ~/amq-streams-2022/4-management ./kafka/bin/zookeeper-server-start.sh ./kafka/config/zookeeper.properties ``` * check port 7075 start with command, open new terminal ```bash netstat -tnlp ``` example result ```bash ... tcp6 0 0 :::7075 :::* LISTEN 16617/java ... ``` * call curl to for check metrics work! ```bash curl http://localhost:7075/metrics ``` example result ```bash ... # TYPE jmx_exporter_build_info gauge jmx_exporter_build_info{version="0.16.1.redhat-00001",name="jmx_prometheus_javaagent",} 1.0 # HELP jmx_config_reload_failure_created Number of times configuration have failed to be reloaded. # TYPE jmx_config_reload_failure_created gauge jmx_config_reload_failure_created 1.668411562813E9 # HELP jmx_config_reload_success_created Number of times configuration have successfully been reloaded. # TYPE jmx_config_reload_success_created gauge jmx_config_reload_success_created 1.668411562812E9 # HELP jvm_memory_pool_allocated_bytes_created Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously. # TYPE jvm_memory_pool_allocated_bytes_created gauge jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'profiled nmethods'",} 1.668411563609E9 jvm_memory_pool_allocated_bytes_created{pool="G1 Old Gen",} 1.668411563616E9 jvm_memory_pool_allocated_bytes_created{pool="G1 Eden Space",} 1.668411563616E9 jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-profiled nmethods'",} 1.668411563616E9 jvm_memory_pool_allocated_bytes_created{pool="G1 Survivor Space",} 1.668411563616E9 jvm_memory_pool_allocated_bytes_created{pool="Compressed Class Space",} 1.668411563616E9 jvm_memory_pool_allocated_bytes_created{pool="Metaspace",} 1.668411563616E9 jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-nmethods'",} 1.668411563616E9 ``` ### JMX Kafka * create kafka broker prometheus config file such as [kafka\_broker.yml](https://github.com/chatapazar/amq-streams-2022/blob/master/4-management/kafka/config/kafka_broker.yml) and save file in /kafka/config of kafka broker node. ```yml ... #startDelaySeconds: 120 lowercaseOutputName: true lowercaseOutputLabelNames: true ... rules: # This is by far the biggest contributor to the number of sheer metrics being produced. # Always keep it on the top for the case of probability when so many metrics will hit the first condition and exit. # "kafka.cluster:type=*, name=*, topic=*, partition=*" # "kafka.log:type=*,name=*, topic=*, partition=*" - pattern: kafka.(\w+)<>Value name: kafka_$1_$2_$3 type: GAUGE labels: topic: "$4" partition: "$5" ... ``` * for kafka broker, edit \~/amq-streams-2022/4-management/kafka/bin/kafka-server-start.sh by add config to load jmxagent jar file and set configuration to file kafka\_broker.yml in previous step such as ```bash #add below command for start jmx exporter at port 7076 #set path to kafka_broker.yml export KAFKA_OPTS=$KAFKA_OPTS' -javaagent:/home/student/amq-streams-2022/4-management/kafka/libs/jmx_prometheus_javaagent-0.16.1.redhat-00001.jar=7076:/home/student/amq-streams-2022/4-management/kafka/config/kafka_broker.yml' exec $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka "$@" ``` * start kafka broker ```bash cd ~/amq-streams-2022/4-management ./kafka/bin/kafka-server-start.sh ./kafka/config/server.properties ``` * check port 7076 start with command, open new terminal ```bash netstat -tnlp ``` example result ```bash ... tcp6 0 0 :::7076 :::* LISTEN 16617/java ... ``` * call curl to for check metrics work! ```bash curl http://localhost:7076/metrics ``` example result ```bash ... # TYPE jmx_config_reload_success_created gauge jmx_config_reload_success_created 1.668413540288E9 # HELP jvm_memory_pool_allocated_bytes_created Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously. # TYPE jvm_memory_pool_allocated_bytes_created gauge jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'profiled nmethods'",} 1.668413541537E9 jvm_memory_pool_allocated_bytes_created{pool="G1 Old Gen",} 1.66841354154E9 jvm_memory_pool_allocated_bytes_created{pool="G1 Eden Space",} 1.66841354154E9 jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-profiled nmethods'",} 1.66841354154E9 jvm_memory_pool_allocated_bytes_created{pool="G1 Survivor Space",} 1.66841354154E9 jvm_memory_pool_allocated_bytes_created{pool="Compressed Class Space",} 1.66841354154E9 jvm_memory_pool_allocated_bytes_created{pool="Metaspace",} 1.66841354154E9 jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-nmethods'",} 1.66841354154E9 ``` ## Kafka Exporter ### Prepare Topic Data * List the topics using Kafka, open new terminal and run command ```bash cd ~/amq-streams-2022/4-management ./kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list ``` * no topic show in terminal * create sample new topic which we will use ```bash ./kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic my-topic --partitions 3 --replication-factor 1 ``` result of create topic command ```bash Created topic my-topic. ``` * List the topics again to see it was created. ```bash ./kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list ``` * Describe the topic to see more details: ```bash ./kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --topic my-topic --describe ``` * start consumer, open new terminal and run command ```bash ./kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --from-beginning --group monitor ``` * Start the console producer for create and send message to topic ```bash cd ~/amq-streams-2022/4-management ./kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic ``` * Wait until it is ready (it should show `>`). and send 3 messages. ```bash >a >b >c > ``` * stop consumer console by type ctrl+c * back to producer console and submit 3 more data ```bash >a >b >c >d >e >f > ``` * stop producer console * List the topics again to see '\_\_consumer\_offsets' was created. ```bash ./kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list ``` example result ```bash __consumer_offsets my-topic ``` * monitor consumer groups: ```bash ./kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --all-groups --list ``` * and describe them: see current-offset, last-offset, lag ```bash ./kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --all-groups --describe ``` example result ```bash Consumer group 'monitor' has no active members. GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID monitor my-topic 0 1 3 2 - - - monitor my-topic 1 1 1 0 - - - monitor my-topic 2 1 2 1 - - - ``` ### Run Kafka Exporter * run kafka exporter ```bash cd ~/amq-streams-2022/4-management ./kafka/bin/kafka_exporter --kafka.version=3.2.3 --kafka.server=localhost:9092 ``` example result ```bash I1114 09:30:43.593105 74303 kafka_exporter.go:792] Starting kafka_exporter (version=1.6.0, branch=HEAD, revision=c021e94dfb808e642d41064c6550cbba87fe30c6) I1114 09:30:43.601413 74303 kafka_exporter.go:963] Listening on HTTP :9308 ``` * call curl to for check metrics work! ```bash curl http://localhost:9308/metrics ``` example result ```bash ... # TYPE kafka_consumergroup_lag_sum gauge kafka_consumergroup_lag_sum{consumergroup="monitor",topic="my-topic"} 3 ... # TYPE kafka_topic_partition_current_offset gauge kafka_topic_partition_current_offset{partition="0",topic="__consumer_offsets"} 0 kafka_topic_partition_current_offset{partition="0",topic="my-topic"} 3 kafka_topic_partition_current_offset{partition="1",topic="__consumer_offsets"} 0 kafka_topic_partition_current_offset{partition="1",topic="my-topic"} 1 ... kafka_topic_partition_current_offset{partition="2",topic="my-topic"} 2 ... kafka_topic_partition_oldest_offset{partition="0",topic="__consumer_offsets"} 0 kafka_topic_partition_oldest_offset{partition="0",topic="my-topic"} 0 kafka_topic_partition_oldest_offset{partition="1",topic="__consumer_offsets"} 0 kafka_topic_partition_oldest_offset{partition="1",topic="my-topic"} 0 ... kafka_topic_partitions{topic="my-topic"} 3 ... ``` --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://chatapazar.gitbook.io/amq-streams-training-2022/amq-streams-2022-training/monitor.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.