Red Hat AMQ Streams Architecture
Last updated
Last updated
make sure to stop all kafka & zookeeper server
by ctrl+c in your terminal or
call kafka-server-stop.sh for kafka broker and call zookeeper-server-stop.sh for zookeeper
clear zookeeper & kafka data
create certificate for enable ssl connection to amq streams cluster
example result of generate.sh (path ~/amq-streams-2022/2-amq-streams-architecture/ssl/keys, don't worry about warning in result!).
First before we start Kafka, we have to start ZooKeeper cluster. We will use 3 node cluster. Start the 3 Zookeeper nodes by running these 3 scripts in different terminals. Don't worry about error after run zookeeper-0.sh (error about QuorumPeer):
Look at the ensemble configuration in the Zookeeper properties files
Check the content of the Zookeeper data dirs in /tmp
Notice the myid file which needs to be created before starting Zookeeper with the node ID
client port
zookeeper server list (server.1,server.2,server.3)
client to server security config (this lab use 'SASL')
server to server security config (this lab use 'SASL')
run start zookeeper command in different terminal (1 shell script 1 terminal)
example result
We will use 3 node Kafka cluster Start the 3 Kafka nodes by running these 3 scripts in different terminals:
broker.id
listeners, advertised listeners, protocols
Zookeeper config
Look at the data dir in /tmp
Look at the tools in ./kafka/bin
SASL, SSL (server to server, kafka to kafka & zookeeper to kafka use SASL Plaintext in this lab)
run start kafka broker command in different terminal
example result
open new terminal
Find and notice the ZK JAR files in ./kafka/libs and ./kafka/bin - Zookeeper is integrated into Kafka distribution (see in 2-amq-streams-architecture/)
Start the ZK client
Browse through the Zookeeper nodes with below command
example result
Exit from zookeeper-shell with ctrl+c
Try to do netcat dump with connected brokers
example result
Kill broker 2 (type ctrl+c in kafka broker terminal #3) and do the netcat again to see how it disappeared
example result after kill broker 2
start broker 2 again with command
example result
example result
Notice the distribution of leaders and the ISR replicas.
Send at least 10 messages (e.g. Message 1, Message 2 etc. to be able to notice the ordering later), exit command with ctrl+c
example result
Read from the whole topic
example result
Notice how the messages are out of order. And check how nicely ordered they are in a single partition. exit old console consumer and call it again and select only partition 0.
example result
Reading from a particular offset (try to change partition and offset to test this command)
example result
View topic description with the leaders and new ISR
example result
Kill broker 2 by ctrl+c in kafka terminal #2 (broker start with 0,1,2)
Look again at the topic description with the leaders which changed and new ISR
example result (partition 2 change leader to broker 0)
Try to consume the messages again to confirm that replication worked and that the messages are still in the topic!
example result
example result
start broker 2 again with command
Look again at the topic description with the leaders which changed and new ISR
example result (broker 2 comeback, but leader not change)
Leadership didn't changed, but all replicas are again ISR
Try to consume the messages again.
review and run for create certificate
Look at the Zookeeper config files in ./configs/zookeeper/, review , zookeeper-2.properties, zookeeper-3.properties
review for authentication information
Look at the Kafka configuration files in ./configs/kafka/, review , server-1.properties, server-2.properties
review for authentication information
