Red Hat AMQ Streams Architecture

Prerequisite

Clean AMQ Streams Data

  • make sure to stop all kafka & zookeeper server

    • by ctrl+c in your terminal or

    • call kafka-server-stop.sh for kafka broker and call zookeeper-server-stop.sh for zookeeper

  • clear zookeeper & kafka data

Generate the certificates

  • create certificate for enable ssl connection to amq streams cluster

  • review generate.sh and run for create certificate

  • example result of generate.sh (path ~/amq-streams-2022/2-amq-streams-architecture/ssl/keys, don't worry about warning in result!).

Start ZooKeeper cluster

First before we start Kafka, we have to start ZooKeeper cluster. We will use 3 node cluster. Start the 3 Zookeeper nodes by running these 3 scripts in different terminals. Don't worry about error after run zookeeper-0.sh (error about QuorumPeer):

  • Look at the Zookeeper config files in ./configs/zookeeper/, review zookeeper-1.properties, zookeeper-2.properties, zookeeper-3.properties

    • Look at the ensemble configuration in the Zookeeper properties files

    • Check the content of the Zookeeper data dirs in /tmp

    • Notice the myid file which needs to be created before starting Zookeeper with the node ID

    • client port

    • zookeeper server list (server.1,server.2,server.3)

    • client to server security config (this lab use 'SASL')

    • server to server security config (this lab use 'SASL')

  • review jaas.config for authentication information

  • run start zookeeper command in different terminal (1 shell script 1 terminal)

    example result

Start Kafka cluster

We will use 3 node Kafka cluster Start the 3 Kafka nodes by running these 3 scripts in different terminals:

  • Look at the Kafka configuration files in ./configs/kafka/, review server-0.properties, server-1.properties, server-2.properties

    • broker.id

    • listeners, advertised listeners, protocols

    • Zookeeper config

    • Look at the data dir in /tmp

    • Look at the tools in ./kafka/bin

    • SASL, SSL (server to server, kafka to kafka & zookeeper to kafka use SASL Plaintext in this lab)

  • review jaas.config for authentication information

  • run start kafka broker command in different terminal

    example result

Zookeeper

Show what Kafka does in Zookeeper

  • open new terminal

  • Find and notice the ZK JAR files in ./kafka/libs and ./kafka/bin - Zookeeper is integrated into Kafka distribution (see in 2-amq-streams-architecture/)

  • Start the ZK client

  • Browse through the Zookeeper nodes with below command

    example result

  • Exit from zookeeper-shell with ctrl+c

  • Try to do netcat dump with connected brokers

    example result

  • Kill broker 2 (type ctrl+c in kafka broker terminal #3) and do the netcat again to see how it disappeared

    example result after kill broker 2

  • start broker 2 again with command

Basics Operation with Cluster

Create topic

example result

Check the created topic

example result

Notice the distribution of leaders and the ISR replicas.

Send some messages

  • Send at least 10 messages (e.g. Message 1, Message 2 etc. to be able to notice the ordering later), exit command with ctrl+c

    example result

Consume messages

  • Read from the whole topic

    example result

  • Notice how the messages are out of order. And check how nicely ordered they are in a single partition. exit old console consumer and call it again and select only partition 0.

    example result

  • Reading from a particular offset (try to change partition and offset to test this command)

    example result

Replication

Test Broker crash

  • View topic description with the leaders and new ISR

    example result

  • Kill broker 2 by ctrl+c in kafka terminal #2 (broker start with 0,1,2)

  • Look again at the topic description with the leaders which changed and new ISR

    example result (partition 2 change leader to broker 0)

Consume messages

  • Try to consume the messages again to confirm that replication worked and that the messages are still in the topic!

    example result

Send some new messages

example result

Start the broker 2 again

  • start broker 2 again with command

  • Look again at the topic description with the leaders which changed and new ISR

    example result (broker 2 comeback, but leader not change)

  • Leadership didn't changed, but all replicas are again ISR

  • Try to consume the messages again.

PreviousIntroduction to Red Hat AMQ StreamsNextConsumer & Producer

Last updated