# Cruise Control * [Cruise Control](#cruise-control) * [Prerequisite](#prerequisite) * [Deploying the Cruise Control Metrics Reporter](#deploying-the-cruise-control-metrics-reporter) * [Prepare Test Topic](#prepare-test-topic) * [Enable Cruise Control](#enable-cruise-control) * [Test Auto Rebalance After Add Broker With Cruise-control](#test-auto-rebalance-after-add-broker-with-cruise-control) * [Stop Server](#stop-server) ## Prerequisite * [Setup Red Hat AMQ Streams Lab](/amq-streams-training-2022/amq-streams-2022-training/setup.md) * [Red Hat AMQ Streams Architecture](/amq-streams-training-2022/amq-streams-2022-training/architecture.md) * Stop all server in previous lab * type ctrl+c in each terminal (stop kafka before stop zookeeper) * check kafka broker and zookeeper process with jps command ```bash jps ``` * clear old data in previous lab ```bash rm -rf /tmp/zookeep* rm -rf /tmp/kaf* ``` ## Deploying the Cruise Control Metrics Reporter * In the Kafka configuration file [server-0.properties](https://github.com/chatapazar/amq-streams-2022/blob/master/2-amq-streams-architecture/configs/kafka/server-0.properties) configure the Cruise Control Metrics Reporter: * Add the CruiseControlMetricsReporter class to the metric.reporters configuration option. Do not remove any existing Metrics Reporters. (add at bottom of properties file) ```properties ... # add metric reporters metric.reporters=com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter cruise.control.metrics.topic.auto.create=true cruise.control.metrics.topic.num.partitions=1 cruise.control.metrics.topic.replication.factor=1 ``` * repeat with server-1.properties, server-2.properties in \~/amq-streams-2022/2-amq-streams-architecture/configs/kafka * start cluster * run start zookeeper command in different terminal (1 shell script 1 terminal) ```bash cd ~/amq-streams-2022/2-amq-streams-architecture/ ./scripts/zookeeper-0.sh ./scripts/zookeeper-1.sh ./scripts/zookeeper-2.sh ``` * run start kafka broker command in different terminal, start only broker 0 and broker 1 (we will start broker 2 later in this lab) ```bash cd ~/amq-streams-2022/2-amq-streams-architecture/ ./scripts/kafka-0.sh ./scripts/kafka-1.sh ``` * check zookeeper & kafka process (3 zookeeper & 2 kafka) ```bash [student@node1 ~]$ jps 1504 QuorumPeerMain 2034 QuorumPeerMain 4362 Jps 3819 Kafka 2557 QuorumPeerMain 3277 Kafka ``` or ```bash [student@node1 ~]$ jcmd 1504 org.apache.zookeeper.server.quorum.QuorumPeerMain /home/student/amq-streams-2022/2-amq-streams-architecture/scripts/../configs/zookeeper/zookeeper-1.properties 2034 org.apache.zookeeper.server.quorum.QuorumPeerMain /home/student/amq-streams-2022/2-amq-streams-architecture/scripts/../configs/zookeeper/zookeeper-2.properties 3819 kafka.Kafka /home/student/amq-streams-2022/2-amq-streams-architecture/scripts/../configs/kafka/server-1.properties 4380 jdk.jcmd/sun.tools.jcmd.JCmd 2557 org.apache.zookeeper.server.quorum.QuorumPeerMain /home/student/amq-streams-2022/2-amq-streams-architecture/scripts/../configs/zookeeper/zookeeper-3.properties 3277 kafka.Kafka /home/student/amq-streams-2022/2-amq-streams-architecture/scripts/../configs/kafka/server-0.properties ``` ## Prepare Test Topic * Create topic "demo" ```bash cd ~/amq-streams-2022/2-amq-streams-architecture/ ./kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic demo --partitions 10 --replication-factor 2 ``` example result ```bash Created topic demo. ``` * Note: replication factor can't larger than available broker. * Check Topic Description ```bash ./kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list ./kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic demo ``` example result, show partition of demo topic in broker 0 & broker 1 ```bash Topic: demo TopicId: gBQpzFJnRyGcvtFniabrVw PartitionCount: 10 ReplicationFactor: 2 Configs: segment.bytes=1073741824 Topic: demo Partition: 0 Leader: 1 Replicas: 1,0 Isr: 1,0 Topic: demo Partition: 1 Leader: 0 Replicas: 0,1 Isr: 0,1 Topic: demo Partition: 2 Leader: 1 Replicas: 1,0 Isr: 1,0 Topic: demo Partition: 3 Leader: 0 Replicas: 0,1 Isr: 0,1 Topic: demo Partition: 4 Leader: 1 Replicas: 1,0 Isr: 1,0 Topic: demo Partition: 5 Leader: 0 Replicas: 0,1 Isr: 0,1 Topic: demo Partition: 6 Leader: 1 Replicas: 1,0 Isr: 1,0 Topic: demo Partition: 7 Leader: 0 Replicas: 0,1 Isr: 0,1 Topic: demo Partition: 8 Leader: 1 Replicas: 1,0 Isr: 1,0 Topic: demo Partition: 9 Leader: 0 Replicas: 0,1 Isr: 0,1 ``` * load data to topic "demo" ```bash ./kafka/bin/kafka-producer-perf-test.sh \ --topic demo \ --throughput -1 \ --num-records 500000 \ --record-size 2048 \ --producer-props acks=all bootstrap.servers=localhost:9092,localhost:9093 ``` * run start kafka broker 2 in different terminal ```bash cd ~/amq-streams-2022/2-amq-streams-architecture/ ./scripts/kafka-2.sh ``` * check all zookeeper & kafka process (3 zookeeper & 3 kafka) ```bash [student@node1 ~]$ jps 1504 QuorumPeerMain 6225 Kafka 2034 QuorumPeerMain 6760 Jps 3819 Kafka 2557 QuorumPeerMain 3277 Kafka ``` * recheck topic information of "demo" don't change after start broker 2 ```bash cd ~/amq-streams-2022/2-amq-streams-architecture/ ./kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic demo ``` ## Enable Cruise Control * reviews cruise-control configuration in [cruisecontrol.properties](https://github.com/chatapazar/amq-streams-2022/blob/master/4-management/cruise-control/config/cruisecontrol.properties), Configure the properties used by Cruise Control and then start the Cruise Control server using the kafka-cruise-control-start.sh script. ```properties ... # The Kafka cluster to control. bootstrap.servers=localhost:9092 # The replication factor of Kafka metric sample store topic sample.store.topic.replication.factor=2 # The configuration for the BrokerCapacityConfigFileResolver (supports JBOD, non-JBOD, and heterogeneous CPU core capacities) #capacity.config.file=config/capacity.json #capacity.config.file=config/capacityCores.json capacity.config.file=config/capacityJBOD.json # The list of goals to optimize the Kafka cluster for with pre-computed proposals default.goals=com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.PotentialNwOutGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.TopicReplicaDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderReplicaDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderBytesInDistributionGoal # The list of supported goals goals=com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.PotentialNwOutGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.TopicReplicaDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderReplicaDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderBytesInDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.kafkaassigner.KafkaAssignerDiskUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.kafkaassigner.KafkaAssignerEvenRackAwareGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.PreferredLeaderElectionGoal # How often should the cached proposal be expired and recalculated if necessary proposal.expiration.ms=60000 # The zookeeper connect of the Kafka cluster zookeeper.connect=localhost:2181 ``` * start cruise-contral at port 9191 ```bash cd ~/amq-streams-2022/4-management/cruise-control ./kafka-cruise-control-start.sh config/cruisecontrol.properties 9191 ``` * check Auto-created topics of cruise-control ```bash cd ~/amq-streams-2022/2-amq-streams-architecture/ ./kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list --exclude-internal ``` example result, show new topic such as "\_\_CruiseControlMetrics", "\_\_KafkaCruiseControlPartitionMetricSamples" and "\_\_KafkaCruiseControlModelTrainingSamples" ```bash __CruiseControlMetrics __KafkaCruiseControlModelTrainingSamples __KafkaCruiseControlPartitionMetricSamples demo ``` * open new terminal, check cruise-control state ```bash curl 'http://localhost:9191/kafkacruisecontrol/state' ``` example result ```bash [student@node1 ~]$ curl 'http://localhost:9191/kafkacruisecontrol/state' MonitorState: {state: RUNNING(0.000% trained), NumValidWindows: (0/0) (NaN%), NumValidPartitions: 0/0 (0.000%), flawedPartitions: 0} ExecutorState: {state: NO_TASK_IN_PROGRESS} AnalyzerState: {isProposalReady: false, readyGoals: []} AnomalyDetectorState: {selfHealingEnabled:[], selfHealingDisabled:[DISK_FAILURE, BROKER_FAILURE, GOAL_VIOLATION, METRIC_ANOMALY, TOPIC_ANOMALY, MAINTENANCE_EVENT], selfHealingEnabledRatio:{DISK_FAILURE=0.0, BROKER_FAILURE=0.0, GOAL_VIOLATION=0.0, METRIC_ANOMALY=0.0, TOPIC_ANOMALY=0.0, MAINTENANCE_EVENT=0.0}, recentGoalViolations:[], recentBrokerFailures:[], recentMetricAnomalies:[], recentDiskFailures:[], recentTopicAnomalies:[], recentMaintenanceEvents:[], metrics:{meanTimeBetweenAnomalies:{GOAL_VIOLATION:0.00 milliseconds, BROKER_FAILURE:0.00 milliseconds, METRIC_ANOMALY:0.00 milliseconds, DISK_FAILURE:0.00 milliseconds, TOPIC_ANOMALY:0.00 milliseconds}, meanTimeToStartFix:0.00 milliseconds, numSelfHealingStarted:0, numSelfHealingFailedToStart:0, ongoingAnomalyDuration=0.00 milliseconds}, ongoingSelfHealingAnomaly:None, balancednessScore:100.000} ``` * review capacity limits for Kafka broker resources in [config/config/capacityJBOD.json](https://github.com/chatapazar/amq-streams-2022/blob/master/4-management/cruise-control/config/capacityJBOD.json) , To apply the same capacity limits to every broker monitored by Cruise Control, set capacity limits for broker ID -1. To set different capacity limits for individual brokers, specify each broker ID and its capacity configuration. ```json { "brokerCapacities":[ { "brokerId": "-1", "capacity": { "DISK": "100000", "CPU": "100", "NW_IN": "10000", "NW_OUT": "10000" }, "doc": "This is the default capacity. Capacity unit used for disk is in MB, cpu is in percentage, network throughput is in KB." }, { "brokerId": "0", "capacity": { "DISK": "100000", "CPU": "100", "NW_IN": "10000", "NW_OUT": "10000" }, "doc": "This overrides the capacity for broker 0." } ] } ``` ## Test Auto Rebalance After Add Broker With Cruise-control * Test Generating optimization proposals with dryrun. ```bash curl -v -X POST 'localhost:9191/kafkacruisecontrol/add_broker?brokerid=2' ``` example result, Review the optimization proposal contained in the response. ```bash * Trying ::1... * TCP_NODELAY set * Connected to localhost (::1) port 9191 (#0) > POST /kafkacruisecontrol/add_broker?brokerid=2 HTTP/1.1 > Host: localhost:9191 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 24 Nov 2022 06:17:17 GMT < Set-Cookie: JSESSIONID=node0vavr3i7abj7ilf5myhd9ixsb8.node0; Path=/ < Expires: Thu, 01 Jan 1970 00:00:00 GMT < User-Task-ID: 01d01442-230e-4f39-9514-5355239f99e1 < Content-Type: text/plain;charset=utf-8 < Cruise-Control-Version: 2.5.89.redhat-00006 < Cruise-Control-Commit_Id: 1e4e2dd8f14df6c409244c11c84c52038597ca99 < Content-Length: 13200 < Server: Jetty(9.4.45.v20220203-redhat-00001) < Optimization has 9 inter-broker replica(593 MB) moves, 0 intra-broker replica(0 MB) moves and 0 leadership moves with a cluster model of 1 recent windows and 100.000% of the partitions covered. Excluded Topics: []. Excluded Brokers For Leadership: []. Excluded Brokers For Replica Move: []. Counts: 3 brokers 149 replicas 4 topics. On-demand Balancedness Score Before (86.026) After(92.120). Provision Status: RIGHT_SIZED. [ 2 ms] Stats for ReplicaCapacityGoal(NO-ACTION): AVG:{cpu: 0.000 networkInbound: 370.684 networkOutbound: 185.337 disk: 656.680 potentialNwOut: 370.674 replicas:49.666666666666664 leaderReplicas:25.0 topicReplicas:12.416666666666666} MAX:{cpu: 0.000 networkInbound: 1112.053 networkOutbound: 556.011 disk: 985.028 potentialNwOut: 556.011 replicas:53 leaderReplicas:27 topicReplicas:22} MIN:{cpu: 0.000 networkInbound: 1112.053 networkOutbound: 556.011 disk: 0.000 potentialNwOut: 0.000 replicas:43 leaderReplicas:22 topicReplicas:0} STD:{cpu: 0.000 networkInbound: 0.000 networkOutbound: 0.000 disk: 464.343 potentialNwOut: 262.106 replicas:4.714045207910317 leaderReplicas:2.1602468994692865 topicReplicas:1.532064692570853 [ 2 ms] Stats for DiskCapacityGoal(NO-ACTION): AVG:{cpu: 0.000 networkInbound: 370.684 networkOutbound: 185.337 disk: 656.680 potentialNwOut: 370.674 replicas:49.666666666666664 leaderReplicas:25.0 topicReplicas:12.416666666666666} MAX:{cpu: 0.000 networkInbound: 1112.053 networkOutbound: 556.011 disk: 985.028 potentialNwOut: 556.011 replicas:53 leaderReplicas:27 topicReplicas:22} MIN:{cpu: 0.000 networkInbound: 1112.053 networkOutbound: 556.011 disk: 0.000 potentialNwOut: 0.000 replicas:43 leaderReplicas:22 topicReplicas:0} STD:{cpu: 0.000 networkInbound: 0.000 networkOutbound: 0.000 disk: 464.343 potentialNwOut: 262.106 replicas:4.714045207910317 leaderReplicas:2.1602468994692865 topicReplicas:1.532064692570853 [ 2 ms] Stats for NetworkInboundCapacityGoal(NO-ACTION): AVG:{cpu: 0.000 networkInbound: 370.684 networkOutbound: 185.337 disk: 656.680 potentialNwOut: 370.674 replicas:49.666666666666664 leaderReplicas:25.0 topicReplicas:12.416666666666666} MAX:{cpu: 0.000 networkInbound: 1112.053 networkOutbound: 556.011 disk: 985.028 potentialNwOut: 556.011 replicas:53 leaderReplicas:27 topicReplicas:22} MIN:{cpu: 0.000 networkInbound: 1112.053 networkOutbound: 556.011 disk: 0.000 potentialNwOut: 0.000 replicas:43 leaderReplicas:22 topicReplicas:0} STD:{cpu: 0.000 networkInbound: 0.000 networkOutbound: 0.000 disk: 464.343 potentialNwOut: 262.106 replicas:4.714045207910317 leaderReplicas:2.1602468994692865 topicReplicas:1.532064692570853 [ 1 ms] Stats for NetworkOutboundCapacityGoal(NO-ACTION): AVG:{cpu: 0.000 networkInbound: 370.684 networkOutbound: 185.337 disk: 656.680 potentialNwOut: 370.674 replicas:49.666666666666664 leaderReplicas:25.0 topicReplicas:12.416666666666666} MAX:{cpu: 0.000 networkInbound: 1112.053 networkOutbound: 556.011 disk: 985.028 potentialNwOut: 556.011 replicas:53 leaderReplicas:27 topicReplicas:22} MIN:{cpu: 0.000 networkInbound: 1112.053 networkOutbound: 556.011 disk: 0.000 potentialNwOut: 0.000 replicas:43 leaderReplicas:22 topicReplicas:0} STD:{cpu: 0.000 networkInbound: 0.000 networkOutbound: 0.000 disk: 464.343 potentialNwOut: 262.106 replicas:4.714045207910317 leaderReplicas:2.1602468994692865 topicReplicas:1.532064692570853 [ 1 ms] Stats for CpuCapacityGoal(NO-ACTION): AVG:{cpu: 0.000 networkInbound: 370.684 networkOutbound: 185.337 disk: 656.680 potentialNwOut: 370.674 replicas:49.666666666666664 leaderReplicas:25.0 topicReplicas:12.416666666666666} MAX:{cpu: 0.000 networkInbound: 1112.053 networkOutbound: 556.011 disk: 985.028 potentialNwOut: 556.011 replicas:53 leaderReplicas:27 topicReplicas:22} MIN:{cpu: 0.000 networkInbound: 1112.053 networkOutbound: 556.011 disk: 0.000 potentialNwOut: 0.000 replicas:43 leaderReplicas:22 topicReplicas:0} STD:{cpu: 0.000 networkInbound: 0.000 networkOutbound: 0.000 disk: 464.343 potentialNwOut: 262.106 replicas:4.714045207910317 leaderReplicas:2.1602468994692865 topicReplicas:1.532064692570853 [ 3 ms] Stats for ReplicaDistributionGoal(FIXED): AVG:{cpu: 0.000 networkInbound: 370.684 networkOutbound: 185.337 disk: 656.680 potentialNwOut: 370.674 replicas:49.666666666666664 leaderReplicas:25.0 topicReplicas:12.416666666666666} MAX:{cpu: 0.000 networkInbound: 1112.053 networkOutbound: 556.011 disk: 985.028 potentialNwOut: 556.011 replicas:52 leaderReplicas:27 topicReplicas:23} MIN:{cpu: 0.000 networkInbound: 1112.053 networkOutbound: 556.011 disk: 0.000 potentialNwOut: 0.000 replicas:45 leaderReplicas:22 topicReplicas:0} STD:{cpu: 0.000 networkInbound: 0.000 networkOutbound: 0.000 disk: 464.343 potentialNwOut: 262.106 replicas:3.2998316455372216 leaderReplicas:2.1602468994692865 topicReplicas:1.8438694748020148 [ 1 ms] Stats for PotentialNwOutGoal(NO-ACTION): AVG:{cpu: 0.000 networkInbound: 370.684 networkOutbound: 185.337 disk: 656.680 potentialNwOut: 370.674 replicas:49.666666666666664 leaderReplicas:25.0 topicReplicas:12.416666666666666} MAX:{cpu: 0.000 networkInbound: 1112.053 networkOutbound: 556.011 disk: 985.028 potentialNwOut: 556.011 replicas:52 leaderReplicas:27 topicReplicas:23} MIN:{cpu: 0.000 networkInbound: 1112.053 networkOutbound: 556.011 disk: 0.000 potentialNwOut: 0.000 replicas:45 leaderReplicas:22 topicReplicas:0} STD:{cpu: 0.000 networkInbound: 0.000 networkOutbound: 0.000 disk: 464.343 potentialNwOut: 262.106 replicas:3.2998316455372216 leaderReplicas:2.1602468994692865 topicReplicas:1.8438694748020148 [ 5 ms] Stats for DiskUsageDistributionGoal(VIOLATED): AVG:{cpu: 0.000 networkInbound: 370.684 networkOutbound: 185.337 disk: 656.680 potentialNwOut: 370.674 replicas:49.666666666666664 leaderReplicas:25.0 topicReplicas:12.416666666666666} MAX:{cpu: 0.000 networkInbound: 1112.053 networkOutbound: 556.011 disk: 687.773 potentialNwOut: 403.316 replicas:52 leaderReplicas:27 topicReplicas:23} MIN:{cpu: 0.000 networkInbound: 1112.053 networkOutbound: 556.011 disk: 595.351 potentialNwOut: 333.606 replicas:48 leaderReplicas:23 topicReplicas:0} STD:{cpu: 0.000 networkInbound: 0.000 networkOutbound: 0.000 disk: 43.368 potentialNwOut: 28.630 replicas:1.699673171197595 leaderReplicas:1.632993161855452 topicReplicas:0.7832093030221935 [ 1 ms] Stats for NetworkInboundUsageDistributionGoal(NO-ACTION): AVG:{cpu: 0.000 networkInbound: 370.684 networkOutbound: 185.337 disk: 656.680 potentialNwOut: 370.674 replicas:49.666666666666664 leaderReplicas:25.0 topicReplicas:12.416666666666666} MAX:{cpu: 0.000 networkInbound: 1112.053 networkOutbound: 556.011 disk: 687.773 potentialNwOut: 403.316 replicas:52 leaderReplicas:27 topicReplicas:23} MIN:{cpu: 0.000 networkInbound: 1112.053 networkOutbound: 556.011 disk: 595.351 potentialNwOut: 333.606 replicas:48 leaderReplicas:23 topicReplicas:0} STD:{cpu: 0.000 networkInbound: 0.000 networkOutbound: 0.000 disk: 43.368 potentialNwOut: 28.630 replicas:1.699673171197595 leaderReplicas:1.632993161855452 topicReplicas:0.7832093030221935 [ 1 ms] Stats for NetworkOutboundUsageDistributionGoal(NO-ACTION): AVG:{cpu: 0.000 networkInbound: 370.684 networkOutbound: 185.337 disk: 656.680 potentialNwOut: 370.674 replicas:49.666666666666664 leaderReplicas:25.0 topicReplicas:12.416666666666666} MAX:{cpu: 0.000 networkInbound: 1112.053 networkOutbound: 556.011 disk: 687.773 potentialNwOut: 403.316 replicas:52 leaderReplicas:27 topicReplicas:23} MIN:{cpu: 0.000 networkInbound: 1112.053 networkOutbound: 556.011 disk: 595.351 potentialNwOut: 333.606 replicas:48 leaderReplicas:23 topicReplicas:0} STD:{cpu: 0.000 networkInbound: 0.000 networkOutbound: 0.000 disk: 43.368 potentialNwOut: 28.630 replicas:1.699673171197595 leaderReplicas:1.632993161855452 topicReplicas:0.7832093030221935 [ 2 ms] Stats for CpuUsageDistributionGoal(NO-ACTION): AVG:{cpu: 0.000 networkInbound: 370.684 networkOutbound: 185.337 disk: 656.680 potentialNwOut: 370.674 replicas:49.666666666666664 leaderReplicas:25.0 topicReplicas:12.416666666666666} MAX:{cpu: 0.000 networkInbound: 1112.053 networkOutbound: 556.011 disk: 687.773 potentialNwOut: 403.316 replicas:52 leaderReplicas:27 topicReplicas:23} MIN:{cpu: 0.000 networkInbound: 1112.053 networkOutbound: 556.011 disk: 595.351 potentialNwOut: 333.606 replicas:48 leaderReplicas:23 topicReplicas:0} STD:{cpu: 0.000 networkInbound: 0.000 networkOutbound: 0.000 disk: 43.368 potentialNwOut: 28.630 replicas:1.699673171197595 leaderReplicas:1.632993161855452 topicReplicas:0.7832093030221935 [ 1 ms] Stats for TopicReplicaDistributionGoal(NO-ACTION): AVG:{cpu: 0.000 networkInbound: 370.684 networkOutbound: 185.337 disk: 656.680 potentialNwOut: 370.674 replicas:49.666666666666664 leaderReplicas:25.0 topicReplicas:12.416666666666666} MAX:{cpu: 0.000 networkInbound: 1112.053 networkOutbound: 556.011 disk: 687.773 potentialNwOut: 403.316 replicas:52 leaderReplicas:27 topicReplicas:23} MIN:{cpu: 0.000 networkInbound: 1112.053 networkOutbound: 556.011 disk: 595.351 potentialNwOut: 333.606 replicas:48 leaderReplicas:23 topicReplicas:0} STD:{cpu: 0.000 networkInbound: 0.000 networkOutbound: 0.000 disk: 43.368 potentialNwOut: 28.630 replicas:1.699673171197595 leaderReplicas:1.632993161855452 topicReplicas:0.7832093030221935 [ 1 ms] Stats for LeaderReplicaDistributionGoal(NO-ACTION): AVG:{cpu: 0.000 networkInbound: 370.684 networkOutbound: 185.337 disk: 656.680 potentialNwOut: 370.674 replicas:49.666666666666664 leaderReplicas:25.0 topicReplicas:12.416666666666666} MAX:{cpu: 0.000 networkInbound: 1112.053 networkOutbound: 556.011 disk: 687.773 potentialNwOut: 403.316 replicas:52 leaderReplicas:27 topicReplicas:23} MIN:{cpu: 0.000 networkInbound: 1112.053 networkOutbound: 556.011 disk: 595.351 potentialNwOut: 333.606 replicas:48 leaderReplicas:23 topicReplicas:0} STD:{cpu: 0.000 networkInbound: 0.000 networkOutbound: 0.000 disk: 43.368 potentialNwOut: 28.630 replicas:1.699673171197595 leaderReplicas:1.632993161855452 topicReplicas:0.7832093030221935 [ 11 ms] Stats for LeaderBytesInDistributionGoal(VIOLATED): AVG:{cpu: 0.000 networkInbound: 370.684 networkOutbound: 185.337 disk: 656.680 potentialNwOut: 370.674 replicas:49.666666666666664 leaderReplicas:25.0 topicReplicas:12.416666666666666} MAX:{cpu: 0.000 networkInbound: 1112.053 networkOutbound: 556.011 disk: 687.773 potentialNwOut: 403.316 replicas:52 leaderReplicas:27 topicReplicas:23} MIN:{cpu: 0.000 networkInbound: 1112.053 networkOutbound: 556.011 disk: 595.351 potentialNwOut: 333.606 replicas:48 leaderReplicas:23 topicReplicas:0} STD:{cpu: 0.000 networkInbound: 0.000 networkOutbound: 0.000 disk: 43.368 potentialNwOut: 28.630 replicas:1.699673171197595 leaderReplicas:1.632993161855452 topicReplicas:0.7832093030221935 Cluster load after adding broker [2]: HOST BROKER RACK DISK_CAP(MB) DISK(MB)/_(%)_ CORE_NUM CPU(%) NW_IN_CAP(KB/s) LEADER_NW_IN(KB/s) FOLLOWER_NW_IN(KB/s) NW_OUT_CAP(KB/s) NW_OUT(KB/s) PNW_OUT(KB/s) LEADERS/REPLICAS localhost, 0,localhost, 10000.000, 686.917/06.87, 1, 0.000, 10000.000, 209.129, 165.970, 10000.000, 209.129, 375.099, 23/48 localhost, 1,localhost, 10000.000, 687.773/06.88, 1, 0.000, 10000.000, 124.477, 278.839, 10000.000, 124.477, 403.316, 25/49 localhost, 2,localhost, 10000.000, 595.351/05.95, 1, 0.000, 10000.000, 222.436, 111.202, 10000.000, 222.404, 333.606, 27/52 * Connection #0 to host localhost left intact ``` * if your found error 500 after call proposals such as ```bash Error processing POST request '/add_broker' due to: 'com.linkedin.kafka.cruisecontrol.exception.KafkaCruiseControlException: com.linkedin.cruisecontrol.exception.NotEnoughValidWindowsException: There are only 0 valid windows when aggregating in range [-1, 1670319259389] for aggregation options (minValidEntityRatio=0.95, minValidEntityGroupRatio=0.00, minValidWindows=1, numEntitiesToInclude=75, granularity=ENTITY)' ``` please wait and re try again because Cruise Control has not collect enough metric data to aggregate to a valid window. (around 2-3 minutes) * Approving an optimization proposal ```bash curl -v -X POST 'localhost:9191/kafkacruisecontrol/add_broker?dryrun=false&brokerid=2' ``` * Check progress ```bash curl 'localhost:9191/kafkacruisecontrol/user_tasks' ``` example result ```bash USER TASK ID CLIENT ADDRESS START TIME STATUS REQUEST URL 081d2454-5c1c-4c9f-be3c-90d282fdf0fe [0:0:0:0:0:0:0:1] 2022-11-24T06:20:30Z InExecution POST /kafkacruisecontrol/add_broker?brokerid=2&dryrun=false 7bb50955-f9e5-45e6-bb6b-6177d2b3dc8e [0:0:0:0:0:0:0:1] 2022-11-24T06:13:33Z Completed GET /kafkacruisecontrol/state 5c0d11d0-0b98-431b-965d-f030e3263835 [0:0:0:0:0:0:0:1] 2022-11-24T06:13:38Z Completed GET /kafkacruisecontrol/state ``` call check until status change to Completed ```bash USER TASK ID CLIENT ADDRESS START TIME STATUS REQUEST URL 7bb50955-f9e5-45e6-bb6b-6177d2b3dc8e [0:0:0:0:0:0:0:1] 2022-11-24T06:13:33Z Completed GET /kafkacruisecontrol/state ... 081d2454-5c1c-4c9f-be3c-90d282fdf0fe [0:0:0:0:0:0:0:1] 2022-11-24T06:20:30Z Completed POST /kafkacruisecontrol/add_broker?brokerid=2&dryrun=false ``` * Check Topic after call cruise-control/add\_broker, some partition move to new broker ```bash cd ~/amq-streams-2022/4-management ./kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic demo ``` example result, show partition of demo topic in broker 0,broker 1 and broker 2 ```bash Topic: demo TopicId: Z5cEJOZdQ_iUsuzXOWvdfw PartitionCount: 10 ReplicationFactor: 2 Configs: segment.bytes=104857600 Topic: demo Partition: 0 Leader: 2 Replicas: 2,0 Isr: 0,2 Topic: demo Partition: 1 Leader: 2 Replicas: 2,1 Isr: 1,2 Topic: demo Partition: 2 Leader: 1 Replicas: 1,0 Isr: 1,0 Topic: demo Partition: 3 Leader: 2 Replicas: 2,1 Isr: 1,2 Topic: demo Partition: 4 Leader: 2 Replicas: 2,0 Isr: 0,2 Topic: demo Partition: 5 Leader: 0 Replicas: 0,1 Isr: 0,1 Topic: demo Partition: 6 Leader: 1 Replicas: 1,2 Isr: 1,2 Topic: demo Partition: 7 Leader: 0 Replicas: 0,1 Isr: 0,1 Topic: demo Partition: 8 Leader: 1 Replicas: 1,0 Isr: 1,0 Topic: demo Partition: 9 Leader: 0 Replicas: 0,2 Isr: 0,2 ``` ## Stop Server * stop cruise-control ```bash cd ~/amq-streams-2022/4-management/cruise-control ./kafka-cruise-control-stop.sh config/cruisecontrol.properties 9191 ``` * stop kafka broker in each terminal with ctrl+c * stop zookeeper in each terminal with ctrl+c --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://chatapazar.gitbook.io/amq-streams-training-2022/amq-streams-2022-training/cruisecontrol.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.