Brokers

Brokers

• A Kafka cluster is composed of multiple brokers (servers)
• Each broker is identified with its ID (integer)
• Each broker contains certain topic partitions
• After connecting to any broker (called a bootstrap broker), you will be connected to the entire cluster
• A good number to get started is 3 brokers, but some big clusters have over 100 brokers.
• You can choose to number brokers starting at 100 (arbitrary)

---------------       ---------------      --------------
|             |       |             |      |            |
| Broker 101  |       | Broker 102  |      | Broker 103 |
|             |       |             |      |            |
---------------       ---------------      --------------

Brokers and Topics

• Example of Topic-A with 3 partitions
• Example of Topic-B with 2 partitions

• Note: Data is distributed and Broker 103 doesn’t have any Topic B data

Concept of Leader for a Partition

• At any time only ONE broker can be a leader for a given partition
• Only that leader can receive and serve data for a partition
• The other brokers will synchronize the data
• Therefore each partition has one leader and multiple ISR (in-sync replica)