Kafka

How to build distributed scalable system with Kafka Concurrency?

How to access Spring Beans from Non-Spring managed classes and pojos ? How to make an API call only when required ? How to avoid making too many API calls ? How to make an API call only when required for every kafka message ? How to make sure your application is performant when using 3rd party REST API calls ?

Kafka Concepts Summarized in a single diagram.

Kafka Guarantees, Messages are appended to a topic-partition in the order they are sent. Consumers read messages in the order stored in a topic-partition

Kafka Brokers Discovery, Every Kafka broker is called a bootstrap server. That means that you only need to connect to one broker and you will be connected to the entire cluster. Zookeeper manages brokers. Kafka can't work without Zookeeper

Kafka stores the offsets at which a consumer group has been reading. The offsets committed live in a Kafka topic named _consumer_offsets. Delivery Semantics for Consumers : Atmost once, Atleast once, Exactly once

Kafka Consumers. Consumers read data from a topic. Consumers know which broker to read from. Each consumer within a group reads from exclusive partitions. What if too many Consumers?

Kafka Producers. Producers can choose to receive acknowledgement of data writes.

Kafka Brokers and Topics. Concept of Leader for a Partition. At any time only ONE broker can be a leader for a given partition. Example of brokers and partition. Concept of Leader for a partition

Kafka Topics, Partitions and Offsets

What is Kafka ? Kafka is a stream-processing platform for handling unified, high-throughput, low-latency real-time data feeds.