Spread the loveKafka Topics List existing topics bin/kafka-topics.sh –zookeeper localhost:2181 –list Purge a topic bin/kafka-topics.sh –zookeeper localhost:2181 –alter –topic mytopic –config retention.ms=1000 … wait a minute … bin/kafka-topics.sh –zookeeper localhost:2181 –alter –topic mytopic –delete-config retention.ms
We have two kinds or types of work flows in Apache Kafka:
- Topic Messaging System
- Queue Messaging System
We will see how it will work.
First we will see work flow of a topic:
It is also called as publish-subscribe messaging so in this scenario producers send messages to a topic at regular intervals, Kafka broker stores all the messages in the partitions configured for the particular topic and it ensures that messages are equally shared between partitions.
If the producer sends two messages and there are two partitions then Kafka will store one message in first partition and the second message in the second partition and then consumers subscribe to a specific topic. Once the consumer subscribes to a topic, Kafka will provide the current offset of the topic to the consumer and also save the offset in the Apache Zookeeper.
Consumer will send request to Kafka in regular intervals like 100 milliseconds for new messages.
Once Kafka receives the messages from producers, It will forward these messages to the consumers and then the consumers will receive the messages and process them. If needed consumer sends an acknowledgement to the Kafka broker saying that I got the messages.
When Kafka receives the acknowledgement, it will change its offset value to the new value and updates in the zookeeper also because all offsets are maintained in the zookeeper.
Note: In Apache Kafka latest versions like 0.9.x.x onwards, we can store the offset in both Apache Kafka and Apache Zookeeper. Apache community is trying to remove the zookeeper dependency and hopefully they will remove zookeeper dependency soon.
The consumers can read next messages correctly even doing server outrageous. so this process will be repeated until the consumer stops the request. consumer has the option to rewind or skip to the desired offset of a topic at any time and read all the subsequent messages. this is how topic Messaging System will work
Queue messaging system also called as point-to-point system
Now we will see workflow of Queue messaging and it is also called as consumer group.
In this case instead of a single consumer, a group of consumers having the same group ID will subscribe to a topic. In previous case only one consumer is subscribing to that particular topic but in this case a group of consumers with the same group ID are subscribing to the topic.
In simple terms consumers subscribing to a topic with the same group ID are considered as a single group and the messages are shared among them, so we will see how it will work.
Here, producer will sends the message to your topic in a regular interval and Kafka stores all the messages in the partitions configured for that particular topic similar to the earlier scenario.
A single consumer subscribes to a specific topic, assume a topic with some group ID then Kafka interact with the consumer in the same way as a publishing subscribe messaging until new consumers subscribe the same topic with the same group ID.
Here we will consider the topic as topic 1 and group IDs Group 1. Once the new consumer arrives Kafka switches its operation to share mole and shares the data between the two consumers. This sharing will go on until the number of consumers reach the number of partitions configured for that particular topic.
Once the number of consumer exceeds the number of partitions the new consumer will not receive any further message until any one of the existing consumer unsubscribes. This scenario arises because each consumer in Kafka will be assigned a minimum of one partition or once all the partitions are assigned to the existing consumers, the new consumer will have to wait. This feature is also called as consumer group in the same way Kafka will provide the best of both the systems in very simple and efficient manner.