Spread the loveKafka Topics List existing topics bin/kafka-topics.sh –zookeeper localhost:2181 –list Purge a topic bin/kafka-topics.sh –zookeeper localhost:2181 –alter –topic mytopic –config retention.ms=1000 … wait a minute … bin/kafka-topics.sh –zookeeper localhost:2181 –alter –topic mytopic –delete-config retention.ms
Messaging system: when we are dealing with large amount of data, we have two main challenges:
First challenge is how to collect large volume of data.
Second challenge is to analyze the collected data.
To overcome those challenges you must need a messaging system.
First of all what is a messaging system ?
A messaging system is responsible for transferring data from one application to another so the applications can just focus on data but not to worry about how to share the data.
The distributed messaging is based on the concept of reliable message Queuing. The messages are queued synchronously between applications and messaging system.
Two types of messaging patterns are available one is point-to-point and the other is publish/subscribe.
Most of the messaging patterns will follow publish and subscribe.
so what is point-to-point messaging system ?
In a point-to-point messaging system messages are persisted in a queue. One or more consumers can consume the messages in the queue but a particular message can be consumed by a maximum of one consumer. Once a consumer reads a message in the queue then the messages will be disappear from that particular queue.
The typical example of this system is an order processing system where each other will be processed by one order processor but multiple order processors can work as well at the same time.
The following diagram explains the structure of a point-to-point messaging system.
So, producer is sending the message to a queue. here only the consumer with the green line will be able to consume a message.
When it comes to publish/subscribe messaging system, the messages are persisted in a topic.
Unlike point-to-point system, consumers can subscribe to one or more topics and consume all the messages in the topic.
In the publish/subscribe system message producers are called publishers and the message consumers are called subscribers
A real-life example for this kind of system is our Dish TV, which publishes different channels like sports, movies and shows and anyone can subscribe to their own set of channels and get them whenever their subscriber channels are available.
If you see the diagram the producer is sending the messages to the topic and all three consumers are able to consume the messages that will be shown in the green line.