Spread the loveKafka Topics List existing topics bin/kafka-topics.sh –zookeeper localhost:2181 –list Purge a topic bin/kafka-topics.sh –zookeeper localhost:2181 –alter –topic mytopic –config retention.ms=1000 … wait a minute … bin/kafka-topics.sh –zookeeper localhost:2181 –alter –topic mytopic –delete-config retention.ms
If you see some background about Kafka security. For versions older than 0.9 security was achieved by maintaining access at network level which was not a good option when we use the multi-tenant cluster.
For the large applications consequently securing Kafka has been one of the most requested features. Security is one of the most important dimensions in today’s world, where everyone wants to access everyone’s data.
Kafka community added a number of features that can be used to increase Kafka cluster security. Addressing security threats are crucial in today’s world as it is threatened by the wide variety of cyber attacks so Apache Kafka can become a good choice for an enterprise messaging system so that is the reason in recent version that is 0.10 they added many features so these features are used individually or together so that we can increase the security in the cluster.
In the next lectures we will see, what are the currently supported security measures in Kafka versions 0.10.x.x.
Authentication of connections between Kafka servers or brokers and clients:
If you see the diagram above, the external clients and the internal clients are trying to talk to or connect to a Kafka cluster. A cluster is nothing but a set of Kafka brokers so in this communication we want to enable the security.
This kind of communication is called connections between brokers and clients so this authentication will provide a security in this layer.
Authentication of connections between Apache Kafka servers and Apache Zookeeper
When it comes to Kafka zookeeper is an important component. It will store all the meta data like it will store consumer offset and we will also see what are the recent changes in 0.10 but for now just remember Kafka will store all the metadata so it is important enable a security or encryption between this layer.
Encryption of data using SSL:
As I mentioned it is performance impact, if you enable SSL definitely your performance is going to degrade. If you use one way authentication it will go bit low if you enable 2 way authentication then definitely it will go around 50 percent.
We have encryption between brokers and clients, between the brokers and broker and other tools. When I say between brokers like to replicate from one server to another server so in that situation we want to encrypt the data so this is about the inter broker communication.
Authorization of read and write or ACL or Access Control List:
If you use only one topic or if you are the only user using this entire cluster then you are good you don’t need this ACL but if you are a part of a managed cluster your cluster. If it is being used by many users then it is difficult to maintain because any user can access any topic so they can send the data they can receive the data. This is not a good practice that is the reason we have to enable ACL so that we can restrict the access on a particular topic level or host level.
Also I want to introduce you two concepts that is:
What is authentication ?
What is authorization ?
Difference between them ?
Currently Kafka supports the below listed ports
First one is plain text, when use plain text we are not enabling any encryption and we are also not enabling any authentication. We will just send the message and receive the message that’s it simple.
The next one is SSL, when you use SSL we are internally encrypting the data and also we are using it as authentication but this is limited authentication.
The next one is SASL, it is also called as Kerberos authentication. It is a vast subject, I have to introduce you about the Kerberos so this is not something which we are covering in this tutorial. If you are aware of how to use Kerberos then I will provide you the material so that you can develop the security.
You can also use both SSL and SASL together. The SSL is used for encryption purpose SASL is used for authentication purpose.
If we enable the security from broker or server side but it is client responsibility to configure correct credentials and also correct port is used then only security feature will work.
If you have anything feel free to reach out to us.