Spread the loveKafka Topics List existing topics bin/kafka-topics.sh –zookeeper localhost:2181 –list Purge a topic bin/kafka-topics.sh –zookeeper localhost:2181 –alter –topic mytopic –config retention.ms=1000 … wait a minute … bin/kafka-topics.sh –zookeeper localhost:2181 –alter –topic mytopic –delete-config retention.ms
In this Apache Zookeeper course, we will learn the complete Apache Zookeeper concepts like:
What is Apache Zookeeper
Why we need Apache Zookeeper
What is Distributed System
Why Coordination is a challenge ?
Apache ZooKeeper Back ground
Apache Zookeeper Architecture
Important Components of Apache Zookeeper
Apache Zookeeper Data Model
Apache Zookeeper ZNodes
Apache Zookeeper Sessions & Watches
Popular Applications/Companies using Apache Zookeeper
Basic Apache Zookeeper CLI or Command Line Operations
WHAT IS APACHE ZOOKEEPER
Apache ZooKeeper is a distributed, open-source coordination service for distributed applications.
It exposes a simple set of primitives that distributed applications can build upon to implement higher level services for synchronization, configuration maintenance, groups and naming.
It is designed to be easy to program and also easy to.
Apache ZooKeeper is implemented in Java. It has bindings with C, Java and Python languages
Apache ZooKeeper is widely used by a large number of organizations, such as Yahoo! Inc., Twitter, Net ix, and Facebook, in their distributed application platforms as a coordination service. We will discuss more about how ZooKeeper is used in the real world in coming lectures
WHY WE NEED APACHE ZOOKEEPER
Coordination services: The process of Integrating or communicating b/w the services in a distributed environment.
Coordination services are notoriously hard to get right. They are especially prone to errors such as race conditions and deadlock.
Race condition − Two or more systems are trying to perform the same task, which actually needs to be done only by a single system at a given time. For example, shared resources should only be modified by a single machine at any given time.
Deadlock − Two or more operations waiting for each other to complete indefinitely.
To make the co-ordination in distributed environment easy, developer came up with ZooKeeper so that we don’t to implement our own coordination services from scratch
WHAT IS DISTRIBUTED SYSTEM
Distributed System: To speak in general terms: multiple computer systems working on a single problem
In Engineering terms: A distributed system is a network that consists of independent computers that are connected using a distribution middleware.
The key features of a distributed system are:
Components in the system are concurrent. It allows resource sharing, including software by systems connected to the network at the same time. The components could be multiple but will generally be autonomous in nature. The systems can be spread across different geographies. Compared to other network models, there is greater fault tolerance in a distributed model. Price/performance ratio is much better.
The key goals of a distributed system include:
Transparency: Hides the complexity of the system and shows itself as a single entity / application.
Reliability: Compared to a single system, a distributed system should be highly capable of being secure, consistent and have a high capability of masking errors.
Performance: Compared to other models, distributed models are expected to give a much-wanted boost to performance.
Scalability: Distributed systems should be scalable with respect to geography, administration or size.
Challenges for the distributed system include:
Security is a big challenge in a distributed environment, especially when using public networks.
Fault tolerance could be tough when the distributed model is built based on unreliable components.
Coordination and resource sharing can be difficult if proper protocols or policies are not in place.
WHY COORDINATION IS A CHALLENGE ?
Now you have an idea on what is Distributed system, let’s drill down deeper into why coordination in a distributed system is a hard problem.
Let’s take the example of doing coordination or configuration management for a distributed application that has many systems. Now, having a master node where the cluster data is stored and other worker nodes or slave nodes get the data from this master node is look like simple.
However, this solution suffers from a potential problem of the master node being a single point of failure.
Even if we assume that the master node is designed to be fault-tolerant, these changes should propagate to all the slave nodes and it is not easy.
Another coordination problem is, To increase availability when we add more sservers to the system. In these situations coordination will be more difficult. It needs careful design and implementation of logic.
APACHE ZOOKEEPER BACK GROUND
Apache ZooKeeper is a software project of the Apache Software Foundation; it provides an open source solution to the various coordination problems in large distributed systems. ZooKeeper was originally developed at Yahoo!
Apache ZooKeeper is implemented in Java
ZooKeeper, as a centralized coordination service, is distributed and highly reliable, running on a cluster of servers called a ZooKeeper ensemble.
We will dive deeper into these concepts in later sections
APACHE ZOOKEEPER ARCHITECTURE
There are two types of nodes are available on zookeeper. 1. Leader node and 2. Follower node
Leader Node – Leader Node is the only node responsible for processing the write requests. All other nodes called followers simply delegate the client write calls to Leader node.We don’t mark any node as leader while setting up Apache ZooKeeper cluster. It instead is elected internally among all the nodes of cluster. Apache ZooKeeper uses the concept of majority for same i.e. Node that gets highest number of votes is elected as Leader.
It is recommended to have odd number of nodes in a cluster for best failover and availability.
E.g. if we create the cluster of four nodes and two nodes go offline for some reason. Apache ZooKeeper will be down as half of the nodes have gone offline as it is not possible to gain majority for Leader node election. However if we create the cluster of five nodes, even if two nodes go offline, Apache ZooKeeper will still be functional as we still have majority of nodes in service.
Follower Nodes – All nodes other than Leader are called Follower Nodes. A follower node is capable of servicing read requests on its own. For write requests, it gets it done through Leader Node. Followers also play an important role of electing a new leader if existing leader node goes down.
IMPORTANT COMPONENTS OF APACHE ZOOKEEPER
Request Processor – This component is only active in Leader Node and is responsible for processing write requests originating from client or follower nodes. Once request processor proesses the write request, it broadcasts the changes to follower nodes so that they can update their state accordingly.
Atomic Broadcast –This component is present in both Leader Node and Follower Nodes. This component is responsible for broadcasting the changes to other nodes (in Leader Node) as well as receiving the change notification (in Follower Nodes).
In-memory Database (Replicated Database) – This in-memory and replicated database is responsible for storing the data in ZooKeeper. Every node contains its own database that enables them to server read requests. In addition to this, data is also written to file system providing recoverability in case of any problems with cluster. In case of write requests, in-memory database is updated only after it has successfully been written to file system.
Client: One of the nodes in our distributed application cluster, access information from the server. For a particular time interval, every client sends a message to the server to let the sever know that the client is alive.
Similarly, the server sends an acknowledgement when a client connects. If there is no response from the connected server, the client automatically redirects the message to another server.
Server: One of the nodes in our ZooKeeper ensemble, provides all the services to clients. Gives acknowledgement to client to inform that the server is alive.
Ensemble: Group of ZooKeeper servers. The minimum number of nodes that is required to form an ensemble is 3.
APACHE ZOOKEEPER DATA MODEL
The name space provided by ZooKeeper is much like a standard file system. A name is a sequence of path elements separated by a slash (/).
Every node in ZooKeeper’s name space is identified by a path and these ZooKeeper nodes arereferred as znode
Each znode can store upto 1MB of data.
These path details has to be absolute, because zookeeper can’t recognize relative paths
APACHE ZOOKEEPER ZNODES
Every node in a ZooKeeper tree is referred to as a znode
Znodes maintain a stat structure that include, Version numbers, ACL ,Timestamp, Data length
Version number − Every znode has a version number, which means every time the data associated with the znode changes, its corresponding version number would also increased.
(ACL) − ACL is basically an authentication mechanism for accessing the znode. It represents read and write permisions on znode.
Timestamp − Timestamp represents time elapsed from znode creation and modification. It is usually represented in milliseconds.
Data length − Total amount of the data stored in a znode is the data length. You can store a maximum of 1MB of data.
Type of znodes: we have 3 types of znodes they are persistence, ephemeral and sequential.
Persistence znode is alive until they’re explicitly deleted, A znode can be deleted by calling the delete API call.
Ephemeral znodes are active until the client connection is alive. When a client gets disconnected from the ZooKeeper ensemble, then the ephemeral znodes get deleted automatically
Sequential znodes can be either persistent or ephemeral. When a new znode is created as a sequential znode, then ZooKeeper sets the path of the znode by attaching a 10 digit unique sequence number to the original name. Sequential znodes play an important role in Locking and Synchronization.
APACHE ZOOKEEPER SESSIONS & WATCHES
Sessions are very important for the operation of ZooKeeper. Requests in a session are executed in FIFO order. Once a client connects to a server, the session will be established and a session id is assigned to the client.
The client sends heartbeats at a particular time interval to keep the session valid. If the ZooKeeper ensemble does not receive heartbeats from a client for more than the period (session timeout) specified at the starting of the service, it decides that the client died.
Session timeouts are usually represented in milliseconds. When a session ends for any reason, the ephemeral znodes created during that session also get deleted.
Watches are a simple mechanism for the client to get notifications about the changes in the ZooKeeper ensemble. Clients can set watches while reading a particular znode. Watches send a notification to the registered client for any of the znode (on which client registers) changes.
Znode changes are modification of data associated with the znode or changes in the znode’s children. Watches are triggered only once. If a client wants a notification again, it must be done through another read operation. When a connection session is expired, the client will be disconnected from the server and the associated watches are also removed.
POPULAR APPLICATIONS/COMPANIES USING APACHE ZOOKEEPER
Popular Applications and Companies using Zookeeper
Some of the better known applications and organizations using ZooKeeper:
- Apache HBase, a Hadoop database uses ZooKeeper for leader election, bootstrapping, server lease management and coordination between servers.
- Apache Solrcloudedition v1.5 uses ZooKeeper for leader election, configuration and other services.
- Yahoo! uses ZooKeeper for leader election, configuration management, sharding, locking, group membership, etc…
- Katta, a scalable, failure tolerant, distributed, data storage for real time access uses ZooKeeper for node, master and index management in the grid.
- Eclipse Communication Framework,a framework for building distributed servers, applications, and tools uses ZooKeeper for its abstract discovery services.
- Deepdyve, online article rental service uses ZooKeeper to manage server state, control index deployment and other tasks.
- AdtroitLogic’s UltraESB,a enterprise service bus that uses ZooKeeper to implement it’s clustering support and the automated round-robin-restart of the complete cluster.
BASIC APACHE ZOOKEEPER CLI OR COMMAND LINE OPERATIONS
From the shell or Terminal, type help to get a listing of commands that can be executed from the client:
[zkshell: 0] help ZooKeeper host:port cmd args get path [watch] ls path [watch] set path data [version] delquota [-n|-b] path quit printwatches on|off createpath data acl stat path [watch] listquota path history setAcl path acl getAcl path sync path redo cmdno addauth scheme auth delete path [version] setquota -n|-b val path
Please contact us for any help.