Fact-checked by Grok 2 weeks ago

Apache ZooKeeper

Apache ZooKeeper is an open-source, distributed coordination service that provides a centralized infrastructure for maintaining information, naming, synchronization, and group services in large-scale distributed applications. Developed originally at Yahoo! to address coordination challenges in clusters like Hadoop, it simplifies the creation of reliable distributed systems by offering a simple set of that reduce the complexity of implementing synchronization and reduce bugs associated with race conditions. As a high-performance for coordination, ZooKeeper enables applications to build such as , group membership, and dynamic updates through its wait-free data objects and efficient handling of read-heavy workloads, supporting tens to hundreds of thousands of with read-to-write ratios from 2:1 to 100:1. At its core, operates as a replicated, centralized of servers that maintains a shared hierarchical resembling a , where is stored in znodes—small files or directories that can hold and support watches for change notifications. It guarantees strict consistency through for all state-changing operations and ordering of requests per client, ensuring reliable coordination even in the presence of failures via its Zab ( Atomic Broadcast) , a leader-based mechanism for propagating updates across the . This architecture makes particularly suited for Internet-scale systems, where it has been deployed to manage services in production environments handling millions of operations daily. Originally created in the mid-2000s at Yahoo! as a general-purpose solution to coordination problems across various distributed applications, ZooKeeper was open-sourced and donated to , becoming a top-level project in November 2008. Since then, it has evolved through community contributions, with ongoing releases as of 2025 (latest stable 3.9.4) enhancing performance, security, and support for distributed frameworks such as Hadoop, while maintaining and focusing on reliability for mission-critical deployments; however, some projects like have transitioned to alternative coordination mechanisms such as KRaft.

History and Development

Origins at Yahoo

Apache ZooKeeper was developed in 2006-2007 by engineers at Research, including Benjamin Reed and Flavio Junqueira, to address persistent coordination challenges in large-scale distributed systems. These challenges arose from the need to manage , , and failure recovery across numerous services, such as the and Fetching Service, where ad-hoc implementations often led to brittle and unreliable applications. The project aimed to provide a centralized, reliable service that would allow developers to focus on core application logic rather than reinventing coordination primitives. Early prototypes emphasized through replication, via automatic recovery mechanisms, and to handle read-heavy workloads typical in Yahoo's infrastructure, ensuring consistent views of shared state even under partial failures. Internally at Yahoo, ZooKeeper was initially deployed for critical tasks including management in the Fetching Service for web crawling, recovery in the to detect crashes and elect leaders, and primitives to coordinate distributed processes. These applications demonstrated its effectiveness in providing guarantees and wait-free operations, quickly leading to widespread adoption within Yahoo's ecosystem before its transition to an open-source Apache project.

Apache Project Evolution and Releases

Apache ZooKeeper entered the Apache Incubator as a subproject of Hadoop in 2008, marking its initial step into the open-source ecosystem under the Apache Software Foundation. It graduated to become a top-level Apache project on November 17, 2010, allowing it to operate independently with its own governance and community. The project's release history began with the first stable version, 3.0.0, issued in November 2008, which established the core framework for distributed coordination. Subsequent major versions introduced significant enhancements: the 3.4.x series (spanning releases from 2011 to 2019) improved access control lists (ACLs) for finer-grained permissions and refined watch mechanisms for more efficient event notifications. The 3.5.x series, starting with alpha releases in 2014 and stabilizing in 2019, added dynamic reconfiguration to enable runtime adjustments to the server ensemble without restarts. The 3.8.x series, starting in 2022 and with the latest stable release 3.8.5 on September 18, 2025, introduced LogBack logging migration, JDK 17 support, and OSGi compatibility. In 2020, the 3.6.x series brought authentication enhancements, including better SASL support and audit logging for improved security tracking. The 3.7.x series, initiated in 2021, emphasized performance optimizations such as reduced memory usage and faster processing in high-load scenarios. The most recent version, 3.9.4 released in August 2025, focuses on security patches, including updates to logback-classic 1.3.15 and slf4j-api 2.0.13 to address CVEs, alongside minor protocol adjustments for compatibility. Over time, evolved from a Hadoop-centric tool for cluster management into a standalone coordination service suitable for diverse distributed systems, broadening its applicability beyond ecosystems. This shift is supported by ongoing community contributions tracked through the Apache system, such as ZOOKEEPER-4891, which implemented critical security updates for logging dependencies. The ZooKeeper community maintains a support policy that prioritizes two active branches—stable and current—with end-of-life announcements approximately six months after a new minor version's release, with downloads available for one year post-EoL and a for critical fixes. For instance, version 3.7.2 reached end-of-life on February 2, 2024, with downloads available until February 2, 2025. The project places strong emphasis on across versions to facilitate enterprise deployments, minimizing disruptions during upgrades.

Overview

Purpose and Design Goals

Apache ZooKeeper serves as a centralized, reliable coordination service designed to maintain configuration information, naming, distributed , and group services for large-scale distributed applications. It enables developers to build robust systems without reinventing coordination primitives, addressing the complexities of managing state across multiple processes in environments like clusters. The design of ZooKeeper emphasizes simplicity through a file-like interface that mimics a hierarchical namespace, allowing applications to avoid implementing custom protocols for coordination tasks. For high performance, it employs in-memory storage to achieve low latency and high throughput, particularly optimized for read-dominant workloads with typical ratios of 10:1 reads to writes. Reliability is ensured via replication across an ensemble of servers, eliminating single points of failure and enabling quick recovery from partial failures, such as through leader election in under 200 milliseconds. Additionally, ZooKeeper enforces strict ordering of operations using ZooKeeper Transaction IDs (zxids), which assign a unique, monotonically increasing identifier to each transaction to guarantee total order and prevent race conditions. ZooKeeper specifically tackles challenges in distributed systems, including partial failures where nodes may crash or become unreachable, race conditions arising from concurrent updates, and the overhead of coordination in large clusters by providing operations and notifications. However, it is not intended as a general-purpose database or distributed ; instead, it focuses on small coordination , with each limited to less than 1 MB to prioritize primitives like ephemeral znodes for session-based management.

Core Features and Benefits

Apache ZooKeeper provides a hierarchical that organizes data in a tree-like structure, where each , known as a znode, can store data and have child nodes, facilitating and naming services. Ephemeral znodes are temporary nodes that are automatically deleted upon the termination of the creating client's session, enabling dynamic tracking of distributed processes such as group membership. Clients can set watches on znodes to receive one-time notifications of changes (with permanent and recursive watches available since version 3.6.0), allowing for efficient event-driven coordination without constant polling. All operations on znodes are atomic, with built-in versioning that includes separate version numbers for data, children, and lists, ensuring conditional updates and preventing conflicts in concurrent modifications. These features deliver significant benefits in performance and reliability for distributed systems. ZooKeeper achieves high throughput, with version 3.2 demonstrating approximately twice the read/write performance compared to version 3.1. Benchmarks on ensembles of up to 50 dual-core 2.1 GHz servers with 4 GB show it handling up to 460,000 read operations per second on a 13-server ensemble and average request latencies around 1.2 ms on a three-server ensemble. Scalability tests using over 30 client simulators on dual 2 GHz servers with high-RPM drives further confirm robust handling of concurrent workloads. Fault tolerance is ensured through a quorum-based replication model, where an of 2f+1 servers can tolerate up to f failures while maintaining via majority for writes. This provides a single, consistent system image across all replicas, as clients always see a coherent view of the regardless of which server they connect to. Leader elections occur with low , typically under 200 ms, minimizing during failures. operates as an with write-ahead disk logging and periodic snapshots for recovery, balancing speed with . Compared to ad-hoc coordination solutions, ZooKeeper's wait-free design reduces the need for custom in applications, while guaranteeing even in unreliable networks through its protocol.

Data Model and Programming Interface

Znodes and Hierarchical Namespace

At the core of ZooKeeper's data model is the znode, a fundamental data unit that organizes information in a hierarchical resembling a . Each znode represents either a file-like entity or a directory-like container, uniquely identified by an absolute path composed of slash-separated elements, such as /app/config, enabling logical organization of distributed coordination data. A znode can store up to 1 of data—typically small payloads in the byte to range for efficiency—along with metadata including version numbers, lists (ACLs) for permissions, and a list of its child znodes. This structure supports atomic operations, ensuring in distributed environments without requiring full data replacement for partial updates. Znodes come in several types to accommodate different coordination needs. Persistent znodes endure beyond the session that created them, remaining in the namespace until explicitly deleted, making them suitable for stable storage. Ephemeral znodes, in contrast, are automatically deleted upon the close of the creating session, often used for transient purposes like representing active nodes in a cluster or heartbeat signals. Ephemeral znodes cannot have children, enforcing their temporary nature. Sequential znodes append a monotonically increasing 10-digit sequence number (e.g., /lock0000000001) to the base path upon creation, facilitating ordered such as in distributed locks or queues. The supports key operations to manipulate znodes: establishes a new znode with initial data and ACLs; deletion removes a znode and its data; getting data retrieves the current content atomically; setting data updates the content while optionally checking versions; and listing children enumerates the immediate sub-znodes without their data. Paths must be absolute forms, disallowing relative references or elements like "." or "..", and adhere to encoding with restrictions such as no null characters. Versioning enhances reliability by associating each znode with distinct counters for data changes (), child list modifications (cversion), and ACL updates (aversion), included in the stat metadata. Operations like setting data can specify an expected version, failing if mismatched to prevent concurrent overwrites and detect conflicts in distributed updates. This mechanism, combined with the tree-like hierarchy, draws an analogy to a , where znodes function as hybrid files and directories but emphasize coordination primitives like over bulk storage.

Sessions, Watches, and Client API

In Apache , a session establishes and maintains a client's to the ZooKeeper ensemble, uniquely identified by a 64-bit generated upon initial . The session includes a configurable timeout, typically set to 30 seconds by clients, though the server enforces limits between twice and twenty times its tickTime parameter (default 2 seconds). To prevent expiration, the client library sends periodic pings to the server, often at intervals of one-third the timeout period, ensuring the remains active even during temporary network issues. If no is received within the timeout, the session expires, automatically deleting any ephemeral znodes associated with it and notifying registered watchers. Upon detecting a disconnection, the client library transparently attempts to reconnect to another server in the ensemble, preserving the if the reconnection occurs before expiration. Watches enable efficient, event-driven notifications for clients monitoring znode changes, eliminating the need for constant polling. Set as one-time triggers during read operations like exists(), getData(), or getChildren(), watches deliver asynchronous events for specific changes, such as NodeCreated, NodeDeleted, NodeDataChanged, or NodeChildrenChanged. Upon triggering, ZooKeeper queues the event for delivery to the client, including the event type, znode path, and session details, but the watch is automatically unregistered after firing or if the session closes. Events guarantee ordered delivery within the session's FIFO request sequence, ensuring the client processes the notification before observing the updated data. Introduced in ZooKeeper 3.6.0, persistent watches via addWatch() allow repeated notifications for ongoing changes without reregistration, including recursive options to monitor entire subtrees. The client offers a , consistent for manipulating the hierarchical , supporting both synchronous and asynchronous operations on a thread-safe handle. Key operations include create(path, data, , flags) for creating znodes with support for ephemeral or sequential types; delete(path, version) for removal, requiring version matching to prevent conflicts; exists(path, watch) to check znode presence and optionally set a watch; getData(path, watch) to retrieve byte array data; setData(path, data, version) to update with optimistic concurrency via version; getChildren(path, watch) to list child names; and sync(path) to force synchronization of the client's local view with the server. Asynchronous variants append callbacks for non-blocking execution, maintaining strict ordering of requests from the same session. Multi-threaded applications can safely share a single instance, as it serializes operations internally. Client initialization uses a straightforward constructor , such as ZooKeeper zk = new ZooKeeper(connectString, sessionTimeoutMs, [watcher](/page/Watcher)), where connectString specifies the ensemble (e.g., "host1:2181,host2:2181"), sessionTimeoutMs defines the timeout in milliseconds, and implements the Watcher interface for handling. Errors are reported via exceptions or codes, with KeeperException.Code.OK (or ZOK in C bindings) indicating success, Code.SESSIONEXPIRED (ZSESSIONEXPIRED) for timeout failures, and others like Code.NOAUTH for permission issues, allowing robust recovery in distributed applications.

Architecture and Implementation

Distributed Ensemble and Replication

Apache ZooKeeper operates as a distributed service through an ensemble, which is a collection of ZooKeeper servers that replicate data to provide high availability and fault tolerance. An ensemble typically consists of an odd number of servers, with 3 or 5 recommended for production environments to establish a clear majority quorum while minimizing resource overhead; for instance, a 3-server ensemble can tolerate one failure without losing quorum. Clients connect to any server in the ensemble, which then routes write operations to the designated leader, ensuring seamless access regardless of the entry point. Data replication in is synchronous and maintains across the through an atomic broadcast mechanism, where all updates are propagated to followers from the leader before acknowledgment. The entire data tree resides in memory on each server for fast access, supplemented by periodic disk snapshots and logs for and ; snapshots capture the at a given ID, while logs record all changes to enable followers to catch up during synchronization. This in-memory approach, combined with persistent storage, allows to recover quickly from failures without relying on external databases. In the ensemble, servers assume distinct roles: the leader processes all write requests and coordinates replication, while followers primarily handle read requests and forward writes to the leader for processing. Followers operate in read-only mode for clients but participate fully in decisions. This architecture supports scalability for thousands of concurrent clients, with availability guaranteed as long as a of servers remain operational—for example, a 5-server ensemble can withstand two failures while continuing service.

Consensus Mechanism (Zab Protocol)

Apache ZooKeeper employs the ZooKeeper Atomic Broadcast (Zab) protocol as its core consensus mechanism to achieve fault-tolerant coordination and ensure consistent state replication across a distributed ensemble of servers. Zab is a leader-based atomic broadcast protocol designed specifically for primary-backup systems like ZooKeeper, providing total order delivery of state changes while supporting high-performance operation in read-heavy workloads. Unlike general-purpose consensus algorithms, Zab emphasizes simplicity, crash recovery, and the ability to handle multiple outstanding transactions without batching, making it well-suited for ZooKeeper's coordination primitives. The Zab protocol operates in three main phases: discovery, synchronization, and broadcast. In the discovery phase, servers elect a leader and establish connections, using a best-effort algorithm to select the server with the most up-to-date history based on quorum votes. The synchronization phase follows, where the newly elected leader brings followers up to date by sending snapshots (SNAP), truncation markers (TRUNC), or differential updates (DIFF) to ensure a consistent prefix of the transaction history; followers then acknowledge with an ACK after processing. Once synchronization completes with quorum acknowledgments, the leader activates broadcast mode, proposing updates to followers for ongoing operations. Leader election in Zab is fast and in-memory, leveraging the FastLeaderElection algorithm to select the server with the highest zxid (ZooKeeper transaction ID) from a quorum, typically completing in under 200 milliseconds to minimize downtime. The process uses epochs to delineate leadership terms, incrementing a 32-bit epoch counter in the 64-bit zxid (composed of epoch and a 32-bit monotonic counter) to ensure uniqueness and ordering across leader changes. During the consensus process, the leader assigns a zxid to each proposed update and broadcasts it to followers, who process and acknowledge (ACK) if it extends their local history; the leader commits the update once a quorum of ACKs is received, guaranteeing its durability and visibility. This quorum-based agreement (majority of servers, e.g., 2f+1 for f faults) ensures that committed operations are irreversible and applied in the same total order across all servers. Fault handling in Zab triggers view changes upon detecting failures via timeouts or lost quorums, prompting servers to revert to the discovery phase for re-election while preserving committed history. The protocol ensures for write operations—meaning each write appears to take effect instantaneously at some point between invocation and response—and for reads and writes, where all processes observe operations in a single global order consistent with their program order. Compared to , Zab differs by recovering full transaction histories rather than single-value instances and incorporating a dedicated phase to resolve causal conflicts without complex ballot numbers.

Use Cases

Configuration Management and Naming

serves as a centralized for data in distributed applications, leveraging its znodes to store small pieces of information, typically in the form of key-value pairs under hierarchical paths such as /config/app, where the znode path acts as the key and its associated data field holds the value. This structure allows applications to maintain a unified view of settings across multiple nodes, avoiding inconsistencies that arise from decentralized file-based or manual propagation methods. Updates to configuration znodes are performed atomically, ensuring that reads and writes occur as indivisible operations, which prevents partial or corrupted states during concurrent modifications. Each znode maintains version numbers for its , children list, and access control lists, requiring clients to specify the expected version in update requests to succeed; a mismatch due to an intervening change results in failure, thus enforcing . Clients can register one-time watches on znodes via operations like getData() or exists(), triggering asynchronous notifications upon changes, which facilitates dynamic reloading of configurations without polling and supports adaptation in running systems. ZooKeeper's naming service utilizes its slash-separated hierarchical to enable and registration, where services can be identified by paths like /services/db1, allowing clients to traverse and query the structure for locating resources. Ephemeral znodes enhance this by tying existence to client sessions; for instance, a service instance creates an ephemeral znode under /services/db1 to register itself and periodically update it via heartbeats, with automatic deletion upon session expiry signaling failure to other participants. In , ZooKeeper stores shared parameters accessible by components like the NameNode, ensuring all nodes operate with synchronized settings for cluster-wide operations. Similarly, in architectures, ZooKeeper's centralized storage mitigates configuration drift by providing a single authoritative source, where services fetch and watch configs to maintain uniformity across deployments. These capabilities deliver a consistent, fault-tolerant view of configurations and names to all participants, reducing errors from manual synchronization and enabling scalable management in large-scale environments.

Synchronization and Leader Election

Apache ZooKeeper provides a set of client-implemented recipes for synchronization primitives, leveraging its hierarchical namespace of znodes and watch notifications to enable distributed coordination without built-in server-side locking mechanisms. These primitives, such as locks, barriers, and queues, are constructed using ephemeral and sequential znodes, ensuring atomic operations and failure detection through session expiration. Leader election, a key synchronization task, follows a similar pattern to select a unique coordinator among participants, preventing issues like split-brain scenarios in clustered systems.

Distributed Locks

ZooKeeper's lock recipe implements a globally synchronous lock to ensure across distributed clients. Clients create ephemeral sequential znodes under a designated lock (e.g., /locks/lock-), where the znode with the lowest sequence number holds the lock. To acquire the lock, a client uses the getChildren() operation to list siblings, identifies the next lowest sequence number, and sets a watch via exists() on that predecessor znode; upon notification of its deletion (due to session expiry or explicit removal), the client checks if it now holds the lowest number. Releasing the lock involves simply deleting the client's znode, which triggers watches for waiting clients and avoids thundering herd effects by chaining notifications sequentially. For shared locks, clients prefix znodes with "read-" or "write-" to allow concurrent readers while excluding writers, adapting the for read-write .

Leader Election

Leader election in ZooKeeper elects a single leader from a group of clients to coordinate tasks, using ephemeral sequential znodes under an election path (e.g., /election/). Each creates a znode like /election/guid-n_, where n is the assigned number, and the lowest-numbered znode designates the leader. Non-leader clients set a watch on the immediate predecessor znode (the one just below their own ) using getChildren() and exists(); if the leader fails, its znode expires, notifying the watcher, who then verifies if it is now the leader by checking for no lower sequences. This design ensures only one client promotes itself at a time, minimizing contention, and ephemeral znodes automatically remove failed . The recipe is fault-tolerant, as session timeouts detect leader crashes, enabling rapid without involvement beyond znode ordering guarantees.

Barriers

Barriers in synchronize a group of es, blocking until all participants reach a common state before proceeding. For a single barrier, clients watch a barrier znode (e.g., /barrier) with exists(); the barrier is lifted when the coordinator deletes the znode, unblocking all watchers. Double barriers, for entry and exit phases, use a barrier with ephemeral znodes (e.g., /b-n/ for the nth ) to track participants; clients create their , then watch the for changes. When the number of children reaches the expected threshold, a "ready" znode is created to signal entry; for exit, clients delete their children and watch until the drops to zero, ensuring ordered progression. This primitive prevents partial execution in distributed computations, such as phased workloads.

Queues and Two-Phase Commit

Distributed process tasks in order using sequential ephemeral znodes under a path (e.g., /queue/), where producers append with create() and consumers retrieve via getChildren() with watches on the to process the lowest-numbered first. extend this by embedding priorities in znode names (e.g., /queue-PRI-X), sorting children lexicographically for higher- items. For two-phase commit, a creates a znode (e.g., /app/Tx) and per-site children (e.g., /app/Tx/s_i); sites vote by writing "commit" or "abort" to their , with all parties watching siblings via getChildren() for unanimous agreement before finalizing. The optimizes by directly notifying sites upon decision, reducing O(n²) watch notifications, though the recipe relies on ZooKeeper's linearizable writes for . In practice, these primitives support critical use cases like HBase's master failover, where handles among master nodes to select an active coordinator, manages server leases to detect region server failures, and ensures single-master operation to avoid during and coordination. All recipes are implemented client-side using , with servers providing strict ordering and broadcasts via the to guarantee consistency.

Client Libraries

Official Bindings (Java and C)

The official Java binding for Apache ZooKeeper is provided in the org.apache.zookeeper package, which forms the core client library for interacting with ZooKeeper ensembles. The primary class in this binding is ZooKeeper, which handles connections to the server, session establishment, and operations such as creating, reading, updating, and deleting znodes. Developers implement the Watcher interface to receive notifications about changes in the ZooKeeper state or specific znodes, enabling event-driven applications. Asynchronous operations are supported through callback mechanisms, where methods like createAsync or getDataAsync queue requests on an internal event thread and invoke user-provided callbacks upon completion. The C binding, known as libzookeeper, offers a procedural that closely mirrors the functionality of the Java binding to ensure equivalence across languages. Key functions include zookeeper_init for initializing a with a specified host string, timeout, and watcher callback, and zoo_create for creating znodes with options like ephemeral or sequential flags. It supports multi-threaded environments through the zhandle_t opaque handle type, which manages I/O threads and event dispatching for concurrent access, with the multi-threaded library (zookeeper_mt) recommended over the single-threaded variant (zookeeper_st) for most use cases. A single-threaded variant, libzookeeper_st, is also available for simpler use cases without threading concerns. Both bindings incorporate robust session management, using a 64-bit and optional for across reconnections, with configurable timeouts ranging from a minimum of twice the server's tick time to a maximum of 20 times that value. Automatic reconnection logic is built-in, allowing clients to maintain sessions by retrying against available servers in the ensemble when disruptions occur. In the Java binding, administrative utilities such as ZooKeeperMain provide a for interactive testing and basic tasks like znode manipulation. Installation of these bindings is straightforward as they are bundled in the official Apache ZooKeeper distribution tarball, available from the project's release page. For , integration requires JDK 8 or later (including LTS versions like JDK 11), with the client library accessible via the provided files. The C binding must be compiled from source using autotools; after extracting the distribution, navigate to the src/c directory, run ./configure, followed by make and optionally make install to build and install libzookeeper.

Community and Third-Party Libraries

The community has developed a range of third-party client libraries for Apache ZooKeeper, extending its usability to additional programming languages and providing higher-level abstractions beyond the official and bindings. These libraries aim to offer idiomatic interfaces, improved error handling, and built-in patterns for common distributed coordination tasks, while maintaining compatibility with ZooKeeper's core . As of November 2025, ZooKeeper's stable version is in the 3.9.x series (latest 3.9.4 released August 2025), and users should verify client compatibility, especially for versions beyond 3.5.x, due to potential changes. In , serves as a prominent high-level client library, inspired by Apache Curator, that simplifies interactions through recipes for locks, leader elections, and queues. It features a unified asynchronous compatible with gevent (version 1.2 or later) and eventlet, enabling non-blocking operations in concurrent applications. implements a pure-Python wire protocol and supports data and children watchers for efficient event handling, with documented compatibility for versions 3.3 through 3.5. The project's last major release was 2.10.0 in January 2024, with limited recent activity; users should test for alignment with 3.9.x. For .NET environments, provides a client library that ports the core , allowing developers to perform operations like creation, data retrieval, and watches in C# applications. It targets .NET Framework 4.0 and above, ensuring broad compatibility within the .NET ecosystem. The library, last updated in 2019 (version 3.4.6.2 based on 3.4.x), has no recent maintenance; for modern .NET support (e.g., .NET 6+), consider forks like vostok/zookeeper.NetEx.fixed, and verify compatibility with 3.9.x. The community utilizes node-zookeeper-client, an event-driven library that mirrors the Java API while adhering to Node.js conventions, including support for promises and EventEmitter for handling connection states and node changes. It enables asynchronous operations such as creating paths, getting children, and setting data, with built-in methods like mkdirp for hierarchical node management. Tested primarily with ZooKeeper 3.4.x, the library's last release was 1.1.1 in July 2021 and is no longer actively maintained; compatibility with ZooKeeper 3.9.x should be verified, and alternative Node.js clients may be preferable. Higher-level abstractions in include , a mature framework that builds on the base client to provide recipes for distributed locks, leader elections, and barriers, reducing for common patterns. incorporates retry policies to handle transient failures and utilities for path construction and manipulation, enhancing reliability in production environments. As an top-level project, it maintains strong compatibility with recent versions, including 3.9.x, through regular releases (latest 5.9.0 as of 2025) and community contributions. Other languages feature specialized wrappers, such as the Foursquare ZooKeeper client, which enhances the API with Scala idioms like functional operations for path creation, recursive deletion, and node monitoring via watchers. In Go, the go-zookeeper library offers a native implementation for ZooKeeper connections, supporting ACLs, sessions, and event handling without external dependencies; its last release (v1.0.4) was in July 2024, and compatibility with ZooKeeper 3.9.x should be confirmed. These libraries emphasize idiomatic error handling and session management, though adoption and update frequency vary by ecosystem.

Adoption and Integrations

Usage in Apache Projects

Apache ZooKeeper serves as an external coordination service in various Apache projects, providing reliable mechanisms for , , and distributed synchronization without being embedded within the applications themselves. This pattern allows projects to leverage ZooKeeper's ensemble for fault-tolerant operations across distributed environments, ensuring and in cluster state. In , enables for the HDFS NameNode through automatic . The ZKFailoverController monitors NameNode health via persistent ZooKeeper sessions and triggers elections to promote a standby NameNode if the active one fails, using exclusive locks to ensure only one active node at a time. It also supports HDFS sharing and job coordination in by facilitating ResourceManager in a similar manner. Apache relies on for critical cluster coordination tasks, including master election to select the active HBase , region server heartbeats to monitor server liveness and manage leases, and management by storing the location of the in a znode for client discovery. Region servers and the register ephemeral znodes in , enabling and ongoing across the distributed database. In Apache Kafka's legacy architecture, ZooKeeper handled broker coordination by storing cluster metadata and broker registrations, topic configuration for dynamic updates and consistency, and controller election to manage partition leadership and state changes. However, since Kafka 3.3 in 2021, the project has transitioned to KRaft mode, a ZooKeeper-free consensus protocol using Kafka Raft for metadata management, with full removal of ZooKeeper support in version 4.0. Other Apache projects integrate ZooKeeper for specialized coordination needs. SolrCloud uses it for leader election among Solr nodes to assign shard leaders and manage distributed indexing state. Apache Spark employs ZooKeeper in standalone mode for high availability, storing cluster state to enable leader election among multiple Masters and recovery after failures, typically recovering in 1-2 minutes. Apache Druid leverages ZooKeeper to manage current cluster state, including service discovery and segment metadata announcements for data loading and querying.

Applications in Commercial and Other Systems

Apache ZooKeeper has seen significant adoption in commercial environments for managing distributed coordination tasks such as , , and . At (now ), ZooKeeper powers the system, a forked and optimized version used primarily for across large-scale , enabling holistic storage and updates without service disruptions. Twitter (now X) leverages ZooKeeper as a service registry to support its naming service for dynamic and in its key-value store, facilitating scalable operations in high-throughput environments. Pinterest employs ZooKeeper for , dynamic , and resilience in its architecture, where it handles node coordination and fault-tolerant updates to support over 500 million users. Rackspace utilizes ZooKeeper in its for coordinating sharding, locking, and resource orchestration in distributed systems like clients and broader environments. Beyond these, other commercial entities have integrated ZooKeeper for similar purposes, including Box for service discovery and Hadoop support, Wealthfront for distributed locking and leader election, and Yahoo for sharding and group membership management, demonstrating its versatility in enterprise-scale deployments. In free software projects outside the Apache ecosystem, ZooKeeper enables high-availability features and clustering. Neo4j uses it in its high-availability components for write-master election and slave coordination to ensure consistent graph database operations. The Akka toolkit incorporates ZooKeeper for cluster management in concurrent, distributed applications, supporting fault-tolerant actor-based systems. Spring Cloud Zookeeper provides integrations for Spring Boot applications, binding configurations and enabling service discovery through ZooKeeper's hierarchical namespace. AdroitLogic's UltraESB relies on ZooKeeper for node coordination and cluster management in enterprise service buses, ensuring scalability and failover. ZooKeeper remains widespread in big data ecosystems, notably through Cloudera Data Platform (CDP), where it serves as the central coordination service for clusters, managing HBase, , and other components in production environments. However, adoption trends show a decline in certain areas; for instance, has fully removed ZooKeeper dependency in version 4.0 released in 2025, transitioning to KRaft mode for management to simplify and improve . Alternatives like etcd are gaining traction for new deployments due to their lighter footprint and Raft-based consensus, particularly in Kubernetes-native systems, though ZooKeeper persists for legacy integrations and established clusters. The Apache ZooKeeper wiki's historical PoweredBy page documents numerous commercial companies and projects, underscoring ZooKeeper's established role in powering thousands of distributed clusters globally.

References

  1. [1]
    Apache ZooKeeper
    Apache ZooKeeper is an open-source server for reliable distributed coordination, maintaining configuration, naming, and providing distributed synchronization.Releases · Documentation · How to Contribute to ZooKeeper · Project
  2. [2]
    [PDF] ZooKeeper: Wait-free coordination for Internet-scale systems - USENIX
    In this paper we discuss our design and implementa- tion of ZooKeeper. With ZooKeeper, we are able to im- plement all coordination primitives that our ...
  3. [3]
    ZooKeeper is a distributed, open-source coordination service for ...
    ZooKeeper is a distributed, open-source coordination service for distributed applications. It exposes a simple set of primitives that distributed ...ZooKeeper: A Distributed... · Design Goals · Guarantees · Implementation
  4. [4]
    Apache ZooKeeper | Dremio
    Initially developed by Yahoo!, Apache ZooKeeper became a top-level project for The Apache Software Foundation in 2008. Over the years, it has evolved, with ...
  5. [5]
    Apache ZooKeeper™ Releases
    The Apache ZooKeeper system for distributed coordination is a high-performance service for building distributed applications.
  6. [6]
    Zookeeper
    Jun 26, 2022 · History. ZooKeeper was originally developed at Yahoo! to streamline the processes running on big data clusters. It was developed to fix the bugs ...
  7. [7]
    Apache ZooKeeper Meets the Dining Philosophers - Instaclustr
    May 9, 2021 · ZooKeeper is a mature technology. It originated as a subproject of Apache Hadoop and evolved to be a top-level project of Apache Software ...
  8. [8]
    ZooKeeper 3.4.0 Release Notes
    These release notes include new developer and user facing incompatibilities, features, and major improvements.Missing: x 2013-2016
  9. [9]
    Release Notes - ZooKeeper - Version 3.6.0
    Release Notes - ZooKeeper - Version 3.6.0. New Feature. ZOOKEEPER-27 - Unique DB identifiers for servers and clients; ZOOKEEPER-1260 - Audit logging in ...
  10. [10]
    Release Notes - ZooKeeper - Version 3.7.0
    Release Notes - ZooKeeper - Version 3.7.0. New Feature. ZOOKEEPER-1112 - Add support for C client for SASL authentication; ZOOKEEPER-3264 - The benchmark ...Missing: x performance
  11. [11]
    Release Notes - ZooKeeper - Version 3.9.4
    ZOOKEEPER-4891 updates logback-classic to 1.3.15 to solve cve issues and slf4j-api to 2.0.13 to meet compatibilty requirement of logback. This could cause ...
  12. [12]
    Update logback to 1.3.15 to fix CVE-2024-12798. - Issues
    Feb 14, 2025 · ZOOKEEPER-4891. Update logback to 1.3.15 to fix CVE-2024-12798 ... I have committed this to master. Switch to desktop version. CancelSuccess.Missing: security | Show results with:security
  13. [13]
  14. [14]
  15. [15]
    ZooKeeper Programmer's Guide - Apache ZooKeeper
    This document is a guide for developers wishing to create distributed applications that take advantage of ZooKeeper's coordination services. It contains ...
  16. [16]
    ZooKeeper 3.2 Performance - Apache Software Foundation
    ZooKeeper 3.2 Performance. Below is a throughput graph of ZooKeeper release 3.2 running on servers with dual 2Ghz Xeon and two SATA 15K RPM drives.
  17. [17]
    ZooKeeper: Because Coordinating Distributed Systems is a Zoo
    ### Summary of ZooKeeper Java Example Content
  18. [18]
  19. [19]
    ZooKeeper Internals
    At the heart of ZooKeeper is an atomic messaging system that keeps all of the servers in sync. Guarantees, Properties, and Definitions.
  20. [20]
    Managing configuration of a distributed system with Apache ...
    In this article, I'm going to show how to use Apache ZooKeeper to design centralized configuration storage for distributed systems as an alternative to file- ...
  21. [21]
    ZooKeeper Recipes and Solutions
    Leader Election. A simple way of doing leader election with ZooKeeper is to use the SEQUENCE|EPHEMERAL flags when creating znodes that represent "proposals ...
  22. [22]
    ZooKeeper Use Cases
    ZooKeeper is a separate service from Flink, which provides highly reliable distributed coordination via leader election and light-weight consistent state ...
  23. [23]
  24. [24]
  25. [25]
  26. [26]
  27. [27]
    ZooKeeper Getting Started Guide
    When a new leader arises, a follower opens a TCP connection to the leader using this port. Because the default leader election also uses TCP, we currently ...ZooKeeper Administrator's Guide · Administrator's Guide · Programmer's Guide
  28. [28]
  29. [29]
  30. [30]
  31. [31]
  32. [32]
    kazoo — kazoo 2.10.0 documentation
    ### Kazoo Features Summary
  33. [33]
    python-zk/kazoo - GitHub
    Kazoo is a high-level Python library that makes it easier to use Apache Zookeeper. kazoo.readthedocs.io. License. Apache-2.0 license.
  34. [34]
    ZooKeeper.Net 3.4.6.2 - NuGet
    May 20, 2015 · ZooKeeper.Net 3.4.6.2 is an Apache ZooKeeper client for .NET, targeting .NET Framework 4.0, and is compatible with higher versions.
  35. [35]
    node-zookeeper-client
    ### Summary of node-zookeeper-client
  36. [36]
    Welcome to Apache Curator | Apache Curator
    Jul 23, 2025 · Apache Curator is a Java/JVM client library for Apache ZooKeeper, a distributed coordination service. It includes a high level API framework and utilities.
  37. [37]
    A ZooKeeper client library in Scala. - GitHub
    ZooKeeper provides a Java client library that's perfectly usable from Scala. This just wraps some idioms and niceties around that library to make it as ...Missing: wrapper | Show results with:wrapper
  38. [38]
    go-zookeeper/zk: Native ZooKeeper client for Go - GitHub
    Native ZooKeeper client for Go. Contribute to go-zookeeper/zk development by creating an account on GitHub.
  39. [39]
    HDFS High Availability Using the Quorum Journal Manager
    Active NameNode election - ZooKeeper provides a simple mechanism to exclusively elect a node as active. If the current active NameNode crashes, another node may ...
  40. [40]
    HBaseUseCases - Confluence Mobile - Apache Software Foundation
    HBase clients use ZooKeeper to find the cluster. Masters and region servers register with ZooKeeper, and the root table location is stored in a znode.
  41. [41]
    Apache Kafka
    Summary of each segment:
  42. [42]
    ZooKeeper Ensemble Configuration :: Apache Solr Reference Guide
    We'll first take a look at the basic configuration for ZooKeeper, then specific parameters for configuring each node to be part of an ensemble.
  43. [43]
    Spark Standalone Mode - Spark 4.0.1 Documentation
    When spark.deploy.recoveryMode is set to ZOOKEEPER, this configuration is used to set the zookeeper URL to connect to. 0.8.1. spark.deploy.zookeeper.dir, None ...Cluster Launch Scripts · Launching Spark Applications · Rest Api
  44. [44]
    ZooKeeper | Apache® Druid
    Apache Druid uses Apache ZooKeeper (ZK) for management of current cluster state. Minimum ZooKeeper versions Apache Druid supports ZooKeeper versions 3.5.x and ...
  45. [45]
    [PDF] Holistic Configuration Management at Facebook - acm sigops
    Oct 4, 2015 · Chubby [6] and ZooKeeper [18] provide coordination services for distributed systems, and can be used to store application metadata. We use Zeus, ...
  46. [46]
    ZooKeeper at Twitter - Blog
    Oct 11, 2018 · Apache ZooKeeper is a system for distributed coordination. ZooKeeper is used at Twitter as the source of truth for storing critical metadata ...
  47. [47]
    PoweredBy - Apache ZooKeeper - Apache Software Foundation
    ### Commercial Companies Using Apache ZooKeeper
  48. [48]
    Spring Cloud Zookeeper
    Spring Cloud Zookeeper provides Apache Zookeeper integrations for Spring Boot apps through autoconfiguration and binding to the Spring Environment.
  49. [49]
    Add a ZooKeeper service | Cloudera on Cloud
    You can add the ZooKeeper service manually. In a production environment, you should deploy ZooKeeper as an ensemble with an odd number of servers.
  50. [50]
    Apache Kafka 4.0.0 Release
    Kafka 4.0 will be fully saying goodbye to ZooKeeper. There will be no support for running in ZK mode, or migrating from ZK mode. This means that administrators ...