Key Value Stores

List of common Key-value datastore for data engineering

Redis

In-memory key-value store with many use cases

Redis is an open-source in-memory data store that can be used as a database, cache, or message broker. It is often used for caching, reducing the load on servers. It offers excellent performance, with the ability to quickly read and write data to memory.

In Redis, a key-value pair is a data structure that consists of a unique key, which is used to identify the data, and a value, which is the data itself. Key-value pairs are the most basic data structure in Redis, and they are used to store and manage data in the database.

• Lightning-Fast Data Access: Redis stores data entirely in memory, which means it can provide sub-millisecond response times.
• Fast writes
• Can handle millions of operations per second

Drawback:

• It is a single-threaded system, which means that it can only process one command at a time. This can limit the performance and scalability of Redis in applications that require high concurrency and parallelism. The I/O threading added in v6.0 helps, but under heavy loads, that core thread can still bottleneck.

Cassandra

Highly available distributed Key-Value store

Cassandra is a NoSQL database designed for handling large amounts of data across many commodity servers, providing high availability without sacrificing performance.

• Scalability: The capability to expand by adding nodes without interrupting operations is crucial for handling growing data demands. Cassandra’s architecture supports continuous growth while maintaining performance stability. Cassandra DB is its horizontal scalability. Cassandra scales linearly.
• Fast writes — Cassandra is known to have a very high throughput, not hindered by its size. 1 million writes per second.
• Fault Tolerance: Its decentralized nature and replication strategies minimize downtime and data loss, making it ideal for business-critical applications where data integrity and availability are non-negotiable.
• Real-Time Performance: Cassandra’s low-latency reads and writes are advantageous for applications that require real-time analytics and quick response times. The database’s performance ensures that data is processed and accessed swiftly, meeting the needs of time-sensitive applications.
• Data Distribution: By distributing data across multiple geographic locations, Cassandra enhances disaster recovery capabilities and provides geographical redundancy. This distribution ensures that data remains available and resilient, even in the face of regional failures.
• Apache Cassandra is written in Java
• Used at large techs like Apple, Discord, Activision, Best Buy, and Netflix.

Drawbacks

• While Cassandra provides superior performance in write operations, read operations are sub-optimal.
• Latency issues are a common problem when managing large amounts of data and requests.
• Cassandra also does not provide support for aggregates and subqueries.
• Cassandra stores the same data multiple times as it is based on queries.
• Cassandra has Java Memory Model (JVM) issues.

DynamoDB

Storing data temporarily for faster retrieval

Amazon DynamoDB is a cloud-native NoSQL primarily key-value database. DynamoDB is cloud-native and runs on Amazon Web Services (AWS).

Why use DynamoDB?

• Managed — provided ‘as-a-Service’ so users would not need to maintain the database
• Scalable — automatically provision hardware on the backend, invisible to the user
• Fast — single-digit millisecond response times. With DAX, it can provide microsecond response times for cached read-heavy workloads
• Durable and highly available — multiple availability zones for failures/disaster recovery
• Flexible — make it easy for users to get started and continuously evolve their database

Drawbacks

• Limited query support
• Complex pricing structure (Indexing, multi-region setup adds cost), DynamoDB can be very expensive
• No joins support
• Vendor lock-in, It is only available in AWS.
• Throughput Constraints — We must provide read and write capacity units when using DynamoDB’s provisioned throughput mechanism.
• The design of DynamoDB prioritizes availability and partition tolerance, yet in some cases, consistency may be lost.

ETCD

Key value store with no single point of failure

ETCD is a strongly consistent, distributed key-value store that provides a reliable way to store data that needs to be accessed by a distributed system or cluster of machines. Data is stored in the form of hierarchical directories forming a standard file system.

Why use etcd?

• Fully replicated: Every node in an etcd cluster has access to the full data store.
• Highly available: etcd is designed to have no single point of failure and gracefully tolerate hardware failures and network partitions.
• Reliably consistent: Every data ‘read’ returns the latest data ‘write’ across all clusters.
• Fast: etcd has been benchmarked at 10,000 writes per second.
• Secure: etcd supports automatic Transport Layer Security (TLS) and optional secure socket layer (SSL) client certificate authentication.
• It gracefully handles leader elections during network partitions and can tolerate machine failure, even in the leader node.
• etcd is consistent due to the Raft algorithm.
• Raft for consensus, gRPC for communication, BoltDB as its K-V storage backend, WAL (Write-Ahead Logging) for data durability and integrity.
• Battle-tested at Kubernetes, CoreDNS, Rook, M3

The name “etcd” originated from two ideas, the unix “/etc” folder (place to store configuration data) and “d”istributed systems.

Drawbacks:

• With more nodes, writes to etcd becomes slower
• Defragmentation issues
• Recommended maximum of size limit 8GB data

Zookeeper

Solution for distributed system coordination and metadata storage

Apache ZooKeeper is a distributed coordination service that manages configuration, synchronization, and naming for large, distributed systems. It provides a centralized infrastructure to maintain and coordinate distributed systems by offering key services like configuration management, naming registry, and leader election, ensuring fault tolerance and consistency.

At its core, ZooKeeper follows a simple architecture, offering an ensemble (a set of ZooKeeper servers), with one leader and several followers. Clients connect to these servers to read/write data stored in a hierarchical namespace, similar to a file system. The core components include nodes (called `znodes`) that hold the metadata, states, and information needed for coordination.

• Leader Election: ZooKeeper can select a leader among multiple services for tasks that need coordination. This is essential in distributed systems where only one component must take control (e.g., to write to a shared resource).
• Service Discovery: ZooKeeper keeps track of all nodes in a system, acting as a registry. If a node goes down, ZooKeeper updates the system, allowing other services to adjust.
• Configuration Management: It stores configuration information in `znodes`. Clients can read the latest configurations and get notified of any changes in real-time.
• Distributed Locks: ZooKeeper provides mechanisms for distributed locks, ensuring that multiple services don’t try to access the same resource at the same time.
• Quorum Consensus: It uses a consensus mechanism for state replication, providing fault tolerance.
• High Availability: ZooKeeper is highly reliable due to its leader-follower architecture, offering availability even if some nodes fail.
• Consistency Guarantees: ZooKeeper guarantees that reads are served consistently, even if some servers are unavailable.
• Fault Tolerance: It remains operational in case of server failures by using quorum consensus to ensure no single point of failure.
• Scalability: It scales well with the size of the cluster and the number of client nodes, allowing it to manage increasingly large distributed systems.
• Notification Mechanism: It notifies clients of changes in the system, allowing real-time updates and action based on state changes.
• Simplified Coordination: By abstracting complex synchronization issues, ZooKeeper simplifies building distributed applications, saving developers time and effort.

Real-World Use Cases of ZooKeeper

• Druid, Flink, Kafka (earlier versions)
• Cassandra, HBase, Ignite, Pulsar

Drawbacks:

• Scalability Issues: Zookeeper is not horizontally write scalable; all writes go through the leader, which can become a bottleneck.

ScyllaDB

Monstrously Fast + Scalable

ScyllaDB offers a similar architecture, data format, and query language as Cassandra, but without Java and its expensive GC pauses. We can improve performance at scale with fewer nodes, reduced administration, and lower infrastructure cost.

Why use ScyllaDB?

• ScyllaDB is written in C++, no JVM overheads
• Fast, scalable, high performance & low latency (sub-millisecond)
• Workload prioritization on a shared cluster
• Unified Row-based cache
• Support global secondary index
• Supports DynamoDB-compatible APIs
• Incremental compaction strategy (ICS) to enhance Cassandra’s existing Size-tiered Compaction Strategy (STCS)
• Server-side scripts using Lua
• Self-tuning
• Trusted at Expedia, Discord, Comcast Rakuten

Apache Ignite

Fastest key-value database that stores data both in memory and on disk

Apache Ignite is an open-source memory-centric distributed platform. We can use it as a database, a caching system for the in-memory data processing.

Ignite is an in-memory computing platform that includes an in-memory data grid (IMDG), in-memory database (IMDB), support for streaming analytics, and a continuous learning framework for machine and deep learning.

Ignite is typically used to:

• Add speed and scalability to existing applications
• Build new, modern, highly performant, and scalable transactional and/or analytical applications
• Build streaming analytics applications, often with Apache Spark, Apache Kafka, and other streaming technologies
• Add continuous machine and deep learning to applications to improve decision automation
• Apache Ignite key-value APIs provide additional capabilities such as automatic synchronization with external databases, near caching, and ACID transactions.
• Apache ignite is introducing RAFT-based replication in version 3.x.

Notable mentions

• FoundationDB (Used at Apple, Snowflake, Firebolt, Datadog, Epic Games)
• DragonFly
• DataStax
• Hazelcast
• RonDB