Python Distributed Caching — Core Concepts

Why distributed caching exists

When a Python application runs on a single server, an in-process cache (like a dictionary or functools.lru_cache) works fine. The moment you scale to multiple servers behind a load balancer, each server maintains its own separate cache. This creates three problems:

1. Inconsistency — Server A caches version 1 of a record while Server B caches version 2.
2. Wasted memory — the same data is duplicated across every server.
3. Cold starts — when a new server joins, its cache is empty and it hammers the database.

A distributed cache solves all three by providing a single shared cache that every server reads from and writes to.

How it works

A distributed cache is a separate service (Redis, Memcached, or a cluster of them) accessible over the network. All application servers connect to it as clients.

The typical flow:

1. Application server receives a request.
2. Checks the distributed cache over the network (~0.5–2ms latency).
3. On hit, returns the cached data.
4. On miss, queries the database, stores the result in the distributed cache, and returns.

Redis vs Memcached

Feature	Redis	Memcached
Data structures	Strings, hashes, lists, sets, sorted sets	Strings only
Persistence	Optional (RDB, AOF)	None
Replication	Built-in primary/replica	Not built-in
Max value size	512 MB	1 MB default
Cluster mode	Native clustering	Client-side sharding
Pub/Sub	Yes	No

Choose Redis when you need data structures beyond simple key-value, persistence, or pub/sub. Choose Memcached when you want a dead-simple, high-throughput string cache with multi-threaded performance.

Cache topologies

Single node

One Redis instance. Simple to operate. Single point of failure. Good for non-critical caches where a restart just means temporary cache misses.

Primary-replica

One primary handles writes; one or more replicas handle reads. Improves read throughput and provides failover. Reads from replicas may be slightly behind the primary.

Cluster

Data is sharded across multiple nodes. Each node owns a subset of keys. Scales horizontally for both reads and writes. More complex to operate.

Consistency considerations

Distributed caches introduce network calls, which means:

• Stale reads — a value could be updated in the database but not yet invalidated in the cache.
• Race conditions — two servers might simultaneously read a cache miss and both write to the cache.
• Partial failures — the cache might be reachable from some servers but not others during a network partition.

Most applications accept eventual consistency from their cache layer. For stronger guarantees, combine cache-aside with explicit invalidation on writes.

When to add a distributed cache

• You have two or more application servers sharing the same data.
• Database queries are too slow or too expensive for your traffic volume.
• You need session sharing across servers without sticky sessions.
• API responses need sub-10ms latency that the database can’t provide.

When a local cache is enough

• Single-server deployment with no scaling plans.
• Data that’s unique to each process (per-request memoization).
• Extremely hot data that’s read thousands of times per second — a local L1 cache in front of the distributed L2 cache can reduce network calls.

Common misconception

Adding a distributed cache doesn’t automatically make your application faster. If your cache hit rate is low (because data changes frequently or access patterns are too random), you’re adding network latency to every read without meaningful savings. Measure hit rates before committing to the architecture.

The one thing to remember: distributed caching unifies the cache layer across all your Python servers — but it only pays off when your read patterns are predictable enough to maintain a high hit rate.