distributed-cache - Human vs. AI, One Design Twice

Overview

distributed-cache is a distributed, in-memory key/value store in Go: an LRU cache with per-item TTL, a consistent-hashing ring that maps keys to nodes, and an HTTP cluster where each node owns a slice of the keyspace and forwards requests for keys it doesn't own. Nothing about that is novel on its own — it's the standard shape of a sharded cache.

What makes this repo interesting is the method. I hand-coded the entire thing first (internal/human) without any AI assistance, then asked an AI to build its own version of the exact same system (internal/ai) behind an identical interface. Both packages expose New(port, peers) and Start(), both have a cache.go, hash_ring.go, and server.go with the same responsibilities, and main.go picks which one to boot with an -impl flag. The goal was never to ship a cache — it was to write a real system myself, then read a second competent take on it side by side and learn from the diff: where the AI reached for a data structure I didn't, where it caught a bug I'd shipped, and where its "better" version bought speed with complexity I should be able to name.

This post walks the architecture through both lenses. The standard library does all the work — no external dependencies.

Architecture

The shared interface and the flag switch

The whole comparison only works because both packages are interchangeable at the boundary. main.go defines a one-method cacheServer interface and constructs the human or AI implementation from the same flags.

// cacheServer is the common interface both packages satisfy: New(port, peers)
// builds one, Start() runs it. main just picks which package to construct from.
type cacheServer interface {
	Start()
}

func newServer(impl, port string, peers []string) cacheServer {
	switch impl {
	case "ai":
		return ai.New(port, peers)
	default:
		return human.New(port, peers)
	}
}

Keeping the surface this narrow was a deliberate constraint. If the two versions diverged in how you invoke them, any performance or correctness difference would be muddied by API differences. Same input, same entrypoint, same lifecycle — so every remaining difference is genuinely about the implementation.

The local store: one lock vs. thirty-two shards

My hand-written cache is the textbook LRU: a map[string]*list.Element for O(1) lookup, a container/list for recency ordering, and a single sync.Mutex guarding both. Set evicts from the back when full; Get moves the touched element to the front and lazily drops expired items.

func (c *Cache) Get(key string) (string, bool) {
	c.mu.Lock()
	defer c.mu.Unlock()
	elem, found := c.items[key]
	if !found {
		return "", false
	}

	if time.Now().After(elem.Value.(*entry).value.ExpriryTime) {
		c.eviction.Remove(elem)
		delete(c.items, key)
		return "", false
	}

	c.eviction.MoveToFront(elem)
	return elem.Value.(*entry).value.Value, true
}

This is correct and easy to reason about, but every operation — including reads — takes the same global lock, so the cache serializes all traffic through one mutex. The AI version attacks exactly that. It splits the store into 32 independent shards, routes each key to a shard by hash, and holds no top-level lock at all:

// shardFor picks a shard by hashing the key. fnv is fast and has good spread;
// masking with (shardCount-1) works only because shardCount is a power of two.
func (c *ShardedCache) shardFor(key string) *cacheShard {
	h := fnv.New32a()
	h.Write([]byte(key))
	return c.shards[h.Sum32()&(shardCount-1)]
}

Two goroutines now contend only when their keys land in the same shard. That's the single biggest throughput win under concurrent load, and it's the kind of move that's obvious in hindsight but that I didn't reach for on the first pass — I optimized for "clearly correct" over "scales under contention."

TTL expiry: scanning everything vs. a min-heap

The difference I found most instructive was expiry. My background collector wakes on a ticker and walks the entire keyspace to find expired items — O(n) per sweep regardless of how many keys actually expired.

for range ticker.C {
	c.mu.Lock()
	now := time.Now()
	for key, item := range c.items {
		if now.After(item.Value.(*entry).value.ExpriryTime) {
			c.eviction.Remove(item)
			delete(c.items, key)
			itemsEvicted += 1
		}
	}
	c.mu.Unlock()
}

The AI kept a per-shard min-heap ordered by expiry time. The collector only pops entries that are actually due and stops the moment the earliest expiry is still in the future, so cleanup cost is proportional to what expired, not to cache size.

func (s *cacheShard) collectExpired() int {
	s.mu.Lock()
	defer s.mu.Unlock()

	now := s.clock.Now()
	evicted := 0
	for s.expiry.Len() > 0 {
		top := s.expiry[0]
		if top.expiresAt.After(now) {
			break // earliest expiry is still in the future -> nothing else is due
		}
		heap.Pop(&s.expiry)
		if top.elem == nil {
			continue // already removed; stale heap node
		}
		if elem, ok := s.items[top.key]; ok && elem == top.elem {
			s.removeElem(elem)
			evicted++
		}
	}
	return evicted
}

The heap isn't free — it adds a push on every Set and requires lazy deletion: when a key is overwritten or LRU-evicted, its stale heap node is tombstoned (its elem set to nil) and skipped when popped. That's a classic space-and-bookkeeping-for-time trade, and being able to state it precisely was half the point of the exercise.

Consistent hashing: one point per node vs. virtual nodes

Both hash rings do the same core thing — hash nodes onto a ring, keep the hashes sorted, and binary-search for the first node clockwise of a key's hash. My GetNode is already O(log n) via sort.Search, with hashes computed once at add-time rather than rehashing on every lookup.

func (h *HashRing) GetNode(key string) Node {
	h.lock.RLock()
	defer h.lock.RUnlock()

	if len(h.hashes) == 0 {
		return Node{}
	}

	keyHash := h.hash(key)
	i := sort.Search(len(h.hashes), func(i int) bool { return h.hashes[i] >= keyHash })
	if i == len(h.hashes) {
		i = 0 // wrap around the ring
	}
	return h.nodeAt[h.hashes[i]]
}

The gap the AI closed here is distribution, not asymptotics. My ring places each physical node at exactly one point, hash(node.ID). With a handful of nodes that produces very lumpy ownership — a single node can own a huge arc of the keyspace purely by where its one hash landed, and removing it dumps all of those keys onto one neighbor. The AI version places each node at ~100 virtual points and averages them out:

func (h *ConsistentHashRing) AddNode(node Node) {
	h.lock.Lock()
	defer h.lock.Unlock()

	for i := 0; i < h.replicas; i++ {
		hash := h.hash(virtualKey(node.ID, i))
		if _, taken := h.ownerOf[hash]; taken {
			continue // extremely rare collision; skip this replica
		}
		h.ownerOf[hash] = node
		h.sorted = append(h.sorted, hash)
	}
	slices.Sort(h.sorted)
}

Many points per node smooths ownership toward even and means removing a node scatters its keys across all remaining nodes instead of one. The cost is ~100 ring entries per node in memory — again, a nameable trade rather than a free lunch.

Coordination: routing a request to its owner

The server layer turns HTTP into ring lookups plus local cache ops plus peer forwarding. A POST /set asks the ring who owns the key; if that's us we store locally and replicate to peers in the background, otherwise we forward. The AI's SetHandler is the clean expression of that flow:

func (cs *Server) SetHandler(w http.ResponseWriter, r *http.Request) {
	var req setRequest
	if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
		http.Error(w, err.Error(), http.StatusBadRequest)
		return
	}

	// A replication request is authoritative: store it locally and stop.
	if r.Header.Get(replicationHeader) != "" {
		cs.cache.Set(req.Key, req.Value, cs.ttl)
		w.WriteHeader(http.StatusOK)
		return
	}

	owner := cs.ring.GetNode(req.Key)
	if owner.ID != cs.self.ID {
		cs.forward(w, owner, r)
		return
	}

	cs.cache.Set(req.Key, req.Value, cs.ttl)
	go cs.replicateSet(req.Key, req.Value)
	w.WriteHeader(http.StatusOK)
}

Every request takes exactly one path and writes its response exactly once. Replication requests carry a header that marks them authoritative so they store verbatim without re-routing or re-replicating — which is what keeps writes from bouncing around the cluster forever. That header discipline is the load-bearing detail of the whole coordination layer.

Difficult Parts

The bug the AI caught in my server

The most humbling part of the exercise: the AI found a real correctness bug in my SetHandler. My version routes the key, and in the owner-vs-forward branch it handles both cases — but then, after that if/else, it unconditionally stores locally, replicates, and calls WriteHeader a second time.

targetNode := cs.hashRing.GetNode(req.Key)
if targetNode.Addr == "self" {
	cs.cache.Set(req.Key, req.Value, 1*time.Hour)
	// ...
	w.WriteHeader(http.StatusOK)
} else {
	cs.forwardRequest(w, targetNode, r)
}
// (no early return above — so a forwarded write ALSO falls through here)

The consequence is that every forwarded write gets stored on the wrong node too, and every request writes its HTTP status twice. The fix is trivial once you see it — an early return in each branch — but I'd read that handler several times and never noticed, because the happy path (a key you own) works fine. It only misbehaves on the forwarding path, which my quick manual tests didn't exercise. It was a good reminder that "the code I wrote and skimmed" and "the code that's correct on every branch" are different claims.

Deciding ownership by ID, not by a magic string

My server decides "do I own this key?" by checking targetNode.Addr == "self" — a sentinel I stuff into the ring for the local node. It works, but it's brittle: I add peers to the ring by their address string while identifying myself with a separately-generated selfID, so the keys the ring is built from and the keys routing compares against can quietly disagree. The AI made every node use its own URL as both its ring ID and its dial address, and compares on owner.ID != cs.self.ID. IDs are unambiguous identity; addresses are for dialing — conflating them is exactly the kind of shortcut that works in a two-node test and breaks in a real cluster.

HTTP client hygiene under forwarding

Forwarding means every node is also an HTTP client to its peers, and that's easy to get subtly wrong. Both versions eventually share a single *http.Client with a 3-second timeout — critical, because a per-call client throws away connection pooling and a client with no timeout lets one dead peer hang a handler forever. The other trap is response bodies: you have to drain and close every response (including replication responses you don't read) or the connection never returns to the pool. Getting forwarding to loop-proof also took an X-Forwarded-For guard so a misrouted request bails with a 508 instead of ping-ponging between two nodes indefinitely.

Making time testable

A quieter thing I took away from the AI version: it hides time.Now() behind a Clock interface so expiry logic can be tested deterministically instead of with time.Sleep. My tests lean on real sleeps, which are slow and flaky. Injecting a fake clock that advances on demand is a small change with an outsized payoff in test reliability — the sort of discipline that's easy to skip when you're the one racing to a working prototype, and easy to appreciate when you read someone else's version that didn't skip it.