Your Kubernetes Controller's Cache: The Silent RAM Buffet Between You and the API Server

This deep dive explains how controller-runtime's informer cache actually works: reconciler reads come from an in-memory store populated via list+watch, not from the API server. That makes reads nearly free but introduces subtle pitfalls — stale reads after writes, hidden linear scans over large object sets, and memory consumption driven by cache scope and indexes. The article walks through the Reflector, delta queue, and Indexer plumbing, clarifying when APIReader is necessary and how to avoid expensive surprises in production clusters.