LRU throws away your most expensive knowledge
Least-recently-used eviction is semantically blind. It happily discards the rare, costly answer you'll pay dearly to regenerate, and keeps the cheap trivia everyone asks. There's a smarter way.
Every cache eventually fills up and has to throw something away. The default rule, evict whatever was used least recently, sounds reasonable until you notice what it optimizes for. It keeps the cheap, popular questions and discards the rare, expensive ones. But the expensive answer is exactly the one you don't want to pay the model to regenerate.
In plain words. Crowkis decides what to evict using more than recency: how often it's used, how unique the knowledge is, and how much it cost to produce in the first place. Rare, costly answers are the last to go, and some can be pinned so they never leave.
The effect is a cache that gets smarter as it fills, not dumber. It protects the long-tail domain knowledge that's painful to recompute and lets the cheap, easily-regenerated trivia churn. You keep the answers that would hurt to lose.
The bottom line
Eviction is a value judgment, so make it about value. Keep the expensive, rare knowledge; let the cheap stuff go. Blind LRU has it exactly backwards.