CSTALE: serve the slightly-old answer now, refresh it behind the scenes
A hard TTL turns a one-second-expired answer into a full model call. CSTALE serves the cached answer past its TTL with a stale flag, so you choose freshness versus latency per request.
Hard expiry is a blunt instrument: the moment an entry's TTL passes, the next request eats a full model call, even if the answer is one second stale and perfectly usable. CSTALE adds the stale-while-revalidate pattern that CDNs have used for years, it returns the cached answer even past its TTL, marked with a stale flag so the caller knows.
In plain words. Instead of a just-expired answer forcing a slow fresh call, Crowkis can hand back the slightly-old one instantly (flagged as stale) and refresh in the background, you decide which answers can tolerate that.
That flag is the control surface. For a dashboard number or a chat reply, a few-seconds-stale answer served instantly is the right call, and you ignore the flag. For a price or a balance, you treat the flag as a signal to kick off a refresh, serve the stale value for snappiness while the fresh one computes, or block for the fresh one when correctness outranks speed. The policy is yours, per request.
flowchart TD Q["incoming query"] --> I["intent classifier"] I --> T["template match"] T --> V["HNSW neighbours"] V --> C["confidence gate"] C --> TR["trust + freshness"] TR -- pass --> A["answer · <1ms"] I -. veto .-> M["(nil) → your model"] T -. veto .-> M V -. veto .-> M C -. veto .-> M TR -. veto .-> M style A fill:#fbe9e8,stroke:#d62221,stroke-width:2.5px style M fill:#f3eee5
The win is that expiry stops being a latency cliff. Instead of every TTL boundary becoming a synchronous model call for some unlucky user, the stale answer covers the gap while the cache refreshes, and the p99 that a hard TTL would spike stays flat.
The bottom line
Freshness and latency are a dial, not a switch. CSTALE hands you the dial, most answers can tolerate a moment of staleness in exchange for never paying the cold-miss tax, and the ones that can't are exactly the ones the flag lets you treat differently.