What is a semantic cache for LLMs? (and why exact-match caching fails)

Your users ask the same things all day, phrased a hundred ways. 'How do refunds work?', 'what's the refund window?', 'refund timeline?' — one intent, three strings. A normal cache keys on the exact bytes, so it treats all three as different questions and you pay the model three times. That's the gap a semantic cache closes.

#Why not just a vector database?

Raw vector similarity is necessary but not sufficient. 'Cancel my subscription' and 'pause my subscription' sit close together in embedding space, yet serving one as the other is a customer-facing mistake. Similarity alone over-serves. A cache needs to know when 'similar' is actually 'safe to reuse.'

Crowkis does a dual lookup: an HNSW embedding search for meaning, plus a structural intent-template match across 12 intent classes. A hit needs both to agree, so paraphrases match while a changed number, entity, or negation does not.

flowchart TD
  Q["incoming query"] --> E["embed + find neighbours"]
  E --> T["structural intent match"]
  T --> C["confidence gate"]
  C --> A["safe reuse, sub-millisecond"]

#The three commands to know

CSET "how do refunds work?" "Refunds take 5-7 business days."
CGET "what's the refund timeline?"   # a paraphrase, still a hit
CSIM "how do refunds work?" "refund process?"   # -> similarity score

On repetitive workloads, where a large share of traffic is repeats or rephrasings, a semantic cache can cut LLM costs up to 60–70%. The exact figure depends on how repetitive your traffic is, but the mechanism is simple: stop paying twice for an answer you already have.

Exact-match caching is blind to wording. Vector-only caching is careless about meaning. A semantic cache has to be both precise and safe.