A million vectors, still instant: the search engine we wrote in Rust
Finding the nearest meaning among a million cached answers, in under a millisecond, without a single external dependency. A look at the pure-Rust HNSW engine underneath Crowkis.
Every cache hit starts with a search problem: given a new question's meaning, find the closest question we've already answered, out of possibly millions. Do it slowly and the cache is pointless; the model call would've been faster. So the search has to be effectively instant, and it has to stay instant as the cache fills up.
In plain words. HNSW is the algorithm most serious vector search uses. Picture a graph where similar items are linked; you start anywhere and 'walk downhill' toward the query, arriving in a handful of hops instead of scanning everything.
We wrote ours from scratch, in Rust, with no C libraries bolted on. That sounds masochistic until you see the payoff: the search understands that a cache entry isn't just a vector, it carries a trust score, a freshness stamp, an intent template. Owning the engine means the reuse checks happen right next to the data, not across five foreign API calls per lookup.
The result: at a million cached meanings, a search still returns in well under a millisecond and almost never misses the true nearest neighbour. Because the whole engine is pure Rust with the model bundled in, the entire thing ships as one small container, nothing to download at runtime, nothing external to trust.
The bottom line
Fast, accurate, self-contained, pick three. The cache in the hot path of every LLM call can't afford to be slow, and it can't afford to hallucinate a neighbour. Owning the search engine is how you get both, and it's why Crowkis fits in one file.