gallery unit · throughput & caching · bottleneck class C

Silent cache-miss guardrail

A cache with a key bug still returns the right answer. It just never hits — and nothing tells you.

Why the functional test can’t see it

• The answer is computed from the query, not the cache. A wrong cache key never produces a wrong result — on a miss the system recomputes correctly. So a test that checks outputs passes whether the cache hit or not.
• The failure is silent by design. Below the minimum length the provider declines to store and returns no error; a volatile field in the prefix breaks exact matching so the key is never identical twice. Either way: correct output, full cost, no signal.

Correctness cannot see cost. The guardrail asserts a different property: on a warm workload — one where requests are known to recur — the cache-hit ratio must clear a floor derived from the repetition structure. A cache that never hits reads 0.00 and is flagged, in about a microsecond.

bug: cache key = f"{timestamp}:{prompt}"   ->  key differs every request  ->  0% hits  ✗
fix: cache key = canonical(prompt)         ->  same key for same prompt   ->  95% hits  ✓

guardrail: on a warm workload, assert hit_ratio >= floor

Evidence

Tier 4 — one warm workload run through four key configs, scored by the guardrail and by a real functional correctness test. The functional test is the sharp part: the thing teams trust to validate a cache passes every silently-broken config. Anchored on Python’s own functools.lru_cache — a real off-the-shelf cache — whose cache_info() shows the identical miss (0 hits / 240 on the volatile key). Synthetic workload; the repetition is known by construction, stated on the page.

make setup && make test && make run   # $0, laptop, no GPU, no network

Honest gaps

The cache models the two documented provider rules (exact-prefix match, minimum cacheable length) rather than integrating with a live provider. The guardrail catches total silence — 0% — cleanly; partial degradation (say 95% → 60% after a config change) needs a tuned floor or trend tracking, not a fixed one. And the workload’s repetition is known here by construction; a production stream’s warmth has to be measured before the floor means anything.