This site'sRAG chat
The chat widget on this page, built against a free tier that allows twenty requests a day. Most of the engineering is in what happens when the model is unavailable — because at that quota, unavailable is the normal case rather than the exception.
Designing for the quota
Retrieval is ordinary: portfolio data is chunked and embedded at build time, then scored by cosine similarity at request time. The interesting part is the degradation path. On a 429 the upstream's own RetryInfo is parsed, persisted in Redis with a matching TTL, and used to short-circuit later requests rather than burning round-trips discovering the same exhaustion again.
Below that sits a bank of pre-written answers embedded with the same model. During fallback the user's already-computed query embedding is scored against them at a higher threshold than normal retrieval — a canned answer should only fire when it genuinely matches. When nothing matches, the honest response is a 503 carrying Retry-After derived from the upstream signal, rather than an opaque failure.
Making the budget go further
- Semantic answer cache: clean responses are stored with their question embedding; a new question above 0.90 similarity is served from cache in ~40ms instead of waiting on the model.
- Cheaper vectors: cache embeddings are truncated from 3072 to 768 dimensions using Matryoshka prefix slicing and re-normalised, cutting vector storage 4× while retrieval quality holds at the threshold.
- Automatic invalidation: the index resets on cold start when the hash of all corpus chunks changes, so editing the source content invalidates the cache without a manual flush.
- No self-repetition: a cached answer is suppressed when its text already appears in the active conversation.
- Truncation is disclosed: per-chunk finish reasons are captured, so a response cut off by a token limit says so instead of just stopping mid-sentence.