Multi-tenant LLMgateway in Go
One dispatch seam sits between callers and every model provider. A canonical request model absorbs the differences between OpenAI and Anthropic wire formats, and the streaming engine hands each dialect back exactly as its clients expect. TTFT and TPOT are measured on the seam that serves the traffic, not estimated beside it.
The problem
Two provider dialects that disagree about almost everything: polymorphic tool_calls versus tool_use / tool_result blocks, different image encodings, and a stop field whose type changes shape depending on who you ask. Translating badly means silently dropping content the caller sent.
Streaming makes it harder. A gateway that buffers to translate destroys the latency it exists to protect, and one that translates carelessly emits frames the client SDK refuses to parse.
The architecture
Everything entering the system becomes a CanonReq — the lingua franca between dialects. Anything it cannot model faithfully rides through untouched in a raw passthrough field rather than being lossily reshaped, so translation never silently discards a payload it did not understand.
// every provider enters through one seam
type ProviderAdapter interface {
Dispatch(ctx, *CanonReq) (Stream, error)
}
// ... and leaves in the dialect its client expects
for chunk := range stream {
switch dialect {
case OpenAI: frame(chunk) // near-passthrough
case Anthropic: synthesize(chunk) // message_* lifecycle
}
}On the way out, OpenAI is near-passthrough with a terminal [DONE]; Anthropic gets a synthesised message_start / content_block_* / message_delta / message_stop lifecycle, with tool-call fragments mapped onto input_json_delta. Four backends sit behind the same seam: OpenAI, Anthropic, Bedrock over PrivateLink, and OpenAI-compatible in-VPC vLLM/SGLang pools.
Details that mattered
- Measured, not estimated: a live loopback TCP harness validates framing overhead against a sub-2ms p50 budget — measured p50 lands effectively sub-microsecond, with p99 between 1.5 and 7ms across runs.
- Backpressure by construction: no unbounded buffer between upstream and client, so a slow reader paces the stream instead of growing memory.
- Cancellation stops spend: context teardown propagates to the provider, so a client disconnect stops paying for tokens nobody will read.
- Atomic swaps under live reads: concurrent ~200ms polls were observing truncated JSON mid-write; every write now goes temp-file → rename, with a retry loop for the Windows case where a reader holding the destination blocks the replace.
- Isolation you can verify: a render-time attestation interlock stops a chart from deploying while claiming an egress guarantee it does not actually enforce.