Closed-loop SGLanginference tuning

The controller originally compared a window mean against its latency target. On real traffic that reads as healthy while users are actively being hurt: the mean sat at 25.5ms inside a 40ms target while p95 was 80.9ms against an 80ms budget. The tail was in violation and the mean never noticed.

The same distortion showed up in prefill-length variance — a mean of roughly 400k sat comfortably under the threshold while p95 measured over 4.1M. Averaging a skewed distribution deletes exactly the signal you are trying to control on.

The loop tried to drain a backlog by growing the chunk size. But under closed-loop load, queue depth is set by client concurrency — not by chunk size — so backlog was an unreachable setpoint. The controller pushed against something it structurally could not move, and read the lack of response as a reason to push harder.

That is a control defect, not a tuning one, so no threshold adjustment would have fixed it. The fix was to change what counts as evidence: a move now has to prove a ≥5% prefill-throughput improvement or it gets reverted, with the load state captured at decision time so a protective move is not blamed for a rise in offered load.

• Real server, real traffic: a live smoke test drives an actual SGLang server and caught two control-path bugs that unit fakes had hidden by encoding contracts the engine never emitted.
• Reproducible load: five named workload shapes driven with exact input token counts, so variance comes from the system rather than the generator.
• Replayable decisions: telemetry capture records the controller's own measurement window, letting a decision be replayed byte-identically after the fact.
• Bidirectional width: prefill batch width was a one-way ratchet; it now widens as well as narrows, seeded from the observed peak so a default-launched server has a lever at all.