Multi Node LLM Serving at Production RPS

Serving an LLM on one or two GPUs can look like a model-server problem. At that size, most tuning happens inside one inference engine: batching behavior, scheduler settings, tensor parallelism, memory limits, and model/runtime choices. But at some point the next useful improvement no longer comes from another engine setting. The decision changes: when do you stop tuning one instance and redesign the serving layer around it? This talk follows the growth path from a 1-2 GPU deployment to several multi-node self-hosted vLLM and SGLang systems. It shows where the problem changes: from tuning a single engine, to choosing between tensor parallelism and data parallelism, to managing routing, load balancing, streaming behavior, KV cache locality, L3 cache, admission control, and rate limits around the model servers. The center of the talk is not a catalog of engine settings, but a decision map for recognizing when the bottleneck has moved into the serving layer. Production cases anchor the talk. First, we tuned both the inference engine and the proxy/connection layer to keep user-visible TTFT p95 under 3 seconds at around 2000 RPS, including bottlenecks that were outside the model server. Second, we increased useful throughput by roughly 1.5x with admission control instead of letting overload destroy latency for everyone. Third, we served frontier coding models under limited resources by combining router-level cache locality with Mooncake/L3 cache to reach 95%+ KV cache reuse. The main lesson is that multi-node LLM serving has to be designed around production load shape, not only around model size or benchmark throughput. That design comes with trade-offs: router-level cache locality can improve reuse while creating stickiness and fairness problems; admission control can protect tail latency while rejecting or delaying some traffic; and moving logic out of the engine can improve throughput while making debugging and operations harder. Attendees will leave with a practical map for moving beyond one LLM inference instance: what to keep inside the engine, what belongs in the serving layer, and how to decide when routing or admission control is the right lever.
Format: TECHNICAL TALK
Location: ROOM A
Speakers: Daniil Ponizov
Level: Intermediate to Advanced
Audience: Registered participants