NVIDIA Advances AI Infrastructure With Disaggregated LLM Inference on Kubernetes

NVIDIA has published detailed technical guidance for deploying disaggregated large language model inference workloads on Kubernetes, a development that could reshape how enterprises manage GPU resources for AI applications. The approach, outlined by NVIDIA engineer Anish Maddipoti, separates the computationally distinct prefill and decode stages of LLM inference into independent services that can scale and optimize separately.

The timing matters. NVIDIA entered production with Dynamo, its inference operating system for AI factories, just last week on March 16. With NVDA stock trading at $176.21 as of March 23—up 2.6% in 24 hours and carrying a $4.26 trillion market cap—the company continues expanding its software ecosystem to complement its dominant hardware position.

Why Disaggregation Changes the Economics

Traditional LLM inference runs both stages on the same hardware, forcing GPUs to alternate between fundamentally different workloads. Prefill—processing the input prompt—is compute-intensive and benefits from high FLOPS. Decode—generating tokens one at a time—is memory-bandwidth-bound and benefits from fast HBM access.

“A single monolithic serving process starts to hit its limits,” Maddipoti writes. By splitting these stages, operators can match GPU resources to each stage’s actual needs rather than compromising on a single approach.

Three practical benefits emerge: different optimization profiles per stage, independent scaling based on actual demand patterns, and better GPU utilization since each stage can saturate its target resource.

The Scheduling Problem

Disaggregation creates orchestration complexity. NVIDIA’s guidance centers on KAI Scheduler, which handles three critical capabilities: gang scheduling (all-or-nothing pod placement), hierarchical gang scheduling for multi-level workloads, and topology-aware placement to colocate tightly coupled pods on nodes with high-bandwidth interconnects like NVLink.

The company’s Grove API allows operators to express all roles—router, prefill workers, decode workers—in a single PodCliqueSet resource. This handles startup dependencies, per-role autoscaling, and topology constraints declaratively rather than through manual coordination.

“Placing a Tensor Parallel group’s pods on the same rack with high-bandwidth NVIDIA NVLink interconnects can mean the difference between fast inference and a network bottleneck,” Maddipoti notes.

Scaling Gets Complicated

Autoscaling disaggregated workloads operates at three levels: per-role, per-Tensor-Parallel-group, and cross-role coordination. The Dynamo planner runs separate prefill and decode scaling loops targeting Time To First Token and Inter-Token Latency SLAs respectively, using time-series models to predict demand.

This matters because there’s an optimal ratio between prefill and decode capacity that shifts with request patterns. Scale prefill 3x without scaling decode and the extra output has nowhere to go—decode bottlenecks and KV cache transfer queues up.

NVIDIA will demonstrate the full stack at KubeCon EU 2026 in Amsterdam, where the company plans to present an end-to-end open source AI inference reference architecture at booth 241.

Image source: Shutterstock