モデルサービングにおけるルーティングの現状State of Routing in Model Serving

Routing has quietly become one of the most consequential layers in modern machine learning infrastructure, and Netflix's overview of its current state highlights why teams operating models at scale increasingly treat traffic management as a first-class engineering concern rather than an afterthought. As organizations move from serving a single model to orchestrating dozens or hundreds of model versions, replicas, and variants simultaneously, the question of how a given inference request reaches the right compute resource directly shapes latency, cost, and reliability.

At its core, model serving routing is the logic that decides which model instance handles an incoming request. In simple deployments this may be a basic round-robin distribution across identical replicas behind a load balancer. But production ML systems rarely stay simple. A recommendation or personalization stack like Netflix's may run multiple model versions concurrently for A/B testing, shadow deployments, canary releases, and gradual rollouts. Each of these patterns imposes different routing requirements: experiments need deterministic assignment of users or sessions to variants, canaries need a controlled percentage of live traffic, and shadow tests need traffic to be mirrored to a new model without affecting the user-facing response.

Load balancing sits underneath much of this. The naive assumption that requests are interchangeable breaks down quickly in inference workloads, where the cost of a request can vary dramatically depending on input size, sequence length, or batch composition. Distributing traffic evenly by request count does not necessarily distribute it evenly by compute. As a result, more sophisticated strategies attempt to balance based on actual load signals such as queue depth, GPU utilization, or observed latency, steering requests away from saturated replicas. Approaches like least-outstanding-requests or latency-aware routing tend to outperform simple round-robin when request costs are heterogeneous, though they add complexity and require reliable telemetry.

Latency optimization is a recurring theme because inference is often on the critical path of user-facing experiences. Routing decisions interact heavily with batching, since many serving frameworks group requests together to use accelerators efficiently. There is an inherent tension here: larger batches improve throughput and hardware utilization but can increase tail latency, while routing for the lowest individual latency can underutilize expensive GPUs. Effective systems appear to treat routing, batching, and autoscaling as a coordinated problem rather than three independent knobs. Locality also matters, as routing requests to replicas that already have the relevant model loaded in memory avoids costly cold starts and model-loading delays.

The broader context is an ecosystem of tools that has matured around these challenges. Open-source serving frameworks such as NVIDIA Triton Inference Server, KServe, Ray Serve, and TorchServe provide building blocks for deployment, while service meshes like Envoy and Istio offer general-purpose traffic routing that ML teams sometimes adapt. More recently, the rise of large language models has intensified interest in specialized routing, including techniques that direct queries to different models based on cost or difficulty, and KV-cache-aware routing that keeps related requests on the same instance to reuse computation. Netflix's discussion reflects a setting where many of these concerns predate the current LLM wave, rooted instead in years of large-scale personalization and recommendation serving.

Routing also functions as a control point for operational safety. By centralizing traffic decisions, teams can implement gradual rollouts, instant rollbacks, circuit breaking, and fallback to a known-good model when a new version misbehaves. This makes routing a key part of the MLOps lifecycle, connecting model deployment to monitoring and experimentation systems. The ability to shift traffic precisely and observe the effect is what allows safe iteration on models that are difficult to fully validate offline.

What emerges from this overview is that routing is less a solved problem than an evolving discipline shaped by the specific shape of each workload. The right strategy depends on whether traffic is dominated by tail latency, throughput, cost, or experimentation needs, and the trade-offs are rarely universal. Netflix's framing is likely intended to share patterns and vocabulary rather than prescribe a single architecture, and it underscores a wider industry trend: as model portfolios grow, the infrastructure that decides where requests go is becoming as important as the models themselves.