ONLINE robertZurich, CHAS214476Delivery context: TLPT / FINMALanguageENDE
robert

BLOG

Inference platform: a sanitized case study

Published 8 min read

How a GitOps-managed vLLM inference platform actually worked: one OpenAI-compatible router in front of per-GPU engine profiles, hot swaps with readiness gates, streaming passthrough, token-accurate telemetry, and an offline model cache.

This case study describes the vLLM router implementation recorded in the platform's June 2026 Git history. It is not a description of every component running today. The platform is private, so hostnames, network addresses, ports, repository paths and model inventory are omitted. The discussion separates implemented mechanisms from operational trade-offs; vendor documentation explains those mechanisms but is not evidence of my deployment.

The problem the router solved

A small set of GPUs serves a changing fleet of models: release cadence is weekly at times, consumers are command-line agents and a web chat expecting an OpenAI-compatible HTTP surface, and every consumer assumes the endpoint stays valid while the model behind it changes. Running one long-lived engine process per model does not scale on a single workstation's memory; running one process per model request destroys latency. The resolution pattern is a coordinate system most serving stacks already speak, so the platform kept the vLLM OpenAI-compatible server (opens in a new tab) as the engine and put a thin, deterministic orchestrator in front of it.

Requirements, concretely:

  • One OpenAI-compatible endpoint per GPU pool, with model identity resolved by request model field.
  • At most one loaded engine process per GPU at any time; load is the scarce resource.
  • Changes to the model fleet declared in configuration, applied through GitOps, never by interactive shell work on the serving node.
  • Requests, tokens, latency, concurrency, swaps and active state visible as metrics, without breaking the offline posture of the serving pod.

Architecture

        agents, CLI clients, web chat
                     │   OpenAI-compatible HTTP, streaming and non-streaming
                     ▼
   ┌───────────────────────────────────────┐
   │           inference router            │
   │  - model-to-profile resolution        │
   │  - readiness gate and swap control    │
   │  - streaming passthrough              │
   │  - metrics endpoint + JSON access log │
   └───────────────────┬───────────────────┘
                       │   localhost HTTP, one address per pool
          ┌────────────┴────────────┐
          ▼                         ▼
  ┌──────────────────┐      ┌──────────────────┐
  │     GPU pool 1   │      │     GPU pool 2   │
  │  one engine      │      │  one engine      │
  │  process loaded  │      │  process loaded  │
  │  at a time       │      │  at a time       │
  └────────┬─────────┘      └────────┬─────────┘
           │ read-only mount         │ read-only mount
           ▼                         ▼
  ┌────────────────────────────────────────────┐
  │   pre-downloaded model cache, offline      │
   └────────────────────────────────────────────┘

Two layers are separated. The router is one FastAPI application per deployment: it handles the OpenAI-compatible API, holds the profile configuration, owns process lifecycle, and exports metrics. A profile declares a model identifier, served names and aliases, a local model path, context limit, startup timeout, environment and engine arguments. The engine is vLLM started as a subprocess, with its HTTP listener bound to loopback. Resolving models from the local cache avoids normal Hugging Face downloads; it does not by itself restrict arbitrary network access.

Deliberate simplifications: the router is a single process and holds no persistent state, so a crash is recoverable by restart, and its configuration is a single JSON document versioned in Git. No service discovery, no distributed state, nothing to reconcile: the platform is one node's GPU fleet, and the cost of pretending otherwise buys nothing at this scale.

Model readiness and hot swaps

The swap is the load-bearing operation. Its contract, as implemented:

  1. Requests drain before the swap. The pool refuses to start a swap while in-flight requests are open; new requests for the incoming profile wait on the same condition variable. A swap never kills a request that is mid-stream.
  2. The old engine stops with a graceful signal and a bounded wait, 90 seconds, before a hard kill.
  3. The incoming engine starts, then the router polls it on a short fixed interval until it reports ready; a process that dies mid-load is detected through its exit status and reported.
  4. Readiness is a real check, not a port probe. The router polls the engine's models endpoint and considers it ready only when the declared model identifier actually appears in the response. The configured startup window (30 minutes by default) bounds how long a cold load may take.
  5. Exactly one profile is loaded per pool, and a status endpoint reports which one, what is switching, in-flight request count and the last error. The platform's own answer to "which model is actually loaded right now" is one HTTP call, not nvidia-smi archaeology.

Cold-start time equals engine load time: seconds for a small quantized model, minutes for a large full-precision checkpoint. That number is a property of the engine, not the router, and the router's startup window admits it rather than pretending swaps are free.

The router preloads each pool's configured default at startup. Choosing a default that matches the main workload avoids its initial on-demand load. This is not a residency guarantee: a request for another profile can still trigger a swap. Workloads that repeatedly alternate between profiles retain the cold-load cost; preloading alone does not solve that contention.

Streaming, exactly preserved

Streaming passthrough is where most naive proxies break, so it was treated as a first-class contract:

  • The router streams chunks as they arrive; it never buffers an SSE response before forwarding.
  • Hop-by-hop headers are stripped in both directions; everything else is forwarded verbatim.
  • For streaming chat completions, the router ensures the response includes usage. If the client did not ask for it, it rewrites the request to add the standard option, so the token accounting below works on streaming paths too. Non-streaming responses are lightly captured for usage extraction; streaming ones keep a bounded tail of the last kilobytes.

Telemetry: what operation looked like

Metrics are hand-rendered Prometheus text from a stdlib-only module, because the serving image runs offline and should not take a client dependency for exposition format. The set was small on purpose:

  • Counters: requests (by pool, GPU, model, status class, streaming flag), prompt tokens and completion tokens (by pool, model, client class).
  • A request-duration histogram with sane, coarse buckets.
  • A swap counter (per pool, from-profile, to-profile).
  • Gauges: active requests and currently-loaded profile index.

Access logging is one JSON line per request: pool, model, profile, client class, status, streaming flag, token counts, duration in milliseconds. Client attribution resolves a request to a small set of named consumer classes and uses it only as a metrics label; it is bookkeeping, deliberately simple, and explicitly not a security boundary.

Swap counters expose model churn, while duration and concurrency metrics help distinguish cold loads from serving bottlenecks. These signals support diagnosis; they do not by themselves establish why a particular client timed out.

Isolation posture

  • Hugging Face offline settings (HF_HUB_OFFLINE=1 and equivalents) keep supported library calls on the local cache. These settings are not an egress firewall; network access must be controlled independently.
  • Models live on a host cache mounted read-only into the pod; only a separate, explicit download step with write access populates it.
  • Only the router's endpoint is reachable through the ingress; engine processes bind to loopback inside the pod.
  • Both the deployment and the profile configuration are GitOps-declared and reconciled from Git; there was no interactive configuration surface on the node.
  • Network policy allowed ingress to the serving pod only from the ingress controller, the web chat and the metrics scraper, so consumers could not bypass the router.

Implemented here versus generic practice

To keep the claim surface honest:

Implemented, as described above Generic guidance, not a claim
Pool-per-GPU facade, one engine process per pool, profile resolution by name and alias Whether one vLLM process per GPU is the right shape for a different fleet
Drain-before-swap with in-flight counting, bounded SIGTERM wait, model-id readiness polling Which engine versions or scheduling policies one should run
Streaming passthrough with forced usage accounting and hand-rolled metrics text Prometheus monitoring best practices
Offline serving against a read-only model cache Any specific Kubernetes distribution or storage product

Limits of this documentation

  • This is a historical implementation snapshot, not a current inventory or a claim that later deployments retained every mechanism.
  • No throughput figures are published: they depend on engine, hardware, batch shape and model, and none of the measurements in private operational records are stable enough to quote as platform-level facts.
  • Client identities and any network-level detail are intentionally omitted; the article deliberately cannot be used to reach the platform.

Evidence