An Open Field Guide to Inference Engineering · Modelplane Docs
A field guide to inference concepts, production operations, tooling, and career development for platform engineers.
A field guide to inference concepts, production operations, tooling, and career development for platform engineers.
Note
Numbers, prices and tool versions are as of October 2026.
This guide is for platform engineers new to inference. If you need to understand how models and GPUs work together and how to run inference in production, this guide will help. You’ll also learn about the tools to use and how to grow into an inference engineering role.
This guide builds on the work of others in the inference community. If you’re interested in inference engineering, continue your learning with these:
In an AI-forward organization, platform teams become inference teams. Coding assistants can’t send source code to third parties, and frontier model bills increase month to month. Support tools also need internal data that a hosted model can’t reach.
Organizations want to run these models themselves, and open models now make that practical.
Hugging Face hosts more than two million open models, and the best of them compete with closed models. DeepSeek released R1 in January 2025, and it matched OpenAI’s o1 on math and coding benchmarks about a month after o1 came out . When you run your own model, you control latency, uptime, and where your data goes. At high volume, you can also cut costs by about 80% .
The work suits platform engineers. The price of a fixed level of AI capability falls roughly 10x a year, and most of that comes from how models are served, not from better GPUs . Upbound, the company behind Crossplane and Modelplane, saw this firsthand. In its Project Champagne blog series, the team ran its own models on Modelplane and found that vLLM and scheduling worked. The time went to TLS, identity, timeouts, storage, IAM and observability.
An inference engineer picks a model, gets it onto the right GPU, keeps it serving, makes it faster and cheaper, and gets paged when it breaks. That’s SRE for a new workload, and most of your skills carry over:
| What you already know | What it becomes in inference |
|---|---|
| Scheduling and node pools | GPU placement with device selectors, taints and exclusive devices |
| Capacity planning | GPU memory math: weights, KV cache and runtime overhead, checked at runtime |
| Load balancing | Cache-aware routing and long-lived streaming connections |
| Autoscaling on CPU | Scaling on queue depth and KV cache use, with cold starts measured in minutes |
| Rollouts | Rollouts of exclusive GPUs, and a version matrix of model, hardware and engine build |
| SLOs and latency percentiles | Time to first token, time per output token and goodput |
| Observability | Engine metrics, real GPU activity and speculative decoding acceptance |
| FinOps | Cost per million tokens and cost per user session |
| Identity and access | Credentials that last as long as an agent session |
A hosted API is still the right call early in a product’s life . Self-hosting pays off when API spend becomes a real line item, when latency or reliability needs outgrow vendors, or when data can’t leave your network. In August 2026, Chris Zeoli counted about 58 open inference roles on public careers boards, with no settled job title.
This section covers the concepts in roughly the order you’ll meet them. Most are about what the engine keeps in GPU memory or what limits its speed.
Viktor Farcic suggests asking when each concept starts to matter. Some matter as soon as you deploy, others only once many people use the model, and some you may never need. The table near the end of this section gives the signal for each.
An inference engine, like vLLM or SGLang , is the server that turns model weights into an API. For a platform engineer, the closest mental model is a stateful database server with a GPU attached. It loads a large dataset into memory at startup and keeps per-session state. It also schedules work internally and answers over HTTP. Ollama on a laptop is the same idea at small scale, and every hosted model API runs one behind the scenes.
| You know this from databases and services | The engine’s counterpart |
|---|---|
| Loading a dataset into the buffer pool at startup | Loading model weights into GPU memory, which can take many minutes |
| Per-session state in a session store | The KV cache, one for each conversation in flight |
| A query planner and connection pool | The scheduler, which batches requests into each GPU pass |
| A wire protocol clients already speak | An OpenAI- or Anthropic-compatible HTTP API |
| Warming caches after a restart | Cold starts: pulling a large image and reloading the weights |
Unlike a database, the engine keeps its working set in GPU memory, not RAM. GPU memory is small and expensive, so most tuning is about deciding how much of it each request gets.
The engine is one layer of four. At the bottom, kernels do the arithmetic. You rarely choose them directly. Above them sits the engine. Above that, a control plane deploys and manages engines, because sooner or later you run more than one. On top, a gateway routes traffic across models and replicas.
Inference stack: four layers and the clients they serve
Requests pass through the gateway to an engine, which runs kernels on the GPU. The control plane deploys and manages the gateway and engines, but no request passes through it.
A token is the unit a language model works in, roughly a word or word fragment. Context windows, prices and speeds are all counted in tokens. When someone quotes tokens per second, they mean this unit.
The weights are fixed, and the KV cache grows with load. Size for both, as you would a database’s footprint and working set.
Weights are the fixed footprint. Most models are released at 2 bytes per parameter (BF16 or FP16), so an 8B model needs about 16 GB, and it has to live in GPU memory, not system RAM. On a 48 GB L40S, that leaves about two-thirds of the card for everything else. That number is the same for one user or a hundred.
The KV cache is the working set. Each new token is computed against every token before it. Rather than recompute that history, the engine keeps the attention keys and values for every token in flight. Its size scales with context length multiplied by the number of concurrent conversations, much like a session store that grows with both session size and user count.
You can estimate the KV cache per token from a model’s config:
KV bytes per token = 2 × layers × KV heads × head dimension × bytes per valueFor Llama 3 70B at FP16 (80 layers, 8 KV heads, head dimension 128), that is about 320 KB per token. One 128K-token conversation needs about 40 GB of cache. At that size, the KV cache usually limits how many users a GPU can serve.
Watch for these sizing mistakes:
Each request runs in two phases on the same GPU, and they are limited by different things .
| Phase | What it does | Limited by | Metric it sets |
|---|---|---|---|
| Prefill | Reads the whole input in parallel, including the system prompt and tool definitions as well as the conversation history. Produces the first token and the KV cache. | Compute | Time to first token (TTFT) |
| Decode | Writes the answer one token at a time. Each token needs a pass that reads all active weights, plus the KV cache, from memory. At long context, reading the cache can cost more than reading the weights. | Memory bandwidth | Time per output token and tokens per second |
Generating one token does one or two floating-point operations per byte read from memory, depending on weight precision. An H100 needs about 295 operations per byte to stay busy. That means one request uses a small fraction of the chip while the compute units wait on memory (Data Gravity ).
Peak compute has grown about 3x every two years, against about 1.6x for memory bandwidth. Nearly every optimization either moves fewer bytes per token or spreads each byte across more tokens.
Because the phases have opposite bottlenecks, a technique that speeds one often does little for the other. That is why benchmarks report TTFT and tokens per second separately.
| Metric | What it measures | Watch out for |
|---|---|---|
| Time to first token (TTFT) | Queueing plus prefill: how long the user waits before anything appears | With reasoning models, the engine’s first token may be hidden thinking. Measure the first token the user sees. |
| Time per output token, or inter-token latency | Decode speed per stream | Speculative decoding and memory bandwidth show up here |
| Tokens per second per user | Interactivity, or the model’s typing speed | Falls as batch size rises |
| Throughput | Total tokens per second across all users | Rises as batch size rises |
| Goodput | Throughput that still meets your latency targets | The number to plan capacity with |
| End-to-end latency | Total request time | Depends on how much the model writes. A healthy 190-second answer is normal for long generations. |
| Percentiles | p50, p90, p99 of each latency | Users feel the tail, not the average |
| Errors and refusals | Failed or rejected requests | Count refusals as errors, or a context cap can look like a speedup |
| Cost per million tokens | GPU cost divided by tokens served | Compare at the load you actually run |
Cost per million tokens converts a GPU price and a throughput into one comparable number:
Cost per million tokens = GPU cost per hour / (tokens per second × 3600) × 1,000,000Spheron’s worked example: a GPU at $2.90 an hour sustaining 1,200 tokens per second costs about $0.67 per million tokens.
Batching more requests together raises total throughput and slows each individual user. Interactive chat and coding favor latency. Overnight batch jobs favor throughput.
It’s the same trade-off as tuning a database for transactions or for analytics. Write down the workload’s constraints before you tune anything.
Each row gives the symptom that tells you it’s time to act, in roughly the order teams need them.
| Concept | What it does | The signal that you need it |
|---|---|---|
| Fitting the weights | Choose a model, precision and GPU so the weights fit with room to spare | The engine fails to start, or starts with almost no KV cache left |
| Continuous batching and paged KV cache | Share each GPU pass across requests, and hand out cache in small blocks | Already on by default in vLLM and SGLang. Check it, don’t build it. |
| Quantization | Store weights in fewer bits, and give the freed memory to the KV cache | Requests queue while the GPU still has compute to spare |
| Prefix caching | Reuse the work already done on repeated prompt openings | Already on by default in vLLM and SGLang. Watch the hit rate once your clients are agents or chat apps that resend long histories |
| Speculative decoding | Draft several tokens and verify them in one pass | Individual responses stream too slowly, even at low load |
| Gateway and control plane | Route between replicas, and manage engines declaratively | You add a second model, a second replica or a second cluster |
| Sharding across GPUs | Split one model with tensor, pipeline or expert parallelism | The model will not fit on one GPU. Plan this before you deploy. |
| Autoscaling | Add and remove replicas on queue depth and KV cache use | Idle GPUs show up on the bill, or queues build at peak hours |
| Disaggregation | Run prefill and decode on separate GPUs | Long prompts stall streaming at scale, and you have RDMA-class networking |
Most teams need only the first few rows on day one. Add the rest when their signals show up.
Coding agents and other agentic workloads stress every part of this picture. Upbound measured this in Part 5 of its Project Champagne series (introduced in Section 1), where about 10 of its engineers ran Claude Code and other coding agents against a self-hosted model. Their traffic looked like this:
In production, every choice trades latency, throughput, cost and quality against each other, usually with too few GPUs. Where a practice comes from Project Champagne, Section 5 has the details.
Each use case sets different requirements. In Project Champagne:
Separate interactive workloads from batch workloads. They need different targets, and often different deployments. Write the pass and fail bar down before you test anything.
A model that fits at startup can still run out of memory under real traffic when fragmentation pushes past the profiled peak. That can happen days later, in the middle of a request.
Most engines turn on continuous batching and paged KV cache by default. The rest are choices you make per model and workload.
| Technique | What it improves | Cost or risk | Use it when |
|---|---|---|---|
| Continuous batching | Throughput. Orca reported 36.9x over a static baseline at the same latency. | Per-user speed drops as batches grow | Always. It is the default in modern engines. |
| Paged KV cache (PagedAttention ) | Concurrency. vLLM cut KV cache waste from 60 to 80% to under 4%. | Little in practice | Always. It is the default in vLLM. |
| Prefix caching | TTFT and prefill cost on repeated openings | Only helps up to the first token that differs | System prompts, RAG, multi-turn chat and agents |
| Weight quantization | Decode always, and prefill only when activations are quantized too, as with FP8. Weight-only formats such as AWQ and GPTQ speed up decode alone. One step down in precision is typically 30 to 50% faster (Inference Engineering). | Quality loss, which you must measure | Use the lowest precision your hardware runs natively and your evals accept. FP8 runs natively on Ada, Hopper and Blackwell GPUs. |
| KV cache quantization | More cache fits, and it moves faster | Errors compound from token to token | When KV cache is the limit |
| Speculative decoding , multi-token prediction | Tokens per second at low batch sizes | Wasted compute when guesses are rejected. Its value shrinks as load rises and spare compute runs out. | Latency matters and output is predictable |
| Tensor parallelism | Fits large models, faster per user | Needs fast GPU-to-GPU links such as NVLink | Large models within one node |
| Expert parallelism | Throughput for MoE models | Routing between experts | MoE models, including across nodes |
| Pipeline parallelism | Fits a model across nodes on ordinary networking | Adds latency to each pass | The model does not fit in one node |
| Prefill and decode disaggregation | Removes interference between phases. DistServe reported 7.4x more requests within latency limits. | Needs RDMA-class networking to move the KV cache | Large scale, long prompts, fast interconnect |
A few rules of thumb:
Set SLOs on what users feel, measured per request at the front door:
Do not set an SLO on total request time, because it depends on how much the model writes. Low-traffic models need a fleet-wide SLO, longer windows or synthetic traffic. Missing data reads as 100% good, so keep a separate alert for metrics that stop arriving. Open source tools such as Sloth and Pyrra turn SLO definitions into burn-rate alerts. For background, see Observability Engineering .
Promote every serving change through tiers. Each tier costs more than the one before and catches a different kind of failure, so most problems show up before users see them:
When an open model scores low, check three places before you blame the model:
Also pin the data the right answer depends on and give every model the same context. Keep test markers where the model cannot read them. Run each test more than once, and prefer deterministic checks over using a model as the judge. A batching server can return different tokens for the same request even at temperature 0, so expect some variance unless the engine runs in a slower batch-invariant mode.
Go deep on the engine. You already know Kubernetes, gateways and observability, so learn what changes for inference. For kernels, reading a profile is enough. Most inference engineering happens higher up the stack.
| Layer | Tools to know | Master first |
|---|---|---|
| GPUs on Kubernetes | NVIDIA GPU Operator , Dynamic Resource Allocation and its NVIDIA driver , DCGM exporter | How GPUs are advertised, requested and placed |
| Kernels and libraries | CUDA , PyTorch , Triton , FlashAttention , torch.compile | Kernel basics, kernel fusion, and how to read a profile |
| Inference engines | vLLM , SGLang , TensorRT-LLM ; Ollama and llama.cpp for local work | vLLM’s core flags: --gpu-memory-utilization, --max-num-seqs, --max-model-len, tensor parallel size, quantization, prefix caching, chunked prefill |
| Distributed serving | NVIDIA Dynamo , llm-d , KServe , LeaderWorkerSet | Multi-GPU and multi-node layouts, and prefill and decode split |
| Gateway and routing | Envoy Gateway , Agent Router (formerly Envoy AI Gateway), Gateway API Inference Extension | Streaming, timeouts, auth policies, model-name and cache-aware routing |
| Control plane | Modelplane , built on Crossplane | Declaring clusters, hardware classes, deployments and caches across environments |
| Observability | Prometheus , Grafana , OpenTelemetry , Loki , Sloth or Pyrra | Engine metrics, real GPU activity, and SLOs on user-facing latency |
| Benchmarking | vLLM’s benchmark tools , NVIDIA AIPerf , GuideLLM , SemiAnalysis AgentX traces | Concurrency sweeps and latency percentiles under realistic prompts |
| Evals | Inspect from the UK AI Security Institute; Hugging Face upskill for testing skills across models | Deterministic graders on the end state of a task |
| Models | Hugging Face Hub , safetensors checkpoints | Reading a model config for layers, heads and context, and checking real file size |
The engine controls batching, memory management, kernel selection and API compatibility. It often matters more to cost and throughput than a hardware upgrade.
| Engine | Why teams pick it | What it costs to operate |
|---|---|---|
| vLLM | The broadest model coverage, plus OpenAI and Anthropic APIs out of the box, so it is the usual first deployment. Hosted by the PyTorch Foundation. | Frequent releases with new flags. Pin versions and re-benchmark on every upgrade. |
| SGLang | A radix-tree prefix cache and a scheduler that favors requests sharing a cached prefix, which suits agents and long multi-turn sessions. Upbound measured its biggest single gain here (Project Champagne, Part 5, 2026). | A smaller community, and new models may need nightly builds before a tagged release works. |
| TensorRT-LLM | Kernels tuned for NVIDIA hardware, including FP4 on Blackwell | NVIDIA hardware only, which ties your serving stack to one vendor. |
| NVIDIA Dynamo | A layer above engines that coordinates many nodes, including split prefill and decode | Another distributed system to run, so use it only at multi-node scale. |
| Ollama , llama.cpp | Local serving for learning and prototyping | Not designed for many concurrent users |
Hugging Face’s TGI was a common choice and is now in maintenance mode, so check release cadence, maintainers and foundation backing before you commit to one. Engines add new flags every few weeks. Re-run your benchmarks when you upgrade, rather than trusting last quarter’s numbers.
Upbound’s own comparison in Project Champagne, Part 5 shows this. The team was serving a coding model to about 10 engineers on two 8-GPU nodes and tested how to grow to a team of 15. On the same hardware and the same agent traffic, switching from vLLM to SGLang was the largest single gain it measured. Adding SGLang’s host-memory cache on top didn’t add a measurable gain, because the KV cache never filled.
Modelplane is an open source project from Upbound, built on Crossplane and released under the Apache 2.0 license. It orchestrates models, inference engines and AI infrastructure across clouds, neoclouds and your own GPUs, with Kubernetes-style declarative APIs. It provisions inference clusters or uses ones you bring. Documentation is at docs.modelplane.ai .
Modelplane splits the work into three layers of ownership. You can use the same split as a mental model even with different tools.
| Who | Owns | Modelplane resources |
|---|---|---|
| Platform team | The fleet: clusters, hardware classes and the gateway | InferenceCluster, InferenceClass, InferenceGateway |
| Model operator, or inference engineer | The models: deployments, services and weight caches | ModelDeployment, ModelService, ModelCache |
| Modelplane | Everything in between, composed from the two above | ModelEndpoint, ModelReplica |
The hierarchy mirrors core Kubernetes one scope up. ModelDeployment → ModelReplica → ModelService → ModelEndpoint parallels Deployment → Pod → Service → Endpoint, across a fleet instead of within one cluster. You bring the engine container and its flags, and Modelplane shapes the topology around them. People and agents never create any of these. They call the InferenceGateway by model name.
Modelplane maps vLLM and SGLang metrics onto common names out of the box, and a MetricMapping adds any other engine, so different engines are judged on the same signals.
For a platform engineer, the second layer is where the new learning is. Upbound found the same thing when it started self-hosting in Project Champagne, Part 1 : the cluster and gateway work was familiar, and the model operator layer held the unknowns.
Much of what you need to know about running inference comes from teams that wrote up what broke. Learn to read those write-ups critically, and write your own (Section 6).
Upbound, the company behind Crossplane and Modelplane, self-hosted chat, coding agents and a CVE-fixing build agent on open models with Modelplane, across AWS and Nebius GPUs. The team published what broke in a five-part series. Highlights, with numbers as of 2026:
GPU-Util with tensor cores busy about 10% of the time. (Part 4)Each post on the Upbound blog tells these stories in detail, with the configs, queries and numbers behind them:
You don’t need to start over. Add inference concepts to the platform skills you have, and show the work in public. The field is young, so newcomers can catch up fast.
Pick the direction closest to the work you already do. Most platform engineers start in the first two and grow toward the third.
| Direction | Focus | Good fit if you |
|---|---|---|
| Inference platform | Clusters, GPU scheduling, gateways, autoscaling, multi-cloud capacity | Enjoy Kubernetes and distributed systems |
| Inference reliability | SLOs, observability, capacity planning, rollouts, incident response | Have carried a pager and run SRE practices |
| Runtime performance | Engine tuning, quantization, speculative decoding, parallelism, kernels | Like profiling and working close to the hardware |
Vishakha Sadhwani’s AI Inference Engineering Roadmap 2026 is a thorough 13-stage map of the field. Coming from platform engineering, you can move through it faster, because you already own much of the infrastructure half. A shorter path:
Each project below fits on a rented GPU or two, and each produces a repo with real numbers. Shut instances down between runs, and use spot capacity for batch experiments.
| Project | What it proves | What to publish |
|---|---|---|
| Serve an 8B model with vLLM on one GPU. Sweep 1, 10 and 50 concurrent users. | You can measure TTFT, time per output token and throughput accurately | Configs, the load script, charts and cost per million tokens |
| Predict how many 32K-token sessions fit on your GPU, then test it | You understand weights, KV cache and runtime overhead | Your prediction, the result and why they differ |
| Compare FP16 and FP8, and a 4-bit format if you can | You can trade quality against speed with evidence | Eval scores next to latency and throughput |
| Run the same prompts with shared content first, then last | You understand prefix caching and prompt structure | Cache hit rate and TTFT for each |
| Deploy on Kubernetes with the GPU Operator, a gateway with TLS and auth, autoscaling on queue depth, and dashboards | You can run inference as a platform | Manifests or compositions, and dashboard definitions |
| Compare vLLM and SGLang on agent-like traffic, one change at a time | You make engine choices with data | The bar, the method and results with intervals |
| Build an eval harness for one agent task with deterministic graders | You can tell a model failure from a harness failure | Tasks, graders and negative controls |
| Reproduce a failure: a timeout, an out-of-memory crash or a stuck rollout | You can operate it, not just deploy it | A short postmortem and runbook |
Many inference roles fill through networks before they reach a careers board. In August 2026, Zeoli found one serving company valued at $17.5B that didn’t list any dedicated inference performance openings. Public work is how people outside those networks get found.
Much of what inference projects still need is platform work. Upbound’s own upstream list from Project Champagne reads like a platform backlog: configurable load balancers, certificate issuance, Recreate rollouts for exclusive devices, an honest Ready signal, engine metrics scraped out of the box, and auth that suits long-running agents.
Projects to follow include vLLM , SGLang , llm-d , KServe , NVIDIA Dynamo , Agent Router (formerly Envoy AI Gateway), the Gateway API Inference Extension , Modelplane and Crossplane . Good first contributions:
Read each project’s contributing guide, start with a small change, and comment on an issue before taking on a large one.
Engines add flags every few weeks, and research reaches production within months.
Don’t expect a consistent job title yet. The same work shows up as inference engineer, performance engineer, model serving engineer or ML infrastructure engineer, and sometimes as AI platform engineer, so search for all of them. Pay reflects how scarce the skills are: in August 2026, Zeoli found an Anthropic inference performance role posted at $350,000 to $850,000, and an OpenAI equivalent at $295,000 to $555,000.
You may not need to change jobs to get started, though. If your company is planning to self-host its first model, put your hand up. Your team will probably end up responsible for that project anyway, and running it is the most direct experience you can get.
Practice answering these. Each one points to the section of this guide that covers it. Several are adapted from Lamhot Siagian’s interview pocket notes on efficient inference (Section 7).
Hugging Face, upskill .
Texto de docs.modelplane.ai tal como lo publicó, leído de su página y mostrado aquí con el formato de El Autómata. El original, en docs.modelplane.ai · Volver a la noticia de El Autómata