Complete Guide to vLLM for LLM Serving

The Complete Guide to vLLM: Fast, Efficient LLM Serving for Production

Running a large language model (LLM) on your own infrastructure sounds simple until you try to serve it to real users. One request at a time works fine. A hundred concurrent users, long prompts, and strict latency targets are a different story. GPUs sit half-idle, memory runs out, and costs climb quickly.

vLLM was built to solve exactly this problem. It is an open-source, high-throughput, memory-efficient engine for LLM inference and serving, and it has become one of the most popular ways to self-host open models such as Llama, Qwen, Mistral, Gemma, and Phi.

This guide covers what vLLM is, why it's fast, how to install it, and how to run it offline and as an API server. We also cover the key configuration options, Docker deployment, and best practices for production. We also look at where this fits for voice and sales products like HireStar.io Voice AI and Revenue Intelligence System, where inference speed shapes how natural every reply feels.

What Is vLLM?

vLLM is an inference and serving engine for LLMs. It originated in research at UC Berkeley and is now maintained by a large open-source community. You give it a model, usually straight from Hugging Face, and it gives you two things:

1
A Python API for fast batch (offline) inference.
2
An OpenAI-compatible HTTP server for online serving, so existing apps built on the OpenAI SDK can point to your own model with minimal code changes.

vLLM handles the serving layer. Your application still owns everything around it: retrieval, prompts, permissions, logging, safety checks, and the user experience.

Why vLLM Is Fast

PagedAttention

When an LLM generates text, it stores intermediate attention data for every token in a structure called the KV cache. Traditional serving systems reserve one large, contiguous block of GPU memory per request, sized for the longest possible output. Most of that memory goes unused, which limits how many requests fit on a GPU.

vLLM introduced PagedAttention, inspired by virtual memory in operating systems. It splits the KV cache into small, fixed-size blocks that are allocated on demand and don't need to be contiguous. The result is far less wasted memory, and more requests running concurrently on the same hardware.

Continuous batching

Older systems use static batching: wait for a batch of requests, process them together, and only start new ones when the whole batch finishes. Short requests end up waiting for long ones.

vLLM uses continuous batching. As soon as one request finishes, a new one takes its place in the batch at the next step. GPUs stay busy, and throughput rises dramatically under real traffic.

Other optimizations

vLLM also includes:

•
Prefix caching: reuses the KV cache for shared prompt prefixes, such as a long system prompt, so repeated work is skipped.
•
Optimized attention kernels such as FlashAttention and FlashInfer.
•
Quantization support, including FP8, INT8, INT4, GPTQ, AWQ, and more, to fit larger models on smaller GPUs.
•
Speculative decoding to speed up generation.
•
Tensor and pipeline parallelism to split large models across multiple GPUs or nodes.
•
Streaming outputs, structured outputs, tool calling, and multi-LoRA serving.

Prerequisites

Before you install, make sure you have:

•
Linux (on Windows, use WSL2 with GPU support).
•
Python 3.10 to 3.13.
•
An NVIDIA GPU with a recent CUDA driver for the simplest setup. AMD ROCm, Intel XPU, Google TPU, AWS Trainium, and other hardware are supported through dedicated builds or plugins.
•
Enough GPU memory for your model. As a rough rule, a model needs about 2 bytes per parameter in FP16/BF16, plus headroom for the KV cache. A 7–8B model typically needs a 24 GB GPU; quantized models need less.
•
A Hugging Face token if you plan to use gated models such as Llama.

Step 1: Install vLLM

The official docs recommend using uv, a fast Python environment manager:

bash

# Create and activate a clean environment
uv venv --python 3.12 --seed
source .venv/bin/activate

# Install vLLM with the right PyTorch build for your CUDA driver
uv pip install vllm --torch-backend=auto

Prefer plain pip? That works too on NVIDIA GPUs:

bash

python -m venv .venv
source .venv/bin/activate
pip install vllm

If you need a gated model, log in to Hugging Face:

bash

huggingface-cli login

Step 2: Offline (Batch) Inference with Python

Offline inference is ideal for processing large datasets: generating summaries, labeling data, running evaluations, and similar jobs.

python

from vllm import LLM, SamplingParams

prompts = [
    "Explain PagedAttention in one sentence.",
    "Write a haiku about GPUs.",
    "What is continuous batching?",
]

sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=200,
)

llm = LLM(model="Qwen/Qwen2.5-1.5B-Instruct")
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(f"Prompt: {output.prompt}")
    print(f"Output: {output.outputs[0].text}\n")

For chat models, use llm.chat() so the model's chat template is applied automatically:

python

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Give me three tips for writing clean code."},
]

outputs = llm.chat(messages, sampling_params)
print(outputs[0].outputs[0].text)

vLLM batches all prompts internally, so passing thousands of prompts at once is much faster than looping one by one.

Step 3: Online Serving with the OpenAI-Compatible API

To serve a model to applications, start the server:

bash

vllm serve Qwen/Qwen2.5-1.5B-Instruct

By default, it listens on http://localhost:8000 and exposes OpenAI-style endpoints, including /v1/chat/completions, /v1/completions, and /v1/models.

Test with curl

bash

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-1.5B-Instruct",
    "messages": [{"role": "user", "content": "Hello! What can you do?"}],
    "max_tokens": 150
  }'

Use the OpenAI Python SDK

python

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-1.5B-Instruct",
    messages=[{"role": "user", "content": "Summarize what vLLM does."}],
    stream=True,
)

for chunk in response:
    print(chunk.choices[0].delta.content or "", end="")

Because the API is OpenAI-compatible, frameworks such as LangChain, LlamaIndex, and most agent libraries can use your vLLM server simply by changing the base URL.

Tip: Compatibility isn't identity. Test chat templates, tool calling, structured output, and streaming behavior with your specific model before swapping out a production endpoint.

Step 4: Key Configuration Options

These are the flags you'll reach for most often with vllm serve:

Flag	What it does
`--host`, `--port`	Address and port for the server (default port 8000).
`--api-key`	Requires clients to send this key, adding basic protection.
`--tensor-parallel-size N`	Splits the model across N GPUs on one machine.
`--pipeline-parallel-size N`	Splits model layers across stages, useful for multi-node setups.
`--gpu-memory-utilization 0.9`	Fraction of GPU memory vLLM may use for weights and KV cache.
`--max-model-len`	Maximum context length. Lower it to save memory if you don't need long contexts.
`--max-num-seqs`	Maximum concurrent sequences per batch.
`--dtype`	Precision, such as auto, bfloat16, or float16.
`--quantization`	Quantization method, e.g. awq, gptq, or fp8.
`--served-model-name`	A friendly model name for clients to use.

Example of serving a larger model across two GPUs:

bash

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --tensor-parallel-size 2 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --api-key your-secret-key \
  --served-model-name llama-8b

Tool calling for AI agents

To let agents call functions through the standard tools parameter, enable automatic tool choice with a parser that matches your model family:

bash

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --enable-auto-tool-choice \
  --tool-call-parser hermes

Check the vLLM docs for the right parser for your model.

Serving LoRA adapters

You can serve several fine-tuned LoRA adapters on top of one base model. This is the pattern behind persona-specific agents such as HireStar.io Voice AI and Revenue Intelligence System, where one base model can power different conversational roles. For example:

bash

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-lora \
  --lora-modules support-bot=./adapters/support sales-bot=./adapters/sales

Clients then pick an adapter by setting "model": "support-bot" in their request.

Step 5: Deploy with Docker

For consistent, reproducible deployments, use the official Docker image:

bash

docker run --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HF_TOKEN=$HF_TOKEN \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen2.5-7B-Instruct

A few notes:

•
Mounting the Hugging Face cache avoids re-downloading models on every restart.
•
--ipc=host (or a larger --shm-size) is needed for PyTorch shared memory, especially with multiple GPUs.
•
Pin a specific image version in production instead of latest.

For Kubernetes, vLLM runs well as a standard deployment behind a service, with GPU node selectors and a readiness probe on the /health endpoint.

Step 6: Monitoring and Performance Tuning

vLLM exposes Prometheus metrics at /metrics, including request throughput, time to first token, generation speed, queue length, and KV cache usage. Connect it to Grafana to watch your service in real time.

When tuning, focus on:

•
Time to first token (TTFT): how fast users see the first word. Critical for chat and voice agents. For HireStar.io Voice AI, TTFT directly shapes how natural a phone conversation feels.
•
Inter-token latency: how smoothly text streams.
•
Throughput: total tokens per second across all users, which drives cost.
•
KV cache utilization: if it's constantly full, requests will queue.

Practical tuning tips:

•
Out of memory? Lower --max-model-len, reduce --max-num-seqs, or use a quantized model.
•
Long, shared system prompts? Make sure prefix caching is on. It's enabled by default in recent versions.
•
Model too big for one GPU? Increase --tensor-parallel-size.
•
Need more throughput? Run multiple replicas behind a load balancer.
•
Benchmark with realistic traffic. vLLM includes benchmarking tools, and your own prompt lengths matter more than generic benchmarks.

Production Best Practices

1
Secure the endpoint. Use --api-key at minimum, and place vLLM behind an API gateway or reverse proxy with authentication, TLS, and rate limiting.
2
Pin versions. vLLM moves fast. Lock the package or Docker image version and test upgrades in staging.
3
Pick the right model size. A well-prompted smaller model often beats a larger one on cost and latency for focused tasks. This is why HireStar.io favors focused, task-tuned models for Voice AI.
4
Evaluate before switching. Measure quality on your real prompts, not just benchmark scores.
5
Plan for scale. Use autoscaling, health checks, and graceful shutdowns so in-flight requests aren't dropped.
6
Log responsibly. Capture metrics and errors, but handle prompt and response data according to your privacy policies.

vLLM vs. Other Options

•
Ollama and llama.cpp are excellent for local development, laptops, and lightweight deployments, but they aren't designed for high-concurrency serving.
•
TensorRT-LLM can deliver top performance on NVIDIA hardware but requires more setup and model-specific optimization.
•
SGLang is another high-performance serving engine with strong support for structured and agentic workloads.
•
Hosted APIs are the fastest way to start, but self-hosting with vLLM gives you control over data, cost at scale, and model choice.

For most teams that want to self-host open models for many concurrent users, vLLM hits a strong balance of performance, flexibility, and ease of use. For latency-sensitive products like HireStar.io Voice AI, these trade-offs matter most.

Final Thoughts

vLLM turns a GPU and an open-source model into a production-ready, OpenAI-compatible API in a few commands. Under the hood, PagedAttention and continuous batching let you serve far more users on the same hardware. On top, features like quantization, multi-GPU support, tool calling, and LoRA serving give you room to grow.

Start small: install it, serve a compact model, and point your existing app at it. Then tune, secure, and scale as your traffic grows. The same approach underpins how HireStar.io thinks about serving LLMs for Voice AI and Revenue Intelligence System.

FAQs – vLLM for Production LLM Serving

•
What is vLLM used for? vLLM is an open-source engine for serving LLMs at high throughput. It offers a Python API for offline batch inference and an OpenAI-compatible HTTP server for online serving.
•
Why is vLLM faster than traditional serving? PagedAttention stores the KV cache in small on-demand blocks, wasting far less GPU memory, and continuous batching keeps the GPU busy by swapping in new requests as others finish.
•
Can I use the OpenAI SDK with vLLM? Yes. Point the SDK's base URL at your vLLM server. Still test chat templates, tool calling, structured output, and streaming with your model before replacing a production endpoint.
•
How much GPU memory do I need? A rough rule is about 2 bytes per parameter in FP16/BF16 plus headroom for the KV cache, so a 7–8B model typically needs a 24 GB GPU. Quantized models need less.
•
How do I serve several fine-tuned models on one GPU setup? Use multi-LoRA serving. Load one base model with --enable-lora and register adapters by name, then clients choose an adapter through the model field.
•
Why does vLLM matter for voice agents? Voice conversations are sensitive to time to first token and concurrency. vLLM's batching and caching help keep both under control, which matters for products like HireStar.io Voice AI.

Planning to self-host LLMs for your product or AI agents? Talk to the HireStar.io team — we'd be happy to help you design a fast, cost-efficient inference stack. Talk to our team