vLLM Serving Guide for High-Throughput LLM Inference

Master vLLM production serving with PagedAttention, continuous batching, multi-GPU tensor parallelism, AWQ quantization, and OpenAI API drop-in compatibility.

Mahmudul Haque Qudrati — CEO & ML Engineer at Pristren

Mahmudul Haque Qudrati

CEO & ML Engineer

May 17, 2026
10 min read
vLLM Serving Guide for High-Throughput LLM Inference Cover Graphic

This vLLM serving guide covers an open-source inference engine engineered for high-throughput, low-latency serving of large language models, delivering up to 24x greater throughput than naive Hugging Face Transformers implementations. Powered by PagedAttention, vLLM dynamically partitions the Key-Value (KV) cache into non-contiguous memory blocks, eliminating memory fragmentation and enabling continuous batching. The guide walks through deploying vLLM with Docker, configuring OpenAI-compatible endpoints, optimizing multi-GPU tensor parallelism, and benchmarking quantization formats.

vLLM Serving Guide: The Memory Bottleneck in Naive LLM Inference

Understanding why standard transformer inference fails under concurrent production loads requires inspecting how attention layers allocate memory. During autoregressive decoding, each generated token computes attention weights against all previous tokens. To avoid recomputing hidden states at every step, transformers cache Key and Value tensors in GPU High Bandwidth Memory (HBM).

In naive inference servers, the KV cache for a request is allocated as a static, contiguous memory block sized for the model's maximum context window (such as 4,096 or 8,192 tokens). This design creates three severe memory inefficiencies:

  1. Internal Fragmentation - When a request only generates 150 tokens out of a 4,096 reservation, over 96 percent of the reserved KV cache memory sits completely unused.
  2. External Fragmentation - Dynamic request lengths fragment available VRAM into isolated memory pockets, preventing new concurrent requests from finding contiguous space even when total free memory is abundant.
  3. Over-Allocation Bottlenecks - Because memory is reserved upfront before actual token generation lengths are known, inference engines artificially restrict concurrent batch sizes to avoid Out-Of-Memory (OOM) crashes.

As a consequence, naive inference servers routinely leave 60 to 80 percent of GPU memory wasted, throttling throughput to single-digit requests per second.

How PagedAttention and Continuous Batching Work

vLLM resolves KV cache memory waste through PagedAttention, an algorithm introduced by UC Berkeley researchers in the original PagedAttention research paper on arXiv. Drawing inspiration from virtual memory paging in operating systems, PagedAttention breaks down the KV cache into fixed-size physical blocks called pages.

Virtual Memory Abstraction for KV Cache

Instead of allocating continuous physical memory, PagedAttention maintains a block table that maps logical token positions to arbitrary physical memory pages across GPU VRAM:

  • Dynamic Page Allocation - Physical pages (typically holding 16 tokens each) are allocated strictly on demand as new tokens are generated.
  • Near-Zero Memory Waste - Waste is confined strictly to the final page of a sequence, reducing overall KV cache fragmentation to below 4 percent.
  • Copy-on-Write Memory Sharing - Multiple decoding paths in parallel sampling or beam search share physical KV pages until generation paths diverge, dramatically reducing memory overhead for complex prompting techniques.

Continuous Batching Architecture

Traditional batching operates in lockstep: a batch of requests executes together until every single request finishes generating. If one request requests 1,000 tokens while others complete in 50 tokens, the entire GPU remains tied up waiting for the slowest request.

vLLM implements continuous batching (also called iteration-level scheduling). As soon as an individual sequence emits an end-of-sequence token, its pages are immediately reclaimed, and a new waiting request from the queue slots into the very next forward pass without pipeline stalls.

Containerized Deployment with Docker and GPU Acceleration

Deploying vLLM via containers ensures standardized CUDA runtimes and reproducible dependency trees. Ensure the host machine has the NVIDIA container toolkit installed via the official Docker Engine installation guide.

Launch an official vLLM container running an OpenAI-compatible API server:

bash
docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai:latest \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --gpu-memory-utilization 0.92 \
  --max-model-len 8192 \
  --tensor-parallel-size 1

The --ipc=host flag grants the container shared memory access required for high-speed inter-process communication, while --gpu-memory-utilization 0.92 allocates 92 percent of available VRAM to model weights and dynamic KV pages.

Consuming the OpenAI-Compatible API

vLLM provides built-in drop-in compatibility with the OpenAI SDK, as documented in the official vLLM OpenAI-compatible server guide. You can integrate self-hosted models into existing Python applications without refactoring application logic:

python
import os
from openai import OpenAI

# Connect to local or remote vLLM endpoint
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-required-for-local-vllm"
)

# Execute streaming chat completion
stream = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are an expert AI systems performance architect."},
        {"role": "user", "content": "Explain how PagedAttention eliminates internal memory fragmentation."}
    ],
    temperature=0.2,
    max_tokens=1024,
    stream=True
)

for chunk in stream:
    content = chunk.choices[0].delta.content or ""
    print(content, end="", flush=True)
print()

Because vLLM reproduces OpenAI streaming chunk payloads, client-side libraries such as Vercel AI SDK and LangChain connect directly by merely overriding the baseURL parameter.

Offline Batch Inference with the Native Python API

For non-streaming bulk tasks such as batch classification, synthetic data generation, or dataset enrichment, vLLM provides a high-throughput native Python engine:

python
from vllm import LLM, SamplingParams

prompts = [
    "Write a production-ready Bash deployment script for Docker.",
    "Explain the difference between sparse and dense vector search.",
    "Summarize the key advantages of multi-worker test sharding.",
    "Provide a minimal Python example of Redis rate limiting."
]

# Configure generation constraints
sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.95,
    max_tokens=512,
    stop=["<|eot_id|>"]
)

# Instantiate engine and load model weights
llm = LLM(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    tensor_parallel_size=1,
    gpu_memory_utilization=0.90
)

# Execute parallel batch generation
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt}")
    print(f"Generated ({len(generated_text.split())} words): {generated_text.strip()}\n---")

The native engine executes parallel token generation across all input prompts simultaneously, saturating tensor cores without the network overhead of HTTP serializers.

Scaling Across Multi-GPU Nodes with Tensor Parallelism

When serving massive model architectures such as 70B parameter models, weights alone exceed the 24GB or 80GB VRAM limits of a single GPU. vLLM implements Megatron-LM tensor parallelism to split attention and MLP weight matrices across multiple devices.

Verify your model architecture against the vLLM supported models directory, then launch the server with distributed tensor slicing:

bash
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 256 \
  --max-num-batched-tokens 16384

Under 4-way tensor parallelism:

  • Self-attention projection matrices (Q, K, V) are sliced column-wise across the 4 GPUs.
  • Attention output projections (O) are sliced row-wise.
  • An all-reduce communication step over NVLink synchronizes intermediate activation vectors between forward passes.

For optimal efficiency, always ensure GPUs are connected via high-bandwidth NVLink interconnects rather than standard PCIe slots to minimize all-reduce latency overhead.

Quantization Strategies Across AWQ, GPTQ, and FP8

Quantization compresses 16-bit floating-point weights into 4-bit or 8-bit precision representations, shrinking VRAM requirements by 50 to 75 percent with minimal degradation in perplexity.

Detailed specifications for modern quantization standards are available in the Hugging Face quantization documentation.

Quantization FormatWeight BitwidthActivation BitwidthTypical VRAM SavingsRecommended HardwareIdeal Workload
FP16 / BF1616-bit16-bitBaseline (0%)Any modern NVIDIA GPUHighest accuracy, research baseline
FP8 (e4m3 / e5m2)8-bit8-bit~50% savingsAda Lovelace, Hopper, BlackwellHigh throughput with near-zero quality loss
AWQ (AutoAWQ)4-bit16-bit~70% savingsAmpere (RTX 3090, A100) or newerProduction high-concurrency 4-bit serving
GPTQ4-bit16-bit~70% savingsAmpere, Turing, PascalEdge deployments with calibrated datasets

To deploy an AWQ-quantized model in vLLM:

bash
python -m vllm.entrypoints.openai.api_server \
  --model casperhansen/llama-3.1-70b-instruct-awq \
  --quantization awq \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.92

Quantizing Llama-3.1-70B with AWQ compresses VRAM footprint from 140GB down to approximately 40GB, allowing a 70-billion parameter model to run comfortably on two 24GB GPUs (such as RTX 4090 or A10G) instead of requiring four A100 accelerators.

Production Monitoring and Prometheus Metrics

Operating self-hosted inference servers in mission-critical environments demands real-time observability. vLLM exposes native Prometheus metrics at the /metrics HTTP route, enabling automated alerting and Grafana dashboard visualization.

Key metrics to monitor in production:

  • vllm:num_requests_running - The count of active requests currently executing on the GPU. Saturated values indicate that traffic is queuing.
  • vllm:num_requests_waiting - Number of pending requests queued in memory. Spikes signal the need to scale up replicas.
  • vllm:gpu_cache_usage_factor - Fraction of allocated GPU KV cache blocks currently occupied. Values consistently above 0.85 indicate impending preemption or queue delays.
  • vllm:time_to_first_token_seconds - Histogram measuring prompt prefill latency (TTFT), directly reflecting perceived user responsiveness.
  • vllm:time_per_output_token_seconds - Inter-token latency during autoregressive token generation.

Configure Prometheus scraping by pointing your Prometheus configuration to port 8000:

yaml
scrape_configs:
  - job_name: "vllm-inference"
    static_configs:
      - targets: ["vllm-service:8000"]

High Availability and Auto-Scaling with Ray Serve

When serving unpredictable consumer traffic, a single instance creates an operational single point of failure. Deploying vLLM on top of Ray Serve enables dynamic replica scaling across multiple GPU worker nodes.

python
import ray
from ray import serve
from vllm.engine.async_llm_engine import AsyncLLMEngine
from vllm.engine.arg_utils import AsyncEngineArgs

@serve.deployment(
    num_replicas=2,
    ray_actor_options={"num_gpus": 1}
)
class VLLMDeployment:
    def __init__(self):
        engine_args = AsyncEngineArgs(
            model="meta-llama/Meta-Llama-3.1-8B-Instruct",
            max_model_len=4096,
            gpu_memory_utilization=0.90
        )
        self.engine = AsyncLLMEngine.from_engine_args(engine_args)

    async def __call__(self, request):
        json_data = await request.json()
        prompt = json_data.get("prompt", "")
        # Forward request to asynchronous execution engine
        results_generator = self.engine.generate(prompt, sampling_params, request_id=request.id)
        final_output = None
        async for request_output in results_generator:
            final_output = request_output
        return {"response": final_output.outputs[0].text}

deployment = VLLMDeployment.bind()

Ray Serve automatically load-balances inbound API requests across worker replicas, handling node failover and horizontal auto-scaling without dropping connections.

Hardware Capacity and Cost Analysis

Selecting appropriate hardware configurations directly governs infrastructure costs and token generation velocity. Here is how common cloud and bare-metal environments compare for production workloads:

Cloud Infrastructure Tiers

  1. Single AWS g5.2xlarge (NVIDIA A10G 24GB)
    • Cost Profile: $1.21/hour on-demand ($870/month) or ~$0.40/hour on spot instances.
    • Capability: Runs Mistral 7B or Llama 8B in FP16 with 2,000+ cached tokens.
    • Throughput: Up to 1,100 tokens per second across batch size 32.
  2. Dual GPU Bare Metal (2x NVIDIA RTX 4090 24GB)
    • Cost Profile: ~$220 to $280/month on dedicated hosting providers.
    • Capability: Runs Llama 70B (AWQ) or dual isolated 8B production models with NVLink or PCIe gen 4.
    • Throughput: Highest token-per-dollar ratio for continuous enterprise API workloads.
  3. Multi-Node Cluster (8x NVIDIA A100 80GB SXM4)
    • Cost Profile: ~$24 to $32/hour on enterprise cloud providers.
    • Capability: High-concurrency unquantized FP16 70B models serving thousands of concurrent users with 32k context windows.

Production Tuning and Optimization Checklist

To extract maximum tokens per second and eliminate pipeline bottlenecks, apply these production guidelines:

  1. Tune Maximum Batched Tokens - Set --max-num-batched-tokens between 16,384 and 32,768 to ensure forward passes fully saturate tensor core compute pipelines.
  2. Prevent OOM with Memory Ceilings - Never set --gpu-memory-utilization above 0.96. Leaving 4 to 8 percent headroom accommodates CUDA context buffers and unexpected prompt bursts.
  3. Pre-Tokenize High-Volume Prompts - When executing large batch operations, tokenize strings before passing them to the execution engine to avoid CPU worker serialization delays.
  4. Enable Prefix Caching - Use --enable-prefix-caching when serving applications with static system prompts or few-shot examples. vLLM reuses cached KV states for identical prompt prefixes across distinct user requests, reducing TTFT (Time-To-First-Token) by up to 85 percent.
  5. Inspect Open Source Development - Track ongoing architectural optimizations and release tags directly on the vLLM open source GitHub repository.

Continue expanding your understanding of open-source artificial intelligence systems with our technical playbooks:


Accelerate Your AI Infrastructure with Pristren

Deploying self-hosted language models at enterprise scale requires rigorous GPU resource management, high-performance API design, and production-ready CI/CD workflows. Pristren partners with organizations to architect, optimize, and deploy robust machine learning systems that slash inference latency and operational expenses. Explore how our engineering team can master your AI infrastructure and bulletproof your release velocity. Schedule an engineering consultation or discover our software engineering services.

Frequently Asked Questions

Why is vLLM faster than standard Hugging Face Transformers for inference?

vLLM utilizes PagedAttention to eliminate KV cache fragmentation by allocating memory dynamically in fixed-size pages. Combined with continuous batching that slots incoming requests into active forward passes without waiting for entire batches to finish, vLLM achieves up to 24x higher throughput.

How does continuous batching differ from traditional static batching?

Static batching forces all requests in a batch to execute together until the longest sequence completes, leaving GPU cores idle as shorter requests finish early. Continuous batching operates at the iteration level, immediately evicting completed sequences and inserting new waiting requests into the next forward pass.

Can vLLM serve models across multiple GPUs simultaneously?

Yes. vLLM natively supports Megatron-LM tensor parallelism using the --tensor-parallel-size flag. This splits model layers across available GPUs to serve massive models such as 70-billion parameter architectures that exceed the memory capacity of a single GPU.

How does AWQ quantization reduce GPU memory requirements in vLLM?

Activation-aware Weight Quantization (AWQ) compresses 16-bit model weights down to 4-bit precision while preserving salient outlier weights. This shrinks total VRAM requirements by approximately 70 percent, allowing 70B models to execute on two 24GB GPUs instead of requiring four 80GB accelerators.

What is prefix caching in vLLM and how does it improve latency?

Prefix caching enables vLLM to identify and reuse already computed KV cache states for shared prompt prefixes, such as standard system instructions or multi-shot examples. This dramatically lowers Time-To-First-Token (TTFT) and conserves GPU compute across repetitive queries.

#vllm#pagedattention#llm-inference#open-source-ai#gpu-serving#quantization

One AI engineering post, weekly

LLM benchmarks, prompt techniques, and token-cost breakdowns, delivered to your inbox.

Comments

Have a question or something to add? Sign in to join the discussion.

Mahmudul Haque Qudrati — CEO & ML Engineer at Pristren

Mahmudul Haque Qudrati

CEO & ML Engineer

Visionary technologist, software engineer, and machine learning specialist. Founder and CEO of Pristren, directing engineering teams that ship production-grade AI/ML pipelines, mission-critical full-stack applications, and developer tooling. Creator of Zlyqor, the unified team workspace platform. Author of 540+ technical guides and benchmark research reports on large language models, agentic workflows, Model Context Protocol (MCP), and modern web stacks.

PristrenZlyqor