Text Generation Inference (TGI): HuggingFace's Production LLM Server

What Is TGI?

Text Generation Inference (TGI) is HuggingFace's production-grade server for deploying large language models. It's the backend that powers HuggingFace Inference Endpoints, and it's available as open-source software you can run yourself.

TGI is designed for one thing: serving LLMs at high throughput with low latency. Everything in it - from the batching algorithm to the quantization support - exists to push more tokens per second out of a given GPU.

One-Command Deployment

docker run --gpus all   --shm-size 1g   -p 8080:80   -v /models:/data   ghcr.io/huggingface/text-generation-inference:latest   --model-id meta-llama/Meta-Llama-3-8B-Instruct   --num-shard 1   --max-input-length 4096   --max-total-tokens 8192

TGI downloads the model weights, loads them onto the GPU, and starts serving an OpenAI-compatible API on port 8080.

OpenAI-Compatible API

TGI implements the OpenAI Messages API, so any code written for OpenAI works without modification:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="tgi",
    messages=[{"role": "user", "content": "What is quantum computing?"}],
    max_tokens=500,
    stream=True,
)

for chunk in response:
    print(chunk.choices[0].delta.content, end="", flush=True)

Continuous Batching

Standard server implementations process one request at a time. TGI's continuous batching algorithm accepts new requests mid-generation, filling GPU capacity that would otherwise sit idle. This dramatically improves throughput under concurrent load - often 5-10x more tokens per second compared to naive sequential serving.

Tensor Parallelism

For models too large for a single GPU, TGI splits tensor computations across multiple GPUs:

--num-shard 4   # splits the model across 4 GPUs

Quantization Support

TGI natively supports GPTQ, AWQ, and bitsandbytes 4-bit quantization. For GPTQ/AWQ, use a pre-quantized model from the Hub:

--model-id TheBloke/Mistral-7B-Instruct-v0.2-GPTQ --quantize gptq

Speculative Decoding

For tasks where output length is predictable (code completion, structured output), speculative decoding uses a small draft model to propose multiple tokens that the main model validates in one forward pass. This can double effective tokens-per-second for compatible workloads.

TGI vs vLLM

Both are production LLM servers with continuous batching. TGI integrates more tightly with the HuggingFace ecosystem and handles gated models (Llama, Gemma) with better authentication support. vLLM has broader model architecture support (including models not on HuggingFace) and a more active community around PagedAttention research. For standard HuggingFace models in a production setting, TGI is the lower-friction choice.

Resources

• TGI GitHub
• TGI docs