Groq LPU: 800 Tokens/sec and Why It Beats GPU for LLM Inference

GPU vs LPU: The Bandwidth Problem

GPUs are designed for training - massively parallel floating-point throughput. During LLM inference, however, the bottleneck is almost never compute: it is memory bandwidth. Each forward pass reads billions of model weights from DRAM, and DRAM bandwidth tops out around 3 TB/s even on an H100. Most of that time is spent waiting for data, not computing.

Groq's Language Processing Unit (LPU) is built differently. It uses a deterministic, single-threaded compute fabric with large on-chip SRAM - no speculative execution, no cache hierarchies, no DRAM reads during the critical path. All model weights for supported model sizes fit entirely in SRAM, which provides 10 - 80x higher effective bandwidth per watt than DRAM-based GPU inference.

The result: GroqCloud delivers 800+ tokens/sec on Llama 3.1 70B, compared to 30 - 60 tokens/sec on a typical GPU-based API.

Supported Models on GroqCloud (2026)

• Llama 3.1 8B Instruct
• Llama 3.1 70B Instruct
• Llama 3.3 70B Versatile
• Gemma 2 9B IT
• Mistral 8x7B (Mixtral)
• Whisper Large v3 (audio)

Getting Started

Install the SDK:

pip install groq

The groq-python SDK mirrors the OpenAI client API:

from groq import Groq

client = Groq(api_key="YOUR_GROQ_API_KEY")

response = client.chat.completions.create(
    model="llama-3.1-70b-versatile",
    messages=[{"role": "user", "content": "Explain the LPU in plain English"}],
    max_tokens=512,
)
print(response.choices[0].message.content)

Streaming Example

pip install groq

stream = client.chat.completions.create(
    model="llama-3.1-8b-instant",
    messages=[{"role": "user", "content": "Write a sorting algorithm in Rust"}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Because Groq processes tokens deterministically and in-order, the time-to-first-token (TTFT) is typically under 200ms even on the 70B model - comparable to a cached local model response.

Free Tier and Cost

GroqCloud has a generous free tier: 14,400 requests/day and 500k tokens/day per model as of early 2026. Paid tiers are priced at approximately $0.05 - 0.59 per million tokens depending on model size - significantly cheaper than equivalent throughput on AWS or Azure GPU instances.

When to Use Groq vs vLLM vs Ollama

Groq is ideal for user-facing applications where latency is the primary UX metric - chat interfaces, voice assistants, and real-time coding tools.