AI Gateways in 2026: Cloudflare vs Portkey vs LiteLLM vs Custom
An AI gateway sits between your application and LLM providers to handle routing, fallback, caching, rate limiting, and cost tracking. This guide compares Cloudflare AI Gateway, LiteLLM, Portkey, and building your own.
Cloudflare AI Gateway is free, runs at Cloudflare's edge, and requires no infrastructure to manage. It works by changing the base URL in your existing OpenAI SDK calls:
from openai import OpenAI
client = OpenAI(
api_key=os.environ["OPENAI_API_KEY"],
# Add your Cloudflare gateway URL as base_url
base_url="https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_name}/openai",
)
# All requests now flow through Cloudflare gateway
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello"}],
)
Features: real-time analytics dashboard, request/response logging, rate limiting by IP or custom header, edge caching for identical requests. The free tier covers most individual and small team use cases.
Limitation: no semantic caching (must be exact match), limited provider fallback configuration.
LiteLLM
LiteLLM is the open-source choice for teams that need 100+ provider support and want to run everything on their own infrastructure:
import litellm
# Same code works for any provider
response = litellm.completion(
model="openai/gpt-4o", # OpenAI
# model="anthropic/claude-3-5-sonnet-20241022", # Anthropic
# model="together_ai/deepseek-ai/DeepSeek-R1", # Together
# model="ollama/llama3.3", # Local Ollama
messages=[{"role": "user", "content": "Hello"}],
)
LiteLLM also ships a proxy server that exposes an OpenAI-compatible API in front of all providers:
Your application calls localhost:4000 (OpenAI format), and LiteLLM handles provider routing, fallback, and retry logic.
Portkey
Portkey is the enterprise-grade option. Key features that differentiate it:
Semantic caching - cache responses to semantically similar (not just identical) requests, using embeddings to match queries. Can reduce LLM spend by 20-40% on repetitive workloads.
Virtual keys - team members use Portkey virtual keys instead of raw provider API keys, enabling centralized key rotation without updating every service
Guardrails - content filtering, PII detection, and output validation rules that run on every request/response
Per-request routing - route different users to different model tiers based on custom attributes
For teams that want full control without managing LiteLLM or paying Portkey fees, a custom gateway built on Hono (Cloudflare Workers) is surprisingly lightweight:
This runs on Cloudflare Workers (zero cold starts, global edge) and gives you complete control over routing logic. The trade-off is maintenance burden.
Which to Choose
Cloudflare AI Gateway - starting out, need basic analytics and caching, want zero infrastructure
LiteLLM - open source requirement, 100+ provider support, self-hosted everything
Portkey - enterprise, need semantic caching and guardrails, willing to pay SaaS fees
Practical deep-dives on LLMs, developer tools, and AI engineering. No filler. Unsubscribe any time.
// written byFIG. AUTH-01
530
Mahmudul Haque Qudrati
CEO & ML Engineer
CEO and ML Engineer at Pristren. Builds AI-powered software for teams and writes about machine learning, LLMs, developer tools, and practical AI applications.
Open Code Review – An AI-powered code review CLI tool: A Practical Overview
Open Code Review is an open-source CLI tool from Alibaba that uses AI to review code changes. It runs locally, supports multiple LLMs, and costs about $0.01 per review. Here's a practical breakdown.
