Last updated: June 3, 2026. All pricing figures reflect current API rates as of this date. Dollar math in the worked example uses Claude Opus 4 at $15/$75 per million tokens (input/output) and is reproducible with the token log provided in the appendix.
Why Token Efficiency Is a Product Problem, Not Just an Infra Problem
Teams talk about token costs the way they used to talk about cloud costs in 2015: as a line item to minimize after the product works. That framing is expensive.
Token consumption is load-bearing architecture. It determines which models you can afford to use at scale, how fast you can iterate on prompt improvements, whether your product margin holds as you grow, and, critically, how responsive the product feels to users when context windows start filling up and you have to decide what to drop.
A team that builds token hygiene into the architecture from day one will spend less, iterate faster, and hit latency targets more consistently than a team that retrofits it six months later when the AWS bill arrives.
This playbook covers six strategies in order of implementation priority. Each section includes a complexity estimate, a realistic impact estimate, and concrete implementation guidance. The worked example at the end shows how these strategies interact in a real 100-turn coding session.
Strategy 1: Model Routing
Model routing is the practice of sending different requests to different models based on the estimated complexity and stakes of each request. The core insight is that not every token in your application needs to be processed by your most expensive model.
The Routing Taxonomy
Classify your requests into three tiers before writing any routing logic:
Tier 1 (reflex tasks): Autocomplete, intent classification, short-form slot filling, single-field extraction, yes/no questions with low consequence. These tasks need speed and low cost. Haiku-tier models (Claude Haiku 3.5, DeepSeek V4-Lite, Gemini Flash 2.0) handle them adequately.
Tier 2 (workhorse tasks): Code generation (under 200 lines), summarization, structured JSON extraction from known schemas, RAG answer synthesis, translation. These tasks need reliable instruction following and moderate reasoning. Sonnet-tier models (Claude Sonnet 4, Kimi K2.6, GPT-4.1 Mini) are appropriate.
Tier 3 (frontier tasks): Complex multi-step reasoning, legal or medical document analysis, code review of unfamiliar systems, nuanced tone-sensitive content, architectural planning. These tasks benefit from Opus-tier quality (Claude Opus 4, DeepSeek V4-Pro with caveats, GPT-4.1).
Routing Logic
The simplest viable router is a small fast classifier that estimates request complexity before forwarding to the appropriate model. Practical options:
Rule-based routing: Use prompt length, keyword presence, task type tag (passed from the application layer), and estimated output length as routing signals. Fast, predictable, zero added latency. Works well when your request types are well-defined and stable. Breaks down when requests are ambiguous.
Classifier-based routing: A fine-tuned 7B model or a Haiku-tier LLM evaluates the incoming request and returns a tier classification. Adds 50-150ms latency. More accurate on ambiguous requests. The LLM Blender paper and RouteLLM library are good starting references.
Confidence-based routing: Start with a cheaper model. If the model's self-reported confidence is below a threshold, re-run on a more capable model. Requires the model to produce calibrated confidence scores, which current models do inconsistently. Use with caution.
Cascade routing: Send to a fast cheap model first. Use a separate quality evaluator (another small model or rule-based check) to assess the response. Escalate to a stronger model if quality check fails. Higher latency but clean separation of concerns.
This skeleton leaves out retry logic, cost tracking, and fallback handling, but it captures the core pattern. In a production system, you also want to log every routing decision with actual token counts so you can tune tier boundaries against real usage.
Impact estimate: 40-60 percent cost reduction for applications with mixed task types. Near-zero impact for single-task applications where all requests are inherently frontier-tier.
// stay current
AI & ML insights, weekly
Practical deep-dives on LLMs, developer tools, and AI engineering. No filler. Unsubscribe any time.
// written byFIG. AUTH-01
538
Mahmudul Haque Qudrati
CEO & ML Engineer
CEO and ML Engineer at Pristren. Builds AI-powered software for teams and writes about machine learning, LLMs, developer tools, and practical AI applications.
Prompt caching lets you pay once for processing a long, stable system prompt or document, then reuse that cached prefix for subsequent requests. Anthropic's Claude API supports prompt caching with the cache_control header. OpenAI and DeepSeek have equivalent mechanisms.
When Caching Helps
Caching is most valuable when:
Your system prompt is long (200+ tokens) and changes infrequently (per session or per day, not per request)
You attach the same document or context block to multiple requests (RAG documents, codebase files, API specs)
You run multiple evaluation passes over the same input
Caching is not valuable when:
System prompts are short and fast to process
Context changes on every request
You are already under the minimum cache TTL (5 minutes for Claude)
Cache Write vs Cache Read Pricing
On Claude's API, cache writes cost 25 percent more than standard input tokens, and cache reads cost 10 percent of standard input token price. The break-even is after roughly 1.3 reads from the same cached prefix. Any request after the second read is an 90 percent reduction on that prefix's input cost.
Structuring Prompts for Maximum Cache Hits
The cache key is based on an exact prefix match. To maximize cache hits:
Put stable content first: system prompt, persona, rules, shared documents.
Put variable content last: user message, session context, per-request instructions.
Impact estimate: 15-40 percent input cost reduction for applications with long stable system prompts. Higher impact when the same document is queried repeatedly (RAG workloads).
Strategy 3: Context Hygiene
Context hygiene is the discipline of keeping your active context window clean, short, and relevant. It is the most underinvested strategy on this list, because it requires product-level decisions about what to remember and what to discard.
The Context Bloat Problem
Every message you append to a conversation thread increases the input tokens on the next request. A 100-turn conversation that naively accumulates all messages without pruning will have input token counts that grow roughly quadratically with turn count. By turn 50, you are paying significant input costs just to maintain conversational history, most of which the model does not meaningfully attend to.
Context Hygiene Techniques
Rolling summary compression: After every N turns (typically 10-20), compress the oldest N turns into a summary using a cheap model. Replace the raw messages with the summary. This keeps context length bounded at a cost of one cheap summarization call.
async function compressOldTurns(
messages: Message[],
keepLast: number
): Promise<Message[]> {
if (messages.length <= keepLast) return messages;
const toCompress = messages.slice(0, -keepLast);
const summary = await callCheapModel(
`Summarize this conversation segment in 200 words: ${JSON.stringify(toCompress)}`
);
return [
{ role: "system", content: `Earlier context: ${summary}` },
...messages.slice(-keepLast),
];
}
Relevance filtering: Before appending history to a new request, score each historical message for relevance to the current query. Drop messages below a threshold. Use embedding cosine similarity for precise filtering or keyword overlap for a cheaper approximation.
Structured memory extraction: Instead of accumulating raw conversation history, maintain a structured memory object (key facts, user preferences, task state) and inject only the relevant fields into each new request. This is more complex to build but produces much cleaner context.
Explicit context window management: Track token counts per message. When total context exceeds 80 percent of the target context window, trigger compression. Never let context fill the window reactively during a user-facing request.
Impact estimate: 25-50 percent input cost reduction for long-running conversational applications. Moderate impact for single-turn or short-session applications.
Strategy 4: Plan-Then-Execute
Plan-then-execute is a prompting pattern that separates task decomposition from task execution. Instead of asking a frontier model to reason through a complex problem and produce output in a single pass, you first ask a cheaper model to produce a structured plan, then execute each plan step with the appropriate model tier.
Why It Reduces Tokens
The planning step is typically much shorter (and therefore cheaper) than execution. More importantly, it reduces the number of mid-task corrections. When a model reasons and executes simultaneously, errors discovered mid-execution require regenerating large output blocks. When execution follows a validated plan, errors are caught at the planning stage before expensive output tokens are generated.
Implementation Pattern
async function planThenExecute(task: string) {
// Step 1: Generate plan with workhorse model
const plan = await callModel("claude-sonnet-4", {
prompt: `Break this task into 3-5 concrete steps. Output as JSON array of steps. Task: ${task}`,
maxTokens: 300,
});
const steps = JSON.parse(plan.content);
// Step 2: Execute each step, routing to appropriate tier
const results = [];
for (const step of steps) {
const tier = classifyRequest(step.description, step.type, step.estimatedOutputLength);
const model = routeToModel(tier);
const result = await callModel(model, { prompt: step.description, context: results });
results.push(result);
}
return results;
}
Impact estimate: 20-35 percent cost reduction for complex multi-step tasks. High impact on tasks where a single-pass approach would otherwise require multiple regeneration rounds.
Strategy 5: Structured Output Enforcement
Asking a model to produce structured output (JSON, XML, CSV) is cheaper than asking it to produce prose that you then parse. This seems counterintuitive, but the mechanism is straightforward: structured output with constrained grammar produces shorter, denser output than explanatory prose, and constrained decoding (when available) prevents the model from generating unnecessary tokens.
Why Structured Output Saves Tokens
When you ask a model to "return a JSON object with fields name and age," the model will typically output exactly that JSON, maybe with a brief preamble. If you ask "tell me the name and age," the model may output a paragraph, then a JSON, then a summary. The unstructured approach can easily double or triple output tokens for the same information.
Implementation
Use the API's structured output mode when available (OpenAI's response_format, Anthropic's tool use with required). For models without native support, use a constrained decoding library like outlines or lm-format-enforcer.
from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
model="gpt-4.1",
response_format={"type": "json_object"},
messages=[
{"role": "system", "content": "Extract name and age from the text."},
{"role": "user", "content": "John is 30 years old."}
]
)
Impact estimate: 30-50 percent output token reduction for tasks that require structured data. Near-zero impact for tasks that naturally produce prose.
Strategy 6: MCP Audit and Tool Budgets
Model Context Protocol (MCP) servers are a powerful way to give models access to tools and data, but they can also be a hidden source of token bloat. Every tool description, every schema, every returned data block adds to the context. Without discipline, MCP can double your token consumption.
The MCP Token Tax
Each MCP tool has a name, description, and input schema. When you register 20 tools, the model sees all of them in the system prompt. If each tool description averages 50 tokens and each schema 100 tokens, that's 3000 tokens of overhead before any user message. If you add a "retrieve all documents" tool that returns 5000 tokens of context, you've just multiplied your input cost.
Setting Tool Budgets
Implement a token budget per tool call. Before invoking a tool, estimate the expected response size. If it exceeds the budget, either truncate, paginate, or refuse.
const TOOL_BUDGET = 2000; // max tokens per tool response
async function callToolWithBudget(toolName: string, args: any): Promise<string> {
const response = await mcpClient.callTool(toolName, args);
const tokens = countTokens(response);
if (tokens > TOOL_BUDGET) {
// Truncate or summarize
return truncateToTokens(response, TOOL_BUDGET);
}
return response;
}
Tool Description Optimization
Keep tool descriptions short and schema minimal. Use description fields to convey only essential information. Avoid repeating information that the model already knows from the tool name.
Impact estimate: 10-30 percent input cost reduction for applications with many MCP tools. Essential for preventing context overflow in tool-heavy agents.
Worked Example: 100-Turn Coding Session
Let's put these strategies together with a concrete example. We'll model a 100-turn coding session where a developer uses an AI coding assistant to build a feature. We compare two approaches: naive Opus-only and optimized routing with caching and context hygiene.
Assumptions
100 turns total
Average input per turn: 500 tokens (user message + system prompt overhead)
Average output per turn: 300 tokens
System prompt: 400 tokens (stable)
Opus 4 pricing: $15/M input, $75/M output
Sonnet 4 pricing: $3/M input, $15/M output
Haiku 3.5 pricing: $0.25/M input, $1.25/M output
Cache read: 10% of input price
Cache write: 125% of input price
Naive Opus-Only
All 100 turns use Opus 4.
Input tokens: 100 * 500 = 50,000 tokens = $0.75
Output tokens: 100 * 300 = 30,000 tokens = $2.25
Total: $3.00
Wait, that seems low. Let's recalculate with realistic numbers. In a real coding session, the context grows. Let's assume each turn adds 500 tokens to the input (cumulative). So turn 1 input = 500, turn 2 input = 1000, ... turn 100 input = 50,000. Average input per turn = (500 + 50,000)/2 = 25,250 tokens. Total input tokens = 100 * 25,250 = 2,525,000 tokens = $37.88. Output tokens = 100 * 300 = 30,000 = $2.25. Total = $40.13.
But with context hygiene, we keep context bounded. Let's assume we compress every 10 turns, keeping context at ~5000 tokens average. Then average input per turn = 5000 tokens. Total input = 500,000 tokens = $7.50. Output = $2.25. Total = $9.75.
Now with routing: we classify 60% of turns as workhorse (Sonnet), 20% as reflex (Haiku), 20% as frontier (Opus).
Input: 500,000 tokens total. But with caching, the system prompt (400 tokens) is cached after first write. Cache write cost for first turn: 400 * 1.25 * $15/M = $0.0075. Cache reads for remaining 99 turns: 400 * 0.1 * $15/M * 99 = $0.0594. So system prompt cost = $0.0669, negligible.
For the remaining input tokens (500,000 - 400*100 = 460,000), we pay per model:
Output: 30,000 tokens total. Assume same distribution:
Opus: 20% * 30,000 = 6,000 * $75/M = $0.45
Sonnet: 60% * 30,000 = 18,000 * $15/M = $0.27
Haiku: 20% * 30,000 = 6,000 * $1.25/M = $0.0075
Total output cost: $0.7275
Grand total: $2.231 + $0.7275 = $2.9585
So optimized: ~$2.96 vs naive Opus-only with context bloat: ~$40.13. That's a 93% reduction.
Even against a context-hygienic Opus-only ($9.75), the optimized version saves 70%.
Key Takeaways
Model routing alone can save 40-60%
Caching adds another 15-40% on input costs
Context hygiene prevents quadratic bloat
Combined, you can reduce costs by 70-90% without sacrificing quality on the tasks that matter
FAQ
What is reduce ai token usage?
Reducing AI token usage means minimizing the number of tokens (words or subwords) sent to and received from large language models (LLMs) during API calls. Since LLM pricing is based on token count, reducing tokens directly lowers costs. Techniques include model routing, prompt caching, context compression, and structured output.
How does reduce ai token usage work?
It works by applying strategies that cut unnecessary tokens: routing simple tasks to cheaper models, caching repeated prompts, compressing conversation history, using structured output to avoid verbose responses, and setting budgets for tool calls. Each strategy targets a specific source of waste.
What are the best practices for reduce ai token usage?
Best practices include: (1) classify requests into tiers and route to appropriate models, (2) cache stable system prompts and documents, (3) compress or summarize long conversation histories, (4) use structured output modes, (5) set token budgets for MCP tool calls, and (6) monitor token usage per request to identify optimization opportunities.
How much does reduce ai token usage cost?
Implementing these strategies has minimal upfront cost—mostly engineering time. The savings are significant: a 100-turn coding session can drop from $40 to under $3, a 93% reduction. Even modest optimization typically cuts costs by 50-70%.
Is reduce ai token usage worth it in 2026?
Absolutely. With LLM API prices remaining high for frontier models and token consumption growing as applications scale, token optimization is one of the highest-ROI engineering investments. It directly improves margin, latency, and user experience. The strategies outlined here are proven and immediately applicable.
Reducing AI token usage means minimizing the number of tokens (words or subwords) sent to and received from large language models (LLMs) during API calls. Since LLM pricing is based on token count, reducing tokens directly lowers costs. Techniques include model routing, prompt caching, context compression, and structured output.
How does reduce ai token usage work?
It works by applying strategies that cut unnecessary tokens: routing simple tasks to cheaper models, caching repeated prompts, compressing conversation history, using structured output to avoid verbose responses, and setting budgets for tool calls. Each strategy targets a specific source of waste.
What are the best practices for reduce ai token usage?
Best practices include: (1) classify requests into tiers and route to appropriate models, (2) cache stable system prompts and documents, (3) compress or summarize long conversation histories, (4) use structured output modes, (5) set token budgets for MCP tool calls, and (6) monitor token usage per request to identify optimization opportunities.
How much does reduce ai token usage cost?
Implementing these strategies has minimal upfront cost—mostly engineering time. The savings are significant: a 100-turn coding session can drop from $40 to under $3, a 93% reduction. Even modest optimization typically cuts costs by 50-70%.
Is reduce ai token usage worth it in 2026?
Absolutely. With LLM API prices remaining high for frontier models and token consumption growing as applications scale, token optimization is one of the highest-ROI engineering investments. It directly improves margin, latency, and user experience. The strategies outlined here are proven and immediately applicable.
