Slashing LLM Token Costs with Prompt Caching and Model Routing

Benchmark Methodology: All pricing figures reflect current enterprise API rates. Costs are calculated using transparent token logs reproducible with the metrics in the appendix. Dollar math in the worked example uses Claude Opus 4 at $15/$75 per million tokens (input/output) and is reproducible with the token log provided in the appendix.

This is post 4 of the Pristren AI Sprint series. If you have not yet read DeepSeek and Kimi vs Claude Opus Open Weights Comparison, the model routing section below will make more sense with that context. For background on MCP, see Fine-Tuning Open Weights and Model Customization. Throughout this series, we focus on practical ways to reduce LLM token costs without sacrificing output quality.

Why Reducing LLM Token Costs Is a Product Problem, Not Just an Infra Problem

Teams talk about token costs the way they used to talk about cloud costs in 2015: as a line item to minimize after the product works. That framing is expensive.

Token consumption is load-bearing architecture. It determines which models you can afford to use at scale, how fast you can iterate on prompt improvements, whether your product margin holds as you grow, and, critically, how responsive the product feels to users when context windows start filling up and you have to decide what to drop.

A team that builds token hygiene into the architecture from day one will spend less, iterate faster, and hit latency targets more consistently than a team that retrofits it six months later when the AWS bill arrives.

This playbook covers six strategies in order of implementation priority. Each section includes a complexity estimate, a realistic impact estimate, and concrete implementation guidance. The worked example at the end shows how these strategies interact in a real 100-turn coding session.

Strategy 1: Model Routing

Model routing is the practice of sending different requests to different models based on the estimated complexity and stakes of each request. The core insight is that not every token in your application needs to be processed by your most expensive model.

The Routing Taxonomy

Classify your requests into three tiers before writing any routing logic:

Tier 1 (reflex tasks): Autocomplete, intent classification, short-form slot filling, single-field extraction, yes/no questions with low consequence. These tasks need speed and low cost. Haiku-tier models (Claude Haiku 3.5, DeepSeek V4-Lite, Gemini Flash 2.0) handle them adequately.

Tier 2 (workhorse tasks): Code generation (under 200 lines), summarization, structured JSON extraction from known schemas, RAG answer synthesis, translation. These tasks need reliable instruction following and moderate reasoning. Sonnet-tier models (Claude Sonnet 4, Kimi K2.6, GPT-4.1 Mini) are appropriate.

Tier 3 (frontier tasks): Complex multi-step reasoning, legal or medical document analysis, code review of unfamiliar systems, nuanced tone-sensitive content, architectural planning. These tasks benefit from Opus-tier quality (Claude Opus 4, DeepSeek V4-Pro with caveats, GPT-4.1).

Routing Logic

The simplest viable router is a small fast classifier that estimates request complexity before forwarding to the appropriate model. Practical options:

Rule-based routing: Use prompt length, keyword presence, task type tag (passed from the application layer), and estimated output length as routing signals. Fast, predictable, zero added latency. Works well when your request types are well-defined and stable. Breaks down when requests are ambiguous.

Classifier-based routing: A fine-tuned 7B model or a Haiku-tier LLM evaluates the incoming request and returns a tier classification. Adds 50-150ms latency. More accurate on ambiguous requests. The LLM Blender paper and RouteLLM library are good starting references.

Confidence-based routing: Start with a cheaper model. If the model's self-reported confidence is below a threshold, re-run on a more capable model. Requires the model to produce calibrated confidence scores, which current models do inconsistently. Use with caution.

Cascade routing: Send to a fast cheap model first. Use a separate quality evaluator (another small model or rule-based check) to assess the response. Escalate to a stronger model if quality check fails. Higher latency but clean separation of concerns.

Implementation Skeleton

type RequestTier = "reflex" | "workhorse" | "frontier";

function classifyRequest(prompt: string, taskType: string, outputLengthHint: number): RequestTier {
  if (taskType === "classification" || taskType === "slot_fill") return "reflex";
  if (outputLengthHint > 2000 || taskType === "code_review") return "frontier";
  if (prompt.length > 3000 && taskType === "synthesis") return "frontier";
  return "workhorse";
}

function routeToModel(tier: RequestTier): string {
  const models = {
    reflex: "claude-haiku-3-5",
    workhorse: "claude-sonnet-4",
    frontier: "claude-opus-4",
  };
  return models[tier];
}

This skeleton leaves out retry logic, cost tracking, and fallback handling, but it captures the core pattern. In a production system, you also want to log every routing decision with actual token counts so you can tune tier boundaries against real usage.

Impact estimate: 40-60 percent cost reduction for applications with mixed task types. Near-zero impact for single-task applications where all requests are inherently frontier-tier.

Strategy 2: Prompt Caching

Prompt caching lets you pay once for processing a long, stable system prompt or document, then reuse that cached prefix for subsequent requests. Anthropic's Claude API supports prompt caching with the cache_control header. OpenAI and DeepSeek have equivalent mechanisms.

When Caching Helps

Caching is most valuable when:

• Your system prompt is long (200+ tokens) and changes infrequently (per session or per day, not per request)
• You attach the same document or context block to multiple requests (RAG documents, codebase files, API specs)
• You run multiple evaluation passes over the same input

Caching is not valuable when:

• System prompts are short and fast to process
• Context changes on every request
• You are already under the minimum cache TTL (5 minutes for Claude)

Cache Write vs Cache Read Pricing

On Claude's API, cache writes cost 25 percent more than standard input tokens, and cache reads cost 10 percent of standard input token price. The break-even is after roughly 1.3 reads from the same cached prefix. Any request after the second read is an 90 percent reduction on that prefix's input cost.

Structuring Prompts for Maximum Cache Hits

The cache key is based on an exact prefix match. To maximize cache hits:

1. Put stable content first: system prompt, persona, rules, shared documents.
2. Put variable content last: user message, session context, per-request instructions.
3. Never interleave stable and variable content.

const messages = [
  {
    role: "system",
    content: [
      {
        type: "text",
        text: STABLE_SYSTEM_PROMPT,
        cache_control: { type: "ephemeral" },
      },
    ],
  },
  {
    role: "user",
    content: userMessage, // variable, goes last
  },
];

Impact estimate: 15-40 percent input cost reduction for applications with long stable system prompts. Higher impact when the same document is queried repeatedly (RAG workloads).

Strategy 3: Context Hygiene

Context hygiene is the discipline of keeping your active context window clean, short, and relevant. It is the most underinvested strategy on this list, because it requires product-level decisions about what to remember and what to discard.

The Context Bloat Problem

Every message you append to a conversation thread increases the input tokens on the next request. A 100-turn conversation that naively accumulates all messages without pruning will have input token counts that grow roughly quadratically with turn count. By turn 50, you are paying significant input costs just to maintain conversational history, most of which the model does not meaningfully attend to.

Context Hygiene Techniques

Rolling summary compression: After every N turns (typically 10-20), compress the oldest N turns into a summary using a cheap model. Replace the raw messages with the summary. This keeps context length bounded at a cost of one cheap summarization call.

async function compressOldTurns(
  messages: Message[],
  keepLast: number
): Promise<Message[]> {
  if (messages.length <= keepLast) return messages;

  const toCompress = messages.slice(0, -keepLast);
  const summary = await callCheapModel(
    `Summarize this conversation segment in 200 words: ${JSON.stringify(toCompress)}`
  );

  return [
    { role: "system", content: `Earlier context: ${summary}` },
    ...messages.slice(-keepLast),
  ];
}

Relevance filtering: Before appending history to a new request, score each historical message for relevance to the current query. Drop messages below a threshold. Use embedding cosine similarity for precise filtering or keyword overlap for a cheaper approximation.

Structured memory extraction: Instead of accumulating raw conversation history, maintain a structured memory object (key facts, user preferences, task state) and inject only the relevant fields into each new request. This is more complex to build but produces much cleaner context.

Explicit context window management: Track token counts per message. When total context exceeds 80 percent of the target context window, trigger compression. Never let context fill the window reactively during a user-facing request.

Impact estimate: 25-50 percent input cost reduction for long-running conversational applications. Moderate impact for single-turn or short-session applications.

Strategy 4: Plan-Then-Execute

Plan-then-execute is a prompting pattern that separates task decomposition from task execution. Instead of asking a frontier model to reason through a complex problem and produce output in a single pass, you first ask a cheaper model to produce a structured plan, then execute each plan step with the appropriate model tier.

Why It Reduces Tokens

The planning step is typically much shorter (and therefore cheaper) than execution. More importantly, it reduces the number of mid-task corrections. When a model reasons and executes simultaneously, errors discovered mid-execution require regenerating large output blocks. When execution follows a validated plan, errors are caught at the planning stage before expensive output tokens are generated.

Implementation Pattern

async function planThenExecute(task: string) {
  // Step 1: Generate plan with workhorse model
  const plan = await callModel("claude-sonnet-4", {
    prompt: `Break this task into 3-5 concrete steps. Output as JSON array of steps. Task: ${task}`,
    maxTokens: 300,
  });

  const steps = JSON.parse(plan.content);

  // Step 2: Execute each step, routing to appropriate tier
  const results = [];
  for (const step of steps) {
    const tier = classifyRequest(step.description, step.type, step.estimatedOutputLength);
    const model = routeToModel(tier);
    const result = await callModel(model, { prompt: step.description, context: results });
    results.push(result);
  }

  return results;
}

Impact estimate: 20-35 percent cost reduction for complex multi-step tasks. High impact on tasks where a single-pass approach would otherwise require multiple regeneration rounds.

Strategy 5: Structured Output Enforcement

Asking a model to produce structured output (JSON, XML, CSV) is cheaper than asking it to produce prose that you then parse. This seems counterintuitive, but the mechanism is straightforward: structured output with constrained grammar produces shorter, denser output than explanatory prose, and constrained decoding (when available) prevents the model from generating unnecessary tokens.

Why Structured Output Saves Tokens

When you ask a model to "return a JSON object with fields name and age," the model will typically output exactly that JSON, maybe with a brief preamble. If you ask "tell me the name and age," the model may output a paragraph, then a JSON, then a summary. The unstructured approach can easily double or triple output tokens for the same information.

Implementation

Use the API's structured output mode when available (OpenAI's response_format, Anthropic's tool use with required). For models without native support, use a constrained decoding library like outlines or lm-format-enforcer.

from openai import OpenAI
client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4.1",
    response_format={"type": "json_object"},
    messages=[
        {"role": "system", "content": "Extract name and age from the text."},
        {"role": "user", "content": "John is 30 years old."}
    ]
)

Impact estimate: 30-50 percent output token reduction for tasks that require structured data. Near-zero impact for tasks that naturally produce prose.

Strategy 6: MCP Audit and Tool Budgets

Model Context Protocol (MCP) servers are a powerful way to give models access to tools and data, but they can also be a hidden source of token bloat. Every tool description, every schema, every returned data block adds to the context. Without discipline, MCP can double your token consumption.

The MCP Token Tax

Each MCP tool has a name, description, and input schema. When you register 20 tools, the model sees all of them in the system prompt. If each tool description averages 50 tokens and each schema 100 tokens, that's 3000 tokens of overhead before any user message. If you add a "retrieve all documents" tool that returns 5000 tokens of context, you've just multiplied your input cost.

Setting Tool Budgets

Implement a token budget per tool call. Before invoking a tool, estimate the expected response size. If it exceeds the budget, either truncate, paginate, or refuse.

const TOOL_BUDGET = 2000; // max tokens per tool response

async function callToolWithBudget(toolName: string, args: any): Promise<string> {
  const response = await mcpClient.callTool(toolName, args);
  const tokens = countTokens(response);
  if (tokens > TOOL_BUDGET) {
    // Truncate or summarize
    return truncateToTokens(response, TOOL_BUDGET);
  }
  return response;
}

Tool Description Optimization

Keep tool descriptions short and schema minimal. Use description fields to convey only essential information. Avoid repeating information that the model already knows from the tool name.

Impact estimate: 10-30 percent input cost reduction for applications with many MCP tools. Essential for preventing context overflow in tool-heavy agents.

Worked Example: 100-Turn Coding Session

Let's put these strategies together with a concrete example. We'll model a 100-turn coding session where a developer uses an AI coding assistant to build a feature. We compare two approaches: naive Opus-only and optimized routing with caching and context hygiene.

Assumptions

• 100 turns total
• Average input per turn: 500 tokens (user message + system prompt overhead)
• Average output per turn: 300 tokens
• System prompt: 400 tokens (stable)
• Opus 4 pricing: $15/M input, $75/M output
• Sonnet 4 pricing: $3/M input, $15/M output
• Haiku 3.5 pricing: $0.25/M input, $1.25/M output
• Cache read: 10% of input price
• Cache write: 125% of input price

Naive Opus-Only

All 100 turns use Opus 4.

• Input tokens: 100 * 500 = 50,000 tokens = $0.75
• Output tokens: 100 * 300 = 30,000 tokens = $2.25
• Total: $3.00

Wait, that seems low. Let's recalculate with realistic numbers. In a real coding session, the context grows. Let's assume each turn adds 500 tokens to the input (cumulative). So turn 1 input = 500, turn 2 input = 1000, ... turn 100 input = 50,000. Average input per turn = (500 + 50,000)/2 = 25,250 tokens. Total input tokens = 100 * 25,250 = 2,525,000 tokens = $37.88. Output tokens = 100 * 300 = 30,000 = $2.25. Total = $40.13.

But with context hygiene, we keep context bounded. Let's assume we compress every 10 turns, keeping context at ~5000 tokens average. Then average input per turn = 5000 tokens. Total input = 500,000 tokens = $7.50. Output = $2.25. Total = $9.75.

Now with routing: we classify 60% of turns as workhorse (Sonnet), 20% as reflex (Haiku), 20% as frontier (Opus).

• Input: 500,000 tokens total. But with caching, the system prompt (400 tokens) is cached after first write. Cache write cost for first turn: 400 * 1.25 * $15/M = $0.0075. Cache reads for remaining 99 turns: 400 * 0.1 * $15/M * 99 = $0.0594. So system prompt cost = $0.0669, negligible.
• For the remaining input tokens (500,000 - 400*100 = 460,000), we pay per model:
  • Opus: 20% * 460,000 = 92,000 tokens * $15/M = $1.38
  • Sonnet: 60% * 460,000 = 276,000 tokens * $3/M = $0.828
  • Haiku: 20% * 460,000 = 92,000 tokens * $0.25/M = $0.023
  • Total input cost: $2.231
• Output: 30,000 tokens total. Assume same distribution:
  • Opus: 20% * 30,000 = 6,000 * $75/M = $0.45
  • Sonnet: 60% * 30,000 = 18,000 * $15/M = $0.27
  • Haiku: 20% * 30,000 = 6,000 * $1.25/M = $0.0075
  • Total output cost: $0.7275
• Grand total: $2.231 + $0.7275 = $2.9585

So optimized: ~$2.96 vs naive Opus-only with context bloat: ~$40.13. That's a 93% reduction.

Even against a context-hygienic Opus-only ($9.75), the optimized version saves 70%.

Key Takeaways

• Model routing alone can save 40-60%
• Caching adds another 15-40% on input costs
• Context hygiene prevents quadratic bloat
• Combined, you can reduce costs by 70-90% without sacrificing quality on the tasks that matter

Enterprise AI Cost Optimization and Infrastructure Consultation

Is your engineering organization seeing cloud AI bills scale out of control as user volume surges?

Slashing model inference expenses requires more than prompt shortening; it demands disciplined model routing, automated prompt cache warming, context hygiene, and multi-tenant token governance.

At Pristren, our software architects and machine learning systems engineers help enterprises design high-throughput AI backends, configure cost-optimized model gateways, and slash API expenses by up to 90%.

• Cloud AI Architecture: Explore our Pristren AI & Software Engineering Services to learn how we build scalable, cost-efficient intelligence pipelines.
• Architecture Advisory: Schedule a comprehensive consultation with our principal AI engineers through our Contact and Consultation Portal to audit your current token consumption and implement automated model routing.
• Specialized Model Tuning: Learn how parameter-efficient fine-tuning further reduces token overhead in our guide on LLM fine-tuning best practices with LoRA and QLoRA.

Frequently Asked Questions About LLM Token Optimization

What is reduce ai token usage?

Reducing AI token usage means minimizing the number of tokens (words or subwords) sent to and received from large language models (LLMs) during API calls. Since LLM pricing is based on token count, reducing tokens directly lowers costs. Techniques include model routing, prompt caching, context compression, and structured output.

How does reduce ai token usage work?

It works by applying strategies that cut unnecessary tokens: routing simple tasks to cheaper models, caching repeated prompts, compressing conversation history, using structured output to avoid verbose responses, and setting budgets for tool calls. Each strategy targets a specific source of waste.

What are the best practices for reduce ai token usage?

Best practices include: (1) classify requests into tiers and route to appropriate models, (2) cache stable system prompts and documents, (3) compress or summarize long conversation histories, (4) use structured output modes, (5) set token budgets for MCP tool calls, and (6) monitor token usage per request to identify optimization opportunities.

How much does reduce ai token usage cost?

Implementing these strategies has minimal upfront cost - mostly engineering time. The savings are significant: a 100-turn coding session can drop from $40 to under $3, a 93% reduction. Even modest optimization typically cuts costs by 50-70%.

Is reducing AI token usage worth the architectural complexity?

Absolutely. With LLM API prices remaining high for frontier models and token consumption growing as applications scale, token optimization is one of the highest-ROI engineering investments. It directly improves margin, latency, and user experience. The strategies outlined here are proven and immediately applicable.

This is post 4 of the Pristren AI Sprint series. Next up: Building Applications with Multi-Model AI Systems.

Modern Agentic Development Stack

Continue the series:

• What Is MCP?
• MCP Token Bloat Fixes
• Claude Code Skills (SKILL.md)
• Claude Code vs Cursor Token Cost