LLM Cost Optimization Cheatsheet: Prompt Caching, Batching and Model Routing

The first LLM bill is usually a pleasant surprise. The second one, after the feature has real users and the prototype has grown into an agent with tools, retrieval and a long system prompt, is usually not.

LLM costs grow in ways that are easy to miss. Agents resend the whole conversation on every turn. RAG stuffs thousands of tokens of context into each call. Reasoning models think for a while before they answer, and thinking tokens are billed as output. A feature that cost a cent per request in testing can cost twenty cents in production, multiplied by every user and every retry.

The good news: a few techniques, applied in the right order, routinely cut costs by half or more without touching answer quality. The most important one, prompt caching, is close to free to adopt if you structure prompts correctly. This cheatsheet covers where the money goes, the main levers, and how to measure whether they work.

1. Where the money goes

Every major provider bills by tokens, with different prices for different kinds of tokens.

Two consequences follow:

• Input volume usually dominates the bill, especially for agents and RAG. Output is more expensive per token, but there is usually much less of it.
• The same input sent repeatedly is the biggest waste. That is exactly what prompt caching fixes.

The agent multiplier

A chat or agent loop resends the full history on each turn. If each turn adds t tokens to the context, the total input over n turns grows roughly with n * n * t / 2. A 30-turn agent session with 2,000 tokens added per turn processes on the order of a million input tokens, even though the final context is only 60,000. Long agent sessions are a quadratic cost problem, and caching turns most of that into cheap cached reads.

2. The levers, in order of effort vs. payoff

The order matters. Caching and batching are pure wins with no quality trade-off, so do them first. Routing and smaller models can save more, but only with an eval suite that tells you when quality drops.

3. Prompt caching

A prompt cache stores the model's processed state for a prefix of your prompt. If the next request starts with exactly the same tokens, the provider reuses that work: you pay a fraction of the input price for the cached part, and the first token arrives sooner.

How the main APIs do it (mid-2026)

Exact discounts and minimums differ by model and change over time. Check the current pricing page before you build a business case on a number.

Structure prompts for cache hits

Caching is all about stable prefixes. The rule: static first, dynamic last.

[tools / function definitions]        <- never changes      (cache)
[system prompt, instructions]         <- rarely changes     (cache)
[long reference docs, few-shot set]   <- per tenant / doc   (cache)
[conversation history]                <- grows, append-only (cache up to last turn)
[the new user message]                <- changes every call (not cached)

Things that silently break caching:

Is caching worth it? A quick rule

With Anthropic-style pricing (write 1.25x, read 0.1x), a cached prefix pays for itself as soon as it is read once within the TTL: 1.25 + 0.1 = 1.35 for two requests, versus 2.0 without caching. Every further hit costs 0.1 instead of 1.0. With automatic caching (OpenAI, Gemini implicit), there is no write premium, so the only work is keeping prefixes stable.

4. Batch APIs: 50% off for things that can wait

Most providers offer a batch interface. You submit a file of requests, the provider processes them asynchronously, and you collect the results later, at a large discount.

Good batch candidates: nightly classification or tagging, document summarization pipelines, embeddings and backfills, eval runs, synthetic data generation, re-processing an archive after a prompt change. Rule of thumb: if no human is waiting, it should probably be a batch.

5. Context and output: send less, generate less

Input side

• Retrieve less, retrieve better. Five relevant chunks beat twenty mediocre ones, for cost and usually for quality. A reranker that lets you send fewer chunks often pays for itself.
• Compact long histories. Summarize older turns, drop tool outputs that are no longer relevant, and keep only the last N turns verbatim. Do it at a few fixed points, not every turn, so caching keeps working between compactions.
• Trim tool definitions. Long tool descriptions and large JSON schemas are sent on every call. Twenty tools at 300 tokens each is 6,000 tokens per request before the user says anything. Load tools on demand when the platform supports it.
• Do not send what code can compute. Dates, totals, lookups: compute them and pass the result, not the raw data.

Output side

• Ask for the format you need. "Return JSON with fields x, y, z" instead of prose that you then parse.
• Set max_tokens to a sensible ceiling per use case, and monitor how often you hit it.
• Right-size reasoning. Reasoning models expose effort or budget controls. Use high effort for hard problems and low effort or no extended thinking for extraction, classification and formatting. Thinking tokens are billed as output, which makes this one of the biggest levers on reasoning-heavy workloads.
• Stop early. Stop sequences and structured outputs keep the model from adding a summary of what it just said.

6. Model routing and cascades

Not every request needs the best model. A large share of traffic in most products is easy: short questions, simple extraction, classification, formatting.

The research supports the idea. FrugalGPT (Stanford, 2023) used a learned cascade over several commercial APIs and reported matching the best single model's performance at up to 98% lower cost on its benchmarks. RouteLLM (LMSYS and UC Berkeley, 2024) trained routers on human preference data and reported cost reductions of more than 2x in some settings without losing response quality. Your numbers will differ. The pattern holds: if much of your traffic is easy, sending it to a cheaper model is the largest saving available.

The catch: routing without evals is gambling. Build a representative task set first, measure each candidate model on it, and only then decide what goes where. Re-check when models or prompts change.

7. Agent-specific cost traps

Set a budget per task (tokens or money) and enforce it in the harness. An agent that hits its budget should stop and report, not quietly keep going.

8. Measure before and after

You cannot optimize what you do not attribute. Log, per request:

• Model, feature or endpoint, tenant or customer.
• Input, cached-read, cache-write, output and reasoning tokens (from the API's usage object).
• Cost, latency (time to first token and total), and the outcome (success, failure, escalation).

Then track a few derived metrics:

9. A Python cost model

A small, provider-neutral calculator to compare scenarios. Plug in your own prices from the provider's pricing page; the numbers below are placeholders.

from dataclasses import dataclass


@dataclass
class Prices:
    """USD per million tokens. Fill in from your provider's pricing page."""
    input: float
    output: float
    cache_read_mult: float = 0.10   # cached input as a fraction of input price
    cache_write_mult: float = 1.25  # use 1.0 for automatic caching (no write premium)


def session_cost(
    p: Prices,
    turns: int,
    static_prefix: int,       # tools + system prompt + fixed docs
    tokens_per_turn: int,     # new user message + tool results added each turn
    output_per_turn: int,
    caching: bool,
    batch_discount: float = 0.0,
) -> float:
    total = 0.0
    history = 0
    for turn in range(turns):
        new_input = tokens_per_turn
        if caching:
            cached = static_prefix + history if turn > 0 else 0
            written = (static_prefix if turn == 0 else 0) + new_input
            fresh = 0
        else:
            cached, written, fresh = 0, 0, static_prefix + history + new_input
        total += (
            fresh * p.input
            + cached * p.input * p.cache_read_mult
            + written * p.input * p.cache_write_mult
            + output_per_turn * p.output
        ) / 1_000_000
        history += new_input + output_per_turn
    return total * (1 - batch_discount)


prices = Prices(input=3.00, output=15.00)  # placeholder prices
for caching in (False, True):
    cost = session_cost(prices, turns=30, static_prefix=8_000,
                        tokens_per_turn=2_000, output_per_turn=500, caching=caching)
    print(f"caching={caching}: ${cost:.2f} per 30-turn session")

With these placeholder numbers, the uncached session costs about $4.39 and the cached one about $0.88. Caching cuts the session cost by about 80%, without touching the model or the prompt content. The calculator simplifies (it assumes every turn hits the cache and ignores TTL expiry), but it is good enough to see which lever matters for your workload.

Why this matters

• Input tokens, repeated, are the usual culprit. Agents and RAG resend the same prefix again and again.
• Prompt caching is the first, cheapest win. Stable prefixes, static content first, append-only history.
• If nobody is waiting, batch it. Half price for latency you do not need.
• Routing saves the most, but only with evals. Cheaper and wrong is the most expensive outcome.
• Measure cost per successful task, not cost per token, and keep a budget on every agent.

Final Thoughts

Cost optimization for LLMs looks like a pricing problem, but it is mostly an engineering problem. The same habits that make systems fast and reliable make them cheap: stable interfaces, no repeated work, asynchronous processing where possible, and measurement before change.

Start with the boring wins. Reorder your prompts so the static parts come first and turn caching on. Move offline jobs to batch. Then build the eval set you need anyway, and use it to decide which requests really need your most expensive model. Most teams find that the answer is "fewer than we thought".

Sources

• Anthropic, Prompt caching: cache_control breakpoints, 5-minute and 1-hour TTLs, write and read multipliers (1.25x, 2x, 0.1x), prefix order and invalidation rules, usage fields.
• Anthropic, Batch processing: 50% discount, 24-hour processing window, combining batches with prompt caching.
• OpenAI, Prompt caching: automatic caching above 1,024 tokens, prompt_cache_key, cached_tokens, retention options.
• OpenAI, Batch API: 50% discount, 24-hour completion window, 50,000 requests per batch.
• Google, Context caching and Batch mode in the Gemini API documentation: implicit and explicit caching, batch pricing and turnaround.
• Chen, L., Zaharia, M., and Zou, J., FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance (2023): LLM cascades, up to 98% cost reduction at matched performance on the paper's benchmarks.
• Ong, I., et al., RouteLLM: Learning to Route LLMs with Preference Data (2024): learned routers, more than 2x cost reduction in some settings without loss of response quality.