COST OPTIMIZATION · UPDATED JULY 2026
8 ways to cut your LLM bill
LLM cost optimization is the practice of reducing what you spend on model API calls without degrading output quality. The main levers are model routing, response caching, token reduction, and provider-native prompt-cache preservation — in roughly that order of impact for most teams. This page covers all of them, vendor-neutral, including which ones you can build yourself this afternoon and which ones are genuinely ongoing maintenance.
Where the money actually goes
Before optimizing anything, it helps to know which pattern is actually inflating the bill. In most codebases it's some mix of these four:
Expensive models on simple work
A frontier model answering something a much cheaper model — or a 7B open model — could handle just as well.
The same question, paid for repeatedly
A common support question or templated task, answered fresh (and billed fresh) every single time it's asked.
Conversation history dragged along
Thousands of tokens of prior context sent to answer one small follow-up, with no compression and no provider cache reuse.
Verbose output, not just verbose input
Output tokens are typically priced higher than input tokens, and models left to their default verbosity generate a lot of them — especially agents and coding tools.
The eight levers
Roughly ordered by how much a typical team saves per engineering-hour invested. Not every lever applies to every workload — the “when it doesn't help” note is as important as the technique itself.
Model routing (right-sizing)
Send simple, low-stakes requests to a cheaper model tier and reserve the expensive model for work that actually needs it. The failure mode is a router that only looks at which model was requested — a good one looks at request signals (length, tools, structured output, a difficulty score) and defaults to the requested model when it isn't clearly safe to downgrade.
- Do it yourself
- A rules engine keyed on prompt length and a few heuristics, revisited as your traffic mix changes.
- When it doesn't help
- Traffic that's already uniformly hard, technical, or long-context has little room to route down.
Exact response caching
Identical request in, identical response out, skip the model call entirely. This is the easiest, safest lever — it never changes output — but it only fires on byte-identical requests, which caps how much of your traffic it can touch.
- Do it yourself
- A hash of the normalized request as a cache key, a TTL, and a place to store responses.
- When it doesn't help
- Traffic where every request is meaningfully unique won't hit an exact cache at all.
Semantic response caching
Recognize a reworded but equivalent prompt — 'capital of France?' vs 'whats the capital of France???' — and serve the prior answer. This is where most of the real risk lives: without guards against changed numbers, negation, roles, or instructions, a semantic cache will eventually serve a confidently wrong answer.
- Do it yourself
- An embedding model, a vector index, a similarity threshold, and — critically — deterministic guards on top of the similarity score, not just the score alone.
- When it doesn't help
- Anything requiring a fresh answer — current events, randomness, personalization — needs cache exclusion, not a higher threshold.
Input token reduction
Trim recognized filler from the newest message without touching conversation history. Mutating history to save tokens is a common mistake — it looks like a win until it invalidates the provider's own prompt-cache discount on that conversation, which usually costs more than the trim saved.
- Do it yourself
- A filler-detection pass on the latest message only, deliberately leaving prior turns byte-for-byte identical.
- When it doesn't help
- Short prompts with no filler to trim get essentially nothing from this lever.
Preserving provider-native prompt caching
Anthropic's cache_control breakpoints and OpenAI's automatic prefix caching give a real discount for reusing a long, unchanged prefix — but any proxy or middleware that rewrites requests can silently break the prefix match and turn every call into a full-price cache miss. Anything sitting between your app and the provider should be tested for this specifically.
- Do it yourself
- Route cache_control-bearing requests through the provider's native API unmodified, and test that a repeated prefix actually reports as a cache hit.
- When it doesn't help
- Single-shot requests with no repeated prefix (no long system prompt, no multi-turn conversation) have nothing to preserve.
Output length discipline
Output tokens are usually priced higher than input tokens, and models default to more verbose answers than most use cases need. A system-prompt instruction or a dedicated 'dense output' mode asking for shorter, less conversational answers can meaningfully cut output-token spend on agentic and coding workloads especially — those generate a high volume of output tokens by nature.
- Do it yourself
- A system-prompt addition, suppressed for tool calls and structured output where verbosity isn't the issue.
- When it doesn't help
- Workloads that already require long-form output (drafting, summarizing long documents) won't benefit and may need the opposite instruction.
Batch APIs for non-real-time work
OpenAI and Anthropic both offer batch processing endpoints at a meaningful discount off standard pricing, in exchange for asynchronous turnaround (typically within 24 hours) instead of an immediate response. For any workload that doesn't need a live answer — nightly summarization, bulk classification, dataset labeling — this is free money that routing and caching don't touch.
- Do it yourself
- Check whether your workload actually needs real-time responses before defaulting to the synchronous API — this is a vendor-native feature, not something a gateway needs to build.
- When it doesn't help
- Anything user-facing and interactive, by definition, can't wait for asynchronous turnaround.
Cross-provider failover and pricing awareness
A provider outage that triggers an uncontrolled retry stampede can spike a bill in minutes. Bounded failover — moving to a backup provider on a defined set of failures (429, 5xx, timeout), at most once per provider — controls both the reliability and the cost side of an incident, rather than let retries run unchecked.
- Do it yourself
- A fixed, ordered fallback chain with a hard cap on retries per provider — never an unbounded retry loop.
- When it doesn't help
- Single-provider setups with no fallback candidate configured have nothing to fail over to.
How to measure whether it's working
Optimization work you can't measure tends to quietly regress. Three rules keep the number honest:
Compute a shadow baseline, not a guess
For every optimized request, calculate what it would have cost with no optimization applied — the model you'd have called, real token counts, current list prices — and compare that to what you actually paid. A savings number without a computed baseline is a marketing number.
Sign the number
A failover onto a pricier model, or a routing decision that turned out wrong, should count AGAINST the total — not get quietly dropped from the calculation. If your only failure mode is silence, you'll never notice when optimization is losing money.
Don't double-count provider-side discounts
Provider-native prompt caching is a real discount, but it's the provider's discount, not your optimization layer's. Keep it in a separate line item, or you'll overstate what your own routing and caching are actually contributing.
This is exactly how SlashSpend's own dashboard computes savings — actual cost vs. a cache-aware unoptimized baseline, signed, with provider prompt-cache reuse tracked separately and never counted as SlashSpend savings.
Build it yourself, or use a gateway?
Every technique above is buildable in-house. The honest question is whether you want to own the ongoing maintenance.
- ✓Full control over routing logic and cache behavior.
- ✓No new vendor, no new bill.
- ○Cache-safety guards, routing edge cases, and model-registry upkeep become your team's ongoing work, not a one-time build.
- ✓Routing, caching guards, and prompt-cache preservation already built and maintained.
- ✓A dashboard computing the shadow baseline for you, the way the section above describes.
- ✓Trades engineering time for a monthly fee — worth checking that the fee's incentive lines up with yours: does it grow with your spend, or stay flat? We wrote about that here.
Where SlashSpend fits
SlashSpend implements levers 1, 2, 3, 4, 5, and 8 above automatically — model routing, exact and semantic caching, input token reduction, provider prompt-cache preservation, and bounded cross-provider failover — behind a one-line base-URL change, with the dashboard computing the shadow-baseline savings number described above for your real traffic.
It doesn't touch levers 6 and 7 (output-length discipline and batch APIs) — those are worth doing directly against your provider regardless of what gateway, if any, you use. And results genuinely depend on your traffic mix: teams with a real share of repeated or simple requests tend to land near a ~40% ceiling; teams with mostly unique, hard, frontier-tier traffic will see less, and we'd rather say that than take a flat fee for nothing.
Frequently asked questions
How much can I realistically save on LLM API costs?
It depends entirely on your traffic mix, not a fixed percentage. If a real share of your calls are repeated questions, templated tasks, or work a cheaper model can safely handle, teams typically see savings in the 20-40% range. If your traffic is mostly unique, hard, frontier-tier prompts, there is less to optimize — sometimes close to nothing. Be skeptical of any vendor that quotes one number without asking about your traffic first.
Does caching AI responses hurt quality or freshness?
Exact caching (identical request in, identical response out) never changes output — it just skips a redundant call. Semantic caching (near-duplicate prompts) can hurt quality if it is not scoped carefully: it needs guards against changed numbers, negation, roles, and instructions, and should be excluded for anything requiring a fresh answer (current events, anything with randomness or personalization). A cache without those guards will eventually serve a wrong answer confidently.
Will routing to a cheaper model make my app worse?
Only if the routing logic is naive. Difficulty-aware routing should look at request signals — length, whether tools or structured output are requested, confidence scoring — and route conservatively, defaulting to the requested model when it isn't clearly safe to downgrade. The failure mode to watch for is a router that only looks at the model name you asked for and ignores what the request actually needs.
What's the difference between semantic caching and provider prompt caching?
They solve different problems. Semantic caching skips the model call entirely for a near-duplicate prompt — it lives in front of the provider. Provider prompt caching (Anthropic's cache_control, OpenAI's automatic prefix caching) is a discount from the provider itself for reusing a long, unchanged prefix within a conversation — the model still runs, just cheaper. A cost-optimization setup should count these separately; conflating a provider's own discount with your tool's savings inflates the numbers.
Do I need a gateway to reduce LLM costs, or can I do this myself?
You can build all of this yourself — routing rules, a cache layer, token trimming — and plenty of teams do. The tradeoff is ongoing engineering time: cache-safety guards, routing edge cases, and keeping a model registry current are real maintenance, not a one-time build. A gateway trades a monthly fee for not owning that maintenance.
How do I measure whether my LLM cost optimization is actually working?
Compute a shadow baseline: what would this exact traffic have cost without any optimization, using real token counts and current list prices, and compare it to what you actually paid. Track it as a signed number, not just savings — a failover onto a pricier model should count against the total, not get quietly excluded. And don't count a provider's own prompt-cache discount as a win for your optimization layer; that's a separate, provider-side effect.
Keep reading
- What is an LLM gateway? — category definition and how routing, caching, and failover actually work under the hood.
- OpenRouter alternatives — an honest comparison, including when to just keep OpenRouter.
- Anthropic prompt cache TTL explained — the actual documented default, and how to request the 1-hour tier explicitly.
- Docs — request controls, response headers, streaming.
- Leaving guide — the one-line rollback, documented before you need it.
Skip the build. Keep the savings.
Change your base URL to SlashSpend and see routing, cache, and failover decisions on every response — or use this page as a checklist to build it yourself.