LLM GATEWAY · ROUTING · CACHE · FAILOVER
What is an LLM gateway?
An LLM gateway is a control layer between your application and model providers. It gives your code one stable API while centralizing routing, caching, failover, policy, and request-level observability. The useful question is not only which provider should answer — but whether an expensive model call needs to happen at all.
SlashSpend is a BYOK, OpenAI-compatible gateway built around that economic question. Point an existing client at one base URL, keep your provider accounts, and add task-difficulty routing, semantic response reuse, token reduction, and cross-provider failover. Read the docs.
One base-URL change in. One base-URL change back out.
requested gpt-5
X-SlashSpend-Model-Used: gpt-5-mini
X-SlashSpend-Cache: miss
X-SlashSpend-Routed: true
X-SlashSpend-Failover: armed
# estimates only — not a guarantee
When do you need an LLM gateway?
A direct provider integration is fine when you have one model, one application, and low operational risk. A gateway starts earning its place when model traffic becomes shared infrastructure instead of a single API call.
You use multiple models or providers
A stable gateway boundary keeps provider-specific formats and credentials out of application logic.
Reliability code is spreading
Centralized failure policy is easier to inspect than different retry and fallback loops in every service.
The bill needs active control
Reporting spend after the invoice lands is not the same as deciding, in the request path, whether a cheaper model or cached response is sufficient.
You need request-level receipts
Routing, cache, failover, latency, and model decisions should be visible for each call — not inferred from a monthly dashboard.
Migration risk matters
A compatible boundary lets teams change providers, routing policy, or gateway vendors without rebuilding every client.
Portkey describes the same production trigger: multiple providers, growing reliability expectations, active cost control, and centralized governance. Source.
LLM proxy vs router vs aggregator vs gateway
The names overlap. The job is what matters.
| Layer | Primary job | Typical decision |
|---|---|---|
| LLM proxy | Put a stable endpoint in front of provider APIs | “Where should this request be forwarded?” |
| LLM router | Choose a model or provider using cost, capability, latency, policy, or health | “Which model or provider should handle this?” |
| Model aggregator | Provide access to many models through one API or account | “Which available model do I want to call?” |
| LLM gateway | Combine proxying with some mix of routing, reliability, policy, cost control, and observability | “How should this request be handled end to end?” |
OpenRouter describes itself as a unified API for hundreds of models with automatic fallbacks and cost-effective provider selection — valuable aggregation and provider routing. SlashSpend can run on top of an OpenRouter key and add a different decision: whether a cheaper model — or no new model call — is enough for the job. OpenRouter quickstart.
How an LLM gateway works
- 1
Your application sends a familiar request
SlashSpend implements the OpenAI-compatible Chat Completions shape used by the OpenAI SDK, LangChain, LiteLLM, the Vercel AI SDK, and plain HTTP clients.
- 2
Request controls define the contract
Per-request headers can bypass cache, routing, compression, session locking, or failover when a workload needs direct behavior.
- 3
The cache checks for work already paid for
Exact and semantic reuse can return an eligible stored response without a provider call. Semantic candidates are constrained by model/provider, response policy, instructions, conversation, and deterministic safety guards; structured-output requests skip semantic reuse.
- 4
Routing asks how much model is actually needed
Conservative substitution only happens after checks for request size, configured targets, prompt-cache breakpoints, tools/structured output, scoring confidence, and key availability.
- 5
The selected provider gets one attempt
Eligible transient failures — 429, 5xx, or time-to-first-byte timeout — can move to the next configured provider. Terminal client errors are returned rather than multiplied into more billed attempts.
- 6
The response comes back with a receipt
Headers identify the model used, routing decision, cache result, failover reason, compression state, latency, and request ID. A metadata-only trace is available by request ID.
Full detail in the docs.
Five controls that move cost and reliability
Model routing
Simple work can step down to a lower-cost tier, while hard, structured, tool-using, or low-confidence requests stay on the requested model. SlashSpend exposes why it routed — or declined to route — through response headers, and supports shadow evaluation before more aggressive substitutions.
Semantic response caching
A useful semantic cache recognizes equivalent intent, not just byte-identical text. It also needs boundaries. SlashSpend uses similarity to find a candidate, then applies deterministic guards and request-contract scoping before serving it. Teams can bypass, version, or flush cache entries when a request must be fresh.
Token reduction
Input compression removes only recognized leading filler from the newest user message and leaves conversation history unchanged. Optional dense-output mode asks for shorter answers and is suppressed for tools and structured output. These are separate controls: one reduces what you send; the other asks the model to return less.
Cross-provider failover
Provider reliability is not one retry loop. A gateway should define which failures qualify, which candidates can hold the request, when to stop, and how the application learns that failover occurred. SlashSpend reports the final provider and failover reason instead of hiding the decision.
Observability
Every optimization needs a receipt. SlashSpend returns routing, model, cache, failover, compression, latency, and request IDs in headers. Persistent logs contain metadata rather than prompt or response text by default; opt-in content debugging is scoped to one request and expires after one hour.
BYOK privacy: know what crosses the gateway
A gateway is in the request path, so its data contract matters more than its privacy adjectives.
In SlashSpend's recommended Local Key Mode, provider keys arrive as request headers, are used in memory for the upstream call, and are not written to the database or logs. Prompt and response content is not logged by default. Durable metadata — model, token counts, latency, cost, and cache/route flags — is retained for 90 days. The response cache has a one-hour default TTL; the semantic index stores a vector rather than prompt text. Security.
SlashSpend is not SOC 2 certified and does not offer a BAA today. Teams with those requirements should treat that as a real selection constraint, not a checkbox to waive.
How to choose an LLM gateway
Use a skeptical checklist. Ask each vendor to show the behavior, not only the feature name.
- ?
Compatibility: Which API shapes, streaming modes, tools, functions, and structured outputs are actually supported?
- ?
Routing scope: Does it choose a provider for the same model, a different model tier, or both?
- ?
Cache safety: What partitions the cache? How are semantic matches guarded, bypassed, versioned, and deleted?
- ?
Failure policy: Which errors trigger failover? Can any provider be called twice? What happens to structured output?
- ?
Observability: Can you inspect the model used, routing rationale, cache result, latency, and failover reason for one request?
- ?
Data handling: Where do provider keys, prompts, responses, vectors, and metadata go? How long are they retained?
- ?
Prompt-cache behavior: Does the gateway preserve provider-native prompt caching or accidentally invalidate it?
- ?
Deployment and compliance: Managed, self-hosted, VPC, regional, SOC 2, BAA — what is real today?
- ?
Exit cost: Can you remove the gateway with a configuration change, or has application code become coupled to it?
- ?
Pricing incentives: Does the vendor earn more when your model bill grows?
Where SlashSpend fits — and where it does not
- ✓You already have provider access and want a cost-optimization layer rather than another marketplace.
- ✓Your traffic includes repeated questions, templated tasks, or simple work currently sent to expensive models.
- ✓You want cross-provider failover and request-level traces without rewriting an OpenAI-compatible client.
- ✓You want BYOK control and a documented one-line rollback path. Leaving guide.
- ○Broad model access and consolidated provider selection are the main job; OpenRouter may be the primary layer.
- ○Fully self-hosted control is the priority; LiteLLM may fit that architecture better.
- ○Nearly every request is unique, difficult, and frontier-tier — SlashSpend's own homepage says such traffic may have little to optimize.
- ○You require SOC 2, a BAA, embeddings, image generation, or audio endpoints today. Security.
Frequently asked questions
Is an LLM gateway the same as an API gateway?
It uses the same architectural idea — a stable control point in front of services — but adds model-specific concerns such as token accounting, model/provider routing, prompt and response caching, streaming normalization, structured-output handling, and model failover.
Does an LLM gateway add latency?
Yes, every proxy adds processing and network overhead. The relevant question is end-to-end behavior. A cache hit can avoid generation entirely, while routing and failover can avoid a slow or unsuitable provider. Measure the gateway's own overhead separately from upstream generation time.
Can I use an LLM gateway with OpenRouter?
Yes. OpenRouter can provide broad model access and provider selection. SlashSpend accepts an OpenRouter key and can add task-difficulty routing, semantic response reuse, token reduction, and a last-resort open-model failover candidate.
How does an LLM gateway reduce cost?
Depending on the product, it may route suitable work to a lower-cost model, return cached responses, preserve provider prompt-cache discounts, reduce input/output tokens, enforce budgets, or choose a lower-cost provider host. Results depend on traffic mix; unique, difficult prompts may offer little room to optimize.
Will routing send hard prompts to a weaker model?
That risk is why routing policy and observability matter. SlashSpend declines substitution for structured/tool-using requests, prompt-cache breakpoints, low-confidence difficulty scores, missing keys, or other failed checks. It reports the decision in response headers and supports shadow evaluation.
What happens if the gateway goes down?
A gateway is still a network dependency. SlashSpend documents fail-open behavior for optional internal layers, cross-provider failover for eligible upstream faults, and a one-line base-URL revert if SlashSpend itself is unavailable.
Put a bouncer on the model bill.
Change your base URL to SlashSpend. Keep your provider keys and accounts. See every cache, route, compression, and failover decision on the response.