Short answer: LLM bill shock happens because most AI features don't cost a fixed amount per use. Each request resends more context than the last, agents make several model calls for one user action, reasoning models bill for thinking you never see, and a few power users drive a large share of the tokens. So the bill grows faster than your user count. The fix is to measure cost per request, per feature and per customer first, then cut the waste in a fixed order.
This guide explains where the extra tokens come from, how to find them in your own logs, which fixes to try first, and when running your own private model is worth it. Vendor billing rules were checked against official documentation in November 2026. The numbers in the worked examples are made up to show the mechanics. They are not benchmarks.
Why the bill grows faster than usage
Most teams budget for AI as requests per user, times users, times a price per request. But providers bill tokens, and the tokens in a request depend on how your product builds it. Three things push that number up:
- Requests get bigger. Chat history, retrieved documents and tool definitions are sent again on every call.
- One user action becomes many calls. Agents plan, call tools, read results and call the model again. Failed calls get retried.
- Output gets more expensive. Reasoning models produce hidden thinking tokens, billed at the output rate, and output usually costs several times more per token than input.
Where the extra tokens come from

1. Chat history is resent every turn
The model has no memory between calls. To continue a conversation, your app sends the whole conversation again. OpenAI's documentation says this directly for chained responses: "all previous input tokens for responses in the chain are billed as input tokens in the API." The same is true when you manage history yourself.
A made-up example shows why this hurts. Say your system prompt is 2,000 tokens and each turn adds 500 tokens of question and answer. Turn 1 sends 2,000 tokens. Turn 20 sends 11,500. Add up every turn and a 20-turn chat sends about 135,000 input tokens. A 40-turn chat sends about 470,000. Twice the turns, about three and a half times the input. The cost per conversation grows much faster than its length, not in a straight line.
2. RAG stuffing
Retrieval-augmented generation (RAG) finds passages from your documents and pastes them into the prompt. It's common to start with a generous setting, such as the top 10 or 20 chunks, because more context feels safer. Every chunk is billed on every request, and in a chat it's billed again on every later turn if you keep it in the history.
3. Agent loops and tool calls
An agent that answers one question might call the model five or ten times: plan, call a search tool, read the result, call another tool, check its work, write the answer. Each call usually carries the full transcript so far, including every tool result. So agent steps compound like chat turns do, inside a single user action.
Tool definitions cost tokens too. Anthropic's pricing page says a tool use system prompt is added whenever tools are present, and that declaring its built-in browser toolset adds about 6,600 input tokens to a request. If you send twenty tool schemas on every call, you pay for them on every call, whether the model uses them or not.
4. Retries
Timeouts, rate-limit errors and responses that fail your JSON validation all trigger retries. A retry resends the full prompt. A retry inside an agent loop can resend the whole transcript. If a validation failure retries three times with the same prompt, one bad request costs four.
5. Reasoning tokens
Reasoning models think before they answer, and you pay for the thinking. OpenAI says reasoning tokens are "not visible via the API" but "still occupy space in the model's context window and are billed as" output tokens. If the request hits the `max_output_tokens` limit during reasoning, you can pay for input and reasoning and get no visible answer at all.
Anthropic's documentation says the same for Claude: thinking tokens are billed as output, and "you are billed for the full thinking process, not the thinking content visible in the response." On newer Claude models, thinking blocks from earlier turns stay in context and are billed again as input. Google's Gemini pricing page labels output prices as "including thinking tokens".
6. Output length
Output tokens usually cost several times more than input tokens on the same model; check your provider's price page for the exact ratio. A prompt change that makes answers longer can raise cost more than one that makes prompts longer.
7. Model upgrades and long-context pricing
Switching to a newer, larger model often changes the price per token, the number of tokens a task needs, and how much the model thinks by default. Long-context pricing differs by vendor too. As of November 2026, Anthropic says Claude 4.6 and later models (except Claude Haiku 5.5) bill a full 1M-token context at standard rates, while Haiku 5.5 charges higher prices for prompts over 100,000 tokens. Google lists higher per-token rates for Gemini 3.1 Pro Preview prompts over 200,000 tokens. Read the pricing page before you upgrade, not after the invoice.
8. Power users
Usage is rarely even. Heavy users paste the largest files and keep the longest threads. On a flat subscription, those customers can cost more in tokens than they pay you. That is where AI gross margin pressure usually starts: not across the whole base, but in a small group of heavy accounts.
Causes, signals and fixes
Use this table to match what you see in your logs to a likely cause.
| Cause | Signal in your logs | First fix |
|---|---|---|
| Chat history resent | Signal in your logs: Input tokens per request rise with turn number | First fix: Summarise or drop old turns; cache the stable prefix |
| RAG stuffing | Signal in your logs: Large, flat input size on every request, even short questions | First fix: Retrieve fewer chunks; rerank; cap context per request |
| Agent loops | Signal in your logs: Many model calls per user action; long tail of 20+ steps | First fix: Cap steps per task; trim tool results before resending |
| Tool definitions | Signal in your logs: High input tokens even on the first call | First fix: Send only the tools each step needs |
| Retries | Signal in your logs: Same request ID or prompt hash billed more than once | First fix: Fix the cause; limit retries; retry only the failed step |
| Reasoning tokens | Signal in your logs: Output tokens far above visible answer length | First fix: Lower the effort or reasoning setting for easy tasks |
| Long output | Signal in your logs: Output tokens per request creeping up after prompt changes | First fix: Set output limits; ask for shorter formats |
| Model upgrade | Signal in your logs: Cost per request jumps on the release date | First fix: Compare cost per task, not price per token, before switching |
| Power users | Signal in your logs: Top few percent of tenants carry a large share of tokens | First fix: Per-tenant quotas; usage-based pricing tiers |
How to measure it
Your provider's invoice tells you the total. It can't tell you which feature or customer caused it. You need your own record of every model call. Log these fields on each request:
- Who and what: tenant ID, user ID, feature name and a request ID that links every call made for one user action.
- Model and settings: model name, reasoning or effort setting, output limit.
- Tokens: input, cached input, output, and reasoning or thinking tokens. OpenAI reports reasoning tokens in `output_tokens_details.reasoning_tokens`. Anthropic reports `output_tokens_details.thinking_tokens` and cache reads in `cache_read_input_tokens`.
- Shape of the work: turn number, agent step number, retry flag, number of retrieved chunks.
- Outcome: success or error, latency, and whether the user accepted or used the result.
Put every model call through one wrapper so nothing skips the log. Then price the tokens and build three views:
- Cost per request, as a distribution. Look at the 95th and 99th percentile, not just the average. Bill shock lives in the tail.
- Cost per feature. Some features cost far more than others for similar value. That is where to focus.
- Cost per customer against revenue per customer. This is your AI gross margin by account. It shows whether pricing, not engineering, is the real fix.
If you're not sure your logs are complete, a technical audit can map every place your product calls a model and what it sends.
The fixes, in the order to try them
Start with changes that cut cost without changing what users see. Leave pricing changes for last, once you know the real numbers.
1. Use prompt caching
Both major providers discount input tokens they have already processed recently, if the start of the prompt is identical. OpenAI enables caching by default on supported models and says cached input is discounted by up to 95%, depending on the model. Anthropic bills cache reads at 0.1 times the base input price, and less on some newer models, with cache writes at 1.25 times for the default five-minute lifetime. On its newest models, OpenAI also bills cache writes at 1.25 times the input price.
The catch is the word identical. OpenAI's documentation says "cache reuse requires the entire rendered prefix to match." Put fixed content first: system prompt, tool definitions, long reference documents. Put anything that changes per request last. A timestamp or user name at the top of the system prompt can stop the cache from ever being used.
2. Trim the context
- Keep the last few turns in full and replace older ones with a short running summary.
- Retrieve fewer, better chunks. Add a reranking step that keeps only the most relevant few, and test on your own questions whether answers hold up.
- Shorten tool results before you send them back. The model rarely needs the full raw API response.
- Set a hard ceiling on input tokens per request, and log when you hit it.
3. Cap agent steps and retries
Give every agent task a maximum number of steps and a token budget. Retry only the step that failed, with backoff, and stop after one or two attempts. If a response fails validation, fix the prompt or use your provider's structured output feature rather than retrying blind.
4. Route easy steps to smaller models
Not every step needs your largest model. Classification, extraction, short rewrites and routing decisions often work well on a small, cheap model. Keep the large model for the hard reasoning. Compare cost per completed task: a cheap model that fails and falls back can cost more than the large one.
5. Control output and reasoning
Set output limits that fit each feature. Ask for the format you need: a short list, a fixed JSON schema, a 100-word summary. On reasoning models, use the lowest effort or reasoning setting that keeps quality acceptable for each task. Anthropic notes that `max_tokens` covers thinking and answer together, so a limit that is too low cuts off answers rather than saving money.
6. Batch anything that isn't live
Overnight reports, document backfills, evaluations and bulk tagging don't need an instant answer. OpenAI's Batch API and Anthropic's Message Batches API both give a 50% discount in exchange for asynchronous processing, with results within 24 hours. Anthropic says most of its batches finish in under an hour, and that batch and caching discounts can be combined.
7. Set per-tenant limits
Give each customer a token or request quota per day or month, with alerts before they hit it. This protects you from runaway agents, scripts and abuse as much as from heavy users.
8. Change the pricing
If cost per customer still varies widely after the fixes above, the problem is the price model. Options include usage tiers, included credits with paid top-ups, a fair-use limit on unlimited plans, or charging separately for the heaviest features. Use your cost-per-customer data to set the limits, so the changes affect the accounts that cause the cost and not everyone else.
Going further: routing, caching, batching and smaller models
The fixes above are where to start. Once you can see cost per request, these go further. Prices in this section are public list prices, checked 20 November 2026, in US dollars per 1M tokens, before tax.
Route by task, with a fallback
The price gap between model tiers is wide. Anthropic lists Claude Haiku 5.5 at $0.10 input and $0.50 output, against $4 and $20 for Claude Opus 5.5: 40 times less. OpenAI lists GPT-6 Luna at $0.10 and $0.50, 20 times less than GPT-6.2 Sol at $2 and $10. Three common ways to route:
- By feature. Fixed rules: tagging, extraction and routing decisions go to the small model; drafting and multi-step reasoning go to the large one. Simple to run and easy to explain.
- By request. A cheap classifier, or the small model itself, decides whether each request is hard enough to need the large model.
- Cascade. Try the small model first, check the output against a schema or test, and escalate only when it fails. Count the failed attempt in the cost, because you pay for it too.
If you route through a gateway, check how fallbacks are billed. OpenRouter, for example, tries the models in your list in order when a call fails, and prices the request using the model that was finally used. Judge every routing rule on an evaluation set drawn from your own traffic, by cost per completed task, before it goes live.
Choose the cache lifetime on purpose
On Anthropic, a cache write costs 1.25 times the base input price for a five-minute lifetime and 2 times for one hour. Reads cost 0.1 times, or 0.05 times on Opus 5.5 and Sonnet 5.5. The arithmetic decides which to use. With the five-minute cache, a prefix reused once already costs less than sending it twice uncached (1.25 + 0.1 against 2). With the one-hour cache, you need at least two reuses within the hour to come out ahead (2 + 0.2 against 3). The longer lifetime suits prefixes that are reused steadily but with gaps of more than five minutes, such as a large shared document behind an internal tool.
Track the cache hit rate for each feature. The OpenTelemetry conventions for generative AI, still an emerging standard, include a `gen_ai.usage.cache_read.input_tokens` attribute for exactly this. A sudden drop usually means something changed at the start of the prompt.
Move work off the live path
Batch processing is 50% off at both OpenAI and Anthropic, and Anthropic's batch discount stacks with its cache pricing. Go through your logs for calls nobody is waiting on: nightly summaries, tagging and embedding backfills, evaluation runs, scheduled reports and re-scoring of old records. Send them to a queue that submits batches, and show users a "ready by" time instead of a spinner.
Make the job smaller, not just the model
A cheaper model tier is one way to shrink the cost. Another is to shrink the job: split one large prompt into steps so the hard step is short, and send each step only the fields it needs. For high-volume, well-defined tasks, a small open model on your own GPUs is a further option, covered below. Whichever you choose, keep a fixed test set from real traffic and rerun it on every model change. A smaller model that fails quietly can cost more to fix than it saves in tokens.
An AI spend audit finds where these savings sit across your LLM, GPU and AI SaaS bills. An LLM observability setup gives you the per-request traces to prove each change worked.
Pricing AI features when tokens are a variable cost
Engineering fixes lower the cost per request. They don't change the shape of the cost: every use still costs you money, and heavy users still cost more. Classic software pricing assumes the next user is nearly free to serve. With AI features, the next request isn't, so the price has to allow for it.
Work out the margin for each feature
Start from your logs, not the price page. Here is a made-up example. An assistant request sends 6,000 input tokens and returns 800 output tokens on a model priced at $2 and $10 per 1M tokens. That is $0.012 of input and $0.008 of output, or $0.02 a request. A user who makes 300 requests a month costs $6. A power user who makes 3,000 costs $60. On a $20 seat, the first leaves room for your other costs; the second loses money every month.
Do this sum for each AI feature, at the median user and at the 95th percentile. Include retries, agent steps and embedding calls, not just the final answer.
Pricing models that fit a variable cost
| Model | How it works | Watch out for |
|---|---|---|
| Seat with fair use | How it works: AI is included in the seat price, with a published usage limit | Watch out for: Set the limit from your 95th-percentile usage, or a few heavy accounts erase the margin |
| Seat plus credits | How it works: Each seat includes a monthly credit allowance, with paid top-ups | Watch out for: Customers need to know what a credit buys. Tie credits to actions, not tokens |
| Usage-based | How it works: Customers pay per action, document or task | Watch out for: Harder for buyers to budget, so they may ask for a monthly cap |
| Paid AI add-on | How it works: AI features sold as a separate tier or add-on | Watch out for: Adoption can stay low if the add-on is hard to try |
| Outcome-based | How it works: Customers pay per resolved ticket, completed review or similar result | Watch out for: You carry the cost of failed attempts. Define a result both sides can count |
Leave room for price changes
Provider prices move in both directions. Google lists Gemini 4 Carbon at $2 input and $10 output per 1M tokens through 30 November 2026, rising to $4 and $20 from 1 December 2026. A feature priced on the old rate with no headroom has just had its model cost doubled.
Price in units customers understand, such as documents summarised or tickets drafted, rather than tokens. Make sure your terms let you change credit rates or usage limits with notice; your lawyers own that wording. And review margin per feature every month alongside the bill, so pricing follows cost instead of trailing it by a quarter.
When private models make sense, and when they don't

Running an open model on your own GPUs changes the cost shape. You stop paying per token and start paying for capacity: GPUs whether they're busy or idle, plus the people to run them. That trade suits some workloads and not others.
| Factor | Favours a pay-per-token API | Favours a private model |
|---|---|---|
| Volume | Favours a pay-per-token API: Low or unpredictable traffic | Favours a private model: Steady, high daily volume |
| Utilisation | Favours a pay-per-token API: GPUs would sit idle most of the day | Favours a private model: GPUs stay busy across the day, or overnight batch work fills the gaps |
| Task difficulty | Favours a pay-per-token API: Needs the strongest frontier model | Favours a private model: Well-defined tasks an open model handles well in your own tests |
| Data rules | Favours a pay-per-token API: Data can go to a third-party provider under your contracts | Favours a private model: Contracts or regulation say data must stay on infrastructure you control |
| Team | Favours a pay-per-token API: No one to own GPUs, upgrades and on-call | Favours a private model: An ops team, or a partner, that can run it |
| Cost predictability | Favours a pay-per-token API: Variable bills are acceptable | Favours a private model: You need a fixed monthly cost |
Be honest about the costs on the private side. GPUs are expensive to rent or buy, and an idle GPU still costs money. Someone has to patch servers, upgrade models, watch latency and handle failures at 3 a.m. You need your own evaluations, because an open model that is good on public tests may still be weaker on your task. And the fixes above still apply: bloated prompts need more GPUs.
A hybrid is common. Keep the hardest, lowest-volume tasks on a frontier API, and move the high-volume, routine steps, or the ones that touch sensitive data, to a private model.
Mono's Own Your AI service sets up open models in a private, isolated section of your own cloud account on rented GPUs, or on servers you own, with personal data removed before the model sees a request and every request logged. Private cloud setups often take two to four weeks. Many teams start in the private cloud and move to their own servers later without rebuilding. If the reason you're looking at private models is a contract that forbids public AI, our post on what BPOs can still automate under AI clauses covers that side. If you're still deciding where AI fits, start with AI adoption.
Sources: OpenAI API documentation: Prompt caching, Batch API, Reasoning models, Conversation state (developers.openai.com, checked November 2026). Anthropic Claude documentation: Prompt caching, Batch processing, Pricing (including long context and tool use pricing), Steering thinking (platform.claude.com, checked November 2026). Google Gemini API pricing (aistudio.google.com/docs/pricing, checked November 2026). List prices for OpenAI, Anthropic and Google models, Anthropic cache pricing, OpenRouter model fallbacks documentation and the OpenTelemetry GenAI semantic conventions, checked 20 November 2026. Prices and rules change often; check your provider's page before acting.
FAQ
What is LLM bill shock?
LLM bill shock is when the bill for an AI feature grows much faster than its usage. It happens because requests get bigger over time: chat history and retrieved documents are resent on every call, agents make several model calls per user action, retries repeat full prompts, and reasoning models bill hidden thinking as output tokens.
Why are our token costs increasing when user numbers are flat?
Usually because each request is getting bigger, not because there are more requests. Check input tokens per request by conversation turn, model calls per user action, retry counts and reasoning tokens. A prompt change, a new tool, a model upgrade or a few heavy customers can raise cost without any growth in users.
Are reasoning tokens billed?
Yes. OpenAI and Anthropic both bill reasoning or thinking tokens as output tokens, even though you don't see them in full. OpenAI reports them as reasoning_tokens and Anthropic as thinking_tokens in the usage data. Google's Gemini pricing also says output prices include thinking tokens.
How much does prompt caching save?
It depends on the provider and how much of each prompt repeats. OpenAI says cached input is discounted by up to 95%, depending on the model. Anthropic bills cache reads at 0.1 times the base input price, or less on some newer models. Savings only apply when the start of the prompt is exactly the same, so put fixed content first.
Is self-hosting an LLM cheaper than using an API?
Sometimes. A private model replaces per-token fees with the fixed cost of GPUs and the people to run them. It tends to pay off for steady, high-volume work that keeps GPUs busy, or when contracts require data to stay on your infrastructure. For low or spiky volume, or tasks that need the strongest frontier model, an API is often cheaper.
