Keeping AI model costs down in production.

The way to keep model costs down in production is to control volume: make fewer calls, send fewer tokens with each one, and use the smallest model that does the job well. The price per token matters far less than the tokens a system spends without anyone noticing.

Nobody gets a surprise model bill in month one. The surprise arrives months later, when someone finally looks at the line and finds it has multiplied without any single decision having caused it. Model spend behaves less like a subscription and more like cloud storage: it accumulates from choices that each looked free at the time.

How are LLM costs calculated?

LLM costs are calculated per call, from the number of tokens you send in and the number of tokens the model sends back. Every request is metered, including the ones that failed, repeated or were never used.

Tokens in and tokens out

A token is a chunk of text, often a short word or a piece of a longer one. Each model family splits text in its own way, so the same paragraph can count as a different number of tokens with different providers.

Input tokens are everything you send: instructions, examples, the user's message, earlier turns of the conversation, retrieved documents and tool definitions. Output tokens are what the model writes back. Output tokens are commonly priced higher than input tokens, though the gap depends on the provider and the model.

Some models also work through internal reasoning before they answer. Depending on the provider, those reasoning tokens can be billed as output even though they never appear in the response, so check how yours reports them.

As an illustrative example, with invented round numbers that are not any provider's rates: say input costs 1 unit per thousand tokens and output costs 4 units per thousand. A call that sends 3,000 tokens and receives 500 costs 3 units for input and 2 for output. Input is six times the tokens but only a little over half the cost, which is why both sides of the call are worth watching.

Why do LLM costs keep rising?

LLM costs keep rising because spend drifts upward through many small changes that are each reasonable on their own. A spike gets noticed. Drift does not, because every increment is defensible and none of them appears on the invoice as a separate line.

Someone adds two more examples to a prompt to fix an edge case. Someone raises a retry limit during an incident and never lowers it. A summarisation step is added before the main call, doubling the requests per item. Meanwhile, usage grows.

None of those is a mistake. The trouble is that they multiply rather than add. Cost per item is roughly calls per item, times tokens per call, times the price per token. As plain arithmetic, if calls per item and tokens per call each rise by a fifth, cost per item rises by just under half, before volume has grown at all.

A model cost problem is usually a volume problem wearing a price tag.

A pilot hides all of this, because a small sample with a few friendly users produces a bill too small to study. Cost at real volume is one of the things a pilot never tests, alongside the other gaps that make AI pilots stall before production.

Where does the money actually go?

Most of the money goes on tokens and calls that do not need to exist. Once a drifted system is instrumented, the spend is rarely where the team expected.

Context that stopped earning its place

Prompts accumulate. Each fix for a failure adds an instruction or an example, and nothing removes them. After months of this, much of the prompt addresses problems that no longer occur, and it is paid for on every single call.

Conversation history resent on every turn

Many model APIs are stateless, which means each call has to include the whole conversation so far. Every new turn pays again for every earlier turn. As an illustrative example, if each turn adds about 500 tokens, the tenth turn sends around 5,000 tokens of input, and the conversation as a whole has sent around 27,500. Even where a provider stores the conversation for you, earlier turns are usually still counted as input, so check how yours bills it.

Silent retries

A timeout, a malformed response, a rate limit: each retry is a full-price call, and retry logic often exists in more than one layer. Many client libraries retry some errors by default, so an application retry wrapped around them, inside a job queue that also retries, multiplies attempts. Three layers that each allow three attempts can make twenty-seven calls for one item. A request your code gave up on may also have finished on the provider's side and been billed anyway.

Reprocessing unchanged inputs

The same document, the same question and the same answer, computed fresh every time because nothing remembered it. A nightly job that reprocesses a whole folder when three files changed, or a summary regenerated on every page load, pays full price for work already done. In many systems this is the largest saving available.

The largest model for every task

The most capable model is usually chosen during development, for the hardest case, and then inherited by everything else. That includes classifying into a few categories, pulling a handful of known fields from a form and short rewrites, which a much smaller and cheaper model often handles just as well.

Agent loops and tool calls

An agent works in steps, and each step is a model call that resends the instructions, the tool definitions and everything that has happened so far. A task that takes a dozen steps pays for that growing context a dozen times. An agent that gets stuck repeats the same tool call or keeps searching for something that is not there, and without a limit it keeps paying. Clear escalation paths that hand an agent's work to a person are as much a cost control as a quality one.

Outputs longer than needed

Models are often wordy by default: they restate the question, explain their reasoning unasked, and wrap a single value in several sentences. When software reads the output, every word beyond the data it needs is spent on the side of the call that is usually priced higher.

How do you track LLM costs per feature?

You track LLM costs per feature by logging token counts, model and a feature tag for every call at the moment it is made. A monthly invoice shows the total and very little else. Attribution at the point of the call answers the questions that matter: which feature costs the most, what one unit of work costs, and whether that number is moving.

Most provider APIs return token counts with every response, so this is mostly a matter of recording what is already there. Add a transaction id so calls can be grouped, an attempt number so retries are visible, and whether the answer came from a cache.

call:
  feature:        invoice_extract
  transaction_id: txn_48213
  model:          small
  tokens_in:      1,240
  tokens_cached:  900
  tokens_out:     180
  attempt:        1
  cache_hit:      false
  outcome:        accepted

Work out cost from those logs with a rate table you maintain, and version it, because prices change and old records should keep the rate they were incurred at. Reconcile the logged total with the invoice each month. A gap usually means calls that bypass the logging, such as a script or an old integration using production keys. This logging belongs in the same layer as the rest of your monitoring, which is the kind of work our backend and cloud engineering covers.

Cost per transaction as the headline number

A transaction is one unit of business work: an invoice processed, a ticket answered, a conversation completed. Cost per transaction is the cost of every call in a period, including retries, escalations and failed attempts, divided by the units of work completed. That is the number worth putting on a dashboard, ahead of total spend.

Total spend rising while cost per transaction falls is a business growing. Total spend flat while cost per transaction rises is a system degrading, and it is the case that would otherwise go unnoticed.

Alerts on movement rather than totals

An alert on cost per transaction moving more than a set amount week-on-week catches drift while it is still a five-minute fix. A threshold on total spend fires only once the damage is done, and it also fires for good reasons, such as a busy month.

Keep one hard ceiling as well, a daily spend limit well above normal, as a backstop against a runaway loop or a leaked key. The signals below usually explain a movement.

How do you reduce LLM costs without hurting quality?

You reduce LLM costs without hurting quality by removing unnecessary calls first, moving work to smaller models second and trimming tokens third, testing every change against an evaluation set. The large numbers are in calls that should never have been made, and the risk to quality rises as you move from removing waste to changing what the model sees.

Effort and risk below are relative and depend on how the system is built.

How does caching reduce token costs?

Caching reduces token costs in two different ways: a response cache skips the model call entirely, while provider-side prompt caching makes the repeated part of a call cheaper. They solve different problems, and most systems with repeated work can use both.

Response caching

A response cache stores the model's answer and returns it when the same request arrives again. The cache key should be a hash of everything that shapes the answer: the model, the full prompt and its version, settings such as temperature, and the input itself.

Models do not always give the same answer twice, but for extraction, classification or summarising an unchanged document, one valid answer is as good as another, so reusing it is safe. Answers that depend on the date, on who is asking or on data that changes need an expiry or a key that accounts for it.

Before the cache, hash documents on arrival so duplicates are caught early, and have scheduled jobs handle only what changed. Semantic caching, which reuses an answer for a question that is merely similar, removes more calls but can return the right answer to the wrong question, so it needs its own evaluation.

Provider-side prompt caching

Many providers offer prompt caching. When the opening of a prompt matches one sent recently, the provider reuses the work it already did on that shared opening, typically charges less for those tokens and often responds faster. It is designed to leave the model's answer unchanged.

Because it matches from the start of the prompt, structure matters. Put stable content first (instructions, examples, tool definitions, a long reference document) and variable content last (the user's message). A timestamp or a customer name near the top breaks the match for everything after it.

The details vary by provider. Some apply it automatically and others need you to mark what to cache, some charge extra to write to the cache, there is often a minimum prompt length, and cached content expires, often after a short idle period. Use the cached token counts in your logs to confirm it is actually happening.

Routing work to the smallest capable model

After caching, the next lever is matching each task to the smallest model that meets the quality bar.

Routing by task

Routing by task means choosing the model for each call site based on what that call does, rather than one default everywhere. Classification, sending a request to the right queue, extracting known fields and short rewrites rarely need the largest model. Drafting a complex reply from several sources, or reasoning over a long contract, may well need it.

Decide with an audit. List every call site with its volume and current model, then run the highest-volume ones against a smaller model on their evaluation sets, because a saving there is multiplied by the most calls. Where a small model falls just short on a narrow, stable task, tuning it on that task can close the gap, and the trade-offs are covered in our guide to choosing between fine-tuning and retrieval.

Model cascades

A model cascade tries a small model first and escalates to a larger one only when the result fails a check. The checks must be cheap and dependable: the output matches the schema, required fields are present, totals add up, a category is one of the allowed values, a date is a real date. A model saying it is confident is a weak check on its own.

Escalated cases pay twice, so a cascade only saves money when the small model passes most of the time. Log the escalation rate per call site. If it creeps up, the inputs have changed or the checks are too strict.

Sending fewer tokens in and out

The remaining lever is the size of each call, and it is where quality is most at risk.

Trimming prompts against an evaluation set

Prompts get shorter far more easily than people expect once someone reads them end to end. Remove instructions for failures that no longer happen, merge rules that say the same thing twice, and keep only the examples that still change the result.

Do it against an evaluation set: a fixed collection of real inputs with known good outcomes, run before and after every change. Without one, trimming is guesswork and a regression surfaces weeks later as complaints. Output varies from run to run, so the comparison needs care, and our guide to testing LLM applications whose output varies covers how to make it meaningful.

Limiting retrieved context and history

Set a token budget for retrieved context, fetch fewer and better passages, drop near-duplicates, and lower the number of passages step by step while watching quality on the evaluation set. Past a certain point, extra passages add cost and noise without improving answers.

For conversations, keep the last few turns word for word and replace older ones with a short running summary or with structured state, such as the order number and what has been agreed. For agents, drop bulky tool results once they have been used, and cut large API responses down to the fields the model needs.

Capping output length and asking for structure

Tell the model the shape and length of answer you want, and ask for structured output wherever software reads the result. Many providers can hold a response to a defined schema, which keeps answers short, removes preamble and takes away a common cause of parse failures and the retries they trigger.

Set a maximum output length on every call too, and treat it as a safety net. A model that hits the limit stops mid-answer rather than writing more concisely, so log truncated responses and fix the prompt or raise the limit when they appear.

Batch jobs, retries and agent limits

Batch processing for work that is not urgent

Plenty of AI work does not need an answer in seconds: overnight document processing, backfills, reclassifying an archive. Some providers offer a batch interface for this, where you submit many requests together and collect the results within a stated window, and some price it lower than the same requests sent one at a time.

If no person is waiting for a pipeline's result, it is a candidate for batching, or at least for running at quieter times when rate limits and the retries they cause are less likely.

Retry policy with caps and backoff

A sound retry policy retries only errors that might succeed next time, a limited number of times, with a growing wait between attempts. Rate limits, timeouts and temporary server errors are worth retrying. An invalid request, an input that is too long or an authentication failure will fail identically the second time.

• Cap attempts per call, and let only one layer retry.
• Wait longer after each failure, with a little randomness so clients do not all retry at once, and honour any retry-after signal from the provider.
• Treat a malformed response as a prompt or schema problem: one repair attempt that includes the error is reasonable, repeating the same request is not.
• Log every attempt against its transaction, so retries show up in cost per transaction.

Step limits and budgets on agents

Every agent needs a maximum number of steps and a token or spend budget per task. When either is reached, the agent stops and hands the task to a person with what it has found so far. The same tool called with the same arguments twice in one task is a common sign of a loop and worth stopping on.

Give each step only the tools it needs, since tool definitions are input tokens on every call.

Budgets, owners and what to leave alone

Budgets, quotas and a named owner

Spend without an owner drifts fastest. Name one person responsible for model spend, with the dashboard, the alerts and the authority to get prompts or models changed when a number moves.

Set a monthly budget per feature, and use separate projects or API keys for production, staging and experiments, so a test run cannot eat the production budget and every call can be attributed. Many providers let you set spend limits or usage alerts at account or project level; use them as a backstop behind your own alerts. For customer-facing features, add quotas per user or per customer account so one heavy or abusive user cannot run up the bill for everyone.

Visibility also depends on where the bill lands. On AI Agency's packages, cloud and model provider costs are billed to the client's own account, so clients see them directly.

What is not worth optimising

Some cost work is engineering time spent on the smallest term. A saving per call only matters when it is multiplied by a lot of calls.

• Shaving a few tokens off a prompt that is already short.
• Tuning an internal tool that runs a few dozen times a week.
• Switching providers over small per-token price differences before measuring where the tokens go.
• Caching requests that almost never repeat, where the cache adds complexity and saves nothing.
• Cutting evaluation runs to save tokens, when they protect every other change.

A monthly cost review routine

A short monthly review, run by the spend owner and whoever maintains the prompts, stops drift coming back.

1. Reconcile logged cost with the provider invoices, and chase any gap.
2. Rank features by total spend and by cost per transaction, and compare both with last month.
3. For the top features, check calls per transaction, retry rate, cache hit rate, escalation rate and average tokens in and out.
4. Read the most expensive transactions of the month end to end, because they show loops, oversized context and wasted retries faster than a chart.
5. List the prompt, model and pipeline changes made during the month and match them to the movements.
6. Choose one or two changes, test them against the evaluation set, and ship them with a note of the effect you expect.
7. Update budgets, quotas and the rate table if usage or provider prices have changed.

Key takeaways

• Model costs are driven by volume: calls made, tokens sent and received, and the model chosen for each call.
• Spend drifts through small, reasonable changes that multiply, so it rarely shows up as a single spike.
• Log tokens, model, feature and attempt on every call, make cost per transaction the headline number, and alert when it moves.
• Remove unnecessary calls first with caching, deduplication and capped retries, then route work to smaller models.
• Trim prompts, retrieved context and output only against an evaluation set, so quality is protected.
• Give agents step limits and budgets, and give model spend a named owner and a monthly review.

Where to start with model spend

Start with visibility. If you cannot say what one transaction costs today, add per-call logging with feature tags before changing anything else, because every later decision depends on it.

Then take the cheapest wins: cache and deduplicate repeated work, cap retries in a single layer, and move stable content to the top of each prompt so provider prompt caching can apply. After that, audit model choice on your highest-volume call sites, with an evaluation set in place before you change a single prompt.

If your bill has been creeping up, or you are designing a new system and want cost built in from the start, schedule a call and we can look at where your tokens are going together.

Frequently asked questions

What is the difference between prompt caching and response caching?

Response caching skips the model call entirely by returning a stored answer when an identical request arrives again. Prompt caching, which many providers offer, still runs the call but charges less for the opening of a prompt that matches one sent recently. Response caching suits repeated work such as reprocessing the same document. Prompt caching suits long, stable instructions shared by many different requests, and most systems with repeated work can use both.

Is a smaller model always cheaper overall?

No, a smaller model is cheaper per token but not always cheaper per completed task. If it fails more often, the retries, escalations to a larger model and human corrections can cost more than using the larger model in the first place. It may also need a longer prompt with more examples to reach the same quality. Compare models on the cost per accepted result from your own evaluation set, not on the price list alone.

How much should an AI feature cost per transaction?

There is no universal figure, because the right cost per transaction depends on what the work is worth. Start from the value of one unit of work, such as the staff time saved on an invoice or the margin on an order, and set a target well below it. Then track actual cost per transaction against that target, and treat a rising trend as a problem even while the total bill still looks small.

Does fine-tuning reduce LLM costs?

Fine-tuning can reduce LLM costs when it lets a smaller model handle a narrow, stable task that previously needed a larger one, or when it removes long instructions and examples from every prompt. It has its own costs too: preparing training data, running the training, evaluating the result and repeating the work when the task changes. How tuned models are priced for use depends on the provider, so compare the full cost at your real volume.

Is self-hosting an open-weight model cheaper than paying per token?

Self-hosting can be cheaper at high, steady volume, but usually is not at low or uneven volume. With a hosted API you pay for what you use. With your own deployment you pay for the hardware whether it is busy or idle, plus the engineering time to run, scale, monitor and update it. Measure your real token volume and its daily pattern first, then compare both options on their total running cost.

How do I stop an AI agent from running up a large bill?

Give every agent a maximum number of steps and a token or spend budget per task, and make it stop and hand the task to a person when either is reached. Detect repeated tool calls with the same arguments, which usually indicate a loop. Add a spend limit at account or project level as a final backstop, and alert on steps per task so a change in behaviour is noticed early.

Why does the same text cost a different number of tokens with different models?

The count differs because each model family uses its own tokeniser, the component that splits text into tokens. The same sentence can become a different number of tokens depending on the model, and the gap is often larger for code, tables and languages other than English. That makes price per token a poor way to compare providers on its own. Run a sample of your real inputs through each option and compare cost per task.

Does setting a maximum output length reduce answer quality?

A maximum output length usually has no effect on how the model writes, but it stops the response once the limit is reached. If the limit is too low, answers end mid-sentence or structured output breaks, which can trigger retries that cost more than the limit saves. Ask for concise answers in the prompt, set the limit comfortably above normal length as a safety net, and log truncated responses so you can adjust it.

Schedule a call