Keeping LLM costs predictable as usage grows

The first month of an AI feature usually costs almost nothing. The fifth month, after it has been rolled out to every customer and someone has added a longer context window "just to be safe," can look very different. We have seen a client's model bill grow ninefold in a quarter while usage only tripled. None of it was fraud or abuse; it was a series of reasonable decisions nobody was measuring. Cost control for LLM features is an infrastructure discipline, and it works best when it is designed in from the start.

Know what each feature costs

The single most useful change is attributing spend to features and customers. Provider dashboards show one total, which tells you nothing about where to act. We route every model call through a thin internal gateway that tags it with the feature name, tenant, prompt version and model, then records input tokens, output tokens, cached tokens and latency.

With that in place, questions that used to take a day of guesswork take a single query:

• Which feature accounts for most of the spend, and is it the one generating most of the value?
• Which tenants are outliers, and are they on a plan that covers their usage?
• Did last week's prompt change increase average tokens per request?
• What does one successful outcome, such as a resolved ticket or processed document, actually cost?

That last metric, cost per successful outcome, is the one to put on the dashboard. Cost per request encourages cheaper calls that fail more often; cost per outcome captures the whole picture.

The levers that move the bill

Trim the context

Input tokens dominate most bills. Retrieval pipelines that stuff twenty chunks into every prompt, chat features that resend the entire history, and system prompts that grew by accretion are the usual causes. Reranking and sending the top five chunks instead of twenty, summarizing older conversation turns, and auditing system prompts for dead instructions commonly cut input tokens by 40% to 60% with no measurable quality loss, as long as you confirm that with an evaluation set.

Cache aggressively

Prompt caching, where the provider reuses a processed prefix across calls, is the largest single saving for features with long, stable system prompts or shared reference documents. Structure prompts so stable content comes first and variable content last. On one document-heavy workload, reordering alone raised the cache hit rate from 12% to 81% and cut input spend by more than half. Separately, cache full responses for deterministic, repeated requests such as classifying identical inputs.

Route to the right model

Not every call needs the most capable model. Classification, extraction and short rewrites often run perfectly well on a smaller, cheaper tier. We route by task type, and occasionally by difficulty: a small model tries first, and low-confidence cases escalate. The eval suite decides which routes are acceptable, not intuition.

Cap the output

Set maximum output tokens per feature, ask for structured output rather than prose when a machine will read it, and use batch processing for anything that does not need a real-time answer. Batch pricing is often around half the interactive rate, and nightly enrichment or backfill jobs rarely need a response within seconds.

A cost regression is a bug. It deserves the same test, alert and rollback path as a latency regression.

Budgets and guardrails in production

Attribution tells you what happened. Guardrails stop surprises from happening in the first place. Our standard setup includes:

1. Per-feature daily budgets with alerts at 70% and 100% of expected spend, sent to the channel of the team that owns the feature.
2. Per-tenant rate limits tied to plan, so a single customer's script cannot consume the month's budget overnight.
3. Loop protection for agents: a hard cap on tool calls and total tokens per task, after which the agent stops and hands off.
4. Cost checks in CI: the evaluation run reports average tokens per case, and a change that raises it by more than 15% requires explicit approval.
5. Provider-level spend limits as a final backstop, set well above normal usage but low enough to prevent disasters.

Agent loops deserve special attention. An agent that retries a failing tool call indefinitely can burn through hundreds of dollars in an hour. Every agent we deploy has a step limit and a token ceiling enforced outside the model.

Forecasting and pricing

With per-outcome costs measured, forecasting becomes arithmetic. Multiply expected outcomes by cost per outcome, add a margin for model price changes and usage growth, and you have a number finance can plan around. If you sell the feature to customers, this is also the basis for sensible pricing. We have helped more than one client discover that their heaviest users were unprofitable and adjust plan limits before it became a real problem.

Model prices have generally fallen over time, but not smoothly, and new model versions sometimes use more tokens for the same task. Re-run cost benchmarks whenever you change models, and keep the ability to switch providers behind your gateway so a price change is a configuration decision, not a rewrite.

Where to start

If you already have AI features in production, the order of work that pays back fastest is usually: add attribution through a gateway, set budgets and alerts, reorder prompts for caching, then trim context and route models with your evaluation set as the referee. Most teams recover 30% to 60% of spend within a month without touching quality.

This work spans our LLM integration practice and our infrastructure team, who build the gateway, dashboards and alerting as part of broader AWS setup and management. For teams running many services, we typically codify budgets and alerts alongside everything else using infrastructure as code.

Get a handle on your model spend

Send us last month's model invoice and a list of the features that use it, and we will identify the biggest savings and quote the work to capture them. Start the conversation here; a fixed-price quote follows within 24 hours.