How to reduce LLM API costs: nine levers, in the order to apply them

When a team tells us their OpenAI or Anthropic bill has doubled, the first question is rarely "which model?". It is "what are you sending?". In almost every workload we review, the money goes on text the application sends to the model, not on the answers that come back. This guide walks through the nine changes we check in every LLM cost audit, in the order we apply them.

Where the money goes

A typical production request carries four kinds of input: a system prompt with instructions and examples, documents pulled in by retrieval, the user's message or ticket, and, for agents and chat, the history of earlier turns. Output is usually a few hundred tokens. Even though output tokens cost more per token, the volume of input usually wins. In our sample audit of a fictional support-software company, input made up 95.7% of all tokens.

That has a practical consequence. The fastest savings rarely come from asking for shorter answers. They come from sending less, sending it more cleverly, or sending it to a cheaper model.

Why order matters

Savings overlap. If you first cut a system prompt in half and then cache it, the caching saving is smaller than it would have been on the original prompt. Add up savings that were each estimated against the original bill and you will promise more than you can deliver. We apply each lever on top of the previous ones, feature by feature, and record only the extra saving each one adds.

The nine levers

1. Trim system prompts

Prompts grow by accretion: a rule added after each incident, the same instruction phrased three ways, a dozen worked examples. Every token is paid for on every call. Remove repeated rules, keep two or three examples, and move rarely needed instructions into the cases that need them. Our Prompt Lab flags repetition and buried instructions automatically.

2. Retrieve fewer documents

Retrieval systems are often set to fetch eight or ten chunks "to be safe". Log which retrieved chunks each answer actually uses and find the rank that covers about 95% of answers. In our sample audit, answers almost never used anything below the sixth result, so the last two of eight were pure cost.

3. Ask for shorter answers

Set an expected length and format that match what users need, based on approved answers, not on a guess. This matters most for drafting and summarising features.

4. Give agents memory and a step limit

Agents resend their whole history at every step, so cost grows faster than the number of steps. Summarise older turns and cap the number of steps, with a hand-off to a person when the cap is reached.

5. Use structured outputs

If a feature expects JSON, ask the provider for schema-constrained output. Every malformed answer you retry is a call you paid for twice.

6. Cache repeat answers

Help centres and FAQ bots see the same questions again and again. Serving identical questions from your own cache costs nothing per hit. Keep personalised or account-specific answers out of it.

7. Route simple work to smaller models

Classification, extraction and simple questions rarely need the largest model. Split each feature into simple and complex requests, test smaller models on both, and route only where quality holds. This is usually the single largest saving, and the one that most needs testing.

8. Turn on provider prompt caching

Both major providers discount input they have seen recently. OpenAI applies caching automatically to prompts longer than 1,024 tokens, and it works best when static content such as instructions and examples comes first and variable content comes last. Anthropic lets you mark cache points explicitly; writing to the five-minute cache costs 1.25 times the normal input rate and the one-hour cache twice the rate, while reads are discounted. The practical rule for both: keep the start of every prompt byte-for-byte identical. A date or a customer name in the first line breaks the cache.

9. Batch work that can wait

Nightly summaries, report generation and backfills do not need an answer in seconds. OpenAI's Batch API and Anthropic's Message Batches both charge 50% less than the standard rate in exchange for asynchronous processing within 24 hours.

Keeping quality where it is

A saving that lowers answer quality is a cost moved elsewhere, usually to your support team. Before changing anything, collect 20 to 50 approved answers per feature: real inputs paired with outputs your team signed off. Replay them on every candidate setup and score the results. We only recommend a change if it stays within one point of the current setup on a 0 to 100 scale, and we test simple and complex requests separately, because a model that handles the easy half well can fail badly on the hard half.

How to measure your own bill

Start with your provider's usage data rather than estimates. OpenAI's Usage API reports input, cached input and output tokens and request counts per day, model and project. Anthropic's usage report gives the same token breakdown, including cache reads and writes. Pair one month of that with a few thousand logged requests and you can see exactly which feature, and which part of each request, drives the bill.

Common mistakes

• Counting savings twice by estimating each change against the original bill.
• Switching models on a hunch without a test set, then discovering the failures in production.
• Breaking the cache by putting timestamps or user names at the start of the prompt.
• Self-hosting too early. Running your own model has a fixed monthly cost; below a certain volume an API is cheaper. Work out your break-even point first.