LLM bills tend to grow in silence. A feature that cost pennies in testing can become a real budget line at scale. These seven techniques usually work, and most are straightforward.
1. Measure per feature first
You cannot fix what you cannot see. Tag each call with feature, customer and model. Often one feature or one bad loop accounts for most of the spend.
2. Use prompt caching
If every call repeats a long system prompt or the same documents, caching reuses that prefix at a heavily discounted rate. Keep stable content at the start of the prompt, and variable content at the end, so the cache can hit.
3. Route by difficulty
Not every task needs your strongest model. Classification, extraction and short summaries often work fine on a small model. Use a router, and escalate to a larger model only when needed. Verify with your evaluation set, because the cheap model must be proven good enough.
4. Send less context
Teams often stuff in more retrieved text “to be safe”. Better retrieval with reranking lets you send fewer passages, cutting cost and often improving answers. Trim conversation history as well, by summarising older turns.
5. Batch what can wait
Overnight reports, bulk enrichment and evaluation runs do not need instant replies. Batch APIs offer substantial discounts for work that can wait.
6. Limit output
Output tokens usually cost more than input. Set sensible maximum lengths and ask for concise formats. Structured output prevents the model from chatting.
7. Cache whole answers
Common questions can be served from a semantic cache without calling a model. Take care with permissions and with data that changes.
Guardrails
Set a budget per feature and per customer, with alerts. Add hard stops on agent loops. Review a cost dashboard weekly. A runaway agent loop is the classic expensive surprise.
Check the quality
Every saving here can damage quality if done blindly. Run your evaluation set before and after each change. A cheaper system that gives worse answers is not a saving.
Try different volumes and models in our LLM Cost Calculator, or ask us about LLMOps.