7 Engineering Practices to Cut Token Costs in AI Applications
Token cost is the first wall an AI application hits on the way from demo to product. This article is not about "switch to a cheaper vendor" — a one-off saving. These are 7 engineering practices you write into your code once, and they keep paying off. Ordered by ROI, each with implementation notes.
Practice 1: model tiering (biggest payoff)
Not every call needs a flagship model. Classification, format conversion, simple rewrites, intent detection — small models are usually good enough, at an order of magnitude lower price.
Implementation: define a tier per task type in your app; the router is usually a dozen lines:
MODEL_TIERS = {
"classify": "small model", # intent, tagging, formatting
"summarize": "mid model", # summaries, rewriting
"reason": "flagship model" # complex reasoning, code generation
}
def route(task_type: str) -> str:
return MODEL_TIERS.get(task_type, MODEL_TIERS["reason"])
Before switching, run a quality assessment per task type (feed real business samples to the small model, spot-check manually). Once quality passes, switch. Behind a unified gateway, switching a model is just changing the model parameter — zero business-code changes.
Practice 2: context trimming — stop stuffing full history into the model
The most common waste in conversational apps is carrying the entire history indiscriminately. Three progressive strategies:
- Sliding window: keep the last N turns plus one summary; compress everything older;
- Retrieval injection (RAG): don't paste whole documents — retrieve relevant chunks first;
- Structured trimming: compress logs and code (strip blank lines, repeated boilerplate) before sending.
Practice 3: use prompt caching (prefix caching)
Most major models support prefix caching: a stable prefix (system prompt, long reference material) that hits cache is billed at a steep discount. Usage notes:
- Put unchanging content first (system prompt → long documents → few-shot examples); variable content (user input, history) goes last;
- Keep the prefix byte-identical — even one extra space can invalidate the cache;
- Never concatenate timestamps or random IDs into the prefix.
For apps with long system prompts or mounted documents, this alone often cuts input cost by half or more.
Practice 4: constrain output — max_tokens and structure
Output tokens are usually pricier than input, and models tend to treat the limit as a target. Two moves:
- Set
max_tokensto a business-appropriate ceiling (300–500 for summaries; more for code); - Ask for structured output (JSON with explicit fields) — the model won't spend three paragraphs of prose on a boolean.
resp = client.chat.completions.create(
model="deepseek-chat",
messages=messages,
max_tokens=500, # reasonable ceiling for summaries
response_format={"type": "json_object"}, # structured output
)
Practice 5: retry and fallback — stop paying for wasted calls
Waste isn't only tokens in normal calls — it's also how you retry failed ones:
- Exponential backoff + jitter: on 429/5xx back off 1s → 2s → 4s with random jitter to avoid retry storms;
- Idempotency guard: attach a request ID to important calls and deduplicate at the gateway so business-layer resubmits don't double-bill;
- Model fallback chain: when the flagship model is rate-limited, automatically degrade to a mid-tier model instead of retrying the same tier.
You'd hand-write all of this on a self-built gateway; some managed gateways (including 4-4 AI-style new-api ecosystems) ship channel switching and retries built-in.
Practice 6: batch APIs for offline work
Offline batch jobs (data cleaning, bulk translation, eval scoring) don't need real-time responses. Models with a batch async interface typically price it at ~50% off, and it doesn't consume online rate limits. The test is simple: would it matter if this result came back tomorrow instead of now? If not, put it in the batch queue.
Practice 7: usage monitoring and budget guards
Optimization without monitoring is blind. Minimum setup:
- Log per business dimension: model / input+output tokens / task type per call — totals alone never reconcile;
- Week-over-week alerts: a sudden doubling of some call type (usually a prompt regression or an infinite loop) must be visible;
- Hard budget caps: set quota limits at the gateway token level so runaway calls can't burn the account.
The 4-4 AI console covers both with usage statistics and token quota settings; on a self-hosted stack you'd build this on the one-api-family logs yourself.
Suggested implementation order
| Order | Practice | Why |
|---|---|---|
| 1 | 7. Monitoring & budget guards | data first — every later optimization needs it; guards prevent incidents |
| 2 | 1. Model tiering | biggest payoff, smallest code |
| 3 | 3. Prefix caching + 2. Context trimming | major wins for long-context apps |
| 4 | 4, 5, 6 | add as your app shape demands |
FAQ
Isn't switching to a cheaper vendor more direct?
It works, but it's a one-off gain with risks around channel stability, rate limits and protocol compatibility. Engineering practices compound: optimize once, benefit as you scale. Do both — engineering first, then vendor shopping.
What if the small model isn't good enough?
Tiering isn't wholesale downgrading. Keep flagship models for quality-sensitive tasks; spot-check small-model outputs, and migrate "good-enough" tasks gradually. A hybrid "small model first-pass + flagship review" architecture also works.
How do I verify cache hit rate?
Watch the cache-metering fields in responses (names vary by API), or compare billing across consecutive calls with identical prefixes. The denominator is "prefix volume that could have been cached".
Related articles
One key for GLM, DeepSeek, Qwen and more — unified billing and usage analytics.
View models & pricing Get started