7 Engineering Practices to Cut Token Costs in AI Applications

Cost optimization · 2026-11-20 · ~9 min read

Token cost is the first wall an AI application hits on the way from demo to product. This article is not about "switch to a cheaper vendor" — a one-off saving. These are 7 engineering practices you write into your code once, and they keep paying off. Ordered by ROI, each with implementation notes.

Practice 1: model tiering (biggest payoff)

Not every call needs a flagship model. Classification, format conversion, simple rewrites, intent detection — small models are usually good enough, at an order of magnitude lower price.

Implementation: define a tier per task type in your app; the router is usually a dozen lines:

MODEL_TIERS = {
    "classify":  "small model",   # intent, tagging, formatting
    "summarize": "mid model",     # summaries, rewriting
    "reason":    "flagship model" # complex reasoning, code generation
}

def route(task_type: str) -> str:
    return MODEL_TIERS.get(task_type, MODEL_TIERS["reason"])

Before switching, run a quality assessment per task type (feed real business samples to the small model, spot-check manually). Once quality passes, switch. Behind a unified gateway, switching a model is just changing the model parameter — zero business-code changes.

Practice 2: context trimming — stop stuffing full history into the model

The most common waste in conversational apps is carrying the entire history indiscriminately. Three progressive strategies:

Rule of thumb: many chat apps can trim more than half of the history with no perceivable quality drop — because early turns rarely influence the current answer anyway.

Practice 3: use prompt caching (prefix caching)

Most major models support prefix caching: a stable prefix (system prompt, long reference material) that hits cache is billed at a steep discount. Usage notes:

  1. Put unchanging content first (system prompt → long documents → few-shot examples); variable content (user input, history) goes last;
  2. Keep the prefix byte-identical — even one extra space can invalidate the cache;
  3. Never concatenate timestamps or random IDs into the prefix.

For apps with long system prompts or mounted documents, this alone often cuts input cost by half or more.

Practice 4: constrain output — max_tokens and structure

Output tokens are usually pricier than input, and models tend to treat the limit as a target. Two moves:

resp = client.chat.completions.create(
    model="deepseek-chat",
    messages=messages,
    max_tokens=500,        # reasonable ceiling for summaries
    response_format={"type": "json_object"},  # structured output
)

Practice 5: retry and fallback — stop paying for wasted calls

Waste isn't only tokens in normal calls — it's also how you retry failed ones:

You'd hand-write all of this on a self-built gateway; some managed gateways (including 4-4 AI-style new-api ecosystems) ship channel switching and retries built-in.

Practice 6: batch APIs for offline work

Offline batch jobs (data cleaning, bulk translation, eval scoring) don't need real-time responses. Models with a batch async interface typically price it at ~50% off, and it doesn't consume online rate limits. The test is simple: would it matter if this result came back tomorrow instead of now? If not, put it in the batch queue.

Practice 7: usage monitoring and budget guards

Optimization without monitoring is blind. Minimum setup:

The 4-4 AI console covers both with usage statistics and token quota settings; on a self-hosted stack you'd build this on the one-api-family logs yourself.

Suggested implementation order

OrderPracticeWhy
17. Monitoring & budget guardsdata first — every later optimization needs it; guards prevent incidents
21. Model tieringbiggest payoff, smallest code
33. Prefix caching + 2. Context trimmingmajor wins for long-context apps
44, 5, 6add as your app shape demands

FAQ

Isn't switching to a cheaper vendor more direct?

It works, but it's a one-off gain with risks around channel stability, rate limits and protocol compatibility. Engineering practices compound: optimize once, benefit as you scale. Do both — engineering first, then vendor shopping.

What if the small model isn't good enough?

Tiering isn't wholesale downgrading. Keep flagship models for quality-sensitive tasks; spot-check small-model outputs, and migrate "good-enough" tasks gradually. A hybrid "small model first-pass + flagship review" architecture also works.

How do I verify cache hit rate?

Watch the cache-metering fields in responses (names vary by API), or compare billing across consecutive calls with identical prefixes. The denominator is "prefix volume that could have been cached".

Related articles

Self-hosting one-api / new-api vs. a Managed AI Gateway: Cost, Ops & Security An honest decision framework comparing self-hosted one-api/new-api with a managed AI gateway across hidden ops Connecting Cursor / Claude Code / Cherry Studio to a Custom AI Gateway Configure Cursor, Claude Code and Cherry Studio to use an OpenAI-compatible custom gateway (including 4-4 AI):
Ready to build?

One key for GLM, DeepSeek, Qwen and more — unified billing and usage analytics.

View models & pricing Get started