Latency, Cost & Reliability Budgets
The three production constraints that kill AI features — and the standard levers for each.
Latency: users feel time-to-first-token, so stream everything user-facing — a response that starts in 400ms feels fast even if it takes 8s to finish. Match model size to task: frontier models for hard reasoning, small fast models for classification and routing; a router that sends easy queries to a cheap model is often the single biggest latency and cost win. Parallelize independent calls; never chain sequentially what you can fan out.
Cost: you pay per token, so the levers are fewer tokens, cheaper tokens, and cached tokens. Prompt caching (supported by the major providers) makes the repeated prefix — system prompt, tool definitions, few-shot examples — dramatically cheaper and faster on every call after the first; structure prompts static-first to exploit it. Cap max output tokens per call site, log cost per feature (not per account), and set alerts before finance does. A surprising amount of spend is usually one retry loop nobody meters.
Reliability: providers have outages and rate limits, and your product inherits them unless you design otherwise. The baseline kit: timeouts on every call, retries with exponential backoff and jitter for transient errors, a fallback model (ideally cross-provider, via your gateway), and graceful degradation in the UX — 'here's a cached answer' beats a spinner that dies. Idempotency matters too: a retried request that charges twice or sends two emails is a reliability fix that created an incident.