Cost, latency and LLMOps

Token economics

Hosted models are priced per token, usually quoted per million tokens, with separate prices for input tokens (everything sent: system prompt, tool definitions, retrieved documents, conversation history and the user's message) and output tokens (everything generated). Output tokens typically cost several times more than input tokens, because input is processed in parallel in one pass (the prefill), whereas output is generated one token at a time, each needing its own pass through the model (the decode). Reasoning models (section 12.1) also bill their internal reasoning tokens as output, even when those tokens are not shown, so the same question can cost very different amounts depending on how long the model thinks.

The cost of one call is simply input tokens × input price + output tokens × output price. The architectural surprises lie in what drives the token counts. In a chat, the whole history is normally sent again with every turn, so input grows with each turn and the total over a conversation grows roughly with the square of its length. Tool definitions and a long system prompt are paid for on every call. Retrieval adds thousands of tokens per question, and agents that loop through tools (section 12.4) make several calls per user request. In most production assistants input tokens dominate the bill even though each one is cheaper.

Self-hosting an open-weight model changes the shape of the cost rather than removing it: you pay for GPU time whether or not it is used, so cost per token depends on utilisation and on how many requests the server can process together. The FinOps habits of section 10.8 apply directly: express cost per unit of business value (per conversation, per resolved ticket, per document processed), compare it with the alternative (a human agent's handling cost), and allocate it to features and teams. The prices in this section are illustrative figures chosen for easy arithmetic, not current list prices; real prices differ between providers and have fallen steeply over time.

Latency: time to first token and generation speed

The latency of a model response has two main parts. Time to first token (TTFT) covers everything before the first output token arrives: your own pre-processing (input guards, retrieval, re-ranking), the network, queueing at the provider, and the prefill, which grows with the number of input tokens. After that, output arrives at the model's generation speed, measured in output tokens per second, so total time is roughly TTFT + output tokens ÷ tokens per second. Reasoning models add a thinking phase before the first visible token, which can take seconds or minutes.

Two consequences follow. First, output length is usually the largest lever on latency: halving a 400-token answer saves more time than any network optimisation. Asking for concise answers, setting a maximum output length and using structured outputs instead of prose all help. Second, every tool call is a full extra round trip with its own prefill and decode, so an agent that makes four tool calls before answering is four times as slow as a single call, whatever the model's speed. Specify and monitor latency as percentiles, not averages (section 2.4): p95 TTFT and p95 total time per turn, because model providers' queueing makes the tail much longer than the median at busy times.

Streaming

Streaming sends tokens to the client as they are generated, typically as server-sent events over HTTP. It does not make the answer finish sooner, but users judge responsiveness largely by when something starts to happen, so a 4-second answer that starts after 1 second feels much faster than one that appears all at once. Streaming has architectural costs. Every proxy, gateway and load balancer on the path must pass the stream through without buffering and keep long-lived connections open. Output guardrails (section 12.6) can no longer inspect the whole answer before the user sees it, so they either check chunks as they pass (and may have to retract text already shown) or hold back the stream by a sentence or so. Structured outputs that drive actions cannot be acted on until they are complete and validated, so stream the prose to the user and keep the actions for the end. Billing, logging and tracing must cope with streams that the client abandons halfway.

Prompt caching and semantic caching

Prompt caching (also called prefix caching or context caching) reuses the provider's computed internal state for an identical prompt prefix that was processed recently. Cached input tokens are billed at a steep discount (some providers charge around a tenth of the normal input price, others less of a discount, and some charge extra to write the cache) and the prefill for that part is skipped, which also lowers TTFT. It is the single most effective cost optimisation for assistants with long, stable instructions. It only works if the prefix is byte-for-byte identical, so prompt layout becomes an architectural concern: put static content first (system prompt, tool definitions, policies, few-shot examples), then slowly changing content (conversation history), and per-request content last (retrieved documents, the new message). A timestamp or user name at the top of the system prompt silently defeats the cache. Self-hosted inference servers such as vLLM offer the same technique, known as automatic prefix caching.

Semantic caching is different: it stores whole responses keyed by the meaning of the question, using embeddings, and returns a stored answer when a new question is similar enough, skipping the model entirely. It can save a great deal for repetitive, non-personal questions (what are your delivery hours?), but it is riskier than it looks. Similar questions can need different answers (can I cancel my order? and cancel my order), the similarity threshold trades hit rate against wrong answers, and the rules of section 7.6 still apply: the key must include everything that changes the answer, cached answers go stale when policies change, and anything personal must never be served from a shared cache. Measure the false-hit rate on a labelled sample before relying on it. A plain exact-match cache for deterministic calls, such as classification at temperature 0, is simpler and safe.

Batching

Not every model call needs an answer in seconds. Nightly summarisation, tagging a backlog of tickets, generating embeddings for a re-index and running evaluation suites can all wait. Most large providers offer batch interfaces that accept a file of requests and return results asynchronously within a stated window (commonly up to 24 hours) at a substantial discount, often around half the normal price, and with separate, larger rate limits. Architecturally this is the queue-based, asynchronous processing of section 9.3: design the job to be idempotent, track each request's status and handle partial failures.

On self-hosted models, batching happens inside the inference server: continuous batching interleaves the decode steps of many concurrent requests on one GPU, multiplying throughput. The trade-off is the familiar one between throughput and latency: larger batches mean cheaper tokens but slower individual responses, so interactive and offline traffic are often served by separate pools with different settings.

Model selection and cascades for cost

The biggest cost decision is which model handles each task, and the answer is usually the smallest one that passes the evaluation suite for that task (section 12.5). Prices between small and large models commonly differ by a factor of ten or more, while many tasks (classifying intent, extracting an order number, answering a status question from structured data) do not need a large model. Routing sends each request to a suitable model based on a cheap classification; a model cascade tries a small model first and escalates to a larger one when the output fails validation or the small model signals low confidence (section 12.2). FrugalGPT (Chen, Zaharia and Zou, 2023) showed cascades matching the quality of the strongest model at a fraction of the cost on their benchmarks.

The expected cost of a two-step cascade is the small model's cost plus the escalation rate times the large model's cost. Cascades therefore pay only when most requests stay on the small model, and they have a latency catch: escalated requests pay for both attempts, so the tail gets slower even as the average improves. Up-front routing avoids the double attempt but depends on the router's accuracy. Other token reductions are often cheaper still: trim the system prompt, retrieve fewer but better chunks, summarise older conversation turns, cap output length, lower the reasoning effort of reasoning models for easy tasks, and for a single very high-volume task, consider fine-tuning or distilling a small model.

Quotas, rate limits and fallback ladders

Model providers enforce rate limits: requests per minute, input and output tokens per minute, concurrent requests and sometimes daily caps, signalled by HTTP 429 responses, usually with a Retry-After header. Your own gateway should enforce quotas too, per user, per tenant and per feature, both to share capacity fairly and to defend against the unbounded consumption risk of section 12.6. The resilience patterns of section 9.4 carry over: timeouts on every call, retries with exponential back-off and jitter only for retryable errors, circuit breakers, bulkheads that stop one noisy feature exhausting the shared quota, and graceful degradation (a canned answer and a link to human support beats a spinner).

A fallback ladder lists alternative models and providers to try in order when the first choice fails, is rate-limited or returns an invalid result. Good ladders have a total time budget so that the user is not kept waiting through every rung, cool a rung down for as long as the provider asks after a 429 instead of hammering it, treat an unparseable or schema-invalid reply as a failure that moves to the next rung, and keep prompts portable between models. Every rung must pass the evaluation suite: an untested fallback model is a silent quality drop exactly when the system is under stress. This course's own AI features work this way. ArchitectBot and the test marker try a ladder of hosted open-weight models with a timeout per model and an overall time budget, pause a model after a rate-limit response for the time the provider asks, and fall back to a second provider as a last resort; web searches have daily and monthly caps, and visitors are rate-limited.

LLMOps: versioning, tracing and cost monitoring

LLMOps is the set of practices for building, releasing and operating applications built on large language models, adapting MLOps and DevOps to components whose behaviour is defined by prompts, models and data as much as by code. Its first rule is that everything that shapes behaviour is a versioned artefact: prompt templates (kept in the repository or in a prompt registry with immutable version identifiers), model identifiers pinned to dated versions rather than floating aliases, tool schemas, retrieval settings, the embedding model and index version (changing the embedding model means re-embedding the whole corpus), guardrail rules and the evaluation datasets. A release is a specific combination of these, deployed through the same pipeline, feature flags and canaries as code (section 10.5), with its evaluation results attached.

The second rule is that every model call is traced. Each call becomes a span in the distributed trace of the user's request (section 9.6) carrying the model, the prompt version, input and output token counts, latency, TTFT, cost and outcome, with child spans for retrieval and tool calls, so that a slow or wrong answer can be followed step by step. OpenTelemetry's semantic conventions for generative AI standardise the names, such as gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model and gen_ai.usage.input_tokens, together with metrics for operation duration and token usage. As of October 2026 these conventions still have Development status and have recently moved to their own repository; names have changed between versions (gen_ai.system became gen_ai.provider.name, for example, and the token metrics are being reorganised), so pin the version your instrumentation emits. Recording the text of prompts and completions is opt-in in the conventions, for good reason: it captures personal data, so it needs the retention and access controls of section 12.8.

The third rule is that cost is monitored like latency. Attribute token spend to feature, tenant, model and prompt version; set budgets with alerts on both absolute spend and unit cost (cost per conversation); and watch for the signatures of trouble: a prompt change that doubled input tokens, an agent stuck in a tool loop, a cache hit rate that collapsed after a deployment, or one account generating a tenth of all traffic. Finally, evaluation runs in the pipeline (section 12.5), so that cost, latency and quality are judged together for every release.

from opentelemetry import trace

tracer = trace.get_tracer("support-assistant")

def ask_model(client, messages, model="medium-2026-05", prompt_version="refund-v14"):
    with tracer.start_as_current_span(f"chat {model}") as span:
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.provider.name", "example-provider")
        span.set_attribute("gen_ai.request.model", model)
        span.set_attribute("app.prompt.version", prompt_version)  # our own attribute
        reply = client.chat(model=model, messages=messages, max_tokens=300)
        span.set_attribute("gen_ai.usage.input_tokens", reply.usage.input_tokens)
        span.set_attribute("gen_ai.usage.output_tokens", reply.usage.output_tokens)
        return reply