Where models fit in an architecture

The model is a dependency, not the architecture

Section 12.1 described a model as a probabilistic function reached through an API. In an architecture it is one container among many, and usually a small part of the code. A customer-support assistant has a chat front end, a conversation store, integrations with order and refund services, a search index, identity, logging, a handoff to human agents and a deployment pipeline; the model call is one line in the middle. The architectural questions are therefore familiar ones asked of a new kind of dependency: where does it run, how is it called, what sits between it and the callers, what happens when it is slow, wrong or unavailable, and who is accountable for its output?

Getting these questions right matters more than choosing the best model of the month. Models are replaced every few months; the gateway, the fallback design, the evaluation set and the human review process last for years. The decisions in this section are the significant, hard-to-reverse ones in the sense of Chapter 1, so they deserve decision records; the choice of a particular model version should be easy to change.

Three places a model can run

Hosted model APIs. The provider runs the model and you pay per token. This covers the frontier providers' own APIs (OpenAI, Anthropic and Google among others) and cloud platforms that resell several vendors' models inside your existing cloud account. It is the fastest way to start and gives access to the most capable models, with no accelerators to buy or operate. The costs are per-token pricing that grows with use, rate limits set by someone else, a dependency on the provider's availability, models that are deprecated on the provider's timetable, and data that leaves your boundary. Contracts, data processing terms, zero- or limited-retention options and regional processing reduce the last concern but do not remove it.

Self-hosted open-weight models. An open-weight model publishes its trained weights, so you can run it on your own or rented accelerators with an inference server such as vLLM. Examples include models from Meta (Llama), Mistral, Alibaba (Qwen), DeepSeek, Google (Gemma) and OpenAI (gpt-oss). You control the data path, the exact version and fine-tuning, and at high, steady volume the cost per token can fall well below API prices. In exchange you take on capacity planning for scarce, expensive hardware, utilisation (idle accelerators cost the same as busy ones), upgrades, security patching and on-call. Licences matter: some open-weight models use permissive licences such as Apache 2.0, while others use custom licences with conditions on use or scale, and most do not publish their training data. The Open Source Initiative's Open Source AI Definition (version 1.0, October 2024) sets a higher bar than open weights alone. A middle path, used by this course's own chatbot, is to call open-weight models through a third-party inference provider: per-token pricing, no hardware, and the freedom to move the same model elsewhere.

On-device models. Small models, often a few billion parameters or fewer and quantised to lower precision, run on phones, laptops or in the browser. They give privacy (data never leaves the device), offline operation, no per-call cost and very low latency, at the price of much lower capability, a wide spread of device performance, larger app downloads and the difficulty of updating models already shipped to millions of devices. Many products combine them: a small on-device model handles quick, private tasks and escalates harder ones to the cloud.

Online inference and batch scoring

The second question is whether a user is waiting. Online inference calls the model in the request path, so latency targets apply: users notice the time to the first visible words, which is why chat answers are streamed, and they abandon a request that takes many seconds. Online calls need timeouts, concurrency limits and capacity for the evening peak, exactly as any synchronous dependency in Chapter 6.

Batch scoring processes many items without anyone waiting: tagging two million menu items with allergens overnight, summarising yesterday's support conversations, re-embedding a document collection. Batch work can use the cheapest adequate model, run when capacity is free and retry calmly. Several providers offer asynchronous batch interfaces that accept a file of requests and return results within hours at a substantial discount. Between the two sits asynchronous inference: a request is queued, the user is told the result will follow (a refund decision by e-mail, a document summary in a few minutes), and workers drain the queue at a controlled rate, using the queues and back pressure of Chapter 9 to absorb peaks and provider rate limits. Moving work out of the online path is one of the most effective cost and reliability decisions available.

An AI gateway between applications and models

Once more than one team or feature calls models, a dedicated AI gateway pays for itself. It plays the role an API gateway plays for microservices, specialised for model traffic. Applications call one internal endpoint with one request format and their own identity; the gateway holds the provider keys, translates to each provider's API, applies routing and fallbacks, enforces per-team quotas and spending budgets, applies rate limits, records every call (model, version, token counts, latency, cost, outcome) for tracing and cost allocation, can cache responses, and offers a place to attach input and output guardrails centrally.

The benefits are architectural: provider choice becomes a configuration decision instead of a code change in every service, keys are no longer scattered across applications, spending is visible per feature, and a provider outage is handled once. The costs are those of any shared intermediary. It is a new critical dependency that must be highly available, it adds a small amount of latency, and its logs become a concentrated store of prompts that may contain personal data, so redaction and retention rules (see privacy by design) apply to it. Open-source and managed gateway products exist, and cloud platforms include gateway features; building a thin one is reasonable for a small system, but quotas, streaming, retries and accounting are more work than they look.

Routing and cascades

Not every request needs the most capable model. Routing decides up front which model should handle a request, using rules (the feature, the customer tier, the language) or a small, fast classifier. In the support assistant, where is my order? can go to a small, cheap model that calls one tool, while a complaint about three separate problems goes to a larger model. A model cascade instead tries a cheaper model first and escalates only when a check says its answer is not good enough. Chen, Zaharia and Zou's FrugalGPT (2023) showed that well-tuned cascades can approach the quality of the strongest model at a fraction of the cost.

A cascade is only as good as its escalation check. Reliable checks are deterministic: the output fails schema validation, a cited source does not exist, a required field is missing, a business rule rejects the proposal. A verifier model scoring the answer is weaker but often useful. A model's own statement of confidence is the least reliable signal and should not be used alone. Cascades also have costs: escalated requests pay for two calls and wait for both, there are now two models to evaluate and keep in step, and the quality ceiling is set by the check, not by the strong model. Measure the escalation rate; if most requests escalate, route instead.

Fallback ladders across models and providers

Model APIs fail in all the ways remote dependencies fail, plus a few of their own: rate-limit responses (HTTP 429) when a quota is exhausted, overload errors at busy times, slow responses that exceed a timeout, regional outages, and models withdrawn on a deprecation date. The patterns of Chapter 9 apply directly, arranged as a fallback ladder: an ordered list of rungs, each tried when the one above fails. A typical ladder is the primary model; the same model in another region or from another host; a different model of similar capability from a second provider; a smaller, faster model; and finally a non-AI response, such as a helpful canned message, the help-centre search results or a handoff to a person.

Design rules for a ladder: give each rung its own timeout and the whole ladder a total deadline, so that the user never waits for four slow failures in a row; respect Retry-After and put a rate-limited model into a cooldown (a simple circuit breaker) instead of hammering it; do not retry errors that will fail again, such as a request that is too large or rejected by a content filter; validate output on every rung, because a lower rung may produce answers in a slightly different shape; evaluate each rung against the same evaluation set, since prompts often need per-model adjustment; and record which rung answered, so that quality dashboards can tell a degraded answer from a normal one.

This course's own AI features are a small, real example. ArchitectBot's answers and the marking of written test answers are generated through a ladder of several open-weight models served by one inference provider, tried in a fixed order of preference. A model that returns a rate-limit response is cooled down for the period the provider asks (within set bounds), a request that fails, times out or returns an unusable answer moves to the next rung, those rungs share an overall time budget, and a second provider serves as the last rung. The design trades some answer quality on the lower rungs for availability: a slightly weaker answer is preferred to none.

Humans in the loop

Human in the loop describes several different designs, and it helps to name which one is meant. Approval before effect: the model drafts, a person approves (a refund above a limit, a reply to a regulator, a change to production). Review after effect: outputs take effect immediately and people review a sample or the flagged cases. Escalation: the model handles what it can and hands the rest to a person, with the conversation and the evidence. Feedback: people label or correct outputs to build evaluation sets and improve prompts.

Each design has architectural consequences. Approval creates a queue that must be sized for peak volume, like the fraud review queue in section 12.1, and adds latency measured in minutes or hours. People who approve hundreds of good suggestions start approving without reading (automation bias), so the interface should show the evidence and the exact action, highlight what is unusual, and track how often reviewers change the proposal; an override rate near zero is a warning, not a success. Handoffs need the full context so the customer does not repeat themselves. The human oversight required by regulation for some uses is more demanding still: the person must understand the system's limits and be able to override it.

Build or buy, and how to choose a model

The build-or-buy question has more than two answers. You can buy a complete AI feature inside a product you already use (a helpdesk vendor's assistant); build on hosted APIs; fine-tune a model for your task; self-host open weights; or, rarely, train your own. Use the strategic lens of domain-driven design: buy for generic capabilities such as meeting transcription, and build where AI touches your core domain and your own data gives you an advantage. Whatever you build, keep provider-specific code behind a port with an adapter per provider, and avoid depending on features only one provider offers unless their value justifies the lock-in.

Choose models with your own evaluation set, not with public leaderboards. Leaderboards measure general tasks that may not resemble yours, and some have been tuned against. Define the criteria and their weights before testing, as with any weighted decision, then shortlist two or three candidates and measure them on a few hundred real examples. Expect the answer to be a combination, such as a small model for routing and a larger one for hard cases, rather than one winner.