Machine learning and large language models for architects

Why architects need to understand the model

Every component in the earlier chapters of this course was deterministic: given the same input and the same state, a service returned the same output, and a test could assert it. A component built on machine learning is different in kind. It was not written as rules; it was fitted to examples, and what it returns is a statistical best guess. A fraud model returns a score, not a verdict. A large language model (LLM) returns fluent text that is usually right and occasionally, confidently, wrong.

An architect does not need to train models, any more than they need to write a database engine. They do need to know a model's properties well enough to design around them, in the same way that Chapter 7 required knowing what an isolation level actually guarantees. Those properties are: how models are made and what that implies about their knowledge; how they fail; how their cost and latency arise; and why their output can never be trusted in the way a database row can. Everything else in this chapter, from retrieval to guardrails, follows from them.

Machine learning in one page

Machine learning builds a function from data instead of from hand-written rules. There are three classic families. In supervised learning the training data contains the right answers (labels): past payments marked fraudulent or genuine, past deliveries with their actual durations. The model learns to map inputs to labels. Predicting a category is classification (fraud or not); predicting a number is regression (minutes until the food arrives). In unsupervised learning there are no labels, and the model finds structure on its own: clusters of similar customers, unusual transactions that fit no pattern, compact numerical representations of text. In reinforcement learning an agent takes actions in an environment and learns from a reward signal rather than from correct answers, for example a dispatch policy rewarded for short delivery times.

Large language models mostly come from a fourth variant, self-supervised learning, in which the labels are taken from the data itself: hide the next word of a sentence and train the model to predict it. Because any text supplies its own labels, models can be trained on vastly more data than anyone could label by hand.

Two phases matter architecturally. Training adjusts the model's internal numbers (its parameters or weights) to reduce error on the training data. It is a heavy batch workload: large data sets, many accelerators (GPUs or similar chips), hours to months of computation, and a pipeline that must be repeatable. Inference is using the trained model to produce outputs for new inputs. It is a serving workload with latency targets, request rates and per-call cost, much like any other service in Chapter 9. Most teams building AI features never train a large model at all; they consume inference from a provider or run someone else's trained weights.

The most useful mental model is that a trained model is a probabilistic function. For an input it produces scores or a probability distribution over possible outputs. Somebody, usually the application, then turns that distribution into a decision: block the payment if the fraud score exceeds 0.8, or pick the next word by sampling. That last step is a design decision with business consequences, and it belongs to the architecture, not to the model.

Evaluation basics: precision, recall and overfitting

Because a model is fitted rather than specified, the only way to know how good it is is to measure it on examples it has not seen. Data is split into a training set (to fit the model), a validation set (to tune choices such as thresholds) and a test set kept back for the final, honest measurement. A model that scores well on its training data but badly on new data is overfitting: it has memorised quirks of the examples instead of learning the general pattern. Data leakage, where information about the answer slips into the inputs (a refund flag that is only set after fraud is confirmed, say), produces models that look excellent in testing and fail in production.

For classification, accuracy alone misleads whenever one class is rare. Count four outcomes: true positives (fraud flagged), false positives (genuine payment flagged), false negatives (fraud missed) and true negatives. Precision is the share of flagged items that really are positive, TP / (TP + FP). Recall is the share of real positives that were caught, TP / (TP + FN). Raising the decision threshold usually raises precision and lowers recall, and the right balance depends on what each kind of mistake costs. The F1 score, the harmonic mean of the two, summarises them in one number when the costs are similar.

Models also degrade after deployment. When the inputs the model sees drift away from the training data (new restaurants, a new payment method, a pandemic changing ordering habits), its accuracy falls silently. Monitoring for this is part of evaluating AI systems, covered later in this chapter.

Deep learning and the transformer

A neural network is a stack of layers, each computing weighted sums of its inputs followed by a simple non-linear function. Training nudges millions or billions of weights, using gradient descent and backpropagation, so that the network's outputs move closer to the targets. Deep learning means networks with many layers, which learn their own internal representations of the input instead of relying on features engineered by hand. It transformed image recognition from about 2012 and speech and translation soon after.

Language was harder because meaning depends on context: in the courier left the bag because it was torn, it refers to the bag. Earlier recurrent networks processed text one word at a time, which made training slow and long-range context fragile. In 2017 Ashish Vaswani and colleagues at Google published Attention Is All You Need, introducing the transformer. Its central mechanism, self-attention, lets every position in the text compute a weighted mix of information from every other position, so it can attend strongly to bag. Because all positions are processed in parallel during training, transformers use modern accelerators efficiently, and their quality kept improving as models, data and computation were scaled up.

Two consequences reach the architect. First, the attention computation grows roughly with the square of the sequence length, which is one reason context length has limits and long prompts cost more time and money (engineering advances have softened, but not removed, this cost). Second, there are different transformer shapes for different jobs: decoder-only models generate text and underlie today's chat models, while encoder models in the line of BERT (2018) read text to classify or represent it, and power many embedding and re-ranking models used in retrieval. Many large models also use a mixture-of-experts design, in which only part of the network is active for each token, reducing inference cost for a given total size.

How a large language model produces text

A model does not read characters or words; it reads tokens, pieces of text from a fixed vocabulary of tens or hundreds of thousands of entries, produced by a tokeniser. Common words are one token; rare words, names and numbers split into several. For English text a common rule of thumb is about four characters, or three-quarters of a word, per token. Other languages and source code often need more tokens for the same content, which matters because prices, limits and latency are all counted in tokens.

The context window is the maximum number of tokens the model can take into account in one call, counting the instructions, the conversation so far, any documents supplied and the answer it writes. In 2026 windows range from a few thousand tokens on small on-device models to around a million or more on some hosted models. A large window is not the same as good use of it: Liu and colleagues (Lost in the Middle, 2023) showed that models use information at the start and end of a long context more reliably than information buried in the middle, and every extra token adds cost and latency.

Generation is next-token prediction in a loop. The model reads the whole input (the prefill phase) and outputs a probability for every token in its vocabulary as the next one. One token is chosen, appended to the input, and the process repeats (the decode phase) until the model emits an end marker or reaches a maximum length. The model caches internal results for tokens it has already processed (the KV cache), so each step only computes the new token, but output is still produced one token at a time. That is why output tokens are slower and usually several times more expensive than input tokens, and why answers are streamed to users as they are produced.

How the next token is chosen is called sampling. Greedy decoding always takes the most probable token. Temperature rescales the probabilities first: below 1 it sharpens them towards the favourite, above 1 it flattens them so less likely tokens appear more often. Top-p (nucleus) sampling keeps only the most probable tokens whose probabilities add up to p, and top-k keeps only the k most probable. Low temperature suits extraction, classification and marking; higher temperature suits brainstorming and varied wording.

Finally, the model is stateless. A model API remembers nothing between calls: a chat feels continuous only because the application sends the whole relevant history with every request. Conversation state therefore lives in your system, its size drives cost on every turn, and deciding what to keep, summarise or drop is a design task (see memory and state in section 12.4).

import math

# raw scores the model gives three candidate next tokens after
# "The capital of Portugal is"
logits = {"Lisbon": 9.0, "Porto": 6.5, "Madrid": 5.0}

def probabilities(logits, temperature):
    scaled = {t: v / temperature for t, v in logits.items()}
    total = sum(math.exp(v) for v in scaled.values())
    return {t: round(math.exp(v) / total, 4) for t, v in scaled.items()}

print(probabilities(logits, 1.0))  # Lisbon 0.9088, Porto 0.0746, Madrid 0.0166
print(probabilities(logits, 0.5))  # Lisbon 0.9930, Porto 0.0067, Madrid 0.0003
print(probabilities(logits, 2.0))  # Lisbon 0.7033, Porto 0.2015, Madrid 0.0952

Embeddings: meaning as numbers

An embedding is a list of numbers (a vector, typically a few hundred to a few thousand numbers long) that represents the meaning of a piece of text, an image or a product, produced by an embedding model. The model is trained so that items with similar meaning land close together: my food arrived cold and the order was not hot when delivered produce nearby vectors although they share almost no words. Closeness is usually measured by cosine similarity, the angle between two vectors.

Embeddings power semantic search, recommendations, de-duplication, clustering and classification, and they are the heart of most retrieval-augmented generation systems. Two architectural facts follow. Embedding models are separate, smaller and much cheaper than generation models, and can easily run on your own infrastructure. And vectors from different embedding models (or different versions of one) are not comparable, so changing the embedding model means re-embedding the whole corpus: a decision that is cheap to make early and expensive to reverse later.

From pretraining to a helpful assistant

A modern chat model is built in stages. Pretraining runs self-supervised next-token prediction over an enormous corpus of text and code, often trillions of tokens. It is by far the most expensive stage, affordable to only a few organisations, and it yields a base model: a powerful text continuer that has absorbed grammar, facts, styles and reasoning patterns, but does not reliably follow instructions. Everything the model knows from pretraining is frozen at a training cut-off date.

Fine-tuning continues training on a much smaller, targeted data set. Instruction tuning fine-tunes on examples of instructions paired with good responses, turning the base model into an assistant. Reinforcement learning from human feedback (RLHF) then asks people to compare pairs of answers, trains a reward model to predict their preferences, and optimises the assistant to score well on it. The approach was described by Christiano and colleagues in 2017 and applied to instruction following in OpenAI's InstructGPT work (Ouyang and colleagues, 2022). Variants replace parts of the pipeline: direct preference optimisation (2023) learns from the comparisons without a separate reward model, and feedback can come from another model following written principles instead of from people.

Organisations can fine-tune too, often with parameter-efficient methods such as LoRA (Hu and colleagues, 2021) that train a small add-on rather than all the weights. Fine-tuning is good at teaching a format, a tone, a narrow task or a domain vocabulary. It is a poor way to teach facts that change, such as prices or policies, because those facts are then baked into weights that cannot cite their source, cannot be updated without retraining and ignore access rights. For changing knowledge, retrieval is usually the better tool; the usual order of attempts is better prompts, then retrieval, then fine-tuning.

Hallucination and non-determinism

A hallucination is output that is fluent and confident but false or unsupported: an invented citation, a function that does not exist in a library, a refund policy the company never had. It is a direct consequence of how the model works. Training rewards plausible continuations, not verified ones, and the model has no built-in way to tell what it knows from what merely sounds right. Hallucination rates vary widely by model and task and fall when the model is given the relevant facts (grounding), allowed and encouraged to say that it does not know, or made to cite sources that the application can check. They do not fall to zero.

Outputs are also non-deterministic. With sampling, the same prompt gives different answers by design. Even at temperature 0, hosted models are not guaranteed to be repeatable: floating-point arithmetic on accelerators is not perfectly associative, and the way a server batches your request with others can change the order of operations and therefore, occasionally, a token. Providers also update models behind unchanged names. Small changes to prompt wording, the order of examples or the surrounding context can change results noticeably.

For an architect these two properties mean that a test asserting one exact output proves little, that the model version and prompt must be recorded with every result, and that any output that matters must be checked by something more reliable than the model itself.

Designing around a probabilistic component

The single most important architectural idea in this chapter is containment. A model is a powerful but unreliable component, so the deterministic code around it decides what the model is allowed to see, what it is allowed to do, and what happens to its output. In hexagonal terms, the model sits behind a port: the domain asks for a draft reply, a classification or an extraction, and the adapter that calls the model also enforces the output contract. The domain rules (refund limits, opening hours, who may see which order) stay in ordinary, testable code.

A useful test for any proposed AI feature is to ask what happens when the model is wrong in the most plausible way. If the answer is a customer sees a slightly clumsy sentence, little control is needed. If it is a refund is paid to the wrong account or a patient is told the wrong dose, the design must make that outcome impossible through validation, permissions and human confirmation, regardless of how good the model is. Models will keep improving; this design principle will not change.