Architecture
Patterns

What Is LLM Architecture?

An LLM architecture is the blueprint that defines how a language model processes text: how layers are stacked, how information flows between them, and how the model scales during training and inference. Every production AI system you interact with today -- chat assistants, coding agents, RAG pipelines, autonomous workflows -- runs on a small family of designs that share one core mechanism: attention.

Understanding these architectures is not academic. The architecture behind a model determines how fast it generates tokens, how much memory it needs per user, how far its context window can stretch, and what it costs to serve. When you pick a model for a production system, you are picking an architecture and all of its trade-offs with it.

This collection covers the decisions that matter in production, from the original transformer design to modern variants like mixture-of-experts and state-space models. Each guide includes the mechanics, the trade-offs, and a build prompt you can run in Claude Code, OpenCode, or Codex.

The Transformer: The Foundation of Modern LLMs

The transformer, introduced in the 2017 paper "Attention Is All You Need," replaced the recurrence of RNNs with self-attention, letting a model process every token in a sequence in parallel instead of one at a time. That single change made training on internet-scale datasets practical, and every frontier model today is a descendant of it.

Transformers come in three families, and the differences matter when you choose a model:

• Decoder-only models (GPT-style) generate text left to right and dominate chat, code generation, and agentic workloads.
• Encoder-only models (BERT-style) build bidirectional representations and excel at classification, search, and embeddings.
• Encoder-decoder models(T5-style) map an input sequence to an output sequence, which still suits translation and structured rewriting.

If you are weighing them for a system design, the architecture comparison guide puts the trade-offs side by side.

How Attention Mechanisms Work

Self-attention lets every token look at every other token and decide how relevant each one is. Each token is projected into three vectors -- a query, a key, and a value -- and the attention score is the scaled dot product between one token's query and every token's key, normalized by a softmax and used to weight the values.

In practice, models run several of these operations at once (multi-head attention), so different heads can specialize: one tracks syntax, another long-range references, another position. That is where a transformer's contextual understanding comes from.

The cost is quadratic: attention between every pair of tokens grows with the square of sequence length, which is why long-context serving gets expensive fast. Sparse attention mechanisms -- sliding windows, dilated patterns, linear approximations -- cut that cost and are how modern models stretch to contexts of hundreds of thousands of tokens.

Mixture of Experts (MoE)

A dense transformer activates every parameter for every token. A mixture-of-experts model replaces the feed-forward layer with a router and a set of expert networks, activating only a few experts per token.

The result is decoupled capacity and compute: a model can carry hundreds of billions of parameters while each token only flows through a handful of them. That is how frontier-class models deliver large-model quality at a fraction of the FLOPs per token. Mixtral and DeepSeek made the approach mainstream, and it is now a default at scale.

The trade-offs are operational rather than conceptual: MoE models need load balancing during training, careful expert placement to fit across GPUs, and more total memory because every expert stays resident even when rarely used. The guide walks through routing, top-k selection, and the failure modes to watch for.

KV-Cache Optimization and Inference Performance

During generation, a transformer would recompute attention over the whole prefix for every new token -- unless it caches. The KV cache stores each layer's keys and values so generation becomes incremental: one token in, one token out.

The catch is that the cache grows with context length and concurrent users, and at scale it becomes the dominant memory consumer in the serving stack. KV-cache optimization -- quantizing the cache, paging it like virtual memory (the idea behind vLLM), sharing it across queries with grouped-query attention, and evicting stale windows -- is what turns a model that serves one user into one that serves thousands.

If you are sizing infrastructure, pair it with the inference serving systems guide, which covers batching, scheduling, and throughput engineering.

Beyond Transformers: SSMs, Multimodal Models, and Reasoning

Three newer directions round out the collection. State-space models such as Mamba replace attention with a recurrent formulation that scales linearly with sequence length, promising cheap long context. Multimodal foundation models extend the architecture to accept images, audio, and video alongside text. And reasoning models apply test-time compute, generating long chains of thought before answering -- trading latency for accuracy on hard problems.

For agentic systems, the tool-augmented LLM system design shows how a model is wired into tools, memory, and orchestration: the architecture of the application, not just the model.