LLM Architecture
Explained

What Is LLM Architecture?

LLM architecture describes a model's internal design: how it represents tokens, mixes information, and computes predictions across its layers. Most widely used language models are transformer-based, while alternatives such as selective state-space models use different sequence-mixing mechanisms.

Architecture affects training and inference behavior, including compute, memory use, and the ways a model can use context. These trade-offs depend on the model, implementation, hardware, and workload; an architecture label alone does not predict quality, speed, or serving cost.

This collection focuses on model architecture and inference. Application architecture is a broader layer: retrieval, tools, permissions, evaluation, and deployment surround the model but do not change its internal architecture. Start with the guides below, then follow their cited papers for the original methods and assumptions.

The Transformer: The Foundation of Modern LLMs

The Transformer was introduced in the 2017 paper "Attention Is All You Need." Its attention-based design allowed sequence elements to be processed in parallel during training, unlike recurrent sequence models. Many current language models use transformer variants, but this is not the only family of sequence models.

Transformers come in three families, and the differences matter when you choose a model:

• Decoder-only models (GPT-style) generate text left to right and dominate chat, code generation, and agentic workloads.
• Encoder-only models (BERT-style) build bidirectional representations and excel at classification, search, and embeddings.
• Encoder-decoder models(T5-style) map an input sequence to an output sequence, which still suits translation and structured rewriting.

If you are weighing them for a system design, the architecture comparison guide puts the trade-offs side by side.

How Attention Mechanisms Work

Self-attention lets every token look at every other token and decide how relevant each one is. Each token is projected into three vectors -- a query, a key, and a value -- and the attention score is the scaled dot product between one token's query and every token's key, normalized by a softmax and used to weight the values.

Multi-head attention runs several learned attention projections. Their behavior is learned from training; a head should not be assumed to map cleanly to a human-readable role such as syntax or position.

In standard full self-attention, token-to-token interactions grow quadratically with sequence length. Sparse and other efficient attention methods change that computation, with trade-offs that depend on the task and implementation. Sparse attention mechanisms include local-window and other restricted patterns; a longer supported context does not by itself mean the model can use every token equally well.

Mixture of Experts (MoE)

A dense transformer activates every parameter for every token. A mixture-of-experts model replaces the feed-forward layer with a router and a set of expert networks, activating only a few experts per token.

An MoE model can have a large total parameter count while activating only selected experts for a given token. Sparse activation changes compute patterns, but does not guarantee a quality or cost advantage: routing, load balancing, memory, and communication also matter. Read the linked papers and model documentation for the specific design being evaluated.

The trade-offs are operational rather than conceptual: MoE models need load balancing during training, careful expert placement to fit across GPUs, and more total memory because every expert stays resident even when rarely used. The guide walks through routing, top-k selection, and the failure modes to watch for.

KV-Cache Optimization and Inference Performance

During autoregressive generation, a KV cache stores keys and values from prior tokens so the model can reuse them for the next token instead of recomputing the full prefix. The cache uses memory and its size depends on factors such as model shape, context length, and concurrent sequences.

The cache can constrain serving capacity for long contexts or many concurrent sequences; its impact should be measured for the target model and workload. KV-cache optimization -- quantizing the cache, paging it like virtual memory (the idea behind vLLM), sharing it across queries with grouped-query attention, and eviction, each with different quality, latency, and memory trade-offs.

If you are sizing infrastructure, pair it with the inference serving systems guide, which covers batching, scheduling, and throughput engineering.

Beyond Transformers: SSMs, Multimodal Models, and Reasoning

Other sequence and system designs address different constraints. State-space models such as Mamba use selective state-space mechanisms; the original Mamba paper describes linear scaling with sequence length, but practical speed and quality depend on the model, hardware, and task. Multimodal foundation models extend the architecture to accept images, audio, and video alongside text. And reasoning models can spend additional inference-time computation on a task. That may increase latency and token use; accuracy effects are task- and method-dependent.

For agentic systems, the tool-augmented LLM system design shows how a model is wired into tools, memory, and orchestration: the architecture of the application, not just the model.