Back to Patterns

State Space Models

Architecture Patterns

Summary

State Space Models (SSMs), popularized by the Mamba architecture, offer a compelling alternative to Transformers for sequence modeling. SSMs replace the attention mechanism with a learned state-space dynamics model that compresses the entire sequence into a hidden state, achieving linear-time inference and theoretically unlimited context length.

Key Characteristics

  • Linear-Time Inference: Processes sequences in O(n) time rather than O(n^2), enabling very long context windows
  • Recurrent Formulation: Operates as a recurrent neural network at inference time for constant-memory generation
  • Selection Mechanism: Modern SSMs (Mamba) learn to selectively propagate or ignore information based on input content
  • Hardware-Aware Design: Optimized for GPU memory hierarchy with parallel scan algorithms

Popular Models

  • Mamba: The pioneering selective SSM that matches Transformer quality with linear-time inference
  • Mamba-2: Simplified architecture using state-space duality theory for improved throughput
  • Jamba: Hybrid architecture combining Mamba layers with Transformer attention layers
  • S4 (Structured State Space Sequence Model): Foundational SSM introducing structured state-space parameterization

Build This Pattern

Copy this prompt and paste it into Claude Code, OpenCode, Codex, or Cursor to implement this pattern.

Explain State Space Models (SSMs) like Mamba. Cover: how SSMs use a recurrent state instead of attention to process sequences in linear time, the structured state-space formulation, how Mamba makes the state time-varying for content-aware processing, comparison to transformers on long-context efficiency (SSMs scale better to 100k+ tokens), and current adoption status (Hybrid SSM-attention models show most promise). Applications: long-document processing, DNA sequence analysis, real-time audio processing, efficient streaming.