Mamba: Linear-Time Sequence Modeling with Selective State Spaces

Introduced selective state space models (SSMs) with hardware-aware parallel scanning, achieving linear-time O(N) context scaling with Transformer-matching quality.

Foundational non-transformer sequence architecture enabling 1M+ token context lengths, deployed in Jamba, Falcon-Mamba, and long-sequence genomic models.

Assumptions

• Making SSM state transition matrices input-dependent allows selective memory gating while retaining hardware-efficient parallel scans

Limitations

• Pure SSMs have lower memory capacity for exact associative recall than full quadratic Transformer attention; hybrid architectures often required