xlstm

What it is

xLSTM is a new Recurrent Neural Network architecture based on ideas of the original LSTM. Through Exponential Gating with appropriate normalization and stabilization techniques and a new Matrix Memory it overcomes the limitations of the original LSTM and shows promising performance on Language Modeling when compared to Transformers or State Space Models.

How it works

The repository includes implementations for xLSTM blocks, sLSTM, and mLSTM variants, with a dedicated xLSTM Large 7B architecture. It provides a standalone single file implementation of the xLSTM Large architecture in xlstm/xlstm_large/model.py and requires the mlstm_kernels package for the optimized kernels. It also exposes xLSTMBlockStack and xLSTMLMModel configurations and demonstrates usage patterns.

Getting started

Minimal Installation steps are outlined:

Create a conda environment from the file `environment_pt240cu124.yaml`.
Install the model code only (i.e. the module `xlstm`) as package:

For using the xLSTM Large 7B model install:

pip install mlstm_kernels

Then install the xlstm package via pip:

pip install xlstm

Or clone from github:

git clone https://github.com/NX-AI/xlstm.git
cd xlstm
pip install -e .

Requirements mention PyTorch versions >=1.8 and provide a conda environment file:

conda env create -n xlstm -f environment_pt240cu124.yaml
conda activate xlstm

For xLSTM Large 7B, mlstm_kernels is required.

Recent releases

Latest release: v2.0.4 (2025-05-28): Fix vanilla sLSTM cell stabilization.

Traction

Stars: 2190, Forks: 185, Open issues: 61

Behind the repo

Not specified in the README excerpt.

Caveats

License: Apache-2.0. The README references environment and kernel dependencies (mlstm_kernels) and Torch compatibility (>=1.8). The 7B model weights are available on HuggingFace as indicated within the document.