MatMul-Free LM provides a Python-based implementation compatible with HuggingFace Transformers that eliminates matrix multiplication in its architecture. It includes pre-trained model zoos, installation requirements, and usage examples.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
MatMul-Free LM is a language model architecture that eliminates the need for Matrix Multiplication (MatMul) operations. This repository provides an implementation of MatMul-Free LM that is compatible with the 🤗 Transformers library.
How it works
The README describes an architecture that uses components like HGRNBitConfig, HGRNBitModel, HGRNBitBlock, and fused bit linear projections (FusedBitLinear) within attention and MLP modules, aiming for MatMul-free operations while remaining HuggingFace-compatible. Specific code structure details are shown in the example initialization:
>>> from mmfreelm.models import HGRNBitConfig
>>> from transformers import AutoModel
>>> config = HGRNBitConfig()
>>> AutoModel.from_config(config)
The model components include embedding layers and a stack of layers with attention norms (RMSNorm) and various projection blocks (i_proj, f_proj, g_proj, o_proj) using fused operations and RMSNorm-based norms.
Getting started
Installation requirements listed:
- PyTorch >= 2.0
- Triton >= 2.2
- einops
pip install -U git+https://github.com/ridgerchu/matmulfreellm
Recent releases
There are no releases documented in this repository snapshot.
Traction
The repository shows:
- stars: 3083
- forks: 201
- open_issues: 25
Behind the repo
This section is not present in the provided facts.
Caveats
License: Apache-2.0 Created: 2024-04-23 Last push: 2025-12-02 Language: Python Topics: large-language-model, linear-transformer, llm