NExT-GPT provides code, data and model weights for an end-to-end multimodal LLM that handles text, image, video, and audio. It includes environment setup, dataset preparation, and training/run scripts, with multiple pretrained components and diffusion-model integrations.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
NExT-GPT is code and models for an ICML 2024 paper describing an end-to-end multimodal LLM that perceives inputs and generates outputs in arbitrary combinations across text, image, video, and audio. The repository also references legacy code and provides a structure for training and deploying multimodal capabilities.
How it works
The approach combines multimodal encoders, an LLM core, and multimodal decoders with projection layers. It encodes inputs via modality-specific encoders, translates representations into language-like forms using a projection layer, and uses an LLM to process information and emit both text and modality signal tokens that guide decoding.
Getting started
Getting started steps include environment setup and cloning the repo, followed by installing dependencies:
conda env create -n nextgpt python=3.8
conda activate nextgpt
# CUDA 12.1
conda install pytorch==2.1.2 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.6 -c pytorch -c nvidia
git clone https://github.com/NExT-GPT/NExT-GPT.git
cd NExT-GPT
pip install -r requirements.txt
Recent releases
The README does not list specific releases; latest release notes are not provided in the visible content.
Traction
Stars: 3639, Forks: 360, Open issues: 81. Language: Python. License: BSD-3-Clause. Created: 2023-08-30. Last push: 2025-05-13.
Behind the repo
Not present in the provided content.
Caveats
The README includes procedures for preparing multiple pretrained checkpoints (ImageBind, Vicuna, diffusion models), and notes about data copyright issues for MosIT data. It references external download links and datasets, with some placeholder links (e.g., MosIT data download link).