LISA is a Python project for reasoning segmentation using large language models, with multi-modal inputs and multi-stage training/inference workflows. It provides data preparation, training, inference, and deployment guidance, plus references to LISA and LISA++ models.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
LISA: Reasoning Segmentation via Large Language Model. It is a project that combines large language models with multi-modal segmentation to produce segmentation masks guided by natural language reasoning.
How it works
The repository includes instructions for data preparation across multiple segmentation datasets, training and validation commands, and inference steps to chat with LISA-13B variants. It covers merging weights from LLaVA, using SAM ViT-H weights, and running chat-based prompts to generate segmentation masks. It exposes separate workflows for training, validation, and deployment.
Getting started
Install dependencies:
pip install -r requirements.txt
pip install flash-attn --no-build-isolation
Training is described with a deepspeed command template:
deepspeed --master_port=24999 train_ds.py \
--version="PATH_TO_LLaVA" \
--dataset_dir='./dataset' \
--vision_pretrained="PATH_TO_SAM" \
--dataset="sem_seg||refer_seg||vqa||reason_seg" \
--sample_rates="9,3,3,1" \
--exp_name="lisa-7b"
And instructions for saving merged LoRA weights and validation:
CUDA_VISIBLE_DEVICES="" python merge_lora_weights_and_save_hf_model.py \
--version="PATH_TO_LLaVA" \
--weight="PATH_TO_pytorch_model.bin" \
--save_path="PATH_TO_SAVED_MODEL"
deepspeed --master_port=24999 train_ds.py \
--version="PATH_TO_LISA_HF_Model_Directory" \
--dataset_dir='./dataset' \
--vision_pretrained="PATH_TO_SAM" \
--exp_name="lisa-7b" \
--eval_only
To chat:
CUDA_VISIBLE_DEVICES=0 python chat.py --version='xinlai/LISA-13B-llama2-v1'
CUDA_VISIBLE_DEVICES=0 python chat.py --version='xinlai/LISA-13B-llama2-v1-explanatory'
Recent releases
Release notes list multiple model releases and online demo links, including LISA and LISA++ variants, with accompanying data and training code availability. The README section shows a sequence of updates from 2023 to 2024, including model releases and paper announcements.
Traction
The repository shows 2673 stars and 206 forks as raw counts. Open issues: 115. Language: Python. License: Apache-2.0. Creation date: 2023-08-01. Last push: 2025-02-16.
Behind the repo
The project references the authors and related papers, including LISA and LISA++ teams, and links to arXiv papers and Hugging Face model pages. No standalone startup/company information is provided beyond the academic collaboration context.
Caveats
The README includes licensing and dataset preparation notes, and provides detailed instructions for training and inference, including specific model versions and paths. It notes legacy model handling with a git checkout command for v0 models and mentions 4-bit/8-bit inference options and VRAM considerations. Also, the Installation and Deployment sections contain commands that must be run verbatim as written.