VLM-R1 is a Python-repo that trains and evaluates a reinforcement-learning-enhanced vision-language model for referring expression tasks, with multi-image and multi-node capabilities. It provides GRPO, LoRA, SFT workflows and data-loading utilities, plus multiple pretrained models.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
VLM-R1: A stable and generalizable R1-style Large Vision-Language Model for Referring Expression Comprehension (REC). The project trains and evaluates models (REC/GRPO) using RL-based methods and integrates multiple training scripts and model variants.
How it works
The repo supports:
- Full fine-tuning for GRPO
- Freeze vision modules via a script option
- LoRA fine-tuning for GRPO
- Multi-node training demonstrations
- Multi-image input training
- Support for adding new multimodal models (e.g., QwenVL, InternVL)
It includes a mechanism to load custom data via jsonl files and provides reward-function customization through grpo_jsonl tooling. The update log shows integration of a reward customization flag and unified training/logging improvements.
Getting started
Setup:
conda create -n vlm-r1 python=3.10
conda activate vlm-r1
bash setup.sh
Training concepts and data preparation are documented under REC/GRPO, with steps to:
- Prepare COCO Train2014 and annotation files
- Configure data_paths and image_folders in run_scripts/run_grpo_rec.sh
- Run the GRPO script:
bash run_scripts/run_grpo_rec.sh - For multi-node training, refer to multinode_training_demo.sh
- For SFT setup, clone LLaMA-Factory and run:
llamafactory-cli train examples/train_full/qwen2_5_vl_full_sft.yaml
For your own data, the repository provides jsonl formats and example scripts, including multi-image input guidance.
Recent releases
Latest three releases:
- v0.2.1 (2025-04-15)
- Changes reference development patches from PRs 177 and 186
- v0.2.0 (2025-03-24)
- Added test_od_r1 and ongoing development
- v0.1.0 (2025-03-17)
- Initial set of data loaders and doc updates
Traction
Stars: 6017 Forks: 385 Open issues: 166
Behind the repo
Not provided in the given data.
Caveats
License: Apache-2.0 Created: 2025-02-06 Last push: 2026-07-07 This repository includes extensive README-backed usage notes, multiple scripts for training GRPO, SFT, LoRA, and multi-image inputs, and references to external datasets and model checkpoints.