star-vector

What it is

StarVector is a multimodal vision-language model for Scalable Vector Graphics (SVG) generation. It can be used to perform image2SVG and text2SVG generation. We pose image generation as a code generation task, using the power of multimodal VLMs

How it works

The model processes both visual and textual inputs to produce SVG code. In image-to-SVG generation, the image is projected into visual tokens and SVG code is generated. In text-to-SVG, the model receives a text instruction and creates an SVG. The LLM is based on StarCoder to transfer coding skills to SVG generation. The project introduces SVG-Stack as training data and SVG-Bench as evaluation benchmarks.

Getting started

Installation:

git clone https://github.com/joanrod/star-vector.git
cd star-vector

conda create -n starvector python=3.11.3 -y
conda activate starvector
pip install --upgrade pip  # enable PEP 660 support
pip install -e .

pip install -e ".[train]"

Upgrade:

git pull
pip install -e .

Quick Start (Image2SVG):

from PIL import Image
from starvector.model.starvector_arch import StarVectorForCausalLM
from starvector.data.util import process_and_rasterize_svg

model_name = "starvector/starvector-8b-im2svg"

starvector = StarVectorForCausalLM.from_pretrained(model_name)

starvector.cuda()
st arvector.eval()

image_pil = Image.open('assets/examples/sample-0.png')
image = starvector.process_images([image_pil])[0].cuda()
batch = {"image": image}

raw_svg = starvector.generate_im2svg(batch, max_length=1000)[0]
svg, raster_image = process_and_rasterize_svg(raw_svg)

From HuggingFace AutoModel:

from PIL import Image
from transformers import AutoModelForCausalLM, AutoTokenizer, AutoProcessor
from starvector.data.util import process_and_rasterize_svg
import torch

model_name = "starvector/starvector-8b-im2svg"

starvector = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16, trust_remote_code=True)
processor = starvector.model.processor
tokenizer = starvector.model.svg_transformer.tokenizer

starvector.cuda()
st arvector.eval()

image_pil = Image.open('assets/examples/sample-18.png')

image = processor(image_pil, return_tensors="pt")['pixel_values'].cuda()
if not image.shape[0] == 1:
    image = image.squeeze(0)
batch = {"image": image}

raw_svg = starvector.generate_im2svg(batch, max_length=4000)[0]
svg, raster_image = process_and_rasterize_svg(raw_svg)

Models

Hugging Face checkpoints exist for image2SVG vectorization:

• starvector-1b-im2svg
• starvector-8b-im2svg These are available via HuggingFace collections and related links in the README.

Datasets - SVG-Bench

SVG-Bench contains 10 datasets and 3 tasks (Image-to-SVG, Text-to-SVG, Diagram-to-SVG). Datasets include SVG-Stack, SVG-Fonts, SVG-Icons, SVG-Emoji, SVG-Diagrams, and others with various train/val/test splits and token lengths. The README lists dataset statistics such as train/val/test counts and token lengths per dataset.

Training

Dependencies: install with

pip install -e ".[train]"

Environment setup examples include HF_HOME, HF_TOKEN, WANDB_API_KEY, OUTPUT_DIR. Training commands include:

• StarVector-1B (Image2SVG) uses accelerate with deepspeed-8-gpu.yaml

accelerate launch --config_file configs/accelerate/deepspeed-8-gpu.yaml starvector/train/train.py config=configs/models/starvector-1b/im2svg-stack.yaml

• StarVector-8B (Image2SVG) uses torchrun with 8 GPUs

torchrun \
  --nproc-per-node=8 \
  --nnodes=1 \
  starvector/train/train.py \
  config=configs/models/starvector-8b/im2svg-stack.yaml

Finetuning sections include Text2SVG and SVG-Bench finetuning commands for both 1B and 8B models, with similar accelerator/torchrun patterns.

Validation on SVG Benchmarks

Validation supports HuggingFace generation backend and vLLM backend, with example commands for StarVector-1B and StarVector-8B on SVG-Stack, using specific config files and dataset names. vLLM setup requires forking and installing a StarVector fork of VLLM.