LongLoRA

What it is

LongLoRA and LongAlpaca are tools and models for long-context fine-tuning of large language models. The repository includes data, models, training scripts, and instructions to extend context length for LLaMA-2 and related architectures, with LoRA-based fine-tuning and optional QLoRA integration.

How it works

The project documents methods to extend context via LoRA-based fine-tuning (LoRA or full fine-tuning) to achieve longer context windows (up to 100k in some models). It provides supervised fine-tuning and data generation pathways, using pre-trained LLaMA-2 variants as bases, and includes compatibility notes for flash-attn and deepspeed configurations.

Getting started

Installation and quick guide:

pip install -r requirements.txt
pip install flash-attn --no-build-isolation

# Follow with selecting a released model or fine-tuning a model

Recent releases

RELEASES (latest 0):

• none

Traction

2690 stars, 282 forks, 54 open issues

License

Code license: Apache-2.0; Data license: CC By NC 4.0; Weight license: CC By NC 4.0

Models

Models include LongAlpaca-7B, -13B, -70B with 32768 token context, and multiple context-extended variants such as Llama-2-7b-longlora-8k-ft, up to 100k contexts for some configurations. Pre-trained weights are based on LLaMA-2 models (hf links provided).

Training

Pre-trained weights list:

• Llama-2-7b-hf
• Llama-2-13b-hf
• Llama-2-70b-hf
• Llama-2-7b-chat-hf
• Llama-2-13b-chat-hf
• Llama-2-70b-chat-hf

Fine-tuning examples are provided, including a 8k to 100k context range with various batch and optimization settings. Example commands are shown for both Fine-tuning and Supervised Fine-tuning.

Fine-tuning (example)

torchrun --nproc_per_node=8 fine-tune.py  \
        --model_name_or_path path_to/Llama-2-7b-hf \
        --bf16 True \
        --output_dir path_to_saving_checkpoints       \
        --cache_dir path_to_cache \
        --model_max_length 8192 \
        --use_flash_attn True \
        --low_rank_training False \
        --num_train_epochs 1  \
        --per_device_train_batch_size 1     \
        --per_device_eval_batch_size 2     \
        --gradient_accumulation_steps 8     \
        --evaluation_strategy "no"     \
        --save_strategy "steps"     \
        --save_steps 1000     \
        --save_total_limit 2     \
        --learning_rate 2e-5     \
        --weight_decay 0.0     \
        --warmup_steps 20     \
        --lr_scheduler_type "constant_with_warmup"     \
        --logging_steps 1     \
        --deepspeed "ds_configs/stage2.json" \
        --tf32 True \
        --max_steps 1000

• Note path_to/ placeholders to be replaced by user paths.
• You may adjust model_max_length and ds_configs accordingly.

Supervised Fine-tuning (example)

torchrun --nproc_per_node=8 supervised-fine-tune.py  \
        --model_name_or_path path_to_Llama2_chat_models \
        --bf16 True \
        --output_dir path_to_saving_checkpoints       \
        --model_max_length 16384 \
        --use_flash_attn True \
        --data_path LongAlpaca-16k-length.json \
        --low_rank_training True \
        --num_train_epochs 5  \
        --per_device_train_batch_size 1     \
        --per_device_eval_batch_size 2     \
        --gradient_accumulation_steps 8     \
        --evaluation_strategy "no"     \
        --save_strategy "steps"     \
        --save_steps 98     \
        --save_total_limit 2     \
        --learning_rate 2e-5     \
        --weight_decay 0.0     \
        --warmup_steps 20     \
        --lr_scheduler_type "constant_with_warmup"     \
        --logging_steps 1     \
        --deepspeed "ds_configs/stage2.json" \
        --tf32 True