ChatGLM2-6B is a bilingual open-source large language model with 32K context support, Multi-Query Attention for faster inference, and multiple deployment options including API, CLI, and web demos. Repository provides usage examples, environment setup, and low-cost deployment through quantization.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
ChatGLM2-6B is an open bilingual dialog model, the second generation of ChatGLM-6B, designed for English-Chinese conversations. It claims improvements in benchmarks (MMLU, CEval, GSM8K, BBH) over the base model and introduces longer context lengths (32K) via FlashAttention, as well as more efficient inference using Multi-Query Attention. The model weights are stated to be open for academic use and, after questionnaire registration, allow free commercial use.
How it works
The model uses a mixture of objectives from GLM and introduces long-context support, multi-query attention, and causal masking for dialogue training to reuse KV caches across turns. It provides quantized inference options (INT4, etc.) and FP16 defaults. It reports performance metrics across MMLU, C-Eval, GSM8K, and BBH, and includes comparisons against ChatGLM-6B and various ChatGLM2 configurations.
Getting started
Environment setup:
git clone https://github.com/THUDM/ChatGLM2-6B
cd ChatGLM2-6B
Install dependencies:
pip install -r requirements.txt
Code usage examples show loading with transformers and generating chat responses:
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm2-6b", trust_remote_code=True)
model = AutoModel.from_pretrained("THUDM/chatglm2-6b", trust_remote_code=True, device='cuda')
model = model.eval()
response, history = model.chat(tokenizer, "你好", history=[])
print(response)
Web/demo start commands:
python web_demo.py
streamlit run web_demo2.py
CLI usage:
python cli_demo.py
API deployment:
python api.py
Curl example:
curl -X POST "http://127.0.0.1:8000" \
-H 'Content-Type: application/json' \
-d '{"prompt": "你好", "history": []}'
Recent releases
Latest release listed is 0 with no assets; update information includes 32K model release on 2023/07/31 and related project updates.
Traction
Stars: 15537 (as provided). Language: Python. License: none listed. Created: 2023-06-24. Last push: 2024-06-27.