ctransformers provides Python bindings for Transformer models implemented in C/C++ via GGML, exposing a unified API for model loading, tokenization, generation, and optional GPU support. It documents supported models, configuration options, and usage examples.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
Python bindings for Transformer models implemented in C/C++ using GGML library. It offers a unified interface for loading models, tokenization, generation, and optional integrations with 🤗 Transformers, LangChain, and GPU options.
How it works
The project exposes a Python API (via the AutoModelForCausalLM, LLM, and related classes) that loads models from local files or Hugging Face hubs, communicates with a C/C++ backend (GGML) under the hood, and provides methods for tokenization, embedding, evaluation, and generation. It supports streaming output and various sampling/configuration parameters.
Getting started
pip install ctransformers
from ctransformers import AutoModelForCausalLM
llm = AutoModelForCausalLM.from_pretrained("/path/to/ggml-model.bin", model_type="gpt2")
print(llm("AI is going to"))
To stream output:
for text in llm("AI is going to", stream=True):
print(text, end="", flush=True)
You can load models from Hugging Face Hub directly:
llm = AutoModelForCausalLM.from_pretrained("marella/gpt-2-ggml")
If a model repo has multiple files, specify a model_file:
llm = AutoModelForCausalLM.from_pretrained("marella/gpt-2-ggml", model_file="ggml-model.bin")
Recent releases
- v0.2.27 (2023-09-10): Skip evaluating tokens that are evaluated in the past; deprecate LLM.
- v0.2.26 (2023-08-30): Add support for 🤗 Transformers
- v0.2.25 (2023-08-29): Add support for GGUF v2, CUDA ROCm; low-level API for add_bos_token and bos_token_id
- v0.2.24 (2023-08-24): Add GGUF format support for Llama and Falcon; support for Code Llama models
- v0.2.23 (2023-08-20): Add mmap and mlock parameters for LLaMA and Falcon; revision option for models on Hugging Face Hub
Traction
Stars: 1885 (as of the provided data) Forks: 143 Open issues: 113
Behind the repo
Linked to a Python binding project for GGML-based transformers; repository created 2023-05-14 and last push 2024-01-28. License: MIT.
Caveats
License: MIT Created: 2023-05-14 Last push: 2024-01-28