LLMCompiler is a Python framework that orchestrates parallel function calling for LLMs, supporting both open-source and closed models, with a workflow to identify parallelizable tasks and manage multi-model prompts. Installation and basic run commands are provided for evaluation scenarios and custom models.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
LLMCompiler is a framework that enables an efficient and effective orchestration of parallel function calling with LLMs, including both open-source and close-source models, by automatically identifying which tasks can be performed in parallel and which ones are interdependent.
How it works
The tool decomposes problems into multiple tasks that can be executed in parallel and computes an optimized orchestration for the function calls. It supports open-source models (e.g., LLaMA) and OpenAI models, with variants for vLLM-serving and Friendli endpoints as described in the usage notes. It provides integration points for specifying tools, in-context examples, and prompts.
Getting started
Install steps from the README:
conda create --name llmcompiler python=3.10 -y
conda activate llmcompiler
git clone https://github.com/SqueezeAILab/LLMCompiler
cd LLMCompiler
pip install -r requirements.txt
Basic runs for evaluation:
export OPENAI_API_KEY="sk-xxx"
python run_llm_compiler.py --benchmark {benchmark-name} --store {store-path} [--logging] [--stream]
For custom models via vLLM:
python run_llm_compiler.py --model_type vllm --benchmark {benchmark-name} --store {store-path} --model_name {vllm-model-name} --vllm_port {vllm-port} [--logging]
Azure/Microsoft/Friendli endpoints and additional options are described in the README:
--model_type azure
pip install friendli-client
python evaluate_results.py --file {store-path}
Getting started (continued)
The README outlines how to run predefined benchmarks (hotpotqa, movie, parallelqa) and how to point to a store to collect results, including latency and predictions.
Recent releases
The latest releases section shows no entries under RELEASES (latest 0): none.
Traction
The repository has 1869 stars and 134 forks as reported in the facts block.
Behind the repo
Not provided in the README or facts block.
Caveats
Metadata:
- License: MIT
- Created: 2023-12-06
- Last push: 2024-07-10
- Primary language: Python
Key operational notes include support for multiple endpoints (OpenAI, vLLM, Azure, Friendli) and the need to set environment variables for API keys. The README emphasizes configuration details for benchmarks and result storage, but does not include explicit licensing caveats beyond the MIT license and does not mention open issues. The releases section has none. All command lines and options are quoted exactly as written in the README.