FastDeploy

What it is

FastDeploy is a Python-based toolkit for high-performance inference and deployment of large language models (LLMs) and vision-language models (VLMs) built on PaddlePaddle. It provides production-grade deployment features and aims to support OpenAI API service and vLLM compatibility, with quantization formats and advanced acceleration techniques.

How it works

• Supports multiple hardware backends including NVIDIA GPUs, KunlunXin XPU, Iluvatar GPUs, Enflame GCU, Hygon DCU, Metax GPUs, and Intel Gaudi.
• Offers features such as load-balanced PD decomposition, KV caching, online/offline inference, and supports various quantization formats (e.g., W8A16, W8A8, W4A16, W4A8, W2A16, FP8).
• Provides disaggregated deployment, speculative decoding, prefix caching, chunked prefill, and router-based load balancing.

Getting started

• Installation guidance references multiple platforms and hardware backends:
  • NVIDIA GPU: ./docs/zh/get_started/installation/nvidia_gpu.md
  • Kunlunxin XPU: ./docs/zh/get_started/installation/kunlunxin_xpu.md
  • Iluvatar GPU: ./docs/zh/get_started/installation/iluvatar_gpu.md
  • Enflame GCU: ./docs/zh/get_started/installation/Enflame_gcu.md
  • Hygon DCU: ./docs/zh/get_started/installation/hygon_dcu.md
  • Metax GPU: ./docs/zh/get_started/installation/metax_gpu.md
  • Intel Gaudi: ./docs/zh/get_started/installation/intel_gaudi.md
• Quick start and deployment guides are available:
  • 10分钟快速部署
  • ERNIE-4.5 部署
  • ERNIE-4.5-VL 部署
  • 离线推理
  • 在线服务
  • 最佳实践

Recent releases

• v2.5.0 released on 2026-04-09: FastDeploy Release 2.5. Release notes indicate new model support for Qwen3-VL and MoE variants, CUDA Graph support, and GLM TP+DP+EP support among other updates.
• Prior entries in Chinese release notes mention v2.4.0 (2026-01-23) with core inference and model support enhancements, including text prompt_logprob/logprob capabilities and ZMQ-based offline/online logprobs, and v2.3.0 (2025-11-11) adding GLM 4.5 text-class models and ERNIE-4.5-VL-28B-A3B-Thinking deployment.

Traction

• Stars: 3703

Behind the repo

• Description indicates a broader ecosystem around ERNIE, llm-serving, and OpenAI/vLLM compatibility as topics; no company accelerator or startup link is provided in the facts.

Caveats

• License: Apache-2.0
• Created: 2022-06-27
• Last push: 2026-08-03
• Issues: 672
• Language: Python
• Releases and changelog are partially in Chinese; latest release notes are available under the release page.