web-llm

What it is

WebLLM is a high-performance in-browser LLM inference engine that runs language model inference inside the browser using WebGPU. It is fully compatible with the OpenAI API and provides streaming, JSON-mode generation, and support for multiple cache backends. It is distributed as an npm package and supports CDN delivery.

How it works

WebLLM runs in-browser with hardware acceleration via WebGPU. It exposes an MLCEngine interface and a CreateMLCEngine factory to initialize and load models. It supports loading models locally in the browser and offers cache backends such as cache, indexeddb, OPFS, and a cross-origin storage backend. It includes a web worker and service worker integration path for offloading computation and offline usage.

Getting started

Installation (package manager):

# npm
npm install @mlc-ai/web-llm
# yarn
yarn add @mlc-ai/web-llm
# or pnpm
pnpm install @mlc-ai/web-llm

CDN delivery example:

import * as webllm from "https://esm.run/@mlc-ai/web-llm";

Create MLCEngine usage:

import { CreateMLCEngine } from "@mlc-ai/web-llm";
const initProgressCallback = (initProgress) => {
  console.log(initProgress);
};
const selectedModel = "Llama-3.1-8B-Instruct-q4f32_1-MLC";
const engine = await CreateMLCEngine(
  selectedModel,
  { initProgressCallback: initProgressCallback }, // engineConfig
);

Cache backends configuration example:

import { CreateMLCEngine, prebuiltAppConfig } from "@mlc-ai/web-llm";
const appConfig = { ...prebuiltAppConfig, cacheBackend: "cross-origin" };
const engine = await CreateMLCEngine("Llama-3.1-8B-Instruct-q4f32_1-MLC", {
  appConfig,
});

Recent releases

Latest releases include:

• v0.2.83 (2026-04-24): Add support for cross-origin storage; Support integrity verification for model artifacts; Add subgroup support and subgroup wasm gating example; Refactor llm_chat.ts function.
• v0.2.82 (2026-03-13): Update failing tests; Add CONTRIBUTING.md; Add CI for tests, build, and security.
• v0.2.81 (2026-02-18): Engine and API overhaul: ChatModule refactored into Engine/MLCEngine, consolidated constructor/reload behavior, multi-model loading, better worker lifecycle, concurrency handling; OpenAI API: mirror.
• v0.2.0 (2023-05-26): Major TypeScript overhaul; npm package; WebWorker support.

Traction

Stars: 18509

Behind the repo

Not included in the provided facts.

Caveats

License: Apache-2.0 Created: 2023-04-13 Last push: 2026-08-04 Open issues: 150 Language: TypeScript Topics include: chatgpt, deep-learning, language-model, llm, tvm, webgpu, webml