WebGPU speeds up large language model responses inside browsers
GPU-Resident Speculative Decoding for In-Browser LLM Inference with WebGPU
Computers and Society
Summary
Running large language models (LLMs) directly inside web browsers usually slows down because of communication and processing limits. The paper presents a new way to keep most computations on the GPU, avoiding delays from back-and-forth with the CPU. This method speeds up generating multiple words at once while keeping answers exactly the same as slower methods. Tests on several models and graphics cards show it can make LLM responses 3 to 8 times faster without losing accuracy.
What this means in practice
- •For web developers: Run faster and private large language model inference directly within web pages without server calls using WebGPU acceleration.
- •For gpu driver teams: Optimize GPU runtimes to handle multi-token speculative decoding workloads efficiently for web-based AI applications.
Authors
Wei Song, Yuxin Cao, Ziqi Ding, Xiao Cheng
Abstract
WebGPU enables private, offline LLM inference inside web pages, but browser dispatch, inter-process communication, synchronous readback and single-token kernels slow decoding and make conventional speculative decoding inefficient. We present a WebGPU runtime that addresses these browser-specific costs. Deterministic small-batch integer kernels accelerate multi-token verification while preserving exact agreement with greedy decoding in the same quantized runtime. A GPU-resident loop performs drafting, verification, acceptance and KV-cache maintenance without a per-token CPU decision. A draft-free context-mixing drafter reuses page content, conversation history and session memory, while a device-aware controller adapts the speculation width and the submission window to measured costs. Across seven models from 0.5B to 8B parameters, 11 tasks and three client GPUs, our runtime decodes 3.5 to 5.4$\times$ faster than WebLLM on a laptop GPU, 4.8 to 6.2$\times$ faster on an integrated GPU and 4.8 to 8.6$\times$ faster on a workstation GPU. The GPU-resident loop and speculation contribute a further 1.8 to 3.2$\times$ over the optimized kernels on the workstation. Outputs match the quantized greedy decoding of the runtime exactly, and the mean GSM8K accuracy matches that of WebLLM. Optional draft heads raise the peak speedup to 10.2$\times$, while session memory provides additional gains without downloading extra weights.