Papers for

gpu driver teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

WebGPU speeds up large language model responses inside browsers

GPU-Resident Speculative Decoding for In-Browser LLM Inference with WebGPU

Abstract: WebGPU enables private, offline LLM inference inside web pages, but browser dispatch, inter-process communication, synchronous readback and single-token kernels slow decoding and make conventional speculative decoding inefficient. We present a WebGPU runtime that addresses these browser-specific costs. Deterministic small-batch integer kernels accelerate multi-token verification while preserving exact agreement with greedy decoding in the same quantized runtime. A GPU-resident loop performs drafting, verification, acceptance and KV-cache maintenance without a per-token CPU decision. A draft-free context-mixing drafter reuses page content, conversation history and session memory, while a device-aware controller adapts the speculation width and the submission window to measured costs. Across seven models from 0.5B to 8B parameters, 11 tasks and three client GPUs, our runtime decodes 3.5 to 5.4$\times$ faster than WebLLM on a laptop GPU, 4.8 to 6.2$\times$ faster on an integrated GPU and 4.8 to 8.6$\times$ faster on a workstation GPU. The GPU-resident loop and speculation contribute a further 1.8 to 3.2$\times$ over the optimized kernels on the workstation. Outputs match the quantized greedy decoding of the runtime exactly, and the mean GSM8K accuracy matches that of WebLLM. Optional draft heads raise the peak speedup to 10.2$\times$, while session memory provides additional gains without downloading extra weights.

Thu 29 OctComputers and Society
The gist
Running large language models (LLMs) directly inside web browsers usually slows down because of communication and processing limits. The paper presents a new way to keep most computations on the GPU, avoiding delays from back-and-forth with the CPU. This method speeds up generating multiple words at once while keeping answers exactly the same as slower methods. Tests on several models and graphics cards show it can make LLM responses 3 to 8 times faster without losing accuracy.
Open → 2610.39761v1