When Words Predict Workload
2026-07-06 • Distributed, Parallel, and Cluster Computing
Distributed, Parallel, and Cluster ComputingComputation and Language
AI summaryⓘ
The authors address problems with standard methods for managing distributed large language model (LLM) tasks, which struggle when processing legally strict text like European patent claims. They introduce a CPU-side Linguistic Resource Forecasting gateway that predicts when complex processing will overwhelm limited GPU memory, allowing smart routing of requests to avoid crashes. In tests, their system greatly reduced errors and kept GPU memory usage safely low, even with varying network delays. The authors’ method improves prediction accuracy and adaptability over simpler static approaches.
large language model (LLM)GPU VRAMEuropean Patent Convention (EPC) Article 84XGBoost predictorKV-cachemulti-model ensemblelatency telemetryout-of-memory (OOM) errorAUROCdynamic routing threshold
Authors
Anubhab Banerjee
Abstract
Standard distributed \ac{llm} schedulers rely on static token counts or rolling latency averages, making them susceptible to failures on statutorily constrained text. On \ac{epo} claims governed by Article 84 \ac{epc}, linguistic rigidity makes human and machine authorship statistically indistinguishable. Resolving this ambiguity mid-flight forces dynamic multi-model ensemble expansion, triggering unpredictable KV-cache and weight-allocation spikes that saturate consumer-grade edge GPU VRAM and cause severe \ac{oom} crashes. To prevent hardware collapse, we propose a CPU-side Linguistic Resource Forecasting (LRF) gateway. The gateway extracts a 16-dimensional text-structure vector and applies an XGBoost predictor to forecast trap-band membership. The resulting escalation probability ($\Pesc$) is evaluated against a dynamic, closed-form routing threshold ($\Tauroute(t)$) computed via real-time latency telemetry. Requests are safely routed to either a local Qwen2.5-7B edge worker or a remote contrastive ensemble (Qwen2.5 7B + 32B) on an NVIDIA H100 \emph{before} any edge GPU memory is allocated. In a 6,000-request live trial, the LRF gateway reduced the operational misroute fraction ($R_{\mathrm{mis}}$) to $0.087$--$0.095$, an order of magnitude below the token-count baseline ($0.849$). Peak edge VRAM remained safely bounded at $\SI{4.82}{\gibi\byte}$ (under the $\SI{8}{\gibi\byte}$ ceiling) across a $27\times$ variation in \ac{wan} delay. The predictor achieved a live-trial AUROC of $0.84$, and the dynamic $\Tauroute(t)$ controller yielded an $8.2\%$ relative reduction in misroutes compared to an equivalent static threshold.