SelectInfer: Selective Neuron Loading and Computation for On-Device LLMs
2026-07-20 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors present SelectInfer, a method to make large language models (LLMs) run more efficiently on small devices like phones. Instead of using the whole model, SelectInfer identifies which parts (neurons) are most important for a task and only loads and computes those. This approach reduces the amount of memory and processing needed without hurting performance much. Their tests show it works well on different tasks, making LLMs easier to use on devices with limited resources.
Large Language ModelsNeuron-level optimizationEdge devicesModel compressionSelective loadingSelective computationModel inferenceMemory footprintQuantizationPruning
Authors
Huzaifa Shaaban Kabakibo, Eric Schniedermeyer, Artem Burchanow, Lin Wang
Abstract
Large Language Models (LLMs) have demonstrated remarkable capabilities across a range of Natural Language Processing (NLP) tasks, but their high computational and memory demands pose significant challenges for deployment on resource-constrained edge devices. Existing approaches to model compression and optimization often rely on coarse-grained pruning or quantization, which can compromise accuracy or require re-training and fine-tuning. In this work, we introduce SelectInfer, a neuron-level optimization framework that enables efficient LLM inference on edge devices through selective neuron loading and computation. By profiling and identifying both task-specific and general-purpose neurons using an offline LLM profiler, SelectInfer implements two key optimizations: selective loading, which reduces memory footprint by selectively loading a subset of neurons that were identified to be most important during the offline stage, and selective computation, which dynamically computes only the most relevant neurons at runtime. Evaluation across multiple datasets shows that SelectInfer achieves significant reductions in memory footprint and computation while preserving task performance, making it a practical step towards enabling LLM deployment on edge devices