Pruning large language models impacts smart home task reliability
What Breaks Under Pruning in Smart Homes, and When? Evaluating LLM Degradation Across Architectures and Task Complexity
Computation and Language
Summary
Using pruning techniques can make large language models (LLMs) run faster and cheaper, but this paper finds that pruning can cause these models to make more mistakes in smart home tasks. The authors tested different kinds of LLMs and pruning methods on smart home commands to see what breaks and when. They discovered that simpler models quickly lose performance when pruned too much, while complex mixture-of-experts models handle pruning better. However, pruning tends to reduce how specifically the model follows instructions and can cause it to refuse commands unnecessarily.
What this means in practice
- •For smart home developers: Select and prune large language models for smart home device command execution without sacrificing reliability on complex tasks.
- •For edge computing engineers: Optimize pruning strategies for deploying large language models on limited hardware while understanding specific failure modes in tool calling.
Authors
Congjing Zhang, Vashishtha Patil, Henning Lange, Usman Aleem
Abstract
Pruning can reduce the deployment cost of large language models (LLMs), but its impact on context-grounded tool calling remains poorly understood. We systematically study pruning-induced degradation in smart-home tool calling across four LLMs spanning dense Transformer, dense hybrid, and mixture-of-experts (MoE) architectures, together with depth, width, hybrid, and expert pruning methods. After post-pruning supervised fine-tuning (SFT), we evaluate more than 19,500 instances from three smart-home datasets. Beyond aggregate task accuracy, we characterize degradation along two dimensions: action components (i.e., operation, device, argument, and value) and task complexity. Our results show that dense models have narrow safe pruning regions followed by sharp degradation, while MoE models tolerate substantially more pruning. Pruning degrades grounded specificity before schema-level intent, and aggressive dense pruning can induce systematic over-refusal. These findings highlight the importance of evaluating pruning beyond aggregate accuracy when selecting pruned LLMs for reliable tool execution.