Small language models follow plans better on offline edge devices
PTC-Decoder: Towards Intelligent SLMs on Offline Resource-Constrained Edge Devices
Artificial Intelligence
Summary
Small language models running on devices with limited power, like satellites, have trouble following multi-step instructions reliably. The authors show that existing methods to make these models follow plans don’t work well because the models ignore plan prompts. They propose PTC-Decoder, a new approach that forces the model to start with a plan step and restricts the model’s output to valid tool names only. This results in better plan-following without needing to retrain the model, improving performance on real satellite tasks.
What this means in practice
- •For embedded system engineers: Enable complex multi-step task execution by small language models on offline, resource-limited devices such as satellites.
- •For industrial iot developers: Improve reliability of on-device AI planning and control for Internet of Things devices operating without internet access.
Authors
Minghui Yu, Ke Mu, Gang Wu
Abstract
Deploying small language models (SLMs) on offline, resource-constrained edge devices such as remote sensing satellites presents a fundamental challenge: their limited reasoning capacity hinders reliable execution of multi-step agent tasks requiring complex tool orchestration. Existing plan-solve paradigms rely on prompt-based enforcement, which our experiments show SLMs almost entirely disregard: weak models fail to invoke the plan. We propose PTC-Decoder (Plan-Tool Constrained Decoder), a training-free, plug-and-play decoder framework that combines (1) a Plan-to-Act paradigm, which elevates planning to an atomic tool and forces its invocation at the first inference step, and (2) TC-Decoder, a deterministic finite automaton that imposes token-level hard constraints on tool names while preserving freedom over parameter generation, thereby retaining SLM reasoning capability. Evaluated on 200 real remote-sensing satellite tasks across 7 SLMs, PTC-Decoder yields a statistically significant mean overall score gain of +1.21 (p<0.01), 95% CI [+1.13, +1.29]), with consistent improvements across models and other datasets. An ablation study that removes TC-Decoder causes substantial performance degradation across all quality metrics without reducing computational cost, confirming TC-Decoder as the primary driver. PTC-Decoder thus offers a lightweight yet effective solution for improving step-level reliability, with final-answer accuracy remaining an open challenge. In essence, we enforce plan adherence by constraining the permissible output vocabulary during inference, without requiring retraining.