AI agent workflows improve PyTorch deployment on device accelerators
AIPC: Evidence-Guided Agent Workflows for PyTorch Project Deployment
Software EngineeringArtificial IntelligenceMachine Learning
Summary
Deploying AI models on specialized hardware like on-device NPUs is complicated because models must be converted and adjusted for different hardware and software layers. The authors propose AIPC, a system that guides AI agents with detailed information and skills to help fix and optimize these models automatically. Their approach organizes the whole deployment process including adapting models, building runtime files, running tests, and tuning speed and accuracy. Tests on various AI models showed that AIPC improved accuracy, speed, and reduced human involvement during deployment. They also explored using a large language model to help automate deployment workflows locally.
What this means in practice
- •For embedded device engineers: Automate deployment and optimize AI models for on-device NPUs to meet accuracy and latency requirements with minimal human intervention.
- •For ai platform developers: Build end-to-end deployment pipelines that integrate agent-driven diagnosis and repair for diverse model types and hardware toolchains.
Authors
Zhanwei Wu, Jianhao Su, Brian Zhang, ShengTing Huang, Weidong Feng, Gang Liu, Zhiguo Li, Wei Huang, Li Tan, Wanjun Li, Zejiong Teng, Changgeng Zhong, Guanping Wu, Han-Hsing Tu
Abstract
Deploying and optimizing a PyTorch inference project on a dedicated accelerator, especially an on-device NPU, requires more than using a framework device API as with CUDA. The project must be converted into hardware-runtime artifacts and adapted to the target toolchain and application interface. Differences in model formats, quantization flows, operator support, and toolchain interfaces create cross-layer problems spanning PyTorch/ONNX models, toolchains, runtimes, and device execution. LLM Agents can invoke tools to address such problems, but may drift because of insufficient domain knowledge, large configuration and repair spaces, or poor path selection. We propose AIPC (AI Porting & Conversion), an Agent method for deploying PyTorch inference projects. AIPC combines structured Skills with toolchain, runtime, and profiling evidence to drive cross-layer diagnosis and repair, organizing model adaptation, artifact building, on-target execution, accuracy repair, and performance optimization into one deployment flow. We evaluate AIPC on the on-device NPU path of Qualcomm QAIRT across vision, speech, LLM, and diffusion models. In two optimization cases, 6/7 Skill-arm Whisper-Small runs meet predefined accuracy and latency criteria versus 0/6 control runs, while YOLO-Worldv2 achieves a 2.791x end-to-end speedup versus 1.599x for the control. Under a pre-fixed deployment contract, Skill, tool permissions, and check conditions, some runs achieve zero in-run human intervention (0-HITL). We also examine Qwen3.8-27B as a locally deployable model for driving an Agent workflow.