Llm agents automate model conversion across multiple edge AI runtimes
From PyTorch to the NPU: LLM-Agent-Driven Model Conversion Across Heterogeneous Inference Runtimes
Software EngineeringDistributed, Parallel, and Cluster Computing
Summary
Deploying AI models on different devices often involves many tricky steps like changing model formats and making sure they run correctly. The authors show how large language model (LLM) agents can automate this process across several popular AI runtimes, such as those from Intel, NVIDIA, and Qualcomm. They break down deployment into clear stages and teach the agent how to handle specific runtime requirements to convert a model from PyTorch into an executable form with less manual work. Their work focuses on engineering practice and how to guide the agent reliably, rather than comparing runtime performance or repair strategies.
What this means in practice
- •For edge ai engineers: Automate deployment of vision models onto various hardware accelerators with less manual runtime adaptation work
- •For embedded device developers: Integrate heterogeneous AI runtime support into device pipelines to streamline AI model deployment across chips
Authors
Jianhao Su, Zhanwei Wu, Chia-Heng Tu, ShengTing Huang
Abstract
Edge AI model deployment is a multi-stage engineering process involving model conversion, operator compatibility handling, runtime integration, and precision verification. While prior work has demonstrated agent-based automation for Qualcomm AI Runtime, the broader edge inference runtime ecosystem, including Intel OpenVINO, Rockchip RKNN, NVIDIA TensorRT, and ONNX Runtime, presents distinct toolchains and optimization strategies. This paper extends AIPC (AI Porting Conversion, an LLM agent-driven methodology for AI model deployment automation previously demonstrated on Qualcomm AI Runtime) to multi-runtime scenarios, proposing an LLM agent-driven approach for automated single-model-to-single-runtime deployment across heterogeneous inference backends, such as Intel OpenVINO, Rockchip RKNN, NVIDIA TensorRT, and ONNX Runtime. We decompose the edge AI deployment into standardized, verifiable stages, and inject runtime-specific domain knowledge into the agent execution flow through agent skills, auxiliary scripts, and staged verification loops. Using representative vision models, we demonstrate that agent-based deployment can complete the conversion from a PyTorch model to its executable inference, targeting OpenVINO for x86/NPU, RKNN for RK3588, TensorRT for NVIDIA GPU, and ONNX Runtime for Qualcomm NPU with a focus on FP16 precision deployment feasibility verification. The contributions of this paper primarily lie in providing multi-runtime deployment engineering practice experience, toolchain mapping analysis, a layout-adaptation and inference-replacement layer that removes manual transpose insertion from the agent's repair burden, and an empirical characterization of agent deviation behavior under structured knowledge injection, rather than large-scale systematic benchmarking or cross-runtime operator repair strategy comparison.