AgentDP automates AI model deployment on varied embedded devices
AgentDP: Towards Automated Model Deployment for Heterogeneous Embedded AI Applications
Artificial Intelligence
Summary
Deploying AI models on different small devices is usually hard and takes a lot of time. The authors developed AgentDP, a system that uses large language models to automatically adjust AI models so they work well on many types of devices. It helps users clearly state what they want, splits AI models to run efficiently on different parts of devices, and reuses previous settings to save time. Tests showed it speeds up the process a lot and improves AI accuracy compared to doing it manually.
What this means in practice
- •For embedded system engineers: Automatically adapt AI models for efficient inference on diverse embedded hardware without manual tuning.
- •For mobile app developers: Deploy pre-trained AI features on various smartphones with minimal manual configuration by automating optimization.
Authors
Leming Shen, Qiang Yang, Xinyu Huang, Zijing Ma, Yuanqing Zheng
Abstract
This paper presents AgentDP, which aims to deploy pre-trained AI models on heterogeneous embedded devices by fully automating model optimization and adaptation (O&A) with Large Language Model (LLM) agents. AgentDP not only significantly reduces human effort via automation but also enhances model inference efficacy by integrating LLMs with device-specific domain knowledge. To deliver a robust automation framework, AgentDP develops novel solutions to address three key technical challenges: 1) To help users elicit clear requirements, we propose a user requirement standardization module; 2) To mitigate the execution environment mismatch between AI models and heterogeneous embedded devices, we design an operator-aware model partition paradigm that enables different parts of the model to be efficiently executed on distinct inference engines. 3) To avoid re-searching optimization hyperparameters for new tasks, we propose a configuration reuse module that heuristically retrieves previous hyperparameter configurations for efficient reuse. Extensive experiments demonstrate that by fully automating the entire O&A process, AgentDP reduces entire model adaptation time by up to 80% compared with manual O&A while achieving higher task accuracy (more than 13% on average) than the baselines. Additionally, a comprehensive user study also exhibits the robust generalizability of AgentDP in adapting to different user requirements for heterogeneous embedded devices.