Mobile inference speed improved by smart scheduling on multiple processors
Partition-Aware Scheduling for Mobile Heterogeneous Inference Co-Execution
Distributed, Parallel, and Cluster ComputingMachine LearningPerformance
Summary
Mobile devices often run AI tasks using both their CPUs and GPUs, but deciding how to split work between them can be tricky. The authors looked at taking both big chunks of work and small pieces inside those chunks to run together on CPUs and GPUs. They found the best way to do this depends on how the AI task is structured. They built a method that guesses the fastest way to organize the work without needing long tests. This method gets close to the best possible speeds while keeping overhead low.
What this means in practice
- •For mobile app developers: Schedule AI model tasks on mobile devices to reduce delay using CPU and GPU together efficiently.
- •For embedded systems engineers: Configure heterogeneous mobile platforms to handle complex inference models with balanced CPU-GPU workloads for faster results.
Authors
Zhuojin Li, Marco Paolieri, Leana Golubchik
Abstract
Modern mobile inference runs on heterogeneous platforms combining mobile GPUs with multiple CPU core clusters. Existing optimizations typically exploit either inter-operator parallelism, by assigning entire operators to CPU cores or to the GPU, or intra-operator parallelism, by partitioning each operator for CPU-GPU co-execution. We consider these two forms of parallelism together, to improve inference latency of tasks that can be represented by a static DAG of operators with predefined input/output tensor shapes (e.g., CNNs or vision transformers). We define the problem of partition-aware DAG scheduling for mobile heterogeneous inference, illustrating that the best strategy depends on the structure of the inference DAG, thus motivating a joint formulation capturing operator partition choices, device assignment, and execution order. We propose an online iterative search framework, which decomposes large DAGs into stages, focuses search on critical operators, and uses latency predictors to estimate partitioned execution without exhaustive profiling. Across representative mobile inference workloads, our approach achieves latency close to an offline solution while keeping scheduling overhead to a fraction of the model initialization cost, allowing platform-specific scheduling at deployment time.