FMplex: Model Virtualization for Serving Extensible Foundation Models
2026-06-08 • Distributed, Parallel, and Cluster Computing
Distributed, Parallel, and Cluster ComputingArtificial IntelligenceMachine LearningOperating Systems
AI summaryⓘ
The authors discuss how foundation models (FMs), used for many AI tasks, are usually duplicated for each specific job, wasting memory and processing power. They introduce FMplex, a system that lets different tasks share a single FM backbone while keeping their own custom parts separate. This approach improves efficiency by reducing memory use and speeding up processing. FMplex also uses a special scheduler to balance work fairly among tasks. Their tests show FMplex lowers latency and supports more tasks compared to previous methods.
foundation modelsmodel servingvirtualizationbatchingschedulertask isolationaccelerator memorylatencyco-location
Authors
Hetvi Shastri, Pragya Sharma, Walid A. Hanafy, David Irwin, Mani Srivastava, Prashant Shenoy
Abstract
Foundation models (FMs) are increasingly used as backbones for downstream tasks across language, vision, time-series, and multimodal applications. Yet existing model-serving systems deploy each customized task as an independent model instance, thereby replicating heavyweight backbones, wasting accelerator memory, and losing opportunities to amortize batching and loading costs. This paper presents FMplex, a serving system that treats FM backbones as a virtualization substrate for deployment sharing. FMplex presents each task with a virtual foundation model (vFM), a logically private FM instance backed by a shared physical FM. This abstraction lets independently customized tasks share a backbone while preserving task-specific extensions, independent lifecycles, and task-level isolation. In addition, we propose a batch-aware fair-queueing scheduler that combines weighted task-level sharing with inter- and intra-task batching across colocated tasks. We implement a FMplex-based serving stack spanning task construction, sharing-aware deployment, and runtime execution. Across 7 FM backbones (16 variants) and 92 downstream tasks, FMplex reduces latency by up to 80% over spatial partitioning and 33.3% over best-effort co-location, while hosting up to 6x more tasks at cluster scale.