Interflow speeds up private edge ai by pipelining randomness generation
Interflow: Breaking the Offline-Online Barrier for Private AI at the Edge
Hardware ArchitectureCryptography and Security
Summary
Running private AI on devices like phones is hard because it needs special secret data that takes up a lot of space and slows things down. The authors found a way to make this secret data while the AI is working, instead of making it all beforehand and storing it. This new method uses hardware and smart scheduling to make AI run faster and use less memory without risking privacy. It works especially well for popular AI models like ResNet and BERT.
What this means in practice
- •For edge device engineers: Increase speed and reduce memory use when running private AI models on resource-limited edge hardware.
- •For ai hardware designers: Design hardware chips that overlap preprocessing with inference to improve throughput in privacy-preserving AI systems.
Authors
Hanieh Totonchi Asl, Ebrahim Nouri, Zhuoran Li, Danella Zhao
Abstract
Privacy-preserving machine learning (PPML) offers a promising path toward bringing powerful AI services to the edge without exposing sensitive user data. However, its practical deployment remains constrained by the high communication, computation, and data-movement costs of secure multi-party computation (MPC), particularly during nonlinear inference. Existing MPC systems typically decouple input-independent correlated randomness generation from online execution, creating substantial memory pressure to buffer single-use cryptographic data. Moreover, limited storage can force a stop-and-refill execution pattern that repeatedly stalls the pipeline to replenish these buffers, bringing randomness generation back onto the critical path. We propose Interflow, an algorithm-hardware co-design that rethinks this separation by batch-wise processing and pipelining correlated-randomness generation with online execution: while the current batch consumes pre-generated randomness to execute secure nonlinear operations, correlation generation proceeds concurrently for the next batch. By leveraging interleaved execution, efficient data movement, and hardware acceleration, the design transforms preprocessing from a serialized, storage-intensive offline phase into a continuously overlapped pipeline that generates correlated randomness closer to the point of consumption, reducing memory footprint and data movement overhead. Implemented on the Zynq UltraScale+ ZCU102, Interflow improves nonlinear evaluation throughput by up to 1.55x for CNN operators and 1.48x for transformer operators. It achieves up to 1.26x speedup for ResNet-50 and up to 1.21x speedup for BERT-base, and reduces memory footprint by up to 37%, compared with baseline frameworks.