Small models keep strong accuracy with new iterative pruning method
DAC-Loop: Distillation-Aware Compression for Remote Sensing CNN
Machine Learning
Summary
Big computer models are great at identifying scenes from satellite images but are too big to run on small devices like drones or phones. The authors show how to make these models much smaller without losing accuracy by carefully removing parts while teaching the small model from the big one. Their method repeats pruning and learning steps, which helps keep the model smart even when over 90% smaller. Tests on two remote sensing datasets confirm this approach works better than usual ways of shrinking models.
What this means in practice
- •For embedded system engineers: Deploy smaller but accurate CNN models for satellite image classification on hardware with limited compute and memory.
- •For mobile app developers: Run efficient remote sensing models on mobile devices for real-time scene recognition without heavy processing power.
Authors
Rico Staedeli, Lorena Raichle, Konstantinos Tzevelekakis, Aron Asefaw, Damian Borth
Abstract
While large-scale models achieve strong results in remote sensing scene classification, deploying them on edge devices remains difficult due to limited compute, memory, and energy budgets. Compact CNN backbones are therefore a practical choice in many real-world settings, offering efficient inference with well-optimized implementations on embedded hardware. To push this efficiency further, we investigate how far a knowledge-distilled student model can be compressed while preserving high accuracy. We propose a Distillation-Aware Compression Loop (DAC-Loop) that integrates (i) knowledge distillation applied both before and after structured channel pruning, and (ii) an iterative compression schedule that alternates structured pruning with unstructured sparsification. Across EuroSAT and NWPU-RESISC45, the DAC-Loop consistently improves the accuracy-compression trade-off relative to standard one-shot pruning and single-stage KD recovery, maintaining strong accuracy even beyond 90% parameter reduction. These results demonstrate that combining distillation and pruning in a unified iterative loop offers an effective path toward compact remote sensing models.