Benchmarking how well old knowledge helps new machine learners
After the Federation: Benchmarking Knowledge Portability to Heterogeneous Newcomers
Computer Vision and Pattern Recognition
Summary
When a system built by many partners stops working together, new systems must still learn from what was left behind. This paper introduces a way to test how useful old knowledge is when used by new systems with different designs. The authors found that some usual methods overestimate how well knowledge transfers, and they offer a better baseline method that works without extra data. Their tests show this method balances accuracy and speed well. They provide a standard test to measure how knowledge sharing works after teams split.
What this means in practice
- •For machine learning engineers: Evaluate the effectiveness and costs of transferring learned knowledge to new models with different architectures after collaboration ends using the FEDNBO benchmark.
- •For mobile app developers: Improve accuracy and reduce configuration time when deploying updated AI models on devices with new hardware by using the proposed knowledge distillation reference.
Authors
Xinyuan Zhao
Abstract
Once historical clients become unavailable, a federation must serve newcomers through the artifact it leaves behind. How much of its knowledge remains useful in an architecture chosen afterward? We introduce FEDNBO, a seven-dataset benchmark that separates knowledge supply, policy identification, and private-model compilation under disjoint data roles. Controlled studies expose inflated oracle gains, architecture-dependent transfer reversals, and a gap between teacher accuracy and private-model utility. Ordinary public-response distillation provides a competitive reference without public task labels or newcomer samples: gains over strengthened local training are 23.52 and 16.62 percentage points on 300 CelebA identities and 300 FedEMNIST writers, respectively. In a complete-pipeline FedEMNIST comparison, this reference reaches 97.13% accuracy with 12.4 seconds of compilation per architecture, versus 97.03% and 67.5 seconds for a HeteroFedTTA reconstruction at the same epoch budget. Equal-count public-input interventions extend support sensitivity to trainable visual backbones. Together, these results establish an auditable benchmark and a strong reference for evaluating the utility, cost and reliability of post-federation knowledge transfer.