Lessons from federated AI training across multiple research institutions
From Pilots to Production: Lessons in Cross-Institutional Federated Training and Artificial Intelligence for Science
Distributed, Parallel, and Cluster Computing
Summary
Some important scientific data can't be combined in one place because of rules or privacy concerns. This means that instead of moving data to a program, the program must move to the data, a process called federated AI. The authors looked at several recent projects where different groups used federated AI to train models safely and reliably. They share practical lessons about making this process dependable, secure, and manageable, and highlight challenges that still need solving. They suggest creating a global group to address these open problems together.
What this means in practice
- •For enterprise it teams: Deploy and maintain federated AI systems across different secure sites with reliable synchronization and privacy safeguards.
- •For cloud infrastructure providers: Support multi-tenant federated AI workloads that respect data sovereignty and enable trusted model training on distributed data.
A position paper. It proposes an approach and reports no results.
Authors
Olivera Kotevska, Max Carlson, Yan Gao, Francis Jeanson, Yijiang Li, William Lindskog, Mohammad Naseri, Minseok Ryu, Sahil Tyagi, Jerry Watkins, Feiyi Wang, Ravi Madduri, Kibaek Kim
Abstract
Many of the most valuable scientific datasets cannot be centralized: they are proprietary, export-controlled, classified, or bound by data-sovereignty restrictions. This inverts the usual paradigm: the model must move to the data, making federated artificial intelligence (AI) core infrastructure for open science. We synthesize lessons from U.S. Department of Energy national laboratories, industry deployments, and the open-source community into a structured account of moving federated AI from pilot demonstrations to dependable, multi-site production. The lessons come from five concurrent efforts spanning leadership-class supercomputers and cloud infrastructure in regulated settings. We organize them around an adapted \textit{five-domain readiness frame} and a stratified view of the stack beneath a trained model: data architecture, privacy and security controls, trust and verification, governance and socio-technical factors, and operations, the layers that decide whether a pilot becomes dependable. The question shifts from ``can we train it?'' to ``can we operate it, audit it, and change it safely?'' We report systems lessons in memory efficiency, reliability, and synchronization; set out what privacy, security, and decentralized trust require in production, and what is not yet validated there; and identify cross-cutting open problems (asynchronous federation, leakage auditing, verification standards, and harmonized data contracts) that we argue warrant a dedicated, international, open-science working group.