Federated learning improves multimodal rgb tracking while respecting privacy
Fed-MAPTrack: Federated Modality-aware Prompt Tuning for RGB-X Tracking
Computer Vision and Pattern Recognition
Summary
Tracking objects in video can use extra sensors like thermal, depth, or event cameras to help, but gathering all this data in one place is hard and raises privacy issues. The authors introduce a method called Fed-MAPTrack that lets different devices train a tracking model together without sharing raw video data. This method separates shared parts of the model for RGB data from those specific to extra sensors, making sure devices without extra sensors can still help train the system. Their method improves tracking accuracy on multiple datasets and shows how carefully sharing model parts and selecting contributors helps make federated multimodal training more effective.
What this means in practice
- •For autonomous vehicle developers: Improve object tracking by combining RGB and thermal or depth sensors while keeping data decentralized and respecting privacy.
- •For surveillance system engineers: Enhance tracking accuracy using heterogeneous sensors across devices without centralizing sensitive video data.
Authors
Xiangyu Liu, Yijie Shen, Yue Yin, Zhitong Yin, Yongqi Pan, Wentao Fan
Abstract
RGB-X tracking complements RGB appearance with thermal, depth, or event observations, but centralized training requires collecting multimodal videos at a single location. Federated training keeps raw videos local, but sensor heterogeneity can create a mismatch between parameter-sharing domains and the clients contributing to them. This mismatch is particularly relevant when RGB-only clients coexist with sensor-equipped clients: their RGB observations remain useful, but they provide no auxiliary measurements for learning sensor-dependent interactions. We propose Fed-MAPTrack, a modality-aware federated prompt tuning framework for RGB-X tracking. The pretrained tracking backbone and prediction head are frozen, and clients optimize and exchange only prompt-related parameters. The server aggregates RGB-shared parameters globally and maintains separate thermal, event, and depth banks for auxiliary-specific, fusion, and normalization parameters. Selective participation retains RGB-only clients' contributions to the global RGB bank while excluding their fusion and normalization updates from modality-specific aggregation. Across three matched training repetitions under a fixed client partition, Fed-MAPTrack improves the mean FedAvg scores by 2.35 AUC points on LasHeR, 3.45 SR points on VisEvent, and 2.11 F-score points on DepthTrack. Among the nine evaluated methods, it achieves the highest mean thermal AUC and depth F-score, while its mean event SR is 0.07 points below that of Independent FL. Controlled ablations show that extending the bucketing scope without a compatible participation rule can reduce performance. These results highlight the need to jointly define the sharing domain and eligible contributors for each parameter group.