Scheduling and Fusion for Multimodal Federated Learning in Energy-constrained Wireless Networks

The rise of privacy-preserving applications, such as medical diagnostics and the Metaverse, highlights the importance of federated learning (FL) for distributed model training at the wireless edge. These applications often rely on multimodal data (e.g., text, images, audio), necessitating advances in multimodal federated learning (MMFL). However, MMFL faces challenges like energy efficiency, multimodal fusion, and heterogeneity. To address these, a scheduling and fusion-based MMFL framework (SFMMFL) is proposed that focuses on improving both the scheduling mechanism and aggregation strategy. To improve the training performance under energy constraint, a Lyapunov-based scheduling algorithm is proposed, in which long-term optimization is transformed into immediate optimization. After that, to tackle the issue of model separation caused by multimodal datasets, a multimodal model aggregation strategy based on Knowledge Distillation (KD) is introduced for multimodal fusion. Convergence analysis proves its feasibility, and simulation results demonstrate that it can achieve faster and more stable convergence performance while improving model training accuracy. Specifically, our proposed SFMMFL can lower the energy consumption of the system by about 20% for computing and 16.67% for transmission.