Task-Oriented Resource Allocation for Cross-Modal Semantic Communications

Multi-modal services are increasingly prevalent in sixth-generation (6G) networks and demand stringent ultra-reliable low-latency communication (uRLLC) requirements. Semantic communications, as an emerging communication paradigm, is able to extract and compress the same information from various modalities, breaking down data transmission barriers and enhancing the quality of multi-modal services. Nonetheless, ensuring accurate and reliable transmission of multi-modal streams remains a critical challenge in dynamic and resource-limited environments. To tackle this challenge, this article proposes a semantic-aware cross-modal resource allocation architecture that capitalizes on the inherent relations and semantic correlations between multi-modal streams to facilitate efficient semantic symbol processing and stream transmission. First, a semantic importance-aware transmission scheme is developed to investigate the temporal, spatial, and semantic correlations of cross-modal streams within tasks. This approach enables flexible arrangement of transmission sequences according to task priorities, thereby reducing the volume of transmitted data. Second, a joint deep reinforcement learning (JDRL) methodology is employed to optimize the multi-modal stream transmission. Multi-agents are designed to jointly process continuous and discrete streams, leveraging cross-modal correlation to improve overall transmission rate while ensuring quality of the transmitted information. Experimental results demonstrate that the proposed resource allocation architecture exhibits exceptional reliability and efficiency.