Robust Rate-Splitting Design for Mixed Dual-Polarized Integrated Satellite-Terrestrial Networks Under Polarization Mismatch

Dual-polarized transmission offers a promising approach to improve spectral efficiency in multiantenna networks by reusing frequency and time resources across orthogonal polarization domains. Building upon this advantage, this paper investigates interference management in mixed dual-polarized integrated satellite-terrestrial networks (MDP-ISTN), comprising a circularly polarized (CP) satellite sub-network and a linearly polarized (LP) terrestrial sub-network. To this end, we employ rate-splitting multiple access (RSMA), which enables flexible non-orthogonal transmission through partial interference decoding and partial interference treating-as-noise. Specifically, to jointly mitigate both inter-network interference between the CP low Earth orbit (LEO) satellite and LP terrestrial sub-networks as well as intra-network interference within each sub-network, we propose an MDP-RSMA framework that incorporates inter-network rate-splitting (RS) with a super-common message together with intra-network RS. Moreover, we account for practical challenges in MDP-ISTN, including polarization mismatch, channel depolarization, and imperfect channel state information at the transmitter. To maximize the minimum user rate among all satellite and terrestrial users, we formulate a robust precoder optimization problem and develop a weighted minimum mean square error (WMMSE)-based algorithm tailored to the proposed MDP-RSMA. Numerical results demonstrate that the proposed scheme significantly improves the minimum user rate over several baseline schemes across diverse MDP-ISTN scenarios.