Multimodal Deep Learning based Automatic Modulation Recognition: Fusion of Signal Modalities

Xin Lin; Qinggeng Guo; Xi Yang; Ping Wu; Shengliang Peng

doi:10.53941/aieng.2026.100005

Abstract

Automatic modulation recognition (AMR) has been becoming an indispensable part in intelligent communications systems, especially for cognitive radio and radio regulation. With the fast development of machine learning in the recent years, deep learning (DL) has been applied to AMR. However, existing DL based AMR methods only rely on a single signal modality, which limits the full utilization of signal features and restricts recognition performance. Thus, this paper proposes to develop multimodal DL based AMR, which is established, in particular, by exploring a variety of modalities to represent the received signals (e.g., in-phase and quadrature sequences, constellation diagram), and then fusing two or more of the modalities at three different stages of the DL architecture. The three stages of the multimodal fusion include the early fusion, intermediate fusion and late fusion in which the multiple signal modalities are fused before, in between and after the DL model, respectively. The algorithms for the three fusion methods are proposed and implemented in experiments. Evaluation of the algorithms is made according to accuracy, complexity and flexibility. The results show that (1) the early fusion exhibits satisfactory classification accuracy, least complexity and not good flexibility; (2) the intermediate fusion gives best accuracy, high complexity and satisfactory classification flexibility; and (3) the late fusion gives low accuracy, high complexity and best flexibility. Moreover, the proposed multimodal DL based AMR algorithms consistently outperform single signal modality approaches under tested conditions, including different DL model structures, sample quantities, and channel models, demonstrating strong generality and universal superiority for automatic modulation recognition.

References

1.
Pateromichelakis, E.; Bulakci, O.; Peng, C.; et al. LAA as a Key Enabler in Slice-Aware 5G RAN: Challenges and Opportunities. IEEE Commun. Stand. Mag. 2018, 2, 29–35.
2.
Ya, T.; Yun, L.; Haoran, Z.; et al. Large-Scale Real-World Radio Signal Recognition with Deep Learning. Chin. J. Aeronaut. 2022, 35, 35–48.
3.
Zou, Y.; Yao, Y.D.; Zheng, B. Cooperative Relay Techniques for Cognitive Radio Systems: Spectrum Sensing and Secondary User Transmissions. IEEE Commun. Mag. 2012, 50, 98–103.
4.
Xin, C.; Song, M. Analysis of the On-Demand Spectrum Access Architecture for CBRS Cognitive Radio Networks. IEEE Trans. Wireless Commun. 2019, 19, 970–978.
5.
Wang, Y.; Wang, J.; Zhang, W.; et al. Deep Learning-Based Cooperative Automatic Modulation Classification Method for MIMO Systems. IEEE Trans. Veh. Technol. 2020, 69, 4575–4579.
6.
Zhang, H.; Farzanullah, M.; Sediq, A.B.; et al. Supervised Contrastive Learning for Uncertainty-Aware Wireless Signal Analysis: A Case Study for Modulation Classification. IEEE Trans. Commun. 2026, 74, 7227–7240.
7.
Wang, G.; Liu, Z.; Zhang, X.; et al. PID: A Parameter-Efficient Isolation Domain-Incremental Learning Framework for Signal Modulation Classification. IEEE Trans. Neural Netw. Learn. Syst. 2026, 37, 1449–1462.
8.
Shi, J.; Hong, S.; Cai, C.; et al. Deep Learning-Based Automatic Modulation Recognition Method in the Presence of Phase Offset. IEEE Access 2020, 8, 42841–42847.
9.
Tunze, G.B.; Huynh-The, T.; Lee, J.M.; et al. Sparsely Connected CNN for Efficient Automatic Modulation Recognition. IEEE Trans. Veh. Technol. 2020, 69, 15557–15568.
10.
Li, L.; Huang, J.; Cheng, Q.; et al. Automatic Modulation Recognition: A Few-Shot Learning Method Based on the Capsule Network. IEEE Wireless Commun. Lett. 2020, 10, 474–477.
11.
Li, R.; Li, L.; Yang, S.; et al. Robust Automated VHF Modulation Recognition Based on Deep Convolutional Neural Networks. IEEE Commun. Lett. 2018, 22, 946–949.
12.
Huang, L.; Pan, W.; Zhang, Y.; et al. Data Augmentation for Deep Learning-Based Radio Modulation Classification. IEEE Access 2019, 8, 1498–1506.
13.
Zhang, F.; Luo, C.; Xu, J.; et al. An Efficient Deep Learning Model for Automatic Modulation Recognition Based on Parameter Estimation and Transformation. IEEE Commun. Lett. 2021, 25, 3287–3290.
14.
Chen, Y.; Dong, B.; Liu, C.; et al. Abandon Locality: Frame-Wise Embedding Aided Transformer for Automatic Modulation Recognition. IEEE Commun. Lett. 2022, 27, 327–331.
15.
Ramachandram, D.; Taylor, G.W. Deep Multimodal Learning: A Survey on Recent Advances and Trends. IEEE Signal Process. Mag. 2017, 34, 96–108.
16.
Liang, M.; Li, Z.; Chen, T.; et al. Integrative Data Analysis of Multi-Platform Cancer Data with a Multimodal Deep Learning Approach. IEEE/ACM Trans. Comput. Biol. Bioinform. 2014, 12, 928–937.
17.
Valada, A.; Oliveira, G.L.; Brox, T.; et al. Deep Multispectral Semantic Scene Understanding of Forested Environments Using Multimodal Fusion. In Proceedings of the 2016 International Symposium on Experimental Robotics, Nagasaki, Japan, 3–8 October 2016; pp. 465–477.
18.
Simonyan, K.; Zisserman, A. Two-Stream Convolutional Networks for Action Recognition in Videos. In Proceedings of the 28th International Conference on Neural Information Processing Systems, Montreal, QC, Canada, 8–13 December 2014; pp. 568–576.
19.
Jain, A.; Singh, A.; Koppula, H.S.; et al. Recurrent Neural Networks for Driver Activity Anticipation via Sensory-Fusion Architecture. In Proceedings of the 2016 IEEE International Conference on Robotics and Automation (ICRA), Stockholm, Sweden, 16–21 May 2016; pp. 3118–3125.
20.
Peng, S.; Sun, S.; Yao, Y.D. A Survey of Modulation Classification Using Deep Learning: Signal Representation and Data Preprocessing. IEEE Trans. Neural Netw. Learn. Syst. 2021, 33, 7020–7038.
21.
Khan, F.N.; Zhong, K.; Al-Arashi,W.H.; et al. Modulation Format Identification in Coherent Receivers Using Deep Machine Learning. IEEE Photonics Technol. Lett. 2016, 28, 1886–1889.
22.
Huang, J.; Yi, A.; Liao, P.; et al. Modulation Format Classification of Probabilistically Shaped M-QAM Signals Based on Nonlinear Power Transformation. In Proceedings of the 2022 Asia Communications and Photonics Conference (ACP), Shenzhen, China, 5–8 November 2022; pp. 782–785.
23.
Peng, S.; Jiang, H.; Wang, H.; et al. Modulation Classification Based on Signal Constellation Diagrams and Deep Learning. IEEE Trans. Neural Netw. Learn. Syst. 2018, 30, 718–727.
24.
Lin, Y.; Tu, Y.; Dou, Z.; et al. Contour Stella Image and Deep Learning for Signal Recognition in the Physical Layer. IEEE Trans. Cogn. Commun. Netw. 2020, 7, 34–46.
25.
Wang, D.; Zhang, M.; Li, Z.; et al. Modulation Format Recognition and OSNR Estimation Using CNN-Based Deep Learning. IEEE Photonics Technol. Lett. 2017, 29, 1667–1670.
26.
Wang, Y.; Liu, M.; Yang, J.; et al. Data-Driven Deep Learning for Automatic Modulation Recognition in Cognitive Radios. IEEE Trans. Veh. Technol. 2019, 68, 4074–4077.
27.
Qi, P.; Zhou, X.; Zheng, S.; et al. Automatic Modulation Classification Based on Deep Residual Networks with Multimodal Information. IEEE Trans. Cogn. Commun. Netw. 2020, 7, 21–33.
28.
Changbo, H.; Guowei, L.; Lijie, H.; et al. Multimodal Feature Fusion Recognition of Modulated Signals Based on Image and Waveform Domain. In Proceedings of the 2020 7th International Conference on Dependable Systems and Their Applications (DSA), Xi’an, China, 28–29 November 2020; pp. 337–342.
29.
Deng, W.; Wang, X.; Huang, Z.; et al. Modulation Classifier: A Few-Shot Learning Semi-Supervised Method Based on Multimodal Information and Domain Adversarial Network. IEEE Commun. Lett. 2022, 27, 576–580.
30.
Zhou, H.; Zhou, Z.; Bai, J. Electromagnetic Signal Modulation Classification Based on Multimodal Features and Reinforcement Learning. In Proceedings of the 2022 International Joint Conference on Neural Networks (IJCNN), Padua, Italy, 18–23 July 2022; pp. 1–7.
31.
Zheng, S.; Zhou, X.; Zhang, L.; et al. Toward Next-Generation Signal Intelligence: A Hybrid Knowledge and Data-Driven Deep Learning Framework for Radio Signal Classification. IEEE Trans. Cogn. Commun. Netw. 2023, 9, 564–579.
32.
Xu, J.; Luo, C.; Parr, G.; et al. A Spatiotemporal Multi-Channel Learning Framework for Automatic Modulation Recognition. IEEE Wireless Commun. Lett. 2020, 9, 1629–1632.
33.
O’Shea, T.J.; Roy, T.; Clancy, T.C. Over-the-Air Deep Learning Based Radio Signal Classification. IEEE J. Sel. Top. Signal Process. 2018, 12, 168–179.
34.
O’Shea, T.; Hoydis, J. An Introduction to Deep Learning for the Physical Layer. IEEE Trans. Cogn. Commun. Netw. 2017, 3, 563–575.
35.
He, K.; Zhang, X.; Ren, S.; et al. Deep Residual Learning for Image Recognition. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778.
36.
Zhang, F.; Luo, C.; Xu, J.; et al. Deep Learning Based Automatic Modulation Recognition: Models, Datasets, and Challenges. Digit. Signal Process. 2022, 129, 103650.

Scilight Press

Author Information

Abstract

Keywords

References

About Scilight

Journals

Publishing Policies

Contact Us