Multi-Modality ViT-cGAN for Low-Dose PET Image Reconstruction

Yaling Fang; Jiahui Yang; Haoran Wan; Shuhan Jin; Shaoya Wang; Yueyang Teng

doi:10.53941/aim.2026.100001

Abstract

Positron emission tomography (PET) is an important medical imaging technique that reflects the molecular activity of tissues and organs by injecting radioactive tracers. Low-dose (LD) PET is gradually being adopted to reduce radiation dose and scanning costs, however this usually leads to increased image noise and artifacts, which can affect clinical diagnosis. Therefore, in order to maintain high-quality PET image generation while utilizing LD-PET data, this paper proposes a multi-modality Vision Transformer-based conditional generative adversarial network (ViT-cGAN) that directly achieves high-quality PET image reconstruction using the corresponding LD-PET sinogram data and computed tomography (CT) images. Specifically, the network incorporates the advantages of Vision Transformer and multi-modality inputs. In addition, an extensive objective function is designed to optimize the network for improving the details and visual quality of the reconstructed images. Experimental results show that our proposed method can effectively reconstruct high-quality PET images, outperforming current state-of-the-art methods.

References

1.
Chen, W. Clinical applications of PET in brain tumors. J. Nucl. Med. 2007, 48, 1468–1481.
2.
Wang, Y.; Zhang, P.; An, L.; et al. Predicting standard-dose PET image from low-dose PET and multimodal MR images using mapping-based sparse representation. Phys. Med. Biol. 2016, 61, 791.
3.
Wang, Y.; Ma, G.; An, L.; et al. Semisupervised tripled dictionary learning for standard-dose PET image prediction using low-dose PET and multimodal MRI. IEEE Trans. Biomed. Eng. 2016, 64, 569–579.
4.
Tang, X.; Ning, R. A cone beam filtered backprojection (CB-FBP) reconstruction algorithm for a circle-plus-two-arc orbit. Med. Phys. 2001, 28, 1042–1055.
5.
Tao, X.; Zhang, H.; Wang, Y.; et al. VVBP-tensor in the FBP algorithm: Its properties and application in low-dose CT reconstruction. IEEE Trans. Med. Imaging 2019, 39, 764–776.
6.
Balda, M.; Hornegger, J.; Heismann, B. Ray contribution masks for structure adaptive sinogram filtering. IEEE Trans. Med. Imaging 2012, 31, 1228–1239.
7.
Manduca, A.; Yu, L.; Trzasko, J.D.; et al. Projection space denoising with bilateral filtering and CT noise modeling for dose reduction in CT. Med. Phys. 2009, 36, 4911–4919.
8.
Dutta, J.; Leahy, R.M.; Li, Q. Non-Local Means Denoising of Dynamic PET Images. PLoS ONE 2013, 8, e81390.
9.
Cong, Y.; Zhang, S.; Lian, Y. K-SVD Dictionary Learning and Image Reconstruction Based on Variance of Image Patches. In Proceedings of the 2015 8th International Symposium on Computational Intelligence and Design (ISCID), Hangzhou, China, 12–13 December 2015.
10.
Yu, X.; Wang, C.; Hu, H.; et al. Low Dose PET Image Reconstruction with Total Variation Using Alternating Direction Method. PLoS ONE 2016, 11, e0166871.
11.
Buades, A.; Coll, B.; Morel, J.M. A Non-Local Algorithm for Image Denoising. In Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), San Diego, CA, USA, 20–25 June 2005.
12.
Zhang, H.; Ma, J.;Wang, J.; et al. Statistical image reconstruction for lowdose CT using nonlocal means-based regularization. Part II: An adaptive approach. Comput. Med. Imaging Graph. 2015, 43, 26–35.
13.
Fumene Feruglio, P.; Vinegoni, C.; Gros, J.; et al. Block matching 3D random noise filtering for absorption optical projection tomography. Phys. MedBiol. 2010, 55, 5401–5415.
14.
Moeskops, P.; Viergever, M.A.; Mendrik, A.M.; et al. Automatic segmentation of MR brain images with a convolutional neural network. IEEE Trans. Med. Imaging 2016, 35, 1252–1261.
15.
Chen, H.; Zhang, Y.; Zhang,W.; et al. Low-dose CT via convolutional neural network. Biomed. Opt. Express 2017, 8, 679–694.
16.
Gong, K.; Guan, J.; Liu, C.C.; et al. PET image denoising using a deep neural network through fine tuning. IEEE Trans. Radiat. Plasma Med. Sci. 2018, 3, 153–161.
17.
Lei, Y.; Harms, J.; Wang, T.; et al. MRI-only based synthetic CT generation using dense cycle consistent generative adversarial networks. Med. Phys. 2019, 46, 3565–3581.
18.
Gong, K.; Guan, J.; Kyungsang, K.; et al. Iterative PET Image Reconstruction Using Convolutional Neural Network Representation. IEEE Trans. Med. Imaging 2019, 38, 675–685.
19.
H¨aggstrom, I.; Schmidtlein, C.R.; Campanella, G.; et al. DeepPET: A deep encoder-decoder network for directly solving the PET image reconstruction inverse problem. Med. Image Anal. 2019, 54, 253–262.
20.
Spuhler, K.; Serrano-Sosa, M.; Cattell, R.; et al. Full-count PET recovery from low-count image using a dilated convolutional neural network. Med. Phys. 2020, 47, 4928–4938.
21.
Wang, Y.; Zhou, L.; Wang, L.; et al. Locality Adaptive Multi-Modality GANs for High-Quality PET Image Synthesis. In Proceedings of the Medical Image Computing and Computer Assisted Intervention—MICCAI 2018: 21st International Conference, Granada, Spain, 16–20 September 2018.
22.
Xue, H.; Zhang, Q.; Zou, S.; et al. LCPR-Net: Low-count PET image reconstruction using the domain transform and cycle-consistent generative adversarial networks. Quant. Imaging Med. Surg. 2021, 11, 749.
23.
Luo, Y.; Zhou, L.; Zhan, B.; et al. Adaptive rectification based adversarial network with spectrum constraint for high-quality PET image synthesis. Med. Image Anal. 2022, 77, 102335.
24.
Fei, Y.; Zu, C.; Jiao, Z.; et al. Classification-aided high-quality PET image synthesis via bidirectional contrastive GAN with shared information maximization. In International Conference on Medical Image Computing and Computer-Assisted Intervention; Springer Nature: Cham, Switzerland, 2022.
25.
Wang, Y.; Yu, B.; Wang, L.; et al. 3D conditional generative adversarial networks for high-quality PET image estimation at low dose. Neuroimage 2018, 174, 550–562.
26.
Liu, Z.; Ye, H.; Liu, H. Deep-learning-based framework for PET image reconstruction from sinogram domain. Appl. Sci. 2022, 12, 8118.
27.
Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; et al. An image is worth 16x16 words: Transformers for image recognition at scale. arxiv 2020, arxiv:2010.11929.
28.
Zhang, Z.; Yu, L.; Liang, X.; et al. TransCT: Dual-path transformer for low dose computed tomography. In Proceedings of the Medical Image Computing and Computer Assisted Intervention—MICCAI 2021: 24th International Conference, Strasbourg, France, 27 September–1 October 2021.
29.
Zheng, H.; Lin, Z.; Zhou, Q.; et al. Multi-transsp: Multimodal transformer for survival prediction of nasopharyngeal carcinoma patients. In International Conference on Medical Image Computing and Computer-Assisted Intervention; Springer Nature: Cham, Switzerland, 2022.
30.
De Wever, W.; Ceyssens, S.; Mortelmans, L.; et al. Additional value of PET-CT in the staging of lung cancer: Comparison with CT alone, PET alone and visual correlation of PET and CT. Eur. Radiol. 2007, 17, 23–32.
31.
Fletcher, J.W.; Kymes, S.M.; Gould, M.; et al. A comparison of the diagnostic accuracy of 18F-FDG PET and CT in the characterization of solitary pulmonary nodules. J. Nucl. Med. 2008, 49, 179–185.
32.
Isola, P.; Zhu, J.Y.; Zhou, T.; et al. Image-to-Image Translation with Conditional Adversarial Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 1125–1134.
33.
Johnson, J.; Alahi, A.; Fei-Fei, L. Perceptual losses for real-time style transfer and super-resolution. In Proceedings of the Computer Vision—ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, 11–14 October 2016.
34.
Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv 2014, arXiv:1409.1556.
35.
Bakr, S.; Gevaert, O.; Echegaray, S.; et al. A radiogenomic dataset of non-small cell lung cancer. Sci. Data 2018, 5, 180202. https://doi.org/10.1038/sdata.2018.202.
36.
Hore, A.; Ziou, D. Image Quality Metrics: PSNR vs. SSIM. In Proceedings of the 2010 20th International Conference on Pattern Recognition, Istanbul, Turkey, 23–26 August 2010.

Scilight Press

Author Information

Abstract

Keywords

References

About Scilight

Journals

Publishing Policies

Contact Us