Background Replacement in Video Conferencing

Kiran Shahi; Yongmin Li

doi:10.53941/ijndi.2023.100004

Abstract

Background replacement is one of the most used features in video conferencing applications by many people, perhaps mainly for privacy protection, but also for other purposes such as branding, marketing and promoting professionalism. However, the existing applications in video conference tools have serious limitations. Most applications tend to generate strong artefacts (while there is a slight change in the perspective of the background), or require green screens to avoid such artefacts, which results in an unnatural background or even exposes the original background to other users in the video conference. In this work, we aim to study the relationship between the foreground and background in real-time videos. Three different methods are presented and evaluated, including the baseline U-Net, the lightweight U-Net MobileNet, and the U-Net MobileNet&ConvLSTM models. The above models are trained on public datasets for image segmentation. Experimental results show that both the lightweight U-Net MobileNet and the U-Net MobileNet& ConvLSTM models achieve superior performance as compared to the baseline U-Net model.

References

1.
Beyer, W. Traveling-matte photography and the blue-screen system: A tutorial paper. J. SMPTE 1965, 74, 217–239. doi: 10.5594/J06054.
2.
Porter, T.; Duff, T. Compositing digital images. ACM SIGGRAPH Comput. Graph. 1984, 18, 253–259. doi: 10.1145/964965.808606.
3.
Rhemann, C.; Rother, C.; Rav-Acha, A.; et al. High resolution matting via interactive trimap segmentation. In Proceedings of 2008 IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, USA, 23–28 June 2008; IEEE: New York, 2008; pp. 1–8. doi: 10.1109/CVPR.2008.4587441.
4.
Boda, J.; Pandya, D. A survey on image matting techniques. In Proceedings of 2018 International Conference on Communication and Signal Processing (ICCSP), Chennai, India, 3–5 April 2018; IEEE: New York, 2018; pp. 765–770. doi: 10.1109/ICCSP.2018.8523834.
5.
Xu, N.; Price, B.; Cohen, S.; et al. Deep image matting. In Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, USA, 21–26 July 2017; IEEE: New York, 2017, pp. 311–320. doi: 10.1109/CVPR.2017.41.
6.
Forte, M.; Pitié, F. F, B, alpha matting. arXiv: 2003.07711, 2020.
7.
Sengupta, S.; Jayaram, V.; Curless, B.; et al. Background matting: The world is your green screen. In Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, USA, 13–19 June 2020; IEEE: New York, 2020; pp. 2288–2297. doi: 10.1109/CVPR42600.2020.00236.
8.
Lin, S.C.; Ryabtsev, A.; Sengupta, S.; et al. Real-time high-resolution background matting. In Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, USA, 20–25 June 2021; IEEE: New York, 2021; pp. 8758–8767. doi: 10.1109/CVPR46437.2021.00865.
9.
Zhu, Q.S.; Heng, P.A.; Shao, L.; et al. What’s the role of image matting in image segmentation? In Proceedings of 2013 IEEE International Conference on Robotics and Biomimetics (ROBIO), Shenzhen, China, 12–14 December 2013; IEEE: New York, 2013; pp. 1695–1698. doi: 10.1109/ROBIO.2013.6739711.
10.
Ronneberger, O.; Fischer, P.; Brox, T. U-Net: Convolutional networks for biomedical image segmentation. In Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany, 5–9 October 2015; Springer: Berlin/Heidelberg, Germany, 2015; pp. 234–241. doi: 10.1007/978-3-319-24574-4_28.
11.
Xie, H.X.; Lin, C.Y.; Zheng, H.; et al. An UNet-based head shoulder segmentation network. In Proceedings of 2018 IEEE International Conference on Consumer Electronics-Taiwan (ICCE-TW), Taichung, China, 19–21 May 2018; IEEE: New York, 2018; pp. 1–2. doi: 10.1109/ICCE-China.2018.8448587.
12.
Kuang, Z.J.; Tie, X.R. Flow-based video segmentation for human head and shoulders. arXiv: 2104.09752, 2021.
13.
Zhang, Z.X.; Liu, Q.J.; Wang, Y.H. Road extraction by deep residual U-Net. IEEE Geosci. Remote Sens. Lett. 2018, 15, 749–753. doi: 10.1109/LGRS.2018.2802944.
14.
Liang, Z.Y.; Guo, K.; Li, X.B.; et al. Person foreground segmentation by learning multi-domain networks. IEEE Trans. Image Process., 2022, 31: 585−597.
15.
Miao, J.; Sun, K.Q.; Liao, X.; et al. Human segmentation based on compressed deep convolutional neural network. IEEE Access, 2020, 8: 167585−167595.
16.
Zhang, S.H.; Dong, X.; Li, H.; et al. PortraitNet: Real-time portrait segmentation network for mobile device. Comput. Graph., 2019, 80: 104−113.
17.
Shi, X.J.; Chen, Z.R.; Wang, H.; et al. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In Proceedings of the 28th International Conference on Neural Information Processing Systems, Montreal Canada, 7–2 December 2015; MIT Press: Cambridge, 2015; pp. 802–810.
18.
Ballas, N.; Yao, L.; Pal, C.; et al. Delving deeper into convolutional networks for learning video representations. In Proceedings of the 4th International Conference on Learning Representations, San Juan, Puerto Rico, 2–4 May 2016; 2016.
19.
Lin, S.C.; Yang, L.J.; Saleemi, I.; et al. Robust high-resolution video matting with temporal guidance. In Proceedings of 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, USA, 3–8 January 2022; IEEE: New York, 2022; pp. 3132–3141. doi: 10.1109/WACV51458.2022.00319.
20.
Azad, R.; Asadi-Aghbolaghi, M.; Fathy, M.; et al. Bi-directional convLSTM U-Net with densley connected convolutions. In Proceedings of 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), Seoul, Korea (South), 27–28 October 2019; IEEE: New York, 2019; pp. 406–415. doi: 10.1109/ICCVW.2019.00052.
21.
Tomar, N. Person segmentation dataset. 2020. Available online: https://www.kaggle.com/datasets/nikhilroxtomar/personsegmentation (accessed on 10 March 2023).
22.
Jordan, J. An overview of semantic image segmentation. 2018. Available online: https://www.jeremyjordan.me/semantic-segmentation/ (accessed on 8 March 2023).
23.
Kandeyang, V.K. Image segmentation: Cross-entropy loss vs dice loss. 2020. Available online: https://www.kaggle.com/getting-started/133156/ (accessed on 6 March 2023).

Scilight Press

Author Information

Abstract

Graphical Abstract

Keywords

References

About Scilight

Journals

Publishing Policies

Contact Us