Conditional Generative Adversarial Net based Feature Extraction along with Scalable Weakly Supervised Clustering for Facial Expression Classification

Ze Chen; Lu Zhang; Jiaming Tang; Jiafa Mao; Weiguo Sheng

doi:10.53941/ijndi.2024.100024

Abstract

Extracting proper features plays a pivotal role in facial expression recognition. In this paper, we propose to extract facial expression features via a conditional generative adversarial net, followed by an algorithmic optimization step. These refined features are subsequently integrated into a scalable weakly supervised clustering framework for facial expression classification. Our results show that the proposed method can achieve an average recognition rate of 85.3%, which significantly outperforms related methods. Further, by employing a residual-based scheme for feature extraction, our method shows superior adaptability compared to algorithms based solely on weakly supervised clustering. Additionally, our method does not require high accurate annotation data and is robust to the noise presented in data sets.

References

1.
Chu, W.S.; De la Torre, F.; Cohn, J.F. Selective transfer machine for personalized facial action unit detection. In 2013 IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, 23–28 June 2013; IEEE: New York, 2013; pp. 3515–3572. doi: 10.1109/CVPR.2013.451
2.
Lucey, P.; Cohn, J.F.; Kanade, T.; et al. The extended Cohn-Kanade dataset (CK+): A complete dataset for action unit and emotion-speciﬁed expression. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition – Workshops, San Francisco, CA, USA, 13–18 June 2010; IEEE: New York, 2010; pp. 94–101. doi: 10.1109/CVPRW.2010.5543262
3.
Valstar, M.F.; Almaev, T.; Girard, J.M.; et al. FERA 2015 - second facial expression recognition and analysis challenge. In 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Ljubljana, Slovenia, 4–8 May 2015; IEEE: New York, 2015; pp. 1–8. doi: 10.1109/FG.2015.7284874
4.
McDuff, D.; El Kaliouby, R.; Senechal, T.; et al. Affectiva-MIT facial expression dataset (AM-FED): Naturalistic and spontaneous facial expressions collected “in-the-wild”. In 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops, Portland, OR, USA, 23–28 June 2013; IEEE: New York, 2013; pp. 881–888. doi: 10.1109/CVPRW.2013.130
5.
Xia, Y.F.; Yu, H.; Wang, X.; et al. Relation-aware facial expression recognition. IEEE Trans. Cogn. Dev. Syst., 2022, 14: 1143−1154. doi: 10.1109/TCDS.2021.3100131
6.
Mirza, M.; Osindero, S. Conditional generative adversarial nets. arXiv preprint arXiv: 1411.1784, 2014.
7.
Nigam, S.; Singh, R.; Misra, A.K. Efficient facial expression recognition using a histogram of oriented gradients in wavelet domain. Multimed. Tools Appl., 2018, 77: 28725−28747. doi: 10.1007/s11042-018-6040-3
8.
Ge, H.L.; Zhu, Z.Y.; Dai, Y.W.; et al. Facial expression recognition based on deep learning. Comput. Methods Programs Biomed., 2022, 215: 106621. doi: 10.1016/j.cmpb.2022.106621
9.
Zhang, H.F.; Su, W.; Yu, J.; et al. Identity–expression dual branch network for facial expression recognition. IEEE Trans. Cogn. Dev. Syst., 2021, 13: 898−911. doi: 10.1109/TCDS.2020.3034807
10.
Sun, W.Y.; Zhao, H.T.; Jin, Z. A visual attention based ROI detection method for facial expression recognition. Neurocomputing, 2018, 296: 12−22. doi: 10.1016/j.neucom.2018.03.034
11.
Michael Revina, I.; Sam Emmanuel, W.R. Facial expression recognition via modified GAD features with PSO-KNN. In 2018 International Conference on Smart Systems and Inventive Technology (ICSSIT), Tirunelveli, India, 13–14 December 2018; IEEE: New York, 2018; pp. 145–149. doi: 10.1109/ICSSIT.2018.8748697
12.
Kim, Y.; Yoo, B.; Kwak, Y.; et al. Deep generative-contrastive networks for facial expression recognition. arXiv preprint arXiv: 1703.07140, 2017.
13.
Zafeiriou, S.; Petrou, M. Sparse representations for facial expressions recognition via l1 optimization. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition – Workshops, San Francisco, CA, USA, 13–18 June 2010; IEEE: New York, 2010; pp. 32–39. doi: 10.1109/CVPRW.2010.5543148
14.
Bazzo, J.J.; Lamar, M.V. Recognizing facial actions using Gabor wavelets with neutral face average difference. In Proceedings of Sixth IEEE International Conference on Automatic Face and Gesture Recognition, Seoul, Korea (South), 19 May 2004; IEEE: New York, 2004; pp. 505–510. doi: 10.1109/AFGR.2004.1301583
15.
Rawal, N.; Stock-Homburg, R.M. Facial emotion expressions in human-robot interaction: A survey. Int. J. Soc. Robot., 2022, 14: 1583−1604. doi: 10.1007/S12369-022-00867-0
16.
Zhang, X.; Zhang, F.F.; Xu, C.S. Joint expression synthesis and representation learning for facial expression recognition. IEEE Trans. Circuits Syst. Video Technol., 2022, 32: 1681−1695. doi: 10.1109/TCSVT.2021.3056098
17.
Yaermaimaiti, Y.; Kari, T.; Zhuang, G.H. Research on facial expression recognition based on an improved fusion algorithm. Nonlinear Eng., 2022, 11: 112−122. doi: 10.1515/nleng-2022-0015
18.
Bao, H.; Ma, T. Feature extraction and facial expression recognition based on Bezier curve. In 2014 IEEE International Conference on Computer and Information Technology, Xi’an, China, 11–13 September 2014; IEEE: New York, 2014; pp. 884–887. doi: 10.1109/CIT.2014.140
19.
Ghahari, A.; Fatmehsari, Y.R.; Zoroofi, R.A. A novel clustering-based feature extraction method for an automatic facial expression analysis system. In Fifth International Conference on Intelligent Information Hiding and Multimedia Signal Processing, Kyoto, Japan, 12–14 September 2009; IEEE: New York, 2019; pp. 1314–1317. doi: 10.1109/IIH-MSP.2009.38
20.
Yang, H.Y.; Ciftci, U.; Yin, L.J. Facial expression recognition by de-expression residue learning. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; IEEE: New York, 2018; pp. 2168–2177. doi: 10.1109/CVPR.2018.00231
21.
Bilen, H.; Pedersoli, M.; Tuytelaars, T. Weakly supervised object detection with convex clustering. In 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; IEEE: New York, 2015; pp. 1081–1089. doi: 10.1109/CVPR.2015.7298711
22.
Prest, A.; Schmid, C.; Ferrari, V. Weakly supervised learning of interactions between humans and objects. IEEE Trans. Pattern Anal. Mach. Intell., 2012, 34: 601−614. doi: 10.1109/TPAMI.2011.158
23.
Bo, Y.; Fan, J.J.; Zhuang, J. Visual analysis of facial expression recognition research based on knowledge graph. In Proceedings of the 4th International Conference on Machine Learning for Cyber Security, Guangzhou, China, 2–4 December 2022; Springer: Berlin/Heidelberg, 2022; pp. 350–357. doi: 10.1007/978-3-031-20102-8_27
24.
Luo, Y.; Wu, J.X.; Zhang, Z.H.; et al. Design of facial expression recognition algorithm based on CNN model. In Proceedings of the 3rd IEEE International Conference on Power, Electronics and Computer Applications, Shenyang, China, 29–31 January 2023; IEEE: New York, 2023; pp. 580–583. doi: 10.1109/ICPECA56706.2023.10075779
25.
Shiomi, T.; Nomiya, H.; Hochin, T. Facial expression intensity estimation considering change characteristic of facial feature values for each facial expression. In Proceedings of the 23rd ACIS International Summer Virtual Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, Kyoto City, Japan, 4–7 July 2022; IEEE: New York, 2022; pp. 15–21. doi: 10.1109/SNPD-Summer57817.2022.00012
26.
Jin, X.F.; Liu, J.Y.; Yue, D. The research and improvement of facial expression recognition algorithm based on convolutional neural network. In Proceedings of the 26th ACIS International Winter Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, Taiyuan, China, 5–7 July 2023; IEEE: New York, 2023; pp. 166–170. doi: 10.1109/SNPD-Winter57765.2023.10224044
27.
Wang, L.; Li, R.F.; Wang, K. A novel automatic facial expression recognition method based on AAM. J. Comput., 2014, 9: 608−617.
28.
Zhao, K.L.; Chu, W.S.; Martinez, A.M. Learning facial action units from web images with scalable weakly supervised clustering. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; IEEE: New York, 2018; pp. 2090–2099. doi: 10.1109/CVPR.2018.00223
29.
Zhu, C.H.; Wen, F.; Sun, J. A rank-order distance based clustering algorithm for face tagging. In CVPR 2011, Colorado Springs, CO, USA, 20–25 June 2011; IEEE: New York, 2011; pp. 481–488. doi: 10.1109/CVPR.2011.5995680
30.
Newman, M.E.J. Modularity and community structure in networks. Proc. Natl. Acad. Sci. USA, 2006, 103: 8577−8582. doi: 10.1073/pnas.0601602103
31.
Lyons, M.; Akamatsu, S.; Kamachi, M.; et al. Coding facial expressions with Gabor wavelets. In Third IEEE International Conference on Automatic Face and Gesture Recognition, Nara, Japan, 14–16 April 1998; IEEE: New York, 1998; pp. 200–205. doi: 10.1109/AFGR.1998.670949
32.
Zhao, G.Y.; Huang, X.H.; Taini, M.; et al. Facial expression recognition from near-infrared videos. Image Vision Comput., 2011, 29: 607−619. doi: 10.1016/j.imavis.2011.07.002
33.
Melacci, S.; Belkin, M. Laplacian support vector machines trained in the primal. J. Mach. Learn. Res., 2011, 12: 1149−1184.
34.
Joachims, T. Transductive inference for text classification using support vector machines. In Proceedings of the Sixteenth International Conference on Machine Learning, Bled, Slovenia, 27–30 June 1999; Morgan Kaufmann Publishers Inc.: San Francisco, 1999; pp. 200–209.
35.
Guo, Y.M.; Zhao, G.Y.; Pietikäinen, M. Dynamic facial expression recognition using longitudinal facial expression atlases. In 12th European Conference on Computer Vision, Florence, Italy, 7–13 October 2012; Springer: Berlin/Heidelberg, 2012; pp. 631–644. doi: 10.1007/978-3-642-33709-3_45
36.
Zhao, G.Y.; Pietikainen, M. Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell., 2007, 29: 915−928. doi: 10.1109/TPAMI.2007.1110
37.
Liu, M.Y.; Shan, S.G; Wang, R.P.; et al. Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition. In Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 23–28 June 2014; IEEE: New York, 2014; pp. 1749–1756. doi: 10.1109/CVPR.2014.226
38.
Zhao, X.Y.; Liang, X.D.; Liu, L.Q.; et al. Peak-piloted deep network for facial expression recognition. In 14th European Conference on Computer Vision, The Netherlands, 11–14 October 2016; Springer: Berlin/Heidelberg, 2016; pp. 425–442. doi: 10.1007/978-3-319-46475-6_27
39.
Liu, S.G.; Li, L.J.; Peng, Y.L.; et al. Improved sparse representation method for image classification. IET Comput. Vision, 2017, 11: 319−330. doi: 10.1049/iet-cvi.2016.0186
40.
Liliana, D.Y.; Basaruddin, T. The fuzzy emotion recognition framework using semantic-linguistic facial features. In 2019 IEEE R10 Humanitarian Technology Conference, Depok, West Java, Indonesia, 12–14 November 2019; IEEE: New York, 2019; pp. 263–268. doi: 10.1109/R10-HTC47129.2019.9042442

Scilight Press

Author Information

Abstract

Keywords

References

About Scilight

Journals

Publishing Policies

Contact Us