Abstract
Emotions expressed on a human face have a significant impact on decisions and arguments on a variety of topics. According to psychological theory, a person’s emotional states can be categorized as the afraid, disgusted, angry, sad, happy, neutral face and surprised. The automatic extraction of these emotions from images of human faces can help in human–computer interaction, among other things. Convolution Neural Network (CNN), Deep Belief Network (DBN), Bi-directional Long Short Term Memory (Bi-LSTM) are some of the existing techniques used to recognize the emotions of a human. This technique has some impacts like low accuracy and high error. To achieve better accuracy, hybrid CNN-SVM (Support Vector Machine) model is designed for classifying emotional state of humans. Initially, preprocessing is used to remove unwanted things from the image dataset. Resizing, Gaussian filter, Median filter, Histogram Equalization and Wiener filters are used in the preprocessing stage. After that, Region of Interest of the preprocessed image is extracted. Then features of the images are extracted based on Local Binary Pattern and Gabor feature technique. These obtained features are fused using the feature fusion process. The fused image data is fed to a hybrid CNN-SVM classifier. The hybrid CNN-SVM classifies the different emotional states of humans. The proposed method achieves an accuracy of 94% for CK_Plus, 86% FER_2013, 78% for KDEF, 96% for KMU_FED and 85% for the TFEID dataset. Thus the proposed human emotion recognition using the CNN-SVM approach produced optimal solutions compared to the existing systems.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data Availability
Not applicable.
References
Padhmashree, V., & Bhattacharyya, A. (2022). Human emotion recognition based on time–frequency analysis of multivariate EEG signal. Knowledge-Based Systems, 238, 107867.
Jiang, D., Wu, K., Chen, D., Tu, G., Zhou, T., Garg, A., & Gao, L. (2020). A probability and integrated learning based classification algorithm for high-level human emotion recognition problems. Measurement, 150, 107049.
Jain, D. K., Shamsolmoali, P., & Sehdev, P. (2019). Extended deep neural network for facial emotion recognition. Pattern Recognition Letters, 120, 69–74.
Karthick, S., & Maniraj, S. (2019). Different medical image registration techniques: A comparative analysis. Current Medical Imaging Formerly Current Medical Imaging Reviews, 15(10), 911–921. https://doi.org/10.2174/1573405614666180905094032
Chowdary, M. K., Nguyen, T. N., & Hemanth, D. J. (2021). Deep learning-based facial emotion recognition for human–computer interaction applications. Neural Computing and Applications, 1–18. https://doi.org/10.1007/s00521-021-06012-8
Cimtay, Y., Ekmekcioglu, E., & Caglar-Ozhan, S. (2020). Cross-subject multimodal emotion recognition based on hybrid fusion. IEEE Access, 8, 168865–168878.
Pal, S., Mukhopadhyay, S., & Suryadevara, N. (2021). Development and progress in sensors and technologies for human emotion recognition. Sensors, 21(16), 5554.
Arunnehru, J., & Kalaiselvi Geetha, M. (2017). Automatic human emotion recognition in surveillance video. Springer, Cham: In Intelligent Techniques in Signal Processing for Multimedia Security.
Zhang, Y., Du, J., Wang, Z., Zhang, J., & Tu, Y. (2018). Attention based fully convolutional network for speech emotion recognition. In 2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) (pp. 1771–1775). IEEE.
Gupta, V., Chopda, M. D., & Pachori, R. B. (2018). Cross-subject emotion recognition using flexible analytic wavelet transform from EEG signals. IEEE Sensors Journal, 19(6), 2266–2274.
Egger, M., Ley, M., & Hanke, S. (2019). Emotion recognition from physiological signal analysis: A review. Electronic Notes in Theoretical Computer Science, 343, 35–55.
Bhattacharyya, A., Tripathy, R. K., Garg, L., & Pachori, R. B. (2020). A novel multivariate-multiscale approach for computing EEG spectral and temporal complexity for human emotion recognition. IEEE Sensors Journal, 21(3), 3579–3591.
Liang, Z., Oba, S., & Ishii, S. (2019). An unsupervised EEG decoding system for human emotion recognition. Neural Networks, 116, 257–268.
Liu, Y., & Fu, G. (2021). Emotion recognition by deeply learned multi-channel textual and EEG features. Future Generation Computer Systems, 119, 1–6.
Jerritta, S., Murugappan, M., Nagarajan, R., & Wan, K. (2011). Physiological signals based human emotion recognition: a review. In 2011 IEEE 7th international colloquium on signal processing and its applications (pp. 410–415). IEEE.
Batbaatar, E., Li, M., & Ryu, K. H. (2019). Semantic-emotion neural network for emotion recognition from text. IEEE Access, 7, 111866–111878.
Hassan, M. M., Alam, M. G. R., Uddin, M. Z., Huda, S., Almogren, A., & Fortino, G. (2019). Human emotion recognition using deep belief network architecture. Information Fusion, 51, 10–18.
Hossain, M. S., & Muhammad, G. (2019). Emotion recognition using deep learning approach from audio–visual emotional big data. Information Fusion, 49, 69–78.
Meng, H., Yan, T., Yuan, F., & Wei, H. (2019). Speech emotion recognition from 3D log-mel spectrograms with deep learning network. IEEE access, 7, 125868–125881.
Bhatti, A. M., Majid, M., Anwar, S. M., & Khan, B. (2016). Human emotion recognition and analysis in response to audio music using brain signals. Computers in Human Behavior, 65, 267–275.
Rahman, Z., Pu, Y. F., Aamir, M., & Ullah, F. (2019). A framework for fast automatic image cropping based on deep saliency map detection and Gaussian filter. International Journal of Computers and Applications, 41(3), 207–217.
Shah, A., Bangash, J. I., Khan, A. W., Ahmed, I., Khan, A., Khan, A., & Khan, A. (2020). Comparative analysis of median filter and its variants for removal of impulse noise from gray scale images. Journal of King Saud University-Computer and Information Sciences, 34(3), 505.
Rao, B. S. (2020). Dynamic histogram equalization for contrast enhancement for digital images. Applied Soft Computing, 89, 106114.
Manju, B. R., & Sneha, M. R. (2020). ECG denoising using wiener filter and Kalman filter. Procedia Computer Science, 171, 273–281.
Pattnaik, G., & Parvathi, K. (2021). Automatic detection and classification of tomato pests using support vector machine based on HOG and LBP feature extraction technique. Singapore: In Progress in Advanced Computing and Intelligent Engineering Springer.
Hassaballah, M., Kenk, M. A., & El-Henawy, I. M. (2020). Local binary pattern-based on-road vehicle detection in urban traffic scene. Pattern Analysis and Applications, 23(4), 1505–1521.
Muthukumar, A., & Kavipriya, A. (2019). A biometric system based on Gabor feature extraction with SVM classifier for finger-Knuckle-print. Pattern Recognition Letters, 125, 150–156.
Hussain, M., Bird, J.J., & Faria, D.R. (2018). A study on cnn transfer learning for image classification. In UK Workshop on computational Intelligence (pp. 191–202). Springer, Cham.
Ahlawat, S., & Choudhary, A. (2020). Hybrid CNN-SVM classifier for handwritten digit recognition. Procedia Computer Science, 167, 2554–2560.
Funding
The authors declare that no funds, grants, or other support were received during the preparation of this manuscript.
Author information
Authors and Affiliations
Contributions
All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by KS. V, Dr. PM P and Dr. MA. The first draft of the manuscript was written by KS. V and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Competing interests
The authors have no relevant financial or non-financial interests to disclose.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Vaidya, K.S., Patil, P.M. & Alagirisamy, M. Hybrid CNN-SVM Classifier for Human Emotion Recognition Using ROI Extraction and Feature Fusion. Wireless Pers Commun 132, 1099–1135 (2023). https://doi.org/10.1007/s11277-023-10650-7
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11277-023-10650-7