Authors : Rohini Mehta; Pulicharla Sai Pravalika; Bellamkonda Venkata Naga Durga Sai; Bharath Kumar P; Ritendu Bhattacharyya; Bharani Kumar Depuru

Volume/Issue : Volume 9 - 2024, Issue 7 - July

Google Scholar : https://tinyurl.com/2jftwaym

Scribd : https://tinyurl.com/yfv5aurt

DOI : https://doi.org/10.38124/ijisrt/IJISRT24JUL721

Note : A published paper may take 4-5 working days from the publication date to appear in PlumX Metrics, Semantic Scholar, and ResearchGate.

Abstract : Behavior analysis involves the detailed process of identifying, modeling, and comprehending the various nuances and patterns of emotional expressions exhibited by individuals. It poses a significant challenge to accurately detect and predict facial emotions, especially in contexts like remote interviews, which have become increasingly prevalent. Notably, many participants struggle to convey their thoughts to interviewers with a happy expression and good posture, which may unfairly diminish their chances of employment, despite their qualifications. To address this challenge, artificial intelligence techniques such as image classification offer promising solutions. By leveraging AI models, behavior analysis can be applied to perceive and interpret facial reactions, thereby paving the way to anticipate future behaviors based on learned patterns to the participants. Despite existing works on facial emotion recognition (FER) using image classification, there is limited research focused on platforms like remote interviews and online courses. In this paper, our primary focus lies on emotions such as happiness, sadness, anger, surprise, eye contact, neutrality, smile, confusion, and stooped posture. We have curated our dataset, comprising a diverse range of sample interviews captured through participants' video recordings and other images documenting facial expressions and speech during interviews. Additionally, we have integrated existing datasets such as FER 2013 and the Celebrity Emotions dataset. Through our investigation, we explore a variety of AI and deep learning methodologies, including VGG19, ResNet50V2, ResNet152V2, Inception-ResNetV2, Xception, EfficientNet B0, and YOLO V8 to analyze facial patterns and predict emotions. Our results demonstrate an accuracy of 73% using the YOLO v8 model. However, we discovered that the categories of happy and smile, as well as surprised and confused, are not disjoint, leading to potential inaccuracies in classification. Furthermore, we considered stooped posture as a non-essential class since the interviews are conducted via webcam, which does not allow for the observation of posture. By removing these overlapping categories, we achieved a remarkable accuracy increase to around 76.88% using the YOLO v8 model.

Keywords : Behavior Analysis, Computer Vision, YOLO, Keras, Tensorflow, Image Classification.

References :

1. Jia Lu, Minh Nguyen, Wei Qi Yan; Deep Learning Methods for Human Behavior Recognition; 2020 35th International Conference on Image and Vision Computing New Zealand (IVCNZ);  https://ieeexplore.ieee.org/abstract/document/9290640
2. N. Mehendale; Facial Emotion Recognition Using Convolutional Neural Networks (FERC); SN Appl. Sci., vol. 2, no.3, 2020; https://link.springer.com/ article/10.1007/s42452-020-2234-1 
3. Shrey Srivastava, Amit Vishvas Divekar, Chandu Anilkumar, Ishika Naik, Ved Kulkarni and V. Pattabiraman; Comparative Analysis Of Deep Learning Image Detection Algorithms; Journal of Big Data volume 8, Article number: 66 (2021); https://journalofbigdata.springeropen.com/articles/10.1186/s40537-021-00434-w
4. Liam Schoneveld a, Alice Othmani b, Hazem Abdelkawy b; Leveraging Recent Advances In Deep Learning For Audio-Visual Emotion Recognition; Pattern Recognition Letters, Volume 146, June 2021, Pages 1-7; https://www.sciencedirect.com/science/ article/abs/pii/S0167865521000878
5. Stefan Studer, Thanh Binh Bui, Christian Drescher, Alexander Hanuschkin, Ludwig Winkler, Steven Peters and Klaus-Robert Müller; Towards CRISP-ML(Q): A Machine Learning Process Model with Quality Assurance Methodology; Mach. Learn. Knowl. Extr. 2021, 3, 392–413; https://doi.org/ 10.3390/make3020020
6. Ismail Olaniyi Muraina; Ideal Dataset Splitting Ratios In Machine Learning Algorithms: General Concerns For Data Scientists And Data Analysts; 7th International Mardin Artuklu Scientific Researches Conference; https://scholar.google.com/ citations?view_op=view_citation&hl=en&user=rXa9qAgAAAAJ&citation_for_view=rXa9qAgAAAAJ:hFOr9nPyWt4C
7. Ahatsham Hayat, Fernando Morgado-Dias; Deep Learning-Based Automatic Safety Helmet Detection System for Construction Safety; Appl. Sci. 2022, 12(16), 8268; https://doi.org/ 10.3390/app12168268  
8. Gaurav Meena, Krishna Kumar Mohbey, Ajay Indian, Sunil Kumar; Sentiment Analysis from Images using VGG19 based Transfer Learning Approach; Procedia Computer Science, Volume 204, 2022;  https://www.sciencedirect.com/science/article/pii/S1877050922007888
9. N. Abbassi, R. Helaly, M. A. Hajjaji, A. Mtibaa; A Deep Learning Facial Emotion Classification system: a VGGNet-19 based approach; 2020 20th International Conference on Sciences and Techniques of Automatic Control and Computer Engineering (STA), Monastir, Tunisia, 2020; https://ieeexplore.ieee.org/document/9329355
10. Evander Banjarnahor, Yohanes Adi Saputra Abraham, Yoseph Siahaan; Fundus Image Classification for Diabetic Retinopathy Using ResNet50V2 and InceptionV32024 IEEE International Conference on Artificial Intelligence and Mechatronics Systems (AIMS); https://ieeexplore.ieee.org/abstract/ document/10512970
11. Usman Haruna, Rozniza Ali, Mustafa Man; A new modification CNN using VGG19 and ResNet50V2 for classification of COVID-19 from X-ray radiograph images; Indonesian Journal of Electrical Engineering and Computer Science Vol. 31, No. 1, July 2023; https://ijeecs.iaescore.com/index.php/ IJEECS/article/view/30991
12. H. Kishan Kondaveeti and M. Vishal Goud; Emotion Detection using Deep Facial Features; IEEE International Conference on Advent Trends in Multidisciplinary Research and Innovation (ICATMRI), Buldhana, India, 2020; https://ieeexplore.ieee.org/abstract/document/9398439
13. Almeida, José, and Fátima Rodrigues; Facial Expression Recognition System for Stress Detection with Deep Learning; ICEIS (1). 2021; https://www.scitepress.org/Papers/2021/104742/104742.pdf
14. Priyadarshini D. Kalwad, Suvarna G. Kanakaraddi, Ashok K. Chikaraddi, T. Preeti & Karuna C. Gull (2022); XCEPTION: Facial Expression Detection Using Deep Learning Techniques; Advances in Intelligent Systems and Computing, vol 1415. Springer, Singapore; https://doi.org/10.1007/978-981-16-7330-6_26
15. R. Angeline, A. Alice Nithya; Deep Human Facial Emotion Recognition: A Transfer Learning Approach Using Efficientnetb0 Model; Journal of Theoretical and Applied Information Technology 102.8 (2024); https://www.jatit.org/volumes/ Vol102No8/37Vol102No8.pdf
16. Mahmoud Jameel Atta Daasan, Mohamad Hafis Izran Bin Ishak; Enhancing Face Recognition Accuracy through Integration of YOLO v8 and Deep Learning: A Custom Recognition Model Approach; Methods and Applications for Modeling and Simulation of Complex Systems, AsiaSim 2023, CCIS, vol. 1911, Springer, Singapore; https://doi.org/10.1007/978-981-99-7240-1_19
17. Kakani, Palak, and Shreya Vyas; Automated Catalog Generation Using Deep Learning;  International Research Journal of Modernization in Engineering Technology and Science Volume:05/Issue:08/August-2023; https://www.irjmets.com/uploadedfiles/paper/ issue_8_august_2023/44010/final/fin_irjmets1692089966.pdf
18. K. K. Pal and K. S. Sudeep; Preprocessing For Image Classification By Convolutional Neural Networks; 2016 IEEE International Conference on Recent Trends in Electronics, Information & Communication Technology (RTEICT), Bangalore, India, 2016; https://ieeexplore.ieee.org/abstract/ document/7808140

19. Kiran Maharana, Surajit Mondal, Bhushankumar Nemade; A Review: Data Pre-Processing And Data Augmentation Techniques; Global Transitions Proceedings, Volume 3, Issue 1, 2022,ISSN 2666-285X;  https://www.sciencedirect.com/science/article/pii/S2666285X22000565