Authors : Arvind Kumar; Arpan Mukherjee

Volume/Issue : Volume 11 - 2026, Issue 3 - March

Google Scholar : https://tinyurl.com/bdevft9s

Scribd : https://tinyurl.com/x4m5kab9

DOI : https://doi.org/10.38124/ijisrt/26mar176

Note : A published paper may take 4-5 working days from the publication date to appear in PlumX Metrics, Semantic Scholar, and ResearchGate.

Abstract : Driver behavior remains a leading factor in road accidents, yet existing monitoring systems typically rely on single data modalities such as facial expressions or speech alone, limiting their reliability and contextual awareness. This work proposes a comprehensive driver behavior monitoring system using multimodal AI, uniquely integrating video, audio, and vehicle speed telemetry — an approach that remains underexplored in existing literature — to predict driver emotions and behaviors in real time. The system analyzes facial cues to detect visual anomalies, processes audio inputs to infer emotional states, and incorporates speed telemetry to provide additional behavioral context. This fusion of modalities is designed to improve classification accuracy and reduce false positives compared to unimodal approaches. Performance evaluation is conducted using benchmark datasets for both video-based and audio-based emotion recognition, with comparative analysis between individual and combined modalities. By addressing the challenges of multimodal integration and real-time processing, this research contributes a novel and effective framework for intelligent driver assistance systems, advancing the goal of enhanced road safety through predictive behavioral intervention. Additionally, this research is being extended to incorporate an intermediate-fusion-based multimodal decision framework, wherein top predictions from image, audio, and vehicle telemetry are jointly fed into a decision model (e.g., Random Forest or SVM) for improved context-aware warning generation. This approach addresses prior limitations of ensemble-style fusion and better aligns with the goals of true multimodal AI.

Keywords : Multimodal Driver Monitoring; Real-Time Emotion Detection; Deep Learning; Audio-Visual Data Fusion; Road Safety; Predictive Behavior Analysis

References :

1. World Health Organization, “Global status report on road safety 2023,” World Health Organization Publications, 2023.
2. F. Qu, N. Dang, B. Furht, and M. Nojoumian, “Comprehensive study of driver behavior monitoring systems using computer vision and machine learning techniques,” Journal of Big Data, vol. 11, p. 32, 2024.
3. Z.-Y. Huang et al., “A study on computer vision for facial emotion recognition,” Scientific Reports, 2023.
4. G. Liu, S. Cai, and C. Wang, “Speech emotion recognition based on emotion perception,” EURASIP Journal on Audio, Speech, and Music Processing, p. 22, 2023.
5. R. Singh, S. Saurav, T. Kumar, R. Saini, A. Vohra, and S. Singh, “Facial expression recognition in videos using hybrid CNN and ConvLSTM,” International Journal of Information Technology, vol. 15, no. 4, pp. 1819–1830, 2023.
6. M. Mohana, P. Subashini, and M. Krishnaveni, “Emotion recognition from facial expressions using hybrid CNN–LSTM network,” International Journal of Pattern Recognition and Artificial Intelligence, vol. 37, no. 8, 2023.
7. B. Niu, Z. Gao, and B. Guo, “Facial expression recognition with LBP and ORB features,” Computational Intelligence and Neuroscience, 2021.
8. Y. Albadawi, M. Takruri, and M. Awad, ‘A review of recent developments in driver drowsiness detection systems’, Sensors, vol. 22, no. 5, p. 2069, 2022.
9. J. Hu, L. Shen, and G. Sun, ‘Squeeze-and-excitation networks’, in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7132–7141.
10. M. M. Kabir, T. A. Anik, M. S. Abid, M. F. Mridha, and M. A. Hamid, “Facial expression recognition using CNN-LSTM approach,” in IEEE international conference on science & contemporary technologies (ICSCT), 2021.
11. M. Selvaraj, R. Bhuvana, and S. Padmaja, “Human speech emotion recognition,” International Journal of Engineering and Technology, vol. 8, no. 1, 2016.
12. H. Aouani and Y. B. Ayed, “Speech emotion recognition with deep learning,” Procedia Computer Science, vol. 176, pp. 251–260, 2020.
13. A. Kumar, “A new fitness function in genetic programming for classification of imbalanced data,” Journal of Experimental & Theoretical Artificial Intelligence, vol. 36, no. 7, pp. 1021–1033, 2024.
14. A. Kumar, P. Maurya, S. M. Tiwari, A. Ali, H. Vasisht, and A. S. Baghel, “Classification of forest cover-type using ensemble of decision tree, random forest and k nearest neighbor,” JIMS8I International Journal of Information Communication and Computing Technology, vol. 10, no. 2, pp. 615–619, 2022.
15. A. Mollahosseini, B. Hasani, and M. H. Mahoor, “Affectnet: A database for facial expression, valence, and arousal computing in the wild,” IEEE transactions on affective computing, vol. 10, no. 1, pp. 18–31, 2017.
16. A. Montoya, D. Holman, SF_data_science, T. Smith, and W. Kan, “State Farm Distracted Driver Detection,”.  https://kaggle.com/competitions/state-farm-distracted-driver-detection.