A Reinforcement Learning-Based Approach for Promoting Mental Health Using Multimodal Emotion Recognition

Abstract

This research aims to enhance mental well-being by addressing symptoms of anxiety and depression through a personalized, culturally specific multimodal emotion prediction system. It employs an emotionally aware Reinforcement Learning (RL) agent to suggest tailored Cognitive Behavioral Therapy (CBT) activities. The study focuses on developing precise, individualized emotion prediction models using facial expressions, vocal tones, and text, and integrates these models with the RL agent for emotionally aware CBT recommendations. The mHealth approach combines deep learning models with RL, achieving accuracies of 72% for facial expressions, 73% for vocal tones, and 86% for text, all fine-tuned for the Sri Lankan context. Validation through real-world use and user feedback consistently demonstrated that each model exceeds 70% accuracy, fulfilling the objective of precise emotion prediction. A weighted algorithm was introduced to refine the emotion prediction experience and personalize forecasts across the three modalities to enhance mental well-being. The RL-enabled agent suggests CBT activities approved by mental health professionals, tailored based on predicted emotions, and delivered through the same mHealth application. The effectiveness of these interventions was assessed using the DASS-21 questionnaire, revealing significant reductions in depression scores (from 21.08 to 13.54) and anxiety scores (from 19.85 to 10.46) in the study group compared to the control group. The study concludes that integrating multimodal emotion prediction models with RL-based CBT suggestions positively impacts mental well-being and contributes to personalized mental health interventions.