Predicting determinants influencing user satisfaction with mental health app: An explainable machine learning approach based on unstructured data

Abstract

In the contemporary digital landscape, the rising concern for mental health has sparked a surge in the use of mental health apps (MHAs) as accessible tools for addressing psychological well-being. Maintaining a high level of user satisfaction (USAT) is important for MHAs in the highly competitive app market. Leveraging BERT (Bidirectional Encoder Representations from Transformers), a state-of-the-art deep learning (DL) model, we perform topic modeling and sentiment analysis on 17,717 user online reviews. Specifically, we employ the BERTopic model to identify the determinants of USAT with MHAs. Utilizing a BERT-base-multilingual-uncase-sentiment model, we perform sentiment analysis to distinguish determinants that elicit satisfaction from those causing dissatisfaction. Also, this study tests and compares six machine learning (ML) algorithms to predict the influence of determinants on USAT with MHAs. The Light Gradient Boosting Machine (LightGBM) emerges as the top performer, showcasing its efficacy in predicting USAT determinants. By using SHAP (Shapley Additive exPlanations), an explainable ML model with cross-validation, we visualize the results of the LightGBM. The SHAP values show that the five most influential determinants of USAT with MHAs include soothing audio experience, smoking cessation support, payment and subscription management, tracking progress and mindful meditation experience. This study facilitates a deeper understanding of user experiences through the identification and prediction of determinants of USAT with MHAs. Understanding these factors and their interplay is essential for developers, clinicians, and stakeholders who aim to enhance MHAs’ services and ultimately improve the well-being of users.