P571Accuracy of a machine learning program to correctly triage incoming SMS text replies from a successful cardiovascular SMS-based secondary prevention program

Abstract

Abstract Background Cardiovascular SMS text programs are effective alternate secondary prevention programs for cardiac risk factor reduction and can be delivered as one-way or two-way communication. However, people text back regularly, leading to staffing costs to monitor replies. If you could reduce the need for staff review by 60–70%, costs and scalability of text programs would substantially improve. Purpose To develop and assess accuracy of a machine-learning (ML) program to “triage” and identify texts requiring review/action. Methods We manually reviewed and classified all replies received from two “TEXT ME” cardiovascular secondary prevention programs. Simultaneously a ML model was developed to classify texts and determine those needing a reply (figure). Comparison of ML models included “Naïve Bayes”, “random forest decision trees”, and “gradient boosted trees”, along with comparison to “convolutional neural network” and “recurrent neural network” classification approaches. “Natural language programming” was evaluated however this presented challenges in relation to text content due to non-standard English grammar, frequent use of non-standard abbreviations, and spelling errors. The ML program was trained with 70% of the data-set and accuracy was tested with 30%. Results Manual review of 3118 text replies revealed that only one text was considered urgent, and only 21% required review/action: categorisation was not straight forward due to complexity of texts often containing more than one sentiment (table). The ML program was able to correctly classify 84% of texts into the designated 12 categories. The sensitivity for correctly identifing the need for health professional review was 94% (6.4% false negatives; 3.6% false positives); but with addition of “heuristics” (e.g. searching for specified keywords, question marks etc) sensitivity increased to 97% (2.9% false negatives; 7.3% false positives). Therefore, health professionals would only have to review 27% (true + false positives) of all text replies. Table 1. SMS manual categorisation (n=3118) REVIEW REQUIRED Health Question/concern Admin request Request to STOP Ceased smoking SMS not delivered Urgent/ distress (13%) (4.5%) (3%) (0.8%) (0.4%) (0.03%) NO REVIEW REQUIRED General statement Statement of thanks Reporting good health Blank message Unrelated/ accidental Emoticon only (33%) (23%) (11%) (6%) (4%) (2.4%) Figure 1. Development process Conclusions The ML program has high sensitivity identifying text replies requiring health professional input and a low false negative rate indicating few messages needing response would be missed. Thus, introduction of the program could significantly reduce the workload of health professionals, leading to substantial improvements in scalability and capacity of text-based programs. The future implications for this technology are vast, including utilisation in other interactive mHealth interfaces and cardiovascular health “apps”. Acknowledgement/Funding National Heart Foundation Vanguard Grant; National Health and Medical Research Council Project Grant