Can automated content analysis be used to assess and improve the use of evidence in mental health policy? A systematic review

Kristel Alla,Florin Oprescu,Harvey A Whiteford,Carla S Meurk,Brian W Head,Wayne D Hall

doi:10.1186/s13643-018-0853-z

Abstract

BackgroundThis review assesses the utility of applying an automated content analysis method to the field of mental health policy development. We considered the possibility of using the Wordscores algorithm to assess research and policy texts in ways that facilitate the uptake of research into mental health policy.MethodsThe PRISMA framework and the McMaster appraisal tools were used to systematically review and report on the strengths and limitations of the Wordscores algorithm. Nine electronic databases were searched for peer-reviewed journal articles published between 2003 and 2016. Inclusion criteria were (1) articles had to be published in public health, political science, social science or health services disciplines; (2) articles had to be research articles or opinion pieces that used Wordscores; and (3) articles had to discuss both strengths and limitations of using Wordscores for content analysis.ResultsThe literature search returned 118 results. Twelve articles met the inclusion criteria. These articles explored a range of policy questions and appraised different aspects of the Wordscores method.DiscussionFollowing synthesis of the material, we identified the following as potential strengths of Wordscores: (1) the Wordscores algorithm can be used at all stages of policy development; (2) it is valid and reliable; (3) it can be used to determine the alignment of health policy drafts with research evidence; (4) it enables existing policies to be revised in the light of research; and (5) it can determine whether changes in policy over time were supported by the evidence. Potential limitations identified were (1) decreased accuracy with short documents, (2) words constitute the unit of analysis and (3) expertise is needed to choose ‘reference texts’.ConclusionsAutomated content analysis may be useful in assessing and improving the use of evidence in mental health policies. Wordscores is an automated content analysis option for comparing policy and research texts that could be used by both researchers and policymakers.

Highlights

This review assesses the utility of applying an automated content analysis method to the field of mental health policy development
Wordscores is an automated content analysis option for comparing policy and research texts that could be used by both researchers and policymakers
This paper focuses on one method, Wordscores, an algorithm that was developed by Laver, Benoit and Garry [25] in 2003

Summary

Introduction

This review assesses the utility of applying an automated content analysis method to the field of mental health policy development. Academics are increasingly expected to inform policy and influence policymakers to produce and implement evidence-based recommendations. This imperative is based on the assumption that evidence-informed policy will improve outcomes and efficiencies [1,2,3]. There have been repeated calls to better incorporate scientific evidence on the most effective interventions into mental health policies and services [10,11,12]. Collie and Livingstone [3] have argued that evidence-informed policy requires tools that facilitate the translation of evidence into effective interventions and policies. Chriqui and Stamatakis [6] echoed the need for ‘systematic and evidence-based approaches to policy development’ (p. 1576)

Objectives

Methods

Results

Conclusion