Using Return on Investment to Evaluate Child Welfare Training Programs

Abstract

The impact of child abuse and neglect has affected many aspects of society beyond just those in child welfare. In 2010, there were 408,425 children in foster care in the United States (U.S. Department of Health and Human Services, 2011), and at least that many children who were living in the homes of their parents but under the jurisdiction of a Child Protective Services (CPS) agency. Researchers have also conservatively estimated that there are over 52,000 CPS workers in the nation providing services to these children (Barth, Lloyd, Christ, Chapman, & Dickinson, 2008). Based on estimates by Wang & Holton (2007), the annual economic impact of child abuse and neglect amortized for inflation is over $111 billion, of which less than 25 percent is directly related to child welfare expenditures. The other expenditures relate to the impact on the health, judicial, and education systems and to the loss to society in terms of productivity. Collins (2008) suggested that the field of child welfare is a very difficult area of practice involving many social problems of children and families that require a high level of expertise from child welfare workers. Collins, Kim, and Amodeo (2010) argued that any best child welfare practice has a worker at the core who can demonstrate knowledge of a content area, attitudes that are supportive of the children and families they serve, and skill in the delivery of a specific service. It is because of this needed expertise that training is such a fundamental part of child welfare services. In fact, training is so critical that the Child and Family Services Reviews (CFSRs), a process in which the federal government holds child welfare services agencies responsible, has included training as one of the seven elements related to the outcomes of safety, permanency, and well-being (van Zyl, Antle, & Barbee, 2011). Public child welfare agencies spend large amounts of resources on training their staff to provide high-quality services (Collins, 2008), which in part is due to the high national annual average turnover rate of 26 percent for child welfare workers (CPS-Human Resource Services [CPS-HRS], 2006). Federal Title IV-E child welfare expenditures for training between 1995 and 2008 averaged $238.6 million annually and accounted for 3.9 percent ($3.3 billion) of the overall $85.1 billion in Title IV-E expenditures (U.S. House of Representatives Ways and Means Committee, 2011). Collins (2008) suggested that the evaluation of such training is important because of the need to ultimately identify the best means of conducting training that leads to best child welfare practices. Salas and Cannon-Bowers (2001) noted that, unfortunately, the evaluation of training is easier said than done. They said that this evaluation is resource intensive, has high costs associated with it, has political implications and, worst of all, could show that the training did not work. In fact, though training is important, there is very little evidence to establish that training has been effective in addressing very complex issues related to child welfare, such as cultural competency, diversity, and antiracism (Johnson, Antle, & Barbee, 2009). The evaluation of the child welfare training has been historically guided by Donald Kirkpatrick's taxonomy of training developed more than 50 years ago (Antle, Barbee, & van Zyl, 2008; Kirkpatrick & Kirkpatrick, 2006). This taxonomy includes the reactions of workers (level 1) such as those indicated by course evaluations, learning (level 2) typically indicated by pre- and posttests, transfer of learning (level 3) as indicated by case reviews, and organizational impact (level 4) as measures of organizational statistics. More recently, others have suggested a fifth level of evaluation called the return on investment (ROI) (J. J. Phillips, 1998;J. J. Phillips & Phillips, 2008). Unfortunately, there have been only two research studies on level 4 evaluation in child welfare in the last 40 years (Antle et al. …