After assessing whether appropriate children are entering programs, the research assesses the degree to which the structure of the programs fits with research on best practices. Assessing program adherence to best practices on an ongoing basis is important for several reasons. First, it provides a measure of program integrity and quality that allows for outcomes to be better understood. Also, assessments of adherence give crucial information to direct quality improvement efforts by identifying the strengths and weaknesses of a program. It also allows for benchmarking the performance of the program in comparison to other similar services. Finally, since program integrity and quality change over time, continuous assessment allows a program to evaluate and recognize when deterioration or improvement is occurring (Fixsen, Blasé, Metz, & Van Dyke, 2013; Perepletchikova, Treat, & Kazdin, 2007).
To assess program adherence to best practices, the Social Research Institute created the Child Welfare Program Assessment (CWPA). The measure draws from a wide range of relevant research including implementation science (Fixsen et al., 2013; Fixsen et al., 2005), skill acquisition and learning principles (Ericsson, 1993; Goldstein, Glick, & Gibbs, 1998), psychotherapy process and outcome literature (Gillies, Taylor, Gray, O’Brien, & D’Abrew, 2012; Kamphaus & Frick, 2002; Ledley et al., 2009), and best practice guidelines for common problems/issues (e.g. Cohen, Mannarino, & Deblinger, 2006; Landenberger & Lipsey, 2005; Macdonald et al., 2012; SAMHSA, 1999; Child Trends, 2011; Waldron & Turner, 2008; Winokur, Rozen, Batchelder, & Valentine, 2006; and Winters, Botzet, & Fahnhorst, 2011). The CWPA also draws from the Correctional Program Checklist (CPC). The CPC is a validated tool developed to assess adherence to best practices and use of evidence-based treatments within delinquency and correctional intervention programs (Holsinger, 1999; Lowenkamp and Latessa, 2003; Lowenkamp, 2003; Lowenkamp and Latessa, 2005a; Lowenkamp and Latessa, 2005b). Some components of the CWPA are consensus-based characteristics; there is widespread agreement among researchers, experts, and clinicians as to the importance of the components for creating an effective program.
The CWPA is divided into the following areas:
- Leadership and Theory: This section reviews the qualifications and duties of the program director including: educational background, experience, involvement in hiring new staff, involvement in training new staff, and supervision of staff. This section also assesses evidence-based practice in developing the program’s model or theory.
- Staff: This section reviews the qualifications, training, and supervision of staff. Staff’s input into program structure, as well as perceived program effectiveness is evaluated, along with staff perception of professional and community support.
- Assessment: This section focuses on assessment of youth in the program. The extent to which youth are appropriate for the service is provided is critical to program success. Effective programs assess youth using structured assessment tools with strong psychometric properties. Services should then be tailored to the individual based on results from these assessments.
- Treatment: This section is divided into methods for engaging youth in treatment and changing target attitudes, behaviors, and symptoms. It reviews the types of treatment used to target identified problem behaviors, behavioral management systems, and methods used to train the youth in new skills. This section evaluates how treatment is targeted to the following content areas: behavioral disorders, trauma, mood disorders, substance abuse, delinquency, and sexual offenses (Appendix B provides further detail on each area).
A CWPA assessment consists of an annual program visit that takes place over 2-3 consecutive days. During these visits the researchers interview the program directors, staff, and youth, they review program materials, and conduct observations of treatment activities. Appendix C details the site visit protocol. After the program visit, the researchers score the assessment and then write a detailed report covering all of the sections outlined above. Each section includes details on the strengths of the program, areas that need improvement, recommendations for improvement, and resources for increasing the effectiveness of the service or program.
The results of the CWPA assessment provided to the program in an in-person feedback session. Additionally, the full report is published to the programs dashboard, where program staff can access it (provided under the Evidence-Based Components header on the evaluation website. The program can also access their full reports from previous CWPA visits. As shown in Figure 3, the most important recommendations are listed on the Summary page of the evaluation website.
Figure 3: CWPA Recommendations