A.     Introduction

The CSAP Training System (CTS) consists of multiple programs designed to assist national and local
organizations, States, and communities in preventing substance abuse. CTS services promote the
coordinated application of prevention strategies by many sectors of society.

The audiences of the CTS include

• Community coalition members and volunteers. This group includes representatives of community organizations, school systems, law enforcement and corrections, businesses, and communities of faith.
• Professional preventionists. Among this group are State alcohol and drug agency personnel and outreach specialists.
• Social service and healthcare professionals and paraprofessionals. This group includes mental health specialists, social workers, nurses, physicians and physicians assistants, dentists and dental hygienists, substance abuse treatment professionals, physical rehabilitation specialists, and staff of healthcare delivery systems.

A report of the CTS National Evaluation, presents data obtained from CTS training participants throughout 1994 and 1995.
 

B.     Evaluation in an Open System

The CSAP Training System, like the communities it serves, functions in an open system. Open systems are highly complex and subject to chaotic influences. For example, there are many influences on substance abuse in a community, both positive (prevention programs, political groups, the school system, and social services) and negative (violence, racism, and poverty.) No problem in an open system can be solved from the top down. Today’s “solution” is soon defeated by the emergence of new factors and influences.

Evaluating the outcomes and impacts of prevention programs is challenging because these programs
operate in open systems. Prevention programs must use multiple interventions simultaneously. As
a result, outcomes and impacts in a community cannot be attributed to any single intervention.

How can the contribution of training to community change be documented? From 1992 through 1996, CSAP’s Training and Evaluation Branch (now the Knowledge Applications Branch) sponsored the development of an evaluation method for use in open systems.

A Ladder of Results

The open systems evaluator examines the total environment and attempts to document the program’s
contribution to the whole. The evaluator and program designer together define the maximum result
desired, and the interim accomplishments necessary to achieve this result. This information helps the
program determine what the gaps in its overall strategy are and how to fill those gaps.

The CTS evaluation uses a “Ladder of Results” (sometimes referenced to as a “hierarchy of results”)
to map the maximum and interim strategies used by participants in CTS training. The Ladder of Results organizes these strategies into seven categories as shown on Figure 1. The assumption of the CTS evaluation is that these follow-through strategies will, in general, differ in their impact on substance abuse in communities. The Results Ladder represents this concept; strategies with a greater potential impact appear higher on the ladder. The evaluation assigns “sharing information with people” and “motivational activities” to lower ranks in the Results Ladder. It assigns a higher rank to getting people engaged in changing their organizations or communities (“environmental changes”), and a still higher rank to “creating healthy alternatives” and “fundamental shifts in professional practices.”

Result Mapping

To permit comparison of results from diverse strategies, the CTS evaluators use a Result Mapping
process. The process relies heavily on the use of result mapping sentences. The evaluator fills in blanks (called “facets”) in a structured sentence to describe characteristics of the trainee, team, or
organization that has generated each result. Facets describing the training event are also filled in. Then a results score is assigned to each sentence. The basic form of the mapping sentences is shown in Figure 2.

Each documented result earns a Result Score that reflects both its quantity and quality. The Result
Score is the product of two multipliers:

• points for the type of result (e.g., creation of a healthy alternative; see Figure 3), and

• points for the number of people reached. As shown in Figure 4, an exponential scale is used to rate the number of people reached. This approach avoids exaggerating the impact of lower-level results (such as distributing pamphlets to 1,000 households) that reach many people. The scoring system is illustrated in Figure 5.

High-scoring training suggests that 1) a training generated high-level results in sufficient numbers,
and 2) a training is likely to contribute to effective prevention programming. The mapped sentences
and their analyses yield baseline data for the next round of planning and provide feedback for future
rounds of curriculum design or training delivery.

A Cyclical Planning and Evaluation Process

The CTS evaluation follows a cyclical process in which the evaluator and training designer work
together on each phase in the development of a training program. These phases include

• Clarification of the intent of training.
• Development of a goal-oriented approach for the training, linked to a hierarchy of intended results.
• Training delivery assessment.
• Gathering information from all participants at the event and in a followup 2 months after training.
• A long-range study of training transfer and application, through tracking of samples of participants at 3-, 6, and 9-month postevent.

Cultural Sensitivity in Training Evaluation

The Center for Substance Abuse Prevention recognizes the importance of cultural sensitivity in
training evaluation. CSAP’s predecessor agency, the Office for Substance Abuse Prevention (OSAP)
produced a monograph entitled Cultural Competence for Evaluators (Orlandi et al., 1992). This monograph explores some of the cross-cultural issues involved in program evaluation. For example,
the introductory chapter in that volume provides an overview on the differing perspectives of evaluators and program staff.

Open systems evaluation strives to develop a partnership between the evaluator and the program. This effort can help to bridge the traditional gap between evaluators and program staff.
 

C.     What Worked Well in the CTS Approach to Evaluation

• Evaluation was an integral part of training planning, design, delivery, and follow up. Understanding what needed to be evaluated and including evaluation throughout the process was crucial to validate, provide feedback for continued development and revision, and to provide sound evidence of program/transfer outcome and integration of new knowledge and skills into practice.
• When a single evaluation staff member served as a consistent contact with the training contract, the evaluation process worked better.
• The tools prepared by the evaluators—especially the Hierarchy of Results—were useful for training development.
• The evaluation focused on outcomes, measuring participants’ intentions and behavior, not simply their knowledge.
• The evaluation covered both process and content in the training design—and focused on the outcomes of both.

D.     What the CTS Should Do Differently in Future Approaches to Evaluation

Evaluate Impact on Change Agents

• Evaluation of impact should focus on activities of facilitators, who are themselves in key national positions and who influence policy, teaching, and clinical practice.

• Projects need an evaluation that lets them know if training is being replicated, what the quality of the replication is, and what they can do to make training more replicable. One issue in replication is the importance of the facilitator. Is a talented, briefed facilitator necessary, or can anyone replicate the training?
• Many CTS trainings are heavily replicated. In other cases, partnerships ask for a copy of a specific curriculum so they can do what they want with it. How can the CTS know, in that case, whether the organization’s use of a curriculum constitutes a replication? And is the organization replicating the curriculum as the trainer presented it, or as it appears on paper? Some pieces inevitably get deemphasized. How many modules can be omitted without damaging the integrity of the training?
• Perhaps training designers need to give up the notion of having their courses replicated precisely. At some point, they must trust that the group requesting the curriculum has done an assessment and knows what is best for their community and target population. Replicating isn’t cloning.

• Evaluation of a system would better be done across contracts, not contract by contract. The CTS should evaluate its cumulative effect on individual project results. Also, the relationship between training and technical assistance should be thoroughly evaluated.
• It is hard to measure systemic change by evaluating the behavior of individuals. Even if training were funded for 20 years, it might never be possible to document the system changes. Results might not be evident until years after the last training. The kinds of shifts the CTS sought will not come in within months. That’s the dilemma of evaluating programs with short lifespans.

• The results of evaluation, and the benefits of evaluation data to ongoing program improvement,should be conveyed to audiences, especially audiences with a distrust of evaluation.

Center for Substance Abuse Prevention (CSAP). 1996. Assessing training results. Rockville, MD:
CSAP. Unpublished document.

Center for Substance Abuse Prevention (CSAP). 1996. Training evaluation report of 1994 profile,
feedback, and follow-up data. Rockville, MD: CSAP. Report produced by Pacific Institute for
Research and Evaluation.

CSAP. 1992. Complexity and prevention forum. Meeting Proceedings. Rockville, MD: CSAP
Kibel, Barry M. 1994. Evaluation of local prevention processes: An open systems model in action.
New Designs for Youth Development, 11 (3): 15-22.

Orlandi, M.A., R. Weston, and L.G. Epstein, eds. 1992. Cultural competence for evaluators. A
guide for alcohol and other drug abuse prevention practitioners working with Ethnic/Racial
Communities. OSAP Cultural Competence Series, 1. Rockville, MD: U.S. Department of
Health and Human Services, Public Health Service.