Two-Year Training Evaluation Report
Center for Substance Abuse Prevention: An Overview
The major responsibility
of the Center for Substance Abuse Prevention (CSAP) is to spearhead the
Federal effort to prevent substance abuse which has been linked to other
national concerns including violence, rising health care costs, teen pregnancy,
and low work productivity. CSAP is one of three Centers in the Substance
Abuse and Mental Health Services Administration (SAMHSA) of the U.S. Department
of Health and Human Services. CSAP administers a variety of grant and contract
programs. Training is a key element of the CSAP prevention agenda.
CSAP Training System
The CSAP Training System (CTS) offers training and related services to assist national, state, and local organizations in making sustained changes needed to prevent substance abuse. CTS programs focus on building the capacity of individuals, organizations and communities to plan and carry out prevention programs; they explore elements in the following areas:
CTS National Evaluation
Evaluation of the CTS is guided by comprehensive conceptual frameworks used to investigate both the development and outcomes of training in substance abuse prevention. Data collection includes use of questionnaires, observation, curriculum review, and on-site and telephone interviews with participants and training staff.
Approximately 600 profile
(pre), feedback (post), and follow-up reports have been prepared on individual
CTS training events. A previous annual cross-training report, multi-site
reports on 19 different types of training, as well as longitudinal case
studies on 15 training events have been compiled.
Two-Year Training Evaluation Report
This report presents CTS evaluation data obtained from profile, feedback, and two month follow-up questionnaires administered to all CTS training participants throughout 1994 and 1995. Descriptive quantitative data have been supplemented with verbatim written comments from respondents following training. In-depth analysis on post-training application and training effects will follow this descriptive data compilation.
For additional information regarding this and other CTS evaluation reports, write to: Center for Substance Abuse Prevention, CTS Training Evaluation Reports, 5600 Fishers Lane, Rockwall II - 9 Floor, Rockville, MD 20857.
Of the approximately 9500 participants who attended CTS trainings in 1994 and 1995, profile, feedback, and two-month follow-up questionnaires were returned by 8272, 6936, and 2154 respondents respectively. This report presents data on 31 different types of CTS training conducted across the United States in 249 separate events. Training ranged from ½ day to 5 days in length. Some training events were part of an extended learning experience.
Who Attended CTS Trainings?
CTS participants varied in gender, age and ethnicity. Of the questionnaire respondents, 69% were female and 31% male. The majority of respondents (81%) were between 25 and 65 years, with 9% under 25 years of age and 10% over 65 years.
n 2 = 8006
The majority of respondents had some previous experience and/or training in substance abuse prevention, with 15% indicating extensive experience and an equal percent indicating little or no experience.
Respondents were well educated: 38% graduate school; 31% college; 22% some college; 5% high school; and 4% grade school.
Nearly half (42%) attended the training as part of a team, and just over half (52%) indicated that someone other than themselves expected something from them as a result of the training. Respondents indicated which, among pre-selected, organizational types they represented:
Why Did Participants Attend?
Three quarters of respondents (75%) indicated that they were moderately to well informed regarding the purposes of training prior to attending. They rated the following reasons for attending the training:
What Were Participants' Responses to Training?
Respondents were generally satisfied with the training and indicated that the training was on target with elements essential to learning application. For example, most respondents indicated that the training provided sufficient examples and time for discussion and practice. They further indicated that they had sufficient help in planning for application of learning and ample opportunity to consider barriers and facilitators to learning application. The below graph represents ratings of training usefulness at the immediate end of training; mean rating was 8.00.
At the end of each training event, respondents rated training content and process. The high mean ratings indicate positive responses to these training elements.
CTS training programs emphasized
the need to prepare participants for post-training application of learning.
Of the 6632 respondents who rated preparation for application, a mean score
of 3.96 was indicated on a five-point scale where 1= not at all and s=
substantially. Other mean ratings of application-related elements of training
are found in the adjacent graph. Overall, ratings indicate respondents
left CTS trainings indicating preparation for application. Future analysis
will explore
characteristics of training,
learners, and community contexts that affect application.
What Were the Immediate Effects of CTS Trainings?
Both quantitative and qualitative data collected at the end of training, revealed that respondents gained a wide range of knowledge and skills. The below graph shows mean ratings of selected effects at the end of training. All mean ratings are above average. Written comments by respondents supported these ratings and indicated that many gained new perspectives and insight into the work of substance abuse prevention.
Were Participants Prepared to Apply Learning?
At the end of training, respondents were asked the extent to which they felt able to apply learning following their return to community or work settings. The below graph shows the percent of respondents rating the extent to which they felt able to apply learning on a ten-point scale where 1= not at all and 10= completely. The mean rating was 7.66.
Did Participants Have Support in the Community?
Community resources and support make a difference in whether and how participants are able to apply training upon returning to their communities. Respondents rated the extent to which support elements found in the adjacent table existed in their community on a five point scale where 1 = not at all and 5 = substantial. T-tests were done to compare mean ratings at post and follow-up with matched data. Significant differences were found. This suggests respondents found less support than they anticipated at the end of the training. Future analysis will explore the impact of community support on application.
The majority of respondents
(90%) left the training encouraged and with the belief that the training
would make a difference in the way they did their job or volunteer activities.
Two months later, 71% of respondents reported that the training had actually
made a difference. The following pages provide examples of these changes.
What Are the Short-Term Effects of Training?
CTS participants were surveyed two or more months following training to assess their participation in substance abuse prevention activities as a result of CTS training. Both quantitative data on post-training prevention activities were sought, along with open-ended, written comments by respondents on training results related to substance abuse prevention. The quantitative data, shown in the graph below, indicate that the approximately 2100 respondents to the follow-up questionnaire are engaged in a broad range of selected prevention activities. 73% of respondents indicated they made moderate to extreme changes in work or volunteer activities following CTS training and 68% indicated they made moderate to extreme increases in their substance abuse prevention activities. The most highly rated follow-up activity is sharing information with others which supports the diffusion role of such training.
Respondents elaborated on
these ratings of CTS training results in open-ended, written comments.
Among the several thousand responses received, those representing “best
practices” are presented here. These responses were chosen for their relationship
to substance abuse prevention, the varied levels of change they represent,
and the attribution of change (at least, in part) by respondents to CTS
training. The responses are drawn from across all CTS trainings and are
presented exactly as respondents wrote them without editing. The qualitative
data were primarily analyzed using six categories of prevention strategies
developed by the Substance Abuse and Mental Health Services Administration
(SAMHSA), CSAP’s parent agency. Since the training itself represents one
of these categories, a seventh category, “Learning,” was added. This category
leads off to show the kinds of learning respondents reported from CTS trainings
and how this was applied in activities in other categories of prevention
strategies.
Participant Responses About CTS Training Results
LEARNING: Respondents reported a wide range of changes in knowledge, affect, and skill as a result of CTS trainings. Among the changes are new knowledge about the effects of substance abuse, existing resources, and prevention strategies. In addition, changes included affirmation of existing prevention practice and extensive network within and across community agencies. Note: Respondents use of ATOD refers to alcohol, tobacco, and other drugs.
Increased knowledge about ATOD and further understanding of my pro-active role and attitude.
Stepping back to look at own feelings, values, prejudices regarding this issue.
Validated my beliefs that substance abuse prevention activities need to be... systematic.
Made me more aware of how ATOD affects the disability of an individual. I am able to research the effect ATOD have on client’s prescription medicine. Before I didn’t know to do this. I am able to share this information with my clients.
I am now more confident to speak about the subject of alcohol and the substance abuse prevention.
A major infusion of hope for the future of all American Indians.
I am still overwhelmed about the IAAM [Institute for African American Mobilization] workshop. Each time... I have the opportunity to share... about this I get so full inside... Please look over my mistakes in spelling, etc., but I can't say enough about the workshop. Count it to my head, not my heart...
Gave me strategies that I know will work.
Changed my attitude related to different cultures and drug abuse.
It has pricked me to collaborate
with others more.... It made me aware that there are others...like myself
who are...interested in reaching those who may be considering using ATOD.
INFORMATION / DISSEMINATION:Participants reported sharing information at work and in their communities to individuals, groups, state agencies, and community organizations through the distribution of literature, the media, workshops, computer networks, and speaking engagements.
Setting up for local newspaper series on substance abuse prevention.
The messages from the workshop have spread among the community and help leaders to be aware of abuse prevention.
We have used PREVline data and services weekly to update our resources. Updated data base of violence prevention programs and expanded our network of contact folk in prevention nation wide.
I have shared much of the information I learned with colleagues in my partnership and my school.
I have ordered many pamphlets, posters and kits from Prevention Works Magazine and have been able to provide all of the NHA residents with information regarding ATOD use and abuse.
Set up pocket size resource information form... to pass out to homeless street kids.
Have made materials and information given out at the workshop available to bureau of substance abuse prevention specialists as well as community based prevention personnel. Also have shared information availability of Prevline/violine with others within my division and community.
Discuss with patients in dental practice the problems that arise from substance use.
I have begun to actively participate in fairs, conferences and other events that allow me to disseminate prevention materials.
I am discussing substance abuse prevention more frequently during my classroom guidance visits.
I have been interviewed for a community issue television show where I shared the need for prevention and other information I’ve learned.
Offered to send information about substance abuse to providers who have called in.
Have begun organized prevention programs with area churches for high school youth, actively using information pamphlets, books that teen are interest in.
Marked waiting room magazine’s tobacco ads.
I have integrated more prevention material in my role as a child care provider.
I have been able to communicate with my neighbors about drug and alcohol problems that exist in the community and try to work with them on suggestions and methods we can work together to solve some problems in our neighborhood.
Wrote article in Black Family’s Association newspaper.
Initiated newsletter to
promulgate prevention activities.
EDUCATION: Activities described by participants included those for youth, clients, other professionals, and volunteers in school, university, practice, and community settings.
We have started a new program in peer leadership training about issues of substance abuse prevention and AID/HIV awareness training. As a result of the workshop, I have paid particular attention to cultural issues and identification.
I continue to train others in mobilizing churches as active partners in prevention.
I spend more time educating my patients about the danger of ATOD, how it impacts their health and I have contacted two other doctors I met at the program regarding implementing a tobacco education lecture.
I have planned workshops for the Hispanic substance abuse program for the community. The training I received...for prevention has helped me to better present the material I have learned.
We have developed a comprehensive education and intervention effort geared to teenagers who are not smoking, but would be likely to.
Trainings for community volunteers were developed.
We are in the process of including more prevention tools in the juvenile drug program.
Drug and alcohol awareness groups with children at ...Boys and Girls club.
Taught a 3rd grade level on substance abuse prevention.
Trained staff to screen all athletes during physical exam.
We are replicating the Institute for African American Mobilization for youth, which we hope to repeat city wide.
My residency program will now offer to new residents opportunities to do rotation in ATOD prevention programs.
We have started a circle of care committees where we can share information with our teens and expectant mothers who drink while they are pregnant.
We have established workshops to train volunteers to work in their local communities.
To train co-workers and community lay persons about substance abuse prevention.
I have organized a community awareness group on drug prevention.
Convinced a local charity
hospital to allow me to help design the prevention program.
ALTERNATIVES: Respondents described a variety of programs they initiated providing healthy alternatives to ATOD use including youth leadership training, community service, cultural activities, services for parent and senior citizens, and recreation.
I have sent out a survey to parents to see how they stand on various prevention issues and to see how much participation we could get from them. We had about a 70% response. Now, we will set up coffee hours, craft night, and other activities as requested.
We have made space available for an Alateen meeting. It is now the only one in our community.
Myself, along with some other community mothers have had several events at our local black YMCA branch for young teens in problem areas of our community. We have had meetings with... different organizations...asking for their support and help with various events such as: recreational day, youth awareness day, fun day, etc.
Increased activities for teens to keep them involved.
During discharge, I encourage patients to use our resources for leisure recreation , rather than ATOD use.
Formed an alliance.. to support development of more assets that support kids to stay drug free.
We are going forward with proposals for a youth center drop-in. Once school resumes, a task force is being developed to seek out more parents involvement and student input.
I am now organizing a mural project for teenagers.
Contacted more agencies to support abuse programs, including the origin of a grandparents as parents support group...the community is involved in this program.
Started teen group as sponsor.
I have made it a point to speak to the kids I work with, that have not had a problem yet with OD, about substance abuse and alternatives. I have talked to the parents that I work with about the examples they set for their children and the cycle of alcohol and drug abuse in the family environment. We have also discussed alternative lifestyles and support.
Worked with teenagers during summer and in process of organizing a group within one school as a satellite school that will be introduced to their schools.
I am working with Japanese students who will enter college...to combine my knowledge of their culture with the empowerment activities I learned at the workshop to try to steer them away from substance abuse.
Set-up drug awareness and self-esteem group for preadolescents.
Expansion of prevention programs into upper school grades.
My ATF has assessed year
long operation and made changes in our programs so that in every activity/project
undertaken, there is always a portion /part for awareness for drug abuse/prevention.
PROBLEM IDENTIFICATION/REFERRAL: Respondents reported making modifications to exiting screening and referral procedures and forms, as well as screening and referral in the community and private practice.
Conducted a survey of students and teaches at middle and high school levels concerning drug /alcohol usage on/off campus. Information to be used to encourage collaboration between parents, teachers, law enforcement folks, clergy, students, and politicians to deal with the issue, as well as to deal with discipline and violence and the adverse effects these issues are having on education.
Making community survey of ATOD areas in the community to be targeted.
I do more careful drug/alcohol screening and...refer patients more frequently.
Developing enhanced substance abuse screening for juvenile delinquents that enter probation.
More referral to family shelter groups.
More consistent screening for substance abuse problems.
Increased follow-up on substance abuse referrals; increased use of substance abuse screening.
I truly feel that the trainings I have done in my community will make a difference. People involved in primary care are much more aware of ATOD problems that they encounter, but prior to this training did not recognize.
Modification of substance abuse screening test.
Asked more in-depth questions of clients; pushed more when given evasive answers to questions; offered more feedback on behavioral changes made by clients.
Now can utilize a systematic assessment process to key in on when assessing ATOD problems.
We remade our substance abuse evaluation and referral forms.
Adopted a screening form to be used with new clients who do not have substance abuse diagnosis.
Incorporate some substance abuse prevention education in our assessments for physicals, obstetric visits, and general medicine, including pediatrics.
Referrals to appropriate agencies with inpatient/outpatient.
Have expanded our nicotine addition intake form to assess for other addictions and set up a referral.
I do oral history of smoking, alcohol and drug use.
Greater emphasis on prevention with non-using clients in case load.
Increased time in assessing substance abuse by clients during history taking during physical exam.
Maintained file on substance abuse as part of information and referral resources.
I spoke to local Rotary members about the risks of substance abuse and how to refer someone for help if problems are suspected.
On job emphasized to collateral’s, communicable disease epidemiology staff who have responsibility for HIV EPI and partner notification in nine countries, that referrals for AOD may be preventive for HIV. Worked on smoothing referral path.
I’ve used CAGE questions during mental status evaluations.
Detect and refer clients
with ATOD problems to appropriate agency or agencies.
COMMUNITY-BASED PROCESSES: Community activities described by respondents included intra- and inter-organizational collaboration, mobilization, recruitment, and service delivery.
We are currently working on developing a statewide council of clergy and lay persons. This council will be addressing the alcohol, tobacco and other drug problems of their faith based communities...the council will help to develop awareness around the state about ATOD abuse. Using the CSAP model will help us to address these issues.
Hospitals attending formed task forces on this issue.
Created better working partnership/networking with other community service provider programs.
Hosted two follow-up GONA meetings [Gathering of Native Americans].
Local interdisciplinary violence prevention team is meeting monthly to proceed with action plan.
Our organization has initiated a state-wide clergy alliance. Its purpose is to mobilize celery in the area of substance abuse prevention and to encourage members to be substance abuse prevention advocates in their home communities.
We have remained organized and have continued to move forward on our plan.
It gave us the initiative to start a substance abuse committee at our hospital and although it is a slow process we are making some gains; currently we are working on drafting policies.
Changes regarding partnership structure are currently in process -- especially in the areas of executive board function and staff development issues.
Since the workshop, we have gotten more volunteers involved. Also, trying to network with other spiritual communities.
Please be advised that as a result of the sessions I have begun to solicit more community help... begun a parent support group in church and through my local board of education...identified areas of concern and have formed various committees to address the problems...begun meeting with other mental health workers to determine what they are doing in their areas in comparison to mine.
Formed a strong team to take needed action to develop Partnership.
We have written our plan, set up the parameters and are awaiting approval for implementation.
As a direct results of the workshop, we have expanded our workforce to include other ethnic groups. We have broadened our radio program to include people of all religious persuasions.
Became a player at the table in two local coalitions.
I’m now able to communicate better with others who did not know what some of us were talking about regarding substance abuse...some of us had become “old hat” around here.
Some of the school and community members are now willing to work with us.
Increased interaction with other service providers that deal with ATOD and Asian populations.
I have joined the tobacco prevention task force.
We have actively sought to communicate to our management council that it is very important to have youth on the management council. They have accepted our challenge and changed he bylaws and youth will be on our management council as of September 1st.
Increased partnerships with schools to support prevention and youth development; increased partnership with businesses to support prevention.
Networked with disciplines internally to create a strategic plan for community.
I have linked with other community organizations to share and receive information concerning alcohol and drug prevention.
Joined a gang awareness/communication action task force.
Worked with the neighborhood to take some practice steps and start an anti-violence campaign/program.
Forming a hospital-wide committee to deal with problem facing nursing in substance abuse.
Currently I am facilitating the merger of three prevention organizations. Our focus is now to stay focused, mobilize at the grass roots level and adequately evaluate what we do. Currently, we are looking at developing a framework that is customer driven and not an agency driven.
In the past few months,
I have spoken to several groups about prevention.
ENVIRONMENTAL: Participants described changes related to policy and resource generation, public policy changes, and reducing environmental risks.
As a result of using your model to identity problems, I helped write a grant for a community project to help strengthen cultural identity and education for Hispanic youths in substance abuse prevention in Michigan.
Created some funding for a grass root organization.
Implementation of new guidelines and rules in agency programming.
Require no smoking in clinic setting; staff not to use tobacco.
Increased involvement with state agencies.
I am working with the executive of COF agency to implement no smoking policy to support group members.
Our organization is beginning
to network more with other agencies.
Summary and Implications
The preceding survey data
from the CTS evaluation demonstrate the diverse and wide-ranging
effects of short-term professional
and community training in substance abuse prevention.
These survey findings
provide one snap shot of CTS effects. For a more complete picture these
descriptive data need further analysis and need to be combined with other
national and local CTS evaluations, as well as other CSAP evaluation efforts,
to understand better the role of training in primary substance abuse prevention.