The purpose of this workshop is to stimulate participants to consider how to involve their communities and stakeholders in the evaluation of their partnership's work. The emphasis is on using outcome and process evaluation as activities parallel to planning and on reporting and using evaluation findings. The workshop also emphasizes that partnerships should involve their evaluation staff in generating the right questions and in reporting the answers to the community and to stakeholders.
Recognize how evaluation findings can be used to deepen stakeholders' and communities' understanding and ownership of, and involvement in, the partnership's work.
Use a participatory group process to generate questions about the effectiveness and efficiency of the partnership ATOD abuse prevention work and rank these questions in order of priority.
Explain the importance of cultural considerations as they relate to community participation in the partnership's evaluation of its work.
Because planning and evaluation are seen as concurrent processes, this unit is linked to the unit on participatory planning. It is also related to the units on community mobilization, cultural consciousness, and stakeholder analysis, because of its emphasis on ways to involve members of the community in evaluation of partnership activities. Since process evaluation is related to assessment of the internal environment of the partnership, this unit is also linked to the workshops on that subject.
The overall planning framework from Unit 1D is not included in participant manuals for this session because it is assumed that there will be a large copy of it posted in each room. If the facilitator wants each participant to have a copy to use as a point of reference, the handout will have to be photocopied ahead of time for distribution during the session.
Trainers should review TSupp-1 and prepare PN-1.
The following resources, which are not in the Trainer Resource Manual, may also be helpful in preparing to deliver this unit:
Galano, J., and J. Nezlek. (1986). Evaluating prevention programs: A training manual. Office of Prevention, Promotion, and Library Services, Department of Mental Health, Mental Retardation, and Substance Abuse Services, Richmond, Commonwealth of Virginia, September.
Hawkins, J.D., and B. Nederhood. (1987). Handbook for evaluating drug and alcohol prevention Programs. DHHS Pub. No. (ADM)87-1512. U.S. Office for Substance Abuse Prevention, ADAMHA, Rockville, MD.
Price, R.H., and S.S. Smith. Guide to evaluating prevention programs in mental health. DHHA Pub. #(ADM) 85-1365, 1985.
Solomon, D.D. (1977). Evaluating community programs. In Tactics and techniques of community practice, ed. F.M. Cox et al., Itasca, IL: F.E. Peacock Publishers, 326-340.
For additional resources, see Planning and Evaluation in the CPI Trainer Resource Manual.
B. Ask participants to write down 2 questions about evaluation that they hope this unit will answer.
C. Going around the room one by one, ask each participant to tell the other participants his/her name, and partnership's name and location, and to read one of the questions that he/she wrote down.
D. Write each question on newsprint exactly as phrased and label it with a letter, leaving room between each question for later comments. Record each question even though some may seem redundant.
E. After going around the group once, ask for any other questions that do not duplicate those on the newsprint and write those down.
F. Ask if each question is clear and, if not, have the person who gave the question clarify it. If the clarification significantly changes the meaning, make notes on the newsprint or rewrite the question to make sure that the wording accurately reflects the question and that everyone clearly understands each question.
G. Ask each person to pick the 3 most important questions from the newsprint and write the letters for those questions on a piece of paper. Then ask them to write the rank order of the three questions (1 = most important). Post PN-1.
H. Ask participants to write the rank number on the row next to the letter label of the question on PN-1.
I. Point out that one could quantify these rankings by calculating an average ranking. Then discuss what the raw numerical rankings show the most important questions to be.
J. Select one of the questions and ask participants to brainstorm some indicators of appropriate answers. (In other words, how would they know whether, and how well, the question was answered?).
K. Write answers on newsprint. Point out that these indicators form a basis that could be used for evaluating this workshop. Note that they participated in, and collaborated on, the outcome indicators.
Tell the group that they have just used a technique called "nominal group process" to elicit members' ideas about the most important questions to be answered about evaluating this workshop.
Content Points
A. Characteristics of this process.
They each played an active part in devising an evaluation.
The result reflects the entire group's thinking.
The process is simple and easy to use.
B. This process could have been much more elaborate, or a number of other participatory processes could have been used. The entire process could be repeated at the same time with other groups of participants, perhaps with different questions and indications that would have to be reconciled.
C. From the list of questions generated, it would be easy for staff to develop methods for evaluating whether, and to what degree, those questions have been answered when the workshop is over. (If there are evaluation staff members present, ask them to give a brief example of how this could be done.)
D. By using such methods, with groups of decision makers, community members, and stakeholders, partnerships can get wide participation in determining what questions concerning impact, outcome, and process to address in the evaluation of the partnership's work.
A. Evaluation of your partnership's work helps demonstrate that you are doing something worthwhile.
B. It helps document your "good works," and that documentation can be used to help others in their planning efforts.
C. Evaluation permits you to direct and redirect your resources more thoughtfully and to make attainment of your partnership's mission more likely.
D. CSAP places much emphasis on partnerships' evaluation of their own planning efforts and of their communities' ATOD abuse prevention activities because of:
The high visibility of partnership activity;
The intention to replicate partnership successes (and avoid its mistakes);
The partnership's accountability to funding sources and other stakeholders.
Documentation and information dissemination requirements for a variety of purposes; and
The congressional mandate for such evaluation.
E. Because evaluation is often seen as not useful, it is often undervalued. Evaluation reports are often read by virtually no one but the preparers of reports and those who authorized them. Two factors that lead to underuse of evaluation information are:
Requirements for evaluation are imposed form outside and not generated from within; and
Evaluations often focus on outcomes or impacts prematurely, before the plans or programs have had the chance to yield valid information.
F. To be effective, evaluation must have a motive. Evaluation conducted without motivation leads to disinterest and outright resistance.
G. One solution to this problem is to include a broad range of participants. This builds a sense of ownership. The planning and evaluation framework discussed earlier in the week calls for the collaboration and participation of all the relevant parties.
H. Five general steps guiding such participation in evaluation are listed in HO-1
(HO-1: Five General Steps for Participatory Evaluation).
Identify and organize relevant stakeholders, communities, and decision makers, both in and outside of the partnership--something already part of the planning framework.
Identify questions to be answered by evaluation.
Select methods of evaluation likely to generate the information needed and wanted by stakeholders, communities, and decision makers.
Use a participatory process between evaluation staff and partnership members to analyze and interpret data generated by evaluation.
Use a collaborative process involving evaluation staff and members of the partnership in dissemination of evaluation findings.
I. "[T]he final measure of . . . evaluation is the extent to which its findings are used by decision makers and other consumers" (Price and Smith, 1985).
J. The end result of participatory evaluation is increased capability and commitment on the part of communities, stakeholders, and decision makers. Communities can be as empowered by evaluation as well as by planning and taking action (HO-2: 10 Building Blocks of the Participatory Evaluation Process).
A concern for the use of evaluation information should be the driving force in evaluation.
Concern for use of evaluation information starts at the outset and continues throughout the evaluation.
Evaluation of prevention planning and programs should be aimed at the interests and information needs of specific, identifiable groups and communities.
The communities, stakeholders, and decision makers who will be using evaluation data should be involved in making decisions about evaluation at every stage in order to ensure their commitment to understanding and using the findings. Participation is the ultimate antidote for the fear and resistance associated with evaluation.
Representatives of various constituencies should come together at the outset of planning to decide on which questions will be given priority.
Careful selection of the communities and stakeholders who will participate in the evaluation process will improve the quality of the outcome.
The professional evaluation staff has a responsibility to train communities and stakeholders in evaluation methods and the uses of evaluation information.
The highest priority of evaluation is to serve the intended use of the intended audience.
Early in the evaluation process, and in collaboration with communities and stakeholders, the partnership needs to consider how various factors that may affect the use of information might diminish or threaten the potential for full and proper use.
Adequate time and money must be allocated to ensure that substantive participation in, and use of, evaluation takes place.
A. Review related definitions listed in T-Supp.-1. (from Webster's 9th New Collegiate Dictionary) (TSupp-1: Definitions of Terms Related to Evaluation)
Content Points
Evaluate: To determine or fix the value, significance, or worth of something, usually by careful appraisal and study, usually in nonmonetary terms (HO-3: Definitions of Evaluation).
Assess: To determine the importance, size, or value of something.
Appraise: To set a value on, evaluate the worth, significance, or status of something, often in monetary terms and with expert guidance.
Analyze: To study and determine the nature and relationships of the parts of a whole; to divide a complex whole into its parts or elements; to distinguish component parts of a whole to discover its true nature or inner relationships.
B. Evaluation is a way to get answers to questions about important subjects (HO-3):
Impact. Is your community's prevention system really working? (That is, have the incidence and prevalence of ATOD abuse been declining as a result of that system's work? How do you know?)
Outcome. Has your partnership been accomplishing its part of the prevention system's goals? (How do you know?)
Process. How has your partnership gone about accomplishing its objectives? How do you know?
C. Evaluation gets answers to the questions by:
Systematically collecting and using information that,
When built into the partnership's process from the start, is an ongoing mechanism that
Helps keep the partnership heading in the right direction.
D. Stereotype of evaluation
Many people believe that evaluation is a set of activities done by professional staff, divorced from the membership and the community, after a program has been planned and put into effect.
This stereotype is misleading and may interfere with proper evaluation.
E. As discussed earlier in the week, evaluation should be part of the process of planning and managing the community partnership (HO-4: Community Prevention Planning Framework).
F. The participatory planning framework (Green and Krueter) shows evaluation as interwoven into a recurrent cycle of planning, implementation, and decision. Evaluation should be part of an ongoing, formative process that accommodates the changing needs and capabilities of the community.
G. Good evaluation provides necessary information to decision makers, stakeholders, and the communities in which the partnership is embedded. It provides this information without compromising accuracy or rigor.
A. Reporting the findings of a participatory evaluation calls for tailored strategies for different audiences because each has special information needs and intended uses of the information. It might mean:
A 2-page executive summary for local legislators;
A 5-page report with charts or graphs for teachers and principles;
A 1-page summary for parents;
A 15-page report to partnership member organizations with a discussion of implications and recommendation;
A 1-page press release for the community in general; or
A 50-page report to CSAP with raw data, protocols used, and other support materials.
B. From the beginning, the partnership must ensure that the right information, in the proper dosage and package, reaches the right people, so that they can use the information for the purposes for which it was intended.
C. In summary, when thinking about evaluation, one should consider who will use the information it provides and what they need or want to know.
A. Divide the group into 3 subgroups. Each is to brainstorm a list of questions for a different part of the external environment of a partnership in a community where ATOD abuse is a serious problem:
One subgroup is to brainstorm questions that they think the community would ask about the goals of the partnership and the effect of the partnership's work on the incidence and prevalence of ATOD abuse. The group should write the list on newsprint and tape it on the wall.
The second subgroup is to brainstorm questions that they think local government officials who are feeling pressure from ATOD abuse problems would ask about the objectives of the partnership. What would officials ask about the effect of the partnership's work on target behaviors and environments of an ATOD abuse problem, such as the incidence of arrests for driving while intoxicated?
The third subgroup is to brainstorm questions that they think a State or regional prevention network interested in replicating the work of the partnership would ask about how the partnership reached its objectives and how it knows what worked and what did not work.
B. After 10 minutes, ask the groups to stop brainstorming. Give each a few minutes to tell the others a few of the questions that it generated. Point out that the exercise simulated only three parts of the environment (a local community, local legislators, and a State or regional prevention network) and the groups had to guess the questions each might ask. In reality, their partnerships need to plan how they will involve all sectors of their environments in generating questions about the effectiveness of the partnership's work.
C. Discuss the processes used at the beginning of the workshop and how they could be adapted for use with these sectors to prioritize the evaluation questions in each of the phases of planning: impact, outcome, and process.
A. Because each partnership has professional evaluation personnel on staff or under contract, partnership members need not be evaluators.
B. Evaluators are not researchers. Researchers frequently focus on conceptual questions unrelated to utility and may try to avoid influence by others. Evaluation is intended for use by decision makers and stakeholders and seeks to be influenced by them.
C. It is important to be sure that evaluators are involved in the partnership's community prevention efforts and that both staff and members understand the importance of involving the community in the evaluation process.
D. A description of a participatory process between the evaluation staff and partnership members is described in a handbook by Hawkins and Nederhood.
The Hawkins and Nederhood handbook describes how each step in evaluation can involve both staff and partnership members. The handbook is called STEPP, the Staff/Team Evaluation of Prevention Programs, and is designed to be used by community groups such as the partnerships. (The Federal Government stopped printing the handbook in 1990, but copies may still be found in libraries and reference collections.)
Back to Table of Contents Cont'd.