Evaluation can be done by in-house staff or by a third-party evaluator. Either way, the staff involved must be knowledgeable about and experienced with the components of an evaluation. A benefit of using internal staff for project evaluation is that they may have working knowledge of the content and be more accessible for spur-of-the-moment meetings, and some of the cost may be covered by your organization's operating budget.
Third-party evaluation may be more complicated to establish and monitor, but the results typically bear more weight, since the evaluator is considered an impartial professional with no vested interest in the outcome of the research. In addition, many funders, including government sources, require professional, third-party evaluation.
To find potential evaluators, ask colleagues who they have used. Funders can tell you about evaluators who have worked on projects they have funded. If you are targeting a specific funder for the lion's share of your budget, you may want to ask your project officers' advice, because they may have specific evaluators they do -- and don't -- respect. You can also contact the National Center for Outreach (NCO) for recommendations and assistance, or check the Outreach Pipeline on the NCO Web site. Look for similar projects, and call the appropriate contact person to find out who is evaluating the project. If your educational outreach materials are tied to national or state standards, you might want your material evaluated by an individual with experience and credibility in those curriculum areas.
Typically, you will want to identify and engage an evaluator as you begin the project-planning process so that your project timetable can appropriately accommodate the time needed for different evaluation phases. You should also include the evaluation plan as part of the proposal you send to funders.
Evaluating an Evaluator
When selecting an evaluator, consider the individual's or company's
- Experience in evaluation and ability to grasp your project quickly
- Knowledge base and strategies for gaining more information about your organization and project
- Attitudes regarding evaluation that suggest a compatibility with your project, evaluation goals, and organizational dynamics
During the initial meeting or conversation, ask your prospective evaluator questions such as the following:
- What is your general experience in evaluation? Has your organization been in business for some time? Do you specialize in one type of evaluation, or a broad range?
Some groups prefer to undertake either formative or summative evaluation, but not both.
- What is your general experience with activities like ours? With participants like ours?
Look for organizations that are familiar with educational outreach in general, and specifically with the type of project you are planning.
- What is your experience with our target audience(s)?
Remember, the audience for the outreach may be wider than for your broadcast program.
- We want to know X about our program. Have you evaluated this type of question before for other projects?
If your main goal is to determine how users navigate your Web site to learn about a particular topic, then you will want to find an evaluator who has experience and facility with the Web medium.
- What evaluation approaches do you think are relevant to our situation? Are the results likely to be more subjective (assessing changes in emotions, beliefs, etc.), objective (offering statistical data), or a combination of the two?
Much of this will depend on the types of questions you ask, but you should be aware of the kind of evaluation that will be acceptable to you and what may be required by your funders.
- What do you consider to be aspects of a strong evaluation? A weak evaluation?
How well does the evaluator articulate components that contribute to a successful evaluation effort? Does he talk about tailoring efforts to each project, using a combination of subjective and objective measures, focus groups, large national surveys, and other approaches as appropriate to different situations?
- What processes and evaluation approaches would you use for this project?
Will the evaluator use a cookie-cutter approach for your project or design one specifically tailored to your needs? For example, a mixture of focus groups, mailed surveys, and telephone interviews may or may not be appropriate to your situation. You want an evaluator to think about the particulars of your project.
- What would you need to know from us to determine appropriate evaluation approaches and methods?
An evaluator will typically need to have several conversations with you and other project members and read all relevant background material to fully understand your project, its goals, and what you will consider important evaluation results. You cannot, however, expect an evaluator to tell you what's important to find out about your resources. This is why you need clearly established goals and objectives for your project from the outset.
- Do you subcontract for any services? If so, can résumés of subcontractors be provided for our information?
You want to be sure that the credentials you're buying are the ones you're actually getting.
- Are you willing to sign a nondisclosure or confidentiality agreement?
Do you want the evaluator to share the results of your project with others without your knowledge or approval? If not, you should be clear about this from the beginning.
- We have a budget of X. What type of plan would you recommend for this range?
Is the evaluator creative in working within your budget? Some larger groups may not take on a project unless the evaluation budget is over a certain amount, such as $10,000.
- How rapidly do you provide feedback, and in what form?
This is particularly important for front-end and formative evaluation, since you will want to get timely feedback that can inform the further development of your activities.
Make sure to ask for references from people with whom a potential evaluator has worked on prior occasions. An evaluator is likely to give you names of satisfied customers only. One way to gather more information is to ask for a list of all clients from the previous year and request permission to contact whomever you choose. Another option is to write down the names of organizations that the evaluator references when you meet and ask to talk directly to those clients. If the evaluator uses them as examples, you should be able to speak with them for more information. You may also ask to see samples of an evaluator's reports to assess if they meet the scientific rigor you want and are compatible with the type of audience you expect to reach with the evaluation results and reports.