In today's increasingly businesslike world, with fewer dollars to go around, pure philanthropic giving is a thing of the past. Corporations, government agencies, and even foundations are taking a hard look at "Return on Investment" (ROI). They want to know more than Nielsen ratings and the number of IP visitors. They are asking: "What difference did our dollars make? What impact did your efforts have?"
Nearly everyone involved in educational outreach has a treasured collection of anecdotes that attest to the value of what we do. But anecdotes aren't enough to convince funders, producers, or even ourselves that educational outreach has an important place in public broadcasting. Moreover, many funders, especially government agencies, require professional, third-party evaluations to document and justify the impact of federal, taxpayer dollars. The good news is that with a little forethought and planning, you can design your activities to include measurable objectives that can be evaluated effectively to determine reach and impact.
Ultimately, evaluation has two main goals:
- To generate information to help improve the effectiveness of your activities
- To demonstrate to you, your constituencies, and current and prospective funders the impact of your activities
Types of Evaluation
Evaluation can occur at three different phases of your project:
Assesses the knowledge, understanding, and needs of the target audience to serve as a baseline prior to component development. Front-end evaluation can also help shape some important design decisions further down the line. This can be accomplished through market research and focus groups.
Gauges the effectiveness of a component while it is in development. The information collected can inform changes in the activity and correct problems before it's too late. For example, a prototype Web feature, rough-cut video, or draft student activity can be used with small groups made up of members of your intended target audience to test usability, appeal, and learning outcomes. The data collected can help guide further revisions.
Helps determine if you have achieved your goals and objectives. Have you reached your target audience? How many individuals did you reach? Have you helped them increase their knowledge of the subject? Are they able to apply this information to new situations? Have they benefited in some way by interacting with the educational outreach material?
Establishing Evaluation Objectives
Not all projects need in-depth front-end, formative, and summative evaluation for every activity or component. From the outset, you need to be aware of your goals and objectives and how you will measure your success. You need to have clearly articulated objectives for each educational outreach component and the project as a whole. Ask yourself: What do I want to achieve with this activity? What do I want end users to know, be able to do, or understand differently after interacting with this resource or activity?
Objectives can be precise and measurable (e.g., in pre- and post-participation knowledge tests, students will demonstrate a significant increase in their understanding of America's role in the Holocaust). Or they can be more subjective (e.g., the resources will encourage educators to teach about jazz and use the PBS program as part of their curriculum). Even for this latter type of objective, there should be measurable results. For example, what percentage of educators reported they used or would use the Jazz material in their curriculum?
Budgeting for Evaluation
There is no formula for how much evaluation activities should cost. Each project will be different, based on the goals of the assessment. A variable rule of thumb is that evaluation should equal approximately 10 percent of the total budget for the components to be evaluated. This is a good starting place from which evaluation can be scaled up or down. If your budget is fixed or you are working within a set limit, advise the prospective evaluator of the target range for the work before requesting a detailed plan.