Continue to Actions and Considerations
The purpose of a NORC program is to bring about the changes that allow seniors to age well in their own communities. A well-designed evaluation helps a NORC program determine whether it is making a difference and accomplishing its intended goals.
Many program planners are unfamiliar with evaluation techniques, and even a bit intimidated by them. But an evaluation does not have to be extensive to show whether activities are benefiting the target population. It needs, instead, to be smart, strategic, and evidence-based.
NORC programs address complex issues, typically involving many interacting factors and multiple players. But each project that addresses these issues does not need to be complex, nor does its evaluation.
The approach to evaluation described here is not intended to assess the overall value of the NORC program model. Similarly, it should not be considered a global assessment of a given NORC program. The goal of evaluation is much simpler – to show that each project undertaken has value to the residents of the community. It pays to start thinking about how to demonstrate a positive effect from the earliest phase of design.
There are two common types of evaluation: process and outcome.
A process evaluation assesses the extent to which a project is operating as intended. (For guidelines on process evaluations, see Designing and Implementing a NORC Program, Guiding Principle #6.)
An outcome evaluation emphasizes the extent to which a specific project has achieved its objectives. That is the focus of the material in this section.
The roadmap presented here has been adapted for NORC programs from Framework for Program Evaluation, a well-known and tested model developed by the Centers for Disease Control and Prevention.
Documenting and evaluating projects promote:
A NORC program incorporates a well-planned evaluation into projects as early as possible in their development.
Committing to an evaluation from the earliest planning stages of a project enables a NORC program to collect baseline data, document activities, and measure change over time. It also allows early findings to be more readily incorporated into the project design. Assessing the effectiveness of a project after it is well underway is much harder to do and not nearly as useful.
The purpose of evaluation is to gather the information needed to find out whether a project is doing what it set out to do and to identify opportunities to make improvements. A well-designed evaluation complements and supports the goals, objectives, and implementation plan developed for the project itself.
Continue to Actions and Considerations
A NORC program incorporates a well-planned evaluation into projects as early as possible in their development.
Get everyone on board about the importance of evaluation. Before designing an evaluation strategy, the NORC program partners need to understand why it is in their self-interest to do so. Some partners may have to be sold on the idea that evaluation is an activity worthy of resources. Until everyone agrees on that, evaluations are likely to be of limited use.
Take a look at similar evaluations. Look for your own previous studies or evaluations conducted in similar environments that may be useful as tools of comparison.
Review your work plan. The work plan you created for each project is a useful guide to what you will be evaluating (see Designing and Implementing a NORC Program, Guiding Principle #5). A work plan spells out your:
Use the logic model to design your evaluation. The logic model helps you to create a picture of how the many elements of any project—including planned activities, resources, and expected outcomes—are supposed to work together.
The first five steps of your logic model—developing your problem statement, goals, objectives, resources, and activities—are generally contained in the work plan.
The data that you collect and analyze for an evaluation can also be used to refine your projects. Check in along the way—by keeping an eye on your findings as you collect them, you may be able to make mid-course corrections, when needed.
All NORC program partners, including the senior residents, have roles to play in designing an evaluation strategy, collecting data, analyzing the findings, and applying them to decision making. They may also participate in surveys, questionnaires, interviews, and focus groups.
Each project evaluation is sharply focused and designed to be practical.
Given limited resources, NORC programs need to design simple and practical evaluations that provide essential information without becoming burdensome. A streamlined approach prevents errors and inefficiencies, including duplication, lost time, and misplaced data.
Keeping focus on the goal—to learn whether projects are making a difference—will help clarify thinking about the best way to shape your evaluation. A credible evaluation should use accurate measurements that support the reported results and conclusions. Completing the logic model will help in organizing the information so that it can be widely used.
Continue to Actions and Considerations
Each project evaluation is sharply focused and designed to be practical.
Decide what data you need to collect and how you will collect it. Generally, you should first document the problem you are addressing—for example, before beginning a project to prevent falls, you need baseline data on the number of senior residents at risk or the percentage of seniors that have received flu shots. Then, you should measure again after a project has been in place for a period of time, typically six months, to make comparisons. (See Deepdale CARES NORC Program Profile.)
Keep your measures simple and streamlined. Sometimes, you can answer a critical question just by counting heads—for example, you can get information about social engagement by counting how many seniors participate in a friendly visiting initiative. The measures you choose depend on what you are evaluating, the audience who will review the findings, and the resources available for the undertaking. Where possible, build on data your NORC program already collects regularly.
Consider both quantitative and qualitative approaches. Combining both approaches helps to confirm your findings and highlights key insights.
Count your outputs. The next step in completing your logic model is to list outputs. Outputs are linked directly to your activities and show the quantifiable results of what you did. Usually, they are measured in terms of volume—the number of classes taught, the number of people served, the amount of material distributed. Outputs alone do not offer insight about the effect of a project, but they are necessary for measuring outcomes.
Identify your outcomes. The final step in completing your logic model is to record outcomes, which are changes in conditions, behavior, attitudes, skills, or knowledge that occur as a result of participating in a project. There are three types of outcomes:
Focus on tracking outputs and immediate outcomes. These are your building blocks. Showing success here suggests a project is making progress towards longer-range outcomes. Long-term outcomes themselves are often difficult to measure because they are subject to many influences and can take a long time to become evident.
Intermediate-term outcomes are often the most practical way to feel confident that a project has made a positive difference in the lives of the people it serves. For example, if ten providers attended a falls prevention training session (that’s the output), began using a standardized falls assessment tool (that’s the immediate outcome), and increased shared care planning among providers (intermediate outcome), you can feel more confident that your project will ultimately reduce the risk of falls themselves (long-term outcome).
A streamlined evaluation can still be rigorous. Recognizing the limits of staff and resources at most NORC programs, measures should be no more complicated or broad than necessary.
Getting early agreement from all partners about what you are measuring, and how, lessens the chance of conflict over time.
To conduct a useful evaluation, a NORC program commits adequate human and financial resources and develops efficient recordkeeping systems.
Evaluating the projects of a NORC program is a practical and purposeful exercise that requires dedicated resources. Having adequate funding and the right people in place allows both the monitoring of progress and any needed revisions.
To keep expectations realistic, ensure clarity in responsibilities, and maximize efficiency, it will be important to:
Continue to Actions and Considerations
To conduct a useful evaluation, a NORC program commits adequate human and financial resources and develops efficient recordkeeping systems.
Determine the knowledge and technical experience you need to conduct a systematic and thorough evaluation. Think through the skills that will allow you to collect, analyze, and disseminate evaluation data.
Consider who among your staff, volunteers, and partners has the skills you need. If there are gaps, try to identify other stakeholders who may be able to provide the expertise you need, or help you find it.
Put one person in charge, but involve everyone. One individual is generally assigned to oversee the development and implementation of a project and the tools for evaluating it. That person should also be responsible for quality control.
At the same time, designing an evaluation and collecting relevant data are tasks to which many people can contribute. Because a NORC program is very much of the community, many players should have opportunities to provide feedback.
Create a system to monitor your data-collection activities. Identify each evaluation activity, the partner or staff person responsible for getting it done, the time it will require, and the resources being allocated to the work.
Document your evaluation. Establishing a record of how and why activities were selected and measured is important for interpreting results and guiding project planning and management. Careful documentation also enables future evaluations to build on systems that have already been tested.
Review your data-collection process regularly. It is possible that your collection methods could be more efficient or give you fuller information. If you can discover this early, you may be able to refine your data-collection strategy.
Where possible, draw on the data you are already collecting so that you do not add to your workload.
Data tracking forms and processes will not create themselves. Give adequate consideration to these time-consuming components of evaluation in order to develop a usable and accurate collection and reporting system. Continue to Guiding Principle #4
A NORC program involves its partners in the process of deriving meaning from its evaluation findings.
The wide range of perspectives that NORC program partners bring to any discussion can enrich an evaluation and inform next steps. Once all data have been collected, they should be reviewed with partners, and everyone should be involved in discussions about whether intended results have been achieved.
Collecting data is only a first step. Meaning still must be drawn from those data. Interpreting findings is a multi-step process that involves:
Continue to Actions and Considerations
A NORC program drills down to identify the component parts of a problem and to set priorities for action.
Use your data to gain insight. Drawing on the information in your logic model, you will begin to answer these questions:
Disseminate findings to program staff, senior residents, and other partners. Create a communications loop so that you can:
Sources to tell you more about past experiences in the community
Consider any limitations to the data. Did you collect all the data as intended in a consistent manner? Are there enough data and the right kind of data to draw conclusions? If not, identify the challenges.
Compare your new data to secondary sources or your own earlier studies. If your findings are consistent with accumulated knowledge, they will seem more reliable. If there are differences or contradictions, you’ll want to understand why.
Make sure core partners agree on key findings before sharing them with the larger community. This process generates trust within the partnership, builds confidence, and increases your chances of gaining support for future evaluations.
Evaluations generate recommendations, not definitive prescriptions.
A well-structured evaluation generates knowledge that the program communicates to the larger community and translates into action.
A good evaluation often leads to new understanding, identifies further challenges, and points the way towards fresh approaches. The ultimate goal is to strengthen the NORC program so that it is better equipped to fulfill its mission.
Ensuring that evaluation findings are disseminated and used appropriately requires deliberate effort. Once core partners agree on the meaning of findings, joint strategies need to be developed to report the results to the larger community and turn that meaning into action. The findings can support decisions about:
Continue to Actions and Considerations
A well-structured evaluation generates knowledge that the program communicates to the larger community and translates into action.
Communicate success. Disseminate findings to partners and stakeholders within the larger community. Showing that change is possible is very empowering. Make a special point of sharing information with people likely to be affected by what is revealed by the evaluation.
Use findings to generate support for the NORC program. Showing that your work makes a difference inspires other people to get involved. For example:
When your NORC program is dynamic and your measurements show positive outcomes, you have excellent building blocks to attract new and diverse sources of funding.
Evaluation is not an end to itself. Results should be used to provide a basis for making decisions about future activities.