Program Evaluations Reveal Vital Data, But They're Often Neglected

Program evaluation should be the cornerstone of community health and education projects, but evaluations are not widely done or are not done well, says Susan Scavo Gallagher, associate director of the Center for Violence and Injury Prevention (CVIP), a center within Health and Human Development Programs at EDC.

Although two major obstacles—a lack of time and funding—often preclude rigorous, multilevel evaluations, "some level of evaluation can and always should be performed," writes Gallagher in an editorial, "Program evaluation—balancing rigor with reality," in the June 2000 issue of the journal Injury Prevention. The editorial accompanied four articles that illustrate the possibilities and limitations inherent in community-based program evaluations.

Most funders understandably want outcome evaluations, but when control groups are impossible to create or the funding and length of studies don't allow for them, other types of evaluations can provide important information, says Gallagher.

For example, formative evaluations can be done that look at whether the program is being designed correctly, and process evaluations can ensure that the program is being implemented correctly.

Often, however, projects begin evaluations at too late a stage when their usefulness is limited or the wrong data are collected. When an evaluation is incorporated into the planning, "you're making sure the program is being implemented correctly and fine-tuning it as you go," says Gallagher.

Gallagher points to one group's experience as a lesson for others. The group, which produced an injury prevention curriculum for elementary school children, attempted to design outcome measures at the end of the program. But, many teachers never even used the curriculum in the first place or only partially implemented it. This would have emerged had a formative evaluation been part of the study design.

"Had they done a focus group on the teachers' impressions of the curriculum in a draft stage, they would not have spent money developing and disseminating something that people didn't want to use," she says.

Another type of evaluation that is often overlooked is qualitative evaluation.

"Quantitative evaluations are certainly important; everyone wants to see numbers, but qualitative evaluations enable you to explain how you got the numbers," says Gallagher. For example, a practitioner may want to do interviews with a community advisory board about their perceptions of a problem both before and after an intervention. "If the board's perceptions are still the same at the end of two years, that's telling you that the program may not have been implemented properly," says Gallagher.

Qualitative evaluations, which can include subjective information, have historically been looked at as "soft" by epidemiologists, says Gallagher. But such knowledge may be critical to getting to the root of a problem, she points out.

You can look at this year's data on arrests for drunk driving in a community and compare it to last year's data, but the change--or lack of change--in the number of arrests does not tell the full story, says Gallagher. "What you really need to understand is what happened after the arrests. Were those who were arrested actually going to court and being convicted? If not, why not?"

In an effort to change the behavior of the police and court system in one community, the local newspaper began publicizing drunk driving arrests and convictions, and printed the names of the offenders. This caused the police department and the courts to became more responsive to the problem, and more convictions resulted.

"If you succeed in changing the environment, a change in the numbers will reflect real-life changes," says Gallagher.