Learnings

What is Program Evaluation?

A Look Into the CX100-Abacus Evaluation Partnership

by Bola Bilikis Yusuf, Miranda Wenhold, Lisa Maria Cross

June 29, 2023

Abacus Evaluation (Abacus) works with the Carolina Across 100 (CX100) team to evaluate the CX100 initiative and its programs. Abacus is a Research Service Center within the Center for Health Equity Research (CHER) at the UNC School of Medicine. Abacus is committed to amplifying the impact and effectiveness of programs, centers, and projects. Abacus’ team brings extensive experience in researching and evaluating diverse programs at local, state, and national levels.

What is Program Evaluation?

Program evaluation assesses the value of a program, project, or policy. It helps to determine whether a program, policy, or project is meeting its intended goals and attaining measurable outcomes. It utilizes systematic methods to collect and analyze data to assess program effectiveness to ensure continuous improvement. (Source)

Importance of Program Evaluation

Evaluation is important because it provides a means to monitor progress toward program goals and offers opportunities for learning and improvement. Evaluation planning should occur alongside program planning to ensure there is a structured and transparent evaluation plan.

The current CX100 evaluation comprises feedback sessions and surveys that have led to process improvements, capturing progress on milestones and shared goals, and ultimately the use of data to identify solutions for lasting community change.

Specific to the CX100 evaluation, program evaluation allows for (1) the tracking of progress towards its goal of building partnerships across communities in North Carolina ensuring that activities and processes are implemented as planned, and (2) continuous improvements of these processes as they occur.

The different types of evaluation can assess different aspects of a program including the short-term, intermediate, and long-term outcomes.

The Abacus team applies different types of program evaluation to the CX100 initiative

  • Formative evaluation: Conducted to ensure that a program is feasible and can be effectively implemented. It is particularly helpful in the early stages of building a program.

    • Aims to provide feedback and insights to improve the intervention’s design, content, and implementation.
    • Helps identify strengths, weaknesses, and areas for refinement before the intervention is fully implemented by gathering data and input from stakeholders.

  • Process evaluation: Focuses on the implementation and delivery of an intervention or program. This evaluation approach is used to determine whether the program activities are being implemented as planned.

    • The findings can be used to inform future program activities.
    • Aims to understand the factors that influence program implementation, identify areas for improvement, and provide insights into the relationship between program delivery and outcomes.

  • Outcome evaluation: Assesses the progress of a program towards its intended outcome objectives in the target populations.

    • Method used to assess the intermediate effects or outcomes of an intervention or program.
    • Determines whether the intervention has achieved its desired results and to identify any unintended consequences.

  • Impact evaluation: Impact evaluation is a systematic process used to assess the effects or outcomes of an intervention or program. It involves collecting and analyzing data to determine the extent to which the intervention has achieved its intended goals and to understand the factors that contributed to its success or failure.

    • The findings can inform decision-making, improve program design and implementation, and enhance accountability and learning within organizations.

Evaluating the CX100 Initiative

Abacus’ evaluation work maps onto the Collective Impact Model used by the CX100 initiative to address different societal problems.​ The Collective Impact Model is a model used to address complex issues requiring large-scale and sustainable improvements. Abacus conducts an evaluation that ensures adherence to this model and its components, which includes a common agenda, continuous communication, shared measurement system, mutually reinforcing activities, and a backbone organization.

Abacus partners with the CX100 team to carry out its vision to form meaningful partnerships with North Carolinians to address longstanding challenges exacerbated by, and new challenges associated with, COVID-19. The Abacus team works in partnership with CX100 stakeholders to ensure evaluation processes are efficient, time-effective, and integrated into program planning and adaptation.

Carolina Across 100 Evaluation: Conceptual Framework

Goal: Carolina Across 100 will form meaningful partnerships in each of North Carolina’s 100 counties to address longstanding challenges exacerbated by, and new challenges associated with, COVID-19.

Conceptual Framework

Photo credit: Steve Remich, Root Media

OSOW teams attending Forum 3 in March 2023

Evaluation activities include:

This survey tracks the community teams’ progress towards their goals by asking about their ability to collect and measure data, the strategies they use to reach out to Opportunity Youth, and their groups/networks.

This survey helps the CX100 team understand how useful their different forms of support are to the community teams by asking about the usefulness of the resources shared, the communications from the CX100 team, and the monthly program manager meetings.

These surveys assess knowledge gained, event satisfaction, and usefulness of event content. Survey findings are shared with the CX100 team to improve subsequent events.

Feedback Data Briefs for Process Improvement

Tracking the CX100 team’s adherence to the Collective Impact Model.

Tracking collaborative teams’ milestones and programmatic progress.

The Takeaway

The CX100 program evaluation ensures that program activities and processes are carried out as closely to plan as possible and allows for continuous improvement by providing information about the community teams’ experiences to the CX100 team.

The OSOW program will end in December 2023. As additional CX100 programming is announced and implemented, the Abacus team is proud to continue to contribute to the project’s goal of responding to the state’s most pressing challenges.

References

Recent Posts