Sow and Reap: The Value of Outcome Harvesting Evaluation in Capturing Capacity Demonstrated by Beneficiaries of Short-Term Capacity Strengthening Projects

Summary:

Global health programs have, for years, aspired to strengthen the capacity of organizations and systems. The iterative and adaptive nature of capacity strengthening interventions and the complex nature of capacity itself makes measuring the outcomes of capacity strengthening activities an increasingly challenging endeavor. Conventional evaluation approaches fall short in capturing a complete picture of change in complex global health programs that involve organizations and systems. Outcome Harvesting is an emerging participatory methodology that embraces the complexity of capacity strengthening. Outcome Harvesting has the ability to capture intended and unintended project outcomes, whether positive or negative. Outcome Harvesting can be used alone or can complement other methods, depending on the needs and scope of the evaluation. This presentation will showcase Outcome Harvesting evaluation case studies from two Breakthrough ACTION country projects Nepal and Rwanda. The case studies will highlight the application of the Outcome Harvesting methodology as well as the diverse project outcomes that it captured. The case studies will reflect on the value of using Outcome Harvesting for evaluating the effects of short-term capacity strengthening projects.

Background/Objectives

Within the larger Breakthrough ACTION project are two social and behavior change capacity strengthening projects Nepal and Rwanda. Both capacity strengthening projects were implemented for short time periods between 2017 and 2020. The Nepal and Rwanda projects were similar in their implementation timelines and objectives of increasing/strengthening capacity of their intended beneficiaries toward coordination, design, implementation and evaluation of effective social and behavior change programs. The projects decided to employ the Outcome Harvesting evaluation methodology to capture whether and/or how, if at all, their activities influenced sustainable capacity shifts in their intended beneficiaries over the short implementation period.

Description Of Intervention And/or Methods/Design

Outcome Harvesting is an evaluation methodology used to identify, verify and analyze outcomes of capacity strengthening activities. The process collects evidence of capacity change ('outcomes') and works backward to assess whether and/or how a project contributed to that change. The methodology identifies key outcomes through a thorough review of existing documentation and engagement of key informants. For the complete list of outcomes, the evaluation requires verification of the outcomes with knowledgeable external and internal sources. The verified outcomes are then thematically analyzed to answer the key evaluation questions. For the Breakthrough ACTION Nepal and Rwanda projects, these were: 1) In what ways have beneficiaries demonstrated important changes in their capacity for improved SBC since the start of the project? 2) To what extent did the Breakthrough ACTION project outcomes exceed or fall short of the project's objectives? 3) How sustainable are the outcomes to which the Breakthrough ACTION project contributed?

Results/Lessons Learned

The evaluation in Rwanda has concluded while the evaluation for Nepal is slated for December 2019. The methodology affords the capture of capacity shifts demonstrated by beneficiaries either in the practice or systematization of a behavior toward effective SBC programming. Even with the short implementation time, Breakthrough ACTION Rwanda was able to capture 28 verified outcomes using Outcome Harvesting, out of which 23 were deemed to be sustainable shifts in capacity. These outcomes demonstrated capacity shifts across the individual, organizational and even system levels and highlighted ways the project met its objectives using the outcomes as narrative. Direct engagement of beneficiaries allowed for those outcomes to be captured of which even the country team was not aware. Such outcomes spoke to the beneficiaries' capacities to apply improved skills towards other avenues of their work, beyond the Breakthrough ACTION project.

Discussion/Implications For The Field

Even though true sustainability of capacity is hard to measure in short-term projects such as Nepal and Rwanda, the shift in recognizing the value and demonstrated use of innovative SBC approaches highlights the ways the beneficiaries contribute to the larger SBC landscape. The process offers insights into the value of using Outcome Harvesting for short capacity strengthening projects trying to qualitatively measure project outcomes. Outcome Harvesting affords projects a preview of how the capacity they built is being applied by the beneficiaries as well as informs future capacity strengthening work in those contexts based on the findings.

Abstract submitted by:

Ruchita Pillai - JHU

Arzum Ciloglu - JHU

Zoe Hendrickson - JHU

Sanjanthi Velu - JHU