Triangulating strengthened capacity: Using multiple research methods to assess changes in social and behavior change capacity in Nepal

Summary:

Breakthrough ACTION (BA) Nepal project aims to strengthen institutional and technical capacity of the Government of Nepal to design, implement, evaluate and coordinate effective social and behavior change (SBC) programs for health. In light of limitations to existing ways of measuring SBC capacity or evaluating capacity strengthening interventions, we describe the design and results of a multi-method approach used in Nepal to triangulate changes in individual, organization, and systems-level capacity in SBC for health programming. BA Nepal integrated 3 evaluation methods:1) mixed methods self-assessments of SBC capacity at baseline and endline with key stakeholders;2) most significant change (MSC) interviews with health professionals and elected representatives;and 3) a complexity-aware qualitative post-hoc review of program achievements, termed Outcome Harvesting, to identify and validate outcomes attributable to the program. Data gathered have been analyzed iteratively to triangulate findings. A comparison of baseline and endline SBC capacity self-assessment scores suggested that, at both local and provincial levels, overall SBC capacity as well as capacity subdomains such as knowledge management, coordination, and collaboration increased from baseline to endline. Common across MSC stories was the acknowledgement of the importance of situation analyses and interactions with communities to inform program design. Institutional changes identified in MSC stories included greater prioritization of SBC in annual planning and budgeting. Learnings from the combination of 3 distinct quantitative and qualitative approaches will be useful for future projects like BA Nepal aiming to measure and evaluate capacity strengthening activities to improve SBC programs for health.

Background/Objectives

The Breakthrough ACTION Nepal project aims to strengthen the institutional and technical capacity of the Government of Nepal to design, implement, evaluate and coordinate effective social and behavior change (SBC) programs for health. A comprehensive monitoring and evaluation approach is necessary to measure changes in capacity and attribute them to programmatic activities. However, approaches to measuring SBC capacity or evaluating capacity strengthening interventions rigorously are limited. We describe the design and results of a multi-method approach used in Nepal to triangulate changes in individual, organization, and institutional capacity in SBC for health programming at multiple levels of government.

Description Of Intervention And/or Methods/Design

BA Nepal integrated three evaluation methods: 1) mixed methods self-assessments of SBC capacity at baseline and endline with key stakeholders; 2) most significant change (MSC) interviews with health professionals and elected representatives; and 3) complexity-aware qualitative post-hoc review of program achievements, termed Outcome Harvesting (OH), to identify and validate outcomes attributable to the program. Capacity assessments, key informant interviews, and OH were conducted at each level of government where the project was working: four local project-specific sites, the provincial government, and the federal SBC counterpart, the National Health Education Information Communication Centre. Data gathered have been analyzed iteratively to triangulate findings. Quantitative assessment scores were integrated with qualitative data gathered through key informant interviews. MSC stories were documented and vetted through participatory workshops and compared with quantitative assessments. OH, currently being implemented, builds on these methods conducting a global post-hoc document review of project accomplishments.

Results/Lessons Learned

A comparison of baseline and endline SBC capacity self-assessment scores suggested that, at both local and provincial levels, overall SBC capacity as well as capacity subdomains such as knowledge management, coordination, and collaboration increased from baseline to endline. Among the 22 MSC stories collected and ten selected by key stakeholders as most significant, participants emphasized an increased knowledge and understanding of SBC, with an emphasis on how to identify community needs and target groups for intervention. Common across stories was the acknowledgement of the importance of situation analyses and interactions with communities to inform program design. Institutional changes identified in MSC stories included greater prioritization of SBC in annual planning and budgeting. Outcome Harvesting results will be presented to complement these findings, with a focus on a verification of Breakthrough ACTION Nepal's concrete contributions to changes in SBC capacity at the individual, organizational, and systems levels.

Discussion/Implications For The Field

Through the combination of three distinct quantitative and qualitative approaches to measuring SBC capacity and evaluate capacity strengthening activities, the Breakthrough ACTION Nepal project had access to invaluable sources of qualitative and quantitative data that informed programmatic activities, scale up, and sustainability strategies. Multiple methods enable data triangulation, with findings illuminating the weaknesses of using one approach independently. Learnings from this integration will be useful for future projects aiming to measure and evaluate rigorously capacity strengthening activities to improve SBC programs for health.

Abstract submitted by:

Lokesh Bhatt - JHU CCP

Zoe Hendrickson - JHU

Pranab Rajbhandari - JHU CCP

Shreejana  K.C. - JHU CCP

TrishAnn Davis - JHU

Leticia  De Los Rios - JHMI

Sanjanthi Velu - JHU