Evaluation of a Complex Health Intervention in Zambia: The case of the Better Health Outcome through Mentorship and Assessment (BHOMA) Applying system wide approaches to measuring health system strengthening: Essential Markers and Impact Pathway

Abstract

Introduction: In many low income countries the delivery of quality health services is hampered by health system-wide barriers which are often interlinked, however empirical evidence on how to assess the level and scope of these barriers is scarce. It has been recognised that taking a more comprehensive approach to assessing these barriers is more likely to provide lessons on what works and why. WHO has been advocating the use of systems wide approaches such as systems thinking to guide intervention design and evaluation. This thesis reports system-wide assessment of a complex health system intervention in Zambia known as Better Health Outcome through Mentorship and Assessment (BHOMA) that aimed to improve service quality at the health facility and influence service demand from the community. Methodology: This study is nested within a cluster randomised trial of the BHOMA intervention that aims to strengthen the health system in three rural districts covering 42 health facilities in Zambia. The main trial has a stepped wedge design where the intervention is being rolled-out to all the 42 health facilities over a period of 4 years. A baseline health facility survey was done in 2011. This was followed by a 12 months post-intervention evaluation survey. At the time of the follow up survey 24 health facilities had received the intervention while 18 had not. Data collection used both quantitative and qualitative methods. The study was guided by a systems thinking theoretical framework which was inspired by the WHO building blocks for health system strengthening. Results: The baseline survey validated tools and indicators for assessing health system building blocks. Research paper 2 applied an innovative measure of health worker motivation which was initially applied in Kenya. The results showed that this simple tool was reliable with cronbach’s alpha of 0.73 for the 21 item measures of health workers’ motivation. Baseline assessment of health worker motivation showed variation in motivation score based on gender and access to training. Research paper 3 tested and applied a new tool for measuring health systems governance at health facility level. The new tool for measuring governance was reliable with the 16 item one side cronbach’s alpha ranging between 0.69-0.74.The tool was simple to use and found to be applicable in the Zambian health care setting. A balanced scorecard approach was applied to measure the baseline health system characteristics for the target districts. Differences in performance were noted by district and residence in most domains with finance and service delivery domains performing poorly in all study districts. Regression modelling showed that children’s clinical observation scores were negatively correlated with drug availability (coeff 20.40, p = 0.02) while Adult clinical observation scores were positively association with adult service satisfaction score (coeff 0.82, p = 0.04) Baseline qualitative results are presented in paper 5. The results showed close linkages between health system building blocks. Challenges noted in service delivery were linked to human resources, medical supplies, information flow, governance and finance building blocks either directly or indirectly. The 12 months post intervention survey applied both quantitative and qualitative methods. Research paper 6 presents 12 months post intervention quantitative results applying the balanced scorecard approach as at baseline. Comparison was made between the control and intervention health facilities. The results showed significant mean differences between intervention (I) and control (C) sites in the following domains: Training domain (Mean I:C;87.5.vs 61.1, mean difference 23.3,p=0.031),adult clinical observation domain (mean I:C;73.3 vs.58.0, mean difference 10.9,p=0.02 ). The 12 months post intervention qualitative evaluation applied systems thinking approach and the conceptual framework developed before the intervention. The findings are presented in research paper 7. The overall results showed that the community had accepted the intervention with increasing demand for services reported in all sites where the BHOMA intervention was implemented. The indications were that in the short term there was increased demand for services but the health workers’ capacity was not severely affected. However, from a systems thinking perspective, it was clear that several unintended consequences also occurred during the implementation of the BHOMA. Conclusion: In evaluation of complex interventions such as the BHOMA attention should be paid to context. Using system wide approaches and triangulating data collection methods seems to be important to successful evaluation of such complex intervention.