Methodological approaches to impact evaluation of integrated HIV programs and services: quasi-experimental methods and multilevel models applied to two case studies in Peru

Abstract

Tight health budgets are increasingly strained in Low and Middle Income Countries (LMICs) through shifting burdens of disease, placing new constraints on how HIV is controlled. As HIV transitions from an acute to a chronic disease, control policies which potentially improve effectiveness, minimise cost or, both warrant investigation. Integration of HIV programs or services with other health system components is one such policy.Integration is a general term for processes of aligning two or more system components towards common goals. Alignment is in response to system inefficiency driven through differentiation of those components. Contention exists, however, as to how effective different integration modalities are within health systems. This is in part due to inherent methodological issues facing empirical evaluation of integration.Four issues limit the scope of evaluating the comparative impact of HIV integration modalities. Firstly, integration is unlikely to be randomised given it is often implemented fixed to health establishments with limited variation in how individuals access it. Secondly, integration is most frequently applied to places as opposed to individuals, which results in clustering. Thirdly, evaluation of integration is frequently performed ex-post meaning policy analysts have little control over the evaluation process. Finally, evaluation is complicated by contention around conceptualisations of integration itself and appropriate indicators reflecting differences in modalities being compared.This thesis aims to understand and address issues in evaluating integration. It primarily explores how methodological issues may be overcome through the combined use of quasi-experiments and MultiLevel regression Models (MLMs). Quasi-experiments exploit aspects of study design to recreate some benefits of randomisation. A class of quasi-experiment, the Propensity Score (PS); created through regression is used to reduce measured systematic biases. MLMs are regression models which allow inclusion of cluster level information. PSs have traditionally been constructed using naive, non-MLM regression. By instead constructing PSs using MLMs, known as a MLM-PS, non-randomisation and clustering may be simultaneously addressed. MLM-PS is sufficiently flexible to be applied ex-post with relatively little information on policy assignment compared to other quasi-experiments. The MLM-PS approach is potentially able to overcome methodological issues inhibiting HIV integration evaluation.To address issues of selecting plausible indicators for evaluation a theoretical framework is constructed for the thesis. It builds upon existing work deconstructing comparative integration modalities mapping them across a standard framework for evaluating quality in health programs. Indicators are described as reflecting the relative impact of comparable integration modalities through plausible mechanisms of action. Feasibility of MLM-PS and the theoretical framework are explored through application to two case studies of HIV integration. Both case studies originate from an implementation of integrated HIV services in Peru. Integration in both case studies has been applied non-randomly and clustered at district level with evaluation conducted ex-post reflecting the common issues described above.The first case study examines the impact of increasing HIV and Antenatal Care (ANC) integration, on the use of essential testing services. MLM-PS is shown to demonstrably reduce bias while controlling for clustering and including important cluster level information in the PS. Using the method, increased integration is shown to be highly effective. The second case study examines the impact of HIV/TB integration on essential testing services. MLM-PS method is also shown to reduce biases incorporating control for clustering, while using conceptually different data. Using the method, higher levels of integration are shown to have mixed results. As a demonstration of robustness, different iterations of MLM-PS for both case studies are explored. Here the importance of using MLMs in the PS when clustering is present is also established.MLM-PS is concluded to be a useful pragmatic alternative when more robust methods are not available. It is flexible enough to be applied when data are limited, and little control is available over evaluation. It is suggested as a potential tool for policy analysts and policy makers to improve estimates of integration impact, or indeed other similar polices. This is with a view to strengthening evidence-based decision making on policies critical to the control of HIV.