Evaluating multi-criteria decision-making in health technology assessment

Abstract

ObjectivesLiterature has emphasized the role of multi-criteria decision analysis (MCDA) in improving consistency and comprehensiveness of health technology assessment (HTA), however there has been no evidence to indicate this as true. We sought to understand if MCDA ensures systematic and comprehensive HTA. MethodsIn Canada, the Ontario Health Technology Advisory Committee (OHTAC) was created as a provincial portal for funding recommendations regarding introduction of nondrug health technologies (HTs). In 2009, a “Decision Determinants Sub-Committee” developed a MCDA framework outlining specific criteria to be considered. We undertook mixed-methods analyses to investigate funding recommendation reports, including types of evidence used to evaluate criteria, strength and quality of evidence, and the relationship between criteria and funding recommendations. ResultsThe findings indicated that comprehensive and consistent use of evidence is lacking even with MCDA. Effectiveness was almost always considered, whereas ethical values were considered in only one-third of reports. Evaluation of quality and strength of evidence declined over the years, and value for money was often not assessed. ConclusionsWe have proposed two reasons for the inconsistent evaluation: (1) insufficient planning for evaluation (i.e., what needs to be evaluated), and (2) lack of evidence. Further research could focus on embedding economic and societal/ethical considerations in the MCDA framework, and specifying a system to evaluate the efficacy of these variables. Such an approach may encourage comprehensive and consistent consideration of diverse types of evidence in HTA.