An Audit of Evidence Review Group Criticisms of Systematic Literature Reviews Conducted To Inform Manufacturers' Submissions To The National Institute for Health and Care Excellence

Abstract

Several systematic literature reviews (SLRs) are required to inform manufacturers' single technology appraisal (STA) submissions to the National Institute for Health and Care Excellence (NICE). An independent Evidence Review Group (ERG) assesses each submission, including the SLRs, with a view to highlighting strengths and weaknesses. As SLRs can be time-consuming and resource-intensive, to help maximise their impact we aimed to review the feedback that ERGs have previously provided on SLRs presented in manufacturers' submissions (MS), identify the methodological and reporting approaches most likely to be criticised, and make recommendations on avoiding these criticisms. MS and ERG reports for STAs submitted to NICE between January 2015–May 2017 were downloaded from the NICE website. Key information was extracted from each MS and ERG report regarding the methodological and reporting approach taken and any relevant ERG comments. Clinical SLRs and SLRs on cost-effectiveness studies were more likely to be criticised than SLRs on health-related quality of life or cost and resource use data. Common methodological criticisms included unaccounted for discrepancies between the scope of the clinical SLR and the final NICE scope, not using or modifying published search filters, and the simultaneous querying of MEDLINE and Embase. The most common reporting criticism was not describing the reviewer process for study selection and data extraction. Methodological approaches criticised by ERGs in some MS were not always criticised by the same or different ERGs in other MS. Despite critique being common, in most cases the ERG did not believe that the SLR limitations had resulted in relevant studies being missed. It is possible to derive recommendations regarding SLR methodology that should help to avoid criticism from NICE ERGs. The importance of high quality and transparent SLR write-ups is also clear; this should be easily achieved with the implementation of quality control processes.