National Institute For Health And Care Excellence Technology Appraisal Research Articles

Evidence Following Conditional NICE Technology Appraisal Recommendations: A Critical Analysis of Methods, Quality and Risk of Bias.

The National Institute for Health and Care Excellence (NICE) may approve health technologies on condition of more evidence generated only in research (OiR) or only with research (OwR). NICE specifies the information needed to comply with its request, although it may not necessarily guarantee good quality and timely evidence for re-appraisal, before reaching a final decision. This study aimed to critically appraise the methods, quality and risk of bias of evidence generated in response to NICE OiR and OwR technology appraisal (TA) and highly specialised technologies (HSTs) recommendations. NICE TAs (between March 2000 and September 2020) and HST evaluations (to October 2023) of medicines were reviewed. Conditional recommendations were analysed to identify the evidence requested by NICE for re-appraisal. The new evidence was analysed for compliance with NICE's request and assessed using the Cochrane Collaboration's tools for risk of bias in randomised trials and the ROBINS-I tool for non-randomised evidence. NICE made 54 conditional recommendations from TAs (13 OiR and 41 OwR) and five conditional recommendations for HSTs (all OwR). Of these, 16 TAs presented additional evidence for re-appraisal (9 OiR [69%] and 7 OwR [17%]) and three HSTs (3 OwR [60%]). Two of the nine re-appraised TAs with OiR recommendation and four of the seven OwR complied fully with NICE's request for further evidence, while all three from the HSTs complied. The majority of re-appraised TAs and HSTs included evidence that was deemed to be at serious, high, moderate or unclear risk of bias. Among the 26 randomised controlled trials from TAs assessed, eight were categorised as having low risk of bias in all domains and ten had at least one domain as a high risk of bias. Reporting was unclear for the remainder. Twenty-two non-randomised studies, primarily single-arm studies, were susceptible to biases mostly due to the selection of participants and to confounding. Two HSTs provided evidence from randomised controlled trials which were classified as unclear or high risk of bias. All non-randomised evidence from HSTs were categorised as moderate or serious risk of bias. There is widespread non-compliance with agreed data requests and important variation in the quality of evidence submitted in response to NICE conditional approval recommendations. Quality standards ought to be stipulated in respect to evidence contributing to re-appraisals following NICE conditional approval recommendations.

Trends in NICE technology appraisals of non-small cell lung cancer drugs over the last decade.

The aim of this study is to analyse the trends in technology appraisals for non-small cell lung cancer (NSCLC) treatments performed by the National Institute for Health and Care Excellence (NICE) over the last ten years. A systematic search was conducted for single technology appraisals of NSCLC drugs in the online NICE database from 2012 to 2022. Search terms used were 'non small cell lung cancer', and 'NSCLC'. Appraisals that were under development or terminated as well as multiple technology appraisals were considered out of scope. In the 30 included appraisals for targeted therapies and immunotherapies within NSCLC, a total of 53 different comparators were included by NICE for 41 assorted indications or subgroups. Partitioned survival models were most frequently used, often including three health states and time horizons of up to 30 years. Throughout the decade the use of indirect comparisons was high and became more established and complex over time. Of all appraisals, 90% positively recommended the treatment for use in the UK. Technology appraisals became more complex over time due to the emergence of targeted therapies and immunotherapies, leading to multiple different indications, subpopulations and comparators that needed to be included in appraisals. Partitioned Survival Analysis (PartSA) models became the cornerstone within NSCLC, with time horizons up to 30 years and over time methods for indirect treatment comparisons became more established. The majority of the appraisals resulted in a positive recommendation for reimbursement.

An empirical ethics study of the coherence of NICE technology appraisal policy and its implications for moral justification

BackgroundAs the UK’s main healthcare priority-setter, the National Institute for Health and Care Excellence (NICE) has good reason to want to demonstrate that its decisions are morally justified. In doing so, it has tended to rely on the moral plausibility of its principle of cost-effectiveness and the assertion that it has adopted a fair procedure. But neither approach provides wholly satisfactory grounds for morally defending NICE’s decisions. In this study we adopt a complementary approach, based on the proposition that a priority-setter's claim to moral justification can be assessed, in part, based on the coherence of its approach and that the reliability of any such claim is undermined by the presence of dissonance within its moral system. This study is the first to empirically assess the coherence of NICE’s formal approach and in doing so to generate evidence-based conclusions about the extent to which this approach is morally justified.MethodsThe study is grounded in the theory, methods and standards of empirical bioethics. Twenty NICE policy documents were coded to identify and classify the normative commitments contained within NICE technology appraisal policy as of 31 December 2021. Coherence was systematically assessed by attempting to bring these commitments into narrow reflective equilibrium (NRE) and by identifying sources of dissonance.FindingsMuch of NICE policy rests on coherent values that provide a strong foundation for morally justified decision-making. However, NICE’s formal approach also contains several instances of dissonance which undermine coherence and prevent NRE from being fully established. Dissonance arises primarily from four sources: i) NICE’s specification of the principle of cost-effectiveness; ii) its approach to prioritising the needs of particular groups; iii) its conception of reasonableness in the context of uncertainty, and iv) its concern for innovation as an independent value.ConclusionAt the time of analysis, the level of coherence across NICE policy provides reason to question the extent to which its formal approach to technology appraisal is morally justified. Some thoughts are offered on why, given these findings, NICE has been able to maintain its legitimacy as a healthcare priority-setter and on what could be done to enhance coherence.

Cross-sectional analysis of use of real-world data in single technology appraisals of oncological medicine by the National Institute for Health and Care Excellence in 2011–2021

ObjectivesThis study aims to identify how real-world data (RWD) have been used in single technology appraisals (STAs) of cancer drugs by the National Institute for Health and Care Excellence (NICE).DesignCross-sectional...

Assessment of quality of data submitted for NICE technology appraisals over two decades

BackgroundThe National Institute for Health and Care Excellence (NICE) pioneered the Health Technology Assessment (HTA) processes and methodologies. Technology appraisals (TAs) focus on pharmaceutical products and clinical and economic data,...

The international HTA database returned incomplete search results for NICE technology appraisals: An exploratory study and discussion of the implications

AbstractIntroductionThe International Network of Agencies for Health Technology Assessment (INAHTA) database offers a single point of access for identifying technology appraisals, in contrast to searching multiple websites directly. The aim of this research is to compare the coverage of the INAHTA and Centre for Reviews and Dissemination (CRD) Health Technology Assessment (HTA) databases with direct searching on the National Institute for Health and Care Excellence (NICE) website to identify Technology Appraisals published by NICE.MethodsNICE Technology Appraisals were downloaded from the NICE website (April 2022). Technology Appraisals were randomized and the first 20 Technology Appraisals constituted data for analysis. The INAHTA and CRD HTA databases were searched to determine if the 20 Technology Appraisals available on the NICE website were also available for retrieval.ResultsCoverage was incomplete. INAHTA: 15 of 20 Technology Appraisals (75%) were not identified via full title or intervention‐specific searches. CRD HTA: 7 of 12 Technology Appraisals (58%) that were published before the last update of the database were not identified.ConclusionFindings indicate that researchers seeking to identify NICE Technology Appraisals should search the NICE website directly. How this finding impacts identification of guidance from other agencies should be evaluated.

OP154 Horses For Courses: Developing A Proportionate Approach To Health Technology Assessment In England

IntroductionThe number and range of technologies that the National Institute for Health and Care Excellence (NICE) evaluates has never been greater, and with that comes an increasing requirement for capacity. However, not all technologies need the full intensity of the current standard evaluation process. This presents an opportunity to differentiate evaluation processes, and in doing so release capacity for more evaluations. NICE has embarked on a project to develop proportionate appraisal processes, allowing promising medicines to move through refined processes that better match the specific needs of individual evaluations.MethodsThe proportionate approach to technology appraisals (PATT) project was initiated in June 2022. Multiple work strands were established to develop and test potential proportionate approaches, focusing mainly on streamlined approaches for appraisals in which a lighter-touch approach is sufficient. By creating an accountability framework which empowered staff across NICE, new processes were developed using test-and-learn principles: piloting key concepts using retrospective reviews and live appraisals and adjusting the approach based on their findings. The impacts of each approach on NICE, on stakeholders and on the individual appraisals were monitored and assessed.ResultsThe project has identified a range of procedural, methodological and operational improvements across the NICE technology appraisal process. Substantial efficiencies have been found, including consolidation of activities, reducing duplication of effort and minimizing disproportionate steps. The test-and-learn approach has proved successful, both in rapidly identifying unsuccessful ideas and in refining and adjusting processes in light of new information and challenges.ConclusionsThe proportionate approach to technology appraisals project represents an important part of improving and streamlining NICE’s approach to health technology assessment, reflecting the increasing demands on the program. Developing proportionate approaches allows efficient use of limited evaluation resources to continue supporting rapid, evidence-based access to innovative technologies. The project also provides a valuable demonstration of an agile, flexible approach to process improvement, paving the way to rapid, dynamic updates in the future.

Evaluation of US oncology electronic health record real-world data to reduce uncertainty in health technology appraisals: a retrospective cohort study

ObjectivesExamine whether data from early access to medicines in the USA can be used to inform National Institute for Health and Care Excellence (NICE) health technology assessments (HTA) in oncology.DesignRetrospective...

HTA61 Do NICE Technology Appraisals Account for Unmet Medical Need in Oncology?

In appraising oncology therapies, National Institute for Health and Care Excellence (NICE) evaluates clinical and cost-effectiveness vs standard of care; the impact of unmet need on recommendations is unclear. We surveyed NICE appraisal outcomes using line of therapy (LoT) and comparator therapy class as surrogate measures of unmet need.

EE87 Are Landmark Survival Models Accepted in National Institute for Health and Care Excellence Health Technology Evaluations?

Novel immuno-oncology therapies have necessitated increasingly complex survival models in health technology evaluations. The National Institute for Health and Care Excellence (NICE) Technical Support Document 21 (TSD21, November 2020) discusses several flexible models, including landmark survival models (LSMs). Considering these advancements in survival modelling, we investigated the use of LSMs in NICE Technology Appraisals (TAs) in oncology.

The estimation of health state utility values in rare diseases: do the approaches in submissions for NICE technology appraisals reflect the existing literature? A scoping review

BackgroundRare diseases negatively impact patients’ quality of life, but the estimation of health state utility values (HSUVs) in research studies and cost–utility models for health technology assessment is challenging.ObjectivesThis study compared the methods for estimating the HSUVs included in manufacturers’ submissions of orphan drugs to the National Institute for Health and Care Excellence (NICE) with those of published studies addressing the same rare diseases to understand whether manufacturers fully exploited the existing literature in developing their economic models.MethodsAll NICE Technology Appraisal (TA) and Highly Specialized Technologies (HST) guidance documents of non-cancer European Medicines Agency (EMA) orphan medicinal products were reviewed and compared with any published primary studies, retrieved via PubMed until November 2020, and estimating HSUVs for the same conditions addressed in manufacturers’ submissions.ResultsWe identified 22 NICE TA/HST appraisal reports addressing 19 different rare diseases. Sixteen reports presented original HSUVs estimated using EQ-5D or Health Utility Index (n = 12), direct methods (n = 2) or mapping (n = 2), while the other six included values obtained from the literature only. In parallel, we identified 111 published studies: 86.6% used preference-based measures (mainly EQ-5D, 60.7%), 12.5% direct techniques, and 2.7% mapping. The collection of values from non-patient populations (using ‘vignettes’) was more frequent in manufacturers’ submissions than in the literature (22.7% vs. 8.0%).ConclusionsThe agreement on methodological choices between manufacturers’ submissions and published literature was only partial. More efforts should be made by manufacturers to accurately reflect the academic literature and its methodological recommendations in orphan drugs submissions.

Incremental benefits of novel pharmaceuticals in the UK: a cross-sectional analysis of NICE technology appraisals from 2010 to 2020

ObjectivesTo evaluate the incremental value of new drugs across disease areas receiving favourable coverage decisions by the UK’s National Institute for Health and Care Excellence (NICE) over the past decade.Design,...

POSC143 Comparison of Model Structures Used in NICE and ICER Cost Effectiveness Evaluations

The Institute for Clinical and Economic Review (ICER) in the United States and the National Institute for Health and Care Excellence (NICE) in United Kingdom produce evidence-based reports on cost-effectiveness of health technologies to inform policymaking and reimbursement decisions. We aimed to compare the degree to which the structure of cost-effectiveness models submitted to NICE and developed by ICER differ. In 2020–2021, ICER published reports in nine disease areas: multiple myeloma, hereditary angioedema, high cholesterol, atopic dermatitis, sickle cell disease, ulcerative colitis, migraine, non-alcoholic steatohepatitis, and beta thalassemia. We identified 17 NICE technology appraisals in the same disease areas published in 2016–2021. Full-text screening of committee papers and ICER reports was conducted by one investigator and abstracted data was validated by a second. Information on treatment comparators, model structure, and approach were extracted. Evidence review group commentaries were considered. Comparisons were based on the adopted model structure and approach to elicit inputs. Data sources and results were not relevant. Among the disease areas, ICER and NICE used different model structures in three, broadly similar structures in four, and similar structures in two evaluations. Among the models with different structures, NICE were more complex with larger numbers of health states (2/3). Among models with broadly similar structures, NICE implemented approaches that more closely captured the effect of new treatments (2/4) with more complex data elicitation approaches (3/4). Among the models with very similar model structures, the key difference was the means of treatment effect measure. Despite NICE and ICER having similar guidelines for conducting cost-effectiveness assessments, there are differences among model structures within the same disease areas. Availability of trial patient-level data to inform submissions to NICE and manufacturers’ intent to capture treatment effect to the best possible extent could be reasons behind the discordance.

Real-world data from expanded access programmes in health technology assessments: a review of NICE technology appraisals

ObjectivesTo quantify and characterise the usage of expanded access (EA) data in National Institute for Health and Care Excellence (NICE) technology appraisals (TAs). EA offers patients who are ineligible for...

Audit of data redaction practices in NICE technology appraisals from 1999 to 2019

ObjectivesTo assess the extent and type of data redaction in all active technology appraisals (TA) and highly specialised technology (HST) evaluations issued by the National Institute for Health and Care...

PNS70 Societal Cost Savings Are Often Not Considered in NICE Decision Making

Broader societal cost savings offered by new technologies, for example from reduced absenteeism and increased productivity at work, less caregiver time or reduced costs associated with travelling to a hospital to receive treatment, are not typically considered by the National Institute for Health and Care Excellence (NICE) when making recommendations for funding. Instead, NICE refers to ‘exceptional cases’ where these may be accepted. The research presented here explores NICE and evidence review group (ERG) critiques of societal cost-saving analyses in submission documents, to understand how these are currently evaluated and their impact on recommendations. We reviewed all technology appraisals (TAs) and highly specialised technology (HST) assessments published between April 2013, when the current NICE methods guidance was issued, and July 2020. Manufacturers included societal cost considerations within economic models in 6% (19/331) of NICE TAs and HST assessments. Societal cost-saving analyses were presented for a range of (mostly chronic) conditions, most commonly neurological conditions or rare diseases. In only two of the 19 appraisals (11%), the inclusion was agreed with NICE in scoping materials. In 84% (16/19), the societal cost-saving analyses were incorporated into scenario analyses; only three submissions considered different elements of societal costs in their base case. NICE commented on societal costs in only 26% of appraisals (5/19; 2/5 included analyses in the base case), while the ERG commented on 47%. NICE accepted the analyses in only two of the five appraisals (HST3 and TA325). In appraisals of new treatments where NICE did not comment on the submitted societal cost analyses, these are unlikely to have contributed heavily to the final funding decision. Given the low acceptance rate of societal cost savings by NICE, funding decisions may not reflect those that would most benefit society.

PNS82 The Methods for Incorporating Carer HRQoL Should be More Clearly Defined By NICE

The broader societal benefits offered by new technologies in terms of quality-adjusted life-years (QALYs) accrued by carers (e.g., a partner, parent, or family member) are accepted for inclusion in the Reference Case for National Institute for Health and Care Excellence (NICE) technology appraisals (TAs) and highly specialised technology (HST) guidance. The research presented here reviews all published TAs and HST assessments to identify those that considered the impact of an intervention on QALYs for carers and how these have incorporated carer health-related quality of life (HRQoL). We reviewed all TAs and HST assessments published between April 2013, when the current NICE methods guidance was issued, and July 2020. Manufacturers included carer HRQoL in their economic models in 5% (17/331) of NICE TAs and HST assessments. Most of these (9/17) concerned neurological conditions; 7/9 were for multiple sclerosis, 6 of which used utility decrements for carers of patients with Alzheimer’s disease. In 2/17 submissions, carer disutilities were derived using vignettes valued through the EuroQol five-dimension questionnaire (EQ-5D) visual analogue scale (VAS). One submission included the QALY loss to a family due to the premature death of a child. The remaining studies used disutilities from the literature. Of the 17 submissions, 9 included carer HRQoL in the base case only, and 2 in the base case and scenario analyses. NICE accepted inclusion of disutilities for carers in all of these. Six included carer HRQoL only in a scenario analysis. NICE rejected 4 of them. Carer HRQoL benefits/analyses are not routinely submitted for NICE decision-making. In the cases where NICE accepted it, various methodologies were applied besides EQ-5D to derive utility decrements. NICE should provide further guidance and evidence to support a consistent approach to including carer/family health outcomes in appraisals.

The use of UK primary care databases in health technology assessments carried out by the National Institute for health and care excellence (NICE)

BackgroundReal world evidence (RWE) is becoming more frequently used in technology appraisals (TAs). This study sought to explore the use and acceptance of evidence from primary care databases, a key source of RWE in the UK, in National Institute for Health and Care Excellence (NICE) technology assessments and to provide recommendations regarding their use in future submissions.MethodsA keyword search was conducted relating to the main primary care databases in the UK on the NICE website. All NICE TAs identified through this search were screened, assessed for duplication and information on the data source and the way the data was used in the submission were extracted. Comments by the evidence review group (ERG) and the appraisal committee were also extracted and reviewed. All data extraction was performed by two independent reviewers and all decisions were reached by consensus with an additional third reviewer.ResultsA total of 52 NICE TAs were identified, 47 used the General Practice Research Database /Clinical Practice Research Datalink (GPRD/CPRD) database, 10 used The Health Improvement Network (THIN) database and 3 used the QResearch databases. Data from primary care databases were used to support arguments regarding clinical need and current treatment in 33 NICE TAs while 36 were used to inform input parameters for economic models. The databases were sometimes used for more than one purpose. The data from the three data sources were generally well received by the ERGs/committees. Criticisms of the data typically occurred where the results had been repurposed from a published study or had not been applied appropriately.ConclusionsThe potential of UK primary care databases in NICE submissions is increasingly being realised, particularly in informing the parameters of economic models. Purpose conducted studies are less likely to receive criticism from ERGs/committees, particularly when providing clinical input into cost effectiveness models.

Assessment of technical errors and validation processes in economic models submitted by the company for NICE technology appraisals.

Economic models play a central role in the decision-making process of the National Institute for Health and Care Excellence (NICE). Inadequate validation methods allow for errors to be included in economic models. These errors may alter the final recommendations and have a significant impact on outcomes for stakeholders. To describe the patterns of technical errors found in NICE submissions and to provide an insight into the validation exercises carried out by the companies prior to submission. All forty-one single technology appraisals (STAs) completed in 2017 by NICE were reviewed and all were on medicines. The frequency of errors and information on their type, magnitude, and impact was extracted from publicly available NICE documentation along with the details of model validation methods used. Two STAs (5 percent) had no reported errors, nineteen (46 percent) had between one and four errors, sixteen (39 percent) had between five and nine errors, and four (10 percent) had more than ten errors. The most common errors were transcription errors (29 percent), logic errors (29 percent), and computational errors (25 percent). All STAs went through at least one type of validation. Moreover, errors that were notable enough were reported in the final appraisal document (FAD) in eight (20 percent) of the STAs assessed but each of these eight STAs received positive recommendations. Technical errors are common in the economic models submitted to NICE. Some errors were considered important enough to be reported in the FAD. Improvements are needed in the model development process to ensure technical errors are kept to a minimum.

A Review of Survival Analysis Methods Used in NICE Technology Appraisals of Cancer Treatments: Consistency, Limitations, and Areas for Improvement.

Objectives. In June 2011, the National Institute for Health and Care Excellence (NICE) Decision Support Unit published a Technical Support Document (TSD) providing recommendations on survival analysis for NICE technology appraisals (TAs). Survival analysis outputs are influential inputs into economic models estimating the cost-effectiveness of new cancer treatments. Hence, it is important that systematic and justifiable model selection approaches are used. This study investigates the extent to which the TSD recommendations have been followed since its publication. Methods. We reviewed NICE cancer TAs completed between July 2011 and July 2017. Information on survival analyses undertaken and associated critiques for overall survival (OS) and progression-free survival were extracted from the company submissions, Evidence Review Group (ERG) reports, and final appraisal determination documents. Results. Information was extracted from 58 TAs. Only 4 (7%) followed all TSD recommendations for OS outcomes. The vast majority (91%) compared a range of common parametric models and assessed their fit to the data (86%). Only a minority of TAs included an assessment of the shape of the hazard function (38%) or proportional hazards assumption (40%). Validation of the extrapolated portion of the survival function using external data was attempted in a minority of TAs (40%). Extrapolated survival functions were frequently criticized by ERGs (71%). Conclusions. Survival analysis within NICE TAs remains suboptimal, despite publication of the TSD. Model selection is not undertaken in a systematic way, resulting in inconsistencies between TAs. More attention needs to be given to assessing hazard functions and validation of extrapolated survival functions. Novel methods not described in the TSD have been used, particularly in the context of immuno-oncology, suggesting that an updated TSD may be of value.