Real-world Evidence Generation Research Articles

Addressing immortal time bias in precision medicine: Practical guidance and methods development.

To compare theoretical strengths and limitations of common immortal time adjustment methods, propose a new approach using multiple imputation (MI), and provide practical guidance for using MI in precision medicine evaluations centered on a real-world case study. Methods comparison, guidance, and real-world case study based on previous literature. We compared landmark analysis, time-distribution matching, time-dependent analysis, and our proposed MI application. Guidance for MI spanned (1) selecting the imputation method; (2) specifying and applying the imputation model; and (3) conducting comparative analysis and pooling estimates. Our case study used a matched cohort design to evaluate overall survival benefits of whole-genome and transcriptome analysis, a precision medicine technology, compared to usual care for advanced cancers, and applied both time-distribution matching and MI. Bootstrap simulation characterized imputation sensitivity to varying data missingness and sample sizes. Case study used population-based administrative data and single-arm precision medicine program data from British Columbia, Canada for the study period 2012 to 2015. While each method described can reduce immortal time bias, MI offers theoretical advantages. Compared to alternative approaches, MI minimizes information loss and better characterizes statistical uncertainty about the true length of the immortal time period, avoiding false precision. Additionally, MI explicitly considers the impacts of patient characteristics on immortal time distributions, with inclusion criteria and follow-up period definitions that do not inadvertently risk biasing evaluations. In the real-world case study, survival analysis results did not substantively differ across MI and time distribution matching, but standard errors based on MI were higher for all point estimates. Mean imputed immortal time was stable across simulations. Precision medicine evaluations must employ immortal time adjustment methods for unbiased, decision-grade real-world evidence generation. MI is a promising solution to the challenge of immortal time bias.

Characterizing Myasthenia Gravis Symptoms, Exacerbations, and Crises From Neurologist's Clinical Notes Using Natural Language Processing.

Background Myasthenia gravis (MG) is a rare, autoantibody neuromuscular disorder characterized by fatigable weakness. Real-world evidence based on administrative and structured datasets regarding MG may miss important details related to the clinical encounter. Examination of free-text clinical progress notes has the potential to illuminate aspects of MG care. Objective The primary objective was to examine and characterize neurologist progress notes in the care of individuals with MG regarding the prevalence of documentation of clinical subtypes, antibody status, symptomatology, and MG deteriorations, including exacerbations and crises.The secondary objectives were to categorize MG deteriorations into practical, objective states as well as examine potential sources of clinical inertia in MG care. Methods We performed a retrospective, cross-sectional analysis of de-identified neurologist clinical notes from 2017 to 2022. A qualitative analysis of physician descriptions of MG deteriorations and a discussion of risks in MG care (risk for adverse effects, risk for clinical decompensation, etc.) was performed. Results Of the 3,085 individuals with MG, clinical subtypes and antibody status identified included gMG (n = 400; 13.0%), ocular MG (n = 253; 8.2%), MG unspecified (2,432; 78.8%), seropositivity for acetylcholine receptor antibody (n = 441; 14.3%), and MuSK antibody (n = 29; 0.9%). The most common gMG manifestations were dysphagia (n = 712; 23.0%), dyspnea (n = 626; 20.3%), and dysarthria (n = 514; 16.7%). In MG crisis patients, documentation of difficulties with MG standard therapies was common (n = 62; 45.2%). The qualitative analysis of MG deterioration types includes symptom fluctuation, symptom worsening with treatment intensification, MG deterioration with rescue therapy, and MG crisis. Qualitative analysis of MG-related risks included the toxicity of new therapies and concern for worsening MG because of changing therapies. Conclusions This study of neurologist progress notes demonstrates the potential for real-world evidence generation in the care of individuals with MG. MG patients suffer fluctuating symptomatology and a spectrum of clinical deteriorations. Adverse effects of MG therapies are common, highlighting the need for effective, less toxic treatments.

Real-World Evidence to Support EU Regulatory Decision Making-Results From a Pilot of Regulatory Use Cases.

Studies using real-world data (RWD) can complement evidence from clinical trials and fill evidence gaps during different stages of a medicine's lifecycle. This review presents the experience resulting from the European Medicines Agency (EMA) pilot to generate RWE to support evaluations by EU regulators and down-stream decision makers from September 2021 to February 2023. A total of 61 research topics were identified for RWE generation during this period, covering a wide range of research questions, primarily generating evidence on medicines safety (22, 36%), followed by questions on the design and feasibility of clinical trials (11, 18%), drug utilization (10, 16%), clinical management (10, 16%), and disease epidemiology. A significant number of questions were related to the pediatric population and/or rare diseases. A total of 27 regulatory-led RWD studies have been conducted. Most studies were descriptive and aimed at estimating incidence and prevalence rates of clinical outcomes including adverse events or to evaluate medicines utilization. The review highlights key learnings to guide further efforts to enable the use and establish the value of real-world evidence (RWE) for regulatory decisions. For instance, there is a need to access additional fit-for-purpose and representative data, and to explore further means to provide timely evidence that meets regulatory timelines. The need for early interactions and close collaboration with study requesters, e.g., from the Agency's scientific Committees, to better understand the research question is equally important. Finally, the review provides our perspective on the way forward to maximize the potential of regulatory-led RWE generation.

Unstructured EMR data hold the key to oncology studies: Methods for validating NLP-extracted phenotypes from EMR notes.

11133 Background: Natural language processing (NLP) parsers are advanced algorithms designed to parse values of interest from free-text stored in Electronic Medical Records (EMR). Given the importance of information stored in free-text for use in oncology research, it is crucial to understand how accurately NLP algorithms are extracting oncology-specific measures of interest. We focus on validating NLP-parsers that identify three measures critical to oncology research by evaluating how closely the NLP-parser results match manual chart abstractor review: AJCC summary cancer stage group (cancer stage), AJCC TNM stage (TNM stage), and surgical treatment of cancer (surgery). Methods: Following deployment of the parsers on 8,000 non-identifiable free-text notes housed in an aggregated non-identifiable U.S. EMR database, manual chart abstraction was performed on a random selection of notes (n=300) dated between 2008 to 2023 to validate the parsers’ accurate capture of cancer stage, TNM stage, and surgical treatment of cancer. We report true positives (TP), false positives (FP) and positive percent value (PPV). Results: The PPV of AJCC Cancer Stage was 99.0% (n=297 TP, n=3 FP). The PPV of the TNM stage NLP-parser was 97.3% for tumor stage (n=292 TP, n=8 FP), 99.0% for nodal stage (n=297 TP, n=3 FP), and 96.7% for presence of metastases (n=290 TP, n=10 FP). The PPV of the surgery parser was 87.8% (n=468 TP, n=65 FP). Conclusions: These NLP-parsers performed very well identifying the measures of interest, providing confidence in use of these derived measures which are central to oncology research. Validation is a necessary initial step in processing real-world data for use in real-world evidence generation, with unique considerations needed in the oncology research space due to key information being documented in EMR free-text rather than structured data fields.

Automating the detection of treatment progression in patients with lung cancer using large language models.

e13620 Background: Accurate assessment of treatment response is fundamental to advancing cancer patient care, particularly in lung cancer, where treatment modalities are diverse and complex. The extraction of treatment response, particularly disease progression information, from Electronic Health Records (EHRs) is essential for several reasons: It facilitates generation of real-world evidence from real-world data, enables personalized treatment planning, and contributes to a broader understanding of cancer therapeutics. However, much of treatment response information is documented in free-text EHR notes and traditional methods of extracting such information from notes are labor-intensive, error-prone, and inefficient, presenting significant barriers to timely and accurate real-world evidence studies. The advancement of Natural Language Processing (NLP) technologies, especially Large Language Models (LLMs), presents a transformative opportunity to automate the extraction of treatment responses. Methods: We used a cohort of 1,953 primary lung cancer patients identified from UPMC Hillman Cancer Center cancer registry and retrieved ~113,000 clinical notes from UPMC clinical data warehouse. We focused on extracting disease progression information, following the RECIST guidelines which define progression as an increase in tumor size or cancer markers after therapy. We selected ~50 notes to perform manual annotations by an experienced oncologist and created a gold standard dataset to validate the NLP models. We fine-tuned a state-of-the-art open-source LLM named LLAMA-2 and compared against a traditional rule-based NLP system for the automated extraction of disease progression from notes. The process of fine-tuning involved adjusting the model's parameters specifically to improve its ability to recognize and classify instances of disease progression. Results: Our analysis demonstrated a significant enhancement in performance with the LLM compared to the traditional rule-based NLP system. LLM exhibited a remarkable increase in sensitivity by ~37%, indicating its superior ability to accurately identify disease progression. Additionally, the model maintained high specificity and PPV, achieving scores nearly comparable to the rule-based system but with a notable improvement in the F1-score by ~14%. Conclusions: Our research highlights the transformative potential of LLM-based NLP algorithms in automating the extraction of treatment responses from free-text EHRs. This methodology not only provides a scalable and efficient mechanism for processing large volumes of clinical text but also significantly enhances the accuracy of lung cancer treatment response assessments. [Table: see text]

OHDSI-compliance: a set of document templates facilitating the implementation and operation of a software stack for real-world evidence generation.

The open-source software offered by the Observational Health Data Science and Informatics (OHDSI) collective, including the OMOP-CDM, serves as a major backbone for many real-world evidence networks and distributed health data analytics platforms. While container technology has significantly simplified deployments from a technical perspective, regulatory compliance can remain a major hurdle for the setup and operation of such platforms. In this paper, we present OHDSI-Compliance, a comprehensive set of document templates designed to streamline the data protection and information security-related documentation and coordination efforts required to establish OHDSI installations. To decide on a set of relevant document templates, we first analyzed the legal requirements and associated guidelines with a focus on the General Data Protection Regulation (GDPR). Moreover, we analyzed the software architecture of a typical OHDSI stack and related its components to the different general types of concepts and documentation identified. Then, we created those documents for a prototypical OHDSI installation, based on the so-called Broadsea package, following relevant guidelines from Germany. Finally, we generalized the documents by introducing placeholders and options at places where individual institution-specific content will be needed. We present four documents: (1) a record of processing activities, (2) an information security concept, (3) an authorization concept, as well as (4) an operational concept covering the technical details of maintaining the stack. The documents are publicly available under a permissive license. To the best of our knowledge, there are no other publicly available sets of documents designed to simplify the compliance process for OHDSI deployments. While our documents provide a comprehensive starting point, local specifics need to be added, and, due to the heterogeneity of legal requirements in different countries, further adoptions might be necessary.

Data Science Methods for Real-World Evidence Generation in Real-World Data.

In the healthcare landscape, data science (DS) methods have emerged as indispensable tools to harness real-world data (RWD) from various data sources such as electronic health records, claim and registry data, and data gathered from digital health technologies. Real-world evidence (RWE) generated from RWD empowers researchers, clinicians, and policymakers with a more comprehensive understanding of real-world patient outcomes. Nevertheless, persistent challenges in RWD (e.g., messiness, voluminousness, heterogeneity, multimodality) and a growing awareness of the need for trustworthy and reliable RWE demand innovative, robust, and valid DS methods for analyzing RWD. In this article, I review some common current DS methods for extracting RWE and valuable insights from complex and diverse RWD. This article encompasses the entire RWE-generation pipeline, from study design with RWD to data preprocessing, exploratory analysis, methods for analyzing RWD, and trustworthiness and reliability guarantees, along with data ethics considerations and open-source tools. This review, tailored for an audience that may not be experts in DS, aspires to offer a systematic review of DS methods and assists readers in selecting suitable DS methods and enhancing the process of RWE generation for addressing their specific challenges.

Expert recommendation for real-world study on the application of human papillomavirus vaccine in the Yangtze River Delta region

Cervical cancer has become a global public health problem that poses a serious threat to women's health. Human papillomavirus (HPV) vaccination is an important primary preventive measure for cervical cancer. With the accelerated application of the HPV vaccine in China, conducting real-world studies (RWS) on the application of HPV vaccines can guide the rational use of the vaccine, which is of great significance for the strategy and practice of accelerating the elimination of cervical cancer in China. The Yangtze River Delta Cervical Cancer and Female Lower Genital Tract Infection Prevention and Control Alliance and Yangtze River Delta Immunization Integration Working Group organized an expert group to develop the "Expert recommendation for real-world study on the application of HPV vaccine in the Yangtze River Delta region". Based on the current situation of medical informatization and HPV vaccination administration in China, as well as literature and cases from domestic and international studies, this study describes the data chain of HPV vaccination and evaluation, introduces the study design, study direction, database construction, data governance and statistical analysis of RWS, and provides expert recommendation on the problems in carrying out the practice of HPV vaccine RWS, aiming to guide the RWS of HPV vaccine application and promote the generation of real-world evidence (RWE) in China.

Maximizing the Value of Real-World Data and Real-World Evidence to Accelerate Healthcare Transformation in China: Summary of External Advisory Committee Meetings.

Use of real-world data (RWD) is gaining wide attention. To bridge the gap between diverse healthcare stakeholders and to leverage the impact of Chinese real-world evidence (RWE) globally, a multi-stakeholder External Advisory Committee (EAC) and EAC meetings were initiated, aiming to elucidate the current and evolving RWD landscape in China, articulate the values of RWE in ensuring Chinese patients' equitable access to affordable medicines and solutions, and identify strategic opportunities and partnerships for expansion of RWE generation in China. Chinese and international experts who are clinicians and academic researchers were selected as EAC members based on their professional background and familiarity with RWD/RWE. Three EAC meetings were held quarterly in 2023. Various topics were presented and discussed for insights and suggestions. Nine experts from China, one from South Korea, and two from Europe were selected as EAC members and attended these meetings. Experts' presentations were summarized by theme, including the RWD landscape and RWE enablement in China, as well as global development of a patient-centric ecosystem. Experts' insights and suggestions on maximizing the RWD/RWE value to accelerate healthcare transformation in China were collected. We concluded that though data access, sharing, and quality are still challenging, RWD is developing to support evidence generation in the medicinal product lifecycle, inform clinical practice, and empower patient management in China. RWD/RWE creates value, accelerates healthcare transformation, and improves patient outcomes. Fostering a patient-centric ecosystem across healthcare stakeholders and maintaining global partnerships and collaboration are essential for unlocking the power of RWD/RWE.

The Use of Real-World Evidence for Regulatory Decisions in China.

There is a growing demand for the use of high-quality real-world evidence (RWE) to support regulatory decision-making worldwide and in China, which highlights the need for conducting literature reviews to evaluate the available data and evidence. This study aims to review the use of RWE in Chinese regulatory decisions and to summarize relevant regulatory and methodological considerations to inform the future use of RWE in China. We identified policy documents, technical guidance documents, and cases on official Chinese government websites and extracted their contents separately. We consulted experts from the National Medical Products Administration (NMPA) and academic institutes and searched case-related articles for enrichment. We also searched and included articles related to the use of RWE/Real-world data in Chinese regulatory decisions. Six trial versions of technical guidance documents, 7 case studies, and 40 articles related to the Chinese regulatory decisions were included in this study. Based on the technical guidance, data quality, and appropriate study design and statistical analysis are the main concerns for RWE generation. The cases and articles related to regulatory decisions revealed 9 main concerns, including data sources and applicability, data quality, strength of existing evidence, appropriate study design and statistical analysis, regulated and transparent process for analysis and evidence generation, product safety and efficacy, product characteristics and clinical needs, ethical considerations and data security, and communicate adequately with regulatory authorities. Among these concerns, data issues are central. Preliminary attempts have been made by the NMPA to promote the use of RWE, but substantial challenges still remain.

Expert survey on real-world data utilization and real-world evidence generation for regulatory decision-making in drug lifecycle in Korea.

As the importance of utilizing real-world data (RWD)/real-world evidence (RWE) for supporting regulatory scientific decision-making continues to grow, experiences and inputs from experts become crucial for developing a systematic and practice-oriented plan for the use of fit-for-purpose RWD/RWE. This study aimed to survey relevant experts from government agencies, industries, and academia to identify prerequisites for the drug life cycle in Korea. The questionnaire comprised the following: (A) the definition and categories of RWD/RWE, (B) the suitability and feasibility of using RWD/RWE at each authorization stage by the types of RWD, and (C) the challenges and solutions for the use of RWD/RWE. A total of 46 respondents completed the online survey, with 89.1% of them having prior experience with RWD/RWE usage. A majority of respondents agreed that RWD can be obtained from various sources. Among these sources, the registry was the most suitable source. It is suitable to compensate for the limitations of randomized control trials and ensure quality in data collection. Though there was consensus among the respondents for the use of RWD/RWE in post-marketing surveillance, the use of such data in new drug application (NDA) was disagreeable. Respondents considered it necessary to write a protocol in advance for RWD collection and RWE generation, for all RWD types. In conclusion, this study examined the perceptions of experts for RWD/RWE use at each approval stage of drugs. The results suggest that guidelines for the fit-for-purpose use of RWD/RWE should be developed via careful deliberation among experts in the future.

Abstract 926: Adaptive universal platform for real-world observational studies (AUPROS): An emerging model for clinical, epidemiologic, and precision oncology research

Abstract Background: Adaptive Universal Platform for Real-world Observational Studies (AUPROS) is an emerging study design and platform for the generation of real-world evidence (RWE) particularly in malignancies with rare molecular aberrations. Here, we evaluate the efficiency of AUPROS as an innovative model for real-world observational studies in oncology. Methods: We conducted a mixed-methods study to evaluate three AUPROS studies at Princess Margaret Cancer Centre either locally or as a part of national and international collaborations: 1) METAL (Molecular Epidemiology of ThorAcic Lesions; n=11,378), 2) THANKS (Translational Head And NecK Study; n=3,506), and 3) CARMA (Canadian Cancers with Rare Molecular Alterations; NCT04151342; n=3,879). Specifically, we reviewed data collected, study elements, protocol language, coordination, institutional review boards (IRBs), and contracts. We also performed stakeholder-directed survey and discussions, analysis of funding, research output, and collaborations, as well as a Strengths-Weaknesses-Opportunities-Threats (SWOT) analysis. Results: AUPROS is an innovative study design that borrows design elements from master protocol trials, adaptive trials, and master observational trials as well as retrospective and ambispective designs, enabling comprehensive data collection. The universality of AUPROS allowed for multi-purpose analyses of various real-world data (RWD) including epidemiological, clinical, patient-reported outcomes, biospecimens, and imaging data. The adaptive nature created opportunities for multi-source funding (e.g., academic, pharmaceutical, philanthropic, among others), as well as national/international collaborations and involvement in consortia (e.g., ILCCO, TCGA, BEACON, and INHANCE). Additionally, AUPROS has allowed the development of an expanding national RWE platform that involves collaborations among medical and radiation oncologists and surgeons across different tumor sites. As assessed by research output analyses, AUPROS provided a platform for enhanced research productivity in different areas and utilizing various data sources (METAL=130; THANKS=31; CARMA=5 publications). Our survey and SWOT analysis identified several cost and operational benefits of AUPROS over conventional observational studies, as well as several challenges pertinent to ethics approvals, sustainability, complex coordination, and data quality. Conclusions: AUPROS is an innovative model for real-world observational studies with significant logistical and methodological benefits over conventional RWD study designs. Our findings may help broaden the use of AUPROS to address emerging needs in the scope of precision oncology and clinico-epidemiological research. Citation Format: Samir H. Barghout, Stavroula Raptis, Luna Jia Zhan, Faisal Al-Agha, M Catherine Brown, Devalben Patel, Geoffrey Liu. Adaptive universal platform for real-world observational studies (AUPROS): An emerging model for clinical, epidemiologic, and precision oncology research [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 926.

Raising the Bar for Real-World Data in Oncology: Approaches to Quality Across Multiple Dimensions.

Electronic health record (EHR)-based real-world data (RWD) are integral to oncology research, and understanding fitness for use is critical for data users. Complexity of data sources and curation methods necessitate transparency into how quality is approached. We describe the application of data quality dimensions in curating EHR-derived oncology RWD. A targeted review was conducted to summarize data quality dimensions in frameworks published by the European Medicines Agency, The National Institute for Healthcare and Excellence, US Food and Drug Administration, Duke-Margolis Center for Health Policy, and Patient-Centered Outcomes Research Institute. We then characterized quality processes applied to curation of Flatiron Health RWD, which originate from EHRs of a nationwide network of academic and community cancer clinics, across the summarized quality dimensions. The primary quality dimensions across frameworks were relevance (including subdimensions of availability, sufficiency, and representativeness) and reliability (including subdimensions of accuracy, completeness, provenance, and timeliness). Flatiron Health RWD quality processes were aligned to each dimension. Relevancy to broad or specific use cases is optimized through data set size and variable breadth and depth. Accuracy is addressed using validation approaches, such as comparison with external or internal reference standards or indirect benchmarking, and verification checks for conformance, consistency, and plausibility, selected on the basis of feasibility and criticality of the variable to the intended use case. Completeness is assessed against expected source documentation; provenance by recording data transformation, management procedures, and auditable metadata; and timeliness by setting refresh frequency to minimize data lags. Development of high-quality, scaled, EHR-based RWD requires integration of systematic processes across the data lifecycle. Approaches to quality are optimized through knowledge of data sources, curation processes, and use case needs. By addressing quality dimensions from published frameworks, Flatiron Health RWD enable transparency in determining fitness for real-world evidence generation.

Use of Healthcare Claims Data to Generate Real-World Evidence on Patients With Drug-Resistant Epilepsy: Practical Considerations for Research.

Objectives: Regulatory bodies, health technology assessment agencies, payers, physicians, and other decision-makers increasingly recognize the importance of real-world evidence (RWE) to provide important and relevant insights on treatment patterns, burden/cost of illness, product safety, and long-term and comparative effectiveness. However, RWE generation requires a careful approach to ensure rigorous analysis and interpretation. There are limited examples of comprehensive methodology for the generation of RWE on patients who have undergone neuromodulation for drug-resistant epilepsy (DRE). This is likely due, at least in part, to the many challenges inherent in using real-world data to define DRE, neuromodulation (including type implanted), and related outcomes of interest. We sought to provide recommendations to enable generation of robust RWE that can increase knowledge of "real-world" patients with DRE and help inform the difficult decisions regarding treatment choices and reimbursement for this particularly vulnerable population. Methods: We drew upon our collective decades of experience in RWE generation and relevant disciplines (epidemiology, health economics, and biostatistics) to describe challenges inherent to this therapeutic area and to provide potential solutions thereto within healthcare claims databases. Several examples were provided from our experiences in DRE to further illustrate our recommendations for generation of robust RWE in this therapeutic area. Results: Our recommendations focus on considerations for the selection of an appropriate data source, development of a study timeline, exposure allotment (specifically, neuromodulation implantation for patients with DRE), and ascertainment of relevant outcomes. Conclusions: The need for RWE to inform healthcare decisions has never been greater and continues to grow in importance to regulators, payers, physicians, and other key stakeholders. However, as real-world data sources used to generate RWE are typically generated for reasons other than research, rigorous methodology is required to minimize bias and fully unlock their value.

Use of Healthcare Claims Data to Generate Real-World Evidence on Patients With Drug-Resistant Epilepsy: Practical Considerations for Research

Objectives: Regulatory bodies, health technology assessment agencies, payers, physicians, and other decision-makers increasingly recognize the importance of real-world evidence (RWE) to provide important and relevant insights on treatment patterns, burden/cost of illness, product safety, and long-term and comparative effectiveness. However, RWE generation requires a careful approach to ensure rigorous analysis and interpretation. There are limited examples of comprehensive methodology for the generation of RWE on patients who have undergone neuromodulation for drug-resistant epilepsy (DRE). This is likely due, at least in part, to the many challenges inherent in using real-world data to define DRE, neuromodulation (including type implanted), and related outcomes of interest. We sought to provide recommendations to enable generation of robust RWE that can increase knowledge of “real-world” patients with DRE and help inform the difficult decisions regarding treatment choices and reimbursement for this particularly vulnerable population. Methods: We drew upon our collective decades of experience in RWE generation and relevant disciplines (epidemiology, health economics, and biostatistics) to describe challenges inherent to this therapeutic area and to provide potential solutions thereto within healthcare claims databases. Several examples were provided from our experiences in DRE to further illustrate our recommendations for generation of robust RWE in this therapeutic area. Results: Our recommendations focus on considerations for the selection of an appropriate data source, development of a study timeline, exposure allotment (specifically, neuromodulation implantation for patients with DRE), and ascertainment of relevant outcomes. Conclusions: The need for RWE to inform healthcare decisions has never been greater and continues to grow in importance to regulators, payers, physicians, and other key stakeholders. However, as real-world data sources used to generate RWE are typically generated for reasons other than research, rigorous methodology is required to minimize bias and fully unlock their value.

A Pragmatic Informatics Approach to Develop Knowledge Tools for Supporting Cardiac Surgical Patients' Mental Health Needs.

A pragmatic informatics approach was developed to create knowledge tools for co-design of a new model of mental healthcare in cardiac surgery The real-world evidence generation leverages existing technological platforms and routine data collections plus tailored brief tools, surveys and qualitative data.

Evaluation of the test–retest and inter-mode comparability of the Impact of Vision Impairment questionnaire in people with chronic eye diseases

PurposeThe main objective of this study is to assess the test–retest and inter-administration mode reliability of the Impact of Vision Impairment profile (IVI), a common patient-reported outcome measure (PROM) for people with chronic eye diseases.MethodsThe IVI was administered to adult patients with stable, chronic eye diseases two to four times per participant (average intervals between administrations 12 to 20 days; maximum two phone interviews, paper administration, electronic administration) by two trained interviewers. Rasch models were fit to the data. Intra-class correlation coefficients (ICCs), mean differences and Cronbach’s alpha between test–retest administrations (two phone interviews) and inter-mode comparisons were calculated.ResultsTwo hundred-sixteen patients (mean age 67 ± 12 years, 40% male) were included in the study. The IVI met all psychometric requirements of the Rasch model, and the division into the domains of functional items (IVI_F) and emotional items (IVI_E) corresponded to the German validation study. ICCs (all for IVI_F and IVI_E, respectively) for the retest administrations were 0.938 and 0.912, and 0.853 and 0.893 for inter-mode comparisons phone/paper, 0.939 and 0.930 for phone/electronic, and 0.937 and 0.920 for paper/electronic (all p < 0.01). Mean differences (all for IVI_F and IVI_E, respectively) for the retest administrations were 2.8% and 0.7% and ranged from 2.0% to 6.2% and from 0.4 % to 4.9% between administration modes. Cronbach’s alpha ranged from 0.886 to 0.944 for retest and inter-mode comparisons.ConclusionDue to the high test–retest reliability and the almost equally high comparability of different modes of administration of the IVI, the study endorses its use as a robust PROM to capture vision-related quality of life. Our results further support the use of the IVI as an endpoint in clinical trials and may simplify implementing it in both clinical trials or real-world evidence generation by offering multiple administration modes with high reliability.

OP114 Health Technology Assessment Of Digital Technologies: Results From Two Past Examples And Thoughts For The Future

IntroductionDigital health care is a rapidly growing and increasingly diverse area of medicine and social care. Health technology assessment (HTA) agencies around the world are engaging with digital health technologies and developing methods for appropriately assessing their value. Given our experience in HTA for pharmaceuticals and how methodologies must differ when assessing medical devices and diagnostics, we must be proactive in rapidly developing useful methods for assessing a variety of digital health technologies.MethodsWe will first present the results of two completed HTAs from the UK and contrast their methods and results, while looking at the UK approach to assessing digital health care in general. Next, we will present preliminary results from an ongoing consensus conference study of HTA’s for integrated digital technologies. This study will use a modified Delphi methodology to develop statements for discussion at a consensus conference. The primary research question is “how digital technologies can help build resilient healthcare systems and how HTAs can assess their value”.ResultsAn HTA conducted in the UK of an artificial intelligence-enabled ambulatory electrocardiogram device resulted in a recommendation for further real-world evidence generation. Another HTA of a digital therapeutic app for insomnia, which was also conducted in the UK, resulted in a positive recommendation but the process was lengthy and needed to be flexible. The preliminary results of an upcoming consensus conference study will also be presented. The consensus panel will consist of a range of stakeholders from digital experts, HTA practitioners, clinicians, health economists, industry representatives, and policy makers.ConclusionsDigital technology is now a broad field with a wide range of very different technologies included in its definition. Questions remain as to whether digital technologies should be assessed using a traditional pharma-based framework for HTA, a MedTech model, or something else entirely. Given our past experience of developing flexible methods for HTA, we have an opportunity to develop useful recommendations and best practices from the ground up.

OP152 Use of Real-world Evidence By The Brazilian Health Technology Assessment Committee (Conitec) For Monitoring Of Health Technologies

IntroductionIn Brazil, the incorporation or disinvestment of health technologies into the Unified Health System (SUS) are advised by the National Committee for Health Technology Incorporation (Conitec). Despite the thorough evaluation carried out by Conitec, the results measured after implementation do not always reflect the economic and clinical impact expected from the incorporation. Thus, real-world evidence (RWE) is essential for monitoring health technologies. The aim of this study was to report how Brazil is using the RWE to obtain additional information about the incorporated technologies.MethodsActions related to the use of RWE for monitoring of technologies incorporated into the SUS were described. The period evaluated was between 2012 and 2022.ResultsThe first Conitec recommendation in which the use of real-life data in the decision-making process was evidenced occurred in 2016. Administrative data from a cohort of patients identified that beta-interferons for Multiple Sclerosis were less effective than the other drugs used in the Brazilian public system. A further eight reports have been published assessing the performance of technologies using administrative data.Another strategy for RWE generation was through the funding of primary studies, highlighting a study with 21 rare diseases and another one to evaluate Zolgensma gene therapy, acquired through court for Spinal Muscular Atrophy. Both studies are ongoing and aim to evaluate the effectiveness, safety, adherence, and cost of medications in the evaluated diseases. Conitec is also following studies in RWE financed by pharmaceutical companies to evaluate effectiveness for incorporated technologies subject to reassessment. Additionally, managed access arrangements have been promoted for generating RWE when there is uncertainty about outcomes.ConclusionsReal-world evidence from administrative data and clinical research allows monitoring after the implementation of technologies in the Unified Health System in Brazil. This makes it possible to reallocate resources in health and contribute for the system sustainability, in addition to generating data that reduce the uncertainties assumed at the time of incorporation.

Real-world evidence generation from patients, their caregivers and physicians supporting clinical, regulatory and guideline decisions: an update on Disease Specific Programmes

Objective To update on and describe the role of Disease Specific Programmes (DSPs), a multi-perspective real-world data (RWD) source, in the context of the evolution of the value and acceptance of real-world evidence (RWE) in clinical, regulatory and guideline decision-making. Methods DSPs are multi-national, multi-subscriber, multi-therapy cross-sectional surveys incorporating retrospective data collection from patient, caregiver and physician perspectives. Information collected covers the patient journey, including treatment/prescribing patterns and rationale, patient-reported outcomes, impact on work and everyday activities, attitudes towards and perceptions of the condition, adherence to treatment and burden of illness. Published peer-reviewed DSP papers were aligned with current key RWE themes identified in the literature, alongside their contribution to RWE. Results RWE themes examined were: using RWE to inform clinical practice, patient and caregiver engagement, RWE role in supporting health technology assessments and regulatory submissions, informing value-driven healthcare decisions, real-world patient subgroup differences and therapeutic inertia/unmet needs; highlighting patients’ and caregivers’ experience of living with a disease, disconnect from their physicians, unmet needs and educational gaps. Conclusions DSPs provide a wealth of RWD in addition to evidence generated by registries, clinical trials and observational research, with wide use for the pharmaceutical industry, government, funding/regulatory bodies, clinical practice guideline insights and, most importantly, informing improvements in people’s lives. The depth, breadth and heritage of information collected via DSPs since 1995 is unparalleled, extending understanding of how diseases are managed by physicians in routine clinical practice and why treatment choices are made, patients’ perceptions of their disease management, and caregiver burden.