ePRO Data Research Articles

Analysis of a Remote Monitoring Program for Symptoms Among Adults With Cancer Receiving Antineoplastic Therapy

Electronic patient-reported outcomes (ePROs) may have the potential to improve cancer care delivery by enhancing patient quality of life, reducing acute care visits, and extending overall survival. However, the optimal cadence of ePRO assessments is unknown. To determine patient response preferences and the clinical value associated with a daily cadence for ePROs for patients receiving antineoplastic treatment. This quality improvement study of adult patients undergoing antineoplastic treatment assessed a remote monitoring program using ePROs that was developed to manage cancer therapy-related symptoms. ePRO data submitted between October 16, 2018 to February 29, 2020, from a single regional site within the Memorial Sloan Kettering Cancer Center network were included. Data were analyzed from April 2020 to January 2022. While undergoing active treatment, patients received a daily ePRO assessment that, based on patient responses, generated yellow (moderate) or red (severe) symptom alerts that were sent to clinicians. The main outcomes assessed included patient response rate, symptom alert frequency, and an analysis of the clinical value of daily ePROs. A total of 217 patients (median [range] age, 66 [31-92] years; 103 [47.5%] women and 114 [52.5%] men) initiating antineoplastic therapy at high risk for symptoms were monitored for a median (range) of 91 (2-369) days. Most patients had thoracic (59 patients [27.2%]), head and neck (48 patients [22.1%]), or gastrointestinal (43 patients [19.8%]) malignant neoplasms. Of 14 603 unique symptom assessments completed, 7349 (50.3%) generated red or yellow symptom alerts. Symptoms commonly generating alerts included pain (665 assessments [23.0%]) and functional status (465 assessments [16.1%]). Most assessments (8438 assessments [57.8%]) were completed at home during regular clinic hours (ie, 9 am-5 pm), with higher response rates on weekdays (58.4%; 95% CI, 57.5%-59.5%) than on weekend days (51.3%; 95% CI, 49.5%-53.1%). Importantly, 284 of 630 unique red alerts (45.1%) surfaced without a prior yellow alert for the same symptom within the prior 7 days; symptom severity fluctuated over the course of a week, and symptom assessments generating a red alert were followed by an acute care visit within 7 days 8.7% of the time compared with 2.9% for assessments without a red alert. These findings suggest that daily ePRO assessments were associated with increased insight into symptom management in patients undergoing antineoplastic treatment and symptom alerts were associated with risk of acute care.

Patient-Reported Outcome Information Collected from Lupus Patients Using a Mobile Application: Compliance and Validation.

ObjectivePatient‐reported outcomes (PROs) can provide critical information concerning the impact of a disease on an individual. Mobile technology to collect PRO data in an electronic format (ePRO) allows for frequent assessment in the person's regular environment. The goal of this study was to assess the compliance with a phone application (app) and validate ePRO information in individuals with systemic lupus erythematosus (SLE).MethodsA smartphone app that collects ePRO data from various clinical instruments was developed. Information was collected by both an ePRO and a paper‐administered instrument as part of a multicenter randomized interventional clinical trial of patients meeting American College of Rheumatology (ACR) criteria for the classification of SLE. To determine agreement between PRO information collected in the different formats, intraclass correlation coefficients (ICCs), paired Student's t tests, and Bland–Altman plots were evaluated. Compliance and Cronbach's alpha were also assessed as a measure of survey reliability.ResultsFor the 62 subjects from diverse ancestral backgrounds, compliance with ePRO completion was high (more than 75%). Cronbach alpha values for PROs indicated moderate to high survey reliability. The vast majority (73.4%) of ICC values were indicative of good to excellent reliability between measurement methods. Bland–Altman plots verified method agreement, and 87% of pairwise t tests yielded an insignificant difference between information collected with the different administration methods.ConclusionThe excellent compliance and the high level of consistency between data collected by paper and that collected by electronic methods indicate that the app provides a reliable means of cataloging real‐time changes in PROs in SLE patients.

Electronic patient-reported outcomes and machine learning in predicting immune-related adverse events of immune checkpoint inhibitor therapies

BackgroundImmune-checkpoint inhibitors (ICIs) have introduced novel immune-related adverse events (irAEs), arising from various organ systems without strong timely dependency on therapy dosing. Early detection of irAEs could result in improved toxicity profile and quality of life. Symptom data collected by electronic (e) patient-reported outcomes (PRO) could be used as an input for machine learning (ML) based prediction models for the early detection of irAEs.MethodsThe utilized dataset consisted of two data sources. The first dataset consisted of 820 completed symptom questionnaires from 34 ICI treated advanced cancer patients, including 18 monitored symptoms collected using the Kaiku Health digital platform. The second dataset included prospectively collected irAE data, Common Terminology Criteria for Adverse Events (CTCAE) class, and the severity of 26 irAEs. The ML models were built using extreme gradient boosting algorithms. The first model was trained to detect the presence and the second the onset of irAEs.ResultsThe model trained to predict the presence of irAEs had an excellent performance based on four metrics: accuracy score 0.97, Area Under the Curve (AUC) value 0.99, F1-score 0.94 and Matthew’s correlation coefficient (MCC) 0.92. The prediction of the irAE onset was more difficult with accuracy score 0.96, AUC value 0.93, F1-score 0.66 and MCC 0.64 but the model performance was still at a good level.ConclusionThe current study suggests that ML based prediction models, using ePRO data as an input, can predict the presence and onset of irAEs with a high accuracy, indicating that ePRO follow-up with ML algorithms could facilitate the detection of irAEs in ICI-treated cancer patients. The results should be validated with a larger dataset.Trial registration Clinical Trials Register (NCT3928938), registration date the 26th of April, 2019

Outcomes of a Mobile App to Monitor Patient-Reported Outcomes in Rheumatoid Arthritis: A Randomized Controlled Trial.

To examine the effects of a smartphone application (app) to monitor longitudinal electronic patient-reported outcomes (ePROs) on patient satisfaction and disease activity in patients with rheumatoid arthritis (RA). We conducted a 6-month randomized controlled trial of care coordination along with an app (intervention) versus care coordination alone (control) in 191 RA patients. Participants in the intervention group were prompted to provide information daily using ePROs. In both the intervention and control groups, a care coordinator contacted participants at 6 and 18 weeks to assess for flares. The main outcome measures were the global satisfaction score from the Treatment Satisfaction Questionnaire for Medication (TSQM), the score from the Perceived Efficacy in Patient-Physician Interactions (PEPPI) Questionnaire, and the Clinical Disease Activity Index (CDAI) score. Groups were similar at baseline. The median TSQM score at 6 months was 83.3 in both groups, and the median PEPPI score at 6 months was 50 in both groups. The median CDAI score at 6 months was 8 in the intervention group versus 10 in the control group. No statistically significant group differences in the medians of TSQM, PEPPI, or CDAI scores at 6 months were detected. Of the 67 intervention participants who completed the exit survey, 90% rated their likelihood of recommending the app as ≥7 of 10. Of the 11 physicians who completed the exit survey, 73% agreed/strongly agreed that they wanted to continue offering the app to patients. A mobile app designed to collect ePRO data on RA symptoms did not significantly improve patient satisfaction or disease activity compared to care coordination alone. However, both patients and physicians reported positive experiences with the app.

Implementation of the Vinehealth application, a digital health tool, into the care of patients living with brain cancer.

e13582 Background: Cancers of the brain lead to significant neurocognitive, physical and psychological morbidities. Digital technologies provide a novel platform to capture and evaluate these needs. Mobile health (mHealth) applications typically focus on one aspect of care rather than addressing the multimodal needs of the demographic of these patients. The Vinehealth application aims to address this by tracking symptoms, delivering machine learning-based personalised educational content, and facilitating reminders for medications and appointments. Where mHealth interventions traditionally lack the evidence-based approach of pharmaceuticals, this study acts as an initial step in the rigorous assessment of a new digital health tool. Methods: A mixed methodology approach was applied to evaluate the Vinehealth application as a care delivery adjunct. Patients with brain cancer were recruited from the day of their procedure ± 7 days. Over a 12-week period, we collected real-world and ePRO data via the application. We assessed qualitative feedback from mixed-methodology surveys and semi-structured interviews at onboarding and after two weeks of application use. Results: Six participants enrolled of whom four downloaded the application; four completed all interviews. One patient set up their device incorrectly and so couldn't receive the questionnaires; excluding this patient, the EQ-5D-5L and EORTC QLQ-BN20 completion rates were 100% and 83% respectively. Average scores (±SD) at onboarding and offboarding were EQ-5D-5L: 2.07±1.28 and 1.73±1.22, and QLQ-BN20: 13.33 and 22.5. In total: 212 symptoms, 174 activity, and 47 medication data points were captured, and 113 educational articles were read. Participants were generally optimistic about application use. All users stated they would recommend Vinehealth and expressed subjective improvements in care. Accessibility issues in the ePRO delivery system which impacted completion rate were identified and have subsequently been fully addressed. Conclusions: This feasibility study showed acceptable patient use, led to a subjective improvement in care, and demonstrated effective collection of real-world and validated ePRO data. This provides a strong basis to further explore the integration of the Vinehealth application into brain cancer care. This study will inform the design of a larger, more comprehensive trial continuing to evaluate improvements in care delivery through data collection, educational support and patient empowerment.

Using mobile wearables to establish sleep bioprofiles in newly diagnosed multiple myeloma (MM) patients.

8040 Background: Passive monitoring using wearables can objectively measure sleep over extended time periods. MM patients (PTs) are susceptible to fluctuating sleep patterns due to pain and dexamethasone (dex) treatment. In this prospective study, we remotely monitored sleep patterns on 40 newly diagnosed MM (NDMM) PTs while administering electronic PT reported outcome (ePRO) surveys. The study aim was to establish sleep bioprofiles during therapy and correlate with ePROs. Methods: Eligible PTs for the study had untreated NDMM and assigned to either Cohort A – PTs < 65 years or Cohort B – PTs ≥ 65 years. PTs were remotely monitored for sleep 1-7 days at baseline [BL] and continuously up to 6 therapy cycles. PTs completed ePRO surveys (EORTC - QLQC30 and MY20) at BL and after each cycle. Sleep data and completed ePRO surveys were synced to Medidata Rave through Sensorlink technology. Associations between sleep measurement trends and QLQC30 scores were estimated using a linear mixed model with a random intercept. Results: Between Feb 2017 - Sep 2019, 40 PTs (21 M and 19 F) were enrolled with 20 in cohort A (mean 54 yrs, 41-64) and 20 in cohort B (mean 71 yrs, 65-82). Regimens included KRd 14(35%), RVd 12(30%), Dara-KRd 8(20%), VCd 5(12.5%), and Rd 1(2.5%). Sleep data was compiled among 23/40 (57.5%) PTs. BL mean sleep was 578.9 min/24 hr for Cohort A vs. 544.9 min/24 hr for Cohort B (p = 0.41, 95% CI -51.5, 119.5). Overall median sleep trends changed for cohort A by -6.3 min/24 hr per cycle (p = 0.09) and for cohort B by +0.8 min/24 hr per cycle (p = 0.88). EPRO data trends include global health +1.5 score/cycle (p = 0.01, 95% CI 0.31, 3.1), physical +2.16 score/cycle (p < 0.001, 95% CI 1.26, 3.07), insomnia -1.6 score/cycle (p = 0.09, 95% CI [-3.47, 0.26]), role functioning +2.8 score/cycle (p = 0.001, 95% CI 1.15, 4.46), emotional +0.3 score/cycle (p = 0.6, 95% CI -0.73, 1.32), cognitive -0.36 score/cycle (p = 0.44, 95% CI -1.29,0.56), and fatigue -0.36 score/cycle (p = 0.4, 95% CI -1.65, 0.93). No association between sleep measurements and ePRO were detected. Difference in sleep on dex days compared to all other days during the sample cycle period for cohort A was 81.4 min/24 hr (p = 0.004, 95% CI 26, 135) and for cohort B was 37.4 min/24 hr (p = 0.35, 95% CI -41, 115). Conclusions: Our study provides insight into wearable sleep monitoring in NDMM. Overall sleep trends in both cohorts do not demonstrate significant gains or losses, and these trends fit with HRQOL ePRO insomnia responses. Upon further examination, we demonstrate objective differences (younger PTs) in intra-cyclic sleep measurements on dex days compared to other cycle days (less sleep by > 1 hr). For older patients, less variation in sleep profiles was detected during dex days, possibly due to higher levels of fatigue or longer sleep duration. Sleep is an integral part of well-being in the cancer patient. Future studies should continue to characterize sleep patterns as it relates to HRQOL.

A learning health systems approach to integrating electronic patient-reported outcomes across the health care organization.

IntroductionFoundational to a learning health system (LHS) is the presence of a data infrastructure that can support continuous learning and improve patient outcomes. To advance their capacity to drive patient‐centered care, health systems are increasingly looking to expand the electronic capture of patient data, such as electronic patient‐reported outcome (ePRO) measures. Yet ePROs bring unique considerations around workflow, measurement, and technology that health systems may not be poised to navigate. We report on our effort to develop generalizable learnings that can support the integration of ePROs into clinical practice within an LHS framework.MethodsGuided by action research methodology, we engaged in iterative cycles of planning, acting, observing, and reflecting around ePRO use with two primary goals: (1) mobilize an ePRO community of practice to facilitate knowledge sharing, and (2) establish guidelines for ePRO use in the context of LHS practice. Multiple, emergent data collection activities generated generalizable guidelines that document the tangible best practices for ePRO use in clinical care. We organized guidelines around thematic areas that reflect LHS structures and stakeholders.ResultsThree core thematic areas (and 24 guidelines) emerged. The theme of governance reflects the importance of leadership, knowledge management, and facilitating organizational learning around best practice models for ePRO use. The theme of integration considers the intersection of workflow, technology, and human factors for ePROs across areas of care delivery. Lastly, the theme of reporting reflects critical considerations for curating data and information, designing system functions and interactions, and presentation of ePRO data to support the translation of knowledge to action.ConclusionsThe guidelines produced from this work highlight the complex, multidisciplinary nature of implementing change within LHS contexts, and the value of action research approaches to enable rapid, iterative learning that leverages the knowledge and experience of communities of practice.

Abstract PO-042: Engaging diverse patients and families in the visualization of cancer-related patient-reported outcomes

Abstract Background: Black patients with cancer report worse symptom burden and management relative to Whites. Use of electronic patient-reported outcomes (ePROs) have been linked to improved cancer-related symptom management, with potential to address inequities in symptom management. Visualization of ePRO symptom data is a critical component of ePRO development; however, engagement of diverse patient and family stakeholders in the visualization of ePRO data has been limited. As part of the Symptom and Healthcare Access Reporting and Evaluation (SHARE) study, we engaged diverse stakeholders in the development of an ePRO tool. Herein, we describe our stakeholder engagement process, as well as feedback and recommendations for optimizing visualization of ePRO data with an equity prioritization lens. Methods: We leveraged a multi-step, mixed-methods, community-engaged research approach to solicit input from diverse patients in active treatment (n= 30 Black; n=49 White) and members of cancer-focused community advisory boards (n=1 Asian; n=8 White). Stakeholders provided feedback on two versions of the ePRO symptom report (Prototype 1[original version] and Prototype 2 [modified version based on initial stakeholder feedback]) through an iterative process. Feedback was obtained via surveys, individual interviews, and group meetings. Results: Compared with White patients, Blacks reported more challenges in using and understanding Prototype 1 of the symptom report. Community advisory board members shared similar concerns regarding ease of use of Prototype 1. In particular, patients and members noted that graphical legends and axes were difficult to interpret, making it difficult to identify which symptoms were worsening or improving. Prototype 2 of the ePRO symptom report incorporated feedback, such as increasing font and graph sizes, combining related symptoms (e.g., nausea and vomiting) into a single graph, and inclusion of color-graded graph schemes corresponding to symptom severity (e.g., green=no symptom; orange=moderate severity), which facilitated ease of use for patients with limited graphical literacy. In comparing Prototype 1 to Prototype 2, community advisory board members reported that Prototype 2 was more visually appealing and easier to use and understand. Conclusions: Overall, feedback from diverse patients and community stakeholder groups resulted in the development of a more usable and patient-centered ePRO symptom summary report. Our work highlights the importance and value of engaging diverse stakeholders in the development of ePRO systems, including the visualization of ePRO data, in order to optimize the in the use of ePRO tools for everyone. Citation Format: Wendi L. Elkins, Neda R. Padilla, Dani Cutts, Mattias Jonsson, Michelle Johnson, Giselle Corbie-Smith, Samuel Cykert, Bryce B. Reeve, Antonia V. Bennett, Donald L. Rosenstein, Hanna Sanoff, Cleo A. Samuel. Engaging diverse patients and families in the visualization of cancer-related patient-reported outcomes [abstract]. In: Proceedings of the AACR Virtual Conference: Thirteenth AACR Conference on the Science of Cancer Health Disparities in Racial/Ethnic Minorities and the Medically Underserved; 2020 Oct 2-4. Philadelphia (PA): AACR; Cancer Epidemiol Biomarkers Prev 2020;29(12 Suppl):Abstract nr PO-042.

Adaptation and Evaluation of a Symptom-Monitoring Digital Health Intervention for Patients With Relapsed and Refractory Multiple Myeloma: Pilot Mixed-Methods Implementation Study.

BackgroundRelapsed and refractory multiple myeloma (RRMM) is a bone marrow cancer that requires systemic treatment, which often results in severe symptom burden. Recent studies have found that electronic patient-reported outcome (ePRO) interventions implemented in the clinic setting have had positive outcomes for other oncology populations. Evidence of the efficacy of a similar approach is lacking for patients with RRMM.ObjectiveRecent recommendations for digital health interventions call for the publication of descriptions of iterative development processes in order to improve reproducibility and comparability. This study is an implementation pilot aiming to evaluate the acceptability and appropriateness of an ePRO intervention for patients with RRMM and to explore its impact on clinic workflow.MethodsA total of 11 patients with RRMM were recruited from the John Theurer Cancer Center in Hackensack, New Jersey. Patients used a mobile app to report on 17 symptoms at 4 sessions, each a week apart. Patients could also report symptoms ad hoc. When reports met predefined thresholds, the clinic was alerted and patients received automated guidance. Study end points were assessed using qualitative and quantitative methods.ResultsA total of 9 patients (mean age 69.7 years) completed the study. Overall, 83% (30/36) of weekly sessions were completed. Patients found the frequency and time required to complete reporting acceptable. All patients agreed that the app was easy to use and understand. Providers felt the alerts they received required refinement. Patients and providers agreed it would be beneficial for patients to report for longer than 4 weeks. Patients felt that the training they received was adequate but contained too much information for a single session. All patients found the symptoms tracked to be appropriate; providers suggested shortening the list. All patients understood how to use the app for weekly reporting but had confusion about using it ad hoc. Providers felt the ad hoc feature could be removed. Neither patients nor providers viewed the in-app data reports but agreed on their potential value. Patients reported benefitting from symptom reporting through increased awareness of their symptoms. Clinic staff reported that app alerts were too numerous and redundant. They had difficulty responding to alerts within their existing workflow, partially because the data were not integrated into the electronic medical record system.ConclusionsOverall, the intervention was found to be acceptable and appropriate for patients with RRMM. Points of friction integrating the intervention into the clinic workflow were identified. Clinic staff provided recommendations for addressing these issues. Once such modifications are implemented, ePRO data from patients with RRMM could be used to inform and improve clinical research and care. This study underlines the importance of an iterative approach to implementation that includes all stakeholders in order to ensure successful adoption.

How to implement routine electronic patient-reported outcome monitoring in oncology rehabilitation.

BackgroundImplemenation of patient‐reported outcomes (PRO) like quality of life can add the patient's perspective to traditional clinical outcomes of cancer rehabilitation in a structured and standardized way.AimTo present useful steps for a successful implementation of routine electronic patient‐reported outcomes (ePRO) monitoring. The presented steps are exemplified by describing the procedure applied in an Austrian inpatient cancer rehabilitation centre.MethodsThe suggested implementation steps are presented based on the structure of the replicating effective programmes framework, which was used for developing a pragmatic implementation strategy.ResultsWe scheduled alternating trainings and process evaluations for audit and enhancement of procedures. In this way, the ePRO participation rates could be improved. Stakeholder involvement led to initiatives that included the integration of ePRO data into the medical discharge letter and the implementation of follow‐up assessments.DiscussionTailored changes in assessment procedures enabled the successful implementation of ePRO, which has been shown to be feasible before and after cancer rehabilitation. The continuous involvement of stakeholders paved the way for further projects initiated by medical staff as users themselves (inclusion of PRO data in the discharge letter and a comprehensive ePRO follow‐up using a versatile online patient portal).

Implementation of Electronic Patient-Reported Outcomes in Routine Cancer Care at an Academic Center: Identifying Opportunities and Challenges.

Electronic patient-reported outcomes (ePROs) can help clinicians proactively assess and manage their patients' symptoms. Despite known benefits, there is limited adoption of ePROs into routine clinical care as a result of workflow and technologic challenges. This study identifies oncologists' perspectives on factors that affect integration of ePROs into clinical workflows. We conducted semistructured qualitative interviews with 16 oncologists from a large academic medical center, across diverse subspecialties and cancer types. Oncologists were asked how they currently use or could imagine using ePROs before, during, and after a patient visit. We used an inductive approach to thematically analyze these qualitative data. Results were categorized into the following three main themes: (1) selection and development of ePRO tool, (2) contextual drivers of adoption, and (3) patient-facing concerns. Respondents preferred diagnosis-based ePRO tools over more general symptom screeners. Although they noted information overload as a potential barrier, respondents described strong data visualization and ease of use as facilitators. Contextual drivers of oncologist adoption include identifying target early adopters, incentivizing uptake through use of ePRO data to support billing and documentation, and emphasizing benefits for patient care and efficiency. Respondents also indicated the need to focus on patient-facing issues, such as patient response rate, timing of survey distribution, and validity and reliability of responses. Respondents identified several barriers and facilitators to successful uptake of ePROs. Understanding oncologists' perspectives is essential to inform both practice-level implementation strategies and policy-level decisions to include ePROs in alternative payment models for cancer care.

FRI0018 USING SELF-REPORTED OUTCOMES TO DETECT NEW-ONSET FLARE IN A REAL-WORLD STUDY OF PARTICIPANTS WITH RHEUMATOID ARTHRITIS - INTERIM RESULTS FROM THE DIGITAL TRACKING OF ARTHRITIS LONGITUDINALLY (DIGITAL) STUDY

Background:Patients with rheumatoid arthritis (RA) experience fluctuating symptoms, increased pain, decreased function and variable quality of life; such changes often occur between visits to clinicians. Digital Tracking of Arthritis Longitudinally (DIGITAL) study2is evaluating the use of electronically captured patient-reported outcomes (ePRO) and passive data collection from a Fitbit device to identify disease worsening in a real-world study of participants (pts) with RA.Objectives:Evaluate agreement between self-reported new-onset flare and ePROs in an interim analysis from DIGITAL using a classification model.Methods:Members of the ArthritisPower registry with RA were invited to participate in DIGITAL. Pts who successfully completed a two-week Lead-in period entered the Main Study in which they wore a smartwatch and provided daily (pain and fatigue numeric rating scales (NRS)) and weekly ePROs, including the OMERACT RA Flare Questionnaire (FLARE) and PROMIS measures. This interim analysis is of ePRO data from pts who completed at least 30 days of the Main Study. A “Yes” response to the FLARE item, “Are you having a flare now?” identified flare. For modeling association between new-onset flare and ePRO, the dataset was split into training (the first 30 days of the Main Study) and test data (Day 31 and following). Within each dataset, repeated binary outcomes (Flare/No Flare) per pt were defined each week. To focus on new-onset flare, within each dataset, outcomes for patient weeks for which flare was present in the previous week were excluded.Candidate variables for the model included baseline and current FLARE score (0-50 scale) and each of its 5 items, daily pain, daily fatigue, and several PROMIS weekly instruments and their lagged values (last week or last 6 days for daily). ‘Baseline’ was calculated in non-flare weeks. Training data was used for logistic regression model selection combining clinical expertise with backward elimination. Performance of the final model was evaluated using test data.Results:The training data was composed of outcomes from 128 pts who reported 388 weekly flare assessments as no flare or onset flare over 2800 days during the first month of the Main Study. Of pts in the training dataset, 92.2% were female, 87.5% white, with mean age (SD) 52.7 (11.0) and years since RA diagnosis 10.4 (10.3); 62.5% were on a biologic. Among those in the training dataset, 58 flare outcomes occurred in 50 (39.1%) unique pts.The test data comprised outcomes from 123 pts who reported 442 weekly flare assessments as no flare or onset flare over 3366 days in which 64 flare outcomes occurred, and primarily included continued observations from pts who contributed to the training dataset.The best-performing model to classify flare in training data included the current and baseline FLARE instrument activity question (i.e. “Considering how active your rheumatoid arthritis has been, how much difficulty have you had when taking part in activities such as work, family life, social events that are typical for you during the last week”), current daily pain, and baseline daily pain average and standard deviation. In test data, this model had an area under the receiver operator curve of 0.81 (Figure). At a cut point requiring specificity to be ≥0.80, sensitivity to detect flare was 0.62 and overall accuracy was 0.78.Conclusion:New-onset flare is common among RA patients, and the FLARE instrument and daily pain scores appear effective to classify it. Evaluation of passive data as a proxy for self-reported new-onset flare is ongoing.

Electronic patient-reported outcomes (ePROs) and machine learning (ML) in predicting the presence and onset of immune-related adverse events (irAEs) of immune checkpoint inhibitor (ICI) therapies.

e14058 Background: ICIs have introduced novel irAEs, arising from various organ systems without strong timely dependency on initiation and discontinuation of the therapy. Early detection of the irAEs could result in improved safety profile of the treatment and better quality of life for patients. Symptom data collected by ePROs could be used as an input for ML based prediction models for early detection of irAEs. Methods: The utilized dataset consisted of two data sources. The first dataset consisted of 16 540 reported symptoms from 33 ICI-treated cancer patients, including 18 monitored symptoms collected using Kaiku Health digital platform. The second dataset included prospectively collected irAE data, including initiation and end dates, CTCAE class, and severity of 26 irAEs (the longest irAE lasted 799 days, and the shortest two days while median duration was 61 days). Two ML models were built using extreme gradient boosting, a well-known classification algorithm. Using the ePRO data, the first model was trained to detect the presence and the second model to detect the onset (0-21 days prior to diagnosis) of irAEs. The dataset was split into training (70 % of the data) and test sets (30 % of the data) by random allocation. The test set was left out from the model training and tuning, and was used only to evaluate the model performance. Results: The model trained to predict the presence of irAEs had an excellent performance with the test dataset. The prediction of the irAE onset was more difficult, but the model performance was still at a very good level. The performance metrics for the ML models are presented in Table. Conclusions: Current study suggests that ML based prediction models, using ePRO data as input for the models, can predict the presence and onset of irAEs with high accuracy. Thus, it indicates that digital symptom monitoring combined with ML could enable the detection of irAEs in ICI-treated cancer patients. The results should be validated with a larger dataset from prospective clinical trials. Clinical trial information: NCT03928938. [Table: see text]

Home ePRO compliance in prostate cancer clinical studies.

2063 Background: Patient-reported outcomes (PRO) and electronic PRO (ePRO) play an important role in the development and approval of cancer products. Regulatory agencies are encouraging the inclusion of PRO-based endpoints that are indicative of clinical benefit in terms of patient symptoms and overall quality of life (QOL). Compliance with completion of ePRO assessments is an important component for obtaining accurate and high-quality data when conducting clinical trials. Traditionally, ePRO data in oncology trials has been collected mainly at clinic visits due to concerns over poor compliance at home. However, since symptoms and QOL can vary widely through a treatment course, it is often necessary to collect ePRO data more frequently in between clinic visits. It has been hypothesized that home completion, length of time in a study, and number of assessments may affect compliance. Methods: To address this hypothesis, ePRO compliance data was analyzed from two clinical studies in prostate cancer. Both studies used a handheld smartphone that contained an application to collect ePRO data. At the randomization visit, subjects completed ePRO assessments in clinic (2-3 questionnaires). Subsequently, all assessments were completed at home, including a daily diary and 1-4 questionnaires completed every 4-8 weeks for up to 14 months. Compliance was calculated as the number of assessments received divided by the number of assessments expected in a given assessment period. To evaluate assessment burden, each assessment period was categorized as requiring a lower number (daily diary and 1 questionnaire) or higher number (daily diary and 2-4 questionnaires) of assessments. Results: A total of 1,040 patients were included in the analysis. Overall compliance at the single clinic visit was 100%, which was expected since it was a required randomization visit. Overall compliance at home over 14 months was 80%. Compliance ranged from 78% to 89% over the duration of the studies, with no effect of time in the study on compliance. Compliance remained high even as patient numbers declined. Compliance when patients were required to complete a lower number of assessments (80%) was similar to compliance when patients were required to complete a higher number of assessments (79%). Compliance by region varied from 72% (Middle East) to 87% (Asia and Eastern Europe). Conclusions: The collection of ePRO at home provided high compliance that did not vary with length of time in the study or due to assessment burden. At home ePRO assessments provide an effective and feasible approach for recording symptoms and QOL in prostate cancer patients.

Predicting onset and continuity of patient-reported symptoms in cancer patients undergoing immune checkpoint inhibitor (ICI) therapies using machine learning

Abstract Background In recent years, ICI therapies have become standard-of-care treatments in several malignancies. ICIs introduced novel toxicities, which can arise from various organ systems or at any time point even after treatment discontinuation. The toxicities can be life threatening, but most are reversible if detected and treated early and therefore, early detection could result in improved safety profile of treatment and better Health related Quality-of-Life. Electronic collection of patient reported symptoms (ePRO) could be used in development of machine learning (ML) based prediction models to enable earlier detection of toxicities compared to traditional follow-up. Methods The training dataset consists of ePRO data of ICI-treated patients collected using Kaiku Health platform, including 21 744 reported symptoms from 72 ICI patients. The ML models were built with an established classification algorithm extreme gradient boosting and trained for 14 symptoms related to ICI toxicities. The validation dataset consisted of 16 884 reported symptoms from 67 cancer patients receiving ICIs and followed with the ePRO tool. Results ML models were able to predict the onset and continuation of 14 symptoms that may indicate the development of ICI toxicities. The model performance assessed using area under curve (AUC), a common performance metric for ML models, is best for dyspnea (0.96) and lowest for headache (0.77). Generally, the models are performing well with an average AUC of 0.86, as shown in the table. Table . 50P AUC values for symptoms related to ICI toxicities from the validation dataset Symptoms AUC Dizziness 0,88 Itching 0,84 Fever 0,90 Diarrhoea 0,78 Stomach pain 0,86 Nausea 0,82 Fatigue 0,90 Rash 0,84 Decreased appetite 0,84 Cough 0,90 Dyspnea 0,96 Joint pain 0,92 Headache 0,77 Chest pain 0,88 Conclusion ML based modeling of ePRO data on ICI treated cancer patients is feasible in predicting onset and continuation of symptoms related to ICI toxicities. The study suggest that ML based approaches could be used in early detection of toxicities. The results should be validated with a dataset collected in a prospective clinical trial. Legal entity responsible for the study Kaiku Health Oy. Funding Oulu University and Finnish Cancer Society. Disclosure J. Ekstrom: Full / Part-time employment: Kaiku Health Oy. H. Virtanen: Full / Part-time employment: Kaiku Health Oy; Shareholder / Stockholder / Stock options: Kaiku Health Oy. J.P. Koivunen: Advisory / Consultancy: Kaiku Health Oy. All other authors have declared no conflicts of interest.

Integration of electronic patient-reported outcomes into clinical workflows within the Epic electronic medical record.

102 Background: Electronic patient-reported outcome measures (ePROs) offer a new strategy for symptom assessment that can improve quality of life and prolong survival in routine cancer care. However, ePRO systems are often separate from existing electronic medical records (EMRs) and not well integrated into oncology clinics. In this pilot project, we assessed the feasibility and utility of integrating ePROs into our existing EMR and clinical workflows. Methods: The 10-question Edmonton Symptom Assessment Scale (ESAS) was integrated into the Epic EMR at three outpatient clinics in the Duke Cancer Institute. Patients with active MyChart accounts were offered the ESAS survey prior to their visit, via the patient portal. ePRO data were routed to clinicians in tabular and graphical formats. A “SmartPhrase” facilitated easy data integration into clinical notes. We subsequently interviewed clinicians and optimized workflows. Several patient engagement strategies were used, including automated messages, phone call reminders, and electronic tablets, to increase response rate. Results: It was feasible to quickly customize and activate an ePRO in Epic. Over 10 months, 161 patients completed 208 ePRO surveys. Initially, 10-20% of eligible patients completed the MyChart questionnaire. Patient engagement strategies, including phone calls and personalized MyChart messages, had little effect. Ultimately, tablets were introduced in the clinic check-in process, increasing response rates to >90%. Clinicians reported positive regard for the system, and an impact on patient symptom management. Clinician workflow optimization resulted in minimal “clicks” in the EMR, and the SmartPhrase was used in 128 clinical notes. Conclusions: Integration of ePROs into the clinical setting poses three challenges: technical implementation, workflow optimization, and patient engagement. While technical implementation is important, it was the easiest to solve, with patient engagement as the greatest barrier. Clinicians value an integrated ePRO system that automatically routes data to the clinical note. The key to successful ePRO integration is in ease of use for both patients and clinicians.

Real-world experience with electronic patient reported outcomes and missing items: "Don't ask me irrelevant questions".

99 Background: Electronic patient-reported outcomes (ePROs) can improve quality of life and prolong survival in patients with cancer by enhancing the detection and tracking of unmet supportive care needs. However, there remain unanswered questions about how to handle missing ePRO data. We hypothesized that patients may skip specific questions they feel do not apply to them. We aimed to examine the relationship between patient demographics and missing items in a real-world ePRO dataset. Methods: We utilized a prospectively collected database of ePROs from oncology clinics administering Patient Care Monitor 2.0 (PCM), a validated symptom survey of 78 items for men and 86 for women. We tabulated the frequency of missing items by item and domain (emotional, functional and symptom-related), and examined these by age, gender, education, race and marital status. Results: In 20,986 encounters there were responses to at least 1 PCM item from 6,933 patients. On average just 1% of items were skipped per encounter. By domain, 12.3% of functional, 8.4% of symptom-related, and 1.6% of emotional constructs contained at least one missing item. The highest frequency of item non-response was seen in older patients (>60yo) and those with high school education or less. The most frequently skipped items included: “attend a paid job” (10.7%), “reduced sexual enjoyment” (3.8%), and “running” (3.7%). These questions may be less relevant to older individuals, who, for example, quit running years earlier or are retired. Men had less missingness overall, except for “cooks for self” and “house work”, which may reflect traditional gender roles in the Southeast US. Conclusions: In a real-world ePRO implementation, items pertaining to more universal issues for cancer patients, like emotional well-being, have much lower rates of missingness, especially compared to functional items like “attend a paid job.” These results suggest that patients differentially complete ePROs based on perceived question relevance to them. The underlying driver behind individual item non-response may itself be an important data point in clinical care, warranting further study and discussion during clinic visits.

Using an ePRO tool to help meet quality metrics in a clinical oncology practice.

171 Background: ASCO has implemented the Quality Oncology Practice Initiative (QOPI), a certification program established to evaluate oncology practice performance. However, practices often lack the necessary resources to meet the required standards and fulfill required metric reporting. The study purpose was to document the effectiveness of using an electronic Patient Reported Outcome (ePRO) system to facilitate compliance with two QOPI standards: pain assessment and depression screening. Methods: This study used a retrospective, observational design with ePRO and clinical data collected via the Patient Care Monitor (PCM), a web-based ePRO system linked to electronic medical record data. All study data were collected as part of routine clinical care at a community oncology practice during a 4-month interval (1/1–4/26/19). Patients at an initial oncology clinic visit completed the PCM Core Symptom survey and the PHQ2/PHQ9 to screen for pain and depression respectively via a handheld e-tablet. Results: Overall, 10,449 patients completed the PCM survey (mean age 63; 30% male/70% female; 62% White, 34% Black). Cancer type was known for 53% of all patients (22% breast, 9% hematologic, 4% lung, 5% colorectal, 3% prostate, 11% other types). Fifteen percent of patients (n = 1584) met the pain threshold (scored 7 or above on a 0-10 scale) that signaled an alert to the physician for further evaluation. About 3% of patients (n = 413) skipped the pain question on the PCM Core Symptom survey. With respect to depression, 5% (n = 559) reported a PHQ2 threshold level that triggered the remaining PHQ9 items. Among those at higher risk for depression, 16% (n = 92) met criteria for moderate to severe depression warranting further clinical evaluation and the PCM signaled a real-time alert to the physician in advance of the patient consultation. Approximately 4% (n = 413) skipped the PHQ2. Conclusions: This study demonstrates that using an ePRO system is an effective way to screen for pain and depression and can be used to: 1) monitor symptoms to enhance physician-patient communication during the consultation; 2) provide an audit trail for QOPI metric reporting. Overall, an ePRO system contributes to creating a culture of excellence at community oncology practices.

Complementing clinical cancer registry data with patient reported outcomes: A feasibility study on routine electronic patient-reported outcome assessment for the Austrian Myelome Registry.

ObjectivesRoutinely assessed patient‐reported outcomes (PROs), such as quality of life (QOL), are important to supplement clinical cancer data but requires rigorous implementation. This study aims at depicting the implementation procedure and evaluating the feasibility of routine electronic PRO monitoring (ePRO) for collecting data supplementing the Austrian Myeloma Registry (AMR).MethodsIntegration of ePRO monitoring into clinical routine was planned according to the Replicating Effective Programs framework. QOL data were assessed regularly during treatment and aftercare at the hematooncological outpatient unit at the Medical University of Innsbruck with the EORTC QLQ‐C30/ +MY20 and the EQ‐5D‐5L. Feasibility and usability testing were performed via a multimethod approach.ResultsWithin the first year, 94.4% of the MM patients (N = 142, mean age 65.4, SD 11.8, 60% male) provided 748 PRO assessment time points overall. Patients and clinicians were satisfied with ePRO monitoring and indicated no to little disruption in clinical routine. Patient preference on assessment time points and completion frequency became evident.ConclusionsComplementing the AMR with ePRO data proved to be feasible. Our findings provide useful insights for healthcare providers considering introducing ePRO monitoring to their units for informing clinical registries as well as individualised feedback to patients alike.

Training on the Use of Technology to Collect Patient-Reported Outcome Data Electronically in Clinical Trials: Best Practice Recommendations from the ePRO Consortium.

Electronic capture of patient-reported outcome (PRO) data has many advantages over paper-based data collection. Regulatory agencies have consistently supported the use of electronic PRO (ePRO) data capture and recommended participant and site staff training on the correct use of electronic data capture systems. The objective of this paper is to outline best practice recommendations for training end users, including site staff and study participants, on the use of ePRO technology in clinical trials to enable consistent, accurate, and complete data collection. Site personnel should be trained on study-specific as well as technology-specific topics and be given instructions on whom to contact to obtain technical support. Optimal training takes place over time using multiple modalities, including hands-on, face-to-face training at an investigator meeting or directly in the clinical site; remote training via webinar or teleconference; interactive on-demand self-paced-training via e-learning modalities; and supplemented by proxy training performed by study clinical research associates. Like site personnel training, study participants should be provided with individual, hands-on training by site staff at the initiation of the trial and in conjunction with interactive electronic training modules that can be accessed on-demand throughout the duration of the trial. The recommendations put forth in this paper provide a structured framework for the training that site personnel and study participants need to optimize the advantages trials can gain from using ePRO data collection systems.