Validated Assessment Tools Research Articles

Translation and validation of chinese version of MDASI immunotherapy for early-phase trials module: a cross-sectional study

BackgroundDuring immunotherapy treatment and survival, identifying symptoms requires a standardized and validated assessment tool. The aim of this study was to translate, validate and use the Chinese version of the Immunotherapy of the M.D. Anderson Symptom Inventory for Early-Phase Trials module (MDASI-Immunotherapy EPT) to assess the symptom burden of cancer patients receiving immunotherapy in China.MethodsThe MDASI-Immunotherapy EPT was translated into Chinese using Brislin’s translation model and the back-translation method. In total, 312 Chinese-speaking colorectal cancer patients receiving immunotherapy were enrolled in the trial from August 2021 to July 2022 after receiving definitive diagnoses in our cancer center. The reliability and validity of the translated version was evaluated.ResultsCronbach’s α values were 0.964 and 0.935 for the symptom severity and interference scales, respectively. Significant correlations were found between the MDASI-Immunotherapy EPT-C and FACT-G scores (-0.617–0.732, P < 0.001). Known-group validity was supported by significant differences in the scores of the four scales grouped by ECOG PS (all P < 0.01). The overall mean subscale scores for the core and interference subscales were 1.92 ± 1.75 and 1.46 ± 1.87, respectively. Fatigue, numbness/tingling, and disturbed sleep had the highest scores for the most serious symptoms.ConclusionThe MDASI-Immunotherapy EPT-C showed adequate reliability and validity for measuring symptoms among Chinese-speaking colorectal cancer patients receiving immunotherapy. The tool could be used in clinical practice and clinical trials to gather patients’ health and quality of life data and manage their symptoms in a timely manner in the future.

DYMUS-Hr self-assessment questionnaire (Croatian version) for dysphagia in multiple sclerosis-validity, reliability, and cross-cultural adaptation.

The prevalence of dysphagia in the early phases of multiple sclerosis is 30-40%, with an estimated of 30% of cases going undiagnosed cases. Such complications can lead to malnutrition, dehydration, and aspiration pneumonia and have a great impact on the quality of life and psychosocial status of a person with MS. The aim of this study was the validation of dysphagia in multiple sclerosis self-assessment questionnaire (DYMUS) in the Croatian language. The cross-cultural adaptation process included a back-forward translation technique of the English language version of DYMUS to the Croatian language, with pilot testing on 30 participants. The validity and reliability of the Croatian version of DYMUS (DYMUS-Hr) was applied to 106 MS patients, with comparison to the Eating Assessment Tool (EAT10), the Water Swallowing Test (WST), and a dichotomous self-assessment question. In the assessment of test-retest reliability, 99 MS patients were included. Internal consistency of DYMUS-Hr was very good (Cronbach's alpha-0.837); Cronbach's alpha was 0.819 for the "dysphagia for solids", and 0.562 for "dysphagia for liquids" subscale. A significant correlation (p < 0.001) was found between DYMUS-Hr and EAT10 (Spearman's rho-0.787), and WST (Spearman's rho-0.483). Construct validity was assessed with the self-assessment question and interpreted with the Mann-Whitney U test. Test-retest reliability showed moderate to substantial Cohen's Kappa reliability for each item. DYMUS-Hr is a valid and reliable screening assessment tool for patients with MS. There is a general lack of awareness about dysphagia symptoms among patients with MS; consequently, this disorder receives inadequate attention and often goes untreated.

The Turkish version of the Casey-Fink Readiness for Practice Scale: A validity and reliability study

AimThe aim was to translate and adapt the Casey-Fink Readiness for Practice Scale (CFRPS) into Turkish and assess its validity and reliability for senior nursing students. BackgroundNursing students’ readiness for practice is important for quality nursing care and to support new graduate nurses and orient them quickly for their professional life. Nurse educators and nurse managers are responsible for develop nursing students’ and new graduate nurses’ readiness for practice. Currently, there is no valid and reliable tool to assess this metric for senior nursing students in Turkey. DesignThe study was conducted using a methodological approach. MethodsThe sample of the study consisted of 179 students who were enrolled in the last year of nursing school across three state universities in one region of Turkey. A socio-demographic form and the Turkish version of the CFRPS were used for data collection. Data were collected online between 12 April – 17 May 2021. Content validity was assessed using expert approval. Confirmatory factor analysis, exploratory factor analysis and structural equation modelling were used to assess validity. Cronbach's alpha and test–retest were performed to assess reliability. ResultsNursing students’ mean age was found to be 22.3 ± 1.12. The content validity index of the scale was calculated as 0.94. Confirmatory and exploratory factor analysis identified 15 items, which could be categorized under one factor, that were obtained differently from the original scale. The factor loads were found to be between 0.39 and 0.70. The Cronbach's alpha of the scale was 0.881. One factor model achieved a good fit. ConclusionsThe study demonstrated that the Turkish version of the CFRPS is a valid and reliable assessment tool to evaluate senior nursing students’ readiness for practice. Information in the Turkish version of the CFRPS was obtained differently from the original scale. Nurse educators can use this tool to assess their students’ progress in readiness for practice before they graduate.

94 A Review of the Utility of the Readmission Risk Scale in Hospitalized Burn Injured Patients

Abstract Introduction The burden of wound care post-hospitalization is often overwhelming, not only for patients and their families but also for home health nurses and intermediate care facilities. Inadequate wound care following hospital discharge can lead to wound deterioration and hospital readmission. These readmissions impact the patient, as well as healthcare institutions, physicians, payor sources, and policy makers. Readmission Risk Scales (RSS) exist but typically focus on those patients with chronic illness. Our hospital’s RRS consists of 16 items: patient without a primary care physician (PCP), active problem list &amp;gt; 2, past medical history &amp;gt; 2, age ≥ 50, lives alone, high fall risk, length of stay &amp;gt; 10 days, mental health disorder, diabetes, heart failure, COPD, alcohol abuse, drug use, uninsured/Medicaid, &amp;gt; 2 ED visits in previous 6 months and prior readmission within 30 days. The purpose of this study was to evaluate the fit of our current RSS for burn patients and identify other variables that may be predictive of readmission in burn patients. Methods This was a retrospective chart review of burn patients admitted over a 1-year period. Primary outcome measures were RSS scores, individual RSS variables, social determinants of health and incidence of readmission within 30 days of discharge. Readmitted patients were compared to those who did not require readmission. Descriptive statistics were calculated, and bivariate and multivariable logistic regressions (forward LR) were used to identify variables that contributed uniquely to readmission. Results A total of 600 burn patients were admitted over the study period of which 449 met inclusion/exclusion criteria. Thirty-two (7%) of these patients were readmitted within 30 days of their initial hospital discharge. In bivariate analyses, only 4 of the current RRS variables were statistically significant predictors of readmission within 30 days, patient without a PCP (p=.019), active problem list &amp;gt; 2 (p&amp;lt; .001), mental health disorder (p=.004) and drug use (p&amp;lt; .00004). However, when these variables were tested simultaneously in one predictive equation, only drug use remained significantly predictive. The total RSS score accounted for only 1.1% of the variance in readmission within 30 days. Conclusions The national readmission rate is 12%, and our overall rate was substantially below this. However, our current Readmission Risk Scale is not predictive in identifying those patients at risk for being readmitted within 30 days of discharge. Further investigation is warranted to identify additional variables to identify at risk patients. Applicability of Research to Practice A valid readmission risk assessment tool, would be beneficial to identify patients who are at risk for readmission and enhancing discharge planning.

DEVELOPMENT OF STUDENT ASSESSMENT INSTRUMENTS FOR CIVIVCS LEARNING IN SMP/MTs

This study focuses on creating a valid and practical assessment instrument for 8th-grade Civics Education (PPKn) students in junior high schools (SMP/MTs). The research employs the Research and Development (R&amp;D) approach, adapting the Borg &amp; Gall model by Sugiyono. The instrument, centered on Pancasila as the foundational principle of the state and national outlook, consists of 20 multiple-choice questions derived from a well-designed question matrix and indicators. Expert validation results show high approval scores for material and language aspects from two validators. The average validation score is 82.06%, affirming the instrument's validity. Educator and student questionnaire responses demonstrate strong practicality support, with average response rates of 83.89% and 87.60%, respectively. Analysis of the 20 questions confirms 18 valid questions, 2 needing revision, establishing the instrument's reliability (reliability coefficient r11 = 0.819). Difficulty analysis reveals 16 easy and 4 moderately difficult questions. Discrimination power assessment indicates 9 adequately differentiating and 11 highly discriminative questions. The study confirms the successful development of a valid and practical assessment tool for enhancing Civics Education in junior high schools, rejecting the null hypothesis.

Critical care nurses’ experiences of caring for patients with iatrogenic opioid withdrawal in the intensive care unit: A qualitative study

ObjectivesTo explore critical care nurses’ experiences of caring for adult patients experiencing iatrogenic opioid withdrawal in the intensive care unit. Research methodology/designA qualitative study with an explorative and descriptive design was conducted. Data were collected through semi-structured interviews and systematic text condensation was used to analyse the data. The study was reported according to the consolidated criteria for reporting qualitative research checklist. Setting and participantsTen critical care nurses, working at three different intensive care units in two university hospitals in Norway. FindingsThree categories were identified in the data analysis. “Subtle signs and symptoms of opioid withdrawal”, lack of a systematic approach to opioid withdrawal, and the prerequisites for appropriate management of opioid withdrawal. Critical care nurses experienced challenges in identifying opioid withdrawal due to subtle and vague signs and symptoms, especially when not knowing their patient or when difficulties were encountered with patient communication. A systematic approach to opioid withdrawal and increased knowledge, definitive plans for weaning, as well as interdisciplinary unity and collaboration, could improve the management of opioid withdrawal. ConclusionValidated assessment tools, systematic strategies, and guidelines are essential for the management of opioid withdrawal in opioid naïve patients in intensive care units. The prerequisites for an appropriate management of opioid withdrawal are an accurate and effective communication among critical care nurses and other healthcare professionals involved in patient care. Implications for clinical practiceThere is a need for a validated assessment tool, systematic strategies, and guidelines for the management of opioid withdrawal in opioid naïve patients in intensive care units. Increased emphasis needs to be placed on the process of identifying iatrogenic opioid withdrawal and improving opioid withdrawal management in the education system and clinical practice.

Validity and Reliability of the activPAL4TM for Measurement of Body Postures and Stepping Activity in 6-12-Year-Old Children.

A link between inappropriate physical behaviour patterns (low physical activity and high sedentary behaviour) and poor health outcomes has been observed. To provide evidence to quantify this link, it is important to have valid and reliable assessment tools. This study aimed to assess the validity and reliability of the activPAL4TM monitor for distinguishing postures and measuring stepping activity of 6-12-year-old children. Thirteen children (8.5 ± 1.8 years) engaged in pre-determined standardised (12 min) and non-standardised (6 min) activities. Agreement, specificity and positive predictive value were assessed between the activPAL4TM and direct observation (DO) (nearest 0.1 s). Between-activPAL4TM (inter-device) and between-observer (inter-rater) reliability were determined. Detection of sitting and stepping time and forward purposeful step count were all within 5% of DO. Standing time was slightly overestimated (+10%) and fast walking/jogging steps underestimated (-20%). For non-standardised activities, activPAL4TM step count matched most closely to combined backward and forward purposeful steps; however, agreement varied widely. The activPAL4TM demonstrated high levels of reliability (ICC(1, 1) > 0.976), which were higher in some instances than could be achieved through direct observation (ICC(2, 1) > 0.851 for non-standardised activities). Overall, the activPAL4TM recorded standardised activities well. However, further work is required to establish the exact nature of steps counted by the activPAL4TM.

The Fear of War Scale (FOWARS): Development and Initial Validation

The emergence of the Ukraine-Russia war in 2022 led to fear, worry and anxiety among individuals, mostly in the immediate neighboring countries of Ukraine. The purpose of the present study was to develop the fear of war scale (FOWARS), aiming to fill the gap in the literature that currently is scarce in valid assessment tools for measuring the fear of war. The sample of the study consisted of N = 1131 participants (n = 670 for group A, for EFA and n = 461 for group B, for CFA). Hungarian speaking participants, mainly from Romania and Hungary. Exploratory factor analysis (EFA) evinced a two-factor model of the newly developed scale, and the final version fulfilled the criteria of the confirmatory factor analysis (CFA). The 13-items FOWARS has robust psychometric properties and proves to be appropriate for a fear of war assessment in the general population. The phenomenon is measured by two factors, namely experiential and physiological dimensions of fear. The scale is available in the present paper in two languages: English and Hungarian.

Methods of physical rehabilitation of elderly people for the prevention and treatment of sarcopenia

The aim of the review is to analyze the prevalence of sarcopenia in the elderly age group, the causes of its occurrence, and to present modern methods of prevention and physical rehabilitation. The study focuses on the relationship between exercise, training effects and physiological mechanisms, as well as the safety of various types of strength, anaerobic and multimodal training, which have a positive impact during the prevention and rehabilitation treatment of sarcopenia. Literature reviews, meta-analyses, and original studies are included that focus on older people in all settings, using validated assessment tools and methods. A literature search was conducted in four electronic databases – PubMed, Cochrane Library, Scopus, Springer, for the period from 2012 to June 30, 2022. There were no restrictions on the language bias of the publication. Search strategy. The keywords used to define the terms of participation in the review are “older/advanced age”, “sarcopenia” and “sarcopenic obesity”. Articles were included if they met the following criteria – cohorts with mean or median age ≥ 60 years and any of the following definitions of sarcopenia: European Working Group on Sarcopenia in the Elderly (EWGSOP), Asian Working Group on Sarcopenia (AWGS), International Working Group on Sarcopenia (IWGS). To ensure comparability of interventions, the review included studies that were conducted for at least 8 weeks, and the distribution of patients by study design was randomized. Also, articles involving hospitalized patients are excluded.

Organizational Practices Advancing Racial Equity (OPARE): Development of a Tool Using an Implementation Science Approach

Organizational practices to advance racial equity are needed to address the systemic nature of racism. Implementation science frameworks offer a comprehensive approach when implementing complex initiatives. We describe the development of a tool to assess organizational practices in advancing racial equity (OPARE). Findings on the acceptability, reliability and preliminary validity are presented based on 98 responses from child and youth mental health agencies across Ontario. The OPARE shows promise as a valid and reliable organizational assessment tool for helping plan, implement and evaluate racial equity efforts in community mental health settings.

Tools for Quality of Life Assessment during Pregnancy: A Narrative Review

Background: Pregnancy is a period of physiological and hormonal alterations that can decrease the quality of life (QoL). Women’s subjective perception of QoL is clinically relevant for assessing the effectiveness of interventions and identifying susceptibility to depression. Objective: This review aims to identify, present, and critically appraise the assessment tools of QoL during pregnancy. Methods: A narrative review of the English-language literature was conducted through the PubMed, CENTRAL, Scopus, and Google Scholar electronic databases. Conclusion: Several generic instruments have been proposed, such as pregnancy-specific and disease- specific questionnaires concerning common diseases during pregnancy. Plasma, salivary and hair cortisol, inflammatory, genetic and epigenetic biomarkers and gut microbiome have also been investigated as potential indicators of maternal anxiety. Their use can quantify the association between pregnancy-specific anxiety and QoL. Application of validated assessment tools of QoL during pregnancy could improve QoL, maternal health interventions, and early identification of susceptibility to maternal depression.

Symptom management in adult brain tumours: A literature review.

To review the literature related to symptom management, clinical significance and related theoretical framework systems in adult patients with brain tumours. As understanding of symptoms or symptom clusters and underlying biologic mechanisms has grown, it is apparent that symptom science is moving forward. Although some progress has been made in the symptom science of solid tumours such as breast and lung neoplasms, insufficient attention has been paid to symptom management for patients suffering from brain tumours. Further research is needed to achieve effective symptom management for these patients. A literature review with a systematic search of symptom management in adult brain tumours. Electronic data bases were searched to obtain relevant published literature on symptom management in adults with brain tumours. This was then analysed and a synthesis of relevant findings is presented. Four significant general themes relating to symptom management of brain tumours in adults were identified: (1) The potential theoretical foundation related to symptom management was revealed. (2) Widely accepted validated scales or questionnaires for the assessment of single symptoms or symptom clusters were recommended. (3) Several symptom clusters and the underlying biologic mechanisms have been reported. (4) Specific symptom interventions for adults with brain tumours were collected and classified as evidence-based or insufficient evidence. There are still many challenges in the effective management of symptoms in adults with brain tumours. The guiding role of theoretical frameworks or models related to symptom management should be utilized in future research. Using the concept of symptom clustering for research into symptoms found in patients with brain tumours, exploring common biological mechanisms for specific symptom clusters and making full use of modern big data resources to build a strong evidence base for an effective intervention or management program may inform the management of symptoms among these patients leading to better results. This is a literature review. The ultimate goal is obviously not only improving the survival rate of patients with brain tumours, but also enhancing their quality of life. Several important findings from our review include the theoretical foundations, validated assessment tools, the assessment of symptom clusters and the underlying biologic mechanism, and the identification of the evidence base for symptom interventions. These are of relevance for managers, researchers and practitioners and may function as a reference to help the effective symptom management for adults with brain tumours.

The “Aberdeen Home Continence Stress Test”: a novel objective assessment tool for female stress urinary incontinence

Introduction and hypothesisClinical trials for stress urinary incontinence (SUI) require a robust, reliable, and responsive tool for objective assessment of SUI post-intervention. The Aberdeen Home Continence Stress Test (HCST) is a novel patient-reported objective assessment tool, aimed to be patient-friendly and reduce attrition rates by avoiding hospital appointments and prolonged pad-wearing. We aim to describe the HCST for the first time and evaluate its reliability, diagnostic accuracy, and response to change.MethodsA secondary analysis of the Single-Incision Mini-Slings (SIMS) study (a prospective multicentre randomised control trial (RCT) comparing two surgical treatments of SUI was performed. In SIMS (n = 600 women), the objective outcome was assessed by the 24-h pad test, while the patient-reported success rates were assessed using the Patient Global Impression of Improvement (PGI-I) at 15 months, 2 years and 3 years post-randomisation. Participants were instructed to perform the HCST before and after the 24-h pad test. The HCST was analysed in relation to reliability, validity, and the relationship between the 24-h pad test and HCST results and finally with regard to its responsiveness to change in PGI-I. (Trial registration-number ISRCTN93264234, registration date 14/01/2014).ResultsCompared to the 24-h pad test, the sensitivity of the HCST ranged from 0.81–0.95, specificity was 0.76–0.79, negative predictive value was 0.96–0.99 and positive predictive value was 0.32–0.43. Reliability was indicated by high-performing Cronbach’s alpha value (> 0.7). An improvement of ≥ 2 leakage groups on the HCST (for example from Large at baseline to Small leakage at follow-up) was strongly associated with patient-reported success on PGI-I (OR 4.38, 95% CI 2.31, 8.31).ConclusionsThe HCST is a valid and reliable patient-reported objective assessment tool that can be used for assessing SUI in surgical trials with good specificity, sensitivity, and consistency.

Evaluation of adherence to the Mediterranean diet with sustainable nutrition knowledge and environmentally responsible food choices.

Dietary patterns and their possible effects on health and the environment are becoming increasingly important. It is thought that nutritionally balanced diets can also be compatible with environmental targets and, therefore, the Mediterranean diet (MD), which is regarded as a sustainable diet model, comes to the fore. This study was carried out to evaluate adherence to the MD with sustainable nutrition knowledge and environmentally responsible food choices and to determine the factors affecting adherence. A questionnaire prepared by the researchers was sent to individuals online and 1732 adults living in Turkey participated in this cross-sectional study. Adherence to the MD was evaluated with the Mediterranean Diet Adherence Screener (MEDAS). In addition, questions were asked about nutritional knowledge and environmentally responsible food choices to evaluate the sustainable nutritional behaviors of individuals. Half of the participants (51.1% of men / 53% of women) adhere to the MD at a moderate level. Even the individuals with the highest adherence to the MD had low compliance with the recommendations for fruit (43.4%) and fish (37.3%) consumption. A one-unit increase in age, sustainable nutrition knowledge score, and environmentally responsible food choices score increases the MD adherence score by 0.08, 0.125, and 0.148 points, respectively (p < 0.005). Individuals with high adherence to the MD avoid consuming genetically modified organism food more (p < 0.001), prefer to consume environmentally labeled foods (p < 0.001), and buy food more from local businesses (p < 0.001), while they prefer to buy imported food less (p = 0.034). The results of this study showed that some strategies should be developed to increase the adaptation of individuals to the MD and sustainable nutritional behaviors. Nutritionally adequate, sustainable, and eco-friendly nutritional behaviors should be encouraged to increase the possible health benefits of nutrition and minimize environmental effects. To promote sustainable nutrition, firstly it is important to determine the knowledge level of individuals concerning sustainable nutrition and, for this purpose, it is thought that an international valid sustainable nutrition knowledge assessment tool is needed.

Assessing bystander first aid: development and validation of a First Aid Quality Assessment (FAQA) tool

BackgroundInjuries are one of the leading causes of death worldwide. Bystanders at the scene can perform first aid measures before the arrival of health services. The quality of first aid measures likely affects patient outcome. However, scientific evidence on its effect on patient outcome is limited. To properly assess bystander first aid quality, measure effect, and facilitate improvement, validated assessment tools are needed.The purpose of this study was to develop and validate a First Aid Quality Assessment (FAQA) tool. The FAQA tool focuses on first aid measures for injured patients based on the ABC-principle, as assessed by ambulance personnel arriving on scene.MethodsIn phase 1, we drafted an initial version of the FAQA tool for assessment of airway management, control of external bleeding, recovery position and hypothermia prevention. A group of ambulance personnel aided presentation and wording of the tool. In phase 2 we made eight virtual reality (VR) films, each presenting an injury scenario where bystander performed first aid. In phase 3, an expert group discussed until consensus on how the FAQA tool should rate each scenario. Followingly, 19 respondents, all ambulance personnel, rated the eight films with the FAQA tool. We assessed concurrent validity and inter-rater agreement by visual inspection and Kendall’s coefficient of concordance.ResultsFAQA-scores by the expert group concurred with ± 1 of the median of the respondents on all first aid measures for all eight films except one case, where a deviation of 2 was seen. The inter-rater agreement was “very good” for three first aid measures, “good” for one, and “moderate” for the scoring of overall quality on first aid measures.ConclusionOur findings show that it is feasible and acceptable for ambulance personnel to collect information on bystander first aid with the FAQA tool and will be of importance for future research on bystander first aid for injured patients.

Psychometric properties of a clinical assessment tool in the postgraduate midwifery programme, Botswana

The psychometric properties of a clinical assessment tool used in the postgraduate midwifery programme in Botswana have not been evaluated. A lack of reliable and valid clinical assessment tools contributes to inconsistencies in clinical assessment in midwifery programmes. This study aimed to evaluate the internal consistency and content validity of a clinical assessment tool used in the postgraduate midwifery programme in Botswana. For internal consistency, we calculated the total-item correlation and Cronbach's alpha coefficient. For content validity, subject matter experts completed a checklist to evaluate the relevance and clarity of each competency in the clinical assessment tool. The checklist included questions with Likert-scale responses, indicating the level of agreement. The clinical assessment tool had a good reliability, with a Cronbach's alpha of 0.837. The corrected item total correlation values ranged from -0.043 to 0.880 and the Cronbach's alpha (if item deleted) ranged from 0.079 to 0.865. Overall content validity ratio was 0.95, and content validity index was 0.97. Item content validity indices ranged from 0.8 to 1.0. The overall scale content validity index was 0.97 and the scale content validity index using universal agreement was 0.75. The clinical assessment tool used in the postgraduate midwifery programme in Botswana has acceptable reliability. Most of the competencies included in the clinical assessment tool were relevant and clear. Certain competencies need to be reviewed to improve the reliability and validity of the clinical assessment tool.Contribution:The clinical assessment tool currently used in the postgraduate midwifery programme in Botswana had acceptable internal consistency reliability and validity.

PD30-04 SURGICAL GESTURES TO MEASURE SURGICAL DISSECTION PERFORMANCE IN APICAL DISSECTION OF ROBOT-ASSISTED RADICAL PROSTATECTOMY

You have accessJournal of UrologyCME1 Apr 2023PD30-04 SURGICAL GESTURES TO MEASURE SURGICAL DISSECTION PERFORMANCE IN APICAL DISSECTION OF ROBOT-ASSISTED RADICAL PROSTATECTOMY Maxwell X. Otiato, Runzhuo Ma, Timothy N. Chu, Elyssa Y. Wong, Christian Wagner, and Andrew J. Hung Maxwell X. OtiatoMaxwell X. Otiato More articles by this author , Runzhuo MaRunzhuo Ma More articles by this author , Timothy N. ChuTimothy N. Chu More articles by this author , Elyssa Y. WongElyssa Y. Wong More articles by this author , Christian WagnerChristian Wagner More articles by this author , and Andrew J. HungAndrew J. Hung More articles by this author View All Author Informationhttps://doi.org/10.1097/JU.0000000000003316.04AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: Surgical performance impacts patient outcomes, yet ideal performance is challenging to objectively quantify. Previously, we established a surgical gesture classification system that deconstructs robotic tissue dissection into discrete basic surgical maneuvers. Here, we evaluate surgical gestures as a novel assessment method by correlating this metric with surgeon experience and technical skill assessment scores in the apical dissection (AD) of robotic-assisted radical prostatectomy (RARP). METHODS: RARPs from 2016-2018 across two international institutions were included. After standardized training, raters decoded AD surgical videos with surgical gestures. Surgeons were grouped into two experience levels by the median robotic caseload (range 80-5,800 cases; median 475 cases). The less experienced group performed <475 robotic cases and more experienced group performed ≥475 robotic cases. Videos were assessed by a validated assessment tool, Dissection Assessment for Robotic Technique (DART), which contains six domains. A Kruskal-Wallis test was used to compare surgical gesture proportion with DART scores from surgeons of different experience levels. Spearman’s correlation coefficients were used to assess the associations between surgical gesture usage and DART. RESULTS: In total, 78 surgical videos from 20 surgeons were included. More experienced surgeons (n=10) utilized larger proportions of cold cut (50% vs 39%, p=0.008) and smaller proportions of peel/push (12% vs 25%, p=0.02) than less experienced surgeons (n=10). Correlations between the surgical gestures and technical skills assessments ranged from -0.40 to 0.32 using the Spearman’s correlation coefficient (p<0.05, Figure 1b). Surgeons who utilized more retraction gestures had a lower retraction score, efficiency score, and total DART score (all p<0.05). Those who used more discrete gestures and spent more time per gesture had a lower efficiency score (both p<0.01). CONCLUSIONS: Surgeons of different experience levels use different surgical gestures during AD. Significant correlations were demonstrated between surgical gestures and dissection technical skills. Surgical gestures can serve as a novel method to teach and evaluate dissection skill. Source of Funding: Research reported in this publication was supported by the National Cancer Institute of the National Institutes of Health under Award Number R01CA273031. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. © 2023 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 209Issue Supplement 4April 2023Page: e832 Advertisement Copyright & Permissions© 2023 by American Urological Association Education and Research, Inc.MetricsAuthor Information Maxwell X. Otiato More articles by this author Runzhuo Ma More articles by this author Timothy N. Chu More articles by this author Elyssa Y. Wong More articles by this author Christian Wagner More articles by this author Andrew J. Hung More articles by this author Expand All Advertisement PDF downloadLoading ...

106 Neurosurgery Resident Feedback through Artificial-Intelligence

OBJECTIVES/GOALS: Surgical training is constrained by duty hour limits, bias, and a trial-and-error learning process. Surgeon skill variation is a healthcare system disparity that can impact patient outcomes. Incorporating validated, standardized assessment tools and machine learning (ML) algorithms may help to standardize and reduce bias in surgeon education. METHODS/STUDY POPULATION: To support assessment tool and ML algorithm development, we are curating an annotated video registry of neurosurgical procedures. Point-of-view video of resident and attending neurosurgeons performing craniotomies is recorded via an eye-tracking headset. A Delphi panel of neurosurgeons will review the video and determine which represent expert versus trainee performance. Neurosurgery attendings will be interviewed to provide descriptions of craniotomies which will be used to develop an assessment rubric. A Delphi panel will determine what rubric components should be maintained. New craniotomy videos will be viewed by attendings in a blinded fashion while completing the assessment rubric. An online feedback platform is being developed allowing residents to prospectively track assessment data. RESULTS/ANTICIPATED RESULTS: We anticipate development of an annotated, institutional video database featuring craniotomies performed by residents and attending neurosurgeons. Using a Delphi approach, we anticipate achieving consensus on which videos reflect expert versus trainee performance. We anticipate development of a novel craniotomy assessment rubric that is both valid and reliable. Our online feedback platform will allow prospective tracking of assessment data from multiple sources and enhanced transparency in the feedback process. The video registry and assessment data will enable development of novel ML algorithms able to recognize craniotomy segments and estimate operator skill. DISCUSSION/SIGNIFICANCE: Building a video registry of procedures, validated assessment tools, and a prototype feedback platform enables a pipeline for ML algorithm development. Together these tools will help to standardize and optimize resident education translating to earlier operative independence, improved patient safety, and reduced bias during surgical training.

Evaluation of communication skills from a 360° perspective at simulated Tele-rehabilitation scenarios

Communication skills (CS) are competencies required by health professionals. Clinical simulation with a simulated/standardized patient (SP) is a good resource for teaching this skill. However, it requires trained teachers to guide the process. Not all teachers have this training. HC short evaluation instruments are required to develop a structured observation during the activity, especially those that can be applied by all the participants to guide and understand the process from a broader perspective. Validate a short HC assessment questionnaire to evaluate these skills from a 360° analysis, i.e., from the learner, facilitator/teacher, and PS perspective. A brief HC questionnaire was created, subjected to construct and content validity, and thus applied to 40 Kinesiology undergraduate students, facilitator/teacher, and PS during a simulated teleconsultation scenario. We analyzed the result of the communication by descriptive statistics, Lashe, Cronbach, KMO, Bartlett, IVC, Fisher and Cohen tests, and exploratory factorial analysis. The questionnaire showed content validity for 5 of 6 items. Construct validity with commonalities over 50% for each item. With good reliability (Cronbach's alpha > 0.79). We found high levels of HC in the students, but the concordance between observers was weak (Cohen's Kappa < 0.4). The short questionnaire for HC is a valid assessment tool during clinical simulation. Incorporating the views and perceptions of all the simulation participants can improve the understanding of health communication.

PD01-11 AUTOMATED SURGICAL SKILLS ASSESSMENT: CONSENSUS BUILDING TOWARDS HUMAN-DERIVED GROUND TRUTH SCORES FOR MACHINE LEARNING ALGORITHMS

You have accessJournal of UrologyCME1 Apr 2023PD01-11 AUTOMATED SURGICAL SKILLS ASSESSMENT: CONSENSUS BUILDING TOWARDS HUMAN-DERIVED GROUND TRUTH SCORES FOR MACHINE LEARNING ALGORITHMS Timothy N. Chu, Daniel I. Sanford, Elyssa Y. Wong, Runzhuo Ma, Cherine H. Yang, Istabraq S. Dalieh, Mitchell G. Goldenberg, and Andrew J. Hung Timothy N. ChuTimothy N. Chu More articles by this author , Daniel I. SanfordDaniel I. Sanford More articles by this author , Elyssa Y. WongElyssa Y. Wong More articles by this author , Runzhuo MaRunzhuo Ma More articles by this author , Cherine H. YangCherine H. Yang More articles by this author , Istabraq S. DaliehIstabraq S. Dalieh More articles by this author , Mitchell G. GoldenbergMitchell G. Goldenberg More articles by this author , and Andrew J. HungAndrew J. Hung More articles by this author View All Author Informationhttps://doi.org/10.1097/JU.0000000000003218.11AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: Artificial Intelligence (AI)-based assessments of surgical skills may provide objective evaluation of surgeon skill in a scalable manner. However, human scoring used as the ground truth reference standard in AI development may introduce subjectivity and error. Here, we aim to assess the value of a previously validated four-round consensus building process to limit rater bias prior to training machine learning (ML) algorithms with human-derived labels. METHODS: Three different datasets derived from VR suturing exercises completed on the Mimic™ Flex VR robotic simulator were included. Deidentified participant videos were provided binary technical scores for various suturing skills (needle positioning, entry angle, needle driving, needle withdrawal) using a validated assessment tool. Different combinations of three blinded and independent human raters achieved score consensus after undergoing a standardized four-round consensus building process (Figure 1a). Proportions of minority scores (e.g., one rater does not agree with two others) remaining after each round were tracked to determine trends in the consensus building process. RESULTS: In total, 5634 suturing skill assessments across all exercises were included. After initial video review (Round 1), 32% of assessments had a reviewer that provided a minority score. Following an individual review of minority scores (Round 2), 800/1816 (44.1%) of the minority scores persisted for group review across the three different exercises. Of the minority scores that persisted to Round 3 (group review), 274/800 (34.3%) became the final ground truth score. Out of all original minority scores, 274/1816 (15.1%) persisted through the entire consensus building process and became the ground truth labels (Figure 1b). CONCLUSIONS: When using multiple human raters, a considerable proportion of initial minority scores were ultimately agreed on as the ground truth skill assessment. This finding was consistent across multiple datasets, domains, and various raters. The standardized consensus-building process is an essential step in the creation of accurate ML models and underscores the significant impact human rater bias has on evaluations of surgical performance. Source of Funding: Research reported in this publication was supported by the National Cancer Institute of the National Institutes of Health under Award Number R01CA251579. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health © 2023 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 209Issue Supplement 4April 2023Page: e67 Advertisement Copyright & Permissions© 2023 by American Urological Association Education and Research, Inc.MetricsAuthor Information Timothy N. Chu More articles by this author Daniel I. Sanford More articles by this author Elyssa Y. Wong More articles by this author Runzhuo Ma More articles by this author Cherine H. Yang More articles by this author Istabraq S. Dalieh More articles by this author Mitchell G. Goldenberg More articles by this author Andrew J. Hung More articles by this author Expand All Advertisement PDF downloadLoading ...