Fleiss Kappa Research Articles

A new hybrid risk assessment method based on Fine-Kinney and ANFIS methods for evaluation spatial risks in nursing homes

Today, as the elderly population in the world increases, the increase in those living in nursing homes causes their problems to be even more important. Spatial hazards cause injury and death most of the time, therefore should be evaluated risks then corrective and preventive actions should be implemented. Fine-Kinney is one of the most widely used risk assessment methods, but it has some shortcomings. One of them is that risk factors such as probability, frequency, and severity are accepted as equally important, but they can have different importance weights in real-life applications. Another is that experts assess the risk magnitudes using their opinions, who usually tend to use linguistic expressions instead of crisp numbers, in incomplete information and uncertain situations. The last is that the experts' experiences are not effectively incorporated into the automation of the risk assessment. The adaptive neuro-fuzzy inference system (ANFIS) method, which is a machine learning method, can overcome all these shortcomings.In this study, a novel hybrid risk assessment method based on Fine-Kinney and ANFIS is developed to predict the class of a new occurring risk. The hybrid method was applied to nursing homes located in Turkey. The risk classes predicted with the hybrid method were compared to ones found in the traditional Fine-Kinney method. It was determined that the prediction accuracy and Fleiss kappa value of the new hybrid method were 95.745% and 0.929 respectively. Thus, the hybrid method can be used instead of the traditional Fine-Kinney method to determine the class of a new risk, because it does not require a large number of experts and provides a faster assessment.

Periprosthetic humerus fractures after shoulder arthroplasty: an evaluation of available classification systems

Periprosthetic proximal humerus fractures (PPHFs) are a detrimental complication of shoulder arthroplasty, yet their characterization and management have been poorly studied. We aimed to determine the intra- and interobserver reliability of 4 previously described PPHF classification systems to evaluate which classifications are the most consistent. We retrospectively reviewed 32 patients (34 fractures) that were diagnosed with a PPHF between 1990 and 2017. Patient electronic medical records and research electronic data capture (REDCap) were used for data collection. Post-PPHF radiographs in multiple views for all 34 cases were organized into an encrypted, randomized Qualtrics survey. Four blinded fellowship-trained shoulder and elbow surgeons graded each fracture using previously reported classification systems by (1) Wright and Cofield (1995), (2) Campbell etal (1998), (3) Worland etal (1999), and (4) Groh etal (2008), along with selecting a preferred management strategy for each fracture. Grading was performed twice with at least 2 weeks between each randomized attempt. Intraobserver reliability was calculated as an unweighted Cohen kappa coefficient between attempt 1 and attempt 2 for each surgeon. Interobserver reliability and agreeability between surgeons' preferred management strategies were calculated for each classification system using Fleiss kappa coefficient. The kappa coefficients were interpreted using the Landis and Koch criteria. The average intraobserver kappa coefficient for each classification was as follows: Wright and Cofield=0.703, Campbell=0.527, Worland=0.637, Groh=0.699. The overall Fleiss kappa coefficient for interobserver reliability for each classification was as follows: Wright and Cofield=0.583, Campbell=0.488, Worland=0.496, Groh=0.483. Interobserver reliability was significantly greater with the Wright and Cofield classification. Using Landis and Koch criteria, all the classification systems assessed demonstrated only moderate interobserver agreement. Additionally, the mean interobserver agreeability kappa coefficient for preferred management strategy was 0.490, indicating only moderate interobserver agreement. There is only moderate interobserver reliability among the 4 PPHF classification systems and the preferred management strategy for the fractures assessed. Of the 4 PPHF classification systems, Wright and Cofield demonstrated the greatest mean intraobserver reliability and overall interobserver reliability. Our study highlights a need for the development of a PPHF classification system that can achieve high intra- and interobserver reliability and that can allow for a standardized treatment algorithm in the management of PPHFs.

Challenging the mechanism of distal biceps tendon rupture using a video analysis study.

The conventionally described mechanism of distal biceps tendon rupture (DBTR) is of a 'considerable extension force suddenly applied to a resisting, actively flexed forearm'. This has been commonly paraphrased as an 'eccentric contracture to a flexed elbow'. Both definitions have been frequently used in the literature with little objective analysis or citation. The aim of the present study was to use video footage of real time distal biceps ruptures to revisit and objectively define the mechanism of injury. An online search identified 61 videos reporting a DBTR. Videos were independently reviewed by three surgeons to assess forearm rotation, elbow flexion, shoulder position, and type of muscle contraction being exerted at the time of rupture. Prospective data on mechanism of injury and arm position was also collected concurrently for 22 consecutive patients diagnosed with an acute DBTR in order to corroborate the video analysis. Four videos were excluded, leaving 57 for final analysis. Mechanisms of injury included deadlift, bicep curls, calisthenics, arm wrestling, heavy lifting, and boxing. In all, 98% of ruptures occurred with the arm in supination and 89% occurred at 0° to 10° of elbow flexion. Regarding muscle activity, 88% occurred during isometric contraction, 7% during eccentric contraction, and 5% during concentric contraction. Interobserver correlation scores were calculated as 0.66 to 0.89 using the free-marginal Fleiss Kappa tool. The prospectively collected patient data was consistent with the video analysis, with 82% of injuries occurring in supination and 95% in relative elbow extension. Contrary to the classically described injury mechanism, in this study the usual arm position during DBTR was forearm supination and elbow extension, and the muscle contraction was typically isometric. This was demonstrated for both video analysis and 'real' patients across a range of activities leading to rupture.Cite this article: Bone Jt Open2022;3(10):826-831.

A 10-Year Review of Intraoral Salivary Gland Tumor Diagnoses: Diagnostic Challenges and Inter-Observer Agreement.

Salivary gland tumors (SGT) are a diverse group of neoplasms arising from the major and minor glands. The oral cavity is the most common site for minor SGT (IMSGT), and these lesions frequently pose a challenge to the pathologist due to overlapping histopathological features and limited material for analysis. Our objective was to determine specific clinical and histopathological features associated with challenges in IMSGT diagnoses and pathologists' agreement. We conducted a retrospective analysis of 248 IMSGT received between 2010 and 2019. We evaluated the diagnostic challenge of the cases by stratifying according to whether a definitive, favored, or indeterminate (challenging) diagnosis was provided. Inter-observer agreement and concordance of biopsy diagnoses with the final diagnoses after tumor resection were evaluated. Of the 248 biopsies, 191 had a definitive diagnosis, 38 favored diagnoses, and 19 were indeterminate. The predominant diagnoses considered for the indeterminate category were pleomorphic adenoma/myoepithelioma (PA), polymorphous adenocarcinoma (PAC), adenoid cystic carcinoma (AdCC), and low-grade adenocarcinoma. Using multivariate analysis of clinical features, younger patient age, smaller tumor size, and larger biopsy size increased the likelihood of a definitive diagnosis (p = 0.014, p = 0.037, p = 0.012). The inter-observer agreement for 68 representative cases was moderate overall (Fleiss's Kappa 0.575) and good for the 40 cases with a definitive diagnosis (Fleiss's Kappa 0.66). Sixty-five biopsy diagnoses were matched with corresponding tumor resection diagnoses and found to show a good concordance (Cramer's V test 0.76). The discordant diagnoses predominantly involved PA, carcinoma exPA, PAC, AdCC, and adenocarcinoma NOS. Diagnostic challenges in IMSGT incisional biopsies were infrequent, especially if multiple pathologists were consulted. PA, PAC, AdCC, and adenocarcinoma NOS were the histologic types more commonly posing diagnostic challenges. Younger patient age, smaller tumor size, and larger biopsy are associated with a definitive diagnosis. This data highlights the importance of appropriate sampling in IMSGT.

Reliability of ultrasound ovarian-adnexal reporting and data system amongst less experienced readers before and after training.

BACKGROUNDThe 2018 ovarian-adnexal reporting and data system (O-RADS) guidelines are aimed at providing a system for consistent reports and risk stratification for ovarian lesions found on ultrasound. It provides key characteristics and findings for lesions, a lexicon of descriptors to communicate findings, and risk characterization and associated follow-up recommendation guidelines. However, the O-RADS guidelines have not been validated in North American institutions or amongst less experienced readers.AIMTo evaluate the diagnostic accuracy and inter-reader reliability of ultrasound O-RADS risk stratification amongst less experienced readers in a North American institution with and without pre-test training.METHODSA single-center retrospective study was performed using 100 ovarian/adnexal lesions of varying O-RADS scores. Of these cases, 50 were allotted to a training cohort and 50 to a testing cohort via a non-randomized group selection process in order to approximately equal distribution of O-RADS categories both within and between groups. Reference standard O-RADS scores were established through consensus of three fellowship-trained body imaging radiologists. Three PGY-4 residents were independently evaluated for diagnostic accuracy and inter-reader reliability with and without pre-test O-RADS training. Sensitivity, specificity, positive predictive value, negative predictive value (NPV), and area under the curve (AUC) were used to measure accuracy. Fleiss kappa and weighted quadratic (pairwise) kappa values were used to measure inter-reader reliability. Statistical significance was P < 0.05.RESULTSMean patient age was 40 ± 16 years with lesions ranging from 1.2 to 22.5 cm. Readers demonstrated excellent specificities (85%-100% pre-training and 91%-100% post-training) and NPVs (89%-100% pre-training and 91-100% post-training) across the O-RADS categories. Sensitivities were variable (55%-100% pre-training and 64%-100% post-training) with malignant O-RADS 4 and 5 Lesions pre-training and post-training AUC values of 0.87-0.95 and 0.94-098, respectively (P < 0.001). Nineteen of 22 (86%) misclassified cases in pre-training were related to mischaracterization of dermoid features or wall/septation morphology. Fifteen of 17 (88%) of post-training misclassified cases were related to one of these two errors. Fleiss kappa inter-reader reliability was ‘good’ and pairwise inter-reader reliability was ‘very good’ with pre-training and post-training assessment (k = 0.76 and 0.77; and k = 0.77-0.87 and 0.85-0.89, respectively).CONCLUSIONLess experienced readers in North America achieved excellent specificities and AUC values with very good pairwise inter-reader reliability. They may be subject to misclassification of potentially malignant lesions, and specific training around dermoid features and smooth vs irregular inner wall/septation morphology may improve sensitivity.

Implementation of a software application in staging and grading of periodontitis cases.

The purpose of this study was to assess the diagnostic accuracy and the inter-rater agreement among general dentists when staging and grading periodontitis cases with the aid of a software application (SA) developed by the Italian Society of Periodontology and Implantology. Ten general dentists were asked to independently assess 25 periodontitis cases using the SA. Accuracy was estimated using quadratic weighted kappa and examiners' percentage of agreement with a reference diagnosis provided by a gold standard examiner. Inter-rater agreement was evaluated using Fleiss kappa statistics. The overall case definition agreed with the reference diagnosis in 53.6% of cases. The agreements for each general dentist's pairwise comparisons against the reference definition were at least substantial in 100% of cases for stage, in 70% of cases for grade and in none of the cases for extent. Fleiss kappa was 0.818, 0.608, and 0.632 for stage, extent, and grade, respectively. The study recognized possible reasons that could lead to decreased accuracy using the SA. Supported by the SA, general dentists have reached substantial inter-rater agreement and highly accurate assignments of stage and grade. However, complete case definitions were correctly diagnosed in slightly over half of the cases.

Automatically Identifying Twitter Users for Interventions to Support Dementia Family Caregivers: Annotated Data Set and Benchmark Classification Models.

BackgroundMore than 6 million people in the United States have Alzheimer disease and related dementias, receiving help from more than 11 million family or other informal caregivers. A range of traditional interventions has been developed to support family caregivers; however, most of them have not been implemented in practice and remain largely inaccessible. While recent studies have shown that family caregivers of people with dementia use Twitter to discuss their experiences, methods have not been developed to enable the use of Twitter for interventions.ObjectiveThe objective of this study is to develop an annotated data set and benchmark classification models for automatically identifying a cohort of Twitter users who have a family member with dementia.MethodsBetween May 4 and May 20, 2021, we collected 10,733 tweets, posted by 8846 users, that mention a dementia-related keyword, a linguistic marker that potentially indicates a diagnosis, and a select familial relationship. Three annotators annotated 1 random tweet per user to distinguish those that indicate having a family member with dementia from those that do not. Interannotator agreement was 0.82 (Fleiss kappa). We used the annotated tweets to train and evaluate support vector machine and deep neural network classifiers. To assess the scalability of our approach, we then deployed automatic classification on unlabeled tweets that were continuously collected between May 4, 2021, and March 9, 2022.ResultsA deep neural network classifier based on a BERT (bidirectional encoder representations from transformers) model pretrained on tweets achieved the highest F1-score of 0.962 (precision=0.946 and recall=0.979) for the class of tweets indicating that the user has a family member with dementia. The classifier detected 128,838 tweets that indicate having a family member with dementia, posted by 74,290 users between May 4, 2021, and March 9, 2022—that is, approximately 7500 users per month.ConclusionsOur annotated data set can be used to automatically identify Twitter users who have a family member with dementia, enabling the use of Twitter on a large scale to not only explore family caregivers’ experiences but also directly target interventions at these users.

Metal artifact reduction in ultra-high-resolution cone-beam CT imaging with a twin robotic X-ray system

Cone-beam computed tomography (CBCT) has been shown to be a powerful tool for 3D imaging of the appendicular skeleton, allowing for detailed visualization of bone microarchitecture. This study was designed to compare artifacts in the presence of osteosynthetic implants between CBCT and multidetector computed tomography (MDCT) in cadaveric wrist scans. A total of 32 scan protocols with varying tube potential and current were employed: both conventional CBCT and MDCT studies were included with tube voltage ranging from 60 to 140 kVp as well as additional MDCT protocols with dedicated spectral shaping via tin prefiltration. Irrespective of scanner type, all examinations were conducted in ultra-high-resolution (UHR) scan mode. For reconstruction of UHR-CBCT scans an additional iterative metal artifact reduction algorithm was employed, an image correction tool which cannot be used in combination with UHR-MDCT. To compare applied radiation doses between both scanners, the volume computed tomography dose index for a 16 cm phantom (CTDIvol) was evaluated. Images were assessed regarding subjective and objective image quality. Without automatic tube current modulation or tube potential control, radiation doses ranged between 1.3 mGy (with 70 kVp and 50.0 effective mAs) and 75.2 mGy (with 140 kVp and 383.0 effective mAs) in UHR-MDCT. Using the pulsed image acquisition method of the CBCT scanner, CTDIvol ranged between 2.3 mGy (with 60 kVp and 0.6 mean mAs per pulse) and 61.0 mGy (with 133 kVp and 2.5 mean mAs per pulse). In essence, all UHR-CBCT protocols employing a tube potential of 80 kVp or more were found to provide superior overall image quality and artifact reduction compared to UHR-MDCT (all p < .050). Interrater reliability of seven radiologists regarding image quality was substantial for tissue assessment and moderate for artifact assessment with Fleiss kappa of 0.652 (95% confidence interval 0.618–0.686; p < 0.001) and 0.570 (95% confidence interval 0.535–0.606; p < 0.001), respectively. Our results demonstrate that the UHR-CBCT scan mode of a twin robotic X-ray system facilitates excellent visualization of the appendicular skeleton in the presence of metal implants. Achievable image quality and artifact reduction are superior to dose-comparable UHR-MDCT and even MDCT protocols employing spectral shaping with tin prefiltration do not achieve the same level of artifact reduction in adjacent soft tissue.

Reliability of the thoracolumbar injury classification and severity score compared with the McAfee classification among young neurosurgeons

Objective: The reliability of the thoracolumbar injury classification and severity (TLICS) score is well established; however, its reliability among young neurosurgeons in particular has not been investigated. This study was designed to identify intra- and inter-observer differences between the TLICS system and the McAfee classification among young neurosurgeons, with the goal of facilitating communication between physicians and treatment decision-making for patients with thoracolumbar injuries.Methods: Six young neurosurgeons reviewed thoracolumbar spinal fracture patients between January 2016 and October 2020 and analyzed thoracolumbar fractures according to the 2 classification systems. The intra- and inter-observer reliability of the TLICS and the McAfee scale was assessed with the Cohen and Fleiss kappa tests.Results: The intra-observer kappa value for the TLICS exhibited excellent reliability (κ=0.85) compared to the McAfee classification (κ=0.79). The inter-observer kappa values for each category of the TLICS were 0.69 (morphology), 0.93 (neurologic status), 0.74 (posterior ligamentous complex), and 0.72 (total score). The kappa value of the McAfee classification was lower (κ=0.52).Conclusion: The TLICS system showed higher reliability than the McAfee classification. The TLICS score showed more consistent results for thoracolumbar spinal fractures and may thus serve as a guideline for young neurosurgeons in treating patients with thoracolumbar fractures.

Development and validation of a new prognostic tool for impacted maxillary canines, 'Easy box': A cone-beam computed tomographic study.

To develop a tool to facilitate simple, easy and rapid assessment of the level of difficulty in disimpaction of an impacted canine using cone-beam computed tomography (CBCT). A cross-sectional study. A panel of four experienced orthodontists formulated a questionnaire capable of portraying the limits of a three-dimensional (3D) area around impacted maxillary canines on CBCT images, which would determine the disimpaction difficulty level. In total, 14 experienced orthodontists graded each item in the questionnaire and assigned scores in the range of 0-5. Average scores obtained outlined the limits of a 3D area, within which an impaction was deemed favourable. This area was termed 'Easy box'. The disimpaction difficulty level of 170 impacted maxillary canines was then assessed using 'Easy box' and the existing tool KPG index for convergent validation. Perfect inter-examiner agreement was obtained as evidenced by Fleiss kappa value of 0.984 for the tool 'Easy box' and 0.948 for the KPG index. There was also perfect agreement between the two methods of assessment (Cohen weighted kappa, k>0.86). The average time taken for assessment using 'Easy box' was 3.3 min compared to 8.8 min for the KPG index. 'Easy box' proves to be a simple, accurate and rapid assessment tool for the prognostic evaluation of impacted canines, using CBCT images. Impacted maxillary canines with cusp tips that lie beyond the limits of the 3D 'Easy box' area with a beta angle greater than 45°, were rated to be more difficult to align.

An emotion analysis dataset of course comment texts in massive online learning course platforms

ABSTRACT Datasets are critical for emotion analysis in the machine learning field. This study aims to explore emotion analysis datasets and related benchmarks in online learning, since, currently, there are very few studies that explore the same. We have scientifically labeled the topic and nine-category emotion of 4715 comment texts in online learning platforms using the “three-person voting label method” based on the “sentence-level” and multi-category labeling dimensions with our self-developed system. After testing the consistency of the labeling results using the Fleiss Kappa method, we found that the consistency of the dataset was about 0.51, representing a moderate strength of agreement. Based on the dataset, the prediction accuracy of the Long-Short Term Memory (LSTM) method is about 0.68. This dataset provides a benchmark for the multi-category emotion dataset in the Chinese online learning field. It can provide a basis for the subsequent solution of emotion analysis, monitoring, and intervention in the education field. It can also provide a reference for constructing subsequent datasets in the education field.

Development of a diagnostic indirect ELISA test for detection of Brucella antibody using recombinant outer membrane protein 16 kDa (rOMP16).

Brucellosis is considered as one of the important global zoonotic diseases that causes medical as well as economic problems especially in tropical countries. The illness has no specific pathognomonic signs; therefore, the rapid and accurate diagnosis of the disease has a very important role in preventing the Brucella spillover and treatment. The purpose of this study was to design a new indirect ELISA test for detection of human brucellosis based on using recombinant Brucella abortus outer membrane protein 16 kDa (rOMP16) as an antigen. OMP16 gene of B. abortus was initially synthesized and cloned in pET-21d vector and then expressed in Escherichia coli cells. The expression was confirmed by the SDS-PAGE, western blotting and dot blotting. The purified protein was coated in ELISA plates and an indirect ELISA was performed on 70 human serum samples. The results were evaluated with a commercial IgG ELISA kit and Rose Bengal plate agglutination tests as reference tests. Diagnostic performance of designed OMP16 ELISA test in comparison with Rose Bengal plate test revealed 100% of sensitivity, 95.00% of specificity and good Fleiss kappa agreement, whereas, where it was compared to commercial ELISA kit, it revealed very good kappa agreement with 100% of sensitivity and 100% of specificity in cut-off value of 0.13. It was concluded that OMP 16 kDa could be acceptable alternative antigen for detecting Brucella IgG antibody with high accuracy.

Comparison of a dichotomous versus trichotomous checklist for neonatal intubation

BackgroundTo compare validity evidence for dichotomous and trichotomous versions of a neonatal intubation (NI) procedural skills checklist.MethodsNI skills checklists were developed utilizing an existing framework. Experts were trained on scoring using dichotomous and trichotomous checklists, and rated recordings of 23 providers performing simulated NI. Videolaryngoscope recordings of glottic exposure were evaluated using Cormack-Lehane (CL) and Percent of Glottic Opening scales. Internal consistency and reliability of both checklists were analyzed, and correlations between checklist scores, airway visualization, entrustable professional activities (EPA), and global skills assessment (GSA) were calculated.ResultsDuring rater training, raters gave significantly higher scores on better provider performance in standardized videos (both p < 0.001). When utilized to evaluate study participants’ simulated NI attempts, both dichotomous and trichotomous checklist scores demonstrated very good internal consistency (Cronbach’s alpha 0.868 and 0.840, respectively). Inter-rater reliability was higher for dichotomous than trichotomous checklists [Fleiss kappa of 0.642 and 0.576, respectively (p < 0.001)]. Sum checklist scores were significantly different among providers in different disciplines (p < 0.001, dichotomous and trichotomous). Sum dichotomous checklist scores correlated more strongly than trichotomous scores with GSA and CL grades. Sum dichotomous and trichotomous checklist scores correlated similarly well with EPA.ConclusionsNeither dichotomous or trichotomous checklist was superior in discriminating provider NI skill when compared to GSA, EPA, or airway visualization assessment. Sum scores from dichotomous checklists may provide sufficient information to assess procedural competence, but trichotomous checklists may permit more granular feedback to learners and educators. The checklist selected may vary with assessment needs.

Upper lip tie: A novel classification scale with improved inter-rater reliability.

ObjectivesUpper lip tie (ULT) is a clinical condition with restrictive attachment of the superior labial frenulum (SLF), which may inhibit flanging of the lips. Objective outcome studies are lacking in part due to unreliable classification systems that rely solely upon a single attachment parameter of the SLF. This study's objectives are to describe a novel 3‐point classification system for ULT and compare its inter‐rater reliability to the Kotlow and Stanford systems.MethodsFive raters used the Kotlow and Stanford systems, as well as our novel 3‐point scale to score images of the SLF from 20 newborns seen at our institution between September 1, 2017 and April 1, 2018. Newborn birth weight, gestational age, and demographic data were collected from the infant's medical record. Fleiss's kappa was used to calculate inter‐rater reliability for all classification systems.ResultsThe parameters for our novel 3‐point classification system for ULT were as follows: length from alveolar edge to frenulum gingival attachment, length of frenulum on stretch, and free‐lip to total‐lip length ratio. Our novel scale yielded the highest inter‐rater reliability of 0.41, compared to 0.24 and 0.25 under the Kotlow and Stanford systems.ConclusionWhile the Kotlow and Stanford systems are based upon a single anatomical parameter, our novel 3‐point classification scale uses three oral parameters that encompass anatomical points of attachment as well as the maximal length of the ULT on stretch. Our classification scheme is the first to incorporate a functional parameter of the SLF, and thereby more fully characterizes ULT.Level of Evidence: Level 4.

Diagnostic Performance and Interreader Agreement of the MRI Clear Cell Likelihood Score for Characterization of cT1a and cT1b Solid Renal Masses: An External Validation Study.

BACKGROUND. The clear cell likelihood score (ccLS) has been proposed for the noninvasive differentiation of clear cell renal cell carcinoma (ccRCC) from other renal neoplasms on multiparametric MRI (mpMRI), though further external validation remains needed. OBJECTIVE. The purpose of our study was to evaluate the diagnostic performance and interreader agreement of the ccLS version 2.0 (v2.0) for characterizing solid renal masses as ccRCC. METHODS. This retrospective study included 102 patients (67 men, 35 women; mean age, 56.9 ± 12.8 [SD] years) who underwent mpMRI between January 2013 and February 2018, showing a total of 108 (≥ 25% enhancing tissue) solid renal masses measuring 7 cm or smaller (83 cT1a [≤ 4 cm] and 25 cT1b [> 4 cm and ≤ 7 cm]), all with a histologic diagnosis. Three abdominal radiologists independently reviewed the MRI examinations using ccLS v2.0. Median reader sensitivity, specificity, and accuracy were computed for predicting ccRCC by ccLS of 4 or greater, and individual reader AUCs were derived. The percentage of masses that were ccRCC was calculated, stratified by ccLS. Interobserver agreement was assessed by the Fleiss kappa statistic. RESULTS. The sample included 45 ccRCCs (34 cT1a, 11 cT1b), 30 papillary renal cell carcinomas (RCCs), 13 chromophobe RCCs, 14 oncocytomas, and six fat-poor angiomyolipomas. Median reader sensitivity, specificity, and accuracy for predicting ccRCC by ccLS of 4 or greater were 85%, 82%, and 83% among cT1a masses and 82%, 100%, and 92% among cT1b masses. The three readers' AUCs for predicting ccRCC by ccLS for cT1a masses were 0.90, 0.84, and 0.89 and for cT1b masses were 0.99, 0.97, and 0.92. Across readers, the percentage of masses that were ccRCC among cT1a masses was 0%, 0%, 20%, 68%, and 93% for ccLS of 1, 2, 3, 4, and 5, respectively; among cT1b masses, the percentage of masses that were ccRCC was 0%, 0%, 32%, 90%, and 100% for ccLS of 1, 2, 3, 4, and 5, respectively. Interobserver agreement among cT1a and cT1b masses for ccLS of 4 or greater was 0.82 and 0.83 and for ccLS of 1-5 overall was 0.65 and 0.62, respectively. CONCLUSION. This study provides external validation of the ccLS, finding overall high measures of diagnostic performance and interreader agreement. CLINICAL IMPACT. The ccLS provides a standardized approach to the noninvasive diagnosis of ccRCC by MRI.

Asian hate speech detection on Twitter during COVID-19.

Coronavirus disease 2019 (COVID-19) started in Wuhan, China, in late 2019, and after being utterly contagious in Asian countries, it rapidly spread to other countries. This disease caused governments worldwide to declare a public health crisis with severe measures taken to reduce the speed of the spread of the disease. This pandemic affected the lives of millions of people. Many citizens that lost their loved ones and jobs experienced a wide range of emotions, such as disbelief, shock, concerns about health, fear about food supplies, anxiety, and panic. All of the aforementioned phenomena led to the spread of racism and hate against Asians in western countries, especially in the United States. An analysis of official preliminary police data by the Center for the Study of Hate & Extremism at California State University shows that Anti-Asian hate crime in 16 of America's largest cities increased by 149% in 2020. In this study, we first chose a baseline of Americans' hate crimes against Asians on Twitter. Then we present an approach to balance the biased dataset and consequently improve the performance of tweet classification. We also have downloaded 10 million tweets through the Twitter API V-2. In this study, we have used a small portion of that, and we will use the entire dataset in the future study. In this article, three thousand tweets from our collected corpus are annotated by four annotators, including three Asian and one Asian-American. Using this data, we built predictive models of hate speech using various machine learning and deep learning methods. Our machine learning methods include Random Forest, K-nearest neighbors (KNN), Support Vector Machine (SVM), Extreme Gradient Boosting (XGBoost), Logistic Regression, Decision Tree, and Naive Bayes. Our Deep Learning models include Basic Long-Term Short-Term Memory (LSTM), Bidirectional LSTM, Bidirectional LSTM with Drop out, Convolution, and Bidirectional Encoder Representations from Transformers (BERT). We also adjusted our dataset by filtering tweets that were ambiguous to the annotators based on low Fleiss Kappa agreement between annotators. Our final result showed that Logistic Regression achieved the best statistical machine learning performance with an F1 score of 0.72, while BERT achieved the best performance of the deep learning models, with an F1-Score of 0.85.

Rainwater Charitable Foundation criteria for the neuropathologic diagnosis of progressive supranuclear palsy

Neuropathologic criteria for progressive supranuclear palsy (PSP) proposed by a National Institute of Neurological Disorders and Stroke (NINDS) working group were published in 1994 and based on the presence of neurofibrillary tangles in basal ganglia and brainstem. These criteria did not stipulate detection methods or incorporate glial tau pathology. In this study, a group of 14 expert neuropathologists scored digital slides from 10 brain regions stained with hematoxylin and eosin (H&E) and phosphorylated tau (AT8) immunohistochemistry. The cases included 15 typical and atypical PSP cases and 10 other tauopathies. Blinded to clinical and neuropathological information, raters provided a categorical diagnosis (PSP or not-PSP) based upon provisional criteria that required neurofibrillary tangles or pretangles in two of three regions (substantia nigra, subthalamic nucleus, globus pallidus) and tufted astrocytes in one of two regions (peri-Rolandic cortices, putamen). The criteria showed high sensitivity (0.97) and specificity (0.91), as well as almost perfect inter-rater reliability for diagnosing PSP and differentiating it from other tauopathies (Fleiss kappa 0.826). Most cases (17/25) had 100% agreement across all 14 raters. The Rainwater Charitable Foundation criteria for the neuropathologic diagnosis of PSP feature a simplified diagnostic algorithm based on phosphorylated tau immunohistochemistry and incorporate tufted astrocytes as an essential diagnostic feature.

Embryologist agreement when assessing blastocyst implantation probability: is data-driven prediction the solution to embryo assessment subjectivity?

Embryologist agreement when assessing blastocyst implantation probability: is data-driven prediction the solution to embryo assessment subjectivity?

Reliability and Construct Validity of the Yale Pharyngeal Residue Severity Rating Scale: Performance on Videos and Effect of Bolus Consistency.

The Yale Pharyngeal Residue Severity Rating Scale (YPRSRS) provides an image-based assessment of pharyngeal residue in the fiberoptic endoscopic evaluation of swallowing (FEES). Its performance was investigated only in FEES frames. This study analyzed the reliability and construct validity of the YPRSRS in FEES videos and the influence of bolus consistency. Thirty pairs of FEES videos and frames, 8 thin liquids (<50 mPa·s), 11 pureed (2583.3 mPa·s at 50 s−1, 697.87 mPa·s at 300 s−1), and 11 solid food; were assessed by 29 clinicians using the YPRSRS; 14 raters re-assessed materials at least 15 days from the first evaluation. Construct validity and intra-rater reliability were assessed using weighted Cohen’s Kappa. Inter-rater reliability was assessed using weighted Fleiss Kappa. Construct validity and inter-rater reliability were almost perfect or excellent for frames (0.82 ≤ k ≤ 0.89) and substantial or intermediate to good for videos (0.67 ≤ k ≤ 0.79). Intra-rater reliability was almost perfect for both frames and videos (k ≥ 0.84). Concerning bolus consistency, thin liquids had significantly lower values of construct validity, intra-, and inter-rater reliability than pureed and solid food. Construct validity and inter-rater reliability were significantly lower for solid food than for pureed food. The YPRSRS showed satisfactory reliability and construct validity also in FEES videos. Reliability was significantly influenced by bolus consistency.

O023 Artificial intelligence in photography of normothermic machine perfusion of livers

Abstract Introduction Livers donated for use in transplantation are underutilised. A lack of clear guidelines regarding the visual assessment of their quality means many borderline grafts are needlessly discarded; macroscopic visual assessment by transplanting clinicians is subjective and inconsistent. Deep learning (DL), a form of artificial intelligence, has previously been used to aid medical image analysis. Normothermic machine perfusion (NMP) is often used to evaluate borderline livers for transplantation. This study aimed to develop an objective DL model to visually assess organ transplantability during NMP. Methods Five DL models were trained and tested on 100 images of donor livers, each labelled with scores from three transplant clinicians on steatosis, perfusion, and transplantability. Models were trained to classify liver transplantability through either image data and transplantability scoring data, or image data and all clinician scoring data. Model accuracy, specificity, and sensitivity were calculated. Evaluations of clinician scoring agreement were carried out using intraclass correlation coefficient (ICC) and Fleiss kappa. Results In the classification of transplantability, the highest performing models achieved training and testing accuracies of 64.3% and 76%, respectively. Sensitivity and specificity ranged between 44.1%–94.1% and 6.3%–75%, respectively. ICC and Fleiss kappa values indicated a fair-to moderate scoring agreement between clinicians. Conclusion The performance of DL image analysis in assessing liver quality during NMP has been modest. Visual assessment during NMP is more challenging than on the back table at organ retrieval. Broader, more varied data sets are required to maximise model performance. Take-home message Livers donated for use in transplantation are underutilised due to a lack of clear guidelines and the subjective nature of their visual assessment. Artificial intelligence can be used to consistently and objectively assess donated livers in order to improve their utilisation.