Moderate Inter-rater Agreement Research Articles

The effect of tin prefiltration on extremity cone-beam CT imaging with a twin robotic X-ray system

IntroductionWhile tin prefiltration is established in various CT applications, its value in extremity cone-beam CT relative to optimized spectra has not been thoroughly assessed thus far. This study aims to investigate the effect of tin filters in extremity cone-beam CT with a twin-robotic X-ray system. MethodsWrist, elbow and ankle joints of two cadaveric specimens were examined in a laboratory setup with different combinations of prefiltration (copper, tin), tube voltage and current–time product. Image quality was assessed subjectively by five radiologists with Fleiss’ kappa being computed to measure interrater agreement. To provide a semiquantitative criterion for image quality, contrast-to-noise ratios (CNR) were compared for standardized regions of interest. Volume CT dose indices were calculated for a 16 cm polymethylmethacrylate phantom. ResultsRadiation dose ranged from 17.4 mGy in the clinical standard protocol without tin filter to as low as 0.7 mGy with tin prefiltration. Image quality ratings and CNR for tin-filtered scans with 100 kV were lower than for 80 kV studies with copper prefiltration despite higher dose (11.2 and 5.6 vs. 4.5 mGy; p < 0.001). No difference was ascertained between 100 kV scans with tin filtration and 60 kV copper-filtered scans with 75% dose reduction (subjective: p = 0.101; CNR: p = 0.706). Fleiss’ kappa of 0.597 (95% confidence interval 0.567–0.626; p < 0.001) indicated moderate interrater agreement. ConclusionConsiderable dose reduction is feasible with tin prefiltration, however, the twin-robotic X-ray system's low-dose potential for extremity 3D imaging is maximized with a dedicated low-kilovolt scan protocol in situations without extensive beam-hardening artifacts. Implications for practiceLow-kilovolt imaging with copper prefiltration provides a superior trade-off between dose reduction and image quality compared to tin-filtered cone-beam CT scan protocols with higher tube voltage.

Acromegaly disease activity according to ACRODAT\xae, a cross-sectional study in Spain: ACROVAL study

ObjectivesTo evaluate disease activity status using the Acromegaly Disease Activity Tool (ACRODAT®) in a cohort of Spanish acromegaly patients, to assess the relationship between the level of disease activity according to both ACRODAT® and the physicians’ clinical evaluation, and to study the potential discrepancies in the perception of symptoms between physicians and patients.DesignMulticenter, observational, descriptive and cross-sectional study.MethodsDisease activity was assessed in adult patients with acromegaly under pharmacological treatment during at least 6 months using ACRODAT®.ResultsAccording to ACRODAT®, 48.2%, 31.8% and 20.0% of a total of 111 patients were classified as having a stable disease (S), mild disease activity (M-DA) and significant disease activity (S-DA) respectively. ACRODAT® classification of disease activity significantly correlated with physicians’ opinion, with a moderate inter-rater agreement and a specificity of 92.45% (PPV = 86.21%). No correlation was found between IGF-I levels and severity of symptoms or quality of life (QoL). A decision to take clinical action was significantly more frequent in S-DA and M-DA patients than S patients but no action was taken on 5 (22.7%) and 27 (77.1%) S-DA and M-DA patients, respectivelyConclusionsACRODAT® detected disease activity in 51.8% of patients. Interestingly, although M-DA and S-DA patients were likely to be in the process of being controlled, action was not always taken on these patients. ACRODAT® is a validated and highly specific tool that may be useful to routinely monitor acromegaly and to identify patients with non-obvious disease activity by incorporating “patient-centred” parameters like symptoms and QoL to the clinical evaluation of acromegaly.

Evaluating Surgical Complexity of Endoscopic Hysterectomy: An Inter-Rater Agreement Study for Novel Scoring Tool

<h3>Study Objective</h3> Evaluate inter-rater reliability of a novel scoring tool for surgical complexity assessment of endoscopic hysterectomy. <h3>Design</h3> Validation study. <h3>Setting</h3> Academic medical center. <h3>Patients or Participants</h3> 11 academic OB/GYNS with varying years of post-training practice and surgical volumes <h3>Interventions</h3> Application of a novel scoring tool to evaluate surgical complexity of 150 sets of surgical images taken in a standardized fashion (global pelvis, anterior cul-de-sac, posterior cul-de-sac, right adnexa, left adnexa). Surgical complexity was staged on a scale of I-IV (low to high complexity). Raters were asked to assess uterine size, number and location of fibroids, adnexal and uterine mobility, need for ureterolysis, and presence of endometriosis or adhesions. <h3>Measurements and Main Results</h3> Number of post-residency years in practice for participating surgeons ranged from 2 – 15, with an average of 8 years. 8 (72.73%) OB/GYNs had completed a fellowship in minimally invasive gynecologic surgery. 6 (54.55%) reported annual volume of hysterectomy >50 cases. Raters reported 95.4% of images were satisfactory for assessment. Of the 150 sets of images, most were found to be Stage I-II complexity (Stage I: 46.4%, Stage II: 30.6%, Stage III: 16.4%, Stage IV: 6.6%). Level of inter-rater agreement regarding Stage I-II vs. III-IV complexity was moderate (k=0.493, CI [0.43-0.56]). Moderate inter-rater agreement persisted between surgeon raters with annual hysterectomy volume > 50 (k=0.484, CI [0.42-0.56]) as well as between surgeon raters with fellowship experience (k=0.498, CI [0.434-0.57]). <h3>Conclusion</h3> Current methods for assessment of surgical complexity for clinical and research purposes rely on seemingly objective markers like uterine weight, estimated blood loss, and operative time, which can be limited or flawed. In contrast, this novel scoring tool has rich and comprehensive evaluation capabilities and achieved moderate inter-rater agreement. The complexity scoring tool can be used in conjunction with or instead of traditional methods for assessment of surgical complexity of endoscopic hysterectomy.

A Crowdsourced Consensus on Supratotal Resection Versus Gross Total Resection for Anatomically Distinct Primary Glioblastoma.

Gross total resection (GTR) of contrast-enhancing tumor is associated with increased survival in primary glioblastoma. Recently, there has been increasing interest in performing supratotal resections (SpTRs) for glioblastoma. To address the published results, which have varied in part due to lack of consensus on the definition and appropriate use of SpTR. A crowdsourcing approach was used to survey 21 neurosurgical oncologists representing 14 health systems nationwide. Participants were presented with 11 definitions of SpTR and asked to rate the appropriateness of each definition. Participants reviewed T1-weighed postcontrast and fluid-attenuated inversion-recovery magnetic resonance imaging for 22 anatomically distinct glioblastomas. Participants were asked to assess the tumor location's eloquence, the perceived equipoise of enrolling patients in a randomized trial comparing gross total to SpTR, and their personal treatment plans. Most neurosurgeons surveyed (n=18, 85.7%) agree that GTR plus resection of some noncontrast enhancement is an appropriate definition for SpTR. Overall, moderate inter-rater agreement existed regarding eloquence, equipoise, and personal treatment plans. The 4 neurosurgeons who had performed >10 SpTRs for glioblastomas in the past year were more likely to recommend it as their treatment plan (P <.005). Cases were divided into 3 anatomically distinct groups based upon perceived eloquence. Anterior temporal and right frontal glioblastomas were considered the best randomization candidates. We established a consensus definition for SpTR of glioblastoma and identified anatomically distinct locations deemed most amenable to SpTR. These results may be used to plan prospective trials investigating the potential clinical utility of SpTR for glioblastoma.

Comparative study on the adequacy of cervical smears using wooden Ayre’s spatula, VS Papcone® sampling device

This is a comparative study on the adequacy of cervical smears obtained using the Papcone® sampling device or wooden Ayre’s spatula conducted from two tertiary health facilities –– Nnamdi Azikiwe University Teaching Hospital Nnewi and Chukwuemeka Odumegwu Ojukwu University Teaching Hospital Awka, in Anambra State, Nigeria. Slides from smears obtained using both devices were read by a cytopathologist blinded for the study. The primary outcome was the proportion of smears with an adequate endocervical component. Significantly higher adequate cervical smears were obtained in 177/192 (92.2%) women using the Papcone® sampling device, compared to 152/192 (79.2%) using wooden Ayre’s spatula (p < 0.001). Kappa analysis showed moderate inter-rater agreement between the two devices. We recommend the use of the Papcone device when it is available, as the adequacy of cervical smears obtained with the Papcone® was better than that obtained using wooden Ayre’s spatula.

Patient and physician shared decision-making behaviors in oncology: Evidence on adequate measurement properties of the iSHARE questionnaires

ObjectivesWe have developed two Dutch questionnaires to assess the shared decision-making (SDM) process in oncology; the iSHAREpatient and iSHAREphysician. In this study, we aimed to determine: scores, construct validity, test-retest agreement (iSHAREpatient), and inter-rater (iSHAREpatient-iSHAREphysician) agreement. MethodsPhysicians from seven Dutch hospitals recruited cancer patients, and completed the iSHAREphysician and SDM-Questionnaire-physician version. Their patients completed the: iSHAREpatient, nine-item SDM-Questionnaire, Decisional Conflict Scale, Combined Outcome Measure for Risk communication And treatment Decision-making Effectiveness, and five-item Perceived Efficacy in Patient-Physician Interactions. We formulated, respectively, one (iSHAREphysician) and 10 (iSHAREpatient) a priori hypotheses regarding correlations between the iSHARE questionnaires and questionnaires assessing related constructs. To assess test-retest agreement patients completed the iSHAREpatient again 1–2 weeks later. ResultsIn total, 151 treatment decision-making processes with unique patients were rated. Dimension and total iSHARE scores were high both in patients and physicians. The hypothesis on the iSHAREphysician and 9/10 hypotheses on the iSHAREpatient were confirmed. Test-retest and inter-rater agreement were>.60 for most items. ConclusionsThe iSHARE questionnaires show high scores, have good construct validity, substantial test-retest agreement, and moderate inter-rater agreement. Practice implicationsResults from the iSHARE questionnaires can inform both physician- and patient-directed efforts to improve SDM in clinical practice.

Deep learning in knee imaging: a systematic review utilizing a Checklist for Artificial Intelligence in Medical Imaging (CLAIM).

Our purposes were (1) to explore the methodologic quality of the studies on the deep learning in knee imaging with CLAIM criterion and (2) to offer our vision for the development of CLAIM to assure high-quality reports about the application of AI to medical imaging in knee joint. A Checklist for Artificial Intelligence in Medical Imaging systematic review was conducted from January 1, 2015, to June 1, 2020, using PubMed, EMBASE, and Web of Science databases. A total of 36 articles discussing deep learning applications in knee joint imaging were identified, divided by imaging modality, and characterized by imaging task, data source, algorithm type, and outcome metrics. A total of 36 studies were identified and divided into: X-ray (44.44%) and MRI (55.56%). The mean CLAIM score of the 36 studies was 27.94 (standard deviation, 4.26), which was 66.53% of the ideal score of 42.00. The CLAIM items achieved an average good inter-rater agreement (ICC 0.815, 95% CI 0.660-0.902). In total, 32 studies performed internal cross-validation on the data set, while only 4 studies conducted external validation of the data set. The overall scientific quality of deep learning in knee imaging is insufficient; however, deep learning remains a promising technology for diagnostic or predictive purpose. Improvements in study design, validation, and open science need to be made to demonstrate the generalizability of findings and to achieve clinical applications. Widespread application, pre-trained scoring procedure, and modification of CLAIM in response to clinical needs are necessary in the future. • Limited deep learning studies were established in knee imaging with mean score of 27.94, which was 66.53% of the ideal score of 42.00, commonly due to invalidated results, retrospective study design, and absence of a clear definition of the CLAIM items in detail. • A previous trained data extraction instrument allowed reaching moderate inter-rater agreement in the application of the CLAIM, while CLAIM still needs improvement in scoring items and result reporting to become a wide adaptive tool in reviews of deep learning studies.

An End-of-Life Communication Performance Rubric: Reliability Assessment.

Nurse educators still struggle with teaching and assessing end-of-life communication skills. Many resources are available to assist in teaching end-of-life communication, but few tools exist to assess performance learning outcomes. Behavior evaluation tools must be user-friendly and provide beneficial student feedback. Quality end-of-life patient care and nursing job satisfaction depend on skillful communication. The purpose of this study was to evaluate the reliability of an end-of-life communication clinical simulation evaluation performance rubric. Moderate interrater consistency and agreement were found between 3 evaluators when assessing the same students. All simulation evaluators agreed that the performance rubric was user-friendly and provided rich feedback for students during simulation debriefing. The performance evaluation tool was moderately effective when evaluating end-of-life communication performance and appears as a worthy framework for other behavior evaluations.

Evaluation of the Psychometric Properties of PainChek\xae in UK Aged Care Residents with advanced dementia

BackgroundThe aim of this study was to further validate PainChek®, an electronic pain assessment instrument, with a population living with dementia in a UK care home.MethodThis study utilised a correlational design to evaluate the psychometric properties of PainChek® when compared to the Abbey Pain Scale (APS). Blinded paired pain assessments were completed at rest and immediately post-movement by a researcher and a nurse. A total of 22 participants with a diagnosis of moderate-to-severe dementia and a painful condition were recruited using opportunity sampling.ResultsOverall, 302 paired assessments were collected for 22 participants. Out of these 179 were conducted during rest and 123 were immediately post-movement. The results demonstrated a positive significant correlation between overall PainChek® pain scores and overall APS pain scores (r = 0.818, N = 302, p < .001, one-tailed), satisfactory internal consistency (α = 0.810), moderate single measure intraclass correlation (ICC = 0.680) and substantial inter-rater agreement (κ = 0.719).ConclusionsPainChek® has demonstrated to be a valid and reliable instrument to assess the presence and severity of pain in people with moderate-to-severe dementia living in aged care.

Accuracy of magnetic resonance imaging in predicting anterior cruciate ligament tear location and tear degree

PurposeThe purpose of this study is to evaluate the reliability of magnetic resonance imaging (MRI) in predicting the location of ACL tears in preoperative planning for anterior cruciate ligament (ACL) repair. MethodsThirty-four patients who underwent ACL repair were retrospectively analyzed to compare intraoperative arthroscopic findings with preoperative MRIs. ResultsFor identifying type I tears, the sensitivity of MRI was 9.0% and the accuracy of MRI was 8.8%. There was moderate interrater agreement between MRI findings for tear location and tear degree. ConclusionMRI alone may not necessarily be accurate in identifying which ACL tears are amenable to repair. Study designRetrospective case series; Level of Evidence: IV;

Tissue sample analysis for post mortem determination of brain edema

The post mortem evaluation of a brain edema is routinely performed by pathologists based on the macroscopic signs during autopsy. This method represents the current gold standard, but is subjective and observer dependent. Therefore, three post mortem evaluation methods of brain samples were analyzed in this work: histology, wet-dry weight and normalized cerebral weight, which was described in 2020 by Bauer et al.Tissue samples from six different regions of 34 brains were collected and examined both by rating of histological slides and by measuring the water content by using a drying oven. The rating of the histological slides, stained with hematoxylin and eosin, was performed by two pathologists independently. For the water content, the wet weight and the dry weight of each sample were set in relation. The normalized cerebral weight was calculated by dividing the brain weight by the brain volume, which were both determined during autopsy and in computed tomography images, respectively.A fair to moderate interrater agreement was obtained for the histologic evaluation and a significant correlation was present between one rater and the wet-dry weight and the normalized cerebral weight method. When classifying according to the gold standard, a significant difference was detected between the edematous and nonedematous cases by using the wet-dry weight method in the cerebral cortex and by using the normalized cerebral weight method. However, the significant correlations and group differences were limited to the aforementioned results.In conclusion, both the histological and the wet-dry weight method show limited benefits for the classification of brain edema and the histology analysis is highly observer dependent. The normalized cerebral weight method, however, reveals a significant effect between the edematous and nonedematous cases when classifying according to the gold standard. Therefore, we suggest to apply this method for the assessment of brain edema since it is objective and rater independent. Nevertheless, the exact evaluation of brain edema remains a challenging task, especially due to the continuous transition between no edema and edema.

Appropriate responses to potential child abuse: The importance of information quality

BackgroundWhen people suspect a child or young person is experiencing, or at risk of, abuse or neglect, they have to decide how to respond. However, the under and over reporting of child welfare issues indicate that people may struggle to identify an appropriate response. ObjectiveTo develop scenarios (for future training and research purposes) that closely resemble the child welfare situations health/allied health practitioners encounter, and for which there is a reasonable level of child protection professional consensus as to what the appropriate response for each situation should be. Participants, setting, and methodsWe developed 285 scenarios from 190 child protection reports made by health/allied health practitioners to two Australian government child welfare agencies, that covered a range of appropriate response pathways and abuse types. An appropriate response pathway for each scenario was identified by having 34 child protection professionals provide their opinion and rationales. ResultsChild protection professionals displayed moderate (e.g., krippendorf’s alpha = 0.58, 95 % CI: 0.52 to 0.62) interrater agreement as to the appropriate response pathway for the scenarios. For 127 of the 285 scenarios (44.56 %), there was strong consensus (K = 0.73, 95 % CI: 0.66 to 0.78). ConclusionProfessional consensus was higher than anticipated from previous research, although still low compared to generally acceptable levels of consensus. Our results suggest several promising avenues to increase professional consensus, such as improving the quality of information that people typically report to child welfare agencies.

Delayed Contrast-Enhanced MR Angiography for the Assessment of Internal Carotid Bulb Patency in the Context of Acute Ischemic Stroke: An Accuracy, Interrater, and Intrarater Agreement Study.

CTA has shown limited accuracy and reliability in distinguishing tandem occlusions and pseudo-occlusions on initial acute stroke imaging. The utility of early and delayed contrast-enhanced MRA in this setting is unknown. We aimed to assess the accuracy and reliability of early and delayed contrast-enhanced MRA for carotid bulb patency in patients with acute ischemic stroke. We retrospectively reviewed patients who had ICA occlusion and underwent thrombectomy with preprocedural early and delayed contrast-enhanced MRA in a single comprehensive stroke center. During 2 sessions, 10 raters independently assessed 32 cases with early contrast-enhanced MRA (with an additional delayed contrast-enhanced MRA sequence during the second reading session). Their judgments were compared with DSA as a reference standard. Accuracy and interrater agreement were measured. Five raters undertook a third reading session to assess intrarater agreement. Accuracy for the assessment of carotid bulb patency with early contrast-enhanced MRA was limited (69%; 95% CI, 59%-79%), with moderate interrater agreement (κ = 0.42; 95% CI, 0.27-0.55). The second reading with an additional delayed contrast-enhanced MRA sequence improved both accuracy (82%; 95% CI, 73%-91%; P < .001) (raters corrected 43%-77% of incorrect diagnoses with early contrast-enhanced MRA alone; mean = 59%) and interrater agreement (κ = 0.56; 95% CI, 0.41-0.73; P = .07). Intrarater agreement was almost perfect, substantial, and moderate for 3, 1, and 1 raters. Early contrast-enhanced MRA has limited accuracy and repeatability for the evaluation of carotid bulb patency in acute ischemic stroke. The additional delayed contrast-enhanced MRA sequence may improve accuracy and reliability.

99mTc-white blood cell SPECT/CT to assess diabetic foot osteomyelitis remission: contribution of semi-quantitative scoring system.

We previously reported that 99mTc-White blood cell (WBC) single-photon emission computed tomography (SPECT/CT) could be a useful tool to assess diabetic foot osteomyelitis (DFO) remission and guide the duration of antibiotic treatment. The aim of the present study was to evaluate the performance and reproducibility of two analysis methods to diagnose DFO remission using 99mTc-WBC-SPECT/CT. 99mTc-WBC-SPECT/CT performed for patients with DFO at the end of antibiotic treatment were retrospectively read by two nuclear physicians (one senior and one junior). Assessment by conventional visual analysis and by the use of a semi-quantitative scoring system, the composite score index (CSI), was performed. The performance and reproducibility of methods were compared between the two nuclear physicians. Successful treatment of DFO was defined by the absence of DFO relapse at the same site within 1 year. A total of 68 patients with 74 DFO were included. Three were excluded from the analysis due to the low quality of SPECT/CT; among the 71 DFO analyzed, 11 (15.5%) had a relapse during follow-up. Performances of 99mTc-WBC-SPECT/CT to predict DFO remission with conventional visual assessment were significantly lower for junior than for senior nuclear physician with moderate inter-rater agreement (Kappa: 0.417). Performances with the use of CSI were similar between the two readers with good inter-rater agreement (Kappa: 0.756). The study found that conventional visual assessment of 99mTc-WBC-SPECT/CT to assess DFO remission requires experience, and supported that CSI could be useful for junior nuclear physician to discriminate residual infections and inflammatory post-treatment uptake.

Pediatric adaptions are needed to improve the diagnostic accuracy of thyroid ultrasound using TI-RADS

Background/PurposeThyroid Imaging Reporting and Data System (TI-RADS) is validated in adults but not yet in children. The purpose of this study was to determine the sensitivity, specificity, and accuracy of TI-RADS in predicting thyroid malignancy for pediatric nodules, and to compare the diagnostic accuracy to the current American Thyroid Association (ATA) guidelines. MethodsA single institution retrospective review was performed of patients younger than 21 years who underwent thyroid nodule fine needle aspiration biopsy (FNAB). Two radiologists were blinded to the pathology and independently classified all biopsied thyroid nodules based on TI-RADS. ATA and TI-RADS guidelines were analyzed to determine the diagnostic sensitivity and specificity of both scoring systems. Results115 patients (median age 15.5 years, 90 females) with 138 nodules were scored using TI-RADS. There was moderate inter-rater agreement between radiologists (Kappa = 0.51; p < 0.0001). Evaluating several potential TI-RADS criteria, 23.2%-68.1% of nodules were recommended for FNAB, compared to 82.6% of nodules using ATA guidelines. Using TI-RADS ≥ 3 (without size cutoff) as an indication for FNAB had 100% sensitivity with no missed suspicious or malignant nodules on cytology or pathology. ConclusionsUsing TI-RADS for diagnostic management of pediatric thyroid nodules improves accuracy in predicting malignancy.

Accuracy of grading pancreatic neuroendocrine tumors on endoscopic ultrasound-guided fine needle aspiration: a multi-institutional study

Presenter: Ammar Javed MD | Johns Hopkins University School of Medicine Background: World Health Organization (WHO) grading system is prognostic in pancreatic neuroendocrine tumors (PanNETs). The concordance between WHO grade on cytological analysis (c-grade) of endoscopic ultrasound-guided fine needle aspiration (EUS-FNA) and histopathological analysis (h-grade) of surgical specimen is reported to be between 60 and 80%. Factors associated with concordance and trends of utilization of EUS-FNA for grading of PanNETs remain poorly understood. Methods: A multicenter retrospective study was performed on patients undergoing resection for PanNETs at four high-volume centers. Patients with functional or syndrome associated tumors, and those who received neoadjuvant therapy were excluded. Factors associated with concordance between c-grade and h-grade and trends of utilization of EUS-FNA for PanNETs over the last two decades were assessed. Results: Of the 1,398 patients included in the study, 704 (50.4%) underwent EUS-FNA; 579 (80.25) were diagnostic of PanNETs and WHO grade was reported for 296 (51.1%) patients. The concordance between c-grade and h-grade was 74.7% with moderate interrater agreement (Kc = 0.42, p<0.001). Significantly higher rates of concordance were observed in patients with smaller tumors ( < 2 vs. ≥2 cm, 70.1% vs. 29.9%, p = 0.030). The highest concordance was 100.0%and was observed in patients with small tumors without perineural invasion undergoing assessment between 2015-2019; perfect interrater agreement (Kc = 1.00, p<0.001). Over the last two decades an increase in the utilization of EUS-FNA from 31.5% to 61.4% was observed (p<0.001). Furthermore, EUS-FNA was more frequently diagnostic of PanNETs (p<0.001), and WHO grade was more frequently reported ( < 0.001). Despite increased utilization of EUS-FNA the rate of concordance did not change (p = 0.338). Conclusion: Higher rates of concordance between c-grade and h-grade are observed in patients with small PanNETs without perineural invasion. While an increase in the utilization and diagnostic accuracy of EUS-FNA has been observed, the concordance remains unchanged, suggesting a need for improved techniques and devices for accurate grading of PanNETs

The inter-rater reliability and individual reviewer performance of the 2012 world heart federation guidelines for the echocardiographic diagnosis of latent rheumatic heart disease

BackgroundIn 2012, the World Heart Federation (WHF) published guidelines for the echocardiographic diagnosis of rheumatic heart disease (RHD). This study assesses individual reviewer performance and inter-rater agreement and reliability on the presence of any RHD, as well classification of RHD based on the 2012 WHF criteria. MethodsFour cardiologists individually reviewed echocardiograms in the context of a randomized clinical trial (ClinicalTrials.gov:NCT03346525) and participated in a blinded adjudication panel. Panel decision was the reference standard for diagnosis. Performance of individual reviewers to panel adjudication was compared through sensitivity and specificity analyses and inter-rater reliability was assessed between individual panelists using Fleiss free marginal multirater kappa. ResultsEchocardiograms from 784 children had two independent reports and panel adjudication. The accuracy of independent reviewers for any RHD had high sensitivity (94%, 95% CI 93–95%) and moderate specificity (62%, 95% CI 53–70%). Sensitivity and specificity for definite RHD was 61.3 (95% CI, 55.3–67.1) and 93.1 (95% CI, 91.6–94.4), with 86.8 (84.7–88.7) and 65.8 (61.0–70.4) for borderline RHD. There was moderate inter-rater agreement (κ = 0.66) on the presence of any RHD while agreement for specific 2012 WHF classification was only fair (κ = 0.51). ConclusionsThe 2012 WHF guidelines are moderately reproducible when used by expert cardiologists. More cases of RHD were diagnosed by an consensus panel than by individual reviewers. A revision to the criteria is now warranted to further increase the reliability of the WHF criteria.

SURG-21. A CROWDSOURCED CONSENSUS ON SUPRATOTAL RESECTION VERSUS GROSS TOTAL RESECTION FOR ANATOMICALLY DISTINCT PRIMARY GLIOBLASTOMA

Abstract Gross total resection (GTR) of contrast-enhancing tumor is associated with significantly increased overall survival in primary glioblastoma (GBM). Even when achieved, recurrence is likely, in part due to malignant cells infiltrating outside enhanced regions. Subsequently, there has been increasing interest in performing supratotal resections (SpTRs) for GBM. Published results have varied in part due to a lack of consensus on the definition of SpTR in GBM and its appropriate use. A crowdsourcing approach was used to survey 21 academic neurosurgical oncologists representing 13 health systems nationwide. Participants’ demographics including fellowship training status, years of experience, and operative volume with various techniques was collected. Participants were presented with 11 definitions of SpTR from published, peer-reviewed studies and asked to rate the appropriateness of each definition. Subsequently, participants reviewed T1-weighed post-contrast and FLAIR MR imaging videos in the axial, coronal, and sagittal planes for 22 GBMs. Participants were asked to assess eloquence of the tumor’s location, perceived equipoise of enrolling patients in a randomized clinical trial comparing GTR to SpTR, and their own personal surgical treatment plans. Most neurosurgeons surveyed (n=18, 85.7%) agree or strongly agree that GTR plus resection of some non-contrast enhancement is an appropriate definition for SpTR. Overall, there was only moderate inter-rater agreement, measured using sample variance and the index of qualitative variation, regarding eloquence, equipoise, and personal treatment plans. Neurosurgeons who performed more than 10 SpTRs for GBMs in the past year were more likely than counterparts to recommend it as their personal treatment plan (p&amp;lt; 0.005). Anterior temporal and right frontal GBMs were considered the best randomization candidates. We established a consensus definition for SpTR of GBM and identified anatomically distinct locations deemed most amenable to SpTR. These results will be used to plan prospective trials further investigating the potential clinical utility of SpTR for GBMs.

Peer review in MRI: A quality improvement programme and pilot study

IntroductionThe aim of this pilot study was to implement a peer review programme to evaluate MRI images and protocols, with the goal of improving quality and standardising protocols. MethodA plan-do-check-act method of action research was adopted. A checklist was designed and two radiographers scored the images (Optimal, Diagnostic or Suboptimal). A sample of five Lumbar Spine examinations were selected and reviewed on two separate occasions, one month apart. A consultant radiologist also scored the selected examinations. ResultsThere was excellent intra-rater reliability for both observers. At first assessment, Cohen's weighted Kappa analysis indicated moderate inter-rater agreement (0.457) rising to substantial agreement (0.606) at second assessment. Variations in sequence parameter settings by the radiographers were noted. Feedback via educational sessions were implemented and enhancement of protocols were made. ConclusionsRadiographers manipulate many parameters during MRI examinations, and image quality can be degraded if sequences are not optimised. This may lead to misinterpretation.Initial results suggest peer review can result in improved image quality, better protocols, and improved staff confidence.There are opportunity costs involved in releasing staff from clinical work and time and resources required for image appraisal training, however we suggest this model can be extended to other departments and modalities as a healthcare improvement initiative. Implications for practiceInitial results suggest peer review processes performed by radiographers can result in improvement to protocols and image quality in MRI. With appropriate training, radiographers can implement and undertake appraisal of image quality in MRI as part of clinical governance. The purpose of feedback must be made clear, and good professional relationships should be forged to allow candid and supportive feedback with a focus on development and education.

Diagnostic accuracy and interobserver variability of CO-RADS in patients with suspected coronavirus disease-2019: a multireader validation study.

ObjectiveTo conduct a multireader validation study to evaluate the interobserver variability and the diagnostic accuracy for the lung involvement by COVID-19 of COVID-19 Reporting and Data System (CO-RADS) score.MethodsThis retrospective study included consecutive symptomatic patients who underwent chest CT and reverse transcriptase-polymerase chain reaction (RT-PCR) from March 2020 to May 2020 for suspected COVID-19. Twelve readers with different levels of expertise independently scored each CT using the CO-RADS scheme for detecting pulmonary involvement by COVID-19. Receiver operating characteristic (ROC) curves were computed to investigate diagnostic yield. Fleiss’ kappa statistics was used to evaluate interreader agreement.ResultsA total of 572 patients (mean age, 63 ± 20 [standard deviation]; 329 men; 142 patients with COVID-19 and 430 patients without COVID-19) were evaluated. There was a moderate agreement for CO-RADS rating among all readers (Fleiss’ K = 0.43 [95% CI 0.42–0.44]) with a substantial agreement for CO-RADS 1 category (Fleiss’ K = 0.61 [95% CI 0.60–0.62]) and moderate agreement for CO-RADS 5 category (Fleiss’ K = 0.60 [95% CI 0.58–0.61]). ROC analysis showed the CO-RADS score ≥ 4 as the optimal threshold, with a cumulative area under the curve of 0.72 (95% CI 66–78%), sensitivity 61% (95% CI 52–69%), and specificity 81% (95% CI 77–84%).ConclusionCO-RADS showed high diagnostic accuracy and moderate interrater agreement across readers with different levels of expertise. Specificity is higher than previously thought and that could lead to reconsider the role of CT in this clinical setting.Key Points• COVID-19 Reporting and Data System (CO-RADS) demonstrated a good diagnostic accuracy for lung involvement by COVID-19 with an average AUC of 0.72 (95% CI 67–75%).• When a threshold of ≥ 4 was used, sensitivity and specificity were 61% (95% CI 52–69%) and 81% (95% CI 76–84%), respectively.• There was an overall moderate agreement for CO-RADS rating across readers with different levels of expertise (Fleiss’ K = 0.43 [95% CI 0.42–0.44]).Electronic supplementary materialThe online version of this article (10.1007/s00330-020-07273-y) contains supplementary material, which is available to authorized users.