Inter-reviewer Variability Research Articles

"Real-world" performance of the Confirm Rx™ SharpSense AF detection algorithm: UK Confirm Rx study.

The novel Confirm Rx™ implantable cardiac monitor (ICM) with SharpSense™ technology incorporates a new P-wave discriminator designed to improve AF detection. This study aimed to evaluate the diagnostic performance of the Confirm Rx™ ICM in detecting AF episodes of varying durations. We conducted a multicenter retrospective analysis of consecutive patients implanted with a Confirm Rx™ ICM (v1.2) across nine UK hospitals, all with documented AF lasting at least 6 min. Electrocardiograms (ECGs) were manually adjudicated by cardiologists. To account for intra- and inter-reviewer variability, a random sample of 10% of ECGs underwent additional review. Disagreements were resolved by a third reviewer. Diagnostic performance was determined by calculating the gross and patient-averaged positive predictive value (PPV) for AF episodes of different duration. The source of false positive (FP) detection was also categorized. Overall, 16,230 individual ECGs from 232 patients were included. The median AF episode duration was 14 min. R-wave amplitude remained stable during follow-up (0.52 ± 0.27 mV [initial] vs. 0.54 ± 0.29 mV [end of follow-up], p = .10). The gross and patient-averaged PPV were 75.0% and 67.0%, respectively. Diagnostic performance (gross) increased with progressively longer AF episodes: 88.0% for ≥1 h, 97.3% for 6 h, and 100% for 24 h. The main source of FP during tachycardia was T-wave oversensing (54.2%), while in non-tachycardic episodes it was predominantly ectopy (71.2%). The AF burden precision was excellent (93.3%). The Confirm Rx™ ICM diagnostic performance was modest for all AF episodes (75%), with accuracy increasing for longer AF episodes.

Abstract 7379: Deep learning based risk stratification of pre-operative CT scans is prognostic of overall survival in kidney cancers

Abstract Introduction: Preoperative risk stratification of kidney cancer patients is critical for treatment selection and improving mortality rates. Current clinical approaches such as cancer grading are limited by inter-reviewer variability, require invasive biopsy procedures, and might not capture the full extent of patient risk. We sought to evaluate a deep learning (DL) model to quantify patient risk and prognosticate overall survival in kidney cancer patients via CT scans. Methods: This retrospective study used preoperative contrast CT scans from 4 public multi-institutional collections with date of death/censoring available. A deep learning (DL) network architecture was developed to analyze input CT scans via a convolutional feature extractor and a Cox proportional hazards-based cost function. The DL model assigned a risk score to each patient, with the median risk used as a cutoff for classifying patients as high-risk or low-risk. Model performance for prognosticating overall survival (OS) was evaluated separately in training, internal validation, and external validation cohorts via Kaplan-Meier analysis in terms of concordance index and hazard ratios, with risk groups compared via Wilcoxon log-rank tests (α = 0.05). Results: In the training/internal validation cohort (N=418, 8 institutions) and external validation cohort (N=102, 1 institution), the median age was 60 and 59 years, the male/female ratios were 62%/38% and 75%/25%, with a median follow up of 2.8 and 3.1 years, respectively. The DL model yielded a training c-index of 0.66 (95% CI: 0.63-0.68), internal validation c-index of 0.73 (95% CI: 0.70-0.76) and an external validation c-index of 0.67 (95% CI: 0.64-0.70). Kaplan-Meier analysis of associated survival curves based on DL-determined high- and low-risk patients demonstrated statistically significant separation in all cohorts: training (p = 0.003, HR = 8.63, 95% CI: 8.28-8.98), internal validation (p = 0.027, HR = 4.89, 95% CI: 4.52-5.26), and external validation (p = 0.01, HR = 6.54, 95% CI: 6.18-6.90). Baseline clinicopathological characteristics were not significantly different between DL high and low risk patients in external validation (chi-square testing; ISUP grade p=0.375, age p=0.824, race p=0.567, and sex p=0.440). ISUP grade was also not a significant predictor of OS in internal validation (HR=1.57, p=0.211, 95% CI: 1.28-1.85) or external validation (p = 0.465, HR=0.16, 95% CI: 0.33-0.74). Conclusion: A deep learning model based on pre-operative CT scans was found to accurately prognosticate overall survival in kidney cancers in a multi-institutional setting, independent of clinicopathological factors. This model is being validated on larger cohorts, and in the context of targeted therapies. Citation Format: Brennan Flannery, Mohsen Hariri, Thomas DeSilvio, Amir Sadri, Jane Nguyen, Erick M. Remer, Smitha Krishnamurthi, Satish E. Viswanath. Deep learning based risk stratification of pre-operative CT scans is prognostic of overall survival in kidney cancers [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 7379.

B-280 Sensitivity and Specificity of Beta-2 Transferrin Gel Electrophoresis With Immunofixation for Evaluating Nasal Cerebrospinal Fluid Leaks

Abstract Background Cerebrospinal fluid (CSF) rhinorrhea usually presents as unilateral clear thin nasal drainage, is caused by a direct communication from the subarachnoid space to the sinonasal cavities and can lead to intracranial infection, pneumocephalus, or death. Diagnosing CSF rhinorrhea can be challenging. While the gold standard for CSF leak confirmation is surgical exploration, testing nasal fluid for beta-2 transferrin (B2Tf) via agarose gel electrophoresis followed by immunofixation (AGEI) remains the initial non-invasive CSF confirmatory test of choice. However, AGEI requires subjective interpretation of immune-stained protein bands. Multiple studies have reported high sensitivity and specificity of B2Tf AGEI (90–100%), but other studies have shown wider ranges from 70–100%. Due to small sample sizes and methodologic differences between studies, comparing results between studies is problematic. The purpose of this study was to assess the sensitivity and specificity of B2Tf AGEI in confirming or excluding CSF rhinorrhea in patients with unilateral clear thin nasal drainage, and to determine the inter-reviewer variability of AGEI interpretation. Methods This was a retrospective observational study of patients who had B2Tf AGEI performed on their unilateral clear nasal drainage from 2020 through 2022. A pathologist and four trained medical laboratory scientists conducted visual reviews of archived gels and their results were compared to the documented CSF rhinorrhea status. Results were also compared to the original test results documented in patients’ charts. Sensitivity and specificity, and inter-reviewer agreement were assessed for B2Tf AGEI confirming or excluding CSF rhinorrhea. The gold standard for confirming CSF rhinorrhea was surgical exploration. The gold standard for excluding CSF rhinorrhea included either negative surgical exploration or rhinorrhea resolution after medical therapy aimed at treating rhinitis or rhinosinusitis. Results Sensitivity and specificity varied for the five reviewers and documented test results, with sensitivities ranging from 65.0 to 86.0% (mean ± SD, 73.0 ± 5.7) and specificities from 61.4 to 88.6% (84.1 ± 11.0). The false positive rate was 8.7%, and false negative rate was 13.9%. Sensitivity and specificity for the documented test results ranging from 57.1 to 82.1% (mean ± SD, 72.1, 9.3) and specificities from 75.0 to 97.26% (91.6 ± 9.4). The abilities of the reviewers to reliability identify a CSF leak was moderate, with kappa statistics ranging from 0.313 to 0.646 (weighted average 62.2, 4.0). Two of the four reviewers demonstrated substantial to almost perfect agreement to the expert reviewer, with the other two demonstrating moderate agreement. Conclusions B2Tf AGEI demonstrated lower sensitivity and specificity than some previous reports, however this study had a larger sample size and very well-defined clinical standards for CSF and non-CSF rhinorrhea. While false results were present in 9 to 14% of cases, the test is non-invasive, and is helpful in stratifying patients into those more or less likely to have CSF rhinorrhea. There was also substantial inter-reviewer variability, highlighting the challenges of subjective interpretation of this testing modality. Larger prospective studies would be helpful to more accurately determine the diagnostic accuracy of B2Tf AGEI, as well as explore practices to help limit inter-reviewer variability.

Comparison of a machine and deep learning model for automated tumor annotation on digitized whole slide prostate cancer histology.

One in eight men will be affected by prostate cancer (PCa) in their lives. While the current clinical standard prognostic marker for PCa is the Gleason score, it is subject to inter-reviewer variability. This study compares two machine learning methods for discriminating between cancerous regions on digitized histology from 47 PCa patients. Whole-slide images were annotated by a GU fellowship-trained pathologist for each Gleason pattern. High-resolution tiles were extracted from annotated and unlabeled tissue. Patients were separated into a training set of 31 patients (Cohort A, n = 9345 tiles) and a testing cohort of 16 patients (Cohort B, n = 4375 tiles). Tiles from Cohort A were used to train a ResNet model, and glands from these tiles were segmented to calculate pathomic features to train a bagged ensemble model to discriminate tumors as (1) cancer and noncancer, (2) high- and low-grade cancer from noncancer, and (3) all Gleason patterns. The outputs of these models were compared to ground-truth pathologist annotations. The ensemble and ResNet models had overall accuracies of 89% and 88%, respectively, at predicting cancer from noncancer. The ResNet model was additionally able to differentiate Gleason patterns on data from Cohort B while the ensemble model was not. Our results suggest that quantitative pathomic features calculated from PCa histology can distinguish regions of cancer; however, texture features captured by deep learning frameworks better differentiate unique Gleason patterns.

Evaluating the Inter-Reviewer Reliability and Evidentiary Grounds for Third-Party Peer-to-Peer Mandated Downgrading of Radiation Therapy Prescriptions

<h3>Purpose/Objective(s)</h3> Health plans employ utilization management strategies to control access to certain treatments in a bid to cut health care spending, frequently requiring prior authorization (PA) and subsequent peer-to-peer (P2P) discussion if the prescribed studies or treatments are not initially approved. In theory, P2P reviewers should follow current, evidence-based clinical guidelines and have knowledge of novel treatments to optimize patient care. We examine P2P determinations to downgrade prescriptions (Rx) of radiotherapy (RT) and evaluate both the evidentiary basis and internal consistency of these decisions, as applied to patients at a large academic radiation medicine department. <h3>Materials/Methods</h3> De-identified departmental billing records from 1/12/21 to 8/12/21 were used to query cases in which an Rx for RT was denied PA and ultimately downgraded on P2P in terms of RT technique, fraction number, or use of image-guided radiation therapy (IGRT). These were cross-identified with patient charts in an IRB-approved electronic medical record, and clinical details reviewed. The P2P record was searched for pair matches for each downgraded Rx (i.e., anatomic site, modality, fractionation), such that the match was approved unaltered. PubMed searches were performed for literature supporting the initial Rx in terms of control, survival, or toxicity outcomes. <h3>Results</h3> 1620 Rx were prescribed during the analyzed period, of which 697 cases were referred for PA. 123 (17.6%) of the latter were denied and referred to P2P. 31 of the 123 (25.2%) resulted in a downgraded Rx, of which 13 (41.9%) were for curative treatment, 8 (25.8%) for treatment of oligometastatic disease (OMD), and 10 (32.3%) for non-OMD palliative treatment. No Rx were downgraded for fraction number. 15 downgraded Rx (48.4%) were pair-matched with Rx that were approved unaltered on P2P. 3 (9.6%) downgraded Rx were matched with approved Rx for higher grade modality (e.g., IMRT over 3DCRT) for equivalent indications. 18 of 31 downgraded Rx (58.1%) were matched with approvals for identical or upgraded Rx, including 10 curative cases, 7 OMD cases, and 2 of 5 cases in which IGRT alone was excluded on P2P. For 17 downgraded Rx (54.8%), literature in support of the original Rx was identified, including at minimum a phase II study, a meta-analysis of > 1000 subjects, or a retrospective review with between 86 and 202 subjects. Multiple phase II studies supported all Rx for OMD that were downgraded. <h3>Conclusion</h3> Our analysis shows that more than half of treatment downgrades resulting from the P2P process can be matched with P2P approvals of the same or upgraded treatment for the same indication. We further show that published evidence could have upheld the initial prescriptions in more than half of P2P downgrades. These results suggest a high degree of inter-reviewer variability and inconsistent fidelity to evidence-based management in P2P decisions to downgrade RT Rx. Multi-institutional quality review of ultimate P2P dispositions is warranted.

Peer review: Risk and risk tolerance

Peer review, commonly used in grant funding decisions, relies on scientists’ ability to evaluate research proposals’ quality. Such judgments are sometimes beyond reviewers’ discriminatory power and could lead to a reliance on subjective biases, including preferences for lower risk, incremental projects. However, peer reviewers’ risk tolerance has not been well studied. We conducted a cross-sectional experiment of peer reviewers’ evaluations of mock primary reviewers’ comments in which the level and sources of risks and weaknesses were manipulated. Here we show that proposal risks more strongly predicted reviewers’ scores than proposal strengths based on mock proposal evaluations. Risk tolerance was not predictive of scores but reviewer scoring leniency was predictive of overall and criteria scores. The evaluation of risks dominates reviewers’ evaluation of research proposals and is a source of inter-reviewer variability. These results suggest that reviewer scoring variability may be attributed to the interpretation of proposal risks, and could benefit from intervention to improve the reliability of reviews. Additionally, the valuation of risk drives proposal evaluations and may reduce the chances that risky, but highly impactful science, is supported.

Computer extracted gland features from H&E predicts prostate cancer recurrence comparably to a genomic companion diagnostic test: a large multi-site study

Existing tools for post-radical prostatectomy (RP) prostate cancer biochemical recurrence (BCR) prognosis rely on human pathologist-derived parameters such as tumor grade, with the resulting inter-reviewer variability. Genomic companion diagnostic tests such as Decipher tend to be tissue destructive, expensive, and not routinely available in most centers. We present a tissue non-destructive method for automated BCR prognosis, termed "Histotyping", that employs computational image analysis of morphologic patterns of prostate tissue from a single, routinely acquired hematoxylin and eosin slide. Patients from two institutions (n = 214) were used to train Histotyping for identifying high-risk patients based on six features of glandular morphology extracted from RP specimens. Histotyping was validated for post-RP BCR prognosis on a separate set of n = 675 patients from five institutions and compared against Decipher on n = 167 patients. Histotyping was prognostic of BCR in the validation set (p < 0.001, univariable hazard ratio [HR] = 2.83, 95% confidence interval [CI]: 2.03–3.93, concordance index [c-index] = 0.68, median years-to-BCR: 1.7). Histotyping was also prognostic in clinically stratified subsets, such as patients with Gleason grade group 3 (HR = 4.09) and negative surgical margins (HR = 3.26). Histotyping was prognostic independent of grade group, margin status, pathological stage, and preoperative prostate-specific antigen (PSA) (multivariable p < 0.001, HR = 2.09, 95% CI: 1.40–3.10, n = 648). The combination of Histotyping, grade group, and preoperative PSA outperformed Decipher (c-index = 0.75 vs. 0.70, n = 167). These results suggest that a prognostic classifier for prostate cancer based on digital images could serve as an alternative or complement to molecular-based companion diagnostic tests.

Inter-reviewer Variability in Interpretation of pH-Impedance Studies: The Wingate Consensus

Ambulatory pH-impedance monitoring evaluates gastroesophageal reflux disease (GERD) symptoms that persist despite empiric treatment trials.1 Although automated analysis of pH-impedance monitoring is known to overestimate non-acid reflux episodes,2 even experts may disagree on individual reflux episodes,3,4 and accurate identification of postreflux swallow-induced peristaltic wave (PSPW). We hypothesized that a formal consensus meeting between expert pH-impedance reviewers would establish definitive criteria for identification of reflux episodes and PSPW, and improve inter-reviewer variability.

Addressing the quality of submissions to ClinicalTrials.gov for registration and results posting: The use of a checklist.

US Federal regulations since the late 1990s have required registration of some clinical trials and submission of results for some of these trials on a public registry, ClinicalTrials.gov. The quality of the submissions made to ClinicalTrials.gov determines the duration of the Quality Control review, whether the submission will pass the review (success), and how many review cycles it will take for a study to be posted. Success rate for all results submitted to ClinicalTrials.gov is less than 25%. To increase the success of investigators' submissions and meet the requirements of registration and submission of results in a timely fashion, the Johns Hopkins ClinicalTrials.gov Program implemented a policy to review all studies for quality before submission. To standardize our review for quality, minimize inter-reviewer variability, and have a tool for training new staff, we developed a checklist. The Program staff learned from major comments received from ClinicalTrials.gov and also reviewed the Protocol Registration and Results System review criteria for registration and results to fully understand how to prepare studies to pass Quality Control review. These were summarized into bulleted points and incorporated into a checklist used by Program staff to review studies before submission. In the period before the introduction of the checklist, 107 studies were submitted for registration with a 45% (48/107) success rate, a mean (SD) of 18.9 (26.72) days in review, and 1.74 (0.78) submission cycles. Results for 44 records were submitted with 11% (5/44) success rate, 115.80 (129.33) days in review, and 2.23 (0.68) submission cycles. In the period after the checklist, 104 studies were submitted for registration with 80% (83/104) success rate, 2.12 (3.85) days in review, and 1.22 (0.46) submission cycles. Results for 22 records were submitted with 41% (9/22) success rate, 39.27 (19.84) days in review, and 1.64 (0.58) submission cycles. Of the 44 results submitted prior to the checklist, 30 were Applicable or Probable Applicable Clinical Trials, with 10% (3/30) being posted within 30 days as required of the National Institutes of Health. For the 22 results submitted after the checklist, 17 were Applicable or Probable Applicable Clinical Trials, with 47% (8/17) being posted within 30 days of submission. These pre- and post-checklist differences were statistically significant improvements. The checklist has substantially improved our success rate and contributed to a reduction in the review days and number of review cycles. If Academic Medical Centers and industry will adopt or create a similar checklist to review their studies before submission, the quality of the submissions can be improved and the duration of review can be minimized.

Standard operating procedure for somatic variant refinement of sequencing data with paired tumor and normal samples

PurposeFollowing automated variant calling, manual review of aligned read sequences is required to identify a high-quality list of somatic variants. Despite widespread use in analyzing sequence data, methods to standardize manual review have not been described, resulting in high inter- and intralab variability.MethodsThis manual review standard operating procedure (SOP) consists of methods to annotate variants with four different calls and 19 tags. The calls indicate a reviewer’s confidence in each variant and the tags indicate commonly observed sequencing patterns and artifacts that inform the manual review call. Four individuals were asked to classify variants prior to, and after, reading the SOP and accuracy was assessed by comparing reviewer calls with orthogonal validation sequencing.ResultsAfter reading the SOP, average accuracy in somatic variant identification increased by 16.7% (p value = 0.0298) and average interreviewer agreement increased by 12.7% (p value < 0.001). Manual review conducted after reading the SOP did not significantly increase reviewer time.ConclusionThis SOP supports and enhances manual somatic variant detection by improving reviewer accuracy while reducing the interreviewer variability for variant calling and annotation.

Correlation of Technetium-99m Macroaggregated Albumin and Yttrium-90 Glass Microsphere Biodistribution in Hepatocellular Carcinoma: A Retrospective Review of Pretreatment Single Photon Emission CT and Posttreatment Positron Emission Tomography/CT

PurposeTo evaluate whether technetium-99 (99mTc)-labeled macroaggregated albumin (MAA) can predict subsequent yttrium-90 (90Y) distribution and imaging response in patients with hepatocellular carcinoma (HCC). MaterialsRetrospective review was performed of records of 83 patients with HCC who underwent 90Y glass microsphere radioembolization with 99mTc-MAA single photon emission computed tomography (SPECT) and 90Y positron emission tomography (PET)/CT between January 2013 and December 2014. Images were fused to segment the whole liver normal tissue (WLNT) and the largest tumors. Fused images were reviewed and analyzed for comparison of absorbed dose (AD) to tumors and WLNT as calculated from 99mTc-MAA SPECT and from 90Y PET/CT, subjective imaging comparison of 99mTc-MAA SPECT and 90Y PET for tumors and WLNT, and correlation of tumoral AD with response on follow-up CT. ResultsFinal analysis included 73 and 63 patients for WLNT and tumor 99mTc-MAA/90Y correlation, respectively, and 62 patients for AD vs response. 99mTc-MAA/90Y limit of agreement for each reviewer was viewed as clinically acceptable only for WLNT (–15 to 15 Gy). AD interreviewer variability was clinically acceptable for WLNT but was too broad for tumor. Mean tumor AD for objective response (78%) was 313 Gy vs 234 Gy for nonresponders. No threshold was found between tumor AD and response (P > .1). Catheter mismatch between 99mTc-MAA and 90Y had a direct impact on AD mismatch between the 2 image sets. Conclusions99mTc-MAA was found to be a poor surrogate to quantitatively predict subsequent 90Y AD to hepatocellular tumors. 99mTc-MAA distribution correlated with 90Y distribution in the normal hepatic parenchyma.

Evaluation of a 5-Tier Scheme Proposed for Classification of Sequence Variants Using Bioinformatic and Splicing Assay Data: Inter-Reviewer Variability and Promotion of Minimum Reporting Guidelines

Splicing assays are commonly undertaken in the clinical setting to assess the clinical relevance of sequence variants in disease predisposition genes. A 5-tier classification system incorporating both bioinformatic and splicing assay information was previously proposed as a method to provide consistent clinical classification of such variants. Members of the ENIGMA Consortium Splicing Working Group undertook a study to assess the applicability of the scheme to published assay results, and the consistency of classifications across multiple reviewers. Splicing assay data were identified for 235 BRCA1 and 176 BRCA2 unique variants, from 77 publications. At least six independent reviewers from research and/or clinical settings comprehensively examined splicing assay methods and data reported for 22 variant assays of 21 variants in four publications, and classified the variants using the 5-tier classification scheme. Inconsistencies in variant classification occurred between reviewers for 17 of the variant assays. These could be attributed to a combination of ambiguity in presentation of the classification criteria, differences in interpretation of the data provided, nonstandardized reporting of results, and the lack of quantitative data for the aberrant transcripts. We propose suggestions for minimum reporting guidelines for splicing assays, and improvements to the 5-tier splicing classification system to allow future evaluation of its performance as a clinical tool.

Variations in risk assessment models may contribute to the existing gap between venous thromboembolism prophylaxis guidelines and adherence

BackgroundRisk assessment models (RAMs) may allow the clinician to determine need for deep vein thrombosis (DVT) prophylaxis. Individual healthcare facilities often develop their own RAMs. The purpose of this study was to determine: 1.) inter-RAM variability in DVT risk factors and contraindications; 2.) inter-rater variability and inter-RAM variability when applying a RAM to a standard case; and 3.) inter-rater and inter-RAM variability in outcome as far as type of prophylaxis. A convenience sample of RAMs was obtained from various institutions and ten reviewers were recruited to apply the RAMs to three patient cases.FindingThe review resulted in 390 separate assessments. Patient 1 did not receive any chemoprophylaxis in 67% of the evaluations, patient 2 in 27% of the evaluations and patient 3 in 2.3% of the evaluations. There was statistically significant variation in the provision of chemoprophylaxis per RAM for patient 1 (p=0.001) and no significant variation for patients 2 and 3. When analyzing the rate of chemoprophylaxis per reviewer, there was statistically significant variation for patients 1 and 2 (p=0.026 and <0.0001 respectively) but not for patient 3 (p=0.123).ConclusionThere may be significant inter-RAM and inter-reviewer variability when utilizing RAMs for assessing DVT risk.

High inter-reviewer variability of spike detection on intracranial EEG addressed by an automated multi-channel algorithm

ObjectiveThe goal of this study was to determine the consistency of human reviewer spike detection and then develop a computer algorithm to make the intracranial spike detection process more objective and reliable. MethodsThree human reviewers marked interictal spikes on samples of intracranial EEGs from 10 patients. The sensitivity, precision and agreement in channel ranking by activity were calculated between reviewers. A computer algorithm was developed to parallel the way human reviewers detect spikes by first identifying all potential spikes on each channel using frequency filtering and then block scaling all channels at the same time in order to exclude potential spikes that fall below an amplitude and slope threshold. Its performance was compared to the human reviewers on the same set of patients. ResultsHuman reviewers showed surprisingly poor inter-reviewer agreement, but did broadly agree on the ranking of channels for spike activity. The computer algorithm performed as well as the human reviewers and did especially well at ranking channels from highest to lowest spike frequency. ConclusionsOur algorithm showed good agreement with the different human reviewers, even though they demonstrated different criteria for what constitutes a ‘spike’ and performed especially well at the clinically important task of ranking channels by spike activity. SignificanceAn automated, objective method to detect interictal spikes on intracranial recordings will improve both research and the surgical management of epilepsy patients.

Panel discussion does not improve reliability of peer review for medical research grant proposals

ObjectivePeer review is the gold standard for evaluating scientific quality. Compared with studies on inter-reviewer variability, research on panel evaluation is scarce. To appraise the reliability of panel evaluations in grant review, we compared scores by two expert panels reviewing the same grant proposals. Our main interest was to evaluate whether panel discussion improves reliability. MethodsThirty reviewers were randomly allocated to one of the two panels. Sixty-five grant proposals in the fields of clinical medicine and epidemiology were reviewed by both panels. All reviewers received 5–12 proposals. Each proposal was evaluated by two reviewers, using a six-point scale. The reliability of reviewer and panel scores was evaluated using Cohen's kappa with linear weighting. In addition, reliability was also evaluated for the panel mean scores (mean of reviewer scores was used as panel score). ResultsThe proportion of large differences (at least two points) was 40% for reviewers in panel A, 36% for reviewers in panel B, 26% for the panel discussion scores, and 14% when the means of the two reviewer scores were used. The kappa for panel score after discussion was 0.23 (95% confidence interval: 0.08, 0.39). By using the mean of the reviewer scores, the panel coefficient was similarly 0.23 (0.00, 0.46). ConclusionThe reliability between panel scores was higher than between reviewer scores. The similar interpanel reliability, when using the final panel score or the mean value of reviewer scores, indicates that panel discussions per se did not improve the reliability of the evaluation.

P32.14: Quality control of standardized 3D acquisition for the examination of the fetal head and abdomen

The purpose of this study was to was to develop and evaluate the feasibility and reproducibility of criteria-based quality control for routine standardized 3d volume acquisition of fetal head and volume. 3D ultrasound was performed in a standardized manner (description) in 30 fetuses at 20–24 weeks at the time of routine mid-trimester scan. A standardized volume acquisition of the fetal head and abdomen were realized by two untrained operators. For the acquisition of fetal head volumes used, the starting plane was defined at the level of the biparital diameter. The scan angle was 50°. For the acquisition of fetal abdomen volumes, the starting plane used was slightly lower than the plane of the abdominal circumference measurement, at the level of the 2 pyelic cavities and the insertion of the umbilical cord. These were then analyzed by two expert reviewers, according to predefined criteria agreed upon on the basis of established standards. Each criteria was quoted 1 if present, yelding to a score of 5 and 7 for fetal head and abdomen volumes respectively. Inter reviewer variability in criteria assessment and mean score were evaluated. Among the 60 volumes, criteria based evaluation demonstrated good adjusted kappa values (> 0.8 in most cases), and high intra-class correlation coefficient. Among the study period, there was a clear trend in an increase of mean score for each operator (P < 0.05). Standardized 3D volumes quality control policy based on scoring is feasible and allows for good inter- and intra-reviewer reproducibility. Besides its potential for audit and quality control, this could also be useful during the training process.

Role of Sagittal Reformatted Computed Tomographic Images in the Evaluation of Orbital Floor Fractures

The sagittal plane computed tomographic (CT) scan has been proposed as the most important radiologic view in the diagnosis of orbital floor fractures. Before the advent of high-resolution CT, several articles were published in the radiology literature debating the need for direct sagittal views, which required cumbersome patient positioning. Today, the ability to easily create high-quality sagittal reformations requires reassessment of the importance of this view in the evaluation of orbital floor fractures. Computed tomographic studies of 24 orbital floor fractures for which coronal and sagittal images were available were included. Five independent reviewers evaluated all image sets and recorded maximum fracture width, depth, posterior shelf length (PSL), and presence of medial wall fracture and of trap door deformity. Direct fracture width measurements were obtained via coronal images, whereas depth and PSL were measured directly in the sagittal plane. Indirect measurements in a given plane were obtained by counting the number of slices in which the fracture was present and multiplying by the slice thickness. Cronbach alpha analysis was used to provide an intraclass correlation coefficient, where greater values signify less interreviewer variability. The Cronbach alpha values for width, depth, and PSL were 64%, 59%, and 85% in the coronal view and 76%, 55%, and 51% in the sagittal view, respectively. The alpha values for presence/absence of medial wall fracture and trap door deformity were greater in the coronal view at 73% and 55%, respectively. The intrareviewer variability was not significantly correlated with the degree of patient rotation or CT scan slice thickness. The interreviewer variability was significantly less among attending reviewer then resident reviewers. There was less interreviewer variability when the measured variables were assessed indirectly (width was best assessed in the sagittal plane, and depth and PSL were best assessed in the coronal view). These findings support the idea that additional views aid the surgeon's ability to further define fracture anatomy than might be possible with any single view, thereby improving preoperative planning and reconstructive strategies.

Comparison of whole liver and small region-of-interest measurements of MRI liver R2* in children with iron overload

Measurement of liver MRI T2* and R2* is emerging as a reliable alternative to liver biopsy for the quantitation of liver iron content. A systematic investigation of the influence of the region-of-interest size and placement has not been conducted. To compare small and whole liver region-of-interest (ROI) MRI R2* values to each other and to biopsy liver iron content in patients with iron overload. Forty-one iron-overloaded patients, ages 7-35 years, underwent biopsy for liver iron content quantitation and MRI for liver R2* measurement within 30 days. Three reviewers independently used small and whole liver ROIs to measure R2*. Inter-reviewer agreement was assessed with the intra-class correlation coefficient (ICC). Associations between R2* and liver iron content were investigated using Spearman's rank-order correlation and Monte Carlo estimated exact P values. Biopsy liver iron content and small and whole liver ROI R2* measurements were strongly associated for all reviewers (all P < 0.0001). Although inter-reviewer agreement was excellent for both ROI methods (ICC = 0.98-0.99), the small ROI technique more frequently led to inter-reviewer differences larger than 75 Hz, slightly higher R2* values, larger standard errors and greater range in values. Small and whole liver ROI techniques are strongly associated with biopsy liver iron content. We found slightly greater inter-reviewer variability in R2* values using the small ROI technique. Because such variability could adversely impact patient management when R2* values are near a threshold of iron chelation therapy, we recommend using a whole liver ROI.

Incidence of Capsular Contracture in Silicone Versus Saline Cosmetic Augmentation Mammoplasty: A Meta-Analysis

Capsular contracture after augmentation mammoplasty occurs at a rate of 15% to 45%. The purpose of the present study was to determine the effect of implant core type (silicone versus saline) on the rate of capsular contracture in augmentation mammoplasty. A systematic review was conducted through a search of three electronic databases. Two reviewers independently scanned titles yielded by the search and identified potentially relevant papers. Inter-reviewer variability and the scientific quality of the articles were assessed. Meta-analysis was performed. Eighty-eight titles of potential relevance were selected from the 393 articles yielded by the search. Inter-rater agreement for selection of potentially relevant articles was 84% (kappa=0.54). Four comparative studies were included in the analysis. Scientific quality scores of the included studies ranged from 5 of 14 to 9 of 14. Three of the four studies reported a higher rate of capsular contracture in patients with silicone implants. A combined odds ratio calculated on two of the studies found a 2.25-fold increased risk of capsular contracture in patients who received silicone implants. Eight series of patients who received cohesive gel silicone implants reported rates of capsular contracture from 0% to 13.6%. Higher rates of capsular contracture were found in patients who received silicone implants when compared with those who received saline implants. However, the scientific quality of the comparative studies to date on this subject is poor. Recent series evaluating cohesive gel implants report relatively low rates of capsular contracture. A randomized controlled trial comparing rates of capsular contracture in cohesive gel and saline implants is recommended.

Evaluation of Data Obtained from Military Disability Medical Administrative Databases for Service Members with Schizophrenia or Bipolar Disorder

We are studying associations between selected biomarkers and schizophrenia or bipolar disorder among military personnel. To assess potential diagnostic misclassification and to estimate the date of illness onset, we reviewed medical records for a subset of cases. Two psychiatrists independently reviewed 182 service medical records retrieved from the Department of Veterans Affairs. Data were evaluated for diagnostic concordance between database diagnoses and reviewers. Interreviewer variability was measured by using proportion of agreement and the kappa statistic. Data were abstracted to estimate date of onset. High levels of agreement existed between database diagnoses and reviewers (proportion, 94.7%; kappa = 0.88) and between reviewers (proportion, 92.3%; kappa = 0.87). The median time between illness onset and initiation of medical discharge was 1.6 and 1.1 years for schizophrenia and bipolar disorder, respectively. High levels of agreement between investigators and database diagnoses indicate that diagnostic misclassification is unlikely. Discharge procedure initiation date provides a suitable surrogate for disease onset.