Gold Standard Set Research Articles

Set-theory based benchmarking of three different variant callers for targeted sequencing

BackgroundNext generation sequencing (NGS) technologies have improved the study of hereditary diseases. Since the evaluation of bioinformatics pipelines is not straightforward, NGS demands effective strategies to analyze data that is of paramount relevance for decision making under a clinical scenario. According to the benchmarking framework of the Global Alliance for Genomics and Health (GA4GH), we implemented a new simple and user-friendly set-theory based method to assess variant callers using a gold standard variant set and high confidence regions. As model, we used TruSight Cardio kit sequencing data of the reference genome NA12878. This targeted sequencing kit is used to identify variants in key genes related to Inherited Cardiac Conditions (ICCs), a group of cardiovascular diseases with high rates of morbidity and mortality.ResultsWe implemented and compared three variant calling pipelines (Isaac, Freebayes, and VarScan). Performance metrics using our set-theory approach showed high-resolution pipelines and revealed: (1) a perfect recall of 1.000 for all three pipelines, (2) very high precision values, i.e. 0.987 for Freebayes, 0.928 for VarScan, and 1.000 for Isaac, when compared with the reference material, and (3) a ROC curve analysis with AUC > 0.94 for all cases. Moreover, significant differences were obtained between the three pipelines. In general, results indicate that the three pipelines were able to recognize the expected variants in the gold standard data set.ConclusionsOur set-theory approach to calculate metrics was able to identify the expected ICCs related variants by the three selected pipelines, but results were completely dependent on the algorithms. We emphasize the importance to assess pipelines using gold standard materials to achieve the most reliable results for clinical application.

OP441 Testing The Sensitivity And Precision Of The Cochrane MEDLINE Randomized Controlled Trial Search Filters

IntroductionThe Cochrane Handbook of Systematic Reviews contains two search filters to find randomized controlled trials (RCT) in Ovid MEDLINE: a sensitivity maximizing RCT filter and a sensitivity and precision maximizing RCT filter. The RCT search strategies were originally published in 1994 have been adapted and updated, most recently in 2008. To determine whether the Cochrane filters are still performing adequately to inform Cochrane reviews, we tested the performance of the Cochrane filters and 36 other MEDLINE filters in a large new gold standard set of relevant records.MethodsWe identified a gold standard set of RCT reports published in 2016 from the Cochrane CENTRAL database of controlled clinical trials. We retrieved the records in Ovid MEDLINE using their PubMed identifiers. Each RCT filter was run in MEDLINE and combined with the gold standard set of records, to determine their sensitivity, precision and f-scores.ResultsThe gold standard comprised 27,617 records and the searches were run on 16 July 2019. The most sensitive RCT filter was Duggan (sensitivity 0.99). The Cochrane sensitivity maximizing RCT filter had a sensitivity of 0.96, but was more precise than Duggan (0.14 compared to 0.04 for Duggan). The most precise RCT filter was Chow, Glanville/Lefebvre, Royle/Waugh, Dumbrique (precision 0.97, sensitivity 0.83). The best precision Cochrane filter was the sensitivity and precision maximising RCT filter.ConclusionsThe Cochrane MEDLINE sensitivity maximizing RCT filter can continue to be used by Cochrane reviewers and CENTRAL compilers as it has very high sensitivity but a more acceptable precision than many higher sensitivity filters. Slightly more sensitive filters are available, but with lower precision than the Cochrane sensitivity maximizing RCT filter. These other filters may be preferred when combining with a subject search when record numbers may be more manageable than searching the whole of MEDLINE.

Performance of deep learning to detect mastoiditis using multiple conventional radiographs of mastoid.

This study aimed to compare the diagnostic performance of deep learning algorithm trained by single view (anterior-posterior (AP) or lateral view) with that trained by multiple views (both views together) in diagnosis of mastoiditis on mastoid series and compare the diagnostic performance between the algorithm and radiologists. Total 9,988 mastoid series (AP and lateral views) were classified as normal or abnormal (mastoiditis) based on radiographic findings. Among them 792 image sets with temporal bone CT were classified as the gold standard test set and remaining sets were randomly divided into training (n = 8,276) and validation (n = 920) sets by 9:1 for developing a deep learning algorithm. Temporal (n = 294) and geographic (n = 308) external test sets were also collected. Diagnostic performance of deep learning algorithm trained by single view was compared with that trained by multiple views. Diagnostic performance of the algorithm and two radiologists was assessed. Inter-observer agreement between the algorithm and radiologists and between two radiologists was calculated. Area under the receiver operating characteristic curves of algorithm using multiple views (0.971, 0.978, and 0.965 for gold standard, temporal, and geographic external test sets, respectively) showed higher values than those using single view (0.964/0.953, 0.952/0.961, and 0.961/0.942 for AP view/lateral view of gold standard, temporal external, and geographic external test sets, respectively) in all test sets. The algorithm showed statistically significant higher specificity compared with radiologists (p = 0.018 and 0.012). There was substantial agreement between the algorithm and two radiologists and between two radiologists (κ = 0.79, 0.8, and 0.76). The deep learning algorithm trained by multiple views showed better performance than that trained by single view. The diagnostic performance of the algorithm for detecting mastoiditis on mastoid series was similar to or higher than that of radiologists.

Which are the most sensitive search filters to identify randomized controlled trials in MEDLINE?

Objective:The Cochrane Handbook of Systematic Reviews contains search filters to find randomized controlled trials (RCTs) in Ovid MEDLINE: one maximizing sensitivity and another balancing sensitivity and precision. These filters were originally published in 1994 and were adapted and updated in 2008. To determine the performance of these filters, the authors tested them and thirty-six other MEDLINE filters against a large new gold standard set of relevant records.Methods:We identified a gold standard set of RCT reports published in 2016 from the Cochrane CENTRAL database of controlled clinical trials. We retrieved the records in Ovid MEDLINE and combined these with each RCT filter. We calculated their sensitivity, relative precision, and f-scores.Results:The gold standard comprised 27,617 records. MEDLINE searches were run on July 16, 2019. The most sensitive RCT filter was Duggan et al. (sensitivity=0.99). The Cochrane sensitivity-maximizing RCT filter had a sensitivity of 0.96 but was more precise than Duggan et al. (0.14 compared to 0.04 for Duggan). The most precise RCT filters had 0.97 relative precision and 0.83 sensitivity.Conclusions:The Cochrane Ovid MEDLINE sensitivity-maximizing RCT filter can continue to be used by Cochrane reviewers and to populate CENTRAL, as it has very high sensitivity and a slightly better precision relative to more sensitive filters. The results of this study, which used a very large gold standard to compare the performance of all known RCT filters, allows searchers to make better informed decisions about which filters to use for their work.

Relationship Between Sensory Symptoms, Mulder's Sign, and Dynamic Ultrasonographic Findings in Morton's Neuroma.

Morton's neuroma (MN) is often a diagnostic dilemma lacking a gold standard set of diagnostic criteria. Advanced imaging of MN is evolving including ultrasonography. The current study aimed to analyze the relationship between ultrasonographic findings and symptoms, clinical data, and operative findings in a subgroup of patients. We evaluated physical examination, ultrasonographic findings, symptoms, and in a subgroup, the operative findings for Morton's neuroma. We analyzed the symptoms, the findings on physical examination, and ultrasonography data and performed a statistical correlation between them. A total of 175 patients were seen for suspected Morton's neuroma during the last 7 years. Neuropathic pain of the toes was reported in 63% of patients. Presence of ultrasonographic findings suggesting Morton's neuroma was observed in 77% of cases. A mild significant relationship between neuropathic pain and positive ultrasonographic findings was observed. We found a strong correlation between ultrasonographic and clinical evaluation, but more than half with clinical negative Morton's neuroma had positive findings at ultrasonography. In the subgroup of operatively treated patients (n = 44) surgery confirmed Morton's neuroma in all patients who had positive ultrasonography findings. The current study suggests that the association of ultrasonographic evaluation and clinical evaluation can be very useful for the management of Morton's neuroma. Our study could help with the development of a multiperspective approach in the diagnosis of Morton's neuroma. Level II, prospective cohort survey study.

Abstract 2118: Using electronic medical records to assemble a lung cancer cohort for prognosis study

Abstract Background: Electronic medical records (EMRs) provide a low-cost means of accessing longitudinal data on large populations with detailed information regarding diagnosis, clinical procedures, medications and tests. A lung cancer cohort assembled from EMR represents a powerful resource for studying prognosis. Methods: EMRs are from the Massachusetts General Hospital and Brigham and Women's Hospital using Partners HealthCare System Research Patient Data Registry from 1988 to October 18, 2018. A previously validated phenotyping algorithm was developed to identify lung cancer patients based on a gold standard set of 200 patients. Demographics and clinical characteristics were extracted from both structured data and clinical notes utilizing natural language processing tools. The quality of EMR cohort was assessed by comparing to Boston Lung Cancer Study (BLCS) cohort using overlapped population. Specifically, we assessed: 1) absolute differences: the agreements of diagnosis dates, histological type and clinical stage; 2) relative differences: the effect sizes of variables on overall survival estimated from Cox regression models. Results: The initial study population included 76,643 patients with a diagnostic code related to primary lung cancer. The phenotyping model identified 42,069 lung cancer patients with sensitivity of 75.2%, specificity of 90.0%, PPV of 94.4% and AUC of 0.927. A total of 5,053 patients overlapped in BLCS and EMR cohorts were used for quality assessment. The diagnosis dates from EMR agreed with diagnosis dates from BLCS cohort with median of 0 days. 10.9%, 7.5%, and 5.1% of the population had an absolute date discrepancy of more than 90 days, 180 days, and one year, respectively. The agreements of histological type and clinical stage were 90.1% and 82.8% respectively. Further analysis for overall survival showed high consistency in the two cohorts as Cox regression models controlled for age, sex, race, smoking status, histological type, stages and treatments yielded similar estimates. Conclusion: We assembled a large lung cancer cohort from EMRs using phenotyping algorithm and extraction strategies combining structured and unstructured data. The quality of analytic data from EMRs were compared with the well curated epidemiology study to ensure their suitability for lung cancer prognosis research. Citation Format: Qianyu Yuan, Tianrun Cai, Chuan Hong, Tianxi Cai, David Christiani. Using electronic medical records to assemble a lung cancer cohort for prognosis study [abstract]. In: Proceedings of the Annual Meeting of the American Association for Cancer Research 2020; 2020 Apr 27-28 and Jun 22-24. Philadelphia (PA): AACR; Cancer Res 2020;80(16 Suppl):Abstract nr 2118.

AB0556 PRIMARY SYSTEMIC SCLEROSIS HEART INVOLVEMENT (PSSCHI): A SYSTEMATIC LITERATURE REVIEW (SLR), CONSENSUS-BASED DEFINITION AND PRELIMINARY VALIDATION.

Background:pSScHI may cause tissue, functional and conduction abnormalities with varied clinical manifestations. The absence of a clear definition of pSScHI impairs the significance and ability of focussed research, frequently not allowing the distinction between primary and secondary involvement.Objectives:We aimed to establish an expert consensus definition for pSScHI, to be used in clinical trials and everyday clinical practice, and to start its validation process.Methods:A SLR for cardiac manifestations and alterations in SSc was conducted using PubMed, Web of Science and Embase. Articles published from inception to December 31st, 2018 were identified. Inclusion criteria included papers in English on adult SSc patients, with heart involvement as outcome. We excluded non-human studies, secondary heart involvement (eg PAH, drugs, infections), reviews and case reports. PRISMA recommendations were followed where applicable. Extracted data were categorized into relevant domains (signs, symptoms, anatomical site involved, physiological abnormalities, pathological changes, prognostic outcomes), which informed the consensus definition. Sixteen senior experts (7 rheumatologists, 8 cardiologists, 1 pathologist) discussed the data and, using a nominal group technique, added expert opinion, provided statements to consider and ranked them. Consensus was attained for agreement >70%. Sixteen clinical cases were evaluated in two rounds to test for face validity, feasibility, inter- and intra-rater reliability and criterion validity (gold standard set by agreed evaluation between expert rheumatologist, cardiologist and methodologist).Results:2593 publications were identified and screened, 251 full texts were evaluated,172 met eligibility criteria. Data from the 7 domains were extracted and used to develop the World Scleroderma Foundation – Heart Failure Association (WSF-HFA) consensus-derived definition of pSSc-HI, as follows:“pSScHI comprises cardiac abnormalities that are predominantly attributable to SSc rather than other causes and/or complications*. pSScHI may be sub-clinical and must be confirmed through diagnostic investigation. The pathogenesis of pSScHI comprises one or more of inflammation, fibrosis and vasculopathy. *Non SSc-specific cardiac conditions (e.g. Ischaemic heart disease, arterial hypertension, drug toxicity, other cardiomyopathy, primary valvular disease) and/or SSc non cardiac conditions (e.g. PAH, Renal involvement, ILD).”Face validity was determined by a 100% agreement on credibility; application was feasible, with a median 60 (5-600) seconds taken per case; inter rater agreement was moderate [mKappa (95%CI) 0.56 (0.46-1.00) and 0.55 (0.44-1.00) for the two rounds] and intra rater agreement was good [mKappa (95%CI) 0.77 (0.47-1,00)]. Content validity was reached based on the wide variety of patients in the SLR, criterion validity was reached with 78 (73-84) % correctness.Conclusion:Using a SLR and modified nominal technique, we have developed a preliminary pSScHI consensus-based definition and started a validation process for it to be used in clinical research and clinical practice.Acknowledgments:Aleksandra Djokovic, Giacomo De Luca, Raluca B. Dumitru,Alessandro Giollo, Marija Polovina, Yossra Atef Suliman, Kostantinos Bratis, Alexia Steelandt, Ivan Milinkovic, Anna Baritussio, Ghadeer Hasan, Anastasia Xintarakou, Yohei Isomura, George Markousis-Mavrogenis, Silvia Bellando-Randone, Lorenzo Tofani, Sophie Mavrogeni, Luna Gargani, Alida L.P. Caforio, Carsten Tschoepe, Arsen Ristic, Karin Klingel, Sven Plein, Elijah Behr, Yannick Allanore, Masataka Kuwana, Christopher Denton, Daniel E. Furst, Dinesh Khanna, Thomas Krieg, Renzo Marcolongo.Disclosure of Interests:Cosimo Bruni Speakers bureau: Actelion, Eli Lilly, Maya H Buch Grant/research support from: Pfizer, Roche, and UCB, Consultant of: Pfizer; AbbVie; Eli Lilly; Gilead Sciences, Inc.; Merck-Serono; Sandoz; and Sanofi, Petar Seferovic: None declared, Marco Matucci-Cerinic Grant/research support from: Actelion, MSD, Bristol-Myers Squibb, Speakers bureau: Acetelion, Lilly, Boehringer Ingelheim

Dose planning variations related to delineation variations in MRI-guided brachytherapy for locally advanced cervical cancer

PurposeTo examine the variability in prescribed dose due to contouring variations in intracavitary image-guided adaptive brachytherapy for cervical cancer. To identify correlations between dosimetric outcomes and delineation uncertainty metrics. Methods and MaterialsA data set from an EMBRACE sub-study on contouring uncertainties was used, consisting of magnetic resonance images of six patients with cervical cancer delineated by 10 experienced observers (target volumes and organs at risk). Two gold standard contours were generated, an expert consensus and the simultaneous truth and performance level estimation. Plans were individually optimised to all of the contour sets (12 in total). Plans were applied to the gold standard contour sets, and dose volume histogram parameters including D90, D98 and D2cm3 were determined. The variability between plans was assessed. Dose volume histogram parameters and delineation uncertainty metrics were correlated using the Spearman's non-parametric rank correlation. ResultsThere is a dosimetric variability between observers, patients and the gold standard contour used for analysis. Approximately 3 Gy D90 EQD210 variability (SD) was observed for the CTVHR and 1.2-3.6 Gy D2cm3 EQD23 for the organs at risk. The maximum geometric dimensions of the delineations are most commonly correlated with dosimetry changes. Although the correlations are similar across gold standards, the direction of these correlations differs, indicating that the dosimetric outcomes are dependent on the contour that the plan is optimised to. ConclusionThis study highlights the dosimetric differences interobserver uncertainty in contouring can have for cervical cancer brachytherapy. The importance of carefully choosing a gold standard from which to benchmark is reiterated.

Integrated Care Search: development and validation of a PubMed search filter for retrieving the integrated care research evidence

BackgroundIntegrated care is an increasingly important principle for organising healthcare. Integrated care models show promise in reducing resource wastage and service fragmentation whilst improving the accessibility, patient-centredness and quality of care for patients. Those needing reliable access to the growing research evidence base for integrated care can be frustrated by search challenges reflective of the topic’s complexity. The aim of this study is to report the empirical development and validation of two search filters for rapid and effective retrieval of integrated care evidence in PubMed. One filter is optimised for recall and the other for precision.MethodsAn Expert Advisory Group comprising international integrated care experts guided the study. A gold standard test set of citations was formed from screening Handbook Integrated Care chapter references for relevance. This set was divided into a Term Identification Set (20%) for determining candidate terms using frequency analysis; a Filter Development Set (40%) for testing performance of term combinations; and a Filter Validation Set (40%) reserved for confirming final filter performance. In developing the high recall filter, recall was steadily increased while maintaining precision at ≥50%. Similarly, the high precision filter sought to maximise precision while keeping recall ≥50%. For each term combination tested, an approximation of precision was obtained by reviewing the first 100 citations retrieved in Medline for relevance.ResultsThe gold standard set comprised 534 citations. The search filter optimised for recall (‘Broad Integrated Care Search’) achieved 86.0–88.3% recall with corresponding low precision (47–53%). The search filter optimised for precise searching (‘Narrow Integrated Care Search’) demonstrated precision of 73–95% with recall reduced to between 55.9 and 59.8%. These filters are now available as one-click URL hyperlinks in the website of International Foundation for Integrated Care.ConclusionsThe Broad and Narrow Integrated Care Search filters provide potential users, such as policy makers and researchers, seamless, reliable and ongoing access to integrated care evidence for decision making. These filters were developed according to a rigorous and transparent methodology designed to circumvent the challenges of information retrieval posed by this complex, multifaceted topic.

Prediction of Protein-Ligand Interaction Based on the Positional Similarity Scores Derived from Amino Acid Sequences.

The affinity of different drug-like ligands to multiple protein targets reflects general chemical–biological interactions. Computational methods estimating such interactions analyze the available information about the structure of the targets, ligands, or both. Prediction of protein–ligand interactions based on pairwise sequence alignment provides reasonable accuracy if the ligands’ specificity well coincides with the phylogenic taxonomy of the proteins. Methods using multiple alignment require an accurate match of functionally significant residues. Such conditions may not be met in the case of diverged protein families. To overcome these limitations, we propose an approach based on the analysis of local sequence similarity within the set of analyzed proteins. The positional scores, calculated by sequence fragment comparisons, are used as input data for the Bayesian classifier. Our approach provides a prediction accuracy comparable or exceeding those of other methods. It was demonstrated on the popular Gold Standard test sets, presenting different sequence heterogeneity and varying from the group, including different protein families to the more specific groups. A reasonable prediction accuracy was also found for protein kinases, displaying weak relationships between sequence phylogeny and inhibitor specificity. Thus, our method can be applied to the broad area of protein–ligand interactions.

Reproducibility, Specificity and Accuracy of Relative Quantification Using Spectral Library-based Data-independent Acquisition.

Currently data-dependent acquisition (DDA) is the method of choice for mass spectrometry-based proteomics discovery experiments, but data-independent acquisition (DIA) is steadily becoming more important. One of the most important requirements to perform a DIA analysis is the availability of suitable spectral libraries for peptide identification and quantification. Several studies were performed addressing the evaluation of spectral library performance for protein identification in DIA measurements. But so far only few experiments estimate the effect of these libraries on the quantitative level.In this work we created a gold standard spike-in sample set with known contents and ratios of proteins in a complex protein matrix that allowed a detailed comparison of DIA quantification data obtained with different spectral library approaches. We used in-house generated sample-specific spectral libraries created using varying sample preparation approaches and repeated DDA measurement. In addition, two different search engines were tested for protein identification from DDA data and subsequent library generation. In total, eight different spectral libraries were generated, and the quantification results compared with a library free method, as well as a default DDA analysis. Not only the number of identifications on peptide and protein level in the spectral libraries and the corresponding DIA analysis results was inspected, but also the number of expected and identified differentially abundant protein groups and their ratios.We found, that while libraries of prefractionated samples were generally larger, there was no significant increase in DIA identifications compared with repetitive non-fractionated measurements. Furthermore, we show that the accuracy of the quantification is strongly dependent on the applied spectral library and whether the quantification is based on peptide or protein level. Overall, the reproducibility and accuracy of DIA quantification is superior to DDA in all applied approaches.Data has been deposited to the ProteomeXchange repository with identifiers PXD012986, PXD012987, PXD012988 and PXD014956.

Forward search path count as an alternative indirect citation impact indicator

Abstract Main path analysis is a popular method for extracting the backbone of scientific evolution from a (paper) citation network. The first and core step of main path analysis, called search path counting, is to weight citation arcs by the number of scientific influence paths from old to new papers. Search path counting shows high potential in scientific impact evaluation due to its semantic similarity to the meaning of scientific impact indicator, i.e. how many papers are influenced to what extent. In addition, the algorithmic idea of search path counting also resembles many known indirect citation impact indicators. Inspired by the above observations, this paper presents the FSPC (Forward Search Path Count) framework as an alternative scientific impact indicator based on indirect citations. Two critical assumptions are made to ensure the effectiveness of FSPC. First, knowledge decay is introduced to weight scientific influence paths in decreasing order of length. Second, path capping is introduced to mimic human literature search and citing behavior. By experiments on two well-studied datasets against two carefully created gold standard sets of papers, we have demonstrated that FSPC is able to achieve surprisingly good performance in not only recognizing high-impact papers but also identifying undercited papers.

Ensembles of natural language processing systems for portable phenotyping solutions.

Manually curating standardized phenotypic concepts such as Human Phenotype Ontology (HPO) terms from narrative text in electronic health records (EHRs) is time consuming and error prone. Natural language processing (NLP) techniques can facilitate automated phenotype extraction and thus improve the efficiency of curating clinical phenotypes from clinical texts. While individual NLP systems can perform well for a single cohort, an ensemble-based method might shed light on increasing the portability of NLP pipelines across different cohorts. We compared four NLP systems, MetaMapLite, MedLEE, ClinPhen and cTAKES, and four ensemble techniques, including intersection, union, majority-voting and machine learning, for extracting generic phenotypic concepts. We addressed two important research questions regarding automated phenotype recognition. First, we evaluated the performance of different approaches in identifying generic phenotypic concepts. Second, we compared the performance of different methods to identify patient-specific phenotypic concepts. To better quantify the effects caused by concept granularity differences on performance, we developed a novel evaluation metric that considered concept hierarchies and frequencies. Each of the approaches was evaluated on a gold standard set of clinical documents annotated by clinical experts. One dataset containing 1,609 concepts derived from 50 clinical notes from two different institutions was used in both evaluations, and an additional dataset of 608 concepts derived from 50 case report abstracts obtained from PubMed was used for evaluation of identifying generic phenotypic concepts only. For generic phenotypic concept recognition, the top three performers in the NYP/CUIMC dataset are union ensemble (F1, 0.634), training-based ensemble (F1, 0.632), and majority vote-based ensemble (F1, 0.622). In the Mayo dataset, the top three are majority vote-based ensemble (F1, 0.642), cTAKES (F1, 0.615), and MedLEE (F1, 0.559). In the PubMed dataset, the top three are majority vote-based ensemble (F1, 0.719), training-based (F1, 0.696) and MetaMapLite (F1, 0.694). For identifying patient specific phenotypes, the top three performers in the NYP/CUIMC dataset are majority vote-based ensemble (F1, 0.610), MedLEE (F1, 0.609), and training-based ensemble (F1, 0.585). In the Mayo dataset, the top three are majority vote-based ensemble (F1, 0.604), cTAKES (F1, 0.531) and MedLEE (F1, 0.527). Our study demonstrates that ensembles of natural language processing can improve both generic phenotypic concept recognition and patient specific phenotypic concept identification over individual systems. Among the individual NLP systems, each individual system performed best when they were applied in the dataset that they were primary designed for. However, combining multiple NLP systems to create an ensemble can generally improve the performance. Specifically, the ensemble can increase the results reproducibility across different cohorts and tasks, and thus provide a more portable phenotyping solution compared to individual NLP systems.

Using natural language processing to construct a metastatic breast cancer cohort from linked cancer registry and electronic medical records data.

ObjectivesMost population-based cancer databases lack information on metastatic recurrence. Electronic medical records (EMR) and cancer registries contain complementary information on cancer diagnosis, treatment and outcome, yet are rarely used synergistically. To construct a cohort of metastatic breast cancer (MBC) patients, we applied natural language processing techniques within a semisupervised machine learning framework to linked EMR-California Cancer Registry (CCR) data.Materials and MethodsWe studied all female patients treated at Stanford Health Care with an incident breast cancer diagnosis from 2000 to 2014. Our database consisted of structured fields and unstructured free-text clinical notes from EMR, linked to CCR, a component of the Surveillance, Epidemiology and End Results Program (SEER). We identified de novo MBC patients from CCR and extracted information on distant recurrences from patient notes in EMR. Furthermore, we trained a regularized logistic regression model for recurrent MBC classification and evaluated its performance on a gold standard set of 146 patients.ResultsThere were 11 459 breast cancer patients in total and the median follow-up time was 96.3 months. We identified 1886 MBC patients, 512 (27.1%) of whom were de novo MBC patients and 1374 (72.9%) were recurrent MBC patients. Our final MBC classifier achieved an area under the receiver operating characteristic curve (AUC) of 0.917, with sensitivity 0.861, specificity 0.878, and accuracy 0.870.Discussion and ConclusionTo enable population-based research on MBC, we developed a framework for retrospective case detection combining EMR and CCR data. Our classifier achieved good AUC, sensitivity, and specificity without expert-labeled examples.

Predicting protein pairs

Protein Networks Biological function is driven by interaction between proteins. High-throughput experimental techniques have provided large datasets of protein interactions in several organisms; however, much combinatorial space remains uncharted. Cong et al. predict protein interfaces by identifying coevolving residues in aligned protein sequences (see the Perspective by Vajda and Emili). In comparison with gold-standard and negative control sets, they show that the accuracy is higher than for proteome-wide two-hybrid and mass spectrometry screens. The approach predicts 1618 protein interactions in Escherichia coli , 682 of which were unanticipated, and 911 interacting pairs in Mycobacterium tuberculosis , most of which had not been previously described. With an expected false-positive rate of between 10 and 20%, the predicted interactions and networks provide an excellent starting point for further study. Science , this issue p. [185][1]; see also p. [120][2] [1]: /lookup/doi/10.1126/science.aaw6718 [2]: /lookup/doi/10.1126/science.aay1440

CoCoScore: context-aware co-occurrence scoring for text mining applications using distant supervision.

MotivationInformation extraction by mining the scientific literature is key to uncovering relations between biomedical entities. Most existing approaches based on natural language processing extract relations from single sentence-level co-mentions, ignoring co-occurrence statistics over the whole corpus. Existing approaches counting entity co-occurrences ignore the textual context of each co-occurrence.ResultsWe propose a novel corpus-wide co-occurrence scoring approach to relation extraction that takes the textual context of each co-mention into account. Our method, called CoCoScore, scores the certainty of stating an association for each sentence that co-mentions two entities. CoCoScore is trained using distant supervision based on a gold-standard set of associations between entities of interest. Instead of requiring a manually annotated training corpus, co-mentions are labeled as positives/negatives according to their presence/absence in the gold standard. We show that CoCoScore outperforms previous approaches in identifying human disease–gene and tissue–gene associations as well as in identifying physical and functional protein–protein associations in different species. CoCoScore is a versatile text mining tool to uncover pairwise associations via co-occurrence mining, within and beyond biomedical applications.Availability and implementationCoCoScore is available at: https://github.com/JungeAlexander/cocoscore.Supplementary information Supplementary data are available at Bioinformatics online.

Distant supervision for treatment relation extraction by leveraging MeSH subheadings.

The growing body of knowledge in biomedicine is too vast for human consumption. Hence there is a need for automated systems able to navigate and distill the emerging wealth of information. One fundamental task to that end is relation extraction, whereby linguistic expressions of semantic relationships between biomedical entities are recognized and extracted. In this study, we propose a novel distant supervision approach for relation extraction of binary treatment relationships such that high quality positive/negative training examples are generated from PubMed abstracts by leveraging associated MeSH subheadings. The quality of generated examples is assessed based on the quality of supervised models they induce; that is, the mean performance of trained models (derived via bootstrapped ensembling) on a gold standard test set is used as a proxy for data quality. We show that our approach is preferable to traditional distant supervision for treatment relations and is closer to human crowd annotations in terms of annotation quality. For treatment relations, our generated training data performs at 81.38%, compared to traditional distant supervision at 64.33% and crowd-sourced annotations at 90.57% on the model-wide PR-AUC metric. We also demonstrate that examples generated using our method can be used to augment crowd-sourced datasets. Augmented models improve over non-augmented models by more than two absolute points on the more established F1 metric. We lastly demonstrate that performance can be further improved by implementing a classification loss that is resistant to label noise.

Asthma as an outcome: Exploring multiple definitions of asthma across birth cohorts in the Environmental influences on Child Health Outcomes Children's Respiratory and Environmental Workgroup

Asthma definitions vary widely across research studies and feature a trade-off between inclusivity and precision. Harmonizing definitions across multiple asthma birth cohort studies highlights these challenges.

Chronic wound assessment and infection detection method

BackgroundNumerous patients suffer from chronic wounds and wound infections nowadays. Until now, the care for wounds after surgery still remain a tedious and challenging work for the medical personnel and patients. As a result, with the help of the hand-held mobile devices, there is high demand for the development of a series of algorithms and related methods for wound infection early detection and wound self monitoring.MethodsThis research proposed an automated way to perform (1) wound image segmentation and (2) wound infection assessment after surgical operations. The first part describes an edge-based self-adaptive threshold detection image segmentation method to exclude nonwounded areas from the original images. The second part describes a wound infection assessment method based on machine learning approach. In this method, the extraction of feature points from the suture area and an optimal clustering method based on unimodal Rosin threshold algorithm that divides feature points into clusters are introduced. These clusters are then merged into several regions of interest (ROIs), each of which is regarded as a suture site. Notably, a support vector machine (SVM) can automatically interpret infections on these detected suture site.ResultsFor (1) wound image segmentation, boundary-based evaluation were applied on 100 images with gold standard set up by three physicians. Overall, it achieves 76.44% true positive rate and 89.04% accuracy value. For (2) wound infection assessment, the results from a retrospective study using confirmed wound pictures from three physicians for the following four symptoms are presented: (1) Swelling, (2) Granulation, (3) Infection, and (4) Tissue Necrosis. Through cross-validation of 134 wound images, for anomaly detection, our classifiers achieved 87.31% accuracy value; for symptom assessment, our classifiers achieved 83.58% accuracy value.ConclusionsThis augmentation mechanism has been demonstrated reliable enough to reduce the need for face-to-face diagnoses. To facilitate the use of this method and analytical framework, an automatic wound interpretation app and an accompanying website were developed.Trial registration201505164RIND, 201803108RSB.

A semi-automated tool for identifying agricultural roadway crashes in crash narratives

Objective: Crash reports contain precoded structured data fields and a crash narrative that can be a source of rich information not included in the structured data. The narrative can be useful for identifying vulnerable roadway users, such as agricultural workers. However, using the narratives often requires manual reviews that are time consuming and costly. The objective of this research was to develop a simple and relatively inexpensive, semi-automated tool for screening crash narratives and expediting the process of identifying crashes with specific characteristics, such as agricultural crashes.Methods: Crash records for Louisiana from 2010 to 2015 were obtained from the Louisiana Department of Transportation (LaDOTD). Records with narratives were extracted and stratified by vehicle type. The majority of analyses focused on a vehicle type of farm equipment (Type T). Two keyword lists, an inclusion list and an exclusion list, were created based on the published literature, subject-matter experts, and findings from a pilot project. Next, a semi-automated tool was developed in Microsoft Excel to identify agricultural crashes. Lastly, the tool’s performance was assessed using a gold standard set of agricultural narratives identified through manual review.Results: The tool reduced the search space (e.g., number of narratives that need manual review) for narratives requiring manual review from 6.7 to 59.4% depending on the research question. Sensitivity was high, with 96.1% of agricultural crash narratives being correctly classified. Of the gold standard agricultural narratives, 58.3% included an equipment keyword and 72.8% included a farm equipment brand.Conclusion: This article provides information on how crash narratives can supplement structured crash data. It also provides an easy-to-implement method to facilitate incorporating narratives into safety research along with keyword lists for identifying agricultural crashes.