Gold Standard Set Research Articles

Development and validation of a geographic search filter for MEDLINE (PubMed) to identify studies about Germany.

While geographic search filters exist, few of them are validated and there are currently none that focus on Germany. We aimed to develop and validate a highly sensitive geographic search filter for MEDLINE (PubMed) that identifies studies about Germany. First, using the relative recall method, we created a gold standard set of studies about Germany, dividing it into 'development' and 'testing' sets. Next, candidate search terms were identified using (i) term frequency analyses in the 'development set' and a random set of MEDLINE records; and (ii) a list of German geographic locations, compiled by our team. Then, we iteratively created the filter, evaluating it against the 'development' and 'testing' sets. To validate the filter, we conducted a number of case studies (CSs) and a simulation study. For this validation we used systematic reviews (SRs) that had included studies about Germany but did not restrict their search strategy geographically. When applying the filter to the original search strategies of the 17 SRs eligible for CSs, the median precision was 2.64% (interquartile range [IQR]: 1.34%-6.88%) versus 0.16% (IQR: 0.10%-0.49%) without the filter. The median number-needed-to-read (NNR) decreased from 625 (IQR: 211-1042) to 38 (IQR: 15-76). The filter achieved 100% sensitivity in 13 CSs, 85.71% in 2 CSs and 87.50% and 80% in the remaining 2 CSs. In a simulation study, the filter demonstrated an overall sensitivity of 97.19% and NNR of 42. The filter reliably identifies studies about Germany, enhancing screening efficiency and can be applied in evidence syntheses focusing on Germany.

Large-scale identification of social and behavioral determinants of health from clinical notes: comparison of Latent Semantic Indexing and Generative Pretrained Transformer (GPT) models

BackgroundSocial and behavioral determinants of health (SBDH) are associated with a variety of health and utilization outcomes, yet these factors are not routinely documented in the structured fields of electronic health records (EHR). The objective of this study was to evaluate different machine learning approaches for detection of SBDH from the unstructured clinical notes in the EHR.MethodsLatent Semantic Indexing (LSI) was applied to 2,083,180 clinical notes corresponding to 46,146 patients in the MIMIC-III dataset. Using LSI, patients were ranked based on conceptual relevance to a set of keywords (lexicons) pertaining to 15 different SBDH categories. For Generative Pretrained Transformer (GPT) models, API requests were made with a Python script to connect to the OpenAI services in Azure, using gpt-3.5-turbo-1106 and gpt-4-1106-preview models. Prediction of SBDH categories were performed using a logistic regression model that included age, gender, race and SBDH ICD-9 codes.ResultsLSI retrieved patients according to 15 SBDH domains, with an overall average PPV ≥\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\ge$$\\end{document} 83%. Using manually curated gold standard (GS) sets for nine SBDH categories, the macro-F1 score of LSI (0.74) was better than ICD-9 (0.71) and GPT-3.5 (0.54), but lower than GPT-4 (0.80). Due to document size limitations, only a subset of the GS cases could be processed by GPT-3.5 (55.8%) and GPT-4 (94.2%), compared to LSI (100%). Using common GS subsets for nine different SBDH categories, the macro-F1 of ICD-9 combined with either LSI (mean 0.88, 95% CI 0.82-0.93), GPT-3.5 (0.86, 0.82-0.91) or GPT-4 (0.88, 0.83-0.94) was not significantly different. After including age, gender, race and ICD-9 in a logistic regression model, the AUC for prediction of six out of the nine SBDH categories was higher for LSI compared to GPT-4.0.ConclusionsThese results demonstrate that the LSI approach performs comparable to more recent large language models, such as GPT-3.5 and GPT-4.0, when using the same set of documents. Importantly, LSI is robust, deterministic, and does not have document-size limitations or cost implications, which make it more amenable to real-world applications in health systems.

Advancements in opportunistic intracranial aneurysm screening: The impact of a deep learning algorithm on radiologists' analysis of T2-weighted cranial MRI

(1) Background: Unruptured Intracranial Aneurysms (UIAs) are common blood vessel malformations, occurring in up to 3 % of healthy adults. Magnetic Resonance Angiography (MRA) is frequently used for the screening of UIAs due to its high resolution in vascular anatomy. However, T2-Weighted Magnetic Resonance Imaging (T2WI) is a standard sequence utilized for the majority of outpatient head scans. By employing a sequence such as T2WI, there is a possible shift towards early detection of UIAs through opportunistic screening. Here, we assess a Deep Learning Algorithm (DLA) developed to assist radiologists in identifying and reporting UIAs on T2WI in a routine clinical setting. (2) Methods: A DLA was trained on a set of 110 patients undergoing an MR head scan with the gold standard set by two radiology experts. Eight radiologists were given a cohort of 50 cranial T2WI studies and asked for a routine report. After a 10-day washout period, they reviewed the same cases randomized and supported by the DLA predictions. We assessed changes in sensitivity, specificity, accuracy, and false positives. (3) Results: During routine reporting, the models’ assistance improved the sensitivity of the eight participants by an average of 36.19 and the accuracy by 10.00 percentage points. (4) Conclusion: Our results indicate the potential benefit of deep learning to improve radiologists' detection of UIAs during routine reporting. From this, we can infer that the combination of T2WI with our DLA supports opportunistic screening, suggesting potential approaches for future research and application.

Annotation of epilepsy clinic letters for natural language processing

BackgroundNatural language processing (NLP) is increasingly being used to extract structured information from unstructured text to assist clinical decision-making and aid healthcare research. The availability of expert-annotated documents for the development and validation of NLP applications is limited. We created synthetic clinical documents to address this, and to validate the Extraction of Epilepsy Clinical Text version 2 (ExECTv2) NLP pipeline.MethodsWe created 200 synthetic clinic letters based on hospital outpatient consultations with epilepsy specialists. The letters were double annotated by trained clinicians and researchers according to agreed guidelines. We used the annotation tool, Markup, with an epilepsy concept list based on the Unified Medical Language System ontology. All annotations were reviewed, and a gold standard set of annotations was agreed and used to validate the performance of ExECTv2.ResultsThe overall inter-annotator agreement (IAA) between the two sets of annotations produced a per item F1 score of 0.73. Validating ExECTv2 using the gold standard gave an overall F1 score of 0.87 per item, and 0.90 per letter.ConclusionThe synthetic letters, annotations, and annotation guidelines have been made freely available. To our knowledge, this is the first publicly available set of annotated epilepsy clinic letters and guidelines that can be used for NLP researchers with minimum epilepsy knowledge. The IAA results show that clinical text annotation tasks are difficult and require a gold standard to be arranged by researcher consensus. The results for ExECTv2, our automated epilepsy NLP pipeline, extracted detailed epilepsy information from unstructured epilepsy letters with more accuracy than human annotators, further confirming the utility of NLP for clinical and research applications.

Artificial intelligence and dental age estimation: development and validation of an automated stage allocation technique on all mandibular tooth types in panoramic radiographs.

Age estimation in forensic odontology is mainly based on the development of permanent teeth. To register the developmental status of an examined tooth, staging techniques were developed. However, due to inappropriate calibration, uncertainties during stage allocation, and lack of experience, non-uniformity in stage allocation exists between expert observers. As a consequence, related age estimation results are inconsistent. An automated staging technique applicable to all tooth types can overcome this drawback.This study aimed to establish an integrated automated technique to stage the development of all mandibular tooth types and to compare their staging performances.Calibrated observers staged FDI teeth 31, 33, 34, 37 and 38 according to a ten-stage modified Demirjian staging technique. According to a standardised bounding box around each examined tooth, the retrospectively collected panoramic radiographs were cropped using Photoshop CC 2021® software (Adobe®, version 23.0). A gold standard set of 1639 radiographs were selected (n31 = 259, n33 = 282, n34 = 308, n37 = 390, n38 = 400) and input into a convolutional neural network (CNN) trained for optimal staging accuracy. The performance evaluation of the network was conducted in a five-fold cross-validation scheme. In each fold, the entire dataset was split into a training and a test set in a non-overlapping fashion between the folds (i.e., 80% and 20% of the dataset, respectively). Staging performances were calculated per tooth type and overall (accuracy, mean absolute difference, linearly weighted Cohen's Kappa and intra-class correlation coefficient). Overall, these metrics equalled 0.53, 0.71, 0.71, and 0.89, respectively. All staging performance indices were best for 37 and worst for 31. The highest number of misclassified stages were associated to adjacent stages. Most misclassifications were observed in all available stages of 31.Our findings suggest that the developmental status of mandibular molars can be taken into account in an automated approach for age estimation, while taking incisors into account may hinder age estimation.

The NICE search filters for treating and managing COVID-19: validation in MEDLINE and Embase (Ovid).

In this paper we report how the United Kingdom's National Institute for Health and Care Excellence (NICE) search filters for treating and managing COVID-19 were validated for use in MEDLINE (Ovid) and Embase (Ovid). The objective was to achieve at least 98.9% for recall and 64% for precision. We did two tests of recall to finalize the draft search filters. We updated the data from an earlier peer-reviewed publication for the first recall test. For the second test, we collated a set of systematic reviews from Epistemonikos COVID-19 L.OVE and extracted their primary studies. We calculated precision by screening all the results retrieved by the draft search filters from a targeted sample covering 2020-23. We developed a gold-standard set to validate the search filter by using all articles available from the "Treatment and Management" subject filter in the Cochrane COVID-19 Study Register. In the first recall test, both filters had 99.5% recall. In the second test, recall was 99.7% and 99.8% in MEDLINE and Embase respectively. Precision was 91.1% in a deduplicated sample of records. In validation, we found the MEDLINE filter had recall of 99.86% of the 14,625 records in the gold-standard set. The Embase filter had 99.88% recall of 19,371 records. We have validated search filters to identify records on treating and managing COVID-19. The filters may require subsequent updates, if new SARS-CoV-2 variants of concern or interest are discussed in future literature.

Developing a phenotype risk score for tic disorders in a large, clinical biobank

Tics are a common feature of early-onset neurodevelopmental disorders, characterized by involuntary and repetitive movements or sounds. Despite affecting up to 2% of children and having a genetic contribution, the underlying causes remain poorly understood. In this study, we leverage dense phenotype information to identify features (i.e., symptoms and comorbid diagnoses) of tic disorders within the context of a clinical biobank. Using de-identified electronic health records (EHRs), we identified individuals with tic disorder diagnosis codes. We performed a phenome-wide association study (PheWAS) to identify the EHR features enriched in tic cases versus controls (n = 1406 and 7030; respectively) and found highly comorbid neuropsychiatric phenotypes, including: obsessive-compulsive disorder, attention-deficit/hyperactivity disorder, autism spectrum disorder, and anxiety (p < 7.396 × 10−5). These features (among others) were then used to generate a phenotype risk score (PheRS) for tic disorder, which was applied across an independent set of 90,051 individuals. A gold standard set of tic disorder cases identified by an EHR algorithm and confirmed by clinician chart review was then used to validate the tic disorder PheRS; the tic disorder PheRS was significantly higher among clinician-validated tic cases versus non-cases (p = 4.787 × 10−151; β = 1.68; SE = 0.06). Our findings provide support for the use of large-scale medical databases to better understand phenotypically complex and underdiagnosed conditions, such as tic disorders.

VISTA: an integrated framework for structural variant discovery.

Structural variation (SV) refers to insertions, deletions, inversions, and duplications in human genomes. SVs are present in approximately 1.5% of the human genome. Still, this small subset of genetic variation has been implicated in the pathogenesis of psoriasis, Crohn's disease and other autoimmune disorders, autism spectrum and other neurodevelopmental disorders, and schizophrenia. Since identifying structural variants is an important problem in genetics, several specialized computational techniques have been developed to detect structural variants directly from sequencing data. With advances in whole-genome sequencing (WGS) technologies, a plethora of SV detection methods have been developed. However, dissecting SVs from WGS data remains a challenge, with the majority of SV detection methods prone to a high false-positive rate, and no existing method able to precisely detect a full range of SVs present in a sample. Previous studies have shown that none of the existing SV callers can maintain high accuracy across various SV lengths and genomic coverages. Here, we report an integrated structural variant calling framework, Variant Identification and Structural Variant Analysis (VISTA), that leverages the results of individual callers using a novel and robust filtering and merging algorithm. In contrast to existing consensus-based tools which ignore the length and coverage, VISTA overcomes this limitation by executing various combinations of top-performing callers based on variant length and genomic coverage to generate SV events with high accuracy. We evaluated the performance of VISTA on comprehensive gold-standard datasets across varying organisms and coverage. We benchmarked VISTA using the Genome-in-a-Bottle gold standard SV set, haplotype-resolved de novo assemblies from the Human Pangenome Reference Consortium, along with an in-house polymerase chain reaction (PCR)-validated mouse gold standard set. VISTA maintained the highest F1 score among top consensus-based tools measured using a comprehensive gold standard across both mouse and human genomes. VISTA also has an optimized mode, where the calls can be optimized for precision or recall. VISTA-optimized can attain 100% precision and the highest sensitivity among other variant callers. In conclusion, VISTA represents a significant advancement in structural variant calling, offering a robust and accurate framework that outperforms existing consensus-based tools and sets a new standard for SV detection in genomic research.

Human-AI teams to improve accuracy and timeliness of oncology trial prescreening: Preplanned interim analysis of a randomized trial.

1524 Background: Identifying eligible patients for oncology clinical trials (“prescreening”) relies on manual chart review by clinical research coordinators (CRCs), which is time-consuming and often inaccurate. Consequently, 70% of patients with cancer who meet trial eligibility criteria are not offered participation. Natural language processing (NLP) may improve the accuracy and timeliness of prescreening. Methods: This was a preplanned interim analysis of a paired-design noninferiority trial, powered to assess timeliness. We adapted NLP algorithms to identify 13 common eligibility criteria related to cancer type/stage, prior systemic therapy, actionable biomarkers, and response criteria. NLP systems performed optical character recognition, entity and relationship extraction, and natural language inference through deep learning and symbolic AI techniques. Deidentified unstructured electronic health records (EHRs) from real-world patients with non-small cell lung (NSCLC) or colorectal (CrCa) cancer were presented to CRCs via a secure platform. Two CRCs were randomized 1:1 to view blocks of 20 charts each with (Human+AI) or without (Human-alone) NLP annotations. In pre-trial assessment, the CRCs had 86% inter-rater agreement. The primary outcome was overall chart-level accuracy, defined as percent of the 13 CRC-coded eligibility items matching a gold standard set, determined by 2-3 clinicians blinded to experimental arms. Paired t-tests compared overall accuracy (noninferiority margin 5%) and criteria-specific accuracy. Significant differences were assessed for superiority using two-sided paired t-tests. Mann-Whitney tests compared the secondary outcome of timeliness (time per chart review). Results: Among 74 (40 NSCLC; 34 CrCa) of a planned 400 patients, overall accuracy for Human+AI was noninferior to Human-alone (78.7% vs. 76.7%, mean difference 2.0%, p&lt;0.001 rejecting inferiority); both were greater than AI-alone (63.5%). Superiority for Human+AI was demonstrated for RECIST response, but not for overall nor other criteria-specific accuracy (Table). Median time per review was lower for Human+AI than Human-alone (34.1 vs 43.9 min, adjusted p=0.05). Conclusions: Human+AI teams can improve timeliness of trial prescreening with noninferior accuracy. This platform is being used for eligibility assessment in an ongoing clinical trial. [Table: see text]

Validation of an interprofessional education search strategy in PubMed to optimize IPE literature searching.

With exponential growth in the publication of interprofessional education (IPE) research studies, it has become more difficult to find relevant literature and stay abreast of the latest research. To address this gap, we developed, evaluated, and validated search strategies for IPE studies in PubMed, to improve future access to and synthesis of IPE research. These search strategies, or search hedges, provide comprehensive, validated sets of search terms for IPE publications. The search strategies were created for PubMed using relative recall methodology. The research methods followed the guidance of previous search hedge and search filter validation studies in creating a gold standard set of relevant references using systematic reviews, having expert searchers identify and test search terms, and using relative recall calculations to validate the searches' performance against the gold standard set. The three recommended search hedges for IPE studies presented had recall of 71.5%, 82.7%, and 95.1%; the first more focused for efficient literature searching, the last with high recall for comprehensive literature searching, and the remaining hedge as a middle ground between the other two options. These validated search hedges can be used in PubMed to expedite finding relevant scholarships, staying up to date with IPE research, and conducting literature reviews and evidence syntheses.

AAclust: k-optimized clustering for selecting redundancy-reduced sets of amino acid scales.

Amino acid scales are crucial for sequence-based protein prediction tasks, yet no gold standard scale set or simple scale selection methods exist. We developed AAclust, a wrapper for clustering models that require a pre-defined number of clusters k, such as k-means. AAclust obtains redundancy-reduced scale sets by clustering and selecting one representative scale per cluster, where k can either be optimized by AAclust or defined by the user. The utility of AAclust scale selections was assessed by applying machine learning models to 24 protein benchmark datasets. We found that top-performing scale sets were different for each benchmark dataset and significantly outperformed scale sets used in previous studies. Noteworthy is the strong dependence of the model performance on the scale set size. AAclust enables a systematic optimization of scale-based feature engineering in machine learning applications. The AAclust algorithm is part of AAanalysis, a Python-based framework for interpretable sequence-based protein prediction, which is documented and accessible at https://aaanalysis.readthedocs.io/en/latest and https://github.com/breimanntools/aaanalysis.

Development and application of pharmacological statin-associated muscle symptoms phenotyping algorithms using structured and unstructured electronic health records data.

Statins are widely prescribed cholesterol-lowering medications in the United States, but their clinical benefits can be diminished by statin-associated muscle symptoms (SAMS), leading to discontinuation. In this study, we aimed to develop and validate a pharmacological SAMS clinical phenotyping algorithm using electronic health records (EHRs) data from Minnesota Fairview. We retrieved structured and unstructured EHR data of statin users and manually ascertained a gold standard set of SAMS cases and controls using the published SAMS-Clinical Index tool from clinical notes in 200 patients. We developed machine learning algorithms and rule-based algorithms that incorporated various criteria, including ICD codes, statin allergy, creatine kinase elevation, and keyword mentions in clinical notes. We applied the best-performing algorithm to the statin cohort to identify SAMS. We identified 16 889 patients who started statins in the Fairview EHR system from 2010 to 2020. The combined rule-based (CRB) algorithm, which utilized both clinical notes and structured data criteria, achieved similar performance compared to machine learning algorithms with a precision of 0.85, recall of 0.71, and F1 score of 0.77 against the gold standard set. Applying the CRB algorithm to the statin cohort, we identified the pharmacological SAMS prevalence to be 1.9% and selective risk factors which included female gender, coronary artery disease, hypothyroidism, and use of immunosuppressants or fibrates. Our study developed and validated a simple pharmacological SAMS phenotyping algorithm that can be used to create SAMS case/control cohort to enable further analysis which can lead to the development of a SAMS risk prediction model.

Methods for drug safety signal detection using routinely collected observational electronic health care data: a systematic review

Signal detection is a crucial step in the discovery of post-marketing adverse drug reactions. There is a growing interest in using routinely collected data to complement established spontaneous report analyses.The aim. This work aims to systematically review the methods for drug safety signal detection using routinely collected healthcare data and their performance, both in general and for specific types of drugs and outcomes.Metodology. We conducted a systematic review following the PRISMA guidelines, and registered a protocol in PROSPERO.Results. The review included 101 articles, among which there were 39 methodological works, 25 performance assessment papers, and 24 observational studies. Methods included adaptations from those used with spontaneous reports, traditional epidemiological designs, methods specific to signal detection with real-world data. More recently, implementations of machine learning have been studied in the literature. Twenty-five studies evaluated method performances, 16 of them using the area under the curve (AUC) for a range of positive and negative controls as their main measure. Despite the likelihood that performance measurement could vary by drug-event pair, only 10 studies reported performance stratified by drugs and outcomes, in a heterogeneous manner. The replicability of the performance assessment results was limited due to lack of transparency in reporting and the lack of a gold standard reference set.

A Network Clustering Algorithm for Protein Complex Detection Fused with Power-Law Distribution Characteristic

Network clustering for mining protein complexes from protein–protein interaction (PPI) networks has emerged as a prominent research area in data mining and bioinformatics. Accurately identifying complexes plays a crucial role in comprehending cellular organization and functionality. Network characteristics are often useful in enhancing the performance of protein complex detection methods. Many protein complex detection algorithms have been proposed, primarily focusing on local micro-topological structure metrics while overlooking the potential power-law distribution characteristic of community sizes at the macro global level. The effective use of this distribution characteristic information may be beneficial for mining protein complexes. This paper proposes a network clustering algorithm for protein complex detection fused with power-law distribution characteristic. The clustering algorithm constructs a cluster generation model based on scale-free power-law distribution to generate a cluster with a dense center and relatively sparse periphery. Following the cluster generation model, a candidate cluster is obtained. From a global perspective, the number distribution of clusters of varying sizes is taken into account. If the candidate cluster aligns with the constraints defined by the power-law distribution function of community sizes, it is designated as the final cluster; otherwise, it is discarded. To assess the prediction performance of the proposed algorithm, the gold standard complex sets CYC2008 and MIPS are employed as benchmarks. The algorithm is compared to DPClus, IPCA, SEGC, Core, SR-MCL, and ELF-DPC in terms of F-measure and Accuracy on several widely used protein–protein interaction networks. The experimental results show that the algorithm can effectively detect protein complexes and is superior to other comparative algorithms. This study further enriches the connection between analyzing complex network topology features and mining network function modules, thereby significantly contributing to the improvement of protein complex detection performance.

Nf-core/clipseq - a robust Nextflow pipeline for comprehensive CLIP data analysis.

Crosslinking and immunoprecipitation (CLIP) technologies have become a central component of the molecular biologists' toolkit to study protein-RNA interactions and thus to uncover core principles of RNA biology. There has been a proliferation of CLIP-based experimental protocols, as well as computational tools, especially for peak-calling. Consequently, there is an urgent need for a well-documented bioinformatic pipeline that enshrines the principles of robustness, reproducibility, scalability, portability and flexibility while embracing the diversity of experimental and computational CLIP tools. To address this, we present nf-core/clipseq - a robust Nextflow pipeline for quality control and analysis of CLIP sequencing data. It is part of the international nf-core community effort to develop and curate a best-practice, gold-standard set of pipelines for data analysis. The standards enabled by Nextflow and nf-core, including workflow management, version control, continuous integration and containerisation ensure that these key needs are met. Furthermore, multiple tools are implemented ( e.g. for peak-calling), alongside visualisation of quality control metrics to empower the user to make their own informed decisions based on their data. nf-core/clipseq remains under active development, with plans to incorporate newly released tools to ensure that pipeline remains up-to-date and relevant for the community. Engagement with users and developers is encouraged through the nf-core GitHub repository and Slack channel to promote collaboration. It is available at https://nf-co.re/clipseq.

Developing a Classification Algorithm for Prediabetes Risk Detection From Home Care Nursing Notes: Using Natural Language Processing.

This study developed and validated a rule-based classification algorithm for prediabetes risk detection using natural language processing from home care nursing notes. First, we developed prediabetes-related symptomatic terms in English and Korean. Second, we used natural language processing to preprocess the notes. Third, we created a rule-based classification algorithm with 31 484 notes, excluding 315 instances of missing data. The final algorithm was validated by measuring accuracy, precision, recall, and the F1 score against a gold standard testing set (400 notes). The developed terms comprised 11 categories and 1639 words in Korean and 1181 words in English. Using the rule-based classification algorithm, 42.2% of the notes comprised one or more prediabetic symptoms. The algorithm achieved high performance when applied to the gold standard testing set. We proposed a rule-based natural language processing algorithm to optimize the classification of the prediabetes risk group, depending on whether the home care nursing notes contain prediabetes-related symptomatic terms. Tokenization based on white space and the rule-based algorithm were brought into effect to detect the prediabetes symptomatic terms. Applying this algorithm to electronic health records systems will increase the possibility of preventing diabetes onset through early detection of risk groups and provision of tailored intervention.

Abstract 2454: Identification of colorectal cancer stem cells from single cell RNA sequencing

Abstract Cancer stem cells (CSCs) play a critical role in metastasis, relapse and therapy resistance in colorectal cancer (CRC). Although the normal lineage of cell development in the intestine has identified many of the genes involved in the induction and maintenance of pluripotency, recent studies of CSCs suggest significant heterogeneity and lack of common surface markers for identification. In this study, we perform single cell RNA sequencing (scRNA-seq) to extract a single cell stemness signature (SCS_sig) for colorectal CSCs identification and as a potential predictor of relapse after surgical resection. A total of 26 fresh surgery specimens (6 colon tumor, 3 colon matched normal, 9 liver metastasis, 8 liver matched normal) from neoadjuvant chemotherapy (NAC) treated patients with single-cell transcriptome generated using 10X Chromium scRNA-seq and Illumina platforms were analyzed. Following extensive quality control and batch effect correction, we identified 20k epithelial cells using canonical epithelial cell marker genes (EPCAM, KRT8 and KRT18) from a total of 111,200 cells, with mean 1113 epithelial cell per sample (excluding liver normal samples).To obtain the CSCs profiling signature, we first annotated a ‘gold-standard’ set of CSC that expressed all canonical colon CSC marker genes (LGR5, ASCL2, EPHB2, PROM1, AXIN2, LEFTY1, RNF43, CD44 and SLC12A2), identifying 346 CSCs (1.71% of epithelial cells). Published stemness signatures were significantly enriched in the gold standard stem cells (false discovery rate (FDR) &amp;lt; 0.05). Then we selected the 50 most significantly up-regulated genes in this gold standard set (relative to other epithelial cells) as a single-cell stemness signature (named SCS_sig hereafter). Interesting, we saw that in every tumor there was a continuum of stemness, rather than distinct stem and differentiated populations. The stemness state by SCS_sig was positively correlated with less differentiated state calculated by CytoTRACE score (r = 0.76, p&amp;lt; 0.001), a computational method that predicts the differentiation state of cells from scRNA-seq data. The SCS_sig was significantly higher in tumor cells compared to normal cells in our cohort as well as the previously published SMC cohort (fold change = 2.52 and p &amp;lt; 0.001, fold change = 1.44 and p &amp;lt; 0.001, respectively). There was notable heterogeneity in the fraction of stem-like tumor cells with in 15 tumor samples, with the tumor with greatest CSC percentage having the shortest time to relapse. The SCS_sig generated from stem-like cells can identify the stemness state of epithelial cells in scRNAseq data. A signature score is more simple and robust than using few or panel of CSCs marker genes to identify single CSC because of the reads dropout in scRNA-seq. Further investigation of the SCS_sig as a predictive and prognostic biomarker in CRC is ongoing in larger patient cohorts. Citation Format: Kangyu Lin, Mohammad Zeineddine, Saikat Chowdhury, Nicholas Hornstein, Sendurai Mani, Scott Kopetz, John Paul Shen. Identification of colorectal cancer stem cells from single cell RNA sequencing [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2023; Part 1 (Regular and Invited Abstracts); 2023 Apr 14-19; Orlando, FL. Philadelphia (PA): AACR; Cancer Res 2023;83(7_Suppl):Abstract nr 2454.

Application of text mining to the development and validation of a geographic search filter to facilitate evidence retrieval in Ovid MEDLINE: An example from the United States.

Given the increasing volume of published research in bibliographic databases, efficient retrieval of evidence is crucial and represents an opportunity to integrate novel techniques such as text mining. To develop and validate a geographic search filter for identifying research from the United States (US) in Ovid MEDLINE. US and non-US citations were collected from bibliographies of evidence-based reviews. Citations were partitioned by US/non-US status and randomly divided to a training and testing set. Using text mining, common one- and two-word terms in title/abstract fields were identified, and frequencies compared between US/non-US citations. Common US-related terms included (as ratio of frequency in US/non-US citations) US populations and geographic terms [e.g., 'Americans' (15.5), 'Baltimore' (20.0)]. Common non-US terms were non-US geographic terms [e.g., 'Japan' (0.04), 'French' (0.05)]. A search filter was developed with 98.3% sensitivity and 82.7% specificity. This search filter will streamline the identification of evidence from the US. Periodic updates may be necessary to reflect changes in MEDLINE's controlled vocabulary. Text mining was instrumental to the development of this search filter. A novel technique generated a gold standard set comprising >20,000 citations. This method may be adapted to develop subsequent geographic search filters.

Development and validation of paired MEDLINE and Embase search filters for cost-utility studies

BackgroundSearch filters are standardised sets of search terms, with validated performance, that are designed to retrieve studies with specific characteristics. A cost–utility analysis (CUA) is the preferred type of economic evaluation to underpin decision-making at the National Institute for Health and Care Excellence (NICE). Until now, when searching for economic evidence for NICE guidelines, we have used a broad set of health economic-related search terms, even when the reviewer’s interest is confined to CUAs alone. METHODS: We developed search filters to retrieve CUAs from MEDLINE and Embase. Our aim was to achieve recall of 90% or better across both databases while reducing the overall yield compared with our existing broad economic filter. We used the relative recall method along with topic expert input to derive and validate 3 pairs of filters, assessed by their ability to identify a gold-standard set of CUAs that had been used in published NICE guidelines. We developed and validated MEDLINE and Embase filters in pairs (testing whether, when used together, they find target studies in at least 1 database), as this is how they are used in practice. We examined the proxy-precision of our new filters by comparing their overall yield with our previous approach using publications indexed in a randomly selected year (2010). RESULTS: All 3 filter-pairs exceeded our target recall and led to substantial improvements in search proxy-precision. Our paired ‘sensitive’ filters achieved 100% recall (95% CI 99.0 to 100%) in the validation set. Our paired ‘precise’ filters also had very good recall (97.6% [95%CI: 95.4 to 98.9%]). We estimate that, compared with our previous search strategy, using the paired ‘sensitive’ filters would reduce reviewer screening burden by a factor of 5 and the ‘precise’ versions would do so by a factor of more than 20. CONCLUSIONS: Each of the 3 paired cost–utility filters enable the identification of almost all CUAs from MEDLINE and Embase from the validation set, with substantial savings in screening workload compared to our previous search practice. We would encourage other researchers who regularly use multiple databases to consider validating search filters in combination as this will better reflect how they use databases in their everyday work.

The Value of Ultrasound-guided Core Needle Biopsy in Differentiating Benign from Malignant Salivary Gland Lesions.

In recent years, core needle biopsy (CNB) technique has received much attention, being used as alternative method of tissue sampling for surgical biopsy of salivary gland tumors (SGTs). The present study aimed to evaluate the value of CNB in differentiating benign from malignant SGTs. Patients with suspected benign or malignant SGTs in imaging were enrolled in this study. All core needle biopsies were performed under ultrasound guidance, i.e. ultrasound-guided Core Needle Biopsy (USCNB). Histological examination of the specimen after surgical excision was regarded as gold standard test and set as reference standard to assess USCNB accuracy for discriminating between ultrasound-visible benign and malignant SGTs. Based on USCNB results, from 36 participants (14 women and 22 men) with SGTs, 44.4% of detected tumors were benign and 55.5% were malignant. Twenty-two patients underwent surgery and postoperative specimen histological examination showed that 59% of excised lesions were benign and 41% were malignant. USCNB and surgical biopsy (SB) findings were completely compatible for 21 patients. Sensitivity, specificity, positive predictive value, negative predictive value and diagnostic accuracy of USCNB were 100% in differentiating malignant from benign lesions. USCNB is a valuable and accurate method of diagnosis with high sensitivity and specificity in distinguishing benign from malignant ultrasound-visible SGTs.