Reported Performance Measures Research Articles

Large language models in neurosurgery: a systematic review and meta-analysis.

Large Language Models (LLMs) have garnered increasing attention in neurosurgery and possess significant potential to improve the field. However, the breadth and performance of LLMs across diverse neurosurgical tasks have not been systematically examined, and LLMs come with their own challenges and unique terminology. We seek to identify key models, establish reporting guidelines for replicability, and highlight progress in key application areas of LLM use in the neurosurgical literature. We searched PubMed and Google Scholar using terms related to LLMs and neurosurgery ("large language model" OR "LLM" OR "ChatGPT" OR "GPT-3" OR "GPT3" OR "GPT-3.5" OR "GPT3.5" OR "GPT-4" OR "GPT4" OR "LLAMA" OR "MISTRAL" OR "BARD") AND "neurosurgery". The final set of articles was reviewed for publication year, application area, specific LLM(s) used, control/comparison groups used to evaluate LLM performance, whether the article reported specific LLM prompts, prompting strategy types used, whether the LLM query could be reproduced in its entirety (including both the prompt used and any adjoining data), measures of hallucination, and reported performance measures. Fifty-one articles met inclusion criteria, and were categorized into six application areas, with the most common being Generation of Text for Direct Clinical Use (n = 14, 27.5%), Answering Standardized Exam Questions (n = 12, 23.5%), and Clinical Judgement and Decision-Making Support (n = 11, 21.6%). The most frequently used LLMs were GPT-3.5 (n = 30, 58.8%), GPT-4 (n = 20, 39.2%), Bard (n = 9, 17.6%), and Bing (n = 6, 11.8%). Most studies (n = 43, 84.3%) used LLMs directly out-of-the-box, while 8 studies (15.7%) conducted advanced pre-training or fine-tuning. Large language models show advanced capabilities in complex tasks and hold potential to transform neurosurgery. However, research typically addresses basic applications and overlooks enhancing LLM performance, facing reproducibility issues. Standardizing detailed reporting, considering LLM stochasticity, and using advanced methods beyond basic validation are essential for progress.

Impact of artificial intelligence on healthcare-associated infection control: a systematic review

Abstract Background Healthcare-associated infections (HAIs) represent a significant public health concern, correlating with increased morbidity, mortality rates and healthcare expenditures. While artificial intelligence (AI) systems offer transformative potential in enhancing HAIs detection and control practices, the actual performance effectiveness of these systems remains uncertain. This systematic review updates a previously published study from 2020 and evaluates the performance of AI-based tools for surveillance, detection, and control of HAIs. Methods PRISMA 2020 guidelines were applied. The study protocol has been registered in PROSPERO (ID: CRD42024513145). PubMed, Embase, Scopus and Web of Science were searched for experimental and observational studies assessing the performance of AI-based tools to detect and control HAIs, published in English. Results From 8,701 articles initially identified, 4,212 records were removed due to duplication. Out of 4,489 papers screened, 147 were included. Studies reported performance measures including sensitivity, specificity, positive and negative predictive values, area under the receiver operating characteristic curve, accuracy, precision, F1 score. Significant heterogeneity was found in the types of technology, infections targeted, health care settings and data sources between studies. Conclusions The increase in published evidence since the previous review reflects the growing interest and use of new technologies such as Large Language Models, showing promising performance in surveillance, early diagnosis and prediction of HAIs. However, the observed heterogeneity in study designs, targeted infections, healthcare settings, and data sources underscores the need for standardised methodologies and robust validation processes to ensure the reliability and comparability of results across different studies. Key messages • The use of AI-based tools has the potential to enhance surveillance, detection, and control of healthcare-associated infections, offering a transformative impact on healthcare systems. • Standardised methodologies and validation processes are needed to ensure comparable results across studies and to maximise the real-world impact of AI tools in HAI surveillance and control efforts.

Predictive Models of Psychological Distress, Quality of Life, and Adherence to Medication in Breast Cancer Patients: A Scoping Review.

An interplay of clinical and psychosocial variables affects breast cancer patients' experiences and clinical trajectories. Several studies investigated the role of socio-demographic, clinical, and psychosocial factors in predicting relevant outcomes in breast cancer care, thus developing predictive models. Our aim is to summarize predictive models for specific psychological and behavioral outcomes: psychological distress, quality of life, and medication adherence. Specifically, we aim to map the determinants of the outcomes of interest, offering a thorough overview of these models. Databases (PubMed, Scopus, Embase) have been searched to identify studies meeting the inclusion criteria: a breast cancer patients' sample, development/validation of a predictive model for selected psychological/behavioral outcomes (ie, psychological distress, quality of life, and medication adherence), and availability of English full-text. Twenty-one papers describing predictive models for psychological distress, quality of life, and adherence to medication in breast cancer were included. The models were developed using different statistical approaches. It has been shown that treatment-related factors (eg, side-effects, type of surgery or treatment received), socio-demographic (eg, younger age, lower income, and inactive occupational status), clinical (eg, advanced stage of disease, comorbidities, physical symptoms such as fatigue, insomnia, and pain) and psychological variables (eg, anxiety, depression, body image dissatisfaction) might predict poorer outcomes. Predictive models of distress, quality of life, and adherence, although heterogeneous, showed good predictive values, as indicated by the reported performance measures and metrics. Many of the predictors are easily available in patients' health records, whereas others (eg, coping strategies, perceived social support, illness perceptions) might be introduced in routine assessment practices. The possibility to assess such factors is a relevant resource for clinicians and researchers involved in developing and implementing psychological interventions for breast cancer patients.

Medicare Care Compare: Publicly reported performance measures

Data transparency helps achieve national care improvement goals.

Quality Improvement Efforts in VA Community Living Centers Following Public Reporting of Performance

ABSTRACT For two decades, the U.S. government has publicly reported performance measures for most nursing homes, spurring some improvements in quality. Public reporting is new, however, to Department of Veterans Affairs nursing homes (Community Living Centers [CLCs]). As part of a large, public integrated healthcare system, CLCs operate with unique financial and market incentives. Thus, their responses to public reporting may differ from private sector nursing homes. In three CLCs with varied public ratings, we used an exploratory, qualitative case study approach involving semi-structured interviews to compare how CLC leaders (n = 12) perceived public reporting and its influence on quality improvement. Across CLCs, respondents said public reporting was helpful for transparency and to provide an “outside perspective” on CLC performance. Respondents described employing similar strategies to improve their public ratings: using data, engaging staff, and clearly defining staff roles vis-à-vis quality improvement, although more effort was required to implement change in lower performing CLCs. Our findings augment those from prior studies and offer new insights into the potential for public reporting to spur quality improvement in public nursing homes and those that are part of integrated healthcare systems.

Screening for Human Trafficking of Minors in Health Care: A Systematic Review

ABSTRACT Human trafficking is a form of exploitation that has significant consequences to trafficked minors. Many victims have accessed health care during their time of exploitation and are not identified. One of the greatest barriers to identification in health care is the lack of screening tools that are reliable, valid, and practical to workflow. The purpose of this manuscript is to provide a systematic review of current screening instruments for human trafficking including their performance measures and applicability to identifying minors. Forty-five instruments were reviewed in full to assess their applicability of identifying minors in the health care setting who are involved in human trafficking. A majority of the instruments reviewed are not peer reviewed nor have they reported performance measures to ensure accuracy of screening results. Only one instrument was validated and developed for the health care setting. Screening for human trafficking in minors should hold the same emphasis as screening for other types of abuse and maltreatment in health care. This article addresses the U.N. Sustainability Goal 16.2 to End abuse, exploitation, trafficking and all forms of violence against and torture of children. Recognizing and addressing the gaps in current practice such as increasing provider knowledge and working toward the development of valid and reliable screening tools will increase the efforts to identify exploited minors and assist in resource utilization to reduce the sequela from human trafficking.

A Systematic Review of Polygenic Models for Predicting Drug Outcomes.

Polygenic models have emerged as promising prediction tools for the prediction of complex traits. Currently, the majority of polygenic models are developed in the context of predicting disease risk, but polygenic models may also prove useful in predicting drug outcomes. This study sought to understand how polygenic models incorporating pharmacogenetic variants are being used in the prediction of drug outcomes. A systematic review was conducted with the aim of gaining insights into the methods used to construct polygenic models, as well as their performance in drug outcome prediction. The search uncovered 89 papers that incorporated pharmacogenetic variants in the development of polygenic models. It was found that the most common polygenic models were constructed for drug dosing predictions in anticoagulant therapies (n = 27). While nearly all studies found a significant association with their polygenic model and the investigated drug outcome (93.3%), less than half (47.2%) compared the performance of the polygenic model against clinical predictors, and even fewer (40.4%) sought to validate model predictions in an independent cohort. Additionally, the heterogeneity of reported performance measures makes the comparison of models across studies challenging. These findings highlight key considerations for future work in developing polygenic models in pharmacogenomic research.

Theory and Implementation of Integral Illumination

Integral imaging, i.e., the use of lenticular optics to display stereoscopic/multiscopic images, is now being used in an array of products including glasses-free 3D displays. This paper describes <i>integral illumination</i>, an adaptation of integral imaging where fine-grained control of plenoptic light fields is used to realize new forms of programmable lighting. Relying on a combination of an imaging apparatus and custom lenticular optics, integral illumination devices can produce high-fidelity illusions of real and imagined light sources (e.g., spotlight, chandelier), replicating their illumination effects. Such devices have potential uses as ambient lighting fixtures, photography/videography equipment, components of artistic installations, etc. The paper will provide a general overview of integral illumination, describing its basic principles, hardware configuration, control mechanism, range of capabilities, and theoretical/practical limitations. We will also present a sample implementation of a working integral illumination device, describe its engineering details, report performance measurements, and discuss possibilities for future improvements and extensions.

Leaf Image-Based Plant Disease Identification Using Color and Texture Features

Identification of plant disease is usually done through visual inspection or during laboratory examination which causes delays resulting in yield loss by the time identification is complete. On the other hand, complex deep learning models perform the task with reasonable performance but due to their large size and high computational requirements, they are not suited to mobile and handheld devices. Our proposed approach contributes automated identification of plant diseases which follows a sequence of steps involving pre-processing, segmentation of diseased leaf area, calculation of features based on the Gray-Level Co-occurrence Matrix (GLCM), feature selection and classification. In this study, six color features and twenty-two texture features have been calculated. Support vector machines is used to perform one-vs-one classification of plant disease. The proposed model of disease identification provides an accuracy of 98.79% with a standard deviation of 0.57 on tenfold cross-validation. The accuracy on a self-collected dataset is 82.47% for disease identification and 91.40% for healthy and diseased classification. The reported performance measures are better or comparable to the existing approaches and highest among the feature-based methods, presenting it as the most suitable method to automated leaf-based plant disease identification. This prototype system can be extended by adding more disease categories or targeting specific crop or disease categories.

Accuracy of risk models used for public reporting of heart transplant center performance

Heart transplant programs and regulatory entities require highly accurate performance metrics to support internal quality improvement activities and national oversight of transplant programs, respectively. We assessed the accuracy of publicly reported performance measures. We used the United Network for Organ Sharing registry to study patients who underwent heart transplantation between January 1, 2016 and June 30, 2018. We used tests of calibration to compare the observed rate of 1-year graft failure to the expected risk of 1-year graft failure, which was calculated for each recipient using the July 2019 method published by the Scientific Registry of Transplant Recipients (SRTR). The primary study outcome was the joint test of calibration, which accounts for both the total number of events predicted (calibration-in-the-large) and dispersion of risk predictions (calibration slope). 6,528 heart transplants were analyzed. The primary test of calibration failed (p <0.0001), indicating poor accuracy of the SRTR model. The calibration-in-the-large statistic (0.63, 95% confidence interval [CI] 0.58-0.68, p < 0.0001) demonstrated overestimation of event rates while the calibration slope statistic (0.56, 95% CI 0.49-0.62, p <0.0001) indicated over-dispersion of event rates. Pre-specified subgroup analyses demonstrated poor calibration for all subgroups (each p <0.01). After recalibration, program-level observed/expected ratios increased by a median of 0.14 (p <0.0001). Risk models employed for publicly-reported graft survival at U.S. heart transplant centers lack accuracy in general and in all subgroups tested. The use of disease-specific models may improve the accuracy of program performance metrics.

Assessment of Variation in Electronic Health Record Capabilities and Reported Clinical Quality Performance in Ambulatory Care Clinics, 2014-2017

Electronic health records (EHRs) are widely promoted to improve the quality of health care, but information about the association of multifunctional EHRs with broad measures of quality in ambulatory settings is scarce. To assess the association between EHRs with different degrees of capabilities and publicly reported ambulatory quality measures in at least 3 clinical domains of care. This cross-sectional and longitudinal study was conducted using survey responses from 1141 ambulatory clinics in Minnesota, Washington, and Wisconsin affiliated with a health system that responded to the Healthcare Information and Management Systems Society Annual Survey and reported performance measures in 2014 to 2017. Statistical analysis was performed from July 10, 2019, through February 26, 2021. A composite measure of EHR capability that considered 50 EHR capabilities in 7 functional domains, grouped into the following ordered categories: no functional EHR, EHR underuser, EHR, neither underuser or superuser, EHR superuser; as well as a standardized composite of ambulatory clinical performance measures that included 3 to 25 individual measures (median, 13 individual measures). In 2014, 381 of 746 clinics (51%) were EHR superusers; this proportion increased in each subsequent year (457 of 846 clinics [54%] in 2015, 510 of 881 clinics [58%] in 2016, and 566 of 932 clinics [61%] in 2017). In each cross-sectional analysis year, EHR superusers had better clinical quality performance than other clinics (adjusted difference in score: 0.39 [95% CI, 0.12-0.65] in 2014; 0.29 [95% CI, -0.01 to 0.59] in 2015; 0.26 [95% CI, -0.05 to 0.56] in 2016; and 0.20 [95% CI, -0.04 to 0.45] in 2017). This difference in scores translates into an approximately 9% difference in a clinic's rank order in clinical quality. In longitudinal analyses, clinics that progressed to EHR superuser status had only slightly better gains in clinical quality between 2014 and 2017 compared with the gains in clinical quality of clinics that were static in terms of their EHR status (0.10 [95% CI, -0.13 to 0.32]). In an exploratory analysis, different types of EHR capability progressions had different degrees of associated improvements in ambulatory clinical quality (eg, progression from no functional EHR to a status short of superuser, 0.06 [95% CI, -0.40 to 0.52]; progression from EHR underuser to EHR superuser, 0.18 [95% CI, -0.14 to 0.50]). Between 2014 and 2017, ambulatory clinics in Minnesota, Washington, and Wisconsin with EHRs having greater capabilities had better composite measures of clinical quality than other clinics, but clinics that gained EHR capabilities during this time had smaller increases in clinical quality that were not statistically significant.

A-064 Executive Functioning in an Acute Pediatric Neurocritical Care Population Recently Discharged from the PICU: Exploring the Associations among Parent Report and Individual Performance Measures

Abstract Objective An emerging literature has identified that PICU survivors face a host of long-term physical, cognitive, emotional, and psychosocial difficulties that stem from the underlying concern and side effects of critical care intervention. Research shows that executive functioning (EF) is particularly vulnerable to sequelae. This study sought to understand associations between parent-report of daily life EF and performance-based measures following neurocritical care. Method Twenty-three children ages 8–16 (M = 12.66 years; 39% male) were screened as part of an integrated (neuropsychology and pediatric critical care) acute phase (3–6-week post-discharge) follow-up clinic. Injuries included children with traumatic brain injury (n = 19), or acquired brain injury (i.e., anoxic brain injury, AVM, acute cerebellitis, hemorrhagic stroke; n = 4). EF outcomes were assessed using the Behavior Rating Inventory of Executive Functioning, Second Edition (BRIEF-2), Delis-Kaplan Executive Function System (D-KEFS) Trail Making Test and Verbal Fluency Test, Children’s Memory Scale Digits Backwards, and Wechsler Intelligence Scale for Children, Fifth Edition (WISC-V) Coding and Symbol Search subtests. Bivariate correlations were used to examine associations among parent-report of EF and performance-based measures of EF in our neurocritical care population. Results Analyses demonstrated significant (□ = .05) correlations between the WISC-V Coding subtest and the BRIEF-2 Shift scale (r = −.44) and Global Executive Composite (GEC; r = −.59). Correlations between D-KEFS Category Fluency and BRIEF-2 Initiate (r = −.51), Working Memory (r = −.44), and GEC (r = −.55) were significant. Inverse correlations demonstrate agreement. Conclusion In the acute recovery phase following neurocritical care, parent-report and specific performance-based measures are only moderately associated, demonstrating the importance of multi-method assessment to detect potential acquired deficits to inform rehabilitation.

Evidence-based Clinical Decision Support Systems for the prediction and detection of three disease states in critical care: A systematic literature review.

Background: Clinical decision support (CDS) systems have emerged as tools providing intelligent decision making to address challenges of critical care. CDS systems can be based on existing guidelines or best practices; and can also utilize machine learning to provide a diagnosis, recommendation, or therapy course. Methods: This research aimed to identify evidence-based study designs and outcome measures to determine the clinical effectiveness of clinical decision support systems in the detection and prediction of hemodynamic instability, respiratory distress, and infection within critical care settings. PubMed, ClinicalTrials.gov and Cochrane Database of Systematic Reviews were systematically searched to identify primary research published in English between 2013 and 2018. Studies conducted in the USA, Canada, UK, Germany and France with more than 10 participants per arm were included. Results: In studies on hemodynamic instability, the prediction and management of septic shock were the most researched topics followed by the early prediction of heart failure. For respiratory distress, the most popular topics were pneumonia detection and prediction followed by pulmonary embolisms. Given the importance of imaging and clinical notes, this area combined Machine Learning with image analysis and natural language processing. In studies on infection, the most researched areas were the detection, prediction, and management of sepsis, surgical site infections, as well as acute kidney injury. Overall, a variety of Machine Learning algorithms were utilized frequently, particularly support vector machines, boosting techniques, random forest classifiers and neural networks. Sensitivity, specificity, and ROC AUC were the most frequently reported performance measures. Conclusion: This review showed an increasing use of Machine Learning for CDS in all three areas. Large datasets are required for training these algorithms; making it imperative to appropriately address, challenges such as class imbalance, correct labelling of data and missing data. Recommendations are formulated for the development and successful adoption of CDS systems.

Electronic health records systems and hospital clinical performance: a study of nationwide hospital data.

Electronic health records (EHRs) were expected to yield numerous benefits. However, early studies found mixed evidence of this. We sought to determine whether widespread adoption of modern EHRs in the US has improved clinical care. We studied hospitals reporting performance measures from 2008-2015 in the Centers for Medicare and Medicaid Services Hospital Compare database that also reported having an EHR in the American Hospital Association 2015 IT supplement. Using interrupted time-series analysis, we examined the association of EHR implementation, EHR vendor, and Meaningful Use status with 11 process measures and 30-day hospital readmission and mortality rates for heart failure, pneumonia, and acute myocardial infarction. A total of 1246 hospitals contributed 8222 hospital-years. Compared to hospitals without EHRs, hospitals with EHRs had significant improvements over time on 5 of 11 process measures. There were no substantial differences in readmission or mortality rates. Hospitals with CPSI EHR systems performed worse on several process and outcome measures. Otherwise, we found no substantial improvements in process measures or condition-specific outcomes by duration of EHR use, EHR vendor, or a hospital's Meaningful Use Stage 1 or Stage 2 status. In this national study of hospitals with modern EHRs, EHR use was associated with better process of care measure performance but did not improve condition-specific readmission or mortality rates regardless of duration of EHR use, vendor choice, or Meaningful Use status. Further research is required to understand why EHRs have yet to improve standard outcome measures and how to better realize the potential benefits of EHR systems.

Sustaining and spreading quality improvement: Decreasing intrapartum malpractice risk.

Malpractice liability is an ongoing problem in obstetrics. However, developing, sustaining, and spreading effective interventions is challenging. The aim of this study is to examine the spread and sustainability of a multilevel integrated practice and coordinated communication model 66 months after its original implementation. Data on labor and delivery patients from 37 hospitals (5 beta sites and 32 expansion sites) were analyzed for the 81-month time period from January 2010 through September 2016. High-risk occurrence rates per 1000 live births decreased by over 70% at both beta and expansion sites. The likelihood of a high-risk occurrence was statistically significantly lower during the final study period than in the preintervention period at both beta sites (odds ratio [OR]=0.218; p<.0001) and expansion sites (OR=0.288; p<.001). The multilevel integrated practice and coordinated communication model was successfully spread and sustained. Key elements contributing to this success included developing and maintaining evidence-based guidelines, ensuring leadership buy-in and support, collecting and reporting performance measures, holding teams accountable, providing training, and ensuring transparent communication.

Detailed Findings From the CLER National Report of Findings 2018.

Detailed Findings From the CLER National Report of Findings 2018.

Diabetes risk scores for Hispanics living in the United States: A systematic review

AimUndiagnosed diabetes is more prevalent among racial/ethnic minorities in the United States (U.S.). Despite the proliferation of risk scores, few have been validated in Hispanics populations. The aim of this study is to systematically review published studies that developed risk scores to identify undiagnosed Type 2 Diabetes Mellitus based on self-reported information that were validated for Hispanics in the U.S. MethodsThe search included PubMed, EMBASE, Cochrane and CINAHL from inception to 2016 without language restrictions. Risk scores whose main outcome was undiagnosed Type 2 diabetes reporting performance measures for Hispanics were included. ResultsWe identified three studies that developed and validated risk scores for undiagnosed diabetes based on questionnaire data. Two studies were conducted in Latin America and one in the U.S. All three studies reported adequate performance (area under the receiving curve (AUC) range between0.68and 0.78). The study conducted in the U.S. reported a higher sensitivity of their risk score for Hispanics than whites. The limited number of studies, small size and heterogeneity of the combined cohorts provide limited evidence of the validity of risk scores for Hispanics. ConclusionsEfforts to develop and validate risk prediction models in Hispanic populations in the U.S are needed, particularly given the diversity of thisfast growing population. Healthcare professionals should be aware of the limitations of applying risk scores developed for the general population on Hispanics.

A Systematic Review of Coding Systems Used in Pharmacoepidemiology and Database Research

Clinical coding systems have been developed to translate real-world healthcare information such as prescriptions, diagnoses and procedures into standardized codes appropriate for use in large healthcare datasets. Due to the lack of information on coding system characteristics and insufficient uniformity in coding practices, there is a growing need for better understanding of coding systems and their use in pharmacoepidemiology and observational real world data research. To determine: 1) the number of available coding systems and their characteristics, 2) which pharmacoepidemiology databases are they adopted in, 3) what outcomes and exposures can be identified from each coding system, and 4) how robust they are with respect to consistency and validity in pharmacoepidemiology and observational database studies. Electronic literature database and unpublished literature searches, as well as hand searching of relevant journals were conducted to identify eligible articles discussing characteristics and applications of coding systems in use and published in the English language between 1986 and 2016. Characteristics considered included type of information captured by codes, clinical setting(s) of use, adoption by a pharmacoepidemiology database, region, and available mappings. Applications articles describing the use and validity of specific codes, code lists, or algorithms were also included. Data extraction was performed independently by two reviewers and a narrative synthesis was performed. A total of 897 unique articles and 57 coding systems were identified, 17% of which included country-specific modifications or multiple versions. Procedures (55%), diagnoses (36%), drugs (38%), and site of disease (39%) were most commonly and directly captured by these coding systems. The systems were used to capture information from the following clinical settings: inpatient (63%), ambulatory (55%), emergency department (ED, 34%), and pharmacy (13%). More than half of all coding systems were used in Europe (59%) and North America (57%). 34% of the reviewed coding systems were utilized in at least 1 of the 16 pharmacoepidemiology databases of interest evaluated. 21% of coding systems had studies evaluating the validity and consistency of their use in research within pharmacoepidemiology databases of interest. The most prevalent validation method was comparison with a review of patient charts, case notes or medical records (64% of reviewed validation studies). The reported performance measures in the reviewed studies varied across a large range of values (PPV 0-100%, NPV 6-100%, sensitivity 0-100%, specificity 23-100% and accuracy 16-100%) and were dependent on many factors including coding system(s), therapeutic area, pharmacoepidemiology database, and outcome. Coding systems vary by type of information captured, clinical setting, and pharmacoepidemiology database and region of use. Of the 57 reviewed coding systems, few are routinely and widely applied in pharmacoepidemiology database research. Indication and outcome dependent heterogeneity in coding system performance suggest that accurate definitions and algorithms for capturing specific exposures and outcomes within large healthcare datasets should be developed on a case-by-case basis and in consultation with clinical experts.

Multiple performance measures are needed to evaluate triage systems in the emergency department

ObjectivesEmergency department triage systems can be considered prediction rules with an ordinal outcome, where different directions of misclassification have different clinical consequences. We evaluated strategies to compare the performance of triage systems and aimed to propose a set of performance measures that should be used in future studies. Study Design and SettingWe identified performance measures based on literature review and expert knowledge. Their properties are illustrated in a case study evaluating two triage modifications in a cohort of 14,485 pediatric emergency department visits. Strengths and weaknesses of the performance measures were systematically appraised. ResultsCommonly reported performance measures are measures of statistical association (34/60 studies) and diagnostic accuracy (17/60 studies). The case study illustrates that none of the performance measures fulfills all criteria for triage evaluation. Decision curves are the performance measures with the most attractive features but require dichotomization. In addition, paired diagnostic accuracy measures can be recommended for dichotomized analysis, and the triage-weighted kappa and Nagelkerke's R2 for ordinal analyses. Other performance measures provide limited additional information. ConclusionWhen comparing modifications of triage systems, decision curves and diagnostic accuracy measures should be used in a dichotomized analysis, and the triage-weighted kappa and Nagelkerke's R2 in an ordinal approach.

Assessing the performance of methodological search filters to improve the efficiency of evidence information retrieval: five literature reviews and a qualitative study.

Effective study identification is essential for conducting health research, developing clinical guidance and health policy and supporting health-care decision-making. Methodological search filters (combinations of search terms to capture a specific study design) can assist in searching to achieve this. This project investigated the methods used to assess the performance of methodological search filters, the information that searchers require when choosing search filters and how that information could be better provided. Five literature reviews were undertaken in 2010/11: search filter development and testing; comparison of search filters; decision-making in choosing search filters; diagnostic test accuracy (DTA) study methods; and decision-making in choosing diagnostic tests. We conducted interviews and a questionnaire with experienced searchers to learn what information assists in the choice of search filters and how filters are used. These investigations informed the development of various approaches to gathering and reporting search filter performance data. We acknowledge that there has been a regrettable delay between carrying out the project, including the searches, and the publication of this report, because of serious illness of the principal investigator. The development of filters most frequently involved using a reference standard derived from hand-searching journals. Most filters were validated internally only. Reporting of methods was generally poor. Sensitivity, precision and specificity were the most commonly reported performance measures and were presented in tables. Aspects of DTA study methods are applicable to search filters, particularly in the development of the reference standard. There is limited evidence on how clinicians choose between diagnostic tests. No published literature was found on how searchers select filters. Interviewing and questioning searchers via a questionnaire found that filters were not appropriate for all tasks but were predominantly used to reduce large numbers of retrieved records and to introduce focus. The Inter Technology Appraisal Support Collaboration (InterTASC) Information Specialists' Sub-Group (ISSG) Search Filters Resource was most frequently mentioned by both groups as the resource consulted to select a filter. Randomised controlled trial (RCT) and systematic review filters, in particular the Cochrane RCT and the McMaster Hedges filters, were most frequently mentioned. The majority indicated that they used different filters depending on the requirement for sensitivity or precision. Over half of the respondents used the filters available in databases. Interviewees used various approaches when using and adapting search filters. Respondents suggested that the main factors that would make choosing a filter easier were the availability of critical appraisals and more detailed performance information. Provenance and having the filter available in a central storage location were also important. The questionnaire could have been shorter and could have included more multiple choice questions, and the reviews of filter performance focused on only four study designs. Search filter studies should use a representative reference standard and explicitly report methods and results. Performance measures should be presented systematically and clearly. Searchers find filters useful in certain circumstances but expressed a need for more user-friendly performance information to aid filter choice. We suggest approaches to use, adapt and report search filter performance. Future work could include research around search filters and performance measures for study designs not addressed here, exploration of alternative methods of displaying performance results and numerical synthesis of performance comparison results. The National Institute for Health Research (NIHR) Health Technology Assessment programme and Medical Research Council-NIHR Methodology Research Programme (grant number G0901496).