Premium LiteratUre Service Research Articles

Translating Clinical Questions by Physicians Into Searchable Queries: Analytical Survey Study.

BackgroundStaying up to date and answering clinical questions with current best evidence from health research is challenging. Evidence-based clinical texts, databases, and tools can help, but clinicians first need to translate their clinical questions into searchable queries. MacPLUS FS (McMaster Premium LiteratUre Service Federated Search) is an online search engine that allows clinicians to explore multiple resources simultaneously and retrieves one single output that includes the following: (1) evidence from summaries (eg, UpToDate and DynaMed), (2) preappraised research (eg, EvidenceAlerts), and (3) non-preappraised research (eg, PubMed), with and without validated bibliographic search filters. MacPLUS FS can also be used as a laboratory to explore clinical questions and evidence retrieval.ObjectiveOur primary objective was to examine how clinicians formulate their queries on a federated search engine, according to the population, intervention, comparison, and outcome (PICO) framework. Our secondary objective was to assess which resources were accessed by clinicians to answer their questions.MethodsWe performed an analytical survey among 908 clinicians who used MacPLUS FS in the context of a randomized controlled trial on search retrieval. Recording account log-ins and usage, we extracted all 1085 queries performed during a 6-month period and classified each search term according to the PICO framework. We further categorized queries into background (eg, “What is porphyria?”) and foreground questions (eg, “Does treatment A work better than B?”). We then analyzed the type of resources that clinicians accessed.ResultsThere were 695 structured queries, after exclusion of meaningless queries and iterations of similar searches. We classified 56.5% (393/695) of these queries as background questions and 43.5% (302/695) as foreground questions, the majority of which were related to questions about therapy (213/695, 30.6%), followed by diagnosis (48/695, 6.9%), etiology (24/695, 3.5%), and prognosis (17/695, 2.5%). This distribution did not significantly differ between postgraduate residents and medical faculty physicians (P=.51). Queries included a median of 3 search terms (IQR 2-4), most often related to the population and intervention or test, rarely related to the outcome, and never related to the comparator. About half of the resources accessed (314/610, 51.5%) were summaries, 24.4% (149/610) were preappraised research, and 24.1% were (147/610) non-preappraised research.ConclusionsOur results, from a large sample of real-life queries, could guide the development of educational interventions to improve clinicians’ retrieval skills, as well as inform the design of more useful evidence-based resources for clinical practice.Trial RegistrationClinicalTrials.gov NCT02038439; https://www.clinicaltrials.gov/ct2/show/NCT02038439

Efficiency of pragmatic search strategies to update clinical guidelines recommendations

BackgroundA major challenge in updating clinical guidelines is to efficiently identify new, relevant evidence. We evaluated the efficiency and feasibility of two new approaches: the development of restrictive search strategies using PubMed Clinical Queries for MEDLINE and the use of the PLUS (McMaster Premium Literature Service) database.MethodsWe evaluated a random sample of recommendations from a national guideline development program and identified the references that would potentially trigger an update (key references) using an exhaustive approach.We designed restrictive search strategies using the minimum number of Medical Subject Headings (MeSH) terms and text words required from the original exhaustive search strategies and applying broad and narrow filters. We developed PLUS search strategies, matching Medical Subject Headings (MeSH) and Systematized Nomenclature of Medicine (SNOMED) terms with guideline topics. We compared the number of key references retrieved by these approaches with those retrieved by the exhaustive approach.ResultsThe restrictive approach retrieved 68.1 % fewer references than the exhaustive approach (12,486 versus 39,136), and identified 89.9 % (62/69) of key references and 88 % (22/25) of recommendation updates. The use of PLUS retrieved 88.5 % fewer references than the exhaustive approach (4,486 versus 39,136) and identified substantially fewer key references (18/69, 26.1 %) and fewer recommendation updates (10/25, 40 %).ConclusionsThe proposed restrictive approach is a highly efficient and feasible method to identify new evidence that triggers a recommendation update. Searching only in the PLUS database proved to be a suboptimal approach and suggests the need for topic-specific tailoring.Electronic supplementary materialThe online version of this article (doi:10.1186/s12874-015-0058-2) contains supplementary material, which is available to authorized users.

Strategies to assess the validity of recommendations: a study protocol

BackgroundClinical practice guidelines (CPGs) become quickly outdated and require a periodic reassessment of evidence research to maintain their validity. However, there is little research about this topic. Our project will provide evidence for some of the most pressing questions in this field: 1) what is the average time for recommendations to become out of date?; 2) what is the comparative performance of two restricted search strategies to evaluate the need to update recommendations?; and 3) what is the feasibility of a more regular monitoring and updating strategy compared to usual practice?. In this protocol we will focus on questions one and two.MethodsThe CPG Development Programme of the Spanish Ministry of Health developed 14 CPGs between 2008 and 2009. We will stratify guidelines by topic and by publication year, and include one CPG by strata.We will develop a strategy to assess the validity of CPG recommendations, which includes a baseline survey of clinical experts, an update of the original exhaustive literature searches, the identification of key references (reference that trigger a potential recommendation update), and the assessment of the potential changes in each recommendation.We will run two alternative search strategies to efficiently identify important new evidence: 1) PLUS search based in McMaster Premium LiteratUre Service (PLUS) database; and 2) a Restrictive Search (ReSe) based on the least number of MeSH terms and free text words needed to locate all the references of each original recommendation.We will perform a survival analysis of recommendations using the Kaplan-Meier method and we will use the log-rank test to analyse differences between survival curves according to the topic, the purpose, the strength of recommendations and the turnover. We will retrieve key references from the exhaustive search and evaluate their presence in the PLUS and ReSe search results.DiscussionOur project, using a highly structured and transparent methodology, will provide guidance of when recommendations are likely to be at risk of being out of date. We will also assess two novel restrictive search strategies which could reduce the workload without compromising rigour when CPGs developers check for the need of updating.

023 Feasibility and Efficiency of Strategies For Updating Clinical Practice Guidelines

BackgroundExhaustive search strategies (ESS) for updating clinical practice guidelines (CPGs) recommendations are laborious and expensive. Highly sensitive and specific alternative search strategies are necessary to improve the efficiency in recommendations...

How Current Are Leading Evidence-Based Medical Textbooks? An Analytic Survey of Four Online Textbooks

BackgroundThe consistency of treatment recommendations of evidence-based medical textbooks with more recently published evidence has not been investigated to date. Inconsistencies could affect the quality of medical care.ObjectiveTo determine the frequency with which topics in leading online evidence-based medical textbooks report treatment recommendations consistent with more recently published research evidence.MethodsSummarized treatment recommendations in 200 clinical topics (ie, disease states) covered in four evidence-based textbooks–UpToDate, Physicians’ Information Education Resource (PIER), DynaMed, and Best Practice–were compared with articles identified in an evidence rating service (McMaster Premium Literature Service, PLUS) since the date of the most recent topic updates in each textbook. Textbook treatment recommendations were compared with article results to determine if the articles provided different, new conclusions. From these findings, the proportion of topics which potentially require updating in each textbook was calculated.Results478 clinical topics were assessed for inclusion to find 200 topics that were addressed by all four textbooks. The proportion of topics for which there was 1 or more recently published articles found in PLUS with evidence that differed from the textbooks’ treatment recommendations was 23% (95% CI 17-29%) for DynaMed, 52% (95% CI 45-59%) for UpToDate, 55% (95% CI 48-61%) for PIER, and 60% (95% CI 53-66%) for Best Practice (χ 2 3=65.3, P<.001). The time since the last update for each textbook averaged from 170 days (range 131-209) for DynaMed, to 488 days (range 423-554) for PIER (P<.001 across all textbooks).ConclusionsIn online evidence-based textbooks, the proportion of topics with potentially outdated treatment recommendations varies substantially.

Research Quality and Newsworthiness of Published Articles are Partial Predictors of Journal Impact Factors

Objective – Determine what characteristics of a journal’s published articles can be used to predict the journal impact factor (JIF).&#x0D; &#x0D; Design – A retrospective cohort study.&#x0D; &#x0D; Setting – The researchers are located at McMaster University, Hamilton, Ontario, Canada.&#x0D; &#x0D; Subjects – The sample consisted of 1,267 clinical research articles from 103 evidence based and clinical journals which were published in 2005 and indexed in the McMaster University Premium LiteratUre Service (PLUS) database and those same journals’ JIF from 2007.&#x0D; &#x0D; Method – The articles were divided 60:40 into a derivation set (760 articles and 99 journals) and a validation set (507 articles and 88 journals). Ten variables which could influence JIF were developed and a multiple linear regression was run on the derivation set and then applied to the validation set.&#x0D; &#x0D; Main Results – The four variables found to be significant were the number of databases which indexed the journal, the number of authors, the quality of research, and the “newsworthiness” of the journal’s published articles.&#x0D; &#x0D; Conclusion – The quality of research and newsworthiness at time of publication of a journal’s articles can predict the journal impact factor with 60% accuracy.

McMaster Premium LiteratUre Service (PLUS) performed well for identifying new studies for updated Cochrane reviews

ObjectiveWe compared the performance of McMaster Premium LiteratUre Service (PLUS) and Clinical Queries (CQs) to that of the Cochrane Controlled Trials Register, MEDLINE, and EMBASE for locating studies added during an update of reviews. Study Design and SettingA sample of new studies in updated Cochrane systematic reviews was used as a reference standard. Searches were performed for each study in each database. Where a new study was not indexed in PLUS, we examined the effect on the review of excluding the study. ResultsNinety-eight updated Cochrane reviews were identified. For the 87 reviews with a usable meta-analysis, PLUS contained all new studies for 13 reviews. No statistically significant difference between PLUS and non-PLUS new studies was found when ratio of odds ratios (RORs) were pooled across 39 reviews (ROR⊕/⊖: 0.99; 95% confidence interval: 0.87–1.14). Thirty-five updated reviews had no new studies indexed in PLUS, but conclusions were seldom altered by addition of new studies. ConclusionsPLUS included less than a quarter of the new studies in Cochrane updates, but most reviews appeared unaffected by the omission of these studies. Reviewers should consider adopting PLUS and CQ filters to improve the efficiency of keeping their reviews up to date.

Systematic reviews and original articles differ in relevance, novelty, and use in an evidence-based service for physicians: PLUS project

Objectives To describe the ratings from physicians, and use by physicians, of high quality, clinically pertinent original articles and systematic reviews from over 110 clinical journals and the Cochrane Database of Systematic Reviews (CDSRs). Study Design and Setting Prospective observational study. Data were collected via an online clinical rating system of relevance and newsworthiness for quality-filtered clinical articles and via an online delivery service for practicing physicians, during the course of the McMaster Premium LiteratUre Service Trial. Clinical ratings of articles in the MORE system by over 1,900 physicians were compared and the usage rates over 13 months of these articles by physicians, who were not raters, were examined. Results Systematic reviews were rated significantly higher than original articles for relevance ( P < 0.001), but significantly lower for newsworthiness ( P < 0.001). Reviews published in the CDSR had significantly lower ratings for both relevance ( P < 0.001) and newsworthiness ( P < 0.001) than reviews published in other journals. Participants accessed reviews more often than original articles ( P < 0.001), and accessed reviews from journals more often than from CDSR ( P < 0.001). Conclusion Physician ratings and the use of high-quality original articles and systematic reviews differed, generally favoring systematic reviews over original articles. Reviews published in journals were rated higher and accessed more often than Cochrane reviews.

McMaster PLUS: A Cluster Randomized Clinical Trial of an Intervention to Accelerate Clinical Use of Evidence-based Information from Digital Libraries

BackgroundPhysicians have difficulty keeping up with new evidence from medical research. MethodsWe developed the McMaster Premium LiteratUre Service (PLUS), an internet-based addition to an existing digital library, which delivered quality- and relevance-rated medical literature to physicians, matched to their clinical disciplines. We evaluated PLUS in a cluster-randomized trial of 203 participating physicians in Northern Ontario, comparing a Full-Serve version (that included alerts to new articles and a cumulative database of alerts) with a Self-Serve version (that included a passive guide to evidence-based literature). Utilization of the service was the primary trial end-point. ResultsMean logins to the library rose by 0.77 logins/month/user (95% CI 0.43, 1.11) in the Full-Serve group compared with the Self-Serve group. The proportion of Full-Serve participants who utilized the service during each month of the study period showed a sustained increase during the intervention period, with a relative increase of 57% (95% CI 12, 123) compared with the Self-Serve group. There were no differences in these proportions during the baseline period, and following the crossover of the Self-Serve group to Full-Serve, the Self-Serve group’s usage became indistinguishable from that of the Full-Serve group (relative difference 4.4 (95% CI −23.7, 43.0). Also during the intervention and crossover periods, measures of self-reported usefulness did not show a difference between the 2 groups. ConclusionA quality- and relevance-rated online literature service increased the utilization of evidence-based information from a digital library by practicing physicians.

Second-Order Peer Review of the Medical Literature for Clinical Practitioners

Most articles in clinical journals are not appropriate for direct application by individual clinicians. To create a second order of clinical peer review for journal articles to determine which articles are most relevant for specific clinical disciplines. A 2-stage prospective observational study in which research staff reviewed all issues of over 110 (number has varied slightly as new journals were added or discarded from review but number has always been over 110) clinical journals and selected each article that met critical appraisal criteria from January 2003 through the present. Practicing physicians were recruited from around the world, excluding Northern Ontario, to the McMaster Online Rating of Evidence (MORE) system and registered as raters according to their clinical disciplines. An automated system assigned each qualifying article to raters for each pertinent clinical discipline, and recorded their online assessments of the articles on 7-point scales (highest score, 7) of relevance and newsworthiness (defined as useful new information for physicians). Rated articles fed an online alerting service, the McMaster Premium Literature Service (PLUS). Physicians from Northern Ontario were invited to register with PLUS and then receive e-mail alerts about articles according to MORE system peer ratings for their own discipline. Online access by PLUS users of PLUS alerts, raters' comments, article abstracts, and full-text journal articles was automatically recorded. Clinical rater recruitment and performance. Relevance and newsworthiness of journal articles to clinical practice in the discipline of the rating physician. Through October 2005, MORE had 2139 clinical raters, and PLUS had 5892 articles with 45 462 relevance ratings and 44 724 newsworthiness ratings collected since 2003. On average, clinicians rated systematic review articles higher for relevance to practice than articles with original evidence and lower for useful new information. Primary care physicians rated articles lower than did specialists (P<.05). Of the 98 physicians who registered for PLUS, 88 (90%) used it on 3136 occasions during an 18-month test period. This demonstration project shows the feasibility and use of a post-publication clinical peer review system that differentiates published journal articles according to the interests of a broad range of clinical disciplines.