Research Study Quality Instrument Score Research Articles

An analysis of vascular surgery education publications reveals an educational shortage

BackgroundAs vascular surgery training continues to evolve with the growth of integrated vascular surgery residency (0+5) programs and the consolidation of fellowship programs, optimizing all aspects of the education for vascular surgery trainees, both fellows and 0+5 residents, has become increasingly important. In the present study, we aimed to determine the prevalence, quality, and content of vascular surgery education publications across journals. MethodsJournal websites (n = 26) and PubMed were queried to identify vascular surgery education publications from 2012 to 2021. The publications were organized into 11 content categories: (1) curriculum, (2) simulation, (3) trainee assessment, (4) program evaluation, (5) wellness/burnout, (6) diversity/inclusion, (7) mentorship/career, (8) case outcomes, (9) perceptions of training, (10) social media, and (11) other. Publication interactivity and quality were measured via PlumX data and Medical Education Research Study Quality Instrument scores. The data were analyzed via univariate analysis and linear regression. ResultsA total of 115 vascular surgery education publications (0.2% [interquartile range (IQR), 0.04%-0.5%] of total publications) were identified from the selected journals. The Journal of Vascular Surgery had the highest proportion (0.8%) of vascular surgery education publications, followed by the Journal of Surgical Education (0.7%) and Annals of Vascular Surgery (0.6%). Vascular surgery journals constituted most (79%) of the publications. Of the authors, 15% (IQR, 0%-25%) had a master's or doctorate degree in education. Senior authors were more often identified as male gender (77%), and more first authors (41%) were identified as female gender. An interactivity analysis showed that there were 10.3 citations (IQR, 12), 33.1 captures (IQR, 34), and 8.4 social media interactions (IQR 14) per publication. The educational quality had a median Medical Education Research Study Quality Instrument score of 11 (IQR, 9-12.5), with 49% of publications having a score greater than the median. Publications on training (44% curriculum and 20% simulation) were significantly more frequent than other topics (P < .001), with no change in the publication content over 10 years (P = .29). The volume of vascular surgery education publications did not change during the study period (P = .13) despite the ongoing changes in the educational environment. ConclusionsDespite the increasing importance placed on vascular surgery education by national vascular societies, publications on vascular surgery education have remained sparse among all journals. Also, the vascular surgery educational content has not changed during the past 10 years, with a primary focus on curriculum and simulation training. Further promotion of vascular surgery educational research is required to increase the quality, volume, and diversity of education publications.

Validity Evidence for Procedure-specific Competence Assessment Tools in General Surgery: A Scoping Review.

This study aims to provide a focused and detailed assessment of the validity evidence supporting procedure-specific operative assessment tools in general surgery. Competency-based assessment tools should be supported by robust validity evidence to be used reliably for evaluation of operative skills. The contemporary framework of validity relies on five sources of evidence: content, response process, internal structure, relation to other variables, and consequences. A systematic search of 8 databases was conducted for studies containing procedure-specific operative assessment tools in general surgery. The validity evidence supporting each tool was assessed and scored in alignment with the contemporary framework of validity. Methodological rigour of studies was assessed with the Medical Education Research Study Quality Instrument. The educational utility of each tool was assessed with the Accreditation Council for Graduate Medical Education framework. There were 28 studies meeting inclusion criteria and 23 unique tools were assessed. Scores for validity evidence varied widely between tools, ranging from 3 - 14 (maximum 15). Medical Education Research Study Quality Instrument scores assessing the quality of study methodology were also variable (8.5-15.5, maximum 16.5). Direct reporting of educational utility criteria was limited. This study has identified a small group of procedure-specific operative assessment tools in general surgery. Many of these tools have limited validity evidence and have not been studied sufficiently to be used reliably in high-stakes summative assessments. As general surgery transitions to competency-based training, a more robust library of operative assessment tools will be required to support resident education and evaluation.

Toward Competency-Based Training: To What Extent Are We Competency-Based?

Time-based training models in plastic surgery vary in exposure, resulting in low confidence levels among graduates. The evolution of postgraduate medical education into a competency-based model to address these issues requires an understanding of interventions described in the plastic surgery literature to identify gaps and guide creation of assessments to demonstrate competence. A systematic search of the MEDLINE, Embase, Cumulative Index to Nursing and Allied Health Literature, PubMed, and Cochrane databases from inception until December of 2017 was conducted using search terms and synonyms of educational interventions reported in plastic surgery. Full texts were retrieved following filtering and data extracted were related to intervention design and execution, involvement of competency assessment, and educational objectives and alignment to Accreditation Council for Graduate Medical Education competencies and Royal College of Physicians and Surgeons of Canada Canadian Medical Education Directives for Specialists roles. Study quality was assessed using Kirkpatrick's levels of learning evaluation, validity evidence, and the Medical Education Research Study Quality Instrument score. Of the initial 4307 results, only 36 interventions met the inclusion criteria. Almost all interventions aligned to medical knowledge and patient care Accreditation Council for Graduate Medical Education competencies. One-fifth of the interventions involved no assessment of competency, whereas most displayed assessment at the level of design as opposed to outcomes. Quality assessment revealed low levels of learning evaluation and evidence of validity; the average Medical Education Research Study Quality Instrument score was 10.9 of 18. A systematic review of educational literature in plastic surgery was conducted to assess the quality of reported educational interventions, and to help guide creating tools that ensure competency acquirement among trainees.

An evaluation of educational interventions aimed at preparing medical students for discharge summary writing: a rapid review of the literature

Whilst discharge summaries are important for ensuring patient continuity of care, they often lack important information. Medical students and newly qualified doctors have attributed this to insufficient teaching, thereby indicating a need to improve discharge summary education within the undergraduate medical curriculum. The aim was to review educational approaches used to prepare undergraduate medical students for discharge summary writing, and their effectiveness Medline and Scopus were systematically searched using keywords, for studies published between 2009 and 2019. Narrative synthesis was used to integrate the findings of the included studies. Study quality was appraised using the Medical Education Research Study Quality Instrument. Six studies were included in this review. Allowing students to write a discharge summary following a real patient encounter and receive written feedback was the most common educational approach used (n = 4). Three of these studies improved student attitudes towards discharge summary writing. One study described the use of a mobile application, which improved discharge summary quality. Another study utilized interprofessional learning but the effect was statistically insignificant. Medical Education Research Study Quality Instrument scores were satisfactory (mean = 12, range = 8.4-15). The literature concerning this topic is limited. The results indicate that providing written feedback to students can positively influence attitudes towards discharge summary writing. Optimisation of the quality of discharge summaries written by medical students using feedback and checklists requires further investigation, as does the use of interprofessional learning. Future studies would additionally benefit from including participant baseline data and control groups, being multicentred and measuring behavioural or patient/healthcare outcomes.

Utility of Supplemental Training to Improve Radiologist Performance in Breast Cancer Screening: A Literature Review

PurposeThe authors evaluate whether supplemental training for radiologists improves their breast screening performance and how this is measured. MethodsA systematic search was conducted in PubMed on August 3, 2017. Articles were included if they described supplemental training for radiologists reading mammograms to improve their breast screening performance and at least one outcome measure was reported. Study quality was assessed using the Medical Education Research Study Quality Instrument. ResultsOf 2,199 identified articles, 18 were included, of which 17 showed improvement on at least one of the outcome measures, for at least one training activity or subgroup. Two measurement approaches were found. For the first approach, measuring performance on test sets, sensitivity, and specificity were the most reported outcomes (8 of 11 studies). Recall rate is the most reported outcome (6 of 7 studies) for the second approach, which measures performance in actual screening practice. The studies were mainly of moderate quality (Medical Education Research Study Quality Instrument score 11.7 ± 1.7), caused by small sample sizes and the lack of a control group. ConclusionsSupplemental training helps radiologists improve their screening performance, despite the mainly moderate quality of the studies. There is a need for better designed studies. Future studies should focus on performance in actual screening practice and should look for methods to isolate the training effect. If test sets are used, focus should be on knowledge about correlation between performance on test sets and actual screening practice.

The Current State of the 0+5 Integrated Vascular Surgery Residency Training Paradigm: A Scoping Review of the Literature

The 0+5 integrated vascular surgery residency program (IVSR) was introduced as a training model toward board certification in vascular surgery over 10 years ago. The pros and cons of this training model have been debated. The purpose of this review is to investigate, using qualitative methods, what is known about the development, implementation, and outcomes of the IVSR paradigm. A systematic search of the literature pertaining to the IVSR training model was conducted to include literature from 2005 to 2016. A search strategy involving use of 4 literature databases, 4 search terms, and 4 inclusion criteria was used. Three independent reviewers screened titles and abstracts for inclusion. Data abstraction was performed by 1 reviewer. Qualitative content analysis was completed using the method of constant comparative analysis associated with a grounded theory design by all 3 reviewers. Of 890 articles initially identified, 33 articles were found to meet inclusion criteria for full review. Nineteen (57%) were research articles with an average Medical Education Research Study Quality Instrument score of 6.3 out of 18. The remaining articles were categorized as editorials, presidential addresses, invited commentaries, and historical summaries. Three major themes related to the IVSR program were identified: context of program development, processes of the program once implemented, and outcomes. The literature on the IVSR paradigm reflects contextual, process, and outcome issues. Research articles are of generally low quality and there is a paucity of analyses of outcome issues. Further research is recommended to identify and understand the outcomes of the model.

Substance Use Education in Schools of Nursing: A Systematic Review of the Literature.

The authors systematically reviewed the nursing literature for articles describing substance use disorders (SUDs) education in schools of nursing. Five literature databases were searched, producing 3107 retrieved articles, of which 12 were included in this review. A Medical Education Research Study Quality Instrument score was calculated for each study. The included studies demonstrated that teaching nursing students about SUDs produced a positive impact on their attitudes, knowledge, and skills.

Simulation in Interprofessional Clinical Education

The research literature regarding interprofessional simulation-based medical education has grown substantially and continues to explore new aspects of this educational modality. The aim of this study was to explore the validation evidence of tools used to assess teamwork and nontechnical skills in interprofessional simulation-based clinical education. This systematic review included original studies that assessed participants' teamwork and nontechnical skills, using a measurement tool, in an interprofessional simulated setting. We assessed the validity of each assessment tool using Kane's framework. Medical Education Research Study Quality Instrument scores for the studies ranged from 8.5 to 17.0. Across the 22 different studies, there were 20 different assessment strategies, in which Team Emergency Assessment Measure, Anesthetist's Nontechnical Skills, and Nontechnical Skills for Surgeons were used more than once. Most assessment tools have been validated for scoring and generalization inference. Fewer tools have been validated for extrapolation inference, such as expert-novice analysis or factor analysis.

Neglect in Undergraduate Medical Education: A Systematic Review

OBJECTIVE: To describe and compare the definitions of neglect used in the medical education literature and the estimated prevalence of neglect reported on clinical rotations. DATA SOURCES: EMBASE and Ovid MEDLINE were searched using these terms: lack of appreciation, lack of guidance, ignored, passed over, unintentional mistreat, unintentional mistreatment, missed learning opportunity, environment-based mistreatment, insidious mistreatment, marginalized learner, neglect, social dominance, medical education, undergraduate medical education, medical student. METHODS OF STUDY SELECTION: Non-English language articles, editorials, commentaries, and letters were excluded using search filters or by hand. Publications were selected for full-text review if they commented on medical students’ perceptions of being ignored, neglected, mistreated, or experiencing an unwelcoming learning environment. Further screening of 1,698 identified only three that focused specifically on neglect. Three additional studies were identified from reference lists or findings of prior literature searches. TABULATION, INTEGRATION AND RESULTS: The six publications included in the review reported the results of three qualitative and three cross-sectional studies. Medical Education Research Study Quality Instrument scores for the cross-sectional studies were low due to limitations in study design: single institution, no comparison group, and unknown validity of evaluation instruments. A discrete definition of neglect was not found, but several authors described a spectrum including “covert, status related abuse” and “environment-based mistreatment” to describe this phenomenon. CONCLUSION: Poor study quality and missing or inconsistent definitions and measures limit the validity of efforts to estimate the prevalence of neglect on different clinical rotations for medical students.

A systematic review of serious games in asthma education.

Serious games may be useful tools for asthma education. The objectives of this systematic review were to identify the available articles on serious games designed to educate patients and the general public about asthma and to assess their impact on patient's knowledge, behavior, and clinical outcomes related to asthma. PubMed, EMBASE, Cochrane Library, PsychInfo, and Web of Science were systematically searched from January 1980 to December 2015 for studies investigating serious games in asthma education. Two investigators independently assessed studies against inclusion criteria and rated those included on indicators of quality. Investigators extracted data on serious games' content and learning objectives, and on outcomes following Kirkpatrick classification. A total of 12 articles were found to be relevant, describing a total of 10 serious games. All serious games were directed toward children, with eight games for children with asthma and two for school-based intervention. The average Medical Education Research Study Quality Instrument score was 13.9 of 18, which is high. Most of the serious games were associated with high rates of satisfaction and improvement in children's knowledge. Seven studies evaluated the impact of serious games on clinical outcomes and found no significant difference relative to control groups. Although serious games designed for asthma education have evolved with advances in technology, results of their evaluation remained similar across studies, with clear improvements in knowledge but little or no change in behaviors and clinical outcomes.

The use of simulation in neurosurgical education and training. A systematic review.

There is increasing evidence that simulation provides high-quality, time-effective training in an era of resident duty-hour restrictions. Simulation may also permit trainees to acquire key skills in a safe environment, important in a specialty such as neurosurgery, where technical error can result in devastating consequences. The authors systematically reviewed the application of simulation within neurosurgical training and explored the state of the art in simulation within this specialty. To their knowledge this is the first systematic review published on this topic to date. The authors searched the Ovid MEDLINE, Embase, and PsycINFO databases and identified 4101 articles; 195 abstracts were screened by 2 authors for inclusion. The authors reviewed data on study population, study design and setting, outcome measures, key findings, and limitations. Twenty-eight articles formed the basis of this systematic review. Several different simulators are at the neurosurgeon's disposal, including those for ventriculostomy, neuroendoscopic procedures, and spinal surgery, with evidence for improved performance in a range of procedures. Feedback from participants has generally been favorable. However, study quality was found to be poor overall, with many studies hampered by nonrandomized design, presenting normal rather than abnormal anatomy, lack of control groups and long-term follow-up, poor study reporting, lack of evidence of improved simulator performance translating into clinical benefit, and poor reliability and validity evidence. The mean Medical Education Research Study Quality Instrument score of included studies was 9.21 ± 1.95 (± SD) out of a possible score of 18. The authors demonstrate qualitative and quantitative benefits of a range of neurosurgical simulators but find significant shortfalls in methodology and design. Future studies should seek to improve study design and reporting, and provide long-term follow-up data on simulated and ideally patient outcomes.

Predictive Validity Evidence for Medical Education Research Study Quality Instrument Scores: Quality of Submissions to JGIM’s Medical Education Special Issue

BackgroundDeficiencies in medical education research quality are widely acknowledged. Content, internal structure, and criterion validity evidence support the use of the Medical Education Research Study Quality Instrument (MERSQI) to measure education research quality, but predictive validity evidence has not been explored.ObjectiveTo describe the quality of manuscripts submitted to the 2008 Journal of General Internal Medicine (JGIM) medical education issue and determine whether MERSQI scores predict editorial decisions.Design and ParticipantsCross-sectional study of original, quantitative research studies submitted for publication.MeasurementsStudy quality measured by MERSQI scores (possible range 5–18).ResultsOf 131 submitted manuscripts, 100 met inclusion criteria. The mean (SD) total MERSQI score was 9.6 (2.6), range 5–15.5. Most studies used single-group cross-sectional (54%) or pre-post designs (32%), were conducted at one institution (78%), and reported satisfaction or opinion outcomes (56%). Few (36%) reported validity evidence for evaluation instruments. A one-point increase in MERSQI score was associated with editorial decisions to send manuscripts for peer review versus reject without review (OR 1.31, 95%CI 1.07–1.61, p = 0.009) and to invite revisions after review versus reject after review (OR 1.29, 95%CI 1.05–1.58, p = 0.02). MERSQI scores predicted final acceptance versus rejection (OR 1.32; 95% CI 1.10–1.58, p = 0.003). The mean total MERSQI score of accepted manuscripts was significantly higher than rejected manuscripts (10.7 [2.5] versus 9.0 [2.4], p = 0.003).ConclusionsMERSQI scores predicted editorial decisions and identified areas of methodological strengths and weaknesses in submitted manuscripts. Researchers, reviewers, and editors might use this instrument as a measure of methodological quality.Electronic supplementary materialThe online version of this article (doi:10.1007/s11606-008-0664-3) contains supplementary material, which is available to authorized users.