Assessment of Student Skills for Critiquing Published Primary Scientific Literature Using a Primary Trait Analysis Scale

Abstract

Instructor evaluation of progressive student skills in the analysis of primary literature is critical for the development of these skills in young scientists. Students in a senior or graduate-level one-semester course in Immunology at a Masters-level comprehensive university were assessed for abilities (primary traits) to recognize and evaluate the following elements of a scientific paper: Hypothesis and Rationale, Significance, Methods, Results, Critical Thinking and Analysis, and Conclusions. We tested the hypotheses that average recognition scores vary among elements and that scores change with time differently by trait. Recognition scores (scaled 1 to 5), and differences in scores were analyzed using analysis of variance (ANOVA), regression, and analysis of covariance (ANCOVA) (n = 10 papers over 103 days). By multiple comparisons testing, we found that recognition scores statistically fell into two groups: high scores (for Hypothesis and Rationale, Significance, Methods, and Conclusions) and low scores (for Results and Critical Thinking and Analysis). Recognition scores only significantly changed with time (increased) for Hypothesis and Rationale and Results. ANCOVA showed that changes in recognition scores for these elements were not significantly different in slope (F1,16 = 0.254, P = 0.621) but the Results trait was significantly lower in elevation (F1,17 = 12.456, P = 0.003). Thus, students improved with similar trajectories, but starting and ending with lower Results scores. We conclude that students have greatest difficulty evaluating Results and critically evaluating scientific validity. Our findings show extant student skills, and the significant increase in some traits shows learning. This study demonstrates that students start with variable recognition skills and that student skills may be learned at differential rates. Faculty can use these findings or the primary trait analysis scoring scale to focus on specific paper elements for which they desire to improve recognition.