In Reply to Yates.

Abstract

We appreciate Yates’ insights on the need for longitudinal measures of self-regulated learning strategies, metacognition, and critical thinking, and we want to explain why we only longitudinally measured the growth of medical students’ knowledge. We agree that collecting multiple measures of these 3 concepts, serially or only at the beginning and the end of the study period, can provide illuminating results. However, a couple of concerns caused us to drop the idea to simplify the model. One is that modeling the growth of the student performance as the outcome measure was the focus of our study. Including multiple measures of the 3 scales and the growth pattern of each concept may have blurred the focus. The other is the sample size of the study. We had only a moderate sample size available and needed to consider whether the parameter estimate could achieve adequate power (> .80) in the complex model. We are glad that the importance of different students’ growth patterns was pointed out in Yates’ letter. Identifying different types of growth patterns and comparing the evolution of self-regulated learning strategies with the growth patterns in medical knowledge would be illuminating. But this model, the growth mixture model, requires a larger sample size to achieve adequate power. The higher the number of hypothesized growth patterns, the larger the required sample size. Standard practice is to use 10 times the number of measured indicators 1 multiplied by the number of growth patterns as the minimum sample size. Therefore, pooling comparable data from multiple institutions would be needed to make this study plausible. In addition, we would need to have a well-developed scale/instrument to provide comparable scaled scores when measuring growth across different time points. Studies examining the psychometric properties of the 3 scales in longitudinal settings are needed first. We have an unbiased estimate only when we have psychometric evidence to justify and validate that the change scores among repeated measures are not due to memory. As Yates pointed out, whether students change their self-regulated strategies or not, those longitudinal patterns’ relationships with the students’ growth patterns can be enlightening. Longitudinal assessment data provide great potential in facilitating the medical education study over time. We hope this continuing discussion leads to improved quality of assessments and encourages more institutions to work together.