Abstract
Physics education researchers (PER) commonly use complete-case analysis to address missing data. For complete-case analysis, researchers discard all data from any student who is missing any data. Despite its frequent use, no PER article we reviewed that used complete-case analysis provided evidence that the data met the assumption of missing completely at random (MCAR) necessary to ensure accurate results. Not meeting this assumption raises the possibility that prior studies have reported biased results with inflated gains that may obscure differences across courses. To test this possibility, we compared the accuracy of complete-case analysis and multiple imputation (MI) using simulated data. We simulated the data based on prior studies such that students who earned higher grades participated at higher rates, which made the data missing at random (MAR). PER studies seldom use MI, but MI uses all available data, has less stringent assumptions, and is more accurate and more statistically powerful than complete-case analysis. Results indicated that complete-case analysis introduced more bias than MI and this bias was large enough to obscure differences between student populations or between courses. We recommend that the PER community adopt the use of MI for handling missing data to improve the accuracy in research studies.
Highlights
Physics education research (PER) commonly handles missing data by using complete-case analysis [1,2]
If the results indicate that complete-case analysis provides inaccurate results compared to multiple imputation (MI), these results could motivate researchers to use MI in their studies
We designed the study to cover a broad range of variables we identified as pertinent to concept inventory data
Summary
Physics education research (PER) commonly handles missing data by using complete-case analysis ( known as listwise deletion, casewise deletion, and matched data) [1,2]. Complete-case analysis removes any individuals who are missing any data from the analysis. This method is common because it is easy to implement. Complete-case analysis produces reliable results so long as the missing data is missing completely at random (MCAR) [3]. For MCAR, the missingness is completely independent of any observed or missing data [7]. We are not aware of any studies in PER that have explicitly tested the MCAR assumption. The few studies that have explicitly compared participants and non-participants using course grades [2,10,11,12] all indicate that students with higher course grades are more likely to provide complete data
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have