Assessment of Educational Research Literacy in Higher Education: Construct validation of the factorial structure of an assessment instrument comparing different treatments of omitted responses

Abstract

The ability to purposefully access, reflect, and use evidence from educational research (Educational Research Literacy) are key competencies of future professionals in educational practice. A test instrument was developed to assess Educational Research Literacy with the competence facets Information Literacy, Statistical Literacy, and Evidence-based Reasoning. Even though there are certain overlaps with generic concepts like critical thinking or problem solving, Educational Research Literacy is acquired within its reference disciplines. This contribution aimed to delve deeper into the question which factorial model is most appropriate. Four competing models were compared: unidimensional, three-dimensional, and two bifactor models. The comparison was based on a study of 1360 students at six German universities and was validated by another study of 753 students at three universities. The results also were examined relative to the scoring of omitted responses and the booklet design used in the first study. The results indicate that the four-dimensional bifactor model was the most appropriate: Educational Research Literacy seems to consist of one dominant factor and three secondary factors. The results also support handling both omitted and not-reached responses as missing information. Subsequently, the results are critically discussed relative to the requirements for assessing and for imparting competencies in higher education. Recommendations for future research are stated.