Developing and Validating an Instrument for Assessing Learning Sciences Competence of Doctoral Students in Education in China

Abstract

Learning science competence refers to a necessary professional competence for educators, which is manifested in their deep understanding of learning science knowledge, positive attitudes, and scientific thinking and skills in conducting teaching practice and research. It is of paramount importance for doctoral students in education to develop their competence in the field of learning sciences. This will enhance their abilities to teach and conduct research, and guide their educational research and practice toward greater sustainability. In order to address the shortcomings of current assessment instruments, we constructed a theoretical model for assessing learning sciences competence based on the PISA 2025 framework and Piaget’s theory of knowledge. A three-dimensional assessment framework was designed, along with an initial instrument. Furthermore, the “Delphi method based on large language models (LLM)” was employed to conduct two rounds of expert consultations with the objective of testing and refining the instrument. Throughout this process, we developed a set of guidelines for engaging AI experts to improve interactions with LLM, including an invitation letter to AI experts, the main body of the questionnaire, and the general inquiry about AI experts’ perspectives. In analyzing the results of the Delphi method, we used the “threshold method” to identify and refine the questionnaire items that performed sub-optimally. This resulted in the final assessment instrument for evaluating learning sciences competence among doctoral students in education. The assessment instrument encompasses three dimensions: the knowledge of learning sciences, application of learning sciences, and attitude towards learning sciences, with a total of 40 items. These items integrate Likert scales and scenario-based questions. Furthermore, the study examined potential limitations in the item design, question type selection, and method application of the assessment instrument. The design and development of the assessment instrument provide valuable references for the standardized monitoring and sustainability development of the learning sciences competence of doctoral students in education.