Aligning objectives with assessment in online courses: Integrating learning analytics and measurement theory

Abstract

Well-designed online courses enhance learning experiences and allow effective development of learners' skills and knowledge. A critical factor contributing to the design of online courses in the higher education settings are well-defined learning objectives that align with course assessments and learning activities. While there are several introspective instruments to evaluate course designs, with the broader adoption of educational technologies and digital tools, there is a wealth of data that offers insights on the alignment of learning objectives to assessments. Such data has paved the way for evidence-based methods of investigating course effectiveness within higher education. This study outlines a methodology for designing and evaluating the alignment between course learning objectives and assessment activities at scale, utilising a combination of learning analytics and measurement theory approaches, more specificially exploratory multi-dimensional item response theory (MIRT) models. We demonstrate the proposed methodology within a professional development MOOC on leadership skills development, where we evaluate the alignemnt between course objectives and reflective writing assessments activities. Our results suggested that the alignment of the existing course objectives to assessment activities can be improved, showing the practical value of the proposed approach. The theoretical and practical implications of this research are further illustrated.