Learning Analytics Community Research Articles

Curriculum Analytics of Course Choices:

In a higher education context, students are expected to take charge of their learning by deciding “what” to learn and “how” to learn. While the learning analytics (LA) community has seen increasing research on the “how” to learn part (i.e., researching methods for supporting students in their learning journey), the “what” to learn part is still underinvestigated. We present a case study of curriculum analytics and its application to a dataset of 243 students of the bachelor’s program in the broad discipline of health sciences to explore the effects of course choices on students’ academic performance. Using curriculum metrics such as grading stringency, course temporal position, and duration, we investigated how course choices differed between high- and low-performing students using both temporal and sequential analysis methods. We found that high-performing students were likely to pick an elective course of low difficulty. It appeared that these students were more strategic in terms of their course choices than their low-performing peers. Generally, low-performing students seemed to have made suboptimal choices when selecting elective courses; e.g., when they picked an elective course of high difficulty, they were less likely to pick a following course of low difficulty. The findings of this study have design implications for researchers, program directors, and coordinators, because they can use the results to (i) update the course sequencing, (ii) guide students about course choices based on their current GPA (such as through course recommendation dashboards), (iii) identify bottleneck courses, and (iv) assist higher education institutions in planning a more balanced course roadmap to help students manage their workload effectively.

LAK of Direction

Learning analytics defines itself with a focus on data from learners and learning environments, with corresponding goals of understanding and optimizing student learning. In this regard, learning analytics research, ideally, should be characterized by studies that make use of data from learners engaged in education systems, should measure student learning, and should make efforts to intervene and improve these learning environments. However, a common concern among members of the learning analytics research community is that these standards are not being met. In two analysis waves, we review a large and comprehensive sample of research articles from the proceedings of the three most recent Learning Analytics and Knowledge conferences, the premier conference venue for learning analytics research, and from articles published during the same time in the Journal of Learning Analytics (over the years of 2020, 2021, and 2022). We find that 37.4% of articles do not analyze data from learners in an education system, 71.1% do not include any measure of learning, and 89.0% of articles do not attempt to intervene in the learning environment. We contrast these findings with the stated definition of learning analytics and infer, like others before us, that scholarship in learning analytics research presently lacks clear direction toward its stated goals. We invite critical discussion of these findings from the learning analytics community, through open peer commentary.

A systematic review of empirical studies using log data from open‐ended learning environments to measure science and engineering practices

AbstractTechnology‐based, open‐ended learning environments (OELEs) can capture detailed information of students' interactions as they work through a task or solve a problem embedded in the environment. This information, in the form of log data, has the potential to provide important insights about the practices adopted by students for scientific inquiry and problem solving. How to parse and analyse the log data to reveal evidence of multifaceted constructs like inquiry and problem solving holds the key to making interactive learning environments useful for assessing students' higher‐order competencies. In this paper, we present a systematic review of studies that used log data generated in OELEs to describe, model and assess scientific inquiry and problem solving. We identify and analyse 70 conference proceedings and journal papers published between 2012 and 2021. Our results reveal large variations in OELE and task characteristics, approaches used to extract features from log data and interpretation models used to link features to target constructs. While the educational data mining and learning analytics communities have made progress in leveraging log data to model inquiry and problem solving, multiple barriers still exist to hamper the production of representative, reproducible and generalizable results. Based on the trends identified, we lay out a set of recommendations pertaining to key aspects of the workflow that we believe will help the field develop more systematic approaches to designing and using OELEs for studying how students engage in inquiry and problem‐solving practices. Practitioner notesWhat is already known about this topic Research has shown that technology‐based, open‐ended learning environments (OELEs) that collect users' interaction data are potentially useful tools for engaging students in practice‐based STEM learning. More work is needed to identify generalizable principles of how to design OELE tasks to support student learning and how to analyse the log data to assess student performance. What this paper adds We identified multiple barriers to the production of sufficiently generalizable and robust results to inform practice, with respect to: (1) the design characteristics of the OELE‐based tasks, (2) the target competencies measured, (3) the approaches and techniques used to extract features from log files and (4) the models used to link features to the competencies. Based on this analysis, we can provide a series of specific recommendations to inform future research and facilitate the generalizability and interpretability of results: Making the data available in open‐access repositories, similar to the PISA tasks, for easy access and sharing. Defining target practices more precisely to better align task design with target practices and to facilitate between‐study comparisons. More systematic evaluation of OELE and task designs to improve the psychometric properties of OELE‐based measurement tasks and analysis processes. Focusing more on internal and external validation of both feature generation processes and statistical models, for example with data from different samples or by systematically varying the analysis methods. Implications for practice and/or policy Using the framework of evidence‐centered assessment design, we have identified relevant criteria for organizing and evaluating the diverse body of empirical studies on the topic and that policy makers and practitioners can use for their own further examinations. This paper identifies promising research and development areas on the measurement and assessment of higher‐order constructs with process data from OELE‐based tasks that government agencies and foundations can support. Researchers, technologists and assessment designers might find useful the insights and recommendations for how OELEs can enhance science assessment through thoughtful integration of learning theories, task design and data mining techniques.

A New Era in Multimodal Learning Analytics: Twelve Core Commitments to Ground and Grow MMLA

Multimodal learning analytics (MMLA) has increasingly been a topic of discussion within the learning analytics community. The Society of Learning Analytics Research is home to the CrossMMLA Special Interest Group and regularly hosts workshops on MMLA during the Learning Analytics Summer Institute (LASI). In this paper, we articulate a set of 12 commitments that we believe are critical for creating effective MMLA innovations. Moreover, as MMLA grows in use, it is important to articulate a set of core commitments that can help guide both MMLA researchers and the broader learning analytics community. The commitments that we describe are deeply rooted in the origins of MMLA and also reflect the ways that MMLA has evolved over the past 10 years. We organize the 12 commitments in terms of (i) data collection, (ii) analysis and inference, and (iii) feedback and data dissemination and argue why these commitments are important for conducting ethical, high-quality MMLA research. Furthermore, in using the language of commitments, we emphasize opportunities for MMLA research to align with established qualitative research methodologies and important concerns from critical studies.

Bridging the Gap Between Theory and Empirical Research in Evaluative Judgment

The value of students developing the capacity to accurately judge the quality of their work and that of others has been widely studied and recognized in higher education literature. To date, much of the research and commentary on evaluative judgment has been theoretical and speculative in nature, focusing on perceived benefits and proposing strategies seen to hold the potential to foster evaluative judgment. The efficacy of the strategies remains largely untested. The rise of educational tools and technologies that generate data on learning activities at an unprecedented scale, alongside insights from the learning sciences and learning analytics communities, provides new opportunities for fostering and supporting empirical research on evaluative judgment. Accordingly, this paper offers a conceptual framework and an instantiation of that framework in the form of an educational tool called RiPPLE for data-driven approaches to investigating the enhancement of evaluative judgment. Two case studies, demonstrating how RiPPLE can foster and support empirical research on evaluative judgment, are presented.

Learning Analytics for Professional and Workplace Learning: A Literature Review

Despite the ubiquity of learning in workplace and professional settings, the learning analytics (LA) community has paid significant attention to such settings only recently. This may be due to the focus on researching formal learning, as workplace learning is often informal, hard to grasp, and not unequivocally defined. This article summarizes the state of the art of workplace learning analytics (WPLA), extracted from a two-iteration systematic literature review. Our in-depth analysis of 52 existing proposals not only provides a descriptive view of the field, but also reflects on researcher conceptions of learning and their influence on the design, analytics, and technology choices made in this area. We also discuss the characteristics of workplace learning that make WPLA proposals different from LA in formal education contexts and the challenges resulting from this. We found that WPLA is gaining momentum, especially in some fields, like healthcare and education. The focus on theory is generally a positive feature in WPLA, but we encourage a stronger focus on assessing the impact of WPLA in realistic settings.

Predicting Student Academic Performance by Means of Associative Classification

The Learning Analytics community has recently paid particular attention to early predict learners’ performance. An established approach entails training classification models from past learner-related data in order to predict the exam success rate of a student well before the end of the course. Early predictions allow teachers to put in place targeted actions, e.g., supporting at-risk students to avoid exam failures or course dropouts. Although several machine learning and data mining solutions have been proposed to learn accurate predictors from past data, the interpretability and explainability of the best performing models is often limited. Therefore, in most cases, the reasons behind classifiers’ decisions remain unclear. This paper proposes an Explainable Learning Analytics solution to analyze learner-generated data acquired by our technical university, which relies on a blended learning model. It adopts classification techniques to early predict the success rate of about 5000 students who were enrolled in the first year courses of our university. It proposes to apply associative classifiers at different time points and to explore the characteristics of the models that led to assign pass or fail success rates. Thanks to their inherent interpretability, associative models can be manually explored by domain experts with the twofold aim at validating classifier outcomes through local rule-based explanations and identifying at-risk/successful student profiles by interpreting the global rule-based model. The results of an in-depth empirical evaluation demonstrate that associative models (i) perform as good as the best performing classification models, and (ii) give relevant insights into the per-student success rate assignments.

Learning Analytics: Pathways to Impact

The field of learning analytics has evolved over the past decade to provide new ways to view, understand and enhance learning activities and environments in higher education. It brings together research and practice traditions from multiple disciplines to provide an evidence base to inform student support and effective design for learning. This has resulted in a plethora of ideas and research exploring how data can be analysed and utilised to not only inform educators, but also to drive online learning systems that offer personalised learning experiences and/or feedback for students. However, a core challenge that the learning analytics community continues to face is how the impact of these innovations can be demonstrated. Where impact is positive, there is a case for continuing or increasing the use of learning analytics, however, there is also the potential for negative impact which is something that needs to be identified quickly and managed. As more institutions implement strategies to take advantage of learning analytics as part of core business, it is important that impact can be evaluated and addressed to ensure effectiveness and sustainability. In this editorial of the AJET special issue dedicated to the impact of learning analytics in higher education, we consider what impact can mean in the context of learning analytics and what the field needs to do to ensure that there are clear pathways to impact that result in the development of systems, analyses, and interventions that improve the educational environment.

Fostering An Impactful Field of Learning Analytics

Over 2019 the breadth of articles published in the Journal of Learning Analytics reflects the diversity of contributions to the field, including quantitative, qualitative and design-based studies and contributions through the different submission types (Research, Practical, Data and Tools, and Book Review). This year we have also published two special sections. The first is a set of empirical papers related to the emerging area of Human-Centered Learning Analytics. In this section, five sets of authors each address the core challenge of how to design and implement learning analytics in ways that are people- rather than technology-centric to achieve impact in the field. The second is a new format for the journal, an invited dialogue around a critical community issue. In this section Neil Selwyn asks the intentionally provocative question “what’s the problem with learning analytics?”, which is engaged with by four respondents from the community, representing a variety of different perspectives. Both of these special sections tackle, in their own way, a central issue for the learning analytics community: how we ensure that the work we do has positive impact in the world.

Book Review: Learning Analytics Explained by Niall Sclater

Sclater’s book Learning Analytics Explained is not unique in its goal to present a practitioner audience with a synopsis of the “what and how” of learning analytics. Throughout LA’s brief history, studies and guides have been produced recommending models and approaches for learning analytics “deployment,” as well as other short commentaries providing an overview of what exactly learning analytics is. These resources have been intended to increase awareness of the field across research and practice, and to facilitate sustainable uptake. Yet, and in spite of the high profile of these guides, the learning analytics community still finds itself lamenting a perceived low uptake of learning analytics within the higher education sector. Sclater, himself a prominent scholar and consultant in the field of learning analytics implementations, is mindful of this present juncture in learning analytics. His observations of the education sector’s rapidly growing interest in learning analytics, alongside a concomitant lack of understanding of what learning analytics actually is and could be, acted as motivations for him to write what he hopes is “a readable and informative summary of the area.”

Understanding Time-Based Trends in Stakeholders’ Choice of Learning Activity Type Using Predictive Models

The learning analytics communities, as well as most learning analytics research, have not frequently focused on time-based trends in the same virtual learning environment over different years of deployment, or on temporal trends in the selection of different activity types over a typical day. This paper contributes to this debate and provides a novel approach to learning analytics using a multinomial logit model for modeling the probabilities of students’ choice of learning activities during the hours of the day over several academic years. An abstraction called activity is introduced, which categorizes individual student’s log accesses to the virtual learning environment into more semantically meaningful categories. Consequently, the activity represents a sequence of semantically meaningful Web accesses related to a particular activity or task that a student of the virtual learning environment performs. This paper includes a comprehensive explanation of the model and an evaluation of the model. This paper introduces a case study, which shows that the multinomial logit model can give useful insight into the course schedule, as it shows what the peak times are for different types of activities. This paper also discusses the possible implications of the results in the context of virtual learning environment management and content improvement at the institutional level.

OnTask: Delivering Data-Informed, Personalized Learning Support Actions

The learning analytics community has matured significantly over the past few years as a middle space where technology and pedagogy combine to support learning experiences. To continue to grow and connect these perspectives, research needs to move beyond the level of basic support actions. This means exploring the use of data to prove richer forms of actions, such as personalized feedback, or hybrid approaches where instructors interpret the outputs of algorithms and select an appropriate course of action. This paper proposes the following three contributions to connect data extracted from the learning experience with such personalized student support actions: 1) a student–instructor centred conceptual model connecting a representation of the student information with a basic set of rules created by instructors to deploy Personalized Learning Support Actions (PLSAs); 2) a software architecture based on this model with six categories of functional blocks to deploy the PLSAs; and 3) a description of the implementation of this architecture as an open-source platform to promote the adoption and exploration of this area.

Bridging the gap between Networked Learning and Learning Analytics

Learning Analytics is a field that is constantly evolving, encompassing new perspectives and expanding into other research fields. This workshop was inspired by the 2012 call for action from Rebecca Ferguson in which she called for building stronger connections to the learning sciences (Ferguson, 2012). Although this call has not remained unanswered, we still find this connection to be insufficient to foster the development of technological solutions needed in the future. This is where the Networked Learning community comes into play. During this workshop, we wish to discover, reflect on and build connections joining the two fields, in the hope that a better understanding of what Networked Learning can offer the Learning Analytics community and vice versa will foster beneficial and productive collaborations in the future.

Learning Analytics Mobile Application for the Bucharest University of Economic Studies’ Students

IntroductionIn the last years there was an increasingly interest in personal and professional development and acquirement of as many information as possible, especially by development of alternative ways of study like online courses or training sessions for free. It is harder to keep the students interested when they have access to a lot of information interactively presented. So, it was necessary to develop a solution to provide proactive feedback and predict success or failure of a student in order to guide the learning process in such a way that offer the necessary information and maintain the students focused.This methodology called learning analytics implies the measurement and analysis of data about learners aimed at optimizing the process of learning, providing better information and a deep understanding of the field of interest for those involved. [1]Of course, there are many other ways to solve the problems in this area, reforming the learning process and introducing interactive teaching methods being only a small part of what can be done. However, it is very important to know the main cause and the factors that led to that problem in order to solve it, so learning analytics came to show the elements essentially to know and the main directions to the solution, making it the most suitable response. Just as it happens in any methodology, there are some limitations for learning analytics. According to [2], some of the limitations for learning analytics are:* Information belongs to different systems and there is not a unique system;* Education-technology industry has not enough standards and is not cooperating;* The governments don't focus on structures and focus on content;* Researchers don't focus on the processes in an ecosystem and focus on algorithms or teaching;* The target of the tools and reports is not the professional capital and capacity building;* In the relation between governments, school, leadership, teachers, students and the community does not exist enough trust.Taking these into account, there is a necessity to develop a solution to solve the main problems connected to learning process. Because of limitations of learning analytics, especially regarding a unified education system for all the institutions, we have focused only on students from Bucharest University of Economic Studies by developing an app for their needs. This software solution is designed to communicate with SIMUR database and to provide some key functionalities for every student, beyond their year of study or their faculty. Learning analytics means various benefits for all stakeholders so this application provides several reports and predictions to help both students and teachers by improving the learning process.As innovations, the current software solution comes with specific elements in order to sup- port students by providing methods of efficient time planning such as Gantt charts or predictions for specific facts as chances for a scholarship.2Related WorkLearning analytics became an important topic when referring to teaching and learning process, to students' success and to new methods of teaching and how useful they are. Frequently, papers related to learning analytics are focusing on what students will know at the end of the learning process and the elements that have an influence to this aspect, but, in order to understand and to use at the proper level the power of learning analytics a deeper look is needed.An analysis of data in the teaching process was done even before the widespread use of technology in order to obtain a feedback about the work done in the previous period, these calculations being done by hand. After that, learning analytics came to automate the process and to take care of all the data.Learning analytics has contributed very much in growth of effectiveness in the education field. Research has begun to be more intense in the learning analytics community after the appearance of massive online courses, online video-based learning, educational applications and the high availability of all kinds of computing devices. …

How Flexible Is Your Data? A Comparative Analysis of Scoring Methodologies across Learning Platforms in the Context of Group Differentiation

Data is flexible in that it is molded by not only the features and variables available to a researcher for analysis and interpretation, but also by how those features and variables are recorded and processed prior to evaluation. “Big Data” from online learning platforms and intelligent tutoring systems is no different. The work presented herein questions the quality and flexibility of data from two popular learning platforms, comparing binary measures of problem-level accuracy, the scoring method typically used to inform learner analytics, with partial credit scoring, a more robust, real-world methodology. This work extends previous research by examining how the manipulation of scoring methodology has the potential to alter outcomes when testing hypotheses, or specifically, when looking for significant differences between groups of students. Datasets from ASSISTments and Cognitive Tutor are used to assess the implications of data availability and manipulation within twelve mathematics skills. A resampling approach is used to determine the size of equivalent samples of high- and low-performing students required to reliably differentiate performance when considering each scoring methodology. Results suggest that in eleven out of twelve observed skills, partial credit offers more efficient group differentiation, increasing analytic power and reducing Type II error. Alternative applications of this approach and implications for the Learning Analytics community are discussed.

Understanding the mediating role of teacher inquiry when connecting learning analytics with design for learning

Discussion amongst the learning analytics community is for the need to link learning analytics with learning design. However, learning design is a complex space, and not all approaches to design for learning are the same. With this proposed integrated practice comes unique challenges for the contemporary educator. In this paper, we focus on the underlying processes of teacher inquiry when connecting design for learning with learning analytics. We propose that valuable connection between learning analytics and design for learning is only realised through the mediation of effective teacher inquiry processes. Hence, we aim to better understand these processes in order to identify teachers’ developmental needs. To this end, we propose a working model and examine learners (as educators) inquiry process in a professional learning workshop. We conclude with implications for the proposed model and professional learning.

A Pattern Language Based Learning Design Studio for an Analytics Informed Inter-Professional Design Community

Over the past ten years, a lot of advances have been made in both the learning design and learning analytics research communities in terms of research outputs. On the other hand, these advances have not made visible impact on the everyday practices of the key stakeholders they aim to serve: teachers and learners. This paper argues that to advance both fields and to connect them into an iterative cycle of continuous improvement, we need a learning design pattern language in the Alexandrian tradition to guide learning design practice, and to connect designers’ intuitive formulation of questions about learning processes and outcomes with evidence-based questions and operationalizable questions. It further describes the pattern language and the online tool, Learning Design Studio that the authors have developed to serve as an organically extensible language and a technology platform to support learning design, and inter-professional collaboration among learning designer and learning analytics communities.

Editorial: Datasets for Learning Analytics

The European LinkedUp and LACE (Learning Analytics Community Exchange) project have been responsible for setting up a series of data challenges at the LAK conferences 2013 and 2014 around the LAK dataset. The LAK datasets consists of a rich collection of full text publications in the domain of Learning Analytics and Educational Data Mining. The LAK dataset offers publicly available, machine-readable versions of research articles from the Learning Analytics and Educational Data Mining communities in various formats, where the main goal is to facilitate research, analysis, and smart explorative applications. Based on the insights gained from these data challenges, the idea was born to make more Learning Analytics data sets publicly available for researchers to get to a more open access data-driven research community within Learning Analytics. With this special section, we publish four data sets that answered the call for data sets by the journal. It is our vision to collect more data sets like these in this initial collection and reward their creators through citing the datasets and connecting new research outcomes to them.

Guest Editorial: Ethics and Privacy in Learning Analytics

The European Learning Analytics Community Exchange (LACE) project is responsible for an ongoing series of workshops on ethics and privacy in learning analytics (EP4LA), which have been responsible for driving and transforming activity in these areas. Some of this activity has been brought together with other work in the papers that make up this special issue. These papers cover the creation and development of ethical frameworks, as well as tools and approaches that can be used to address issues of ethics and privacy. This editorial suggests that it is worth taking time to consider the often intertangled issues of ethics, data protection and privacy separately. The challenges mentioned within the special issue are summarised in a table of 22 challenges that are used to identify the values that underpin work in this area. Nine ethical goals are suggested as the editors’ interpretation of the unstated values that lie behind the challenges raised in this paper.

Ethics and Privacy as Enablers of Learning Analytics

This issue of the Journal of Learning Analytics features a special section on ethics and privacy that is guest edited by a team of researchers involved in the European Learning Analytics Community Exchange (LACE) project. The issue also features a paper that looks at the use of new methods for the measurement of self-regulated learning. This editorial concludes with a summary of the future changes in the editorial team of the journal.