Learning Analytics Interventions Research Articles

An Extended Learning Analytics Framework Integrating Machine Learning and Pedagogical Approaches for Student Performance Prediction and Intervention

AbstractThe use of educational data mining and machine learning to analyse large data sets collected by educational institutions has the potential to discover valuable insights for decision-making. One such area that has gained attention is to predict student performance by analysing large educational data sets. In the relevant literature, many studies have focused on developing prediction models on student performance but comparatively less work exists on actions taken based on predicted at-risk students and evaluating their impact. Learning Analytics Intervention (LAI) studies have emerged as an approach that aims to address this gap. In LAI studies, student risk levels are predicted and disseminated to relevant stakeholders (academics, administrators and students) using learning analytics (LA) tools for targeted interventions. The interventions themselves are mainly left under the discretion of the academics and/or administrators, who are aware of the learning context and have the authority to make decisions, with LA tools facilitating this process. LAI studies have shown success in improving outcomes (e.g. improve pass rates, retention, grades), but their uptake has been slow. The main impediment to piloting LAIs by academics has been the lack of access to LAI infrastructure, which requires institutional investments to develop predictive models collecting data from diverse IT systems. Another challenge in LAIs is the development of effective interventions. This paper presents an extended LAI framework, termed Student Performance Prediction and Action (SPPA), which provides access to LAI infrastructure for academics to pilot LAIs in their courses without the need for institution-wide efforts. SPPA and its features are seamlessly accessed via a web browser and academics can develop course-specific predictive models based on historical course assessment data. Furthermore, SPPA integrates sound pedagogical approaches and provides relevant information (such as students’ knowledge gaps, personalised study plans) to assist academics in providing effective interventions. SPPA was evaluated by a number of academics piloting LAIs in their courses. Quantitative and qualitative data was collected and analysed. Academics were able to provide effective interventions using SPPA and also had a positive outlook on using SPPA and its features. SPPA is also provided as an open-source project for further development and can be a catalyst for widescale uptake in LAIs. Furthermore, a model for continuous improvement in LAIs is outlined along with a number of areas for future research and development.

Sequence analysis and process mining perspectives to goal setting: What distinguishes business students with high and low self-efficacy beliefs?

This study investigates the relationship between students' self-efficacy beliefs, goal-setting, and learning tactics in an online business course. Using sequence analysis and process mining techniques, we analyzed log data from 209 students to identify distinct learning tactics and their association with self-efficacy beliefs, inferred from initial goal-setting and final grades. Four learning tactics were identified, with significant differences observed between students with high and low self-efficacy beliefs. High self-efficacy students demonstrated more varied and adaptive tactics, including greater use of quizzes and project-focused activities. In contrast, low self-efficacy students and those who didn't set goals showed less engagement and fewer monitoring activities. The project-focused tactic had the strongest correlation with final grades and goal achievement. Goal-setting at the course's start was linked to more effective learning behaviors and better outcomes. The study reveals how students' online learning behavior changes based on initial goal-setting and subsequent reflection. These findings contribute to research on self-regulated learning in online environments and offer practical implications for designing online courses and learning analytics interventions. Understanding these differences enables the development of targeted interventions to improve learning tactics and self-efficacy beliefs among students, ultimately enhancing their ability to achieve learning goals in online educational settings.

Evaluating the student performance prediction and action framework through a learning analytics intervention study

AbstractLearning analytics intervention (LAI) studies aim to identify at-risk students early during an academic term using predictive models and facilitate educators to provide effective interventions to improve educational outcomes. A major impediment to the uptake of LAI is the lack of access to LAI infrastructure by educators to pilot LAI, which typically requires substantial institution-wide efforts and investment to collect related data sets and develop accurate predictive models that identify at-risk students and also provide tools to facilitate interventions. This paper presents a novel LAI framework, termed Student Performance Prediction and Action (SPPA), that facilitates educators to seamlessly provide LAIs in their courses avoiding the need for large-scale institution-wide efforts and investments. Educators develop course-specific predictive models using historical course assessment data. In learning analytics, providing effective interventions is a challenge. SPPA utilises pedagogy principles in course design and interventions to facilitate effective interventions by providing insights into students’ risk levels, gaps in students’ knowledge, and personalised study/revision plans addressing knowledge gaps. SPPA was evaluated in a large undergraduate course on its ability to predict at-risk students and facilitate effective interventions as well as its ease of use by academics. The results are encouraging with high performance of predictive models, facilitating effective interventions leading to significant improved educational outcomes with positive feedback and uptake by academics. With its advantages, SPPA has the potential to catalyse and influence wide-scale adoption in LAIs.

Implementing learning analytics interventions to support student agency in knowledge building

ABSTRACT Research and development of Learning Analytics (LA) have created new ways to support students’ learning. However, our understanding of teachers’ roles when implementing LA in classroom practices remains nascent. This study investigates how teachers can implement LA to support students’ agency in directing their own inquiry, when engaging in a digital pedagogy approach known as Knowledge Building (KB) on an online platform called Knowledge Forum (KF). Together with a teacher experienced in KB and KF, we co-constructed interventions anchored on two easy-to-use LA tools in KF, the Scaffold Tracker and Word Cloud. These LA interventions were enacted with a Grade 5 class of 18 students and another Grade 6 class of 18 students. Three teacher roles were identified: detecting behaviour unproductive to collaboration, mediating between LA and student self-assessment, and framing student-generated lines of inquiry in actionable ways. We coded KF notes for the discourse moves they reflected and found that after the teacher implemented the LA interventions, notes that sustained inquiry increased sharply with students’ focus shifting from individual to collective knowledge and from superficial explanations to deep understandings. We discuss the implications of these findings on the development of ways to bridge LA research and practice.

Learning Analytics Intervention Using Prompts and Feedback for Measurement of e-Learners’ Socially-Shared Regulated Learning

The future of university learning in Sub-Saharan Africa has become increasingly digitally transformed by both e-Learning, and learning analytics, post-COVID-19 pandemic. Learning analytics intervention is critical for effective support of socially-shared regulated learning skills, which are crucial for twenty-first-century e-Learners. Socially-shared regulation is the major determinant of successful collaborative e-learning. However, most e-learners lack such skills thereby facing socio-cognitive challenges, due to the unavailability of intelligent support during learning. This research aims to investigate and understand the effect of Learning Analytics instructional support using feedback and prompts, on e-learners’ SSRL indicators. A theoretical model was derived from these factors and built from selected features. Both survey data and behavioral trace data were employed in the Learning analytics-based intervention. In this paper, only a segment of the data is discussed. The e-learners’ perceptions and feedback confirmed that Learning Analytics-based interventions using prompts and feedback are effective in promoting SSRL in collaborative e-learning contexts. The findings indicated that the success of SSRLA-based intervention be tied to support from instructors and academic counselors, particularly feedback on previous problems and quizzes. This will improve e-learners’ SSRL skills for quality educational experience, hence motivate e-learners, and help lecturers to identify at-risk learners in web programming problem-based courses. In conclusion, without adequate utilization of the Learning Analytics interventional trace data, critical information about learners’ behavior patterns in terms of their online interactivity with the course activities and their SSRL profiles and strategies cannot be disclosed leading to little improvement of e-Learning interventions.

A Systematic Review of Learning Analytics

The learning management system (LMS) is widely used in educational settings to support teaching and learning practices. The usage log data, generated by both learners and instructors, enables the development and implementation of learning analytics (LA) interventions aimed at facilitating teaching and learning activities. To examine the current status of the development and empirical impacts of learning analytics–incorporated interventions within LMSs on improving teaching and learning practices, we conducted a systematic review that examined 27 articles published from 2012 through 2023. The outcomes of this review provided valuable insights into the design and development of learning analytics–incorporated interventions implemented on LMSs and empirical evidence of the impacts of these interventions, along with implications to inform future design and applications.

I-Ntervene: applying an evidence-based learning analytics intervention to support computer programming instruction

Apart from good instructional design and delivery, effective intervention is another key to strengthen student academic performance. However, intervention has been recognized as a great challenge. Most instructors struggle to identify at-risk students, determine a proper intervention approach, trace and evaluate whether the intervention works. This process requires extensive effort and commitment, which is impractical especially for large classes with few instructors. This paper proposes a platform, namely i-Ntervene, that integrates Learning Management System (LMS) automatic code grader, and learning analytics features which can empower systematic learning intervention for large programming classes. The platform supports instructor-pace courses on both Virtual Learning Environment (VLE) and traditional classroom setting. The platform iteratively assesses student engagement levels through learning activity gaps. It also analyzes subject understanding from programming question practices to identify at-risk students and suggests aspects of intervention based on their lagging in these areas. Students’ post-intervention data are traced and evaluated quantitatively to determine effective intervention approaches. This evaluation method aligns with the evidence-based research design. The developed i-Ntervene prototype was tested on a Java programming course with 253 first-year university students during the Covid-19 pandemic in VLE. The result was satisfactory, as the instructors were able to perform and evaluate 12 interventions throughout a semester. For this experimental course, the platform revealed that the approach of sending extrinsic motivation emails had more impact in promoting learning behavior compared to other types of messages. It also showed that providing tutorial sessions was not an effective approach to improving students’ subject understanding in complex algorithmic topics. i-Ntervene allows instructors to flexibly trial potential interventions to discover the optimal approach for their course settings which should boost student’s learning outcomes in long term.

Adaptive or adapted to: Sequence and reflexive thematic analysis to understand learners' self‐regulated learning in an adaptive learning analytics dashboard

AbstractThe real‐time and granularized learning information and recommendations available from adaptive learning technology can provide learners with feedback that is personalized. However, at an individual level, learners often experience technological and pedagogical conflicts. Learners have more freedom to accept, ignore or reject the feedback while also having the challenges of building learning strategies and utilizing learning information that requires self‐regulated learning skills. Given the conflicts, both understanding how learners learn and providing support for learners to be more self‐regulated in the learning environment are imperative. This investigation explores how learners processed their learning in an adaptive technology‐integrated learning analytics dashboard (ALAD). It employed mixed‐methods using a lens of self‐regulated learning (SRL). Three groups were identified based on clustering analysis of the learners' usage of warm‐up (WU) tests. Sequence analysis revealed the time trends of each group's interactions with course content. Reflexive thematic analysis brought insights on how learners built their learning strategies (eg, ways of using WU tests and submodule assessments) and how they monitored and controlled their learning. It showed their dynamic interactions with core adaptive learning analytics dashboard elements. Challenges such as difficulties in rehearsing and monitoring through segmented course content arose from the new structural changes. We suggest the need of future improvement to individual learning support through the learning analytics dashboard to be more diverse and dynamic (real‐time) over the course of learning while reducing potential undesirable consequences. Practitioner notesWhat is already known about this topic One purpose of learning analytics and adaptive learning is to help learners identify learning goals and take action to achieve their goals. Learning analytics intervention showed support in learners' reflection phase SRL. However, it is not clear how to better support actionable and strategic changes to learning. Learning improvement through learning analytics interventions varies depending on how learners utilize feedback and monitor their learning progress, interacting with their digital learning environments. What this paper adds Learners established certain learning strategies with core adaptive learning analytics dashboard elements (eg, use of assessments, monitoring strategies). Learners' perceptions about learning support from the ALAD were built by interacting with the learners' task and cognitive conditions. Based on the perceptions, individuals' various SRL strategies and the need for diverse support were found. Monitoring and rehearsals can be challenging when course content is broken down for individuals' support. Implications for practice and/or policy Courses with adaptive learning analytics dashboards need to be designed more carefully, considering possible undesirable consequences, and to improve SRL support. Learners need more diverse learning support at an individual level based on how they interact with the learning environment.

Supporting self-regulated learning with learning analytics interventions – a systematic literature review

During the past years scholars have shown an increasing interest in supporting students' self-regulated learning (SRL). Learning analytics (LA) can be applied in various ways to identify a learner’s current state of self-regulation and support SRL processes. It is important to examine how LA has been used to identify the need for support in different phases of SRL cycle, which channels are used to mediate the intervention and how efficient and impactful the intervention is. This will help the learners to achieve the anticipated learning outcomes. The systematic literature review followed PRISMA 2020 statement to examine studies that applied LA interventions to enhance SRL. The search terms used for this research identified 753 papers in May 2021. Of these, 56 studies included the elements of LA, SRL, and intervention. The reviewed studies contained various LA interventions aimed at supporting SRL, but only 46% of them revealed a positive impact of an intervention on learning. Furthermore, only four studies reported positive effects for SRL and covered all three SRL phases (planning, performance, and reflection). Based on the findings of this literature review, the key recommendation is for all phases of SRL to be considered when planning interventions to support learning. In addition, more comparative research on this topic is needed to identify the most effective interventions and to provide further evidence on the effectiveness of interventions supporting SRL.

Personalized learning analytics intervention approach for enhancing student learning achievement and behavioral engagement in blended learning

Personalized learning analytics intervention approach for enhancing student learning achievement and behavioral engagement in blended learning

Exploring the Effectiveness of Learning Path Recommendation based on Felder-Silverman Learning Style Model: A Learning Analytics Intervention Approach

A fixed learning path for all learners is a major drawback of virtual learning systems. An online learning path recommendation system has the advantage of offering flexibility to select appropriate learning content. Learning Analytics Intervention (LAI) provides several educational benefits, particularly for low-performing students. Researchers employed an LAI approach in this work to recommend personalised learning paths to students pursuing online courses depending on their learning styles. It was accomplished by developing a Learning Path Recommendation Model (LPRM) based on the Felder–Silverman Learning Style Model (FSLSM) and evaluating its efficacy. The data were analysed with the help of a dataset from the Moodle Research Repository, and different learning paths were recommended using a sequence matching algorithm. The effectiveness of this approach was tested in two groups of learners using the independent two-sample t-test, a statistical testing tool. The experimental evaluation revealed that learners who followed the suggested learning path performed better than those who followed the learning path without any recommendations. This enhanced learning performance exemplifies the effects of learning analytics intervention .

Developing a Learning Analytics Intervention in E-learning to Enhance Students’ Learning Performance: A Case Study

Developing a Learning Analytics Intervention in E-learning to Enhance Students’ Learning Performance: A Case Study

About or with Teachers? A Systematic Review of Learning Analytics Interventions to Support Teacher Professional Development

Learning analytics (LA) allows aggregate data about users’ to support decision making. Although teachers are recognised as important stakeholders, little is yet explored about the role that LA can play in teacher professional development (TPD). This paper aimed to conduct a systematic review of the use of LA in TPD context, focusing specifically on intervention studies, classifying purposes and methods as well as beneficiaries’ engagement and lessons learned. Search terms identified 189 papers and 31 studies were selected based on the inclusion criteria. The results show that most studies adopted data-driven approaches to monitoring teacher behaviours, through automatic extraction of logs in technology-enhanced learning environments. The perspectives, benefits and limitations in the application of LA to TPD are finally presented.

To Opt in or to Opt Out? Predicting Student Preference for Learning Analytics-Based Formative Feedback

Teachers’ work is increasingly augmented with intelligent tools that extend their pedagogical abilities. While these tools may have positive effects, they require use of students’ personal data, and more research into student preferences regarding these tools is needed. In this study, we investigated how learning strategies and study engagement are related to students’ willingness to share data with learning analytics (LA) applications and whether these factors predict students’ opt-in for LA-based formative feedback. Students (N=158) on a self-paced online course set their personal completion goals for the course and chose to opt in for or opt out of personalized feedback based on their progress toward their goal. We collected self-reported measures regarding learning strategies, study engagement, and willingness to share data for learning analytics through a survey (N=73). Using a regularized partial correlation network, we found that although willingness to share data was weakly connected to different aspects of learning strategies and study engagement, students with lower self-efficacy were more hesitant to share data about their performance. Furthermore, we could not sufficiently predict students’ opt-in decisions based on their learning strategies, study engagement, or willingness to share data using logistic regression. Our findings underline the privacy paradox in online privacy behavior: theoretical unwillingness to share personal data does not necessarily lead to opting out of interventions that require the disclosure of personal data. Future research should look into why students opt in for or opt out of learning analytics interventions.

The challenge of learning analytics implementation: lessons learned


 
 
 Despite broad interest in learning analytics across the Australian Higher Education sector, there remains few examples of institution-wide implementations. Learning analytics implementation is currently under-theorised with little knowledge of the complexities that mediate the systemic uptake required for an institution-wide implementation. It has been established that approaches to learning analytics that are exclusively top-down or bottom-up, are insufficient for successful implementation across an enterprise. Drawing upon an award-winning and institution-wide learning analytics intervention that has been used across almost 5,000 unit offerings, this paper formulates an initial set of theory informed design principles that can help learning analytics practitioners mediate the complexities of institution-wide implementation.
 
 

Changing the recent past to reduce ongoing dropout: an early learning analytics intervention for an online statistics course

ABSTRACT Practitioners of the statistics course embedded in a computer science programme at a fully online university were concerned with the high dropout rate. In the academic year 2018–19, they decided to carry out a two-phase project in order to address this issue. In the first phase, an early classifier to identify students at risk of dropping out of the 2018–19 statistics course was determined. The second phase was planned to design and implement an early intervention based on the results of the first phase. This article presents the analysis of this intervention. In the 2019–20 online statistics course, 35 students did not submit the first few quizzes before the respective deadlines. They were the target of this intervention, which gave them the chance to change their recent past by submitting the unsubmitted quizzes. Students were encouraged to change by means of an email message. The aim of the intervention was to reduce the dropout rate among the targeted students. Our analysis also includes the students’ perception of the intervention collected via a set of interviews. The results show that the intervention clearly had a beneficial effect for some students.

Learning Analytics Intervention Improves Students’ Engagement in Online Learning

Learning Analytics Intervention Improves Students’ Engagement in Online Learning

Where is the Learning in Learning Analytics? A Systematic Literature Review on the Operationalization of Learning-Related Constructs in the Evaluation of Learning Analytics Interventions

Learning technologies enable interventions in the learning process aiming to improve learning. Learning analytics provides such interventions based on analysis of learner data, which are believed to have beneficial effects on both learning and the learning environment. Literature reporting on the effects of learning analytics interventions on learning allows us to assess in what way learning analytics improves learning. No standard set of operational definitions for learning affected by learning analytics interventions is available. We performed a systematic literature review of 1932 search hits, which yielded 62 key studies. We analyzed how affected learning was operationalized in these key studies and classified operational definitions into three categories: 1) learning environment ; 2) learning process; and 3) learning outcome . A deepening analysis yielded a refined classification scheme with 11 subcategories. Most of the analyzed studies relate to either learning outcome or learning process . Only nine of the key studies relate to more than one category. Given the complex nature of applying learning analytics interventions in practice, measuring the effects on a wider spectrum of aspects can give more insight into the workings of learning analytics interventions on the different actors, processes, and outcomes involved. Based on the results of our review, we recommend making deliberate decisions on the (multiple) aspects of learning one tries to improve by applying learning analytics. Our refined classification with examples of operational definitions may help both academics and practitioners doing so, as it allows for a more structured, grounded, and comparable positioning of learning analytics benefits.

A review of learning analytics intervention in higher education (2011–2018)

Intervention has long been practised in higher education to provide assistance for at-risk or underachieving learners. With the development of learning analytics, the delivery of intervention has been informed by data-driven approaches to identify learners’ problems and provide them with just-in-time and personalised support. However, intervention has been claimed to be the greatest challenge in learning analytics and has yet to be widely implemented. This paper reviews 24 case studies of learning analytics intervention in higher education. The cases were categorised and summarised according to their objectives, the data used, the intervention methods, the outcomes obtained and the challenges encountered. The results show that intervention practices have focused most frequently on increasing students’ study performance, offering personalised feedback and improving student retention. The frequent types of data involved students’ online learning behaviours, study performance, demographics and course selection information. The most commonly used intervention methods involved offering personalised recommendations and visualising learning data. The interventions have led to outcomes such as enhancing study performance, retention and course registration, as well as productivity and effectiveness in learning and teaching. The challenges covered a wide range of aspects, including the scalability of intervention, conditions for implementing intervention, limitations of the channels for delivering intervention and the evaluation of intervention effectiveness. The results suggest that learning analytics intervention has the potential to further extend its scope of practices to serve a wider range of purposes, but more studies on the empirical evidence, even with null or negative results, are needed to support its long-term effectiveness and sustainability.

The efficacy of learning analytics interventions in higher education: A systematic review

AbstractEducational institutions are increasingly turning to learning analytics to identify and intervene with students at risk of underperformance or discontinuation. However, the extent to which the current evidence base supports this investment is currently unclear, and particularly so in relation to the effectiveness of interventions based on predictive models. The aim of the present paper was to conduct a systematic review and quality assessment of studies on the use of learning analytics in higher education, focusing specifically on intervention studies. Search terms identified 689 papers, but only 11 studies evaluated the effectiveness of interventions based on learning analytics. These studies highlighted the potential of such interventions, but the general quality of the research was moderate, and left several important questions unanswered. The key recommendation based on this review is that more research into the implementation and evaluation of scientifically driven learning analytics is needed to build a solid evidence base for the feasibility, effectiveness and generalizability of such interventions. This is particularly relevant when considering the increasing tendency of educational institutions around the world to implement learning analytics interventions with only little evidence of their effectiveness. Practitioner NotesWhat is already known about this topic? Drop‐out rates and underachivement is a significant issue at most Western universities. Learning analytics have been shown to predict student performance and risk of dropping out. Interventions based on learning analytics have emerged in recent years, some reportedly successful. What this paper adds The paper also reviews and synthesizes the evidence on the effectiveness of learning analytics interventions targeting student underperformance, experience and discontinuation. The paper compares and contrasts past and current learning analytics methods and foci, and makes recommendations for the future research and practice. It critically synthesizes the current evidence base on learning analytics interventions, which is a field that is in constant flux and development. Implications for practice and/or policy The paper focuses on an increasing part of higher education with the goal of validating learning analytics methods and usefulness. The paper makes evidence‐based recommendations for institutions wishing to implement learning analytics programs and/or interventions. The paper makes evidence‐based recommendations for instructors as well as researchers in the field.