Learning Analytics Interventions Research Articles

A Systematic Review of Learning Analytics

The learning management system (LMS) is widely used in educational settings to support teaching and learning practices. The usage log data, generated by both learners and instructors, enables the development and implementation of learning analytics (LA) interventions aimed at facilitating teaching and learning activities. To examine the current status of the development and empirical impacts of learning analytics–incorporated interventions within LMSs on improving teaching and learning practices, we conducted a systematic review that examined 27 articles published from 2012 through 2023. The outcomes of this review provided valuable insights into the design and development of learning analytics–incorporated interventions implemented on LMSs and empirical evidence of the impacts of these interventions, along with implications to inform future design and applications.

I-Ntervene: applying an evidence-based learning analytics intervention to support computer programming instruction

Apart from good instructional design and delivery, effective intervention is another key to strengthen student academic performance. However, intervention has been recognized as a great challenge. Most instructors struggle to identify at-risk students, determine a proper intervention approach, trace and evaluate whether the intervention works. This process requires extensive effort and commitment, which is impractical especially for large classes with few instructors. This paper proposes a platform, namely i-Ntervene, that integrates Learning Management System (LMS) automatic code grader, and learning analytics features which can empower systematic learning intervention for large programming classes. The platform supports instructor-pace courses on both Virtual Learning Environment (VLE) and traditional classroom setting. The platform iteratively assesses student engagement levels through learning activity gaps. It also analyzes subject understanding from programming question practices to identify at-risk students and suggests aspects of intervention based on their lagging in these areas. Students’ post-intervention data are traced and evaluated quantitatively to determine effective intervention approaches. This evaluation method aligns with the evidence-based research design. The developed i-Ntervene prototype was tested on a Java programming course with 253 first-year university students during the Covid-19 pandemic in VLE. The result was satisfactory, as the instructors were able to perform and evaluate 12 interventions throughout a semester. For this experimental course, the platform revealed that the approach of sending extrinsic motivation emails had more impact in promoting learning behavior compared to other types of messages. It also showed that providing tutorial sessions was not an effective approach to improving students’ subject understanding in complex algorithmic topics. i-Ntervene allows instructors to flexibly trial potential interventions to discover the optimal approach for their course settings which should boost student’s learning outcomes in long term.

Adaptive or adapted to: Sequence and reflexive thematic analysis to understand learners' self‐regulated learning in an adaptive learning analytics dashboard

AbstractThe real‐time and granularized learning information and recommendations available from adaptive learning technology can provide learners with feedback that is personalized. However, at an individual level, learners often experience technological and pedagogical conflicts. Learners have more freedom to accept, ignore or reject the feedback while also having the challenges of building learning strategies and utilizing learning information that requires self‐regulated learning skills. Given the conflicts, both understanding how learners learn and providing support for learners to be more self‐regulated in the learning environment are imperative. This investigation explores how learners processed their learning in an adaptive technology‐integrated learning analytics dashboard (ALAD). It employed mixed‐methods using a lens of self‐regulated learning (SRL). Three groups were identified based on clustering analysis of the learners' usage of warm‐up (WU) tests. Sequence analysis revealed the time trends of each group's interactions with course content. Reflexive thematic analysis brought insights on how learners built their learning strategies (eg, ways of using WU tests and submodule assessments) and how they monitored and controlled their learning. It showed their dynamic interactions with core adaptive learning analytics dashboard elements. Challenges such as difficulties in rehearsing and monitoring through segmented course content arose from the new structural changes. We suggest the need of future improvement to individual learning support through the learning analytics dashboard to be more diverse and dynamic (real‐time) over the course of learning while reducing potential undesirable consequences. Practitioner notesWhat is already known about this topic One purpose of learning analytics and adaptive learning is to help learners identify learning goals and take action to achieve their goals. Learning analytics intervention showed support in learners' reflection phase SRL. However, it is not clear how to better support actionable and strategic changes to learning. Learning improvement through learning analytics interventions varies depending on how learners utilize feedback and monitor their learning progress, interacting with their digital learning environments. What this paper adds Learners established certain learning strategies with core adaptive learning analytics dashboard elements (eg, use of assessments, monitoring strategies). Learners' perceptions about learning support from the ALAD were built by interacting with the learners' task and cognitive conditions. Based on the perceptions, individuals' various SRL strategies and the need for diverse support were found. Monitoring and rehearsals can be challenging when course content is broken down for individuals' support. Implications for practice and/or policy Courses with adaptive learning analytics dashboards need to be designed more carefully, considering possible undesirable consequences, and to improve SRL support. Learners need more diverse learning support at an individual level based on how they interact with the learning environment.

Supporting self-regulated learning with learning analytics interventions – a systematic literature review

During the past years scholars have shown an increasing interest in supporting students' self-regulated learning (SRL). Learning analytics (LA) can be applied in various ways to identify a learner’s current state of self-regulation and support SRL processes. It is important to examine how LA has been used to identify the need for support in different phases of SRL cycle, which channels are used to mediate the intervention and how efficient and impactful the intervention is. This will help the learners to achieve the anticipated learning outcomes. The systematic literature review followed PRISMA 2020 statement to examine studies that applied LA interventions to enhance SRL. The search terms used for this research identified 753 papers in May 2021. Of these, 56 studies included the elements of LA, SRL, and intervention. The reviewed studies contained various LA interventions aimed at supporting SRL, but only 46% of them revealed a positive impact of an intervention on learning. Furthermore, only four studies reported positive effects for SRL and covered all three SRL phases (planning, performance, and reflection). Based on the findings of this literature review, the key recommendation is for all phases of SRL to be considered when planning interventions to support learning. In addition, more comparative research on this topic is needed to identify the most effective interventions and to provide further evidence on the effectiveness of interventions supporting SRL.

Personalized learning analytics intervention approach for enhancing student learning achievement and behavioral engagement in blended learning

Personalized learning analytics intervention approach for enhancing student learning achievement and behavioral engagement in blended learning

Exploring the Effectiveness of Learning Path Recommendation based on Felder-Silverman Learning Style Model: A Learning Analytics Intervention Approach

A fixed learning path for all learners is a major drawback of virtual learning systems. An online learning path recommendation system has the advantage of offering flexibility to select appropriate learning content. Learning Analytics Intervention (LAI) provides several educational benefits, particularly for low-performing students. Researchers employed an LAI approach in this work to recommend personalised learning paths to students pursuing online courses depending on their learning styles. It was accomplished by developing a Learning Path Recommendation Model (LPRM) based on the Felder–Silverman Learning Style Model (FSLSM) and evaluating its efficacy. The data were analysed with the help of a dataset from the Moodle Research Repository, and different learning paths were recommended using a sequence matching algorithm. The effectiveness of this approach was tested in two groups of learners using the independent two-sample t-test, a statistical testing tool. The experimental evaluation revealed that learners who followed the suggested learning path performed better than those who followed the learning path without any recommendations. This enhanced learning performance exemplifies the effects of learning analytics intervention .

Developing a Learning Analytics Intervention in E-learning to Enhance Students’ Learning Performance: A Case Study

Developing a Learning Analytics Intervention in E-learning to Enhance Students’ Learning Performance: A Case Study

About or with Teachers? A Systematic Review of Learning Analytics Interventions to Support Teacher Professional Development

Learning analytics (LA) allows aggregate data about users’ to support decision making. Although teachers are recognised as important stakeholders, little is yet explored about the role that LA can play in teacher professional development (TPD). This paper aimed to conduct a systematic review of the use of LA in TPD context, focusing specifically on intervention studies, classifying purposes and methods as well as beneficiaries’ engagement and lessons learned. Search terms identified 189 papers and 31 studies were selected based on the inclusion criteria. The results show that most studies adopted data-driven approaches to monitoring teacher behaviours, through automatic extraction of logs in technology-enhanced learning environments. The perspectives, benefits and limitations in the application of LA to TPD are finally presented.

To Opt in or to Opt Out? Predicting Student Preference for Learning Analytics-Based Formative Feedback

Teachers’ work is increasingly augmented with intelligent tools that extend their pedagogical abilities. While these tools may have positive effects, they require use of students’ personal data, and more research into student preferences regarding these tools is needed. In this study, we investigated how learning strategies and study engagement are related to students’ willingness to share data with learning analytics (LA) applications and whether these factors predict students’ opt-in for LA-based formative feedback. Students (N=158) on a self-paced online course set their personal completion goals for the course and chose to opt in for or opt out of personalized feedback based on their progress toward their goal. We collected self-reported measures regarding learning strategies, study engagement, and willingness to share data for learning analytics through a survey (N=73). Using a regularized partial correlation network, we found that although willingness to share data was weakly connected to different aspects of learning strategies and study engagement, students with lower self-efficacy were more hesitant to share data about their performance. Furthermore, we could not sufficiently predict students’ opt-in decisions based on their learning strategies, study engagement, or willingness to share data using logistic regression. Our findings underline the privacy paradox in online privacy behavior: theoretical unwillingness to share personal data does not necessarily lead to opting out of interventions that require the disclosure of personal data. Future research should look into why students opt in for or opt out of learning analytics interventions.

The challenge of learning analytics implementation: lessons learned


 
 
 Despite broad interest in learning analytics across the Australian Higher Education sector, there remains few examples of institution-wide implementations. Learning analytics implementation is currently under-theorised with little knowledge of the complexities that mediate the systemic uptake required for an institution-wide implementation. It has been established that approaches to learning analytics that are exclusively top-down or bottom-up, are insufficient for successful implementation across an enterprise. Drawing upon an award-winning and institution-wide learning analytics intervention that has been used across almost 5,000 unit offerings, this paper formulates an initial set of theory informed design principles that can help learning analytics practitioners mediate the complexities of institution-wide implementation.
 
 

Changing the recent past to reduce ongoing dropout: an early learning analytics intervention for an online statistics course

ABSTRACT Practitioners of the statistics course embedded in a computer science programme at a fully online university were concerned with the high dropout rate. In the academic year 2018–19, they decided to carry out a two-phase project in order to address this issue. In the first phase, an early classifier to identify students at risk of dropping out of the 2018–19 statistics course was determined. The second phase was planned to design and implement an early intervention based on the results of the first phase. This article presents the analysis of this intervention. In the 2019–20 online statistics course, 35 students did not submit the first few quizzes before the respective deadlines. They were the target of this intervention, which gave them the chance to change their recent past by submitting the unsubmitted quizzes. Students were encouraged to change by means of an email message. The aim of the intervention was to reduce the dropout rate among the targeted students. Our analysis also includes the students’ perception of the intervention collected via a set of interviews. The results show that the intervention clearly had a beneficial effect for some students.

Learning Analytics Intervention Improves Students’ Engagement in Online Learning

Learning Analytics Intervention Improves Students’ Engagement in Online Learning

Where is the Learning in Learning Analytics? A Systematic Literature Review on the Operationalization of Learning-Related Constructs in the Evaluation of Learning Analytics Interventions

Learning technologies enable interventions in the learning process aiming to improve learning. Learning analytics provides such interventions based on analysis of learner data, which are believed to have beneficial effects on both learning and the learning environment. Literature reporting on the effects of learning analytics interventions on learning allows us to assess in what way learning analytics improves learning. No standard set of operational definitions for learning affected by learning analytics interventions is available. We performed a systematic literature review of 1932 search hits, which yielded 62 key studies. We analyzed how affected learning was operationalized in these key studies and classified operational definitions into three categories: 1) learning environment ; 2) learning process; and 3) learning outcome . A deepening analysis yielded a refined classification scheme with 11 subcategories. Most of the analyzed studies relate to either learning outcome or learning process . Only nine of the key studies relate to more than one category. Given the complex nature of applying learning analytics interventions in practice, measuring the effects on a wider spectrum of aspects can give more insight into the workings of learning analytics interventions on the different actors, processes, and outcomes involved. Based on the results of our review, we recommend making deliberate decisions on the (multiple) aspects of learning one tries to improve by applying learning analytics. Our refined classification with examples of operational definitions may help both academics and practitioners doing so, as it allows for a more structured, grounded, and comparable positioning of learning analytics benefits.

A review of learning analytics intervention in higher education (2011–2018)

Intervention has long been practised in higher education to provide assistance for at-risk or underachieving learners. With the development of learning analytics, the delivery of intervention has been informed by data-driven approaches to identify learners’ problems and provide them with just-in-time and personalised support. However, intervention has been claimed to be the greatest challenge in learning analytics and has yet to be widely implemented. This paper reviews 24 case studies of learning analytics intervention in higher education. The cases were categorised and summarised according to their objectives, the data used, the intervention methods, the outcomes obtained and the challenges encountered. The results show that intervention practices have focused most frequently on increasing students’ study performance, offering personalised feedback and improving student retention. The frequent types of data involved students’ online learning behaviours, study performance, demographics and course selection information. The most commonly used intervention methods involved offering personalised recommendations and visualising learning data. The interventions have led to outcomes such as enhancing study performance, retention and course registration, as well as productivity and effectiveness in learning and teaching. The challenges covered a wide range of aspects, including the scalability of intervention, conditions for implementing intervention, limitations of the channels for delivering intervention and the evaluation of intervention effectiveness. The results suggest that learning analytics intervention has the potential to further extend its scope of practices to serve a wider range of purposes, but more studies on the empirical evidence, even with null or negative results, are needed to support its long-term effectiveness and sustainability.

The efficacy of learning analytics interventions in higher education: A systematic review

AbstractEducational institutions are increasingly turning to learning analytics to identify and intervene with students at risk of underperformance or discontinuation. However, the extent to which the current evidence base supports this investment is currently unclear, and particularly so in relation to the effectiveness of interventions based on predictive models. The aim of the present paper was to conduct a systematic review and quality assessment of studies on the use of learning analytics in higher education, focusing specifically on intervention studies. Search terms identified 689 papers, but only 11 studies evaluated the effectiveness of interventions based on learning analytics. These studies highlighted the potential of such interventions, but the general quality of the research was moderate, and left several important questions unanswered. The key recommendation based on this review is that more research into the implementation and evaluation of scientifically driven learning analytics is needed to build a solid evidence base for the feasibility, effectiveness and generalizability of such interventions. This is particularly relevant when considering the increasing tendency of educational institutions around the world to implement learning analytics interventions with only little evidence of their effectiveness. Practitioner NotesWhat is already known about this topic? Drop‐out rates and underachivement is a significant issue at most Western universities. Learning analytics have been shown to predict student performance and risk of dropping out. Interventions based on learning analytics have emerged in recent years, some reportedly successful. What this paper adds The paper also reviews and synthesizes the evidence on the effectiveness of learning analytics interventions targeting student underperformance, experience and discontinuation. The paper compares and contrasts past and current learning analytics methods and foci, and makes recommendations for the future research and practice. It critically synthesizes the current evidence base on learning analytics interventions, which is a field that is in constant flux and development. Implications for practice and/or policy The paper focuses on an increasing part of higher education with the goal of validating learning analytics methods and usefulness. The paper makes evidence‐based recommendations for institutions wishing to implement learning analytics programs and/or interventions. The paper makes evidence‐based recommendations for instructors as well as researchers in the field.

Facilitating Student Success in Introductory Chemistry with Feedback in an Online Platform

Instructional technologists and faculty in post-secondary institutions have increasingly adopted learning analytics interventions such as dashboards that provide real-time feedback to students to support student’ ability to regulate their learning. But analyses of the effectiveness of such interventions can be confounded by measures of students’ prior learning as well as their baseline level of self-regulated learning. For this research study, we sought to examine whether the frequency of accessing a dashboard was associated with learning outcomes after matching subjects on confounding variables. And because prior research has suggested that measures of prior learning are associated with students’ likelihood to use learning analytics interventions, we sought to adequately control for learners’ likelihood to access the feedback by using a propensity score matching with a non-binary treatment variable. We administered the Motivated Strategies for Learning Questionnaire and also collected demographic information for a propensity score matching process. Users’ frequency of accessing the intervention was categorized as High, Moderate, or Low/No usage. After matching users on characteristics associated with dashboard usage (gender, high school GPA, and the “Test Anxiety” and “Self Efficacy” factors) we found that both the “High” and “Moderate” users achieved significantly higher course grades than the “Low/No” users. The results suggest learners benefited from regularly accessing the feedback, but extreme amounts of usage were not necessary to achieve a positive effect. We discuss the implications for recommending how students use learning analytics interventions without excessively accessing feedback.

Analytics4Action Evaluation Framework: A Review of Evidence-Based Learning Analytics Interventions at the Open University UK

There is an urgent need to develop an evidence-based framework for learning analytics whereby stakeholders can manage, evaluate, and make decisions about which types of interventions work well and under which conditions. In this article, we will work towards developing a foundation of an Analytics4Action Evaluation Framework (A4AEF) that is currently being tested and validated at the Open University UK. By working with 18 introductory large-scale modules for a period of two years across the five faculties and disciplines within the OU, Analytics4Action provides a bottom-up-approach for working together with key stakeholders within their respective contexts. A holistic A4AEF has been developed to unpack, understand and map the six key steps in the evidence-based intervention process. By means of an exemplar in health and social science, a practical illustration of A4AEF is provided. In the next 3–5 years, we hope that a rich, robust evidence-base will be presented to show how learning analytics can help teachers to make informed, timely and successful interventions that will help each learner to achieve the module’s learning outcomes.

Investigating student motivation in the context of a learning analytics intervention during a summer bridge program

Summer bridge programs are designed to improve retention and academic success among at-risk populations in postsecondary education by focusing on successful skills, behaviors, and high impact practices that promote academic performance. Recent research on these programs has focused primarily on how students’ incoming demographics and prior academic performance predict academic performance at the university level. This study investigated changes in students’ academic motivation orientations over the course of one bridge program, and how a learning analytics-based intervention was employed by academic advisors to inform their face-to-face meetings with students. The results of our study show that students’ mastery orientation decreased over the course of the bridge program, and indicate that students’ exposure to displays of their academic performance negatively predicts this change. The findings suggest that student perceptions of their goals and formative performance need to be carefully considered in the design of learning analytics interventions since the resulting tools can affect students’ interpretations of their own data as well as their subsequent academic success.

Learning Analytics for Online Discussions: Embedded and Extracted Approaches

This paper describes an application of learning analytics that builds on an existing research program investigating how students contribute and attend to the messages of others in asynchronous online discussions. We first overview the E-Listening research program and then explain how this work was translated into analytics that students and instructors could use to reflect on their discussion participation. Two kinds of analytics were designed: some embedded in the learning environment to provide students with real-time information on their activity in-progress; and some extracted from the learning environment and presented to students in a separate digital space for reflection. In addition, we describe the design of an intervention though which use of the analytics can be introduced as an integral course activity. Findings from an initial implementation of the application indicated that the learning analytics intervention supported changes in students’ discussion participation. Five issues for future work on learning analytics in online discussions are presented. One, unintentional versus purposeful change; two, differing changes prompted by the same analytic; three, importance of theoretical buy-in and calculation transparency for perceived analytic value; four, affective components of students’ reactions; and five, support for students in the process of enacting analytics-driven changes.