Learning Design and Learning Analytics to Improve Higher Education: A Systematic Literature Review

The rapid recent use of learning analytics (LA) in higher education, specifically during the COVID-19 pandemic, allows the monitoring of users' behavior while learning. Using LA may promote students' learning outcomes but also intrude into their privacy. This study aimed to explore students’ behaviour and perceptions towards privacy and data protection when using LA for pedagogical needs, examine the privacy trade-off of students' willingness to share personal information in exchange for pedagogical benefits and understand the predicting variables for this privacy trade-off. A model was developed containing five groups of influencing variables (demographic data, perceptions, feelings, behaviour and awareness) on the privacy trade-off. A total of 1,014 students completed an online questionnaire. The results found that students do care about their privacy but are not aware of privacy and data protection regulations. They are willing to trade off privacy for pedagogical benefits, and they trust their academic institutions, but they want transparency. Age, a sense of security in the academic institution, behaviour, data misuse concern and institution management of students' personal information are the significant predictors for a privacy trade-off. It is important to engage students in the process as they are the main beneficiaries of LA and build trust between them and the institution. Implications for practice or policy: Academic institution should actively provide information to raise privacy awareness. Academic institutions should engage students in the process of using LA to create a high degree of trust. Universities should endorse LA policy, fostering it for pedagogical purposes. Instructors may utilise LA to enrich student learning, respecting their willingness to share pedagogical data. Academic institutions should provide a high level of transparency in order to build students’ trust in their institutions.

Data-driven decision-making and data-intensive research are becoming prevalent in many sectors of modern society, i.e. healthcare, politics, business, and entertainment. During the COVID-19 pandemic, huge amounts of educational data and new types of evidence were generated through various online platforms, digital tools, and communication applications. Meanwhile, it is acknowledged that education lacks computational infrastructure and human capacity to fully exploit the potential of big data. This paper explores the use of Learning Analytics (LA) in higher education for measurement purposes. Four main LA functions in the assessment are outlined: (a) monitoring and analysis, (b) automated feedback, (c) prediction, prevention, and intervention, and (d) new forms of assessment. The paper concludes by discussing the challenges of adopting and upscaling LA as well as the implications for instructors in higher education.

The current landscape of learning analytics in higher education

“ETEC 565A: Understanding Learning Analytics” was a new course offered in a Master of Educational Technology programme in UBC in January 2019. In order to support students exploring learning analytics in a more relevant way to them, the final project allows students to choose their own learning analytics adventure. This presentation will be a showcase and a reflection of learning from our final (group) project. Our group wanted to focus on learning design and learning analytics. There has been a lot of interests in learning analytics in higher education, it has been appearing in the EDUCAUSE’s Horizon report for many years as a technology to adopt. We also know that learning technologies should support the educational goals. Therefore, it is important for us to understand, how would one combine learning design with learning analytics. Our instructor guided to Lockyer, Heathcote and Dawson (2013)’s work. Their article presented “learning design as a form of documentation of pedagogical intent that can provide the context for making sense of diverse sets of analytic data” (p.1439). Lockyer et al (2013) explored using the checkpoint and process analytics as broad categories of learning analytics and how through this documentation of pedagogical intent and the related learning analytics that can be collected, could support pedagogical actions.
 
 Our instructor has given us permission to use our course as an example to apply Lockyer et al (2013)’s framework and conduct our analysis on the course. Our group also had access to some learning analytics data ourselves, through the “Threadz” tool for analysing our discussion forum activities. For all other types of data that we did not have access to, we made comments on what the data could be used for, and if it would provide enough information to assess if the pedagogical intent was met. Comments were also made about potential data that could have been better for informing pedagogical actions but were either not possible to get or too difficult and unpractical.
 
 In the presentation, I will share some backgrounds for the Lockyer et al (2013)’s framework for aligning learning analytics with learning design, how one could use the framework to document their own course design and identify potential learning analytics data sources or the lack of. As the framework provides teachers and designers a tool to think, plan and reflect with. I will discuss some of our group’s findings and reflections from the analysis of our own online course. Then discuss about potentials of using such framework on a more traditional face-to-face course.
 
 As institutions and courses collect more data about their students, it is useful to have a framework to help teachers think about how they might use the learning analytics data to support their students through examining and documenting their pedagogical intents. It is also important to note what the existing data can and cannot do to support pedagogical goals.

Purpose The analysis of data collected from user interactions with educational and information technology has attracted much attention as a promising approach to advancing our understanding of the learning process. This promise motivated the emergence of the field of learning analytics and supported the education sector in moving toward data-informed strategic decision making. Yet, progress to date in embedding such data-informed processes has been limited. The purpose of this paper is to address a commonly posed question asked by educators, managers, administrators and researchers seeking to implement learning analytics – how do we start institutional adoption of learning analytics? Design/methodology/approach A narrative review is performed to synthesize the existing literature on learning analytics adoption in higher education. The synthesis is based on the established models for the adoption of business analytics and finding two projects performed in Australia and Europe to develop and evaluate approaches to adoption of learning analytics in higher education. Findings The paper first defines learning analytics and touches on lessons learned from some well-known case studies. The paper then reviews the current state of institutional adoption of learning analytics by examining evidence produced in several studies conducted worldwide. The paper next outlines an approach to learning analytics adoption that could aid system-wide institutional transformation. The approach also highlights critical challenges that require close attention in order for learning analytics to make a long-term impact on research and practice of learning and teaching. Originality/value The paper proposed approach that can be used by senior leaders, practitioners and researchers interested in adoption of learning analytics in higher education. The proposed approach highlights the importance of the socio-technical nature of learning analytics and complexities pertinent to innovation adoption in higher education institutions.

Learning Analytics (LA) is a field of research and practice that uses data analysis to comprehend and optimize learning and the environment in which learning takes place. As an AI tool in higher education, LA is expected to improve student learning and support the academic community in teaching delivery, institutional management, long-term research and development, innovation, data-driven decision-making, and more. We have conducted a literature review to explore these issues and examine the added value of LA in higher education. We have focused on the key issues that educational institutions need to consider to get the most out of LA use. The findings of this review reveal that the proper use of the LA toolkit can enhance the development of an appropriate educational environment through the careful determination of ethics and policies that support the main institutional objective, and the study of opportunities, challenges, and trends in the sector. The key challenges of using AI tools like LA in Higher Education are data privacy and protection, data ownership, data heterogeneity, potential biases in AI algorithms, and the need for alignment of institutional strategies for LA with pedagogical approaches. The trends highlight the current advances in LA that give added value in higher education.

In a context where learning mediated by technology has gained prominence in higher education, learning analytics has become a powerful tool to collect and analyse data with the aim of improving students’ learning. However, learning analytics is part of a young community and its developments deserve further exploration. Some critical stances claim that learning analytics tends to underplay the complexity of teaching-learning processes. By means of both a bibliometric and a content analysis, this paper examines the publication patterns on learning analytics in higher education and their main challenges. 385 papers that were published in WoScc and SciELO indexes between 2013 and 2019 were identified and analysed. Learning analytics is a vibrant and fast-developing community. However, it continues to face multiple and complex challenges, especially regarding students’ learning and their implications. The paper concludes by distinguishing between a practice-based and management-oriented community of learning analytics and an academic-oriented community. Within both communities, though, it seems that the focus is more on analytics than on learning.

This workshop aims to promote strategic planning for learning analytics in higher education through developing institutional policies. While adoption of learning analytics is predominantly seen in small-scale and bottom-up patterns, it is believed that a systemic implementation can bring the widest impact to the education system and lasting benefits to learners. However, the success of it highly depends on the adopted strategy that meets the needs of various stakeholders and systematically pushes the institution towards achieving its targets. It is imperative to develop a learning analytics policy that ensures a practice that is valid, effective and ethical.

Learning analytics is defined as collecting, analysis, and reporting of data about learners inside a system. Users’ interaction, which is a form of digital traces can be processed and identified into pattern. In comparison with subjective methods such as surveys and questionnaires, learning analytics provide deeper insight on students’ behaviour and assist learning institution with data-informed decision-making regarding students’ learning issues. This paper reviews on existing literature on the latest application and the trends of learning analytics in higher education over the last few years, within the period of pre and post pandemic era. As online learning was adapted rapidly during the pandemic, learning analytics became an important tool to assess students’ learning. The literature review was obtained from Scopus and Web of Science databases by using Boolean search techniques with the right keywords. This study reviewed and provided a comprehensive analysis according to certain criteria of searched studies from the year 2018 to 2023. The result showed that the research trend in learning analytics was fluctuating in Scopus and gradually declining in Web of Science due to the pandemic and shifts in research interest. In this post pandemic era, Artificial Intelligence (AI) is trending upwards, and researchers are putting interest in integrating AI with learning analytics in education. As such, this paper could help educators see the trends of LA over the last few years, as well as the innovative practices recently.

Increased adoption of educational technologies, the emergence of digital classroom concepts, and the interest in big data innovations have led to a growing awareness of the potential implementation of learning analytics to support learning development in educational institutions. However, most studies have focused on the implementation of learning analytics in higher education. As a result, research evidence and studies on actual and ongoing implementation in pre-higher education are still scarce. Therefore, there is a need to understand better the implementation of learning analytics at primary and secondary from the current learning analytics literature. This systematic literature review (SLR) aimed to identify learning analytics research that focuses on implementing learning analytics at pre-higher education levels, including in pre-school, primary, or secondary school. This SLR was carried out based on the SALSA framework to determine the protocol, search, appraisal, synthesis, analysis, and reporting approaches. The findings of the SLR support the arguments that the implementation of learning analytics in school is still in its infancy, where the implementation has been observed mostly in developed countries. Most of the Implementation is aimed at descriptive analytics, focusing on the purpose of monitoring, analysis, and feedback. The literature review has shown a lack of research on the implementation of learning analytics at the primary and secondary education levels.

This paper presents the results of a review of eight policies for learning analytics of relevance for higher education, and discusses how these policies have tried to address prominent challenges in the adoption of learning analytics, as identified in the literature. The results show that more considerations need to be given to establishing communication channels among stakeholders and adopting pedagogy-based approaches to learning analytics. It also reveals the shortage of guidance for developing data literacy among end-users and evaluating the progress and impact of learning analytics. Moreover, the review highlights the need to establish formalised guidelines to monitor the soundness, effectiveness, and legitimacy of learning analytics. As interest in learning analytics among higher education institutions continues to grow, this review will provide insights into policy and strategic planning for the adoption of learning analytics.

Despite the growing popularity of learning analytics in higher education, its development status in Asia was barely studied. This paper reports a study on the development of learning analytics in higher education in Asia. Semi-structured interviews were conducted with eight senior managers or senior academics from various tertiary institutions in Asia. The participants were asked about their institutions’ position on learning analytics, the progress in its implementation, factors leading to effective implementation, and challenges encountered, if any. The results showed that in those institutions where learning analytics has been implemented, it aimed mainly at enhancing student retention, pedagogy and student learning experience. Its effective implementation relies on support from senior management, and taking students’ views into account in decision-making. The participants’ institutions encountered difficulties due to teachers’ and students’ concerns, such as the increased workload and data privacy, as well as technical issues in data collection, processing and analysis. In short, though starting late in Asia, learning analytics has been gradually gaining attention and is being implemented. The future directions of research and practices in learning analytics are also discussed.

ABSTRACTIn recent years, the application of technological innovation in higher education has become more and more widely spread, and technological innovation has been improving the level of education. In the research of higher education with innovation technology, one of the main focuses is on the dynamic data which can lay a foundation for the analysis of educational activities by learning analytics. The dynamic data created by technological innovation will become the key basis for analytical research and development in higher education. The methods and analysis results of learning analytics will directly affect decision-making and strategy about higher education. In this paper, we use bibliometric and visualisation methods to review the literature, in order to highlight the development of learning analytics in higher education. Using bibliometric analysis, our study depicts the development process of the main methods used in learning analytics, and summarises the current situation in this field, which increases the level of understanding provided by those studies. Finally, we summarise the research hotspots and study trends, which will be useful for future study in this field.

In order to progress in a data-driven society, higher education institutions should strive to integrate learning analytics into their strategic and operational activities. The purpose of this paper is to propose the Higher Education Learning Analytics Capability Maturity Model (HELA-CMM), a capability maturity model for adoption of learning analytics in higher education institutions. The development of the HELA-CMM was based on design science research paradigm and a procedural model for maturity models design. After conducting an extended literature review, a combination of nominal group techniques and focus group were used. The subject-matter experts participating in a nominal focus group elaborated on the structure and evaluated relative importance of its categories and capabilities utilizing the pairwise comparison technique. The proposed maturity model comprises 28 capabilities grouped into eight categories: (1) Teaching / Learning / Assessment, (2) Data Management and Analytics, (3) Ethics, Privacy, Legal Issues, (4) People Competences, (5) Management / Leadership, (6) Culture, (7) Infrastructure, and (8) Quality Assurance. The maturity model will be further developed through an iterative process and evaluated at several higher education institutions in Europe. With the HELA-CMM, higher education institutions will have a tool that on the one hand enables benchmarking and assessment, and on the other hand provides guidance for improving the maturity of learning analytics adoption.

Higher education institutions recently have been drawing on methods from learning analytics to make decisions about learners' academic progress, predictions about future performance and to recognise potential issues. As the use of learning analytics in higher education is a relatively new area of practice and research, the intent of this paper is to provide an overview of learning analytics including a summary of some exemplar tools. Finally we conclude the paper with a discussion on challenges and ethical issues.