Learning Management Systems Evaluation Research Articles

Evaluation of Learning Management Systems for Success Factors

Evaluating an information system for success is key. Evaluating success factors of a Learning Management System (LMS) is essential in perspective of information systems success in a developing country context. eLearning is vital to educational system considering its benefits and impacts, particularly in accessing learning from remote areas, suitable for different learners’ categories, minimal resource utilization in terms of cost and time. Through the literature reviewed on eLearning and Information System, the desire to determine the variables that measure the success factors for information systems continues. Existing Information System (IS) success models do not sufficiently evaluate eLearning in developing country, Uganda inclusive as the current IS success models are generic in nature. This study aimed to describe a model of information system success tailored to eLearning system, Kampala International University as a case study, a Uganda’s context. To address the above objective, a field study was conducted, using questionnaire to determine factors for information systems success in Uganda, a sample size of 370 respondents were used from population size of 5500 using Sloven formula. The respondents comprise of 340 students, 27 lecturers and 4 administrators. The identified success factors are skills & training, infrastructure and management support. Data were cleaned and analyzed using Statistical Package for Social Scientists version 20.0 (SPSS). This study adopted Delone and Mclean’s Information System Success model (2016) and extended it using factors obtained from the field study survey. Analysis was performed to evaluate the model. Results of the study showed that all the independent variables management support, infrastructure, skills and training are positively related to the dependent variable of intention to and use of information system. There exists a strong relationship between the multiple independent variables and the dependent variable. All factors identified, has positive impact in explaining the variation in intention to use and use of the system with r coefficient of 0.343, 0.406 and 0.406 respectively. The results of the study presented a model of success factors for Learning Management System, eLearning specifically. For future research, this study recommends conducting qualitative studies to delve deeper into the nuanced perceptions and experiences of learners and teachers as well as looking into policies that can promote eLearning especially in developing countries.

Usability Evaluation of Learning Management Systems: Google Classroom, Microsoft Teams and Padlet

In education, it is crucial to make the students enjoy during the class. Many platforms have been introduced for online classes, especially since the COVID-19 pandemic; thus, it is significant to assess the student's perception of the usability of those platforms. This research aims to evaluate the usability of the selected platforms, Google Classroom, Microsoft Teams, and Padlet. The System Usability Scale is used to measure the usability of each Learning Management System platform. A survey consisting of ten questions is distributed to a group of students. The result indicates that Google Classroom got the highest usability score, suggesting that it provides students with a more user-friendly and intuitive experience. Although Microsoft Teams received the lowest usability score, ironically, most students prefer to use it compared to the other LMSs. These findings emphasise that it is important to consider factors beyond usability alone when assessing students' platform preferences. Understanding these factors can assist educators in making informed decisions about selecting and implementing LMS platforms.

A Model for Usability Evaluation of Learning Management Systems

Learning management systems (LMSs) are increasingly popular in education. The usability of these systems must be examined to ensure they meet the needs of both students and faculty. To address the limitations of existing models for evaluating the usability of LMS, this research proposes a new model that combines objective measures collected from eye-tracking with subjective measures collected from retrospective think-aloud protocols and questionnaires. This model evaluates the effectiveness, efficiency, satisfaction, and learnability of LMSs. The proposed model was applied to study the usability of the LMS used at King Saud University (Blackboard) and was compared with other existing usability evaluation models. The findings indicate that the proposed model fulfilled its intended purpose and provided more comprehensive results than existing models.

Scrutinizing Learning Management Systems in Practice: An Applied Time Series Research in Higher Education

This study examined the use of Advancity Learning Management Systems (ALMS) and the Moodle Learning Management Systems (LMS) in learning settings, as well as online exams, within the framework of Transactional Distance Theory. With 146 college students (nfemale = 102, nmale = 44) as voluntary participants, data was gathered through an online questionnaire. A time series design was used for two different LMS sessions, and participants who voluntarily participated in ALMS and Moodle LMS sessions were matched. The findings revealed that while Moodle and ALMS both receive relatively similar assessment ratings for online exams, Moodle scored better in terms of learning setting. When factors of the Learning Management Systems Evaluation Scale (LMSES) based on Transactional Distance Theory were compared, the dialogue and autonomy factors were significantly higher for Moodle LMS than for ALMS. When online exams in the LMS were compared, there was no significant difference between ALMS and Moodle LMS, and for both LMS, the reliability factor was a determinant indicator than the other factors. As a result, in assessing and using an LMS, choices should be based on how well the LMS characteristics address an institution’s demands.

Perceived usability evaluation of learning management systems via axiomatic design with a real life application

PurposeUsage of learning management systems (LMSs) has become widespread with the disruption of face-to-face educations after the COVID-19 pandemic. There are several software products, usually named as LMS to enable and support distance education. However, selection of a suitable LMS is a complex multiple criteria decision making (MCDM) problem that requires consideration of many criteria and inputs from different parties like students, academicians, education managers, etc. Usability evaluation of LMS is one of the critical steps in deciding which LMS system to be adapted. There are several studies related to usability evaluation of LMS in the literature, but utilization of MCDM methods and real life case studies are very rare. Based on this motivation, perceived usability evaluation of SAKAI-LMS that is in use at an academic department is performed by employing axiomatic design procedure (ADP). This paper aims to discuss the aforementioned issues.Design/methodology/approachADP is considered as a suitable MCDM method for perceived usability evaluation as it allows an easy approach to data fusion and setting performance targets for decision makers. A questionnaire is developed to collect data from three types of system users about predetermined usability criteria and their importance. After detailed statistical analyses and weighting criteria via analytical hierarch process (AHP), ADP is carried out to evaluate usability of the LMS.FindingsIt is found that the proposed ADP based approach is easy to apply in practical circumstances and able to quantify perceived usability of the LMSs.Research limitations/implicationsThe proposed approach provides an easy and practical evaluation of perceived usability of the LMSs for decision makers who are responsible for the implementation of LMSs. The developed novel and practical MCDM-based perceived usability approach for LMS in this study has been verified through a real life case study at an academic department. Perceived usability results, therefore, reflects only the views of this focus group and are not generalizable.Originality/valueFirst time in the literature, a comprehensive ADP based MCDM approach is proposed based on the analyses of the related literature and information gathered from the system users.

Evaluation of Learning Management System (LMS) Canvas amidst Pandemic: Students' Perspectives

Due to the spread of the Covid-19 virus, the Pandemic Era shifted face-to-face learning to online distance learning, which faced many challenges. One of the challenges is the decreasing level of student motivation and interactivity. LMS to organize learning materials, assignments, and other learning activities is an alternative that facilitates the learning process during the pandemic. Through LMS, lecturers can share materials, give assignments, and provide feedback to students through an integrated online system. This research aims to examine the effectiveness of using LMS Canvas at Sampoerna University, especially pedagogic quality during the pandemic. In this study, researchers used a questionnaire adapted from the LMS Evaluation model by Mtebe & Raisamo (2014) by adding pedagogical quality, including teaching quality, interactivity quality, and student understanding quality. The results of the study state that these three components have a positive effect on user satisfaction and the use of LMS.

Development of the learning management systems evaluation scale based on transactional distance theory

This study aimed to develop the Learning Management Systems Evaluation Scale (LMSES) with reference to Transactional Distance Theory (TDT). At the first stage of the scale development, it was observed that 19 items with three factors explained 63.73% of the total variance. The variance amounts explained by the scale factors are Dialogue (D=8 items, 23.06%), Structure (S=7 items, 25.74%), and Autonomy (A=4 items, 14.93%). As a result of the confirmatory factor analysis performed at the second stage of the scale development, it was determined that the scale had a sufficient reliability coefficient (Cronbach's alpha S=0.90, D=0.89, A=0.82) and fit indices (χ2=252.78; sd=146; χ2/sd =1.73; CFI=0.95, NFI=0.90, GFI=0.89, AGFI=0.85, SRMR=0.06, RMSEA=0.06; p<0.001). The overall Cronbach's alpha coefficient of the scale was calculated as 0.94. Furthermore, the structural reliability of the scale (Sω=0.90, SAVE=0.57; Dω=0.88, DAVE=0.49; Aω=0.82, AAVE=0.54) and the results of the HTMT (rS-D=0.68, rA-D=0.80, rA-S=0.76) correlation ratio analysis were examined, and the evidence for the validity of the scale was supported. According to these research findings, the LMSES has the potential to provide a statistically verified measurement structure based on TDT and support advanced theoretical and statistical studies.

Evaluation of Learning Management System (LMS) Canvas amidst Pandemic: Students' Perspectives

Due to the spread of the Covid-19 virus, the Pandemic Era shifted face-to-face learning to online distance learning, which faced many challenges. One of the challenges is the decreasing level of student motivation and interactivity. LMS to organize learning materials, assignments, and other learning activities is an alternative that facilitates the learning process during the pandemic. Through LMS, lecturers can share materials, give assignments, and provide feedback to students through an integrated online system. This research aims to examine the effectiveness of using LMS Canvas at Sampoerna University, especially pedagogic quality during the pandemic. In this study, researchers used a questionnaire adapted from the LMS Evaluation model by Mtebe & Raisamo (2014) by adding pedagogical quality, including teaching quality, interactivity quality, and student understanding quality. The results of the study state that these three components have a positive effect on user satisfaction and the use of LMS.

User Experience Matters: Does One size Fit all? Evaluation of Learning Management Systems

Learning Management Systems (LMS) are in use for years and are still crucial assets for both teachers and students for teaching and learning. There is a wide variety of LMS available for institutions to administer learning in and out of the classes. However, deciding to pick an LMS to integrate is a critical decision that affects both instructors and students. On the other hand, although there is previous research LMS from students' perspective, there is a lack of research in the literature from instructors' standpoint. This research aims to gain a deeper understanding of how in-service teachers, enrolled as students in the study program of instructional technology, use of LMS in terms of user experience and satisfaction of three well-known LMS, including Canvas, Blackboard, and Moodle. The results show a significant difference between the three LMS, Blackboard, Moodle, and Canvas, in terms of users’ subjective satisfaction, ease of use, and functionality.

Quality Facetsof a UX LMS Affective Model for the Design and Evaluation of Learning Management Systems

This study looked into extant literature to elicit a mapping relationship for the affective component of Bloom’s learning taxonomy and a theoretical affective model for the design and evaluation of the affective experiences of users of and learners on learning management system (LMS) platforms. The study found that no prior affective model exists specifically for the design and evaluation of the affective experiences of LMS users. The study then mapped the affective component of the Bloom’s taxonomy of learning with the conceptual affective model and captured corresponding suitable quality facets/dimensions that influence and contribute to the model. This model is particularly appropriate for the design and evaluation of the affectivity of LMS platforms.

Fuzzy Vikor Application for Learning Management Systems Evaluation in Higher Education

Adopting learning management systems (LMS) in higher education has become a major focus of interest to implement e-learning. Evaluating the quality of LMS is important to improve learner outcomes and promote teaching strategy. Many LMSs are emerging and thus assisting higher institutions to choose the adequate LMS becomes crucial especially under fuzzy environment where uncertainties and subjectivities are considered. Because of this, the paper proposes a quality framework inspired from ISOLIEC 9126 to evaluate and rank proprietary, open source and cloud-based LMSs. Then a Fuzzy Vikor (VlseKriterijumska Optimizacija I Kompromisno Resenje) technique is applied for instantiating the proposed framework criteria and selecting alternatives from three LMSs adopted in Saudi Arabia universities. The obtained results show that the most important criteria for decision makers in these institutions are equally understandability and time behavior. In addition, the open source Moodle was set as the appropriate LMS to meet higher institutions standards.

Perceptions of learning management system quality, satisfaction, and usage: Differences among students of the arts

The model of information system success has been used to evaluate students’ satisfaction with the system quality, information quality, and service quality of learning management systems. This study extends the model by considering the pedagogical dimensions of instructional quality, learning quality, and interaction quality as well as how perceived usage frequency influences students’ quality perceptions. Through a sample of 376 higher education students, this study validated a five-factor quality structure for the evaluation of learning management systems with respect to the perceptions of students of the arts. Regression analyses showed differences among the quality factors that predicted satisfaction for arts students perceiving infrequent, average frequency, and frequent use. Although the quality of instructional experiences predicted satisfaction for all student groups, information quality significantly predicted satisfaction for only infrequent and average frequency users. System quality was a significant predictor of satisfaction for only average frequency and frequent users, whereas only frequent users perceived the quality of learning outcomes to be a significant predictor of satisfaction. The theoretical, instructional, and institutional implications for higher education institutions are discussed.

The Evaluation of Learning Management Systems by Using Fuzzy AHP, Fuzzy TOPSIS and an Integrated Method: A Case Study

Distance Education has become an important educational model with the development of information and communication technologies in the world. It has also become increasingly used model in Turkey. In our country, the number of universities and educational organizations offering distance education programs increase day by day. Distance education is performed by some special software called Learning Management System (LMS). There are both commercial LMSs and several open-source LMSs that have cost advantages. It is difficult to select an appropriate LMS for many institutions to meet their standards. The decision making and selection process is crucial, because LMS functions are different from each other, and they all have various features. Because of this, in this study, it is targeted to help and facilitate the decision-making process of LMS for institutions. In this study, Fuzzy AHP (Analytic Hierarchy Process), Fuzzy TOPSIS (Technique for Order Preference by Similarity to Ideal Solution) and an integrated method were used for evaluating criteria and deciding alternatives. To help accomplish this aim, questionnaire forms was designed and used for data collection. These methods were compared with each other. Results were evaluated. According to results among different criteria, the most important criteria for selecting LMS alternatives was found Content Management and Development (96.3%) and between the alternatives Moodle was found the best alternative with the degree of 36.3%.

Retraction statement: The application of a multi‐attribute decision‐making algorithm to learning management systems evaluation

The above article, published online on 3 February 2010 in Wiley Online Library (http://wileyonlinelibrary.com) has been retracted by agreement between the Journal Editors, Carina Girvan, Sara Hennessy, Manolis Mavrikis, Sara Price and Niall Winters, and John Wiley & Sons Ltd. on behalf of British Educational Research Association. The Retraction has been agreed due to unauthorized use of material contained in an unpublished MSc thesis: Momani, A. (2008) Easy Way to Evaluate LMSs (EW‐LMS), Department of Computer and Information Systems, Near East University, Nicosia, North Cyprus.Reference Cavus, N. (2011). . British Journal of Educational Technology, 42, 19–30. doi: 10.1111/j.1467-8535.2009.01033.x

A new expert system for learning management systems evaluation based on neutrosophic sets

AbstractThere has been a sudden increase in the usage of Learning Management Systems applications to support learner's learning process in higher education. Many studies in learning management system evaluation are implemented under complete information, while the real environment has uncertainty aspects. As these systems were described by development organizations with uncertainty terms such as vague, imprecise, ambiguity and inconsistent, that is why traditional evaluation methods may not be effective. This paper suggests neutrosophic logic as a better option to simulate human thinking than fuzzy logic because unlike fuzzy logic, it is able to handle indeterminacy of information which expresses the percentage of unknown parameters. As previous studies suggested neutrosophic decision maker and neutrosophic expert systems as future work in ecommerce and e‐learning applications, this paper presents neutrosphic expert system for learning management systems evaluation. Information for building and validating the neutrosophic expert system is collected from five experts using surveys, and then analysis is done by using Fuzzytech 5.54d software. Finally, the comparison between fuzzy expert system and neutrosophic expert system results show that the neutrosophic logic is capable of representing uncertainty in human thinking for evaluating Learning Management Systems.

Perceived usability evaluation of learning management systems: Empirical evaluation of the System Usability Scale

<h4>Perceived usability affects greatly student’s learning effectiveness and overall learning experience, and thus is an important requirement of educational software. The System Usability Scale (SUS) is a well-researched and widely used questionnaire for perceived usability evaluation. However, surprisingly few studies have used SUS to evaluate the perceived usability of learning management systems (LMSs). This paper presents an empirical evaluation of the SUS questionnaire in the context of LMSs’ perceived usability evaluation. Eleven studies involving 769 students were conducted, in which participants evaluated the usability of two LMSs (eClass and Moodle) used within courses of their curriculum. It was found that the perceived usability of the evaluated LMSs is at a satisfactory level (mean SUS score 76.27). Analysis of the results also demonstrated the validity and reliability of SUS for LMSs’ evaluation, and that it remains robust even for small sample sizes. Moreover, the following SUS attributes were investigated in the context of LMSs evaluation: gender, age, prior experience with the LMS, Internet self-efficacy, attitude towards the Internet and usage frequency of the LMS.</h4>

Students' evaluation of learning management systems in the personal computer and smartphone computing environments

The objective of this study is twofold: 1) to investigate an impact of software quality attributes of a learning management system (LMS) on students' satisfaction in both the personal computer and the smartphone settings; 2) to examine whether LMS quality factors have different impacts in the smartphone context and the PC context. We explore five quality attributes of LMS (capability, usability, performance, reliability and documentation). Data from a survey of 193 students were analysed using ordered logit regression. Findings showed that while only usability and reliability significantly affected user satisfaction in the PC context; all the quality attributes except documentation had a significant influence on user satisfaction in the smartphone setting. We also found that reliability was twice as important to user satisfaction in the smartphone context as in the PC context. The results imply that LMS quality attributes have different impacts on students' satisfaction in the smartphone context from the PC context.

Selecting a learning management system (LMS) in developing countries: instructors' evaluation

Learning management systems (LMSs) contain hidden costs, unclear user environments, bulky developer and administration manuals, and limitations with regard to interoperability, integration, localization, and bandwidth requirements. Careful evaluation is required in selecting the most appropriate LMS for use, and this is a general problem in web-based education. One possibility is to automate this evaluation process using computer-aided techniques. In this article, the researcher has focused on the idea of automating the evaluation process and has developed a computer-aided system named Easy Way to Evaluate LMSs (EW-LMS), for the quick and efficient evaluation of LMSs, especially for developing countries, where the idea of using an LMS may be relatively new. The investigation described in this article was carried out with 88 voluntary higher education instructors to test the usefulness of the developed system. Two types of quantitative questionnaires were used during the study. The results obtained indicate that the instructors were highly satisfied using the developed system, and the EW-LMS is technically sound and well developed. Theevaluation system can help instructors to select the best LMS appropriate to their educational needs and their type of usage in the shortest possible time, with little effort, without any specialized technical knowledge/training and by following simple, user-friendly steps. The system is oriented to anyone interested in evaluating and using an LMS in web-based education. Those individuals who may be interested in using this software are instructors, students, or anyone else who may have an interest in LMSs, and educational organizations such as universities, schools, and institutes.

Considerations for recruiting contributions to anonymised data sets

For the improvement and evaluation of learning management systems as well as concepts and tools for personal history, the availability of suitable data has become increasingly important. In this paper we discuss considerations for creating such a data set and report on our experiences while recruiting contributions for the Web History Repository project (WHR). We focus on measures and techniques for addressing privacy issues and strategies for motivating people to contribute. We briefly report on the outcomes of the first rounds of analysis of the data set and discuss implications for educational data mining.

Retracted: The application of a multi-attribute decision-making algorithm to learning management systems evaluation

Recently, because of the rapid increase in the popularity of the Internet, the delivery of learning programmes has gradually shifted from local desktop to online-based applications. As more and more technological tools become available for online education, there is an increasing interest among educators and other professionals in the application of these tools with regard to online courses. In large-scale operations, online learning management systems (LMSs) can save cost and time. LMSs are software applications that comprise a suite of tools for learning and teaching online. In conventional educational settings, online LMSs can help to improve the speed and effectiveness of the educational process, communication among learners, and also between staff and students. There are a large number of open-source and commercially available LMSs on the marketplace, and one of the problems facing an administrator or instructor is how to choose the LMS that can meet his/her needs. This paper describes a software system developed by the author to aid in the selection of a suitable LMS. The software is a web-based decision support system, titled Easy Way to Evaluate LMSs, and is based on using a multi-attribute decision-making algorithm for the effective selection of an LMS.