Putting ECD into Practice: The Interplay of Theory and Data in Evidence Models within a Digital Learning Environment

In recent years the educational community has increasingly embraced digital technologies for the purposes of developing alternative learning environments, providing diagnostic feedback, and fostering the development of socalled 21st -century skills. This special issue is dedicated to bridging recent work from the disciplines of educational and psychological assessment and educational data mining (EDM) via the assessment design and implementation framework of evidence-centered design (ECD). It consists of a series of five papers: one conceptual paper on ECD, three applied case studies that use ECD and EDM tools, and one simulation study that relies on ECD for its design and EDM for its implementation. In this introduction, we discuss the underlying rationales for the special issue in more detail, provide a short introduction to ECD, and describe the focus of the five selected papers.

This special issue of JEDM was dedicated to bridging work done in the disciplines of educational and psychological assessment and educational data mining (EDM) via the assessment design and implementation framework of evidence-centered design (ECD). It consisted of a series of five papers: one conceptual paper on ECD, three applied case studies that use ECD and EDM tools, and one simulation study that relies on ECD for its design and EDM for its implementation. In this reflection piece we discuss some of the key lessons that we have learned from the articles in this special issue with respect to the instructional utility of the digital learning environments, the nature of the statistical methodologies used, and the added value of the ECD framework for the work conducted in these projects.

The opportunities for delivering effective PD has dramatically increased with the recent growth of web application capabilities. However, our methodologies for making meaningful inferences about these digital learning environments have remained limited. This dissertation argues that valid measurements of research constructs in online PD environments require three components to be successful: 1) a methodology for reliably recording user behavior, 2) an assessment framework for building the connection between constructs and observable behavior and 3) a statistical analysis approach that provides post-hoc methodologies for recognizing patterns in the observable behavior. The first manuscript in this dissertation conducts a review of the literature on methods used in a sample of existing studies and suggests that a combination of click-track data as the methodology for recording user behavior, Evidence Centered Design (ECD) as the assessment framework and Educational Data Mining (EDM) as the statistical analysis approach has the potential to provide insights in online PD environments. The second manuscript uses EDM methodologies to investigate construct assumptions and user behavior patterns through click-track data. The third manuscript uses a combination of ECD and EDM methodologies to build a measure for evaluating teacher engagement in an online PD environment. The dissertation provides case studies for the use of these combined methodologies, which show promise as a viable strategy for researching and understanding online PD environments. Insights and limitations of using click-track data and directions for further research are also discussed.

Evidence-centered design (ECD) is a comprehensive framework for describing the conceptual, computational and inferential elements of educational assessment. It emphasizes the importance of articulating inferences one wants to make and the evidence needed to support those inferences. At first blush, ECD and educational data mining (EDM) might seem in conflict: structuring situations to evoke particular kinds of evidence, versus discovering meaningful patterns in available data. However, a dialectic between the two stances increases understanding and improves practice. We first introduce ECD and relate its elements to the broad range of digital inputs relevant to modern assessment. We then discuss the relation between EDM and psychometric activities in educational assessment. We illustrate points with examples from the Cisco Networking Academy, a global program in which information technology is taught through a blended program of face-to-face classroom instruction, an online curriculum, and online assessments.

Introduction. The article is devoted to the problem of preparing future teachers for providing moral education of school students within the digital learning environment. The purpose of the study is to compare the attitudes of academics, who are developing the system of preparing future teachers for providing moral education of school students within the digital learning environment with students’ opinions about the effectiveness of its components. Materials and Methods. The research follows the systemic, activity-based and personality-oriented methodological approaches. The systemic approach allows to study the preparation of future teachers for providing moral education of schoolchildren in the digital learning environment as a system. The activity-based approach makes it possible to identify specific types of activities within the system. The personality-oriented approach contributes to selecting the contents according to the interests of future teachers. In order to achieve the purpose of the study, the authors used theoretical (analysis of scholarly literature, analysis of university work experience, generalization) and empirical (ranking, survey, interview) research methods. Statistical data processing was conducted by means of the Mann-Whitney U-test. The sample consisted of Education undergraduates (n = 122), newly-qualified teachers (the graduates of the university) (n =10), and academics (n=40). Results. The results of the theoretical research demonstrate that preparation of future teachers for providing moral education of school students within the digital learning environment has systemic characteristics and is implemented in the process of studying education disciplines, work placements, independent learning, participation in extra-curricular activities organized by the university, their own school experience, information obtained from the Internet, and studying experience of teaching staff at schools. The research supports the opinion that transferring skills between learning environments is a complicated task. Consequently, targeted work is required on order to prepare future teachers for educating school students in the digital learning environment. In the process of empirical research, it was revealed that the most effective ways of teacher training today are work placements, education disciplines, and personal learning experience of prospective teachers. The authors emphasized that students consider traditional methods of preparation, for example, independent learning, to be more effective than obtaining information from the Internet, while the academics have the opposite point of view. The results of the study have proven that the balance between the real and digital environments is significant for effective teacher preparation. Conclusions. The article concludes that the system of teacher education should be supplemented by the electronic learning environment which comprises theoretical and practical issues of providing moral education of school students within the digital learning environment. The preferences of future teachers and academics largely coincide, that can contribute to the effectiveness of education. However, academics overestimate the theoretical component of the curriculum.

In this article we examine educational assessment in the 21st century. Digital learning environments emphasize learning in action. In such environments, assessments need to focus on performance in context rather than on tests of abstracted and isolated skills and knowledge. Digital learning environments also provide the potential to assess performance in context, because digital tools make it possible to record rich streams of data about learning in progress. But what assessment methods will use this data to measure mastery of complex problem solving—the kind of thinking in action that takes place in digital learning environments? Here we argue that one way to address this challenge is through evidence-centered design1—a framework for developing assessments by systematically linking models of understanding, observable actions, and evaluation rubrics to provide evidence of learning. We examine how evidence-centered design can address the challenge of assessment in new media learning environments by present...

Introduction. The paper is devoted to the problem of comparing the readiness of schools, students and future teachers for moral educational activities in the digital learning environment. In order to solve the research problem, it is necessary to compare forms of moral education activities in the digital learning environment which are offered by modern schools and the expectations of students, as well as the preferences and skills of Education students – future teachers. Thus, the purpose of the research is to compare the preferences of students and the forms actually used in the content of education in the digital learning environment. Materials and Methods. The method of analysis of information from official school websites was applied. Based on its results, a written survey of pupils and future teachers was carried out in order to select / reject the choice of each form of activity for the subsequent ranking of the forms of moral educational work in the digital learning environment. The authors analyzed the websites of selected Chelyabinsk schools where teaching internship of Education students was carried out. The sample included 135 secondary and high school students (59 male and 76 female) and 34 university students (2 male and 32 female). Statistical data processing was carried out on the basis of the Mann-Whitney U- test and the Wilcoxon T-test. Results. The authors summarized the range of forms of moral educational activities implemented by schools, which included online festivals, workshops, shifts, online theaters, thematic meetings, online projects, class hours, memos, educating activities, video lessons, school media channels, online newspapers, promotions, publication of collections, competitions, online forums. It was found that a wide range of activities was offered only in one educational setting. The rest limited themselves to participation in municipal events. Meanwhile, it was emphasized that schoolchildren showed interest in the proposed activities. The authors noted that workshops, online projects, promotions, competitions and shifts aroused the greatest interest. The main research result consisted in comparing the preferences of students and future teachers regarding the identified forms of work. Based on the Mann-Whitney U-criterion (Uemp=116.5 at a significance level of p≤0.01), the authors found that pupils and students had close preferences for the forms of educational activities in the digital learning environment. However, the Wilcoxon T-test (Temp=5.5) showed that students, being prepared to conduct moral educational activities, did not aim at implementing them in the digital learning environment. Conclusions. As a result of the study, the authors came to the conclusion that the preferences of pupils and future teachers basically coincided, which could potentially give a good result of moral educational work in the digital learning environment. However, firstly, schools have not formed such a wide range of moral educational activities in the digital learning environment as they have done in traditional offline education. Secondly, future teachers consider the digital nature of the educational environment as a demotivating factor, demonstrating the desire to educate offline, by their own example. They do not tend to look for a ‘fundamentally different’ organization of education in a digital educational environment.

National educational standards stress the importance of science and mathematics learning for today’s students. However, across disciplines, students frequently struggle to meet learning goals about core concepts like energy. Digital learning environments enhanced with artificial intelligence hold the promise to address this issue by providing individualized instruction and support for students at scale. Scaffolding and feedback, for example, are both most effective when tailored to students’ needs. Providing individualized instruction requires continuous assessment of students’ individual knowledge, abilities, and skills in a way that is meaningful for providing tailored support and planning further instruction. While continuously assessing individual students’ science and mathematics learning is challenging, intelligent tutoring systems show that it is feasible in principle. However, the learning environments in intelligent tutoring systems are typically not compatible with the vision of how effective K-12 science and mathematics learning looks like. This leads to the challenge of designing digital learning environments that allow for both – meaningful science and mathematics learning and the reliable and valid assessment of individual students’ learning. Today, digital devices such as tablets, laptops, or digital measurement systems increasingly enter science and mathematics classrooms. In consequence, students’ learning increasingly produces rich product and process data. Learning Analytics techniques can help to automatically analyze this data in order to obtain insights about individual students’ learning, drawing on general theories of learning and relative to established domain specific models of learning, i.e., learning progressions. We call this approach Learning Progression Analytics (LPA). In this manuscript, building of evidence-centered design (ECD), we develop a framework to guide the development of learning environments that provide meaningful learning activities and data for the automated analysis of individual students’ learning – the basis for LPA and scaling individualized instruction with artificial intelligence.

Two decades of game concepts in digital learning environments – A bibliometric study and research agenda

The objective of the paper is to examine the nature of students’ digital learning environments to understand the interplay of institutional systems and tools that are managed by the students themselves. The paper is based on a study of 128 students’ digital learning environments. The objectives of the study are 1) to provide an overview of tools for students’ study activities, 2) to identify the most used and most important tools for students and 3) to discover which activities the tools are used for. The empirical study reveals that the students have a varied use of digital media. Some of the most used tools in the students’ digital learning environments are Facebook, Google Drive, tools for taking notes, and institutional systems. Additionally, the study shows that the tools meet some very basic demands of the students in relation to collaboration, communication, and feedback. Finally, the study shows that most of the important tools are not related to the systems provided by the educational institutions. Based on the study, the paper concludes with a discussion of how institutional systems connect to the other tools in the students’ practices, and how we can qualify students’ digital learning environments in relation to existing and emerging needs.

Analog learning environments are often combined with, or in some cases completely replaced by digital environments – a trend that has been particularly accelerated during the COVID-19 pandemic. To gain insight into students’ perceptions, practices, and experiences regarding the use of digital technologies and learning in digital environment, a qualitative study was conducted employing in-depth interview techniques. The research sample consisted of 14 students from the University of Novi Sad, with equal representation from the different fields: social sciences and humanities, natural and mathematical sciences, technical and technological sciences, medical sciences, and the arts. By exploring students’ perspectives, our study aimed to contribute to the body of research conducted in the post-pandemic period. The research results were interpreted within four sections, starting with an examination of the broader context and the use of digital technologies in students’ daily lives, including entertainment and relaxation, communication and connectivity, and extending to information and learning. The second section focuses on the use of digital technologies in the context of learning, while the third section is dedicated to the application of digital technologies and learning in a digital environment during the teaching process. Following this, the study explores perceptions and possibilities for the coexistence of different teaching models. Based on the reviewed literature and the results of the conducted research, the conclusion is drawn that the pandemic, as a contextual factor, potentially influenced students’ negative attitudes toward online education, given that it was a case of “emergency remote education” rather than planned online education. On the other hand, the primary purpose of using digital technologies and learning in digital environment is identified as accessing learning materials.In this context, the study of Open Educational Resources (OER) may gain increasing importance in the future. Although all students report using digital technologies in the context of learning and teaching, differences in the scope, frequency, and manner of their application are noticeable. Among the variables studied, affiliation with a specific field of study was found to be more relevant for participants’ responses compared to gender and average grade.

Learning environments are ubiquitous. Schools, universities, workplaces, professional organizations, community groups, families, religious groups, museums, after‐school programs, and peer groups are environments that provide opportunities for learning and socialization. As digital technologies saturate our lives and participation in online environments soars, both the number and the variety of digital learning environments are growing exponentially. Video games, social networking sites, open courses, and a variety of social and networked technologies provide individuals with opportunities to learn content and competencies, and adopt new behavior patterns. Growing numbers of learners are taking courses online (Allen and Seaman 2013) and governments, elected officials, and interest groups are encouraging the development of digital learning opportunities (e.g., European Commission 2013). Digital learning environments are central to endeavors to design, develop, and deliver digital learning opportunities. While the LMS (e.g., Blackboard, Desire2Learn, Moodle, Instructure Canvas) is often used in such initiatives, a number of other digital learning environments have been adopted (e.g., blogging platforms, social media, and other standalone digital environments). We note that the terms LMS and VLE refer to the same technology. The term LMS is used in this chapter. The notion of the learning environment is associated with the constructivist movement (Wilson 1995, 27), as emphasis has moved from the individual (e.g., student, instructor), to the context, to the place and space surrounding learning and instruction. Wilson argues: “learning environments seem intrinsically fuzzy and ill‐defined. That is, an environment that is good for learning cannot be fully prepackaged and defined. If students are involved in choosing learning activities and controlling pace and direction, a level of uncertainty and uncontrolledness comes into play.” The increasing use of digital technologies in education gave rise to the notion of the digital learning environment. Yet, digital learning environments are also ill‐defined. Digital Learning Environments

Skepticism and learning difficulties in a digital environment at the Bachelor's and Master's levels: are preconceptions valid?

The digital learning environment comprises various resources - didactically transformed and untransformed information, and mediated communication. Students’ information behaviour combines both actions characteristic of the traditional educational process and specific for the digital environment, based on digital tools and user interactions. Students’ information behaviour in the digital environment is considered as an indicator of their engagement in various educational activities that contribute to the personalisation of learning. The results of a survey on students’ preferences of information resources in the digital environment show that learners use a variety of information sources, but they mainly apply the methods of work in the “traditional” learning paradigm. They insufficiently use the digital environment potential of collaboration, knowledge exchange, and knowledge extraction from authentic sources. Obtained data indicates problems in students’ information culture and shortcomings in the methodological support of students’ autonomous work. Based on the results, recommendations on creating conditions for developing students’ prospective strategies of interaction with digital resources are proposed. These recommendations include a gradual increase of the authentic digital learning resources, an account of students’ information preferences, and a particular attention to the management issues in the digital learning environment.

The article deals whih the results of the experimental verification of the methodology of use the information and digital learning environment as a tool for the development of teachers’ civic competence. The experimental verification was carried out during a comprehensive study in 2018-2019. The authors of this study diagnosed the level of developing teachers’ civic competence by questionnaires (incoming and outgoing) in the control (122 people) and experimental (129 people) groups. The description of criteria, levels and descriptors of assessment of civic competence of a teacher in an information and digital learning environment is given. Particular attention is paid to the attitude of teachers, school principals, methodologists and in-service teacher training specialists to the use of information and communication technologies (ICT) to create and develop an information and digital learning environment in educational institutions, and the use of digital tools in in-service teacher training institutions for civic education in schools. The purpose of the article is to prove the effectiveness of the methodology of using the information and digital learning environment as a tool for increasing teachers’ level of civic competence and to highlight the process of the implementation of civic education in Ukrainian schools. In this study we have developed the main components of the methodology of using the information and digital learning environment as a means of developing teachers’ civic competence, which constitutes the novelty of the study. The information and digital learning environment is an effective tool for raising the level of teachers’ civic competence and a tool for implementing civic education in secondary school, which is the practical significance of the study.