Evidence-centered Design Research Articles

Assessing the Grammatical Rules Proficiency among Polytechnic and Colleges of Education Students in Anambra State Using Evidence Centered Design Assessment

Language assessment is a broad concern in Applied Linguistics because it focuses on the measurement and judgment of learners’ proficiency, and the efficiency of the learning practice used by the learners as well as the teaching method employed by language teachers. This research paper adopted Evidence Centered Design (ECD) as its theoretical framework to form the bedrock of this study. The ECD methodology is an assessment approach or design which underscores the central sole of evidentiary reasoning in assessment. ECD assessment deisgn is used in this study to explore the efficiency of Evidence Centered Design in the assessment of English language proficiency level of the Igbo speakers of 200 undergraduates from three polytechnics and one college of education in Anambra State, Nigeria. The study adopted qualitative and quantitative research design, primary and secondary source of data to help the researcher make predictions and judge the communicative competence of these learners by using diagnostic model of test to determine their interactive knowledge in the use of parts of speech. The study used simple percentage which is the statistical analysis acceptable in Language studies, to analyze the data collected from the questionnaire and the test. The findings from this study proved the validity of ECD assessment in determining language proficiency.

Designing measures of complex collaborations with participatory, evidence-centered design.

An increasingly popular form of collaboration involves forming partnerships among researchers, educators, and community members to improve or transform education systems through research inquiry. However, not all partnerships are successful. The field needs valid, reliable, and useful measures to help with assessing progress toward partnership goals. In this community case study, we present a participatory, mixed-methods approach for creating measures to assess the progress of education research-practice partnerships (RPPs). The case illustrates a novel approach to measurement design, driven by perspectives and feedback of over 300 members of 80 partnerships. As a result, the measures align with the values and practices of the very collaborations the measures were intended to assess.

Feasibility of virtual reality and machine learning to assess personality traits in an organizational environment.

Personality plays a crucial role in shaping an individual's interactions with the world. The Big Five personality traits are widely used frameworks that help describe people's psychological behaviours. These traits predict how individuals behave within an organizational setting. In this article, we introduce a virtual reality (VR) strategy for relatively scoring an individual's personality to evaluate the feasibility of predicting personality traits from implicit measures captured from users interacting in VR simulations of different organizational situations. Specifically, eye-tracking and decision-making patterns were used to classify individuals according to their level in each of the Big Five dimensions using statistical machine learning (ML) methods. The virtual environment was designed using an evidence-centered design approach. The dimensions were assessed using NEO-FFI inventory. A random forest ML model provided 83% accuracy in predicting agreeableness. A k-nearest neighbour ML model provided 75%, 75%, and 77% accuracy in predicting openness, neuroticism, and conscientiousness, respectively. A support vector machine model provided 85% accuracy for predicting extraversion. These analyses indicated that the dimensions could be differentiated by eye-gaze patterns and behaviours during immersive VR. Eye-tracking measures contributed more significantly to this differentiation than the behavioural metrics. Currently, we have obtained promising results with our group of participants, but to ensure the robustness and generalizability of our findings, it is imperative to replicate the study with a considerably larger sample. This study demonstrates the potential of VR and ML to recognize personality traits.

Defining Graduate Academic Yiddish Proficiency: Results of an Evidence-Based Study

In the field of second language pedagogy, it has become increasingly common to consider the real-world usage for language when strategizing goals and curriculum development for language instruction. Emerging from a reverse design perspective, which prioritizes desired outcomes as a starting place for curricular design, language instructors identify and define the knowledge, skills, and abilities (KSAs) they aim for their students to acquire. In cases in which existing proficiency guidelines are not well aligned with the real-world language use that a particular course is targeting, it is becoming increasingly common for instructors to design Languages for Special Purposes (LSP) courses that reflect the unique uses certain bodies of students may have for the language. This paper considers one such case, that of Yiddish for Academic Purposes. Using domain analysis, a multidimensional research framework that supports and undergirds the development of new LSP courses in an assessment-driven proficiency-oriented reverse design framework and evidence-centered design (ECD), this study presents a series of target KSAs for Yiddish for Academic Purposes, on the basis of which curriculum developers could build assessments and subsequently curricula aligned with one another and with the specific language usage unique to the real-world language use domain of graduate academic Yiddish. This process of domain analysis could be replicated for other languages when academic usage is considered as a specific purpose for which an LSP course could be developed. This study is particularly relevant to the development of LSP courses for less commonly taught languages (LCTLs), which tend to be under-resourced and under-researched. Examining academic applications of LCTLs is particularly essential for those languages for which there are fewer, or more constrained, other “real world” applications for the language outside of academic use than there are for more commonly spoken languages.

Developing a measure to evaluate education research-practice partnerships

Abstract Research practice partnerships (RPPs) are an increasingly popular approach for bringing together research and practice in ways that can improve and transform education. To date, however, evaluators have lacked measures to evaluate such partnerships. This paper describes a project to develop validity evidence for a survey to evaluate the effectiveness of RPPs in education. The survey is intended to be used in conjunction with interview, observation, and other evidence to draw inferences about the effectiveness of projects within RPPs and support partnership improvement. The measurement effort followed an evidence-centered design approach to developing and evaluating the validity of the survey measure for its intended uses, collecting and analyzing data from sixty-five different RPPs. Results indicate the scales were reliable overall, but that measurement was not entirely consistent for educators and researchers. Implications for appropriate use of these measures in the context of evaluating RPPs are discussed.

Extending Principles of Evidence-Centered Design for Diverse Populations: K–12 English Learners with the Most Significant Cognitive Disabilities

ABSTRACT Designing an effective K–12 alternate English language proficiency assessment (ELP) for English learners with significant cognitive disabilities (ELSCDs) is a substantial undertaking, particularly given the absence of design models for such an assessment and limited empirical research to guide test design teams. These challenges emphasized the need to follow a rigorous and principled approach in order to establish an appropriate design. This paper describes how Evidence-Centered Design (ECD) – a comprehensive and flexible assessment design framework – was applied throughout the design process of such an assessment. By extending the principles of ECD, a maximally accessible alternate ELP assessment was designed and implemented, and evidence was collected to support validity. Implications for this assessment design effort include findings for the education and assessment of ELSCDs, the application of ECD to the assessment of diverse test takers, and to a general movement in the field toward more personalized assessments.

Concept and implementation of an adaptive digital hearing training system for cochlear implant aftercare

Within the context of an interdisciplinary research project, we created acutting-edge prototype of an adaptive digital auditory training system designed for cochlear implant (CI) users. By leveraging the evidence-centered design (ECD) framework, we integrated adynamic difficulty adjustment feature that tailors the experience to the unique performance capabilities of each individual user. The ECD provides aconceptual design framework suitable for complex assessments of competence and dynamic performance. In the first phase, the domain of hearing was first defined in the context of CI users. In the development phase the three core models of the ECD, the competence model, the evidence model, and the task model, were developed and implemented. In addition, an asset pool of sound and language files was created, which included comprehensive linguistic feature descriptions for calculating item difficulties. Based on the requirements described, an adaptive exercise generator, an AI service, and other components were implemented. This included the development of agame environment and adashboard for patient data management. The exercises' difficulty levels were determined based on various parameters (e.g., sound, word frequency and number of words, grammatical properties) in combination with defined task types and levels. An adaptive digital auditory training system can help to supervise and train CI patients in acontinuous, interactive process based on their individual needs. We see the ECD as an effective way to build auser-based adaptive system.

Theory Matters

Abstract: Following an extended perspective of evidence-centered design, this study provides a methodological exemplar of the theory-based construction of process indicators from log data. We investigated decision-making processes in web search as the target construct, assuming that individuals follow a heuristic search (focusing on search results vs. websites as a primary information source) and stopping rule (following a satisficing vs. sampling strategy). Drawing on these assumptions, we describe our reasoning for identifying the empirical evidence needed and selecting an assessment to obtain this evidence to derive process indicators that represent groups differentiated by search and stopping rule combinations. To evaluate our approach, we reanalyzed the processing behavior of 150 university students who were requested in four tasks to select a specific website from a list of five search results. We determined the process indicators per item and conducted multiple cluster analyses to investigate group recovery. For each item, we found three clusters, two of which matched our assumptions. Additionally, we explored the consistency of students’ cluster membership across items and investigated their relationship with students’ skills in evaluating online information. Based on the results, we discuss the tradeoff between construct breadth and process elaboration for deriving meaningful process indicators.

Digital Literacy Assessment: Methodology, Conceptual Model and Measurement Tool

The development of new technologies, the extensive implementation of digital devices in all spheres of society contributed to the appearance of a new type of human literacy - digital literacy. Today, it ranks alongside other basic literacies such as numeracy, writing, and reading. Digital literacy has become an indicator to the new conditions of life, involving the use of digital technologies in education, family and professional activities; to some extent, it has become an indicator of the socialization of a modern person.
 Based on the analysis of international approaches to the assessment of digital literacy and related constructs, a conceptual model of its measurement is proposed, in which digital literacy is considered as a complex latent construct.
 The process of developing digital literacy assessment tools based on the Principled Assessment Design paradigm and the Evidence-Centered Design method is presented. 
 An analysis of the results of digital literacy studies of students in grades 7-8 made it possible to confirm the quality of the developed tools for measuring digital literacy based on authentic performance-based tasks, to confirm the reliability and validity of the results of its measurement.

Applying a Culturally Responsive Pedagogical Framework to Design and Evaluate Classroom Performance-Based Assessments in Hawai‘i

ABSTRACT Previous writings focus on why centering assessment design around students’ cultural, social, and/or linguistic diversity is important and how performance-based assessment can support such aims. This article extends previous work by describing how a culturally responsive classroom assessment framework was created from a culturally responsive education (CRE) pedagogical framework. The goal of the framework was to guide the design and evaluation of curriculum-embedded, classroom performance assessments. Components discussed include: modification of evidence-centered design processes, teacher and/or student adaptation of construct irrelevant aspects of task prompts, addition of cultural meaningfulness questions to think alouds, and revision of task quality review protocols to promote CRE design features. Future research is needed to explore the limitations of the framework applied, and the extent to which students perceive the classroom summative assessments designed do indeed allow them to better show all they know and can do in ways related to their cultural, social, and/or linguistic identities.

When a machine detects student reasoning: a review of machine learning-based formative assessment of mechanistic reasoning

In chemistry, reasoning about the underlying mechanisms of observed phenomena lies at the core of scientific practices. The process of uncovering, analyzing, and interpreting mechanisms for explanations and predictions requires a specific kind of reasoning: mechanistic reasoning. Several frameworks have already been developed that capture the aspects of mechanistic reasoning to support its formative assessment. However, evaluating mechanistic reasoning in students’ open responses is a time- and resource-intense, complex, and challenging task when performed by hand. Emerging technologies like machine learning (ML) can automate and advance the formative assessment of mechanistic reasoning. Due to its usefulness, ML has already been applied to assess mechanistic reasoning in several research projects. This review focuses on 20 studies dealing with ML in chemistry education research capturing mechanistic reasoning. We developed a six-category framework based on the evidence-centered design (ECD) approach to evaluate these studies in terms of pedagogical purpose, rubric design, construct assessment, validation approaches, prompt structure, and sample heterogeneity. Contemporary effective practices of ML-based formative assessment of mechanistic reasoning in chemistry education are emphasized to guide future projects by these practices and to overcome challenges. Ultimately, we conclude that ML has advanced replicating, automating, and scaling human scoring, while it has not yet transformed the quality of evidence drawn from formative assessments.

Establishing the Validity and Reliability of the LOCUS Assessments

The development of assessments as part of the funded LOCUS project is described. The assessments measure students’ conceptual understanding of statistics as outlined in the GAISE PreK–12 Framework. Results are reported from a large-scale administration to 3,430 students in grades 6 through 12 in the United States. Items were designed to assess levels of understanding as well as components of the statistical problem solving process as articulated in the GAISE framework. We discuss details of how the model used to develop the LOCUS assessments guided the gathering of evidence for validity and reliability arguments. Three types of validity evidence are presented: Validity Evidence via Evidence-Centered Design, Validity and Reliability Evidence via Item Analysis, and Validity Evidence via Item Calibration.

Domain Analysis as a Multidimensional Research Framework: Evidence-Based Alignment for LSP Research, Assessment, and Curricula

Language for specific purposes (LSP) pedagogy has as its goal the delivery of the specific knowledge, skills, and abilities (KSAs) one must possess in order to function in professional real-world language use contexts. Delivery of such a promise requires well-defined outcomes. This paper argues that when these outcomes are not readily identifiable through theoretically explained data or empirically collected evidence, one must turn to the process of domain analysis as a research framework. The paper first contextualizes domain analysis in an assessment-driven proficiency-oriented reverse design framework (Baumann et al., n.d.) and evidence-centered design (ECD) (Mislevy & Haertel, 2006; Mislevy, 2011). Next, it describes the steps of domain analysis as a multidimensional and systematic research framework to analyze and define language use in specific LSP contexts. It concludes with a discussion of why LSP domain analyses are essential to ensure alignment across LSP research, assessment, and curricula, regardless of purpose or discipline.

Virtual reality stimulation and organizational neuroscience for the assessment of empathy

This study aimed to evaluate the viability of a new procedure based on machine learning (ML), virtual reality (VR), and implicit measures to discriminate empathy. Specifically, eye-tracking and decision-making patterns were used to classify individuals according to their level in each of the empathy dimensions, while they were immersed in virtual environments that represented social workplace situations. The virtual environments were designed using an evidence-centered design approach. Interaction and gaze patterns were recorded for 82 participants, who were classified as having high or low empathy on each of the following empathy dimensions: perspective-taking, emotional understanding, empathetic stress, and empathetic joy. The dimensions were assessed using the Cognitive and Affective Empathy Test. An ML-based model that combined behavioral outputs and eye-gaze patterns was developed to predict the empathy dimension level of the participants (high or low). The analysis indicated that the different dimensions could be differentiated by eye-gaze patterns and behaviors during immersive VR. The eye-tracking measures contributed more significantly to this differentiation than did the behavioral metrics. In summary, this study illustrates the potential of a novel VR organizational environment coupled with ML to discriminate the empathy dimensions. However, the results should be interpreted with caution, as the small sample does not allow general conclusions to be drawn. Further studies with a larger sample are required to support the results obtained in this study.

Toward learning progression analytics — Developing learning environments for the automated analysis of learning using evidence centered design

National educational standards stress the importance of science and mathematics learning for today’s students. However, across disciplines, students frequently struggle to meet learning goals about core concepts like energy. Digital learning environments enhanced with artificial intelligence hold the promise to address this issue by providing individualized instruction and support for students at scale. Scaffolding and feedback, for example, are both most effective when tailored to students’ needs. Providing individualized instruction requires continuous assessment of students’ individual knowledge, abilities, and skills in a way that is meaningful for providing tailored support and planning further instruction. While continuously assessing individual students’ science and mathematics learning is challenging, intelligent tutoring systems show that it is feasible in principle. However, the learning environments in intelligent tutoring systems are typically not compatible with the vision of how effective K-12 science and mathematics learning looks like. This leads to the challenge of designing digital learning environments that allow for both – meaningful science and mathematics learning and the reliable and valid assessment of individual students’ learning. Today, digital devices such as tablets, laptops, or digital measurement systems increasingly enter science and mathematics classrooms. In consequence, students’ learning increasingly produces rich product and process data. Learning Analytics techniques can help to automatically analyze this data in order to obtain insights about individual students’ learning, drawing on general theories of learning and relative to established domain specific models of learning, i.e., learning progressions. We call this approach Learning Progression Analytics (LPA). In this manuscript, building of evidence-centered design (ECD), we develop a framework to guide the development of learning environments that provide meaningful learning activities and data for the automated analysis of individual students’ learning – the basis for LPA and scaling individualized instruction with artificial intelligence.

Evidence centered design framework and dynamic bayesian network for modeling learning progression in online assessment system.

An overarching mission of the educational assessment community today is strengthening the connection between assessment and learning. To support this effort, researchers draw variously on developments across technology, analytic methods, assessment design frameworks, research in learning domains, and cognitive, social, and situated psychology. The study lays out the connection among three such developments, namely learning progressions, evidence-centered assessment design (ECD), and dynamic Bayesian modeling for measuring students’ advancement along learning progression in a substantive domain. Their conjunction can be applied in both formative and summative assessment uses. In addition, this study conducted an application study in domain of beginning computer network engineering for illustrating the ideas with data drawn from the Cisco Networking Academy’s online assessment system.

Validity Arguments Meet Artificial Intelligence in Innovative Educational Assessment

AbstractWe have dramatically advanced our ability to create rich, complex, and effective assessments across a range of uses through technology advancement. Artificial Intelligence (AI) enabled assessments represent one such area of advancement—one that has captured our collective interest and imagination. Scientists and practitioners within the domains of organizational and workforce assessment have increasingly used AI in assessment, and its use is now becoming more common in education. While these types of solutions offer their users the promise of efficiency, effectiveness, and a “wow factor,” users need to maintain high standards for validity and fairness in high stakes settings. Due to the complexity of some AI methods and tools, this requirement for adherence to standards may challenge our traditional approaches to building validity and fairness arguments. In this edition, we review what these challenges may look like as validity arguments meet AI in educational assessment domains. We specifically explore how AI impacts Evidence‐Centered Design (ECD) and development from assessment concept and coding to scoring and reporting. We also present information on ways to ensure that bias is not built into these systems. Lastly, we discuss future horizons, many that are almost here, for maximizing what AI offers while minimizing negative effects on test takers and programs.

영어 읽기 평가에 대한 재고찰: 증거기반평가모델의 적용

This paper discusses the limitations of the current practices of English reading assessment in the Korean educational context based on the concept of 'cognitive validity.' It then introduces the Evidence-centered Design (ECD) model as a framework that can guide English teachers and test developers in developing a reading assessment. The paper illustrates how the framework can be applied to the assessment formats and practices widely used in Korean middle and high schools. The ECD framework can help English teachers reconsider reading assessment practices commonly implemented in Korea. The framework contributes to enabling them to focus on the three critical, interrelated questions: what ability to measure with what task(s), how to score students' responses, and how to interpret the test results. Teachers' conscious application of the ECD framework would lead to a more valid and theoretically more sound reading assessment. Such an assessment is expected to align better with teaching and eventually bring a positive washback in English language learning.

Applying Evidence-Centered Design to Measure Psychological Resilience: The Development and Preliminary Validation of a Novel Simulation-Based Assessment Methodology.

Modern technologies have enabled the development of dynamic game- and simulation-based assessments to measure psychological constructs. This has highlighted their potential for supplementing other assessment modalities, such as self-report. This study describes the development, design, and preliminary validation of a simulation-based assessment methodology to measure psychological resilience—an important construct for multiple life domains. The design was guided by theories of resilience, and principles of evidence-centered design and stealth assessment. The system analyzed log files from a simulated task to derive individual trajectories in response to stressors. Using slope analyses, these trajectories were indicative of four types of responses to stressors: thriving, recovery, surviving, and succumbing. Using Machine Learning, the trajectories were predictive of self-reported resilience (Connor-Davidson Resilience Scale) with high accuracy, supporting construct validity of the simulation-based assessment. These findings add to the growing evidence supporting the utility of gamified assessment of psychological constructs. Importantly, these findings address theoretical debates about the construct of resilience, adding to its theory, supporting the combination of the “trait” and “process” approaches to its operationalization.

Measuring Students’ Critical Thinking in Online Environment: Methodology, Conceptual Framework and Tasks Typology

Today critical thinking is one of the key competencies of the modern world. The abundance and availability of various information ( in particular due to the spread of electronic devices and the Internet) suggest that people, regardless of age, need to be able to effectively navigate and evaluate information, draw their own conclusions and use arguments when making decisions. Research of critical thinking in terms of the possibilities of its evaluation and development began more than 60 years ago and has a wide and heterogeneous field of theoretical approaches. In this article, we explore the theoretical possibility of measuring the complex latent construct “critical thinking in an online environment” using the Evidence- Centered Design (ECD) methodology. It allows one to build a chain of arguments that substantiate the conclusion about the level of critical thinking development in the respondent and thereby ensure the fairness of the assessment process. The measurement goes from theoretical assumptions about the nature of the construct to the search for empirical evidence - observable actions in the testing process, allowing to draw reasonable conclusions about the respondents. The result of the work is a theoretical framework for assessing critical thinking in an online environment for university students and its operationalization through the relevant observed behavior of the target audience, which allows obtaining valid data on the severity of critical thinking. This is the first step to create an instrument in Russian with a confirmed psychometric quality. The key feature of the tool is that the student does not work in a simulated environment where information sources are selected by developers, but in an open online environment, therefore, he can use any available materials to solve the task.