Evidence-centered Assessment Design Research Articles

Evidence centered design framework and dynamic bayesian network for modeling learning progression in online assessment system.

An overarching mission of the educational assessment community today is strengthening the connection between assessment and learning. To support this effort, researchers draw variously on developments across technology, analytic methods, assessment design frameworks, research in learning domains, and cognitive, social, and situated psychology. The study lays out the connection among three such developments, namely learning progressions, evidence-centered assessment design (ECD), and dynamic Bayesian modeling for measuring students’ advancement along learning progression in a substantive domain. Their conjunction can be applied in both formative and summative assessment uses. In addition, this study conducted an application study in domain of beginning computer network engineering for illustrating the ideas with data drawn from the Cisco Networking Academy’s online assessment system.

Examining an Acid–Base Laboratory Practical Assessment from the Perspective of Evidence-Centered Design

This paper uses evidence-centered assessment design (ECD) to structure and analyze a laboratory activity where students collect and represent titration curve data to determine acid strength in terms of molarity and dissociation constant. Following principles of multidimensional learning, the activity prompts students to demonstrate knowledge of chemistry concepts related to acid–base titrations, the science and engineering practice of using mathematics, and the cross-cutting concept of patterns. The work involved writing two multidimensional learning performances, themselves elaborated into five distinct evidence statements. In the process of coding, we further differentiated the five evidence statements into 13 components, each aligned with at least one place in the task where students could provide evidence of that component to produce results that describe how frequently the students meet the evidence statements. We describe how the evidence statements and their components give insight into whether the students are demonstrating knowledge, skills, and thinking patterns. The data consisted of laboratory reports and postlab questions of 45 students in a general chemistry course at a large public urban university. We examine the data to address the research questions “How does a process of evidence-centered assessment design with associated learning performances provide a structure for research on how students can demonstrate multidimensional learning about chemistry content knowledge?” and “How does an assessment using evidence-centered design and multidimensional learning provide insight about teaching and assessment of learning about the use of titration curve laboratory data in acid–base chemistry?” The results provide insight into the challenges and opportunities associated with having a rich, deep examination of student multidimensional learning across a two-week laboratory assessment, including the way that grading rubrics can impact whether students fully engage with all aspects of the task. Patterns observed within the student work supported conclusions about the nature of student engagement in disciplinary concepts, the practice of using mathematics, and the use of patterns on acid–base titration tasks. We also develop ways to characterize particular components for whether the component can provide single-dimensional evidence, basic multidimensional evidence by requiring use of a concept or practice as a tool, linked multidimensional evidence because a concept or practice is needed to make sense of the phenomenon in a new way, or integrated multidimensional evidence of the different dimensions working together for sense-making.

The American Chemical Society General Chemistry Performance Expectations Project: From Task Force to Distributed Process for Implementing Multidimensional Learning

This contribution reports the process and outcomes of a multiyear effort that supported institutions in reforming general chemistry using multidimensional learning approaches in the form of performance expectations. Performance expectations are based on evidence-centered assessment design principles and describe what learners should be able to do with their knowledge, a subtle (but profound) shift from traditional course learning goals. The effort grew from the recommendations of a task force appointed in 2015 by the American Chemical Society’s Division of Chemical Education and Society Committee on Education. This task force recommended a participatory process for creating general chemistry performance expectations that was distributed over, and also coordinated across, multiple institutions. With support from the ACS Education Division, this recommendation was enacted through workshops which supported faculty in developing activities and assessments that integrated content, science and engineering practices, and cross-cutting concepts—a three-dimensional structure based on the National Research Council report A Framework for K-12 Education. From these workshops, a group of faculty committed to implementing three-dimensional performance expectations in their courses evolved. In practice, these faculty found that their institutional work resulted in designing learning performances that, while also three-dimensional, were of narrower content focus than is typical of performance expectations. This development process also led faculty to use the structures of evidence-centered design and multidimensional learning to document learning activity designs in new ways, generating a consensus activity structure. As examples of how faculty used this consensus activity structure as a new way to examine student learning and performances, development artifacts from four of the participating institutions is presented.

Utilizing Cognitive Load Theory and Evidence-Centered Design to Inform the Design of Game-Based Learning Environments

Digital game-based learning (DGBL) environments are increasingly utilized to facilitate classroom instruction. For the game in our study, a formative stealth assessment tool, in the form of an intelligent tutoring system (ITS) is guided by evidence-centered assessment design (ECD). Cognitive Load Theory and ECD are utilized as diagnostic tools to analyze upsurges in hints delivered by the ITS and inform game design revisions that will promote improved learner support and learning outcomes.

Assessing Information and Communication Technology Competence of Students: Approaches, Tools, Validity and Reliability of Results

The study describes the Information and Communication Literacy Test (the ICL Test), an instrument to measure information and communication technology competence in middle school students (grades 7–9). An overview of the existing instruments proves that there are no other ICT competence assessment instruments applicable to Russian students today. The assessment was constructed using an innovative systematic approach (evidence-centered assessment design, ECD). The ICL Test results were calculated using Bayesian networks, which are more effective than the Item Response Theory in this case. The ICL Test revealed a high level of construct, content, convergent, divergent and face validity as well as good reliability. The instrument conforms to both Russian and international educational standards according to Webb’s criteria for alignment. The ICL Test results were used to assess ICT competence and identify the factors that influence its development in regions of Russia, Armenia and Belarus. The conclusion is that the ICL Test can be applied to both Russian and foreign education systems.

How Task Features Impact Evidence From Assessments Embedded in Simulations and Games

One of the key ideas of evidence-centered assessment design (ECD) is that task features can be deliberately manipulated to change the psychometric properties of items. ECD identifies a number of roles that task-feature variables can play, including determining the focus of evidence, guiding form creation, determining item difficulty and discrimination, characterizing proficiency, and producing task variants. Assessment developers can use these task features to manipulate the psychometric properties of both conventional assessment formats and complex tasks embedded in simulations and games.Simulations and games tasks present additional challenges: even defining what corresponds to an item can be difficult. Often task features are determined by game or simulation logic rather than psychometric design. Despite these difficulties, the roles for task features identified in ECD are useful for analyzing the psychometric properties of embedded assessments in simulations and games. This article compares a conventional test of mathematics-problem-solving ability using word problems to an assessment of conceptual physics, creativity, and conscientiousness embedded in the game Newton’s Playground, describing how the task roles play out in each setting and how they can be used to manipulate the evidence provided by an assessment.

A Framework for Structuring Learning Assessment in a Massively Multiplayer Online Educational Game

Educational games offer an opportunity to engage and inspire students to take interest in science, technology, engineering, and mathematical (STEM) subjects. Unobtrusive learning assessment techniques coupled with machine learning algorithms can be utilized to record students' in-game actions and formulate a model of the students' knowledge without interrupting the students' play. This paper introduces “Experiment Centered Assessment Design” (XCD), a framework for structuring a learning assessment feedback loop. XCD builds on the “Evidence Centered Assessment Design” (ECD) approach, which uses tasks to elicit evidence about students and their learning. XCD defines every task as an experiment in the scientific method, where an experiment maps a test of factors to observable outcomes. This XCD framework was applied to prototype quests in a massively multiplayer online (MMO) educational game. Future work would build upon the XCD framework and use machine learning techniques to provide feedback to students, teachers, and researchers.

Problem solving learning environments and assessment: A knowledge space theory approach

This paper explores the relation between problem solving learning environments (PSLEs) and assessment concepts. The general framework of evidence-centered assessment design is used to describe PSLEs in terms of assessment concepts, and to identify similarities between the process of assessment design and of PSLE design. We use a recently developed PSLE, the ProNIFA system, to illustrate the concepts of student model, evidence model, and task model, concepts that provide for a close link between problem solving teaching software and assessment concepts. We also introduce ProNIFA because it uses a mathematical method developed in psychometric test theory-Competency-based Knowledge Space Theory (CbKST)-for building a student model based on observations on students' problem solving performance. The experiences made with methods such as CbKST lead us to the conclusion that the time has come to more frequently integrate assessment components into PSLEs, and to use problem solving and simulation environments as part of assessment environments. This will contribute to making assessment more authentic and less obtrusive, and making PSLEs more relevant in formal educational settings.

Evidence-Centered Assessment Design and the Advanced Placement Program®: A Psychometrician's Perspective

This paper provides an overview of evidence-centered assessment design (ECD) and some general information about of the Advanced Placement (AP®) Program. Then the papers in this special issue are discussed, as they relate to the use of ECD in the revision of various AP tests. This paper concludes with some observations about the need to validate various claims that are often made about ECD.

Representing Targets of Measurement Within Evidence-Centered Design

In the last few years, the Advanced Placement (AP) Program® has used evidence-centered assessment design (ECD) to articulate the knowledge, skills, and abilities to be taught in the course and measured on the summative exam for four science courses, three history courses, and six world language courses; its application to calculus and English language arts subjects is in progress. The purpose of this article is to describe the methodology that was used with subject-matter experts (SMEs) to articulate the content and skills important in the domain, and then the iterative processes that were used to articulate the claims and evidence to represent the targets of instruction for AP courses, and by extension, the targets of measurement for the AP exams. Discussion will focus on how the use of ECD provides a strong foundation for ensuring the alignment among curriculum, instruction, and assessment while at the same time enhances the validity argument for test score interpretation.

Evidence-Centered Assessment Design as a Foundation for Achievement-Level Descriptor Development and for Standard Setting

In many large-scale assessment programs, achievement level descriptors (ALDs) provide a critical role in communicating what scores on the assessment mean and in interpreting what examinees know and are able to do based on their test performance. Based on their test performance, examinees are often classified into performance categories. The descriptions of what these performance categories mean are communicated through the ALDs. In addition to providing supporting information for score interpretations, ALDs may also serve an important role in setting the cut scores on the score scale that determine the categorization of examinees' scores into the performance categories. The standard setting process is used to establish these cut scores. The purpose of this article is to provide a methodology to directly connect evidence-centered assessment design (ECD) to score interpretation and use through the development of ALDs and to explore how ECD-based ALDs can be leveraged in standard setting.

The Promises and Challenges of Implementing Evidence-Centered Design in Large-Scale Assessment

The cornerstone of evidence-centered assessment design (ECD) is an evidentiary argument that requires that each target of measurement (e.g., learning goal) for an assessment be expressed as a claim to be made about an examinee that is relevant to the specific purpose and audience(s) for the assessment. The observable evidence required to warrant each claim is also articulated. In turn, the claims and evidence shape the design of assessment opportunities for students to demonstrate what they have learned, whether that opportunity is a classroom activity or a multiple-choice item on a high-stakes assessment. Once identified, the characteristics of these assessment opportunities are referred to as task models, each capable of generating multiple assessment tasks. Taken together, the claims, evidence, and task models constitute the evidentiary argument. The benefits and challenges of implementing ECD in the Advanced Placement Program are addressed.

Claims, Evidence, and Achievement-Level Descriptors as a Foundation for Item Design and Test Specifications

Evidence-centered assessment design (ECD) explicates a transparent evidentiary argument to warrant the inferences we make from student test performance. This article describes how the vehicles for gathering student evidence—task models and test specifications—are developed. Task models, which are the basis for item development, flow directly from the claims we want to make about students and the observable evidence required to warrant those claims. The claims and evidence, and therefore, the task models, are directly connected to the underlying construct as defined by the achievement-level descriptors (ALDs). Test specifications must balance the claims and evidence that distinguish students at each achievement level with the psychometric characteristics necessary for reliability and comparability. The explicit relationships among claims, evidence, task models, items, achievement levels, and test specifications provide an evidentiary argument for making valid inferences about student test performance. The iterative nature, challenges, and resulting benefits of this evidence-centered approach to item design and test specifications are discussed.

Application of Evidence-Centered Assessment Design to the Advanced Placement Redesign: A Graphic Restatement

The foregoing articles constitute what I consider a comprehensive and clear description of the redesign process of a major assessment. The articles serve to illustrate the problems that will need to be addressed by large-scale assessments in the twenty-first century. Primary among them is how to organize the development of such assessments to meet a new set of demands without sacrificing quality, and even taking the opportunity to increase quality. Furthermore, these articles provide very concrete steps in solving those problems. My main goal in this article is to provide a unified graphic restatement of their well-explicated process. In addition I will raise a few questions that may be worth considering.

EVIDENCE‐CENTERED ASSESSMENT DESIGN FOR REASONING ABOUT ACCOMMODATIONS FOR INDIVIDUALS WITH DISABILITIES IN NAEP READING AND MATHEMATICS

ABSTRACTAccommodations play a key role in enabling individuals with disabilities to participate in the National Assessment of Educational Progress (NAEP) and other large‐scale assessments. However, it can be difficult to know how accommodations affect the validity of results, thus making it difficult to determine which accommodations should be allowed. This study describes recent extension of evidence‐centered assessment design (ECD) for reasoning about the impact of accommodations and other accessibility features (e.g., universal design features) on the validity of assessment results, using examples from NAEP reading and mathematics. The study found that the ECD‐based techniques were useful in analyzing the effects of accommodations and other accessibility features on validity. Such design capabilities may increase assessment designers' capacity to employ accessibility features without undermining validity.

Implications of Evidence‐Centered Design for Educational Testing

Evidence‐centered assessment design (ECD) provides language, concepts, and knowledge representations for designing and delivering educational assessments, all organized around the evidentiary argument an assessment is meant to embody. This article describes ECD in terms of layers for analyzing domains, laying out arguments, creating schemas for operational elements such as tasks and measurement models, implementing the assessment, and carrying out the operational processes. We argue that this framework helps designers take advantage of developments from measurement, technology, cognitive psychology, and learning in the domains. Examples of ECD tools and applications are drawn from the Principled Assessment Design for Inquiry (PADI) project. Attention is given to implications for large‐scale tests such as state accountability measures, with a special eye for computer‐based simulation tasks.

A BRIEF INTRODUCTION TO EVIDENCE‐CENTERED DESIGN

ABSTRACTEvidence‐centered assessment design (ECD) is an approach to constructing educational assessments in terms of evidentiary arguments. This paper provides an introduction to the basic ideas of ECD, as well as some of the terminology and models that have been developed to implement the approach. In particular, it presents the high‐level models of the Conceptual Assessment Framework (CAF) and the Four‐process Delivery Architecture for assessment delivery systems. Special attention is given to the role of probability‐based reasoning in accumulating evidence across task performances, in terms of belief about unobservable variables that characterize the knowledge, skills, and/or abilities of students. This is the role traditionally associated with psychometric models, such as those of item response theory (IRT) and latent class models. To unify the ideas and to provide a foundation for extending probability‐based reasoning in assessment applications more broadly, however, a more general expression in terms of graphical models is indicated. This brief overview of evidence‐centered design provides the reader with a feel for where and how graphical models fit into the larger enterprise of educational and psychological assessment. A simple example based on familiar large‐scale standardized tests such as the Graduate Record Examinations® (GRE®) is used to fix ideas.