Evidence-centered Design Framework Research Articles

Designing formative assessments of early childhood computational thinking

With growing interest in supporting the development of computational thinking (CT) in early childhood, there is also need for new assessments that serve multiple purposes and uses. In particular, there is a need to understand the design of formative assessments that can be used during classroom instruction to provide feedback to teachers and children in real-time. In this paper, we report on an empirical study and advance a new unit of observational analysis for formative assessment that we call an indicator of a knowledge refinement opportunity or as a shorthand, KRO indicators. We put forth a new framework for conceptualizing the design of formative assessments that builds on the Evidence Centered Design framework but centers identification and analysis of indicators of knowledge refinement opportunities. We illustrate a number of key indicators through empirical examples drawn from video recordings of Kindergarten classroom lessons.

Evidence centered design framework and dynamic bayesian network for modeling learning progression in online assessment system.

An overarching mission of the educational assessment community today is strengthening the connection between assessment and learning. To support this effort, researchers draw variously on developments across technology, analytic methods, assessment design frameworks, research in learning domains, and cognitive, social, and situated psychology. The study lays out the connection among three such developments, namely learning progressions, evidence-centered assessment design (ECD), and dynamic Bayesian modeling for measuring students’ advancement along learning progression in a substantive domain. Their conjunction can be applied in both formative and summative assessment uses. In addition, this study conducted an application study in domain of beginning computer network engineering for illustrating the ideas with data drawn from the Cisco Networking Academy’s online assessment system.

영어 읽기 평가에 대한 재고찰: 증거기반평가모델의 적용

This paper discusses the limitations of the current practices of English reading assessment in the Korean educational context based on the concept of 'cognitive validity.' It then introduces the Evidence-centered Design (ECD) model as a framework that can guide English teachers and test developers in developing a reading assessment. The paper illustrates how the framework can be applied to the assessment formats and practices widely used in Korean middle and high schools. The ECD framework can help English teachers reconsider reading assessment practices commonly implemented in Korea. The framework contributes to enabling them to focus on the three critical, interrelated questions: what ability to measure with what task(s), how to score students' responses, and how to interpret the test results. Teachers' conscious application of the ECD framework would lead to a more valid and theoretically more sound reading assessment. Such an assessment is expected to align better with teaching and eventually bring a positive washback in English language learning.

Measuring Students’ Ability to Engage in Scientific Inquiry: A New Instrument to Assess Data Analysis, Explanation, and Argumentation

ABSTRACTWe describe the conceptualization, design, development, validation, and testing of a summative instrument that measures high school students’ ability to analyze and evaluate data, construct scientific explanations, and formulate scientific arguments in biology and chemistry disciplinary contexts. Data from 1,405 students were analyzed to evaluate the properties of the instrument. Student measurement separation reliability was 0.71 with items showing satisfactory fit to the Partial Credit Model. The use of the Evidence-Centered Design framework during the design and development process provided a strong foundation for the validity argument. Additional evidence for validation were also gathered. The strengths of the instrument lie in its relatively brief time for administration and a unique approach that integrates science practice and disciplinary knowledge, while simultaneously seeking to decouple their measurement. This research models how to design assessments that align to the National Research Council’s framework and informs the design of Next Generation Science Standards-aligned assessments.

Validity of a Special Education Teacher Observation System

ABSTRACTThis manuscript describes the comprehensive validation work undertaken to develop the Recognizing Effective Special Education Teachers (RESET) observation system, which was designed to provide evaluations of special education teachers’ ability to effectively implement evidence-based practices and to provide specific, actionable feedback to teachers on how to improve instruction. Following the guidance for developing effective educator evaluation systems, we employed the Evidence-Centered Design framework, articulated the claims and inferences to be made with RESET, and conducted a series of studies to collect evidence to evaluate its validity. Our efforts and results to date are described, and implications for practice and further research are discussed.

The Expanded Evidence-Centered Design (e-ECD) for Learning and Assessment Systems: A Framework for Incorporating Learning Goals and Processes Within Assessment Design.

Evidence-centered design (ECD) is a framework for the design and development of assessments that ensures consideration and collection of validity evidence from the onset of the test design. Blending learning and assessment requires integrating aspects of learning at the same level of rigor as aspects of testing. In this paper, we describe an expansion to the ECD framework (termed e-ECD) such that it includes the specifications of the relevant aspects of learning at each of the three core models in the ECD, as well as making room for specifying the relationship between learning and assessment within the system. The framework proposed here does not assume a specific learning theory or particular learning goals, rather it allows for their inclusion within an assessment framework, such that they can be articulated by researchers or assessment developers that wish to focus on learning.

Preliminary Development of the Hand and Arm Function Measure for People With Neurological Conditions

Abstract Date Presented 4/1/2017 The Hand and Arm Function Measure was newly developed for people with neurological conditions. It advances occupational performance measurement with use of an evidence-centered design framework for scale development, engagement of stakeholders, and combination of self-report and performance. Primary Author and Speaker: Namrata Grampurohit Contributing Authors: Brian Dudgeon, Valerie Kelly, Catherine Taylor, Deborah Kartin

Learning analysis of K-12 students’ online problem solving: a three-stage assessment approach

ABSTRACTProblem solving is considered a fundamental human skill. However, large-scale assessment of problem solving in K-12 education remains a challenging task. Researchers have argued for the development of an enhanced assessment approach through joint effort from multiple disciplines. In this study, a three-stage approach based on an evidence-centered design framework is proposed to analyze problem-solving behavior, abilities, and performance. The approach is applied to assess the online problem solving of 554 students in a Shanghai primary school. The study reveals four clusters with distinctive problem-solving behavior, abilities, and performance. The findings of this approach also corroborate the results of the Programme for International Student Assessment of Shanghai students’ problem-solving performance. The implications and limitations of this study are also discussed.

Cognitively Diagnostic Assessments and the Cognitive Diagnosis Model Framework

This paper aims to identify the utility of and the need for cognitively diagnostic assessments (CDAs) in conjunction with cognitive diagnosis models (CDMs), and to outline various considerations involved in their development and use. We begin by contrasting the CDA/CDM framework against existing assessment frameworks, which are typically based on item response theory or classical test theory, and show that CDAs used in the CDM context can provide valuable diagnostic information that could enhance classroom instruction and learning. We then detail how the components of a CDA fit into the assessment triangle framework, as well as the evidence-centered design framework. Attribute identification and item development in the context of CDA are discussed, and examples from relevant research are provided. Details of CDMs, which are the statistical models that underpin the practical implementations of CDAs, are also discussed.