The Transition to Computer-Based Assessment New Approaches to Skills Assessment and Implications for Large-scale Testing

Abstract

In educational large-scale assessments such as PISA only recently an increasing interest in measuring cross-curricular competencies can be observed. These are now discovered as valuable aspects of school achievement. Complex problem solving (CPS) describes an interesting construct for the diagnostics of domain-general competencies. Here, we present MicroDYN, a new approach for computer-based assessment of CPS. We introduce the new concept, describe proper software and present first results. At last, we provide an outlook for further research and specify necessary steps to take in the effort to measure CPS on an individual level. ______________________________________ Until recently, psychological assessment of aptitudes and abilities has relied almost entirely on paper-and-pencil-based testing. As computers emerged, these were discovered as efficient means to measure abilities. This development has led to new technologies and assessment procedures such as Computer Adaptive Testing (CAT) as is outlined widely in this volume. However, not only has measurement become more efficient through computer-based assessment. Additionally, new constructs not measurable in traditional formats now can be assessed by computer-based procedures (see Patrick Kyllonen, this volume). Among others, complex problem solving being inherently dynamic is one of these new constructs that rely on interaction between task and subject. We will introduce complex problem solving as research topic and present ways to measure problem-solving competencies in an innovative way. First results and open-access software are presented showing how new constructs over time might emerge. Complex problem solving within dynamic systems has been an area of major interest in experimental research over the last decades (for a review see Blech & Funke, 2005). Comparatively little research has been conducted about CPS in the context of individual differences even though some efforts have been made (e.g. Beckmann, 1994; Wagener, 2001). However, embedded in the recent development of large-scale assessments in educational settings, cross-curricular competencies such as CPS have been discovered as valuable aspects of school achievement (Klieme, Leutner, & Wirth, 2005). Starting from a practical point of view, applied implications of CPS are frequently found in everyday life. Many activities can be described within this formal framework ranging from medical emergencies over evaluating one’s monthly expenses to handling ticket machines at train stations. These activities involve situations comprising of the following characteristics: • Different variables influence one or more outcomes (interconnectedness), • the underlying system is not static (dynamics), • exhaustive information and evaluation of the situation may not be obtained (intransparency). A first successful approach towards measuring CPS (CPS and dynamic problem solving are identical; we argue that CPS is in itself always dynamic as opposed to analytical problem solving) in a large-scale context was conducted in PISA 1999 (Wirth & Funke, 2005). Students had to explore and control a system (embedded in the context of space travel) with different states that could be changed by activating or deactivating various switches (e.g. on/off; start/land). A system with qualitatively differing states that can be altered by the user is commonly called a finite automaton. Comparable to a finite automaton is the approach outlined below. These systems differ, however, from the qualitative approach by using only quantitatively different states (e.g. continuum from low to high). The finite automaton used in PISA could explain additional variance in student achievement after controlling for general intelligence. Furthermore, factor analytical results, structural equation models and multidimensional scaling suggested that CPS, analytical problem solving, domain specific literacy and general intelligence are correlated and yet separable constructs with CPS being best separable from the others (Wirth, Leutner, & Klieme, 2005). These results indicate construct validity and in particular convergent and divergent validity for CPS. However, the finite automaton used in