Complex Problem-solving Skills Research Articles

Overcoming the PBL Assessment Challenge: Design and Development of the Incremental Thesaurus for Assessing Causal Maps (ITACM)

Because of the lack of tools available to assess problem-solving skills, teachers often revert to more traditional instructional approaches (e.g. lecture-based, memorization) that fail to prepare learners for the complexity of dynamic work environments. To overcome this challenge, technology solutions are needed that accurately and efficiently assess complex problem-solving skills such as causal reasoning. Moreover, these tools must be valid and reliable so instructors can accurately assess student learning. This emergent report details the design and development of Incremental Thesaurus for Assessing Causal Maps. As will be described, the software offers three unique features: (a) analysis of causal map with little or no manipulation of the original file; (b) a growing repository of terms that supports efficient assessment and (c) ability to codify the level of concept complexity using the structure–behavior–function framework.

The logic of success: the relation between complex problem-solving skills and university achievement

The successful completion of a university degree program is accompanied by multiple complex opportunities and challenges, which require students to react accordingly with the skills necessary to meet them. Therefore, the aim of this study was to investigate the role of complex-problem solving (CPS) skills in undergraduate students’ university success in two independent samples. In Study 1, 165 university students completed a measure of intelligence as well as a measure of CPS. In addition, students’ university grade point averages (GPAs) and their subjective evaluation of academic success were collected. CPS made a significant contribution to the explanation of GPAs and the subjective success evaluations even when controlling for intelligence. To further investigate this effect, Study 2 relied on an independent and more heterogeneous sample of 216 university students. The findings of Study 1 were essentially replicated in Study 2. Thus, the combined results demonstrate a link between individual differences in CPS and the abilities necessary to be academically successful in university education.

Fluid Ability (Gf) and Complex Problem Solving (CPS).

Complex problem solving (CPS) has emerged over the past several decades as an important construct in education and in the workforce. We examine the relationship between CPS and general fluid ability (Gf) both conceptually and empirically. A review of definitions of the two factors, prototypical tasks, and the information processing analyses of performance on those tasks suggest considerable conceptual overlap. We review three definitions of CPS: a general definition emerging from the human problem solving literature; a more specialized definition from the “German School” emphasizing performance in many-variable microworlds, with high domain-knowledge requirements; and a third definition based on performance in Minimal Complex Systems (MCS), with fewer variables and reduced knowledge requirements. We find a correlation of 0.86 between expert ratings of the importance of CPS and Gf across 691 occupations in the O*NET database. We find evidence that employers value both Gf and CPS skills, but CPS skills more highly, even after controlling for the importance of domain knowledge. We suggest that this may be due to CPS requiring not just cognitive ability but additionally skill in applying that ability in domains. We suggest that a fruitful future direction is to explore the importance of domain knowledge in CPS.

Appropriateness of action learning in the physical and virtual spaces: a discussion

Paramedics are autonomous health professionals who must graduate into the workforce with complex problem-solving skills, an ability to exercise critical thinking, and clinical reasoning skills to challenging situations, if they are to practise safely. Face-to-face action learning (AL) has long been acknowledged for contributing to the development of such skills, by providing a platform for structured thinking, group working, and a tool for developing participant's ability to ask insightful questions ( Pedler et al, 2005 ) and more recently, action learning in the virtual space is becoming increasingly popular. Virtual action learning has emerged for reasons including globalisation, where the emergence of multi-national and dispersed students has become commonplace ( Dickenson et al, 2010 ) and advances in technology that have allowed more collaborative communication.This discussion paper provides an introduction and background to action learning (AL), virtual action learning (VAL) and explores the appropriateness of these paradigms as a tool for embedding reflective practice and problem-solving skills among undergraduate student paramedics.

Complex Problems in Entrepreneurship Education: Examining Complex Problem‐Solving in the Application of Opportunity Identification

In opening up the black box of what entrepreneurship education (EE) should be about, this study focuses on the exploration of relationships between two constructs: opportunity identification (OI) and complex problem‐solving (CPS). OI, as a domain‐specific capability, is at the core of entrepreneurship research, whereas CPS is a more domain‐general skill. On a conceptual level, there are reasons to believe that CPS skills can help individuals to identify potential opportunities in dynamic and nontransparent environments. Therefore, we empirically investigated whether CPS relates to OI among 113 masters students. Data is analyzed using multiple regressions. The results show that CPS predicts the number of concrete ideas that students generate, suggesting that having CPS skills supports the generation of detailed, potential business ideas of good quality. The results of the current study suggest that training CPS, as a more domain‐general skill, could be a valuable part of what should be taught in EE.

Individual differences in students’ complex problem solving skills: How they evolve and what they imply.

Changes in the demands posed by increasingly complex workplaces in the 21st century have raised the importance of nonroutine skills such as complex problem solving (CPS). However, little is known about the antecedents and outcomes of CPS, especially with regard to malleable external factors such as classroom climate. To investigate the relations between CPS and other constructs, we had Finnish 6th-grade students complete a test battery that included CPS tasks, fluid reasoning, classroom climate, and academic outcomes such as school grades and academic potential (N = 1,670). The working memory test was administered to a subsample of students (N = 357). A latent multilevel analysis suggests that (a) fluid reasoning, working memory, and classroom climate influenced CPS skills, and (b) CPS skills exhibited some incremental value in explaining school grades after controlling for cognitive ability, although the largest part of CPS’ relations to the outcomes was due to its overlap with other cognitive abilities. Further, on the class level, classroom climate showed a significant indirect effect on school grades via its influence on between-class differences in CPS. On the basis of this pattern of results, we argue that classroom climate is likely to be an important antecedent of CPS skills. Hence, we suggest that future research further explore how CPS is related to malleable factors such as classroom climate and extend analyses on the predictive validity of CPS to include real-world outcomes beyond the academic setting. (PsycINFO Database Record (c) 2016 APA, all rights reserved)

Content validity of game-based assessment: case study of a serious game for ICT managers in training

Serious games foster the acquisition of complex problem-solving skills. Assessment of such skills should be in line with instruction, and within a serious game environment its content validity should equal face-to-face assessment. Research on assessment in serious gaming has remained rather scarce. This article shows how assessment can be implemented in serious gaming in a way that assures content validity. The core of the authors’ validation method entails mapping learning activities (as contained in the game scenario) on performance indicators and outputs (as derived from formal attainment levels). They present how they have elaborated and applied the method for an assessment game for ICT managers in secondary vocational education. They describe the procedure and extent to which this assessment is content-valid compared to face-to-face assessment.

Assessing creative potential as student outcome: On the applicability of the TCT-DP in repeated measures

Within a constantly changing society, the ability to act creatively in various domains is becoming increasingly important. Schools are expected to react to this by educating their students in complex problem-solving skills and creative thinking (OECD, 2014). Teachers are therefore asked to foster creativity, which implies a need to learn more about assessing creativity or creative potential. When evaluating programs that are designed to foster creativity in and out of school, it is necessary to assess creativity as a criterion, even though this is a challenging thing to do. The difficulties become particularly apparent when it comes to investigating creativity development. The present study uses the Test for Creative Thinking-Drawing Production (Urban & Jellen, 1996) to take a closer look at the measurability of creativity during childhood. German students were tested three times while at elementary school (at the beginning of the first grade and at the end of the second and fourth grade). We tested measurement equivalence and found partially strong measurement invariance in a subset of the rating categories suggested in the test’s manual. The proposed model is discussed theoretically and empirically, while keeping in mind that it is essentially a preliminary finding until replication studies have been conducted. The study indicates how creativity could be taken into account as a relevant student outcome in large-scale assessments.

Design and Implementation of the Game-Design and Learning Program

Design involves solving complex, ill-structured problems. Design tasks are consequently, appropriate contexts for children to exercise higher-order thinking and problem-solving skills. Although creating engaging and authentic design contexts for young children is difficult within the confines of traditional schooling, recently, game-design has emerged as an alternative context to provide young children with opportunities to practice design and thinking skills. Despite the increasing interest from educators and researchers to use game-design as a platform to teach young students higher-order thinking skills, literature documenting design and development of such learning experiences has been scarce. This paper provides a detailed account of how a complex network of pedagogies, theories, and technologies were brought together to design and develop the Game Design and Learning (GDL) program, with the purpose of teaching students basics of computer programming, and to give them hands-on experiences in game-design, and teach them complex problem-solving skills. The GDL program can serve as an example for efforts aiming to create similar technology-rich environments.

Complex problem solving and intelligence: A meta-analysis

The purpose of this meta-analysis is to examine the nature and magnitude of the relation between complex problem-solving skills (CPS) and intelligence, a topic that has been widely discussed and that has instigated a vast array of partially contradicting findings in the past. Theoretically, researchers have hypothesized the two constructs to be everything from completely separate to identical. Over the course of almost four decades, empirical studies yielded results in support of both arguments. Our meta-analysis of 47 studies containing 60 independent samples and a total sample size of 13,740 participants revealed a substantial correlation of CPS and intelligence with an average effect size of M(g)=.433. In addition, we investigated whether the operationalization of CPS and intelligence moderated this correlation. Whereas there were no significant correlation differences considering the operationalization of intelligence, the approach used to measure CPS moderated the correlation of CPS and intelligence. Especially the most recent approach towards the assessment of CPS yielded the strongest associations between the two constructs. Implications for existing theories and future research are discussed.

The complex route to success: complex problem-solving skills in the prediction of university success

ABSTRACTSuccessful completion of a university degree is a complex matter. Based on considerations regarding the demands of acquiring a university degree, the aim of this paper was to investigate the utility of complex problem-solving (CPS) skills in the prediction of objective and subjective university success (SUS). The key finding of this study was that CPS was not only strongly correlated to objective university success but also explained university grade point averages (GPA) of N = 78 German business students even after intelligence and high-school GPA were controlled for. In addition, priority setting while working on a complex problem explained self-rated SUS. Given this incremental validity over the most established predictors of objective university success and its versatility, we argue that CPS may be a valuable addition to existing predictors in university applicant selection.

What do employers pay for employees’ complex problem solving skills?

We estimate the market value that employers assign to the complex problem solving (CPS) skills of their employees, using individual-level Mincer-style wage regressions. For the purpose of the study, we collected new and unique data using psychometric measures of CPS and an extensive background questionnaire on employees’ personal and work history. The data were collected in 16 firms (23 establishments) in Germany, Spain, South Africa, Denmark, Slovakia, Switzerland, and France in the period 2012–2014. We find significant economic returns to CPS in our sample. One standard deviation higher CPS is associated with 10–20% higher hourly wages. The returns to CPS are sizeable even after controlling for fluid intelligence, suggesting that CPS probably captures skills important for modern production that are beyond what general intelligence tests can measure.

Extending the Assessment of Complex Problem Solving to Finite State Automata

Abstract. Recent advancements in the assessment of Complex Problem Solving (CPS) build on the use of homogeneous tasks that enable the reliable estimation of CPS skills. The range of problems featured in established instruments such as MicroDYN is consequently limited to a specific subset of homogeneous complex problems. This restriction is problematic when looking at domain-specific examples of complex problems, which feature characteristics absent from current assessment instruments (e.g., threshold states). We propose to utilize the formal framework of Finite State Automata (FSA) to extend the range of problems included in CPS assessment. An approach based on FSA, called MicroFIN, is presented, translated into specific tasks, and empirically investigated. We conducted an empirical study (N = 576), (1) inspecting the psychometric features of MicroFIN, (2) relating it to MicroDYN, and (3) investigating the relations to a measure of reasoning (i.e., CogAT). MicroFIN (1) exhibited adequate measurement characteristics and multitrait-multimethod models indicated (2) the convergence of latent dimensions measured with MicroDYN. Relations to reasoning (3) were moderate and comparable to the ones previously found for MicroDYN. Empirical results and corresponding explanations are discussed. More importantly, MicroFIN highlights the feasibility of expanding CPS assessment to a larger spectrum of complex problems.

Employability in the 21st Century: Complex (Interactive) Problem Solving and Other Essential Skills

Neubert, Mainert, Kretzschmar, and Greiff (2015) plea to integrate the 21st century skills of complex problem solving (CPS) and collaborative problem solving (ColPS) in the assessment and development suite of industrial and organizational (I-O) psychologists, given the expected increase in nonroutine and interactive tasks in the new workplace. At the same time, they promote new ways of assessing these skills using computer-based microworlds, enabling the systematic variation of problem features in assessment. Neubert and colleagues’ (2015) suggestions are a valuable step in connecting differential psychologists’ models of human differences and functioning with human resources professionals’ interest in understanding and predicting behavior at work. We concur that CPS and ColPS are important transversal skills, useful for I-O psychologists, but these are only two babies of a single family, and the domain of 21st century skills includes other families of a different kind that are also with utility for I-O psychologists. The current contribution is meant to broaden this interesting discussion in two important ways. We clarify that CPS and ColPS need to be considered in the context of a wider set of 21st century skills with an origin in the education domain, and we highlight a number of crucial steps that still need to be taken before “getting started” (Neubert et al., 2015, p. last page of the discussion) with this taxonomic framework. But first, we feel the need to slightly reframe the relevance of considering 21st century skills in I-O psychology by shifting the attention from narrow task-related skills to the broader domain of career management competencies.

The influence of higher order cognitive capacities on leader organizational continuance and retention: The mediating role of developmental experiences

The relationship between cognitive capacities and retention or turnover has received scant attention in the extant literature. The few findings that have been reported show mixed to no linear effects of general mental ability on organizational continuance. In this study, we examined the association of more specific higher order cognitive capacities including complex problem solving skills and divergent thinking with officer continuance in the U.S. Army. We also tested the role of developmental experiences as a partial mediator of this relationship. Our sample included 640 officers who completed measures of these skills and of their career experiences in 1992–1993. To this sample, we added years of service from date of commissioning to 2008, as well as data on whether officers experienced particular assignments considered to be challenging and developmental. Our findings support the association of complex problem solving and divergent thinking skills with leader continuance. We also found that this effect is partially mediated by challenging developmental experiences. Thus, we provide stronger evidence than in prior studies for a linear relationship between cognitive abilities and continuance in an organization.

Sometimes less is more: Comparing the validity of complex problem solving measures

The world we live in is becoming increasingly connected and dynamic. With this increase in complexity in people's everyday lives, there is a rising demand for instruments assessing the skills necessary to deal with complex situations. Therefore, both researchers and practitioners have frequently used measures of complex problem solving skills (CPS) since their development in the 1970s. These classical measures of CPS have been questioned because of their insufficient psychometric qualities. In response, newer measures of CPS with a stronger focus on psychometric qualities were developed. However, research comparing the validity of classical measures of CPS to the new measures is limited. The aim of this study is to compare classical measures of CPS to measures following the multiple complex systems approach (MCS), a new approach towards CPS assessment, with regard to their validity. Based on a university student sample (N=339), we compared the Tailorshop, an established classical measure of CPS to three different MCS tests (MicroDYN, Genetics Lab, MicroFIN). Examining construct validity, MCS tests correlated significantly higher with each other than with the Tailorshop and this difference was not attributable exclusively to a higher share of reasoning within the MCS tests. Further, in predicting school grades, MCS tests were found to have incremental validity over the Tailorshop but not over reasoning and the Tailorshop combined. These findings provide further empirical support for the proposition of MCS tests as valid alternatives to classical measures for the assessment of CPS and add to the literature on the relation between CPS and intelligence.

Student publishing: future scholars as change agents

Purpose – The purpose of this paper is to highlight undergraduates as an emergent student–scholar author group and to encourage institutions to take a future-oriented view, focusing greater attention to and support of undergraduate’s publishing. Design/methodology/approach – Highlighting benefits derived from undergraduate research (UR) experiences and publishing taken from the literature and experienced through local practice (Providence College), presenting pedagogical models for transforming students into independent thinkers (students as scholars) and responding to business and non-profit leader graduate skills requests of higher education, this paper argues for the need to cultivate graduate attributes (requisite 21st century workforce skills, abilities and behaviors), especially graduate demonstrated articulation and communication (publication) skills and abilities. Findings – The conclusions drawn in this paper align with the literatures’ support of derived benefits from UR experiences and its completion through articulation and communication (publication). Final remarks reiterate that critical thinking, complex problem-solving and communication (publication) skills and abilities demonstrate graduate agency and preparedness for meeting 21st century challenges. Originality/value – This paper layers several pedagogical engagement-based teacher–learner models, highlights benefits of undergraduates’ completing the research process through communication (publication) and underscores the importance of cultivating 21st century graduate agency.

Toward an integrated model for designing assessment systems: An analysis of the current status of computer-based assessments in science

Drawing upon an integrated model proposed by Bennett and Bejar (1998), this review study examined how 66 computer-based science assessments (CBSAs) in basic science and medicine took advantage of advanced technologies. The model regarded a CBSA as an integrated system, included several assessment components (e.g., assessment purpose, measured construct, test and task design, examinee interface, and scoring procedure), and emphasized the interplay among these components. Accordingly, this study systematically analyzed the item presentations of interactive multimedia, the constructs measured, the response formats in formative and summative assessments, the scoring procedures, the adaptive test activities administrated based on the algorithms related to the item response theory (IRT) and other rules beyond IRT, and the strategies for the automatic provision of informative hints and feedbacks in the CBSAs. Our analysis revealed that although only 19 out of 66 assessments took advantage of dynamic and interactive media for item presentations, CBSAs with these media could measure integrated understanding of science phenomena and complex problem-solving skills. However, we also found that limitations in automated scoring may lead to infrequent use of the automated provision of hints and feedbacks with open-ended and extended responses. These findings suggest the interrelatedness of the assessment components, and thus we argue that designers should repeatedly consider the relationships between components of CBSAs to ensure the validity of the assessments. Finally, we also indicate the issues for future research in computer-based assessments.

Computer-based assessment of Complex Problem Solving: concept, implementation, and application

Complex Problem Solving (CPS) skills are essential to successfully deal with environments that change dynamically and involve a large number of interconnected and partially unknown causal influences. The increasing importance of such skills in the 21st century requires appropriate assessment and intervention methods, which in turn rely on adequate item construction, delivery, and scoring. The lack of assessment tools, however, has slowed down research on and understanding of CPS. This paper first presents the MicroDYN framework for assessing CPS, which is based on linear structural equation systems with input and output variables and opaque relations among them. Second, a versatile assessment platform, the CBA Item Builder, which allows the authoring, delivery, and scoring of CPS tasks for scientific and educational purposes is introduced. Third, we demonstrate the potential of such a tool for research by reporting an experimental study illustrating the effect of domain specific content knowledge on performance in CPS tasks both on an overall performance and on a process level. The importance of accessible and versatile technical platforms not only for assessment and research but also for intervention and learning are discussed with a particular focus on educational contexts.

The Project Model of Clinical Education: Eight Pedagogical Principles to Maximize Student Learning and Social Justice Impact

In clinical legal education, there is growing interest in the development of project-based clinical work, which includes a broad range of activities, such as legislative and policy reform, community economic development and community legal education. In the project model, students use non-litigation advocacy strategies to solve challenging legal problems for clients and engage a broad range of multidimensional legal skills, including complex problem-solving, strategic planning, project management, and professional communication skills. Clinical scholarship on project-based learning has suggested that key pedagogical methods, particularly maximizing role assumption and student ownership of clinic work, must be compromised in projects due to the inherent complexity of the model. Because the project model holds such great potential for creating systemic change and serving communities, clinicians who develop projects often struggle to navigate the balance between social justice impact and pedagogical goals. In response to these and other challenges of projects in clinical education, this article argues that through intentional and goal-driven planning, clinicians can design project-based learning experiences that meet social justice goals while also maximizing student ownership and learning. To assist clinicians in developing projects that are successful from pedagogical and social justice perspectives, this article offers eight pedagogical principles, transferable across clinical contexts, for the design and supervision of this emerging model of clinical education.