Consensual Assessment Technique Research Articles

Creativity in sports: The crucial role of motor skills

What enables athletes to perform creative actions in their sports? In the last two decades, sports studies have predominantly assumed that divergent thinking (DT) is the key ingredient and have developed sport-specific DT tests to assess players’ creativity. However, a fundamental question remains: Do players who score high on sport-specific DT tests perform more creative actions on the field? The DT approach in sports has recently been challenged by the hypothesis that creative actions are primarily enabled (or limited) by players’ motor skill repertoires. Here, predictions based on the DT and motor skill approach are tested in the domain of soccer. Studies 1a and b tested male under-12 players, and Study 2 tested female under-19 players of a professional soccer club. All three studies indicated no correlation between the creativity of actions performed on the field (rated with the Consensual Assessment Technique) and players’ soccer-specific DT. In contrast, moderate to high correlations between on-field creativity and motor skills were found. These results challenge the DT approach in sports and the validity of sport-specific DT tests, highlighting the crucial role of sensorimotor skills in performing creative actions in sports or other domains in which creative performance is embodied through motor actions.

Development of an artificial intelligence curriculum design for children in Taiwan and its impact on learning outcomes

In the digital age, the application of Artificial Intelligence (AI) has become an irreversible trend, with its potential in the field of education being particularly noteworthy. However, there are currently few AI education programs for children in Taiwan, and there is a lack of systematic teaching resources and methods, which poses a major challenge to the promotion of AI education. To address this challenge, this study designed a tailor-made AI curriculum for children in Taiwan, aimed at enhancing their foundational knowledge in the AI field and skills in using generative AI. This study was conducted in Taiwan, involving 30 elementary school students from grades 3 and 4, employing a single-group pre-test and post-test research design. Data were collected and analyzed through pre-and post-tests on AI knowledge quizzes and AI knowledge self-assessment, as well as the expert consensual assessment technique and the pupils’ attitude toward technology survey questionnaire. Research shows that students who participated in the AI course significantly improved their test scores on AI knowledge before and after the course. The students AI knowledge increased by 62.75%, demonstrating the course’s effectiveness and showing a positive attitude towards AI technology. Additionally, the student’s project outcomes demonstrated a high level of creativity. The students exhibited an enhanced interest and positive attitude towards learning AI, expressing a willingness to participate in more AI educational courses. This work provides valuable experience and guidance for the future integration of AI technology in children’s education, offering practical guidelines for teachers and researchers on how to effectively teach AI knowledge, as well as serving as a robust reference for educational policy makers in formulating strategies to promote AI education.

The relationship between creativity and language as measured by linguistic maturity and text production

Creativity is the result of several converging elements, including cultural, environmental, and social factors. The context in which people live their lives not only affects their creativity; it also affects how they acquire and use language. This study shows how a person's context and linguistic development, as measured by linguistic maturity, affects their creativity. 249 students aged between 13 and 15, participated in the study by completing an online test measuring their linguistic maturity and creativity. This study used Guilford's (1967) Alternative Uses Test following the Consensual Assessment Technique. Regression models revealed a positive and significant influences of linguistic maturity on creativity. Greater semantic memory networks allow for more efficient searches, thus enabling creativity. Therefore, greater semantic richness leads to greater linguistic maturity, resulting in increased creativity. Our findings confirm a relationship between language skills and creativity (i.e., linguistic maturity and text production) at a deeper level than previously studied. This is relevant for both curriculum development and instructional design. In this sense, tools as ChatGPT that produce text from millions of documents, push the student to be mediator of all this linguistic richness. Furthermore, measuring linguistic maturity and text production provides another means to measure creativity, transcendent, considering that having proper approaches to measure creativity is a challenge.

Stability of Assessment Using Creativity‐Related Indices

AbstractThe aim of this study is to determine how many raters are required for the design products proposed in this study to obtain stable ratings of creativity over time using the Consensual Assessment Technique (CAT) and to examine the creativity subscales, namely, originality and practicality, and their composite scores in a similar manner. The results revealed that creativity assessment using the CAT was stable over time if the assessment was conducted by about 10 to 15 raters. On the contrary, originality and practicality assessment was found to be stable over time with about three raters. However, when the sum of the originality and practicality scores was used as a composite score, the assessment was found to be unstable. This could be attributed to the negative correlation between the originality and practicality assessments of the products. Further, when the product of the originality and practicality scores or the Euclidean distance within these dimensions was used, stable scores were obtained over time by using about 15 to 20 raters.

Assessing Creativity in the Language Classroom with the Consensual Assessment Technique

ABSTRACT In this study, The Consensual Assessment Technique is used to measure the creativity of 100 teaching artifacts from world language classrooms. These teaching artifacts were self-selected by K-12 world language teachers and categorized as creative or uncreative. The first set of artifacts was collected prior to the COVID-19 pandemic, in spring of 2019; the second set of artifacts was collected during the COVID-19 pandemic, in the winter of 2020, after several months of online teaching. The two sets of artifacts were randomly mixed and content experts rated the creativity of the 100 total artifacts. High levels of inter-rater reliability were obtained. Results indicated that creativity was not impacted by the move to online instruction during the pandemic and that use of target language (e.g. Spanish, French) may have a role to play in creativity.

Examining the effects of creativity, collaboration, creative diversity and autonomy on team creative performance

The present study investigated the multidimensional relationships among some of the factors which affect the creative performance of teams, including creativity, collaboration, diversity, and autonomy in educational settings. Two hundred and thirty one (231) university students of Informatics worked in teams for a whole semester, and their projects were evaluated for creativity with the Consensual Assessment Technique. Structural equation modeling and path analysis with bootstrapping were used to analyze the collected data and examine relationships. According to the results, the creative diversity of teams had a significant positive relationship with their creative performance, whereas the aggregated individual creativity of members did not have a significant effect. Collaboration among team members had a positive impact on the teams’ creative outputs and also partially mediated the relationship between creative diversity and creative performance. Autonomy of members correlated significantly with their creativity, however, at the team level, there was not a significant relationship between autonomy and creative performance. Theoretical and practical implications are discussed and future research directions are proposed.

How powerful are Chinese characters' effects on children's creativity?

This paper conducts a preliminary evaluation of a learning strategy for Chinese native speakers to learn Chinese characters more effectively. The study utilizes a newly developed creative instruction approach called a Two-Stage learning Model (TSLM) and investigates its effects on students' verbal creativity and creative writing. The researchers developed this learning strategy based on children's self-generated stories, rhymes, and pictures of new Chinese characters they are learning. Learning in this manner is a natural approach because the Chinese people developed characters from drawings of natural objects, items from everyday life, and abstract concepts. This study adopted a quasi-experimental design with 54 participants (27 boys and 27 girls) in the 4th grade in an urban elementary school in Taiwan. The experimental group underwent TSLM instruction for eight weeks, and the control group received regular instruction. Participants were asked to use imagination and draw a character's picture, then explain their drawings by writing stories or making rhymes of the assigned characters. Three quantitative instruments were applied: New Creative Thinking Test (NCTT) for measuring divergent-thinking skills; Consensual Assessment Technique (CAT) for evaluating creative products; and Chinese characters' recall test for assessing the number of characters recalled. Results indicated enhancement of three constructs of NCTT: fluency, flexibility, and originality. Moreover, they showed that children who underwent treatment scored higher on the creative writing tasks; the recall test results were insignificant suggesting that the difference between groups in creative performance was due to TSLM instruction. The study provides the first demonstration in real educational settings of how Chinese characters could promote verbal creativity.

Artificial intelligence as relational artifacts in creative learning

ABSTRACT Artificial Intelligence (AI) has significantly advanced in creating professional-level media content. In creative education, determining how students can benefit without becoming dependent on them is a challenge. In this study, researchers conducted an exploratory experiment that positioned AI as a relational artifact to students in a series of drawing activities and examined the potential impact of affective relations with machines in socio-cultural creative learning. The resulting artifacts, observations, and interview transcripts were analyzed using the Consensual Assessment Technique and a grounded theory approach. The study's results indicate that the design professors reliably evaluated the student drawings as more creative than the AI drawings, but neither demonstrated a consistent increase in creativity. However, the presence of AI engaged the students to explore different approaches to artistic prompts. We theorize that AI can be mediated as a learning artifact for transformative creativity if the students perceive their relationship with AI as empathetic and collaborative.

Associations between drawing creativity, task-related divergent thinking, and other creative subprocesses

Although the majority of research investigating the creative process has focused on divergent thinking, few studies have explored the role task-related divergent thinking has in creative production. Due to this emphasis on divergent thinking, many other cognitive processes relevant to creativity remain underresearched. This study sought to determine the associations between task-related measures of divergent thinking and expert ratings of drawing creativity. Additionally, this study explored whether the creativity of drawings based on self-generated ideas would be enhanced by an elaborative strategy involving the exploration of an original idea, evaluation of the intermediate status of a drawing, and subsequent refinement of creative goals. Undergraduate university students (N =107) were divided into two groups: a treatment group that was prompted to engage in stages involving exploration, evaluation, and goal refinement and a control group that engaged in alternative activities not presumed to enhance drawing creativity. Using the Consensual Assessment Technique, seven judges trained in the visual arts provided ratings of drawing creativity. Of the different divergent thinking measures explored, the average originality of generated ideas, ideational flexibility, and novelty of selected ideas were most strongly associated with creativity ratings, and an interaction indicated that selected idea novelty was a stronger predictor among control group participants. Although this interaction illustrated a difference between control and treatment groups, creativity ratings did not vary across groups, suggesting that merely prompting exploration, evaluation, and goal refinement is not sufficient to improve drawing creativity.

Gauging Ghanaian secondary visual arts teachers' levels of creativity: Four C trajectory of creativity

Drawing on the Four C Model of Creativity and Consensual Assessment Technique (CAT) as frameworks, the multi-site qualitative case study explored the creative practice of 16 Ghanaian secondary visual arts teachers in the Sekondi-Takoradi Metropolis, as the 2019 National Pre-Tertiary Curriculum Framework mandates creativity. The researchers used the interpretivist and constructivist paradigms for the study, which employed interviews, documents review and participant observations to gather data from eight secondary schools. The data were analysed using thematic analysis, with two categories of teachers emerging: teacher-practitioners and non-teacher-practitioners. The practising teachers were categorised under little-c and Pro-c—none fell within the Big-C category. Practising teachers were seen to be more successful in enacting creativity in their teaching practice, meeting the aims of the curriculum framework. We recommend that education providers and stakeholders should encourage the artistic practice of visual arts teachers and make it an integral part of professional practice.

Children’s Imagination of the Divine: Creativity Across Religions

ABSTRACT Earlier studies have concluded that the religious sphere is less open to creativity than are other areas of human activities. Also, it has been suggested that artistic freedom and creative expression are unwelcome in the domain of religious iconography. In the present study, we address this subject with regard to children’s artistic expression of the divine (n = 1703) in five different cultural and religious environments: Japanese (Buddhism and Shinto), Iranian (Islam), Russian-Buryat (Buddhism, shamanism), Russian Slavic (Christian Orthodoxy) and French-speaking Swiss (Catholic and reformed Christianity). The Consensual Assessment Technique was used to access the creativity. The results globally supported the previous research on the positive role that domain knowledge plays in creativity. It is likely that the exposure to knowledge in the religious domain through education would not negatively impact on children’s creativity in the domain of pictorial representations of the divine, at least in secularized societies. Results also revealed that a child’s creative expression could be a function of his or her age. Obviously, with age and knowledge of the religious domain, older children would go more often beyond the simplistic or conventional representations of the divine. As for gender, no effect was found in four out of five samples.

The manipulation of task constraints and the effect on creative performance of youth soccer players

Creativity is a popular topic that has been increasingly researched over the years. Recently, creativity is suggested as a relational concept (Zahno & Hossner, 2020). This conceptualization implies that creativity is not solely an individual ability. By contrast, creative actions emerge in the athlete’s search to adapt to task and environmental constraints (Orth et al., 2017; Withagen & van der Kamp, 2018). Manipulating task constraints that serve to increase variability might, therefore, evoke creative actions. Such manipulations in soccer are often conducted in training situations with small sided games. This was recently shown with a numerical assessment of creative actions comparing small sided soccer games (SSGs) with 11v11 matches in elite adult soccer players (Caso & van der Kamp, 2020). They found that players performed more actions, a broader action repertoire and more creative actions in smaller formats (5v5, 6v6 and 7v7) than in an 11v11 format. The purely quantitative method of Caso & van der Kamp (2020), however, uses an arbitrarily 5% benchmark for originality. More importantly, although subjective, the coaches opinions are not meaningless and should not be neglected.
 Our research examined whether the latter task manipulation also increased the action repertoire and creative performance of elite youth soccer players (O11-O13). A numerical method was introduced, assessing creative performances as a frequency of the action types emerging in an environment. A notational video-analysis identified all (not-) functional actions. Next, a threshold defined whether the action was original. This threshold has been determined with expert coaches ranking action types following the procedure of the consensual assessment technique (CAT; Amabile, 1996). If the action was both original and functional, then it was defined a creative action. The data consisted of 293 minutes small sided game (4v4) and 263 minutes regular sided game (11v11). Results indicated that small sided games hold relative more actions, more functional action and more creative actions compared with regular sided games. This is in line with previous studies, that small sided games induce more variation of actions which increase the possible emergence of creative actions (Orth et al., 2017).
 References
 Amabile, T. M., Amabile, T. M., Collins, M. A., Conti, R., Phillips, E., Picariello, M., Ruscio, J., & Whitney, D. (1996). Creativity in Context: Update to The Social Psychology of Creativity (1st ed.). Routledge.
 Caso, S., & van der Kamp, J. (2020). Variability and creativity in small-sided conditioned games among elite soccer players. Psychology of Sport and Exercise, 48, Article 101645. https://doi.org/10.1016/j.psychsport.2019.101645
 Orth, D., van der Kamp, J., Memmert, D., & Savelsbergh, G. J. P. (2017). Creative motor actions as emerging from movement variability. Frontiers in Psychology, 8, Article 1903. https://doi.org/10.3389/fpsyg.2017.01903
 Withagen, R., & van der Kamp, J. (2018). An ecological approach to creativity in making. New Ideas in Psychology,49, 1–6. https://doi.org/10.1016/j.newideapsych.2017.11.002
 Zahno, S., & Hossner, E.-J. (2020). On the issue of ceveloping creative players in team sports: A systematic review and critique from a functional perspective. Frontiers in Psychology, 11, Article 575475. https://doi.org/10.3389/fpsyg.2020.575475

Is Curiosity Killed by the CAT? A Divergent, Open-Ended, and Generative (DOG) Approach to Creativity Assessment

ABSTRACT The consensual assessment technique (CAT) represents one of the most popular evaluation techniques used by researchers to assess creative artifacts. In this paper we discuss how the prototypical use of the CAT, while useful for identifying unambiguous examples of creative artifacts, can inadvertently kill the curiosity of researchers interested in interpreting and understanding more ambiguous, contested, and divergent examples of creative expression. More specifically, we open the article by briefly describing the CAT and how it can simultaneously help and hinder researchers in making judgments about the creativity of artifacts in and across domains. We then introduce a methodological elaboration on the CAT, called the Divergent, Open-Ended, and Generative (DOG) approach to interpreting creative artifacts. Next, we present a secondary analysis of CAT data to illustrate the DOG approach and how it can extend insights about creativity when used in conjunction with the CAT. We close with a discussion of implications for creativity theory and research.

A study of the evaluation metrics for generative images containing combinational creativity

AbstractIn the field of content generation by machine, the state-of-the-art text-to-image model, DALL⋅E, has advanced and diverse capacities for the combinational image generation with specific textual prompts. The images generated by DALL⋅E seem to exhibit an appreciable level of combinational creativity close to that of humans in terms of visualizing a combinational idea. Although there are several common metrics which can be applied to assess the quality of the images generated by generative models, such as IS, FID, GIQA, and CLIP, it is unclear whether these metrics are equally applicable to assessing images containing combinational creativity. In this study, we collected the generated image data from machine (DALL⋅E) and human designers, respectively. The results of group ranking in the Consensual Assessment Technique (CAT) and the Turing Test (TT) were used as the benchmarks to assess the combinational creativity. Considering the metrics’ mathematical principles and different starting points in evaluating image quality, we introduced coincident rate (CR) and average rank variation (ARV) which are two comparable spaces. An experiment to calculate the consistency of group ranking of each metric by comparing the benchmarks then was conducted. By comparing the consistency results of CR and ARV on group ranking, we summarized the applicability of the existing evaluation metrics in assessing generative images containing combinational creativity. In the four metrics, GIQA performed the closest consistency to the CAT and TT. It shows the potential as an automated assessment for images containing combinational creativity, which can be used to evaluate the images containing combinational creativity in the relevant task of design and engineering such as conceptual sketch, digital design image, and prototyping image.

Comparing Expert, Quasi-Expert, and Novice Evaluations of Award-Winning Design Products Using the Consensual Assessment Technique

ABSTRACT The consensus-based assessment has long been a prevalent methodology employed in “panel crit” settings in design education and professional design awards. Acknowledging the subjective nature of design evaluation and its importance to the design development process, we report two studies investigating the effect of three levels of design expertise (expert, quasi-expert, and novice) on the evaluation of award-winning lamp designs using the consensual assessment technique (CAT). The pilot study (N = 30) preliminarily examined expertise-related differences in evaluation outcome (i.e., ratings and rankings) and the use of the CAT instrument. In a follow-up study (N = 135), we tested four hypotheses using the refined CAT instrument. Results showed a significant interaction effect between expertise group and design on evaluation. Correlations between groups showed that quasi-experts were more similar to novices than experts. This finding contributes to the discussion on whether quasi-experts should be considered proper substitutes for experts. Furthermore, the reliability of ratings on individual CAT scale items varied among groups, suggesting that the degree to which raters were consistent within a group depends upon which aspect of design quality they were assessing. Taken together, the findings provide valuable insights into the influence of design education and experience on judging design quality.

Relationship between personality types and creativity: A study on novice architecture students

The first milestone and the foundation in every architect’s life is architectural education. As creativity is the core of successful architecture, focusing on its development in the pedagogical years would enhance every architect’s future career. Architectural education tends to give a one-size-fits-all approach to educating architects and helping with their creative potential, which neglects the students’ different levels of creativity along with their personalities’ individuality. This study aims to enhance each architect’s architectural pedagogical experience by acknowledging the student’s personality and creativity level which would help tailor a customized experience that would develop their full potential. This will be achieved by examining the relationship between personality types and creativity through the Myers-Briggs Personality Type Indicator (MBTI) and the Consensual Assessment Technique (CAT). The study is employed in the Architecture Department, Alexandria University - Egypt, where questionnaires were distributed to first- and second-year students along with design projects to be assessed by Architectural Department professors. Responses were analyzed using Pearson’s Chi-square analysis with Fisher’s exact test. The results indicated a strong connection between Intuition/Sensing and the creativity of students.

The Relationships between Cognitive Styles and Creativity: The Role of Field Dependence-Independence on Visual Creative Production.

Previous studies explored the relationships between field dependent-independent cognitive style (FDI) and creativity, providing misleading and unclear results. The present research explored this problematic interplay through the lens of the Geneplore model, employing a product-oriented task: the Visual Creative Synthesis Task (VCST). The latter requires creating objects belonging to pre-established categories, starting from triads of visual components and consists of two steps: the preinventive phase and the inventive phase. Following the Amabile’s consensual assessment technique, three independent judges evaluated preinventive structures in terms of originality and synthesis whereas inventions were evaluated in terms of originality and appropriateness. The Embedded Figure Test (EFT) was employed in order to measure the individual’s predisposition toward the field dependence or the field independence. Sixty undergraduate college students (31 females) took part in the experiment. Results revealed that field independent individuals outperformed field dependent ones in each of the four VCST scores, showing higher levels of creativity. Results were discussed in light of the better predisposition of field independent individuals in mental imagery, mental manipulation of abstract objects, as well as in using their knowledge during complex tasks that require creativity. Future research directions were also discussed.

Creativity Ratings of Fashion Outfits Presented on Instagram: Does Gender Matter?

Abstract Rationale: Creativity assessment can be influenced by rater characteristics, including social group membership, such as gender. As raters are often male, the gender composition of rater panels in the Consensual Assessment Technique (CAT) could introduce unintended implicit biases into this measurement methodology. The present study analyzed such biases by examining gender differences in creativity assessment. Method: We applied the CAT and asked male (n = 26) and female (n = 39) judges to rate the creativity of fashion outfits presented on Instagram. We then examined gender differences in mean creativity ratings and rater consistency (inter-rater reliability). In an additional qualitative analysis, we analyzed implicit theories of creativity of female and male raters by comparing the criteria that these raters applied when assessing creativity. Results: We found no systematic support for gender differences in the level of creativity ratings, but observed that rating consistency was significantly higher for female than for male judges. Additional content analysis suggested that female and male raters attached different relative importance to various assessment criteria, indicating gender differences in rating criteria. Discussion: Our study suggests that rater panel composition can indeed affect aspects of creativity assessment, although we do not obtain strong support for a gender-related bias in the CAT methodology.

If a Picture is Worth 1000 Words, Is a Word Worth 1000 Features for Design Metric Estimation?

Abstract A picture is worth a thousand words, and in design metric estimation, a word may be worth a thousand features. Pictures are awarded this worth because they can encode a plethora of information. When evaluating designs, we aim to capture a range of information as well, including usefulness, uniqueness, and novelty of a design. The subjective nature of these concepts makes their evaluation difficult. Still, many attempts have been made and metrics developed to do so, because design evaluation is integral to the creation of novel solutions. The most common metrics used are the consensual assessment technique (CAT) and the Shah, Vargas-Hernandez, and Smith (SVS) method. While CAT is accurate and often regarded as the “gold standard,” it relies on using expert ratings, making CAT expensive and time-consuming. Comparatively, SVS is less resource-demanding, but often criticized as lacking sensitivity and accuracy. We utilize the complementary strengths of both methods through machine learning. This study investigates the possibility of using machine learning to predict expert creativity assessments from more accessible nonexpert survey results. The SVS method results in a text-rich dataset about a design. We utilize these textual design representations and the deep semantic relationships that words and sentences encode to predict more desirable design metrics, including CAT metrics. We demonstrate the ability of machine learning models to predict design metrics from the design itself and SVS survey information. We show that incorporating natural language processing (NLP) improves prediction results across design metrics, and that clear distinctions in the predictability of certain metrics exist. Our code and additional information about our work are available on the MIT DeCoDE Lab website.1

Development of the R library “jrt”: Automated item response theory procedures for judgment data and their application with the consensual assessment technique.

Development of the R library “jrt”: Automated item response theory procedures for judgment data and their application with the consensual assessment technique.