Collegiate Learning Assessment Research Articles

How do self-regulation and effort in test-taking contribute to undergraduate students’ critical thinking performance?

ABSTRACT Critical thinking is a multifaceted construct involving a set of skills and affective dispositions together with self-regulation. The aim of this study was to explore how self-regulation and effort in test-taking contribute to undergraduate students’ performance in critical thinking assessment. The data were collected in 18 higher education institutions in Finland. A total of 2402 undergraduate students at the initial and final stages of their bachelor degree programmes participated in the study. An open-ended performance task, namely the Collegiate Learning Assessment (CLA+) International, was assigned to assess students’ critical thinking, and a self-report questionnaire was used to measure self-regulation and effort in test-taking. Information on test-taking time was also utilised in the analysis. The interrelations between the variables were analysed with correlations and structural equation models. The results indicate that self-regulation in test-taking has only indirect effects on critical thinking performance task scores, with effort and time as mediating variables. More precisely, planning contributed to critical thinking performance indirectly through test-taking time and effort, while monitoring had no significant relation to critical thinking performance. The findings did not differ between the initial-stage and final-stage students. The model explained a total of 36% of the variation in the critical thinking performance task scores for the initial-stage students and 27% for the final-stage students. The findings indicate that performance-based assessments should be carefully designed and implemented to better capture the multifaceted nature of critical thinking.

Effectiveness of Collegiate Learning Assessment Task-based Ebook to Empower Critical Explanatory with Different Problem-Solving Style Types

Effectiveness of Collegiate Learning Assessment Task-based Ebook to Empower Critical Explanatory with Different Problem-Solving Style Types

Linguistic, Contextual, and Experiential Equivalence Issues in the Adaptation of a Performance-Based Assessment of Generic Skills in Higher Education

This qualitative study investigated the various linguistic, contextual, and experiential equivalence issues embedded in a performance-based instrument aimed at assessing generic skills in higher education. A rigorous translation and adaptation process (American English to Finnish) was conducted on one instrument, namely Collegiate Learning Assessment (CLA+) International. The data were obtained from cognitive laboratories (n = 13), with think-alouds and follow-up interviews conducted among Finnish undergraduate students. Content logs were created, and the data were analyzed thematically. The findings revealed that linguistic and contextual equivalence issues were more prominent than experiential ones. The findings underline how important – and potentially problematic – it is for a test to measure the same construct in a different language and culture. To achieve adequate measurement equivalence, a detailed qualitative analysis of linguistic, contextual, and experiential equivalence should be conducted as part of test adaptation.

The dynamic relationship between response processes and self-regulation in critical thinking assessments

Our aim was to explore higher education students’ response and self-regulatory processes plus the relationship between these, as evidenced in two types of performance-based critical thinking tasks included in the Collegiate Learning Assessment (CLA+) International instrument. The data collection consisted of 20 cognitive laboratories. The data were analyzed using a qualitative approach. The tasks were found to trigger different response and self-regulatory processes. Overall, the performance task evoked more holistic processes than the selected-response questions, in which students’ processes were more question-oriented. The results also indicated the entanglement of students’ response and self-regulation processes. Three self-regulation groups were identified. Students with versatile self-regulation skills were able to complete the task thoroughly, whereas students with moderate self-regulation skills faced challenges in monitoring and evaluating their performance. Students who were lacking in self-regulation struggled both with the task as a whole and their own progress. Implications for higher education are discussed.

Exploring internal structure of a performance-based critical thinking assessment for new students in higher education

Critical thinking is a combination of complex cognitive skills. It is important for the successful acquisition of disciplinary skills in higher education and thus, it is a valuable competency for a new student. The complex nature of critical thinking leads to challenges for its assessment even in performance assessments such as CLA + International (Collegiate Learning Assessment). The aim with this study is to examine internal associations of a critical thinking assessment for new students in higher education. The sample consisted of 1469 first-year students in 18 higher education institutions in Finland. An open-ended performance task and multiple-choice tasks were used to assess six measures of critical thinking, namely analysis and problem solving, writing effectiveness, writing mechanics, scientific and quantitative reasoning, critical reading and critiquing an argument. Exploratory and confirmatory factor analyses were used to analyse the data. A latent component indicated by the measures derived from the performance task was identified. The measures derived from the multiple-choice tasks did not form a factorial structure. Multiple-choice questions are problematic in critical thinking assessment as they focus on individual skills instead of holistic use of skills. Implications for assessment development and higher education are discussed.

Correlates of the (soft) skills of Italian undergraduate students

Our research investigates the correlates of the soft skills owned by Italian undergraduate students (first-cycle degree programmes) at almost the end of their academic career. We take advantage of a unique data set provided by the National Agency for the Evaluation of the University and Research Systems, which contains information from both the adaptation of the US Collegiate Learning Assessment Plus test to the Italian context and the evaluation of the research quality of Italian universities. The research derives and discusses a set of relationships between test scores and some observed characteristics of students, institutions and study programmes. Results draw a general picture of the students who are preparing to enrol in second-cycle degree programmes or to enter the labour market, and show how – along with individual characteristics – institutional and socio-economic backgrounds affect students’ human capital endowments.

Exploring the Relationship between the Collegiate Learning Assessment, Student Learning Activities, and Study Behaviors: Implications for Colleges and Universities

Globally, institutions of higher learning have attempted to utilize various methods to assess student learning outcomes and simultaneously determine what factors influence targeted performance measures. The National Survey of Student Engagement is a popular instrument many colleges and universities employ to gain an understanding of what student behaviors are linked to such desired outcomes as graduation and persistence. Internationally, other universities are exploring the concept of student engagement as a means to assess college environments. Recently, calls from stakeholders have prompted institutions to use assessments to demonstrate so-called twenty-first century skills such as critical thinking; the Collegiate Learning Assessment is one such tool. This study reports preliminary efforts to link the NSSE engagement indicators to CLA performance at a medium-sized Historically Black College or University. Results indicate that the NSSE indicators are (1) poor predictors of student GPA, (2) of the ten indicators of the NSSE, only one was found to be a significant predictor of CLA performance, and (3) of the items comprising this indicator, only half are associated with CLA outcomes.

Engagement and Higher Order Skill Proficiency of First and Second Year Medical Students: A Comparison between IUSM's Legacy and Reformed Curricula

A large, multicampus, public medical school underwent curricular renewal emphasizing a student‐centered approach with 50% of all course contact time devoted to active learning. Determining the impact of active learning on student engagement and higher order skill (HOS) proficiency was the primary aim of this study. The study compared levels of engagement and HOS between 1) students participating in the legacy curriculum (LC) to those in the renewed curriculum (RC); 2) students in their first year (MS1) of medical school to their second year (MS2). Engagement was determined using a validated Survey of Student Engagement (Ahlfeldt, 2005). HOS proficiency was assessed using the Collegiate Learning Assessment (CLA+), professionally developed and validated by the Council for Aid to Education (https://cae.org/flagship‐assessments‐cla‐cwra/cla/). Results indicated that MS1 students in the RC demonstrated significantly higher engagement than LC students (39.0±7.0 vs 36.3±5.3), and continued to increase engagement significantly from MS1 to MS2 (40.8±5.3). However, there were no differences in HOS proficiency when comparing RC to LC, or MS1 to MS2. Interestingly, RC MS1 students in the lowest HOS quartile (1688.8±53.1) demonstrated significantly increased HOS when re‐tested during MS2 (1809.5±86.8). This phenomenon was not seen in LC students. Implementation of RC resulted in higher levels of student engagement than LC, but no differences in HOS were found. These results may be explained by a “ceiling” effect present in this group of highly developed learners.Support or Funding InformationNAThis abstract is from the Experimental Biology 2019 Meeting. There is no full text article associated with this abstract published in The FASEB Journal.

Assessing Student Engagement and Higher Order Skill Proficiency During the First and Second Year of Medical School – A Comparison Between the Legacy and Recently Reformed Active Learning Curricula

Background and Hypothesis: This study set-out to determine if: 
 
 students enrolled in Indiana University’s reformed curricula (RC) demonstrate higher levels of engagement (E) and higher order skill (HOS) proficiency than students prior to reform in the legacy curriculum (LC). 
 
 
 students increase E and HOS from first year of medical school (MS1) to second year (MS2). 
 students performing in lowest HOS quartile during MS1 will demonstrate greater gains in HOS by MS2 than students in higher quartiles. 
 
 Experimental Design or Project Methods: Determined E using a validated self-report survey (Ahlfeldt, 2007). Assessed HOS using the standardized Collegiate Learning Assessment (CLA+), professionally developed and validated by the Council for Aid to Education (https://cae.org/flagship-assessments-cla-cwra/cla/). Statistical analysis was preliminary; further analysis to be completed by statistician. Between group comparison of LC and RC via t test assuming unequal variance; paired t test for within group comparison MS1 to MS2 (significance p<0.05). 
 Results: Students in RC increased E significantly from MS1 (39.0±7.0) to MS2 (40.8±5.3) and demonstrate significantly higher E than LC MS2 students (36.3±5.3). There were no differences in HOS proficiency when comparing RC to LC, or MS1 to MS2. Students in RC in the lowest quartile for HOS during MS1 (1688.8±53.1) significantly increased when re-tested during MS2 (1809.5±86.8). 
 Conclusion and Potential Impact: Curricular reform resulted in higher E when compared to LC. Despite increased E, there were no related changes in HOS. Results from quartile analysis agreed with previous reports that active learning preferentially benefits lower performing students (Koles, 2010).

AN ENGINEERING DESIGN COURSE TO DEVELOP AND ASSESS CRITICAL THINKING AND PROBLEM SOLVING

A challenging new engineering design course is developed as part of the Engineering Design and Practice Sequence in the Civil Engineering program. This course engages students in a cyclical design process where they plan, build, test, and evaluate a model-scale tidal current turbine. They then use their own observations and analysis to iteratively inform, improve and re-test their design.The two objectives of this paper are to provide a description of the development and structure of this design course, and to assess student learning. The Final Design Reports were externally evaluated using the Valid Assessment of Learning in Undergraduate Education rubrics. Students also completed a standardized test called the Collegiate Learning Assessment as an objective evaluation of longitudinal learning gains. The Civil Engineering students demonstrated significant improvement in critical thinking, problem solving, and written communication skills.

Comparison of standardized assessment methods: logistics, costs, incentives and use of data

Critical thinking, problem solving and communication are fundamental elements of undergraduate education, but methods for assessing these skills across an institution are susceptible to logistical, motivational and financial issues. Queen’s University conducted two research studies investigating the use of standardised tests to assess cognitive skill development across the institution. Synthesis of results from implementing the Collegiate Learning Assessment (CLA+), the Critical Thinking Assessment Test (CAT), and the HEIghtenTM test found that student test effort was a significant factor, and effort level correlated with performance at r = .33. Test incentives were also a significant factor; effort levels for the $25 financial incentive group were one standard deviation higher than effort for the in-class test group. A dedicated computer lab was the preferred option for computer-based testing. A paper-based test was found to be much simpler to administer, but test results were not available for a long time, therefore limiting the usefulness of data. The true cost of tests was greater than the price of the instrument; recruitment, training, proctoring and marking costs need to be included in the calculation. Generally speaking, alignment of test objectives with student or course objectives, and timeliness of data, were key for participation and motivation.

Engagement and higher order skill proficiency of students completing a medical physiology course in three diverse learning environments.

The primary aim of this study was to determine whether levels of student engagement, higher order skill proficiency, and knowledge acquisition demonstrated by medical students would differ when completing the same course in three diverse learning environments. Following Institutional Review Board approval, 56 first-year medical students, registered at the same medical school but attending class at three different campus centers, were enrolled in the study. All participants were completing a medical physiology course that utilized the same learning objectives but relied on different faculty incorporating diverse methodologies (percentage of class devoted to active learning strategies), course format (6-wk block vs. 17-wk semester), and student attendance. Students completed a validated survey of student engagement (SSE), a proctored online problem-based assessment of higher order skill proficiency [Collegiate Learning Assessment (CLA+); http://cae.org ], and the National Board of Medical Examiners (NBME) Physiology subject exam. In this limited sample, results indicate no significant differences between campus sites for any of the variables assessed. Levels of engagement were lower than expected compared with published values for graduate students. Higher order skill proficiency assessed by CLA+ was significantly higher than values reported for college seniors nationally. Surprisingly, SSE offered no prediction of performance on CLA+ or NBME, as there were no significant correlations between variables. These data indicate that, although first-year medical students may not perceive themselves as highly engaged, they are adept in using higher order skills and excel in meeting course learning objectives, regardless of learning environment.

Determining Impact of Learning Environment on Student Engagement, Bloom's Higher Order Skill Proficiency, and Exam Performance of First Year Medical Students

The primary aim of this study was to determine if levels of student engagement, higher order skill proficiency, and knowledge acquisition demonstrated by medical students would differ when completing the same course in three diverse learning environments. Following IRB approval, 54 first year medical students registered at the same medical school, but attending class at three different campus centers were enrolled in the study. All participants were completing a Medical Physiology course that utilized the same learning objectives but incorporated diverse methodologies (percent of class devoted to active learning strategies), course format (6 week block versus 17 week semester), and student attendance rates. Students completed a validated survey of student engagement (SSE), a proctored online problem‐based assessment of higher order skill proficiency (Collegiate Learning Assessment (CLA+) (http://cae.org), and the National Board of Medical Examiners (NBME) Physiology subject exam. Results indicated levels of engagement were lower than expected when compared to published values for graduate students. Higher order skill proficiency assessed by CLA+ was significantly higher than values reported for college seniors nationally. Importantly, there were no significant differences between campus sites for any of the variables assessed. Surprisingly, assessed higher order skill proficiency (CLA+) offered no prediction of performance on NBME as there were no significant correlations between the SSE and CLA+, or between CLA+ and NBME scores. These data indicate that although first year medical students may not perceive themselves as highly engaged, they are adept in using higher order skills and excel in meeting course objectives regardless of learning environment.This abstract is from the Experimental Biology 2018 Meeting. There is no full text article associated with this abstract published in The FASEB Journal.

Threats to the validity of the Collegiate Learning Assessment (CLA+) as a measure of critical thinking skills and implications for Learning Gain

The University of Reading Learning Gain project is a three-year longitudinal project to test and evaluate a range of available methodologies and to draw conclusions on what might be the right combination of instruments for the measurement of Learning Gain in higher education. This paper analyses the validity of a measure of critical thinking skills, the Collegiate Learning Assessment (CLA+) and the implications of using this standardised test as a proxy for Learning Gain. The paper reviews five inferences regarding the interpretations and use of test scores: construct representation, scoring, generalisation, extrapolation and decision-making. Each section reviews some of the available evidence in support of the claims the CLA+ makes and the threats to their validity. The possible impact of these issues on Learning Gain in the UK is considered.

USING THE CLA+ TO TRACK DEVELOPMENT OF COGNITIVE SKILLS FROM FIRST TO THIRD YEAR: ACCESSIBILITY AND FACTORS FOR SUCCESS

This paper provides results from the first three years of a 4-year longitudinal assessment project on the development of cognitive skills using the Collegiate Learning Assessment (CLA+). The CLA+ is an online test, designed to measure cognitive skills (CS) (critical thinking, problem solving and written communication, with sub-scores reporting scientific and quantitative reasoning, critical reading and evaluation, and critiquing an argument). Cognitive skills are fundamental elements of engineering programs and central to the practice of engineering. The Canadian Engineering Accreditation Board’s (CEAB) requires programs to assess their student’s graduate attributes and have a system in place to use the assessment for curriculum improvement. CLA+ assessment constructs relate to the CEAB attributes of problem analysis, investigation and communication. The testing was initiated as part of a strategy to track development of cognitive skills and to inform course improvement efforts. The testing was embedded in a range of engineering courses. Student achievement on written communication outcomes appeared to fall over time, and critiquing an argument was identified as an area of weakness, specifically for the first year students. Strategies were subsequently implemented as part of a curriculum improvement initiative. With the goal of maximizing success for all students, we investigated differences in achievement based on sex, language, and parental education, as well as the amount of effort put into the test. There were significant differences in achievement, prompting concerns about motivation and relevance to the engineering discipline. Students without English as a first language demonstrated negative gains over time. Parental education level was the strongest predictor of performance, but the most significant factor impacting on students’ scores was self-reported effort.

Editors’ Notes

Editors’ Notes

Prior education of Open University students contributes to their capability in critical thinking

The focus of this study is on Open University students’ entry-level critical thinking skills. The research questions were: how are students’ age, and level and discipline of previous education related to critical thinking skills; is the level and discipline of previous education connected to the accuracy of students’ self-evaluation of their critical thinking skills? Totally 138 students participated altogether. Critical thinking skills were measured by the constructed-response task of the Collegiate Learning Assessment and retrospective self-evaluation scale. The results showed that students’ critical thinking skills, as measured by both the constructed-response task and the self-evaluation scale, differed statistically between groups of different educational levels. The students with higher education degrees had the most developed critical thinking skills. Furthermore, the students who had previous higher education studies in hard sciences succeeded better in critical thinking than those of soft sciences. Between age-groups there were no differences in critical thinking. Students with prior higher education studies evaluated their skills more accurately. A variety of teaching and learning methods and ample feedback are needed in order to learn critical thinking. In addition, to learn metacognitive skills, students should rehearse self-evaluation skills.

Testing the Factorial Equivalence of the Collegiate Learning Assessment Performation Task Diagnostic Instrument Across Lower Class and Upper Class Predominantly Black College Students

Objective: This study was aimed at determining the external validity of the psychometric properties of a two-factor Collegiate Learning Assessment Performance Task Diagnostic Instrument (CLAPTDI) for use in assessing learning skills among predominantly black college students. The construct validity of the two factors CLAPTDI had been established in a previous study exploratory and confirmatory factor analyses. Establishing the external validity involved conducting a multi-group test of the measurement instrument’s factorial scores equivalence across panels of lower class and upper class students from a predominantly black college. Method: The study relied on a strict test of equivalence categorization by focusing on tests for invariance across the two groups with respect to factor loadings, intercepts, and error factor loadings by estimating the difference in chi-square goodness-of-fit statistic and comparative fit index (CFI). Sets of measurement and structural parameters were put to the test in a logically ordered and increasingly restrictive manner. Results: The analyses found that the CLAPTDI scale’s factorial measurement structure was invariant across lower class and upper class PBC students. Conclusion: The collegiate learning assessment performance task diagnostic instrument with two latent factors and five observed variables is a valid measurement scale for assessing the level of analytic reasoning and problem solving learning among predominantly black college students.

Assessing learning gains of university students in Hong Kong adopting the Collegiate Learning Assessment Plus (CLA+)

AbstractThe present study attempted to examine students’ learning gains in critical thinking, problem solving, and effective communication after 1-year university study under a new 4-year undergraduate curriculum based on students’ performance on the Collegiate Learning Assessment Plus (CLA+). One hundred and fifty freshmen and 150 sophomores at The Hong Kong Polytechnic University (PolyU) completed the CLA+ in 2013–2014 academic year. Results showed that sophomores had better performance in CLA+ than freshmen on problem solving, effective communication, and critical thinking. Compared with freshmen, fewer sophomores were classified as ‘Below Basic’ at mastering the related skills. The findings support the effectiveness of the new 4-year undergraduate curriculum, particularly the General University Requirements, in promoting students’ development in desired graduate attributes. However, students’ language barrier might affect their CLA+ scores which led to the small effect size in the differences between freshmen and sophomores.

The Professoriate in an Age of Assessment and Accountability: Understanding Faculty Response to Student Learning Outcomes Assessment and the Collegiate Learning Assessment

The Professoriate in an Age of Assessment and Accountability: Understanding Faculty Response to Student Learning Outcomes Assessment and the Collegiate Learning Assessment