Analyzing Student Cognition in Problem Solving via Online Surveys

Abstract

Metacognitive studies provide invaluable insights into common misconceptions and may shed light on other barriers to learning, such as a fear of math. Among these studies, the think-aloud method is a powerful tool to provide context and explain underlying factors in student learning. However, they often require transcription services, are difficult to conduct in an online environment, and depend on students committing the necessary time to volunteer. To address these challenges, we created an anonymous online survey to mimic a think-aloud study for majors and non-majors introductory biology courses that could give insight to student learning. The survey asked students to predict the probability a child would inherit a fictional disorder in order to analyze students’ problem-solving skills to a question that cannot easily be found on the internet. Two follow-up questions asked the students to describe how they arrived at the response and their emotional state during solving the problem. Based on a previous pilot study, a minimum character limit was required (450 and 100 characters, respectively) to allow concept and application analyses as well as affective perception during the problem-solving process. We adapted an existing framework for coding problem solving based on concept (e.g. understanding concepts related to the problem, such as homo/heterozygous), connection (understanding connections between concepts), and skills (methods to solve the problem). Emotional state and emotional cause were also coded. Three biology faculty participated in three calibration sessions as additional codes were discussed. Final coding was completed by the authors independently, and discrepancies with no majority were discussed and decided by consensus. The dataset was reduced using stepwise AIC to determine the most relevant variables, which were then factored into a logistic regression and post hoc Dunn tests. A total of 181 students (76 non-majors, 105 majors) responded. Students were 40% male and 55% female. Mode age category was 26-30 (21-25 to 51-55). Two answers were discarded as unresponsive, and 91 of responses were marked correct. AIC analysis identified majors vs. non-majors and the codes Concept, Connection, and Emotional State as the best predictors of entering a correct answer. In the overall logistic regression, majors vs. non-majors, Concept, and Connection were significant predictors of getting an answer correct. Dunn tests revealed several factors increasing the likelihood a student answered correctly: identifying relevant concepts vs. unrelated or no concepts (z = -5.56 and -2.27; p = .00 and .035, respectively); drawing explicit or implicit connections instead of none (z = 7.00, 3.93 and 0.85; p = .00, .0001 and .60, respectively); and majors were significantly more likely to answer correctly than nonmajors. Although a notable portion of students expressed confusion or irritation regarding minimum response requirements, nearly all provided insight into their cognitive processes. Overall, carefully designed surveys may offer a promising, if comparatively limited, alternative to think-alouds for metacognition studies when student time and faculty resources are limited.