Cognitive Tutor Algebra Research Articles

Flow with an intelligent tutor: A latent variable modeling approach to tracking flow during artificial tutoring.

Increasing use of intelligent tutoring systems in education calls for analytic methods that can unravel students' learning behaviors. In this study, we explore a latent variable modeling approach for tracking learning flow during computer-interactive artificial tutoring. The study considers three models that give discrete profiles of a latent process: the (i) latent class model, (ii) latent transition model, and (iii) hidden Markov model. We illustrate application of each model using example log data from Cognitive Tutor Algebra I and suggest analytic procedures of drawing learning flow. Through experimental application, we show that the models can reveal substantive information about students' learning behaviors and have potential utility for describing the learning flow. The models differed in the assumptions and data constraints but yielded consistent findings on the flow states and interaction modalities. Based on our experiential analyses, we discuss strengths and limitations of the models and illuminate areas of future development.

Student Log-Data from a Randomized Evaluation of Educational Technology: A Causal Case Study

Randomized evaluations of educational technology produce log data as a bi-product: highly granular data on student and teacher usage. These datasets could shed light on causal mechanisms, effect heterogeneity, or optimal use. However, there are methodological challenges: implementation is not randomized and is only defined for the treatment group, and log datasets have a complex structure. This article discusses three approaches to help surmount these issues. One approach uses data from the treatment group to estimate the effect of usage on outcomes in an observational study. Another, causal mediation analysis, estimates the role of usage in driving the overall effect. Finally, principal stratification estimates overall effects for groups of students with the same “potential” usage. We analyze hint data from an evaluation of the Cognitive Tutor Algebra I curriculum using these three approaches, with possibly conflicting results: the observational study and mediation analysis suggest that hints reduce posttest scores, while principal stratification finds that treatment effects may be correlated with higher rates of hint requests. We discuss these mixed conclusions and give broader methodological recommendations.

The role of mastery learning in an intelligent tutoring system: Principal stratification on a latent variable

Students in Algebra I classrooms typically learn at different rates and struggle at different points in the curriculum—a common challenge for math teachers. Cognitive Tutor Algebra I (CTA1), an educational computer program, addresses such student heterogeneity via what they term “mastery learning,” where students progress from one section of the curriculum to the next by demonstrating appropriate “mastery” at each stage. However, when students are unable to master a section’s skills even after trying many problems, they are automatically promoted to the next section anyway. Does promotion without mastery impair the program’s effectiveness? At least in certain domains, CTA1 was recently shown to improve student learning on average in a randomized effectiveness study. This paper uses student log data from that study in a continuous principal stratification model to estimate the relationship between students’ potential mastery and the CTA1 treatment effect. In contrast to extant principal stratification applications, a student’s propensity to master worked sections here is never directly observed. Consequently we embed an item-response model, which measures students’ potential mastery, within the larger principal stratification model. We find that the tutor may, in fact, be more effective for students who are more frequently promoted (despite unsuccessfully completing sections of the material). However, since these students are distinctive in their educational strength (as well as in other respects), it remains unclear whether this enhanced effectiveness can be directly attributed to aspects of the mastery learning program.

How readability and topic incidence relate to performance on mathematics story problems in computer-based curricula.

Solving mathematics story problems requires text comprehension skills. However, previous studies have found few connections between traditional measures of text readability and performance on story problems. We hypothesized that recently developed measures of readability and topic incidence measured by text-mining tools may illuminate associations between text difficulty and problem-solving measures. We used data from 3,216 middle and high school students from 10 schools using the Cognitive Tutor Algebra program; these schools were geographically, socioeconomically, racially, and ethnically diverse. We found that several indicators of the readability and topic of story problems were associated with students’ tendency to give correct answers and request hints in Cognitive Tutor. We further examined the individual skill of writing an algebraic expression from a story scenario, and examined students at the lowest performing schools in the sample only, and found additional associations for these subsets. Key readability and topic categories that were related to problem-solving measures included word difficulty, text length, pronoun use, sentence similarity, and topic familiarity. These findings are discussed in the context of models of mathematics story problem solving and previous research on text comprehension. (PsycINFO Database Record (c) 2016 APA, all rights reserved)

Effectiveness of Cognitive Tutor Algebra I at Scale

This article examines the effectiveness of a technology-based algebra curriculum in a wide variety of middle schools and high schools in seven states. Participating schools were matched into similar pairs and randomly assigned to either continue with the current algebra curriculum for 2 years or to adopt Cognitive Tutor Algebra I (CTAI), which uses a personalized, mastery-learning, blended-learning approach. Schools assigned to implement CTAI did so under conditions similar to schools that independently adopt it. Analysis of posttest outcomes on an algebra proficiency exam finds no effects in the first year of implementation, but finds evidence in support of positive effects in the second year. The estimated effect is statistically significant for high schools but not for middle schools; in both cases, the magnitude is sufficient to improve the median student’s performance by approximately eight percentile points.

Using adaptive learning technologies to personalize instruction to student interests: The impact of relevant contexts on performance and learning outcomes.

Adaptive learning technologies are emerging in educational settings as a means to customize instruction to learners’ background, experiences, and prior knowledge. Here, a technology-based personalization intervention within an intelligent tutoring system (ITS) for secondary mathematics was used to adapt instruction to students’ personal interests. We conducted a learning experiment where 145 ninth-grade Algebra I students were randomly assigned to 2 conditions in the Cognitive Tutor Algebra ITS. For 1 instructional unit, half of the students received normal algebra story problems, and half received matched problems personalized to their out-of-school interests in areas such as sports, music, and movies. Results showed that students in the personalization condition solved problems faster and more accurately within the modified unit. The impact of personalization was most pronounced for 1 skill in particular—writing symbolic equations from story scenarios—and for 1 group of students in particular—students who were struggling to learn within the tutoring environment. Once the treatment had been removed, students who had received personalization continued to write symbolic equations for normal story problems with increasingly complex structures more accurately and with greater efficiency. Thus, we provide evidence that interest-based interventions can promote robust learning outcomes—such as transfer and accelerated future learning—in secondary mathematics. These interest-based connections may allow for abstract ideas to become perceptually grounded in students’ experiences such that they become easier to grasp. Adaptive learning technologies that utilize interest may be a powerful way to support learners in gaining fluency with abstract representational systems.

Scripted collaborative learning with the cognitive tutor algebra

With the aim to promote students’ mathematics learning, we extended the Cognitive Tutor Algebra (CTA), a computer-based tutoring system for high school mathematics, to a collaborative setting. Furthermore we developed a collaboration script to support students’ interactions. In an experimental classroom study, we compared three conditions: scripted collaborative learning, unscripted collaborative learning, and individual learning. After a 2-day learning phase, posttests assessed individual and collaborative reproduction of knowledge and skills, and future learning. First, with the collaboration script we aimed to improve students’ interaction. Second, we assumed that due to an improved interaction students would benefit more from the learning opportunities during collaboration and, in consequence, their learning would increase as compared with the other conditions. To investigate the first assumption, we compared the interaction of a scripted dyad and an unscripted dyad. The in-depth process analyses revealed a positive impact of the script on student collaboration and problem solving during scripted interaction and in subsequent unscripted interaction. While this effect was mirrored in the learning gains of the two dyads, we could not establish a general learning effect in the quantitative between-condition comparison of student performance. Particularly for students with low prior knowledge, the removal of the script in the test phase initially entailed a decline in reproduction performance as students had to get used to the unscripted problem-solving situation. A notable finding was, however, that the collaborative conditions yielded the same outcomes as the individual condition in the individual reproduction test even though students had solved fewer problems during the learning phase and had only solved them collaboratively.

How Curriculum and Classroom Achievement Predict Teacher Time on Lecture- and Inquiry-Based Mathematics Activities.

This study drew on data from a large, randomized trial of Cognitive Tutor Algebra (CTA) in high-poverty settings to investigate how mathematics curricula and classroom achievement related to teacher reports of time spent on inquiry-based and lecture-based mathematics activities. We found that teachers using the CTA curriculum reported more time on inquiry-based activities and less time on teacher lecture activities overall compared to non-CTA teachers. However, both CTA and non-CTA teachers of the highest-achieving students spent more time on inquiry-based activities compared to teachers of lower-achieving students. Additionally, CTA teachers in classrooms with the most low-achieving and non-gifted students reported almost as much time on lecturebased activities as their non-CTA counterparts. Qualitative findings suggest that CTA teachers engaged in more traditional lecture-based activities and fewer inquiry-based activities when they thought their lower-achieving students could not tackle the reading open-ended activities in the curriculum without explicit demonstration and traditional practice problems. CTA thus appeared to increase inquiry-based activities in teachers’ classrooms overall. However, lower-achieving students may have had needs unaddressed by the CTA curriculum. These findings thus imply that districts should think carefully about how to implement CTA and—potentially—other inquiry-based curricula in order to support teachers of students with the highest needs and least preparation.

CTRL: A research framework for providing adaptive collaborative learning support

There is evidence suggesting that providing adaptive assistance to collaborative interactions might be a good way of improving the effectiveness of collaborative activities. In this paper, we introduce the Collaborative Tutoring Research Lab (CTRL), a research-oriented framework for adaptive collaborative learning support that enables researchers to combine different types of adaptive support, particularly by using domain-specific models as input to domain-general components in order to create more complex tutoring functionality. Additionally, the framework allows researchers to implement comparison conditions by making it easier to vary single factors of the adaptive intervention. We evaluated CTRL by designing adaptive and fixed support for a peer tutoring setting, and instantiating the framework using those two collaborative scenarios and an individual tutoring scenario. As part of the implementation, we integrated pre-existing components from the Cognitive Tutor Algebra (CTA) with custom-built components. The three conditions were then compared in a controlled classroom study, and the results helped us to contribute to learning sciences research in peer tutoring. CTRL can be generalized to other collaborative scenarios, but the ease of implementation relates to the complexity of the existing components used. CTRL as a framework has yielded a full implementation of an adaptive support system and a controlled evaluation in the classroom.

INTEGRATING COLLABORATION AND INTELLIGENT TUTORING DATA IN THE EVALUATION OF A RECIPROCAL PEER TUTORING ENVIRONMENT

Intelligent tutoring systems have been successful at increasing student mathematics learning, but may be further improved with the addition of collaborative activities. We have extended the Cognitive Tutor Algebra, a successful intelligent tutoring system for individual learning, with a reciprocal peer tutoring activity designed to increase conceptual learning. While using our peer tutoring environment, students take on tutor and tutee roles, and engage in both problem-solving actions and dialogue. In a classroom study, we randomly assigned 62 participants to three conditions (adaptive assistance to peer tutoring, fixed assistance to peer tutoring, and individual learning). All conditions yielded significant learning gains, but there were no differences between conditions in final outcomes. There were significant process differences, however. We assessed student interaction using problem-solving information logged by the intelligent tutoring system and collaborative dialogue captured in a chat window. Our analysis integrated these multiple data sources in order to better understand how collaborative dialogue and problem-solving actions might lead to conceptual learning. This rich data sheds light on how students benefitted from the reciprocal peer tutoring activity: Peer tutors learned when they reflected on tutee problem-solving actions, and tutees learned when the tutor's help was responsive to those actions.