Think-aloud Protocol Analysis Research Articles

Using a tablet-based composition marking recording system to conduct think-aloud for composition rating research.

Language teacher’s composition rating process has become an issue of growing concern in writing research over the past three decades. It is evident that a substantial number of composition rating research have employed concurrent think-aloud protocol analysis (TAPA) as a major method of qualitative data collection and data analysis. For the previous studies that adopted the method of TAPA in exploring the composition marking process, they were mostly confined to the study on L1/L2 English language education and not much has been done specifically on L1 Chinese language education. None of the previous studies employing TAPA were conducted with a computer marking platform. In the past, researchers generally videotaped and recorded teacher participants’ think-aloud protocol during the composition marking process. However, the presence of the camera put a certain amount of psychological pressure on the participant and affected his/her performance in the rating task. To address these research gaps, a camera-free, tablet-based composition marking system is developed allowing writing researchers to examine the composition marking behavior of Chinese writing teachers with the method of TAPA conducted in a tablet-assisted marking environment. The system is able to eliminate the psychological pressure on the teacher participants as in video recording and to record their process of marking compositions systematically for later analysis.

Understanding the Connection Between Epistemic Beliefs and Internet Searching

Within the context of exploring an ill-structured task using the Google search engine, this study examined (a) the connections between general epistemic beliefs and the complexity of learners' knowledge exploration processes (i.e., learning complexity) and (b) the role of activating learners' task-oriented epistemic beliefs (i.e., epistemic activation) in affecting their learning processes. Survey was used to collect participants' general epistemic beliefs, and direct analysis of participants' think-aloud protocols documenting their knowledge exploration processes was conducted to measure their learning complexity. Results revealed positive epistemology-learning relationships in the way that learners with complex epistemic beliefs are more likely to invest efforts in integrating knowledge, building flexible knowledge representations, evaluating information credibility, engaging in inquiry, and being learning-oriented during their ill-structured Internet search activities. Epistemic activation seemed to provide an opportunity for learners to contemplate varying advanced strategies to evaluate the quality of web information, trigger their efforts to search for alternative views during learning, and enhance their awareness of task complexity and subjectivity. In addition, learners with complex epistemic beliefs seemed to benefit more from epistemic activation in terms of using advanced strategies for web information evaluation. This research contributes to (a) theoretical understandings of epistemology in connection to learning complexity when learning is open-ended and (b) pedagogical practices of using the epistemic activation strategy to promote deep learning in Internet-based learning environments.

Using think-aloud protocol analysis to gain in-depth insights into upper primary school children's self-regulated learning

Recently, research considering SRL among primary school students has increased since it has been acknowledged that even young students can regulate their learning. However, in-depth and longitudinal information regarding upper primary school children's SRL is still lacking, especially of students at risk of school failure. Through a longitudinal multiple case study eight at-risk students were followed during two successive school years by using think-aloud protocols (six measurement occasions). The results confirm that at-risk upper primary students do engage in SRL, however on a rather superficial level. Moreover, their strategy use remained rather stable over time. Furthermore, differential results in the Sudoku and text studying task confirm that SRL can vary across tasks and domains. Regarding the assessment of SRL, the current study illustrates that think-aloud protocols are valuable tools to provide in-depth information, while also indicating the need for a multi-method design, especially in order to grasp motivational aspects of SRL.

Promoting university students’ metacognitive regulation through peer learning: the potential of reciprocal peer tutoring

Although successful learning in university education can be advanced by students’ competence to self-regulate their learning, students often possess insufficient metacognitive regulation skills to regulate their learning adequately. The present study investigates changes in university students’ adoption of metacognitive regulation after participating in reciprocal peer tutoring (RPT). A quasi-experimental pretest–posttest design was adopted, involving an experimental (n = 51) and two control groups; CG1 (n = 24) and CG2 (n = 22). Experimental students participated in a RPT intervention during a complete semester. Metacognitive regulation was assessed by means of think-aloud protocol analysis. Results indicate that RPT is promising to promote metacognitive regulation. Experimental students increasingly adopt monitoring, evaluation, and orientation and significantly evolve towards deep-level regulation from pretest to posttest. Except for an increased use of low-level comprehension monitoring, none of the evolutions in experimental students’ regulation could be discerned for students in both control groups.

Concurrent think-aloud protocols to assess elementary design students

Initiatives to integrate engineering design in the elementary science classroom have become increasingly evident in both national reform documents and classroom practice. Missing from these efforts is a purposeful attempt to capture students’ designerly thinking and dialogues as they engage in the process. The purpose of this study was to investigate how elementary school students approach and engage in engineering design using concurrent think-aloud protocols. Data from seven concurrent think-aloud protocols among triads of elementary students across seven classrooms were analyzed to identify how students conceptualize design. Researchers employed a transfer problem and think-aloud protocol analysis to assess students’ transfer of learning from classroom science based engineering design-based experiences. Results indicate that elementary student triad design teams were able to define a design problem, identify constraints and criteria, and generate multiple design ideas to solve the problem. Protocol timelines were generated using NVivo software to capture sequence of the triads’ coded cognitive strategies crucial in understanding which triads used a systematic approach to solving the problem from triads that randomly brainstormed ideas. If design is to become a pedagogical approach to teaching science or other STEM-related subjects, attention must be given to how students learn design and function within design. Concurrent think-aloud protocol provides a promising means of assessment of such efforts.

Try, Try, Try Again: The Process of Designing New History Assessments

This article considers the design process for new formative history assessments. Over the course of 3 years, my colleagues from the Stanford History Education Group and I designed, piloted, and revised dozens of History Assessments of Thinking (HATs). As we created HATs, we sought to gather information about their cognitive validity, the relationship between the constructs targeted by the assessments, and the cognitive processes students used to answer them. Three case studies trace the development of different HATs through analyses of draft assessments, student responses, and think-aloud protocols. Design principles specific to formative history assessments emerged from these analyses: (1) assessments must be historically accurate, (2) assessments must target specific historical constructs, (3) assessment structure must align with targeted constructs, (4) assessments must yield useful information for teachers, and (5) pilot data are indispensable for refining HATs. These finding suggest the need for increased attention on the construction and validation of new assessment materials for the history classroom.

Reading Strategies and Reading Comprehension Retention with and without a Pop-up Dictionary by Beginning Learners of Chinese

This study is aimed at identifying reading strategies of beginning learners of Chinese as a foreign language (CFL) with and without a pop-up dictionary and at determining if learners retain the reading comprehension gained from using the dictionary. Beginning CFL learners at a Midwestern university answered questions about their reading strategies after reading passages in Chinese with and without a pop-up dictionary, did think-aloud protocols, and wrote recalls. Qualitative analyses of answers and thinkaloud protocols identified generally-used strategies both with and without the pop-up dictionary: engaging in bottom-up strategies, monitoring comprehension, using the first language, and focusing on lexical cues. Qualitative analyses also revealed different reading strategies in obtaining character and word meanings and in constructing sentence meanings. Statistical results indicated that much of the improvement gained from using the pop-up dictionary during the first reading was not retained over a short period of time. It may be concluded that the use of the pop-up dictionary may not alter CFL learners’ general reading strategies, yet the dictionary does provide additional resources to facilitate meaning construction. Hence, it is relevant to guide students to effectively use the pop-up dictionary.

Usability testing of Avoiding Diabetes Thru Action Plan Targeting (ADAPT) decision support for integrating care-based counseling of pre-diabetes in an electronic health record

PurposeUsability testing can be used to evaluate human–computer interaction (HCI) and communication in shared decision making (SDM) for patient–provider behavioral change and behavioral contracting. Traditional evaluations of usability using scripted or mock patient scenarios with think-aloud protocol analysis provide a way to identify HCI issues. In this paper we describe the application of these methods in the evaluation of the Avoiding Diabetes Thru Action Plan Targeting (ADAPT) tool, and test the usability of the tool to support the ADAPT framework for integrated care counseling of pre-diabetes. The think-aloud protocol analysis typically does not provide an assessment of how patient–provider interactions are effected in “live” clinical workflow or whether a tool is successful. Therefore, “Near-live” clinical simulations involving applied simulation methods were used to compliment the think-aloud results. This complementary usability technique was used to test the end-user HCI and tool performance by more closely mimicking the clinical workflow and capturing interaction sequences along with assessing the functionality of computer module prototypes on clinician workflow. We expected this method to further complement and provide different usability findings as compared to think-aloud analysis. Together, this mixed method evaluation provided comprehensive and realistic feedback for iterative refinement of the ADAPT system prior to implementation. MethodsThe study employed two phases of testing of a new interactive ADAPT tool that embedded an evidence-based shared goal setting component into primary care workflow for dealing with pre-diabetes counseling within a commercial physician office electronic health record (EHR). Phase I applied usability testing that involved “think-aloud” protocol analysis of eight primary care providers interacting with several scripted clinical scenarios. Phase II used “near-live” clinical simulations of five providers interacting with standardized trained patient actors enacting the clinical scenario of counseling for pre-diabetes, each of whom had a pedometer that recorded the number of steps taken over a week. In both phases, all sessions were audio-taped and motion screen-capture software was activated for onscreen recordings. Transcripts were coded using iterative qualitative content analysis methods. ResultsIn Phase I, the impact of the components and layout of ADAPT on user's Navigation, Understandability, and Workflow were associated with the largest volume of negative comments (i.e. approximately 80% of end-user commentary), while Usability and Content of ADAPT were representative of more positive than negative user commentary. The heuristic category of Usability had a positive-to-negative comment ratio of 2.1, reflecting positive perception of the usability of the tool, its functionality, and overall co-productive utilization of ADAPT. However, there were mixed perceptions about content (i.e., how the information was displayed, organized and described in the tool).In Phase II, the duration of patient encounters was approximately 10min with all of the Patient Instructions (prescriptions) and behavioral contracting being activated at the end of each visit. Upon activation, providers accepted the pathway prescribed by the tool 100% of the time and completed all the fields in the tool in the simulation cases. Only 14% of encounter time was spent using the functionality of the ADAPT tool in terms of keystrokes and entering relevant data. The rest of the time was spent on communication and dialog to populate the patient instructions. In all cases, the interaction sequence of reviewing and discussing exercise and diet of the patient was linked to the functionality of the ADAPT tool in terms of monitoring, response-efficacy, self-efficacy, and negotiation in the patient-provider dialog. There was a change from one-way dialog to two-way dialog and negotiation that ended in a behavioral contract. This change demonstrated the tool's sequence, which supported recording current exercise and diet followed by a diet and exercise goal setting procedure to reduce the risk of diabetes onset. ConclusionsThis study demonstrated that “think-aloud” protocol analysis with “near-live” clinical simulations provided a successful usability evaluation of a new primary care pre-diabetes shared goal setting tool. Each phase of the study provided complementary observations on problems with the new onscreen tool and was used to show the influence of the ADAPT framework on the usability, workflow integration, and communication between the patient and provider. The think-aloud tests with the provider showed the tool can be used according to the ADAPT framework (exercise-to-diet behavior change and tool utilization), while the clinical simulations revealed the ADAPT framework to realistically support patient-provider communication to obtain behavioral change contract. SDM interactions and mechanisms affecting protocol-based care can be more completely captured by combining “near-live” clinical simulations with traditional “think-aloud analysis” which augments clinician utilization. More analysis is required to verify if the rich communication actions found in Phase II compliment clinical workflows.

Using on-line and off-line measures to explore fifth and sixth graders' text-learning strategies and schematizing skills

This study explores the (a) text-learning strategies and (b) schematizing skills of pre-adolescents with varying achievement levels by means of the analysis of think-aloud protocols, traces, and pen movements. Twenty fifth- and sixth-grade students from two elementary schools participated. Results show the use and combination of various text-learning strategies during learning from text, mostly applied at a surface level. Notwithstanding the large variation in students' individual strategy repertoires, four main text-learning approaches could be distinguished. No achievement level differences in text-learning strategy use were found. As to students' schematizing skills, analyses illustrate the great difficulty students experience with spatially and hierarchically representing text information. Students paid limited attention in the construction process to assisting metacognitive processes. Surprisingly, low achievers spent significantly more time on these metacognitive processes. This multi-method assessment of text-learning strategies in general and schematizing skills in particular provide fruitful avenues for future intervention research.

Measuring critical components of digital literacy and their relationships with learning

The growing prominence of the Internet, and other digital environments, as educational tools requires research regarding learners' digital literacy. We argue that two critical aspects of digital literacy are the ability to effectively plan and monitor the efficacy of strategies used to search and manage the wealth of information available online, and the knowledge to appropriately vet and integrate those information sources. Therefore, digital literacy requires effective self-regulated learning (SRL) skills, and availing epistemic cognition (EC). Although numerous researchers and scholars have examined the role of SRL in online learning (e.g., Efklides, 2011; Lee & Tsai, 2010; Williams & Hellman, 2004; Winters, Greene, & Costich, 2008), there is a need for additional empirical research on how SRL and EC interact, and relate to learning in digital environments. In this study, we used a powerful, but little-used data collection methodology, think-aloud protocol (TAP) analysis, to investigate the relations among SRL, EC, and learning gains with 20 college students who studied vitamins on the Internet. We also contributed to the literature by exploring alternative techniques for preparing, analyzing, and representing these data, accounting for the strengths and challenges of TAPs. We found that, on average, participants did increase their understanding as a result of learning with the Internet, and that a data-driven approach to understanding relations among SRL, EC, and learning yielded the most powerful representation of these phenomena. Our study has implications for future research on digital literacy using TAPs, as well as the relative contribution of SRL and EC, as aspects of digital literacy, to online learning.

To code or not to code: Dilemmas in analysing think-aloud protocols in learning strategies research

Abstract Analysis of qualitative data depends on the overall design and the nature of the research questions. Specifically, the approach to coding and analysis should be coherent with the way data is elicited and the epistemological perspective that situates the overall design. This article illustrates these points by using an example from the analysis of think-aloud protocols which revealed patterns of strategy choice and use in completing a listening task among a group of primary school pupils in Singapore. Throughout the discussion, I focus on the major dilemmas that arose during the process of coding and analysis. Two dimensions of these dilemmas were identified, those that pertain to the nature of qualitative data analysis and those that are specifically related to language learning strategies.

Experience and Opportunity Identification: The Role of Intuition and Cognitive Versatility

The issue of “why, when and how” some people, but not others, identify opportunities is central to our understanding of entrepreneurship. Research suggests that entrepreneurs become more adept at identifying opportunities as they gain experience, but the cognitive processes underlying this relationship remain unclear. Our study explores this gap by asking: To what extent can the relationship between entrepreneurial experience and opportunity identification be explained by cognitive strategy? Seventy-four technology-entrepreneurs completed a think-aloud protocol analysis exercise (Ericsson & Simon, 1993) where they generated potential business ideas for three innovative technologies, together with an online survey. Extending previous studies, we show that cognitive versatility, which is an information processing strategy characterized by high levels of both intuition and analysis, together with an ability to switch between them as needed (see Hodgkinson & Clarke, 2007), mediates the relationship between experience and opportunity identification. Not only do experienced entrepreneurs use their intuition to help them identify potentially lucrative opportunities, they also employ a process of analysis to complement their intuition and to ensure that intuition is not leading them astray.

The Role of L1 in L2 Idiom Comprehension

The present study investigated the role of first language (L1) transfer in Iranian EFL learners' second language (L2) idiom comprehension. It was also sought to understand whether there is any significant difference between learners of different proficiency levels and their use of L1 in decoding L2 idioms. To do this, the participants of different levels of language proficiency were asked to participate in this study. The L2 idioms were categorized based on their similarity to L1 into three groups of identical, similar and different. A think-aloud protocol analysis was performed and participants were asked to verbalize their thoughts as they read the target idioms in order to detect the strategies they used. The results showed that the most favoured strategy used by learners of different levels was translation. Translation to L1 (Persian) was also the most-frequent strategy in decoding similar, identical and different types of idioms. It was also revealed that generally the participants of different levels were significantly different from each other in using strategies.

Pitfall or scaffolding? Starting-point pull in configuration tasks.

In configuration problems, such as the construction of a weekly study schedule, decision makers must assemble a combination of parts under a set of constraints. Interactions may be present between the parts, and more than a single objective function may exist, such as minimizing the number of days on campus and maximizing the interest level of the courses. Little is known about the decision-making processes involved in configuration tasks. Building on problem-solving and resource-allocation literature, the current study examines the effect of the starting points on performance in these tasks. We predicted that task constraints serve as starting points for the construction process that pull the entire configuration toward them. In 2 experiments, we asked participants to construct a weekly course schedule. We manipulated the course-offering sets so that starting the configuration with one of the task constraints (mandatory courses or a requirement to be on campus on specific days) and myopically building the configuration around it would either help or hinder the effort to reach a good schedule. By several convergent measures, participants succeeded more with the "Asset" course-offering sets than with the "Obstacle" sets. Think-aloud protocol analysis also supported this starting-point pull effect. The results of Experiment 3 show that for a starting point to pull the configuration, it has to be a constraint on the task. All 3 experiments show how building the configuration myopically around the starting points unduly affects the end result.

Fostering University Students’ Metacognitive Regulation Through Peer Tutoring

University students often possess insufficient metacognitive skills to self-regulate their learning adequately. This study investigates the impact of reciprocal peer tutoring (RPT) on students’ adoption of metacognitive skills and on their approach to metacognitive regulation. A quasi experimental pretest-posttest design was adopted, involving an experimental (n=51) and two comparison groups; CG1 (n=24) and CG2 (n=22). Experimental students participated in the RPT-intervention. Students’ regulation was assessed using think-aloud protocol analysis. Results indicate that RPT-students increasingly use metacognitive orientation, monitoring, and evaluation at posttest; evolutions which are not shown in the comparison groups. Significant effects on students’ metacognitive planning could not be distinguished. Despite a significant evolution towards more deep-level comprehension monitoring for RPT-students, all participants’ metacognitive regulation remains dominantly low-level.

Iranian EFL Students’ Perceptions and Preferences for Teachers’ Written Feedback: Do Students’ ideas Reflect Teachers’ Practice?

The present study-both qualitative and quantitative--explored fifty EFL learners’ preferences for receiving error feedback on different grammatical units as well as their beliefs about teacher feedback strategies. The study also examined the effect of the students’ level of writing ability on their views about the importance of teacher feedback on different error types. Data was gathered through the administration of a questionnaire, verbal protocol analysis, and students’ writing scores. The results of repeated measures, multivariate analysis of variance, and frequency counts revealed that the majority of the students expect and value teachers’ written feedback on the following surface-level errors: transitional words, sentence structure, verb tenses, adverbs, punctuation, prepositions, and spelling, respectively. The results of think-aloud protocol analysis indicated that students’ beliefs about the importance of feedback on different grammatical units are formed as a result of the teacher’s practice and his emphasis on certain types of feedback and feedback strategies. Finally, the findings of thestudy showed that the L2 learners’ level of writing ability influences their views about the importance of feedback on errors pertinent to particular grammatical units.

Integrating usability testing and think-aloud protocol analysis with “near-live” clinical simulations in evaluating clinical decision support

PurposeUsability evaluations can improve the usability and workflow integration of clinical decision support (CDS). Traditional usability testing using scripted scenarios with think-aloud protocol analysis provide a useful but incomplete assessment of how new CDS tools interact with users and clinical workflow. “Near-live” clinical simulations are a newer usability evaluation tool that more closely mimics clinical workflow and that allows for a complementary evaluation of CDS usability as well as impact on workflow. MethodsThis study employed two phases of testing a new CDS tool that embedded clinical prediction rules (an evidence-based medicine tool) into primary care workflow within a commercial electronic health record. Phase I applied usability testing involving “think-aloud” protocol analysis of 8 primary care providers encountering several scripted clinical scenarios. Phase II used “near-live” clinical simulations of 8 providers interacting with video clips of standardized trained patient actors enacting the clinical scenario. In both phases, all sessions were audiotaped and had screen-capture software activated for onscreen recordings. Transcripts were coded using qualitative analysis methods. ResultsIn Phase I, the impact of the CDS on navigation and workflow were associated with the largest volume of negative comments (accounting for over 90% of user raised issues) while the overall usability and the content of the CDS were associated with the most positive comments. However, usability had a positive-to-negative comment ratio of only 0.93 reflecting mixed perceptions about the usability of the CDS. In Phase II, the duration of encounters with simulated patients was approximately 12min with 71% of the clinical prediction rules being activated after half of the visit had already elapsed. Upon activation, providers accepted the CDS tool pathway 82% of times offered and completed all of its elements in 53% of all simulation cases. Only 12.2% of encounter time was spent using the CDS tool. Two predominant clinical workflows, accounting for 75% of all cases simulations, were identified that characterized the sequence of provider interactions with the CDS. These workflows demonstrated a significant variation in temporal sequence of potential activation of the CDS. ConclusionsThis study successfully combined “think-aloud” protocol analysis with “near-live” clinical simulations in a usability evaluation of a new primary care CDS tool. Each phase of the study provided complementary observations on problems with the new onscreen tool and was used to refine both its usability and workflow integration. Synergistic use of “think-aloud” protocol analysis and “near-live” clinical simulations provide a robust assessment of how CDS tools would interact in live clinical environments and allows for enhanced early redesign to augment clinician utilization. The findings suggest the importance of using complementary testing methods before releasing CDS for live use.

Exploring the potential impact of reciprocal peer tutoring on higher education students’ metacognitive knowledge and regulation

It is widely recognized that metacognition is an important mediator for successful and high-level learning, especially in higher education. Nevertheless, a majority of higher education students possess insufficient metacognitive knowledge and regulation skills to self-regulate their learning adequately. This study explores the potential of reciprocal peer tutoring to promote both university students’ metacognitive knowledge and their metacognitive regulation skills. The study was conducted in a naturalistic higher education setting, involving 67 students tutoring each other during a complete semester. A multi-method pretest–posttest design was used combining a self-report questionnaire, assessing students’ metacognitive knowledge and their perceived metacognitive skilfulness, with the analysis of think-aloud protocols, revealing students’ actual use of metacognitive strategies. Results indicate no significant pretest to posttest differences in students’ metacognitive knowledge, nor in their perception of metacognitive skill use. In contrast, significant changes are observed in students’ actual metacognitive regulation. At posttest, students demonstrate significantly more frequent and more varied use of metacognitive regulation, especially during the orientation, monitoring, and evaluation phases. Furthermore, our findings point to an increase in more profound and higher-quality strategy use at posttest.

Developing and designing online engineering ethics instruction for international graduate students

The present project embarked on an educational intervention, consisting of a series of online ethics learning modules, to aid international graduate students in overcoming the acculturation barriers to understanding and inculcating normative ethical obligations associated with engineering practice and research in the United States. A fundamental initial step in the process of helping international, as well as domestic, engineering graduate students embrace ethical obligations is to provide clear instruction on fundamental engineering ethical principles and values relevant in the United States. Most institutes of higher education do not have a cohesive approach to basic graduate engineering ethics instruction, much less materials that have been calibrated for international students (National Science Foundation, http://www.nspe.org/Ethics/index.html , 2009). Herein the authors describe our instructional intervention, as well as to document the development, design, and assessment of the learning modules intended to provide students with a framework for learning ethical precepts and applying them in the engineering field. Think-Aloud Protocol and Cognitive Task Analysis results were used to improve the content modules and learning experience. Initial pilot findings indicate that the content modules increased student knowledge acquisition compared to pre-test performance, indicating a step-forward in the formulation of a useful learning tool for graduate engineering ethics instruction.

Decisional strategy determines whether frame influences treatment preferences for medical decisions.

Decision makers are influenced by the frame of information such that preferences vary depending on whether survival or mortality data are presented. Research is inconsistent as to whether and how age impacts framing effects. This paper presents two studies that used qualitative analyses of think-aloud protocols to understand how the type of information used in the decision making process varies by frame and age. In Study 1, 40 older adults, age 65 to 89, and 40 younger adults, age 18 to 24, responded to a hypothetical lung cancer scenario in a within-subject design. Participants received both a survival and mortality frame. Qualitative analyses revealed that two main decisional strategies were used by all participants: one strategy reflected a data-driven decisional process, whereas the other reflected an experience-driven process. Age predicted decisional strategy, with older adults less likely to use a data-driven strategy. Frame interacted with strategy to predict treatment choice; only those using a data-driven strategy demonstrated framing effects. In Study 2, 61 older adults, age 65 to 98, and 63 younger adults, age 18 to 30, responded to the same scenarios as in Study 1 in a between-subject design. The results of Study 1 were replicated, with age significantly predicting decisional strategy and frame interacting with strategy to predict treatment choice. Findings suggest that framing effects may be more related to decisional strategy than to age.