Retrospective Think-aloud Protocols Research Articles

Aligning Teaching Philosophy Statements with Practice: An Evidence-Based Approach Using Retrospective Think-Aloud Protocols

Teaching philosophy statements are often declarations of beliefs interspersed with descriptions and metaphors. The disjuncture between the stated philosophy and actual teaching has been raised by numerous academics. This case study addresses the neglected area of grounding teaching philosophies on actual teacher behaviour rather than on espoused beliefs. This study includes a replicable framework for teachers to create evidence-based teaching philosophy statements through a systematic investigation of their actual teaching practices. A retrospective think-aloud protocol was used to recount a lesson. Using a transcript of the recount, the teacher’s actions were identified, extracted, and justified following pre-determined protocols. References to theoretical and empirical studies supporting or contradicting the justifications were checked in the research literature. To counteract potential self-bias, colleagues’ views on the reasons selected were surveyed. The discrepancy between the teacher’s justification of actions and the peers’ perspectives revealed hitherto hidden idiosyncrasies and underlying values.

A Model for Usability Evaluation of Learning Management Systems

Learning management systems (LMSs) are increasingly popular in education. The usability of these systems must be examined to ensure they meet the needs of both students and faculty. To address the limitations of existing models for evaluating the usability of LMS, this research proposes a new model that combines objective measures collected from eye-tracking with subjective measures collected from retrospective think-aloud protocols and questionnaires. This model evaluates the effectiveness, efficiency, satisfaction, and learnability of LMSs. The proposed model was applied to study the usability of the LMS used at King Saud University (Blackboard) and was compared with other existing usability evaluation models. The findings indicate that the proposed model fulfilled its intended purpose and provided more comprehensive results than existing models.

Deconstructing Immersion in the Experience Economy Framework for Immersive Dining Experiences through Mixed Reality.

In this study, we test the immersive character in an interactive content narrative developed for Microsoft HoloLens 2 mixed reality glasses in the dining context. We use retrospective think aloud protocol (RTAP) and galvanic skin response (GSR) to explore different types of immersion that can be created through interactive content narratives. Leaning on the core dimensions of the experience economy, we expand the current understanding on the role of immersion through integration of four immersive experience facilitators. The study revealed that these immersive experience facilitators occur simultaneously and can be enhanced through interactive content narrative design. Perceived novelty and curiosity were identified as key determinants to keep consumers engaged in the immersive experience and engage with the content. The study verifies the use of galvanic skin response in combination with retrospective think aloud protocol as a suitable approach to measure emotional engagement potential in interpreting consumers' recollection of immersive experiences.

Communication strategy use of EFL learners in videoconferencing, virtual world and face-to-face environments

AbstractThis study investigates the communication strategy (CS) use of English as a foreign language (EFL) learners in videoconferencing (VC), virtual world (VW), and face-to-face environments. The study was conducted with 30 senior Turkish undergraduate EFL students. The data were collected via video and audio recordings of three opinion-exchange tasks, a background and post-task questionnaire, and an interview that includes a retrospective think-aloud protocol. The participants worked in groups of five to complete the opinion-exchange tasks in each of the three environments. The findings indicated that the participants made use of a wide range of CSs, and although some of the CSs differed, mostly the same types were employed in all the environments. However, the results revealed that the frequency of CSs showed variance among environments, with the highest number in the VC environment and the lowest in the VW. It was possible to establish a connection between the differences in the frequency and the types of strategy use with the distinctive features of the environments, the proficiency level of the participants, and the type of the tasks that was utilized. Additively, 10 new strategy types were discovered.

An Exploratory Study of Culturally Familiar or Unfamiliar Texts Contributing to Reading Comprehension in EFL Context

The current study examines the impact of culturally relevant background knowledge on second-year Saudi EFL learners' reading comprehension. The purpose of this study is to contribute to a better understanding of culturally diverse texts and to enhance the teaching-learning process by utilizing the background knowledge of local learners. This qualitative study employed a think-aloud retrospective interview methodology to help readers express their in-depth thoughts, interpret, and reflect on familiar and unfamiliar cultural passages. Thematic analysis was used to classify think-aloud retrospective interview protocol on two passages, which were audio-recorded, transcribed, and coded for specified categories. The qualitative data were analyzed to examine if there was a link between EFL respondents' cultural background knowledge and the passages. A total of five English-major undergraduate respondents from Majmaah University took part in retrospective think-aloud interview protocols. The findings of this study revealed that EFL undergraduate respondents' cultural background knowledge had a favorable impact on a culturally relevant topic. The findings of this study revealed that culturally relevant topics differed significantly from culturally irrelevant topics.

Personal Viewpoint Ethnography Utilizing GoPro™ Cameras to Capture Student Information Seeking in School Libraries

This exploratory study investigated supports and services academic libraries currently provide that assist in the fulfillment of information needs. Utilizing wayfinding, think-aloud protocol (TAP), retrospective think-aloud protocol (RTAP), and a GoProTM camera, the actions, thoughts, and feelings of an autistic college freshman and a neurotypical peer were recorded as they navigated their campus library in search of materials. The library website, virtual maps to resource locations, and library workers served equally as enablers and barriers to both students in their information seeking. This proof-of-concept study demonstrates that participant viewpoint ethnography is a viable methodology that can be applied in school library research, with specific adaptations and examples provided.

Exploring attention on a retailer’s homepage: an eye-tracking & qualitative research study

ABSTRACT This article explores why some areas of a retailer’s homepage receive more attention than others. An eye-tracking study followed by retrospective think-aloud protocols and qualitative semi-structured interviews were conducted with a purposive sample of 52 females (aged 20–70). Findings revealed the visual hierarchy of a retail homepage: 1. Rotating Banner, 2. Main Menu Bar, 3. Left drop-down menu, 4. Static images below banner, 5. Search bar. Qualitative data revealed that some features attracted attention due to them being useful, serving a functional purpose (menu bars, search bar), or due to them being interesting (static banner images) but others did so inadvertently because they were distracting and irritating (rotating banner). This shows that attention can be due to both positive and negative factors. This article contributes to knowledge by providing a rich interpretation of eye-tracking data and attention on a retail homepage, showing the interplay between top-down and bottom-up attention on a user’s shopping journey. This is valuable for future academic research as well as practitioners. This study also highlights the importance of using live websites for research to increase ecological validity.

Older Adults’ Concurrent and Retrospective Think-Aloud Verbalizations for Identifying User Experience Problems of VR Games

AbstractWhile virtual reality (VR) games are beneficial for older adults to improve their physical functions and cognitive abilities, VR research often does not include older adults. Our review of the proceedings of major HCI conferences (i.e. ASSETS, CHI, CHI PLAY, CSCW and DIS) between 2016 and 2020 shows that only three out of 352 VR-related papers involved older adults. Consequently, older adults tend to encounter user experience (UX) problems with VR. One common way to identify UX problems is to conduct usability testing with think-aloud (TA) protocols. As VR games tend to be perceptually and physically demanding, older adults might need to allocate more resources to VR content and interaction and thus have fewer resources for thinking aloud. This raises the question of whether TA protocols are still a viable approach to detecting UX problems of VR games for older adult participants. To answer this question, we conducted usability testing with older adults who played two common types of VR games (i.e. the exergame and experience game) using concurrent and retrospective TA protocols (i.e. CTA and RTA), which are widely used in the industry. We analyzed participants’ TA verbalizations and uncovered how different categories of verbalizations indicate UX problems. We further show how older adults perceived the effects of thinking aloud on their game experiences in two TA protocols and offer design implications.

Capturing the Information-seeking of Students Employing a Wearable Camera: Discussions of Application in School Libraries

Utilizing wayfinding, think-aloud protocol (TAP), retrospective think-aloud protocol (RTAP), and a wearable camera, the actions, thoughts and feelings of an autistic college freshman and a neurotypical peer are recorded as they navigate their campus library in search of materials. The library website, virtual maps to resource locations, and library workers served equally as enablers and barriers to both students in their information seeking. This proof of concept study demonstrates that participant viewpoint ethnography is a viable research methodology that could be applied in school libraries for both neurotypical and autistic students.

The impact of translation modality on user experience: an eye-tracking study of the Microsoft Word user interface

This paper presents results of the effect of different translation modalities on users when working with the Microsoft Word user interface. An experimental study was set up with 84 Japanese, German, Spanish, and English native speakers working with Microsoft Word in three modalities: the published translated version, a machine translated (MT) version (with unedited MT strings incorporated into the MS Word interface) and the published English version. An eye-tracker measured the cognitive load and usability according to the ISO/TR 16982 guidelines: i.e., effectiveness, efficiency, and satisfaction followed by retrospective think-aloud protocol. The results show that the users’ effectiveness (number of tasks completed) does not significantly differ due to the translation modality. However, their efficiency (time for task completion) and self-reported satisfaction are significantly higher when working with the released product as opposed to the unedited MT version, especially when participants are less experienced. The eye-tracking results show that users experience a higher cognitive load when working with MT and with the human-translated versions as opposed to the English original. The results suggest that language and translation modality play a significant role in the usability of software products whether users complete the given tasks or not and even if they are unaware that MT was used to translate the interface.

The effect of combined strategy instruction on reading comprehension

This study investigated the effectiveness of combined strategy instruction on reading comprehension, students’ perceptions of combined strategy training in reading instruction, and teachers’ perceptions about combined reading strategy instruction and their experiences during strategy instruction. Four upper-intermediate classes (two as control groups and two as experimental groups) participated in the study. The experimental group received four-week long combined strategy instruction while the control group followed the current reading syllabus without strategy instruction. During the four-week study, Chamot and O'Malley's (1994) strategy instruction model, Cognitive Academic Language Learning Approach (CALLA), was followed for the most part. Prior to and after the four-week study an International English Language Testing System (IELTS) reading test was given to the students to assess their reading comprehension. Retrospective think-aloud protocols were used after the post-reading test in order to gather evidence on the use of strategies during the post-test. Following the treatment, a questionnaire was administered to the experimental group students in order to explore their perceptions of the strategy instruction program. Finally, the instructors of the experimental classes were interviewed about their experiences during the treatment period. The data analysis showed that the experimental group showed significantly greater improvement on the reading test after the four-week study. Furthermore, the retrospective think-aloud protocols demonstrated that experimental group students employed a broad range of strategies during the post-reading test. The analysis of the questionnaire and interviews revealed that combined strategy instruction had a positive impact on both teachers and students.

‘Dream now, travel later’: pre-travel online destination experiences on destination websites

Destination marketing organisations (DMOs) seek to provide positive pre-travel online destination experiences (ODEs) to attract tourists. Thereby, DMOs understand official destination websites (ODWs) as central sources of information influencing tourists’ travel decisions. Although experiential marketing theory postulates that customers are reached through sensory, affective, intellectual, behavioural or social experiences, this theory has rarely been applied to analysing tourist experiences on ODWs. Past research and theories remain similar to models from product brands, yet fail to acknowledge the peculiarities of destination experiences. This research explores how users of ODWs experience unfamiliar tourist destinations in the pre-travel phase. To gain a deeper understanding of the nature of ODEs on ODWs, a qualitative multi-method study was conducted involving eye-tracking, retrospective think-aloud protocols, semi-structured interviews and video observations with 15 German millen­nials selected via purposeful sampling. Data was analysed in a qualitative directed content analysis following an abductive approach. Findings expand on previous theory by adding a spatio-temporal experience dimension. In the pre-travel phase, potential tourists explore the spatio-temporal accessibility of expected experiences and the experience density in the destin­ation. Furthermore, this research provides new insights into the different dimensions of ODEs and proposes an advanced conceptual framework.

Exploring the roles of motivation and cognition in label-usage using a combined eye-tracking and retrospective think aloud approach

This study investigates the factors affecting consumers' motivation to engage with food product labelling in the new product context. Using yogurt as a case food, due to its positive association with health, enjoyment and convenience, this study uses eye-tracking experiments, a retrospective think-aloud protocol and semi-structured interviews, to bring to light the conscious and subconscious mechanisms associated with label usage, in order to explore the cognitive processes underlying usage of labels for new product offerings and situate these within the participant's personal context. Key information usage and decision-making strategies and the factors which give rise to these are identified. Findings suggest that consumer involvement is being shaped by the perceived importance of negative consequences (i.e. risk importance) and is further reflected in the label usage strategies identified, which align to the negotiability and specificity of health-related goals. Integration of eye-tracking, retrospective think-aloud and interview data reveal that although labelling cues promoted non-volitional attention through design features, in the absence of ‘personal motivational relevance’, information attended to was discounted from the evaluation process, confirming the importance of needs-based information provision.

Cognitive grammar learning strategies in the acquisition of tense-aspect morphology in L3 Catalan

ABSTRACTThis study investigates grammar learning strategies and the acquisition of L2 tense-aspect morphology through the retrospective think-aloud processes of a group of L3 learners of Catalan. A total of 18 students of Catalan in a university setting in Catalonia were asked to select one of three tense-aspect forms (preterite, imperfect, and perfect) in a short narrative written in the past. Retrospective think-aloud protocols were used to investigate what they were thinking when they made the decision to choose the appropriate tense-aspect form. A bottom-up analysis identified nine grammar learning strategies that participants used to select the appropriate past tense forms. The results on the frequency of use of the strategies show that they were often used in combination and that the most common ones were translation, aspect, adverbial, and tense strategies. The analysis of strategy use according to type of morphology and selection of the correct form in the narrative showed that strategy use is related both to specific tense-aspect forms as well as to accuracy in the selection of forms.

A comparison of three think-aloud protocols used to evaluate a voice intelligent agent that expresses emotions

ABSTRACTThis study proposes two adapted think-aloud protocols for the evaluation of a voice intelligent agent. In the adapted retrospective think-aloud (RTA) protocol, users verbalise their thoughts based on the chat history after task-completion. In the adapted interactive think-aloud (ITA) protocol, users verbalise their thoughts regarding the intelligent agent being evaluated without the help of a facilitator. This study compares these two protocols with the classical think-aloud protocol (CTA) for evaluating an intelligent agent in terms of task time and verbal utterances. The influence of the intelligent agent’s emotional expression is also considered. The results suggest RTA is suitable for collecting user experience and causal explanation of utterances, CTA for collecting recommendation and prediction utterances, and ITA for collecting problem formulation and recommendation utterances. Furthermore, CTA and RTA can collect more total utterances, while CTA and ITA are influenced less the VIA’s emotional expression. This study provides guidelines by which future evaluators can choose suitable think-aloud protocols.

Conceptualizing the information seeking of college students on the autism spectrum through participant viewpoint ethnography

Due to increased numbers of diagnoses, targeted programs and initiatives, more students with autism spectrum disorder (ASD) are attending college (Heflin & Isbell, 2012; Zager & Alpern, 2007) but academic librarians and their staffs have not been trained to optimally serve this growing population. Utilizing wayfinding, think aloud protocol (TAP), retrospective think aloud protocol (RTAP), and a wearable camera, the actions, thoughts and feelings of a student with ASD and a neurotypical peer are evidenced as they navigate their campus library in search of materials. The library website, virtual maps to resource locations, and library workers served equally as enablers and barriers to both students in their information seeking. This study demonstrates that participant viewpoint ethnography is a viable research methodology for both neurotypical college students and those with autism.

Reasons for relating representations when reading digital multimodal science information

This exploratory study examined 16 participants’ reasons for relating representations when reading online information that presented weather forecasts comprising a variety of representations. They were told to advise a friend, who was planning to paint the exterior of his house, based on the online information. The participants’ multimodal reading was tracked with a SMI RED (50 Hz) eye-tracker, and their reasons for relating representations were investigated through a retrospective think-aloud protocol cued by the recording of the individual’s reading behaviour and a stand-alone interview. In line with expected behaviour, results suggest that decision-making processes influenced the construction of reading paths. More interestingly, results showed that the participants’ reasons for making transitions between representations were to control and compare information. The discussion focuses on this study’s unique contribution in terms of new conjectures for future research.

Using think-aloud protocols and interviews in investigating writers’ composing processes: combining concurrent and retrospective data

ABSTRACTResearch comparing the use of retrospective interviews (RIs) and think-aloud protocols (TAPs) as two data sources of writers’ composing processes is almost nonexistent. Trying to address this research gap, the present study examined the data obtained from the task-specific RIs and TAPs about the composing processes of 30 second language (L2) writers. Each participant thought aloud by concurrently verbalizing thoughts while performing an argumentative writing task, and then was interviewed immediately after the think-aloud session about their composing processes. The RI and TAP data was analyzed and compared in terms of five aspects of L2 writers’ composing: pre-writing stage, while-writing planning, first language (L1) use, composing problems and problem-solving, and text reviewing and changing. Though the results generally indicate that concurrent verbalizations provide much richer data than RIs about L2 writers’ composing processes, RIs were found to be a valuable source for revealing the ‘why’ of such processes. Besides, the gap between the data provided by the two sources differed from one composing aspect to another. Recommendations for how to optimally combine and make use of the two data sources in writing process research are provided.

Fall Prevention Self-Assessments Via Mobile 3D Visualization Technologies: Community Dwelling Older Adults' Perceptions of Opportunities and Challenges.

BackgroundIn the field of occupational therapy, the assistive equipment provision process (AEPP) is a prominent preventive strategy used to promote independent living and to identify and alleviate fall risk factors via the provision of assistive equipment within the home environment. Current practice involves the use of paper-based forms that include 2D measurement guidance diagrams that aim to communicate the precise points and dimensions that must be measured in order to make AEPP assessments. There are, however, issues such as “poor fit” of equipment due to inaccurate measurements taken and recorded, resulting in more than 50% of equipment installed within the home being abandoned by patients. This paper presents a novel 3D measurement aid prototype (3D-MAP) that provides enhanced measurement and assessment guidance to patients via the use of 3D visualization technologies.ObjectiveThe purpose of this study was to explore the perceptions of older adults with regard to the barriers and opportunities of using the 3D-MAP application as a tool that enables patient self-delivery of the AEPP.MethodsThirty-three community-dwelling older adults participated in interactive sessions with a bespoke 3D-MAP application utilizing the retrospective think-aloud protocol and semistructured focus group discussions. The system usability scale (SUS) questionnaire was used to evaluate the application’s usability. Thematic template analysis was carried out on the SUS item discussions, think-aloud, and semistructured focus group data.ResultsThe quantitative SUS results revealed that the application may be described as having “marginal-high” and “good” levels of usability, along with strong agreement with items relating to the usability (P=.004) and learnability (P<.001) of the application. Four high-level themes emerged from think-aloud and focus groups discussions: (1) perceived usefulness (PU), (2) perceived ease of use (PEOU), (3) application use (AU) and (4) self-assessment (SA). The application was seen as a useful tool to enhance visualization of measurement guidance and also to promote independent living, ownership of care, and potentially reduce waiting times. Several design and functionality recommendations emerged from the study, such as a need to manipulate the view and position of the 3D furniture models, and a need for clearer visual prompts and alternative keyboard interface for measurement entry.ConclusionsParticipants perceived the 3D-MAP application as a useful tool that has the potential to make significant improvements to the AEPP, not only in terms of accuracy of measurement, but also by potentially enabling older adult patients to carry out the data collection element of the AEPP themselves. Further research is needed to further adapt the 3D-MAP application in line with the study outcomes and to establish its clinical utility with regards to effectiveness, efficiency, accuracy, and reliability of measurements that are recorded using the application and to compare it with 2D measurement guidance leaflets.

Korean EFL Students’ Strategy Use in Gap-Filling Inference Items

This paper explores Korean EFL students’ test-taking strategy use in gap-filling inference items of the College Scholastic Aptitude Test (CSAT) through retrospective think-aloud protocols. Eight college freshmen participated in this study, and verbally reported the process of answering three gap-filling inference items in their L1. Students with higher proficiency levels tended to use language learner strategies with more variety, while students of lower levels often relied on familiar words and distractors as avoidance strategies. Also, the position of the gap in each item had an effect on the order and portions of reading. Unusual reading patterns were more frequently found from those who checked the gap first, when the gap was located in the middle or at the end of the passage. To foster positive washback effect of the test items on learning, this paper suggests that teachers encourage students to effectively employ language learner strategies in order to comprehend the text rather than rely on test-wiseness.