Think-aloud Protocols Research Articles

Assessing the content quality of essays in content and language integrated learning: Exploring the construct from subject specialists’ perspectives

Assessing the content of learners’ compositions is a common practice in second language (L2) writing assessment. However, the construct definition of content in L2 writing assessment potentially underrepresents the target competence in content and language integrated learning (CLIL), which aims to foster not only L2 proficiency but also critical thinking skills and subject knowledge. This study aims to conceptualize the construct of content in CLIL by exploring subject specialists’ perspectives on essays’ content quality in a CLIL context. Eleven researchers of English as a lingua franca (ELF) rated the content quality of research-based argumentative essays on ELF submitted in a CLIL course and produced think-aloud protocols. This study explored some essay features that have not been considered relevant in language assessment but are essential in the CLIL context, including the accuracy of the content, presence and quality of research, and presence of elements required in academic essays. Furthermore, the findings of this study confirmed that the components of content often addressed in language assessment (e.g., elaboration and logicality) are pertinent to writing assessment in CLIL. The manner in which subject specialists construe the content quality of essays on their specialized discipline can deepen the current understanding of content in CLIL.

The Importance of Social Cues When Browsing Appearance-Focused Social Media Content: A Think Aloud Protocol Analysis Using Fitspiration Images and Instagram Feed Browsing

ABSTRACT Previous research has demonstrated an impact of exposure to appearance-focused social media images on measures of female body image. However, the processes through which social media users consider and respond to appearance-focused content whilst browsing, or the degree to which this is reflective of their reactions to images typically encountered during every day browsing, are not fully understood. Think Aloud Protocol (TAP) was used to collect real-time, concurrent responses to Instagram content during two browsing tasks. Female participants browsed a set of 20 predefined fitspiration images, embedded within an online profile, and their own Instagram feed. Content analyses revealed that patterns of attentional focus in both tasks indicated the natural evaluation of images to extend far beyond simple comparison with the physical appearance of women. Participants engaged in more appearance appraisals when browsing fitspiration images compared to when browsing their own feed. A framework for understanding natural engagement with Instagram content is described which identifies the importance of extended social information when observing and judging appearance-focused images, via social media, that may be harmful to body image.

Assessing a nurse-assisted eHealth intervention posthospital discharge in adult patients with non-communicable diseases: a protocol for a feasibility study

IntroductionA growing number of patients with non-communicable diseases (NCDs), such as heart failure (HF) and colorectal cancer (CRC), are prone to comorbidity, a high rate of readmissions and complex healthcare...

The Role of Technology in Undergraduate Bioscience Laboratory Learning: Bridging the Gap between Theory and Practice

Integration of technology is widespread in laboratory teaching, whose purpose includes building theoretical understanding and practical skills. How second-year bioscience undergraduate students at a UK university use technology to construct their understanding of laboratory-based topics was investigated using a concurrent think-aloud protocol in the laboratory, followed by semi-structured interviews. Analysis of think-aloud data used socially shared metacognitive coding since students may co-construct their understanding in these collaborative spaces. This analysis demonstrated that participants used technology within the laboratory either as a tool to conduct their experiment or, as a source of information to help them understand, apply or perform their experimental task. Semi-structured interviews demonstrated that students integrated technology into all aspects of their laboratory learning. Eight out of the ten participants described using technology to help them make connections between theory and practice as part of post-laboratory activities such as analysing or conducting further research on the topic. A survey of UK bioscience undergraduate modules found that 22% of modules did not use post-laboratory activities, suggesting that more scaffolding of post-laboratory activities could provide bioscience students with greater integration of practical and theoretical understanding and consequently meaningful laboratory learning.

Exploring and modeling the reading-writing connection in EFL integrated writing.

Reading-to-write tasks have increasingly been used in high-stakes language tests worldwide; however, the nature of the reading-writing connection is not well understood. This study utilized a mixed-methods approach to ground descriptions of EFL cognitive processes and identify process interaction patterns to determine how reading and writing were connected. Grounded theory analysis of fourteen EFL learners' writing think-aloud protocols showed that students engaged in an interactive composing process involving source reading, comprehension monitoring, planning, language monitoring, narration monitoring, and continuity evaluation. We also conducted a confirmatory factor model study on 486 EFL learners' responses to a self-developed writing questionnaire, which covered five factors, including reading monitoring, narration monitoring, ideational planning, continuity evaluation, and skill integration. The findings showed that reading monitoring was the only factor that had a direct and significant impact on skill integration, a composite factor covering discourse synthesizing and source using processes. Based on the discussion of the theoretical, empirical, and pedagogical implications of the current findings, we called for more studies to explore the use of three pillar skills-reading, writing, and language use-to support EFL integrated writing. We also suggested that test designers include explicit rating descriptor(s) for source using to evaluate reading comprehension, and instructors enhance reading instruction to improve task performance.

THE EFFECTS OF STRATEGY TRAINING ON THE USE OF GLOBAL READING STRATEGIES: IMPLICATIONS FROM THINK-ALOUD AND EYE-TRACKING DATA

Considering the critical role of readings strategies in L2 reading this study aimed to investigate the impact of strategy training on Turkish EFL learners’ use of global reading strategies. 23 freshmen, enrolled in the ELT Department, participated in the study, and the data were collected through eye tracking and thinkaloud protocols. The results are encouraging for the use of reciprocal teaching, one of the comprehensionenhancing approaches to teach reading, in the EFL contexts. Eye tracking results, hand in hand with the think-aloud protocols’ findings, revealed that the participants utilised global reading strategies more frequently to a statistically significant degree after the strategy instruction. Thus, it was concluded that strategy training was useful for the participants to improve their use of global reading strategies, and a recent technological tool can be purposefully integrated into the data collection procedure of educational research studies. Pedagogical implications on the strategy training for the employment of global reading strategies that would help students improve reading skills need to be given attention by teachers.

How do older adults receiving aged care services understand and respond to the EQ-5D-5L? A think-aloud study in residential care

PurposeThe EQ-5D-5L is a preference-based instrument for measuring and valuing health-related quality of life (HRQoL). The EQ-5D-5L has been used extensively in economic evaluation, including in aged care. However, older adults’ understanding of the EQ-5D-5L has not been comprehensively investigated to date. This research aimed to assess older adults’ understanding of the EQ-5D-5L using a think-aloud protocol with two cognition groups: no cognitive impairment and mild/moderate cognitive impairment.MethodsParticipants’ cognition was assessed using the Standardised Mini-Mental State Examination (SMMSE). Face-to face interviews were conducted with concurrent and retrospective think-aloud encouraged through verbal probing. Audio recordings were transcribed, and qualitative analysis, informed by the Tourangeau four-stage Response Model (comprehension, retrieval, decision process, response process) was conducted in NVivo.ResultsIn total, 46 older adults (age 65 +) were recruited from 10 residential care facilities across South Australia (n = 25 no cognitive impairment, n = 21 mild/moderate cognitive impairment). Comprehension, retrieval, judgement and response mapping issues were common across all cognition levels and EQ-5D-5L dimensions. The two dimensions resulting in the most response issues were usual activities and personal care.ConclusionOlder adults may bring a different understanding to the EQ-5D-5L descriptive system than that expected given testing with general population samples. Dimension descriptors that are more relevant to this population may facilitate responses that better align with the underlying EQ-5D-5L concept model.

Rhetorical Strategies for Teaching Essay Writing: A Case Study Involving Saudi ESL Students

The use of rhetorical strategy in teaching writing has been a topic of discussion in ESL training programs since most teaching methodologies do not give the mother tongue a substantial role. This article presents findings from research on the rhetorical techniques pre-service teachers employ in writing both before and after receiving process-based writing instruction. The study focused on determining the writing techniques student teachers employ to complete an argumentative essay, and finding out how much pre-service teachers’ usage of writing methods is improved by using a rhetorical process approach to writing. The significance of the present study lies in the fact that it examines the writing abilities of Saudi students with different levels of language abilities and evaluates how rhetorical strategies help them in writing English easefully and with little or no flaws. The fundamental question to be probed through this study is to what extent rhetorical strategies can help teach successful writing to Saudi ESL learners. The data was obtained via the performance of an essay task utilizing two think-aloud protocols and based on a sample of 98 student teachers from Saudi Arabia. The findings of this study show rhetorical strategies such as generating ideas and rewriting are very essential in writing final essays by students. It is clear from the results that after students have mastered the choosing ideas process, they may include concepts that are relevant to the text. Professors should spend time, particularly teaching this kind of approach to help students become competent method users and writers generally, regardless of their audience. In concluding writing an article, it seemed like the strategy of translating from the native tongue to a foreign language was being utilized

CHANGES IN COGNITION AND NEUROCOGNITION WHEN THINKING ALOUD DURING DESIGN

AbstractThe think-aloud protocol provides researchers an insight into the designer's mental state, but little is understood about how thinking aloud influences design. The study presented in this paper sets out to measure the cognitive and neurocognitive changes in designers when thinking aloud. Engineering students (n=50) were randomly assigned to the think-aloud or control group. Students were outfitted with a functional near-infrared spectroscopy band. Students were asked to design a personal entertainment system. The think-aloud group spent significantly less time designing. Their design sketches included significantly fewer words. The think-aloud group also required significantly more resources in the left and right dorsolateral prefrontal cortex (DLPFC). The left DLPFC is often recruited for language processing, and the right DLPFC is involved in visual representation and problem-solving. The faster depletion of neurocognitive resources may have contributed to less time designing. Thinking aloud influences design cognition and neurocognition, but these effects are only now becoming apparent. More research and the adoption of neuroscience techniques can help shed light on these differences.

The influence of raters and scoring procedures on EIT outcomes: An exploratory study

This mixed-methods study examines the role of rating modality (aural vs. written) and rater characteristics (language background, linguistics training) on outcomes on a Spanish elicited imitation task (EIT) and provides insight into rater cognition during the scoring process. Thirty Spanish-English bilingual raters were divided into two conditions—one that scored test-takers’ audio-recorded oral EIT productions, the other that scored written transcriptions. Within each condition, raters varied according to their language background (native vs. nonnative Spanish speaker) and linguistics training (no training, minimal training, moderate training, extensive training). All raters scored the same six 30-item EITs using Ortega et al.’s (2002) rubric. Ten raters additionally completed a think-aloud protocol during their scoring. Ordinal logistic mixed-effects modelling was used to explore quantitative interrater differences in scoring (k = 5,400), and content analysis (Krippendorf, 2018) was employed to investigate patterns in the verbal reports. Results reveal significant differences in EIT outcomes according to rating modality and language background but not linguistics training. Raters scoring in the aural condition and native Spanish speakers rated test items higher than their written condition and nonnative Spanish-speaking counterparts. Analyses of the verbal reports suggest some systematic differences in rating considerations according to rating modality such as differences introduced by punctuation in the written modality or by the need to determine what language was produced in the aural modality. The findings contribute novel information for EIT validity arguments and have implications for rating procedures for future EIT studies.

A Think-Aloud Study of Novice Debugging

Debugging is a core skill required by programmers, yet we know little about how to effectively teach the process of debugging. The challenges of learning debugging are compounded for novices who lack experience and are still learning the tools they need to program effectively. In this work, we report a case study in which we used a think-aloud protocol to gain insight into the behaviour of three students engaged in debugging tasks. Our qualitative analysis reveals a variety of helpful practices and barriers that limit the effectiveness of debugging. We observe that comprehension, evidence-based activities, and workflow practices all contribute to novice debugging success. Lack of sustained effort, precision, and methodical processes negatively impact debugging effectiveness. We anticipate that understanding how students engage in debugging tasks will aid future work to address ineffective behaviours and promote effective debugging activities.

INVESTIGATING THE COGNITIVE PROCESSES IN POST-EDITING ACTIVITY USING THINKING ALOUD PROTOCOLS

The study aims to investigate the cognitive processes and meta-cognitive strategies used by the post-editor when doing the post-editing activity. It examines the way the post editor reads, thinks, speculates, and makes decision to the machine translation results and correcting the problems founds from the translated text. Thinking Aloud Protocols (TAPs) is used as the approach of the study to record the meta-cognitive strategies applied by the post editor. Several instruments such as video recording tools, machine-translated text, and timer are operated during the data collection. From the study, it is shown that reasoning and problem solving become the most running strategy from the post-editor. It occurs since the post-editor mostly looks for the problems and tries to find a way of correcting mistakes from the translation. The result also shows that the post-editor tries to comprehend the problems and decides the accurate decision when song the post-editing. The cognitive process and meta-cognitive strategies are quite different between the translator and the post-editor.

Factors behind L2 English learners’ performance of oppositional speech acts: a look at pragmatic-related episodes (PREs) during thinking aloud

Abstract The current study looked at the influence of different L1s, different proficiency levels, and other factors on the think-alouds in connection with oppositional speech acts by eighty tertiary-level Korean and Chinese learners of L2 English at two proficiency levels. The research instruments consisted of think-aloud protocols elicited during the learners’ production of written oppositional speech acts in the form of responses to the other learners’ essays, questionnaires, and focus group interviews. In particular, in the think-aloud protocols the number and content of pragmatic-related episodes (PREs) were identified. The results indicated that proficiency level, but not L1, significantly impacted the learners’ think-alouds. Other factors were revealed to include intensification of claims, focus on face issues, pragmatic transfer, and insufficient syntactic/semantic knowledge. These factors related to the sub-categories of the PREs involving ‘lexical upgrading’ and ‘external downgrading’.

ReflAct: Formative assessment for teacher reflection in video-coaching settings

Video coaching appears to be a promising approach for teachers' professional development, but is not automatically productive. To our knowledge, there are no tools to help coaches to formatively assess the quality of reflective talk. Such a tool is needed to help coaches realize the potential of reflective video coaching conversations. Key features of a minimally viable prototype were explored through a mixed-methods proof-of-concept study, focusing on (perceptions of) added value, compatibility, clarity, and tolerance. A total of 17 participants with insight into teachers' reflective practices participated in an online study. Data were collected through think-aloud protocols, interviews, surveys, and use logs. Findings showed modest, yet concrete evidence that a tool would be viable and is worth developing further. First, users indicated that it could add value in terms of development and calibration that outweighed the associated costs. Second, the tool was deemed compatible with improvement needs, though respondents pointed out that attention should be paid to user differences, and to strengthening familiarization opportunities. Third, seven sub-factors require explanation to increase the tool's clarity. Finally, the tool appeared tolerant of various modalities of use, but reasoning about the nature of reflective conversations requires support. Together, the results yield quality criteria that can be used for further development and evaluation, and the findings extend current theory on the collaborative learning and reflection of (pre- and in-service) teacher educators. Overall, this study illustrates how digital technology can facilitate operationalization of ambiguous processes like reflective talk, and highlights the importance of attending to the implementation early on.

Faculty Process for Reviewing and Utilizing a School’s Course Evaluation Comments

ObjectiveThis study aimed to explore faculty engagement with qualitative comments from course evaluations. MethodsCourse faculty from the University of North Carolina Eshelman School of Pharmacy were recruited via email to participate in a 30-minute interview session. Previous course evaluation comments were adapted to create a de-identified mock evaluation. Six interviews were conducted via Zoom, consisting of a think-aloud protocol based on the mock course evaluation followed by a cognitive interview focused on goals and current utilization of comments, and common patterns and issues sought by faculty. Interview transcripts were manually cleaned and de-identified. Transcripts were inductively coded by 1 researcher using MAXQDA. ResultsThree overarching themes were identified: general faculty process for reviewing comments (ie, how faculty perceive and analyze comments), comments utilization for course change (ie, how faculty utilize comments in making course changes), and faculty analysis strategy (ie, faculty approach to locating common patterns in evaluation comments). The most common subthemes included usefulness of comments, methods for tracking comment patterns, and challenges with the large number of comments each semester. ConclusionFaculty provided useful insight and feedback regarding the current state of the course evaluation process that can be used to improve the structure, organization, and utilization of course evaluations by faculty. These findings could inform the creation of the course evaluation comment automated analysis program in the next stage of an ongoing project.

Designing audit and feedback dashboards for dentists to monitor their opioid prescribing

Background and objectivePrescription drug abuse is a major factor leading to drug overdose deaths in the US and dentists are one of the leading prescribers of opioid pain medication. Knowing that Audit & Feedback (A&F) dashboards are an effective tool and are used as quality improvement interventions, we aimed to develop such dashboards personalized for dental providers which could allow them to monitor their own opioid prescribing performance. MethodsIn this paper we report on the process for designing the A&F dashboards for dentists which were developed by using an iterative human-centered design process. The results obtained from each iteration were used to enrich the information needs analyses, provide function testing, and guide the design decisions of the next iteration. ResultsEngaging dentists in the development and refinement of the dashboards while using the think-aloud protocol for user-testing, provided rapid feedback and identified areas that were confusing and needed either a redesign or additional explanatory content. The final version of dashboards consisted of displaying necessary information through easy to interpret visualizations and interactive features. These included providing access to current national and organizational prescribing guidelines, displaying changes in individual prescribing behavior over time, comparing individual prescribing rate to peer group rate and target rate, displaying procedure specific prescribing, integrating patient reported post-operative dental pain experience and providing navigation and interpretation tips for users. The dashboards were easy to learn and understand for the dentists and were deemed as worth using often in dental practice. ConclusionOur research was able to demonstrate the creation of useful and usable A&F dashboards using data from electronic dental records and patient surveys, for dentists to effectively monitor their opioid prescribing behavior. Efficacy of the dashboards will be tested in future work.

Metacognition & learning process: Using think-aloud protocol (TAP) to understand students and their teacher’s reflection processes during a problem-solving situation

Introduction. Today, the act of teaching has become increasingly intricate. Multiple fields of science now aid in comprehending this complexity, enabling instructors to support learners through-out their educational journey.Aim. The main aim of this study is to determine cognitive and metacognitive thinking process of students and their teachers during a problem-solving situation.Research methodology and methods. The authors used think-aloud protocol (TAP) in which students (12 participants at the same level) were asked to verbalise their thoughts during a learning activity (math exercise and written production). Similarly, during a pedagogical intervention, nine participants with varying levels of professional experience were required to articulate their professional practices as educators. It should be noted that our approach is purely qualitative following Ericsson and Simon’s approach, from data collection step to coding system and processing of these data.Results and scientific novelty. The results showed that half of the students solved well what was asked in the problem-solving situation, contrary to the other participants who found particular difficulties in each type of situation proposed (in math and in written production). For the teachers, their verbalisations tend towards three aspects with a degree of dominance for each teacher. The authors consider hat their research is a first step towards a new approach of evaluation of the teaching-learning act that includes both the teacher and the learners simultaneously.Practical significance. The results obtained can be used by pedagogical practitioners to better understand how their learners think on the one hand and develop their professional practices on the other.

"A Great Way to Start the Conversation": Evidence for the Use of an Adolescent Mental Health Chatbot Navigator for Youth at Risk of HIV and Other STIs.

Chatbot use is increasing for mobile health interventions on sensitive and stigmatized topics like mental health because of their anonymity and privacy. This anonymity provides acceptability to sexual and gendered minority youth (ages 16-24) at increased risk of HIV and other STIs with poor mental health due to higher levels of stigma, discrimination, and social isolation. This study evaluates the usability of Tabatha-YYC, a pilot chatbot navigator created to link these youth to mental health resources.Tabatha-YYC was developed using a Youth Advisory Board (n = 7). The final design underwent user testing (n = 20) through a think-aloud protocol, semi-structured interview, and a brief survey post-exposure which included the Health Information Technology Usability Evaluation Scale.The chatbot was found to be an acceptable mental health navigator by participants.This study provides important design methodology considerations and key insights into chatbot design preferences of youth at risk of STIs seeking mental health resources.

EFL Pre-service Teachers' Open Internet Reading Processing and Its Implication for Teaching: A Guided Think-Aloud Study

The mixed-method study explores EFL pre-service teachers' cognitive processes during open Internet reading in the teaching practice. The data were analyzed through t-test and think-aloud protocol (TAP). The quantitative findings indicated that the participants in the TAP group performed better in reading performance than the non-TAP’s. The qualitative results indicated that the TAP group applied their reading processing through problem identification, selection, and elaboration.

A Study on the Effects of Lexical Processing Strategies in Incidental Vocabulary Acquisition While Reading

Research on vocabulary acquisition in SLA has revealed that a large proportion of vocabulary is acquired without overt intention. This paper analyzes the relevance of different lexical processing strategies for incidental vocabulary acquisition while reading involving 56 native Chinese speakers who are studying English at a local university in China. The lexical processing strategies which the participants adopted to analyze unknown words are discussed based on the cognitive processes involved, namely implicit/explicit cognitive processes and top-down/bottom-up cognitive processes. According to the introspective data collected during a think-aloud protocol as well as the results of a subsequent vocabulary retention test, we examine the acquisition effects of different strategies. The results indicate that students can learn vocabulary incidentally through implicit processing, though it has a significantly lower acquisition rate than that of the explicit processing strategies. With regard to the dichotomy of top-down and bottom-up cognitive processes, the bottom-up processing strategy demonstrates better acquisition effects than the top-down strategy. Finally, a multilevel cognitive model of factors that contribute to incidental vocabulary acquisition is developed in an attempt to provide theoretical and practical implications to L2 vocabulary teaching and learning in China.