Think-aloud Technique Research Articles

User-Centered Design and Usability of Voxe as a Pediatric Electronic Patient-Reported Outcome Measure Platform: Mixed Methods Evaluation Study.

Electronic patient-reported outcome measures (ePROMs) are standardized digital instruments integrated into clinical care to collect subjective data regarding patients' health-related quality of life, functional status, and symptoms. In documenting patient-reported progress, ePROMs can guide treatment decisions and encourage measurement-based care practices. Voxe is a pediatric and user-centered ePROM platform for patients with chronic health conditions. We aimed to describe the user-centered design approach involving feedback from end users and usability testing of Voxe's platform features to support implementation in a pediatric health care setting. Purposive sampling was used to recruit patients aged 8-17 years from 2 chronic illness populations in 2 pediatric hospitals in Canada. Patients' health care team members were also purposively recruited. One-on-one iterative testing sessions were conducted digitally by research team members with participants to obtain feedback on the appearance and functionalities of the Voxe platform prototype. Patients and health care providers (HCPs) completed Voxe-related task-based activities. International Organization for Standardization key performance indicators were tracked during HCP task-based activities. HCPs also completed the System Usability Scale. To test platform usability, the think-aloud technique was used by participants during the completion of structured tasks. After completing all task-based activities, patient participants selected 5 words from the Microsoft Desirability Toolkit to describe their overall impression and experience with the Voxe platform. Qualitative data about likes, dislikes, and ease of use were collected through semistructured interviews. Feedback testing sessions were conducted with patients and HCPs until Voxe was acceptable to participating end users, with no further refinements identified. Quantitative and qualitative data analysis were completed using descriptive statistics and content analysis. A total of 49 patients and 38 HCPs were recruited. Patients were positive about Voxe's child-centered design characteristics and notification settings. HCPs rated Voxe as user-friendly and efficient, with the time to complete tasks decreasing over time. HCPs were satisfied with the Voxe platform functionalities and identified the value of Voxe's system notifications, summarized display of ePROM results, and its capacity to integrate with electronic medical records. Patients' and HCPs' high satisfaction rates with the Voxe prototype highlight the importance of being responsive to user suggestions from the inception of eHealth platform developments to ensure their efficient and effective design. This paper describes the user-centered creation and usability testing of Voxe as an ePROM platform for implementation into clinical care for pediatric patients with chronic health conditions. As a patient-facing platform that can be integrated into electronic medical records, Voxe aligns with measurement-based care practices to foster quality patient-centered approaches to care. End users' positive feedback and evaluation of the platform's user-friendliness and efficiency suggest that Voxe represents a valuable and promising solution to systematically integrate patient-related outcome (PRO) data into complex and dynamic clinical health care settings. RR2-10.1136/bmjopen-2021-053119.

The impact of unfolding case studies combined with think-aloud strategies on the clinical reasoning and self-directed learning abilities of postgraduate students: A mixed methods study

AimTo conduct unfolding case studies and test their effectiveness in improving clinical reasoning, teamwork and self-directed learning among postgraduate students. BackgroundPostgraduate students, as advanced nursing professionals in clinical settings, are the driving force behind the rapid development of the nursing profession. Effective clinical reasoning is a fundamental nursing skill that postgraduate students must cultivate, having a direct impact on patient health outcomes. The development and usation of unfolding case studies, which reflect the evolving conditions of patients, combined with think-aloud teaching methods, can enhance postgraduate students' clinical reasoning abilities and foster communication and self-reflection, thereby achieving this goal. DesignMixed methods design. MethodsThe Nurses' Clinical Reasoning Scale and the Nursing Students' Self-Directed Learning Ability Scale were used to evaluate the clinical reasoning and self-directed learning abilities of nursing students. A qualitative exploratory design with a think-aloud interview technique was employed to explore the clinical reasoning process of nursing students in unfolding cases. Twenty-one nursing students completed the questionnaire survey. Data analysis was conducted using Wilcoxon signed-rank tests, Spearman correlations, regressions and inductive content analysis. ResultsAfter implementing the case study, there was a statistically significant improvement in students' clinical reasoning, self-directed learning and teamwork abilities. Think-aloud analysis revealed that the cognitive strategies most employed by students in clinical reasoning were 'Making choices', 'Forming relationships', 'Searched for information' and 'Drawing conclusions'. ConclusionUnfolding case studies combined with think-aloud strategies provide a conducive learning environment for postgraduate students, effectively enhancing their clinical reasoning and self-directed learning abilities.

Development and psychometric evaluation of a questionnaire for the assessment of depression in primary care: a cross-sectional study

ObjectivesTo develop a new questionnaire for the diagnostic assessment of depression adapted to the primary care setting by combining psychiatric criteria and heuristics of general practitioners (GPs). Psychometric evaluation of...

Are the content and usability of a new direct observation tool adequate for assessing competency in delivering person-centred care: a think-aloud study with patients and healthcare professionals in Sweden

ObjectiveTo evaluate the content and usability of a new direct observation tool for assessing competency in delivering person-centred care based on the Gothenburg Centre for Person-Centred Care (gPCC) framework.DesignThis is...

Accessibility, Relevance, and Impact of a Symptom Monitoring Tool for Home Hospice Care: Theory Elaboration and Qualitative Assessment.

Early users found Engagement and Visualization to Improve Symptoms in Oncology Care (ENVISION), a web-based application designed to improve home management of hospice patients' symptoms and support patients' and family caregivers' well-being, to be generally useful and easy to use. However, they also raised concerns about potential challenges users with limited technological proficiency might experience. We sought to concurrently accomplish two interrelated study aims: (1) to develop a conceptual framework of digital inclusivity for health information systems and (2) to apply the framework in evaluating the digital inclusivity of the ENVISION application. We engaged ENVISION users (N=34) in a qualitative study in which data were collected via direct observation, think-aloud techniques, and responses to open-ended queries. Data were analyzed via theory elaboration and basic qualitative description. Accessibility, relevance, and impact were identified as 3 essential considerations in evaluating a health system's digital inclusivity. Study findings generally supported ENVISION's digital inclusivity, particularly concerning its perceived relevance to the work of family caregivers and hospice clinicians and its potentially positive impact on symptom management and quality of life. Limitations to ENVISION's digital inclusivity centered around issues of accessibility, particularly operability among individuals with limited technological knowledge and skills. The Accessibility, Relevance, and Impact conceptual framework of digital inclusivity for health information systems can help identify opportunities to strengthen the digital inclusivity of tools, such as ENVISION, intended for use by a broad and diverse range of users.

Qualitative study exploring the design of a patient-reported symptom-based risk stratification system for suspected head and neck cancer referrals: protocol for work packages 1 and 2 within the EVEREST-HN programme

IntroductionBetween 2009/2010 and 2019/2020, England witnessed an increase in suspected head and neck cancer (sHNC) referrals from 140 to 404 patients per 100 000 population. 1 in 10 patients are...

Exploring Comprehension Strategies of Modular Process Models: A Combined Eye-Tracking and Concurrent Think-Aloud Study.

The study of complex process models often encounters challenges in terms of comprehensibility. This paper explores using modularization as a strategy to mitigate such challenges, notably the reduction in complexity. Previous research has delved into the comprehensibility of modularized process models, yet an unresolved question about the cognitive factors at play during their comprehension still needs to be answered. Addressing the latter, the paper presents findings from an innovative study combining eye-tracking and concurrent think-aloud techniques involving 25 participants. The study aimed to comprehend how individuals comprehend process models when presented in three different modular formats: flattened process models, models with grouped elements, and models with subprocesses. The results shed light on varying comprehension strategies employed by participants when navigating through these modularized process models. The paper concludes by suggesting avenues for future research guided by these insights.

Modification of the Paediatric Gastro-oesophageal Reflux Disease Symptom and Quality of Life Questionnaire (PGSQ) for children with cerebral palsy: a preliminary study

ObjectiveGastro-oesophageal reflux disease (GORD) is a common condition affecting children, characterised by the passage of gastric contents into the oesophagus causing pain, vomiting and regurgitation. Children with neurodisability (such as...

사회초년생을 위한 경력개발 지원 챗봇(Career Development Supporting Chatbot, CDSC) 시나리오 디자인

Purpose: The purpose is to design a Career Development Supporting Chatbot (CDSC) scenario for early careers.
 Method: A literature study was conducted to derive the chatbot's core values, support processes, and resources for chatbot design. FGI was conducted by four experts to review the content validity of the design plan of the derived chatbot scenario, and a study was conducted based on the design plan. After developing a prototype, think-aloud technique was applied to 7 early careers. To ensure the reliability of the overall study, triangulation verification of the collected data was conducted.
 Results: A scenario for a chatbot to support career development for beginners was developed, a chatbot prototype was implemented based on this.
 Conclusion: it is necessary to expand the chatbot's functions and develop a chatbot that can provide customized information to users based on the diagnosed career anchor type and qualitative data collected through interviews.

Influence of Reading Strategies on ESL Students’ Reading Comprehension in Secondary Schools in Malé, Maldives

The use of reading strategies plays an integral part in the reading comprehension level of students. However, English as a Second Language (ESL) learners are unaware of the impact of reading strategies on their reading comprehension level. Moreover, teachers also lack a solid foundation for teaching reading strategies to students and are unable to understand the reading strategies preferences of the students. Only a few studies have been conducted highlighting the significance of reading strategies on students’ comprehension levels, especially in the context of Maldivian secondary schools. The main objective of this study is to identify the use of reading strategies to improve the reading comprehension level of ESL learners in secondary school. This research employed an experimental design. A total of 118 students from grades 9 and 10 from a secondary school participated in the study. Two instruments were utilized to collect data: a questionnaire, a pre-test, and a post-test. Statistical analysis revealed a strong positive correlation between reading strategies and comprehension achievement. Moreover, a significant difference in strategy use was found between male and female students. Recommendations include implementing school-wide reading programs, using think-aloud technique to model strategies, and consistent teacher training in Maldives as well as monitoring of students' reading strategies to improve comprehension and learning opportunities.

Thought Processes as Data in Eye Care Education and Research.

Working memory is a component of the human memory system in which all thought processes take place during an activity. Asking participants to verbalize their thoughts (think-aloud method) is a well-established technique in cognitive psychology and computer science for obtaining access to mental processes in order to investigate in-performance cognitions. Few attempts have been made to use such data in eye care research. This article provides a concise overview of the think-aloud technique and discusses potential research areas where this method can produce useful data.

Remote Moderated Usability Testing of a Mobile Phone App for Remote Monitoring of Pregnant Women at High Risk of Preeclampsia in Karachi, Pakistan

This study assessed the usability of the smartphone app, named “Raabta” from the perspective of pregnant women at high risk of preeclampsia to improve the Raabta app for future implementation. Think-aloud and task-completion techniques were used with a purposive sample of 14 pregnant women at high risk of preeclampsia. The sessions were audio-recorded and later professionally transcribed for thematic analysis. The study generated learnings associated with four themes: improving the clarity of instructions, messaging, and terminology; accessibility for non-tech savvy and illiterate Urdu users; enhancing visuals and icons for user engagement; and simplifying navigation and functionality. Overall, user feedback emphasized the importance of enhancing the clarity of instructions, messaging, and terminology within the Raabta app. Voice messages and visuals were valued by users, particularly among the non-tech savvy and illiterate Urdu users, as they enhance accessibility and enable independent monitoring. Suggestions were made to enhance user engagement through visual improvements such as enhanced graphics and culturally aligned color schemes. Lastly, users highlighted the need for improved navigation both between screens and within screens to enhance the overall user experience. The Raabta app prototype will be modified based on the feedback of the users to address the unique needs of diverse groups.

Exploring differences in self-regulated learning strategy use between high- and low-performing students in introductory programming: An analysis of eye-tracking and retrospective think-aloud data from program comprehension

Previous studies have reported mixed results regarding the relationship between students’ use of self-regulated learning (SRL) strategies and their performance in introductory programming courses. These studies were constrained by their reliance on self-report questionnaires as a means of collecting and analysing data. To address this limitation, this study aimed to employ eye-tracking and retrospective think-aloud techniques to identify differences in SRL strategy use for program comprehension tasks between high-performing students (N = 31) and low-performing students (N = 31) in an undergraduate programming course. All participants attended individual eye-tracking sessions to comprehend two Python program codes with different constructs. Their eye-tracking data and video-recalled retrospective think-aloud data were captured and recorded for analysis. The findings reveal that higher-order cognitive skills, such as elaboration and critical thinking, were mostly adopted by high-performing students, while basic cognitive and resource management strategy, such as rehearsal and help-seeking, were mostly employed by low-performing students when comprehending the program codes. This study not only demonstrates the design of combining eye-tracking and retrospective think-aloud data to explore students’ use of SRL strategies but also provides evidence to support the notion that program comprehension is a complex process that cannot be effectively addressed by employing merely rudimentary strategies, such as repetitively reading the same code segment. In the future, researchers could explore the possibility of using a webcam to monitor and assess students’ online programming processes and provide feedback based on their eye movements. They could also examine the effects of SRL strategies training on students’ motivation, engagement, and performance in various types of programming activities.

Face Validity of Four Preference-Weighted Quality-of-Life Measures in Residential Aged Care: A Think-Aloud Study.

There is an increased use of preference-weighted quality-of-life measures in residential aged care to guide resource allocation decisions or for quality-of-care assessments. However, little is known about their face validity (i.e., how understandable, appropriate and relevant the measures are 'on their face' when respondents complete them). The aim of this study was to assess the face validity of four preference-weighted measures (i.e., EQ-5D-5L, EQ-HWB, ASCOT, QOL-ACC) in older people living in residential aged care. Qualitative cognitive think-aloud interviews were conducted using both concurrent and retrospective think-aloud techniques. To reduce burden, each resident completed two measures, with the four measures randomised across participants. Audio recordings were transcribed and framework analysis was used for data analysis, based on an existing framework derived from the Tourangeau four-stage response model. In total, 24 interviews were conducted with residents living across three residential aged care facilities in Melbourne, Australia. Response issues were identified across all four measures, often related to comprehension and difficulty selecting a response level due to double-barrelled and ambiguous items that have different meanings in the residential aged care context. We also identified issues related to understanding instructions, non-adherence to the recall period, and noted positive responding that requires attention when interpreting the data. Our findings provide further evidence on the appropriateness of existing measures, indicating numerous response issues that require further research to guide the selection process for research and practice.

A Web-Based Tool to Report Adverse Drug Reactions by Community Pharmacists in Australia: Usability Testing Study.

Adverse drug reactions (ADRs) are unintended and harmful events associated with medication use. Despite their significance in postmarketing surveillance, quality improvement, and drug safety research, ADRs are vastly underreported. Enhanced digital-based communication of ADR information to regulators and among care providers could significantly improve patient safety. This paper presents a usability evaluation of the commercially available GuildCare Adverse Event Recording system, a web-based ADR reporting system widely used by community pharmacists (CPs) in Australia. We developed a structured interview protocol encompassing remote observation, think-aloud moderating techniques, and retrospective questioning to gauge the overall user experience, complemented by the System Usability Scale (SUS) assessment. Thematic analysis was used to analyze field notes from the interviews. A total of 7 CPs participated in the study, who perceived the system to have above-average usability (SUS score of 68.57). Nonetheless, the structured approach to usability testing unveiled specific functional and user interpretation issues, such as unnecessary information, lack of system clarity, and redundant data fields-critical insights not captured by the SUS results. Design elements like drop-down menus, free-text entry, checkboxes, and prefilled or auto-populated data fields were perceived as useful for enhancing system navigation and facilitating ADR reporting. The user-centric design of technology solutions, like the one discussed herein, is crucial to meeting CPs' information needs and ensuring effective ADR reporting. Developers should adopt a structured approach to usability testing during the developmental phase to address identified issues comprehensively. Such a methodological approach may promote the adoption of ADR reporting systems by CPs and ultimately enhance patient safety.

Cognitive Testing of Items Measuring Movement Behaviours in Young Children Aged Zero to Five Years: Development of the Movement Behaviour Questionnaires for -Baby (MBQ-B) and -Child (MBQ-C).

This paper describes the cognitive interview phase of the development of two brief surveys, the Movement Behaviour Questionnaire-Baby (MBQ-B) and Movement Behaviour Questionnaire-Child (MBQ-C), which measure the duration of physical activity, screen time, and sleep of children aged 0-5 years. The aims were (1) review the format, content, and clarity of questionnaire items and response options, (2) understand how parents retrieve, encode, and formulate responses when asked about their child's movement behaviours, and (3) identify potential sources of response error and make appropriate modifications. Interviews with parents of children aged 0-5 years were conducted using concurrent think-aloud techniques and probing questions. Parents reviewed the MBQ-B and/or MBQ-C depending on the developmental stage of their child(ren). Twenty-nine interviews were conducted with 20 parents, over four iterative rounds. Participants recalled usual family routines and rules when estimating the duration/frequency of behaviours. To estimate active play, parents referred to the child's daily routine considering wake and bedtimes, naps, and mealtimes. Participants were influenced by the examples provided, being unable to interpret these as exemplars only. Decomposing general items into specific questions with examples was well received. Use of numeracy skills when estimating duration was evident. Interviews informed revisions to item wording, examples, and recall prompts, which will be taken forward into the MBQ-B and MBQ-C validation studies. Utilising cognitive interviewing can enhance confidence that questionnaire items are correctly interpreted and understood by participants.

Usability Study of an Electronic Medical Record From the Nurse Practitioners' Practice: A Qualitative Study Using the Think-Aloud Technique.

Ensuring the usability of electronic medical records (EMRs) is crucial for healthcare providers to offer efficient, effective, and safe patient care. Nurse practitioners (NPs) are integral to the healthcare system and are essential in managing patient workflows. However, few studies assess NPs' perspectives on how EMR usage affects workflow and patient care quality. In this study, the "think-aloud technique" was utilized for usability testing. It involves observing users (NPs) as they complete their everyday tasks on the EMR while vocalizing their thoughts and emotions. This method has been proven reliable and valid through various research, such as a systematic review. The EMR system used by NPs can create a heavy cognitive workload, have limited functionality, can lead to unintended errors, and may experience technical difficulties. The EMR system used by NPs is challenging due to three main issues: high cognitive workload, limited system functionality, and technical problems. To improve the system, it is recommended to reduce the cognitive burden by customizing the user interface to fit the NPs' needs, enhancing the system's functionality by adding essential features and fixing any technical issues.

Teachers’ Beliefs and Practice on Providing EFL Written Corrective Feedback

<p class="05IsiAbstrak"><span>This study analyzes how and why writing instructors provide written corrective feedback (WCF) on academic writing tasks. It was an ethnography study conducted by the Department of English Education at University in Semarang, Indonesia. This study investigated and analyzed the attitudes of teachers to get insight into classroom methods for delivering written corrective feedback (CF) on student writing. The major objective of the study was to evaluate instructors' beliefs and practices. Therefore, the teachers' perspectives on providing written corrective feedback (WCF) were required to determine the true answers and provide corrective feedback. The conceptual basis for this study was developed from Borg's (2003) concept of teacher cognition (beliefs) and practices, as well as the impediments to written CF reported by several writers. This recent study conducted interviews with three English Education Department instructors who instructed writing classes. This investigation utilized classroom observation, the think-aloud technique, and in-depth interviews. The data was examined using Spradley's technique (1979, 1980). The study of the data revealed both congruence and discord between the beliefs and actions of lecturers. For instance, the instructors' focus and number of writing CF were consistent. In addition, it was demonstrated that extraneous influences affected the lecturers' application of their theories.</span></p>

OC-081 DIASTASIS RECTI REPAIR – DEVELOPMENT OF A PATIENT REPORTED OUTCOME MEASURE THROUGH QUALITATIVE ASSESSMENT

Abstract Aim Diastasis recti may lead to undesirable symptoms including aesthetic dissatisfaction and decreased quality of life (QoL). Currently, there is no validated disease-specific patient reported outcomes (PRO) measure to assess patient satisfaction and facilitate personalized care after repair. Our study aimed to develop a diastasis-specific PRO-measure through qualitative assessment within this population. Methods Diastasis recti repair patients were enrolled and semi-structured cognitive interviews using the think-aloud technique were conducted by a trained qualitative research member of the Mixed Methods Research Laboratory. Patients answered a 23-item questionnaire, derived from BODY-Q and AHQ, addressing four components–Physical, Appearance, Surgical Team, and Satisfaction with Abdomen. Results Thirteen patients who underwent diastasis repair completed the interviews, which were conducted an average of 353 days after surgery. All participants reported the questions in the survey were easy to interpret and assessed different constructs. Most questions in the study were relevant and easily interpreted, however, a few participants reported some items were unclear, specifically related to discerning between size and shape of their abdomen. When asked about aspects of their experience not addressed by the survey, patients mentioned sensory changes around the incision, and accordingly a question was added to the instrument. Conclusion As a first step toward creation of a diastasis recti PRO-measure, we present the results of qualitative patient assessment. Patients found the instrument clear and relevant to their experience. A void remains to optimize PROs after diastasis repair; therefore, our PRO-measure will provide surgeons a qualitative tool for quantitative assessment of their QoL.

Revision of the Advanced Cancer Patients’ Distress Scale (ACPDS): a mixed-methods study among palliative patients and healthcare professionals in Germany

ObjectivesTo revise the 37-item Advanced Cancer Patients’ Distress Scale (ACPDS) regarding its content, comprehensibility, applicability, and relevance by healthcare professionals (HCPs) and patients in order to enhance an existing instrument...