Think-aloud Protocols Research Articles

A data-driven approach to optimizing clinical study eligibility criteria

ObjectiveFeasible, safe, and inclusive eligibility criteria are crucial to successful clinical research recruitment. Existing expert-centered methods for eligibility criteria selection may not be representative of real-world populations. This paper presents a novel model called OPTEC (OPTimal Eligibility Criteria) based on the Multiple Attribute Decision Making method boosted by an efficient greedy algorithm. MethodsIt systematically identifies the optimal criteria combination for a given medical condition with the optimal tradeoff among feasibility, patient safety, and cohort diversity. The model offers flexibility in attribute configurations and generalizability to various clinical domains. The model was evaluated on two clinical domains (i.e., Alzheimer's disease and Neoplasm of pancreas) using two datasets (i.e., MIMIC-III dataset and NewYork-Presbyterian/Columbia University Irving Medical Center (NYP/CUIMC) database). ResultsWe simulated the process of automatically optimizing eligibility criteria according to user-specified prioritization preferences and generated recommendations based on the top-ranked criteria combination accordingly (top 0.41–2.75%) with OPTEC. Harnessing the power of the model, we designed an interactive criteria recommendation system and conducted a case study with an experienced clinical researcher using the think-aloud protocol. ConclusionsThe results demonstrated that OPTEC could be used to recommend feasible eligibility criteria combinations, and to provide actionable recommendations for clinical study designers to construct a feasible, safe, and diverse cohort definition during early study design.

Investigation of Prospective Mathematics Teachers’ Proof Completion Processes Supported By Key Ideas

The aim of this research was to examine prospective mathematics teachers’ completion processes of mathematical proofs supported by key ideas in the field of abstract algebra. Prospective teachers were asked to complete the parts of a proof left incomplete using those key ideas. It was decided together with three academics who are experts in the field of algebra which theorems to use and which proof sections to leave missing in the proof completion form. The proof completion form consisted of 5 proofs and was applied to 5 participants. The participants' processes of completing the proofs were evaluated in semi-structured interviews supported by the think-aloud method. Interviews were recorded by video and transcribed, and descriptive analysis was performed. The findings obtained were analyzed with think-aloud protocols. According to the results, key ideas can be said to play an active role in the proof process and the teaching of proof. In addition, developing activities consisting of proofs supported by key ideas can contribute to prospective teachers’ abilities to prove without memorization and internalize proof processes.

A Visual Analytic Tool (VIADS) to Assist the Hypothesis Generation Process in Clinical Research: Mixed Methods Usability Study

Visualization can be a powerful tool to comprehend data sets, especially when they can be represented via hierarchical structures. Enhanced comprehension can facilitate the development of scientific hypotheses. However, the inclusion of excessive data can make visualizations overwhelming. We developed a visual interactive analytic tool for filtering and summarizing large health data sets coded with hierarchical terminologies (VIADS). In this study, we evaluated the usability of VIADS for visualizing data sets of patient diagnoses and procedures coded in the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM). We used mixed methods in the study. A group of 12 clinical researchers participated in the generation of data-driven hypotheses using the same data sets and time frame (a 1-hour training session and a 2-hour study session) utilizing VIADS via the think-aloud protocol. The audio and screen activities were recorded remotely. A modified version of the System Usability Scale (SUS) survey and a brief survey with open-ended questions were administered after the study to assess the usability of VIADS and verify their intense usage experience with VIADS. The range of SUS scores was 37.5 to 87.5. The mean SUS score for VIADS was 71.88 (out of a possible 100, SD 14.62), and the median SUS was 75. The participants unanimously agreed that VIADS offers new perspectives on data sets (12/12, 100%), while 75% (8/12) agreed that VIADS facilitates understanding, presentation, and interpretation of underlying data sets. The comments on the utility of VIADS were positive and aligned well with the design objectives of VIADS. The answers to the open-ended questions in the modified SUS provided specific suggestions regarding potential improvements for VIADS, and the identified problems with usability were used to update the tool. This usability study demonstrates that VIADS is a usable tool for analyzing secondary data sets with good average usability, good SUS score, and favorable utility. Currently, VIADS accepts data sets with hierarchical codes and their corresponding frequencies. Consequently, only specific types of use cases are supported by the analytical results. Participants agreed, however, that VIADS provides new perspectives on data sets and is relatively easy to use. The VIADS functionalities most appreciated by participants were the ability to filter, summarize, compare, and visualize data. RR2-10.2196/39414.

Tell Us What You Really Think: A Think Aloud Protocol Analysis of the Verbal Cognitive Reflection Test.

The standard interpretation of cognitive reflection tests assumes that correct responses are reflective and lured responses are unreflective. However, prior process-tracing of mathematical reflection tests has cast doubt on this interpretation. In two studies (N = 201), we deployed a validated think-aloud protocol in-person and online to test how this assumption is satisfied by the new, validated, less familiar, and non-mathematical verbal Cognitive Reflection Test (vCRT). Verbalized thoughts in both studies revealed that most (but not all) correct responses involved reflection and that most (but not all) lured responses lacked reflection. The think-aloud protocols seemed to reflect business-as-usual performance: thinking aloud did not disrupt test performance compared to a control group. These data suggest that the vCRT usually satisfies the standard interpretation of the reflection tests (albeit not without exceptions) and that the vCRT can be a good measure of the construct theorized by the two-factor explication of 'reflection' (as deliberate and conscious).

Designing Virtual Natural Environments for Older Adults: Think-Aloud Study

Spending time in natural environments is beneficial for human health, but many older adults have limited or no access to natural environments. Virtual reality technology may be a means to facilitate nature experiences, and so, there is a need for knowledge on how to design virtual restorative natural environments for older adults. The aim of this study was to identify, implement, and test older adults' preferences and ideas regarding virtual natural environments. A total of 14 older adults (mean age 75, SD 5.9 years) participated in an iterative process to design such an environment. We used think-aloud protocols and qualitative content analysis and established questionnaires that targeted usability, affective aspects, and side effects. These data guided the design decisions for incremental implementations of a prototype. The participants' preferences included trueness to reality in terms of rendition and behavior; traces of human activity and natural processes that trigger the imagination and provide believability; the ability to roam, explore, and interact with the environment; and a familiar, relatable environment that evokes memories. The iterative design process resulted in a prototype featuring many of the participants' ideas and preferences, including a seated locomotion technique, animals, a boat ride, the discovery of a boat wreck, and apple picking. The questionnaire results indicated high perceived usability, interest, and enjoyment; low pressure and tension; moderate value and usefulness; and negligible side effects. We suggested 3 principles for virtual natural environments for older adults: realness, interactivity, and relatedness. Virtual natural environments should also provide a diversity of content and activities to accommodate the heterogeneity in older adults' preferences. These results can contribute to a framework for designing virtual natural environments for older adults. However, these findings need to be tested and potentially revised in future studies.

New Ways for Patients to Make Sense of Their Electronic Health Record Data Using the Discovery Web Application: Think-Aloud Evaluation Study.

In the United States, patients can access their electronic health record (EHR) data through patient portals. However, current patient portals are mainly focused on a single provider, with very limited data sharing capabilities and put low emphasis on independent sensemaking of the EHR data. This makes it very challenging for patients to switch between different portals and aggregate the data to obtain a complete picture of their medical history and to make sense of it. Owing to this fragmentation, patients are exposed to numerous inconveniences such as medical errors, repeated tests, and limited self-advocacy. To overcome the limitations of EHR patient portals, we designed and developed Discovery-a web-based application that aggregates EHR data from multiple providers and present them to the patient for efficient exploration and sensemaking. To learn how well Discovery meets the patients' sensemaking needs and what features should such applications include, we conducted an evaluation study. We conducted a remote study with 14 participants. In a 60-minute session and relying on the think-aloud protocol, participants were asked to complete a variety of sensemaking tasks and provide feedback upon completion. The audio materials were transcribed for analysis and the video recordings of the users' interactions with Discovery were annotated to provide additional context. These combined textual data were thematically analyzed to surface themes that reflect how participants used Discovery's features, what sensemaking of their EHR data really entails, and what features are desirable to support that process better. We found that Discovery provided much needed features and could be used in a variety of everyday scenarios, especially for preparing and during clinical visits and also for raising awareness, reflection, and planning. According to the study participants, Discovery provided a robust set of features for supporting independent exploration and sensemaking of their EHR data: summary and quick overview of the data, finding prevalence, periodicity, co-occurrence, and pre-post of medical events, as well as comparing medical record types and subtypes across providers. In addition, we extracted important design implications from the user feedback on data exploration with multiple views and nonstandard user interface elements. Patient-centered sensemaking tools should have a core set of features that can be learned quickly and support common use cases for a variety of users. The patients should be able to detect time-oriented patterns of medical events and get enough context and explanation on demand in a single exploration view that feels warm and familiar and relies on patient-friendly language. However, this view should have enough plasticity to adjust to the patient's information needs as the sensemaking unfolds. Future designs should include the physicians in the patient's sensemaking process and improve the communication in clinical visits and via messaging.

Cognitive and Metacognitive Processes Demonstrated by Pharmacy Students When Making Therapeutic Decisions

Objective. To characterize the types of cognitive and metacognitive processes demonstrated by third-year pharmacy students during a therapeutic reasoning activity.Methods. A qualitative, descriptive study following a think-aloud protocol was used to analyze the cognitive (analytical) and metacognitive processes observed by third-year pharmacy students as they completed a 25-minute therapeutic reasoning activity. Using a deductive codebook developed from literature about reasoning, two independent coders characterized processes from students' audio-recorded, transcribed think-aloud episodes while making therapeutic decisions about simulated clinical cases.Results. A total of 40 think-aloud episodes were transcribed among the cohort. Categorization of the think-aloud transcriptions revealed a series of cognitive analytical and metacognitive processes demonstrated by students during the therapeutic decision-making activity. A total of 1792 codes were categorized as analytical processes, falling into six major themes: 69% gathering information (1232/1792), 13% processing information (227/1792), 7% making assessments (133/1792), 1% synthesizing information (19/1792), 7% articulating evidence (117/1792), and 4% making a recommendation (64/1792). In comparison to gathering information, a much lower frequency of processing and assessment was observed for students, particularly for those that were unable to resolve the case. Students' movement between major analytical processes co-occurred commonly with metacognitive processes. Of the 918 codes categorized as metacognitive processes, two major themes arose: 28% monitoring for knowledge or emotions (257/918) and 72% controlling the planning of next steps or verification of correct information (661/918). Sequencing the codes and co-occurrences of processes allowed us to propose an integrated cognitive/metacognitive model of therapeutic reasoning for students.Conclusion. This study categorizes the cognitive (analytical) and metacognitive processes engaged during pharmacy students' therapeutic reasoning process. The findings can inform current instructional practices and further research into educational activities that can strengthen pharmacy students' therapeutic reasoning skills.

Exploring the Process and Strategies of Chinese–English Abstract Writing Using Machine Translation Tools

The present study explores English as a foreign language (EFL) learners’ processes and strategies when using machine translation (MT) tools in academic abstract writing. Eight EFL graduate students were introduced to translation-friendly writing strategies using Google Translate and were required to produce an English abstract with the aid of a machine translation tool. The study used qualitative and quantitative approaches in data collection and analysis. A triangulation process was developed and implemented, including think-aloud protocols during the writing session, surveys, and individual interviews after the writing session. The findings suggested that the translation-friendly writing strategies introduced to the participants were useful in enhancing the quality of their writing. Each participant demonstrated individual strategic uses of MT. Among the various strategies reported, back translation was the most commonly adopted one; that is, they first composed an abstract in Chinese (L1) and engaged in multiple rounds of translation between Chinese and English using MT; when problems were identified in the English abstract, they modified the Chinese abstract using translation-friendly writing strategies to enhance the quality of MT translation output. Most of the translation problems identified by the participants were related to non-academic expressions. While participants were satisfied with the quality of the abstracts produced with the aid of MT, they raised ethical concerns regarding the use of MT in academic writing. These findings suggest that MT has fundamentally changed the process of academic writing in English and call for the re-examination of the purpose of academic writing instruction and the approaches employed.

A Model for Usability Evaluation of Learning Management Systems

Learning management systems (LMSs) are increasingly popular in education. The usability of these systems must be examined to ensure they meet the needs of both students and faculty. To address the limitations of existing models for evaluating the usability of LMS, this research proposes a new model that combines objective measures collected from eye-tracking with subjective measures collected from retrospective think-aloud protocols and questionnaires. This model evaluates the effectiveness, efficiency, satisfaction, and learnability of LMSs. The proposed model was applied to study the usability of the LMS used at King Saud University (Blackboard) and was compared with other existing usability evaluation models. The findings indicate that the proposed model fulfilled its intended purpose and provided more comprehensive results than existing models.

Early validity and reliability evidence for the American Board of Emergency Medicine Virtual Oral Examination.

The American Board of Emergency Medicine (ABEM) in-person Oral Certification Examination (OCE) was halted abruptly in 2020 due to the COVID-19 pandemic. The OCE was reconfigured to be administered in a virtual environment starting in December 2020. The purpose of this investigation was to determine whether there was sufficient validity and reliability evidence to support the continued use of the ABEM virtual Oral Examination (VOE) for certification decisions. This retrospective, descriptive study used multiple data sources to provide validity evidence and reliability data. Validity evidence focused on test content, response processes, internal structure (e.g., internal consistency and item response theory), and the consequences of testing. A multifaceted Rasch reliability coefficient was used to measure reliability. Study data were from two 2019 in-person OCEs and the first four VOE administrations. There were 2279 physicians who took the 2019 in-person OCE examination and 2153 physicians who took the VOE during the study period. Among the OCE group, 92.0% agreed or strongly agreed that the cases on the examination were cases that an emergency physician should be expected to see; 91.1% of the VOE group agreed or strongly agreed. A similar pattern of responses given to a question about whether the cases on the examination were cases that they had seen. Additional evidence of validity was obtained by the use of the EM Model, the process for case development, the use of think-aloud protocols, and similar test performance patterns (e.g., pass rates). For reliability, the Rasch reliability coefficients for the OCE and the VOE during the study period were all >0.90. There was substantial validity evidence and reliability to support ongoing use of the ABEM VOE to make confident and defensible certification decisions.

Usability Evaluation with Eye Tracking: The Case of a Mobile Augmented Reality Application with Historical Images for Urban Cultural Heritage

Eye-tracking technologies have matured significantly in recent years and have become more affordable and easier to use. We investigated how eye-tracking technology can be applied to evaluate the usability of mobile augmented reality applications with historical images for urban cultural heritage. The experiment involved a series of complex user evaluation sessions, combining semi-structured interviews, observations, think-aloud protocol, SUS questionnaire, and product reaction cards, complemented by eye tracking, to gather insights on the Spotlight Timisoara AR mobile application, part of a digital storytelling multiplatform for the city of Timisoara (Romania), soon to be European Capital of Culture in 2023. The results indicate strong and weak aspects of the application, both as expressed by the participants and as derived from analyzing the eye-tracking data. The paper also lists the main challenges we identified in using eye-tracking equipment to evaluate the usability of such mobile augmented reality applications for urban outdoor heritage.

Miscommunication and Employee Power Dynamics May Affect Student Navigation of Library Resources

A Review of: Almeida, N., & Tidal, J. (2022). Library wayfinding and ESOL students: Communication challenges and empathy-based intervention. portal: Libraries and the Academy, 22(2), 453–474. https://doi.org/10.1353/pla.2022.0025 Objective – To map the experiences of students of English for speakers of other languages (ESOL) navigating an academic library. Design – A wayfinding study to evaluate how students navigate a library. Setting – An urban-based academic library at an institution of higher education. Subjects – Students of English for speakers of other languages (ESOL). Methods – A mixed methods study including visual recordings, web screen capture, interviews, and surveys. Subjects were recruited through email. Twelve participants were selected and given an initial screening survey. They were given four tasks to complete: Find a book in the stacks, find a book in the reserves, find a DVD in media, and find a database. They were equipped with a GoPro camera and were given a think-aloud protocol (TAP). They were then given a post-task debriefing interview. Qualitative data were analyzed and coded. Quantitative data like success of task and time to completion were also recorded. Main Results – Success rate varied among tasks: Finding a book in reserves had the highest rate at 75%, while finding a database had the lowest at 50%. Time also varied from 12 minutes to find a book in the stacks to just under 6 minutes to find a database. Seven of the 12 participants indicated they had prior library experience; however, they still encountered skill gaps. They lacked familiarity with the space, policies, website, and terminology. Participants also struggled with library jargon and inconsistent use of jargon among staff and librarians. Conclusion – The researchers discovered there were discrepancies between language used in signs, directions provided by staff, and information provided on the website. Signage was important because several participants made remarks on lack a familiarity with the library space. They would get lost and anxious. In addition, the video recordings and subsequent discussions among the staff and librarians showed issues arising from the power dynamics in the library organization. Staff felt pressured to provide reference services when librarians were unavailable due to staffing shortages, which led to miscommunication. These conclusions lead to empathy-based training to address language discrepancies and experiences among staff. It also provided additional rationale for hiring.

At the Crossroads of Translation and Cognition: Tracing Decision Making in Translation through Think-Aloud Protocol

At the Crossroads of Translation and Cognition: Tracing Decision Making in Translation through Think-Aloud Protocol

The Utilisation of Test-Taking Strategies by Saudi EFL High Schoolers in Al-Ahsa Region

It is highly recommended that educators know the importance of test-wiseness and test-unwiseness strategies which impact learners' performance on tests. This study aims to identify the common strategies learners adopt to achieve a high score and the most significant challenges that the learners face when taking the tests concerning the design of the tests and the clarity of instructions and questions. A group of six high school students has participated in the study. They have been given a vocabulary test with two types of questions (open and closed questions). The data has been collected by direct observation and a think-aloud protocol. The participants has utilized various test-taking strategies, which have helped them arrange ideas and gather information, such as asking about instructions, excluding choices, reading all questions before answering, and making a random choice of answers. Other students have utilized some TTS unwisely, such as skipping reading questions and instructions or reading them partially and choosing random answers without processing the whole question for meaning. The study helps to understand test-wise and test-unwiseness strategies and differentiate between conscious and unconscious strategies used by the participants. It has also suggested that educators should understand the impact of these strategies on their tests as this would help reduce the incidence of poorly framed questions and ambiguous instructions for learners. The study also raised the question of whether TTS should be taught to students or not. A topic that could further this investigation to reach wider insights and implications.

A novel intelligent agent-based framework for appropriate stream selection from perceptive of career counseling.

Picking a career stream profoundly influences people's abilities in different ways. Nowadays, choosing the correct career stream in advanced education is troublesome, as the number of people wanting to be in specific specializations is growing. Therefore, it is essential to be able to select the appropriate career path. This article proposes a system that can suggest streams in advanced education schools. This system is influenced by agent-based stream proposal systems (ASPS). The proposed system aims to make picking out the correct stream to study at school simpler for an individual. Different streams are evaluated based on seven pre-characterized models. In our system, three unique sorts of tests, learning styles, and coaching were coordinated in a way that caused the system to recognize an individual's interests and limits to an area of study. A sample of 238 participants was recruited for our questionnaire on accessibility, user-friendliness, accuracy, and satisfaction with the system. The incorporation of learning styles and coaching proved helpful in the study. The reliability and validity were proven in addition to incorporating a thinking-aloud protocol and immediate evaluation in the pre-, during and post-tests. To a large extent, respondents were satisfied with the model, as was revealed in the Likert scale response frequencies and percentages. The proposed system can be applied to undergraduates choosing engineering, medicine, arts and science degrees.

Cognitive behaviors in Indonesian-English translation: Exploring translation processes

This present study investigated the cognitive behaviors in the processes of Indonesian-English translation. Two research questions were employed to find out; 1) what types of cognitive behaviors emerge in the translation processes, 2) and which types of cognitive behavior are the most dominant in the translation process? The data were taken from screen recording of translator activities in video supplemented by think-aloud protocols in TransCon6_ID.mp4 entitled "Panorama Sumba." The data were analysed qualitatively by following the steps: transcribing, identifying, classifying, and interpreting (Bailey, 2008). The result reveals that the translator uses several cognitive behaviors such as generating ideas, revising, elaborating, clarifying, retrieving, rehearsing, referencing or resourcing, and summarizing. However, the translator's most dominant behavior is referencing by seeking the dictionaries to solve the translation problem.

Is viewing a painting like reading a story?: Trans-symbolic comprehension processes and aesthetic responses across two media

ABSTRACT A large body of research has outlined how mental models are formed by comprehending texts, yet relatively less work has been conducted in the field of comprehending artworks. Trans-symbolic comprehension (TSC) processes have been theorized to partially account for how mental models are formed across media. The current study tested whether participants use these processes similarly across reading stories and viewing paintings. The current study also tested whether the frequencies of TSC processes predict the aesthetic responses of understanding, interest, and pleasure. Participants typed out their thoughts while viewing paintings and reading texts. The think-aloud protocols were then parsed and coded for TSC processes. Results indicated similarities in TSC processes across both texts and paintings; however, the association between the TSC processes and aesthetic responses were greater for the texts than for the paintings. Overall, the results provide support for the TSC framework.

Development of a Digital Behavioral Intervention to Reduce the Consumption of Sugar-Sweetened Beverages Among Rural Appalachian Adults: Multiphased, Human-Centered Design Approach

To avoid the low engagement and limited efficacy of digital behavioral health interventions, robust human-centered design (HCD) processes are needed. The primary objective of this study was to describe a flexible, step-by-step HCD process to develop digital behavioral health interventions by illustrating iSIPsmarter as an example. iSIPsmarter is a digital intervention for reducing the consumption of sugar-sweetened beverages (SSBs) that comprises 6 internet-based cores metered out over time to deliver the program content, an integrated SMS text message strategy to engage users in reporting SSB behaviors, and an electronic cellular-enabled scale for in-home weighing. The secondary objective is to illustrate the key components and characteristics of iSIPsmarter that resulted from the HCD process. The methods were guided by the Model for Internet Interventions and by best practices in HCD and instructional design processes (eg, rapid prototype development and think-aloud protocol). The 3-phased (ie, contextual, prototype testing, end user testing phases) process followed in this study included a series of 13 semistructured one-on-one interviews with 7 advisory team participants from the targeted Appalachian user group. The interviews were content coded by 2 researchers and then deductively coded to the suggested areas of digital behavioral health interventions. The participants provided rich perspectives pertaining to iSIPsmarter's appearance, behavioral prescriptions, burdens, content, delivery, message, participation, and assessment. These inputs included requests for built-in flexibility to account for varying internet and SMS text message accessibility among users; ideas to resolve the issues and problems encountered when using the prototypes, including those related to navigation and comprehension of content; ideas to enhance personalized feedback to support motivation and goal setting for SSB consumption and weight; and feedback to refine the development of realistic and relatable vignettes. The participants were able to interact with multiple prototype drafts, allowing researchers to capture and incorporate feedback related to the iSIPsmarter dashboard, daily SSB and weight diaries, action planning, core content, interactions, and vignettes. Using scientific models and established processes is critical for building robust and efficacious interventions. By applying an existing model and HCD and instructional design processes, we were able to identify assumptions and address the key areas of the iSIPsmarter intervention that were hypothesized to support users' engagement and promote behavior change. As evidenced by the rich feedback received from the advisory team members and the resulting iSIPsmarter product, the HCD methodology was instrumental in the development process. Although the final iSIPsmarter content is specific to improving SSB consumption behaviors among adults in rural areas, the intent is that this HCD process will have wide applications in the development of digital behavioral health interventions across multiple geographic and behavioral contexts.

Clinical research staff perceptions on a natural language processing-driven tool for eligibility prescreening: An iterative usability assessment

BackgroundParticipant recruitment is a barrier to successful clinical research. One strategy to improve recruitment is to conduct eligibility prescreening, a resource-intensive process where clinical research staff manually reviews electronic health records data to identify potentially eligible patients. Criteria2Query (C2Q) was developed to address this problem by capitalizing on natural language processing to generate queries to identify eligible participants from clinical databases semi-autonomously. ObjectiveWe examined the clinical research staff’s perceived usability of C2Q for clinical research eligibility prescreening. MethodsTwenty clinical research staff evaluated the usability of C2Q using a cognitive walkthrough with a think-aloud protocol and a Post-Study System Usability Questionnaire. On-screen activity and audio were recorded and transcribed. After every-five evaluators completed an evaluation, usability problems were rated by informatics experts and prioritized for system refinement. There were four iterations of system refinement based on the evaluation feedback. Guided by the Organizational Framework for Intuitive Human-computer Interaction, we performed a directed deductive content analysis of the verbatim transcriptions. ResultsEvaluators aged from 24 to 46 years old (33.8; SD: 7.32) demonstrated high computer literacy (6.36; SD:0.17); female (75 %), White (35 %), and clinical research coordinators (45 %). C2Q demonstrated high usability during the final cycle (2.26 out of 7 [lower scores are better], SD: 0.74). The number of unique usability issues decreased after each refinement. Fourteen subthemes emerged from three themes: seeking user goals, performing well-learned tasks, and determining what to do next. ConclusionsThe cognitive walkthrough with a think-aloud protocol informed iterative system refinement and demonstrated the usability of C2Q by clinical research staff. Key recommendations for system development and implementation include improving system intuitiveness and overall user experience through comprehensive consideration of user needs and requirements for task completion.

A Proposal for Multi-scale Topographic Map Symbols for Roads and Buildings in Brazilian Urban Areas

ABSTRACT Topographic maps are widely used to provide geospatial information on digital environments, and the principles of cartographic communication positively impact its design. This paper proposes new road and building symbols in mapping Brazilian urban areas, aiming to increase the effectiveness of map-reading tasks (MRTs) in a multi-scale context. For this purpose, colour theory and semiotics were applied to develop new symbols and a remote test through an adapted think-aloud protocol. The results showed that users performed better with the proposed symbols than the previous ones.