Cognitive Walkthrough Research Articles

Prospective Validation and Usability Evaluation of a Mobile Diagnostic App for Obstructive Sleep Apnea

Background/Objectives: Obstructive sleep apnea (OSA) classification relies on polysomnography (PSG) results. Current guidelines recommend the development of clinical prediction algorithms in screening prior to PSG. A recent intuitive and user-friendly tool (OSABayes), based on a Bayesian network model using six clinical variables, has been proposed to quantify the probability of OSA. Our aims are (1) to validate OSABayes prospectively, (2) to build a smartphone app based on the proposed model, and (3) to evaluate app usability. Methods: We prospectively included adult patients suspected of OSA, without suspicion of other sleep disorders, who underwent level I or III diagnostic PSG. Apnea–hypopnea index (AHI) and OSABayes probabilities were obtained and compared using the area under the ROC curve (AUC [95%CI]) for OSA diagnosis (AHI ≥ 5/h) and higher severity levels (AHI ≥ 15/h) prediction. We built the OSABayes app on ‘App Inventor 2’, and the usability was assessed with a cognitive walkthrough method and a general evaluation. Results: 216 subjects were included in the validation cohort, performing PSG levels I (34%) and III (66%). OSABayes presented an AUC of 83.6% [77.3–90.0%] for OSA diagnosis and 76.3% [69.9–82.7%] for moderate/severe OSA prediction, showing good response for both types of PSG. The OSABayes smartphone application allows one to calculate the probability of having OSA and consult information about OSA and the tool. In the usability evaluation, 96% of the proposed tasks were carried out. Conclusions: These results show the good discrimination power of OSABayes and validate its applicability in identifying patients with a high pre-test probability of OSA. The tool is available as an online form and as a smartphone app, allowing a quick and accessible calculation of OSA probability.

Applying the Cognitive Walkthrough for Implementation Strategies methodology to inform the redesign of a selection-quality implementation toolkit for use in schools.

Implementation strategies are key to enhancing thetranslation of new innovations but there is a need to systematically design and tailor strategies to match the targeted implementation context and address determinants. There are increasing methods to inform the development and tailoring of implementation strategies to maximize their usability, feasibility, and appropriateness in new settings such as the Cognitive Walkthrough for Implementation Strategies (CWIS) approach. The aim of the current project is to apply the CWIS approach to inform the redesign of a multifaceted selection-quality implementation toolkit entitled Adoption of Curricular supports Toolkit: Systematic Measurement of Appropriateness and Readiness for Translation in Schools (ACT SMARTS) for use in middle and high schools. We systematically applied CWIS as the second part of a community-partnered iterative redesign of ACT SMARTS for schools to evaluate the usability and inform further toolkit redesign areas. We conducted three CWIS user testing sessions with key end users of school district administrators (n = 3), school principals (n = 6), and educators (n = 6). Our CWIS application revealed that end users found ACT SMARTS acceptable and relevant but anticipate usability issues engaging in the ACT SMARTS process. Results informed the identification of eleven usability issues and corresponding redesign solutions to enhance the usability of ACT SMARTS for use in middle and high schools. Results indicated the utility of CWIS in assessing implementation strategy usability in service of informing strategy modification as part of our broader redesign to improve alignment with end user, end recipient, and setting needs. Recommendations regarding the use of this participatory approach are discussed.

Pengembangan Antarmuka Pengguna (User Interface) Sistem Pengelolaan Beasiswa Berbasis Website dengan Pendekatan Cognitive Walkthrough

An effective user interface (UI) is crucial in a web-based scholarshipmanagement system to ensure a good user experience and easy accessibility toinformation. However, many users still encounter difficulties using the existingsystem, necessitating a better design. This study aims to evaluate and improve theuser interface of the scholarship management system using the CognitiveWalkthrough approach. The Cognitive Walkthrough approach is used to identifyusability issues in the user interface. A prototype of the design solution was thentested through usability testing involving seven evaluators and five different taskscenarios using Maze. The results show that all evaluators were able to completethe task scenarios with a category 1 rating (no issues), although four evaluators experienced minor issues in task scenarios 1 and 2. Evaluators were slightlyconfused when searching for the latest requirements information and the ineffectiveuse of buttons. By implementing the suggested improvements based on the findingsfrom the Cognitive Walkthrough, the user interface in the scholarship managementsystem can be significantly enhanced, facilitating users in accessing the requiredinformation.

User-Centered Design for Designing and Evaluating a Prototype of a Data Collection Tool to Submit Information About Incidents of Violence Against Sex Workers: Multiple Methods Approach.

Sex workers face an epidemic of violence in the United States. However, violence against sex workers in the United States is underreported. Sex workers hesitate to report it to the police because they are frequently punished themselves; therefore, an alternative for reporting is needed. We aim to apply human-centered design methods to create and evaluate the usability of the prototype interface for ReportVASW (violence against sex worker, VASW) and identify opportunities for improvement. This study explores ways to improve the prototype of ReportVASW, with particular attention to ways to improve the data collection tool. Evaluation methods included cognitive walkthrough, system usability scale, and heuristic evaluation. End users were enthusiastic about the idea of a website to document violence against sex workers. ReportVASW scored 90 on the system usability scale. The tool scored neutral on consistency, and all other responses were positive toward the app, with most being strong. Many opportunities to improve the interface were identified. Multiple methods identified multiple issues to address. Most changes are not overly complex, and the majority were aesthetic or minor. Further development of the ReportVASW data collection tool is worth pursuing.

Usability Evaluation of the Revised Color Me Healthy Symptom Assessment App: Perspectives of Children and Parents.

The Color Me Healthy symptom reporting app was co-designed with school-age children with cancer, their parents, and clinicians. Preliminary studies demonstrated its feasibility and acceptability; however, children and parents identified the need for additional refinements. Guided by the Technology Acceptance Model and principles of user-centered design, this study supported refinements to the Color Me Healthy user interface and evaluation of its usability. As the programming team completed builds of the app, school-age children with cancer and their parents participated in cognitive walkthrough usability evaluations and qualitative interviews. Usability logs documented the completion of key tasks related to reporting pain and review of child-reported data. Parents completed the Technology Acceptance Model Perceived Usefulness Scale (TAM-PUS). Interview responses were analyzed using qualitative content analysis. Fourteen children (median age 8.5 years; range 6-12) and 14 parents (median age 38.5 years; range 34-49) participated in one of three usability evaluation cycles. After the third cycle, children and parents navigated the app and completed key tasks independently. Median TAM-PUS scores were 6 (range 6-8), indicating high perceived usefulness. Qualitative analyses indicated that children regarded the app as easy and fun to use. Parents emphasized the app's developmental relevance for their child and for themselves as the child's caregiver. This study demonstrates the perceived usefulness and perceived ease of use of the revised Color Me Healthy app. Optimizing the usability of the Color Me Healthy app with attention to the user needs of children and parents positions the app for wider-scale clinical implementation.

Assessing gender bias in the software used in computer science and software engineering education

Women are underrepresented in Computer Science (CS)/ Software Engineering (SE) and other technology related degrees. As undergraduates, they are also less likely to persist with CS/SE studies than men enrolled in those same courses. Gender correlated differences in personal characteristics, behaviour, and preferences mean that course design decisions may introduce unintended bias. To address this issue, we drew inspiration from the GenderMag method. GenderMag uses personas with evidence-based gender differences in problem-solving traits to detect usability issues in software. In this paper we investigate the personal qualities of CS and SE students, and how these influence their CS/SE learning journey. A series of persona development workshops were held to gather an extensive and unique qualitative dataset capturing the prior experiences, preferences, learning styles, motivations, goals, frustrations, and constraints of CS/SE students. Gender differences were used to construct preliminary male and female student personas. These personas were used in cognitive walkthroughs of software applications commonly used in education, and their performance compared to GenderMag’s Tim and Abi. While the student personas were less effective and lacked specificity compared to Abi, they were able to identify issues not detectable with GenderMag. Furthermore, the findings show the utility of persona development workshops as a data collection method and introduce a comprehensive list of CS/SE student qualities that may inspire future investigations.

Applying cognitive walkthrough methodology to improve the usability of an equity-focused implementation strategy

BackgroundOur research team partnered with primary care and quality improvement staff in Federally Qualified Community Health Centers (CHCs) to develop Partnered and Equity Data-Driven Implementation (PEDDI) to promote equitable implementation of evidence-based interventions. The current study used a human-centered design methodology to evaluate the usability of PEDDI and generate redesign solutions to address usability issues in the context of a cancer screening intervention.MethodsWe applied the Cognitive Walkthrough for Implementation Strategies (CWIS), a pragmatic assessment method with steps that include group testing with end users to identify and prioritize usability problems. We conducted three facilitated 60-min CWIS sessions with end users (N = 7) from four CHCs that included scenarios and related tasks for implementing a colorectal cancer (CRC) screening intervention. Participants rated the likelihood of completing each task and identified usability issues and generated ideas for redesign solutions during audio-recorded CWIS sessions. Participants completed a pre-post survey of PEDDI usability. Our research team used consensus coding to synthesize usability problems and redesign solutions from transcribed CWIS sessions.ResultsUsability ratings (scale 0–100: higher scores indicating higher usability) of PEDDI averaged 66.3 (SD = 12.4) prior to the CWIS sessions. Scores averaged 77.8 (SD = 9.1) following the three CWIS sessions improving usability ratings from “marginal acceptability” to “acceptable”. Ten usability problems were identified across four PEDDI tasks, comprised of 2–3 types of usability problems per task. CWIS participants suggested redesign solutions that included making data fields for social determinants of health and key background variables for identifying health equity targets mandatory in the electronic health record and using asynchronous communication tools to elicit ideas from staff for adaptations.ConclusionsUsability ratings indicated PEDDI was in the acceptable range following CWIS sessions. Staff identified usability problems and redesign solutions that provide direction for future improvements in PEDDI. In addition, this study highlights opportunities to use the CWIS methodology to address inequities in the implementation of cancer screening and other clinical innovations in resource-constrained healthcare settings.

Special Issue on Informatics Education: < Integrating Diversity, Equity, Inclusion, and Accessibility into a Data Storytelling Model for Health Informatics Education >.

Health informatics education is pivotal in integrating diversity, equity, inclusion, and accessibility (DEIA) principles into curricula and leveraging data with equity considerations. Integrating clinically driven data with other datasets is crucial to comprehensive understanding of patient care demographics, experiences, and outcomes to create equity-minded data storytelling. Publicly available Healthy People 2030 (HP2030) resources complement academic EHRs, supporting tailored learning activities in informatics education to enhance educational utility through a DEIA lens. This case report describes the expansion of an existing DEI checklist to an updated DEIA checklist for preparing future informaticians to collect and critically evaluate DEIA features using this checklist in creating equity-minded data storytelling. An equity-minded data storytelling model and the HP2030 framework were utilized to develop the DEIA checklist. We employed an informal cognitive walkthrough to expand the DEIA checklist and evaluate the DEIA measures or characteristics within datasets from the HP2030 social determinants of health (SDOH) 5 topics using this checklist. We reviewed 76 available SDOH-related datasets and added 6 measures to "demographics" and 7 to "skills, abilities, & accessibility" of the DEIA checklist. Our evaluation of the DEIA checklist verified HP2030's inclusion of all measures, except "religions/beliefs." All DEIA measures were linked to equity and accessibility, 1 in inclusion, and the inclusion of 3 characteristics comprising the category "language" and 6 characteristics comprising the category "images." Results highlighted the accessibility and comprehensiveness of HP2030 demographic data resources, considering SDOH factors and promoting inclusive data representation to address health disparities. The DEIA checklist provides a structured tool in facilitating unbiased data collection and visualization of SDOH-related data in data storytelling through an equity-informed lens. Integrating an equity-minded data storytelling with frameworks like HP2030 enriches health informatics education, broadens students' understanding of health disparities, and supports evidence-based interventions for improved health outcomes.

Face and content validity of a holistic assessment questionnaire to assess cancer-related fatigue after breast cancer

ABSTRACT Background and objective Cancer-related fatigue (CRF) affects the quality of life after breast cancer. In a previous study, we developed a 72-item questionnaire that assesses CRF from a holistic point of view; named the Holistic Assessment of CRF (HA-CRF) questionnaire. The current study assessed the face and content validity of the HA-CRF questionnaire. Methods Using a mixed-method approach, ten breast cancer survivors (BCS) did a cognitive walkthrough of the HA-CRF via an app followed by a semi-structured interview about relevancy and essentiality (qualitative). In addition, ten health care professionals (HCPs) assessed the relevancy, clarity, and essentiality of each item via a questionnaire (quantitative). Results BCS indicated minor textual improvement for four items and six items were not completely clear. The app was considered easy to use and the HC-CRF was on average completed in 18 minutes. The HA-CRF questionnaire provided openness about fatigue and gave the feeling of being heard. The items were helpful and induced self-awareness. HCPs indicated 71% of items being very clear or minor revisions proposed by the minority, with 64% of items being essential and 92% considered relevant. Conclusions The HA-CRF showed good face and excellent content validity. Further research is needed to assess its ability to monitor in daily life.

Integration of Equity, Diversity, Inclusivity, and Indigeneity Principles into Usability Heuristic Evaluations.

Cognitive walkthrough is a form of usability testing that considers the perspective of the end users to identify issues related to user experience and web design. This project aims to enhance traditional heuristic evaluation methods with consideration of equity, diversity, inclusivity, and indigeneity (EDI-I) principles. The authors provide suggestions that align with modern informatics advancements, aiming for inclusive design systems and the elimination of systemic barriers.

Applying the Cognitive Walkthrough for Implementation Strategies Methodology to Inform the Redesign of a Selection-Quality Implementation Toolkit for Use in Schools.

Implementation strategies are key to enhancing translation of new innovations but there is a need to systematically design and tailor strategies to match the targeted implementation context and address determinants. There are increasing methods to inform the redesign and tailoring of implementation strategies to maximize their usability, feasibility, and appropriateness in new settings such as the Cognitive Walkthrough for Implementation Strategies (CWIS) approach. The aim of the current project is to apply the CWIS approach to inform the redesign of a multifaceted selection-quality implementation toolkit entitled ACT SMARTS for use in middle and high schools. We systematically applied CWIS as the second part of a community-partnered iterative redesign of ACT SMARTS for schools to evaluate the usability and inform further toolkit redesign areas. We conducted three CWIS user testing sessions with key end users of school district administrators, school principals, and educators. Our CWIS application revealed that end users found ACT SMARTS acceptable and relevant but anticipate usability issues engaging in the ACT SMARTS process. Results informed the identification of eleven usability issues and corresponding redesign solutions to enhance the usability of ACT SMARTS for use in middle and high schools. Results indicated the utility of CWIS in assessing implementation strategy usability in service of informing strategy tailoring and redesign to improve alignment with user and setting needs. Recommendations regarding the use of this participatory approach are discussed.

Design and Evaluation of UI/UX of Siwi Village in West Papua Province Using Cognitive Walkthrough

Design and Evaluation of UI/UX of Siwi Village in West Papua Province Using Cognitive Walkthrough

Integrating Heuristic Evaluation and Cognitive Walkthrough in Usability Evaluation of Mobile Application

This study aims to present the usability evaluation of mobile applications. The growth of mobile applications has been spread to any type of digital activities, especially for paying bills or ordering things. Unfortunately, the lack of interaction design of the application makes the application not easy to use and learn. This study proposed a combination of two usability evaluation methods which are heuristic evaluation and cognitive walkthrough. Myindihome is an application that will be studied to present this evaluation. Those selected methods is employed to evaluate based on the 10’s Nielsen heuristic principle and the Cognitive of the evaluator during the group determination. The heuristic evaluation and cognitive walkthrough are evaluated by 3 experts or evaluators. The result of the heuristic evaluation revealed 3 major issues and 6 minor issues. Whereas the cognitive walkthrough determination revealed that 1 critical main menu needs to be re-designed. Thus, the interaction design of the application in some parts is not easy to learn and not efficient. It is expected this study can be adopted by mobile developers to produce an ease-of-learn and efficient application.

Empathetic and Emotive Design Heuristics: Preliminary Results of Their Application to Evaluating Survey User Interfaces.

Empathetic and emotive design is becoming increasingly important in the digital age. In this research we describe the results of a combined cognitive walkthrough and heuristic evaluation using newly developed, empirically derived empathy or emotive design heuristics. We applied the heuristics to the evaluation of four commonly used survey platforms. Our preliminary findings revealed that the heuristics performed effectively in scoring survey platforms on their level of empathy. Survey platforms that are highly empathetic were scored highest.

User-centric avatar design: A cognitive walkthrough approach for metaverse in virtual education

Metaverse, once a concept confined to science fiction, has emerged as a transformative reality during the digital era. As this immersive virtual world gains prominence, the role of avatars and the digital representations of users in shaping educational experiences within a metaverse becomes increasingly crucial. This study examined the design of avatars and their effectiveness in virtual education. Using the cognitive walkthrough (CW) method, a well-established user-centric assessment approach, this study examined how users engage in avatar design in virtual educational settings, highlighting their cognitive processes. Through the CW approach, this study identified usability issues and learning challenges, providing valuable insights into the user experience. The investigation focuses on the VRoid platform for avatar design, selected for its user-friendly interface and robust customization features. Participants (n ​= ​12; six men and six women) from diverse educational backgrounds were carefully selected to offer a comprehensive range of perspectives. They performed avatar design tasks that closely mirrored the learning environment of the metaverse, followed by in-depth post-exercise interviews. The findings of this study highlight the essential role of avatar design in enhancing user engagement and improving the learning outcomes of virtual education. Furthermore, we provide practical recommendations for educators and designers to effectively leverage avatar innovation. In a continuously evolving era of metaverse, this research contributes significantly to the ongoing discourse on user-centered design in educational settings, thereby influencing the future trajectory of virtual education.

A Step Toward Better Care: Understanding What Caregivers and Residents in Assisted Living Facilities Value in Health Monitoring Systems

The past several decades have seen significant advances in monitoring older adults' health and well-being. However, creating viable, practical monitoring systems for informing caregivers requires understanding which behaviors and signs to track and what approaches best present that information. To investigate how technology can be leveraged to better augment caregivers' workflows, we take a multi-stage, qualitative approach to gain insights into the needs of caregivers and the older adults receiving care. Specifically, we use a series of domain expert interviews, cognitive walkthroughs, and semi-structured interviews with residents, and we synthesize our takeaways using thematic analysis at each phase. Our results show that this type of monitoring technology has great potential to reduce the effort needed by caregivers to complete their responsibilities and communicate with their teams. Additionally, we found that older adults are receptive to the technology but their privacy and autonomy must be prioritized for the sake of their mental wellbeing. These insights will facilitate greater intelligent interface development for Person-Centered Care by identifying important design considerations and vital features that require system support.

BRIDGING THEORY AND PRACTICE: COMPARATIVE INSIGHTS INTO EVALUATING USER INTERFACES

Abstract Keywords Cite This Article as User interface (UI) evaluation is an important aspect of software development, ensuring that software or systems are efficient and user satisfactory. Among the various usability testing methods available, Cognitive Walkthrough Method, Heuristic Evaluation Method and Concurrent Usability Testing Method stand out as most popular approaches. This research paper presents a comparative analysis of these testing methods, emphasizing their strengths, flaws, and applicability in different situations. Through a detailed analysis of each testing method's procedural steps, and practical concerns, we aim to provide insights into their effectiveness and suitability for evaluating User Interface (UI) designs. Additionally, we discuss potential synergies and inadequacies when combining these testing methods in practice. Our results and findings contribute to a deeper understanding of UI evaluation techniques and provide guidance for selecting the most appropriate approach for testing team’s specific needs.

Centering School Leaders’ Expertise: Usability Evaluation of a Leadership-Focused Implementation Strategy to Support Tier 1 Programs in Schools

Although there are a growing number of implementation strategies to increase the adoption, fidelity, and sustainment of evidence-based practices (EBP), they are often complex and bulky, which can interfere with their widespread application. To address these challenges, the Helping Educational Leaders Mobilize Evidence (HELM) strategy was created as an adaptation of the evidence-based Leadership and Organizational Change for Implementation (LOCI) implementation strategy to enhance elementary school principals’ use of strategic implementation leadership to support the adoption and delivery of Tier 1 (i.e., universal social, emotional, and behavioral) EBP. In service of its iterative development, a human-centered design methodology was employed to increase the successful uptake and use of HELM. The Cognitive Walkthrough for Implementation Strategies (CWIS), a novel mixed-methods approach to evaluate implementation strategy usability, was applied to identify and test HELM strategy tasks of critical importance. A sample of 15 elementary school principals participated in group cognitive walkthrough testing as either school principal recipients or HELM coaches. Both user types rated the strategy as acceptable (principal M = 77.8, SD = 15.5; coach M = 87.5, SD = 7.9). Five usability issues were identified using a highly structured common usability issue framework and provided direction for the generation of redesign solutions to be incorporated in a subsequent version of the HELM strategy. The evaluation of the strategy’s usability improved its alignment with user needs, expectations, and contextual constraints, rendering a more usable strategy and broadly applicable information surrounding the development of psychosocial implementation strategies in real-world settings.

Towards a Patient-Centered Design of a Cancer Telerehabilitation System.

Successful implementation of telehealth platforms requires a detailed understanding of patient's needs, preferences, and attitudes toward a home-based platform. The goal of this study was to identify patient-centered characteristics of a cancer rehabilitation system based on cognitive evaluation of user interface and semi-structured qualitative interviews. Quantitative and qualitative feedback from 29 patients with metastatic urogenital cancer was collected after using a cancer telerehabilitation system. Heuristic evaluation, cognitive walkthrough, and analysis of qualitative interviews demonstrated a high level of support for the concept of home-based cancer telerehabilitation by cancer patients. Post-task surveys demonstrated sufficient usability and satisfaction scores from the participants. The patients provided valuable and insightful comments on how to further improve the functionality and interface of the platform. Further improvement of the system usability, consistency, and accessibility based on the patient-centered design principles will significantly facilitate the implementation of cancer telerehabilitation in clinical practice.

Development of Ward Visitor Management System in the COVID-19 Epidemic.

The purpose of this study was to design, develop, and deploy a visitor management system (VMS) to effectively manage visitors during COVID-19. The VMS was designed using the User-Centered Design (UCD) methodology. The iterative process of UCD includes 3 interviews and 5 usability tests and cognitive walkthrough cycles. This system comprised six parts: the WEB server provides visit scheduling service; the database server stores visit data and provides visit data services; the mobile application server provides security checks and scanning services; the electronic medical record (EMR) server provides ward data service; the Internet application gateway provides health code data service and exchanges health code data with the Sichuan Tianfu Health Code platform, and the service bus enables the centralized exchange and integration of visit data. The visit management system optimizes the workflow of ward visitors, improves staff productivity, and reduces the risk of infection transmission.