User Testing Interviews Research Articles

Continuous adaptation of conversation aids for uterine fibroids treatment options in a four-year multi-center implementation project

BackgroundFibroids are non-cancerous uterine growths that can cause symptoms impacting quality of life. The breadth of treatment options allows for patient-centered preference. While conversation aids are known to facilitate shared decision making, the implementation of these aids for uterine fibroids treatments is limited. We aimed to develop two end-user-acceptable uterine fibroids conversation aids for an implementation project. Our second aim was to outline the adaptations that were made to the conversation aids as implementation occurred.MethodsWe used a multi-phase user-centered participatory approach to develop a text-based and picture-enhanced conversation aid for uterine fibroids. We conducted a focus group with project stakeholders and user-testing interviews with eligible individuals with symptomatic uterine fibroids. We analyzed the results of the user-testing interviews using Morville’s Honeycomb framework. Spanish translations of the conversation aids occurred in parallel with the English iterations. We documented the continuous adaptations of the conversation aids that occurred during the project using an expanded framework for reporting adaptations and modifications to evidence-based interventions (FRAME).ResultsThe first iteration of the conversation aids was developed in December 2018. Focus group participants (n = 6) appreciated the brevity of the tools and suggested changes to the bar graphs and illustrations used in the picture-enhanced version. User-testing with interview participants (n = 9) found that both conversation aids were satisfactory, with minor changes suggested. However, during implementation, significant changes were suggested by patients, other stakeholders, and participating clinicians when they reviewed the content. The most significant changes required the addition or deletion of information about treatment options as newer research was published or as novel interventions were introduced into clinical practice.ConclusionsThis multi-year project revealed the necessity of continuously adapting the uterine fibroids conversation aids so they remain acceptable in an implementation and sustainability context. Therefore, it is important to seek regular user feedback and plan for the need to undertake updates and revisions to conversation aids if they are going to be acceptable for clinical use.

Engaging communities to inform the development of a diverse cohort of cancer survivors: formative research for the eat move sleep study (EMOVES)

BackgroundThere are more than 18 million cancer survivors in the United States. Yet, survivors of color remain under-represented in cancer survivorship research (Saltzman et al. in Contemp Clin Trials Commun 29:100986, 2022; Pang et al. in J Clin Oncol 34:3992–3999, 2016; Lythgoe et al. in Prostate Cancer Prostatic Dis 24:1208–1211, 2021). Our long-term goal is to enroll and follow a cohort of historically under-represented cancer survivors, to better understand modifiable risk factors that influence clinical and quality of life outcomes in these populations. Towards that goal, we describe herein how we applied community-based participatory research approaches to develop inclusive study materials for enrolling such a cohort.MethodsWe implemented community engagement strategies to inform and enhance the study website and recruitment materials for this cohort including: hiring a dedicated engagement coordinator/community health educator as a member of our team; working with the Helen Diller Family Comprehensive Cancer Center Office of Community Engagement (OCE) and Community Advisory Board members; presenting our educational, research, and study recruitment materials at community events; and establishing a community advisory group specifically for the study (4 individuals). In parallel with these efforts, 20 semi-structured user testing interviews were conducted with diverse cancer survivors to inform the look, feel, and usability of the study website.ResultsEngagement with community members was a powerful and important approach for this study’s development. Feedback was solicited and used to inform decisions regarding the study name (eat move sleep, EMOVES), logo, study website content and imagery, and recruitment materials. Based on community feedback, we developed additional educational materials on healthy groceries and portion size in multiple languages and created a study video.ConclusionsIncluding an engagement coordinator as a permanent team member, partnering with the institutional community outreach and engagement resources (i.e., OCE), and allocating dedicated time and financial support for cultivating relationships with stakeholders outside the university were critical to the development of the study website and materials. Our community guided strategies will be tested as we conduct enrollment through community advisor networks and via the state cancer registry.

Misunderstood terms and concepts identified through user testing of educational materials for fragile X premutation: "Not weak or fragile?"

Complicated genetic mechanisms and unpredictable health risks associated with the FMR1 premutation can result in challenges for patient education when the diagnosis is made in a newborn. From October 15, 2018, to December 10, 2021, North Carolina parents could obtain FMR1 premutation results about their newborns through a voluntary expanded newborn screening research study. The study provided confirmatory testing, parental testing, and genetic counseling. We developed web-based educational materials to augment information about fragile X premutation conveyed by a genetic counselor. Many genetics education materials are developed for the lay population. However, relatively little research is published on how well individuals understand these materials. We conducted three rounds of iterative user testing interviews to help refine web-based educational materials that support understanding and self-paced learning. The participants included 25 parents with a 2-year college degree or less and without a child identified with fragile X syndrome, premutation, or gray-zone allele. Content analysis of interview transcripts resulted in iterative changes and ultimately saturation of findings. Across all rounds of interviews, there were two terms that were commonly misunderstood (fragile and carrier) and two terms that elicited initial misconceptions that were overcome by participants. Many also had difficulty understanding the relationship between fragile X premutation and fragile X syndrome as well as appreciating the implications of having a "fragile X gene." Website layout, formatting, and graphics also influenced comprehension. Despite iterative changes to the content, certain issues with understandability persisted. The findings support the need for user testing to identify misconceptions that may interfere with understanding and using genetic information. Here, we describe a process used to develop and refine evidence-based, understandable parental resources on fragile X premutation. Additionally, we provide recommendations to address ongoing educational challenges and discuss the potential impact of bias on the part of expert content developers.

User testing of a Scottish Intercollegiate Guideline Network public guideline for the parents of children with autism

BackgroundThe Scottish Intercollegiate Guidelines Network (SIGN) is the leading national clinical guideline producer in Scotland. Improved design and dissemination of guidelines produced for the public can empower people to take an active role in self-management and shared decision-making. The public version of the guideline examined covered getting assessed and diagnosed with autism, and approaches that can help. The aim of this study was to test a public version of a guideline for the parents of children and young people with autism, implement improvements, and identify what works in making it usable and accessible.MethodsWe recruited mothers from across Scotland. User testing involved formal ‘think aloud’ semi-structured interviews that guided users through the booklet. Interviews took place individually and were recorded and transcribed. Key findings were identified and themed using the honeycomb user experience model.ResultsFourteen user-testing interviews were conducted. Facilitators for usability and desirability of the guideline included the chunking of text, consistent use of colour and boxes to highlight important information. Simple language, written in a tone of partnership, helped to engage mothers. Value arose from the guidelines ability to explain the process of diagnosis and make mothers feel empowered in their relationships with healthcare professionals. There was a lack of consensus on the usefulness of rating the strength of evidence and recommendations.ConclusionThere was a marked similarity between what was important to the mothers and what has been found to be important to other groups. The involvement of service users and carers in the guidelines development was key to its credibility. One size does not fit all in presenting evidence-based recommendations to the public and it is a challenge to provide sufficient information while avoiding information overload. Recommendations and evidence levels are suitable for use in public versions, but these should be kept as simple as possible.

Development of a Health Information Technology Tool for Behavior Change to Address Obesity and Prevent Chronic Disease Among Adolescents: Designing for Dissemination and Sustainment Using the ORBIT Model.

Health information technology (HIT) has not been broadly adopted for use in outpatient healthcare settings to effectively address obesity in youth, especially among disadvantaged populations that face greater barriers to good health. A well-designed HIT tool can deliver behavior change recommendations and provide community resources to address this gap, and the Obesity-Related Behavioral Intervention Trials (ORBIT) model can guide its development and refinement. This article reports the application of the ORBIT model to (1) describe the characteristics and design of a novel HIT tool (the PREVENT tool) using behavioral theory, (2) illustrate the use of stakeholder-centered “designing for dissemination and sustainability” principles, and (3) discuss the practical implications and directions for future research. Two types of stakeholder engagement (customer discovery and user testing) were conducted with end users (outpatient healthcare teams). Customer discovery interviews (n = 20) informed PREVENT tool components and intervention targets by identifying (1) what healthcare teams (e.g., physicians, dietitians) identified as their most important “jobs to be done” in helping adolescents who are overweight/obese adopt healthy behaviors, (2) their most critical “pains” and “gains” related to overweight/obesity treatment, and (3) how they define success compared to competing alternatives. Interviews revealed the need for a tool to help healthcare teams efficiently deliver tailored, evidence-based behavior change recommendations, motivate patients, and follow-up with patients within the constraints of clinic schedules and workflows. The PREVENT tool was developed to meet these needs. It facilitates prevention discussions, delivers tailored, evidence-based recommendations for physical activity and food intake, includes an interactive map of community resources to support behavior change, and automates patient follow-up. Based on Self-Determination Theory, the PREVENT tool engages the patient to encourage competence and autonomy to motivate behavior change. The use of this intentional, user-centered design process should increase the likelihood of the intended outcomes (e.g., behavior change, weight stabilization/loss) and ultimately increase uptake, implementation success, and long-term results. After initial tool development, user-testing interviews (n = 13) were conducted using a think-aloud protocol that provided insight into users' (i.e., healthcare teams) cognitive processes, attitudes, and challenges when using the tool. Overall, the PREVENT tool was perceived to be useful, well-organized, and visually appealing.

User testing a patient information resource about potential complications of vaginally inserted synthetic mesh

BackgroundVaginal mesh implants are medical devices used in a number of operations to treat stress urinary incontinence and pelvic organ prolapse. Although many of these operations have delivered good outcomes, some women have experienced serious complications that have profoundly affected their quality of life. To ensure that evolving patient information is up-to-date, accurate and appropriate, the Transvaginal Mesh Oversight Group ‘user-tested’ a newly developed Scottish patient resource, the first to focus exclusively on the issue of complications. The aim of this research was to gather feedback on usability, content, language and presentation to inform the development of the resource from a user perspective.MethodsThe experience of using the patient resource was captured through semi-structured interviews that followed a ‘think-aloud’ protocol. The interviewer observed each participant as they went through the resource, asking questions and making field notes. Participants’ comments were then categorised using a validated model of user experience and subsequently analysed thematically.ResultsThirteen people participated in the user testing interviews, including women with lived experience of mesh implants (n = 7), a convenience sample of staff working for Healthcare Improvement Scotland (n = 5) and a patient’s carer (n = 1). The majority of participants considered the resource as clear and helpful. Respondents reported that some presentational aspects promoted usability and understandability, including the use of a font that is easy to read, bullet lists, coloured headings and simple language. Barriers included the reliance on some technical language and an explicit anatomical diagram. Participants endorsed the valuable role of health professionals as co-mediators of patient information.ConclusionsThe findings illustrate the value of undertaking in-depth user-testing for patient information resources before their dissemination. The study highlighted how the direct guidance or navigation of a patient information resource by a health professional could increase its salience and accuracy of interpretation by patients, their families and carers. These insights may also be useful to other developers in improving patient information.

User-Centered Development and Patient Acceptability Testing of a Health-Coaching Intervention to Enhance Cancer Survivorship Follow-up in Primary Care.

We describe an iterative three-phase approach used to develop a cancer survivorship health-coaching intervention to guide self-management and follow-up care for post-treatment breast, colorectal, and prostate cancer survivors. Informed by theory (e.g., Cognitive-Social Health Information Processing Model (C-SHIP)), relevant literature, and clinical guidelines, we engaged in a user-centered design process. In phase I, we conducted depth interviews with survivors of breast (n = 34), prostate (n = 4), and colorectal (n = 6) cancers to develop a health coaching prototype. In phase II, we utilized user-testing interviews (n = 9) to test and refine the health coaching prototype. For both phases, we used a template analysis independently coding each interview. In phase I, majority (n = 34, 81%) of survivors were positive about the utility of health coaching. Among these survivors (n = 34), the top areas of identified need were emotional support (44%), general health information (35.3%), changes in diet and exercise (29.3%), accountability and motivation (23.5%), and information about treatment effects (17.7%). The prototype was developed and user-tested and refined in phase III to address the following concerns: (1) the amount of time for calls, (2) density of reading materials, (3) clarity about health coaches' role, (4) customization. Collectively, this resulted in the development of the Extended Cancer Educational for Long-Term Cancer Survivors health-coaching (EXCELSHC) program, which represents the first cancer survivorship follow-up program to support follow-up care designed-for-dissemination in primary care settings. EXCELSHC is being tested in a clinical efficacy trial. Future research will focus on program refinement and testing for effectiveness in primary care.

Development of the WHO Antenatal Care Recommendations Adaptation Toolkit: a standardised approach for countries

BackgroundIncreasingly, WHO recommendations are defined by context-specific factors and WHO is developing strategies to ensure that recommendations are successfully adapted and implemented at country level. This manuscript describes the development of a toolkit to support governments to adapt the WHO recommendations on antenatal care (ANC) for a positive pregnancy experience for their context in a systematic manner.MethodsThe toolkit was developed in three steps. It was created with input from methodologists and regional implementation experts (Step 1) followed by a user-testing phase (Step 2), implemented during country stakeholder meetings. User testing consisted of stakeholder interviews that were transcribed, and data was categorised according to the content analysis method. Suggestions for toolkit improvement and issues identified during the interviews were assessed as serious, moderately serious or minor/cosmetic.ResultsA total of 22 stakeholders – comprising five Ministry of Health (MoH) consultants, four MoH policy-makers, and 13 advisors/implementers – from Burkina Faso, India, Rwanda and Zambia participated in user-testing interviews during stakeholder meetings held in each country between August 2018 and February 2019. Most stakeholders had a medical or nursing background and half were women. Overall, responses to the toolkit were positive, with all stakeholders finding it useful and desirable. User testing interviews highlighted four serious, four moderately serious and five minor/cosmetic issues to be managed. These were addressed in the final step (Step 3), an updated version of the WHO ANC Recommendations Adaptation Toolkit, comprised of two main components – a baseline assessment tool with spreadsheets for data entry and a Slidedoc®, a dual-purpose document for reading and presentation, outlining the qualitative data that shaped the women-centred perspective of the guidelines, accompanied by an instruction manual detailing the components’ use.ConclusionsThe WHO ANC Recommendations Adaptation Toolkit was developed to support countries to systematically adapt the WHO ANC recommendations for country contexts. Using this approach, similar tools can be developed to support guideline implementation across different health domains and the continuum of care.

Testing Usability and Feasibility of a Mobile Educator Tool for Pediatric Diabetes Self-Management: Mixed Methods Pilot Study.

BackgroundMobile interventions hold promise as an intervention modality to engage children in improving diabetes self-management education, attitudes, and behaviors.ObjectiveThis pilot study aimed to explore the usability, acceptability, and feasibility of delivering a mobile diabetes educational tool to parent-child pairs in a clinical setting.MethodsThis mixed methods pilot study comprised two concurrent phases with differing study participants. Phase 1 used user testing interviews to collect qualitative data on the usability and acceptability of the tool. Phase 2 used a single-arm pre- and poststudy design to quantitatively evaluate the feasibility and preliminary efficacy of the intervention. Study participants (English-speaking families with youth aged 5-14 years with insulin-dependent diabetes) were recruited from an urban hospital in Massachusetts, United States. In phase 1, parent-child pairs were invited to complete the intervention together and participate in 90-min user testing interviews assessing the tool’s usability and acceptability. Interview transcripts were analyzed using a directed content analysis approach. In phase 2, parent-child pairs were invited to complete the intervention together in the clinical setting. Measures included parental and child knowledge, attitudes, and behaviors related to diabetes management (self-report surveys) and child hemoglobin A1c levels (medical record extractions); data were collected at baseline and 1-month follow-up. Pre- and postoutcomes were compared using paired t tests and the Fisher exact test.ResultsA total of 11 parent-child pairs (N=22) participated in phase 1 of the study, and 10 parent-child pairs (N=20) participated in phase 2 of the study. Participants viewed the mobile educational tool as acceptable (high engagement and satisfaction with the layout, activities, and videos) and identified the areas of improvement for tool usability (duration, directions, and animation).ConclusionsThe findings from this pilot study suggest that the mobile educational tool is an informative, engaging, and feasible way to deliver diabetes self-management education to parents and children in an urban hospital setting. Data will inform future iterations of this mobile diabetes educational intervention to improve usability and test intervention efficacy.

Development and usability testing of the e-EXCELS tool to guide cancer survivorship follow-up care.

The aim of this study is to describe a user-centered (e.g., cancer survivors and clinicians) development process of an eHealth tool designed to facilitate self-management of cancer survivorship follow-up care. Guided by Cognitive-Social Health Information Processing (C-SHIP) model and informed by core self-management skills, we engaged in a user-centered design process. In phase I, we conducted in-depth interviews with survivors of breast (n = 33), prostate (n = 4), and colorectal (n = 6) cancers, and (n = 9) primary care providers to develop content and design of the web tool. Phase II utilized iterative user testing interviews (n = 9) to test the web-based tool prototype. Data from both phases were independently coded using a template/content analytic approach. The top 5 functions identified in phase I for the web-based platform included: (a) educational materials to learn and prepare for health encounters (80%); (b) questions for health providers (74%); (c) ability to track contact information of providers (67%); (d) provide general information (64%); and, (e) support information (62%). Users of the prototype reported patient burden, tool fatigue, introduction timing of the tool, relevance, and security/privacy as concerns in phase II. This study demonstrates the value of using a theoretically informed and user-centered design process to develop relevant and patient-centered eHealth resources to support cancer survivorship. A larger study is needed to establish the efficacy of this eHealth tool as an intervention to improve adherence to follow-up care guidelines.

Decision-Making about Healthcare Related Tests and Diagnostic Strategies: User Testing of GRADE Evidence Tables.

ObjectiveTo develop guidance on what information to include and how to present it in tables summarizing the evidence from systematic reviews of test accuracy following the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach.MethodsTo design and refine the evidence tables, we used an iterative process based on the analysis of data from four rounds of discussions, feedback and user testing. During the final round, we conducted one-on-one user testing with target end users. We presented a number of alternative formats of evidence tables to participants and obtained information about users’ understanding and preferences.ResultsMore than 150 users participated in initial discussions and provided their formal and informal feedback. 20 users completed one-on-one user testing interviews. Almost all participants preferred summarizing the results of systematic reviews of test accuracy in tabular format rather than plain text. Users generally preferred less complex tables but found presenting sensitivity and specificity estimates only as too simplistic. Users found the presentation of test accuracy for several values of prevalence initially confusing but modifying table layout and adding sample clinical scenarios for each prevalence reduced this confusion. Providing information about clinical consequences of testing result was viewed as not feasible for authors of systematic reviews.ConclusionWe present the current formats for tables presenting test accuracy following the GRADE approach. These tables can be developed using GRADEpro guidelines development tool (www.guidelinedevelopment.org or www.gradepro.org) and are being further developed into electronic interactive tables that will suit the needs of different end users. The formatting of these tables, and how they influence result interpretation and decision-making will be further evaluated in a randomized trial.

076 Enhancing the Acceptance and Implementation of GRADE Summary Tables for Evidence about Diagnostic Tests

BackgroundThe GRADE Working Group developed Summary tables adapted to summarise and present evidence from diagnostic test accuracy (DTA) systematic reviews.ObjectiveTo develop guidance on what information to include in these summary...

Can we improve recruitment to trials and informed consent by improving participant information sheets? - A nested RCT

Methods CASPER is a UK NIHR-funded trial of Collaborative Care to prevent depression in adults aged 75+. During the CASPER set-up phase we produced an information sheet compliant with National Research Ethics Service (NRES) guidance (the ‘standard sheet’). We decided to try to enhance recruitment by improving the PIS (the ‘enhanced sheet’). The nested study comprises: first, development and testing of an ‘enhanced sheet’; second, comparison of the ‘standard’ and ‘enhanced’ versions in a nested RCT. The comprehensibility of the NRES-approved standard sheet was assessed by asking cohorts of 10 older adults to take part in individual user testing interviews. After reading the sheet, participants were asked to find answers in the sheet to 18 factual questions and show their understanding of found information. The sheet was then re-written, re-structured and re-designed, drawing on graphics and writing expertise and the user testing data. The ‘enhanced sheet’ was user-tested on further cohorts of 10 people and amended according to the data obtained. In a nested RCT, potential CASPER participants will be posted either the ‘standard’ or ‘enhanced’ sheet, to assess effects on interest in participation and recruitment. Responders will be asked to complete an abbreviated version of the Joffe Quality of Informed Consent measure [1]. We will also conduct 2 focus groups with participants to explore the role of the sheet in their decision to participate (or not).