LLM-powered assistant with electrotactile feedback to assist blind and low vision people with maps and routes preview

Tactile graphics are widely used to present maps and statistical diagrams to blind and low vision (BLV) people, with accessibility guidelines recommending their use for graphics where spatial relationships are important. Their use is expected to grow with the advent of commodity refreshable tactile displays. However, in stark contrast to visual information graphics, we lack a clear understanding of the benefts that well-designed tactile information graphics offer over text descriptions for BLV people. To address this gap, we introduce a framework considering the three components of encoding, perception and cognition to examine the known benefts for visual information graphics and explore their applicability to tactile information graphics. This work establishes a preliminary theoretical foundation for the tactile-frst design of information graphics and identifes future research avenues.

Advancements in accessibility technologies such as low-cost swell form printers or refreshable tactile displays promise to allow blind or low-vision (BLV) people to analyze data by transforming visual representations directly to tactile representations. However, it is possible that design guidelines derived from experiments on the visual perception system may not be suited for the tactile perception system. We investigate the potential mismatch between familiar visual encodings and tactile perception in an exploratory study into the strategies employed by BLV people to measure common graphical primitives converted to tactile representations. First, we replicate the Cleveland and McGill study on graphical perception using swell form printing with eleven BLV subjects. Then, we present results from a group interview in which we describe the strategies used by our subjects to read four common chart types. While our results suggest that familiar encodings based on visual perception studies can be useful in tactile graphics, our subjects also expressed a desire to use encodings designed explicitly for BLV people. Based on this study, we identify gaps between the perceptual expectations of common charts and the perceptual tools available in tactile perception. Then, we present a set of guidelines for the design of tactile graphics that accounts for these gaps. Supplemental material is available at https://osf.io/3nsfp/?view_only=7b7b8dcbae1d4c9a8bb4325053d13d9f.

Screen readers are audio-based software that Blind and Low Vision (BLV) people use to interact with computing devices, such as tablets and smartphones. Although this technology has significantly improved the accessibility of touchscreen devices, the sequential nature of audio limits the bandwidth of information users can receive and process. We introduce TapNav, an adaptive spatiotactile screen reader prototype developed to interact with touchscreen interfaces spatially. TapNav's screen reader provides adaptive auditory feedback that, in combination with a tactile overlay, conveys spatial information and location of interface elements on-screen. We evaluated TapNav with 12 BLV users who interacted with TapNav to explore a data visualization and interact with a bank transactions application. Our qualitative findings show that touch points and spatially constrained navigation helped users anticipate outcomes for faster exploration, and offload cognitive load to touch. We provide design guidelines for creating tactile overlays for adaptive spatiotactile screen readers and discuss their generalizability beyond our exploratory data analysis and everyday application navigation scenarios.

Augmented reality (AR) applications have been shown to improve accessibility for people with low vision by enhancing the visibility of surrounding objects. Yet, prior studies mostly examined controlled settings and often focused on developing solutions for specific functional challenges. A human remote assistant powered with AR capabilities may provide a flexible solution that addresses a variety of scenarios. To examine how AR-based remote assistance can help low-vision people in real-world settings, we examined the scenario of visiting a museum, which requires coping with a variety of tasks, from navigating between museum rooms to the accessibility of museum exhibits. We conducted a qualitative user study at an archeological museum with 11 low-vision participants who toured a predefined path while receiving real-time auditory explanations and AR annotations from a remote assistant. Our results reveal that the AR-based remote assistance improved the museum experience, assisting in mobility within the museum and the visibility of artifacts. The use of remote assistance proved to be dynamic and flexible, enabling real-time in-place annotations that helped guide low-vision participants, providing a stronger feeling of security. While acknowledging the transformative potential, participants highlighted challenges in accuracy and responsiveness, emphasizing the need to improve the design of real-time AR annotations. Our study is the first to examine the use of AR remote assistance in meeting the dynamic and diverse needs of people with low vision, illustrating both its potential and the challenges for this population.

Avoiding obstacles while navigating is a challenge for people with low vision, who have impaired yet functional vision, which impacts their mobility, safety, and independence. This study investigates the impact of using Augmented Reality (AR) to enhance the visibility of obstacles for people with low vision. Twenty-five participants (14 with low vision and 11 typically sighted) wore smart glasses and completed a real-world obstacle course under two conditions: with obstacles enhanced using 3D AR markings and without any enhancement (i.e., passthrough only - control condition). Our results reveal that AR enhancements significantly decreased walking time, with the low vision group demonstrating a notable reduction in time. Additionally, the path length was significantly shorter with AR enhancements. The decrease in time and path length did not lead to more collisions, suggesting improved obstacle avoidance. Participants also reported a positive user experience with the AR system, highlighting its potential to enhance mobility for low vision users. These results suggest that AR technology can play a critical role in supporting the independence and confidence of low vision individuals in mobility tasks within complex environments. We discuss design guidelines for future AR systems to assist low vision people.

The accessibility to everyone, including visually impaired people, is one of the key aspects of art in public spaces. Nowadays, the issue of providing blind and low vision people with the opportunity to enjoy artworks is only beginning to be addressed; thus, this practice has not yet become widespread. The physiological and cognitive characteristics of people with visual impairments as well as existing examples of solutions to this issue are analysed in the article. Based on the conducted analysis, a theoretical framework is developed, enabling people with visual impairments to fully perceive art in public spaces.

Object recognition technologies hold the potential to support blind and low-vision people in navigating the world around them. However, the gap between benchmark performances and practical usability remains a significant challenge. This paper presents a study aimed at understanding blind users' interaction with object recognition systems for identifying and avoiding errors. Leveraging a pre-existing object recognition system, URCam, fine-tuned for our experiment, we conducted a user study involving 12 blind and low-vision participants. Through in-depth interviews and hands-on error identification tasks, we gained insights into users' experiences, challenges, and strategies for identifying errors in camera-based assistive technologies and object recognition systems. During interviews, many participants preferred independent error review, while expressing apprehension toward misrecognitions. In the error identification task, participants varied viewpoints, backgrounds, and object sizes in their images to avoid and overcome errors. Even after repeating the task, participants identified only half of the errors, and the proportion of errors identified did not significantly differ from their first attempts. Based on these insights, we offer implications for designing accessible interfaces tailored to the needs of blind and low-vision users in identifying object recognition errors.

Two-factor Authentication (also known as 2FA or two-step verification) is an authentication method that provides an extra layer of protection to ensure online account security. 2FA methods are used along with other primary authentication methods like PINs and Passwords to verify that the person trying to access any digital account is the person they are claiming to be. However, 2FA methods can be inaccessible for blind and low vision (BLV) users due to the requirement of multiple steps, apps, and/or devices for authentication. In addition, it can be a security risk as screen readers may read out the verification codes to bystanders. To address this, we present Haptic2FA, a haptic-based authentication method to improve 2FA accessibility for BLV users. Here, as a part of the 2FA process, the users are sent a 'haptic pattern' (similar to a one-time passcode in traditional 2FA methods) that they are required to enter or select for verification. Through a usability study with 10 BLV participants, we evaluated haptic patterns and input methods for the haptic patterns in the Haptic2FA method. Through the findings, we discuss the accessibility and usability of the Haptic2FA method.

Abstract Which adjectives tend to occur as attributive (the cute/red dress) versus predicative (the dress is cute/red) and why? Building on findings from Wiegand et al. (2013. Predicative adjectives: An unsupervised criterion to extract subjective adjectives. In Lucy Vanderwende, Hal DauméIII & Katrin Kirchhoff (eds.), Proceedings of the 2013 conference of the North American chapter of the Association for Computational Linguistics : Human language technologies (NAACL-HLT), 534–539. Atlanta, GA: Association for Computational Linguistics) and Vartiainen (2013. Subjectivity, indefiniteness and semantic change. English Language and Linguistics 17(1). 157–179), this paper argues that subjective adjectives such as cute tend to be placed in predicative position not just because they often describe discourse-new information, but because this position serves to foreground information that the hearer may disagree with. This claim is supported using data from the Corpus of Contemporary American English (Davies, Mark. 2008. The corpus of contemporary American English: One billion words, 1990-present. Available at: https://www.english-corpora.org/coca/) combined with human annotations for subjectivity from Scontras et al. (2017. Subjectivity predicts adjective ordering preferences. Open Mind 1(1). 53–66) et seq.; and data from image captions versus descriptions (for seeing versus low-vision people) from the National Gallery of Art. A production experiment manipulates the discourse context to further show that adjectives tend to be placed in predicative position when they express controversial information. Overall, this paper explores how the lexical semantics of adjectives shapes the pragmatic contexts in which they tend to be used, which in turn shapes the syntax of the sentences using them.

To estimate utility values associated with visual loss using EuroQol (EQ-5D) questionnaire, the impact of low-vision (LV) device use on utilities and the contribution of Instrumental Activities of Daily Living (IADL) score in patients attending vision rehabilitation (VR) services enrolled in the Italian Device & Aids Register (D.A.Re). This is a multicenter, prospective, cross-sectional study. D.A.Re. collects general and clinical information, vision-specific variables, use of electronic devices and quality of life questionnaires. A total of 442 patients (75.0±16.6 years, 275 female) were included, 88 (19.9%) used specialised electronic LV devices, and 116 (26.2%) used smartphones and tablets. Users of smartphones and tablets were younger than non-users (67.5 vs. 77.6 years, p<0.001), but overall, their age ranged between 20 and 93. Stronger associations were found between vision-specific variables and IADL score compared to EQ-5D score. In multivariable age-adjusted models, the utility value of using smartphones and tablets on EQ-5D score was 0.12 (p<0.01), slightly larger than that of 1.0 logMAR difference (-0.09, p<0.01) or visual field damage within 10° of fixation (-0.10, p<0.01). Use of portable low-vision electronic devices and being employed or student (vs. retired) was also associated with better utility values (0.12 and 0.15, respectively, p<0.05). Visual loss is associated with loss of utilities in Italian patients attending VR services, whereas special-purpose electronic aids, and smartphone and tablet use are associated with better utility values. We found that IADL may be more sensitive to visual loss than EQ-5D and could be a valid health-related quality of life outcome in trials on VR.

This article introduces the “Mobile Bio-Eye-Tronic System” which is an artificial vision system for the impaired (blind) people and also low vision (sight loss) people. “Mobile Bio-Eye-Tronic System” is a completely original project and unique to the author of this article. There are 45 million visually impaired (blind) people in the world and 135 million low vision (sight loss) people. 60% of blindness in the world is treatable and 20% is preventable. 25 million people are blind in Europe, 12 million in America, 9 million in India, 6 million in China, and 7 million in Africa. In Turkey, this number is approximately 300 thousand. Based on these statistics, the main aim of this article is to appeal to hundreds of thousands of people, to help them fulfill their daily activities, even partially, to improve the quality of life of these visually impaired people and to restore their health. In addition, the scientific and technical studies to be carried out on this subject will contribute to the enrichment of the literature on the subject and will also be beneficial for scientific and technical progress. When the current studies on bionic eyes are examined, there is no other system in the literature that obtains results using a mobile phone camera and software. The bionic eye stated in this article will be a first in this respect.

This article introduces the “Mobile Bio-Eye-Tronic System” which is an artificial vision system for the impaired (blind) people and also low vision (sight loss) people. “Mobile Bio-Eye-Tronic System” is a completely original project and unique to the author of this article. There are 45 million visually impaired (blind) people in the world and 135 million low vision (sight loss) people. 60% of blindness in the world is treatable and 20% is preventable. 25 million people are blind in Europe, 12 million in America, 9 million in India, 6 million in China, and 7 million in Africa. In Turkey, this number is approximately 300 thousand. Based on these statistics, the main aim of this article is to appeal to hundreds of thousands of people, to help them fulfill their daily activities, even partially, to improve the quality of life of these visually impaired people and to restore their health. In addition, the scientific and technical studies to be carried out on this subject will contribute to the enrichment of the literature on the subject and will also be beneficial for scientific and technical progress. When the current studies on bionic eyes are examined, there is no other system in the literature that obtains results using a mobile phone camera and software. The bionic eye stated in this article will be a first in this respect. [1-29]

Visual art facilitates expression, communication, and connection, yet it remains inaccessible to those who are visually-impaired and those who lack the resources to understand the techniques and history of art. In this work, I propose the development of a generative AI model that generates a description and interpretation of a given artwork. Such research can make art more accessible, support art education, and improve the ability of AI to understand and translate between creative media. Development will begin with a formative study to assess the needs and preferences of blind and low vision people and art experts. Following the formative study, the basic approach is to train the model on a database of artworks and their accompanying descriptions, predict sentiments from extracted visual data, and generate a paragraph closely resembling training textual data and incorporating sentiment analysis. The model will then be evaluated quantitatively through metrics like METEOR and qualitatively through Turing tests in an iterative process.

Low vision is a visual impairment that falls short of blindness but cannot be corrected by eyeglasses or contact lenses. While current low vision aids (e.g., magnifier, CCTV) support basic vision enhancements, such as magnification and contrast enhancement, these enhancements often arbitrarily alter a user's full field of view without considering the user's context, such as their visual abilities, tasks, and environmental factors. As a result, these low vision aids are not sufficient or preferred by low vision users in many important tasks. Augmented reality (AR) technology presents a unique opportunity to enhance low vision people's visual experience by automatically recognizing the surrounding environment and presenting tailored visual augmentations. In this talk, I will talk about how we design and build intelligent AR systems to support low vision people in visual tasks, such as a head-mounted AR system that presents visual cues to orient users' attention in a visual search task, as well as a projection-based AR system that projects visual highlights on the stair edges to support safe stair navigation. I will conclude my talk by discussing our future research direction on AR for low vision accessibility.

ABSTRACT Maps are frequently employed in the natural hazard mitigation planning (NHMP) process for analyze a community’s vulnerability to hazards and illustrating the character of potential hazards. But because the encoded information of these maps relies on visual access, blind or low-vision (B/LV) people who want to contribute to their community’s NHMP efforts are therefore effectively excluded from any stage of the process involving maps. In response, we investigate tactile flood maps as an accessible tool for NHMP. Our study proposes a workflow for creating thematic tactile flood maps using existing resources, methods, and design conventions, and then evaluates those thematic tactile maps to understand user confidence amongst B/LV users. This work contributes a proposed configuration of tactile mapping resources to be used for NHMP and evaluates user confidence while using those resources. Results suggest that B/LV users respond positively to using tactile maps in a high-stakes context such as NHMP, and that tactile maps can expand the number and diversity of people who are able to contribute to NHMP. To maximize contributions, we recommend that future tactile map research invest a greater amount of attention to developing resources for B/LV people to create, edit, and distribute maps themselves.

The usage of smartphones is increasingly widespread, and the usefulness of mobile applications as low-vision aids is evident but not thoroughly examined. In this study, we surveyed people with low vision to assess the usability of common, preloaded mobile applications, to evaluate the usage of typical assistive technologies of smartphones, and to measure the usefulness, and usability of recent software advancements that can be used as visual aids. We invited 134 low-vision individuals to participate, and 45 of them met the eligibility criteria and completed an in-person survey. The eligibility criteria were as follows: aged 18 years or older and mentally competent, visual acuity worse than 0.4 logMAR with best-corrected glasses in the better-seeing eye, ownership of a smartphone and familiarity with visual assistive technologies. All testing scenarios were carried out using the participants' smartphones, either with Android or iOS operating systems. Participants reported the usefulness and ease of use for common visual display enhancements (i.e., text size, bold text, increased contrast, inverted colors, and dark mode), audio feedback capabilities, four primary preloaded apps (Dialer, Clock, Calculator, and Calendar), and four usage scenarios that serve as low-vision aids (magnify with camera, hard-copy text-to-speech, voice typing, and voice commands). Participants also indicated whether they could use the apps or execute the scenarios independently. The Dialer and Clock apps, text enhancements, camera magnification, and voice typing were rated as highly useful, while the Calendar application received lower ratings. Most of the selected apps or services were rated as easy to use, with lower ratings recorded for the Calendar and Select to Speak ones. Considering the positive results across all options, this collection of apps and services proved useful for all age groups, regardless of gender, technological familiarity, or education. The feedback received in this study can help towards improving the everyday lives of low-vision people as well as informing the design of apps and assistive features, guiding future research and development to enhance visual accessibility on mobile computing devices.

To investigate the characteristics of electronic device users, specifically smartphones and tablets, in the Device & Aids Register (D.A.Re), from several low-vision rehabilitation services in Italy. We collected general and clinical information about ocular and systemic diseases, visual function, reading speed and Instrumental Activities of Daily Living (IADL) questionnaire score. Technological details of each optical and electronic device, (including screen size, touch-screen and OCR functions, text-to-speech function) were also collected. 1218 patients (752 females and 466 males) were included in our analysis, mean age 71.5 (±18.8) years. Users of electronic aids (n.237) were slightly younger (67 vs 72 years, p < 0.001) than non-users (n.981), had a worse reading speed (38 vs 65 words/minute), critical print size (43 vs 28 print size, p < 0.001), poorer visual acuity (VA)(1.0 logMAR or less: 30% non-users vs 73% users, p < 0.001) and more commonly visual field restriction within 10° (23% vs 14%, p = 0.001). A similar proportion of users and non-users were retired (about 70%) and about 16-17% were employed. The use of portable electronic devices (5″or less, p < 0.001; 6″ to 18″ screen size, p = 0.017) was associated with better IADL scores, and the use of stand devices with worse IADL score (p < 0.001); Furthermore, using smartphones and tablets (193 subjects) was strongly associated with better IADL scores. We found that using electronic devices, and especially smartphone and tablets, were associated with better vision-related quality of life in low-vision people attending rehabilitation services. While this association does not mean causality, these findings seemed robust to confounder adjustment.

Gait is an individual's walking pattern, and it is a significant part of daily living activities. Quantitative gait assessments, like spatiotemporal parameters (STPs), are related to the functional conditions to provide useful information. This study reviewed the comprehensive differences in spatiotemporal gait variability measures between visually impaired people and the sighted. The search strategy was performed in three databases (PubMed/MEDLINE, Web of Science, and Scopus) from the start date to October 2022, and the utilized keywords for this search are related to gait and blindness. This review considered only those studies that evaluated gait parameters in people with visual impairment and blind people without any limitations in age and gender. In this review, studies without a control group (sighted people) were excluded. The Newcastle-Ottawa Scale (NOS) was applied for critical appraisal. Six full manuscripts were included. The sample size ranged from 19 to 91. The mean modified NOS critical appraisal scores for cross-sectional studies were 6.0. In these studies, among nine STPs: stride length, walking speed, stance and swing phase, step width, cadence, step length, double support, and single support, at least five and at most seven factors were examined. The gait pattern of blind and low-vision people is characterized by a slower walking speed, shorter stride length, increased step width, decreased cadence, prolonged duration of double support, and reduced single support compared to the controls.

BackgroundPersonal health technologies, including wearable tracking devices and mobile apps, have great potential to equip the general population with the ability to monitor and manage their health. However, being designed for sighted people, much of their functionality is largely inaccessible to the blind and low-vision (BLV) population, threatening the equitable access to personal health data (PHD) and health care services.ObjectiveThis study aims to understand why and how BLV people collect and use their PHD and the obstacles they face in doing so. Such knowledge can inform accessibility researchers and technology companies of the unique self-tracking needs and accessibility challenges that BLV people experience.MethodsWe conducted a web-based and phone survey with 156 BLV people. We reported on quantitative and qualitative findings regarding their PHD tracking practices, needs, accessibility barriers, and work-arounds.ResultsBLV respondents had strong desires and needs to track PHD, and many of them were already tracking their data despite many hurdles. Popular tracking items (ie, exercise, weight, sleep, and food) and the reasons for tracking were similar to those of sighted people. BLV people, however, face many accessibility challenges throughout all phases of self-tracking, from identifying tracking tools to reviewing data. The main barriers our respondents experienced included suboptimal tracking experiences and insufficient benefits against the extended burden for BLV people.ConclusionsWe reported the findings that contribute to an in-depth understanding of BLV people’s motivations for PHD tracking, tracking practices, challenges, and work-arounds. Our findings suggest that various accessibility challenges hinder BLV individuals from effectively gaining the benefits of self-tracking technologies. On the basis of the findings, we discussed design opportunities and research areas to focus on making PHD tracking technologies accessible for all, including BLV people.