Simulated Phosphene Research Articles

Learning Scene Representations for Human-assistive Displays Using Self-attention Networks

Video-see-through (VST) augmented reality (AR) is widely used to present novel augmentative visual experiences by processing video frames for viewers. Among VST AR systems, assistive vision displays aim to compensate for low vision or blindness, presenting enhanced visual information to support activities of daily living for the vision impaired/deprived. Despite progress, current assistive displays suffer from a visual information bottleneck, limiting their functional outcomes compared to healthy vision. This motivates the exploration of methods to selectively enhance and augment salient visual information. Traditionally, vision processing pipelines for assistive displays rely on hand-crafted, single-modality filters, lacking adaptability to time-varying and environment-dependent needs. This article proposes the use of Deep Reinforcement Learning (DRL) and Self-attention (SA) networks as a means to learn vision processing pipelines for assistive displays. SA networks selectively attend to task-relevant features, offering a more parameter—and compute-efficient approach to RL-based task learning. We assess the feasibility of using SA networks in a simulation-trained model to generate relevant representations of real-world states for navigation with prosthetic vision displays. We explore two prosthetic vision applications, vision-to-auditory encoding, and retinal prostheses, using simulated phosphene visualisations. This article introduces SA-px, a general-purpose vision processing pipeline using self-attention networks, and SA-phos, a display-specific formulation targeting low-resolution assistive displays. We present novel scene visualisations derived from SA image patches importance rankings to support mobility with prosthetic vision devices. To the best of our knowledge, this is the first application of self-attention networks to the task of learning vision processing pipelines for prosthetic vision or assistive displays.

Retinal prosthetic vision simulation: temporal aspects

Objective. The perception of individuals fitted with retinal prostheses is not fully understood, although several retinal implants have been tested and commercialized. Realistic simulations of perception with retinal implants would be useful for future development and evaluation of such systems. Approach. We implemented a retinal prosthetic vision simulation, including temporal features, which have not been previously simulated. In particular, the simulation included temporal aspects such as persistence and perceptual fading of phosphenes and the electrode activation rate. Main results. The simulated phosphene persistence showed an effective reduction in flickering at low electrode activation rates. Although persistence has a positive effect on static scenes, it smears dynamic scenes. Perceptual fading following continuous stimulation affects prosthetic vision of both static and dynamic scenes by making them disappear completely or partially. However, we showed that perceptual fading of a static stimulus might be countered by head-scanning motions, which together with the persistence revealed the contours of the faded object. We also showed that changing the image polarity may improve simulated prosthetic vision in the presence of persistence and perceptual fading. Significance. Temporal aspects have important roles in prosthetic vision, as illustrated by the simulations. Considering these aspects may improve the future design, the training with, and evaluation of retinal prostheses.

Full gaze contingency provides better reading performance than head steering alone in a simulation of prosthetic vision

The visual pathway is retinotopically organized and sensitive to gaze position, leading us to hypothesize that subjects using visual prostheses incorporating eye position would perform better on perceptual tasks than with devices that are merely head-steered. We had sighted subjects read sentences from the MNREAD corpus through a simulation of artificial vision under conditions of full gaze compensation, and head-steered viewing. With 2000 simulated phosphenes, subjects (n = 23) were immediately able to read under full gaze compensation and were assessed at an equivalent visual acuity of 1.0 logMAR, but were nearly unable to perform the task under head-steered viewing. At the largest font size tested, 1.4 logMAR, subjects read at 59 WPM (50% of normal speed) with 100% accuracy under the full-gaze condition, but at 0.7 WPM (under 1% of normal) with below 15% accuracy under head-steering. We conclude that gaze-compensated prostheses are likely to produce considerably better patient outcomes than those not incorporating eye movements.

Sensory augmentation to aid training with retinal prostheses

Objective. Retinal prosthesis recipients require rehabilitative training to learn the non-intuitive nature of prosthetic ‘phosphene vision’. This study investigated whether the addition of auditory cues, using The vOICe sensory substitution device (SSD), could improve functional performance with simulated phosphene vision. Approach. Forty normally sighted subjects completed two visual tasks under three conditions. The phosphene condition converted the image to simulated phosphenes displayed on a virtual reality headset. The SSD condition provided auditory information via stereo headphones, translating the image into sound. Horizontal information was encoded as stereo timing differences between ears, vertical information as pitch, and pixel intensity as audio intensity. The third condition combined phosphenes and SSD. Tasks comprised light localisation from the Basic Assessment of Light and Motion (BaLM) and the Tumbling-E from the Freiburg Acuity and Contrast Test (FrACT). To examine learning effects, twenty of the forty subjects received SSD training prior to assessment. Main results. Combining phosphenes with auditory SSD provided better light localisation accuracy than either phosphenes or SSD alone, suggesting a compound benefit of integrating modalities. Although response times for SSD-only were significantly longer than all other conditions, combined condition response times were as fast as phosphene-only, highlighting that audio-visual integration provided both response time and accuracy benefits. Prior SSD training provided a benefit to localisation accuracy and speed in SSD-only (as expected) and Combined conditions compared to untrained SSD-only. Integration of the two modalities did not improve spatial resolution task performance, with resolution limited to that of the higher resolution modality (SSD). Significance. Combining phosphene (visual) and SSD (auditory) modalities was effective even without SSD training and led to an improvement in light localisation accuracy and response times. Spatial resolution performance was dominated by auditory SSD. The results suggest there may be a benefit to including auditory cues when training vision prosthesis recipients.

Semantic and structural image segmentation for prosthetic vision.

Prosthetic vision is being applied to partially recover the retinal stimulation of visually impaired people. However, the phosphenic images produced by the implants have very limited information bandwidth due to the poor resolution and lack of color or contrast. The ability of object recognition and scene understanding in real environments is severely restricted for prosthetic users. Computer vision can play a key role to overcome the limitations and to optimize the visual information in the prosthetic vision, improving the amount of information that is presented. We present a new approach to build a schematic representation of indoor environments for simulated phosphene images. The proposed method combines a variety of convolutional neural networks for extracting and conveying relevant information about the scene such as structural informative edges of the environment and silhouettes of segmented objects. Experiments were conducted with normal sighted subjects with a Simulated Prosthetic Vision system. The results show good accuracy for object recognition and room identification tasks for indoor scenes using the proposed approach, compared to other image processing methods.

Perceptual learning in a non-human primate model of artificial vision.

Visual perceptual grouping, the process of forming global percepts from discrete elements, is experience-dependent. Here we show that the learning time course in an animal model of artificial vision is predicted primarily from the density of visual elements. Three naïve adult non-human primates were tasked with recognizing the letters of the Roman alphabet presented at variable size and visualized through patterns of discrete visual elements, specifically, simulated phosphenes mimicking a thalamic visual prosthesis. The animals viewed a spatially static letter using a gaze-contingent pattern and then chose, by gaze fixation, between a matching letter and a non-matching distractor. Months of learning were required for the animals to recognize letters using simulated phosphene vision. Learning rates increased in proportion to the mean density of the phosphenes in each pattern. Furthermore, skill acquisition transferred from trained to untrained patterns, not depending on the precise retinal layout of the simulated phosphenes. Taken together, the findings suggest that learning of perceptual grouping in a gaze-contingent visual prosthesis can be described simply by the density of visual activation.

Active confocal imaging for visual prostheses

There are encouraging advances in prosthetic vision for the blind, including retinal and cortical implants, and other “sensory substitution devices” that use tactile or electrical stimulation. However, they all have low resolution, limited visual field, and can display only few gray levels (limited dynamic range), severely restricting their utility. To overcome these limitations, image processing or the imaging system could emphasize objects of interest and suppress the background clutter. We propose an active confocal imaging system based on light-field technology that will enable a blind user of any visual prosthesis to efficiently scan, focus on, and “see” only an object of interest while suppressing interference from background clutter. The system captures three-dimensional scene information using a light-field sensor and displays only an in-focused plane with objects in it. After capturing a confocal image, a de-cluttering process removes the clutter based on blur difference. In preliminary experiments we verified the positive impact of confocal-based background clutter removal on recognition of objects in low resolution and limited dynamic range simulated phosphene images. Using a custom-made multiple-camera system based on light-field imaging, we confirmed that the concept of a confocal de-cluttered image can be realized effectively.

Improved visual performance in letter perception through edge orientation encoding in a retinal prosthesis simulation

Objective. Stimulation strategies for retinal prostheses predominately seek to directly encode image brightness values rather than edge orientations. Recent work suggests that the generation of oriented elliptical phosphenes may be possible by controlling interactions between neighboring electrodes. Based on this, we propose a novel stimulation strategy for prosthetic vision that extracts edge orientation information from the intensity image and encodes it as oriented elliptical phosphenes. We test the hypothesis that encoding edge orientation via oriented elliptical phosphenes leads to better alphabetic letter recognition than standard intensity-based encoding. Approach. We conduct a psychophysical study with simulated phosphene vision with 12 normal-sighted volunteers. The two stimulation strategies were compared with variations of letter size, electrode drop-out and spatial offsets of phosphenes. Main results. Mean letter recognition accuracy was significantly better with the new proposed stimulation strategy (65%) compared to direct grayscale encoding (47%). All examined parameters—stimulus size, phosphene dropout, and location shift—were found to influence the performance, with significant two-way interactions between phosphene dropout and stimulus size as well as between phosphene dropout and phosphene location shift. The analysis delivers a model of perception performance. Significance. Displaying available directional information to an implant user may improve their visual performance. We present a model for designing a stimulation strategy under the constraints of existing retinal prostheses that can be exploited by retinal implant developers to strategically employ oriented phosphenes.

Optimizing Chinese Character Displays Improves Recognition and Reading Performance of Simulated Irregular Phosphene Maps

A visual prosthesis may elicit an irregular phosphene map relative to a regular electrode array. This study used simulated irregular phosphene maps as a way of optimizing the display methods of Chinese characters (CCs) to improve recognition and reading performance. TWENTY SUBJECTS WITH NORMAL OR CORRECTED SIGHT PARTICIPATED IN TWO EXPERIMENTS (9 FEMALES, 11 MALES, 2030 YEARS OF AGE). EXPERIMENT 1: two character display methods were proposed: selecting phosphenes covered by character strokes on a simulated phosphene array (projection method) and finding the phosphene closest to the expected location in some range of an irregular phosphene array as a substitute (nearest neighbor search [NNS] method). The recognition accuracy of CCs was investigated using six levels for the coverage ratio of stroke and phosphene area and for search range, respectively, for two methods, for several irregularity levels. Experiment 2: reading accuracy (RA) and reading efficiency (RE) were measured using the regular array correspondence and NNS methods. EXPERIMENT 1: projection and NNS methods were significantly affected by coverage ratio or search range. NNS significantly improved CC recognition accuracy to the highest at 81.3 ± 2.7% and 59.1 ± 5.2%, respectively, for different irregularity levels, compared with the projection method. Experiment 2: RA and RE significantly decreased as the distortion level increased; NNS significantly improved RA (from approximately 40% to >80%) and RE (from approximately 13 char/min to >40 char/min) when reading more irregular paragraphs. The performance of CC recognition and paragraph reading when using an irregular phosphene array can be improved through optimizing the display method.

Estimation of Simulated Phosphene Size Based on Tactile Perception

Clinical trials have successfully shown that a visual prosthesis can elicit visual perception (phosphenes) in the visual field. Psychophysical studies based on simulated prosthetic vision offer an effective means to evaluate and refine prosthetic vision. We designed three experiments to examine the effect of phosphene luminance, flicker rate, and eccentricity on the ability to estimate simulated phosphene sizes using tactile perception. Thirty subjects participated in the three experiments. There was a linear increase in reported size as visual stimulus size increased. Judgment was significantly affected by stimulus luminance and eccentricity (P < 0.05) but not by flicker rates. Brighter stimuli were perceived as being larger, and the more eccentric the position, the larger the estimated size. These simulation studies, although idealized, suggested that tactile perception is a potential way to estimate phosphene sizes.

Chinese Character Recognition Using Simulated Phosphene Maps

A visual prosthetic device may produce phosphene maps in which individual phosphene characteristics can be altered. This study was an investigation of the ability of normally sighted subjects to recognize Chinese characters (CCs) after altering simulated phosphene maps. Thirty volunteers with normal or corrected visual acuity of 20/20 were recruited. CC recognition accuracy and response time were investigated while one parameter was changed (distortion, pixel dropout percentage, pixel size variability, or pixel gray level) or different combinations of three parameters were used. Five hundred CCs consisting of 1 to 16 strokes were used for the character sets. CC recognition accuracy and response times respectively decreased and increased when distortion, dropout, and pixel size variability increased. Gray levels did not significantly affect the results, except when eight levels were used. To maintain an 80% accuracy rate, there should be a distortion index (k) of no more than 0.2 (irregularity), a pixel dropout of 20%, and a pixel size range of 1 to 16 mm (7-112 min arc). Only a combination of a k=0.1 distortion index, a dropout of 10%, and a pixel size range of 1.33 to 12 mm (9.3-84 min arc) achieved a goal of ≥80% accuracy. Distortion, dropout percentage, and pixel size variability have a significant impact on pixelated CC recognition. Although at present the visual ability of prosthesis users is limited, it should be possible to extend this to CC recognition and reading in the future. The results will help visual prosthesis researchers determine the effects of altering phosphene maps and improve outcomes for patients.

Dispersion and Accuracy of Simulated Phosphene Positioning Using Tactile Board

The evaluation of phosphene is one of the most important things after the electrode array's implantation. Some experiments have been used to study the basic rules on people using their tactile perception in describing the characteristics of simulated phosphenes, such as the dispersion, accuracy, and response time. In order to describe the simulated phosphenes more systematically and scientifically, an improved phosphene-positioning method using tactile board was designed in this study. Using tactile board to guide the tactile perception, the dispersion was limited to less than 6 mm, while the largest mean error was 8.1 mm, which was nearly equal to the minimal resolution of tactile board (8 mm). The response time greatly increased because of the standard processes in using tactile board. Moreover, the long-term experiment to repeat the procedure could improve and sustain the subjects' performance in good results.

C-Sight Visual Prostheses for the Blind

Visual prostheses based on a stimulating microelectrode array to restore vision offer a promising approach for the blind and has become a rapidly growing scientific field in neurorehabilitation engineering. A number of research groups from major developed countries lead the research activities in this field. The goal of the C-Sight Project is to develop an implantable microelectronic medical device that will restore useful vision to blind patients. Retinitis pigmentosa (RP) and age-related macular degeneration are the two leading causes for blindness, for which there have been no effective treatments, both surgically and biologically, until now.The ultimate goal of our project is to develop a completely implantable visual prosthesis based on a penetrating microelectrode array at the optical nerve; to implement this, it requires many technical advances. However, in our study, an implantable microcamera, the wearable information processor, and the multichannel neurostimulator are investigated. Electrophysiological experiments were also performed to provide evidence for the feasibility of our approach. Some related psychophysical studies including simulated phosphene positioning and recognition of pixelized images are also reported in this article.

Reading speed with a pixelized vision system.

A visual prosthesis based on electrical stimulation of the visual cortex with an array of penetrating electrodes is expected to produce pixelized visual images consisting of punctate spots of light (phosphenes). We measured reading speed in subjects viewing text with optically simulated phosphene fields in order to obtain estimates of the following design parameters for such an electrode array: pixel number, pixel spacing, and visual-field size. Comparisons were made between scanning the text with eye movements and scanning the text with head movements. The results indicate that a 25 x 25 array of pixels representing four letters of text projected on a foveal visual field of 1.7 degrees is sufficient to provide reading rates near 170 words/min with scrolled text and near 100 words/min with fixed text.