Integration Of Visual Information Research Articles

Exploring AKAPs in visual signaling.

The complex nature of the retina demands well-organized signaling to uphold signal accuracy and avoid interference, a critical aspect in handling a variety of visual stimuli. A-kinase anchoring proteins (AKAPs), known for binding protein kinase A (PKA), contribute to the specificity and efficiency of retinal signaling. They play multifaceted roles in various retinal cell types, influencing photoreceptor sensitivity, neurotransmitter release in bipolar cells, and the integration of visual information in ganglion cells. AKAPs like AKAP79/150 and AKAP95 exhibit distinct subcellular localizations, impacting synaptic transmission and receptor sensitivity in photoreceptors and bipolar cells. Furthermore, AKAPs are involved in neuroprotective mechanisms and axonal degeneration, particularly in retinal ganglion cells. In particular, AKAP6 coordinates stress-specific signaling and promotes neuroprotection following optic nerve injury. As our review underscores the therapeutic potential of targeting AKAP signaling complexes for retinal neuroprotection and enhancement, it acknowledges challenges in developing selective drugs that target complex protein-protein interactions. Overall, this exploration of AKAPs provides valuable insights into the intricacies of retinal signaling, offering a foundation for understanding and potentially addressing retinal disorders.

Role of the left posterior middle temporal gyrus in shape recognition and its reconstruction during drawing: A study combining transcranial magnetic stimulation and functional near infrared spectroscopy.

There are numerous reports of enhanced or emerged visual arts abilities in patients with semantic impairment. These reports led to the theory that a loss of function on the language side of the brain can result in changes of ability to draw and/or to paint. Further, the left posterior middle temporal gyrus (l-pMTG) has been revealed to contribute to the higher control semantic mechanisms with objects recognition and integration of visual information, within a widely distributed network of the left hemisphere. Nevertheless, the theory has not been fully studied in neural bases. The aim of this study is to examine role of the l-pMTG on shape recognition and its reconstruction within drawing behavior, by using a combining method of the repetitive transcranial magnetic stimulation (rTMS) and functional near-infrared spectroscopy (fNIRS). Eighteen healthy participants received a low frequency inhibitory rTMS to their l-pMTG during the drawing task of the Benton Visual Retention Test (BVRT). There was a significant decrease of the mean accuracy of reproductions in the Complex designs of the BVRT, compared to the Simple and Medium designs. The fNIRS data showed strong negative correlations with the results of the BVRT. Though our hypothesis had a contradiction that rTMS would have inhibited the brain activity in the stimulated site, the results suggest that shape recognition and its reconstruction such as the BVRT require neural activations of the l-TL as well as that of the l-pMTG.

Gait Adaptability and the Effect of Ocular Disorders on Visually Guided Walking in Parkinson's Disease.

Gait disorders are a disabling feature of Parkinson's disease (PD). To avoid falls, people with PD should be able to adequately adapt their gait. This requires correct response inhibition and integration of visual information. In this small pilot study, we investigated PD-related impairments in gait adaptability and the influence of ocular disorders thereon. Compared with controls, persons with PD were less able to adapt their gait in unexpected situations (U = 21.5, p = 0.013), with only a small influence of ocular disorders on precision stepping (U = 6, p = 0.012 in the ML-direction and in the AP-direction, (U = 20, p = 0.456). This shows that people with PD have more difficulty with precision stepping than healthy controls and experience more problems with adapting their gait. We found only a small impact of ocular disorders on successfully execute precision stepping. The ability to adapt gait, particularly in challenging environmental conditions or with impaired vision, may provide a useful assessment and training option for fall prevention in PD.

Deep Aphantasia: a visual brain with minimal influence from priors or inhibitory feedback?

The authors are both self-described congenital aphantasics, who feel they have never been able to have volitional imagined visual experiences during their waking lives. In addition, Loren has atypical experiences of a number of visual phenomena that involve an extrapolation or integration of visual information across space. In this perspective, we describe Loren's atypical experiences of a number of visual phenomena, and we suggest these ensue because her visual experiences are not strongly shaped by inhibitory feedback or by prior expectations. We describe Loren as having Deep Aphantasia, and Derek as shallow, as for both a paucity of feedback might prevent the generation of imagined visual experiences, but for Loren this additionally seems to disrupt activity at a sufficiently early locus to cause atypical experiences of actual visual inputs. Our purpose in describing these subjective experiences is to alert others to the possibility of there being sub-classes of congenital aphantasia, one of which-Deep Aphantasia, would be characterized by atypical experiences of actual visual inputs.

CLIPSyntel: CLIP and LLM Synergy for Multimodal Question Summarization in Healthcare

In the era of modern healthcare, swiftly generating medical question summaries is crucial for informed and timely patient care. Despite the increasing complexity and volume of medical data, existing studies have focused solely on text-based summarization, neglecting the integration of visual information. Recognizing the untapped potential of combining textual queries with visual representations of medical conditions, we introduce the Multimodal Medical Question Summarization (MMQS) Dataset. This dataset, a major contribution of our work, pairs medical queries with visual aids, facilitating a richer and more nuanced understanding of patient needs. We also propose a framework, utilizing the power of Contrastive Language Image Pretraining(CLIP) and Large Language Models(LLMs), consisting of four modules that identify medical disorders, generate relevant context, filter medical concepts, and craft visually aware summaries. Our comprehensive framework harnesses the power of CLIP, a multimodal foundation model, and various general-purpose LLMs, comprising four main modules: the medical disorder identification module, the relevant context generation module, the context filtration module for distilling relevant medical concepts and knowledge, and finally, a general-purpose LLM to generate visually aware medical question summaries. Leveraging our MMQS dataset, we showcase how visual cues from images enhance the generation of medically nuanced summaries. This multimodal approach not only enhances the decision-making process in healthcare but also fosters a more nuanced understanding of patient queries, laying the groundwork for future research in personalized and responsive medical care. Disclaimer: The article features graphic medical imagery, a result of the subject's inherent requirements.

Manipulating Visually Aware Federated Recommender Systems and Its Countermeasures

Federated recommender systems (FedRecs) have been widely explored recently due to their capability to safeguard user data privacy. These systems enable a central server to collaboratively learn recommendation models by sharing public parameters with clients, providing privacy-preserving solutions. However, this collaborative approach also creates a vulnerability that allows adversaries to manipulate FedRecs. Existing works on FedRec security already reveal that items can easily be promoted by malicious users via model poisoning attacks, but all of them mainly focus on FedRecs with only collaborative information (i.e., user–item interactions). We contend that these attacks are effective primarily due to the data sparsity of collaborative signals. In light of this, we propose a method to address data sparsity and model poisoning threats by incorporating product visual information. Intriguingly, our empirical findings demonstrate that the inclusion of visual information renders all existing model poisoning attacks ineffective. Nevertheless, the integration of visual information also introduces a new avenue for adversaries to manipulate federated recommender systems, as this information typically originates from external sources. To assess such threats, we propose a novel form of poisoning attack tailored for visually aware FedRecs, namely image poisoning attacks, where adversaries can gradually modify the uploaded image with human-unaware perturbations to manipulate item ranks during the FedRecs’ training process. Moreover, we provide empirical evidence showcasing a heightened threat when image poisoning attacks are combined with model poisoning attacks, resulting in easier manipulation of the federated recommendation systems. To ensure the safe utilization of visual information, we employ a diffusion model in visually aware FedRecs to purify each uploaded image and detect the adversarial images. Extensive experiments conducted with two FedRecs on two datasets demonstrate the effectiveness and generalization of our proposed attacks and defenses.

Electrophysiological signatures of visual temporal processing deficits in developmental dyslexia.

Developmental dyslexia (DD) is a common neurodevelopmental disorder that affects reading ability despite normal intelligence and education. In search of core deficits, previous evidence has linked DD with impairments in temporal aspects of perceptual processing, which might underlie phonological deficits as well as inefficient graphemic parsing during reading. However, electrophysiological evidence for atypical temporal processing in DD is still scarce in the visual modality. Here, we investigated the efficiency of both temporal segregation and integration of visual information by means of event-related potentials (ERPs). We confirmed previous evidence of a selective segregation deficit in dyslexia for stimuli presented in rapid succession (<80 ms), despite unaffected integration performance. Importantly, we found a reduced N1 amplitude in DD, a component related to the allocation of attentional resources, which was independent of task demands (i.e., evident in both segregation and integration). In addition, the P3 amplitude, linked to working memory and processing load, was modulated by task demands in controls but not in individuals with DD. These results suggest that atypical attentional sampling in dyslexia might weaken the quality of information stored in visual working memory, leading to behavioral and electrophysiological signatures of atypical temporal segregation. These results are consistent with some existing theories of dyslexia, such as the magnocellular theory and the "Sluggish Attentional Shifting" framework, and represent novel evidence for neural correlates of decreased visual temporal resolution in DD.

Nonadditive integration of visual information in ensemble processing

Statistically summarizing information from a stimulus array into an ensemble representation (e.g., the mean) improves the efficiency of visual processing. However, little is known about how the brain computes the ensemble statistics. Here, we propose that ensemble processing is realized by nonadditive integration, rather than linear averaging, of individual items. We used a linear regression model approach to extract EEG responses to three levels of information: the individual items, their local interactions, and their global interaction. The local and global interactions, representing nonadditive integration of individual items, elicited rapid and independent neural responses. Critically, only the neural representation of the global interaction predicted the precision of the ensemble perception at the behavioral level. Furthermore, spreading attention over the global pattern to enhance ensemble processing directly promoted rapid neural representation of the global interaction. Taken together, these findings advocate a global, nonadditive mechanism of ensemble processing in the brain.

Species-specific wiring of cortical circuits for small-world networks in the primary visual cortex.

Long-range horizontal connections (LRCs) are conspicuous anatomical structures in the primary visual cortex (V1) of mammals, yet their detailed functions in relation to visual processing are not fully understood. Here, we show that LRCs are key components to organize a "small-world network" optimized for each size of the visual cortex, enabling the cost-efficient integration of visual information. Using computational simulations of a biologically inspired model neural network, we found that sparse LRCs added to networks, combined with dense local connections, compose a small-world network and significantly enhance image classification performance. We confirmed that the performance of the network appeared to be strongly correlated with the small-world coefficient of the model network under various conditions. Our theoretical model demonstrates that the amount of LRCs to build a small-world network depends on each size of cortex and that LRCs are beneficial only when the size of the network exceeds a certain threshold. Our model simulation of various sizes of cortices validates this prediction and provides an explanation of the species-specific existence of LRCs in animal data. Our results provide insight into a biological strategy of the brain to balance functional performance and resource cost.

Musical expertise shapes visual-melodic memory integration.

Music can act as a mnemonic device that can elicit multiple memories. How musical and non-musical information integrate into complex cross-modal memory representations has however rarely been investigated. Here, we studied the ability of human subjects to associate visual objects with melodies. Musical laypersons and professional musicians performed an associative inference task that tested the ability to form and memorize paired associations between objects and melodies ("direct trials") and to integrate these pairs into more complex representations where melodies are linked with two objects across trials ("indirect trials"). We further investigated whether and how musical expertise modulates these two processes. We analyzed accuracy and reaction times (RTs) of direct and indirect trials in both groups. We reasoned that the musical and cross-modal memory demands of musicianship might modulate performance in the task and might thus reveal mechanisms that underlie the association and integration of visual information with musical information. Although musicians showed a higher overall memory accuracy, non-musicians' accuracy was well above chance level in both trial types, thus indicating a significant ability to associate and integrate musical with visual information even in musically untrained subjects. However, non-musicians showed shorter RTs in indirect compared to direct trials, whereas the reverse pattern was found in musicians. Moreover, accuracy of direct and indirect trials correlated significantly in musicians but not in non-musicians. Consistent with previous accounts of visual associative memory, we interpret these findings as suggestive of at least two complimentary mechanisms that contribute to visual-melodic memory integration. (I) A default mechanism that mainly operates at encoding of complex visual-melodic associations and that works with surprising efficacy even in musically untrained subjects. (II) A retrieval-based mechanism that critically depends on an expert ability to maintain and discriminate visual-melodic associations across extended memory delays. Future studies may investigate how these mechanisms contribute to the everyday experience of music-evoked memories.

Cascaded normalizations for spatial integration in the primary visual cortex of primates.

Spatial integration of visual information is an important function in the brain. However, neural computation for spatial integration in the visual cortex remains unclear. In this study, we recorded laminar responses in V1 of awake monkeys driven by visual stimuli with grating patches and annuli of different sizes. We find three important response properties related to spatial integration that are significantly different between input and output layers: neurons in output layers have stronger surround suppression, smaller receptive field (RF), and higher sensitivity to grating annuli partially covering their RFs. These interlaminar differences canbe explained by a descriptive model composed of two global divisions (normalization) and a local subtraction. Our results suggest suppressions with cascaded normalizations (CNs) are essential for spatial integration and laminar processing in the visual cortex. Interestingly, the features of spatial integration in convolutional neural networks, especially in lower layers, are different from our findings in V1.

Use-dependent increase in attention to the prosthetic foot in patients with lower limb amputation

Patients with lower limb amputation experience “embodiment” while using a prosthesis, perceiving it as part of their body. Humans control their biological body parts and receive appropriate information by directing attention toward them, which is called body-specific attention. This study investigated whether patients with lower limb amputation similarly direct attention to prosthetic limbs. The participants were 11 patients with lower limb amputation who started training to walk with a prosthesis. Attention to the prosthetic foot was measured longitudinally by a visual detection task. In the initial stage of walking rehabilitation, the index of attention to the prosthetic foot was lower than that to the healthy foot. In the final stage, however, there was no significant difference between the two indexes of attention. Correlation analysis revealed that the longer the duration of prosthetic foot use, the greater the attention directed toward it. These findings indicate that using a prosthesis focuses attention akin to that of an individual’s biological limb. Moreover, they expressed that the prosthesis felt like a part of their body when they could walk independently. These findings suggest that the use of prostheses causes integration of visual information and movement about the prosthesis, resulting in its subjective embodiment.

Neural alterations of emotion processing in atypical trajectories of psychotic-like experiences

The aim of this study was to investigate the neural bases of facial emotion processing before the onset of clinical psychotic symptoms in youth belonging to well-defined developmental trajectories of psychotic-like experiences (PLEs). A unique sample of 86 youths was recruited from a population-based sample of over 3800 adolescents who had been followed from 13 to 17 years of age. Three groups were identified based on validated developmental trajectories: a control trajectory with low and decreasing PLEs, and two atypical trajectories with moderate to elevated baseline PLEs that subsequently decreased or increased. All had functional magnetic resonance imaging data collected during a facial emotion processing task. Functional activation and connectivity data were analyzed for different contrasts. The increasing PLE trajectory displayed more positive psychotic symptoms while the decreasing trajectory exhibited more negative symptoms relative to the control group. During face processing, both atypical trajectories displayed decreased activations of the right inferior frontal gyrus (IFG), while the increasing trajectory displayed a negative signal in the precentral gyrus. The increasing PLE trajectory also displayed impaired connectivity between the amygdala, ventromedial prefrontal cortex, and cerebellum, and between the IFG, precuneus, and temporal regions, while the decreasing trajectory exhibited reduced connectivity between the amygdala and visual regions during emotion processing. Both atypical PLE trajectories displayed alterations in brain regions involved in attention salience. While the increasing trajectory with more positive symptoms exhibited dysconnectivity in areas that influence emotion salience and face perception, the decreasing trajectory with more negative symptoms had impairments in visual information integration areas. These group-specific features might account for the differential symptom expression.

Recent Treatment Advances in Amblyopia.

Occlusion therapy has a long history as the gold standard treatment for amblyopia. Over the past two decades, large multicenter randomized controlled trials and objective dose-monitoring studies have characterized the effects of refractive correction, patching, and atropine penalization, providing insights into the impact of factors such as age and treatment dose. More recent approaches, whose development has been accelerated by advances in technology, are designed to provide different stimulation to the amblyopic eye and the fellow eye. This review explores a variety of such dichoptic approaches, categorized according to whether they primarily feature requisite use of the amblyopic eye in the face of fellow-eye masking, integration of visual information from both eyes, or reduction of stimulus salience in the fellow eye. It is still unclear whether dichoptic treatments are superior to traditional, low-cost treatment methods or whether their therapeutic mechanisms are fundamentally different from those of established treatments.

Graph-based linguistic and visual information integration for on-site occupational hazards identification

Construction sites are hazardous with various potential hazards that can occur at any time. The combination of different factors always causes the construction fatalities, and the majority of these fatalities could be prevented if workers followed on-site regulatory rules. However, compliance of regulatory rules is not strictly enforced among workers due to all kinds of reasons. Although previously proposed vision-based approaches are available for occupational hazards identification, the practicality is limited by the lack of automated understanding and adaptability to regulatory rules changes. In response to these gaps, this paper proposes a novel graph-based framework that integrates linguistic and visual information to process regulatory rule sentences and images for on-site occupational hazards identification. Particularly, a regulatory rules processing approach is presented to automatically extract and represent the key linguistic information of regulatory rules and a vision-based image scene information understanding approach is introduced to process on-site images by the combination of deep learning-based object detection and individual detection using geometric relationships analysis. Additionally, an automated reasoning approach is proposed to provide the integration of the processed linguistic and visual information and perform hazards identification. The hazards of two scenes, i.e., “working on height” and “operating a grinder”, were successfully identified with significantly higher performance compared to the baseline model.

Dynamic networks of P300-related process.

P300 as an effective biomarker to index attention and memory has been widely used for brain-computer interface, cognitive evaluation, and clinical diagnosis. To evoke clear P300, an oddball paradigm consisting of two types of stimuli, i.e., infrequent target stimuli and frequent standard stimuli, is usually used. However, to simply and quickly explore the P300-related process, previous studies predominately focused on the target condition but ignored the fusion of target and standard conditions, as well as the difference of brain networks between them. Therefore, in this study, we used the hidden Markov model to investigate the fused multi-conditional electroencephalogram dataset of P300, aiming to effectively identify the underlying brain networks and explore the difference between conditions. Specifically, the inferred networks, including their transition sequences and spatial distributions, were scrutinized first. Then, we found that the difference between target and standard conditions was mainly concentrated in two phases. One was the stimulation phase that mainly related to the cortical activities of the postcentral gyrus and superior parietal lobule, and the other corresponded to the response phase that involved the activities of superior and medial frontal gyri. This might be attributed to distinct cognitive functions, as the stimulation phase is associated with visual information integration whereas the response phase involves stimulus discrimination and behavior control. Taken together, the current work explored dynamic networks underlying the P300-related process and provided a complementary understanding of distinct P300 conditions, which may contribute to the design of P300-related brain-machine systems.

Nonadditive Integration of Visual Information in Ensemble Representation

Nonadditive Integration of Visual Information in Ensemble Representation

When You Do Not Get the Whole Picture: Scene Perception After Occipital Cortex Lesions.

Background: Occipital cortex lesions (OCLs) typically result in visual field defects (VFDs) contralateral to the damage. VFDs are usually mapped with perimetry involving the detection of point targets. This, however, ignores the important role of integration of visual information across locations in many tasks of everyday life. Here, we ask whether standard perimetry can fully characterize the consequences of OCLs. We compare performance on a rapid scene discrimination task of OCL participants and healthy observers with simulated VFDs. While the healthy observers will only suffer the loss of part of the visual scene, the damage in the OCL participants may further compromise global visual processing.Methods: VFDs were mapped with Humphrey perimetry, and participants performed two rapid scene discrimination tasks. In healthy participants, the VFDs were simulated with hemi- and quadrant occlusions. Additionally, the GIST model, a computational model of scene recognition, was used to make individual predictions based on the VFDs.Results: The GIST model was able to predict the performance of controls regarding the effects of the local occlusion. Using the individual predictions of the GIST model, we can determine that the variability between the OCL participants is much larger than the extent of the VFD could account for. The OCL participants can further be categorized as performing worse, the same, or better as their VFD would predict.Conclusions: While in healthy observers the extent of the simulated occlusion accounts for their performance loss, the OCL participants’ performance is not fully determined by the extent or shape of their VFD as measured with Humphrey perimetry. While some OCL participants are indeed only limited by the local occlusion of the scene, for others, the lesions compromised the visual network in a more global and disruptive way. Yet one outperformed a healthy observer, suggesting a possible adaptation to the VFD. Preliminary analysis of neuroimaging data suggests that damage to the lateral geniculate nucleus and corpus callosum might be associated with the larger disruption of rapid scene discrimination. We believe our approach offers a useful behavioral tool for investigating why similar VFDs can produce widely differing limitations in everyday life.

Effect of visual input on syllable parsing in a computational model of a neural microcircuit for speech processing

Objective. Seeing a person talking can help us understand them, particularly in a noisy environment. However, how the brain integrates the visual information with the auditory signal to enhance speech comprehension remains poorly understood. Approach. Here we address this question in a computational model of a cortical microcircuit for speech processing. The model consists of an excitatory and an inhibitory neural population that together create oscillations in the theta frequency range. When stimulated with speech, the theta rhythm becomes entrained to the onsets of syllables, such that the onsets can be inferred from the network activity. We investigate how well the obtained syllable parsing performs when different types of visual stimuli are added. In particular, we consider currents related to the rate of syllables as well as currents related to the mouth-opening area of the talking faces. Main results. We find that currents that target the excitatory neuronal population can influence speech comprehension, both boosting it or impeding it, depending on the temporal delay and on whether the currents are excitatory or inhibitory. In contrast, currents that act on the inhibitory neurons do not impact speech comprehension significantly. Significance. Our results suggest neural mechanisms for the integration of visual information with the acoustic information in speech and make experimentally-testable predictions.

Transsaccadic integration of visual information is predictive, attention-based, and spatially precise.

Eye movements produce shifts in the positions of objects in the retinal image, but observers are able to integrate these shifting retinal images into a coherent representation of visual space. This ability is thought to be mediated by attention-dependent saccade-related neural activity that is used by the visual system to anticipate the retinal consequences of impending eye movements. Previous investigations of the perceptual consequences of this predictive activity typically infer attentional allocation using indirect measures such as accuracy or reaction time. Here, we investigated the perceptual consequences of saccades using an objective measure of attentional allocation, reverse correlation. Human observers executed a saccade while monitoring a flickering target object flanked by flickering distractors and reported whether the average luminance of the target was lighter or darker than the background. Successful task performance required subjects to integrate visual information across the saccade. A reverse correlation analysis yielded a spatiotemporal “psychophysical kernel” characterizing how different parts of the stimulus contributed to the luminance decision throughout each trial. Just before the saccade, observers integrated luminance information from a distractor located at the post-saccadic retinal position of the target, indicating a predictive perceptual updating of the target. Observers did not integrate information from distractors placed in alternative locations, even when they were nearer to the target object. We also observed simultaneous predictive perceptual updating for two spatially distinct targets. These findings suggest both that shifting neural representations mediate the coherent representation of visual space, and that these shifts have significant consequences for transsaccadic perception.