Visual Information Research Articles

Multimodal In‐Sensor Computing Implemented by Easily‐Fabricated Oxide‐Heterojunction Optoelectronic Synapses

AbstractThe advent of big data and the Internet of Things has created urgent demands for in‐sensor computing hardware with multimodal perception that can effectively resolve the inefficiency, high latency, and excessive energy consumption challenges faced by conventional sensory systems. Here, a simple‐structured optoelectronic synaptic device of In2O3·SnO2/Nb:SrTiO3 (ITO/NSTO) heterostructure is proposed, which vividly demonstrates in‐sensor computing and multimodal perception capabilities. First, with the ingenious synaptic responses of the device under both optical and electrical stimuli, a multimodal in‐sensor neuromorphic computing system capable of concurrently perceiving and processing visual and auditory information is constructed. Using this multimodal system to perform a human emotion recognition task, the misjudgment arising from single‐modal cognition can thereby be effectively avoided. Second, utilizing the integrated sensing and processing functions, along with the dynamic memories of the device array, a neuromorphic vision system is implemented for real‐time monitoring of moving vehicles, which displays high recognition efficiency and accuracy. This research not only provides an optoelectronic synaptic device that is low‐cost and easy to mass‐produce, but also paves the way for next‐generation in‐sensor computing that can efficiently process multimodal signals.

The shadow drama

Abstract This study examines how discursive polarization between majority populations and so-called non-Western immigrant identities is enabled via verbal and visual metaphors in outputs by the Human Rights Service (HRS), a prominent Norwegian extreme-right media outlet. Focusing especially on the HRS’s use of visual primary metaphors of cold and darkness, a contribution is made to the existing literature regarding how right-wing outlets construct an image of immigrants and Muslims as threatening Others. As such, the potential polarizing outcomes of the HRS’s visual primary metaphors are theorized to arise from a capacity to invite certain forms of embodied cognition and implicitly associate the target identities with a range of negative emotions. Ultimately, the HRS’s visual primary metaphors of cold and darkness are best understood as polarization vehicles that tacitly support anti-social biases by leveraging the rapidity and efficiency with which subjects can respond emotionally to visual information—especially fear triggers.

A Novel Multi-modal Sentiment Analysis Based on Multiple Kernel Learning with Margin-Dimension Constraint

Most of the sentiment analysis studies focus on the sentimental classification of pictures in the video, ignoring the spatio-temporal information of the sequence of picture frames as well as text and audio information. The multiple kernel learning is a new hotspot in the field of nuclear machine learning, capable of handling multiple modalities. For multiple kernel learning, it is easy to ignore the basic features that are not discriminative, and cannot make full use of the base features of different modes. This paper puts forward a novel multi-modal fusion model for sentiment analysis, in which a multiple kernel learning algorithm based on convolution margin-dimension constraint is proposed for feature fusion. Moreover, the 3D convolutional neural network is used to extract the features of visual information, and the multiple kernel learning algorithm based on margin-dimension constraint is used to fuse visual, text and audio sentiment features. Experiments conducted on the MOUD and IEMOCAP sentiment databases show that the proposed model outperforms existing models in the field of multi-modal sentiment analysis research.

Human Visual Pathways for Action Recognition Versus Deep Convolutional Neural Networks: Representation Correspondence in Late But Not Early Layers.

Deep convolutional neural networks (DCNNs) have attained human-level performance for object categorization and exhibited representation alignment between network layers and brain regions. Does such representation alignment naturally extend to other visual tasks beyond recognizing objects in static images? In this study, we expanded the exploration to the recognition of human actions from videos and assessed the representation capabilities and alignment of two-stream DCNNs in comparison with brain regions situated along ventral and dorsal pathways. Using decoding analysis and representational similarity analysis, we show that DCNN models do not show hierarchical representation alignment to human brain across visual regions when processing action videos. Instead, later layers of DCNN models demonstrate greater representation similarities to the human visual cortex. These findings were revealed for two display formats: photorealistic avatars with full-body information and simplified stimuli in the point-light display. The discrepancies in representation alignment suggest fundamental differences in how DCNNs and the human brain represent dynamic visual information related to actions.

Neuropsychological features of somatic symptom disorder and depression/anxiety in Taiwan: An analysis based on the comorbidity status

Individuals with somatic symptom disorder (SSD) often have comorbid depression or anxiety, but whether SSD is associated with specific neuropsychological functions has yet to be fully examined. We analyzed which neuropsychological features are more closely associated with SSD, anxiety, and depression. In this case-control study, we recruited 140 individuals with SSD, 104 individuals with affective disorders without SSD, and 159 healthy controls in Taiwan. We collected DSM-5 diagnoses, questionnaire scores, and performance on eight tasks from the Cambridge Neuropsychological Test Automated Battery (CANTAB) for each participant. Several CANTAB tasks involving attention, executive function, and social cognition showed significant group differences. In the adjusted analysis, the tasks significantly associated with SSD were the Match to Sample Visual Search (MTS) and the Emotion Recognition Task (ERT). Among the questionnaires, the Cognitions about Body and Health Questionnaire showed the most significant associations with the tasks, specifically with Rapid Visual Information Processing, MTS, Paired Associates Learning, Spatial Working Memory, Intra-Extra Dimensional Set Shift, and ERT. We conclude that the MTS and ERT tasks show significant relationships with both SSD diagnosis and related questionnaires. These tasks primarily involve selective attention and negative emotion regulation.

We May Be Visually or Hearing Impaired, but We Need Access to Visual and Auditory Information

This study explored access to information by visually and hearing-impaired patrons at the selected public libraries in Drakenstein Municipality: Paarl, Drakenstein, and Groenheuwel. The three Drakenstein Municipality public libraries were selected because they collaborate with the South African Library for the Blind (SALB) mini-library project to provide information services for the visually and hearing-impaired patrons. Participants were selected through purposive and snowball sampling within targeted public libraries and Drakenstein communities, respectively. Qualitative data were collected through interviews with visually and hearing-impaired patrons, SALB project coordinators, sign language interpreters, library assistants, and librarians responsible for providing service to visually and hearing-impaired patrons. Document analysis and observation were also used to supplement the data collected through interviews. The data were analyzed using content and thematic analysis. The study revealed that none of the three libraries had a specific policy or set of guidelines in place to meet the information needs of visually- and hearing-impaired patrons. Furthermore, the study established that the visually impaired have limited access to braille resources and audiobook collections. It is recommended that the three libraries develop a policy addressing information access issues for visually and hearing-impaired patrons.

Integrating vision and somatosensation does not improve the accuracy and response time when estimating area and perimeter of rectangles in primary school

BackgroundProblem-solving and learning in mathematics involves sensory perception and processing. Multisensory integration may contribute by enhancing sensory estimates. This study aims to assess if combining visual and somatosensory information improves elementary students' perimeter and area estimates. Methods87 4th graders compared rectangles with respect to area or perimeter either solely using visual observation or additionally with somatosensory information. Three experiments targeted different task aspects. Statistical analyses tested success rates and response times. ResultsContrary to expectations, adding somatosensory information did not boost success rates for area and perimeter comparison. Response time even increased with adding somatosensory information. Children's difficulty in accurately tracing figures negatively impacted the success rate of area comparisons. DiscussionResults suggest visual observation alone suffices for accurately estimating and comparing area and perimeter of rectangles in 4th graders. ImplicationsCareful deliberation on the inclusion of somatosensory information in mathematical tasks concerning perimeter and area estimations of rectangles is recommended.

Multimodal Machine Translation Based on Enhanced Knowledge Distillation and Feature Fusion

Existing research on multimodal machine translation (MMT) has typically enhanced bilingual translation by introducing additional alignment visual information. However, picture form requirements in multimodal datasets pose important constraints on the development of MMT because this requires a form of alignment between image, source text, and target text. This limitation is especially compounded by the fact that the inference phase, when aligning images, is not directly available in a conventional neural machine translation (NMT) setup. Therefore, we propose an innovative MMT framework called the DSKP-MMT model, which supports machine translation by enhancing knowledge distillation and feature refinement methods in the absence of images. Our model first generates multimodal features from the source text. Then, the purified features are obtained through the multimodal feature generator and knowledge distillation module. The features generated through image feature enhancement are subsequently further purified. Finally, the image–text fusion features are generated as input in the transformer-based machine translation reasoning task. In the Multi30K dataset test, the DSKP-MMT model has achieved a BLEU of 40.42 and a METEOR of 58.15, showing its ability to improve translation effectiveness and facilitating utterance communication.

Reinforcing Visual Content Integrity through Image Restoration and AI Recognition

A number of factors can cause photographs to deteriorate over time, losing their clarity and aesthetic appeal. The goal of this research is to help restore priceless memories because images act as memory triggers and better image quality can increase clarity and visual comfort. This research is important because it may aid in the recovery and preservation of special memories that were captured in damaged photos. Restoring an image's visual quality has the power to arouse feelings and create closer ties to the past.AI has made great progress in several areas in the recent past. However, questions concerning the veracity and authenticity of visual content have been raised due to the possibility of false information and the spread of deepfakes. We set out to create an automated system that could accurately distinguish between photographs taken by hand and images created by artificial intelligence as a solution to this problem. Furthermore, there are some useful advantages to improving image quality, including better vitality, higher clarity, and less eye strain when viewing the restored photos. All things considered, this study shows how deep learning using GANs works well for image restoration, emphasising its usefulness in retrieving priceless memories and improving visual content for wider uses. The findings of this study demonstrate the potential benefits that AI-driven image restoration methods could have on our daily interactions with visual information in the digital age as well as our personal life.

Decoding dynamic visual scenes across the brain hierarchy.

Understanding the computational mechanisms that underlie the encoding and decoding of environmental stimuli is a crucial investigation in neuroscience. Central to this pursuit is the exploration of how the brain represents visual information across its hierarchical architecture. A prominent challenge resides in discerning the neural underpinnings of the processing of dynamic natural visual scenes. Although considerable research efforts have been made to characterize individual components of the visual pathway, a systematic understanding of the distinctive neural coding associated with visual stimuli, as they traverse this hierarchical landscape, remains elusive. In this study, we leverage the comprehensive Allen Visual Coding-Neuropixels dataset and utilize the capabilities of deep learning neural network models to study neural coding in response to dynamic natural visual scenes across an expansive array of brain regions. Our study reveals that our decoding model adeptly deciphers visual scenes from neural spiking patterns exhibited within each distinct brain area. A compelling observation arises from the comparative analysis of decoding performances, which manifests as a notable encoding proficiency within the visual cortex and subcortical nuclei, in contrast to a relatively reduced encoding activity within hippocampal neurons. Strikingly, our results unveil a robust correlation between our decoding metrics and well-established anatomical and functional hierarchy indexes. These findings corroborate existing knowledge in visual coding related to artificial visual stimuli and illuminate the functional role of these deeper brain regions using dynamic stimuli. Consequently, our results suggest a novel perspective on the utility of decoding neural network models as a metric for quantifying the encoding quality of dynamic natural visual scenes represented by neural responses, thereby advancing our comprehension of visual coding within the complex hierarchy of the brain.

Automatic driving image matching via Random Sample Consensus (RANSAC) and Spectral Clustering (SC) with monocular camera.

In today's big data era, with the development of the Internet of Things (IoT) technology and the trend of autonomous driving prevailing, visual information has shown a blowout increase, but most image matching algorithms have problems such as low accuracy and low inlier rates, resulting in insufficient information. In order to solve the problem of low image matching accuracy and low inlier rate in the field of autonomous driving, this research innovatively applies spectral clustering (SC) in the field of data analysis to image matching in the field of autonomous driving, and a new image matching algorithm "SC-RANSAC" based on SC and Random Sample Consensus (RANSAC) is proposed. The datasets in this research are collected based on the monocular cameras of autonomous driving cars. We use RANSAC to obtain the initial inlier set and the SC algorithm to filter RANSAC's outliers and then use the filtered inliers as the final inlier set. In order to verify the effectiveness of the algorithm, it shows the matching effect from three angles: camera translation, rotation, and rotation and translation. SC-RANSAC is also compared with RANSAC, graph-cut RANSAC, and marginalizing sample consensus by using two different types of datasets. Finally, we select three representative pictures to test the robustness of the SC-RANSAC algorithm. The experimental results show that SC-RANSAC can effectively and reliably eliminate mismatches in the initial matching results; has a high inlier rate, real-time performance, and robustness; and can be effectively applied in the environment of autonomous driving.

Sub-bundle based analysis reveals the role of human optic radiation in visual working memory.

White matter (WM) functional activity has been reliably detected through functional magnetic resonance imaging (fMRI). Previous studies have primarily examined WM bundles as unified entities, thereby obscuring the functional heterogeneity inherent within these bundles. Here, for the first time, we investigate the function of sub-bundles of a prototypical visual WM tract-the optic radiation (OR). We use the 7T retinotopy dataset from the Human Connectome Project (HCP) to reconstruct OR and further subdivide the OR into sub-bundles based on the fiber's termination in the primary visual cortex (V1). The population receptive field (pRF) model is then applied to evaluate the retinotopic properties of these sub-bundles, and the consistency of the pRF properties of sub-bundles with those of V1 subfields is evaluated. Furthermore, we utilize the HCP working memory dataset to evaluate the activations of the foveal and peripheral OR sub-bundles, along with LGN and V1 subfields, during 0-back and 2-back tasks. We then evaluate differences in 2bk-0bk contrast between foveal and peripheral sub-bundles (or subfields), and further examine potential relationships between 2bk-0bk contrast and 2-back task d-prime. The results show that the pRF properties of OR sub-bundles exhibit standard retinotopic properties and are typically similar to the properties of V1 subfields. Notably, activations during the 2-back task consistently surpass those under the 0-back task across foveal and peripheral OR sub-bundles, as well as LGN and V1 subfields. The foveal V1 displays significantly higher 2bk-0bk contrast than peripheral V1. The 2-back task d-prime shows strong correlations with 2bk-0bk contrast for foveal and peripheral OR fibers. These findings demonstrate that the blood oxygen level-dependent (BOLD) signals of OR sub-bundles encode high-fidelity visual information, underscoring the feasibility of assessing WM functional activity at the sub-bundle level. Additionally, the study highlights the role of OR in the top-down processes of visual working memory beyond the bottom-up processes for visual information transmission. Conclusively, this study innovatively proposes a novel paradigm for analyzing WM fiber tracts at the individual sub-bundle level and expands understanding of OR function.

Audiovisual integration and sensory dominance effects in older adults with subjective cognitive decline: Enhanced redundant effects and stronger fusion illusion susceptibility.

Subjective cognitive decline (SCD) refers to individuals' perceived decline in memory and/or other cognitive abilities relative to their previous level of performance. Sensory decline is one of the main manifestations of decline in older adults with SCD. The efficient integration of visual and auditory information, known as audiovisual integration, is a crucial perceptual process. This study aims to evaluate audiovisual integration in older adults with SCD. We adopted the audiovisual detection task, the Colavita task, and the Sound-Induced Flash Illusion (SIFI) task to evaluate the audiovisual integration by examining both redundant and illusory effects. Older adults diagnosed with SCD (N=50, mean age=67.8 years) and a control group of non-SCD older adults (N=51, mean age=66.5 years) were recruited. All participants took part in the three aforementioned experiments. The outcomes showed that a redundant effect occurred in both SCD and non-SCD older adults, with SCD older adults gaining more benefits in audiovisual detection task. Moreover, an equivalent amount of the visual dominance effect was observed among both SCD and non-SCD older adults in Colavita task. In addition, older adults with SCD perceived an equal fission illusion but a bigger fusion illusion compared with non-SCD older adults in SIFI task. Overall, older adults with SCD exhibit increased audiovisual redundant effects and stronger fusion illusion susceptibility compared to non-SCD older adults. Besides, visual dominance was observed in both groups via the Colavita task, with no significant difference between non-SCD and SCD older adults. These findings implied that audiovisual integration might offer a potential way for the identification of SCD.

The role of general cognitive skills in integrating visual and linguistic information during sentence comprehension: individual differences across the lifespan

Individuals exhibit massive variability in general cognitive skills that affect language processing. This variability is partly developmental. Here, we recruited a large sample of participants (N = 487), ranging from 9 to 90 years of age, and examined the involvement of nonverbal processing speed (assessed using visual and auditory reaction time tasks) and working memory (assessed using forward and backward Digit Span tasks) in a visual world task. Participants saw two objects on the screen and heard a sentence that referred to one of them. In half of the sentences, the target object could be predicted based on verb-selectional restrictions. We observed evidence for anticipatory processing on predictable compared to non-predictable trials. Visual and auditory processing speed had main effects on sentence comprehension and facilitated predictive processing, as evidenced by an interaction. We observed only weak evidence for the involvement of working memory in predictive sentence comprehension. Age had a nonlinear main effect (younger adults responded faster than children and older adults), but it did not differentially modulate predictive and non-predictive processing, nor did it modulate the involvement of processing speed and working memory. Our results contribute to delineating the cognitive skills that are involved in language-vision interactions.

Use of Interactive Food Labels to Increase Confidence in Preparing Produce among College Students.

College students may have limited access to produce and may lack confidence in preparing it, but cooking videos can show how to make healthy dishes. The Cognitive Theory of Multimedia Learning suggests that learning is enhanced when visual and auditory information is presented considering cognitive load (e.g., highlighting important concepts, eliminating extraneous information, and keeping the video brief and conversational). The purpose of this project was to pilot test a food label for produce grown at an urban university and assess whether student confidence in preparing produce improved after using the label and QR code to view a recipe video developed using principles from the Cognitive Theory of Multimedia Learning. The video showed a student preparing a salad with ingredients available on campus. Students indicated the label was helpful and reported greater perceived confidence in preparing lettuce after viewing the label and video (mean confidence of 5.60 ± 1.40 before vs. 6.14 ± 0.89 after, p = 0.016, n = 28). Keeping the video short and providing ingredients and amounts onscreen as text were cited as helpful. Thus, a brief cooking video and interactive label may improve confidence in preparing produce available on campus. Future work should determine whether the label impacts produce consumption and if it varies depending on the type of produce used.

Morphological Characteristics of Retinal Ganglion Cells in the Retinas of Giant Pandas (Ailuropoda melanoleuca).

Vision plays a crucial role in the survival of animals, and the visual system has particularly selectively evolved in response to the visual environment, ecological niche, and species habitats in vertebrate species. To date, a horizontal streak of retinal ganglion cell (RGC) distribution pattern is observed across mammal species. Here, we report that the giant panda's vertically oriented visual streak, combined with current evidence of the animal's forward-placed eyes, ocular structure, and retinal neural topographic distribution patterns, presents the emergence of a well-adapted binocular visual system. Our results suggest that the giant panda may use a unique way to processing binocular visual information. Results of mathematical simulation are in favor of this hypothesis. The topographic distribution properties of RGCs reported here could be essential for understanding the visual adaptation and evolution of this living fossil.

Deep autoencoder based image enhancement approach with hybrid feature extraction for plant disease detection using supervised classification

Plant leaf diseases pose significant threats to global agriculture, leading to reduced crop yields and economic losses. Rapid and accurate disease detection is essential for timely interventions and sustainable farming practices. This study presents an innovative approach for plant leaf disease detection by integrating wavelet analysis, color, and texture features, coupled with autoencoder denoising and support vector machine (SVM) classification. Wavelet analysis is employed to extract multi-resolution features, capturing intricate details at different scales. Furthermore, color and texture characteristics are extracted to encompass a broad spectrum of visual information crucial for distinguishing diseases. The Autoencoder model helps to enhance the feature representation that mitigates the impact of noise and irrelevant data. The SVM classifier is utilized to learn complex patterns and accurately classify different disease classes. The combined model of wavelet, color, and texture attributes, in combination with autoencoder denoising and SVM classification, markedly enhances the precision and efficiency of disease detection in contrast to conventional methods. The system's performance is evaluated using a PlantVillage dataset, showcasing its adaptability to different plant species and disease types. The overall performance is obtained as 98.60%, 97.25%, 96.89%, and 97.20% in terms of accuracy, precision, recall, and F-Score, respectively.

Saccades and Microsaccades Coupling During Free-Throw Shots in Basketball Players.

We investigated the role of saccades and microsaccades when different levels of basketball players were engaged in an ecological free-throw condition. All participants made more correct than incorrect shoots, with a movement time initiation shorter in amateurs than in near-expert groups. Near-experts had more stable gaze fixation than amateurs, with higher microsaccade rate and duration and lower peak velocity. Amateurs showed higher saccade rate, peak velocity, and amplitude than near-experts. The temporal sequence of near-experts' microsaccade rate increased after the saccade peak; on the contrary, in amateurs, the saccade peak is shown after the decrement in microsaccade rates. The spatiotemporal characteristics of microsaccades and saccades may reflect an optimal sampling method by which the brain discretely acquires visual information and can differentiate between participants who use a fixation before the critical movement time and participants who move their eyes to catch more visual cues to make decisions.

Weakly supervised low-dose computed tomography denoising based on generative adversarial networks.

Low-dose computed tomography (LDCT) is a diagnostic imaging technique designed to minimize radiation exposure to the patient. However, this reduction in radiation may compromise computed tomography (CT) image quality, adversely impacting clinical diagnoses. Various advanced LDCT methods have emerged to mitigate this challenge, relying on well-matched LDCT and normal-dose CT (NDCT) image pairs for training. Nevertheless, these methods often face difficulties in distinguishing image details from nonuniformly distributed noise, limiting their denoising efficacy. Additionally, acquiring suitably paired datasets in the medical domain poses challenges, further constraining their applicability. Hence, the objective of this study was to develop an innovative denoising framework for LDCT images employing unpaired data. In this paper, we propose a LDCT denoising network (DNCNN) that alleviates the need for aligning LDCT and NDCT images. Our approach employs generative adversarial networks (GANs) to learn and model the noise present in LDCT images, establishing a mapping from the pseudo-LDCT to the actual NDCT domain without the need for paired CT images. Within the domain of weakly supervised methods, our proposed model exhibited superior objective metrics on the simulated dataset when compared to CycleGAN and selective kernel-based cycle-consistent GAN (SKFCycleGAN): the peak signal-to-noise ratio (PSNR) was 43.9441, the structural similarity index measure (SSIM) was 0.9660, and the visual information fidelity (VIF) was 0.7707. In the clinical dataset, we conducted a visual effect analysis by observing various tissues through different observation windows. Our proposed method achieved a no-reference structural sharpness (NRSS) value of 0.6171, which was closest to that of the NDCT images (NRSS =0.6049), demonstrating its superiority over other denoising techniques in preserving details, maintaining structural integrity, and enhancing edge contrast. Through extensive experiments on both simulated and clinical datasets, we demonstrated the superior efficacy of our proposed method in terms of denoising quality and quantity. Our method exhibits superiority over both supervised techniques, including block-matching and 3D filtering (BM3D), residual encoder-decoder convolutional neural network (RED-CNN), and Wasserstein generative adversarial network-VGG (WGAN-VGG), and over weakly supervised approaches, including CycleGAN and SKFCycleGAN.

Ordered spatial configuration protects representations of dissimilar items and reduces the similarity effect in visual working memory.

Although similarity could improve visual working memory (VWM) performance, it remains unclear how the spatial configuration of visual information influences the similarity effect in VWM. We explored this question by manipulating the orderliness of spatial configuration (ordered vs. scrambled) in the simultaneous (Experiment 1) and sequential (Experiment 2) change detection tasks. The results showed that similarity improved VWM performance when memory items were presented simultaneously and sequentially. For the simultaneous memory array containing similar and dissimilar items, the performance of the ordered spatial configuration was better than that of the scrambled spatial configuration when probing dissimilar items, while no such difference was found when probing similar items. Further, the similarity effect value in the scrambled spatial configuration was higher than that in the ordered spatial configuration. For the sequential memory array containing similar and dissimilar items, spatial configuration did not affect the similarity effect in VWM. Taken together, these findings suggest that spatial configuration could modulate the similarity effect when memory items are presented simultaneously, in which the ordered spatial configuration protects representations of dissimilar items and reduces the similarity effect in VWM. Our study provides additional evidence for the role of spatial configuration in the similarity effect in VWM, and supports the hierarchical model.