Visual Information Research Articles

P-CNN: Percept-CNN for semantic segmentation

ABSTRACT The task of image segmentation remains a fundamental challenge, in the field of computer vision. Convolutional Neural Networks (CNNs) have achieved significant success in this field, yet there are some limitations in the conventional approach. The process of accurate, pixel-wise image annotation is time-consuming, as well as requires more human effort. These problems are addressed by the proposed method called as Percept-CNN (P-CNN), which extracts the power of percepts, pixels responsible for the higher activations at each level. From each layer, percepts are extracted during the forward propagation. These percepts are then passed onto the subsequent layers, enabling the model to focus only on the useful visual information. The proposed method with Percept Convolution can potentially eliminate the complex and time-consuming task of image annotation without affecting the segmentation accuracy. Since the model focuses only on the useful salient visual information, it tends to reduce the extraction of the redundant features, which doesn’t really contribute towards the final goal. This makes the model to be more robust, accurate and efficient. The proposed model was able to perform semantic segmentation without pixelwise annotations with an accuracy of 67% when tested on Oxford IIIT pet dataset.

Superalloys fracture process inference based on overlap analysis of 3D models

Superalloy materials exhibit susceptibility to fracture failures stemming from the influence of thermomechanical factors. To comprehensively understand the fracture mechanisms, material properties, root causes of failure, and the subsequent optimization of alloys, a detailed analysis of the internal fracture process and the morphological traits of the fracture surface is imperative. Traditional analysis of fracture surfaces solely relies on 2D images, thus lacking crucial 3D information. Although in situ experiments can capture the fracture process, their effectiveness is confined to the specimen’s surface, precluding insight into internal changes. Here we introduce an integrated framework encompassing the process of 3D reconstruction of fracture surfaces, aiming to enhance the visual information obtained with micron-level accuracy, visual intuitiveness and sense of depth. Additionally, this framework also facilitates the scrutiny and inference of internal fracture processes. These results demonstrate that under specific service conditions, material deformation fracture probably stems from a combination of surface cracking and internal cracking rather than exclusively one or the other. Overall, our description and analysis of internally initiated cracking due to defects within the specimens can be beneficial in guiding future alloy design and optimization efforts.

Two "What" Networks in the Human Brain.

Ungerleider and Mishkin, in their influential work that relied on detailed anatomical and ablation studies, suggested that visual information is processed along two distinct pathways: the dorsal "where" pathway, primarily responsible for spatial vision, and the ventral "what" pathway, dedicated to object vision. This strict division of labor has faced challenges in light of compelling evidence revealing robust shape and object selectivity within the putative "where" pathway. This article reviews evidence that supports the presence of shape selectivity in the dorsal pathway. A comparative examination of dorsal and ventral object representations in terms of invariance, task dependency, and representational content reveals similarities and differences between the two pathways. Both exhibit some level of tolerance to image transformations and are influenced by tasks, but responses in the dorsal pathway show weaker tolerance and stronger task modulations than those in the ventral pathway. Furthermore, an examination of their representational content highlights a divergence between the responses in the two pathways, suggesting that they are sensitive to distinct features of objects. Collectively, these findings suggest that two networks exist in the human brain for processing object shapes, one in the dorsal and another in the ventral visual cortex. These studies lay the foundation for future research aimed at revealing the precise roles the two "what" networks play in our ability to understand and interact with objects.

A comparison of balance between real and virtual environments: differences, role of visual cues and full-body avatars, a quasi-experimental clinical study

Virtual rehabilitation using Virtual Reality (VR) technology is a promising novel approach to rehabilitation. However, postural responses in VR differ significantly from real life. The introduction of an avatar or visual cues in VR could help rectify this difference. An initial session was used to assess static and dynamic balance performances between VR and real life to set the reference values. A second session involved three VR conditions applied in a randomised order: i.e. full-body avatar, enhanced visual cues, or a combination of both conditions. Performances of the centre of pressure (COP) were recorded on a force plate. Seventy (70) people took part in the first session and 74 in the second. During the first session, a significant difference was observed in left static, right static and right dynamic COP distance (respectively SMD = − 0.40 [− 0.73, − 0.06], p = 0.02, − 0.33 [− 0.67, 0.00], p = 0.05, SMD = − 0.61 [− 0.95, − 0.27], p < 0.001) and a non-significant difference in the left dynamic, SMD = − 0.22 [− 0.56, 0.11], p = 0.19). During the second session it was observed that this difference was corrected mainly by reinforced visual information and to a lesser extent by the presence of a full-body avatar. Balance disruption triggered by the use of virtual reality can be offset by vertical visual information and/or by the presence of a full-body avatar. Further research is required on the effects of a full-body avatar.

GAT-Based Bi-CARU with Adaptive Feature-Based Transformation for Video Summarisation

Nowadays, video is a common social media in our lives. Video summarisation has become an interesting task for information extraction, where the challenge of high redundancy of key scenes leads to difficulties in retrieving important messages. To address this challenge, this work presents a novel approach called the Graph Attention (GAT)-based bi-directional content-adaptive recurrent unit model for video summarisation. The model makes use of the graph attention approach to transform the visual features of interesting scene(s) from a video. This transformation is achieved by a mechanism called Adaptive Feature-based Transformation (AFT), which extracts the visual features and elevates them to a higher-level representation. We also introduce a new GAT-based attention model that extracts major features from weight features for information extraction, taking into account the tendency of humans to pay attention to transformations and moving objects. Additionally, we integrate the higher-level visual features obtained from the attention layer with the semantic features processed by Bi-CARU. By combining both visual and semantic information, the proposed work enhances the accuracy of key-scene determination. By addressing the issue of high redundancy among major information and using advanced techniques, our method provides a competitive and efficient way to summarise videos. Experimental results show that our approach outperforms existing state-of-the-art methods in video summarisation.

Multimodal In‐Sensor Computing Implemented by Easily‐Fabricated Oxide‐Heterojunction Optoelectronic Synapses

AbstractThe advent of big data and the Internet of Things has created urgent demands for in‐sensor computing hardware with multimodal perception that can effectively resolve the inefficiency, high latency, and excessive energy consumption challenges faced by conventional sensory systems. Here, a simple‐structured optoelectronic synaptic device of In2O3·SnO2/Nb:SrTiO3 (ITO/NSTO) heterostructure is proposed, which vividly demonstrates in‐sensor computing and multimodal perception capabilities. First, with the ingenious synaptic responses of the device under both optical and electrical stimuli, a multimodal in‐sensor neuromorphic computing system capable of concurrently perceiving and processing visual and auditory information is constructed. Using this multimodal system to perform a human emotion recognition task, the misjudgment arising from single‐modal cognition can thereby be effectively avoided. Second, utilizing the integrated sensing and processing functions, along with the dynamic memories of the device array, a neuromorphic vision system is implemented for real‐time monitoring of moving vehicles, which displays high recognition efficiency and accuracy. This research not only provides an optoelectronic synaptic device that is low‐cost and easy to mass‐produce, but also paves the way for next‐generation in‐sensor computing that can efficiently process multimodal signals.

The shadow drama

Abstract This study examines how discursive polarization between majority populations and so-called non-Western immigrant identities is enabled via verbal and visual metaphors in outputs by the Human Rights Service (HRS), a prominent Norwegian extreme-right media outlet. Focusing especially on the HRS’s use of visual primary metaphors of cold and darkness, a contribution is made to the existing literature regarding how right-wing outlets construct an image of immigrants and Muslims as threatening Others. As such, the potential polarizing outcomes of the HRS’s visual primary metaphors are theorized to arise from a capacity to invite certain forms of embodied cognition and implicitly associate the target identities with a range of negative emotions. Ultimately, the HRS’s visual primary metaphors of cold and darkness are best understood as polarization vehicles that tacitly support anti-social biases by leveraging the rapidity and efficiency with which subjects can respond emotionally to visual information—especially fear triggers.

A Novel Multi-modal Sentiment Analysis Based on Multiple Kernel Learning with Margin-Dimension Constraint

Most of the sentiment analysis studies focus on the sentimental classification of pictures in the video, ignoring the spatio-temporal information of the sequence of picture frames as well as text and audio information. The multiple kernel learning is a new hotspot in the field of nuclear machine learning, capable of handling multiple modalities. For multiple kernel learning, it is easy to ignore the basic features that are not discriminative, and cannot make full use of the base features of different modes. This paper puts forward a novel multi-modal fusion model for sentiment analysis, in which a multiple kernel learning algorithm based on convolution margin-dimension constraint is proposed for feature fusion. Moreover, the 3D convolutional neural network is used to extract the features of visual information, and the multiple kernel learning algorithm based on margin-dimension constraint is used to fuse visual, text and audio sentiment features. Experiments conducted on the MOUD and IEMOCAP sentiment databases show that the proposed model outperforms existing models in the field of multi-modal sentiment analysis research.

Human Visual Pathways for Action Recognition Versus Deep Convolutional Neural Networks: Representation Correspondence in Late But Not Early Layers.

Deep convolutional neural networks (DCNNs) have attained human-level performance for object categorization and exhibited representation alignment between network layers and brain regions. Does such representation alignment naturally extend to other visual tasks beyond recognizing objects in static images? In this study, we expanded the exploration to the recognition of human actions from videos and assessed the representation capabilities and alignment of two-stream DCNNs in comparison with brain regions situated along ventral and dorsal pathways. Using decoding analysis and representational similarity analysis, we show that DCNN models do not show hierarchical representation alignment to human brain across visual regions when processing action videos. Instead, later layers of DCNN models demonstrate greater representation similarities to the human visual cortex. These findings were revealed for two display formats: photorealistic avatars with full-body information and simplified stimuli in the point-light display. The discrepancies in representation alignment suggest fundamental differences in how DCNNs and the human brain represent dynamic visual information related to actions.

Neuropsychological features of somatic symptom disorder and depression/anxiety in Taiwan: An analysis based on the comorbidity status

Individuals with somatic symptom disorder (SSD) often have comorbid depression or anxiety, but whether SSD is associated with specific neuropsychological functions has yet to be fully examined. We analyzed which neuropsychological features are more closely associated with SSD, anxiety, and depression. In this case-control study, we recruited 140 individuals with SSD, 104 individuals with affective disorders without SSD, and 159 healthy controls in Taiwan. We collected DSM-5 diagnoses, questionnaire scores, and performance on eight tasks from the Cambridge Neuropsychological Test Automated Battery (CANTAB) for each participant. Several CANTAB tasks involving attention, executive function, and social cognition showed significant group differences. In the adjusted analysis, the tasks significantly associated with SSD were the Match to Sample Visual Search (MTS) and the Emotion Recognition Task (ERT). Among the questionnaires, the Cognitions about Body and Health Questionnaire showed the most significant associations with the tasks, specifically with Rapid Visual Information Processing, MTS, Paired Associates Learning, Spatial Working Memory, Intra-Extra Dimensional Set Shift, and ERT. We conclude that the MTS and ERT tasks show significant relationships with both SSD diagnosis and related questionnaires. These tasks primarily involve selective attention and negative emotion regulation.

We May Be Visually or Hearing Impaired, but We Need Access to Visual and Auditory Information

This study explored access to information by visually and hearing-impaired patrons at the selected public libraries in Drakenstein Municipality: Paarl, Drakenstein, and Groenheuwel. The three Drakenstein Municipality public libraries were selected because they collaborate with the South African Library for the Blind (SALB) mini-library project to provide information services for the visually and hearing-impaired patrons. Participants were selected through purposive and snowball sampling within targeted public libraries and Drakenstein communities, respectively. Qualitative data were collected through interviews with visually and hearing-impaired patrons, SALB project coordinators, sign language interpreters, library assistants, and librarians responsible for providing service to visually and hearing-impaired patrons. Document analysis and observation were also used to supplement the data collected through interviews. The data were analyzed using content and thematic analysis. The study revealed that none of the three libraries had a specific policy or set of guidelines in place to meet the information needs of visually- and hearing-impaired patrons. Furthermore, the study established that the visually impaired have limited access to braille resources and audiobook collections. It is recommended that the three libraries develop a policy addressing information access issues for visually and hearing-impaired patrons.

Multimodal Machine Translation Based on Enhanced Knowledge Distillation and Feature Fusion

Existing research on multimodal machine translation (MMT) has typically enhanced bilingual translation by introducing additional alignment visual information. However, picture form requirements in multimodal datasets pose important constraints on the development of MMT because this requires a form of alignment between image, source text, and target text. This limitation is especially compounded by the fact that the inference phase, when aligning images, is not directly available in a conventional neural machine translation (NMT) setup. Therefore, we propose an innovative MMT framework called the DSKP-MMT model, which supports machine translation by enhancing knowledge distillation and feature refinement methods in the absence of images. Our model first generates multimodal features from the source text. Then, the purified features are obtained through the multimodal feature generator and knowledge distillation module. The features generated through image feature enhancement are subsequently further purified. Finally, the image–text fusion features are generated as input in the transformer-based machine translation reasoning task. In the Multi30K dataset test, the DSKP-MMT model has achieved a BLEU of 40.42 and a METEOR of 58.15, showing its ability to improve translation effectiveness and facilitating utterance communication.

Reinforcing Visual Content Integrity through Image Restoration and AI Recognition

A number of factors can cause photographs to deteriorate over time, losing their clarity and aesthetic appeal. The goal of this research is to help restore priceless memories because images act as memory triggers and better image quality can increase clarity and visual comfort. This research is important because it may aid in the recovery and preservation of special memories that were captured in damaged photos. Restoring an image's visual quality has the power to arouse feelings and create closer ties to the past.AI has made great progress in several areas in the recent past. However, questions concerning the veracity and authenticity of visual content have been raised due to the possibility of false information and the spread of deepfakes. We set out to create an automated system that could accurately distinguish between photographs taken by hand and images created by artificial intelligence as a solution to this problem. Furthermore, there are some useful advantages to improving image quality, including better vitality, higher clarity, and less eye strain when viewing the restored photos. All things considered, this study shows how deep learning using GANs works well for image restoration, emphasising its usefulness in retrieving priceless memories and improving visual content for wider uses. The findings of this study demonstrate the potential benefits that AI-driven image restoration methods could have on our daily interactions with visual information in the digital age as well as our personal life.

Decoding dynamic visual scenes across the brain hierarchy.

Understanding the computational mechanisms that underlie the encoding and decoding of environmental stimuli is a crucial investigation in neuroscience. Central to this pursuit is the exploration of how the brain represents visual information across its hierarchical architecture. A prominent challenge resides in discerning the neural underpinnings of the processing of dynamic natural visual scenes. Although considerable research efforts have been made to characterize individual components of the visual pathway, a systematic understanding of the distinctive neural coding associated with visual stimuli, as they traverse this hierarchical landscape, remains elusive. In this study, we leverage the comprehensive Allen Visual Coding-Neuropixels dataset and utilize the capabilities of deep learning neural network models to study neural coding in response to dynamic natural visual scenes across an expansive array of brain regions. Our study reveals that our decoding model adeptly deciphers visual scenes from neural spiking patterns exhibited within each distinct brain area. A compelling observation arises from the comparative analysis of decoding performances, which manifests as a notable encoding proficiency within the visual cortex and subcortical nuclei, in contrast to a relatively reduced encoding activity within hippocampal neurons. Strikingly, our results unveil a robust correlation between our decoding metrics and well-established anatomical and functional hierarchy indexes. These findings corroborate existing knowledge in visual coding related to artificial visual stimuli and illuminate the functional role of these deeper brain regions using dynamic stimuli. Consequently, our results suggest a novel perspective on the utility of decoding neural network models as a metric for quantifying the encoding quality of dynamic natural visual scenes represented by neural responses, thereby advancing our comprehension of visual coding within the complex hierarchy of the brain.

Language conditioned multi-scale visual attention networks for visual grounding

Visual grounding (VG) is a task that requires to locate a specific region in an image according to a natural language expression. Existing efforts on the VG task are divided into two-stage, one-stage and Transformer-based methods, which have achieved high performance. However, most of the previous methods extract visual information at a single spatial scale and ignore visual information at other spatial scales, which makes these models unable to fully utilize the visual information. Moreover, the insufficient utilization of linguistic information, especially failure to capture global linguistic information, may lead to failure to fully understand language expressions, thus limiting the performance of these models. To better address the task, we propose a language conditioned multi-scale visual attention network (LMSVA) for visual grounding, which can sufficiently utilize visual and linguistic information to perform multimodal reasoning, thus improving performance of model. Specifically, we design a visual feature extractor containing a multi-scale layer to get the required multi-scale visual features by expanding the original backbone. Moreover, we exploit pooling the output of the pre-trained Bidirectional Encoder Representations from Transformers (BERT) model to extract sentence-level linguistic features, which can enable the model to capture global linguistic information. Inspired by the Transformer architecture, we present the Visual Attention Layer guided by Language and Multi-Scale Visual Features (VALMS), which is able to better learn the visual context guided by multi-scale visual and linguistic features, and facilitates further multimodal reasoning. Extensive experiments on four large benchmark datasets, including ReferItGame, RefCOCO, RefCOCO+ and RefCOCOg, demonstrate that our proposed model achieves the state-of-the-art performance.

Sub-bundle based analysis reveals the role of human optic radiation in visual working memory.

White matter (WM) functional activity has been reliably detected through functional magnetic resonance imaging (fMRI). Previous studies have primarily examined WM bundles as unified entities, thereby obscuring the functional heterogeneity inherent within these bundles. Here, for the first time, we investigate the function of sub-bundles of a prototypical visual WM tract-the optic radiation (OR). We use the 7T retinotopy dataset from the Human Connectome Project (HCP) to reconstruct OR and further subdivide the OR into sub-bundles based on the fiber's termination in the primary visual cortex (V1). The population receptive field (pRF) model is then applied to evaluate the retinotopic properties of these sub-bundles, and the consistency of the pRF properties of sub-bundles with those of V1 subfields is evaluated. Furthermore, we utilize the HCP working memory dataset to evaluate the activations of the foveal and peripheral OR sub-bundles, along with LGN and V1 subfields, during 0-back and 2-back tasks. We then evaluate differences in 2bk-0bk contrast between foveal and peripheral sub-bundles (or subfields), and further examine potential relationships between 2bk-0bk contrast and 2-back task d-prime. The results show that the pRF properties of OR sub-bundles exhibit standard retinotopic properties and are typically similar to the properties of V1 subfields. Notably, activations during the 2-back task consistently surpass those under the 0-back task across foveal and peripheral OR sub-bundles, as well as LGN and V1 subfields. The foveal V1 displays significantly higher 2bk-0bk contrast than peripheral V1. The 2-back task d-prime shows strong correlations with 2bk-0bk contrast for foveal and peripheral OR fibers. These findings demonstrate that the blood oxygen level-dependent (BOLD) signals of OR sub-bundles encode high-fidelity visual information, underscoring the feasibility of assessing WM functional activity at the sub-bundle level. Additionally, the study highlights the role of OR in the top-down processes of visual working memory beyond the bottom-up processes for visual information transmission. Conclusively, this study innovatively proposes a novel paradigm for analyzing WM fiber tracts at the individual sub-bundle level and expands understanding of OR function.

The role of general cognitive skills in integrating visual and linguistic information during sentence comprehension: individual differences across the lifespan

Individuals exhibit massive variability in general cognitive skills that affect language processing. This variability is partly developmental. Here, we recruited a large sample of participants (N = 487), ranging from 9 to 90 years of age, and examined the involvement of nonverbal processing speed (assessed using visual and auditory reaction time tasks) and working memory (assessed using forward and backward Digit Span tasks) in a visual world task. Participants saw two objects on the screen and heard a sentence that referred to one of them. In half of the sentences, the target object could be predicted based on verb-selectional restrictions. We observed evidence for anticipatory processing on predictable compared to non-predictable trials. Visual and auditory processing speed had main effects on sentence comprehension and facilitated predictive processing, as evidenced by an interaction. We observed only weak evidence for the involvement of working memory in predictive sentence comprehension. Age had a nonlinear main effect (younger adults responded faster than children and older adults), but it did not differentially modulate predictive and non-predictive processing, nor did it modulate the involvement of processing speed and working memory. Our results contribute to delineating the cognitive skills that are involved in language-vision interactions.

Use of Interactive Food Labels to Increase Confidence in Preparing Produce among College Students.

College students may have limited access to produce and may lack confidence in preparing it, but cooking videos can show how to make healthy dishes. The Cognitive Theory of Multimedia Learning suggests that learning is enhanced when visual and auditory information is presented considering cognitive load (e.g., highlighting important concepts, eliminating extraneous information, and keeping the video brief and conversational). The purpose of this project was to pilot test a food label for produce grown at an urban university and assess whether student confidence in preparing produce improved after using the label and QR code to view a recipe video developed using principles from the Cognitive Theory of Multimedia Learning. The video showed a student preparing a salad with ingredients available on campus. Students indicated the label was helpful and reported greater perceived confidence in preparing lettuce after viewing the label and video (mean confidence of 5.60 ± 1.40 before vs. 6.14 ± 0.89 after, p = 0.016, n = 28). Keeping the video short and providing ingredients and amounts onscreen as text were cited as helpful. Thus, a brief cooking video and interactive label may improve confidence in preparing produce available on campus. Future work should determine whether the label impacts produce consumption and if it varies depending on the type of produce used.

Integrating vision and somatosensation does not improve the accuracy and response time when estimating area and perimeter of rectangles in primary school

BackgroundProblem-solving and learning in mathematics involves sensory perception and processing. Multisensory integration may contribute by enhancing sensory estimates. This study aims to assess if combining visual and somatosensory information improves elementary students' perimeter and area estimates. Methods87 4th graders compared rectangles with respect to area or perimeter either solely using visual observation or additionally with somatosensory information. Three experiments targeted different task aspects. Statistical analyses tested success rates and response times. ResultsContrary to expectations, adding somatosensory information did not boost success rates for area and perimeter comparison. Response time even increased with adding somatosensory information. Children's difficulty in accurately tracing figures negatively impacted the success rate of area comparisons. DiscussionResults suggest visual observation alone suffices for accurately estimating and comparing area and perimeter of rectangles in 4th graders. ImplicationsCareful deliberation on the inclusion of somatosensory information in mathematical tasks concerning perimeter and area estimations of rectangles is recommended.

M[formula omitted]ixup: A multi-modal data augmentation approach for image captioning

Despite the great success, most models in image captioning (IC) are still stuck in the dilemma of generating simple and non-discriminative captions. In this paper, we study this problem from the perspective of data augmentation and propose a novel method called Multi-modal Mixup (M3ixup). Compared with the original Mixup strategy designed for image classification, the proposed M3ixup has three novel designs to mix IC samples from the aspects of visual features, sentence embeddings and loss values, respectively. In practice, M3ixup can not only enrich the diversity of IC training data, but also enforce the model to focus more on visual information for captioning, thereby alleviating the negative effect of dataset bias and addressing the issue of simple captioning. To validate M3ixup, we apply it to three baseline models and conduct extensive experiments on MS COCO. The experimental results demonstrate that our proposed M3ixup can not only improve the discriminability and quality of generated captions, but also help the baseline models obtain obvious performance gains, i.e., improving the CIDEr scores of the state-of-the-art model from 133.8 to 135.3 on off-line testing and 135.4 to 137.1 on online testing.