Visual Mechanisms Research Articles

Pedestrian Re-identification Algorithm Based on Visual Attention-positive Sample Generation Network Deep Learning Model

In view of the strong generalizability and self-learning capabilities of deep learning models, many scholars have studies how to apply deep learning theory in the pedestrian re-identification field. However, a number of problems persist in practically applying deep learning in this field, including determining how to make full use of the features of the sequence information in the salient region of an image and addressing the data gap between data-driven deep learning models and pedestrian re-identification tasks. In view of these problems, in this paper, a re-identification method is proposed based on a visual common attention mechanism. Initially, the method focuses on the local area of the image at the location specified by the given coordinates. Next, under the constraint of pedestrian image pairing tags, it focuses on the sequence of salient regions of image pairs based on deep learning techniques. Then, the global features and the local attention features are cascaded into joint features for use in pedestrian re-identification. To address the data gap between deep learning models and pedestrian re-identification, a new strategy for generating difficult positive samples is proposed primarily through a positive sample that mainly involves a positive sample generation network, a difficult positive sample conversion network, and a dual-stream twin network. We using this network, a large number of positive samples can be obtained to train the data-driven neural network and solve the re-identification task. The above ideas are combined to propose a pedestrian re-identification algorithm based on a visual attention-positive sample generation network deep learning model. The experimental results show that the method proposed in this paper not only achieves better recognition results than other deep learning methods, but also adapts well to a variety of databases. In addition, the method proposed in this paper is more robust to occluded pedestrian images than other deep learning methods. In addition, the method proposed in this paper is more robust than other deep learning methods for occluding pedestrian images.

ASI-DBNet: An Adaptive Sparse Interactive ResNet-Vision Transformer Dual-Branch Network for the Grading of Brain Cancer Histopathological Images.

Brain cancer is the deadliest cancer that occurs in the brain and central nervous system, and rapid and precise grading is essential to reduce patient suffering and improve survival. Traditional convolutional neural network (CNN)-based computer-aided diagnosis algorithms cannot fully utilize the global information of pathology images, and the recently popular vision transformer (ViT) model does not focus enough on the local details of pathology images, both of which lead to a lack of precision in the focus of the model and a lack of accuracy in the grading of brain cancer. To solve this problem, we propose an adaptive sparse interaction ResNet-ViT dual-branch network (ASI-DBNet). First, we design the ResNet-ViT parallel structure to simultaneously capture and retain the local and global information of pathology images. Second, we design the adaptive sparse interaction block (ASIB) to interact the ResNet branch with the ViT branch. Furthermore, we introduce the attention mechanism in ASIB to adaptively filter the redundant information from the dual branches during the interaction so that the feature maps delivered during the interaction are more beneficial. Intensive experiments have shown that ASI-DBNet performs best in various baseline and SOTA models, with 95.24% accuracy in four grades. In particular, for brain tumors with a high degree of deterioration (Grade III and Grade IV), the highest diagnostic accuracies achieved by ASI-DBNet are 97.93% and 96.28%, respectively, which is of great clinical significance. Meanwhile, the gradient-weighted class activation map (Grad_cam) and attention rollout visualization mechanisms are utilized to visualize the working logic behind the model, and the resulting feature maps highlight the important distinguishing features related to the diagnosis. Therefore, the interpretability and confidence of the model are improved, which is of great value for the clinical diagnosis of brain cancer.

Supervised Visual Attention for Simultaneous Multimodal Machine Translation

There has been a surge in research in multimodal machine translation (MMT), where additional modalities such as images are used to improve translation quality of textual systems. A particular use for such multimodal systems is the task of simultaneous machine translation, where visual context has been shown to complement the partial information provided by the source sentence, especially in the early phases of translation. In this paper, we propose the first Transformer-based simultaneous MMT architecture, which has not been previously explored in simultaneous translation. Additionally, we extend this model with an auxiliary supervision signal that guides the visual attention mechanism using labelled phrase-region alignments. We perform comprehensive experiments on three language directions and conduct thorough quantitative and qualitative analyses using both automatic metrics and manual inspection. Our results show that (i) supervised visual attention consistently improves the translation quality of the simultaneous MMT models, and (ii) fine-tuning the MMT with supervision loss enabled leads to better performance than training the MMT from scratch. Compared to the state-of-the-art, our proposed model achieves improvements of up to 2.3 BLEU and 3.5 METEOR points.

Driving as well as on a Sunny Day? Predicting Driver's Fixation in Rainy Weather Conditions via a Dual-Branch Visual Model

Dear Editor, Traffic driving is a dynamic and complicated task in which drivers are required to pay close attention to the important targets or regions to maintain safe margins. Rainy weather conditions make it more challenging with factors such as low visibility, raindrops, pedestrian with umbrellas, wipers, etc. Studies showed that rainy condition affects driving safety significantly [1], [2]. It is reported that, in raining weather condition, the odds for a fatal accident are 3.340 times higher on highways than on streets [3]. An investigation of the relationship between rainfall and fatal crashes in Texas from 1994 to 2018 on fatality analysis reporting system (FARS) database illustrated that rain-related fatal crashes represented about 6.8% of the total fatal crashes on average, moreover, the proportion showed high variability at the annual, monthly, and hourly time scales [4]. Therefore, raining is a complex and critical factor for road safety planning and management. In fact, the traffic environment is a dynamic scene with multiple sources of information, including important targets that are highly relevant to the current driving task as well as irrelevant targets that may distract the driving task [5]. Driven by the visual selective attention mechanism, experienced drivers often focus their attention on the most important regions and only show concern for objects related to driving safety in those salient regions. This selective attention mechanism [6], [7] helps drivers reduce the interference of irrelevant scene information and guarantee the driving safety. Understanding the selective attention mechanism of experienced drivers and then simulating the efficient saliency detection process in rainy conditions may help driving a car in rainy conditions as well as on a sunny day.

Review: Ludified, edited by Marko Ciciliani, Barbara Lüneburg, and Andreas Pirchner

Review: <i>Ludified</i>, edited by Marko Ciciliani, Barbara Lüneburg, and Andreas Pirchner

IA-Net$:$ An Inception–Attention-Module-Based Network for Classifying Underwater Images From Others

To distinguish underwater images from natural images is one of the challenge of collecting and generation of underwater image data. Common image classification and recognition models classify the objects in an image depending on the saliency while suppressing the background. In this article, an inception–attention network (IA-Net), a convolutional neural network (CNN)-based model to classify the underwater images from natural images is reported, in which an inception–attention (I-A) module is constructed to simulate the visual correlation mechanism of classifying images taken from special environments such as fog, nighttime and under water. It is illustrated that the context background is as important as the salient object when understanding the underwater images. We executed experiments on a data set, which consists of 4000 underwater images and 5000 nonunderwater images, and demonstrate that the proposed IA-Net achieves an accuracy of 99.3 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> on underwater image classification, which is significantly better than classical image classification networks, such as AlexNet, InceptionV3, and ResNet. In addition, the comparative experiments prove that the IA-Net is superior to other networks when distinguishing underwater images from foggy, nighttime images and fish images taken in nonunderwater environments, although these images have indistinguishable characteristics with underwater images. Moreover, we demonstrate the I-A structure we proposed can be used to boost the performance of the existing object recognition networks. By substituting the inception module with the I-A module, the Inception-ResNetV2 network achieves a 10.7 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> top-1 error rate on the subset of ILSVRC-2012, which further illustrates the effectiveness of the correlation between the image background and subjective perception in improving the performance of the visual analysis tasks.

Android Device Malware Classification Framework Using Multistep Image Feature Extraction and Multihead Deep Neural Ensemble

The incidence of malicious threats to computer systems has increased with the increasing use of Android devices and high-speed Internet. Malware visualization mechanism can analyze a computer whenever a software or system crash occurs because of malicious activity. This paper presents a new malware classification approach to recognize such Android device malware families by capturing suspicious processes in the form of different size color images. Important local and global characteristics of color images are extracted through a combined local and global feature descriptor (structure based local and statistical based global combined texture analysis) to reduce the training complexity of neural networks. A multihead ensemble of neural networks is proposed to increase network classification performance by merging prediction results from weak learners (convolutional neural network + gated recurrent unit) and using them as learning input to a multi-layer perceptron meta learner. Two public datasets of Android device malware are used to evaluate the classification and detection performance of the proposed approach. A baseline is established to compare the classification performance of the proposed approach with those of state-of-the-art and previous malware detection approaches. The proposed multihead ensemble improved the malware classification performance, with up to 97.8%, accuracy with the R2-D2 dataset and 94.1% accuracy with the MalNet dataset. The overall results show that a multihead ensemble with multi-step feature extraction is a practical approach to classify and detect Android malware.

Detection of Dense Citrus Fruits by Combining Coordinated Attention and Cross-Scale Connection with Weighted Feature Fusion

The accuracy detection of individual citrus fruits in a citrus orchard environments is one of the key steps in realizing precision agriculture applications such as yield estimation, fruit thinning, and mechanical harvesting. This study proposes an improved object detection YOLOv5 model to achieve accurate the identification and counting of citrus fruits in an orchard environment. First, the latest visual attention mechanism coordinated attention module (CA) was inserted into an improved backbone network to focus on fruit-dense regions to recognize small target fruits. Second, an efficient two-way cross-scale connection and weighted feature fusion BiFPN in the neck network were used to replace the PANet multiscale feature fusion network, giving effective feature corresponding weights to fully fuse the high-level and bottom-level features. Finally, the varifocal loss function was used to calculate the model loss for better model training results. The results of the experiments on four varieties of citrus trees showed that our improved model proposed to this study could effectively identify dense small citrus fruits. Specifically, the recognized AP (average precision) reached 98.4%, and the average recognition time was 0.019 s per image. Compared with the original YOLOv5 (including deferent variants of n, s, m, l, and x), the increase in the average accuracy precision of the improved YOLOv5 ranged from 7.5% to 0.8% while maintaining similar average inference time. Four different citrus varieties were also tested to evaluate the generalization performance of the improved model. The method can be further used as a part in a vision system to provide technical support for the real-time and accurate detection of multiple fruit targets during mechanical picking in citrus orchards.

KSRFB-net: detecting and identifying butterflies in ecological images based on human visual mechanism

KSRFB-net: detecting and identifying butterflies in ecological images based on human visual mechanism

Lightweight single image deraining algorithm incorporating visual saliency

There are still some challenges in the task of single image rain removal, such as artefact remnant, background over-smooth, and increasingly complex and heavy-weight network architecture. Especially too heavy-weight network to fit outdoor detection devices or mobile devices. To address the above challenges, we propose a lightweight single image Deraining algorithm incorporating visual attention saliency mechanisms (LDVS). The proposed network consists of five blocks and two convolution operations, where each block consists of a dilation convolution module and a convolutional block attention module (CBAM). Specifically, visual saliency module CBAM is used for accurate localization of rain streak, and further the combinations of dilated convolution with CBAM is used to extract feature maps of rain streaks faithfully, which is able to remove artefact remnant while maintaining background details. A good tradeoff is presented between the network's weight size and effect of rain removal. Specifically, with only 48,268 parameters, the proposed model can achieve a guaranteed performance. Extensive experiments on a few typical rainy scenarios on synthetic and real-world datasets have demonstrated that to achieve the same level of performance, the proposed method has far smaller size than most of the baselines under both qualitative and quantitative analyses.

Usability of deep learning and H&E images predict disease outcome-emerging tool to optimize clinical trials

Understanding factors that impact prognosis for cancer patients have high clinical relevance for treatment decisions and monitoring of the disease outcome. Advances in artificial intelligence (AI) and digital pathology offer an exciting opportunity to capitalize on the use of whole slide images (WSIs) of hematoxylin and eosin (H&E) stained tumor tissue for objective prognosis and prediction of response to targeted therapies. AI models often require hand-delineated annotations for effective training which may not be readily available for larger data sets. In this study, we investigated whether AI models can be trained without region-level annotations and solely on patient-level survival data. We present a weakly supervised survival convolutional neural network (WSS-CNN) approach equipped with a visual attention mechanism for predicting overall survival. The inclusion of visual attention provides insights into regions of the tumor microenvironment with the pathological interpretation which may improve our understanding of the disease pathomechanism. We performed this analysis on two independent, multi-center patient data sets of lung (which is publicly available data) and bladder urothelial carcinoma. We perform univariable and multivariable analysis and show that WSS-CNN features are prognostic of overall survival in both tumor indications. The presented results highlight the significance of computational pathology algorithms for predicting prognosis using H&E stained images alone and underpin the use of computational methods to improve the efficiency of clinical trial studies.

A Novel Image Edge Detection Method Based on the Asymmetric STDP Mechanism of the Visual Path

The detection of image edges plays an important role for image processing. In view of the fact that these existing methods cannot effectively detect the edge of the image when facing the image with rich details. This paper proposes a novel method of asymmetric spike-timing-dependent plasticity (STDP) image edge detection based on the visual physiological mechanism. In the proposed method, the original image is preprocessed by the Gabor filter to simulate the visual physiological orientation characteristics to obtain the image information in different directions, and the orientation feature fusion is used to reconstruct the primary edge feature information of the image. Then, based on the mechanism of the visual nervous system, a neuron network composed of dynamic synapses based on the asymmetric STDP mechanism is constructed to further process it to obtain impulse response images. In order to eliminate disturbance of the neuron’s system noise on the impulse response image, the impulse response image is filtered by a Gaussian filter. Then, the lateral inhibition between neurons is applied to refine the filtered image edges. Finally, the result is normalized, and the final edge of the experimental image is obtained. Experimental results based on the colony image data set collected in the laboratory indicate that the proposed method achieved better performance than these state-of-the-art methods; meanwhile, the AUC value remains above 0.6.

Tuning electronic structure via CoS clusters for visual photocatalytic H2 production and mechanism insight

Clusters with this unique atomic arrangement and configuration are highly potential cocatalysts, but designing efficient clusters and clarifying the mechanism of action remains a challenge. Herein, CoS clusters are in-situ grown on CdS nanorods, which leads to the existence of Cd-S-Co bonds. It is found that the Cd-S-Co bonds could induce interfacial charge rearrangements. Theoretical calculations reveal that the charge redistribution of heterostructures at interfaces is conducive to modulating d-band center, reducing thermodynamic energy barriers, and improving carrier separation. The optimized CdS/CoS composite shows a great photocatalytic H2 production rate (65.7 mmol h−1 g−1), such a H2 evolution rate exceeded most reported CdS nanorods photocatalyst. This study provides an experimental and theoretical foundation for the preparation of cluster-semiconductors photocatalytic materials and explores the transfer pathway between clusters and catalysts at the atomic level, and deeply discusses the mechanism of their catalytic performance.

Revisit the circle of representation in Web 2.0

ABSTRACT Visual perception plays a leading role in tourism destination marketing. In the era of Web 2.0, it is critical to understand whether and how the circle of representation has changed in the visual representation. The research combines photo data from the internet and interview data to explore the visual representation mechanism of TDI. Through both content and thematic analysis, the research finds that: (1) diverse sources result in differences and similarities between the projected and perceived images of the destination; (2) the circle of representation is still valid. The icons were photographed repeatedly due to the impacts of social context; (3) new iconic images emerged to satisfy tourists’ needs to construct meaningful experiences. The emerging images became a new component of the circle of representation. Applications for destination marketing are made accordingly.

Multiscale Brain-Like Neural Network for Saliency Prediction on Omnidirectional Images

Current top-performing saliency prediction methods of omnidirectional images (ODIs) depend on deep feedforward convolutional neural networks (CNNs), benefiting from their powerful multiscale representation ability. Although these methods adopt deep feedforward CNNs to achieve superb performance in saliency prediction task, they have the following limitations: 1) these deep feedforward CNNs are difficult to map to ventral stream structure of the brain visual system due to their vast number of layers and missing biologically important connections, such as recurrence and 2) most deep feedforward CNNs represent the multiscale features in a layerwise manner. To tackle these issues, models that could learn multiscale features yet share the similarities with human brain are needed. In this article, we propose a novel multiscale brain-like network (MBN) model to predict saliency of head fixations on ODIs. Specifically, our proposed model consists of two major modules: 1) a brain-like CORnet-S module and 2) a multiscale feature extraction module. The CORnet-S module is a lightweight backbone network with four anatomically mapped areas (V1, V2, V4, and IT) and it can simulate the visual processing mechanism of ventral visual stream in the human brain. The multiscale feature extraction module is inspired by the multiscale brain structure, which represents multiscale features at a granular level and increases the range of receptive fields for each network layer. Extensive experiments and ablation studies conducted on two major benchmarks demonstrate the superiority of the proposed MBN model over the state-of-the-art methods.

Visual Hierarchy Design of Map Site Information in Thematic Meteorological Interface

The visual map elements of the thematic meteorological interface have an essential impact on the user’s processing efficiency. Based on the different visual hierarchies of map site information, including shape, colour, and area, this paper explored the influence of site information’s visual hierarchy change mechanism on user behaviour performance. The results were as follows: 1) The greater the number of shapes, the more difficult it was to complete the task. Therefore, the shape of map sites could be formulated according to the type of meteorological disaster. 2) Under the same comparison level, when the number of colours for multiple map sites was 6, the user’s task operation efficiency was best. 3) The area of map sites was reasonably enlarged within a specific range that could improve the operation efficiency of interface tasks and balance the information complexity of map sites. The results have certain reference value for the visual hierarchy design of map sites in the meteorological interface.

Illusional Perspective across Humans and Bees.

For two centuries, visual illusions have attracted the attention of neurobiologists and comparative psychologists, given the possibility of investigating the complexity of perceptual mechanisms by using relatively simple patterns. Animal models, such as primates, birds, and fish, have played a crucial role in understanding the physiological circuits involved in the susceptibility of visual illusions. However, the comprehension of such mechanisms is still a matter of debate. Despite their different neural architectures, recent studies have shown that some arthropods, primarily Hymenoptera and Diptera, experience illusions similar to those humans do, suggesting that perceptual mechanisms are evolutionarily conserved among species. Here, we review the current state of illusory perception in bees. First, we introduce bees’ visual system and speculate which areas might make them susceptible to illusory scenes. Second, we review the current state of knowledge on misperception in bees (Apidae), focusing on the visual stimuli used in the literature. Finally, we discuss important aspects to be considered before claiming that a species shows higher cognitive ability while equally supporting alternative hypotheses. This growing evidence provides insights into the evolutionary origin of visual mechanisms across species.

Video-based assessment of intraoperative surgical skill.

Surgeons' skill in the operating room is a major determinant of patient outcomes. Assessment of surgeons' skill is necessary to improve patient outcomes and quality of care through surgical training and coaching. Methods for video-based assessment of surgical skill can provide objective and efficient tools for surgeons. Our work introduces a new method based on attention mechanisms and provides a comprehensive comparative analysis of state-of-the-art methods for video-based assessment of surgical skill in the operating room. Using a dataset of 99 videos of capsulorhexis, a critical step in cataract surgery, we evaluated image feature-based methods and two deep learning methods to assess skill using RGB videos. In the first method, we predict instrument tips as keypoints and predict surgical skill using temporal convolutional neural networks. In the second method, we propose a frame-wise encoder (2D convolutional neural network) followed by a temporal model (recurrent neural network), both of which are augmented by visual attention mechanisms. We computed the area under the receiver operating characteristic curve (AUC), sensitivity, specificity, and predictive values through fivefold cross-validation. To classify a binary skill label (expert vs. novice), the range of AUC estimates was 0.49 (95% confidence interval; CI = 0.37 to 0.60) to 0.76 (95% CI = 0.66 to 0.85) for image feature-based methods. The sensitivity and specificity were consistently high for none of the methods. For the deep learning methods, the AUC was 0.79 (95% CI = 0.70 to 0.88) using keypoints alone, 0.78 (95% CI = 0.69 to 0.88) and 0.75 (95% CI = 0.65 to 0.85) with and without attention mechanisms, respectively. Deep learning methods are necessary for video-based assessment of surgical skill in the operating room. Attention mechanisms improved discrimination ability of the network. Our findings should be evaluated for external validity in other datasets.

Eye-tracking as a window into primate social cognition.

Over the past decade, noninvasive, restraint-free eye-tracking research with primates has transformed our understanding of primate social cognition. The use of this technology with many primate species allows for the exploration and comparison of how these species attend to and understand social agents and interactions. The ability to compare and contrast the cognitive capacities of various primate species, including humans, provides insight into the evolutionary mechanisms and selective pressures that have likely shaped social cognition in similar and divergent ways across the primate order. In this review, we begin by discussing noninvasive behavioral methods used to measure primate gaze and attention before the introduction of noninvasive, restraint-free eye-tracking methodologies. Next, we focus on findings from recent eye-tracking research on primate social cognition, beginning with simple visual and search mechanisms. We then discuss the results that have built on this basic understanding of how primates view images and videos, exploring discrimination and knowledge of social agents, following social cues, tracking perspectives and predicting behavior, and the combination of eye-tracking and other behavioral and physiological methods. Finally, we discuss some future directions of noninvasive eye-tracking research on primate social cognition and current eye-tracking work-in-progress that builds on these previous studies, investigating underexplored socio-cognitive capacities and utilizing new methodologies.

A Lightweight Efficient Person Re-Identification Method Based on Multi-Attribute Feature Generation

Person re-identification (re-ID) technology has attracted extensive interests in critical applications of daily lives, such as autonomous surveillance systems and intelligent control. However, light-weight and efficient person re-ID solutions are rare because the limited computing resources cannot guarantee accuracy and efficiency in detecting person features, which inevitably results in performance bottleneck in real-time applications. Aiming at this research challenge, this study developed a lightweight framework for generation of the person multi-attribute feature. The framework mainly consists of three sub-networks each conforming to a convolutional neural network architecture: (1) the accessory attribute network (a-ANet) grasps the person ornament information for an accessory descriptor; (2) the body attribute network (b-ANet) captures the person region structure for a body descriptor; and (3) the color attribute network (c-ANet) forms the color descriptor to maintain the consistency of the color of the person(s). Inspired by the human visual processing mechanism, these descriptors (each “descriptor” corresponds to the attribute of an individual person) are integrated via a tree-based feature-selection method to construct a global “feature”, i.e., a multi-attribute descriptor of the person serving as the key to identify the person. Distance learning is then exploited to measure the person similarity for the final person re-identification. Experiments have been performed on four public datasets to evaluate the proposed framework: CUHK-01, CUHK-03, Market-1501, and VIPeR. The results indicate that (1) the multi-attribute feature outperforms most of the existing feature-representation methods by 5–10% at rank@1 in terms of the cumulative matching curve criterion; and (2) the time required for recognition is as low as O(n) for real-time person re-ID applications.