Image Saliency Research Articles

Landmark-aware autonomous odometry correction and map pruning for planetary rovers

Planetary rover autonomous localization is paramount for a planetary surface exploration mission. However, existing methods demonstrate limited localization accuracy, mostly due to the unstructured texture characterization of planetary surface. In response, this study presents a novel Neural Radiance Field (NeRF) driven visual odometry correction method that allows for high-precision 6-DoF rover pose estimation and local map pruning. First, an innovative image saliency evaluation approach, combining binarization and feature detection, is introduced to meticulously select landmarks that are conducive to rover re-localization. Subsequently, we conduct 3D reconstruction and rendering of the chosen landmarks based on a-priori knowledge of planetary surface images and their Neural Radiance Field (NeRF) models. High-precision odometry correction is achieved through the optimization of photometric loss between NeRF rending images and real images. Simultaneously, the odometry correction mechanism is employed in an autonomous manner to refine the NeRF model of the corresponding landmark, leading to an improved local map and gradually enhanced rover localization accuracy. Numerical simulation and experiment trials are carried out to evaluate the performance of the proposed method, results of which demonstrate state-of-the-art rover re-localization accuracy and local map pruning.

Gaze behaviors during free viewing revealed differences in visual salience processing across four major psychiatric disorders: a mega-analysis study of 1012 individuals.

Aberrant salience processing has been proposed as a pathophysiological mechanism underlying psychiatric symptoms in patients with schizophrenia. The gaze trajectories of individuals with schizophrenia have been reported to be abnormal when viewing an image, suggesting anomalous visual salience as one possible pathophysiological mechanism associated with psychiatric diseases. This study was designed to determine whether visual salience is affected in individuals with schizophrenia, and whether this abnormality is unique to patients with schizophrenia. We examined the gaze behaviors of 1012 participants recruited from seven institutes (550 healthy individuals and 238, 41, 50 and 133 individuals with schizophrenia, bipolar disorder, major depressive disorder and autism spectrum disorder, respectively) when they looked at stationary images as they liked, i.e., free-viewing condition. We used an established computational model of salience maps derived from low-level visual features to measure the degree to which the gaze trajectories of individuals were guided by visual salience. The analysis revealed that the saliency at the gaze of individuals with schizophrenia were higher than healthy individuals, suggesting that patients' gazes were guided more by low-level image salience. Among the low-level image features, orientation salience was most affected. Furthermore, a general linear model analysis of the data for the four psychiatric disorders revealed a significant effect of disease. This abnormal salience processing depended on the disease and was strongest in patients with schizophrenia, followed by patients with bipolar disorder, major depressive disorder, and autism spectrum disorder, suggesting a link between abnormalities in salience processing and strength/frequency for psychosis of these disorders.

Design of Chinese painting style classification model based on multi-layer aggregation CNN

This study delves deeply into exploring the artistic value of traditional Chinese painting (TCP) and aims to bridge the gap between its fundamental characteristics and the realm of human emotions. To achieve this, a novel convolutional neural network (CNN)-based classification model for TCP emotions is proposed. By thoroughly analyzing the distinct emotional mapping relationships inherent in TCP, a comprehensive framework is developed. Notably, emotional feature regions are accurately extracted using image saliency, and a multi-layer aggregation recalibrated emotional feature module is seamlessly integrated into the CNN network structure. This integration strengthens the activation of features that significantly influence TCP emotions. Moreover, a method of multi-category weighted activation localization is skillfully employed to classify Chinese painting emotions within the CMYK (cyan magenta yellow black) color space. The empirical results convincingly demonstrate that our algorithm surpasses existing approaches such as VGG-19, GoogLeNet, ResNet-50, and WSCNet in the task of TCP emotion recognition, achieving an impressive accuracy of 92.36%, the largest error is 0.191. This improvement signifies the advancements made by our model in accurately capturing and understanding the emotional nuances within TCP. By outperforming previous methods, our research contributes to the convergence of multimedia technology and cultural education.

Pulp grade monitoring using binocular image through multi-scale feature cross-attention fusion network and saliency map constraint

Pulp grade is an important indicator for performance monitoring in froth flotation process. Previous studies have shown that the pulp grade can be predicted more accurately by using the stereo vision information of froth images. However, due to the low intra-class variation among froth images under the identical working conditions and the similarity in shape and texture among bubbles, it is challenging for current binocular image-based methods for grade prediction. Therefore, to accurately predict the key performance indicators in flotation process, a prediction model based on binocular image fusion is proposed in this paper. First, a calculation method of froth image saliency map is proposed, and the saliency map is used as a priori knowledge to guide the prediction model to learn the characteristics of the region of interest in the image. This measure aims to solve the problem of difficulty in grade prediction caused by low intra-class differences in froth images. Then, a multi-scale feature cross-attention fusion network is introduced, wherein the multi-scale features of the left and right views serve as attention mechanism gating signals to extract common feature from binocular image. After that, a pulp grade prediction model is developed based on Video Transformer Network. The results on actual industrial flotation datasets show that, compared with other pulp grade prediction methods, our proposed approach reduces the mean absolute error and root mean square error by 22.84% and 23.55%, respectively.

Automated Identification of Clinically Relevant Regions in Glaucoma OCT Reports Using Expert Eye Tracking Data and Deep Learning.

To propose a deep learning-based approach for predicting the most-fixated regions on optical coherence tomography (OCT) reports using eye tracking data of ophthalmologists, assisting them in finding medically salient image regions. We collected eye tracking data of ophthalmology residents, fellows, and faculty as they viewed OCT reports to detect glaucoma. We used a U-Net model as the deep learning backbone and quantized eye tracking coordinates by dividing the input report into an 11 × 11 grid. The model was trained to predict the grids on which fixations would land in unseen OCT reports. We investigated the contribution of different variables, including the viewer's level of expertise, model architecture, and number of eye gaze patterns included in training. Our approach predicted most-fixated regions in OCT reports with precision of 0.723, recall of 0.562, and f1-score of 0.609. We found that using a grid-based eye tracking structure enabled efficient training and using a U-Net backbone led to the best performance. Our approach has the potential to assist ophthalmologists in diagnosing glaucoma by predicting the most medically salient regions on OCT reports. Our study suggests the value of eye tracking in guiding deep learning algorithms toward informative regions when experts may not be accessible. By suggesting important OCT report regions for a glaucoma diagnosis, our model could aid in medical education and serve as a precursor for self-supervised deep learning approaches to expedite early detection of irreversible vision loss owing to glaucoma.

A novel framework for concealed weapons detection using passive terahertz imaging

ABSTRACT In this work, a novel framework for non-destructive, non-intrusive and completely automatic concealed weapon detection under human clothing for entry control and security check applications using passive terahertz (THz) images has been given. The technique models the problem of concealed weapon detection as that of binary segmentation, where the pixels corresponding to concealed weapons are foreground while the remaining are background. The core idea of the proposed framework is to first oversegment the THz image into superpixels and subsequently multiple handcrafted features are extracted from each superpixel. Finally, a machine learning-based classifier is used to perform binary classification thereby classifying each superpixel as foreground or background such that each pixel in a foreground superpixel is a foreground pixel and vice-versa for background. It is worth mentioning that this technique does not require a perfect segmentation to extract features from, rather an oversegmentation suffices. Furthermore, both global features like those based on image saliency and local features like those based on intensity are extracted considering the image properties. The detailed experiments and comparative analysis confirm that the proposed technique efficiently detects concealed weapons, exhibiting superior performance compared to state-of-the-art binary segmentation methods for the purpose.

Underwater Image Enhancement Based on Luminance Reconstruction by Multi-Resolution Fusion of RGB Channels.

Underwater image enhancement technology is crucial for the human exploration and exploitation of marine resources. The visibility of underwater images is affected by visible light attenuation. This paper proposes an image reconstruction method based on the decomposition-fusion of multi-channel luminance data to enhance the visibility of underwater images. The proposed method is a single-image approach to cope with the condition that underwater paired images are difficult to obtain. The original image is first divided into its three RGB channels. To reduce artifacts and inconsistencies in the fused images, a multi-resolution fusion process based on the Laplace-Gaussian pyramid guided by a weight map is employed. Image saliency analysis and mask sharpening methods are also introduced to color-correct the fused images. The results indicate that the method presented in this paper effectively enhances the visibility of dark regions in the original image and globally improves its color, contrast, and sharpness compared to current state-of-the-art methods. Our method can enhance underwater images in engineering practice, laying the foundation for in-depth research on underwater images.

A multimodal neural network with gradient blending improves predictions of survival and metastasis in sarcoma

The objective of this study is to develop a multimodal neural network (MMNN) model that analyzes clinical variables and MRI images of a soft tissue sarcoma (STS) patient, to predict overall survival and risk of distant metastases. We compare the performance of this MMNN to models based on clinical variables alone, radiomics models, and an unimodal neural network. We include patients aged 18 or older with biopsy-proven STS who underwent primary resection between January 1st, 2005, and December 31st, 2020 with complete outcome data and a pre-treatment MRI with both a T1 post-contrast sequence and a T2 fat-sat sequence available. A total of 9380 MRI slices containing sarcomas from 287 patients are available. Our MMNN accepts the entire 3D sarcoma volume from T1 and T2 MRIs and clinical variables. Gradient blending allows the clinical and image sub-networks to optimally converge without overfitting. Heat maps were generated to visualize the salient image features. Our MMNN outperformed all other models in predicting overall survival and the risk of distant metastases. The C-Index of our MMNN for overall survival is 0.77 and the C-Index for risk of distant metastases is 0.70. The provided heat maps demonstrate areas of sarcomas deemed most salient for predictions. Our multimodal neural network with gradient blending improves predictions of overall survival and risk of distant metastases in patients with soft tissue sarcoma. Future work enabling accurate subtype-specific predictions will likely utilize similar end-to-end multimodal neural network architecture and require prospective curation of high-quality data, the inclusion of genomic data, and the involvement of multiple centers through federated learning.

Saliency-based video summarization for face anti-spoofing

With the growing availability of databases for face presentation attack detection, researchers are increasingly focusing on video-based face anti-spoofing methods that involve hundreds to thousands of images for training the models. However, there is currently no clear consensus on the optimal number of frames in a video to improve face spoofing detection. Inspired by the visual saliency theory, we present a video summarization method for face anti-spoofing detection that aims to enhance the performance and efficiency of deep learning models by leveraging visual saliency. In particular, saliency information is extracted from the differences between the Laplacian and Wiener filter outputs of the source images, enabling the identification of the most visually salient regions within each frame. Subsequently, the source images are decomposed into base and detail images, enhancing the representation of the most important information. Weighting maps are then computed based on the saliency information, indicating the importance of each pixel in the image. By linearly combining the base and detail images using the weighting maps, the method fuses the source images to create a single representative image that summarizes the entire video. The key contribution of the proposed method lies in demonstrating how visual saliency can be used as a data-centric approach to improve the performance and efficiency for face presentation attack detection. By focusing on the most salient images or regions within the images, a more representative and diverse training set can be created, potentially leading to more effective models. To validate the method’s effectiveness, a simple CNN–RNN deep learning architecture was used, and the experimental results showcased state-of-the-art performance on four challenging face anti-spoofing datasets.

Video Salient Object Detection Via Multi-level Spatiotemporal Bidirectional Network Using Multi-scale Transfer Learning

Video saliency prediction aims to resemble human visual attention by identifying the most relevant and significant elements in a video frame or sequence. This task becomes notably intricate in scenarios characterized by dynamic elements such as rapid motion, occlusions, blur, background variations, and nonrigid deformations. Therefore, the inherent complexity of human visual attention behavior during dynamic scenes necessitates the assessment of both temporal and spatial data. Existing video saliency frameworks often falter under such conditions, and relying solely on image saliency models neglects crucial temporal information in videos. This study presents a new Video Salient Object Detection via Multi-level Spatiotemporal Bidirectional Network using Multi-scale Transfer Learning (MSB-Net) to address the problem of identifying significant objects in videos. The proposed MSB-Net achieves notable results for a given sequence of frames by employing multi-scale transfer learning with an encoder and decoder approach to acquire knowledge and saliency map attributes spatially and temporally. The proposed MSB-Net model has bidirectional LSTM (Long Short-Term Memory) and CNN (Convolutional Neural Network) components. The VGG16 (Video Geometry Group) and VGG19 architectures extract multi-scale features from the input video frames. Evaluation of diverse datasets, namely DAVIS-T, SegTrack-V2, ViSal, VOS-T, and DAVSOD-T, demonstrates the model's effectiveness, outperforming other competitive models based on parameters such as MAE, F-measure, and S-measure.

Research on image saliency detection based on deep neural network

AbstractAs a hot research field at present, computer vision is devoted to the rapid acquisition and application of target information from images or videos by simulating human visual mechanism. In order to improve the accuracy and efficiency of image detection, image saliency region detection technology has received more and more attention in the field of computer vision research; an important research content in the field, the core part of which lies in the research on algorithms related to feature extraction and saliency calculation of targets. This paper analyzes the multi‐feature fusion saliency detection model and visual saliency calculation process, and based on the existing algorithm, by improving the VGG16 network, a fully convolutional network saliency detection algorithm is proposed. The qualitative and quantitative experimental results show that compared with the four mainstream methods of BL, GS, SF, and RFCN, our algorithm not only improves the accuracy of salient object detection, but also effectively solves the problem of target edge blur. Therefore, this study has improved the accuracy and efficiency of saliency detection, which can not only promote the development of computer vision technology, but also provide support for research in the field of image processing.

TranSalNet+: Distortion-aware saliency prediction

Predicting the saliency of images affected by distortion is a challenging but emerging research problem. Given a distorted image, we wish to accurately predict saliency as perceived by humans. A recent distortion-aware saliency benchmark – the CUDAS database – reveals the inadequacy of existing saliency models in handling distorted images. In this paper, we devise a deep learning Distortion-Aware Saliency Module (DASM) that enables capturing saliency features related to image distortions, and integrates this module into a saliency prediction architecture. To achieve the high expressive capability of DASM using supervised learning, we create a dedicated dataset that draws upon a large-scale saliency dataset and machine-generated image quality assessments. Experimental results demonstrate the superior performance of the proposed model in predicting the saliency of distorted images.

Video saliency detection using modified high efficiency video coding and background modelling

Video saliency has a profound effect on our lives with its compression efficiency and precision. There have been several types of research done on image saliency but not on video saliency. This paper proposes a modified high efficiency video coding (HEVC) algorithm with background modelling and the implication of classification into coding blocks. This solution first employs the G-picture in the fourth frame as a long-term reference and then it is quantized based on the algorithm that segregates using the background features of the image. Then coding blocks are introduced to decrease the complexity of the HEVC code, reduce time consumption and overall speed up the process of saliency. The solution is experimented upon with the dynamic human fixation 1K (DHF1K) dataset and compared with several other state-of-the-art saliency methods to showcase the reliability and efficiency of the proposed solution.

Enhancing Livestock Detection: An Efficient Model Based on YOLOv8

Maintaining a harmonious balance between grassland ecology and local economic development necessitates effective management of livestock resources. Traditional approaches have proven inefficient, highlighting an urgent need for intelligent solutions. Accurate identification of livestock targets is pivotal for precise livestock farming management. However, the You Only Look Once version 8 (YOLOv8) model exhibits limitations in accuracy when confronted with complex backgrounds and densely clustered targets. To address these challenges, this study proposes an optimized CCS-YOLOv8 (Comprehensive Contextual Sensing YOLOv8) model. First, we curated a comprehensive livestock detection dataset encompassing the Qinghai region. Second, the YOLOv8n model underwent three key enhancements: (1) incorporating a Convolutional Block Attention Module (CBAM) to accentuate salient image information, thereby boosting feature representational power; (2) integrating a Content-Aware ReAssembly of FEatures (CARAFE) operator to mitigate irrelevant interference, improving the integrity and accuracy of feature extraction; and (3) introducing a dedicated small object detection layer to capture finer livestock details, enhancing the recognition of smaller targets. Experimental results on our dataset demonstrate the CCS-YOLOv8 model’s superior performance, achieving 84.1% precision, 82.2% recall, 84.4% mAP@0.5, 60.3% mAP@0.75, 53.6% mAP@0.5:0.95, and 83.1% F1-score. These metrics reflect substantial improvements of 1.1%, 7.9%, 5.8%, 6.6%, 4.8%, and 4.7%, respectively, over the baseline model. Compared to mainstream object detection models, CCS-YOLOv8 strikes an optimal balance between accuracy and real-time processing capability. Its robustness is further validated on the VisDrone2019 dataset. The CCS-YOLOv8 model enables rapid and accurate identification of livestock age groups and species, effectively overcoming the challenges posed by complex grassland backgrounds and densely clustered targets. It offers a novel strategy for precise livestock population management and overgrazing prevention, aligning seamlessly with the demands of modern precision livestock farming. Moreover, it promotes local environmental conservation and fosters sustainable development within the livestock industry.

A Two-branch Edge Guided Lightweight Network for infrared image saliency detection

In the dynamic landscape of saliency detection, convolutional neural networks have emerged as catalysts for innovation, but remain largely tailored for RGB imagery, falling short in the context of infrared images, particularly in memory-restricted environments. These existing approaches tend to overlook the wealth of contour information vital for a nuanced analysis of infrared images. Addressing this notable gap, we introduce the novel Two-branch Edge Guided Lightweight Network (TBENet), designed explicitly for the robust analysis of infrared image saliency detection. The main contributions of this paper are as follows. First, we formulate the saliency detection task as two subtasks, contour enhancement and foreground segmentation. Therefore, the TBENet is divided into two specialized branches: a contour prediction branch for extracting target contour and a saliency map generation branch for separating the foreground from the background. The first branch employs an encoder–decoder architecture to meticulously delineate object contours, serving as a guiding blueprint for the second branch. This latter segment adeptly integrates spatial and semantic data, creating a precise saliency map that is refined further by an innovative edge-weighted contour loss function. Second, to enhance feature integration capabilities, we propose depthwise multi-scale and multi-cue modules, facilitating sophisticated feature aggregation. Third, a high-level linear bottleneck module is devised to ensure the extraction of rich semantic information, and by replacing the standard convolution with the depthwise convolution, it is beneficial to reduce model complexity. Additional, we reduce the number of channels of the feature maps from each stage of the decoder to further enhance the lightweight of the model. Last, we construct a novel infrared ship dataset Small-IRShip to train and evaluate our proposed model. Experimental results on the homemade dataset Small-IRShip and two publicly available datasets, namely RGB-T and IRSTD-1k, demonstrate TBENet’s superior performance over state-of-the-art methods, affirming its effectiveness in harnessing edge information and incorporating advanced feature integration strategies.

3D Pop-Ups: Omnidirectional image visual saliency prediction based on crowdsourced eye-tracking data in VR

The prediction of the visual saliency of omnidirectional images in VR is valuable for understanding visual behaviors. However, the equipment cost, software setups, hardware operation, and other constraints in acquiring eye-tracking data of omnidirectional images for visual saliency prediction would lead to a low training efficiency and prediction performance. Therefore, this paper proposed a crowdsourcing method based on recall fixations, which was used to collect and construct an omnidirectional image with eye-tracking dataset called CrowdSourcing360, which contained 16,200 pieces of data on 180 images. Using this dataset, a visual saliency prediction model CSnet360 was trained. Experiments demonstrated that the visual saliency prediction performance of the CSnet360 outperformed most existing models even without using actual gaze fixations. Finally, a VR interior design assistance prototype system was built and the preliminary study results indicated that the system could help designers to improve the quality of their design solutions.

Visual Saliency Detection using Deep Learning

Salient object discovery models mimic the gesture of human beings and capture the most salient region/ object from the images or scenes. This field has numerous important operations in both computer vision and pattern recognition tasks. Despite hundreds of models proposed in this field, it still has a large room for exploration. This paper demonstrates a detailed overview of the recent progress of saliency discovery models in terms of heuristic- grounded ways and deep literacy- grounded ways. We've bandied and reviewed it’s co-related fields, similar as Eye obsession- vaticination, RGBD salient- object- discovery, co- saliency object discovery, and videotape- saliency- discovery models. Image saliency object discovery can fleetly prize useful information from image scenes and further assay it. At present, the traditional saliency target discovery technology still has the edge of outstanding target that can not be well saved. Convolutional neural network( CNN) can prize largely general deep features from the images and effectively express the essential point information of the images. This paper designs a model which applies CNN in deep saliency object discovery tasks. It can efficiently optimize the edges of focus objects and realize largely effective image saliency discovery through multilayer nonstop point birth, refinement of layered boundary, and original saliency point emulsion. The experimental result shows that the proposed system can achieve further robust saliency discovery to acclimate itself to complex background terrain.

Quantifying task-related gaze.

Competing theories attempt to explain what guides eye movements when exploring natural scenes: bottom-up image salience and top-down semantic salience. In one study, we apply language-based analyses to quantify the well-known observation that task influences gaze in natural scenes. Subjects viewed ten scenes as if they were performing one of two tasks. We found that the semantic similarity between the task and the labels of objects in the scenes captured the task-dependence of gaze (t(39) = 13.083; p < 0.001). In another study, we examined whether image salience or semantic salience better predicts gaze during a search task, and if viewing strategies are affected by searching for targets of high or low semantic relevance to the scene. Subjects searched 100 scenes for a high- or low-relevance object. We found that image salience becomes a worse predictor of gaze across successive fixations, while semantic salience remains a consistent predictor (X2(1, N=40) = 75.148, p < .001). Furthermore, we found that semantic salience decreased as object relevance decreased (t(39) = 2.304; p = .027). These results suggest that semantic salience is a useful predictor of gaze during task-related scene viewing, and that even in target-absent trials, gaze is modulated by the relevance of a search target to the scene in which it might be located.

A Unified Visual and Linguistic Semantics Method for Enhanced Image Captioning

Image captioning, also recognized as the challenge of transforming visual data into coherent natural language descriptions, has persisted as a complex problem. Traditional approaches often suffer from semantic gaps, wherein the generated textual descriptions lack depth, context, or the nuanced relationships contained within the images. In an effort to overcome these limitations, we introduce a novel encoder–decoder framework called A Unified Visual and Linguistic Semantics Method. Our method comprises three key components: an encoder, a mapping network, and a decoder. The encoder employs a fusion of CLIP (Contrastive Language–Image Pre-training) and SegmentCLIP to process and extract salient image features. SegmentCLIP builds upon CLIP’s foundational architecture by employing a clustering mechanism, thereby enhancing the semantic relationships between textual and visual elements in the image. The extracted features are then transformed by a mapping network into a fixed-length prefix. A GPT-2-based decoder subsequently generates a corresponding Chinese language description for the image. This framework aims to harmonize feature extraction and semantic enrichment, thereby producing more contextually accurate and comprehensive image descriptions. Our quantitative assessment reveals that our model exhibits notable enhancements across the intricate AIC-ICC, Flickr8k-CN, and COCO-CN datasets, evidenced by a 2% improvement in BLEU@4 and a 10% uplift in CIDEr scores. Additionally, it demonstrates acceptable efficiency in terms of simplicity, speed, and reduction in computational burden.

Multi‐exposure embeddings for graph learning: Towards high dynamic range image saliency prediction

AbstractIdentifying saliency in high dynamic range (HDR) images is a fundamentally important issue in HDR imaging, and plays critical roles towards comprehensive scene understanding. Most of existing studies leverage hand‐crafted features for HDR image saliency prediction, lacking the capabilities of fully exploiting the characteristics of HDR image (i.e. wider luminance range and richer colour gamut). Here, systematical studies are carried out on HDR image saliency prediction by proposing a new framework to single out the contributions from multi‐exposure images. Specifically, inspired by the mechanism of HDR imaging, the method first utilizes graph neural networks to model the relations among multi‐exposure images and the tone‐mapped image obtained from an HDR image, enabling more discriminative saliency‐related feature representations. Subsequently, the saliency features driven by global semantic knowledge are aggregated from the tone‐mapped image through enhancing global context‐aware semantic information. Finally, a fusion module is designed to integrate saliency‐oriented feature representations originated from multi‐exposure images and the tone‐mapped image, producing the saliency maps of HDR images. Moreover, a new challenging HDR eye fixation database (HDR‐EYEFix) is created, expecting to further contribute the research on HDR image saliency prediction. Experiment results show that the method obtains superior performance compared to the state‐of‐the‐art methods.