Saliency Detection Method Research Articles

A virtual-reality spatial matching algorithm and its application on equipment maintenance support: System design and user study

Equipment maintenance support is an important technical measure to maintain the equipment’s expected performance. However, the current maintenance supports are mainly completed by maintainers under the guidance of technical manual or additional experts, which may be insufficient for some advanced equipment with rapid update rate and complex inner structure. The rising technology of augmented reality (AR) provides a new solution for equipment maintenance support, while one of the key issues limiting the practical application of AR in maintenance field is the spatial matching issue between virtual space and reality space. In this paper, a virtual-reality spatial matching algorithm is designed to accurately superimpose the virtual information to the corresponding actual scene on the AR glasses. In this algorithm, two methods are proposed to help achieve the stable matching of virtual space and reality space. In detail, to obtain the saliency map with less background interference and improved saliency detection accuracy, a saliency detection method is designed based on the super-pixel segmentation. To deal with the problems of uneven distribution on the feature points and weak robustness to the light changes, a feature extraction and matching method is proposed for acquiring the feature point matching set with the utilization of the obtained saliency map. Finally, an immersive equipment maintenance support system (IEMSS) is developed based on this spatial matching algorithm, which provides the maintainers with immediate and immersive guidance to improve the efficiency and safety in the maintenance task, as well as offers maintenance training for inexperienced maintainers with expanded virtual information in case of limited experts. Several comparative experiments are implemented to verify the effectiveness of proposed methods, and a user study of real system application is carried out to further evaluate the superiority of these methods when applied in the IEMSS.

Cascaded Aggregation Convolution Network for Salient Grain Pests Detection.

Pest infestation poses significant threats to grain storage due to pests' behaviors of feeding, respiration, excretion, and reproduction. Efficient pest detection and control are essential to mitigate these risks. However, accurate detection of small grain pests remains challenging due to their small size, high variability, low contrast, and cluttered background. Salient pest detection focuses on the visual features that stand out, improving the accuracy of pest identification in complex environments. Drawing inspiration from the rapid pest recognition abilities of humans and birds, we propose a novel Cascaded Aggregation Convolution Network (CACNet) for pest detection and control in stored grain. Our approach aims to improve detection accuracy by employing a reverse cascade feature aggregation network that imitates the visual attention mechanism in humans when observing and focusing on objects of interest. The CACNet uses VGG16 as the backbone network and incorporates two key operations, namely feature enhancement and feature aggregation. These operations merge the high-level semantic information and low-level positional information of salient objects, enabling accurate segmentation of small-scale grain pests. We have curated the GrainPest dataset, comprising 500 images showcasing zero to five or more pests in grains. Leveraging this dataset and the MSRA-B dataset, we validated our method's efficacy, achieving a structure S-measure of 91.9%, and 90.9%, and a weighted F-measure of 76.4%, and 91.0%, respectively. Our approach significantly surpasses the traditional saliency detection methods and other state-of-the-art salient object detection models based on deep learning. This technology shows great potential for pest detection and assessing the severity of pest infestation based on pest density in grain storage facilities. It also holds promise for the prevention and control of pests in agriculture and forestry.

Saliency Detection Based on Multiple-Level Feature Learning.

Finding the most interesting areas of an image is the aim of saliency detection. Conventional methods based on low-level features rely on biological cues like texture and color. These methods, however, have trouble with processing complicated or low-contrast images. In this paper, we introduce a deep neural network-based saliency detection method. First, using semantic segmentation, we construct a pixel-level model that gives each pixel a saliency value depending on its semantic category. Next, we create a region feature model by combining both hand-crafted and deep features, which extracts and fuses the local and global information of each superpixel region. Third, we combine the results from the previous two steps, along with the over-segmented superpixel images and the original images, to construct a multi-level feature model. We feed the model into a deep convolutional network, which generates the final saliency map by learning to integrate the macro and micro information based on the pixels and superpixels. We assess our method on five benchmark datasets and contrast it against 14 state-of-the-art saliency detection algorithms. According to the experimental results, our method performs better than the other methods in terms of F-measure, precision, recall, and runtime. Additionally, we analyze the limitations of our method and propose potential future developments.

An efficient targeted design for real-time defect detection of surface defects

In practical industrial applications, the inference speed of deep learning models directly affects the efficiency of industrial production. Therefore, the lightweight real-time detection method of surface defects is an essential task in the industrial process. We need to achieve a favorable balance between efficiency and accuracy since the rising demand for production efficiency. However, most of the existing pixel-level detection methods 1) often adopt huge computational overhead to learn rich features, resulting in slow inference speed and 2) show a performance degradation when applied to different industrial surface defect scenarios. To this end, we propose an efficient targeted design (ETD) for real-time defect detection of surface defects. It consists of two branches: (i) an efficient feature enhancement branch, with global aggregation module (GAM) and cross-scale guide module (CGM) to gradually enhance defect features, and (ii) an edge posterior branch, with verification module (VM) and scale interaction module (SIM) to implicitly guide the boundary details of defects. Specifically, while inheriting this framework, we reconsider the relationship between precision, parameters, and speed so that our model can be applied to different industrial scenarios. Extensive experimental results on four datasets indicate that ETD outperforms other leading saliency detection methods. Meanwhile, our method ETD-S achieves 347 FPS on ESDIs-SOD dataset, 254 FPS on Crack500 dataset, 227 FPS on NRSD-MN dataset and 273 FPS on DAGM dataset. Additionally, we conduct real-time analysis of ETD on an intelligent paradigm for industrial surface defect detection, further demonstrating its efficacy in practical scenarios. ETD demonstrates effective detection performance while achieving a lightweight architecture, which can be implemented using various deep learning frameworks, showcasing substantial potential for real-time surface defect detection. The source code and dataset are publicly available at https://github.com/VDT-2048/ETD.

Video saliency-recognition by applying custom spatio temporal fusion technique

<span lang="EN-US">Video saliency detection is a major growing field with quite few contributions to it. The general method available today is to conduct frame wise saliency detection and this leads to several complications, including an incoherent pixel-based saliency map, making it not so useful. This paper provides a novel solution to saliency detection and mapping with its custom spatio-temporal fusion method that uses frame wise overall motion colour saliency along with pixel-based consistent spatio-temporal diffusion for its temporal uniformity. In the proposed method section, it has been discussed how the video is fragmented into groups of frames and each frame undergoes diffusion and integration in a temporary fashion for the colour saliency mapping to be computed. Then the inter group frame are used to format the pixel-based saliency fusion, after which the features, that is, fusion of pixel saliency and colour information, guide the diffusion of the spatio temporal saliency. With this, the result has been tested with 5 publicly available global saliency evaluation metrics and it comes to conclusion that the proposed algorithm performs better than several state-of-the-art saliency detection methods with increase in accuracy with a good value margin. All the results display the robustness, reliability, versatility and accuracy.</span>

Hybrid time-spatial video saliency detection method to enhance human action recognition systems

Since digital media has become increasingly popular, video processing has expanded in recent years. Video processing systems require high levels of processing, which is one of the challenges in this field. Various approaches, such as hardware upgrades, algorithmic optimizations, and removing unnecessary information, have been suggested to solve this problem. This study proposes a video saliency map based method that identifies the critical parts of the video and improves the system’s overall performance. Using an image registration algorithm, the proposed method first removes the camera’s motion. Subsequently, each video frame’s color, edge, and gradient information are used to obtain a spatial saliency map. Combining spatial saliency with motion information derived from optical flow and color-based segmentation can produce a saliency map containing both motion and spatial data. A nonlinear function is suggested to properly combine the temporal and spatial saliency maps, which was optimized using a multi-objective genetic algorithm. The proposed saliency map method was added as a preprocessing step in several Human Action Recognition (HAR) systems based on deep learning, and its performance was evaluated. Furthermore, the proposed method was compared with similar methods based on saliency maps, and the superiority of the proposed method was confirmed. The results show that the proposed method can improve HAR efficiency by up to 6.5% relative to HAR methods with no preprocessing step and 3.9% compared to the HAR method containing a temporal saliency map.

Saliency detection of textured 3D models based on multi-view information and texel descriptor

Saliency-driven mesh simplification methods have shown promising results in maintaining visual detail, but effective simplification requires accurate 3D saliency maps. The conventional mesh saliency detection method may not capture salient regions in 3D models with texture. To address this issue, we propose a novel saliency detection method that fuses saliency maps from multi-view projections of textured models. Specifically, we introduce a texel descriptor that combines local convexity and chromatic aberration to capture texel saliency at multiple scales. Furthermore, we created a novel dataset that reflects human eye fixation patterns on textured models, which serves as an objective evaluation metric. Our experimental results demonstrate that our saliency-driven method outperforms existing approaches on several evaluation metrics. Our method source code can be accessed at https://github.com/bkballoon/mvsm-fusion and the dataset can be accessed at 10.5281/zenodo.8131602.

Drogue detection for autonomous aerial refueling via hybrid pigeon-inspired optimized color opponent and saliency aggregation

Drogue detection is one of the challenging tasks in autonomous aerial refueling due to the requirement for accuracy and rapidity. Saliency detection based on image intrinsic cues can achieve fast detection, but with poor accuracy. Recent studies reveal that optimization-based methods provide accurate and quick solutions for saliency detection. This paper presents a hybrid pigeon-inspired optimization method, the optimized color opponent, that aims to adjust the weight of color opponent channels to detect the drogue region. It can optimize the weights in the selected aerial refueling scene offline, and the results are applied for drogue detection in the scene. A novel algorithm aggregated by the optimized color opponent and robust background detection is presented to provide better precision and robustness. Experimental results on benchmark datasets and aerial refueling images show that the proposed method successfully extracts the saliency region or drogue and exhibits superior performance against the other saliency detection methods with intrinsic cues. The algorithm designed in this paper is competent for the drogue detection task of autonomous aerial refueling.

A three-layer decomposition method based on structural texture perception for fusion of CT and MRI images

The fusion of computed tomography (CT) images and magnetic resonance imaging (MRI) images has made essential contribution to clinical diagnosis because of its ability of complementary visual information enhancement and redundancy elimination. However, some methods that involve upsampling and downsampling operations may lose information during image processing, which affects the fusion results. In this paper, a medical image fusion algorithm based on structural texture perception is proposed. The proposed method demonstrates enhanced performance in preserving both detailed texture information and energy information of source images. First, the source image is decomposed into a signal strength layer, a base layer, and a texture layer based on the proposed three-layer decomposition framework. Then, the signal strength layers are fused using a saliency detection method based on iterative least squares. The texture layer is fused using the principle of spatial frequency maximization. For the fusion of the base layer, a non-linear function is designed to calculate the fusion weights. Finally, the final fusion result is obtained through image reconstruction. The proposed algorithm is compared with nine state-of-the-art fusion algorithms to verify its superiority. Experimental results show that the proposed method can effectively preserve the intensity information of CT images and the detailed texture of MRI images.

Exploring Focus and Depth-Induced Saliency Detection for Light Field.

An abundance of features in the light field has been demonstrated to be useful for saliency detection in complex scenes. However, bottom-up saliency detection models are limited in their ability to explore light field features. In this paper, we propose a light field saliency detection method that focuses on depth-induced saliency, which can more deeply explore the interactions between different cues. First, we localize a rough saliency region based on the compactness of color and depth. Then, the relationships among depth, focus, and salient objects are carefully investigated, and the focus cue of the focal stack is used to highlight the foreground objects. Meanwhile, the depth cue is utilized to refine the coarse salient objects. Furthermore, considering the consistency of color smoothing and depth space, an optimization model referred to as color and depth-induced cellular automata is improved to increase the accuracy of saliency maps. Finally, to avoid interference of redundant information, the mean absolute error is chosen as the indicator of the filter to obtain the best results. The experimental results on three public light field datasets show that the proposed method performs favorably against the state-of-the-art conventional light field saliency detection approaches and even light field saliency detection approaches based on deep learning.

Retracted: Intelligent Image Saliency Detection Method Based on Convolution Neural Network Combining Global and Local Information

Retracted: Intelligent Image Saliency Detection Method Based on Convolution Neural Network Combining Global and Local Information

Video saliency-detection using custom spatiotemporal fusion method

There have been several researches done in the field of image saliency but not as much as in video saliency. In order to increase precision and accuracy during compression, reduce coding complexity and time consumption along with memory allocation problems with our proposed solution. It is a modified high-definition video compression (HEVC) pixel based consistent spatiotemporal diffusion with temporal uniformity. It involves taking apart the video into groups of frames, computing colour saliency, integrate temporal fusion, pixel saliency fusion is conducted and then colour information guides the diffusion process for the spatiotemporal mapping with the help of permutation matrix. The proposed solution is tested on a publicly available extensive dataset with five global saliency valuation metrics and is compared with several other state-of-the-art saliency detection methods. The results display and overall best performance amongst all other candidates.

Looking at Boundary: Siamese Densely Cooperative Fusion for Salient Object Detection

Though deep learning-based saliency detection methods have achieved gratifying performance recently, the predicted saliency maps still suffer from the boundary challenge. From the perspective of foreground-background separation, this article attempts to extract the edge information of objects by exploiting the difference between different color channels in the RGB color space and establishes a novel multicolor contrast extraction (MCE) mechanism to improve the learning ability of exquisite boundary information of the network. To make full use of the MCE outputs and RGB colors, and well depict and capture the complementary information between them, we devise a novel Siamese densely cooperative fusion (DCF) network (SDFNet) for saliency detection, which consists of two effective components: boundary-directed feature learning (BDFL) and DCF. The BDFL provides joint learning for both MCE and RGB modalities through a Siamese network, while the DCF module is devised for complementary feature discovery, in order to effectively combine the features learned from two modalities. Experiments on five well-known benchmark datasets demonstrate that the proposed method outperforms the state-of-the-art approaches in terms of different evaluation metrics. We provide a detailed analysis of these results and indicate that our joint modeling of MCE and RGB colors helps to better capture the object details, especially in the object boundaries.

Multi-scale attention and boundary enhancement with long-range dependency for salient object detection

Existing saliency detection methods have achieved great progress in extracting multi-level features, however it is a challenging problem to catch accurate long-range dependencies that can enhance the accuracy of semantic information. To address this, a Transformer-based multi-scale attention and boundary enhancement with long-range dependency (MSBE) network is proposed in this paper. A multi-scale attention enhancement module (MSAEM) is designed to reduce the redundant or noisy features and generate a high-quality feature representation by integrating multiple attentional features with diverse perspectives. The high-quality features are then fed into the triple Transformer encoder embedding module (TEM) to enhance high-level semantic features by learning long-range dependencies across layers. In the decoder part, a cross-layer feature fusion module (CLFFM) and boundary enhancement module (BEM) are designed to improve the effect of feature fusion and get accurate prediction results. Extensive experiments on six challenging public datasets demonstrate that the proposed method achieves competitive performance.

Maritime Target Saliency Detection for UAV Based on the Stimulation Competition Selection Mechanism of Raptor Vision

A maritime target saliency detection method inspired by the stimulation competition and selection mechanism of raptor vision is presented for the airborne vision system of unmanned aerial vehicle (UAV) in an unknown maritime environment. The stimulation competition and selection mechanism in the visual pathway of raptor vision based on the phenomenon of raptor capturing prey in complex scenes are studied. Then, the mathematical model of the stimulation competition and selection mechanism of raptor vision is established and employed for the salient object detection. Popular image datasets and practical scene datasets are applied to verify the effectiveness of the presented method. Results show that the detection performance of the proposed method is better than that of other comparison methods. The proposed algorithm provides an idea for maritime target salient detection and cross-domain joint mission for UAV or other unmanned equipment.

TBINet: fabric defect detection based on a top-down and bottom-up inference network

A key aspect of textile production is the use of automatic defect detection methods for quality control, which is a critical part of the current fabric defect detection system. Benefiting from the explosive development of convolutional neural networks, it has been proved to be feasible to apply it to salient object detection model in fabric defect detection, but how to learn more robust features and better feature fusion for saliency are still complicated tasks. This article presents a novel saliency detection method for fabric defect detection based on both top-down and bottom-up saliency inference combined with two-way information. The top-down process is to infer high-level saliency by gradually using higher-level and richer semantic features on the basis of the backbone. Then a self-fusion enhanced representation module is proposed, which simulates the parallel processing mechanism with residual connection in a top-down manner to generate robust features and effectively characterize the texture features of complex fabrics. In addition, in the down-top process, the interactive feature fusion module is designed, which is used for coarse-to-fine saliency estimation, where the high-level saliency is gradually integrated with finer lower-level features to obtain a fine-grained result. Finally, fabric defect detection is localized by segmenting the generated saliency map. Extensive experimental results on two kinds of fabric image datasets have demonstrated that the proposed TBINet can localize the defect region with high accuracy, and perform favorably against seven state-of-the-art methods on eight widely tested metrics.

Graph learning model for saliency detection in thermal pedestrian videos

Thermal imaging is becoming popular recently due to its all-weather and all-time capability. However, video saliency detection, an active topic in computer vision, has not been well studied for thermal videos. This paper proposes an effective saliency detection method for thermal pedestrian videos. Unlike graph-based saliency detection methods with fixed graph structures, we introduce a saliency-guided graph learning model (SGL), which integrates a multi-view graph and saliency to achieve joint graph learning and saliency diffusion. Then, we apply SGL to thermal video saliency detection by constructing the objectness descriptor based motion saliency and a multi-view spatiotemporal graph. Furthermore, a challenging thermal pedestrian video dataset IPV with 67 sequences and 1462 frames in total is built to compensate for the insufficient scale and complexity of existing datasets. Extensive experiments on a public dataset and our newly built dataset demonstrate the superior performance of our method. Compared to 11 state-of-the-art (SOTA) video saliency detection methods, the proposed approach achieves the best performance of 74.0 %, 75.1 %, and 1.8 % in F-measure, S-measure, and MAE respectively, on the IPV dataset. It also achieves better performance than SOTA on the public dataset GTFD with 69.9 %, 75.6 %, and 1.1 %, respectively.

Lightweight saliency detection method for real-time localization of livestock meat bones

Existing salient object detection networks are large, have many parameters, are bulky and take up a lot of computational resources. Seriously hinder its application and promotion in boning robot. To solve this problem, this paper proposes a lightweight saliency detection algorithm for real-time localization of livestock meat bones. First, a lightweight feature extraction network based on multi-scale attention is constructed in the encoding stage. To ensure that more adequate salient object features are extracted with fewer parameters. Second, the fusion of jump connections is introduced in the decoding phase. Used to capture fine-grained semantics and coarse-grained semantics at full scale. Finally, we added a residual refinement module at the end of the backbone network. For optimizing salient target regions and boundaries. Experimental results on both publicly available datasets and self-made Pig leg X-ray (PLX) datasets show that. The proposed method is capable of ensuring first-class detection accuracy with 40 times less parameters than the conventional model. In the most challenging SOD dataset. The proposed algorithm in this paper achieves a value of Fωβ of 0.699. And the segmentation of livestock bones can be effectively performed on the homemade PLX dataset. Our model has a detection speed of 5fps on industrial control equipment.

A uniform transformer-based structure for feature fusion and enhancement for RGB-D saliency detection

RGB-D saliency detection integrates information from both RGB images and depth maps to improve the prediction of salient regions under challenging conditions. The key to RGB-D saliency detection is to fully mine and fuse information at multiple scales across the two modalities. Previous approaches tend to apply the multi-scale and multi-modal fusion separately via local operations, which fails to capture long-range dependencies. Here we propose a transformer-based structure to address this issue. The proposed architecture is composed of two modules: an Intra-modality Feature Enhancement Module (IFEM) and an Inter-modality Feature Fusion Module (IFFM). IFFM conducts a sufficient feature fusion by integrating features from multiple scales and two modalities over all positions simultaneously. IFEM enhances feature on each scale by selecting and integrating complementary information from other scales within the same modality before IFFM. We show that transformer is a uniform operation which presents great efficacy in both feature fusion and feature enhancement, and simplifies the model design. Extensive experimental results on five benchmark datasets demonstrate that our proposed network performs favorably against most state-of-the-art RGB-D saliency detection methods. Furthermore, our model is efficient for having relatively smaller FLOPs and model size compared with other methods.

Biological Eagle-eye Inspired Target Detection for Unmanned Aerial Vehicles Equipped with a Manipulator

Inspired by eagle eye mechanisms, the structure and information processing characteristics of the eagle’s visual system are used for the target capture task of an unmanned aerial vehicle (UAV) with a mechanical arm. In this paper, a novel eagle-eye inspired multi-camera sensor and a saliency detection method are proposed. A combined camera system is built by simulating the double fovea structure on the eagle retina. A saliency target detection method based on the eagle midbrain inhibition mechanism is proposed by measuring the static saliency information and dynamic features. Thus, salient targets can be accurately detected through the collaborative work between different cameras of the proposed multi-camera sensor. Experimental results show that the eagle-eye inspired visual system is able to continuously detect targets in outdoor scenes and that the proposed algorithm has a strong inhibitory effect on moving background interference.