Saliency Features Research Articles

Filter-PCA-Based Process Monitoring and Defect Identification During Climbing Helium Arc Welding Process Using DE-SVM

In-process defect detection for climbing helium arc welding of aluminum alloy is challenging due to the highly complex nature of the heat and mass transfer process. This paper presents a process monitoring and defects identification method for weld quality monitoring during climbing helium arc welding based on molten pool visual sensing. High contrast molten pool images were acquired by the designed passive vision sensor. Then, an image processing algorithm based on Otsu's method and visual saliency features was proposed to extract multi-region features from the acquired images in real-time. An optimal feature subset selection approach based on filter and principal component analysis(Filter-PCA) was developed to select the components related to welding defects. Finally, a novel prediction model using Filter-PCA and the support vector machine based on differential evolution algorithm(DE-SVM) was established to identify undercut weld, snake-like weld, and sound weld. The experiment results obtained in the actual production line of 2219 aluminum alloy rocket propellent tank show that the proposed method is remarkably effective and robust. This research provides a technical approach for early identification and early warning of defects during climbing helium arc welding of aluminum alloy. It also provides a basis for stability and online control of weld formation quality under time-varying welding pose.

Salient object detection algorithm based on diversity features and global guidance information

Aiming at the problems of traditional salient object detection methods such as fuzzy boundary and insufficient information integrity, a salient object detection network composed of feature diversity enhancement module, global information guidance module and feature fusion module is proposed. Firstly, asymmetric convolution, cavity convolution and common convolution are spliced to form a feature diversity enhancement module to extract different types of spatial features corresponding to each feature layer. Secondly, the global information guidance module transmits the information captured by the coordinate attention mechanism to each feature layer through the global guidance stream, so as to learn the semantic relationship between different feature layers and alleviate the dilution effect; Finally, the feature fusion module receives the high-level features output from the previous layer, the low-level features of the corresponding layer and the global context information generated by the global information guidance module, and the cascade feature diversity enhancement module gradually integrates the multi-level features to refine the saliency feature map. Comparative experiments on five public data sets show that this method can achieve the highest values of 0.959 and 0.030 in F-measure and MAE. Compared with other seven advanced algorithms, it has better detection performance.

Semantic Segmentation of Transmission Corridor 3D Point Clouds Based on CA-PointNet++

Automated extraction of key points from three-dimensional (3D) point clouds in transmission corridors provides technical support for digital twin construction and risk management of the power grid. However, accurately and efficiently segmenting the point clouds of transmission corridors remains a challenging problem. Traditional segmentation methods for transmission corridors suffer from low accuracy and poor generalization ability, and the potential of deep learning in this field has been overlooked. Therefore, the PointNet++ deep learning model is employed as the backbone network for the segmentation of 3D point clouds in transmission corridors. Additionally, given the distinct distribution of key components, an end-to-end CA-PointNet++ architecture is proposed by integrating the Coordinate Attention (CA) module with PointNet++. This approach captures long-distance spatial contextual features and improves feature saliency for more precise segmentation. Furthermore, CA-PointNet++ is evaluated on a dataset of 3D point clouds collected by unmanned aerial vehicles (UAV) equipped with Light Detection and Ranging (LiDAR) for inspecting transmission corridors. The results show that CA-PointNet++ achieved 93.7% overall accuracy (OA) and 67.4% mean Intersection over Union (mIoU). Comparative studies with established deep learning models confirm that our proposed CA-PointNet++ exhibits high accuracy and strong generalization ability for point cloud segmentation tasks in transmission corridors.

Similarity evaluation of graphic design based on deep visual saliency features

The creativity of an excellent design work generally comes from the inspiration and innovation of its main visual features. The similarity among primary visual elements stands as a paramount indicator when it comes to identifying plagiarism in design concepts. This factor carries immense importance, especially in safeguarding cultural heritage and upholding copyright protection. This paper aims to develop an efficient similarity evaluation scheme for graphic design. A novel deep visual saliency feature extraction generative adversarial network is proposed to deal with the problem of lack of training examples. It consists of two networks: One predicts a visual saliency feature map from an input image and the other takes the output of the first to distinguish whether a visual saliency feature map is a predicted one or ground truth. Unlike traditional saliency generative adversarial networks, a residual refinement module is connected after the encoding and decoding network. Design importance maps generated by professional designers are used to guide the network training. A saliency-based segmentation method is developed to locate the optimal layout regions and notice insignificant regions. Priorities are then assigned to different visual elements. Experimental results show the proposed model obtains state-of-the-art performance among various similarity measurement methods.

Pedestrian Re-ID based on feature consistency and contrast enhancement

For robust person re-identification(Re-ID), the key is effectively learning the features of body parts and their long-distance dependence. ResNet and Transformer are respectively good at learning local dependence and long-distance dependence between region features due to their respective special structures. In order to fully integrate the advantages of the two models, we propose a novel person Re-ID framework that effectively incorporates pixel-level region features, posture-level relation features and the long-distance dependence of region features. Specifically, we design a Semantic Correction Module (SCM) that corrects pixel-level region features and posture-level relation features in a masked manner to generate discriminative fine-grained features with high pose semantics. Considering the semantic inconsistency between relation features and region features, we propose a Contrastive Association Module (CAM) to interactively enhances the long-distance correlation and local saliency of features in a self-attention way. Finally, to improve the robustness of local and global features, we construct a CAM layer to enhance the representation of features based on their potential relationships. Extensive experiment results on general and occlusion datasets demonstrate that our approach performs favorably against the state-of-the-art methods, e.g. 96% Rank-1 on Market-1501.

CasViGE: Learning robust point cloud registration with cascaded visual-geometric encoding

In this work, we present Cascaded Visual-Geometric Encoding (CasViGE), a novel inter-modality feature learning method to improve point cloud registration via the visual information from RGB images. Registering point clouds based on solely the 3D geometric structure suffers from severe matching ambiguity caused by repeated geometric patterns and geometrically less discriminative regions. For this reason, recent methods attempt to inject the visual information from RGB images to obtain more accurate correspondences. They usually first extract the visual and the geometric features independently and then fuse them by point-wise concatenation. However, as 2D and 3D convolutions have different inductive biases, this simplistic method ignores the intrinsic correlation between the two modalities, which harms the distinctiveness of the point descriptors. To address this issue, CasViGE iteratively fuses the inter-modality features by leveraging the inductive biases of both 2D and 3D convolutions, which better considers the correlation between the two modalities. A geometry-centric encoding module first promotes the visual features with 3D convolutions in the geometric space to explicitly embed the geometric information. Next, a vision-centric encoding module casts the fused features back to the image space and enforces the local feature saliency and correlation with 2D convolutions. Extensive experiments on 3DMatch and 3DLoMatch benchmarks have demonstrated the efficacy of our method. Our CasViGE is plug-and-play and attains significant improvements consistently on various point cloud registration methods.

What about the Latent Space? The Need for Latent Feature Saliency Detection in Deep Time Series Classification

Saliency methods are designed to provide explainability for deep image processing models by assigning feature-wise importance scores and thus detecting informative regions in the input images. Recently, these methods have been widely adapted to the time series domain, aiming to identify important temporal regions in a time series. This paper extends our former work on identifying the systematic failure of such methods in the time series domain to produce relevant results when informative patterns are based on underlying latent information rather than temporal regions. First, we both visually and quantitatively assess the quality of explanations provided by multiple state-of-the-art saliency methods, including Integrated Gradients, Deep-Lift, Kernel SHAP, and Lime using univariate simulated time series data with temporal or latent patterns. In addition, to emphasize the severity of the latent feature saliency detection problem, we also run experiments on a real-world predictive maintenance dataset with known latent patterns. We identify Integrated Gradients, Deep-Lift, and the input-cell attention mechanism as potential candidates for refinement to yield latent saliency scores. Finally, we provide recommendations on using saliency methods for time series classification and suggest a guideline for developing latent saliency methods for time series.

Object identification in cerebral visual impairment characterized by gaze behavior and image saliency analysis

Individuals with cerebral visual impairment (CVI) have difficulties identifying common objects, especially when presented as cartoons or abstract images. In this study, participants were shown a series of images of ten common objects, each from five possible categories ranging from abstract black & white line drawings to color photographs. Fifty individuals with CVI and 50 neurotypical controls verbally identified each object and success rates and reaction times were collected. Visual gaze behavior was recorded using an eye tracker to quantify the extent of visual search area explored and number of fixations. A receiver operating characteristic (ROC) analysis was also carried out to compare the degree of alignment between the distribution of individual eye gaze patterns and image saliency features computed by the graph-based visual saliency (GBVS) model. Compared to controls, CVI participants showed significantly lower success rates and longer reaction times when identifying objects. In the CVI group, success rate improved moving from abstract black & white images to color photographs, suggesting that object form (as defined by outlines and contours) and color are important cues for correct identification. Eye tracking data revealed that the CVI group showed significantly greater visual search areas and number of fixations per image, and the distribution of eye gaze patterns in the CVI group was less aligned with the high saliency features of the image compared to controls. These results have important implications in helping to understand the complex profile of visual perceptual difficulties associated with CVI.

Sea surface height data reconstruction via inter and intra layer features based on dual attention

Understanding of geoscience relies on spatio-temporal continuous fields, such as Sea Surface Height (SSH). Data reconstruction for restoring spatio-temporal continuous and gridded maps from incomplete observations and inaccuracy interpolation products has long been a crucial challenge in the marine geoscience. Despite remarkable progress, most existing learning-based reconstruction methods neglect to fully and discriminatively utilize the inter-layer and intra-layer features. The inter-layer semantics are complementary, and the significance of intra-layer components varies with frequency. To address these issues, we propose a multi-layer Feature Combination Network based on Attention mechanism (FCANet) for SSH data reconstruction. Specifically, a novel trainable Multi-layer Feature Combination Block (MFCB) is developed to enrich the features through inter-layer dependencies. Dual attention mechanism is introduced into the MFCB to enhance saliency of intra-layer features by adaptively rescaling the spatial-wise and channel-wise features. Furthermore, experimental results on SSH variable demonstrate the superiority of our FCANet over state-of-the-art reconstruction methods in the metrics of RMSE and SSIM.

Search and insight processes in card sorting games.

Insight problems are particularly interesting, because problems which require restructuring allow researchers to investigate the underpinnings of the Aha-experience, creativity and out of the box thinking. There is a need for new insight tasks to probe and extend the limits of existing theories and cognitive frameworks. To shed more light on this fascinating issue, we addressed the question: Is it possible to convey a well-known card sorting game into an insight task? We introduced different conditions and tested them via two online experiments (N = 546). Between the conditions we systematically varied the available perceptual features, and the existence of non-obvious rules. We found that our card sorting game elicited insight experience. In the first experiment, our data revealed that solution strategies and insight experience varied by the availability and saliency of perceptual features. The discovery of a non-obvious rule, which is not hinted at by perceptual features, was most difficult. With our new paradigm, we were able to construe ambiguous problems which allowed participants to find more than one solution strategy. Interestingly, we realized interindividual preferences for different strategies. The same problem drove strategies which either relied on feature integration or on more deliberate strategies. The second experiment varied the degree of independence of a sorting rule from the standard rules which were in accordance with prior knowledge. It was shown that the more independent the hidden rule was, the more difficult the task became. In sum, we demonstrated a new insight task which extended the available task domains and shed light on sequential and multi-step rule learning problems. Finally, we provided a first sketch of a cognitive model that should help to integrate the data within the existing literature on cognitive models and speculated about the generalizability of the interplay of prior knowledge modification and variation for problem solving.

Multisensor Infrared and Visible Image Fusion via Double Joint Edge Preservation Filter and Nonglobally Saliency Gradient Operator

Multi-sensor infrared and visible image fusion is to extract image features captured by multi-sensor devices to generate a fusion image that contains both infrared thermal radiation and visible texture details. Based on this purpose, this paper proposes a novel infrared and visible image fusion method via double joint edge preservation filter and non-globally saliency gradient operator. First, the source images are finely decomposed by the proposed filter, which has the merits of multi-level scale awareness and double joint edge preservation, into the base, infrared detail, and visible detail layers. Furthermore, to ensure that the overall appearance of the fused image is maximally similar to the source images, a maximum-absolute rule is introduced to process the base layers. Then, a non-globally saliency gradient operator constructed by three different kinds of basic operators is presented to dominate the fusion of the detail layers, aiming to filter out the redundant information and recover the saliency features. Finally, the merged image is obtained by summating the fused base and detail layers. The experimental results demonstrate that the proposed method not only fully extracts the visible details and infrared thermal target of the source images but also generates a fused image with excellent subjective performance and objective indicators compared to other state-of-the-art fusion methods.

Enhancing 360 Video Streaming through Salient Content in Head-Mounted Displays.

Predicting where users will look inside head-mounted displays (HMDs) and fetching only the relevant content is an effective approach for streaming bulky 360 videos over bandwidth-constrained networks. Despite previous efforts, anticipating users' fast and sudden head movements is still difficult because there is a lack of clear understanding of the unique visual attention in 360 videos that dictates the users' head movement in HMDs. This in turn reduces the effectiveness of streaming systems and degrades the users' Quality of Experience. To address this issue, we propose to extract salient cues unique in the 360 video content to capture the attentive behavior of HMD users. Empowered by the newly discovered saliency features, we devise a head-movement prediction algorithm to accurately predict users' head orientations in the near future. A 360 video streaming framework that takes full advantage of the head movement predictor is proposed to enhance the quality of delivered 360 videos. Practical trace-driven results show that the proposed saliency-based 360 video streaming system reduces the stall duration by 65% and the stall count by 46%, while saving 31% more bandwidth than state-of-the-art approaches.

Adaptive Visual Saliency Feature Enhancement of CBCT for Image-Guided Radiotherapy

Unlike the high imaging radiation dose of computed tomography (CT), cone-beam CT (CBCT) has smaller radiation dose and presents less harm to patients. Therefore, CBCT is often used for target delineation, dose planning, and postoperative evaluation in the image-guided radiotherapy (IGRT) of various cancers. In the process of IGRT, CBCT images usually need to be collected multiple times in a radiotherapy stage for postoperative evaluation. The effectiveness of radiotherapy is measured by comparing and analyzing the registered CBCT and the source CT image obtained before radiotherapy. Hence, the registration of CBCT and CT is the most important step in IGRT. CBCT images usually have poor visual effects due to the small imaging dose used, which adversely affects the registration performance. In this paper, we propose a novel adaptive visual saliency feature enhancement method for CBCT in IGRT. Firstly, we denoised CBCT images using a structural similarity based low-rank approximation model (SSLRA) and then enhanced the denoised results with a visual saliency feature enhancement (VSFE)-based method. Experimental results show that the enhancement performance of the proposed method is superior to the comparison enhancement algorithms in visual objective comparison. In addition, the extended experiments prove that the proposed enhancement method can improve the registration accuracy of CBCT and CT images, demonstrating their application prospects in IGRT-based cancer treatment.

Multi-Guidance CNNs for Salient Object Detection

Feature refinement and feature fusion are two key steps in convolutional neural networks–based salient object detection (SOD). In this article, we investigate how to utilize multiple guidance mechanisms to better refine and fuse extracted multi-level features and propose a novel multi-guidance SOD model dubbed as MGuid-Net. Since boundary information is beneficial for locating and sharpening salient objects, edge features are utilized in our network together with saliency features for SOD. Specifically, a self-guidance module is applied to multi-level saliency features and edge features, respectively, which aims to gradually guide the refinement of lower-level features by higher-level features. After that, a cross-guidance module is devised to mutually refine saliency features and edge features via the complementarity between them. Moreover, to better integrate refined multi-level features, we also present an accumulative guidance module, which exploits multiple high-level features to guide the fusion of different features in a hierarchical manner. Finally, a pixelwise contrast loss function is adopted as an implicit guidance to help our network retain more details in salient objects. Extensive experiments on five benchmark datasets demonstrate our model can identify salient regions of an image more effectively compared to most of state-of-the-art models.

Leveraging Saliency in Single-Stage Multi-Label Concrete Defect Detection Using Unmanned Aerial Vehicle Imagery

Visual inspection of concrete structures using Unmanned Areal Vehicle (UAV) imagery is a challenging task due to the variability of defects’ size and appearance. This paper proposes a high-performance model for automatic and fast detection of bridge concrete defects using UAV-acquired images. Our method, coined the Saliency-based Multi-label Defect Detector (SMDD-Net), combines pyramidal feature extraction and attention through a one-stage concrete defect detection model. The attention module extracts local and global saliency features, which are scaled and integrated with the pyramidal feature extraction module of the network using the max-pooling, multiplication, and residual skip connections operations. This has the effect of enhancing the localisation of small and low-contrast defects, as well as the overall accuracy of detection in varying image acquisition ranges. Finally, a multi-label loss function detection is used to identify and localise overlapping defects. The experimental results on a standard dataset and real-world images demonstrated the performance of SMDD-Net with regard to state-of-the-art techniques. The accuracy and computational efficiency of SMDD-Net make it a suitable method for UAV-based bridge structure inspection.

CAE-Net: Cross-Modal Attention Enhancement Network for RGB-T Salient Object Detection

RGB salient object detection (SOD) performs poorly in low-contrast and complex background scenes. Fortunately, the thermal infrared image can capture the heat distribution of scenes as complementary information to the RGB image, so the RGB-T SOD has recently attracted more and more attention. Many researchers have committed to accelerating the development of RGB-T SOD, but some problems still remain to be solved. For example, the defective sample and interfering information contained in the RGB or thermal image hinder the model from learning proper saliency features, meanwhile the low-level features with noisy information result in incomplete salient objects or false positive detection. To solve these problems, we design a cross-modal attention enhancement network (CAE-Net). First, we concretely design a cross-modal fusion (CMF) module to fuse cross-modal features, where the cross-attention unit (CAU) is employed to enhance the two modal features, and channel attention is used to dynamically weigh and fuse the two modal features. Then, we design the joint-modality decoder (JMD) to fuse cross-level features, where the low-level features are purified by higher level features, and multi-scale features are sufficiently integrated. Besides, we add two single-modality decoder (SMD) branches to preserve more modality-specific information. Finally, we employ a multi-stream fusion (MSF) module to fuse three decoders’ features. Comprehensive experiments are conducted on three RGB-T datasets, and the results show that our CAE-Net is comparable to the other methods.

Recognition of aggressive behavior of group-housed pigs based on CNN-GRU hybrid model with spatio-temporal attention mechanism

Aggressive behavior of group-housed pigs seriously affects farm economy and animal welfare. Automatic and accurate recognition of aggressive behavior of group-housed pigs is thus important for farm production management. This study proposes a hybrid model that combines convolutional neural network (CNN) and gated recurrent unit (GRU) to differentiate aggressive and other behaviors from surveillance videos. The CNN network served as a spatial feature extractor to learn appearance representations of behavior in each individual frame, while the GRU network served as a temporal feature extractor to learn motion representations of behavior in a behavior episode. More importantly, to focus on the saliency features in both the spatial domain and the temporal domain of behavior, a specific spatio-temporal attention mechanism was designed and integrated in the CNN-GRU hybrid model to improve the effect of aggressive behavior recognition. To evaluate the proposed model, a behavior video dataset consisting of 5530 behavior episodes about 10 piglets. The accuracy of the proposed hybrid model conducted on the test set were 94.8 %. The results showed that the proposed hybrid model integrated with spatio-temporal attention performed better than the model with an independent spatial or temporal attention and the model without attention mechanism, and achieved a competitive performance of aggressive behavior recognition over the state-of-the-art approaches. We shared our behavior video dataset at https://github.com/IPCLab-NEAU/Aggressive-Behavior-Recognition for precision livestock farming research community.

Infrared-visible image fusion method based on sparse and prior joint saliency detection and LatLRR-FPDE

In this paper, a new infrared and visible image fusion method based on saliency detection and LatLRR-FPDE is proposed to ensure the saliency of the fused target and improve the clarity of the texture detail information of the latent low-rank representation (LatLRR) fusion method. In the feature representation stage, source images are decomposed into low-rank and sparse parts based on LatLRR. Then Fourth order Partial Differential Equation (FPDE) is used to extract high-frequency detail information from the low-rank part. In the fusion decision stage, according to the characteristic difference of the decomposed sub-bands, a multi-scale fusion scheme is carried out using rules such as EM (expectation-maximization), average (AVG), and the maximum absolute value. To guarantee the integrity of thermal radiation features in the target area in fusion results, an infrared image saliency detection algorithm is proposed, which utilizes the fusion information obtained from the sparse and prior saliency feature. Specifically, the algorithm uses convex hull center prior and contrast prior to obtain the background-suppressed saliency map. Meanwhile, to make full use of the high-dimensional saliency features in the image decomposition stage, the prior saliency map and the sparse saliency map are fused to improve the accuracy of saliency detection without the demand to extract redundant feature, which largely increases time-efficiency of the model. Finally, infrared feature compensation is performed on the target region of the fused image based on the saliency mapping strategy to make the final fused image having a better infrared target feature representation. The experiments are based on registered public datasets and compared with various popular image fusion algorithms. The subjective and objective results show that the proposed algorithm can effectively keep the integrity of salient infrared features and has superior visual quality.

Multi-scale saliency features fusion model for person re-identification

Multi-scale saliency features fusion model for person re-identification

BiTCAN: A emotion recognition network based on saliency in brain cognition.

In recent years, with the continuous development of artificial intelligence and brain-computer interfaces, emotion recognition based on electroencephalogram (EEG) signals has become a prosperous research direction. Due to saliency in brain cognition, we construct a new spatio-temporal convolutional attention network for emotion recognition named BiTCAN. First, in the proposed method, the original EEG signals are de-baselined, and the two-dimensional mapping matrix sequence of EEG signals is constructed by combining the electrode position. Second, on the basis of the two-dimensional mapping matrix sequence, the features of saliency in brain cognition are extracted by using the Bi-hemisphere discrepancy module, and the spatio-temporal features of EEG signals are captured by using the 3-D convolution module. Finally, the saliency features and spatio-temporal features are fused into the attention module to further obtain the internal spatial relationships between brain regions, and which are input into the classifier for emotion recognition. Many experiments on DEAP and SEED (two public datasets) show that the accuracies of the proposed algorithm on both are higher than 97%, which is superior to most existing emotion recognition algorithms.