Visual Saliency Algorithm Research Articles

Research on Texture Feature Recognition of Regional Architecture Based on Visual Saliency Model

Architecture is a representative of a city. It is also a spatial carrier of urban culture. Identifying the architectural features in a city can help with urban transformation and promote urban development. The use of visual saliency models in regional architectural texture recognition can effectively enhance the effectiveness of regional architectural texture recognition. In this paper, the improved visual saliency model first enhances the texture images of regional buildings through histogram enhancement technology, and uses visual saliency algorithms to extract the visual saliency of the texture features of regional buildings. Then, combined with the maximum interclass difference method of threshold segmentation, the visual saliency image is segmented to achieve accurate target recognition. Finally, the feature factor iteration of the Bag of Visual Words model and the function classification of support vector machines were used to complete the recognition of regional architectural texture features. Through experimental verification, the constructed regional architectural texture feature recognition method based on visual saliency model can effectively enhance the recognition image. This method performs well in boundary contour separation and visual saliency, with an average recognition rate of 0.814 for texture features in different building scenes, indicating high stability.

UAV Image Target Recognition Algorithm based on Machine Learning

Target recognition in aerial operation of unmanned aerial vehicle (UAV) has the characteristics of complex environment, easy deformation of moving image target, single target feature and small volume. To solve these problems, different advantages of visual saliency algorithm in time, space and frequency domains are studied. The candidate region of the target is extracted, and the final target is further determined by using region feature extraction and SVM. The recognition results on the test data sets show that the recall rate of the MDVS algorithm used in the study exceeds 99% on both test sets, and the F1 index is also above 85%. After adding SVM classifier for classification and recognition, the precision rate has been significantly improved, and the values of the three indicators have exceeded 90%. The recognition algorithm proposed for UAV images, compared with several common recognition algorithms, is effective and has better recognition effect and robustness.

Fabric defect detection using cartoon–texture image decomposition model and visual saliency method

Fabric defect detection is of great importance in the modern textile industry. In actual production, subjective factors usually affect manual detection, which is prone to problems such as false and missed detection. With the development of computer vision, a large number of fabric defect automatic detection algorithms have been proposed. Traditional algorithms rely too heavily on setting parameters manually, while deep learning algorithms have expensive training and computing costs. Given these limitations, a new fabric defect detection method based on the latest cartoon texture image decomposition model, visual saliency algorithm, and mathematical morphology, is proposed in this article. A digital image acquisition system is also designed and constructed. Therefore, the self-made dataset used in the experiment is composed of self-collected images and network public images. To further evaluate the performance of the proposed method, this study conducted fabric defect detection experiments and comparison experiments based on the self-made dataset. The results show that this method can successfully detect fabric defects, having high detection accuracy and efficiency. Combining subjective vision and objective evaluation, comparison experiments prove that this method is superior to other common fabric defect detection methods and has the highest value of accuracy and F1-score. This research provides a new method and technical support for fabric defect detection and other fields.

Pancreas segmentation using a dual-input v-mesh network.

Accurate segmentation of the pancreas from abdomen scans is crucial for the diagnosis and treatment of pancreatic diseases. However, the pancreas is a small, soft and elastic abdominal organ with high anatomical variability and has a low tissue contrast in computed tomography (CT) scans, which makes segmentation tasks challenging. To address this challenge, we propose a dual-input v-mesh fully convolutional network (FCN) to segment the pancreas in abdominal CT images. Specifically, dual inputs, i.e., original CT scans and images processed by a contrast-specific graph-based visual saliency (GBVS) algorithm, are simultaneously sent to the network to improve the contrast of the pancreas and other soft tissues. To further enhance the ability to learn context information and extract distinct features, a v-mesh FCN with an attention mechanism is initially utilized. In addition, we propose a spatial transformation and fusion (SF) module to better capture the geometric information of the pancreas and facilitate feature map fusion. We compare the performance of our method with several baseline and state-of-the-art methods on the publicly available NIH dataset. The comparison results show that our proposed dual-input v-mesh FCN model outperforms previous methods in terms of the Dice similarity coefficient (DSC), positive predictive value (PPV), sensitivity (SEN), average surface distance (ASD) and Hausdorff distance (HD). Moreover, ablation studies show that our proposed modules/structures are critical for effective pancreas segmentation.

Detection of Bobbin Yarn Surface Defects by Visual Saliency Analysis

In order to solve the problem of unstable quality of the traditional bobbin yarn surface defect detection, a method for detecting the surface defect of bobbin yarn using visual saliency analysis is proposed. Firstly, the Difference of Gaussian algorithm is used to suppress the texture of the image, and the contrast between the defect area and the background is enhanced. Then, the method of wavelet threshold is applied to filter the noise in the images after the Difference of Gaussian, and the detailed features of the defect area are preserved while the noise interference is eliminated. Finally, the frequency tuned visual saliency algorithm is used to extract and segment the defective regions accurately. After the process about the Difference of Gaussian and wavelet threshold denoising, the original image of the bobbin yarn can make the image highlight the feature information of the defect area on the basis of ensuring elimination of the background texture and noise, and facilitate the detection of the defect area by combining visual saliency analysis. In the detection of the accuracy and integrity of the defect area, the detection method of this paper is better than the control method, and the defect segmentation result is more accurate. The proposed method for detecting the surface defect of the bobbin can not only completely eliminate the influence of the background texture of the bobbin surface, but also accurately and completely detect the surface defect of the bobbin, and the defect details remain intact. In this paper, the effectiveness of the algorithm is further verified by objective tests. The precision and recall rate of the test are 95.15 % and 98.00 %, respectively, which can meet the needs of actual detection.

Dynamic Zero-Parallax-Setting Techniques for Multi-View Autostereoscopic Display

The objective of this paper is to research a dynamic computation of Zero-Parallax-Setting (ZPS) for multi-view autostereoscopic displays in order to effectively alleviate blurry 3D vision for images with large disparity. Saliency detection techniques can yield saliency map which is a topographic representation of saliency which refers to visually dominant locations. By using saliency map, we can predict what attracts the attention, or region of interest, to viewers. Recently, deep learning techniques have been applied in saliency detection. Deep learning-based salient object detection methods have the advantage of highlighting most of the salient objects. With the help of depth map, the spatial distribution of salient objects can be computed. In this paper, we will compare two dynamic ZPS techniques based on visual attention. They are 1) maximum saliency computation by Graphic-Based Visual Saliency (GBVS) algorithm and 2) spatial distribution of salient objects by a convolutional neural networks (CNN)-based model. Experiments prove that both methods can help improve the 3D effect of autostereoscopic displays. Moreover, the spatial distribution of salient objects-based dynamic ZPS technique can achieve better 3D performance than maximum saliency-based method.

The Effect of Eye Movements in Response to Different Types of Scenes Using a Graph-Based Visual Saliency Algorithm

Saliency is the quality of an object that makes it stands out from neighbouring items and grabs viewer attention. Regarding image processing, it refers to the pixel or group of pixels that stand out in an image or a video clip and capture the attention of the viewer. Our eye movements are usually guided by saliency while inspecting a scene. Rapid detection of emotive stimuli an ability possessed by humans. Visual objects in a scene are also emotionally salient. As different images and clips can elicit different emotional responses in a viewer such as happiness or sadness, there is a need to measure these emotions along with visual saliency. This study was conducted to determine whether the existing available visual saliency models can also measure emotional saliency. A classical Graph-Based Visual Saliency (GBVS) model is used in the study. Results show that there is low saliency or salient features in sad movies with at least a significant difference of 0.05 between happy and sad videos as well as a large mean difference of 76.57 and 57.0, hence making these videos less emotionally salient. However, overall visual content does not capture emotional salience. The applied Graph-Based Visual Saliency model notably identified happy emotions but could not analyze sad emotions.

High Efficiency Video Coding Compliant Perceptual Video Coding Using Entropy Based Visual Saliency Model

In past years, several visual saliency algorithms have been proposed to extract salient regions from multimedia content in view of practical applications. Entropy is one of the important measures to extract salient regions, as these regions have high randomness and attract more visual attention. In the context of perceptual video coding (PVC), computational visual saliency models that utilize the charactertistics of the human visual system to improve the compression ratio are of paramount importance. To date, only a few PVC schemes have been reported that use the visual saliency model. In this paper, we conduct the first attempt to utilize entropy based visual saliency models within the high efficiency video coding (HEVC) framework. The visual saliency map generated for each input video frame is optimally thresholded to generate a binary saliency mask. The proposed HEVC compliant PVC scheme adjusts the quantization parameter according to visual saliency relevance at the coding tree unit (CTU) level. Efficient CTU level rate control is achieved by allocating bits to salient and non-salient CTUs by adjusting the quantization parameter values according to their perceptual weighted map. The attention based on information maximization has shown the best performance on newly created ground truth dataset, which is then incorporated in a HEVC framework. An average bitrate reduction of is achieved by the proposed HEVC compliant PVC scheme with the same perceptual quality and a nominal increase in coding complexity of when compared with HEVC reference software. Moreover, the proposed PVC scheme performs better than other HEVC based PVC schemes when encoded at low data rates.

Echoic log-surprise: A multi-scale scheme for acoustic saliency detection

Perceptual signals such as acoustic or visual cues carry a massive amount of information. From a human perspective, this problem is solved by means of cognitive mechanisms related to attention. In particular, saliency is a property of particular stimuli that makes them stand from others to allow the brain to take decisions about their relevance in the process of exploring the world.For artificial intelligence systems it is advantageous to mimic these mechanisms. Visual saliency algorithms have been successfully employed in tasks such as medical diagnosis, detection of violent scenes, environment understanding made by robots, etc. In contrast, computational models of the acoustic saliency mechanisms are less extended. In this context, we propose a novel acoustic saliency algorithm to be used by intelligent and expert systems facing tasks such as sound detection and classification, early alarm, surveillance, robotic exploration of the surroundings, among many other applications.This technique, we termed echoic log-surprise, combines an unsupervised statistical approach based on Bayesian log-surprise and the biological concept of echoic or Auditory Sensory Memory. Our algorithm computes several independent log-surprise cues in parallel considering a wide range of memory values, with the aim of leveraging saliency information from different temporal scales. Then, we explore several statistical metrics to combine these multi-scale signals in a single temporal saliency signal including Renyi entropy, Jensen-Shannon divergence, Cramer or Bhattacharyya distances. We have adopted Acoustic Event Detection tasks as adequate proxies to test its performance. Results show that the proposed echoic log-surprise method outperforms classical acoustic detection techniques commonly deployed in intelligent and expert systems, such as energy thresholding or voice activity detection, and it also achieves better results than some other state-of-the-art acoustic saliency algorithms, such as Kalinli’s and conventional log-surprise.

A Computationally Efficient Visual Saliency Algorithm Suitable for an Analog CMOS Implementation.

Computer vision algorithms are often limited in their application by the large amount of data that must be processed. Mammalian vision systems mitigate this high bandwidth requirement by prioritizing certain regions of the visual field with neural circuits that select the most salient regions. This work introduces a novel and computationally efficient visual saliency algorithm for performing this neuromorphic attention-based data reduction. The proposed algorithm has the added advantage that it is compatible with an analog CMOS design while still achieving comparable performance to existing state-of-the-art saliency algorithms. This compatibility allows for direct integration with the analog-to-digital conversion circuitry present in CMOS image sensors. This integration leads to power savings in the converter by quantizing only the salient pixels. Further system-level power savings are gained by reducing the amount of data that must be transmitted and processed in the digital domain. The analog CMOS compatible formulation relies on a pulse width (i.e., time mode) encoding of the pixel data that is compatible with pulse-mode imagers and slope based converters often used in imager designs. This letter begins by discussing this time-mode encoding for implementing neuromorphic architectures. Next, the proposed algorithm is derived. Hardware-oriented optimizations and modifications to this algorithm are proposed and discussed. Next, a metric for quantifying saliency accuracy is proposed, and simulation results of this metric are presented. Finally, an analog synthesis approach for a time-mode architecture is outlined, and postsynthesis transistor-level simulations that demonstrate functionality of an implementation in a modern CMOS process are discussed.

RIPL

Specialized FPGA implementations can deliver higher performance and greater power efficiency than embedded CPU or GPU implementations for real-time image processing. Programming challenges limit their wider use, because the implementation of FPGA architectures at the register transfer level is time consuming and error prone. Existing software languages supported by high-level synthesis (HLS), although providing a productivity improvement, are too general purpose to generate efficient hardware without the use of hardware-specific code optimizations. Such optimizations leak hardware details into the abstractions that software languages are there to provide, and they require knowledge of FPGAs to generate efficient hardware, such as by using language pragmas to partition data structures across memory blocks. This article presents a thorough account of the Rathlin image processing language (RIPL), a high-level image processing domain-specific language for FPGAs. We motivate its design, based on higher-order algorithmic skeletons, with requirements from the image processing domain. RIPL’s skeletons suffice to elegantly describe image processing stencils, as well as recursive algorithms with nonlocal random access patterns. At its core, RIPL employs a dataflow intermediate representation. We give a formal account of the compilation scheme from RIPL skeletons to static and cyclostatic dataflow models to describe their data rates and static scheduling on FPGAs. RIPL compares favorably to the Vivado HLS OpenCV library and C++ compiled with Vivado HLS. RIPL achieves between 54 and 191 frames per second (FPS) at 100MHz for four synthetic benchmarks, faster than HLS OpenCV in three cases. Two real-world algorithms are implemented in RIPL: visual saliency and mean shift segmentation. For the visual saliency algorithm, RIPL achieves 71 FPS compared to optimized C++ at 28 FPS. RIPL is also concise, being 5x shorter than C++ and 111x shorter than an equivalent direct dataflow implementation. For mean shift segmentation, RIPL achieves 7 FPS compared to optimized C++ on 64 CPU cores at 1.1, and RIPL is 10x shorter than the direct dataflow FPGA implementation.

Deep Salient Feature Based Anti-Noise Transfer Network for Scene Classification of Remote Sensing Imagery

Remote sensing (RS) scene classification is important for RS imagery semantic interpretation. Although tremendous strides have been made in RS scene classification, one of the remaining open challenges is recognizing RS scenes in low quality variance (e.g., various scales and noises). This paper proposes a deep salient feature based anti-noise transfer network (DSFATN) method that effectively enhances and explores the high-level features for RS scene classification in different scales and noise conditions. In DSFATN, a novel discriminative deep salient feature (DSF) is introduced by saliency-guided DSF extraction, which conducts a patch-based visual saliency (PBVS) algorithm using “visual attention” mechanisms to guide pre-trained CNNs for producing the discriminative high-level features. Then, an anti-noise network is proposed to learn and enhance the robust and anti-noise structure information of RS scene by directly propagating the label information to fully-connected layers. A joint loss is used to minimize the anti-noise network by integrating anti-noise constraint and a softmax classification loss. The proposed network architecture can be easily trained with a limited amount of training data. The experiments conducted on three different scale RS scene datasets show that the DSFATN method has achieved excellent performance and great robustness in different scales and noise conditions. It obtains classification accuracy of 98.25%, 98.46%, and 98.80%, respectively, on the UC Merced Land Use Dataset (UCM), the Google image dataset of SIRI-WHU, and the SAT-6 dataset, advancing the state-of-the-art substantially.

A Newly Developed Ground Truth Dataset for Visual Saliency in Videos

Visual saliency models aim to detect important and eye catching portions in a scene by exploiting human visual system characteristics. The effectiveness of visual saliency models is evaluated by comparing saliency maps with a ground truth data set. In recent years, several visual saliency computation algorithms and ground truth data sets have been proposed for images. However, there is lack of ground truth data sets for videos. A new human labeled ground truth is prepared for video sequences that are commonly used in video coding. The selected videos are from different genres including conversational, sports, outdoor, and indoor having low, medium, and high motion. Saliency mask is obtained for each video by nine different subjects, which are asked to label the salient region in each frame in the form of a rectangular bounding box. A majority voting criteria is used to construct a final ground truth saliency mask for each frame. Sixteen different state-of-the-art visual saliency algorithms are selected for comparison and their effectiveness is computed quantitatively on the newly developed ground truth. It is evident from results that multiple kernel learning and spectral residual-based saliency algorithms perform best for different genres and motion-type videos in terms of F-measure and execution time, respectively.

Exploiting visual saliency for assessing the impact of car commercials upon viewers

Content based video indexing and retrieval (CBVIR) is a lively area of research which focuses on automating the indexing, retrieval and management of videos. This area has a wide spectrum of promising applications where assessing the impact of audiovisual productions emerges as a particularly interesting and motivating one. In this paper we present a computational model capable to predict the impact (i.e. positive or negative) upon viewers of car advertisements videos by using a set of visual saliency descriptors. Visual saliency provides information about parts of the image perceived as most important, which are instinctively targeted by humans when looking at a picture or watching a video. For this reason we propose to exploit visual information, introducing it as a new feature which reflects high-level semantics objectively, to improve the video impact categorization results. The suggested salience descriptors are inspired by the mechanisms that underlie the attentional abilities of the human visual system and organized into seven distinct families according to different measurements over the identified salient areas in the video frames, namely population, size, location, geometry, orientation, movement and photographic composition. Proposed approach starts by computing saliency maps for all the video frames, where two different visual saliency detection frameworks have been considered and evaluated: the popular graph based visual saliency (GBVS) algorithm, and a state-of-the-art DNN-based approach. Then, frame-level salience descriptors are extracted from these maps. Next, pooled statistics are used to collapse the obtained frame-level values into video-level descriptors. Finally, a Logistic regression classifier is built upon the subset of video-level features resulting from a feature selection stage. Experimental validation, conducted on a publicly available corpus of 138 commercials collected from YouTube, shows that the proposed salience descriptors are indicative of the impact upon viewers and achieve a similar performance when compared to a method purely based on aesthetics. Besides, the combined approach, exploiting both saliency and aesthetics together, ultimately results in better performance than what can be achieved individually. In addition, the seven families of salience descriptors defined are also compared in terms of classification performance. Finally, a similar study is also performed targeting the distinct pooling techniques used in the video-level feature computation.

Applications of saliency analysis in focus image fusion

In the study of autofocus technology, we propose an image fusion method based on saliency analysis, which can solve the problem of all in focus. First, the focal area in the source image is positioned by the graph-based visual saliency (GBVS) algorithm, and then the watershed and morphological methods are used to obtain the closed area of the saliency graph and the pseudo-focus region is removed. The defocused region is processed by the Shearlet transform, and the SML operator is used to choose the fusion parts. Finally, the precise focused region and the processed defocused region are fused into an all in focus image. Experiments show that the fused image of our method is clear and rich in detail, which has the best visual effect, and improves more than 5% in definition and fusion compared with traditional methods.

GPU accelerated Foreign Object Debris Detection on Airfield Pavement with visual saliency algorithm

We present a GPU-based implementation of visual saliency algorithm to detect foreign object debris(FOD) on airfield pavement with effectiveness and efficiency. Visual saliency algorithm is introduced in FOD detection for the first time. We improve the image signature algorithm to target at FOD detection in complex background of pavement. First, we make pooling operations in obtaining saliency map to improve recall rate. Then, connected component analysis is applied to filter candidate regions in saliency map to get the final targets in original image. Besides, we map the algorithm to GPU-based kernels and data structures. The parallel version of the algorithm is able to get the results with 23.5 times speedup. Experimental results elucidate that the proposed method is effective to detect FOD real-time.

Structure augmented monocular saliency for planetary rovers

This paper proposes a novel object detection method based on the visual saliency model in order to reliably detect objects such as rocks from single monocular planetary images. The algorithm takes advantage of the relatively homogeneous and distinct albedos present in planetary environments such as Mars or the Moon to extract a Digital Terrain Model of a scene using photoclinometry. The Digital Terrain Model is then incorporated into a bottom-up visual saliency algorithm to augment objects that protrude out of the ground. This Structure Augmented Monocular Saliency algorithm (SAMS) improves the accuracy and reliability of detecting objects in a planetary environment with no training requirements, greater robustness and lower computational complexity than 3D saliency models. Comprehensive analysis of the proposed method is performed using three challenging benchmark datasets. The results show that the Structure Augmented Monocular Saliency (SAMS) algorithm performs better than against commonly used visual saliency models on the same datasets.

Multi-focus image fusion algorithm based on focused region extraction

Detection of the focused region is the key issue of the multi-focus image fusion algorithm. The blurred boundary of the focused region increases the difficulty of identifying focused regions accurately. According to the principles that the focused region of the source image should be directly fused as much as possible, this paper proposes a novel multi-focus image fusion algorithm based on saliency analysis, Graph-based visual saliency (GBVS) algorithm is used to locate the focused region in the source image. Then, watershed and morphological methods are used to get the enclosed region of the saliency map and remove the pseudo-focus region. Finally, the focused region is fused directly and the remaining regions are fused based on the Shearlet transform. Experiments demonstrate that the proposed approach can accurately extract the focused region, the fused image preserves more features and it is superior to traditional methods in subjective and objective evaluations.

Visual Salience and Reference Resolution in Simulated 3-D Environments

In this paper we present a novel false colouring-based visual saliency algorithm and illustrate how it is used in the situated language interpreter (SLI) system to ground a reference resolution framework for natural language interfaces to 3-D simulated environments. The visual saliency algorithm allows us to dynamically maintain a model of the evolving visual context. The visual saliency scores associated with the elements in the context model can be used to resolve underspecified references.