High-level Vision Applications Research Articles

Frequency-guidance Collaborative Triple-branch Network for single image dehazing

Image dehazing is a crucial pre-processing step in many high-level vision applications, e.g., crowd surveillance, automatic driving, and remote sensing. Recently, learning-based methods have achieved promising performance by designing various convolutional neural networks (CNNs). However, most existing CNNs are based on a single feature branch and cannot fully utilize the low-frequency and high-frequency features of haze, which inevitably affects the quality of the restored image. To this end, we develop an effective Frequency-guidance Collaborative Triple-branch Network (FCT-Net) for image dehazing. It casts dehazing as a dual-frequency component restoration task and deals with pertinent features based on a triple-branch-like architecture. Specifically, our FCT-Net can be decomposed into four components: low-frequency branch (LFB), high-frequency branch (HFB), progressive fusion branch (PFB), and image reconstruction tail (IRT). The LFB aims to mine more contextual information with dual-path feature modulation block (DFMB), while the HFB aims to recover the missing structural detail via mixed attention residual block (MARB). PFB is a fusion branch built between LFB and HFB, where different feature streams (low- and high-frequency information) can be integrated together through continuous selective progressive fusion (SPF) modules. Finally, our IRT exploits all these derived features to generate the clear haze-free image. Furthermore, we utilize a multi-strategy loss to optimize our FCT-Net from the perspectives of pixel domain and feature domain, which can achieve better image recovery quality. Extensive experiments on different benchmark datasets show that our FCT-Net is able to get comparable results with other state-of-the-art methods.

Rethinking the necessity of image fusion in high-level vision tasks: A practical infrared and visible image fusion network based on progressive semantic injection and scene fidelity

Image fusion aims to integrate complementary characteristics of source images into a single fused image that better serves human visual observation and machine vision perception. However, most existing image fusion algorithms primarily focus on improving the visual appeal of fused images. Although there are some semantic-driven methods that consider semantic requirements of downstream applications, none of them have demonstrated the potential of image-level fusion compared to feature-level fusion, which fulfills high-level vision tasks directly on multi-modal features rather than on a fused image. To overcome these limitations, this paper presents a practical infrared and visible image fusion network based on progressive semantic injection and scene fidelity constraints, termed PSFusion. First of all, the sparse semantic perception branch extracts sufficient semantic features, which are then progressively integrated into the fusion network using the semantic injection module to fulfill the semantic requirements of high-level vision tasks. The scene fidelity path within the scene restoration branch is devised to ensure that the fusion features contain complete information for reconstructing the source images. Additionally, the contrast mask and salient target mask are employed to construct the fusion loss to maintain impressive visual effects of fusion results. In particular, we provide quantitative and qualitative analyses to demonstrate the potential of image-level fusion compared to feature-level fusion for high-level vision tasks. With the rapid advancement of large-scale models, image-level fusion can expeditiously leverage the advantages of multi-modal data and state-of-the-art (SOTA) unimodal segmentation to achieve superior performance. Furthermore, extensive comparative experiments demonstrate the superiority of our PSFusion over SOTA image-level fusion alternatives in terms of visual appeal and high-level semantics. Even under harsh circumstances, our method offers satisfactory fusion results to serve subsequent high-level vision applications. The source code is available at https://github.com/Linfeng-Tang/PSFusion.

Hierarchical Spatiotemporal Graph Regularized Discriminative Correlation Filter for Visual Object Tracking.

Visual object tracking is a fundamental and challenging task in many high-level vision and robotics applications. It is typically formulated by estimating the target appearance model between consecutive frames. Discriminative correlation filters (DCFs) and their variants have achieved promising speed and accuracy for visual tracking in many challenging scenarios. However, because of the unwanted boundary effects and lack of geometric constraints, these methods suffer from performance degradation. In the current work, we propose hierarchical spatiotemporal graph-regularized correlation filters for robust object tracking. The target sample is decomposed into a large number of deep channels, which are then used to construct a spatial graph such that each graph node corresponds to a particular target location across all channels. Such a graph effectively captures the spatial structure of the target object. In order to capture the temporal structure of the target object, the information in the deep channels obtained from a temporal window is compressed using the principal component analysis, and then, a temporal graph is constructed such that each graph node corresponds to a particular target location in the temporal dimension. Both spatial and temporal graphs span different subspaces such that the target and the background become linearly separable. The learned correlation filter is constrained to act as an eigenvector of the Laplacian of these spatiotemporal graphs. We propose a novel objective function that incorporates these spatiotemporal constraints into the DCFs framework. We solve the objective function using alternating direction methods of multipliers such that each subproblem has a closed-form solution. We evaluate our proposed algorithm on six challenging benchmark datasets and compare it with 33 existing state-of-the art trackers. Our results demonstrate an excellent performance of the proposed algorithm compared to the existing trackers.

DesmokeNet: A Two-Stage Smoke Removal Pipeline Based on Self-Attentive Feature Consensus and Multi-Level Contrastive Regularization

In image processing, smoke may degrade visibility and deteriorate the performance of high-level vision applications. Therefore, single image smoke removal is crucial for computer vision. Currently, existing smoke removal algorithms mainly leverage handcrafted priors. Moreover, these methods usually apply haze removal methods to perform smoke removal due to the similarity between smoke and haze. However, these methods cannot sufficiently address the degradation of thick smoke and may suffer from residual smoke and color distortion problems due to the non-global and non-homogeneous distribution of smoke. In this paper, to solve the aforementioned problems, an end-to-end deep neural network called DesmokeNet is proposed. We construct a two-stage recovered pipeline to remove the smoke in different thicknesses. The light and thick smoke is first removed locally by the smoke removal network (SRN). The missing pixels in the thick smoke are then recovered by the pixel compensation network (PCN). Moreover, we proposed the thickness-aware pixel loss and the dark channel loss to suppress the residual smoke. To further increase the discriminative ability of the DesmokeNet, we proposed self-attentive feature consensus loss and multi-level contrastive regularization loss to improve the performance of smoke removal. Finally, to train the proposed method, we construct the first large-scale dataset containing synthetic and real-world data. Extensive experiments show that the proposed method outperforms favorably against other state-of-the-art methods quantitatively and qualitatively.

Accurate and Fast Image Denoising via Attention Guided Scaling.

Image denoising is a classical topic yet still a challenging problem, especially for reducing noise from the texture information. Feature scaling (e.g., downscale and upscale) is a widely practice in image denoising to enlarge receptive field size and save resources. However, such a common operation would lose some visual informative details. To address those problems, we propose fast and accurate image denoising via attention guided scaling (AGS). We find that the main informative feature channel and visual primitives during the scaling should keep similar. We then propose to extract the global channel-wise attention to maintain main channel information. Moreover, we propose to collect global descriptors by considering the entire spatial feature. And we then distribute the global descriptors to local positions of the scaled feature, based on their specific needs. We further introduce AGS for adversarial training, resulting in a more powerful discriminator. Extensive experiments show the effectiveness of our proposed method, where we clearly surpass all the state-of-the-art methods on most popular synthetic and real-world denoising benchmarks quantitatively and visually. We further show that our network contributes to other high-level vision applications and improves their performances significantly.

Deep Learning Based Multi-Channel Intelligent Attack Detection for Data Security

Deep learning methods, e.g., convolutional neural networks (CNNs) and Recurrent Neural Networks (RNNs), have achieved great success in image processing and natural language processing especially in high level vision applications such as recognition and understanding. However, it is rarely used to solve information security problems such as attack detection studied in this paper. Here, we move forward a step and propose a novel multi-channel intelligent attack detection method based on long short term memory recurrent neural networks (LSTM-RNNs). To achieve high detection rate, data preprocessing, feature abstraction, and multi-channel training and detection are seamlessly integrated into an end-to-end detection framework. Data preprocessing provides high-quality data for subsequent processing, then different types of features are extracted from the processed data. Multi-channel processing is used to generate classifiers by training neural networks with different types of features, which preserve attack features of input vectors and classify the attack from normal data. With the results of the classifier's attack detection, we introduce a voting algorithm to decide whether the input data is an attack or not. Experimental results validate that the proposed attack detection method greatly outperforms several attack detection methods that use feature detection and Bayesian or SVM classifiers.

Deep Ego-Motion Classifiers for Compound Eye Cameras.

Compound eyes, also known as insect eyes, have a unique structure. They have a hemispheric surface, and a lot of single eyes are deployed regularly on the surface. Thanks to this unique form, using the compound images has several advantages, such as a large field of view (FOV) with low aberrations. We can exploit these benefits in high-level vision applications, such as object recognition, or semantic segmentation for a moving robot, by emulating the compound images that describe the captured scenes from compound eye cameras. In this paper, to the best of our knowledge, we propose the first convolutional neural network (CNN)-based ego-motion classification algorithm designed for the compound eye structure. To achieve this, we introduce a voting-based approach that fully utilizes one of the unique features of compound images, specifically, the compound images consist of a lot of single eye images. The proposed method classifies a number of local motions by CNN, and these local classifications which represent the motions of each single eye image, are aggregated to the final classification by a voting procedure. For the experiments, we collected a new dataset for compound eye camera ego-motion classification which contains scenes of the inside and outside of a certain building. The samples of the proposed dataset consist of two consequent emulated compound images and the corresponding ego-motion class. The experimental results show that the proposed method has achieved the classification accuracy of 85.0%, which is superior compared to the baselines on the proposed dataset. Also, the proposed model is light-weight compared to the conventional CNN-based image recognition algorithms such as AlexNet, ResNet50, and MobileNetV2.

CAMEL: An Adaptive Camera With Embedded Machine Learning-Based Sensor Parameter Control

Today’s cameras are designed to approximate what they observe in a manner that preserves entropy. However, time critical autonomous applications such as autonomous driving, surveillance and defense systems require task critical information at the highest quality. With rapid advances in frame rates and resolutions, observing scenes at the highest quality raises concerns for the transmission bandwidth. In this paper, we introduce a new paradigm of smart camera that captures only task-critical information at the highest quality. Embedded deep neural network (DNN) algorithms within the camera enhance quality of information through real-time control of sensor parameters. We show the hardware feasibility of this camera by demonstrating a 3D-stacked architecture with a Digital Pixel Sensor (DPS). We demonstrate a number of high-level vision applications that benefit through this task-guided control including object detection, object tracking and activity recognition. Finally, we present the unique challenges faced created as a result of feedback and show how software/hardware innovations can be used to mitigate them.

Distribution-Oriented Aesthetics Assessment With Semantic-Aware Hybrid Network

Image aesthetics assessment has emerged as a hot topic in recent years due to its potential in numerous high-level vision applications. In this paper, distinguished from existing studies relying on a single label, we propose quantifying image aesthetics by a distribution over multiple quality levels. The distribution-based representation characterizes the disagreement among users’ aesthetic preferences regarding the same image, and is also compatible with the traditional task of aesthetic label prediction. Our framework is developed based on fully convolutional networks and enables inputs of varying sizes. In this way, we circumvent the fixed-size constraint of prevalent convolutional neural networks, and avoid the risk of impairing the intrinsic aesthetic appeal of images. Moreover, given the fact that aesthetic perceiving is coupled with semantic understanding, we present a novel semantic-aware hybrid NEtwork (SANE), which harvests the information from object categorization and scene recognition to enhance image aesthetics assessment. Experiments on two benchmark datasets have well verified the effectiveness of our approach in both scenarios of aesthetic distribution prediction and aesthetic label prediction, and highlighted the benefits of input preserving as well as semantic understanding for images.

Unsupervised deep context prediction for background estimation and foreground segmentation

Background estimation is a fundamental step in many high-level vision applications, such as tracking and surveillance. Existing background estimation techniques suffer from performance degradation in the presence of challenges such as dynamic backgrounds, photometric variations, camera jitters, and shadows. To handle these challenges for the purpose of accurate background estimation, we propose a unified method based on Generative Adversarial Network (GAN) and image inpainting. The proposed method is based on a context prediction network, which is an unsupervised visual feature learning hybrid GAN model. Context prediction is followed by a semantic inpainting network for texture enhancement. We also propose a solution for arbitrary region inpainting using the center region inpainting method and Poisson blending technique. The proposed algorithm is compared with the existing state-of-the-art methods for background estimation and foreground segmentation and outperforms the compared methods by a significant margin.

An End-to-End Compression Framework Based on Convolutional Neural Networks

Deep learning, e.g., convolutional neural networks (CNNs), has achieved great success in image processing and computer vision especially in high level vision applications such as recognition and understanding. However, it is rarely used to solve low-level vision problems such as image compression studied in this paper. Here, we move forward a step and propose a novel compression framework based on CNNs. To achieve high-quality image compression at low bit rates, two CNNs are seamlessly integrated into an end-to-end compression framework. The first CNN, named compact convolutional neural network (ComCNN), learns an optimal compact representation from an input image, which preserves the structural information and is then encoded using an image codec (e.g., JPEG, JPEG2000 or BPG). The second CNN, named reconstruction convolutional neural network (RecCNN), is used to reconstruct the decoded image with high-quality in the decoding end. To make two CNNs effectively collaborate, we develop a unified end-to-end learning algorithm to simultaneously learn ComCNN and RecCNN, which facilitates the accurate reconstruction of the decoded image using RecCNN. Such a design also makes the proposed compression framework compatible with existing image coding standards. Experimental results validate that the proposed compression framework greatly outperforms several compression frameworks that use existing image coding standards with state-of-the-art deblocking or denoising post-processing methods.

Motion Segmentation from Surveillance Video using modified Hotelling's T-Square Statistics

Motion segmentation is an important task in video surveillance and in many high-level vision applications. This paper proposes two generic methods for motion segmentation from surveillance video sequences captured from different kinds of sensors like aerial, Pan Tilt and Zoom (PTZ), thermal and night vision. Motion segmentation is achieved by employing Hotelling's T-Square test on the spatial neighborhood RGB color intensity values of each pixel in two successive temporal frames. Further, a modified version of Hotelling's T-Square test is also proposed to achieve motion segmentation. On comparison with Hotelling's TSquare test, the result obtained by the modified formula is better with respect to computational time and quality of the output. Experiments along with the qualitative and quantitative comparison with existing method have been carried out on the standard IEEE PETS (2006, 2009 and 2013) and IEEE Change Detection (2014) dataset to demonstrate the efficacy of the proposed method in the dynamic environment and the results obtained are encouraging.

Real-time, parallel segmentation of high-resolution images on multi-core platforms

Many high-level vision applications apply image segmentation during preprocessing to improve their overall execution efficiency. The applied segmentation technique must, therefore, be highly efficient, or otherwise would counteract any system performance improvement. Existing segmentation approaches often fail to meet real-time processing standards and exhibit extremely slow frame rates when applied to high-resolution images. This paper presents a highly optimized serial implementation of a novel image segmentation approach known as leap segmentation, which achieves frame rates exceeding that of the state-of-the art: it segments more than 80 fps on 640 × 360 images and more than 20 fps on high-resolution (1,280 × 720) images. (All images used for evaluation in this paper use a 24-bit red-green-blue color representation with 8 bits per color). We analyze leap segmentation further for areas of possible parallelization and restructure it for use on a parallel processing system to achieve additional speed-up. On a multi-core, mobile processing system with four threads, our multi-core leap segmentation implementation achieves frame rates of over 114 fps on 640 × 360 images and more than 31 fps on 1,280 × 720 images, thus easily exceeding real-time processing standards.

A Fuzzy-Neural approach for estimation of depth map using focus

Depth map is used for recovery of three-dimensional structure of the object which is required in many high level vision applications. In this paper, we present a new algorithm for the estimation of depth map for three-dimensional shape recovery. This algorithm is based on Fuzzy-Neural approach using shape from focus (SFF). A Fuzzy Inference System (FIS) is designed for the calculation of the depth map and an initial set of membership functions and fuzzy rules are proposed. Then Neural Network is used to train the FIS. The training is done using back propagation algorithm in combination with the least squares method. Hence, a new set of input membership functions are generated while discarding the initial ones. Lastly, the trained FIS is used to obtain final depth map. The results are compared with five other methods including the traditional SFF method and the Focused Image Surface SFF method (FISM). Six different types of objects are used for testing the proposed algorithm.

A novel algorithm for estimation of depth map using image focus for 3D shape recovery in the presence of noise

Three-dimensional shape recovery from one or multiple observations is a challenging problem of computer vision. In this paper, we present a new Focus Measure for the estimation of a depth map using image focus. This depth map can subsequently be used in techniques and algorithms leading to the recovery of a three-dimensional structure of the object, a requirement of a number of high level vision applications. The proposed Focus Measure has shown robustness in the presence of noise as compared to the earlier Focus Measures. This new Focus Measure is based on an optical transfer function implemented in the Fourier domain. The results of the proposed Focus Measure have shown drastic improvements in estimation of a depth map, with respect to the earlier Focus Measures, in the presence of various types of noise including Gaussian, Shot, and Speckle noises. The results of a range of Focus Measures are compared using root mean square error and correlation metric measures.

Consideration of illumination effects and optimization of window size for accurate calculation of depth map for 3D shape recovery

Obtaining an accurate and precise depth map is the ultimate goal for 3D shape recovery. For depth map estimation, one of the most vital parts is the initial selection of the focus measure and processing the images with the selected focus measure. Although, many focus measures have been proposed in the literature but not much attention has been paid to the factors affecting those focus measures as well as the manner the images are processed with those focus measures. In this paper, for accurate calculation of depth map, we consider the effects of illumination on the depth map as well as the selection of the window size for application of the focus measures. The resulting depth map can further be used in techniques and algorithms leading to recovery of three-dimensional structure of the object which is required in many high-level vision applications. It is shown that the illumination effects can directly result in incorrect estimation of depth map if proper window size is not selected during focus measure computation. Further, it is shown that the images need some kind of pre-processing to enhance the dark regions and shadows in the image. For this purpose, an adaptive enhancement algorithm is proposed for pre-processing. In this paper, we prove that without such pre-processing for image enhancement and without the use of proper window size for the estimation of depth maps, it is not possible to obtain the accurate depth map.