Multi-scale Fusion Strategy Research Articles

LFNet: A Novel Bidirectional Recurrent Convolutional Neural Network for Light-Field Image Super-Resolution.

The low spatial resolution of light-field image poses significant difficulties in exploiting its advantage. To mitigate the dependency of accurate depth or disparity information as priors for light-field image super-resolution, we propose an implicitly multi-scale fusion scheme to accumulate contextual information from multiple scales for super-resolution reconstruction. The implicitly multi-scale fusion scheme is then incorporated into bidirectional recurrent convolutional neural network, which aims to iteratively model spatial relations between horizontally or vertically adjacent sub-aperture images of light-field data. Within the network, the recurrent convolutions are modified to be more effective and flexible in modeling the spatial correlations between neighboring views. A horizontal sub-network and a vertical sub-network of the same network structure are ensembled for final outputs via stacked generalization. Experimental results on synthetic and real-world data sets demonstrate that the proposed method outperforms other state-of-the-art methods by a large margin in peak signal-to-noise ratio and gray-scale structural similarity indexes, which also achieves superior quality for human visual systems. Furthermore, the proposed method can enhance the performance of light field applications such as depth estimation.

Deeply Supervised Depth Map Super-Resolution as Novel View Synthesis

Deep convolutional neural network (DCNN) has been successfully applied to depth map super-resolution and outperforms existing methods by a wide margin. However, there still exist two major issues with these DCNN based depth map super-resolution methods that hinder the performance: i) The low-resolution depth maps either need to be up-sampled before feeding into the network or substantial deconvolution has to be used; and ii) The supervision (high-resolution depth maps) is only applied at the end of the network, thus it is difficult to handle large up-sampling factors, such as $\times 8, \times 16$. In this paper, we propose a new framework to tackle the above problems. First, we propose to represent the task of depth map super-resolution as a series of novel view synthesis sub-tasks. The novel view synthesis sub-task aims at generating (synthesizing) a depth map from different camera pose, which could be learned in parallel. Second, to handle large up-sampling factors, we present a deeply supervised network structure to enforce strong supervision in each stage of the network. Third, a multi-scale fusion strategy is proposed to effectively exploit the feature maps at different scales and handle the blocking effect. In this way, our proposed framework could deal with challenging depth map super-resolution efficiently under large up-sampling factors (e.g. $\times 8, \times 16$). Our method only uses the low-resolution depth map as input, and the support of color image is not needed, which greatly reduces the restriction of our method. Extensive experiments on various benchmarking datasets demonstrate the superiority of our method over current state-of-the-art depth map super-resolution methods.

Infrared and Visible Image Fusion Combining Interesting Region Detection and Nonsubsampled Contourlet Transform

The most fundamental purpose of infrared (IR) and visible (VI) image fusion is to integrate the useful information and produce a new image which has higher reliability and understandability for human or computer vision. In order to better preserve the interesting region and its corresponding detail information, a novel multiscale fusion scheme based on interesting region detection is proposed in this paper. Firstly, the MeanShift is used to detect the interesting region with the salient objects and the background region of IR and VI. Then the interesting regions are processed by the guided filter. Next, the nonsubsampled contourlet transform (NSCT) is used for background region decomposition of IR and VI to get a low-frequency and a series of high-frequency layers. An improved weighted average method based on per-pixel weighted average is used to fuse the low-frequency layer. The pulse-coupled neural network (PCNN) is used to fuse each high-frequency layer. Finally, the fused image is obtained by fusing the fused interesting region and the fused background region. Experimental results demonstrate that the proposed algorithm can integrate more background details as well as highlight the interesting region with the salient objects, which is superior to the conventional methods in objective quality evaluations and visual inspection.

Decolorization by Fusion

Decolorization aims at converting color images into grayscale images while preserving their original contrast and color discriminability. In this paper, we introduce an original fusion-based decolorization approach. Our algorithm employs as inputs the three color channels $R$ , $G$ , and $B$ , and an additional input related to the Helmholtz-Kohlrausch effect. To blend those inputs, we adopt a multiscale fusion strategy to prevent the artifacts arising from a pixel-wise application of weight maps and use several weight maps that respectively control saliency, exposure, and saturation. The new operator has been tested successfully on a large data set of both natural and synthetic images. It is competitive compared with modern optimization-based methods in terms of decolorized image visual quality while offering the advantage of being computationally simple, and temporally consistent when decolorizing video sequences.

A multi-scale fusion scheme based on haze-relevant features for single image dehazing

Abstract Outdoor images are often degraded by aerosols suspending in atmosphere in bad weather conditions like haze. To cope with this phenomenon, researchers have proposed many approaches and single image based techniques draw attention mostly. Recently, a fusion-based strategy achieves good results, which derives two enhanced images from single image and blends them to recover haze-free image. However, there are still some deficiencies in the fusion-input images and weight maps, which leads their restoration less natural. In this paper, we propose a multi-scale fusion scheme for single image dehazing. We first use an adaptive color normalization to eliminate a common phenomenon, color distortion, in haze condition. Then two enhanced images, including our newly presented local detail enhanced image, are derived to be blended. Thereafter, five haze-relevant features of dark channel, clarity, saliency, luminance and chromatic are investigated since those can serve as weight maps for fusion. Dark channel, clarity and saliency features are finally selected due to their expression abilities and less interconnection. The fusion is processed with a pyramid strategy layer-by-layer. The multi-scale blended images are combined in a bottom-up manner. At last quantitative experiments demonstrate that our approach is effectiveness and yields better results than other methods.

Change Detection Based on Deep Features and Low Rank

In this letter, we address the problem of change detection for remote sensing images from the perspective of visual saliency computation. The proposed method incorporates low-rank-based saliency computation and deep feature representation. First, multilevel convolutional neural network (CNN) features are extracted for superpixels generated using SLIC, in which a fixed-size CNN feature can be formed to represent each superpixel. Then, low-rank decomposition is applied to the change features of the two input images to generate saliency maps that indicate change probabilities of each pixel. Finally, binarized change map can be obtained with a simple threshold. To deal with scale variations, a multiscale fusion strategy is employed to produce more reliable detection results. Extensive experiments on Google Earth and GF-2 images demonstrate the feasibility and effectiveness of the proposed method.

Multi-Layer Model Based on Multi-Scale and Multi-Feature Fusion for SAR Images

A multi-layer classification approach based on multi-scales and multi-features (ML–MFM) for synthetic aperture radar (SAR) images is proposed in this paper. Firstly, the SAR image is partitioned into superpixels, which are local, coherent regions that preserve most of the characteristics necessary for extracting image information. Following this, a new sparse representation-based classification is used to express sparse multiple features of the superpixels. Moreover, a multi-scale fusion strategy is introduced into ML–MFM to construct the dictionary, which allows complementation between sample information. Finally, the multi-layer operation is used to refine the classification results of superpixels by adding a threshold decision condition to sparse representation classification (SRC) in an iterative way. Compared with traditional SRC and other existing methods, the experimental results of both synthetic and real SAR images have shown that the proposed method not only shows good performance in quantitative evaluation, but can also obtain satisfactory and cogent visualization of classification results.

Color Balance and Fusion for Underwater Image Enhancement.

We introduce an effective technique to enhance the images captured underwater and degraded due to the medium scattering and absorption. Our method is a single image approach that does not require specialized hardware or knowledge about the underwater conditions or scene structure. It builds on the blending of two images that are directly derived from a color-compensated and white-balanced version of the original degraded image. The two images to fusion, as well as their associated weight maps, are defined to promote the transfer of edges and color contrast to the output image. To avoid that the sharp weight map transitions create artifacts in the low frequency components of the reconstructed image, we also adapt a multiscale fusion strategy. Our extensive qualitative and quantitative evaluation reveals that our enhanced images and videos are characterized by better exposedness of the dark regions, improved global contrast, and edges sharpness. Our validation also proves that our algorithm is reasonably independent of the camera settings, and improves the accuracy of several image processing applications, such as image segmentation and keypoint matching.

Multi-Scale Analysis Strategies in PRNU-Based Tampering Localization

Accurate unsupervised tampering localization is one of the most challenging problems in digital image forensics. In this paper, we consider a photo response non-uniformity analysis and focus on the detection of small forgeries. For this purpose, we adopt a recently proposed paradigm of multi-scale analysis and discuss various strategies for its implementation. First, we consider a multi-scale fusion approach, which involves combination of multiple candidate tampering probability maps into a single, more reliable decision map. The candidate maps are obtained with sliding windows of various sizes and thus allow to exploit the benefits of both the small- and large-scale analyses. We extend this approach by introducing modulated threshold drift and content-dependent neighborhood interactions, leading to improved localization performance with superior shape representation and easier detection of small forgeries. We also discuss two novel alternative strategies: a segmentation-guided approach, which contracts the decision statistic to a central segment within each analysis window and an adaptive-window approach, which dynamically chooses analysis window size for each location in the image. We perform extensive experimental evaluation on both synthetic and realistic forgeries and discuss in detail practical aspects of parameter selection. Our evaluation shows that the multi-scale analysis leads to significant performance improvement compared with the commonly used single-scale approach. The proposed multi-scale fusion strategy delivers stable results with consistent improvement in various test scenarios.

Robust degraded face recognition with novel face representation and multiscale fusion

Recognizing degraded faces from low-resolution and blurry images is a common yet challenging task. We propose appealing solutions to this problem without any image reconstruction, without any limitation to blur type, and only using high-quality samples to design a classifier. Short-term Fourier transform (STFT) is an effective and concise transform for face recognition, yet its effectiveness depends on two important issues: one is face representation construction from STFT; the other is a scale-named window size of STFT. To deal with the first issue, we explore the increased discrimination brought by joint coding and using multiple frequency combinations. Specifically, we propose a novel local descriptor in which information of a pixel coming from two frequencies is jointly encoded and multiple two-frequency combinations are jointly utilized so as to construct a more descriptive and discriminative face representation. To deal with the second issue, we propose a multiscale fusion strategy that extracts multiple descriptors corresponding with multiple window sizes of STFT followed by equal weighted summation of outputs given by multiple scales. The experiments conducted on face databases confirm that state-of-the-art performance has been achieved by the proposed novel face representation and multiscale fusion strategy.

Data Fusion Strategy for Multiscale Surface Measurements

Interdisciplinary research efforts have started focusing on the development of multiscale models and development of designer multiscale surfaces exhibiting specific properties at different scales for a specific purpose. With the rapid evolution of these new engineered surfaces for microelectromechanical systems (MEMS), microfluidics, etc., there is a strong need for developing tools to measure and characterize these surfaces at different scales. In order to obtain all meaningful details of the surface at various required scales, one is left with the only option of measuring the surface using multiple technologies using a combination of instruments. The majority of hardware-based approaches focus on the development of systems housing multiple technologies/capabilities into a single frame. These systems enable the user to obtain different surface maps using various technologies, but the user does not readily have the ability to combine all the obtained data into one single dataset. The effective approach toward multiscale measurement and characterization would be to use the individual measurement tools and finding a method to relate the individual coordinate systems and use an offline virtual tool to unify, manipulate, segment, merge, and retrieve data. Shape primitives and focus-based fusion strategies cannot be used as every data point in the data sets under consideration has to be treated as essentially at optimal focus. A multiscale data fusion strategy results in edge effects on nonplanar and high aspect ratio surfaces. An optimized fusion strategy, the “FWR method,” for the surface metrology domain is proposed where the subimages obtained from discrete wavelet frame (DWF) were separated into three regimes—form, waviness, and roughness—and fusion was not performed on subimages in the form regime. This approach effectively eliminates the edge effects. Individual data-point-level fusion was successfully demonstrated on Fresnel microlens array surface data as a case study of a nondirectional engineered surface with high aspect ratio.