Multi-Scale Recurrent Network Research Articles

High Efficiency Deep-learning Based Video Compression

Although deep learning technique has achieved significant improvement on image compression, but its advantages are not fully explored in video compression, which leads to the performance of deep-learning-based video compression (DLVC) is obviously inferior to that of hybrid video coding framework. In this article, we proposed a novel network to improve the performance of DLVC from its most important modules, including Motion Process (MP), Residual Compression (RC), and Frame Reconstruction (FR). In MP, we design a split second-order attention and multi-scale feature extraction module to fully remove the warping artifacts from multi-scale feature space and pixel space, which can help reduce the distortion in the following process. In RC, we propose a channel selection mechanism to gradually drop redundant information while preserving informative channels for a better rate-distortion performance. Finally, in FR, we introduce a residual multi-scale recurrent network to improve the quality of the current reconstructed frame by progressively exploiting temporal context information between it and its several previous reconstructed frames. Extensive experiments are conducted on the three widely used video compression datasets (HEVC, UVG, and MCL-JVC), and the performance demonstrates the superiority of our proposed approach over the state-of-the-art methods.

Multivariate Time Series Forecasting Using Multiscale Recurrent Networks With Scale Attention and Cross-Scale Guidance.

Multivariate time series (MTS) forecasting is considered as a challenging task due to complex and nonlinear interdependencies between time steps and series. With the advance of deep learning, significant efforts have been made to model long-term and short-term temporal patterns hidden in historical information by recurrent neural networks (RNNs) with a temporal attention mechanism. Although various forecasting models have been developed, most of them are single-scale oriented, resulting in scale information loss. In this article, we seamlessly integrate multiscale analysis into deep learning frameworks to build scale-aware recurrent networks and propose two multiscale recurrent network (MRN) models for MTS forecasting. The first model called MRN-SA adopts a scale attention mechanism to dynamically select the most relevant information from different scales and simultaneously employs input attention and temporal attention to make predictions. The second one named as MRN-CSG introduces a novel cross-scale guidance mechanism to exploit the information from coarse scale to guide the decoding process at fine scale, which results in a lightweight and more easily trained model without obvious loss of accuracy. Extensive experimental results demonstrate that both MRN-SA and MRN-CSG can achieve state-of-the-art performance on five typical MTS datasets in different domains. The source codes will be publicly available at https://github.com/qguo2010/MRN.

Capsule Endoscopy Image Deblurring Method Based on Multi-scale Recurrent Network

During the process of acquiring capsule endoscope images, image motion blur may result from errors made by the operating physician. A multi-scale recurrent attention network is proposed to address the issue of motion blur in collected images. Considering the issue of over-exposure during image acquisition in capsule endoscopy, which can adversely affect network generation results, an image preprocessing module has been incorporated into the network to eliminate highlights caused by over-exposure in input images. To enhance network sampling efficiency, the encoder is equipped with a Convolutional Block Attention Module (CBAM) to extract fuzzy features more effectively. Additionally, depthwise separable convolution is employed to reduce parameter count. Since the structure of images obtained through capsule endoscopy is relatively simple and stable, we have opted for the mean square error as our loss function due to its heightened sensitivity to data errors. The experimental results demonstrate that the proposed network achieves a Peak Signal-to-Noise Ratio (PSNR) of 31.096 and a Structural Similarity Index Measure (SSIM) of 0.925, indicating a significant improvement over DeblurGAN-v2 and Pix2pix.

Multi-Scale Cyclic Image Deblurring Based on PVC-Resnet

Aiming at the non-uniform blurring of image caused by optical system defects or external interference factors, such as camera shake, out-of-focus, and fast movement of object, a multi-scale cyclic image deblurring model based on a parallel void convolution-Resnet (PVC-Resnet) is proposed in this paper, in which a multi-scale recurrent network architecture and a coarse-to-fine strategy are used to restore blurred images. The backbone network is built based on Unet codec architecture, where a PVC-Resnet module designed by combinations of parallel dilated convolution and residual network is constructed in the encoder of the backbone network. The convolution receptive field is expanded with parallel dilated convolution to extract richer global features. Besides, a multi-scale feature extraction module is designed to extract the shallow features of different scale targets in blurred images, and then the extracted features are sent to the backbone network for feature refinement. The SSIM loss function and the L1 loss function are combined to construct the SSIM-L1 joint loss function for the optimization of the overall network to ensure that the image restoration at different stages can be optimized. The experimental results show that the average peak signal-to-noise ratio (PSNR) of the proposed model on different data sets is as high as 32.84 dB, and the structural similarity (SSIM) reaches 0.9235. and statistical structural similarity (Stat-SSIM) of 0.9249 on different datasets. Compared with other methods, the deblurred images generated by this method are superior to the methods proposed by Nah et al., Kupyn et al. and Cho S J et al., especially on the calibration board data set. The model proposed in this paper applies parallel dilated convolution and SSIM-L1 joint loss function to improve the performance of the network so that the edge and texture details of the restored image are clearer.

Modelling novelty detection in the thalamocortical loop.

In complex natural environments, sensory systems are constantly exposed to a large stream of inputs. Novel or rare stimuli, which are often associated with behaviorally important events, are typically processed differently than the steady sensory background, which has less relevance. Neural signatures of such differential processing, commonly referred to as novelty detection, have been identified on the level of EEG recordings as mismatch negativity (MMN) and on the level of single neurons as stimulus-specific adaptation (SSA). Here, we propose a multi-scale recurrent network with synaptic depression to explain how novelty detection can arise in the whisker-related part of the somatosensory thalamocortical loop. The "minimalistic" architecture and dynamics of the model presume that neurons in cortical layer 6 adapt, via synaptic depression, specifically to a frequently presented stimulus, resulting in reduced population activity in the corresponding cortical column when compared with the population activity evoked by a rare stimulus. This difference in population activity is then projected from the cortex to the thalamus and amplified through the interaction between neurons of the primary and reticular nuclei of the thalamus, resulting in rhythmic oscillations. These differentially activated thalamic oscillations are forwarded to cortical layer 4 as a late secondary response that is specific to rare stimuli that violate a particular stimulus pattern. Model results show a strong analogy between this late single neuron activity and EEG-based mismatch negativity in terms of their common sensitivity to presentation context and timescales of response latency, as observed experimentally. Our results indicate that adaptation in L6 can establish the thalamocortical dynamics that produce signatures of SSA and MMN and suggest a mechanistic model of novelty detection that could generalize to other sensory modalities.

Closed-set automatic speaker identification using multi-scale recurrent networks in non-native children.

Children may benefit from automatic speaker identification in a variety of applications, including child security, safety, and education. The key focus of this study is to develop a closed-set child speaker identification system for non-native speakers of English in both text-dependent and text-independent speech tasks in order to track how the speaker's fluency affects the system. The multi-scale wavelet scattering transform is used to compensate for concerns like the loss of high-frequency information caused by the most widely used mel frequency cepstral coefficients feature extractor. The proposed large-scale speaker identification system succeeds well by employing wavelet scattered Bi-LSTM. While this procedure is used to identify non-native children in multiple classes, average values of accuracy, precision, recall, and F-measure are being used to assess the performance of the model in text-independent and text-dependent tasks, which outperforms the existing models.

UNet Based Multi-Scale Recurrent Network for Lightweight Video Deblurring

UNet Based Multi-Scale Recurrent Network for Lightweight Video Deblurring

Blind motion deblurring via [formula omitted] sparse representation

The method of blind deblurring based on machine learning can effectively deal with the blurred images in the real world. However, existing multi-level architectures can lead to problems such as the inability to reserve edges, the expected introduction of artifacts and ghosts when deblurring. Multi methods have found that using L0 norm to realize image sparse representation can help keeping main structure of images. In this paper, an edge extraction module based on L0 sparse representation is proposed to preserve the edge of images, which is embedded in a multi-scale recurrent network(SRN). When the current scale transmits information to the next scale, edge enhancement is performed using the edge extraction module. Furthermore, considering the correlation among pixels and the correlation among channels, we introduce dual-attention mechanism into the encoder-decoder structure. The deblurring experiment was carried out on the GOPRO dataset. Comparing with 5 state-of-art methods qualitatively and quantitatively, the experimental results show that the proposed method can better preserve the image edges and effectively avoid the artifact of the image. And the peak signal-to-noise ratio of the proposed method are improved compared with other methods.

Octave Residual Network for Image Motion Deblurring

Image motion deblurring technology is an essential topic. Traditional methods with a high algorithm complexity when estimating the blur kernel require a lot of calculations. To solve this problem, we proposed Octave convolution residual block, and proposed Octave Residual Network (ORN) based on this block. ORN is a Generative Adversarial Networks, where the generator is a multi-scale recurrent network, and the discriminator is a deep neural network. ORN restore sharp images in an end-to-end manner where blur is caused by various sources. We present multi-scale loss function that mimics conventional coarse-to-fine approaches. To verify the validity of ORN, we trained our model on GOPRO dataset, the peak signal-to-noise ratio (PSNR) of the model reached 30.04, and the operation time of each step is reduced by 0.04 seconds. Experimental results show that our method achieves the state-of-the-art performance of image motion deblurring not only in performance but also in training speed.

Deep Multi-Scale Recurrent Network for Synthetic Aperture Radar Images Despeckling

For the existence of speckles, many standard optical image processing methods, such as classification, segmentation, and registration, are restricted to synthetic aperture radar (SAR) images. In this work, an end-to-end deep multi-scale recurrent network (MSR-net) for SAR image despeckling is proposed. The multi-scale recurrent and weights sharing strategies are introduced to increase network capacity without multiplying the number of weights parameters. A convolutional long short-term memory (convLSTM) unit is embedded to capture useful information and helps with despeckling across scales. Meanwhile, the sub-pixel unit is utilized to improve the network efficiency. Besides, two criteria, edge feature keep ratio (EFKR) and feature point keep ratio (FPKR), are proposed to evaluate the performance of despeckling capacity for SAR, which can assess the retention ability of the despeckling algorithm to edge and feature information more effectively. Experimental results show that our proposed network can remove speckle noise while preserving the edge and texture information of images with low computational costs, especially in the low signal noise ratio scenarios. The peak signal to noise ratio (PSNR) of MSR-net can outperform traditional despeckling methods SAR-BM3D (Block-Matching and 3D filtering) by more than 2 dB for the simulated image. Furthermore, the adaptability of optical image processing methods to real SAR images can be enhanced after despeckling.