Kernels In Convolutional Neural Networks Research Articles

Learnable Gabor Kernels in Convolutional Neural Networks for Seismic Interpretation Tasks

Learnable Gabor Kernels in Convolutional Neural Networks for Seismic Interpretation Tasks

Unleashing the full potential of hyperspectral imaging: Decoupled image and frequency-domain spatial–spectral framework

Hyperspectral image classification (HSIC) is a rapidly developing field that utilizes deep learning methods. However, the reliance on convolutional neural networks (CNNs) for spectral–spatial feature extraction presents certain limitations. Specifically, the use of the fixed-position convolutional kernels in CNNs hinders their ability to capture fully the spectral information around the spatial central pixel, thereby overlooking critical differences between features. To address this issue, a decoupled image- and frequency-domain spectral–spatial framework for HSIC was developed in this study. This method incorporates image- and frequency-domain-based multiscale learnable convolutional attention to refine the differentiating features of the different feature distributions. Additionally, a novel frequency-domain information enhancement module was designed to extract the structural shape and texture details under the semantic constraints of the frequency phase, complementing the image domain to improve the extracted feature maps. Furthermore, a simple and efficient hierarchical feature representation module was introduced to extract both local and global information effectively from the fused features. The experimental results obtained using three open datasets and a practical hyperspectral image of the Gaofen-5 satellite demonstrate that the proposed method outperforms other state-of-the-art HSIC methods.

Dead pixel test using effective receptive field

Deep neural networks have been used in various fields, but their internal behavior in how they understand images is not well known. In this study, we discuss two counterintuitive properties of convolutional neural networks (CNNs). First, we evaluated the size of the receptive field of CNNs with their classification accuracy. Previous studies have attempted to increase the size of the receptive field for performance gain. However, we observed that some CNNs with a smaller receptive field can achieve higher classification accuracy. In this regard, we claim that a larger receptive field does not guarantee improved classification accuracy. Second, using the effective receptive field, we examined the contribution of each pixel to the output of CNN. Intuitively, each pixel is expected to equally contribute to the final output, but we found that there exist pixels in a partially dead state with little contribution to the output. We reveal that the reason for dead pixels lies in even stride operations with odd-sized kernels in CNN and propose a kernel padding method to remove the dead pixels. We demonstrated the vulnerability of CNNs with dead pixels when we detect a noise or small box that is on dead pixels. Our findings on dead pixels should be understood and considered in practical applications of CNN.

A Low-Power Hardware Architecture for Real-Time CNN Computing.

Convolutional neural network (CNN) is widely deployed on edge devices, performing tasks such as objective detection, image recognition and acoustic recognition. However, the limited resources and strict power constraints of edge devices pose a great challenge to applying the computationally intensive CNN models. In addition, for the edge applications with real-time requirements, such as real-time computing (RTC) systems, the computations need to be completed considering the required timing constraint, so it is more difficult to trade off between computational latency and power consumption. In this paper, we propose a low-power CNN accelerator for edge inference of RTC systems, where the computations are operated in a column-wise manner, to realize an immediate computation for the currently available input data. We observe that most computations of some CNN kernels in deep layers can be completed in multiple cycles, while not affecting the overall computational latency. Thus, we present a multi-cycle scheme to conduct the column-wise convolutional operations to reduce the hardware resource and power consumption. We present hardware architecture for the multi-cycle scheme as a domain-specific CNN architecture, which is then implemented in a 65 nm technology. We prove our proposed approach realizes up to 8.45%, 49.41% and 50.64% power reductions for LeNet, AlexNet and VGG16, respectively. The experimental results show that our approach tends to cause a larger power reduction for the CNN models with greater depth, larger kernels and more channels.

Label-based, Mini-batch Combinations Study for Convolutional Neural Network Based Fluid-film Bearing Rotor System Diagnosis

This paper suggests label-based, mini-batch methods for convolutional neural network (CNN) based diagnosis of fluid-film bearing rotor systems. Rather than using random mini-batches in the training process, mini-batches are generated based on the label information. Label information is a critical factor for robust diagnosis. Five different types of label-based mini-batches are proposed and their performance is compared to the conventional random mini-batch method. In addition, sensitivity analysis of kernels in convolutional neural networks is suggested as a method to analyze the performance variation. A case study of a fluid-film bearing rotor system is used to show the effect of the proposed methods. The case study results indicate a wide range of performance variation among the proposed mini-batch methods. Of the examined methods, the equally labeled mini-batch approach presents the best performance. Moreover, the results of the kernel sensitivity analysis show that the use of properly sensitive kernels does positively affect the overall performance of the CNN.

Going Deeper into OSNR Estimation with CNN

As optical performance monitoring (OPM) requires accurate and robust solutions to tackle the increasing dynamic and complicated optical network architectures, we experimentally demonstrate an end-to-end optical signal-to-noise (OSNR) estimation method based on the convolutional neural network (CNN), named OptInception. The design principles of the proposed scheme are specified. The idea behind the combination of the Inception module and finite impulse response (FIR) filter is elaborated as well. We experimentally evaluate the mean absolute error (MAE) and root-mean-squared error (RMSE) of the OSNR monitored in PDM-QPSK and PDM-16QAM signals under various symbol rates. The results suggest that the MAE reaches as low as 0.125 dB and RMSE is 0.246 dB in general. OptInception is also proved to be insensitive to the symbol rate, modulation format, and chromatic dispersion. The investigation of kernels in CNN indicates that the proposed scheme helps convolutional layers learn much more than a lowpass filter or bandpass filter. Finally, a comparison in performance and complexity presents the advantages of OptInception.

Naive Gabor Networks for Hyperspectral Image Classification.

Recently, many convolutional neural network (CNN) methods have been designed for hyperspectral image (HSI) classification since CNNs are able to produce good representations of data, which greatly benefits from a huge number of parameters. However, solving such a high-dimensional optimization problem often requires a large number of training samples in order to avoid overfitting. In addition, it is a typical nonconvex problem affected by many local minima and flat regions. To address these problems, in this article, we introduce the naive Gabor networks or Gabor-Nets that, for the first time in the literature, design and learn CNN kernels strictly in the form of Gabor filters, aiming to reduce the number of involved parameters and constrain the solution space and, hence, improve the performances of CNNs. Specifically, we develop an innovative phase-induced Gabor kernel, which is trickily designed to perform the Gabor feature learning via a linear combination of local low-frequency and high-frequency components of data controlled by the kernel phase. With the phase-induced Gabor kernel, the proposed Gabor-Nets gains the ability to automatically adapt to the local harmonic characteristics of the HSI data and, thus, yields more representative harmonic features. Also, this kernel can fulfill the traditional complex-valued Gabor filtering in a real-valued manner, hence making Gabor-Nets easily perform in a usual CNN thread. We evaluated our newly developed Gabor-Nets on three well-known HSIs, suggesting that our proposed Gabor-Nets can significantly improve the performance of CNNs, particularly with a small training set.

Dynamic Motion Estimation and Evolution Video Prediction Network

Future video prediction provides valuable information that helps a computer machine understand the surrounding environment and make critical decisions in real-time. However, long-term video prediction remains a challenging problem due to the complicated spatiotemporal dynamics in a video. In this paper, we propose a dynamic motion estimation and evolution (DMEE) network model to generate unseen future videos from the observed videos in the past. Our primary contribution is to use trained kernels in convolutional neural network (CNN) and long short-term memory (LSTM) architectures, adapted to each time step and sample position, to efficiently manage spatiotemporal dynamics. DMEE uses the motion estimation (ME) and motion update (MU) kernels to predict the future video using an end-to-end prediction-update process. In the prediction, the ME kernel estimates the temporal changes. In the update step, the MU kernel combines the estimates with the previously generated frames as reference frames using a weighted average. The kernels are not only used for a current frame, but also are evolved to generate successive frames to enable temporally specific filtering. We perform qualitative performance analysis and quantitative performance analysis based on the peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and video classification score developed for examining the visual quality of the generated video. It is demonstrated with experiments that our algorithm provides better qualitative and quantitative performance superior to the current state-of-the-art algorithms. Our source codes are available in <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Nayoung-Kim-ICP/Video-Generation</uri> .

Fusing HOG and convolutional neural network spatial–temporal features for video‐based facial expression recognition

Video-based facial expression recognition (VFER) is the fundamental feature of various computer vision applications. Visual features are the key factors for facial expression recognition. However, the gap between the visual features and the emotions is large. In order to bridge the gap, the proposed method utilises convolutional neural networks (CNNs) and histogram of oriented gradient (HOG) to obtain the more comprehensive feature for VFER. Firstly, it extracts shallow features from the video frame through a number of convolutional kernels in CNNs, which has the characteristics of displacement, scale and deformation invariance. Then, the HOG is employed to extract HOG features from CNN's shallow features, which are strongly correlated with facial expressions. Finally, the support vector machine (SVM) is employed to conduct the task of facial expression recognition. The extensive experiments on RML, CK+ and AFEW5.0 database show that this framework takes on the promising performance and outperforming the state of the arts.

Deep eigen-filters for face recognition: Feature representation via unsupervised multi-structure filter learning

Training deep convolutional neural networks (CNNs) often requires high computational cost and a large number of learnable parameters. To overcome this limitation, one solution is computing predefined convolution kernels from training data. In this paper, we propose a novel three-stage approach for filter learning alternatively. It learns filters in multiple structures including standard filters, channel-wise filters and point-wise filters which are inspired from variations of CNNs’ convolution operations. By analyzing the linear combination between learned filters and original convolution kernels in pre-trained CNNs, the reconstruction error is minimized to determine the most representative filters from the filter bank. These filters are used to build a network followed by HOG-based feature extraction for feature representation. The proposed approach shows competitive performance on color face recognition compared with other deep CNNs-based methods. Besides, it provides a perspective of interpreting CNNs by introducing the concepts of advanced convolutional layers to unsupervised filter learning.

Hyperspectral Image Classification Based on Spectral and Spatial Information Using Multi-Scale ResNet

Hyperspectral imaging (HSI) contains abundant spectrums as well as spatial information, providing a great basis for classification in the field of remote sensing. In this paper, to make full use of HSI information, we combined spectral and spatial information into a two-dimension image in a particular order by extracting a data cube and unfolding it. Prior to the step of combining, principle component analysis (PCA) is utilized to decrease the dimensions of HSI so as to reduce computational cost. Moreover, the classification block used during the experiment is a convolutional neural network (CNN). Instead of using traditionally fixed-size kernels in CNN, we leverage a multi-scale kernel in the first convolutional layer so that it can scale to the receptive field. To attain higher classification accuracy with deeper layers, residual blocks are also applied to the network. Extensive experiments on the datasets from Pavia University and Salinas demonstrate that the proposed method significantly improves the accuracy in HSI classification.

A Dual Neural Architecture Combined SqueezeNet with OctConv for LiDAR Data Classification.

Light detection and ranging (LiDAR) is a frequently used technique of data acquisition and it is widely used in diverse practical applications. In recent years, deep convolutional neural networks (CNNs) have shown their effectiveness for LiDAR-derived rasterized digital surface models (LiDAR-DSM) data classification. However, many excellent CNNs have too many parameters due to depth and complexity. Meanwhile, traditional CNNs have spatial redundancy because different convolution kernels scan and store information independently. SqueezeNet replaces a part of 3 × 3 convolution kernels in CNNs with 1 × 1 convolution kernels, decomposes the original one convolution layer into two layers, and encapsulates them into a Fire module. This structure can reduce the parameters of network. Octave Convolution (OctConv) pools some feature maps firstly and stores them separately from the feature maps with the original size. It can reduce spatial redundancy by sharing information between the two groups. In this article, in order to improve the accuracy and efficiency of the network simultaneously, Fire modules of SqueezeNet are used to replace the traditional convolution layers in OctConv to form a new dual neural architecture: OctSqueezeNet. Our experiments, conducted using two well-known LiDAR datasets and several classical state-of-the-art classification methods, revealed that our proposed classification approach based on OctSqueezeNet is able to provide competitive advantages in terms of both classification accuracy and computational amount.

Blood Cell Classification Based on Hyperspectral Imaging With Modulated Gabor and CNN.

Cell classification, especially that of white blood cells, plays a very important role in the field of diagnosis and control of major diseases. Compared to traditional optical microscopic imaging, hyperspectral imagery, combined with both spatial and spectral information, provides more wealthy information for recognizing cells. In this paper, a novel blood cell classification framework, which combines a modulated Gabor wavelet and deep convolutional neural network (CNN) kernels, named as MGCNN, is proposed based on medical hyperspectral imaging. For each convolutional layer, multi-scale and orientation Gabor operators are taken dot product with initial CNN kernels. The essence is to transform the convolutional kernels into the frequency domain to learn features. By combining characteristics of Gabor wavelets, the features learned by modulated kernels at different frequencies and orientations are more representative and discriminative. Experimental results demonstrate that the proposed model can achieve better classification performance than traditional CNNs and widely used support vector machine approaches, especially as training small-sample-size situations.

Automatic Kernel Size Determination for Deep Neural Networks Based Hyperspectral Image Classification

Considering kernels in Convolutional Neural Networks (CNNs) as detectors for local patterns, K-means neural network proposes to cluster local patches extracted from training images and then fixate those kernels as the representative patches in each cluster without further training. Thus the amount of labeled samples necessitated for training can be greatly reduced. One key property of those kernels is their spatial size which determines their capacity in detecting local patterns and is expected to be task-specific. However, most of literatures determine the spatial size of those kernels in a heuristic way. To address this problem, we propose to automatically determine the kernel size in order to better adapt the K-means neural network for hyperspectral imagery classification. Specifically, a novel kernel-size determination scheme is developed by measuring the clustering performance of local patches with different sizes. With the kernel of determined size, more discriminative local patterns can be detected in the hyperspectral imagery, with which the classification performance of K-means neural network can be obviously improved. Experimental results on two datasets demonstrate the effectiveness of the proposed method.

Epithelium-Stroma Classification via Convolutional Neural Networks and Unsupervised Domain Adaptation in Histopathological Images.

Epithelium-stroma classification is a necessary preprocessing step in histopathological image analysis. Current deep learning based recognition methods for histology data require collection of large volumes of labeled data in order to train a new neural network when there are changes to the image acquisition procedure. However, it is extremely expensive for pathologists to manually label sufficient volumes of data for each pathology study in a professional manner, which results in limitations in real-world applications. A very simple but effective deep learning method, that introduces the concept of unsupervised domain adaptation to a simple convolutional neural network (CNN), has been proposed in this paper. Inspired by transfer learning, our paper assumes that the training data and testing data follow different distributions, and there is an adaptation operation to more accurately estimate the kernels in CNN in feature extraction, in order to enhance performance by transferring knowledge from labeled data in source domain to unlabeled data in target domain. The model has been evaluated using three independent public epithelium-stroma datasets by cross-dataset validations. The experimental results demonstrate that for epithelium-stroma classification, the proposed framework outperforms the state-of-the-art deep neural network model, and it also achieves better performance than other existing deep domain adaptation methods. The proposed model can be considered to be a better option for real-world applications in histopathological image analysis, since there is no longer a requirement for large-scale labeled data in each specified domain.