Shallow Convolutional Neural Network Research Articles

A Novel Time-Incremental End-to-End Shared Neural Network with Attention-Based Feature Fusion for Multiclass Motor Imagery Recognition.

In the research of motor imagery brain-computer interface (MI-BCI), traditional electroencephalogram (EEG) signal recognition algorithms appear to be inefficient in extracting EEG signal features and improving classification accuracy. In this paper, we discuss a solution to this problem based on a novel step-by-step method of feature extraction and pattern classification for multiclass MI-EEG signals. First, the training data from all subjects is merged and enlarged through autoencoder to meet the need for massive amounts of data while reducing the bad effect on signal recognition because of randomness, instability, and individual variability of EEG data. Second, an end-to-end sharing structure with attention-based time-incremental shallow convolution neural network is proposed. Shallow convolution neural network (SCNN) and bidirectional long short-term memory (BiLSTM) network are used to extract frequency-spatial domain features and time-series features of EEG signals, respectively. Then, the attention model is introduced into the feature fusion layer to dynamically weight these extracted temporal-frequency-spatial domain features, which greatly contributes to the reduction of feature redundancy and the improvement of classification accuracy. At last, validation tests using BCI Competition IV 2a data sets show that classification accuracy and kappa coefficient have reached 82.7 ± 5.57% and 0.78 ± 0.074, which can strongly prove its advantages in improving classification accuracy and reducing individual difference among different subjects from the same network.

Shallow Convolutional Neural Networks for Human Activity Recognition Using Wearable Sensors

Due to rapid development of sensor technology, human activity recognition (HAR) using wearable inertial sensors has recently become a new research hotspot. Deep learning, especially convolutional neural network (CNN) that can automatically learn intricate activity features have gained a lot of attention in ubiquitous HAR task. Most existing CNNs process sensor input by extracting channel-wise features, and the information from each channel can be separately propagated in a hierarchical way from lower layers to higher layers. As a result, they typically overlook information exchange among channels within the same layer. In this article, we first propose a shallow CNN that considers cross-channel communication in HAR scenario, where all channels in the same layer have a comprehensive interaction to capture more discriminative features of sensor input. One channel can communicate with all other channels by graph neural network to remove redundant information accumulated among channels, which is more beneficial for deploying lightweight deep models. Extensive experiments are conducted on multiple benchmark HAR datasets, namely UCI-HAR, OPPORTUNITY, PAMAP2 and UniMib-SHAR, which indicates that the proposed method enables shallower CNNs to aggregate more useful information, and surpasses baseline deep networks and other competitive methods. The inference speed is evaluated via deploying the HAR systems on an embedded system.

Deep Edge Computing for Videos

This paper provides a modular architecture with deep neural networks as a solution for real-time video analytics in an edge-computing environment. The modular architecture consists of two networks of Front-CNN (Convolutional Neural Network) and Back-CNN, where we adopt Shallow 3D CNN (S3D) as the Front-CNN and a pre-trained 2D CNN as the Back-CNN. The S3D (i.e., the Front CNN) is in charge of condensing a sequence of video frames into a feature map with three channels. That is, the S3D takes a set of sequential frames in the video shot as input and yields a learned 3 channel feature map (3CFM) as output. Since the 3CFM is compatible with the three-channel RGB color image format, we can use the output of the S3D (i.e., the 3CFM) as the input to a pre-trained 2D CNN of the Back-CNN for the transfer-learning. This serial connection of Front-CNN and Back-CNN architecture is end-to-end trainable to learn both spatial and temporal information of videos. Experimental results on the public datasets of UCF-Crime and UR-Fall Detection show that the proposed S3D-2DCNN model outperforms the existing methods and achieves state-of-the-art performance. Moreover, since our Front-CNN and Back-CNN modules have a shallow S3D and a light-weighted 2D CNN, respectively, it is suitable for real-time video recognition in edge-computing environments. We have implemented our CNN model on NVIDIA Jetson Nano Developer as an edge-computing device to show its real-time execution.

On Convolutional Neural Networks for Chest X-ray Classification

Pneumonia is an infectious disease accounting for one fifth of the deaths of children under five worldwide. It is also a cause for adult hospital admissions with mortality rate of almost 25% in patients over 75 years. Pneumonia is curable and mortality can be prevented if it is diagnosed. Preferred diagnostic technique is chest X-ray image examination. The lack of radiology equipment of trained clinicians reduces the chance of the majority of people to be properly diagnosed. In the paper we propose a shallow convolutional neural network architecture, further fine-tuned with chose of the Adam optimizer which is split tested in the current experimental work. We showed that the automatic detection of pneumonia in chest X-ray images is possible with accuracy higher than 90% using 3 blocks of 2 convolutional 2D layers with max-pooling for feature extraction and a flatten output block comprising two dense layers, trained over 10 epochs. Therefore, the application for CNN is a viable solution as a supplement to the decision-making process of pneumonia X-ray diagnostics.

An Intelligent Diagnosis Method of the Working Conditions in Sucker-Rod Pump Wells Based on Convolutional Neural Networks and Transfer Learning

In recent years, deep learning models represented by convolutional neural networks have shown incomparable advantages in image recognition and have been widely used in various fields. In the diagnosis of sucker-rod pump working conditions, due to the lack of a large-scale dynamometer card data set, the advantages of a deep convolutional neural network are not well reflected, and its application is limited. Therefore, this paper proposes an intelligent diagnosis method of the working conditions in sucker-rod pump wells based on transfer learning, which is used to solve the problem of too few samples in a dynamometer card data set. Based on the dynamometer cards measured in oilfields, image classification and preprocessing are conducted, and a dynamometer card data set including 10 typical working conditions is created. On this basis, using a trained deep convolutional neural network learning model, model training and parameter optimization are conducted, and the learned deep dynamometer card features are transferred and applied so as to realize the intelligent diagnosis of dynamometer cards. The experimental results show that transfer learning is feasible, and the performance of the deep convolutional neural network is better than that of the shallow convolutional neural network and general fully connected neural network. The deep convolutional neural network can effectively and accurately diagnose the working conditions of sucker-rod pump wells and provide an effective method to solve the problem of few samples in dynamometer card data sets.

Defect Detection Method of Aluminum Profile Surface Using Deep Self-Attention Mechanism Under Hybrid Noise Conditions

Noisy images will inevitably exist in the image acquisition process of the aluminum profile surface (APS) due to the limitations of the imaging system. And, the overlap of different types of image noise will increase the difficulty of image-based detection for APS. Addressing this problem, we propose an APS defect detection method based on a deep self-attention mechanism (DSAM) under hybrid noise conditions. Firstly, we use the residual learning strategy to obtain defect feature maps from noisy images through identity mapping. This strategy overcomes the limitations of the shallow convolutional neural network (CNN) on feature extraction. Then, the self-attention mechanism achieves refined feature extraction by adding corresponding weight matrices to the defect feature maps. The mechanism improves the defect feature extraction ability under hybrid noise conditions. Finally, a softmax classification layer is added to the feature output layer to establish the mapping relationship between the feature space and the APS defect space. And the APS defect detection result under hybrid noise conditions is obtained. Experimental results demonstrated that the proposed method can achieve higher recognition accuracy under hybrid noise conditions.

Subband Adaptive Image Deblocking Using Wavelet Based Convolutional Neural Networks

In this paper, we propose subband adaptive image deblocking using wavelet based convolutional neural networks (CNNs). We build wavelet based CNNs for image deblocking to achieve subband adaptive reconstruction. First, we perform subband adaptive processing after the discrete wavelet transform (DWT) on the input image. For low frequency subband (LL), we use a simple and effective shallow CNN to restore the low frequency component, while for high frequency subbands (LH, HL, and HH) we utilize multi-kernel convolution to capture multiscale features and restore sparse high frequency components. Then, we conduct mixed convolution of dilated convolution and standard convolution to expand the receptive field while introducing channel and spatial attentions to adjust the proportion of different subbands and spatial coordinates. Various experiments on Classic5 and LIVE1 datasets show that the proposed method successfully recovers sharp edges and clear textures in highly compressed images while removing compression artifacts such as blocking and banding. Moreover, the proposed method achieves comparable state-of-the-art performance on compression artifact removal in terms of both visual quality and quantitative measurements.

Chest X-Ray Image Preprocessing for Disease Classification

Research has proven that images with blur, low contrast or other deficiencies are detrimental to accuracy of classification models. Working with a dataset of blurry, hard-to-see images creates an uncertainty in whether the machine learning model is able to accurately identify objects. These hard to identify images require the use of metadata on the image to be able to truly understand and classify the image accuracy within a model. To overcome the necessity of metadata for classification of hard to identify images, we propose a method of identifying the unfit images for purposes of cleaning the dataset prior to neural network training. In our work, we use Sobel and Scharr operators-based edge detectors to produce an image with detected boundaries among the elements of the original X-Ray. The resulting images are used to train a shallow CNN to classify the image as clear or lacking in quality. Our proposed method identifies the clear, quality images with 95% accuracy and can be utilized in future disease classification models. We define the proposed approach as image preprocessing aiming to weed out infit, blurry, low/high intensity X-Rays and create a dataset suitable for disease identification.

A Real-Time Patient-Specific Sleeping Posture Recognition System Using Pressure Sensitive Conductive Sheet and Transfer Learning

Sleeping is an indispensable activity of human beings. Sleeping postures have a significant effect on sleeping quality and health. A real-time low-cost sleeping posture recognition system with high privacy and good user experience is desired. In this article, we propose a sleeping posture recognition system based on a low-cost pressure sensor array which consists of conductive fabric and conductive wires. The sensor array is deployed as a bedsheet with 32 rows and 32 columns resulting in 1024 nodes. An Arduino Nano performs data collection using a 10-bit Analog to Digital Converter (ADC). The sampling rate of the overall sensor array is 0.4 frame/sec. Six health-related sleeping postures of five participants can be recognized by a shallow Convolutional Neural Network (CNN) deployed on a Personal Computer (PC). The system accuracy achieved 84.80% using the standard training-test method and 91.24% using the transfer learning-based subject-specific method. The real-time processing speed achieved 434 us/frame.

Ensemble Learning of Convolutional Neural Network, Support Vector Machine, and Best Linear Unbiased Predictor for Brain Age Prediction: ARAMIS Contribution to the Predictive Analytics Competition 2019 Challenge

We ranked third in the Predictive Analytics Competition (PAC) 2019 challenge by achieving a mean absolute error (MAE) of 3.33 years in predicting age from T1-weighted MRI brain images. Our approach combined seven algorithms that allow generating predictions when the number of features exceeds the number of observations, in particular, two versions of best linear unbiased predictor (BLUP), support vector machine (SVM), two shallow convolutional neural networks (CNNs), and the famous ResNet and Inception V1. Ensemble learning was derived from estimating weights via linear regression in a hold-out subset of the training sample. We further evaluated and identified factors that could influence prediction accuracy: choice of algorithm, ensemble learning, and features used as input/MRI image processing. Our prediction error was correlated with age, and absolute error was greater for older participants, suggesting to increase the training sample for this subgroup. Our results may be used to guide researchers to build age predictors on healthy individuals, which can be used in research and in the clinics as non-specific predictors of disease status.

WLnet: Towards an Approach for Robust Workload Estimation Based on Shallow Neural Networks

Electroencephalography (EEG) is a non-invasive technology used for the human brain-computer interface. One of its important applications is the evaluation of the mental state of an individual, such as workload estimation. In previous works, common spatial pattern feature extraction methods have been proposed for the EEG-based workload detection. Recently, several novel methods were introduced to detect EEG pattern workloads. However, it is still unknown which one of these methods is the one that offers the best performance for the workload EEG pattern feature detection. In this article, four methods were used to extract workload EEG features: (a) common spatial pattern feature extraction; (b) temporally constrained sparse group spatial pattern feature extraction; (c) EEGnet; and (d) the new proposed shallow convolutional neural network for workload estimation (WLnet). The classification accuracy of these four methods was compared. Experimental results demonstrate that the proposed WLnet achieved the best detection accuracy in both stress and non-stress conditions. We believe that the proposed methods may be relevant to real-life applications of mental workload estimation.

DeepInterpolation: fusion of multiple interpolations and CNN to obtain super‐resolution

The authors propose architectures that learn end-to-end mapping functions to improve the spatial resolution of the input natural images. The models are unique in forming non-linear combinations of three image interpolation techniques using the convolutional neural network. Another proposed architecture uses a skip connection with nearest-neighbour interpolation, achieving almost similar results. The architectures have been carefully designed to ensure that the reconstructed images lie precisely in the manifold of high-resolution images, thereby preserving the high-frequency components with fine details. They have compared with the state-of-the-art and recent deep learning-based natural image super-resolution techniques and found that their methods can preserve the sharp details in the image, while also obtaining comparable or better peak-signal-to-noise ratio values than them. Since their methods use image interpolations and a shallow convolutional neural network (CNN) with a fewer number of smaller filters, the computational cost is kept low. They have reported the results of the best two proposed architectures on five standard data sets for an upscale factor of 2. Their methods generalise well in most cases, which is evident from the better results obtained with increasingly complex data sets. For four times upscaling, they have designed similar architectures for comparing with other methods.

Hybrid-COVID: a novel hybrid 2D/3D CNN based on cross-domain adaptation approach for COVID-19 screening from chest X-ray images.

The novel Coronavirus disease (COVID-19), which first appeared at the end of December 2019, continues to spread rapidly in most countries of the world. Respiratory infections occur primarily in the majority of patients treated with COVID-19. In light of the growing number of COVID-19 cases, the need for diagnostic tools to identify COVID-19 infection at early stages is of vital importance. For decades, chest X-ray (CXR) technologies have proven their ability to accurately detect respiratory diseases. More recently, with the availability of COVID-19 CXR scans, deep learning algorithms have played a critical role in the healthcare arena by allowing radiologists to recognize COVID-19 patients from their CXR images. However, the majority of screening methods for COVID-19 reported in recent studies are based on 2D convolutional neural networks (CNNs). Although 3D CNNs are capable of capturing contextual information compared to their 2D counterparts, their use is limited due to their increased computational cost (i.e. requires much extra memory and much more computing power). In this study, a transfer learning-based hybrid 2D/3D CNN architecture for COVID-19 screening using CXRs has been developed. The proposed architecture consists of the incorporation of a pre-trained deep model (VGG16) and a shallow 3D CNN, combined with a depth-wise separable convolution layer and a spatial pyramid pooling module (SPP). Specifically, the depth-wise separable convolution helps to preserve the useful features while reducing the computational burden of the model. The SPP module is designed to extract multi-level representations from intermediate ones. Experimental results show that the proposed framework can achieve reasonable performances when evaluated on a collected dataset (3 classes to be predicted: COVID-19, Pneumonia, and Normal). Notably, it achieved a sensitivity of 98.33%, a specificity of 98.68% and an overall accuracy of 96.91%

Improving Alzheimer's stage categorization with Convolutional Neural Network using transfer learning and different magnetic resonance imaging modalities

BackgroundAlzheimer's Disease (AD) is a neurodegenerative disease characterized by progressive loss of memory and general decline in cognitive functions. Multi-modal imaging such as structural MRI and DTI provide useful information for the classification of patients on the basis of brain biomarkers. Recently, CNN methods have emerged as powerful tools to improve classification using images. New MethodIn this paper, we propose a transfer learning scheme using Convolutional Neural Networks (CNNs) to automatically classify brain scans focusing only on a small ROI: e.g. a few slices of the hippocampal region. The network's architecture is similar to a LeNet-like CNN upon which models are built and fused for AD stage classification diagnosis. We evaluated various types of transfer learning through the following mechanisms: (i) cross-modal (sMRI and DTI) and (ii) cross-domain transfer learning (using MNIST) (iii) a hybrid transfer learning of both types. ResultsOur method shows good performances even on small datasets and with a limited number of slices of small brain region. It increases accuracy with more than 5 points for the most difficult classification tasks, i.e., AD/MCI and MCI/NC. Comparison with Existing Method(s)Our methodology provides good accuracy scores for classification over a shallow convolutional network. Besides, we focused only on a small region; i.e., the hippocampal region, where few slices are selected to feed the network. Also, we used cross-modal transfer learning. ConclusionsOur proposed method is suitable for working with a shallow CNN network for low-resolution MRI and DTI scans. It yields to significant results even if the model is trained on small datasets, which is often the case in medical image analysis.

Test-time adaptable neural networks for robust medical image segmentation.

Convolutional Neural Networks (CNNs) work very well for supervised learning problems when the training dataset is representative of the variations expected to be encountered at test time. In medical image segmentation, this premise is violated when there is a mismatch between training and test images in terms of their acquisition details, such as the scanner model or the protocol. Remarkable performance degradation of CNNs in this scenario is well documented in the literature. To address this problem, we design the segmentation CNN as a concatenation of two sub-networks: a relatively shallow image normalization CNN, followed by a deep CNN that segments the normalized image. We train both these sub-networks using a training dataset, consisting of annotated images from a particular scanner and protocol setting. Now, at test time, we adapt the image normalization sub-network for each test image, guided by an implicit prior on the predicted segmentation labels. We employ an independently trained denoising autoencoder (DAE) in order to model such an implicit prior on plausible anatomical segmentation labels. We validate the proposed idea on multi-center Magnetic Resonance imaging datasets of three anatomies: brain, heart and prostate. The proposed test-time adaptation consistently provides performance improvement, demonstrating the promise and generality of the approach. Being agnostic to the architecture of the deep CNN, the second sub-network, the proposed design can be utilized with any segmentation network to increase robustness to variations in imaging scanners and protocols. Our code is available at: https://github.com/neerakara/test-time-adaptable-neural-networks-for-domain-generalization.

A Computationally Efficient Single-Channel EEG Sleep Stage Scoring Approach using Simple Structured CNN

Automatic sleep stage classification has been a hot trend since hand-crafted feature engineering is highly inefficient. However, current studies of automatic sleep stage scoring focus more on designing complex neural network structures to improve the model performance while neglecting the model efficiency. This causes both lengthy training time and highly demanding hardware are needed for model training, which is not favorable for future industrial applications. This work proves the concept that the simple model, such as a shallow Convolutional Neural Network (CNN) combining the proper data processing techniques, can achieve a comparable model performance (overall accuracy of 79.0 %) to the complex model (overall accuracy of 74.9-82.0 %). The designed model in this work also significantly improves the model efficiency by reducing the number of learnable parameters in the neural network. This approach provides a new insight into automatic sleep stage scoring study as well as other deep learning studies that the data processing and the model design are equally important.

Parameter Distribution Balanced CNNs.

Convolutional neural network (CNN) is the primary technique that has greatly promoted the development of computer vision technologies. However, there is little research on how to allocate parameters in different convolution layers when designing CNNs. We research mainly on revealing the relationship between CNN parameter distribution, i.e., the allocation of parameters in convolution layers, and the discriminative performance of CNN. Unlike previous works, we do not append more elements into the network, such as more convolution layers or denser short connections. We focus on enhancing the discriminative performance of CNN through varying its parameter distribution under strict size constraint. We propose an energy function to represent the CNN parameter distribution, which establishes the connection between the allocation of parameters and the discriminative performance of CNN. Extensive experiments with shallow CNNs on three public image classification data sets demonstrate that the CNN parameter distribution with a higher energy value will promote the model to obtain better performance. According to the motivated observation, the problem of finding the optimal parameter distribution can be transformed into an optimization problem of finding the biggest energy value. We present a simple yet effective guideline that uses balanced parameter distribution to design CNNs. Extensive experiments on ImageNet with three popular backbones, i.e., AlexNet, ResNet34, and ResNet101, demonstrate that the proposed guideline can make consistent improvements upon different baselines under strict size constraint.

A Topological Neural-Based Scheme for Classification of Faults in Induction Machines

This article presents a data-driven approach for the classification of faults in induction machines. The designed scheme involves newly engineered features extracted from the line current signals, which provides an improved fault discrimination. For this purpose, a topological-based fast projection technique (curvilinear component analysis) is used as a tool to reduce the dimensionality of the data and interpret the feature behavior. Consequently, a shallow convolutional neural network has been designed to classify the three-phase stator current signals. Experimental tests at different operating conditions have assessed the procedure, confirming its effectiveness and suitability for online and real-time diagnostics.

Convolutional Neural Network Based Ensemble Approach for Homoglyph Recognition

Homoglyphs are pairs of visual representations of Unicode characters that look similar to the human eye. Identifying homoglyphs is extremely useful for building a strong defence mechanism against many phishing and spoofing attacks, ID imitation, profanity abusing, etc. Although there is a list of discovered homoglyphs published by Unicode consortium, regular expansion of Unicode character scripts necessitates a robust and reliable algorithm that is capable of identifying all possible new homoglyphs. In this article, we first show that shallow Convolutional Neural Networks are capable of identifying homoglyphs. We propose two variations, both of which obtain very high accuracy (99.44%) on our benchmark dataset. We also report that adoption of transfer learning allows for another model to achieve 100% recall on our dataset. We ensemble these three methods to obtain 99.72% accuracy on our independent test dataset. These results illustrate the superiority of our ensembled model in detecting homoglyphs and suggest that our model can be used to detect new homoglyphs when increasing Unicode characters are added. As a by-product, we also prepare a benchmark dataset based on the currently available list of homoglyphs.

Superpixel-Based Shallow Convolutional Neural Network (SSCNN) for Scanned Topographic Map Segmentation

Motivated by applications in topographic map information extraction, our goal was to discover a practical method for scanned topographic map (STM) segmentation. We present an advanced guided watershed transform (AGWT) to generate superpixels on STM. AGWT utilizes the information from both linear and area elements to modify detected boundary maps and sequentially achieve superpixels based on the watershed transform. With achieving an average of 0.06 on under-segmentation error, 0.96 on boundary recall, and 0.95 on boundary precision, it has been proven to have strong ability in boundary adherence, with fewer over-segmentation issues. Based on AGWT, a benchmark for STM segmentation based on superpixels and a shallow convolutional neural network (SCNN), termed SSCNN, is proposed. There are several notable ideas behind the proposed approach. Superpixels are employed to overcome the false color and color aliasing problems that exist in STMs. The unification method of random selection facilitates sufficient training data with little manual labeling while keeping the potential color information of each geographic element. Moreover, with the small number of parameters, SCNN can accurately and efficiently classify those unified pixel sequences. The experiments show that SSCNN achieves an overall F1 score of 0.73 on our STM testing dataset. They also show the quality of the segmentation results and the short run time of this approach, which makes it applicable to full-size maps.