Existing Convolutional Neural Networks Architectures Research Articles

Quantized CNN-based efficient hardware architecture for real-time hand gesture recognition

Nowadays, Convolutional Neural Networks (CNN) have been widely adopted for vision-based hand gesture recognition. Several existing CNN architectures designed for gesture classification perform well with high accuracy but require a high memory footprint and processing when deployed on low-power embedded devices. To address this issue, we present a quantized CNN-based efficient framework for meeting real-life hand gesture recognition challenges. The proposed quantized CNN is designed and implemented using the FINN-based pipelined streaming architecture on an FPGA. Moreover, hardware-based optimizations are used to minimize the resources needed and to achieve fast memory access. Experimental results demonstrate that the developed recognition system achieves an average accuracy of 92% on the numeral database of Indian Sign Language (ISL). Additionally, our optimized design attains an inference latency of 0.85ms for real-time single gesture prediction on the PYNQ Zynq Ultrascale (ZU) FPGA, consuming only 3.63 W of power. The proposed design achieves a better trade-off between hardware resource utilization and speed performance, over previous designs.

IMU-Based Fitness Activity Recognition Using CNNs for Time Series Classification.

Mobile fitness applications provide the opportunity to show users real-time feedback on their current fitness activity. For such applications, it is essential to accurately track the user's current fitness activity using available mobile sensors, such as inertial measurement units (IMUs). Convolutional neural networks (CNNs) have been shown to produce strong results in different time series classification tasks, including the recognition of daily living activities. However, fitness activities can present unique challenges to the human activity recognition task (HAR), including greater similarity between individual activities and fewer available data for model training. In this paper, we evaluate the applicability of CNNs to the fitness activity recognition task (FAR) using IMU data and determine the impact of input data size and sensor count on performance. For this purpose, we adapted three existing CNN architectures to the FAR task and designed a fourth CNN variant, which we call the scaling fully convolutional network (Scaling-FCN). We designed a preprocessing pipeline and recorded a running exercise data set with 20 participants, in which we evaluated the respective recognition performances of the four networks, comparing them with three traditional machine learning (ML) methods commonly used in HAR. Although CNN architectures achieve at least 94% test accuracy in all scenarios, two traditional ML architectures surpass them in the default scenario, with support vector machines (SVMs) achieving 99.00 ± 0.34% test accuracy. The removal of all sensors except one foot sensor reduced the performance of traditional ML architectures but improved the performance of CNN architectures on our data set, with our Scaling-FCN reaching the highest accuracy of 99.86 ± 0.11% on the test set. Our results suggest that CNNs are generally well suited for fitness activity recognition, and noticeable performance improvements can be achieved if sensors are dropped selectively, although traditional ML architectures can still compete with or even surpass CNNs when favorable input data are utilized.

Fine Tuned of DenseNET121 to Classify NTT Weaving Motifs on Mobile Application

The problem of classifying Woven Fabric Motifs through pattern recognition can be addressed using Convolutional Neural Networks (CNNs). Existing CNN architectures like VGG, ResNet, MobileNet, and DenseNet offer diverse propagation methods. These architectures, trained on datasets like imagenet, have demonstrated competence in solving large-scale classification tasks. The CNN model trained on the ImageNet dataset, hereinafter referred to as the pre-trained model, can be utilized to address the classification issue of NTT woven fabric motifs. This involves retraining the model using a new output layer and dataset, a method known as Transfer Learning. In addition to Transfer Learning, this research employs Fine Tuning, which entails retraining several classification layers. The pre-trained model used in this research is DenseNet121. This model was chosen because it does not require too much storage space and has good classification performance so that it can be embedded in smartphones. The results of this study indicate that of the three pre-trained models tested (DenseNet121, MobileNetV2, and ResNet50V2), the pre-trained Model DenseNet121 is the model that has the highest accuracy and the smallest loss, namely 92.58% accuracy and 29.62% Loss. Tests on mobile devices also show that from 130 test data, this model gets an accuracy of 99.23%. Overall, the classification model of NTT woven fabric motifs embedded in mobile devices can be used as an alternative to help the community or people who want to learn about NTT woven fabric motifs.

A Jackknife-Inspired Deep Learning Approach to Subject-Independent Classification of EEG

One of the significant challenges in Brain–Computer Interface (BCI) is to develop a classifier that can decode users’ mental states based on electroencephalogram (EEG) data collected from independent subjects. The focus of such subject-independent (SI) classification is justified because it can lead to BCIs that do not require a user-specific calibration process. In recent years, the emergence of deep neural networks (DNNs) has significantly improved the performance of EEG classification. Among various deep learning techniques, the training efficiency and the performance of Convolutional Neural Networks (CNNs), in particular, have led to several state-of-the-art architectures for accurate classification of EEG. Not surprisingly, the efforts to improve the performance of these architectures for EEG classification have been ramped up in recent years. In this regard, a trivial approach is to train and tune a large number of architectures and hyperparameters and hope to improve upon the existing results. In contrast with this ad hoc approach, here we put forward a systematic method inspired by the jackknife estimation to improve the performance of existing CNN architectures. Using EEGNet and ShallowConvNet as archetypical, our empirical results show that the proposed “Delete-a-Subject Jackknife” (DASJ) technique can potentially improve the performance of existing CNN architectures for SI classification of EEG.

Simultaneous denoising and reconstruction of distributed acoustic sensing seismic data via a multicascade deep-learning method

Distributed acoustic sensing (DAS) has been moderately used in seismic exploration as a consequence of its advantages, such as low cost, high resolution, high sensitivity, and strong resistance to high temperatures. Nonetheless, seismic data recorded by the DAS system often are seriously contaminated by a massive amount of background noise and recording gaps, which also decrease the quality of DAS seismic data. Building on its success in numerous fields of data processing, convolutional neural networks (CNNs) also have been applied to seismic data denoising as well as reconstruction and achieved good performance compared with traditional methods. However, relatively few CNN-based studies have been done on the simultaneous denoising and reconstruction of seismic data, especially for the complex DAS seismic data with a low signal-to-noise ratio. In this study, we have developed a multicascade network structure in combination with a channel attention mechanism (MCA-Net) to achieve the simultaneous denoising and reconstruction of DAS seismic data by only using a unified trained model. Particularly, our MCA-Net consists of three branches of cascade networks that can extract abundant features from DAS seismic data with different resolutions. Furthermore, we deploy some channel attention modules and aggressive blocks to MCA-Net to fuse these aforementioned multiresolution features and thus improving the feature extraction ability of the whole network. Meanwhile, a high-quality training data set composed of paired clean complete and noisy incomplete patches is used to train MCA-Net in a supervised fashion. Some synthetic and real DAS seismic data are used to demonstrate the effectiveness of MCA-Net, and our method exhibits better denoising and reconstruction performance than some traditional methods and two existing CNN architectures.

Class attention to regions of lesion for imbalanced medical image recognition

Automated medical image classification is the key component in intelligent diagnosis systems. However, most medical image datasets contain plenty of samples of common diseases and just a handful of rare ones, leading to major class imbalances. Currently, it is an open problem in intelligent diagnosis to effectively learn from imbalanced training data. In this paper, we propose a simple yet effective framework, named Class Attention to REgions of the lesion (CARE), to handle data imbalance issues by embedding attention into the training process of Convolutional Neural Networks (CNNs). The proposed attention module helps CNNs attend to lesion regions of rare diseases, therefore helping CNNs to learn their characteristics more effectively. In addition, this attention module works only during the training phase and does not change the architecture of the original network, so it can be directly combined with any existing CNN architecture. The CARE framework needs bounding boxes to represent the lesion regions of rare diseases. To alleviate the need for manual annotation, we further developed variants of CARE by leveraging the traditional saliency methods or a pretrained segmentation model for bounding box generation. Results show that the CARE variants with automated bounding box generation are comparable to the original CARE framework with manual bounding box annotations. A series of experiments on an imbalanced skin image dataset and a pneumonia dataset indicates that our method can effectively help the network focus on the lesion regions of rare diseases and remarkably improves the classification performance of rare diseases.

Adaptive pooling-based convolution factorization for deploying CNNs on energy-constrained IoT edge devices

Convolutional neural networks (CNNs) involve a tremendous amount of multiply-and-accumulate (MAC) computations, leading to energy issues when deployed on energy-constrained IoT edge devices for computer vision purposes. However, a great number of these computations are ineffectual since they lead to meaningless zero activations, which could be avoided if predicted appropriately. This has been the focus of many studies, whereas it has only been done through zero-activation simplifications. This paper goes beyond those simplifications through a convolution factorization approach to convert the MAC-intensive convolution operations into less complex pooling operations. The main characteristic of the proposed idea is that, given a target accuracy, the applicability and type of factorization are adaptively decided for each individual activation. This is achieved by providing a lightweight multi-class CNN-based predictor, namely RedZAP. The selected types of pooling operations for our convolution factorizations are supported by existing CNN architectures. The experimental results show that, given a top-1 accuracy loss constraint of 1%, the proposed approach saves another 8% MAC operations in comparison to the existing zero-prediction approaches in the literature.

Optimal Design of Convolutional Neural Network Architectures Using Teaching–Learning-Based Optimization for Image Classification

Convolutional neural networks (CNNs) have exhibited significant performance gains over conventional machine learning techniques in solving various real-life problems in computational intelligence fields, such as image classification. However, most existing CNN architectures were handcrafted from scratch and required significant amounts of problem domain knowledge from designers. A novel deep learning method abbreviated as TLBOCNN is proposed in this paper by leveraging the excellent global search ability of teaching–learning-based optimization (TLBO) to obtain an optimal design of network architecture for a CNN based on the given dataset with symmetrical distribution of each class of data samples. A variable-length encoding scheme is first introduced in TLBOCNN to represent each learner as a potential CNN architecture with different layer parameters. During the teacher phase, a new mainstream architecture computation scheme is designed to compute the mean parameter values of CNN architectures by considering the information encoded into the existing population members with variable lengths. The new mechanisms of determining the differences between two learners with variable lengths and updating their positions are also devised in both the teacher and learner phases to obtain new learners. Extensive simulation studies report that the proposed TLBOCNN achieves symmetrical performance in classifying the majority of MNIST-variant datasets, displays the highest accuracy, and produces CNN models with the lowest complexity levels compared to other state-of-the-art methods due to its promising search ability.

BlockCrime: Blockchain and Deep Learning-Based Collaborative Intelligence Framework to Detect Malicious Activities for Public Safety

Detecting malicious activity in advance has become increasingly important for public safety, economic stability, and national security. However, the disparity in living standards incites the minds of certain undesirable members of society to commit crimes, which may disrupt society’s stability and mental calm. Breakthroughs in deep learning (DL) make it feasible to address such challenges and construct a complete intelligent framework that automatically detects such malicious behaviors. Motivated by this, we propose a convolutional neural network (CNN)-based Xception model, i.e., BlockCrime, to detect crimes and improve public safety. Furthermore, we integrate blockchain technology to securely store the detected crime scene locations and alert the nearest law enforcement authorities. Due to the scarcity of the dataset, transfer learning has been preferred, in which a CNN-based Xception model is used. The redesigned Xception architecture is evaluated against various assessment measures, including accuracy, F1 score, precision, and recall, where it outperforms existing CNN architectures in terms of train accuracy, i.e., 96.57%.

DAFT: A universal module to interweave tabular data and 3D images in CNNs

Prior work on Alzheimer’s Disease (AD) has demonstrated that convolutional neural networks (CNNs) can leverage the high-dimensional image information for diagnosing patients. Beside such data-driven approaches, many established biomarkers exist and are typically represented as tabular data, such as demographics, genetic alterations, or laboratory measurements from cerebrospinal fluid. However, little research has focused on the effective integration of tabular data into existing CNN architectures to improve patient diagnosis. We introduce the Dynamic Affine Feature Map Transform (DAFT), a general-purpose module for CNNs that incites or represses high-level concepts learned from a 3D image by conditioning feature maps of a convolutional layer on both a patient’s image and tabular clinical information. This is achieved by using an auxiliary neural network that outputs a scaling factor and offset to dynamically apply an affine transformation to the feature maps of a convolutional layer. In our experiments on AD diagnosis and time-to-dementia prediction, we show that the DAFT is highly effective in combining 3D image and tabular information by achieving a mean balanced accuracy of 0.622 for diagnosis, and mean c-index of 0.748 for time-to-dementia prediction, thus outperforming all baseline methods. Finally, our extensive ablation study and empirical experiments reveal that the performance improvement due to the DAFT is robust with respect to many design choices.

Autonomous CNN (AutoCNN): A data-driven approach to network architecture determination

Designing a Convolutional Neural Networks (CNN) is a complex task and requires expert knowledge to optimize the performance and network architecture. In this paper, a novel data-driven approach is proposed to determine the architecture of CNN models. The proposed Autonomous Convolutional Neural Networks (AutoCNNThe executable code and original numerical results can be downloaded from (https://tinyurl.com/AutoCNN)) algorithm introduces data driven strategies for addition of new convolutional layers, pruning of redundant filters and training cycle optimization. AutoCNN is evaluated using MNIST, MNIST-rot-back-image, Fashion MNIST and the ADHD200 datasets to measure the performance on small datasets with varied feature distributions. The results indicate that AutoCNN optimizes the CNN network architecture and helps maximise the classification performance. The data-driven network determination approach introduced in this paper was found to not only provides competitive performance similar to existing evolutionary computation based network determination algorithms in literature, but was found to be an effective optimization tool to improve the performance of existing CNN architectures. Further, the AutoCNN was found to highly immune to noise in the dataset and has proven to be effective method to transfer knowledge between related datasets. Therefore, the AutoCNN is a highly versatile CNN architecture determination tool that has a wide range of applications in the field of autonomous driving, medical image analysis, image enhancement, camera based security monitoring and image based fault detection.

Harmonic convolutional networks based on discrete cosine transform

Convolutional neural networks (CNNs) learn filters in order to capture local correlation patterns in feature space. We propose to learn these filters as combinations of preset spectral filters defined by the Discrete Cosine Transform (DCT). Our proposed DCT-based harmonic blocks replace conventional convolutional layers to produce partially or fully harmonic versions of new or existing CNN architectures. Using DCT energy compaction properties, we demonstrate how the harmonic networks can be efficiently compressed by truncating high-frequency information in harmonic blocks thanks to the redundancies in the spectral domain. We report extensive experimental validation demonstrating benefits of the introduction of harmonic blocks into state-of-the-art CNN models in image classification, object detection and semantic segmentation applications.

Semi-supervised generative and discriminative adversarial learning for motor imagery-based brain\u2013computer interface

Convolutional neural networks (CNNs), which can recognize structural/configuration patterns in data with different architectures, have been studied for feature extraction. However, challenges remain regarding leveraging advanced deep learning methods in BCIs. We focus on problems of small-sized training samples and interpretability of the learned parameters and leverages a semi-supervised generative and discriminative learning framework that effectively utilizes synthesized samples with real samples to discover class-discriminative features. Our framework learns the distributional characteristics of EEG signals in an embedding space using a generative model. By using artificially generated and real EEG signals, our framework finds class-discriminative spatio-temporal feature representations that help to correctly discriminate input EEG signals. It is noteworthy that the framework facilitates the exploitation of real, unlabeled samples to better uncover the underlying patterns inherent in a user’s EEG signals. To validate our framework, we conducted experiments comparing our method with conventional linear models by utilizing variants of three existing CNN architectures as generator networks and measuring the performance on three public datasets. Our framework exhibited statistically significant improvements over the competing methods. We investigated the learned network via activation pattern maps and visualized generated artificial samples to empirically justify the stability and neurophysiological plausibility of our model.

Ensemble Convolutional Neural Networks With Support Vector Machine for Epilepsy Classification Based on Multi-Sequence of Magnetic Resonance Images

Classification of brain abnormalities as a pathological cue of epilepsy based on magnetic resonance (MR) images is essential for diagnosis. There are some types of brain structural abnormalities as a pathological cue of epilepsy. To identify it, a neurologist can involve some sequence of MR images at a time. Existing algorithms for abnormalities classification usually involve only one or two sequences of MR images. In this paper, we proposed ensemble convolutional neural networks with a support vector machine (SVM) scheme to classify brain abnormalities (epilepsy) vs. non-epilepsy based on the axial multi-sequence of MR images. The convolutional neural network (CNN) models on the proposed method are base-learner models with different architectures and have low parameters. The performance improvement on the proposed method is made by combining the output of the base-learner models and the combination of predictions from these models. The combination of predictions uses majority voting, weighted majority voting, and weighted average. Henceforth, the combined output becomes input in the meta-learning process with SVM for the final classification. The dataset for evaluation is the axial multi-sequences of MR images that include abnormal brain structures causing epilepsy and non-epilepsy with various subjects’ histories. The experimental results show the proposed method can obtain an accuracy average and F <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> -score of 86.37% and 90.75%, respectively, and an improvement of accuracy of 6.7%-18.19% against the CNN models on the base-learner and 2.54%-2.65% against the combination of predictions. With these results, the proposed architecture also provides better performance compared to the two existing CNN architectures.

Mesh-Conv: Convolution operator with mesh resolution independence for flow field modeling

Convolutional neural network (CNN) is a useful model in deep-learning and has become a promising alternative for flow field modeling due to its capability of spatial information representation. However, the standard convolution operator is only suitable for uniform Cartesian mesh, rather than non-uniform structured or unstructured mesh. In this work, a new convolution operator named Mesh-Conv is designed to allow CNN to model flow field without requiring pixelation preprocessing. This is achieved by decomposing standard convolution elements, combining the unique data type of the flow field, and introducing the concept of local weights to incorporate data structure information. The differences are clarified among the new operator, the standard convolution operator, and the graph convolution operator. In addition, new geometric features and loss functions are designed to further improve the model performance for flow field modeling. Numerical experiments show that the Mesh-Conv operator can obtain higher modeling accuracy than the standard convolution operator. The correlation coefficients of the flow fields predicted by the model are all above 0.999, and there is no checkerboard phenomenon. The Mesh-Conv operator is independent of mesh resolution to a certain extent, so its training and testing losses to the dataset with mixed mesh resolution are reduced by 63% and 55% respectively, compared with the standard convolution operator, which greatly relaxes the restriction on the data structure. Moreover, the Mesh-Conv operator is easy to be extended to most existing CNN architectures.

A few filters are enough: Convolutional neural network for P300 detection

Over the past decade convolutional neural networks (CNNs) have become the driving force of an ever-increasing set of applications, achieving state-of-the-art performance. Modern CNN architectures are often composed of many convolutional and some fully connected layers, and have thousands or millions of parameters. CNNs have shown to be effective in the detection of Event-Related Potentials from electroencephalogram (EEG) signals, notably the P300 component which is frequently employed in Brain-Computer Interfaces (BCIs). However, for this task, the increase in detection rates compared to approaches based on human-engineered features has not been as impressive as in other areas and might not justify such a large number of parameters. In this paper, we study the performance of existing CNN architectures with diverse complexities for single-trial within-subject and cross-subject P300 detection on four different datasets. We also proposed SepConv1D, a very simple CNN architecture consisting of a single depthwise separable 1D convolutional layer followed by a fully connected Sigmoid classification neuron. We found that with as few as four filters in its convolutional layer and an overall small number of parameters, SepConv1D obtained competitive performances in the four datasets. We believe these results may represent an important step towards building simpler, cheaper, faster, and more portable BCIs.

CNN-Based Automatic Modulation Classification for Beyond 5G Communications

In this letter, we propose an improved convolutional neural network (CNN)-based automatic modulation classification network (IC-AMCNet), an algorithm to classify the modulation type of a wireless signal. Since adaptive coding and modulation is widely used in wireless communication, high accuracy and short computing time of classifier is needed. Compared with the existing CNN architectures, we adjusted the number of layers and added new type of layers to comply with the estimated latency standards in beyond fifth-generation (B5G) communications. According to the simulation results, the proposed scheme significantly outperforms the previous works in terms of both classification accuracy and computing time.

QuadRoad: An Ensemble of CNNs for Road Segmentation

Abstract Automatic identification of (paved) roads is crucial for applications such as autonomous navigation and urban planning. We use high-resolution aerial images, airborne Lidar, and low-resolution multi-spectral images to perform pixel-wise segmentation of road pixels. We utilize multi-modal data along with binary and multi-class ground truth information to train convolutional neural networks (CNNs) and modify an existing CNN architecture to enable training on the lower resolution multi-spectral images. We propose an ensemble of CNNs, called QuadRoad, which combines the results of four trained CNN models to boost the binary road segmentation accuracy. QuadRoad achieves higher accuracy than existing road segmentation techniques.

Depth-Wise Separable Convolutions and Multi-Level Pooling for an Efficient Spatial CNN-Based Steganalysis

For steganalysis, many studies showed that convolutional neural network (CNN) has better performances than the two-part structure of traditional machine learning methods. Existing CNN architectures use various tricks to improve the performance of steganalysis, such as fixed convolutional kernels, the absolute value layer, data augmentation and the domain knowledge. However, some designing of the network structure were not extensively studied so far, such as different convolutions (inception, xception, etc.) and variety ways of pooling(spatial pyramid pooling, etc.). In this paper, we focus on designing a new CNN network structure to improve detection accuracy of spatial-domain steganography. First, we use $3\times 3$ kernels instead of the traditional $5\times 5$ kernels and optimize convolution kernels in the preprocessing layer. The smaller convolution kernels are used to reduce the number of parameters and model the features in a small local region. Next, we use separable convolutions to utilize channel correlation of the residuals, compress the image content and increase the signal-to-noise ratio (between the stego signal and the image signal). Then, we use spatial pyramid pooling (SPP) to aggregate the local features and enhance the representation ability of features by multi-level pooling. Finally, data augmentation is adopted to further improve network performance. The experimental results show that the proposed CNN structure is significantly better than other five methods such as SRM, Ye-Net, Xu-Net, Yedroudj-Net and SRNet, when it is used to detect three spatial algorithms such as WOW, S-UNIWARD and HILL with a wide variety of datasets and payloads.

SCNN: A General Distribution Based Statistical Convolutional Neural Network with Application to Video Object Detection

Various convolutional neural networks (CNNs) were developed recently that achieved accuracy comparable with that of human beings in computer vision tasks such as image recognition, object detection and tracking, etc. Most of these networks, however, process one single frame of image at a time, and may not fully utilize the temporal and contextual correlation typically present in multiple channels of the same image or adjacent frames from a video, thus limiting the achievable throughput. This limitation stems from the fact that existing CNNs operate on deterministic numbers. In this paper, we propose a novel statistical convolutional neural network (SCNN), which extends existing CNN architectures but operates directly on correlated distributions rather than deterministic numbers. By introducing a parameterized canonical model to model correlated data and defining corresponding operations as required for CNN training and inference, we show that SCNN can process multiple frames of correlated images effectively, hence achieving significant speedup over existing CNN models. We use a CNN based video object detection as an example to illustrate the usefulness of the proposed SCNN as a general network model. Experimental results show that even a nonoptimized implementation of SCNN can still achieve 178% speedup over existing CNNs with slight accuracy degradation.