Fast Neural Network Research Articles

Fast Region Convolutional Neural Network Lane and Road Defect Detection for Autonomous Vehicle Application

Fast Region Convolutional Neural Network Lane and Road Defect Detection for Autonomous Vehicle Application

The use of fast molecular descriptors and artificial neural networks approach in organochlorine compounds electron ionization mass spectra classification

Developing of theoretical tools can be very helpful for supporting new pollutant detection. Nowadays, a combination of mass spectrometry and chromatographic techniques are the most basic environmental monitoring methods. In this paper, two organochlorine compound mass spectra classification systems were proposed. The classification models were developed within the framework of artificial neural networks (ANNs) and fast 1D and 2D molecular descriptor calculations. Based on the intensities of two characteristic MS peaks, namely, [M] and [M-35], two classification criterions were proposed. According to criterion I, class 1 comprises [M] signals with the intensity higher than 800 NIST units, while class 2 consists of signals with the intensity lower or equal than 800. According to criterion II, class 1 consists of [M-35] signals with the intensity higher than 100, while signals with the intensity lower or equal than 100 belong to class 2. As a result of ANNs learning stage, five models for both classification criterions were generated. The external model validation showed that all ANNs are characterized by high predicting power; however, criterion I-based ANNs are much more accurate and therefore are more suitable for analytical purposes. In order to obtain another confirmation, selected ANNs were tested against additional dataset comprising popular sunscreen agents disinfection by-products reported in previous works.

Private Model Compression via Knowledge Distillation

The soaring demand for intelligent mobile applications calls for deploying powerful deep neural networks (DNNs) on mobile devices. However, the outstanding performance of DNNs notoriously relies on increasingly complex models, which in turn is associated with an increase in computational expense far surpassing mobile devices’ capacity. What is worse, app service providers need to collect and utilize a large volume of users’ data, which contain sensitive information, to build the sophisticated DNN models. Directly deploying these models on public mobile devices presents prohibitive privacy risk. To benefit from the on-device deep learning without the capacity and privacy concerns, we design a private model compression framework RONA. Following the knowledge distillation paradigm, we jointly use hint learning, distillation learning, and self learning to train a compact and fast neural network. The knowledge distilled from the cumbersome model is adaptively bounded and carefully perturbed to enforce differential privacy. We further propose an elegant query sample selection method to reduce the number of queries and control the privacy loss. A series of empirical evaluations as well as the implementation on an Android mobile device show that RONA can not only compress cumbersome models efficiently but also provide a strong privacy guarantee. For example, on SVHN, when a meaningful (9.83,10−6)-differential privacy is guaranteed, the compact model trained by RONA can obtain 20× compression ratio and 19× speed-up with merely 0.97% accuracy loss.

A novel adaptive and fast deep convolutional neural network for bearing fault diagnosis under different working conditions

Deep learning method is gradually applied in the field of mechanical equipment fault diagnosis because it can learn complex and useful features automatically from the vibration signals. Among the many intelligent diagnostic models, convolutional neural network has been gradually applied to intelligent fault diagnosis of bearings due to its advantages of local connection and weight sharing. However, there are still some drawbacks. (1) The training process of convolutional neural network is slow and unstable. It has more training parameters. (2) It cannot perform well under different working conditions, such as noisy environment and different workloads. In this paper, a novel model named adaptive and fast convolutional neural network with wide receptive field is presented to overcome the aforementioned deficiencies. The prime innovations include the following. First, a deep convolutional neural network architecture is constructed using the scaled exponential linear unit activation function and global average pooling. The model has fewer training parameters and can converge rapidly and stably. Second, the model has a wide receptive field with two medium and three small length convolutional kernels. It also has high diagnostic accuracy and robustness when the environment is noisy and workloads are changed compared with other models. Furthermore, to demonstrate how the wide receptive field convolutional neural network model works, the reasons for high model performance are analyzed and the learned features are also visualized. Finally, the wide receptive field convolutional neural network model is verified by the vibration dataset collected in the background of high noise, and the results indicate that it has high diagnostic performance.

Non-Invasive Optical Blood Glucose Measurement based on Discrete Fourier Transform and Fast Artificial Neural Network: Fasting Normal Glucose Participants Case Study

A prototype non-invasive blood glucose level measurement optical device (NI-BGL-MOD) has been developed. The NI-BGL-MOD uses a discrete Fourier transform (DFT) method and a fast artificial neural network algorithm to optimize device performance. The appropriate light-emitting diode for the sensory module was selected based on near-infrared spectrophotometry of a blood glucose model and human blood. DFT is implemented in an analog-to-digital converter module. An in vitro trial using the blood glucose model along with a clinical trial involving 110 participants were conducted to evaluate the performance of the prototype. The root-mean-square error of the prototype was 10.8 mg/dl in the in vitro trial and 3.64 mg/dl in the clinical trial, which is lower than the ISO-15197:2016 mandated value of 10 mg/dl. In each trial, consensus error grid analysis indicated that the measurement error was within the safe range. The sensitivity and specificity of the prototype were 0.83 (0.36, 1.00) and 0.90 (0.55, 1.00) in the in vitro trial and 0.81 (0.75, 0.85) and 0.83 (0.78, 0.87) in the clinical trial, respectively. In general, the proposed NI-BGL-MOD demonstrated good performance than gold-standard measurement. Key words: Non-invasive blood glucose measurement, optical device, discrete Fourier transform, multi-formulatric regression, fast artificial neural network

Pedestrian and Cyclist Detection and Intent Estimation for Autonomous Vehicles: A Survey

As autonomous vehicles become more common on the roads, their advancement draws on safety concerns for vulnerable road users, such as pedestrians and cyclists. This paper presents a review of recent developments in pedestrian and cyclist detection and intent estimation to increase the safety of autonomous vehicles, for both the driver and other road users. Understanding the intentions of the pedestrian/cyclist enables the self-driving vehicle to take actions to avoid incidents. To make this possible, development of methods/techniques, such as deep learning (DL), for the autonomous vehicle will be explored. For example, the development of pedestrian detection has been significantly advanced using DL approaches, such as; Fast Region-Convolutional Neural Network (R-CNN) , Faster R-CNN and Single Shot Detector (SSD). Although DL has been around for several decades, the hardware to realise the techniques have only recently become viable. Using these DL methods for pedestrian and cyclist detection and applying it for the tracking, motion modelling and pose estimation can allow for a successful and accurate method of intent estimation for the vulnerable road users. Although there has been a growth in research surrounding the study of pedestrian detection using vision-based approaches, further attention should include focus on cyclist detection. To further improve safety for these vulnerable road users (VRUs), approaches such as sensor fusion and intent estimation should be investigated.

Adversarial environment reinforcement learning algorithm for intrusion detection

Intrusion detection is a crucial service in today’s data networks, and the search for new fast and robust algorithms that are capable of detecting and classifying dangerous traffic is essential to deal with changing threats and increasing detection difficulty. In this work, we present a new intrusion detection algorithm with an excellent prediction performance. The prediction is based on a classifier which is a simple and extremely fast neural network. The classifier implements a policy function that is trained with a novel reinforcement learning model, where the behavior of the environment is adjusted in parallel with the learning process.Intrusion detection frameworks are based on a supervised learning paradigm that uses a training dataset composed of network features and associated intrusion labels. In this work, we integrate this paradigm with a reinforcement learning algorithm that is normally based on interaction with a live environment (not a pre-recorded dataset). To perform the integration, the live environment is replaced by a simulated one.The principle of this approach is to provide the simulated environment with an intelligent behavior by, first, generating new samples by randomly extracting them from the training dataset, generating rewards that depend on the goodness of the classifier's predictions, and, second, by further adjusting this initial behavior with an adversarial objective in which the environment will actively try to increase the difficulty of the prediction made by the classifier. In this way, the simulated environment acts as a second agent in an adversarial configuration against the original agent (the classifier). We prove that this architecture increases the final performance of the classifier.This work presents the first application of adversarial reinforcement learning for intrusion detection, and provides a novel technique that incorporates the environment's behavior into the learning process of a modified reinforcement learning algorithm.We prove that the proposed algorithm is adequate for a supervised learning problem based on a labeled dataset. We validate its performance by comparing it with other well-known machine learning models for two datasets. The proposed model outperforms the other models in the weighted Accuracy (>0.8) and F1 (>0.79) metrics, and especially excels in the results for the under-represented labels.

Fast orthogonal recurrent neural networks employing a novel parametrisation for orthogonal matrices

Training Recurrent Neural Networks (RNNs) is challenging due to the vanishing/exploding gradient problem. Recent progress suggests to solve this problem by constraining the recurrent transition matrix to be unitary/orthogonal during training, but all of which are either limited-capacity, or involve time-consuming operators, e.g., evaluation for the derivation of lengthy matrix chain multiplication, the matrix exponential, or the singular value decomposition. This paper addresses this problem based on the exponentials of sparse antisymmetric matrices with one or more nonzero columns and an equal number of nonzero rows from a geometric view. An analytical expression is presented to simplify the computation of the sparse antisymmetric matrix exponential, which is actually a novel formula for parameterizing orthogonal matrices. The algorithms of this paper are fast, tunable, and full-capacity, where the target variable is updated by optimizing a matrix multiplier, instead of using the explicit gradient descent. Experiments demonstrate the superior performance of our proposed algorithms.

Evaluation of Deep Learning Neural Networks for Surface Roughness Prediction Using Vibration Signal Analysis

The use of surface roughness (Ra) to indicate product quality in the milling process in an intelligent monitoring system applied in-process has been developing. From the considerations of convenient installation and cost-effectiveness, accelerator vibration signals combined with deep learning predictive models for predicting surface roughness is a potential tool. In this paper, three models, namely, Fast Fourier Transform-Deep Neural Networks (FFT-DNN), Fast Fourier Transform Long Short Term Memory Network (FFT-LSTM), and one-dimensional convolutional neural network (1-D CNN), are used to explore the training and prediction performances. Feature extraction plays an important role in the training and predicting results. FFT and the one-dimensional convolution filter, known as 1-D CNN, are employed to extract vibration signals’ raw data. The results show the following: (1) the LSTM model presents the temporal modeling ability to achieve a good performance at higher Ra value and (2) 1-D CNN, which is better at extracting features, exhibits highly accurate prediction performance at lower Ra ranges. Based on the results, vibration signals combined with a deep learning predictive model could be applied to predict the surface roughness in the milling process. Based on this experimental study, the use of prediction of the surface roughness via vibration signals using FFT-LSTM or 1-D CNN is recommended to develop an intelligent system.

Skeleton-based automatic generation of Labanotation with neural networks

Labanotation is one of the most widely used notation systems and plays a powerful role in recording and archiving traditional folk dance. We propose an end-to-end method for generating Labanotation from motion capture data by identifying the movement of each part of the human body and by assigning corresponding Labanotation symbols. Our method is mainly highlighted in the following aspects: first, we design simple yet highly discriminative skeleton features that can accurately represent human movements; and second, for the recognition of upper limb movements, we adopt fast and efficient extreme-learning neural networks, and for the recognition of lower limb movements, we employ powerful long short-term memory networks. It is worth mentioning that this is the first time that neural networks have been applied to the field of Labanotation generation. Experimental results show that our approach achieves much better recognition accuracy than previous work.

R-CNN-Based Ship Detection from High Resolution Remote Sensing Imagery

Offshore and inland river ship detection has been studied on both synthetic aperture radar (SAR) and optical remote sensing imagery. However, the classic ship detection methods based on SAR images can cause a high false alarm ratio and be influenced by the sea surface model, especially on inland rivers and in offshore areas. The classic detection methods based on optical images do not perform well on small and gathering ships. This paper adopts the idea of deep networks and presents a fast regional-based convolutional neural network (R-CNN) method to detect ships from high-resolution remote sensing imagery. First, we choose GaoFen-2 optical remote sensing images with a resolution of 1 m and preprocess the images with a support vector machine (SVM) to divide the large detection area into small regions of interest (ROI) that may contain ships. Then, we apply ship detection algorithms based on a region-based convolutional neural network (R-CNN) on ROI images. To improve the detection result of small and gathering ships, we adopt an effective target detection framework, Faster-RCNN, and improve the structure of its original convolutional neural network (CNN), VGG16, by using multiresolution convolutional features and performing ROI pooling on a larger feature map in a region proposal network (RPN). Finally, we compare the most effective classic ship detection method, the deformable part model (DPM), another two widely used target detection frameworks, the single shot multibox detector (SSD) and YOLOv2, the original VGG16-based Faster-RCNN, and our improved Faster-RCNN. Experimental results show that our improved Faster-RCNN method achieves a higher recall and accuracy for small ships and gathering ships. Therefore, it provides a very effective method for offshore and inland river ship detection based on high-resolution remote sensing imagery.

Pedestrian and cyclist detection based on deep neural network fast R-CNN

In this article, a unified joint detection framework for pedestrian and cyclist is established to realize the joint detection of pedestrian and cyclist targets. Based on the target detection of fast regional convolution neural network, a deep neural network model suitable for pedestrian and cyclist detection is established. Experiments for poor detection results for small-sized targets and complex and changeable background environment; various network improvement schemes such as difficult case extraction, multilayer feature fusion, and multitarget candidate region input were designed to improve detection and to solve the problems of frequent false detections and missed detections in pedestrian and cyclist target detection. Results of experimental verification of the pedestrian and cyclist database established in Beijing’s urban traffic environment showed that the proposed joint detection method for pedestrians and cyclists can realize the stable tracking of joint detection and clearly distinguish different target categories. Therefore, an important basis for the behavior decision of intelligent vehicles is provided.

Evaluation of the robusticity of mutual fund performance in Ghana using Enhanced Resilient Backpropagation Neural Network (ERBPNN) and Fast Adaptive Neural Network Classifier (FANNC)

Mutual fund investment continues to play a very important role in the world financial markets especially in developing economies where the capital market is not very matured and tolerant of small scale investors. The total mutual fund asset globally as at the end of 2016 was in excess of $40.4 trillion. Despite its success there are uncertainties as to whether mutual funds in Ghana obtain optimal performance relative to their counterparts in United States, Luxembourg, Ireland, France, Australia, United Kingdom, Japan, China and Brazil. We contribute to the extant literature on mutual fund performance evaluation using a collection of more sophisticated econometric models. We selected six continuous historical years that is 2010–2011, 2012–2013 and 2014–2015 to construct a mutual fund performance evaluation model utilizing the fast adaptive neural network classifier (FANNC), and to compare our results with those from an enhanced resilient back propagation neural networks (ERBPNN) model. Our FANNC model outperformed the existing models in terms of processing time and error rate. This makes it ideal for financial application that involves large volume of data and routine updates.

A gas source declaration scheme based on a tetrahedral sensor structure in three-dimensional airflow environments.

A gas source declaration scheme based on a tetrahedral sensor structure in three-dimensional airflow environments is proposed. First, a tetrahedral sensor structure was established. Based on the tetrahedral structure, the gas source declaration problem was converted into a two-class classification issue. Then a classification algorithm combining an extreme learning machine (ELM, a fast neural network classifier) with a gas mass flux criterion is proposed. A novel calculation method for the mass flux through a closed tetrahedral surface is presented, and a mass flux criterion was developed which acts as a training sample filter for the ELM. The source declaration scheme was validated by using both regular and irregular tetrahedron experiments.

Prediction of human assessments of dairy odor utilizing a fast gas chromatograph and neural networks

A method based on hedonic tone was developed and applied to evaluate human assessments of odor emitted from dairy operations using a fast gas chromatograph and neural networks. A general pleasantness scale ranging from −11 (extremely unpleasant) to +11 (extremely pleasant) was used to collect human responses. The panelists were able to identify the difference between various samples, and gave individually consistent responses for the same sample. The measurements of a fast gas chromatograph, called the zNose, were trained using Artificial Neural Networks (ANNs) to predict the human assessments. Three ANNs, Levenberg-Marquardt Back-propagation Neural Network (LMBNN), Scaled Conjugate Gradient Back-propagation (CGBNN), and Resilient Back-propagation Neural Network (RPBNN), were applied to connect human assessments and instrument measurements. In separate validation, zNose-LMBNN model showed superiority in four criteria, Mean Square Error (MSE), Correlation Coefficient (R), probability within 10% range to target, and probability within 5% range to target. The optimal model outputs represented human response as high as 67% within the 10% range and 44% within the 5% range of the targets. In addition, the model outputs have a good linear relationship with the targets (R = 0.53).

Fast Sparse Deep Neural Networks: Theory and Performance Analysis

In this paper, fast sparse deep neural networks that aim to offer an alternative way of learning in a deep structure are proposed. We examine some optimization algorithms for traditional deep neural networks and find that deep neural networks suffer from a time-consuming training process because of a large number of connecting parameters in layers and layers. To reduce time consumption, we propose fast sparse deep neural networks, which mainly consider the following two aspects in the design of the network. One is that the parameter learning at each hidden layer is given utilizing closed-form solutions, which is different from the BP algorithm with iterative updating strategy. Another aspect is that fast sparse deep neural networks use the summation method of a multi-layer linear approximation to estimate the output target, which is a different way from most deep neural network models. Unlike the traditional deep neural networks, fast sparse deep neural networks can achieve excellent generalization performance without fine-tuning. In addition, it is worth noting that fast sparse deep neural networks can also effectively overcome the shortcomings of the extreme learning machine and hierarchical extreme learning machine. Compared to the existing deep neural networks, enough experimental results on benchmark datasets demonstrate that the proposed model and optimization algorithms are feasible and efficient.

Atrial Fibrillation Detection Using an Improved Multi-Scale Decomposition Enhanced Residual Convolutional Neural Network

Atrial fibrillation, the most common sustained arrhythmia, is still a big challenge for researchers in the medical field. Many studies attempt to realize intelligent classification of AF based on deep learning methods. However, many of the studies focused on investigations of relatively simple datasets collected from a relatively small number of subjects. On the other hand, sophisticated preprocessing is usually adopted to analyse the ECG signals. These two factors significantly affect the generalization ability of the trained models for complicated data sets collected from a large number of subjects. In order to address this problem, an improved multi-scale decomposition enhanced residual convolutional neural network is proposed. The proposed method is applied to the large single-lead ECG dataset provided by the PhysioNet/CinC Challenge 2017, and good classification accuracy is suggested by the testing results. In the proposed method, the original ECG record with a large difference in length is re-segmented into short samples of 9 s. Then, using the derived wavelet frame decomposition, the segmented short samples are decomposed and reconstituted into sub-signal samples of different scales. We trained the fast down-sampling residual convolutional neural networks (FDResNets) with the original short-signal dataset and the reconstructed dataset of each scale. The transfer learning technique is then applied to couple the three FDResNets with good performance into a multi-scale decomposition enhanced residual convolutional neural network (MSResNet). The FDResNet trained by the [0, 9.375 Hz] reconstruction dataset achieved the best performance. After six-fold cross-validation, the average test accuracy reached 87.12%, and the average comprehensive F1 score reached 85.29%. The average test accuracy of the multi-scale residual neural network reached 92.1%, and the average overall F1 score reached 89.9%.

Improving resolution of medical images with deep dense convolutional neural network

SummaryDoctors always desire high‐resolution medical images to have accurate diagnosis. Super‐resolution (SR) is a technology that can improve the resolution of medical images. Convolutional neural network (CNN)–based SR methods have achieved desired performance in natural images. In this paper, we apply a deep dense SR (DDSR) convolutional neural networks model to two types of medical images, including Computerized Tomography (CT) images and Magnetic Resonance imaging (MRI) images. This network densely connects every hidden layer to learn high‐level features, which was first proposed for object recognition. A set of medical images is used for experiments. We compare the performance of DDSR with three state‐of‐the‐art SR network models, including SR Convolutional Neural Network (SRCNN), Fast SR Convolutional Neural Network (FSRCNN), and Very Deep SR Convolutional Neural Network (VDSR). Both the objective indices and subjective evaluations are used for comparison. The results show that the proposed network has better performances both on CT and MRI images.

Deep convolutional extreme learning machines: Filters combination and error model validation

In recent years, deep convolutional neural network models have been increasingly used in various computer vision tasks, like plate number recognition, object recognition, automatic digit recognition, and medical applications supporting diagnosis by signals or images. A disadvantage of these networks is the long training time. It can take days to adjust weights with iterative methods based on gradient descent. This can be an obstacle in applications that need frequent training or in real time. Fast convolutional networks avoid gradient-based methods by efficiently defining filters in feature extraction and weights in classification. The issue is how to set the convolutional filter banks, since they are not learned by the backpropagation of gradients? In this work we propose a deep fast convolutional neural network based on extreme learning machine and a fixed bank of filters. We demonstrate that our model is feasible to be used in cost-effective non-specialized computer hardware, performing the training task faster than models running on GPUs. Results were generated on EMNIST dataset representing the widely studied problem of digit recognition. We provide a deep convolutional extreme learning machine (CELM) with two feature extraction stages and combinations of selected filters. For the proposed network, we find that the empirical generalization error is explained by the error model based on a theorem by Rahimi and Retch. In comparison to the state-of-the-art, the proposed network resulted in superior accuracy as well as competitive training time, even in relation to approaches that employ processing in GPUs.

LightweightNet: Toward fast and lightweight convolutional neural networks via architecture distillation

In recent years, deep neural networks have achieved remarkable successes in many pattern recognition tasks. However, the high computational cost and large memory overhead hinder them from applications on resource-limited devices. To address this problem, many deep network acceleration and compression methods have been proposed. One group of methods adopt decomposition and pruning techniques to accelerate and compress a pre-trained model. Another group designs single compact unit to stack their own networks. These methods are subject to complicated training processes, or lack of generality and extensibility. In this paper, we propose a general framework of architecture distillation, namely LightweightNet, to accelerate and compress convolutional neural networks. Rather than compressing a pre-trained model, we directly construct the lightweight network based on a baseline network architecture. The LightweightNet, designed based on a comprehensive analysis of the network architecture, consists of network parameter compression, network structure acceleration, and non-tensor layer improvement. Specifically, we propose the strategy of low-dimensional features of fully-connected layers for substantial memory saving, and design multiple efficient compact blocks to distill convolutional layers of baseline network with accuracy-sensitive distillation rule for notable time saving. Finally, it can effectively reduce the computational cost and the model size by >4× with negligible accuracy loss. Benchmarks on MNIST, CIFAR-10, ImageNet and HCCR (handwritten Chinese character recognition) datasets demonstrate the advantages of the proposed framework in terms of speed, performance, storage and training process. In HCCR, our method even outperforms traditional handcrafted features-based classifiers in terms of speed and storage while maintaining state-of-the-art recognition performance.