Parameter reduction in convolutional neural networks with kernel transposition

Artificial neural networks (ANN) which include deep learning neural networks (DNN) have problems such as the local minimal problem of Back propagation neural network (BPNN), the unstable problem of Radial basis function neural network (RBFNN) and the limited maximum precision problem of Convolutional neural network (CNN). Performance (training speed, precision, etc.) of BPNN, RBFNN and CNN are expected to be improved. Main works are as follows: Firstly, based on existing BPNN and RBFNN, Wavelet neural network (WNN) is implemented in order to get better performance for further improving CNN. WNN adopts the network structure of BPNN in order to get faster training speed. WNN adopts the wavelet function as an activation function, whose form is similar to the radial basis function of RBFNN, in order to solve the local minimum problem. Secondly, WNN-based Convolutional wavelet neural network (CWNN) method is proposed, in which the fully connected layers (FCL) of CNN is replaced by WNN. Thirdly, comparative simulations based on MNIST and CIFAR-10 datasets among the discussed methods of BPNN, RBFNN, CNN and CWNN are implemented and analyzed. Fourthly, the wavelet-based Convolutional Neural Network (WCNN) is proposed, where the wavelet transformation is adopted as the activation function in Convolutional Pool Neural Network (CPNN) of CNN. Fifthly, simulations based on CWNN are implemented and analyzed on the MNIST dataset. Effects are as follows: Firstly, WNN can solve the problems of BPNN and RBFNN and have better performance. Secondly, the proposed CWNN can reduce the mean square error and the error rate of CNN, which means CWNN has better maximum precision than CNN. Thirdly, the proposed WCNN can reduce the mean square error and the error rate of CWNN, which means WCNN has better maximum precision than CWNN.

In order to mine information from medical health data and develop intelligent application-related issues, the multi-modal medical health data feature representation learning related content was studied, and several feature learning models were proposed for disease risk assessment. In the aspect of medical text feature learning, a medical text feature learning model based on convolutional neural network is proposed. The convolutional neural network text analysis technology is applied to the disease risk assessment application. The medical data feature representation adopts the deep learning method. The learning and extraction of different disease characteristics use the same method to realize the versatility of the model. A simple preprocessing of the experimental data samples, including its power frequency denoising and lead convolution regularization, constructs a convolutional neural network for medical data feature advancement and intelligent recognition. On the basis of it, several sets of experiments were carried out to discuss the influence of the convolution kernel and the choice of learning rate on the experimental results. In addition, comparative experiments with support vector machine, BP neural network and RBF neural network are carried out. The results show that the convolutional neural network used in this paper shows obvious advantages in recognition rate and training speed compared with other methods. In the aspect of time series data feature learning, a multi-channel convolutional self-encoding neural network is proposed. Analyze the connection between fatigue and emotional abnormalities and define the concept of emotional fatigue. The proposed multi-channel convolutional neural network is used to learn the data features, and the convolutional self-encoding neural network is used to learn the facial image data features. These two characteristics and the collected physiological data are combined to perform emotional fatigue detection. An emotional fatigue detection demonstration platform for multi-modal data feature fusion is established to realize data acquisition, emotional fatigue detection and emotional feedback. The experimental results verify the validity, versatility and stability of the model.

Brain tumor segmentation with deep convolutional symmetric neural network

A convolutional fuzzy min-max neural network

The electronic nose is a gas detection instrument using the bionic olfactory mechanism, which usually consists of the gas sensor array and the gas classification algorithm. Furthermore, the capability of the gas classification algorithm is critical to the reliable accuracy of gas recognition. The process of gas classification usually involves pattern recognition of multiple time-related gas sensor response curves. Traditional gas classification algorithms are mainly machine learning methods, such as PCA, LDA, ICA, SVM, KNN, etc. These algorithms are relatively cumbersome because we need to extract handcrafted features before using them. Deep learning also has applications in electronic nose gas classification algorithms[1-4] which improve the accuracy of classification result, but most of deep neural networks have complex structures and consume huge computing resources. The spiking neural network is the third generation of artificial neural network, and its spiking neuron model which is more bionic than previous artificial neurons can process spike sequence signals[5]. The spiking neural network model has a simple structure with higher computational efficiency, and takes up less computational resources. Moreover, its time attribute makes it more suitable for processing information about time series. In order to simultaneously take advantage of the efficient feature extraction of the convolutional neural network and the high computational efficiency of the spiking neural network, our team converted the convolutional neural network into a convolutional spiking neural network(CSNN) and applied it to gas classification. The activation function layer in the traditional convolutional layer was replaced with the spiking neuron layer which used the IF or LIF spiking neuron model to transform the continuous values passed by the convolutional later into discrete values so as to achieve the transmission of spikes between layers. The first convolutional spiking layer was used as a spiking encoder, so the spiking encoding method such as Gaussian encoding was not used. The spike-firing-frequency output by the last layer of neurons was calculated to obtain the classification result. The probability that a gas sample belonged to a certain class was proportional to the spike-firing-frequency of the corresponding neuron of the class. Our team built a convolutional spiking neural network model with 9 layers of convolutional spiking layer and 2 layers of fully connected spiking layer, and used the food spoilage gas dataset collected by us and open source gas mixtures dataset[6] to evaluate the capability of our model. With regard to the gas mixtures dataset, ethylene, methane, CO and their mixed state need to be classified. After training, CSNN achieved the test accuracy of 92.6%, and the other algorithms’ test accuracy were 92.9% of ResNet-18, 91.2% of one-dimensional deep convolutional neural network(1D-DCNN) and 88.5% of SVM. As for the food spoilage gas dataset, 30 types of spoiled meat, vegetables, fruits and their mixed state samples were measured. The first task was to classify the major categories of spoiled food, furtuer, the 30 types of spoiled food odor samples were going to be divided into 4 categories: fresh food, spoiled meat, spoiled vegetables and spoiled fruits. After training, CSNN achieved the test accuracy of 81.4% which had a certain accuracy improvement comparing with 80.6% of ResNet-18, 80.1% of 1D-DCNN and 77.3% of SVM. The second task was to classify the subcategories of spoiled fruits, that is, 8 classes of spoiled fruit odor samples should be classified. After training, CSNN could achieve high test accuracy of 90.7%, and the accuracy of other algorithms was 88.8% of ResNet-18, 87.1% of 1D-DCNN and 77.9% of PCA+ANN. The CSNN output of a spoiled watermelon sample is shown in Figure 1.In conclusion, CSNN had similar odor classification performance to ResNet-18, but the computing resources occupied by CSNN was only 1/5 of ResNet-18. This research shows that the spiking neural network has the advantages of high odor classification accuracy, great calculation efficiency and occupying few computing resources. It is suitable as a gas classification algorithm of electronic nose and for further development.

Introduction. The implementation of information technologies in various spheres of public life dictates the creation of efficient and productive systems for entering information into computer systems. In such systems it is important to build an effective recognition module. At the moment, the most effective method for solving this problem is the use of artificial multilayer neural and convolutional networks. The purpose of the paper. This paper is devoted to a comparative analysis of the recognition results of handwritten characters of the Azerbaijani alphabet using neural and convolutional neural networks. Results. The analysis of the dependence of the recognition results on the following parameters is carried out: the architecture of neural networks, the size of the training base, the choice of the subsampling algorithm, the use of the feature extraction algorithm. To increase the training sample, the image augmentation technique was used. Based on the real base of 14000 characters, the bases of 28000, 42000 and 72000 characters were formed. The description of the feature extraction algorithm is given. Conclusions. Analysis of recognition results on the test sample showed: as expected, convolutional neural networks showed higher results than multilayer neural networks; the classical convolutional network LeNet-5 showed the highest results among all types of neural networks. However, the multi-layer 3-layer network, which was input by the feature extraction results; showed rather high results comparable with convolutional networks; there is no definite advantage in the choice of the method in the subsampling layer. The choice of the subsampling method (max-pooling or average-pooling) for a particular model can be selected experimentally; increasing the training database for this task did not give a tangible improvement in recognition results for convolutional networks and networks with preliminary feature extraction. However, for networks learning without feature extraction, an increase in the size of the database led to a noticeable improvement in performance. Keywords: neural networks, feature extraction, OCR.

Most speech separation studies in monaural channel use only a single type of network, and the separation effect is typically not satisfactory, posing difficulties for high quality speech separation. In this study, we propose a convolutional recurrent neural network with an attention (CRNN-A) framework for speech separation, fusing advantages of two networks together. The proposed separation framework uses a convolutional neural network (CNN) as the front-end of a recurrent neural network (RNN), alleviating the problem that a sole RNN cannot effectively learn the necessary features. This framework makes use of the translation invariance provided by CNN to extract information without modifying the original signals. Within the supplemented CNN, two different convolution kernels are designed to capture information in both the time and frequency domains of the input spectrogram. After concatenating the time-domain and the frequency-domain feature maps, the feature information of speech is exploited through consecutive convolutional layers. Finally, the feature map learned from the front-end CNN is combined with the original spectrogram and is sent to the back-end RNN. Further, the attention mechanism is further incorporated, focusing on the relationship among different feature maps. The effectiveness of the proposed method is evaluated on the standard dataset MIR-1K and the results prove that the proposed method outperforms the baseline RNN and other popular speech separation methods, in terms of GNSDR (gloabl normalised source-to-distortion ratio), GSIR (global source-to-interferences ratio), and GSAR (gloabl source-to-artifacts ratio). In summary, the proposed CRNN-A framework can effectively combine the advantages of CNN and RNN, and further optimise the separation performance via the attention mechanism. The proposed framework can shed a new light on speech separation, speech enhancement, and other related fields.

The quality of classification is crucial in medical applications. Especially when it comes to confirm that the patient does not have a malignant tumor. An example of such an application is a binary classification of breast tumor malignancy based on histopathological images. This paper explains the most popular attention mechanism in convolution neural networks as follows. Convolutional Block Attention Module, Attention Augmented Convolution, and Attention Guided Convolutional Neural Networks. Four neural networks are built and compared. Each is evaluated in the classification problem of histopathological images of breast cancer. On the basis of the results, it is clear that some attentional neural networks can outperform standard convolutional networks in the classification of breast cancer. In our investigation, the convolution networks reached an accuracy level of 90% and an AUC-ROC of 95.9%. It is worse compared to the Convolutional Block Attention Module Network (accuracy 90.7%, AUC-ROC 96.9%) and the Attention-Guided Convolutional Network (accuracy 91.2%, AUC-ROC 96.6%). Attention-augmented convolution remains behind the standard convolutional network, achieving 88.9% accuracy and 94.8% AUC-ROC. The Attention-Guided Convolution Network was the best network of all four. We also compared precision, NPV, sensitivity, specificity, and \(F_{1}\)-score. We came to the conclusion that the Convolutional Block Attention Module network has the highest NPV (90.8%) and sensitivity (96.2%), while the Attention-Guided Convolutional Neural Network scored the highest in precision (92.4%) and specificity (82.9%).KeywordsConvolutional neural networksAttentional neural networksBreast cancerHistopathologic imagesConvolutional block attention moduleAttention-augmented convolutionAttention-guided convolutional neural network

Convolutional neural network (CNN) is recognized as state of the art of deep learning algorithm, which has a good ability on the image classification and recognition. The problems of CNN are as follows: the precision, accuracy and efficiency of CNN are expected to be improved to satisfy the requirements of high performance. The main work is as follows: Firstly, wavelet convolutional neural network (wCNN) is proposed, where wavelet transform function is added to the convolutional layers of CNN. Secondly, wavelet convolutional wavelet neural network (wCwNN) is proposed, where fully connected neural network (FCNN) of wCNN and CNN are replaced by wavelet neural network (wNN). Thirdly, image classification experiments using CNN, wCNN and wCwNN algorithms, and comparison analysis are implemented with MNIST dataset. The effect of the improved methods are as follows: (1) Both precision and accuracy are improved. (2) The mean square error and the rate of error are reduced. (3) The complexitie of the improved algorithms is increased.

Tunnel boring machine vibration-based deep learning for the ground identification of working faces

Monitoring tool usage in surgery videos using boosted convolutional and recurrent neural networks.

In this paper, we investigate the performance of two deep learning paradigms for the audio-based tasks of acoustic scene, environmental sound and domestic activity classification. In particular, a convolutional recurrent neural network (CRNN) and pre-trained convolutional neural networks (CNNs) are utilised. The CRNN is directly trained on Mel-spectrograms of the audio samples. For the pre-trained CNNs, the activations of one of the top layers of various architectures are extracted as feature vectors and used for training a linear support vector machine (SVM).Moreover, the predictions of the two models—the class probabilities predicted by the CRNN and the decision function of the SVM—are combined in a decision-level fusion to achieve the final prediction. For the pre-trained CNN networks we use as feature extractors, we further evaluate the effects of a range of configuration options, including the choice of the pre-training corpus. The system is evaluated on the acoustic scene classification task of the IEEE AASP Challenge on Detection and Classification of Acoustic Scenes and Events (DCASE 2017) workshop, ESC-50 and the multi-channel acoustic recordings from DCASE 2018, task 5. We have refrained from additional data augmentation as our primary goal is to analyse the general performance of the proposed system on different datasets. We show that using our system, it is possible to achieve competitive performance on all datasets and demonstrate the complementarity of CRNNs and ImageNet pre-trained CNNs for acoustic classification tasks. We further find that in some cases, CNNs pre-trained on ImageNet can serve as more powerful feature extractors than AudioSet models. Finally, ImageNet pre-training is complimentary to more domain-specific knowledge, either in the form of the convolutional recurrent neural network (CRNN) trained directly on the target data or the AudioSet pre-trained models. In this regard, our findings indicate possible benefits of applying cross-modal pre-training of large CNNs to acoustic analysis tasks.

Identification of crops is an important topic in the agricultural domain. Hyperspectral remote sensing data are very useful for crop feature extraction and classification. Remote sensing data is an unstructured data and Convolutional Neural Network (CNN) can work on unstructured data efficiently. This paper presents an evaluation of CNN for crop classification using the Indian Pines standard dataset obtained from the AVIRIS sensor and the study area dataset obtained from the EO-1hyperion sensor. Optimized CNN has been tuned by training the model on different parameters. It has been compared with two classification algorithms: Deep Neural Network (DNN) and Convolutional Autoencoder. According to the test results, the proposed optimized CNN model provided better results as compared to the other two methods. CNN has given 97 ± 0.58% overall accuracy for the Indian Pines standard dataset and 78 ± 2.43% for our study area dataset.

The evolution in the field of Artificial Intelligent (AI) with its training algorithms make AI very important in different aspect of the life. The prediction problem of behavior of dynamical control system is one of the most important issue that the AI can be employed to solve it. In this paper, a Convolutional Multi-Spike Neural Network (CMSNN) is proposed as smart system to predict the response of nonlinear dynamical systems. The proposed structure mixed the advantages of Convolutional Neural Network (CNN) with Multi -Spike Neural Network (MSNN) to generate the smart structure. The CMSNN has the capability of training weights based on a proposed training algorithm. The simulation results demonstrated that the proposed structure has the ability to predict the response of dynamical systems more powerful than with the CNN. The proposed structure is more powerful than the CNN by 28.33% in terms of minimizing the root mean square error.

paper, we focus on exploring the relationship between weave patterns and their mechanical properties in woven fiber composites through Machine Learning. Specifically, we explore the interactions between woven architectures and in-plane stiffness properties through Deep Convolutional Neural Network (DCNN) and Generative Adversarial Network (GAN). Our research is important for exploring how woven composite’s pattern is related to its mechanical properties and accelerating woven composite design as well as optimization. We focus on two tasks: (1) Stiffness prediction: Predicting in-plane stiffness properties for given weave patterns. Our DCNN extracts high-level features through several convolutional and fully connected layers to determine the final predictions. (2) Weave pattern prediction: Predicting weave patterns for target stiffness properties, which can be treated as the reverse task of the first one. Due to many-to-one mapping between weave patterns and the composite properties, we utilize a Decoder Neural Network as our baseline model and compare its performance with GAN and Genetic Algorithm. We represent the weave patterns as 2D checkerboard models and use finite element analysis (FEA) to determine in-plane stiffness properties, which serve as input data for our ML framework. We show that: (1) for stiffness prediction, DCNN can predict stiffness values for a given weave pattern with relatively high accuracy (above 93%); (2) for weave pattern prediction, the GAN model gives the best prediction accuracy (above 92%) while Decoder Neural Network has the best time efficiency. HAOTIAN FENG