Multi-Scale convolutional neural networks integrated with self-attention for motor imagery EEG decoding

Objective To propose a multi-scale convolutional neural network (CNN) based lesions detection method of fundus image, and evaluate its application in diabetic retinopathy (DR) assisted diagnosis. Methods A multi-scale CNN based on lesions detection method of fundus image was proposed.Compared with the existing detection methods, the problem of poor robustness based on threshold segmentation and morphological segmentation was overcome.The idea of multi-scale grids detection without relying on manual pixel-by-pixel labeling was adopted in this algorithm, and the detection performance of small lesions was significantly improved.In addition, multiple DR lesions with high accuracy could be detected by the proposed loss function under the condition of weak labels and small data sets. Results At the level of lesions, the sensitivity and specificity of hard exudation lesions detection were 92.17% and 97.17%, respectively.Compared with single-scale method, the sensitivity and accuracy of multi-scale method proposed in this paper increased by 7.41% and 5.02%, respectively, and compared with other algorithm using the same public dataset IDRiD, the specificity of this algorithm increased by 55.82%.This method could effectively detect the lesions in fundus images, and could give the basic range of the lesions.The average detection time of fundus images with a large number of lesions was 1.59 seconds. Conclusions The DR lesions in the fundus image can be quickly and reliably identified, the location information of the lesions can be marked, and the influence of subjective factors can be reduced by using this algorithm, and it can be used to assist the clinician to conduct more effectively. Key words: Artificial intelligence; Diabetic retinopathy/diagnosis; Fundus color photograph; Multi-scale convolutional neural network

Controlled cooling technology is widely used in hot-rolled steel plate production lines. The final cooling temperature directly affects the microstructure and properties of steel plates, but cooling and heat transfer constitutes a nonlinear process, which is difficult to be accurately described using a mathematical model. In order to improve the accuracy of the controlled cooling temperature, a multi-scale convolutional neural network is used to predict the final cooling temperature. Convolution kernels with different sizes are introduced in the layer of a multi-scale convolutional neural network. This structure can simultaneously extract the feature information of different sizes and improve the perceptual power of the network model. The measured steel plate thickness, speed, header flow, and other variables are taken as input. The final cooling temperature is taken as the output and predicted using a multi-scale convolutional neural network. The results show that the multi-scale convolution neural network prediction model has strong generalization and nonlinear fitting ability. Compared with the traditionally structured BP neural network and convolution neural network (CNN), the mean square error (MSE) of the multi-scale convolutional neural network decreased by 24.7% and 12.2%, the mean absolute error (MAE) decreased by 19.6% and 7.97%, and the coefficient of determination (R2) improved by 4.26% and 2.65%, respectively. The final cooling temperature traditional structure by the multi-scale CNN agreed with the actual temperature within ±10% error bands. As the prediction accuracy improved, the multi-scale CNN can be effectively applied to hot-rolled steel plate production.

Objective. Electroencephalography (EEG) is an important kind of bioelectric signal for measuring physiological activities of the brain, and motor imagery (MI) EEG has significant clinical application prospects. Convolutional neural network has become a mainstream algorithm for MI EEG classification, however lack of subject-specific data considerably restricts its decoding accuracy and generalization performance. To address this challenge, a novel transfer learning (TL) framework using auxiliary dataset to improve the MI EEG classification performance of target subject is proposed in this paper. Approach. We developed a multi-source deep domain adaptation ensemble framework (MSDDAEF) for cross-dataset MI EEG decoding. The proposed MSDDAEF comprises three main components: model pre-training, deep domain adaptation, and multi-source ensemble. Moreover, for each component, different designs were examined to verify the robustness of MSDDAEF. Main results. Bidirectional validation experiments were performed on two large public MI EEG datasets (openBMI and GIST). The highest average classification accuracy of MSDDAEF reaches 74.28% when openBMI serves as target dataset and GIST serves as source dataset. While the highest average classification accuracy of MSDDAEF is 69.85% when GIST serves as target dataset and openBMI serves as source dataset. In addition, the classification performance of MSDDAEF surpasses several well-established studies and state-of-the-art algorithms. Significance. The results of this study show that cross-dataset TL is feasible for left/right-hand MI EEG decoding, and further indicate that MSDDAEF is a promising solution for addressing MI EEG cross-dataset variability.

The construction of health indicators (HI) is an essential parts of prognostics and health management (PHM). A good health indicator can accurately reflect the mechanical degradation process. Currently, most health indicator construction methods rely on abundant expert knowledge, but expert knowledge may be challenging to obtain. To solve this problem, this paper proposes a HI construction method based on unsupervised parallel multiscale convolutional long short-term memory neural network (PMCLSTM). Firstly, multiscale convolutional neural network (MSCNN) and multiscale long short-term memory neural network (MSLSTM) were used to extract multi-dimensional local and global features in parallel. Under the action of the LSTM neural network, the local jitter caused by CNN can be effectively reduced. Then, the linear regression (LR) model was used to fuse and reduce the acquired features to construct feature vectors. Finally, the relative similarity of the feature vector between the initial sample data and the current sample data is calculated, and the normalization method is used to construct the health indicator. The model is validated on the railway wagon wheel wear dataset and compared quantitatively with some advanced methods using two metrics, monotonicity and trendability. The experimental results show that the constructed HI is improved in both monotonicity and trendability, and the predicted remaining useful life of the wheel is in good agreement with the real remaining useful life, which can effectively identify the mechanical degradation process.

As an important transmission part, planetary gearbox is mainly used to output power and change the transmission power ratio of the whole transmission system. It is widely used in various complex mechanical equipment. Because the planetary gearbox often operates under heavy load and high speed, the gears and bearings will inevitably wear and malfunction, which will cause casualties and economic losses in serious cases. In order to accurately diagnose the fault of planetary gearbox, this paper design a fault diagnosis method based on multi-scale convolution neural network, and built a fault diagnosis simulation test-bed. At the same time, this paper compares the diagnosis accuracies of the multi-scale convolutional neural network with the traditional convolutional neural network and multi-core convolutional neural network. The result shows that the multiscale convolutional neural network has high recognition accuracy and is suitable for the fault diagnosis of planetary gearbox.

Brain-computer interface (BCI) based on motor imagery (MI) electroencephalogram (EEG) decoding helps motor-disabled patients to communicate with external devices directly, which can achieve the purpose of human-computer interaction and assisted living. MI EEG decoding has a core problem which is extracting as many multiple types of features as possible from the multi-channel time series of EEG to understand brain activity accurately. Recently, deep learning technology has been widely used in EEG decoding. However, the variability of the simple network framework is insufficient to satisfy the complex task of EEG decoding. A multi-scale fusion convolutional neural network based on the attention mechanism (MS-AMF) is proposed in this paper. The network extracts spatio temporal multi-scale features from multi-brain regions representation signals and is supplemented by a dense fusion strategy to retain the maximum information flow. The attention mechanism we added to the network has improved the sensitivity of the network. The experimental results show that the network has a better classification effect compared with the baseline method in the BCI Competition IV-2a dataset. We conducted visualization analysis in multiple parts of the network, and the results show that the attention mechanism is also convenient for analyzing the underlying information flow of EEG decoding, which verifies the effectiveness of the MS-AMF method.

Background: New drugs are costly, time-consuming, and often accompanied by safety concerns. With the development of deep learning, computer-aided drug design has become more mainstream, and convolutional neural networks and graph neural networks have been widely used for drug–target affinity (DTA) prediction. Objective: The paper proposes a method of predicting DTA using graph convolutional networks and multiscale convolutional neural networks. Methods: We construct drug molecules into graph representation vectors and learn feature expressions through graph attention networks and graph convolutional networks. A three-branch convolutional neural network learns the local and global features of protein sequences, and the two feature representations are merged into a regression module to predict the DTA. Results: We present a novel model to predict DTA, with a 2.5% improvement in the consistency index and a 21% accuracy improvement in terms of the mean squared error on the Davis dataset compared to DeepDTA. Morever, our method outperformed other mainstream DTA prediction models namely, GANsDTA, WideDTA, GraphDTA and DeepAffinity. Conclusion: The results showed that the use of multiscale convolutional neural networks was better than a single-branched convolutional neural network at capturing protein signatures and the use of graphs to express drug molecules yielded better results.

BackgroundEntity relation extraction technology can be used to extract entities and relations from medical literature, and automatically establish professional mapping knowledge domains. The classical text classification model, convolutional neural networks for sentence classification (TEXTCNN), has been shown to have good classification performance, but also has a long-distance dependency problem, which is a common problem of convolutional neural networks (CNNs). Recurrent neural networks (RNN) address the long-distance dependency problem but cannot capture text features at a specific scale in the text.MethodsTo solve these problems, this study sought to establish a model with a multi-scale convolutional recurrent neural network for Sentence Classification (TEXTCRNN) to address the deficiencies in the 2 neural network structures. In entity relation extraction, the entity pair is generally composed of a subject and an object, but as the subject in the entity pair of medical literature is always omitted, it is difficult to use this coding method to obtain general entity position information. Thus, we proposed a new coding method to obtain entity position information to re-establish the relationship between subject and object and complete the entity relation extraction.ResultsBy comparing the benchmark neural network model and 2 typical multi-scale TEXTCRNN models, the TEXTCRNN [bidirectional long- and short-term memory (BiLSTM)] and TEXTCRNN [double-layer stacking gated recurrent unit (GRU)], the results showed that the multi-scale CRNN model had the best F1 value performance, and the TEXTCRNN (double-layer stacking GRU) was more capable of entity relation classification when the same entity word did not belong to the same entity relation.ConclusionsThe experimental results of the entity relation extraction from Pharmacopoeia of the People’s Republic of China—Guidelines for Clinical Drug Use—Volume of Chemical Drugs and Biological Products showed that entity relation extraction could effectively proceed using the new labeling method. Additionally, compared to typical neural network models, including the TEXTCNN, GRU, and BiLSTM, the multi-scale convolutional recurrent neural network structure had advantages across several evaluation indicators.

Whole-slide imaging (WSI) is the digitization of conventional glass slides. Automatic computer-aided diagnosis (CAD) based on WSI enables digital pathology and the integration of pathology with other data like genomic biomarkers. Numerous computational algorithms have been developed for WSI, with most of them taking the image patches cropped from the highest resolution as the input. However, these models exploit only the local information within each patch and lost the connections between the neighboring patches, which may contain important context information. In this paper, we propose a novel multi-scale convolutional network (ConvNet) to utilize the built-in image pyramids of WSI. For the concentric image patches cropped at the same location of different resolution levels, we hypothesize the extra input images from lower magnifications will provide context information to enhance the prediction of patch images. We build corresponding ConvNets for feature representation and then combine the extracted features by 1) late fusion: concatenation or averaging the feature vectors before performing classification, 2) early fusion: merge the ConvNet feature maps. We have applied the multi-scale networks to a benchmark breast cancer WSI dataset. Extensive experiments have demonstrated that our multiscale networks utilizing the WSI image pyramids can achieve higher accuracy for the classification of breast cancer. The late fusion method by taking the average of feature vectors reaches the highest accuracy (81.50%), which is promising for the application of multi-scale analysis of WSI.

Motor imagery EEG recognition based on conditional optimization empirical mode decomposition and multi-scale convolutional neural network

Daily acquisition of large amounts of aerial and satellite images has facilitated subsequent automatic interpretations of these images. One such interpretation is object detection. Despite the great progress made in this domain, the detection of multi-scale objects, especially small objects in high resolution satellite (HRS) images, has not been adequately explored. As a result, the detection performance turns out to be poor. To address this problem, we first propose a unified multi-scale convolutional neural network (CNN) for geospatial object detection in HRS images. It consists of a multi-scale object proposal network and a multi-scale object detection network, both of which share a multi-scale base network. The base network can produce feature maps with different receptive fields to be responsible for objects with different scales. Then, we use the multi-scale object proposal network to generate high quality object proposals from the feature maps. Finally, we use these object proposals with the multi-scale object detection network to train a good object detector. Comprehensive evaluations on a publicly available remote sensing object detection dataset and comparisons with several state-of-the-art approaches demonstrate the effectiveness of the presented method. The proposed method achieves the best mean average precision (mAP) value of 89.6%, runs at 10 frames per second (FPS) on a GTX 1080Ti GPU.

We propose Multi-scale convolutional recurrent neural networks (MCRNN) and data augmentation methods to detect polyphonic sound event with few training data. MCRNN consists of Multi-scale convolutional neural networks (MCNN) and recurrent neural networks (RNN). MCNN concatenates the higher level features extracted using multiple convolution kernels with different scales from the time domain and frequency domain at the same time. RNN is able to capture the longer term temporal context characteristics. A novel background spectrum random replacement (BSRR) data augmentation method is applied to expand training data, which uses standard normal distribution data with randomly selected position and length instead of the original time-domain, frequency-domain or time-frequency domain background spectrum features. Our method is tested on the datasets of DCASE 2019 Task3 (T3). The experimental results showed that the MCRNN and BSRR data augmentation method are efficient. We achieved better results than the first place and the single advanced on the T3 by applying BSRR and SpecAugment data augmentation method simultaneously. On the evaluation dataset (T3-eval), our best result shows 0.05 and 0.975 of error rate (ER) and F1 respectively. Our method got the best performance and relatively improved 17% and 1% than the corresponding values of the single advanced.

The purpose of crowd counting is to estimate the number of pedestrians in crowd images. Crowd counting or density estimation is an extremely challenging task in computer vision, due to large scale variations and dense scene. Current methods solve these issues by compounding multi-scale Convolutional Neural Network with different receptive fields. In this paper, a novel end-to-end architecture based on Multi-Scale Adversarial Convolutional Neural Network (MSA-CNN) is proposed to generate crowd density and estimate the amount of crowd. Firstly, a multi-scale network is used to extract the globally relevant features in the crowd image, and then fractionally-strided convolutional layers are designed for up-sampling the output to recover the loss of crucial details caused by the earlier max pooling layers. An adversarial loss is directly employed to shrink the estimated value into the realistic subspace to reduce the blurring effect of density estimation. Joint training is performed in an end-to-end fashion using a combination of Adversarial loss and Euclidean loss. The two losses are integrated via a joint training scheme to improve density estimation performance.We conduct some extensive experiments on available datasets to show the significant improvements and supremacy of the proposed approach over the available state-of-the-art approaches.

In this study, a multi-scale high-density convolutional neural network (MHCNN) classification method for spatial cognitive ability assessment was proposed, aiming at achieving the binary classification of task-state EEG signals before and after spatial cognitive training. Besides, the multi-dimensional conditional mutual information method was used to extract the frequency band features of the EEG data. And the coupling features under the combination of multi-frequency bands were transformed into multi-spectral images. At the same time, the idea of Densenet was introduced to improve the multi-scale convolutional neural network. Firstly, according to the discreteness of multispectral EEG image features, two-scale convolution kernels were used to calculate and learn useful channel and frequency band feature information in multispectral image data. Secondly, to enhance feature propagation and reduce the number of parameters, the dense network was connected after the multi-scale convolutional network, and the learning rate change function of the stochastic gradient descent algorithm was optimized to objectively evaluate the training effect. The experimental results showed that compared with the classical convolution neural network (CNN) and multi-scale convolution neural network, the proposed MHCNN had better classification performance in the six frequency band combinations with the highest accuracy of 98%: Theta-Alpha2-Gamma, Alpha2-Beta2-Gamma, Beta1-Beta2-Gamma, Theta-Beta2-Gamma, Theta- Alpha1-Gamma, and Alpha1-Alpha2-Gamma. By comparing the classification results of six frequency band combinations, it was found that the combination of the Theta-Beta2-Gamma band had the best classification effect. The MHCNN classification method proposed in this research could be used as an effective biological indicator of spatial cognitive training effect and could be extended to other brain function evaluations.

A high precision intrusion detection system for network security communication based on multi-scale convolutional neural network