Multi-scale Convolution Research Articles

Multitask Touch Gesture and Emotion Recognition Using Multiscale Spatiotemporal Convolutions With Attention Mechanism

During human-robot interaction, touch gesture and emotion recognition (TGER) can help robots perceive and infer human intentions and emotional states, and establish a more harmonious human-robot relationship. Conventional handcrafted feature-based machine learning methods and end-to-end deep learning networks have been used for TGER. However, existing TGER methods are mainly based on single-task models, i.e., the touch gestures and emotions are recognized separately with different models, which suffer from relatively low recognition accuracy, high computational complexity, or high memory burden. To solve these existing problems, this paper proposed a multi-scale spatiotemporal convolutions with attention mechanism for touch gesture and emotion recognition (MUSCAT), and established a multi-task TGER framework for an overall model construction. MUSCAT uses a factorized spatiotemporal convolutional neural network to sequentially extract spatial and temporal features, and multi-scale convolution is used in temporal convolution. The attention mechanisms are applied after spatial convolution, temporal convolution and point-wise convolution to respectively focus on the key information of spatial, temporal and channel dimensions. The proposed MUSCAT model has a deeper structure than the standard three-dimensional convolution, which gains better performance with fewer model parameters. Extensive experiments were conducted on two touch datasets, and the results verify the feasibility of the proposed method.

Multi-input Unet model based on the integrated block and the aggregation connection for MRI brain tumor segmentation

With the growth of data information and the development of computer equipment, it is extremely time-consuming and laborious to rely on the traditional manual segmentation of brain medical images. To solve the above problems, this paper proposes a multi-input Unet model based on the integrated block and the aggregation connection to achieve efficient and accurate segmentation of tumor structure. Besides, this paper solves the low-resolution problem in sagittal and coronal planes, which can effectively improve memory efficiency. The proposed algorithm is innovative in three aspects. Firstly, by inputting the mask images which can effectively represent the tumor location characteristics, it can provide more information about the spatial relationship to alleviate the problems of fuzzy boundary and low contrast between the lesion region and healthy brain tissue. Then, the integrated block extracts the tumor local information in different receptive domains by a multi-scale convolution kernel. The aggregation connection realizes the implicit deep connection of context information, which combines the shallow and deep information of the brain with strong geometric spatial relationships. Meanwhile, to effectively alleviate the waste of memory resources caused by redundant and background information in medical images, the amount of calculation in model training is reduced by dimension reduction of the feature map. An ablation experiment is used to verify the innovation of the proposed algorithm on the BraTS dataset, which compares with the state-of-the-art brain tumor segmentation methods. The precision of the proposed multi-input Unet model for the whole tumor and core lesion is 0.92 and 0.90, respectively.

Research on deep learning method recognition and a classification model of grassland grass species based on unmanned aerial vehicle hyperspectral remote sensing

AbstractIdentifying grass species in grasslands based on unmanned aerial vehicle hyperspectral remote sensing is the basis and premise of hyperspectral remote sensing when applied to grassland degradation monitoring and research. The small targets and mixed pixels involved grass species identification in grasslands creates problems, making identification cumbersome and classification accuracy difficult. This study involved the construction of an unmanned aerial vehicle hyperspectral remote sensing system using hyperspectral data of grass species in desert habitats that had been collected under natural light. A multi‐resolution combined with a 1 × 1 feature map was formed by multiscale convolution, and grass species data were extracted from hyperspectral fine‐grained feature data from grasslands. A recognition and classification model for degradation indicator species CNN was constructed using max pooling to retain the maximum amount of feature detail and up‐sampling, reconstructing the feature space and feature fusion to smooth the edge texture of the data and enhance the weak data to alleviate the imbalance among samples. The results showed that the overall identification accuracy of the model for grassland species reached 98.78%, and the kappa coefficient reached 0.92, realizing the high‐precision identification of grassland species, which laid the foundation for grassland species detection and research based on unmanned aerial vehicle hyperspectral imagery. In addition, the proposed degradation indicator species CNN model provides a useful reference for the identification and classification of small targets with mixed pixels.

RGB-D Image Semantic Segmentation Based on Multi-Modal Adaptive Convolution

With the availability of consumer RGB-D sensors, lots of research uses both color information and depth information for semantic segmentation. However, most previous studies simply fuse RGB features and depth features with equal-weight concatenating or summing, which may fail to effectively use complementary information between RGB information and depth information. On the other hand, previous works construct multi-scale representation by utilizing multi-scale convolution kernel with the fixed-parameter, which may lead to parameter redundancy and fail to perform online self-adaption. To effectively utilize the internal context information of multi-modal features, an RGB-D image semantic segmentation network is proposed by introducing a multi-modal adaptive convolution module. The multi-scale adaptive convolution kernel is generated dynamically, and the context information of multi-modal features is embedded into the multi-scale convolution filters effectively. Compared with the traditional multi-scale convolution kernel, proposed method has higher computational efficiency and better accuracy. Experimental results on the public RGB-D indoor semantic segmentation datasets SUN RGB-D and NYU Depth v2 show that the pixel accuracy, mean pixel accuracy, and mean IoU of proposed method is 82.5%, 62.0%, 50.6% and 77.1%, 64.2%, 50.8% Respectively, which outperforms all existing RGB-D semantic segmentation methods.

SVDistNet: Self-Supervised Near-Field Distance Estimation on Surround View Fisheye Cameras

A 360° perception of scene geometry is essential for automated driving, notably for parking and urban driving scenarios. Typically, it is achieved using surround-view fisheye cameras, focusing on the near-field area around the vehicle. The majority of current depth estimation approaches focus on employing just a single camera, which cannot be straightforwardly generalized to multiple cameras. The depth estimation model must be tested on a variety of cameras equipped to millions of cars with varying camera geometries. Even within a single car, intrinsics vary due to manufacturing tolerances. Deep learning models are sensitive to these changes, and it is practically infeasible to train and test on each camera variant. As a result, we present novel camera-geometry adaptive multi-scale convolutions which utilize the camera parameters as a conditional input, enabling the model to generalize to previously unseen fisheye cameras. Additionally, we improve the distance estimation by pairwise and patchwise vector-based self-attention encoder networks. We evaluate our approach on the Fisheye WoodScape surround-view dataset, significantly improving over previous approaches. We also show a generalization of our approach across different camera viewing angles and perform extensive experiments to support our contributions. To enable comparison with other approaches, we evaluate the front camera data on the KITTI dataset (pinhole camera images) and achieve state-of-the-art performance among self-supervised monocular methods. An overview video with qualitative results is provided at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://youtu.be/bmX0UcU9wtA</uri> . Baseline code and dataset will be made public. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><xref ref-type="fn" rid="fn1">1</xref></sup>

High-precision autocollimation method based on a multiscale convolution neural network for angle measurement.

A high-precision autocollimation method based on multiscale convolution neural network (MSCNN) for angle measurement is proposed. MSCNN is integrated with the traditional measurement model. Using the multiscale representation learning ability of MSCNN, the relationship between spot shape (large-scale feature), gray distribution (small-scale feature), and the influence of aberration and assembly error in the collimating optical path is extracted. The constructed accurate nonlinear measurement model directly improves the uncertainty of angle measurement. Experiments demonstrate that the extended uncertainty reaches 0.29 arcsec (k = 2), approximately 7 times higher than that with the traditional measurement principle, and solves the nonlinear error caused by aberration and assembly error in the autocollimation system. Additionally, this method has a good universality and can be applied to other autocollimation systems.

Prediction of protein secondary structure based on an improved channel attention and multiscale convolution module.

Prediction of the protein secondary structure is a key issue in protein science. Protein secondary structure prediction (PSSP) aims to construct a function that can map the amino acid sequence into the secondary structure so that the protein secondary structure can be obtained according to the amino acid sequence. Driven by deep learning, the prediction accuracy of the protein secondary structure has been greatly improved in recent years. To explore a new technique of PSSP, this study introduces the concept of an adversarial game into the prediction of the secondary structure, and a conditional generative adversarial network (GAN)-based prediction model is proposed. We introduce a new multiscale convolution module and an improved channel attention (ICA) module into the generator to generate the secondary structure, and then a discriminator is designed to conflict with the generator to learn the complicated features of proteins. Then, we propose a PSSP method based on the proposed multiscale convolution module and ICA module. The experimental results indicate that the conditional GAN-based protein secondary structure prediction (CGAN-PSSP) model is workable and worthy of further study because of the strong feature-learning ability of adversarial learning.

Mathematical Statistics Technology in the Educational Grading System of Preschool Students

Abstract The current education system for preschool students has relatively small differences and cannot fully reflect the true mastery of students. The article designs a cognitive system for preschool students’ educational scoring based on mathematical statistics. The article uses the attention vector and knowledge acceptance in children’s teaching as a multi-scale convolution level to analyze the effect of education. The study results found that the evaluation system can analyze the quality of children’s education while also considering children’s learning emotions.

Improved YOLOv4 for Pedestrian Detection and Counting in UAV Images.

UAV (unmanned aerial vehicle) captured images have small pedestrian targets and loss of key information after multiple down sampling, which are difficult to overcome by existing methods. We propose an improved YOLOv4 model for pedestrian detection and counting in UAV images, named YOLO-CC. We used the lightweight YOLOv4 for pedestrian detection, which replaces the backbone with CSPDarknet-34, and two feature layers are fused by FPN (Feature Pyramid Networks). We expanded the perception field using multiscale convolution based on the high-level feature map and generated the population density map by feature dimension reduction. By embedding the density map generation method into the network for end-to-end training, our model can effectively improve the accuracy of detection and counting and make feature extraction more focused on small targets. Our experiments demonstrate that YOLO-CC achieves 21.76 points AP50 higher than that of the original YOLOv4 on the VisDrone2021-counting data set while running faster than the original YOLOv4.

A text classification method based on a convolutional and bidirectional long short-term memory model

Text classification refers to labelling text with specified labels, and it is widely used in public opinion supervision, spam detection, and other fields. However, due to the complex semantics of natural language and the difficulty of extracting semantic features, users of traditional methods encounter difficulties when trying to achieve better classification results. In response to this problem, a text classification method based on the CBM (Convolutional and Bi-LSTM Model) model, which can extract shallow local semantic features and deep global semantic features, is proposed. First, the text is vectorised using the Glove model in the embedding layer. Then, the vector text is sent to the Multiscale Convolutional Neural Network (MCNN) and the Bidirectional Long Short-Term Memory network (Bi-LSTM) respectively. The Bi-LSTM layer is also designed in the present work with use of mixed attention to extract deeper semantic features. Finally, the MCNN features and Bi-LSTM features are fused and sent to the softmax layer for classification. Experimental results show that the model can significantly improve the accuracy of text classification.

Remaining Useful Life Prediction Model for Rolling Bearings Based on MFPE-MACNN.

Aiming to resolve the problem of redundant information concerning rolling bearing degradation characteristics and to tackle the difficulty faced by convolutional deep learning models in learning feature information in complex time series, a prediction model for remaining useful life based on multiscale fusion permutation entropy (MFPE) and a multiscale convolutional attention neural network (MACNN) is proposed. The original signal of the rolling bearing was extracted and decomposed by resonance sparse decomposition to obtain the high-resonance and low-resonance components. The multiscale permutation entropy of the low-resonance component was calculated. Moreover, the locally linear-embedding algorithm was used for dimensionality reduction to remove redundant information. The multiscale convolution module was constructed to learn the feature information at different time scales. The attention module was used to fuse the feature information and input it into the remaining useful life prediction module for evaluation. The appropriate network structure and parameter configuration were determined, and a multiscale convolutional attention neural network was designed to determine the remaining useful life prediction model. The results show that the method demonstrates effectiveness and superiority in degrading the feature information representation and improving the remaining useful life prediction accuracy compared with other models.

EEG Identity Authentication in Multi-Domain Features: A Multi-Scale 3D-CNN Approach.

Electroencephalogram (EEG) authentication has become a research hotspot in the field of information security due to its advantages of living, internal, and anti-stress. However, the performance of identity authentication system is limited by the inherent attributes of EEG, such as low SNR, low stability, and strong randomness. Researchers generally believe that the in-depth fusion of features can improve the performance of identity authentication and have explored among various feature domains. This experiment invited 70 subjects to participate in the EEG identity authentication task, and the experimental materials were visual stimuli of the self and non-self-names. This paper proposes an innovative EEG authentication framework, including efficient three-dimensional representation of EEG signals, multi-scale convolution structure, and the combination of multiple authentication strategies. In this work, individual EEG signals are converted into spatial–temporal–frequency domain three-dimensional forms to provide multi-angle mixed feature representation. Then, the individual identity features are extracted by the various convolution kernel of multi-scale vision, and the strategy of combining multiple convolution kernels is explored. The results show that the small-size and long-shape convolution kernel is suitable for ERP tasks, which can obtain better convergence and accuracy. The experimental results show that the classification performance of the proposed framework is excellent, and the multi-scale convolution method is effective to extract high-quality identity characteristics across feature domains. The results show that the branch number matches the EEG component number can obtain the excellent cost performance. In addition, this paper explores the network training performance for multi-scale module combination strategy and provides reference for deep network construction strategy of EEG signal processing.

High impedance fault detection method in distribution network based on improved Emanuel model and DenseNet

The paper proposes a high impedance fault (HIF) detection method in distribution network based on improved Emanuel and DenseNet. Firstly, the improved Emanuel model is introduced to accurately simulate HIF under different working conditions. Secondly, an improved DenseNet model is developed to implement HIF detection, the model uses multi-scale convolution kernels to extract different scale fault information, applies dual-channel Dense Block to transmit fault features more effectively, which can alleviate the vanishing gradient problem. ELU is employed as an activation function to expedite convergent speed of iterative operation. The comparative experiments verify the effectiveness of the proposed method.

Motor Imagery EEG Classification Based on Transfer Learning and Multi-Scale Convolution Network.

For the successful application of brain-computer interface (BCI) systems, accurate recognition of electroencephalography (EEG) signals is one of the core issues. To solve the differences in individual EEG signals and the problem of less EEG data in classification and recognition, an attention mechanism-based multi-scale convolution network was designed; the transfer learning data alignment algorithm was then introduced to explore the application of transfer learning for analyzing motor imagery EEG signals. The data set 2a of BCI Competition IV was used to verify the designed dual channel attention module migration alignment with convolution neural network (MS-AFM). Experimental results showed that the classification recognition rate improved with the addition of the alignment algorithm and adaptive adjustment in transfer learning; the average classification recognition rate of nine subjects was 86.03%.

Multi-scale convolution neural network with residual modules for determination of drugs in human hair using surface-enhanced Raman spectroscopy with a gold nanorod film self-assembled by inverted evaporation

Detection of illegal drug users is crucial in controlling drug-related crimes, reducing drug prevalence, and protecting human lives to ensure social stability. In this study, surface-enhanced Raman spectroscopy (SERS) and deep learning networks were employed to determine methamphetamine, ketamine, and morphine in human hair. Drugs were obtained from hair through alkaline hydrolysis and liquid–liquid extraction, and gold nanorods were employed to prepare the self-assembled film as the SERS substrate by inverted evaporation. The film showed good uniformity and excellent sensitivity, with a relative standard deviation of 15.6% and a detection limit of at least 10-10 M in the SERS detection of crystal violet. The spectra of methamphetamine, ketamine, and morphine at 0.05–1.0, 0.1–2.0, and 0.1–2.0 ng/mg were obtained, and the three drugs could be detected. Inception, a multi-scale feature extraction network, was combined with residual modules (Inception–ResNet) to develop the identification models of drugs, and the effect of spectral input form as a vector or matrix was explored. Inception–ResNet with input form of matrix outweighed other methods with 100.00%, 100.00%, and 99.23% accuracies in the training, validation, and prediction sets, respectively. In brief, SERS and Inception–ResNet with the spectra in matrix form provide an efficient and accurate determination of drugs in human hair, enabling the retrospective evaluation of drug use, and the method will be anticipated to detect excitant, poison, and toxic chemicals in human hair.

Multi-scale attention mechanism residual neural network for fault diagnosis of rolling bearings

Rolling bearing fault diagnosis is crucial to improve industrial safety and reliability. In recent years, intelligent fault diagnosis method represented by deep learning (DL) has been receiving increasing attention. In order to ameliorate the full training of the deep network, improve the model accuracy, and perfect the analysis of mechanical vibration signals with huge amount of information, a multi-scale attention mechanism residual network (MSA-ResNet) fault diagnosis method is proposed in this paper. First, an attention mechanism block is introduced to construct a new type of residual block combination. Second, a multi-scale structure is constructed by choosing an appropriate convolution kernel size. Finally, the overall framework of MSA-ResNet is constructed for efficient training and failure pattern recognition. The MSA-ResNet algorithm introduces an attention mechanism in each residual module of the residual network (ResNet), which improves the sensitivity to features. The features of different scales are obtained through the multi-scale convolution kernel, and the multi-scale feature extraction of complex nonlinear mechanical vibration signals is realized. The processing of original vibration signal rarely involves artificial interference, which is more conducive to industrial application of the proposed method. Diagnostic experiments are conducted on bearing datasets from the Case Western Reserve University (CWRU) and the Machinery Failure Prevention Technology (MFPT) to verify the effectiveness of the proposed method. The results illustrating the rolling bearing fault diagnosis method based on MSA-ResNet have advantages in multi-scale feature extraction, noise immunity, and fault classification accuracy.

Birdsong classification based on ensemble multi-scale convolutional neural network

With the intensification of ecosystem damage, birds have become the symbolic species of the ecosystem. Ornithology with interdisciplinary technical research plays a great significance for protecting birds and evaluating ecosystem quality. Deep learning shows great progress for birdsongs recognition. However, as the number of network layers increases in traditional CNN, semantic information gradually becomes richer and detailed information disappears. Secondly, the global information carried by the entire input may be lost in convolution, pooling, or other operations, and these problems will weaken the performance of classification. In order to solve such problems, based on the feature spectrogram from the wavelet transform for the birdsongs, this paper explored the multi-scale convolution neural network (MSCNN) and proposed an ensemble multi-scale convolution neural network (EMSCNN) classification framework. The experiments compared the MSCNN and EMSCNN models with other CNN models including LeNet, VGG16, ResNet101, MobileNetV2, EfficientNetB7, Darknet53 and SPP-net. The results showed that the MSCNN model achieved an accuracy of 89.61%, and EMSCNN achieved an accuracy of 91.49%. In the experiments on the recognition of 30 species of birds, our models effectively improved the classification effect with high stability and efficiency, indicating that the models have better generalization ability and are suitable for birdsongs species recognition. It provides methodological and technical scheme reference for bird classification research.

Multi-Scale Convolution-Capsule Network for Crop Insect Pest Recognition

Accurate crop insect pest identification in fields is useful to control pests and beneficial to agricultural yield and quality. However, it is a difficult and challenging problem due to the crop insect pests being small with various sizes, postures, shapes, and disorganized backgrounds. Multi-scale convolution-capsule network (MSCCN) is constructed for crop insect pest identification. It consists of a multi-scale convolution module, capsule network (CapsNet) module, and SoftMax classification module. Multi-scale convolution is used to extract the multi-scale discriminative features, CapsNet is employed to encode the hierarchical structure of the size-variant insect pests in the crop images, and Softmax is adopted for insect pest identification. MSCCN combines the advantages of convolutional neural network (CNN), CapsNet, and multi-scale CNN, and can learn multi-scale robust features from pest images of different shapes and sizes for pest recognition and identify various morphed pests. Experimental results on the crop pest image dataset show that this method has a good recognition rate of 91.4%.

Formula omitted]-net: Dual supervised medical image segmentation with multi-dimensional self-attention and diversely-connected multi-scale convolution

Although U-Net and its variants have achieved some great successes in medical image segmentation tasks, their segmentation performances for small objects are still unsatisfactory. Therefore, in this work, a new deep model, ω-Net, is proposed to achieve more accurate medical image segmentations. The advancements of ω-Net are mainly threefold: First, it incorporates an additional expansive path into U-Net to import an extra supervision signal and obtain a more effective and robust image segmentation by dual supervision. Then, a multi-dimensional self-attention mechanism is further developed to highlight salient features and suppress irrelevant ones consecutively in both spatial and channel dimensions. Finally, to reduce semantic disparity between the feature maps of the contracting and expansive paths, we further propose to integrate diversely-connected multi-scale convolution blocks into the skip connections, where several multi-scale convolutional operations are connected in both series and parallel. Extensive experimental results on three abdominal CT segmentation tasks show that (i) ω-Net greatly outperforms the state-of-the-art image segmentation methods in medical image segmentation tasks; (ii) the proposed three advancements are all effective and essential for ω-Net to achieve the superior performances; and (iii) the proposed multi-dimensional self-attention (resp., diversely-connected multi-scale convolution) is more effective than the state-of-the-art attention mechanisms (resp., multi-scale solutions) for medical image segmentations. The code will be released online after this paper is formally accepted.

A Multimodel Fusion Method for Cardiovascular Disease Detection Using ECG.

Objective. Electrocardiogram (ECG) is an important diagnostic tool that has been the subject of much research in recent years. Owing to a lack of well-labeled ECG record databases, most of this work has focused on heartbeat arrhythmia detection based on ECG signal quality. Approach. A record quality filter was designed to judge ECG signal quality, and a random forest method, a multilayer perceptron, and a residual neural network (RESNET)-based convolutional neural network were implemented to provide baselines for ECG record classification according to three different principles. A new multimodel method was constructed by fusing the random forest and RESNET approaches. Main Results. Owing to its ability to combine discriminative human-crafted features with RESNET deep features, the proposed new method showed over 88% classification accuracy and yielded the best results in comparison with alternative methods. Significance. A new multimodel fusion method was presented for abnormal cardiovascular detection based on ECG data. The experimental results show that separable convolution and multiscale convolution are vital for ECG record classification and are effective for use with one-dimensional ECG sequences.