Feature Fusion Layer Research Articles

Small Aircraft Detection Based on Feature Enhancement and Context Information

In the field of civil aviation safety and air security, research on identification and tracking of small aircraft is very important. Aiming at the problem of low recognition rate of long-distance small air targets, this study proposes a method of GST-YOLOv3 combined with DeepSORT algorithm to detect and track small aircraft. The proposed GST-YOLOv3 uses a ghost module to extract effective features and simplify the network. In feature extraction layer, the spatial pyramid pooling layer is added to fuse local features and global features to enhance the features of small targets. The Transformer module is embedded in the feature fusion layer to focus on global context information. In the multiscale detection layer, a new large-scale feature map detection layer is formed by fusing shallow features and deep features to obtain a smaller receptive field. To avoid frame loss of video detection, the DeepSORT algorithm is used for real-time tracking. In this study, the self-made data set Aplanes and the public data set VisDroneDET are used to verify the effect of the proposed GST-YOLOv3. The computation of the GST-YOLOv3 was reduced by 45.8% compared with YOLOv3; the missed detection rate of airplanes decreased by 2.5%; and the detection precision, recall rate, F1 value, and average precision/0.5 value of small aircraft reached 98.4, 98.7, 98.5, and 98.6%, respectively, which are better than those of similar literature.

BgNet: Classification of benign and malignant tumors with MRI multi-plane attention learning.

To propose a deep learning-based classification framework, which can carry out patient-level benign and malignant tumors classification according to the patient's multi-plane images and clinical information. A total of 430 cases of spinal tumor, including axial and sagittal plane images by MRI, of which 297 cases for training (14072 images), and 133 cases for testing (6161 images) were included. Based on the bipartite graph and attention learning, this study proposed a multi-plane attention learning framework, BgNet, for benign and malignant tumor diagnosis. In a bipartite graph structure, the tumor area in each plane is used as the vertex of the graph, and the matching between different planes is used as the edge of the graph. The tumor areas from different plane images are spliced at the input layer. And based on the convolutional neural network ResNet and visual attention learning model Swin-Transformer, this study proposed a feature fusion model named ResNetST for combining both global and local information to extract the correlation features of multiple planes. The proposed BgNet consists of five modules including a multi-plane fusion module based on the bipartite graph, input layer fusion module, feature layer fusion module, decision layer fusion module, and output module. These modules are respectively used for multi-level fusion of patient multi-plane image data to realize the comprehensive diagnosis of benign and malignant tumors at the patient level. The accuracy (ACC: 79.7%) of the proposed BgNet with multi-plane was higher than that with a single plane, and higher than or equal to the four doctors' ACC (D1: 70.7%, p=0.219; D2: 54.1%, p<0.005; D3: 79.7%, p=0.006; D4: 72.9%, p=0.178). Moreover, the diagnostic accuracy and speed of doctors can be further improved with the aid of BgNet, the ACC of D1, D2, D3, and D4 improved by 4.5%, 21.8%, 0.8%, and 3.8%, respectively. The proposed deep learning framework BgNet can classify benign and malignant tumors effectively, and can help doctors improve their diagnostic efficiency and accuracy. The code is available at https://github.com/research-med/BgNet.

RD-YOLO: An Effective and Efficient Object Detector for Roadside Perception System.

In recent years, intelligent driving technology based on vehicle-road cooperation has gradually become a research hotspot in the field of intelligent transportation. There are many studies regarding vehicle perception, but fewer studies regarding roadside perception. As sensors are installed at different heights, the roadside object scale varies violently, which burdens the optimization of networks. Moreover, there is a large amount of overlapping and occlusion in complex road environments, which leads to a great challenge of object distinction. To solve the two problems raised above, we propose RD-YOLO. Based on YOLOv5s, we reconstructed the feature fusion layer to increase effective feature extraction and improve the detection capability of small targets. Then, we replaced the original pyramid network with a generalized feature pyramid network (GFPN) to improve the adaptability of the network to different scale features. We also integrated a coordinate attention (CA) mechanism to find attention regions in scenarios with dense objects. Finally, we replaced the original Loss with Focal-EIOU Loss to improve the speed of the bounding box regression and the positioning accuracy of the anchor box. Compared to the YOLOv5s, the RD-YOLO improves the mean average precision (mAP) by 5.5% on the Rope3D dataset and 2.9% on the UA-DETRAC dataset. Meanwhile, by modifying the feature fusion layer, the weight of RD-YOLO is decreased by 55.9% while the detection speed is almost unchanged. Nevertheless, the proposed algorithm is capable of real-time detection at faster than 71.9 frames/s (FPS) and achieves higher accuracy than the previous approaches with a similar FPS.

JGRCAN: A Visual Question Answering Co-Attention Network via Joint Grid-Region Features

In recent years, region features extracted from target detection networks have played an important role in visual question answering. The region features only extract the areas that are related to the target, but they lose a lot of nontarget context information and fine-grained details. On the contrary, the grid feature does not lose the details of nontargets but is not conducive to the recognition of the counting question of multiple small targets in the image. To solve this problem, this paper proposes a visual question answering network via joint grid-region features (JGRCAN), which consists of a feature extraction layer, co-attention layer, and fusion layer. The feature extraction layer includes extracting grid features and region features from the image and text features from the question and extracting multivisual feature representation and question feature representation through the co-attention layer to output attention weight and attention feature representation, respectively. The proposed approach effectively integrates grid features and region features, realizes the complementary advantages of region features and grid features, and is able to accurately focus on areas of the image that are relevant to the answer to the question. The results show that the overall classification accuracy of the algorithm on the test-dev and test-std subsets of VQA-v2 is 70.87% and 71.18%, respectively. Compared with baseline models, our proposed JGRCAN has good performance.

Improved Cotton Seed Breakage Detection Based on YOLOv5s

Convolutional neural networks have been widely used in nondestructive testing of agricultural products. Aiming at the problems of missing detection, false detection, and slow detection, a lightweight improved cottonseed damage detection method based on YOLOv5s is proposed. Firstly, the focus element of the YOLOv5s backbone network is replaced by Denseblock, simplifying the number of modules in the backbone network layer, reducing redundant information, and improving the feature extraction ability of the network. Secondly, the collaborative attention (CA) mechanism module is added after the SPP pooling layer, and a large target detection layer is reduced to guide the network to pay more attention to the location, channel, and dimension information of small targets. Thirdly, Ghostconv is used instead of the conventional convolution layer in the neck feature fusion layer to reduce the amount of floating-point calculation and speed up the reasoning speed of the model. The CIOU loss function is selected as the border regression loss function to improve the recall rate of the model. Lastly, the model was verified using an ablation experiment and compared with the YOLOv4, Yolov5s, and SSD-VGG16 network models. The accuracy, recall rate, and map value of the improved network model were 92.4%, 91.7%, and 98.1%, respectively, and the average recognition time of each image was 97 fps. The results show that the improved network can effectively solve the problem of missing detection, reduce false detection, and have better recognition performance. This method can provide technical support for real-time and accurate detection of damaged cottonseed in a cottonseed screening device.

Abnormal behavior recognition based on feature fusion C3D network

The real-time detection and recognition ability of human action recognition in a video surveillance system is a key problem in an intelligent surveillance system. Because the behavior recognition for video surveillance systems is affected by the complexity of the scene, the classification performance of the behavior recognition models is not satisfactory. To increase the processing efficiency of the network and solve the problem of low classification accuracy of human action recognition, we designed a deep learning model based on three-dimensional (3D) convolutional network multiscale feature fusion to reduce the impact of constant appearance changes, background clutter, and pedestrian occlusion. The model alternately uses 3D convolution and 3D pooling operations to extract temporal and spatial features between consecutive frames after data preprocessing, and then uses a feature pyramid structure to select three sets of feature layers with different scales. The model performs deconvolution operations in a bottom to up order and fuses with the features of the previous layer, then downsampling and high-level feature layer fusion are performed sequentially from top to bottom. Using the newly generated the highest-level feature layer to realize abnormal behavior recognition. The C3D network algorithm based on feature fusion proposed in this paper is compared with the three most advanced methods of C3D, R3D, and R ( 2 + 1 ) D on the pedestrian abnormal action recognition (PAAR) dataset and the same parameters, and the accuracy is significantly improved.

Power Equipment Fault Diagnosis Method Based on Energy Spectrogram and Deep Learning.

With the development of industrial manufacturing intelligence, the role of rotating machinery in industrial production and life is more and more important. Aiming at the problems of the complex and changeable working environment of rolling bearings and limited computing ability, fault feature information cannot be effectively extracted, and the current deep learning model is difficult to be compatible with lightweight and high efficiency. Therefore, this paper proposes a fault detection method for power equipment based on an energy spectrum diagram and deep learning. Firstly, a novel two-dimensional time-frequency feature representation method and energy spectrum feature map based on wavelet packet transform is proposed, and an energy spectrum feature map dataset is made for subsequent diagnosis. This method can realize multi-resolution analysis, fully extract the feature information contained in the fault signal, and accelerate the convergence of the subsequent diagnosis model. Secondly, a lightweight residual dense convolutional neural network model (LR-DenseNet) is proposed. This model combines the advantages of residual learning and a dense connection, and can not only extract deep features more easily, but can also effectively use shallow features. Then, based on the lightweight residual dense convolutional neural network model, an LR-DenseSENet model is proposed. By introducing the transfer learning strategy and adding the channel domain, an attention mechanism is added to the channel feature fusion layer, with the accuracy of detection up to 99.4%, and the amount of parameter calculation greatly reduced to one-fifth of that of VGG. Finally, through an experimental analysis, it is verified that the fault detection model designed in this paper based on the combination of an energy spectrum feature map and LR-DenseSENet achieves a satisfactory detection effect.

Research on Fingerprint Security Based on Improved Yolo Algorithm

The quantitative identification technology based on the statistical law of fingerprint features has become a new research difficulty and focus, and the automatic detection and classification of fingerprint features are the basis for realizing automatic fingerprint feature statistics. In this paper, a YOLO-based fingerprint feature detection method was proposed. First, a fingerprint feature dataset was established, which contained a total of 4,000 annotated fingerprint images; then, according to the characteristics of small size and dense distribution of fingerprint feature points, the YOLO network structure was improved, the original large target feature detection layer by 32-fold downsampling was deleted, and a new small feature fusion layer was added; the FPN, PAN, and SPP structures were used to achieve local and global feature extraction through multiple-scale fusion methods; finally, the SE channel attention mechanism module was added to effectively enhance the model robustness and detection ability of dense small objects. The experimental results show that compared with the improved FP-YOLO model of the original model, when the detection speed is basically unchanged, the mAP0.5 value is increased from 93.0% to 97.4%, and the weight is reduced by 3/4.

Kiwifruit Detection Method in Orchard via an Improved Light-Weight YOLOv4

In order to detect kiwifruit quickly and accurately in orchard environments for the picking robot, this paper proposed a detection method based on a lightweight YOLOv4-GhostNet network. The implementations of the method are as follows: The original CSP-Darknet53 backbone network model was replaced by GhostNet, a feature layer facilitating small object detection was introduced in the feature fusion layer, and part of the ordinary convolution was replaced by a combination of 1 × 1 convolution and depth-separable convolution to reduce the computational pressure caused by the fused feature layer. The parameters of the new network are reduced, and the generalization ability of the model is improved by loading pre-training weights and freezing some layers. The trained model was tested, and the results showed that the detection performances were better than that of the original YOLOv4 network. The F1 value, map, and precision were improved on the test set, which were 92%, 93.07%, and 90.62%, respectively. The size of weight parameters was reduced to 1/6 of the original YOLOv4 network, and the detection speed reached 53 FPS. Therefore, the method proposed in this study shows the features of fast recognition, lightweight parameters, and high recognition accuracy, which can provide technical support for vision systems of kiwifruit picking robots.

Improved Algorithm for Insulator and Its Defect Detection Based on YOLOX.

Aerial insulator defect images have some features. For instance, the complex background and small target of defects would make it difficult to detect insulator defects quickly and accurately. To solve the problem of low accuracy of insulator defect detection, this paper concerns the shortcomings of IoU and the sensitivity of small targets to the model regression accuracy. An improved SIoU loss function was proposed based on the regular influence of regression direction on the accuracy. This loss function can accelerate the convergence of the model and make it achieve better results in regressions. For complex backgrounds, ECA (Efficient Channel Attention Module) is embedded between the backbone and the feature fusion layer of the model to reduce the influence of redundant features on the detection accuracy and make progress in the aspect. As a result, these experiments show that the improved model achieved 97.18% mAP which is 2.74% higher than before, and the detection speed could reach 71 fps. To some extent, it can detect insulator and its defects accurately and in real-time.

Convolutional Neural Network Approach Based on Multimodal Biometric System with Fusion of Face and Finger Vein Features.

In today’s information age, how to accurately identify a person’s identity and protect information security has become a hot topic of people from all walks of life. At present, a more convenient and secure solution to identity identification is undoubtedly biometric identification, but a single biometric identification cannot support increasingly complex and diversified authentication scenarios. Using multimodal biometric technology can improve the accuracy and safety of identification. This paper proposes a biometric method based on finger vein and face bimodal feature layer fusion, which uses a convolutional neural network (CNN), and the fusion occurs in the feature layer. The self-attention mechanism is used to obtain the weights of the two biometrics, and combined with the RESNET residual structure, the self-attention weight feature is cascaded with the bimodal fusion feature channel Concat. To prove the high efficiency of bimodal feature layer fusion, AlexNet and VGG-19 network models were selected in the experimental part for extracting finger vein and face image features as inputs to the feature fusion module. The extensive experiments show that the recognition accuracy of both models exceeds 98.4%, demonstrating the high efficiency of the bimodal feature fusion.

Feature layer fusion of linear features and empirical mode decomposition of human EMG signal

To explore the influence of the fusion of different features on recognition, this paper took the electromyogram (EMG) signals of rectus femoris under different motions (walk, step, ramp, squat, and sitting) as signals, linear features (time-domain features (variance (VAR) and root mean square (RMS)), frequency-domain features (mean frequency (MF) and mean power frequency (MPF)), and nonlinear features (EMD) of the signals were extracted. Two feature fusion algorithms, the series splicing method and complex vector method, were designed, which were verified by a double hidden layer error back propagation (BP) neural network. Results show that with the increase of the types and complexity of feature fusions, the recognition rate of the EMG signal to actions is gradually improved. When the EMG signal is used in the series splicing method, the recognition rate of time-domain ​+ ​frequency-domain ​+ ​empirical mode decomposition (TD ​+ ​FD ​+ ​EMD) splicing is the highest, and the average recognition rate is 92.32%. And this value is raised to 96.1% by using the complex vector method, and the variance of the BP system is also reduced.

Research on Improved YOLOv4-based Pedestrian Detection Model

Pedestrian safety affects many fields such as driverless vehicles. In order to improve the precision of the pedestrian detection method and distinguish people-like objects from pedestrians, an improved YOLOv4 pedestrian detection method is proposed. Firstly, RepVGG Block is introduced into the feature extraction layer and feature fusion layer of YOLOv4 to improve the feature extraction ability of the network and reduce the loss of feature information. Then the SENet attention mechanism is introduced to make the algorithm focus more on the useful information. Finally, the SIoU loss function is introduced to the regression of the pedestrian target frame, which improves the convergence speed and reduces the blindness of the target frame. Experimental results show that, compared with the original YOLOv4 algorithm, this improved algorithm has higher detection precision, and it can also distinguish people from people-like objects on the published Pedestrian Detection Data set, with a detection precision of 83.7%.

Multi-sensor signals with parallel attention convolutional neural network for bearing fault diagnosis

Rolling bearing fault signals are non-smooth, non-linear, and susceptible to background noise interference. A feature layer fusion model combining multi-sensor signals and parallel attention convolutional neural networks is proposed and applied to the fault diagnosis work of rolling bearings. First, a multi-channel parallel convolutional neural network model is constructed according to the number of sensors, and the multi-sensor signals are fed to each parallel channel separately. Second, due to the different strengths of shock features within each channel and signal, the attention mechanism is introduced into each parallel channel, the fault features with strong shock characteristics are extracted, and the feature extraction capability for different sensor signals is improved. Finally, the extracted feature information is fused in the concatenate layer, the fused features are input to the fully connected layer, and the diagnosis results are output through the Softmax layer. The experimental results show that the model can effectively fuse multi-sensor signal features, and its recognition accuracy is greatly improved over that of the single sensor, providing a feasible method for bearing fault diagnosis.

Seismic impedance inversion using a multi‐input neural network with a two‐step training strategy

ABSTRACTDeep learning has shown excellent performance in simulating complex nonlinear mappings from the seismic data to elastic parameters. However, seismic acoustic impedance estimated from a direct mapping from seismic waveform data to P‐wave impedance (single‐input network) is hampered by the limited frequency bands. In this paper, we propose to incorporate the low‐frequency impedance model to constrain the inversion (multi‐input network). We add a feature fusion layer to force the lateral smoothness. Besides, usually, a given seismic survey is likely to contain only a few well logs, which is insufficient for conventional deep‐learning ‐based methods to learn the complex mapping from seismic data to elastic parameters. The problem is compounded by the fact that a network trained with synthetic data (compensated for the lack of logs) cannot be directly used for field data. Therefore, we propose to use transfer learning to mitigate this issue. The multi‐input neural network is trained using synthetics and real data in two stages. We carry out experiments to demonstrate that the two‐step training multi‐input network approach has high accuracy in the time direction, excellent continuity in the lateral direction and favourable robustness. Synthetic and field data examples demonstrate that the proposed network can accurately predict impedance even with limited logging data, which provides a reference for oil and gas exploration in the actual production process.

Swin-YOLOv5: Research and Application of Fire and Smoke Detection Algorithm Based on YOLOv5.

Accurate monitoring of fire and smoke plays an irreplaceable role in preventing fires and safeguarding the safety of citizens' lives and property. The network structure of YOLOv5 is simple, but using convolution to extract features will lead to some problems such as limited receptive field, poor feature extraction ability, and insufficient feature integration. In view of the current defects of YOLOv5 target detection algorithm, a new algorithm model named Swin-YOLOv5 was proposed in this work. Swin transformation mechanism was introduced into YOLOv5 network, which enhanced the receptive field and feature extraction ability of the model without changing the depth of the model. In order to enrich the feature map splicing method of weighted Concat and enhance the feature fusion ability of model pairs, the feature splicing method of three output heads of feature fusion layer network was improved. The feature fusion module was further modified, and the weighted feature splicing method was introduced to improve the network feature fusion ability. Experiments showed that, compared with the original algorithm, the rising rate of mAP@0.5 (mean average precision, IoU=0.5) of the improved algorithm was 0.7%, the mAP@0.5:0.95 was increased by 4.5%, and the target detection speed with high accuracy was accelerated by 1.8 FPS (frames per second) under the same experimental dataset. The improved algorithm could more accurately detect the targets that were not detected or detected inaccurately by the original algorithm, which embodied the adaptability of real scene detection and had practical significance. This work provided an opportunity for the application of fire-smoke detection in forest and indoor scenes and also developed a feasible idea for feature extraction and fusion of YOLOv5.

Transmission Line Object Detection Method Based on Label Adaptive Allocation

Inspection of the integrality of components and connecting parts is an important task to maintain safe and stable operation of transmission lines. In view of the fact that the scale difference of the auxiliary component in a connecting part is large and the background environment of the object is complex, a one-stage object detection method based on the enhanced real feature information and the label adaptive allocation is proposed in this study. Based on the anchor-free detection algorithm FCOS, this method is optimized by expanding the real feature information of the adjacent feature layer fusion and the semantic information of the deep feature layer, as well as adaptively assigning the label through the idea of pixel-by-pixel detection. In addition, the grading ring image is sliced in original data to improve the proportion of bolts in the dataset, which can clear the appearance features of small objects and reduce the difficulty of detection. Experimental results show that this method can eliminate the background interference in the GT (ground truth) as much as possible in object detection process, and improve the detection accuracy for objects with a narrow shape and small size. The evaluation index AP (average precision) increased by 4.1%. Further improvement of detection accuracy lays a foundation for the realization of efficient real-time patrol inspection.

Multilevel feature fusion and edge optimization network for self-supervised monocular depth estimation

Monocular depth estimation is an essential step in scene geometry understanding. However, the depth maps predicted by the existing methods have the problems of loss of small target details and blurred edges. To this end, we propose a monocular depth estimation method based on multilevel feature fusion and edge optimization to obtain depth maps with rich depth details and precise edges. First, we improved the encoder part to make it better adapted for the depth estimation task. In addition, we fill the U-shaped network with a dense feature fusion layer (Dense-FL) to capture global context information and then combine the proposed structure attention module to further enhance the obtained feature information, so the depth map contains more detailed information. Finally, we design an edge optimization module to incorporate edge features into the training process and combine the proposed reweighted loss and image edge detail loss to constrain the network, so as to further improve the learning ability of the model to the edge of the object. The experimental results on the KITTI dataset show that the depth prediction results obtained by our method perform better at small targets and the edges of objects in the scene, and the structure of the objects is more complete. Generalization experiments on the Cityscapes and Sceneflow datasets also further confirmed the effectiveness and superiority of our proposed method.

A Multifeature Fusion Short-Term Traffic Flow Prediction Model Based on Deep Learnings

Short-term traffic flow prediction is an important component of intelligent transportation systems, which can support traffic trip planning and traffic management. Although existing predicting methods have been applied in the field of traffic flow prediction, they cannot capture the complex multifeatures of traffic flows resulting in unsatisfactory short-term traffic flow prediction results. In this paper, a multifeature fusion model based on deep learning methods is proposed, which consists of three modules, namely, a CNN-Bidirectional GRU module with an attention mechanism (CNN-BiGRU-attention) and two Bidirectional GRU modules with an attention mechanism (BiGRU-attention). The CNN-BiGRU-attention module is used to extract local trend features and long-term dependent features of the traffic flow, and the two BiGRU-attention modules are used to extract daily and weekly periodic features of the traffic flow. Moreover, a feature fusion layer in the model is used to fuse the features extracted by each module. And then, the number of neurons in the model, the loss function, and other parameters such as the optimization algorithm are discussed and set up through simulation experiments. Finally, the multifeature fusion model is trained and tested based on the training and test sets from the data collected from the field. And the results indicate that the proposed model can better achieve traffic flow prediction and has good robustness. Furthermore, the multifeature fusion model is compared and analyzed against the baseline models with the same dataset, and the experimental results show that the multifeature fusion model has superior predictive performance compared to the baseline models.

Fusion of THz-TDS and NIRS Based Detection of Moisture Content for Cattle Feed

As an essential index to evaluate feed quality, feed moisture content which is too high or too low will impose an adverse impact on feed nutritional value. Therefore, the quantitative analysis of feed moisture content is significant. In this paper, the detection of feed moisture content based on terahertz (THz) and near-infrared (NIR) spectroscopy and data fusion technology of THz and NIR (THz-NIR) was investigated. First, feed samples with different water content (29.46%–49.46%) were prepared, and THz (50–3000 μm) and NIR (900–1700 nm) spectral data of samples was collected and preprocessed, and the feed samples were divided into correction set and verification set by 2:1. Second, the spectral data was fused through the head-to-tail splicing, and the feed moisture content prediction model was established combined with partial least squares regression (PLSR). Third, competitive adaptive reweighting sampling (CARS) was applied to extract spectral characteristic variables for feature layer fusion, and the feed moisture content prediction model in feature level was constructed combined with PLSR. Finally, the evaluation parameters validation set correlation coefficient (Rp), the root mean square error of prediction (RMSEP), and the residual predictive deviation (RPD) were employed to evaluate the prediction effect of the model. The results indicated that THz, NIR spectra, and data fusion technology could quickly and effectively predict feed moisture content. Among them, the characteristic layer spectral data fusion model achieved the optimal prediction effect while Rp, RMSEP, and RPD reached 0.9933, 0.0069, and 8.7386 respectively. In conclusion, compared with the prediction model established by single THz and NIR spectrum, THz-NIR spectrum data fusion could more accurately predict feed moisture content and provide certain theoretical and technical support for inspirations and methods for quantitative analysis of feed moisture content of livestock and poultry.