Feature Map Fusion Research Articles

Flood Detection in Polarimetric SAR Data Using Deformable Convolutional Vision Model

Floods represent a significant natural hazard with the potential to inflict substantial damage on human society. The swift and precise delineation of flood extents is of paramount importance for effectively supporting flood response and disaster relief efforts. In comparison to optical sensors, Synthetic Aperture Radar (SAR) sensor data acquisition exhibits superior capabilities, finding extensive application in flood detection research. Nonetheless, current methodologies exhibit limited accuracy in flood boundary detection, leading to elevated instances of both false positives and false negatives, particularly in the detection of smaller-scale features. In this study, we proposed an advanced flood detection method called FWSARNet, which leveraged a deformable convolutional visual model with Sentinel-1 SAR images as its primary data source. This model centered around deformable convolutions as its fundamental operation and took inspiration from the structural merits of the Vision Transformer. Through the introduction of a modest number of supplementary parameters, it significantly extended the effective receptive field, enabling the comprehensive capture of intricate local details and spatial fluctuations within flood boundaries. Moreover, our model employed a multi-level feature map fusion strategy that amalgamated feature information from diverse hierarchical levels. This enhancement substantially augmented the model’s capability to encompass various scales and boost its discriminative power. To validate the effectiveness of the proposed model, experiments were conducted using the ETCI2021 dataset. The results demonstrated that the Intersection over Union (IoU) and mean Intersection over Union (mIoU) metrics for flood detection achieved impressive values of 80.10% and 88.47%, respectively. These results surpassed the performance of state-of-the-art (SOTA) models. Notably, in comparison to the best results documented on the official ETCI2021 dataset competition website, our proposed model in this paper exhibited a remarkable 3.29% improvement in flood prediction IoU. The experimental outcomes underscore the capability of the FWSARNet method outlined in this paper for flood detection using Synthetic Aperture Radar (SAR) data. This method notably enhances the accuracy of flood detection, providing essential technical and data support for real-world flood monitoring, prevention, and response efforts.

Robust RGB-T Tracking via Graph Attention-Based Bilinear Pooling.

RGB-T tracker possesses strong capability of fusing two different yet complementary target observations, thus providing a promising solution to fulfill all-weather tracking in intelligent transportation systems. Existing convolutional neural network (CNN)-based RGB-T tracking methods often consider the multisource-oriented deep feature fusion from global viewpoint, but fail to yield satisfactory performance when the target pair only contains partially useful information. To solve this problem, we propose a four-stream oriented Siamese network (FS-Siamese) for RGB-T tracking. The key innovation of our network structure lies in that we formulate multidomain multilayer feature map fusion as a multiple graph learning problem, based on which we develop a graph attention-based bilinear pooling module to explore the partial feature interaction between the RGB and the thermal targets. This can effectively avoid uninformed image blocks disturbing feature embedding fusion. To enhance the efficiency of the proposed Siamese network structure, we propose to adopt meta-learning to incorporate category information in the updating of bilinear pooling results, which can online enforce the exemplar and current target appearance obtaining similar sematic representation. Extensive experiments on grayscale-thermal object tracking (GTOT) and RGBT234 datasets demonstrate that the proposed method outperforms the state-of-the-art methods for the task of RGB-T tracking.

MHT: A multiscale hourglass-transformer for remaining useful life prediction of aircraft engine

Remaining useful life (RUL) prediction of aircraft engines is significant in the health monitoring, operation, and maintenance of aircraft. Capturing more comprehensive device degradation trends at different time scales and extracting long-term dependencies effectively among elements in long time series are two challenges in the field of aircraft engine RUL estimation. To address the aforementioned challenges, this paper proposes a novel multiscale Hourglass-Transformer (MHT) aircraft engine RUL prognostics. Specifically, an hourglass-shaped multiscale feature extractor (HME) is designed based on one-dimensional convolutional neural network, which can scale the time sequence into multi-time scales for feature fusion. Then, a transformer network is employed to further extract features from the fused feature map and output the RUL. To enhance inter-scale data attention, a pyramid self-attention mechanism is employed in both the encoder and decoder. Finally, the superiority and effectiveness of this approach are verified on the Commercial Modular Aero-Propulsion System Simulation (C-MAPSS) dataset. Furthermore, the robustness and generalization capability of this method are further validated on New Commercial Modular Aero-Propulsion System Simulation (N-CMAPSS) dataset.

Distilling a Powerful Student Model via Online Knowledge Distillation.

Existing online knowledge distillation approaches either adopt the student with the best performance or construct an ensemble model for better holistic performance. However, the former strategy ignores other students' information, while the latter increases the computational complexity during deployment. In this article, we propose a novel method for online knowledge distillation, termed feature fusion and self-distillation (FFSD), which comprises two key components: FFSD, toward solving the above problems in a unified framework. Different from previous works, where all students are treated equally, the proposed FFSD splits them into a leader student set and a common student set. Then, the feature fusion module converts the concatenation of feature maps from all common students into a fused feature map. The fused representation is used to assist the learning of the leader student. To enable the leader student to absorb more diverse information, we design an enhancement strategy to increase the diversity among students. Besides, a self-distillation module is adopted to convert the feature map of deeper layers into a shallower one. Then, the shallower layers are encouraged to mimic the transformed feature maps of the deeper layers, which helps the students to generalize better. After training, we simply adopt the leader student, which achieves superior performance, over the common students, without increasing the storage or inference cost. Extensive experiments on CIFAR-100 and ImageNet demonstrate the superiority of our FFSD over existing works. The code is available at https://github.com/SJLeo/FFSD.

LF-YOLOv4: a lightweight detection model for enhancing the fusion of image features of surface defects in lithium batteries

In response to the low accuracy of traditional methods for detecting surface defects in lithium batteries, as well as the problems of large model size and high computational complexity in current detection models, this article proposes a new lightweight LF-YOLOv4 model that enhances image feature fusion. Firstly, replace the CSPDarknet53 backbone network in YOLOv4 with a lightweight MobileNetv2 network, thereby greatly reducing the computational parameters of the network while ensuring the ability to extract features. Secondly, in order to further reduce the number of model parameters and computational complexity, and minimize potential accuracy loss as much as possible, an improved depthwise separable convolution (DSC-SE-HsId) in this article was studied, which replaced some ordinary convolutions in the Neck and Head networks. Finally, in order to compensate for the partial accuracy loss caused by lightweight operations, and also to fuse feature maps of different scales to obtain more complete feature information, a new lightweight adaptive spatial feature fusion module (LSE-ASFF) in this article was studied and embedded behind the existing path aggregation network (PANet). In order to verify the performance and widespread applicability of the improved model, we conducted tests using the self-built lithium battery surface defect dataset, and the steel surface defect dataset provided by Northeastern University. Moreover, in order to prevent the network training from over fitting due to too few defect samples in the self-built dataset, we also used image enhancement to expand the dataset. And the experiment results show that the improved model proposed in this article achieves the highest TOPSIS score in both experimental datasets. Among them, compared with YOLOv4 on the self-built dataset, our improved model not only increases mAP50 by 2.97%, reaching 97.83%, but also has model parameters of only 18.16% of the original model, floating point of operations of only 13.87% of the original model, model size of only 21.02% of the original model, and model training time shortened by 30.67% compared to the original model. At last, the effectiveness and superiority of the improved model in this article are demonstrated through example analysis and comparison.

Feature fusion strategy and improved GhostNet for accurate recognition of fish feeding behavior

In aquaculture, accurate detection of fish feeding intensity is a critical step for establishing an on-demand feeding system. In this paper, a novel fish feeding intensity detection method based on the fusion of multiple features (Mel spectrogram, STFT, and CQT feature map) and the LC-GhostNet lightweight network was proposed as an experimental object with Oplegnathus. First, a dataset of feeding sounds of Oplegnathus punctatus was built, in which there are four types: “strong”, “medium”, “weak” and “none”. Next, the Mel, STFT, and CQT feature maps of the feeding sound were extracted using the Librosa library, and then these feature maps were fused by feature image stitching. Lastly, the fused audio feature maps were fed to LC-GhostNet for further feature extraction and classification. Experimental results indicated that the accuracy of the fused feature maps was 97.941% when used as input, which was 4.053%, 7.207%, and 3.003% higher than that of the single feature Mel, STFT, and CQT, respectively. The method proposed in this paper is more lightweight and effective than the mainstream methods, and the accuracy has been improved. Based on this automatic and non-destructive method of obtaining fish feeding information, feeding decisions can be effectively optimized.

LHFNet: A Fast and Accurate Object Detector Based on CenterNet

This paper proposes LHFNet, an improved object detection method based on CenterNet, which achieves a better speed-accuracy trade-off. The main contributions of this paper are as follows: We design a lightweight Hourglass network to reduce the number of parameters and computations of CenterNet, thus improving the detection speed. We introduce an intermediate feature map fusion module to enhance the feature extraction capability of Hourglass, which compensates for the possible performance degradation caused by the network simplification. We apply an attention module to the final heat map to improve the image feature representation. We design and optimize a series of loss functions to train LHFNet effectively. Through the research in this paper, we demonstrate that the feature map fusion and the attention module greatly improve the performance of LHFNet. We evaluate our method on the COCO dataset and achieve a high accuracy at a fast speed (Tesla V100), which outperforms other object detection methods in terms of both accuracy and efficiency.

YOLACTFusion: An instance segmentation method for RGB-NIR multimodal image fusion based on an attention mechanism

The tomato plant’s main-stem is a feasible lead for robotic searching the grows discretely-growing targets of harvesting, pruning or pollinating. Owing to the highlighted reflection characteristics of the main-stem in the near-infrared (NIR) waveband, this study proposes a multimodal hierarchical fusion method (YOLACTFusion) based on the attention mechanism, to achieve an instance segmentation of the main-stem from similar-colored differentiation (i.e., green leaf and green fruit) in robotic vision systems. The model inputs RGB images and 900–1100 nm NIR images into two ResNet50 backbone networks and uses a parallel attention mechanism to fuse feature maps of various scales together into the head network, to improve the segmentation performance of the main-stem of RGB images. The loss function for the multimodal image weights the original loss on the RGB image and the position offset loss and classification loss on the NIR image. Furthermore, the local depthwise separable convolution is used for the backbone network, and Conv-BN layers are merged to reduce the computational complexity. The results show that the precision and recall of YOLACTFusion of the main-stem detection, respectively reached 93.90 % and 62.60 %; and the precision and recall of instance segmentation reached 95.12 % and 63.41 %, respectively. Compared to YOLACT, the mean average precision (mAP) of YOLACTFusion is increased from 39.20 % to 46.29 %, the model size is reduced from 199.03 MB to 165.52 MB, while the image processing efficiency remains similar. The overall results show that the multimodal instance segmentation method proposed in this study significantly improves the detection and segmentation of tomato main-stems under a similar-colored background, which would be a potential method for improving agricultural robot’s visual perception.

Quad-Pol SAR Data Reconstruction from Dual-Pol SAR Mode Based on a Multiscale Feature Aggregation Network

Polarimetric synthetic aperture radar (PolSAR) is widely used in remote sensing applications due to its ability to obtain full-polarization information. Compared to the quad-pol SAR, the dual-pol SAR mode has a wider observation swath and is more common in most SAR systems. The goal of reconstructing quad-pol SAR data from the dual-pol SAR mode is to learn the contextual information of dual-pol SAR images and the relationships among polarimetric channels. This work is dedicated to addressing this issue, and a multiscale feature aggregation network has been established to achieve the reconstruction task. Firstly, multiscale spatial and polarimetric features are extracted from the dual-pol SAR images using the pretrained VGG16 network. Then, a group-attention module (GAM) is designed to progressively fuse the multiscale features extracted by different layers. The fused feature maps are interpolated and aggregated with dual-pol SAR images to form a compact feature representation, which integrates the high- and low-level information of the network. Finally, a three-layer convolutional neural network (CNN) with a 1 × 1 convolutional kernel is employed to establish the mapping relationship between the feature representation and polarimetric covariance matrices. To evaluate the quad-pol SAR data reconstruction performance, both polarimetric target decomposition and terrain classification are adopted. Experimental studies are conducted on the ALOS/PALSAR and UAVSAR datasets. The qualitative and quantitative experimental results demonstrate the superiority of the proposed method. The reconstructed quad-pol SAR data can better sense buildings’ double-bounce scattering changes before and after a disaster. Furthermore, the reconstructed quad-pol SAR data of the proposed method achieve a 97.08% classification accuracy, which is 1.25% higher than that of dual-pol SAR data.

CEMLB-YOLO: Efficient Detection Model of Maize Leaf Blight in Complex Field Environments

Northern corn leaf blight is a severe fungal disease that adversely affects the health of maize crops. In order to prevent maize yield decline caused by leaf blight, we propose the YOLOv5-based object detection lightweight models to rapidly detect maize leaf blight disease in complex scenarios. Firstly, the Crucial Information Position Attention Mechanism (CIPAM) enables the model to focus on retaining critical information during downsampling to reduce information loss. We introduce the Feature Restructuring and Fusion Module (FRAFM) to extract deep semantic information and make the feature map fusion across maps at different scales more effective. Thirdly, we add the Mobile Bi-Level Transformer (MobileBit) to the feature extraction network to help the model understand complex scenes more effectively and cost-effectively. The experimental results demonstrate that the proposed model achieves 87.5% mAP@0.5 accuracy on the NLB dataset, which is 5.4% higher than the original model.

GMDNet: An Irregular Pavement Crack Segmentation Method Based on Multi-Scale Convolutional Attention Aggregation

Pavement cracks are the primary type of distress that cause road damage, and deep-learning-based pavement crack segmentation is a critical technology for current pavement maintenance and management. To address the issues of segmentation discontinuity and poor performance in the segmentation of irregular cracks faced by current semantic segmentation models, this paper proposes an irregular pavement crack segmentation method based on multi-scale convolutional attention aggregation. In this approach, GhostNet is first introduced as the model backbone network for reducing parameter count, with dynamic convolution enhancing GhostNet’s feature extraction capability. Next, a multi-scale convolutional attention aggregation module is proposed to cause the model to focus more on crack features and thus improve the segmentation effect on irregular cracks. Finally, a progressive up-sampling structure is used to enrich the feature information by gradually fusing feature maps of different depths to enhance the continuity of segmentation results. The experimental results on the HGCrack dataset show that GMDNet has a lighter model structure and higher segmentation accuracy than the mainstream semantic segmentation algorithms, achieving 75.16% of MIoU and 84.43% of F1 score, with only 7.67 M parameters. Therefore, the GMDNet proposed in this paper can accurately and efficiently segment irregular cracks on pavements that are more suitable for pavement crack segmentation scenarios in practical applications.

Multisource neural network feature map fusion: An efficient strategy to detect plant diseases

The use of RGB cameras or multispectral imaging systems can provide a wide range of applications for crop monitoring, plant phenotyping and disease detection. Although several approaches have been proposed, they increasingly use convolutional neural network-based architectures, which have, however, become increasingly cumbersome for improving classification results and difficult to train with few labeled data. Other increasingly popular approaches consist of using an ensemble of convolutional neural networks, in which each model solves a different problem. Since the inference is time- and resource-consuming due to the execution of multiple models, recent works have focused on transferring knowledge from an ensemble of models to a compact model to obtain better performance. In this paper, we propose an original approach that improves both accuracy and speed by reusing feature maps extracted by heterogeneous models from different data. Linked to each model, a transformation block allows keeping the correct number of feature maps and changing their dimension if necessary. To generate the feature maps, we only need the first layers of the ensemble models, thus taking advantage of ensemble learning methods, while adding only a few layers of a second model dedicated to aggregation of features. This approach allows an ensemble of models to be combined with different architectures that can process different data, such as several representations of the same input image or multispectral images, while being fast enough at the inference stage. This approach is adapted to hierarchical classification tasks by re-exploiting the same feature maps with different transformation blocks, offering accuracy gains in tasks not handled by the ensemble model. The results are provided for the PlantVillage dataset, with RGB images converted to three different color spaces, and for a custom Grapevine Yellow dataset, with multispectral images acquired with two different multispectral cameras.

A new real-time image semantic segmentation framework based on a lightweight deep convolutional encoder-decoder architecture for robotic environment sensing

Background: Image semantic segmentation can be understood as the allocation of a predefined category label to each pixel in the image to achieve the region segmentation of the image. Different categories in the image are identified with different colors. While achieving pixel classification, the position information of pixel points of different categories in the image is retained. Purpose: Due to the influence of background and complex environment, the traditional semantic segmentation methods have low accuracy. To alleviate the above problems, this paper proposes a new real-time image semantic segmentation framework based on a lightweight deep convolutional encoder-decoder architecture for robotic environment sensing. Methodology: This new framework is divided into three stages: encoding stage, decoding stage and dimension reduction stage. In the coding stage, a cross-layer feature map fusion (CLFMF) method is proposed to improve the effect of feature extraction. In the decoding stage, a new lightweight decoder (LD) structure is designed to reduce the number of convolutional layers to speed up model training and prediction. In the dimension reduction stage, the convolution dimension reduction method (CDR) is presented to connect the encoder and decoder layer by layer to enhance the decoder effect. Results: Compared with other state-of-the-art image semantic segmentation methods, we conduct comparison experiments on datasets Cityscapes, SUN RGB-D, CamVid, KITTI. The Category iIoU combined with the proposed method is more than 70%, and the Category IoU is as high as 89.7%. Conclusion: The results reflect that the new method can achieve the better semantic segmentation effect.

PCBSNet: A Pure Convolutional Bilateral Segmentation Network for Real-Time Natural Scene Text Detection

Scene text detection is a fundamental research work in the field of image processing and has extensive application value. Segmentation-based methods have time-consuming feature processing, while post-processing algorithms are excellent. Real-time semantic segmentation methods use lightweight backbone networks for feature extraction and aggregation but lack effective post-processing methods. The pure convolutional network improves model performance by changing key components. Combining the advantages of three types of methods, we propose a Pure Convolutional Bilateral Segmentation Network (PCBSNet) for real-time natural scene text detection. First, we constructed a bilateral feature extraction backbone network to significantly improve detection speed. The low extraction detail branch captures spatial information, while the efficient semantic extraction branch accurately captures semantic features through a series of micro designs. Second, we built an efficient attention aggregation module to guide the efficient and adaptive aggregation of features from the two branches. The fused feature map undergoes feature enhancement to obtain more accurate and reliable feature representation. Finally, we used differentiable binarization post-processing to construct text instance boundaries. To evaluate the effectiveness of the proposed model, we compared it with mainstream lightweight models on three datasets: ICDAR2015, MSRA-TD500, and CTW1500. The F-measure scores were 82.9%, 82.8%, and 78.9%, respectively, and the FPS were 59.1, 94.3, and 75.5 frames per second. We also conducted extensive ablation experiments on the ICDAR2015 dataset to validate the rationality of the proposed improvements. The obtained results indicate that the proposed model significantly improves inference speed while enhancing accuracy and demonstrates good competitiveness compared to other advanced detection methods. However, when faced with curved text, the detection performance of PCBSNet needs to be improved.

Multiscale Feature Fusion for Hyperspectral Marine Oil Spill Image Segmentation

Oil spills have always been a threat to the marine ecological environment; thus, it is important to identify and divide oil spill areas on the ocean surface into segments after an oil spill accident occurs to protect the marine ecological environment. However, oil spill area segmentation using ordinary optical images is greatly interfered with by the absorption of light by the deep sea and the distribution of algal organisms on the ocean surface, and it is difficult to improve segmentation accuracy. To address the above problems, a hyperspectral ocean oil spill image segmentation model with multiscale feature fusion (MFFHOSS-Net) is proposed. Specifically, the oil spill segmentation dataset was created using hyperspectral image data from NASA for the Gulf of Mexico oil spill, small-size images after the waveband filtering of the hyperspectral images were generated and the oil spill images were annotated. The model makes full use of having different layers with different characteristics by fusing feature maps of different scales. In addition, an attention mechanism was used to effectively fuse these features to improve the oil spill region segmentation accuracy. A case study, ablation experiments and model evaluation were also carried out in this work. Compared with other models, our proposed method achieved good results according to various evaluation metrics.

FFA-YOLOv7: Improved YOLOv7 Based on Feature Fusion and Attention Mechanism for Wearing Violation Detection in Substation Construction Safety

Ensuring compliance with safety regulations regarding wearing is essential for the safety and security of those working on substation construction sites. However, relying on supervisors to monitor workers in real time on the work site or through remote surveillance videos is both unreasonable and inefficient. A deep learning network approach named FFA-YOLOv7 is presented in this study that utilizes an improved version of YOLOv7 to detect violations of worker wearing in real time during power construction site surveillance. In YOLOv7, the feature pyramid network (FPN) of the neck stage is constructed through continuous upsampling and skip connections for feature fusion, after continuous downsampling of the backbone. However, this process can result in the loss of precise shallow position information. To tackle this issue, we have introduced a novel feature fusion pathway to the FPN architecture, enabling each layer not only to fuse feature maps from the same level during the downsampling course but also to fuse feature maps from shallower levels. This approach combines precise positional information from shallow layers with rich semantic information from deep layers. Additionally, we utilized attention after feature fusion in each layer to optimize the feature map fusion effect and achieve better detection accuracy performance. In order to conduct comparative experiments, we trained six variations of the YOLO model as detectors using a dataset gathered from realistic construction sites. The experimental results indicate that our proposed FFA-YOLOv7 attained a detection precision of 95.92% and a recall rate of 97.13%, demonstrating a high level of accuracy and a low rate of missed detections. These outcomes effectively satisfy the requirements for robust and accurate detection of real-world power construction violations.

A novel UNet segmentation method based on deep learning for preferential flow in soil

Accurate binary images based on dye-tracing experiments are fundamental for understanding preferential flow in soil. Nevertheless, general image-processing software, such as Photoshop, Image-Pro Plus, and ImageJ, cannot automatically and accurately binarize dye-strained regions for the areas with uneven staining and blurred boundaries, causing errors in the subsequent analysis of preferential flow characteristics. Therefore, this study aimed to develop a novel UNet segmentation method based on a dual-scale attention residual module (DARM-UNet), to improve the segmentation accuracy of preferential flow images and assist in digital soil descriptions. The proposed method is a two-level nested U-structure, i.e., a dual-scale attention residual module (DARM) on the lowest level and the typical architecture of UNet at the top level. On the lowest level, the DARM obtained improved contextual information and expand the receptive field based on the attention mechanism and the dual-scale convolution, which was used to improve the recognition ability of uneven dye and low-contrast regions. At the top level, the typical architecture of the UNet model was followed to fuse feature maps of different resolutions, in which each stage is filled by a DARM block. Compared with general image processing software and deep learning methods, the DARM-UNet method showed higher preferential flow segmentation accuracies (95.71% and 93.29%), recall rates (91.91% and 89.29%), and harmonic means (92.72% and 91.82%) for the natural secondary and hazelnut shrub forests, respectively. The four preferential flow indicators indicated that the flow patterns of the natural secondary and hazelnut shrub forests varied with soil depth. In natural secondary forests, the preferential flow started from the surface and was of the primary flow type, whereas in hazelnut shrub forests, it occurred only in the deeper layer, following the matrix flow and concentrating in the 0–16 cm soil layer. This study demonstrated that the proposed DARM-UNet method could effectively identify and segment preferential flow of different forests, especially the areas with uneven staining and blurred boundaries.

Prediction of pulse wave of target organ damage in hypertension based on multiple feature fusion

With respect to less efficiency and low accuracy of predicting on hypertensive target organ damage, this article proposes a fusion prediction model combining pulse-taking with inquiry diagnosis of traditional Chinese medicine to accomplish the efficient and non-invasive diagnosis. Regarding the class imbalance of inquiry diagnosis samples, an Eliminated random forest algorithm is proposed to select efficient features and reduce the impact of class imbalance on classification performance via cluster-based under-sampling algorithm. As to low discriminability of hypertensive time-domain pulse wave samples, time-domain pulse wave is transformed to the frequency-domain MFCC feature maps, and fuse feature maps of inquiry diagnosis scale for predicting hypertension target organ damage. In the article, the clinical 608 cases of hypertensive target organ damage are from Longhua Hospital affiliated to Shanghai University of Chinese Medicine and Hospital of Integrated Traditional Chinese and Western Medicine concerning pulse-taking and inquiry diagnosis. The evaluation indicators of 5-Fold cross-validation classification, i.e. F1-score, Accuracy, Precision, Sensitivity, AUC, are 97.31%, 98.72%, 97.71%, 97.04%, 99.13% respectively, which are higher than those of the other typical models. In addition, this article also studies the correlation between classification of pulse-taking or inquiry diagnosis and its features, and analyzes the feature importance ranking on pulse-taking and inquiry diagnosis, which aids clinicians to seek the occurrence mechanisms of hypertensive target organ damage, and find the effective measurements for timely prevention and treatment.

Application of Multi-Scale Convolutional Neural Networks and Extreme Learning Machines in Mechanical Fault Diagnosis

Extracting fault features in mechanical fault diagnosis is challenging and leads to low diagnosis accuracy. A novel fault diagnosis method using multi-scale convolutional neural networks (MSCNN) and extreme learning machines is presented in this research, which was conducted in three stages: First, the collected vibration signals were transformed into images using the continuous wavelet transform. Subsequently, an MSCNN was designed to extract all detailed features of the original images. The final feature maps were obtained by fusing multiple feature layers. The parameters in the network were randomly generated and remained unchanged, which could effectively accelerate the calculation. Finally, an extreme learning machine was used to classify faults based on the fused feature maps, and the potential relationship between the fault and labels was established. The effectiveness of the proposed method was confirmed. This method performs better in mechanical fault diagnosis and classification than existing methods.

Facial Expression Recognition With Visual Transformers and Attentional Selective Fusion

Facial Expression Recognition (FER) in the wild is extremely challenging due to occlusions, variant head poses, face deformation and motion blur under unconstrained conditions. Although substantial progresses have been made in automatic FER in the past few decades, previous studies were mainly designed for lab-controlled FER. Real-world occlusions, variant head poses and other issues definitely increase the difficulty of FER on account of these information-deficient regions and complex backgrounds. Different from previous pure CNNs based methods, we argue that it is feasible and practical to translate facial images into sequences of visual words and perform expression recognition from a global perspective. Therefore, we propose the Visual Transformers with Feature Fusion (VTFF) to tackle FER in the wild by two main steps. First, we propose the attentional selective fusion (ASF) for leveraging two kinds of feature maps generated by two-branch CNNs. The ASF captures discriminative information by fusing multiple features with the global-local attention. The fused feature maps are then flattened and projected into sequences of visual words. Second, inspired by the success of Transformers in natural language processing, we propose to model relationships between these visual words with the global self-attention. The proposed method is evaluated on three public in-the-wild facial expression datasets (RAF-DB, FERPlus and AffectNet). Under the same settings, extensive experiments demonstrate that our method shows superior performance over other methods, setting new state of the art on RAF-DB with 88.14%, FERPlus with 88.81% and AffectNet with 61.85%. The cross-dataset evaluation on CK+ shows the promising generalization capability of the proposed method.