Fusion Block Research Articles

MULTI-SCALE ATTENTION BASED U-NET MODEL FOR LIVER TUMOR SEGMENTATION

It is essential to automatically evaluate the position and size of the liver tumour for radiologists, diagnosis, and the clinical process. Many U-Net-based variants have been suggested in recent years to enhance the segmentation results for medical image segmentation, but they are unable to describe the global spatial and channel relationships among lesion regions. To overcome this issues, we proposed a novel network called Multi-scale Attention UNet (MA-UNet) to address this problem by adding a self-attention mechanism into our approach to adaptively combine local features with their global dependencies. The attention mechanism of the MA-UNet allows it to capture complex contextual dependencies. Position-wise Attention Block and Multi-scale Fusion Attention Block are the two blocks that we have developed. The feature interdependencies in spatial dimensions, which represent the spatial dependencies between pixels in a global view, are modelled using the Position-wise Attention Block. A multi-scale semantic feature fusion attention block is also used to capture the channel dependencies between any feature map. On the MICCAI 2017 LiTS Competition dataset, we assess our methodology. Compared to other cutting-edge methods, the suggested way performs better. The Dice and VOE values of liver tumors segmentation are 0.749 ± 0.08 and 0.21 ± 0.06 respectively. KEYWORDS: Liver tumor segmentation, Attention mechanism, Deep learning.

Tuberculosis chest X-ray detection using CNN-based hybrid segmentation and classification approach

Tuberculosis still significantly impacts the world's population, with more than 10 million people getting sick each year. Researchers have focused on developing computer-aided diagnosis (CAD) technology based on X-ray imaging to support the identification of tuberculosis, and deep learning is a popular and efficient method. However, deep learning-based CAD approaches usually ignore the relationship between the two vision tasks of specific region segmentation and classification. In this research, we introduced a novel TB-UNet, which is based on dilated fusion block (DF) and Attention block (AB) block for accurate segmentation of lungs regions and achieved the highest results in terms of Precision (0.9574), Recall (0.9512), and F1score (0.8988), IoU (0.8168) and Accuracy (0.9770). We also proposed TB-DenseNet which is based on five dual convolution blocks, DenseNet-169 layer, and a feature fusion block for the precise classification of tuberculosis images. The experiments have been performed on three chest X-ray (CXR) datasets, segmented images, and original images are fed to TB-DenseNet for better classification. Furthermore, the proposed method is tested against simultaneously three different diseases, such as Pneumonia, COVID-19, and Tuberculous. The highest results are achieved in terms of Precision (0.9567), Recall (0.9510), F1score (0.9538), and Accuracy (0.9510). The achieved results reflect that our proposed method produces the highest accuracy compared to the state-of-the-art methods. The source code is available at: https://github.com/ahmedeqbal/TB-DenseNet.

MFEFNet: Multi-scale feature enhancement and Fusion Network for polyp segmentation

The polyp segmentation technology based on computer-aided can effectively avoid the deterioration of polyps and prevent colorectal cancer. To segment the polyp target precisely, the Multi-Scale Feature Enhancement and Fusion Network (MFEFNet) is proposed. First of all, to balance the network's predictive ability and complexity, ResNet50 is designed as the backbone network, and the Shift Channel Block (SCB) is used to unify the spatial location of feature mappings and emphasize local information. Secondly, to further improve the network's feature-extracting ability, the Feature Enhancement Block (FEB) is added, which decouples features, reinforces features by multiple perspectives and reconstructs features. Meanwhile, to weaken the semantic gap in the feature fusion process, we propose strong associated couplers, the Multi-Scale Feature Fusion Block (MSFFB) and the Reducing Difference Block (RDB), which are mainly composed of multiple cross-complementary information interaction modes and reinforce the long-distance dependence between features. Finally, to further refine local regions, the Polarized Self-Attention (PSA) and the Balancing Attention Module (BAM) are introduced for better exploration of detailed information between foreground and background boundaries. Experiments have been conducted under five benchmark datasets (Kvasir-SEG, CVC-ClinicDB, CVC-ClinicDB, CVC300 and CVC-ColonDB) and compared with state-of-the-art polyp segmentation algorithms. The experimental result shows that the proposed network improves Dice and mean intersection over union (mIoU) by an average score of 3.4% and 4%, respectively. Therefore, extensive experiments demonstrate that the proposed network performs favorably against more than a dozen state-of-the-art methods on five popular polyp segmentation benchmarks.

NAG-Net: Nested attention-guided learning for segmentation of carotid lumen-intima interface and media-adventitia interface

Cardiovascular diseases (CVD), as the leading cause of death in the world, poses a serious threat to human health. The segmentation of carotid Lumen-intima interface (LII) and Media-adventitia interface (MAI) is a prerequisite for measuring intima-media thickness (IMT), which is of great significance for early screening and prevention of CVD. Despite recent advances, existing methods still fail to incorporate task-related clinical domain knowledge and require complex post-processing steps to obtain fine contours of LII and MAI. In this paper, a nested attention-guided deep learning model (named NAG-Net) is proposed for accurate segmentation of LII and MAI. The NAG-Net consists of two nested sub-networks, the Intima-Media Region Segmentation Network (IMRSN) and the LII and MAI Segmentation Network (LII-MAISN). It innovatively incorporates task-related clinical domain knowledge through the visual attention map generated by IMRSN, enabling LII-MAISN to focus more on the clinician’s visual focus region under the same task during segmentation. Moreover, the segmentation results can directly obtain fine contours of LII and MAI through simple refinement without complicated post-processing steps. To further improve the feature extraction ability of the model and reduce the impact of data scarcity, the strategy of transfer learning is also adopted to apply the pretrained weights of VGG-16. In addition, a channel attention-based encoder feature fusion block (EFFB-ATT) is specially designed to achieve efficient representation of useful features extracted by two parallel encoders in LII-MAISN. Extensive experimental results have demonstrated that our proposed NAG-Net outperformed other state-of-the-art methods and achieved the highest performance on all evaluation metrics.

RsALUNet: A reinforcement supervision U-Net-based framework for multi-ROI segmentation of medical images

A new multiple region of interest (multi-ROI) segmentation framework, RsALUNet, is proposed in this paper, whose backbone was U-Net. Rs represented the reinforcement-supervision strategy by utilizing adversarial learning (AL) between U-Net’s decoders and an additional discriminator, which was based on the differences among the segmentation results (diffSegs) and labels of multi-ROI (diffLabels). As the AL progressed, diffSegs was increasingly similar to diffLabels, and this further contributed to a more accurate segmentation of each member in the multi-ROI. In addition to the Rs strategy, three blocks were proposed to enhance RsALUNet, namely, a dilated convolution chain providing diverse and large receptive fields to accommodate different target sizes, a fusion block integrating features of large targets to small ones to optimize the segmentation of the latter, and a location-encoder block extracting multi-scale positional information to enhance the model's attention to the ROI. RsALUNet was evaluated through three multi-ROI segmentation tasks using different imaging modalities, including X-ray, ultrasound, and magnetic-resonance (MR) imaging. The mean Dice coefficient (Dice) increased from 1.1% to 8.5% compared to the other frameworks. The results demonstrate the promising adaptability and extendibility of our strategy and RsALUNet for multi-ROI segmentation in X-ray, ultrasound, and MR images.

Hierarchical graph neural network with adaptive cross-graph fusion for remaining useful life prediction

Multi-sensor monitoring data provide abundant information resources for complex machine systems, which facilitates monitoring the degradation process of machinery and ensuring the reliability of the industrial process. However, previous prognostic methods focus more on the sequential data obtained from multi-sensors, while ignoring the underlying prior structural information of the equipment. To fully leverage the structural information into the modeling process, and thus improve the remaining useful life (RUL) prediction performance, a hierarchical graph neural network with adaptive cross-graph fusion (HGNN-ACGF) method for RUL prediction is proposed in this study. In the HGNN-ACGF method, a hierarchical graph consisting of a sensor graph and a module graph is constructed by introducing the structural information to fully model the degradation trend information of the complex machine system. Besides, the graph neural network (GNN) is adopted to learn the representation at both the module graph and sensor graph, and an adaptive cross-graph fusion (ACGF) block is proposed. Owing to the cross-graph fusion block, the representation from different graphs can be fused adaptively by considering the relative importance between different modules and sensors. To verify the proposed method, the experiments were conducted on a set of degradation data sets of aircraft engines provided by the Commercial Modular Aero-Propulsion System Simulation. The experimental results show that the proposed method has superior performance in RUL prediction over the state-of-the-art methods.

Fusion and Allocation Network for Light Field Image Super-Resolution

Light field (LF) images taken by plenoptic cameras can record spatial and angular information from real-world scenes, and it is beneficial to fully integrate these two pieces of information to improve image super-resolution (SR). However, most of the existing approaches to LF image SR cannot fully fuse the information at the spatial and angular levels. Moreover, the performance of SR is hindered by the ability to incorporate distinctive information from different views and extract informative features from each view. To solve these core issues, we propose a fusion and allocation network (LF-FANet) for LF image SR. Specifically, we have designed an angular fusion operator (AFO) to fuse distinctive features among different views, and a spatial fusion operator (SFO) to extract deep representation features for each view. Following these two operators, we further propose a fusion and allocation strategy to incorporate and propagate the fusion features. In the fusion stage, the interaction information fusion block (IIFB) can fully supplement distinctive and informative features among all views. For the allocation stage, the fusion output features are allocated to the next AFO and SFO for further distilling the valid information. Experimental results on both synthetic and real-world datasets demonstrate that our method has achieved the same performance as state-of-the-art methods. Moreover, our method can preserve the parallax structure of LF and generate faithful details of LF images.

Learning multi-view and centerline topology connectivity information for pulmonary artery–vein separation

Background:Automatic pulmonary artery–vein separation has considerable importance in the diagnosis and treatment of lung diseases. However, insufficient connectivity and spatial inconsistency have always been the problems of artery–vein separation. Methods:A novel automatic method for artery–vein separation in CT images is presented in this work. Specifically, a multi-scale information aggregated network (MSIA-Net) including multi-scale fusion blocks and deep supervision, is proposed to learn the features of artery–vein and aggregate additional semantic information, respectively. The proposed method integrates nine MSIA-Net models for artery–vein separation, vessel segmentation, and centerline separation tasks along with axial, coronal, and sagittal multi-view slices. First, the preliminary artery–vein separation results are obtained by the proposed multi-view fusion strategy (MVFS). Then, centerline correction algorithm (CCA) is used to correct the preliminary results of artery–vein separation by the centerline separation results. Finally, the vessel segmentation results are utilized to reconstruct the artery–vein morphology. In addition, weighted cross-entropy and dice loss are employed to solve the class imbalance problem. Results:We constructed 50 manually labeled contrast-enhanced computed CT scans for five-fold cross-validation, and experimental results demonstrated that our method achieves superior segmentation performance of 97.7%, 85.1%, and 84.9% on ACC, Pre, and DSC, respectively. Additionally, a series of ablation studies demonstrate the effectiveness of the proposed components. Conclusion:The proposed method can effectively solve the problem of insufficient vascular connectivity and correct the spatial inconsistency of artery–vein.

SDTF-Net: Static and dynamic time–frequency network for Speech Emotion Recognition

Speech Emotion Recognition (SER) is an important but challenging research topic in Human–Computer interaction (HCI). As a crucial component influencing the model’s performance, in SER, feature extraction needs to comprehensively consider the static local states, the dynamic change tendency, and the time–frequency characteristics of speech signals to generate meaningful emotional representations and further improve the SER performance. To address this aim, we propose the Static and Dynamic Time–frequency Network called SDTF-Net. To reduce the interference between different features, in SDTF-Net, we introduce two parallel modules, i.e., Time–Frequency Convolutional Neural Network (TF-CNN) and Time–Frequency Transformer (TF-Trans), to generate the static and dynamic emotional features based on the time–frequency information, respectively. Specifically, in the TF-Trans module, we design a three-step strategy to learn the relatively complicated dynamic feature, comprising two parallel Bi-direction Multi-scale Dynamic Activation (BMDA) blocks, two parallel transformer (Trans) blocks, and a time–frequency fusion (TF-Cross) block. Subsequently, we propose an SD-Cross module to effectively fuse the learned static and dynamic features, generating the final meaningful emotional representations. Given the problem of data sparsity, moreover, we employ the data augmentation method called SpecAugment to improve the learning ability of SDTF-Net. From the experimental results on IEMOCAP and MSP-IMPROV datasets, we observe that SDTF-Net outperforms the state-of-the-art methods. Furthermore, extensive ablation experiments further indicate the rationality of the SDTF-Net structure and the necessity of considering dynamic and static features based on time–frequency information comprehensively in SER.

BSNet: Boundary-semantic-fusion network for farmland parcel mapping in high-resolution satellite images

Mapping farmland parcels using satellite images is essential for agricultural remote sensing applications. Loss of spatial details and positioning of parcel boundaries are the main challenges in available deep convolution network (DCN) models. This study developed a boundary-semantic-fusion DCN (BSNet) model for delineating farmland parcels from high-resolution satellite images. Central to this method is the combination between shallow-level boundary features with accurate spatial positioning and deep-level semantic features for category identification. First, a general deep convolution framework consisting of boundary, semantic and fusion blocks was implemented for farmland parcel mapping. Second, a particular serial structure with a detaching operation for linking the boundary and semantic blocks was explored to maintain the spatial details and fine-scale boundaries in feature learning. Third, an encoder-decoder fusion block was developed to integrate the boundary and semantic features to produce the final parcel maps. We validated the proposed model with different high-resolution satellite images in two study areas. The experimental results, with improvements greater than 4% in the F1 score and 6% in the IoU score relative to other comparative methods, illustrate the effectiveness of the proposed model for fine-scale farmland parcel mapping.

A vector conversion method of building indoor space data based on attribute classification

Aiming at the problems of low data conversion efficiency, low accuracy and low data utilization rate after conversion in traditional methods, this paper proposes a vector conversion method for building indoor space data based on attribute classification. Firstly, the transformation process of data vectors is analyzed. Secondly, block matching detection and fusion recognition were carried out on the building interior space images, and fuzzy feature extraction method was used to optimize the collection and feature recognition of the building interior space data. Then, the attribute classification method is used to obtain the condition attribute and decision attribute of the data, and realize the building interior space data mining. Then, the K-means algorithm is used to cluster the indoor spatial data samples, and the wavelet transform method is used to de-noise the noisy data in advance. Finally, the obtained data is processed by vector transformation. The experimental results show that the data conversion efficiency of this method is high, and the data conversion accuracy and data utilization rate have been improved.

Fault Diagnosis for Rolling Bearings Based on Multiscale Feature Fusion Deep Residual Networks

Deep learning, due to its excellent feature-adaptive capture ability, has been widely utilized in the fault diagnosis field. However, there are two common problems in deep-learning-based fault diagnosis methods: (1) many researchers attempt to deepen the layers of deep learning models for higher diagnostic accuracy, but degradation problems of deep learning models often occur; and (2) the use of multiscale features can easily be ignored, which makes the extracted data features lack diversity. To deal with these problems, a novel multiscale feature fusion deep residual network is proposed in this paper for the fault diagnosis of rolling bearings, one which contains multiple multiscale feature fusion blocks and a multiscale pooling layer. The multiple multiscale feature fusion block is designed to automatically extract the multiscale features from raw signals, and further compress them for higher dimensional feature mapping. The multiscale pooling layer is constructed to fuse the extracted multiscale feature mapping. Two famous rolling bearing datasets are adopted to evaluate the diagnostic performance of the proposed model. The comparison results show that the diagnostic performance of the proposed model is superior to not only several popular models, but also other advanced methods in the literature.

Video Based Cocktail Causal Container for Blood Pressure Classification and Blood Glucose Prediction.

With the development of modern cameras, more physiological signals can be obtained from portable devices like smartphone. Some hemodynamically based non-invasive video processing applications have been applied for blood pressure classification and blood glucose prediction objectives for unobtrusive physiological monitoring at home. However, this approach is still under development with very few publications. In this paper, we propose an end-to-end framework, entitled cocktail causal container, to fuse multiple physiological representations and to reconstruct the correlation between frequency and temporal information during multi-task learning. Cocktail causal container processes hematologic reflex information to classify blood pressure and blood glucose. Since the learning of discriminative features from video physiological representations is quite challenging, we propose a token feature fusion block to fuse the multi-view fine-grained representations to a union discrete frequency space. A causal net is used to analyze the fused higher-order information, so that the framework can be enforced to disentangle the latent factors into the related endogenous association that corresponds to down-stream fusion information to improve the semantic interpretation. Moreover, a pair-wise temporal frequency map is developed to provide valuable insights into extraction of salient photoplethysmograph (PPG) information from fingertip videos obtained by a standard smartphone camera. Extensive comparisons have been implemented for the validation of cocktail causal container using a Clinical dataset and PPG-BP benchmark. The root mean square error of 1.329±0.167 for blood glucose prediction and precision of 0.89±0.03 for blood pressure classification are achieved in Clinical dataset.

Lipid-mediated interplay between host restriction factor Serinc5 and HIV-1 ENV glycoprotein.

Human immunodeficiency virus (HIV-1) is a pathogenic retrovirus that continues to be a severe threat to public health. HIV-1 is an enveloped virus that enters and infects cells through cell-virus membrane fusion, mediated by HIV-1 envelope glycoprotein, Env.The host restriction factor, Serinc5, impairs this process. It reduces HIV infection rates by incorporating itself into the viral membrane and blocking fusion of the virus with the plasma membrane of uninfected cells. The exact restriction mechanism of Serinc5 remains unknown. Studies suggest that Serinc5 disrupts Env distribution at the membrane and inhibits the protein's fusion function. We hypothesize that Serinc5 acts on Env by altering the physical properties of the surrounding membrane. In this study, we evaluate the nature of the membrane properties affected by Serinc5. Applying the fluorescent probe FLIPPER-TR to measure membrane order in viral particles with and without Serinc5, we discovered that Serinc5 slightly increases lipid order. HIV-1 bearing Env with a truncated cytoplasmic tail (∆CT) avoids infection restriction by Serinc5. We used isolated giant plasma membrane vesicles (GPMVs) to quantify Serinc5 and Env partitioning into specific lipid domains. Our results indicate that Serinc5 and Env partition into the liquid-disordered lipid domains of the plasma membrane. Interestingly, mutant Env ∆CT showed different lipid phase preferences than Serinc5 and wild-type Env. We discuss how these different lipid interactions might affect entry of progeny viruses into cells by membrane fusion.

Simultaneous superresolution reconstruction and distortion correction for single-shot EPI DWI using deep learning.

Single-shot (SS) EPI is widely used for clinical DWI. This study aims to develop an end-to-end deep learning-based method with a novel loss function in an improved network structure to simultaneously increase the resolution and correct distortions for SS-EPI DWI. Point-spread-function (PSF)-encoded EPI can provide high-resolution, distortion-free DWI images. A distorted image from SS-EPI can be described as the convolution between a PSF function with a distortion-free image. The deconvolution process to recover the distortion-free image can be achieved with a convolution neural network, which also learns the mapping function between low-resolution SS-EPI and high-resolution reference PSF-EPI to achieve superresolution. To suppress the oversmoothing effect, we proposed a modified generative adversarial network structure, in which a dense net with gradient map guidance and a multilevel fusion block was used as the generator. A fractional anisotropy loss was proposed to utilize the diffusion anisotropy information among diffusion directions. In vivo brain DWI data were used to test the proposed method. The results show that distortion-corrected high-resolution DWI images with restored structural details can be obtained from low-resolution SS-EPI images by taking advantage of the high-resolution anatomical images. Additionally, the proposed network can improve the quantitative accuracy of diffusion metrics compared with previously reported networks. Using high-resolution, distortion-free EPI-DWI images as references, a deep learning-based method to simultaneously increase the perceived resolution and correct distortions for low-resolution SS-EPI was proposed. The results show that DWI image quality and diffusion metrics can be improved.

MFR-DTA: a multi-functional and robust model for predicting drug-target binding affinity and region.

Recently, deep learning has become the mainstream methodology for drug-target binding affinity prediction. However, two deficiencies of the existing methods restrict their practical applications. On the one hand, most existing methods ignore the individual information of sequence elements, resulting in poor sequence feature representations. On the other hand, without prior biological knowledge, the prediction of drug-target binding regions based on attention weights of a deep neural network could be difficult to verify, which may bring adverse interference to biological researchers. We propose a novel Multi-Functional and Robust Drug-Target binding Affinity prediction (MFR-DTA) method to address the above issues. Specifically, we design a new biological sequence feature extraction block, namely BioMLP, that assists the model in extracting individual features of sequence elements. Then, we propose a new Elem-feature fusion block to refine the extracted features. After that, we construct a Mix-Decoder block that extracts drug-target interaction information and predicts their binding regions simultaneously. Last, we evaluate MFR-DTA on two benchmarks consistently with the existing methods and propose a new dataset, sc-PDB, to better measure the accuracy of binding region prediction. We also visualize some samples to demonstrate the locations of their binding sites and the predicted multi-scale interaction regions. The proposed method achieves excellent performance on these datasets, demonstrating its merits and superiority over the state-of-the-art methods. https://github.com/JU-HuaY/MFR.

Dilated convolution network with edge fusion block and directional feature maps for cardiac MRI segmentation.

Cardiac magnetic resonance imaging (MRI) segmentation task refers to the accurate segmentation of ventricle and myocardium, which is a prerequisite for evaluating the soundness of cardiac function. With the development of deep learning in medical imaging, more and more heart segmentation methods based on deep learning have been proposed. Due to the fuzzy boundary and uneven intensity distribution of cardiac MRI, some existing methods do not make full use of multi-scale characteristic information and have the problem of ambiguity between classes. In this paper, we propose a dilated convolution network with edge fusion block and directional feature maps for cardiac MRI segmentation. The network uses feature fusion module to preserve boundary information, and adopts the direction field module to obtain the feature maps to improve the original segmentation features. Firstly, multi-scale feature information is obtained and fused through dilated convolutional layers of different scales while downsampling. Secondly, in the decoding stage, the edge fusion block integrates the edge features into the side output of the encoder and concatenates them with the upsampled features. Finally, the concatenated features utilize the direction field to improve the original segmentation features and generate the final result. Our propose method conducts comprehensive comparative experiments on the automated cardiac diagnosis challenge (ACDC) and myocardial pathological segmentation (MyoPS) datasets. The results show that the proposed cardiac MRI segmentation method has better performance compared to other existing methods.

MsGoF: Breast lesion classification on ultrasound images by multi-scale gradational-order fusion framework

Background and Objective: Predicting the malignant potential of breast lesions based on breast ultrasound (BUS) images is a crucial component of computer-aided diagnosis system for breast cancers. However, since breast lesions in BUS images generally have various shapes with relatively low contrast and present complex textures, it still remains challenging to accurately identify the malignant potential of breast lesions. Methods: In this paper, we propose a multi-scale gradational-order fusion framework to make full advantages of multi-scale representations incorporating with gradational-order characteristics of BUS images for breast lesions classification. Specifically, we first construct a spatial context aggregation module to generate multi-scale context representations from the original BUS images. Subsequently, multi-scale representations are efficiently fused in feature fusion block that is armed with special fusion strategies to comprehensively capture morphological characteristics of breast lesions. To better characterize complex textures and enhance non-linear modeling capability, we further propose isotropous gradational-order feature module in the feature fusion block to learn and combine multi-order representations. Finally, these multi-scale gradational-order representations are utilized to perform prediction for the malignant potential of breast lesions. Results: The proposed model was evaluated on three open datasets by using 5-fold cross-validation. The experimental results (Accuracy: 85.32%, Sensitivity: 85.24%, Specificity: 88.57%, AUC: 90.63% on dataset A; Accuracy: 76.48%, Sensitivity: 72.45%, Specificity: 80.42%, AUC: 78.98% on dataset B) demonstrate that the proposed method achieves the promising performance when compared with other deep learning-based methods in BUS classification task. Conclusions: The proposed method has demonstrated a promising potential to predict malignant potential of breast lesion using ultrasound image in an end-to-end manner.

Cascaded information enhancement and cross-modal attention feature fusion for multispectral pedestrian detection

Multispectral pedestrian detection is a technology designed to detect and locate pedestrians in Color and Thermal images, which has been widely used in automatic driving, video surveillance, etc. So far most available multispectral pedestrian detection algorithms only achieved limited success in pedestrian detection because of the lacking take into account the confusion of pedestrian information and background noise in Color and Thermal images. Here we propose a multispectral pedestrian detection algorithm, which mainly consists of a cascaded information enhancement module and a cross-modal attention feature fusion module. On the one hand, the cascaded information enhancement module adopts the channel and spatial attention mechanism to perform attention weighting on the features fused by the cascaded feature fusion block. Moreover, it multiplies the single-modal features with the attention weight element by element to enhance the pedestrian features in the single-modal and thus suppress the interference from the background. On the other hand, the cross-modal attention feature fusion module mines the features of both Color and Thermal modalities to complement each other, then the global features are constructed by adding the cross-modal complemented features element by element, which are attentionally weighted to achieve the effective fusion of the two modal features. Finally, the fused features are input into the detection head to detect and locate pedestrians. Extensive experiments have been performed on two improved versions of annotations (sanitized annotations and paired annotations) of the public dataset KAIST. The experimental results show that our method demonstrates a lower pedestrian miss rate and more accurate pedestrian detection boxes compared to the comparison method. Additionally, the ablation experiment also proved the effectiveness of each module designed in this paper.

Emotion Recognition from Large-Scale Video Clips with Cross-Attention and Hybrid Feature Weighting Neural Networks.

The emotion of humans is an important indicator or reflection of their mental states, e.g., satisfaction or stress, and recognizing or detecting emotion from different media is essential to perform sequence analysis or for certain applications, e.g., mental health assessments, job stress level estimation, and tourist satisfaction assessments. Emotion recognition based on computer vision techniques, as an important method of detecting emotion from visual media (e.g., images or videos) of human behaviors with the use of plentiful emotional cues, has been extensively investigated because of its significant applications. However, most existing models neglect inter-feature interaction and use simple concatenation for feature fusion, failing to capture the crucial complementary gains between face and context information in video clips, which is significant in addressing the problems of emotion confusion and emotion misunderstanding. Accordingly, in this paper, to fully exploit the complementary information between face and context features, we present a novel cross-attention and hybrid feature weighting network to achieve accurate emotion recognition from large-scale video clips, and the proposed model consists of a dual-branch encoding (DBE) network, a hierarchical-attention encoding (HAE) network, and a deep fusion (DF) block. Specifically, the face and context encoding blocks in the DBE network generate the respective shallow features. After this, the HAE network uses the cross-attention (CA) block to investigate and capture the complementarity between facial expression features and their contexts via a cross-channel attention operation. The element recalibration (ER) block is introduced to revise the feature map of each channel by embedding global information. Moreover, the adaptive-attention (AA) block in the HAE network is developed to infer the optimal feature fusion weights and obtain the adaptive emotion features via a hybrid feature weighting operation. Finally, the DF block integrates these adaptive emotion features to predict an individual emotional state. Extensive experimental results of the CAER-S dataset demonstrate the effectiveness of our method, exhibiting its potential in the analysis of tourist reviews with video clips, estimation of job stress levels with visual emotional evidence, or assessments of mental healthiness with visual media.