Gated Fusion Network Research Articles

MGFNet: An MLP-dominated gated fusion network for semantic segmentation of high-resolution multi-modal remote sensing images

MGFNet: An MLP-dominated gated fusion network for semantic segmentation of high-resolution multi-modal remote sensing images

MGFNet: Cross-scene Crowd Counting via Multistage Gated Fusion Network

Existing crowd counting methods are mainly trained and tested in similar scenarios. When the testing and training scenarios of the model are different, the counting accuracy of these methods will sharply decrease, which seriously limits their practical application. To address this problem, we propose a multistage gated fusion network (MGFNet) for cross-scene crowd counting. MGFNet is primarily composed of dynamic gated convolution units (DGCU) and multilevel scale attention blocks (MSAB) modules. Specifically, DGCU uses a dynamic gating path to supplement detailed information to reduce the loss of crowd information and overestimation of background in different scenarios. MSAB calibrates crowd information at different scales and perspectives in different scenes by generating attention maps with discriminative information. In addition, we used a new global local consistency loss to optimize the model to adapt to changes in crowd density and distribution. Extensive experiments on four different types of scene counting benchmarks show that the proposed MGFNet achieves superior cross-scene counting performance.

Progressively-orthogonally-mapped EfficientNet for action recognition on time-range-Doppler signature

Although 2D radar signal representations, such as spectrograms and range-Doppler maps have been widely used for target recognition, 3D time-range-Doppler (TRD) has been less studied, partially because of the difficulties in extracting features from the TRD representation, i.e., shallow 3D neural networks have limited discriminant power, but repeatedly applying 3D convolutions will lead to an oversized 3D network. A hybrid 3D–2D network architecture, Progressively-Orthogonally-Mapped EfficientNet (POMEN), is proposed to address these challenges. More specifically, the proposed POMEN utilizes 3D convolutions in the earlier stages to capture the information embedded in the sparse 3D TRD representation, and to avoid the oversized feature map caused by excessively applying 3D convolutions, we propose to progressively map the 3D features into three sets of 2D features corresponding to the range-time signature, range-Doppler map and time-Doppler signature (spectrogram), respectively. Subsequently, 2D EfficientNet blocks were designed to extract discriminant information from the three sets of 2D feature maps. This hybrid 3D–2D network design effectively extracts features from the 3D TRD representation, thereby avoiding oversized features from full-sized 3D networks and the information loss of 2D networks on 2D representations. Finally, a homogeneous gated fusion network was designed to fuse the three sets of 2D features. The proposed method was evaluated on the UGRS, MIMOGR, and mmWRWD datasets. The experimental results for all datasets demonstrate that the proposed POMEN significantly and consistently outperforms the state-of-the-art models in both 2D and 3D representations.

Multi-scale recurrent attention gated fusion network for single image dehazing

The purpose of single image dehazing is to eliminate the bad influence of haze on images, so as to maintain more scene information of images. In recent years, the convolutional neural networks (CNN) have made significant contributions to single image dehazing. However, the visual quality of dehazed images still needs to be further improved. In view of the problems of single-scale shallow image feature extraction and the insufficient use of intermediate layer features in existing dehazing networks, we propose an end-to-end Multi-scale Recurrent Attention Gated Fusion Network (MRAGFN) to address the image dehazing task. We cascade three Dual Attention Fusion (DAF) modules to progressively form three haze-relevant features map, meanwhile, we adopt downsampling operation on the input to produce global feature map, which are used to weight the three feature maps to compensate for the missing of single-scale feature information. We present Feature Enhancement Module (FEM) to enhance the feature representation ability of these weighted feature maps. We design Recurrent Attention Gated Fusion (RAGF) module by adding attention mechanism and gating mechanism to gradually obtain more refined features based on these weighted features while eliminating redundant features. Experimental results on different hazy images demonstrate that the proposed dehazing network can restore the haze-free images and perform better than the state-of-the-art dehazing networks in terms of the objective indicators (such as PSNR, SSIM)and the subjective visual quality.

MSGFNet: Multi-Scale Gated Fusion Network for Remote Sensing Image Change Detection

Change detection (CD) stands out as a pivotal yet challenging task in the interpretation of remote sensing images. Significant developments have been witnessed, particularly with the rapid advancements in deep learning techniques. Nevertheless, challenges such as incomplete detection targets and unsmooth boundaries remain as most CD methods suffer from ineffective feature fusion. Therefore, this paper presents a multi-scale gated fusion network (MSGFNet) to improve the accuracy of CD results. To effectively extract bi-temporal features, the EfficientNetB4 model based on a Siamese network is employed. Subsequently, we propose a multi-scale gated fusion module (MSGFM) that comprises a multi-scale progressive fusion (MSPF) unit and a gated weight adaptive fusion (GWAF) unit, aimed at fusing bi-temporal multi-scale features to maintain boundary details and detect completely changed targets. Finally, we use the simple yet efficient UNet structure to recover the feature maps and predict results. To demonstrate the effectiveness of the MSGFNet, the LEVIR-CD, WHU-CD, and SYSU-CD datasets were utilized, and the MSGFNet achieved F1 scores of 90.86%, 92.46%, and 80.39% on the three datasets, respectively. Furthermore, the low computational costs and small model size have validated the superior performance of the MSGFNet.

Gated Fusion Network With Progressive Modality Enhancement for Sentiment Analysis

Gated Fusion Network With Progressive Modality Enhancement for Sentiment Analysis

Fake News Detection Based on Cross-Modal Message Aggregation and Gated Fusion Network

Fake News Detection Based on Cross-Modal Message Aggregation and Gated Fusion Network

STGNet: Short‐term residential load forecasting with spatial–temporal gated fusion network

AbstractWith the continuous transformation of the global power system, ensuring the flexibility and stability of the power system has become a direction that the global energy industry has been striving for. This puts forward higher requirements for load forecasting, especially high‐precision short‐term residential load forecasting. However, the high volatility and uncertainty of electric consumption make this problem challenging. Existing methods usually focus on capturing temporal correlations and ignore the underlying spatial correlations in historical loads, which is crucial for accurate forecasting. In this paper, we propose a novel spatial–temporal gated fusion network (STGNet) for short‐term residential load forecasting including not only individual residential load but also aggregated load. By seeking both adaptive neighbors and temporal similarity neighbors, we construct a dynamic graph based on a data‐driven method. On this basis, we provide an adaptive graph gated fusion mechanism to capture fine‐grained node‐specific spatial dependence more accurately and comprehensively. Finally, a bidirectional gated recurrent unit (BiGRU) is utilized to learn the temporal dependence of load data. Extensive experiments on real‐world datasets are conducted and demonstrate the superiority of STGNet over the existing approaches.

Image expression of time series data of wearable IMU sensor and fusion classification of gymnastics action

Human activity recognition (HAR) based on multimodal wearable motion sensors is a valuable technology, and gymnastics action recognition is one of its important application scenarios. However, this field faces challenges such as excessive sensor node deployment, hindering effective utilization of time-series sensor data, and low accuracy in action recognition. To address these issues, this paper introduces a gymnastics action recognition system comprising a hardware setup and an action recognition framework that combines a time series data imaging method with a novel three-channel convolutional model ViTGS. Firstly, a cost-effective single-node sensor system equipped with a 6-axis IMU is devised to efficiently capture gymnastics action data. Secondly, three one-dimensional time series data image encoding methods, Gramian Angular Field (GAF), Recurrence Plot (RP), and Markov Transition Field (MTF), are employed to capture the temporal features, phase space features, and state transition features of the data respectively. Finally, the ViTGS model is proposed, which uses the ViT_B/16 network to extract the features of each channel image, and uses Gated Fusion Network (GFN) to realize the effective fusion of different channel features and uses SVM as the recognition layer to further improve the classification accuracy. The experimental results show that using single-node (left wrist) sensor data, an accuracy of 98.56% is obtained on the acceleration dataset, and an accuracy of 99.04% is obtained on the angular velocity dataset. The method of this paper is not only applicable to gymnastics action recognition but also can be generalized to other human action recognition fields.

GFNet: gated fusion network for video saliency prediction

GFNet: gated fusion network for video saliency prediction

Exploring Semantic Relations for Social Media Sentiment Analysis

With the massive social media data available online, the conventional single modality emotion classification has developed into more complex models of multimodal sentiment analysis. Most existing works simply extracted image features at a coarse level, resulting in the absence of partially detailed visual features. Besides, social media data usually contain multiple images, while existing works considered a single image case and used only one image for representing visual features. In fact, it is nontrivial to extend the single image case to the multiple images case, due to the complex relations among multiple images. To solve the above issues, in this paper, we propose a <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">G</b> ated <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</b> usion <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S</b> emantic <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R</b> elation (GFSR) network to explore semantic relations for social media sentiment analysis. In addition to inter-relations between visual and textual modalities, we also exploit intra-relations among multiple images, potentially improving the sentiment analysis performance. Specifically, we design a gated fusion network to fuse global image embeddings and the corresponding local Adjective Noun Pair (ANP) embeddings. Then, apart from textual relations and cross-modal relations, we employ the multi-head cross attention mechanism between images and ANPs to capture similar semantic contents. Eventually, the updated textual and visual representations are concatenated for the final sentiment prediction. Extensive experiments are conducted on real-world <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Yelp</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Flickr30k</i> datasets, showing that our GFSR can improve about 0.10% to 3.66% in terms of accuracy on the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Yelp</i> dataset with multiple images, and achieve the best accuracy for two classes and the best macro F1 for three classes on the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Flickr30k</i> dataset with a single image.

Sea ice image classification based on ResFPG network and heterogeneous data fusion

ABSTRACT Sea ice detection has played an important role in climate protection and strategic deployment. Due to the autonomous learning characteristics of deep learning, it has been gradually applied to the classification of remote sensing sea ice images. At present, deep learning models are mostly used in sea ice classification of single source remote sensing data. Due to the limitations of single source data and the information loss of deep learning model in the process of feature extraction layer by layer, it is inevitable to encounter a bottleneck in sea ice detection requiring fine classification. To solve the above problems, this paper proposes a sea ice image classification method based on ResNet16-feature pyramid networks-spatial pyramid pooling-gated fusion network (ResFPG) and heterogeneous data fusion. In the feature extraction part, the method uses the improved ResNet16 to extract the multi-level feature information of sea ice in synthetic aperture radar data and optical data, reducing information loss in the feature extraction process, then mines and fuses the low-level spatial information and high-level semantic information through the improved feature pyramid networks (FPN), and then collects and fuses the output features of different scales through the spatial pyramid pooling (SPP) network. In the feature fusion part, a gated feature-level fusion strategy is designed to further improve the overall classification accuracy by adaptively adjusting the feature contribution of two heterogeneous sources of data through the gated fusion network (GFN). In order to verify the effectiveness of this method, we use two sets of heterogeneous sea ice remote sensing data located in Hudson Bay area for experiments. The experimental results show that compared with other image classification methods, the proposed method fully excavates and integrates the multi-scale and multi-level features in heterogeneous data, effectively distinguishes the feature contribution, and achieves better classification results (97.14% and 95.85%).

Multi-Branch Gated Fusion Network: A Method That Provides Higher-Quality Images for the USV Perception System in Maritime Hazy Condition

Image data acquired by unmanned surface vehicle (USV) perception systems in hazy situations is characterized by low resolution and low contrast, which can seriously affect subsequent high-level vision tasks. To obtain high-definition images under maritime hazy conditions, an end-to-end multi-branch gated fusion network (MGFNet) is proposed. Firstly, residual channel attention, residual pixel attention, and residual spatial attention modules are applied in different branch networks. These attention modules are used to focus on high-frequency image details, thick haze area information, and contrast enhancement, respectively. In addition, the gated fusion subnetworks are proposed to output the importance weight map corresponding to each branch, and the feature maps of three different branches are linearly fused with the importance weight map to help obtain the haze-free image. Then, the network structure is evaluated based on the comparison with pertinent state-of-the-art methods using artificial and actual datasets. The experimental results demonstrate that the proposed network is superior to other previous state-of-the-art methods in the PSNR and SSIM and has a better visual effect in qualitative image comparison. Finally, the network is further applied to the hazy sea–skyline detection task, and advanced results are still achieved.

Nighttime Image Dehazing Based on Multi-Scale Gated Fusion Network

In this paper, we propose an efficient algorithm to directly restore a clear image from a hazy input, which can be adapted for nighttime image dehazing. The proposed algorithm hinges on a trainable neural network realized in an encoder–decoder architecture. The encoder is exploited to capture the context of the derived input images, while the decoder is employed to estimate the contribution of each input to the final dehazed result using the learned representations attributed to the encoder. The constructed network adopts a novel fusion-based strategy which derives three inputs from an original input by applying white balance (WB), contrast enhancing (CE), and gamma correction (GC). We compute pixel-wise confidence maps based on the appearance differences between these different inputs to blend the information of the derived inputs and preserve the regions with pleasant visibility. The final clear image is generated by gating the important features of the derived inputs. To train the network, we introduce a multi-scale approach to avoid the halo artifacts. Extensive experimental results on both synthetic and real-world images demonstrate that the proposed algorithm performs favorably against the state-of-the-art dehazing for nighttime images.

A Gated Fusion Network for Dynamic Saliency Prediction

Predicting saliency in videos is a challenging problem due to complex modeling of interactions between spatial and temporal information, especially when ever-changing, dynamic nature of videos is considered. Recently, researchers have proposed large-scale data sets and models that take advantage of deep learning as a way to understand what is important for video saliency. These approaches, however, learn to combine spatial and temporal features in a static manner and do not adapt themselves much to the changes in the video content. In this article, we introduce the gated fusion network for dynamic saliency (GFSalNet), the first deep saliency model capable of making predictions in a dynamic way via the gated fusion mechanism. Moreover, our model also exploits spatial and channelwise attention within a multiscale architecture that further allows for highly accurate predictions. We evaluate the proposed approach on a number of data sets, and our experimental analysis demonstrates that it outperforms or is highly competitive with the state of the art. Importantly, we show that it has a good generalization ability, and moreover, exploits temporal information more effectively via its adaptive fusion scheme.

Gated fusion network for SAO filter and inter frame prediction in Versatile Video Coding

In order to achieve higher coding efficiency, the Versatile Video Coding (VVC) standard includes several new components at the expense of an increase in decoder computational complexity. These technologies often create ringing and contouring effects on the reconstructed frames at a low bit rate and introduce blurring and distortion. To smooth those visual artifacts, the H.266/VVC framework supports four post-processing filter operations. The state-of-the-art CNN-based in-loop filters prefer to deploy multiple networks for various quantization parameters and frame resolutions, which increases training resources and subsequently becomes overhead at decoder frame reconstruction. This paper presents a single deep-learning-based model for sample adaptive off-set (SAO) non-linear filtering operation on the decoder side, uses feature correlation among adjacent frames, and substantiates the merits of intra–inter frame quality enhancement. We introduced a variable filter size dual multi-scale convolutional neural network (D-MSCNN) to attenuate the compression artifact and incorporated strided deconvolution to restore the high-frequency details on the distorted frame. Our model follows sequential training across all QP values and updates the model weights. Using data augmentation, weight fusion, and residual learning, we demonstrated that our model could be trained effectively by transferring the convolution prior feature indices to the decoder to produce a dense output map. The Objective measurements demonstrate that the proposed method outperforms the baseline VVC method in PSNR, MS-SSIM, and VMAF metrics and achieves an average of 5.16% bit rate saving on different test sequence categories.

A cross-channel multi-scale gated fusion network for recognizing construction and demolition waste from high-resolution remote sensing images

ABSTRACT Timely and accurate survey of construction and demolition waste (C&DW) distribution is of significance for managing C&DW and enhancing the quality of the urban environment, especially in rapidly urbanizing country of China. Automatic C&DW recognition using high-resolution remote sensing images is an extremely important surveying method. Existing C&DW recognition methods can only obtain a rough spatial distribution of C&DW. Mapping precise C&DW distribution on large-scale remote sensing images is still challenging. In this paper, to solve the problem of precise C&DW recognition, a novel deep learning algorithm based on high-resolution remote sensing images is proposed, which we refer to as the cross-channel multi-scale gated fusion network (CCMGNet). CCMGNet extracts depth features of C&DW from RGB and NIR bands by using two independent encoding streams. A gated fusion layer and a multi-scale attention module are designed to effectively fuse the features from two encoding streams. We manually labelled the sample set used for training and testing on 26 remote sensing images captured by the Gaofen-2 (GF-2) satellite. These images were taken in nine typical cities in China over the time period from 2019 to 2021. The results showed that the proposed method is effective in recognizing C&DW, realizing 89.42% and 84.52% in precision and IoU, and is superior to other state-of-the-art deep learning algorithms and existing C&DW recognition methods. The effectiveness of all significant components in CCMGNet was confirmed by ablation experiments. The proposed method was applied to extract C&DW regions in large-scale remote sensing images, with an example of a GF-2 image of Beijing. The high extraction efficiency and satisfactory visual effect demonstrate the potential of the proposed method for surveying the spatial distribution of C&DW. The novel method will play a crucial role in automatically surveying the spatial distribution and size of C&DW across the country.

Cross-document attention-based gated fusion network for automated medical licensing exam

One of the applications of machine-learning in the medical industry is to automatically learn knowledge from medical textbooks and transfer medical knowledge into diagnosis abilities. Because of complex nature of medical issues, the learning process usually requires multiple knowledge documents to form a comprehensive reasoning chain for diagnosis, which increases the difficulty of the automatic learning process. Existing models for multiple document comprehension either concatenate multiple documents together for inference or reason on every document independently. In this paper, we propose a Co-Attention-based Multi-document Inference (CAMI) framework for better reasoning over multiple documents. The proposed framework makes use of not only the attentional information among questions, answers and support documents but also the complementary attentional information across different documents. In addition, a gated fusion network is designed to fuse the cross-document information. The proposed model outperforms the state-of-the-art methods on Chinese National Medical Licensing Examination (CNMLE) dataset, ClinicQA, which contains 27,432 plain text documents and 13,827 CNMLE questions. We intend to make it publicly available as the first clinical OpenQA dataset.

CEGFNet: Common Extraction and Gate Fusion Network for Scene Parsing of Remote Sensing Images

Scene parsing of high spatial resolution (HSR) remote sensing images has achieved notable progress in recent years by the adoption of convolutional neural networks. However, for scene parsing of multimodal remote sensing images, effectively integrating complementary information remains challenging. For instance, the decrease in feature map resolution through a neural network causes loss of spatial information, likely leading to blurred object boundaries and misclassification of small objects. In addition, object scales on a remote sensing image vary substantially, undermining the parsing performance. To solve these problems, we propose an end-to-end common extraction and gate fusion network (CEGFNet) to capture both high-level semantic features and low-level spatial details for scene parsing of remote sensing images. Specifically, we introduce a gate fusion module to extract complementary features from spectral data and digital surface model data. A gate mechanism removes redundant features in the data stream and extracts complementary features that improve multimodal feature fusion. In addition, a global context module and a multilayer aggregation decoder handle scale variations between objects and the loss of spatial details due to downsampling, respectively. The proposed CEGFNet was quantitatively evaluated on benchmark scene parsing datasets containing HSR remote sensing images, and it achieved state-of-the-art performance.

CMGFNet: A deep cross-modal gated fusion network for building extraction from very high-resolution remote sensing images

The extraction of urban structures such as buildings from very high-resolution (VHR) remote sensing imagery has improved dramatically, thanks to recent developments in deep multimodal fusion models. However, Due to the variety of colour intensities with complex textures of building objects in VHR images and the low quality of the digital surface model (DSM), it is challenging to develop the optimal cross-modal fusion network that takes advantage of these two modalities. This research presents an end-to-end cross-modal gated fusion network (CMGFNet) for extracting building footprints from VHR remote sensing images and DSMs data. The CMGFNet extracts multi-level features from RGB and DSM data by using two separate encoders. We offer two methods for fusing features in two modalities: Cross-modal and multi-level feature fusion. For cross-modal feature fusion, a gated fusion module (GFM) is proposed to combine two modalities efficiently. The multi-level feature fusion fuses the high-level features from deep layers with shallower low-level features through a top-down strategy. Furthermore, a residual-like depth-wise separable convolution (R-DSC) is introduced to enhance the performance of the up-sampling process and decrease the parameters and time complexity in the decoder section. Experimental results from challenging datasets show that the CMGFNet outperforms other state-of-the-art models. The efficacy of all significant elements is also confirmed by the extensive ablation study.