Decoder Module Research Articles

Coarse Mask Guided Interactive Object Segmentation.

Interactive object segmentation aims to produce object masks with user interactions, such as clicks, bounding boxes, and scribbles. Click point is the most popular interactive cue for its efficiency, and related deep learning methods have attracted lots of interest in recent years. Most works encode click points as gaussian maps and concatenate them with images as the model's input. However, the spatial and semantic information of gaussian maps would be noised through multiple convolution layers and won't be fully exploited by top layers for mask prediction. To pass click information to top layers exactly and efficiently, we propose a coarse mask guided model (CMG) which predicts coarse masks with a coarse module to guide the object mask prediction. Specifically, the coarse module encodes user clicks as query features and enriches their semantic information with backbone features through transformer layers, coarse masks are generated based on the enriched query feature and fed into CMG's decoder. Benefiting from the efficiency of transformer, CMG's coarse module and decoder module are lightweight and computationally efficient, making the interaction process more smooth. Experiments on several segmentation benchmarks demonstrate the effectiveness of our method, and we get new state-of-the-art results compared with previous works.

Direction Consistency‐Guided Lightweight Power Line Detection Network for Aerial Images

Accurate detection of power lines in aerial images is of great significance in ensuring grid security. However, complex power line scenarios and the thin and light structure of power lines both make it difficult to detect power lines accurately. Most of the existing approaches use traditional deep learning methods, using networks with a large number of parameters, computation, and memory occupation, thus making them not lightweight enough to perform on mobile devices. Based on this, a lightweight power line detection network based on direction consistency and location attention is proposed. The network is designed with a coordinate‐aware feature extraction layer, which performs feature extraction by four‐layer stacking to achieve faster inference speed while ensuring the network has fewer parameters. This layer is also able to sense the coordinates of the center pixel of the convolution in the image during the convolution process, thus preserving the location information of the power lines. In order to enhance the power line representation, a two‐stage context‐guided module is later utilized to simultaneously learn local features, surrounding context, and global context. Then, the features are input into a Gaussian kernel estimation module and features are aggregated in the corresponding directions through Gaussian kernels of eight different directions. The main directions of the power lines in the image and the corresponding Gaussian convolution kernels are obtained by filtering the feature responses. In addition, a kernel‐guided decoder module is proposed to take advantage of the estimated power line features in the main direction of Gaussian kernel aggregation. This module can effectively enhance the power line representation and maintain the continuity of power lines. Meanwhile, low‐level features are introduced to recover the edge details to realize high performance in distinguishing dense power lines. Both ablation experiments and comparison experiments on the transmission towers and power lines aerial‐image and Power Line Aerial Image Dataset show that the proposed power line detection network has a good segmentation performance in complex scenarios. The proposed method performs the best in the comparison experiments, improving over the suboptimal method by 3.51% on average for the max F‐measure metric.

Center Weighted Convolution and GraphSAGE Cooperative Network for Hyperspectral Image Classification

Hyperspectral image (HSI) classification is one of the basic tasks of remote sensing image processing, which is to predict the label of each HSI pixel. Convolution neural network (CNN) and graph convolution neural network (GCN) have become the current research focus due to their outstanding performance in the field of HSI classification in recent years. However, GCN is a transductive learning method, which needs all nodes to participate in the training process to get the node embedding. Graph sample and aggregation (GraphSAGE) is an important branch of graph neural network, which can flexibly aggregate new neighbor nodes in non-Euclidean data of any structure, and capture long-range contextual relationships. Superpixel-based GraphSAGE can not only integrate the global spatial relationship of data, but also further reduce its computing cost. CNN can extract pixel-level features in a small area, and our center attention module (CAM) and center weighted convolution (CW-Conv) can also improve the feature extraction ability of CNN by enhancing the dominant position of target pixels. In order to make full use of the advantages of CNN and GraphSAGE, we propose a center weighted convolution and GraphSAGE (CW-SAGE) cooperative network for HSI classification. Specifically, graph simple and aggregate branch is constructed by superpixel-based encoder and decoder modules, then pixel-level features are extracted by a central attention convolutional neural network. Finally, the features of the two branches are spliced together for feature fusion. We conduct experiments on three hyperspectral datasets and compare the results with other current state-of-the-art methods. A series of experiments demonstrate the advantages of our method.

Building Extraction From Very High-Resolution Remote Sensing Images Using Refine-UNet

Accurate building extraction from very high-resolution (VHR) remote sensing images plays an important role in urban dynamic monitoring, planning, and management. However, it is still a challenging task to achieve building extraction with high accuracy and integrity due to diverse building appearances and more complex ground background in VHR remote sensing images. Recently, Unity Networking (UNet) has been proven to be capable of feature extraction and semantic segmentation of remote sensing images. However, UNet cannot achieve sufficient multi-scale and multi-level features with larger receptive fields. To address these problems, an improved network based on UNet structure (Refine-UNet) is proposed for extracting buildings from the VHR images. The proposed Refine-UNet mainly consists of encoder module, decoder module, and refine skip connection scheme. The refine skip connection scheme is composed of an atrous spatial convolutional pyramid pooling (ASPP) module and several improved depthwise separable convolution (IDSC) modules. Experimental results on the Jilin-1 VHR datasets with a spatial resolution of 0.75 m demonstrate that compared with UNet, PSPNet, DeepLabV3+, and SegNet, the proposed Refine-UNet can obtain more accurate building extraction results and achieve the best Precision of 95.1% and intersection over union (IoU) of 87.0%, respectively, indicating the great practical potential.

Arbitrary-Shape Scene Text Detection via Visual-Relational Rectification and Contour Approximation

One trend in the latest bottom-up approaches for arbitrary-shape scene text detection is to determine the links between text segments using Graph Convolutional Networks (GCNs). However, the performance of these bottom-up methods is still inferior to that of state-of-the-art top-down methods even with the help of GCNs. We argue that a cause of this is that bottom-up methods fail to make proper use of visual-relational features, which results in accumulated false detection, as well as the error-prone route-finding used for grouping text segments. In this paper, we improve classic bottom-up text detection frameworks by fusing the visual-relational features of text with two effective false positive/negative suppression (FPNS) mechanisms and developing a new shape-approximation strategy. First, dense overlapping text segments depicting the ‘`characterness’' and ‘`streamline’' properties of text are constructed and used in weakly supervised node classification to filter the falsely detected text segments. Then, relational features and visual features of text segments are fused with a novel Location-Aware Transfer (LAT) module and Fuse Decoding (FD) module to jointly rectify the detected text segments. Finally, a novel multiple-text-map-aware contour-approximation strategy is developed based on the rectified text segments, instead of the error-prone route-finding process, to generate the final contour of the detected text. Experiments conducted on five benchmark datasets demonstrate that our method outperforms the state-of-the-art performance when embedded in a classic text detection framework, which revitalizes the strengths of bottom-up methods.

Human Motion Segmentation via Velocity-Sensitive Dual-Side Auto-Encoder.

Human motion segmentation (HMS) aims to segment a long human action video into a bunch of short and meaningful action clips. Existing supervised learning approaches need a large amount of training data which may be costly in real-world scenario, while most unsupervised clustering methods cannot fully explore the temporal correlations among human motions and hard to achieve promising performances. In our paper, we design a novel unsupervised framework, called Velocity-Sensitive Dual-Side Auto-Encoder (VSDA), for HMS task. Specifically, a multi-neighbor auto-encoder (MNA) is proposed to extract informative temporal features, which fully explores the local temporal patterns of human motions. In addition, a long-short distance encoding (LSE) strategy is designed. It constrains the encoded representations of close (short-distance) frames becoming similar while the representations of far-away (long-distance) frames becoming distinctive. Similarly, this strategy is also deployed on the decoded outputs as the long-short distance decoding (LSD) module. The LSE/LSD guides the learning process explicitly and implicitly to achieve the dual-side structure. Moreover, we consider the energy variations during the human motion to propose the velocity-sensitive (VS) guidance mechanism for further model improvement. VSDA leverages the temporal characteristics of human motion and derives promising HMS performance. Comprehensive experiments on six real-world human motion datasets illustrate the effectiveness of our proposed model.

Future Event Prediction Based on Temporal Knowledge Graph Embedding

Accurate prediction of future events brings great benefits and reduces losses for society in many domains, such as civil unrest, pandemics, and crimes. Knowledge graph is a general language for describing and modeling complex systems. Different types of events continually occur, which are often related to historical and concurrent events. In this paper, we formalize the future event prediction as a temporal knowledge graph reasoning problem. Most existing studies either conduct reasoning on static knowledge graphs or assume knowledges graphs of all timestamps are available during the training process. As a result, they cannot effectively reason over temporal knowledge graphs and predict events happening in the future. To address this problem, some recent works learn to infer future events based on historical event-based temporal knowledge graphs. However, these methods do not comprehensively consider the latent patterns and influences behind historical events and concurrent events simultaneously. This paper proposes a new graph representation learning model, namely Recurrent Event Graph ATtention Network (RE-GAT), based on a novel historical and concurrent events attention-aware mechanism by modeling the event knowledge graph sequence recurrently. More specifically, our RE-GAT uses an attention-based historical events embedding module to encode past events, and employs an attention-based concurrent events embedding module to model the associations of events at the same timestamp. A translation-based decoder module and a learning objective are developed to optimize the embeddings of entities and relations. We evaluate our proposed method on four benchmark datasets. Extensive experimental results demonstrate the superiority of our RE-GAT model comparing to various baselines, which proves that our method can more accurately predict what events are going to happen.

Jointly Fusing Multi-Scale Spatial-Logical Brain Networks: A Neural Decoding Method.

Functional magnetic resonance imaging (fMRI) is a methodology for measuring human brain activities. It has become more and more popular in neural decoding due to its noninvasive. Neural decoding aims to establishing models to reconstruct external stimuli or features of stimuli from known brain responses, so that we can understand the principles of brain functions such as emotion, cognition and language. Neural decoding based on fMRI is of great significance for further understanding the mechanism of brain operation. Most existing studies take multi-scale topology information of brain networks obtained from fMRI into account in neural decoding. However, they always ignore the simultaneous modeling of network structure and hemodynamic response, thus leading to information loss. In addition, current multi-scale methods usually only utilize spatial or logical reasoning relationship of brain networks, which brings challenge to precise neural decoding. In this work, we present a novel and robust multi-scale spatial and logical reasoning learning framework (MSLR) for fMRI-based neural decoding. Specifically, we first design graph signal wavelet generation module to combine brain network topology and node information to construct multi-scale representation of brain networks in a local to global manner. Then, we develop multi-scale information fusion module that can simultaneously model the spatial and logical reasoning relationship of brain networks, it can also learn discriminative multi-scale features with brain state transition. Finally, we construct a neural decoding module to predict the brain states. We evaluated the framework on the public Human Connectome Project (HCP) dataset that included 986 participants. The experimental results with support vector machine (SVM) outperform current state-of-the-art methods on four evaluation metrics (accuracy: 91.58, kappa coefficient: 0.883, macro F1: 0.865 and hamming distance: 0.105) under 19 different stimuli spanning 7 different cognitive tasks. The interpretation of the learned multi-scale representation replicates neuroscientific findings from previous fMRI studies and renews the multi-scale information flow pattern of brain network in neural decoding.

End-to-End Handwritten Paragraph Text Recognition Using a Vertical Attention Network.

Unconstrained handwritten text recognition remains challenging for computer vision systems. Paragraph text recognition is traditionally achieved by two models: the first one for line segmentation and the second one for text line recognition. We propose a unified end-to-end model using hybrid attention to tackle this task. This model is designed to iteratively process a paragraph image line by line. It can be split into three modules. An encoder generates feature maps from the whole paragraph image. Then, an attention module recurrently generates a vertical weighted mask enabling to focus on the current text line features. This way, it performs a kind of implicit line segmentation. For each text line features, a decoder module recognizes the character sequence associated, leading to the recognition of a whole paragraph. We achieve state-of-the-art character error rate at paragraph level on three popular datasets: 1.91% for RIMES, 4.45% for IAM and 3.59% for READ 2016. Our code and trained model weights are available at https://github.com/FactoDeepLearning/VerticalAttentionOCR.

MISNet: Multiscale Cross-Layer Interactive and Similarity Refinement Network for Scene Parsing of Aerial Images

Although progress has been made in multisource data scene parsing of natural scene images, extracting complex backgrounds from aerial images of various types and presenting the image at different scales remain challenging. Various factors in high-resolution aerial images (HRAIs), such as imaging blur, background clutter, object shadow, and high resolution, substantially reduce the integrity and accuracy of object segmentation. By applying multisource data fusion, as in scene parsing of natural scene images, we can solve the aforementioned problems through the integration of auxiliary data into HRAIs. To this end, we propose a multiscale cross-layer interactive and similarity refinement network (MISNet) for scene parsing of HRAIs. First, in a feature fusion optimization module, we extract, filter, and optimize multisource features and further guide and optimize the features using a feature guidance module. Second, a multiscale context aggregation module increases the receptive field, captures semantic information, and extracts rich multiscale background features. Third, a dense decoding module fuses the global guidance information and high-level fused features. We also propose a joint learning method based on feature similarity and a joint learning module to obtain deep multilevel information, enhance feature generation, and fuse multiscale and global features to enhance network representation for accurate scene parsing of HRAIs. Comprehensive experiments on two benchmark HRAIs datasets indicate that our proposed MISNet is qualitatively and quantitatively superior to similar state-of-the-art models.

A Thorough Benchmark and a New Model for Light Field Saliency Detection.

Compared with current RGB or RGB-D saliency detection datasets, those for light field saliency detection often suffer from many defects, e.g., insufficient data amount and diversity, incomplete data formats, and rough annotations, thus impeding the prosperity of this field. To settle these issues, we elaborately build a large-scale light field dataset, dubbed PKU-LF, comprising 5,000 light fields and covering diverse indoor and outdoor scenes. Our PKU-LF provides all-inclusive representation formats of light fields and offers a unified platform for comparing algorithms utilizing different input formats. For sparking new vitality in saliency detection tasks, we present many unexplored scenarios (such as underwater and high-resolution scenes) and the richest annotations (such as scribble annotations, bounding boxes, object-/instance-level annotations, and edge annotations), on which many potential attention modeling tasks can be investigated. To facilitate the development of saliency detection, we systematically evaluate and analyze 16 representative 2D, 3D, and 4D methods on four existing datasets and the proposed dataset, furnishing a thorough benchmark. Furthermore, tailored to the distinct structural characteristics of light fields, a novel symmetric two-stream architecture (STSA) network is proposed to predict the saliency of light fields more accurately. Specifically, our STSA incorporates a focalness interweavement module (FIM) and three partial decoder modules (PDM). The former is designed to efficiently establish long-range dependencies across focal slices, while the latter aims to effectively aggregate the extracted coadjutant features in a mutual-enhancement way. Extensive experiments demonstrate that our method can significantly outperform the competitors.

A construction schedule management method of large-scale construction project based on BIM model

In order to overcome the problems of long response time and small number of manageable indicators existing in traditional construction project schedule management methods, a new construction schedule management method based on BIM model is designed in this paper. The construction progress data acquisition and decoding module circuit is set to complete the construction progress data acquisition, and the K-means algorithm is used to preprocess the construction progress data. First, the construction project progress is decomposed; then, the large-scale construction project is divided into different progress management levels by the WBS analysis method; a functional information module is established; the construction project progress data is imported into the BIM model; and finally, the BIM information function management of the method is realised. The experimental results show that the proposed method has low response time and multiple schedule management indicators, and the shortest response time of the proposed method is only 1.1 s.

Rice Plant Counting, Locating, and Sizing Method Based on High-Throughput UAV RGB Images.

Rice plant counting is crucial for many applications in rice production, such as yield estimation, growth diagnosis, disaster loss assessment, etc. Currently, rice counting still heavily relies on tedious and time-consuming manual operation. To alleviate the workload of rice counting, we employed an UAV (unmanned aerial vehicle) to collect the RGB images of the paddy field. Then, we proposed a new rice plant counting, locating, and sizing method (RiceNet), which consists of one feature extractor frontend and 3 feature decoder modules, namely, density map estimator, plant location detector, and plant size estimator. In RiceNet, rice plant attention mechanism and positive-negative loss are designed to improve the ability to distinguish plants from background and the quality of the estimated density maps. To verify the validity of our method, we propose a new UAV-based rice counting dataset, which contains 355 images and 257,793 manual labeled points. Experiment results show that the mean absolute error and root mean square error of the proposed RiceNet are 8.6 and 11.2, respectively. Moreover, we validated the performance of our method with two other popular crop datasets. On these three datasets, our method significantly outperforms state-of-the-art methods. Results suggest that RiceNet can accurately and efficiently estimate the number of rice plants and replace the traditional manual method.

Language Model Agnostic Gray-Box Adversarial Attack on Image Captioning

Adversarial susceptibility of neural image captioning is still under-explored due to the complex multi-model nature of the task. We introduce a GAN-based adversarial attack to effectively fool encoder-decoder based image captioning frameworks. Unique to our attack is the systematic disruption of the internal representation of an image at the encoder stage which allows control over the captions generated at the decoder stage. We cause the desired disruption with an input perturbation that promotes similarity between the features of the input image with a target image of our choice. The target image provides a convenient handle to control the incorrect captions in our method. We do not assume any knowledge of the decoder module, which makes our attack ‘gray-box’. Moreover, our attack remains agnostic to the type of decoder module, thereby proving effective for RNNs as well as Transformers as the language models. This makes our attack highly pragmatic.

Image Segmentation of Intestinal Polyps using Attention Mechanism based on Convolutional Neural Network

The intestinal polyp is one of the common intestinal diseases, which is characterized by protruding lining tissue of the colon or rectum. Considering that they may become cancerous, they should be removed by surgery as soon as possible. In the past, it took a lot of manpower and time to identify and diagnose intestinal polyps, which greatly affected the treatment efficiency of medical staff. Because the polyp part looks similar to the normal structure of the human body, the probability of human eye misjudgment is high. Therefore, it is necessary to use advanced computer technology to segment the intestinal polyp image. In the model established in this paper, an image segmentation method based on convolution neural network is proposed. The Har-DNet backbone network is used as the encoder in the model, and its feature processing results are converted into three feature images of different sizes, which are input to the decoding module. In the decoding process, each output first expands the receptive field module and then fuses the feature image processed by the attention mechanism. The fusion results are input to the density aggregation module for processing to improve the operation efficiency and accuracy of the model. The experimental results show that compared with the previous Pra-Net model and Har-DNet MSEG model, the accuracy and precision of this method are greatly improved, and can be applied to the actual medical image recognition process, thus improving the treatment efficiency of patients.

Joint transformer progressive self‐calibration network for low light enhancement

AbstractWhen the lighting conditions are poor and the environmental light is weak, the image captured by the imaging device often has lower brightness and is accompanied by a lot of noise. The paper designs a progressive self‐calibration network model (PSCNet) for recovering high‐quality low‐light‐enhanced images. First, shallow features in low‐light images can be better focused and extracted with the help of attention mechanism. Next, the feature mapping is passed to the encoder and decoder modules, where the transformer and encoder‐decoder jump connection structures can be better combined with the semantic information of the context to learn rich deep feature information. Finally, the self‐calibration module can adaptively cascade the features decoded by the decoder and input them into the residual attention module quickly and accurately. Meanwhile, the LBP features of the image are also fused into the feature information of the residual attention module to enhance the detailed texture information of the image. Qualitative analysis and quantitative comparison of a large number of experimental results show that this method outperforms existing methods.

MSSTNet: A Multi-Scale Spatiotemporal Prediction Neural Network for Precipitation Nowcasting

Convolution-based recurrent neural networks and convolutional neural networks have been used extensively in spatiotemporal prediction. However, these methods tend to concentrate on fixed-scale spatiotemporal state transitions and disregard the complexity of spatiotemporal motion. Through statistical analysis, we found that the distribution of the spatiotemporal sequence and the variety of spatiotemporal motion state transitions exhibit some regularity. In light of these statistics and observations, we propose the Multi-scale Spatiotemporal Neural Network (MSSTNet), an end-to-end neural network based on 3D convolution. It can be separated into three major child modules: a distribution feature extraction module, a multi-scale motion state capture module, and a feature decoding module. Furthermore, the MSST unit is designed to model multi-scale spatial and temporal information in the multi-scale motion state capture module. We first conduct the experiments on the MovingMNIST dataset, which is the most commonly used dataset in the field of spatiotemporal prediction, MSSTNet can achieve state-of-the-art results for this dataset, and ablation experiments demonstrate that the MSST unit has positive significance for spatiotemporal prediction. In addition, this paper applies the model to valuable precipitation nowcasting, due to efficiently capturing the multi-scale information of distribution and motion, the new MSSTNet model can predict the real-world radar echo more accurately.

SPNet: A novel deep neural network for retinal vessel segmentation based on shared decoder and pyramid-like loss

Segmentation of retinal vessel images is critical to the diagnosis of retinopathy. Recently, convolutional neural networks have shown significant ability to extract the blood vessel structure. However, it remains challenging to refined segmentation for the capillaries and the edges of retinal vessels due to thickness inconsistencies and blurry boundaries. In this paper, we propose a novel deep neural network for retinal vessel segmentation based on shared decoder and pyramid-like loss (SPNet) to address the above problems. Specifically, we introduce a decoder-sharing mechanism to capture multi-scale semantic information, where feature maps at diverse scales are decoded through a sequence of weight-sharing decoder modules. Also, to strengthen characterization on the capillaries and the edges of blood vessels, we define a residual pyramid architecture which decomposes the spatial information in the decoding phase. A pyramid-like loss function is designed to compensate possible segmentation errors progressively. Experimental results on public benchmarks show that the proposed method outperforms the backbone network and most state-of-the-art methods, especially in the regions of the capillaries and the vessel contours. In addition, performances on cross-datasets verify that SPNet shows stronger generalization ability.

VieCap4H Challenge 2021: A transformer-based method for Healthcare Image Captioning in Vietnamese

The automatic image caption generation is attractive to both Computer Vision and Natural Language Processing research community because it lies in the gap between these two fields. Within the vieCap4H contest organized by VLSP 2021, we participate and present a Transformer-based solution for image captioning in the healthcare domain. In detail, we use grid features as visual presentation and pre-training a BERT-based language model from PhoBERT-base pre-trained model to obtain language presentation used in the Adaptive Decoder module in the RSTNet model. Besides, we indicate a suitable schedule with the self-critical training sequence (SCST) technique to achieve the best results. Through experiments, we achieve an average of 30.3% BLEU score on the public-test round and 28.9% on the private-test round, which ranks 3rd and 4th, respectively. Source code is available at https://github.com/caodoanh2001/uit-vlsp-viecap4h-solution.

A lightweight multi-dimension dynamic convolutional network for real-time semantic segmentation.

Semantic segmentation can address the perceived needs of autonomous driving and micro-robots and is one of the challenging tasks in computer vision. From the application point of view, the difficulty faced by semantic segmentation is how to satisfy inference speed, network parameters, and segmentation accuracy at the same time. This paper proposes a lightweight multi-dimensional dynamic convolutional network (LMDCNet) for real-time semantic segmentation to address this problem. At the core of our architecture is Multidimensional Dynamic Convolution (MDy-Conv), which uses an attention mechanism and factorial convolution to remain efficient while maintaining remarkable accuracy. Specifically, LMDCNet belongs to an asymmetric network architecture. Therefore, we design an encoder module containing MDy-Conv convolution: MS-DAB. The success of this module is attributed to the use of MDy-Conv convolution, which increases the utilization of local and contextual information of features. Furthermore, we design a decoder module containing a feature pyramid and attention: SC-FP, which performs a multi-scale fusion of features accompanied by feature selection. On the Cityscapes and CamVid datasets, LMDCNet achieves accuracies of 73.8 mIoU and 69.6 mIoU at 71.2 FPS and 92.4 FPS, respectively, without pre-training or post-processing. Our designed LMDCNet is trained and inferred only on one 1080Ti GPU. Our experiments show that LMDCNet achieves a good balance between segmentation accuracy and network parameters with only 1.05 M.