Hierarchical Decoder Research Articles

IVA-former: invisible–visible query guided amodal mask measurement network for desktop object via hierarchical transformer

Abstract Instance segmentation of desktop objects is important for service robots. Most of the previous works for desktop environments are restricted to measuring the visible area of target objects. However, when a target object is placed behind another, the algorithm that only performs visible area segmentation is unable to provide accurate appearance information for the occluded objects. To solve this problem, we propose the invisible–visible query guided amodal mask measurement network based on a hierarchical transformer for desktop scenes, which can perceive the entire appearance of objects in the presence of occlusions. In this method, the RGB-D backbone is adopted to fuse the features from both RGB and depth images. Then, the pixel decoder is used to generate multi-scale feature maps. We then adopt a hierarchical transformer decoder to predict invisible, visible, and amodal masks simultaneously. To enhance the associations between the three prediction branches, we propose a query transform module to transfer object queries between adjacent branches. Since amodal masks are a combination of invisible and visible masks, we propose an invisible–visible mixture loss that takes masks from both invisible and visible branches to further supervise the network. Our method is trained on synthetic datasets for desktop objects and evaluated on both visible and amodal real-world datasets. Compared to other recent segmentation algorithms, our method achieves competitive performance.

DeMambaNet: Deformable Convolution and Mamba Integration Network for High-Precision Segmentation of Ambiguously Defined Dental Radicular Boundaries.

The incorporation of automatic segmentation methodologies into dental X-ray images refined the paradigms of clinical diagnostics and therapeutic planning by facilitating meticulous, pixel-level articulation of both dental structures and proximate tissues. This underpins the pillars of early pathological detection and meticulous disease progression monitoring. Nonetheless, conventional segmentation frameworks often encounter significant setbacks attributable to the intrinsic limitations of X-ray imaging, including compromised image fidelity, obscured delineation of structural boundaries, and the intricate anatomical structures of dental constituents such as pulp, enamel, and dentin. To surmount these impediments, we propose the Deformable Convolution and Mamba Integration Network, an innovative 2D dental X-ray image segmentation architecture, which amalgamates a Coalescent Structural Deformable Encoder, a Cognitively-Optimized Semantic Enhance Module, and a Hierarchical Convergence Decoder. Collectively, these components bolster the management of multi-scale global features, fortify the stability of feature representation, and refine the amalgamation of feature vectors. A comparative assessment against 14 baselines underscores its efficacy, registering a 0.95% enhancement in the Dice Coefficient and a diminution of the 95th percentile Hausdorff Distance to 7.494.

Bridging the Domain Gap in Scene Flow Estimation via Hierarchical Smoothness Refinement

This article introduces SmoothFlowNet3D, an innovative encoder-decoder architecture specifically designed for bridging the domain gap in scene flow estimation. To achieve this goal, SmoothFlowNet3D divides the scene flow estimation task into two stages: initial scene flow estimation and smoothness refinement. Specifically, SmoothFlowNet3D comprises a hierarchical encoder that extracts multi-scale point cloud features from two consecutive frames, along with a hierarchical decoder responsible for predicting the initial scene flow and further refining it to achieve smoother estimation. To generate the initial scene flow, a cross-frame nearest-neighbor search operation is performed between the features extracted from two consecutive frames, resulting in forward and backward flow embeddings. These embeddings are then combined to form the bidirectional flow embedding, serving as input for predicting the initial scene flow. Additionally, a flow smoothing module based on the self-attention mechanism is proposed to predict the smoothing error and facilitate the refinement of the initial scene flow for more accurate and smoother estimation results. Extensive experiments demonstrate that the proposed SmoothFlowNet3D approach achieves state-of-the-art performance on both synthetic datasets and real LiDAR point clouds, confirming its effectiveness in enhancing scene flow smoothness.

Floorplanning with Edge-aware Graph Attention Network and Hindsight Experience Replay

In this article, we focus on chip floorplanning, which aims to determine the location and orientation of circuit macros simultaneously, so the chip area and wirelength are minimized. As the highest level of abstraction in hierarchical physical design, floorplanning bridges the gap between the system-level design and the physical synthesis, whose quality directly influences downstream placement and routing. To tackle chip floorplanning, we propose an end-to-end reinforcement learning (RL) methodology with a hindsight experience replay technique. An edge-aware graph attention network (EAGAT) is developed to effectively encode the macro and connection features of the netlist graph. Moreover, we build a hierarchical decoder architecture mainly consisting of transformer and attention pointer mechanism to output floorplan actions. Since the RL agent automatically extracts knowledge about the solution space, the previously learned policy can be quickly transferred to optimize new unseen netlists. Experimental results demonstrate that, compared with state-of-the-art floorplanners, the proposed end-to-end methodology significantly optimizes area and wirelength on public GSRC and MCNC benchmarks.

Distraction-aware hierarchical learning for vascular structure segmentation in intravascular ultrasound images

Vascular structure segmentation in intravascular ultrasound (IVUS) images plays an important role in pre-procedural evaluation of percutaneous coronary intervention (PCI). However, vascular structure segmentation in IVUS images has the challenge of structure-dependent distractions. Structure-dependent distractions are categorized into two cases, structural intrinsic distractions and inter-structural distractions. Traditional machine learning methods often rely solely on low-level features, overlooking high-level features. This way limits the generalization of these methods. The existing semantic segmentation methods integrate low-level and high-level features to enhance generalization performance. But these methods also introduce additional interference, which is harmful to solving structural intrinsic distractions. Distraction cue methods attempt to address structural intrinsic distractions by removing interference from the features through a unique decoder. However, they tend to overlook the problem of inter-structural distractions. In this paper, we propose distraction-aware hierarchical learning (DHL) for vascular structure segmentation in IVUS images. Inspired by distraction cue methods for removing interference in a decoder, the DHL is designed as a hierarchical decoder that gradually removes structure-dependent distractions. The DHL includes global perception process, distraction perception process and structural perception process. The global perception process and distraction perception process remove structural intrinsic distractions then the structural perception process removes inter-structural distractions. In the global perception process, the DHL searches for the coarse structural region of the vascular structures on the slice of IVUS sequence. In the distraction perception process, the DHL progressively refines the coarse structural region of the vascular structures to remove structural distractions. In the structural perception process, the DHL detects regions of inter-structural distractions in fused structure features then separates them. Extensive experiments on 361 subjects show that the DHL is effective (e.g., the average Dice is greater than 0.95), and superior to ten state-of-the-art IVUS vascular structure segmentation methods.

Visual-guided scene-aware audio generation method based on hierarchical feature codec and rendering decision

Visually guided spatial sound generation (VGSSG) is a well-suited multimodal learning method for dealing with recorded videos. However, existing methods are difficult to be directly applied to spatial sound generation for movie clips. This is mainly due to (1) the existence of Cinematic Audiovisual Language (CAL) in movies, which makes it difficult to construct spatial sound mapping models directly through data-driven based methods. (2) The problem of the inadequate model performance, which is caused by the excessive heterogeneous gap between audiovisual modal information. To solve the aforementioned problems, we propose a VGSSG method based on CAL decision-making and hierarchical feature coding and decoding, which effectively accomplishes spatial sound generation based on the CAL of movies. Specifically, to solve the problem of CAL modeling, a multimodal information-guided movie audio rendering decision maker is established, which can decide the rendering strategy based on the CAL of the current clip. To narrow the heterogeneous gap that hinders the fusion between audiovisual modal data, we propose a codec structure based on hierarchical fusion of audiovisual features and full-scale skip-connections, which improves the efficiency of the comprehensive utilization of audiovisual modal data, and demonstrates the effectiveness of adopting shallow features in VGSSG task. We integrate both 2-channel and 6-channel spatial audio generation into a unified framework. In addition, we establish a movie audiovisual bimodal dataset with hand-crafted CAL annotations. Experimentally, we demonstrate that compared with the existing methods, our method has higher performance in terms of reducing generation distortion.

Data repair accelerating scheme for erasure-coded storage system based on FPGA and hierarchical parallel decoding structure

Data repair accelerating scheme for erasure-coded storage system based on FPGA and hierarchical parallel decoding structure

A SURVEY ON SUMMARIZATION OF LEGAL TEXTS

The task of summarizing legal texts is of paramount importance in facilitating access to legal knowledge. This survey paper critically reviews and analyzes diverse methodologies applied to the challenging task of summarizing legal texts. In the supervised realm, models such as GIST, SummaRuNNer, and BERTSUM have gained prominence, demonstrating prowess in extractive summarization techniques. Semi-supervised methods, leveraging innovative techniques such as TF-IDF Scores and Sentence Embeddings, have proven effective in handling the intricate structure of court judgments. Unsolicited approaches, CaseSummarizer and exemplified by USLT-LEGAL-BERT, address the challenges of summarization without the luxury of labeled data, showcasing their adaptability to diverse legal domains. Furthermore, the paper explores legal document retrieval strategies, incorporating document vector embeddings and deep learning techniques, underscoring the significance of domain-specific approaches. The study encompasses a variety of models, from Legal-BERT to hierarchical encoder decoder frameworks, showcasing their applicability in handling the complexity inherent in legal texts. Additionally, the survey delves into BERT-based techniques for legal classification tasks, emphasizing the crucial role of context and introducing innovative methods such as "Stride 64." In the realm of legal text simplification, the survey scrutinizes the KIS model and graph-based approaches, revealing nuanced trade- offs between readability, fluency, and meaning preservation. The research culminates with an exploration of the team's participation in LegalEval 2023 shared tasks, underscoring the prowess of deep learning models in tasks such as rhetorical role classification and legal named entity extraction.This survey paper provides a nuanced and comprehensive perspective on the evolving landscape of legal document summarization, offering valuable insights into current methodologies, challenges, and potential future directions for advancing the field. Keywords- Legal text Summarization,Supervised Learning, Extractive Summarization,Semi-supervised methods,Deep learning Techniques,Legal Document retrieval.

A label information fused medical image report generation framework

Medical imaging is an important tool for clinical diagnosis. Nevertheless, it is very time-consuming and error-prone for physicians to prepare imaging diagnosis reports. Therefore, it is necessary to develop some methods to generate medical imaging reports automatically. Currently, the task of medical imaging report generation is challenging in at least two aspects: (1) medical images are very similar to each other. The differences between normal and abnormal images and between different abnormal images are usually trivial; (2) unrelated or incorrect keywords describing abnormal findings in the generated reports lead to mis-communications. In this paper, we propose a medical image report generation framework composed of four modules, including a Transformer encoder, a MIX-MLP multi-label classification network, a co-attention mechanism (CAM) based semantic and visual feature fusion, and a hierarchical LSTM decoder. The Transformer encoder can be used to learn long-range dependencies between images and labels, effectively extract visual and semantic features of images, and establish long-term dependent relationships between visual and semantic information to accurately extract abnormal features from images. The MIX-MLP multi-label classification network, the co-attention mechanism and the hierarchical LSTM network can better identify abnormalities, achieving visual and text alignment fusion and multi-label diagnostic classification to better facilitate report generation. The results of the experiments performed on two widely used radiology report datasets, IU X-RAY and MIMIC-CXR, show that our proposed framework outperforms current report generation models in terms of both natural linguistic generation metrics and clinical efficacy assessment metrics. The code of this work is available online at https://github.com/watersunhznu/LIFMRG.

MSwinUNet: A multi-modal U-shaped swin transformer for supervised change detection

Convolutional neural networks (CNNs) have received significant attention for change detection (CD) on multimodal remote sensing images, but they struggle to capture global cues due to the locality of convolution operations. In contrast, the transformer can learn global semantic information by dividing the input image into patches, adding position encodings, and utilizing the self-attention mechanism. Motivated by this, we propose mSwinUNet, a novel end-to-end multi-modal model with swin-transformer-based and U-shaped siamese network architectures for supervised CD using Sentinel-1 Synthetic Aperture Radar (SAR) and Sentinel-2 Multispectral Imager (MSI) data. mSwinUNet contains multi-modal encoder with difference module, bottleneck, and fused decoder, and all of them are based on swin transformer. Firstly, tokenized multi-modal bitemporal image patches are fed into multiple Siamese encoder branches to extract multi-level multi-modal difference feature maps in parallel. Subsequently, the last level multi-modal difference maps are fused to generate the smallest scale change map in the bottleneck. Then, the hierarchical decoder incorporates patch expansion and fusion operations to fuse multi-scale difference and change maps, effectively recuperating the details of the change information. Finally, the last patch expansion and a linear projection are applied to output the final change map, which preserves the identical spatial resolution as the input image. Extensive experiments have shown that mSwinUNet outperforms several the state-of-the-art multi-modal CD methods on OSCD dataset and the corresponding Sentinel-1 SAR data.

Overlap-Aware Hierarchical Decoder for point cloud registration

Extracting high-quality correspondences is a critical challenge in current feature-learning based point cloud registration methods. Recently, coarse-to-fine network structures have shown great potential in addressing this challenge. Inspired by such structures, we investigate the effectiveness of two-stage network optimization in matching and propose a non-keypoint registration model called Overlap-Aware Hierarchical Decoder (OAH-Net). The construction of this model focuses on reducing outliers in the matching results and improving the understanding of geometric transformation invariance. To achieve this goal, we propose a point-to-point perception module for encoding the paired point clouds and perceiving the overlapping regions, as well as a pyramid hierarchical decoder for decoding multi-level features. We design an optimal matching mechanism adapted to the pyramid structure to handle accurate correspondences from multi-level matching, thereby improving the accuracy of the final registration. Experimental evaluations conducted on indoor and outdoor benchmark tests validate the outstanding performance of OAH-Net, which not only ensures low parameter count and computational efficiency but also demonstrates accuracy and stability comparable to state-of-the-art models.

Features Split and Aggregation Network for Camouflaged Object Detection.

Higher standards have been proposed for detection systems since camouflaged objects are not distinct enough, making it possible to ignore the difference between their background and foreground. In this paper, we present a new framework for Camouflaged Object Detection (COD) named FSANet, which consists mainly of three operations: spatial detail mining (SDM), cross-scale feature combination (CFC), and hierarchical feature aggregation decoder (HFAD). The framework simulates the three-stage detection process of the human visual mechanism when observing a camouflaged scene. Specifically, we have extracted five feature layers using the backbone and divided them into two parts with the second layer as the boundary. The SDM module simulates the human cursory inspection of the camouflaged objects to gather spatial details (such as edge, texture, etc.) and fuses the features to create a cursory impression. The CFC module is used to observe high-level features from various viewing angles and extracts the same features by thoroughly filtering features of various levels. We also design side-join multiplication in the CFC module to avoid detail distortion and use feature element-wise multiplication to filter out noise. Finally, we construct an HFAD module to deeply mine effective features from these two stages, direct the fusion of low-level features using high-level semantic knowledge, and improve the camouflage map using hierarchical cascade technology. Compared to the nineteen deep-learning-based methods in terms of seven widely used metrics, our proposed framework has clear advantages on four public COD datasets, demonstrating the effectiveness and superiority of our model.

Design and Optimization of Hierarchical Demodulator and Decoder for Layered Coded OFDM-IM With NR-LDPC

Design and Optimization of Hierarchical Demodulator and Decoder for Layered Coded OFDM-IM With NR-LDPC

Multi-phase features interaction transformer network for liver tumor segmentation and microvascular invasion assessment in contrast-enhanced CT.

Precise segmentation of liver tumors from computed tomography (CT) scans is a prerequisite step in various clinical applications. Multi-phase CT imaging enhances tumor characterization, thereby assisting radiologists in accurate identification. However, existing automatic liver tumor segmentation models did not fully exploit multi-phase information and lacked the capability to capture global information. In this study, we developed a pioneering multi-phase feature interaction Transformer network (MI-TransSeg) for accurate liver tumor segmentation and a subsequent microvascular invasion (MVI) assessment in contrast-enhanced CT images. In the proposed network, an efficient multi-phase features interaction module was introduced to enable bi-directional feature interaction among multiple phases, thus maximally exploiting the available multi-phase information. To enhance the model's capability to extract global information, a hierarchical transformer-based encoder and decoder architecture was designed. Importantly, we devised a multi-resolution scales feature aggregation strategy (MSFA) to optimize the parameters and performance of the proposed model. Subsequent to segmentation, the liver tumor masks generated by MI-TransSeg were applied to extract radiomic features for the clinical applications of the MVI assessment. With Institutional Review Board (IRB) approval, a clinical multi-phase contrast-enhanced CT abdominal dataset was collected that included 164 patients with liver tumors. The experimental results demonstrated that the proposed MI-TransSeg was superior to various state-of-the-art methods. Additionally, we found that the tumor mask predicted by our method showed promising potential in the assessment of microvascular invasion. In conclusion, MI-TransSeg presents an innovative paradigm for the segmentation of complex liver tumors, thus underscoring the significance of multi-phase CT data exploitation. The proposed MI-TransSeg network has the potential to assist radiologists in diagnosing liver tumors and assessing microvascular invasion.

Load prediction of integrated energy systems for energy saving and carbon emission based on novel multi-scale fusion convolutional neural network

The integrated energy system plays an important role in the energy conservation, emission reduction and the resource-efficient utilization. Accurate load forecasting is a significant basis for the optimal scheduling of the integrated energy system. The integrated energy system has coupling interaction between different energy sources in production, distribution and storage. However, the traditional method cannot effectively extract multi-scale features and utilize the coupling information between the multivariate energy sources. In this paper, a novel multi-scale fusion convolutional neural network integrating the bi-directional long short-term memory network and multi-domains hierarchical decoding is proposed to extract and analyze multivariate load data coupling in the integrated energy system data. The multi-scale fusion convolutional neural network is constructed by the multi-dimension convolution layer to obtain multi-scale feature of the integrated energy system data. Meanwhile, the bi-directional long short-term memory network is applied to extract the time dependencies of the integrated energy system data. Finally, the multi-domains hierarchical decoding extracts the coupling characteristics of different domains to predict multiple domains values. Compared with the backpropagation neural network, the support vector machine, the short and long-time memory network, the bi-directional long short-term memory network, and the convolutional neural network - bi-directional long short-term memory network, the proposed method achieves state-of-the-art results in terms of the mean absolute percentage error with 0.365 %, the explained variance score with 99.987 % and the R2-score with 99.984 %, which proves the effectiveness of the proposed model in the load prediction of the integrated energy system. In addition, the proposed method can provide operational guidance for energy production and storage. Through the operation guidance of the proposed method, the carbon emissions are reduced by 238791.58 kg every week.

A Fast Attention-Guided Hierarchical Decoding Network for Real-Time Semantic Segmentation.

Semantic segmentation provides accurate scene understanding and decision support for many applications. However, many models strive for high accuracy by adopting complex structures, decreasing the inference speed, and making it challenging to meet real-time requirements. Therefore, a fast attention-guided hierarchical decoding network for real-time semantic segmentation (FAHDNet), which is an asymmetric U-shaped structure, is proposed to address this issue. In the encoder, we design a multi-scale bottleneck residual unit (MBRU), which combines the attention mechanism and decomposition convolution to design a parallel structure for aggregating multi-scale information, making the network perform better at processing information at different scales. In addition, we propose a spatial information compensation (SIC) module that effectively uses the original input to make up for the spatial texture information lost during downsampling. In the decoder, the global attention (GA) module is used to process the feature map of the encoder, enhance the feature interaction in the channel and spatial dimensions, and enhance the ability to mine feature information. At the same time, the lightweight hierarchical decoder integrates multi-scale features to better adapt to different scale targets and accurately segment objects of different sizes. Through experiments, FAHDNet performs outstandingly on two public datasets, Cityscapes and Camvid. Specifically, the network achieves 70.6% mean intersection over union (mIoU) at 135 frames per second (FPS) on Cityscapes and 67.2% mIoU at 335 FPS on Camvid. Compared to the existing networks, our model maintains accuracy while achieving faster inference speeds, thus enhancing its practical usability.

Transformer-Based Multi-Scale Feature Integration Network for Video Saliency Prediction

Most cutting-edge video saliency prediction models rely on spatiotemporal features extracted by 3D convolutions due to its local contextual cues acquirement ability. However, the shortage of 3D convolutions is that it cannot effectively capture long-term spatiotemporal dependencies in videos. To address this limitation, we propose a novel Transformer-based Multi-scale Feature Integration Network (TMFI-Net) for video saliency prediction, where the proposed TMFI-Net consists of a semantic-guided encoder and a hierarchical decoder. Firstly, embarking on the Transformer-based multi-level spatiotemporal features, the semantic-guided encoder enhances the features by inserting the high-level feature into each level feature via a top-down pathway and a longitudinal connection, which endows the multi-level spatiotemporal features with rich contextual information. In this way, the features are steered to give more concerns to saliency regions. Secondly, the hierarchical decoder employs a multi-dimensional attention (MA) module to elevate features along channel, temporal, and spatial dimensions jointly. Successively, the hierarchical decoder deploys a progressive decoding block to conduct an initial saliency prediction, which provides a coarse localization of saliency regions. Lastly, considering the complementarity of different saliency predictions, we integrate all initial saliency prediction results into the final saliency map. Comprehensive experimental results on four video saliency datasets firmly demonstrate that our model achieves superior performance when compared with the state-of-the-art video saliency models. The code is available at https://github.com/wusonghe/TMFI-Net.

Number-enhanced representation with hierarchical recursive tree decoding for math word problem solving

Automatic solving math word problems (MWPs) is a number-intensive application in natural language processing (NLP). However, these existing methods are far from achieving acceptable levels of numeracy learning. As a result, the performance of these models is limited for mathematical reasoning. In addition, the mainstream tree decoder suffers from early-stage information loss, resulting in an unsatisfactory performance for complex problems with more operators. In this paper, we propose NERHRT (Number-Enhanced Representation with Hierarchical Recursive Tree Decoding), a simple yet effective number embedding method that produces the numerical reasoning via the decimal notation-based embedding and a dual-direction graph attention network. In addition, a hierarchical recursive tree-structured decoder is introduced to aggregate information from all ancestor nodes. Experiments show that our approach obtains the best performance on four popular benchmark datasets, and beats the state-of-the-art models with a large margin.

Transformer-based hierarchical dynamic decoders for salient object detection

Global context and global contrast are crucial clues for Salient Object Detection (SOD) in images. Most advanced SOD methods exploit CNN-based architectures, achieving impressive results. However, these methods have intrinsic limitations in capturing long-range global information since a CNN extracts feature in local sliding windows. In contrast, transformers exploit a self-attention mechanism to extract features, gaining a powerful capability of learning global cues. Nonetheless, a pure transformer-based network consumes a large computational overhead and easily suffers from attention collapse, as it goes deeper. To address this issue, in this paper, we propose a Transformer-based Hierarchical Dynamic Decoder (T-HDDNet) for image salient object detection. Specifically, our T-HDDNet employs the transformer to encode each image patch into multi-level and multi-resolution features based on the long-range dependencies among pixels. To obtain an accurate saliency map of high resolution, we develop a dynamic dual upsampling mechanism to enlarge feature spatial size in a data-driven manner, together with a dynamic feature fusion unit. Ultimately, the hierarchical dynamic decoders built on the basis of these two units are used to attain the final saliency progressively. Extensive experimental results show that the proposed method achieves the best performance on all benchmarks, in comparison with state-of-the-art technologies.

BrightsightNet: A lightweight progressive low-light image enhancement network and its application in “Rainbow” maglev train

To address the low-light image (LLI) problem in train driving scenarios, this paper proposes a progressive and lightweight network called BrightsightNet for LLI enhancement. First, to overcome the problem of insufficient local exposure level, two structurally identical light curve parameter estimation sub-networks are used for light enhancement in turn. Second, for real-time inference, an efficient feature extraction operator is proposed that combines depth-separable convolution and attention mechanism. Third, the overall network uses encoder–decoder architecture. For the encoder, the output features of the three layers are fused through skip connections to form an information-rich feature map. For the decoder, a hierarchical decoding approach is used to predict the light curve parameters through the three convolution layers sequentially. Experimental results show that BrightsightNet achieves a user study score (USR) of 4.43 on the proposed dataset, outperforming Zero-DCE++, SCI, RetinexDIP, and RUAS by 0.51, 0.86, 0.64, and 1.39, respectively. Moreover, BrightsightNet has parameters of only 2.6K and a single inference time of 0.052 s, which is an innovative and practical solution for low-light image enhancement in train driving scenarios, contributing to safer and more reliable train operations.