Hierarchical Encoding Research Articles

Online monitoring and evaluation for conceptual design of product morphological structure driven by the retrospective analysis of cognitive behaviour

Monitoring and understanding designers’ design behaviour holds significant value for optimising the design paradigm and enhancing its quality and efficiency. This paper focuses on computer-aided conceptual design of product morphological structures, presenting a design monitoring and evaluation method driven by retrospective analysis of the association of designers’ cognitive behaviour. To begin with the monitoring perspective, the article analyzes the characteristics and collection methods of three types of design cognitive behaviour data. It also provides a behaviour element sequence transcription model, encompassing cognitive behaviour encoding, sketch behaviour encoding, and hierarchical encoding of design problems. Subsequently, a quantitative analysis index system is established with three dimensions: cognitive behaviour order, design state credibility, and thinking transition stability. By retrospectively analyzing the evolutionary relationships among fundamental design cognitive behavioural elements and referring to knowledge of design cognitive behavioural laws, designers’ cognitive preference type and state fluctuations can be distinguished. Finally, a hybrid fuzzy multi-attribute evaluation method is applied to filter various transcriptional combinations and calculate a comprehensive evaluation score. A real conceptual design experiment has preliminarily demonstrated the effectiveness of the introduced method.

CIS-UNet: Multi-class segmentation of the aorta in computed tomography angiography via context-aware shifted window self-attention

Advancements in medical imaging and endovascular grafting have facilitated minimally invasive treatments for aortic diseases. Accurate 3D segmentation of the aorta and its branches is crucial for interventions, as inaccurate segmentation can lead to erroneous surgical planning and endograft construction. Previous methods simplified aortic segmentation as a binary image segmentation problem, overlooking the necessity of distinguishing between individual aortic branches. In this paper, we introduce Context-Infused Swin-UNet (CIS-UNet), a deep learning model designed for multi-class segmentation of the aorta and thirteen aortic branches. Combining the strengths of Convolutional Neural Networks (CNNs) and Swin transformers, CIS-UNet adopts a hierarchical encoder–decoder structure comprising a CNN encoder, a symmetric decoder, skip connections, and a novel Context-aware Shifted Window Self-Attention (CSW-SA) module as the bottleneck block. Notably, CSW-SA introduces a unique adaptation of the patch merging layer, distinct from its traditional use in the Swin transformers. CSW-SA efficiently condenses the feature map, providing a global spatial context, and enhances performance when applied at the bottleneck layer, offering superior computational efficiency and segmentation accuracy compared to the Swin transformers. We evaluated our model on computed tomography (CT) scans from 59 patients through a 4-fold cross-validation. CIS-UNet outperformed the state-of-the-art Swin UNetR segmentation model by achieving a superior mean Dice coefficient of 0.732 compared to 0.717 and a mean surface distance of 2.40 mm compared to 2.75 mm. CIS-UNet’s superior 3D aortic segmentation offers improved accuracy and optimization for planning endovascular treatments. Our dataset and code will be made publicly available at https://github.com/mirthAI/CIS-UNet.

AIRHF-Net: an adaptive interaction representation hierarchical fusion network for occluded person re-identification

To tackle the high resource consumption in occluded person re-identification, sparse attention mechanisms based on Vision Transformers (ViTs) have become popular. However, they often suffer from performance degradation with long sequences, omission of crucial information, and token representation convergence. To address these issues, we introduce AIRHF-Net: an Adaptive Interaction Representation Hierarchical Fusion Network, named AIRHF-Net, designed to enhance pedestrian identity recognition in occluded scenarios. Our approach begins with the development of an Adaptive Local-Window Interaction Encoder (AL-WIE), which aims to overcome the inherent subjective limitations of traditional sparse attention mechanisms. This innovative encoder merges window attention, adaptive local attention, and interaction attention, facilitating automatic localization and focusing on visible pedestrian regions within images. It effectively extracts contextual information from window-level features while minimizing the impact of occlusion noise. Additionally, recognizing that ViTs may lose spatial information in deeper structural layers, we implement a Local Hierarchical Encoder (LHE). This component segments the input sequence in the spatial dimension, integrating features from various spatial positions to construct hierarchical local representations that substantially enhance feature discriminability. To further augment the quality and breadth of datasets, we adopt an Occlusion Data Augmentation Strategy (ODAS), which bolsters the model’s capacity to extract critical information under occluded conditions. Extensive experiments demonstrate that our method achieves improved performance on the Occluded-DukeMTMC dataset, with a rank-1 accuracy of 69.6% and an mAP of 61.6%.

Ray Tracing Animated Displaced Micro‐Meshes

AbstractWe present a new method that allows efficient ray tracing of virtually artefact‐free animated displaced micro‐meshes (DMMs) [MMT23] and preserves their low memory footprint and low BVH build and update cost. DMMs allow for compact representation of micro‐triangle geometry through hierarchical encoding of displacements. Displacements are computed with respect to a coarse base mesh and are used to displace new vertices introduced during 1 : 4 subdivision of the base mesh. Applying non‐rigid transformation to the base mesh can result in silhouette and normal artefacts (see Figure 1) during animation. We propose an approach which prevents these artefacts by interpolating transformation matrices before applying them to the DMM representation. Our interpolation‐based algorithm does not change DMM data structures and it allows for efficient bounding of animated micro‐triangle geometry which is essential for fast tessellation‐free ray tracing of animated DMMs.

Transformer-Integrated Hybrid Convolutional Neural Network for Dose Prediction in Nasopharyngeal Carcinoma Radiotherapy.

Radiotherapy is recognized as the major treatment of nasopharyngeal carcinoma. Rapid and accurate dose prediction can improve the efficiency of the treatment planning process and the quality of radiotherapy plans. Currently, deep learning-based methods have been widely applied to dose prediction for radiotherapy treatment planning. However, it is important to note that existing models based on Convolutional Neural Networks (CNN) often overlook long-distance information. Although some studies try to use Transformer to solve the problem, it lacks the ability of CNN to process the spatial information inherent in images. Therefore, we propose a novel CNN and Transformer hybrid dose prediction model. To enhance the transmission ability of features between CNN and Transformer, we design a hierarchical dense recurrent encoder with a channel attention mechanism. Additionally, we propose a progressive decoder that preserves richer texture information through layer-wise reconstruction of high-dimensional feature maps. The proposed model also introduces object-driven skip connections, which facilitate the flow of information between the encoder and decoder. Experiments are conducted on in-house datasets, and the results show that the proposed model is superior to baseline methods in most dosimetric criteria. In addition, the image analysis metrics including PSNR, SSIM, and NRMSE demonstrate that the proposed model is consistent with ground truth and produces promising visual effects compared to other advanced methods. The proposed method could be taken as a powerful clinical guidance tool for physicists, significantly enhancing the efficiency of radiotherapy planning. The source code is available at https://github.com/CDUTJ102/THCN-Net .

Revamping Image-Recipe Cross-Modal Retrieval with Dual Cross Attention Encoders

The image-recipe cross-modal retrieval task, which retrieves the relevant recipes according to food images and vice versa, is now attracting widespread attention. There are two main challenges for image-recipe cross-modal retrieval task. Firstly, a recipe’s different components (words in a sentence, sentences in an entity, and entities in a recipe) have different weight values. If a recipe’s different components own the same weight, the recipe embeddings cannot pay more attention to the important components. As a result, the important components make less contribution to the retrieval task. Secondly, the food images have obvious properties of locality and only the local food regions matter. There are still difficulties in enhancing the discriminative local region features in the food images. To address these two problems, we propose a novel framework named Dual Cross Attention Encoders for Cross-modal Food Retrieval (DCA-Food). The proposed framework consists of a hierarchical cross attention recipe encoder (HCARE) and a cross attention image encoder (CAIE). HCARE consists of three types of cross attention modules to capture the important words in a sentence, the important sentences in an entity and the important entities in a recipe, respectively. CAIE extracts global and local region features. Then, it calculates cross attention between them to enhance the discriminative local features in the food images. We conduct the ablation studies to validate our design choices. Our proposed approach outperforms the existing approaches by a large margin on the Recipe1M dataset. Specifically, we improve the R@1 performance by +2.7 and +1.9 on the 1k and 10k testing sets, respectively.

Hierarchical Shared Encoder With Task-Specific Transformer Layer Selection for Emotion-Cause Pair Extraction

Hierarchical Shared Encoder With Task-Specific Transformer Layer Selection for Emotion-Cause Pair Extraction

Hierarchical stacked spatiotemporal self-attention network for sea surface temperature forecasting

Sea surface temperature (SST) is a highly complex spatiotemporal variable, which stems from its susceptibility to non-linear dynamical processes and substantial spatiotemporal variability. In particular, accurately forecasting small-scale SST is a formidable challenge due to the compounded effects of diverse physical processes spanning across various scales. In this study, we employ deep learning methods to mine the ocean’s evolutionary patterns, as the ocean’s dynamic mechanisms are inherently embedded in spatiotemporal data. We propose a hierarchical stacked spatiotemporal self-attention mechanism (HSSSA) network architecture. The hierarchical stacked encoder–decoder architecture provides the capability for feature fusion and extraction at different scales. The spatial self-attention and temporal self-attention modules simultaneously focus on information from different spatial locations and time steps, allowing the exploration of spatiotemporal patterns in the complex dynamics of the ocean. The experiments are conducted on a high-resolution East China Sea dataset (1/10°×1/10°) to demonstrate the forecast performance of the proposed model for refined ocean variables. The 15-day forecasts indicate that the HSSSA method outperforms the EOF-ARIMA and CNN-Transformer methods.

TBiSeg: A transformer-based network with bi-level routing attention for inland waterway segmentation

Unmanned surface vehicles (USVs) for inland waterways have recently attracted increasing attention in various fields. Accurate detection in navigable regions is crucial for ensuring USV safety in autonomous navigation. However, the complex and variable environment of inland waterways, such as confusable textures and irregular edge details, continues to pose some problems in existing methods. Therefore, to acquire navigable regions, this study proposed TBiSeg, a Vision Transformer-based efficient inland waterway segmentation network, for obtaining pixel-level results. Bi-level routing attention is used to improve the Transformer block, which enhances the understanding of inland water textures. Additionly, this study combined global and local attention through a hierarchical encoder–decoder architecture. To simulate inland waterway scenes as accurately as possible, this study used two representative public datasets for data integration and data augmentation, and conducted testing and cross-validating using multiple inland waterway datasets. Results demonstrated that the model performed better than current state-of-the-art models in segmentation accuracy and robustness in complex inland waterway environments while showing impressive generalization. The datasets and code used in this paper is available at https://github.com/dawnnazzz/TBiSeg.

ALKU-Net: Adaptive Large Kernel Attention Convolution Network for Lung Nodule Segmentation

The accurate segmentation of lung nodules in computed tomography (CT) images is crucial for the early screening and diagnosis of lung cancer. However, the heterogeneity of lung nodules and their similarity to other lung tissue features make this task more challenging. By using large receptive fields from large convolutional kernels, convolutional neural networks (CNNs) can achieve higher segmentation accuracies with fewer parameters. However, due to the fixed size of the convolutional kernel, CNNs still struggle to extract multi-scale features for lung nodules of varying sizes. In this study, we propose a novel network to improve the segmentation accuracy of lung nodules. The network integrates adaptive large kernel attention (ALK) blocks, employing multiple convolutional layers with variously sized convolutional kernels and expansion rates to extract multi-scale features. A dynamic selection mechanism is also introduced to aggregate the multi-scale features obtained from variously sized convolutional kernels based on selection weights. Based on this, we propose a lightweight convolutional neural network with large convolutional kernels, called ALKU-Net, which integrates the ALKA module in a hierarchical encoder and adopts a U-shaped decoder to form a novel architecture. ALKU-Net efficiently utilizes the multi-scale large receptive field and enhances the model perception capability through spatial attention and channel attention. Extensive experiments demonstrate that our method outperforms other state-of-the-art models on the public dataset LUNA-16, exhibiting considerable accuracy in the lung nodule segmentation task.

SuPrNet: Super Proxy for 4D occupancy forecasting

Spacetime (4D) occupancy forecasting explicitly models the future development of self-driving vehicles (SDVs) themselves and their surrounding environment, which is highly beneficial for their downstream tasks. Currently, using point cloud forecasting as a proxy to self-supervisedly optimize the 4D occupancy forecasting framework has significant application prospects due to its independence from labeled data. However, existing methods have the following shortcomings: (1) Insufficient mining of spatiotemporal information in the scene. (2) Neglecting the extraction of regional deep features. (3) Inadequate learning ability for spatiotemporal predictive learning. Based on this, we propose SuPrNet, constructing a ”Super Proxy” to address the aforementioned issues. Specifically, SuPrNet first embeds the spatial and temporal information of the current frame into point-wise features through a 4D hybrid representation combined with multi-planar and grid features (4DPG). Then, a Hierarchical Encoder–Decoder Module (HED) with sparsity consideration is used to structure the point cloud and extract deep features. Thanks to the structured feature map representation, SuPrNet proceeds to use a spatiotemporal predictive learning module (STL) to implicitly predict future feature maps. Finally, we decode these feature maps to a specific size as the forecasting of future occupancy. The 4DPG, HED, and STL in the model are designed to address the aforementioned issues, and experiments on Nuscene and KITTI-Odometry show that our method achieves the best 4D occupancy forecasting and point cloud forecasting accuracy, surpassing previous state-of-the-art models by approximately 30%. Additionally, we have expanded and validated the application prospects of SuPrNet through various downstream tasks. Our project is available at https://github.com/AlanLiangC/4DPCF.

SSCFormer: Revisiting ConvNet-Transformer Hybrid Framework From Scale-Wise and Spatial-Channel-Aware Perspectives for Volumetric Medical Image Segmentation.

Accurate and robust medical image segmentation is crucial for assisting disease diagnosis, making treatment plan, and monitoring disease progression. Adaptive to different scale variations and regions of interest is essential for high accuracy in automatic segmentation methods. Existing methods based on the U-shaped architecture respectively tackling intra- and inter-scale problem with a hierarchical encoder, however, are restricted by the scope of multi-scale modeling. In addition, global attention and scaling attention in regions of interest have not been appropriately adopted, especially for the salient features. To address these two issues, we propose a ConvNet-Transformer hybrid framework named SSCFormer for accurate and versatile medical image segmentation. The intra-scale ResInception and inter-scale transformer bridge are designed to collaboratively capture the intra- and inter-scale features, facilitating the interaction of small-scale disparity information at a single stage with large-scale from multiple stages. Global attention and scaling attention are cleverly integrated from a spatial-channel-aware perspective. The proposed SSCFormer is tested on four different medical image segmentation tasks. Comprehensive experimental results show that SSCFormer outperforms the current state-of-the-art methods.

Hierarchical Encoding and Fusion of Brain Functions for Depression Subtype Classification

Hierarchical Encoding and Fusion of Brain Functions for Depression Subtype Classification

Medical image super-resolution via transformer-based hierarchical encoder–decoder network

Medical image super-resolution via transformer-based hierarchical encoder–decoder network

Bridging the Domain Gap in Scene Flow Estimation via Hierarchical Smoothness Refinement

This article introduces SmoothFlowNet3D, an innovative encoder-decoder architecture specifically designed for bridging the domain gap in scene flow estimation. To achieve this goal, SmoothFlowNet3D divides the scene flow estimation task into two stages: initial scene flow estimation and smoothness refinement. Specifically, SmoothFlowNet3D comprises a hierarchical encoder that extracts multi-scale point cloud features from two consecutive frames, along with a hierarchical decoder responsible for predicting the initial scene flow and further refining it to achieve smoother estimation. To generate the initial scene flow, a cross-frame nearest-neighbor search operation is performed between the features extracted from two consecutive frames, resulting in forward and backward flow embeddings. These embeddings are then combined to form the bidirectional flow embedding, serving as input for predicting the initial scene flow. Additionally, a flow smoothing module based on the self-attention mechanism is proposed to predict the smoothing error and facilitate the refinement of the initial scene flow for more accurate and smoother estimation results. Extensive experiments demonstrate that the proposed SmoothFlowNet3D approach achieves state-of-the-art performance on both synthetic datasets and real LiDAR point clouds, confirming its effectiveness in enhancing scene flow smoothness.

Stories behind decisions: Towards interpretable malware family classification with hierarchical attention

Malware family classification is essential for understanding malware code and behavioral characteristics, significantly reducing analysis time and response time for emerging malware. However, existing malware family classification methods are limited for two reasons. (1) The feature representation learned from a single information source is not sufficiently discriminative for large-scale malware family classification. (2) The lack of interpretability of classification results makes it difficult to provide intuitive guidance for malware analysis. To tackle the issues, we propose MalAtt, a novel malware family classification method for interpretable and large-scale classification. Specifically, a unified feature engineering approach is proposed to obtain integrated embedding from heterogeneous static and dynamic information. Then, a hierarchical attention encoder module is employed to progressively extract semantic features from the static and dynamic embedding. A collaborative attention module is introduced to model the association between high-level static and dynamic features. Finally, MalAtt is pluggable for different classifiers, such as SVM and RF, to implement malware family classification. For evaluations, we carefully collected a dataset from 62 malware families containing over 20,070 in-the-wild samples. MalAtt is proved to be effective when classifying large-scale malware families, with an accuracy of 93.26%. Besides, the ablation study demonstrates the rationality of the proposed modules. Additionally, we provide a case study to show how MalAtt intuitively guides in-depth malware analysis by highlighting partial input features with high attention scores.

Bidirectional interaction of CNN and Transformer for image inpainting

Image inpainting requires filling the corrupted regions with plausible contents and fitting seamlessly with the surroundings. Currently, CNN-dominated inpainting models still produce artifacts inconsistent with surroundings when facing large-scale corrupted regions. This is attributed to the locality and weight sharing of convolutions, which limits the ability of CNNs to learn the correlations between corrupted contents and distant contexts explicitly. In contrast, Transformer-dominated models are experts in modeling such long-range correlations, but they often deteriorate local details due to the lack of locality. A potential inpainting model should incorporate the advantages of CNN and Transformer. To do this, we propose an inpainting-oriented concurrent hybrid network, termed bidirectional interaction dual-stream network (BIDS-Net), which can sufficiently integrate the advantages of CNN and Transformer. Specifically, we adopt a CNN stream to capture rich local patterns to achieve detail reconstruction and refinement, and a Transformer stream to model long-range contextual correlations to fully borrow the relevant information within a global field of view. More importantly, we devise a bidirectional feature interaction (BFI) module based on a selective feature fusion method, which aims to enhance the locality of the transformer stream and the long-range correlation awareness of the CNN stream to realize full complementary advantages. In addition, we build both of the streams upon a hierarchical encoder–decoder structure to facilitate multi-scale context reasoning and improve computational efficiency. The comprehensive experimental results on three public datasets demonstrate the superiority of the proposed BIDS-Net in image inpainting task.

Enhance audio-visual segmentation with hierarchical encoder and audio guidance

As one of the pivotal technologies leading towards embodied intelligence, audio-visual segmentation is geared towards achieving precise segmentation of sounding objects, offering vast application prospects in scenarios such as emergency rescue and natural exploration. Nevertheless, the performance of audio-visual segmentation technology encounters limitations stemming from challenges related to the adaptation and fusion of cross-modal information encoding, as well as the decoding and generation of masks. To address these issues, this paper explores the adaptation of multi-modal information based on a shared encoder by employing a neural architecture search method to design a hierarchical encoder cooperation module for enhanced information interaction. An intermediate loss is leveraged to help the encoder to keep spatial knowledge reserved. Furthermore, an audio-guided class-aware decoder is devised to guide the generation of masks. Our approach has yielded competitive experimental results across multiple datasets, thus substantiating its effectiveness.

Structural and positional ensembled encoding for Graph Transformer

In the Transformer architecture, positional encoding is a vital component because it provides the model with information about the structure and position of data. In Graph Transformer, there have been attempts to introduce different positional encodings and inject additional structural information. Therefore, in terms of integrating positional and structural information, we propose a Structural and Positional Ensembled Graph Transformer (SPEGT). We developed SPEGT by noting the different properties of structural and positional encodings of graphs and the similarity of their computational processes. We have set a unified component that integrates the functionalities: (i) Random Walk Positional Encoding, (ii) Shortest Path Distance between each node, and (iii) Hierarchical Cluster Encoding. We find a problem with a well-known positional encoding and experimentally verify that combining it with other encodings can solve their problem. In addition, SPEGT outperforms previous models on a variety of graph datasets. We also show that SPEGT using unified positional encoding, performs well on structurally indistinguishable graph data through error case analysis.

Multi-scale learnable key-channel attention network for point cloud classification and segmentation

The 3D point cloud, a common format of 3D data, finds extensive applications in fields like remote sensing, surveying, robotics, and more. Addressing the challenges posed by insufficient weighted feature information and the offset between weighted results and task expectations in the attention mechanism, this paper proposes a Multi-scale Learnable Key-channel Attention Network (MLKNet). First, we introduce a feature feedback-repair module to mitigate the impact of information loss in the feature embedding process. This module aims to fully embed the original data into a high-dimensional feature space, ensuring a rich supply of feature information for subsequent transformer modules. Second, an efficient hierarchical local feature encoder extract and aggregate local features from point clouds at various scales, thereby significantly enhancing the model's capability to represent geometric structures. Third, a novel learnable key-channel attention module allows tasks to influence the feature selection and weighting process directly, make the highlighted features as close to task expectations as possible, effectively enhancing the network's perception of global semantic information. Our method was benchmarked on various tasks where we achieved overall accuracy (OA) of 92.3% on the ModelNet40 classification task and achieved instance mean intersection over union (ins. mIoU) of 87.6% on the ShapeNet-part segmentation task. The results indicate the superior performance of our method.