Global Feature Map Research Articles

BreakNet: discontinuity-resilient multi-scale transformer segmentation of retinal layers

Visible light optical coherence tomography (vis-OCT) is gaining traction for retinal imaging due to its high resolution and functional capabilities. However, the significant absorption of hemoglobin in the visible light range leads to pronounced shadow artifacts from retinal blood vessels, posing challenges for accurate layer segmentation. In this study, we present BreakNet, a multi-scale Transformer-based segmentation model designed to address boundary discontinuities caused by these shadow artifacts. BreakNet utilizes hierarchical Transformer and convolutional blocks to extract multi-scale global and local feature maps, capturing essential contextual, textural, and edge characteristics. The model incorporates decoder blocks that expand pathways to enhance the extraction of fine details and semantic information, ensuring precise segmentation. Evaluated on rodent retinal images acquired with prototype vis-OCT, BreakNet demonstrated superior performance over state-of-the-art segmentation models, such as TCCT-BP and U-Net, even when faced with limited-quality ground truth data. Our findings indicate that BreakNet has the potential to significantly improve retinal quantification and analysis.

TSRDet: A Table Structure Recognition Method Based on Row-Column Detection

As one of the most commonly used and important data carriers, tables have the advantages of high structuring, strong readability and strong flexibility. However, in reality, tables usually present various forms, such as Excel, images, etc. Among them, the information in the table image cannot be read directly, let alone further applied. Therefore, the research related to image-based table recognition is crucial. It contains the table structure recognition and the table content recognition. Among them, table structure recognition is the most important and difficult task because the table structure is abstract and changeable. In order to address this problem, we propose an innovative table structure recognition method, named TSRDet (Table Structure Recognition based on object Detection). It includes a row-column detection method, named SACNet (StripAttention-CenterNet) and the corresponding post-processing. SACNet is an improved version of the original CenterNet. The specific improvements include the following: firstly, we introduce the Swin Transformer as the encoder to obtain the global feature map of the image. Then, we propose a plug-and-play row-column attention module, including a channel attention module and a row-column spatial attention module. It improves the detection accuracy of rows and columns by capturing long-range row-column feature maps in the image. After completing the row-column detection, this paper also designs a simple and fast post-processing to generate the table structure based on the row-column detection results. Experimental results show that for row-column detection, SACNet has high detection accuracy, even at a high IoU threshold. Specifically, when the threshold is 0.75, its mAP of row detection and column detection still exceeds 90%, which is 91.40% and 92.73% respectively. In addition, in the comparative experiment with the existing object detection methods, SACNet’s performance was significantly better than that of all others. For table structure recognition, the TEDS-Struct score of TSRDet is 95.7%, which shows competitive performance in table structure recognition, and verifies the rationality and superiority of the proposed method.

Fine-Grained High-Resolution Remote Sensing Image Change Detection by SAM-UNet Change Detection Model

Remote sensing image change detection is crucial for urban planning, environmental monitoring, and disaster assessment, as it identifies temporal variations of specific targets, such as surface buildings, by analyzing differences between images from different time periods. Current research faces challenges, including the accurate extraction of change features and the handling of complex and varied image contexts. To address these issues, this study proposes an innovative model named the Segment Anything Model-UNet Change Detection Model (SCDM), which incorporates the proposed center expansion and reduction method (CERM), Segment Anything Model (SAM), UNet, and fine-grained loss function. The global feature map of the environment is extracted, the difference measurement features are extracted, and then the global feature map and the difference measurement features are fused. Finally, a global decoder is constructed to predict the changes of the same region in different periods. Detailed ablation experiments and comparative experiments are conducted on the WHU-CD and LEVIR-CD public datasets to evaluate the performance of the proposed method. At the same time, validation on more complex DTX datasets for scenarios is supplemented. The experimental results demonstrate that compared to traditional fixed-size partitioning methods, the CERM proposed in this study significantly improves the accuracy of SOTA models, including ChangeFormer, ChangerEx, Tiny-CD, BIT, DTCDSCN, and STANet. Additionally, compared with other methods, the SCDM demonstrates superior performance and generalization, showcasing its effectiveness in overcoming the limitations of existing methods.

Human–object interaction detection algorithm based on graph structure and improved cascade pyramid network

Aiming at the problem of insufficient use of human–object interaction (HOI) information and spatial location information in images, we propose a human–object​ interaction detection network based on graph structure and improved cascade pyramid. This network is composed of three branches, namely, graph branch, human–object branch and human pose branch. In graph branch, we propose a Graph-based Interactive Feature Generation Algorithm (GIFGA) to address the inadequate utilization of interaction information. GIFGA constructs an initial dense graph model by taking humans and objects as nodes and their interaction relationships as edges. Then, by traversing each node, the graph model is updated to generate the final interaction features. In human pose branch, we propose an Improved Cascade Pyramid Network (ICPN) to tackle the underutilization of spatial location information. ICPN extracts human pose features and maps both the object bounding boxes and extracted human pose maps onto the global feature map to capture the most discriminative interaction-related region features within the global context. Finally, the features from the three branches are fed into a Multi-Layer Perceptron (MLP) for fusion and then classified for recognition. Experimental results demonstrate that our network achieves mAP of 54.93% and 28.69% on the V-COCO and HICO-DET datasets, respectively.

Multi-scale recurrent attention gated fusion network for single image dehazing

The purpose of single image dehazing is to eliminate the bad influence of haze on images, so as to maintain more scene information of images. In recent years, the convolutional neural networks (CNN) have made significant contributions to single image dehazing. However, the visual quality of dehazed images still needs to be further improved. In view of the problems of single-scale shallow image feature extraction and the insufficient use of intermediate layer features in existing dehazing networks, we propose an end-to-end Multi-scale Recurrent Attention Gated Fusion Network (MRAGFN) to address the image dehazing task. We cascade three Dual Attention Fusion (DAF) modules to progressively form three haze-relevant features map, meanwhile, we adopt downsampling operation on the input to produce global feature map, which are used to weight the three feature maps to compensate for the missing of single-scale feature information. We present Feature Enhancement Module (FEM) to enhance the feature representation ability of these weighted feature maps. We design Recurrent Attention Gated Fusion (RAGF) module by adding attention mechanism and gating mechanism to gradually obtain more refined features based on these weighted features while eliminating redundant features. Experimental results on different hazy images demonstrate that the proposed dehazing network can restore the haze-free images and perform better than the state-of-the-art dehazing networks in terms of the objective indicators (such as PSNR, SSIM)and the subjective visual quality.

Automatic Medical Image Segmentation with Vision Transformer

Automatic image segmentation is vital for the computer-aided determination of treatment directions, particularly in terms of labelling lesions or infected areas. However, the manual labelling of disease regions is inconsistent and a time-consuming assignment. Meanwhile, radiologists’ comments are exceedingly subjective, regularly impacted by personal clinical encounters. To address these issues, we proposed a transformer learning strategy to automatically recognize infected areas in medical images. We firstly utilize a parallel partial decoder to aggregate high-level features and then generate a global feature map. Explicit edge attention and implicit reverse attention are applied to demonstrate boundaries and enhance their expression. Additionally, to alleviate the need for extensive labeled data, we propose a segmentation network combining propagation and transformer architectures that requires only a small amount of labeled data while leveraging fundamentally unlabeled images. The attention mechanisms are integrated within convolutional networks, keeping their global structures intact. Standalone transformers connected straightforwardly and receiving image patches can also achieve impressive segmentation performance. Our network enhanced the learning ability and attained a higher quality execution. We conducted a variety of ablation studies to demonstrate the adequacy of each modelling component. Experiments conducted across various medical imaging modalities illustrate that our model beats the most popular segmentation models. The comprehensive results also show that our transformer architecture surpasses established frameworks in accuracy while better preserving the natural variations in anatomy. Both quantitatively and qualitatively, our model achieves a higher overlap with ground truth segmentations and improved boundary adhesion.

Identification and Localization of Indolent and Aggressive Prostate Cancers Using Multilevel Bi-LSTM.

Identifying indolent and aggressive prostate cancers is a critical problem for optimal treatment. The existing approaches of prostate cancer detection are facing challenges as the techniques rely on ground truth labels with limited accuracy, and histological similarity, and do not consider the disease pathology characteristics, and indefinite differences in appearance between the cancerous and healthy tissue lead to many false positive and false negative interpretations. Hence, this research introduces a comprehensive framework designed to achieve accurate identification and localization of prostate cancers, irrespective of their aggressiveness. This is accomplished through the utilization of a sophisticated multilevel bidirectional long short-term memory (Bi-LSTM) model. The pre-processed images are subjected to multilevel feature map-based U-Net segmentation, bolstered by ResNet-101 and a channel-based attention module that improves the performance. Subsequently, segmented images undergo feature extraction, encompassing various feature types, including statistical features, a global hybrid-based feature map, and a ResNet-101 feature map that enhances the detection accuracy. The extracted features are fed to the multilevel Bi-LSTM model, further optimized through channel and spatial attention mechanisms that offer the effective localization and recognition of complex structures of cancer. Further, the framework represents a promising approach for enhancing the diagnosis and localization of prostate cancers, encompassing both indolent and aggressive cases. Rigorous testing on a distinct dataset demonstrates the model's effectiveness, with performance evaluated through key metrics which are reported as 96.72%, 96.17%, and 96.17% for accuracy, sensitivity, and specificity respectively utilizing the dataset 1. For dataset 2, the model achieves the accuracy, sensitivity, and specificity values of 94.41%, 93.10%, and 94.96% respectively. These results surpass the efficiency of alternative methods.

DFP-Net: A Crack Segmentation Method Based on a Feature Pyramid Network

Timely detection of defects is essential for ensuring safe and stable operation of concrete buildings. Automatic segmentation of concrete buildings’ surfaces is challenging due to the high diversity of crack appearance, the detailed information, and the unbalanced proportion of crack pixels and background pixels. In this work, the Double Feature Pyramid Network is designed for high-precision crack segmentation. Our work reached the state-of-the-art level in crack segmentation, with key contributions outlined as follows: firstly, considering the diversity of crack shapes, the network constructs a feature pyramid containing three feature extraction backbones to extract the global feature map with three scale input images. In particular, due to the biggest challenge being too much single-pixel crack area, the targeted feature pyramid based on the high-resolution is added to extract adequate shallow semantic information. Lastly, designing a cascade feature fusion unit to aggregate the extracted multi-dimensional feature maps and obtain the final prediction. Compared with existing crack detection methods, the superior performance of this method has been verified based on extensive experiments, with Pixel Accuracy of 65.99%, Intersection over Union of 44.71%, and Recall of 62.95%, providing a reliable and efficient solution for the health monitoring and maintenance of concrete structures. This work contributes to the advancement of research and practical applications in related fields, offering robust support for the monitoring and maintenance of concrete structures.

SPNet: a size-variant progressive network for aero-optical thermal radiation effects correction.

When an aircraft is flying at a high speed, the airflow meets the optical cover and is compressed, resulting in aero-optical thermal radiation effects that degrade image quality. In this paper, based on the inherent characteristic that the degrade level of the thermal radiation bias field remains consistent regardless of image size, a size-variant progressive aero-optical thermal radiation effects correction network (SPNet) is proposed. First, SPNet uses two sub-networks to progressively correct degraded image, first and second sub-networks are responsible for learning coarse and accurate thermal radiation bias fields respectively. Second, we introduce the multi-scale feature upsampling module (MFUM) to leverage the multi-scale information of the features and promote inter-channel information interaction. Third, we propose an adaptive feature fusion module (AFFM) to dynamically fuse features from different scales by assigning different weights. At last, a multi-head self-attention feature extraction module (MSFEM) is proposed to extract global information feature maps. Compared with state-of-the-art thermal radiation effects correction methods, experiments on both simulated and real degraded images demonstrate the performance of our proposed method.

An amalgamation of vision transformer with convolutional neural network for automatic lung tumor segmentation.

Lung cancer has the highest mortality rate. Its diagnosis and treatment analysis depends upon the accurate segmentation of the tumor. It becomes tedious if done manually as radiologists are overburdened with numerous medical imaging tests due to the increase in cancer patients and the COVID pandemic. Automatic segmentation techniques play an essential role in assisting medical experts. The segmentation approaches based on convolutional neural networks have provided state-of-the-art performances. However, they cannot capture long-range relations due to the region-based convolutional operator. Vision Transformers can resolve this issue by capturing global multi-contextual features. To explore this advantageous feature of the vision transformer, we propose an approach for lung tumor segmentation using an amalgamation of the vision transformer and convolutional neural network. We design the network as an encoder-decoder structure with convolution blocks deployed in the initial layers of the encoder to capture the features carrying essential information and the corresponding blocks in the final layers of the decoder. The deeper layers utilize the transformer blocks with a self-attention mechanism to capture more detailed global feature maps. We use a recently proposed unified loss function that combines cross-entropy and dice-based losses for network optimization. We trained our network on a publicly available NSCLC-Radiomics dataset and tested its generalizability on our dataset collected from a local hospital. We could achieve average dice coefficients of 0.7468 and 0.6847 and Hausdorff distances of 15.336 and 17.435 on public and local test data, respectively.

Semantic Segmentation and Depth Estimation Based on Residual Attention Mechanism.

Semantic segmentation and depth estimation are crucial components in the field of autonomous driving for scene understanding. Jointly learning these tasks can lead to a better understanding of scenarios. However, using task-specific networks to extract global features from task-shared networks can be inadequate. To address this issue, we propose a multi-task residual attention network (MTRAN) that consists of a global shared network and two attention networks dedicated to semantic segmentation and depth estimation. The convolutional block attention module is used to highlight the global feature map, and residual connections are added to prevent network degradation problems. To ensure manageable task loss and prevent specific tasks from dominating the training process, we introduce a random-weighted strategy into the impartial multi-task learning method. We conduct experiments to demonstrate the effectiveness of the proposed method.

Substation rotational object detection based on multi-scale feature fusion and refinement

In modern energy systems, substations are the core of electricity transmission and distribution. However, similar appearance and small size pose significant challenges for automatic identification of electrical devices. To address these issues, we collect and annotate the substation rotated device dataset (SRDD). Further, feature fusion and feature refinement network (F3RNet) are constructed based on the classic structure pattern of backbone-neck-head. Considering the similar appearance of electrical devices, the deconvolution fusion module (DFM) is designed to enhance the expression of feature information. The balanced feature pyramid (BFP) is embedded to aggregate the global feature. The feature refinement is constructed to adjust the original feature maps by considering the feature alignment between the anchors and devices. It can generate more accurate feature vectors. To address the problem of sample imbalance between electrical devices, the gradient harmonized mechanism (GHM) loss is utilized to adjust the weight of each sample. The ablation experiments are conducted on the SRDD dataset. F3RNet achieves the best detection performance compared with classical object detection networks. Also, it is verified that the features from global feature maps can effectively recognize the similar and small devices.

Attribute‐guided transformer for robust person re‐identification

AbstractRecent studies reveal the crucial role of local features in learning robust and discriminative representations for person re‐identification (Re‐ID). Existing approaches typically rely on external tasks, for example, semantic segmentation, or pose estimation, to locate identifiable parts of given images. However, they heuristically utilise the predictions from off‐the‐shelf models, which may be sub‐optimal in terms of both local partition and computational efficiency. They also ignore the mutual information with other inputs, which weakens the representation capabilities of local features. In this study, the authors put forward a novel Attribute‐guided Transformer (AiT), which explicitly exploits pedestrian attributes as semantic priors for discriminative representation learning. Specifically, the authors first introduce an attribute learning process, which generates a set of attention maps highlighting the informative parts of pedestrian images. Then, the authors design a Feature Diffusion Module (FDM) to iteratively inject attribute information into global feature maps, aiming at suppressing unnecessary noise and inferring attribute‐aware representations. Last, the authors propose a Feature Aggregation Module (FAM) to exploit mutual information for aggregating attribute characteristics from different images, enhancing the representation capabilities of feature embedding. Extensive experiments demonstrate the superiority of our AiT in learning robust and discriminative representations. As a result, the authors achieve competitive performance with state‐of‐the‐art methods on several challenging benchmarks without any bells and whistles.

FDNet: Focal Decomposed Network for efficient, robust and practical time series forecasting

This paper presents FDNet: a Focal Decomposed Network for efficient, robust and practical time series forecasting. We break away from conventional deep time series forecasting formulas which obtain prediction results from universal feature maps of input sequences. In contrary, FDNet neglects universal correlations of input elements and only extracts fine-grained local features from input sequence. We show that: (1) Deep time series forecasting with only fine-grained local feature maps of input sequence is feasible upon theoretical basis. (2) By abandoning global coarse-grained feature maps, FDNet overcomes distribution shift problem caused by changing dynamics of time series which is common in real-world applications. (3) FDNet is not dependent on any inductive bias of time series except basic auto-regression, making it general and practical. Moreover, we propose focal input sequence decomposition method which decomposes input sequence in a focal manner for efficient and robust forecasting when facing Long Sequence Time series Input (LSTI) problem. FDNet achieves competitive forecasting performances on six real-world benchmarks and reduces prediction MSE by 38.4% on average compared with other thirteen SOTA baselines. The source code is available at https://github.com/OrigamiSL/FDNet.

An encoder‐decoder framework with dynamic convolution for weakly supervised instance segmentation

AbstractIn the systems of industrial robotics and autonomous vehicles, instance segmentation is widely employed. However, manually labelling an object outline is time‐consuming. In order to reduce annotation costs, we present a weakly supervised instance segmentation method in this article. A deeply convolutional network is first used to construct multi‐scale feature maps for each object in the input image. After that, the encoder‐decoder framework with dynamic convolution is utilised to enhance model capacity and efficiency, while avoiding the issues of anchor design, proposal selection, and RoIAlign implementation. In particular, Dynamic Heads are used in the encoder to create dynamic convolution kernels, while Instance Heads are used in the decoder to provide the global feature map. With dynamic convolution, each instance can be segmented independently, reducing interference with other instances and improving segmentation accuracy. Under the supervision of projection loss and pixel point colour pairing loss, the contours of each object are finally outlined. On the PASCAL VOC and MS COCO datasets, the proposed method is competitive with more sophisticated approaches. In the VOC dataset, segmentation performance achieved 37.6% average precision with ResNet‐101 and FPN networks. The extensively visualised results demonstrate the effectiveness of the proposed encoder‐decoder framework with dynamic convolution.

Multifeature Fusion-Based Object Detection for Intelligent Transportation Systems

The detection of 3D objects with high precision from point cloud data has become a crucial research topic in intelligent transportation systems. By effectively modeling global and local features, it can be acquired the state-of-the-art detector for 3D object detection. Nevertheless, regarding the previous work on feature representations, volumetric generation or point learning methods have difficulty building the relationships between local features and global features. Thus, we propose a multi-feature fusion network (MFFNet) to improve detection precision for 3D point cloud data by combining the global features from 3D voxel convolutions with the local features from the point learning network. Our algorithm is an end-to-end detection framework that contains a voxel convolutional module, a local point feature module and a detection head. Significantly, MFFNet constructs the local point feature set with point learning and sampling and the global feature map through 3D voxel convolution from raw point clouds. The detection head can use the obtained fusion feature to predict the position and category of the examined 3D object, so the proposed method can obtain higher precision than existing approaches. An experimental evaluation on the KITTI 3D object detection dataset obtain 97% MAP (Mean Average Precision) and Waymo Open dataset obtain 80% MAP, which proves the efficiency of the developed feature fusion representation method for 3D objects, and it can achieve satisfactory location accuracy.

Target Detection Algorithm Incorporating Visual Expansion Mechanism and Path Syndication

Because the lack of semantic information exchange between characteristic layers, the SSD (Single Shot multibox Detector) algorithm has insufficient detection performance. To address this problem, a detection algorithm called VPE-SSD (Visual Path Enhancement SSD) by incorporating a visual expansion mechanism and path syndication proposed in this paper. Firstly, a visual expansion mechanism is added to the shallow characteristic layer to increase the perceptual field. This enables the semantic information in the shallow layer to be more fully utilized by the network. It can also achieve the purpose of enhancing the expressiveness of the shallow feature layer. Then, the processed deep and shallow characteristic layers are fed into the path syndication module for bi-directional fusion. This improves the global information of the feature layers and generates multi-scale global feature maps. Next, to enhance the detailed information of deep characteristics and improve their expression, the deep characteristic enhancement module is applied to the last three characteristic maps. Finally, using the blended attention module to reduce the negative interference and improve the expression of characteristic maps during target detection. The experimental analysis of the VPE-SSD algorithm is conducted on VOC and COCO, and the mAP is 83.4% and 48.4%. From the result, VPE-SSD algorithm can make better use of the different size characteristic information which helps to improve the performance of the algorithm.

C-Net: Cascaded convolutional neural network with global guidance and refinement residuals for breast ultrasound images segmentation

Background and objectiveBreast lesions segmentation is an important step of computer-aided diagnosis system. However, speckle noise, heterogeneous structure, and similar intensity distributions bring challenges for breast lesion segmentation. MethodsIn this paper, we presented a novel cascaded convolutional neural network integrating U-net, bidirectional attention guidance network (BAGNet) and refinement residual network (RFNet) for the lesion segmentation in breast ultrasound images. Specifically, we first use U-net to generate a set of saliency maps containing low-level and high-level image structures. Then, the bidirectional attention guidance network is used to capture the context between global (low-level) and local (high-level) features from the saliency map. The introduction of the global feature map can reduce the interference of surrounding tissue on the lesion regions. Furthermore, we developed a refinement residual network based on the core architecture of U-net to learn the difference between rough saliency feature maps and ground-truth masks. The learning of residuals can assist us to obtain a more complete lesion mask. ResultsTo evaluate the segmentation performance of the network, we compared with several state-of-the-art segmentation methods on the public breast ultrasound dataset (BUSIS) using six commonly used evaluation metrics. Our method achieves the highest scores on six metrics. Furthermore, p-values indicate significant differences between our method and the comparative methods. ConclusionsExperimental results show that our method achieves the most competitive segmentation results. In addition, we apply the network on renal ultrasound images segmentation. In general, our method has good adaptability and robustness on ultrasound image segmentation.

LamNet: A Lesion Attention Maps-Guided Network for the Prediction of Choroidal Neovascularization Volume in SD-OCT Images.

Choroidal neovascularization (CNV) volume prediction has an important clinical significance to predict the therapeutic effect and schedule the follow-up. In this paper, we propose a Lesion Attention Maps-Guided Network (LamNet) to automatically predict the CNV volume of next follow-up visit after therapy based on 3-dimentional spectral-domain optical coherence tomography (SD-OCT) images. In particular, the backbone of LamNet is a 3D convolutional neural network (3D-CNN). In order to guide the network to focus on the local CNV lesion regions, we use CNV attention maps generated by an attention map generator to produce the multi-scale local context features. Then, the multi-scale of both local and global feature maps are fused to achieve the high-precision CNV volume prediction. In addition, we also design a synergistic multi-task predictor, in which a trend-consistent loss ensures that the change trend of the predicted CNV volume is consistent with the real change trend of the CNV volume. The experiments include a total of 541 SD-OCT cubes from 68 patients with two types of CNV captured by two different SD-OCT devices. The results demonstrate that LamNet can provide the reliable and accurate CNV volume prediction, which would further assist the clinical diagnosis and design the treatment options.

Learned Compression Framework With Pyramidal Features and Quality Enhancement for SAR Images

Current image compression algorithms based on transforms can achieve ideal performance for natural images, but do not do well with synthetic aperture radar (SAR) images. We propose a learned compression framework with pyramidal features and quality enhancement to fully exploit the redundancy among image pixels and to improve the compression bitrate and reconstruction quality. Based on the variational autoencoder (VAE) architecture, pyramidal decomposition is performed at the first autoencoder to extract both global and coarse feature maps. The latent distribution is modeled by the second hyperprior autoencoder with a single-Gaussian model for more accurate and flexible entropy estimation. Universal quantization is applied to consolidate the entropy estimation accuracy of the hyperprior network. To further improve reconstruction quality, a residual dense network (RDN) is adopted to fully capture local and global features. Experimental results demonstrate that the proposed framework provides a better rate-distortion tradeoff than standard codecs such as JPEG, JPEG2000, and learning-based methods on both the Sandia and ICEYE datasets.