Multi-level Feature Maps Research Articles

A Boundary-Enhanced Decouple Fusion Segmentation Network for Diagnosis of Adenomatous Polyps.

Adenomatous polyps, a common premalignant lesion, are often classified into villous adenoma (VA) and tubular adenoma (TA). VA has a higher risk of malignancy, whereas TA typically grows slowly and has a lower likelihood of cancerous transformation. Accurate classification is essential for tailored treatment. In this study, we develop a deep learning-based approach for the localization and classification of adenomatous polyps using endoscopic images. Specifically, a pre-trained EGE-UNet is first adopted to extract regions of interest from original images. Multi-level feature maps are then extracted by the feature extraction pipeline (FEP). The deep-level features are fed into the Pyramid Pooling Module (PPM) to capture global contextual information, and the squeeze body edge (SBE) module is then used to decouple the body and edge parts of features, enabling separate analysis of their distinct characteristics. The Group Aggregation Bridge (GAB) and Boundary Enhancement Module (BEM) are then applied to enhance the body features and edge features, respectively, emphasizing their structural and morphological characteristics. By combining the features of the body and edge parts, the final output can be obtained. Experiments show the proposed method achieved promising results on two private datasets. For adenoma vs. non-adenoma classification, It achieved a mIoU of 91.41%, mPA of 96.33%, mHD of 11.63, and mASD of 2.33. For adenoma subclassification (non-adenomas vs. villous adenomas vs. tubular adenomas), it achieved a mIoU of 91.21%, mPA of 94.83%, mHD of 13.75, and mASD of 2.56. These results demonstrate the potential of our approach for precise adenomatous polyp classification.

Multi-query and multi-level enhanced network for semantic segmentation

Plain transformer-based methods have achieved promising performance on semantic segmentation recently. These methods adopt a single set of class queries to predict masks of different semantic categories based on multi-level feature maps. We argue that this single-query design cannot fully exploit diverse information of different levels for improved semantic segmentation. To address this issue, we propose a multi-query and multi-level enhanced network for semantic segmentation (named QLSeg). Our QLSeg first performs multi-level feature enhancement on plain transformer to improve feature discriminability. Afterwards, we introduce multi-query decoder to respectively extract feature embeddings and predict mask logits at different levels, where feature embeddings are adaptively merged for classification and mask logits are summed for output masks. In addition, we introduce masked attention-to-mask to focus on local regions with the same class. We perform the experiments on three widely-used semantic segmentation datasets: ADE20K, COCO-Stuff-10K, and PASCAL-Context. Our proposed QLSeg achieves competitive results on all these three datasets.

Accurate segmentation of liver tumor from multi-modality non-contrast images using a dual-stream multi-level fusion framework

The use of multi-modality non-contrast images (i.e., T1FS, T2FS and DWI) for segmenting liver tumors provides a solution by eliminating the use of contrast agents and is crucial for clinical diagnosis. However, this remains a challenging task to discover the most useful information to fuse multi-modality images for accurate segmentation due to inter-modal interference. In this paper, we propose a dual-stream multi-level fusion framework (DM-FF) to, for the first time, accurately segment liver tumors from non-contrast multi-modality images directly. Our DM-FF first designs an attention-based encoder–decoder to effectively extract multi-level feature maps corresponding to a specified representation of each modality. Then, DM-FF creates two types of fusion modules, in which a module fuses learned features to obtain a shared representation across multi-modality images to exploit commonalities and improve the performance, and a module fuses the decision evidence of segment to discover differences between modalities to prevent interference caused by modality’s conflict. By integrating these three components, DM-FF enables multi-modality non-contrast images to cooperate with each other and enables an accurate segmentation. Evaluation on 250 patients including different types of tumors from two MRI scanners, DM-FF achieves a Dice of 81.20%, and improves performance (Dice by at least 11%) when comparing the eight state-of-the-art segmentation architectures. The results indicate that our DM-FF significantly promotes the development and deployment of non-contrast liver tumor technology.

Multiple-in-Single-Out Object Detector Leveraging Spiking Neural Membrane Systems and Multiple Transformers.

Most existing multi-scale object detectors depend on multi-level feature maps. The Feature Pyramid Networks (FPN) is a significant architecture for object detection that utilizes these multi-level feature maps. However, the use of FPN also increases the detector's complexity. For object detection methods that only use a single-level feature map, the detection performance is limited to some extent because the single-level feature map cannot balance deep semantic information and shallow detail information. We introduce a novel detector- the Spiking Neural P Multiple-in-Single-out (SNPMiSo) detector to address these challenges. The SNPMiSo detector is constructed based on SNP-like neurons. In SNPMiSo, we employ two kinds of Transformers to boost the important features across different-level feature maps separately. After enhancing the features, we use an incremental upsampling module to upsample and merge the two feature maps. This combined feature map is input into the NAF dilated residual module and the NAF dual-branch detection head. This process allows us to extract multi-scale features and carry out detection tasks. Our tests show promising results: On the COCO dataset, SNPMiSo attains an Average Precision (AP) of 38.7, an improvement of 1.0 AP over YOLOF.In addition, SNPMiSo demonstrates a quicker detection speed, outperforming some advanced multi-level and single-level object detectors.

CariesXrays: Enhancing Caries Detection in Hospital-Scale Panoramic Dental X-rays via Feature Pyramid Contrastive Learning

Dental caries has been widely recognized as one of the most prevalent chronic diseases in the field of public health. Despite advancements in automated diagnosis across various medical domains, it remains a substantial challenge for dental caries detection due to its inherent variability and intricacies. To bridge this gap, we release a hospital-scale panoramic dental X-ray benchmark, namely “CariesXrays”, to facilitate the advancements in high-precision computer-aided diagnosis for dental caries. It comprises 6,000 panoramic dental X-ray images, with a total of 13,783 instances of dental caries, all meticulously annotated by dental professionals. In this paper, we propose a novel Feature Pyramid Contrastive Learning (FPCL) framework, that jointly incorporates feature pyramid learning and contrastive learning within a unified diagnostic paradigm for automated dental caries detection. Specifically, a robust dual-directional feature pyramid network (D2D-FPN) is designed to adaptively capture rich and informative contextual information from multi-level feature maps, thus enhancing the generalization ability of caries detection across different scales. Furthermore, our model is augmented with an effective proposals-prototype contrastive regularization learning (P2P-CRL) mechanism, which can flexibly bridge the semantic gaps among diverse dental caries with varying appearances, resulting in high-quality dental caries proposals. Extensive experiments on our newly-established CariesXrays benchmark demonstrate the potential of FPCL to make a significant social impact on caries diagnosis.

Deeply Unsupervised Patch Re-Identification for Pre-Training Object Detectors.

Unsupervised pre-training aims at learning transferable features that are beneficial for downstream tasks. However, most state-of-the-art unsupervised methods concentrate on learning global representations for image-level classification tasks instead of discriminative local region representations, which limits their transferability to region-level downstream tasks, such as object detection. To improve the transferability of pre-trained features to object detection, we present Deeply Unsupervised Patch Re-ID (DUPR), a simple yet effective method for unsupervised visual representation learning. The patch Re-ID task treats individual patch as a pseudo-identity and contrastively learns its correspondence in two views, enabling us to obtain discriminative local features for object detection. Then the proposed patch Re-ID is performed in a deeply unsupervised manner, appealing to object detection, which usually requires multi-level feature maps. Extensive experiments demonstrate that DUPR outperforms state-of-the-art unsupervised pre-trainings and even the ImageNet supervised pre-training on various downstream tasks related to object detection.

CLFR-Det: Cross-level feature refinement detector for tiny-ship detection in SAR images

Ship detection in synthetic aperture radar (SAR) images is an important and active topic, due to the characteristics of SAR images involving all-time and all-weather imaging. However, complex backgrounds caused by speckle noise and inshore land, coupled with information deficiency of tiny ships, pose a great challenge for tiny-ship detection in SAR images. To tackle this problem, we present a cross-level feature refinement detector (CLFR-Det) that utilizes features reflecting different levels and distinct semantics (classification and localization). To enrich the semantic information of tiny ships, our approach incorporates a cross-level modulated deformable convolution to aggregate features from relevant positions across multi-level feature maps. This is supported by a spatially-informed multi-scale feature refinement mechanism that combines the features for classification and those for localization. We implement a uniform IoU-weighted adaptive training sample selection method for equitably distributing the impact of positive samples from targets of various sizes during the training process. A generalized IoU loss between ground-truth and preliminary bounding box is further proposed to supervise the learning process of the CLFR-Det, with uncertainty weights incorporated to dynamically depict the levels of disparate losses, enabling adequate training across different tasks. Also, we construct a novel tiny SAR ship detection dataset to comprehensively evaluate the effectiveness of our system, in conjunction with the use of publicly available SSDD and HRSID datasets. Experimental investigations demonstrate that CLFR-Det generally surpasses state-of-the-art performance for multi-scale ship detection, particularly for the detection of tiny ships.

A deep learning-based dose prediction method for evaluation of radiotherapy treatment planning

[Purpose]To address the issue of low accuracy in dose distribution prediction in radiotherapy, we propose a deep learning-based model for predicting three-dimensional dose distribution in tumor radiation therapy. The model utilizes quantitative evaluation methods to assess the treatment plans. [Methods]We selected a dataset of 130 cervical cancer patients, including CT images and target region files. A deep learning U-Net model based on convolutional neural networks and residual blocks was employed to automatically extract multi-scale and multi-level feature maps of CT images, target regions, and anatomical structures of critical organs for intensity-modulated radiation therapy (IMRT) treatment plans and perform three-dimensional dose distribution prediction. Quantitative analysis methods, including error measures such as maximum dose (Dmax), mean dose (Dmean), V20, and D95, were used. [Results]For cervical cancer cases, DVH (dose-volume histogram) graphs were generated based on the evaluation results, directly reflecting the differences between the actual and predicted doses. The actual errors met the basic requirements, and a quantitative evaluation approach was used to optimize the dosimetric parameters. The specific quantification results are: DSC: 86.52 ± 9.31, 95% HD: 3.74 ± 1.49, JD: 0.112 ± 0.026, MSD: 0.067 ± 0.031. [Conclusion]Through training the deep learning model, we have successfully captured the complex nonlinear relationship between IMRT plan feature map parameters and three-dimensional dose distribution. In practical clinical applications, this trained model can accurately predict personalized three-dimensional dose distribution for new patients and effectively assess treatment plans in a quantitative manner. The source code is available at: https://github.com/xiebw9509/Radiotherapy_dose_prediction.

Transform-Based Feature Map Compression Method for Video Coding for Machines (VCM)

The burgeoning field of machine vision has led to the development by the Moving Picture Experts Group (MPEG) of a new type of compression technology called video coding for machines (VCM), to enhance machine recognition through video information compression. This research proposes a principal component analysis (PCA)-based compression methodology for multi-level feature maps extracted from the feature pyramid network (FPN) structure. Unlike current PCA-based studies that independently carry out PCA for each feature map, our approach employs a generalized basis matrix and mean vector derived from channel correlations by a generalized PCA process to eliminate the need for a PCA process. Further compression is achieved by amalgamating high-dimensional feature maps, capitalizing on the spatial redundancy within these multi-level feature maps. As a result, the proposed VCM encoder forgoes the PCA process, and the generalized data do not incur any compression loss. It only requires compressing the coefficients for each feature map using versatile video coding (VVC). Experimental results demonstrate superior performance by our method over all feature anchors for each machine vision task, as specified by the MPEG-VCM common test conditions, outperforming previous PCA-based feature map compression methods. Notably, it achieved an 89.3% BD-rate reduction for instance segmentation tasks.

EAAINet: An Element-Wise Attention Network With Global Affinity Information for Accurate Indoor Visual Localization

Visual localization, a vital component of many visual applications, has been tackled by scene coordinates regression (SCoRe) methods that leverage neural networks to predict scene coordinates, followed by a PnP algorithm to recover camera pose. However, these methods do not consider the relationship between image patches, known as relative features or affinity information, which is instrumental for network to perform complete scene parsing. Besides, owing to the visual similarity between image patches, these methods are weak in extracting reliable absolute features that represent the context information of the image patches, resulting in inferior localization performance. In response, we propose EAAINet that is based on classical SCoRe approaches and consists of two novel modules: the Global Affinity Aggregation Module (GAAM) and the Element-wise Attention Module (EAM). Specifically, GAAM employs an interval sampling strategy to sample image patches to construct sparse graph neural networks (GNNs), from which global affinity information between image patches is retrieved, hence ensuring precise scene parsing. EAM integrates multi-level features to generate reliable absolute features to regress accurate scene coordinates, with the key insight that the structure information is essential to differentiate similar image patches and the semantic information assists in modeling regression problems. Technically, EAM predicts element-wise soft attention masks to reconcile multi-level feature maps, enabling efficient feature fusion. Positional encoding and uncertainty modeling are also employed to enhance visual localization performance. Experimental results show that EAAINet significantly outperforms the state-of-the-arts on multiple benchmarks with faster speed and less model parameters. Source code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/DK-HU/EAAINet</uri> .

Multiscale features integration based multiple-in-single-out network for object detection

The single-level feature map-based object detection has been a challenging task due to the feature scale limitation. Therefore, enriching multiscale information of single-level features is considered a promising approach to deal with this challenge. Although most existing methods have attempted to augment the feature scale of single-level features, the detection performance is still unsatisfactory because these methods mine multiscale features only based on a one-level feature map. To address this problem, we propose a multiple-in-single-out network (MiSoNet) to integrate multiscale information from multilevel feature maps into a single-level feature map. To achieve this, MiSoNet’s key component is equipped with two cascaded modules: a multilevel feature integration module (MFIM) and a depthwise convolutional residual encoder (DWEncoder). Specifically, MFIM adaptively fuses features of inconsistent semantics and scales from multilevel feature maps. DWEncoder stacks several residual blocks with depthwise convolutions to extract multiscale contexts in the single feature map, which can further extend the scale range of the receptive fields. Extensive experiments are conducted on the Common Objects in Context (COCO) dataset, where the MiSoNet achieves a 41.0AP, which surpasses the YOLOF by 1.4AP with negligible computational overhead. Moreover, the MiSoNet, with fewer parameters and FLOPs, outperforms some advanced detectors based on the feature pyramid network.

FERA-Net: A Building Change Detection Method for High-Resolution Remote Sensing Imagery Based on Residual Attention and High-Frequency Features

Buildings can represent the process of urban development, and building change detection can support land use management and urban planning. However, existing building change detection models are unable to extract multi-scale building features effectively or fully utilize the local and global information of the feature maps, such as building edges. These defections affect the detection accuracy and may restrict further applications of the models. In this paper, we propose the feature-enhanced residual attention network (FERA-Net) to improve the performance of the ultrahigh-resolution remote sensing image change detection task. The FERA-Net is an end-to-end network with a U-shaped encoder–decoder structure. The Siamese network is used as the encoder with an attention-guided high-frequency feature extraction module (AGFM) extracting building features and enriching detail information, and the decoder applies a feature-enhanced skip connection module (FESCM) to aggregate the enhanced multi-level differential feature maps and gradually recover the change feature maps in this structure. The FERA-Net can generate predicted building change maps by the joint supervision of building change information and building edge information. The performance of the proposed model is tested on the WHU-CD dataset and the LEVIR-CD dataset. The experimental results show that our model outperforms the state-of-the-art models, with 93.51% precision and a 92.48% F1 score on the WHU-CD dataset, and 91.57% precision and an 89.58% F1 score on the LEVIR-CD dataset.

A Content-Adaptive Hierarchical Deep Learning Model for Detecting Arbitrary-Oriented Road Surface Elements Using MLS Point Clouds

Accurate and automatic detection of road surface element (such as road marking or manhole cover) information is the basis and key to many applications. To efficiently obtain the information of road surface element, we propose a content-adaptive hierarchical deep learning model to detect arbitrary-oriented road surface elements from mobile laser scanning (MLS) point clouds. In the model, we design a densely connected feature integration module (DCFM) to connect and reorganize feature maps of each stage in the backbone network. Besides, we propose a hierarchical prediction module (HPM) to innovatively use the reorganized feature maps to recognize different types of road surface elements, and thus, semantic information of road surface element can be adaptively expressed on multilevel feature maps. We also add a cascade structure (CS) in the head of model to detect the target efficiently, which can learn the offset between the predicted minimum bounding box of road surface element and ground truth. In experiments, we prove that the proposed method mainly contributed by HPM can maintain robust detection performance, even in the cases of unbalanced category number or overlapping of road surface elements. The experiments also prove that the proposed DCFM can improve the recognition effects of small targets. The CS for predicting boundary offset can detect each target more accurately. We also integrate the designed modules into some rotation detectors, e.g., the EAST and R3Det, and achieve the state-of-the-art results in three road scenes with different categories and uneven distribution of road surface elements, which further shows the effectiveness of the proposed method.

DSD-MatchingNet: Deformable sparse-to-dense feature matching for learning accurate correspondences

BackgroundExploring correspondences across multiview images is the basis of various computer vision tasks. However, most existing methods have limited accuracy under challenging conditions. MethodTo learn more robust and accurate correspondences, we propose DSD-MatchingNet for local feature matching in this study. First, we develop a deformable feature extraction module to obtain multilevel feature maps, which harvest contextual information from dynamic receptive fields. The dynamic receptive fields provided by the deformable convolution network ensure that our method obtains dense and robust correspondence. Second, we utilize sparse-to-dense matching with symmetry of correspondence to implement accurate pixel-level matching, which enables our method to produce more accurate correspondences. ResultExperiments show that our proposed DSD-MatchingNet achieves a better performance on the image matching benchmark, as well as on the visual localization benchmark. Specifically, our method achieved 91.3% mean matching accuracy on the HPatches dataset and 99.3% visual localization recalls on the Aachen Day-Night dataset.

A Transformer-Based Network for Change Detection in Remote Sensing Using Multiscale Difference-Enhancement.

Recently, transformer-based change detection methods have achieved remarkable performance by sophisticated architectures for extracting powerful feature representations. However, due to the existence of various noises in bitemporal images, there are problems such as loss of semantic objects and incompleteness that will occur in change detection. The existing transformer-based approaches do not fully address this issue. In this paper, we propose a transformer-based multiscale difference-enhancement U-shaped network and call it TUNetCD, for change detection in remote sensing. The encoder, which is composed of a multilayer Swin-Transformer block structure, can extract multilevel feature maps, further enhance these multilevel feature maps using a Swin-Transformer feature difference map processing module, and finally obtain the final change map using a lightweight decoder. We conducted comprehensive experiments on two publicly available benchmark datasets, LEVIR-CD and DSIFN-CD, to verify the effectiveness of the method, and our method outperformed other advanced transformer-based methods.

Residual attentive feature learning network for salient object detection

Salient object detection is one of the basic challenging problems in the area of computer vision. Traditional saliency models usually utilize handcrafted features and various prior cues to locate and segment salient objects from complicated surroundings. Recently, with the development of convolutional neural networks (CNNs), most of existing approaches explore the combination of multi-level feature maps for improving detection performance. However, how to learn powerful feature is still a challenge, and the multi-level and multi-scale feature maps are usually aggregated without distinction. In this paper, we propose a novel salient object detection approach named residual attentive feature learning (RAFL) to uniformly highlight salient objects and effectively suppress background noises. In our proposed RAFL, a global perception (GP) module is designed to obtain rich global features of input image. A pyramid feature extraction (PFE) module is utilized to capture the rich context of side-output information. Moreover, an attentive feature selection (AFS) module is applied to refine these side-output features and generate attentive features, so as to find out the important feature channels having high responses to the salient objects. Finally, a residual feature learning (RFL) module is proposed to make the shallow layers generate more accurate saliency maps with the help of the deep layers which contain more global semantic information. Experimental results show that our proposed approach performs favorably against 18 state-of-the-art representative methods on five benchmark datasets.

STCNet: spatiotemporal cross network for industrial smoke detection

Industrial smoke emissions present a serious threat to natural ecosystems and human health. Prior works have shown that using computer vision techniques to identify smoke is a low-cost and convenient method. However, translucent smoke detection is a challenging task because of the irregular contours and complex motion state. To overcome these problems, we propose a novel spatiotemporal cross network (STCNet) to recognize industrial smoke emissions. The proposed STCNet involves a spatial pathway to extract appearance features and a temporal pathway to capture smoke motion information. Our STCNet is more targeted and goal oriented for dealing with translucent, nonrigid smoke objects. The spatial path can easily recognize obvious nonsmoking objects such as trees and buildings, and the temporal path can highlight the obscure traces of motion smoke. Our STCNet achieves the mutual guidance of multilevel spatiotemporal information by bidirectional feature fusion on multilevel feature maps. Extensive experiments on public datasets show that our STCNet achieves clear improvements against the best competitors by 6.2%. We also perform in-depth ablation studies on STCNet to explore the impacts of different feature fusion methods for the entire model. The code will be available at https://github.com/Caoyichao/STCNet.

Difference in oral microbial composition between chronic periodontitis patients with and without diabetic nephropathy

BackgroundTo investigate the difference in the structural composition of salivary flora between chronic periodontitis patients with and without diabetic nephropathy (DN).MethodsThirty salivary samples of 15 chronic periodontitis patients with DN (DN group) and 15 chronic periodontitis patients with diabetes but without DN (DM group) were subjected to pyrosequencing of polymerase chain reaction-amplified 16 s ribosomal RNA genes. After diversity testing, the differential flora were analyzed. The sequencing results were compared with GenBank database to determine the type of differential flora using species composition analysis, hierarchical cluster analysis, principal co-ordinate analysis, and species difference analysis.ResultsThere were significant between-group differences with respect to Gemella, Selenomonas spp, Lactobacillales_unclassified, Bacteria-unclassified and Abiotrophia (p < 0.05). Compared with DM group, the relative abundance of Selenomonas spp. in DN group was significantly higher; the area under the receiver operating characteristic curve of Selenomonas spp. was 0.713 (P < 0.05). Multi-level biological identification and feature maps indicated that Selenomonas spp. might be used as a potential biomarker for DN patients. On binary logistic regression analysis, increase of Selenomonas spp. was related with DN.ConclusionsWe found significant between-group differences in the structural composition of oral flora. The increase in the relative abundance of Selenomonas spp. may be associated with DN in patients with chronic periodontitis.

MINet: Multilevel Inheritance Network-Based Aerial Scene Classification

Scene classification of aerial images is the basis of automatic recognition of complex scenes, and it is also a challenging computer vision task. In recent years, with the rapid development of deep learning, the semantic feature extraction method based on a convolutional neural network (CNN) has made great progress. Moreover, a recent study indicates that combining the semantic information of deep-layer features with the detailed texture information of shallow-layer features in CNN can further improve the performance of classification. In this letter, an end-to-end multilevel feature-based network named multilevel inheritance network (MINet) is proposed for aerial scene classification. First, the feature extraction module based on the feature pyramid network (FPN) is used to get multilevel feature maps. In the process of merging shallow features, high-level semantics of deep-layer are inherited. Then, an attention mechanism is added after the multilevel features to reduce the interference of redundant information and noise. Finally, we use a feature fusion module to automatically learn the weight of each feature layer and make a comprehensive decision. The effectiveness of the proposed method is verified in AID, WHU-RS19 and NWPU-RESISC45 datasets. Results show that the proposed method achieves competitive classification accuracy.

Collection-CAM: A Faster Region-Based Saliency Method Using Collection-Wise Mask Over Pyramidal Features

Due to the black-box nature of deep networks, making explanations of their decision-making is extremely challenging. A solution is using post-hoc attention mechanisms with the deep network to verify the decision basis. However, those methods have problems such as gradient noise and false confidence. In addition, existing saliency methods either have limited performance by using only the last convolution layer or suffer from large computational overhead. In this work, we propose the Collection-CAM, which generates an attention map with low computational overhead while utilizing multi-level feature maps. First, the Collection-CAM searches for the most appropriate form of the partition through bottom-up clustering and clustering validation process. Then the Collection-CAM applies different pre-processing procedures on the shallow feature map and final feature map to overcome the false positiveness when applied without distinction. By combining collection-wise masks according to their contribution to the confidence score, the Collection-CAM completes the attention map generation process. Experimental results on ImageNet1k, UC Merced, and CUB dataset and various deep network models demonstrate that the Collection-CAM not only can synthesize a saliency map with a better visual explanation but also requires significantly lower computational overhead compared to those of region-based saliency methods.