Multi-scale Context Research Articles

Multi‐source heterogeneous iris segmentation method based on lightweight convolutional neural network

Iris segmentation is a critical step in the iris recognition system. Since the quality of iris database taken under different camera sensors varies greatly, thus most existing iris segmentation methods are designed for a particular collection device. Meanwhile, many iris segmentation methods based on convolutional neural networks (CNNs) require a lot of computational costs and hardware costs (storage space), which are not suitable for deploying on low-performance devices. To address the above problems, an accurate and efficient heterogeneous iris segmentation network is proposed in this paper. First, the authors design an efficient feature extraction network, which combines depth-wise separable convolution with traditional convolution to greatly reduce model parameters and computational cost while maintaining segmentation accuracy. Then, a Multi-scale Context Information Extraction Module (MCIEM) is proposed to extract multi-scale spatial information at a more granular level and enhance the discriminability of the iris region. Finally, a Multi-layer Feature Information Fusion Module (MFIFM) is proposed to reduce the loss of information during the downsampling process. Experimental results on multi-source heterogeneous iris database show that the proposed network can not only achieve state-of-the-art performance but is also more efficient in terms of required parameters, calculated load, and storage space.

GLFormer: Global and Local Context Aggregation Network for Temporal Action Detection

As the core component of video analysis, Temporal Action Localization (TAL) has experienced remarkable success. However, some issues are not well addressed. First, most of the existing methods process the local context individually, without explicitly exploiting the relations between features in an action instance as a whole. Second, the duration of different actions varies widely; thus, it is difficult to choose the proper temporal receptive field. To address these issues, this paper proposes a novel network, GLFormer, which can aggregate short, medium, and long temporal contexts. Our method consists of three independent branches with different ranges of attention, and these features are then concatenated along the temporal dimension to obtain richer features. One is multi-scale local convolution (MLC), which consists of multiple 1D convolutions with varying kernel sizes to capture the multi-scale context information. Another is window self-attention (WSA), which tries to explore the relationship between features within the window range. The last is global attention (GA), which is used to establish long-range dependencies across the full sequence. Moreover, we design a feature pyramid structure to be compatible with action instances of various durations. GLFormer achieves state-of-the-art performance on two challenging video benchmarks, THUMOS14 and ActivityNet 1.3. Our performance is 67.2% and 54.5% AP@0.5 on the datasets THUMOS14 and ActivityNet 1.3, respectively.

SLT-Net: A codec network for skin lesion segmentation

Automatic segmentation of skin lesions is beneficial for improving the accuracy and efficiency of melanoma diagnosis. However, due to variation in the size and shape of the lesion areas and the low contrast between the edges of the lesion and the normal skin tissue, this task is very challenging. The traditional convolutional neural network based on codec structure lacks the capability of multi-scale context information modeling and cannot realize information interaction of skip connections at the various levels, which limits the segmentation performance. Therefore, a new codec structure of skin lesion Transformer network (SLT-Net) was proposed and applied to skin lesion segmentation in this study. Specifically, SLT-Net used CSwinUnet as the codec to model the long-distance dependence between features and used the multi-scale context Transformer (MCT) as the skip connection to realize information interaction between skip connections across levels in the channel dimension. We have performed extensive experiments to verify the effectiveness and superiority of our proposed method on three public skin lesion datasets, including the ISIC-2016, ISIC-2017, and ISIC-2018. The DSC values on the three data sets reached 90.45%, 79.87% and 82.85% respectively, higher than most of the state-of-the-art methods. The excellent performance of SLT-Net on these three datasets proved that it could improve the accuracy of skin lesion segmentation, providing a new benchmark reference for skin lesion segmentation tasks. The code is available at https://github.com/FengKaili-fkl/SLT-Net.git.

MonoPoly: A practical monocular 3D object detector

3D object detection plays a pivotal role in driver assistance systems and has practical requirements for small storage and fast inference. Monocular 3D detection alternatives abandon the complexity of LiDAR setup and pursues the effectiveness and efficiency of the vision scheme. In this work, we propose a set of anchor-free monocular 3D detectors called MonoPoly based on the keypoint paradigm. Specifically, we design a polynomial feature aggregation sampling module to extract multi-scale context features for auxiliary training and alleviate classification and localization misalignment through an attention-aware loss. Extensive experiments show that the proposed MonoPoly series achieves an excellent trade-off between performance and model size while maintaining real-time efficiency on KITTI and nuScenes datasets.

Monocular depth estimation with spatially coherent sliced network

Monocular depth estimation from a single still image is one of the most challenging fundamental problems in scenes understanding. As a pixel level regression problem, the inherent continuous large range of depth itself causes three main difficulties: (1) The unbalance between large value and small value during regressing; (2) Exploiting multi-scale contextual information; (3) Preserving spatial and semantic structures. To overcome these difficulties, this paper presents a novel spatially coherent sliced network for monocular depth estimation (SCS-Net). It first uses feature pyramids network to form the feature fusions of hierarchical feature maps. Then, the depth is sliced to supervise the estimation in different ranges generated from the feature fusions of multi-scale contexts. The holistic depth is invoked as the supervised signal of aggregated feature maps to ensure the learning of global structure of the scene. The self-spatial-attention mechanism further takes advantage of the semantics of objects and the spatial pixel relations to maintain the coherence of space and semantics in depth sliced estimation. Finally, the regulations of second-order depth information further make the estimated boundaries not too smooth. Numerous ablation experiments and comparisons on three popular indoor and outdoor benchmark datasets indicate the effectiveness and robustness of the proposed approach. • Spatially coherent sliced network regresses sliced depth from feature fusions pyramidally. • Our network introduces the scale and shift regression to maintain spatial coherence. • Self-spatial-attention mechanism refine potentially compromised semantic coherence. • Experimental results show competitive results and promising performance in urban road scenes.

A multi-branch hierarchical attention network for medical target segmentation

Medical imaging techniques have been widely used in modern clinical disease diagnosis and treatment programs. The captured medical images can well reflect the conditions of the human body tissues, which are significantly helpful to the doctors to determine the existence or the severity of the disease. In this paper, we develop a hierarchical attentive high-resolution convolutional network (AttHRNet) for segmenting targets of interest from medical images aiming to improve the automated processing standard and the intelligent interpretation quality of the medical images. The AttHRNet is an improved version of the high-resolution network (HRNet) structure with three novel modules. First, built with an improved HRNet structure assisted by a multiscale context augmentation (MSCA) module as the feature extraction backbone, the AttHRNet can produce a set of high-quality, strong-semantic feature maps at different resolutions. The MSCA module functions to reduce the information loss during feature downsampling. Second, designed with an effective feature attention principle, the feature encoding quality in each branch can be significantly promoted by concentrating on the informative and salient feature encodings across both channels and spatial locations. Furthermore, formulated with a hierarchical segmentation scheme, the output feature maps can be further augmented by including the semantic-level category exploitation (SLCE) module with a global perspective. The SLCE module allows the information from lower resolution segmentations to inform higher resolution segmentations. Through quantitative examinations, visual verifications, and comparative evaluations on four medical image datasets, we convince the promising applicability and competitive superiority of the AttHRNet in medical target segmentation issues.

Warp-based edge feature reinforcement network for medical image segmentation.

Rapid and accurate segmentation of medical images can provide important guidance in the early stages of life-threatening diseases. However, fuzzy edges and high similarity with the background in images usually cause undersegmentation or oversegmentation. To solve these problems. We propose a novel edge features-reinforcement (EFR) module that uses relative frequency changes before and after warping images to extract edge information. Then, the EFR module leverages deep features to guide shallow features to produce a band-shaped edge attention map for reinforcing the edge region of all channels. We also propose a multiscale context exploration (MCE) module to fuse multiscale features and to extract channel and spatial correlations, which allows a model to focus on the parts that contribute most to the final segmentation. We construct EFR-Net by embedding EFR and MCE modules on the encoder-decoder architecture. We verify EFR-Net's performance with four medical datasets: retinal vessel segmentation dataset DRIVE, endoscopic polyp segmentation dataset CVC-ClinicDB, dermoscopic image dataset ISIC2018, and aortic true lumen dataset Aorta-computed tomography (CT). The proposed model achieves Dice similarity coefficients (DSCs) of 81.61%, 92.87%, 89.87%, and 96.98% on DRIVE, CVC-ClinicDB, ISIC2018, and Aorta-CT, respectively, which are better than those of current mainstream methods. In particular, the DSC of polyp segmentation increased by 3.87%. Through quantitative and qualitative research, our method is determined to surpass current mainstream segmentation methods, and EFR modules can effectively improve the edge prediction effect of color images and CT images. The proposed modules are easily embedded in other encoder-decoder architectures, which has the potential to be applied and expanded.

Scale adaption-guided human face detection

Anchor-free based object detection has recently seen important progress benefiting from the advances in convolution neural networks. However, the detection performance for human faces is not so satisfactory. First of all, many existing anchor-free methods only focus on a certain scale of the feature map, such a mechanism often fails to perceive the important multi-scale context, resulting in a low recall rate of faces with large scale variations. To solve this problem, we propose to boost the face detection by adaptive learning to perceive the focal scale. To be specific, we design an online scale adaptation strategy to heuristically guide each layer detector to detect faces of different scales in multi-branch structures, which reduces outliers and improves recall rates. In additional, we also argue that the detection head with single convolution layer widely used in anchor-free methods is not robust enough to image context. Therefore, we augment the network by a context-aware detection module. The module dynamically generates different detectors for different input images based on their context to adapt to their image features, reducing the dependence on feature extraction ability of backbone network, and avoiding feature deviations in different scenes. Extensive experiments demonstrate that our method achieves significant performance gains compared to previous anchor-free methods and is comparable to the most advanced anchor-based face detection methods.

Detection Range Enhancement for Φ-OTDR Using Semantic Image Segmentation

Aiming to reduce the complexity of Φ-OTDR system introduced by traditional distributed optical amplification technology in the pursuit of long-distance sensing, a novel detection range enhancement method with semantic image segmentation based on deep learning is presented. In this study, the Rayleigh backscattered curves for detecting vibration signals are used to construct the temporal-spatial image. The encoder-decoder structure and atlas spatial pyramid pool model are adopted to capture the boundary of vibration area and multi-scale context information from the temporal-spatial image. The semantic extraction of vibration position is then realized by binary classification of image pixels. In experiments, this method significantly improves the signal-to-noise ratio of the vibration location results. The signal-to-noise ratio can reach 37.84 dB, 34.28 dB, 34.09 dB, and 32.17 dB at 4 km, 10 km, 20 km, and 40 km, respectively. With this approach, the vibration detection range is enhanced to about 80 km without adding any traditional distributed optical amplification structure. Furthermore, within this enhanced detection range, the vibration signals with different frequencies and different intensities can be located and multi-point vibration can also be verified experimentally. This method provides a novel means for the detection range enhancement of Φ-OTDR system.

AMFuse: Add–Multiply-Based Cross-Modal Fusion Network for Multi-Spectral Semantic Segmentation

Multi-spectral semantic segmentation has shown great advantages under poor illumination conditions, especially for remote scene understanding of autonomous vehicles, since the thermal image can provide complementary information for RGB image. However, methods to fuse the information from RGB image and thermal image are still under-explored. In this paper, we propose a simple but effective module, add–multiply fusion (AMFuse) for RGB and thermal information fusion, consisting of two simple math operations—addition and multiplication. The addition operation focuses on extracting cross-modal complementary features, while the multiplication operation concentrates on the cross-modal common features. Moreover, the attention module and atrous spatial pyramid pooling (ASPP) modules are also incorporated into our proposed AMFuse modules, to enhance the multi-scale context information. Finally, in the UNet-style encoder–decoder framework, the ResNet model is adopted as the encoder. As for the decoder part, the multi-scale information obtained from our proposed AMFuse modules is hierarchically merged layer-by-layer to restore the feature map resolution for semantic segmentation. The experiments of RGBT multi-spectral semantic segmentation and salient object detection demonstrate the effectiveness of our proposed AMFuse module for fusing the RGB and thermal information.

Predicting RNA solvent accessibility from multi-scale context feature via multi-shot neural network

Knowledge of RNA solvent accessibility has recently become attractive due to the increasing awareness of its importance for key biological process. Accurately predicting the solvent accessibility of RNA is crucial for understanding its 3D structure and biological function. In this study, we develop a novel computational method, termed M2pred, for accurately predicting the solvent accessibility of RNA from sequence-based multi-scale context feature. In M2pred, three single-view features, i.e., base-pairing probabilities, position-specific frequency matrix, and a binary one-hot encoding, are first generated as three feature sources, and immediately concatenated to engender a super feature. Secondly, for the super feature, the matrix-format features of each nucleotide are extracted using an initialized sliding window technique, and regularly stacked into a cube-format feature. Then, using multi-scale context feature extraction strategy, a pyramid feature constructed of contextual feature of four scales related to target nucleotides is extracted from the cube-format feature. Finally, a customized multi-shot neural network framework, which is equipped with four different scales of receptive fields mainly integrating several residual attention blocks, is designed to dig discrimination information from the contextual pyramid feature. Experimental results demonstrate that the proposed M2pred achieve a high prediction performance and outperforms existing state-of-the-art prediction methods of RNA solvent accessibility.

Content‐augmented feature pyramid network with light linear spatial transformers for object detection

As one of the prevalent components, feature pyramid network (FPN) is widely used in current object detection models for improving multi-scale object detection performance. However, its feature fusion mode is still in a misaligned and local manner, thus limiting the representation power. To address the inherited defects of FPN, a novel architecture termed content-augmented feature pyramid network (CA-FPN) is proposed in this paper. Firstly, a global content extraction module (GCEM) is proposed to extract multi-scale context information. Secondly, lightweight linear spatial Transformer connections are added in the top-down pathway to augment each feature map with multi-scale features, where a linearized approximate self-attention function is designed for reducing model complexity. By means of the self-attention mechanism in Transformer, it is no longer needed to align feature maps during feature fusion, thus solving the misaligned defect. By setting the query scope to the entire feature map, the local defect can also be solved. Extensive experiments on COCO and PASCAL VOC datasets demonstrated that the CA-FPN outperforms other FPN-based detectors without bells and whistles and is robust in different settings.

Landmark Localization for Cephalometric Analysis Using Multiscale Image Patch-Based Graph Convolutional Networks.

Accurate and robust cephalometric image analysis plays an essential role in orthodontic diagnosis, treatment assessment and surgical planning. This paper proposes a novel landmark localization method for cephalometric analysis using multiscale image patch-based graph convolutional networks. In detail, image patches with the same size are hierarchically sampled from the Gaussian pyramid to well preserve multiscale context information. We combine local appearance and shape information into spatialized features with an attention module to enrich node representations in graph. The spatial relationships of landmarks are built with the incorporation of three-layer graph convolutional networks, and multiple landmarks are simultaneously updated and moved toward the targets in a cascaded coarse-to-fine process. Quantitative results obtained on publicly available cephalometric X-ray images have exhibited superior performance compared with other state-of-the-art methods in terms of mean radial error and successful detection rate within various precision ranges. Our approach performs significantly better especially in the clinically accepted range of 2 mm and this makes it suitable in cephalometric analysis and orthognathic surgery.

NPCNet: Jointly Segment Primary Nasopharyngeal Carcinoma Tumors and Metastatic Lymph Nodes in MR Images.

Nasopharyngeal carcinoma (NPC) is a malignant tumor whose survivability is greatly improved if early diagnosis and timely treatment are provided. Accurate segmentation of both the primary NPC tumors and metastatic lymph nodes (MLNs) is crucial for patient staging and radiotherapy scheduling. However, existing studies mainly focus on the segmentation of primary tumors, eliding the recognition of MLNs, and thus fail to comprehensively provide a landscape for tumor identification. There are three main challenges in segmenting primary NPC tumors and MLNs: variable location, variable size, and irregular boundary. To address these challenges, we propose an automatic segmentation network, named by NPCNet, to achieve segmentation of primary NPC tumors and MLNs simultaneously. Specifically, we design three modules, including position enhancement module (PEM), scale enhancement module (SEM), and boundary enhancement module (BEM), to address the above challenges. First, the PEM enhances the feature representations of the most suspicious regions. Subsequently, the SEM captures multiscale context information and target context information. Finally, the BEM rectifies the unreliable predictions in the segmentation mask. To that end, extensive experiments are conducted on our dataset of 9124 samples collected from 754 patients. Empirical results demonstrate that each module realizes its designed functionalities and is complementary to the others. By incorporating the three proposed modules together, our model achieves state-of-the-art performance compared with nine popular models.

UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-Wise Perspective with Transformer

Most recent semantic segmentation methods adopt a U-Net framework with an encoder-decoder architecture. It is still challenging for U-Net with a simple skip connection scheme to model the global multi-scale context: 1) Not each skip connection setting is effective due to the issue of incompatible feature sets of encoder and decoder stage, even some skip connection negatively influence the segmentation performance; 2) The original U-Net is worse than the one without any skip connection on some datasets. Based on our findings, we propose a new segmentation framework, named UCTransNet (with a proposed CTrans module in U-Net), from the channel perspective with attention mechanism. Specifically, the CTrans (Channel Transformer) module is an alternate of the U-Net skip connections, which consists of a sub-module to conduct the multi-scale Channel Cross fusion with Transformer (named CCT) and a sub-module Channel-wise Cross-Attention (named CCA) to guide the fused multi-scale channel-wise information to effectively connect to the decoder features for eliminating the ambiguity. Hence, the proposed connection consisting of the CCT and CCA is able to replace the original skip connection to solve the semantic gaps for an accurate automatic medical image segmentation. The experimental results suggest that our UCTransNet produces more precise segmentation performance and achieves consistent improvements over the state-of-the-art for semantic segmentation across different datasets and conventional architectures involving transformer or U-shaped framework. Code: https://github.com/McGregorWwww/UCTransNet.

SBDF-Net: A versatile dual-branch fusion network for medical image segmentation

In the field of medical image analysis, image segmentation can help doctors diagnose diseases and plan treatments. U-net has become an important network in biomedical image segmentation. Inspired by U-Net, we propose a dual-branch encoder for aggregating multi-scale context information. A novelty Shuffle Grouped Fusion Module is used to fuse cross-scale information between dual branches. In addition, Skip Connection + calibrates the features extracted by encoder to optimize the feature mapping. Finally, Three-branch Down-sampling Block and Two-branch Up-sampling Block are designed to reduce the feature loss produced by sampling operations. We have evaluated the performance of our network on four datasets. The IoU, Dice and Sensitivity of the model reach 86.45%, 92.95% and 93.36% on the 2018 Data Science Bowl dataset, 81.85%, 89.35% and 88.90% on the GLAS dataset, 80.92%, 87.63% and 87.19% on the Kvasir-SEG dataset, 91.54%, 95.48% and 94.72% on the Aortic Dissection dataset. The experimental results show that our proposed model is superior to U-Net and other advanced segmentation networks in many metrics. The proposed model is available at https://github.com/1998supper/SBDF-Net.

Revisiting moment-closure methods with heterogeneous multiscale population models

Stochastic chemical kinetics at the single-cell level give rise to heterogeneous populations of cells even when all individuals are genetically identical. This heterogeneity can lead to nonuniform behaviour within populations, including different growth characteristics, cell-fate dynamics, and response to stimuli. Ultimately, these diverse behaviours lead to intricate population dynamics that are inherently multiscale: the population composition evolves based on population-level processes that interact with stochastically distributed single-cell states. Therefore, descriptions that account for this heterogeneity are essential to accurately model and control chemical processes. However, for real-world systems such models are computationally expensive to simulate, which can make optimisation problems, such as optimal control or parameter inference, prohibitively challenging. Here, we consider a class of multiscale population models that incorporate population-level mechanisms while remaining faithful to the underlying stochasticity at the single-cell level and the interplay between these two scales. To address the complexity, we study an order-reduction approximations based on the distribution moments. Since previous moment-closure work has focused on the single-cell kinetics, extending these techniques to populations models prompts us to revisit old observations as well as tackle new challenges. In this extended multiscale context, we encounter the previously established observation that the simplest closure techniques can lead to non-physical system trajectories. Despite their poor performance in some systems, we provide an example where these simple closures outperform more sophisticated closure methods in accurately, efficiently, and robustly solving the problem of optimal control of bioproduction in a microbial consortium model.

A Deep Learning-Based Semantic Segmentation Architecture for Autonomous Driving Applications

In recent years, the development of smart transportation has accelerated research on semantic segmentation as it is one of the most important problems in this area. A large receptive field has always been the center of focus when designing convolutional neural networks for semantic segmentation. A majority of recent techniques have used maxpooling to increase the receptive field of a network at an expense of decreasing its spatial resolution. Although this idea has shown improved results in object detection applications, however, when it comes to semantic segmentation, a high spatial resolution also needs to be considered. To address this issue, a new deep learning model, the M-Net is proposed in this paper which satisfies both high spatial resolution and a large enough receptive field while keeping the size of the model to a minimum. The proposed network is based on an encoder-decoder architecture. The encoder uses atrous convolution to encode the features at full resolution, and instead of using heavy transposed convolution, the decoder consists of a multipath feature extraction module that can extract multiscale context information from the encoded features. The experimental results reported in the paper demonstrate the viability of the proposed scheme.

HrreNet: semantic segmentation network for moderate and high-resolution satellite images

ABSTRACT Semantic segmentation is a commonly used intelligent interpretation method. However, semantic segmentation via deep learning usually faces several problems: resolution loss due to downsampling, difficulty in obtaining context information, and imprecise object boundaries. These problems lead to a poor performance when using satellite images instead of conventional images and limit the value of intelligent interpretation of satellite images in practical application. In this study, we construct a Qilian Ecological Resource Extraction Dataset (QERED) with multiple scene satellite images of Qilian after preprocessing via correction, fusion, mosaic, and annotation. We then propose a segmentation network, namely, High-resolution Resource Extracting Network (HrreNet), by using high-resolution feature representation, multi-scale context fusion, boundary refinement with relearning, and structural similarity loss. Experiments on QERED show that HrreNet greatly improves performance on small-size objects and slightly improves performance on larger-size objects. HrreNet achieves the best result of 73.36% mIoU on a moderate-resolution satellite dataset CCFD and 80.87% mIoU on a high-resolution satellite dataset Vaihingen.

CIMFNet: Cross-Layer Interaction and Multiscale Fusion Network for Semantic Segmentation of High-Resolution Remote Sensing Images

Semantic segmentation of remote sensing images has received increasing attention in recent years; however, using a single imaging modality limits the segmentation performance. Thus, digital surface models have been integrated into semantic segmentation to improve performance. Nevertheless, existing methods based on neural networks simply combine data from the two modalities, mostly neglecting the similarities and differences between multimodal features. Consequently, the complementarity between multimodal features cannot be exploited, and excess noise is introduced during feature processing. To solve these problems, we propose a multimodal fusion module to explore the similarities and differences between features from the two information modalities for adequate fusion. In addition, although downsampling operations such as pooling and striding can improve the feature representativeness, they discard spatial details and often lead to segmentation errors. Thus, we introduce hierarchical feature interactions to mitigate the adverse effects of downsampling and introduce a two-way interactive pyramid pooling module to extract multiscale context features for guiding feature fusion. Extensive experiments performed on two benchmark datasets show that the proposed network integrating our novel modules substantially outperforms state-of-the-art semantic segmentation methods. The code and results can be found at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/NIT-JJH/CIMFNet</uri> .