Image Context Information Research Articles

Bi-directional information interaction for multi-modal 3D object detection in real-world traffic scenes

Multimodal 3D object detection methods are poorly adapted to real-world traffic scenes due to sparse distribution of point clouds and misalignment multimodal data during actual collection. Among the existing methods, they focus on high-quality open-source datasets, with performance relying on the accurate structural representation of point clouds and the precise mapping relationship between point clouds and images. To solve the above challenges, this paper proposes a multimodal feature-level fusion method based on the bi-directional interaction between image and point cloud. To overcome the sparsity issue in asynchronous multi-modal data, a point cloud densification scheme based on visual guidance and point cloud density guidance is proposed. This scheme can generate object-level virtual point clouds even when the point cloud and image are misaligned. To deal with the unalignment issue between point cloud and image, a bi-directional interaction module based on image-guided interaction with key points of point clouds and point cloud-guided interaction with image context information is proposed. It achieves effective feature fusion even when the point cloud and image are misaligned. The experiments on the VANJEE and KITTI datasets demonstrated the effectiveness of the proposed method, with average precision improvements of 6.20% and 1.54% compared to the baseline.

Extracting shallow water depth from the fusion of multi-temporal ICESat-2 data and multi-spectral imageries

With the development of satellite remote sensing and laser altimetry data, the fusion of laser altimetry data and multi-spectral images for shallow water depth inversion has become an economically convenient way. However, there are few methods that take into account the temporal dimension of data to integrate the results of water depth inversion in multiple temporal phases. In response to this issue, this article proposed a shallow water depth inversion method that integrates multi-source and multi-temporal remote sensing data. This method utilized the random forest (RF) algorithm to estimate the water depth values at different time, taking into account the contextual information of Sentinel-2 imagery and using the overall least squares as the theoretical model to fuse the multi-temporal water depth inversion results. This article took the Yongle Islands in the South China Sea as the research area and conducted the shallow water inversion experiments using ICESat-2 (Ice, Cloud and Land Elevation Satellite 2) data from 5 temporal phases and one Sentinel-2 imagery. The results show that the mean value of RMSE (root mean square error) and R2 (determination coefficient) of the proposed method using single temporal imagery was 1.53m and 0.7610, which outperformed the inversion accuracy of traditional methods that ignore image context information. The RMSE and R2 of the multi-temporal fusion model was 1.15m and 0.8622, which was 0.08m and 0.0199m higher than existing median filtering fusion method.

Sparse spatial transformers for few-shot learning

Learning from limited data is challenging because data scarcity leads to a poor generalization of the trained model. A classical global pooled representation will probably lose useful local information. Many few-shot learning methods have recently addressed this challenge using deep descriptors and learning a pixel-level metric. However, using deep descriptors as feature representations may lose image contextual information. Moreover, most of these methods independently address each class in the support set, which cannot sufficiently use discriminative information and task-specific embeddings. In this paper, we propose a novel transformer-based neural network architecture called sparse spatial transformers (SSFormers), which finds task-relevant features and suppresses task-irrelevant features. Particularly, we first divide each input image into several image patches of different sizes to obtain dense local features. These features retain contextual information while expressing local information. Then, a sparse spatial transformer layer is proposed to find spatial correspondence between the query image and the full support set to select task-relevant image patches and suppress task-irrelevant image patches. Finally, we propose using an image patch-matching module to calculate the distance between dense local representations, thus determining which category the query image belongs to in the support set. Extensive experiments on popular few-shot learning benchmarks demonstrate the superiority of our method over state-of-the-art methods. Our source code is available at \url{https://github.com/chenhaoxing/ssformers}.

Garment Segmentation Network Based on Texture and Semantic Decoding Module

With the great development of the apparel e-commerce industry, more and more people are engaged in this field. In reality, clothing is easily affected by background, perspective, occlusion, and other factors, so clothing segmentation is not accurate enough. This paper proposes a clothing segmentation network based on the texture and semantic decoding module of clothing. Based on the swing transformer network, this network proposes a clothing texture analysis module (CTAM) and a clothing semantic decoding module (CSDM). In the coding process, CTAM extracts clothing texture features to enhance spatial information, while CSDM uses mixed attention to enhance image context information during decoding to improve the accuracy of clothing segmentation. The experimental results show that compared with the original model, the average pixel accuracy of the proposed method on the LIP data set is increased by about 2%, and the average crossover ratio is increased by about 1.5%. The segmentation network significantly improves indistinguishable garment boundaries and the accuracy of garment image segmentation.

Multi-Pooling Context Network for Image Semantic Segmentation

With the development of image segmentation technology, image context information plays an increasingly important role in semantic segmentation. However, due to the complexity of context information in different feature maps, simple context capture operations can easily cause context information omission. Rich context information can better classify categories and improve the quality of image segmentation. On the contrary, poor context information will lead to blurred image category segmentation and an incomplete target edge. In order to capture rich context information as completely as possible, we constructed a Multi-Pooling Context Network (MPCNet), which is a multi-pool contextual network for the semantic segmentation of images. Specifically, we first proposed the Pooling Context Aggregation Module to capture the deep context information of the image by processing the information between the space, channel, and pixel of the image. At the same time, the Spatial Context Module was constructed to capture the detailed spatial context of images at different stages of the network. The whole network structure adopted the form of codec to better extract image context. Finally, we performed extensive experiments on three semantic segmentation datasets (Cityscapes, ADE20K, and PASCAL VOC2012 datasets), which fully proved that our proposed network effectively alleviated the lack of context extraction and verified the effectiveness of the network.

Dual Gradient Based Snow Attentive Desnowing

In this paper, we propose a novel dual gradient-based desnowing algorithm that can accurately remove snow from a scene by characterizing snow particles. To localize snow in an image, we present a gradient-based snow activation map that can be estimated using snow classification. To recognize various patterns in the shapes and trajectories of snow particles, we introduce a gradient-based snow edge map. Using these two gradients, we estimate an accurate snow attention mask that is subsequently used for snow removal. In addition, we propose a translucency-aware context restoration network to handle various degrees of snow transparency and thus, prevent our method from losing the image context information during desnowing. Experimental results demonstrate that the proposed method considerably outperforms other state-of-the-art desnowing algorithms quantitatively and qualitatively.

Semantic segmentation for remote sensing images based on an AD-HRNet model

ABSTRACT Semantic segmentation for remote sensing images faces challenges of unbalanced category weight, rich context causing difficulties of recognition, blurred boundaries of multi-scale objects, and so on. To address these problems, we propose a new model by combining HRNet with attention mechanisms and dilated convolution, denoted as: AD-HRNet for the semantic segmentation of remote sensing images. In the framework of AD-HRNet, we obtained the weight value of each category based on an improved weighted cross-entropy function by introducing the median frequency balance method to solve the issue of class weight imbalance. The Shuffle-CBAM module with channel attention and spatial attention in AD-HRNet framework was applied to extract more global context information of images through slightly increasing the amount of computation. To address the problem of blurred boundaries caused by multi-scale object segmentation and edge segmentation, we developed an MDC-DUC module in AD-HRNet framework to capture the context information of multi-scale objects and the edge information of many irregular objects. Taking Postdam, Vaihingen, and SAMA-VTOL datasets as materials, we verified the performance of AD-HRNet by comparing with eight typical semantic segmentation models. Experimental results shown that AD-HRNet increases the mIoUs to 75.59% and 71.58% based on the Postdam and Vaihingen datasets, respectively.

ESTAN: Enhanced Small Tumor-Aware Network for Breast Ultrasound Image Segmentation.

Breast tumor segmentation is a critical task in computer-aided diagnosis (CAD) systems for breast cancer detection because accurate tumor size, shape, and location are important for further tumor quantification and classification. However, segmenting small tumors in ultrasound images is challenging due to the speckle noise, varying tumor shapes and sizes among patients, and the existence of tumor-like image regions. Recently, deep learning-based approaches have achieved great success in biomedical image analysis, but current state-of-the-art approaches achieve poor performance for segmenting small breast tumors. In this paper, we propose a novel deep neural network architecture, namely the Enhanced Small Tumor-Aware Network (ESTAN), to accurately and robustly segment breast tumors. The Enhanced Small Tumor-Aware Network introduces two encoders to extract and fuse image context information at different scales, and utilizes row-column-wise kernels to adapt to the breast anatomy. We compare ESTAN and nine state-of-the-art approaches using seven quantitative metrics on three public breast ultrasound datasets, i.e., BUSIS, Dataset B, and BUSI. The results demonstrate that the proposed approach achieves the best overall performance and outperforms all other approaches on small tumor segmentation. Specifically, the Dice similarity coefficient (DSC) of ESTAN on the three datasets is 0.92, 0.82, and 0.78, respectively; and the DSC of ESTAN on the three datasets of small tumors is 0.89, 0.80, and 0.81, respectively.

Context‐guided ground truth sampling for multi‐modality data augmentation in autonomous driving

AbstractData augmentation is an important pre‐processing step for object detection in 2D image and 3D point clouds. However, studies on multimodal data augmentation are extremely limited compared to single‐modal work. Moreover, simultaneously ensuring consistency and rationality when pasting both image and point cloud samples is a major challenge in multimodal methods. In this study, a novel multimodal data augmentation method based on ground truth sampling (GT sampling) is proposed for generating content‐rich synthetic scenes. A GT database and scene ground database based on the raw training set is initially built, following which the context of the image and point cloud is used to guide the paste location and filtering strategy of the GT samples. The proposed method can avoid the cluttered features caused by random pasting of samples; the image context information can help the model to learn the correlation between the object and the environment more comprehensively, and the point cloud context information can reduce occlusion in the case of long‐distance objects. The effectiveness of the proposed strategy is demonstrated on the publicly available KITTI dataset. Utilizing the multimodal 3D detector MVXNet as an implementation tool, our experiments evaluate different superimposition strategies ranging from context‐free sample pasting methods to context‐guided new training scenes. In comparison with existing GT sampling methods, our method exhibits a relative performance improvement of 15% on benchmark datasets. In ablation studies, our sample pasting strategy achieves a +2.81% gain compared with previous work. In conclusion, considering the multimodal context of modelled objects is crucial for placing them in the correct environment.

Vectorized dataset of roadside noise barriers in China using street view imagery

Abstract. Roadside noise barriers (RNBs) are important urban infrastructures to ensure that cities remain liveable. However, the absence of accurate and large-scale geospatial data on RNBs has impeded the increasing progress of rational urban planning, sustainable cities, and healthy environments. To address this problem, this study creates a vectorized RNB dataset in China using street view imagery and a geospatial artificial intelligence framework. First, intensive sampling is performed on the road network of each city based on OpenStreetMap, which is used as the georeference for downloading 6×106 Baidu Street View (BSV) images. Furthermore, considering the prior geographic knowledge contained in street view images, convolutional neural networks incorporating image context information (IC-CNNs) based on an ensemble learning strategy are developed to detect RNBs from the BSV images. The RNB dataset presented by polylines is generated based on the identified RNB locations, with a total length of 2667.02 km in 222 cities. Last, the quality of the RNB dataset is evaluated from two perspectives, i.e., the detection accuracy and the completeness and positional accuracy. Specifically, based on a set of randomly selected samples containing 10 000 BSV images, four quantitative metrics are calculated, with an overall accuracy of 98.61 %, recall of 87.14 %, precision of 76.44 %, and F1 score of 81.44 %. A total length of 254.45 km of roads in different cities are manually surveyed using BSV images to evaluate the mileage deviation and overlap level between the generated and surveyed RNBs. The root mean squared error for the mileage deviation is 0.08 km, and the intersection over union for overlay level is 88.08 % ± 2.95 %. The evaluation results suggest that the generated RNB dataset is of high quality and can be applied as an accurate and reliable dataset for a variety of large-scale urban studies, such as estimating the regional solar photovoltaic potential, developing 3D urban models, and designing rational urban layouts. Besides that, the benchmark dataset of the labeled BSV images can also support more work on RNB detection, such as developing more advanced deep learning algorithms, fine-tuning the existing computer vision models, and analyzing geospatial scenes in BSV. The generated vectorized RNB dataset and the benchmark dataset of labeled BSV imagery are publicly available at https://doi.org/10.11888/Others.tpdc.271914 (Chen, 2021).

MSK-UNET: A Modified U-Net Architecture Based on Selective Kernel with Multi-Scale Input for Pavement Crack Detection

Pavement crack condition is a vitally important indicator for road maintenance and driving safety. However, due to the interference of complex environment, such as illumination, shadow and noise, the automatic crack detection solution cannot meet the requirements of accuracy and efficiency. In this paper, we present an extended version of U-Net framework, named MSK-UNet, for pavement crack to solve these challenging problems. Specifically, first, the U-shaped network structure is chosen as the framework to extract more hierarchical representation. Second, we introduce selective kernel (SK) units to replace U-Net’s standard convolution blocks for obtaining the receptive fields with distinct scales. Third, multi-scale input layer establishes an image pyramid to retain more image context information at the encoder stage. Finally, a hybrid loss function including generalized Dice loss with Focal loss is employed. In addition, a regularization term is defined to reduce the impact of imbalance between positive and negative samples. To evaluate the performance of our algorithm, some tests were conducted on DeepCrack dataset, AsphaltCrack300 dataset and Crack500 dataset. Experimental results show that our approach can detect various crack types with diverse conditions, obtains a better performance in precision, recall and [Formula: see text]-score, with 97.43%, 96.95% and 97.01% precision values, 82.51%, 93.33% and 87.58% recall values and 95.33%, 99.24% and 98.55% [Formula: see text]-score values, respectively.

FECC-Net: A Novel Feature Enhancement and Context Capture Network Based on Brain MRI Images for Lesion Segmentation

In recent years, the increasing incidence of morbidity of brain stroke has made fast and accurate segmentation of lesion areas from brain MRI images important. With the development of deep learning, segmentation methods based on the computer have become a solution to assist clinicians in early diagnosis and treatment planning. Nevertheless, the variety of lesion sizes in brain MRI images and the roughness of the boundary of the lesion pose challenges to the accuracy of the segmentation algorithm. Current mainstream medical segmentation models are not able to solve these challenges due to their insufficient use of image features and context information. This paper proposes a novel feature enhancement and context capture network (FECC-Net), which is mainly composed of an atrous spatial pyramid pooling (ASPP) module and an enhanced encoder. In particular, the ASPP model uses parallel convolution operations with different sampling rates to enrich multi-scale features and fully capture image context information in order to process lesions of different sizes. The enhanced encoder obtains deep semantic features and shallow boundary features in the feature extraction process to achieve image feature enhancement, which is helpful for restoration of the lesion boundaries. We divide the pathological image into three levels according to the number of pixels in the real mask area and evaluate FECC-Net on an open dataset called Anatomical Tracings of Lesions After Stroke (ATLAS). The experimental results show that our FECC-Net outperforms mainstream methods, such as DoubleU-Net and TransUNet. Especially in small target tasks, FECC-Net is 4.09% ahead of DoubleU-Net on the main indicator DSC. Therefore, FECC-Net is encouraging and can be relied upon for brain MRI image applications.

DualMMP-GAN: Dual-scale multi-modality perceptual generative adversarial network for medical image segmentation

Multi-modality magnetic resonance imaging (MRI) can reveal distinct patterns of tissue in the human body and is crucial to clinical diagnosis. But it still remains a challenge to obtain diverse and plausible multi-modality MR images due to expense, noise, and artifacts. For the same lesion, different modalities of MRI have big differences in context information, coarse location, and fine structure. In order to achieve better generation and segmentation performance, a dual-scale multi-modality perceptual generative adversarial network (DualMMP-GAN) is proposed based on cycle-consistent generative adversarial networks (CycleGAN). Dilated residual blocks are introduced to increase the receptive field, preserving structure and context information of images. A dual-scale discriminator is constructed. The generator is optimized by discriminating patches to represent lesions with different sizes. The perceptual consistency loss is introduced to learn the mapping between the generated and target modality at different semantic levels. Moreover, generative multi-modality segmentation (GMMS) combining given modalities with generated modalities is proposed for brain tumor segmentation. Experimental results show that the DualMMP-GAN outperforms the CycleGAN and some state-of-the-art methods in terms of PSNR, SSMI, and RMSE in most tasks. In addition, dice, sensitivity, specificity, and Hausdorff95 obtained from segmentation by GMMS are all higher than those from a single modality. The objective index obtained by the proposed methods are close to upper bounds obtained from real multiple modalities, indicating that GMMS can achieve similar effects as multi-modality. Overall, the proposed methods can serve as an effective method in clinical brain tumor diagnosis with promising application potential.

Airborne SAR Radiometric Calibration Based on Improved Sliding Window Integral Method.

To verify the performance of the high-resolution fully polarimetric synthetic aperture radar (SAR) sensor carried by the Xinzhou 60 remote-sensing aircraft, we used corner reflectors to calibrate the acquired data. The target mechanism in high-resolution SAR images is more complex than it is in low-resolution SAR images, the impact of the point target pointing error on the calibration results is more obvious, and the target echo signal of high-resolution images is more easily affected by speckle noise; thus, more accurate extraction of the point target position and the response energy is required. To solve this problem, this paper introduces image context information and proposes a method to precisely determine the integration region of the corner reflector using sliding windows based on the integral method. The validation indicates that the fully polarimetric SAR sensor on the Xinzhou 60 remote-sensing aircraft can accurately reflect the radiometric characteristics of the ground features and that the integral method can obtain more stable results than the peak method. The sliding window allows the position of the point target to be determined more accurately, and the response energy extracted from the image via the integral method is closer to the theoretical value, which means that the high-resolution SAR system can achieve a higher radiometric calibration accuracy. Additionally, cross-validation reveals that the airborne SAR images have similar quality levels to Sentinel-1A and Gaofen-3 images.

High dynamic range imaging via gradient-aware context aggregation network

Obtaining a high dynamic range (HDR) image from multiple low dynamic range images with different exposures is an important step in various computer vision tasks. One of the ongoing challenges in the field is to generate HDR images without ghosting artifacts. Motivated by an observation that such artifacts are particularly noticeable in the gradient domain, in this paper, we propose an HDR imaging approach that aggregates the information from multiple LDR images with guidance from image gradient domain. The proposed method generates artifact-free images by integrating the image gradient information and the image context information in the pixel domain. The context information in a large area helps to reconstruct the contents contaminated by saturation and misalignments. Specifically, an additional gradient stream and the supervision in the gradient domain are applied to incorporate the gradient information in HDR imaging. To use the context information captured from a large area while preserving spatial resolution, we adopt dilated convolutions to extract multi-scale features with rich context information. Moreover, we build a new dataset containing 40 groups of real-world images from diverse scenes with ground truth to validate the proposed model. The samples in the proposed dataset include more challenging moving objects inducing misalignments. Extensive experimental results demonstrate that our proposed model outperforms previous methods on different datasets in terms of both quantitative measure and visual perception quality.

Automatic coronary artery segmentation algorithm based on deep learning and digital image processing

The automatic segmentation of coronary artery in coronary computed tomography angiography (CCTA) image is of great significance for clinicians to evaluate patients with coronary heart disease. When a 3D image is limited by the amount of available GPU memory, reducing the resolution of 3D image will easily lead to the loss of image detail information. Taking patches of image as input cannot make full use of image context information. Image segmentation based on deep learning is difficult to recover perfect smooth edges. The use of smooth loss function may filter out some small lesions on the coronary artery. In this paper, we present a novel CCTA image segmentation framework that combines deep learning and digital image processing algorithms to address these challenging problems. We first use V-Net to process the CCTA image with lower resolution, and get the basic feature map (rough segmentation result) with the same resolution as the original CCTA image. Then, the original CCTA image is concatenated to the basic feature map and input it into the patch-based cascaded V-shaped module to obtain a accurate coronary artery segmentation image. Finally, the center points of coronary segmentation image and the basic gradient image of the original coronary image are obtained by morphological operation. The center points of coronary artery segmentation image are used as seed points, region growing is performed on the binary basic gradient image until the white contour boundary is searched, so as to obtain a coronary segmentation result with full segmentation and smooth edges. The proposed method is analyzed quantitatively and qualitatively, and the results show that the method is better than the mainstream baseline. The ablation experiment also proved the effectiveness of each module.

Impact of Image Context on Deep Learning for Classification of Teeth on Radiographs.

Objectives: We aimed to assess the impact of image context information on the accuracy of deep learning models for tooth classification on panoramic dental radiographs. Methods: Our dataset contained 5008 panoramic radiographs with a mean number of 25.2 teeth per image. Teeth were segmented bounding-box-wise and classified by one expert; this was validated by another expert. Tooth segments were cropped allowing for different context; the baseline size was 100% of each box and was scaled up to capture 150%, 200%, 250% and 300% to increase context. On each of the five generated datasets, ResNet-34 classification models were trained using the Adam optimizer with a learning rate of 0.001 over 25 epochs with a batch size of 16. A total of 20% of the data was used for testing; in subgroup analyses, models were tested only on specific tooth types. Feature visualization using gradient-weighted class activation mapping (Grad-CAM) was employed to visualize salient areas. Results: F1-scores increased monotonically from 0.77 in the base-case (100%) to 0.93 on the largest segments (300%; p = 0.0083; Mann–Kendall-test). Gains in accuracy were limited between 200% and 300%. This behavior was found for all tooth types except canines, where accuracy was much higher even for smaller segments and increasing context yielded only minimal gains. With increasing context salient areas were more widely distributed over each segment; at maximum segment size, the models assessed minimum 3–4 teeth as well as the interdental or inter-arch space to come to a classification. Conclusions: Context matters; classification accuracy increased significantly with increasing context.

DFFAN: Dual Function Feature Aggregation Network for Semantic Segmentation of Land Cover

Analyzing land cover using remote sensing images has broad prospects, the precise segmentation of land cover is the key to the application of this technology. Nowadays, the Convolution Neural Network (CNN) is widely used in many image semantic segmentation tasks. However, existing CNN models often exhibit poor generalization ability and low segmentation accuracy when dealing with land cover segmentation tasks. To solve this problem, this paper proposes Dual Function Feature Aggregation Network (DFFAN). This method combines image context information, gathers image spatial information, and extracts and fuses features. DFFAN uses residual neural networks as backbone to obtain different dimensional feature information of remote sensing images through multiple downsamplings. This work designs Affinity Matrix Module (AMM) to obtain the context of each feature map and proposes Boundary Feature Fusion Module (BFF) to fuse the context information and spatial information of an image to determine the location distribution of each image’s category. Compared with existing methods, the proposed method is significantly improved in accuracy. Its mean intersection over union (MIoU) on the LandCover dataset reaches 84.81%.

Modeling and Optimization of Semantic Segmentation for Track Bed Foreign Object Based on Attention Mechanism

The problem of foreign object intrusion onto the track bed often occurs in the actual operation process of high-speed railways. To solve the problem, we propose an anomaly detection method for the ballastless track bed, which is based on semantic segmentation. Firstly, we put forward the RFODLab semantic segmentation network according to the randomness of foreign objects distribution, and a small proportion of target pixels in the track image. The segmentation results of track image obtained through this model can be used to obtain the accurate pixel information of foreign objects. To further improve the recall and precision, the channel attention mechanism is introduced for the backbone network of the model to aggregate the context information of images, which achieves the weighted constraints of the model on the area to be recognized. Furthermore, to improve the model performance affected by unbalanced sample category distribution during the anomaly detection, we modify the loss function by balancing distribution of each category. The experimental results show that our proposed method can effectively segment various types of anomalies on the ballastless track bed including broken elastic strips, animal carcasses, and fallen pieces. The precision of anomaly detection on the test set can reach 90% while the recall can be maintained at more than 95%. The anomaly detection results on actual lines also verify the effectiveness of the method.

A Serial-Parallel Self-Attention Network Joint With Multi-Scale Dilated Convolution

Semantic segmentation is a high-level task in the field of computer vision, which paves the way for the realization of a complete understanding of the scene, and has been widely used in automatic driving, human-computer interaction, virtual reality, and other aspects. Recently, the semantic segmentation method of convolutional neural networks with deep structure has been more accurate and efficient than other methods. However, there are some problems in these methods, such as the loss of information caused by the down-sampling operation, the lack of usage of image context information, and the neglect of the relationship between spatial features and channel features. To solve these problems, a novel self-attention network based on the series-parallel structure is proposed in the paper. Firstly, a multi-scale dilated convolution backbone network is constructed by combining the dilated convolution and the residual network, which makes up for the information loss caused by the restriction of the receptor field in the ordinary network and improves the richness of extracted features. Secondly, the self-attention modules are stacked with serial and parallel structures, which can effectively extract the contextual information of space, channel, and space-channel and fully integrate them. Finally, the proposed algorithm is tested extensively and compared with the existing classical algorithms. The experimental results show that the proposed algorithm achieves state-of-the-art performance on the public dataset.