Context Aggregation Network Research Articles

Multi-Encoder Context Aggregation Network for Structured and Unstructured Urban Street Scene Analysis

Multi-Encoder Context Aggregation Network for Structured and Unstructured Urban Street Scene Analysis

Compensating for Local Ambiguity With Encoder-Decoder in Urban Scene Segmentation

Semantic segmentation plays a critical role in scene understanding for self-driving vehicles. A line of efforts has proven that global context matters in urban scene segmentation due to massive scale changes. However, we find that existing methods suffer from local ambiguities when dissipating continuous local context, i.e. scrambling to a huge receptive field of global cues by coarse pooling. To this end, this paper proposes a new Context Aggregation Module (CAM) that consists of two primary components: context encoding using no coarse pooling but encoder-decoders with appropriate sampling scales and gated fusion that extends gate attention mechanism to balance different-scale context during feature fusion. Weeding out coarse pooling and applying the encoder-decoder inherits the merits of exploring global context while avoiding the drawback of losing local contextual continuity. We then construct a Context Aggregation Network (CANet) and conduct extensive evaluations on challenging autonomous driving benchmarks of Cityscapes, CamVid and BDD100K. Consistently improved results evidence the effectiveness. Notably, we attain competitive mIoU 82.7% on Cityscapes and optimal mIoU 80.5% on CamVid.

GLFormer: Global and Local Context Aggregation Network for Temporal Action Detection

As the core component of video analysis, Temporal Action Localization (TAL) has experienced remarkable success. However, some issues are not well addressed. First, most of the existing methods process the local context individually, without explicitly exploiting the relations between features in an action instance as a whole. Second, the duration of different actions varies widely; thus, it is difficult to choose the proper temporal receptive field. To address these issues, this paper proposes a novel network, GLFormer, which can aggregate short, medium, and long temporal contexts. Our method consists of three independent branches with different ranges of attention, and these features are then concatenated along the temporal dimension to obtain richer features. One is multi-scale local convolution (MLC), which consists of multiple 1D convolutions with varying kernel sizes to capture the multi-scale context information. Another is window self-attention (WSA), which tries to explore the relationship between features within the window range. The last is global attention (GA), which is used to establish long-range dependencies across the full sequence. Moreover, we design a feature pyramid structure to be compatible with action instances of various durations. GLFormer achieves state-of-the-art performance on two challenging video benchmarks, THUMOS14 and ActivityNet 1.3. Our performance is 67.2% and 54.5% AP@0.5 on the datasets THUMOS14 and ActivityNet 1.3, respectively.

DCAN: Improving Temporal Action Detection via Dual Context Aggregation

Temporal action detection aims to locate the boundaries of action in the video. The current method based on boundary matching enumerates and calculates all possible boundary matchings to generate proposals. However, these methods neglect the long-range context aggregation in boundary prediction. At the same time, due to the similar semantics of adjacent matchings, local semantic aggregation of densely-generated matchings cannot improve semantic richness and discrimination. In this paper, we propose the end-to-end proposal generation method named Dual Context Aggregation Network (DCAN) to aggregate context on two levels, namely, boundary level and proposal level, for generating high-quality action proposals, thereby improving the performance of temporal action detection. Specifically, we design the Multi-Path Temporal Context Aggregation (MTCA) to achieve smooth context aggregation on boundary level and precise evaluation of boundaries. For matching evaluation, Coarse-to-fine Matching (CFM) is designed to aggregate context on the proposal level and refine the matching map from coarse to fine. We conduct extensive experiments on ActivityNet v1.3 and THUMOS-14. DCAN obtains an average mAP of 35.39% on ActivityNet v1.3 and reaches mAP 54.14% at IoU@0.5 on THUMOS-14, which demonstrates DCAN can generate high-quality proposals and achieve state-of-the-art performance. We release the code at https://github.com/cg1177/DCAN.

ECANet: enhanced context aggregation network for single image dehazing

ECANet: enhanced context aggregation network for single image dehazing

A novel multi-exposure fusion approach for enhancing visual semantic segmentation of autonomous driving

Visual semantic segmentation is a key technology to realize scene understanding for autonomous driving and its accuracy is affected by the light changes in images. This paper proposes a novel multi-exposure fusion approach to visual semantic enhancement of autonomous driving. Firstly, a multi-exposure image sequence is aligned to construct a stable image input. Secondly, high contrast regions of multi-exposure image sequences are evaluated by context aggregation network (CAN) to predict image weight map. Finally, the high-quality image is generated by weighted fusion of multi-exposure image sequences. The proposed approach is validated by using Cityscapes’ HDR dataset and real environment data. The experimental results show that the proposed method effectively restores lost features in the light changing images and enhances accuracy of subsequent semantic segmentation.

Analysis of Deep Learning-Based Phase Retrieval Algorithm Performance for Quantitative Phase Imaging Microscopy.

Quantitative phase imaging has been of interest to the science and engineering community and has been applied in multiple research fields and applications. Recently, the data-driven approach of artificial intelligence has been utilized in several optical applications, including phase retrieval. However, phase images recovered from artificial intelligence are questionable in their correctness and reliability. Here, we propose a theoretical framework to analyze and quantify the performance of a deep learning-based phase retrieval algorithm for quantitative phase imaging microscopy by comparing recovered phase images to their theoretical phase profile in terms of their correctness. This study has employed both lossless and lossy samples, including uniform plasmonic gold sensors and dielectric layer samples; the plasmonic samples are lossy, whereas the dielectric layers are lossless. The uniform samples enable us to quantify the theoretical phase since they are established and well understood. In addition, a context aggregation network has been employed to demonstrate the phase image regression. Several imaging planes have been simulated serving as input and the label for network training, including a back focal plane image, an image at the image plane, and images when the microscope sample is axially defocused. The back focal plane image plays an essential role in phase retrieval for the plasmonic samples, whereas the dielectric layer requires both image plane and back focal plane information to retrieve the phase profile correctly. Here, we demonstrate that phase images recovered using deep learning can be robust and reliable depending on the sample and the input to the deep learning.

Improvement of noisy images filtered by bilateral process using a multi-scale context aggregation network

Deep learning has recently received a lot of attention as a feasible solution to a variety of artificial intelligence difficulties. Convolutional neural networks (CNNs) outperform other deep learning architectures in the application of object identification and recognition when compared to other machine learning methods. Speech recognition, pattern analysis, and image identification, all benefit from deep neural networks. When performing image operations on noisy images, such as fog removal or low light enhancement, image processing methods such as filtering or image enhancement are required. The study shows the effect of using Multi-scale deep learning Context Aggregation Network CAN on Bilateral Filtering Approximation (BFA) for de-noising noisy CCTV images. Data-store is used tomanage our dataset, which is an object or collection of data that are huge to enter in memory, it allows to read, manage, and process data located in multiple files as a single entity. The CAN architecture provides integral deep learning layers such as input, convolution, back normalization, and Leaky ReLu layers to construct multi-scale. It is also possible to add custom layers like adaptor normalization (µ) and adaptive normalization (Lambda) to the network. The performance of the developed CAN approximation operator on the bilateral filtering noisy image is proven when improving both the noisy reference image and a CCTV foggy image. The three image evaluation metrics (SSIM, NIQE, and PSNR) evaluate the developed CAN approximation visually and quantitatively when comparing the created de-noised image over the reference image.Compared with the input noisy image, these evaluation metrics for the developed CAN de-noised image were (0.92673/0.76253, 6.18105/12.1865, and 26.786/20.3254) respectively

Edge Guided Context Aggregation Network for Semantic Segmentation of Remote Sensing Imagery

Semantic segmentation of remote sensing imagery (RSI) has obtained great success with the development of deep convolutional neural networks (DCNNs). However, most of the existing algorithms focus on designing end-to-end DCNNs, but neglecting to consider the difficulty of segmentation in imbalance categories, especially for minority categories in RSI, which limits the performance of RSI semantic segmentation. In this paper, a novel edge guided context aggregation network (EGCAN) is proposed for the semantic segmentation of RSI. The Unet is employed as backbone. Meanwhile, an edge guided context aggregation branch and minority categories extraction branch are designed for a comprehensive enhancement of semantic modeling. Specifically, the edge guided context aggregation branch is proposed to promote entire semantic comprehension of RSI and further emphasize the representation of edge information, which consists of three modules: edge extraction module (EEM), dual expectation maximization attention module (DEMA), and edge guided module (EGM). EEM is created primarily for accurate edge tracking. According to that, DEMA aggregates global contextual features with different scales and the edge features along spatial and channel dimensions. Subsequently, EGM cascades the aggregated features into the decoder process to capture long-range dependencies and further emphasize the error-prone pixels in the edge region to acquire better semantic labels. Besides this, the exploited minority categories extraction branch is presented to acquire rich multi-scale contextual information through an elaborate hybrid spatial pyramid pooling module (HSPP) to distinguish categories taking a small percentage and background. On the Tianzhi Cup dataset, the proposed algorithm EGCAN achieved an overall accuracy of 84.1% and an average cross-merge ratio of 68.1%, with an accuracy improvement of 0.4% and 1.3% respectively compared to the classical Deeplabv3+ model. Extensive experimental results on the dataset released in ISPRS Vaihingen and Potsdam benchmarks also demonstrate the effectiveness of the proposed EGCAN over other state-of-the-art approaches.

Cuff-Less Blood Pressure Prediction from ECG and PPG Signals Using Fourier Transformation and Amplitude Randomization Preprocessing for Context Aggregation Network Training

This research proposes an algorithm to preprocess photoplethysmography (PPG) and electrocardiogram (ECG) signals and apply the processed signals to the context aggregation network-based deep learning to achieve higher accuracy of continuous systolic and diastolic blood pressure monitoring than other reported algorithms. The preprocessing method consists of the following steps: (1) acquiring the PPG and ECG signals for a two second window at a sampling rate of 125 Hz; (2) separating the signals into an array of 250 data points corresponding to a 2 s data window; (3) randomizing the amplitude of the PPG and ECG signals by multiplying the 2 s frames by a random amplitude constant to ensure that the neural network can only learn from the frequency information accommodating the signal fluctuation due to instrument attachment and installation; (4) Fourier transforming the windowed PPG and ECG signals obtaining both amplitude and phase data; (5) normalizing both the amplitude and the phase of PPG and ECG signals using z-score normalization; and (6) training the neural network using four input channels (the amplitude and the phase of PPG and the amplitude and the phase of ECG), and arterial blood pressure signal in time-domain as the label for supervised learning. As a result, the network can achieve a high continuous blood pressure monitoring accuracy, with the systolic blood pressure root mean square error of 7 mmHg and the diastolic root mean square error of 6 mmHg. These values are within the error range reported in the literature. Note that other methods rely only on mathematical models for the systolic and diastolic values, whereas the proposed method can predict the continuous signal without degrading the measurement performance and relying on a mathematical model.

Multi-Resolution Context Aggregation Network for Single Image Rain Removal

Severe weather such as rain will cause serious degradation of image quality and affect the accuracy of computer vision algorithms. In order to better extract the features of multi-scale rain streaks and restore the important detailed feature information of the image, a new single image rain removal based on multi-resolution context aggregation network is proposed. Firstly, the shuffling operation is used to convert a single resolution input image into a multi-spatial resolution input, which enables the network to rapidly expand the acceptance field at the low spatial resolution and extract more refined rain streak features at high spatial resolution. The low-resolution rain streak features are aggregated from top to bottom into high spatial resolution, and the network is guided to extract multi-scale rain streak information. Secondly, the multi-resolution feature enhancement block is used to refine the details of the image at different resolutions to prevent the loss or blurring of the details in the rain-removed image. This method uses local residual densely connect blocks and squeeze-excitation network to enhance the ability and efficiency of the network to extract features. Finally, the mixed loss function constructed in this paper is used to obtain higher evaluation index values while improving the human visual perception of rain removal images. The experimental results show that proposed method achieves significant rain removal effects on the Rain100H, Rain100L, and Rain12 synthetic datasets and real rain datasets. Compared with the existing algorithms, the proposed method can achieve significant improvement in qualitative and quantitative indicators and has a high degree of detail retention.

Multiscale Supervision-Guided Context Aggregation Network for Single Image Dehazing

End-to-end learning-based image dehazing methods tend to overdehaze or underdehaze in real scenes due to inefficient feature extraction and feature fusion. In this letter, we propose a multiscale supervision-guided context aggregation network (MSGCAN) based on two principles: improving feature extraction and enhancing feature mapping. To improve feature extraction, an attention-guided context aggregation (AGCA) module is adopted to merge context features extracted by several residual dense blocks (RDB). Moreover, we output these aggregated context features on each scale and form multiscale supervision to enhance feature mapping and ensure that the extracted features on each scale contain more realistic details. The experimental results show that the proposed MSGCAN performs better than other state-of-the-art dehazing methods in both synthetic and real-world scenes.

Multiscale Context Aggregation Network for Building Change Detection Using High Resolution Remote Sensing Images

The existing methods of building change detection (CD) using remote sensing (RS) images are still deficient in handling scale variation and class imbalance problems, indicating a decrease in the robustness of small-object detection and pseudo-change information. Thus, a novel building CD framework called the multiscale context aggregation network (MSCANet) is proposed. The high-resolution network is integrated into the feature extracting stage to maintain high-resolution representations throughout the whole process. Then, multiscale context information is aggregated using a scale-aware feature pyramid module (FPM). Recognition performance can be improved from discriminant feature representation learning by using a channel–spatial attention module. Furthermore, a class-balanced loss is proposed to reduce the impact of class imbalance in long-tail datasets. Experimental results from using the LEVIR-CD and SZTAKI AirChange benchmark datasets prove the superiority of the MSCANet over the other baseline methods, with improved maximum F1 scores of 5.28 and 8.47, respectively.

DCACNet: Dual context aggregation and attention-guided cross deconvolution network for medical image segmentation

Background and Objective: Segmentation is a key step in biomedical image analysis tasks. Recently, convolutional neural networks (CNNs) have been increasingly applied in the field of medical image processing; however, standard models still have some drawbacks. Due to the significant loss of spatial information at the coding stage, it is often difficult to restore the details of low-level visual features using simple deconvolution, and the generated feature maps are sparse, which results in performance degradation. This prompted us to study whether it is possible to better preserve the deep feature information of the image in order to solve the sparsity problem of image segmentation models. Methods: In this study, we (1) build a reliable deep learning network framework, named DCACNet, to improve the segmentation performance for medical images; (2) propose a multiscale cross-fusion encoding network to extract features; (3) build a dual context aggregation module to fuse the context features at different scales and capture more fine-grained deep features; and (4) propose an attention-guided cross deconvolution decoding network to generate dense feature maps. We demonstrate the effectiveness of the proposed method on two publicly available datasets. Results: DCACNet was trained and tested on the prepared dataset, and the experimental results show that our proposed model has better segmentation performance than previous models. For 4-class classification (CHAOS dataset), the mean DSC coefficient reached 91.03%. For 2-class classification (Herlev dataset), the accuracy, precision, sensitivity, specificity, and Dice score reached 96.77%, 90.40%, 94.20%, 97.50%, and 97.69%, respectively. The experimental results show that DCACNet can improve the segmentation effect for medical images. Conclusion: DCACNet achieved promising results on the prepared dataset and improved segmentation performance. It can better retain the deep feature information of the image than other models and solve the sparsity problem of the medical image segmentation model.

High dynamic range imaging via gradient-aware context aggregation network

Obtaining a high dynamic range (HDR) image from multiple low dynamic range images with different exposures is an important step in various computer vision tasks. One of the ongoing challenges in the field is to generate HDR images without ghosting artifacts. Motivated by an observation that such artifacts are particularly noticeable in the gradient domain, in this paper, we propose an HDR imaging approach that aggregates the information from multiple LDR images with guidance from image gradient domain. The proposed method generates artifact-free images by integrating the image gradient information and the image context information in the pixel domain. The context information in a large area helps to reconstruct the contents contaminated by saturation and misalignments. Specifically, an additional gradient stream and the supervision in the gradient domain are applied to incorporate the gradient information in HDR imaging. To use the context information captured from a large area while preserving spatial resolution, we adopt dilated convolutions to extract multi-scale features with rich context information. Moreover, we build a new dataset containing 40 groups of real-world images from diverse scenes with ground truth to validate the proposed model. The samples in the proposed dataset include more challenging moving objects inducing misalignments. Extensive experimental results demonstrate that our proposed model outperforms previous methods on different datasets in terms of both quantitative measure and visual perception quality.

Attention-guided chained context aggregation for semantic segmentation

The way features propagate in Fully Convolutional Networks is of momentous importance to capture multi-scale contexts for obtaining precise segmentation masks. This paper proposes a novel series-parallel hybrid paradigm called the Chained Context Aggregation Module (CAM) to enrich feature representation. CAM gains features of various spatial scales through chain-connected ladder-style information flows and fuses them in a two-stage process, namely pre-fusion and re-fusion. The serial flow continuously increases receptive fields of output neurons and those in parallel encode different region-based contexts. Each information flow is a shallow encoder-decoder with appropriate down-sampling scales to sufficiently capture contextual information. We further adopt an attention model in CAM to guide feature re-fusion. Based on these developments, we construct the Chained Context Aggregation Network (CANet), which employs an asymmetric decoder to recover precise spatial details of prediction maps. We conduct extensive experiments on six challenging datasets, including Pascal VOC 2012, Pascal Context, Cityscapes, CamVid, SUN-RGBD and GATECH. Results evidence that CANet achieves state-of-the-art or competitive performance.

Deep learning-based single-shot phase retrieval algorithm for surface plasmon resonance microscope based refractive index sensing application

A deep learning algorithm for single-shot phase retrieval under a conventional microscope is proposed and investigated. The algorithm has been developed using the context aggregation network architecture; it requires a single input grayscale image to predict an output phase profile through deep learning-based pattern recognition. Surface plasmon resonance imaging has been employed as an example to demonstrate the capability of the deep learning-based method. The phase profiles of the surface plasmon resonance phenomena have been very well established and cover ranges of phase transitions from 0 to 2π rad. We demonstrate that deep learning can be developed and trained using simulated data. Experimental validation and a theoretical framework to characterize and quantify the performance of the deep learning-based phase retrieval method are reported. The proposed deep learning-based phase retrieval performance was verified through the shot noise model and Monte Carlo simulations. Refractive index sensing performance comparing the proposed deep learning algorithm and conventional surface plasmon resonance measurements are also discussed. Although the proposed phase retrieval-based algorithm cannot achieve a typical detection limit of 10–7 to 10–8 RIU for phase measurement in surface plasmon interferometer, the proposed artificial-intelligence-based approach can provide at least three times lower detection limit of 4.67 × 10–6 RIU compared to conventional intensity measurement methods of 1.73 × 10–5 RIU for the optical energy of 2500 pJ with no need for sophisticated optical interferometer instrumentation.

Real-time Semantic Segmentation with Context Aggregation Network

With the increasing demand of autonomous systems, pixelwise semantic segmentation for visual scene understanding needs to be not only accurate but also efficient for potential real-time applications. In this paper, we propose Context Aggregation Network, a dual branch convolutional neural network, with significantly lower computational costs as compared to the state-of-the-art, while maintaining a competitive prediction accuracy. Building upon the existing dual branch architectures for high-speed semantic segmentation, we design a high resolution branch for effective spatial detailing and a context branch with light-weight versions of global aggregation and local distribution blocks, potent to capture both long-range and local contextual dependencies required for accurate semantic segmentation, with low computational overheads. We evaluate our method on two semantic segmentation datasets, namely Cityscapes dataset and UAVid dataset. For Cityscapes test set, our model achieves state-of-the-art results with mIOU of 75.9%, at 76 FPS on an NVIDIA RTX 2080Ti and 8 FPS on a Jetson Xavier NX. With regards to UAVid dataset, our proposed network achieves mIOU score of 63.5% with high execution speed (15 FPS).

Congested Crowd Counting via Adaptive Multi-Scale Context Learning.

In this paper, we propose a novel congested crowd counting network for crowd density estimation, i.e., the Adaptive Multi-scale Context Aggregation Network (MSCANet). MSCANet efficiently leverages the spatial context information to accomplish crowd density estimation in a complicated crowd scene. To achieve this, a multi-scale context learning block, called the Multi-scale Context Aggregation module (MSCA), is proposed to first extract different scale information and then adaptively aggregate it to capture the full scale of the crowd. Employing multiple MSCAs in a cascaded manner, the MSCANet can deeply utilize the spatial context information and modulate preliminary features into more distinguishing and scale-sensitive features, which are finally applied to a 1 × 1 convolution operation to obtain the crowd density results. Extensive experiments on three challenging crowd counting benchmarks showed that our model yielded compelling performance against the other state-of-the-art methods. To thoroughly prove the generality of MSCANet, we extend our method to two relevant tasks: crowd localization and remote sensing object counting. The extension experiment results also confirmed the effectiveness of MSCANet.

HCANet: A Hierarchical Context Aggregation Network for Semantic Segmentation of High-Resolution Remote Sensing Images

Many practical applications of high-resolution remote sensing images (HRRSIs) are based on semantic segmentation. However, due to the complex ground object information contained in remote sensing images, it is difficult to make precise semantic segmentation of HRRSIs. In this letter, we proposed a hierarchical context aggregation network (HCANet) for the semantic segmentation of HRRSIs. The HCANet has an encoder-decoder structure which is similar to UNet. In the HCANet, we designed two Compact Atrous Spatial Pyramid Pooling (CASPP and CASPP+) modules. The CASPP modules replace the copy and crop operation in UNet to extract the multiscale context information of the multisemantic features of ResNet. The CASPP+ module is embedded in the middle layer of HCANet’s decoder to provide a strong aggregation path of contextual information. In the decoder of HCANet, the multiscale context information obtained by CASPP modules is hierarchically merged layer by layer for the semantic segmentation of HRRSIs. We compared our method with several of the most advanced methods on the ISPRS Vaihingen and Potsdam data sets. The final results demonstrate that our method can achieve outstanding performance.