Challenging Scenes Research Articles

High Dynamic Range Imaging for Dynamic Scenes With Large-Scale Motions and Severe Saturation

Two key challenges exist in high dynamic range (HDR) imaging from multiexposure low dynamic range (LDR) images for dynamic scenes: 1) aligning the input images with large-scale foreground motions and 2) recovering large saturated regions from a limited number of input LDR images. To tackle these challenges, several deep convolutional neural networks have been proposed that have made significant progress. However, these methods tend to suffer from ghosting and saturation artifacts when applied to some challenging scenes. In this article, we propose an end-to-end deformable HDR imaging network, called DHDRNet, which attempts to alleviate these problems by building an effective aligning module and adopting self-guided attention. First, we analyze the alignment process in the HDR imaging task and correspondingly design a pyramidal deformable module (PDM) that aligns LDR images at multiple scales and reconstructs aligned features in a coarse-to-fine manner. In this way, the proposed DHDRNet can handle large-scale complex motions and suppress ghosting artifacts caused by misalignments. Moreover, we adopt self-guided attention to reduce the influence of saturated regions during the aligning and merging processes, which helps suppress artifacts and retain fine details in the final HDR image. Extensive qualitative and quantitative comparisons demonstrate that the proposed model outperforms the existing start-of-the-art methods and that it is robust to challenging scenes with large-scale motions and severe saturation. The source code is available at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Tx000/DHDRNet</uri> .

A Point-Line VIO System With Novel Feature Hybrids and With Novel Line Predicting-Matching

Weak texture and motion blur are always challenging problems for visual-inertial odometry (VIO) systems. To improve accuracy of VIO systems in the challenging scenes, we propose a point-line-based VIO system with novel feature hybrids and with novel predicting-matching for long line track. Point-line features with shorter tracks are categorized into “MSCKF” features and with longer tracks into “SLAM” features. Especially, “SLAM” lines are added into the state vector to improve accuracy of the proposed system. Besides, to ensure the reliability and stability of detection and tracking of line features, we also propose a new “Predicting-Matching” line segment tracking method to increase the track lengths of line segments. Experimental results show that the proposed method outperforms the state-of-the-art methods of VINS-Mono [1], PL-VINS [2] and OpenVINS [3]) on both a public dataset and a collected dataset in terms of accuracy. The collected dataset is full of extremely weak textures and motion blurs. On this dataset, the proposed method also obtains better accuracy than ORB-SLAM3 [4].

JoKDNet: A joint keypoint detection and description network for large-scale outdoor TLS point clouds registration

Registration of large-scale outdoor Terrestrial Laser Scanning (TLS) point clouds remains many challenges in the scenes with symmetric and repetitive elements (e.g., park, forest, and tunnel), the weak geometric features (e.g., underground excavation), and dramatically changes in different phases (e.g., mountain). To address these issues, a novel neural network JoKDNet is proposed to jointly learn the keypoint detection and feature description to improve the feasibility and accuracy of point clouds registration. Firstly, a novel keypoint detection module is introduced to automatically learn the score of each sampled point and regard the most significant Top-k sampled points as the detected keypoints. Secondly, an enhanced feature description module is proposed to learn the feature representation of each keypoint by fusing the hierarchical local features and context features. Thirdly, a loss function is designed to make the detected keypoints more distinguishable for matching, which simultaneously maximizes the feature distance between non-corresponding keypoints and minimizes the feature distance of corresponding keypoints. Finally, the distance matrix module and RANdom SAmple Consensus (RANSAC) are utilized to determine the correspondences of source and target point clouds for the transformation calculation. Comprehensive experiments show that the JoKDNet performs effectively on five challenging scenes (e.g., park, forest, tunnel, underground excavation, and mountain) from two datasets (WHU-TLS and ETH-TLS) in terms of registration errors, and robustness to varying scenes, with the maximum rotation error less than 0.06° and maximum translation error less than 0.84 m without ICP.

Disruptor-Aware Interval-Based Response Inconsistency for Correlation Filters in Real-Time Aerial Tracking

Aerial object tracking approaches based on discriminative correlation filter (DCF) have attracted wide attention in the tracking community due to their impressive progress recently. Many studies introduce temporal regularization into the DCF-based framework to achieve a more robust appearance model and further enhance the tracking performance. However, existing temporal regularization approaches usually utilize the information of two consecutive frames, which are not robust enough due to limited information. Although some methods attempt to incorporate abundant training samples and generally improve the tracking performance, these improvements are at the expense of significantly increased computing consumption. Besides, most existing methods introduce historical information directly without denoising, which means that background noises are also introduced into the filter training and may degrade the tracking accuracy. To tackle the drawbacks mentioned earlier, this work proposes a novel aerial object tracking approach to exploit disruptor-aware interval-based response inconsistency, i.e., IBRI tracker. The proposed method is able to incorporate historical interval information by utilizing responses in the filter training process, thereby obtaining a robust tracking performance while maintaining the real-time speed. Moreover, to reduce the disruptions caused by similar object, partial occlusion, and other challenging scenes, a novel disruptor-aware scheme based on response bucketing is introduced to detect the disruptor and enforce a spatial penalty for the disruptive area around the tracked object. Exhausted experiments on multiple well-known challenging aerial tracking benchmarks demonstrate the accuracy and robustness of the proposed IBRI tracker against other 35 state-of-the-art trackers. With a real-time speed of ~32 frames/s on a single CPU, the proposed approach can be applied for typical aerial platforms to achieve aerial visual object tracking efficiently.

HDPL: a hybrid descriptor for points and lines based on graph neural networks

PurposeThis paper aims to design a novel feature descriptor to improve the performance of feature matching in challenge scenes, such as low texture and wide-baseline scenes. Common descriptors are not suitable for low texture scenes and other challenging scenes mainly owing to encoding only one kind of features. The proposed feature descriptor considers multiple features and their locations, which is more expressive.Design/methodology/approachA graph neural network–based descriptors enhancement algorithm for feature matching is proposed. In this paper, point and line features are the primary concerns. In the graph, commonly used descriptors for points and lines constitute the nodes and the edges are determined by the geometric relationship between points and lines. After the graph convolution designed for incomplete join graph, enhanced descriptors are obtained.FindingsExperiments are carried out in indoor, outdoor and low texture scenes. The experiments investigate the real-time performance, rotation invariance, scale invariance, viewpoint invariance and noise sensitivity of the descriptors in three types of scenes. The results show that the enhanced descriptors are robust to scene changes and can be used in wide-baseline matching.Originality/valueA graph structure is designed to represent multiple features in an image. In the process of building graph structure, the geometric relation between multiple features is used to establish the edges. Furthermore, a novel hybrid descriptor for points and lines is obtained using graph convolutional neural network. This enhanced descriptor has the advantages of both point features and line features in feature matching.

Estimating Obstacle Maps for USVs Based on a Multistage Feature Aggregation and Semantic Feature Separation Network

Obstacle map estimation based on efficient semantic segmentation networks is promising for improving the environmental awareness of unmanned surface vehicles (USVs). However, existing networks perform poorly in challenging scenes with small obstacles, scenery reflections, boat wakes, and visual ambiguities caused by unfavorable weather conditions. In this paper, we address the small obstacle segmentation problem by learning representations of obstacles at multiple scales. An efficient multistage feature aggregation (MFA) module is proposed, which utilizes fully separable convolutions of different sizes to capture and fuse multiscale context information from different stages of a backbone network. In addition, a novel feature separation (FS) loss function based on Gaussian mixture model is presented, which encourages the MFA module to enforce separation among different semantic features, thereby providing a robust and discriminative representation in various challenging scenes. Building upon the MFA module and the FS loss function, we present a fast multistage feature aggregation and semantic feature separation network (FASNet) for obstacle map estimation of USVs. An extensive evaluation was conducted on a challenging public dataset (MaSTr1325). We validated that various lightweight semantic segmentation models achieved consistent performance improvement when our MFA module and FS loss function were adopted. The evaluation results showed that the proposed FASNet outperformed state-of-the-art lightweight models and achieved 96.71% mIoU and > 1.5% higher obstacle-class IoU than the second-best network, while running over 58 fps.

GSAP: A Global Structure Attention Pooling Method for Graph-Based Visual Place Recognition

The Visual Place Recognition problem aims to use an image to recognize the location that has been visited before. In most of the scenes revisited, the appearance and view are drastically different. Most previous works focus on the 2-D image-based deep learning method. However, the convolutional features are not robust enough to the challenging scenes mentioned above. In this paper, in order to take advantage of the information that helps the Visual Place Recognition task in these challenging scenes, we propose a new graph construction approach to extract the useful information from an RGB image and a depth image and fuse them in graph data. Then, we deal with the Visual Place Recognition problem as a graph classification problem. We propose a new Global Pooling method—Global Structure Attention Pooling (GSAP), which improves the classification accuracy by improving the expression ability of the Global Pooling component. The experiments show that our GSAP method improves the accuracy of graph classification by approximately 2–5%, the graph construction method improves the accuracy of graph classification by approximately 4–6%, and that the whole Visual Place Recognition model is robust to appearance change and view change.

HDROmni: Optical Extension of Dynamic Range for Panoramic Robot Vision

This paper presents the design method to optically extend the perceptible dynamic range of panoramic vision for robotics. The large range of scene radiance is captured in real-time by a combination of paraboloidal mirrors, a telecentric lens and neutral density filters. The first prototype made of 4 mirrors and 3 neutral density filters increases 100 times the perceptible dynamic range of the used camera. Results in various challenging scenes compare the new camera to low dynamic range ones. They show that the dynamic range quality of images doubles, on average. They also show significant benefit for feature-based approaches, i.e. much more matched features, and direct ones, i.e. much higher positioning accuracy of direct visual servoing.

RigidFusion: Robot Localisation and Mapping in Environments With Large Dynamic Rigid Objects

This work presents a novel RGB-D SLAM approach to simultaneously segment, track and reconstruct the static background and large dynamic rigid objects that can occlude major portions of the camera view. Previous approaches treat dynamic parts of a scene as outliers and are thus limited to a small amount of changes in the scene, or rely on prior information for all objects in the scene to enable robust camera tracking. Here, we propose to treat all dynamic parts as one rigid body and simultaneously segment and track both static and dynamic components. We, therefore, enable simultaneous localisation and reconstruction of both the static background and rigid dynamic components in environments where dynamic objects cause large occlusion. We evaluate our approach on multiple challenging scenes with large dynamic occlusion. The evaluation demonstrates that our approach achieves better motion segmentation, localisation and mapping without requiring prior knowledge of the dynamic object's shape and appearance.

Salient object detection via a boundary-guided graph structure

Graph-based salient object detection methods have gained more and more attention recently. However, existing works fail to separate effectively salient object and background in some challenging scenes. Inspired by this observation, we propose an effective salient object detection method based on a novel boundary-guided graph structure. More specifically, the input image is firstly segmented into a series of superpixels. Then we integrate two prior cues to generate the coarse saliency map, a novel weighting mechanism is proposed to balance the proportion of two prior cues according to their performance. Secondly, we propose a novel boundary-guided graph structure to explore deeply the intrinsic relevance between superpixels. Based on the proposed graph structure, an iterative propagation mechanism is constructed to refine the coarse saliency map. Experimental results on four datasets show adequately the superiority of the proposed method than other state-of-the-art methods.

Pairwise Coarse Registration of Indoor Point Clouds Using 2D Line Features

Registration is essential for terrestrial LiDAR (light detection and ranging) scanning point clouds. The registration of indoor point clouds is especially challenging due to the occlusion and self-similarity of indoor structures. This paper proposes a 4 degrees of freedom (4DOF) coarse registration method that fully takes advantage of the knowledge that the equipment is levelled or the inclination compensated for by a tilt sensor in data acquisition. The method decomposes the 4DOF registration problem into two parts: (1) horizontal alignment using ortho-projected images and (2) vertical alignment. The ortho-projected images are generated using points between the floor and ceiling, and the horizontal alignment is achieved by the matching of the source and target ortho-projected images using the 2D line features detected from them. The vertical alignment is achieved by making the height of the floor and ceiling in the source and target points equivalent. Two datasets, one with five stations and the other with 20 stations, were used to evaluate the performance of the proposed method. The experimental results showed that the proposed method achieved 80% and 63% successful registration rates (SRRs) in a simple scene and a challenging scene, respectively. The SRR in the simple scene is only lower than that of the keypoint-based four-point congruent set (K4PCS) method. The SRR in the challenging scene is better than all five comparison methods. Even though the proposed method still has some limitations, the proposed method provides an alternative to solve the indoor point cloud registration problem.

Edge-Aware Multi-Level Interactive Network for Salient Object Detection of Strip Steel Surface Defects

The performance of the salient object detection of strip surface defects has been promoted largely by deep learning based models. However, due to the complexity of strip surface defects, the existing models perform poorly in the challenging scenes such as noise disturbance, and low contrast between defect regions and background. Meanwhile, the detection results of existing models often suffer from coarse boundary details. Therefore, we propose a novel saliency model, namely an Edge-aware Multi-level Interactive Network, to detect the defects from the strip steel surface. Concretely, our model adopts the U-shape architecture where the two crucial points are the interactive feature integration and the edge-guided saliency fusion. Firstly, except the skip connection that combines the same stage of encoder and decoder, we deploy another connection, where the features from adjacent levels of encoder are transferred to the same stage of decoder. By this way, we are able to provide an effective fusion of multi-level deep features, yielding a well depiction for defects. Secondly, to give well-defined boundaries for prediction results, we add the edge extraction branch after each decoder block, where the progressive feature aggregation endows the edge with precise details and complete object cues. Meanwhile, together with the edge extraction branches, we deploy the saliency prediction branch at each decoder stage. After that, coupled with the fine edge information, we fuse all outputs of saliency prediction branches into the final saliency map, where the edge cue steers the saliency result to pay more attention to the boundary details. Following this way, we can provide a high-quality saliency map which can accurately locate and segment the defects. Extensive experiments are performed on the public dataset, and the results prove the effectiveness and robustness of our model which consistently outperforms the state-of-the-art models.

Dynamic Selective Network for RGB-D Salient Object Detection.

RGB-D saliency detection is receiving more and more attention in recent years. There are many efforts have been devoted to this area, where most of them try to integrate the multi-modal information, i.e. RGB images and depth maps, via various fusion strategies. However, some of them ignore the inherent difference between the two modalities, which leads to the performance degradation when handling some challenging scenes. Therefore, in this paper, we propose a novel RGB-D saliency model, namely Dynamic Selective Network (DSNet), to perform salient object detection (SOD) in RGB-D images by taking full advantage of the complementarity between the two modalities. Specifically, we first deploy a cross-modal global context module (CGCM) to acquire the high-level semantic information, which can be used to roughly locate salient objects. Then, we design a dynamic selective module (DSM) to dynamically mine the cross-modal complementary information between RGB images and depth maps, and to further optimize the multi-level and multi-scale information by executing the gated and pooling based selection, respectively. Moreover, we conduct the boundary refinement to obtain high-quality saliency maps with clear boundary details. Extensive experiments on eight public RGB-D datasets show that the proposed DSNet achieves a competitive and excellent performance against the current 17 state-of-the-art RGB-D SOD models.

Artefact‐free image stitching via a better normed seam‐cutting energy function

Image stitching, as the important field of computer graphics and vision, has received much attention in recent years. Image stitching techniques are generally decomposed into two phases: image alignment, which aligns target images with the reference images; and image composition, which fixes ghosting and visual artefacts. This work aims to propose a new strategy for the seam-cutting method which provides visually appealing result. Seam-cutting is one of the most influential methods in image composition, which can relieve artefacts and produce plausible results. However, it is observed that the state-of-the-art seam-cutting approaches usually lead to undesirable seams in some challenging scenes. Here, the authors put forward a novel seam-cutting method by defining a new energy function. This method uses 5 / 2 power of L 1 norm as a colour difference which can magnify the weight of colour distinction to avoid undesirable seams and artefacts. The proposed method can be easily implemented. The test images are collected from the public available challenging datasets and taken by ourselves. Experiments demonstrate that the proposed method can create comparable or even better stitching results compared to other state-of-the-art seam-cutting approaches.

Global-Guided Selective Context Network for Scene Parsing.

Recent studies on semantic segmentation are exploiting contextual information to address the problem of inconsistent parsing prediction in big objects and ignorance in small objects. However, they utilize multilevel contextual information equally across pixels, overlooking those different pixels may demand different levels of context. Motivated by the above-mentioned intuition, we propose a novel global-guided selective context network (GSCNet) to adaptively select contextual information for improving scene parsing. Specifically, we introduce two global-guided modules, called global-guided global module (GGM) and global-guided local module (GLM), to, respectively, select global context (GC) and local context (LC) for pixels. When given an input feature map, GGM jointly employs the input feature map and its globally pooled feature to learn its global contextual demand based on which per-pixel GC is selected. While GLM adopts low-level feature from the adjacent stage as LC and synthetically models the input feature map, its globally pooled feature and LC to generate local contextual demand, based on which per-pixel LC is selected. Furthermore, we combine these two modules as a selective context block and import such SCBs in different levels of the network to propagate contextual information in a coarse-to-fine manner. Finally, we conduct extensive experiments to verify the effectiveness of our proposed model and achieve state-of-the-art performance on four challenging scene parsing data sets, i.e., Cityscapes, ADE20K, PASCAL Context, and COCO Stuff. Especially, GSCNet-101 obtains 82.6% on Cityscapes test set without using coarse data and 56.22% on ADE20K test set.

DeepSLAM: A Robust Monocular SLAM System With Unsupervised Deep Learning

In this article, we propose DeepSLAM, a novel unsupervised deep learning based visual simultaneous localization and mapping (SLAM) system. The DeepSLAM training is fully unsupervised since it only requires stereo imagery instead of annotating ground-truth poses. Its testing takes a monocular image sequence as the input. Therefore, it is a monocular SLAM paradigm. DeepSLAM consists of several essential components, including Mapping-Net, Tracking-Net, Loop-Net, and a graph optimization unit. Specifically, the Mapping-Net is an encoder and decoder architecture for describing the 3-D structure of environment, whereas the Tracking-Net is a recurrent convolutional neural network architecture for capturing the camera motion. The Loop-Net is a pretrained binary classifier for detecting loop closures. DeepSLAM can simultaneously generate pose estimate, depth map, and outlier rejection mask. In this article, we evaluate its performance on various datasets, and find that DeepSLAM achieves good performance in terms of pose estimation accuracy, and is robust in some challenging scenes.

RFRN: A recurrent feature refinement network for accurate and efficient scene text detection

Scene text detection plays a vital role for scene text understanding, but arbitrary-shaped text detection remains a significant challenge. To extract discriminative features, most recent state-of-the-art methods adopt heavy networks, resulting in parameter redundancy and inference inefficiency. For accurate and efficient scene text detection, in this paper we propose a novel recurrent feature refinement network (RFRN). RFRN, as a recurrent segmentation framework, contains a recurrent path augmentation that refines the previous feature maps as inner states, which not only helps improve the segmentation quality, but also fully facilitates the reuse of parameters and low computational cost. During testing, RFRN discards redundant prediction procedures for efficient inference, and achieves a good balance between speed and accuracy of inference. We conduct experiments on four challenging scene text benchmarks, CTW1500, Total-Text, ICDAR2015 and ICDAR2017-MLT, which include curved texts and multi-oriented texts with complex background. The results show that the proposed RFRN achieves competitive performance on detection accuracy while maintaining computational efficiency.

A quality improvement method for 3D laser slam point clouds based on geometric primitives of the scan scene

ABSTRACT 3D laser simultaneous localization and mapping (SLAM) technology is one of the most efficient methods to capture spatial information. However, the low-accuracy SLAM point cloud limits its application in many fields. To address the problem, a laser SLAM point cloud quality improvement method based on geometric primitive (PCQI-GP) is proposed, which mainly includes extraction of reference datum and point cloud data correction. More specifically, the point cloud data quality is evaluated through extracting the geometric primitives of the scanning scene using the random sampling consistency algorithm, and the high-accuracy point cloud is defined as the reference datum. Furthermore, the primitive parameters extracted from reference datum are adopted to construct constraint conditions, and the coordinates of drifting point cloud data are corrected. Experiments are conducted with data collected by a handheld laser scan system in two challenging scenes to evaluate the PCQI-GP method. Compared with the theoretical design values, the correction accuracy of the PCQI-GP method is less than 3 cm. The experimental results demonstrate that the proposed method can effectively improve the quality of the laser SLAM point cloud in the area without global navigation satellite system (GNSS) signal and sufficient feature points.

TRBACF: Learning temporal regularized correlation filters for high performance online visual object tracking

Correlation filter-based trackers (CFTs) have recently shown remarkable performance in the field of visual object tracking. The advantage of these trackers originates from their ability to convert time-domain calculations into frequency domain calculations. However, a significant problem of these CFTs is that the model is insufficiently robust when the tracking scenarios are too complicated, meaning that the ideal tracking performance cannot be acquired. Recent work has attempted to resolve this problem by reducing the boundary effects from modeling the foreground and background of the object target effectively (e.g., CFLB, BACF, and CACF). Although these methods have demonstrated reasonable performance, they are often affected by occlusion, deformation, scale variation, and other challenging scenes. In this study, considering the relationship between the current frame and the previous frame of a moving object target in a time series, we propose a temporal regularization strategy to improve the BACF tracker (denoted as TRBACF), a typical representative of the aforementioned trackers. The TRBACF tracker can efficiently adjust the model to adapt the change of the tracking scenes, thereby enhancing its robustness and accuracy. Moreover, the objective function of our TRBACF tracker can be solved by an improved alternating direction method of multipliers, which can speed up the calculation in the Fourier domain. Extensive experimental results demonstrate that the proposed TRBACF tracker achieves competitive tracking performance compared with state-of-the-art trackers.

Scene Segmentation With Dual Relation-Aware Attention Network.

In this article, we propose a Dual Relation-aware Attention Network (DRANet) to handle the task of scene segmentation. How to efficiently exploit context is essential for pixel-level recognition. To address the issue, we adaptively capture contextual information based on the relation-aware attention mechanism. Especially, we append two types of attention modules on the top of the dilated fully convolutional network (FCN), which model the contextual dependencies in spatial and channel dimensions, respectively. In the attention modules, we adopt a self-attention mechanism to model semantic associations between any two pixels or channels. Each pixel or channel can adaptively aggregate context from all pixels or channels according to their correlations. To reduce the high cost of computation and memory caused by the abovementioned pairwise association computation, we further design two types of compact attention modules. In the compact attention modules, each pixel or channel is built into association only with a few numbers of gathering centers and obtains corresponding context aggregation over these gathering centers. Meanwhile, we add a cross-level gating decoder to selectively enhance spatial details that boost the performance of the network. We conduct extensive experiments to validate the effectiveness of our network and achieve new state-of-the-art segmentation performance on four challenging scene segmentation data sets, i.e., Cityscapes, ADE20K, PASCAL Context, and COCO Stuff data sets. In particular, a Mean IoU score of 82.9% on the Cityscapes test set is achieved without using extra coarse annotated data.