Local Feature Points Research Articles

PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detection

3D object detection is receiving increasing attention from both industry and academia thanks to its wide applications in various fields. In this paper, we propose Point-Voxel Region-based Convolution Neural Networks (PV-RCNNs) for 3D object detection on point clouds. First, we propose a novel 3D detector, PV-RCNN, which boosts the 3D detection performance by deeply integrating the feature learning of both point-based set abstraction and voxel-based sparse convolution through two novel steps, i.e., the voxel-to-keypoint scene encoding and the keypoint-to-grid RoI feature abstraction. Second, we propose an advanced framework, PV-RCNN++, for more efficient and accurate 3D object detection. It consists of two major improvements: sectorized proposal-centric sampling for efficiently producing more representative keypoints, and VectorPool aggregation for better aggregating local point features with much less resource consumption. With these two strategies, our PV-RCNN++ is about 3times faster than PV-RCNN, while also achieving better performance. The experiments demonstrate that our proposed PV-RCNN++ framework achieves state-of-the-art 3D detection performance on the large-scale and highly-competitive Waymo Open Dataset with 10 FPS inference speed on the detection range of 150m times 150m.

A biosensing system employing nonlinear dynamic analysis-assisted neural network for drug-induced cardiotoxicity assessment

Preclinical investigation of drug-induced cardiotoxicity is of importance for drug development. To evaluate such cardiotoxicity, in vitro high-throughput interdigitated electrode-based recording of cardiomyocytes mechanical beating is widely used. To automatically analyze the features from the beating signals for drug-induced cardiotoxicity assessment, artificial neural network analysis is conventionally employed and signals are segmented into cycles and feature points are located in the cycles. However, signal segmentation and location of feature points for different signal shapes require design of specific algorithms. Consequently, this may lower the efficiency of research and the applications of such algorithms in signals with different morphologies are limited. Here, we present a biosensing system that employs nonlinear dynamic analysis-assisted neural network (NDANN) to avoid the signal segmentation process and directly extract features from beating signal time series. By processing beating time series with fixed time duration to avoid the signal segmentation process, this NDANN-based biosensing system can identify drug-induced cardiotoxicity with accuracy over 0.99. The individual drugs were classified with high accuracies over 0.94 and drug-induced cardiotoxicity levels were accurately predicted. We also evaluated the generalization performance of the NDANN-based biosensing system in assessing drug-induced cardiotoxicity through an independent dataset. This system achieved accuracy of 0.85–0.95 for different drug concentrations in identification of drug-induced cardiotoxicity. This result demonstrates that our NDANN-based biosensing system has the capacity of screening newly developed drugs, which is crucial in practical applications. This NDANN-based biosensing system can work as a new screening platform for drug-induced cardiotoxicity and improve the efficiency of bio-signal processing.

Semantic Segmentation of 3D Point Clouds Based on High Precision Range Search Network

Semantic segmentation for 3D point clouds plays a critical role in the construction of 3D models. Due to the sparse and disordered natures of the point clouds, semantic segmentation of such unstructured data yields technical challenges. A recently proposed deep neural network, PointNet, delivers attractive semantic segmentation performance, but it only exploits the global features of point clouds without incorporating any local features, limiting its ability to recognize fine-grained patterns. For that, this paper proposes a deeper hierarchical structure called the high precision range search (HPRS) network, which can learn local features with increasing contextual scales. We develop an adaptive ball query algorithm that designs a comprehensive set of grouping strategies. It can gather detailed local feature points in comparison to the common ball query algorithm, especially when there are not enough feature points within the ball range. Furthermore, compared to the sole use of either the max pooling or the mean pooling, our network combining the two can aggregate point features of the local regions from hierarchy structure while resolving the disorder of points and minimizing the information loss of features. The network achieves superior performance on the S3DIS dataset, with a mIoU declined by 0.26% compared to the state-of-the-art DPFA network.

DSACNN: Dynamically local self-attention CNN for 3D point cloud analysis

The point cloud is a common 3D representation widely applied in CAX engineering due to its simple data representation and rich semantic information. However, discrete and unordered 3D data structures make it difficult for point clouds to understand semantic information and make them unsuitable for applying standard operators. In this paper, to enhance machine perception of 3D semantic information, we propose a novel approach that can not only directly process point cloud data by a novel convolution-like operator but also dynamically pay attention to local semantic information. First, we design a novel dynamic local self-attention mechanism that can dynamically and flexibly focus on top-level information of the receptive field to learn and understand subtle features. Second, we propose a dynamic self-attention learning block, which adopts the proposed dynamic local self-attention learning convolution operation to directly deal with disordered and irregular point clouds to learn global and local point features while dynamically learning the important local semantic information. Third, the proposed operation can be compatibly applied as an independent component in popular architectures to improve the perception of local semantic information. Numerous experiments demonstrate the advantage of our method for point cloud tasks on datasets from both CAD data and scan data of complex real-world scenes.

Real-time and high precision feature matching between blur aerial images.

When aerial cameras get aerial remote sensing images, the defocus will occur because of reasons such as air pressure, temperature and ground elevation changes, resulting in different image sharpness of continual aerial remote sensing images. Nowadays, the rapidly developing feature matching algorithm will rapidly reduce the registration rate between images with different image sharpness. Therefore, in order to enable aerial cameras to get image sharpness parameters according to the locations of aerial image feature points with inconsistent sharpness, this paper proposes a feature matching algorithm between aerial images with different sharpness by using DEM data and multiple constraints. In this paper, the feature matching range is extended according to the modified aerial imaging model and the nonlinear soft margin support vector machine. Then the relative moving speed and its variation of the feature points in the image are obtained by using the extended L-k optical flow, and finally the epipolar geometric constraint is introduced. To locate the feature points is obtained under multiple constraints, there is no need to calculate the feature point descriptors, and some mismatched point pairs are corrected, which improves the matching efficiency and precision. The experimental results show the feature matching precision of this algorithm is more than 90%, and the running time and matching precision can meet various application needs of aerial cameras.

Automatic registration of urban high-resolution remote sensing images based on characteristic spatial objects

Automatic registration of high-resolution remote sensing images (HRRSIs) has always been a severe challenge due to the local deformation caused by different shooting angles and illumination conditions. A new method of characteristic spatial objects (CSOs) extraction and matching is proposed to deal with this difficulty. Firstly, the Mask R-CNN model is utilized to extract the CSOs and their positioning points on the images automatically. Then, an encoding method is provided to encode each object with its nearest adjacent 28 objects according to the object category, relative distance, and relative direction. Furthermore, a code matching algorithm is applied to search the most similar object pairs. Finally, the object pairs need to be filtered by position matching to construct the final control points for automatic image registration. The experimental results demonstrate that the registration success rate of the proposed method reaches 88.6% within a maximum average error of 15 pixels, which is 28.6% higher than that of conventional optimization method based on local feature points. It is reasonable to believe that it has made a beneficial contribution to the automatic registration of HRRSIs more accurately and efficiently.

Optimization Method of Temperature Measuring Point Layout for Steel-Concrete Composite Bridge Based on TLS-IPDP

An optimization method of temperature measurement point layout for steel-concrete composite bridges based on the total least squares improved piecewise Douglas–Peucker (TLS-IPDP) algorithm was proposed to solve the problem that the traditional temperature measurement data cannot reflect the actual temperature gradient (TG) due to the position of measurement points on different paths is not reasonable. The characteristic curves of TG for the most unfavorable period and annual period are extracted from the finite element model. The rationality of the proposed method is illustrated by two typical steel-concrete composite beams with steel plates and steel boxes. By improving the classical Douglas–Peucker (DP) algorithm, the TLS-IPDP algorithm proposed in this paper has a better approximation effect on the original data. Compared with the traditional temperature measuring point arrangement method, the TLS-IPDP algorithm optimized arrangement in this paper realized the measuring point arrangement with different variable spacing under different paths; the temperature gradient curve obtained was closer to the real temperature distribution, and had higher accuracy in the region with a large gradient. In addition, the proposed method has the function of manually specifying the location of feature points and reserving the required number. The optimized arrangement of measuring points can meet the requirements of measuring points number and measurement accuracy. The method presented in this paper can provide a useful reference for temperature data acquisition and sensor layout for health monitoring of steel-composite bridges.

3D Animation Automatic Generation System Design Based on Deep Learning.

In the field of 3D animation design and generation, the expression generation method of animation is not obvious due to the lack of image details, which leads to the lack of realism of the generated animation expressions. In order to solve this problem, a deep learning-based animation character expression generation method is proposed. The method, based on the real facial expression images, uses improved deep learning to design cascade classifiers, extracts facial expression feature images from real images, softens image edges, and enhances feature details. The content and style of images are unified, the loss function is designed from the content constraints and style constraints, the judgment network is optimized, and the feature information is fused under the constraints of the loss function to generate the facial expressions of animated characters. The experimental results show that the design based on the feature point location of the improved deep learning expression generation method is accurate, the Pearson correlation coefficient between the input image and the generated image is high, the root mean square error is small, and the realism of the generated facial expression is enhanced.

Research on occlusion block face recognition based on feature point location

AbstractAiming at improving the poor face recognition accuracy under occlusion, a multipose block occlusion face recognition method based on feature point location is proposed. Face segmentation is carried out according to the location results of face feature points and occlusion areas. The local features of each face block are extracted by deep convolution neural network. The dynamic adaptive weighting method is used to give different weights to the face block information, and the occluded face recognition is completed according to the results of face segmentation, which effectively reduces the impact of pose change and occlusion on face recognition. The experiment is analyzed from two aspects: frontal occlusion and multipose face occlusion. The results show that fewer blocks in the frontal occlusion experiment are conducive to maintaining good local integrity and relatively good recognition performance; When the proportion of frontal occlusion reaches 50%, the recognition rate of our algorithm can still be up to 92.68%, which is significantly better than other algorithms.

An accurate pose measurement method of workpiece based on rapid extraction of local feature points

An accurate pose measurement method of workpiece based on rapid extraction of local feature points

A depth-colour image registration method based on local feature point extraction

A binocular camera composed of colour and depth cameras is an accurate and inexpensive tool for recognising fruit trees to plan spraying paths. For this purpose, however, the accurate visual registration of the depth and colour cameras is a problem that must be solved. This paper proposes a local feature point extraction (LFPE) algorithm for depth-colour image registration, which automatically extracts the feature points of local regions in the colour and depth images for registration. First, the binarised colour and depth images are divided into local regions after edge extraction, and the regions in which the edge features match at several pixels between images are selected. Then, these regions are again divided into smaller regions to detect local extrema of the pixels as feature points. Finally, the matching feature points are calculated based on the Euclidean distance and are used to optimise the affine transformation matrix. To verify the accuracy and effectiveness of the algorithm, it is compared with the FAST, SURF and BRISK algorithms. The experimental results show that the accuracy of the proposed LFPE algorithm is 1.6 times that of BRISK, 3.5 times that of FAST and 40.1 times that of SURF. Therefore, it is more suitable for registration between depth and colour images. • Proposed a depth-colour image registration method for orchard precision spraying. • A local feature point extraction (LFPE) algorithm is proposed to match images. • The registration accuracy of LFPE is much better than other similar algorithms.

QRS complexes and T waves localization in multi-lead ECG signals based on deep learning and electrophysiology knowledge

Accurate locations of key feature points for QRS complexes and T waves in electrocardiograms (ECGs) play a vital role in the analysis of cardiovascular diseases, such as arrhythmia and myocardial infarction. Because accuracy is affected by specific morphological changes of QRS waves and ST-T segments, traditional methods of detecting these waves need to be improved, which depend on empirical parameters and lack of robustness. This paper described a novel QRS complex and T wave localization method based on a U-Net framework combining the convolution neural network and long short-term memory with ensemble learning for multi-lead ECG signals. Using multi-lead ECG as an input, the model extracted strong temporal correlation features and fine morphological features in the training process, and its output was the probability value for every sample point. Dynamic threshold adaptive adjustment strategies during decision-making were then used to locate QRS complexes and T waves. Based on electrophysiology knowledge (EK), this method considered each lead information and was used to reduce missed and false detections. The proposed method based on deep learning and EK was also validated by three public databases. The experimental results indicated that the proposed method achieved a comparable or even better performance, when compared with present state-of-art methods. The currently proposed method produced results with high sensitivity and positive predictive values ranging from 99.45% to 100% for the locations of QRS complex waves, and from 99.99% to 97.88% for the locations of T waves from three datasets. Moreover, definitive analyses for missed and false detections of ECG signals were also used. These signals contained specific QRS complexes and T waves, such as higher amplitude T waves and QRS complexes, which contained fine structures, inverted U waves behind T waves, and waves along with atrial fibrillation and ventricular premature events.

DAN-SuperPoint: Self-Supervised Feature Point Detection Algorithm with Dual Attention Network.

In view of the poor performance of traditional feature point detection methods in low-texture situations, we design a new self-supervised feature extraction network that can be applied to the visual odometer (VO) front-end feature extraction module based on the deep learning method. First, the network uses the feature pyramid structure to perform multi-scale feature fusion to obtain a feature map containing multi-scale information. Then, the feature map is passed through the position attention module and the channel attention module to obtain the feature dependency relationship of the spatial dimension and the channel dimension, respectively, and the weighted spatial feature map and the channel feature map are added element by element to enhance the feature representation. Finally, the weighted feature maps are trained for detectors and descriptors respectively. In addition, in order to improve the prediction accuracy of feature point locations and speed up the network convergence, we add a confidence loss term and a tolerance loss term to the loss functions of the detector and descriptor, respectively. The experiments show that our network achieves satisfactory performance under the Hpatches dataset and KITTI dataset, indicating the reliability of the network.

An improved lightweight deep neural network with knowledge distillation for local feature extraction and visual localization using images and LiDAR point clouds

Visual localization nowadays is a research hotspot in computer vision and photogrammetry. It can provide meter level or higher localization accuracy under the conditions without GPS signals. However, achieving efficient, robust and high-accuracy visual localization under the condition of day-night changes is still challenging. To deal with this problem, we develop an improved lightweight deep neural network with knowledge distillation to efficiently extract deep local features from imagery while maintaining strong robustness for day-night visual localization. Furthermore, to further improve the accuracy of visual localization, we use aligned dense LiDAR point clouds and imagery collected by a new portable camera-LiDAR integrated device to build a prior map, and directly utilize the 2D-3D correspondences between 2D local feature points extracted by our lightweight network and 3D laser points retrieved from the prior map for localization. Moreover, we build our own ground truth point cloud dataset at 5 cm accuracy to evaluate the accuracy of the constructed prior map as well as a day-night dataset including prior map and verification data for the evaluation of the proposed visual localization method. The experimental results prove that our visual localization method achieves a balance between the efficiency and robustness while improving localization accuracy for day-night visual localization. In a comparison with a variety of state-of-the-art local feature extraction methods based on deep neural networks, our lightweight network has the least number of parameters (0.2 million) and reaches the highest feature extraction efficiency (92 frames per second), which is on par with that of the classic real-time ORB feature extraction method. Furthermore, our network remains competitive with other advanced deep local feature extraction networks in feature matching and day-night visual localization. In addition, evaluations performed on our own dataset demonstrate that our visual localization method using images and LiDAR point clouds provides a localization error of 1.2 m under the conditions of day-night changes, which is much smaller than those achieved by a state-of-the-art, purely visual localization method.

Road rage detection algorithm based on fatigue driving and facial feature point location

Road rage detection algorithm based on fatigue driving and facial feature point location

Fast Extraction Algorithm for Local Edge Features of Super-Resolution Image

Image super-resolution is getting popularity these days in diverse fields, such as medical applications and industrial applications. The accuracy is imperative on image super-resolution. The traditional approaches for local edge feature point extraction algorithms are merely based on edge points for super-resolution images. The traditional algorithms are used to calculate the geometric center of gravity of the edge line when it is near, resulting in a low feature recall rate and unreliable results. In order to overcome these problems of lower accuracy in the existing system, an attempt is made in this research work to propose a new fast extraction algorithm for local edge features of super-resolution images. This paper primarily focuses on the super-resolution image reconstruction model, which is utilized to extract the super-resolution image. The edge contour of the super-resolution image feature is extracted based on the Chamfer distance function. Then, the geometric center of gravity of the closed edge line and the nonclosed edge line are calculated. The algorithm emphasizes on polarizing the edge points with the center of gravity to determine the local extreme points of the upper edge of the amplitude-diameter curve and to determine the feature points of the edges of the super-resolution image. The experimental results show that the proposed algorithm consumes 0.02 seconds to extract the local edge features of super-resolution images with an accuracy of up to 96.3%. The experimental results show that our proposed algorithm is an efficient method for the extraction of local edge features from the super-resolution images.

A Real-Time CPU-GPU Embedded Implementation of a Tightly-Coupled Visual-Inertial Navigation System

In autonomous navigation technologies, the Multi-State Constraint Kalman Filter (MSCKF) is one of the most accurate and robust tightly-coupled fusion frameworks for Visual-Inertial Navigation (VIN). However, the adoption of the MSCKF VIN system in real-time embedded applications depends heavily on an efficient implementation of its tangled pipeline. This work initially proposes a novel parallel multi-thread implementation of the MSCKF VIN pipeline on an embedded CPU-enabled hardware that has speeded up the per-epoch processing time of the pipeline by 41% compared to the conventional sequential implementation. The heart of the MSCKF pipeline’s visual backend is an inertially-aided 3D localization of visual feature points that are reduced to a set of nonlinear optimization problems which were conventionally solved in a serial fashion using the single-objective Gauss-Newton optimization algorithm. This work leveraged the parallel architecture of an embedded GPU and further proposes an efficient parallel implementation of a multi-objective Gauss-Newton algorithm. Integration of the proposed GPU-accelerated feature localization technique in the MSCKF parallel pipeline has resulted in 33% faster per-epoch processing time and consequently, the satisfaction of strict real-time constraints. The proposed parallel MSCKF VIN pipelines have been developed using C++ and CUDA on the NVIDIA Jetson TX2 embedded board. Experimental evaluations on a real visual-inertial odometry dataset have been provided to validate the efficacy and real-time performance enhancement of the proposed parallel implementation.

QBB: Quantile-Based Binarization of 3D Point Cloud Descriptors

Local 3D point feature descriptors play an important role in many areas of computer vision, such as object recognition, registration, etc. There are many well-functioning feature descriptors, but they are typically real-valued and multidimensional vectors, leading to high computational complexity in nearest neighbor searches. To overcome this challenge, methods binarizing real-valued descriptors have emerged. In this paper, we first investigate the available binarization methods and standalone binary feature descriptors and show that existing binarization techniques cannot generally achieve good performance for arbitrary feature descriptors. To remedy this problem, we propose a new binarization method called quantile-based binarization (QBB) that can be applied to any real-valued feature descriptors. It analyses the distribution of feature descriptors that is then used to form meaningful groups along each dimension. To this end, QBB computes quantiles of the empirical distribution and the interval lengths (bin sizes) defined by quantile boundaries. Finally, it assigns a binary code to each group and concatenate them to get the final binary descriptor. QBB is able to adaptively compute the number of bits based on a capacity constraint, i.e., with the appropriate capacity setting, the resulting binary descriptor can be used on devices with lower computational power. We evaluate the descriptiveness of well-known descriptors binarized by QBB and compare them to state-of-the-art methods. According to our evaluation, QBB is able to create binary descriptors whose descriptiveness is closer to the real-valued descriptors than prior approaches. Finally, we also show that QBB can even compete with standalone binary feature descriptors.

Local neighborhood feature point extraction and matching for point cloud alignment

Local neighborhood feature point extraction and matching for point cloud alignment

Residual 3-D Scene Flow Learning With Context-Aware Feature Extraction

Scene flow estimation is the task to predict the point-wise or pixel-wise 3-D displacement vector between two consecutive frames of point clouds or images, which has important application in fields such as service robots and autonomous driving. Although many previous works have explored greatly on scene flow estimation based on point clouds, there are two problems that have not been noticed or well solved before: 1) points of adjacent frames in repetitive patterns tend to be wrongly associated due to similar spatial structure in their neighborhoods and 2) scene flow between adjacent frames of point clouds with long-distance movement tends to be inaccurately estimated. To solve the first problem, a novel context-aware set convolution layer is proposed in this article to exploit contextual structure information of Euclidean space and learn soft aggregation weights for local point features. This design is inspired by human perception of contextual structure information during scene understanding with repetitive patterns. For the second problem, an explicit residual flow learning structure is proposed in the residual flow refinement layer to cope with long-distance movement. The experiments and ablation study on FlyingThings3D and Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) scene flow datasets demonstrate the effectiveness of each proposed component. The qualitative results show that the problems of ambiguous interframe association and long-distance movement estimation are well handled. Quantitative results on both FlyingThings3D and KITTI scene flow datasets show that the proposed method achieves state-of-the-art performance, surpassing all previous works to the best of our knowledge by at least 26.22% on FlyingThings3D dataset and 25.42% on KITTI scene flow dataset for 3-D endpoint error (EPE3D) metric. The source codes are available at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/IRMVLab/CARFlow</uri> .