Features For Visual Tracking Research Articles

Visual Odometry of a Low-Profile Pallet Robot Based on Ortho-Rectified Ground Plane Image from Fisheye Camera

This study presents a visual-only odometry technique of a low-profile pallet robot using image feature tracking in ground plane images generated from a fisheye camera. The fisheye camera is commonly used in many robot vision applications because it provides a larger field of view (FoV) around a robot. However, because of the large radial distortion, the fisheye image is generally converted to a pinhole image for visual feature tracking or matching. Although the radial distortion can be eliminated via image undistortion with the lens calibration parameters, it causes several side effects, such as degraded image resolution and a significant reduction in the FoV. In this paper, instead of using the pinhole model, we propose to generate a ground plane image (GPI) from the fisheye image. GPI is a virtual top-view image that only contains the ground plane at the front of the robot. First, the original fisheye image is projected to several virtual pinhole images to generate a cubemap. Second, the front and bottom faces of the cubemap are projected to a GPI. Third, the GPI is homographically transformed again to further reduce image distortion. As a result, an accurate ortho-rectified ground plane image is obtained from the virtual top-view camera. For visual odometry using the ortho-rectified GPI, a number of 2D motion vectors are obtained using feature extraction and tracking between the previous and current frames in the GPI. By calculating a scaled motion vector, which is the measurement of the virtual wheel encoder of the mobile robot, we estimate the velocity and steering angle of the virtual wheel using the motion vector. Finally, we estimate the pose of the mobile robot by applying a kinematic model to the mobile robot.

Multi-Modal Feature Constraint Based Tightly Coupled Monocular Visual-LiDAR Odometry and Mapping

In this paper, we present a novel multi-sensor fusion framework for tightly coupled monocular visual-LiDAR odometry and mapping. Compared to previous visual-LiDAR fusion frameworks, our proposed framework leverages more constraints among LiDAR features and visual features and integrates that in a tightly coupled approach. Specifically, the framework starts with a preprocess module which contains LiDAR feature extraction, visual feature extraction and tracking, and visual feature depth recover. Then a frame-to-frame odometry module is established by fusing visual feature tracking and LiDAR feature match between frames, aiming to provide a coarse pose estimation for next module. Finally, to refine the pose and build a multi-modal map, we introduce a multi-modal mapping module that tightly couple multi-modal feature constraints by matching or registering multi-modal features to multi-modal map. In addition, the proposed fusion framework also functions well in sensor-degraded environment (texture-less or structure-less), which increases its robustness. The effectiveness and performance of the proposed fusion framework are demonstrated and evaluated on the public KITTI odometry benchmark, and results show that our proposed fusion framework achieves comparable performance compared with the state-of-the-art visual-LiDAR odometry frameworks.

Unsupervised Monocular Visual Odometry for Fast-Moving Scenes Based on Optical Flow Network with Feature Point Matching Constraint.

Robust and accurate visual feature tracking is essential for good pose estimation in visual odometry. However, in fast-moving scenes, feature point extraction and matching are unstable because of blurred images and large image disparity. In this paper, we propose an unsupervised monocular visual odometry framework based on a fusion of features extracted from two sources, that is, the optical flow network and the traditional point feature extractor. In the training process, point features are generated for scene images and the outliers of matched point pairs are filtered by FlannMatch. Meanwhile, the optical flow network constrained by the principle of forward-backward flow consistency is used to select another group of corresponding point pairs. The Euclidean distance between the matching points found by FlannMatch and the corresponding point pairs by the flow network is added to the loss function of the flow network. Compared with SURF, the trained flow network shows more robust performance in complicated fast-motion scenarios. Furthermore, we propose the AvgFlow estimation module, which selects one group of the matched point pairs generated by the two methods according to the scene motion. The camera pose is then recovered by Perspective-n-Point (PnP) or the epipolar geometry. Experiments conducted on the KITTI Odometry dataset verify the effectiveness of the trajectory estimation of our approach, especially in fast-moving scenarios.

A Coupled Visual and Inertial Measurement Units Method for Locating and Mapping in Coal Mine Tunnel.

Mobile robots moving fast or in scenes with poor lighting conditions often cause the loss of visual feature tracking. In coal mine tunnels, the ground is often bumpy and the lighting is uneven. During the movement of the mobile robot in this scene, there will be violent bumps. The localization technology through visual features is greatly affected by the illumination and the speed of the camera movement. To solve the localization and mapping problem in an environment similar to underground coal mine tunnels, we improve a localization and mapping algorithm based on a monocular camera and an Inertial Measurement Unit (IMU). A feature-matching method that combines point and line features is designed to improve the robustness of the algorithm in the presence of degraded scene structure and insufficient illumination. The tightly coupled method is used to establish visual feature constraints and IMU pre-integration constraints. A keyframe nonlinear optimization algorithm based on sliding windows is used to accomplish state estimation. Extensive simulations and practical environment verification show that the improved simultaneous localization and mapping (SLAM) system with a monocular camera and IMU fusion can achieve accurate autonomous localization and map construction in scenes with insufficient light such as coal mine tunnels.

VISUAL ODOMETRY OF A MOBILE PALETTE ROBOT USING GROUND PLANE IMAGE FROM A FISHEYE CAMERA

Abstract. In this paper, we present a method of mobile robot’s visual odometry using the visual feature tracking in the ground plane image generated from the fisheye image. In order to extract the feature information on the ground, we use a fisheye camera that has a larger FOV than a general pinhole camera, so that we can capture more information on the ground plane. However, due to the large distortion, it is difficult to extract the visual features in the fisheye image. The distortion can be eliminated, but various problems arise, such as a decrease in the resolution of an image or losing the wide angle of the fisheye camera. We propose the EUCM-Cubemap projection model to convert the fisheye image into the cubemap image without losing the FOV of the fisheye image. And we create the Ground Plane Image, a virtual image that vertically looks at the ground from a cube map image. So, the ground plane image is generated so that it is captured from a virtual camera perpendicular to the ground. In the ground plane image, the motion vector obtained by feature tracking between previous and current frames is proportional to the actual robot’s motion in the 2D ground plane. Thus, if we know the actual scale of the motion vector, we can estimate the mobile robot’s velocity and steering angle on the virtual wheel generated by the ground plane image. The scale of the vector can be estimated using the position and focal length of the camera. Using these parameters, we estimate the mobile robot’s pose by applying the bicycle kinematic model. Experimental results show that the proposed method can replace other conventional odometry methods for mobile robots. And, in the future, it is expected to be used in a variety of fields such as visual-based control or path planning.

Improved In-Flight Estimation of Inertial Biases through CDGNSS/Vision Based Cooperative Navigation.

This paper discusses the exploitation of a cooperative navigation strategy for improved in-flight estimation of inertial sensors biases on board unmanned aerial vehicles. The proposed multi-vehicle technique is conceived for a “chief” Unmanned Aerial Vehicle (UAV) and relies on one or more deputy aircrafts equipped with Global Navigation Satellite System (GNSS) antennas for differential positioning which also act as features for visual tracking. Combining carrier-phase differential GNSS and visual estimates, it is possible to retrieve accurate inertial-independent attitude information, thus potentially enabling improved bias estimation. Camera and carrier-phase differential GNSS measurements are integrated within a 15 states extended Kalman filter. Exploiting an ad hoc developed numerical environment, the paper analyzes the performance of the cooperative approach for inertial biases estimation as a function of number of deputies, formation geometry and distances, and absolute and relative dynamics. It is shown that exploiting two deputies it is possible to improve biases estimation, while a single deputy can be effective if changes of relative geometry and dynamics are also considered. Experimental proofs of concept based on two multi-rotors flying in formation are presented and discussed. The proposed framework is applicable beyond the domain of small UAVs.

Hybrid Cascade Filter With Complementary Features for Visual Tracking

Different features describe different aspects of the object. Individually tailoring proper features for visual tracking is crucial to obtain high performance. In this letter, we propose a hybrid cascade filter to fuse handcrafted and deep features for exploiting their strengths. We complement the deep representation with handcrafted features to achieve better localization accuracy, as well as build a hybrid cascade structure using multiple observation models to achieve better robustness. Furthermore, a coarse-to-fine searching strategy is used for lowering the computational cost. Extensive experimental results on two benchmark datasets show that the proposed method performs favorably against the state-of-the-art trackers.

Anytime Computation and Control for Autonomous Systems

The correct and timely completion of the sensing and action loop is of utmost importance in safety critical autonomous systems. Crucial to the performance of this feedback control loop are the computation time and accuracy of the estimator which produces state estimates used by the controller. These state estimators often use computationally expensive perception algorithms like visual feature tracking. With on-board computers on autonomous robots being computationally limited, the computation time of such an estimation algorithm can at times be high enough to result in poor control performance. We develop a framework for codesign of anytime estimation and robust control algorithms, taking into account computation delays and estimation inaccuracies. This is achieved by constructing an anytime estimator from an off-the-shelf perception-based estimation algorithm and obtaining a trade-off curve for its computation time versus estimation error. This is used in the design of a robust predictive control algorithm that at run-time decides a contract, or operation mode, for the estimator in addition to controlling the dynamical system to meet its control objectives at a reduced computation energy cost. This codesign provides a mechanism through which the controller can use the tradeoff curve to reduce estimation delay at the cost of higher inaccuracy, while guaranteeing satisfaction of control objectives. Experiments on a hexrotor platform running a visual-based algorithm for state estimation show how our method results in up to a 10% improvement in control performance while simultaneously saving 5%–6% in computation energy as compared to a method that does not leverage the codesign.

Localization-Aware Meta Tracker Guided With Adversarial Features

Deep learning has recently shown great potentials in learning powerful features for visual tracking. However, most deep learning-based trackers neglect localization accuracy in the evaluation process of candidates. What’s more, they usually over-rely on the discriminative features in a single frame in the training process. Consequently, they may fail when the discriminative features are occluded or changed in the tracking phase. In this paper, we propose a novel localization-aware meta tracker (LMT) guided with adversarial features to address the above issues. First of all, we design a novel intersection over union guided method to effectively balance the problem of classification and localization accuracy. To further improve the robustness of our classifier, we creatively use adversarial features during offline training phase. Those adversarial features can effectively guide the classifier in learning how to better deal with the situation where the discriminative features are occluded or changed. Finally, benefiting from meta learning, our algorithm only needs to perform one iterative update on the first frame and it can perform well on the tracking sceneries. The extensive experiments demonstrate the outstanding performance of our LMT compared with the state-of-the-art trackers on three benchmarks: OTB-2015, VOT-2016, and VOT-2018.

Robust L1 tracker with CNN features

Recently, L1 tracker has been widely applied and received great success in visual tracking. However, most L1 trackers use only the image intensity for sparse representation, which is insufficient to represent the object especially when drastic appearance changes occur. Convolutional neural network (CNN) has demonstrated remarkable capability in a wide range of computer vision fields, and features extracted from different convolutional layers have different characteristics. In this paper, we propose a novel sparse representation model with convolutional features for visual tracking. Besides, to alleviate the redundancy from high-dimensional convolutional features, a feature selection method is adopted to remove noisy and irrelevant feature maps, which can reduce computation redundancy and improve tracking accuracy. Different from traditional sparse representation based tracking methods, our model not only exploits convolutional features to improve the robustness for describing the object appearance but also uses the trivial templates to model both reconstruction errors caused by sparse representation and the eigen-subspace representation. In addition, an unified objective function is proposed and a customized APG method is developed to effectively solve the optimization problem. Numerous qualitative and quantitative evaluations demonstrate that our tracker outperforms other state-of-the-art trackers in a wide range of tracking scenarios.

Robust and real-time pose tracking for augmented reality on mobile devices

This paper addresses robust and ultrafast pose tracking on mobile devices, such as smartphones and small drones. Existing methods, relying on either vision analysis or inertial sensing, are either too computational heavy to achieve real-time performance on a mobile platform, or not sufficiently robust to address unique challenges in mobile scenarios, including rapid camera motions, long exposure time of mobile cameras, etc. This paper presents a novel hybrid tracking system which utilizes on-device inertial sensors to greatly accelerate the visual feature tracking process and improve its robustness. In particular, our system adaptively resizes each video frame based on inertial sensor data and applies a highly efficient binary feature matching method to track the object pose in each resized frame with little accuracy degradation. This tracking result is revised periodically by a model-based feature tracking method (Hare et al. 2012) to reduce accumulated errors. Furthermore, an inertial tracking method and a solution of fusing its results with the feature tracking results are employed to further improve the robustness and efficiency. We first evaluate our hybrid system using a dataset consisting of 16 video clips with synchronized inertial sensing data and then assess its performance in a mobile augmented reality application. Experimental results demonstrated our method’s superior performance to a state-of-the-art feature tracking method (Hare et al. 2012), a direct tracking method (Engel et al. 2014) and the Vuforia SDK (Ibanez and Figueras 2013), and can run at more than 40 Hz on a standard smartphone. We will release the source code with the pubilication of this paper.

Orientation estimation in modern wearables with visual feature tracking

In this paper we propose and examine a robust method for orientation estimation of wearables (like Google Glass) through bias compensation of gyroscope. Through integration of raw angular rates, the resulting orientation estimate will drift, due to the non-zero bias of the measured gyroscope was also accumulated. A simple error model was constructed for the measurement capabilities of the device, in terms of inertial sensing of rotation. Using the benefits of feature point displacements (as optical flow) from the camera of the device, a sensor fusion algorithm was developed to lower, and compensate the bias. The introduced orientation estimator and bias removal method is based on adaptive reliability filter for the optical flow feature points. To fuse the remaining feature point displacements into one single value, various aggregation methods have been tested, and a maximum a-posteriori estimator was applied on the dataset. To further lower the bias from the gyroscope, the output of the system was used as a feedback to calculate a bias estimate. To measure the performance and investigate the benefits of the asynchronous work-flow, the optical flow was performed on various devices in real time. To validate the results of the simulations, real world measurements were performed with industrial robots. Our reliable, adaptive filter matched our expectations, could compensate the bias well, and the integrated output of the system had no significant drift.

Visual Servoing Trajectory Tracking of Nonholonomic Mobile Robots Without Direct Position Measurement

Localization is one of the most difficult and costly problems in mobile robotics. To avoid this problem, this paper presents a new controller for the trajectory tracking of nonholonomic mobile robots using visual feedback without direct position measurement. This controller works on the basis of a novel adaptive algorithm for estimating the global position of the mobile robot online using natural visual features measured by a vision system and its orientation and velocity measured by odometry and Attitude and Heading Reference System (IMU&Compass) sensors. The nonholonomic motion constraint of mobile robots is fully taken into account, compared with most of the existing visual servo controllers for mobile robots. The Lyapunov theory is used to prove that the proposed adaptive visual servo controller gives rise to asymptotic tracking of a desired trajectory and convergence of the position estimation to the actual position. A graphical processing unit is adopted to implement the proposed adaptive controller in parallel to achieve real-time detection and tracking of visual features. Experiments on a mobile robot are conducted to validate the effectiveness and robust performance of the proposed controller.

A fuzzy inference approach to template-based visual tracking

The tracking of visual features using appearance models is a well studied but still open area of computer vision. In the absence of knowledge about the structural constraints of the tracked object, the validity of the model can be compromised if only appearance information is used. We propose a fuzzy inference scheme that can be used to selectively update a given template-based model in tracking tasks. This allows us to track moving objects under translation, rotation, and scale changes with minimal feature drift. Moreover, no rigidity constraint needs to be enforced on the moving target. Some experiments have been performed using several targets, and the results are very close to the ground truth paths. The computational cost of our approach is low enough to allow its application in real-time tracking using modest hardware requirements.

Motion perception: behavior and neural substrate.

Visual motion perception is vital for survival. Single-unit recordings in primate primary visual cortex (V1) have revealed the existence of specialized motion sensing neurons; perceptual effects such as the motion after-effect demonstrate their importance for motion perception. Human psychophysical data on motion detection can be explained by a computational model of cortical motion sensors. Both psychophysical and physiological data reveal at least two classes of motion sensor capable of sensing motion in luminance-defined and texture-defined patterns, respectively. Psychophysical experiments also reveal that motion can be seen independently of motion sensor output, based on attentive tracking of visual features. Sensor outputs are inherently ambiguous, due to the problem of univariance in neural responses. In order to compute stimulus direction and speed, the visual system must compare the responses of many different sensors sensitive to different directions and speeds. Physiological data show that this computation occurs in the visual middle temporal (MT) area. Recent psychophysical studies indicate that information about spatial form may also play a role in motion computations. Adaptation studies show that the human visual system is selectively sensitive to large-scale optic flow patterns, and physiological studies indicate that cells in the middle superior temporal (MST) area derive this sensitivity from the combined responses of many MT cells. Extraretinal signals used to control eye movements are an important source of signals to cancel out the retinal motion responses generated by eye movements, though visual information also plays a role. A number of issues remain to be resolved at all levels of the motion-processing hierarchy. WIREs Cogni Sci 2011 2 305-314 DOI: 10.1002/wcs.110 For further resources related to this article, please visit the WIREs website Additional Supporting Information may be found in http://www.lifesci.sussex.ac.uk/home/George_Mather/Motion/index.html.

Robust Real-time 3D Detection of Obstructed Head and Hands in Indoors Environments

We address the need for robust detection of obstructed human features in complex environments, with a focus on intelligent surgical UIs. In our setup, real-time detection is used to find features without the help of local (spatial or temporal) information. Such a detector is used to validate, correct or reject the output of the visual feature tracking, which is locally more robust, but drifts over time. In Operating Rooms (OR), surgeons’ faces and hands are typically obstructed by sterile clothing and tools, making statistical and/or feature-based feature detection approaches ineffective. We propose a new method for head and hands detection that relies on geometric information from disparity maps, locally refined by color processing. We have applied our method to a surgical mock- up scene, as well as to images gathered during real surgery. Running in a realtime, continuous detection loop, our detector successfully found more than 97% of target features, with very few false positives (less than 0.7%).