Trifocal Tensor Research Articles

HIGH-RESOLUTION SATELLITE IMAGE TRIFOCAL TENSOR SOLUTION

Abstract. This study presents an efficient algorithm for solving the affine trifocal tensor problem in high-resolution satellite imagery. Because the imaging mode of high-resolution satellites is usually a line-center projection and the field of view is small, it can be approximated as an affine camera for orientation. However, the nature of high-resolution satellite trifocal tensors has not been effectively studied or discussed. To address the shortcomings of the current trifocal tensor direct linear algorithm, this study proposes a minimally parameterized trifocal calculation method that can stably calculate the affine trifocal tensors of high-resolution satellites. Efficacy was demonstrated using ZY3-02 and Pleiades-Neo data.

Estimation of Extrinsic Parameters With Trifocal Tensor for Intelligent Vehicle-Mounted Cameras

In this article, the extrinsic parameter estimation of a camera mounted on an intelligent vehicle is addressed. The trifocal tensor is utilized to construct vision dynamics which relates image coordinates, velocity signals, and extrinsic parameters. Artificial visual patterns such as chessboards and planar reference objects used in homography-based methods are no longer required. An auxiliary tensor decouples the rotational extrinsic parameters from the translational ones. A key frame strategy is adopted to deal with the field of view constraint and an unknown distance is eliminated from the vision dynamics to counter the scale change caused by key frame switching. The Lyapunov method is used to design nonlinear observers, which estimate the extrinsic parameters at each time step based all collected valid historical data. Performance of the proposed method is verified by both simulation and experimental results.

Towards real-time 3D visualization with multiview RGB camera array.

A real-time 3D visualization (RT3DV) system using a multiview RGB camera array is presented. RT3DV can process multiple synchronized video streams to produce a stereo video of a dynamic scene from a chosen view angle. Its design objective is to facilitate 3D visualization at the video frame rate with good viewing quality. To facilitate 3D vision, RT3DV estimates and updates a surface mesh model formed directly from a set of sparse key points. The 3D coordinates of these key points are estimated from matching 2D key points across multiview video streams with the aid of epipolar geometry and trifocal tensor. To capture the scene dynamics, 2D key points in individual video streams are tracked between successive frames. We implemented a proof of concept RT3DV system tasked to process five synchronous video streams acquired by an RGB camera array. It achieves a processing speed of 44 milliseconds per frame and a peak signal to noise ratio (PSNR) of 15.9 dB from a viewpoint coinciding with a reference view. As a comparison, an image-based MVS algorithm utilizing a dense point cloud model and frame by frame feature detection and matching will require 7 seconds to render a frame and yield a reference view PSNR of 16.3 dB.

Robust line segment matching across views via ranking the line-point graph

Line segment matching in two or multiple views is helpful to 3D reconstruction and pattern recognition. To fully utilize the geometry constraint of different features for line segment matching, a novel graph-based algorithm denoted as GLSM (Graph-based Line Segment Matching) is proposed in this paper, which includes: (1) the employment of three geometry types, i.e., homography, epipolar, and trifocal tensor, to constrain line and point candidates across views; (2) the method of unifying different geometry constraints into a line-point association graph for two or multiple views; and (3) a set of procedures for ranking, assigning, and clustering with the line-point association graph. The experimental results indicate that GLSM can obtain sufficient matches with a satisfactory accuracy in both two and multiple views. Moreover, GLSM can be employed with large image datasets. The implementation of GLSM will be available soon at https://skyearth.org/research/.

Parallel feature based calibration method for a trinocular vision sensor.

In this paper, a new method to calibrate a trinocular vision sensor is presented. A planar target with several parallel lines is utilized. The trifocal tensor of three image planes can be calculated out according to line correspondences. Compatible essential matrix between each two cameras can be obtained. Then, rotation matrix and translation matrix can be deduced base on singular value decomposition of their corresponding essential matrix. In our proposed calibration method, image rectification is carried out to remove perspective distortion. As the feature utilized is straight line, precise point to point correspondence is not necessary. Experimental results show that our proposed calibration method can obtain precise results. Moreover, the trifocal tensor can also give a strict constraint for feature matching as descripted in our previous work. Root mean square error of measured distances is 0.029 mm with regards to the view field of about 250×250 mm. As parallel feature exists widely in natural scene, our calibration method also provides a new approach for self-calibration of a trinocular vision sensor.

A novel adaptive federated filter for GNSS/INS/VO integrated navigation system

In order to solve the problem of decreased navigation performance of the Global Navigation Satellite System (GNSS)/inertial navigation system (INS) integrated navigation systems in GNSS-denied environments, and to improve the navigation accuracy and robustness of the navigation system, a novel adaptive federated filter with a feedback scheme for a GNSS/INS/visual odometry (VO) integrated navigation system is proposed in this paper. A visual-inertial odometry system model with a multi-state constraint Kalman filter structure based on a polar geometry and trifocal tensor geometry between different images is established, which can provide better navigation accuracy in GNSS-denied environments. Moreover, a new method to obtain the information allocation factor according to the different navigation performances of local filters is deduced in this paper, which has low computational complexity and a simple structure. Meanwhile, an abnormal measurement detection algorithm based on fuzzy logic is proposed to detect the abnormal measurements of local filters. The results of the vehicle experiment with the publicly available real-world KITTI dataset show that the proposed algorithm can obtain reliable navigation results in GNSS-denied environments and improve the navigation accuracy and robustness of the GNSS/INS/VO integrated navigation system.

Robust adaptive filter using fuzzy logic for tightly‐coupled visual inertial odometry navigation system

In this study, a robust adaptive filter using fuzzy logic for tightly-coupled visual-inertial odometry (VIO) navigation system is proposed. First, the authors use the epipolar geometry and trifocal tensor geometry as the measurement models of the VIO navigation system, which can avoid calculating the three-dimensional position of feature points. Second, the camera poses corresponding to the three images are corrected in the filter to form a multi-state constraint Kalman filter. Finally, when the measurement noise statistics changes or the erroneous measurement occurs in practical applications, the proposed method utilises Takagi–Sugeno (T–S) fuzzy logic to determine a scalar factor according to the divergence degree parameter. Then, the scalar factor is applied to the innovation covariance matrix and the filter gain, which improves the navigation accuracy and robustness of the VIO navigation system. The experimental results testing with the publicly available real-world KITTI dataset demonstrate the effectiveness of the proposed method.

Accurate Line-Based Relative Pose Estimation With Camera Matrices

While most monocular structure-from-motion frameworks rely on sparse keypoints, it has long been acknowledged that lines represent an alternative, higher-order feature with high accuracy, repeatability, and abundant availability in man-made environments. Its exclusive use, however, is severely complicated by its inability to resolve the common bootstrapping scenario of two-view geometry. Even with stereo cameras, a one-dimensional disparity space, as well as ill-posed triangulations of horizontal lines make the realization of purely line-based tracking pipelines difficult. The present paper successfully leverages the redundancy in camera matrices to alleviate this shortcoming. We present a novel stereo trifocal tensor solver and extend it to the case of two camera matrix view-points. Our experiments demonstrate superior behavior with respect to both 2D-2D and 3D-3D alternatives. We furthermore outline the camera matrix’s ability to continuously and robustly bootstrap visual motion estimation pipelines via integration into a robust, purely line-based visual odometry pipeline. The result leads to state-of-the-art tracking accuracy comparable to what is achieved by point-based stereo or even dense depth camera alternatives.

Improved Sensors Based on Scheimpflug Conditions and Multi-Focal Constraints

Binocular vision sensor composed of two traditional camerasis widely utilized in many fields. However, there are still twodeficiencies, i.e. limited measurement area and weakconstraint for feature matching. In this paper, we firstintroduced measurement principle of the Scheimpflug camera and its calibrationmethod. Then an improved binocular vision sensor composed of two Scheimpflug cameras is constructed. Measurement area is maximized.Then, a trinocular vision sensor as an example of the multi-viewstereo vision sensor is analyzed and multi-focal constraint basedmeasurement model is given. Feature matching according to theconstraint of a trifocal tensor is strict, which is verified by relatedexperiments. Calibration procedures for the camera and improved visionsensor are detailed. Experiment results show that measurement area ofour improved vision sensor increases up to 86.2% (when tilted angleis 4°). Root mean square error of distance is0.107 0 / 00 with regard to the measurementarea.

Visual Navigation Features Selection Algorithm Based on Instance Segmentation in Dynamic Environment

Ego-motion estimation, as one of the core technologies of unmanned systems, is widely used in autonomous robot navigation, unmanned driving, augmented reality and other fields. With the development of computer vision, there has been considerable interest in ego-motion estimation with visual navigation. One of the core technologies in Visual navigation is using the matching feature points between consecutive image frames to estimate pose. Since the feature-based method performed under the assumption of a static environment, it susceptive to the dynamic targets. Visual navigation in the dynamic environment has become an important research issue. This paper proposed a practical and robust features selection algorithm of visual navigation which avoids using the feature points on dynamic objects. Firstly, according to the instance segmentation of deep neural network, the objects are classified into potential dynamic and static categories. Subsequently, the matching features on the potential moving objects are used to update vehicle state respectively, meanwhile, the relevant reprojection error of other feature points on the background could be calculated. Eventually, the result of whether the target is moving or not will be judged by the reprojection error, and the features on dynamic targets are removed. To illustrate the effectiveness of the features selection method in the dynamic environment, the proposed algorithm is merged into an MSCKF based on tri-focal tensor geometry, and it has been evaluated in a public dataset. Experimental results demonstrated the effectiveness of the proposed method.

Trifocal tensor-based 6-DOF visual servoing

This paper proposes a trifocal tensor-based approach for six-degree-of-freedom visual servoing. The trifocal tensor model among the current, desired, and reference views is constructed to describe the geometric relationship of the system. More precisely, to ensure the computation consistency of trifocal tensor, a virtual reference view is introduced by exploiting the transfer relationships between the initial and desired images. Instead of resorting to explicit estimation of the camera pose, a set of visual features with satisfactory decoupling properties are constructed from the tensor elements. Based on the selected features, a visual controller is developed to regulate the camera to a desired pose, and an adaptive update law is used to compensate for the unknown distance scale factor. Furthermore, the system stability is analyzed via Lyapunov-based techniques, showing that the proposed controller can achieve almost global asymptotic stability. Both simulation and experimental results are provided to demonstrate the effectiveness and robustness of our approach under different conditions and case studies.

GENERATION OF A BENCHMARK DATASET USING HISTORICAL PHOTOGRAPHS FOR AN AUTOMATED EVALUATION OF DIFFERENT FEATURE MATCHING METHODS

Abstract. This contribution shows the generation of a benchmark dataset using historical images. The difficulties when working with historical images are pointed out and structured in three categories. Especially large viewpoint differences, image artifacts and radiometric differences lead to weak matching results with classical feature matching approaches. The necessity of publishing an own benchmark dataset is emphasized when comparing to existing datasets which are partly using synthetic data, well-known orientation or strictly categorized image differences. The presented image dataset consists at the moment of 24 images which are oriented in image triples using the properties of the Trifocal Tensor as a more stable image geometry. In the following, three different feature detectors and descriptors that have already been proven well on historical images (MSER, ORB, RIFT) are evaluated using the new benchmark dataset. Then, several outlier removal methods were applied on the detected features. The tests show that for the entirety of image pairs RIFT performs slightly better than the other two methods. Nonetheless, for some image pairs MSER significantly improves the matching score but even so, historical image pairs are difficult to be matched with the presented methods due to challenging outlier removal. Still, the estimated projective relative orientation could be used in an autocalibration approach to place the images in a metric scene.

Pedestrian Detection for Autonomous Vehicle Using Multi-Spectral Cameras

Pedestrian detection is a critical feature of autonomous vehicle or advanced driver assistance system. This paper presents a novel instrument for pedestrian detection by combining stereo vision cameras with a thermal camera. A new dataset for vehicle applications is built from the test vehicle recorded data when driving on city roads. Data received from multiple cameras are aligned using trifocal tensor with pre-calibrated parameters. Candidates are generated from each image frame using sliding windows across multiple scales. A reconfigurable detector framework is proposed, in which feature extraction and classification are two separate stages. The input to the detector can be the color image, disparity map, thermal data, or any of their combinations. When applying to convolutional channel features, feature extraction utilizes the first three convolutional layers of a pre-trained convolutional neural network cascaded with an AdaBoost classifier. The evaluation results show that it significantly outperforms the traditional histogram of oriented gradients features. The proposed pedestrian detector with multi-spectral cameras can achieve 9% log-average miss rate. The experimental dataset is made available at http://computing.wpi.edu/dataset.html.

Developing Computationally Efficient Nonlinear Cubature Kalman Filtering for Visual Inertial Odometry

This paper presents a computationally efficient sensor-fusion algorithm for visual inertial odometry (VIO). The paper utilizes trifocal tensor geometry (TTG) for visual measurement model and a nonlinear deterministic-sampling-based filter known as cubature Kalman filter (CKF) to handle the system nonlinearity. The TTG-based approach is developed to replace the computationally expensive three-dimensional-feature-point reconstruction in the conventional VIO system. This replacement has simplified the system architecture and reduced the processing time significantly. The CKF is formulated for the VIO problem, which helps to achieve a better estimation accuracy and robust performance than the conventional extended Kalman filter (EKF). This paper also addresses the computationally efficient issue associated with Kalman filtering structure using cubature information filter (CIF), the CKF version on information domain. The CIF execution avoids the inverse computation of the high-dimensional innovation covariance matrix, which in turn further improves the computational efficiency of the VIO system. Several experiments use the publicly available datasets for validation and comparing against many other VIO algorithms available in the recent literature. Overall, this proposed algorithm can be implemented as a fast VIO solution for high-speed autonomous robotic systems.

Trinocular vision system based on cameras with tilt–shift lens

In multi-camera vision system, two major deficiencies are the limited field of view and the difficulty of feature matching. In this paper, a camera with tilt–shift lens is designed and a trinocular vision system based on the camera is detailed. The tilt–shift lens, which meets the Scheimpflug condition, can enlarge the overlapping area of the trinocular vision system. The trifocal tensor which can provide a strong constraint is determined, and the trifocal tensor is used for feature matching and 3D reconstruction in this paper.

Minimum Sigma Set SR-UKF for Quadrifocal Tensor-based Binocular Stereo Vision-IMU Tightly-coupled System

This paper presents a binocular vision-IMU (Inertial Measurement Unit) tightly-coupled structure based on a Minimum sigma set Square-Root Unscented Kalman Filter (M-SRUKF) for real time navigation applications. Though the M-SRUKF has only half the sigma points of the SRUKF, it has the same accuracy as the SRUKF when applied to the binocular vision-IMU tightly-coupled system. As the Kalman filter flow is a kind of square-root system, the stability of the system can be guaranteed. The measurement model and the outlier rejection model of this tightly-coupled system not only utilises the epipolar constraint and the trifocal tensor geometry constraint between the consecutive two image pairs, but also uses the quadrifocal tensor geometry among four views. The structure of the binocular vision-IMU tightly-coupled system is in the form of an error state, and the time updates of the state and the state covariance are directly estimated without using Unscented Transformation (UT). Experiments are carried out based on an outdoor land vehicle open source dataset and an indoor Micro Aerial Vehicle (MAV) open source dataset. Results clearly show the effectiveness of the proposed new mechanisation.

Monocular Visual Odometry Based on Trifocal Tensor Constraint

For the problem of real-time precise localization in the urban street, a monocular visual odometry based on Extend Kalman fusion of optical-flow tracking and trifocal tensor constraint is proposed. To diminish the influence of moving object, such as pedestrian, we estimate the motion of the camera by extracting the features on the ground, which improves the robustness of the system. The observation equation based on trifocal tensor constraint is derived, which can form the Kalman filter alone with the state transition equation. An Extend Kalman filter is employed to cope with the nonlinear system. Experimental results demonstrate that, compares with Yu’s 2-step EKF method, the algorithm is more accurate which meets the needs of real-time accurate localization in cities.

Rapid and robust initialization for monocular visual inertial navigation within multi-state Kalman filter

Sensor-fusion based navigation attracts significant attentions for its robustness and accuracy in various applications. To achieve a versatile and efficient state estimation both indoor and outdoor, this paper presents an improved monocular visual inertial navigation architecture within the Multi-State Constraint Kalman Filter (MSCKF). In addition, to alleviate the initialization demands by appending enough stable poses in MSCKF, a rapid and robust Initialization MSCKF (I-MSCKF) navigation method is proposed in the paper. Based on the trifocal tensor and sigma-point filter, the initialization of the integrated navigation can be accomplished within three consecutive visual frames. Thus, the proposed I-MSCKF method can improve the navigation performance when suffered from shocks at the initial stage. Moreover, the sigma-point filter is applied at initial stage to improve the accuracy for state estimation. The state vector generated at initial stage from the proposed method is consistent with MSCKF, and thus a seamless transition can be achieved between the initialization and the subsequent navigation in I-MSCKF. Finally, the experimental results show that the proposed I-MSCKF method can improve the robustness and accuracy for monocular visual inertial navigations.

Vision-aided localization and navigation based on trifocal tensor geometry

In this paper, a new method for vision-aided navigation based on trifocal tensor geometry is presented. The main goal of the proposed method is to estimate the position of vehicles in global positioning system-denied environments, using a standard inertial navigation system and only a single camera. The geometric trifocal tensor relationship between three images is used as measurement information from the camera, and the primary contribution of this work is the derivation of a measurement model that is able to express the geometric constraints of the trifocal tensor in the global frame. This measurement model does not require including the three-dimensional feature positions in the state vector. In other words, the proposed method does not entail reconstructing the environment. Rather, the method only considers the vehicle state. The vision-aided inertial navigation algorithm that we propose has computational complexity only with regard to the number of features at the current time, and the algorithm is capable of estimating the pose in real environments. Experiments were conducted to show the effectiveness of the proposed method in simulations and real environments.

On Some Properties of Calibrated Trifocal Tensors

In two-view geometry, the essential matrix describes the relative position and orientation of two calibrated images. In three views, a similar role is assigned to the calibrated trifocal tensor. It is a particular case of the (uncalibrated) trifocal tensor and thus it inherits all its properties but, due to the smaller degrees of freedom, satisfies a number of additional algebraic constraints. Some of them are described in this paper. More specifically, we define a new notion --- the trifocal essential matrix. On the one hand, it is a generalization of the ordinary (bifocal) essential matrix, and, on the other hand, it is closely related to the calibrated trifocal tensor. We prove the two necessary and sufficient conditions that characterize the set of trifocal essential matrices. Based on these characterizations, we propose three necessary conditions on a calibrated trifocal tensor. They have a form of 15 quartic and 99 quintic polynomial equations. We show that in the practically significant real case the 15 quartic constraints are also sufficient.