Epipolar Lines Research Articles

Registration of Terrestrial LiDAR and Panoramic Imagery Using the Spherical Epipolar Line and Spherical Absolute Orientation Model

Registration between terrestrial LiDAR and optical imagery plays a crucial role in information fusion. However, it is difficult to find reliable correlations among the different feature information of optical imagery and LiDAR point clouds. Therefore, in order to achieve high-precision registration of heterogeneous sensors, a method based on spherical epipolar line and spherical absolute orientation is proposed in this paper. The method firstly projects the LiDAR point clouds into spherical images based on the spherical imaging model and derives the spherical epipolar line equation. Then the relative and absolute orientations of the spherical LiDAR images and the optical images are performed based on manually selected control points. Finally, based on Harris corner extraction, combined with the geometric constraints of the spherical epipolar line and absolute orientation, dense matching between optical and LIDAR images are achieved, and all matching points are used as control points for registration to improve the accuracy of manually selected points registration. Multiple sets of test data are acquired outdoors using a FARO Focus S laser scanner, a Z + F IMAGER 5010C laser scanner, and a Ladybug5+ panoramic camera. The experimental results show that the method in this paper is practical and improves the accuracy of manual points selection registration, and the degree of improvement is related to the number of successfully matched corner points.

A linear pushbroom satellite image epipolar resampling method for digital surface model generation

Epipolar resampling, also known as stereo rectification, is a prerequisite for optical satellite image stereo matching in digital surface model (DSM) generation. In epipolar resampling, image pairs are warped and thus can be matched pixel-to-pixel along the epipolar line. Different from the pinhole imaging model of frame cameras, the multi-center projection model of linear pushbroom sensors does not obey the epipolar geometric constraint, making it impossible to conjugate epipolar lines from stereo images. The epipolar resampling methods can resolve some of the inherent epipolar error, but not completely. The topographic relief brings high disparity values between epipolar image pairs, posing a challenge to the subsequent stereo matching to recover disparity map. In this paper, we introduce a novel epipolar resampling method for satellite imagery based on ortho rectification. This method takes full advantage of a rational function model to establish a connection between image space and object space coordinates, and thereby transform the two-dimensional disparity in ortho rectification products into one dimension. We propose a coarse-to-fine DSM generation pipeline to overcome the high-disparity-problem without additional time costs or precision decrement. We conducted experiments on various satellite stereo images, and validated both the extreme suppression on epipolar error as well as the feasibility of the proposed coarse-to-fine DSM generation pipeline. The experimental results show that the generated epipolar image pairs from our proposed method effectively solved the high disparity problem. Furthermore, the novel DSM generation method obtained an internal error below one pixel for a rectified stereo image pair in mountain area, and the lowest vertical error in the IARPA MVS dataset when compared with the common open-source programs and commercial software.

EPIPOLAR LINE-BASED LATERAL VIBRATION MEASUREMENT BY USING TWO CAMERAS

Abstract. Vibration measurement techniques can be categorized into contact-type and non-contact-type techniques. These types of techniques can add mass-loading to a lightweight structure resulting in the negative performance of a structure, because sensors, high contrast speckles or targets should be mounted on a structure. Moreover, non-contact-type vibration measurement techniques have only been tested to detect vibrations using a single camera. As the vibrations occurring at the opposite sides of a rotating structure in a region of interest (ROI) can be different from each other. For 3-dimensional (3D) vibration measurement, the same position in videos acquired from two cameras should be used. Because the videos acquired by two cameras placed perpendicular to the structure can be used to detect the vibrations in the x-direction as well as y-direction. In this study, an epipolar line-based corresponding point selection on a rotating cylindrical structure was performed, to extract the same ROIs from videos recorded by two cameras. A fundamental matrix was constructed by using the targets attached on the structure and in the background. The coordinates of the mid-pixel of the ROI in a video acquired by one camera was used to determine the epipolar line for the same ROI in the video acquired by another camera. Then an edge-based vibration measurement technique was applied to measure the vibration in the extracted ROIs. The results were used to reconstruct a 3D vibration signal. The 3D vibration measurement results can be used to effectively recognize the deformations resulting in the negative performance of a structure.

EPIPOLAR GEOMETRY BETWEEN PHOTOGRAMMETRY AND COMPUTER VISION – A COMPUTATIONAL GUIDE

Abstract. Stereo image orientation is one of the major topics in computer vision, photogrammetry, and robotics. The stereo vision problem solution represents the basic element of the multi-view Structure from Motion SfM in computer vision and photogrammetry.A successfully reconstructed stereo image geometry is based on solving the epipolar constraint using the fundamental matrix which is based on the projective geometry in computer vision. However, in photogrammetry, the problem is well known as relative orientation and there is a different solution that is based on the euclidean geometry using collinearity or coplanarity equations.A lot of literature and discussions were found in the last decades to solve the epipolar geometry problem. However, there is still no clear description to compare between solutions introduced using both projective and euclidean solutions and which method of the relative image orientation is mostly preferred.To the best of our knowledge, computing and plotting the epipolar lines using photogrammetric collinearity and coplanarity equations is not shown before in the educational litrature. In this paper, a detailed mathematical solution of the epipolar geometry will be shown using both photogrammetric and computer vision techniques. This is aimed to remove any confusion for new learners in using the current methods in both scientific fields and show that using any technique should lead to comparable results with advantages and disadvantages.

Stereoscopic deflectometry with a curved screen.

Deflectometry has been widely used in topography measurement of specular surface. In deflectometry with a curved screen, the range of the gradient and height field of the measured specular surface can be effectively expanded compared to deflectometry with a plane screen. As stereo deflectometry measures gradient with high accuracy, the specular surface is reconstructed by integrating the gradient. In this paper, a stereo deflectometry with a curved screen is proposed in the aspect of system calibration and measuring principle. A pair of cameras and deflectometric system are calibrated simultaneously to obtain the camera parameters and relationship between two cameras and the curved screen. Then, pairs of rectified fringe patterns are demodulated to obtain two pairs of rectified absolute phase maps and homologous points are searched along epipolar line with the principle of the same normal direction on specular surface. Finally, simulated and actual experiments are carried out and the results show high accuracy and stability.

A marker-less monocular vision point positioning method for industrial manual operation environments

Vision-assisted technologies in industrial manual operation such as augmented reality (AR) are increasingly popular. They require high positioning accuracy and robustness to operate normally. However, narrow spaces, moving hands, or tools may occlude or obscure local visual features of operation environments and thus affect the positioning accuracy and robustness of operating position. The resultant misguidance may even cause misoperation of operators. This paper proposes a marker-less monocular vision point positioning method for vision-assisted manual operation in industrial environments. The proposed method can accurately and robustly locate the target point of operation using constraint minimization method even if the target area has no corresponding visual features in the case of occlusion and improper illumination. The proposed method has three phases: intersection generation, intersection optimization, and target point solving. In the intersection generation phase, a certain number of intersections of epipolar lines are generated as candidate target points using fundamental matrices. Here, the solving constraint is converted from point-to-line to point-to-points. In the intersection optimization phase, the intersections are optimized to two different sets through the iterative linear fitting and geometric mean absolute error methods. Here, the solving constraint is further converted from point-to-points to point-to-point sets. In the target point solving phase, the target point is solved as a constrained minimization problem based on the distribution constraint of the two intersection sets. Here, the solving constraint is finally converted from point-to-point sets to point-to-point, and the unique optimal solution is obtained as the target point. The experimental results show that this method has a better accuracy and robustness than the traditional homography matrix method for the practical industrial operation scenes.

TIDE: Temporally Incremental Disparity Estimation via Pattern Flow in Structured Light System

We introduced Temporally Incremental Disparity Estimation Network (TIDE-Net), a learning-based technique for disparity computation in mono-camera structured light systems. In our hardware setting, a static pattern is projected onto a dynamic scene and captured by a monocular camera. Different from most former disparity estimation methods that operate in a frame-wise manner, our network acquires disparity maps in a temporally incremental way. Specifically, We exploit the deformation of projected patterns (named <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">pattern flow</i> ) on captured image sequences, to model the temporal information. Notably, this newly proposed pattern flow formulation reflects the disparity changes along the epipolar line, which is a special form of optical flow. Tailored for pattern flow, the TIDE-Net, a recurrent architecture, is proposed and implemented. For each incoming frame, our model fuses correlation volumes (from current frame) and disparity (from former frame) warped by pattern flow. From fused features, the final stage of TIDE-Net estimates the residual disparity rather than the full disparity, as conducted by many previous methods. Interestingly, this design brings clear empirical advantages in terms of efficiency and generalization ability. Using only synthetic data for training, our extensitve evaluation results (w.r.t. both accuracy and efficienty metrics) show superior performance than several SOTA models on unseen real data. The code will be available on <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><uri>https://github.com/CodePointer/TIDENet</uri></i> soon.

Image stitching by disparity-guided multi-plane alignment

Image stitching aims to warp and align two or more images with overlapping areas to generate a panorama with a larger field of view. Due to the wide baseline or the abrupt depth changes caused by the foreground objects, the phenomenon that adjacent pixels or regions in one image are not adjacent in another image happens. It is difficult to avoid severe parallax artifacts and get good alignment results when stitching such images. In this paper, focusing on the images with large parallax, we design a multi-plane alignment algorithm guided by the disparity map. The concept of average tolerable parallax is proposed to help distinguish one background plane and multiple foreground objects from the image, which are robust to small parallax. In order to obtain the reliable disparity map of the image pairs got from any camera with unknown camera parameters, we propose an optimal homography estimation method based on the relative projection biases. This helps to satisfy the common epipolar line constraint when applying stereo matching modules. Experimental results demonstrate that our algorithm provides accurate stitching results on images with large parallax, and outperforms other existing state-of-the-art methods both qualitatively and quantitatively.

Fast and Reliable Vision-Based Navigation for Real Time Kinematic Applications

Abstract: Automatic Image Matching (AIM) is the term used to identify the automatic detection of corresponding points located on the overlapping areas of multiple images. AIM is extensively used with Mobile Mapping System (MMS) for different engineering applications, such as highway infrastructure mapping, monitoring of road surface quality and markings, telecommunication, emergency response, and collecting data for Geographical Information Systems (GIS). Robotics community and Simultaneous Localization And Mapping (SLAM) based applications are other important areas that require fact and welldistributed AIM for robust vision navigation solutions. Different robust feature detection methods are commonly used for AIM, such as Scale Invariant Feature Transform (SIFT), Principal Component Analysis (PCA)–SIFT and Speeded Up Robust Features (SURF). The performance of such techniques have been widely investigated and compared, showing high capability to provide reliable and precise results. However, these techniques are still limited to be used for real and nearly real time SLAM based applications, such as intelligent Robots and low-cost Unmanned Aircraft Vehicles (UAV) based on vision navigation. The main limitations of these AIM techniques are represented in the relatively long processing time and the random distribution of matched points over the common area between images. This paper works on overcoming these two limitations, providing extremely fast AIM with well- distributed common points for robust real time vision navigation. Digital image pyramid, Epipolar line and 2D transformation have been utilized for limiting the size of search windows significantly and determining the rotating angle and scale level of features, reducing the overall processing time considerably. Using limited number of well-distributed common points has also helped to speed up the automatic matching besides providing robust vision navigation solution. The idea has been tested with terrestrial MMS images, and surveying UAV aerial images. The results reflect the high capability of the followed technique in providing fast and robust AIM for real-time SLAM based applications. Keywords: Automatic Image Matching, Epipolar Line, Image Pyramid, SLAM, Vision Navigation, Real Time, Vision Navigation.

Active Stereo Vision System with Rotated Structured Light Patterns and Two-Step Denoising Process for Improved Spatial Resolution

Three-dimensional (3D) scanning systems are widely used nowadays for multiple applications. If the object is too large, it is a challenge to provide the detailed characteristics of the scanned object. Improving the spatial resolution of the scanning process is essential in improving the quality of the reconstruction results. Accordingly, this study has proposed an active stereo vision system for improving the spatial resolution. It is well known that the reconstruction quality of stereo vision systems is severely impaired for objects with limited features on their surfaces, thereby, in the present study, an active stereo vision system is proposed in which stripe patterns are projected onto the object surface with various rotation angles in order to facilitate a more accurate and complete detection of the matching points. Notably, the point clouds obtained by the projected patterns at each rotation angle are formed within the same world coordinate system. As a result, the final reconstructed 3D object can be obtained by merging sets of point clouds directly without the need for a prior registration process. In practice, the quality of the reconstruction results obtained by stereo vision systems is commonly degraded by the effects of noise. Therefore, this study also proposes a two-step (multi-decoding pattern and Epipolar line) method for removing the incorrect matching points and improving the reconstruction results accordingly. The performance of the proposed reconstruction system is evaluated experimentally and compared with that of a general active stereo vision system. The experimental results show that the active stereo vision system presented in the present study provides a low-cost, convenient and effective approach for performing the reconstruction of 3D objects with high spatial resolution.

Spatially Consistent Transformer for Colorization in Monochrome-Color Dual-Lens System.

We study the colorization problem in monochrome-color dual-lens camera systems, i.e. colorizing the gray image from the monochrome camera using the color image from the color camera as reference. In related methods, cost volume based CNN methods achieve the state-of-the-art results, but they are costly in GPU memory due to building the 4D cost volume. Recently, some slice-wise cross-attention based methods are proposed for related problems. The slice-wise cross-attention has much less costs in GPU memory but directly using them for this colorization problem cannot generate competing results. We make use of the non-local computation property of cross-attention to propose a transformer based method. To overcome the limitations of straight-forward slice-wise cross-attention, we propose the spatially consistent cross-attention (SCCA) block to encourage pixels of slices across different epipolar lines in the gray image to find spatially consistent correspondence with pixels of the reference color image. And, to further reduce the memory cost while keeping the colorization accuracy, we design a pyramid processing strategy to cascade a series of SCCA blocks with smaller slice size and perform the colorization from coarse to fine. To extract more powerful image features, we use several regional self-attention (RSA) blocks with U-style connections. Experimental results show that we outperform the state-of-the-art methods largely on the synthesized datasets of Cityscapes, Sintel, and SceneFlow, and the real monochrome-color dual-lens dataset.

A Robust Two-Stage Registration Algorithm for Large Optical and SAR Images

Accurate optical and synthetic aperture radar (SAR) image registration is crucial to multisensor remote sensing applications. Though several algorithms have been proposed, the practical implementation that directly matches large and high-resolution optical and SAR images remains underexplored by the community. Equipped with the satellite positioning parameters, optical and SAR images can be roughly registered based on geographic coordinates. However, the relative positioning accuracy is still dozens, even hundreds of pixels due to the inaccuracies of sensor parameters and elevation. Consequently, we propose a robust registration algorithm, which consists of two stages, where the horizontal positioning errors can be reduced in the first stage, and then, we fine-tune the correspondences in the second stage. Specifically, we propose a novel template matching method based on the dilated convolutional feature (DCF) and epipolar-oriented phase correlation. DCF is constructed by a depthwise-separable dilated convolution with multichannel gradients, which are generated by the Sobel operator for optical images and a ratio of exponentially weighted averages (ROEWA) operator for SAR images. Due to the large reception field of dilated convolution, DCF can retain invariance even for large relative positioning errors. The epipolar-oriented rectangle template, which stretches along the epipolar line, is then proposed to capture more overlapping areas compared to square templates. Furthermore, the outlier removal is implemented in the coordinate system of optical images to avoid the effect of range compression in SAR images. Inliers are finally used to refine the rational polynomial coefficients (RPCs) based on the bundle adjustment technique. Experimental results on high-resolution optical and SAR image products of various scenarios demonstrate the effectiveness of the proposed registration framework. The relative positioning errors of the refined RPCs can be reduced from hundreds of pixels to the subpixel level.

Ego-Motion Estimation With Stereo Cameras Using Efficient 3D–2D Edge Correspondences

This article presents a new edge-based ego-motion estimation method for stereo cameras using 3D–2D edge alignment. To filter out prominent edgelets from Canny edge extractor, a region growing algorithm is developed to remove cluttered edges out of raw edgelets. Edge matching between stereo image pairs is guided by sparse feature correspondence using optical flow. The disparity is then refined using an inverse compositional algorithm, which is far more computationally efficient than searching along the epipolar line. The relative camera motion is solved by 3D–2D edge alignment based on an iterative closest point (ICP) algorithm, minimizing the point-to-tangent distance. A gradient-orientation-based similarity measurement is proposed, and the optimal correspondence is searched between the 3-D edge point projected onto the current image and the extracted edge pixels in the current frame with 1-nearest neighbor. In addition, the inverse depth of edge points in the reference frame is optimized. Experiments show that the proposed method can provide centimeter-level relative pose error (RPE), which has a better performance than the point-based approach efficient perspective-n-point (EPnP) and the edge-based approach direct edge-alignment (DE-A).

Spatial-Angular Versatile Convolution for Light Field Reconstruction

Spatial-angular separable convolution (SAS-conv) has been widely used for efficient and effective 4D light field (LF) feature embedding in different tasks, which mimics a 4D convolution by alternatively operating on 2D spatial slices and 2D angular slices. In this paper, we argue that, despite its global intensity modeling capabilities, SAS-conv can only embed local geometry information into the features, resulting in inferior performances in the regions with textures and occlusions. Because the epipolar lines are highly related to the scene depth, we introduce the concept of spatial-angular correlated convolution (SAC-conv). By alternating 2D convolutions on the vertical and horizontal epipolar slices, SAC-conv can embed global and robust geometry information into the features. We verify that SAS-conv and SAC-conv are skilled at different aspects of 4D LF feature embedding through a detailed feature and error analysis. Based on their complementarity, we further combine SAS-conv and SAC-conv by a parallel residual connection, forming a new spatial-angular versatile convolution (SAV-conv) module. We conduct comprehensive experiments on two representative LF reconstruction tasks, i.e., LF angular super-resolution and LF spatial super-resolution. Both the quantitative and qualitative results demonstrate that, without any extra parameters, networks upgraded with our proposed SAV-conv notably outperform those upgraded with SAS-conv and achieve a new state-of-the-art performance. Codes are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Joechann0831/SAV_conv</uri> .

Cross View Capture for Stereo Image Super-Resolution

Stereo image super-resolution exploits additional features from cross view image pairs for high resolution (HR) image reconstruction. Recently, several new methods have been proposed to investigate cross view features along epipolar lines to enhance the visual perception of recovered HR images. Despite the impressive performance of these methods, global contextual features from cross view images are left unexplored. In this paper, we propose a cross view capture network (CVCnet) for stereo image super-resolution by using both global contextual and local features extracted from both views. Specifically, we design a cross view block to capture diverse feature embeddings from the views in stereo vision. In addition, a cascaded spatial perception module is proposed to redistribute each location in feature maps according to the weight it occupies to make the extraction of features more effective. Extensive experiments demonstrate that our proposed CVCnet outperforms the state-of-the-art image super-resolution methods to achieve the best performance for stereo image super-resolution tasks. The source code is available at https://github.com/xyzhu1/CVCnet.

Large-curvature specular surface phase measuring deflectometry with a curved screen

Deflectometry has been widely used in topography measurement of the specular surface due to the merits of full-field data acquisition, automatic data processing, high accuracy, and large dynamic range. Existing deflectometry-based methods, such as model phase measuring deflectometry, direct phase measuring deflectometry, and stereo deflectometry, employ a plane screen, so they have a small field view. The characteristics of curved screens can effectively expand the range of the gradient and height field of the measured specular surface. This paper presents a novel deflectometric technique by using a curved screen to measure large-curvature specular surface. In order to obtain a model of the curved screen, a stereo vision method has been proposed to capture fringes displayed on the curved screen. After phase correction along epipolar lines and homologous points matching, the point cloud containing location and phase is obtained. A virtual plane fixed on the screen is assumed to gain posture and position of the screen. The model of the curved screen is transferred to the deflectometric system through the virtual plane. After calibrating the deflectometric system, height of specular objects can be obtained. Simulated and actual experiments on measuring specular objects have been carried out and the results show high accuracy and large field view for large-curvature specular surface of the proposed technique.

Feature-extraction and motion-measurement method for noncooperative space targets

<p indent="0mm">Space targets served on the orbit have a noncooperative characteristic. Thus, it is difficult to extract their features and measure their motions. To resolve this problem, a feature-extraction and motion-measurement method for noncooperative space targets is proposed in this paper. First, threshold segmentation is performed with the images captured using the binocular camera to separate the satellite-rocket docking ring section from the surrounding background. Additionally, the Canny operator method is used to detect the edge of the satellite-rocket docking ring. Based on the analysis of the characteristic of the edge’s shape, Hough transformation is employed to extract the line feature of the cross. The contour of the satellite-rocket docking ring is fitted using the least square method. Furthermore, the parallel equivalent projection and the epipolar line alignment methods are used to match the images of the left and right cameras in the convergence binocular vision. Finally, the relative position of the satellite-rocket docking ring is obtained through a three-dimensional reconstruction of the feature points on the satellite-rocket docking ring and cross. Additionally, the dual-vector pose-determination method is employed to obtain the relative pose of the satellite-rocket docking ring. The ground simulation test shows that this algorithm can measure the relative position and pose of the satellite-rocket docking ring effectively. At a distance of <sc>1.5 m</sc> from the target, the precision of the relative position is <sc>15 mm,</sc> and that of the relative pose is 1.12°. This research can achieve an accurate measurement of the relative position and pose of the noncooperative target’s docking ring, providing the measurement basis and important support for on-orbit service missions, such as the grasping, repairing, refueling, and module replacement of noncooperative space targets; thus, it has important practical application value.

Reidentification-Based Automated Matching for 3D Localization of Workers in Construction Sites

The location information of entities in construction sites, such as workers and construction machines, is valuable in project management and safety. Therefore, as nonintrusive and accurate solutions, various vision-based methods have been proposed to track entities in construction sites and obtain their three-dimensional (3D) coordinates. However, most existing vision-based methods realize 3D localizations by basing entity matching on the epipolar line, which brings instability in entity matching due to the calculation error of the epipolar line or failure to match entities when multiple entities are located on the same epipolar. To solve this problem, a novel framework based on reidentification is proposed to automatically match workers across two camera views, thereby obtaining their 3D coordinates in construction sites. In this framework, deep-learning-based computer vision algorithms are firstly used to detect and track workers in two camera views. Then, the reidentification (ReID) algorithm is applied to utilize tracked workers’ visual features to match the workers across both two camera views and different frames. As a result, for every matched pair, the worker’s pixel locations in two camera views can be obtained to calculate the 3D coordinates through triangulation. The implementation of videos recorded from a real construction project proves the feasibility and accuracy of this framework. Specifically, through utilizing the ReID algorithm to match workers, the framework achieves competitive results on workers matching with precision, recall, and accuracy of more than 99%, 93%, and 93%. Furthermore, it also effectively addresses the practical problems of ID repetition and ID switching. Meanwhile, this paper extends the application scenarios of reidentification algorithms in construction sites, thereby contributing to the future application of multiple-camera vision-based methods in construction sites.

The importance of three dimensional coronary artery reconstruction accuracy when computing virtual fractional flow reserve from invasive angiography

Three dimensional (3D) coronary anatomy, reconstructed from coronary angiography (CA), is now being used as the basis to compute ‘virtual’ fractional flow reserve (vFFR), and thereby guide treatment decisions in patients with coronary artery disease (CAD). Reconstruction accuracy is therefore important. Yet the methods required remain poorly validated. Furthermore, the magnitude of vFFR error arising from reconstruction is unkown. We aimed to validate a method for 3D CA reconstruction and determine the effect this had upon the accuracy of vFFR. Clinically realistic coronary phantom models were created comprosing seven standard stenoses in aluminium and 15 patient-based 3D-printed, imaged with CA, three times, according to standard clinical protocols, yielding 66 datasets. Each was reconstructed using epipolar line projection and intersection. All reconstructions were compared against the real phantom models in terms of minimal lumen diameter, centreline and surface similarity. 3D-printed reconstructions (n = 45) and the reference files from which they were printed underwent vFFR computation, and the results were compared. The average error in reconstructing minimum lumen diameter (MLD) was 0.05 (± 0.03 mm) which was < 1% (95% CI 0.13–1.61%) compared with caliper measurement. Overall surface similarity was excellent (Hausdorff distance 0.65 mm). Errors in 3D CA reconstruction accounted for an error in vFFR of ± 0.06 (Bland Altman 95% limits of agreement). Errors arising from the epipolar line projection method used to reconstruct 3D coronary anatomy from CA are small but contribute to clinically relevant errors when used to compute vFFR.

Moving Object Detection with Single Moving Camera and IMU Sensor using Mask R-CNN Instance Image Segmentation

This paper describes a new method for the moving object detection using the IMU sensor and instance image segmentation. In the proposed method, the feature points are extracted by the detector, and the initial fundamental matrix is calculated from the IMU data. Next, the epipolar line is used to classify the extracted feature points. From the background feature point matching, fundamental matrix is calculated iteratively to minimize the error of classification. After the feature point classification, image segmentation is used to enhance the quality of the classification result. The proposed method is implemented and tested with real-world driving videos, and compared with the previous works.