Camera Inertial Measurement Unit Research Articles

Real-Time Digital Video Stabilization of Bioinspired Robotic Fish Using Estimation-and-Prediction Framework

The rhythmic movement of bioinspired robotic fish brings about undesirable visual jitter. Note that the unstable camera path of this kind of robot is characterized by obvious regularity and predictability, of which traditional stabilization methods have not made full advantage. This article proposes a novel estimation-and-prediction framework for real-time digital video stabilization of bioinspired robotic fish. First, based on the attitude information of an inertial measurement unit (IMU), a camera–IMU model is established, where the homography transformation with eight degrees of freedom (DOFs) is reduced to translation transformation with two DOFs. Second, traditional optical flow and gray projection methods as well as a novel translation estimation network are employed to estimate the translations between consecutive frames. Third, a lightweight long short-term memory (LSTM) network is constructed, allowing remarkable prediction and smoothing of the camera path. Finally, aquatic experiments under various scenarios are conducted on a manta-inspired robot, demonstrating the effectiveness of the proposed method. Specifically, compared with the state-of-the-art commercial offline stabilization software, our online stabilization algorithm achieves approximate visual stability and remarkably faster stabilization speed. The obtained results shed light on visual sensing and control applications of bioinspired underwater vehicles.

Observability Analysis and Keyframe-Based Filtering for Visual Inertial Odometry With Full Self-Calibration

Camera–inertial measurement unit (IMU) sensor fusion has been extensively studied in recent decades. Numerous observability analysis and fusion schemes for motion estimation with self-calibration have been presented. However, it has been uncertain whether the intrinsic parameters of both the camera and the IMU are observable under general motion. To answer this question, by using the Lie derivatives, we first prove that for a rolling shutter (RS) camera–IMU system, all the intrinsic and extrinsic parameters, camera time offset, and readout time of the RS camera are observable with an unknown landmark. To our knowledge, we are the first to present such a proof. Next, to validate this analysis and to solve the drift issue of a structureless filter during standstills, we develop a keyframe-based sliding window filter (KSWF) for odometry and self-calibration, which works with a monocular RS camera or stereo RS cameras. Though the keyframe concept is widely used in vision-based sensor fusion, to our knowledge, the KSWF is the first of its kind to support self-calibration. Our simulation and real data tests have validated that it is possible to fully calibrate the camera–IMU system using the observations of opportunistic landmarks under diverse motion. Real data tests confirmed previous allusions that keeping landmarks in the state vector can remedy the drift in standstill and showed that the keyframe-based scheme is an alternative solution.

Rotation alignment of a camera-IMU system using a single affine correspondence.

We propose an accurate and easy-to-implement method on rotation alignment of a camera-inertial measurement unit (IMU) system using only a single affine correspondence in the minimal case. The known initial rotation angles between the camera and IMU are utilized; thus, the alignment model can be formulated as a polynomial equation system based on homography constraints by expressing the rotation matrix in a first-order approximation. By solving the equation system, we can recover the rotation alignment parameters. Furthermore, more accurate alignment results can be achieved with the joint optimization of multiple stereo image pairs. The proposed method does not require additional auxiliary equipment or a camera's particular motion. The experimental results on synthetic data and two real-world data sets demonstrate that our method is efficient and precise for the camera-IMU system's rotation alignment.

Visual–inertial state estimation with camera and camera–IMU calibration

In the last two decades, the Visual–Inertial Odometry (VIO) has recently received much attention for efficient and accurate ego-motion estimation of unmanned vehicle systems. In particular, the VIO includes only an Inertial Measurement Unit (IMU) and a camera. In this paper, we present a novel calibration approach for accurate deployment of monocular VIO. For this purpose, the hybrid optimization algorithm is used for calibrating the camera intrinsic and camera–IMU extrinsic calibration, automatically and without knowing the mechanical configuration. It is a professional work to carefully calibrate the intrinsic and extrinsic parameters, and it is required to repeat this work when the mechanical configuration of the sensor suite changes. Quantitative comparisons our method with the offline conventional calibration method on the KITTI dataset verify the efficacy and accuracy of the proposed method. We also demonstrate the performance of the proposed approach in large scale outdoor experiments.

Monocular Visual–Inertial State Estimation With Online Initialization and Camera–IMU Extrinsic Calibration

There have been increasing demands for developing microaerial vehicles with vision-based autonomy for search and rescue missions in complex environments. In particular, the monocular visual–inertial system (VINS), which consists of only an inertial measurement unit (IMU) and a camera, forms a great lightweight sensor suite due to its low weight and small footprint. In this paper, we address two challenges for rapid deployment of monocular VINS: 1) the initialization problem and 2) the calibration problem. We propose a methodology that is able to initialize velocity, gravity, visual scale, and camera–IMU extrinsic calibration on the fly. Our approach operates in natural environments and does not use any artificial markers. It also does not require any prior knowledge about the mechanical configuration of the system. It is a significant step toward plug-and-play and highly customizable visual navigation for mobile robots. We show through online experiments that our method leads to accurate calibration of camera–IMU transformation, with errors less than 0.02 m in translation and 1° in rotation. We compare out method with a state-of-the-art marker-based offline calibration method and show superior results. We also demonstrate the performance of the proposed approach in large-scale indoor and outdoor experiments.

Visual-Inertial Sensor Fusion: Localization, Mapping and Sensor-to-Sensor Self-calibration

Visual and inertial sensors, in combination, are able to provide accurate motion estimates and are well suited for use in many robot navigation tasks. However, correct data fusion, and hence overall performance, depends on careful calibration of the rigid body transform between the sensors. Obtaining this calibration information is typically difficult and time-consuming, and normally requires additional equipment. In this paper we describe an algorithm, based on the unscented Kalman filter, for self-calibration of the transform between a camera and an inertial measurement unit (IMU). Our formulation rests on a differential geometric analysis of the observability of the camera—IMU system; this analysis shows that the sensor-to-sensor transform, the IMU gyroscope and accelerometer biases, the local gravity vector, and the metric scene structure can be recovered from camera and IMU measurements alone. While calibrating the transform we simultaneously localize the IMU and build a map of the surroundings, all without additional hardware or prior knowledge about the environment in which a robot is operating. We present results from simulation studies and from experiments with a monocular camera and a low-cost IMU, which demonstrate accurate estimation of both the calibration parameters and the local scene structure.