Motion Estimation System Research Articles

Self-Supervised Learning of Perceptually Optimized Block Motion Estimates for Video Compression.

Block based motion estimation is integral to inter prediction processes performed in hybrid video codecs. Prevalent block matching based methods that are used to compute block motion vectors (MVs) rely on computationally intensive search procedures. They also suffer from the aperture problem, which tends to worsen as the block size is reduced. Moreover, the block matching criteria used in typical codecs do not account for the resulting levels of perceptual quality of the motion compensated pictures that are created upon decoding. Towards achieving the elusive goal of perceptually optimized motion estimation, we propose a search-free block motion estimation framework using a multi-stage convolutional neural network, which is able to conduct motion estimation on multiple block sizes simultaneously, using a triplet of frames as input. This composite block translation network (CBT-Net) is trained in a self-supervised manner on a large database that we created from publicly available uncompressed video content. We deploy the multi-scale structural similarity (MS-SSIM) loss function to optimize the perceptual quality of the motion compensated predicted frames. Our experimental results highlight the computational efficiency of our proposed model relative to conventional block matching based motion estimation algorithms, for comparable prediction errors. Further, when used to perform inter prediction in AV1, the MV predictions of the perceptually optimized model result in average Bjontegaard-delta rate (BD-rate) improvements of -1.73% and -1.31% with respect to the MS-SSIM and Video Multi-Method Assessment Fusion (VMAF) quality metrics, respectively, as compared to the block matching based motion estimation system employed in the SVT-AV1 encoder.

Real-time siamese multiple object tracker with enhanced proposals

Maintaining the identity of multiple objects in real-time video is a challenging task, as it is not always feasible to run a detector on every frame. Thus, motion estimation systems are often employed, which either do not scale well with the number of targets or produce features with limited semantic information. To solve the aforementioned problems and allow the tracking of dozens of arbitrary objects in real-time, we propose SiamMOTION. SiamMOTION includes a novel proposal engine that produces quality features through an attention mechanism and a region-of-interest extractor fed by an inertia module and powered by a feature pyramid network. Finally, the extracted tensors enter a comparison head that efficiently matches pairs of exemplars and search areas, generating quality predictions via a pairwise depthwise region proposal network and a multi-object penalization module. SiamMOTION has been validated on five public benchmarks, achieving leading performance against current state-of-the-art trackers. Code available at: https://www.github.com/lorenzovaquero/SiamMOTION

A Novel LiDAR-IMU-Odometer Coupling Framework for Two-Wheeled Inverted Pendulum (TWIP) Robot Localization and Mapping with Nonholonomic Constraint Factors.

This paper proposes a method to solve the problem of localization and mapping of a two-wheeled inverted pendulum (TWIP) robot on approximately flat ground using a Lidar–IMU–Odometer system. When TWIP is in motion, it is constrained by the ground and suffers from motion disturbances caused by rough terrain or motion shaking. Combining the motion characteristics of TWIP, this paper proposes a framework for localization consisting of a Lidar-IMU-Odometer system. This system formulates a factor graph with five types of factors, thereby coupling relative and absolute measurements from different sensors (including ground constraints) into the system. Moreover, we analyze the constraint dimension of each factor according to the motion characteristics of TWIP and propose a new nonholonomic constraint factor for the odometry pre-integration constraint and ground constraint factor in order to add them naturally to the factor graph with the robot state node on SE(3). Meanwhile, we calculate the uncertainty of each constraint. Utilizing such a nonholonomic constraint factor, a complete lidar–IMU–odometry-based motion estimation system for TWIP is developed via smoothing and mapping. Indoor and outdoor experiments show that our method has better accuracy for two-wheeled inverted pendulum robots.

Optimizing a Feature-Based Motion Tracking System for Prospective Head Motion Estimation in MRI and PET/MRI

Patient head motion during magnetic resonance imaging (MRI) of the brain causes artifacts that limit diagnostic value. Prospective motion correction can, in principle, allow artifact-free imaging for any MRI sequence. Optical motion tracking systems are effective at measuring head motion for prospective motion correction but normally rely on attached markers, which are cumbersome and can move relative to the head. In this study, we optimized and benchmarked an optical motion tracking system that uses in-bore stereo-optical cameras to track feature patches on the forehead rather than attached markers. Feature patches were applied to the forehead in the form of an easily removable ink stamp or temporary tattoo. This approach has flexibility in camera positioning and allows tracking of a variety of surfaces on the human face, even those lacking prominent contours. This is advantageous given the limited space and line-of-sight with typical head coils. We optimized the accuracy and efficiency of two key components of the method: 1) the geometry of the stereo camera setup and 2) the feature detection and matching algorithm used for pose estimation. The absolute accuracy of the optimized feature-based method was compared with a commercial marker-based motion tracking system. Estimates of head motion of 15 volunteers simultaneously tracked by the two systems resulted in average positional discrepancies of less than 3 mm using a stamp and submillimeter average discrepancy using a feature-dense tattoo. These results suggest the feasibility of obtaining accurate head motion data for prospective motion compensated MRI using optimized feature-based motion tracking.

Development of a ground test system for motion estimation of decommissioned spacecraft

Development of a ground test system for motion estimation of decommissioned spacecraft

A robust end-to-end deep learning framework for detecting Martian landforms with arbitrary orientations

With increasingly massive amounts of high-resolution images of Mars, automated detection of geological landforms on Mars has received widespread interest. It is significant for acquiring knowledge of distant planetary surfaces and processes, or manifold onboard applications such as spacecraft motion estimation and obstacle avoidance. This is a challenging task, not only because of the multiple sizes of targets and complex image backgrounds, but also the various orientations of some bar-shaped landforms in satellite images captured from the top view. The existing methods for directed landform detection require several pre or post-processing operations to extract possible regions of interest and final detection results with orientation, which are very time consuming. In this paper, a new end-to-end deep learning framework is developed for detecting arbitrarily-directed landforms. This framework, named Rotated-SSD (Single Shot MultiBox Detector, SSD), can locate and identify different landforms on Mars in one pass, by using rotatable anchor-box based mechanism. To enhance its robustness against angle variation of the targets and complex backgrounds, a new efficient match strategy is proposed for anchoring default boxes to ground truth boxes in the model training process and an autoencoder-based unsupervised pre-training operation is introduced to improve both the model training and inference performance. The proposed framework is tested for detection of bar-shaped buttes and dark slope streaks on satellite images. The detection results show that our framework can significantly contribute to onboard motion estimation systems. The comparative results demonstrate that the proposed match strategy outperforms other state-of-the-art match strategies with regard to model training efficiency and prediction accuracy. The pre-training strategy can facilitate the training of deep architectures in case of limited available training data.

Accurate Realtime Motion Estimation Using Optical Flow on an Embedded System

Motion estimation has become one of the most important techniques used in realtime computer vision application. There are several algorithms to estimate object motions. One of the most widespread techniques consists of calculating the apparent velocity field observed between two successive images of the same scene, known as the optical flow. However, the high accuracy of dense optical flow estimation is costly in run time. In this context, we designed an accurate motion estimation system based on the calculation of the optical flow of a moving object using the Lucas–Kanade algorithm. Our approach was applied on a local treatment region implemented into Raspberry Pi 4, with several improvements. The efficiency of our accurate realtime implementation was demonstrated by the experimental results, showing better performance than with the conventional calculation.

Non-rigid image registration using a modified fuzzy feature-based inference system for 3D cardiac motion estimation

Background and objectiveNon-rigid image registration is a well-established method for estimating cardiac motion on 3D echocardiographic images. However, such images have relatively poor spatio-temporal resolution making registration challenging. Some of the main challenges are extracting features relevant to the registration problem and defining a suitable geometrical transformation to be applied. The latter can be tackled using a fuzzy inference system considering its potential in transformation modeling. From this point of view, feature-based image registration can be considered an identification problem in which the transformation parameters are computed through an optimization process. This study, thus, aims to estimate cardiac motion on 3D echocardiographic images based on feature-based non-rigid image registration through sets of modified fuzzy rules. MethodsThe 3D volume features were extracted with the popular scale-invariant feature transform (SIFT) descriptors in 3D space. Sets of fuzzy rules were generated according to the extracted features to register every two consecutive frames. Finally, some supplementary rules modified the registration rule for estimating cardiac motion. ResultsApplying the fuzzy feature-based inference system on the STRAUS synthetic database showed the proposed method to be competitive with other well-established registration algorithms in terms of tracking error and accuracy of strain estimates. The proposed algorithm yielded a tracking error of 1 mm and a relative circumferential strain error of 0.82±4.69%. In addition, the potential of the proposed algorithm for clinical applications was confirmed by evaluating its performance on an in-vivo database called CETUS. ConclusionThis paper proposes a novel registration method based on fuzzy logic which was shown to enable tracking complex cardiac deformations in 3D echocardiographic images with high accuracy.

Translational Motion Estimation Using Kinect

Study on robot navigation has been progressing significantly. However, research of autonomous navigation is still lacking in environments with changing layout. For supporting robot navigation system, rotational and translational motion estimation need to be improved. Low cost sensor such as Kinect can be used as alternative to laser to percept changing in the vicinity of robot. There is still problem in developing motion estimation using Kinect. Narrowness in the sensor field of view limits its ability to track robot motion. This problem emerges specially in translational motion estimation. This paper aims to develop a simple approach based on RGB-D data for translational motion estimation of a Kinect sensor. RGB-D data consist of RGB and Depth images. In previous research, camera was positioned in the direction of longitudinal translational motion. This paper differs from previous works in terms of sensor setting that is placed in the direction of lateral translation movements. Point features are matched between pairs of RGB frame to get interesting pixels. Depth image provides distance information for these pixels in synchronization with feature tracking process. Law of cosines is then applied to these depth pixels to get position translation. Results show that our approach have decreased RMSE to 0.0382 meter better than previous works. In the future research, this translational motion estimation system will be implemented in mobile robot system to support navigation function.

Design and Implementation of Smart Multi-Touch Interface Using Special Purpose CORDIC Processor

This paper presents a smart and embedded multi-touch system for consumer electronics. The infrared (IR) camera is utilized to capture the coordinate of the IR blob from IR stylus or human finger in an emitted IR field. This system allows the consumer to easily convert large surface, projector screen, or conference table into multi-touch interface. The keystone distortion of IR camera is compensated by using homography. To improve the user experience, the calibration of homography is automatically triggered by accelerometer sensor. The special purpose (SP) processor with the parallel COordinate Rotation Digital Computer (CORDIC) architecture computes the observation-to-track association, and the computation result updates the motion status of each tracked trajectory. These motion data support the multi-touch tracking system to evaluate the sliding window of each tracked trajectory for reducing the tracking error. Additionally, the motion data are reused by the adaptive trajectory estimation (TE) system to smooth the touch trajectory and mitigate the measurement noise. This research presents multiple autonomous functions to provide a user-friendly multi-touch interface for consumer electronics. These include the automatic calibration system, the multi-touch tracking architecture with motion estimation system, and the adaptive TE system.

Rao‐Blackwellised particle filtering for low‐cost encoder/INS/GNSS integrated vehicle navigation with wheel slipping

This study presents a Rao-Blackwellised particle filter (RBPF)-based encoder/inertial navigation system (INS)/global navigation satellite system (GNSS) integration method for improving the navigation performance of an autonomous land vehicle (ALV) with wheel slipping. In contrast to traditional integration methods, the proposed integration method introduces an overall wheel slip consideration for the ALV, which greatly improves the accuracy of the velocity estimation, especially when the inertial sensor is low cost. Additionally, the proposed integrated system uses double-difference pseudorange measurements instead of single point positioning results provided by low-cost GNSS receivers, which greatly improves the accuracy of the position estimation. To verify the navigation performance of the proposed integrated system, comparisons between the states estimated by the proposed system, the EKF-based integrated system and the joint wheel-slip and motion-estimation system are provided. The results of the experiment show that the proposed integrated system has the highest accuracy in both the position estimation and the velocity estimation among the three compared systems, and can improve the navigation performance during GNSS signals outages.

Calibration procedures of a vision-based system for relative motion estimation between satellites flying in proximity

Relative Measurements systems represent a key technology for next generation space missions that require proximity operations between satellites. Before on-orbit validation, the sensors and algorithms need to be validated in laboratory employing a good fiducial reference of the relative motion and specific calibration procedures concerning the estimation of roto-translation matrices between different reference frames. This paper presents a set of calibration procedures that allow to assess the accuracy in estimating the relative pose between a Target spacecraft, equipped with a set of square markers, and an Inspector satellite moving in proximity and hosting a monocular camera. An external Motion Capture system is used to track the motion of a set of spherical markers attached to both the Target and the Inspector, providing a reliable fiducial reference for the relative pose between the two spacecraft. The proposed calibration procedures were tested using the SPARTANS hardware facility of the University of Padova.

Energy-Aware Motion and Disparity Estimation System for 3D-HEVC With Run-Time Adaptive Memory Hierarchy

The popularization of multimedia services has pushed forward the development of 2D/3D video-capable embedded mobile devices. Such devices require efficient energy/memory-management strategies to deal with severe memory/processing requirements and limited energy supply. Therefore, we propose a motion and disparity estimation (ME and DE) system—the most memory/processing demanding encoding steps—for the 3D High Efficiency Video Coding (3D-HEVC) standard. It was designed for low energy consumption, featuring a run-time adaptive memory hierarchy. The processing unit employs flexible coding order and optimizations to reduce the computational effort by exploring the inter-channel and inter-view redundancies. The memory hierarchy features window-based prefetching, data reuse, subsampling, and dynamic voltage scaling controlled by our depth-based dynamic search window resizing algorithm. Memory results demonstrate an average on-chip energy reduction of 79% in comparison to the widely used Level-C solution for a 45-nm technology. The proposed energy-aware ME and DE system dissipates 7.55 W while processing three HD 1080p views (video + depth) at 30 frames per second and presents a mean energy consumption of 0.107 J per access unit. To the best of our knowledge, this is the first work that proposes a real-time ME/DE system for the 3D-HEVC standard with an adaptive memory hierarchy.

Video stabilization performance enhancement for low-texture videos

Digital video stabilization (DVS) aims to remove irregular global motion effects from an image sequence. This work aims at developing a real-time video stabilization algorithm for rectifying high-frequency jitter in marine surveillance applications. A DVS system consists of a global motion estimation system and motion correction system. The development of global motion estimation system resistant to failures in low texture videos is the primary goal. Due to the computational advantage and inherent properties, the phase correlation method is adopted as the basic global motion estimation algorithm. The basic algorithm is then modified to adapt to the varying texture content of the video sequences under consideration. An adaptive phase correlation-based global motion estimation is suggested and verified on the videos of varying textures.

An Effective Multi-Cue Positioning System for Agricultural Robotics

The self-localization capability is a crucial component for Unmanned Ground Vehicles (UGV) in farming applications. Approaches based solely on visual cues or on low-cost GPS are easily prone to fail in such scenarios. In this paper, we present a robust and accurate 3D global pose estimation framework, designed to take full advantage of heterogeneous sensory data. By modeling the pose estimation problem as a pose graph optimization, our approach simultaneously mitigates the cumulative drift introduced by motion estimation systems (wheel odometry, visual odometry, ...), and the noise introduced by raw GPS readings. Along with a suitable motion model, our system also integrates two additional types of constraints: (i) a Digital Elevation Model and (ii) a Markov Random Field assumption. We demonstrate how using these additional cues substantially reduces the error along the altitude axis and, moreover, how this benefit spreads to the other components of the state. We report exhaustive experiments combining several sensor setups, showing accuracy improvements ranging from 37% to 76% with respect to the exclusive use of a GPS sensor. We show that our approach provides accurate results even if the GPS unexpectedly changes positioning mode. The code of our system along with the acquired datasets are released with this paper.

Odometry-Vision-Based Ground Vehicle Motion Estimation With SE(2)-Constrained SE(3) Poses

This paper focuses on the motion estimation problem of ground vehicles using odometry and monocular visual sensors. While the keyframe-based batch optimization methods become the mainstream approach in mobile vehicle localization and mapping, the keyframe poses are usually represented by SE(3) in vision-based methods or SE(2) in methods based on range scanners. For a ground vehicle, this paper proposes a new SE(2)-constrained SE(3) parameterization of its poses, which can be easily achieved in the batch optimization framework using specially formulated edges. Utilizing such a parameterization of poses, a complete odometry-vision-based motion estimation system is developed. The system is designed in a commonly used structure of graph optimization, providing high modularity and flexibility for further implementation or adaptation. Its superior performance in terms of accuracy on a ground vehicle platform is validated by real-world experiments in industrial indoor environments.

Reliability and Validity of Attitude and Heading Reference System Motion Estimation in a Novel Mirror Therapy System

In this study, we conducted an experiment for validating a motion estimation system used in novel mirror therapy. The proposed motion estimation system consists of two parts. The first part is motion estimation using attitude and heading reference system sensors. The system attaches a reference frame to a patient to provide flexibility to the patient’s movements. The second part is a compensation algorithm that uses principal component analysis (PCA). Ten subjects performed simple reaching tasks for the validation experiment. The results showed success rate of 89.67% under normal conditions. The PCA-based compensation algorithm increases the success rate from 89.67 to 92.89%. Accurate capturing of healthy arm motion will make effective robot-aided mirror therapy systems feasible.

Register array‐based sum of absolute difference processor with parallel memory system for fast motion estimation

Register array‐based sum of absolute difference processor with parallel memory system for fast motion estimation

An error-based micro-sensor capture system for real-time motion estimation**Research supported by the National Natural Science Foundation of China (Nos. 61431017, 81272166).

A wearable micro-sensor motion capture system with 16 IMUs and an error-compensatory complementary filter algorithm for real-time motion estimation has been developed to acquire accurate 3D orientation and displacement in real life activities. In the proposed filter algorithm, the gyroscope bias error, orientation error and magnetic disturbance error are estimated and compensated, significantly reducing the orientation estimation error due to sensor noise and drift. Displacement estimation, especially for activities such as jumping, has been the challenge in micro-sensor motion capture. An adaptive gait phase detection algorithm has been developed to accommodate accurate displacement estimation in different types of activities. The performance of this system is benchmarked with respect to the results of VICON optical capture system. The experimental results have demonstrated effectiveness of the system in daily activities tracking, with estimation error 0.16 ± 0.06 m for normal walking and 0.13 ± 0.11 m for jumping motions.

The algorithm and VLSI architecture of a high efficient motion estimation with adaptive search range for HEVC systems

This paper presents a novel algorithm and VLSI architecture of motion estimation (ME) for high efficiency video coding systems. The proposed algorithm examines a much smaller set of search candidates and thus greatly reduces the computational complexity. Furthermore, in order to strike a balance between the quality of the Video and the efficiency of the system, this algorithm possesses the advantages that the number of search candidates is adaptive to the characteristic of the Video content. The simulation results show that, compared to the HM reference software, the proposed algorithm leads to a 96% reduction in search candidates with only 1.98% increment in average bitrate. Based on this algorithm, a hardware-efficient VLSI architecture of ME is designed and implemented with 90 nm technology. The experimental results show that, occupying the area complexity of 274.5 kGE, the presented design achieves 60 frames per second with resolution of 3840 × 2160 at the frequency of 201 MHz. The proposed ME system enhances the hardware efficiency by at least 50% compared to the prior works.