Object Motion Estimation Research Articles

Self-supervised rigid object 3-D motion estimation from monocular video

Independent object 3D motion estimation is a fundamental problem in 3D computer vision. Directly segmenting and estimating rigid object 3D motion from a consistent video frame is an ill-posed problem. We present a self-supervised framework for segmenting moving independent rigid objects and estimating their motion information including location, driving direction, and speed from a monocular video. Specifically, we first estimate depth, optical flow, and camera pose between a pair of video frames, and then synthesize a new 3D viewpoint from this pair. Subsequently, the Motion Recurrent All-Pairs Field Transforms (MRAFT) module is introduced to extract 3D scene flow and a motion area binary mask from a pair of images and depth. After that, a Rigid Object Motion Estimation Module (ROMEM) with a slot attention mechanism is proposed to extract rigid object motion masks from a multi-layer motion field, including optical flow, depth changes, refined scene flow, and motion masks. Finally, 2D images and 3D scene reconstruction errors are used to facilitate self-supervised training for rigid object motion. Experiments on the FlyingThings3D and KITTI datasets demonstrate that our method outperforms other advanced algorithms in estimating depth, optical flow, scene flow, and rigid moving object masks, demonstrating the benefits of our approach.

Siamese Multi-Scale Adaptive Search Network for Remote Sensing Single-Object Tracking

With the development of remote sensing earth observation technology, object tracking has gained attention for its broad application prospects in computer vision. However, object tracking is challenging owing to the background clutter, occlusion, and scale variation that often appear in remote sensing videos. Many existing trackers cannot accurately track the object for remote sensing videos with complex backgrounds. Several tracking methods can handle just one situation, such as occlusion. In this article, we propose a Siamese multi-scale adaptive search (SiamMAS) network framework to achieve object tracking for remote sensing videos. First, a multi-scale cross correlation is presented to obtain a more discriminative model and comprehensive feature representation, improving the performance of the model to handle complex backgrounds in remote sensing videos. Second, an adaptive search module is employed that augments the Kalman filter with a partition search strategy for object motion estimation. The Kalman filter is adopted to re-detect the object when the network cannot track the object in the current frame. Moreover, the partition search strategy can help the Kalman filter accomplish a more accurate region-proposal selection. Finally, extensive experiments on remote sensing videos taken from Jilin-1 commercial remote sensing satellites show that the proposed tracking algorithm achieves strong tracking performance with 0.913 precision while running at 37.528 frames per second (FPS), demonstrating its effectiveness and efficiency.

Temporal Estimation of Non-Rigid Dynamic Human Point Cloud Sequence Using 3D Skeleton-Based Deformation for Compression.

This paper proposes an algorithm for transmitting and reconstructing the estimated point cloud by temporally estimating a dynamic point cloud sequence. When a non-rigid 3D point cloud sequence (PCS) is input, the sequence is divided into groups of point cloud frames (PCFs), and a key PCF is selected. The 3D skeleton is predicted through 3D pose estimation, and the motion of the skeleton is estimated by analyzing the joints and bones of the 3D skeleton. For the deformation of the non-rigid human PC, the 3D PC model is transformed into a mesh model, and the key PCF is rigged using the 3D skeleton. After deforming the key PCF into the target PCF utilizing the motion vector of the estimated skeleton, the residual PC between the motion compensation PCF and the target PCF is generated. If there is a key PCF, the motion vector of the target PCF, and a residual PC, the target PCF can be reconstructed. Just as compression is performed using pixel correlation between frames in a 2D video, this paper compresses 3D PCFs by estimating the non-rigid 3D motion of a 3D object in a 3D PC. The proposed algorithm can be regarded as an extension of the 2D motion estimation of a rigid local region in a 2D plane to the 3D motion estimation of a non-rigid object (human) in 3D space. Experimental results show that the proposed method can successfully compress 3D PC sequences. If it is used together with a PC compression technique such as MPEG PCC (point cloud compression) in the future, a system with high compression efficiency may be configured.

Vision-Based Real-Time Motion Estimation of Ground Vehicles from Aerial Imagery

This research presents vision-based maneuvering object motion estimation in case of occlusion. Unequal dimension Interactive Multiple Model (UDIMM) approach is applied to increase the motion prediction accuracy when no measurement is available. Current deep learning-based multi-object tracking algorithms cannot track the objects when an occlusion exists, even for quite a short time. The main reason is the motion model used in tracking algorithms. If the measurement coming from the object detector is not available, the prediction model propagates the Kalman Filter’s motion model. In this period, the object may maneuver or accelerate/decelerate so that the tracker cannot assign the tracklets successfully. Eventually, the ID switch occurs, and object tracking gets lost. Using the UDIMM approach, track loss due to the ID switch is highly reduced compared to the simple constant velocity motion model used in the StrongSort multi-object tracker algorithm. A two-dimensional simulation environment is created to compare the ground truth data with the estimation results. Performance results of different filters are compared.

DTFS-eHarris: A High Accuracy Asynchronous Corner Detector for Event Cameras in Complex Scenes

The event camera, a new bio-inspired vision sensor with low latency and high temporal resolution, has brought great potential and demonstrated a promising application in machine vision and artificial intelligence. Corner detection is a key step of object motion estimation and tracking. However, most existing event-based corner detectors, such as G-eHarris and Arc*, lead to a huge number of redundant or wrong corners, and cannot strike a balance between the accuracy and real-time performance, especially in complex scenes with high texture that require higher computational costs. To address these issues, we propose an asynchronous corner detection method: a double threshold filter with Sigmoid eHarris (DTFS-eHarris) and an asynchronous corner tracker. The main contributions are that a double threshold filter is designed to reduce the redundant events and the improved Sigmoid function is utilized to represent the Surface of Active Events (Sigmoid*-SAE). We selected four scenes—shapes, dynamic, poster and boxes—from the public event camera dataset DAVIS240C to compare with the existing state-of-the-art hybrid method; our method has shown more than a 10% reduction in false positive rate and a 5% and 20% improvement in accuracy and throughput, respectively. The evaluations indicate that DTFS-eHarris shows a significant improvement, especially in complex scenes. Thus, it is anticipated to enhance the real-time performance and feature detection accuracy for future robotic applications.

Motion Estimation of Non-Cooperative Space Objects Based on Monocular Sequence Images

The motion estimation of non-cooperative space objects is an important prerequisite for successfully completing on-orbit servicing, debris removing and asteroid exploration missions. The motion estimation of non-cooperative space objects is a challenging task because of the lack of prior information about the object in unknown scenarios. In this paper, a motion estimation method of non-cooperative space objects for autonomous navigation based on a monocular sequence of images is proposed. The rotational attitude estimation of the non-cooperative space object is realized by matching the feature points of adjacent frame images, and the conditions required for translational estimation are discussed. This method has small calculation costs and runs stably, which meets the requirements of real-time computation on-board. As a result, this method implements the estimation of the complete unknown information of the object, including the attitude quaternion and angular velocity. Numerical simulations and results verify the effectiveness of the proposed method.

Pose and Motion Estimation of Free-Flying Objects: Aerodynamics, Constrained Filtering, and Graph-Based Feature Tracking

In this article, we investigate the problem of dynamically estimating the instantaneous position, orientation, velocity, and angular velocity of an arbitrarily shaped object during its free flight based on image frames taken simultaneously by two high-speed cameras. Aerodynamic effects, including drag, lift, and Magnus forces, are modeled to describe the object’s flight. Observables are derived from combining dynamics with a camera projection model assuming two-view geometry, via the use of multiple quaternions. The state of the composed system can then be estimated via constrained Kalman filtering, to which a solution is presented for the case of multiple quadratic constraints. To keep track of appearing and disappearing visual features during flight, the estimation algorithm employs a graph matching-based technique to maintain a set of evolving hypotheses through evaluation, pruning, and addition. Experiments conducted over various objects have either provided validation against motions independently estimated using multiple accelerometers, or formed verification by matching flight images against projections based on the state estimates.

Modified block-matching algorithm for moving object tracking in video surveillance

Video surveillance has risen as one of the most promising methods for people who live alone in their dwellings. Few video surveillance innovations have recently been introduced. However, due to various changes in illumination, abrupt shifts in target appearance, identical non-target artifacts in the background, and occlusions, developing a reliable video surveillance algorithm remains a difficult challenge. This work attempts to introduce a new framework for moving object detection and tracking by following four major phases: “Video-to-Frame Conversion, Pre-Processing, Background Subtraction, Feature-Based Multi-object Detection, Multi-object Tracking by Filtering”. Initially, in the Video-to-Frame Conversion process, the recorded input video clips are transformed into distinct frames. During pre-processing, the noise is removed from the video frame using a filtering approach, and thereby the nature of the images will be enhanced. In the proposed work, a Weiner filter is used to remove noise and other undesirable features during the pre-processing. Then, to distinguish the frontal areas of objects, background subtraction is performed using the neutrosophic set in noiseless video frames (pre-processed frames). The objects in the background-subtracted frames are separated using Improved Region Growing (IRG) segmentation model in the Feature-Based Multi-object Detection phase. The objects in the frames are determined from this segmented image. The Modified Full Search Algorithm is being used to track the object (motion estimation) on the video frame after it has been identified in the segmented phase. The Modified full search block matching algorithm (MFSA) is introduced in this research work to find the appropriate mobility. Promising results have been obtained by the proposed work, and also the mathematical excellence of the new method is also proven over other state-of-the-art models.

A novel enhanced algorithm for efficient human tracking

<p><span>Tracking moving objects has been an issue in recent years of computer vision and image processing and human tracking makes it a more significant challenge. This category has various aspects and wide applications, such as autonomous deriving, human-robot interactions, and human movement analysis. One of the issues that have always made tracking algorithms difficult is their interaction with goal recognition methods, the mutable appearance of variable aims, and simultaneous tracking of multiple goals. In this paper, a method with high efficiency and higher accuracy was compared to the previous methods for tracking just objects using imaging with the fixed camera is introduced. The proposed algorithm operates in four steps in such a way as to identify a fixed background and remove noise from that. This background is used to subtract from movable objects. After that, while the image is being filtered, the shadows and noises of the filmed image are removed, and finally, using the bubble routing method, the mobile object will be separated and tracked. Experimental results indicated that the proposed model for detecting and tracking mobile objects works well and can improve the motion and trajectory estimation of objects in terms of speed and accuracy to a desirable level up to in terms of accuracy compared with previous methods.</span></p>

Enhanced the moving object detection and object tracking for traffic surveillance using RBF-FDLNN and CBF algorithm

For traffic surveillance systems (TSS), Moving object detection and tracking is the key technology. On account of moving object-orientation variation, changing weather conditions, moving objects’ appearance, and non-target objects in the background, it's very hard to detection some objects in a traffic sequence of the video frame. Though many methods were introduced to detect and track moving objects in a traffic environment, they could not attain a desirable output. Hence, this work proposes an enhanced system for automatic moving object detection (MOD) and object tracking (OT) using RBF-FDLNN and CFR algorithms. The noises existent in a video frame are initially filtered using a GESFA filter. Next, the conversion of the RGB frame to HSL is done, and the background regions are eradicated from the frame using IGMM. After background removal (BGR), the Motion Estimation (ME) of object and OT are performed using GCS and CFR algorithm, respectively. Subsequently, the required features in the frames are extracted and fed into the RBF-FDLNN classifier for performing object detection (OD). The performance of the proposed method RBF-FDLNN classifier is better than other existing methods in the given video frame. Lastly, the outcomes are proffered to corroborate the proposed method’s effectiveness.

Novel object motion estimation method for industrial vision systems in aligning machines

Industrial vision systems have been adopted in many manufacturing processes for monitoring manufacturing procedures. With the recent proposed concept of industrial information integration engineering (IIIE) (Da Xu, 2020 [1]), the adoption of industrial vision systems involves a wider IIIE context as a data acquisition measurement, and thus higher level intelligence like control automation is expected. Industrial vision systems have been used for object alignment in aligning machines. By coping with a pneumatic device, misaligned objects can be detected by vision systems and rejected. Since alignment efficiency depends on the appropriateness of the blown timing/pressure, which is casually manifested in the motion of the observed blown objects, a motion estimation method for industrial vision systems must be adopted. Although conventional motion estimation methods in this field typically use pre-prepared templates, this paper proposed a novel object motion estimation method based on the properties of industrial images. A bounding box within a confined observation area is first initialized for each object, and then motion estimation is achieved by updating the bounding box following the expectation–maximization principle. Experiments revealed that our proposed method achieved robust and continuous motion estimation of the parts and reduced the processing time compared to conventional template matching methods.

Real-Time High Speed Motion Prediction Using Fast Aperture-Robust Event-Driven Visual Flow.

Optical flow is a crucial component of the feature space for early visual processing of dynamic scenes especially in new applications such as self-driving vehicles, drones and autonomous robots. The dynamic vision sensors are well suited for such applications because of their asynchronous, sparse and temporally precise representation of the visual dynamics. Many algorithms proposed for computing visual flow for these sensors suffer from the aperture problem as the direction of the estimated flow is governed by the curvature of the object rather than the true motion direction. Some methods that do overcome this problem by temporal windowing under-utilize the true precise temporal nature of the dynamic sensors. In this paper, we propose a novel multi-scale plane fitting based visual flow algorithm that is robust to the aperture problem and also computationally fast and efficient. Our algorithm performs well in many scenarios ranging from fixed camera recording simple geometric shapes to real world scenarios such as camera mounted on a moving car and can successfully perform event-by-event motion estimation of objects in the scene to allow for predictions of upto 500 ms i.e., equivalent to 10 to 25 frames with traditional cameras.

Conditional random field tracking model based on a visual long short term memory network

In dense pedestrian tracking, frequent object occlusions and close distances between objects cause difficulty when accurately estimating object trajectories. In this study, a conditional random field tracking model is established by using a visual long short term memory network in the three dimensional space and the motion estimations jointly performed on object trajectory segments. Object visual field information is added to the long short term memory network to improve the accuracy of the motion related object pair selection and motion estimation. To address the uncertainty of the length and interval of trajectory segments, a multimode long short term memory network is proposed for the object motion estimation. The tracking performance is evaluated using the PETS2009 dataset. The experimental results show that the proposed method achieves better performance than the tracking methods based on independent motion estimation.

End-to-End Learning of Object Motion Estimation from Retinal Events for Event-Based Object Tracking

Event cameras, which are asynchronous bio-inspired vision sensors, have shown great potential in computer vision and artificial intelligence. However, the application of event cameras to object-level motion estimation or tracking is still in its infancy. The main idea behind this work is to propose a novel deep neural network to learn and regress a parametric object-level motion/transform model for event-based object tracking. To achieve this goal, we propose a synchronous Time-Surface with Linear Time Decay (TSLTD) representation, which effectively encodes the spatio-temporal information of asynchronous retinal events into TSLTD frames with clear motion patterns. We feed the sequence of TSLTD frames to a novel Retinal Motion Regression Network (RMRNet) to perform an end-to-end 5-DoF object motion regression. Our method is compared with state-of-the-art object tracking methods, that are based on conventional cameras or event cameras. The experimental results show the superiority of our method in handling various challenging environments such as fast motion and low illumination conditions.

Vision Tracking: A Survey of the State-of-the-Art

Vision tracking is a well-studied framework in vision computing. Developing a robust visual tracking system is challenging because of the sudden change in object motion, cluttered background, partial occlusion and camera motion. In this study, the state-of-the art visual tracking methods are reviewed and different categories are discussed. The overall visual tracking process is divided into four stages—object initialization, appearance modeling, motion estimation, and object localization. Each of these stages is briefly elaborated and related researches are discussed. A rapid growth of visual tracking algorithms is observed in last few decades. A comprehensive review is reported on different performance metrics to evaluate the efficiency of visual tracking algorithms which might help researchers to identify new avenues in this area. Various application areas of the visual tracking are also discussed at the end of the study.

Occlusion Boundary: A Formal Definition & Its Detection via Deep Exploration of Context.

Occlusion boundaries contain rich perceptual information about the underlying scene structure and provide important cues in many visual perception-related tasks such as object recognition, segmentation, motion estimation, scene understanding, and autonomous navigation. However, there is no formal definition of occlusion boundaries in the literature, and state-of-the-art occlusion boundary detection is still suboptimal. With this in mind, in this paper we propose a formal definition of occlusion boundaries for related studies. Further, based on a novel idea, we develop two concrete approaches with different characteristics to detect occlusion boundaries in video sequences via enhanced exploration of contextual information (e.g, local structural boundary patterns, observations from surrounding regions, and temporal context) with deep models and conditional random fields. Experimental evaluations of our methods on two challenging occlusion boundary benchmarks (CMU and VSB100) demonstrate that our detectors significantly outperform the current state-of-the-art. Finally, we empirically assess the roles of several important components of the proposed detectors to validate the rationale behind these approaches.

Correlation Based Low Complex Video Frame Interpolation

Frame Interpolation is one of the main stages in video processing. Video coding standards skip some in-between frames for efficient compression and coding. At decoder the common approach to reconstruct the skipped frame using Motion Compensated Frame Interpolation (MCFI)methods. In MCFI, computational complexity is very high as calculation of Block Matching Algorithm, Motion Vectors (MV), Motion Estimation(ME) and Prediction logic of objects in motion between the frames, increases the complexity in MCFI method. A more feasible approach with minimum computational complexity using block level correlation is proposed in this paper. Errored MV at the decoder results in holes, occlusions, blurring and edge deformations in the interpolated frame. This proposal minimizes afore mentioned effects along with minimum complexity. The results are simulated in terms of peak-signal-to-noise-ratio (PSNR) and structural similarity index (SSIM).

Fast hierarchical wavelet-domain motion estimation for arbitrarily shaped visual objects

Motion estimation is widely used for video compression as it reduces the memory requirements of any video file while maintaining high visual quality. But the computational cost consumed by motion estimation/compensation usually accounts for more than 50%, even to 80%, of the total cost of the encoder. So researchers try to develop fast motion estimation algorithms to lower the computational complexity. Moreover, few studies have been concentrated on the efficient motion estimation of arbitrarily shaped visual objects, which particularly pays insufficient attention to boundary continuity of the motion-compensated objects as well. In this research, we present a hierarchical wavelet-domain approach to motion estimation of visual objects with arbitrary shapes. We explore the guiding role of binary alpha-plane in assisting motion estimation of visual objects, and bring forward a binary alpha-plane matching scheme. It constrains the intensity-based motion estimation into a small subset of initial candidate vectors in the search window. This both accelerates search speed and helps to produce coherent object boundary. Moreover, we employ the low-band-shift technique to eliminate the wavelet’s shift-variance effect. And based on the observation that high frequency subbands of wavelet contribute little to calculating motion vectors, a new matching function is derived for fast wavelet-domain motion estimation. Coupled with the alpha-plane matching and the devised matching function, a hierarchical low-band-based motion estimation approach is further proposed. Through executing a refined search around the initial candidate vectors using only the low frequency subband at each scale, our approach achieves a satisfying balance between prediction accuracy and computational efficiency. Experimental results show that by using the standard accuracy measurements and the benchmark video sequences, the performance of the proposed approach is close to the full search with a matching quality over 99%, and with 95.80% reduction in complexity.

SCOM: Spatiotemporal Constrained Optimization for Salient Object Detection.

This paper presents a novel model for video salient object detection called spatiotemporal constrained optimization model (SCOM), which exploits spatial and temporal cues, as well as a local constraint, to achieve a global saliency optimization. For a robust motion estimation of salient objects, we propose a novel approach to modeling the motion cues from optical flow field, the saliency map of the prior video frame and the motion history of change detection, which is able to distinguish the moving salient objects from diverse changing background regions. Furthermore, an effective objectness measure is proposed with intuitive geometrical interpretation to extract some reliable object and background regions, which provided as the basis to define the foreground potential, background potential, and the constraint to support saliency propagation. These potentials and the constraint are formulated into the proposed SCOM framework to generate an optimal saliency map for each frame in a video. The proposed model is extensively evaluated on the widely used challenging benchmark data sets. Experiments demonstrate that our proposed SCOM substantially outperforms the state-of-the-art saliency models.

Object Tracking with LiDAR: Monitoring Taxiing and Landing Aircraft

Mobile light detection and ranging (LiDAR) sensors used in car navigation and robotics, such as the Velodyne’s VLP-16 and HDL-32E, allow for sensing the surroundings of the platform with high temporal resolution to detect obstacles, tracking objects and support path planning. This study investigates the feasibility of using LiDAR sensors for tracking taxiing or landing aircraft close to the ground to improve airport safety. A prototype system was developed and installed at an airfield to capture point clouds to monitor aircraft operations. One of the challenges of accurate object tracking using the Velodyne sensors is the relatively small vertical field of view (30°, 41.3°) and angular resolution (1.33°, 2°), resulting in a small number of points of the tracked object. The point density decreases with the object–sensor distance, and is already sparse at a moderate range of 30–40 m. The paper introduces our model-based tracking algorithms, including volume minimization and cube trajectories, to address the optimal estimation of object motion and tracking based on sparse point clouds. Using a network of sensors, multiple tests were conducted at an airport to assess the performance of the demonstration system and the algorithms developed. The investigation was focused on monitoring small aircraft moving on runways and taxiways, and the results indicate less than 0.7 m/s and 17 cm velocity and positioning accuracy achieved, respectively. Overall, based on our findings, this technology is promising not only for aircraft monitoring but for airport applications.