Object State Estimation Research Articles

A comprehensive survey of space robotic manipulators for on-orbit servicing.

On-Orbit Servicing (OOS) robots are transforming space exploration by enabling vital maintenance and repair of spacecraft directly in space. However, achieving precise and safe manipulation in microgravity necessitates overcoming significant challenges. This survey delves into four crucial areas essential for successful OOS manipulation: object state estimation, motion planning, and feedback control. Techniques from traditional vision to advanced X-ray and neural network methods are explored for object state estimation. Strategies for fuel-optimized trajectories, docking maneuvers, and collision avoidance are examined in motion planning. The survey also explores control methods for various scenarios, including cooperative manipulation and handling uncertainties, in feedback control. Additionally, this survey examines how Machine learning techniques can further propel OOS robots towards more complex and delicate tasks in space.

DLOAM: Real-time and Robust LiDAR SLAM System Based on CNN in Dynamic Urban Environments

Dynamic object detection, state estimation, and map-building are crucial for autonomous robot systems and intelligent transportation applications in urban scenarios. Most current LiDAR Simultaneous Localization and Mapping (SLAM) systems operate on the assumption that the observed environment is static. However, the overall accuracy and robustness of a SLAM system can be compromised by dynamic objects in the environment. Aiming at the problem of inaccurate odometry estimation and wrong mapping caused by the existing LiDAR SLAM method which cannot detect the dynamic objects, we study the SLAM problem of robots and unmanned vehicles equipped with LiDAR traveling in the dynamic urban scenes. We propose a fast LiDAR-only model-free dynamic objects detection method, which uses the spatial and temporal information of point cloud through a convolutional neural network (CNN), and the detection accuracy is improved by 35 use spatial information. We further integrate it into a state-of-the-art LiDAR SLAM framework to improve the SLAM performance. Firstly, the range image constructed by LiDAR point cloud is used for ground extraction and non-ground point clustering. Then, the motion of objects in the scene is estimated by the difference between adjacent frames, and the segmented objects are further divided into dynamic objects and static objects by their motion features. After that, the stable feature points are extracted from the static objects. Finally, the pose transformation of adjacent frames is solved by matching feature point pairs. We evaluated the accuracy and robustness of our system on datasets with different challenging dynamic environments, and the results show our system has significant improvements in accuracy and robustness of odometry and mapping, while still maintain real-time performance, which is sufficient for autonomous robot systems and intelligent transportation applications in urban scenarios.

The Method for Generating a Set of Reference Images for Assessing the Condition of Critical Infrastructure Facilities Using Mobile Robots

The purpose of this work is to improve the accuracy of critical infrastructure condition assessment using mobile robots by considering the geometric distortions of the current images during the formation of a set of reference images. The goal is achieved by determining the sampling step values by angles and sighting height without loss of accuracy. The most important result is the determination of acceptable discretization values in the range of angles and heights of a correlation-extreme navigation system. The significance of the obtained results is in solving the problem of forming a set of reference images, which will reduce the impact of changes in the geometry of sighting on the accuracy of the evaluation of objects. A special feature of the results obtained is the establishment of maximum permissible sampling steps in angles and heights of sight to ensure the required accuracy of object state estimation. When forming a set of reference images, the sampling step by height should be (0.06....0.11)% and (0.12....0.2)% relative to the initial flight altitude for the sighting surface with normal and high object saturation, respectively. The angular sampling step is 10...17 degrees and 6...10 degrees, respectively, for the same surface types. The difference from known works is that the perspective and scale distortions are considered at the stage of formation of a set of reference images, which ensures high accuracy of the system functioning in conditions of orientation and sighting geometry changes.

Lidar Object Perception Framework for Urban Autonomous Driving: Detection and State Tracking Based on Convolutional Gated Recurrent Unit and Statistical Approach

This article describes the development and implementation of a 3D lidar perception framework to guarantee the precise cognition of the surrounding environment for urban autonomous driving. The proposed framework consists of two different detection modules operating in parallel: a deep learning-based and a geometric model-free cluster-based method. The first module utilizes the convolutional gated recurrent unit (ConvGRU)-based residual network (CGRN). The module aims to repredict 3D objects based on results from a continuous single-frame detection network. A vision-fusion methodology based on 2D projection is adopted for postprocessing in the first module. The second module utilizes geometric model-free area (GMFA) cluster detection and is designed to cope with false-negative cases of unclassified objects from the prior module. For the second module, a cluster variance-based ground removal is conducted to prevent false-positive cases. A kinematic model-based particle filter (PF) is then applied to estimate the dynamic states of detection. The suggested framework has been developed with real-time operation in mind, to be implemented in autonomous vehicles equipped with automotive lidars and low-cost cameras. The test results show that the framework with CGRN and GMFA successfully improved the surrounding object detection and state estimation accuracy in urban autonomous driving.

Adaptive Part Mining for Robust Visual Tracking.

Visual tracking aims to estimate object state in a video sequence, which is challenging when facing drastic appearance changes. Most existing trackers conduct tracking with divided parts to handle appearance variations. However, these trackers commonly divide target objects into regular patches by a hand-designed splitting way, which is too coarse to align object parts well. Besides, a fixed part detector is difficult to partition targets with arbitrary categories and deformations. To address the above issues, we propose a novel adaptive part mining tracker (APMT) for robust tracking via a transformer architecture, including an object representation encoder, an adaptive part mining decoder, and an object state estimation decoder. The proposed APMT enjoys several merits. First, in the object representation encoder, object representation is learned by distinguishing target object from background regions. Second, in the adaptive part mining decoder, we introduce multiple part prototypes to adaptively capture target parts through cross-attention mechanisms for arbitrary categories and deformations. Third, in the object state estimation decoder, we propose two novel strategies to effectively handle appearance variations and distractors. Extensive experimental results demonstrate that our APMT achieves promising results with high FPS. Notably, our tracker is ranked the first place in the VOT-STb2022 challenge.

Parameterized Particle Filtering for Tactile-Based Simultaneous Pose and Shape Estimation

Object state and shape estimation is essential in many robotic manipulation tasks (e.g., in-hand manipulation, insertion). While such estimation is typically relied on visual perception, for tasks to be carried out in a vision-degraded or vision-denied environment, haptics becomes the reliable source of perception. In this letter, we propose the use of parameterized particle filtering to estimate object pose and shape in 3D space using tactile feedback. This approach is able to estimate with high accuracy using contact information of the object with a collision surface from a rough initial estimation. In comparison to conventional particle filtering, this approach significantly reduces the number of particles required for a satisfactory estimation, making it applicable for pose and shape estimation, where the number of degrees of freedom is high or even uncertain. Moreover, the proposed method can automatically choose the fastest-convergent contact action during the pose estimation stage to shorten the time required. A set of experiments in both simulation and on a real-world robot have been conducted to validate the proposed method and compare against the state-of-the-art approach in the literature. Results from both sets of experiments show that the proposed method can determine the pose and shape of the objects with very high accuracy within a small number of iterations.

Development of object state estimation method in intelligent decision support systems

A method of object state estimation in intelligent decision support systems (DSS) has been developed. The essence of the method is to ensure a high-quality analysis of the current state of the analyzed object. The key difference of the developed method is the use of an advanced genetic algorithm. The advanced genetic algorithm is used when constructing a fuzzy cognitive model and increases the efficiency of identifying factors and relationships between them by simultaneously finding a solution by several individuals. The objective and complete analysis is achieved using advanced fuzzy temporal models of the object state, taking into account the type of uncertainty and noise of initial data. The method also contains an improved procedure for processing initial data under a priori uncertainty, an improved procedure for training artificial neural networks and an improved procedure for topological analysis of the structure of fuzzy cognitive models. The essence of the training procedure is the training of synaptic weights of the artificial neural network, the type and parameters of the membership function, as well as the architecture of individual elements and the architecture of the artificial neural network as a whole. The method increases the efficiency of data processing at the level of 11–15 % using additional advanced procedures. The proposed method can be used in DSS of automated control systems (artillery units, special-purpose geographic information systems). It can also be used in DSS for aviation and air defense ACS, as well as in DSS for logistics ACS of the Armed Forces

MultiPDF particle filtering in state estimation of nonlinear objects

This paper presents a new particle filter algorithm (MultiPDF) for state estimation of nonlinear systems. The proposed method is a modification of the standard particle filter approach. Due to the strong need for the acceleration of calculations and an improvement in the estimation quality of state estimation, the authors propose a method which enables one to divide the main particle filter into smaller sub-filters with an accordingly smaller number of particles for each one of them. The algorithm has been implemented for various numbers of particles and subordinate parallel filters. Estimation quality has been checked for nine nonlinear objects (both one- and multidimensional) and evaluated through the quality index, average root-mean-squared error. The computation time of the particle filter algorithm for several hardware configurations has been compared. Based on the obtained results, it can be concluded that, besides the computation acceleration, the parallelization of the particle filter’s operation also improves the estimation quality.

Learning-Based Autonomous UAV System for Electrical and Mechanical (E&M) Device Inspection.

The inspection of electrical and mechanical (E&M) devices using unmanned aerial vehicles (UAVs) has become an increasingly popular choice in the last decade due to their flexibility and mobility. UAVs have the potential to reduce human involvement in visual inspection tasks, which could increase efficiency and reduce risks. This paper presents a UAV system for autonomously performing E&M device inspection. The proposed system relies on learning-based detection for perception, multi-sensor fusion for localization, and path planning for fully autonomous inspection. The perception method utilizes semantic and spatial information generated by a 2-D object detector. The information is then fused with depth measurements for object state estimation. No prior knowledge about the location and category of the target device is needed. The system design is validated by flight experiments using a quadrotor platform. The result shows that the proposed UAV system enables the inspection mission autonomously and ensures a stable and collision-free flight.

Modeling of Deformable Objects for Robotic Manipulation: A Tutorial and Review.

Manipulation of deformable objects has given rise to an important set of open problems in the field of robotics. Application areas include robotic surgery, household robotics, manufacturing, logistics, and agriculture, to name a few. Related research problems span modeling and estimation of an object's shape, estimation of an object's material properties, such as elasticity and plasticity, object tracking and state estimation during manipulation, and manipulation planning and control. In this survey article, we start by providing a tutorial on foundational aspects of models of shape and shape dynamics. We then use this as the basis for a review of existing work on learning and estimation of these models and on motion planning and control to achieve desired deformations. We also discuss potential future lines of work.

Laser-scanner-based object state estimation and tracking control algorithms of autonomous truck using single-wheel driving module

This paper describes state estimation and tracking control algorithms for use in an autonomous truck using a single-wheel driving module and simulation-based performance evaluation with actual data from a one-layer laser scanner. In order to construct the tracking control algorithm, the one-layer laser scanner has been used to detect the object as two-dimensional point cloud data. The tracking control algorithm consists of three routines—perception, decision, and control. In the perception routine, the coordinate transformation, downsizing, clustering, and state estimation have been conducted for calculation of the object states, such as position and velocity, using point data from the one-layer laser scanner. The velocity components of the object have been estimated based on the extended Kalman filter, and the desired straight path has been derived using the estimated velocity of the object. To track the object reasonably, lateral error and yaw angle error have been defined with respect to the desired straight path. The error dynamics has been derived using a single-wheel driving module, equipped with a planar truck model based on the vehicle dynamics. Using the derived planar truck model, the optimal steering input for tracking the object has been computed based on the linear quadratic regulator. The actual point cloud data from the one-layer laser scanner has been used to conduct reasonable performance evaluation of the tracking control algorithm proposed in this study. The actual data-based performance evaluation of the algorithm has been conducted in the MATLAB/SIMULINK environment with various scenarios. The performance evaluation results show that the tracking control algorithm proposed in this study can manipulate the single-wheel driving module of the autonomous truck to track the object soundly, based on the state estimation algorithm.

Grid-Based Object Tracking With Nonlinear Dynamic State and Shape Estimation

Object tracking is crucial for planning safe maneuvers of mobile robots in dynamic environments, in particular for autonomous driving with surrounding traffic participants. Multi-stage processing of sensor measurement data is thereby required to obtain abstracted high-level objects, such as vehicles. This also includes sensor fusion, data association, and temporal filtering. Often, an early-stage object abstraction is performed, which, however, is critical, as it results in information loss regarding the subsequent processing steps. We present a new grid-based object tracking approach that, in contrast, is based on already fused measurement data. The input is thereby pre-processed, without abstracting objects, by the spatial grid cell discretization of a dynamic occupancy grid, which enables a generic multi-sensor detection of moving objects. On the basis of already associated occupied cells, presented in our previous work, this paper investigates the subsequent object state estimation. The object pose and shape estimation thereby benefit from the freespace information contained in the input grid, which is evaluated to determine the current visibility of extracted object parts. An integrated object classification concept further enhances the assumed object size. For a precise dynamic motion state estimation, radar Doppler velocity measurements are integrated into the input data and processed directly on the object-level. Our approach is evaluated with real sensor data in the context of autonomous driving in challenging urban scenarios.

Learning Multi-Task Correlation Particle Filters for Visual Tracking.

In this paper, we propose a multi-task correlation particle filter (MCPF) for robust visual tracking. We first present the multi-task correlation filter (MCF) that takes the interdependencies among different object parts and features into account to learn the correlation filters jointly. Next, the proposed MCPF is introduced to exploit and complement the strength of a MCF and a particle filter. Compared with existing tracking methods based on correlation filters and particle filters, the proposed MCPF enjoys several merits. First, it exploits the interdependencies among different features to derive the correlation filters jointly, and makes the learned filters complement and enhance each other to obtain consistent responses. Second, it handles partial occlusion via a part-based representation, and exploits the intrinsic relationship among local parts via spatial constraints to preserve object structure and learn the correlation filters jointly. Third, it effectively handles large scale variation via a sampling scheme by drawing particles at different scales for target object state estimation. Fourth, it shepherds the sampled particles toward the modes of the target state distribution via the MCF, and effectively covers object states well using fewer particles than conventional particle filters, thereby resulting in robust tracking performance and low computational cost. Extensive experimental results on four challenging benchmark datasets demonstrate that the proposed MCPF tracking algorithm performs favorably against the state-of-the-art methods.

Robust object tracking via multi-cue fusion

Object tracking is one of the fundamental problems and an active research area in computer vision during the last decade. Although a wide variety of methods have been proposed, the long-term object tracking is still a challenging problem when dealing with occlusion, out-of-view, scale and illumination variation. To address these challenges, we propose a robust visual object tracking method based on binocular vision in this paper. Our method formulates the object tracking problem in a multi-cue fusion framework which allows our system recover from tracking drift and occlusion. For each channel of the binocular cameras, the coarse object state is estimated by combining the information of motion model, detection and online tracker. Stereo filter is designed to check the object candidate consistency of the two channels. The final object state estimation is determined by fusing the two-channel information and binocular geometry constraints. Experimental results demonstrate the effectiveness of proposed method.

Robust Object Tracking With Discrete Graph-Based Multiple Experts.

Variations of target appearances due to illumination changes, heavy occlusions, and target deformations are the major factors for tracking drift. In this paper, we show that the tracking drift can be effectively corrected by exploiting the relationship between the current tracker and its historical tracker snapshots. Here, a multi-expert framework is established by the current tracker and its historical trained tracker snapshots. The proposed scheme is formulated into a unified discrete graph optimization framework, whose nodes are modeled by the hypotheses of the multiple experts. Furthermore, an exact solution of the discrete graph exists giving the object state estimation at each time step. With the unary and binary compatibility graph scores defined properly, the proposed framework corrects the tracker drift via selecting the best expert hypothesis, which implicitly analyzes the recent performance of the multi-expert by only evaluating graph scores at the current frame. Three base trackers are integrated into the proposed framework to validate its effectiveness. We first integrate the online SVM on a budget algorithm into the framework with significant improvement. Then, the regression correlation filters with hand-crafted features and deep convolutional neural network features are introduced, respectively, to further boost the tracking performance. The proposed three trackers are extensively evaluated on three data sets: TB-50, TB-100, and VOT2015. The experimental results demonstrate the excellent performance of the proposed approaches against the state-of-the-art methods.

Visual Object Tracking With Partition Loss Schemes

Object tracking is a fundamental task for building vision systems of automatic transportation. Despite demonstrated success in this active research field, it is still difficult to cope with complicated appearance changes caused by background clutters, illumination change, scale variation, deformation, rotation, and occlusion, etc. Due to these challenging factors, a target bounding box tends to easily contain a disturbing context of background, which may lead to an inaccurate localization if some key parts of the foreground object share an excess of information loss. In this paper, we propose an online algorithm using the local loss features to alleviate the drift problem during tracking. An adaptive block-division appearance model is constructed to exploit patch-based loss representations by decomposing sample region sequences into a set of subblocks. The basic purpose of partition coefficients is to indicate local relevance and effectively capture spatial correlation through measuring image similarity. Namely, they can strengthen positive effects of discriminative patches within locating bounding boxes and weaken negative impacts of disturbing context possibly included in the surrounding regions. The object state estimation of a moving target is then formulated as an integrated likelihood evaluation on the ensemble loss. Experimental results on a suite of representative video sequences of realistic scenarios demonstrate the superiority of the proposed method to several state-of-the-art tracking approaches in terms of both accuracy and robustness.

A novel approach of multi-stage tracking for precise localization of target in video sequences

Visual tracking methods are mostly based on single stage state estimation that limitedly caters to precise localization of target under dynamic environment such as occlusion, object deformation, rotation, scaling and cluttered background. In order to address these issues, we introduce a novel multi-stage coarse-to-fine tracking framework with quick adaptation to environment dynamics. The key idea of our work is to propose two-stage estimation of object state and to develop an adaptive fusion model. Coarse estimation of object state is achieved using optical flow and multiple fragments are generated around this approximation. Precise localization of object is obtained through evaluation of these fragments using three complementary cues. Adaptation of proposed tracker to dynamic environment changes is quick due to incorporation of context sensitive cue reliability, which encompass its direct application for development of expert system for video surveillance. In addition, proposed framework caters to object rotation and scaling through a random walk state model and rotation invariant features. The proposed tracker is evaluated over eight- benchmarked color video sequences and competitive results are obtained. As an average of the outcomes, we achieved mean center location error (in pixels) of 6.791 and F-measure of 0.78. Results demonstrate that proposed tracker not only outperforms various state-of-the-art trackers but also effectively caters to various dynamic environments.

Clustering for filtering: Multi-object detection and estimation using multiple/massive sensors

Advanced multi-sensor systems are expected to combat the challenges that arise in object recognition and state estimation in harsh environments with poor or even no prior information, while bringing new challenges mainly related to data fusion and computational burden. Unlike the prevailing Markov-Bayes framework that is the basis of a large variety of stochastic filters and the approximate, we propose a clustering-based methodology for multi-sensor multi-object detection and estimation (MODE), named clustering for filtering (C4F), which abandons unrealistic assumptions with respect to the objects, background and sensors. Rather, based on cluster analysis of the input multi-sensor data, the C4F approach needs no prior knowledge about the latent objects (whether quantity or dynamics), can handle time-varying uncertainties regarding the background and sensors such as noises, clutter and misdetection, and does so computationally fast. This offers an inherently robust and computationally efficient alternative to conventional Markov–Bayes filters for dealing with the scenario with little prior knowledge but rich observation data. Simulations based on representative scenarios of both complete and little prior information have demonstrated the superiority of our C4F approach.

Analysis of spatters in laser welding with beam oscillation: A machine vision approach

Spatter formation is still an issue in welding, particularly in laser welding. This research takes up the topic and proposes a machine vision approach for spatter tracking in high speed image series. After a description of the multi hypothesis tracking method, in which a Kalman-filter is used for optimal object state estimation, the tracking is applied on high speed images from an experimental series on laser welding with beam oscillation using stainless steel. On the basis of its results three different spatter formation mechanisms could be identified and described. Those were spatter formation by material ablation, by periodic re-entry of the laser spot into the melt pool and by melt pool dynamics.

Multiple-Object Space Surveillance Tracking Using Finite-Set Statistics

The dynamic tracking of objects is, in general, concerned with state estimation using imperfect data. Multiple object tracking adds the difficulty of encountering unknown associations between the collected data and the objects. State estimation of objects necessitates the prediction of uncertainty through nonlinear (in the general case) dynamical systems and the processing of nonlinear (in the general case) measurement data in order to provide corrections that refine the system uncertainty, where the uncertainty may be non-Gaussian in nature. The sensors, which provide the measurement data, are imperfect with possible misdetections, false alarms, and noise-affected data. The resulting measurements are inherently unassociated upon reception. In this paper, a Bayesian method for tracking an arbitrary, but known, number of objects is developed. The method is based on finite-set statistics coupled with finite mixture model representations of the multiobject probability density function. Instead of relying on first-moment approximations, such as the probability hypothesis density filter, to the full multiobject Bayesian posterior, as is often done for multiobject filtering, the proposed method operates directly on the exact Bayesian posterior. Results are presented for application of the method to the problem of tracking multiple space objects using synthetic line-of-sight data.