Object Orientation Estimation Research Articles

Joint stereo 3D object detection and implicit surface reconstruction

We present a new learning-based framework S-3D-RCNN that can recover accurate object orientation in SO(3) and simultaneously predict implicit rigid shapes from stereo RGB images. For orientation estimation, in contrast to previous studies that map local appearance to observation angles, we propose a progressive approach by extracting meaningful Intermediate Geometrical Representations (IGRs). This approach features a deep model that transforms perceived intensities from one or two views to object part coordinates to achieve direct egocentric object orientation estimation in the camera coordinate system. To further achieve finer description inside 3D bounding boxes, we investigate the implicit shape estimation problem from stereo images. We model visible object surfaces by designing a point-based representation, augmenting IGRs to explicitly address the unseen surface hallucination problem. Extensive experiments validate the effectiveness of the proposed IGRs, and S-3D-RCNN achieves superior 3D scene understanding performance. We also designed new metrics on the KITTI benchmark for our evaluation of implicit shape estimation.

Asynchronous Method of Simultaneous Object Position and Orientation Estimation with Two Transmitters

Asynchronous Method of Simultaneous Object Position and Orientation Estimation with Two Transmitters

PolarMesh: A Star-Convex 3D Shape Approximation for Object Pose Estimation

In this letter, we introduce PolarMesh as a star-convex approximation of a 3D object based on spherical projection and can be applied to monocular object pose and shape estimation. The proposed PolarMesh can be stored in a discrete 2D map that allows a trivial conversion between it and the object surface, as the adjacent map elements form faces of the 3D model. The fixed-grid 2D nature of this object surface approximation ensures that it can be estimated from an image patch of the object with an auto-encoder neural network. Moreover, the PolarMesh not only encodes the object shape but also its orientation. Consequently, it is used for accurate estimation of object orientation and can be extended to the full 6D pose estimation using a projection-based method for translation. An exhaustive quantitative evaluation on LineMod, LineMod-Occlusion and HomebrewedDB shows the competitive performance of the proposed method compared to other monocular RGB pose estimation pipelines. Furthermore, we tested the generalization capabilities of the representation on the recent NOCS dataset for category-level pose estimation and achieved very compelling results in terms of predicted rotations.

Revealing Sea Turtle Behavior in Relation to Fishing Gear Using Color-Coded Spatiotemporal Motion Patterns With Deep Neural Networks

Incidental capture, or bycatch, of marine species is a global conservation concern. Interactions with fishing gear can cause mortality in air-breathing marine megafauna, including sea turtles. Despite this, interactions between sea turtles and fishing gear—from a behavior standpoint—are not sufficiently documented or described in the literature. Understanding sea turtle behavior in relation to fishing gear is key to discovering how they become entangled or entrapped in gear. This information can also be used to reduce fisheries interactions. However, recording and analyzing these behaviors is difficult and time intensive. In this study, we present a machine learning-based sea turtle behavior recognition scheme. The proposed method utilizes visual object tracking and orientation estimation tasks to extract important features that are used for recognizing behaviors of interest with green turtles (Chelonia mydas) as the study subject. Then, these features are combined in a color-coded feature image that represents the turtle behaviors occurring in a limited time frame. These spatiotemporal feature images are used along a deep convolutional neural network model to recognize the desired behaviors, specifically evasive behaviors which we have labeled “reversal” and “U-turn.” Experimental results show that the proposed method achieves an average F1 score of 85% in recognizing the target behavior patterns. This method is intended to be a tool for discovering why sea turtles become entangled in gillnet fishing gear.

Two-Dimensional Position and Orientation Estimation Using a 5.8 GHz RFID System

Precise estimation of object position and orientation using RF tags is an emerging application of RFID technology. In this work, we present a novel wireless two-dimensional position and orientation estimation and tracking technique that illustrates the concept of sensor fusion-using multiple, disparate sensor measurements to synthesize a single, superior estimate of position and orientation. The technique is implemented using a 5.8 GHz RFID system that consists of three readers and a customized tag equipped with a 3-axis accelerometer, 3-axis gyroscope and 3-axis magnetometer. Received signal strengths and packet-to-packet phase differences at the reader are used to measure radial tag positions and radial tag velocities, and backscattered data packets provide measurements of the tag's linear accelerations, angular velocities and orientations. A hybrid inertial microwave reflectometry (HIMR) algorithm is used to produce motion-capture grade estimates of tag position and orientation; in experiments, a two-dimensional position accuracy of less than 13 mm was achieved. The HIMR algorithm is designed using physics-based modeling equations, modifications are proposed to accommodate temporary link loss, and observability analyses clarify the foundations of the estimator design method.

A multi-task Faster R-CNN method for 3D vehicle detection based on a single image

Vehicle detection is an important part of robot environmental perception. In this paper, a 3D vehicle detection method using a single image is proposed to generate the 3D space coordinate information of the object using monocular vision for autonomous driving. The proposed method works under the multi-task framework and integrates 2D object detection, 3D object detection, orientation estimation and key point detection into one unified deep convolution neural network (DCNN) which could be trained by end-to-end learning. Besides, our proposed method is built by modifying Fast R-CNN using multi-task learning, and thus our proposed method is named multi-task Faster R-CNN (MT-Faster R-CNN). The experiments on KITTI dataset are conducted to evaluate our proposed method and the other 3D vehicle detection methods. The experimental results demonstrate that our proposed method is competitive and could significantly assist autonomous driving.

A fast self-attention cascaded network for object detection in large scene remote sensing images

Aiming at the real-time detection of multiple objects and micro-objects in large-scene remote sensing images, a cascaded convolutional neural network real-time object-detection framework for remote sensing images is proposed, which integrates visual perception and convolutional memory network reasoning. The detection framework is composed of two fully convolutional networks, namely, the strengthened object self-attention pre-screening fully convolutional network (SOSA-FCN) and the object accurate detection fully convolutional network (AD-FCN). SOSA-FCN introduces a self-attention module to extract attention feature maps and constructs a deep feature pyramid to optimize the attention feature maps by combining convolutional long-term and short-term memory networks. It guides the acquisition of potential sub-regions of the object in the scene, reduces the computational complexity, and enhances the network’s ability to extract multi-scale object features. It adapts to the complex background and small object characteristics of a large-scene remote sensing image. In AD-FCN, the object mask and object orientation estimation layer are designed to achieve fine positioning of candidate frames. The performance of the proposed algorithm is compared with that of other advanced methods on NWPU_VHR-10, DOTA, UCAS-AOD, and other open datasets. The experimental results show that the proposed algorithm significantly improves the efficiency of object detection while ensuring detection accuracy and has high adaptability. It has extensive engineering application prospects.

2D-Key-Points-Localization-Driven 3D Aircraft Pose Estimation

In this paper, we are interesting in inferring 3D pose estimation of aircraft object leveraging 2D key-points localization. Monocular vision based pose estimation for aircraft can be widely utilized in airspace tasks like flight control system, air traffic management, autonomous navigation and air defense system. Nonetheless, prior methods using directly regression or classification can not meet the requirements of high precision in aircraft pose estimation context, other approaches using PnP algorithms that need additional information such as template 3D model or depth as prior knowledge. These methods do not exploit to full advantage the correlation information between 2D key-points and 3D pose. In this paper, we present a multi-branch network, named AirPose network, using convolutional neural network to address 3D pose estimation based on 2D key-points information. In the meantime, a novel feature fusion method is explored to enable orientation estimation branch adequately exploit key-points information. Our feature fusion method significantly decreases 3D pose estimation error also avoids the involvement of RANSAC based PnP algorithms. To address the problem that there is no available dedicated aircraft 3D pose dataset for training and testing, we build a visual simulation platform on Unreal Engine 4 applying domain randomization (DR) skill, named AKO platform, which generates aircraft images automatically labeled with 3D orientation and key-points location. The dataset is called AKO dataset. We implement a series of ablation experiments to evaluate our framework for aircraft object detection, key-points localization and orientation estimation on AKO dataset. Experiments show that our proposed AirPose network leveraging AKO dataset can achieve convincing results for each of the tasks.

Efficient collection and automatic annotation of real-world object images by taking advantage of post-diminished multiple visual markers

ABSTRACTTo collect a human-annotated dataset for training deep convolutional neural networks is a very time-consuming and laborious process. To reduce this burden, we previously proposed an automated annotation by placing one visual marker above the detection target object in the training phase. However, in this approach, occasionally the marker hides the object surface. To avoid this issue, we propose placing a pedestal with multiple markers at the bottom of the object. If we use multiple markers, the object can be annotated even when the object hides some of the markers. Besides that, the simple modification of placing the markers on the bottom allows the use of simple background masking to avoid the neural network learning the remaining markers in the training image as a feature of the object. Background masking can completely remove the markers during the training process. Experiments showed the proposed vision system using our automatic object annotation outperformed the vision system using manual annotation in terms of object detection, orientation estimation, and 2D position estimation while reducing the time required for dataset collection from 16.1 hours to 7.30 hours.

Real-Time Object Detection in Remote Sensing Images Based on Visual Perception and Memory Reasoning

Aiming at the real-time detection of multiple objects and micro-objects in large-scene remote sensing images, a cascaded convolutional neural network real-time object-detection framework for remote sensing images is proposed, which integrates visual perception and convolutional memory network reasoning. The detection framework is composed of two fully convolutional networks, namely, the strengthened object self-attention pre-screening fully convolutional network (SOSA-FCN) and the object accurate detection fully convolutional network (AD-FCN). SOSA-FCN introduces a self-attention module to extract attention feature maps and constructs a depth feature pyramid to optimize the attention feature maps by combining convolutional long-term and short-term memory networks. It guides the acquisition of potential sub-regions of the object in the scene, reduces the computational complexity, and enhances the network’s ability to extract multi-scale object features. It adapts to the complex background and small object characteristics of a large-scene remote sensing image. In AD-FCN, the object mask and object orientation estimation layer are designed to achieve fine positioning of candidate frames. The performance of the proposed algorithm is compared with that of other advanced methods on NWPU_VHR-10, DOTA, UCAS-AOD, and other open datasets. The experimental results show that the proposed algorithm significantly improves the efficiency of object detection while ensuring detection accuracy and has high adaptability. It has extensive engineering application prospects.

Hybrid Polarimetric GPR Calibration and Elongated Object Orientation Estimation

Ground penetrating radar (GPR) has been widely applied to the detection of subsurface elongated targets, such as underground pipes, concrete rebars, and subsurface fractures. The orientation angle of a subsurface elongated target can hardly be delineated by a commercial single-polarization GPR system. In this paper, a hybrid dual-polarimetric GPR system, which consists of a circularly polarized transmitting antenna and two linearly polarized receiving antenna, is employed to detect buried elongated objects. A polarimetric calibration experiment using a gridded trihedral is carried out to correct the imbalances and cross talk between the two receiving channels. A full-polarimetric scattering matrix is extracted from the double-channel GPR signals reflected from a buried elongated object. An improved Alford rotation method is proposed to estimate the orientation angle of the elongated object from the extracted scattering matrix, and its accuracy is validated by a numerical test. A laboratory experiment was further conducted to detect five metal rebar buried in dry sand at different orientation angle relative to the GPR scan direction. The maximum relative error of the estimated angles of the buried rebars in the migrated GPR images is less than 5%. It is concluded that radar polarimetry can provide not only richer information than single-polarization GPR, but also a reliable approach for orientation estimation of a subsurface elongated object.

RECOVERING THE 3D POSE AND SHAPE OF VEHICLES FROM STEREO IMAGES

Abstract. The precise reconstruction and pose estimation of vehicles plays an important role, e.g. for autonomous driving. We tackle this problem on the basis of street level stereo images obtained from a moving vehicle. Starting from initial vehicle detections, we use a deformable vehicle shape prior learned from CAD vehicle data to fully reconstruct the vehicles in 3D and to recover their 3D pose and shape. To fit a deformable vehicle model to each detection by inferring the optimal parameters for pose and shape, we define an energy function leveraging reconstructed 3D data, image information, the vehicle model and derived scene knowledge. To minimise the energy function, we apply a robust model fitting procedure based on iterative Monte Carlo model particle sampling. We evaluate our approach using the object detection and orientation estimation benchmark of the KITTI dataset (Geiger et al., 2012). Our approach can deal with very coarse pose initialisations and we achieve encouraging results with up to 82 % correct pose estimations. Moreover, we are able to deliver very precise orientation estimation results with an average absolute error smaller than 4°.

DETECTION AND 3D MODELLING OF VEHICLES FROM TERRESTRIAL STEREO IMAGE PAIRS

Abstract. The detection and pose estimation of vehicles plays an important role for automated and autonomous moving objects e.g. in autonomous driving environments. We tackle that problem on the basis of street level stereo images, obtained from a moving vehicle. Processing every stereo pair individually, our approach is divided into two subsequent steps: the vehicle detection and the modelling step. For the detection, we make use of the 3D stereo information and incorporate geometric assumptions on vehicle inherent properties in a firstly applied generic 3D object detection. By combining our generic detection approach with a state of the art vehicle detector, we are able to achieve satisfying detection results with values for completeness and correctness up to more than 86%. By fitting an object specific vehicle model into the vehicle detections, we are able to reconstruct the vehicles in 3D and to derive pose estimations as well as shape parameters for each vehicle. To deal with the intra-class variability of vehicles, we make use of a deformable 3D active shape model learned from 3D CAD vehicle data in our model fitting approach. While we achieve encouraging values up to 67.2% for correct position estimations, we are facing larger problems concerning the orientation estimation. The evaluation is done by using the object detection and orientation estimation benchmark of the KITTI dataset (Geiger et al., 2012).

3D Object Proposals Using Stereo Imagery for Accurate Object Class Detection.

The goal of this paper is to perform 3D object detection in the context of autonomous driving. Our method aims at generating a set of high-quality 3D object proposals by exploiting stereo imagery. We formulate the problem as minimizing an energy function that encodes object size priors, placement of objects on the ground plane as well as several depth informed features that reason about free space, point cloud densities and distance to the ground. We then exploit a CNN on top of these proposals to perform object detection. In particular, we employ a convolutional neural net (CNN) that exploits context and depth information to jointly regress to 3D bounding box coordinates and object pose. Our experiments show significant performance gains over existing RGB and RGB-D object proposal methods on the challenging KITTI benchmark. When combined with the CNN, our approach outperforms all existing results in object detection and orientation estimation tasks for all three KITTI object classes. Furthermore, we experiment also with the setting where LIDAR information is available, and show that using both LIDAR and stereo leads to the best result.

Set-based modeling and observer design for planar structure from motion

Sets that are defined either from closed contours or from a set of points are generic descriptors for several kinds of objects in binary images. In this paper, we derive a novel model in the space of sets and design an observer for the proposed model to estimate the depth and orientation of planar objects from a camera. This problem is well known as structure from motion. When the objects are only partially projected on the image plane of the camera, our model makes object depth and orientation estimation possible without feature tracking and matching between distinct image frames (i.e., the so-called correspondence problem), which is an advantage over the image moments-based model. However, the proposed model fails in some situations (the failures can be seen as brief instabilities), and it is not always continuous. To compensate for these drawbacks, we designed a fast observer based on L1 control theory with a binary signal for the proposed model. Stability analysis with respect to a certain asymptotic instability ratio is also presented in this paper. The effectiveness of the proposed model and observer is demonstrated through simulations and experimental results.

3D Traffic Scene Understanding From Movable Platforms.

In this paper, we present a novel probabilistic generative model for multi-object traffic scene understanding from movable platforms which reasons jointly about the 3D scene layout as well as the location and orientation of objects in the scene. In particular, the scene topology, geometry, and traffic activities are inferred from short video sequences. Inspired by the impressive driving capabilities of humans, our model does not rely on GPS, lidar, or map knowledge. Instead, it takes advantage of a diverse set of visual cues in the form of vehicle tracklets, vanishing points, semantic scene labels, scene flow, and occupancy grids. For each of these cues, we propose likelihood functions that are integrated into a probabilistic generative model. We learn all model parameters from training data using contrastive divergence. Experiments conducted on videos of 113 representative intersections show that our approach successfully infers the correct layout in a variety of very challenging scenarios. To evaluate the importance of each feature cue, experiments using different feature combinations are conducted. Furthermore, we show how by employing context derived from the proposed method we are able to improve over the state-of-the-art in terms of object detection and object orientation estimation in challenging and cluttered urban environments.

GEOMETRIC TRANSFORM INVARIANT TEXTURE ANALYSIS WITH MODIFIED CHEBYSHEV MOMENTS BASED ALGORITHM

Texture based Geometric invariance, which comprises of rotation scale and translation (RST) invariant is finding application in various areas including industrial inspection, estimation of object range and orientation, shape analysis, satellite imaging, and medical diagnosis. Moments based techniques, apart from being computationally simple as compared to other RST invariant texture operators, are also robust in presence of noise. Zernike moments (ZM) based techniques are one of the well-established methods used for texture identification. As ZM are continuous moments, when discretization is done for implementation, errors are introduced. Error, calculated as difference between theoretically computed values and simulated values is proved to be prominent for fine textures. Therefore, a novel approach to detect RST invariant textures present in image is presented in this paper. This approach calculates discrete Chebyshev moments (CM) of log-polar transformed images to achieve rotation and scale invariance. The image is made translation invariant by shifting it to its centroid. The data is collected as samples from Brodatz and Vistex data sets. Zernike moments and its modifications, along with proposed scheme are applied to the same and Performance evaluation apart from RST invariance is noise sensitivity and redundancy. The performance is also compared with circular Mellin Feature extractors.

Principal Axes Estimation Using the Vibration Modes of Physics-Based Deformable Models

This paper addresses the issue of accurate, effective, computationally efficient, fast, and fully automated 2-D object orientation and scaling factor estimation. The object orientation is calculated using object principal axes estimation. The approach relies on the object's frequency-based features. The frequency-based features used by the proposed technique are extracted by a 2-D physics-based deformable model that parameterizes the objects shape. The method was evaluated on synthetic and real images. The experimental results demonstrate the accuracy of the method, both in orientation and the scaling estimations.

A real-time model-based method for 3-D object orientation estimation in outdoor scenes

This letter presents a new method for real-time determination of the three-dimensional (3-D) orientation, i.e., the rotation angle with respect to one of the principal object axes, of a moving known object from a monocular image sequence. The method is composed by three steps: 1) extraction of the morphological skeleton from binary images, 2) projection of the skeleton function on two planes and its analytical approximation by nonuniform rational B-splines, and 3) comparison with a set of data stored into a model database. Several experiments performed on real images prove the method's validity.

A genetic algorithm for intelligent imaging from quantum-limited data

Abstract A parallel genetic algorithm is presented for 2-D object recognition and simulataneous estimation of object position, magnification and orientation from quantum-limited sensor data. Traditional approaches to this problem are based on matching a concise set of features (boundaries, corners, moments, etc.) from the sensor data to a corresponding set of model features. These approaches break down at low SNR due to a deluge of artifacts among the data features, and inconsistencies arising from the lack of optimal interaction between high-level and low-level vision processes. As a first step towards overcoming the above hurdles, this paper presents a drastic departure from conventional vision-based approaches that (i) avoids the computation of features from noisy data, and (ii) uses a synergistic interaction of high-level and low-level vision processes to avoid inconsistencies. The combined vision problem is posed as a large-scale global optimization over a single objective function that directly involves the sensor data, the noise model and object templates. The optimization is accomplished using a genetic algorithm that runs on a parallel computer with 40 Transputers. Experimental results are presented which demonstrate robust operation and high accuracy with quantum-limited (5–10 events/pixel) data.