Stereo Tracking Research Articles

Research on the rapid 3D measurement of satellite antenna reflectors using stereo tracking technique

Measurement and analysis of surface accuracy are crucial steps in the manufacturing process of satellite antenna reflectors. The conventional close-range photogrammetry requires attaching target points on the surface, which is time-consuming and may damage the surface quality. To overcome these drawbacks, a rapid measurement technique of non-contact three-dimensional (3D) shapes is proposed, which is based on the integration of a stereo tracking method and a structured light measurement method. This paper focuses on addressing the issue of obtaining pose transformation matrices for measurement devices in motion within this method. Initially, a plane tracking scheme and tracking plate pattern is designed. Furthermore, the feature points of the tracking plate are extracted from the target area of image processing. Moreover, the pose transformation matrix of the tracking plate relative to the fixed world coordinate system at different positions is obtained using the triangulation method. Finally, a measurement experiment is conducted, resulting in a stereo tracking scheme error within 0.5%.

A Demonstration of Three-Satellite Stereo Winds

Stereo tracking of clouds from multiple satellites permits the simultaneous retrieval of an atmospheric motion vector (“wind”) and its height in the atmosphere. The direct measurement of height is a major advantage of stereo methods over observations made from a single satellite where the height must be inferred from infrared brightness temperatures. A pair of operational geostationary satellites over the Americas provides stereo coverage where their two fields of view intersect. Stereo coverage can be extended to nearly a full hemisphere with a third satellite. We demonstrate this configuration with the operational GOES-R constellation of GOES-16 (east) and GOES-17 (west) augmented by GOES-18 in its central test slot and use the 500-m resolution Advanced Baseline Imager Band 2. We examine the consistency of the pairwise products created from GOES-18 and -16 versus GOES-18 and -17 and create a fused triple-GOES product that spans nearly the full hemisphere seen from GOES-18. We also examine the retrieval of ground points observed under clear skies and compare their retrievals to zero speed and known terrain heights. The results are compatible with a wind accuracy about 0.1 m/s with height assignment uncertainty of 175 m.

A progressive online external parameter calibration and initialization method for stereo-IMU system

PurposeThis paper aims to present a pipeline to progressively deal with the online external parameter calibration and estimator initialization of the Stereo-inertial measurement unit (IMU) system, which does not require any prior information and is suitable for system initialization in a variety of environments.Design/methodology/approachBefore calibration and initialization, a modified stereo tracking method is adopted to obtain a motion pose, which provides prerequisites for the next three steps. Firstly, the authors align the pose obtained with the IMU measurements and linearly calculate the rough external parameters and gravity vector to provide initial values for the next optimization. Secondly, the authors fix the pose obtained by the vision and restore the external and inertial parameters of the system by optimizing the pre-integration of the IMU. Thirdly, the result of the previous step is used to perform visual-inertial joint optimization to further refine the external and inertial parameters.FindingsThe results of public data set experiments and actual experiments show that this method has better accuracy and robustness compared with the state of-the-art.Originality/valueThis method improves the accuracy of external parameters calibration and initialization and prevents the system from falling into a local minimum. Different from the traditional method of solving inertial navigation parameters separately, in this paper, all inertial navigation parameters are solved at one time, and the results of the previous step are used as the seed for the next optimization, and gradually solve the external inertial navigation parameters from coarse to fine, which avoids falling into a local minimum, reduces the number of iterations during optimization and improves the efficiency of the system.

Ambiguity-Free Optical-Inertial Tracking for Augmented Reality Headsets.

The increasing capability of computing power and mobile graphics has made possible the release of self-contained augmented reality (AR) headsets featuring efficient head-anchored tracking solutions. Ego motion estimation based on well-established infrared tracking of markers ensures sufficient accuracy and robustness. Unfortunately, wearable visible-light stereo cameras with short baseline and operating under uncontrolled lighting conditions suffer from tracking failures and ambiguities in pose estimation. To improve the accuracy of optical self-tracking and its resiliency to marker occlusions, degraded camera calibrations, and inconsistent lighting, in this work we propose a sensor fusion approach based on Kalman filtering that integrates optical tracking data with inertial tracking data when computing motion correlation. In order to measure improvements in AR overlay accuracy, experiments are performed with a custom-made AR headset designed for supporting complex manual tasks performed under direct vision. Experimental results show that the proposed solution improves the head-mounted display (HMD) tracking accuracy by one third and improves the robustness by also capturing the orientation of the target scene when some of the markers are occluded and when the optical tracking yields unstable and/or ambiguous results due to the limitations of using head-anchored stereo tracking cameras under uncontrollable lighting conditions.

Joint Deep and Depth for Object-Level Segmentation and Stereo Tracking in Crowds

Tracking multiple people in crowds is a fundamental and essential task in the multimedia field. It is often hindered by difficulties, such as dynamic occlusion between objects, cluttered background, and abrupt illumination changes. To respond to this need, in this paper, we combine deep and depth to build a stereo tracking system for crowds. The core of the system is the fusion of the advantages of deep learning and depth information, which is exploited to achieve object segmentation and improve the multiobject tracking performance in severe occlusion. More specifically, first, to obtain more accurate detection observations in the tracking system, we present a novel object-level segmentation method. This method combines the effective detection results of deep learning with depth information to obtain precise object segmentation results. Then, we integrate the segmentation results and three-dimensional (3-D) information to extract 2-D and 3-D characteristics to represent the target, and design three similarity models to realize a stereo tracking method through data association in crowds. Finally, we build a diverse stereo dataset including various challenging indoor and outdoor scenes. The comprehensive experiments verify the effective and robust tracking performance of our system in various scenarios, and the system has rich output results including segmentation results, target distance, and tracking results. Moreover, the qualitative and quantitative comparison results show that the proposed algorithm not only has good object segmentation performance but also improves the tracking performance of completely and partially occluded objects, which is superior to the tested state-of-the-art tracking approaches.

Aerodynamic Coefficients of Free-Flying Cubes in Hypersonic Flowfield

Aerodynamic coefficients of a cube depending on a broad-range attitude change have been measured using the free-flight technique in a hypersonic flowfield. Experiments were performed therefore in the hypersonic wind tunnel H2K at the German Aerospace Center (DLR) in Cologne. The free-flight technique in H2K has allowed achieving a continuous rotation of the cube without any sting interferences in a broad angular range of 90°. This motion of the model during the free flight has been acquired using a nonintrusive high-speed stereo tracking system. A marker-based tracking algorithm has been applied to reconstruct the three-dimensional flight trajectories and attitudes to determine the resulting forces and moments. By using high-speed schlieren photography, the unsteady flow structures have been recorded. Pitch-angle-dependent aerodynamic coefficients of rotating cubes have been observed.

Real-Time Monocular Three-Dimensional Motion Tracking Using a Multithread Active Vision System

In this study, we developed a monocular stereo tracking system to be used as a marker-based, three-dimensional (3-D) motion capture system. This system aims to localize dozens of markers on multiple moving objects in real time by switching five hundred different views in 1 s. The ultrafast mirror-drive active vision used in our catadioptric stereo tracking system can accelerate a series of operations for multithread gaze control with video shooting, computation, and actuation within 2 ms. By switching between five hundred different views in 1 s, with real-time video processing for marker extraction, our system can function asJvirtual left and right pan-tilt tracking cameras, operating at 250/Jfps to simultaneously capture and processJpairs of 512 × 512 stereo images with different views via the catadioptric mirror system. We conducted several real-time 3-D motion experiments to capture multiple fast-moving objects with markers. The results demonstrated the effectiveness of our monocular 3-D motion tracking system.

Finding an appropriate headphone-based stereo track playback methods by referencing acoustic characters of professional recording studios

Most Red Book CD or streaming 2-track music mixed and mastered in a professional recording studio. However, headphone listener experience different to its original studio condition. The recent headphone has decent frequency response, but the sounds do not match especially in soundstage (width and depth of music mix in a horizontal plane) perspective. Although Recording studios are designed to minimize unwanted reflections and resonances, every studio has its own room characteristics of natural room frequency response. In other words, “the acoustical room characteristic” added on “finalized mix” is the real representation of the mastered 2-track music. In this paper, we tried to find the ultimate headphone-based listening methods by referencing the professional recording studios.

Quantification of fast pinna motions in rhinolophid and hipposiderid bats

As part of their biosonar behaviors, rhinolophid (family Rhinolophidae) and hipposiderid bats (family Hipposideridae) both show conspicuous motions of their outer ears (pinna). These motions coincide with pulse emission and echo reception in time and could hence have a functional relevance for the encoding of sensory information. However, a quantitative in-depth characterization of these motions is still needed to derive detailed hypotheses for their function. To accomplish this, dense sets of landmark points have been placed on the pinna to provide for dense spatial coverage of its surface over the course of a motion cycle. Occlusion-free stereo tracking of the landmarks was accomplished with an array of four high-speed video cameras. Customized methods based on motion prediction have been used to track landmark points across video frames. The results have been used to construct accurate, continuous estimates of pinna surface motion. These estimates reveal that the pinna surface is subject to heterogeneous patterns of displacements and velocities within each motion cycle. Experiments with simplified robotic reproductions to understand the acoustic implications of these pinna surface motions are currently in progress. Once the signal transformations that result from the pinna motions are understood, the question of functional relevance can be addressed.

Monocular Stereo Measurement Using High-Speed Catadioptric Tracking.

This paper presents a novel concept of real-time catadioptric stereo tracking using a single ultrafast mirror-drive pan-tilt active vision system that can simultaneously switch between hundreds of different views in a second. By accelerating video-shooting, computation, and actuation at the millisecond-granularity level for time-division multithreaded processing in ultrafast gaze control, the active vision system can function virtually as two or more tracking cameras with different views. It enables a single active vision system to act as virtual left and right pan-tilt cameras that can simultaneously shoot a pair of stereo images for the same object to be observed at arbitrary viewpoints by switching the direction of the mirrors of the active vision system frame by frame. We developed a monocular galvano-mirror-based stereo tracking system that can switch between 500 different views in a second, and it functions as a catadioptric active stereo with left and right pan-tilt tracking cameras that can virtually capture 8-bit color images each operating at 250 fps to mechanically track a fast-moving object with a sufficient parallax for accurate 3D measurement. Several tracking experiments for moving objects in 3D space are described to demonstrate the performance of our monocular stereo tracking system.

Fusion of freehand SPECT and ultrasound: First experience in preoperative localization of sentinel lymph nodes.

Radioguided sentinel lymph node biopsy (SLNB) is the standard of care in breast cancer and melanoma. Additional preoperative Single-photon emission computed tomography (SPECT/CT) for improved anatomical co-registration of the SLNs causes additional radiation exposure and is time-consuming and expensive. The aim of this prospective study was to evaluate a novel approach involving real-time fusion of freehand SPECT (fhSPECT) and ultrasound (US) for anatomical co-registration of SLNs. From February 2015 to February 2016, 153 patients were included in this prospective study. All patients underwent lymphoscintigraphy according to practical guidelines and 151 (118 cases of breast cancer, 30 cutaneous malignancies, and three cases of vulvar cancer) of the 153 patients were additionally investigated with fhSPECT-US. FhSPECT connected to a hand-held gamma detector generates three-dimensional images of the radioactivity distribution in the scanned area. For co-registration and real-time fusion of fhSPECT and subsequently performed US, an infrared stereo tracking system was used. In all patients an SLN was found on lymphoscintigraphy, and the fhSPECT detected corresponding hotspots in all but one patient. In 72% of patients and 73% of lymph node basins, real-time anatomical co-registration with US was feasible. The rate of success in achieving good co-registration increased from 60 to 75% after training by a radiologist specialized in breast imaging. A higher co-registration rate (78%) was observed in patients with only one SLN than in those with two SLNs (68%) or three or more SLNs (0%). Real-time fusion of fhSPECT and US for preoperative anatomical co-registration of SLNs is feasible. However, before this approach can completely replace preoperative lymphatic imaging, further technical developments are needed.

Catadioptric Stereo Tracking for Three Dimensional Shape Measurement of MRI Guided Catheters.

The recent introduction of Magnetic Resonance Imaging (MRI)-actuated steerable catheters lays the ground work for increasing the efficacy of cardiac catheter procedures. The MRI, while capable of imaging the catheter for tracking and control, does not fulfill all of the needs required to identify and develop a complete catheter model. Specifially, the frequency response of the catheter must be identified to ensure stable control of the catheter system. This requires a higher frequency imaging than the MRI can achieve. This work uses a catadioptric stereo camera system consisting of a mirror and a single camera in order to track a MRI actuated catheter inside a MRI machine. The catadioptric system works in parallel to the MRI and is capable of recording the catheter at 60 fps for post processing. The accuracy of the catadioptric system is verified in imaging conditions that would be found inside the MRI. The stereo camera is then used to track a catheter as it is actuated inside the MRI.

PRECISE HEAD TRACKING IN HEARING APPLICATIONS

Abstract. The paper gives an overview about two research projects, both dealing with optical head tracking in hearing applications. As part of the project “Development of a real-time low-cost tracking system for medical and audiological problems (ELCoT)” a cost-effective single camera 3D tracking system has been developed which enables the detection of arm and head movements of human patients. Amongst others, the measuring system is designed for a new hearing test (based on the “Mainzer Kindertisch”), which analyzes the directional hearing capabilities of children in cooperation with the research project ERKI (Evaluation of acoustic sound source localization for children). As part of the research project framework “Hearing in everyday life (HALLO)” a stereo tracking system is being used for analyzing the head movement of human patients during complex acoustic events. Together with the consideration of biosignals like skin conductance the speech comprehension and listening effort of persons with reduced hearing ability, especially in situations with background noise, is evaluated. For both projects the system design, accuracy aspects and results of practical tests are discussed.

Toward real-time endoscopically-guided robotic navigation based on a 3D virtual surgical field model.

The challenge is to accurately guide the surgical tool within the three-dimensional (3D) surgical field for robotically-assisted operations such as tumor margin removal from a debulked brain tumor cavity. The proposed technique is 3D image-guided surgical navigation based on matching intraoperative video frames to a 3D virtual model of the surgical field. A small laser-scanning endoscopic camera was attached to a mock minimally-invasive surgical tool that was manipulated toward a region of interest (residual tumor) within a phantom of a debulked brain tumor. Video frames from the endoscope provided features that were matched to the 3D virtual model, which were reconstructed earlier by raster scanning over the surgical field. Camera pose (position and orientation) is recovered by implementing a constrained bundle adjustment algorithm. Navigational error during the approach to fluorescence target (residual tumor) is determined by comparing the calculated camera pose to the measured camera pose using a micro-positioning stage. From these preliminary results, computation efficiency of the algorithm in MATLAB code is near real-time (2.5 sec for each estimation of pose), which can be improved by implementation in C++. Error analysis produced 3-mm distance error and 2.5 degree of orientation error on average. The sources of these errors come from 1) inaccuracy of the 3D virtual model, generated on a calibrated RAVEN robotic platform with stereo tracking; 2) inaccuracy of endoscope intrinsic parameters, such as focal length; and 3) any endoscopic image distortion from scanning irregularities. This work demonstrates feasibility of micro-camera 3D guidance of a robotic surgical tool.

Augmented Reality Navigation With Automatic Marker-Free Image Registration Using 3-D Image Overlay for Dental Surgery

Computer-assisted oral and maxillofacial surgery (OMS) has been rapidly evolving since the last decade. State-of-the-art surgical navigation in OMS still suffers from bulky tracking sensors, troublesome image registration procedures, patient movement, loss of depth perception in visual guidance, and low navigation accuracy. We present an augmented reality navigation system with automatic marker-free image registration using 3-D image overlay and stereo tracking for dental surgery. A customized stereo camera is designed to track both the patient and instrument. Image registration is performed by patient tracking and real-time 3-D contour matching, without requiring any fiducial and reference markers. Real-time autostereoscopic 3-D imaging is implemented with the help of a consumer-level graphics processing unit. The resulting 3-D image of the patient's anatomy is overlaid on the surgical site by a half-silvered mirror using image registration and IP-camera registration to guide the surgeon by exposing hidden critical structures. The 3-D image of the surgical instrument is also overlaid over the real one for an augmented display. The 3-D images present both stereo and motion parallax from which depth perception can be obtained. Experiments were performed to evaluate various aspects of the system; the overall image overlay error of the proposed system was 0.71mm.

The Research and Application of Artificial Intelligence in the Field of Intelligent Robot

The research and application of artificial intelligence has a very wide range in intelligent robot field. Intelligent robot can not only make use of artificial intelligence gain access to external data, information, (such as stereo vision system, face recognition and tracking, etc.), and then deal with it so as to exactly describe external environment, and complete a task independently, owing the ability of learning knowledge, but also have self-many kinds of artificial intelligence like judgment and decision making, processing capacity and so on. It can make corresponding decision according to environmental changes. Its application range is expanding. In deep sea exploration, star exploration, mineral exploration, heavy pollution, domestic service, entertainment clubs, health care and so on, the figure of intelligent robots artificial intelligence application can all be seen.

Stereo Vision for Bus Traffic Conflict Investigation

In this paper an imaging system for bus traffic conflict investigation is presented. The system employs geo-referenced stereo sequences and tracking procedure to provide much greater information regarding pre-crash and crash events than what is currently available. The exploitation of the traffic conflict technique potentiality as a surrogate safety measure could constitute an effective tool in understanding how the driver interacts and adapts its behavior with respect to the vehicle, the road characteristics, the traffic control devices and the environment. Experiments performed on real data acquired in urban context confirm the effectiveness of the system for the traffic conflicts measurement and the driver behavior analysis.

Stereo Visual Tracking Within Structured Environments for Measuring Vehicle Speed

We present a novel visual tracking method for measuring the speed of a moving vehicle within a structured environment using stationary stereo cameras. In the proposed method, visual stereo tracking and motion estimation in 3-D are integrated within the framework of particle filtering. The visual tracking processes in the two views are coupled with each other since they are dependent upon the same 3-D motion and correlated in the observations. Considering that the vehicle's motion is physically constrained by the environment, we further utilize the path constraint reconstructed from stereo views to reduce the uncertainty about the vehicle's motion and improve the accuracy for both tracking and speed measuring. The proposed method overcomes the challenges arising from the limitation of depth accuracy in a long-range stereo, and the experiments on the synthetic and real-world sequences have demonstrated its effectiveness and accuracy in both the tracking performance and the speed measurement.

A Two-Stage Assignment-Based Algorithm for Asynchronous Multisensor Bearings-Only Tracking

We present a two-stage centralized algorithm for tracking multiple targets using spatially distributed bearings-only sensors that report the observations asynchronously. The sensors are assumed to be passive, i.e., they detect the energy emitted by the targets of interest to measure the bearing. The number of targets in the surveillance region is unknown a priori and the targets can enter or leave the surveillance region at any time. The measurement origin is also unknown since a detection can be due to the target being tracked, from a new target, or from clutter. In the first stage (the initialization stage) the proposed algorithm forms local bearings-only (mono) tracks for each sensor and combines these tracks to generate complete kinematic (stereo) tracks in the Cartesian coordinate frame. Once stereo tracks are formed, in the second stage, which is called the stereo tracking stage, bearing measurements are directly used to update the stereo tracks. This separation of the initialization and maintenance of the stereo tracks is lacking in many existing algorithms and results in improved performance. In this work we used the assignment-based technique to solve various data association problems that arise due to measurement origin uncertainty. Through extensive simulations we show that the proposed algorithm achieves better tracking accuracy while being computationally simpler than existing algorithms.

Real-Time 3D Stereo Tracking and Localizing of Spherical Objects with the iCub Robotic Platform

Visual pattern recognition is a basic capability of many species in nature. The skill of visually recognizing and distinguishing different objects in the surrounding environment gives rise to the development of sensory-motor maps in the brain, with the consequent capability of object reaching and manipulation. This paper presents the implementation of a real-time tracking algorithm for following and evaluating the 3D position of a generic spatial object. The key issue of our approach is the development of a new algorithm for pattern recognition in machine vision, the Least Constrained Square-Fitting of Ellipses (LCSE), which improves the state of the art ellipse fitting procedures. It is a robust and direct method for the least-square fitting of ellipses to scattered data. In this work we applied it to the iCub humanoid robotics platform simulator and real robot. We used it as a base for a circular object localization within the 3D surrounding space. We compared its performance with the Hough Transform and the state of the art ellipse fitting algorithms, in terms of robustness (succes/failure in the object detection) and fitting precision. Our experiments involve robustness against noise, occlusion, and computational complexities analyses.