Stereo Research Articles

Infrared simulation of large-scale urban scene through LOD.

The growing use of infrared (IR) imaging systems places increasing demands for simulating infrared images of real scenes. Utilizing images captured from unmanned aerial vehicles (UAV), we propose a semi-automatic pipeline to generate large-scale IR urban scenes in the form of levels of detail (LODs). It significantly reduces the cost of labor and time while providing detailed IR structures. Starting from the surface meshes generated by multi-view stereo (MVS) systems, we produce watertight LODs via semantic segmentation and structure-aware approximation. For each LOD, we divide the surfaces into triangle facets of specific scales. For each facet, one material attribute is attached, and the heat balance equations are solved to obtain the temperature. Three strategies are proposed to accelerate the thermal distribution calculation. Finally, by synthesizing the radiance distribution, the whole IR scenes are generated and rendered. Experiments on real urban scenes show that the proposed pipeline could effectively simulate IR scenes of large-scale urban scenes.

Photometric stereo with quasi-point light source

This paper presents a complete computational framework for the photometric stereo system composed of some LED lamps and a single web-camera. To describe the lighting condition accurately, the angular and distance attenuation factors, as well as the LED radiance property, are considered in the modeling of the light source. The calibration of the system's parameters is a critical step for our setup, and a two-step calibration procedure is proposed in this work. A multiple-sphere-based approach is first used to estimate the light source positions. Then, a reference-plane-based approach is applied to estimate the principal optical axis of each light source and its principal energy. For each surface point on the target, the direction and intensity of its incidental light ray can be precisely determined by the calibration parameters and the quasi-point light model. Based on the initial 3D reconstruction result, an iterative procedure is applied to optimize the model's parameters for each surface point with respect to its initial estimated depth. In the experiments, both regular and free-form surfaces including a real human face are used. The results show that accurate 3D reconstruction results can be obtained by the proposed approach based on the low-cost setup.

Modeling 3D human body with a smart vest

A novel human body reconstruction method based on measuring human body feature curves is proposed in this paper. Feature curves reflecting the cross section shape of human body are obtained by reconstructing the shapes of flexible tapes on an easy-to-operate smart vest that is dressed on the user. These tapes are specially marked to make the length of curves and spatial position on each marker perceptible, when the user rotates in front of a binocular stereo system. Taking these perceived geometric information as the input, precise feature curves are computed by solving a constrained optimization function. Individual human model is then generated by mapping the feature curves onto human shape space based on a multilayer neural network. Comparative experiments prove that our approach provides a low-costs way to reconstruct human models with competitive geometric precision (about 8mm point-to-point distance). Our approach can be potentially applied in many human body related product industries.

High-fidelity facial reflectance and geometry inference from an unconstrained image

We present a deep learning-based technique to infer high-quality facial reflectance and geometry given a single unconstrained image of the subject, which may contain partial occlusions and arbitrary illumination conditions. The reconstructed high-resolution textures, which are generated in only a few seconds, include high-resolution skin surface reflectance maps, representing both the diffuse and specular albedo, and medium- and high-frequency displacement maps, thereby allowing us to render compelling digital avatars under novel lighting conditions. To extract this data, we train our deep neural networks with a high-quality skin reflectance and geometry database created with a state-of-the-art multi-view photometric stereo system using polarized gradient illumination. Given the raw facial texture map extracted from the input image, our neural networks synthesize complete reflectance and displacement maps, as well as complete missing regions caused by occlusions. The completed textures exhibit consistent quality throughout the face due to our network architecture, which propagates texture features from the visible region, resulting in high-fidelity details that are consistent with those seen in visible regions. We describe how this highly underconstrained problem is made tractable by dividing the full inference into smaller tasks, which are addressed by dedicated neural networks. We demonstrate the effectiveness of our network design with robust texture completion from images of faces that are largely occluded. With the inferred reflectance and geometry data, we demonstrate the rendering of high-fidelity 3D avatars from a variety of subjects captured under different lighting conditions. In addition, we perform evaluations demonstrating that our method can infer plausible facial reflectance and geometric details comparable to those obtained from high-end capture devices, and outperform alternative approaches that require only a single unconstrained input image.

A data-driven photometric stereo algorithm

Representing the complicated reflectance of non-Lambertian material is one of the difficulty in photometric stereo. This paper presents a Gaussian process based data-driven photometric stereo method to address this problem. The method makes use of Gaussian process, a non-parametric Bayesian inference model, to establish a global plausible model to represent non-linear reflectance property of non-Lambertian material based on measured bidirectional reflectance distribution functions. The statistic nature of the Gaussian process makes the method has much better flexibility and accuracy in describing the complicated reflectance of various non-Lambterian materials so that a better estimation accuracy can be achieved for a photometric stereo system. The validity of the proposed method is verified by both computer simulation and real experiment. The results show that the proposed method has much better accuracy and efficiency in estimating surface normal than other existing methods.

How Much Stereoscopic Effect Does Laparoscopy Need? Controlled, Prospective Randomized Trial on Surgical Task Efficiency in Standardized Phantom Tasks.

To regain 2-eyed vision in laparoscopy, dual-channeled optics have been introduced. With this optics design, the distance between the 2 front lenses defines how much stereoscopic effect is seen. This study quantifies the impact of an enhanced and a reduced stereo effect on surgical task efficiency. A prospective single-blinded study was performed with 20 laparoscopic novices in an inanimate experimental setting. A standard bichannelled stereo system was used to perform a suturing and knotting task. The working distance and the task size were scaled to vary the stereo effect and, thereby, simulate hypothetic stereo optics with enhanced and reduced optical bases. The task performances were timed, and the number of trials for stitching out was counted. The participants finally filled out a questionnaire to collect subjective impressions. The increase of the stereo effect by 50% caused no objective improvement in laparoscopic knotting compared with typical 3D (control group with stereo basis of 4.5 mm). But ergonomic disadvantages (headache) were subjectively reported in 1 of 20 cases in the questionnaire. The reduction of the stereo effect by one-third led to a significantly longer average execution time. There was no significant dependence found between stereo effect and number of stich-out trials, stitching precision, or knotting quality. Considering laparoscopy, it does not seem advisable to enhance the stereo effect because of ergonomic problems. Otherwise, a miniaturization of the 3D scope (5 mm version) is problematic because its benefit mostly shrinks with the reduced stereo effect.

Meet the Editors: Dietrich A. Volmer

Meet the Editors: Dietrich A. Volmer

A hierarchical stereo matching algorithm based on adaptive support region aggregation method

Stereo matching is a fundamental process in many application fields. An accurate depth information is useful for stereo systems to separate occluding image components. In the conception of stereo matching algorithms, various works rely on the Census Transform (CT) as a cost computation step, due to its robustness against radiometric changes. In this paper, we propose a new variant of the CT cost function which incorporates edge side information for further compensating radiometric changes. We demonstrate that the proposed variant matching cost improves significantly the quality of the disparity results. In addition, an aggregation method, based on the adaptive support region, is implemented into a hierarchical fusion processing scheme, by incorporating low frequency information into high frequency using a robust exponential function. This allows a multi-scale interaction within the adaptive cost aggregation. Hence, significant erroneous disparities are reduced, especially in textureless regions. Experiments were conducted on the KITTI benchmark and the obtained results have shown the potential merits of the proposed framework.

THE THREE-DIMENSIONAL FLOW SIMULATION OF A FLAPPING WING

This study investigates three-dimensional (3D) flow fields around a flapping wing with a span of 20 cm. Numerical simulations are performed by varying the simulation mesh density for a half-flapping-wing profile with wind tunnel domain for flapping stroke angles of 53° and 90°. Wind velocities ranging from 1 to 3 m/s, variation in flapping frequency of 14-15 Hz, and corresponding lift characteristics are obtained. Comparative evaluation of numerical and experimental lift force data reveals the translational and rotational lift phenomena of Dickinson’s mechanisms. The stream line that flows over the 3D wing profile is found to be in good agreement with the existing smoke trace experimental data. The motions of a flapping wing obtained through 3D stereo photography and COMSOL Multiphysics simulation are compared and the aerodynamic characteristics are exploited. The streamwise and spanwise 3D velocity fields with intercepting planes at three chord sections are explored, and the reverse Karman vortex sheet is observed. The comparison of the simulation and experimental results for a 3D flapping wing provides insight into the motion characteristics of a flapping wing and flapping wing aerodynamic analysis of micro air vehicles.

Wide-Baseline Stereo-Based Obstacle Mapping for Unmanned Surface Vehicles.

This paper proposes a wide-baseline stereo-based static obstacle mapping approach for unmanned surface vehicles (USVs). The proposed approach eliminates the complicated calibration work and the bulky rig in our previous binocular stereo system, and raises the ranging ability from 500 to 1000 m with a even larger baseline obtained from the motion of USVs. Integrating a monocular camera with GPS and compass information in this proposed system, the world locations of the detected static obstacles are reconstructed while the USV is traveling, and an obstacle map is then built. To achieve more accurate and robust performance, multiple pairs of frames are leveraged to synthesize the final reconstruction results in a weighting model. Experimental results based on our own dataset demonstrate the high efficiency of our system. To the best of our knowledge, we are the first to address the task of wide-baseline stereo-based obstacle mapping in a maritime environment.

Long-term clinical effect of aspheric multifocal intraocular lens implantation for developmental cataracts in childhood

Objective To observe the long-term visual quality and clinical effect of aspheric diffractive multifocal intraocular lens (IOL) implantation for congenital and developmental cataracts in childhood. Methods A retrospective cohort study on multifocal IOL implantation for congenital and developmental cataracts in childhood (aged 8 to 14 years) was performed in the First Affiliated Hospital of Zhengzhou University from August 2013 to January 2015.The clinical data of 67 eyes from 46 congenital cataract patients who received phacoemulsification with IOL implantation were collected.The AMO (ZMB00) IOL was implanted in 34 eyes of 24 patients in the multifocal IOL group, and Bausch & Lomb (MI60) IOL was implanted in 33 eyes of 22 patients in the monofocal IOL group.The distance, intermediate and near vision acuity were analyzed in 3, 6 and 12 months after surgery, including uncorrected visual acuity (UCVA) and best corrected visual acuity (BCVA). The wavefront aberrations, modulation transfer functions (MTF) and stereopsis were obtained with iTrace Analysis System, Optec-6500 Visual Functional Analyzer and Titmus Stereo Test Chart, respectively in 12 months after surgery.The near additional degree, removing glasses rate and myopic shift were compared between the two groups in postoperative 12 months. Results The intermediate and near UCVA in the multifocal IOL group were obviously better than those in the monofocal IOL group 3, 6 and 12 months after surgery (intermediate: Z=-3.74, -4.36, -3.66; all at P=0.00.near: Z=-2.67, -2.50, -2.33; all at P 0.05). The MTFs under the 5.0 mm optical zone and 5, 10, 15, 20, 25, 30 c/d in the multifocal IOL group were insignificantly lower than those in the monofocal IOL group (all at P>0.05). In addition, the near stereopsis, near additional degree and myopic shift (ΔD) were reduced in the multifocal IOL group compared with monofocal IOL group (both at P 0.05). The myopic shifting rates were 16.7% and 83.3% in the multifocal IOL group and monofocal IOL group, with a significant difference between the two groups (χ2=15.02, P=0.00). Conclusions The aspheric multifical IOL implantation can achieve good and stable distance, intermediate and near visual acuities, provide better near stereopsis, reduce postoperative dependence on spectacles and decrease the incidence of myopic shift in child cataract patients. Key words: Congenital cataract; Intraocular lens; Prosthesis implantation; Aspheric; Multifocal; Wavefront aberration; Modulation transfer function; Childhood

Design and test of robotic harvesting system for cherry tomato

Harvesting of fresh-eating cherry tomato was highly costly on labor and time. In order to achieve mechanical harvesting for the fresh-eating tomato, a new harvesting robot was designed, which consisted of a stereo visual unit, an end-effector, manipulator, a fruit collector, and a railed vehicle. The robot configuration and workflow design focused on the special cultivating condition. Three key parts were introduced in detail: a railroad vehicle capably moving on both ground and rail was adopted as the robot’s carrier, a visual servo unit was used to identify and locate the mature fruits bunch, and the end-effector to hold and separate the fruit bunch was designed based on the stalk’s mechanical features. The field test of the new developed robot was conducted and the results were analyzed. The successful harvest rate of the robot was 83%, however, each successful harvest averagely needed 1.4 times attempt, and a single successful harvesting cycle cost 8 s excluding the time cost on moving. Keywords: cherry tomato, harvesting robot, visual servo, configuration design, field test DOI: 10.25165/j.ijabe.20181101.2853 Citation: Feng Q C, Zou W, Fan P F, Zhang C F, Wang X. Design and test of robotic harvesting system for cherry tomato. Int J Agric & Biol Eng, 2018; 11(1): 96–100.

A multimodal 3D framework for fire characteristics estimation

In the last decade we have witnessed an increasing interest in using computer vision and image processing in forest fire research. Image processing techniques have been successfully used in different fire analysis areas such as early detection, monitoring, modeling and fire front characteristics estimation. While the majority of the work deals with the use of 2D visible spectrum images, recent work has introduced the use of 3D vision in this field. This work proposes a new multimodal vision framework permitting the extraction of the three-dimensional geometrical characteristics of fires captured by multiple 3D vision systems. The 3D system is a multispectral stereo system operating in both the visible and near-infrared (NIR) spectral bands. The framework supports the use of multiple stereo pairs positioned so as to capture complementary views of the fire front during its propagation. Multimodal registration is conducted using the captured views in order to build a complete 3D model of the fire front. The registration process is achieved using multisensory fusion based on visual data (2D and NIR images), GPS positions and IMU inertial data. Experiments were conducted outdoors in order to show the performance of the proposed framework. The obtained results are promising and show the potential of using the proposed framework in operational scenarios for wildland fire research and as a decision management system in fighting.

Effective Data-Driven Calibration for a Galvanometric Laser Scanning System Using Binocular Stereo Vision.

A new solution to the problem of galvanometric laser scanning (GLS) system calibration is presented. Under the machine learning framework, we build a single-hidden layer feedforward neural network (SLFN）to represent the GLS system, which takes the digital control signal at the drives of the GLS system as input and the space vector of the corresponding outgoing laser beam as output. The training data set is obtained with the aid of a moving mechanism and a binocular stereo system. The parameters of the SLFN are efficiently solved in a closed form by using extreme learning machine (ELM). By quantitatively analyzing the regression precision with respective to the number of hidden neurons in the SLFN, we demonstrate that the proper number of hidden neurons can be safely chosen from a broad interval to guarantee good generalization performance. Compared to the traditional model-driven calibration, the proposed calibration method does not need a complex modeling process and is more accurate and stable. As the output of the network is the space vectors of the outgoing laser beams, it costs much less training time and can provide a uniform solution to both laser projection and 3D-reconstruction, in contrast with the existing data-driven calibration method which only works for the laser triangulation problem. Calibration experiment, projection experiment and 3D reconstruction experiment are respectively conducted to test the proposed method, and good results are obtained.

Calibration Comparison of Single Camera and Stereo Camera

Calibration Comparison of Single Camera and Stereo Camera

Semi-automatic Extraction of 3D Building Boundary Using DSM from Stereo Images Matching

Semi-automatic Extraction of 3D Building Boundary Using DSM from Stereo Images Matching

A One-Shot FPGA Based Depth Acquisition System

This paper introduces a field-programmable gate array (FPGA)-based one-shot depth acquisition system, which is a kind of Kinect-like Depth sensing. The proposed implementation integrates the random binary pattern with an improved matching method, which is able to obtain dense and precise depth maps. Scene images captured by a camera are transferred to FPGA (Xilinx Virtex7 XC7VX690T-2FFG1761C) through a CamerLink interface, and the acquired depth results are sent to host computer through a high-speed USB3.0 interface. An elaborate FPGA architecture is designed to realize efficient and accurate calculations. The proposed system outperforms the existing stereo systems, not only on the quality of depth acquisition, but also on the reconstruction speed for which our system achieves 90 fps at a resolution of $1280\times960$ . Besides, Kinect-like depth sensing has an inherent advantage in deployments, much easier than other kinds of systems. Our proposed system has extensive applications not only in the field of vision processing, but also in the fields where block matching is needed.

A Stereo Approach to Wildfire Smoke Detection: The Improvement of the Existing Methods by Adding a New Dimension

In this paper, we present a novel approach to visual smoke detection based on stereo vision. General smoke detection is usually performed by analyzing the images from remote cameras using various computer vision techniques. The literature on smoke detection shows a variety of approaches, and the focus of this paper is the improvement of the general smoke detection process by introducing stereo vision. Two cameras are used to estimate the distance and size of the detected phenomena based on stereo triangulation. Using this information, the minimum size and overall dynamics of the detected regions are further examined to ensure the elimination of false alarms induced by various phenomena (such as the movement of objects located at short distances from the camera). Such false alarms could easily be detected by the proposed stereo system, allowing the increase of the sensitivity and overall performance of the detection. We analyzed the requirements of such system in terms of precision and robustness to possible error sources, especially when dealing with detection of smoke at various distances from the camera. For evaluation, three existing smoke detection methods were tested and the results were compared to their newly implemented stereo versions. The results demonstrated better overall performance, especially a decrease in false alarm rates for all tested methods.

Experiments on mobile robot stereo vision system calibration under hardware imperfection

Calibration is essential for any robot vision system for achieving high accuracy in deriving objects metric information. One of typical requirements for a stereo vison system in order to obtain better calibration results is to guarantee that both cameras keep the same vertical level. However, cameras may be displaced due to severe conditions of a robot operating or some other circumstances. This paper presents our experimental approach to the problem of a mobile robot stereo vision system calibration under a hardware imperfection. In our experiments, we used crawler-type mobile robot «Servosila Engineer». Stereo system cameras of the robot were displaced relative to each other, causing loss of surrounding environment information. We implemented and verified checkerboard and circle grid based calibration methods. The two methods comparison demonstrated that a circle grid based calibration should be preferred over a classical checkerboard calibration approach.

Metode vizuelnog upravljanja robotskih sistema - pregled stanja u oblasti istraživanja

This paper surveys the methods used for visual servoing of robotic systems, where the main focus is on mobile robot systems. The three main areas of research include the Direct Visual Servoing, stereo vision systems, and artificial intelligence in visual servoing. The standard methods such as Image-Based Visual Servoing (IBVS) and Position-Based Visual Servoing (PBVS) are analyzed and compared with the new method named Direct Visual Servoing (DVS). DVS methods have better accuracy, compared to IBVS and PBVS, but have limited convergence area. Because of their high accuracy, DVS methods are suitable for integration into hybrid systems. Furthermore, the use of the stereo systems for visual servoing is comprehensively analyzed. The main contribution of the stereo system is the accurate depth estimation, which is critical for many visual servoing tasks. The use of artificial intelligence (AI) in visual servoing purposes has also gained popularity over the years. AI techniques give visual servoing controllers the ability to learn by using predefined examples or empirical knowledge. The learning ability is crucial for the implementation of robotic systems in a real-world dynamic manufacturing environment. Also, we analyzed the use of visual odometry in combination with a visual servoing controller for creating more robust and reliable positioning system.