Stereo Vision Research Articles

Measuring 3D Video Quality of Experience (QoE) Using A Hybrid Metric Based on Spatial Resolution and Depth Cues.

A three-dimensional (3D) video is a special video representation with an artificial stereoscopic vision effect that increases the depth perception of the viewers. The quality of a 3D video is generally measured based on the similarity to stereoscopic vision obtained with the human vision system (HVS). The reason for the usage of these high-cost and time-consuming subjective tests is due to the lack of an objective video Quality of Experience (QoE) evaluation method that models the HVS. In this paper, we propose a hybrid 3D-video QoE evaluation method based on spatial resolution associated with depth cues (i.e., motion information, blurriness, retinal-image size, and convergence). The proposed method successfully models the HVS by considering the 3D video parameters that directly affect depth perception, which is the most important element of stereoscopic vision. Experimental results show that the measurement of the 3D-video QoE by the proposed hybrid method outperforms the widely used existing methods. It is also found that the proposed method has a high correlation with the HVS. Consequently, the results suggest that the proposed hybrid method can be conveniently utilized for the 3D-video QoE evaluation, especially in real-time applications.

Design and calibration of the 6-DOF motion tracking system integrated on the Stewart parallel manipulator.

Accurate pose measurement is crucial for parallel manipulators (PM). This study designs a novel integrated 6-DOF motion tracking system to achieve precise online pose measurement. However, the presence of geometric errors introduces imperfections in the accuracy of the measured pose. Based on the displacement information of six grating rulers, measurement pose is obtained through forward kinematics. By comparing the measurement results with the actual pose information captured by stereo vision, measurement errors can be obtained. A closed-loop vector-based kinematic model and an error model are established, and then the geometric errors are identified with the least-squares method. Finally, the geometric calibration experiments are conducted, and the results show that the measurement accuracy has significantly improved, with the average position error decreasing from 3.148 mm to 0.036 mm, and the average orientation error is decreased from 0.225° to 0.022°.

Research on the vision system of lychee picking robot based on stereo vision

In this study, we propose a development scheme of litchi robot vision system based on stereo vision. The system includes three components: image acquisition, image analysis, and image processing. The obtained image can be corrected according to the light environment conditions with the corresponding algorithm for image color, calculate the Hue color layer, analyze the external characteristics and positional data (2D) of litchi, obtain the 3D coordinate information of fruits and vegetables, and provide to the running device for route planning and vegetable and fruit picking claw tools for picking. The integration test results for the robot arm and claw tool demonstrate the following: (1) Using image acquisition, the system can acquire high-quality images in real-time via a combination of automatic exposure and white balance conversion. (2) Using image analysis techniques such as background separation, edge detection, and external feature search, it is possible to successfully calculate the location coordinates of the litchi's tip, center, and bottom. (3) Using image processing, the claw of the picking machine can determine the condition of overlapping lychees and their state of ripening based on the depth information of feature points used in the picking operation. (4) Under the working range of 18.4 cm × 33.6 cm, the success rate of picking litchi with a diameter of 28 mm, 34 mm, and 42 mm was 93.3%, 91.1 %, and 84.4%, respectively, exceeding the design expectation level.

3D POINT CLOUD COMPLETION USING TERRAIN-CONTINUOUS CONSTRAINTS AND DISTANCE-WEIGHTED INTERPOLATION FOR LUNAR TOPOGRAPHIC MAPPING

Abstract. Stereo vision has been proven to be an efficient tool for 3D reconstruction in lunar topographic mapping. However, point clouds reconstructed from pairs of stereo images always suffer from occlusions and illumination changes, especially in Lunar environments, resulting in incomplete geometric information. 3D point cloud completion is usually required for refining photogrammetric point clouds and enabling further applications. In this work, we address the problem of completing and refining 3D photogrammetric point clouds based on the assumption that 3D terrain should be continuous and with consistent slope change. We proposed a generalized strategy for 3D point cloud completion of lunar topographic mapping, including distance-weighted point cloud interpolation, terrian-continuous constrained outlier detection, and contour-based hole filling. We carried out experiments on two datasets of point clouds generated from 12 pairs and 6 pairs of stereo LROC NAC images covering the Apollo 17 and the Chang’E-4 landing sites, respectively. As a result, the holes in the initial DTM have been smoothly filled and the completeness of the whole DTM has been greatly improved. The incomplete area of the experimental areas has dropped by 100% and 93%, respectively. Finally, we constructed DTM with a resolution of 10 m covering a 33 km × 60 km area of the Apollo 17 landing site with RMSE of 4 m and a 12 km × 56 km area of Chang’E-4 landing site with RMSE of 4 m compared with LOLA laser points as a reference.

Analysis of the Teaching Path of Animation Film Majors Integrating Digital Visual Space in the Context of Digital Media Art

Abstract In this study, we first adopt the dynamic stereo vision spatial coordinate measurement method of non-zoom cameras, establish a coordinate measurement model based on differential GPS, and study the application of the spatial coordinates of dynamic stereo digital vision systems in animation design. Secondly, the internal and external parameters of the camera are defined as the static error of the dynamic stereo vision system so as to provide an easy and quick-to-use design tool for the teaching of animation film majors. Finally, a test sequence animation sample set is used to verify the teaching effect of digital visual space based on animation design. The results show that in the multi-comparison test of animation quality against pixel bit rate, the animation design samples were found to have frame rates in the range of 5-30 Fps. And under the condition of having extra bandwidth, the animation feature learning evaluation dimension is 2589, and increasing the animation visual frame rate can improve the output picture quality of animation more effectively. It shows that the integration of digital visual space in the context of digital media art can allow students to have enough project sources for learning and practicing so as to improve their professional ability.

A novel three-dimensional surface reconstruction method for fabric appearance evaluation

Traditional fabric appearance evaluation method mainly relies on subjective and manual operation, which is not uniform and time-consuming. In this paper, one set of self-developed 3D reconstruction system and method based on photometric stereo is proposed for objective fabric appearance evaluation. A photometric stereo image acquisition system equipped with multi-directional lighting sources is developed for the 3D surface reconstruction. Lighting and camera calibration is conducted to obtain the position of light sources and the intrisic parameters of digital camera. Our experimental results show that 3D fabric surface could be reconstructed effectively based on photometric stereo vision method, it has the potential to be used for fabric appearance evaluation.

Light Field Display Based on Monocular Stereo Vision

Light Field Display Based on Monocular Stereo Vision

Energy-Efficient ReS2-Based Optoelectronic Synapse for 3D Object Reconstruction and Recognition.

The neuromorphic vision system (NVS) equipped with optoelectronic synapses integrates perception, storage, and processing and is expected to address the issues of traditional machine vision. However, owing to their lack of stereo vision, existing NVSs focus on 2D image processing, which makes it difficult to solve problems such as spatial cognition errors and low-precision interpretation. Consequently, inspired by the human visual system, an NVS with stereo vision is developed to achieve 3D object recognition, depending on the prepared ReS2 optoelectronic synapse with 12.12 fJ ultralow power consumption. This device exhibits excellent optical synaptic plasticity derived from the persistent photoconductivity effect. As the cornerstone for 3D vision, color planar information is successfully discriminated and stored in situ at the sensor end, benefiting from its wavelength-dependent plasticity in the visible region. Importantly, the dependence of the channel conductance on the target distance is experimentally revealed, implying that the structure information on the object can be directly captured and stored by the synapse. The 3D image of the object is successfully reconstructed via fusion of its planar and depth images. Therefore, the proposed 3D-NVS based on ReS2 synapses for 3D objects achieves a recognition accuracy of 97.0%, which is much higher than that for 2D objects (32.6%), demonstrating its strong ability to prevent 2D-photo spoofing in applications such as face payment, entrance guard systems, and others.

Full field-of-view virtual reality goggles for mice

Visual virtual reality (VR) systems for head-fixed mice offer advantages over real-world studies for investigating the neural circuitry underlying behavior. However, current VR approaches do not fully cover the visual field of view of mice, do not stereoscopically illuminate the binocular zone, and leave the lab frame visible. To overcome these limitations, we developed iMRSIV (Miniature Rodent Stereo Illumination VR)-VR goggles for mice. Our system is compact, separately illuminates each eye for stereo vision, and provides each eye with an ∼180° field of view, thus excluding the lab frame while accommodating saccades. Mice using iMRSIV while navigating engaged in virtual behaviors more quickly than in a current monitor-based system and displayed freezing and fleeing reactions to overhead looming stimulation. Using iMRSIV with two-photon functional imaging, we found large populations of hippocampal place cells during virtual navigation, global remapping during environment changes, and unique responses of place cell ensembles to overhead looming stimulation.

Comparison-embedded evidence-CNN model for fuzzy assessment of wear severity using multi-dimensional surface images

AbstractWear topography is a significant indicator of tribological behavior for the inspection of machine health conditions. An intelligent in-suit wear assessment method for random topography is here proposed. Three-dimension (3D) topography is employed to address the uncertainties in wear evaluation. Initially, 3D topography reconstruction from a worn surface is accomplished with photometric stereo vision (PSV). Then, the wear features are identified by a contrastive learning-based extraction network (WSFE-Net) including the relative and temporal prior knowledge of wear mechanisms. Furthermore, the typical wear degrees including mild, moderate, and severe are evaluated by a wear severity assessment network (WSA-Net) for the probability and its associated uncertainty based on subjective logic. By integrating the evidence information from 2D and 3D-damage surfaces with Dempster–Shafer (D–S) evidence, the uncertainty of severity assessment results is further reduced. The proposed model could constrain the uncertainty below 0.066 in the wear degree evaluation of a continuous wear experiment, which reflects the high credibility of the evaluation result.

Optical Flow-Based Stereo Visual Odometry With Dynamic Object Detection

Autonomous vehicles (AVs) play an important role in the next-generation intelligent transportation system (ITS), which requires AVs to have the ability of rapid decision-making and control. As a part of ITS, simultaneous localization and mapping (SLAM) technology is the basis for AVs to operate in a complex and unknown environment. In this article, a stereo vision odometry method in a dynamic environment is proposed, which can not only effectively overcome the influence of dynamic objects but also detect the position of dynamic objects. An optical flow filtering algorithm based on the quantitative histogram (QH) and the optical flow angle histogram (OFAH) of the feature points is proposed to obtain dynamic points. Furthermore, a multifeature fusion mechanism is used to perform binary segmentation and get the bounding box of dynamic object. Experiments show that the proposed method can improve the accuracy of pose estimation and detect moving objects in a dynamic environment.

Fine-grained damage detection of cement concrete pavement based on UAV remote sensing image segmentation and stitching

A hybrid model for pavement damage segmentation, stitching, and detection based on unmanned aerial vehicle (UAV) remote-sensing pavement images was proposed as high-resolution panoramic images are required to analyze pavement damage feature parameters during the inspection process. Based on target recognition, active stereo vision was used to determine the autonomy awareness of the UAV system in the 3D structure. Then, incremental search strategy was employed to triangulate scale-invariant feature transform (SIFT) point features based on geometric constraints. Perspective-n-point and random sample consensus (PNP-RANSAC) algorithm were combined to find the best matching surface for mis-matching rejection, and the trigonometric function theory was used to fuse images. Subsequently, the semantic segmentation framework based on deep learning context encoder network (CE-Net) was employed to train suitable models to simultaneously detect and segment pavement damages. Finally, the stitching technique and hue, saturation, value (HSV) segmentation technique were combined to obtain the feature parameters to be detected. This method achieved fast and accurate stitching, feature parameter segmentation, and pavement health condition assessment of long-threaded pavement disease image datasets. The splicing accuracy rate and the global feature segmentation overlap rate were greater than 84% and 80%, respectively. The experimental data show that the method is able to detect full-width pavement breakage parameters better than existing optical sensor-based automotive or drone inspection methods.

Fully automatic system for fish biomass estimation based on deep neural network

The approach for estimating biomass in non-contact, free-swimming fish has encountered difficulties such as fish body occlusion, bending, non-orthogonal angles, and low efficiency. To address these issues, this study had combined fish posture recognition (using deep learning technology) with biomass estimation (utilizing stereo vision technology) for the first time, and developed a fast, precise, and fully automatic fish biomass estimation system. The improved single-stage target detection algorithm significantly improved the direct detection and extraction of high-quality images of moving fish in real-time, eliminating the need for manual processing of images that may have imperfect posture. Fish body length and height were measured in the real world using binocular stereo vision technology. Finally, the fish body weight can be estimated by considering the relationship among their body length, height, and weight. The experiment confirmed that the system had successfully avoided influencing factors that could affect fully automatic estimation. The results demonstrated a strong linear relationship between the estimated and measured fish body weights, with a mean relative error (MRE) of 2.87%. There were no significant differences between the estimated and measured weights (p = 0.94). The MRE of the multi-factor model was much lower than that of the single-factor model (length-weight of 8.86% and height-weight of 7.41%). The results indicate that the system developed is a highly effective approach to fully automated biomass estimation. This can be used to guide actual production and further study of the mechanism of fish growth.

A stereo vision SLAM with moving vehicles tracking in outdoor environment

A stereo vision SLAM with moving vehicles tracking in outdoor environment

Design and Development of Traditional Elements in Visual Communication Design under the Integration of Virtual Reality Technology

Abstract This paper proposes a method based on visual communication to generate two-dimensional motion graphics that can enhance the perception of depth and three-dimensional structure in stereoscopic vision. Through eye tracking and calibration, gaze points in 3D space are calculated and applied to Chinese traditional elements in visual communication design. Feature vectors and improved plain Bayesian classifier methods are used to achieve user gesture recognition in virtual design and construct interaction design scenarios based on virtual reality technology. Lastly, the performance impact of augmented reality technology on traditional element design is investigated through a questionnaire survey. Based on the eight samples of conventional elements, the evaluation factors with the highest scores were found to be 10. In the subjective assessment of visual communication design, traditional components fall between mediocre and good, according to the ratings of all 18 assessment factors, which range from 0.2 to 1. The study presented in this paper effectively integrates virtual reality technology into the graphical design of traditional elements, significantly enhancing their visual expression.

A markless 3D human motion data acquisition method based on the binocular stereo vision and lightweight open pose algorithm

This study aimed to achieve precise 3D human motion data acquisition without the need for markers. Specifically, the proposed system utilized two affordable cameras to capture human motion from diverse angles. It developed a lightweight Open Pose algorithm to extract 2D human body joints from recorded videos. Subsequently, a stereo-matching method was applied to derive accurate 3D human body joint data. Comprehensive experiments with 10 healthy participants showcased the system's exceptional performance, demonstrating superior measurement accuracy compared to established standards like the NOKOV system (RMSE values consistently below 9°) and commonly used Microsoft Kinect devices (V2 and Azure). This innovative binocular vision system presents a new approach to motion capture, potentially offering a cost-effective and precise tool for practical applications across various scenarios (e.g., motor functional assessment, virtual avatar streaming), contributing significantly to both scientific and academic communities.

How Do Most Young Moderate Hyperopes Avoid Strabismus?

Uncorrected hyperopic children must overcome an apparent conflict between accommodation and vergence demands to focus and align their retinal images. This study tested hypotheses about simultaneous accommodation and vergence performance of young hyperopes to gain insight into ocular motor strategies used to maintain eye alignment. Simultaneous eccentric photorefraction and Purkinje image tracking were used to assess accommodative and vergence responses of 26 adult emmetropes (AE) and 94 children (0-13 years) viewing cartoons. Children were habitually uncorrected (CU) (spherical equivalent refractive error [SE] -0.5 to +4 D), corrected and aligned (CCA), or corrected with a history of refractive esotropia (CCS). Accommodative and vergence accuracy, dissociated heterophoria, and vergence/accommodation ratios in the absence of retinal disparity cues were measured for 33- and 80-cm viewing distances. In binocular viewing, median accommodative lags for 33 cm were 1.0 D (AE), 1.33 D (CU), 1.25 D (CCA), and 1.0 D (CCS). Median exophorias at 80 and 33 cm were 1.2 and 4.5 pd (AE), 0.8 and 2.5 pd (CU), and 0 and 1.2 pd (CCA), respectively. Without disparity cues, most response vergence/accommodation ratios were between 1 and 2 meter angle/D (∼5-10 pd/D) (69% of AE, 44% of CU, 60% of CCA, and 50% of CCS). Despite apparent conflict in motor coupling, uncorrected hyperopes were typically exophoric and achieved adultlike accuracy of both vergence and accommodation simultaneously, indicating ability to compensate for conflicting demands rather than bias to accurate vergence while tolerating inaccurate accommodation. Large lags and esophoria are therefore atypical. This analysis provides normative guidelines for clinicians and a deeper mechanistic understanding of how hyperopes avoid strabismus.

Method for three-dimensional reconstruction of dynamic stereo vision based on line structured light using global optimization

To overcome the limitation of the measurement range of static stereo vision, a three-dimensional (3-D) reconstruction method of dynamic stereo vision based on line structured light for metal surface is presented in this paper. The dynamic stereo vision is mainly composed of binocular cameras and a laser line generator, and moves along a horizontal guide rail. The intrinsic parameters of the cameras and the rotation and translation of the right camera relative to the left camera are previously obtained by Zhang’s method. To obtain the 3-D coordinates of the line structured light fringe in each position, the sub-pixel coordinates of the fringe center are extracted by Steger algorithm, and a joint calibration method between single camera and a laser line generator is proposed. The joint calibration method calculates the plane equation of the line structured light by using multiple checkerboard images with projected structured light line. To unify the 3-D coordinates of structured light fringe in each position to the reference coordinate system, the rotation and translation of the camera in each position relative to the reference position (called absolute pose) should be solved. The direct linear transformation method is firstly used to calculate the rough relative pose between adjacent frames, and principle of re-projection error minimization in object space is used to obtain the refined relative pose. Instead of using step-by-step transfer method, the absolute pose of the camera in each position is eventually calculated by using global optimization algorithm after the relative pose between all adjacent frames are obtained. Computer simulation and abundant experimental data validate the effectiveness of proposed method. The proposed 3-D reconstruction method using global optimization is respectively compared with step-by-step transfer method based on pixel re-projection error minimization and re-projection error minimization in object space. The experimental results reflect that our method is superior to step-by-step transfer method, and the 3-D reconstruction accuracy is improved by 28.05 % after global optimization.

Analysis of independent risk factors for acute acquired comitant esotropia.

To explore the risk factors for acute acquired comitant esotropia (AACE). A retrospective cohort study was conducted by analyzing 83 patients (case group) with AACE who underwent strabismus correction surgery from January 1, 2021 to June 30, 2022. Totally 73 outpatient volunteers were recruited during the same period as the normal control group. The case group's binocular vision time, near and distance esotropia angle, and near stereo vision function were recorded, and the age, gender, refractive status, and best-corrected visual acuity (BCVA) of both groups were analyzed. Additionally, multiple logistic regression analysis was conducted using an eye usage condition questionnaire to determine the independent risk factors for AACE. In the case group, 61 patients (73.49%) had myopia, with a mean equivalent spherical power (SE) of -3.35±3.31 D (range: +2.75 to -10.62 D) of the right eye and -2.87±3.35 D (range: +2.75 to -11.12 D) of the left eye. The average duration of diplopia in the case group was 29.83±35.72mo, of which 80 patients (96.39%) were primarily with distance diplopia. The near and distance esotropia angle after wearing glasses were 52.36±20.95 prism degree (PD) and 56.71±19.54 PD, respectively, and there was no statistically significant difference between the two (t=1.38, P=0.169). The incidence of improper glasses wearing and unhealthy eye habits in the case group was significantly higher than those in the control group (P<0.05). Close-up work without glasses [β=2.30, odds ratio (OR)=10, 95% confidence interval (CI) 2.35-42.51, P=0.002] and near work in supine position (β=1.80, OR=6.02, 95%CI 3.29-11.02, P<0.001) were independent risk factors for AACE. Patients with AACE mainly present with distance diplopia, and there is a high degree of variation in myopia. Near work without wearing glasses and in supine position are independent risk factors for AACE.

Neural correlates of virtual reality-based attention training: An fMRI study

Theoretical backgroundVirtual Reality technology is increasingly used in attention rehabilitation for functional training purposes. However, the neural mechanisms by which Virtual Reality can affect attentional functioning are still unclear. The current study's objective is to examine the effects of stereoscopic vs. monoscopic presentation on neural processing during a visual attention task. MethodThirty-two healthy participants performed a visual attention task in an immersive virtual environment that was displayed via MR-compatible video goggles in an MRI scanner. The paradigm altered between trials that required active engagement with the task and mere observation trials. Furthermore, the form of binocular presentation switched between monoscopic and stereoscopic presentation. ResultsAnalyses yielded evidence for increased activation in stereoscopic compared to monoscopic trials in the tertiary visual cortex area V3A as well as elevated activation in the dorsal attention network when engaging in the attention task. An additional ROI analysis of area V3A revealed significantly lower attentional engagement costs in stereoscopic conditions. DiscussionResults support previous findings suggesting that V3A is involved in binocular depth perception. Furthermore, heightened activation in V3A following stereoscopic presentation seemed to facilitate attentional engagement with the task. Considering that V3A is the origin of the dorso-dorsal, ventro-dorsal, and ventral visual processing pathways, we regard it as a gating area that decides which kind of visual perception is processed.