Binocular Images Research Articles

Evaluation of monocular and binocular contrast perception on virtual reality head-mounted displays.

Visualization of medical images on a virtual reality (VR) head-mounted display (HMD) requires binocular fusion of a stereoscopic pair of graphical views. However, current image quality assessment on VR HMDs for medical applications has been primarily limited to time-consuming monocular optical bench measurement on a single eyepiece. As an alternative to optical bench measurement to quantify the image quality on VR HMDs, we developed a WebXR test platform to perform contrast perceptual experiments that can be used for binocular image quality assessment. We obtained monocular and binocular contrast sensitivity responses (CSRs) from participants on a Meta Quest 2 VR HMD using varied interpupillary distance (IPD) configurations. The perceptual result shows that contrast perception on VR HMDs is primarily affected by optical aberration of the VR HMD. As a result, monocular CSR degrades at a high spatial frequency greater than 4 cycles per degree when gazing at the periphery of the display field of view, especially for mismatched IPD settings consistent with optical bench measurements. On the contrary, binocular contrast perception is dominated by the monocular view with superior image quality measured by the contrast. We developed a test platform to investigate monocular and binocular contrast perception by performing perceptual experiments. The test method can be used to evaluate monocular and/or binocular image quality on VR HMDs for potential medical applications without extensive optical bench measurements.

The Adversarial Robust and Generalizable Stereo Matching for Infrared Binocular Based on Deep Learning

Despite the considerable success of deep learning methods in stereo matching for binocular images, the generalizability and robustness of these algorithms, particularly under challenging conditions such as occlusions or degraded infrared textures, remain uncertain. This paper presents a novel deep-learning-based depth optimization method that obviates the need for large infrared image datasets and adapts seamlessly to any specific infrared camera. Moreover, this adaptability extends to standard binocular images, allowing the method to work effectively on both infrared and visible light stereo images. We further investigate the role of infrared textures in a deep learning framework, demonstrating their continued utility for stereo matching even in complex lighting environments. To compute the matching cost volume, we apply the multi-scale census transform to the input stereo images. A stacked sand leak subnetwork is subsequently employed to address the matching task. Our approach substantially improves adversarial robustness while maintaining accuracy on comparison with state-of-the-art methods which decrease nearly a half in EPE for quantitative results on widely used autonomous driving datasets. Furthermore, the proposed method exhibits superior generalization capabilities, transitioning from simulated datasets to real-world datasets without the need for fine-tuning.

Pulp grade monitoring using binocular image through multi-scale feature cross-attention fusion network and saliency map constraint

Pulp grade is an important indicator for performance monitoring in froth flotation process. Previous studies have shown that the pulp grade can be predicted more accurately by using the stereo vision information of froth images. However, due to the low intra-class variation among froth images under the identical working conditions and the similarity in shape and texture among bubbles, it is challenging for current binocular image-based methods for grade prediction. Therefore, to accurately predict the key performance indicators in flotation process, a prediction model based on binocular image fusion is proposed in this paper. First, a calculation method of froth image saliency map is proposed, and the saliency map is used as a priori knowledge to guide the prediction model to learn the characteristics of the region of interest in the image. This measure aims to solve the problem of difficulty in grade prediction caused by low intra-class differences in froth images. Then, a multi-scale feature cross-attention fusion network is introduced, wherein the multi-scale features of the left and right views serve as attention mechanism gating signals to extract common feature from binocular image. After that, a pulp grade prediction model is developed based on Video Transformer Network. The results on actual industrial flotation datasets show that, compared with other pulp grade prediction methods, our proposed approach reduces the mean absolute error and root mean square error by 22.84% and 23.55%, respectively.

Multi-line laser scanning reconstruction with binocularly speckle matching and trained deep neural networks

A multi-line laser scanning system for 3D topography measurement is proposed. This method not only has the advantages of high precision of laser scanning technology, but also has high reconstruction efficiency. In this paper, speckle reconstruction technique, multi-line laser technique and Biocular reconstruction technique are used to construct a 3D reconstruction system, and test equipment is built, and the problems existing in the system establishment process are actually studied. In order to solve the problem of mismatching in binocular multi-line laser matching, a method to sort out the correspondence of multiple laser lines in binocular images based on speckle matching results is proposed. In order to optimize the multi-line laser matching effect, a speckle matching network based on deep learning is proposed, which integrates the grayscale images of the left and right cameras as supplementary information, and takes the speckle image and grayscale image as the input of the network model to obtain more accurate and edge-complete matching results. Finally, the matching results of the multi-line laser and the camera calibration parameters were used to reconstruct the object point cloud. Experimental results show that the proposed speckle matching method can make binocular multiline laser point cloud reconstruction more robust and stable than the traditional method, and through the accuracy analysis of the system, it is proved that the average measurement accuracy of the proposed method can reach 0.05 mm.

Reconstructing the local structures of Chinese ancient architecture using unsupervised depth estimation

Digitalization of ancient architectures is one of the effective means for the preservation of heritage structures, with 3D reconstruction based on computer vision being a key component of such digitalization techniques. However, Chinese ancient architectures are located in mountainous areas, and existing 3D reconstruction methods fall short in restoring the local structures of these architectures. This paper proposes a self-attention-guided unsupervised single image-based depth estimation method, providing innovative technical support for the reconstruction of local structures in Chinese ancient architectures. First, an attention module is constructed based on features extracted from architectural images learned by the encoder, and then embedded into the encoder-decoder to capture the interdependencies across local features. Second, a disparity map is generated using the loss constraint network, including reconstruction matching, smoothness of the disparity, and left-right disparity consistency. Third, an unsupervised architecture based on binocular image pairs is constructed to remove any potential adverse effects due to unknown scale or estimated pose errors. Finally, with the known baseline distance and camera focal length, the disparity map is converted into the depth map to perform the end-to-end depth estimation from a single image. Experiments on the our architecture dataset validates our method, and it performs well also well on KITTI.

Artificial Intelligence for Early Detection of Pediatric Eye Diseases Using Mobile Photos

Identifying pediatric eye diseases at an early stage is a worldwide issue. Traditional screening procedures depend on hospitals and ophthalmologists, which are expensive and time-consuming. Using artificial intelligence (AI) to assess children's eye conditions from mobile photographs could facilitate convenient and early identification of eye disorders in a home setting. To develop an AI model to identify myopia, strabismus, and ptosis using mobile photographs. This cross-sectional study was conducted at the Department of Ophthalmology of Shanghai Ninth People's Hospital from October 1, 2022, to September 30, 2023, and included children who were diagnosed with myopia, strabismus, or ptosis. A deep learning-based model was developed to identify myopia, strabismus, and ptosis. The performance of the model was assessed using sensitivity, specificity, accuracy, the area under the curve (AUC), positive predictive values (PPV), negative predictive values (NPV), positive likelihood ratios (P-LR), negative likelihood ratios (N-LR), and the F1-score. GradCAM++ was utilized to visually and analytically assess the impact of each region on the model. A sex subgroup analysis and an age subgroup analysis were performed to validate the model's generalizability. A total of 1419 images obtained from 476 patients (225 female [47.27%]; 299 [62.82%] aged between 6 and 12 years) were used to build the model. Among them, 946 monocular images were used to identify myopia and ptosis, and 473 binocular images were used to identify strabismus. The model demonstrated good sensitivity in detecting myopia (0.84 [95% CI, 0.82-0.87]), strabismus (0.73 [95% CI, 0.70-0.77]), and ptosis (0.85 [95% CI, 0.82-0.87]). The model showed comparable performance in identifying eye disorders in both female and male children during sex subgroup analysis. There were differences in identifying eye disorders among different age subgroups. In this cross-sectional study, the AI model demonstrated strong performance in accurately identifying myopia, strabismus, and ptosis using only smartphone images. These results suggest that such a model could facilitate the early detection of pediatric eye diseases in a convenient manner at home.

SED-RCNN-BE: A SE-Dual channel RCNN network optimized binocular estimation model for automatic size estimation of free swimming fish in aquaculture

Size of fish body is one of the most important morphological characteristics for cultured fish and plays a key role to enable effective management in aquaculture. Automatic, accurate and real-time monitoring of fish body size is therefore essential for improving aquaculture production. Computer vision facilitated the non-contact estimation of fish size and reduce labor consumption. While, current computer vision estimation techniques are still restricted in multi-scale size and multi-pose-variant of free-swimming fish, which limit the relevance application to aquaculture. Hence, we propose a novel method to facilitate automatic and accurate size estimation of free-swimming fish for growth monitoring of fish in aquaculture. This method comprised of an improved keypoint detection module based on Keypoint RCNN (SE-D-KP-RCNN) network and size estimation module based on CREStereo matching and spatial plane curve fitting. In the detection module, a squeeze and excitation Dual channel pooling layer (SE-D) is designed for the backbone network to improve the keypoint detection accuracy for free-swimming fish. In the estimation module, the global parallax map of binocular image is produced by CREStereo matching to obtain the 3D information of detection keypoint and build curve fitting of multi-pose variant fish body for height and length estimation. We implement and test our proposed method in different sizes and poses fish and the comprehensive experimental results show that the average accuracy of size estimation can reached 93.18% in the real aquaculture scene. This is important to help breeders to obtain underwater fish size information accurately and improve the efficiency for aquaculture management.

16‐1: A Model for the Appearance of Interocular Colorimetric Differences in Binocular XR Displays

Many extended reality (XR) devices present different views to the left and right eyes. Unwanted colorimetric differences between these views can cause perceptual artifacts that degrade binocular image quality. We present an image‐computable model designed to predict the appearance of binocular views with colorimetric differences in XR displays. The model is fitted to data from a recent perceptual study in which people provided multidimensional responses about the appearance of stimuli simulating an optical see‐through augmented reality device with interocular intensity differences. This work can be used to create preliminary assessments of binocular artifact appearance and inform XR display design.

Extraction of Corn Plant Phenotypic Parameters with Keypoint Detection and Stereo Images

Corn is a global crop that requires the breeding of superior varieties. A crucial aspect of the breeding process is the accurate extraction of phenotypic parameters from corn plants. The existing challenges in phenotypic parameter extraction include low precision, excessive manual involvement, prolonged processing time, and equipment complexity. This study addresses these challenges by opting for binocular cameras as the data acquisition equipment. The proposed stereo corn phenotype extraction algorithm (SCPE) leverages binocular images for phenotypic parameter extraction. The SCPE consists of two modules: the YOLOv7-SlimPose model and the phenotypic parameter extraction module. The YOLOv7-SlimPose model was developed by optimizing the neck component, refining the loss function, and pruning the model based on YOLOv7-Pose. This model can better detect bounding boxes and keypoints with fewer parameters. The phenotypic parameter extraction module can construct the skeleton of the corn plant and extract phenotypic parameters based on the coordinates of the keypoints detected. The results showed the effectiveness of the approach, with the YOLOv7-SlimPose model achieving a keypoint mean average precision (mAP) of 96.8% with 65.1 million parameters and a speed of 0.09 s/item. The phenotypic parameter extraction module processed one corn plant in approximately 0.2 s, resulting in a total time cost of 0.38 s for the entire SCPE algorithm to construct the skeleton and extract the phenotypic parameters. The SCPE algorithm is economical and effective for extracting phenotypic parameters from corn plants, and the skeleton of corn plants can be constructed to evaluate the growth of corn as a reference. This proposal can also serve as a valuable reference for similar functions in other crops such as sorghum, rice, and wheat.

Quasi-3D: reducing convergence effort improves visual comfort of head-mounted stereoscopic displays

The diffusion of virtual reality urges to solve the problem of vergence-accommodation conflict arising when viewing stereoscopic displays, which causes visual stress. We addressed this issue with an approach based on reducing ocular convergence effort. In virtual environments, vergence can be controlled by manipulating the binocular separation of the virtual cameras. Using this technique, we implemented two quasi-3D conditions characterized by binocular image separations intermediate between 3D (stereoscopic) and 2D (monoscopic). In a first experiment, focused on perceptual aspects, ten participants performed a visuo-manual pursuit task while wearing a head-mounted display (HMD) in head-constrained (non-immersive) condition for an overall exposure time of ~ 7 min. Passing from 3D to quasi-3D and 2D conditions, progressively resulted in a decrease of vergence eye movements—both mean convergence angle (static vergence) and vergence excursion (dynamic vergence)—and an increase of hand pursuit spatial error, with the target perceived further from the observer and larger. Decreased static and dynamic vergence predicted decreases in asthenopia trial-wise. In a second experiment, focused on tolerance aspects, fourteen participants performed a detection task in near-vision while wearing an HMD in head-free (immersive) condition for an overall exposure time of ~ 20 min. Passing from 3D to quasi-3D and 2D conditions, there was a general decrease of both subjective and objective visual stress indicators (ocular convergence discomfort ratings, cyber-sickness symptoms and skin conductance level). Decreased static and dynamic vergence predicted the decrease in these indicators. Remarkably, skin conductance level predicted all subjective symptoms, both trial-wise and session-wise, suggesting that it could become an objective replacement of visual stress self-reports. We conclude that relieving convergence effort by reducing binocular image separation in virtual environments can be a simple and effective way to decrease visual stress caused by stereoscopic HMDs. The negative side-effect—worsening of spatial vision—arguably would become unnoticed or compensated over time. This initial proof-of-concept study should be extended by future large-scale studies testing additional environments, tasks, displays, users, and exposure times.

Research on image deformation monitoring algorithm based on binocular vision

Monitoring the deformation of critical structures such as tunnel、buildings and bridges is critical to ensure the safety of infrastructures. Non-contact deformation monitoring technology based on digital imagery has emerged as a prevalent and effective monitoring method, which enables cost-effective and efficient tracking of structural deformations. This paper aims to develop an automated deformation monitoring methodology using the binocular vision. First, standard preprocessing of digital images is performed through the integration of filtering algorithms and image enhancement techniques. Furthermore, by combining the Scale Invariant Feature Transform (SIFT) feature matching algorithm, the inverse compositional gauss–newton iterative algorithm, and interpolation methodologies, the planar displacement of the monitored targets can be calculated precisely, with sub-pixel precision being achieved. Subsequently, by adopting an enhanced Semi-Global Matching (SGM) algorithm, stereo matching of binocular images were accomplished. This process yields a disparity map, from which the depth map is computed. Then, precise measurement of the vertical displacement of the monitoring target can be obtained. This enables accurate measurement of three-dimensional displacements of the monitored targets. Finally, the developed methodology undergoes a comparative assessment against traditional measurement methods and existing algorithms to validate its measurement accuracy. The results prove that the deformation measurement technique based on binocular vision proposed in this paper captures structural deformation effectively. The accuracy surpasses that of existing measurement algorithms, and its efficiency outperforms traditional measurement methods.

Research on YOLOv5 Vehicle Detection and Positioning System Based on Binocular Vision

Vehicle detection and location is one of the key sensing tasks of automatic driving systems. Traditional detection methods are easily affected by illumination, occlusion and scale changes in complex scenes, which limits the accuracy and robustness of detection. In order to solve these problems, this paper proposes a vehicle detection and location method for YOLOv5(You Only Look Once version 5) based on binocular vision. Binocular vision uses two cameras to obtain images from different angles at the same time. By calculating the difference between the two images, more accurate depth information can be obtained. The YOLOv5 algorithm is improved by adding the CBAM attention mechanism and replacing the loss function to improve target detection. Combining these two techniques can achieve accurate detection and localization of vehicles in 3D space. The method utilizes the depth information of binocular images and the improved YOLOv5 target detection algorithm to achieve accurate detection and localization of vehicles in front. Experimental results show that the method has high accuracy and robustness for vehicle detection and localization tasks.

DRC: Chromatic aberration intensity priors for underwater image enhancement

Underwater imaging technology is a crucial tool for monitoring marine flora and fauna. However, selective light absorption and scattering properties of water make underwater imagery frequently appear blurred and exhibit color biases, hindering the extraction of vital aquacultural insights. To address this challenge, we propose a method, namely DRC, which is a holistic approach to enhancing underwater image clarity and color fidelity. This method comprises three integral components: D-procedure, R-procedure, and C-procedure. The D-procedure intricately accounts for the trichromatic underwater attenuation dynamics, formulating a chromatic aberration intensity prior. This prior counters the disparities in degradation levels seen in conventional single-channel prior depth estimations, achieving a dynamic depth representation approaching binocular image precision. The R-procedure, utilizing an adaptive dark-pixel prior, pinpoints corresponding points across varied depth zones to counteract backscattering, thereby mitigating the image’s hazy appearance. The C-procedure bolsters image luminance and color fidelity through opponent channel rectification and amalgamates pronounced image contrasts and intricate details via Gaussian pyramid fusion. The method was tested on several publicly available datasets and compared with nine popular underwater image enhancement techniques. Both subjective and objective assessments underscore the superiority of our DRC method over existing underwater image enhancement techniques.

Stereo attention-based all-in-one super-resolution for robot-assisted minimally invasive surgery.

Robot-assisted minimally invasive surgery (MIS) faces challenges in obtaining high-quality imaging results due to the limited spatial environment. In this paper, we present an all-in-one image super-resolution (SR) algorithm designed to tackle this challenge. By utilizing the stereo information from binocular images, we effectively convert low-resolution images into high-resolution ones. Our model architecture amalgamates the prowess of Convolutional Neural Networks (CNNs) and Transformers, capitalizing on the advantages of both methodologies. To achieve super-resolution across all scale factors, we employ a trainable upsampling module within our proposed network. We substantiate the effectiveness of our method through extensive quantitative and qualitative experiments. The results of our evaluations provide strong evidence supporting the superior performance of our approach in enhancing the quality of surgical images. Our method improves the resolution and thus the overall image quality, which allows the surgeon to perform precise operations conveniently. Simultaneously, it also facilitates the scaling of the region of interest (ROI) to achieve high-quality visualization during surgical procedures. Furthermore, it has the potential to enhance the image quality during telesurgery.

Matching cost function analysis and disparity optimization for low-quality binocular images

State-of-the-art dense stereo matching algorithms have achieved excellent performance, demonstrating a capability to attain precise matching in most areas. However, current such methods rarely achieve this when images are captured under poor conditions. To improve the accuracy of the algorithm in such cases, this paper introduces a post-optimization algorithm to rectify matching errors and enhance outcomes. The main research areas of this paper include three aspects. (1) Disparities are classified into reliable and unreliable results based on the analysis of geometric matching relationships, local features in the images, and components within the matching cost function; (2) Subsequent analysis of horizontal image features identifies local characteristic indices calculated through integration along the horizontal axis, which establish specific matching criteria, forming the foundation for a cost volume that encompasses these distinct matches; (3) A redefined matching cost function is applied to previously classified unreliable results to rectify matching errors. This energy function is based on the cost volume above. Experimental results validate the efficacy of the proposed post-optimization algorithm, reducing the average matching errors from 8.66% to 5.85%.

A Novel Approach for Simultaneous Localization and Dense Mapping Based on Binocular Vision in Forest Ecological Environment

The three-dimensional reconstruction of forest ecological environment by low-altitude remote sensing photography from Unmanned Aerial Vehicles (UAVs) provides a powerful basis for the fine surveying of forest resources and forest management. A stereo vision system, D-SLAM, is proposed to realize simultaneous localization and dense mapping for UAVs in complex forest ecological environments. The system takes binocular images as input and 3D dense maps as target outputs, while the 3D sparse maps and the camera poses can be obtained. The tracking thread utilizes temporal clue to match sparse map points for zero-drift localization. The relative motion amount and data association between frames are used as constraints for new keyframes selection, and the binocular image spatial clue compensation strategy is proposed to increase the robustness of the algorithm tracking. The dense mapping thread uses Linear Attention Network (LANet) to predict reliable disparity maps in ill-posed regions, which are transformed to depth maps for constructing dense point cloud maps. Evaluations of three datasets, EuRoC, KITTI and Forest, show that the proposed system can run at 30 ordinary frames and 3 keyframes per second with Forest, with a high localization accuracy of several centimeters for Root Mean Squared Absolute Trajectory Error (RMS ATE) on EuRoC and a Relative Root Mean Squared Error (RMSE) with two average values of 0.64 and 0.2 for trel and Rrel with KITTI, outperforming most mainstream models in terms of tracking accuracy and robustness. Moreover, the advantage of dense mapping compensates for the shortcomings of sparse mapping in most Smultaneous Localization and Mapping (SLAM) systems and the proposed system meets the requirements of real-time localization and dense mapping in the complex ecological environment of forests.

Binocular Image Dehazing via a Plain Network Without Disparity Estimation

Heavy haze leads to severely degraded visual quality for images, and thus the performance of high level image-based tasks such as object detection and semantic segmentation is deteriorated. It is necessary and important to design an effective dehazing method for the computer vision system. It is well known that image haze is a function of depth and binocular images can predict the depth. Existing binocular dehazing methods conduct disparity estimation and dehazing jointly to enhance each other. However, a small error in disparity gives rise to a large variation in depth and in the estimation of haze-free images. To alleviate the problem, we propose a plain binocular image dehazing network in this paper, called BidNet, to dehaze both the left and right images simultaneously. BidNet does not explicitly perform disparity estimation that is time-consuming and well-known to be challenging. Instead, we design a stereo transformation module to mine the relationship and correlation between binocular images, making the best of varying information of cross views. Additionally, we design a Stereo Foggy Cityscapes dataset extended from the Foggy Cityscapes dataset for training the proposed BidNet. Extensive experimental results demonstrate that BidNet significantly outperforms the SOTA dehazing methods on the synthetic stereo foggy datasets as well as in real stereo foggy scenes. Experimental results show that jointly dehazing binocular image pairs is mutually beneficial, which is better than only dehazing left images. Furthermore, when applying BidNet to preprocess foggy inputs, large improvements are obtained in the performance of object detection, instance segmentation, semantic segmentation, and stereo-based 3D object detection.

Avatar-Based Picture Exchange Communication System Enhancing Joint Attention Training for Children With Autism.

Children with Autism Spectrum Disorder (ASD) often struggle with social communication and feel anxious in interactive situations. The Picture Exchange Communication System (PECS) is commonly used to enhance basic communication skills in children with ASD, but it falls short in reducing social anxiety during therapist interactions and in keeping children engaged. This paper proposes the use of virtual character technology alongside PECS training to address these issues. By integrating a virtual avatar, children's communication skills and ability to express needs can be gradually improved. This approach also reduces anxiety and enhances the interactivity and attractiveness of the training. After conducting a T-test, it was found that PECS assisted by a virtual avatar significantly improves children's focus on activities and enhances their behavioral responsiveness. To address the problem of poor accuracy of gaze estimation in unconstrained environments, this study further developed a visual feature-based gaze estimation algorithm, the three-channel gaze network (TCG-Net). It utilizes binocular images to refine the gaze direction and infer the primary focus from facial images. Our focus was on enhancing gaze tracking accuracy in natural environments, crucial for evaluating and improving Joint Attention (JA) in children during interactive processes.TCG-Net achieved an angular error of 4.0 on the MPIIGaze dataset, 5.0 on the EyeDiap dataset, and 6.8 on the RT-Gene dataset, confirming the effectiveness of our approach in improving gaze accuracy and the quality of social interactions.

Poster Session I: Perspective-correct rendering for active observers.

Stereoscopic, head-tracked display systems can show users realistic, world-locked virtual objects and environments (i.e., rendering perspective-correct binocular images with accurate motion parallax). However, discrepancies between the rendering pipeline and physical viewing conditions can lead to perceived instability in the rendered content resulting in reduced immersion and, potentially, visually-induced motion sickness. Precise requirements to achieve perceptually stable world-locked rendering (WLR) are unknown due to the challenge of constructing a wide field of view, distortion-free display with highly accurate head and eyetracking. We present a custom-built system capable of rendering virtual objects over real-world references without perceivable drift under such constraints. This platform is used to study acceptable errors in render camera position for WLR in augmented and virtual reality scenarios, where we find an order of magnitude difference in perceptual sensitivity. We conclude with an analytic model which examines changes to apparent depth and visual direction in response to camera displacement errors, and visual direction is highlighted as a potentially important consideration for WLR alongside depth errors from incorrect disparity.

Restoration of Binocular Images Degraded by Optical Scattering through Estimation of Atmospheric Coefficients.

A binocular vision-based approach for the restoration of images captured in a scattering medium is presented. The scene depth is computed by triangulation using stereo matching. Next, the atmospheric parameters of the medium are determined with an introduced estimator based on the Monte Carlo method. Finally, image restoration is performed using an atmospheric optics model. The proposed approach effectively suppresses optical scattering effects without introducing noticeable artifacts in processed images. The accuracy of the proposed approach in the estimation of atmospheric parameters and image restoration is evaluated using synthetic hazy images constructed from a well-known database. The practical viability of our approach is also confirmed through a real experiment for depth estimation, atmospheric parameter estimation, and image restoration in a scattering medium. The results highlight the applicability of our approach in computer vision applications in challenging atmospheric conditions.