Stereo Vision Research Articles

Fully Distributed Shape Sensing of a Flexible Surgical Needle Using Optical Frequency Domain Reflectometry for Prostate Interventions.

In minimally invasive procedures such as biopsies and prostate cancer brachytherapy, accurate needle placement remains challenging due to limitations in current tracking methods related to interference, reliability, resolution or image contrast. This often leads to frequent needle adjustments and reinsertions. To address these shortcomings, we introduce an optimized needle shape-sensing method using a fully distributed grating-based sensor. The proposed method uses simple trigonometric and geometric modeling of the fiber using optical frequency domain reflectometry (OFDR), without requiring prior knowledge of tissue properties or needle deflection shape and amplitude. Our optimization process includes a reproducible calibration process and a novel tip curvature compensation method. We validate our approach through experiments in artificial isotropic and inhomogeneous animal tissues, establishing ground truth using 3D stereo vision and cone beam computed tomography (CBCT) acquisitions, respectively. Our results yield an average RMSE ranging from 0.58 ± 0.21 mm to 0.66 ± 0.20 mm depending on the chosen spatial resolution, achieving the submillimeter accuracy required for interventional procedures.

Investigating the Link Between Subjective Depth Perception Deficits and Objective Stereoscopic Vision Deficits in Individuals With Acquired Brain Injury.

Individuals with acquired brain injury have reported subjective complaints of depth perception deficits, but few have undergone objective assessments to confirm these deficits. As a result, the literature currently lacks reports detailing the correlation between subjective depth perception deficits and objective stereoscopic vision deficits in individuals with acquired brain injury, particularly those cases that are characterized by a clearly defined lesion. To investigate this relationship, we recruited three individuals with acquired brain injury who experienced depth perception deficits and related difficulties in their daily lives. We had them take neurologic, ophthalmological, and neuropsychological examinations. We also had them take two types of stereoscopic vision tests: a Howard-Dolman-type stereoscopic vision test and the Topcon New Objective Stereo Test. Then, we compared the results with those of two control groups: a group with damage to the right hemisphere of the brain and a group of healthy controls. Performance on the two stereoscopic vision tests was severely impaired in the three patients. One of the patients also presented with cerebral diplopia. We identified the potential neural basis of these deficits in the cuneus and the posterior section of the superior parietal lobule, which play a role in vergence fusion and are located in the caudal region of the dorso-dorsal visual pathway, which is known to be crucial not only for visual spatial perception, but also for reaching, grasping, and making hand postures in the further course of that pathway.

Advances in the application of stereo vision in aquaculture with emphasis on fish: A review

AbstractThe effective implementation of machine vision has played a crucial role in advancing intelligent aquaculture across various domains. Stereo vision, as a branch of machine vision, has become a mainstream technology in aquaculture. Its distinctive capability to conduct comprehensive underwater monitoring from multiple angles, unaffected by object occlusion has propelled it to the forefront of aquaculture applications. This article offers a comprehensive review of the diverse applications of stereo vision in aquaculture spanning from its inception to present. The exploration encompasses its role in crucial areas such as biomass estimation and behavioural analysis, which include fish counting, weight estimation, swimming behaviour, feeding behaviour and abnormal behaviour. Furthermore, the paper delves into the advantages of stereo vision over traditional 2D machine vision approaches, while also acknowledging limitations, and identifying future challenges that must be addressed to fully leverage its potential in aquaculture. The review emphasizes the prospect of advancement in deep learning stereo‐matching algorithms specifically designed for underwater environments to catalyse a breakthrough in stereo vision technology. In summary, this review aims to provide researchers and practitioners with a better understanding of the current development of stereo vision in aquaculture, optimizing stereo vision technology and better serving the aquaculture field.

Advancing defense capabilities through integration of electro-optical systems and computer vision technologies

This paper presents a comprehensive study on the integration of cutting-edge technologies to enhance defense capabilities, focusing on object detection, tracking, and distance measurement tasks. Also, the integration of IMX219-77 cameras and Nvidia Jetson Nano for object detection, tracking, and distance measurement in electro-optical systems is proposed. The decision to combine these components was based on their individual strengths: the high resolution and wide field of view of the IMX219-77 cameras, and the low-power, high-performance capabilities of the Jetson Nano. Software-wise, OpenCV and GStreamer were chosen for their widespread use and capabilities in computer vision and multimedia processing tasks, respectively. The MOSSE algorithm was selected for object tracking due to its speed, efficiency, and resilience to lighting changes. Additionally, Depth Estimation (Stereo Vision) was employed for distance measurement by analyzing images captured from a pair of cameras. The proposed algorithm involves transmitting images from the cameras to the Jetson Nano, where OpenCV and GStreamer are utilized to process the data. The MOSSE algorithm is employed for object detection and tracking, while Depth Estimation generates depth maps to measure distances between objects and the camera. The study's results demonstrate accurate distance measurements with a margin of error ranging from 0 to 2 mm, falling below the acceptable threshold mentioned in academic literature. This aligns with similar studies in the field, showcasing the efficacy of the proposed integration for object detection, tracking, and distance measurement in electro-optical systems.

Autonomous thermal tracking reveals spatiotemporal patterns of seabird activity relevant to interactions with floating offshore wind facilities

Planning is underway for placement of infrastructure needed to begin offshore wind (OSW) energy generation along the West Coast of the United States and elsewhere in the Pacific Ocean. In contrast to the primarily nearshore windfarms currently in the North Atlantic, the seabird communities inhabiting Pacific Wind Energy Areas (WEAs) include significant populations of species that fly by dynamic soaring, a behavior dependent on wind and in which flight height increases steeply with wind speed. Therefore, a more precise and detailed assessment of their 3D airspace use is needed to better understand the potential collision risks that OSW turbines may present to these seabirds. Toward this end, a novel technology called the ThermalTracker-3D (TT3D), which uses thermal imaging and stereo vision, was developed to render high-resolution (on average within ±5 m) flight tracks and related behavior of seabirds. The technology was developed and deployed on a wind-profiling LiDAR buoy in the Humboldt WEA, located 34 to 57 km off California’s coast. During the at-sea deployment between 24 May and 13 August 2021, the TT3D successfully tracked birds moving between 10 and 500 m from the device, around the clock, and in all weather conditions; a total of 1407 detections and their corresponding 3D flight trajectories were recorded. Mean altitudes of detections ranged 6-295 m above sea level (asl). Considering the degree of overlap with anticipated rotor swept zones (RSZ), which extend 25-260 m asl, 79% of detected birds (per m3 of airspace) moved below the RSZ, 21% moved at heights overlapping the RSZ, and another 0.04% occurred at heights exceeding the RSZ. The high-resolution tracks provided valuable insight into seabird space use, especially at heights that make them vulnerable to collision during various environmental conditions (e.g., darkness, strong winds). Observations made by the TT3D will be useful in filling critical knowledge gaps related to estimating collision and avoidance between seabirds and OSW facilities in the Pacific and elsewhere. Future research will focus on enhancing the TT3D’s identification capabilities to the lowest taxon through validation studies and artificial intelligence, further contributing to seabird conservation efforts associated with OSW.

OmniGlasses: an optical aid for stereo vision CNNs to enable omnidirectional image processing

Stereo vision is a key technology for 3D scene reconstruction from image pairs. Most approaches process perspective images from commodity cameras. These images, however, have a very limited field of view and only picture a small portion of the scene. In contrast, omnidirectional images, also known as fisheye images, exhibit a much larger field of view and allow a full 3D scene reconstruction with a small amount of cameras if placed carefully. However, omnidirectional images are strongly distorted which make the 3D reconstruction much more sophisticated. Nowadays, a lot of research is conducted on CNNs for omnidirectional stereo vision. Nevertheless, a significant gap between estimation accuracy and throughput can be observed in the literature. This work aims to bridge this gap by introducing a novel set of transformations, namely OmniGlasses. These are incorporated into the architecture of a fast network, i.e., AnyNet, originally designed for scene reconstruction on perspective images. Our network, Omni-AnyNet, produces accurate omnidirectional distance maps with a mean absolute error of around 13 cm at 48.4 fps and is therefore real-time capable.

Evaluation of the photosensory characteristics of the lateral and pineal eyes of Plioplatecarpus (Squamata, Mosasauridae) based on an exceptionally preserved specimen from the Bearpaw Shale (Campanian, Upper Cretaceous) of southern Alberta

ABSTRACT The skull of Plioplatecarpus possesses large orbits and a large parietal foramen. Although the large orbits suggest that Plioplatecarpus possessed large eyes with enhanced visual performance, none of the previously described skulls include a sclerotic ring, and so it has been impossible in the past to reconstruct their size or optical properties. A recently collected skull with a sclerotic ring preserved in situ provides for the first time the data to do so. The dimensions of the sclerotic ring demonstrate that, although the f-number of the eyeball was not low enough to support the hypothesis that Plioplatecarpus was adapted to low light conditions, the eyeball was nevertheless large and possessed increased visual acuity and sensitivity that enabled the mosasaur to operate efficiently in both light-rich and light-poor conditions and possibly other conditions of reduced visibility. Proportions of this skull also suggest that Plioplatecarpus possessed a relatively wide binocular field of vision, providing stereoscopic vision, and further increasing both acuity and sensitivity, enhancing its ability to effectively detect and track quickly moving objects under a wide range of light conditions. A greatly enlarged parietal foramen implies the presence of a large parietal eye, the significance of which is uncertain, but it may have enhanced the ability to regulate migratory behavior.

Research on Monocular Depth Sensing Method Based on Liquid Zoom Imaging

Monocular stereo vision has excellent application prospects in the field of microrobots. On the basis of the geometric model of bifocal imaging, this paper proposes a monocular depth perception method by liquid zoom imaging. Firstly, the configuration of a monocular liquid vision system for depth measurement is presented, and the working mechanism of the system is analyzed through theoretical derivation. Then, to eliminate the influence of optical axis drift induced by the liquid gravity factor on the measurement results, the target image area is used as the calculation feature instead of the image vector length. A target area calculation method based on chain code classification and strip segmentation is proposed. Furthermore, in response to the fluctuation problem of liquid lens focal power caused by factors such as temperature and object distance, a dynamic focal length model of the liquid zoom imaging system is constructed after precise calibration of the focal power function. Finally, a testing experiment is designed to validate the proposed method. The experimental results show that the average error of depth perception methods is 4.30%, and its measurement time is only on the millisecond scale. Meanwhile, the proposed method has good generalization performance.

A comparative study of stereo-dependent SSVEP targets and their impact on VR-BCI performance.

Steady-state visual evoked potential brain-computer interfaces (SSVEP-BCI) have attracted significant attention due to their ease of deployment and high performance in terms of information transfer rate (ITR) and accuracy, making them a promising candidate for integration with consumer electronics devices. However, as SSVEP characteristics are directly associated with visual stimulus attributes, the influence of stereoscopic vision on SSVEP as a critical visual attribute has yet to be fully explored. Meanwhile, the promising combination of virtual reality (VR) devices and BCI applications is hampered by the significant disparity between VR environments and traditional 2D displays. This is not only due to the fact that screen-based SSVEP generally operates under static, stable conditions with simple and unvaried visual stimuli but also because conventional luminance-modulated stimuli can quickly induce visual fatigue. This study attempts to address these research gaps by designing SSVEP paradigms with stereo-related attributes and conducting a comparative analysis with the traditional 2D planar paradigm under the same VR environment. This study proposed two new paradigms: the 3D paradigm and the 3D-Blink paradigm. The 3D paradigm induces SSVEP by modulating the luminance of spherical targets, while the 3D-Blink paradigm employs modulation of the spheres' opacity instead. The results of offline 4-object selection experiments showed that the accuracy of 3D and 2D paradigm was 85.67 and 86.17% with canonical correlation analysis (CCA) and 86.17 and 91.73% with filter bank canonical correlation analysis (FBCCA), which is consistent with the reduction in the signal-to-noise ratio (SNR) of SSVEP harmonics for the 3D paradigm observed in the frequency-domain analysis. The 3D-Blink paradigm achieved 75.00% of detection accuracy and 27.02 bits/min of ITR with 0.8 seconds of stimulus time and task-related component analysis (TRCA) algorithm, demonstrating its effectiveness. These findings demonstrate that the 3D and 3D-Blink paradigms supported by VR can achieve improved user comfort and satisfactory performance, while further algorithmic optimization and feature analysis are required for the stereo-related paradigms. In conclusion, this study contributes to a deeper understanding of the impact of binocular stereoscopic vision mechanisms on SSVEP paradigms and promotes the application of SSVEP-BCI in diverse VR environments.

Leveraging Mobile Net and Yolo Algorithm for Enhanced Perception in Autonomous Driving

This project presents a cutting-edge autonomous navigation system designed to enhance spatial understanding and object recognition for vehicles. Equipped with cameras on both the left and right sides, by capturing comprehensive view of the surroundings, utilizing stereo vision for crucial depth information. The integration of Mobile Net and YOLO (You Only Look Once) algorithms is central to achieving real-time object detection and recognition. Mobile Net is employed for efficient feature extraction, ensuring optimal computational efficiency. Simultaneously, YOLO plays a pivotal role in rapid and accurate identification of objects within the captured images, contributing to the robustness of spatial understanding crucial for autonomous navigation. The result is a comprehensive and reliable autonomous navigation system, showcasing the effectiveness of combining cutting-edge technologies for improved real time decision making.

Wearable apparatus for correction of visual alignment under torsional strabismus.

A wearable optical apparatus that compensates for eye misalignment (strabismus) to correct for double vision (diplopia) is proposed. In contrast to prism lenses, commonly used to compensate for horizontal and/or vertical misalignment, the proposed approach is able to compensate for any combination of horizontal, vertical, and torsional misalignment. If the action of the extraocular muscles is compromised (e.g., by nerve damage), a patient may lose their ability to maintain visual alignment, negatively affecting their binocular fusion and stereo depth perception capability. Torsional misalignment cannot be mitigated by standard Fresnel prism lenses. Surgical procedures intended to correct torsional misalignment may be unpredictable. A wearable device able to rectify visual alignment and restore stereo depth perception without surgical intervention could potentially be of great value to people with strabismus. We propose a novel lightweight wearable optical device for visual alignment correction. The device comprises two mirrors and a Fresnel prism, arranged in such a way that together they rotationally shift the view seen by the affected eye horizontally, vertically, and torsionally. The extent of the alignment correction on each axis can be arbitrarily adjusted according to the patient's particular misalignment characteristics. The proposed approach was tested by computer simulation, and a prototype device was manufactured. The prototype device was tested by a strabismus patient exhibiting horizontal and torsional misalignment. In these tests, the device was found to function as intended, allowing the patient to enjoy binocular fusion and stereo depth perception while wearing the device for daily activities over a period of several months. The proposed device is effective in correcting arbitrary horizontal, vertical, and torsional misalignment of the eyes. The results of the initial testing performed are highly encouraging. Future study is warranted to formally assess the effectiveness of the device on multiple test patients.

Image Processing Techniques in Virtual and Augmented Reality: A Comprehensive Overview

Abstract: Virtual Reality (VR) and Augmented Reality (AR) have become increasingly popular technologies with diverse applications ranging from entertainment and gaming to healthcare and education. Central to the immersive experiences offered by VR and AR are advanced image processing techniques. This report provides a comprehensive overview of the image processing techniques commonly employed in VR and AR systems, including image recognition, depth sensing, stereo vision, and more. Additionally, it discusses their applications, challenges, and future directions in the field.

Estimating cost of pothole repair from digital images using Stereo Vision and Artificial Neural Network

A significant amount of road maintenance cost goes into pothole repairs. The primary cost factors related to potholes are their size and depth, as larger and thicker potholes incur higher repair costs. However, existing methods for estimating pothole repair in developing countries rely on manual size measurements, which is time consuming, labor intensive, subjective and can lead to poor estimation of repair cost. This paper presents a system that can automatically determine the size of potholes from digital images and estimate the cost of repair. In this study, the stereo vision method was used to automatically estimate the depths of potholes from digital camera images. A feed-forward backward propagation Artificial Neural Network (ANN) was trained using pothole images acquired using mobile phones. The predicted depths and sizes of the potholes were then used to estimate the quantity of materials required to fill the potholes and subsequently, the cumulative cost of repair. Marking out and manual size measurements were performed for twenty randomly selected potholes in the Ugbowo Campus of the University of Benin, Nigeria. These measurements were compared against the estimated sizes of potholes predicted by the ANN model. A system was developed to automatically compute these material costs and considering other cost components such as transportation, labor, and equipment. Results obtained showed that the mean errors for depth, width and height estimates were 3.403%, 3.789% and 5.2617% respectively. Consequently, the developed system correctly estimated the cost of repair of the potholes considered in this study. A significant contribution of the paper is the speed and convenience of acquiring pothole data using a mobile phones without the need for on spot assessment of potholes or use of relatively more expensive stereoscopic camera setup.

Nonlinearly Optimized Dual Stereo Visual Odometry Fusion

Visual odometry (VO) is an important problem studied in robotics and computer vision in which the relative camera motion is computed through visual information. In this work, we propose to reduce the error accumulation of a dual stereo VO system (4 cameras) computing 6 degrees of freedom poses by fusing two independent stereo odometry with a nonlinear optimization. Our approach computes two stereo odometries employing the LIBVISO2 algorithm and later merge them by using image correspondences between the stereo pairs and minimizing the reprojection error with graph-based bundle adjustment. Experiments carried out on the KITTI odometry datasets show that our method computes more accurate estimates (measured as the Relative Positioning Error) in comparison to the traditional stereo odometry (stereo bundle adjustment). In addition, the proposed method has a similar or better odometry accuracy compared to ORB-SLAM2 and UCOSLAM algorithms.

Dense monocular depth estimation for stereoscopic vision based on pyramid transformer and multi-scale feature fusion

Stereoscopic display technology plays a significant role in industries, such as film, television and autonomous driving. The accuracy of depth estimation is crucial for achieving high-quality and realistic stereoscopic display effects. In addressing the inherent challenges of applying Transformers to depth estimation, the Stereoscopic Pyramid Transformer-Depth (SPT-Depth) is introduced. This method utilizes stepwise downsampling to acquire both shallow and deep semantic information, which are subsequently fused. The training process is divided into fine and coarse convergence stages, employing distinct training strategies and hyperparameters, resulting in a substantial reduction in both training and validation losses. In the training strategy, a shift and scale-invariant mean square error function is employed to compensate for the lack of translational invariance in the Transformers. Additionally, an edge-smoothing function is applied to reduce noise in the depth map, enhancing the model's robustness. The SPT-Depth achieves a global receptive field while effectively reducing time complexity. In comparison with the baseline method, with the New York University Depth V2 (NYU Depth V2) dataset, there is a 10% reduction in Absolute Relative Error (Abs Rel) and a 36% decrease in Root Mean Square Error (RMSE). When compared with the state-of-the-art methods, there is a 17% reduction in RMSE.

Redefining the Laparoscopic Spatial Sense: AI-Based Intra- and Postoperative Measurement from Stereoimages

A significant challenge in image-guided surgery is the accurate measurement task of relevant structures such as vessel segments, resection margins, or bowel lengths. While this task is an essential component of many surgeries, it involves substantial human effort and is prone to inaccuracies. In this paper, we develop a novel human-AI-based method for laparoscopic measurements utilizing stereo vision that has been guided by practicing surgeons. Based on a holistic qualitative requirements analysis, this work proposes a comprehensive measurement method, which comprises state-of-the-art machine learning architectures, such as RAFT-Stereo and YOLOv8. The developed method is assessed in various realistic experimental evaluation environments. Our results outline the potential of our method achieving high accuracies in distance measurements with errors below 1 mm. Furthermore, on-surface measurements demonstrate robustness when applied in challenging environments with textureless regions. Overall, by addressing the inherent challenges of image-guided surgery, we lay the foundation for a more robust and accurate solution for intra- and postoperative measurements, enabling more precise, safe, and efficient surgical procedures.

An Improved TransMVSNet Algorithm for Three-Dimensional Reconstruction in the Unmanned Aerial Vehicle Remote Sensing Domain.

It is important to achieve the 3D reconstruction of UAV remote sensing images in deep learning-based multi-view stereo (MVS) vision. The lack of obvious texture features and detailed edges in UAV remote sensing images leads to inaccurate feature point matching or depth estimation. To address this problem, this study improves the TransMVSNet algorithm in the field of 3D reconstruction by optimizing its feature extraction network and costumed body depth prediction network. The improvement is mainly achieved by extracting features with the Asymptotic Pyramidal Network (AFPN) and assigning weights to different levels of features through the ASFF module to increase the importance of key levels and also using the UNet structured network combined with an attention mechanism to predict the depth information, which also extracts the key area information. It aims to improve the performance and accuracy of the TransMVSNet algorithm's 3D reconstruction of UAV remote sensing images. In this work, we have performed comparative experiments and quantitative evaluation with other algorithms on the DTU dataset as well as on a large UAV remote sensing image dataset. After a large number of experimental studies, it is shown that our improved TransMVSNet algorithm has better performance and robustness, providing a valuable reference for research and application in the field of 3D reconstruction of UAV remote sensing images.

Irregular boundaries stereo images dataset creating using depth estimation model

AbstractThis paper introduces a stereoscopic image and depth dataset created using a deep learning model. It addresses the challenge of obtaining accurate and annotated stereo image pairs with irregular boundaries for deep learning model training. Stereoscopic image and depth dataset provides a unique resource for training deep learning models to handle irregular boundary stereoscopic images, which are valuable for real-world scenarios with complex shapes or occlusions. The dataset is created using monocular depth estimation, a state-of-the-art depth estimation model, and it can be used in applications like rectifying images, estimating depth, detecting objects, and autonomous driving. Overall, this paper presents a novel dataset that demonstrates its effectiveness and potential for advancing stereo vision and developing deep learning models for computer vision applications.

Stereo Vision for Plant Detection in Dense Scenes.

Automated precision weed control requires visual methods to discriminate between crops and weeds. State-of-the-art plant detection methods fail to reliably detect weeds, especially in dense and occluded scenes. In the past, using hand-crafted detection models, both color (RGB) and depth (D) data were used for plant detection in dense scenes. Remarkably, the combination of color and depth data is not widely used in current deep learning-based vision systems in agriculture. Therefore, we collected an RGB-D dataset using a stereo vision camera. The dataset contains sugar beet crops in multiple growth stages with a varying weed densities. This dataset was made publicly available and was used to evaluate two novel plant detection models, the D-model, using the depth data as the input, and the CD-model, using both the color and depth data as inputs. For ease of use, for existing 2D deep learning architectures, the depth data were transformed into a 2D image using color encoding. As a reference model, the C-model, which uses only color data as the input, was included. The limited availability of suitable training data for depth images demands the use of data augmentation and transfer learning. Using our three detection models, we studied the effectiveness of data augmentation and transfer learning for depth data transformed to 2D images. It was found that geometric data augmentation and transfer learning were equally effective for both the reference model and the novel models using the depth data. This demonstrates that combining color-encoded depth data with geometric data augmentation and transfer learning can improve the RGB-D detection model. However, when testing our detection models on the use case of volunteer potato detection in sugar beet farming, it was found that the addition of depth data did not improve plant detection at high vegetation densities.

Innovative K-Means based machine learning method for determination of non-uniform image coordinate system in panoramic imaging: a case study with Ladybug2 camera.

Currently, the practical implementations of panoramic cameras range from vehicle navigation to space studies due to their 360-degree imaging capability in particular. In this variety of uses, it is possible to calculate three-dimensional coordinates from a panoramic image, especially using the Direct Linear Transformation (DLT) method. There are several types of omnidirectional cameras which can be classified mainly as central and non-central cameras for 360-degree imaging. The central omnidirectional cameras are those which satisfy the single-viewpoint characteristic. Multi-camera systems are usually developed for applications for which two-image stereo vision is not flexible enough to capture the environment surrounding a moving platform. Although the technology based on multi-view geometry is inexpensive, accessible, and highly customizable, multi-camera panoramic imaging systems pose a difficulty in obtaining a single projection center for the cameras. In this study, not only a defining method of the non-uniform image coordinate system is suggested by means of the K-Means algorithm for a single panoramic image, captured with a Ladybug2 panoramic camera in the panoramic calibration room but also the use of an elliptical panoramic projection coordinate system definition by Singular Value Decomposition (SVD) method in panoramic view. The results of the suggested method have been compared with the DLT algorithm for a single panoramic image which defined a conventional photogrammetric image coordinate system.