Accurate Depth Perception Research Articles

Channel Interaction and Transformer Depth Estimation Network: Robust Self-Supervised Depth Estimation Under Varied Weather Conditions

Monocular depth estimation provides low-cost environmental information for intelligent systems such as autonomous vehicles and robots, supporting sustainable development by reducing reliance on expensive, energy-intensive sensors and making technology more accessible and efficient. However, in practical applications, monocular vision is highly susceptible to adverse weather conditions, significantly reducing depth perception accuracy and limiting its ability to deliver reliable environmental information. To improve the robustness of monocular depth estimation in challenging weather, this paper first utilizes generative models to adjust image exposure and generate synthetic images of rainy, foggy, and nighttime scenes, enriching the diversity of the training data. Next, a channel interaction module and Multi-Scale Fusion Module are introduced. The former enhances information exchange between channels, while the latter effectively integrates multi-level feature information. Finally, an enhanced consistency loss is added to the loss function to prevent the depth estimation bias caused by data augmentation. Experiments on datasets such as DrivingStereo, Foggy CityScapes, and NuScenes-Night demonstrate that our method, CIT-Depth, exhibits superior generalization across various complex conditions.

Assessing Depth Perception in VR and Video See-Through AR: A Comparison on Distance Judgment, Performance, and Preference.

Spatial User Interfaces along the Reality-Virtuality continuum heavily depend on accurate depth perception. However, current display technologies still exhibit shortcomings in the simulation of accurate depth cues, and these shortcomings also vary between Virtual or Augmented Reality (VR, AR: eXtended Reality (XR) for short). This article compares depth perception between VR and Video See-Through (VST) AR. We developed a digital twin of an existing office room where users had top erform five depth-dependent tasks in VR and VST AR. Thirty-two participants took part in a user study using a 1 × 4 within-subjects design. Our results reveal higher misjudgment rates in VST AR due to conflicting depth cues between virtual and physical content. Increased head movements observed in participants were interpreted as a compensatory response to these conflicting cues. Furthermore, a longer task completion time in the VST AR condition indicates a lower task performance in VST AR. Interestingly, while participants rated the VR condition as easier and contrary to the increased misjudgments and lower performance with the VST AR display, a majority still expressed a preference for the VST AR experience. We discuss and explain these findings with the high visual dominance and referential power of the physical content in the VST AR condition, leading to a higher spatial presence and plausibility.

Impact of Background, Foreground, and Manipulated Object Rendering on Egocentric Depth Perception in Virtual and Augmented Indoor Environments.

This research investigated how the similarity of the rendering parameters of background and foreground objects affected egocentric depth perception in indoor virtual and augmented environments. We refer to the similarity of the rendering parameters as visual 'congruence'. Study participants manipulated the depth of a sphere to match the depth of a designated target peg. In the first experiment, the sphere and peg were both virtual, while in the second experiment, the sphere is virtual and the peg is real. In both experiments, depth perception accuracy was found to depend on the levels of realism and congruence between the sphere, pegs, and background. In Experiment 1, realistic backgrounds lead to overestimation of depth, but resulted in underestimation when the background was virtual, and when depth cues were applied to the sphere and target peg. In Experiment 2, background and target pegs were real but matched with the virtual sphere; in comparison to Experiment 1, realistically rendered targets prompted an underestimation and more accuracy with the manipulated object. These findings suggest that congruence can affect distance estimation and the underestimation effect in the AR environment resulted from increased graphical fidelity of the foreground target and background.

Viewpoint-dependent highlight depiction with microdisparity for autostereoscopic displays

The rendering of specular highlights is a critical aspect of 3D rendering on autostereoscopic displays. However, the conventional highlight rendering techniques on autostereoscopic displays result in depth conflicts between highlights and diffuse surfaces. To address this issue, we propose a viewpoint-dependent highlight depiction method with head tracking, which incorporates microdisparity of highlights in binocular parallax and preserves the motion parallax of highlights. Our method was found to outperform physical highlight depiction and highlight depiction with microdisparity in terms of depth perception and realism, as demonstrated by experimental results. The proposed approach offers a promising alternative to traditional physical highlights on autostereoscopic displays, particularly in applications that require accurate depth perception.

Effect of expansive optic flow and lateral motion parallax on depth estimation with normal and artificially reduced acuity.

When an observer moves in space, the retinal projection of a stationary object either expands if the motion is toward the object or shifts horizontally if the motion contains a lateral component. This study examined the impact of expansive optic flow and lateral motion parallax on the accuracy of depth perception for observers with normal or artificially reduced acuity and asked whether any benefit is due to the continuous motion or to the discrete object image displacement. Stationary participants viewed a virtual room on a computer screen. They used an on-screen slider to estimate the depth of a target object relative to a reference object after seeing 2-second videos simulating five conditions: static viewing, expansive optic flow, and lateral motion parallax in either continuous motion or image displacement. Ten participants viewed the stimuli with normal acuity in Experiment 1 and 11 with three levels of artificially reduced acuity in Experiment 2. Linear regression models represented the relationship between the depth estimates of participants and the ground truth. Lateral motion parallax produced more accurate depth estimates than expansive optic flow and static viewing. Depth perception with continuous motion was more accurate than that with displacement under mild and moderate, but not severe, acuity reduction. For observers with both normal and artificially reduced acuity, lateral motion parallax was more helpful for object depth estimation than expansive optic flow, and continuous motion parallax was more helpful than object image displacement.

Depth Completion in Autonomous Driving: Adaptive Spatial Feature Fusion and Semi-Quantitative Visualization

The safety of autonomous driving is closely linked to accurate depth perception. With the continuous development of autonomous driving, depth completion has become one of the crucial methods in this field. However, current depth completion methods have major shortcomings in small objects. To solve this problem, this paper proposes an end-to-end architecture with adaptive spatial feature fusion by encoder–decoder (ASFF-ED) module for depth completion. The architecture is built on the basis of the network architecture proposed in this paper, and is able to extract depth information adaptively with different weights on the specified feature map, which effectively solves the problem of insufficient depth accuracy of small objects. At the same time, this paper also proposes a depth map visualization method with a semi-quantitative visualization, which makes the depth information more intuitive to display. Compared with the currently available depth map visualization methods, this method has stronger quantitative analysis and horizontal comparison ability. Through experiments of ablation study and comparison, the results show that the method proposed in this paper exhibits a lower root-mean-squared error (RMSE) and better small object detection performance on the KITTI dataset.

Effects of Luminance Contrast on Depth Perception in Optical See-Through Augmented Reality

ABSTRACT Augmented reality (AR) technology presents the rendered virtual object in the real world whose depth cues can assist the depth estimation of the virtual object. For many AR applications, the successful interaction between the real world and the virtual object relies on the accuracy of depth perception. In the past, the research of depth perception in AR mainly focused on the impact of virtual object properties, including the size, color, and shading model. In this study, an experiment has been conducted to investigate the influence of ambient luminance, virtual object luminance, and shading model which is a luminance-related parameter on depth perception using an optical see-through head-mounted display. Results show that a higher luminance contrast between the real scene and the virtual object leads to a lower accuracy of depth perception in AR.

Design, integration, and evaluation of a robotic peach packaging system based on deep learning

Fruit packaging is one of the most time-consuming and labor-intensive tasks during postharvest commercialization. With the aging of the global population, it is necessary to apply robot to replace the manual manipulation. However, robotic packaging for fragile fruit is more complicated and difficult, compared with other postharvest processes, such as quality detection and grading. In this study, for better positioning accuracy and grasping robustness, we developed a prototype for peach packaging robot based on deep learning. First, the dataset for peach object detection was built, and YOLO v5 models with different width and depth were trained in an end-to-end manner on this dataset. Considering the requirements on both accuracy and real-time performance in fruit postharvest processing, YOLO v5 - S was finally adopted as the peach detection model for robotic manipulation, which can achieve mAP-50 = 0.996 on validation set and runs at 142.86 fps on RTX 3060. Next, the “Eye-on-Base” hand eye calibration method was used to solve the coordinate transformation matrix from the camera coordinate system to the robot base coordinate system. The results on landmark positioning experiment showed that the positioning error along X and Y axes was 4.87 mm and 5.00 mm on average, respectively. The positioning error alone Z direction was 18.47 mm on average, which was caused mainly by the depth perception error of the RGB-D camera. In the grasping experiment, the influence of the depth perception accuracy was explored, the grasping success rate for small, medium, and large size of peaches was 100 %, 97 %, and 97 %, respectively. Besides, the entire pipeline proposed in this study took 252.81 ms on average for depth perception, object detection, coordinate transformation, and grasping path planning. Finally, the early stage bruise of the peaches was also evaluated through SFDI technology. In general, this research provided a feasible and reliable scheme for the fruit packaging robot, which is potential to be deployed in postharvest commercialization.

Shedding Light on Cast Shadows: An Investigation of Perceived Ground Contact in AR and VR.

Virtual objects in augmented reality (AR) often appear to float atop real world surfaces, which makes it difficult to determine where they are positioned in space. This is problematic as many applications for AR require accurate spatial perception. In the current study, we examine how the way we render cast shadows-which act as an important monocular depth cue for creating a sense of contact between an object and the surface beneath it-impacts spatial perception. Over two experiments, we evaluate people's sense of surface contact given both traditional and non-traditional shadow shading methods in optical see-through augmented reality (OST AR), video see-through augmented reality (VST AR), and virtual reality (VR) head-mounted displays. Our results provide evidence that nontraditional shading techniques for rendering shadows in AR displays may enhance the accuracy of one's perception of surface contact. This finding implies a possible tradeoff between photorealism and accuracy of depth perception, especially in OST AR displays. However, it also supports the use of more stylized graphics like non-traditional cast shadows to improve perception and interaction in AR applications.

Deep learning of monocular depth, optical flow and ego-motion with geometric guidance for UAV navigation in dynamic environments

Computer vision-based depth estimation and visual odometry provide perceptual information useful for robot navigation tasks like obstacle avoidance. However, despite the proliferation of state-of-the-art convolutional neural network (CNN) models for monocular depth, ego-motion and optical flow estimation, a relatively low volume of work has been reported on their practical applications in unmanned aerial vehicle (UAV) navigation. This is due to well-known challenges — embedded hardware constraints, viewpoint variations, scarcity of aerial image datasets, and intricacies of dynamic environments. We address these limitations to facilitate real-world deployment of CNN in UAV navigation. First, we devise efficient confidence weighted adaptive network (Cowan) training framework that iteratively leverages intermediate prediction confidences to enforce cross-task consistency over corresponding image regions. This achieves competitive accuracy with a lightweight CNN capable of real-time execution on resource-constrained embedded systems. Second, we devise a test-time refinement method that adapts the network to dynamic environments while simultaneously improving accuracy. To accomplish this, we first update ego-motion using pose information from on-board inertial measurement unit (IMU). Then, we decompose the UAV’s motion into constituent vectors, and for each axis, we formulate geometric relationships between depth and translation. Based on this information, we triangulate corresponding points acquired through optical flow. Finally, we enforce geometric consistency between the initially updated pose and triangulated depth. Cowan with geometric guided refinement (Cowan-GGR) achieves significant accuracy and robustness. Field tests show the proposed model is capable of accurate depth and object-level motion perception in real-world dynamic environments, thus proving its efficacy in facilitating UAV navigation.

LiDAR Based Self-Driving Car

LiDAR, typically used as an acronym for “’light detection and ranging’”, is essentially a sonar that uses pulsed laser waves to map the distance to surrounding objects. It is used by a large number of autonomous vehicles to navigate environments in real time. Its advantages include impressively accurate depth perception, which allows LiDAR to know the distance to an object to within a few centimetres, up to 60 metres away. It’s also highly suitable for 3D mapping, which means returning vehicles can then navigate the environment predictably —a significant benefit for most self-driving technologies. One of the key strengths of LiDAR is the number of areas that show potential for improvement. These include solid-state sensors, which could reduce its cost tenfold, sensor range increases of up to 200m, and 4-dimensional LiDAR, which senses the velocity of an object as well as its position in 3-D space. However, despite these exciting advances, LiDAR is still hindered by a key factor; its significant cost. LiDAR is not the only self-driving detection technology, with cameras as the major rival, championed by Tesla as the best way forward. Elon Musk has described LiDAR as “a fool’s errand” and “unnecessary”. The argument runs that humans drive based only on ambient visible light, so robots should equally be able to. A camera is significantly smaller and cheaper than LiDAR (although more of them are needed), and has the advantage of seeing in better resolution and in colour, meaning it can read traffic lights and signs. However, cameras have a wide host of characteristics that make them tricky to use in common driving conditions. Whereas LiDAR uses near infra-red light, cameras use visible light, and are thus more susceptible to issues when faced with rain, fog, or even some textures. In addition, LiDARs do not depend on ambient light, generating their own light pulses, whereas cameras are more sensitive to sudden light changes, direct sunlight and even raindrops

Design and analysis of depth cues on depth perception in interactive mixed reality simulation systems

AbstractImplementing an interactive mixed reality simulation system is an economical and risk‐free method for skill training. However, research to enhance the accuracy of mixed reality simulation has always been a challenge. One of the key indicators of a simulation's accuracy is the user's depth perception. Adding depth cues to the simulation scene can effectively increase the accuracy of depth perception. Therefore, in this study, we designed five depth cues to find the one most beneficial for depth perception: (1) depth bar, (2) location ball, (3) fixed window, (4) 3D wireframe, and (5) movable silhouettes. These depth cues were tested using an experimental system with two sets of experimental settings. The first experiment compared the five depth cues at five different viewing distances and showed that the movable silhouettes benefited depth perception accuracy the most. The second experiment compared depth perception accuracies among three different horizontal tilt settings, showing that depth perception accuracy is dependent on the horizontal tilt. Furthermore, different depth perception accuracies were obtained with and without depth cues, proving that the effects from the horizontal tilts were independent of depth cues. The results of these studies provide references for designing interactive mixed reality simulation systems.

Two arms-three instruments robot-assisted laparoscopic hysterectomy: A reproducible technique

Although laparoscopic hysterectomy has been used for more than 3 decades, it is not universally adopted due to steep learning curve. The robotic platform can bridge this gap and reduce the need for open hysterectomy with enhanced dexterity and accurate depth perception by 3D vision and wristed intuitive movements. This technical note introduces a two arms-three instruments “Sinha-Apollo technique” for da Vinci Si system for performing robotic-assisted laparoscopic hysterectomy in simplified and reproducible steps.

Millimeter Wave MIMO-Based Depth Maps for Wireless Virtual and Augmented Reality

Augmented and virtual reality systems (AR/VR) are rapidly becoming key components of the wireless landscape. For immersive AR/VR experience, these devices should be able to construct accurate depth perception of the surrounding environment. Current AR/VR devices rely heavily on using RGB-D depth cameras to achieve this goal. The performance of these depth cameras, however, has clear limitations in several scenarios, such as the cases with shiny objects, dark surfaces, and abrupt color transition among other limitations. In this paper, we propose a novel solution for AR/VR depth map construction using mmWave MIMO communication transceivers. This is motivated by the deployment of advanced mmWave communication systems in future AR/VR devices for meeting the high data rate demands and by the interesting propagation characteristics of mmWave signals. Accounting for the constraints on these systems, we develop a comprehensive framework for constructing accurate and high-resolution depth maps using mmWave systems. In this framework, we developed new sensing beamforming codebook approaches that are specific for the depth map construction objective. Using these codebooks, and leveraging tools from successive interference cancellation, we develop a joint beam processing approach that can construct high-resolution depth maps using practical mmWave antenna arrays. Extensive simulation results highlight the potential of the proposed solution in building accurate depth maps. Further, these simulations show the promising gains of mmWave based depth perception compared to RGB-based approaches in several important use cases.

Simple Stereoscopic Display of 3-Dimensional Living Heart Anatomy Relevant to Electrophysiological Practice.

Comprehensive appreciation of the 3-dimensional anatomy of the living heart is essential for accurate diagnosis and establishment of safe interventional treatment by clinical electrophysiologists. Three-dimensional images displayed on a 2-dimensional surface usually undergo distortion and are not considered a real 3-dimensional representation because they cannot provide accurate depth perception. Currently, the availability of 3-dimensional visualization techniques, including 3-dimensional printing, 3-dimensional projectors, 3-dimensional monitors, and virtual reality, is limited owing to their limited user-friendliness and high costs. The authors discuss an alternative conventional approach of a cross-eyed method for viewing stereoscopically displayed images for better appreciation of real 3-dimensional images based on a simple technique.

Overestimation and contraction biases of depth information stored in working memory depend on spatial configuration.

Depth perception is essential for effective interaction with the environment. Although the accuracy of depth perception has been studied extensively, it is unclear how accurate the depth information is stored in working memory. In this study, we investigated the accuracy and systematic biases of depth representation by a delayed estimation task. The memory array consisted of items presented at various stereoscopic depth positions, and the participants were instructed to estimate the depth position of one target item after a retention interval. We examined the effect of spatial configuration by comparing the memory performance in the whole-display condition where non-target memory items were present during retrieval with that in the single-display condition where non-target memory items were absent. In the single-display condition, we found an overestimation bias that the depth estimates were farther than the corresponding depth positions defined by disparity, and a contraction bias that the stored depth positions near the observer were overestimated and those far from the observer were underestimated. The magnitude of these biases increased with the number of to-be-stored items. However, in the whole-display condition, the overestimation bias was corrected and the contraction bias did not increase with the number of to-be-stored items. Our findings suggested that the number of to-be-stored items could affect the accuracy of depth working memory, and its effect depended crucially on whether the information of spatial configuration of memory display was available at the retrieval stage.

Compact single-shot metalens depth sensors inspired by eyes of jumping spiders

Jumping spiders (Salticidae) rely on accurate depth perception for predation and navigation. They accomplish depth perception, despite their tiny brains, by using specialized optics. Each principal eye includes a multitiered retina that simultaneously receives multiple images with different amounts of defocus, and from these images, distance is decoded with relatively little computation. We introduce a compact depth sensor that is inspired by the jumping spider. It combines metalens optics, which modifies the phase of incident light at a subwavelength scale, with efficient computations to measure depth from image defocus. Instead of using a multitiered retina to transduce multiple simultaneous images, the sensor uses a metalens to split the light that passes through an aperture and concurrently form 2 differently defocused images at distinct regions of a single planar photosensor. We demonstrate a system that deploys a 3-mm-diameter metalens to measure depth over a 10-cm distance range, using fewer than 700 floating point operations per output pixel. Compared with previous passive depth sensors, our metalens depth sensor is compact, single-shot, and requires a small amount of computation. This integration of nanophotonics and efficient computation brings artificial depth sensing closer to being feasible on millimeter-scale, microwatts platforms such as microrobots and microsensor networks.

A Distortion Model of Laser Sheet for a Laser Line Scanner With Large Fan Angle

This article proposes an analytical method of computing distorted laser sheet of a laser line scanner and compensating it. In this article, a distortion model of laser sheet generated from a laser line scanner with large fan angle is derived from physical principles to achieve accurate depth perception even around its edges on which significant depth estimation errors occur in existing algorithms. From a laser beam incident obliquely on a contact surface of two cylindrical lenses, a curved laser sheet is expressed in terms of the nonzero incident angle by the laws of geometrical optics. From the mathematical model of the distorted laser sheet, the incident angle is estimated through an optimization technique and then its estimate is used for depth computation. It is shown through simulations and experiments that the proposed distortion model enables a proper compensation scheme and hence the distortion-compensated depth estimation errors are reduced all over the range of interest, specially much around the both side edges of the laser sheet. Quantitatively in comparison to the compensation-free method, the depth estimation error is improved from 225 to 28 mm on average and from 1781 to 187 mm for worst-case scenarios near edges of a laser sheet.

What are the skills that represent expert-level laparoscopic suturing? A Delphi Study.

As the field of laparoscopic surgery grows, the need for a valid assessment of laparoscopic surgical skills becomes apparent. The objective of this study was to identify laparoscopic suturing skills reflective of performing at an expert level. A Delphi Study was conducted with expert laparoscopic surgeons. Tasks were considered to meet criteria for consensus after 80% of survey participants answered "strongly agree." Results of the previous round of surveys were shared with participants at the start of the next round. The predefined endpoint for the Delphi was set as maximum of three rounds, reaching 80% consensus on each skill, or if > 50% of initial respondents fail to return for subsequent surveys. After three rounds of surveys completed by 13 advanced laparoscopic surgeons, 8 observed skills met consensus (> 80% selected "strongly agree"): forehand suturing, continuous suturing, avoiding tissue trauma, achieving a technically acceptable final product (tight closure), tying a secure knot at the end of suturing, maintaining/checking for appropriate tightness of running suture, accurate depth perception, and loading the needle in < 3 attempts. Skills that approached consensus (with > 50% ranking them as "strongly agree") included correct penetration of target tissue, ability to create a surgeon's knot, awareness of needle tip position, confined space suturing, following the curve of the needle, smooth movements of instruments, loading the needle correctly, adjusting for tissue thickness, and bimanual dexterity. Using Delphi methodology, we identified the skills that are representative of expert laparoscopic suturing. This information can be used to create an assessment tool for this domain. Validation of such a tool will help advance the surgical field towards true competency-based assessment of expertise.

Use of cues in virtual reality depends on visual feedback

3D motion perception is of central importance to daily life. However, when tested in laboratory settings, sensitivity to 3D motion signals is found to be poor, leading to the view that heuristics and prior assumptions are critical for 3D motion perception. Here we explore an alternative: sensitivity to 3D motion signals is context-dependent and must be learned based on explicit visual feedback in novel environments. The need for action-contingent visual feedback is well-established in the developmental literature. For example, young kittens that are passively moved through an environment, but unable to move through it themselves, fail to develop accurate depth perception. We find that these principles also obtain in adult human perception. Observers that do not experience visual consequences of their actions fail to develop accurate 3D motion perception in a virtual reality environment, even after prolonged exposure. By contrast, observers that experience the consequences of their actions improve performance based on available sensory cues to 3D motion. Specifically, we find that observers learn to exploit the small motion parallax cues provided by head jitter. Our findings advance understanding of human 3D motion processing and form a foundation for future study of perception in virtual and natural 3D environments.