Stereo Video Research Articles

Formulating facial mesh tracking as a differentiable optimization problem: a backpropagation-based solution

Facial mesh tracking enables the production of topologically consistent 3D facial meshes from stereo video input captured by calibrated cameras. This technology is an integral part of many digital human applications, such as personalized avatar creation, audio-driven 3D facial animation, and talking face video generation. Currently, most facial mesh tracking methods are built on computer graphics techniques, which involve complex procedures and often necessitate human annotation within pipelines. As a result, these approaches are difficult to implement and hard to generalize across various scenarios. We propose a backpropagation-based solution that formulates facial mesh tracking as a differentiable optimization problem called the BPMT. Our solution leverages visual clues extracted from the stereo input to estimate vertex-wise geometry and texture information. The BPMT is composed of two steps: automatic face analysis and mesh tracking. In the first step, a range of visual clues are automatically extracted from the input, including facial point clouds, multi-view 2D landmarks, 3D landmarks in the world coordinate system, motion fields, and image masks. The second step can be viewed as a differentiable optimization problem, with constraints comprising stereo video input and facial clues. The primary objective is to achieve topologically consistent 3D facial meshes across frames. Additionally, the parameters to be optimized encompass the positions of free-form deformed vertices and a shared texture UV map. Furthermore, the 3D morphable model (3DMM) is introduced as a form of regularization to enhance the convergence of the optimization process. Leveraging the fully developed backpropagation software, we progressively register the facial meshes to the recorded object, generating high-quality 3D faces with consistent topologies. The BPMT requires no manual labeling within the pipeline, making it suitable for producing large-scale stereo facial data. Moreover, our method exhibits a high degree of flexibility and extensibility, positioning it as a promising platform for future research in the community.

Surgical Tattoos in Infrared: A Dataset for Quantifying Tissue Tracking and Mapping.

Quantifying performance of methods for tracking and mapping tissue in endoscopic environments is essential for enabling image guidance and automation of medical interventions and surgery. Datasets developed so far either use rigid environments, visible markers, or require annotators to label salient points in videos after collection. These are respectively: not general, visible to algorithms, or costly and error-prone. We introduce a novel labeling methodology along with a dataset that uses said methodology, Surgical Tattoos in Infrared (STIR). STIR has labels that are persistent but invisible to visible spectrum algorithms. This is done by labelling tissue points with IR-fluorescent dye, indocyanine green (ICG), and then collecting visible light video clips. STIR comprises hundreds of stereo video clips in both in vivo and ex vivo scenes with start and end points labelled in the IR spectrum. With over 3,000 labelled points, STIR will help to quantify and enable better analysis of tracking and mapping methods. After introducing STIR, we analyze multiple different frame-based tracking methods on STIR using both 3D and 2D endpoint error and accuracy metrics. STIR is available at https://dx.doi.org/10.21227/w8g4-g548.

Stereoscopic video deblurring transformer

Stereoscopic cameras, such as those in mobile phones and various recent intelligent systems, are becoming increasingly common. Multiple variables can impact the stereo video quality, e.g., blur distortion due to camera/object movement. Monocular image/video deblurring is a mature research field, while there is limited research on stereoscopic content deblurring. This paper introduces a new Transformer-based stereo video deblurring framework with two crucial new parts: a self-attention layer and a feed-forward layer that realizes and aligns the correlation among various video frames. The traditional fully connected (FC) self-attention layer fails to utilize data locality effectively, as it depends on linear layers for calculating attention maps The Vision Transformer, on the other hand, also has this limitation, as it takes image patches as inputs to model global spatial information. 3D convolutional neural networks (3D CNNs) process successive frames to correct motion blur in the stereo video. Besides, our method uses other stereo-viewpoint information to assist deblurring. The parallax attention module (PAM) is significantly improved to combine the stereo and cross-view information for more deblurring. An extensive ablation study validates that our method efficiently deblurs the stereo videos based on the experiments on two publicly available stereo video datasets. Experimental results of our approach demonstrate state-of-the-art performance compared to the image and video deblurring techniques by a large margin.

DCM2Net: an improved face recognition model for panoramic stereoscopic videos.

The panoramic stereo video has brought a new visual experience for the audience with its immersion and stereo effect. In panoramic stereo video, the face is an important element. However, the face image in panoramic stereo video has varying degrees of deformation. This brings new challenges to face recognition. Therefore, this paper proposes a face recognition model DCM2Net (Deformable Convolution MobileFaceNet) for panoramic stereo video. The model mainly integrates the feature information between channels during feature fusion, redistributes the information between channels in the deeper part of the network, and fully uses the information between different channels for feature extraction. This paper also built a panoramic stereo video live system, using the DCM2Net model to recognize the face in panoramic stereo video, and the recognition results are displayed in the video. After experiments on different datasets, the results show that our model has better results on popular datasets and panoramic datasets.

Stereo Vision Conversion from Planar Videos Based on Temporal Multiplane Images

With the rapid development of 3D movie and light-field displays, there is a growing demand for stereo videos. However, generating high-quality stereo videos from planar videos remains a challenging task. Traditional depth-image-based rendering techniques struggle to effectively handle the problem of occlusion exposure, which occurs when the occluded contents become visible in other views. Recently, the single-view multiplane images (MPI) representation has shown promising performance for planar video stereoscopy. However, the MPI still lacks real details that are occluded in the current frame, resulting in blurry artifacts in occlusion exposure regions. In fact, planar videos can leverage complementary information from adjacent frames to predict a more complete scene representation for the current frame. Therefore, this paper extends the MPI from still frames to the temporal domain, introducing the temporal MPI (TMPI). By extracting complementary information from adjacent frames based on optical flow guidance, obscured regions in the current frame can be effectively repaired. Additionally, a new module called masked optical flow warping (MOFW) is introduced to improve the propagation of pixels along optical flow trajectories. Experimental results demonstrate that the proposed method can generate high-quality stereoscopic or light-field videos from a single view and reproduce better occluded details than other state-of-the-art (SOTA) methods. https://github.com/Dio3ding/TMPI

Estimating enhanced fish production on restored shellfish reefs using automated data collection from underwater videos

Abstract Shellfish reefs across the world have been heavily degraded by mechanical harvesting, disease and declining water quality. In southern Australia, where substantial losses have occurred, government and non‐government efforts to restore functioning reefs are now underway with a strong focus on enhancing fish productivity. However, the capacity of these restored reefs to enhance fish production remains unknown, hampering estimates of return on investment. We quantify the density differences of newly recruited juvenile fish and other nekton on these restored reefs, relative to those on unrestored, unstructured habitat. Fish were surveyed at three paired reef‐unstructured locations using 169 unbaited stereo video deployments during three periods over 12 months (2022–2023). We used automation software, FishID, to automatically identify, size and count fish in videos. We subsequently applied known growth and mortality parameters to model enhancement of fish productivity. Sixteen species occurred as new recruits, with all but two found at higher densities on reefs than unstructured habitat. Enhancement of fish production from subtidal restored shellfish reefs from a single year's cohort is estimated to be, on average, 6186 kg ha−1 year−1 (SD 1802) after enough time has elapsed for all species to have matured. Species harvested commercially or recreationally contributed 98% of that production (6083 kg ha−1 year−1, SD 1797). Enhancement varied greatly among locations, ranging from 12,738 kg ha−1 year−1 (SD 2894), which is the highest yet recorded anywhere, to 1.4 kg ha−1 year−1 (SD 0.9). The lack of juvenile fish at the location with the lowest estimated enhancement might be explained by the impact of overfishing on recruitment of key species or by an abundance of alternative habitat for juvenile fish. Synthesis and applications. The combination of underwater videos with automated data extraction provides a reliable, cost‐effective method for surveying fish on oyster reefs. By quantifying enhanced fish productivity on reefs, we provide estimates that will underpin calculations of ecological, social and financial benefits, supporting the business case for scaling‐up restoration efforts.

IterDepth: Iterative Residual Refinement for Outdoor Self-Supervised Multi-Frame Monocular Depth Estimation

Self-supervised monocular depth estimation has been a challenging task in computer vision for a long time, and it relies on only monocular or stereo video for its supervision. To address the challenge, we propose a novel multi-frame monocular depth estimation method called IterDepth, which is based on an iterative residual refinement network. IterDepth extracts depth features from consecutive frames and computes a 3D cost volume measuring the difference between current and previous features transformed by PoseCNN (pose estimation convolutional neural network). We reformulate depth prediction as a residual learning problem, revamping the dominating depth regression to enable high-accuracy multi-frame monocular depth estimation. Specifically, we design a gated recurrent depth fusion unit that seamlessly blends depth features from the cost volume, image features, and the depth prediction. The unit updates the hidden states and refines the depth map through iterative refinement, achieving more accurate predictions than existing methods. Our experiments on the KITTI dataset demonstrate that IterDepth is 7× faster in terms of FPS (frames per second) than the recent state-of-the-art DepthFormer model with competitive performance. We also test IterDepth on the Cityscapes dataset to showcase its generalization capability in other real-world environments. Moreover, IterDepth can balance accuracy and computational efficiency by adjusting the number of refinement iterations and performs competitively with other CNN-based monocular depth estimation approaches. Code will be made publicly available.

Viewing the Forest in 3-D: How Spherical Stereo Videos Enable Low-Cost Reconstruction of Forest Plots

Viewing the Forest in 3-D: How Spherical Stereo Videos Enable Low-Cost Reconstruction of Forest Plots

Effect of Stereo Video Acquisition Distortion on Visually Induced Motion Sickness

Effect of Stereo Video Acquisition Distortion on Visually Induced Motion Sickness

Ventral wing hairs provide tactile feedback for aerial prey capture in the big brown bat, Eptesicus fuscus.

Bats rely on their hand-wings to execute agile flight maneuvers, to grasp objects, and cradle young. Embedded in the dorsal and ventral membranes of bat wings are microscopic hairs. Past research findings implicate dorsal wing hairs in airflow sensing for flight control, but the function of ventral wing hairs has not been previously investigated. Here, we test the hypothesis that ventral wing hairs carry mechanosensory signals for flight control, prey capture, and handling. To test this hypothesis, we used synchronized high-speed stereo video and audio recordings to quantify flight and echolocation behaviors of big brown bats (Eptesicus fuscus) engaged in an aerial insect capture task. We analyzed prey-capture strategy and performance, along with flight kinematics, before and after depilation of microscopic hairs from the bat's ventral wing and tail membranes. We found that ventral wing hair depilation significantly impaired the bat's prey-capture performance. Interestingly, ventral wing hair depilation also produced increases in the bat's flight speed, an effect previously attributed exclusively to airflow sensing along the dorsal wing surface. These findings demonstrate that microscopic hairs embedded in the ventral wing and tail membranes of insectivorous bats provide mechanosensory feedback for prey handling and flight control.

Stereoscopic Video Quality Assessment in the Context of Internet of Things

Under the background of Internet of Things (IoT), users pursue a higher quality of immer-sive visual experience, including virtual reality (VR), augmented reality and stereoscopic information. In order to ensure the quality of experience (QoE), an effective quality assessment method is proposed to accurately determine the quality of information. Specifically, for the large amount of stereo video data, the representative sequence of stereo video is extract to reduce the data volume. Inspired by binocular channel theory, operators of the difference map and the novel fusion map are executed on these sequences. Then a dynamic texture descriptor, known as volume local binary pattern (VLBP), is employed to represent the spatio-temporal domain to predict the stereo video quality. Experiments have been executed on two public databases and results show that our method has excellent quality prediction capability.

Transformer with Hybrid Attention Mechanism for Stereo Endoscopic Video Super Resolution

With stereo cameras becoming widely used in invasive surgery systems, stereo endoscopic images provide important depth information for delicate surgical tasks. However, the small size of sensors and their limited lighting conditions lead to low-quality and low-resolution endoscopic images and videos. In this paper, we propose a stereo endoscopic video super-resolution method using transformer with a hybrid attention mechanism named HA-VSR. Stereo video SR aims to reconstruct high-resolution (HR) images from corresponding low-resolution (LR) videos. In our method, the stereo correspondence and temporal correspondence are incorporated into the HA-VSR model. Specifically, the Swin transformer architecture is utilized in proposed framework with hybrid attention mechanisms. The parallel attention mechanism is utilized by using the symmetry and consistency of left and right images, and the temporal attention mechanism is utilized by using the consistency of consecutive frames. Detailed quantitative evaluation and experiments on two datasets show the proposed model achieves advanced SR reconstruction performance, showing that the proposed stereo VSR framework outperforms alternative approaches.

Soft tissue monitoring of the surgical field: detection and tracking of breast surface deformations.

Deformable object tracking is common in the computer vision field, with applications typically focusing on nonrigid shape detection and usually not requiring specific 3D point localization.In surgical guidance however, accurate navigation is intrinsically linked to precise correspondence of tissue structure. This work presents a contactless, automated fiducial acquisition method using stereo video of the operating field to provide reliable fiducial localization for an image guidance framework in breast conserving surgery. On n=8 breasts from healthy volunteers, the breast surface was measured throughout the full range of arm motion in a supine mock-surgical position. Using hand-drawn inked fiducials, adaptive thresholding, and KAZE feature matching, precise three-dimensional fiducial locations were detected and tracked through tool interference, partial and complete marker occlusions, significant displacements and nonrigid shape distortions. Compared to digitization with a conventional optically tracked stylus, fiducials were automatically localized with 1.6 ± 0.5 mm accuracy and the two measurement methods did not significantly differ. The algorithm provided an average false discovery rate <0.1% with all cases' rates below 0.2%. On average, 85.6 ± 5.9% of visible fiducials were automatically detected and tracked, and 99.1 ± 1.1% of frames provided only true positive fiducial measurements, which indicates the algorithm achieves a data stream that can be used for reliable on-line registration. Tracking is robust to occlusions, displacements, and most shape distortions. This work-flow friendly data collection method provides highly accurate and precise three-dimensional surface data to drive an image guidance system for breast conserving surgery.

Internet of Things for Diagnosis of Alzheimer’s Disease: A Multimodal Machine Learning Approach Based on Eye Movement Features

Alzheimer’s disease (AD) is a degenerative neurological disease that occurs in the elderly with typical symptoms of decline in cognition, manifested by eye movement behaviors. The key to AD treatment requires early detection of cognitive impairment, which relies on frequent medical screening. This paper proposes an Internet of Things (IoT) architecture constructed with eye-tracker (ET) nodes and cloud-based diagnosis enabled by machine learning (ML), which can provide convenient screening of oculomotor abnormalities and automatic identification of early-stage AD. The bespoke ET nodes collect three-dimensional (3-D) oculomotor responses from diverse stereo video stimulation trials and transmit data into a dedicated multimodal machine learning (MMML) algorithm in the cloud. The algorithm incorporates multimodal features extracted from diverse oculomotor types to enhance the classification accuracy and optimized data dimension reduction in feature fusion to improve the classifier’s performance. From evaluation, the proposed method can distinguish AD patients from the control group with 86% accuracy (ACC), 78% true positive rate (TPR), and 90% positive predictive value (PPV). The results confirm the effectiveness of our MMML algorithm in AD diagnosis with the fusion of multimodal oculomotor features and prove the feasibility of our IoT-powered eye-tracking solution for AD screening.

Coral assemblage structure and condition of the San Felipe Keys National Park, Cuba

Coral reefs provide ecosystem services to humans, essential to protection through different conservation strategies. However, planning and implementing marine conservation management actions must be based on the best available science. Information gaps have resulted in limited knowledge of conservation priorities and a weak conflict resolution and decision-making capacity. The San Felipe Keys National Park (SFKNP), located in the southwestern region of Cuba, is an understudied area with 40 km long forereefs, providing services mainly associated with fisheries (lobster and finfish). This work analyzed the hermatypic coral assemblage structure and condition within the SFKNP and the influence of geographic and environmental factors in its variation. Using stereo videos, we assessed nine coral reef sites between 2013 and 2015, generating 597 transects, in which 10341 colonies, 46 species, and 17 functional entities were found. The dominant species (Siderastrea siderea, Porites astreoides, Agaricia agaricites, and Orbicella faveolata) are mainly massive species with higher depth range, and that visibility in the bottom and habitat heterogeneity explained 57% of coral assemblages’ variation. Besides, the percentage of living coral cover (10%–15%), the prevalence of small-sized colonies (22 cm average), and the high incidence of recent mortality (2%–8% of colonies) indicate the presence of environmental stressors in the SFKNP, such as water turbidity, generated by the natural runoff from the Gulf of Batabanó. These results highlight the importance of enhancing the awareness of the competent authorities and promoting integrated management of the entire western region of the Gulf of Batabanó.

Viewpoint-Controllable Telepresence: A Robotic-Arm-Based Mixed-Reality Telecollaboration System.

In mixed-reality (MR) telecollaboration, the local environment is remotely presented to a remote user wearing a virtual reality (VR) head-mounted display (HMD) via a video capture device. However, remote users frequently face challenges in naturally and actively manipulating their viewpoints. In this paper, we propose a telepresence system with viewpoint control, which involves a robotic arm equipped with a stereo camera in the local environment. This system enables remote users to actively and flexibly observe the local environment by moving their heads to manipulate the robotic arm. Additionally, to solve the problem of the limited field of view of the stereo camera and limited movement range of the robotic arm, we propose a 3D reconstruction method combined with a stereo video field-of-view enhancement technique to guide remote users to move within the movement range of the robotic arm and provide them with a larger range of local environment perception. Finally, a mixed-reality telecollaboration prototype was built, and two user studies were conducted to evaluate the overall system. User study A evaluated the interaction efficiency, system usability, workload, copresence, and user satisfaction of our system from the remote user's perspective, and the results showed that our system can effectively improve the interaction efficiency while achieving a better user experience than two traditional view-sharing techniques based on 360 video and based on the local user's first-person view. User study B evaluated our MR telecollaboration system prototype from both the remote-user side and the local-user side as a whole, providing directions and suggestions for the subsequent design and improvement of our mixed-reality telecollaboration system.

Immersive Free‐Viewpoint Panorama Rendering from Omnidirectional Stereo Video

AbstractIn this paper, we tackle the challenging problem of rendering real‐world 360° panorama videos that support full 6 degrees‐of‐freedom (DoF) head motion from a prerecorded omnidirectional stereo (ODS) video. In contrast to recent approaches that create novel views for individual panorama frames, we introduce a video‐specific temporally‐consistent multi‐sphere image (MSI) scene representation. Given a conventional ODS video, we first extract information by estimating framewise descriptive feature maps. Then, we optimize the global MSI model using theory from recent research on neural radiance fields. Instead of a continuous scene function, this multi‐sphere image (MSI) representation depicts colour and density information only for a discrete set of concentric spheres. To further improve the temporal consistency of our results, we apply an ancillary refinement step which optimizes the temporal coherency between successive video frames. Direct comparisons to recent baseline approaches show that our global MSI optimization yields superior performance in terms of visual quality. Our code and data will be made publicly available.

A Deep-Learning Based Pipeline for Estimating the Abundance and Size of Aquatic Organisms in an Unconstrained Underwater Environment from Continuously Captured Stereo Video

The utilization of stationary underwater cameras is a modern and well-adapted approach to provide a continuous and cost-effective long-term solution to monitor underwater habitats of particular interest. A common goal of such monitoring systems is to gain better insight into the dynamics and condition of populations of various marine organisms, such as migratory or commercially relevant fish taxa. This paper describes a complete processing pipeline to automatically determine the abundance, type and estimate the size of biological taxa from stereoscopic video data captured by the stereo camera of a stationary Underwater Fish Observatory (UFO). A calibration of the recording system was carried out in situ and, afterward, validated using the synchronously recorded sonar data. The video data were recorded continuously for nearly one year in the Kiel Fjord, an inlet of the Baltic Sea in northern Germany. It shows underwater organisms in their natural behavior, as passive low-light cameras were used instead of active lighting to dampen attraction effects and allow for the least invasive recording possible. The recorded raw data are pre-filtered by an adaptive background estimation to extract sequences with activity, which are then processed by a deep detection network, i.e., Yolov5. This provides the location and type of organisms detected in each video frame of both cameras, which are used to calculate stereo correspondences following a basic matching scheme. In a subsequent step, the size and distance of the depicted organisms are approximated using the corner coordinates of the matched bounding boxes. The Yolov5 model employed in this study was trained on a novel dataset comprising 73,144 images and 92,899 bounding box annotations for 10 categories of marine animals. The model achieved a mean detection accuracy of 92.4%, a mean average precision (mAP) of 94.8% and an F1 score of 93%.

Ambient sound levels, directionality, and transport velocity around a seamount measured by a deep-drifting acoustic vector sensor

In September 2022 an opto-acoustic drifter was deployed several times in the vicinity of the Kelvin seamount (38°49’N, 64°2’W) in the North Atlantic Ocean, which rises to a depth of 1500 m from a 5 km deep abyssal plain. The “SQUALL-E” drifter incorporates stereo video cameras, vertical (1.75 m aperture) and tetrahedral hydrophone arrays, and a 2-D Geospectrum M35 acoustic vector sensor. One overnight deployment descended to 250 m depth, and a another to 400 m depth, which resulted in highly different drift rates. The M35 sensor demonstrated the ability to estimate the directionality of ambient sounds between 1 and nearly 30 kHz. Sperm whales, delphinids, various vessels, and other discrete and distributed sources were detected. For both deployments we present distributions of ambient sound levels, azimuthal and elevation angles, and transport velocity. We discuss how these distributions evolved throughout the deployments. [Work sponsored by ONR TFO program.]

Statistical and Dynamical Characteristics of Extreme Wave Crests Assessed with Field Measurements from the North Sea

Abstract Wave crests of unexpected height and steepness pose a danger to activities at sea, and long-term field measurements provide important clues for understanding the environmental conditions that are conducive to their generation and behavior. We present a novel dataset of high-frequency laser altimeter measurements of the sea surface elevation gathered over a period of 18 years from 2003 to 2020 on an offshore platform in the central North Sea. Our analysis of crest height distributions in the dataset shows that mature, high sea states with high spectral steepness and narrow directional spreading exhibit crest height statistics that significantly deviate from standard second-order models. Conversely, crest heights in developing sea states with similarly high steepness but wide directional spread correspond well to second-order theory adjusted for broad frequency bandwidth. The long-term point time series measurements are complemented with space–time stereo video observations from the same location, collected during five separate storm events during the 2019/20 winter season. An examination of the crest dynamics of the space–time extreme wave crests in the stereo video dataset reveals that the crest speeds exhibit a slowdown localized around the moment of maximum crest elevation, in line with prevailing theory on nonlinear wave group dynamics. Extending on previously published observations focused on breaking crests, our results are consistent for both breaking and nonbreaking extreme crests. We show that wave crest steepness estimated from time series using the linear dispersion relation may overestimate the geometrically measured crest steepness by up to 25% if the crest speed slowdown is not taken into account. Significance Statement Better understanding of the statistics and dynamical behavior of extreme ocean surface wave crests is crucial for improving the safety of various operations at sea. Our study provides new, long-term field evidence of the combined effects of wave field steepness and directionality on the statistical distributions of crest heights in storm conditions. Moreover, we show that the dynamical characteristics of extreme wave crests are well described by recently identified nonlinear wave group dynamics. This finding has implications, for example, for wave force calculations and the treatment of wave breaking in numerical wave models.