Stereo Camera Setup Research Articles

Stereo Camera Setup for 360° Digital Image Correlation to Reveal Smart Structures of Hakea Fruits.

About forty years after its first application, digital image correlation (DIC) has become an established method for measuring surface displacements and deformations of objects under stress. To date, DIC has been used in a variety of in vitro and in vivo studies to biomechanically characterise biological samples in order to reveal biomimetic principles. However, when surfaces of samples strongly deform or twist, they cannot be thoroughly traced. To overcome this challenge, different DIC setups have been developed to provide additional sensor perspectives and, thus, capture larger parts of an object's surface. Herein, we discuss current solutions for this multi-perspective DIC, and we present our own approach to a 360° DIC system based on a single stereo-camera setup. Using this setup, we are able to characterise the desiccation-driven opening mechanism of two woody Hakea fruits over their entire surfaces. Both the breaking mechanism and the actuation of the two valves in predominantly dead plant material are models for smart materials. Based on these results, an evaluation of the setup for 360° DIC regarding its use in deducing biomimetic principles is given. Furthermore, we propose a way to improve and apply the method for future measurements.

Rapid validation of water wave metamaterials in a desktop-scale wave measurement system

Metamaterials have a unique ability to manipulate wave phenomena beyond their natural capabilities, and they have shown great promise in electromagnetic and acoustic wave control. However, their exploration in hydrodynamics remains limited. This article introduces a novel desktop-scale wave measurement system, specifically designed for the rapid prototyping and validation of water wave metamaterials. By utilizing 3D printing, the system accelerates the transition from theoretical designs to practical testing, offering a versatile and user-friendly platform. This is further enhanced by a synchronized stereo-camera setup and advanced data processing algorithms, enabling precise measurement and reconstruction of water wave behavior. Our experimental results demonstrate the system’s effectiveness in capturing intricate interactions between engineered structures and water waves. This significantly advances rapid prototyping for water wave metamaterial research, underscoring the system’s potential to catalyze further innovation in this emerging field.

MosaicMVS: Mosaic-Based Omnidirectional Multi-View Stereo for Indoor Scenes

We present MosaicMVS, a novel learning-based depth estimation framework for a mosaic-based omnidirectional multi-view stereo (MVS) camera setup. It uses a regular field of view (FOV) MVS network for an omnidirectional imaging setup with explicit consideration of hypothetical voxel-wise FOV overlaps. The resulting depth predictions are accurate and agree on the omnidirectional multi-view geometry. Unlike existing MVS setups, MosaicMVS camera setup can be easily applied to omnidirectional indoor scenes without having to account for con-straints such as intricate epipolar constraints and the distortion of omnidirectional cameras. We validate the effectiveness of our framework on a new challenging indoor dataset in terms of depth estimation, reconstruction, and view synthesis. We also present new evaluation metric to check reconstruction performance using post-processed masks for accurate evaluation without any ground truth depth map or laser-scanned reconstructions. Experimental results show that our framework outperforms the state-of-the-art MVS methods in a large margin in all test scenes.

COMPREHENSIVE POTHOLE DETECTION SYSTEM FOR ROAD MAINTENANCE AND SAFETY USING IMAGE PROCESSING AND STEREO VISION

This paper introduces a systematic approach to pothole detection, emphasizing its significance in road maintenance and safety. The proposed system is comprised of meticulously designed modules that collectively contribute to achieving accurate and effective results. Commencing with data collection through cameras capturing road areas of interest, the images undergo manual cropping and resizing for standardization. The core principle of the system revolves around image enhancement, involving grayscale conversion, application of blurring techniques, and adjustments to brightness and contrast. Automatic thresholding extracts essential information encoded within pixels, paving the way for precise edge detection refined through morphological operations. The focus then shifts to pothole detection, incorporating a stereo camera setup to calculate depth and disparity of road surfaces. Critical steps, including image rectification, correspondence matching, and disparity calculations, contribute to the accurate identification and delineation of potholes using depth thresholds. Subsequent modules employ K-Means clustering to segregate regions of interest from the image background, and post-processing steps, including filtering, masking, and refinement, fine-tune the images. Utilizing the HSV color space for grayscale image refinement and the connected component method to isolate white pixel objects further enhance the system's capability. The final step involves the addition of bounding boxes around identified potholes, streamlining their identification process. This comprehensive methodology ensures effective image processing, precise pothole detection, and contributes significantly to road safety and maintenance efforts.

Using multi-sine excitation and rigid body motion compensation in randomly sampled camera-based experimental modal analysis to improve SNR

Camera-based Experimental Modal Analysis (EMA) is able to measure full-field vibration modes contactlessly. However, camera measurements still suffer from a lower sampling frequency and lower Signal-to-Noise Ratio (SNR) compared to accelerometers and laser vibrometers. Since regular sampling confines the results to the Nyquist frequency (i.e., half the frame rate of a camera), in a previous paper we have proposed an approach that exploits images randomly sampled in time, allowing to measure modes beyond the Nyquist frequency. In this paper we employ such an approach and we propose a novel excitation scheme to increase the SNR, i.e., the specimen under investigation is excited at selected resonance frequencies, which results in high amplitudes of the structural responses. These frequencies are measured by accelerometers during a pre-test, and a multi-sine signal containing these frequency components is generated as structural excitation through a shaker. Furthermore, as Rigid Body Motion (RBM) due to pseudo-free suspension is not desired for EMA, in this paper we also propose a novel optimization workflow for RBM compensation. The accuracy of the proposed approach with multi-sine excitation is first evaluated numerically by reconstructing steady-state response signals. Secondly, the whole workflow (i.e., multi-sine excitation and RBM compensation) is validated on an experimental case with a 3D component and a stereo camera setup. From a sequence of randomly sampled images equivalent to a frame rate below 50fps (corresponding to an equivalent Nyquist frequency below 25Hz), four modes up to 250Hz (i.e., ten times higher than the Nyquist frequency) are extracted, which correlate well with the modes predicted through a finite element simulation.

Multi Cost Function Fuzzy Stereo Matching Algorithm for Object Detection and Robot Motion Control

Stereo matching algorithms work with multiple images of a scene, taken from two viewpoints, to generate depth information. Authors usually use a single matching function to generate similarity between corresponding regions in the images. In the present research, the authors have considered a combination of multiple data costs for disparity generation. Disparity maps generated from stereo images tend to have noisy sections. The presented research work is related to a methodology to refine such disparity maps such that they can be further processed to detect obstacle regions. A novel entropy based selective refinement (ESR) technique is proposed to refine the initial disparity map. The information from both the left disparity and right disparity maps are used for this refinement technique. For every disparity map, block wise entropy is calculated. The average entropy values of the corresponding positions in the disparity maps are compared. If the variation between these entropy values exceeds a threshold, then the corresponding disparity value is replaced with the mean disparity of the block with lower entropy. The results of this refinement are compared with similar methods and was observed to be better. Furthermore, in this research work, the v-disparity values are used to highlight the road surface in the disparity map. The regions belonging to the sky are removed through HSV based segmentation. The remaining regions which are our ROIs, are refined through a u-disparity area-based technique. Based on this, the closest obstacles are detected through the use of k-means segmentation. The segmented regions are further refined through a u-disparity image information-based technique and used as masks to highlight obstacle regions in the disparity maps. This information is used in conjunction with a kalman filter based path planning algorithm to guide a mobile robot from a source location to a destination location while also avoiding any obstacle detected in its path. A stereo camera setup was built and the performance of the algorithm on local real-life images, captured through the cameras, was observed. The evaluation of the proposed methodologies was carried out using real life out door images obtained from KITTI dataset and images with radiometric variations from Middlebury stereo dataset.

MANGO: low-cost open source needle guidance

ABSTRACT Validating experiments in intra-operative planning for percutaneous interventions requires to localise the needle with a tracking system. As commercial optical trackers are expensive, we present MANGO, a low-cost OpenSource solution for needle tracking. We first present the design of our system, which localises coloured orbs using a stereo-camera setup. Based on this setup, we investigate the static and dynamic error of the system, and experiment with signal filtering to improve tracking performance. With the MANGO, we are able to meet the positional accuracy margin of 5 mm usually required in percutaneous interventions. Orientational accuracy, measured by needle tip localisation accuracy as a proxy measure, exceeds this margin. We plan to investigate this further in future research. Releasing our code, data and 3D models, we aim to provide a low-cost and reproducible platform for needle tracking and validation of intra-operative planning in research environments.

Effect of Mach Number and Pitching Eigenfrequency on Transonic Buffet Onset

The transonic flow around an elastically suspended supercritical airfoil was experimentally investigated in order to enhance knowledge of buffet boundary dependencies and the corresponding self-excited fluid–structure interaction (buffeting). For that purpose, an experimental setup was designed, manufactured, and integrated in the Trisonic Wind Tunnel Munich. The design consisted of a rigid, two-dimensional, supercritical airfoil (OAT15A) with optional pitching degree of freedom and variable torsional spring stiffness. High-speed background-oriented schlieren measurements were used to observe the shock with its dynamics, while a high-speed stereo camera setup for correlation-based deformation measurements was implemented to track the dynamics of the structural motion. Pre-buffet and buffet flows were analyzed by a continuous increase of the angle of attack. The detailed observation of the shock position with increasing angle of attack exhibited a clear effect of Mach number and pitch eigenfrequency on the buffet phenomenon. The corresponding onset boundary clearly shifted into the “pre-buffet” regime given specific structural settings. Furthermore, the resulting fluid–structure interaction of shock and coupled pitch–heave motion showed characteristics of structural frequency lock-in for the pitch-to-buffet frequency ratios of 1.2 and the mode veering region for ratios of 1 and 0.9, indicating the transition region from fluid mode flutter to structural mode flutter.

Miniaturization of a coherent monocular structured illumination system for future combination with digital holography

Holographic and 3D-measurement processes are an often-used tool in industry, medicine, and scientific applications. While small deviations of objects can be visualized by holographic means with high accuracy, optical systems with active structured illumination are a reliable source of absolute 3D-information in these fields. The combination of digital holography with structured illumination allows to simultaneously measure deformations and absolute 3D coordinates but also requires coherent light and has already been demonstrated in principle with a stereo camera setup. Multi-camera systems are limited to certain setup sizes given by the volume and distance of the detectors. Reducing the system to a one-camera (monocular) setup reduces space and acquisition costs. By using a multi-aperture illumination source an extremely high projection rate could be realized and reduced to a monocular approach with a novel voxel-calibration technique, while the projection system itself still requires a large amount of space. In this paper we present a miniaturized, monocular 3D-measurement system that works with repeatable, coherent speckles, generated by a fiber-coupled laser whose light was distributed by a fiber-switch to a diffuser plate connected with a measurement-head, also including a camera. By addressing different fibers through the switch, varying but repeatable patterns are generated. The size of the device (diameter < 3 cm) is now mainly limited by the volume of the camera. A first 3D-reconstruction of an object and an outlook for a combination of this system with digital holography is given, allowing absolute 3D-coordinates and relative deviations of object points to be measured simultaneously.

Motion trajectory estimation of salmon using stereo vision

A main concern for the aquaculture industry is the fish behaviour and welfare. Motion trajectory analysis of salmon at aquaculture farming sites with respect to certain aquaculture operations aims to provide information about the behaviour and possibly stress level of the farmed salmon and may help to generate a general welfare indicator index. Towards this aim we present an innovative computer vision and machine learning based approach for motion trajectory estimation of salmon. Video footage was recorded with a stereo camera setup. Deep learning based object detection was performed to detect particular features. We focused on tracking the fish eyes and heads as a reliable indicators of the fish's position. Feature matching and subsequent 3D reconstruction was performed to calculate the 3D position of the fish from which trajectories of the fish movement were estimated. Related experiments were conducted at an aquaculture research facility under natural lighting conditions and extracted trajectories allowed a qualitative verification. The developed method was verified using synthetic ground truth data produced with an open source computer graphics software for quantifiable performance metrics.

Optimizing a Feature-Based Motion Tracking System for Prospective Head Motion Estimation in MRI and PET/MRI

Patient head motion during magnetic resonance imaging (MRI) of the brain causes artifacts that limit diagnostic value. Prospective motion correction can, in principle, allow artifact-free imaging for any MRI sequence. Optical motion tracking systems are effective at measuring head motion for prospective motion correction but normally rely on attached markers, which are cumbersome and can move relative to the head. In this study, we optimized and benchmarked an optical motion tracking system that uses in-bore stereo-optical cameras to track feature patches on the forehead rather than attached markers. Feature patches were applied to the forehead in the form of an easily removable ink stamp or temporary tattoo. This approach has flexibility in camera positioning and allows tracking of a variety of surfaces on the human face, even those lacking prominent contours. This is advantageous given the limited space and line-of-sight with typical head coils. We optimized the accuracy and efficiency of two key components of the method: 1) the geometry of the stereo camera setup and 2) the feature detection and matching algorithm used for pose estimation. The absolute accuracy of the optimized feature-based method was compared with a commercial marker-based motion tracking system. Estimates of head motion of 15 volunteers simultaneously tracked by the two systems resulted in average positional discrepancies of less than 3 mm using a stamp and submillimeter average discrepancy using a feature-dense tattoo. These results suggest the feasibility of obtaining accurate head motion data for prospective motion compensated MRI using optimized feature-based motion tracking.

Fast Detection of Ground Plane in Stereo Vision Systems Based on Iso-Disparity Strip Pattern

In this paper, first a closed-form formulation is analytically derived for the pattern of iso-disparity strip widths associated with the ground plane in disparity map images. This formulation calculates the iso-disparity strip widths based on the parameters of the stereo camera setup ( i.e. , height, tilt angle, and baseline of the cameras) used to capture the images. The proposed formulation was then statistically validated using the results achieved in the case of a bank of stereo image pairs with different stereo camera setup parameters. Validation of the proposed formulation was in the presence of the strip width imperfections that appear in reality, and were shown to be in the form of additive Gaussian noise. Using the formulation proposed for iso-disparity strip widths, the paper then introduces a novel approach for ground plane detection based on the concept of iso-disparity strips in disparity map images. To add to the robustness of the proposed procedure against strip width noise, Cumulative Moving Average (CMA) and a dynamic thresholding technique are used to find the width of the strips corresponding to the ground plane. According to experimental results for synthesized and captured datasets, while exhibiting sufficiently low false positive rates (1.38% and 5.83%), the proposed method detects the ground plane with average true positive rates of as high as 80% and 54% in 65 ms and 224 ms, respectively.

The hunter and the hunted-A 3D analysis of predator-prey interactions between three-spined sticklebacks (Gasterosteus aculeatus) and larvae of different prey fishes.

Predator-prey interactions play a key life history role, as animals cope with changing predation risk and opportunities to hunt prey. It has recently been shown that the hunting success of sticklebacks (Gasterosteus aculeatus) targeting fish larvae is dependent on both the size of the prey and the prior exposure of its species to stickleback predation. The purpose of the current study was to identify the behavioural predator-prey interactions explaining the success or failure of sticklebacks hunting larvae of three potential prey species [roach (Rutilus rutilus), perch (Perca fluviatilis) and whitefish (Coregonus wartmannii)] in a 3D environment. Trials were carried out for each prey species at four different size classes in a standardised laboratory setup and were recorded using a slow motion, stereo camera setup. 75 predator-prey interactions including both failed and successful hunts were subject to the analysis. 3D track analysis indicated that sticklebacks applied different strategies. Prey with less complex predator escape responses, i.e. whitefish larvae, were hunted using a direct but stealthy approach ending in a lunge, while the behaviourally more complex roach and perch larvae were hunted with a faster approach. A multivariate logistic regression identified that slow average speed and acceleration of the prey in the initial stages of the hunt increased the probability of stickleback success. Furthermore, predators adjusted their swimming direction more often when hunting larger whitefish compared to smaller whitefish. The results suggest that appropriate and adequately timed avoidance behaviours, which vary between prey species and ontogenetic stages, significantly increase the chances of outmanoeuvring and escaping stickleback predation. Small whitefish larvae can reach similar levels of swimming performance compared to older conspecifics, but display ineffective anti-predator behaviours, resulting in higher hunting success for sticklebacks. Thus, the development of appropriate anti-predator behaviours depending on size appears to be the crucial factor to escaping predation.

OV$^{2}$SLAM: A Fully Online and Versatile Visual SLAM for Real-Time Applications

Many applications of Visual SLAM, such as augmented reality, virtual reality, robotics or autonomous driving, require versatile, robust and precise solutions, most often with real-time capability. In this work, we describe OV <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> SLAM, a fully online algorithm, handling both monocular and stereo camera setups, various map scales and frame-rates ranging from a few Hertz up to several hundreds. It combines numerous recent contributions in visual localization within an efficient multi-threaded architecture. Extensive comparisons with competing algorithms shows the state-of-the-art accuracy and real-time performance of the resulting algorithm. For the benefit of the community, we release the source code: https://github.com/ov2slam/ov2slam.

Vision-Based Tactile Sensor Mechanism for the Estimation of Contact Position and Force Distribution Using Deep Learning.

This work describes the development of a vision-based tactile sensor system that utilizes the image-based information of the tactile sensor in conjunction with input loads at various motions to train the neural network for the estimation of tactile contact position, area, and force distribution. The current study also addresses pragmatic aspects, such as choice of the thickness and materials for the tactile fingertips and surface tendency, etc. The overall vision-based tactile sensor equipment interacts with an actuating motion controller, force gauge, and control PC (personal computer) with a LabVIEW software on it. The image acquisition was carried out using a compact stereo camera setup mounted inside the elastic body to observe and measure the amount of deformation by the motion and input load. The vision-based tactile sensor test bench was employed to collect the output contact position, angle, and force distribution caused by various randomly considered input loads for motion in X, Y, Z directions and rotational motion. The retrieved image information, contact position, area, and force distribution from different input loads with specified 3D position and angle are utilized for deep learning. A convolutional neural network VGG-16 classification modelhas been modified to a regression network model and transfer learning was applied to suit the regression task of estimating contact position and force distribution. Several experiments were carried out using thick and thin sized tactile sensors with various shapes, such as circle, square, hexagon, for better validation of the predicted contact position, contact area, and force distribution.

Design of an end-effector for robot-assisted ultrasound-guided breast biopsies

PurposeThe biopsy procedure is an important phase in breast cancer diagnosis. Accurate breast imaging and precise needle placement are crucial in lesion targeting. This paper presents an end-effector (EE) for robotic 3D ultrasound (US) breast acquisitions and US-guided breast biopsies. The EE mechanically guides the needle to a specified target within the US plane. The needle is controlled in all degrees of freedom (DOFs) except for the direction of insertion, which is controlled by the radiologist. It determines the correct needle depth and stops the needle accordingly.MethodIn the envisioned procedure, a robotic arm performs localization of the breast, 3D US volume acquisition and reconstruction, target identification and needle guidance. Therefore, the EE is equipped with a stereo camera setup, a picobeamer, US probe holder, a three-DOF needle guide and a needle stop. The design was realized by prototyping techniques. Experiments were performed to determine needle placement accuracy in-air. The EE was placed on a seven-DOF robotic manipulator to determine the biopsy accuracy on a cuboid phantom.ResultsNeedle placement accuracy was 0.3 ± 1.5 mm in and 0.1 ± 0.36 mm out of the US plane. Needle depth was regulated with an accuracy of 100 µm (maximum error 0.89 mm). The maximum holding force of the stop was approximately 6 N. The system reached a Euclidean distance error of 3.21 mm between the needle tip and the target and a normal distance of 3.03 mm between the needle trajectory and the target.ConclusionAn all in one solution was presented which, attached to a robotic arm, assists the radiologist in breast cancer imaging and biopsy. It has a high needle placement accuracy, yet the radiologist is in control like in the conventional procedure.

Scanning stereo-PLIF method for free surface measurements in large 3D domains

In this work, we extend a planar laser-induced fluorescence method for free surface measurements to a three-dimensional domain using a stereo-camera system, a scanning light sheet, and a modified self-calibration procedure. The stereo-camera set-up enables a versatile measurement domain with self-calibration, improved accuracy, and redundancy (e.g., possibility to overcome occlusions). Fluid properties are not significantly altered by the fluorescent dye, which results in a non-intrusive measurement technique. The technique is validated by determining the free surface of a hydraulic flow over an obstacle and circular waves generated after droplet impact. Free surface waves can be accurately determined over a height of L=100 mm in a large two-dimensional domain (y(x,z) = 120times 62 mm^2), with sufficient accuracy to determine small amplitude variations (eta approx 0.2 mm). The temporal resolution ({varDelta }t = 19 ms) is only limited by the available scanning equipment (f = 1 kHz rate). For other applications, this domain can be scaled as needed.Graphic abstract

3D acquisition and stereo-camera calibration by active devices: A unique structured light encoding framework

The use of light encoding techniques is widespread in the field of 3D surface reconstruction. This paper presents a stereo-camera calibration methodology, which integrates structured light encoding with an active digital device. The structured light encoding approach is proposed to unambiguously solve the stereo matching issue for stereo-camera setups. A sequence of vertical and horizontal binary striped patterns, combined with a checkerboard pattern, is displayed by a high-resolution LCD screen, which is used as calibration board. A bundle adjustment technique is adopted to simultaneously adjust both camera parameters and screen geometry, as part of the stereo-camera calibration process, thus taking into account the possible inaccuracies of the digital display. The same structured light approach, with small variants, is projected by a multimedia digital projector to carry out 3D surface reconstruction. The proposed methodology defines a comprehensive framework for the development of a 3D optical scanner, from calibration to 3D acquisition, which has been validated by measuring primitive surfaces and reconstructing free-form shapes with different stereo-camera setups.

Optimization-based extrinsic calibration of a three-dimensional sensor composed of an array projector and a single camera

We previously presented a technique calibrating a multiaperture array projector (MAAP) for performing high-speed three-dimensional (3-D) surface topology measurements with aperiodic sinusoidal fringe structured illumination. Although a stereo-camera setup used to be required for triangulation, the presented MAAP calibration technique re-enables photogrammetric 3-D measurements to be made using a single CCD or CMOS camera. For such a monocular-view photogrammetric system, intrinsic camera calibration, intrinsic projector calibration, and extrinsic system calibration are vital to obtain high-measurement performance. However, there is currently no such technique that can comprehensively perform the required extrinsic system calibration when using an MAAP and a single-camera setup. A comprehensive optimization-based method is proposed and the resulting measurement performance is evaluated. With the new calibration technique, we are able to achieve a surface standard deviation of ∼50 μm.

A camera model for cameras with hypercentric lenses and some example applications

We propose a camera model for cameras with hypercentric lenses. Because of their geometry, hypercentric lenses allow to image the top and the sides of an object simultaneously. This makes them useful for certain inspections tasks, for which otherwise multiple images would have to be acquired and stitched together. After describing the projection geometry of hypercentric lenses, we derive a camera model for hypercentric lenses that is intuitive for the user. Furthermore, we describe how to determine the parameter values of the model by calibrating the camera with a planar calibration object. We also apply our camera model to two example applications: in the first application, we show how two cameras with hypercentric lenses can be used for dense 3D reconstruction. For an efficient reconstruction, the images are rectified such that corresponding points occur in the same image row. Standard rectification methods would result in perspective distortions in the images that would prevent stereo matching algorithms from robustly establishing correspondences. Therefore, we propose a new rectification method for objects that are approximately cylindrical in shape, which enables a robust and efficient reconstruction. In the second application, we show how to unwrap cylindrical objects to simplify further inspection tasks. For the unwrapping, the pose of the cylinder must be known. We show how to determine the pose of the cylinder based on a single camera image and based on two images of a stereo camera setup.