Projector-camera System Research Articles

Real-time motion-induced error reduction for phase-shifting profilometry with projection points tracking method

Fringe projection profilometry is a significant method for three-dimensional measurement due to its non-contact and high accuracy. However, the motion-induced error will lead to the loss of measurement accuracy in dynamic scenes due to the disruption of the phase measurement process. In this paper, we introduce a novel motion error model that considers object motion causes the misalignment of the projection points on the camera, and propose the projection points tracking method to reduce the motion-induced error. First, the speckle pattern is added to projection sequences, after which the projection point displacements are determined using the digital image correlation (DIC) method between the adjacent speckle patterns. Finally, the fringe patterns are corrected using the image remapping method to calculate the 3D shape. Quantitative analysis and dynamic measurement experiments verify the feasibility of the proposed method. Different from 3D-DIC, we use one camera-projector system to realize high-accuracy measurement in dynamic scenes.

Physics-Based Efficient Full Projector Compensation Using Only Natural Images.

Achieving practical full projector compensation requires the projection display to adapt quickly to textured projection surfaces and unexpected movements without interrupting the display procedure. A possible solution to achieve this involves using a projector and an RGB camera and correcting both color and geometry by directly capturing and analyzing the projected natural image content, without the need for additional patterns. In this study, we approach full projector compensation as a numerical optimization problem and present a physics-based framework that can handle both geometric calibration and radiometric compensation for a Projector-camera system (Procams), using only a few sampling natural images. Within the framework, we decouple and estimate the Procams' factors, such as the response function of the projector, the correspondence between the projector and camera, and the reflectance of projection surfaces. This approach provides an interpretable and flexible solution to adapt to the changes in geometry and reflectance caused by movements. Benefitting from the physics-based scheme, our method guarantees both accurate color calculation and efficient movement and reflectance estimation. Our experimental results demonstrate that our method surpasses other state-of-the-art end-to-end full projector compensation methods, with superior image quality, reduced computational time, lower memory consumption, greater geometric accuracy, and a more compact network architecture.

Color crosstalk compensation method for color phase-shifting fringe projection profilometry based on the phase correction matrix.

Color phase-shifting fringe projection profilometry is one of the single-shot three-dimensional shape measurement techniques. The color crosstalk of the projector-camera system yields undesired phase errors when using phase-shifting method. In this paper, a color crosstalk compensation method based on phase correction matrix is proposed. In this method, the phase correction matrix is established to compensate the deviations between the actual phase-shift values in the acquired fringes and the standard ones in the ideal fringes. Only two fringe patterns are utilized to obtain the phase correction matrix. The quadratic equations for calculating the actual phase-shift values of the fringes in the three color channels are derived. The actual phase-shift values and the corresponding standard ones are employed to form the equilibrium equations for computing the phase correction coefficients in the matrix. Experimental results demonstrate the feasibility of the proposed method and it can effectively reduce the induced overall phase error caused by the color crosstalk.

Multi-Frequency Nonlinear Methods for 3D Shape Measurement of Semi-Transparent Surfaces Using Projector-Camera Systems.

Measuring the 3D shape of semi-transparent surfaces with projector-camera 3D scanners is a difficult task because these surfaces weakly reflect light in a diffuse manner, and transmit a large part of the incident light. The task is even harder in the presence of participating background surfaces. The two methods proposed in this paper use sinusoidal patterns, each with a frequency chosen in the frequency range allowed by the projection optics of the projector-camera system. They differ in the way in which the camera-projector correspondence map is established, as well as in the number of patterns and the processing time required. The first method utilizes the discrete Fourier transform, performed on the intensity signal measured at a camera pixel, to inventory projector columns illuminating directly and indirectly the scene point imaged by that pixel. The second method goes beyond discrete Fourier transform and achieves the same goal by fitting a proposed analytical model to the measured intensity signal. Once the one (camera pixel) to many (projector columns) correspondence is established, a surface continuity constraint is applied to extract the one to one correspondence map linked to the semi-transparent surface. This map is used to determine the 3D point cloud of the surface by triangulation. Experimental results demonstrate the performance (accuracy, reliability) achieved by the proposed methods.

A Monocular Projector-Camera System Using Modular Architecture.

This paper presents a monocular projector-camera (procam) system using modular architecture based on relay optics. Conventional coaxial procam systems cannot support (1) online changes to lens settings (zoom and focus) and (2) wide-angle projection mapping. We develop design guidelines for a proposed procam system that would solve these restrictions and address the proposed system's unique technical issue of crosstalk between the camera and projector pixels. We conducted experiments using prototypes to validate the feasibility of the proposed framework. First, we confirmed that the proposed crosstalk reduction technique worked well. Second, we found our technique could achieve correct alignment of a projected image onto a moving surface while changing the zoom and focus of the objective lens. The monocular procam system also achieved radiometric compensation where a surface texture was visually concealed by pixel-wise control of a projection color based on the captured results of offline color pattern projections. Finally, we demonstrated the high expandability of our modular architecture, through the creation of a high dynamic range projection.

High-speed 360° 3D shape measurement based on virtual stereo phase unwrapping method

360° three-dimensional (3D) shape measurement is of great significance in the fields of defect detection and reverse modeling. Fringe projection profilometry (FPP) is widely used due to the advantages of high-accuracy and full-field measurement, but the movement of object corrupts the phase measurement. Stereo phase unwrapping (SPU) technique is adopted to improve imaging efficiency, but requires multi-cameras which resulting in high system complexity and cost. In this work, we proposed virtual stereo phase unwrapping (VSPU) method for high-speed 360° 3D shape measurement. This method uses the virtual camera-projector systems formed by the rotation of object for phase unwrapping which only requires single camera-projector to achieve 360° 3D shape measurement. The phase consistency checking in SPU method is modified. Experiments demonstrate that proposed method has strong robustness and can achieve 360° 3D shape measurement in 47 s with an accuracy of 100 μm during continuous rotation of the object.

External Ventricular Drain Placement Teleproctoring Using a Novel Camera-Projector Navigation System: A Proof-of-Concept Study

Teleproctoring is an emerging method of bedside clinical teaching; however, its feasibility has been limited by the available technologies. The use of novel tools that incorporate 3-dimensional environmental information and feedback might offer better bedside teaching options for neurosurgical procedures, including external ventricular drain placement. A platform with a camera-projector system was used to proctor medical students on placing external ventricular drains on an anatomic model as a proof-of-concept study. Three-dimensional depth information of the model and surrounding environment was captured by the camera system and provided to the proctor who could provide projected annotations in a geometrically compensated manner onto the head model in real time. The medical students were randomized to identify Kocher's point on the anatomic model with or without the navigation system. The time required to identify Kocher's point and the accuracy were measured as a proxy for determining the effectiveness of the navigation proctoring system. Twenty students were enrolled in the present study. Those in the experimental group identified Kocher's point an average of 130 seconds faster than did the control group (P < 0.001). The mean diagonal distance from Kocher's point was 8.0 ± 4.29 mm for the experimental group compared with 23.6 ± 21.98 mm for the control group (P= 0.053). Of the 10 students randomized to the camera-projector system arm, 70% were accurate to within 1 cm of Kocher's point compared with 40% of the control arm (P > 0.05). Camera-projector systems for bedside procedure proctoring and navigation are a viable and valuable technology. We demonstrated its viability for external ventricular drain placement as a proof-of-concept. However, the versatility of this technology indicates that that it could be useful for a variety of even more complex neurosurgical procedures.

Projection Mapping-Based Interactive Gimmick Picture Book with Visual Illusion Effects

In this study, we proposed an electronic gimmick picture book system based on projection mapping for early childhood education in ordinary households. In our system, the visual effects of pop-up gimmicks in books are electronically reproduced by combining projection mapping, three-dimensional expression, and visual illusion effects. In addition, the proposed projector-camera system can provide children effective interaction experiences, which will have positive effects on their early childhood education. The performance of the proposed system was validated by projection experiments.

Point-by-point visual enhancement with spatially and spectrally tunable laser illumination.

Vision is responsible for most of the information that humans perceive of the surrounding world. Many studies attempt to enhance the visualization of the entire scene by optimizing and tuning the overall illumination spectrum. However, by using a spatially uniform illumination spectrum for the entire scene, only certain global color shifts with respect to a reference illumination spectrum can be realized, resulting in moderate visual enhancement. In this paper, a new visual enhancement method is presented that relies on a spatially variable illumination spectrum. Such an approach can target much more dedicated visual enhancements by optimizing the incident illumination spectrum to the surface reflectance at each position. First, a geometric calibration of the projector-camera system is carried out for determining the spatial mapping from the projected pixel grid to the imaged pixel grid. Secondly, the scene is segmented for implementing the visual enhancement approach. And finally, one of three visual enhancement scenarios is applied by projecting the required color image onto the considered segmented scene. The experimental results show that the visual salience of the scene or region of interest can be efficiently enhanced when our proposed method is applied to achieve colorfulness enhancement, hue tuning, and background lightness reduction.

Dark-Channel Enhanced-Compensation Net: An end-to-end inner-reflection compensation method for immersive projection system.

Immersive projection display system is widely adopted in virtual reality and various exhibition halls. How to maintain high display quality in an immersive projection environment with uneven illumination and the color deviation caused by the inter-reflection of light is still a challenging task. In this paper, we innovatively propose a deep learning-based radiation compensation for an L-shaped projector-camera system. This method employs complex reflection phenomena to simulate the light transport processing in an L-shaped environment, we also designed a Dark-Channel Enhanced-Compensation Net (DECNet) which composed of a convolutional neural network called Compensation Net, a DarkChannelNet and another subnet (such as sensing network) aiming at achieving high-quality reproduction of projected display images. The final output of DECNet is the compensation image to be projected. It is always a critical problem to establish appropriate evaluation and analysis indexes throughout the research of light pollution compensation algorithms. In this paper, PSNR, SSIM, and RMSE are proposed to quantitatively analyze the image quality. The experimental results show that this method has certain advantages in reducing the inter-reflection of the projection plane. And our method could also well replace the traditional process using the backlight transmission matrix. It can be concluded to a certain that this method can be extended to other more complex projection environments with strong scalability and inclusiveness.

SVBRDF Estimation Using a Normal Sorting Technique

Spatially varying bidirectional reflectance distribution functions (SVBRDFs) play an important role in appearance modeling of real-world surfaces. Automatic capture of these surface properties is highly desirable, but many techniques only partially capture these properties or use complicated setups to do so. Micro surface roughness variations are especially difficult to capture using image-based methods. In this paper, we propose a novel approach towards estimating the complete SVBRDFs of surfaces using a portable projector-camera system made of standard consumer-grade components. Our approach uses insights about the relations between the illumination and viewing geometry and captured image statistics to estimate surface reflectance properties. Our technique should be of great value to practitioners seeking to model and render the geometric and reflectance properties of complex real-world surfaces.

An Adaptive Illumination Optimization Method for Local Overexposed Image

In order to solve the local overexposure caused by uneven surface reflectance, this paper proposes a fast-adaptive illumination control method with a camera-projector system. At first, an image is captured by the camera and the local overexposed area is segmented using saliency detection. Then the calculated image is projected onto the object by the projector as corrective illumination. The calculation process includes the inversion of the gray value in the overexposed area and the adjustment based on the position and depth information of the object. The high-exposure saturated regional which affects the target recognition is thus reduced, and the original illumination intensity is reserved for the other regions. This process is iterated until the optimal illumination is achieved. The resulting image for each iteration is evaluated using Blind/no Reference Image Space Quality Estimator (BRISQUE). When BRISQUE value reaches the minimum, a high-quality image is achieved. The experiments show that the proposed approach can significantly improve the speed of obtaining normally exposed images, and this system provides new ideas for industry image acquisition.

Analysis of phase errors introduced by single-shot color phase-shifting profilometry on colored objects

We aim to report an error analysis on single-shot profilometry when projecting a composite colored fringe pattern onto a surface containing different colors. Several studies have analyzed white surfaces using profilometry; however, when color is incorporated, new considerations must be taken into account. In this work, it is found theoretically and corroborated experimentally that each color introduces local phase variations that render in phase errors with a severity that is related to the crosstalk matrix of the projector–camera system. It is also shown that the phase errors are phase-shifting dependent and could be reduced with appropriate phase recovery algorithms. Experimental results are presented and compared with those obtained using a standard temporal four-step phase-shifting method.

Pro-Cam SSfM: projector–camera system for structure and spectral reflectance from motion

Image-based reconstruction of an object’s 3D shape having the wavelength-by-wavelength spectral reflectance property enables higher-fidelity object 3D modeling compared with typical RGB-based modeling. In this paper, we propose a novel projector–camera system for practical and low-cost acquisition of a dense object 3D model with the spectral reflectance property. Different from existing spectral 3D data acquisition systems that use a dedicated multispectral camera or light, we use a standard RGB camera and an off-the-shelf projector as active illumination for both the 3D reconstruction and the spectral reflectance estimation. We first propose a calibration-free multi-view structured-light method to reconstruct the 3D points while estimating the intrinsic parameters and the poses of both the camera and the projector, which are alternately moved around the object during our image acquisition procedure. We then exploit the projector for multispectral imaging and estimate the spectral reflectance of each 3D point based on a novel spectral reflectance estimation method considering the geometric relationship between the reconstructed 3D points and the estimated projector positions. Experimental results on both synthetic and real data demonstrate that our system can precisely acquire a dense spectral 3D model using off-the-shelf devices.

Slope Disparity Gating: System and Applications

Active illumination systems which have the ability to selectively image photons that arrive from a specified surface geometry some distance away have several applications in robotics, autonomous vehicles, and surveillance. In this paper, we present a new technique for sloped disparity gating utilizing a synchronized projector-camera system consisting of a raster scanning laser projector and a rolling shutter camera. This system enables disparity-based triangulation of light from specific sloped planar surfaces in the imaging volume. We demonstrate how to control the slope and thickness of these planar surfaces using hardware parameters of pixel clock, synchronization delay, and exposure. Using our system, we present a new decomposition of the light transport matrix in a scene as a function of disparity. We perform applications including real-time image masking and imaging and relighting in scattering media. We also demonstrate applications of the proposed light transport probing technique like novel disparity-dependent relighting, light transport-aware foreground/background separation and green screening, and improved visualization and relighting in scattering media.

Projector Compensation Framework Using Differentiable Rendering

Projection mapping is a widely adopted technique in various applications. A typically projector-camera system consists of a projector and camera pair where a projector emits an application-specific input image onto a surface, and the camera captures the projected output image. Given an ideal projection surface with white-colored planar geometries, the camera (or a viewer) can observe the projected output image without any visual distortion. The surface, however, can have arbitrary shapes and textured colors in practice, and it often introduces visually distracting artifacts to the output image. It results in lowering the viewing experience drastically. We propose a projector compensation framework that adjusts the projector input image so that the camera can see the projected output image with a much-reduced distortion. Our key contribution is to model the real projection mapping process with a virtual but controllable light simulation and optimize the projector input using differentiable rendering. We demonstrate that our new framework produces a more accurate output than state-of-the-art methods given complex projection surfaces.

Design guidelines for human–robot interaction with assistive robot manipulation systems

Abstract The design of user interfaces (UIs) for assistive robot systems can be improved through the use of a set of design guidelines presented in this article. As an example, the article presents two different UI designs for an assistive manipulation robot system. We explore the design considerations from these two contrasting UIs. The first is referred to as the graphical user interface (GUI), which the user operates entirely through a touchscreen as a representation of the state of the art. The second is a type of novel UI referred to as the tangible user interface (TUI). The TUI makes use of devices in the real world, such as laser pointers and a projector–camera system that enables augmented reality. Each of these interfaces is designed to allow the system to be operated by an untrained user in an open environment such as a grocery store. Our goal is for these guidelines to aid researchers in the design of human–robot interaction for assistive robot systems, particularly when designing multiple interaction methods for direct comparison.

Vision-Based Inspection Approach Using a Projector-Camera System for Off-Site Quality Control in Modular Construction: Experimental Investigation on Operational Conditions

AbstractIn modular construction, quality control is a crucial step in meeting quality requirements, leading to the completion of a project within the planned schedule and cost. Currently, quality i...

An Automatic 3D Point Cloud Registration Method Based on Biological Vision

When measuring surface deformation, because the overlap of point clouds before and after deformation is small and the accuracy of the initial value of point cloud registration cannot be guaranteed, traditional point cloud registration methods cannot be applied. In order to solve this problem, a complete solution is proposed, first, by fixing at least three cones to the target. Then, through cone vertices, initial values of the transformation matrix can be calculated. On the basis of this, the point cloud registration can be performed accurately through the iterative closest point (ICP) algorithm using the neighboring point clouds of cone vertices. To improve the automation of this solution, an accurate and automatic point cloud registration method based on biological vision is proposed. First, the three-dimensional (3D) coordinates of cone vertices are obtained through multi-view observation, feature detection, data fusion, and shape fitting. In shape fitting, a closed-form solution of cone vertices is derived on the basis of the quadratic form. Second, a random strategy is designed to calculate the initial values of the transformation matrix between two point clouds. Then, combined with ICP, point cloud registration is realized automatically and precisely. The simulation results showed that, when the intensity of Gaussian noise ranged from 0 to 1 mr (where mr denotes the average mesh resolution of the models), the rotation and translation errors of point cloud registration were less than 0.1° and 1 mr, respectively. Lastly, a camera-projector system to dynamically measure the surface deformation during ablation tests in an arc-heated wind tunnel was developed, and the experimental results showed that the measuring precision for surface deformation exceeded 0.05 mm when surface deformation was smaller than 4 mm.

Mobile projective augmented reality for collaborative robots in construction

Augmenting virtual construction information directly in a physical environment is promising to increase onsite productivity and safety. However, it has been found that using off-the-self augmented reality (AR) devices, such as goggles or helmets, could potentially cause more health, safety, and efficiency concerns in complex real-world construction projects, due to the restricted field of view and the non-negligible weight of those devices. To address these issues, we propose a mobile projective AR (MPAR) framework in which the AR device is detached from human workers and carried by one or more mobile collaborative robots (co-robots). MPAR achieves glassless AR that is visible to the naked eye using a camera-projector system to superimpose virtual 3D information onto planar or non-planar physical surfaces. Since co-robots often need to move during the operation, we design algorithms to ensure consistent mobile projection with two major components: projector pose estimation and projection image generation. For planar surfaces, MPAR is achieved by a homography-based pose estimation and image warping. For non-planar surfaces, MPAR uses iterative closest point (ICP) for pose estimation and common graphics pipelines to generate projection images. We conducted both qualitative and quantitative experiments to validate the feasibility of MPAR in a laboratory setting, by projecting 1) as-planned building information onto a planar surface, and 2) as-built 3D information onto a piece-wise planar surface. Our evaluation demonstrated centimeter-level projection accuracy of MPAR from different distances and angles to the two types of surfaces.