Camera Projection Research Articles

P‐59: A Pupil Distance based Monocular Camera Eye Tracking Algorithm for Naked Eye 3D Display

Naked eye 3D display makes it possible for viewers to watch stereoscopic scenes without wearing any auxiliary equipment. Eye tracking is an essential technology for viewers to see high quality stereo scenes at different positions. This paper proposed a pupil distance‐based monocular camera eye tracking algorithm, which detected pupils from face and estimated head pose to acquire the distances between eyes and the display. The method further adopted the camera projection to predict the 3D pupil positions. Experimental results demonstrated that the proposed algorithm had high accuracy and was effective on naked eye 3D display devices.

Camera distance helps 3D hand pose estimated from a single RGB image

Most existing methods for RGB hand pose estimation use root-relative 3D coordinates for supervision. However, such supervision neglects the distance between the camera and the object (i.e., the hand). The camera distance is especially important under a perspective camera, which controls the depth-dependent scaling of the perspective projection. As a result, the same hand pose, with different camera distances can be projected into different 2D shapes by the same perspective camera. Neglecting such important information results in ambiguities in recovering 3D poses from 2D images. In this article, we propose a camera projection learning module (CPLM) that uses the scale factor contained in the camera distance to associate 3D hand pose with 2D UV coordinates, which facilities to further optimize the accuracy of the estimated hand joints. Specifically, following the previous work, we use a two-stage RGB-to-2D and 2D-to-3D method to estimate 3D hand pose and embed a graph convolutional network in the second stage to leverage the information contained in the complex non-Euclidean structure of 2D hand joints. Experimental results demonstrate that our proposed method surpasses state-of-the-art methods on the benchmark dataset RHD and obtains competitive results on the STB and D+O datasets.

Critical configurations for two projective views, a new approach

This article develops new techniques to classify critical configurations for 3D scene reconstruction from images taken by unknown cameras. Generally, all information can be uniquely recovered if enough images and image points are provided, but there are certain cases where unique recovery is impossible; these are called critical configurations. In this paper, we use an algebraic approach to study the critical configurations for two projective cameras. We show that all critical configurations lie on quadric surfaces, and classify exactly which quadrics constitute a critical configuration. The paper also describes the relation between the different reconstructions when unique reconstruction is impossible.

Multiple Projector Camera Calibration by Fiducial Marker Detection

Multiple Projector Camera Calibration by Fiducial Marker Detection

3D human pose estimation in multi-view operating room videos using differentiable camera projections

ABSTRACT 3D human pose estimation in multi-view operating room (OR) videos is a relevant asset for person tracking and action recognition. However, the surgical environment makes it challenging to find poses due to sterile clothing, frequent occlusions and limited public data. Methods specifically designed for the OR are generally based on the fusion of detected poses in multiple camera views. Typically, a 2D pose estimator such as a convolutional neural network (CNN) detects joint locations. Then, the detected joint locations are projected to 3D and fused over all camera views. However, accurate detection in 2D does not guarantee accurate localisation in 3D space. In this work, we propose to directly optimise for localisation in 3D by training 2D CNNs end-to-end based on a 3D loss that is backpropagated through each camera’s projection parameters. Using videos from the MVOR dataset, we show that this end-to-end approach outperforms optimisation in 2D space.

Deep-learning-based parallel single-pixel imaging for efficient 3D shape measurement in the presence of strong interreflections by using sampling Fourier strategy

Strong interreflections cause intractable systematic errors to traditional 3D shaping methods, for example, fringe projection profilometry. Parallel single-pixel imaging (PSI) captures the light transport coefficients between the camera projector pair, thus overcoming the influence of interreflections. However, PSI requires a large number of measurements, causing poor capturing efficiency. To achieve efficient 3D shape measurement in the presence of strong interreflections, we present a deep-learning-based parallel single-pixel imaging method (dlPSI), featuring adopting a deep-learning network to achieve accurate light transport coefficient reconstruction and using sampling Fourier strategy to reduce measurements. The deep-learning network is designed to suppress the noise caused by undersampling the Fourier domains. Experiments prove that our dlPSI with the characteristic-based deterministic Fourier sampling strategy achieves high-quality 3D shape measurement while saving 94% of measurements.

WLTOG: An optimization approach for wild large‐range target omnidirectional geolocation based on monocular PTZ camera

AbstractThe pan‐tilt‐zoom (PTZ) cameras are increasingly popular in urban management and resource protection due to their omnidirectional coverage and reasonable cost. Traditionally, the research of monocular camera geolocation focuses on small‐range ones with a central projection camera under ideal conditions, making it challenging to meet the requirements of omnidirectional coverage and large‐range geolocation for PTZ cameras. In this article, a wild large‐range target omnidirectional geolocation (WLTOG) algorithm was proposed. Firstly, the WLTOG algorithm is used to transform the target in a single image to the camera's optical axis, thereby obtaining the camera posture value. After that, by using a high‐order overdetermined equation, it fits the nonlinear mapping relationship between multiple pairs of PTZ camera postures and the geospatial coordinates. At last, it realizes the bidirectional calculation between pixel coordinates and geospatial coordinates. The experimental PTZ camera is mounted on a communication tower in a wild environment. Therefore, this method can be utilized to construct an omnidirectional geolocation relationship that meets the requirements of omnidirectional geolocation within 945 m based on unknown accurate internal and external orientation elements. The WLTOG algorithm is conducive to the geometric measurement of video and GIS‐augmented video surveillance.

Fast and Accurate Pose Estimation with Unknown Focal Length Using Line Correspondences.

Estimating camera pose is one of the key steps in computer vison, photogrammetry and SLAM (Simultaneous Localization and Mapping). It is mainly calculated based on the 2D-3D correspondences of features, including 2D-3D point and line correspondences. If a zoom lens is equipped, the focal length needs to be estimated simultaneously. In this paper, a new method of fast and accurate pose estimation with unknown focal length using two 2D-3D line correspondences and the camera position is proposed. Our core contribution is to convert the PnL (perspective-n-line) problem with 2D-3D line correspondences into an estimation problem with 3D-3D point correspondences. One 3D line and the camera position in the world frame can define a plane, the 2D line projection of the 3D line and the camera position in the camera frame can define another plane, and actually the two planes are the same plane, which is the key geometric characteristic in this paper's estimation of focal length and pose. We establish the transform between the normal vectors of the two planes with this characteristic, and this transform can be regarded as the camera projection of a 3D point. Then, the pose estimation using 2D-3D line correspondences is converted into pose estimation using 3D-3D point correspondences in intermediate frames, and, lastly, pose estimation can be finished quickly. In addition, using the property whereby the angle between two planes is invariant in both the camera frame and world frame, we can estimate the camera focal length quickly and accurately. Experimental results show that our proposed method has good performance in numerical stability, noise sensitivity and computational speed with synthetic data and real scenarios, and has strong robustness to camera position noise.

Single-Shot Three-Dimensional Reconstruction Using Grid Pattern-Based Structured-Light Vision Method

Structured-light vision methods are widely employed for three-dimensional reconstruction. As a typical structured light pattern, grid pattern is extensively applied in single-shot three-dimensional reconstruction. The uniqueness of the grid feature retrieval is critical to the reconstruction. Most methods using grid pattern utilize the epipolar constraint to retrieve the correspondence. However, the low calibration accuracy of the camera–projector stereo system may impact the correspondence retrieval. An approach using grid pattern-based structured-light vision method is proposed. The grid pattern-based structured-light model was combined with the camera model and the multiple light plane equations. An effective extraction method of the grid stripe features was investigated. The system calibration strategy, based on coplanar constraint, is presented. The experimental setup consisted of a camera and an LED projector. Experiments were carried out to verify the accuracy of the proposed method.

Pose and Motion Estimation of Free-Flying Objects: Aerodynamics, Constrained Filtering, and Graph-Based Feature Tracking

In this article, we investigate the problem of dynamically estimating the instantaneous position, orientation, velocity, and angular velocity of an arbitrarily shaped object during its free flight based on image frames taken simultaneously by two high-speed cameras. Aerodynamic effects, including drag, lift, and Magnus forces, are modeled to describe the object’s flight. Observables are derived from combining dynamics with a camera projection model assuming two-view geometry, via the use of multiple quaternions. The state of the composed system can then be estimated via constrained Kalman filtering, to which a solution is presented for the case of multiple quadratic constraints. To keep track of appearing and disappearing visual features during flight, the estimation algorithm employs a graph matching-based technique to maintain a set of evolving hypotheses through evaluation, pruning, and addition. Experiments conducted over various objects have either provided validation against motions independently estimated using multiple accelerometers, or formed verification by matching flight images against projections based on the state estimates.

Technical note: Rapid and high-resolution deep learning-based radiopharmaceutical imaging with 3D-CZT Compton camera and sparse projection data.

The Compton camera (CC) has great potential in nuclear medicine imaging due to the high detection efficiency and the ability to simultaneously detect multi-energy radioactive sources. However, the finite resolution of the detectors will degrade the images that the real-world CC can obtain. Besides, the CC sometimes can be limited by the detection efficiency, leading to difficulty in using sparse projection data to realize high-resolution reconstruction with short-time measurement, which limits its clinical application for real-time or rapid radiopharmaceutical imaging. To overcome the difficulty and promote the usage of the CC in radiopharmaceutical imaging, we present a deep learning (DL)-based CC reconstruction method to realize rapid and high-resolution imaging with short-time measurement. We developed a DL-based algorithm MCBP-CCnet via Monte Carlo sampling-based back projection and a dedicated convolutional neural network, called CC-Net, to realize the rapid and high-resolution reconstruction with sparse projection data. A CC prototype based on a single three-dimensional position-sensitive CdZnTe (3D-CZT) detector was used to demonstrate the feasibility of our proposed method. The simulations and experiments of radiopharmaceutical imaging used the 3D-CZT CC and [18 F]NaF. A 3D-printing mouse phantom was also further used to evaluate the performance of the proposed method in animal molecular imaging. The simulation and experimental results showed that the proposed method could realize the images reconstruction within 5s for list-mode projection data and realized a rapid reconstruction within 35s for experimental radiopharmaceutical imaging based on the 3D-printing mouse phantom, as well as realized the high-resolution imaging with an accuracy of within 0.78mm in terms of the sparse projection data that only contained hundreds of events. Besides, the deviations between the reconstructed radiative activities and the exact values were less than 1.51%. The results demonstrated that the proposed method could realize the rapid and high-resolution CC reconstruction with sparse projection data obtained by the 3D-CZT CC and realize the high-resolution radiopharmaceutical imaging. The study in this paper also demonstrated the potential and feasibility of future applications of a 3D-CZT CC for real-time high-resolution radiopharmaceutical imaging with short-time measurement.

Applying SLAM Algorithm Based on Nonlinear Optimized Monocular Vision and IMU in the Positioning Method of Power Inspection Robot in Complex Environment

Under China’s Intelligent Electric Power Grid (IEPG), the research on IEPG inspection mode is of great significance. This work aims to improve the positioning and navigation performance of IEPG inspection robots in a complex environment. First, it reviews the monocular camera projection and the Inertial Measurement Unit (IMU) models. It also discusses the tight-coupling monocular Vision Inertial Navigation System (VINS) and the initialization theory of the Simultaneous Localization and Mapping (SLAM) system. Nonlinear optimization for SLAM by the Gauss–Newton Method (GNM) is established. Accordingly, this work proposes the SLAM system based on tight-coupling monocular VINS. The EuRoC dataset data sequence commonly used in visual-inertial algorithm testing in IEPG is used for simulation testing. The proposed SLAM system’s attitude and position estimation errors are analyzed on different datasets. The results show that the errors of roll, pitch, and yaw angle are acceptable. The errors of the X, Y, and Z axes are within 40 cm, meeting the positioning requirements of an Unmanned Aerial Vehicle (UAV). Meanwhile, the Root Mean Square Error (RMSE) evaluates the improvement of positioning accuracy by loop detection. The results testify that loop detection can reduce the RMSE and improve positioning accuracy. The attitude estimation tests the angle changes of pitch, roll, and yaw angles with time under a single rotation condition. The estimated value of the proposed SLAM algorithm is compared with the real value through Absolute Trajectory Error (ATE). The results show that the real value and the estimated value of attitude error can coincide well. Thus, the proposed SLAM algorithm is effective for positioning and navigation. ATE can also be controlled within ±2.5°, satisfying the requirements of navigation and positioning accuracy. The proposed SLAM system based on tight-coupling monocular VINS presents excellent positioning and navigation accuracy for the IEPG inspection robot. The finding has a significant reference value in the later research of IEPG inspection robots.

Analysis of phase errors introduced by single-shot color phase-shifting profilometry on colored objects

We aim to report an error analysis on single-shot profilometry when projecting a composite colored fringe pattern onto a surface containing different colors. Several studies have analyzed white surfaces using profilometry; however, when color is incorporated, new considerations must be taken into account. In this work, it is found theoretically and corroborated experimentally that each color introduces local phase variations that render in phase errors with a severity that is related to the crosstalk matrix of the projector–camera system. It is also shown that the phase errors are phase-shifting dependent and could be reduced with appropriate phase recovery algorithms. Experimental results are presented and compared with those obtained using a standard temporal four-step phase-shifting method.

Dancing Human Rights: Politics in Archive

In his solo work Archive, Israeli choreographer Arkadi Zaides provides a physical and choreographic interpretation of footage collected by the Israeli humanitarian group B’Tselem as part of its “Camera Project” initiative. Arkadi Zaides conducts a self-analysis of the modern Israeli body by extracting, imitating, and repeating this material. As isolated choreographic material, the gestures reform in an ongoing act of abstraction: as a critical pointing at absence, they defy any metaphysics of presence, activating an archive of gestures, focusing on the aesthetic and political potential for the disappearance, pausing poses and positions — literal and figurative, ambiguous positions. Here, deconstruction techniques call into question the linearity of a history’s causes and effects. This paper will discuss the synthesis of the following questions to illuminate the methodology and political impact of Zaides’ Archive. How does the duality of Archive help it to serve as a tool for criticising Israeli society? How does the implementation of Zaides’ forms of expression and dance techniques change the role of the Israeli people in Israeli-Palestinian conflicts? To what extent does Archive engage with real politics, both locally and globally?

A static and fast calibration method for line scan camera based on cross-ratio invariance

This paper presents a new calibration method of line scan camera model based on cross-ratio invariance using a calibration pattern on the top of the gauge block. Instead of using the 3D pattern, planner pattern with precise motion platform, or pre-calibrated matrix camera, only a planner pattern with gauge blocks is used in this new method. A group of data on the line scan camera projection plane is obtained by placing the planner pattern at a different height. Line scan camera parameters were calibrated through a derived mathematical model. This method is simple, inexpensive and easy to use. The experimental results demonstrated that the average reprojection error of this method is 0.0431, which is 62.2% lower than the traditional method, which proves the feasibility and effectiveness of this method.

Робота системи технічного зору по елементах поверхні космічного апарата в умовах її неповної видимості

Вирішується задача відстеження переміщення реперних об’єктів на зображеннях, що отримуються системою технічного зору, призначеною для вимірювання відносного положення і орієнтації некооперованих космічних апаратів на малих відстанях у складі системи забезпечення автоматичного стикування. Дана задача виникає при формуванні вихідних даних для так званої задачі PnP, в якій положення та орієнтація тіла у просторі визначаються за зображеннями контрольних точок поверхні цього тіла, отриманими камерою перспективної проєкції. При цьому положення контрольних точок відносно системи координат цільового тіла мають бути відомими. Оскільки цільовим тілом виступає абстрактний некооперований космічний апарат, для визначення контрольних точок можуть використовуватися лише наявні видимі неоднорідності його поверхні. Тому виникає запитання, як вибирати контрольні точки, щоб вони були прив’язані до поверхні тіла і могли бути знайдені з високою точністю на зображеннях за різних зовнішніх умов зйомки. Одним із варіантів вирішення є використання для прив’язки дрібнорозмірних конструктивних елементів некооперованого космічного апарата, які з великою ймовірністю будуть присутні на його поверхні. Тоді прив’язка контрольної точки здійснюється до відомої форми поверхні відповідного елемента, а визначення її положення на зображенні виконується через розпізнавання відповідного образу. У даній статті розглядається випадок використання системи технічного зору з монокулярною камерою. Розглядається етап роботи на близьких відстанях, коли в полі зору камери знаходиться ділянка поверхні стикування з розміщеними на ній елементами, що використовуються як реперні. Запропоновано метод знаходження на зображеннях положення точок прив’язки окремого класу реперних об’єктів при виконанні періодичної зйомки.

Pro-Cam SSfM: projector–camera system for structure and spectral reflectance from motion

Image-based reconstruction of an object’s 3D shape having the wavelength-by-wavelength spectral reflectance property enables higher-fidelity object 3D modeling compared with typical RGB-based modeling. In this paper, we propose a novel projector–camera system for practical and low-cost acquisition of a dense object 3D model with the spectral reflectance property. Different from existing spectral 3D data acquisition systems that use a dedicated multispectral camera or light, we use a standard RGB camera and an off-the-shelf projector as active illumination for both the 3D reconstruction and the spectral reflectance estimation. We first propose a calibration-free multi-view structured-light method to reconstruct the 3D points while estimating the intrinsic parameters and the poses of both the camera and the projector, which are alternately moved around the object during our image acquisition procedure. We then exploit the projector for multispectral imaging and estimate the spectral reflectance of each 3D point based on a novel spectral reflectance estimation method considering the geometric relationship between the reconstructed 3D points and the estimated projector positions. Experimental results on both synthetic and real data demonstrate that our system can precisely acquire a dense spectral 3D model using off-the-shelf devices.

Consensus-Based Optimization for 3D Human Pose Estimation in Camera Coordinates

3D human pose estimation is frequently seen as the task of estimating 3D poses relative to the root body joint. Alternatively, we propose a 3D human pose estimation method in camera coordinates, which allows effective combination of 2D annotated data and 3D poses and a straightforward multi-view generalization. To that end, we cast the problem as a view frustum space pose estimation, where absolute depth prediction and joint relative depth estimations are disentangled. Final 3D predictions are obtained in camera coordinates by the inverse camera projection. Based on this, we also present a consensus-based optimization algorithm for multi-view predictions from uncalibrated images, which requires a single monocular training procedure. Although our method is indirectly tied to the training camera intrinsics, it still converges for cameras with different intrinsic parameters, resulting in coherent estimations up to a scale factor. Our method improves the state of the art on well known 3D human pose datasets, reducing the prediction error by 32% in the most common benchmark. We also reported our results in absolute pose position error, achieving 80 mm for monocular estimations and 51 mm for multi-view, on average. Source code is available at https://github.com/dluvizon/3d-pose-consensus .

3D Reconstruction of Non-Lambertian Surfaces by Perspective Shape-From-Shading With Fast Viscosity Solution

Shape-from-shading (SFS) is an important method to reconstruct three-dimensional (3D) shape of a surface in photometry and computer vision. Lambertian surface reflectance and orthographic camera projection are two fundamental assumptions which generally result in undesirable reconstructed results since inaccurate imaging model is adopted. In this paper, we propose a new fast 3D shape reconstruction approach via the SFS method relaxing the two assumptions. To this end, the Oren-Nayar reflectance and perspective projection models are used to establish an image irradiance equation which depicts the relationship between the 3D shape of non-Lambertian surfaces and its corresponding two-dimensional (2D) shading image. Considering the light attenuation of the near point source, the image irradiance equation is transformed into a static Hamilton-Jacobi partial differential equation (PDE) by solving a quadratic equation. The viscosity solution of the resultant Hamilton-Jacobi PDE is approximated by using optimal control theory and iterative fast marching method starting from a viscosity supersolution. The performance of the proposed approach is evaluated on both synthetic and real-world images and the experimental results demonstrate that the proposed approach is accurate and fast.

Vision-Based Cable Displacement Measurement Using Side View Video.

Recent tragedies around the world have shown how accidents in the cable-stayed bridges can wreak havoc on the society. To ensure the safety of the cable-stayed bridges, several studies have estimated the cable tension force using the vibration of cables. Most of these methods for estimating the tension of a cable start with measuring the displacement of the cable. Recent development of commercial cameras provide opportunity for more convenient and efficient method for measuring the displacement of cable. However, traditional vision-based displacement measurement methods require the assumption that the movement of the cable should be measured in parallel to the camera plane. This assumption limits the installation location of the camera when measuring the displacement of a cable. Therefore, this study introduces a new vision-based cable displacement measurement system that can measure the displacement of a cable in various locations even when the camera is installed in the side of the cable. The proposed method consists of three phases: (1) camera projection matrix estimation, (2) cable tracking in the image coordinate, and (3) cable displacement estimation in the world coordinate. To validate the performance of the proposed method, a simulation-based validation test, a lab-scale validation test, and an on-site validation test were conducted. The simulation-based validation test verified the performance of the proposed method in an ideal condition, and the lab-scale validation test showed the performance of the method in physical environment. Finally, the on-site validation test showed that the proposed method can measure the cable displacement with a side view camera.