Camera Projection Research Articles

Overview of Structure from Motion

In this paper is given an overview of tools and techniques for obtain information about the geometry of 3D scenes from 2D images. The generating three-dimensional structure from a series of 2D images or video of scenes is known as Structure from Motion (SfM). A central tenet of structure from motion is that, given the position of a feature in one image, it is possible to find the corresponding position of the same feature in successive images. We described methods for the simultaneous recovery of 3D points and camera projection matrices using corresponding image points in multiple views. Structure from Motion (SfM) is a fascinating field within computer vision that seeks to reconstruct a three-dimensional structure of the environment from a sequence of two-dimensional images. At the heart of SfM lies a set of sophisticated algorithms that enable the extraction of spatial information from a series of images. Two public repositories that may be of use are Open SfM and Colmap.

Real-time accident risk identification for freeway weaving segments based on video analytics

Accurately identifying the risk of traffic accidents in real time for specific road sections is crucial for implementing proactive traffic control strategies. The existing studies have focused on real-time prediction of collisions between two vehicles, identification of accident risks on road segments relies heavily on historical accident data. This study proposes a framework for identifying road section accident risks at the granular level by utilizing traffic conflicts as an intermediary. In this framework, vehicles are detected and tracked in real time and the pixel coordinates are then efficiently converted into world coordinates based on the camera projection principle. A binary logit model is constructed to identify road section accident risks using carefully selected macroscopic traffic flow parameters. The proposed framework was validated on two one-sided ramp weaves in Jinan, China, achieving an accuracy of 76.2 %. The findings will serve as a foundation for traffic safety monitoring and proactive traffic control.

W-VSLAM: A Visual Mapping Algorithm for Indoor Inspection Robots.

In recent years, with the widespread application of indoor inspection robots, high-precision, robust environmental perception has become essential for robotic mapping. Addressing the issues of visual-inertial estimation inaccuracies due to redundant pose degrees of freedom and accelerometer drift during the planar motion of mobile robots in indoor environments, we propose a visual SLAM perception method that integrates wheel odometry information. First, the robot's body pose is parameterized in SE(2) and the corresponding camera pose is parameterized in SE(3). On this basis, we derive the visual constraint residuals and their Jacobian matrices for reprojection observations using the camera projection model. We employ the concept of pre-integration to derive pose-constraint residuals and their Jacobian matrices and utilize marginalization theory to derive the relative pose residuals and their Jacobians for loop closure constraints. This approach solves the nonlinear optimization problem to obtain the optimal pose and landmark points of the ground-moving robot. A comparison with the ORBSLAM3 algorithm reveals that, in the recorded indoor environment datasets, the proposed algorithm demonstrates significantly higher perception accuracy, with root mean square error (RMSE) improvements of 89.2% in translation and 98.5% in rotation for absolute trajectory error (ATE). The overall trajectory localization accuracy ranges between 5 and 17 cm, validating the effectiveness of the proposed algorithm. These findings can be applied to preliminary mapping for the autonomous navigation of indoor mobile robots and serve as a basis for path planning based on the mapping results.

An event-triggered background-oriented schlieren technique combined with dynamic projection using dynamic mirror device

Abstract This paper reports a high-frequency event-triggered background-oriented schlieren (BOS) technique using a combination of an event-triggered camera and dynamic projection. To combine the advantages of continuous and pulsed illumination for the event-triggered camera, a novel background pattern is first developed to incorporate static and dynamic textures generated through projection utilizing a dynamic mirror device (DMD). Then, a specific post-processing algorithm is proposed to reconstruct frames with high time accuracy from event data. This technique allows for the continuous observation and capturing of flows at 4000 frames per second (FPS) with a very low cost, breaking through the short operating times of current high-frame-rate BOS. Moreover, the proposed BOS technique can visualize the flow in real-time with high temporal accuracy, a capability that is challenging to achieve with traditional BOS. To examine the proposed technique, BOS experiments were conducted on a sweeping jet actuator with various inlet pressure. The sweeping dynamics and the start-up process of the sweeping jet at various inlet pressure were visualized and investigated. It is found that the proposed event-triggered BOS can continuously visualize and record the jet flow at a resolution of 1280 × 720 pixels with an equivalent frame rate of up to 4000 FPS. The oscillation frequency of the sweeping jet was found to increase linearly with increasing inlet pressure. It reaches 117.2 Hz at an inlet pressure of 0.5 Mpa. Within the first ten milliseconds or so of start-up, the shape of the sweep was found to be symmetrical. Within the next hundred milliseconds, the jet commences to sweep and saturates. The start-up time of the sweeping jet was quantitatively measured and was observed to decrease with increased inlet pressures.

Semantic 3D reconstruction-oriented image dataset for building component segmentation

In image-driven 3D building reconstruction, instance segmentation is fundamental to pixel-wise building component detection, which can be fused with 3D data like point clouds and meshes via camera projection for semantic reconstruction. While deep learning-based segmentation has obtained promising results, it relies heavily on large-scale datasets for training. Unfortunately, existing large-scale image datasets often include irrelevant objects that obstruct building components, making them unsuitable for 3D building reconstruction. This paper addresses this gap by introducing a large-scale building image dataset to facilitate building component segmentation for 3D reconstruction. The dataset comprises 3378 images captured from both interiors and exteriors of 36 university buildings, annotated with 49,380 object instances across 11 classes. Rigorous quality control measures were employed during data collection and annotation. Evaluation of five typical deep learning-based instance segmentation models demonstrates the dataset's suitability for training and its value as a benchmark dataset for building component segmentation.

Physical Layer Encryption for CO-OFDM Systems Enabled by Camera Projection Scrambler

In this paper, we propose a camera projection approach to enhance the physical layer security of coherent optical orthogonal frequency division multiplexing (CO-OFDM) systems. The data are converted to the new location by the camera projection module in the encryption system, where the 5D hyperchaotic system provides the keys for the camera projection module. The simulated 16QAM CO-OFDM security system over 80 km SSMF is shown to provide a key space of about 9 × 1090 through the five-dimensional (5D) hyperchaotic system, making it impossible for eavesdroppers to obtain valid information, and the peak-to-average power ratio (PAPR) is reduced by about 0.8 dB.

86‐3: Binocular Camera Eye Tracking Algorithm for Naked Eye 3D Display

Through real‐time acquisition of eye position using eye tracking, viewers can experience stereoscopic scenes from the naked eye 3D display. This paper proposes binocular camera eye tracking method that utilizes disparity to perceive the actual inter‐pupillary distance of different users. This method estimates the head posture based on facial landmark points and exploits camera projection to determine the spatial coordinate of the pupil. Furthermore, collaborative filtering approach is introduced to mitigate the jitter in pupil position. The experimental results demonstrate that the proposed algorithm enhances the performance in eye tracking compared to monocular method.

Robust autonomous vehicle control by leveraging multi-stage MPC and quantized CNN in HIL Framework

This study presents a novel path planning and path-following system designed explicitly for real-world driving conditions in autonomous vehicles. The pipeline considers uncertainties in the dynamic parameters of the vehicle, which are common in real-world scenarios. To address these uncertainties, a robust multi-stage model predictive control approach is developed to generate optimal control outputs. A customized convolutional neural network, Nano-FastSCNN, quantized to half-precision, is employed for accurately segmenting left and right lane boundaries. To ensure that our visual planner is robust against adverse weather conditions, we have rigorously evaluated its generalization capabilities under challenging weather types. The centerline of these lanes is then selected as the desired path and transformed into the camera reference frame by leveraging the camera projection matrix. The controller utilizes this reference trajectory to guide the vehicle along the desired trajectory. To demonstrate real-time processing capabilities, the entire pipeline, including the vision system module and controller, is implemented in a hardware-in-the-loop setup utilizing three NVIDIA Jetson devices to demonstrate the scalability of our architecture. Comprehensive evaluations demonstrate that the pipeline achieves an inference speed of 15.6 FPS, even on the less powerful NVIDIA Jetson Nano. Moreover, the system successfully alleviates the detrimental effects of uncertainties and reliably follows the desired path.

A Geometric Model for Polarization Imaging on Projective Cameras

A Geometric Model for Polarization Imaging on Projective Cameras

LightGyro: A Batteryless Orientation Measuring Scheme Based on Light Reflection

In industrial production, the orientation of facility components can indicate whether the facility is on a regular operating track. For example, when a component gets loose, the orientation variation of the component would exceed the normal range. A common approach for orientation measurement is to attach an inertial measurement unit (IMU) to the target device. However, the IMU requires additional power maintenance. This article presents LightGyro, a cheap and efficient batteryless scheme to measure the orientation, in which we attach a reflective film to the target device and use a camera to capture the light spot on the reflective film. The basic idea of LightGyro is to extract the light spots in the captured frame and use their pixel coordinates to infer the orientation. It is difficult to recognize a single light spot, because the spot lacks distinctive features. To solve the problem, we switch light sources on and off to regulate the appearance of light spots and utilize frame subtraction to extract light spots. The depth of field of light spot is lost in the process of camera projection, which is necessary for the orientation measurement. To address the issue, we propose a light array-based reflection model to extract the depth of field from the relative positions of multiple light spots. To the best of our knowledge, this is the first work to utilize reflection to measure orientation. Experiment results show that the orientation error of LightGyro decreases with the increasing length of the reflection route and the orientation error can achieve less than 1 ˆ .

Event-triggered background-oriented schlieren: high-frequency visualization of a heated jet flow.

This Letter reports a novel, to our knowledge, event-triggered background-oriented schlieren (EBOS) technique using a combination of an event-triggered camera and pulsed laser speckle projection. The BOS images are reconstructed using the event data generated by the pulsed laser speckle projection and then processed to obtain the density and temperature distribution of the flow. This technique enables continuous visualization and recording of flows at kFPS frame rates with a very low cost, breaking through the short operating times of existing high-frame-rate BOSs. To examine the event-triggered BOS technique, tests are conducted on a hot air gun. The measured temperature distribution coincides with the thermocouple data with an error of no more than 10.8%. Measurements taken during the start-up of the hot air gun demonstrate that the presented technique can measure the evolution of the jet temperature for at least 150 s, as well as capture the localized unsteady turbulent structure in the heated jet flow.

INFLUENCE ANALYSIS OF THREE DIFFERENT CAMERA CALIBRATION METHODS ON A CAMERA-PROJECTOR MEASURING SYSTEM

The challenges arising from mass production with high quality level require advanced error measurement systems and optical metrology systems. Nowadays, it is more frequent to find this type of controls at the manufacturing industry into the quality control procedures as long as the need for a faster capture speed and precision has increased. However, these techniques can be improved. The accuracy of 3D measurement systems formed by a camera-projector pair depends directly on the calibration procedure used. The projector is usually modelled as the inverse projection of a pin-hole camera, which presents the option to calibrate the system by two different approaches: as a whole system or separately. The most common approach is the use of a camera previously calibrated, followed by the calibration of the projector. Studies show that the uncertainty of the camera parameters from its calibration propagates to the projector parameters and several authors conclude that the three most widespread are: Tsai, Zhang and Direct Linear Calibration. The objective of this study is to have a clear comprehension of the relationships between the camera and projector parameters and of how uncertainty is propagated to the measurement system error. Therefore, the three calibration methods previously mentioned and some of them possible combinations are analyzed.

Metrology of Short-Length Measurers-Development of a Comparator for the Calibration of Measurers Based on Image Processing and Interferometric Measurements.

For the calibration of linear scales, comparators are generally used. Comparators are devices that enable the movement of an evaluation apparatus over a calibrated scale along a linear base with high precision. The construction of a comparator includes a movable carriage that carries the device for the evaluation of the position of the given edge of the line scale relative to the beginning of the scale. In principle, it involves a camera capturing the scale of the measurer, where the position of the camera's projection center is measured using an interferometer. This article addresses the development of a comparator assembled from low-cost components, as well as the description of systematic influences related to the movement of individual parts of the system, such as the inclination and rotation of the camera and directional and height deviations during the carriage's movement. This article also includes an evaluation of the edge of the given scale with subpixel accuracy, addressing distortion elimination and excluding the influences of impurities or imperfections on the scale. The proposed solution was applied to linear-scale measurers, such as leveling rods with coded and conventional scales and measuring tapes. The entire process of measurement and evaluation was automated.

Estimation of planar angles from non-orthogonal imaging.

Photogrammetry-based methods are traditionally used to estimate the geometrical parameters using optical images. These methods employ specific equipment, computationally sophisticated and expensive algorithms, and utilize projective geometry to reconstruct real-life scenes up to a scale. In this work, we used a computationally less-expensive method for sparse reconstruction to estimate the planar angles using two-view geometry and linear algorithms from non-orthogonal images acquired by a smartphone camera. First, intrinsic camera parameters were determined. Next, scale-invariant feature transform was used to identify the correspondence points from each pair of images. Epipolar constraint was applied on all these points to determine the essential matrix using the eight-point algorithm. Thereafter, extrinsic camera parameters were estimated from the essential matrix and combined with the intrinsic matrix to get the camera projection matrix. Finally, linear triangulation was used to get the sparse point cloud representing the scene. Planar angles were estimated by backprojecting the chosen image points and applying simple vector algebra on the obtained 3D points. The method was successful in estimating the planar angles in less than 10s on non-curved edges with an average error of 3% by using only ten images. Given the simplicity of methods used, this technique can be integrated into a smartphone for on-site measurements as well as large deformations.

Revisiting Viewing Graph Solvability: an Effective Approach Based on Cycle Consistency.

In the structure from motion, the viewing graph is a graph where the vertices correspond to cameras (or images) and the edges represent the fundamental matrices. We provide a new formulation and an algorithm for determining whether a viewing graph is solvable, i.e., uniquely determines a set of projective cameras. The known theoretical conditions either do not fully characterize the solvability of all viewing graphs, or are extremely difficult to compute because they involve solving a system of polynomial equationswith a large number of unknowns. The main result of this paper is a method to reduce the number of unknowns by exploiting cycle consistency. We advance the understanding of solvability by (i) finishing the classification of all minimal graphs up to 9 nodes, (ii) extending the practical verification of solvability to minimal graphs with up to 90 nodes, (iii) finally answering an open research question by showing that finite solvability is not equivalent to solvability, and (iv) formally drawing the connection with the calibrated case (i.e., parallel rigidity). Finally, we present an experiment on real data that shows that unsolvable graphs may appear in practice.

Critical configurations for three projective views

The problem of structure from motion is concerned with recovering the 3-dimensional structure of an object from a set of 2-dimensional images taken by unknown cameras. Generally, all information can be uniquely recovered if enough images and point correspondences are provided, yet there are certain cases where unique recovery is impossible; these are called \emph{critical configurations}. We use an algebraic approach to study the critical configurations for three projective cameras. We show that all critical configurations lie on the intersection of quadric surfaces, and classify exactly which intersections constitute a critical configuration.

Inter-sphere consistency-based calibration for camera–projector pairs

We propose an inter-sphere consistency-based method for camera–projector pairs (CPPs) calibration. Previous sphere-based calibration works have only considered independent constraints from each sphere, and thus require at least three spheres for calibration. In this paper, it is shown that additional constraints can be extracted from consistency between estimated parameters from different spheres, which we term the inter-sphere consistency (ISC). Such ISC constraints facilitate more flexible and easy-to-use calibration using only two spheres, which has been considered a challenging ill-posed problem and never been well addressed before. Extensive experimental results have been provided for the validation.

Seismic performance assessment of medical equipment using experimentally validated rolling and toppling nonlinear models

During the last decades, several hospitals stopped service due to severe seismic damage to nonstructural components and medical equipment. This article investigates the dynamic behavior of medical equipment deployed in a full-scale, five-story building with two different support conditions, base isolated (BI) and fixed to the base (FB). Two nonlinear mathematical models, namely, rolling and toppling, are used to simulate the observed experimental responses using recorded data and the camera projection technique (CPT). CPT generated the possibility of measuring equipment horizontal displacements, slips, rotations, rocking, and toppling responses. The Euler–Lagrange formulation, along with the Stribeck friction model, was used to numerically model the rolling and in-plane rotation behavior of a four-wheel mobile cart (4WMC), which is considered to represent the trajectory of equipment supported on caster wheels. Besides, a simple two-dimensional (2D) rigid block model was used to describe the rocking and toppling behavior of locked and free-standing equipment. It was found that the 4WMC model was sensitive to the platform and wheels’ initial orientation angles, as well as the static and kinetic friction coefficients. As it should be, the toppling model was sensitive to the block dimensions and the intensity of the shaking. It was concluded that both numerical models successfully predicted the equipment rolling, in-plane rotation, rocking, and toppling behavior, as long as neither severe lateral impacts nor significant changes in the equipment mass occur during the motion.

Scalable Spatial Memory for Scene Rendering and Navigation

Neural scene representation and rendering methods have shown promise in learning the implicit form of scene structure without supervision. However, the implicit representation learned in most existing methods is non-expandable and cannot be inferred online for novel scenes, which makes the learned representation difficult to be applied across different reinforcement learning (RL) tasks. In this work, we introduce Scene Memory Network (SMN) to achieve online spatial memory construction and expansion for view rendering in novel scenes. SMN models the camera projection and back-projection as spatially aware memory control processes, where the memory values store the information of the partial 3D area, and the memory keys indicate the position of that area. The memory controller can learn the geometry property from observations without the camera's intrinsic parameters and depth supervision. We further apply the memory constructed by SMN to exploration and navigation tasks. The experimental results reveal the generalization ability of our proposed SMN in large-scale scene synthesis and its potential to improve the performance of spatial RL tasks.

Performance of NeRF and its optimized models on generating images from 2D to 3D of animation or game character

Neural Radiance Fields (NeRF) has recently emerged as a promising approach for single-image 3D scene reconstruction. Despite this, most studies on NeRF have focused on real-world data or items with established camera projection connections, rather than snapshots of video game characters that may not have a distinct camera projection connection. Although some research has been conducted on the use of animatable mannequins, the majority of these studies have used photographs of real people as their data source. To date, there has been limited research into the use of NeRF for animation or game characters. In this paper, we compare and evaluate a range of NeRF models on a dataset of animated game characters. The models considered include original NeRF, Block-NeRF, Mip-NeRF, and Pixel-NeRF, each of which has its own design choices and trade-offs. Our results show that Mip-NeRF models are effective at generating high-quality images of game characters on our dataset, while Pixel-NeRF performs poorly on our dataset. Despite these findings, our research demonstrates the feasibility of using NeRF-based models to create realistic character models or animations for video games, with potential applications in game development and virtual reality.