Camera Projection Research Articles

CALIBRATION PROCEDURES IN MID FORMAT CAMERA SETUPS

Abstract. A growing number of mid-format cameras are used for aerial surveying projects. To achieve a reliable and geometrically precise result also in the photogrammetric workflow, awareness on the sensitive parts is important. The use of direct referencing systems (GPS/IMU), the mounting on a stabilizing camera platform and the specific values of the mid format camera make a professional setup with various calibration and misalignment operations necessary. An important part is to have a proper camera calibration. Using aerial images over a well designed test field with 3D structures and/or different flight altitudes enable the determination of calibration values in Bingo software. It will be demonstrated how such a calibration can be performed. The direct referencing device must be mounted in a solid and reliable way to the camera. Beside the mechanical work especially in mounting the camera beside the IMU, 2 lever arms have to be measured in mm accuracy. Important are the lever arms from the GPS Antenna to the IMU's calibrated centre and also the lever arm from the IMU centre to the Camera projection centre. In fact, the measurement with a total station is not a difficult task but the definition of the right centres and the need for using rotation matrices can cause serious accuracy problems. The benefit of small and medium format cameras is that also smaller aircrafts can be used. Like that, a gyro bases stabilized platform is recommended. This causes, that the IMU must be mounted beside the camera on the stabilizer. The advantage is, that the IMU can be used to control the platform, the problematic thing is, that the IMU to GPS antenna lever arm is floating. In fact we have to deal with an additional data stream, the values of the movement of the stabiliser to correct the floating lever arm distances. If the post-processing of the GPS-IMU data by taking the floating levers into account, delivers an expected result, the lever arms between IMU and camera can be applied. However, there is a misalignment (bore side angle) that must be evaluated by photogrammetric process using advanced tools e.g. in Bingo. Once, all these parameters have been determined, the system is capable for projects without or with only a few ground control points. But which effect has the photogrammetric process when directly applying the achieved direct orientation values compared with an AT based on a proper tiepoint matching? The paper aims to show the steps to be done by potential users and gives a kind of quality estimation about the importance and quality influence of the various calibration and adjustment steps.

On the Generation of Ground Truth Data for Depth Reconstruction

Acquiring data sets for stereo or multi-view reconstruction with known ground truth is costly and time consuming. Freely available data sets mostly focus on specific scene conditions and might not reflect the intended application scenario. We propose a GPU accelerated ray tracing framework that allows to generate realistic images with adjustable camera properties and projection geometry. Due to the nature of ray tracing, ground truth data can easily be obtained. The system also supports projectors as used e.g. for structured light systems. This facilitates to study interaction effects between different pattern designs, camera and projector properties, as well as the utilized reconstruction algorithm.

손의 기하학적인 특성을 적용한 실시간 비전 기반 증강현실 시스템

본 논문에서는 손을 이용한 컴퓨터 비전 기술 기반의 증강 현실 시스템을 제안한다. 입력 영상에 가상의 물체를 정합하기 위해서는 정확한 특징점 추출과 추적 기술이 중요하다. 기존의 마커를 이용한 증강현실 시스템은 매우 안정성이 있지만 마커가 없이는 증강이 불가능하고 증강된 물체를 조작하는데 제한적인 상황을 제공하는 경향이 있다. 제안한 시스템은 손의 기하학적인 특성을 고려하여 적응적 최적 타원 검출 방법을 통해 손끝점을 특징점으로 추출한다. 그리고 손바닥 중심점을 기준으로 한 최단 거리 검출 방법을 이용하여 손끝점의 움직임을 추출해 가상의 객체를 안정적으로 정합한다. 실험을 통하여 특징점 추출이 약 82.0%의 정확도를 보였고 특징점 판별과 추적 성능 평가에서는 단지 약 1.8%와 2.0%의 오류를 보였다. 또한 제안한 시스템이 가상 객체의 안정적인 증강 측면에서, 효과적으로 카메라 사영 행렬을 획득하여 마커를 이용한 시스템을 대체할 수 있음을 확인하였다. In this paper, we propose an AR(augmented reality) system using user's bare hand based on computer vision. It is important for registering a virtual object on the real input image to detect and track correct feature points. The AR systems with markers are stable but they can not register the virtual object on an acquired image when the marker goes out of a range of the camera. There is a tendency to give users inconvenient environment which is limited to control a virtual object. On the other hand, our system detects fingertips as fiducial features using adaptive ellipse fitting method considering the geometric characteristics of hand. It registers the virtual object stably by getting movement of fingertips with determining the shortest distance from a palm center. We verified that the accuracy of fingertip detection over 82.0% and fingertip ordering and tracking have just 1.8% and 2.0% errors for each step. We proved that this system can replace the marker system by tacking a camera projection matrix effectively in the view of stable augmentation of virtual object.

A compact system to extract topography information from scenes viewed by a miniaturized submersible explorer

In images taken underwater, it is generally difficult to correctly extract distances and geometric information of objects. Different techniques, collectively referred to as photogrammetry, exist to measure features in images. One of these is to project a reference pattern onto an object in a scene viewed by a camera, and register the distortion of this pattern, to calculate the shape of, and distance to, that object. This method is implemented here on a miniaturized submersible explorer equipped with, among many other instruments, a camera. Diffractive optical elements (DOEs) have been designed and manufactured using microsystems technology, to, together with a laser diode, camera, and in-house developed software, provide a compact system for projecting reference patterns and analyzing their deformations. The system has been characterized by measuring the distances and angles of objects in a water tank, and attempting to reproduce their shapes. The range of operation of the system, verified to be at least one meter, is limited by the compact mounting in the small submersible and the cameras’ performance. The system was found to work well under turbid conditions as well as in water containing larger particles. Together with a vehicle-mounted camera, the compact and low-power DOE laser projection system enables topographical measurement.

Application of S-transform profilometry in eliminating nonlinearity in fringe pattern

The S-transform, conceptually viewed as an extension of the short-time Fourier transform and the wavelet transform, features a time-frequency representation known for its local spectral property and multiresolution strategy. It has been introduced in optical three-dimensional shape measurement based on a fringe projection technique recently. In this paper, an application of S-transform for demodulating fringe patterns affected by the nonlinearity has been discussed. Two methods based on the S-transform, called S-transform ridge method and S-transform filtering method, are used to eliminate the phase errors caused by nonlinear factors of the projector and CCD camera. The theoretical representations of S-transform ridge method and S-transform filtering method in fringe analysis are given. The computer simulation and experiment are carried out to verify our research. Compared with the reconstructions of the windowed Fourier transform, the wavelet transform and the S-transform ridge method, the S-transform filtering method gives a better result under considering the influence of the fringe nonlinearity.

Camera Calibration by Hybrid Hopfield Network and Self- Adaptive Genetic Algorithm

A new approach based on hybrid Hopfield neural network and self-adaptive genetic algorithm for camera calibration is proposed. First, a Hopfield network based on dynamics is structured according to the normal equation obtained from experiment data. The network has 11 neurons, its weights are elements of the symmetrical matrix of the normal equation and keep invariable, whose input vector is corresponding to the right term of normal equation, and its output signals are corresponding to the fitting coefficients of the camera's projection matrix. At the same time an innovative genetic algorithm is presented to get the global optimization solution, where the cross-over probability and mutation probability are tuned self-adaptively according to the evolution speed factor in longitudinal direction and the aggregation degree factor in lateral direction, respectively. When the system comes to global equilibrium state, the camera's projection matrix is estimated from the output vector of the Hopfield network, so the camera calibration is completed. Finally, the precision analysis is carried out, which demonstrates that, as opposed to the existing methods, such as Faugeras's, the proposed approach has high precision, and provides a new scheme for machine vision system and precision manufacture.

Projection ray intersecting location-based multicolour pseudo-random coded projected active vision method

3D reconstruction has been recognised as an important cue for tasks like active computer vision and image understanding. However, most existing techniques need to calibrate the intrinsic and extrinsic parameters of the projector, making the calculation very complex. This paper presents an active vision method of multicolour pseudo-random encoded projected for recovering the 3D shape of surface from one calibrated image, based on projection ray intersecting location. This method combines geometric and photometric information in order to reconstruct 3D shape only calibrating camera parameters and projection rays. Based on the thought of ray intersecting location, by projecting pseudo-random coded pattern onto surface of object, using a projected ray of a feature point at projector side, and an imaging ray of the same feature point received by camera, the 3D reconstruction is implemented by seeking the intersection point of the projected ray and the imaging ray. We show a number of examples to demonstrate the accuracy of the method. Experimental results on the real world images show that the proposed method reconstructs the 3D shape of objects very efficiently.

A rendering framework for multiscale views of 3D models

Images that seamlessly combine views at different levels of detail are appealing. However, creating such multiscale images is not a trivial task, and most such illustrations are handcrafted by skilled artists. This paper presents a framework for direct multiscale rendering of geometric and volumetric models. The basis of our approach is a set of non-linearly bent camera rays that smoothly cast through multiple scales. We show that by properly setting up a sequence of conventional pinhole cameras to capture features of interest at different scales, along with image masks specifying the regions of interest for each scale on the projection plane, our rendering framework can generate non-linear sampling rays that smoothly project objects in a scene at multiple levels of detail onto a single image. We address two important issues with non-linear camera projection. First, our streamline-based ray generation algorithm avoids undesired camera ray intersections, which often result in unexpected images. Second, in order to maintain camera ray coherence and preserve aesthetic quality, we create an interpolated 3D field that defines the contribution of each pinhole camera for determining ray orientations. The resulting multiscale camera has three main applications: (1) presenting hierarchical structure in a compact and continuous manner, (2) achieving focus+context visualization, and (3) creating fascinating and artistic images.

A New Solution for Camera Calibration and Real-Time Image Distortion Correction in Medical Endoscopy–Initial Technical Evaluation

Medical endoscopy is used in a wide variety of diagnostic and surgical procedures. These procedures are renowned for the difficulty of orienting the camera and instruments inside the human body cavities. The small size of the lens causes radial distortion of the image, which hinders the navigation process and leads to errors in depth perception and object morphology. This article presents a complete software-based system to calibrate and correct the radial distortion in clinical endoscopy in real time. Our system can be used with any type of medical endoscopic technology, including oblique-viewing endoscopes and HD image acquisition. The initial camera calibration is performed in an unsupervised manner from a single checkerboard pattern image. For oblique-viewing endoscopes the changes in calibration during operation are handled by a new adaptive camera projection model and an algorithm that infer the rotation of the probe lens using only image information. The workload is distributed across the CPU and GPU through an optimized CPU+GPU hybrid solution. This enables real-time performance, even for HD video inputs. The system is evaluated for different technical aspects, including accuracy of modeling and calibration, overall robustness, and runtime profile. The contributions are highly relevant for applications in computer-aided surgery and image-guided intervention such as improved visualization by image warping, 3-D modeling, and visual SLAM.

CCD characterization and measurements automation

Modern mosaic cameras have grown both in size and in number of sensors. The required volume of sensor testing and characterization has grown accordingly. For camera projects as large as the LSST, test automation becomes a necessity. A CCD testing and characterization laboratory was built and is in operation for the LSST project. Characterization of LSST study contract sensors has been performed. The characterization process and its automation are discussed, and results are presented. Our system automatically acquires images, populates a database with metadata information, and runs express analysis. This approach is illustrated on 55Fe data analysis. 55Fe data are used to measure gain, charge transfer efficiency and charge diffusion. Examples of express analysis results are presented and discussed.

High resolution acquisition of detailed surfaces with lens-shifted structured light

We present a novel 3D geometry acquisition technique at high resolution based on structured light reconstruction with a low-cost projector–camera system. Using a 1D mechanical lens-shifter extension in the projector light path, the projected pattern is shifted in subpixel scale steps with a granularity of up to 2048 steps per projected pixel, which opens up novel possibilities in depth accuracy and smoothness for the acquired geometry. Combining the mechanical lens-shifter extension with a multiple phase shifting technique yields a measuring range of 120×80 mm while at the same time providing a high depth resolution of better than 100 μ m . Reaching beyond depth resolutions achieved by conventional structured light scanning approaches with projector–camera systems, depth layering effects inherent to conventional techniques are fully avoided. Relying on low-cost consumer products only, we reach an area resolution of down to 55 μ m (limited by the camera). We see two main benefits. First, our acquisition setup can reconstruct finest details of small cultural heritage objects such as antique coins and thus digitally preserve them in appropriate precision. Second, our accurate height fields are a viable input to physically based rendering in combination with measured material BRDFs to reproduce compelling spatially varying, material-specific effects.

Phoenix in Motion: A Dance for the Camera Project for Urban Youth

Phoenix in Motion: A Dance for the Camera Project for Urban Youth is a hybrid dance for the camera project that centered on capturing a sense of community and place in the urban neighborhoods of Ph...

Camera–projector matching using unstructured video

This paper presents a novel approach for matching 2-D points between a video projector and a digital camera. Our method is motivated by camera–projector applications for which the projected image needs to be warped to prevent geometric distortion. Since the warping process often needs geometric information on the 3-D scene obtained from a triangulation, we propose a technique for matching points in the projector to points in the camera based on arbitrary video sequences. The novelty of our method lies in the fact that it does not require the use of pre-designed structured light patterns as is usually the case. The backbone of our application lies in a function that matches activity patterns instead of colors. This makes our method robust to pose, severe photometric and geometric distortions. It also does not require calibration of the color response curve of the camera–projector system. We present quantitative and qualitative results with synthetic and real-life examples, and compare the proposed method with the scale invariant feature transform (SIFT) method and with a state-of-the-art structured light technique. We show that our method performs almost as well as structured light methods and significantly outperforms SIFT when the contrast of the video captured by the camera is degraded.

Colorimetric color reproduction framework for screen relaxation of projection display

With the mobility developing of the projection display, the images are usually wanted to be projected onto the surround surfaces. Unlike the uniform white screen for the conventional projection display, the surround surfaces are usually colored with the random patterns. Herewith, the existed display technologies always failed on these surfaces due to the disability of the color reproduction. In order that the projection display can also obtain an expected rendering even on the patterned surfaces, a novel color reproduction framework was developed in this paper. This framework was focused on the achievement of the colorimetric color reproduction of the input image and composed of three processes: at first, the colorimetric characterization was accomplished to define the characterization color space and describe the workflow of the color reproduction based on the configuration of the projector–camera system. Then, the reproduction objective was tailored according to the original color of the input image and the reflectance of the surface. Finally, the desired input image was predicted and projected onto the surface to reconstruct the objective image. Several ISO images and printed screens were used to validate the performance of the developed framework. The experimental results demonstrated that the expected objectives could be properly achieved on various projection surfaces for different test images.

A flexible and rapid micro-adjustment algorithm for structured light 3D measurement system with camera–projector

The structured light 3D measurement system with camera–projector has been widely studied and applied because of its characteristics of simple structure. In practice, we find that slight shock, vibration, or long-term applying may slightly reduce its level of measurement accuracy; thus, correcting the system parameters is occasionally necessary. However, most calibration algorithms are complicated and time-consuming procedures, not suitable for flexible and rapid precise calibration operation. To address this limitation, a flexible and rapid system parameter micro-optimization algorithm based on an implicit projection model is proposed. With a flexible measurement of an arbitrary 3D gauge, the system parameters are optimized rapidly by Levenberg-Marquardt (L-M) algorithm; thus, the high degree of accuracy of the system can be restored. Experiments were performed to validate the availability and reliability of the adjustment method. The algorithm can meet the convenient and real-time accuracy maintenance in practical application for structured light 3D measurement system.

Passivity-based Visual Motion Observer with Panoramic Camera for Pose Control

This paper considers the vision-based estimation and pose control with a panoramic camera via passivity approach. First, a hyperbolic projection of a panoramic camera is presented. Next, using standard body-attached coordinate frames (the world frame, mirror frame, camera frame and object frame), we represent the body velocity of the relative rigid body motion (position and orientation). After that, we propose a visual motion observer to estimate the relative rigid body motion from the measured camera data. We show that the estimation error system with a panoramic camera has the passivity which allows us to prove stability in the sense of Lyapunov. The visual motion error system which consists of the estimation error system and the pose control error system preserves the passivity. After that, stability and L2-gain performance analysis for the closed-loop system are discussed via Lyapunov method and dissipative systems theory, respectively. Finally, simulation and experimental results are shown in order to confirm the proposed method.

An Road Obstacle Detection Method Based on Monocular Measurement

Obstacle detection in the intelligent vehicle vision navigation system occupies a very important role. The studies for the obstacles detecting, especially Monocular Measurement from the computer vision, simplifying monocular vision system to camera projection model. Getting the conversion relation between image coordinate and the world coordinate system through the geometry derivation to establish the measurement model and achieve the obstacle measurement. The experiment proved that the error of this measurement model selected is within the acceptable range.

Linearized Motion Estimation for Articulated Planes

In this paper, we describe the explicit application of articulation constraints for estimating the motion of a system of articulated planes. We relate articulations to the relative homography between planes and show that these articulations translate into linearized equality constraints on a linear least-squares system, which can be solved efficiently using a Karush-Kuhn-Tucker system. The articulation constraints can be applied for both gradient-based and feature-based motion estimation algorithms and to illustrate this, we describe a gradient-based motion estimation algorithm for an affine camera and a feature-based motion estimation algorithm for a projective camera that explicitly enforces articulation constraints. We show that explicit application of articulation constraints leads to numerically stable estimates of motion. The simultaneous computation of motion estimates for all of the articulated planes in a scene allows us to handle scene areas where there is limited texture information and areas that leave the field of view. Our results demonstrate the wide applicability of the algorithm in a variety of challenging real-world cases such as human body tracking, motion estimation of rigid, piecewise planar scenes, and motion estimation of triangulated meshes.

Augmenting Physical Objects by a Ubiquitous Spatial Projection Display Operating in the Visible Light Spectrum

Recently, mobile projector applications have become cutting-edge research for augmenting the physical world in a ubiquitous manner. By coupling a small projector with sensing devices, one can create a system that is able to recognize the physical world and provide appropriate augmentation via projection in real time. A popular sensing device for projection systems is a camera because it can acquire information about the projected image and the environment simultaneously. However, previous research in this field has focused on developing novel interactive applications and has paid scant attention to developing a system that retains absolute ubiquity for real-world scenarios. In this paper, we propose a vision-based framework that can be applied to any projector–camera paired system that requires absolute ubiquity in device and setup. The framework focuses on dealing with two unpredictable factors of the environment: the unknown geometry of the projection surface and the unknown number of physical objects to be augmented. Three problems are introduced, together with our proposed solutions. First, the problem of geometric calibration between the two devices is solved by using a beam splitter, enabling our framework to perform calibration precisely regardless of the 3D geometry of the environment. Second, we propose a multiple-target tracking approach to enable concurrent augmenting with infinite physical objects. Finally, we describe our solution for preventing real-time projection and real-time camera sensing from interfering with each other. By synchronizing the two devices at an appropriate time, the visual appearance of the environment is not changed when being seen by the camera, and visual analysis of the environment can be performed in a conventional manner. Using a small-scale laboratory setting, experiments were conducted to evaluate the accuracy and speed of all aspects of our proposed approach.

Color Information Presentation for Color Vision Defectives by Using a Projector Camera System

Color perception varies from individual to individual. It is difficult for a person with defective cones in the retina to recognize the difference between specific colors. We propose a method for presenting color information by using a projector camera system. The system projects border lines or color names on real object surfaces for specific color combinations. Registration of the projected images and real objects is executed by using projected markers. The effectiveness of the proposed method is verified through experiments.