Image Plane Of Camera Research Articles

Seedling Leaf Morphology Measurement Based on Photometric Stereo

In order to achieve the on-line nondestructive assessment for vegetable seedling growth, the method and system for measuring the seedling’s leaf morphology based on photometric stereo vision, was introduced in this paper. Through parsing the image luminance information of the seedling leaves under 4 LED light sources, the discrete normal vectors of leaf’s surface could be determined based on photometric stereo vision. According to the normal vectors’ distribution, the leaf’s space plane was fit. And the leaf’s inclination angle was measured as the angle between the fitting plane and camera imaging plane, thus its actual length and size also could be accurately measured. As the test result shown, compared with the manual measurement, the auto-measurement method in this paper has an average error of 3.71mm for leaf length, and 62.57mm2 for leaf size, which was valuable for developing the automatic seedling monitoring system.

ArUco/Gaze Tracking in Real Environments

The emergence of affordable mobile eye-trackers has allowed to study gaze behavior in real-world environments. However, the gaze mapping from recorded video to a static reference image is a complex and open problem. Finding a reference image within the video frames, i.e., image matching, can give satisfying results, but occluded or overlapped objects are almost impossible to locate using this technique. We suggest using ArUco fiducial markers (and their associated software library available in OpenCV) to map gaze to dynamic Areas Of Interest (AOIs) within a reference image. Although such markers have been used previously, technical details of marker detection and mapping have been sparse. The current approach consists of three steps: (1) define an AOI using markers, then (2) resolve any conflict among overlapping AOIs, and (3) map the gaze point to the reference image. A dynamic AOI can be defined using one or more corner markers. When camera rotations are limited and the object is relatively orthogonal to the camera, it is possible to define an AOI using only one corner marker. When the camera rotates, its pose estimation is required to project corner points to the camera image plane. An AOI can also be defined with four corner markers, which has the advantage of robustness with respect to camera rotations, and no a priori required knowledge of the physical dimensions of the object. The two approaches can be combined, e.g., when using four corner markers and one of the markers is most (due to occlusion or view angle), the basis vectors can be used to interpolate the position of the lost marker. When two or more AOIs overlap and all the markers are tracked, gaze should be marked on the AOI closer to the camera. The distance to an object can be defined knowing the length of the object, the number of pixels spanned on the image and the pre-computed camera focal parameter. Once the AOIs are defined and marker overlaps are resolved, the gaze point can be mapped to the coordinates of the reference image using homography.

Adaptive Visual Servoing of Unmanned Aerial Vehicles in GPS-Denied Environments

This paper presents an adaptive output feedback-based visual servoing law for a quadrotor unmanned aerial vehicle equipped with a single downward facing camera. The objective is to regulate the relative position and yaw of the vehicle to a planar target consisting of multiple points using a minimal sensor set, i.e., an inertial measurement unit and a vision sensor. A set of first order image moment features, defined in the image plane of a virtual camera with zero roll and pitch motion, is used for visual servoing. It has been observed in previous work that various system uncertainties, such as aerodynamics constants and attitude estimation bias, result in steady-state errors if not being compensated. By treating those uncertainties as unknown system parameters, an adaptive backstepping controller is developed. As the given visual servoing law is an output feedback controller, the translational velocity measurement from the global position system is not required. The asymptotic stability of the error dynamics is proven. Experimental results are provided to demonstrate the controller performance.

Robust geometric, phase and colour structured light projection system calibration

This paper introduces a new comprehensive procedure for both geometric and colour calibration of structured light system. In order to perform both geometric and colour calibration procedure, a new calibration artifact is proposed. The intrinsic and extrinsic parameters of projector and camera are estimated by using an extended pinhole camera model with a tangential and radial distortion. Camera image plane coordinates are obtained by extracting features from images of a calibration artifact. Projector image plane coordinates are calculated on the basis of continuous phase maps obtained from a fringe pattern phase reconstruction procedure. In order to stereo calibrate camera-projector system, pairs of corresponding image plane points are calculated with subpixel accuracy. In addition, one of three pattern views is used in colour calibration. RGB values of a colour field pattern detected by camera and their reference values are compared. This comparison leads to derivation of a colour transformation matrix. The performance of the proposed method is tested by measuring plane, sphere and distance reference. Also 360 degrees complex object 3D model from a set of measurements is obtained. Residual mean errors for all tests performed are calculated.

Plume propagation direction determination with SO<sub>2</sub> cameras

Abstract. SO2 cameras are becoming an established tool for measuring sulfur dioxide (SO2) fluxes in volcanic plumes with good precision and high temporal resolution. The primary result of SO2 camera measurements are time series of two-dimensional SO2 column density distributions (i.e. SO2 column density images). However, it is frequently overlooked that, in order to determine the correct SO2 fluxes, not only the SO2 column density, but also the distance between the camera and the volcanic plume, has to be precisely known. This is because cameras only measure angular extents of objects while flux measurements require knowledge of the spatial plume extent. The distance to the plume may vary within the image array (i.e. the field of view of the SO2 camera) since the plume propagation direction (i.e. the wind direction) might not be parallel to the image plane of the SO2 camera. If the wind direction and thus the camera–plume distance are not well known, this error propagates into the determined SO2 fluxes and can cause errors exceeding 50 %. This is a source of error which is independent of the frequently quoted (approximate) compensation of apparently higher SO2 column densities and apparently lower plume propagation velocities at non-perpendicular plume observation angles.Here, we propose a new method to estimate the propagation direction of the volcanic plume directly from SO2 camera image time series by analysing apparent flux gradients along the image plane. From the plume propagation direction and the known location of the SO2 source (i.e. volcanic vent) and camera position, the camera–plume distance can be determined. Besides being able to determine the plume propagation direction and thus the wind direction in the plume region directly from SO2 camera images, we additionally found that it is possible to detect changes of the propagation direction at a time resolution of the order of minutes. In addition to theoretical studies we applied our method to SO2 flux measurements at Mt Etna and demonstrate that we obtain considerably more precise (up to a factor of 2 error reduction) SO2 fluxes. We conclude that studies on SO2 flux variability become more reliable by excluding the possible influences of propagation direction variations.

Design, fabrication, and testing of a Shack-Hartmann sensor with an automatic registration feature.

In this research, design, construction, and testing of an innovative Shack-Hartmann sensor are described. As the most critical component, a polymer microlens array is injection molded and mounted on a board-level CMOS camera such that the focal plane of the microlens array is on the camera's image plane. To allow for automatic registration of the spots of the measured area, a diffusing surface was created at the center of the lens array in the same diamond machining process in an uninterrupted operation. This unique diffusing surface does not generate an image spot. The no-spot feature functions as the reference in the measurement on the camera's image plane. Using this unique feature, large global tip-tilt error can be detected and eliminated. In this research, both experiments and simulation have shown that the Shack-Hartmann sensor built using low cost components is capable of precision wavefront detection. This research also demonstrated that automatic registration based on the diffusing surface is simple and reliable.

Real-time depth controllable integral imaging pickup and reconstruction method with a light field camera

In this paper, we develop a real-time depth controllable integral imaging system. With a high-frame-rate camera and a focus controllable lens, light fields from various depth ranges can be captured. According to the image plane of the light field camera, the objects in virtual and real space are recorded simultaneously. The captured light field information is converted to the elemental image in real time without pseudoscopic problems. In addition, we derive characteristics and limitations of the light field camera as a 3D broadcasting capturing device with precise geometry optics. With further analysis, the implemented system provides more accurate light fields than existing devices without depth distortion. We adapt an f-number matching method at the capture and display stage to record a more exact light field and solve depth distortion, respectively. The algorithm allows the users to adjust the pixel mapping structure of the reconstructed 3D image in real time. The proposed method presents a possibility of a handheld real-time 3D broadcasting system in a cheaper and more applicable way as compared to the previous methods.

State transformation-based dynamic visual servoing for an unmanned aerial vehicle

abstractIn this paper, we propose a visual servoing control for a quadrotor unmanned aerial vehicle (UAV) which is based on a state transformation technique. The UAV is equipped with a single downwards facing camera, and the motion control objective is the regulation of relative displacement and yaw to a stationary visual target located on the ground. The state transformation is defined by a system of partial differential equations (PDEs) which eliminate roll and pitch rate dependence in the transformed image feature kinematics. A method for computing the general solutions of these PDEs is given, and we show a particular solution reduces to an established virtual camera approach. We treat point and line cases and introduce image moment features defined in the virtual camera image plane. Robustness of the control design is improved by accounting for attitude measurement bias, and uncertainty in thrust gain, mass, and image feature depth. The asymptotic stability of the closed-loop is proven. The method is based on a simple proportional-integral-derivative (PID) structure which can be readily implemented on-board. Experimental results show improved performance relative to previous work.

Adaptive Image-Based Trajectory Tracking Control of Wheeled Mobile Robots With an Uncalibrated Fixed Camera

In this paper, the uncalibrated image-based trajectory tracking control problem of wheeled mobile robots will be studied. The motion of the wheeled mobile robot can be observed using an uncalibrated fixed camera on the ceiling. Different from traditional vision-based control strategies of wheeled mobile robots in the fixed camera configuration, the camera image plane is not required to be parallel to the motion plane of the wheeled mobile robots and the camera can be placed at a general position. To guarantee that the wheeled mobile robot can efficiently track its desired trajectory, which is specified by the desired image trajectory of a feature point at the forward axis of the wheeled mobile robot, we will propose a new adaptive image-based trajectory tracking control approach without the exact knowledge of the camera intrinsic and extrinsic parameters and the position parameter of the feature point. To eliminate the nonlinear dependence on the unknown parameters from the closed-loop system, a depth-independent image Jacobian matrix framework for the wheeled mobile robots will be developed such that unknown parameters in the closed-loop system can be linearly parameterized. In this way, adaptive laws can be designed to estimate the unknown parameters online, and the depth information of the feature point can be allowed to be time varying in this case. The Lyapunov stability analysis will also be performed to show asymptotical convergence of image position and velocity tracking errors of the wheeled mobile robot. The simulation results based on a two-wheeled mobile robot will be given in this paper to illustrate the performance of the proposed approach as well. The experimental results based on a real wheeled mobile robot will also be provided to validate the proposed approach.

The geometry of confocal curves for passing through a door

This paper presents a geometric approach to the problem of steering a robot subject to nonholonomic constraints through a door by using only visual measurements coming from a single fixed on-board monocular camera. The door is represented by two landmarks located on its vertical supports. After exploring the geometric structure that naturally emerges from the problem statement, e.g., bundle of hyperbolae, ellipses, and circles, we exploit this planar geometry to provide stabilizing feedback control laws to drive the vehicle through the middle of the door. Using visual servoing, we prove that this geometry can be directly measured in the camera image plane. Hence, we provide an image-based control scheme, avoiding the use of a state observer. Simulations in a realistic scenario and experiments are provided to show the effectiveness of the feedback control laws.

Study of Pose Detection Algorithm for Specific Object Based on Monocular Vision

This paper brings forward a monocular vision detection algorithm in allusion to the specific object’s pose based on Hausdorff Distance. At first, according to the mathematic model which has been established, the 2D template sequence is generated by projecting the specific object in different poses into the image plane of a virtual camera. Then, in order to predigest calculations and accelerate the matching speed during the image matching, the algorithm adopts the local-mean Hausdorff Distance as matching estimate and adopts the search strategy based on hiberarchy which reduces the searching rang by the threshold method before accurate match. In the end of this paper, the experiment that measures the poses of the clamp via the different clamp images respectively is given to testify the validity and speediness of this algorithm.

Using a three-dimensional optical simulation to investigate uncertainty in image-based dimensional measurements

Many advanced dimensional measurements exist that incorporate two-dimensional images of the object to be measured, such as structured-light measurements and photogrammetry. The measurements are quite complex and often suitable calibration artifacts are not available; consequently, a traceable uncertainty analysis is difficult. Advanced optical simulation packages can be used to assess important aspects of the uncertainty. These packages can simulate a virtual object to be measured, simulate a virtual camera, and trace rays as they leave a light source, scatter from the object, propagate through the camera, and arrive at the camera image plane. The result is a virtual image of the object whose dimensions are known. The virtual image(s) can then be processed as part of a dimensional measurement and the results compared to the known dimensions of the object. This allows different uncertainty aspects to be isolated and explored. It can be used to validate and understand the current methods of estimating measurement uncertainty, explore optimal measurement conditions, and/or test data analysis algorithms. These types of measurements involve a camera calibration process and the technique can also be used to check this step. This paper demonstrates the approach as applied to a photogrammetry measurement of a box.

Fusing target information from multiple views for robust visual tracking

In this study, the authors address the problem of tracking a single target in a calibrated multi‐camera surveillance system with information on its location in the first frame of each view. Recently, tracking with online multiple instance learning (OMIL) has been shown to give promising tracking results. However, it may fail in a real surveillance system because of problems arising from target orientation, scale or illumination changes. In this study, the authors show that fusing target information from multiple views can avoid these problems and lead to a more robust tracker. At each camera node, an efficient OMIL algorithm is used to model target appearance. To update the OMIL‐based classifier in one view, a co‐training strategy is applied to generate a representative set of training bags from all views. Bags extracted from each view hold a unique weight depending on similarity of target appearance between the current view and the view which contains the classifier that needs to be updated. In addition, target motion on a camera's image plane is modelled by a modified particle filter guided by the corresponding object two‐dimensional (2D) location and fused 3D location. Experimental results demonstrate that the proposed algorithm is robust for human tracking in challenging scenes.

Temporal synchronization in mobile sensor networks using image sequence analysis

This paper addresses the problem of estimating the temporal synchronization in mobile sensors' networks, by using image sequence analysis of their corresponding scene dynamics. Unlike existing methods, which are frequently based on adaptations of techniques originally designed for wired networks with static topologies, or even based on solutions specially designed for ad hoc wireless sensor networks, but that have a high energy consumption and a low scalability regarding the number of sensors, this work proposes a novel approach that reduces the problem of synchronizing a general number $$N$$ N of sensors to the robust estimation of a single line in $${\mathbb {R}}^{N+1}$$ R N + 1 . This line captures all temporal relations between the sensors and can be computed without any prior knowledge of these relations. It is assumed that (1) the network's mobile sensors cross the field of view of a stationary calibrated camera that operates with constant frame rate and (2) the sensors trajectories are estimated with a limited error at a constant sampling rate, both in the world coordinate system and in the camera's image plane. Experimental results with real-world and synthetic scenarios demonstrate that our method can be successfully used to determine the temporal alignment in mobile sensor networks.

航天时间延迟积分CCD相机摆扫成像快速几何校正设计与分析

High-resolution space-borne remote sensors usually adopt mobile imaging, combining pushbroom and whiskbroom modes. In these modes, to achieve fast and accurate splicing and imaging application, a mathematical modeling method is proposed to match optical image motion tracing. Through the analysis of impact of scan angle and earth surface, the degrees of velocity mismatch and imaging deformation are computed and fast geometric correction is achieved by compensating the image shift for camera image plane at each point. Finally, physics time delayed and integration (TDI) CCD imaging simulation system is used for the simulation analysis of imaging. Simulation results show that the image shift amount increases and the image distortion turns serious with the increase of the scan angle. Velocity mismatch imaging is geometrically rapidly corrected under different scanning angles using optical tracing matching model. The mean square errors of simulation and experimental imaging quality are analyzed. The difference reaches to -0.000011 between simulation and experimental imaging and the mean square error of the corrected image turns small. The results show that this method can meet the demand of the ground satellite camera imaging simulation. The calibration method has high efficiency and is easy to splice.

Calculation Method of Earth-Atmosphere Stray Light Illuminance on Low-orbit Space Cameras

The earth-atmosphere stray light can lower contrast ratio and SNR of spatial optical remote sensors, or even fail the system at certain conditions. Errors in the calculation of the stray light illuminance based on point source are very large, for the satellite’s altitude is only several hundred kilometers and far less than the radius of the earth. This paper proposed a stray light illuminance calculating method which viewed the earth as a plane stray source and introduced the method in detail. Using the calculating method, earth-atmosphere stray light illuminance on the camera image plane for a low-orbit space camera at a typical position is calculated, and the results were compared with those calculated by viewing the earth as a point source, which evidently show that, when considering the earth-atmosphere stray light’s effects on low-orbit space cameras, it’s necessary to treat the earth as a plane extended light source

Touchscreen Everywhere: On Transferring a Normal Planar Surface to a Touch-Sensitive Display

We address how a human-computer interface with small device size, large display, and touch-input facility can be made possible by a mere projector and camera. The realization is through the use of a properly embedded structured light sensing scheme that enables a regular light-colored table surface to serve the dual roles of both a projection screen and a touch-sensitive display surface. A random binary pattern is employed to code structured light in pixel accuracy, which is embedded into the regular projection display in a way that the user perceives only regular display but not the structured pattern hidden in the display. With the projection display on the table surface being imaged by a camera, the observed image data, plus the known projection content, can work together to probe the 3-D workspace immediately above the table surface, like deciding if there is a finger present and if the finger touches the table surface, and if so, at what position on the table surface the contact is made. All the decisions hinge upon a careful calibration of the projector-camera-table surface system, intelligent segmentation of the hand in the image data, and exploitation of the homography mapping existing between the projector's display panel and the camera's image plane. Extensive experimentation including evaluation of the display quality, hand segmentation accuracy, touch detection accuracy, trajectory tracking accuracy, multitouch capability and system efficiency are shown to illustrate the feasibility of the proposed realization.

Projector calibration with error surface compensation method in the structured light three-dimensional measurement system

In the structured light three-dimensional measurement system, calibration is the key to the measurement accuracy; however, conventional calibration methods for the projector are either too complex or inaccurate. Therefore, we propose a simple and accurate method to calibrate the projector. In this method, the calibration points in the camera image plane can be mapped to the projector according the homography of the planar projection. In addition, an error surface compensation method is developed to minimize mapping errors caused by lens distortion of the camera and projector. As a result, the projector can be calibrated with the same method as the camera. Experiments are conducted to verify the effectiveness of the proposed method.

Hardware/Software Co-Design of 2D-to-3D Video Conversion on FPGA

With advances in technology, 3D video technology becomes possible and attractive. However, there are still many pre-recorded 2D videos/images which need to get transferred to 3D. Hence this paper presents a high quality view synthesis algorithm and architecture for 2D-to-3D video conversion. During the process of view synthesis, the monocular depth information together with the intermediate view is synthesized to the left-eye and right-eye view. The proposed view synthesis algorithm consists of two parts: 3D image warping and inpainting (hole filling). 3D image warping transforms a 2D camera image plane to a 3D coordinate plane. However the integer grid points of the reference are warped to irregularly spaced points in the virtual view, resulting in occlusion problems. Thus inpainting is needed to fix the virtual images. The proposed algorithm shows an improved PSNR gain of 0.2~1.5dB. We adopt hardware/software co-design to accomplish the proposed view synthesis algorithm. For this we implemented the image inpainting on a FPGA device and the remaining algorithm in software.

Adaptive and robust algorithms and tests for visual-based navigation of a space robotic manipulator

Optical navigation for guidance and control of robotic systems is a well-established technique from both theoretic and practical points of view. According to the positioning of the camera, the problem can be approached in two ways: the first one, “hand-in-eye”, deals with a fixed camera, external to the robot, which allows to determine the position of the target object to be reached. The second one, “eye-in-hand”, consists in a camera accommodated on the end-effector of the manipulator. Here, the target object position is not determined in an absolute reference frame, but with respect to the image plane of the mobile camera. In this paper, the algorithms and the test campaign applied to the case of the planar multibody manipulator developed in the Guidance and Navigation Lab at the University of Rome La Sapienza are reported with respect to the eye-in-hand case. In fact, being the space environment the target application for this research activity, it is quite difficult to imagine a fixed, non-floating camera in the case of an orbital grasping maneuver. The classic approach of Image Base Visual Servoing considers the evaluation of the control actions directly on the basis of the error between the current image of a feature and the image of the same feature in a final desired configuration. Both simulation and experimental tests show that such a classic approach can fail when navigation errors and actuation delays are included. Moreover, changing light conditions or the presence of unexpected obstacles can lead to a camera failure in target acquisition. In order to overcome these two problems, a Modified Image Based Visual Servoing algorithm and an Extended Kalman Filtering for feature position estimation are developed and applied. In particular, the filtering shows a quite good performance if target's depth information is supplied. A simple procedure for estimating initial target depth is therefore developed and tested. As a result of the application of all the novel approaches proposed, the experimental test campaign shows a remarkable increase in the robustness of the guidance, navigation and control systems.