Camera Reference Frame Research Articles

Robust autonomous vehicle control by leveraging multi-stage MPC and quantized CNN in HIL Framework

This study presents a novel path planning and path-following system designed explicitly for real-world driving conditions in autonomous vehicles. The pipeline considers uncertainties in the dynamic parameters of the vehicle, which are common in real-world scenarios. To address these uncertainties, a robust multi-stage model predictive control approach is developed to generate optimal control outputs. A customized convolutional neural network, Nano-FastSCNN, quantized to half-precision, is employed for accurately segmenting left and right lane boundaries. To ensure that our visual planner is robust against adverse weather conditions, we have rigorously evaluated its generalization capabilities under challenging weather types. The centerline of these lanes is then selected as the desired path and transformed into the camera reference frame by leveraging the camera projection matrix. The controller utilizes this reference trajectory to guide the vehicle along the desired trajectory. To demonstrate real-time processing capabilities, the entire pipeline, including the vision system module and controller, is implemented in a hardware-in-the-loop setup utilizing three NVIDIA Jetson devices to demonstrate the scalability of our architecture. Comprehensive evaluations demonstrate that the pipeline achieves an inference speed of 15.6 FPS, even on the less powerful NVIDIA Jetson Nano. Moreover, the system successfully alleviates the detrimental effects of uncertainties and reliably follows the desired path.

Polarized Skylight Navigation Using Full Sky Imager

Navigation is a skill used by all life. Although many plants and animals have developed natural navigation methods, humans typically employ a compass. A magnetic compass relies on the Earth's magnetic field. In the presence of other magnetic fields, such as in many electrical systems, its efficacy is degraded. For this reason, a new form of compass not reliant on the orientation of a magnetic field is investigated.
 The proposed method uses the atmospheric polarization of sunlight to determine heading. As light rays hit the atmosphere, they are scattered by gas molecules. During a scattering event, the E-vector of a ray is uniformly oriented with respect to the sun position, as described by the Rayleigh model. E-vector angles (AoLP) are consistently orthogonal to the solar meridian. This feature of the Rayleigh model can be used to determine the solar meridian from a polarized sky image.
 Sky images were taken using a polarizing camera and fed to a computer for classification. The classifier was based on a method published by Lu et al. It has three steps:
 
 Extract the AoLP from intensity data.
 Generate a binary image by selecting pixels representing an orthogonal AoLP.
 Apply a Hough transform to the binary image to extract heading.
 
 During testing, the compass was aligned with north. Then, an image was taken and classified. The solar azimuth angle in the camera reference frame was read as 125°. The recognized solar meridian with respect to north was 131°. Since the camera was aligned with north, these angles are comparable. The compass error was estimated at ±6°.
 The help of my colleagues has been integral, and I’d like to thank them. Truly, the work done on this project was made possible by ongoing guidance from Dr. Muhammad Alam. I am grateful for his support.

Tracker/Camera Calibration for Accurate Automatic Gaze Annotation of Images and Videos.

Modern appearance-based gaze tracking algorithms require vast amounts of training data, with images of a viewer annotated with "ground truth" gaze direction. The standard approach to obtain gaze annotations is to ask subjects to fixate at specific known locations, then use a head model to determine the location of "origin of gaze". We propose using an IR gaze tracker to generate gaze annotations in natural settings that do not require the fixation of target points. This requires prior geometric calibration of the IR gaze tracker with the camera, such that the data produced by the IR tracker can be expressed in the camera's reference frame. This contribution introduces a simple tracker/camera calibration procedure based on the PnP algorithm and demonstrates its use to obtain a full characterization of gaze direction that can be used for ground truth annotation.

STRUCTURE-FROM-MOTION PROCESSING OF AERIAL ARCHIVE PHOTOGRAPHS: SENSITIVITY ANALYSES PAVE THE WAY FOR QUANTIFYING GEOMORPHOLOGICAL CHANGES SINCE 1978 IN LA RÉUNION ISLAND

Abstract. Landscapes have been photographed dozens of times at scales ca. 1/25,000 and better since World War II. Scans are distributed freely online (e.g. remonterletemps.ign.fr). In parallel, Structure-from-Motion (SFM) software made photogrammetric processing easy to non-specialists. Yet puzzling questions crop up to use both: (i) Can raw scans be used as is? (ii) Can Ground Control Points (GCP) and checkpoints be safely collected from a web portal? (iii) How many parameters are sufficient for camera interior orientation? (iv) Are single flight camera networks sufficient to constrain camera models compared to multiple flights? (v) Are photogrammetric Digital Surface Models (DSM) fit for quantifying landslide activity? Processing of scanned black-and-white 1/27,000 photographs from IGN flown in May 1978 over Cirque de Salazie in La Réunion Island answer these questions. We find that raw scanned photographs need translation, rotation and cropping to match the camera reference frame. GCP and Check point coordinates collected on geoportail.gouv.fr with assumed accuracy of 10 m, achieved ca. 7 m accurate SFM registration. The optimal camera model uses only 4 parameters: f, cx, cy and K1. Compared to a 2015 lidar Digital Terrain Model (DTM), the 0.66 m/pixel DSM of 1978 has a median deviation of −1.39 m ± 3.34 m (Median Absolute Deviation) which is comparable to GCP quality. Elevation difference more importantly reveals, for the first time, the 37 years and 13 cyclones cumulated landslides pattern on Cirque de Salazie. Photographic archives hold decades-long 3D history. SFM is a game changer for landslide risk mitigation planning.

Experiment for pose estimation of uncooperative space debris using stereo vision

Acquiring the relative pose between an uncooperative target and a chaser satellite poses a unique problem for any active debris removal mission. A lack of prior knowledge regarding the target's motion, mass distribution and shape limits the possibilities for accurately tracking the target. In this paper, a stereo-camera pair, mounted on a chaser satellite, is used to extract unique features on the surface of an unknown, uncooperative target using the scale invariant feature transform (SIFT). The features are used as measurement input to an extended Kalman Filter (EKF) that makes use of the simultaneous localisation and mapping (SLAM) approach. The orientation and position of the target relative to the chaser is estimated, along with the target's linear and angular velocities. This motion is estimated relative to the camera reference frame (CRF) while the system simultaneously calculates the shape and size of the target. A simulation environment is created to test and verify the estimation algorithm. The integration of the feature extractor with the EKF is tested using real camera data. A laboratory experiment was conducted to capture image sequences of a moving target. A number of practical considerations is highlighted when such a system is to be applied to a mission. Results show that stereo cameras, along with the EKF-SLAM approach is a viable method for fully autonomous space debris mitigation systems.

Recognition and Pose Estimation of Auto Parts for an Autonomous Spray Painting Robot

The autonomous operation of industrial robots with minimal human supervision has always been in high demand. To prepare the autonomous operation of a car part spray painting robot, novel object detection, and pose estimation algorithms have been developed in this paper. The object detection part used principal components analysis (PCA) to reduce the dimension of three-dimensional (3-D) point cloud to 2-D binary image. Distance measure between the auto and cross correlation of the binary features was established to find out the similarity between them. Resultantly, the type of auto part was successfully obtained. Furthermore, iterative closest point (ICP) algorithm was used to estimate the pose difference of the auto part with respect to the camera reference frame, which was mounted on the robot. An issue with ICP's lack of robustness to local minimum was solved by the combination of ICP and genetic algorithm (GA). This allowed the optimization of pose error and addressed the problem of local minimum entrapment in ICP. For experimental validation: the proposed object recognition pipeline was implemented in both serial and parallel programming paradigms. The results were obtained for the acquired point clouds of side body car parts and compared with the major 3-D object detection systems in terms of computational cost. Pose estimation error was calculated with both ICP and the modified point set registration schemes, and it was shown to be decreasing in the case of later. All shown results supported the research claims.

A passive camera based determination of a non-cooperative and unknown satellite's pose and shape

The relevance of autonomy in space systems during rendezvous and docking operations has been lately increasing. At the scope, a robust GNC architecture is required, which strictly relies on the navigation system's performance and must assure both high efficiency and safety, i.e. low errors and no collisions with the target satellite. One of the most explored fields is the optical navigation one. Using passive optical sensors such as cameras can give high benefit in terms of characterization of the observed scene, thus enlarging the consciousness of what is going on in the mission scenario. The present research investigates the development of a filter which can estimate the shape and relative attitude, position and velocity of a non-cooperative, possibly unknown satellite orbiting around Earth, observed by a camera and a distance sensor mounted on a chaser satellite, whose objective is to successfully complete a docking maneuver. The image taken at a certain time is processed, features are extracted from it and matched with the ones extracted from the image at the previous time step. The matched features along with the relative distance measured by the distance sensor are merged inside an unscented Kalman filter, which predicts, updates and improves the state's estimate throughout the iterations. The expedient used in the filter is to give a 3D characterization to the 2D features used as measurements. The filter estimates the 3D coordinates of these points, i.e. the target's shape, in the camera reference frame, which depend on the target's attitude dynamics and the chaser's relative orbital dynamics. Thus, the target's attitude parameters, i.e. the quaternions, and angular velocity vector, the relative position and velocity vectors and the tracked 3D points are all included in the state vector and estimated by the filter. Subsequently, the 3D point coordinates are determined in the body reference frame. By doing this for all the tracked points, a 3D map of the target can be built.

Prospective motion correction using coil-mounted cameras: Cross-calibration considerations.

Optical prospective motion correction substantially reduces sensitivity to motion in neuroimaging of human subjects. However, a major barrier to clinical deployment has been the time-consuming cross-calibration between the camera and MRI scanner reference frames. This work addresses this challenge. A single camera was mounted onto the head coil for tracking head motion. Two new methods were developed: (1) a rapid calibration method for camera-to-scanner cross-calibration using a custom-made tool incorporating wireless active markers, and (2) a calibration adjustment method to compensate for table motion between scans. Both methods were tested at 1.5T and 3T in vivo. Simulations were performed to determine the required mechanical tolerance for repositioning of the camera. The rapid calibration method is completed in a short (<30 s) scan, which is carried out only once per installation. The calibration adjustment method requires no extra scan time and runs automatically whenever the system is used. The mechanical tolerance analysis indicates that most motion (90% reduction in voxel displacement) could be corrected even with far larger camera repositioning errors than are observed in practice. The methods presented here allow calibration of sufficient quality to be carried out and maintained with no additional technologist workload. Magn Reson Med 79:1911-1921, 2018. © 2017 International Society for Magnetic Resonance in Medicine.

A Combined Microphone and Camera Calibration Technique With Application to Acoustic Imaging

We present a calibration technique for an acoustic imaging microphone array, combined with a digital camera. Computer vision and acoustic time of arrival data are used to obtain microphone coordinates in the camera reference frame. Our new method allows acoustic maps to be plotted onto the camera images without the need for additional camera alignment or calibration. Microphones and cameras may be placed in an ad-hoc arrangement and, after calibration, the coordinates of the microphones are known in the reference frame of a camera in the array. No prior knowledge of microphone positions, inter-microphone spacings, or air temperature is required. This technique is applied to a spherical microphone array and a mean difference of 3 mm was obtained between the coordinates obtained with this calibration technique and those measured using a precision mechanical method.

An EPID based method for performing high accuracy calibration between an optical external marker tracking device and the LINAC reference frame

With the increasing use of external 3D optical tracking cameras to guide modern radiation therapy procedures, it has become vitally important to have an accurate camera to linear accelerator (LINAC) reference frame calibration. To eliminate errors present in current calibration procedures based on the manual hand alignment of a device using the light field crosshairs and in room guidance lasers, a semiautomated quantitative calibration approach requiring only use of an electronic portal imaging device (EPID) was developed. A phantom comprised of seven highly IR reflective plastic BBs was placed on the LINAC treatment couch and imaged with both a 3D stereoscopic IR imager and the on board megavoltage (MV) EPID imager. Having knowledge of the optically determined 3D positions and projected EPID images of the BBs, simulated annealing was used to optimize the location of the BBs in the LINAC frame using four different optimization functions. Singular value decomposition was then used to calculate the transformation matrix between the camera and LINAC reference frames. Results were then compared to a traditional camera calibration method for overall accuracy. Using modeled data, the simulated annealing process was able to determine the actual locations of the BBs with a RMSE of 0.23 mm. Using projection images acquired with an MV imager, the process was able to determine locations of BBs within .26 mm. The results depend on the choice of optimization function. Results show that the method can be used to provide highly accurate spatial registration between an external 3D imaging reference frame and the LINAC frame. The experimental MV imager results, while not as precise as the simulated results, exceed 1 mm accuracy and the current accepted AAPM TG-142 standard of ≤2 mm positioning accuracy.

Flaws in the Computer Algorithm for Reconstructing a Scene from Two Projections

In 1981 Longuet-Higgins represented the world point by two vectors in the two camera reference frames and developed the essential matrix. Such a matrix is a relation between the corresponding image points on the two images of a world point on a rigid scene. The essential matrix is independent of the position and orientation of the cameras used to capture the two views. The calculation of the essential matrix requires the knowledge of at least five accurate pairs of corresponding points. The unavailability of a procedure that fulfills such a requirement led researchers to focus their attention on developing estimation methods of the essential matrix without questioning the mathematical correctness of its derivation. In this paper, we identify and expose flaws in Longuet-Higgins' derivation of the essential matrix. These flaws are the result of mixing up between the scalar product of vectors in a single reference frame and the transformation of vectors from one reference frame to another.

GPU implementation of a road sign detector based on particle swarm optimization

Road Sign Detection is a major goal of the Advanced Driving Assistance Systems. Most published work on this problem share the same approach by which signs are first detected and then classified in video sequences, even if different techniques are used. While detection is usually performed using classical computer vision techniques based on color and/or shape matching, most often classification is performed by neural networks. In this work we present a novel modular and scalable approach to road sign detection based on Particle Swarm Optimization, which takes into account both shape and color to detect signs. In our approach, in particular, the optimization of a single fitness function allows both to detect a sign belonging to a certain category and, at the same time, to estimate its position with respect to the camera reference frame. To speed up processing, the algorithm implementation exploits the parallel computing capabilities offered by modern graphics cards and, in particular, by the Compute Unified Device Architecture by nVIDIA. The effectiveness of the approach has been assessed on both synthetic and real video sequences, which have been successfully processed at, or close to, full frame rate.

On the Definition of Curve's Frame at the Points of inflection

AbstractThis paper deals with the problem of defining a frame along the curve at the point of inflection such that the intrinsic properties of the curve are reflected in the defined frame and the frame can be used as a local frame of the body moving on the curve. It is natural to use the Frenet Serret Frames of the curve whenever we need to define a local reference frame along the curve, but the problem with Frenet Serret Frame is that, it is not defined at the point of inflection. The problem gets serious if the point of inflection drags along for some times. i.e. if the direction of the unit tangent is not changing for some time. In such cases if we want to use the Frenet frame of the curve as the local reference frame of a body (a local frame for some sensor e.g camera reference frame) moving along the same curve, it will suffer from the problem of indeterminacy at the point of inflection. Therefore we have to define a frame which exists at all the points of the curve. This defined frame must retain the intrinsic properties of the curve. The Rotation Minimizing Frame is proposed as a solution to the problem in this paper.

Vision-Based Road-Following Using Proportional Navigation

This paper describes a new approach for autonomous road following for an unmanned air vehicle (UAV) using a visual sensor. A road is defined as any continuous, extended, curvilinear feature, which can include city streets, highways, and dirt roads, as well as forest-fire perimeters, shorelines, and fenced borders. To achieve autonomous road-following, this paper utilizes Proportional Navigation as the basis for the guidance law, where visual information is directly fed back into the controller. The tracking target for the Proportional Navigation algorithm is chosen as the position on the edge of the camera frame at which the road flows into the image. Therefore, each frame in the video stream only needs to be searched on the edge of the frame, thereby significantly reducing the computational requirements of the computer vision algorithms. The tracking error defined in the camera reference frame shows that the Proportional Navigation guidance law results in a steady-state error caused by bends and turns in the road, which are perceived as road motion. The guidance algorithm is therefore adjusted using Augmented Proportional Navigation Guidance to account for the perceived road accelerations and to force the steady-state error to zero. The effectiveness of the solution is demonstrated through high-fidelity simulations, and with flight tests using a small autonomous UAV.

Efficient linear solution of exterior orientation

This paper concerns an efficient algorithm for the solution of the exterior orientation problem. Orthogonal decompositions are used to first isolate the unknown depths of feature points in the camera reference frame, allowing the problem to be reduced to an absolute orientation with scale problem, which is solved using the singular value decomposition (SVD). The key feature of this approach is the low computational cost compared to existing approaches.