Camera Projection Research Articles

Design guidelines for human–robot interaction with assistive robot manipulation systems

Abstract The design of user interfaces (UIs) for assistive robot systems can be improved through the use of a set of design guidelines presented in this article. As an example, the article presents two different UI designs for an assistive manipulation robot system. We explore the design considerations from these two contrasting UIs. The first is referred to as the graphical user interface (GUI), which the user operates entirely through a touchscreen as a representation of the state of the art. The second is a type of novel UI referred to as the tangible user interface (TUI). The TUI makes use of devices in the real world, such as laser pointers and a projector–camera system that enables augmented reality. Each of these interfaces is designed to allow the system to be operated by an untrained user in an open environment such as a grocery store. Our goal is for these guidelines to aid researchers in the design of human–robot interaction for assistive robot systems, particularly when designing multiple interaction methods for direct comparison.

Self-Localization of Mobile Robots Using a Single Catadioptric Camera with Line Feature Extraction.

This paper presents a novel self-localization technique for mobile robots using a central catadioptric camera. A unified sphere model for the image projection is derived by the catadioptric camera calibration. The geometric property of the camera projection model is utilized to obtain the intersections of the vertical lines and ground plane in the scene. Different from the conventional stereo vision techniques, the feature points are projected onto a known planar surface, and the plane equation is used for depth computation. The 3D coordinates of the base points on the ground are calculated using the consecutive image frames. The derivation of motion trajectory is then carried out based on the computation of rotation and translation between the robot positions. We develop an algorithm for feature correspondence matching based on the invariability of the structure in the 3D space. The experimental results obtained using the real scene images have demonstrated the feasibility of the proposed method for mobile robot localization applications.

LOW-COST DGPS ASSISTED AERIAL TRIANGULATION FOR SUB-DECIMETRIC ACCURACY WITH NON-RTK UAVS

Abstract. The possibility of equipping UAVs with lightweight GNSS receivers in order to estimate the camera position within a photogrammetric block allows for a reduction of the number of Ground Control Points (GCP), saving time during the field work and decreasing operational costs. Additionally, this makes it possible to build photogrammetric models even in morphologically complex areas or in emergency situations. This work is proposing a non-intrusive and low-cost procedure to retrieve the coordinates of the camera projection centre with decimetric accuracy. The method was designed and tested with the quadcopter DJI Matrice 210 V2 drone equipped with a DJI ZENMUSE X5S camera and an Emlid reach M, a low-cost, single-frequency (L1) GNSS receiver. GNSS observations are post-processed in PPK in order to obtain the UAV trajectory. Synchronization between the camera and the GNSS receiver is achieved by looking at the camera triggering timestamps in flight telemetry data, without requiring an electronic connection between camera and the GNSS that may be troublesome with commercial UAVs. Two surveys were carried out, respectively to calibrate and validate the procedure. The validation test evidenced the possibility of obtaining the coordinates of the camera projection centres with decimetric accuracy. The centre of projections can then be employed for GNSS-assisted aerial triangulation as input of the bundle block adjustment. Provided that at least one GCP is used, it is possible to reach centimetric accuracy on the ground.

3D SURFACE RECONSTRUCTION FROM MULTI-DATE SATELLITE IMAGES

Abstract. The reconstruction of accurate three-dimensional environment models is one of the most fundamental goals in the field of photogrammetry. Since satellite images provide suitable properties for obtaining large-scale environment reconstructions, there exist a variety of Stereo Matching based methods to reconstruct point clouds for satellite image pairs. Recently, a Structure from Motion (SfM) based approach has been proposed, which allows to reconstruct point clouds from multiple satellite images. In this work, we propose an extension of this SfM based pipeline that allows us to reconstruct not only point clouds but watertight meshes including texture information. We provide a detailed description of several steps that are mandatory to exploit state-of-the-art mesh reconstruction algorithms in the context of satellite imagery. This includes a decomposition of finite projective camera calibration matrices, a skew correction of corresponding depth maps and input images as well as the recovery of real-world depth maps from reparameterized depth values. The paper presents an extensive quantitative evaluation on multi-date satellite images demonstrating that the proposed pipeline combined with current meshing algorithms outperforms state-of-the-art point cloud reconstruction algorithms in terms of completeness and median error. We make the source code of our pipeline publicly available.

Preclinical evaluation of a markerless, real-time, augmented reality guidance system for robot-assisted radical prostatectomy.

Intra-operative augmented reality (AR) during surgery can mitigate incomplete cancer removal by overlaying the anatomical boundaries extracted from medical imaging data onto the camera image. In this paper, we present the first such completely markerless AR guidance system for robot-assisted laparoscopic radical prostatectomy (RALRP) that transforms medical data from transrectal ultrasound (TRUS) to endoscope camera image. Moreover, we reduce the total number of transformations by combining the hand-eye and camera calibrations in a single step. Our proposed solution requires two transformations: TRUS to robot, [Formula: see text], and camera projection matrix, [Formula: see text] (i.e., the transformation from endoscope to camera image frame). [Formula: see text] is estimated by the method proposed in Mohareri et al. (in J Urol 193(1):302-312, 2015). [Formula: see text] is estimated by selecting corresponding 3D-2D data points in the endoscope and the image coordinate frame, respectively, by using a CAD model of the surgical instrument and a preoperative camera intrinsic matrix with an assumption of a projective camera. The parameters are estimated using Levenberg-Marquardt algorithm. Overall mean re-projection errors (MRE) are reported using simulated and real data using a water bath. We show that [Formula: see text] can be re-estimated if the focus is changed during surgery. Using simulated data, we received an overall MRE in the range of 11.69-13.32 pixels for monoscopic and stereo left and right cameras. For the water bath experiment, the overall MRE is in the range of 26.04-30.59 pixels for monoscopic and stereo cameras. The overall system error from TRUS to camera world frame is 4.05mm. Details of the procedure are given in supplementary material. We demonstrate a markerless AR guidance system for RALRP that does not need calibration markers and thus has the capability to re-estimate the camera projection matrix if it changes during surgery, e.g., due to a focus change.

A Real Time Bolometer Tomographic Reconstruction Algorithm in Nuclear Fusion Reactors

In tokamak nuclear fusion reactors, one of the main issues is to know the total emission of radiation, which is mandatory to understand the plasma physics and is very useful to monitor and control the plasma evolution. This radiation can be measured by means of a bolometer system that consists in a certain number of elements sensitive to the integral of the radiation along straight lines crossing the plasma. By placing the sensors in such a way to have families of crossing lines, sophisticated tomographic inversion algorithms allow to reconstruct the radiation tomography in the 2D poloidal cross-section of the plasma. In tokamaks, the number of projection cameras is often quite limited resulting in an inversion mathematic problem very ill conditioned so that, usually, it is solved by means of a grid-based, iterative constrained optimization procedure, whose convergence time is not suitable for the real time requirements. In this paper, to illustrate the method, an assumption not valid in general is made on the correlation among the grid elements, based on the statistical distribution of the radiation emissivity over a set of tomographic reconstructions, performed off-line. Then, a regularization procedure is carried out, which merge highly correlated grid elements providing a squared coefficients matrix with an enough low condition number. This matrix, which is inverted offline once for all, can be multiplied by the actual bolometer measures returning the tomographic reconstruction, with calculations suitable for real time application. The proposed algorithm is applied, in this paper, to a synthetic case study.

Semantic MapNet: Building Allocentric Semantic Maps and Representations from Egocentric Views

We study the task of semantic mapping – specifically, an embodied agent (a robot or an egocentric AI assistant) is given a tour of a new environment and asked to build an allocentric top-down semantic map (‘what is where?’) from egocentric observations of an RGB-D camera with known pose (via localization sensors). Importantly, our goal is to build neural episodic memories and spatio-semantic representations of 3D spaces that enable the agent to easily learn subsequent tasks in the same space – navigating to objects seen during the tour (‘Find chair’) or answering questions about the space (‘How many chairs did you see in the house?’). Towards this goal, we present Semantic MapNet (SMNet), which consists of: (1) an Egocentric Visual Encoder that encodes each egocentric RGB-D frame, (2) a Feature Projector that projects egocentric features to appropriate locations on a floor-plan, (3) a Spatial Memory Tensor of size floor-plan length×width×feature-dims that learns to accumulate projected egocentric features, and (4) a Map Decoder that uses the memory tensor to produce semantic top-down maps. SMNet combines the strengths of (known) projective camera geometry and neural representation learning. On the task of semantic mapping in the Matterport3D dataset, SMNet significantly outperforms competitive baselines by 4.01−16.81% (absolute) on mean-IoU and 3.81−19.69% (absolute) on Boundary-F1 metrics. Moreover, we show how to use the spatio-semantic allocentric representations build by SMNet for the task of ObjectNav and Embodied Question Answering. Project page: https://vincentcartillier.github.io/smnet.html.

Direct Solution for Estimating the Location of the Central Projection Camera by Four Control Points

Direct Solution for Estimating the Location of the Central Projection Camera by Four Control Points

Planet–Sun Sensor Revisited

Since the seminal work of Daniele Mortari (“Moon-Sun Attitude Sensor,” Journal of Spacecraft and Rockets, Vol. 34, No. 3, 1997, pp. 360–364), the concept of an attitude sensor using images of illuminated celestial bodies has been pushed forward through the years. The basic idea consists of extracting two independent directions from the image of a celestial body, namely, the camera-to-planet and the planet-to-sun directions. The former is estimated from the center of an ellipse fitted to the imaged limb points and the latter from the symmetry axis of the illuminated region. These assumptions, however, only hold for far-distant spherical targets. In this work, the problem is reformulated in the framework of projective camera transformations of quadrics and conics, and an algorithm estimating the line of sight to the planet and the illumination direction from the limb and terminator ellipses, respectively, is presented. The method is applicable to any ellipsoidlike celestial body having known orientation. The algorithm is first validated on synthetically generated images and then tested using real pictures of Dione and Enceladus satellites gathered from Cassini spacecraft. Results show that the sensor concept returns rms errors in the order of the angular width of a pixel in computing the nadir direction, and subdegree accuracy in computing the sun direction.

Enhancing object detection for autonomous driving by optimizing anchor generation and addressing class imbalance

Object detection has been one of the most active topics in computer vision for the past years. Recent works have mainly focused on pushing the state-of-the-art in the general-purpose COCO benchmark. However, the use of such detection frameworks in specific applications such as autonomous driving is yet an area to be addressed. This study presents an enhanced 2D object detector based on Faster R-CNN that is better suited for the context of autonomous vehicles. Two main aspects are improved: the anchor generation procedure and the performance drop in minority classes. The default uniform anchor configuration is not suitable in this scenario due to the perspective projection of the vehicle cameras. Therefore, we propose a perspective-aware methodology that divides the image into key regions via clustering and uses evolutionary algorithms to optimize the base anchors for each of them. Furthermore, we add a module that enhances the precision of the second-stage header network by including the spatial information of the candidate regions proposed in the first stage. We also explore different re-weighting strategies to address the foreground-foreground class imbalance, showing that the use of a reduced version of focal loss can significantly improve the detection of difficult and underrepresented objects in two-stage detectors. Finally, we design an ensemble model to combine the strengths of the different learning strategies. Our proposal is evaluated with the Waymo Open Dataset, which is the most extensive and diverse up to date. The results demonstrate an average accuracy improvement of 6.13% mAP when using the best single model, and of 9.69% mAP with the ensemble. The proposed modifications over the Faster R-CNN do not increase computational cost and can easily be extended to optimize other anchor-based detection frameworks.

An image matching optimization algorithm based on pixel shift clustering RANSAC

This paper focuses on improving the accuracy of image matching by eliminating the residual mismatches in the matching results of standard RANSAC. Based on pixel shift clustering and RANSAC algorithms, a matching optimization algorithm called pixel shift clustering RANSAC, PSC-RANSAC in short, is proposed in this paper. Firstly, the pixel shift model of space point from two perspectives are established by parallax principle and camera projection model. Then, based on the established pixel shift model, density peaks clustering (DPC) algorithm is used to select the mismatches out to enhance the accuracy of image matching. Meanwhile the comparisons among PSC-RANSAC, standard RANSAC, progressive sample consensus and graph-cut RANSAC show that PSC-RANSAC can more effectively and robustly eliminate the residual mismatches in initial matching results. The proposed method provides an effective tool for optimization on image matching.

Armed with a Camera: Gender, Human Rights, and Visual Documentation in Israel/Palestine

This article offers an explanation for the use of the documenter’s gender as part of human rights politics. I examine the case of the Camera Project of B’Tselem, an Israeli human rights organization working in the Occupied Palestinian Territories. The organization, which distributes video cameras to Palestinian residents, celebrates the participation of women in the documenting project and declares that the women’s videos capture a unique point of view. In the article I question why the gender of a given witness would be emphasized as essential since there is no clear distinction in the visual outcome—that is, in the videos—between those taken by female and male photographers. I analyze visual material from the project and combine this examination with interviews with Palestinian women photographers, all of which reveals that their gender identity is not crucial to the message that the videos try to impart about Israel’s occupation. Drawing on critical feminist scholarship on the gendering of warfare and international law, I argue that preexisting cultural ideas about gender affect the organization’s strategic framing process.

About the Contributors

About the Contributors

Characterisation of a multi-view fringe projection system based on the stereo matching of rectified phase maps

Multi-view fringe projection systems can be effective solutions to address the limitations imposed by the limited field of view, line-of-sight issues and occlusions when measuring the geometry of complex objects, associated with single camera–projector systems. However, characterisation of a multi-view system is challenging since it requires the cameras and projectors to be in a common global coordinate system. We present a method for characterising a multi-view fringe projection system which does not require the characterisation of the projector. The novelty of the method lies in determining the correspondences in the phase domain using the rectified unwrapped phase maps and triangulating the matched phase values to reconstruct the three-dimensional shape of the object. A benefit of the method is that it does not require registration of the point clouds acquired from multiple perspectives. The proposed method is validated by experiment and comparison with a conventional system and a contact coordinate measuring machine.

Robust Reflectance Estimation for Projection-Based Appearance Control in a Dynamic Light Environment.

We present a novel method that robustly estimates the reflectance, even in an environment with dynamically changing light. To control the appearance of an object by using a projector-camera system, an appropriate estimate of the object's reflectance is vital to the creation of an appropriate projection image. Most conventional estimation methods assume static light conditions; however, in practice, the appearance is affected by both the reflectance and environmental light. In an environment with dynamically changing light, conventional reflectance estimation methods require calibration every time the conditions change. In contrast, our method requires no additional calibration because it simultaneously estimates both the reflectance and environmental light. Our method is based on the concept of creating two different light conditions by switching the projection at a rate higher than that perceived by the human eye and captures the images of a target object separately under each condition. The reflectance and environmental light are then simultaneously estimated by using the pair of images acquired under these two conditions. We implemented a projector-camera system that switches the projection on and off at 120 Hz. Experiments confirm the robustness of our method when changing the environmental light. Further, our method can robustly estimate the reflectance under practical indoor lighting conditions.

Parallel Single-Pixel Imaging: A General Method for Direct\u2013Global Separation and 3D Shape Reconstruction Under Strong Global Illumination

We present parallel single-pixel imaging (PSI), a photography technique that captures light transport coefficients and enables the separation of direct and global illumination, to achieve 3D shape reconstruction under strong global illumination. PSI is achieved by extending single-pixel imaging (SI) to modern digital cameras. Each pixel on an imaging sensor is considered an independent unit that can obtain an image using the SI technique. The obtained images characterize the light transport behavior between pixels on the projector and the camera. However, the required number of SI illumination patterns generally becomes unacceptably large in practical situations. We introduce local region extension (LRE) method to accelerate the data acquisition of PSI. LRE perceives that the visible region of each camera pixel accounts for a local region. Thus, the number of detected unknowns is determined by local region area, which is extremely beneficial in terms of data acquisition efficiency. PSI possesses several properties and advantages. For instance, PSI captures the complete light transport coefficients between the projector–camera pair, without making specific assumptions on measured objects and without requiring special hardware and restrictions on the arrangement of the projector–camera pair. The perfect reconstruction property of LRE can be proven mathematically. The acquisition and reconstruction stages are straightforward and easy to implement in the existing projector–camera systems. These properties and advantages make PSI a general and sound theoretical model to decompose direct and global illuminations and perform 3D shape reconstruction under global illumination.

Star Identification and Attitude Determination With Projective Cameras

Images of starfields collected by a projective camera are useful for a variety of scientific and engineering purposes. This utility is exemplified by star trackers, which are amongst the most commonly used sensors for determining the attitude of modern spacecraft. While the literature on star identification and star-based attitude determination is extensive, most algorithms are developed in an ad hoc manner. This work provides a comprehensive and systematic framework for invariant-based star identification and shows most past star identification algorithms to be special cases within this framework. The new star identification framework is found to motivate new problems in attitude determination and sensor self-calibration. Specifically, new algorithms are presented for simultaneous attitude determination and camera calibration for a generic wide field-of-view sensor using a single starfield image. In the special case where camera focal length is the only unknown calibration parameter, attitude determination performance of the new algorithm is indiscernible from a perfectly calibrated camera.

Research on Quantitative Calculation Method of Lane-Line Region of Interest and Structural Constraints in Constant Speed Cruise Mode

Appropriate region of interest (ROI) and lane structure restriction conditions can improve the accuracy of lane detection. Most previously reported approaches require human experience to set up the constraint conditions of the ROI and lane structure. Few of these studies involved quantitative calculation methods of the constraint conditions of the ROI and lane structure. To solve this problem, constant-speed cruise mode, which relies on lane recognition, was examined. A vehicle dynamics model with a charge-coupled device (CCD) camera was constructed based on the vehicle dynamics and camera projection. The projection of lane line feature points from the world coordinate system to the image coordinate system under different vehicle positions and postures was analyzed. A quantitative calculation method of the ROI lateral constraint equation and the lane structure constraint equation in constant-speed cruise mode is given. Actual road test data showed that the constraint condition calculation method not only agreed with the actual road test results but also improved the real-time accuracy.

Fixed Viewpoint Mirror Surface Reconstruction Under an Uncalibrated Camera.

This paper addresses the problem of mirror surface reconstruction, and proposes a solution based on observing the reflections of a moving reference plane on the mirror surface. Unlike previous approaches which require tedious calibration, our method can recover the camera intrinsics, the poses of the reference plane, as well as the mirror surface from the observed reflections of the reference plane under at least three unknown distinct poses. We first show that the 3D poses of the reference plane can be estimated from the reflection correspondences established between the images and the reference plane. We then form a bunch of 3D lines from the reflection correspondences, and derive an analytical solution to recover the line projection matrix. We transform the line projection matrix to its equivalent camera projection matrix, and propose a cross-ratio based formulation to optimize the camera projection matrix by minimizing reprojection errors. The mirror surface is then reconstructed based on the optimized cross-ratio constraint. Experimental results on both synthetic and real data are presented, which demonstrate the feasibility and accuracy of our method.

Retracted] Design of 3D Modeling Face Image Library in Multimedia Film and Television

The development of 3D modeling technology has promoted the development of the multimedia film and television industry. This article is aimed at studying the design of 3D modeling facial image library in multimedia film and television, at providing a more comprehensive facial image library for the multimedia film and television industry, at breaking the shackles of the traditional film and television industry with 3D technology, and at continuously surpassing traditional film and television media forms. This article deeply explores the background development of multimedia film and television and the characteristics of the development of new media. Starting from 3D technology, it extracts facial features of characters, transforms image data through deep autoencoders, and uses local binarization mode to perform the original facial image is subjected to texture feature extraction. In this paper, a number of experimental subjects were selected, and the subjects were photographed from the left, front, and right from multiple angles. Through the pinhole camera projection imaging process, the internal and external parameters of the camera were adjusted. In the process of 3D image construction, the image is first selected for feature detection, then the corresponding vector information and geometric conditions are matched to construct a 3D matrix, and the facial structure image is obtained by triangulation. This article compares the 3D production software on the market and selects the Maya platform suitable for building this system. The global constraint information is obtained by training some sample images. When searching the test image, find the appropriate feature point position according to the structural matching degree of the local image. When each search is completed, the global information will be used for constraint, so as to output reasonable feature information. The average residual range of the human face image constructed in this paper is 0.25‐0.45, and the maximum residual error does not exceed 4.0. The experimental method in this paper has good stability and robustness. Using the COM transmission model can make experimenters not need to think too much about the underlying details. This face animation‐driven simulation scheme can achieve more vivid facial expressions.