Synthetic Camera Research Articles

Using synthetic camera poses for camera calibration in soccer videos

Using synthetic camera poses for camera calibration in soccer videos

Detecting change in graffiti using a hybrid framework

AbstractGraffiti, by their very nature, are ephemeral, sometimes even vanishing before creators finish them. This transience is part of graffiti's allure yet signifies the continuous loss of this often disputed form of cultural heritage. To counteract this, graffiti documentation efforts have steadily increased over the past decade. One of the primary challenges in any documentation endeavour is identifying and recording new creations. Image‐based change detection can greatly help in this process, effectuating more comprehensive documentation, less biased digital safeguarding and improved understanding of graffiti. This paper introduces a novel and largely automated image‐based graffiti change detection method. The methodology uses an incremental structure‐from‐motion approach and synthetic cameras to generate co‐registered graffiti images from different areas. These synthetic images are fed into a hybrid change detection pipeline combining a new pixel‐based change detection method with a feature‐based one. The approach was tested on a large and publicly available reference dataset captured along the Donaukanal (Eng. Danube Canal), one of Vienna's graffiti hotspots. With a precision of 87% and a recall of 77%, the results reveal that the proposed change detection workflow can indicate newly added graffiti in a monitored graffiti‐scape, thus supporting a more comprehensive graffiti documentation.

Synthetic data generation for the continuous development and testing of autonomous construction machinery

Abstract The development and testing of autonomous systems require sufficient meaningful data. However, generating suitable scenario data is a challenging task. In particular, it raises the question of how to narrow down what kind of data should be considered meaningful. Autonomous systems are characterized by their ability to cope with uncertain situations, i.e. complex and unknown environmental conditions. Due to this openness, the definition of training and test scenarios cannot be easily specified. Not all relevant influences can be sufficiently specified with requirements in advance, especially for unknown scenarios and corner cases, and therefore the “right” data, balancing quality and efficiency, is hard to generate. This article discusses the challenges of automated generation of 3D scenario data. We present a training and testing loop that provides a way to generate synthetic camera and Lidar data using 3D simulated environments. Those can be automatically varied and modified to support a closed-loop system for deriving and generating datasets that can be used for continuous development and testing of autonomous systems.

Feasibility of the EDICAM camera for runaway electron detection in JT-60SA disruptions

The visible camera system EDICAM (Event Detection Intelligent Camera), recently installed on JT-60SA, is simulated to assess whether it can be used for measuring synchrotron radiation from relativistic runaway electrons. In this simulation, the SOFT synthetic synchrotron diagnostic framework is used to compute the synthetic synchrotron camera images from a JT-60SA-like disruption modelled with the DREAM disruption simulation code. In the studied scenario, a large amount of argon is added to the plasma, and a disruption is simulated by starting a prescribed exponential temperature drop and finishing with further cooling provided by the argon in a self-consistent simulation of the current quench. The background plasma evolution is calculated by DREAM self-consistently with the fast electron population, which is modelled kinetically. The resulting runaway electron distribution function along with the parameters of the EDICAM visible camera system are used as an input to the SOFT framework to assess the feasibility of the camera for runaway electron detection. We find that the runaway electron beam formed in the disruption can produce synchrotron radiation observable by the EDICAM system, thus enabling the use of the EDICAM for the characterization of runaway electron beams.

Angular sampling of a monochromatic, wide-field-of-view camera to augment next-generation Earth radiation budget satellite observations

Abstract. Earth radiation budget (ERB) satellite observations require conversion of the measured radiance, which is a remotely sensed quantity, to a derived irradiance, which is the relevant energy balance quantity routinely used in modeling and analysis of the climate system. The state-of-the-art approach for radiance-to-irradiance conversion taken by the Clouds and the Earth's Radiant Energy System (CERES) benefits from the exhaustive sampling of radiance anisotropy by multiple CERES instruments across many years. Unfortunately, the CERES approach is not easily extended to new ERB spectral channels that lack previous sampling, such as the “split-shortwave” planned to be part of the next-generation ERB mission Libera. As an alternative approach, the capability of a monochromatic, wide-field-of-view camera to provide dense angular sampling in a much shorter time frame is assessed. We present a general concept for how this can be achieved and quantify the proficiency of a camera to provide rapid angular distribution model (ADM) generation for the new Libera ultraviolet and visible (VIS) sub-band. A single mid-visible camera wavelength (555 nm) is shown to be ideal for representing the VIS sub-band, requiring only basic scene stratification for 555 nm to VIS conversion. Synthetic camera sampling with realistic operating constraints also demonstrates that the angular radiance field of various scenes can be well populated within a single day of sampling, a notable advance over existing approaches. These results provide a path for generating observationally based VIS ADMs with minimal lag time following Libera's launch. Coupled with efforts to utilize a camera for scene identification, this may also pave the way for future ERB satellite systems to develop stand-alone irradiance products for arbitrary sets of spectral channels, opening up new measurement and science possibilities.

ALIVE: Adaptive-Chromaticity for Interactive Low-light Image and Video Enhancement

Ray tracing remains of interest to Computer Graphics community with its elegant framing of how light interacts with objects, being able to easily support multiple light sources, and simple framework of merging synthetic and real cameras. Recent trends to provide implementations at the chip-level means raytracing’s constant quest of realism would propel its usage in real-time applications. AR/VR, Animations, 3DGames Industry, 3D-large scale simulations, and future social computing platforms are just a few examples of possible major impact. Raytracing is also appealing to HCI community because raytracing extends well along the 3D-space and time, seamlessly blending both synthetic and real cameras at multiple scales to support storytelling. This presentation will include a few milestones from my work such as the Slicing Extent technique and Directed Safe Zones. Our recent applications of applying machine learning techniques creating novel synthetic views, which could also provide a future doorway to handle dynamic scenes with more compute power as needed, will also be presented. It is once again renaissance for ray tracing which for last 50+ years has remained the most elegant technique for modeling light phenomena in virtual worlds at whatever scale compute power could support.

Fusion of Inverse Synthetic Aperture Radar and Camera Images for Automotive Target Tracking

Automotive targets undergoing turns in road junctions offer large synthetic apertures over short dwell times to automotive radars that can be exploited for obtaining fine cross-range resolution. Likewise, the wide bandwidths of the automotive radar signal yield high-range resolution profiles. Together, they are exploited for generating inverse synthetic aperture radar (ISAR) images that offer rich information regarding the target vehicle's size, shape, and trajectory which is useful for object recognition and classification. However, a key requirement for ISAR is translation motion compensation and estimation of the turning velocity of the target. State-of-the-art algorithms for motion compensation trade-off between computational complexity and accuracy. An alternative low complexity method is to use an additional sensor for tracking the target motion. In this work, we propose to exploit computer vision algorithms to identify the radar target object in the sensor field-of-view (FoV) with high accuracy. Further, we propose to track the target vehicle's motion through fusion of vision and radar data. Vision data facilitates the accurate estimation of the lateral position of the target and its lateral velocity which complements the radar's capability of accurate estimation of range and radial velocity. Through simulations and experimental evaluations with a monocular camera and Texas Instrument's millimeter wave radar, we demonstrate the effectiveness of sensor fusion for accurate target tracking for translational motion compensation and generation of high quality ISAR images.

Sparse representation-based demosaicking method for joint chromatic and polarimetric imagery

Division of focal plane (DoFP) polarization camera has become mainstream among various polarimetric imaging systems whose sensor is equipped with Bayer filter and directional polarization filter, so joint chromatic and polarimetric image demosaicking is essential. However, since 15 or 14 pixels are missed out of 16 pixels in color polarization mosaic images, it’s challenging to adopt a method to reconstruct full-resolution images well. In this paper, we propose a demosaicking model based on sparse coding. The model introduces RGB-polarization channels correlation, adaptive sub-dictionaries and non-local self-similarity restrictions, and combines them into an optimal problem to solve. The experimental results, including reconstructing full-resolution synthetic images and camera captured real scenes, demonstrate that our proposed method outperforms the current state-of-the-art method in terms of quantitative measures and visual quality.

Forward modelling of [formula omitted] camera view in ST40 informed by experimental data

Embedding diagnostics in future pilot plants will be a challenging task, because of space- and irradiation-related concerns. Relying on high-fidelity synthetic diagnostics would then be valuable.The 3D Monte-Carlo ray-tracing code CHERAB allows the development of numerous synthetic spectroscopic diagnostics. Focus of the present work is the introduction of new CHERAB models. The forward modelling of a synthetic Dα camera in ST40, the privately funded, high-field spherical tokamak, owned and operated by Tokamak Energy Ltd, and the comparison against experimental data is chosen as a testbed for quality assessment. Main output of the study then consists of estimates of the neutral particle densities throughout the chamber, of crucial relevance within edge plasma studies.Starting from simple analytical models, a 2D Dα source in the poloidal plane is generated. However, the centre column limited plasmas in ST40 display an intrinsically-3D Dα emission, mostly localised around the discrete poloidal limiters on the centre column, not captured by any axisymmetric source model. Hence, a novel methodology is introduced in CHERAB to approximate the 3D non-toroidally-symmetric pattern via a piece-wise emission distribution.Irrespective of the geometry of the emission and size of the tokamak, the pronounced non-homogeneity in the edge plasma emission requires sub-millimetric (∼ power fall-off length) spatial resolution to guarantee an accurate estimate of the peak emission. Minimising the associated burden via implementation of a non-uniform source sampling algorithm, which is a modification of the standard CHERAB uniform sampling, results in a >10-fold reduction of the computational cost.The significantly-shortened simulation time also makes the inclusion of more sophisticated models affordable. Of potential appeal in view of highly-detached divertors, the approximation of optically thin plasma is dropped, and photon-plasma interactions are accounted for. Brand-new CHERAB models able to take into account phenomena of photon absorption and scattering are so introduced.

Parallax Inference for Robust Temporal Monocular Depth Estimation in Unstructured Environments.

Estimating the distance to objects is crucial for autonomous vehicles, but cost, weight or power constraints sometimes prevent the use of dedicated depth sensors. In this case, the distance has to be estimated from on-board mounted RGB cameras, which is a complex task especially for environments such as natural outdoor landscapes. In this paper, we present a new depth estimation method suitable for use in such landscapes. First, we establish a bijective relationship between depth and the visual parallax of two consecutive frames and show how to exploit it to perform motion-invariant pixel-wise depth estimation. Then, we detail our architecture which is based on a pyramidal convolutional neural network where each level refines an input parallax map estimate by using two customized cost volumes. We use these cost volumes to leverage the visual spatio-temporal constraints imposed by motion and make the network robust for varied scenes. We benchmarked our approach both in test and generalization modes on public datasets featuring synthetic camera trajectories recorded in a wide variety of outdoor scenes. Results show that our network outperforms the state of the art on these datasets, while also performing well on a standard depth estimation benchmark.

Anisotropic diffusion as a proxy model for the estimation of heat-loads on plasma-facing components

To facilitate the estimation of heat loads on plasma-facing components in fusion devices in various different magnetic geometries, a heat load proxy model was developed based on anisotropic diffusion. In this work, this model is compared to the so-called field-line diffusion approach. To facilitate the evaluation of these models, a novel synthetic camera-based approach for obtaining heat load distributions from Monte Carlo samples was also developed and implemented. With the assistance of this synthetic camera, heat load predictions for the Wendelstein 7-X divertor were obtained and compared with infrared camera observations. It was found that the anisotropic diffusion-based model achieved a closer match to infrared camera observations, while still being suitable in computational effort for large magnetic configuration database scans.

Challenges of Labelling Unknown Seabed Munition Dumpsites from Acoustic and Optical Surveys: A Case Study at Skagerrak

The disposal of unexploded ordnance (UXOs) at sea is a global problem. The mapping and remediation of historic UXOs can be assisted by autonomous underwater vehicles (AUVs) carrying sensor payloads such as synthetic aperture sonar (SAS) and optical cameras. AUVs can image large areas of the seafloor in high resolution, motivating an automated approach to UXO detection. Modern methods commonly use supervised machine learning which requires labelled examples from which to learn. This work investigates the often-overlooked labelling process and resulting dataset using an example historic UXO dumpsite at Skagerrak. A counterintuitive finding of this work is that optical images cannot be relied on for ground truth as a significant number of UXOs visible in SAS images are not in optical images, presumed buried. Given the lack of ground truth, we use an ordinal labelling scheme to incorporate a measure of labeller uncertainty. We validate this labelling regime by quantifying label accuracy compared to optical labels with high confidence. Using this approach, we explore different taxonomies and conclude that grouping objects into shells, bombs, debris, and natural gave the best trade-off between accuracy and discrimination.

Deep homography estimation in dynamic surgical scenes for laparoscopic camera motion extraction

ABSTRACT Current laparoscopic camera motion automation relies on rule-based approaches or only focuses on surgical tools. Imitation Learning (IL) methods could alleviate these shortcomings, but have so far been applied to oversimplified setups. Instead of extracting actions from oversimplified setups, in this work we introduce a method that allows to extract a laparoscope holder’s actions from videos of laparoscopic interventions. We synthetically add camera motion to a newly acquired dataset of camera motion free da Vinci surgery image sequences through a novel homography generation algorithm. The synthetic camera motion serves as a supervisory signal for camera motion estimation that is invariant to object and tool motion. We perform an extensive evaluation of state-of-the-art (SOTA) Deep Neural Networks (DNNs) across multiple compute regimes, finding our method transfers from our camera motion free da Vinci surgery dataset to videos of laparoscopic interventions, outperforming classical homography estimation approaches in both, precision by , and runtime on a CPU by .

Design Study of an Ultrahigh Resolution Brain SPECT System Using a Synthetic Compound-Eye Camera Design With Micro-Slit and Micro-Ring Apertures.

In this paper, we discuss the design study for a brain SPECT imaging system, referred to as the HelmetSPECT system, based on a spherical synthetic compound-eye (SCE) gamma camera design. The design utilizes a large number ( ~500) of semiconductor detector modules, each coupled to an aperture with a very narrow opening for high-resolution SPECT imaging applications. In this study, we demonstrate that this novel system design could provide an excellent spatial resolution, a very high sensitivity, and a rich angular sampling without scanning motion over a clinically relevant field-of-view (FOV). These properties make the proposed HelmetSPECT system attractive for dynamic imaging of epileptic patients during seizures. In ictal SPECT, there is typically no prior information on where the seizures would happen, and both the imaging resolution and quantitative accuracy of the dynamic SPECT images would provide critical information for staging the seizures outbreak and refining the plans for subsequent surgical intervention.We report the performance evaluation and comparison among similar system geometries using non-conventional apertures, such as micro-ring and micro-slit, and traditional lofthole apertures. We demonstrate that the combination of ultrahigh-resolution imaging detectors, the SCE gamma camera design, and the micro-ring and micro-slit apertures would offer an interesting approach for the future ultrahigh-resolution clinical SPECT imaging systems without sacrificing system sensitivity and FOV.

Pix2pix Conditional Generative Adversarial Networks for Scheimpflug Camera Color-Coded Corneal Tomography Image Generation.

PurposeTo assess the ability of pix2pix conditional generative adversarial network (pix2pix cGAN) to create plausible synthesized Scheimpflug camera color-coded corneal tomography images based upon a modest-sized original dataset to be used for image augmentation during training a deep convolutional neural network (DCNN) for classification of keratoconus and normal corneal images.MethodsOriginal images of 1778 eyes of 923 nonconsecutive patients with or without keratoconus were retrospectively analyzed. Images were labeled and preprocessed for use in training the proposed pix2pix cGAN. The best quality synthesized images were selected based on the Fréchet inception distance score, and their quality was studied by calculating the mean square error, structural similarity index, and the peak signal-to-noise ratio. We used original, traditionally augmented original and synthesized images to train a DCNN for image classification and compared classification performance metrics.ResultsThe pix2pix cGAN synthesized images showed plausible subjectively and objectively assessed quality. Training the DCNN with a combination of real and synthesized images allowed better classification performance compared with training using original images only or with traditional augmentation.ConclusionsUsing the pix2pix cGAN to synthesize corneal tomography images can overcome issues related to small datasets and class imbalance when training computer-aided diagnostic models.Translational RelevancePix2pix cGAN can provide an unlimited supply of plausible synthetic Scheimpflug camera color-coded corneal tomography images at levels useful for experimental and clinical applications.

Role of JETPEAK database in validation of synthetic neutron camera diagnostics and ASCOT- AFSI fast particle and fusion product calculation chain in JET

JETPEAK is a comprehensive stationary-state database and a work environment for JET diagnostics data, processed data and simulations suitable for a wide variety of analytical tasks. It enables systematic modelling and efficient interpretation of synthetic neutron diagnostics and validation of fast particle and fusion product calculations. We present a processing chain connecting the ASCOT orbit following code and AFSI fusion source simulator Monte-Carlo codes, providing synthetic diagnostics data for systematic comparison with measured data and model validation. The ASCOT code defines the reactant distribution functions and the additional AFSI module creates fusion products as 4D (R, z, E, pitch) distributions, which serve as input for synthetic diagnostics calculations. In this contribution we present synthetic neutron emission rates for the JET neutron camera lines of sight. First results show a fair agreement of the synthetic camera results with the horizontal JET neutron cameras and systematic differences, which are not understood, between the ratios of neutron emission measurements and modelled for the vertical and horizontal cameras.

Filament identification in wide-angle high speed imaging of the mega amp spherical tokamak.

A new tomographic inversion technique is presented for the identification of plasma filaments in wide-angle visible camera data. The technique works on the assumption that background subtracted images of filaments can be represented as a superposition of uniformly emitting magnetic equilibrium field lines. A large collection of equilibrium magnetic field lines is traced and projected onto the camera field of view and combined to form a geometry matrix describing the coordinate transformation from magnetic field aligned coordinates to image pixel coordinates. Inverting this matrix enables the reprojection of the emission in the camera images onto a field aligned basis, from which filaments are readily identifiable. The inversion is a poorly conditioned problem which is overcome using a least-squares approach with Laplacian regularization. Blobs are identified using the "watershed" algorithm and 2D Gaussians are fitted to get the positions, widths, and amplitudes of the filaments. A synthetic camera diagnostic generating images containing experimentally representative filaments is utilized to rigorously benchmark the accuracy and reliability of the technique. 74% of synthetic filaments above the detection amplitude threshold are successfully detected, with 98.8% of detected filaments being true positives. The accuracy with which filament properties and their probability density functions are recovered is discussed, along with sources of error and methods to minimize them.

Validating fast-ion wall-load IR analysis-methods against W7-X NBI empty-torus experiment

The first neutral beam injection (NBI) experiments in Wendelstein 7-X (W7-X) stellarator were conducted in the summer of 2018. The NBI system is used to heat the magnetically confined plasma by neutralising an accelerated hydrogen ion beam and directing it into the plasma, where the resulting energetic ions release their energy to heat the plasma. The modelling of the NBI fast ion experiments has commenced, including estimation of the shine-through and the orbit-losses. The stellarator has a wide-angle infra red (IR) imaging system to monitor the machine plasma facing component surface temperatures. This work validates the NBI model “Beamlet Based NBI (BBNBI)” and the newly written synthetic IR camera model. The validation is accomplished by comparing the measured and the synthetic IR camera measurements of an experiment where the NBI was injected into the vacuum vessel without a plasma. A good qualitative and quantitative match was found. This agreement is further supported by spectroscopic and calibration measurements of the NBI and and IR camera systems.

Plumetrack: Flux calculation software for UV cameras

Ultraviolet (UV) cameras are increasingly employed to map and measure SO2 abundances in volcanic emissions to the atmosphere. The main purpose of this is to estimate mass fluxes of SO2, which requires estimation of the transport velocity of the plume. In this paper, we present Plumetrack, open-source Python based software for computing SO2 fluxes from calibrated UV camera images. Designed to be the final component in UV camera processing toolchains, Plumetrack provides functionality for velocity estimation using optical-flow, flux calculation and error estimates. It can be used interactively via a graphical user interface or for batch processing via a commandline interface. We discuss the features and implementation details of Plumetrack, describe in detail a new flux calculation algorithm and demonstrate its performance on a set of synthetic UV camera images. The new algorithm is found to out perform the established flux calculation method, especially for highly spatiotemporally variable plumes. Furthermore, we show that the Plumetrack software may be successfully used with data from other imaging systems such as standard video cameras.

Reconstruction of recycling flux from synthetic camera images, evaluated for the Wendelstein 7-X startup limiter

The interpretation of spectroscopic measurements in the edge region of high-temperature plasmas can be guided by modeling with the EMC3-EIRENE code. A versatile synthetic diagnostic module, initially developed for the generation of synthetic camera images, has been extended for the evaluation of the inverse problem in which the observable photon flux is related back to the originating particle flux (recycling). An application of this synthetic diagnostic to the startup phase (inboard) limiter in Wendelstein 7-X (W7-X) is presented, and reconstruction of recycling from synthetic observation of emission is evaluated. It is shown that recycling—in terms of re-ionization within the view cone of the camera—is underestimated by a factor of about 3 at the peak location and overestimated in between, and that emission related to molecular dissociation can locally yield an additional factor of 2 for the ionization to photon conversion. Fitted conversion factors are extracted from the simulation results which include a dependence on the plasma density.