Synthetic Scenes Research Articles

Synthetic Scene Character Generator and Ensemble Scheme with the Random Image Feature Method for Japanese and Chinese Scene Character Recognition

Synthetic Scene Character Generator and Ensemble Scheme with the Random Image Feature Method for Japanese and Chinese Scene Character Recognition

Time-of-flight camera characterization with functional modeling for synthetic scene generation.

In this manuscript, we design, describe, and present a functional model of Time-of-Flight (ToF) cameras. The model can be used to generate randomized scenes that incorporate depth scenarios with various objects at various depths with varied orientations and illumination intensity. In addition to the potential to generate any random depth scenario, the camera, pixels, and binning are modelled incorporating radial distortion based on camera intrinsic and extrinsic. The model also includes ToF artifacts such as Signal Noise, Crosstalk and Multipath. We measured experimentally the Noise in Time-of-Flight. We experimentally fitted, and simulated with state-of-the art Simulator the Crosstalk effect, and characterized multipath according with the existing literature. Our work can be used to generate as many images as needed for neural network (NN) training and testing. The proposed approach can also be used to benchmark and evaluate both End-to-End ToF algorithms as well as specialized algorithms for denoising, unwrapping, crosstalk, and multipath correction.

Multi-Scale Fusion Networks with RGB Image Features for Depth Map Completion

<p indent="0mm">Currently, researchers cannot directly train end-to-end model for depth image completion because of lacking paired “incomplete-complete” RGB-D datasets. To address this problem, a random mask-based method which is combined with “real-synthetic” data for joint training strategy is proposed to construct paired incomplete-complete RGB-D data. This method generates depth maps with different missing ratios based on random masks, and uses synthetic scene datasets to construct missing regions of depth map with high fidelity truth values. Based on this strategy, a multi-scale depth map completion network is constructed, which fuses the corresponding RGB image features. The proposed network extracts RGB image features and depth map features with different scales from RGB image branches and depth map branches. Then, in the feature fusion branch, the RGB image features and depth map features are fused at different scales, which makes that the rich semantic features of RGB images and the information of depth maps can be effectively integrated for depth map completion. Experiments on the NYU-Depth V2 dataset show that in depth completion tasks with different missing ratios, the average threshold accuracy of this method is 0.98, and the mean relative error is about 0.061. Compared with the existing methods based on neural networks and optimizing sparse equations, this method improves the threshold accuracy by an average of 0.02, and the mean relative error decreases by an average of 0.027.

Multi-Scale Fusion Networks with RGB Image Features for Depth Map Completion

<p indent=0mm>Currently, researchers cannot directly train end-to-end model for depth image completion because of lacking paired “incomplete-complete” RGB-D datasets. To address this problem, a random mask-based method which is combined with “real-synthetic” data for joint training strategy is proposed to construct paired incomplete-complete RGB-D data. This method generates depth maps with different missing ratios based on random masks, and uses synthetic scene datasets to construct missing regions of depth map with high fidelity truth values. Based on this strategy, a multi-scale depth map completion network is constructed, which fuses the corresponding RGB image features. The proposed network extracts RGB image features and depth map features with different scales from RGB image branches and depth map branches. Then, in the feature fusion branch, the RGB image features and depth map features are fused at different scales, which makes that the rich semantic features of RGB images and the information of depth maps can be effectively integrated for depth map completion. Experiments on the NYU-Depth V2 dataset show that in depth completion tasks with different missing ratios, the average threshold accuracy of this method is 0.98, and the mean relative error is about 0.061. Compared with the existing methods based on neural networks and optimizing sparse equations, this method improves the threshold accuracy by an average of 0.02, and the mean relative error decreases by an average of 0.027.

Multi-Spectral Image Synthesis for Crop/Weed Segmentation in Precision Farming

An effective perception system is a fundamental component for farming robots, as it enables them to properly perceive the surrounding environment and to carry out targeted operations. The most recent methods make use of state-of-the-art machine learning techniques to learn a valid model for the target task. However, those techniques need a large amount of labeled data for training. A recent approach to deal with this issue is data augmentation through Generative Adversarial Networks (GANs), where entire synthetic scenes are added to the training data, thus enlarging and diversifying their informative content. In this work, we propose an alternative solution with respect to the common data augmentation methods, applying it to the fundamental problem of crop/weed segmentation in precision farming. Starting from real images, we create semi-artificial samples by replacing the most relevant object classes (i.e., crop and weeds) with their synthesized counterparts. To do that, we employ a conditional GAN (cGAN), where the generative model is trained by conditioning the shape of the generated object. Moreover, in addition to RGB data, we take into account also near-infrared (NIR) information, generating four channel multi-spectral synthetic images. Quantitative experiments, carried out on three publicly available datasets, show that (i) our model is capable of generating realistic multi-spectral images of plants and (ii) the usage of such synthetic images in the training process improves the segmentation performance of state-of-the-art semantic segmentation convolutional networks.

Towards synthesized training data for semantic segmentation of mobile laser scanning point clouds: Generating level crossings from real and synthetic point cloud samples

This paper presents a method for synthesizing mobile laser scanning point clouds of railroad level crossings that can be used to train neural networks for point cloud segmentation. The method arranges point cloud samples representing individual objects into new scenes using a set of simple placement rules. The point cloud samples can be cropped from real point clouds, created from 3D mesh models, or procedurally generated using mathematical functions. The scenes can consist of one or more types of samples, making it possible to combine real and synthetic data. The findings show that a network trained on scenes generated from real point cloud samples resulted in a better overall F1-score compared to a network that was trained using real scenes. Also, the performance of a network trained on a very small amount of real scenes can be improved by adding fully synthetic scenes to the training data.

SMGAN: A self-modulated generative adversarial network for single image dehazing

Single image dehazing has become a key prerequisite for most high-level computer vision tasks since haze severely degrades the input images. Traditional prior-based methods dehaze images by some assumptions concluded from haze-free images, which recover high-quality results but always cause some halos or color distortion. Recently, many methods have been using convolutional neural networks to learn the haze-relevant features and then retrieve the original images. These learning-based methods achieve better performance in synthetic scenes but can hardly restore a clear image with discriminative texture when applied to real-world images, mainly because these networks are trained on synthetic datasets. To solve these problems, a self-modulated generative adversarial network for single image dehazing named SMGAN is proposed. The SMGAN inputs prior-dehazed images into a parameter-shared encoder to produce some latent information of these dehazed images. During the hazy image decoding process, the latent information is sent to self-modulated batch normalization layers, which makes the network fit in real haze removal. Moreover, consider that there are some over-enhanced regions in the guidance images, and a refine module is proposed to alleviate the negative information. The proposed SMGAN combines the advantages of prior-based methods and learning-based methods, which provides superior performance compared with the state-of-the-art methods on both synthetic and real-word datasets.

LenSelect: Object Selection in Virtual Environments by Dynamic Object Scaling

AbstractWe present a novel selection technique for VR called LenSelect. The main idea is to decrease the Index of Difficulty (ID) according to Fitts’ Law by dynamically increasing the size of the potentially selectable objects. This facilitates the selection process especially in cases of small, distant or partly occluded objects, but also for moving targets. In order to evaluate our method, we have defined a set of test scenarios that covers a broad range of use cases, in contrast to often used simpler scenes. Our test scenarios include practically relevant scenarios with realistic objects but also synthetic scenes, all of which are available for download. We have evaluated our method in a user study and compared the results to two state-of-the-art selection techniques and the standard ray-based selection. Our results show that LenSelect performs similar to the fastest method, which is ray-based selection, while significantly reducing the error rate by 44%.

Polyphonic training set synthesis improves self-supervised urban sound classification.

Machine listening systems for environmental acoustic monitoring face a shortage of expert annotations to be used as training data. To circumvent this issue, the emerging paradigm of self-supervised learning proposes to pre-train audio classifiers on a task whose ground truth is trivially available. Alternatively, training set synthesis consists in annotating a small corpus of acoustic events of interest, which are then automatically mixed at random to form a larger corpus of polyphonic scenes. Prior studies have considered these two paradigms in isolation but rarely ever in conjunction. Furthermore, the impact of data curation in training set synthesis remains unclear. To fill this gap in research, this article proposes a two-stage approach. In the self-supervised stage, we formulate a pretext task (Audio2Vec skip-gram inpainting) on unlabeled spectrograms from an acoustic sensor network. Then, in the supervised stage, we formulate a downstream task of multilabel urban sound classification on synthetic scenes. We find that training set synthesis benefits overall performance more than self-supervised learning. Interestingly, the geographical origin of the acoustic events in training set synthesis appears to have a decisive impact.

Contrastive Self-supervised Representation Learning Using Synthetic Data

Learning discriminative representations with deep neural networks often relies on massive labeled data, which is expensive and difficult to obtain in many real scenarios. As an alternative, self-supervised learning that leverages input itself as supervision is strongly preferred for its soaring performance on visual representation learning. This paper introduces a contrastive self-supervised framework for learning generalizable representations on the synthetic data that can be obtained easily with complete controllability. Specifically, we propose to optimize a contrastive learning task and a physical property prediction task simultaneously. Given the synthetic scene, the first task aims to maximize agreement between a pair of synthetic images generated by our proposed view sampling module, while the second task aims to predict three physical property maps, i.e., depth, instance contour maps, and surface normal maps. In addition, a feature-level domain adaptation technique with adversarial training is applied to reduce the domain difference between the realistic and the synthetic data. Experiments demonstrate that our proposed method achieves state-of-the-art performance on several visual recognition datasets.

SREC‐RT: A Structure for Ray Tracing Rounded Edges and Corners

AbstractMan‐made objects commonly exhibit rounded edges and corners generated through their manufacturing processes. The variation of surface normals at these confined locations produces shading details that are visually essential to the realism of synthetic scenes. The more specular the surface, the finer and more prominent its highlights. However, most geometric modellers represent rounded edges and corners with dense polygonal meshes that are limited in terms of smoothness, while tremendously increasing scene complexity. This paper proposes a non‐invasive method (i.e. that does not modify the original geometry) for the modelling and rendering of smooth edges and corners from any input polygonal geometry defined with infinitely sharp edges. At the heart of our contribution is a geometric structure that automatically and accurately defines the geometry of edge and corner rounded areas, as well as the topological relationships at edges and vertices. This structure, called SREC‐RT, is integrated in a ray‐tracing‐based acceleration structure in order to determine the region of interest of each rounded edge and corner. It allows systematic rounding of all edges and vertices without increasing the 3D scene geometric complexity. While the underlying rounded geometry can be of any type, we propose a practical ray‐edge and ray‐corner intersection based on parametric surfaces. We analyse comparisons generated with existing methods. Our results present the advantages of our method, including extreme close‐up views of surfaces with a much higher quality for very little additional memory, and reasonable computation time overhead.

Gaussian-Wiener Representation and Hierarchical Coding Scheme for Focal Stack Images

Focal stack images (FoSIs) are a set of 2D images that captured one scene with serial focal depths. The redundancies of FoSIs mainly come from gradual focused depths changes rather than motion of objects. Conventional coding schemes cannot fully exploit such redundancies, leading to coding inefficiency. In this paper, we propose a new Gaussian-Wiener representation to model the gradual focused depths changes among FoSIs. In the representation, image degradation-restoration relations are utilized to describe the focus-defocus changing characteristics of FoSIs. Based on this representation, we propose a new hierarchical coding scheme for fully exploiting the inter-frame redundancies of FoSIs. In the scheme, a Gaussian-Wiener representation based inter prediction (GWR-IP) is presented by embedding Gaussian convolution and Wiener deconvolution into normal video encoder. Block-wise focus-defocus changing of FoSIs can be predicted in bi-directional manner by solving optimization problem. For higher coding efficiency, a Gaussian-Wiener representation based hierarchical prediction structure (GWR-HPS) is also designed and applied in the coding scheme. The proposed coding scheme is performed on 10 test sequences, including 5 synthetic scenes and 5 realistic scenes. Experimental results show that proposed coding scheme can obtain 2.640 dB PSNR gains and 51.830% bitrate savings on average of all test sequences in Low Delay P configuration, 2.123 dB PSNR gains and 43.975% bitrate savings in Low Delay B configuration, and 1.044 dB PSNR gains and 26.078% bitrate savings in Random Access configuration. Particularly, it achieves up to 65.544% bit rate savings and 3.901 dB PSNR increments for test sequence I09 in Low Delay P configuration. Furthermore, ablation test demonstrates that Gaussian representation contributes more on coding performance than Wiener representation and GWR-HPS.

Anthropogenic CO&amp;lt;sub&amp;gt;2&amp;lt;/sub&amp;gt; monitoring satellite mission: the need for multi-angle polarimetric observations

Abstract. Atmospheric aerosols have been known to be a major source of uncertainties in CO2 concentrations retrieved from space. In this study, we investigate the added value of multi-angle polarimeter (MAP) measurements in the context of the Copernicus Anthropogenic Carbon Dioxide Monitoring (CO2M) mission. To this end, we compare aerosol-induced XCO2 errors from standard retrievals using a spectrometer only (without MAP) with those from retrievals using both MAP and a spectrometer. MAP observations are expected to provide information about aerosols that is useful for improving XCO2 accuracy. For the purpose of this work, we generate synthetic measurements for different atmospheric and geophysical scenes over land, based on which XCO2 retrieval errors are assessed. We show that the standard XCO2 retrieval approach that makes no use of auxiliary aerosol observations returns XCO2 errors with an overall bias of 1.12 ppm and a spread (defined as half of the 15.9–84.1 percentile range) of 2.07 ppm. The latter is far higher than the required XCO2 accuracy (0.5 ppm) and precision (0.7 ppm) of the CO2M mission. Moreover, these XCO2 errors exhibit a significantly larger bias and scatter at high aerosol optical depth, high aerosol altitude, and low solar zenith angle, which could lead to worse performance in retrieving XCO2 from polluted areas where CO2 and aerosols are co-emitted. We proceed to determine MAP instrument specifications in terms of wavelength range, number of viewing angles, and measurement uncertainties that are required to achieve XCO2 accuracy and precision targets of the mission. Two different MAP instrument concepts are considered in this analysis. We find that for either concept, MAP measurement uncertainties on radiance and degree of linear polarization should be no more than 3 % and 0.003, respectively. A retrieval exercise using MAP and spectrometer measurements of the synthetic scenes is carried out for each of the two MAP concepts. The resulting XCO2 errors have an overall bias of −0.004 ppm and a spread of 0.54 ppm for one concept, and a bias of 0.02 ppm and a spread of 0.52 ppm for the other concept. Both are compliant with the CO2M mission requirements; the very low bias is especially important for proper emission estimates. For the test ensemble, we find effectively no dependence of the XCO2 errors on aerosol optical depth, altitude of the aerosol layer, and solar zenith angle. These results indicate a major improvement in the retrieved XCO2 accuracy with respect to the standard retrieval approach, which could lead to a higher data yield, better global coverage, and a more comprehensive determination of CO2 sinks and sources. As such, this outcome underlines the contribution of, and therefore the need for, a MAP instrument aboard the CO2M mission.

A Switching-Coupled Backend for Simultaneous Localization and Dynamic Object Tracking

Simultaneous localization and object tracking (SLOT) is essentially important for autonomous systems. Tightly-coupled and loosely-coupled methods are two commonly used back-end frameworks for the state-of-the-art solutions of SLOT problem. However, some inherent limitations exist in these two frameworks. In particular, the tightly-coupled method is usually disturbed by the poor observations of some dynamic objects, and the performance of a loosely-coupled one completely depends on that of classical static simultaneous localization and mapping (SLAM) process. Motivated by these observations, we propose a novel switching-coupled back-end solution and theoretically derive its concrete form using probability representation. Based on the switching strategy and the proposed objects classification criteria where the object uncertainty, observation quality and prior information are jointly considered, the dynamic objects' states are flexibly coupled with camera's state and static landmarks' states. For implementation, the measurement constraints of “good” dynamic objects and static landmarks are simultaneously leveraged to perform SLAM and good object tracking (SLAMGOT) process, and those of “bad” objects are used for bad object tracking (BOT) process based on the obtained camera state. Extensive evaluations on synthetic scenes, KITTI datasets and real-world experiments demonstrate the performance of the proposed method.

ClusterSLAM: A SLAM backend for simultaneous rigid body clustering and motion estimation

We present a practical backend for stereo visual SLAM which can simultaneously discover individual rigid bodies and compute their motions in dynamic environments. While recent factor graph based state optimization algorithms have shown their ability to robustly solve SLAM problems by treating dynamic objects as outliers, their dynamic motions are rarely considered. In this paper, we exploit the consensus of 3D motions for landmarks extracted from the same rigid body for clustering, and to identify static and dynamic objects in a unified manner. Specifically, our algorithm builds a noise-aware motion affinity matrix from landmarks, and uses agglomerative clustering to distinguish rigid bodies. Using decoupled factor graph optimization to revise their shapes and trajectories, we obtain an iterative scheme to update both cluster assignments and motion estimation reciprocally. Evaluations on both synthetic scenes and KITTI demonstrate the capability of our approach, and further experiments considering online efficiency also show the effectiveness of our method for simultaneously tracking ego-motion and multiple objects.

Deferred neural lighting

We present deferred neural lighting, a novel method for free-viewpoint relighting from unstructured photographs of a scene captured with handheld devices. Our method leverages a scene-dependent neural rendering network for relighting a rough geometric proxy with learnable neural textures. Key to making the rendering network lighting aware are radiance cues: global illumination renderings of a rough proxy geometry of the scene for a small set of basis materials and lit by the target lighting. As such, the light transport through the scene is never explicitely modeled, but resolved at rendering time by a neural rendering network. We demonstrate that the neural textures and neural renderer can be trained end-to-end from unstructured photographs captured with a double hand-held camera setup that concurrently captures the scene while being lit by only one of the cameras' flash lights. In addition, we propose a novel augmentation refinement strategy that exploits the linearity of light transport to extend the relighting capabilities of the neural rendering network to support other lighting types (e.g., environment lighting) beyond the lighting used during acquisition (i.e., flash lighting). We demonstrate our deferred neural lighting solution on a variety of real-world and synthetic scenes exhibiting a wide range of material properties, light transport effects, and geometrical complexity.

In-Situ and Aircraft Reflectance Measurement Effectiveness for CAL/VAL Activities: A Study over Railroad Valley

This paper aims to assess the relationship between the surface reflectance derived from ground based and aircraft measurements. The parameters of the Rahman–Pinty–Verstraete (RPV) and Ross Thick-LiSparse (RTLS) kernel based bi-directional reflectance distribution functions (BRDF), have been derived using actual measurements of the hemispherical-directional reflectance factor (HDRF), collected during different campaigns over the Railroad Valley Playa. The effect of the atmosphere, including that of the diffuse radiation on bi-directional reflectance factor (BRF) parameter retrievals, assessed using 6S model simulations, was negligible for the low turbidity conditions of the site under investigation (τ550≤0.05). It was also shown that the effects of the diffuse radiation on RPV spectral parameters retrieval is linear for the isotropic parameter ρ0 and the scattering parameter Θ, and can be described with a second order polynomial for the k-Minnaert parameter. In order to overcome the lack of temporal collocations between aircraft and in-situ measurements, Monte Carlo 3-D radiative transfer simulations mimicking in-situ and remote sensing techniques were performed on a synthetic parametric meshed scene defined by merging Landsat and Multianglhe Imaging Spectroradiometer (MISR) remote sensing reflectance data. We simulated directional reflectance measurements made at different heights for PARABOLA and CAR, and analyzed them according to practices adopted for real measurements, consisting of the inversion of BRF functions and the calculation of the bi-hemispherical reflectance (BHR). The difference of retrievals against the known benchmarks of kernel parameters and BHR is presented. We associated an uncertainty of up to 2% with the retrieval of area averaged BHR, independently of flight altitudes and the BRF model used for the inversion. As expected, the local nature of PARABOLA data is revealed by the difference of the anisotropic kernel parameters with the corresponding parameters retrieved from aircraft loops. The uncertainty of the resultant BHR fell within ±3%.

Unconstrained Bilingual Scene Text Reading Using Octave as a Feature Extractor

Reading text and unified text detection and recognition from natural images are the most challenging applications in computer vision and document analysis. Previously proposed end-to-end scene text reading methods do not consider the frequency of input images at feature extraction, which slows down the system, requires more memory, and recognizes text inaccurately. In this paper, we proposed an octave convolution (OctConv) feature extractor and a time-restricted attention encoder-decoder module for end-to-end scene text reading. The OctConv can extract features by factorizing the input image based on their frequency. It is a direct replacement of convolutions, orthogonal and complementary, for reducing redundancies and helps to boost the reading text through low memory requirements at a faster speed. In the text reading process, features are first extracted from the input image using Feature Pyramid Network (FPN) with OctConv Residual Network with depth 50 (ResNet50). Then, a Region Proposal Network (RPN) is applied to predict the location of the text area by using extracted features. Finally, a time-restricted attention encoder-decoder module is applied after the Region of Interest (RoI) pooling is performed. A bilingual real and synthetic scene text dataset is prepared for training and testing the proposed model. Additionally, well-known datasets including ICDAR2013, ICDAR2015, and Total Text are used for fine-tuning and evaluating its performance with previously proposed state-of-the-art methods. The proposed model shows promising results on both regular and irregular or curved text detection and reading tasks.

Simulated JWST Data Sets for Multispectral and Hyperspectral Image Fusion

Abstract The James Webb Space Telescope (JWST) will provide multispectral and hyperspectral infrared images of a large number of astrophysical scenes. Multispectral images will have the highest angular resolution, while hyperspectral images (e.g., with integral field unit spectrometers) will provide the best spectral resolution. This paper aims at providing a comprehensive framework to generate an astrophysical scene and to simulate realistic hyperspectral and multispectral data acquired by two JWST instruments, namely, NIRCam Imager and NIRSpec IFU. We want to show that this simulation framework can be resorted to assess the benefits of fusing these images to recover an image of high spatial and spectral resolutions. To do so, we make a synthetic scene associated with a canonical infrared source, the Orion Bar. We develop forward models including corresponding noises for the two JWST instruments based on their physical features. JWST observations are then simulated by applying the forward models to the aforementioned synthetic scene. We test a dedicated fusion algorithm we developed on these simulated observations. We show that the fusion process reconstructs the high spatio-spectral resolution scene with a good accuracy on most areas, and we identify some limitations of the method to be tackled in future works. The synthetic scene and observations presented in the paper can be used, for instance, to evaluate instrument models, pipelines, or more sophisticated algorithms dedicated to JWST data analysis. Besides, fusion methods such as the one presented in this paper are shown to be promising tools to fully exploit the unprecedented capabilities of the JWST.

Three Dimensional Radiative Effects in Passive Millimeter/Sub-Millimeter All-sky Observations

This study was conducted to quantify the errors prompted by neglecting three-dimensional (3D) effects, i.e., beam-filling and horizontal photon transport effects, at millimeter/sub-millimeter wavelengths. This paper gives an overview of the 3D effects that impact ice cloud retrievals of both current and proposed (Ice Cloud Imager) satellite instruments operating at frequencies of ≈186.3 and ≈668 GHz. The 3D synthetic scenes were generated from two-dimensional (2D) CloudSat (Cloud Satellite) observations over the tropics and mid-latitudes using a stochastic approach. By means of the Atmospheric Radiative Transfer Simulator (ARTS), three radiative transfer simulations were carried out: one 3D, one independent beam approximation (IBA), and one-dimensional (1D). The comparison between the 3D and IBA simulations revealed a small horizontal photon transport effect, with IBA simulations introducing mostly random errors and a slight overestimation (below 1 K). However, performing 1D radiative transfer simulations results in a significant beam-filling effect that increases primarily with frequency, and secondly, with footprint size. For a sensor footprint size of 15 km, the errors induced by neglecting domain heterogeneities yield root mean square errors of up to ≈4 K and ≈13 K at 186.3 GHz and 668 GHz, respectively. However, an instrument operating at the same frequencies, but with a much smaller footprint size, i.e., 6 km, is subject to smaller uncertainties, with a root mean square error of ≈2 K at 186.3 GHz and ≈7.1 K at 668 GHz. When designing future satellite instruments, this effect of footprint size on modeling uncertainties should be considered in the overall error budget. The smallest possible footprint size should be a priority for future sub-millimeter observations in light of these results.