Semi-synthetic Data Research Articles

Deep authoring - an AI Tool set for creating immersive MultiMedia experiences

We introduce a fully automated 360° video processing pipeline using a hierarchical combination of Artificial Intelligence (AI) modules to create immersive volumetric XR experiences. Two critical productions tasks (person segmentation and depth estimation) are addressed with a parallel Deep Neural Network (DNN) pipeline that combines instance segmentation, person detection, pose estimation, camera stabilization, neural tracking, 3D face detection, hair masking, and monocular 360° depth computation in a single and robust tool set. To facilitate the rapid uptake of these techniques we provide a detailed review of AI-based methods to address these problems (complete with links to recommended open source implementations) as well as references to existing authoring tools in the market. Our key contributions include a method to create semi-synthetic data sets for data auto-augmentation and using this technique to generate over 3.8 m images as part of a concise evaluation and subsequent retraining of DNNs for person detection tasks. Furthermore, we apply the same techniques to develop a spherical DNN for monocular depth estimation with a Free Viewpoint Video (FVV) capture system and a novel method to generate 3D human shapes and pose mannequins for training. To evaluate the performance of our AI authoring tool set we address four challenging production tasks and demonstrate the practical use of our solution with videos showing processed output.

Optimizing Revenue While Showing Relevant Assortments at Scale

Scalable real-time assortment optimization has become essential in e-commerce operations due to the need for personalization and the availability of a large variety of items. While this can be done when there are simplistic assortment choices to be made, the optimization process becomes difficult when imposing constraints on the collection of relevant assortments based on insights by store-managers and historically well-performing assortments. We design fast and flexible algorithms based on variations of binary search that find the (approximately) optimal assortment in this difficult regime. In particular, we revisit the problem of large-scale assortment optimization under the multinomial logit choice model without any assumptions on the structure of the feasible assortments. We speed up the comparison steps using advances in similarity search in the field of information retrieval/machine learning. For an arbitrary collection of assortments, our algorithms can find a solution in time that is sub-linear in the number of assortments, and for the simpler case of cardinality constraints - linear in the number of items (existing methods are quadratic or worse). Empirical validations using a real world dataset (in addition to experiments using semi-synthetic data based on the Billion Prices dataset and several retail transaction datasets) show that our algorithms are competitive even when the number of items is $\sim 10^5$ ($10\times$ larger instances than previously studied).

A latent capture history model for digital aerial surveys.

We anticipate that unmanned aerial vehicles will become popular wildlife survey platforms. Because detecting animals from the air is imperfect, we develop a mark-recapture line transect method using two digital cameras, possibly mounted on one aircraft, which cover the same area with a short time delay between them. Animal movement between the passage of the cameras introduces uncertainty in individual identity, so individual capture histories are unobservable and are treated as latent variables. We obtain the likelihood for mark-recapture line transects without capture histories by automatically enumerating all possibilities within segments of the transect that contain ambiguous identities, instead of attempting to decide identities in a prior step. We call this method "Latent Capture-history Enumeration" (LCE). We include an availability model for species that are periodically unavailable for detection, such as cetaceans that are undetectable while diving. External data are needed to estimate the availability cycle length, but not the mean availability rate, if the full availability model is employed. We compare the LCE method with the recently developed cluster capture-recapture method (CCR), which uses a Palm likelihood approximation, providing the first comparison of CCR with maximum likelihood. The LCE estimator has slightly lower variance, more so as sample size increases, and close to nominal coverage probabilities. Both methods are approximately unbiased. We illustrate with semisynthetic data from a harbor porpoisesurvey.

Deep Convolutional Neural Network for Microseismic Signal Detection and Classification

Reliable automatic microseismic waveform detection with high efficiency, precision, and adaptability is the basis of stability analysis of the surrounding rock mass. In this paper, a convolutional neural network (CNN)-based microseismic detection network (CNN-MDN) model was established and well trained to a high degree of accuracy using a dataset with 16,000 preprocessed waveforms. By comparison with other methods, 4000 waveforms were tested to evaluate the precision, recall, and F1-score. The results revealed that the CNN-MDN demonstrated the highest performance in microseismic detection. Moreover, the low sensitivity of the CNN-MDN to noise of different intensities was proved by testing on semi-synthetic data. The model also possesses good generalization ability and superior performance capability for microseismic detection under different geological structure backgrounds, and it can correctly detect the microseismic events with Mw ≥ 0.5. These preliminary results show that the CNN-MDN can be directly applied to unprocessed microseismic data and has great potential in real-time microseismic monitoring applications.

Minimax-Optimal Policy Learning Under Unobserved Confounding

We study the problem of learning personalized decision policies from observational data while accounting for possible unobserved confounding. Previous approaches, which assume unconfoundedness, that is, that no unobserved confounders affect both the treatment assignment as well as outcome, can lead to policies that introduce harm rather than benefit when some unobserved confounding is present as is generally the case with observational data. Instead, because policy value and regret may not be point-identifiable, we study a method that minimizes the worst-case estimated regret of a candidate policy against a baseline policy over an uncertainty set for propensity weights that controls the extent of unobserved confounding. We prove generalization guarantees that ensure our policy is safe when applied in practice and in fact obtains the best possible uniform control on the range of all possible population regrets that agree with the possible extent of confounding. We develop efficient algorithmic solutions to compute this minimax-optimal policy. Finally, we assess and compare our methods on synthetic and semisynthetic data. In particular, we consider a case study on personalizing hormone replacement therapy based on observational data, in which we validate our results on a randomized experiment. We demonstrate that hidden confounding can hinder existing policy-learning approaches and lead to unwarranted harm although our robust approach guarantees safety and focuses on well-evidenced improvement, a necessity for making personalized treatment policies learned from observational data reliable in practice. This paper was accepted by Hamid Nazerzadeh, big data analytics.

Deep learning for scanning electron microscopy: Synthetic data for the nanoparticles detection

Deep learning algorithms are one of most rapid developing fields into the modern computation technologies. One of the bottlenecks into the implementation of such advaced algorithms is their requirement for a large amount of manually-labelled data for training. For the general-purpose tasks, such as general purpose image classification/detection the huge images datasets are already labelled and collected. For more subject specific tasks (such as electron microscopy images treatment), no labelled data available. Here I demonstrate that a deep learning network can be successfully trained for nanoparticles detection using semi-synthetic data. The real SEM images were used as a textures for rendered nanoparticles at the surface. Training of RetinaNet architecture using transfer learning can be helpful for the large-scale particle distribution analysis. Beyond such applications, the presented approach might be applicable to other tasks, such as image segmentation.

Anomaly Detection and Attribution in Network

In this article, we address the problem of not only id entifying phenomena, but also attributing the phenomenon to the movement that induces it. This causes to a combinatorial optimisation problem, which is prohibitively expensive. Instead we design two anomaly detection algorithms that are small in complexity. The first is based on the system for cross-entropy (CE), which identifies flow anomalies and labels flow anomalies. The second algorithm detects anomalies through GLRT on aggregated flow transformation a compact low-dimensional representation of raw traffic flows. The two algorithms complement each other and allow the network operator to use the algorithm for flow aggregation first so that device irregularities can be identified easily. After discovery of an exception, the user Can analyse further that individual flows are anomalous using CE-based algorithm. We perform extensive performance tests and trials on synthetic and semi-synthetic data with our algorithms, as well as real Internet traffic data gathered from the MAWI database, and finally make recommendations as to their usability.

Anomaly Detection and Attribution in Network

In this article, we address the problem of not only id entifying phenomena, but also attributing the phenomenon to the movement that induces it. This causes to a combinatorial optimisation problem, which is prohibitively expensive. Instead we design two anomaly detection algorithms that are small in complexity. The first is based on the system for cross-entropy (CE), which identifies flow anomalies and labels flow anomalies. The second algorithm detects anomalies through GLRT on aggregated flow transformation a compact low-dimensional representation of raw traffic flows. The two algorithms complement each other and allow the network operator to use the algorithm for flow aggregation first so that device irregularities can be identified easily. After discovery of an exception, the user Can analyse further that individual flows are anomalous using CE-based algorithm. We perform extensive performance tests and trials on synthetic and semi-synthetic data with our algorithms, as well as real Internet traffic data gathered from the MAWI database, and finally make recommendations as to their usability.

Underwater Object Classification in Sidescan Sonar Images Using Deep Transfer Learning and Semisynthetic Training Data

Sidescan sonars are increasingly used in underwater search and rescue for drowning victims, wrecks and airplanes. Automatic object classification or detection methods can help a lot in case of long searches, where sonar operators may feel exhausted and therefore miss the possible object. However, most of the existing underwater object detection methods for sidescan sonar images are aimed at detecting mine-like objects, ignoring the classification of civilian objects, mainly due to lack of dataset. So, in this study, we focus on the multi-class classification of drowning victim, wreck, airplane, mine and seafloor in sonar images. Firstly, through a long-term accumulation, we built a real sidescan sonar image dataset named SeabedObjects-KLSG, which currently contains 385 wreck, 36 drowning victim, 62 airplane, 129 mine and 578 seafloor images. Secondly, considering the real dataset is imbalanced, we proposed a semisynthetic data generation method for producing sonar images of airplanes and drowning victims, which uses optical images as input, and combines image segmentation with intensity distribution simulation of different regions. Finally, we demonstrate that by transferring a pre-trained deep convolutional neural network (CNN), e.g. VGG19, and fine-tuning the deep CNN using 70% of the real dataset and the semisynthetic data for training, the overall accuracy on the remaining 30% of the real dataset can be eventually improved to 97.76%, which is the highest among all the methods. Our work indicates that the combination of semisynthetic data generation and deep transfer learning is an effective way to improve the accuracy of underwater object classification.

High-dimensional regression in practice: an empirical study of finite-sample prediction, variable selection and ranking

Penalized likelihood approaches are widely used for high-dimensional regression. Although many methods have been proposed and the associated theory is now well developed, the relative efficacy of different approaches in finite-sample settings, as encountered in practice, remains incompletely understood. There is therefore a need for empirical investigations in this area that can offer practical insight and guidance to users. In this paper, we present a large-scale comparison of penalized regression methods. We distinguish between three related goals: prediction, variable selection and variable ranking. Our results span more than 2300 data-generating scenarios, including both synthetic and semisynthetic data (real covariates and simulated responses), allowing us to systematically consider the influence of various factors (sample size, dimensionality, sparsity, signal strength and multicollinearity). We consider several widely used approaches (Lasso, Adaptive Lasso, Elastic Net, Ridge Regression, SCAD, the Dantzig Selector and Stability Selection). We find considerable variation in performance between methods. Our results support a “no panacea” view, with no unambiguous winner across all scenarios or goals, even in this restricted setting where all data align well with the assumptions underlying the methods. The study allows us to make some recommendations as to which approaches may be most (or least) suitable given the goal and some data characteristics. Our empirical results complement existing theory and provide a resource to compare methods across a range of scenarios and metrics.

Efficient Convolutional Neural Networks for Depth-Based Multi-Person Pose Estimation

Achieving robust multi-person 2D body landmark localization and pose estimation is essential for human behavior and interaction understanding as encountered for instance in HRI settings. Accurate methods have been proposed recently, but they usually rely on rather deep Convolutional Neural Network (CNN) architecture, thus requiring large computational and training resources. In this paper, we investigate different architectures and methodologies to address these issues and achieve fast and accurate multi-person 2D pose estimation. To foster speed, we propose to work with depth images, whose structure contains sufficient information about body landmarks while being simpler than textured color images and thus potentially requiring less complex CNNs for processing. In this context, we make the following contributions. i) we study several CNN architecture designs combining pose machines relying on the cascade of detectors concept with lightweight and efficient CNN structures; ii) to address the need for large training datasets with high variability, we rely on semi-synthetic data combining multi-person synthetic depth data with real sensor backgrounds; iii) we explore domain adaptation techniques to address the performance gap introduced by testing on real depth images; iv) to increase the accuracy of our fast lightweight CNN models, we investigate knowledge distillation at several architecture levels which effectively enhance performance. Experiments and results on synthetic and real data highlight the impact of our design choices, providing insights into methods addressing standard issues normally faced in practical applications, and resulting in architectures effectively matching our goal in both performance and speed.

MITRE: inferring features from microbiota time-series data linked to host status

Longitudinal studies are crucial for discovering causal relationships between the microbiome and human disease. We present MITRE, the Microbiome Interpretable Temporal Rule Engine, a supervised machine learning method for microbiome time-series analysis that infers human-interpretable rules linking changes in abundance of clades of microbes over time windows to binary descriptions of host status, such as the presence/absence of disease. We validate MITRE’s performance on semi-synthetic data and five real datasets. MITRE performs on par or outperforms conventional difficult-to-interpret machine learning approaches, providing a powerful new tool enabling the discovery of biologically interpretable relationships between microbiome and human host (https://github.com/gerberlab/mitre/).

Rate optimal estimation and confidence intervals for high-dimensional regression with missing covariates

We consider the problems of estimation and of constructing component-wise confidence intervals in a sparse high-dimensional linear regression model when some covariates of the design matrix are missing completely at random. We analyze a variant of the Dantzig selector for estimating the regression model and we use a de-biasing argument to construct component-wise confidence intervals. We also complement our mathematical study in the supplementary materials with extensive simulations on synthetic and semi-synthetic data that show the accuracy of our asymptotic predictions for finite sample sizes.

Abstract 1650: Deconvolution of copy number alterations combining bulk and single-cell genomic data

Abstract Characterizing the evolution of clonal cell populations in tumor progression is a challenging task given pervasive intratumor heterogeneity (ITH). Bulk DNA sequencing remains the dominant technology for large cohorts, but requires computationally deconvolving clonal substructure from bulk data, an error-prone process with limited resolution. Single-cell sequencing (SCS) is a promising alternative but is not yet practical at the scales needed for large cohorts. To address limitations of both bulk and SCS approaches, we developed strategies for deconvolving ITH by combining bulk sequencing with limited SCS data with specific focus on copy number alterations (CNAs), which are amenable to low-depth SCS and particularly challenging for deconvolution. We introduce two methods based on non-negative matrix factorization (NMF) of bulk data. One method extends the NMF optimization objective with a penalty for deviation of inferred clones from small numbers of observed SCS samples. The other combines clonal deconvolution with tumor phylogeny inference, balancing deconvolution quality against a minimum evolution cost for incorporating inferred and observed single cells into a reconstruction of the history of clonal evolution. We validated the methods on a set of semi-synthetic data derived from true low-depth SCS data consisting of 393 single cells derived from three regions each of two human glioblastoma cases. These data were used to call mean copy numbers at 9934 genomic loci. We artificially mixed the data to generate synthetic bulk samples of known clonal composition and applied each method to correctly reconstruct unobserved single cells and their clonal structures from three, six, or nine semi-synthetic bulk samples plus six single cells each per trial. For three/six/nine bulk samples, we achieved mean RMSD of copy number inference of 1.507/1.483/1.406 for pure NMF without SCS, 0.657/0.641/0.588 for the phylogeny-free SCS method, and 0.479/0.497/0.462 for the phylogeny-based SCS method. RMSD of mixture fractions describing the clonal composition were 0.245/0.242/0.243 for pure NMF without SCS, 0.215/0.198/0.206 for the phylogeny-free SCS method, and 0.215/0.215/0.224 for the phylogeny-based SCS method. The results show that we can substantially improve on bulk CNA deconvolution using limited SCS data, providing a way to balance advantages of pure bulk and pure SCS. The work also supports the value of a principled evolutionary model in inferring accurate clonal structure. Citation Format: Haoyun Lei, Bochuan Lyu, E. Michael Gertz, Alejandro A. Schaeffer, Xulian Shi, Kui Wu, Guibo Li, Liqin Xu, Yong Hou, Michael Dean, Russell Schwartz. Deconvolution of copy number alterations combining bulk and single-cell genomic data [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2019; 2019 Mar 29-Apr 3; Atlanta, GA. Philadelphia (PA): AACR; Cancer Res 2019;79(13 Suppl):Abstract nr 1650.

Hierarchical Bayesian model to inferPL(Z)relations usingGaiaparallaxes

In a recent study we analysed period–luminosity–metallicity (PLZ) relations for RR Lyrae stars using theGaiaData Release 2 (DR2) parallaxes. It built on a previous work that was based on the firstGaiaData Release (DR1), and also included period–luminosity (PL) relations for Cepheids and RR Lyrae stars. The method used to infer the relations fromGaiaDR2 data and one of the methods used forGaiaDR1 data was based on a Bayesian model, the full description of which was deferred to a subsequent publication. This paper presents the Bayesian method for the inference of the parameters ofPL(Z) relations used in those studies, the main feature of which is to manage the uncertainties on observables in a rigorous and well-founded way. The method encodes the probability relationships between the variables of the problem in a hierarchical Bayesian model and infers the posterior probability distributions of thePL(Z) relationship coefficients using Markov chain Monte Carlo simulation techniques. We evaluate the method with several semi-synthetic data sets and apply it to a sample of 200 fundamental and first-overtone RR Lyrae stars for whichGaiaDR1 parallaxes and literatureKs-band mean magnitudes are available. We define and test several hyperprior probabilities to verify their adequacy and check the sensitivity of the solution with respect to the prior choice. The main conclusion of this work, based on the test with semi-syntheticGaiaDR1 parallaxes, is the absolute necessity of incorporating the existing correlations between the period, metallicity, and parallax measurements in the form of model priors in order to avoid systematically biased results, especially in the case of non-negligible uncertainties in the parallaxes. The relation coefficients obtained here have been superseded by those presented in our recent paper that incorporates the findings of this work and the more recentGaiaDR2 measurements.

The Use of Informed Priors in Biclustering of Gene Expression with the Hierarchical Dirichlet Process.

We motivate and describe the application of Hierarchical Dirichlet Process (HDP) models to the "soft" biclustering of gene expression data, in which we obtain modules (biclusters) where the affiliation of genes and samples with the modules are weighted, instead of being hard memberships. As a distinct contribution, we propose a method which HDP is informed with prior beliefs, significantly increasing the quality of the biclustering in terms of both the correctness of the number of modules inferred, and the precision of these modules, especially when evidence is sparse. We outline two such informed priors; one based on co-expression relationships inherent in the data, the other based on an externally provided regulatory network. We validate these results and compare the performance of our approach to Weighted Gene Correlation Network Analysis (WGCNA), another model that features weighted modules. We have, to this end, performed experiments on semi-synthetic data. The results show that HDP, with the addition of a well-informed prior, is able to capture the correct number of modules with increased accuracy. Furthermore, the model becomes robust to changes in the strength of the prior. We conclude by discussing these results and the benefits provided by our approach for gene expression analysis and network validation.

Inertia-Constrained Pixel-by-Pixel Nonnegative Matrix Factorisation: A Hyperspectral Unmixing Method Dealing with Intra-Class Variability

Blind source separation is a common processing tool to analyse the constitution of pixels of hyperspectral images. Such methods usually suppose that pure pixel spectra (endmembers) are the same in all the image for each class of materials. In the framework of remote sensing, such an assumption is no longer valid in the presence of intra-class variability due to illumination conditions, weathering, slight variations of the pure materials, etc. In this paper, we first describe the results of investigations highlighting intra-class variability measured in real images. Considering these results, a new formulation of the linear mixing model is presented leading to two new methods. Unconstrained pixel-by-pixel NMF (UP-NMF) is a new blind source separation method based on the assumption of a linear mixing model, which can deal with intra-class variability. To overcome the limitations of UP-NMF, an extended method is also proposed, named Inertia-constrained Pixel-by-pixel NMF (IP-NMF). For each sensed spectrum, these extended versions of NMF extract a corresponding set of source spectra. A constraint is set to limit the spreading of each source’s estimates in IP-NMF. The proposed methods are first tested on a semi-synthetic data set built with spectra extracted from a real hyperspectral image and then numerically mixed. We thus demonstrate the interest of our methods for realistic source variabilities. Finally, IP-NMF is tested on a real data set and it is shown to yield better performance than state of the art methods.

Towards End-to-End Acoustic Localization Using Deep Learning: From Audio Signals to Source Position Coordinates.

This paper presents a novel approach for indoor acoustic source localization using microphone arrays, based on a Convolutional Neural Network (CNN). In the proposed solution, the CNN is designed to directly estimate the three-dimensional position of a single acoustic source using the raw audio signal as the input information and avoiding the use of hand-crafted audio features. Given the limited amount of available localization data, we propose, in this paper, a training strategy based on two steps. We first train our network using semi-synthetic data generated from close talk speech recordings. We simulate the time delays and distortion suffered in the signal that propagate from the source to the array of microphones. We then fine tune this network using a small amount of real data. Our experimental results, evaluated on a publicly available dataset recorded in a real room, show that this approach is able to produce networks that significantly improve existing localization methods based on SRP-PHAT strategies and also those presented in very recent proposals based on Convolutional Recurrent Neural Networks (CRNN). In addition, our experiments show that the performance of our CNN method does not show a relevant dependency on the speaker’s gender, nor on the size of the signal window being used.

Change detection in multispectral images based on multiband structural information

ABSTRACTChange vector analysis (CVA) and spectral angle mapper (SAM) are usually used to generate difference image in change detection of multispectral images. Although CVA and SAM can describe the difference between multispectral images, they are defined mathematically and lack support of human visual system (HVS) theory. Advanced structural similarity (ASSIM) complies with the pattern that human perceives the changes occurred in an objective scene. Nevertheless, ASSIM was designed for single band images and cannot be used for extracting multiband structural information from multispectral images directly. Therefore, we first propose two strategies to extract multiband structural information from multiband images. Then, we propose the approaches based on multiband structural information for change detection in multispectral images. Experimental results from one semisynthetic data set and two real data sets acquired by Sentinel-2A and QuickBird satellites validate the effectiveness of the proposed approaches.

Gamma-Ray Source Detection With Small Sensors

Large detectors can give better background characterization and can detect radiation sources at larger standoff distances. Small detectors, on the other hand, are less expensive, can often get closer to source materials, and can access places that large detectors cannot (e.g., indoor environments). We systematically quantify the impact of detector size and number on source detection in area search applications. We analyze theoretical upper bounds on source detectability and establish first-order approximations thereof. We demonstrate that these approximations give good comparisons of different choices of sensor size and number for practical source detection algorithms using semisynthetic data. Our results indicate that multiple small detectors, in conjunction, can offer superior overall operational utility compared with a single large detector. Finally, we identify differences in detector response and likelihood of a close approach as key determinants of the relative utility of small sensors.