Simulated Synthetic Data Research Articles

Transforming Finance Into Vision: Concurrent Financial Time Series as Convolutional Nets.

We present a novel representation for multiple synchronized financial time series as images, motivated by deep learning methods in machine vision. The research pursues two related strands of inquiry. The first is to transform concurrent synchronized time series analysis-one that is prevalent in Finance and other domains-into a machine vision problem so that the standard deep learning machinery such as convolutional nets can be applied to the transformed problem. The second line of inquiry pursues the idea of transfer learning, where learning occurs on synthetic simulated data corresponding to a finite set of lead-lag relationships in the concurrent time series, and the learned model is applied out of the box to the application domain, in our case, Finance. The successful application of transfer learning, however, requires that a relationship exists between the simulated and real-world data that the learner is able to discern. This relationship helps to bias the learner toward learning things that will be useful in the application domain. We demonstrate the application of our trained model for identifying data-driven regime shifts in financial time series data. We present an analysis of the results and discuss some of the useful properties of the representation and directions for future research.

Delaunay Triangulation-Based Spatial Clustering Technique for Enhanced Adjacent Boundary Detection and Segmentation of LiDAR 3D Point Clouds.

In spatial data with complexity, different clusters can be very contiguous, and the density of each cluster can be arbitrary and uneven. In addition, background noise that does not belong to any clusters in the data, or chain noise that connects multiple clusters may be included. This makes it difficult to separate clusters in contact with adjacent clusters, so a new approach is required to solve the nonlinear shape, irregular density, and touching problems of adjacent clusters that are common in complex spatial data clustering, as well as to improve robustness against various types of noise in spatial clusters. Accordingly, we proposed an efficient graph-based spatial clustering technique that employs Delaunay triangulation and the mechanism of DBSCAN (density-based spatial clustering of applications with noise). In the performance evaluation using simulated synthetic data as well as real 3D point clouds, the proposed method maintained better clustering and separability of neighboring clusters compared to other clustering techniques, and is expected to be of practical use in the field of spatial data mining.

Bayesian Spectral Deconvolution Based on Poisson Distribution: Bayesian Measurement and Virtual Measurement Analytics (VMA)

In this paper, we propose a new method of Bayesian measurement for spectral deconvolution, which regresses spectral data into the sum of unimodal basis function such as Gaussian or Lorentzian functions. Bayesian measurement is a framework for considering not only the target physical model but also the measurement model as a probabilistic model, and enables us to estimate the parameter of a physical model with its confidence interval through a Bayesian posterior distribution given a measurement data set. The measurement with Poisson noise is one of the most effective system to apply our proposed method. Since the measurement time is strongly related to the signal-to-noise ratio for the Poisson noise model, Bayesian measurement with Poisson noise model enables us to clarify the relationship between the measurement time and the limit of estimation. In this study, we establish the probabilistic model with Poisson noise for spectral deconvolution. Bayesian measurement enables us to perform virtual and computer simulation for a certain measurement through the established probabilistic model. This property is called "Virtual Measurement Analytics(VMA)" in this paper. We also show that the relationship between the measurement time and the limit of estimation can be extracted by using the proposed method in a simulation of synthetic data and real data for XPS measurement of MoS$_2$.

Copula index for detecting dependence and monotonicity between stochastic signals

This paper introduces a nonparametric copula-based index for detecting the strength and monotonicity structure of linear and nonlinear statistical dependence between pairs of random variables or stochastic signals. Our index, termed Copula Index for Detecting Dependence and Monotonicity (CIM), satisfies several desirable properties of measures of association, including most of Rényi’s properties, the data processing inequality (DPI), and consequently self-equitability. Synthetic data simulations reveal that the statistical power of CIM compares favorably to other state-of-the-art measures of association that are proven to satisfy the DPI. Simulation results with real-world data reveal CIM’s unique ability to detect the monotonicity structure among stochastic signals to find interesting dependencies in large datasets. Additionally, simulations show that CIM compares favorably to estimators of mutual information when discovering Markov network structure.

Bayesian inverse estimation of urban CO2 emissions: Results from a synthetic data simulation over Salt Lake City, UT

Top-down, data-driven models possess ample power to improve the accuracy of bottom-up carbon dioxide (CO2) emission inventories, and more work is needed to explore the merger of top-down and bottom-up estimates to better inform the metrics used to monitor global CO2 fluxes. Here we present a Bayesian inverse modeling framework over Salt Lake City, Utah, which utilizes available CO2 emission inventories to establish a synthetic data simulation aimed at exploring model uncertainties. Prescribing a high-resolution, urban-scale data product (Hestia) as the “true” emissions in the model, we combine prior emissions with an atmospheric transport model to derive modeled afternoon CO2 enhancements at six monitoring sites within the Salt Lake Valley during the month of September 2015. A global high-resolution gridded emissions data product (ODIAC) is used as the prior, and objective uncertainty structures are defined for both the a priori estimates and the transport model-data relationship which consider non-negligible spatial and temporal covariances. Optimized (posterior) emissions over the Salt Lake Valley agree closely with the assumed “true” emissions during afternoon times, while results including unconstrained times (e.g. night-time) lack such agreement. Both spatial and temporal correlations of prior errors were found to be necessary for obtaining a robust posterior estimate. Model sensitivity analyses are performed, which examine correlation length and time scales, model-data mismatch error, and measurement site network variability. Through these analyses, one measurement site is identified as being particularly prone to introducing bias into posterior emissions due to influences from a nearby point source. Increasing model-data mismatch error at this site is shown to reduce bias in the posterior without significantly compromising agreement with monthly averaged true emissions.

Gibbs Markov random fields with continuous values based on the modified planar rotator model

We introduce a novel Gibbs Markov random field for spatial data on Cartesian grids based on the modified planar rotator (MPR) model of statistical physics. The MPR captures spatial correlations using nearest-neighbor interactions of continuously-valued spins and does not rely on Gaussian assumptions. The only model parameter is the reduced temperature, which we estimate by means of an ergodic specific energy matching principle. We propose an efficient hybrid Monte Carlo simulation algorithm that leads to fast relaxation of the MPR model and allows vectorization. Consequently, the MPR computational time for inference and simulation scales approximately linearly with system size. This makes it more suitable for big data sets, such as satellite and radar images, than conventional geostatistical approaches. The performance (accuracy and computational speed) of the MPR model is validated with conditional simulation of Gaussian synthetic and non-Gaussian real data (atmospheric heat release measurements and Walker-lake DEM-based concentrations) and comparisons with standard gap-filling methods.

Random noise attenuation by 3D Multi-directional vector median filter

Noise attenuation is an important data processing step in exploration geophysics, which is used to extract useful information or improve the signal to noise ratio (SNR) from the raw measurements. The vector median filter (VMF), as an extension of the scalar median filter (SMF), can efficiently suppress random noise and preserve discontinuities of the spreading structures within the data, but it faces questions of how to define a minimum distance and what appropriate additional constraints should be introduced for a reasonable unique median-output vector. Additionally, to avoid uncertainty in data processing using the 2D-VMF method and to more reasonably describe the 3D seismic cube, this study introduces a 3D multi-directional VMF. The new 3D-MDVMF method primarily consists of two steps: 1) detecting the main local dip using some trial angles with a similarity judgement and 2) applying the filter with the constraints for noise suppression and structure enhancement. Synthetic simulated data and marine data are tested. The results show that the proposed method can enhance local dip detection, separate the random noise, and well preserve discontinuities of the structures. This method might be significant for general noise attenuation for data containing spreading structures. KeywordsNoise attenuationMedian filterSeismic dataMulti-directional operator

Deadline Is Not Enough: Importance-Aware Transmission Control Protocol for Server-Centric Data Centers

An important challenge for modern data centers is to provide users with low-latency, high-quality, and interactive services. To improve effective application throughput, much work has been done to take into account application deadline in the design of the network flow schedule for transmission of the queried data. This raises a critical question: does application-level throughput necessarily translate to better quality of service for users? We note that queries responses typically include a set of semantically related documents (within a flow), and that while some documents are highly related to that query, others are not. This motivates us to consider the degree of a document’s semantic relatedness in the design of an efficient data transmission scheme for data centers. To understand the performance upper bound, we derive a mixed integer linear programming (MILP) model for the above problem, which can be regarded as the schedule of a network subflow problem and is hence complicated. Thus, we propose the importance-aware transmission control protocol (ITCP), a distributed event-driven rate-based delivery control protocol, for server-centric data center networks. ITCP jointly considers the subflow importance, size, and deadline, as well as the characteristics of the underlying infrastructure to maximize the goodput of the most relevant responses of a query. The results of both real and synthetic data simulations show that ITCP outperforms recent works in terms of mean average precision of top $k$ data and the sum of application-level importance.

Iterative deblending using the POCS algorithm in the approximate flattened domain

Abstract We proposed an improved method to eliminate the interference generated by simultaneous-source acquisition, which can help shorten the acquisition period and improve the quality of seismic data. An iterative mathematical framework is devised, which uses the projection onto convex sets algorithm to estimate the blending noise subtracted from the pseudo-deblended data to separate the blended data in an iterative way. Differently to the conventional method using the coherent-promoting operator only based on the curvelet transform, we combine the curvelet transform and the approximate flattened operator (AFO) to improve the deblended result, which can flatten seismic events approximately to preserve the details of useful signals. This is the first time that the AFO and the curvelet transform are combined to enhance the effect of the coherent-promoting operator and improve the performance of deblending. To display the advantages of the improved method, we use both simulated synthetic data and field data examples to compare and analyse the deblended results using our method and the conventional method, and confirm that the improved method can perform better.

Multivariate Granger causality unveils directed parietal to prefrontal cortex connectivity during task-free MRI

While a large body of research has focused on the study of functional brain “connectivity”, few investigators have focused on directionality of brain-brain interactions which, in spite of the mostly bidirectional anatomical substrates, cannot be assumed to be symmetrical. We employ a multivariate Granger Causality-based approach to estimating directed in-network interactions and quantify its advantages using extensive realistic synthetic BOLD data simulations to match Human Connectome Project (HCP) data specification. We then apply our framework to resting state functional MRI (rs-fMRI) data provided by the HCP to estimate the directed connectome of the human brain. We show that the functional interactions between parietal and prefrontal cortices commonly observed in rs-fMRI studies are not symmetrical, but consists of directional connectivity from parietal areas to prefrontal cortices rather than vice versa. These effects are localized within the same hemisphere and do not generalize to cross-hemispheric functional interactions. Our data are consistent with neurophysiological evidence that posterior parietal cortices involved in processing and integration of multi-sensory information modulate the function of more anterior prefrontal regions implicated in action control and goal-directed behaviour. The directionality of functional connectivity can provide an additional layer of information in interpreting rs-fMRI studies both in health and disease.

Times Varying Spectral Coherence Investigation of Cardiovascular Signals Based on Energy Concentration in Healthy Young and Elderly Subjects by the Adaptive Continuous Morlet Wavelet Transform

ObjectiveThe aims of this study, to investigate the interaction among heart rate variability (HRV), respiratory, systolic arterial blood pressure variability (SABPV) and systolic arterial pressure interval variability (APIV) signals for understanding of cardiovascular control. MethodsIn this study, three methods referred as adaptive continuous Morlet wavelet transform (ADCMWT), adaptive Stockwell transform (ADST) and adaptive modified Stockwell transform (ADMST) was used to assess the accuracy (AC) of time-varying spectral coherence (TVSC). The adaptation of these estimators was based on maximum energy concentration measurement. The capability to correct temporal localization of time–frequency regions was validated on synthetic time series data modeled as dynamic characteristics of cardiovascular signals. ResultsThe results on synthetic simulated data show that the ADCMWT method allows for the temporal localization of the time–frequency regions with higher accuracy (AC > 96.074% for SNR ≥ 0 dB), compared to ADST (AC > 90.71% for SNR ≥ 0 dB) and ADMST (AC > 84.45% for SNR ≥ 5 dB). Further, the ADCMWT was applied to real cardiovascular data obtained from Fantasia standard data base and grouped as, 8 young subjects (4M + 4F, age range 23–32) and 8 elderly subjects (4M + 4F, age range 70–82) for estimating the TVSC in low frequency (LF) band (0.04 Hz–0.15 Hz) and high frequency (HF) band (0.15 Hz–0.4 Hz) of HRV spectrum. The global result depict that the median value of interquartile range of coherency between HRV-SABPV and HRV-APIV signals in LF and HF band were significantly (p=0.00001) lower in elderly group subjects compared to young group subjects. The coupling between HRV-Respiratory signals in LF band was not significantly affected with the aging of healthy subjects. However, this coupling in HF band significantly reduced in elderly compare to young group subjects (p=0.0125). ConclusionThe comparative study shows that the time-varying spectra and accurate localization of coupling between two physiological signals can be affected by energy concentration. The ADCMWT at w0=6, could be an alternative, possibly more suitable and highly accurate method for assessment and detection of time varying spectral and coherence components of cardiovascular time series.

The Multi-strand Graph for a PTZ Tracker

High-resolution images can be used to resolve matching ambiguities between trajectory fragments (tracklets), which is one of the main challenges in multiple target tracking. A PTZ camera, which can pan, tilt and zoom, is a powerful and efficient tool that offers both close-up views and wide area coverage on demand. The wide-area view makes it possible to track many targets while the close-up view allows individuals to be identified from high-resolution images of their faces. A central component of a PTZ tracking system is a scheduling algorithm that determines which target to zoom in on. In this paper we study this scheduling problem from a theoretical perspective, where the high resolution images are also used for tracklet matching. We propose a novel data structure, the Multi-Strand Tracking Graph (MSG), which represents the set of tracklets computed by a tracker and the possible associations between them. The MSG allows efficient scheduling as well as resolving -- directly or by elimination -- matching ambiguities between tracklets. The main feature of the MSG is the auxiliary data saved in each vertex, which allows efficient computation while avoiding time-consuming graph traversal. Synthetic data simulations are used to evaluate our scheduling algorithm and to demonstrate its superiority over a na\"ive one.

Prediction of individual outcomes for asthma sufferers.

We consider the problem of individual-specific medication level recommendation (initiation, removal, increase, or decrease) for asthma sufferers. Asthma is one of the most common chronic diseases in both adults and children, affecting 8% of the US population and costing $37-63 billion/year in the United States of America. Asthma is a complex disease, whose symptoms may wax and wane, making it difficult for clinicians to predict outcomes and prognosis. Improved ability to predict prognosis can inform decision making and may promote conversations between clinician and provider around optimizing medication therapy. Data from the US Medical Expenditure Panel Survey (MEPS) years 2000-2010 were used to fit a longitudinal model for a multivariate response of adverse events (Emergency Department or in-patient visits, excessive rescue inhaler use, and oral steroid use). To reduce bias in the estimation of medication effects, medication level was treated as a latent process which was restricted to be consistent with prescription refill data. This approach is demonstrated to be effective in the MEPS cohort via predictions on a validation hold out set and a synthetic data simulation study. This framework can be easily generalized to medication decisions for other conditions as well.

Entropy Ensemble Filter: A Modified Bootstrap Aggregating (Bagging) Procedure to Improve Efficiency in Ensemble Model Simulation

Over the past two decades, the Bootstrap AGGregatING (bagging) method has been widely used for improving simulation. The computational cost of this method scales with the size of the ensemble, but excessively reducing the ensemble size comes at the cost of reduced predictive performance. The novel procedure proposed in this study is the Entropy Ensemble Filter (EEF), which uses the most informative training data sets in the ensemble rather than all ensemble members created by the bagging method. The results of this study indicate efficiency of the proposed method in application to synthetic data simulation on a sinusoidal signal, a sawtooth signal, and a composite signal. The EEF method can reduce the computational time of simulation by around 50% on average while maintaining predictive performance at the same level of the conventional method, where all of the ensemble models are used for simulation. The analysis of the error gradient (root mean square error of ensemble averages) shows that using the 40% most informative ensemble members of the set initially defined by the user appears to be most effective.

Inferring catchment flow path responses using a data-driven model: an exploratory study based on a generalized additive model

ABSTRACTIdentifying physical catchment processes from streamflow data, such as quick- and slow-flow paths, remains challenging. This study is designed to explore whether a flexible nonparametric regression model (generalized additive model, GAM) can be used to infer different flow paths. This assumes that the data relationship in data-driven models is also a reflection of catchment physical processes. The GAM, using time-lagged flow covariates, was fitted to synthetic rainfall–runoff data simulated using simple linear reservoirs. Partial plots of the time-lagged covariates show that the model could differentiate simple and more complex flow paths in simulated synthetic data with short and long memory systems and varying between dry and wet climates. Further analysis of data from real catchments showed that the model could differentiate catchments dominated by slow flow and by quick flow. Therefore, this study indicates that GAM can be used to identify catchment storages and delay processes from streamflow data.

Impact of hydroclimate parameter uncertainty on system yield

Water resource system behaviour is commonly evaluated using Monte Carlo simulation of synthetic streamflow and rainfall data. Fundamental to this process is the estimation of stochastic model parameters by calibration to short observed records. The uncertainty in key hydroclimate parameters, such as the mean, variance and serial autocorrelation, flows through to the simulated hydroclimate data. This study considers the subsequent influence of model parameter uncertainty on the calculated yield of a case study water supply system. A procedure is developed to estimate the system yield for individual samples of model parameters, providing a probability distribution of plausible system yields. The uncertainty in hydroclimate parameters is found to have an appreciable impact on the uncertainty in system yield – of the order of +/−10%. This uncertainty is likely to surpass, potentially by an order of magnitude or more, the influence of many choices made by water managers in both the modelling and operation of water resource systems.

Calibration and Validation of a Detailed Architectural Canopy Model Reconstruction for the Simulation of Synthetic Hemispherical Images and Airborne LiDAR Data

Canopy density measures such as the Leaf Area Index (LAI) have become standardized mapping products derived from airborne and terrestrial Light Detection And Ranging (aLiDAR and tLiDAR, respectively) data. A specific application of LiDAR point clouds is their integration into radiative transfer models (RTM) of varying complexity. Using, e.g., ray tracing, this allows flexible simulations of sub-canopy light condition and the simulation of various sensors such as virtual hemispherical images or waveform LiDAR on a virtual forest plot. However, the direct use of LiDAR data in RTMs shows some limitations in the handling of noise, the derivation of surface areas per LiDAR point and the discrimination of solid and porous canopy elements. In order to address these issues, a strategy upgrading tLiDAR and Digital Hemispherical Photographs (DHP) into plausible 3D architectural canopy models is suggested. The presented reconstruction workflow creates an almost unbiased virtual 3D representation of branch and leaf surface distributions, minimizing systematic errors due to the object–sensor relationship. The models are calibrated and validated using DHPs. Using the 3D models for simulations, their capabilities for the description of leaf density distributions and the simulation of aLiDAR and DHP signatures are shown. At an experimental test site, the suitability of the models, in order to systematically simulate and evaluate aLiDAR based LAI predictions under various scan settings is proven. This strategy makes it possible to show the importance of laser point sampling density, but also the diversity of scan angles and their quantitative effect onto error margins.

Improving Student Results in the Crystal Violet Chemical Kinetics Experiment

Despite widespread use in general chemistry laboratories, the crystal violet chemical kinetics experiment frequently suffers from erroneous student results. Student calculations for the reaction order in hydroxide often contain large asymmetric errors, pointing to the presence of systematic error. Through a combination of in silico synthetic data simulations and laboratory experimentation, we report on five potential sources of error: (1) the kinetic flooding approximation, (2) solution concentration mistakes, (3) signal processing, (4) spectrometer limitations, and (5) particulates in the solution. The majority of this asymmetric error can be eliminated by constraining the data collection region within spectrometer limitations, specifically 0.7–0.1 Abs for a SpectroVis+ spectrophotometer. The consistent detection of particulate formation during the later course of the reaction is an aspect of the reaction not previously explored. Guidelines for optimizing laboratory procedure, instruments, and software a...

Cautions on the treatment of non-detect results for environmental forensics

ABSTRACTWhile censored data (i.e., containing non-detect [ND] results) may provide useful information in the context of risk assessment, their utility in chemical forensics is less clear. Substitution of NDs can create invasive patterns and may obscure the true forensic features in the underlying chemistry and sample compositions. The effect of substituting numeric values for NDs on the results of common chemical forensics multivariate analyses is illustrated using synthetic simulated data and laboratory measurements of soil samples from Colorado. The analyses of simulated and laboratory data demonstrate the impact of substituting values for NDs on multivariate analyses and indicate the potential for incorrect conclusions about the true forensic features within the data. Recommendations are provided for how to investigate and visualize the impact of NDs on chemical forensic multivariate analyses.

New satellite project Aerosol-UA: Remote sensing of aerosols in the terrestrial atmosphere

We discuss the development of the Ukrainian space project Aerosol-UA which has the following three main objectives: (1) to monitor the spatial distribution of key characteristics of terrestrial tropospheric and stratospheric aerosols; (2) to provide a comprehensive observational database enabling accurate quantitative estimates of the aerosol contribution to the energy budget of the climate system; and (3) quantify the contribution of anthropogenic aerosols to climate and ecological processes. The remote sensing concept of the project is based on precise orbital measurements of the intensity and polarization of sunlight scattered by the atmosphere and the surface with a scanning polarimeter accompanied by a wide-angle multispectral imager–polarimeter.Preparations have already been made for the development of the instrument suite for the Aerosol-UA project, in particular, of the multi-channel scanning polarimeter (ScanPol) designed for remote sensing studies of the global distribution of aerosol and cloud properties (such as particle size, morphology, and composition) in the terrestrial atmosphere by polarimetric and spectrophotometric measurements of the scattered sunlight in a wide range of wavelengths and viewing directions from which a scene location is observed. ScanPol is accompanied by multispectral wide-angle imager–polarimeter (MSIP) that serves to collect information on cloud conditions and Earth’s surface image. Various components of the polarimeter ScanPol have been prototyped, including the opto-mechanical and electronic assemblies and the scanning mirror controller. Preliminary synthetic data simulations for the retrieval of aerosol parameters over land surfaces have been performed using the Generalized Retrieval of Aerosol and Surface Properties (GRASP) algorithm. Methods for the validation of satellite data using ground-based observations of aerosol properties are also discussed. We assume that designing, building, and launching into orbit a multi-functional high-precision scanning polarimeter and an imager–polarimeter should make a significant contribution to the study of natural and anthropogenic aerosols and their climatic and ecological effects.