Large Multivariate Data Sets Research Articles

Effective Data Reduction Using Discriminative Feature Selection Based on Principal Component Analysis

Effective data reduction must retain the greatest possible amount of informative content of the data under examination. Feature selection is the default for dimensionality reduction, as the relevant features of a dataset are usually retained through this method. In this study, we used unsupervised learning to discover the top-k discriminative features present in the large multivariate IoT dataset used. We used the statistics of principal component analysis to filter the relevant features based on the ranks of the features along the principal directions while also considering the coefficients of the components. The selected number of principal components was used to decide the number of features to be selected in the SVD process. A number of experiments were conducted using different benchmark datasets, and the effectiveness of the proposed method was evaluated based on the reconstruction error. The potency of the results was verified by subjecting the algorithm to a large IoT dataset, and we compared the performance based on accuracy and reconstruction error to the results of the benchmark datasets. The performance evaluation showed consistency with the results obtained with the benchmark datasets, which were of high accuracy and low reconstruction error.

Holistic integration of omics data reveals the drivers that shape the ecology of microbial meat spoilage scenarios.

The use of omics data for monitoring the microbial flow of fresh meat products along a production line and the development of spoilage prediction tools from these data is a promising but challenging task. In this context, we produced a large multivariate dataset (over 600 samples) obtained on the production lines of two similar types of fresh meat products (poultry and raw pork sausages). We describe a full analysis of this dataset in order to decipher how the spoilage microbial ecology of these two similar products may be shaped differently depending on production parameter characteristics. Our strategy involved a holistic approach to integrate unsupervised and supervised statistical methods on multivariate data (OTU-based microbial diversity; metabolomic data of volatile organic compounds; sensory measurements; growth parameters), and a specific selection of potential uncontrolled (initial microbiota composition) or controlled (packaging type; lactate concentration) drivers. Our results demonstrate that the initial microbiota, which is shown to be very different between poultry and pork sausages, has a major impact on the spoilage scenarios and on the effect that a downstream parameter such as packaging type has on the overall evolution of the microbial community. Depending on the process, we also show that specific actions on the pork meat (such as deboning and defatting) elicit specific food spoilers such as Dellaglioa algida, which becomes dominant during storage. Finally, ecological network reconstruction allowed us to map six different metabolic pathways involved in the production of volatile organic compounds involved in spoilage. We were able connect them to the different bacterial actors and to the influence of packaging type in an overall view. For instance, our results demonstrate a new role of Vibrionaceae in isopropanol production, and of Latilactobacillus fuchuensis and Lactococcus piscium in methanethiol/disylphide production. We also highlight a possible commensal behavior between Leuconostoc carnosum and Latilactobacillus curvatus around 2,3-butanediol metabolism. We conclude that our holistic approach combined with large-scale multi-omic data was a powerful strategy to prioritize the role of production parameters, already known in the literature, that shape the evolution and/or the implementation of different meat spoilage scenarios.

Anomaly Detection Paradigm for Multivariate Time Series Data Mining for Healthcare

Time series data are significant, and are derived from temporal data, which involve real numbers representing values collected regularly over time. Time series have a great impact on many types of data. However, time series have anomalies. We introduce an anomaly detection paradigm called novel matrix profile (NMP) to solve the all-pairs similarity search problem for time series data in the healthcare. The proposed paradigm inherits the features from two state-of-the-art algorithms: Scalable Time series Anytime Matrix Profile (STAMP) and Scalable Time-series Ordered-search Matrix Profile (STOMP). The proposed NMP caches the output in an easy-to-access fashion for single- and multidimensional data. The proposed NMP can be used on large multivariate data sets and generates approximate solutions of high quality in a reasonable time. It is implemented on a Python platform. To determine its effectiveness, it is compared with the state-of-the-art matrix profile algorithms, i.e., STAMP and STOMP. The results confirm that the proposed NMP provides higher accuracy than the compared algorithms.

Optimal Data Reduction of Training Data in Machine Learning-Based Modelling: A Multidimensional Bin Packing Approach

In these days, when complex, IT-controlled systems have found their way into many areas, models and the data on which they are based are playing an increasingly important role. Due to the constantly growing possibilities of collecting data through sensor technology, extensive data sets are created that need to be mastered. In concrete terms, this means extracting the information required for a specific problem from the data in a high quality. For example, in the field of condition monitoring, this includes relevant system states. Especially in the application field of machine learning, the quality of the data is of significant importance. Here, different methods already exist to reduce the size of data sets without reducing the information value. In this paper, the multidimensional binned reduction (MdBR) method is presented as an approach that has a much lower complexity in comparison on the one hand and deals with regression, instead of classification as most other approaches do, on the other. The approach merges discretization approaches with non-parametric numerosity reduction via histograms. MdBR has linear complexity and can be facilitated to reduce large multivariate data sets to smaller subsets, which could be used for model training. The evaluation, based on a dataset from the photovoltaic sector with approximately 92 million samples, aims to train a multilayer perceptron (MLP) model to estimate the output power of the system. The results show that using the approach, the number of samples for training could be reduced by more than 99%, while also increasing the model’s performance. It works best with large data sets of low-dimensional data. Although periodic data often include the most redundant samples and thus provide the best reduction capabilities, the presented approach can only handle time-invariant data and not sequences of samples, as often done in time series.

Variable targeting and reduction in large vector autoregressions with applications to workforce indicators

We develop statistical tools for time series analysis of large multivariate datasets, when a few core series are of principal interest and there are many potential auxiliary predictive variables. The methodology, based on Vector Autoregressions (VAR), handles the case where unrestricted fitting is precluded by a large number of series and a huge parameter space. In particular, we adopt a forecast error criterion and use Granger-causality tests in a sequential manner to build a VAR model that targets the main variables. This approach affects variable reduction (or equivalently, sparsity restrictions) in a computationally fast way that remains feasible for large dimensions. The search for the best model results in a VAR, fitted with a selection of supporting series, that has the best possible forecast performance with respect to the core variables. We apply the statistical methodology to model real Gross Domestic Product and the national Unemployment Rate, two time series widely monitored by economists and policy-makers, based on a large set of Quarterly Workforce Indicators comprising various major sectors of the economy and different measures of labor market conditions.

A semi-parametric, state-space compartmental model with time-dependent parameters for forecasting COVID-19 cases, hospitalizations and deaths.

Short-term forecasts of the dynamics of coronavirus disease 2019 (COVID-19) in the period up to its decline following mass vaccination was a task that received much attention but proved difficult to do with high accuracy. However, the availability of standardized forecasts and versioned datasets from this period allows for continued work in this area. Here, we introduce the Gaussian infection state space with time dependence (GISST) forecasting model. We evaluate its performance in one to four weeks ahead forecasts of COVID-19 cases, hospital admissions and deaths in the state of California made with official reports of COVID-19, Google’s mobility reports and vaccination data available each week. Evaluation of these forecasts with a weighted interval score shows them to consistently outperform a naive baseline forecast and often score closer to or better than a high-performing ensemble forecaster. The GISST model also provides parameter estimates for a compartmental model of COVID-19 dynamics, includes a regression submodel for the transmission rate and allows for parameters to vary over time according to a random walk. GISST provides a novel, balanced combination of computational efficiency, model interpretability and applicability to large multivariate datasets that may prove useful in improving the accuracy of infectious disease forecasts.

Improving Subsurface Characterisation with ‘Big Data’ Mining and Machine Learning

Large databases of legacy hydrocarbon reservoir and well data provide an opportunity to use modern data mining techniques to improve our understanding of the subsurface in the presence of uncertainty and improve predictability of reservoir properties. A data mining approach provides a way to screen dependencies in reservoir and fluid data and enable subsurface specialists to estimate absent properties in partial or incomplete datasets. This allows for uncertainty to be managed and reduced. An improvement in reservoir characterisation using machine learning results from the capacity of machine learning methods to detect and model hidden dependencies in large multivariate datasets with noisy and missing data. This study presents a workflow applied to a large basin-scale reservoir characterization database. The study aims to understand the dependencies between reservoir attributes in order to allow for predictions to be made to improve the data coverage. The machine learning workflow comprises the following steps: (i) exploratory data analysis; (ii) detection of outliers and data partitioning into groups showing similar trends using clustering; (iii) identification of dependencies within reservoir data in multivariate feature space with self-organising maps; and (iv) feature selection using supervised learning to identify relevant properties to use for predictions where data are absent. This workflow provides an opportunity to reduce the cost and increase accuracy of hydrocarbon exploration and production in mature basins.

Students’ informal statistical inferences through data modeling with a large multivariate dataset

ABSTRACT In today’s age of information, the use of data is very powerful in making informed decisions. Data analytics is a field that is interested in identifying and interpreting trends and patterns within big data to make data-driven decisions. We focus on informal statistical inference and data modeling as a means of developing students’ data analytics skills in school. In this study, we examine how students apply the data modeling process to draw informal inferences when exploring trends, patterns and relationships in a real dataset using technological tools, such as CODAP and Excel. We analyzed 17–18-year-old students’ written reports on their explorations of data supplied by third parties. Students used a variety of statistical measures and visualizations to account for variability in analyzing data. They tended to make statements with certainty in their inferences and predictions beyond the data. When the pattern in the data was uncertain, they were inclined to use contextual knowledge to remain certain in their claims.

Predictive geologic mapping from geophysical data using self-organizing maps: A case study from Baie Verte, Newfoundland, Canada

Self-organizing maps (SOMs) are a type of unsupervised artificial neural networks clustering tool. SOMs are used to cluster large multivariate data sets. They can identify patterns and trends in the geophysical maps of an area and generate proxy geology maps, known as remote predictive mapping. We have applied SOMs to magnetic, radiometric, and gravity data sets compiled from multiple modern and legacy data sources over the Baie Verte Peninsula, Newfoundland, Canada. The regional and local geologic maps available for this area and knowledge from numerous geologic studies has enabled the accuracy of SOM-based predictive mapping to be assessed. Proxy geology maps generated by primary clustering directly from the SOMs and secondary clustering using a k-means approach reproduced many geologic units identified by previous traditional geologic mapping. Of the combinations of data sets tested, the combination of magnetic data, primary radiometric data and their ratios, and Bouguer gravity data gave the best results. We found that using reduced-to-the-pole residual intensity or using the analytic signal as the magnetic data were equally useful. The SOM process was unaffected by gaps in the coverage of some of the data sets. The SOM results could be used as input into k-means clustering because this method requires no gaps in the data. The subsequent k-means clustering resulted in more meaningful proxy geology maps than were created by the SOM alone. In regions where the geology is poorly known, these proxy maps can be useful in targeting where traditional, on-the-ground geologic mapping would be most beneficial, which can be especially useful in parts of the world where access is difficult and expensive.

High Performance Multivariate Geospatial Statistics on Manycore Systems

Modeling and inferring spatial relationships and predicting missing values of environmental data are some of the main tasks of geospatial statisticians. These routine tasks are accomplished using multivariate geospatial models and the cokriging technique. The latter requires the evaluation of the expensive Gaussian log-likelihood function, which has impeded the adoption of multivariate geospatial models for large multivariate spatial datasets. However, this large-scale cokriging challenge provides a fertile ground for supercomputing implementations for the geospatial statistics community as it is paramount to scale computational capability to match the growth in environmental data coming from the widespread use of different data collection technologies. In this article, we develop and deploy large-scale multivariate spatial modeling and inference on parallel hardware architectures. To tackle the increasing complexity in matrix operations and the massive concurrency in parallel systems, we leverage low-rank matrix approximation techniques with task-based programming models and schedule the asynchronous computational tasks using a dynamic runtime system. The proposed framework provides both the dense and the approximated computations of the Gaussian log-likelihood function. It demonstrates accuracy robustness and performance scalability on a variety of computer systems. Using both synthetic and real datasets, the low-rank matrix approximation shows better performance compared to exact computation, while preserving the application requirements in both parameter estimation and prediction accuracy. We also propose a novel algorithm to assess the prediction accuracy after the online parameter estimation. The algorithm quantifies prediction performance and provides a benchmark for measuring the efficiency and accuracy of several approximation techniques in multivariate spatial modeling.

A nonparametric algorithm for automatic classification of large multivariate statistical data sets and its application

A nonparametric algorithm for automatic classification of large statistical data sets is proposed. The algorithm is based on a procedure for optimal discretization of the range of values of a random variable. A class is a compact group of observations of a random variable corresponding to a unimodal fragment of the probability density. The considered algorithm of automatic classification is based on the «compression» of the initial information based on the decomposition of a multidimensional space of attributes. As a result, a large statistical sample is transformed into a data array composed of the centers of multidimensional sampling intervals and the corresponding frequencies of random variables. To substantiate the optimal discretization procedure, we use the results of a study of the asymptotic properties of a kernel-type regression estimate of the probability density. An optimal number of sampling intervals for the range of values of one- and two-dimensional random variables is determined from the condition of the minimum root-mean square deviation of the regression probability density estimate. The results obtained are generalized to the discretization of the range of values of a multidimensional random variable. The optimal discretization formula contains a component that is characterized by a nonlinear functional of the probability density. An analytical dependence of the detected component on the antikurtosis coefficient of a one-dimensional random variable is established. For independent components of a multidimensional random variable, a methodology is developed for calculating estimates of the optimal number of sampling intervals for random variables and their lengths. On this basis, a nonparametric algorithm for the automatic classification is developed. It is based on a sequential procedure for checking the proximity of the centers of multidimensional sampling intervals and relationships between frequencies of the membership of the random variables from the original sample of these intervals. To further increase the computational efficiency of the proposed automatic classification algorithm, a multithreaded method of its software implementation is used. The practical significance of the developed algorithms is confirmed by the results of their application in processing remote sensing data.

Learning multivariate new physics

We discuss a method that employs a multilayer perceptron to detect deviations from a reference model in large multivariate datasets. Our data analysis strategy does not rely on any prior assumption on the nature of the deviation. It is designed to be sensitive to small discrepancies that arise in datasets dominated by the reference model. The main conceptual building blocks were introduced in D’Agnolo and Wulzer (Phys Rev D 99 (1), 015014. https://doi.org/10.1103/PhysRevD.99.015014. arXiv:1806.02350 [hep-ph], 2019). Here we make decisive progress in the algorithm implementation and we demonstrate its applicability to problems in high energy physics. We show that the method is sensitive to putative new physics signals in di-muon final states at the LHC. We also compare our performances on toy problems with the ones of alternative methods proposed in the literature.

An integrative machine learning approach to discovering multi-level molecular mechanisms of obesity using data from monozygotic twin pairs.

We combined clinical, cytokine, genomic, methylation and dietary data from 43 young adult monozygotic twin pairs (aged 22–36 years, 53% female), where 25 of the twin pairs were substantially weight discordant (delta body mass index > 3 kg m−2). These measurements were originally taken as part of the TwinFat study, a substudy of The Finnish Twin Cohort study. These five large multivariate datasets (comprising 42, 71, 1587, 1605 and 63 variables, respectively) were jointly analysed using an integrative machine learning method called group factor analysis (GFA) to offer new hypotheses into the multi-molecular-level interactions associated with the development of obesity. New potential links between cytokines and weight gain are identified, as well as associations between dietary, inflammatory and epigenetic factors. This encouraging case study aims to enthuse the research community to boldly attempt new machine learning approaches which have the potential to yield novel and unintuitive hypotheses. The source code of the GFA method is publically available as the R package GFA.

A review of machine learning applications in wildfire science and management

Artificial intelligence has been applied in wildfire science and management since the 1990s, with early applications including neural networks and expert systems. Since then, the field has rapidly progressed congruently with the wide adoption of machine learning (ML) methods in the environmental sciences. Here, we present a scoping review of ML applications in wildfire science and management. Our overall objective is to improve awareness of ML methods among wildfire researchers and managers, as well as illustrate the diverse and challenging range of problems in wildfire science available to ML data scientists. To that end, we first present an overview of popular ML approaches used in wildfire science to date and then review the use of ML in wildfire science as broadly categorized into six problem domains, including (i) fuels characterization, fire detection, and mapping; (ii) fire weather and climate change; (iii) fire occurrence, susceptibility, and risk; (iv) fire behavior prediction; (v) fire effects; and (vi) fire management. Furthermore, we discuss the advantages and limitations of various ML approaches relating to data size, computational requirements, generalizability, and interpretability, as well as identify opportunities for future advances in the science and management of wildfires within a data science context. In total, to the end of 2019, we identified 300 relevant publications in which the most frequently used ML methods across problem domains included random forests, MaxEnt, artificial neural networks, decision trees, support vector machines, and genetic algorithms. As such, there exists opportunities to apply more current ML methods — including deep learning and agent-based learning — in the wildfire sciences, especially in instances involving very large multivariate datasets. We must recognize, however, that despite the ability of ML models to learn on their own, expertise in wildfire science is necessary to ensure realistic modelling of fire processes across multiple scales, while the complexity of some ML methods such as deep learning requires a dedicated and sophisticated knowledge of their application. Finally, we stress that the wildfire research and management communities play an active role in providing relevant, high-quality, and freely available wildfire data for use by practitioners of ML methods.

Bilinear and trilinear modelling of three-way data obtained in two factor designed metabolomics studies

Metabolomic studies of biological samples using experimentally designed experiments at different levels produce large multivariate datasets which can be arranged in three-way data structures and modelled using bilinear and trilinear factor decomposition methods. The goal of these studies is the discovery of the hidden sources of data variability to facilitate their biochemical interpretation. In this paper, the relationship between the effects of the experimental design factors, the structure of the generated three-way datasets and their more appropriate modelling (bilinear or trilinear) are investigated. As example of study, the effects of the dose of a chemical drug on the changes over time in the concentration of lipids in multiple samples of a biological organism are investigated in detail. Different scenarios are considered depending on the type of effects and interactions between the experimental factors. The optimal data modelling results are obtained in case of having reproducible multiplicative effects between the experimental design factors, because in this case the data decomposition can be performed using a trilinear model and the correct lipid profiles are recovered. In the other data scenarios, even in the presence of only additive effects and no interaction between design factors, the correct recovery of the different lipid profiles describing the behavior of the system is not guaranteed and the subsequent rotation ambiguities associated to the bilinear model decompositions can still be present.

Insights from Self-Organizing Maps for Predicting Accessibility Demand for Healthcare Infrastructure

As urban populations grow worldwide, it becomes increasingly important to critically analyse accessibility—the ease with which residents can reach key places or opportunities. The combination of ‘big data’ and advances in computational techniques such as machine learning (ML) could be a boon for urban accessibility studies, yet their application in this field remains limited. In this study, we provided detailed predictions of healthcare accessibility across a rapidly growing city and related them to socio-economic factors using a combination of classical and modern data analysis methods. Using the City of Surrey (Canada) as a case study, we clustered high-resolution income data for 2016 and 2022 using principal component analysis (PCA) and a powerful ML clustering tool, the self-organising map (SOM). We then combined this with door-to-door travel times to hospitals and clinics, calculated using a simple open-source tool. Focusing our analysis on senior populations (65+ years), we found that higher income clusters are projected to become more prevalent across Surrey over our study period. Low income clusters have on average better accessibility to healthcare facilities than high income clusters in both 2016 and 2022. Population growth will be the biggest accessibility challenge in neighbourhoods with good existing access to healthcare, whereas income change (both positive and negative) will be most challenging in poorly connected neighbourhoods. A dual accessibility problem may arise in Surrey: first, large senior populations will reside in areas with access to numerous and close-by, clinics, putting pressure on existing facilities for specialised services. Second, lower-income seniors will increasingly reside in areas poorly connected to healthcare services, which may impact accessibility equity. We demonstrate that combining PCA and SOM clustering techniques results in novel insights for predicting accessibility at the neighbourhood level. This allows for robust planning policy recommendations to be drawn from large multivariate datasets.

Enhancing the Radiometric Map of Australia

The Radiometric Map of Australia shows the distribution of potassium (% K), uranium (ppm eU) and thorium (ppm eTh) over Australia. A suite of image enhancement and data integration techniques can be used to enhance the value of these data for both mineral exploration and environmental mapping. Gradient-enhanced ternary and pseudo-colour image enhancements are now routinely used for the presentation and interpretation of gridded radioelement data. Where digital elevation data are available, these colour representations can be draped over the elevation data to form 3D perspective views, or hill-shaded derivatives of the DEM can be embedded into the ternary imagery as an intensity component. These are useful because the radioelement response can then be interpreted within the context of the relative position of anomalous features in the landscape. However, subtle variations in the concentrations of K, U, and Th are best interpreted using the ratios of the radioelements. The U/Th and U2/Th ratios are important indicators of uranium mineralization. The Th/K ratio is widely used for the detection of several styles of mineralization associated with K alteration. For the interpretation of map data, classification and clustering methods can be used to assist pattern recognition and are useful for the rapid assessment of large multivariate datasets. Automatic edge detection procedures can be used to speed up the annotation of unit boundaries. Residual modelling techniques can be used to highlight potential anomalies in the data.

A Model for Large Multivariate Spatial Datasets

A Model for Large Multivariate Spatial Datasets

Self Organising Maps - A Case Study of Broken Hill

The principle aim of the research was to overcome the challenges faced by modern geophysical data analysts, particularly those working with large multivariate datasets using Self Organising Maps (SOM). SOM is an unsupervised learning technique for multivariate data, which works by taking multiple geophysical datasets for an area of interest, and integrating them to illustrate trends. Once developed, our method drastically lowered the time required for an analyst to examine and identify trends and relations across a broad range of geophysical, geochemical and other data layers. It also revealed hidden relations and distinct populations within correlated layers. Our study shows that SOM continues to be a powerful tool in accelerating the interpretation process. This includes the separation of features into distinct geological units, even without any preliminary map inputs to the SOM process. It also highlights SOM’s ability to highlight variation in cover, which has been identified as a key aspect moving forward in Australia’s mining future, when considering the vast expanses of Australia covered in sub cropping rock. In the future as data continue to grow and overlap, SOM will play an important role in highlighting these relations in soil cover and outcrop geology.

Computational method for discovery of biomarker signatures from large, complex data sets

We present an efficient method for identifying of reliable biomarker panels from large multivariate data sets that typically result from experiments that monitor changes in RNA, small molecule, or protein abundance. Our computational methodology is developed and validated on the toxicogenomics database Drug Matrix that in its largest category contains 1656 recognition targets, characterized by the toxicant, dose and time (or duration) of the exposure. We were able to recognize both individual experimental conditions (compound, dose and time combinations) and the cases where the values for dose and time variables fall within the intervals in the training data, but do not match the training data exactly. Inclusion of gene expression information for multiple organs improved accuracy of recognition. Inclusion of time response information into consideration allowed us to develop particularly accurate marker panels for a large number of targets: we were able to recognize 176 compounds (out of 316) at greater than 90% accuracy. The presented methodology has an immediate application for discovery of diagnostic biomarker panels for exposure to various toxicity hazards, and may also be useful for development of biological markers for medical applications.