Real Datasets Research Articles

Electrical resistivity tomography inversion combining deep variational autoencoder and stochastic adaptive sampling

Geophysical methods, such as electrical resistivity tomography (ERT), can be used to imaging the near-surface electrical resistivity as field measurements depend on the subsurface porosity, water saturation and fluid salinity. ERT has been widely applied to investigate mineral and groundwater resources, and in archaeological, environmental, and engineering studies. The prediction of subsurface electrical conductivity from ERT data requires solving a geophysical inverse problem. For near-surface characterization studies, this is often accomplished with deterministic inverse methods. These methods linearize the problem around an initial solution, and their smoothness depends on an imposed a priori spatial regularization term. Depending on this parameterization, these methodologies might struggle to capture the natural variability of the subsurface. Moreover, deterministic solutions have limited capabilities for uncertainty assessment. On the contrary, stochastic inverse methods can assess uncertainties by predicting multiple model realizations that fit similarly the recorded ERT data. However, they are often more computationally expensive than deterministic solutions. Deep learning algorithms based on deep generative models have been used to re-parametrize model and data spaces into low-dimensional domains and efficiently solve geophysical inverse problems. However, within this context, uncertainty assessment is challenging. We propose a deep convolutional variational autoencoder (VAE) coupled with stochastic adaptive optimization to perform stochastic ERT inversion. Geostatistical simulations of electrical resistivity are used as training data set of the VAE. After training, the VAE generates electrical resistivity models that reproduce the statistics and spatial continuity patterns of the training data set. Then, the VAE latent space is iteratively perturbed and updated with adaptive stochastic sampling based on the misfit between observed and predicted ERT data. The proposed methodology is illustrated in two-dimensional synthetic and real data sets to illustrate the ability of the proposed method to predict reliable electrical resistivity models while generating multiple possible scenarios for uncertainty assessment.

Defective regression models for cure rate data with competing risks

ABSTRACT In this paper, we propose a novel method for the analysis of cure rate data with competing risks using defective distributions. We develop two defective regression models for the analysis of competing risk data subjected to random right censoring. The proposed models enable us to estimate the cure fraction directly from the model. Simultaneously, we also estimate the regression parameters corresponding to each cause of failure using the method of maximum likelihood. We conduct a simulation study to evaluate the finite sample performance of the proposed estimators. The practical usefulness of the procedures is illustrated using two real-life data sets.

An ordered approach on geometric vitality function for risk measurement and its application in stock market analysis

In reliability analysis, the residual life of a component or system plays a key role in explaining a component's present condition as well as how long it will continue to function as intended under specific circumstances, thereby preventing premature failure and saving money. Based on this, Nair and Rajesh [Geometric vitality function and its applications to reliability. IAPQR Trans. 2000;25(1):1–8] proposed a new measure called the geometric vitality function that explains the system's failure pattern. In the present study, we have introduced the geometric vitality function in the context of order statistics by means of the relevation transform and also analysed the monotone and ordering properties of the proposed measure. The application of the proposed measure in the stock markets are explored in the present study. Finally, to estimate the proposed measure using a real life data set, a simple estimator has been introduced as well.

Comparative Analysis of Market Clearing Mechanisms for Peer-to-Peer Energy Market Based on Double Auction

This paper aims to develop an optimisation-based price bid generation mechanism for the sellers and buyers in a double-auction-aided peer-to-peer (P2P) energy trading market. With consumers being prosumers through the continuous adoption of distributed energy resources, P2P energy trading models offer a paradigm shift in energy market operation. Thus, it is essential to develop market models and mechanisms that can maximise the incentives for participation in the P2P energy market. In this sense, the proposed approach focuses on maximising profit at the sellers, as well as maximising cost savings at the buyers. The bids generated from the proposed approach are integrated with three different market clearing mechanisms, and the corresponding market clearing prices are compared. A numerical analysis is performed on a real-life dataset from Ausgrid to demonstrate the bids generated from sellers/buyers, as well as the associated market clearing prices throughout different months of the year. It can be observed that the market clearing prices are lower when the solar generation is higher. The statistical analysis demonstrates that all three market clearing mechanisms can achieve a consistent market clearing price within a range of 5 cents/kWh for 50% of the time when trading takes place.

On the Discretization of the Weibull-G Family of Distributions: Properties, Parameter Estimates, and Applications of a New Discrete Distribution

In this article, we introduce a new three-parameter distribution called the discrete Weibull exponential (DWE) distribution, based on the use of a discretization technique for the Weibull-G family of distributions. This distribution is noteworthy, as its probability mass function presents both symmetric and asymmetric shapes. In addition, its related hazard function is tractable, exhibiting a wide range of shapes, including increasing, increasing–constant, uniform, monotonically increasing, and reversed J-shaped. We also discuss some of the properties of the proposed distribution, such as the moments, moment-generating function, dispersion index, Rényi entropy, and order statistics. The maximum likelihood method is employed to estimate the model’s unknown parameters, and these estimates are evaluated through simulation studies. Additionally, the effectiveness of the model is examined by applying it to three real data sets. The results demonstrate that, in comparison to the other considered distributions, the proposed distribution provides a better fit to the data.

Online Learning and Decision Making Under Generalized Linear Model with High-Dimensional Data

We propose a minimax concave penalized multiarmed bandit algorithm under the generalized linear model (G-MCP-Bandit) for decision-makers facing high-dimensional data in an online learning and decision-making environment. We demonstrate that in the data-rich regime, the G-MCP-Bandit algorithm attains the optimal cumulative regret in the sample size dimension and a tight bound in the covariate dimension and the significant covariate dimension. In the data-poor regime, the G-MCP-Bandit algorithm maintains a tight regret upper bound. In addition, we develop a local linear approximation method, the two-step weighted Lasso procedure, to identify the minimax concave penalty (MCP) estimator for the G-MCP-Bandit algorithm when samples are not independent and identically distributed. Under this procedure, the MCP estimator can match the oracle estimator with high probability and converge to the true parameters at the optimal convergence rate. Finally, through experiments based on both synthetic and real data sets, we show that the G-MCP-Bandit algorithm outperforms other benchmarking algorithms in terms of cumulative regret and that the benefits of the G-MCP-Bandit algorithm increase in the data’s sparsity level and the size of the decision set. This paper was accepted by J. George Shanthikumar, big data analytics. Funding: This work was supported by the National Natural Science Foundation of China (NSFC) [Grants W2441021, 72371172, 72342023, and 71929101] and the National Science Foundation (NSF) [Grant DMS 1820702]. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2022.01557 .

Impact of Data Particle Divide Depth Level on Effectiveness of Hypergeometrical Divide Classifier

Gravitational classifiers belong to the supervised machine learning area, and the basic element that they process is a data particle. So far, many algorithms have been presented in the world literature. They focus on creating a data particle and determining its two important parameters - a centroid and a mass. Hypergeometrical Divide is one of the latest algorithms in this group, which focuses on reducing the amount of processing data and keeping relevant information. A proportion of data to information depends on the data particle divide depth level. Its properties and application potential have been researched, and this article is the next step of the work. The aim of the research described in this article was to determine relation of the depth level value of data particle divide to the effectiveness of the Hypergeometrical Divide algorithm. The research was conducted on 7 real data sets with different characteristics, applying methods and measures of evaluating artificial intelligence algorithms described in the literature. 63 measurements were performed. As a result, the effectiveness of the Hypergeometrical Divide method was defined at each of available data particle divide depth levels for each of used databases.

Almost exact recovery in noisy semi-supervised learning

Abstract Graph-based semi-supervised learning methods combine the graph structure and labeled data to classify unlabeled data. In this work, we study the effect of a noisy oracle on classification. In particular, we derive the maximum a posteriori (MAP) estimator for clustering a degree corrected stochastic block model when a noisy oracle reveals a fraction of the labels. We then propose an algorithm derived from a continuous relaxation of the MAP, and we establish its consistency. Numerical experiments show that our approach achieves promising performance on synthetic and real data sets, even in the case of very noisy labeled data.

Novelty detection for long-term diagnostic data with Gaussian and non-Gaussian disturbances using a support vector machine

Abstract In condition monitoring lack of properly balanced data sets with faulty and healthy cases makes proper condition recognition very challenging. In many cases, one may have good condition data only as the machine is unique and there is no other example. This issue is addressed by proposing a support vector machine (SVM) for novelty detection applied to health index (HI) data. In this scheme, the moving window approach has been utilised in which the simple statistical parameterisation of the data is carried out. Then the model in the multidimensional (mD) space is constructed whose shape is defined by an estimated hypersphere border. If the data lies inside the border then it can be used to re-train the model. Whereas if it is outside the border then it cannot be recognized as a healthy case. The size of the mD hypersphere (for m=2) describes the location of the good-condition data cloud as a potential feature. If the size of the data cloud is growing, it means more dispersion of the data. The efficiency of the method is tested on simulated and well-known real data sets having Gaussian and non-Gaussian disturbances.

Refining penalized ridge regression: a novel method for optimizing the regularization parameter in genomic prediction.

The popularity of genomic selection as an efficient and cost-effective approach to estimate breeding values continues to increase, due in part to the significant saving in phenotyping. Ridge regression is one of the most popular methods used for genomic prediction; however, its efficiency (in terms of prediction performance) depends on the appropriate tunning of the penalization parameter. In this paper we propose a novel, more efficient method to select the optimal penalization parameter for Ridge regression. We compared the proposed method with the conventional method to select the penalization parameter in 14 real data sets and we found that in 13 of these, the proposed method outperformed the conventional method and across data sets the gains in prediction accuracy in terms of Pearson's correlation was of 56.15%, with not-gains observed in terms of normalized mean square error. Finally, our results show evidence of the potential of the proposed method, and we encourage its adoption to improve the selection of candidate lines in the context of plant breeding.

A New Bivariate Survival Model: The Marshall-Olkin Bivariate Exponentiated Lomax Distribution with Modeling Bivariate Football Scoring Data

This paper focuses on applying the Marshall-Olkin approach to generate a new bivariate distribution. The distribution is called the bivariate exponentiated Lomax distribution, and its marginal distribution is the exponentiated Lomax distribution. Numerous attributes are examined, including the joint reliability and hazard functions, the bivariate probability density function, and its marginals. The joint probability density function and joint cumulative distribution function can be stated analytically. Different contour plots of the joint probability density function and joint reliability and hazard rate functions of the bivariate exponentiated Lomax distribution are given. The unknown parameters and reliability and hazard rate functions of the bivariate exponentiated Lomax distribution are estimated using the maximum likelihood method. Also, the Bayesian technique is applied to derive the Bayes estimators and reliability and hazard rate functions of the bivariate exponentiated Lomax distribution. In addition, maximum likelihood and Bayesian two-sample prediction are considered to predict a future observation from a future sample of the bivariate exponentiated Lomax distribution. A simulation study is presented to investigate the theoretical findings derived in this paper and to evaluate the performance of the maximum likelihood and Bayes estimates and predictors. Furthermore, the real data set used in this paper comprises the scoring times from 42 American Football League matches that took place over three consecutive independent weekends in 1986. The results of utilizing the real data set approve the practicality and flexibility of the bivariate exponentiated Lomax distribution in real-world situations, and the bivariate exponentiated Lomax distribution is suitable for modeling this bivariate data set.

A Threefold Approach for Enhancing Fuzzy Interpolative Reasoning: Case Study on Phishing Attack Detection Using Sparse Rule Bases

Fuzzy systems are powerful modeling systems for uncertainty applications. In contrast to traditional crisp systems, fuzzy systems offer the opportunity to extend the binary decision to continuous space, which could offer benefits for various application areas such as intrusion detection systems (IDSs), because of their ability to measure the degree of attacks instead of making a binary decision. Furthermore, fuzzy systems offer a suitable environment that is able to deal with uncertainty. However, fuzzy systems face a critical challenge represented by the sparse fuzzy rules. Typical fuzzy systems demand complete fuzzy rules in order to offer the required results. Additionally, generating complete fuzzy rules can be difficult due to many factors, such as a lack of knowledge base or limited data availability, such as in IDS applications. Fuzzy rule interpolation (FRI) was introduced to overcome this limitation by generating the required interpolation results in cases with sparse fuzzy rules. This work introduces a threefold approach designed to address the cases of missing fuzzy rules, which uses a few fuzzy rules to handle the limitations of missing fuzzy rules. This is achieved by finding the interpolation condition of neighboring fuzzy rules. This procedure was accomplished based on the concept of factors (which determine the degree to which each neighboring fuzzy rule contributes to the interpolated results, in cases of missing fuzzy rules). The evaluation procedure for the threefold approach was conducted using the following two steps: firstly, using the FRI benchmark numerical metrics, the results demonstrated the ability of the threefold approach to generate the required results for the various benchmark scenarios. Secondly, using a real-life dataset (phishing attacks dataset), the results demonstrated the effectiveness of the suggested approach to handle cases of missing fuzzy rules in the area of phishing attacks. Consequently, the suggested threefold approach offers an opportunity to reduce the number of fuzzy rules effectively and generate the required results using only a few fuzzy rules.

Cervical Cancer Prediction Based on Imbalanced Data Using Machine Learning Algorithms with a Variety of Sampling Methods

Cervical cancer affects a large portion of the female population, making the prediction of this disease using Machine Learning (ML) of utmost importance. ML algorithms can be integrated into complex, intelligent, agent-based systems that can offer decision support to resident medical doctors or even experienced medical doctors. For instance, an experienced medical doctor may diagnose a case but need expert support that related to another medical specialty. Data imbalance is frequent in healthcare data and has a negative influence on predictions made using ML algorithms. Cancer data, in general, and cervical cancer data, in particular, are frequently imbalanced. For this study, we chose a messy, real-life cervical cancer dataset available in the Kaggle repository that includes large amounts of missing and noisy values. To identify the best imbalanced technique for this medical dataset, the performances of eleven important resampling methods are compared, combined with the following state-of-the-art ML models that are frequently applied in predictive healtchare research: K-Nearest Neighbors (KNN) (with k values of 2 and 3), binary Logistic Regression (bLR), and Random Forest (RF). The studied resampling methods include seven undersampling methods and four oversampling methods. For this dataset, the imbalance ratio was 12.73, with a 95% confidence interval ranging from 9.23% to 16.22%. The obtained results show that resampling methods help improve the classification ability of prediction models applied to cervical cancer data. The applied oversampling techniques for handling imbalanced data generally outperformed the undersampling methods. The average balanced accuracy for oversampling was 77.44%, compared to 62.28% for undersampling. When detecting the minority class, oversampling achieved an average score of 60.80%, while undersampling scored 41.36%. The logistic regression classifier had the greatest impact on balanced techniques, while random forest achieved promising performance, even before applying balancing techniques. Initially, KNN2 outperformed KNN3 across all metrics, including balanced accuracy, for which KNN2 achieved 53.57%, compared to 52.71% for KNN3. However, after applying oversampling techniques, KNN3 significantly improved its balanced accuracy to 73.78%, while that of KNN2 increased to 63.89%. Additionally, KNN3 outperformed KNN2 in minority class performance, scoring 55.72% compared to KNN2’s 33.93%.

Characteristics of the GLLE Family of Lifetime Distributions, and Applications

ABSTRACT. A Generalized Log-Logistic (GLL) family of lifetime distributions is one in which any pair of distributions are related through a GLL transformation, for some (non-negative) value of the transformation parameter (the odds function of the second distribution is the power of the odds function of the first distribution). We consider GLL families generated from an exponential distribution. We derive some useful characteristics of these distributions; moreover, we discuss the hazard rates and Kullback-Leibler divergence, then illustrate the usefulness of this distribution family by fitting real data sets.

Two sample test for covariance matrices in ultra-high dimension

In this paper, we propose a new test for testing the equality of two population covariance matrices in the ultra-high dimensional setting that the dimension is much larger than the sizes of both of the two samples. Our proposed methodology relies on a data splitting procedure and a comparison of a set of well selected eigenvalues of the sample covariance matrices on the split data sets. Compared to the existing methods, our methodology is adaptive in the sense that (i). it does not require specific assumption (e.g., comparable or balancing, etc.) on the sizes of two samples; (ii). it does not need quantitative or structural assumptions of the population covariance matrices; (iii). it does not need the parametric distributions or the detailed knowledge of the moments of the two populations. Theoretically, we establish the asymptotic distributions of the statistics used in our method and conduct the power analysis. We justify that our method is powerful under weak alternatives. We conduct extensive numerical simulations and show that our method significantly outperforms the existing ones both in terms of size and power. Analysis of two real data sets is also carried out to demonstrate the usefulness and superior performance of our proposed methodology. An R package UHDtst is developed for easy implementation of our proposed methodology.

HFS-CSR: A hierarchical feature selection method based on correlation and structural redundancy

Feature selection (FS) is a crucial step in data engineering, as it not only saves time and resources for model training but also improves the prediction performance. In many realistic data sets, the feature space is formed by binary features which are related via generalization-specialization relationships (a.k.a., hierarchical feature space). These interdependencies among features, which can be described via a directed acyclic graph, present new challenges for classical FS methods. In this paper, a Hierarchical Feature Selection method based on Correlation and Structural Redundancy (HFS-CSR) is proposed to address it. In HFS-CSR, the correlation between features and the class label is measured based on Fisher Score whereas the structural redundancy among features are assessed quantitatively via hierarchical redundancy and depth penalties. The importance of features are determined by combining them adaptively for feature filtering. Experiments are conducted based on two kinds of typical data sets with hierarchical feature space: aging gene data sets and NLP data sets. The results demonstrate that HFS-CSR can achieve better performance compared with seven existing FS methods.

A novel mixed-integer linear programming formulation for continuous-time inventory routing

Inventory management, vehicle routing, and delivery scheduling decisions are simultaneously considered in the context of the inventory routing problem. This paper focuses on the continuous-time version of this problem where, unlike its more traditional discrete-time counterpart, the distributor is required to guarantee that inventory levels are maintained within the desired intervals at any moment of the planning horizon. In this work, we develop a compact mixed-integer linear programming formulation to model the continuous-time inventory routing problem. We further discuss means to expedite its solution process, including the adaptation of well-known rounded capacity inequalities to tighten the formulation in the context of a branch-and-cut algorithm. Through extensive computational studies on a suite of 90 benchmark instances from the literature, we show that our branch-and-cut algorithm outperforms the state-of-the-art approach. We also consider a new set of 63 instances adapted from a real-life dataset and show our algorithm’s practical value in solving instances with up to 20 customers to guaranteed optimality.

Simulating the Cellular Context in Synthetic Datasets for Cryo-Electron Tomography.

Cryo-electron tomography (cryo-ET) allows to visualize the cellular context at macromolecular level. To date, the impossibility of obtaining a reliable ground truth is limiting the application of deep learning-based image processing algorithms in this field. As a consequence, there is a growing demand of realistic synthetic datasets for training deep learning algorithms. In addition, besides assisting the acquisition and interpretation of experimental data, synthetic tomograms are used as reference models for cellular organization analysis from cellular tomograms. Current simulators in cryo-ET focus on reproducing distortions from image acquisition and tomogram reconstruction, however, they can not generate many of the low order features present in cellular tomograms. Here we propose several geometric and organization models to simulate low order cellular structures imaged by cryo-ET. Specifically, clusters of any known cytosolic or membrane-bound macromolecules, membranes with different geometries as well as different filamentous structures such as microtubules or actin-like networks. Moreover, we use parametrizable stochastic models to generate a high diversity of geometries and organizations to simulate representative and generalized datasets, including very crowded environments like those observed in native cells. These models have been implemented in a multiplatform open-source Python package, including scripts to generate cryo-tomograms with adjustable sizes and resolutions. In addition, these scripts provide also distortion-free density maps besides the ground truth in different file formats for efficient access and advanced visualization. We show that such a realistic synthetic dataset can be readily used to train generalizable deep learning algorithms.

A New Mixture of Two Components of Exponentiated Family with Applications to Real Life Data Sets

A New Mixture of Two Components of Exponentiated Family with Applications to Real Life Data Sets

Uncovering hidden alignments in two-dimensional point fields

The problem of mapping hidden alignments of points in data sets of two-dimensional points is of significant interest in many geoscience disciplines. In this paper, we revisit this issue and provide a new algorithm, insights, and results. The statistical significance of alignments is assessed by using percentile confidence intervals estimated by a Monte Carlo procedure in which important issues, such as the shape of the geometric support and the possible non-homogeneity of the point density (i.e., clustering effects), have been considered. The procedure is not limited to the simplest case of occurrence and the chance of triads (alignments of three points in a plane) but has been extended to k-ads with k arbitrarily large. The important issue of scale, when searching for point alignments, has also been taken into account. Case studies using synthetic and real data sets are provided to illustrate the methodology and the claims.