Associations In Large Datasets Research Articles

Improving Association Rules Accuracy in Noisy Domains Using Instance Reduction Techniques

Association rules’ learning is a machine learning method used in finding underlying associations in large datasets. Whether intentionally or unintentionally present, noise in training instances causes overfitting while building the classifier and negatively impacts classification accuracy. This paper uses instance reduction techniques for the datasets before mining the association rules and building the classifier. Instance reduction techniques were originally developed to reduce memory requirements in instance-based learning. This paper utilizes them to remove noise from the dataset before training the association rules classifier. Extensive experiments were conducted to assess the accuracy of association rules with different instance reduction techniques, namely: Decremental Reduction Optimization Procedure (DROP) 3, DROP5, ALL K-Nearest Neighbors (ALLKNN), Edited Nearest Neighbor (ENN), and Repeated Edited Nearest Neighbor (RENN) in different noise ratios. Experiments show that instance reduction techniques substantially improved the average classification accuracy on three different noise levels: 0%, 5%, and 10%. The RENN algorithm achieved the highest levels of accuracy with a significant improvement on seven out of eight used datasets from the University of California Irvine (UCI) machine learning repository. The improvements were more apparent in the 5% and the 10% noise cases. When RENN was applied, the average classification accuracy for the eight datasets in the zero-noise test enhanced from 70.47% to 76.65% compared to the original test. The average accuracy was improved from 66.08% to 77.47% for the 5%-noise case and from 59.89% to 77.59% in the 10%-noise case. Higher confidence was also reported in building the association rules when RENN was used. The above results indicate that RENN is a good solution in removing noise and avoiding overfitting during the construction of the association rules classifier, especially in noisy domains.

Detecting Unbiased Associations in Large Data Sets.

Maximal information coefficient (MIC) explores the associations between pairwise variables in complex relationships. It approaches the correlation by optimized partition on the axis. However, when the relationships meet special noise, MIC may overestimate the correlated value, which leads to the misidentification of the relationship without noiseless. In this article, a novel method of weighted information coefficient mean (WICM) is proposed to detect unbiased associations in large data sets. First, we mathematically analyze the cause of giving an abnormal correlation value to a noisy relationship. Then, the WICM is presented in two core steps. One is to detect the potential overestimation from the relationships with high value, and the other is to rectify the overestimation by calculating information coefficient mean instead of just selecting the maximum element in the characteristic matrix. Finally, experiments in functional relationships and real-world data relationships show that the overestimation can be solved by WICM with both feasibility and effectiveness.

Combining power of different methods to detect associations in large data sets.

Exploring the relationship between factors of interest is a fundamental step for further analysis on various scientific problems such as understanding the genetic mechanism underlying specific disease, brain functional connectivity analysis. There are many methods proposed for association analysis and each has its own advantages, but none of them is suitable for all kinds of situations. This brings difficulties and confusions to practitioner on which one to use when facing a real problem. In this paper, we propose to combine power of different methods to detect associations in large data sets. It goes as combining the weaker to be stronger. Numerical results from simulation study and real data applications show that our new framework is powerful. Importantly, the framework can also be applied to other problems. Availability: The R script is available at https://jiangdata.github.io/resources/DM.zip.

Detecting Three‐Dimensional Associations in Large Data Set

The associations detection among variables in the large dataset is recently important due to the rapid growth rate of data. The interested associations can provide references for solving the problems such as dimension reduction and feature selection. Many methods have done on the associations detection of pairwise variables. The multi-dimensional variables, especially three-dimensional variables, is rarely studied. The relationships among them cannot be revealed by the detection of pairwise variables methods. A new method of Maximal three-dimensional information coefficient (MTDIC) is proposed which is able to indicate the associations of three-dimensional variables. The correlation coefficient is calculated from the three-dimensional mutual information. The World Health Organization (WHO) data and the Tara data are selected to evaluate their associations. The experiment is verified by comparing the coefficient results with the Distance correlation (Dcor). The accurate association strength is obtained by an iterative optimization procedure on sorting descending order of coefficients. The MTDIC performs better than the Dcor in generality and equitability properties.

Big data analysis techniques to address polypharmacy in patients \u2013 a scoping review

BackgroundPolypharmacy is a key challenge in healthcare especially in older and multimorbid patients. The use of multiple medications increases the potential for drug interactions and for prescription of potentially inappropriate medications. eHealth solutions are increasingly recommended in healthcare, with big data analysis techniques as a major component. In the following we use the term analysis of big data as referring to the computational analysis of large data sets to find patterns, trends, and associations in large data sets collected from a wide range of sources in contrast to using classical statistics programs. It is hypothesized that big data analysis is able to reveal patterns in patient data that would not be identifiable using conventional methods of data analysis. The aim of this review was to evaluate whether there are existing big data analysis techniques that can help to identify patients consuming multiple drugs and to assist in the reduction of polypharmacy in patients.MethodsA computerized search was conducted in February 2019 and updated in May 2020, using the PubMed, Web of Science and Cochrane Library databases. The search strategy was defined by the principles of a systematic search, using the PICO scheme. All studies evaluating big data analytics about patients consuming multiple drugs were considered. Two researchers assessed all search results independently to identify eligible studies. The data was then extracted into standardized tables.ResultsA total of 327 studies were identified through the database search. After title and abstract screening, 302 items were removed. Only three studies were identified as addressing big data analysis techniques in patients with polypharmacy. One study extracted antipsychotic polypharmacy data, the second introduced a decision support system to evaluate side-effects in patients with polypharmacy and the third evaluated a decision support system to identify polypharmacy-related problems in individuals.ConclusionsThere are few studies to date which have used big data analysis techniques for identification and management of polypharmacy. There may be a need to further explore interdisciplinary collaboration between computer scientists and healthcare professionals, to develop and evaluate big data analysis techniques that can be implemented to manage polypharmacy.

Machine-learning Algorithm to Predict Hypotension Based on High-fidelity Arterial Pressure Waveform Analysis.

WHAT THIS ARTICLE TELLS US THAT IS NEW: BACKGROUND:: With appropriate algorithms, computers can learn to detect patterns and associations in large data sets. The authors' goal was to apply machine learning to arterial pressure waveforms and create an algorithm to predict hypotension. The algorithm detects early alteration in waveforms that can herald the weakening of cardiovascular compensatory mechanisms affecting preload, afterload, and contractility. The algorithm was developed with two different data sources: (1) a retrospective cohort, used for training, consisting of 1,334 patients' records with 545,959 min of arterial waveform recording and 25,461 episodes of hypotension; and (2) a prospective, local hospital cohort used for external validation, consisting of 204 patients' records with 33,236 min of arterial waveform recording and 1,923 episodes of hypotension. The algorithm relates a large set of features calculated from the high-fidelity arterial pressure waveform to the prediction of an upcoming hypotensive event (mean arterial pressure < 65 mmHg). Receiver-operating characteristic curve analysis evaluated the algorithm's success in predicting hypotension, defined as mean arterial pressure less than 65 mmHg. Using 3,022 individual features per cardiac cycle, the algorithm predicted arterial hypotension with a sensitivity and specificity of 88% (85 to 90%) and 87% (85 to 90%) 15 min before a hypotensive event (area under the curve, 0.95 [0.94 to 0.95]); 89% (87 to 91%) and 90% (87 to 92%) 10 min before (area under the curve, 0.95 [0.95 to 0.96]); 92% (90 to 94%) and 92% (90 to 94%) 5 min before (area under the curve, 0.97 [0.97 to 0.98]). The results demonstrate that a machine-learning algorithm can be trained, with large data sets of high-fidelity arterial waveforms, to predict hypotension in surgical patients' records.

Robust dependence measure for detecting associations in large data set

In this paper, we proposed a new statistical dependency measure for two random vectors based on copula, called copula dependency coefficient (CDC). The CDC is proved to be robust to outliers and easy to be implemented. Especially, it is powerful and applicable to high-dimensional problems. All these properties make CDC practically important in related applications. Both experimental and application results show that CDC is a good robust dependence measure for association detecting.

Risks of Different Testosterone Preparations: Too Much, Too Little, Just Right.

tio [HR], 1.26; 95% CI, 1.18-1.35), hospitalization (HR, 1.16; 95% CI, 1.13-1.19), and death (HR, 1.34; 95% CI, 1.15-1.56) compared with testosterone gel or patch formulations. The outcomes of the analysis support the hypothesis that intermittent pharmacologic levels of testosterone, which are inevitable with an injectable depo-testosterone administration every 2 to 3 weeks, result in an association with an increase in the combined outcome of cardiovascular and cerebrovascular morbidity and death. The absolute risk between testosterone therapy groups was small in this epidemiologic analysis, but the risk does raise concern considering the marked increase in testosterone prescriptions both in the United States and internationally 5 and prior associations in large data sets of increased risks in men receiving vs those not receiving treatment. 3,4 In addition, Layton and colleagues 1 did not include a comparison group of men not receiving testosterone, so overall risks and benefits of the therapy cannot be compared between the groups. Layton et al 1 acknowledge the limitations of the study, including lack of information concerning the indication for treatment and only partial data concerning testosterone levels before initiating or during therapy. However, their findings are consistent with those of other large epidemiologic studies 3,4 and the recent alarming US Food and Drug Administration report that more than 25% of testosterone prescriptions in the United States are written without determination of a level and that more than 30% of patients receiving testosterone therapy do not have follow-up laboratory testing. 5 Taken together, these

Equitability, mutual information, and the maximal information coefficient.

How should one quantify the strength of association between two random variables without bias for relationships of a specific form? Despite its conceptual simplicity, this notion of statistical "equitability" has yet to receive a definitive mathematical formalization. Here we argue that equitability is properly formalized by a self-consistency condition closely related to Data Processing Inequality. Mutual information, a fundamental quantity in information theory, is shown to satisfy this equitability criterion. These findings are at odds with the recent work of Reshef et al. [Reshef DN, et al. (2011) Science 334(6062):1518-1524], which proposed an alternative definition of equitability and introduced a new statistic, the "maximal information coefficient" (MIC), said to satisfy equitability in contradistinction to mutual information. These conclusions, however, were supported only with limited simulation evidence, not with mathematical arguments. Upon revisiting these claims, we prove that the mathematical definition of equitability proposed by Reshef et al. cannot be satisfied by any (nontrivial) dependence measure. We also identify artifacts in the reported simulation evidence. When these artifacts are removed, estimates of mutual information are found to be more equitable than estimates of MIC. Mutual information is also observed to have consistently higher statistical power than MIC. We conclude that estimating mutual information provides a natural (and often practical) way to equitably quantify statistical associations in large datasets.

Detecting Novel Associations in Large Data Sets

Identifying interesting relationships between pairs of variables in large data sets is increasingly important. Here, we present a measure of dependence for two-variable relationships: the maximal information coefficient (MIC). MIC captures a wide range of associations both functional and not, and for functional relationships provides a score that roughly equals the coefficient of determination (R(2)) of the data relative to the regression function. MIC belongs to a larger class of maximal information-based nonparametric exploration (MINE) statistics for identifying and classifying relationships. We apply MIC and MINE to data sets in global health, gene expression, major-league baseball, and the human gut microbiota and identify known and novel relationships.

A Correlation for the 21st Century

A novel statistical approach has been developed that can uncover nonlinear associations in large data sets.