Mining Large Datasets Research Articles

Secondary halo bias through cosmic time

Context. The spatial distribution of dark matter halos carries cosmological and astrophysical information. Cosmological information can be considered to be contained in the connection between halo main properties and the large-scale halo bias, while the astrophysical information would be encoded in the scaling relations between halo properties. The combination of these two contributions leads to the effect of secondary halo bias. Aims. Our goal is to measure the signal of secondary halo bias as a function of a variety of intrinsic and environmental halo properties and to characterize its statistical significance as a function of cosmological redshift. Methods. Using fixed and paired N-body simulations of dark-matter halos – the UNIT simulation – with masses above ∼1011 M⊙h−1 identified over a wide range of cosmological redshifts (0 < z < 5), we explored the behavior of the scaling relations among different halo properties. We included novel environmental properties based on the halo distribution as well as the underlying dark-matter field. We implemented an object-by-object estimator of large-scale effective bias and tested its validity against standard approaches. With a bias assigned to each tracer, we performed a statistical analysis aimed at characterizing the distribution of the bias and the signal of the secondary halo bias. Results. We show how the halo scaling relations linking direct probes of the halo potential well do not depend on the environment. On the contrary, links between the halo mass and the so-called set of secondary halo properties are sensitive to the cosmological environment, mainly to under-dense regions. We show that the signal of secondary bias is derived statistically from secondary correlations beyond the standard link to the halo mass. Conclusions. We show that the secondary bias arises through nonlocal and/or environmental properties related either to the halo distribution or to the properties of the underlying dark-matter field. In particular, properties such as the tidal field (a measure of the anisotropy of the density field) and the local Mach number (a measure of the local kinetic temperature of the halo distribution) generate the signals of the secondary bias with the highest significance. We propose applications of the assignment of individual bias for the generation of mock catalogs containing the signal of secondary bias, as well as a series of cosmological analyses aimed at mining large galaxy datasets.

A unique, ring-like radio source with quadrilateral structure detected with machine learning

ABSTRACT We report the discovery of a unique object in the MeerKAT Galaxy Cluster Legacy Survey (MGCLS) using the machine learning anomaly detection framework astronomaly. This strange, ring-like source is 30′ from the MGCLS field centred on Abell 209, and is not readily explained by simple physical models. With an assumed host galaxy at redshift 0.55, the luminosity (1025 W Hz−1) is comparable to powerful radio galaxies. The source consists of a ring of emission 175 kpc across, quadrilateral enhanced brightness regions bearing resemblance to radio jets, two ‘ears’ separated by 368 kpc, and a diffuse envelope. All of the structures appear spectrally steep, ranging from −1.0 to −1.5. The ring has high polarization (25 per cent) except on the bright patches (<10 per cent). We compare this source to the Odd Radio Circles recently discovered in ASKAP data and discuss several possible physical models, including a termination shock from starburst activity, an end-on radio galaxy, and a supermassive black hole merger event. No simple model can easily explain the observed structure of the source. This work, as well as other recent discoveries, demonstrates the power of unsupervised machine learning in mining large data sets for scientifically interesting sources.

Dual roles of hexokinase 2 in shaping microglial function by gating glycolytic flux and mitochondrial activity.

Microglia continuously survey the brain parenchyma and actively shift status following stimulation. These processes demand a unique bioenergetic programme; however, little is known about the metabolic determinants in microglia. By mining large datasets and generating transgenic tools, here we show that hexokinase 2 (HK2), the most active isozyme associated with mitochondrial membrane, is selectively expressed in microglia in the brain. Genetic ablation of HK2 reduced microglial glycolytic flux and energy production, suppressed microglial repopulation, and attenuated microglial surveillance and damage-triggered migration in male mice. HK2 elevation is prominent in immune-challenged or disease-associated microglia. In ischaemic stroke models, however, HK2 deletion promoted neuroinflammation and potentiated cerebral damages. The enhanced inflammatory responses after HK2 ablation in microglia are associated with aberrant mitochondrial function and reactive oxygen species accumulation. Our study demonstrates that HK2 gates both glycolytic flux and mitochondrial activity to shape microglial functions, changes of which contribute to metabolic abnormalities and maladaptive inflammation in brain diseases.

Country-wide solar power load profile for Germany 2015 to 2019: The impact of system curtailments on prediction models

A large (43824 h) country-wide solar power load profile (LP), solar irradiance and meteorological dataset (ten variables) for Germany covering years 2015 to 2019 is compiled and forecast with eight machine learning and deep learning (ML/DL) algorithms. Analysis reveals that system curtailments are likely responsible for some outlying predictions. The adaptive boosting (ADA) and random forest (RF) algorithms outperform convolutional neural networks and other algorithms in supervised prediction and forecasting tasks with the dataset. Once tuned with 2015 to 2018 data, ADA and RF forecast 2019 hourly data with root mean squared error (RMSE) of < 0.03. Similar forecasting accuracy is achieved using smaller datasets of three months of historical data from 2015 to 2018 to forecast hourly LP in each month of 2019. Forecasts for July 2019 are associated with the highest errors (RMSE = 0.41). The transparent open box (TOB) algorithm, because it can reveal details of data matching contributions to each of its forecasts, is used to data mine the July 2019 forecasts and conduct in-depth outlier analysis. It reveals nine extreme outliers that each overestimate LP based on irradiance and meteorological inputs. Analysis suggests that substantial overestimates leading to high RMSE for July 2019 are likely due to system curtailments. It is beneficial to combine less transparent (e.g., ADA and RF) with more transparent (e.g. TOB) ML algorithms to accurately forecast and data mine large solar power data sets. It also indicates that solar LP forecasts cannot always rely on irradiance and meteorological variables in isolation. There is a need to be mindful of system constraints and market conditions when predicting LP on a country-wide basis.

Algorithm of Apriori-Based Rural Tourism Driving Factors and Its System Optimization

Rural tourism, as a vital component of tourism, is critical to the development of rural economies, farmers’ income, rural civilization, new rural construction, and urban-rural interaction. Simultaneously, as the size and complexity of data sets grow larger, how to improve the efficiency of association rule algorithms for mining large data sets has become a hot topic in association rule mining. Rural tourism development that is cultural and creative not only contributes to rural revitalization, but also to the preservation and inheritance of rural culture. The Apriori algorithm is the most widely used and influential algorithm for mining Boolean association rules, and the majority of current algorithms are extensions of the Apriori algorithm. Demand, supply, marketing, and support forces of rural tourism, which are the core driving force of rural tourism development, are formed by the basic needs of each subsystem of rural tourism. One of the main methods is to promote the sustainable and healthy development of rural tourism in accordance with the nature, characteristics, and laws of rural tourism destination construction, in order to create a dynamic system for long-term development and establish a rural tourism development dynamic system. The study of rural tourism driving factors and their system optimization is proposed in this paper. The main tourism dynamic system is adopted by the association rule algorithm of Apriori, the driving factors of rural tourism development are analyzed in the paper, and the system optimization method is proposed, all based on the Apriori algorithm. In terms of support, the Apriori algorithm is 0.436 higher than the CD algorithm and 0.568 higher than the SVM algorithm, and the Apriori algorithm can greatly reduce database size and improve record reading speed. As a result, the findings of this paper can be used to improve the spatial layout of rural tourism and to develop urban-rural tourism.

Identifying High-Risk Phenotypes Using Three-Way Feature Interactions in Multiple Myeloma: An Analysis of Publically Available Mmrf Commpass Study

The introduction of next generation sequencing has exponentially increased the number of potential disease targets from a limited panel of probes to markers across the entire genome. Previous studies of NGS data in multiple myeloma have identified key features associated with outcome in additive models, typically showing involvement of TP53, gain of 1q, MYC translocations, and others.We modified the multi-factor dimensionality reduction method (MDR) method, a computationally efficient method for detecting non-linear patterns of gene-gene interactions in genome-wide association studies (GWAS) first proposed by Richie et al, to identify three-way interactions significantly associated with outcome in MM. Specifically, we altered the ranking strategy of the survival adapted Cox-MDR method of Lee et al from balanced accuracy (average of sensitivity and specificity) to the absolute sum of deviance residuals. Due to the high level of censoring (94 events in 559 cases), balanced accuracy ranking favored interactions that identified few high risk cases and failed to validate well on external data.The publically available MMRF CoMMpass trial database (release version IA10) was utilized as primary training set for our MDR method. We included in our comparisons 164 RNA-SEQ genes with significant association with overall survival, the 20 most abundant non-synonymous, non-IG variants, and 24 estimated whole chromosome arm copy number gains or loss calls from the long insert WGS data of 559 baseline cases with outcome and clinical data (median follow-up 1.8 years). We searched for all three-way interactions across 210 total features resulting in the ranking of over 1.5 million unique three-way interactions, adjusted for age, ISS, creatinine level, and treatment by modified MDR. The top three-way RNA-expression interactions from our discovery phase were validated on 540 cases with GEP microarray and clinical data from UAMS Total Therapy 2 and 3 with median follow-up of 4.66 years (publically available data from GSE2658, GSE31161, and GSE24080).Of the top 100 three-way gene expression interactions discovered that we could validate on Affymetrix GEP data, 84 were significant in the UAMS TT2 and TT3 data set (log rank p-value < 0.05). HMGB3, a proposed inhibitor of myeloid differentiation, and DDX51, known to be differentially methylated in ALL, were the most commonly involved genes in the top significant interactions across both data sets. Pathway analysis of the 150 most frequently observed genes across the top 5,000 interactions in CoMMpass revealed an enrichment of DNA replication and cell cycle associated genes: RFC5, MCM2, MCM3, MCM4, MCM6, CCNA2, CDK2, etc.Within the top interactions that included whole arm copy number variants, gain of 1q and loss of 13q were the most frequently observed. We note that the top three-way interactions with gain of 1q paired exclusively with ALYREF, a transcriptional promoter, and one additional gene. Of the top interactions that included a SNV, mutations of DIS3 and DNAH5 were the most frequently observed. Many of the top interactions including DIS3 mutations paired with HMGB3 or TRIB3, a negative regulator of NF-kB and sensitizer to apoptosis, and one additional gene.MDR models are capable of identifying complex interactions of features that are not strictly additive. We observed a variety of interaction models, and present below an example where presence of 2 or more features resulted in overall negative outcome while presence of 1 or fewer did not. This situation was observed for the interaction of high expression of HMGB3, DDX51, and PSMB2, a bortezomib sensitizer in MM cell lines, where high expression in 2 or more of these genes imparted a negative outcome in CoMMpass and TT2/TT3 data sets. In the TT2/TT3 data set, cases identified as high risk from this interaction were enriched for the PR subtype and GEP70 HR.This modified MDR method is an effective tool at mining large genomic data sets for all k -way interactions associated with outcome among binary or multi-level features, adjusted for clinical covariates. Our modifications to MDR method allow us to identify more relevant interactions and increase reliability in validation data. This method could potentially further knowledge of the complex relationships between key features in genomic landscape of MM by discovering novel interactions that identify high-risk phenotypes. [Display omitted] DisclosuresBergsagel:Phosplatin Therapeutics: Research Funding.

SOMEnv: An R package for mining environmental monitoring datasets by Self-Organizing Map and k-means algorithms with a graphical user interface

The Self-Organizing Map (SOM) algorithm belongs to the family of artificial neural networks. It is an unsupervised method that requires no a priori knowledge regarding experimental data classification. Further, it can deal with large datasets and non-linear problems, providing powerful visualization features for outcome exploration on 2D maps. For environmental pollution assessments other unsupervised techniques are widely used, such as principal component and hierarchical cluster analyses, but their application for mining large datasets and properly visualizing the results is limited, making them difficult to use for handling of large datasets obtained by high frequency environmental monitoring.This study presents an R package (SOMEnv) that allows non-expert users to elaborate by SOM algorithm environmental variables (pollutants and/or chemical physical properties) recorded with high frequency for a long monitoring period. Additionally, SOMEnv can also be used for elaborating small datasets derived from uneven sampling. All the calculations and outcome visualizations can be done using a graphical user interface (GUI), meaning that experience in R software coding is not necessary, and only a basic knowledge regarding the employed algorithm is needed to interpret the results.The benefits of the SOMEnv package are that (i) both the software environment and tool are freely available; (ii) it is able to handle large datasets; (iii) it provides heuristic rules for SOM initialization; (iv) it has a built-in GUI for performing calculations and visualizing the results.Moreover, it comes with a wide range of visualizations, several of which are dedicated to high frequency data monitoring. An example of application is presented. The package is freely available on the Comprehensive R Archive Network (CRAN) repository.

Hiding sensitive association rules using the optimal electromagnetic optimization method and a dynamic bit vector data structure

Hiding the association rules is one of the methods used to protect sensitive information in data-mining processes. Its goal is to transform the original dataset so that the support for, or the reliability of, sensitive rules is reduced below the minimum threshold. Then these sensitive rules cannot be exploited, while the rules that are non-sensitive can still be exploited normally. Many methods have been proposed for hiding the association rules. However, most of these methods are very slow and consume a large amount of storage space. Consequently, they are not suitable when mining large datasets. Recently, the electromagnetic field optimization (EFO4ARH) method was proposed, and it was found to hide the sensitive association rules better than the other methods. To increase mining efficiency further, this paper proposes a new workaround called EFODBV4ARH. This technique applies a dynamic bit vector data structure in combination with the electromagnetic field optimization method. Experimental results indicate that EFODBV4ARH is significantly more efficient than EFO4ARH.

Semi-supervised classification via simultaneous label and discriminant embedding estimation

Graph-based semi-supervised learning has recently proved to be a powerful paradigm for processing and mining large datasets. The main advantage relies on the fact that these methods can be useful in propagating a small set of known labels to a large set of unlabeled data. However, the lack of labeled data in real applications may affect the semi-supervised learning’s performance. This paper introduces a new semi-supervised framework for simultaneous linear feature extraction and label propagation. The proposed method simultaneously estimates a discriminant transformation, as well as the unknown label, by exploiting both labeled and unlabeled data. The learning model’s unknowns are to be estimated by integrating two types of graph-based smoothness constraints. The resulting semi-supervised model is expected to learn more discriminative information. Experiments are conducted on nine public image datasets. These experimental results show that the proposed method’s performance can be better than that of many state-of-the-art graph-based semi-supervised algorithms.

GeneHunt for rapid domain-specific annotation of glycoside hydrolases

The identification of glycoside hydrolases (GHs) for efficient polysaccharide deconstruction is essential for the development of biofuels. Here, we investigate the potential of sequential HMM-profile identification for the rapid and precise identification of the multi-domain architecture of GHs from various datasets. First, as a validation, we successfully reannotated >98% of the biochemically characterized enzymes listed on the CAZy database. Next, we analyzed the 43 million non-redundant sequences from the M5nr data and identified 322,068 unique GHs. Finally, we searched 129 assembled metagenomes retrieved from MG-RAST for environmental GHs and identified 160,790 additional enzymes. Although most identified sequences corresponded to single domain enzymes, many contained several domains, including known accessory domains and some domains never identified in association with GH. Several sequences displayed multiple catalytic domains and few of these potential multi-activity proteins combined potentially synergistic domains. Finally, we produced and confirmed the biochemical activities of a GH5-GH10 cellulase-xylanase and a GH11-CE4 xylanase-esterase. Globally, this “gene to enzyme pipeline” provides a rationale for mining large datasets in order to identify new catalysts combining unique properties for the efficient deconstruction of polysaccharides.

A Review on Mining Large Unstructured Datasets to Find Top-K Competitors

A Review on Mining Large Unstructured Datasets to Find Top-K Competitors

Distributed Mining of Contrast Patterns

In this paper we propose a novel algorithm for mining contrast patterns using a distributed, map-reduce like framework. Contrast patterns describe differences between contrasted data sets and have previously been used for building highly accurate classifiers. However, mining for contrast patterns is a computationally expensive task and existing algorithms are designed to run in a sequential manner on a single machine. Consequently, existing approaches are unable to handle dense, high volume and high dimensional databases. Our algorithm addresses this problem by partitioning the search-space for contrast patterns into small, independent units. These units can be mined in parallel, providing a scalable solution for mining large data sets. Using three different real-world data sets we test an implementation of our algorithm on a Spark cluster. Results of these tests indicate that our algorithm achieves a high-degree of parallelism and scalability.

A Survey on Issues of Decision Tree and Non-Decision Tree Algorithms

Decision tree and non-decision tree approaches are effectively used in many diverse areas such as speech recognition, radar signal classification, satellite signal classification, medical diagnosis, remote sensing, expert systems, and weather forecasting and so on. Even though classification has been studied widely in the past, many of the algorithms are designed only for memory-resident data, thus limiting their suitability for data mining large data sets. The volume of data in databases is growing to quite large sizes, both in the number of attributes, instances and class labels. Decision tree learning from a very huge set of records in a database is quite complex task and is usually a very slow process, which is often beyond the capabilities of existing computers. This paper is an attempt to summarize the proposed approaches, tools etc. for decision tree learning with emphasis on optimization of constructed trees and handling large datasets. Further, we also discussed and summarized various non-decision tree approaches like Neural Networks, Support Vector Machines, and Naive Bayes and so on.

Functional cooperation between KCa3.1 and TRPC1 channels in human breast cancer: Role in cell proliferation and patient prognosis.

Intracellular Ca2+ levels are important regulators of cell cycle and proliferation. We, and others, have previously reported the role of KCa3.1 (KCNN4) channels in regulating the membrane potential and the Ca2+ entry in association with cell proliferation. However, the relevance of KC3.1 channels in cancer prognosis as well as the molecular mechanism of Ca2+ entry triggered by their activation remain undetermined. Here, we show that RNAi-mediated knockdown of KCa3.1 and/or TRPC1 leads to a significant decrease in cell proliferation due to cell cycle arrest in the G1 phase. These results are consistent with the observed upregulation of both channels in synchronized cells at the end of G1 phase. Additionally, knockdown of TRPC1 suppressed the Ca2+ entry induced by 1-EBIO-mediated KCa3.1 activation, suggesting a functional cooperation between TRPC1 and KCa3.1 in the regulation of Ca2+ entry, possibly within lipid raft microdomains where these two channels seem to co-localize. We also show significant correlations between KCa3.1 mRNA expression and poor patient prognosis and unfavorable clinical breast cancer parameters by mining large datasets in the public domain. Together, these results highlight the importance of KCa3.1 in regulating the proliferative mechanisms in breast cancer cells as well as in providing a promising novel target in prognosis and therapy.

A Survey of approaches for mining large data sets

Data mining environment creates a lot of information, that should be investigated, examples must be removed from that to pick up learning. In this new period with blast of information both organized and unstructured, in the field of genomics, meteorology, science, ecological examination and numerous others, it has gotten to be hard to process, oversee and break down examples utilizing customary databases and architectures. Thus, a legitimate structural engineering ought to be comprehended to pick up information about the Big Data.This paper presents a review of various algorithms from necessary for handling such large data set. These algorithms define various structures and methods implemented to handle Big Data, also in the paper are listed various tool that were developed for analyzing them.

Accelerating Adverse Outcome Pathway Development Using Publicly Available Data Sources.

The adverse outcome pathway (AOP) concept links molecular perturbations with organism and population-level outcomes to support high-throughput toxicity (HTT) testing. International efforts are underway to define AOPs and store the information supporting these AOPs in a central knowledge base; however, this process is currently labor-intensive and time-consuming. Publicly available data sources provide a wealth of information that could be used to define computationally predicted AOPs (cpAOPs), which could serve as a basis for creating expert-derived AOPs in a much more efficient way. Computational tools for mining large datasets provide the means for extracting and organizing the information captured in these public data sources. Using cpAOPs as a starting point for expert-derived AOPs should accelerate AOP development. Coupling this with tools to coordinate and facilitate the expert development efforts will increase the number and quality of AOPs produced, which should play a key role in advancing the adoption of HTT testing, thereby reducing the use of animals in toxicity testing and greatly increasing the number of chemicals that can be tested.

Classification of Concept-Drifting Data Streams using Optimized Genetic Algorithm

Data Stream Mining is the process of extracting knowledge structures from continuous, rapid data records. In these applications, the main goal is to predict the class or value of new instances in the data stream given some knowledge about the class membership or values of previous instances in the data stream. Machine learning techniques can be used to learn this prediction task from labeled examples in an automated fashion.In many applications which are in non-stationary environments, the distribution underlying the instances or the rules underlying their labeling may change over time, i.e., the class or the target value to be predicted may change over time. This problem is referred to as Concept drift[8]. Evolutionary Computations like Genetic Algorithm is a strong rule based classification algorithm which is used for mining static small data sets and inefficient for large data streams. Evolutionary Algorithms are one of the population optimization techniques done by calculating fitness evaluation measures using gene reproduction, crossover, mutation and selection of the individual gene mechanisms. If the Genetic Algorithm can be made scalable and adaptable by reducing its I/O intensity, it will become an efficient and effective tool for mining large data sets like data streams.In this paper a scalable and adaptable online genetic algorithm is proposed to mine classification rules for the data streams with concept drifts. The results of the proposed method are comparable with the other standard methods which are used for mining the data streams.

Enhancing Map-Reduce Mechanism for BigData with Density-Based Clustering

Map-Reduce is software framework that allows certain type of parallelizable or distributable problems involving bulky data sets to be solve using computing clusters. This paper presents a hybrid Map-Reduce framework that gathers computations resources from different clusters and runs Map-Reduce jobs across them. The mechanism is realized using DBSCAN clustering algorithms among Map-Reduce, parallel processing framework over clusters. However, the instant accomplishment of algorithms undergoes from efficiency problem for higher execution time as well as inadequate memory. This paper presents an efficient DBSCAN clustering method for mining large datasets with apache Hadoop and Map-Reduce that will reduce time of accessible algorithms and dataset from the dissimilar location will work simultaneously from single node and find the appropriate outcome in distributed environment

Parallel Associative Classification Data Mining Frameworks Based MapReduce

Associative classification (AC) is a research topic that integrates association rules with classification in data mining to build classifiers. After dissemination of the Classification-based Association Rule algorithm (CBA), the majority of its successors have been developed to improve either CBA's prediction accuracy or the search for frequent ruleitems in the rule discovery step. Both of these steps require high demands in processing time and memory especially in cases of large training data sets or a low minimum support threshold value. In this paper, we overcome the problem of mining large training data sets by proposing a new learning method that repeatedly transforms data between line and item spaces to quickly discover frequent ruleitems, generate rules, subsequently rank and prune rules. This new learning method has been implemented in a parallel Map-Reduce (MR) algorithm called MRMCAR which can be considered the first parallel AC algorithm in the literature. The new learning method can be utilised in the different steps within any AC or association rule mining algorithms which scales well if contrasted with current horizontal or vertical methods. Two versions of the learning method (Weka, Hadoop) have been implemented and a number of experiments against different data sets have been conducted. The ground bases of the comparisons are classification accuracy and time required by the algorithm for data initialization, frequent ruleitems discovery, rule generation and rule pruning. The results reveal that MRMCAR is superior to both current AC mining algorithms and rule based classification algorithms in improving the classification performance with respect to accuracy.

Interactive analysis of large cancer copy number studies with Copy Number Explorer.

Summary: Copy number abnormalities (CNAs) such as somatically-acquired chromosomal deletions and duplications drive the development of cancer. As individual tumor genomes can contain tens or even hundreds of large and/or focal CNAs, a major difficulty is differentiating between important, recurrent pathogenic changes and benign changes unrelated to the subject’s phenotype. Here we present Copy Number Explorer, an interactive tool for mining large copy number datasets. Copy Number Explorer facilitates rapid visual and statistical identification of recurrent regions of gain or loss, identifies the genes most likely to drive CNA formation using the cghMCR method and identifies recurrently broken genes that may be disrupted or fused. The software also allows users to identify recurrent CNA regions that may be associated with differential survival.Availability and Implementation: Copy Number Explorer is available under the GNU public license (GPL-3). Source code is available at: https://sourceforge.net/projects/copynumberexplorer/Contact: scott.newman@emory.edu