Large-scale Data Sources Research Articles

Incremental learning of system log formats

System logs come in a large and evolving variety of formats, many of which are semi-structured and/or non-standard. As a consequence, off-the-shelf tools for processing such logs often do not exist, forcing analysts to develop their own tools, which is costly and timeconsuming. In this paper, we present an incremental algorithm that automatically infers the format of system log files. From the resulting format descriptions, we can generate a suite of data processing tools automatically. The system can handle large-scale data sources whose formats evolve over time. Furthermore, it allows analysts to modify inferred descriptions as desired and incorporates those changes in future revisions.

Probabilistic Record Linkage of Infection Records and Death Registrations: A Tool to Strengthen Surveillance

An important element for many infectious disease surveillance programmes is their capacity to monitor not only the incidence of infection, but also the associated mortality. The ability to monitor post-infection mortality is dependent on outcome information being collected through the surveillance reports, or on infections being precisely specified on death certificates. For many infectious diseases, neither of these sources provides a reliable source of this information, so a method for linking infection and death registration data is needed. Given that surveillance data often lacks a unique patient identifier, a probabilistic record linkage method was developed to reliably bring together large-scale data sources to identify deaths following infection. The method was developed using Streptococcus pneumonia infection records but with wider applicability to other infectious disease surveillance programmes. Evaluation of the mechanism was undertaken by tracing patients through a central health service database. Results of the evaluation showed a positive predictive value of 97.7-99.8% for correctly identifying deaths following infection, and a negative predictive value of 90.2-98.0%. The successful application of probabilistic matching to link infections and death registrations paves the way for a new era in infectious disease surveillance in the UK, with its potential application to augment a wide array of ongoing surveillance programmes with information on patient outcome.

Automatic accuracy assessment via hashing in multiple-source environment

Accuracy is a most important data quality dimension and its assessment is a key issue in data management. Most of current studies focus on how to qualitatively analyze accuracy dimension and the analysis depends heavily on experts’ knowledge. Seldom work is given on how to automatically quantify accuracy dimension. Based on Jensen–Shannon divergence (JSD) measure, we propose accuracy of data can be automatically quantified by comparing data with its entity’s most approximation in available context. To quickly identify most approximation in large scale data sources, locality-sensitive hashing ( LSH) is employed to extract most approximation at multiple levels, namely column, record and field level. Our approach can not only give each data source an objective accuracy score very quickly as long as context member is available but also avoid human’s laborious interaction. As an automatic accuracy assessment solution in multiple-source environment, our approach is distinguished, especially for large scale data sources. Theory and experiment show our approach performs well in achieving metadata on accuracy dimension.

Data in the Field of Adult Education and Lifelong Learning: Present Situation, Improvements and Challenges

Over the last years, political and scientific debates have stressed the growing importance of adult education. Currently important research questions call not only for data sources that collect detailed information on adult education with repeated measurements and in different cohorts, but they should also include data on other life spheres such as education and working histories, partnership and household information, as well as competence development. In Germany, there are several large-scale datasets containing information on adult education. While general panel studies do not provide a systematic overview of educational activities of adults, studies focusing on adult education are either small-scale or cross-sectional and contain little context information. A study that covers information on all educational activities in the life course as well as repeated competence assessment is still missing. In part, these deficits will be resolved by large-scale longitudinal studies focused on adults and education that were either recently conducted or are currently prepared. Thus, we do not call for new data sources on adult education. What is far more important in the next years is analyzing the data of the new large-scale data sources thoroughly, but also developing new theoretical approaches to adult education.

A new paradigm: Data-aware scheduling in grid computing

Efficient and reliable access to large-scale data sources and archiving destinations in a widely distributed computing environment brings new challenges. The insufficiency of the traditional systems and existing CPU-oriented batch schedulers in addressing these challenges has yielded a new emerging era: data-aware schedulers. In this article, we discuss the limitations of the traditional CPU-oriented batch schedulers in handling the challenging data management problem of large-scale distributed applications; give our vision for the new paradigm in data-intensive scheduling; and elaborate on our case study: the Stork data placement scheduler.

OR and the management of electronic services

This review paper is meant to provide some insight to the relation between operations research and the management of electronic services. We use the term electronic services, within the broader e-business context, to refer to any service that is delivered to the customer via electronic means, mainly the Internet as a new channel to communicate with customers, but also through mobile and other digital communications. During the last years we have experienced an explosion in the development of electronic services, which is creating abundant opportunities for operations research (OR) applications, required to empower several types of technology-mediated interactions, exploit the large-scale digital data sources, and automate the recurring operational decisions. In this paper we summarise the various aspects in which operations research may support the management of electronic services, taking into account the different characteristics of alternative electronic media, such as the Internet, mobile communications, interactive television etc.

Welfare Costs of Inflation in a Menu Cost Model

Recent years have seen substantial prog? ress in our understanding of pricing decisions at the micro level. Access to large-scale data sources on prices of individual products has given us rich information on how frequently prices change, by what magnitude, and how the aggregation of price changes in individual firms maps into aggregate inflation (see, among many others, Mark Bils and Peter J. Klenow 2005). These facts about price movements are a key input in recent quantitative models of the aggre? gate effects of nominal rigidities (e.g., Mikhail Golosov and Robert E. Lucas 2007). In this paper, we use both the new facts and the new quantitative models to revisit an old question, namely, that of quantifying the welfare benefits of low inflation. Our model includes two

Investigation of expert rule bases, logistic regression, and non-linear machine learning techniques for predicting response to antiretroviral treatment

The extreme flexibility of the HIV type-1 (HIV-1) genome makes it challenging to build the ideal antiretroviral treatment regimen. Interpretation of HIV-1 genotypic drug resistance is evolving from rule-based systems guided by expert opinion to data-driven engines developed through machine learning methods. The aim of the study was to investigate linear and non-linear statistical learning models for classifying short-term virological outcome of antiretroviral treatment. To optimize the model, different feature selection methods were considered. Robust extra-sample error estimation and different loss functions were used to assess model performance. The results were compared with widely used rule-based genotypic interpretation systems (Stanford HIVdb, Rega and ANRS). A set of 3,143 treatment change episodes were extracted from the EuResist database. The dataset included patient demographics, treatment history and viral genotypes. A logistic regression model using high order interaction variables performed better than rule-based genotypic interpretation systems (accuracy 75.63% versus 71.74-73.89%, area under the receiver operating characteristic curve [AUC] 0.76 versus 0.68-0.70) and was equivalent to a random forest model (accuracy 76.16%, AUC 0.77). However, when rule-based genotypic interpretation systems were coupled with additional patient attributes, and the combination was provided as input to the logistic regression model, the performance increased significantly, becoming comparable to the fully data-driven methods. Patient-derived supplementary features significantly improved the accuracy of the prediction of response to treatment, both with rule-based and data-driven interpretation systems. Fully data-driven models derived from large-scale data sources show promise as antiretroviral treatment decision support tools.

Prioritization of gene regulatory interactions from large-scale modules in yeast

BackgroundThe identification of groups of co-regulated genes and their transcription factors, called transcriptional modules, has been a focus of many studies about biological systems. While methods have been developed to derive numerous modules from genome-wide data, individual links between regulatory proteins and target genes still need experimental verification. In this work, we aim to prioritize regulator-target links within transcriptional modules based on three types of large-scale data sources.ResultsStarting with putative transcriptional modules from ChIP-chip data, we first derive modules in which target genes show both expression and function coherence. The most reliable regulatory links between transcription factors and target genes are established by identifying intersection of target genes in coherent modules for each enriched functional category. Using a combination of genome-wide yeast data in normal growth conditions and two different reference datasets, we show that our method predicts regulatory interactions with significantly higher predictive power than ChIP-chip binding data alone. A comparison with results from other studies highlights that our approach provides a reliable and complementary set of regulatory interactions. Based on our results, we can also identify functionally interacting target genes, for instance, a group of co-regulated proteins related to cell wall synthesis. Furthermore, we report novel conserved binding sites of a glycoprotein-encoding gene, CIS3, regulated by Swi6-Swi4 and Ndd1-Fkh2-Mcm1 complexes.ConclusionWe provide a simple method to prioritize individual TF-gene interactions from large-scale transcriptional modules. In comparison with other published works, we predict a complementary set of regulatory interactions which yields a similar or higher prediction accuracy at the expense of sensitivity. Therefore, our method can serve as an alternative approach to prioritization for further experimental studies.

Election bloggers: Methods for determining political influence

This paper discusses early findings and methodological pitfalls of a study of a hyper–mediated election campaign in Canada. Web logs — or “blogs” — serve as the primary object of study. The paper focuses on possible sources of large scale data (aggregated blogger posts) and methods of determining the political influence of bloggers. A series of methodologies are proposed to resolve the over–reliance upon information aggregators and blog search engines provided by Google and Technorati.

Validation and interpretation of Web users’ sessions clusters

Understanding users’ navigation on the Web is important towards improving the quality of information and the speed of accessing large-scale Web data sources. Clustering of users’ navigation into sessions has been proposed in order to identify patterns and similarities which are then managed in the context of Web users oriented applications (searching, e-commerce, etc.). This paper deals with the problem of assessing the quality of user session clusters in order to make inferences regarding the users’ navigation behavior. A common model-based clustering algorithm is used to result in clusters of Web users’ sessions. These clusters are validated by using a statistical test, which measures the distances of the clusters’ distributions to infer their dissimilarity and distinguishing level. Furthermore, a visualization method is proposed in order to interpret the relation between clusters. Using real data sets, we illustrate how the proposed analysis can be applied in popular application scenarios to reveal valuable associations among Web users’ navigation sessions.

An assessment of machine and statistical learning approaches to inferring networks of protein-protein interactions

Abstract Protein-protein interactions (PPI) play a key role in many biological systems. Over the past few years, an explosion in availability of functional biological data obtained from high-throughput technologies to infer PPI has been observed. However, results obtained from such experiments show high rates of false positives and false negatives predictions as well as systematic predictive bias. Recent research has revealed that several machine and statistical learning methods applied to integrate relatively weak, diverse sources of large-scale functional data may provide improved predictive accuracy and coverage of PPI. In this paper we describe the effects of applying different computational, integrative methods to predict PPI in Saccharomyces cerevisiae. We investigated the predictive ability of combining different sets of relatively strong and weak predictive datasets. We analysed several genomic datasets ranging from mRNA co-expression to marginal essentiality. Moreover, we expanded an existing multi-source dataset from S. cerevisiae by constructing a new set of putative interactions extracted from Gene Ontology (GO)- driven annotations in the Saccharomyces Genome Database. Different classification techniques: Simple Naive Bayesian (SNB), Multilayer Perceptron (MLP) and K-Nearest Neighbors (KNN) were evaluated. Relatively simple classification methods (i.e. less computing intensive and mathematically complex), such as SNB, have been proven to be proficient at predicting PPI. SNB produced the “highest” predictive quality obtaining an area under Receiver Operating Characteristic (ROC) curve (AUC) value of 0.99. The lowest AUC value of 0.90 was obtained by the KNN classifier. This assessment also demonstrates the strong predictive power of GO-driven models, which offered predictive performance above 0.90 using the different machine learning and statistical techniques. As the predictive power of single-source datasets became weaker MLP and SNB performed better than KNN. Moreover, predictive performance saturation may be reached independently of the classification models applied, which may be explained by predictive bias and incompleteness of existing “Gold Standards”. More comprehensive and accurate PPI maps will be produced for S. cerevisiae and beyond with the emergence of largescale datasets of better predictive quality and the integration of intelligent classification methods.

E-Business and Management Science: Mutual Impacts (Part 1 of 2)

This begins a two-part commentary on management science and e-business, the theme of this two-part special issue. After explaining the topical clusters that give organization to both parts, we pose two key questions concerning the impact of the emerging digital economy on management science research: What fundamentally new research questions arise, and what kind of research enables progress on them. We sketch the papers appearing in this part from the perspective of both these questions, and offer summary comments on the first question based on the papers in both parts. The principal conclusion is that the digital economy is giving birth to new research questions in three main ways (not all independent): by enabling and popularizing several types of technology-mediated interactions, by spawning large-scale digital data sources, and by creating recurring operational decisions that need to be automated.