Large-scale Network Data Research Articles

Fractal multi-level organisation of human groups in a virtual world.

Humans are fundamentally social. They form societies which consist of hierarchically layered nested groups of various quality, size, and structure. The anthropologic literature has classified these groups as support cliques, sympathy groups, bands, cognitive groups, tribes, linguistic groups, and so on. Anthropologic data show that, on average, each group consists of approximately three subgroups. However, a general understanding of the structural dependence of groups at different layers is largely missing. We extend these early findings to a very large high-precision large-scale internet-based social network data. We analyse the organisational structure of a complete, multi-relational, large social multiplex network of a human society consisting of about 400,000 odd players of an open-ended massive multiplayer online game for which we know all about their various group memberships at different layers. Remarkably, the online players' society exhibits the same type of structured hierarchical layers as found in hunter-gatherer societies. Our findings suggest that the hierarchical organisation of human society is deeply nested in human psychology.

Variability of metabolite levels is linked to differential metabolic pathways in Arabidopsis's responses to abiotic stresses.

Constraint-based approaches have been used for integrating data in large-scale metabolic networks to obtain insights into metabolism of various organisms. Due to the underlying steady-state assumption, these approaches are usually not suited for making predictions about metabolite levels. Here, we ask whether we can make inferences about the variability of metabolite levels from a constraint-based analysis based on the integration of transcriptomics data. To this end, we analyze time-resolved transcriptomics and metabolomics data from Arabidopsis thaliana under a set of eight different light and temperature conditions. In a previous study, the gene expression data have already been integrated in a genome-scale metabolic network to predict pathways, termed modulators and sustainers, which are differentially regulated with respect to a biochemically meaningful data-driven null model. Here, we present a follow-up analysis which bridges the gap between flux- and metabolite-centric methods. One of our main findings demonstrates that under certain environmental conditions, the levels of metabolites acting as substrates in modulators or sustainers show significantly lower temporal variations with respect to the remaining measured metabolites. This observation is discussed within the context of a systems-view of plasticity and robustness of metabolite contents and pathway fluxes. Our study paves the way for investigating the existence of similar principles in other species for which both genome-scale networks and high-throughput metabolomics data of high quality are becoming increasingly available.

Optimization of Inter-network Bandwidth Resources for Large-Scale Data Transmission

In order to solve problems such as low resources utilization of data center backbone bandwidth and expensive daily expenses caused by the tidal effect in the transition of large-scale network data, the new optimization method of bandwidth resource for bulk data transfer is proposed. This method finishes the bulk data transfer by taking full advantage of the idle channel bandwidth. Firstly, the dynamic idle network bandwidth resource was switched into the static flow network; and then the max-min fair multi-commodity flow model is built on the basis of static network; finally, the iterative technique is used to solve the programming model for getting the maximum transport flows and the corresponding transmission path of bulk data transfer. The simulation method was used to test this optimization project. The results show that the optimization method proposed by the paper can significantly improve the utilization of data center backbone bandwidth resource, and the maximum amount of data transmission and the speed of data transfer have been obviously increased

Critical transitions in social network activity

A large variety of complex systems in ecology, climate science, biomedicine and engineering have been observed to exhibit tipping points, where the internal dynamical state of the system abruptly changes. For example, such critical transitions may result in the sudden change of ecological environments and climate conditions. Data and models suggest that detectable warning signs may precede some of these drastic events. This view is also corroborated by abstract mathematical theory for generic bifurcations in stochastic multi-scale systems. Whether the stochastic scaling laws used as warning signs are also present in social networks that anticipate a-priori {\it unknown} events in society is an exciting open problem, to which at present only highly speculative answers can be given. Here, we instead provide a first step towards tackling this formidable question by focusing on a-priori {\it known} events and analyzing a social network data set with a focus on classical variance and autocorrelation warning signs. Our results thus pertain to one absolutely fundamental question: Can the stochastic warning signs known from other areas also be detected in large-scale social network data? We answer this question affirmatively as we find that several a-priori known events are preceded by variance and autocorrelation growth. Our findings thus clearly establish the necessary starting point to further investigate the relation between abstract mathematical theory and various classes of critical transitions in social networks.

BANKSAFE: Visual analytics for big data in large-scale computer networks

The enormous growth of data in the last decades led to a wide variety of different database technologies. Nowadays, we are capable of storing vast amounts of structured and unstructured data. To address the challenge of exploring and making sense out of big data using visual analytics, the tight integration of such backend services is needed. In this article, we introduce BANKSAFE, which was built for the VAST Challenge 2012 and won the outstanding comprehensive submission award. BANKSAFE is based on modern database technologies and is capable of visually analyzing vast amounts of monitoring data and security-related datasets of large-scale computer networks. To better describe and demonstrate the visualizations, we utilize the Visual Analytics Science and Technology (VAST) Challenge 2012 as case study. Additionally, we discuss lessons learned during the design and development of BANKSAFE, which are also applicable to other visual analytics applications for big data.

Saturn: Range Queries, Load Balancing and Fault Tolerance in DHT Data Systems

In this paper, we present Saturn, an overlay architecture for large-scale data networks maintained over Distributed Hash Tables (DHTs) that efficiently processes range queries and ensures access load balancing and fault-tolerance. Placing consecutive data values in neighboring peers is desirable in DHTs since it accelerates range query processing; however, such a placement is highly susceptible to load imbalances. At the same time, DHTs may be susceptible to node departures/failures and high data availability and fault tolerance are significant issues. Saturn deals effectively with these problems through the introduction of a novel multiple ring, order-preserving architecture. The use of a novel order-preserving hash function ensures fast range query processing. Replication across and within data rings (termed vertical and horizontal replication) forms the foundation over which our mechanisms are developed, ensuring query load balancing and fault tolerance, respectively. Our detailed experimentation study shows strong gains in range query processing efficiency, access load balancing, and fault tolerance, with low replication overheads. The significance of Saturn is not only that it effectively tackles all three issues together - i.e., supporting range queries, ensuring load balancing, and providing fault tolerance over DHTs - but also that it can be applied on top of any order-preserving DHT enabling it to dynamically handle replication and, thus, to trade off replication costs for fair load distribution and fault tolerance.

Detection of traffic changes in large-scale backbone networks: The case of the Spanish academic network

Network management systems produce a huge amount of data in large-scale networks. For example, the Spanish academic network features hundreds of access and backbone links, each of which produces a link utilization time series. For the purpose of detecting relevant changes in traffic load a visual inspection of all such time series is required. As a result, the operational expenditure increases. In this paper, we present an on-line change detection algorithm to identify the relevant change points in link utilization, which are presented to the network manager through a graphical user interface. Consequently, the network manager only inspects those links that show a stationary and statistically significant change in the link load.

Regression Analysis of Asymmetric Pairs in Large-Scale Network Data

In social network analysis, many estimation methods have been developed over the past three decades. Due to the computational complexity for analyzing large-scale social network data, however, those methods cannot be applied effectively. On the other hand, the structure of large-scale network data is often sparse so that the information loss by ignoring symmetric pairs is rather limited. Hence, we propose an asymmetric pairs regression (APR) approach to study the social network relationship. Accordingly, the computation of parameter estimations is simple and the theoretical properties can be obtained via the established logistic regression model. Simulation studies and an empirical example are presented to illustrate the usefulness of APR.

Regression Analysis of Asymmetric Pairs in Large-Scale Network Data

In social network analysis, many estimation methods have been developed over the past three decades. Due to the computational complexity for analyzing large-scale social network data, however, those methods cannot be applied effectively. On the other hand, the structure of large-scale network data is often sparse so that the information loss by ignoring symmetric pairs is rather limited. Hence, we propose an asymmetric pairs regression (APR) approach to study the social network relationship. Accordingly, the computation of parameter estimations is simple and the theoretical properties can be obtained via the established logistic regression model. Simulation studies and an empirical example are presented to illustrate the usefulness of APR.

Sequential Influence Models in Social Networks

The spread of influence among individuals in a social network can be naturally modeled in a probabilistic framework, but it is challenging to reason about differences between various models as well as to relate these models to actual social network data. Here we consider two of the most fundamental definitions of influence, one based on a small set of "snapshot'' observations of a social network and the other based on detailed temporal dynamics. The former is particularly useful because large-scale social network data sets are often available only in snapshots or crawls. The latter however provides a more detailed process model of how influence spreads. We study the relationship between these two ways of measuring influence, in particular establishing how to infer the more detailed temporal measure from the more readily observable snapshot measure. We validate our analysis using the history of social interactions on Wikipedia; the result is the first large-scale study to exhibit a direct relationship between snapshot and temporal models of social influence.

Edward Elgar's Facebook Page

In just a handful of years, social networking tools have become part of our daily work and conversation. In just a handful of years, a tiny moment in human history, we have become a world of social networker. Social networking tools, as seen in the various Internet sites, are part of our daily work and conversation. In embracing these ideas, we are starting to forget that we did not always view society as a nexus of friends and connections. Our ability to recognize and analyze social networks matured in parallel with our ability to build and manage large-scale data networks.

Disconnects in presumed heterosexual HIV transmission in Malawi

Obtaining large-scale sexual network data is difficult [1] and doing so in a poor country is especially difficult. We congratulate Helleringer and Kohler [2] on their pioneering study of a sexual network on Likoma Island, Malawi. Their approach represents an important step toward appropriate research designs for elucidating HIV transmission dynamics in sub-Saharan Africa. Despite this achievement, their results undermine their assertion that the Likoma sexual network sustains epidemic HIV transmission. In fact, their findings add to the already considerable evidence against penile–vaginal intercourse as the primary driver of HIV transmission in sub-Saharan Africa [3]. In moderate prevalence Likoma, Helleringer and Kohler observed an ‘apparently paradoxical distribution of HIV prevalence’. They found that persons in the core (bicomponents) of the main component of the sexual network were less likely to be HIV infected than those in peripheral positions in the main component or in small network components. This finding is at odds with prior empirical research, which indicates that, when significant HIV transmission occurs, persons located in the core of the corresponding risk network are more likely to be infected. In one high prevalence community, persons in the core of the injection and sexual network had considerably higher HIV prevalence and more frequent risk behaviors than those in peripheral or more isolated positions [4]. In that study, injection and homosexual risk behaviors were positively associated with prevalent HIV infection, but heterosexual risk behaviors were not. Similarly, among heterosexuals in a very low prevalence setting with little ongoing HIV transmission, infected persons were, with one exception, isolates, in small components, or on the periphery of the main component of the risk network [5,6]. In Likoma, it is surprising that self-reported symptoms of sexually transmitted infections (STI) are almost evenly distributed across the sexual network, and are not concentrated in the core as expected. Furthermore, no sexual risk variable (condom use, number of sex partners, STI symptoms, or age difference between sex partners) shows a meaningful association with prevalent HIV infection. Taken together, these observations suggest that the measured sexual network might not underlie HIV transmission in Likoma, and that other unmeasured blood and sexual exposures might be involved. To establish the primary modes of HIV transmission in sub-Saharan Africa with confidence, more appropriate research designs are required. Specifically, it is critical to assess blood and sexual exposures comprehensively (beyond just injections and vaginal intercourse) for incident cases and controls, trace their corresponding contacts and sequence-infected persons' HIV isolates [7–9].

Mixture models and exploratory analysis in networks.

Networks are widely used in the biological, physical, and social sciences as a concise mathematical representation of the topology of systems of interacting components. Understanding the structure of these networks is one of the outstanding challenges in the study of complex systems. Here we describe a general technique for detecting structural features in large-scale network data that works by dividing the nodes of a network into classes such that the members of each class have similar patterns of connection to other nodes. Using the machinery of probabilistic mixture models and the expectation-maximization algorithm, we show that it is possible to detect, without prior knowledge of what we are looking for, a very broad range of types of structure in networks. We give a number of examples demonstrating how the method can be used to shed light on the properties of real-world networks, including social and information networks.

Efficient Algorithms for Detecting Signaling Pathways in Protein Interaction Networks

The interpretation of large-scale protein network data depends on our ability to identify significant substructures in the data, a computationally intensive task. Here we adapt and extend efficient techniques for finding paths and trees in graphs to the problem of identifying pathways in protein interaction networks. We present linear-time algorithms for finding paths and trees in networks under several biologically motivated constraints. We apply our methodology to search for protein pathways in the yeast protein-protein interaction network. We demonstrate that our algorithm is capable of reconstructing known signaling pathways and identifying functionally enriched paths and trees in an unsupervised manner. The algorithm is very efficient, computing optimal paths of length 8 within minutes and paths of length 10 in about three hours.

ExperNet: an intelligent multiagent system for WAN management

The authors describe ExperNet, an intelligent multiagent system developed to assist in managing large-scale data networks. The system assists network operators at various nodes of a WAN to detect and diagnose hardware failures and network traffic problems, suggesting the most feasible solution through a Web-based interface.