Thousands Of Machines Research Articles

Analysis of high volumes of network traffic for Advanced Persistent Threat detection

Advanced Persistent Threats (APTs) are the most critical menaces to modern organizations and the most challenging attacks to detect. They span over long periods of time, use encrypted connections and mimic normal behaviors in order to evade detection based on traditional defensive solutions. We propose an innovative approach that is able to analyze efficiently high volumes of network traffic to reveal weak signals related to data exfiltrations and other suspect APT activities. The final result is a ranking of the most suspicious internal hosts; this rank allows security specialists to focus their analyses on a small set of hosts out of the thousands of machines that typically characterize large organizations. Experimental evaluations in a network environment consisting of about 10K hosts show the feasibility and effectiveness of the proposed approach. Our proposal based on security analytics paves the way to novel forms of automatic defense aimed at early detection of APTs in large and continuously varying networked systems.

Energy and Makespan Tradeoffs in Heterogeneous Computing Systems using Efficient Linear Programming Techniques

Resource management for large-scale high performance computing systems pose difficult challenges to system administrators. The extreme scale of these modern systems require task scheduling algorithms that are capable of handling at least millions of tasks and thousands of machines. These large computing systems consume vast amounts of electricity leading to high operating costs. System administrators try to simultaneously reduce operating costs and offer state-of-the-art performance; however, these are often conflicting objectives. Highly scalable algorithms are necessary to schedule tasks efficiently and to help system administrators gain insight into energy/performance trade-offs of the system. System administrators can examine this trade-off space to quantify how much a difference in the performance level will cost in electricity, or analyze how much performance can be expected within an energy budget. In this study, we design a novel linear programming based resource allocation algorithm for a heterogeneous computing system to efficiently compute high quality solutions for simultaneously minimizing energy and makespan. These solutions are used to bound the Pareto front to easily trade-off energy and performance. The new algorithms are highly scalable in both solution quality and computation time compared to existing algorithms, especially as the problem size increases.

Analysis and Review the Data Using Big Data Hadoop

Big information is pool of huge and complicated information sets so it becomes tough to method information exploitation management tools. The term ‘Big Data’ illustrates innovative method and knowledge to capture, store, distribute, handle and evaluate petabyte or larger-sized datasets with high-speed and totally different structures. Huge knowledge may be structured, unstructured or semi-structured, leading to incapability of standard knowledge management ways. With the quick evolution of information, information storage and networking assortment capability, massive information area unit quickly growing altogether science and engineering domains. Knowledge is generated from numerous totally different sources and might arrive within the system at numerous rates. So as to method these giant amounts of information in a cheap and economical approach, similarity are employed. Huge knowledge may be knowledge whose scale, diversity, and quality need new design, techniques, algorithms, and analytics to manage it and extract price and hidden information from it. The analysis of huge information typically tough because it often involves assortment of mixed information supported completely different patterns or rules. The challenges embrace capture, storage, search, sharing, analysis, and visualization. The trend to massive information sets is owing to the additional info drawn from analysis of one large set of connected information, compared to separate smaller sets with constant total quantity of information. Massive data processing is that the ability of extracting helpful info from streams of information or datasets, that owing to its rate, variability and volume. This paper argues applications of huge processing model and conjointly massive data processing. Hadoop is that the core platform for structuring huge knowledge, and solves the matter of constructing it helpful for analytics functions. Hadoop is Associate in nursing open supply software system project that permits the distributed process of huge knowledge sets across clusters of goods servers. It’s designed to rescale from one server to thousands of machines, with a awfully high degree of fault tolerance.

Recent Developments on Security and Reliability in Large-Scale Data Processing with MapReduce

The demand to access to a large volume of data, distributed across hundreds or thousands of machines, has opened new opportunities in commerce, science, and computing applications. MapReduce is a paradigm that offers a programming model and an associated implementation for processing massive datasets in a parallel fashion, by using non-dedicated distributed computing hardware. It has been successfully adopted in several academic and industrial projects for Big Data Analytics. However, since such analytics is increasingly demanded within the context of mission-critical applications, security and reliability in MapReduce frameworks are strongly required in order to manage sensible information, and to obtain the right answer at the right time. In this paper, the authors present the main implementation of the MapReduce programming paradigm, provided by Apache with the name of Hadoop. They illustrate the security and reliability concerns in the context of a large-scale data processing infrastructure. They review the available solutions, and their limitations to support security and reliability within the context MapReduce frameworks. The authors conclude by describing the undergoing evolution of such solutions, and the possible issues for improvements, which could be challenging research opportunities for academic researchers.

Energy efficiency in big data complex systems: a comprehensive survey of modern energy saving techniques

Abstract The growing need of computation and processing has led to the generation of data centers. These data centers are usually comprised of hundreds of thousands of servers and other components. This complicated arrangement of the systems lead to the adoption of complex systems. Complex systems prevail in our society as combination of lots of entities, e.g., immune system, human brain and ecosystems. The adoption and interaction of the entities is possible through nonlinear interactions. The interaction between the components of the complex system is carried out in distributed fashion. Big data which is comprised of thousands of machines is also considered to be a form of complex adaptive systems which makes use of large entities, components and nonlinear interactions with each other. The development of such a complex systems raises certain challenges. Apart from management, energy is the most concerned one which is the core discussion of this research. This paper, surveys the state of the art on modern tools, techniques, architectures and algorithms which has been proposed and deployed to achieve energy efficiency in big data over the period of 2007–2015. We group existing approaches aimed at achieving energy efficiency in the complex paradigm of big data. In this categorization, we aim to provide an easy and concise view of the underlined model adapted by each approach in the context of big data.

The Evolution of Global Scale Filesystems for Scientific Software Distribution

Delivering complex software across a worldwide distributed system is a major challenge in high-throughput scientific computing. The problem arises at different scales for many scientific communities that use grids, clouds, and distributed clusters to satisfy their computing needs. For high-energy physics (HEP) collaborations dealing with large amounts of data that rely on hundreds of thousands of cores spread around the world for data processing, the challenge is particularly acute. To serve the needs of the HEP community, several iterations were made to create a scalable, user-level filesystem that delivers software worldwide on a daily basis. The implementation was designed in 2006 to serve the needs of one experiment running on thousands of machines. Since that time, this idea evolved into a new production global-scale filesystem serving the needs of multiple science communities on hundreds of thousands of machines around the world.

Scalable linear programming based resource allocation for makespan minimization in heterogeneous computing systems

Resource management for large-scale high performance computing systems poses difficult challenges to system administrators. The extreme scale of these modern systems require task scheduling algorithms that are capable of handling at least millions of tasks and thousands of machines. Highly scalable algorithms are necessary to efficiently schedule tasks to maintain the highest level of performance from the system. In this study, we design a novel linear programming based resource allocation algorithm for heterogeneous computing systems to efficiently compute high quality solutions for minimizing makespan. The novel algorithm tightly bounds the optimal makespan from below with an infeasible schedule and from above with a fully feasible schedule. The new algorithms are highly scalable in terms of solution quality and computation time as the problem size increases because they leverage similarity in tasks and machines. This novel algorithm is compared to existing algorithms via simulation on a few example systems.

Design and development of a medical big data processing system based on Hadoop.

Secondary use of medical big data is increasingly popular in healthcare services and clinical research. Understanding the logic behind medical big data demonstrates tendencies in hospital information technology and shows great significance for hospital information systems that are designing and expanding services. Big data has four characteristics--Volume, Variety, Velocity and Value (the 4 Vs)--that make traditional systems incapable of processing these data using standalones. Apache Hadoop MapReduce is a promising software framework for developing applications that process vast amounts of data in parallel with large clusters of commodity hardware in a reliable, fault-tolerant manner. With the Hadoop framework and MapReduce application program interface (API), we can more easily develop our own MapReduce applications to run on a Hadoop framework that can scale up from a single node to thousands of machines. This paper investigates a practical case of a Hadoop-based medical big data processing system. We developed this system to intelligently process medical big data and uncover some features of hospital information system user behaviors. This paper studies user behaviors regarding various data produced by different hospital information systems for daily work. In this paper, we also built a five-node Hadoop cluster to execute distributed MapReduce algorithms. Our distributed algorithms show promise in facilitating efficient data processing with medical big data in healthcare services and clinical research compared with single nodes. Additionally, with medical big data analytics, we can design our hospital information systems to be much more intelligent and easier to use by making personalized recommendations.

Hadoop, MapReduce and HDFS: A Developers Perspective

Abstract The applications running on Hadoop clusters are increasing day by day. This is due to the fact that organizations have found a simple and efficient model that works well in distributed environment. The model is built to work efficiently on thousands of machines and massive data sets using commodity hardware. HDFS and MapReduce is a scalable and fault-tolerant model that hides all the complexities for Big Data analytics. Since Hadoop is becoming increasingly popular, understanding technical details becomes essential. This fact inspired us to explore Hadoop and its components in-depth. The process of analysing, examining and processing huge amount of unstructured data to extract required information has been a challenge. In this paper we discuss Hadoop and its components in detail which comprise of MapReduce and Hadoop Distributed File System (HDFS). MapReduce engine uses JobTracker and TaskTracker that handle monitoring and execution of job. HDFS a distributed file-system which comprise of NameNode, DataNode and Secondary NameNode for efficient handling of distributed storage purpose. The details provided can be used for developing large scale distributed applications that can exploit computational power of multiple nodes for data and compute intensive applications.

Classification of Post-deployment Performance Diagnostic Techniques for Large-scale Software Systems

Today's large-scale software systems (LSSs) such as Facebook, Google, Amazon and many other contemporary datacenters comprise hundreds or thousands of machines running complex applications that require high availability and responsiveness. These LSSs must be carefully monitored for performance bottlenecks before a serious harm is done. Performance analysts have to deal with the tedious task of monitoring the performance of these LSSs to avoid any service level agreements (SLA) violations and to ensure their failure free operations. There do exist several post-deployment performance diagnostic (PPD) techniques for to help analysts diagnose performance problems in the field, i.e., after the software is deployed. However, there is no classification of the proposed PPD techniques to understand their objectives and characteristics. In this paper, we classify the existing PPD techniques along multiple categories. The classification of PPD techniques will provide a guideline for performance analysts and practitioners of LSS to choose techniques suitable for their need. Moreover, the classification will also help researcher understand and fill gaps, i.e., dedicate their research efforts to categories that have received little attention in the past.

On improvement of cloud virtual machine availability with virtualization fault tolerance mechanism

Virtualization, particularly in the field of cloud computing, is a common strategy to improve existing computing resources. Hadoop, one of the Apache projects, is designed to scale up from single servers to thousands of machines, each offering local computation and storage capabilities. However, how to guarantee both stability and reliability of virtualization have become important topics. In this article, to reach this goal we used current open-source software and platforms, for instance, the Xen-Hypervisor virtualization technology, and the OpenNebula virtual machines management tool. After extending components capabilities, we developed a mechanism to support our ideas and reached high availability with Hadoop that is also called as virtualization fault tolerance (VFT). We considered a practical problem, i.e., the single-point-of-failure issue that occurs frequently in virtualization systems, and the experimental results confirm that the downtime interval can be greatly shortened even if failure occurred. As a result, VFT is useful not only for Hadoop applications, but also for more areas in cluster-based systems.

Root cause detection in a service-oriented architecture

Large-scale websites are predominantly built as a service-oriented architecture. Here, services are specialized for a certain task, run on multiple machines, and communicate with each other to serve a user's request. An anomalous change in a metric of one service can propagate to other services during this communication, resulting in overall degradation of the request. As any such degradation is revenue impacting, maintaining correct functionality is of paramount concern: it is important to find the root cause of any anomaly as quickly as possible. This is challenging because there are numerous metrics or sensors for a given service, and a modern website is usually composed of hundreds of services running on thousands of machines in multiple data centers. This paper introduces MonitorRank, an algorithm that can reduce the time, domain knowledge, and human effort required to find the root causes of anomalies in such service-oriented architectures. In the event of an anomaly, MonitorRank provides a ranked order list of possible root causes for monitoring teams to investigate. MonitorRank uses the historical and current time-series metrics of each sensor as its input, along with the call graph generated between sensors to build an unsupervised model for ranking. Experiments on real production outage data from LinkedIn, one of the largest online social networks, shows a 26% to 51% improvement in mean average precision in finding root causes compared to baseline and current state-of-the-art methods.

Practical Compute Capacity Management for Virtualized Datacenters

We present CCM (Cloud Capacity Manager) - a prototype system and its methods for dynamically multiplexing the compute capacity of virtualized datacenters at scales of thousands of machines, for diverse workloads with variable demands. Extending prior studies primarily concerned with accurate capacity allocation and ensuring acceptable application performance, CCM also sheds light on the tradeoffs due to two unavoidable issues in large scale commodity datacenters: (i) maintaining low operational overhead given variable cost of performing management operations necessary to allocate resources, and (ii) coping with the increased incidences of these operations' failures. CCM is implemented in an industry-strength cloud infrastructure built on top of the VMware vSphere virtualization platform and is currently deployed in a 700 physical host datacenter. Its experimental evaluation uses production workload traces and a suite of representative cloud applications to generate dynamic scenarios. Results indicate that the pragmatic cloud-wide nature of CCM provides up to 25% more resources for workloads and improves datacenter utilization by up to 20%, compared to the common alternative approach of multiplexing capacity within multiple independent smaller datacenter partitions.

Blink and it's done

In this demonstration, we present BlinkDB, a massively parallel, sampling-based approximate query processing framework for running interactive queries on large volumes of data. The key observation in BlinkDB is that one can make reasonable decisions in the absence of perfect answers. BlinkDB extends the Hive/HDFS stack and can handle the same set of SPJA (selection, projection, join and aggregate) queries as supported by these systems. BlinkDB provides real-time answers along with statistical error guarantees, and can scale to petabytes of data and thousands of machines in a fault-tolerant manner. Our experiments using the TPC-H benchmark and on an anonymized real-world video content distribution workload from Conviva Inc. show that BlinkDB can execute a wide range of queries up to 150x faster than Hive on MapReduce and 10--150x faster than Shark (Hive on Spark) over tens of terabytes of data stored across 100 machines, all with an error of 2--10%.

SCOPE: parallel databases meet MapReduce

Companies providing cloud-scale data services have increasing needs to store and analyze massive data sets, such as search logs, click streams, and web graph data. For cost and performance reasons, processing is typically done on large clusters of tens of thousands of commodity machines. Such massive data analysis on large clusters presents new opportunities and challenges for developing a highly scalable and efficient distributed computation system that is easy to program and supports complex system optimization to maximize performance and reliability. In this paper, we describe a distributed computation system, Structured Computations Optimized for Parallel Execution (Scope), targeted for this type of massive data analysis. Scope combines benefits from both traditional parallel databases and MapReduce execution engines to allow easy programmability and deliver massive scalability and high performance through advanced optimization. Similar to parallel databases, the system has a SQL-like declarative scripting language with no explicit parallelism, while being amenable to efficient parallel execution on large clusters. An optimizer is responsible for converting scripts into efficient execution plans for the distributed computation engine. A physical execution plan consists of a directed acyclic graph of vertices. Execution of the plan is orchestrated by a job manager that schedules execution on available machines and provides fault tolerance and recovery, much like MapReduce systems. Scope is being used daily for a variety of data analysis and data mining applications over tens of thousands of machines at Microsoft, powering Bing, and other online services.

ICanCloud: A Flexible and Scalable Cloud Infrastructure Simulator

Simulation techniques have become a powerful tool for deciding the best starting conditions on pay-as-you-go scenarios. This is the case of public cloud infrastructures, where a given number and type of virtual machines (in short VMs) are instantiated during a specified time, being this reflected in the final budget. With this in mind, this paper introduces and validates iCanCloud, a novel simulator of cloud infrastructures with remarkable features such as flexibility, scalability, performance and usability. Furthermore, the iCanCloud simulator has been built on the following design principles: (1) it's targeted to conduct large experiments, as opposed to others simulators from literature; (2) it provides a flexible and fully customizable global hypervisor for integrating any cloud brokering policy; (3) it reproduces the instance types provided by a given cloud infrastructure; and finally, (4) it contains a user-friendly GUI for configuring and launching simulations, that goes from a single VM to large cloud computing systems composed of thousands of machines.

CAP and Cloud Data Management

Novel systems that scale out on demand, relying on replicated data and massively distributed architectures with clusters of thousands of machines, particularly those designed for real-time data serving and update workloads, amply illustrate the realities of the CAP theorem. The featured Web extra is a video interview with Yahoo's Raghu Ramakrishnan about CAP and the cloud.

Cluman: Advanced cluster management for the large-scale infrastructures

The recent uptake of multi-core computing has produced a rapid growth of virtualisation and cloud computing services. With the increased use of the many-core processors this trend will likely accelerate and computing centres will be faced with the management of the tens of thousands of the virtual machines. Furthermore, these machines will likely be geographically distributed and need to be allocated on demand. In order to cope with such complexity we have designed and developed an advanced cluster management system that can execute administrative tasks targeting thousands of machines as well as provide an interactive high-density visualisation of the fabrics. The job management subsystem can perform complex tasks while following their progress and output and report aggregated information back to the system administrators. The visualisation subsystem can display tree maps of the infrastructure elements with data and monitoring information, thus providing a very detailed overview of the large clusters at a glance. The initial experience with development and testing of the system will be presented as well as an evaluation of its performance.

An Adaptive and Scalable Replication Protocol on Power Smart Grids

Cloud based storage systems are known to provide high scalability and reliability overcoming the traditional constraints of static distributed systems. The processing capacity over thousands of machines makes this approach especially suitable for many environments. In particular, we focus on power networks. These systems are currently decentralizing their architectures due to the growth of renewable sources and the increasing power demand which are obstacles to the traditional power network radial distribution. This new decentralized architecture, which demands computing abilities for network monitoring and improving customer services, is denoted as power smart grid. This paper proposes a new scalable dynamic storage architecture and its associated replication protocol, with its correctness proof, aimed to store data with different consistency levels. In addition it is able to perform some parallel data computations adapting itself to the physical and dynamic power smart grid layout.

Design & Implementation of Secure AODV in Multicast Routing to Detect DDOS Attack

The wireless ad hoc network is particularly vulnerable to DOS attacks due to its features of open medium, dynamic changing topology, cooperative algorithms, decentralization of the protocols, and lack of a clear line of defense is a growing problem in networks today. In Mobile Ad hoc Networks (MANET), various types of Denial of Service Attacks (DOS) are possible because of the inherent limitations of its routing protocols. In this paper we will secure the MANET from the DDOS attack. DDOS attacks are similar to DOS attacks but there is a difference between them and that is DDOS attacks involve breaking in to hundreds or thousands of machines, so for this reason, this attack called Distributed. Very often, systems that use for attack is a part of the networks and users of these systems don’t know about that, their systems used for attack to another systems. This kind of attack, consume more bandwidth and uses more sources in network. . In this work, we study the effect of one of the important attacks that called DDOS in MANET on most vulnerability protocol that named AODV. The product of this study is detection of DDOS attack by using AODV (adhoc on demand distance vector) protocol. Proposed scheme is distributed in nature it has the capability to prevent Distributed DOS (DDOS) as well.