Large-scale Data Processing Systems Research Articles

BlockEdge: Building Bridges to Safe and Efficient Distributed Systems

Abstract: In the ever-evolving landscape of distributed systems, ensuring safety, efficiency, and scalability remains a paramount challenge. BlockEdge emerges as a pioneering framework designed to address these critical issues by leveraging the principles of blockchain technology and advanced consensus mechanisms. This abstract outlines the key features, innovations, and potential impacts of BlockEdge on the realm of distributed computing. BlockEdge integrates blockchain's immutable ledger properties with a novel consensus algorithm tailored for distributed systems. Unlike traditional blockchain applications that prioritize decentralization for financial transactions, BlockEdge focuses on enhancing the performance and reliability of distributed applications. The framework employs a hybrid consensus model that combines Byzantine Fault Tolerance (BFT) with Proof-of-Stake (PoS), optimizing both security and energy efficiency. A standout feature of BlockEdge is its modular architecture, which allows seamless interoperability between different types of distributed networks. This modularity facilitates the integration of various consensus protocols, catering to the specific needs of diverse applications, from IoT networks to large-scale data processing systems. By enabling secure and efficient cross-chain communication, BlockEdge effectively mitigates the silo effect prevalent in current distributed system designs.

Architecture Design and Optimization of Large-scale Data Processing Systems in Cloud Computing Environments

Architecture Design and Optimization of Large-scale Data Processing Systems in Cloud Computing Environments

Scaling Distributed Database Joins by Decoupling Computation and Communication

To process a large volume of data, modern data management systems use a collection of machines connected through a network. This paper proposes frameworks and algorithms for processing distributed joins—a compute- and communication-intensive workload in modern data-intensive systems. By exploiting multiple processing cores within the individual machines, we implement a system to process database joins that parallelizes computation within each node, pipelines the computation with communication, parallelizes the communication by allowing multiple simultaneous data transfers (send/receive). Our experimental results show that using only four threads per node the framework achieves a 3.5x gains in intra-node performance while compared with a single-threaded counterpart. Moreover, with the join processing workload the cluster-wide performance (and speedup) is observed to be dictated by the intra-node computational loads; this property brings a near-linear speedup with increasing nodes in the system, a feature much desired in modern large-scale data processing system.

TiSEFE: Time Series Evolving Fuzzy Engine for Network Traffic Classification

Monitoring and analyzing network traffic are very crucial in discriminating the malicious attack. As the network traffic is becoming big, heterogeneous, and very fast, traffic analysis could be considered as big data analytic task. Recent research in big data analytic filed has produces several novel large-scale data processing systems. However, there is a need for a comprehensive data processing system to extract valuable insights from network traffic big data and learn the normal and attack network situations. This paper proposes a novel evolving fuzzy system to discriminate anomalies by inspecting the network traffic. After capturing traffic data, the system analyzes it to establish a model of normal network situation. The normal situation is a time series data of an ordered sequence of traffic information variable values at equally spaced time intervals. The performance has been analyzed by carrying out several experiments on real-world traffic dataset and under extreme difficult situation of high-speed networks. The results have proved the appropriateness of time series evolving fuzzy engine for network classification.

Thinking with GDPR: A guide to better system design

Europe’s General Data Protection Regulation (GDPR) has a fearsome reputation as “the law that can fine you €20 million.” But behind that scary slogan lies a text that can be a very helpful guide to designing data processing systems. This paper explores that side of the GDPR: how understanding it can produce more effective - and more trustworthy - systems. Three popular myths often take designers down the wrong track: that GDPR is about stopping processing, is about users, and is about consent. Instead we consider, from a design perspective, the GDPR’s source material, its Principles, and its Lawful Bases for processing. Three examples - from the field of education, but widely applicable - show how “thinking with GDPR” has improved both the effectiveness and safety of large-scale data processing systems.

Cloud Computing Cloud Computing in Remote Sensing : High Performance Remote Sensing Data Processing in a Big data Environment

Multi-area and multi-faceted remote sensing (SAR) datasets are widely used due to the increasing demand for accurate and up-to-date information on resources and the environment for regional and global monitoring. In general, the processing of RS data involves a complex multi-step processing sequence that includes several independent processing steps depending on the type of RS application. The processing of RS data for regional disaster and environmental monitoring is recognized as computationally and data demanding.Recently, by combining cloud computing and HPC technology, we propose a method to efficiently solve these problems by searching for a large-scale RS data processing system suitable for various applications. Real-time on-demand service. The ubiquitous, elastic, and high-level transparency of the cloud computing model makes it possible to run massive RS data management and data processing monitoring dynamic environments in any cloud. via the web interface. Hilbert-based data indexing methods are used to optimally query and access RS images, RS data products, and intermediate data. The core of the cloud service provides a parallel file system of large RS data and an interface for accessing RS data from time to time to improve localization of the data. It collects data and optimizes I/O performance. Our experimental analysis demonstrated the effectiveness of our method platform.

Design analysis and Work load characteristics of a Micro data Center

Data centers are large-scale data storage and processing systems. It is made up of a number of servers that must be capable of handling large amount of data. As a result, data centers generate a significant quantity of heat, which must be cooled and kept at an optimal temperature to avoid overheating. To address this problem, thermal analysis of the data center is carried out using numerical methods. The CFD model consists of a micro data center, where conjugate heat transfer effects are studied. A micro data center consists of servers aligned with air gaps alternatively and cooling air is passed between the air gaps to remove heat. In the present work, the design of data center rack is made in such a way that the cold air is in close proximity to servers. The temperature and airflow in the data center are estimated using the model. The air gap is optimally designed for the cooling unit. Temperature distribution of various load configurations is studied. The objective of the study is to find a favorable loading configuration of the micro data center for various loads and effectiveness of distribution of load among the servers.

Research on Application of Computer Big Data in Mathematical Polymorphic Game Algorithm Theory

In large-scale game activities, the computer big data system is collected, and then large-scale real-time data collection, processing and control systems are used to bring mathematical algorithm models, and state information reliability algorithms are used to illustrate the system control machine design and its Implementation based on VMS. This article calculates the most by calculation. The main design ideas in the game mathematical model, the research found that the system controller proposed by the system is a method to solve the multi-state process synchronization problem in the Client/Server computing system, and can effectively deal with the algorithm application research in the model.

A Survey on Data-driven Performance Tuning for Big Data Analytics Platforms

Many research works deal with big data platforms looking forward to data science and analytics. These are complex and usually distributed environments, composed of several systems and tools. As expected, there is a need for a closer look at performance issues. In this work, we review performance tuning strategies in the big data environment. We focus on data-driven tuning techniques, discussing the use of database inspired approaches. Concerning big data and NoSQL stores, performance tuning issues are quite different from the so-called conventional systems. Many existing solutions are mostly ad-hoc activities that do not fit for multiple situations. But there are some categories of data-driven solutions that can be taken as guidelines and incorporated into general-purpose auto-tuning modules for big data systems. We examine typical performance tuning actions, discussing available solutions to support some of the tuning process's primary activities. We also discuss recent implementations of data-driven performance tuning solutions for big data platforms. We propose an initial classification based on the domain state-of-the-art and present selected tuning actions for large-scale data processing systems. Finally, we organized existing works towards self-tuning big data systems based on this classification and presented general and system-specific tuning recommendations. We found that most of the literature pieces evaluate the use of tuning actions at the physical design perspective, and there is a lack of self-tuning machine-learning-based solutions for big data systems.

A Survey on Spark Ecosystem: Big Data Processing Infrastructure, Machine Learning, and Applications

With the explosive increase of big data in industry and academic fields, it is necessary to apply large-scale data processing systems to analysis Big Data. Arguably, Spark is state of the art in large-scale data computing systems nowadays, due to its good properties including generality, fault tolerance, high performance of in-memory data processing, and scalability. Spark adopts a flexible Resident Distributed Dataset (RDD) programming model with a set of provided transformation and action operators whose operating functions can be customized by users according to their applications. It is originally positioned as a fast and general data processing system. A large body of research efforts have been made to make it more efficient (faster) and general by considering various circumstances since its introduction. In this survey, we aim to have a thorough review of various kinds of optimization techniques on the generality and performance improvement of Spark. We introduce Spark programming model and computing system, discuss the pros and cons of Spark, and have an investigation and classification of various solving techniques in the literature. Moreover, we also introduce various data management and processing systems, machine learning algorithms and applications supported by Spark. Finally, we make a discussion on the open issues and challenges for large-scale in-memory data processing with Spark.

Integration of large-scale data processing systems and traditional parallel database technology

In 2009 we explored the feasibility of building a hybrid SQL data analysis system that takes the best features from two competing technologies: large-scale data processing systems (such as Google MapReduce and Apache Hadoop) and parallel database management systems (such as Greenplum and Vertica). We built a prototype, HadoopDB, and demonstrated that it can deliver the high SQL query performance and efficiency of parallel database management systems while still providing the scalability, fault tolerance, and flexibility of large-scale data processing systems. Subsequently, HadoopDB grew into a commercial product, Hadapt, whose technology was eventually acquired by Teradata. In this paper, we provide an overview of HadoopDB's original design, and its evolution during the subsequent ten years of research and development effort. We describe how the project innovated both in the research lab, and as a commercial product at Hadapt and Teradata. We then discuss the current vibrant ecosystem of software projects (most of which are open source) that continued HadoopDB's legacy of implementing a systems level integration of large-scale data processing systems and parallel database technology.

DESIGN OF COMPREHENSIVE FRAMEWORK ON OPTIMIZATION METHODS IN DISTRIBUTED CLUSTERS

MapReduce is a popular, open source programming paradigm to handle big data which is an industry standard large scale data processing system used by many companies like Yahoo, Google, Facebook, etc. The YARN framework uses low resource fairness algorithms such as FIFO, Capacity, Fair, DRF scheduler, whereas these schedulers are not suitable for heterogeneous Hadoop clusters. Therefore, an Enhanced Combined Regression Ranking (eCRRYARN) algorithm was proposed to enhance resource fairness. The proposed algorithm uses linear regression model to estimate the expected resources to be availed by the tenants. The order ranking is given to the estimated resource and the resources shared as per the ranking provided. Hence, the Hierarchical Hadoop Cluster Resource Sharing (HHCRS) algorithm has been adopted for hierarchical distributed cluster aiming to design a cost effective cluster for organization which is spread across the globe.

The Transition from A Priori to A Posteriori Information: Bayesian Procedures in Distributed Large-Scale Data Processing Systems

The procedure of transition from a priori to a posteriori information for a linear experiment in the context of Big Data systems is considered. At first glance, this process is fundamentally sequential, namely: as a result of observation, a priori information is transformed into a posteriori information, which is later interpreted as a priori for the next observation, etc. It is shown that such a procedure can be parallelized and unified due to the transformation of both the measurement results and the original a priori information into some special type. The properties of various forms of information representation are studied and compared. This approach makes it possible to effectively scale the Bayesian estimation procedure and, thus, adapt it to the problems of processing large amounts of distributed data.

Broadband and scalable optical coupling for silicon photonics using polymer waveguides

Abstract We present optical coupling schemes for silicon integrated photonics circuits that account for the challenges in large-scale data processing systems such as those used for emerging big data workloads. Our waveguide based approach allows to optimally exploit the on-chip optical feature size, and chip- and package real-estate. It further scales well to high numbers of channels and is compatible with state-of-the-art flip-chip die packaging. We demonstrate silicon waveguide to polymer waveguide coupling losses below 1.5 dB for both the O- and C-bands with a polarisation dependent loss of <1 dB. Over 100 optical silicon waveguide to polymer waveguide interfaces were assembled within a single alignment step, resulting in a physical I/O channel density of up to 13 waveguides per millimetre along the chip-edge, with an average coupling loss of below 3.4 dB measured at 1310 nm.

SCARF: A container-based approach to cloud-scale digital forensic processing

The rapid growth of raw data volume requiring forensic processing has become one of the top concerns of forensic analysts. At present, there are no readily available solutions that provide: a) open and flexible integration of existing forensic tools into a processing pipeline; and b) scale-out architecture that is compatible with common cloud technologies.Containers, lightweight OS-level virtualized environments, are quickly becoming the preferred architectural unit for building large-scale data processing systems. We present a container-based software framework, SCARF, which applies this approach to forensic computations. Our prototype demonstrates its practicality by providing low-cost integration of both custom code and a variety of third-party tools via simple data interfaces. The resulting system fits well with the data parallel nature of most forensic tasks, which tend to have few dependencies that limit parallel execution.Our experimental evaluation shows that for several types of processing tasks–such as hashing, indexing and bulk processing–performance scales almost linearly with the addition of hardware resources. We show that the software engineering effort to integrate new tools is quite modest, and all the critical task scheduling and resource allocation are automatically managed by the container orchestration runtime–Docker Swarm, or similar.

PipsCloud: High performance cloud computing for remote sensing big data management and processing

Massive, large-region coverage, multi-temporal, multi-spectral remote sensing (RS) datasets are employed widely due to the increasing requirements for accurate and up-to-date information about resources and the environment for regional and global monitoring. In general, RS data processing involves a complex multi-stage processing sequence, which comprises several independent processing steps according to the type of RS application. RS data processing for regional environmental and disaster monitoring is recognized as being computationally intensive and data intensive.We propose pipsCloud to address these issues in an efficient manner, which combines recent cloud computing and HPC techniques to obtain a large-scale RS data processing system that is suitable for on-demand real-time services. Due to the ubiquity, elasticity, and high-level transparency of the cloud computing model, massive RS data management and data processing for dynamic environmental monitoring can all be performed on the cloud via Web interfaces. A Hilbert-R+-based data indexing method is employed for the optimal querying and access of RS images, RS data products, and interim data. In the core platform beneath the cloud services, we provide a parallel file system for massive high-dimensional RS data, as well as interfaces for accessing irregular RS data to improve data locality and optimize the I/O performance. Moreover, we use an adaptive RS data analysis workflow management system for on-demand workflow construction and the collaborative processing of a distributed complex chain of RS data, e.g., for forest fire detection, mineral resources detection, and coastline monitoring. Our experimental analysis demonstrated the efficiency of the pipsCloud platform.

Pull-based load distribution in large-scale heterogeneous service systems

The model is motivated by the problem of load distribution in large-scale cloud-based data processing systems. We consider a heterogeneous service system, consisting of multiple large server pools. The pools are different in that their servers may have different processing speed and/or different buffer sizes (which may be finite or infinite). We study an asymptotic regime in which the customer arrival rate and pool sizes scale to infinity simultaneously, in proportion to some scaling parameter $n$. Arriving customers are assigned to the servers by a "router", according to a {\em pull-based} algorithm, called PULL. Under the algorithm, each server sends a "pull-message" to the router, when it becomes idle; the router assigns an arriving customer to a server according to a randomly chosen available pull-message, if there are any, or to a random server, otherwise. Assuming sub-critical system load, we prove asymptotic optimality of PULL. Namely, as system scale $n\to\infty$, the steady-state probability of an arriving customer experiencing blocking or waiting, vanishes. We also describe some generalizations of the model and PULL algorithm, for which the asymptotic optimality still holds.

Extracting Functional Dependencies in Large Datasets Using MapReduce Model

Over the last few years, data are generated in large volume at a faster rate and there has been a remarkable growth in the need for large scale data processing systems. As data grows larger in size, data quality is compromised. Functional dependencies representing semantic constraints in data are important for data quality assessment. Executing functional dependency discovery algorithms on a single computer is hard and laborious with large data sets. MapReduce provides an enabling technology for large scale data processing. The open-source Hadoop implementation of MapReduce has provided researchers a powerful tool for tackling large-data problems in a distributed manner. The objective of this study is to extract functional dependencies between attributes from large datasets using MapReduce programming model. Attribute entropy is used to measure the inter attribute correlations, and exploited to discover functional dependencies hidden in the data.

BestPeer++: A Peer-to-Peer Based Large-Scale Data Processing Platform

The corporate network is often used for sharing information among the participating companies and facilitating collaboration in a certain industry sector where companies share a common interest. It can effectively help the companies to reduce their operational costs and increase the revenues. However, the inter-company data sharing and processing poses unique challenges to such a data management system including scalability, performance, throughput, and security. In this paper, we present BestPeer++, a system which delivers elastic data sharing services for corporate network applications in the cloud based on BestPeer—a peer-to-peer (P2P) based data management platform. By integrating cloud computing, database, and P2P technologies into one system, BestPeer++ provides an economical, flexible and scalable platform for corporate network applications and delivers data sharing services to participants based on the widely accepted pay-as-you-go business model. We evaluate BestPeer++ on Amazon EC2 Cloud platform. The benchmarking results show that BestPeer++ outperforms HadoopDB, a recently proposed large-scale data processing system, in performance when both systems are employed to handle typical corporate network workloads. The benchmarking results also demonstrate that BestPeer++ achieves near linear scalability for throughput with respect to the number of peer nodes.

A Hybrid System Engineering Approach for Engineered Resilient Systems: Combining Traditional and Agile Techniques to Support Future System Growth

Accommodating new requirements and user driven features is the major challenge of engineered resilient systems. An enterprise- centric system-engineering paradigm provides a holistic framework for guiding future growth of systems. This paradigm requires new processes to support such growth. Traditional system engineering approaches, while helpful in developing system structure and framework, are not adaptive to future growth. New user desired features or requirements can be added to legacy systems, but typically entail unfortunate tradeoffs. To meet new requirements, engineered resilient systems have adopted a hybrid approach of using traditional and agile system engineering. Traditional system engineering approaches handle original concept development and design requirements, while the specific implementation is agile. The hybrid approach presents some application issues such as abstraction (how much of the systems should be designed before agile implementation). Others include how to facilitate user features requiring major design changes; and what are acceptable tradeoffs. These issues are particularly problematic during verification and validation phases when a system fails to meet a particular requirement, which necessitates a design change. In this paper, we present a real world case study of a large-scale data processing and analysis system. Design and implementation of this system took place over two years using a mix of traditional and agile system engineering approaches. This paper presents the current best practices, customization, and the issues of abstraction, design agility, user accommodation, and tradeoffs while using hybrid system engineering approaches. We include lessons learned from the case study and suggest future guidelines.