Distributed Data Processing System Research Articles

WITHDRAWN: Efficient fragmentation and allocation on clustering in distributed environment (FACE)

Distributed data processing systems require a reliable storage structure that ensures consistent, high-scale, and scalable data access for all users. Data fragmentation and subsequent allocation are important aspects of a flexible distributed database system. Existing techniques may result in significant data loss while performing data allocation and fragmentation. To overcome these issues a novel fragmentation and Allocation on Clustering in Distributed Environment (FACE) is proposed for reliable storage with scalable data access for all users. Initially, the user query is given to distributed database system (DDBS), then the query is fed to the query parser. The query parser parses and converts a textual query into a query object. After parsing the query data, it is passed to the query optimizer, which optimizes the query to access the data more efficiently. In the next step, the query is fragmented using dual (horizontal and vertical) fuzzy C means clustering. In Dual fuzzy C means clustering, the vertical fragmentation focuses on clustering attribute values that are frequently accessed together, and horizontal fragmentation focuses on grouping records that meet data selection criteria. After fragmentation, the query is allocated using sequential rule mining, in which queries occur in sequential order. The allocated query is optimized using the Efficient Fragment Security (EFS) algorithm to normalize the parameters to achieve better results. The performance metrics of the proposed techniques are calculated based on Final Prediction Error (FPE), and Execution time. The proposed FACE achieves a lower execution time of the 1.56 s. The result shows that the proposed FACE decreases the overall execution time lower than the 4.70 s, 4.46 s, 3.3 Sec, and 3.14 s in MAPA, DMA, BBE, and RATS-HM respectively.

Method of planning data processing tasks in distributed systems with limited information about available resources

In today's digital landscape, distributed data processing systems (DDPs) are becoming increasingly critical to efficiently process, analyze, and manage large volumes of data. These systems are often used in commercial, scientific and social domains to process complex data in real-time or batch mode. One of the key components of such systems is task scheduling, which is an extremely complex process, particularly when information about resource requirements is not complete or accurate. The subject of research are algorithms, methods and approaches used for scheduling tasks between nodes in distributed systems. The purpose of the study is to create an optimized method of task planning in the RSOD with limited availability of information about available resources. The task of the research: to analyze the limitations of modern methods for scheduling tasks in distributed data processing systems (DDS); optimize the method of scheduling tasks based on metadata between RSOD nodes, based on the methodology of searching for nearest neighbors using the method of localized hashing and the algebra of finite predicates; develop the architecture of the software solution and its implementation based on the optimized method; test the algorithm on the example of a video decoding task. The following methods were used: statistical algorithms and techniques such as classification and cluster analysis were used to predict resource requirements, and visualization techniques assisted in the analysis and interpretation of results. As a result of the work: the limitations of modern methods for the distribution of tasks in distributed data processing systems (DDPs) were analyzed; an optimized method of task planning based on metadata in RSOD was created, based on the methodology of searching for nearest neighbors using the method of localized hashing and the algebra of finite predicates; the processes in the modified nearest neighbor search algorithm are detailed; the architecture of the software solution was developed, which integrates an optimized method of task planning based on metadata and resource allocation; validation of the software solution was carried out with the help of a practical scenario – the use of the created algorithm in the planning task for decoding video information. The conclusions of this study confirmed that the proposed method, based on the methodology of localized hashing and the use of finite redicate algebra, is effective even with insufficient or limited information about resource needs. This highlights the possibility of using dynamic scheduling strategies to adapt to changing load conditions and resource availability.

ReactiveFnJ: A choreographed model for Fork-Join Workflow in Serverless Computing

Function-as-a-Service (FaaS) is an event-based reactive programming model where functions run in ephemeral stateless containers for short duration. For building complex serverless applications, function composition is crucial to coordinate and synchronize the workflow of an application. Some serverless orchestration systems exist, but they are in their primitive state and do not provide inherent support for non-trivial workflows like, Fork-Join. To address this gap, we propose a fully serverless and scalable design model ReactiveFnJ for Fork-Join workflow. The intent of this work is to illustrate a design which is completely choreographed, reactive, asynchronous, and represents a dynamic composition model for serverless applications based on Fork-Join workflow. Our design uses two innovative patterns, namely, Relay Composition and Master-Worker Composition to solve execution time-out challenges. As a Proof-of-Concept (PoC), the prototypical implementation of Split-Sort-Merge use case, based on Fork-Join workflow is discussed and evaluated. The ReactiveFnJ handles embarrassingly parallel computations, and its design does not depend on any external orchestration services, messaging services, and queue services. ReactiveFnJ facilitates in designing fully automated pipelines for distributed data processing systems, satisfying the Serverless Trilemma in true essence. A file of any size can be processed using our effective and extensible design without facing execution time-out challenges. The proposed model is generic and can be applied to a wide range of serverless applications that are based on the Fork-Join workflow pattern. It fosters the choreographed serverless composition for complex workflows. The proposed design model is useful for software engineers and developers in industry and commercial organizations, total solution vendors and academic researchers.

A Network Load Perception Based Task Scheduler for Parallel Distributed Data Processing Systems

A Network Load Perception Based Task Scheduler for Parallel Distributed Data Processing Systems

A Scalable and Economical Method for Distributed Data Processing

This research paper presents a distributed data processing approach that involves the establishment of virtual machines, the creation of a distributed system, and the processing of data to obtain desired results. The proposed method aims to provide a simple and cost-effective solution for distributed data processing, with the ability to scale infrastructure according to the specific needs. Furthermore, a distributed data processing system is introduced, comprising virtual machines equipped with specialized software to facilitate the establishment of the distributed system. The method offers practical advantages in terms of implementation simplicity, reduced infrastructure costs, and improved resource utilization.

On the big data processing algorithms for finding frequent sequences

AbstractSequential pattern mining algorithms extract trendy sequence appearances inside ordered transactional datasets such as market basket datasets. There is a lack of research employing big data processing techniques to locate frequent sequences on large‐scale datasets. Furthermore, there is a need for optimized sequential pattern mining algorithms that run on ordered one‐dimensional sequences. We also observe a lack of sequential pattern search studies in the literature, where the focus is centered around multi‐dimensional data sequences. Existing approaches that deal with ordered one‐dimensional datasets suffer from scalability issues as the amount of data to be analyzed is enormous. This research investigates the big data processing techniques used to find frequent sequences in large‐scale datasets. It also proposes a scalable sequence pattern mining algorithm called Sequential Pattern Acquisition by Reducing Search Space (SPARSS) designed for distributed data processing systems that efficiently handle large datasets containing sequential one‐element data. It introduces a prototype implementation of SPARSS and provides information on the SPARSS's memory and time requirements, which were calculated as part of experimental studies on a real‐world dataset. The results confirm our expectations and demonstrate SPARSS's superior scalability and run‐time efficiency compared to other distributed algorithms.

Accounting, analysis and control in the conditions of applying information technologies

The subject of research is the accounting of analysis and control in the conditions of information technology. The purpose of the article is to determine the state and trends of the development of information technologies and automated accounting systems of scientific and applied importance. The methodological basis of the study is the general scientific and special methods of scientific cognition used at the empirical and theoretical levels of research: experiment, observation, comparison, systemic approach, analysis and synthesis, induction and deduction, abstraction. Results of work. The most promising means of technical support of automated accounting systems are proved - local and global electronic networks and distributed data processing systems. The principles and development of information technologies will accelerate the organization and content of the implementation of automated accounting systems, changes in its methods and technologies, as well as related scientific, theoretical and practical knowledge by scientists, practitioners and students of higher education in the development of algorithms. Accounting, financial and management reporting, performing the tasks of the accounting process in an integrated automated system. Field of application of results: at Ukrainian enterprises for accounting, analysis and control, as well as in the initial process of teaching the discipline "Information systems and technologies at the enterprise," "Information systems and technologies in accounting and taxation," "Information systems and technologies in finance". Conclusions. Every day, more and more information technologies are developing to facilitate the recording and processing of documents. Today, state bodies are increasingly using informatization of management processes and introducing them into their systems. Almost all government agencies use programs that save time on a unified approach to comparing information, reduce manual processing of data, allow the storage and use of data, and also allow the use of electronic services that significantly facilitate and simplify accounting and information systems.

Adaptive On-the-Fly Changes in Distributed Processing Pipelines.

Distributed data processing systems have become the standard means for big data analytics. These systems are based on processing pipelines where operations on data are performed in a chain of consecutive steps. Normally, the operations performed by these pipelines are set at design time, and any changes to their functionality require the applications to be restarted. This is not always acceptable, for example, when we cannot afford downtime or when a long-running calculation would lose significant progress. The introduction of variation points to distributed processing pipelines allows for on-the-fly updating of individual analysis steps. In this paper, we extend such basic variation point functionality to provide fully automated reconfiguration of the processing steps within a running pipeline through an automated planner. We have enabled pipeline modeling through constraints. Based on these constraints, we not only ensure that configurations are compatible with type but also verify that expected pipeline functionality is achieved. Furthermore, automating the reconfiguration process simplifies its use, in turn allowing users with less development experience to make changes. The system can automatically generate and validate pipeline configurations that achieve a specified goal, selecting from operation definitions available at planning time. It then automatically integrates these configurations into the running pipeline. We verify the system through the testing of a proof-of-concept implementation. The proof of concept also shows promising results when reconfiguration is performed frequently.

ANALYSIS OF THE EFFECTIVENESS OF CONSENSUS-BUILDING ALGORITHMS IN DISTRIBUTED DATA PROCESSING SYSTEMS BASED ON BLOCKCHAIN TECHNOLOGY

The authors of the article consider consensus-building algorithms that are central to the life cycle of transactional exchange between nodes of a DDPS (Distributed Data Processing System) based on blockchain technology. The term under consideration is formally defined. It is proved that when developing these algorithms, it is necessary to choose the main indicators and methods for calculating them, which can be used to evaluate their effectiveness. The process of false forking is described in detail, and the probability of its occurrence is calculated in order to assess the security of data stored in the distributed ledger. When analyzing existing consensus-building algorithms based on statistical data, it is shown that the Byzantine Fault Tolerance Delegated Proof-of-Stake (BFT-DPoS) algorithm has a higher level of efficiency not only in terms of security, but also in terms of performance compared to other algorithms. There is a lack of comprehensive evaluation of candidates for winning Registrar nodes that generate (mining) blocks and distribute them to verifier nodes. It is concluded that the elimination of the above-mentioned drawback would provide a more correct construction of the distributed ledger, since most of the existing algorithms are focused on applicability only in cryptocurrency systems and are not universal. To solve this problem, we propose the need to synthesize the BFT-DPoS and Proof-of-Importance algorithms in order to make it applicable to any type of DDPS using distributed ledger technology (blockchain).

Applying machine learning and parallel data processing for attack detection in IoT

Internet of Things (IoT) networks are kind of computer networks for which the problem of information security and, in particular, computer attack detection is acute. For solving this task the paper proposes a joint application of methods of machine learning and parallel data processing. The structure of basic classifiers is determined, which are designed for detecting the attacks in IoT networks, and a new approach to their combining is proposed. The statement of classification problem is formed in which the integral indicator of effectiveness is the ratio of accuracy to time of training and testing. For enhancing the speed of training and testing we propose the usage of the distributed data processing system Spark and multi-threaded mode. Moreover, a dataset pre-processing procedure is suggested, which leads to a significant reduction of the training sample volume. An experimental assessment of the proposed approach shows that the attack detection accuracy in IoT networks approaches 100 percent, and the speed of dataset processing increases in proportion to the number of parallel threads.

Інтелектуальна навчальна система підготовки диспетчерів управління повітряним рухом

В статті розглянуті основні методи організації інтелектуального навчального середовища підготовки диспетчерів управління повітряним рухом. Обґрунтована необхідність створення інтелектуальної навчальної системи в адаптивних тренажерах диспетчерів управління повітряним рухом. Інтелектуальні навчальні системи повинні базуватись на основі синтезу імітаційно-моделюючих комплексів у вигляді розподілених систем обробки даних для імітації поведінки середовища навчання. В роботі сформовані вимоги, що висуваються до побудови мультиагентного середовища інтелектуальної навчальної системи підготовки диспетчерів управління повітряним рухом. З метою забезпечення моделювання інтелектуального поводження об'єктів, що входять у віртуальне середовище навчання, запропоновано створення інтелектуальних об'єктів, як елементів мультиагентної системи з використанням методів планування дій. Запропоновано підхід удосконалення та розширення функціональних можливостей системи підготовки диспетчерів управління повітряним рухом. Запропонована архітектура інтелектуального агента навчальної системи підготовки диспетчерів управління повітряним рухом створена на базі елементів InterRRa архітектури, що забезпечує взаємодію агента з зовнішнім середовищем та іншими агентами через модель фізичного представлення об'єкта. Наведена математична модель інтелектуального агента в якій враховано можливість здійснення впливу на зовнішнє середовище. Розроблена модель мультиагентного середовища інтелектуальної навчальної системи яка забезпечує ідентифікацію ситуації в підсистемі підготовки і прийняття рішень, що виконує передачу управління на відповідний рівень ієрархії системи поводження інтелектуального агента. Особливістю розробленої структури мультиагентного середовища інтелектуальної навчальної системи є використання моделі поведінки інтелектуальних агентів, що забезпечують змінну поведінку і можливість рішення задач підготовки і прийняття рішень своїх подальших дій за допомогою різних моделей поведінки.

GF-3 data real-time processing method based on multi-satellite distributed data processing system

GF-3 data real-time processing method based on multi-satellite distributed data processing system

Speeding up AutoTuning of the Memory Management Options in Data Analytics

Many solutions used towards building autonomous (or, self-driving) data processing systems today are trying to leverage the “black box” algorithm of Bayesian Optimization (BO) both due to its wider applicability and the theoretical guarantees provided on the quality of results produced. The black-box approach, however, could be time and labor-intensive; or otherwise get stuck in a local minima. We study an important problem of auto-tuning the memory allocation for applications running on modern distributed data processing systems. A simple “white-box” model is developed which can quickly separate good configurations from bad ones. To combine the benefits of the two approaches to tuning, we build a framework called Guided Bayesian Optimization (GBO) that uses the white-box model as a guide during the Bayesian Optimization exploration process. An evaluation carried out on Apache Spark using industry-standard benchmark applications shows that GBO consistently provides performance speedups across the application workload with the magnitude of savings being close to 2x.

The Problem of the Optimal Placing of the Information-Technological Reserve in Distributed Data Processing Systems

It is noted, that information-technological reserve is a new type of information redundancy, the use of which in distributed automated informational control systems allows to increase the efficiency of their performance while processing the typical user requests. The task of optimal allocation of identical copies of information-technological reserve throughout the nodes of the distributed system is formulated in the form of a minisum problem of finding the p-median of a graph. The algorithm is proposed for solving this problem and the example of the solution is given. The brief analysis of the solution algorithm results is performed.

Creation of Information-Technological Reserve in Distributed Data Processing Systems

The main stages are presented of creating an information-technological data reserve in distributed automated informational control systems. The reserve is formed on the basis of the results of the analysis of subject areas, regular requests to the system and the analysis of the execution sequence of requests processing procedures. The tasks are set of the reserve structure design and of the optimal allocation of reserve copies among computer network nodes. It is noted that this type of reserve allows reducing the processing time of typical, regular requests to the system due to the fact that the reserve contains the data sets, prepared beforehand and later used at requests processing.

Simulation approach for improving the computing network topology and performance of the China IHEP Data Center

The paper describes the project intended to improve the computing network topology and performance of the China IHEP Data Center taking into account growing numbers of hosts, experiments and computing resources. The analysis of the computing performance of the IHEP Data Center in order to optimize its distributed data processing system is a really hard problem due to the great scale and complexity of shared computing and storage resources between various HEP experiments. In order to fulfil the requirements, we adopt the simulation program SyMSim that was developed at the Laboratory of Information Technologies of the Joint Institute for Nuclear Research. This simulation system is focused on improving the efficiency of the grid-cloud structures development by using the work quality indicators of some real system. SyMSim facilitates making a decision regarding required equipment and resources. The simulation uses input parameters from the data base of the IHEP computing infrastructure, besides we use some data of the BESIII experiments to indicate workflow and data flow parameters for simulation three different cases of organizing IHEP computing infrastructure. The first simulation results show that the proposed approach allows us to make an optimal choice of the network topology improving its performance and saving resources.

Scalable Data Management on Modern Networks

As data processing evolves towards large scale, distributed platforms, the network will necessarily play a substantial role in achieving efficiency and performance. Modern high-speed networks such as InfiniBand, RoCE, or Omni-Path provide advanced features such as Remote-Direct-Memory-Access (RDMA) that have shown to improve the performance and scalability of distributed data processing systems. Furthermore, switches and network cards are becoming more flexible while programmability at all levels (aka, software-defined networks) opens up many possibilities to tailor the network to data processing applications and to push processing down to the network elements. In this paper, we discuss opportunities and present our recent research results to redesign scalable data management systems for the capabilities of modern networks.

The Berlin Big Data Center (BBDC)

Abstract The last decade has been characterized by the collection and availability of unprecedented amounts of data due to rapidly decreasing storage costs and the omnipresence of sensors and data-producing global online-services. In order to process and analyze this data deluge, novel distributed data processing systems resting on the paradigm of data flow such as Apache Hadoop, Apache Spark, or Apache Flink were built and have been scaled to tens of thousands of machines. However, writing efficient implementations of data analysis programs on these systems requires a deep understanding of systems programming, prohibiting large groups of data scientists and analysts from efficiently using this technology. In this article, we present some of the main achievements of the research carried out by the Berlin Big Data Cente (BBDC). We introduce the two domain-specific languages Emma and LARA, which are deeply embedded in Scala and enable declarative specification and the automatic parallelization of data analysis programs, the PEEL Framework for transparent and reproducible benchmark experiments of distributed data processing systems, approaches to foster the interpretability of machine learning models and finally provide an overview of the challenges to be addressed in the second phase of the BBDC.

Дослідження методів і технологій інтеграції онтологічної моделі з реляційними даними

A study of methods and technological solutions on Data Mapping concerning the integration of an ontological and relational data models has been carried out.The main objective of the investigation is to accelerate and reduce the cost of constructing some ontological models for systems that process distributed data in a heterogeneous environment.At present, the theoretical basis for the integration of various data models (Data Mapping) is a di-rection that has been defined as Ontology-based data integration. The theoretical and practical develop-ment of this area is the Ontology-Based Data Access (OBDA) approach, which integrates ontological models presented in the form of RDF graphs with relational data.The methodology of Data Mapping application for distributed data processing systems has been developed. As an example the process of data models consolidation for Ukrainian State Budget monitor-ing system is given. The database of Ukrainian State Budget monitoring system consists of many rela-tional tables, which contain reports of the State Treasury on the implementation of revenue and expendi-ture parts of both state and municipal budgets, as well as the regional section. In addition, the database contains data on lending and arrears of budget institutions.The ontology model connects to data sources through a declarative specification provided in terms of display, including classes and their properties.The given application converts a SPARQL queries into a SQL query to the relational database. The generated SQL query may be execute by the Oracle 11g database driver, which returns the result as a data snapshot. Then, to improve the performance of SPARQL queries, it should be used the semantic query optimization method.Indicative application of the methodology on the example of the construction of an ontological model of the system of monitoring the state budget of Ukraine in Protege 5. It has been demonstrated the results of the execution of the SPARQL-queries to the relational data of the budget process under the Oracle 11g Database. It has been shown the directions semantic optimization for SPARQL-queries, which allow improving obtained data quality.The proposed methodology allows: to integrate data presented by different models — ontological and relational knowledge acquisition; to overcome constraints when databases based on outdated data models are being merged with modern ontological-oriented systems; to eliminate data redundancy in designing knowledge-based systems.

Optimizing distributed data stream processing by tracing

Heterogeneous mobile, sensor, IoT, smart environment, and social networking applications have recently started to produce unbounded, fast, and massive-scale streams of data that have to be processed “on the fly”. Systems that process such data have to be enhanced with detection for operational exceptions and with triggers for both automated and manual operator actions. In this paper, we illustrate how tracing in distributed data processing systems can be applied to detecting changes in data and operational environment to maintain the efficiency of heterogeneous data stream processing systems under potentially changing data quality and distribution. By the tracing of individual input records, we can (1) identify outliers in a web crawling and document processing system and use the insights to define URL filtering rules; (2) identify heavy keys, such as NULL, that should be filtered before processing; (3) give hints to improve the key-based partitioning mechanisms; and (4) measure the limits of overpartitioning if heavy thread-unsafe libraries are imported.By using Apache Spark as illustration, we show how various data stream processing efficiency issues can be mitigated or optimized by our distributed tracing engine. We describe and qualitatively compare two different designs, one based on reporting to a distributed database and another based on trace piggybacking. Our prototype implementation consists of wrappers suitable for JVM environments in general, with minimal impact on the source code of the core system. Our tracing framework is the first to solve tracing in multiple systems across boundaries and to provide detailed performance measurements suitable for automated optimization, not just debugging.