Distributed Data Analysis Research Articles

Improvement of substation Monitoring aimed to improve its efficiency with the help of Big Data Analysis**

Abstract Data analysis has become most widespread field of research and it has extended in almost every field of study. Considering the recent trends and developments in the field of communication and information technology, there is a scope of combining the monitoring of substation equipment with big data analysis technology. That will result in an improved data analysis ability, information sharing and utilization rate of monitoring data. In the proposed work, the authors have introduced the big data analysis and its corresponding application in the monitoring of substations. Basic concepts and the procedures of the typical data analysis for general problems are also discussed. As a main part of the paper, different types of distributed data analysis techniques have been proposed, in which two relational online analysis, namely Hive and Impala and one H Base multidimensional online analysis are important. These data analysis techniques are proposed considering the analysis efficiency, storage performance from the business development requirements point of view of the substation. The result obtained depicts that the proposed model has an advantage in storage overhead and roll-up performance, when compared with the traditional method, although the data loading speed is approximately 1.7–1.9 times of the traditional model. Some experiments are carried out in order to verify the validity of the model.

Distributed Data Analysis and Reliable Operation of Cyberphysical Systems

The articles in this special section focuses on distributed data analysis and the reliable operations of cyber-physical systems.

Distributed learning on 20 000+ lung cancer patients – The Personal Health Train

Background and purposeAccess to healthcare data is indispensable for scientific progress and innovation. Sharing healthcare data is time-consuming and notoriously difficult due to privacy and regulatory concerns. The Personal Health Train (PHT) provides a privacy-by-design infrastructure connecting FAIR (Findable, Accessible, Interoperable, Reusable) data sources and allows distributed data analysis and machine learning. Patient data never leaves a healthcare institute. Materials and methodsLung cancer patient-specific databases (tumor staging and post-treatment survival information) of oncology departments were translated according to a FAIR data model and stored locally in a graph database. Software was installed locally to enable deployment of distributed machine learning algorithms via a central server. Algorithms (MATLAB, code and documentation publicly available) are patient privacy-preserving as only summary statistics and regression coefficients are exchanged with the central server. A logistic regression model to predict post-treatment two-year survival was trained and evaluated by receiver operating characteristic curves (ROC), root mean square prediction error (RMSE) and calibration plots. ResultsIn 4 months, we connected databases with 23 203 patient cases across 8 healthcare institutes in 5 countries (Amsterdam, Cardiff, Maastricht, Manchester, Nijmegen, Rome, Rotterdam, Shanghai) using the PHT. Summary statistics were computed across databases. A distributed logistic regression model predicting post-treatment two-year survival was trained on 14 810 patients treated between 1978 and 2011 and validated on 8 393 patients treated between 2012 and 2015. ConclusionThe PHT infrastructure demonstrably overcomes patient privacy barriers to healthcare data sharing and enables fast data analyses across multiple institutes from different countries with different regulatory regimens. This infrastructure promotes global evidence-based medicine while prioritizing patient privacy.

Distributed data analysis with ROOT RDataFrame

Widespread distributed processing of big datasets has been around for more than a decade now thanks to Hadoop, but only recently higher-level abstractions have been proposed for programmers to easily operate on those datasets, e.g. Spark. ROOT has joined that trend with its RDataFrame tool for declarative analysis, which currently supports local multi-threaded parallelisation. However, RDataFrame’s programming model is general enough to accommodate multiple implementations or backends: users could write their code once and execute it as-is locally or distributedly, just by selecting the corresponding backend. This abstract introduces PyRDF, a new python library developed on top of RDataFrame to seamlessly switch from local to distributed environments with no changes in the application code. In addition, PyRDF has been integrated with a service for web-based analysis, SWAN, where users can dynamically plug in new resources, as well as write, execute, monitor and debug distributed applications via an intuitive interface.

Differentially Private Maximum Consensus: Design, Analysis and Impossibility Result

In network systems, calculating maximum element plays a fundamental role for distributed data analysis. Maximum consensus algorithm can accomplish the calculation in a fully distributed way. However, the data for communication are privacy-sensitive in many scenarios, making nodes unwilling to participate in the distributed calculation process. How to ensure privacy and accuracy for distributed calculation of maximum element is essential while challenging. In this paper, we first prove that exact maximum consensus and differential privacy cannot be guaranteed simultaneously. Then, to provide sufficient privacy preservation for nodes, we propose a novel differentially private maximum consensus (DPMC) algorithm where nodes perturb their initial states with Laplacian noises. We prove that DPMC algorithm preserves differential privacy and give the analytical expression of privacy preserving degree. Furthermore, the proposed algorithm also guarantees finite-time convergence and achieves high convergence accuracy. The performance tradeoff is finally analyzed and evaluated by extensive numerical simulations.

Guest Editorial Emerging Computing Offloading for IoTs: Architectures, Technologies, and Applications

Billions of Internet of Things (IoT) devices, e.g., sensors and RFIDs, are arising around us providing not only computing-intensive, but also delay-sensitive services, ranging from augmented/virtual realities to distributed data analysis and artificial intelligence. Notably, the low response latency for IoT services is achieved at the cost of computing complexity that far exceeds the capabilities of IoT devices. To feed this trend, multiple computing paradigms are emerging, such as mobile transparent computing (TC), edge computing, and fog computing. These paradigms employ more resourceful edge devices, e.g., small-scale servers, smart phones, and laptops, to assist the low-end IoT devices. By offloading the computing-intensive tasks to the edge devices, it is expected to converge the data collection at IoT devices and the data processing at edge devices to provision computing-intensive and delay-sensitive services. However, many issues remain in the application of computing offloading which impede its flourishing in IoTs. To name a few, what are the killer APPs that need computing offloading for performance boost? How to partition an encapsulated APP into offloadable code blocks for remote loading? How to determine which code blocks or computing tasks should be offloaded to edge servers? How to customize the communication protocol to guarantee the coherence of computation offloading?

BEANS – distributed data analysis for numerical simulations

Abstractbeans is a tool for distributed data analysis. It provides web and command line interface for data analysis and plotting for huge datasets. beans is written in a general form and can be used in any field of research to analyze the data. The main purpose of beans is to provide to the community a versatile tool to store, analyze and then visualize any amount of scientific data (e.g. numerical simulation, observations).

Everyone counts? Design considerations in online citizen science

Effective classification of large datasets is a ubiquitous challenge across multiple knowledge domains. One solution gaining in popularity is to perform distributed data analysis via online citizen science platforms, such as the Zooniverse. The resulting growth in project numbers is increasing the need to improve understanding of the volunteer experience; as the sustainability of citizen science is dependent on our ability to design for engagement and usability. Here, we examine volunteer interaction with 63 projects, representing the most comprehensive collection of online citizen science project data gathered to date. Together, this analysis demonstrates how subtle project design changes can influence many facets of volunteer interaction, including when and how much volunteers interact, and, importantly, who participates. Our findings highlight the tension between designing for social good and broad community engagement, versus optimizing for scientific and analytical efficiency.

Информационные инструменты распределенного анализа данных спутникового мониторинга растительных ареалов при проведении специальных экспертиз

Информационные инструменты распределенного анализа данных спутникового мониторинга растительных ареалов при проведении специальных экспертиз

Training Normal Bayes Classifier on Distributed Data

The paper describes an approach to parallelization of Normal Bayes classifier training algorithm for distributed data. In the process of distributed data analysis and the algorithm performance, the results fail to join properly. Due to this, the algorithm is to be performed in a distributed manner. For this purpose, we use representation of the algorithm as a sequential composition of functions. The algorithm is parallelized to work with data distributed horizontally and vertically. This allows placing parallel functions of the algorithm at data nodes. Experiments show that transfer of computations to sources allow to decrease training time and network traffic. We implement the algorithm variants as an extension of the industrial-strength Java-based library Xelopes.

Distributed gene clinical decision support system based on cloud computing

BackgroundThe clinical decision support system can effectively break the limitations of doctors’ knowledge and reduce the possibility of misdiagnosis to enhance health care. The traditional genetic data storage and analysis methods based on stand-alone environment are hard to meet the computational requirements with the rapid genetic data growth for the limited scalability.MethodsIn this paper, we propose a distributed gene clinical decision support system, which is named GCDSS. And a prototype is implemented based on cloud computing technology. At the same time, we present CloudBWA which is a novel distributed read mapping algorithm leveraging batch processing strategy to map reads on Apache Spark.ResultsExperiments show that the distributed gene clinical decision support system GCDSS and the distributed read mapping algorithm CloudBWA have outstanding performance and excellent scalability. Compared with state-of-the-art distributed algorithms, CloudBWA achieves up to 2.63 times speedup over SparkBWA. Compared with stand-alone algorithms, CloudBWA with 16 cores achieves up to 11.59 times speedup over BWA-MEM with 1 core.ConclusionsGCDSS is a distributed gene clinical decision support system based on cloud computing techniques. In particular, we incorporated a distributed genetic data analysis pipeline framework in the proposed GCDSS system. To boost the data processing of GCDSS, we propose CloudBWA, which is a novel distributed read mapping algorithm to leverage batch processing technique in mapping stage using Apache Spark platform.

When does the increased mortality risk appear in rheumatoid arthritis? A distributed data analysis comparing mortality in two Canadian provinces

IntroductionRheumatoid arthritis (RA) is chronic inflammatory arthritis. For decades studies showed that RA patients died earlier than their general population counterparts. Some inception cohorts have failed to detect an increased mortality risk, possibly due to limited follow-up or to improvement in mortality risk in cohorts of more recent onset.
 Objectives and ApproachWe evaluated mortality risk in RA patients and estimated when the increased risk appears. Using a common protocol, we conducted distributed analyses using administrative data, of incident RA patients in British Columbia (BC) and Ontario (ON) over 2000-2015. We identified all RA patients (using validated criteria), and identified non-RA comparators, matched 1:2 on age, sex and index years. Adjusted hazard ratios (HRs) were estimated using multivariable Cox regression, controlling for comorbidities and other factors. To estimate when the increased risk appeared we included an interaction with follow-up time, to detect if and how the HR varied by RA duration.
 ResultsAmong 13834 RA patients in BC (27668 comparators), 66% were female with a mean age of 58 years at cohort entry. Among 27405 RA patients in ON (54810 comparators), 70% were female with a mean age of 56 years. The prevalence of individual comorbidities was comparable across RA cohorts. During follow-up, 23% of RA patients in each province died, with corresponding crude mortality rates of 2.3 deaths per 100 person-years in both provinces. Multivariable analyses detected an increased mortality risk in RA by 6 years of follow-up, with a linear relationship suggesting further increase over time. By 10 years, the adjusted HR was 1.14 (95% CI 1.07,1.22) in BC and 1.13 (95% CI 1.08,1.18) in ON.
 Conclusion/ImplicationsIn 2 large Canadian RA inception cohorts, a small increased mortality risk appeared after 6 years of RA duration and increased to a 14% (in BC) and 13% (in ON) increased mortality risk after 10 years, suggesting increased efforts to prevent disease progression and optimizing comorbidity management are needed.

An Approach of RDD Optimization in Big Data Analytics

We live in the information age and new information is produced in each and every second. The information can be worlds data which is responsible for major technological changes that can bring new ways in decision making. Keeping this data for further analysis and computation is a difficult task. Several studies are done in this arena, making it effective for future figuring. Processing or analysing such huge amount of data is a challenging task. All the existing technologies contain certain bottlenecks and performance overheads, diverse challenges like scalability. Spark is the commonly used data analysis framework. Map Reduce is a computing paradigm and a popular model for distributed data analysis. This paper gives a survey about some enormous information technologies, how it will deal with huge data, and the difficulties in existing advances, and has additionally learned about a portion of the execution bottlenecks and preventive techniques, and the concentration at that point moves to the Resilient Distributed Dataset (RDD), and how it is optimized.

Role of the ATLAS Grid Information System (AGIS) in Distributed Data Analysis and Simulation

In modern high-energy physics experiments, particular attention is paid to the global integration of information and computing resources into a unified system for efficient storage and processing of experimental data. Annually, the ATLAS experiment performed at the Large Hadron Collider at the European Organization for Nuclear Research (CERN) produces tens of petabytes raw data from the recording electronics and several petabytes of data from the simulation system. For processing and storage of such super-large volumes of data, the computing model of the ATLAS experiment is based on heterogeneous geographically distributed computing environment, which includes the worldwide LHC computing grid (WLCG) infrastructure and is able to meet the requirements of the experiment for processing huge data sets and provide a high degree of their accessibility (hundreds of petabytes). The paper considers the ATLAS grid information system (AGIS) used by the ATLAS collaboration to describe the topology and resources of the computing infrastructure, to configure and connect the high-level software systems of computer centers, to describe and store all possible parameters, control, configuration, and other auxiliary information required for the effective operation of the ATLAS distributed computing applications and services. The role of the AGIS system in the development of a unified description of the computing resources provided by grid sites, supercomputer centers, and cloud computing into a consistent information model for the ATLAS experiment is outlined. This approach has allowed the collaboration to extend the computing capabilities of the WLCG project and integrate the supercomputers and cloud computing platforms into the software components of the production and distributed analysis workload management system (PanDA, ATLAS).

A Modeling Language for MapReduce Programing in a Storage System Perspective

MapReduce is a powerful distributed data analysis programming model. It runs on big data storage systems and processes data in a parallel way. An appropriate way to ensure the correctness of MapReduce programs is formal method analysis, which requires firstly a formal model of MapReduce. In this paper we propose a modeling language to establish the formal model of the MapReduce framework. Unlike other approaches, our language describes the processing of data in the MapReduce programs from a perspective of underlying files and blocks, so that the details of data processing can be clearly demonstrated. The language is based on our previous work, a language describing the management of massive data storage systems, with extensions from two aspects: block content data refinement and concurrency support. Based on our language, the features of the MapReduce programming model can be discussed.

Model selection with distributed SCAD penalty

ABSTRACTIn this paper, we focus on the feature extraction and variable selection of massive data which is divided and stored in different linked computers. Specifically, we study the distributed model selection with the Smoothly Clipped Absolute Deviation (SCAD) penalty. Based on the Alternating Direction Method of Multipliers (ADMM) algorithm, we propose distributed SCAD algorithm and prove its convergence. The results of variable selection of the distributed approach are same with the results of the non-distributed approach. Numerical studies show that our method is both effective and efficient which performs well in distributed data analysis.

A comparison of different database technologies for the CMS AsyncStageOut transfer database

AsyncStageOut (ASO) is the component of the CMS distributed data analysis system (CRAB) that manages users transfers in a centrally controlled way using the File Transfer System (FTS3) at CERN. It addresses a major weakness of the previous, decentralized model, namely that the transfer of the user’s output data to a single remote site was part of the job execution, resulting in inefficient use of job slots and an unacceptable failure rate.Currently ASO manages up to 600k files of various sizes per day from more than 500 users per month, spread over more than 100 sites. ASO uses a NoSQL database (CouchDB) as internal bookkeeping and as way to communicate with other CRAB components. Since ASO/CRAB were put in production in 2014, the number of transfers constantly increased up to a point where the pressure to the central CouchDB instance became critical, creating new challenges for the system scalability, performance, and monitoring. This forced a re-engineering of the ASO application to increase its scalability and lowering its operational effort.In this contribution we present a comparison of the performance of the current NoSQL implementation and a new SQL implementation, and how their different strengths and features influenced the design choices and operational experience. We also discuss other architectural changes introduced in the system to handle the increasing load and latency in delivering output to the user.

Peer sampling gossip-based distributed clustering algorithm for unstructured P2P networks

Clustering, as an unsupervised learning method and an important process in data mining, is an aspect of large and distributed data analysis. In many applications, such as peer-to-peer systems, huge volumes of data are distributed between multiple sources. Analysis of these volumes of data and identifying appropriate clusters is challenging due to transmission, processing and storage costs. In this paper, a gossip-based distributed clustering algorithm for P2P networks called Efficient GBDC-P2P is proposed, based on an improved gossip communicative approach by combining the peer sampeling and CYCLON protocol and the idea of partitioning-based data clustering. This algorithm is appropriate for data clustering in unstructured P2P networks, and it is adapted to the dynamic conditions of these networks. In the Efficient GBDC-P2P algorithm, distributed peers perform clustering operation in a distributed way only through local communications with their neighbors. Our approach does not rely on the central server to carry out data clustering task and without the need to synchronize operations. Evaluation results verify the efficiency of our proposed algorithm for data clustering in unstructured P2P networks. Furthermore, comparative analyses with other well-established distributed clustering approaches demonstrate the superior accuracy of the proposed method.

Vehicle-assist resilient information and network system for disaster management

After big disasters, a damaged area can be out of contact because of severe damage of existing network infrastructures. Meanwhile, high demands for network connections to the disaster area will arise to collect damage information and disseminate rescue instructions. In this paper, we design a vehicle-assist resilient information and network system for disaster management, despite of the Internet unavailability. It contains three main components: (1) smartphone apps that provide functions of SOS reporting, life and medical resources request/provision, and safe road navigation; (2) mobile stations that assist data exchange between smartphone apps and servers; (3) geo-distributed servers that collect user data, conduct distributed data analysis, and make disaster management decisions. Since the vehicle-assist network is critical to connect isolated smartphones and servers, we continue to study the scheduling problem of mobile stations. Given a number of disaster management tasks, such as sensing, information collection, and message dissemination, we propose online algorithms that schedules mobile stations for disaster management tasks with the objective of maximizing the total weight of finished tasks, without any knowledge of future task arrivals. We derive the competitive ratio of our proposed algorithms and conduct extensive simulations for performance evaluation.

Integration of end-user Cloud storage for CMS analysis

End-user Cloud storage is increasing rapidly in popularity in research communities thanks to the collaboration capabilities it offers, namely synchronisation and sharing. CERN IT has implemented a model of such storage named, CERNBox, integrated with the CERN AuthN and AuthZ services. To exploit the use of the end-user Cloud storage for the distributed data analysis activity, the CMS experiment has started the integration of CERNBox as a Grid resource. This will allow CMS users to make use of their own storage in the Cloud for their analysis activities as well as to benefit from synchronisation and sharing capabilities to achieve results faster and more effectively. It will provide an integration model of Cloud storages in the Grid, which is implemented and commissioned over the world’s largest computing Grid infrastructure, Worldwide LHC Computing Grid (WLCG).In this paper, we present the integration strategy and infrastructure changes needed in order to transparently integrate end-user Cloud storage with the CMS distributed computing model. We describe the new challenges faced in data management between Grid and Cloud and how they were addressed, along with details of the support for Cloud storage recently introduced into the WLCG data movement middleware, FTS3. The commissioning experience of CERNBox for the distributed data analysis activity is also presented.