Distributed Processing System Research Articles

Smart Sensory Furniture Based on WSN for Ambient Assisted Living

Ubiquitous computing has been defined as “machines that fit the human environment instead of forcing humans to enter theirs.” An example of this type of approach is “Smart Sensory Furniture” (SSF) Project. SSF is an ambient assisted living system that allows inferring a potential dangerous action of an elderly person living alone at home. This inference is obtained by a specific sensory layer with sensor nodes fixed into furniture and a reasoning layer embedded in a PC that learns from the users’ behavioral patterns and advices when the system detects unusual patterns. This paper aims to explain the SSF sensory layer, which is a distributed signal processing system in a network of sensing objects massively distributed, physically coupled, wirelessly networked, and energy limited. A complete set of experimental test has been carried out. The results show the level of accuracy for each type of sensors and potential use. Finally, the power consumption was experimentally measured and the results show the low maintenance requirements of this solution. The complete system design is described and discussed, including the node mesh details, as well as the type of sensors and actuators and other aspects, such as integration issues and solutions.

Toward a Comprehensive Understanding of EEG and its Analyses

Background: EEG is the oldest tool for studying human cognitive function, but it is blamed to be useless because its poor spatial resolution despite its excellent temporal discrimination. Such comments arise from a reductionist point of view about the cerebral function. However, if the brain is assumed to be a distributed processing system recruiting different types of cells widely distribute over the cortex, then EEG temporal resolution and the many different tools available for its analysis, turn it the tool of choice to investigate human cognition. Proposal: To better understand the different types of information encoded in the recorded cortical electrical activity, a clear model of the cortical structure and function of the different cortical column layers is presented and discussed. Taking this model into consideration, different available techniques for EEG analysis are discussed, under the assumption that tool combination is a necessity to provide a full comprehension of dynamics of the cortical activity supporting human cognition. Methodology: The present approach combines many of the existing methods of analysis to extract better and richer information from the EEG, and proposes additional analysis to better characterize many of the EEG components identified by these different methods. Analysis: Data on language understanding published elsewhere are used to illustrate how to use this combined and complex EEG analysis to disclose important details of cognitive cerebral dynamics, which reveal that cognitive neural circuits are scale free networks supporting entrainment of a large number of cortical column assemblies distributed all over the cortex. Conclusions: Reasoning is then assumed to result from a well orchestrated large scale entrainment.

Seismic Hazard Visualization from Big Simulation Data: Cluster Analysis of Long-Period Ground-Motion Simulation Data

This paper describes a method of extracting the relation between the ground-motion characteristics of each area and a seismic source model, based on ground-motion simulation data output in planar form for many earthquake scenarios, and the construction of a parallel distributed processing system where this method is implemented. The extraction is realized using two-stage clustering. In the first stage, the ground-motion indices and scenario parameters are used as input data to cluster the earthquake scenarios within each evaluation mesh. In the second stage, the meshes are clustered based on the similarity of earthquake-scenario clustering. Because the mesh clusters can be correlated to the geographical space, it is possible to extract the relation between the ground-motion characteristics of each area and the scenario parameters by examining the relation between the mesh clusters and scenario clusters obtained by the two-stage clustering. The results are displayed visually; they are saved as GeoTIFF image files. The system was applied to the long-period ground-motion simulation data for hypothetical megathrust earthquakes in the Nankai Trough. This confirmed that the relation between the extracted ground-motion characteristics of each area and scenario parameters is in agreement with the results of ground-motion simulations.

Electricity Production Data Processing and Management Based on Hadoop and Spark

The modern power system would continue rapidly create a huge quantity of complex user data. Relational database will not be able to cope with its future development. By combined Spark with Hadoop, proposal a new distributed data management and processing system. Design and optimize the model with power and load calculation, line loss calculation and other typical business based on Spark. Test results show that the system and model would maintain the reliability of the data storage, has higher efficiency than the traditional Hadoop model.

SPL

Big data is revolutionizing how all sectors of our economy do business, including telecommunication, transportation, medical, and finance. Big data comes in two flavors: data at rest and data in motion. Processing data in motion is stream processing . Stream processing for big data analytics often requires scale that can only be delivered by a distributed system, exploiting parallelism on many hosts and many cores. One such distributed stream processing system is IBM Streams. Early customer experience with IBM Streams uncovered that another core requirement is extensibility, since customers want to build high-performance domain-specific operators for use in their streaming applications. Based on these two core requirements of distribution and extensibility, we designed and implemented the Streams Processing Language (SPL). This article describes SPL with an emphasis on the language design, distributed runtime, and extensibility mechanism. SPL is now the gateway for the IBM Streams platform, used by our customers for stream processing in a broad range of application domains.

Architectures and Codecs for Real-Time Light Field Streaming

Light field 3D displays represent a major step forward in visual realism, providing glasses-free spatial vision of real or virtual scenes. Applications that capture and process live imagery have to process data captured by potentially tens to hundreds of cameras and control tens to hundreds of projection engines making up the human perceivable 3D light field using a distributed processing system. The associated massive data processing is difficult to scale beyond a specific number and resolution of images, limited by the capabilities of the individual computing nodes. The authors therefore analyze the bottlenecks and data flow of the light field conversion process and identify possibilities to introduce better scalability. Based on this analysis they propose two different architectures for distributed light field processing. To avoid using uncompressed video data all along the processing chain, the authors also analyze how the operation of the proposed architectures can be supported by existing image/video codecs. © 2017 Society for Imaging Science and Technology.

An experimental comparison of partitioning strategies in distributed graph processing

In this paper, we study the problem of choosing among partitioning strategies in distributed graph processing systems. To this end, we evaluate and characterize both the performance and resource usage of different partitioning strategies under various popular distributed graph processing systems, applications, input graphs, and execution environments. Through our experiments, we found that no single partitioning strategy is the best fit for all situations, and that the choice of partitioning strategy has a significant effect on resource usage and application run-time. Our experiments demonstrate that the choice of partitioning strategy depends on (1) the degree distribution of input graph, (2) the type and duration of the application, and (3) the cluster size. Based on our results, we present rules of thumb to help users pick the best partitioning strategy for their particular use cases. We present results from each system, as well as from all partitioning strategies implemented in one common system (PowerLyra).

Aggregation Operator for Assignment of Resources in Distributed Systems

In distributed processing systems it is often necessary to coordinate the allocation of shared resources that should be assigned to processes in the modality of mutual exclusion; in such cases, the order in which the shared resources will be assigned to processes that require them must be decided; in this paper we propose an aggregation operator (which could be used by a shared resources manager module) that will decide the order of allocation of the resources to the processes considering the requirements of the processes (shared resources) and the state of the distributed nodes where the processes operate (their computational load).

Synergistic Analysis of Evolving Graphs

Evolving graph processing involves repeating analyses, which are often iterative, over multiple snapshots of the graph corresponding to different points in time. Since the snapshots of an evolving graph share a great number of vertices and edges, traditional approaches that process these snapshots one at a time without exploiting this overlap contain much wasted effort on both data loading and computation, making them extremely inefficient. In this article, we identify major sources of inefficiencies and present two optimization techniques to address them. First, we propose a technique for amortizing the fetch cost by merging fetching of values for different snapshots of the same vertex. Second, we propose a technique for amortizing the processing cost by feeding values computed by earlier snapshots into later snapshots. We have implemented these optimizations in two distributed graph processing systems, namely, GraphLab and ASPIRE. Our experiments with multiple real evolving graphs and algorithms show that, on average fetch amortization speeds up execution of GraphLab and ASPIRE by 5.2× and 4.1× , respectively. Amortizing the processing cost yields additional average speedups of 2× and 7.9×, respectively.

Intelligent and independent processes for overcoming big graphs

Poor locality as a natural property of graph data structures causes enormous amount of network traffic in large-scale distributed graph processing systems. Moreover, data transmission through the n...

Design of multi‐actor distributed processing systems: A game‐theoretical approach

The manufacturing of a final product could be the result of a value chain involving several processing plants distributed across several distinct owners; a feature that may prevent the application of classical process design approaches that depend on a centralized treatment of the complete processing network. In this article we propose and develop a game‐theoretical framework and specific methodologies, which allow the optimal design of distributed processing systems, through the decentralized strategies of independent actors. The resulting process design corresponds to a Nash Equilibrium point among the interacting actors. Its optimality and the justification of the independent strategies that led to it, are theoretically based on (and constrained by) the properties of the 2‐level Lagrangian approach. The article also discusses the use of penalty‐term approaches, which can extend the applicability of the proposed framework and design methodologies to problems for which the underlying convexity assumptions of the 2‐level Lagrangian approach may not be possible to ascertain. A series of case studies illustrate the application of the proposed ideas to distributed processing networks of various structures. © 2016 American Institute of Chemical Engineers AIChE J, 62: 3369–3391, 2016

A distributed query execution engine of big attributed graphs.

A graph is a popular data model that has become pervasively used for modeling structural relationships between objects. In practice, in many real-world graphs, the graph vertices and edges need to be associated with descriptive attributes. Such type of graphs are referred to as attributed graphs. G-SPARQL has been proposed as an expressive language, with a centralized execution engine, for querying attributed graphs. G-SPARQL supports various types of graph querying operations including reachability, pattern matching and shortest path where any G-SPARQL query may include value-based predicates on the descriptive information (attributes) of the graph edges/vertices in addition to the structural predicates. In general, a main limitation of centralized systems is that their vertical scalability is always restricted by the physical limits of computer systems. This article describes the design, implementation in addition to the performance evaluation of DG-SPARQL, a distributed, hybrid and adaptive parallel execution engine of G-SPARQL queries. In this engine, the topology of the graph is distributed over the main memory of the underlying nodes while the graph data are maintained in a relational store which is replicated on the disk of each of the underlying nodes. DG-SPARQL evaluates parts of the query plan via SQL queries which are pushed to the underlying relational stores while other parts of the query plan, as necessary, are evaluated via indexless memory-based graph traversal algorithms. Our experimental evaluation shows the efficiency and the scalability of DG-SPARQL on querying massive attributed graph datasets in addition to its ability to outperform the performance of Apache Giraph, a popular distributed graph processing system, by orders of magnitudes.

CLUSTOM-CLOUD: In-Memory Data Grid-Based Software for Clustering 16S rRNA Sequence Data in the Cloud Environment.

High-throughput sequencing can produce hundreds of thousands of 16S rRNA sequence reads corresponding to different organisms present in the environmental samples. Typically, analysis of microbial diversity in bioinformatics starts from pre-processing followed by clustering 16S rRNA reads into relatively fewer operational taxonomic units (OTUs). The OTUs are reliable indicators of microbial diversity and greatly accelerate the downstream analysis time. However, existing hierarchical clustering algorithms that are generally more accurate than greedy heuristic algorithms struggle with large sequence datasets. To keep pace with the rapid rise in sequencing data, we present CLUSTOM-CLOUD, which is the first distributed sequence clustering program based on In-Memory Data Grid (IMDG) technology–a distributed data structure to store all data in the main memory of multiple computing nodes. The IMDG technology helps CLUSTOM-CLOUD to enhance both its capability of handling larger datasets and its computational scalability better than its ancestor, CLUSTOM, while maintaining high accuracy. Clustering speed of CLUSTOM-CLOUD was evaluated on published 16S rRNA human microbiome sequence datasets using the small laboratory cluster (10 nodes) and under the Amazon EC2 cloud-computing environments. Under the laboratory environment, it required only ~3 hours to process dataset of size 200 K reads regardless of the complexity of the human microbiome data. In turn, one million reads were processed in approximately 20, 14, and 11 hours when utilizing 20, 30, and 40 nodes on the Amazon EC2 cloud-computing environment. The running time evaluation indicates that CLUSTOM-CLOUD can handle much larger sequence datasets than CLUSTOM and is also a scalable distributed processing system. The comparative accuracy test using 16S rRNA pyrosequences of a mock community shows that CLUSTOM-CLOUD achieves higher accuracy than DOTUR, mothur, ESPRIT-Tree, UCLUST and Swarm. CLUSTOM-CLOUD is written in JAVA and is freely available at http://clustomcloud.kopri.re.kr.

Seismic Hazard Visualization from Big Simulation Data: Construction of a Parallel Distributed Processing System for Ground Motion Simulation Data

We have developed a data mining system of parallel distributed processing system which is applicable to the large-scale and high-resolution numerical simulation of ground motion by transforming into ground motion indices and their statistical values, and then visualize their values for the seismic hazard information. In this system, seismic waveforms at many locations calculated for many possible earthquake scenarios can be used as input data. The system utilizes Hadoop and it calculates the ground motion indices, such as PGV, and statistical values, such as maximum, minimum, average, and standard deviation of PGV, by parallel distributed processing with MapReduce. The computation results are being an output as GIS (Geographic Information System) data file for visualization. And this GIS data is made available via the Web Map Service (WMS). In this study, we perform two benchmark tests by applying three-component synthetic waveforms at about 80,000 locations for 10 possible scenarios of a great earthquake in Nankai Trough to our system. One is the test for PGV calculation processing. Another one is the test for PGV data mining processing. A maximum of 10 parallel processing are tested for both cases. We find that our system can hold the performance even when the total tasks is larger than 10. This system can enable us to effectively study and widely distribute to the communities for disaster mitigation since it is built with data mining and visualization for hazard information by handling a large number of data from a large-scale numerical simulation.

Automatic Workflow Scheduling Tuning for Distributed Processing Systems

Modern scientific applications are composed of various methods, techniques and models to solve complicated problems. Such composite applications commonly are represented as workflows. Workflow scheduling is a well-known optimization problem, for which there is a great amount of solutions. Most of the algorithms contain parameters, which affect the result of a method. Thus, for the efficient scheduling it is important to tune parameters of the algorithms. Moreover, performance models, which are used for the estimation of obtained solutions, are crucial parts of workflow scheduling. In this work we present a combined approach for automatic parameters tuning and performance models construction in the background of the WMS lifecycle. Algorithms tuning is provided by hyper-heuristic genetic algorithm, whereas models construction is performed via symbolic regression methods. Developed algorithm was evaluated using CLAVIRE platform and is applicable for any distributed computing systems to optimize the execution of composite applications.

A Scalable Product Recommendations Using Collaborative Filtering in Hadoop for Bigdata

The growth of data and information causes the need of next-generation databases and data science tools. Most of the business needs a service recommendation system which have been used by millions of users. Day by day, the amount of customers, products and information has grown rapidly, yielding the big data analysis problem for service recommender systems. Consequently, conventional recommender service systems often suffer from lack of scalability and efficiency problems when processing or analysis of this data on a large scale. To avoid these problems, a novel recommendations system using collaborative filtering algorithm is implemented in Apache Hadoop leveraging MapReduce paradigm for Bigdata. Apache Hadoop is an open framework for Distributed processing systems can process large volumes of data. It can be used for offline processing and not suitable for low latency analytics. Port data onto the next generation databases like HBase and optimize the performance of it. For the product recommendations the Amazon dataset is used. Proposed Framework have significant improvement in performance compared to conventional tools.

맵리듀스 기반 DFP-Tree를 이용한 클러스터링 알고리즘

ABSTRACT As BigData is issued, many applications that operate based on t he results of data analysis have been developed, typically applications are products recommend service of e-commerce appli cation service system, search service on the search engine serv ice and friend list recommend system of social network service.In this paper, we suggests a decision frequent pattern tree tha t is combined the origin frequent pattern tree that is mining s imilar pattern to appear in the data set of the existing data mining t echniques and decision tree based on the theory of computer sci ence. The decision frequent pattern tree algorithm improves about pro blem of frequent pattern tree that have to make some a lot’s pattern so it is to hard to analyze about data. We also proposes to mod el for a Mapredue framework that is a programming model to help to operate in distributed environment. ☞ keyword : SData Mining, Frequent-Pattern tree, Clustering Algor ithms, Distributed Processing System, Recommendation System, Map Reduce

도로 침수영역의 탐색을 위한 빅데이터 분석 시스템 연구

The frequency of natural disasters because of global warming is gradually increasing, risks of flooding due to typhoon and torrential rain have also increased. Among these causes, the roads are flooded by suddenly torrential rain, and then vehicle and personal injury are happening. In this respect, because of the possibility that immersion of a road may occur in a second, it is necessary to study the rapid data collection and quick response system. Our research proposes a big data analysis system based on the collected information and a variety of system information collection methods for searching flooded road areas by torrential rains. The data related flooded roads are utilized the SNS data, meteorological data and the road link data, etc. And the big data analysis system is implemented the distributed processing system based on the Hadoop platform.

Security in big data

Security in big data

큐브리드 샤드 분산 데이터베이스에서 집계/분석 함수의 분산 처리 시스템 개발

Database Shard is a technique that can be queried and stored by dividing one logical table into multiple databases horizontally. In order to analyze the shard data with aggregate or analysis functions, a process is required that integrates partial results on each shard database. In this paper, we introduce the design and implementation of a distributed processing system for aggregation and analysis on the CUBRID Shard distributed database, which is an open source database management system. The implemented system can accelerate the analysis onto multiple shards of partitioned tables; it shows efficient aggregation on shard distributed databases compared to stand-alone databases.