Parallel Processing Engine Research Articles

Research on Application of Computer Big Data Technology in Real-time Monitoring System of Bridge Structure Safety

The bridge structure safety real-time monitoring system based on big data evaluates the bridge safety status and real-time warning under the comprehensive consideration of various factors that affect the service performance of the bridge. The system platform uses a distributed file system with high fault tolerance and a parallel data processing engine with high computational efficiency. It has high reliability, availability and storage efficiency, and is easy to expand. Furthermore, the multi-factor analysis method is used to fully explore the implicit correlation between the real-time data of the various sensors of the bridge, and the bridge service performance evaluation model is accurately established by analysing the data correlation, and real-time warning of the bridge safety status is given. In addition, the system platform uses a variety of model verification methods to evaluate the effectiveness of the model to ensure the reliability of the big data analysis method.

Deadline-Aware Cost Optimization for Spark

We present OptEx, a closed-form model of job execution on Apache Spark, a popular parallel processing engine. To the best of our knowledge, OptEx is the first work that analytically models job completion time on Spark. The model can be used to estimate the completion time of a given Spark job on a cloud, with respect to the size of the input dataset, the number of iterations, and the number of nodes comprising the underlying cluster. Experimental results demonstrate that OptEx yields a mean relative error of 6 percent in estimating the job completion time. Furthermore, the model can be applied for estimating the cost-optimal cluster composition for running a given Spark job on a cloud under a completion deadline specified in the SLO (i.e., Service Level Objective). We show experimentally that OptEx is able to correctly estimate the required cluster composition for running a given Spark job under a given SLO deadline with an accuracy of 98 percent. We also provide a tool which can classify Spark jobs into job categories based on bisimilarity analysis on lineage graphs collected from the given jobs.

Parallel seed selection method for overlapping community detection in social network

Social network analysis is one of the key areas of research during modern times. The social network is growing with more users and the ties between them day by day. This reason brings out many research queries and new conclusions from this area. Overlapping community detection in the social network is one such research problem which has acquired interest among researchers nowadays. Earlier, the investigation was in finding out algorithms to detect communities in the network sequentially. There are many distinguished findings toward overlapping community detection. Due to the velocity of data in the current era, the available algorithms will be a bit sluggish in processing the data. The proposed algorithm uses parallel processing engine to resolve this delay problem in the current scenario. The algorithm in parallel finds out the superior seed set in the network and expands it in parallel to find out the community. The work shows amazing improvement in the runtime and also detects quality groups in the network.

Introduction to the Special Issue on Mobile Cloud Applications and Challenges

We are happy to present this special issue of thee Scientific Journal Scalable Computing: Practice and Experience. In this special issue on Mobile Cloud Applications and Challenges (Volume 19 No 4 December 2018), we have selected five papers, which gone through a peer review according to the journal policy. All the papers represents novel results in the field of Mobile and Cloud Applications.
 
 The first paper presents overview of mobile cloud computing, cloudlet technology, security and privacy issues and limitations of mobile cloud commuting. The second paper presents a spatial partition, global index and map reduce operation were studied. The trail results that the proposed indexing cloud framework performs improved results. The third paper proposes an authentication model along with data security in a public cloud storage environment which successful detects the unauthenticated access or any anomaly in data. The fourth paper analyses the student success ratio which uses a cloud based technology to implement and design SaaS. Graph based complex network are used for analysing the course. The fifth paper presents Overlapping community detection in social networks. The proposed algorithm uses parallel processing engine to resolve the delay problem.

A high performance query analytical framework for supporting data-intensive climate studies

Climate observations and model simulations produce vast amounts of data. The unprecedented data volume and the complexity of geospatial statistics and analysis requires efficient analysis of big climate data to investigate global problems such as climate change, natural disasters, diseases, and other environmental issues. This paper introduces a high performance query analytical framework to tackle these challenges by leveraging Hive and cloud computing technologies. With this framework, we propose grid transformation, a new perspective for complex climate analysis that applies a series of atomic transformations to terabytes of climate data using SQL-style query (HiveQL). Specifically, we introduce four types of grid transformations (temporal, spatial, local, and arithmetic) to support a broad range of climate analyses, from the basic spatiotemporal aggregation to more sophisticated anomaly detection. Each query is processed as MapReduce tasks in a highly scalable Hadoop cluster as the parallel processing engine. Big climate data are directly stored and managed in a Hadoop Distributed File System without any data format conversion. A prototype is developed to evaluate the feasibility and performance of the framework. Experimental results show that complex and data-intensive climate analysis can be conducted using intuitive SQL queries with good flexibility and performance. This research provides a building block and practical insights in establishing a cyberinfrastructure that provides a high performance and collaborative environment for data-intensive geospatial applications in climate science.

Big data-enabled multiscale serviceability analysis for aging bridges☆

This work is dedicated to constructing a multi-scale structural health monitoring system to monitor and evaluate the serviceability of bridges based on the Hadoop Ecosystem (MS-SHM-Hadoop). By taking the advantages of the fault-tolerant distributed file system called the Hadoop Distributed File System (HDFS) and high-performance parallel data processing engine called MapReduce programming paradigm, MS-SHM-Hadoop features include high scalability and robustness in data ingestion, fusion, processing, retrieval, and analytics. MS-SHM-Hadoop is a multi-scale reliability analysis framework, which ranges from nationwide bridge-surveys, global structural integrity analysis, and structural component reliability analysis. This Nationwide bridge survey uses deep-learning techniques to evaluate the bridge serviceability according to real-time sensory data or archived bridge-related data such as traffic status, weather conditions and bridge structural configuration. The global structural integrity analysis of a targeted bridge is made by processing and analyzing the measured vibration signals incurred by external loads such as wind and traffic flow. Component-wise reliability analysis is also enabled by the deep learning technique, where the input data is derived from the measured structural load effects, hyper-spectral images, and moisture measurement of the structural components. As one of its major contributions, this work employs a Bayesian network to formulate the integral serviceability of a bridge according to its components serviceability and inter-component correlations. Here the inter-component correlations are jointly specified using a statistics-oriented machine learning method (e.g., association rule learning) or structural mechanics modeling and simulation.

Customized Travel Planner using MapReduce and Approximation Algorithm

It is fun to travel but painful to arrange the trip. When travelers start off planning they need flights, accommodation and attractions. Which scattered across multiple websites on the internet? Traveler spends time scouting each of them for the best deals, and gets the attraction reviews from established planners in the market. It will be always good if traveler gives specified designations and time he wants to spend for the trip and some platform will automatically did everything for traveler with added bonus of optimal and customized itineraries. This system is designed for such travelers to design customized itineraries which will be optimal and consist of Point-Of-Interest (POI) selected by traveler, rather than go and visit the traditional and static trip plans by many travel agencies. This system is two-stage processing system for cost effective and optimal results. First is preprocessing stage works offline uses parallel processing engine as MapReduce to precompute Single-Day Itineraries. In further stage which is online the precomputed itineraries are combined to give multiday itineraries. These itineraries produced are optimal as per travelers selected Point-OfInterest (POI).Here Greedy Approximation Algorithm is used to combine the single day itineraries. In this way TeamOrienteering-Problem (TOP) is transferred to Set-Packing Problem another NP-complete problem.

INDREX: In-database relation extraction

The management of text data has a long-standing history in the human mankind. A particular common task is extracting relations from text. Typically, the user performs this task with two separate systems, a relation extraction system and an SQL-based query engine for analytical tasks. During this iterative analytical workflow, the user must frequently ship data between these systems. Worse, the user must learn to manage both systems. Therefore, end users often desire a single system for both analytical and relation extraction tasks.We propose INDREX, a system that provides a single and comprehensive view of the whole process combining both relation extraction and later exploitation with SQL. The system permits a data warehouse style extract-transform-load of generic relations extracted from text documents and can support additional text mining analysis libraries or systems. Once generic relations are loaded, the user can define SQL queries on the extracted relations to discover higher level semantics or to join them with other relational data.For executing this powerful task, our system extends the SQL-based analytical capabilities of a columnar-based massively parallel query processing engine with a broad set of user-defined functions and a data model that supports this task. Our white-box approach permits INDREX to benefit from built-in query optimization and indexing techniques of the underlaying query execution engine.Applications that support both text mining and analytical workflows leverage new analytical platforms based on the MapReduce framework and its open source Hadoop implementation. We compare our system against this base line. We measure execution times for common workflows and demonstrate orders of magnitude improvement in execution time using INDREX.

A demonstration of iterative parallel array processing in support of telescope image analysis

In this demonstration, we present AscotDB, a new tool for the analysis of telescope image data. AscotDB results from the integration of ASCOT, a Web-based tool for the collaborative analysis of telescope images and their metadata, and SciDB, a parallel array processing engine. We demonstrate the novel data exploration supported by this integrated tool on a 1 TB dataset comprising scientifically accurate, simulated telescope images. We also demonstrate novel iterative-processing features that we added to SciDB in order to support this use-case.

Fine grain pipeline architecture for high performance phase-based optical flow computation

Accurate motion analysis of real life sequences is a very active research field due to its multiple potential applications. Currently, new technologies offer us very fast and accurate sensors that provide a huge quantity of data per second. Processing these data streams is very expensive (in terms of computing power) for general purpose processors and therefore, is beyond processing capabilities of most current embedded devices. In this work, we present a specific hardware architecture that implements a robust optical flow algorithm able to process input video sequences at a high frame rate and high resolution, up to 160 fps for VGA images. We describe a superpipelined datapath of more than 85 stages (some of them configured with superscalar units able to process several data in parallel). Therefore, we have designed an intensive parallel processing engine. System speed (frames per second) produces fine optical flow estimations (by constraining the actual motion ranges between consecutive frames) and the phase-based method confers the system robustness to image noise or illumination changes. In this work, we analyze the architecture of different frame rates and input image noise levels. We compare the results with other approaches in the state of the art and validate our implementation using several hardware platforms.

A High-Performance Three-Engine Architecture for H.264/AVC Fractional Motion Estimation

Variable-block-size motion estimation (VBSME) is one of the contributors to H.264/Advanced Video Coding (AVC)'s excellent coding efficiency. Due to its high computational complexity, however, VBSME needs acceleration for real-time high-resolution applications. We propose a high-performance hardware architecture for H.264/AVC fractional motion estimation. Our architecture consists of three parallel processing engines, one for 4 × 4 and 8 × 8 blocks, one for 8 × 4 and 4 × 8 blocks, and another for the remaining type of blocks. In addition, we propose a resource-sharing scheme which saves 33% of hardware cost for the computation of the sum of absolute transformed difference. Synthesized into a Taiwan Semiconductor Manufacturing Company (TSMC) 180-nm CMOS cell library, our 321-K gate design only needs to run at 154 MHz when encoding a 1920 ×1088 video at 30 frames per second. Compared with a most comparable previous work that consumes 311 K gates and runs at 200 MHz, our proposed architecture is more efficient.

A real‐time music synthesis engine for physical modeling of plucked string instruments.

Music synthesis continues to offer huge potential possibilities for the creation of new musical instruments. One of the promising music synthesis techniques is physical modeling which produces output sounds that resemble much more closely their physical counterparts since it offers the potential of more intuitive control. However, this results in tremendous computational and I/O requirements, prohibiting real‐time use in composition and live performance. To meet the performance requirements, we introduce a parallel processing engine. Whereas commercial digital signal processors such as TI TMS320C6x families use silicon area for large multiported register files, large caches, and deeply pipelined functional units, our parallel processing engine contain many more simple processing elements (PEs) for the same silicon area. As a result, our engine often employs thousands of PEs while possibly distributing and colocating PEs with the data I/O to minimize storage and data communication requirements. In this paper, we implemented physical modeling of a representative plucked Korean string instrument, called Gayageum, which has 12 silk strings. Our engine achieved 12‐notes music synthesis in real‐time at 44.1 kHz sampling rate for the physical modeling algorithms. This is in contrast to TI TMS320C6x, which achieves only single‐note music synthesis. [Work supported by KOSEF, R01‐2008‐000‐20493‐0.]

Integrated Memory Controllers with Parallel Coherence Streams

Previous work in scalable hardware distributed shared memory (DSM) multiprocessors has established the critical and dominant role that protocol processing bandwidth (or its inverse, occupancy) plays in determining overall performance in architectures with standalone memory/coherence controllers. However, with recent architectural trends toward integrated (on-chip) memory controllers and the well-known fact that processor frequency is increasing more rapidly than memory systems, we must ask whether parallel coherence processing engines (either multiple integrated protocol processors/cores or multiple protocol threads) are needed in DSM machines constructed from modern processor architectures and, if so, when. We construct a useful analytical model to give the designer insight into when parallel coherence streams will improve performance and verify our model via detailed simulation on 64-threaded microbenchmarks and parallel applications and on single-node multiprogrammed workloads. Surprisingly, and contrary to related work, we find that, in these architectures, adding a second coherence engine has almost no impact on performance. Further, for less-tuned applications that suffer from hot spots (contentious requests to the same memory line), additional engines offer no benefit whatsoever. Even with double the memory bandwidth (or channels), an additional coherence processing stream yields only slight performance improvement. Only for a special class of DSM machines employing directoryless broadcast protocols over unordered interconnects does parallel "snoop" processing offer reasonable performance improvement for communication-intensive applications. Overall, given the architectural trends, this is good news for DSM designers who want to minimize the resources necessary (protocol threads or integrated protocol processor cores for maintaining internode coherence, respectively) to create SMTp-based or multi-CMP-based scalable DSM machines using directory protocols.

Large detector array and real-time processing and elemental image projection of X-ray and proton microprobe fluorescence data

A detector concept is described that integrates a large solid-angle detector array developed at Brookhaven National Laboratory and a high speed pipelined parallel processing engine developed at CSIRO for machine vision, with an embedded implementation of the Dynamic Analysis method for fluorescence spectra deconvolution and image projection, to yield a detection system capable of energy-dispersive detection, spectral deconvolution and real-time elemental imaging at ∼10 8 events per second for PIXE elemental imaging using the nuclear microprobe and SXRF elemental imaging using the synchrotron X-ray microprobe.

Automatically partitioning packet processing applications for pipelined architectures

Modern network processors employs parallel processing engines (PEs) to keep up with explosive internet packet processing demands. Most network processors further allow processing engines to be organized in a pipelined fashion to enable higher processing throughput and flexibility. In this paper, we present a novel program transformation technique to exploit parallel and pipelined computing power of modern network processors. Our proposed method automatically partitions a sequential packet processing application into coordinated pipelined parallel subtasks which can be naturally mapped to contemporary high-performance network processors. Our transformation technique ensures that packet processing tasks are balanced among pipeline stages and that data transmission between pipeline stages is minimized. We have implemented the proposed transformation method in an auto-partitioning C compiler product for Intel Network Processors. Experimental results show that our method provides impressive speed up for the commonly used NPF IPv4 forwarding and IP forwarding benchmarks. For a 9-stage pipeline, our auto-partitioning C compiler obtained more than 4X speedup for the IPv4 forwarding PPS and the IP forwarding PPS (for both the IPv4 traffic and IPv6 traffic).

Multiprocessor 3D vision system for pick and place

A 3D vision system implemented upon a locally developed transputer-based hybrid parallel processing engine named MARVIN (Multiprocessor ARchitecture for VIsioN), hosted by a SUN workstation, is described. In addition to the recovery of scene descriptions from edge-based binocular stereo, the system incorporates a parallel model matching algorithm which is able to accurately locate modelled objects within such scenes. The competence of this vision system is demonstrated by visually guiding a robot arm to pick up various objects in a cluttered scene with a total processing time of approximately ten seconds.

Use of transputers in a 3-D positron emission tomograph

The use of a VME-bus-based transputer network as a parallel processing engine for positron volume imaging (PVI) is discussed. The authors find that the speedups of parallel networks depend on two major factors, the ratio of computation to communication for a task and the size of the task, and give a simple model to explore the limits on speedups. Through actual implementation it is shown that real-time PVI data acquisition can be achieved with about 20 transputer nodes, and it is estimated that three-dimensional (3-D) image reconstruction can be achieved within 10 min using 200 nodes. Larger images and a larger number of histograms can readily be accommodated using the same parallel algorithms, as the model presented places no limits on the size of the images. The versatility and scalability of transputers makes them very suitable for use in PVI tomographs in that the same transputers can be used for speeding up data acquisition, image reconstruction, and display.

A new technique for visual motion alarm

A new technique for motion detection and location is proposed and illustrated. It is intended for interactive visual surveillance systems. Absolute values of the differences between pairs of images belonging to a sequence taken by a still camera are computed. The absolute values are thresholded and projected onto horizontal, vertical, and two diagonal directions. Motion detection and location are derived from the backprojection image. The projections and the backprojection require two passes through a parallel pipeline processing engine (P 3E). Examples are presented.