Distributed Processing Environment Research Articles

An Efficient Algorithm for Fast Service Edge Selection in Cloud-Based Telco Networks

Telecommunication operators are increasingly integrating computational infrastructure into their networks at different location levels, including the network edge. This makes a highly distributed processing environment a reality, which is expected to enable next-generation services. This article proposes a novel and efficient algorithm to determine the best service execution locations through the “service edge,” a concept that groups services in categories according to their requirements and benefits from the flexibility of distributed cloud resources. The article is focused on describing the algorithm so that it can be integrated into the telco operators' management and orchestration systems. Simulation results underpin the practical feasibility of the proposed algorithm.

Implementation of a hybrid Lagrangian filtered density function–large eddy simulation methodology in a dynamic adaptive mesh refinement environment

Multispecies mixing processes play an important role in many engineering, biological, and environmental applications. Since simulating mixing flows can be useful to understand its physics and to study industrial issues, this work aims to develop the basis of a methodology able to simulate the physics of multiple-species mixing flows, using a hybrid large eddy simulation/Lagrangian filtered density function (FDF) method on an adaptive, block-structured mesh. A computational model of notional particles transport on a distributed processing environment is built using a parallel Lagrangian map. This map connects the Lagrangian information with the Eulerian framework of the in-house code MFSim, in which transport equations are solved. The Lagrangian composition FDF method, through the Monte Carlo technique, performs algebraic calculations over an ensemble of notional particles and provides composition fields statistically equivalent to those obtained by finite volume numerical solution of partial differential equations. Finally, to maintain high accuracy in the system of stochastic differential equations solver when an adaptive mesh refinement environment is used, a methodology for ensuring mass conservation is developed to preserve at least the statistical moments up to order two, even in the case of annihilation or cloning of a large number of notional particles in one time step, ensuring the applicability of Lagrangian FDF methods in dynamically adaptive grid refinement.

A fuzzy-based approach for resource management in SDN-VANETs: Effect of trustworthiness on assessment of available edge computing resources

Vehicular Ad hoc Networks (VANETs) aim to improve the efficiency and safety of transportation systems by enabling communication between vehicles and roadside units, without relying on a central infrastructure. However, since there is a tremendous amount of data and significant number of resources to be dealt with, data and resource management become their major issues. Cloud, Fog and Edge computing, together with Software Defined Networking (SDN) are anticipated to provide flexibility, scalability and intelligence in VANETs while leveraging distributed processing environment. In this paper, we consider this architecture and implement and compare two Fuzzy-based Systems for Assessment of Neighboring Vehicles Processing Capability (FS-ANVPC1 and FS-ANVPC2) to determine the processing capability of neighboring vehicles in Software Defined Vehicular Ad hoc Networks (SDN-VANETs). The computational, networking and storage resources of vehicles comprise the Edge Computing resources in a layered Cloud-Fog-Edge architecture. A vehicle which needs additional resources to complete certain tasks and process various data can use the resources of the neighboring vehicles if the requirements to realize such operations are fulfilled. The proposed systems are used to assess the processing capability of each neighboring vehicle and based on the final value, it can be determined whether the edge layer can be used by the vehicles in need. FS-ANVPC1 takes into consideration the available resources of the neighboring vehicles and the predicted contact duration between them and the present vehicle, while FS-ANVPC2 includes in addition the vehicles trustworthiness value. Our systems take also into account the neighboring vehicles’ willingness to share their resources and determine the processing capability for each neighbor. We evaluate the proposed systems by computer simulations. The evaluation results show that FS-ANVPC1 decides that helpful neighboring vehicles are the ones that are predicted to be within the vehicle communication range for a while and have medium/large amount of available resources. FS-ANVPC2 considers the same neighboring vehicles as helpful neighbors only if they have at least a moderate trustworthiness value ( VT = 0.5). When VT is higher, FS-ANVPC2 takes into consideration also neighbors with less available resources.

Optimal Task Assignment in Distributed Systems through Greedy algorithm

A distributed processing environment consists of one or more applications spread over several computers. These computers may be geographically separated from one another. The applications may be executed on different platforms using different operating systems and telecommunication protocols. In short, a distributed operating environment offers a variety of information system solutions regardless of the location of the user, user’s operating system or the equipment using by user. Performance enhancement of the distributed networks is a major and challenging problem for the researchers. In this paper, algorithm designed, have to allocate m tasks to n processors (m>n) in the environment of distributed processing. These m tasks are to be assigned on the n processors of system through Greedy algorithm of task scheduling. The starting time and finishing time of the processing of a task is considered and denoted by Si and Fi where i= 1, 2, 3, 4, 5…… m. In order to evaluate the optimal time, present algorithm obtains the set of assigned and on assigned tasks. Based on these sets allocation has been made. The method is presented into algorithmic form and several sets of input data have been implemented to test the effectiveness of the algorithm.

Computational storage: an efficient and scalable platform for big data and HPC applications

In the era of big data applications, the demand for more sophisticated data centers and high-performance data processing mechanisms is increasing drastically. Data are originally stored in storage systems. To process data, application servers need to fetch them from storage devices, which imposes the cost of moving data to the system. This cost has a direct relation with the distance of processing engines from the data. This is the key motivation for the emergence of distributed processing platforms such as Hadoop, which move process closer to data. Computational storage devices (CSDs) push the “move process to data” paradigm to its ultimate boundaries by deploying embedded processing engines inside storage devices to process data. In this paper, we introduce Catalina, an efficient and flexible computational storage platform, that provides a seamless environment to process data in-place. Catalina is the first CSD equipped with a dedicated application processor running a full-fledged operating system that provides filesystem-level data access for the applications. Thus, a vast spectrum of applications can be ported for running on Catalina CSDs. Due to these unique features, to the best of our knowledge, Catalina CSD is the only in-storage processing platform that can be seamlessly deployed in clusters to run distributed applications such as Hadoop MapReduce and HPC applications in-place without any modifications on the underlying distributed processing framework. For the proof of concept, we build a fully functional Catalina prototype and a CSD-equipped platform using 16 Catalina CSDs to run Intel HiBench Hadoop and HPC benchmarks to investigate the benefits of deploying Catalina CSDs in the distributed processing environments. The experimental results show up to 2.2× improvement in performance and 4.3× reduction in energy consumption, respectively, for running Hadoop MapReduce benchmarks. Additionally, thanks to the Neon SIMD engines, the performance and energy efficiency of DFT algorithms are improved up to 5.4× and 8.9×, respectively.

A survey on quality-assurance approximate stream processing and applications

The massive growth of data now being made available from a variety of sources leads to an increased demand for fast data processing to extract value from the data. In data streams, processing data requires computational power and data storage capabilities that have not kept pace with the data collection abilities. For these reasons approximate computations have been developed to handle both computational issues as well as the storage issues especially related to real-time data streams. In this paper, we first propose a comprehensive study of approximate computing techniques for data streams. We classify common approximate techniques as data-driven and computing-driven methods, and also discuss the combination of the two methods in emerging distributed processing environments. Based on existing approximate methods, we then detail the research on data quality management including quality evaluation and monitoring. The challenges are grouped into several research themes including pre-evaluation, data learning, approximation processing, and quality measurement. The aim of the paper is to provide researchers with a guide for how to make effective systematic strategies for approximate stream processing.

BigFeel—A Distributed Processing Environment for the Integration of Sentiment Analysis Methods

Abstract Sentiment analysis has been the main focus of plenty of research efforts, particularly justified by its commercial significance, both for consumers and businesses. Thus, many methods have been proposed so far, and the most prominent have been compared in terms of effectiveness. Nonetheless, the literature is deficient when it comes to assessing the efficiency of these methods for processing large volumes of data. In this study, we performed an experimental assessment of the efficiency of 22 methods in total, whose implementations were available. We also proposed and assessed an environment for distributed processing methods for sentiment analysis, using the Apache Spark platform, named BigFeel. In this environment, the existing methods, outlined to run in a non-distributed way, can be adapted, without altering their source code, to run in a distributed manner. The experimental results reveal that (i) few methods are efficient in their native form, (ii) the methods improve their efficiency after having been integrated into BigFeel, (iii) some of them, which were unfeasible to process a large dataset, became viable when deployed in a computer cluster and (iv) some methods can only handle small datasets, even in a distributed manner.

C2Net: A Network-Efficient Approach to Collision Counting LSH Similarity Join

Similarity join of two datasets $P$ and $Q$ is a primitive operation that is useful in many application domains. The operation involves identifying pairs $(p,q)$ , in the Cartesian product of $P$ and $Q$ such that $(p,q)$ satisfies a stipulated similarity condition. In a high-dimensional space, an approximate similarity join based on locality-sensitive hashing (LSH) provides a good solution while reducing the processing cost with a predictable loss of accuracy. A distributed processing framework such as MapReduce allows the handling of large and high-dimensional datasets. However, network cost estimation frequently turns into a bottleneck in a distributed processing environment, thus resulting in a challenge of achieving faster and more efficient similarity join. This paper focuses on collision counting LSH-based similarity join in MapReduce and proposes a network-efficient solution called C2Net to improve the utilization of MapReduce combiners. The solution uses two graph partitioning schemes: (i) minimum spanning tree for organizing LSH buckets replication; and (ii) spectral clustering for runtime collision counting task scheduling. Experiments have shown that, in comparison to the state of the art, the proposed solution is able to achieve 20 percent data reduction and 50 percent reduction in shuffle time.

분산 처리 환경에서 동적 분할 그리드 색인 기반 근사 k-최근접 질의 처리

분산 처리 환경에서 동적 분할 그리드 색인 기반 근사 k-최근접 질의 처리

Открытая система обработки текстов в наукометрии и библиометрии

The paper examines the model of an open information system for text analyzing and processing for solving scientometric and bibliometric problems, as well as special aspects of its implementation on the basis of the service-oriented platform for rapid prototyping Java Spring MVC VerliOKa. The key features of the proposed solution are the ability to implement basic algorithms for text data analyzing and processing in the form of RESTful services, executing of these processes in a distributed processing environment, creating combined algorithms based on them, and also the ability to build an applied programming interface for creating client-server systems with mobile and fixed client subsystems in the field of science and education.

함정 전투체계를 위한 분산처리 환경 기반 메시지 정의 및 관리 시스템의 개발

함정 전투체계는 생존성과 임무효과의 극대화를 위해 수집된 정보를 종합하고, 실시간으로 처리하여 최적의 전투능력을 제공하도록 고안되었다. 전투체계의 소프트웨어 컴포넌트는 함정 규모에 따라 수십 개에서 수천 개에 이르며, 각 컴포넌트 간 통신은 메시지 지향 미들웨어인 DDS(Data Distribution Service)를 통해 이뤄진다. DDS를 통한 전투체계 메시지 인터페이스는 MDMS(Message Define & Management System)를 통해 관리되어 왔으나, 체계 규모가 증가함에 따라 개발 효율이 저하되는 등 다양한 문제점들이 대두되었다. 따라서, 본 논문에서는 분산 처리 환경 운용이 가능한 차세대 메시지 정의 및 관리 시스템을 개발하고, 이를 통해 기존의 문제들을 해결하고자 한다. 실험 결과는 XMDMS(neXt Message Define & Management System)가 기존 시스템에 비해 처리 지연 및 네트워크 성능이 향상됨을 보여준다.

2-6 分散処理環境下における映像再生システムの試作

2-6 分散処理環境下における映像再生システムの試作

Distributed snapshot maintenance in wide-column NoSQL databases using partitioned incremental ETL pipelines

Wide-column NoSQL databases are an important class of NoSQL (Not only SQL) databases which scale horizontally and feature high access performance on sparse tables. With current trends towards big Data Warehouses (DWs), it is attractive to run existing business intelligence/data warehousing applications on higher volumes of data in wide-column NoSQL databases for low latency by mapping multidimensional models to wide-column NoSQL models or using additional SQL add-ons. For examples, applications like retail management can run over integrated data sets stored in big DWs or in the cloud to capture current item-selling trends. Many of these systems also employ Snapshot Isolation (SI) as a concurrency control mechanism to achieve high throughput for read-heavy workloads. SI works well in a DW environment, as analytical queries can now work on (consistent) snapshots and are not impacted by concurrent update jobs performed by online incremental Extract-Transform-Load (ETL) flows that refresh fact/dimension tables. However, the snapshot made available in the DW is often stale, since at the moment when an analytical query is issued, the source updates (e.g. in a remote retail store) may not have been extracted and processed by the ETL process in time due to high input data volume or slow processing speed. This staleness may cause incorrect results for time-critical decision support queries. To address this problem, snapshots which are supposed to be accessed by analytical queries need to be first maintained by corresponding ETL flows to reflect source updates based on given freshness needs. Snapshot maintenance in this work means maintaining the distributed data partitions that are required by a query. Since most NoSQL databases are not ACID compliant and do not provide full-fledged distributed transaction support, snapshot may be inconsistently derived when its data partitions are updated by different ETL maintenance jobs.This paper describes an extended version of HBelt system [1] which tightly integrates the wide-column NoSQL database HBase with a clustered & pipelined ETL engine. Our objective is to efficiently refresh HBase tables with remote source updates while a consistent snapshot is guaranteed across distributed partitions for each scan request in analytical queries. A consistency model is defined and implemented to address so-called distributed snapshot maintenance. To achieve this, ETL jobs and analytical queries are scheduled in a distributed processing environment. In addition, a partitioned, incremental ETL pipeline is introduced to increase the performance of ETL (update) jobs. We validate the efficiency gain in terms of data pipelining and data partitioning using the TPC-DS benchmark, which simulates a modern decision support system for a retail product supplier. Experimental results show that high query throughput can be achieved in HBelt when distributed, refreshed snapshots are demanded.

Unified Platform for AI and Big Data Analytics

This paper describes an integrated platform for machine learning and big data analysis. The integrated platform is configured in a way that builds a large distributed data processing environment in the computing environment that makes up the NVIDIA AI platform. In addition, this paper describes the background of this idea selection and the use of the software to build the unified platform. The technical details are shown in terms of how to create the proposed platform. In the anlaysis section, the methodology is provided and also the steps are explained as to how to use this integration platform. Finally, the expected effects are elaborated in the conclusion section.

Distributed data processing and analysis environment for neutron scattering experiments at CSNS

China Spallation Neutron Source (CSNS) is the first high-performance pulsed neutron source in China, which will meet the increasing fundamental research and technique applications demands domestically and overseas. A new distributed data processing and analysis environment has been developed, which has generic functionalities for neutron scattering experiments. The environment consists of three parts, an object-oriented data processing framework adopting a data centered architecture, a communication and data caching system based on the C/S paradigm, and data analysis and visualization software providing the 2D/3D experimental data display. This environment will be widely applied in CSNS for live data processing.

GRASP-based resource re-optimization for effective big data access in federated clouds

Federated cloud organizations, spanning across multiple networked sites that provide both computing and storage resources, can be considered the state-of-the-art solutions for providing multi-tenant runtime services in modern distributed processing environments. In these scenarios, by re-optimizing the communication paths between virtual machines and big data sources, at evenly spaced interval or when required by circumstances, the overall communication and runtime resource utilization on the cloud infrastructure is re-balanced, so that more virtual machines can be allowed to access the needed big data sources with adequate bandwidth, thereby significantly improving the perceived performance and quality of service. The problem of re-optimization is tackled with a powerful meta-heuristic, the greedy randomized adaptive search procedure (GRASP), augmented by path re-linking. In order to evaluate the proposed approach, extensive simulations have been performed, leading to very interesting results, demonstrating the effectiveness and validity of the underlying ideas and their applicability to real large-scale federated cloud scenarios.

무선 센서 네트워크에서의 분산 컴퓨팅 모델

Wireless sensor networks offer a distributed processing environment. Many sensor nodes are deployed in fields that have limited resources such as computing power, network bandwidth, and electric power. The sensor nodes construct their own networks automatically, and the collected data are sent to the sink node. In these traditional wireless sensor networks, network congestion due to packet flooding through the networks shortens the network life time. Clustering or in-network technologies help reduce packet flooding in the networks. Many studies have been focused on saving energy in the sensor nodes because the limited available power leads to an important problem of extending the operation of sensor networks as long as possible. However, we focus on the execution time because clustering and local distributed processing already contribute to saving energy by local decision-making. In this paper, we present a cooperative processing model based on the processing timeline. Our processing model includes validation of the processing, prediction of the total execution time, and determination of the optimal number of processing nodes for distributed processing in wireless sensor networks. The experiments demonstrate the accuracy of the proposed model, and a case study shows that our model can be used for the distributed application.

Cloud-scale genomic signals processing classification analysis for gene expression microarray data.

As microarray data available to scientists continues to increase in size and complexity, it has become overwhelmingly important to find multiple ways to bring inference though analysis of DNA/mRNA sequence data that is useful to scientists. Though there have been many attempts to elucidate the issue of bringing forth biological inference by means of wavelet preprocessing and classification, there has not been a research effort that focuses on a cloud-scale classification analysis of microarray data using Wavelet thresholding in a Cloud environment to identify significantly expressed features. This paper proposes a novel methodology that uses Wavelet based Denoising to initialize a threshold for determination of significantly expressed genes for classification. Additionally, this research was implemented and encompassed within cloud-based distributed processing environment. The utilization of Cloud computing and Wavelet thresholding was used for the classification 14 tumor classes from the Global Cancer Map (GCM). The results proved to be more accurate than using a predefined p-value for differential expression classification. This novel methodology analyzed Wavelet based threshold features of gene expression in a Cloud environment, furthermore classifying the expression of samples by analyzing gene patterns, which inform us of biological processes. Moreover, enabling researchers to face the present and forthcoming challenges that may arise in the analysis of data in functional genomics of large microarray datasets.

Electric Power Communication Network Alarm Data Acquisition System Based on CORBA Northbound Interface

With the development of communication technology, the increasing scale of power communication network is bigger and bigger. Based on the important significance of electric power communication network to establish an alarm information acquisition system is necessary. CORBA, as a distributed processing environment interconnect solutions in hardware and software, is widely used in the field of network management for its high versatility. In this paper we use real-time CORBA platform TAO as a network management platform to develop a kind of electric power communication network alarm acquisition system.

Recent Developments in Mobile Cloud Scheduling: State-of-the-Art, Challenges and Perspectives

Cloud computing became recently one of the most popular multi-layer distributed computational and data processing environments with various types of services, distributed data storages and resources. With rapid development of mobile technologies, computational clouds have been transformed into the systems with dynamically changing topology and flexible infrastructure through integration with the mobile devices and mobile users as the whole system nodes and actors. The aim of this paper is to provide a comprehensive study and critical comparative analysis of the recent developments in the Mobile Clouds with a new energy optimization criterion scheduling.