Distributed Processing Method Research Articles

FUZZY MODEL OF ELECTRICITY CONTROL WITH WIRELESS INFORMATION PROCESSED ON GPU

Methods of processing information transmitted through wireless networks with software development are investigated in the work. Innovative methods of data transmission such as optical technologies, quantum data transmission and wireless data transmission technologies are disclosed. It is noted that in the modern understanding, the concept of distributed computing defines the process of convergence (convergence) of distributed processing methods, such as GRID, cloud and fog computing, with the combination of virtual cluster systems (grid clusters, cloud clusters and fog clusters) into a single information communication and computing system . It is emphasized that, unlike cellular modems, ZigBee technology nodes have microcontrollers with a pre-installed operating system and flash memory, which allows solving simple computational tasks in real time before sending data. It is advisable to solve such tasks within the framework of a multi-agent approach, which will increase the efficiency of the use of sensor nodes and the entire sensor network. The advantages of the multi-agent technology of fog computing based on sensor nodes of the wireless network of the ZigBee standard are revealed. The method of multi-agent processing of sensory information and its main components are described. The architecture of the system of distributed sensor data processing is outlined, which includes 4 hardware and software levels: Terminal sensor nodes and controllers of measuring devices and automation devices that implement fuzzy calculations; Coordinators, sensor segment routers and cellular modems that collect, protect and transmit sensor data to the processing center; A data processing center that includes a cluster of servers for GRID calculations and a cloud data storage server; Client devices to access cloud storage, computing cluster servers, and distributed fog computing terminals. It is emphasized that indicators and forecast results can be stored on distributed sensor nodes or transmitted for accumulation in cloud storage for further extraction and intelligent processing in the GRID cluster of the data center.

Scalability improvement of simplified, secure distributed processing with decomposition data

Much research on secure and safe AI analysis methods for users has been conducted. They are 1) secure multiparty computation (SMC), 2) quasi-homomorphic encryption, 3) federated learning (FL), and so on. It is known that both utility and confidentiality are essential for machine learning using confidential data. However, there is a trade-off between them. In the previous paper, we proposed a learning method for secure distributed processing using decomposition data and parameters. The characteristic feature of this method is to achieve high confidentiality through distributed processing with decomposed data and parameters. On the other hand, the realization of machine learning through the distribution and integration of decomposition data leads to an increase in computational complexity and a deterioration of computational accuracy. In particular, while increasing the number of servers improves confidentiality, this problem becomes more pronounced. In this paper, we propose scalability improvement of simplified, secure distributed processing methods of BP and NG methods to suppress the computational complexity associated with an increase in the number of servers and demonstrate its effectiveness through a numerical simulation.

DOA Estimation for Heterogeneous Wideband Sources Based on Adaptive Space-Frequency Joint Processing

For direction-of-arrival (DOA) estimation of heterogeneous wideband sources, we propose a new adaptive space-frequency joint processing algorithm. The algorithm is implemented in the sparse Bayesian learning (SBL) DOA framework, which is named as ASF-SBL algorithm in this paper. The traditional SBL-based DOA methods suffer from the problem of erroneous DOA estimation due to the structural mismatch between the invariant prior and the sparse coefficients. To solve this problem, the ASF-SBL algorithm employs a new space-frequency correlation prior model that can be adaptively changed to fit heterogeneous DOA scenarios. Specifically, nine alternative space-frequency structural patterns are constructed to represent the joint space-frequency characteristics of spatial sparse signals. By evaluating the space-frequency correlation of the sparse coefficients updated in each iteration under SBL framework, a suitable pattern is selected from the nine choices to determine the adaptive prior of each coefficient. This adaptive method leads to accurate DOA estimation in different wideband sources scenarios. In addition, we introduce a distributed processing method to extend the ASF-SBL algorithm to two-dimensional DOA estimation. This extension is achieved by decoupling the DOA estimation into two one-dimensional estimations. The decoupling avoids the problems of a huge redundant dictionary and excessive computational complexity caused by the combination of azimuth and elevation angles. Numerical simulations show that the ASF-SBL algorithm is superior to existing algorithms in DOA estimation of heterogeneous sources.

Distributed processing method for deep learning in wireless sensor networks

Distributed processing method for deep learning in wireless sensor networks

Distributed processing method for multi‐GNSS/SINS integration system

Integration of global navigation satellite system (GNSS) and strap-down inertial navigation system (SINS) in pseudo-range and pseudo-range rate measurements level has been demonstrated as an effective way to provide more reliable navigation solutions. Under this condition, the dimension of measurements matrix of the integration filter increases dramatically along with more GNSSs included in the integration, traditional centralized integration filter processing method cannot address the computational load well. In this study, a distributed integration filter processing method is proposed for the integration of multiple GNSSs and SINS using the pseudo-range and pseudo-range rates as the measurements. Measurement difference inner a same satellite constellation is conducted to remove the clock related variables, which reduces the dimension of the state vector. Secondly, a federated Kalman filter is employed to obtain global optimal estimation of the state variables, which reduces the measurement vector dimension. Mathematical model of the distributed processing model with the simplified integration filter scheme is given in detail. Through the simulation based on a dynamic trajectory, the performance of the proposed method is investigated and compared with the centralized integration processing method. Results show that new method has analogous position and velocity accuracy with better computation efficiency.

Active Sampling Rate Calibration Method for Acoustic Sensor Networks

The sampling rate mismatches among nodes severely degrade the performances of distributed signal processing methods in acoustic sensor networks. An active sampling rate calibration method based on the multiple signal classification (MUSIC) algorithm is proposed to solve this problem. Specifically, a pre-defined sinusoidal signal is used as the calibration sound source, and the sampling rate mismatch problem is formulated based on the Hadamard product between output signals of pairwise nodes. The Hadamard product is then filtered through the pre-designed filters to obtain the target sinusoidal signals whose frequencies linearly correlate with the sampling rate mismatch. Finally, the sampling rate differences among nodes are estimated by the root-MUSIC algorithm and compensated by the sinc-interpolation. The proposed method can effectively calibrate the sampling rate mismatch even under severe noise and reverberation. Moreover, it has a strong tolerance for deviations among the microphone frequency responses. Experimental results reveal the validity of the proposed method.

Edge computing health model using P2P-based deep neural networks

Currently, significant amounts of IoT data are being collected as big data for deep neural network learning algorithms that are used to extract meaningful information from big data and applied to various areas. However, a neural network’s over-fitting problem and rising computational costs associated with high levels of time complexity are obstacles to deep learning computations. Because of these problems, response delays are prevalent in big data learning processes and data extraction processes that use deep neural networks, which exponentially increase the cost of data extraction. Therefore, the amount of computation must be controlled so that data processing models can be used practically, and various dedicated devices can be used to process big data, including methods such as distributed processing methods. In general, a P2P method is a typical distributed processing method. P2P is based on traffic distribution and cooperation, and edge computing is a method that increases efficiency by locally processing large volumes of data produced and only transmitting essential information. If P2P method-based edge computing techniques are used, an effective parallel processing model can be constructed, reducing the computing requirements of a central server and the load of a network. In this paper, we propose an edge computing health model using P2P-based deep neural networks. The proposed method is used to process health big data in edge computing. To construct the model, multiple edge nodes are required, and the edge node modules are deployed at locations where health data are produced and directly connected to the deep neural network model. At the edge nodes, a modularized deep neural network is constructed, thereby implementing a parallel big data processing system. In addition, a single server is required for gathering the results of a neural network model, and because the server only collects evaluation results and provides them to the users, the response time delay is improved. In this paper, to evaluate an improved response time, a regular server model and an edge computing health model are constructed separately, and positive numerical values are extracted from the experiments. The extraction results confirm that a combination of parallel processing models and deep neural network techniques can be used to distribute and process computing operations of big data size, ensuring an effective system for reducing response time delay.

BigFeel—A Distributed Processing Environment for the Integration of Sentiment Analysis Methods

Abstract Sentiment analysis has been the main focus of plenty of research efforts, particularly justified by its commercial significance, both for consumers and businesses. Thus, many methods have been proposed so far, and the most prominent have been compared in terms of effectiveness. Nonetheless, the literature is deficient when it comes to assessing the efficiency of these methods for processing large volumes of data. In this study, we performed an experimental assessment of the efficiency of 22 methods in total, whose implementations were available. We also proposed and assessed an environment for distributed processing methods for sentiment analysis, using the Apache Spark platform, named BigFeel. In this environment, the existing methods, outlined to run in a non-distributed way, can be adapted, without altering their source code, to run in a distributed manner. The experimental results reveal that (i) few methods are efficient in their native form, (ii) the methods improve their efficiency after having been integrated into BigFeel, (iii) some of them, which were unfeasible to process a large dataset, became viable when deployed in a computer cluster and (iv) some methods can only handle small datasets, even in a distributed manner.

PPR-partitioning: a distributed graph partitioning algorithm based on the personalized PageRank vectors in vertex-centric systems

Relations among data items can be modeled with graphs in most of big data sets such as social networks’ data. This modeling creates big graphs with many vertices and edges. Balanced k-way graph partitioning is a common problem with big graphs. It has many applications in several fields. There are many approximate solutions for this problem; however, most of them do not have enough scalability for big graph partitioning and cannot be executed in a distributed manner. Vertex-centric model has been introduced recently as a scalable distributed processing method for big graphs. There are a few methods for graph partitioning based on this model. Existing approaches only consider one-step neighbors of vertices for graph partitioning and do not consider neighbors with higher steps. In this paper, a distributed method is introduced based on vertex-centric model for balanced k-way graph partitioning. This method applies the personalized PageRank vectors of vertices and partitions to decide how vertices are joined partitions. This method has been implemented in the Giraph system. The proposed method has been evaluated with several synthetic and real graphs. Experimental results have shown that this method has scalability for partitioning big graphs. It was also found that this method produces partitions with higher quality compared to the state-of-the-art stream-based methods and distributed methods based on vertex-centric programming model. Its result is close to the results of Metis method.

Full Support for Efficiently Mining Multi-Perspective Declarative Constraints from Process Logs

Declarative process management has emerged as an alternative solution for describing flexible workflows. In turn, the modelling opportunities with languages such as Declare are less intuitive and hard to implement. The area of process discovery covers the automatic discovery of process models. It has been shown that the performance of process mining algorithms, particularly when considering the multi-perspective declarative process models, are not satisfactory. State-of-the-art mining tools do not support multi-perspective declarative models at this moment. We address this open research problem by proposing an efficient mining framework that leverages the latest big data analysis technology and builds upon the distributed processing method MapReduce. The paper at hand further completes the research on multi-perspective declarative process mining by extending our previous work in various ways; in particular, we introduce algorithms and descriptions for the full set of commonly accepted types of MP-Declare constraints. Additionally, we provide a novel implementation concept allowing an easy introduction and discovery of customised constraint templates. We evaluated the mining performance and effectiveness of the presented approach on several real-life event logs. The results highlight that, with our efficient mining technique, multi-perspective declarative process models can be extracted in reasonable time.

하둡 및 Spark 기반 공간 통계 핫스팟 분석의 분산처리 방안 연구

하둡 및 Spark 기반 공간 통계 핫스팟 분석의 분산처리 방안 연구

Distributed processing of a GPS receiver network for a regional ionosphere map

This paper proposes a distributed processing method applicable to GPS receivers in a network to generate a regional ionosphere map accurately and reliably. For accuracy, the proposed method is operated by multiple local Kalman filters and Kriging estimators. Each local Kalman filter is applied to a dual-frequency receiver to estimate the receiver’s differential code bias and vertical ionospheric delays (VIDs) at different ionospheric pierce points. The Kriging estimator selects and combines several VID estimates provided by the local Kalman filters to generate the VID estimate at each ionospheric grid point. For reliability, the proposed method uses receiver fault detectors and satellite fault detectors. Each receiver fault detector compares the VID estimates of the same local area provided by different local Kalman filters. Each satellite fault detector compares the VID estimate of each local area with that projected from the other local areas. Compared with the traditional centralized processing method, the proposed method is advantageous in that it considerably reduces the computational burden of each single Kalman filter and enables flexible fault detection, isolation, and reconfiguration capability. To evaluate the performance of the proposed method, several experiments with field collected measurements were performed.

Spatiotemporal Tracking of Ocean Current Field With Distributed Acoustic Sensor Network

A distributed networked underwater sensor (DNUS) system can provide ocean measurements over a wide area with a large number of sensors. This paper studies the estimation of the ocean current field observed by a DNUS system. Considering that the current field is correlated in time and space, we present a statistically-based acoustic travel time difference tomography method based on a Kalman filter (KF) to reconstruct and track the ocean current field. A spatiotemporal autoregressive (AR) model is used to describe the time evolution of the current field. In the spatiotemporal AR model, the observation region is divided into subtriangle grids. The subtriangles are partitioned into clusters and each cluster is assigned with one AR coefficient. Moreover, the AR coefficients are updated adaptively with the past estimated current velocities. The proposed method is verified with the synthetic observational data generated by a barotropic ocean model. Compared with the regular distributed processing method, the proposed ocean current field reconstruction and tracking method achieves a lower region-integrated root-mean-square error (RMSE). In addition, by making use of the spatiotemporal correlation, the proposed method is robust to the measurement link failure and burst errors in the DNUS system.

Observation data processing methods in SINS/multi-GNSS integrated navigation simulation

In this paper, two methods are proposed, namely the unified processing method and the distributed processing method, to process the global navigation satellite system observation data in integrated navigation simulation which uses the strapdown inertial navigation system as the reference system and the multi-global navigation satellite systems as the sub-systems. The unified processing method takes all the global navigation satellite systems as a whole as one sub-system while the distributed processing method takes each global navigation satellite system as one sub-system. The centralized filter and federated filter are utilized respectively to process the global navigation satellite system observation data in the unified processing method and the distributed processing method. The mathematical models of the unified processing method and the distributed processing method are given in detail. Through theoretical derivation and mathematical simulations, the performances of the unified processing method and the distributed processing method are investigated and compared, showing that while they have the same position (velocity) accuracy, the distributed processing method offers better efficiency than the unified processing method especially when the number of global navigation satellite systems is large (>3).

Spark based Distributed Processing Method for Sharing Big Traffic Data in Realtime

이 논문에서는 사용자(차량 및 이륜차 운전자, 보행자) 들에게 교통안전 및 편의 서비스를 제공하기 위해 사용자 각자의 위치와 사용자의 스마트 기기 및 전용 단말기로 수집한 교통 이벤트를 실시간으로 서로 공유하는 방법을 제안한다. 사용자들의 위치 및 수집한 교통 이벤트는 실시간으로 서버로 전송되고, 서버는 전송된 교통 이벤트를 위치별, 시간별로 색인하여 유사하다고 판단되는 교통 이벤트를 분류하여 저장한다. 이 논문에서는 사용자의 위치 데이터 및 교통 이벤트 스트림을 실시간으로 처리하기 위해서 위치데이터에 대한 색인 기법과 이를 이용한 연속질의 기법을 제안한다. 제안하는 색인 기법을 이용해서 유사 교통 이벤트를 빠르게 분류하여 불필요한 교통 이벤트 공유를 피한다. 제안하는 방법을 평가하기 위해서 실험을 통해 유사한 교통정보에 대한 분류 기법과 실시간 공유가 가능한지를 보기 위해 연속질의 처리 결과의 정확성을 측정한다.

In-memory Distributed Processing Method for Traffic Big Data to Analyze and Share Traffic Events in Real Time among Social Groups

In this paper, we propose an in-memory distributed processing method that can rapidly process vehicle location and traffic event data using Spark Streaming. The proposed system enables to share information about surrounding vehicles, pedestrians, and traffic events in real time with drivers who use the WEVING service. In the proposed method, vehicle location and traffic event streams are indexed using the grid indexing technique according to time, and the continuous range query method is processed based on the index. Also, traffic events are grouped based on occurrence time, location, content, and road segment of the traffic event transferred in real time in order to avoid duplicated traffic events. Through experiments, we show that the proposed method is able to deduplicate similar traffic events efficiently.

Large-Scale Data Mining and Distributed Processing in Big Data Internet

With the development of Internet, various Internet-based large-scale data are facing increasing competition. With the hope of satisfying the need of data query, it is necessary to use data mining and distributed processing. As a consequence, this paper proposes a large-scale data mining and distributed processing method based on decision tree algorithm.

위성 데이터 분산처리 시스템 최적화 및 안정화

The goal of this paper is to provide performance improvement and stability for satellite data correction of some distortions due to cloud or radiance through distributed processing on cluster. To do this, we proposed and implemented SGE(Sun Grid Engine) based distributed processing methods using local storages and a status table. In the verification, the experiment result revealed that the proposed system on seven nodes improved the processing speed by 138.81% as compare to the existing system and provided good stability as well. This result showed that the proposed distributed processing work is more appropriate to process CPU bound jobs than I/O bound jobs. We expect that the proposed system will give scientists improved analysis performance in various fields and near-real time analysis services.

An AIPS-based, distributed processing method for large radio interferometric datasets

The data output rates of modern radio interferometric telescopes make the traditional data reduction process impractical in many cases. We report on the implementation of a lightweight infrastructure, named AIPSLite, that enables the deployment of AIPS interferometric processing routines on distributed systems in an autonomous and fault tolerant manner. We discuss how this approach was used to search for sources of 6.7 GHz methanol maser emission in the Cep A region with the European VLBI Network (EVN). The field was searched out to a radius of 1.25 min^(−1) at milli-arcsecond spatial resolution and 1024 frequency channels with 0.088 km s^(−1) velocity resolution. The imaged data was on the order of 30 TB. Processing was performed on 128 processors of the Irish Centre for High End Computing (ICHEC) linux cluster with a run time of 42 h, and a total of 212 CPU days.

全解探索型計算の分散処理手法

全解探索型計算の分散処理手法