Shuffle Stage Research Articles

A two-stage adaptive penalty method based on co-evolution for constrained evolutionary optimization

Penalty function method is popular for constrained evolutionary optimization. However, it is non-trivial to set a proper penalty factor for a constrained optimization problem. This paper takes advantage of co-evolution to adjust the penalty factor and proposes a two-stage adaptive penalty method. In the co-evolution stage, the population is divided into multiple subpopulations, each of which is associated with a penalty factor. Through the co-evolution of these subpopulations, the performance of penalty factors can be evaluated. Since different penalty factors are used, the subpopulations will evolve along different directions. Thus, exploration can be enhanced. In the shuffle stage, all subpopulations are merged into a population and the best penalty factor from the co-evolution stage is used to guide the evolution. In this manner, the information interaction among subpopulations can be facilitated; thus, exploitation can be promoted. By executing these two stages iteratively, the feasible optimum could be obtained finally. In the two-stage evolutionary process, the search algorithm is designed based on two trial vector generation strategies of differential evolution. Additionally, a restart mechanism is designed to help the population avoid stagnating in the infeasible region. Extensive experiments demonstrate the effectiveness of the proposed method.

Over-the-Air Computation of Large-Scale Nomographic Functions in MapReduce Over the Edge Cloud Network

Motivated by increasing powerful edge devices with data-intensive computing and limited storage size, we study a MapReduce-based wireless distributed computing framework by allocating a portion of files in the remote data center to the network edge and utilizing computation and memory resources at the edge. Our framework is composed of three step phases: 1) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Map</i> ; 2) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Shuffle</i> ; and 3) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Reduce</i> . However, in the data shuffling stage, shuffling many data accounts for a large amount of the total running time over wireless interference networks will degrade its performance. Moreover, data shuffling between pervasive edge devices with limited spectrum bandwidth is very challenging. Today, many devices focus on computing functions rather than collecting all the individual wireless data centers. Therefore, we can use over-the-air computation (AirComp) technology to reliably compute multiple target functions by harnessing interference in the multiple-access channel with a higher computation efficiency than the traditional orthogonal multiaccess scheme that combats interference. We study a mixed-timescale optimization of the transmitting–receiving (Tx-Rx) policy and file allocation to minimize the averaged computation mean-squared error (MSE) under the power constraint of each device. File allocation control is adaptive to the long-term statistical channel state information (CSI), while the Tx-Rx policy is adaptive to the CSI and file allocation strategy. We decompose the problem into a short-term Tx-Rx policy and a long-term file allocation control problem to tackle the joint nonconvex optimization. Simulation results indicate the effectiveness of our proposed two-timescale algorithm and the advantages of our computation framework over the state-of-the-art baselines.

A Strategy of Parallel SLIC Superpixels for Handling Large-Scale Images over Apache Spark

Superpixel segmentation algorithms are widely used in the image processing field. The size of the large-scale images usually exceeds the memory of a single machine given that the size of image data has increased rapidly in recent years. This leads to big challenges for implementing sequential superpixel segmentation methods, although these algorithms have good scalability. Additionally, segmentation of large-scale images over a distributed cluster is a feasible solution. Nevertheless, it is challenging to transplant sequential superpixel algorithms directly to a distributed environment, as usually there are incomplete object problems in the border area of image tiles. To overcome the incomplete object problems, one approach is to build a distributed strategy based on a sequential SLIC superpixel segmentation algorithm over a distributed cluster organized by Apache Spark. In our research, the decomposed image tiles were divided into two categories—even tiles and odd tiles. The even tiles were first segmented by the SLIC algorithm, then the cluster centers and buffer sizes of even tiles were extracted and switched to odd tiles. During the shuffle stage, the odd tiles acquired pixels from adjacent even tiles according to the buffer sizes, and then the buffered odd tiles were segmented by the SLIC algorithm with the help of the shared cluster centers. The superpixels with shared cluster centers were generated in even tiles and remained in order to enlarge the odd tiles rather than redundant computing of specific areas to modify incomplete superpixels well. Specifically, this strategy employs the shared variables to transmit intermediate results and the shuffle operations were carried out among approximately half of the entire image tiles, which reduces the communications further. The distributed strategy was evaluated in terms of the accuracy and execution efficiency, which revealed that the proposed strategy could not only get better F-measure values but is also implemented faster relative to the repeat calculation strategy, especially for limited calculation resources. Therefore, the proposed strategy is more suitable for superpixel segmentation algorithms. In addition, this research accumulates experience for expanding the abundant sequential algorithms to the distributed environment and provides more solutions for large-scale image processing demands.

Intermediate data placement and cache replacement strategy under Spark platform

Spark is widely used due to its high performance caching mechanism and high scalability, which still causes uneven workloads and produces useless intermediate caching results when faced with data-intensive applications. A data placement strategy based on an improved reservoir sampling algorithm is proposed to solve the problem of intermediate data tilt in the shuffle stage of Spark. Compared with the traditional sampling algorithm, the amount of intermediate data is accumulated while sampling. The data skew measurement model is used to classify data into skewed data, and non-skewed and coarse-grained, and fine-grained placement algorithms are designed. To further improve Spark's system memory utilization and cache hit rate, an adaptive cache replacement algorithm is proposed to maximize cache gain. We analyze the operational dependencies and propose a cache gain model. Compared with the traditional method, the two known and unknown job arrival rates are considered separately to obtain an online adaptive cache replacement strategy that maximizes cache gain. Experimental results show that our data placement strategy effectively reduces Spark applications' execution time and improves the load balance of reduce tasks. Meanwhile, the proposed adaptive cache replacement strategy effectively reduces Spark's average completion time and improves the memory utilization and cache hit rate.

A Strategy of Parallel Seed-Based Image Segmentation Algorithms for Handling Massive Image Tiles over the Spark Platform

The volume of remote sensing images continues to grow as image sources become more diversified and with increasing spatial and spectral resolution. The handling of such large-volume datasets, which exceed available CPU memory, in a timely and efficient manner is becoming a challenge for single machines. The distributed cluster provides an effective solution with strong calculation power. There has been an increasing number of big data technologies that have been adopted to deal with large images using mature parallel technology. However, since most commercial big data platforms are not specifically developed for the remote sensing field, two main issues exist in processing large images with big data platforms using a distributed cluster. On the one hand, the quantities and categories of official algorithms used to process remote sensing images in big data platforms are limited compared to large amounts of sequential algorithms. On the other hand, the sequential algorithms employed directly to process large images in parallel over a distributed cluster may lead to incomplete objects in the tile edges and the generation of large communication volumes at the shuffle stage. It is, therefore, necessary to explore the distributed strategy and adapt the sequential algorithms over the distributed cluster. In this research, we employed two seed-based image segmentation algorithms to construct a distributed strategy based on the Spark platform. The proposed strategy focuses on modifying the incomplete objects by processing border areas and reducing the communication volume to a reasonable size by limiting the auxiliary bands and the buffer size to a small range during the shuffle stage. We calculated the F-measure and execution time to evaluate the accuracy and execution efficiency. The statistical data reveal that both segmentation algorithms maintained high accuracy, as achieved in the reference image segmented in the sequential way. Moreover, generally the strategy took less execution time compared to significantly larger auxiliary bands and buffer sizes. The proposed strategy can modify incomplete objects, with execution time being twice as fast as the strategies that do not employ communication volume reduction in the distributed cluster.

A Distributed Low-Complexity Coding Solution for Large-Scale Distributed FFT

In distributed computing, a number of available helper nodes assist in completing a task for the master node. In such setups, the failure or straggling of even a single helper node can significantly increase the processing time. Therefore, coded distributed computing has been the subject of many recent studies. A problem that arises in some setups is that the master’s decoding complexity may exceed the complexity of self-computation, rending distributed computing useless. One such case is distributed large-scale FFT, where many helper nodes are required. In this work, we propose a novel distributed coded FFT, where the master’s load is significantly lower than the existing work. The gain is obtained by (1) using a novel distributed FFT structure which allows for reliable distributed coding at the Shuffle stage, and (2) using Raptor codes which enjoy a linear complexity at the cost of a small number of extra helper nodes. Numerical results are provided to support the benefits of our proposed solution and to optimize design parameters.

Hadoop Performance Analysis Model with Deep Data Locality

Background: Hadoop has become the base framework on the big data system via the simple concept that moving computation is cheaper than moving data. Hadoop increases a data locality in the Hadoop Distributed File System (HDFS) to improve the performance of the system. The network traffic among nodes in the big data system is reduced by increasing a data-local on the machine. Traditional research increased the data-local on one of the MapReduce stages to increase the Hadoop performance. However, there is currently no mathematical performance model for the data locality on the Hadoop. Methods: This study made the Hadoop performance analysis model with data locality for analyzing the entire process of MapReduce. In this paper, the data locality concept on the map stage and shuffle stage was explained. Also, this research showed how to apply the Hadoop performance analysis model to increase the performance of the Hadoop system by making the deep data locality. Results: This research proved the deep data locality for increasing performance of Hadoop via three tests, such as, a simulation base test, a cloud test and a physical test. According to the test, the authors improved the Hadoop system by over 34% by using the deep data locality. Conclusions: The deep data locality improved the Hadoop performance by reducing the data movement in HDFS.

ENHANCEMENT ON TRAFFIC AWARE PARTITION IN MAPREDUCE USING CLUSTERING TECHNIQUES

The Map reduce is a programming model for handling and processing the huge data sets using map and reduce tasks in parallel distributing. To increase the execution of mapreduce many number of activities have been made, but they ignore to deal with network traffic produced in shuffle stage. The existing mapreduce traffic-aware partitions suffer from partition skew issue, where the output of map tasks is unevenly distributed among reduces tasks. Existing arrangements take after a comparative rule that re partitions workload among diminish undertakings. In any case, those methodologies frequently cause elite overhead because of the segment estimate expectation and re-partitioning. The proposed work chooses dynamic data aware parallel with k-Means algorithm (DDAP-kM), a framework that provides dynamic partitioning skew reduction and clustering map reduce jobs. These works cope with partitioning skew by adjusting run time resource allocation to reduce tasks. By the experimental results network traffic cost is compared in terms of traffic aware partition algorithm and DDAP-kM algorithm.

Efficient MapReduce algorithms for triangle listing in billion-scale graphs

This paper addresses the classical triangle listing problem, which aims at enumerating all the tuples of three vertices connected with each other by edges. This problem has been intensively studied in internal and external memory, but it is still an urgent challenge in distributed environment where multiple machines across the network can be utilized to achieve good performance and scalability. As one of the de facto computing methodologies in distributed environment, MapReduce has been used in some of existing triangle listing algorithms. However, these algorithms usually need to shuffle a huge amount of intermediate data, which seriously hinders their scalability on large scale graphs. In this paper, we propose a new triangle listing algorithm in MapReduce, FTL, which utilizes a light weight data structure to substantially reduce the intermediate data transferred during the shuffle stage, and also is equipped with multiple-round techniques to ease the burden on memory and network bandwidth when dealing with graphs at billion scale. We prove that the size of the intermediate data can be well bounded near to the number of triangles in the graph. To further reduce the shuffle size and memory cost, we also propose improved algorithms based on a compact data structure, and present several optimization techniques to accelerate the computation and reduce the memory consumption. The extensive experimental results show that our algorithms outperform existing competitors by several times on both synthetic graphs and real world graphs.

A novel encryption scheme for colored image based on high level chaotic maps

In the following paper, the novel method for color image encryption has proposed based on high level chaotic maps . We introduced two novel chaotic maps "Cosinus-Arcsinus (CA)" and "Sinus-Power Logistic (SPL)", which have better chaotic behaviour against other available chaotic maps. Our scheme like other image encryption schemes has two main phases, which are pixel shuffling and pixel diffusion. We made an efficient chaotic permutation method, which is extremely dependent on plain image. The proposed method compared with other availabale permutation methods in pixel shuffling stage has a better performance with a lower computational overhead. Another advantage of our permutation method is less correlation between adjacent pixels in the permuted image, which causes high confusion levels with lower iterations. In pixel diffusion phase, we introduced coupled map based on SPL map to change each color component distinctly. Moreover, we used CA map to generate three chaotic sequences for deriving initial seeds of coupled map in a random manner. Followed by three chaotic matrix will be created to change pixels color component values. The proposed diffusion phase implies desirable uniformly distributation in color histogram of encrypted image and makes the scheme robust against statistical attacks. In addition, creating secret keys in a large size and high sensitivity to the original pictures leads acceptable results in the average of NPCR (99.67), UACI (33.45), and resistance against brute-force attack. The experimental results reveal that the new image encryption scheme has the charectristics of `secure encryption algorithm' such as, large key space, high security and high sensitivity.

Phase–Reconfigurable Shuffle Optimization for Hadoop MapReduce

Hadoop MapReduce is a leading open source framework that supports the realization of the Big Data revolution and serves as a pioneering platform in ultra large amount of information storing and processing. However, tuning a MapReduce system has become a difficult task because a large number of parameters restrict its performance, many of which are related with shuffle, a complicated phase between map and reduce functions, including sorting, grouping, and HTTP transferring. During shuffle phase, a large mount of time is spent on disk I/O due to the low speed of data throughput. In this paper, we build a mathematical model to judge the computing complexity of different operating orders within map-side shuffle, so that a faster execution can be achieved through reconfiguring the order of sorting and grouping. Furthermore, a three-dimensional exploring space of the performance is expanded, with which, some sampled features during shuffle stage, such as key number, spilling file number, and the variances of intermediate results, are collected to support the evaluation of computing complexity of each operating order. Thus, an optimized reconfiguration of map-side shuffle architecture can be achieved within Hadoop without extra disk I/O induced. Comparing with the original Hadoop implementation, the results show that our reconfigurable architecture gains up to 2.37χ speedup to finish the map-side shuffle work.

Design and Implementation of an Efficient General-Purpose Median Filter Network

Median filtering, an image enhancement technique, is commonly used to filter out certain types of noise from an image. In this technique each pixel value in the image is replaced by the median value of the pixels in a window centered around the pixel. This paper discusses the design and implementation of median filters using pipelined multistage networks. The first part of the multistage network is a bitonic sequencer, followed by a shuffle stage feeding the bitonic sorter. The bitonic sorter shuffles the data at every stage, which is implemented as an Omega network. Earlier designs for such filters have used a number of basic unit cells interconnected in a sorting network. In the best known VLSI implementation currently in use, the delay and the number of basic unit cells, increase as O(n) and O(n2), respectively, for an (n = s × s) window of pixels. The improvements in the basic unit cell and the network layout topology presented here reduce the number of cells used to O(n log2n) and the delay to O(log2n).