Shuffle Operation Research Articles

Very Large-Scale Neighborhood Search for Steel Hot Rolling Scheduling Problem With Slab Stack Shuffling Considerations

We study a steel hot rolling scheduling (HRS) problem considering slab stack shuffling (SSS), which is a kind of key issues in the steel strip production. The HRS problem is to select suitable slabs for a predefined sequence of hot rolling slots, from their respective candidate slab sets so that the number of shuffling operations of slabs is minimized. Different from the most researches, we consider a situation where there are intersections among candidate slab sets and shuffled slabs will not return original stacks. Basing on a special index method of slabs, we present an integer linear programming (ILP) to compute approximation solutions of the problem. In order to improve the solutions obtained by the ILP model, we propose a very large-scale neighborhood (VLSN) search algorithm. In the VLSN search process, the solutions of HRS problems are improved by an iteration procedure that partial solutions are interactively destroyed and repaired. In each iteration, partitioned hot rolling slots correspond to a sub-problem. In the repair operations, we propose an efficient branch-and-bound algorithm for solving sub-problem. The computational results on a number of different scale simulated instances show that the VLSN search algorithm is efficient for solving this kind of HRS problems.

Automedian sets of permutations: direct sum and shuffle

Given a set A⊆Sn of m permutations of {1,2,…,n} and a distance function d, the median problem consists in finding the set M(A) of all the permutations that are the “closest” of this set A. In this article we study the automedian case of the problem, i.e. when A=M(A), under the Kendall-τ distance. We show that automedian sets of permutations are closed under the direct sum operation and also, when some balancing properties are imposed on these sets, under the shuffle operation. These results allow us to derive a parallel algorithm that computes the medians of any separable set of permutations in O(k!+mn), where k is the length of its longest inseparable component.

Low-dose dental CT image enhancement using a multiscale feature sensing network

Purpose:The damage to humans caused by CT radiation dose has always been a health concern. Consequently, people prefer to use low-dose CT for examinations. A common method of dose reduction in CT is to reduce the number of samples of the projection data. However, when the dose is reduced by downsampling, artifacts and noise appear in the reconstructed CT images, degrading their quality significantly and resulting in information loss. In this work, we propose a deep learning-based method to improve the quality of low-dose dental CT images. Method:The proposed method involves a new deep neural network whose backbone is composed of two blocks; each basic block in network is designed to have three paths that sense and capture image features at different scales. The large-scale feature information is used to correct the extraction and learning of the smaller-scale information. Moreover, a shuffling operation is introduced to better preserve the feature information when expanding the size of the feature map. Results:We compare the proposed network with the DnCNN and REDCNN networks on a clinical dental CT dataset. The comparison and analysis of various quantitative results and image details in the experiment show that the proposed network corrects artifacts, suppresses noise and preserves image detail information more accurately than do the compared networks. In addition, the proposed network alleviates the over-smoothing phenomenon in the restored images to a certain extent. Conclusions:We conducted research on incompletely sampled dental CT data and proposed a multiscale feature-sensing deep neural network to improve the quality of low-dose dental CT images. The experimental results demonstrate that the proposed network effectively enhances the quality of low-dose dental CT images, making it helpful for promoting wider applications of low-dose technology in cone-beam dental CT.

Magnet

Over the past decade, Apache Spark has become a popular compute engine for large scale data processing. Similar to other compute engines based on the MapReduce compute paradigm, the shuffle operation, namely the all-to-all transfer of the intermediate data, plays an important role in Spark. At LinkedIn, with the rapid growth of the data size and scale of the Spark deployment, the shuffle operation is becoming a bottleneck of further scaling the infrastructure. This has led to overall job slowness and even failures for long running jobs. This not only impacts developer productivity for addressing such slowness and failures, but also results in high operational cost of infrastructure. In this work, we describe the main bottlenecks impacting shuffle scalability. We propose Magnet, a novel shuffle mechanism that can scale to handle petabytes of daily shuffled data and clusters with thousands of nodes. Magnet is designed to work with both on-prem and cloud-based cluster deployments. It addresses a key shuffle scalability bottleneck by merging fragmented intermediate shuffle data into large blocks. Magnet provides further improvements by co-locating merged blocks with the reduce tasks. Our benchmarks show that Magnet significantly improves shuffle performance independent of the underlying hardware. Magnet reduces the end-to-end runtime of Linkedln's production Spark jobs by nearly 30%. Furthermore, Magnet improves user productivity by removing the shuffle related tuning burden from users.

Real-Time Classification of Rubber Wood Boards Using an SSR-Based CNN

The classification of wood types plays an important role in many fields, especially in construction industry and furniture manufacturing. In order to manufacture rubber wood furniture with highly uniform color and texture, wood boards of different colors and textures should be classified elaborately. Many traditional methods have been applied in wood classification relying on extracting features using handcrafted descriptors designed by experienced experts, but it is not easy to construct robust features in various conditions. In this article, we present a split-shuffle-residual (SSR)-based CNN that can learn features automatically from wood images for real-time classification of rubber wood boards. Specifically, we introduce an SSR module that combines channel split and shuffle operations with residual structure to reduce the computation cost while maintaining high classification accuracy. In each module, the input is split into two low-dimensional branches, and the channel shuffle operation is used to enable the information communication between the input and the two separated branches, which is regarded as the feature reuse that enlarges network capacity without increasing complexity. The comprehensive experiments demonstrate that our algorithm outperforms other traditional classification methods and the state-of-the-art deep learning classification networks, yielding an accuracy of 94.86%. Furthermore, the analysis of running time indicates that the SSR-based CNN can be employed for wood classification in real time, which takes only 26.55 ms to handle a single image.

Evolutionary Aggregation Approach for Multihop Energy Metering in Smart Grid for Residential Energy Management

The communication infrastructure is an important part to provide the reliability for energy management in the smart grid environment. With the aim of reducing the infrastructure cost for residential energy management, this article introduces a more complex multihop wireless remote metering network model. A novel evolutionary aggregation algorithm (EAA) is proposed to obtain the minimum number and locations of the local data centers (powerful nodes) in a 2-hop wireless remote metering network which has an arbitrary number of smart meters (ordinary nodes) with arbitrary transmission ranges. In the novel 2-hop EAA, the article designs and implements two novel adaptive operations (the switch operation and the shuffle operation) to improve the algorithm performance. Then the article extends the 2-hop EAA method to a more generic n-hop EAA which could obtain the optimal result in an n-hop (n > 2) smart meter network. Comprehensive case studies and numerical statistical analyses demonstrate that the EAA could efficiently achieve the optimal results in an n-hop (n> = 2) smart meter network environment; and the novel switch and shuffle operations could efficiently improve the performance of the evolutionary algorithm. The connectivity of the smart meter network could be fulfilled with the minimum number of the powerful nodes, from which the infrastructure cost for residential energy network could be minimized.

Algebraic Properties of Parikh Matrices of Binary Picture Arrays

A word is a finite sequence of symbols. Parikh matrix of a word is an upper triangular matrix with ones in the main diagonal and nonnegative integers above the main diagonal which are counts of certain scattered subwords in the word. On the other hand, a picture array, which is a rectangular arrangement of symbols, is an extension of the notion of a word to two dimensions. Parikh matrices associated with a picture array have been introduced, and their properties have been studied. Here, we obtain certain algebraic properties of Parikh matrices of binary picture arrays based on the notions of power, fairness, and a restricted shuffle operator extending the corresponding notions studied in the case of words. We also obtain properties of Parikh matrices of arrays formed by certain geometric operations.

Load Balancing Mechanism Based on Linear Regression Partition Prediction in Spark

With the rise of cloud computing, the internet of things, social networks and other technologies, information data is expanding rapidly, and traditional processing and storage systems have been difficult to deal with massive data. Spark is a fast and efficient MapReduce implementation developed after Hadoop. However, the shuffle operation in spark will cause the data set on some working nodes to be too large, while other nodes may be idle, which will affect the performance of spark jobs. This phenomenon is called data skew. Aiming at the problem of data skew in spark platform, this paper proposes a load balancing mechanism based on linear regression partition prediction: SP-LRP (Spark load balancing mechanism based on Linear Regression Partition), SP-LRP predicts the partition size of the reduce tasks at run-time, leverages the skew detection algorithm to identify the skew partition and adjusts the task resource allocation according to the fine-grained resource allocation algorithm. We use the benchmark dataset to evaluate SP-LRP, and compare the average execution time of two algorithms and the load balancing degree of reducers in different situations. The experimental confirmed the efficiency of SP-LRP in their respective usage scenarios.

Hyperspectral Image Classification Based on a Shuffled Group Convolutional Neural Network with Transfer Learning

Convolutional neural networks (CNNs) have been widely applied in hyperspectral imagery (HSI) classification. However, their classification performance might be limited by the scarcity of labeled data to be used for training and validation. In this paper, we propose a novel lightweight shuffled group convolutional neural network (abbreviated as SG-CNN) to achieve efficient training with a limited training dataset in HSI classification. SG-CNN consists of SG conv units that employ conventional and atrous convolution in different groups, followed by channel shuffle operation and shortcut connection. In this way, SG-CNNs have less trainable parameters, whilst they can still be accurately and efficiently trained with fewer labeled samples. Transfer learning between different HSI datasets is also applied on the SG-CNN to further improve the classification accuracy. To evaluate the effectiveness of SG-CNNs for HSI classification, experiments have been conducted on three public HSI datasets pretrained on HSIs from different sensors. SG-CNNs with different levels of complexity were tested, and their classification results were compared with fine-tuned ShuffleNet2, ResNeXt, and their original counterparts. The experimental results demonstrate that SG-CNNs can achieve competitive classification performance when the amount of labeled data for training is poor, as well as efficiently providing satisfying classification results.

DRGAN: a deep residual generative adversarial network for PET image reconstruction

Positron emission tomography (PET) image reconstruction from low-count projection data and physical effects is challenging because the inverse problem is ill-posed and the resultant image is usually noisy. Recently, generative adversarial networks (GANs) have also shown their superior performance in many computer vision tasks and attracted growing interests in medical imaging. In this work, the authors proposed a novel model [deep residual generative adversarial network (DRGAN)] based on GANs for the reduction of streaking artefacts and the improvement of PET image quality. An innovative feature of the proposed method is that the authors trained a generator to produce `residual PET map' (RPM) for image representation, rather than generate PET images directly. DRGAN used two discriminators (critics) to enforce anatomically realistic PET images and RPM. To better boost the contextual information, the authors designed residual dense connections followed with pixel shuffle operations (RDPS blocks) that encourage feature reuse and prevent losing resolution. Both simulation data and real clinical PET data are used to evaluate the proposed method. Compared with other state-of-the-art methods, the quantification results show that DRGAN can achieve better performance in bias-variance trade-off and provide comparable image quality. Their results were rigorously evaluated by one radiologist at the Shanxi Cancer Hospital.

Go Wider: An Efficient Neural Network for Point Cloud Analysis via Group Convolutions

In order to achieve a better performance for point cloud analysis, many researchers apply deep neural networks using stacked Multi-Layer-Perceptron (MLP) convolutions over an irregular point cloud. However, applying these dense MLP convolutions over a large amount of points (e.g., autonomous driving application) leads to limitations due to the computation and memory capabilities. To achieve higher performances but decrease the computational complexity, we propose a deep-wide neural network, named ShufflePointNet, which can exploit fine-grained local features, but also reduce redundancies using group convolution and channel shuffle operation. Unlike conventional operations that directly apply MLPs on the high-dimensional features of a point cloud, our model goes “wider” by splitting features into groups with smaller depth in advance, having the respective MLP computations applied only to a single group, which can significantly reduce complexity and computation. At the same time, we allow communication between groups by shuffling the feature channel to capture fine-grained features. We further discuss the multi-branch method for wider neural networks being also beneficial to feature extraction for point clouds. We present extensive experiments for shape classification tasks on a ModelNet40 dataset and semantic segmentation task on large scale datasets ShapeNet part, S3DIS and KITTI. Finally, we carry out an ablation study and compare our model to other state-of-the-art algorithms to show its efficiency in terms of complexity and accuracy.

A Spark-based Apriori algorithm with reduced shuffle overhead

Mining frequent itemset is considered as a core activity to find association rules from transactional datasets. Among the different well-known approaches to find frequent itemsets, the Apriori algorithm is the earliest proposed. Many attempts have been made to adopt the Apriori algorithm for large-scale datasets. But the bottlenecks associated with Apriori like/such as repeated scans of the input dataset, generation of all the candidate itemsets prior to counting their support value, etc., reduce the effectiveness of Apriori for large-size datasets. When the data size is large, even distributed and parallel implementations of Apriori using the MapReduce framework does not perform well. This is due to the iterative nature of the algorithm that incurs high disk overhead. In each iteration, the input dataset is scanned that resides on disk, causing the high disk I/O. Apache Spark implementations of Apriori show better performance due to in-memory processing capabilities. It makes iterative scanning of datasets faster by keeping it in a memory abstraction called resilient distributed dataset (RDD). An RDD keeps datasets in the form of key-value pairs spread across the cluster nodes. RDD operations require these key-value pairs to be redistributed among cluster nodes in the course of processing. This redistribution or shuffle operation incurs communication and synchronization overhead. In this manuscript, we propose a novel approach, namely the Spark-based Apriori algorithm with reduced shuffle overhead (SARSO). It utilizes the benefits of Spark’s parallel and distributed computing environment, and it is in-memory processing capabilities. It improves the efficiency further by reducing the shuffle overhead caused by RDD operations at each iteration. In other words, it restricts the movement of key-value pairs across the cluster nodes by using a partitioning method and hence reduces the necessary communication and synchronization overhead incurred by the Spark shuffle operation. Extensive experiments have been conducted to measure the performance of the SARSO on benchmark datasets and compared with an existing algorithm. Experimental results show that the SARSO has better performance in terms of running time and scalability.

Stacked lossless deconvolutional network for remote sensing image restoration

Satellite image restoration from its degraded observation has a significant impact on the remote sensing industry. There are many potential applications that can directly benefit from this technique. A convolutional neural network (CNN) has been recently explored in image restoration and achieved remarkable performance. However, most deep CNN architectures in the literature do not properly handle the inherent trade-off between localization accuracy and the use of global context, which is vital for satellite images covering a ultrabroad area. We present a stacked lossless deconvolutional network (SLDN) for remote sensing image restoration. We fully exploit global context information while guaranteeing the recovery of fine details. Specifically, we design a lossless pooling by reformulating the pixel shuffle operator and incorporate it with a shallow deconvolutional network. The resulting lossless deconvolution blocks are stacked one by one to enlarge the receptive fields without any information loss. We further propose an attentive skip connection and progressive learning scheme to improve gradient flows throughout the SLDN. The SLDN can reconstruct high-quality satellite images without noticeable artifacts. An extensive ablation study is also provided to show that all the components proposed are useful for remote sensing image restoration. Experimental comparisons on various restoration tasks, including super-resolution, denoising, and compression artifact reduction, demonstrate the superiority of the proposed method over state-of-the-art methods both qualitatively and quantitatively.

Deep Fusion Local-Content and Global-Semantic for Image Inpainting

The upsampling layers are adopted in almost all the existing encoder-decoder based generative adversarial networks (GANs), which have shown promising results in the image inpainting field. However, existing upsampling layers (e.g. deconvolution and bilinear interpolation) suffer from two limitations: (1) they obtain few semantic information from the global structure. (2) upsampling layer could hardly capture the local content details. To eliminate the above issues, we propose a deep Fusion local-content and global-semantic (DFLG) model that is both effective and general. The DFLG model mainly consists of four components: the Local Content-Response (LCR) module, the pixel-shuffle operator, the Global Semantic-Aware (GSA) module and the reassembly module. Firstly, the LCR module divides the channel into several groups, then utilizes the squeeze-and-excitation mechanism within each group to capture the correlation between channels. Secondly, the pixel shuffle operator reshapes depth on the channel space into width and height on the spatial space, which transforms the correlation within groups on the channel space into correlation within patches on the spatial space. Next, the GSA module employs a patch-based spatial attention mechanism to calculate the correlation between different patches. Finally, the reassembly module refines the feature map. Furthermore, we propose a novel loss function called Attention Loss (ATLoss), which guides the network to concentrate on regions with obvious artifacts. The experiments on CelebA-HQ, Places2, and Paris StreetView datasets demonstrate the effectiveness of our proposed methods in image inpainting tasks and the capability of obtaining images with higher quality.

Design and Evaluation of an RDMA-aware Data Shuffling Operator for Parallel Database Systems

The commoditization of high-performance networking has sparked research interest in the RDMA capability of this hardware. One-sided RDMA primitives, in particular, have generated substantial excitement due to the ability to directly access remote memory from within an application without involving the TCP/IP stack or the remote CPU. This article considers how to leverage RDMA to improve the analytical performance of parallel database systems. To shuffle data efficiently using RDMA, one needs to consider a complex design space that includes (1) the number of open connections, (2) the contention for the shared network interface, (3) the RDMA transport function, and (4) how much memory should be reserved to exchange data between nodes during query processing. We contribute eight designs that capture salient tradeoffs in this design space as well as an adaptive algorithm to dynamically manage RDMA-registered memory. We comprehensively evaluate how transport-layer decisions impact the query performance of a database system for different generations of InfiniBand. We find that a shuffling operator that uses the RDMA Send/Receive transport function over the Unreliable Datagram transport service can transmit data up to 4× faster than an RDMA-capable MPI implementation in a 16-node cluster. The response time of TPC-H queries improves by as much as 2×.

Practical card-based implementations of Yao's millionaire protocol

Yao's millionaire protocol enables Alice and Bob to know whether or not Bob is richer than Alice by using a public-key cryptosystem without revealing the actual amounts of their properties. In this paper, we present simple and practical implementations of Yao's millionaire protocol using a physical deck of playing cards; we straightforwardly implement the idea behind Yao's millionaire protocol so that even non-experts can easily understand their correctness and secrecy. Our implementations are based partially on the previous card-based scheme proposed by Nakai, Tokushige, Misawa, Iwamoto, and Ohta; their scheme admits players' private actions on a sequence of cards called Private Permutation (PP), implying that a malicious player could make an active attack (for example, he/she could exchange some of the cards stealthily when doing such a private action). By contrast, our implementations rely on a familiar shuffling operation called a random cut, and hence, they can be conducted completely publicly so as to avoid any active attack. More specifically, we present two card-based implementations of Yao's millionaire protocol; one uses a two-colored deck of cards (which consists of black and red cards), and the other uses a standard deck of playing cards. Furthermore, we also provide card-based protocols that rely on a logical circuit representing the comparison.

Various dimensional colour image encryption based on non-overlapping block-level diffusion operation

This paper proposes a colour image encryption scheme to encrypt colour images of arbitrary sizes. In this scheme, a fixed block size (3 × 8) based block-level diffusion operation is performed to encrypt arbitrary sized images. The proposed technique overcomes the limitation of performing block-level diffusion operations in arbitrary sized images. This method first performs bit-plane decomposition and concatenation operation on the three components (blue, green, and red) of the colour image. Second it performs row and column shuffling operation using the Logistic-Sine System. Then the proposed scheme executes block division and fixed block-level diffusion (exclusive-OR) operation using the key image generated by the Piece-wise Linear Chaotic Map. At last, the cipher image is generated by combining the diffused blocks. In addition, the SHA-256 hashing on plain image is used to make chaotic sequences unique in each encryption process and to protect the ciphertext against the known-plaintext attack and the chosen-plaintext attack. Simulation results and various parameter analysis demonstrate the algorithm’s best performance in image encryption and various common attacks.

Incorporating super-operators in big-data query optimizers

The cost of big-data analytics is dominated by shuffle operations that induce multiple disk reads, writes and network transfers. This paper proposes a new class of optimization rules that are specifically aimed at eliminating shuffles where possible. The rules substitute multiple shuffle inducing operators ( Join, UnionAll, Spool, GroupBy ) with a single streaming operator which implements an entire sub-query. We call such operators super-operators. A key challenge with adding new rules that substitute sub-queries with super-operators is that there are many variants of the same sub-query that can be implemented via minor modifications to the same super-operator. Adding each as a separate rule leads to a search space explosion. We propose several extensions to the query optimizer to address this challenge. We propose a new abstract representation for operator trees that captures all possible sub-queries that a super-operator implements. We propose a new rule matching algorithm that can efficiently search for abstract operator trees. Finally we extend the physical operator interface to introduce new parametric super-operators. We implement our changes in SCOPE, a state-of-the-art production big-data optimizer used extensively at Microsoft. We demonstrate that the proposed optimizations provide significant reduction in both resource cost (average 1.7x) and latency (average 1.5x) on several production queries, and do so without increasing optimization time.

SP-BRAIN: scalable and reliable implementations of a supervised relevance-based machine learning algorithm

In this work, new implementations of the U-BRAIN (Uncertainty-managing Bach Relevance-Based Artificial Intelligence) supervised machine learning algorithm are described. The implementations, referred as SP-BRAIN (SP stands for Spark), aim to efficiently process large datasets. Given the iterative nature of the algorithm together with its dependence on in-memory data, a non-standard MapReduce paradigm is applied, taking into account several memory and performance problems, e.g., the granularity of the MAP task, the reduction in the shuffling operation, caching, partial data recomputing, and usage of clusters. The implementations benefit the whole Hadoop ecosystem components, such as HDFS, Yarn, and streaming. Testing is performed in cloud execution environments, using different configurations with up to 128 cores. The performance of the new implementations is evaluated on three known datasets, and the findings are compared to the ones of a previous U-BRAIN parallel implementation. The results show a speedup up to 20 × with a good scalability and reliability in cluster environments.

Secure implementations of a random bisection cut

By using a deck of cards, it is possible to realize a secure multiparty computation. In particular, since a new shuffling operation, called a random bisection cut, was devised in 2009, many efficient card-based protocols have been designed. The random bisection cut functions in the following manner. A sequence of cards is bisected, and the two halves are shuffled. This results in two possible cases depending on whether the two halves of the card sequence are swapped. As only two possibilities exist when a random bisection cut is performed, it has been suggested that information regarding the outcome of the shuffle could sometimes be leaked visually. Thus, in this paper we propose some methods for securely implementing a random bisection cut without leaking such information.