Amounts Of Data In Parallel Research Articles

Optimal Device Selection in Federated Learning for Resource-Constrained Edge Networks

Low latency, resource efficiency, and data privacy are some of the crucial requirements in modern communication networks. Federated learning can efficiently address these issues by utilizing the data at the network edge and processing massive amounts of data in parallel at the edge devices, thus ensuring data privacy and low latency. For a large-scale federated learning task spanning many devices, challenges arise due to device heterogeneity, data variability, and limited network resources. Optimal selection of edge devices participating in federated learning is essential to attaining resilient, reliable, and resource-efficient edge networks. In this context, this paper proposes an optimal device selection method to minimize redundant data training and improve network resource utilization without affecting the performance of federated learning over resource-constrained edge networks. The proposed optimal device selection method aims to minimize network resource demands while maximizing data diversity within the aggregated model. The performance of the proposed federated learning framework is evaluated using a publicly available image dataset of handwritten digits, EMNIST (an extended version of the MNIST dataset). Experimental results indicate that the proposed framework can obtain accuracy convergence performance on par with conventional federated learning methods while significantly reducing device usage (up to 50%) and resource utilization (up to 30%) while reaching 99% of achievable accuracy. The proposed method can therefore be effectively applied to resource-constrained edge networks.

IMapC: Inner MAPping Combiner to Enhance the Performance of MapReduce in Hadoop

Hadoop is a framework for storing and processing huge amounts of data. With HDFS, large data sets can be managed on commodity hardware. MapReduce is a programming model for processing vast amounts of data in parallel. Mapping and reducing can be performed by using the MapReduce programming framework. A very large amount of data is transferred from Mapper to Reducer without any filtering or recursion, resulting in overdrawn bandwidth. In this paper, we introduce an algorithm called Inner MAPping Combiner (IMapC) for the map phase. This algorithm in the Mapper combines the values of recurring keys. In order to test the efficiency of the algorithm, different approaches were tested. According to the test, MapReduce programs that are implemented with the Default Combiner (DC) of IMapC will be 70% more efficient than those that are implemented without one. To make computations significantly faster, this work can be combined with MapReduce.

Multi-dimensional data analysis technology of business application system based on Spark framework

The current multidimensional data analysis technology uses the data approximation to obtain deep data information in the data cube, which requires repeated traversal of the full link data, resulting in low processing efficiency and unable to analyze a large amount of data in parallel. Therefore, a multi-dimensional data analysis technology based on Spark framework for business application system is proposed to optimize the defects of traditional technology. Spark technology is used to build a full-link data analysis framework, and the data warehouse logic is designed to build a full-link data warehouse. Genetic algorithm is used to mine the full-link data of the business system, and the data are stored in the data warehouse, and the data warehouse and the sequential structure neural network are established. In the sequential structure neural network, multi-dimensional data analysis is realized from multiple angles and aspects. The technical feasibility verification results show that the research technology shortens about 14.7% of the time on average, significantly improves the analysis efficiency and can conduct concurrent analysis on a large number of data, which is better than the application effect of traditional analysis technology.

A Diverse Data Augmentation Strategy for Low-Resource Neural Machine Translation

One important issue that affects the performance of neural machine translation is the scale of available parallel data. For low-resource languages, the amount of parallel data is not sufficient, which results in poor translation quality. In this paper, we propose a diversity data augmentation method that does not use extra monolingual data. We expand the training data by generating diversity pseudo parallel data on the source and target sides. To generate diversity data, the restricted sampling strategy is employed at the decoding steps. Finally, we filter and merge origin data and synthetic parallel corpus to train the final model. In the experiment, the proposed approach achieved 1.96 BLEU points in the IWSLT2014 German–English translation tasks, which was used to simulate a low-resource language. Our approach also consistently and substantially obtained 1.0 to 2.0 BLEU improvement in three other low-resource translation tasks, including English–Turkish, Nepali–English, and Sinhala–English translation tasks.

Smart healthcare framework for ambient assisted living using IoMT and big data analytics techniques

Abstract In the era of pervasive computing, human living has become smarter by the latest advancements in IoMT (Internet of Medical Things), wearable sensors and telecommunication technologies in order to deliver smart healthcare services. IoMT has the potential to revolutionize the healthcare industry. IoMT interconnects wearable sensors, patients, healthcare providers and caregivers via software and ICT (Information and Communication Technology). AAL (Ambient Assisted Living) enables integration of new technologies to be part of our daily life activities. In this paper, we have provided a novel smart healthcare framework for AAL to monitor the physical activities of elderly people using IoMT and intelligent machine learning algorithms for faster analysis, decision making and better treatment recommendations. Data is collected from multiple wearable sensors placed on subject’s left ankle, right arm, and chest, is transmitted through IoMT devices to the integrated cloud and data analytics layer. To process huge amounts of data in parallel, Hadoop MapReduce techniques are used. Multinomial Naive Bayes classifier, which fits into the MapReduce paradigm, is utilized to recognize the motion experienced by different body parts and provides higher scalability and better performance with parallel processing when compared to serial processor. Our proposed framework predicts 12 physical activities with an overall accuracy of 97.1%. This can be considered as an optimal solution for recognizing physical activities to remotely monitor health conditions of elderly people.

Ternary content-addressable memory with MoS2 transistors for massively parallel data search

Ternary content-addressable memory (TCAM) is specialized hardware that can perform in-memory search and pattern matching for data-intensive applications. However, achieving TCAMs with high search capacity, good area efficiency and good energy efficiency remains a challenge. Here, we show that two-transistor–two-resistor (2T2R) transition metal dichalcogenide TCAM (TMD-TCAM) cells can be created by integrating single-layer MoS2 transistors with metal-oxide resistive random-access memories (RRAMs). The MoS2 transistors have very low leakage currents and can program the RRAMs with exceptionally robust current control, enabling the parallel search of very large numbers of data bits. These TCAM cells also exhibit remarkably large resistance ratios (R-ratios) of up to 8.5 × 105 between match and mismatch states. This R-ratio is comparable to that of commercial TCAMs using static random-access memories (SRAMs), with the key advantage that our 2T2R TCAMs use far fewer transistors and have zero standby power due to the non-volatility of RRAMs. By integrating two-dimensional MoS2 transistors with metal-oxide resistive random-access memories, two-transistor–two-resistor ternary content-addressable memory cells can be created, which could be used to search large amounts of data in parallel.

Many-to-many and Completely Parallel-data-free Voice Conversion Based on Eigenspace DNN

Media conversion of image, text, speech, etc., generally requires a large amount of parallel data for training a conversion model. Recently, methods for training the model using no or a small amount of parallel data draw researchers’ attention. In many-to-many voice conversion, since it is often hard to collect parallel data from every pair of speakers, the conversion models requiring no parallel data are desired. Conventional many-to-many voice conversion models required a large amount of prestored parallel data to acquire prior knowledge of the entire speaker space. Then, a specific model from an arbitrary speaker to another can be realized by adapting a few model parameters. Although these conversion models certainly do not use parallel data in an adaptation step, they still use parallel data for prior training. In this study, we aim at realizing completely parallel-data-free and many-to-many voice conversion. The proposed method uses both Eigenvoice Gaussian mixture models EVGMM and Deep neural network DNN. EVGMM is a many-to-many conversion model that constructs the entire speaker space called eigenspace by analyzing mean vectors of Gaussian mixture models and it is used in our method to decompose training speakers’ features into their eigenspace components. By using the speaker features and the obtained components as pseudo parallel data, multiple DNNs are trained to realize conversion between them. With these DNNs, features of any target speaker can be represented by a weighted sum of the components. It should be noted that all the processes of our proposal do not require any parallel data. A key technique is to estimate covariance terms of EVGMM with no parallel data. Experiments indicate that individuality scores of the proposed method using no parallel data are comparable enough to those of a baseline system trained with parallel data.

Performance comparison between Hadoop and Spark frameworks using HiBench benchmarks

SummaryBig Data has become one of the major areas of research for cloud service providers due to a large amount of data produced every day and the inefficiency of traditional algorithms and technologies to handle these large amounts of data. Big Data with its characteristics such as volume, variety, and veracity (3V) requires efficient technologies to process in real time. To solve this problem and to process and analyze this vast amount of data, there are many powerful tools like Hadoop and Spark, which are mainly used in the context of Big Data. They work following the principles of parallel computing. The challenge is to specify which Big Data's tool is better depending on the processing context. In this paper, we present and discuss a performance comparison between two popular Big Data frameworks deployed on virtual machines. Hadoop MapReduce and Apache Spark are used to efficiently process a vast amount of data in parallel and distributed mode on large clusters, and both of them suit for Big Data processing. We also present the execution results of Apache Hadoop in Amazon EC2, a major cloud computing environment. To compare the performance of these two frameworks, we use HiBench benchmark suite, which is an experimental approach for measuring the effectiveness of any computer system. The comparison is made based on three criteria: execution time, throughput, and speedup. We test Wordcount workload with different data sizes for more accurate results. Our experimental results show that the performance of these frameworks varies significantly based on the use case implementation. Furthermore, from our results we draw the conclusion that Spark is more efficient than Hadoop to deal with a large amount of data in major cases. However, Spark requires higher memory allocation, since it loads the data to be processed into memory and keeps them in caches for a while, just like standard databases. So the choice depends on performance level and memory constraints.

Neural machine translation for low-resource languages without parallel corpora

The problem of a total absence of parallel data is present for a large number of language pairs and can severely detriment the quality of machine translation. We describe a language-independent method to enable machine translation between a low-resource language (LRL) and a third language, e.g. English. We deal with cases of LRLs for which there is no readily available parallel data between the low-resource language and any other language, but there is ample training data between a closely-related high-resource language (HRL) and the third language. We take advantage of the similarities between the HRL and the LRL in order to transform the HRL data into data similar to the LRL using transliteration. The transliteration models are trained on transliteration pairs extracted from Wikipedia article titles. Then, we automatically back-translate monolingual LRL data with the models trained on the transliterated HRL data and use the resulting parallel corpus to train our final models. Our method achieves significant improvements in translation quality, close to the results that can be achieved by a general purpose neural machine translation system trained on a significant amount of parallel data. Moreover, the method does not rely on the existence of any parallel data for training, but attempts to bootstrap already existing resources in a related language.

Using machine learning to optimize parallelism in big data applications

In-memory cluster computing platforms have gained momentum in the last years, due to their ability to analyse big amounts of data in parallel. These platforms are complex and difficult-to-manage environments. In addition, there is a lack of tools to better understand and optimize such platforms that consequently form the backbone of big data infrastructure and technologies. This directly leads to underutilization of available resources and application failures in such environment. One of the key aspects that can address this problem is optimization of the task parallelism of application in such environments. In this paper, we propose a machine learning based method that recommends optimal parameters for task parallelization in big data workloads. By monitoring and gathering metrics at system and application level, we are able to find statistical correlations that allow us to characterize and predict the effect of different parallelism settings on performance. These predictions are used to recommend an optimal configuration to users before launching their workloads in the cluster, avoiding possible failures, performance degradation and wastage of resources. We evaluate our method with a benchmark of 15 Spark applications on the Grid5000 testbed. We observe up to a 51% gain on performance when using the recommended parallelism settings. The model is also interpretable and can give insights to the user into how different metrics and parameters affect the performance.

Relational Query Optimization Technique using Space Efficient File Formats of Hadoop for the Big Data Warehouse System

Objectives: File structure and storage becomes a challenging issue while processing huge amount of data in parallel and distributed environment. To increase processing capabilities an appropriate file structure must be implemented. Methods/Statistical Analysis: In our approach we have imported the data from the available relational databases like Oracle or MySql to Hive using Sqoop and analyzed the query processing based upon different file storage formats. We have focused on the Parquet, Sequence, RC file and ORC file format for query analysis in MapReduce framework on top of Hadoop. Findings: Understanding dynamic behavior of user buying habits in different web services and product recommendation using social media, e-marketing etc. the MapReduce based data warehousing system plays vital role to perform the Big Data analytic in a parallel and distributed environment. In such type of analysis the data structure used to store the data for parallel query processing effect the performance of Big Data warehouse system. During the analysis of huge amount of relational data in a parallel and distributed system few issues should be taken care to increase the query performance and optimization. These are 1. Faster loading of huge amount of relational data into the Big Data warehouse. 2. Optimized file format to efficiently manage the storage system. 3. Faster query processing by increasing the throughput. Our findings explained appropriate file formats to store the huge amount of relational data in the Big Data warehouse system based upon HDFS and MapReduce framework known as Hive and evaluated the performance of query processing in multi node Hadoop cluster. Application/Improvements: The cost of parallel query processing has been reduced as well as distributed storage efficiency increased by choosing appropriate file structure in Big Data warehouse systems.

Design and development of a medical big data processing system based on Hadoop.

Secondary use of medical big data is increasingly popular in healthcare services and clinical research. Understanding the logic behind medical big data demonstrates tendencies in hospital information technology and shows great significance for hospital information systems that are designing and expanding services. Big data has four characteristics--Volume, Variety, Velocity and Value (the 4 Vs)--that make traditional systems incapable of processing these data using standalones. Apache Hadoop MapReduce is a promising software framework for developing applications that process vast amounts of data in parallel with large clusters of commodity hardware in a reliable, fault-tolerant manner. With the Hadoop framework and MapReduce application program interface (API), we can more easily develop our own MapReduce applications to run on a Hadoop framework that can scale up from a single node to thousands of machines. This paper investigates a practical case of a Hadoop-based medical big data processing system. We developed this system to intelligently process medical big data and uncover some features of hospital information system user behaviors. This paper studies user behaviors regarding various data produced by different hospital information systems for daily work. In this paper, we also built a five-node Hadoop cluster to execute distributed MapReduce algorithms. Our distributed algorithms show promise in facilitating efficient data processing with medical big data in healthcare services and clinical research compared with single nodes. Additionally, with medical big data analytics, we can design our hospital information systems to be much more intelligent and easier to use by making personalized recommendations.

Single-scan: a fast star-join query processing algorithm

A data warehouse can store very large amounts of data that should be processed in parallel in order to achieve reasonable query execution times. The MapReduce programming model is a very convenient way to process large amounts of data in parallel on commodity hardware clusters. A very popular query used in data warehouses is star-join. In this paper, we present a fast and efficient star-join query execution algorithm built on top of a MapReduce framework called Hadoop. By using dynamic filters against dimension tables, the algorithm needs a single scan of the fact table, which means a significant reduction of input/output operations and computational complexity. Also, the algorithm requires only two MapReduce iterations in total-one to build the filters against dimension tables and one to scan the fact table. Our experiments show that the proposed algorithm performs much better than the existing solutions in terms of execution time and input/output. Copyright © 2014 John Wiley & Sons, Ltd.

Handling Big Data in Medical Imaging: Iterative Reconstruction with Large-Scale Automated Parallel Computation.

The primary goal of this project is to implement the iterative statistical image reconstruction algorithm, in this case maximum likelihood expectation maximum (MLEM) used for dynamic cardiac single photon emission computed tomography, on Spark/GraphX. This involves porting the algorithm to run on large-scale parallel computing systems. Spark is an easy-to- program software platform that can handle large amounts of data in parallel. GraphX is a graph analytic system running on top of Spark to handle graph and sparse linear algebra operations in parallel. The main advantage of implementing MLEM algorithm in Spark/GraphX is that it allows users to parallelize such computation without any expertise in parallel computing or prior knowledge in computer science. In this paper we demonstrate a successful implementation of MLEM in Spark/GraphX and present the performance gains with the goal to eventually make it useable in clinical setting.

A HIGH SPEED VLSI ARCHITECTURE FOR DIGITAL SPEECH WATERMARKING WITH COMPRESSION

The need to provide a copy right protection on digital watermarking to multimedia data like speech, image or video is rapidly increasing with an intensification in the application in these areas. Digital watermarking has received a lot of attention in the past few years. A hardware system based solely on DSP processors are fast but may require more area, cost or power if the target application requires a large amount of parallel processing. An FPGA co-processor can provide as many as 550 parallel multiply and accumulate operations on a single device, but FPGAs excel at processing large amounts of data in parallel, as they are not optimized as processors for tasks such as periodic coefficient updates, decision- making control tasks. Combination of both the FPGA and DSP processor delivers an attractive solution for a wide range of applications. A hardware implementation of digital speech watermarking combined with speech compression, encryption on heterogeneous platform is made in this paper. It is observed that the proposed architecture is able to attain high speed while utilizing optimal resources in terms of area.

SmartJoin: a network-aware multiway join for MapReduce

MapReduce is an effective tool for processing large amounts of data in parallel using a cluster of processors or computers. One common data processing task is the join operation, which combines two or more datasets based on values common to each. In this paper, we present a network aware multi-way join for MapReduce (SmartJoin) that improves performance and considers network traffic when redistributing workload amongst reducers. SmartJoin achieves this by dynamically redistributing tuples directly between reducers with an intelligent network aware algorithm. We show that our presented technique has significant potential to minimize the time required to join multiple datasets. In our evaluation, we show that SmartJoin has up to 39 % improvement compared to the non-redistribution method, a 26.8 % improvement over random redistribution and 27.6 % improvement over worst join redistribution.

Simulation and Data-Processing Framework for Hybrid Synthetic Aperture THz Systems Including THz-Scattering

We present a simulation and data-processing framework that incorporates full 3-D configuration, simulation, and reconstruction functionalities for hybrid synthetic aperture THz systems, which combine mechanically scanned real imaging projection (1-D) and synthetic imaging reconstruction (2-D). The simulation is based on a ray-casting approach including models for reflections at rough surfaces and angular antenna sensitivity. A near real-time GPU reconstruction is introduced to process large amounts of data in parallel. Techniques for fast fusion of reconstructed data from several acquisition directions are presented. They serve as basis to visualize a fully consistent 3-D object. The modular framework allows a fast, easy, and efficient evaluation and prediction of system properties including scattering behavior for these types of scanner setups.

Traffic classification combining flow correlation and ensemble classifier

Traffic classification has wide applications in network measurements, network security and quality of service. Recent research tends to apply machine learning methods based on flow statistical features to improve traffic classification performance. In this paper, we propose a novel non-parametric approach for traffic classification, which can improve the classification performance effectively by incorporating correlated information into the classification process. Meanwhile, our system employs a lightweight modular architecture, which combines a series of simple linear binary classifiers, each of which can be efficiently implemented and trained on vast amounts of flow data in parallel, to achieve scalability while attaining high accuracy. A large number of experiments are carried out on real traffic data to validate the proposed approach. The results show that the traffic classification performance can be improved significantly while meeting the scalability and stability requirements of large networks.

Large-Scale Data Processing Using MapReduce in Cloud Computing Environment

The computer industry is being challenged to developmethods and techniques for affordable data processing on large datasets at optimum response times. The technical challenges in dealing with the increasing demand to handle vast quantities of data is daunting and on the rise. One of the recent processing models with a more efficient and intuitive solution to rapidly process large amount of data in parallel is called MapReduce. It is a framework defining a template approach of programming to perform large-scale data computation on clusters of machines ina cloud computing environment. MapReduce provides automatic parallelization and distribution of computation based on several processors. It hides the complexity of writing parallel and distributed programming code. This paper provides a comprehensive systematic review and analysis of large-scale dataset processing and dataset handling challenges and requirements in a cloud computing environment by using the M apReduce framework and its open -source implementation Hadoop. We defined requirements for MapReduce syst ems to perform large-scale data processing. We also proposed the MapReduce framework and one implementation of this framework on Amazon Web Services. At the end of the paper, we presented an experimentation of running MapReduce system in a cloud environment. This paper outlines one of the best techniques to process large datasets is MapReduce;it also can help developers to do parallel and distributed computation in a cloud environment.

Survey on Task Assignment Techniques in Hadoop

is an implementation for processing large scale data parallelly. Actual benefits of MapReduce occur when this framework is implemented in large scale, shared nothing cluster. MapReduce framework abstracts the complexity of running distributed data processing across multiple nodes in cluster. Hadoop is open source implementation of MapReduce framework, which processes the vast amount of data in parallel on large clusters. In Hadoop pluggable scheduler was implemented, because of this several algorithms have been developed till now. This paper presents the different schedulers used for Hadoop.