Data Processing Jobs Research Articles

Accelerating Industrial IoT Acoustic Data Separation With In-Network Computing

Acoustic data from the Industrial Internet of Things (IIoT) are widely used in anomaly detection because audio information reflects richer internal statuses of monitored working machines than the video does. Since multiple acoustic data sources interfere with each other by nature, source data estimation is a prerequisite of subsequent anomaly detection. Existing schemes often use a centralized manner to separate full data on a remote node in clouds. However, such a centralized manner may delay reactions to anomalies due to data transmission delay and the complexity of solving data separation problems. This article shows that the data separation phase can be substantially accelerated with an in-network computing approach. The key idea is to offload data processing jobs to intermediate network nodes along the forwarding path. We first propose a distributed algorithm so that the data separation jobs can be done in a progressive manner; likewise, we modify the forwarding layer in order to eliminate hop-by-hop data transmission delay that hurts the performance of using in-network computing. We further derive theoretical upper and lower bounds of the required number of intermediate nodes that achieve the maximum acceleration. We also implement our proposed solution in a full-stack network emulator. Based on an open and professional data set, evaluation results justify the feasibility and advantages of our idea with nearly 32.18% acceleration on total processing time. This work exemplifies the convergence of IIoT, edge, and clouds.

Outsourcing Data Processing Jobs With Lithops

Unexpectedly, the rise of serverless computing has also collaterally started the “democratization” of massive-scale data parallelism. This new trend heralded by PyWren pursues to enable untrained users to execute single-machine code in the cloud at massive scale through platforms like AWS Lambda. Driven by this vision, this article presents <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Lithops</i> , which carries forward the pioneering work of PyWren to better exploit the innate parallelism of à la MapReduce tasks atop several Functions-as-a-Service platforms such as AWS Lambda, IBM Cloud Functions, Google Cloud Functions or Knative. Instead of waiting for a cluster to be up and running in the cloud, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Lithops</i> makes easy the task of spawning hundreds and thousands of cloud functions to execute a large job in a few seconds from start. With Lithops, for instance, users can painlessly perform exploratory data analysis from within a Jupyter notebook, while it is the Lithops’s engine which takes care of launching the parallel cloud functions, loading dependencies, automatically partitioning the data, etc. In this article, we describe the design and innovative features of Lithops and evaluate it using several representative applications, including sentiment analysis, Monte Carlo simulations, and hyperparameter tuning. These applications manifest the Lithops’ ability to scale single-machine code computations to thousands of cores. And very importantly, without the need of booting a cold cluster or keeping a warm cluster for occasional tasks.

Face Recognition Model Optimization Research Based on Embedded Platform

The development of information technology has promoted the expansion of the application field of facial recognition technology. Its mainstream recognition methods rely on deep learning algorithms for calculation, but the problem of large data computation brought by its system makes it difficult to apply in embedded platform devices. As a result, this study focuses on improving recognition systems built on lightweight backend networks and builds an embedded platform system environment using multi-scale feature fusion, anchor box size optimization, the addition of channel attention mechanism weighted features, affine face alignment, and file compilation. The experimental results showed that when the number of iterations was 300, the loss value (0.46) of the improved embedded algorithm was much smaller than that of other comparison algorithms (1.42, 1.73, 2.01), and its ACC value (0.924) was significantly better than other comparison algorithms (0.915, 0.909, 0.894). The minimum system testing time consumption was 7 ms. This deep learning embedded facial recognition algorithm has high recognition accuracy and fast running speed, and is less limited by environmental conditions and data types.It is ideally suited for use in embedded hardware devices, broadening the scope of equipment matching and facial recognition algorithms’ applications. As a result, it is better suited to satisfy the demands of embedded devices and massive data processing jobs.

Collaborative Cluster Configuration for Distributed Data-Parallel Processing: A Research Overview

Many organizations routinely analyze large datasets using systems for distributed data-parallel processing and clusters of commodity resources. Yet, users need to configure adequate resources for their data processing jobs. This requires significant insights into expected job runtimes and scaling behavior, resource characteristics, input data distributions, and other factors. Unable to estimate performance accurately, users frequently overprovision resources for their jobs, leading to low resource utilization and high costs.In this paper, we present major building blocks towards a collaborative approach for optimization of data processing cluster configurations based on runtime data and performance models. We believe that runtime data can be shared and used for performance models across different execution contexts, significantly reducing the reliance on the recurrence of individual processing jobs or, else, dedicated job profiling. For this, we describe how the similarity of processing jobs and cluster infrastructures can be employed to combine suitable data points from local and global job executions into accurate performance models. Furthermore, we outline approaches to performance prediction via more context-aware and reusable models. Finally, we lay out how metrics from previous executions can be combined with runtime monitoring to effectively re-configure models and clusters dynamically.

Development of an Artificially Intelligent Advising System for Saudi Medical Transcription

In this world of IT related services and computer science, various new technologies are emerging. This technology is focused mainly on the ease of human being and their survival amid Covid 19. Medical Transcription is one of the highly professional IT related services. It involves highly skilled data processing job for medical field. The common system which prevails around the world nowadays is to visit the hospital and get the cure. However, once a patient which is a hospital and gets the medicine which is diagnosed by a doctor. He comes back to his house and take the medicine to get well. But in this complete process there is no guarantee of responding to a medicine in such a manner such that the problem, which had arisen will not happen in the future. Thus, it becomes very obvious that we don’t have a solution in which the healthcare benefits can be obtained from using IT and services. As per the old system. The records of the patient’s health are maintained in a file normally for a good hospital. The system maintains the medical records for each patient, however medical record helps in avoiding chances of omission of some of the important findings in the records. But in this scenario of globalization across the world, mobility of human being had been a very important part of its daily lifestyle. Thus, we can very well say that the system which prevails around the world nowadays is not very easy. Since the files that are written in the hospital for the medical health care services is not approachable at a very convenient rate. It is also not at all very sure that the disease that had been cured by the doctor shall never revert back again. When the disease comes back again, the patient needs a date to visit the doctor. This type of free visit to the doctor is required at every instance whenever a man feel some sick-ness. The sickness that has been encountered by the human being is not at all mortal. It can return back whenever it is required and there are certain symptoms in which the disease can be observed. This is a loss of time and effort. It is a very strong understanding method and the use of artificial intelligence can be done in order to observe the routine of the person and its healthcare. A system could be designed in which depending on the past histories of the medical ailments and the treatment of a person can be utilized in order to predict the nature of its human anatomy. A system which is artificially so intelligent that it can be able to give prediction for what might happen in the future for a particular man. Thus, this artificial intelligence system will be helpful enough to give warning to a human being at various times whenever he is happy, so that he will be saved from the problems that might arise in the future.

Serverless Geospatial Data Processing Workflow System Design

Geospatial data and related technologies have become an increasingly important aspect of data analysis processes, with their prominent role in most of them. Serverless paradigm have become the most popular and frequently used technology within cloud computing. This paper reviews the serverless paradigm and examines how it could be leveraged for geospatial data processes by using open standards in the geospatial community. We propose a system design and architecture to handle complex geospatial data processing jobs with minimum human intervention and resource consumption using serverless technologies. In order to define and execute workflows in the system, we also propose new models for both workflow and task definitions models. Moreover, the proposed system has new Open Geospatial Consortium (OGC) Application Programming Interface (API) Processes specification-based web services to provide interoperability with other geospatial applications with the anticipation that it will be more commonly used in the future. We implemented the proposed system on one of the public cloud providers as a proof of concept and evaluated it with sample geospatial workflows and cloud architecture best practices.

Resource Recommender for Cloud-Edge Engineering

The interaction between artificial intelligence (AI), edge, and cloud is a fast-evolving realm in which pushing computation close to the data sources is increasingly adopted. Captured data may be processed locally (i.e., on the edge) or remotely in the clouds where abundant resources are available. While many emerging applications are processed in situ due primarily to their data intensiveness and short-latency requirement, the capacity of edge resources remains limited. As a result, the collaborative use of edge and cloud resources is of great practical importance. Such collaborative use should take into account data privacy, high latency and high bandwidth consumption, and the cost of cloud usage. In this paper, we address the problem of resource allocation for data processing jobs in the edge-cloud environment to optimize cost efficiency. To this end, we develop Cost Efficient Cloud Bursting Scheduler and Recommender (CECBS-R) as an AI-assisted resource allocation framework. In particular, CECBS-R incorporates machine learning techniques such as multi-layer perceptron (MLP) and long short-term memory (LSTM) neural networks. In addition to preserving privacy due to employing edge resources, the edge utility cost plus public cloud billing cycles are adopted for scheduling, and jobs are profiled in the cloud-edge environment to facilitate scheduling through resource recommendations. These recommendations are outputted by the MLP neural network and LSTM for runtime estimation and resource recommendation, respectively. CECBS-R is trained with the scheduling outputs of Facebook and grid workload traces. The experimental results based on unseen workloads show that CECBS-R recommendations achieve a ∼65% cost saving in comparison to an online cost-efficient scheduler (BOS), resource management service (RMS), and an adaptive scheduling algorithm with QoS satisfaction (AsQ).

An Enhanced Mathematical Model For Cloud Based Data Oriented Job Analysis

Data processing or data analytics is the common functionality that is attached to most of the real world applications. The amount of processing required for data oriented tasks or jobs are quiet high. To resolve the processing issue the most common approach deployed is by using a high performance cluster. Setting a cluster over real time infrastructure leads to a very expensive solution. A Cloud based infrastructure remains as an ideal support for setting up a cluster. Managing the cluster over a Cloud is a challenging task as allocation of infrastructure based on the task schedule is a critical parameter. The proposed mathematical model introduces a strategy to allocate the infrastructure and manage the load of the cluster based on Queuing model. The experimental setup is made on top of private cloud and Hadoop based data processing jobs are tested. The proposed data oriented resource optimizer enhances the performance of the cluster by balancing the increased load due to data processing jobs. The result shows the enhanced improvement in performance compared to default resource manager.

Mitigating Curtailment and Carbon Emissions through Load Migration between Data Centers

Summary As the share of variable renewable energy (VRE) grows in the electric grid, so does the risk of curtailment. While energy storage and hydrogen production have been proposed as solutions to the curtailment problem, they often pose technological and economic challenges. Here, we analyze the potential of data center load migration for mitigating curtailment and greenhouse gas (GHG) emissions. Using historical hourly electricity generation, curtailment, and typical data center server utilization data, we simulate the effect of migrating data center workloads from the fossil-fuel-heavy PJM to the renewable-heavy CAISO. The results show that load migration within the existing data center capacity during the curtailment hours in CAISO has the potential to reduce 113–239 KtCO2e per year of GHG emissions and absorb up to 62% of the total curtailment with negative abatement costs in 2019. Our study demonstrates the overlooked role that data centers can play for VRE integration and GHG emissions mitigation.

Improved Hadoop Cluster Performance by Dynamic Load and Resource Aware Speculative Execution and Straggler Node Detection

The big data is one of the fastest growing technologies, which can to handle huge amounts of data from various sources, such as social media, web logs, banking and business sectors etc. In order to pace with the changes in the data patterns and to accommodate the requirements of big data analytics, the platform for storage and processing such as Hadoop, also requires great advancements. Hadoop, an open source project executes the big data processing job in map and reduce phases and follows master-slave architecture. A Hadoop MapReduce job can be delayed if one of its many tasks is being assigned to an unreliable or congested machine. To solve this straggler problem, a novel algorithm design of speculative execution schemes for parallel processing clusters, from an optimization perspective, under different loading conditions is proposed. For the lightly loaded case, a task cloning scheme, namely, the combined file task cloning algorithm, which is based on maximizing the overall system utility, a straggler detection algorithm is proposed based on a workload threshold. The detection and cloning of tasks assigned with the stragglers only will not be enough to enhance the performance unless cloning of tasks is allocated in a resource aware method. So, a method is proposed which identifies and optimizes the resource allocation by considering all possible aspects of cluster performance balancing. One main issue arises due to the pre configuration of distinct map and reduce slots based on the number of files in the input folder. This can cause severe under-utilization of slot as map slots might not be fully utilized with respect to the input splits. To solve this issue, an alternative technique of Hadoop Slot Allocation is introduced in this paper by keeping the efficient management of slots model. The combine file task cloning algorithm combines the files which are less than the size of a single data block and executes them in the highly performing data node. On implementing these efficient cloning and combining techniques on a heavily loaded cluster after detecting the straggler, machine is found to reduce the elapsed time of execution to an average of 40%. The detection algorithm improves the overall performance of the heavily loaded cluster by 20% of the total elapsed time in comparison with the native Hadoop algorithm.

Bigflow: A General Optimization Layer for Distributed Computing Frameworks

As data volumes grow rapidly, distributed computations are widely employed in data-centers to provide cheap and efficient methods to process large-scale parallel datasets. Various computation models have been proposed to improve the abstraction of distributed datasets and hide the details of parallelism. However, most of them follow the single-layer partitioning method, which limits developers to express a multi-level partitioning operation succinctly. To overcome the problem, we present the NDD (Nested Distributed Dataset) data model. It is a more compact and expressive extension of Spark RDD (Resilient Distributed Dataset), in order to remove the burden on developers to manually write the logic for multi-level partitioning cases. Base on the NDD model, we develop an open-source framework called Bigflow, which serves as an optimization layer over computation engines from most widely used processing frameworks. With the help of Bigflow, some advanced optimization techniques, which may only be applied by experienced programmers manually, are enabled automatically in a distributed data processing job. Currently, Bigflow is processing about 3 PB data volumes daily in the data-centers of Baidu. According to customer experience, it can significantly save code length and improve performance over the intuitive programming style.

Production Operations Management System (POMS) for Fermilab Experiments

The Production Operations Management System (POMS) software allows production teams and analysis groups across multiple Fermilab experiments to launch, modify and monitor large scale campaigns of related Monte Carlo or data processing jobs, and currently manages the majority of production computing of experiments at Fermilab. POMS provides a web service interface that enables automated job submission on distributed resources according to customers’ requests, as well as subsequent monitoring of those submissions as well as recovery of failed submissions, debugging and record keeping. POMS interfaces with existing HEP data access, processing, movement and monitoring tools at Fermilab, including Jobsub (for job submission), SAM (for data management), and Landscape (for monitoring), and in combination with them handles the creation of intermediate datasets of files for multistage campaigns. POMS visualizes these campaigns with an interactive GUI interface. An important feature of POMS is a one-to-one connection between the GUI campaign visualizer and a text based representation of the complete campaign configuration. An extensible library of template campaign configurations is available. The templates are user modifiable and map cleanly to and from the GUI description of the campaign. This paper will discuss the current usage, history, and some technical details of POMS, as well as planned future extensions, particularly interfacing with Rucio as well as SAM for file movement.

Profit Maximization of Big Data Jobs in Cloud Using Stochastic Optimization

Reserved instances offered by cloud providers make it possible to reserve resources and computing capacity for a specific period of time. One should pay for all the hours of that time interval; in exchange, the hourly rate is significantly lower than on-demand instances. Reserved Instances can significantly reduce the monetary cost of resources needed to process big data applications in cloud. However, purchases of these instances are non-refundable, and hence, one should be able to estimate the required resources prior to purchase to avoid over-payment. It becomes important especially when the results obtained by big data job has monetary value, such as business intelligence applications. But, estimating the resource demand of big data processing jobs is hard because of numerous factors that affect them such as data locality, data skew, stragglers, internal settings of big data processing framework, interference among instances, instances availability, etc. To maximize the profit of processing such big data jobs in cloud considering fluctuating nature of their resource demand, as well as reserved instances limitations, we propose Reserved Instances Stochastic Allocation (RISA) approach. Using historical traces of resource demand of big data jobs submitted by user, RISA leverages stochastic optimization to determine the amount of resources needed to be reserved for that user to maximize the profit. Our evaluation using real-world traces shows that RISA can increase the net profit by up to 10x, compared to previous approaches. RISA can also find solutions as close as 2% to the best possible solution.

Keddah

As a distributed system, Hadoop heavily relies on the network to complete data-processing jobs. While the traffic generated by Hadoop jobs is critical for job execution performance, the actual behaviour of Hadoop network traffic is still poorly understood. This lack of understanding greatly complicates research relying on Hadoop workloads. In this article, we explore Hadoop traffic through empirical traces. We analyse the generated traffic of multiple types of MapReduce jobs, with varying input sizes, and cluster configuration parameters. We present Keddah, a toolchain for capturing, modelling, and reproducing Hadoop traffic, for use with network simulators to better capture the behaviour of Hadoop. By imitating the Hadoop traffic generation process and considering the YARN resource allocation, Keddah can be used to create Hadoop traffic workloads, enabling reproducible Hadoop research in more realistic scenarios.

Aquarium: an automatic data-processing and experiment information management system for biological macromolecular crystallography beamlines

With the popularity of hybrid pixel array detectors, hundreds of diffraction data sets are collected at a biological macromolecular crystallography (MX) beamline every day. Therefore, the manual processing and recording procedure will be a very time-consuming and error-prone task. Aquarium is an automatic data processing and experiment information management system designed for synchrotron radiation source MX beamlines. It is composed of a data processing module, a daemon module and a web site module. Before experiments, the sample information can be registered into a database. The daemon module will submit data processing jobs to a high-performance cluster as soon as the data set collection is completed. The data processing module will automatically process data sets from data reduction to model building if the anomalous signal is available. The web site module can be used to monitor and inspect the data processing results.

The Era of Artificial Intelligence and Big Data Provides Knowledge Services for the Publishing Industry in China

With the advent of the Internet, we have entered into the information age, which makes it possible and easier to obtain a large number of representatives data. On the other hand, with the rapid improvement of computer hardware, and software especially the computers speed, and the ability of the computer has been greatly improved. Data-driven and algorithm applications of modern artificial intelligence models have been widely used in various fields. Solving problems through machine learning has become a way of thinking that all industries are willing to study to reduce labor costs and improve processing efficiency. Beijing Formax Co., Ltd. takes advantage of its own data processing jobs to apply machine learning technology to some projects, and has achieved good results. This paper focuses on the application of artificial intelligence and Beijing Formax’s knowledge service for the publishing industry in the era of big data including several typical cases.

An Application of Visual Analytics Methods to Cluster and Categorize Data Processing Jobs in High Energy and Nuclear Physics Experiments

An Application of Visual Analytics Methods to Cluster and Categorize Data Processing Jobs in High Energy and Nuclear Physics Experiments

Time- and Cost- Efficient Task Scheduling across Geo-Distributed Data Centers

Typically called big data processing, analyzing large volumes of data from geographically distributed regions with machine learning algorithms has emerged as an important analytical tool for governments and multinational corporations. The traditional wisdom calls for the collection of all the data across the world to a central data center location, to be processed using data-parallel applications. This is neither efficient nor practical as the volume of data grows exponentially. Rather than transferring data, we believe that computation tasks should be scheduled near the data, while data should be processed with a minimum amount of transfers across data centers. In this paper, we design and implement Flutter , a new task scheduling algorithm that reduces both the completion times and the network costs of big data processing jobs across geographically distributed data centers. To cater to the specific characteristics of data-parallel applications, in the case of optimizing the job completion times only, we first formulate our problem as a lexicographical min-max integer linear programming (ILP) problem, and then transform the ILP problem into a nonlinear program problem with a separable convex objective function and a totally unimodular constraint matrix, which can be further solved using a standard linear programming solver efficiently in an online fashion. In the case of improving both time- and cost- efficiency, we formulate the general problem as an ILP problem and we find out that solving an LP problem can achieve the same goal in the real practice. Our implementation of Flutter is based on Apache Spark, a modern framework popular for big data processing. Our experimental results have shown convincing evidence that Flutter can shorten both job completion times and network costs by a substantial margin.

OLD WINE WITH A NEW LABEL : RIGHTS OF DATA SUBJECTS UNDER GDPR

Recent reforms in data privacy protection framework in European Union have lead to enactment of General Data Protection Regulation (GDPR). However, it remains debatable if GDPR would lead to significant improvement in the protection of privacy rights of individuals, which is always considered the fundamental right. The advent of technology and movement of data across geographical barriers and outsourcing of data processing jobs to countries outside the EU necessitated enactments of GDPR. An analysis is done to demonstrate that though some of the provision of GDPR remain generically remain similar to the Data Protection Directive, GDPR has incorporated some new provisions by choosing the ‘regulation’ as an instrument of law for better harmonisation, expensing the ‘right to be forgotten, legitimisation the role of consent, providing data protection by design and default, increasing accountability of data controllers and expanding the scope of provision of the directive to extra territorial jurisdiction would be remain to be seen whether GDPR is an old wine with the new label or something else in a wine bottle.

Old Wine with a New Label: Rights of Data Subjects Under GDPR

Recent reforms in the data privacy protection framework in the European Union have led to the enactment of the General Data Protection Regulation (GDPR). However, it remains debatable whether the GDPR will lead to significant improvements in the protection of the privacy rights of individuals, which are always classified as fundamental rights. The advent of technology, the movement of data across geographical barriers, and the outsourcing of data processing jobs to countries outside the EU necessitated the enactment of the GDPR. Although some of the provisions of the GDPR remain generically similar to analogous provisions in the 1995 Data Protection Directive, the GDPR differs in some respects. As a “Regulation,” not a “Directive,” the GDPR is better equipped to overcome the problem of harmonization across the EU member-states. Furthermore, the GDPR incorporates the ‘right to be forgotten,’ clarifies and fortifies the concept of consent, provides data protection by design and default, increases the accountability of data controllers, and expands the jurisdictional scope of the Directive to include some extra-territorial (i.e., non-EU) data processors and controllers. It remains to be seen whether the GDPR is an old wine with the new label or something new in a wine bottle.