Big Data Processing Architecture Research Articles

A reference architecture for serverless big data processing

Despite significant advances in data management systems in recent decades, the processing of big data at scale remains very challenging. While cloud computing has been well-accepted as a solution to address scalability needs, cloud configuration and operation complexity persist and often present themselves as entry barriers, especially for novice data analysts. Serverless computing and Function-as-a-Service (FaaS) platforms have been suggested to reduce such entry barriers by shifting configuration and operational responsibilities from the application developer to the FaaS platform provider. Naturally, “serverless data processing (SDP)”, that is, using FaaS for (big) data processing, has received increasing interest in recent years.However, FaaS platforms were never intended to support large data processing tasks primarily. SDP, therefore, manifests itself through workarounds and adaptations on the application level, addressing various quirks and limitations of the FaaS platforms in use for data processing needs. This, in turn, creates tensions between the platforms and the applications using them, again encouraging the constant (re-)design of both. Consequently, we present lessons learned from a series of application and platform re-designs that address these tensions, leading to the development of an SDP reference architecture and a platform instantiation and implementation thereof called CREW. Mitigating the tensions through the process of application platform co-design proves to reduce both entry barriers and costs significantly. In some experiments, CREW outperforms traditional, non-SDP big data processing frameworks by factors.

Data Modifications in Blockchain Architecture for Big-Data Processing.

Due to the immutability of blockchain, the integration with big-data systems creates limitations on redundancy, scalability, cost, and latency. Additionally, large amounts of invaluable data result in the waste of energy and storage resources. As a result, the demand for data deletion possibilities in blockchain has risen over the last decade. Although several prior studies have introduced methods to address data modification features in blockchain, most of the proposed systems need shorter deletion delays and security requirements. This study proposes a novel blockchain architecture called Unlichain that provides data-modification features within public blockchain architecture. To achieve this goal, Unlichain employed a new indexing technique that defines the deletion time for predefined lifetime data. The indexing technique also enables the deletion possibility for unknown lifetime data. Unlichain employs a new metadata verification consensus among full and meta nodes to avoid delays and extra storage usage. Moreover, Unlichain motivates network nodes to include more transactions in a new block, which motivates nodes to scan for expired data during block mining. The evaluations proved that Unlichain architecture successfully enables instant data deletion while the existing solutions suffer from block dependency issues. Additionally, storage usage is reduced by up to 10%.

Big Data Processing over Cloud Computing Environment

The topics of this special paper are mainly devoted to the most recent research, development and applications in the field of cloud computing, big data processing in the various fields, e.g. Web blog, search engine, image processing, and industry. Among all of the submitted manuscripts, 15 papers were selected and included in this special paper. Creative thoughts and interesting inspirations are presented, discussed, and disseminated in this paper. First, there are three papers on the cloud architecture and optimization for big data processing. J. Peng, et al. discussed the problems of load balance for massive data processing on cloud considering the data processing efficiency and nodes’ capability. A load balance approach for massive data, based on the consistent hashing method, was presented to improve the data processing equalization and high processing efficiency. Large-scale sensor networks and internet of things are one of the most important application fields in big data. Y. Sun, et al. proposed a cloud computing system to support PaaS in energy power applications—PROXZONE. PROXZONE provides the cloud services uniform monitoring and authorization, and cloud message service, which can support good performance when dealing with big data. Meanwhile, S. Xu, et al. discussed an improved cooperative dynamic cluster model in multitasks mobile network, and proposed a cooperative message forwarding mechanism to improve the network performance.

A State of Art Survey for Big Data Processing and NoSQL Database Architecture

A State of Art Survey for Big Data Processing and NoSQL Database Architecture

Examining a generic streaming architecture for smart manufacturing's Big data processing in Anomaly detection: A review and a proposal

The smart manufacturing industry has witnessed a rapid increase in data generation due to the integration of sensors, IoT devices, and other advanced technologies. With this huge amount of data, the need for efficient data processing methods becomes critical for identifying anomalies in real-time. With the rise of Industry 4.0 practices, digitally enabled manufacturing units are shifting their focus towards Smart Manufacturing paradigm for better productivity, throughput and increased business volume. Traditionally digital manufacturing units have considered different AI approaches like Neural Network, Statistical Methods, Deep Learning etc. to detect and predict anomalies in their production lines. But with the Smart Manufacturing ecosystem, a manufacturing unit must integrate manufacturing intelligence in real-time across entire production lines through sensor data of IOT devices. Hence the traditional anomaly detection systems fall short to respond well, under the changed scenario, where large volumes of unstructured and varied types of data are being generated at high velocity, to be processed at (soft) real time. The article reviews the current state-of-the-art in big data processing for anomaly detection in smart manufacturing. The review covers various aspects such as data collection, data processing, anomaly detection, and real-time monitoring. The current paper also proposes a novel stateful data streaming computational model for big data processing in smart manufacturing units which conceptually lays the foundation on top of which any discrete anomaly detection engine would be able to work. The proposed architecture has several benefits, including its ability to handle the large volume, velocity, and variety of data generated in smart manufacturing. The architecture can be applied to various smart manufacturing applications, including predictive maintenance, quality control, and supply chain optimization. It is expected that this proposed architecture will pave the way for the development of more efficient and effective smart manufacturing systems in the future.

Handling Big Microarray Data: A Novel Approach to Design Accurate Fuzzy-Based Medical Expert System

The genes data produced by microarray experiments is complex in terms of dimensions and samples. It consumes a lot of computation power and time when it is processed for a disease analysis while working with an expert system. At the same time, data can help doctors identify a patient’s health condition if it is presented in a meaningful way and processed on time. Several methods have been proposed to reduce the dimensions of medical microarray data and optimize its search space with minimal accuracy loss. However, the discretization of continuous gene-values in the process of dimension reduction is failed to preserve the inherent meaning of genes. Also, ensuring high accuracy and interpretability in the reduction process may result in extra processing time, which is unfavorable for time-critical applications. To overcome these issues, in this paper, we propose a dimension reduction method in conjunction with an expert system optimization approach, while keeping an accuracy-interpretability-speedy tradeoff in mind. To accomplish this, we use a fuzzy rough set on f-information to identify meaningful genes without changing their original values. We propose a conditionally guided particle swarm optimization for faster knowledge acquisition, where the velocity is adjusted based on a predefined update probability, resulting in a faster search. A big data processing architecture is designed using the Hadoop ecosystem along with a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MapReduce</i> -equivalent algorithm of the proposed method for speedy processing, enabling parallel processing on microarray data to reduce dimensions and perform classification through knowledge extraction. The proposed method is thoroughly tested on eleven microarray datasets by considering accuracy-interpretability-speed tradeoff. The results show that the proposed method is effective in identifying disease-causing genes while also understanding the patient’s genetic profile with only a few operations and a small amount of CPU time. Statistical tests are also run to validate the proposed method’s efficacy in comparison to other methods.

Research on Distributed Calculation of Flight Parameters Based on Hadoop

With the improvement of launch vehicle technology and the increase of launch missions, under the intensive launch tasks, the contradiction between the large-scale calculation demand of flight parameters of launch vehicle and the traditional standalone calculation mode is increasingly prominent, which is mainly reflected in the slow calculation speed, low processing efficiency, limited bandwidth bottleneck, and single point fault. Big data processing architecture Hadoop’s distributed computing framework MapReduce, running in low cost cluster, is innovative applied in the large-scale calculation of launch vehicle’s flight parameters, relying on its distributed storage, coordination and load balancing mechanism. The method effectively improves computing efficiency, breakthroughs performance bottleneck and avoids single point failure. Compared with the traditional standalone deployment and pseudo-distributed cluster deployment based on Hadoop, the fully distributed cluster deployment based on Hadoop is the optimal deployment for calculating flight parameters. The results show that the parallel calculation of flight parameters based on Hadoop could save time used by 51% with the consistency of calculation results, which is the same as the standalone deployment.

Investigation of Coal Preparation for Life Cycle by Using Building Information Modeling (BIM): A Case Study

Increasing the efficiency of coal preparation is the key link of mine production. This study proposed a framework of coal preparation for life cycle based on Building Information Modeling (BIM), in which the data resources can be retransmitted, and the integration of BIM application links has a high level. In this paper, the kappa big data processing architecture is used to realize the integration and unification of stream data and batch data processing process. By using deep learning method and multimodal data fusion method, the multimodal data association fusion is realized, and Bentley software is adopted for verification and integration. Taking a coal preparation plant as an engineering case study, the proposed framework is applied to build a big data subplatform of three-dimensional visual management. The following is suggested: (1) the three-dimensional visual review of the coal preparation online is realized, and the efficiency of the design results is improved; (2) the production and operation management of the coal preparation are carried out through the established platform, which indicated that the proportion of clean coal increased by 1.97%, and the average increase percentage for the total output is about 5.6%. Results from the case study show that the framework proposed in this study can provide efficient reference for coal washing in the coal preparation.

Monitoring and Analysis of Distributed New Energy Resources Based on the Internet of Things

In order to accurately monitor and analyze new energy resources in real time, the analysis and design of a distributed new energy resource monitoring device based on the Internet of Things cloud platform is carried out, and a Hadoop distributed new energy system based on the Internet of Things cloud computing is designed. It analyzes the computing architecture of parallel collection and distributed storage of new energy resource monitoring data, analyzes the infrastructure system of the distributed system, and builds the big data processing architecture and computing service architecture for the electricity information collection system based on the Hadoop cluster. Tasks are processed in parallel to improve the computing efficiency of electricity consumption big data; at the same time, in order to solve the problem of insufficient storage space caused by the sharp increase in the amount of data in the electricity consumption information collection system and the difficulty of data interaction at all levels, a distributed database is developed. The modular design of the parallel data collection system of HBase and the data warehouse tool Hive, and the distributed data calculation design based on the MapReduce method have improved the efficiency of data collection and the unified storage of heterogeneous data. The research provides reference for realizing an efficient and safe new energy resource monitoring data processing system.

Big Data Processing and Intelligent Analysis System for Students’ Learning State Based on PaddlePaddle

This paper studies the construction idea, method and process of an intelligent learning state analysis system on the background of big data acquisition and processing. Based on the software and hardware of Baidu PaddlePaddle, the intelligent subsystem of feature extraction, face recognition and learning state recognition in the process of teaching process is built. Based on the idea of distributed scalability, big data processing architecture for large-scale data acquisition, storage and analysis is designed. The actual scene shows that this is an intelligent system used for students learning state analysis, which is supported by big data processing, can quickly carry out the teaching attendance, examination verification and class state detection, and can be widely used in learning state analysis of all kinds of schools at all levels.

Big Data Processing Architecture for Smart Farming

In the era of Big data, data-driven farming is changing the agricultural businesses thanks to the use of modern technologies such as the Internet of Things (IoT) sensors, drones, and farm monitoring. IoT devices produce a massive amount of precious agri-data, which are collected and analyzed in real-time using innovative application tools. This combination of technology, known as “Smart farming”, helps various stakeholders in the agri-ecosystem to monitor crops in real-time, as well as maximize productivity and profitability in farm and business operations with the minimum efforts. Although many Smart farming solutions have been introduced, both from industry and academia, universal applicability of these approaches for other farms, unfortunately, is not feasible. Most of these solutions are based on a home-made non-standard Big data processing architecture. In this paper, we propose WALLeSMART, a cloud-based Smart farming management system, applied to the Wallonia region of Belgium. The framework introduces a general architecture to address the challenges of acquisition, processing, storing, and visualization of very large amounts of data, both in batch and real-time basis. An initial prototype has been developed and tested with various farms showing prominent results.

The problem of analysis of big web data and the use of data mining technology for processing and searching patterns in big web data on a practical example

The purpose of the work is to study the current problems and prospects of the solution for processing big data received or stored in the Internet (web data), as well as the possibility of practical realization of Data Mining technology for big web data on practical example. Materials and methods. The study included a review of bibliographic sources on big data analysis problems.Data Mining technology was used to analyze large web data, as well as computer modeling of a practical problem using the C # programming language and creating a DDL database structure for accumulating web data.Results. In the course of the work, the specifics of big data were described, the main characteristics of big data were highlighted, and modern approaches to processing big data were analyzed. A brief description of the horizontal-scalable architecture and the BI-solution architecture for big data processing is given. The problems of processing large web data are formulated: limiting the speed of access to data, providing access via network protocols through general-purpose networks.An example showing the approach to processing large web data was also implemented. Based on the idea of big data, the described complexities of web data processing and the methods of Data Mining, techniques were proposed for effectively solving the practical problem of processing and searching patterns in a large data array.The following classes have been developed in the C # programming language:Class of receiving web data via the Internet; Data conversion class;Intelligent data processing class;Created DDL script that creates a structure for the accumulation of web data.A single UML class diagram has been developed.The constructed system of data and classes allows to solve the main part of the problems of processing large web data and perform intelligent processing using Data Mining technology in order to solve the problem posed of identifying certain records in a large array. The combination of object-oriented approach, neural networks and BI-analysis to filter data will speed up the process of data processing and obtaining the result of the studyConclusion. According to the results of the study, it can be argued that the current state of technology for analyzing large web data allows you to efficiently process data objects, identify patterns, get hidden data and get full-fledged statistical data.The obtained results can be used both for the purpose of the initial study of big data processing technologies, and as a basis for developing an already real application for analyzing web data. The use of neural networks and the created universal classes-handlers makes the created architecture flexible and self-learning, and the class declarations and the base DDL structure will greatly simplify the development of program code.

On construction of an energy monitoring service using big data technology for the smart campus

In this study, we combine cloud computing with big data processing techniques to build a real-time energy monitoring system for smart campus. The monitor plat-form collects the electricity usage in campus buildings through smart meters and environmental sensors, and processes the huge amount of data by big data processing techniques. A Hadoop ecosystem is built on top of big data processing architecture to improve the capacity of big data storage and processing ability. Moreover, we compare the performance of Hive and HBase in searching energy data, and the performance of relational database and big data distributed database for data search. We also identify abnormal electrical condition through the MapReduce framework, and compared the difference of performances between Spark and Hadoop in real-time processing. The proposed system has been implemented in Tunghai University campus. It enables administrators to observe the real-time electricity usage and analyze historical data anytime and from anyplace.

A five-layer architecture for big data processing and analytics

A five-layer architecture for big data processing and analytics

A five-layer architecture for big data processing and analytics

Big data technologies have attracted much attention in recent years. The academia and industry have reached a consensus, that is, the ultimate goal of big data is about transforming 'big data' to '...

A New Data Processing Architecture for Multi-Scenario Applications in Aviation Manufacturing

The development of industry 4.0 has spurred the transformation of traditional manufacturing into modern industrial Internet-of-Things. The most notable feature during this transition is the improvement of digitization and intelligence based on the massive data drives. In such a data-driven environment, the processing, storage, and utilization of the industry data get more and more important. Usually, the traditional data processing architecture runs as a one-way streamline, which cannot adapt to the different requirements of the multi-scenario application. This paper proposed a new industrial big data processing architecture called Phi architecture, which can realize many functions such as batch data processing and stream data processing, distributed storage and access, and real-time control. Compared with other data processing architecture, the Phi architecture combined with edge computing and feedback control has the ability to deal with the different demands in aviation manufacturing. Next, the new architecture is designed for microservices pattern, which improves the flexibility and stability of the architecture, and makes it independent operated in multi-scenarios, such as state monitoring of workshop, adaptive data acquisition, feedback control, and user-oriented information classification. As a proof of concept, the architecture has been tested in a simulation digital manufacturing workshop. The results verify the improved effectiveness of the Phi architecture on the data feedback control and real-time processing. And, the development of microservices architecture greatly improves the efficiency, adaptability, and extensibility of the manufacturing process.

Big Data Processing and Analytics Platform Architecture for Process Industry Factories

This paper describes the architecture of a cross-sectorial Big Data platform for the process industry domain. The main objective was to design a scalable analytical platform that will support the collection, storage and processing of data from multiple industry domains. Such a platform should be able to connect to the existing environment in the plant and use the data gathered to build predictive functions to optimize the production processes. The analytical platform will contain a development environment with which to build these functions, and a simulation environment to evaluate the models. The platform will be shared among multiple sites from different industry sectors. Cross-sectorial sharing will enable the transfer of knowledge across different domains. During the development, we adopted a user-centered approach to gather requirements from different stakeholders which were used to design architectural models from different viewpoints, from contextual to deployment. The deployed architecture was tested in two process industry domains, one from the aluminium production and the other from the plastic molding industry.

Fog Based Intelligent Transportation Big Data Analytics in The Internet of Vehicles Environment: Motivations, Architecture, Challenges, and Critical Issues

The intelligent transportation system (ITS) concept was introduced to increase road safety, manage traffic efficiently, and preserve our green environment. Nowadays, ITS applications are becoming more data-intensive and their data are described using the “5Vs of Big Data”. Thus, to fully utilize such data, big data analytics need to be applied. The Internet of vehicles (IoV) connects the ITS devices to cloud computing centres, where data processing is performed. However, transferring huge amount of data from geographically distributed devices creates network overhead and bottlenecks, and it consumes the network resources. In addition, following the centralized approach to process the ITS big data results in high latency which cannot be tolerated by the delay-sensitive ITS applications. Fog computing is considered a promising technology for real-time big data analytics. Basically, the fog technology complements the role of cloud computing and distributes the data processing at the edge of the network, which provides faster responses to ITS application queries and saves the network resources. However, implementing fog computing and the lambda architecture for real-time big data processing is challenging in the IoV dynamic environment. In this regard, a novel architecture for real-time ITS big data analytics in the IoV environment is proposed in this paper. The proposed architecture merges three dimensions including intelligent computing (i.e. cloud and fog computing) dimension, real-time big data analytics dimension, and IoV dimension. Moreover, this paper gives a comprehensive description of the IoV environment, the ITS big data characteristics, the lambda architecture for real-time big data analytics, several intelligent computing technologies. More importantly, this paper discusses the opportunities and challenges that face the implementation of fog computing and real-time big data analytics in the IoV environment. Finally, the critical issues and future research directions section discusses some issues that should be considered in order to efficiently implement the proposed architecture.

Big Data Processing Architecture for Radio Signals Empowered by Deep Learning: Concept, Experiment, Applications and Challenges

In modern society, the demand for radio spectrum resources is increasing. As the information carriers of wireless transmission data, radio signals exhibit the characteristics of big data in terms of volume, variety, value, and velocity. How to uniformly handle these radio signals and obtain value from them is a problem that needs to be studied. In this paper, a big data processing architecture for radio signals is presented and a new approach of end-to-end signal processing based on deep learning is discussed in detail. The radio signal intelligent search engine is used as an example to verify the architecture, and the system components and experimental results are introduced. In addition, the applications of the architecture in cognitive radio, spectrum monitoring, and cyberspace security are introduced. Finally, challenges are discussed, such as unified representation of radio signal features, distortionless compression of wideband sampled data, and deep neural networks for radio signals.

RedEdge: A Novel Architecture for Big Data Processing in Mobile Edge Computing Environments

We are witnessing the emergence of new big data processing architectures due to the convergence of the Internet of Things (IoTs), edge computing and cloud computing. Existing big data processing architectures are underpinned by the transfer of raw data streams to the cloud computing environment for processing and analysis. This operation is expensive and fails to meet the real-time processing needs of IoT applications. In this article, we present and evaluate a novel big data processing architecture named RedEdge (i.e., data reduction on the edge) that incorporates mechanism to facilitate the processing of big data streams near the source of the data. The RedEdge model leverages mobile IoT-termed mobile edge devices as primary data processing platforms. However, in the case of the unavailability of computational and battery power resources, it offloads data streams in nearer mobile edge devices or to the cloud. We evaluate the RedEdge architecture and the related mechanism within a real-world experiment setting involving 12 mobile users. The experimental evaluation reveals that the RedEdge model has the capability to reduce big data stream by up to 92.86% without compromising energy and memory consumption on mobile edge devices.