Real-time Big Data Research Articles

Real Time Big Data Analysis by Using Apache Kudu and NoSQL Redis in Web Applications

In the recent years data processing in Big Data applications are moving from a batch processing model to a live streaming processing model. The live streaming processing model allows to process data in a real time or near real time manner. In many cases, there are need of combining data from various sources - both new generated data and data previously gathered and archived. In each case a different storage mechanism is appropriate to achieve best flexibility and to avoid the so-called bottleneck when processing data. In this article we review some of the possibilities of using Apache Kudu and NoSQL Redis systems which made them suitable to be used together for fast processing of streaming data.

Real-Time Healthcare Monitoring System using Online Machine Learning and Spark Streaming

The real-time monitoring and tracking systems play a critical role in the healthcare field. Wearable medical devices with sensors, mobile applications, and health cloud have continuously generated an enormous amount of data, often called streaming big data. Due to the higher speed of the streaming data, it is difficult to ingest, process, and analyze such huge data in real-time to make real-time actions in case of emergencies. Using traditional methods that are inadequate and time-consuming. Therefore, there is a significant need for real-time big data stream processing to guarantee an effective and scalable solution. So, we proposed a new system for online prediction to predict health status using Spark streaming framework. The proposed system focuses on applying streaming machine learning models (i.e. streaming linear regression with SGD) on streaming health data events ingested to spark streaming through Kafka topics. The experimental results are done on the historical medical datasets (i.e. diabetes dataset, heart disease dataset, and breast cancer dataset) and generated dataset which is simulated to wearable medical sensors. The historical datasets have shown that the accuracy improvement ratio obtained using the diabetes disease dataset is the highest one with respect to the other two datasets with an accuracy of 81%. For generated datasets, the online prediction system has achieved accuracy with 98% at 5 seconds window size. Beyond this, the experimental results have proofed that the online prediction system can online learn and update the model according to the new data arrival and window size.

A Data Security Enhanced Access Control Mechanism in Mobile Edge Computing

Mobile edge computing, with characteristics of position awareness, mobile support, low latency, decentralization, and distribution, has received widespread attention from industry and academia, and has been applied to application areas such as intelligent transportation, smart city, and real-time big data analysis. However, it also brings the new security threats, especially data security threats during data access that leads to unauthorized/unauthorized access, alteration and disclosure of data, affecting the confidentiality and integrity of the data. Therefore, access control, as an important method to ensure the security of user data during data access, began to be applied to mobile edge computing. However, the existing access control has the disadvantages of coarse-grain, poor flexibility and accuracy, lack of internal attack considerations, etc., which cannot meet the needs of data security in practical applications of mobile edge computing. In this paper, a data security enhanced Fine-Grained Access Control mechanism (FGAC) is proposed to ensure data security during data access in mobile edge computing. In FGAC, a dynamic fine-grained trusted user grouping scheme based on attributes and metagraphs theory was first designed. Secondly, the scheme was combined with the traditional role-based access control mechanism to assign roles to users based on user group credibility. And then, based on attribute matching the user authentication further verifies whether the user is allowed to perform the access operations to achieve fine-grained data protection. Experimental results show that FGAC can effectively identify malicious users and make group adjustments, while achieving fine-grained access control and assure the data security during the data access process in mobile edge computing.

Conceptual Model for Connected Vehicles Safety and Security using Big Data Analytics

The capability of Connected Vehicles (CVs) connecting to a nearby vehicle, surrounding infrastructure and cyberspace presents a high risk in the aspect of safety and security of the CV and others. Data volume generated from the sensors and infrastructure in CVs environment are enormous. Thus, CVs implementations require a real-time big data processing and analytics to detect any anomaly in the CVs’s environment which are physical layer, network layer and application layer. CVs are exposed to various vulnerabilities associated with exploitations or malfunctions of the components in each layer that could result in various safety and security event such as congestion and collision. The safety and security risks added an extra layer of required protection for the CVs implementation that need to be studied and refined. To address this gap, this research aims to determine the basic components of safety and security for CVs implementation and propose a conceptual model for safety and security in CVs by applying the machine learning and deep learning techniques. The proposed model is highly correlated to safety and security and could be applied in congestion and collision prediction.

Smart Industrial Value Creation, Cyber-Physical Production Networks, and Real-Time Big Data Analytics in Sustainable Internet of Things-based Manufacturing Systems

Smart Industrial Value Creation, Cyber-Physical Production Networks, and Real-Time Big Data Analytics in Sustainable Internet of Things-based Manufacturing Systems

The real-time big data processing method based on LSTM or GRU for the smart job shop production process

With the wide application of intelligent sensors and internet of things (IoT) in the smart job shop, a large number of real-time production data is collected. Accurate analysis of the collected data can help producers to make effective decisions. Compared with the traditional data processing methods, artificial intelligence, as the main big data analysis method, is more and more applied to the manufacturing industry. However, the ability of different AI models to process real-time data of smart job shop production is also different. Based on this, a real-time big data processing method for the job shop production process based on Long Short-Term Memory (LSTM) and Gate Recurrent Unit (GRU) is proposed. This method uses the historical production data extracted by the IoT job shop as the original data set, and after data preprocessing, uses the LSTM and GRU model to train and predict the real-time data of the job shop. Through the description and implementation of the model, it is compared with KNN, DT and traditional neural network model. The results show that in the real-time big data processing of production process, the performance of the LSTM and GRU models is superior to the traditional neural network, K nearest neighbor (KNN), decision tree (DT). When the performance is similar to LSTM, the training time of GRU is much lower than LSTM model.

Virtual technology of cache and real-time big data distribution in cloud computing big data center

By comparing several cloud computing of big data network center during COVID-19, this paper proposes a new topology model, which realizes two functions of cloud computing big data center caching and big data real-time distribution. In addition, cloud computing network requires higher performance than traditional application big data center, which makes the consideration of network platform construction performance different from the traditional understanding. During COVID-19, we deeply understood the underlying attributes of cloud, combined with the topology model, we can realize the decoupling of cloud computing big data system, change the situation of direct connection between upstream and downstream, and have more reliable and efficient transmission of message and command big data.

A new Internet of Things architecture for real-time prediction of various diseases using machine learning on big data environment

A number of technologies enabled by Internet of Thing (IoT) have been used for the prevention of various chronic diseases, continuous and real-time tracking system is a particularly important one. Wearable medical devices with sensor, health cloud and mobile applications have continuously generating a huge amount of data which is often called as streaming big data. Due to the higher speed of the data generation, it is difficult to collect, process and analyze such massive data in real-time in order to perform real-time actions in case of emergencies and extracting hidden value. using traditional methods which are limited and time-consuming. Therefore, there is a significant need to real-time big data stream processing to ensure an effective and scalable solution. In order to overcome this issue, this work proposes a new architecture for real-time health status prediction and analytics system using big data technologies. The system focus on applying distributed machine learning model on streaming health data events ingested to Spark streaming through Kafka topics. Firstly, we transform the standard decision tree (DT) (C4.5) algorithm into a parallel, distributed, scalable and fast DT using Spark instead of Hadoop MapReduce which becomes limited for real-time computing. Secondly, this model is applied to streaming data coming from distributed sources of various diseases to predict health status. Based on several input attributes, the system predicts health status, send an alert message to care providers and store the details in a distributed database to perform health data analytics and stream reporting. We measure the performance of Spark DT against traditional machine learning tools including Weka. Finally, performance evaluation parameters such as throughput and execution time are calculated to show the effectiveness of the proposed architecture. The experimental results show that the proposed system is able to effectively process and predict real-time and massive amount of medical data enabled by IoT from distributed and various diseases.

Real-time BigData and Predictive Analytical Architecture for healthcare application

The amount of data produced within health informatics has grown to be quite vast. The large volume of data generated by various vital sign monitoring devices needs to be analysed in real time to alert the care providers about changes in a patients condition. Data processing in real time has complex challenges for the large volume of data. The real-time system should be able to collect millions of events per seconds and handle parallel processing to extract meaningful information efficiently. In our study, we have proposed a real-time BigData and Predictive Analytical Architecture for healthcare application. The proposed architecture comprises three phases: (1) collection of data, (2) offline data management and prediction model building and (3) real-time processing and actual prediction. We have used Apache Kafka, Apache Sqoop, Hadoop, MapReduce, Storm and logistic regression to predict an emergency condition. The proposed architecture can perform early detection of emergency in real time, and can analyse structured and unstructured data like Electronic Health Record (EHR) to perform offline analysis to predict patient’s risk for disease or readmission. We have evaluated prediction performance on different benchmark datasets to detect an emergency condition of any patient in real time and possibility of readmission.

Urban commuting dynamics in response to public transit upgrades: A big data approach.

Public transit, especially urban rail systems, plays a vital role in shaping commuting patterns. Compared with census data and survey data, large-scale and real-time big data can track the impacts of urban policy implementations at finer spatial and temporal scales. Therefore, this study proposed a multi-level analytical framework using transit smartcard data to examine urban commuting dynamics in response to rail transit upgrades. The study area was Shenzhen, one of the most highly urbanized and densely populated cities in China, which provides the opportunity to examine the effects of rail transit upgrades on commuting patterns in a rapidly developing urban context. Changes in commuting patterns were examined at three levels: city, region, and individual. At the city level, we considered the average commuting time, commuting speed, and commuting distance across the whole city. At the region level, we analyzed changes in the job accessibility of residential zones. Finally, this study evaluated the potential effects of rail transit upgrades on the jobs-housing relationship at the individual level. Difference-in-difference models were used for causal inference between rail transit upgrades and commuting patterns. In the very short term, the opening of new rail transit lines resulted in no significant changes in overall commuting patterns across the whole city; however, two effects of rail transit upgrades on commuting patterns were identified. First, rail transit upgrades enhanced regional connectivity between residential zones and employment centers, thus improving job accessibility. Second, rail transit improvement increased the commuting distances of individuals and contributed to the separation of workplaces and residences. This study provides meaningful insights into the effects of rail transit upgrades on commuting patterns.

Manufacturing big data ecosystem: A systematic literature review

Abstract Advanced manufacturing is one of the core national strategies in the US (AMP), Germany (Industry 4.0) and China (Made-in China 2025). The emergence of the concept of Cyber Physical System (CPS) and big data imperatively enable manufacturing to become smarter and more competitive among nations. Many researchers have proposed new solutions with big data enabling tools for manufacturing applications in three directions: product, production and business. Big data has been a fast-changing research area with many new opportunities for applications in manufacturing. This paper presents a systematic literature review of the state-of-the-art of big data in manufacturing. Six key drivers of big data applications in manufacturing have been identified. The key drivers are system integration, data, prediction, sustainability, resource sharing and hardware. Based on the requirements of manufacturing, nine essential components of big data ecosystem are captured. They are data ingestion, storage, computing, analytics, visualization, management, workflow, infrastructure and security. Several research domains are identified that are driven by available capabilities of big data ecosystem. Five future directions of big data applications in manufacturing are presented from modelling and simulation to real-time big data analytics and cybersecurity.

Deep Learning Based on 1-D Ensemble Networks Using ECG for Real-Time User Recognition

The postmobile era will go beyond using individual smart devices and allow for user interaction by connecting various devices with sensing capabilities, such as smartphones, wearable devices, automobiles, and the Internet of Things. Wearable devices can continuously collect a variety of information on the users and their environment as the devices are worn in daily life. Because of this, real-time big data analysis technology is needed. This paper proposes a deep learning-based ensemble network model for improving the performance and overcoming the problems, which can occur on a single network. This model is designed so that the features produced by n number of single networks are combined and relearned. In addition, different parameter values are used on each single network, and the data used in the experiments are generated by the fiducial point method, which uses feature point detection, and the nonfiducial point method for periods of 1 sec and n sec. In the experiment results, in the case of fiducial point-based ECG signals, the ensemble network recognition performance shows a maximum of 0.8% higher accuracy than that of the single network. In the case of a 1 sec period nonfiducial point-based ECG signal, the ensemble network recognition performance is a minimum of 0.4% and a maximum of 1% higher than that of the single network. In the case of an n sec period, there is a maximum difference of 1.3%, and the proposed ensemble network shows better performance than the single network.

A study of developments and applications of mixed reality cubicles and their impact on learning

PurposeThe purpose of this paper is to report on developments and applications of mixed reality cubicles and their impacts on learning in higher education. This paper investigates and presents the cost effective application of augmented reality (AR) as a mixed reality technology via or to mobile devices such as head-mounted devices, smart phones and tablets. Discuss the development of mixed reality applications for mobile (smartphones and tablets) devices leading up to the implementation of a mixed reality cubicle for immersive three dimensional (3D) visualizations.Design/methodology/approachThe approach adopted was to limit the considerations to the application of AR via mobile platforms including head-mounted devices with focus on smartphones and tablets, which contain basic feedback–to-user channels such as speakers and display screens. An AR visualization cubicle was jointly developed and applied by three collaborating institutions. The markers, acting as placeholders acts as identifiable reference points for objects being inserted in the mixed reality world. Hundreds of participants comprising academics and students from seven different countries took part in the studies and gave feedback on impact on their learning experience.FindingsResults from current study show less than 30 percent had used mixed reality environments. This is lower than expected. About 70 percent of participants were first time users of mixed reality technologies. This indicates a relatively low use of mixed reality technologies in education. This is consistent with research findings reported that educational use and research on AR is still not common despite their categorization as emerging technologies with great promise for educational use.Research limitations/implicationsCurrent research has focused mainly on cubicles which provides immersive experience if used with head-mounted devices (goggles and smartphones), that are limited by their display/screen sizes. There are some issues with limited battery lifetime for energy to function, hence the need to use rechargeable batteries. Also, the standard dimension of cubicles does not allow for group visualizations. The current cubicle has limitations associated with complex gestures and movements involving two hands, as one hand are currently needed for holding the mobile phone.Practical implicationsThe use of mixed reality cubicles would allow and enhance information visualization for big data in real time and without restrictions. There is potential to have this extended for use in exploring and studying otherwise inaccessible locations such as sea beds and underground caves. Social implications – Following on from this study further work could be done to developing and application of mixed reality cubicles that would impact businesses, health and entertainment.Originality/valueThe originality of this paper lies in the unique approach used in the study of developments and applications of mixed reality cubicles and their impacts on learning. The diverse composition in nature and location of participants drawn from many countries comprising of both tutors and students adds value to the present study. The value of this research include amongst others, the useful results obtained and scope for developments in the future.

Real-time big data processing for instantaneous marketing decisions: A problematization approach

The collection of big data from different sources such as the internet of things, social media and search engines has created significant opportunities for business-to-business (B2B) industrial marketing organizations to take an analytical view in developing programmatic marketing approaches for online display advertising. Cleansing, processing and analyzing of such large datasets create challenges for marketing organizations — particularly for real-time decision making and comparative implications. Importantly, there is limited research for such interplays. By utilizing a problematization approach, this paper contributes through the exploration of links between big data, programmatic marketing and real-time processing and relevant decision making for B2B industrial marketing organizations that depend on big data-driven marketing or big data-savvy managers. This exploration subsequently encompasses appropriate big data sources and effective batch and real-time processing linked with structured and unstructured datasets that influence relative processing techniques. Consequently, along with directions for future research, the paper develops interdisciplinary dialogues that overlay computer-engineering frameworks such as Apache Storm and Hadoop within B2B marketing viewpoints and their implications for contemporary marketing practices.

Intelligent Embedded Vision for Summarization of Multiview Videos in IIoT

Nowadays, video sensors are used on a large scale for various applications, including security monitoring and smart transportation. However, the limited communication bandwidth and storage constraints make it challenging to process such heterogeneous nature of Big Data in real time. Multiview video summarization (MVS) enables us to suppress redundant data in distributed video sensors settings. The existing MVS approaches process video data in offline manner by transmitting them to the local or cloud server for analysis, which requires extra streaming to conduct summarization, huge bandwidth, and are not applicable for integration with industrial Internet of Things (IIoT). This article presents a light-weight convolutional neural network (CNN) and IIoT-based computationally intelligent (CI) MVS framework. Our method uses an IIoT network containing smart devices, Raspberry Pi (RPi) (clients and master) with embedded cameras to capture multiview video data. Each client RPi detects target in frames via light-weight CNN model, analyzes these targets for traffic and crowd density, and searches for suspicious objects to generate alert in the IIoT network. The frames of each client RPi are encoded and transmitted with approximately 17.02% smaller size of each frame to master RPi for final MVS. Empirical analysis shows that our proposed framework can be used in industrial environments for various applications such as security and smart transportation and can be proved beneficial for saving resources. 1 1 [Online]. Available: https://github.com/tanveer-hussain/Embedded-Vision-for-MVS .

A framework of smart pipeline system and its application on multiproduct pipeline leakage handling

The management system is essential to ensure the smooth operation of the long-distance pipeline. Here, a framework of a smart pipeline system and its application on multiproduct pipeline network are proposed where a pipeline monitoring and accidental leak handling system based on real-time Big Data and Distributed Computing for integrity management is introduced. A novel construction of data-chain and module-chain in the management of pipeline integrity is given. New computational modules for locating a leak, calculating leakage volume and executing shutdown procedure are proposed for emergency treatment. A new optimal model aiming at minimal ALV (accumulative leakage volume) is established to obtain a pipeline shutdown scheme. Several experiments relying on multiproduct pipelines are implemented to test the effectiveness of the framework and evaluate the applicability of the modules. The results demonstrate that the estimations of the starting time, location, coefficient, and leakage volume could be fulfilled within the duration of a negative pressure wave going through the selected pipeline. Meanwhile, the pre-forming of the pipeline shutdown procedure as well as obtaining the ALV could be achieved. The outcomes could provide guidance for the construction of smart pipeline network, pipeline shutdown management, prediction of leakage influential range and subsequent incident investigations.

Log Data Modeling and Acquisition in Supporting SaaS Software Performance Issue Diagnosis

Data logging is helpful for the operation and maintenance manager of SaaS-based solutions to diagnose performance issues. However, long-running SaaS software may generate huge amounts of log data which is difficult to analyze, and it lacks a systematic approach to collect the running log and lacks a unified data structure to normalize the performance-related data. All these threaten the timeliness of SaaS performance issue diagnosis. In this paper, we propose an architecture for log collection and analysis to support the assessment of performance and diagnosis of performance issues of SaaS-based application in cloud computing. The architecture has the three-tier structure and includes a pivot data model to integrate heterogeneous log. The two high-level metrics in the model of Average Response Time (ART) and Request Timeout Rate (RTR) are calculated by statistical measurement and the lower-level metrics are monitored in real-time. Operation and maintenance managers can evaluate the performance of SaaS software based on the high-level metrics, then timely locate the issues from the low-level metrics and take appropriate measures. Thereupon, this study presents the general-purpose technique for the architecture to support real-time big log data collection, access, computation, storage. The proposal has been implemented and validated in a case study.

IoT Analysis and Processing System using Sensed Data for Laboratory Safety Management

This study proposes and implements a real-time message analysis processing system using sensed data for safety management of a laboratory composed of Internet of Things (IoT) sensor devices. Existing IoT systems have difficulty storing data in a relational database system due to the different data formats between sensor devices, and they also have preprocessing overhead due to nonstandard communications protocols. Furthermore, they are centralized systems with unstable service continuity, and it is difficult to add new functions or extend the system because of the complex connections among sensor devices and services. Studies on real-time analysis and processing of large volumes of sensor data are also insufficient. To address these problems, the proposed system uses IoT sensor devices, based on standard communications protocols, and message queue brokers based on standard messaging protocols. The message queue brokers can facilitate adding sensor devices or services by delivering the sensor data from the gateway and central server to the services. Services can continue, even in the event of network trouble, because the services that had been provided from the central server are now provided through smart gateways. The central server can provide risk analysis and prevention services for laboratory safety management through Apache Spark, a real-time big data analysis and processing platform. Through the experiments in this study, the proposed real-time message analysis processing system for laboratory safety management was implemented. The message queue brokers between the standard Message Queuing Telemetry Transport (MQTT) protocol and the smart gateway receiving the data were built into sensor devices equipped with network and sensor modules. Exchange routing keys are also specified by topic and service for each sensor device in line with the laboratory safety management environment. Data transmission processing is handled by a federated connection of the gateway brokers and the central server brokers.

Designing a Smart Transportation System: An Internet of Things and Big Data Approach

Big data analytics are widely used in many areas such as efficient designing and planning of smart transportation, smart control systems, smart cities, smart communities, and more. However, analyzing big data for smart control systems has many challenges and issues using conventional engineering techniques. These challenges include processing big data in real time, fast processing, and efficient decision and management. In this article, we design a model for analyzing transportation data with Hadoop along with Spark to handle real-time transportation data. The system is further divided into four layers: data collection and acquisition, network, data processing, and application. Each layer is designed in a way to process and manage data in a well-organized format. Similarly, the data is tested through Hadoop and Spark in the data processing layer. The data is disseminated to a smart community citizen using the proposed event and decision mechanism based on named data networking. The proposed system is tested for transportation datasets from various authentic sources. The results show processing of data and real-time dissemination with citizens in less possible time. The Hadoop ecosystem along with Spark generate highly accurate results. Further, the significance of the proposed architecture is that it can be used in generic vehicular network scenarios.

Integrating Artificial and Human Intelligence: A Partnership for Responsible Innovation in Biomedical Engineering and Medicine.

Historically, the term "artificial intelligence" dates to 1956 when it was first used in a conference at Dartmouth College in the US. Since then, the development of artificial intelligence has in part been shaped by the field of neuroscience. By understanding the human brain, scientists have attempted to build new intelligent machines capable of performing complex tasks akin to humans. Indeed, future research into artificial intelligence will continue to benefit from the study of the human brain. While the development of artificial intelligence algorithms has been fast paced, the actual use of most artificial intelligence (AI) algorithms in biomedical engineering and clinical practice is still markedly below its conceivably broader potentials. This is partly because for any algorithm to be incorporated into existing workflows it has to stand the test of scientific validation, clinical and personal utility, application context, and is equitable as well. In this context, there is much to be gained by combining AI and human intelligence (HI). Harnessing Big Data, computing power and storage capacities, and addressing societal issues emergent from algorithm applications, demand deploying HI in tandem with AI. Very few countries, even economically developed states, lack adequate and critical governance frames to best understand and steer the AI innovation trajectories in health care. Drug discovery and translational pharmaceutical research stand to gain from AI technology provided they are also informed by HI. In this expert review, we analyze the ways in which AI applications are likely to traverse the continuum of life from birth to death, and encompassing not only humans but also all animal, plant, and other living organisms that are increasingly touched by AI. Examples of AI applications include digital health, diagnosis of diseases in newborns, remote monitoring of health by smart devices, real-time Big Data analytics for prompt diagnosis of heart attacks, and facial analysis software with consequences on civil liberties. While we underscore the need for integration of AI and HI, we note that AI technology does not have to replace medical specialists or scientists and rather, is in need of such expert HI. Altogether, AI and HI offer synergy for responsible innovation and veritable prospects for improving health care from prevention to diagnosis to therapeutics while unintended consequences of automation emergent from AI and algorithms should be borne in mind on scientific cultures, work force, and society at large.