Big Data Streams Research Articles

Applying Apache Spark on Streaming Big Data for Health Status Prediction

Big data applications in healthcare have provided a variety of solutions to reduce costs, errors, and waste. This work aims to develop a real-time system based on big medical data processing in the cloud for the prediction of health issues. In the proposed scalable system, medical parameters are sent to Apache Spark to extract attributes from data and apply the proposed machine learning algorithm. In this way, healthcare risks can be predicted and sent as alerts and recommendations to users and healthcare providers. The proposed work also aims to provide an effective recommendation system by using streaming medical data, historical data on a user’s profile, and a knowledge database to make the most appropriate real-time recommendations and alerts based on the sensor’s measurements. This proposed scalable system works by tweeting the health status attributes of users. Their cloud profile receives the streaming healthcare data in real time by extracting the health attributes via a machine learning prediction algorithm to predict the users’ health status. Subsequently, their status can be sent on demand to healthcare providers. Therefore, machine learning algorithms can be applied to stream health care data from wearables and provide users with insights into their health status. These algorithms can help healthcare providers and individuals focus on health risks and health status changes and consequently improve the quality of life.

A Review on Big Data Stream Processing Applications: Contributions, Benefits, and Limitations

The amount of data in our world has been rapidly keep growing from time to time. Â In the era of big data, the efficient processing and analysis of big data using machine learning algorithm is highly required, especially when the data comes in form of streams. There is no doubt that big data has become an important source of information and knowledge in making decision process. Nevertheless, dealing with this kind of data comes with great difficulties; thus, several techniques have been used in analyzing the data in the form of streams. Many techniques have been proposed and studied to handle big data and give decisions based on off-line batch analysis. Today, we need to make a constructive decision based on online streaming data analysis. Many researchers in recent years proposed some different kind of frameworks for processing the big data streaming. In this work, we explore and present in detail some of the recent achievements in big data streaming in term of contributions, benefits, and limitations. As well as some of recent platforms suitable to be used for big data streaming analytics. Moreover, we also highlight several issues that will be faced in big data stream processing. In conclusion, it is hoped that this study will assist the researchers in choosing the best and suitable framework for big data streaming projects.

The Metamorphosis (of RAM3S)

The real-time analysis of Big Data streams is a terrific resource for transforming data into value. For this, Big Data technologies for smart processing of massive data streams are available, but the facilities they offer are often too raw to be effectively exploited by analysts. RAM3S (Real-time Analysis of Massive MultiMedia Streams) is a framework that acts as a middleware software layer between multimedia stream analysis techniques and Big Data streaming platforms, so as to facilitate the implementation of the former on top of the latter. RAM3S has been proven helpful in simplifying the deployment of non-parallel techniques to streaming platforms, such as Apache Storm or Apache Flink. In this paper, we show how RAM3S has been updated to incorporate novel stream processing platforms, such as Apache Samza, and to be able to communicate with different message brokers, such as Apache Kafka. Abstracting from the message broker also provides us with the ability to pipeline several RAM3S instances that can, therefore, perform different processing tasks. This represents a richer model for stream analysis with respect to the one already available in the original RAM3S version. The generality of this new RAM3S version is demonstrated through experiments conducted on three different multimedia applications, proving that RAM3S is a formidable asset for enabling efficient and effective Data Mining and Machine Learning on multimedia data streams.

Image based statistical process monitoring via partial first order stochastic dominance

The continuously evolving digitalized manufacturing industry is pushing quality engineers to face new and complex challenges. Quality data formats are evolving from simple univariate or multivariate characteristics to big data streams consisting of sequences of images and videos in the visible or infrared range; manufacturing processes are moving from series production to more and more customized applications. In this framework, novel methods are needed to monitor and keep under statistical control the process. This study presents two novel process monitoring techniques that rely on the partial first order stochastic dominance (PFOSD) concept, applicable to in-line analysis of video image data aiming at signaling out-of-control process states. Being non-parametric, they allow dealing with complex underlying dynamics and wildly varying distributions that represent the natural process conditions. A motivating case study in metal additive manufacturing is presented, where the proposed methodology enables the in-line and in-situ detection of anomalous patterns in thermal videos captured during the production of zinc samples. Performances are investigated and compared in the presence of both simulated and real data.

117 Nutrition Modelling: What Can the Pet Field Learn (or Steal) from Recent Directions in Other Species?

Abstract Nutrition modelling has been the cornerstone of feed formulation and diet optimization in animal production systems for decades. Since the 1970s and 1980s, mechanistic models of nutrient digestion, absorption, metabolism, growth and milk/egg production have been developed and implemented to (1) amass our cumulative biological knowledge and develop theories of regulation, (2) identify knowledge gaps, and (3) propose means to manipulate nutrient dynamics in the animal. At the nutrient and metabolite level, many commonalities exist and parallels found between species. In fact, several second generation models originate from other species or research fields, and many current/existing models may be advanced by examination and consideration of models developed in other species. Many such mathematical models are implemented in practice as ‘decision support systems’ or ‘opportunity analysis tools’, in order to examine a variety of (feeding or management) scenarios for their potential outcomes, with the goal of providing targeted nutrition, improving performance, reducing cost and minimizing environmental impact. More recently, partnering artificial intelligence/machine learning modelling methodologies with newly available big data streams has ushered in a new era of possibilities for data extraction and modelling in animal systems. The niche for this type of modelling in animal production appears to be (1) pattern recognition (e.g. disease detection, activity) and (2) strong predictive/forecasting abilities (e.g. bodyweight, milk, egg production). There also appears strong potential for these two seemingly divergent modelling approaches to be integrated – for example, in precision feeding systems, or in utilizing the abundance of sensor data to better drive or develop causal-pathway based mechanistic models. This talk will broadly review trends and advances in agriculture animal species modelling, and suggest what may be borrowed, stolen or serve as inspiration to advance nutrition models in companion species.

An Adaptive Authenticated Model for Big Data Stream SAVI in SDN-Based Data Center Networks

With the rapid development of data-driven and bandwidth-intensive applications in the Software Defined Networking (SDN) northbound interface, big data stream is dynamically generated with high growth rates in SDN-based data center networks. However, a significant issue faced in big data stream communication is how to verify its authenticity in an untrusted environment. The big data stream traffic has the characteristics of security sensitivity, data size randomness, and latency sensitivity, putting high strain on the SDN-based communication system during larger spoofing events in it. In addition, the SDN controller may be overloaded under big data stream verification conditions on account of the fast increase of bandwidth-intensive applications and quick response requirements. To solve these problems, we propose a two-phase adaptive authenticated model (TAAM) by introducing source address validation implementation- (SAVI-) based IP source address verification. The model realizes real-time data stream address validation and dynamically reduces the redundant verification process. A traffic adaptive SAVI that utilizes a robust localization method followed by the Sequential Probability Ratio Test (SPRT) has been proposed to ensure differentiated executions of the big data stream packets forwarding and the spoofing packets discarding. The TAAM model could filter out the unmatched packets with better packet forwarding efficiency and fundamental security characteristics. The experimental results demonstrate that spoofing attacks under big data streams can be directly mitigated by it. Compared with the latest methods, TAAM can achieve desirable network performance in terms of transmission quality, security guarantee, and response time. It drops 97% of the spoofing attack packets while consuming only 9% of the controller CPU utilization on average.

An overview of complex data stream ensemble classification

With the rapid development of information technology, data streams in various fields are showing the characteristics of rapid arrival, complex structure and timely processing. Complex types of data streams make the classification performance worse. However, ensemble classification has become one of the main methods of processing data streams. Ensemble classification performance is better than traditional single classifiers. This article introduces the ensemble classification algorithms of complex data streams for the first time. Then overview analyzes the advantages and disadvantages of these algorithms for steady-state, concept drift, imbalanced, multi-label and multi-instance data streams. At the same time, the application fields of data streams are also introduced which summarizes the ensemble algorithms processing text, graph and big data streams. Moreover, it comprehensively summarizes the verification technology, evaluation indicators and open source platforms of complex data streams mining algorithms. Finally, the challenges and future research directions of ensemble learning algorithms dealing with uncertain, multi-type, delayed, multi-type concept drift data streams are given.

A novel mathematical modeling in shift in emotion for gauging the social influential in big data streams with hybrid sarcasm detection

SummaryOnline social network is a platform that plays an essential role in identifying the emotional values of user‐generated content such as blogs, posts, and comments along with their influential factors. Especially on Twitter, network users are growing worldwide day by day and creating a massive amount of data that is not analyzed effectively in a quick way. Identifying the most influential persons on the social network is also a challenging task over the wide range of real‐time applications like recommendation systems. Now, to handle these situations, this article proposes a novel approach for prediction of information diffusion that includes emotion recognition with sarcasm detection based influence spreader identification (PID‐ERSDISI). The proposed method uses the user‐generated posts for emotion recognition in tandem with sarcasm detection both implicitly and explicitly. This approach helps to gauge the leverage that influences spreaders and also enhances the prediction accuracy of information diffusion in a better way. The implementation of the proposed work executed their task one after another in the following way, namely, sarcasm detection, emotional‐level computation, breakpoint computation, breakpoint validation, influential user generation, and information diffusion. After the successful implementation of this proposed PID‐ERSDIS, it produced prominent results against other state‐of‐art methods.

Audiovisual speaker indexing for Web-TV automations

The current paper introduces a multimodal framework to provide Web-TV automations for live broadcasting and overall big streaming data management. The term indexing refers to the spatiotemporal localization of speakers participating in a discussion panel. Multiple modalities acting in parallel form the data-driven decision-making pipeline. The automated workflow includes the tasks of active speaker detection and localization, frame selection, and creation of a semantically annotated database. For improved performance and robustness, an information fusion model is proposed, which makes use of different audio and visual modalities. Audio-driven Voice Activity Detection follows the Enhanced Temporal Integration methodology applied on a standard audio feature set. For the localization of the dominant audio source, the argument that maximizes the General Cross-Correlation method is calculated. The visual modalities include face and mouth detection and Visual Voice Activity Detection. A Long Short Term Memory network is trained with mouth image sequences to determine voice activity. The values of the audio and visual Voice Activity Detection modules, as well as the General Cross-Correlation result, are used to train an Adaptive Neuro-Fuzzy model, which is responsible for the final decision. Experimental results prove the superiority of the information fusion approach compared to unimodal audio or visual models.

SBIoT: Scalable Broker Design for Real Time Streaming Big Data in the Internet of Things Environment

Thanks to the recent development in the technology number of IoT devices increased dramatically. Therefore, industries have been started to use IoT devices for their business processes. Many systems can be done automatically thanks to them. For this purpose, there is a server to process sensors data. Transferring these data to the server without any loss has crucial importance for the accuracy of IoT applications. Therefore, in this thesis a scalable broker for real time streaming data is proposed. Open source technologies, which are NoSql and in-memory databases, queueing, full-text index search, virtualization and container management orchestration algorithms, are used to increase efficiency of the broker. Firstly, it is planned to be used for the biggest airport in Turkey to determine the staff location. Considering the experiment analysis, proposed system is good enough to transfer data produced by devices in that airport. In addition to this, the system can adapt to device increase, which means if number of devices increasing in time, number of nodes can be increased to capture more data.

Data and Digital Solutions to Support Surveillance Strategies in the Context of the COVID-19 Pandemic.

Background: In order to prevent spread and improve control of infectious diseases, public health experts need to closely monitor human and animal populations. Infectious disease surveillance is an established, routine data collection process essential for early warning, rapid response, and disease control. The quantity of data potentially useful for early warning and surveillance has increased exponentially due to social media and other big data streams. Digital epidemiology is a novel discipline that includes harvesting, analysing, and interpreting data that were not initially collected for healthcare needs to enhance traditional surveillance. During the current COVID-19 pandemic, the importance of digital epidemiology complementing traditional public health approaches has been highlighted.Objective: The aim of this paper is to provide a comprehensive overview for the application of data and digital solutions to support surveillance strategies and draw implications for surveillance in the context of the COVID-19 pandemic and beyond.Methods: A search was conducted in PubMed databases. Articles published between January 2005 and May 2020 on the use of digital solutions to support surveillance strategies in pandemic settings and health emergencies were evaluated.Results: In this paper, we provide a comprehensive overview of digital epidemiology, available data sources, and components of 21st-century digital surveillance, early warning and response, outbreak management and control, and digital interventions.Conclusions: Our main purpose was to highlight the plausible use of new surveillance strategies, with implications for the COVID-19 pandemic strategies and then to identify opportunities and challenges for the successful development and implementation of digital solutions during non-emergency times of routine surveillance, with readiness for early-warning and response for future pandemics. The enhancement of traditional surveillance systems with novel digital surveillance methods opens a direction for the most effective framework for preparedness and response to future pandemics.

On‐Line Fault Diagnosis Model of Distribution Transformer Based on Parallel Big Data Stream and Transfer Learning

Aiming at the problem of calculation speed for distribution transformer on‐line fault diagnosis, and the shortage of on‐line monitoring type and routine test data, an on‐line fault diagnosis model of distribution transformer based on parallel big data stream and transfer learning is established. First, the fault identification feasibility based on the present on‐line monitoring data of distribution transformer is analyzed, and the main indicators of on‐line fault identification are proposed. Second, the on‐line fault identification method based on ARIMA is proposed, on this basis, the on‐line fault identification model of distribution transformer based on big data stream is established, and the distribution transformers that possess hidden fault are selected, which would be diagnosed further. In order to improve the efficiency, the model is completed on Storm platform in parallel form. Then, the distribution transformer fault diagnosis indicator system is construct. In order to achieve fault diagnosis of the above selected distribution transformers, the effective fault information from other distribution transformers is extracted using the transfer learning algorithm TrAdaBoost, which is used as auxiliary data for the fault diagnosis tool training of the distribution transformer to be diagnosed. At the same time, the model is completed on Storm platform to improve the efficiency. Finally, based on the distribution transformer fault data, the distribution transformer fault diagnosis is simulated, and results show that the fault diagnosis is accurate and efficient. © 2021 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.

Distributed Fusion of Heterogeneous Remote Sensing and Social Media Data: A Review and New Developments

Despite the wide availability of remote sensing big data from numerous different Earth Observation (EO) instruments, the limitations in the spatial and temporal resolution of such EO sensors (as well as atmospheric opacity and other kinds of interferers) have led to many situations in which using only remote sensing data cannot fully meet the requirements of applications in which a (near) real-time response is needed. Examples of these applications include floods, earthquakes, and other kinds of natural disasters, such as typhoons. To address this issue, social media data have gradually been adopted to fill possible gaps in the analysis when remote sensing data are lacking or incomplete. In this case, the fusion of heterogeneous big data streams from multiple data sources introduces significant demands from a computational viewpoint. In order to meet these challenges, distributed computing is increasingly viewed as a feasible solution to parallelize the analysis of massive data coming from different sources (e.g., remote sensing and social media data). In this article, we provide an overview of available and new distributed strategies to address the computational challenges brought by massive heterogeneous data processing and fusion for real-time environmental monitoring and decision-making. The 2013 Boulder (Colorado) flood event is taken as a case study to evaluate several new distributed data fusion frameworks. Experimental results demonstrate that the proposed distributed frameworks are suitable in terms of response time and computational requirements for fusing large-volume heterogeneous data sources.

A Cyber-Physical System and Graph-Based Approach for Transportation Management in Smart Cities

In the last decade, technological advancements in the cyber-physical system have set the basis for real-time and context-aware services to ease human lives. The citizens, especially travelers, want to experience a safe, healthy, and timely journey to their destination. Smart and on-ground real-time traffic analysis helps authorities further improve decision-making to ensure safe and convenient traveling. In this paper, we proposed a transport-control model that exploits cyber-physical systems (CPS) and sensor-technology to continuously monitor and mine the big city data for smart decision-making. The system makes use of travel-time, traffic intensity, vehicle’s speed, and current road conditions to construct a weighted city graph representing the road network. Traditional graph algorithms with efficient implementation technologies are employed to respond to commuters’ and authorities’ needs in order to achieve a smart and optimum transportation system. To efficiently process the incoming big data streams, the proposed architecture uses the Apache GraphX tool with several parallel processing nodes, along with Spark and Hadoop that ultimately provide better performance against various state-of-the-art solutions. The system is thoroughly evaluated in terms of system throughput and processing time, revealing that the proposed system is efficient, robust, and scalable.

Measuring the Effectiveness of Adaptive Random Forest for Handling Concept Drift in Big Data Streams.

We are living in the age of big data, a majority of which is stream data. The real-time processing of this data requires careful consideration from different perspectives. Concept drift is a change in the data’s underlying distribution, a significant issue, especially when learning from data streams. It requires learners to be adaptive to dynamic changes. Random forest is an ensemble approach that is widely used in classical non-streaming settings of machine learning applications. At the same time, the Adaptive Random Forest (ARF) is a stream learning algorithm that showed promising results in terms of its accuracy and ability to deal with various types of drift. The incoming instances’ continuity allows for their binomial distribution to be approximated to a Poisson distribution. In this study, we propose a mechanism to increase such streaming algorithms’ efficiency by focusing on resampling. Our measure, resampling effectiveness (), fuses the two most essential aspects in online learning; accuracy and execution time. We use six different synthetic data sets, each having a different type of drift, to empirically select the parameter of the Poisson distribution that yields the best value for . By comparing the standard ARF with its tuned variations, we show that ARF performance can be enhanced by tackling this important aspect. Finally, we present three case studies from different contexts to test our proposed enhancement method and demonstrate its effectiveness in processing large data sets: (a) Amazon customer reviews (written in English), (b) hotel reviews (in Arabic), and (c) real-time aspect-based sentiment analysis of COVID-19-related tweets in the United States during April 2020. Results indicate that our proposed method of enhancement exhibited considerable improvement in most of the situations.

Robust Subspace Tracking Algorithms in Signal Processing: A Brief Survey

Principal component analysis (PCA) and subspace estimation (SE) are popular data analysis tools and used in a wide range of applications. The main interest in PCA/SE is for dimensionality reduction and low-rank approximation purposes. The emergence of big data streams have led to several essential issues for performing PCA/SE. Among them are (i) the size of such data streams increases over time, (ii) the underlying models may be time-dependent, and (iii) problem of dealing with the uncertainty and incompleteness in data. A robust variant of PCA/SE for such data streams, namely robust online PCA or robust subspace tracking (RST), has been introduced as a good alternative. The main goal of this paper is to provide a brief survey on recent RST algorithms in signal processing. Particularly, we begin this survey by introducing the basic ideas of the RST problem. Then, different aspects of RST are reviewed with respect to different kinds of non-Gaussian noises and sparse constraints. Our own contributions on this topic are also highlighted.

Online one pass clustering of data streams based on growing neural gas and fuzzy inference systems

AbstractThe clustering of big data streams has become a challenging task due to time and space constraints of the hardware and decreasing accuracy when the dimensionality of input data grows in time. In this paper, fuzzy growing neural gas is introduced, an online fuzzy approach for clustering data streams based on the growing neural gas algorithm, by adopting more restrictive criteria for selecting the winner nodes in the topological graph constructed at each iteration of the algorithm. The algorithm is tested on public datasets, and the results show improvements over existing clustering methods.

An edge-cloud-aided incremental tensor-based fuzzy c-means approach with big data fusion for exploring smart data

Recently, smart data has attracted great attention in the smart city community since it can provide valuable information to support intelligent services such as planning, monitoring, and decision making. However, it imposes a big challenge to explore smart data from big data gathered from smart city with various advanced fusion and analysis approaches. This paper proposes an incremental tensor-based fuzzy c-means approach (IT-FCM) for obtaining smart data from continuously generated big data. Specifically, a weighted version of the tensor-based fuzzy c-means approach (T-FCM) is firstly proposed to cluster the dataset that combines the previous cluster centroids and the new generated data. Aiming to improve the clustering efficiency, the old data objects are represented by the centroids to avoid repeat clustering. Furthermore, this paper presents an edge-cloud-aided clustering scheme to fuse big data from different sources and perspectives and further to implement co-clustering on the fused datasets for exploring smart data. Finally, the proposed IT-FCM approach is evaluated by comparing with T-FCM regarding clustering accuracy and efficiency on two different datasets in the experiments. The results state that IT-FCM outperforms T-FCM in clustering streaming big data in terms of accuracy and efficiency for obtaining smart data.

An Incremental Tensor-Train Decomposition for Cyber-Physical-Social Big Data

Cyber-physical-social big data generated from ubiquitous devices and diverse spaces generally are multi-source, heterogeneous, and deeply intertwined. To efficiently analyze and handle the ubiquitous cyber-physical-social big data, tensor is considered as an effective tool, but the curse of dimensionality is still the main bottleneck of tensor-based big data analysis. Tensor networks can considerably alleviate or overcome it through the tensor approximate theory. Therefore, this paper focuses on developing an efficient big data processing framework based on tensor networks and providing an incremental tensor train decomposition approach for the streaming big data. Concretely, this paper first presents a hierarchical cyber-physical-social big data processing framework composed of three planes, namely, data representation and decomposition, data storage and processing, and data analysis and service, in which tensor train (TT) and quantized TT decompositions are particularly introduced to remarkably overcome the curse of dimensionality. Besides, to efficiently handle the continuous streaming big data and avoid the repeated decomposition for the history data, an incremental tensor train decomposition (ITTD) approach is proposed and the complexities are further analyzed in detail. Experimental results demonstrate that ITTD demonstrably outperforms the nonincremental TT decomposition in execution time on the precise of guaranteeing the nearly equal approximation error.

The Use of Smart Home Speakers by the Elderly: Exploratory Analyses and Potential for Big Data

Contemporary smart home speakers (Amazon Echo™, Google Nest™) represent an opportunity to capture streams of big data that represent user interactions. Once captured, these data streams may be analyzed to understand use patterns for different demographics. In this research, we focus on one demographic, the aging-at-home elderly. We describe how such data streams may be captured with appropriate safeguards from smart home speakers deployed in the homes of the elderly individuals. We analyze the data to develop visual representations and sonifications to discover several patterns about aggregate use, use intensity, use progression over time, and types of use. Based on these, we draw tentative conclusions about the aging-at-home elderly make use of smart speakers at home, identify some problems, and describe how the empirical findings may be used to discover new voice skills that can contribute to the quality of life for the elderly. The paper concludes with a discussion of possibilities for applying the approach we have demonstrated for larger data sets.