Big Data Streams Research Articles

A review on sentiment discovery and analysis of educational big‐data

AbstractSentiment discovery and analysis (SDA) aims to automatically identify the underlying attitudes, sentiments, and subjectivity towards a certain entity such as learners and learning resources. Due to its enormous potential for smart education, SDA has been deemed as a powerful technique for identifying and classifying sentiments from multimodal and multisource data over the whole process of education. For big educational data streams, SDA faces challenges in unimodal feature selection, sentiment classification, and multimodal fusion. As such, a large body of studies in the literature explores diverse approaches to SDA for educational applications. This paper provides a self‐contained, uniform overview of the SDA techniques for education. In particular, we focus on prominent studies in unimodal sentiment features and classifications (e.g., text, audio, and visual). In addition, we present a novel SDA framework of multimodal fusions, together with description of their crucial components. Based on this framework, we review different approaches to SDA on education from the perspectives of approaches and applications. After comprehensively reviewing the SDA techniques on education, we present the trends and prospectives of the future SDA research under ubiquitous education contexts.This article is categorized under: Application Areas > Education and Learning

Tree stream mining algorithm with Chernoff-bound and standard deviation approach for big data stream

We propose a Chernoff-bound approach and examine standard deviation value to enhance the accuracy of the existing fast incremental model tree with the drift detection (FIMT-DD) algorithm. It is a data stream mining algorithm that can observe and form a model tree from a large dataset. The standard FIMT-DD algorithm uses the Hoeffding bound for its splitting criterion. The use of the simplified Chernoff bound is proposed for splitting a tree more accurately than the Hoeffding bound. We verify our proposed Chernoff bound and standard deviation value examination by evaluating several large real-world datasets that consists of 100,000,000 instances. The result shows that our proposed method has an improvement in term of accuracy compared to the Hoeffding bound in the FIMT-DD algorithm. Error value measurements demonstrate that the Chernoff bound approach contributes to smaller error value compared to the standard method. In term of overall simulation time, the Chernoff bound approach increases the duration of simulation time and utilizes more memory compared to Hoeffding bound. However, Chernoff bound is much faster to get a smaller error compared to the standard algorithm.

Transformation-Based Streaming Workflow Allocation on Geo-Distributed Datacenters for Streaming Big Data Processing

The cost-minimization problem for streaming workflow (SW) has already become increasingly important and even critical in stream big data processing, particularly for geographically distributed datacenters, because of its huge demand on computing and communicating resources. Existing virtual machine (VM) allocation algorithms in cloud computing have been widely applied to batch-processing models; however, none of them can be successfully applied to SW because: 1) they failed to adapt the continuous execution characteristic of SW; and 2) most of them are all based on the assumption that the price of traffic and VMs among datacenters are uniform. In this paper, we propose a transformation-based SW allocation algorithm with the goal of cost-minimization for stream big data processing in geographically distributed datacenters, considering the characteristics of SW and price heterogeneity among geographically distributed datacenters. We first propose a cost-aware workflow transformation framework based on eight well-designed and verified transformation rules for cost reduction to adapt the continuous execution characteristic of SW. We then formulate the joint VM-traffic optimization problem and show that it is NP-hard. To produce the optimal solution in polynomial time, we then transform the SW allocation problem into the minimum-cost maximum-flow problem, considering both traffic and VMs price heterogeneity. Finally, our experimental results validate the high cost efficiency of our approach with lower computing and communicating costs by optimizing the workflow specification and joint VM-traffic cost optimization.

A Semantic Approach for Outlier Detection in Big Data Streams

In recent years, the world faced a big revolution in data generation and collection technologies. The volume, velocity and veracity of data have changed drastically and led to new types of challenges related to data analysis, modeling and prediction. One of the key challenges is related to the semantic analysis of textual data especially in big data streams settings. The existing solutions focus on either topic analysis or the sentiment analysis. Moreover, the semantic outlier detection over data streams as one of the key problems in data mining and data analysis fields has less focus. In this paper, we introduce a new concept of semantic outlier through which the topic of the textual data is considered as the primary content of the data stream while the sentiment is considered as the context in which the data has been generated and affected. Also, we propose a framework for semantic outlier detection in big data streams which incorporates the contextual detection concepts. The advantage of the proposed concept is that it incorporates both topic and sentiment analysis into one single process; while at the same time the framework enables the implementation of different algorithms and approaches for semantic analysis.

A study of IoT malware activities using association rule learning for darknet sensor data

Along with the proliferation of Internet of Things (IoT) devices, cyberattacks towards these devices are on the rise. In this paper, we present a study on applying Association Rule Learning to discover the regularities of these attacks from the big stream data collected on a large-scale darknet. By exploring the regularities in IoT-related indicators such as destination ports, type of service, and TCP window sizes, we succeeded in discovering the activities of attacking hosts associated with well-known classes of malware programs. As a case study, we report an interesting observation of the attack campaigns before and after the first source code release of the well-known IoT malware Mirai. The experiments show that the proposed scheme is effective and efficient in early detection and tracking of activities of new malware on the Internet and hence induces a promising approach to automate and accelerate the identification and mitigation of new cyber threats.

Auto-scaling for real-time stream analytics on HPC cloud

There are very-high-volume streaming data in the cyber world today. With the popularization of 5G technology, the streaming Big Data grows larger. Moreover, it needs to be analyzed in real time. We propose a new strategy HPC2-ARS to enable streaming services on HPC platforms. This strategy includes a three-tier high-performance cloud computing (HPC2) platform and a novel autonomous resource-scheduling (ARS) framework. The HPC2 platform is our de facto base platform for research on streaming service. It has three components: Tianhe-2 high-performance computer, custom OpenStack cloud computing software, and Apache Storm stream data analytic system. Our ARS framework ensures real-time response on unpredictable and fluctuating stream, especially streaming Big Data in the 5G era. This strategy addresses an essential problem in the convergence of HPC Cloud, Big Data, and streaming service. Specifically, Our ARS framework provides theoretical and practical solutions for resource provisioning, placement, and scheduling optimization. Extensive experiments have validated the effectiveness of the proposed strategy.

Is health-care data the new blood?

Is health-care data the new blood?

Crowdbreaks: Tracking Health Trends Using Public Social Media Data and Crowdsourcing.

In the past decade, tracking health trends using social media data has shown great promise, due to a powerful combination of massive adoption of social media around the world, and increasingly potent hardware and software that enables us to work with these new big data streams. At the same time, many challenging problems have been identified. First, there is often a mismatch between how rapidly online data can change, and how rapidly algorithms are updated, which means that there is limited reusability for algorithms trained on past data as their performance decreases over time. Second, much of the work is focusing on specific issues during a specific past period in time, even though public health institutions would need flexible tools to assess multiple evolving situations in real time. Third, most tools providing such capabilities are proprietary systems with little algorithmic or data transparency, and thus little buy-in from the global public health and research community. Here, we introduce Crowdbreaks, an open platform which allows tracking of health trends by making use of continuous crowdsourced labeling of public social media content. The system is built in a way which automatizes the typical workflow from data collection, filtering, labeling and training of machine learning classifiers and therefore can greatly accelerate the research process in the public health domain. This work describes the technical aspects of the platform, thereby covering the functionalities at its current state and exploring its future use cases and extensions.

PAS3-HSID: a Dynamic Bio-Inspired Approach for Real-Time Hot Spot Identification in Data Streams

Hot spot identification is a very relevant problem in a wide variety of areas such as health care, energy or transportation. A hot spot is defined as a region of high likelihood of occurrence of a particular event. To identify hot spots, location data for those events is required, which is typically collected by telematics devices. These sensors are constantly gathering information, generating very large volumes of data. Current state-of-the-art solutions are capable of identifying hot spots from big static batches of data by means of variations of clustering or instance selection techniques that pre-process the original input data, providing the most relevant locations. However, these approaches neglect to address changes in hot spots over time. This paper presents a dynamic bio-inspired approach to detect hot spots in big data streams. This computational intelligence method is designed and applied to the transportation sector as a case study to identify incidents in the roads caused by heavy goods vehicles. We adapt an immune-based algorithm to account for the temporary aspect of hot spots inspired by the idea of pheromones, which is then subsequently implemented using Apache Spark Streaming. Experimental results on real datasets with up to 4.5 million data points—provided by a telematics company—show that the algorithm is capable of quickly processing large streaming batches of data, as well as successfully adapting over time to detect hot spots. The outcome of this method is twofold, both reducing data storage requirements and demonstrating resilience to sudden changes in the input data (concept drift).

Lattice-Modeled Information Flow Control of Big Sensing Data Streams for Smart Health Application

Internet of Things (IoT) provides a promising opportunity to build powerful data analytics systems with real time event detection for smart health, and therefore wearable IoT has become a rising source of big data streams for smart health, for which security needs to be assured by detecting real-time event to avoid malicious activities, and meanwhile to control the information leakage of big sensing data streams. I refer to this as an information flow control problem. To address this problem, this paper proposes a static lattice model for information flow control over big sensing data streams. I initialize two static lattices, i.e., sensor lattice for wearable sensors and user lattice for users, and then static lattices aim to process the flow control model faster, because I am dealing with high volume and velocity of data streams. The experimental evaluation and results of the information flow model show that it can excellently handle the incoming big data streams with low latency and buffer requirement.

Fuzzy-Folded Bloom Filter-as-a-Service for Big Data Storage in the Cloud

With the ongoing trend of smart and Internet-connected objects being deployed across a broad range of applications, there is also a corresponding increase in the amount of data movement across different geographical regions. This, in turn, poses a number of challenges with respect to big data storage across multiple locations, including cloud computing platform. For example, the underlying distributed file system has a large number of directories and files in the form of gigantic trees, which are difficult to parse in polynomial time. Moreover, with the exponential increase of big data streams (i.e., unbounded sets of continuous data flows), challenges associated with indexing and membership queries are compounded. The capability to process such significant amount of data with high accuracy can have significant impact on decision-making and formulation of business and risk-related strategies, particularly in our current Industrial Internet of Things environment (IIoT). However, existing storage solutions are deterministic in nature. In other words, they tend to consume considerable memory and CPU time to yield accurate results. This necessitates the design of efficient quality of service-aware IIoT applications that are able to deal with the challenges of data storage and retrieval in the cloud computing environment. In this paper, we present an effective space-effective strategy for massive data storage using bloom filter (BF). Specifically, in the proposed scheme, the standard BF is extended to incorporate fuzzy-enabled folding approach, hereafter referred to as fuzzy folded BF (FFBF). In FFBF, fuzzy operations are used to accommodate the hashed data of one BF into another to reduce storage requirements. Evaluations on UCI ML AReM and Facebook datasets demonstrate the efficacy of FFBF, in terms of dealing with approximately 1.9 times more data as compared to using the standard BF. This is also achieved without affecting the false positive rate and query time.

Nearest Neighbor Classification for High-Speed Big Data Streams using Spark

Nearest Neighbor Classification for High-Speed Big Data Streams using Spark

Adaptive network diagram constructions for representing big data event streams on monitoring dashboards

Critical systems that produce big data streams can require human operators to monitor these event streams for changes of interest. Automated systems which oversee many tasks can still have a need for the ‘human-in-the-loop’ operator to evaluate whether an intervention is required due to a lack of suitable training data initially offered to the system which would allow a correct course of actions to be taken. In order for an operator to be capable of reacting to real-time events, the visual depiction of the event data must be in a form which captures essential associations and is readily understood by visual inspection. A similar requirement can be found during inspections on activity protocols in a large organization where a code of correct conduct is prescribed and there is a need to oversee whether the activity traces match the expectations, with minimal delay. The methodology presented here addresses these concerns by providing an adaptive window sizing measurement for subsetting the data, and subsequently produces a set of network diagrams based upon event label co-occurrence networks. With an intuitive method of network construction the amount of time required for operators to learn how to monitor complex event streams of big datasets can be reduced.

A review on big data real-time stream processing and its scheduling techniques

Over the last decade, several interconnected disruptions have happened in the large scale distributed and parallel computing landscape. The volume of data currently produced by various activities of the society has never been so big and is generated at an increasing speed. Data that is received in real-time can become way too valuable at the time it arrives and supports valuable decision making. Systems for managing data streams is not a recently developed concept but its becoming more important due to the multiplication of data stream sources in the context of IoT. This paper refers to the unique processing challenges posed by the nature of streams, and the related mechanisms used to face them in the big data era. Several cloud systems emerged to enable distributed processing of streams of big data. Distributed stream management systems (DSMS) along with their strengths and limitations are presented and compared. Computations in these systems demand elaborate orchestration over a collection of machines. Consequently, a classification and literature review on these systems' scheduling techniques and their enhancements is also provided.

Fractal methods in intelligent technologies for processing large data streams

In this work a new approach to the construction of models and logic circuits of algorithms and procedures for information technology processing and analysis of big data streams within the fractal paradigm is describes. In this case, data streams are defined as information objects whose physical nature can be arbitrary. The information object is investigated apart from any model or scheme, the logical scheme of intellectual technology is built in the form of: facts, regularities and reality. Fractal methods form the framework of the logical, algorithmic and content essence of the approach. The basic premise of the approach is as follows. First, the processing and analysis of data stream to determine whether it forms a fractal structure and construct a phase portrait of data stream as an information object. Second, to distinguish the areas of fractal percolation and aggregation in the multifractal structure of the stream, the phase portrait is used. Third, to estimate the spatial and temporal scales of fractal percolation and aggregation processes.

Efficient resource scheduling for the analysis of Big Data streams

Efficient resource scheduling for the analysis of Big Data streams

On-line prediction remaining useful life for ball bearings via grey NARX

The Huge vibration data are generated continuously by many sensors in daily high-speed rotating machinery operations. Accurate online prediction based on big vibration data streaming can reduce the risks related to failures and avoid service disruptions. This paper presents a hybrid nonlinear autoregressive network with exogenous inputs (NARX) model to forecast the remaining useful life of ball bearings through health index based on big vibration data streaming. This approach is validated by a real data from PRONOSTIA experimentation platform and industrial test rig compared with backpropagation neural network (BP), Elman and general regression neural network (GRNN) prediction model. Root mean square error, mean absolute error and correlation coefficient were used as performance indexes to evaluate the prediction accuracy of these models. The mean absolute error, the root mean square error and the correlation coefficient of hybrid NARX model evaluation index are 2.04, 2.85 and 0.98 respectively. It shows that the model presented in this paper has higher prediction accuracy. It can meet the needs of actual ball bearing remaining useful life prediction and also provide reference in other fields.

Efficient deadline-aware scheduling for the analysis of Big Data streams in public Cloud

The emergence of Big Data has had a profound impact on how data are analyzed. Open source distributed stream processing platforms have gained popularity for analyzing streaming Big Data as they provide low latency required for streaming Big Data applications using Cloud resources. However, existing resource schedulers are still lacking the efficiency and deadline meeting that Big Data analytical applications require. Recent works have already considered streaming Big Data characteristics to improve the efficiency and the likelihood of deadline meeting for scheduling in the platforms. Nevertheless, they have not taken into account the specific attributes of analytical application, public Cloud utilization cost and delays caused by performance degradation of leasing public Cloud resources. This study, therefore, presents BCframework, an efficient deadline-aware scheduling framework used by streaming Big Data analysis applications based on public Cloud resources. BCframework proposes a scheduling model which considers public Cloud utilization cost, performance variation, deadline meeting and latency reduction requirements of streaming Big Data analytical applications. Furthermore, it introduces two operator scheduling algorithms based on both a novel partitioning algorithm and an operator replication method. BCframework is highly adaptable to the fluctuation of streaming Big Data and the performance degradation of public Cloud resources. Experiments with the benchmark and real-world queries show that BCframework can significantly reduce the latency and utilization cost and also minimize deadline violations and provisioned virtual machine instances.

Optimizing the confidence bound of count-min sketches to estimate the streaming big data query results more precisely

A count-min sketch is a probabilistic data structure, which serves as a frequency table of events to process a stream of big data. It uses hash functions to map events to frequencies. Querying a count-min sketch returns the targeted event along with an estimated frequency, which is not less than the actual frequency. The estimated error, i.e., the difference between the estimated frequency and the actual, can be measured by a pre-defined confidence bound. However, the bound originally defined is too loose. The reason is that the Markov inequality used to derive the bound does not perform well. In this paper, based on binomial distribution and central limit theorem, we define a tighter bound. We indicate that the reliability of the bound is related to the deviation of data, which can be measured by the data’s coefficient of standard deviation. Our extensive experiments well support the effectiveness and efficiency of the new bound.

Outlier detection techniques for big data streams: focus on cyber security

In recent years, detecting outliers in big data streams has become a main challenge in several domains (e.g., medical monitoring, government security, information security, natural disasters, and online financial frauds). In fact, unlike regular static data, streams raise many issues like high multidimensionality, dynamic data distribution, unpredictable relationships, data sequences, uncertainty and transiency. Most of the proposed approaches can handle some of these issues but not all. In addition, they provide limited considerations with regard to scalability and performance. Real-world applications require high performance, resources optimisation and real-time responsiveness when detecting outliers. This is useful to extract knowledge, detect incidents and predict patterns changes. In this paper, we review and compare recent studies in detecting outliers for data streams. We investigate how researchers improved the outcome of different models and monitoring systems, especially in the context of cyber security.