Big Streams Research Articles

THORIN: an Efficient Module for Federated Access and Threat Mitigation in Big Stream Cloud Architectures

In order to make cloud services attractive for several IT organizations, it is necessary to provide access control and to implement safe and reliable mechanisms of Identity and Access Management (IAM). In this work, we focus on security issues and challenges in the design and implementation of cloud architectures and, in particular, for the management of Big Stream applications in Internet of Things (IoT) scenarios. The proposed work introduces a new set of modules allowing a federated access control policy for cloud users. An analysis of possible threats and attacks against the proposed Big Stream platform is presented, investigating the system performance in terms of detection and elimination of malicious nodes. In particular, we propose a new module, denoted as Traffic Handler Orchestrator & Rapid Intervention (THORIN), which is very efficient in counteracting botnet-based threats.

A Dynamic Drilling Sampling Method and Evaluation Model for Big Streaming Data

The big data sampling method for real-time and high-speed streaming data is prone to lose the value and information of a large amount of discrete data, and it is not easy to make an efficient and accurate evaluation of the value characteristics of streaming data. The SDSLA sampling method based on mineral drilling exploration can evaluate the valuable information of streaming data containing many discrete data in real-time, but when the range of discrete data is irregular, it has low sampling accuracy for discrete data. Based on the SDSLA algorithm, we propose a dynamic drilling sampling method SDDS, which takes well as the analysis unit, dynamically changes the size and position of the well, and accurately locates the position and range of discrete data. A new model SDVEM is further proposed for data valuation, which evaluates the sample set from discrete, centralized, and overall dimensions. Experiments show that compared with the SDSLA algorithm, the sample sampled by the SDDS algorithm has higher evaluation accuracy, and the probability distribution of the sample is closer to the original streaming data, with the AOCV indicator being nearly 10% higher. In addition, the SDDS algorithm can achieve over 90% accuracy, recall, and F1 score for training and testing neural networks with small sampling rates, all of which are higher than the SDSLA algorithm. In summary, the SDDS algorithm not only accurately evaluates the value characteristics of streaming data but also facilitates the training of neural network models, which has important research significance in big data estimation.

Cuckoo Search Augmented MapReduce for Predictive Scheduling With Big Stream Data

Handling an information stream is a basic report for streaming application. There were numerous strategies which help during Bigdata streaming, however it can't deal with the tremendous information. To advance the productivity with least time intricacy, a Cuckoo Search Augmented Map Reduce for Predictive Scheduling (CSA-MRPS) system is presented. This technique incorporates cycles in preprocessing and prescient booking for stream information examination. In preprocessing, nonstop information streams are discretized utilizing Khiops and it begins from the constant time spans, consolidates the closest time as indicated by the Chi-square worth with lesser time intricacy. MapReduce work is applied to discretized information for prescient investigation utilizing Multi-Objective Ranked Cuckoo Search Optimization (MRCSA). It characterize the target capacities for the handling units, for example, CPU time, memory utilization, transfer speed use and energy utilization. Thus, CSA-MRPS Mechanism predicts the asset upgraded preparing unit with high position through the planning system.

Machine Anthropology: A View from International Relations

International relations are made up of thick layers of meaning and big streams of data. How can we capture the nuances and scales of increasingly digitalised world politics, taking advantage of the possibilities that come with ‘big data’ and ‘digital methods’ in our discipline of International Relations (IR)? What is needed, we argue, is a methodological twin-move of making big data thick and thick data big. Taking diplomacy, one of IR's core practices as our case, we illustrate how anthropological and computational approaches can be merged in IR research. We report from our experiences with the project DIPLOFACE: Diplomatic Face-Work between Confidential Negotiations and Public Display, investigating how digital communication technologies influence both the study and conduct of age-old and traditionally analogue practices of inter-state diplomacy.

Religious Moderatism versus Conservatism in Indonesia: Controversy of ‘Islam Nusantara’ during Joko Widodo’s Administration

The current trend of social and political fracture among Indonesian people has permeated many aspects of life, including religious domain. While moderate Muslims are still majority, the voice of conservative and also element of radical Muslim groups has shown a great militancy in campaigning for their agenda. Rivalry among those groups in the public discourse is unavoidable. This study examines the contested idea of ‘Islam Nusantara’ in the effort to campaign for moderate and friendly Islam in the early of Joko Widodo’s administration. Since conservative and radical factions have rejected the idea of Pancasila state, accusing of other Muslims of being kafir (apostasy) and the spirit to eliminate local cultures, their counterparts use different agenda and approach. The moderate Muslims use the concept of Islam Nusantara to campaign for the concept of Islam and nationalism, respect for diversity and cultural approaches. This study reveals the different point of view of Muslim groups regarding the concept and practice of Islam Nusantara which has become a controversy since the Nahdlatul Ulama (NU) used it as the main theme in the 33rd congress. Although ‘Islam Nusantara’ is not a new term in the discourse of Indonesian Islam, the contestation and consensus have produced a new fragmentation of religious opinions among Indonesian Muslims. This research aims to explore the sociology of knowledge among Indonesian Muslims whose inclination has been devided into two big streams of moderate and conservative wing.

Using pattern recognition to characterise heating behaviour in residential buildings

ABSTRACT The understanding of energy-related occupant behaviour and its better reproduction in building energy analysis has recently become a primary field of interest. The combination of computing ease and the availability of big streams of high resolution building performance data enhance the ability to study this behaviour. In this context, data mining methods are increasingly employed. The aim of this study is to propose a data mining methodology for the characterization of heating behaviour in residential buildings. The methodology consists of two multivariate statistical analysis methods, namely Principal Component Analysis (PCA) followed by cluster analysis. The methods were applied on monitored gas consumption data of five dwellings in Italy. Findings support literature indicating that people heat their homes in different ways. It was found that households do not always follow a different heating schedule on weekends, have very different temperature preferences and operate the heating system at different hours during the day. In fact, some households may change heating practices over the heating season. The highlight of the proposed methodology is the insightful and simple way that PCA can extract succinct information about the heating behaviour of the user.

D2IA: User-defined interval analytics on distributed streams

Nowadays, modern Big Stream Processing Solutions (e.g. Spark, Flink) are working towards being the ultimate framework for streaming analytics. In order to achieve this goal, they started to offer extensions of SQL that incorporate stream-oriented primitives such as windowing and Complex Event Processing (CEP). The former enables stateful computation on infinite sequences of data items while the latter focuses on the detection of events pattern. In most of the cases, data items and events are considered instantaneous, i.e., they are single time points in a discrete temporal domain. Nevertheless, a point-based time semantics does not satisfy the requirements of a number of use-cases. For instance, it is not possible to detect the interval during which the temperature increases until the temperature begins to decrease, nor for all the relations this interval subsumes. To tackle this challenge, we present D2IA; a set of novel abstract operators to define analytics on user-defined event intervals based on raw events and to efficiently reason about temporal relationships between intervals and/or point events. We realize the implementation of the concepts of D2IA on top of Flink, a distributed stream processing engine for big data.

Diamond Sketch: Accurate Per-flow Measurement for Big Streaming Data

Per-flow measurement is a critical issue in computer networks, and one of its key tasks is to count the number of packets in each flow (for big streaming data). The literature has demonstrated that sketch is the most memory-efficient data structure for the counting task, and is widely used in distributed systems. Existing sketches often use many counters that are of the same size to record the number of packets in a flow, thus the counters are forced to be large enough to accommodate the size of the largest flow. Unfortunately, as most flows are small (i.e., mice flows) and only a very few flows are large (i.e., elephant flows), many counters represent very small values, which is a waste of memory. Sketches are often stored in fast but expensive memory (e.g., SRAM), thus it is critical to achieve high memory efficiency. To address this issue, we propose a novel sketch, namely the Diamond sketch. The Diamond sketch is composed of atom sketches, and each atom sketch uses small counters. The key idea of Diamond is to dynamically assign an appropriate number of atom sketches to each flow on demand, thus optimizing memory efficiency . Experimental results show that the Diamond sketch outperforms the best of the five typical sketches by up to 508.3 times in terms of relative error while keeping comparable speed. We made the source code of all the six sketches available on GitHub [1] .

Benchmarking big data systems: A survey

With the enormous growth on the availability and usage of Big Data storage and processing systems, it has become essential to assess the various performance aspects of these systems so that we can carefully understand their strong and weak aspects. In practice, currently, when an individual/enterprise aims to develop a Big Data storage and processing solution for harnessing the knowledge inside their data, they will get challenged by the availability of several frameworks from which they need to select. This is a challenging task which needs to directed by with good knowledge about various perspectives of such systems. Additionally, the choice normally vary from one scenario to another according to the essential needs of the application. In practice, there is no single benchmark study which can cover the different types of big data processing requirements, systems, application scenarios and metrics. Several benchmarks and benchmarking studies have been developed where each study focuses on some representative type of frameworks and only consider some aspects to cover. In this article, we provide a comprehensive survey and analysis of the state-of-the-art of benchmarking the different types of big data systems (e.g., NoSQL databases, Big SQL engines, Big Streaming engines, Big Graph Processing engines, Big Machine/Deep Learning engines). Additionally, we highlight some of the significant open challenges and missing requirements of current benchmarks of big data systems with suggestions of directions for future extensions and improvements.

Phish Webpage Classification Using Hybrid Algorithm of Machine Learning and Statistical Induction Ratios

Although the conventional machine learning-based anti-phishing techniques outperform their competitors in phishing detection, they are still targeted by zero-hour phish webpages due to their constraints of phishing induction. Therefore, phishing induction must be boosted up with the extraction of new features, the selection of robust subsets of decisive features, the active learning of classifiers on a big webpage stream. In this paper, we propose a hybrid feature-based classification algorithm (HFBC) for decisive phish webpage classification. HFBC hybridises two statistical criteria optimised feature occurrence (OFC) and phishing induction ratio (PIR) with the induction settings of the most salient machine learning algorithms, Naive bays and decision tree. Additionally, we propose two constituent algorithms of features extraction and features selection for holistic phish webpage characterisation. The superiority of our proposed approach is justified and proven throughout chronological, real-time, and comparative analyses against existing machines learning-based anti-phishing techniques.

Sentiment analysis on big sparse data streams with limited labels

Sentiment analysis is an important task in order to gain insights over the huge amounts of opinionated texts generated on a daily basis in social media like Twitter. Despite its huge amount, standard supervised learning methods won’t work upon such sort of data due to lack of labels and the impracticality of (human) labeling at this scale. In this work, we leverage distant supervision and semi-supervised learning to annotate a big stream of tweets from 2015 which consists of 228 million tweets without retweets (and 275 million with retweets). We present the insights from our annotation process regarding the effect of different semi-supervised learning approaches, namely Self-Learning, Co-Training and Expectation–Maximization. Moreover, we propose two annotation modes, the batch mode where all labeled and unlabeled data are available to the algorithms from the beginning and a lightweight streaming mode that processes the data in batches based on their arrival time in the stream. Our experiments show that stream processing with a sliding window of three months achieves comparable results to batch processing while being more efficient. Finally, to tackle the class imbalance problem, as our dataset is imbalanced toward the positive sentiment class, and its aggravation by the semi-supervised learning methods, we employ data augmentation in the semi-supervised learning process in order to equalize the class distribution. Our results show that semi-supervised learning coupled with data augmentation outperforms significantly the default semi-supervised annotation process. We make the so-called TSentiment15 sentiment-annotated dataset available to the community to be used for evaluation purposes and for developing new methods.

Applying big data and stream processing to the real estate domain

ABSTRACTIn this paper, we propose an architecture that combines Big Data and Stream Processing which can be applied to the Real Estate Domain. Our approach consists of a specialisation of Lambda architecture and it is inspired by some aspects of Kappa architecture. As a proof of this solution, we show a prototype developed following it and a comparison of the three architecture quality models. Finally, we highlight the differences between the proposed architecture and similar ones and draw some future lines following the present approach.

Adaptive network diagram constructions for representing big data event streams on monitoring dashboards

Critical systems that produce big data streams can require human operators to monitor these event streams for changes of interest. Automated systems which oversee many tasks can still have a need for the ‘human-in-the-loop’ operator to evaluate whether an intervention is required due to a lack of suitable training data initially offered to the system which would allow a correct course of actions to be taken. In order for an operator to be capable of reacting to real-time events, the visual depiction of the event data must be in a form which captures essential associations and is readily understood by visual inspection. A similar requirement can be found during inspections on activity protocols in a large organization where a code of correct conduct is prescribed and there is a need to oversee whether the activity traces match the expectations, with minimal delay. The methodology presented here addresses these concerns by providing an adaptive window sizing measurement for subsetting the data, and subsequently produces a set of network diagrams based upon event label co-occurrence networks. With an intuitive method of network construction the amount of time required for operators to learn how to monitor complex event streams of big datasets can be reduced.

The method of correct calculation of regulators based on numerical optimization

Automation as a branch of engineering science forms and studies methods of control of objects, the main essence of which is to form the structure of the regulator and calculate its coefficients. The number of regulator structures is limited, the most common of them is a serial PID, i.e. a regulator having a proportional, integral and derivative channels connected in parallel, and the regulator itself is switched in series with the object. Thus, most of the publications on automation are connected with the calculation of numerical values of the regulator’s coefficient and analysis of the results of its use. Along with the publication of the results of such a calculation for increasingly complex objects, there are still rather big stream of papers, that solve the problem of calculating the regulator for objects, the control of which is no longer of great complexity due to the level of the achieved technology. Therefore, the question of dividing the classes of tasks into those that have already been solved, and those whose solution has yet to be found is relevant. It is required to establish a minimum level of complexity of such problems, and it is also useful to compile a list of obsolete methods, the use of which without cardinal modification is fruitless in comparison with the potential of other methods, which became possible due to the latest achievements of theory and rapid development of computing facilities. Some scientific schools traditionally use obsolete approaches. One of the reasons for this may be their lack of awareness of the latest achievements; another reason may be inept use of these methods, which generates errors and disappointments in them. This paper reveals similar errors and suggests methods of correct calculation. When used correctly, the numerical optimization is the most effective tool for the said problems.

Dagstuhl Seminar on Big Stream Processing

Stream processing can generate insights from big data in real time as it is being produced. This paper reports findings from a 2017 seminar on big stream processing, focusing on applications, systems, and languages.

DeepAnT: A Deep Learning Approach for Unsupervised Anomaly Detection in Time Series

Traditional distance and density-based anomaly detection techniques are unable to detect periodic and seasonality related point anomalies which occur commonly in streaming data, leaving a big gap in time series anomaly detection in the current era of the IoT. To address this problem, we present a novel deep learning-based anomaly detection approach (DeepAnT) for time series data, which is equally applicable to the non-streaming cases. DeepAnT is capable of detecting a wide range of anomalies, i.e., point anomalies, contextual anomalies, and discords in time series data. In contrast to the anomaly detection methods where anomalies are learned, DeepAnT uses unlabeled data to capture and learn the data distribution that is used to forecast the normal behavior of a time series. DeepAnT consists of two modules: time series predictor and anomaly detector. The time series predictor module uses deep convolutional neural network (CNN) to predict the next time stamp on the defined horizon. This module takes a window of time series (used as a context) and attempts to predict the next time stamp. The predicted value is then passed to the anomaly detector module, which is responsible for tagging the corresponding time stamp as normal or abnormal. DeepAnT can be trained even without removing the anomalies from the given data set. Generally, in deep learning-based approaches, a lot of data are required to train a model. Whereas in DeepAnT, a model can be trained on relatively small data set while achieving good generalization capabilities due to the effective parameter sharing of the CNN. As the anomaly detection in DeepAnT is unsupervised, it does not rely on anomaly labels at the time of model generation. Therefore, this approach can be directly applied to real-life scenarios where it is practically impossible to label a big stream of data coming from heterogeneous sensors comprising of both normal as well as anomalous points. We have performed a detailed evaluation of 15 algorithms on 10 anomaly detection benchmarks, which contain a total of 433 real and synthetic time series. Experiments show that DeepAnT outperforms the state-of-the-art anomaly detection methods in most of the cases, while performing on par with others.

PERBANDINGAN ANTARA FILSAFAT EKONOMI ISLAM DAN BARAT

Economic philosophy is an undeveloped theme in department of economy. Whereas during the 20th Century there is a clash of two big ideologies that caused by differences in economic philosophy, they are the capitalism and the socialism. These Two big streams often used as comparison object. This Writing, look at the two big streams in a same frame, that is the thought that arise in western worldview which different with Islamic worldview. So it is interesting to study the comparison between philosophy of Islamic economic and western economic through branchs of philosopy, they are ontology, epistemology and axiology.

Katahi Ti

The book “The Tail of the Fish” was publised in 1968 and written by a Te Aupouri kuia, Matire Kereama (nee: Hoeft) of the far north of Aotearoa, New Zealand. I grew up with this book as my grandmother would read the stories to me at bedtime. Although my comprehension of each story was very vague and unrelatable to my life at that time, today, I find myself totally absorbed by the historical content and knowledge encapsulated in each chapter.
 I completed a Masters of Applied Indigenous Knowledge at Te Wananga o Aotearoa in 2017, entitled; Tales of the singing fish: He tangi wairua. I compsed twelve waiata (Maori songs) of which ten of the waiata was information extracted from ten chapters of the book. The other two waiata were composed specifically for my people of the Te Rarawa tribe, namely, Ahipara.
 Parents today can scarcely believe how the Māori children of long ago survived without cupboards full of ready-to-eat food. Yet they did survive wonderfully well and were healthy because they lived active outdoor lives, which developed their physical and resistance to disease. Quite a big stream flowed through Hauturu settlement and contained many swimming pools.

A Virtual Learning Architecture Enhanced by Fog Computing and Big Data Streams

In recent years, virtual learning environments are gaining more and more momentum, considering both the technologies deployed in their support and the sheer number of terminals directly or indirectly interacting with them. This essentially means that every day, more and more smart devices play an active role in this exemplary Web of Things scenario. This digital revolution, affecting education, appears clearly intertwined with the earliest forecasts of the Internet of Things, envisioning around 50 billions heterogeneous devices and gadgets to be active by 2020, considering also the deployment of the fog computing paradigm, which moves part of the computational power to the edge of the network. Moreover, these interconnected objects are expected to produce more and more significant streams of data, themselves generated at unprecedented rates, sometimes to be analyzed almost in real time. Concerning educational environments, this translates to a new type of big data stream, which can be labeled as educational big data streams. Here, pieces of information coming from different sources (such as communications between students and instructors, as well as students’ tests, etc.) require accurate analysis and mining techniques in order to retrieve fruitful and well-timed insights from them. This article presents an overview of the current state of the art of virtual learning environments and their limitations; then, it explains the main ideas behind the paradigms of big data streams and of fog computing, in order to introduce an e-learning architecture integrating both of them. Such an action aims to enhance the ability of virtual learning environments to be closer to the needs of all the actors in an educational scenario, as demonstrated by a preliminary implementation of the envisioned architecture. We believe that the proposed big stream and fog-based educational framework may pave the way towards a better understanding of students’ educational behaviors and foster new research directions in the field.

THORIN: an Efficient Module for Federated Access and Threat Mitigation in Big Stream Cloud Architectures

In order to make cloud services attractive for several IT organizations, it is necessary to provide access control and to implement safe and reliable mechanisms of Identity and Access Management (IAM). In this work, we focus on security issues and challenges in the design and implementation of cloud architectures and, in particular, for the management of Big Stream applications in Internet of Things (IoT) scenarios. The proposed work introduces a new set of modules allowing a federated access control policy for cloud users. An analysis of possible threats and attacks against the proposed Big Stream platform is presented, investigating the system performance in terms of detection and elimination of malicious nodes. In particular, we propose a new module, denoted as Traffic Handler Orchestrator & Rapid Intervention (THORIN), which is very efficient in counteracting botnet-based threats.