Today's Big Data Research Articles

Robust and efficient memory management in Apache AsterixDB

SummaryTraditional relational database systems handle data by dividing their memory into sections such as a buffer cache and working memory, assigning a memory budget to each section to efficiently manage a limited amount of overall memory. They also assign memory budgets to memory‐intensive operators such as sorts and joins and control the allocation of memory to these operators; each memory‐intensive operator attempts to maximize its memory usage to reduce disk I/O cost. Implementing such memory‐intensive operators requires a careful design and application of appropriate algorithms that properly utilize memory. Today's Big Data management systems need the ability to handle large amounts of data similarly, as it is unrealistic to assume that truly big data will fit into memory. In this article, we share our memory management experiences in Apache AsterixDB, an open‐source Big Data management software platform that scales out horizontally on shared‐nothing commodity computing clusters. We describe the implementation of AsterixDB's memory‐intensive operators and their designs related to memory management. We also discuss memory management at the global (cluster) level. We conducted an experimental study using several synthetic and real datasets to explore the impact of this work. We believe that future Big Data management system builders can benefit from these experiences.

A Review: Big Data Technologies with Hadoop Distributed Filesystem and Implementing M/R

Today Big Data, is any set of data that is larger than the capacity to be processed using traditional database tools to capture, share, transfer, store, manage and analyze within an acceptable time frame; from the point of view of service providers, Organizations need to deal with a large amount of data for the purpose of analysis. And IT department are facing tremendous challenge in protecting and analyzing these increased volumes of information. The reason organizations are collecting and storing more data than ever before is because their business depends on it. The type of information being created is no more traditional database-driven data referred to as structured data rather it is data that include documents, images, audio, video, and social media contents known as unstructured data or Big Data. Big Data Analytics is a way of extracting value from these huge volumes of information, and it drives new market opportunities and maximizes customer retention. Moreover, this paper focuses on discussing and understanding Big Data technologies and Analytics system with Hadoop distributed filesystem (HDFS). This can help predict future, obtain information, take proactive actions and make way for better strategic decision making.

Technical Perspective

When problems are scaled to "big data," researchers must often come up with new solutions, leveraging ideas from multiple research areas - as we frequently witness in today's big data techniques and tools for machine learning, bioinformatics, and data visualization. Beyond these heavily studied topics, there exist other classes of general problems that need to be rethought at scale. One such problem is that of large-scale signal reconstruction [4]: taking a set of observations of relatively low dimensionality, and using them to reconstruct a high-dimensional, unknown signal. This class of problems arises when we can only observe a subset of a complex environment that we are seeking to model - for instance, placing a few sensors and using their readings to reconstruct an environment's temperature, or monitoring multiple points in a network and using the readings to estimate end-to-end network traffic, or using 2D slices to reconstruct a 3D image.

Big Data from A to Z

The rapid development of technology over the past 20 years has led to explosive data growth in various industries, including defense industries, healthcare. The analysis of generated Big Data has recently been addressed by many researchers, because today's Big Data analysis are one of the most important and most profitable areas of development in Data Science and companies that are able to extract valuable knowledge among the massive amount of data at logical time can earn significant advantages . Accordingly, in this survey, we investigate definition of the Big Data and the data sources. Also look at advantages, challenges, applications, analysis and platforms used in the Big Data.

Academic Decision Support System for Choosing Information Systems Sub Majors Programs using Decision Tree Algorithm

Background: Educational data mining is an emerging trend, especially in today Big Data Era. Numerous method and technique already been implemented in order to improve its process to gain better understanding of the educational process and to extract knowledge from various related data, but the implementation of these methods into Decision support system (DSS) application still limited, especially regarding help to choose university sub majors .Objective: To design an academic decision support system (DSS) by adopting Theory of Reasoned Action (TRA) concept and using Data Mining as a factor analytic apporach to extract rules for its knowledge model.Methods: We implemented factor analysis method and decision tree method of C.45 to produce rules of the impact course of the sub- majors and the job interest as the basic rules of the DSS.Results: The proposed academic decision support system able to give sub majors recommendations in accordance with student interest and competence, with 79.03% of precision and 61.11% of recall. Moreover, the system also has a dashboard feature that shows the information about the statistic of students in each sub majors.Conclusion: C.45 algorithm and factor analysis are suitable to build a knowledge model for Academic Decision Support System for Choosing Information System Sub Majors Bachelor Programs. This system could also help the academic adviser on monitoring and make decision accordance with that academic information

Practical Privacy-Preserving MapReduce Based K-Means Clustering Over Large-Scale Dataset

Clustering techniques have been widely adopted in many real world data analysis applications, such as customer behavior analysis, targeted marketing, digital forensics, etc. With the explosion of data in today's big data era, a major trend to handle a clustering over large-scale datasets is outsourcing it to public cloud platforms. This is because cloud computing offers not only reliable services with performance guarantees, but also savings on in-house IT infrastructures. However, as datasets used for clustering may contain sensitive information, e.g., patient health information, commercial data, and behavioral data, etc, directly outsourcing them to public cloud servers inevitably raise privacy concerns. In this paper, we propose a practical privacy-preserving K-means clustering scheme that can be efficiently outsourced to cloud servers. Our scheme allows cloud servers to perform clustering directly over encrypted datasets, while achieving comparable computational complexity and accuracy compared with clusterings over unencrypted ones. We also investigate secure integration of MapReduce into our scheme, which makes our scheme extremely suitable for cloud computing environment. Thorough security analysis and numerical analysis carry out the performance of our scheme in terms of security and efficiency. Experimental evaluation over a 5 million objects dataset further validates the practical performance of our scheme.

A utility-based news recommendation system

News platforms exhibit both the challenges as well as opportunities for enhancing the functionalities of recommendation systems in today's big data environment. Novel use of big data storage and programming models can improve news recommendation systems through efficient handling and analysis of clickstream data and a better understanding of users' interests. Most existing approaches to news recommendation consider users' clicks as the implicit feedback to understand user behaviors. However, “clicks” may not be an effective indicator of real user interests. We address this problem by developing a novel news recommendation system based on a news utility model. Given the new utility model, we propose a two stage news recommendation framework. The framework first generates article-level recommendation rules based on the utility model, then integrates the notion of utility and probabilistic topic models and generates topic-level recommendation rules. We argue that the proposed utility-based news recommendation system also addresses the news cold start problem which is one of the most challenging obstacles for news agencies. We evaluate the framework on a massive real dataset (two billion records) obtained from a major newspaper (i.e., The Globe and Mail) in Canada and show that it outperforms the existing methods.

The role of privacy policy on consumers’ perceived privacy

With today's big data and analytics capability, access to consumer data provides competitive advantage. Analysis of consumers' transactional data helps organizations to understand customer behaviors and preferences. However, prior to capitalizing on the data, organizations ought to have effective plans for addressing consumers' privacy concerns because violation of consumer privacy brings long-term reputational damage. This paper proposes and tests a Privacy Boundary Management Model, explaining how consumers formulate and manage their privacy boundary. It also analyzes the effect of the five dimensions of privacy policy (Fair Information Practices) on privacy boundary formation to assess how customers link these dimensions to the effectiveness of privacy policy. Survey data was collected from 363 customers who have used online banking websites for a minimum of six months. Partial Least Square results showed that the validated research model accounts for high variance in perceived privacy. Four elements of the Fair Information Practice Principles (access, notice, security, and enforcement) have significant impact on perceived effectiveness of privacy policy. Perceived effectiveness in turn significantly influences perceived privacy control and perceived privacy risk. Perceived privacy control significantly influences trust and perceived privacy. Perceived privacy concern and trust also significantly influence perceived privacy.

Real-Time Bayesian Nonparametric Prediction of Solvency Risk

Insurance regulation often dictates that insurers monitor their solvency risk in real time and take appropriate actions whenever the risk exceeds their tolerance level. Bayesian methods are appealing for prediction problems thanks to their ability to naturally incorporate both sample variability and parameter uncertainty into a predictive distribution. However, handling data arriving in real time requires a flexible nonparametric model, and the Monte Carlo methods necessary to evaluate the predictive distribution in such cases are not recursive and can be too expensive to rerun each time new data arrives. In this paper, we apply a recently-developed alternative perspective on Bayesian prediction based on copulas. This approach facilitates recursive Bayesian prediction without computing a posterior, allowing insurers to perform real time updating of risk measures to assess solvency risk, and providing them with a tool for carrying out dynamic risk management strategies in today's big data' era.

Special issue on advanced techniques for cloud data storage and collaborative systems

Cloud computing paradigm builds on the foundations of distributed computing, grid computing, virtualisation, service orientation, etc. Cloud storage, which is one of the most attractive cloud services, offers numerous benefits from both the technology and functionality perspectives such as increased availability, flexibility, and functionality. Therefore, Cloud-based platforms have become fundamental to collaborative and data management systems. However, the new Cloud environment creates major collaborative and data management challenges such as efficient and reliable remote storage, online data query processing, data integrity, outsourcing computation, and secure virtualisation, among others. Such issues need to be carefully studied and solved to enable the wide deployment and adoption of Cloud computing in businesses, industry, etc. The purpose of this special issue is to publish recent advances in Cloud-based collaborative systems and Cloud storage. The special issue comprises five high-quality papers, which are arranged as follows. De Maio et al in the first paper1 “Social Media Marketing through Time-Aware Collaborative Filtering,” define a time-aware collaborative filtering for estimating users' interest along the time in Twitter. Their approach uses text analysis services to semantically annotate tweets' content and to track concepts considering post frequencies along the time. A model-based approach implementing K-Nearest Neighbors is used to estimate user's similarity representing their profile by sampling user's interest with three different techniques: Vectorial Representation, Symbolic Aggregation Approximation, and Median. The authors present experimental results comparing these techniques and performing model training in different time windows. The second paper2 “SERNOTATE: An Automated Approach for Business Service Description Annotation for efficient Service Retrieval and Composition” by Chotipant et al proposes an automated approach, namely, SERNOTATE, for business service description annotation for efficient service retrieval and composition. To that end, the authors employ new semantic-based linking approaches, namely, Extended Case-based Reasoning, Vector-based and Classification-based, that automatically annotate business services to relevant service concepts. The experimental results test and validate the applicability of the proposed approaches to the automatic annotation of business service descriptions to service concepts on a real-world dataset. In the third paper,3 “Sensor Data Management in the Cloud: Data Storage, Data Ingestion, and Data Retrieval,” the authors address an increasing need to capture, store, and analyse the dynamic semistructured data from sensors. A similar growth of semistructured data in the modern Web has led to the creation of NoSQL data stores for scalability, availability, and performance, whereas large-scale data processing frameworks for parallel analysis. The authors study how sensor data management can benefit from MongoDB with Apache Spark technologies, specifically for ingesting high-velocity sensor data and parallel retrieval of high volume data. The performance of MongoDB sharding and no-sharding databases with Apache Spark are evaluated to identify the right software environment for sensor data management. Park et al in the fourth paper,4 “Study on the SDN-IP based Solution of Well-known Bottleneck Problems in Private Sector of National R&E Network for Big Data Transfer,” deal with issues arising in Software Defined Networking (SDN). The key feature of SDN networks is the provisioning of dynamic service architecture. Therefore, it is more appropriate for the big data science computing trends, for handling today's big data science, which requires massive parallel processing and a constant demand for additional capacity and on demand connectivity. The authors bring the experience from KREONET (Korea Research Environment Open Network) and Korea National R&E Network, which adopt SDN as a platform service for big data scientific application. The big data application for KREONET SDN service comprises high-energy physics, astrophysics, biology/genetics, meteorology, artificial satellite data, etc. The last paper5 “Improved Outsourced Private Set Intersection Protocol Based on Polynomial Interpolation” by Yang et al investigates issues in private set intersection (PSI) protocols, which enable two parties to compute the intersection of their inputs without compromising anything about the datasets beyond the intersection. The authors present a variant of delegated private set intersection protocol secure in the semi-honest model under RSA assumption as well as an efficient and secure outsourcing computation algorithm for RSA crypto-system. Based on this algorithm, the authors transform a variant of delegated private set intersection protocol into an improved outsourced one, which enables the clients to perform only simple modular multiplication for computing during the execution of the protocol. Compared with the state of the art, the proposed protocol has a great advantage in efficiency. The editor of this special issue would like to thank all authors for their contributions and the reviewers for their constructive and useful feedback to authors. I would like to thank Prof Geoffrey Fox, the Editor-in-Chief of Concurrency and Computation: Practice and Experience, for the opportunity to edit this special issue and his support.

A Study on the Innovation of U-Education of Party Members from the Perspective of Big Data

Potential cognition of the social development system centered on economic construction has been changed. To a certain extent, it has formed the new concept of "world outlook and views on life and values" for the younger generation. It is imperative that innovation on the form of education of party members in the new era be enforced. In today's big data, problems in the education of party members can be concluded from the analysis of data collection and mining. To explore the U-Education methods and modes for special party members of a certain age group and create an "internet Party Construction platform +" can effectively deal with the shortcomings of the existing education, and through more convenient and quicker APP operation, widely used mobile phones can effectively conduct education and study, and data statistics function of the platform can be used for evaluation and analysis to seek a more perfect education mode.

Two birds with one stone: Classifying positive and unlabeled examples on uncertain data streams

Abstract An important feature characteristic of the data streams in many of today's big data applications is the intrinsic uncertainty, which could happen for both item occurrence and attribute value. While this has already posed great challenges for fundamental data mining tasks such as classification, things are made even more complicated by the fact that completely-labeled examples are usually unavailable in such settings, leaving researchers the only option to learn classifiers on partially-labeled examples on uncertain data streams. Furthermore, there will be concept drift on evolving data streams. To address these challenges, this paper therefore focuses on the study of learning from positive and unlabeled examples (PU) on uncertain data streams. To the best of our knowledge, this paper is the first work to address the uncertainty issue in both item occurrence (occurrence level) and attribute value (attribute level) for the problem of PU learning over streaming data. Firstly, we propose an algorithm to classify positive and unlabeled examples with both uncertainties (PUU). The algorithm extracts reliable positive and negative examples by clustering-based method, and then trains the classifier with Weighted Extreme Learning Machine (Weighted ELM). Secondly, we propose an algorithm of PU learning over uncertain data streams (PUUS). It adopts ensemble model and trains base classification by PUU. In order to detect concept drift, PUUS uses cluster set similarities between the current data block and history data block. We propose different update strategies for different concept drift to adapt to evolving uncertain data streams. Experimental results show that PUUS can effectively classify uncertain data streams with just positive and unlabeled data, while achieving in the meantime good performance in detecting and handling concept drifts.

A BAD demonstration

Nearly all of today's Big Data systems are passive in nature. We demonstrate our Big Active Data ("BAD") system, a scalable system that continuously and reliably captures Big Data and facilitates the timely and automatic delivery of new information to a large population of interested users as well as supporting analyses of historical information. We built our BAD project by extending an existing scalable, open-source BDMS (AsterixDB [1]) in this active direction. In this demonstration, we allow our audience to participate in an emergency notification application built on top of our BAD platform, and highlight its capabilities.

Finding useful data across multiple biomedical data repositories using DataMed.

The value of broadening searches for data across multiple repositories has been identified by the biomedical research community. As part of the US National Institutes of Health (NIH) Big Data to Knowledge initiative, we work with an international community of researchers, service providers and knowledge experts to develop and test a data index and search engine, which are based on metadata extracted from various data sets in a range of repositories. DataMed is designed to be, for data, what PubMed has been for the scientific literature. DataMed supports the findability and accessibility of data sets. These characteristics—along with interoperability and reusability—compose the four FAIR principles to facilitate knowledge discovery in today's big data–intensive science landscape.

Spatial Data Infrastructures

Spatial data infrastructure (SDI) is the infrastructure that facilitates the discovery, access, management, distribution, reuse, and preservation of digital geospatial resources. These resources may include maps, data, geospatial services, and tools. As cyberinfrastructures, SDIs are similar to other infrastructures, such as water supplies and transportation networks, since they play fundamental roles in many aspects of the society. These roles have become even more significant in today's big data age, when a large volume of geospatial data and Web services are available. From a technological perspective, SDIs mainly consist of data, hardware, and software. However, a truly functional SDI also needs the efforts of people, supports from organizations, government policies, data and software standards, and many others. In this chapter, we will present the concepts and values of SDIs, as well as a brief history of SDI development in the U.S. We will also discuss the components of a typical SDI, and will specifically focus on three key components: geoportals, metadata, and search functions. Examples of the existing SDI implementations will also be discussed.

Entity reconciliation in big data sources: A systematic mapping study

The entity reconciliation (ER) problem aroused much interest as a research topic in today's Big Data era, full of big and open heterogeneous data sources. This problem poses when relevant information on a topic needs to be obtained using methods based on: (i) identifying records that represent the same real world entity, and (ii) identifying those records that are similar but do not correspond to the same real-world entity. ER is an operational intelligence process, whereby organizations can unify different and heterogeneous data sources in order to relate possible matches of non-obvious entities. Besides, the complexity that the heterogeneity of data sources involves, the large number of records and differences among languages, for instance, must be added. This paper describes a Systematic Mapping Study (SMS) of journal articles, conferences and workshops published from 2010 to 2017 to solve the problem described before, first trying to understand the state-of-the-art, and then identifying any gaps in current research. Eleven digital libraries were analyzed following a systematic, semiautomatic and rigorous process that has resulted in 61 primary studies. They represent a great variety of intelligent proposals that aim to solve ER. The conclusion obtained is that most of the research is based on the operational phase as opposed to the design phase, and most studies have been tested on real-world data sources, where a lot of them are heterogeneous, but just a few apply to industry. There is a clear trend in research techniques based on clustering/blocking and graphs, although the level of automation of the proposals is hardly ever mentioned in the research work.

Research on Optimization of Pooling System and Its Application in Drug Supply Chain Based on Big Data Analysis.

Reform of drug procurement is being extensively implemented and expanded in China, especially in today's big data environment. However, the pattern of supply mode innovation lags behind procurement improvement. Problems in financial strain and supply break frequently occur, which affect the stability of drug supply. Drug Pooling System is proposed and applied in a few pilot cities to resolve these problems. From the perspective of supply chain, this study analyzes the process of setting important parameters and sets out the tasks of involved parties in a pooling system according to the issues identified in the pilot run. The approach is based on big data analysis and simulation using system dynamic theory and modeling of Vensim software to optimize system performance. This study proposes a theoretical framework to resolve problems and attempts to provide a valuable reference for future application of pooling systems.

Obtaining Good Heuristics for Big Data Broadcasting Problem Using Novel Pipeline Approach.

Today big data computing is a critical challenge faced by many industries .Everyday up to terabytes to petabytes of the data is generated in the cloud computing paradigm. Thus there is a drastic demand for building a service to distribute, manage and process massive data sets that has risen tremendously. Engineers and researchers are dealing with these large dataset. In this paper, we investigate the big data broadcasting problem for single source node to broadcast a big chunk of data to a set of nodes with the objective of minimizing the maximum completion time. These nodes may be present in the same data center or across geo distributed data centers. We model the big data broadcasting problem into a LOCKSTEP BROADCAST TREE (LSBT).The main idea of this model is to define a basic unit of upload bandwidth r, such that a node with capacity c broadcast a data to a set of nodes at the rate r. We divide the broadcast data into m chunks. These data chunks can then be broadcasted using a pipeline approach.

Real-time stream processing for Big Data

Abstract With the rise of the web 2.0 and the Internet of things, it has become feasible to track all kinds of information over time, in particular fine-grained user activities and sensor data on their environment and even their biometrics. However, while efficiency remains mandatory for any application trying to cope with huge amounts of data, only part of the potential of today's Big Data repositories can be exploited using traditional batch-oriented approaches as the value of data often decays quickly and high latency becomes unacceptable in some applications. In the last couple of years, several distributed data processing systems have emerged that deviate from the batch-oriented approach and tackle data items as they arrive, thus acknowledging the growing importance of timeliness and velocity in Big Data analytics. In this article, we give an overview over the state of the art of stream processors for low-latency Big Data analytics and conduct a qualitative comparison of the most popular contenders, namely Storm and its abstraction layer Trident, Samza and Spark Streaming. We describe their respective underlying rationales, the guarantees they provide and discuss the trade-offs that come with selecting one of them for a particular task.

Breaking BAD: A Data Serving Vision for Big Active Data.

Virtually all of today's Big Data systems are passive in nature. Here we describe a project to shift Big Data platforms from passive to active. We detail a vision for a scalable system that can continuously and reliably capture Big Data to enable timely and automatic delivery of new information to a large pool of interested users as well as supporting analyses of historical information. We are currently building a Big Active Data (BAD) system by extending an existing scalable open-source BDMS (AsterixDB) in this active direction. This first paper zooms in on the Data Serving piece of the BAD puzzle, including its key concepts and user model.