Big Data Analytics Framework Research Articles

Cross-Cloud MapReduce for Big Data

MapReduce plays a critical role as a leading framework for big data analytics. In this paper, we consider a geo-distributed cloud architecture that provides MapReduce services based on the big data collected from end users all over the world. Existing work handles MapReduce jobs by a traditional computation-centric approach that all input data distributed in multiple clouds are aggregated to a virtual cluster that resides in a single cloud. Its poor efficiency and high cost for big data support motivate us to propose a novel data-centric architecture with three key techniques, namely, cross-cloud virtual cluster, data-centric job placement, and network coding based traffic routing. Our design leads to an optimization framework with the objective of minimizing both computation and transmission cost for running a set of MapReduce jobs in geo-distributed clouds. We further design a parallel algorithm by decomposing the original large-scale problem into several distributively solvable subproblems that are coordinated by a high-level master problem. Finally, we conduct real-world experiments and extensive simulations to show that our proposal significantly outperforms the existing works.

Unraveling environmental justice in ambient PM2.5 exposure in Beijing: A big data approach

Air pollution imposes significant environmental and health risks worldwide and is expected to deteriorate in the coming decade as cities expand. Measuring population exposure to air pollution is crucial to quantifying risks to public health. In this work, we introduce a big data analytics framework to model residents' stay and commuters' travel exposure to outdoor PM2.5 and evaluate their environmental justice, with Beijing as an example. Using mobile phone and census data, we first infer travel demand of the population to derive residents' stay activities in each analysis zone, and then focus on commuters and estimate their travel routes with a traffic assignment model. Based on air quality observations from monitoring stations and a spatial interpolation model, we estimate the outdoor PM2.5 concentrations at a 500-m grid level and map them to road networks. We then estimate the travel exposure for each road segment by multiplying the PM2.5 concentration and travel time spent on the road. By combining the estimated PM2.5 exposure and housing price harnessed from online housing transaction platforms, we discover that in the winter, Beijing commuters with low wealth level are exposed to 13% more PM2.5 per hour than those with high wealth level when staying at home, but exposed to less PM2.5 by 5% when commuting the same distance (due to lighter traffic congestion in suburban areas). We also find that the residents from the southern suburbs of Beijing have both lower level of wealth and higher stay- and travel- exposure to PM2.5, especially in the winter. These findings inform more equitable environmental mitigation policies for future sustainable development in Beijing. Finally, or the first time in the literature, we compare the results of exposure estimated from passive data with subjective measures of perceived air quality (PAQ) from a survey. The PAQ data was collected via a mobile-app. The comparison confirms consistencies in results and the advantages of the big data for air pollution exposure assessments.

Beyond Hadoop: Spark and Storm

More than a decade after Hadoop appeared on the scene as an open-source framework for big data analysis, much of its independent benefits has revolutionized fast processing of the data. Its development has triggered a number of improvisations for specific needs of data processing, depending on the nature of the processing techniques at different levels of computation. These upgrades have not only shot up the processing speed but also capable to counter intensive real-time computation requirements. This paper highlights the challenges posed by the basic Hadoop architecture and compares with the remedies as provided by Spark and Storm frameworks.

Conceptual frameworks for social and cultural Big Data analytics: Answering the epistemological challenge

This paper aims to contribute to the development of tools to support an analysis of Big Data as manifestations of social processes and human behaviour. Such a task demands both an understanding of the epistemological challenge posed by the Big Data phenomenon and a critical assessment of the offers and promises coming from the area of Big Data analytics. This paper draws upon the critical social and data scientists’ view on Big Data as an epistemological challenge that stems not only from the sheer volume of digital data but, predominantly, from the proliferation of the narrow-technological and the positivist views on data. Adoption of the social-scientific epistemological stance presupposes that digital data was conceptualised as manifestations of the social. In order to answer the epistemological challenge, social scientists need to extend the repertoire of social scientific theories and conceptual frameworks that may inform the analysis of the social in the age of Big Data. However, an ‘epistemological revolution’ discourse on Big Data may hinder the integration of the social scientific knowledge into the Big Data analytics.

Operational Risk Management: Integrating Big Data Analytics for Proactive Decision-Making

The impact of climate change has made responsible risk management a major research topic during the past 20 years. In conjunction with societal problems that affect the economies and cultures in which they function, industrial risks can release dangerous pollutants into the natural world. Advances in information and communication technology, particularly big data analytics, can contribute to the creation of fresh perspectives that enable the detection of business risks whose operations are unstable and the implementation of remedial actions. Although risk management has been the subject of numerous research, there are few that examine the impact of BDA. This study strives to offer a big data analytic framework that integrates a pipeline of statistical testing, data visualization, and machine learning algorithms to interpret market information. The applicability of our framework in recognizing and managing risks is demonstrated through a case study of the global commodity market. Extensive proof-of-concept experimentations validated the efficiency and effectiveness of the argued framework by providing useful insights about market behavior, which can lead the decision-making process to get informed risk management.

Big data analytics framework to analyse student's performance

Performance of the students is the most important element for any educational institution. Knowing the areas where student lags proficiency helps the student as well as the teacher to improve and make the education system to be in par with this competitive world. Aim of this paper is to provide a framework that can analyse the performance of the students. This framework was experimented with a first semester course which has 5,962 students registered. Students were given a problem, taught programming concepts relevant to solving it and they are made to experiment them immediately on a machine. Three categories of exercises practice problems; assessment problems and challenging task were given to the students through an online portal. Rubrics were designed to evaluate the performance of students. A scalable MapReduce-based analytics model using Spark framework was developed and results were visualised. Then K-means clustering algorithm was also applied to group the students based on their performance.

A Novel Framework for Big Data Analytics in Business Intelligence

International Journal of Computer Sciences and Engineering (A UGC Approved and indexed with DOI, ICI and Approved, DPI Digital Library) is one of the leading and growing open access, peer-reviewed, monthly, and scientific research journal for scientists, engineers, research scholars, and academicians, which gains a foothold in Asia and opens to the world, aims to publish original, theoretical and practical advances in Computer Science,Information Technology, Engineering (Software, Mechanical, Civil, Electronics & Electrical), and all interdisciplinary streams of Computing Sciences. It intends to disseminate original, scientific, theoretical or applied research in the field of Computer Sciences and allied fields. It provides a platform for publishing results and research with a strong empirical component. It aims to bridge the significant gap between research and practice by promoting the publication of original, novel, industry-relevant research.

A Novel Framework and Enhanced QoS Big Data Protocol for Smart City Applications.

Various heterogeneous devices or objects will be integrated for transparent and seamless communication under the umbrella of Internet of things (IoT). This would facilitate the open access of data for the growth of various digital services. Building a general framework of IoT is a complex task because of the heterogeneity in devices, technologies, platforms and services operating in the same system. In this paper, we mainly focus on the framework for Big Data analytics in Smart City applications, which being a broad category specifies the different domains for each application. IoT is intended to support the vision of Smart City, where advance technologies will be used for communication to improve the quality of life of citizens. A novel approach is proposed in this paper to enhance energy conservation and reduce the delay in Big Data gathering at tiny sensor nodes used in IoT framework. To implement the Smart City scenario in terms of Big Data in IoT, an efficient (optimized in quality of service) wireless sensor network (WSN) is required where communication of nodes is energy efficient. Thus, a new protocol, QoS-IoT(quality of service enabled IoT), is proposed on the top layer of the proposed architecture (the five-layer architecture consists of technology, data source, data management, application and utility programs) which is validated over the traditional protocols.

A Two-Stage Big Data Analytics Framework with Real World Applications Using Spark Machine Learning and Long Short-Term Memory Network

Every day we experience unprecedented data growth from numerous sources, which contribute to big data in terms of volume, velocity, and variability. These datasets again impose great challenges to analytics framework and computational resources, making the overall analysis difficult for extracting meaningful information in a timely manner. Thus, to harness these kinds of challenges, developing an efficient big data analytics framework is an important research topic. Consequently, to address these challenges by exploiting non-linear relationships from very large and high-dimensional datasets, machine learning (ML) and deep learning (DL) algorithms are being used in analytics frameworks. Apache Spark has been in use as the fastest big data processing arsenal, which helps to solve iterative ML tasks, using distributed ML library called Spark MLlib. Considering real-world research problems, DL architectures such as Long Short-Term Memory (LSTM) is an effective approach to overcoming practical issues such as reduced accuracy, long-term sequence dependency, and vanishing and exploding gradient in conventional deep architectures. In this paper, we propose an efficient analytics framework, which is technically a progressive machine learning technique merged with Spark-based linear models, Multilayer Perceptron (MLP) and LSTM, using a two-stage cascade structure in order to enhance the predictive accuracy. Our proposed architecture enables us to organize big data analytics in a scalable and efficient way. To show the effectiveness of our framework, we applied the cascading structure to two different real-life datasets to solve a multiclass and a binary classification problem, respectively. Experimental results show that our analytical framework outperforms state-of-the-art approaches with a high-level of classification accuracy.

Big data analytics: Machine learning and Bayesian learning perspectives—What is done? What is not?

Big data analytics provides an interdisciplinary framework that is essential to support the current trend for solving real‐world problems collaboratively. The progression of big data analytics framework must be clearly understood so that novel approaches can be developed to advance this state‐of‐the‐art discipline. An ignorance of observing the progression of this fast‐growing discipline may lead to duplications in research and waste of efforts. Its main companion field, machine learning, helps solve many big data analytics problems; therefore, it is also important to understand the progression of machine learning in the big data analytics framework. One of the current research efforts in big data analytics is the integration of deep learning and Bayesian optimization, which can help the automatic initialization and optimization of hyperparameters of deep learning and enhance the implementation of iterative algorithms in software. The hyperparameters include the weights used in deep learning, and the number of clusters in Bayesian mixture models that characterize data heterogeneity. The big data analytics research also requires computer systems and software that are capable of storing, retrieving, processing, and analyzing big data that are generally large, complex, heterogeneous, unstructured, unpredictable, and exposed to scalability problems. Therefore, it is appropriate to introduce a new research topic—transformative knowledge discovery—that provides a research ground to study and develop smart machine learning models and algorithms that are automatic, adaptive, and cognitive to address big data analytics problems and challenges. The new research domain will also create research opportunities to work on this interdisciplinary research space and develop solutions to support research in other disciplines that may not have expertise in the research area of big data analytics. For example, the research, such as detection and characterization of retinal diseases in medical sciences and the classification of highly interacting species in environmental sciences can benefit from the knowledge and expertise in big data analytics.This article is categorized under: Application Areas > Science and Technology Technologies > Machine Learning Fundamental Concepts of Data and Knowledge > Big Data Mining

Semantic Orientation Based Decision Making Framework for Big Data Analysis of Sporadic News Events

The growing public endorsement of social media has changed public life dramatically. Public views and suggestions have now become important for both organizations and individuals. Big data scientists and data mining analysts are increasingly moving their attention toward sentiment analysis because of the growing rate of user-generated contents over microblogging sites. Sentiment analysis is a research field related to computationally identifying public views, feelings, recommendations, opinions and sentiments about focused entities. Research literature shows traces of research work on product and movie reviews for better decision making using big data analysis. Big data analytics offer remarkable opportunities to individuals as well as organizations by providing proficient decision making frameworks and improved forecasting models. The sociopolitical collaboration has gained much attention from online users over the past few years. In this research we analyzed public views, sentiments and opinions shared on social media about a democratic participatory activity called Azadi-March, which was held in Pakistan with participation of online users from all over the world. We carried out computational semantic orientation on public tweets for analyzing public awareness and the effects of online communication through social media over the real world public decision making. We employed unsupervised approach for identification and scoring of tweets. We used lexicon based approach in which annotated lexica are used for scoring verbs, adverbs and other parts of speech. A corpus is used for scoring adjectives and informal opinion indicators. Emoji, exclamatory statements and other additional features are incorporated for supplementary analysis. We noticed that emoticons and NetLingo play significant role in sentiment orientation. Opinion groups are generated from all retrieved tweets and aggregate sentiment weights of opinion groups are computed. The findings of this study indicate that our proposed lexicon based approach outperforms the contemporary machine learning techniques by achieving 86% average accuracy at sentence level sentiment analysis.

Big Data Approach as an Institutional Innovation to Tackle Hong Kong’s Illegal Subdivided Unit Problem

While applications of big data have been extensively studied, discussion is mostly made from the perspectives of computer science, Internet services, and informatics. Alternatively, this article takes the big data approach as an institutional innovation and uses the problem of illegal subdivided units (ISUs) in Hong Kong as a case study. High transaction costs incurred in identification of suspected ISUs and associated enforcement actions lead to a proliferation of ISUs in the city. We posit that the deployment of big data analytics can lower these transaction costs, enabling the government to tackle the problem of illegal accommodations. We propose a framework for big data collection, analysis, and feedback. As the findings of a structured questionnaire survey reveal, building professionals believed that the proposed framework could reduce transaction costs of ISU identification. Yet, concerns associated with the big data approach like privacy and predictive policing were also raised by the professionals.

A novel big data analytics framework for smart cities

The emergence of smart cities aims at mitigating the challenges raised due to the continuous urbanization development and increasing population density in cities. To face these challenges, governments and decision makers undertake smart city projects targeting sustainable economic growth and better quality of life for both inhabitants and visitors. Information and Communication Technology (ICT) is a key enabling technology for city smartening. However, ICT artifacts and applications yield massive volumes of data known as big data. Extracting insights and hidden correlations from big data is a growing trend in information systems to provide better services to citizens and support the decision making processes. However, to extract valuable insights for developing city level smart information services, the generated datasets from various city domains need to be integrated and analyzed. This process usually referred to as big data analytics or big data value chain. Surveying the literature reveals an increasing interest in harnessing big data analytics applications in general and in the area of smart cities in particular. Yet, comprehensive discussions on the essential characteristics of big data analytics frameworks fitting smart cities requirements are still needed. This paper presents a novel big data analytics framework for smart cities called “Smart City Data Analytics Panel — SCDAP”. The design of SCDAP is based on answering the following research questions: what are the characteristics of big data analytics frameworks applied in smart cities in literature and what are the essential design principles that should guide the design of big data analytics frameworks have to serve smart cities purposes? In answering these questions, we adopted a systematic literature review on big data analytics frameworks in smart cities. The proposed framework introduces new functionalities to big data analytics frameworks represented in data model management and aggregation. The value of the proposed framework is discussed in comparison to traditional knowledge discovery approaches.

Cyber forensics framework for big data analytics in IoT environment using machine learning

Forensic analyst skills are at stake for processing of growing data from IoT based environment platforms. Tangible sources often have the size limits, but that’s not the case for communication traffic source. Hence, increasing the thirst for an efficient benchmarking for big data analysis. Available solutions to date have used an anomaly-based approach or have proposed approaches based on the deviation from a regular pattern. To tackle the seized bytes, authors have proposed an approach for big data forensics, with efficient sensitivity and precision. In the presented work, a generalized forensic framework has been proposed that use Google’s programming model, MapReduce as the backbone for traffic translation, extraction, and analysis of dynamic traffic features. For the proposed technique, authors have used open source tools like Hadoop, Hive, and Mahout and R. Apart from being open source, these tools support scalability and parallel processing. Also, comparative analysis of globally accepted machine learning models of P2P malware analysis in mocked real-time is presented. Dataset from CAIDA was taken and executed in parallel to validate the proposed model. Finally, the forensic performance metrics of the model shows the results with the sensitivity of 99%.

Big Data Systems Meet Machine Learning Challenges: Towards Big Data Science as a Service

Abstract Recently, we have been witnessing huge advancements in the scale of data we routinely generate and collect in pretty much everything we do, as well as our ability to exploit modern technologies to process, analyze and understand this data. The intersection of these trends is what is, nowadays, called Big Data Science. Big Data Science requires scalable architectures for storing and processing data. Cloud computing represents a practical and cost-effective solution for supporting Big Data storage, processing and for sophisticated analytics applications. We analyze in details the building blocks of the software stack for supporting Big Data Science as a commodity service for data scientists. In addition, we analyze and classify the state-of-the-art of big data analytics frameworks, available today mostly on Clouds, based on their supported service models. Furthermore, we provide various insights about the latest ongoing developments and open challenges in this domain.

Barriers to big data analytics in manufacturing supply chains: A case study from Bangladesh

Recently, big data (BD) has attracted researchers and practitioners due to its potential usefulness in decision-making processes. Big data analytics (BDA) is becoming increasingly popular among manufacturing companies as it helps gain insights and make decisions based on BD. However, there many barriers to the adoption of BDA in manufacturing supply chains. It is therefore necessary for manufacturing companies to identify and examine the nature of each barrier. Previous studies have mostly built conceptual frameworks for BDA in a given situation and have ignored examining the nature of the barriers to BDA. Due to the significance of both BD and BDA, this research aims to identify and examine the critical barriers to the adoption of BDA in manufacturing supply chains in the context of Bangladesh. This research explores the existing body of knowledge by examining these barriers using a Delphi-based analytic hierarchy process (AHP). Data were obtained from five Bangladeshi manufacturing companies. The findings of this research are as follows: (i) data-related barriers are most important, (ii) technology-related barriers are second, and (iii) the five most important components of these barriers are (a) lack of infrastructure, (b) complexity of data integration, (c) data privacy, (d) lack of availability of BDA tools and (e) high cost of investment. The findings can assist industrial managers to understand the actual nature of the barriers and potential benefits of using BDA and to make policy regarding BDA adoption in manufacturing supply chains. A sensitivity analysis was carried out to justify the robustness of the barrier rankings.

A framework for big data analytics approach to failure prediction of construction firms

This study explored use of big data analytics (BDA) to analyse data of a large number of construction firms to develop a construction business failure prediction model (CB-FPM). Careful analysis of literature revealed financial ratios as the best form of variable for this problem. Because of MapReduce’s unsuitability for iteration problems involved in developing CB-FPMs, various BDA initiatives for iteration problems were identified. A BDA framework for developing CB-FPM was proposed. It was validated by using 150,000 datacells of 30,000 construction firms, artificial neural network, AmazonElastic ComputeCloud, Apache Spark and the R software. The BDA CB-FPM was developed in eight seconds while the same process without BDA was aborted after nine hours without success. This shows the issue of not wanting to use large dataset to develop CB-FPM due to tedious duration is resolvable by applying BDA technique. The BDA CB-FPM largely outperformed an ordinary CB-FPM developed with a dataset of 200 construction firms, proving that use of larger sample size with the aid of BDA, leads to better performing CB-FPMs. The high financial and social cost associated with misclassifications (i.e. model error) thus makes adoption of BDA CB-FPMs very important for, among others, financiers, clients and policy makers.

A unified framework for big data acquisition, storage, and analytics for demand response management in smart cities

With an increased usage of information and communication technologies (ICT) in the smart cities, the data generated from different smart devices has increased manifolds. This data is heterogeneous in nature which varies with respect to time and exhibits the properties of all essential V’s for big data. Therefore, to handle such an enormous amount of data, various big data processing techniques are required. To cope up with these issues, this paper presents a tensor-based big data management technique to reduce the dimensionality of data gathered from the Internet-of-Energy (IoE) environment in a smart city. The core data is extracted out of the gathered data by using tensor operations such as-matricization, vectorization and tensorization with the help of higher-order singular value decomposition. This core data is then stored on the cloud in the reduced form. After reducing the dimensionality of data, it is used for providing many services in smart cities; and its application to provide demand response (DR) services has been discussed in this paper. For this purpose, support vector machine (SVM)-based classifier is used to classify the end-users (residential and commercial) into normal, overloaded and underloaded categories from the core data. Once such users are identified to take part in DR mechanism, utilities then generate commands to handle their DR in order to alter load requirements so that the overall load is optimized. Results obtained on Open Energy Information and PJM dataset clearly indicate the supremacy of the proposed tensor-based scheme over the traditional scheme for DR management.

GeoFog4Health: a fog-based SDI framework for geospatial health big data analysis

Spatial Data Infrastructure (SDI) is an important framework for sharing geospatial big data using the web. Integration of SDI with cloud computing led to emergence of Cloud-SDI as a tool for transmission, processing and analysis of geospatial data. Fog computing is a paradigm where embedded computers are employed to increase the throughput and reduce latency at the edge of the network. In this study, we developed and evaluated a Fog-based SDI framework named GeoFog4Health for mining analytics from geo-health big data. We built prototypes using Intel Edison and Raspberry Pi for studying the comparative performance. We conducted a case study on Malaria vector-borne disease positive maps of Maharastra state in India. The proposed framework had provision of lossless data compression for reduced data transfer. Also, overlay analysis of geospatial data was implemented. In addition, we discussed energy savings, cost analysis and scalability of the proposed framework with respect to efficient data processing. We compared the performance of the proposed framework with the state-of-the-art Cloud-SDI in terms of analysis time. Results and discussions showed the efficacy of the proposed system for enhanced analysis of geo-health big data generated from a variety of sensing frameworks.

Big Data Analytics based on PANFIS MapReduce

In this paper, a big data analytic framework is introduced for processing high-frequency data stream. This framework architecture is developed by combining an advanced evolving learning algorithm namely Parsimonious Network Fuzzy Inference System (PANFIS) with MapReduce parallel computation, where PANFIS has the capability of processing data stream in large volume. Big datasets are learnt chunk by chunk by processors in MapReduce environment and the results are fused by rule merging method, that reduces the complexity of the rules. The performance measurement has been conducted, and the results are showing that the MapReduce framework along with PANFIS evolving system helps to reduce the processing time around 22 percent in average in comparison with the PANFIS algorithm without reducing performance in accuracy.