Big Data Volume Research Articles

Efficient revocable CP-ABE for big data access control in cloud computing

Due to huge volume of big data, cloud is a better choice to store big data. Since the cloud is not trustworthy, privacy and access control is a big concern. Ciphertext policy attribute-based encryption (CP-ABE) is a promising technique to enable both privacy and access control in the cloud. However, directly applying CP-ABE scheme for big data in the cloud is a challenging task because of revocation. Existing CP-ABE with revocation schemes are lacking in efficiency. In this paper, we propose an efficient revocable CP-ABE (R-CP-ABE) scheme for big data access control in cloud using proxy-based updates in which the proxy server performs the ciphertext and secret key updates instead of data owner and data user respectively during revocation. This outsourced updates during revocation reduces the communication and computation overhead of data owner and data users. In security analysis, we prove that our R-CP-ABE scheme is secure against chosen plain-text and user collusion attacks. In addition, we also show that our scheme achieves forward and backward secrecy. The performance analysis demonstrates that our method is efficient when comparing with existing schemes.

Smart city application operation using machine learning algorithm

Machine learning is an up-and-coming area of research and its eventual application in smart cities are broadly recognized. The basic need of human being is increasing as they work together with diverse devices and also, they make available a lot of feedback. Many smart devices produce high volume of data and that can be retrieved and reviewed by humans. Applications are not permanent as it increases day to day life. Machine learning approach is the best adaptive solution for these data generated by different smart devices and smart city applications. Fast development in software, hardware with high speed internet connection provides big volume of data to this objective world. The explanation involvement of this paper is a machine learning application survey towards smart city.

A Tensor-Based Big-Data-Driven Routing Recommendation Approach for Heterogeneous Networks

Telecommunication networks are evolving toward a data-center-based architecture, which includes physical network functions, virtual network functions, as well as various types of management and orchestration systems. The primary purpose of this type of heterogeneous network is to provide efficient and convenient communication services for users. However, the diverse factors of a heterogeneous network such as bandwidth, delay, and communication protocol, bring great challenges for routing recommendations. In addition, the growing volume of big data and the explosive deployment of heterogeneous networks have started a new era of applying big data technologies to implement routing recommendations. In this article, a tensor-based big-data-driven routing recommendation framework, including the edge plane, fog plane, cloud plane, and application plane, is proposed. In this framework, a tensor-based, holistic, hierarchical approach is introduced to generate efficient routing paths using tensor decomposition methods. Also, a tensor matching method including the controlling tensor, seed tensor, and orchestration tensor is employed to realize routing recommendation. Finally, a case study is used to demonstrate the key processing procedures of the proposed framework.

Big Data Processing Technologies in Distributed Information Systems

The analysis of Big data technologies was provided. An example of MapReduce paradigm application, uploading of big volumes of data, processing and analyzing of unstructured information and its distribution into the clustered database was provided. The article summarizes the concept of "big data". Examples of methods for working with arrays of unstructured data are given. The parallel system Resilient Distributed Datasets (RDD) is organized. The class of basic database operations was realized: database con-nection, table creation, getting in line id, returning all elements of the database, update, delete and create the line.

Artificial Intelligence in Crime Counteraction, Prediction, Prevention and Evolution

Crime prediction, prevention and counteraction with the use of modern technologies should, according to the authors, become a priority task for the state, along with the development of economy, education, medicine and the enhancement of defense capacity. The article describes the concepts of «artificial intelligence», «machine learning», «big data», «deep learning», «neural networks» from the standpoint of how they are used both by criminals and by law enforcement bodies and courts. The authors examine the application of technologies which use artificial intelligence, hi tech crime (fishing, drones, fake information, bots, and so on). They outline modern software solutions based on artificial intelligence and aimed at counteracting crime: software that analyzes big volumes of data, processing of stream videos, facial recognition, contextual searching platforms, etc. The authors also describe the existing resources for predictive analytics (in particular, inter-agency experimental software «Artificial Intelligence in Police Work and Investigation of Criminal Offences»; software for recognizing people based on fragments of their tattoos; facial recognition of people after plastic surgeries in pictures and stream videos, with the generation of variants of their original appearance; platform of contextual intelligence Nigel; system Mayhem and others) and how they can be used to predict both crimes in general and individual criminal behavior. The authors also outline ethical dilemmas connected with legal decisions made by artificial intelligence regarding specific people. They present examples of using artificial intelligence for crime prevention (software COMPAS, criminal community’s psychometric prediction system, Harm Assessment Risk Tool, analytical software complex CEG, crime prediction system PredPol, ePOOLICE system, Palantir software, Russian system «Artificial intelligence»). They also outline the indicators of the early crime prevention system: indicators of matching, lagging, cyclical and counter-cyclical indicators. The authors state that Russia is lagging behind other countries in its use of artificial intelligence in law enforcement and suggest adopting the Modern Strategy of Crime Counteraction, Prediction and Prevention. Possible directions of this strategy are described.

A novel predicted replication strategy in cloud storage

Data replication is widely used in cloud storage and data grid to improve the parallel service efficiency and the performance of system, which can promote the file availability and system load balancing, reducing the response time with multiple copies. But high volume of big data gives a new, enormous and rigorous challenge to data storage and business access of cloud storage, specially to the quality of cloud services. In this paper, a novel dynamic predicted replication strategy (DPRS) combining with the access frequency of files and prediction method is proposed to predict the future access of each file and calculating the optional number of replicas based on the real access and future access periodically. The experiment results show that DPRS can availably decrease the response time of a file request and reduce the additional cost of the cloud storage system simultaneously.

TripImputor: Real-Time Imputing Taxi Trip Purpose Leveraging Multi-Sourced Urban Data

Travel behavior understanding is a long-standing and critically important topic in the area of smart cities. Big volumes of various GPS-based travel data can be easily collected, among which the taxi GPS trajectory data is a typical example. However, in GPS trajectory data, there is usually little information on travelers’ activities, thereby they can only support limited applications. Quite a few studies have been focused on enriching the semantic meaning for raw data, such as travel mode/purpose inferring. Unfortunately, trip purpose imputation receives relatively less attention and requires no real-time response. To narrow the gap, we propose a probabilistic two-phase framework named TripImputor , for making the real-time taxi trip purpose imputation and recommending services to passengers at their dropoff points. Specifically, in the first phase, we propose a two-stage clustering algorithm to identify candidate activity areas (CAAs) in the urban space. Then, we extract fine-granularity spatial and temporal patterns of human behaviors inside the CAAs from foursquare check-in data to approximate the priori probability for each activity, and compute the posterior probabilities (i.e., infer the trip purposes) using Bayes’ theorem. In the second phase, we take a sophisticated procedure that clusters historical dropoff points and matches the dropoff clusters and CAAs to immerse the real-time response. Finally, we evaluate the effectiveness and efficiency of the proposed two-phase framework using real-world data sets, which consist of road network, check-in data generated by over 38 000 users in one year, and the large-scale taxi trip data generated by over 19 000 taxis in a month in Manhattan, New York City, USA. Experimental results demonstrate that the system is able to infer the trip purpose accurately, and can provide recommendation results to passengers within 1.6 s in Manhattan on average, just using a single normal PC.

Exploring New Angles to Analyze Student Load Data

Over the past 25 years, performance measurement has gained salience in higher education, and with the explosion of structured data and the impact of business analytics and intelligence systems, there are new angles by which big volumes of data can be analyzed. Using traditional analytical approaches, pairs of reciprocal cohorts are considered as two separate discrete entities; therefore, basis of analysis are individual pairs of values, using statistical measures such as average, mean or median, of the total population. Missing in traditional approaches is a holistic performance measure in which the shape of the comparable cohorts is being compared to the overall cohort population (vector-based analysis). The purpose of this research is to examine shape analysis, using a Cosine similarity measure to distil new perspectives on performance measures in higher education. Cosine similarity measures the angle between the two vectors, regardless of the impact of their magnitude. Therefore, the more similar behavior of the two comparing entities can be interpreted as more similar orientation or smaller angle between the two vectors. The efficacy of the proposed method is experimented on the three Colleges of RMIT University from 2011 to 2016, and analyzes the shape of different cohorts. The current research also compared the performance of Cosine similarity with two other distance measures: Euclidean and Manhattan distance. The experimental results, using vector-based techniques, provide new insights to analyzing patterns of student load distribution and provide additional angles by orientation instead of magnitude / volume comparison.

QoE-driven big data management in pervasive edge computing environment

In the age of big data, services in the pervasive edge environment are expected to offer end-users better Quality-of-Experience (QoE) than that in a normal edge environment. However, the combined impact of the storage, delivery, and sensors used in various types of edge devices in this environment is producing volumes of high-dimensional big data that are increasingly pervasive and redundant. Therefore, enhancing the QoE has become a major challenge in high-dimensional big data in the pervasive edge computing environment. In this paper, to achieve high QoE, we propose a QoE model for evaluating the qualities of services in the pervasive edge computing environment. The QoE is related to the accuracy of high-dimensional big data and the transmission rate of this accurate data. To realize high accuracy of high-dimensional big data and the transmission of accurate data through out the pervasive edge computing environment, in this study we focused on the following two aspects. First, we formulate the issue as a high-dimensional big data management problem and test different transmission rates to acquire the best QoE. Then, with respect to accuracy, we propose a Tensor-Fast Convolutional Neural Network (TF-CNN) algorithm based on deep learning, which is suitable for high-dimensional big data analysis in the pervasive edge computing environment. Our simulation results reveal that our proposed algorithm can achieve high QoE performance.

Massive Volume of Unstructured Data and Storage Space Optimization- a Review

Nowadays the volume of digital data generated and used by enterprises is increasing at an enormous rate. The survey says that more than 80% of data that were generated in the last two years are unstructured in nature. Hence storage space requirement for storing this big volume of unstructured data is very high. It has gained attention to large-scale storage systems. Deduplication is a space efficient method mainly used to solve storage space optimization problem. This paper focuses on the effect of massive volume of unstructured data and review various storage optimization techniques and survey of various storage types. In addition, it elaborates specific challenges with regard to storage optimization using deduplication and technology that handles a huge amount of unstructured data.

Fine-Grained Big Traffic Data Reverse-charge System: A Method of Saving Expenses

The number of mobile users and the volume of big traffic data generated by user terminals are dramatically increased with diverse applications. Development in different economic and social sectors not only require higher network speed but also need low-cost on information exchanges. The market needs a complete set of solutions, which can provide a comprehensive management system for the operation and billing of a fine-grained data traffic system. This study addresses issues related to big traffic data reverse-charge for customers, which is a new type of consumption model that enables customers to save money with the cooperation of operators and online enterprises. We apply virtual private networking to identify apps, so as to help operators easily calculate the big traffic data of each app and reverse the related charges from the consumer to the relevant online enterprise. Online enterprises are willing to pay the expenses of some of their apps to enhance the user experience of customers exploring wireless services. To prove the popularity of our proposed system, we have experimentally employed the proposed reverse charge system, and participants’ responses to questionnaires indicate a win−win situation for customers as well as online enterprises. We hope that the proposed system will be implemented in practice, subject to customers, online enterprises, and operators agreeing to upgrade conventional charging infrastructures.

Lossless Pruned Naive Bayes for Big Data Classifications

Abstract In a fast growing big data era, volume and varieties of data processed in Internet applications drastically increase. Real-world search engines commonly use text classifiers with thousands of classes to improve relevance or data quality. These large scale classification problems lead to severe runtime performance challenges, so practitioners often resort to fast approximation techniques. However, the increase in classification speed comes at a cost, as approximations are lossy, mis-assigning classes relative to the original reference classification algorithm. To address this problem, we introduce a Lossless Pruned Naive Bayes (LPNB) classification algorithm tailored to real-world, big data applications with thousands of classes. LPNB achieves significant speed-ups by drawing on Information Retrieval (IR) techniques for efficient posting list traversal and pruning. We show empirically that LPNB can classify text up to eleven times faster than standard Naive Bayes on a real-world data set with 7205 classes, with larger gains extrapolated for larger taxonomies. In practice, the achieved acceleration is significant as it can greatly cut required computation time. In addition, it is lossless: the output is identical to standard Naive Bayes, in contrast to extant techniques such as hierarchical classification. The acceleration does not rely on the taxonomy structure, and it can be used for both hierarchical and flat taxonomies.

Improvement of air pollution prediction in a smart city and its correlation with weather conditions using metrological big data

Smart cities are an important concept for urban development. This concept addresses many current critical urban problems including traffic and environmental pollution. As utilization of the Internet of things and technology in smart cities increases, large volumes of big data are generated and collected by sensors embedded at different places in the city, which present a real-time display of what is happening throughout the city at all times. Such data should be processed and analyzed as a response to ensure effectiveness and improvement in quality of provided services; correct use and analysis of such data is valuable. Big data mining is the most effective method for analyzing such data. In this paper, aiming to increase speed and accuracy in predicting real levels of air pollution, its location, and effects of weather conditions on density of air pollution, a K-means clustering algorithm using the Mahout library is used as a big data mining tool on datasets of a city pulse project. Results of this study show that temperature, low air pressure, relative increase in moisture, and wind speed are causes of low pollution density at the cleanest point of the city. The SSE evaluation metric shows the high speed of this clustering method for big data, and results obtained from employing RMSE = 0.632 and MSE = 0.488 statistical measures indicate the high efficiency and accuracy of the proposed method in predicting air pollution.

A novel approach to probabilistic characterisation of neural firing patterns

BackgroundThe advances in extracellular neural recording techniques result in big data volumes that necessitate fast, reliable, and automatic identification of statistically similar units. This study proposes a single framework yielding a compact set of probabilistic descriptors that characterise the firing patterns of a single unit. New methodProbabilistic features are estimated from an inter-spike-interval time series, without assumptions about the firing distribution or the stationarity. The first level of proposed firing patterns decomposition divides the inter-spike intervals into bursting, moderate and idle firing modes, yielding a coarse feature set. The second level identifies the successive bursting spikes, or the spiking acceleration/deceleration in the moderate firing mode, yielding a refined feature set. The features are estimated from simulated data and from experimental recordings from the lateral prefrontal cortex in awake, behaving rhesus monkeys. ResultsAn efficient and stable partitioning of neural units is provided by the ensemble evidence accumulation clustering. The possibility of selecting the number of clusters and choosing among coarse and refined feature sets provides an opportunity to explore and compare different data partitions. ConclusionsThe estimation of features, if applied to a single unit, can serve as a tool for the firing analysis, observing either overall spiking activity or the periods of interest in trial-to-trial recordings. If applied to massively parallel recordings, it additionally serves as an input to the clustering procedure, with the potential to compare the functional properties of various brain structures and to link the types of neural cells to the particular behavioural states.

Semantic Web in the Age of Big Data: A Perspective: Will We Drown in a Data Tsunami or Enter a Knowledge Utopia?

We are awash with "Big Data" to this very day because of the technological advancements made during the past decade. The notion of Big Data [1] refers to the datasets which are gigantic in size to be processed by conventional databases and management techniques (volume), are extremely diverse so that no single data model can capture all elements of the data (variety) and are produced or gathered at an unprecedented scale (velocity) [1]. Because of this sheer volume, variety, and velocity of big data, enterprises are facing data heterogeneity, diversity and complexity challenges. However, this big data era came with big opportunities by resolving the associated challenges, so it could transform our traditional way of decision-making. Enterprises with the technical expertise of managing big data are now replacing their usual guesswork and laborious legacy data modeling based decision making processes with facts derived from big data [2].

Cloud and IoT based disease prediction and diagnosis system for healthcare using Fuzzy neural classifier

Recently, the mobile health care (m-healthcare) applications with Internet of Things (IoT) are providing the various dimensionalities and the online services. These applications have provided a new platform to the millions of people for getting benefit over the health tips frequently for living a healthy life. After the introduction of IoT technology and the related devices which are used in medical field, strengthened the various features of these healthcare online applications. The huge volume of big data is generated by IoT devices in healthcare environment. Cloud computing technology is used to handle the large volume of data and also provide the ease of use. In this scenario, cloud based applications are playing major role in this fast world. These medical applications are also used the Cloud Computing technology for secured storage and accessibility. For availing better services to the people over the online healthcare applications, we propose a new Cloud and IoT based Mobile Health care application for monitoring and diagnosing the serious diseases. Here, a new framework is developed for the public. In this work, a new systematic approach is used for the diabetes diseases and the related medical data is generated by using the UCI Repository dataset and the medical sensors for predicting the people who has affected with diabetes severely. In addition, we propose a new classification algorithm called Fuzzy Rule based Neural Classifier for diagnosing the disease and the severity. The experiments have been conducted by the standard UCI Repository dataset and the real health records which are collected from various hospitals. The experimental results show that the performance of the proposed work which outperforms the existing systems for disease prediction.

IoT-Driven Automated Object Detection Algorithm for Urban Surveillance Systems in Smart Cities

Automated object detection algorithm is an important research challenge in intelligent urban surveillance systems for Internet of Things (IoT) and smart cities applications. In particular, smart vehicle license plate recognition and vehicle detection are recognized as core research issues of these IoT-driven intelligent urban surveillance systems. They are key techniques in most of the traffic related IoT applications, such as road traffic real-time monitoring, security control of restricted areas, automatic parking access control, searching stolen vehicles, etc. In this paper, we propose a novel unified method of automated object detection for urban surveillance systems. We use this novel method to determine and pick out the highest energy frequency areas of the images from the digital camera imaging sensors, that is, either to pick the vehicle license plates or the vehicles out from the images. Our proposed method can not only help to detect object vehicles rapidly and accurately, but also can be used to reduce big data volume needed to be stored in urban surveillance systems.

ADVANCED TECHNOLOGIES OF BIG DATA RESEARCH IN DISTRIBUTED INFORMATION SYSTEMS

Context. Considered question correct interpretation information flow in distributed information systems. The object of study methods are promotion “big data” on cluster system.Objective. Is the study promising areas and technology for the analysis of structures data in distributed information systems.Method. The big data tendency prospects as well as timeliness of the problem are studied in this paper. The principles of work with them are addressed. Big data processing technologies are provided. The analysis of each one is performed. An example of “MapReduce” paradigm application, uploading of big volumes of data, processing and analyzing of unstructured information and its distribution into the clustered database is provided. The article summarizes the concept of “big data”. Examples of methods for working with arrays of unstructured data. Dedicated scientific guidance for analyzing big data. The principles of unstructured data in distributed information systems. Driven work platform “Hadoop MapReduce” and “Apache Spark”. Analyzed their properties and given the differences. An analysis of comparative performance against both platforms – the performance of the number of iterations. Consider ways to create RDD: parallelization transmitted collection program and a link to an external file system in “Hadoop”. There is an example rozparalelenoyi system RDD. Proposed work lone class for basic database operations: database connection, create a table, a table, get in line id, returning all elements of the database, update, delete and create the line.Results. The analysis Models Spark and Hadoop MapReduce for phased construction distributed information system. built up SparkConf object, containing information about applique and is the final version of the experiment.Conclusions. Conducted experiment confirmed efficiency the proposed method, are capable process horizontal data arrays, that parallelization by defective presentation of information. These promising areas of analyze structure data for the purpose of forecast results and create algorithms advanced correlation, contributing new understanding activity distributed information systems further research can consist in wide use information systems, that would provide a full range technological process adaptation information flows in clusters.

About a method for compressing x-ray computed microtomography data

The management of scientific data is of high importance especially for experimental techniques that produce big data volumes. Such a technique is x-ray computed tomography (CT) and its community has introduced advanced data formats which allow for better management of experimental data. Rather than the organization of the data and the associated meta-data, the main topic on this work is data compression and its applicability to experimental data collected from a synchrotron-based CT beamline at the Elettra-Sincrotrone Trieste facility (Italy) and studies images acquired from various types of samples. This study covers parallel beam geometry, but it could be easily extended to a cone-beam one. The reconstruction workflow used is the one currently in operation at the beamline. Contrary to standard image compression studies, this manuscript proposes a systematic framework and workflow for the critical examination of different compression techniques and does so by applying it to experimental data. Beyond the methodology framework, this study presents and examines the use of JPEG-XR in combination with HDF5 and TIFF formats providing insights and strategies on data compression and image quality issues that can be used and implemented at other synchrotron facilities and laboratory systems. In conclusion, projection data compression using JPEG-XR appears as a promising, efficient method to reduce data file size and thus to facilitate data handling and image reconstruction.

QoE-Driven Big Data Architecture for Smart City

In the era of big data, the applications/services of the smart city are expected to offer end users better QoE than in a conventional smart city. Nevertheless, various types of sensors will produce an increasing volume of big data along with the implementation of a smart city, where we face redundant and diverse data. Therefore, providing satisfactory QoE will become the major challenge in the big-data-based smart city. In this article, to enhance the QoE, we propose a novel big data architecture consisting of three planes: the data storage plane, the data processing plane, and the data application plane. The data storage plane stores a wide variety of data collected by sensors and originating from different data sources. Then the data processing plane filters, analyzes, and processes the ocean of data to make decisions autonomously for extracting high-quality information. Finally, the application plane initiates the execution of the events corresponding to the decisions delivered from the data processing plane. Under this architecture, we particularly use machine learning techniques, trying to acquire accurate data and deliver precise information to end users. Simulation results indicate that our proposals could achieve high QoE performance for the smart city.