Big Data Scenarios Research Articles

A Streaming model for Generalized Rayleigh with extension to Minimum Noise Fraction.

The Rayleigh quotient optimization is the maximization of a rational function, or a max-min problem, with simultaneous maximization of the numerator function and minimization of the denominator function. Here, we describe a low-rank, streaming solution for Rayleigh quotient optimization applicable for big-data scenarios where the data matrix is too large to be fully loaded into main memory. We apply this for a maximization of the Signal to Noise ratio of big-data, of very large static and dynamic data. Our implementation is shown to achieve faster processing time compared to a standard data read into memory. We demonstrate the trade-offs with synthetic and real data, on different scales to validate the approach in terms of accuracy, speed and storage.

Experimental Evaluation of Open Source Data Mining Tools: R, Rapid Miner and KNIME

In the current scenario of Big Data, open source Data Mining tools are very popular in business data analytics. The paper presents a comprehensive study of three most popular open source data mining tools – R, RapidMiner and KNIME. The tools are compared by implementing them on two real datasets. Performance is evaluated by creating a decision tree of the datasets taken. Our objective is to find the best tool for classification. The study can help researchers, developers and users in selecting a tool for accuracy in their data analysis and prediction. Experiments depict that accuracy level of the tool changes with the quantity and quality of the dataset. The results show that RapidMiner is the best tool followed by KNIME and R.

A Framework and Use Cases for Big Data in Procurement

Big data has become a buzzword that is broadly discussed in media and board rooms alike. However, in the procurement function there are few visible practical applications of a full-fledged big data scenario so far. Some academic research has been conducted as well as several practitioners' surveys, but insights on how procurement can profit from big data applications are very fragmented. Therefore, the following article portrays a conceptual framework for big data applications in the procurement function connecting different data sources, skilled teams, and big data analytics. Furthermore, a full array of existing and new use cases for procurement is derived.

Even big data is not enough: need for a novel reference modelling for forensic document authentication

With the emergence of big data, deep learning (DL) approaches are becoming quite popular in many branches of science. Forensic science is no longer an exception. However, there are certain problems in forensic science where the solutions would hardly benefit from the recent advances in DL algorithms. Document authentication is one such problem where we can have many reference samples, and with the big data scenario probably we would have even more number of reference samples but number of defective or forged samples will remain an issue. Experts often encounter situations where there is no or hardly a scanty number of forged samples available. In such situation, employment of data-hungry algorithms would be inefficient as they will not be able to learn the forged samples properly. This paper addresses this problem and proposes a novel reference modelling framework for forensic document authentication. The approach is based on Mahalanobis space. Two questioned document examination problems have been studied to show the effectiveness of our reference modelling algorithm which has also been compared to a commonly used learning approach, namely neural network-based classification.

From Big to Smart Data: Iterative ensemble filter for noise filtering in Big Data classification

The quality of the data is directly related to the quality of the models drawn from that data. For that reason, many research is devoted to improve the quality of the data and to amend errors that it may contain. One of the most common problems is the presence of noise in classification tasks, where noise refers to the incorrect labeling of training instances. This problem is very disruptive, as it changes the decision boundaries of the problem. Big Data problems pose a new challenge in terms of quality data due to the massive and unsupervised accumulation of data. This Big Data scenario also brings new problems to classic data preprocessing algorithms, as they are not prepared for working with such amounts of data, and these algorithms are key to move from Big to Smart Data. In this paper, an iterative ensemble filter for removing noisy instances in Big Data scenarios is proposed. Experiments carried out in six Big Data datasets have shown that our noise filter outperforms the current state-of-the-art noise filter in Big Data domains. It has also proved to be an effective solution for transforming raw Big Data into Smart Data.

Enabling Computational Storage Through FPGA Neural Network Accelerator for Enterprise SSD

Computational storage is an emerging concept in big data scenario where the demand to process ever-growing storage workloads is outpacing traditional compute server architectures. To enable this paradigm there is a call for developing accelerators that off-load some of the management routines that are usually demanded to the smartness inside the storage. For enterprise solid-state drives (SSD) this translates into a dedicated hardware that exploits the interconnection fabric of the host with the goal of improving SSD reliability/performance. In this brief, we have developed an field-programmable gate array-based neural network accelerator for the moving read reference shift prediction in enterprise SSD. The accelerator high prediction accuracy (up to 99.5%), low latency (6.5 $\mu \text{s}$ per prediction), and low energy consumption (19.5 $\mu \text{J}$ ) opens up unprecedented usage models in the storage environment.

The impact of columnar file formats on SQL‐on‐hadoop engine performance: A study on ORC and Parquet

SummaryColumnar file formats provide an efficient way to store data to be queried by SQL‐on‐Hadoop engines. Related works consider the performance of processing engine and file format together, which makes it impossible to predict their individual impact. In this work, we propose an alternative approach: by executing each file format on the same processing engine, we compare the different file formats as well as their different parameter settings. We apply our strategy to two processing engines, Hive and SparkSQL, and evaluate the performance of two columnar file formats, ORC and Parquet. We use BigBench (TPCx‐BB), a standardized application‐level benchmark for Big Data scenarios. Our experiments confirm that the file format selection and its configuration significantly affect the overall performance. We show that ORC generally performs better on Hive, whereas Parquet achieves best performance with SparkSQL. Using ZLIB compression brings up to 60.2% improvement with ORC, while Parquet achieves up to 7% improvement with Snappy. Exceptions are the queries involving text processing, which do not benefit from using any compression.

A Hierarchical Dimension Reduction Approach for Big Data with Application to Fault Diagnostics

Abstract About four zetta bytes of data, which falls into the category of big data, is generated by complex manufacturing systems annually. Big data can be utilized to improve the efficiency of an aging manufacturing system, provided, several challenges are handled. In this paper, a novel methodology is presented to detect faults in manufacturing systems while overcoming some of these challenges. Specifically, a generalized distance measure is proposed in conjunction with a novel hierarchical dimension reduction (HDR) approach. It is shown that the HDR can tackle challenges that are frequently observed during distance calculation in big data scenarios, such as norm concentration, redundant dimensions, and a non-invertible correlation matrices. Subsequently, a probabilistic methodology is developed for isolation and detection of faults. Here, Edgeworth expansion based expressions are derived to approximate the density function of the data. The performance of the dimension reduction methodology is demonstrated to be efficient with simulation results involving the use of big data sets. It is shown that HDR is able to explain almost 90% of the total information. Furthermore, the proposed dimension reduction methodology is seen to outperform standard dimension reduction approaches and is able to improve the performance of standard classification methodologies in high dimensional scenarios.

Semantic-aware data quality assessment for image big data

Data quality (DQ) assessment is essential for realizing the promise of big data by judging the value of data in advance. Relevance, an indispensable dimension of DQ, focusing on “fitness for requirement”, can arouse the user’s interest in exploiting the data source. It has two-level evaluations: (1) the amount of data that meets the user’s requirements; (2) the matching degree of these relevant data. However, there lack works of DQ assessment at dimension of relevance, especially for unstructured image data which focus on semantic similarity. When we try to evaluate semantic relevance between an image data source and a query (requirement), there are three challenges: (1) how to extract semantic information with generalization ability for all image data? (2) how to quantify relevance by fusing the quantity of relevant data and the degree of similarity comprehensively? (3) how to improve assessing efficiency of relevance in a big data scenario by design of an effective architecture?To overcome these challenges, we propose a semantic-aware data quality assessment (SDQA) architecture which includes off-line analysis and on-line assessment. In off-line analysis, for an image data source, we first transform all images into hash codes using our improved Deep Self-taught Hashing (IDSTH) algorithm which can extract semantic features with generalization ability, then construct a graph using hash codes and restricted Hamming distance, next use our designed Semantic Hash Ranking (SHR) algorithm to calculate the importance score (rank) for each node (image), which takes both the quantity of relevant images and the degree of semantic similarity into consideration, and finally rank all images in descending order of score. During on-line assessment, we first convert the user’s query into hash codes using IDSTH model, then retrieve matched images to collate their importance scores, and finally help the user determine whether the image data source is fit for his requirement. The results on public dataset and real-world dataset show effectiveness, superiority and on-line efficiency of our SDQA architecture.

ENERGY-AWARE SECURITY ROUTING PROTOCOL FOR WSN IN BIG-DATA APPLICATIONS

The dense deployment of the wireless sensor networks has caused enormous data flow leading to a Big-Data generation. To enable a continuous transmission for the huge volume of data packets, it becomes necessary to readapt the routing protocol to facilitate the routing in handling the Big-data scenario. Since energy is the main constraint in the wireless sensor network, as the sensor are battery powered, and the routing methods consuming enormous of the energy for route discovery thereby reducing the network life time, many conventional methods were developed to address the problem of energy consumption with the, increased network latency, overhead, security issues and delay. So the paper proposes the energy-aware routing protocol that could handle the security issues, arising in the enormous data generation satisfying the QOS requirements. The node with substantial resources are selected to overcome the problems of energy consumption and the network life time handling the security issues , delay, network latency and the overhead problem. Further validation of the proposed method in the Network Simulator-3 is performed to evaluate the network latency, packet delivery ratio, throughput, power consumption, network lifetime and Cost.

Benchmarking framework for class imbalance problem using novel sampling approach for big data

The traditional techniques of machine learning always need to be strengthened for dealing with cosmic nature of big data for systematic and methodical learning. The unbalanced distribution of classes in big data, popularly known as imbalanced big data chases the problem of learning to a much higher level. The conventional methods are being progressively modified to handle and curtail the problem of learning from imbalanced datasets in the context of big data at the data level and algorithmic level. In the current study, a cluster heads based data level sampling solution which inherits edge of K-Means and Fuzzy C-Means clustering approaches is applied. The proposed approach is evaluated with three different classifiers namely Support Vector Machines, Decision Tree and k-Nearest Neighbor and compared with conventional SMOTE algorithm. The experiment has shown promising results with an increment of 8.09% and 35.71% in terms of accuracy and AUC respectively, for all imbalanced datasets. This work imparts a baseline comparison of solutions for imbalanced classification at data level in big data scenario and proposes an efficient clustering-based solution for same.

Efficient Attribute-Based Encryption with Privacy-Preserving Key Generation and Its Application in Industrial Cloud

Due to the rapid development of new technologies such as cloud computing, Internet of Things (IoT), and mobile Internet, the data volumes are exploding. Particularly, in the industrial field, a large amount of data is generated every day. How to manage and use industrial Big Data primely is a thorny challenge for every industrial enterprise manager. As an emerging form of service, cloud computing technology provides a good solution. It receives more and more attention and support due to its flexible configuration, on-demand purchase, and easy maintenance. Using cloud technology, enterprises get rid of the heavy data management work and concentrate on their main business. Although cloud technology has many advantages, there are still many problems in terms of security and privacy. To protect the confidentiality of the data, the mainstream solution is encrypting data before uploading. In order to achieve flexible access control to encrypted data, attribute-based encryption (ABE) is an outstanding candidate. At present, more and more applications are using ABE to ensure data security. However, the privacy protection issues during the key generation phase are not considered in the current ABE systems. That is to say, the key generation center (KGC) knows both of attributes and corresponding keys of each user. This problem is especially serious in the industrial big data scenario, because it will cause great damage to the business secrets of industrial enterprises. In this paper, we design a new ABE scheme that protects user’s privacy during key issuing. In our new scheme, we separate the functionality of attribute auditing and key generating to ensure that the KGC cannot know user’s attributes and that the attribute auditing center (AAC) cannot obtain the user’s secret key. This is ideal for many privacy-sensitive scenarios, such as industrial big data scenario.

Entropic Statistical Description of Big Data Quality in Hotel Customer Relationship Management.

Customer Relationship Management (CRM) is a fundamental tool in the hospitality industry nowadays, which can be seen as a big-data scenario due to the large amount of recordings which are annually handled by managers. Data quality is crucial for the success of these systems, and one of the main issues to be solved by businesses in general and by hospitality businesses in particular in this setting is the identification of duplicated customers, which has not received much attention in recent literature, probably and partly because it is not an easy-to-state problem in statistical terms. In the present work, we address the problem statement of duplicated customer identification as a large-scale data analysis, and we propose and benchmark a general-purpose solution for it. Our system consists of four basic elements: (a) A generic feature representation for the customer fields in a simple table-shape database; (b) An efficient distance for comparison among feature values, in terms of the Wagner-Fischer algorithm to calculate the Levenshtein distance; (c) A big-data implementation using basic map-reduce techniques to readily support the comparison of strategies; (d) An X-from-M criterion to identify those possible neighbors to a duplicated-customer candidate. We analyze the mass density function of the distances in the CRM text-based fields and characterized their behavior and consistency in terms of the entropy and of the mutual information for these fields. Our experiments in a large CRM from a multinational hospitality chain show that the distance distributions are statistically consistent for each feature, and that neighbourhood thresholds are automatically adjusted by the system at a first step and they can be subsequently more-finely tuned according to the manager experience. The entropy distributions for the different variables, as well as the mutual information between pairs, are characterized by multimodal profiles, where a wide gap between close and far fields is often present. This motivates the proposal of the so-called X-from-M strategy, which is shown to be computationally affordable, and can provide the expert with a reduced number of duplicated candidates to supervise, with low X values being enough to warrant the sensitivity required at the automatic detection stage. The proposed system again encourages and supports the benefits of big-data technologies in CRM scenarios for hotel chains, and rather than the use of ad-hoc heuristic rules, it promotes the research and development of theoretically principled approaches.

Fast kernel extreme learning machine for ordinal regression

Ordinal regression is a special kind of machine learning problem, which aims to label patterns with an ordinal scale. Due to the ubiquitous existence of the ordering information in many practical cases, ordinal regression has received much attention and can be found in a great variety of applications. Meanwhile, Kernel Extreme Learning Machine (KELM), as the extension of Extreme Learning Machine in the framework of kernel learning, has shown its strength in many machine learning tasks. Nevertheless, existing KELM methods have paid little attention to ordinal regression especially in large-scale situations. In this paper, we consider the kernel technique and ordinal scales in labels at the same time, and propose a new KELM model for ordinal regression by exploiting a quadratic cost-sensitive encoding scheme. In order to make the training process more efficient in the big data scenario, a fast algorithm is designed based on the low rank approximation. The incomplete Cholesky factorization and the Sherman–Morrison–Woodbury formula are utilized together to avoid computing the inverse of the kernel matrix. The time complexity of the new algorithm is thus linear with the number of the training instances, which makes it suitable to the large-scale occasions. Numerical experiments on multiple public datasets validate the effectiveness and efficiency of the proposed methods.

NR‐IQA for noise‐affected images using singular value decomposition

This study presents an efficient no-reference image quality assessment (NR-IQA) technique to assess the quality of images affected by noise. The proposed technique is based on two characteristics of the human eye (retina), namely the presence of centre-surround receptive field and visualisation utilising different spatial frequency channels. In the proposed technique, the authors model centre-surround receptive field using difference of Gaussians (DoG), whereas to mimic multiple frequencies in the centre-surround receptive field, they compute multiple DoG images of different values of standard deviations generated for different frequencies. Furthermore, the singular value decomposition-based features are obtained from the generated DoG images to estimate the image quality. The proposed technique does not require any training, neither based on distorted/original images nor based on subjective human scores, to assess the image quality. The performance of the proposed technique is being analysed on LIVE, TID08, CSIQ and SD-IVL databases and it shows that the proposed technique outperforms recently proposed NR and no-training/training-based IQA techniques. Experimental validation of the proposed technique in the big-data scenario of 10,000 noisy images also shows encouraging results.

A Big Data implementation of the MANTIS reference architecture for predictive maintenance

This article presents the implementation of a reference architecture for cyber-physical systems to support condition-based maintenance of industrial assets. It also focuses on describing the data analysis approach to manage predictive maintenance of clutch-brake assets fleet over the previously defined MANTIS reference architecture. Proposals for both the architecture and data analysis implementation support working on Big Data scenarios, due to the usage of related technologies, such as Hadoop Distributed File System, Kafka or Apache Spark. The techniques are (1) root cause analysis powered by attribute-oriented induction clustering and (2) remaining useful life powered by time series forecasting. The work has been conducted in a real use case within the H2020 European project MANTIS.

Evolutionary Design of Linguistic Fuzzy Regression Systems with Adaptive Defuzzification in Big Data Environments

This paper is positioned in the area of the use of cognitive computation techniques to design intelligent systems for big data scenarios, specifically the use of evolutionary algorithms to design data-driven linguistic fuzzy rule-based systems for regression and control. On the one hand, data-driven approaches have been extensively employed to create rule bases for fuzzy regression and control from examples. On the other, adaptive defuzzification is a well-known mechanism used to significantly improve the accuracy of fuzzy systems. When dealing with large-scale scenarios, the aforementioned methods must be redesigned to allow scalability. Our proposal is based on a distributed MapReduce schema, relying on two ideas: first, a simple adaptation of a classic data-driven method to quickly obtain a set of rules, and, second, a novel scalable strategy that uses evolutionary adaptive defuzzification to achieve better behavior through cooperation among rules. Some different regression problems were used to validate our methodology through an experimental study developed and included at the end of our paper. Therefore, the proposed approach allows scalability while tackling applications of linguistic fuzzy rule-based systems for regression with adaptive defuzzification in large-scale data scenarios. This paper thus examines the use of some relevant techniques for cognitive computing when working with a vast volume of examples, a common occurrence when dealing with the design of artificial intelligent systems that perform reasoning in a similar way as humans.

Heterogeneous Distributed Big Data Clustering on Sparse Grids

Clustering is an important task in data mining that has become more challenging due to the ever-increasing size of available datasets. To cope with these big data scenarios, a high-performance clustering approach is required. Sparse grid clustering is a density-based clustering method that uses a sparse grid density estimation as its central building block. The underlying density estimation approach enables the detection of clusters with non-convex shapes and without a predetermined number of clusters. In this work, we introduce a new distributed and performance-portable variant of the sparse grid clustering algorithm that is suited for big data settings. Our computed kernels were implemented in OpenCL to enable portability across a wide range of architectures. For distributed environments, we added a manager–worker scheme that was implemented using MPI. In experiments on two supercomputers, Piz Daint and Hazel Hen, with up to 100 million data points in a ten-dimensional dataset, we show the performance and scalability of our approach. The dataset with 100 million data points was clustered in 1198 s using 128 nodes of Piz Daint. This translates to an overall performance of 352 TFLOPS . On the node-level, we provide results for two GPUs, Nvidia’s Tesla P100 and the AMD FirePro W8100, and one processor-based platform that uses Intel Xeon E5-2680v3 processors. In these experiments, we achieved between 43% and 66% of the peak performance across all computed kernels and devices, demonstrating the performance portability of our approach.

An efficient and secure privacy preserving multi-cloud storage framework for mobile devices

ABSTRACT In today's scenario of Big Data, different mobile devices like phone, tablets are used to access data stored in the cloud storage services as Media Fire, G Drive, DropBox, etc. Regardless the cloud storage services are much accessible; there is a need to focus on the issues related to resource consumption when these cloud storage services used on resource-constrained mobile devices. The demand for smart mobile devices is increasing rapidly because users want to access their data anywhere anytime. With this, the need for more storage capacity is also growing to store users personal data. Cloud storage is an answer to fulfill storage demand of mobile devices. With the advantage of cloud storage service, it also consumes a lot of mobile device resources as processing (CPU), data (usage of the Internet), and battery and provides limited free storage. However, most users are eager for freely available storage. Different multi-cloud storage services are available to increase the freely available storage for users, but at the same time they are not secure and consuming a lot of mobile device resources. So efficient and secure privacy-preserving multi-cloud storage framework required for resource-constrained mobile devices.

ETL Framework for Real-Time Business Intelligence over Medical Imaging Repositories

In the last decades, the amount of medical imaging studies and associated metadata has been rapidly increasing. Despite being mostly used for supporting medical diagnosis and treatment, many recent initiatives claim the use of medical imaging studies in clinical research scenarios but also to improve the business practices of medical institutions. However, the continuous production of medical imaging studies coupled with the tremendous amount of associated data, makes the real-time analysis of medical imaging repositories difficult using conventional tools and methodologies. Those archives contain not only the image data itself but also a wide range of valuable metadata describing all the stakeholders involved in the examination. The exploration of such technologies will increase the efficiency and quality of medical practice. In major centers, it represents a big data scenario where Business Intelligence (BI) and Data Analytics (DA) are rare and implemented through data warehousing approaches. This article proposes an Extract, Transform, Load (ETL) framework for medical imaging repositories able to feed, in real-time, a developed BI (Business Intelligence) application. The solution was designed to provide the necessary environment for leading research on top of live institutional repositories without requesting the creation of a data warehouse. It features an extensible dashboard with customizable charts and reports, with an intuitive web-based interface that empowers the usage of novel data mining techniques, namely, a variety of data cleansing tools, filters, and clustering functions. Therefore, the user is not required to master the programming skills commonly needed for data analysts and scientists, such as Python and R.