Big Data Knowledge Discovery Research Articles

Sophisticated methods for noise filtering, subgroup discovery, and classification in big data analysis

Big Data is a popular research area where a vast amount of data is created, replicated, and consumed by society. The quality of the data used directly influences big data knowledge discovery. The existence of noise is the most prevalent problem influencing data quality. The following techniques were developed to reduce noise in data with a distributed setting: Homogenous Ensemble for Big Data (HME-BD) and Heterogeneous Ensemble for Big Data (HTE-BD). In this article, the performance of HTE-BD is improved further by developing Enhanced HTE-BD (EHTE-BD), which combines Logistic Regression based Support Vector Machine (LR-SVM) in conjunction with RF, LR, and KNN to reduce noisy data. Furthermore, the Multi-Objective Evolutionary Fuzzy Method for Subgroup Discovery throughout Big Data (MEFASD-BD) was used to resolve the multi-objective optimization challenge, and the Non-Dominated Sorting Genetic Algorithm-II (NSGA-II) was utilized to handle the rising dimensionality issue through subgroup discovery. To address the NSGA-II’s slow convergence rate, an Improved Multi-Objective Meta-Heuristic Fuzzy approach for discovering subgroups in big data is described, that contains a meta-heuristic method for subgroup discovery known as the Multi-Objective Differential Search Algorithm (MODSA). It selects the most relevant subgroups from vast amounts of data, reducing the data’s dimensionality. The Fuzzy Deep Neural Network (FDNN) classifier assesses the main subgroups. By removing noisy data and selecting the most relevant subgroups, the performance of FDNN in classifying vast amounts of data is improved.

Big Data Knowledge Discovery as a Service: Recent Trends and Challenges

Big Data Knowledge Discovery as a Service: Recent Trends and Challenges

Efficient Algorithms for Dynamic Incomplete Decision Systems

Attribute reduction is a crucial problem in the process of data mining and knowledge discovery in big data. In incomplete decision systems, the model using tolerance rough set is fundamental to solve the problem by computing the redact to reduce the execution time. However, these proposals used the traditional filter approach so that the reduct was not optimal in the number of attributes and the accuracy of classification. The problem is critical in the dynamic incomplete decision systems which are more appropriate for real-world applications. Therefore, this paper proposes two novel incremental algorithms using the combination of filter and wrapper approach, namely IFWA_ADO and IFWA_DEO, respectively, for the dynamic incomplete decision systems. The IFWA_ADO computes reduct incrementally in cases of adding multiple objects while IFWA_DEO updates reduct when removing multiple objects. These algorithms are also verified on six data sets. Experimental results show that the filter-wrapper algorithms get higher performance than the other filter incremental algorithms.

µBIGMSA-Microservice-Based Model for Big Data Knowledge Discovery: Thinking Beyond the Monoliths

Enterprise thrives on software applications that are built to fulfil the core business requirements. A single business application can offer a cluster of capabilities to generate value from processing huge amount of data often termed as Big Data. The time-based requirements of these applications are satisfied frequently by applying monolithic approaches with increased complexity and less scalability. Traditional approaches for Big Data Analytics suffer from overpriced, excessive and irrelevant data transfer owing to the constricted coupling amongst computing resources and data processing logic. Service-oriented approach came into existence as a new paradigm to enable applications to be rendered as service for better flexibility and scalability. Service orientation architecture avoids monolithic style but web services, one of its major implementation encourages monolith development of software application. Thus building a scalable, robust, resilient, cost-effective and optimum solution is one of the major requirements for outsized data. New software development style Microservices offer low degree of coupling and smaller size. This work reviews the existing and prevalent approaches like monolithic architecture in this area along with their drawbacks. This work also proposes a generic microservice model µBIGMSA for handling Knowledge Discovery in Big Data. Reference applications are implemented using proposed model. The effectiveness of the proposed model is evaluated by comparing the reference application with the monolithic application using various software metrics.

Big Data Knowledge Discovery Platforms: A 360 Degree Perspective

Big Datais a buzzword affecting nearly every domain and providing different set new opportunity for the development of knowledge discovery process. Although it comes with challengeslike abundance, extensiveness and diversity, timeliness and dynamism, messiness and vagueness, and with an uncertainty as all the data generated does not relates to any specific question and can be associated with another process or activity. To address these challenges are certainly cannot be handled by the traditional infrastructure, platforms and frameworks. New analytical techniques and high performance computing architecture came into picture to handle this explosion. These platforms and architecture are giving a cutting edge to the Big Data Knowledge Discovery process by using Artificial Intelligence, Machine Learning and Expert systems. This study encompasses a comprehensive review of Big Data analytical platforms and frameworks with their comparative analysis. A Knowledge Discovery architecture for Big Data Analytics is also proposed while considering the fundamental aspect of gaining insights from Big Data sets and focus of this analysis is to provide the open challenges associated with these techniques and future research directions.

Special Issue on Knowledge Discovery in Big Data (KDBD)

Special Issue on Knowledge Discovery in Big Data (KDBD)

Assessing reliability of Big Data Knowledge Discovery process

Extracting knowledge from Big Data is the process of transforming this data into actionable information. The exponential growth of data has initiated a myriad of new opportunities, and made data become the most valuable raw material of production for many organizations. Mining Big Data is coupled with some challenges, known as the 3V’s of Big Data: Volume, Variety and Velocity. However, a major challenge that needs to be addressed, and often is ignored in the literature, concerns reliability.Actually, data is agglomerated from multiple disparate sources, and each of Knowledge Discovery (KDD) process steps may be carried out by different organizations. These considerations lead us to ask a critical question that is weather the information we have at each step is reliable enough to proceed to the next one?This paper therefore aims to provide a framework that automatically assesses reliability of the knowledge discovery process. We focus on Linked Open Data (LOD) as a source of data, as it constitutes a relevant data provider in many Big Data applications. However, our framework can also be adapted for unstructured data. This framework will assist scientists to automatically and efficiently measure the reliability of each KDD process stage as well as detect unreliable steps that should be revised. Following this methodology, KDD process will be optimized and therefore produce knowledge with higher quality.

A Fast Projection-Based Algorithm for Clustering Big Data.

With the fast development of various techniques, more and more data have been accumulated with the unique properties of large size (tall) and high dimension (wide). The era of big data is coming. How to understand and discover new knowledge from these data has attracted more and more scholars' attention and has become the most important task in data mining. As one of the most important techniques in data mining, clustering analysis, a kind of unsupervised learning, could group a set data into objectives(clusters) that are meaningful, useful, or both. Thus, the technique has played very important role in knowledge discovery in big data. However, when facing the large-sized and high-dimensional data, most of the current clustering methods exhibited poor computational efficiency and high requirement of computational source, which will prevent us from clarifying the intrinsic properties and discovering the new knowledge behind the data. Based on this consideration, we developed a powerful clustering method, called MUFOLD-CL. The principle of the method is to project the data points to the centroid, and then to measure the similarity between any two points by calculating their projections on the centroid. The proposed method could achieve linear time complexity with respect to the sample size. Comparison with K-Means method on very large data showed that our method could produce better accuracy and require less computational time, demonstrating that the MUFOLD-CL can serve as a valuable tool, at least may play a complementary role to other existing methods, for big data clustering. Further comparisons with state-of-the-art clustering methods on smaller datasets showed that our method was fastest and achieved comparable accuracy. For the convenience of most scholars, a free soft package was constructed.

ProTraS: A probabilistic traversing sampling algorithm

In the process of knowledge discovery in big data, sampling is a technological brick that can be included in a more general framework to speed up existing algorithms and contribute to the scalability issue. Two challenging and connected problems arise with complexity: tuning and timing. ProTraS11A sample code is available at: http://frederic.rosresearch.free.fr/mydata/homepage/. is a new algorithm that fulfills both requirements. It is driven by a unique parameter, the sampling cost. The cost is overestimated by the maximum within group distance and the group cardinality. It is an iterative algorithm, at each step a new representative is added, chosen as the farthest-first traversal item from the representative in the group with the highest probability of cost reduction. The novel algorithm is robust to noise and time optimized. A detailed comparison with alternative algorithms, conducted on various synthetic and real world data sets, shows that the proposal yields competitive results in terms of quality of representation for clustering, sampling size and sampling time.

A general framework for big data knowledge discovery and integration

SummaryData structure description, conceptual modeling, and logic reasoning for knowledge discovery are three critical factors for the integration of information with heterogeneity. In particular, technologies of NoSQL databases and Internet of Things raise an urgent requirement for a uniform expression of heterogeneous data, and little attention has been paid to researches on the integration of NoSQL databases with traditional data models, as well as the semantic description of big data. To tackle these problems, in this paper, a concept‐and‐relation‐oriented grid data model called GODM model is first proposed based on the definitions of Monad, Compounder, Relation, etc. Then, the GODM model is utilized to uniformly describe traditional data models and NoSQL data models, which eliminates structure differences of heterogeneous data. Next, based on the GODM relation mechanism, an extendable semantic system is built up by choosing SHOIQ(D) description logic as the example to establish the correspondence with GODM grammar subset, providing a fundamental support for semantic integration and knowledge discovery of heterogeneous data. After that, comprehensive comparisons with GODM and other models are made, especially the distinctions between GODM and OWL on the aspects of relation mechanism, hybrid schema, description logic, grammatical constructors, etc. Besides, experimental evaluations and analyses on time and space efficiencies of some primary common data models are conducted after the proposal of a general evaluation model, with the results showing that the GODM model has great advantage on properties of expressiveness, flexibility, etc, particularly time and space efficiency. In summary, the GODM model describes heterogeneous data from both aspects of data structure and semantic relationship and realizes a hybrid schema reconciling the schemaful and schemaless data models, making it especially suitable for dynamic data integration and knowledge discovery from big data models.

Machine learning in pain research.

Machine learning in pain research.

Semantic genetic programming for fast and accurate data knowledge discovery

Big data knowledge discovery emerged as an important factor contributing to advancements in society at large. Still, researchers continuously seek to advance existing methods and provide novel ones for analysing vast data sets to make sense of the data, extract useful information, and build knowledge to inform decision making. In the last few years, a very promising variant of genetic programming was proposed: geometric semantic genetic programming. Its difference with the standard version of genetic programming consists in the fact that it uses new genetic operators, called geometric semantic operators, that, acting directly on the semantics of the candidate solutions, induce by definition a unimodal error surface on any supervised learning problem, independently from the complexity and size of the underlying data set. This property should improve the evolvability of genetic programming in presence of big data and thus makes geometric semantic genetic programming an extremely promising method for mining vast amounts of data. Nevertheless, to the best of our knowledge, no contribution has appeared so far to employ this new technology to big data problems. This paper intends to fill this gap. For the first time, in fact, we show the effectiveness of geometric semantic genetic programming on several complex real-life problems, characterized by vast amounts of data, coming from several different application domains.

NYSOL: A User-Centric Framework for Knowledge Discovery in Big Data

NYSOL: A User-Centric Framework for Knowledge Discovery in Big Data

Outlier Detection by Interaction with Domain Experts

We present a method for improving the detection of outlying Fire Service's reports based on domain knowledge and dialogue with Fire & Rescue domain experts. The outlying report is considered as an element which is significantly different from the remaining data. We follow the position of Professor Andrzej Skowron that effective algorithms in data mining and knowledge discovery in big data should incorporate an interaction with domain experts or/and be domain oriented. Outliers are defined and searched on the basis of domain knowledge and dialogue with experts. We face the problem of reducing high data dimensionality without loosing specificity and real complexity of reported incidents. We solve this problem by introducing a knowledge based generalization level intermediating between analyzed data and experts domain knowledge. In our approach we use the Formal Concept Analysis methods for both generation of the appropriate categories from data and as tools supporting communication with domain experts. We conducted two experiments in finding two types of outliers in which outlier detection was supported by domain experts.