Mining Maximal Frequent Itemsets Research Articles

Automated data mining framework for building energy conservation aided by generative pre-trained transformers (GPT)

Data mining technologies have showed promising capabilities in extracting building operation patterns from massive amounts of building operational data for energy conservation. However, the number of the extracted operation patterns is always large. It is time-consuming and tedious for users to find valuable operation patterns among them. To overcome this barrier, this study proposes an automated data mining framework based on maximal frequent itemset mining and generative pre-trained transformers (GPT). An improved maximal frequent itemset mining-based data mining method is developed to extract non-redundant operation patterns from numerous building operational data for reducing the number of the extracted operation patterns. A template-based prompt generation method is proposed to transform the extracted operation patterns into prompts. The prompts are inputted into GPT to determine whether there are energy waste patterns hidden in the extracted operation patterns. It liberates humans from the tedious work on analyzing the extracted operation patterns. The framework is applied to analyze the one-year operational data from a real-world building chiller plant system for verifying its performance. Most of the energy waste patterns in this system are detected successfully, such as valve faults, low chilled water outlet temperature, small chilled and cooling water temperature differences, and improper coordinated control among devices. The detection accuracy of GPT is 89.17 % for energy waste patterns and 99.48 % for normal operation patterns. The response time and cost of GPT are 6747.60 s and $17.68, respectively.

Unsupervised Learning for Feature Selection: A Proposed Solution for Botnet Detection in 5G Networks

The world has seen exponential growth in deploying Internet of Things (IoT) devices. In recent years, connected IoT devices have surpassed the number of connected non-IoT devices. The number of IoT devices continues to grow and they are becoming a critical component of the national infrastructure. IoT devices' characteristics and inherent limitations make them attractive targets for hackers and cyber criminals. Botnet attack is one of the serious threats on the Internet today. This article proposes pattern-based feature selection methods as part of a machine learning (ML)-based botnet detection system. Specifically, two methods are proposed: the first is based on the most dominant pattern feature values and the second is based on maximal frequent itemset mining. The proposed feature selection method uses Gini impurity and an unsupervised clustering method to select the most influential features automatically. The evaluation results show that the proposed methods have improved the performance of the detection system. The developed system has a true positive rate of 100% and a false positive rate of 0% for best performing models. In addition, the proposed methods reduce the computational cost of the system as evidenced by the detection speed of the system.

An Incremental Interesting Maximal Frequent Itemset Mining Based on FP‐Growth Algorithm

Frequent itemset mining is the most important step of association rule mining. It plays a very important role in incremental data environments. The massive volume of data creates an imminent need to design incremental algorithms for the maximal frequent itemset mining in order to handle incremental data over time. In this study, we propose an incremental maximal frequent itemset mining algorithms that integrate subjective interestingness criterion during the process of mining. The proposed framework is designed to deal with incremental data, which usually come at different times. It extends FP‐Max algorithm, which is based on FP‐Growth method by pushing interesting measures during maximal frequent itemset mining, and performs dynamic and early pruning to leave uninteresting frequent itemsets in order to avoid uninteresting rule generation. The framework was implemented and tested on public databases, and the results found are promising.

CL-MAX: a clustering-based approximation algorithm for mining maximal frequent itemsets

The problem of frequent itemset mining is one of the more important problems in data mining which has been extensively employed across a wide range of other relevant tasks such as market basket analysis in marketing, or text analysis in text mining applications. The majority of the deterministic frequent itemset mining algorithms which have been proposed in recent years use some sort or another of an optimal data structures to reduce the overall execution time of the algorithm. In this paper, however, we have tried instead to introduce an approximation algorithm which works by converting the problem into a clustering problem where similar transactions are grouped together. Each cluster centroid represents an itemset which may be assumed to be a candidate frequent itemsets. The validity of this assumption is simply verified by calculating the support count of these itemsets. Those who meet the min-support condition are considered to be an actual frequent itemset. As for the remaining itemsets, they are then passed to MAFIA which extract all maximal frequent itemsets therefrom. Experimentations made on several well-known and diverse datasets show that the proposed algorithm performs almost always faster, and in some cases up to 10 times faster, than the existing deterministic algorithms, and all this by retaining up to 95% of its accuracy.

K-PbC: an improved cluster center initialization for categorical data clustering

The performance of a partitional clustering algorithm is influenced by the initial random choice of cluster centers. Different runs of the clustering algorithm on the same data set often yield different results. This paper addresses that challenge by proposing an algorithm named k-PbC, which takes advantage of non-random initialization from the view of pattern mining to improve clustering quality. Specifically, k-PbC first performs a maximal frequent itemset mining approach to find a set of initial clusters. It then uses a kernel-based method to form cluster centers and an information-theoretic based dissimilarity measure to estimate the distance between cluster centers and data objects. An extensive experimental study was performed on various real categorical data sets to draw a comparison between k-PbC and state-of-the-art categorical clustering algorithms in terms of clustering quality. Comparative results have revealed that the proposed initialization method can enhance clustering results and k-PbC outperforms compared algorithms for both internal and external validation metrics.

An efficient approach based on selective partitioning for maximal frequent itemsets mining

We present a maximal frequent itemset (MFI) mining algorithm based on selective partitioning called SelPMiner. It makes use of a novel data format named Itemset-count tree—a compact and optimized representation in the form of partition that reduces memory requirement. It also does selective partitioning of the database, which reduces runtime to scan database. As the algorithm progressively searches for longer frequent itemsets in a depth-first manner, it creates new partitions with even smaller sizes having less dimensions and unique data instances, which results in faster support counting. SelPMiner uses a number of optimizations to prune the search space. We also prove upper bounds on the amount of memory consumed by these partitions. Experimental comparisons of the SelPMiner algorithm with popular existing fastest MFI mining algorithms on different types of datasets show significant speedup in computation time for many cases. SelPMiner works especially well when the minimum support is low and consumes less memory.

Mining Recent Maximal Frequent Itemsets Over Data Streams with Sliding Window

Abstract: The huge number of data streams makes it impossible to mine recent frequent itemsets. Due to the maximal frequent itemsets can perfectly imply all the frequent itemsets and the number is much smaller, therefore, the time cost and the memory usage for mining maximal frequent itemsets are much more efficient. This paper proposes an improved method called Recent Maximal Frequent Itemsets Mining (RMFIsM) to mine recent maximal frequent itemsets over data streams with sliding window. The RMFIsM method uses two matrixes to store the information of data streams, the first matrix stores the information of each transaction and the second one stores the frequent 1-itemsets. The frequent p-itemsets are mined with “extension” process of frequent 2-itemsets, and the maximal frequent itemsets are obtained by deleting the sub-itemsets of long frequent itemsets. Finally, the performance of the RMFIsM method is conducted by a series of experiments, the results show that the proposed RMFIsM method can mine recent maximal frequent itemsets efficiently

SiBIC: A Tool for Generating a Network of Biclusters Captured by Maximal Frequent Itemset Mining.

Biclustering extracts coexpressed genes under certain experimental conditions, providing more precise insight into the genetic behaviors than one-dimensional clustering. For understanding the biological features of genes in a single bicluster, visualizations such as heatmaps or parallel coordinate plots and tools for enrichment analysis are widely used. However, simultaneously handling many biclusters still remains a challenge. Thus, we developed a web service named SiBIC, which, using maximal frequent itemset mining, exhaustively discovers significant biclusters, which turn into networks of overlapping biclusters, where nodes are gene sets and edges show their overlaps in the detected biclusters. SiBIC provides a graphical user interface for manipulating a gene set network, where users can find target gene sets based on the enriched network. This chapter provides a user guide/instruction of SiBIC with background of having developed this software. SiBIC is available at http://utrecht.kuicr.kyoto-u.ac.jp:8080/sibic/faces/index.jsp .

A novel process-based association rule approach through maximal frequent itemsets for big data processing

The maximal frequent itemsets issue in big data processing has become a hot research topic. Most of the previous work on big data processing directly analyzes the data through the existing approaches, which would cause problems of redundant computation, high time complexity, and large storage space. To solve the problems, this paper proposes a Heuristic MapReduce-based Association rule approach through Maximal frequent itemsets mining, HMAM. The main idea is: At first, by directly operating on the transaction database, we allocate transactions to different processing nodes and group all transactions according to dimension. Then, we screen the most frequent transactions from each transaction set using the Bitmap-Sort and obtain best-transaction-set through aggregating all the transaction-elects of each transaction set. The current candidate maximal frequent itemsets can be acquired by removing sub-transactions in terms of the inclusion relations of the items in best-transaction-set. At the same time, each subset of sub-transactions in the candidate maximal frequent itemsets is discarded from all transaction sets. Then the final candidate maximal frequent itemsets can be obtained by iteration until each transaction set is empty. Finally, we achieve the acquisition of maximal frequent itemsets by employing the minimum support threshold. The experimental results demonstrate that compared with the existing approaches, HMAM significantly avoids producing a large number of candidate itmesets resulting from join operation, accelerates the speed of mining the maximal frequent itemsets, and improves the utilization rate of resources simultaneously.

A Matrix based Maximal Frequent Itemset Mining Algorithm without Subset Creation

A Matrix based Maximal Frequent Itemset Mining Algorithm without Subset Creation

Research of Association Rules Algorithm in Data Mining

With the rapid development of information technology, the data is becoming more and more large, the data mining technology has been applied to many fields in the life. In view of the existing data mining algorithms complexity is too high, the execution time too long, the article applies association rules in data mining technology, analysis summarized the existing advantages and disadvantages of the classic data mining association rules algorithm, is proposed for mining maximum frequent itemsets MFP - Miner algorithm. At last, by experimental simulation result shows that the proposed algorithm performance superiority, its complexity and execution time significantly shortened, the practical guiding significance.

A novel approach for data stream maximal frequent itemsets mining

This paper proposes a novel algorithm AMMFI based on self-adjusting and orderly compound policy to solve the problems of existing algorithms for mining maximal frequent itemsets in a data stream. The proposed algorithm processes the data stream based on sliding window technique and scans data stream fragments single-pass to obtain and store frequent itemsets in frequent itemsets list. It then constructs a self-adjusting and orderly FP-tree, dynamically adjusts the tree structure with the insertion of itemsets, uses mixed subset pruning method to reduce the search space, and merges nodes with the same min_sup in identical branch. Finally, orderly compound FP-tree is generated and it avoids superset checking in the process of mining maximal frequent itemsets. Detailed simulation analysis demonstrates that the presented algorithm is of high efficiency of space and time and is more stable.

Frequent Data Mining in Data Publishing for Privacy Preservation

Weighted frequent pattern mining is suggested to find out more important frequent pattern by considering different weights of each item. Weighted Frequent Patterns are generated in weight ascending and frequency descending order by using prefix tree structure. These generated weighted frequent patterns are applied to maximal frequent item set mining algorithm. Maximal frequent pattern mining can reduces the number of frequent patterns and keep sufficient result information. In this paper, we proposed an efficient algorithm to mine maximal weighted frequent pattern mining over data streams. A new efficient data structure i.e. prefix tree and conditional tree structure is used to dynamically maintain the information of transactions. Here, three information mining strategies (i.e. Incremental, Interactive and Maximal) are presented. The detail of the algorithms is also discussed. Our study has submitted an application to the Electronic shop Market Basket Analysis. Experimental studies are performed to evaluate the good effectiveness of our algorithm..

MaxMining: A Novel Algorithm for Mining Maximal Frequent Itemset

We present a new algorithm for mining maximal frequent itemsets, MaxMining, from big transaction databases. MaxMining employs the depth-first traversal and iterative method. It re-represents the transaction database by vertical tidset format, travels the search space with effective pruning strategies which reduces the search space dramatically. MaxMining removes all the non-maximal frequent itemsets to get the exact set of maximal frequent itemsets directly, no need to enumerate all the frequent itemsets from smaller ones step by step. It backtracks to the proper ancestor directly, needless level by level, ignoring those redundant frequent itemsets. We found that MaxMining can be more effective to find all the maximal frequent itemsets from big databases than many of proposed algorithms with ordinary pruning strategies.

A distributed maximal frequent itemset mining with multi agents system on bitmap join indexes selection

The amount of information in a data warehouse tends to be extremely large and queries may involve several complex join and aggregate operations at the same time. By using the right indices, the database administrator can speed up these OLAP queries and dramatically shorten processing times. However, selection of an optimal set of indices is a very hard task because of the exponential number of attribute candidates that can be used in the selection process. Addressing this problem, we propose a new approach with two main phases. The first involves pruning the search space to reduce the number of indices candidates. To that end, we use a distributed maximal itemsets mining approach based on a multi agent system that can significantly reduce the complexity of the selection process. We also incorporate a convertible anti-monotone constraint that contains information on the profit of index. The second phase uses also a multi agent's architecture to select final indices using a subset of attribute candidates. This final configuration will provide benefit to OLAP queries, but will also respect the disk space constraint. We validate our proposed approach using an experimental evaluation.

Integrating monitor alarms with laboratory test results to enhance patient deterioration prediction

Patient monitors in modern hospitals have become ubiquitous but they generate an excessive number of false alarms causing alarm fatigue. Our previous work showed that combinations of frequently co-occurring monitor alarms, called SuperAlarm patterns, were capable of predicting in-hospital code blue events at a lower alarm frequency. In the present study, we extend the conceptual domain of a SuperAlarm to incorporate laboratory test results along with monitor alarms so as to build an integrated data set to mine SuperAlarm patterns. We propose two approaches to integrate monitor alarms with laboratory test results and use a maximal frequent itemsets mining algorithm to find SuperAlarm patterns. Under an acceptable false positive rate FPRmax, optimal parameters including the minimum support threshold and the length of time window for the algorithm to find the combinations of monitor alarms and laboratory test results are determined based on a 10-fold cross-validation set. SuperAlarm candidates are generated under these optimal parameters. The final SuperAlarm patterns are obtained by further removing the candidates with false positive rate>FPRmax. The performance of SuperAlarm patterns are assessed using an independent test data set. First, we calculate the sensitivity with respect to prediction window and the sensitivity with respect to lead time. Second, we calculate the false SuperAlarm ratio (ratio of the hourly number of SuperAlarm triggers for control patients to that of the monitor alarms, or that of regular monitor alarms plus laboratory test results if the SuperAlarm patterns contain laboratory test results) and the work-up to detection ratio, WDR (ratio of the number of patients triggering any SuperAlarm patterns to that of code blue patients triggering any SuperAlarm patterns). The experiment results demonstrate that when varying FPRmax between 0.02 and 0.15, the SuperAlarm patterns composed of monitor alarms along with the last two laboratory test results are triggered at least once for [56.7–93.3%] of code blue patients within an 1-h prediction window before code blue events and for [43.3–90.0%] of code blue patients at least 1-h ahead of code blue events. However, the hourly number of these SuperAlarm patterns occurring in control patients is only [2.0–14.8%] of that of regular monitor alarms with WDR varying between 2.1 and 6.5 in a 12-h window. For a given FPRmax threshold, the SuperAlarm set generated from the integrated data set has higher sensitivity and lower WDR than the SuperAlarm set generated from the regular monitor alarm data set. In addition, the McNemar’s test also shows that the performance of the SuperAlarm set from the integrated data set is significantly different from that of the SuperAlarm set from the regular monitor alarm data set. We therefore conclude that the SuperAlarm patterns generated from the integrated data set are better at predicting code blue events.

Efficiently Mining Maximal Frequent Itemsets Based on Digraph

In this paper, a novel algorithm for mining maximal frequent itemsets is presented, which has a pre-processing phase where a digraph is constructed. The digraph represents the frequent 2-itemsets which play an important role on the performance of data mining. Then the search for maximal frequent itemsets is done in the digraph. Experiments show that the proposed algorithm is efficient for all types of data.

Algorithms of Mining Maximum Frequent Itemsets Based on Compression Matrix

Association rule mining is one of the most important and well researched techniques of data mining, the key procedure of the association rule mining is to find frequent itemsets , the frequent itemsets are easily obtained by maximum frequent itemsets. so finding maximum frequent itemsets is one of the most important strategies of association data mining. Algorithms of mining maximum frequent itemsets based on compression matrix are introduced in this paper. It mainly obtains all maximum frequent itemsets by simply removing a set of rows and columns of transaction matrix, which is easily programmed recursive algorithm. The new algorithm optimizes the known association rule mining algorithms based on matrix given by some researchers in recent years, which greatly reduces the temporal complexity and spatial complexity, and highly promotes the efficiency of association rule mining.

Enhanced Index-Based GenMax for Frequent Item Set Mining

In many data mining applications such as the discovery of association rules, strong rules, and many other important discovery tasks, mining frequent item sets is a fundamental and essential problem. Methods have been implemented for mining frequent item sets using a prefix-tree structure, for storing compressed information GenMax is used for mining maximal frequent item sets. It uses a technique called progressive focusing to perform maximal checking, and differential set propagation to perform fast frequency computation. GenMax algorithm was not implemented for closed frequent item set. The proposal in this paper present an improved index based enhancement on Genmax algorithm for effective fast and less memory utilized pruning of maximal frequent item and closed frequent item sets. The extension induces a search tree on the set of frequent closed item sets thereby we can completely enumerate closed item sets without duplications. The memory use of mining the maximal frequent item set does not depend on the number of frequent closed item sets. The proposed model reduces the number of disk (Input and Outputs) I/Os and make frequent item set mining scale to large transactional databases. Experimental results shows a comparison of improved index based GenMax and existing GenMax for efficient pruning of maximal frequent and closed frequent item sets in terms of item precision and fastness.

SiBIC: A Web Server for Generating Gene Set Networks Based on Biclusters Obtained by Maximal Frequent Itemset Mining

Detecting biclusters from expression data is useful, since biclusters are coexpressed genes under only part of all given experimental conditions. We present a software called SiBIC, which from a given expression dataset, first exhaustively enumerates biclusters, which are then merged into rather independent biclusters, which finally are used to generate gene set networks, in which a gene set assigned to one node has coexpressed genes. We evaluated each step of this procedure: 1) significance of the generated biclusters biologically and statistically, 2) biological quality of merged biclusters, and 3) biological significance of gene set networks. We emphasize that gene set networks, in which nodes are not genes but gene sets, can be more compact than usual gene networks, meaning that gene set networks are more comprehensible. SiBIC is available at http://utrecht.kuicr.kyoto-u.ac.jp:8080/miami/faces/index.jsp.