Data Chunks Research Articles

Machine Learning-Based Boosted Regression Ensemble Combined with Hyperparameter Tuning for Optimal Adaptive Learning.

Over the past couple of decades, many telecommunication industries have passed through the different facets of the digital revolution by integrating artificial intelligence (AI) techniques into the way they run and define their processes. Relevant data acquisition, analysis, harnessing, and mining are now fully considered vital drivers for business growth in these industries. Machine learning, a subset of artificial intelligence (AI), can assist, particularly in learning patterns in big data chunks, intelligent extrapolative extraction of data and automatic decision-making in predictive learning. Firstly, in this paper, a detailed performance benchmarking of adaptive learning capacities of different key machine-learning-based regression models is provided for extrapolative analysis of throughput data acquired at the different user communication distances to the gNodeB transmitter in 5G new radio networks. Secondly, a random forest (RF)-based machine learning model combined with a least-squares boosting algorithm and Bayesian hyperparameter tuning method for further extrapolative analysis of the acquired throughput data is proposed. The proposed model is herein referred to as the RF-LS-BPT method. While the least-squares boosting algorithm is engaged to turn the possible RF weak learners to form stronger ones, resulting in a single strong prediction model, the Bayesian hyperparameter tuning automatically determines the best RF hyperparameter values, thereby enabling the proposed RF-LS-BPT model to obtain desired optimal prediction performance. The application of the proposed RF-LS-BPT method showed superior prediction accuracy over the ordinary random forest model and six other machine-learning-based regression models on the acquired throughput data. The coefficient of determination (Rsq) and mean absolute error (MAE) values obtained for the throughput prediction at different user locations using the proposed RF-LS-BPT method range from 0.9800 to 0.9999 and 0.42 to 4.24, respectively. The standard RF models attained 0.9644 to 0.9944 Rsq and 5.47 to 12.56 MAE values. The improved throughput prediction accuracy of the proposed RF-LS-BPT method demonstrates the significance of hyperparameter tuning/optimization in developing precise and reliable machine-learning-based regression models. The projected model would find valuable applications in throughput estimation and modeling in 5G and beyond 5G wireless communication systems.

Implementasi Algoritma Apriori untuk Mendapatkan Pola Kelulusan Mahasiswa

Data transactions that occur every day will become chunks of data in an institution's database. These chunks of data have no value if mining is not carried out in obtaining interesting knowledge and information. The purpose of this study was to analyze chunks of graduation data for STIE-Graha Karya Muara Bulian students, in order to obtain patterns formed from predetermined variables. The method used is the Apriori Algorithm. This is in accordance with the main function of this algorithm, which is to analyze the frequent itemset. The results showed that for the 2-itemset data the highest support value was the combination of Female Gender and Work with a value of 34%, while the highest confidence value was in the combination of a Grade Point Average above 3.5 and Work with a score of 85%. As for the 3-itemset, the highest support value is in the combination if the GPA is above 3.5, gender is female and the work status reaches 14% and confidence is 90%.

IoT Data Analysis Using Different Clustering Algorithms: A Survey

There has been a major change in the rise of data creation and consumption due to the quick development of contemporary technology and the advent of the Internet of Things (IoT). With this shift comes a significant challenge in data management, with the goal of extracting useful information and patterns from the vast amount of unstructured IoT data. However, due to some limitations, we are still encountering issues and challenges in obtaining patterns and meaning information, and thus formulating a research agenda to get a sense of the challenges that one might encounter while conducting research on clustering of this massive chunk of unstructured IoT data. In light of these considerations, this study looks at some clustering methods that may be utilized when working with unstructured IoT data.

Deterministic Sampling Classifier with weighted Bagging for drifted imbalanced data stream classification

One of the most critical data analysis tasks is the streaming data classification, where we may also observe the concept drift phenomenon, i.e., changing the decision model’s probabilistic characteristics. From a practical point of view, we may face this type of banking, medicine, or cybersecurity task to enumerate only a few. A vital characteristic of these problems is that the classes we are interested in (e.g., fraudulent transactions, treats, or serious diseases) are usually infrequent, which hinders the classification system design. The paper presents a novel algorithm DSCB (Deterministic Sampling Classifier with weighted Bagging) employs data preprocessing methods and weighted bagging technique to classify non-stationary imbalanced data stream. It builds models based on an incoming data chunk, but it also takes previously arrived instances into account. The proposed approach has been evaluated based on a wide range of computer experiments carried out on real and artificially generated data streams with various imbalance ratios, label noise levels, and concept drift types. The results confirmed that the weighted bagging ensemble coupled with data preprocessing could outperform state-of-the-art methods.

Pengelompokan Barang Menggunakan Metode K-Means Clustering Berdasarkan Hasil Penjualan Di Toko Widya Bengkulu

The Widya shop sells various kinds of hijab products that are tailored to the needs of fashion development. The Widya Bengkulu store also sells various kinds of women's needs such as clothes, accessories, bags, wallets, soft lenses, cosmetics and other fancy items. So far, Widya Store has not used computers as a data processing medium. All processes of selling goods and inventory of goods are still carried out with book records, and do not yet have an application that can help the process of managing the data of these goods. In addition, in managing inventory, Toko Widya only sees stock based on sales results, if the stock on one of the items runs out, an order will be made to the supplier.
 Implementation of the K-Means Clustering Method in Grouping Goods Based on Sales Results at the Widya Bengkulu Store was made using the Visual Basic .Net programming language and SQL Server 2008 database. In the Grouping of Goods Based on Sales Results at the Widya Bengkulu Store, it will be processed to seek knowledge from chunks of data, namely goods sales data. Cluster I is a Very Selling Item, Cluster II is a Self Selling Item, and Cluster III is a Less Selling Item. Based on the results of the tests that have been carried out, the application of Grouping Goods Based on Sales Results at the Widya Bengkulu Store can provide information on the best-selling items sold at the Widya Store.

Fast continuous streaming sort in big streaming data environment under fixed-size single storage.

Big streaming data environment concerns a complicated scenario where data to be processed continuously flow into a processing unit and certainly cause a memory overflow problem. This obstructs the adaptation of deploying all existing classic sorting algorithms because the data to be sorted must be entirely stored inside the fixed-size storage including the space in internal and external storage devices. Generally, it is always assumed that the size of each data chunk is not larger than the size of storage (M) but in fact the size of the entire stream (n) is usually much larger than M. In this paper, a new fast continuous streaming sorting is proposed to cope with the constraint of storage overflow. The algorithm was tested with various real data sets consisting of 10,000 to 17,000,000 numbers and different storage sizes ranging from 0.01n to 0.50n. It was found that the feasible lower bound of storage size is 0.35n with 100% sorting accuracy. The sorting time outperforms bubble sort, quick sort, insertion sort, and merge sort when data size is greater than 1,000,000 numbers. Remarkably, the sorting time of the proposed algorithm is 1,452 times less than the sorting time of external merge sort and 28.1767 times less than the sorting time of streaming data sort. The time complexity of proposed algorithm is O(n) while the space complexity is O(M).

Review Paper on LoRa based technologies for Vehicular and Tracking applications

Abstract: LoRa stands For Long Range. It is a low powered wide area network derived from chirp spread spectrum technology and encodes information through chirp pulses on to the radio waves. The range of LoRa greatly varies from few meters to approximately 100 Kilometers depending on the mode of operation. The main advantage of LoRa against Bluetooth, WiFi and ZigBee is its long range and it is well suited for transmission of small chunks of data with low bit rates. LoRa is generally operated at 915MHz, 868MHz and 433MHz which are Sub-Giga Heartz bands. These are a part of the ISM bands which are reserved for Industrial and Medical Purposes Internationally which are free of cost and do not require any kind of licensing. Vehicular communication is a form of exchanging messages between various vehicular and roadside units regarding their location, speed and much more. The LoRa technology when integrated with Vehicular Communication establishment of green corridors for Emergency vehicles by clearing the traffic and help in reaching the destination with minimal time delay. Keywords: LoRa, ISM Bands, Vehicular Communication

A Unified Vista and Juxtaposed Study on Sorting Algorithms

Data is the new fuel. With the expansion of the global technology, the in-creasing standards of living and with modernization, data values have caught a great height. Now a days, nearly all top MNCs feed on data. Now, to store all this data is a prime concern for all of them, which is relieved by the Data Structures, the systematic way of storing data. Now, once these data are stored and charged in secure vaults, it’s time to utilize them in the most efficient way. Now, there are a lot of operations that needs to be performed on these massive chunks of data, like searching, sorting, inserting, deleting, merging and so more. In this paper, we would be comparing all the major sorting algorithms that have prevailed till date. Further, work have been done and an inequality in dimension of time between the four Sorting algorithms, Bubble, Selection, Insertion, Merge, that have been discussed in the paper have been proposed.

Continuous discovery of Causal nets for non-stationary business processes using the Online Miner

Capturing business process specifics using a model is essential to effectively manage, control, and instruct the process participants with their roles and tasks. A normative process model is an invaluable source of information, not only for human inspection but also for software supporting and controlling the process. The actual process execution likely deviates from the normative model and the magnitude of deviation usually increases over time due to process evolution. A descriptive process model is far more informative when comes to the analysis of the actual execution of the process and spotting the deviations from the norm. However, handcrafting the descriptive model using process-related documents is prohibitively laborious. To automate the discovery of the descriptive process models, we propose Online Miner (OM), an algorithm that continuously builds process models using a stream of events raised by the process. OM first builds a sound process model using a historical event log and then incrementally adapts this model to new data chunks. OM represents the model using a Causal-net – a dependency graph of activities in which arcs are bound with each other to reflect parallel dependencies and exclusive alternatives. OM finds the maximal bindings being consistent with the event log. OM produces sound and perfect fit and generalizing well models. OM quickly reacts to the concept drift spot in the event log by seamlessly adapting the model. OM helps business analytics experts with their everyday tasks of monitoring, auditing, and enhancing business processes.

Employing chunk size adaptation to overcome concept drift

Modern analytical systems must process streaming data and correctly respond to data distribution changes. The phenomenon of changes in data distributions is called concept drift, and it may harm the quality of the used models. Additionally, the possibility of concept drift appearance causes that the used algorithms must be ready for the continuous adaptation of the model to the changing data distributions. This work focuses on non-stationary data stream classification, where a classifier ensemble is used. To keep the ensemble model up to date, the new base classifiers are trained on the incoming data blocks and added to the ensemble while, at the same time, outdated models are removed from the ensemble. One of the problems with this type of model is the fast reaction to changes in data distributions. We propose the new Chunk Adaptive Restoration framework that can be adapted to any block-based data stream classification algorithm. The proposed algorithm adjusts the data chunk size in the case of concept drift detection to minimize the impact of the change on the predictive performance of the used model. The experimental research, backed up with the statistical tests, has proven that Chunk Adaptive Restoration significantly reduces the model’s restoration time.

New incremental SVM algorithms for human activity recognition in smart homes

Smart homes are equipped with several sensor networks to keep an eye on both residents and their environment, to interpret the current situation and to react immediately. Handling large scale dataset of sensory events on real time to enable efficient interventions is challenging and very difficult. To deal with these data flows and challenges, traditional streaming data classification approaches can be boosted by use of incremental learning. In this paper, we presented two new Incremental SVM methods to improve the performance of SVM classification in the context of human activity recognition tasks. Two feature extraction methods elaborated by refining dependency sensor extraction feature and focusing on the last sensor event only have been suggested. On the other hand, a clustering based approach and a similarity based approach have been suggested to boost learning performance of the incremental SVM algorithms capitalizing on the relationship between data chunk and support vectors of previous chunk. We demonstrate through several simulations on two major publicly available data sets (Aruba and Tulum), the feasibility and improvements in learning and classification performances in real time achieved by our proposed methods over the state-of-the-art. For instance, we have shown that the introduced similarity-based incremental learning is 5 to 9 times faster than other methods in terms of training performances. Similarly, the introduced Last-state sensor feature method induces at least 5% improvement in terms of F1-score when using baseline SVM classifier.

An Incremental Learning Method Based on Dynamic Ensemble RVM for Intrusion Detection

Due to the dynamic changes of network data over time, static intrusion detection systems cannot adapt well to the behavioral characteristics of the input network data, resulting in reduced detection accuracy. In addition, continuous input data streams will bring huge challenges to resource storage and computing costs. Therefore, we propose an intrusion detection method of dynamic ensemble incremental learning (DEIL-RVM), and realize a dynamically adjusted ensemble intrusion detection model. In which a new overall misclassification probability weight value (OMPW) based on incremental set or data chunk is designed as the basis for updating the ensemble model, and it can be used to prune and replace the poor base component in the ensemble model. We presented a probabilistic decision function taking into account the posterior probability of each base RVM model dividing the sample into each category. The RVM with high sparsity is used as the base component to obtain the good balance between the accuracy, robustness and resource consumption, which can sacrifice less time and storage cost in ensemble incremental learning while achieving higher detection accuracy and stability in network data streams.

Comparisonal Analysis Of Even-Rodeh Algorithm Code And Fibonacci Code Algorithm For Text File Compression

The growing requirement towards bigger storage space is the main cause of new compression techniques being developed. By doing compression, big chunks of data will be lowered in size to save storage space. In this research we will be using the Even-Rodeh Code and Fibonacci Code Algorithm, the performance will be benchmarked with bitrate, compression ratio, space saving, compression and decompression time in text files. Compression is done by reading strings on text files, and then the Even-Rodeh Code and Fibonacci Code Algorithm makes a string code and performs compression. The compression results are *.erc and *.fib files containing character information and string bit which can be decompressed. The decompression result is the original text file which can be saved using the extensions *.doc, *.docx, *.pdf. In this system test the sample used are strings that contains one type of character (Homogen String) and strings that contain several types of character (heterogeneous string) saved in text files *.doc,*.docx, *.pdf. In homogeneous string compressions we can observe that the Fibonacci Code is more efficient in compression ratio, averaging at 0.25 and faster decompression time at 0.011 millisecond average compared to the Even-Rodeh Code Algorithm. In heterogeneous string compression the Fibonacci Code performs better than the Even-Rodeh Code on an average compression ratio of 0.65 and a shorter average decompression time of 0.293 milliseconds.

Weighted Ensemble with one-class Classification and Over-sampling and Instance selection (WECOI): An approach for learning from imbalanced data streams

Learning from imbalanced data streams is one of the challenges associated with classification algorithms and learning classifiers. The goal of this paper is to propose and validate a new approach for learning from data streams, with reference to the problem of class-imbalanced data. A hybrid approach for changing the class distribution towards a more balanced dataset using over-sampling and instance selection techniques is discussed. The proposed approach is based on the integration of a weighted ensemble classification and a technique to deal with the problem of class imbalance, and is called Weighted Ensemble with one-class Classification and Over-sampling and Instance selection (WECOI). Our approach assumes that classifiers are induced from incoming blocks of instances, called data chunks. These data chunks consist of incoming instances from different classes, and a balance between them is reached through our hybrid approach. These data chunks are then used to induce classifier ensembles. The proposed approach is validated experimentally using several selected benchmark datasets, and the results of computational experiments are presented and discussed. The results show that the proposed approach for eliminating class imbalance in data streams can help increase the performance of online learning algorithms. This is an extended version of a paper that was presented at the International Conference on Computational Science in 2021 (ICCS-2021) (Czarnowski, 2021). This version has been enriched with a deeper analysis of the results obtained from a modified variant of the originally proposed approach.

Hybrid dynamic chunk ensemble model for multi-class data streams

In the analysis more specifically in the classification of continuous data stream using machine learning algorithms joint occurrence of concept drift and imbalanced issue becomes more provocative. Also, imbalance issue is again more challenging when the data stream is multi-class with minority class and that is too with data-difficulty factors. Incremental learning with ensemble models found more promising in handling theses issues. But most of the approaches are for two-class data streams which can’t be utilized for multiclass data streams. In this paper we have designed hybrid dynamic chunk ensemble model (HDCEM) for the classification of multi-class insect-data stream for handling imbalance and concept drift issue. To deal with imbalance issue we have proposed effective split bagging algorithm which has achieved better performance on minority class recall and F-measure on arriving dynamic chunks of data from multi-class data stream. HDCEM model can adapt to abrupt and gradual drift because it has combined features of both online and chunk-based learning together. It has achieved average 78% minority class recall in abrupt insect data stream and 71% in gradual drift insect stream.

Carving of the OOXML document from volatile memory using unsupervised learning techniques

With the drastic rise in digital crimes, emphasis is laid on the digital forensics to solve crimes. Evidences that are being extracted from the digital devices are now considered enough to prove a criminal guilty in the court room. While gathering the proofs from a system, a significant venue to look at is the primary memory (commonly named as RAM). A RAM holds the information about the current state of the system. It often occurs that the metadata of a file is deleted or is not available to the digital investigator. In such cases, the retrieval of the file from RAM becomes very challenging as data is placed at random locations in RAM and their allocation tables are not valid if power supply to the system is removed. While carving these instances that are randomly scattered across the RAM, their usefulness in law process cannot be ensured. Microsoft Open Office XML file Format (OOXML FF) is one of the most widely used format, yet it is not much explored in forensics (and carving). Our research intends to improve the technique of carving of an OOXML file where we have employed clustering to collect the chunks of same data based on some similarity feature. Numerous OOXML files are used in our experiments, where we have extracted and rearranged their textual contents using clustering techniques i.e. K mean and Hierarchical clustering. The results are quite encouraging and show that our proposed method can be used for carving of OOXML format. Our technique of extraction of OOXML document from the RAM reduces the hassle and saves ample time of the digital investigator who would otherwise have to go through every document available on the system to find the concerned document.

Pyramid-based Scatterplots Sampling for Progressive and Streaming Data Visualization

We present a pyramid-based scatterplot sampling technique to avoid overplotting and enable progressive and streaming visualization of large data. Our technique is based on a multiresolution pyramid-based decomposition of the underlying density map and makes use of the density values in the pyramid to guide the sampling at each scale for preserving the relative data densities and outliers. We show that our technique is competitive in quality with state-of-the-art methods and runs faster by about an order of magnitude. Also, we have adapted it to deliver progressive and streaming data visualization by processing the data in chunks and updating the scatterplot areas with visible changes in the density map. A quantitative evaluation shows that our approach generates stable and faithful progressive samples that are comparable to the state-of-the-art method in preserving relative densities and superior to it in keeping outliers and stability when switching frames. We present two case studies that demonstrate the effectiveness of our approach for exploring large data.

Security-Aware and Efficient Data Deduplication for Edge-Assisted Cloud Storage Systems

Data deduplication at the network edge significantly improves communication efficiency in edge-assisted cloud storage systems. With the increasing concern about data privacy, secure deduplication has been proposed to provide data security while supporting deduplication. Since conventional secure deduplication schemes are mainly based on deterministic encryption, they are vulnerable to frequency analysis attacks. Some recent research has focused on this problem, where several works studied the trade-off between deduplication efficiency and resistance to frequency analysis attacks. However, no existing work can provide different deduplication efficiency and protection for different data chunks. In this paper, we propose a security-aware and efficient data deduplication scheme for edge-assisted cloud storage systems. It not only improves the efficiency of deduplication but also reduces information leakage caused by frequency analysis attacks. In particular, we first define the security level for chunks to measure the security needs of users. Then, we develop an encryption scheme with multiple levels of security for deduplication. It provides higher security protection for chunks with higher security levels, while sacrificing security to achieve higher deduplication efficiency for chunks with lower security levels. We also analyze the security of our proposed scheme. Evaluations on the real-world datasets show the efficiency of our design.

Incremental Learning Method for Data with Delayed Labels

Most research on machine learning tasks relies on the availability of true labels immediately after making a prediction. However, in many cases, the ground truth labels become available with a non-negligible delay. In general, delayed labels create two problems. First, labelled data is insufficient because the label for each data chunk will be obtained multiple times. Second, there remains a problem of concept drift due to the long period of data. In this work, we propose a novel incremental ensemble learning when delayed labels occur. First, we build a sliding time window to preserve the historical data. Then we train an adaptive classifier by labelled data in the sliding time window. It is worth noting that we improve the TrAdaBoost to expand the data of the latest moment when building an adaptive classifier. It can correctly distinguish the wrong types of source domain sample classification. Finally, we integrate the various classifiers to make predictions. We apply our algorithms to synthetic and real credit scoring datasets. The experiment results indicate our algorithms have superiority in delayed labelling setting.

An Ameliorated Multiattack Network Anomaly Detection in Distributed Big Data System-Based Enhanced Stacking Multiple Binary Classifiers

The growth of the internet of things (IoT) generates new processing, networking infrastructure, data storage, and management capabilities. This massive data volume may be used to provide high-value information for decision support, forecasting, business intelligence, data-intensive science research, etc. However, due to the nature of IoT in distribution, virtualization, cloud integration, internet connectivity, the IoT environment is prone to various cyber-attacks and security issues, including an inability to access data chunks at the in/outcoming stream like worms, malware, virtual machine escape, distributed denial of service attack (DDoS), etc. Hence, the increasing frequency and potency of recent attacks and the constantly evolving attack vectors necessitate the development of improved detection approaches. Therefore, this paper proposes a distributed computing-based security model to safeguard big data systems. The proposed ensemble multi binary attack model (EMBAM) is an Intrusion Detection System (IDS) that offers a unique anomaly-based IDS to detect normal behaviour and abnormal attack(s), e.g., threats in a network. The EMBAM ensemble multiple binary classifiers into a single model by stacking. The core binary model is a decision tree classifier with hyperparameters optimized using the grid search method. Multiple binary classifiers’ usage allows each binary classifier to adopt the limitations of the others in determining the intercorrelation of the available features and the attack to achieve a high detection rate and a low false alarm rate. The improvement suggestions and outcomes insights by the EMBAM confirm its superiority in inaccuracy, detection rate, and efficiency. Two datasets have been used to validate the experiments, i.e., UNSW-NB15 and CICIDS2017. Empirical analysis of the experimental profile of the EMBAM has been discussed with eight-plus state-of-the-art methods using performance metrics like accuracy, precision, detection rate, false alarm rate, and F1-score. The EMBAM can recognize multiple attack types as a star plug and play advantage in a highly dynamic scheme. The proposed approach outperforms existing approaches over the UNSW-NB15 dataset and yields competitive results over the CICIDS2017 dataset.