Large Dimensional Data Research Articles

An Improved Design for a Cloud Intrusion Detection System Using Hybrid Features Selection Approach with ML Classifier

The rapid adoption of cloud computing has introduced new security concerns, particularly regarding data privacy and protection from sophisticated cyberattacks. Intrusion Detection Systems (IDSs) based on Machine Learning (ML) offer improved detection of malicious behaviour in network traffic. However, existing IDS solutions struggle with large-dimensional data and imbalanced datasets, resulting in higher false positive rates (FPR) and reduced detection rates (DR). In this paper, we propose an improved Cloud IDS leveraging a hybrid features selection approach using Information Gain (IG), Chi-square (CS), and Particle Swarm Optimization (PSO). To address the challenge of imbalanced datasets, the Synthetic Minority Over-sampling Technique (SMOTE) is employed. The Random Forest (RF) classifier is used to detect and classify attacks. The system is evaluated on the UNSW-NB15 and Kyoto datasets, achieving accuracies of over 98% and 99%, respectively, outperforming other methods in terms of detection accuracy.

Deep Learning - Methods to Amplify Epidemiological Data Collection and Analyses.

Deep learning is a subfield of artificial intelligence and machine learning based mostly on neural networks and often combined with attention algorithms that has been used to detect and identify objects in text, audio, images, and video. Serghiou and Rough (Am J Epidemiol. 0000;000(00):0000-0000) present a primer for epidemiologists on deep learning models. These models provide substantial opportunities for epidemiologists to expand and amplify their research in both data collection and analyses by increasing the geographic reach of studies, including more research subjects, and working with large or high dimensional data. The tools for implementing deep learning methods are not quite yet as straightforward or ubiquitous for epidemiologists as traditional regression methods found in standard statistical software, but there are exciting opportunities for interdisciplinary collaboration with deep learning experts, just as epidemiologists have with statisticians, healthcare providers, urban planners, and other professionals. Despite the novelty of these methods, epidemiological principles of assessing bias, study design, interpretation and others still apply when implementing deep learning methods or assessing the findings of studies that have used them.

USE OF ARTIFICIAL INTELLIGENCE AND AUDIT ANALYTICS IN INTERNAL AUDIT PROCESSES IN THE PUBLIC SECTOR

The impact of digitization on management processes in the public sector has led to changes in their roles, corporate activities, objectives, and corporate requirements. Consequently, as the scope and responsibilities of the public sector have changed and expanded due to digitization, the organization, storage, and processing of the increasing dimension of data resulting from public services have become more complex. This has also accelerated the handling of artificial intelligence in internal audits in the public sector. In the public sector, the large dimension of data related to citizens personal information, public tenders, contracts, and suppliers makes it difficult to control and oversee this data. Therefore, there is a trend in the public sector to handle Public Sector Internal Audit Artificial Intelligence models and audit analytics to facilitate Internal Audit while also automating the monitoring and tracking of corporate activities. From this viewpoint, this study explores the advantages of handling Artificial Intelligence models and audit analytics in internal audit. This study attempts to contribute to the literature by providing policy recommendations for professionals and researchers.

Efficient and Fast Algorithm for Attribute Reduction of Large Dimensional Data Using Rough Set Theory on Graphics Processing Unit

Efficient and Fast Algorithm for Attribute Reduction of Large Dimensional Data Using Rough Set Theory on Graphics Processing Unit

Time dimension feature extraction and classification of high-dimensional large data streams based on unsupervised learning

In order to solve the problem of low accuracy of time dimension feature extraction and classification of high-dimensional large data streams, this paper proposes a time dimension feature extraction and classification algorithm of high-dimensional large data streams based on unsupervised learning. Analyze the trend of high-dimensional data flow changes under machine learning, and achieve dimensionality reduction of high-dimensional large traffic time dimensional data through local save projection. Analyze the spatial relationship between feature attributes and feature space, segment and fit high-dimensional big data streams and time dimensional feature data streams, further segment time dimensional sequences using sliding windows, and complete feature extraction through discrete dyadic wavelet transform. According to the clustering algorithm, cluster the time dimension feature data stream, calculate the cosine similarity of the feature data, model the time dimension feature stream of training samples, use the feature classification function to minimize the classification loss, and use unsupervised learning to achieve the final classification task. The test results show that this method can improve the temporal feature extraction and classification accuracy streams.

Intrusion Detection using Deep Learning

The focus of cloud computing nowadays has been reshaping the digital epoch, in which clients now face serious concerns about the security and privacy of their data hosted in the cloud, as well as increasingly sophisticated and frequent cyber attacks. Therefore, it has become imperative for both individuals and organizations to implement a robust intrusion detection system (IDS) capable of monitoring packets in the network, distinguishing between benign and malicious behaviour, and detecting the type of attacks. IDS based on ML are efficient and precise in spotting network threats. Yet, for large dimensional data sizes, the performance of these systems decreases. Thus, it is critical to build a suitable feature selection approach that selects necessary features without having an impact on the classification process or causing information loss. Furthermore, training ML models on unbalanced datasets show a rising false positive rate (FPR) and a lowering detection rate (DR). In this paper, we present an improved cloud IDS designed by incorporating the synthetic minority over-sampling technique (SMOTE) to address the imbalanced data issue, and for feature selection, we propose to use a hybrid approach that includes three techniques: information gain (IG), chi-square (CS), and particle swarm optimization (PSO). Finally, the random forest (RF) model is utilized for detecting and classifying various types of attacks. The suggested system has been verified by the UNSW-NB15 and Kyoto datasets, achieving accuracies of over 98% and 99% in the multi-class classification scenario, respectively. It was noticed that an intrusion detection system with fewer informative features would operate more effectively. The simulation results significantly outperform other methodologies proposed in the related work in terms of different evaluation metrics.

Authentication of selected white wines by geographical origin using ICP spectrometric and chemometric analysis

An important aspect of assessing the authenticity of wines is its geographical origin. The aim of the work is to authenticate by geographical origin according to the data of the ICP-spectrometric and chemometric analysis of elemental "images" of wines produced from white grape varieties Chardonnay, Riesling and Muscat grown in four regions of the Krasnodar Territory, Russia. The difference in the contents of Al, Ba, Ca and Rb in wines was found depending on the variety, and Al, Ba, Rb, Fe, Li, Sr – depending on the region of grape growth. Different models of the experimental data processing were used for attribution of the produced varieties of wine to the area of the grape's growth. The criterion for the quality of the constructed models was the accuracy of the attribution of a wine variety to the area of the grape's growth (%). Analysis of the elemental analysis data of 153 wine samples showed that in terms of attribution accuracy, automated neural networks (100 %) are preferred among machine learning methods, followed by support vector machines (98.69 %) and general discriminant analysis (94.77 %). The applied mathematical models enabled the revealing of the cluster structure of the analyzed wine varieties and their attribution to the area of a grape growth with high accuracy. Sr, Li and Fe concentrations in wines were found as the dominating predictors in the constructed models for definition of the geographical origin of wines. The combination of ICP-spectrometric analysis data with the capabilities of statistical modeling of machine learning methods focused on large-dimensional data made it possible to successfully solve small-dimensional problems of the definition of the geographical origin of wines by their elemental composition and variety.

Anomaly attribute mining method for topological nodes of power system based on graph theory

Due to the large scale, high dimension and time series characteristics of power system data, and the normal samples far exceed the abnormal samples, the sample imbalance phenomenon occurs, and it is difficult to mine the abnormal attributes of nodes. Therefore, a method of mining abnormal attributes of topological nodes of power systems based on graph theory is developed. The logical relation between nodes is analyzed by graph theory, and directed graph and undirected graph are obtained. The topology structure of the power system is constructed, the noise in data is removed by the adaptive clustering algorithm, the weight of topological nodes is set, and the abnormal malicious attacks of topological nodes are discovered by the double threshold method. The experimental results show that the proposed method can accurately detect the abnormal topological nodes in the network, perform better in the energy consumption of the neighbor topological nodes, and greatly reduce the energy consumption of the neighbor topological nodes during mining.

Economic analysis of world cities using improved deep shallow learning network with intelligent shell game optimization

An economic analysis is required to determine the financial status of every city throughout the world The economic growth of a city depends on multiple factors like health, finances, transport, industry, and so on. Therefore, it is necessary to have a user-friendly factor for handling the economic data regarding the financial evaluation outcomes The main motive behind this research work is to tackle the issue of low prediction accuracy of the conventional economic and financial growth trend assumption model by developing a model using the latest deep network technology. The data regarding the economic status of a city is collected from standard online sources. The collected data are given to the preprocessing for economic prediction in any city throughout the world phase. After that the optimal attributes from the preprocessed data are extracted with the help of a newly suggested Accuracy-based Shell Game Optimization (ASGO) algorithm Consequently, the chosen optimal attributes are given as input to the final prediction stage. The economy prediction of a city is done using the Optimized and Deep Shallow Learning Network (ODSLN). The parameters in the ODSLN are tuned using the same ASGO algorithm. This helps in enhancing the prediction functionality of the deployed model over large dimensional data. The developed model is validated with standard performance metrics against other conventional prediction models. Throughout the result analysis, the developed model attains a 94% accuracy rate and 93% sensitivity rate which is much better than the existing approaches. The efficiency of the suggested deep learning-based economic prediction model is evaluated against the recently developed model based on several performance measures.

Stochastic first-order learning for large-scale flexibly tied Gaussian mixture models

Gaussian Mixture Models (GMMs) are one of the most potent parametric density models used extensively in many applications. Flexibly-tied factorization of the covariance matrices in GMMs is a powerful approach for coping with the challenges of common GMMs when faced with high-dimensional data and complex densities which often demand a large number of Gaussian components. However, the expectation–maximization algorithm for fitting flexibly-tied GMMs still encounters difficulties with streaming and very large dimensional data. To overcome these challenges, this paper suggests the use of first-order stochastic optimization algorithms. Specifically, we propose a new stochastic optimization algorithm on the manifold of orthogonal matrices. Through numerous empirical results on both synthetic and real datasets, we observe that stochastic optimization methods can outperform the expectation–maximization algorithm in terms of attaining better likelihood, needing fewer epochs for convergence, and consuming less time per each epoch.

A study on ship hull form transformation using convolutional autoencoder

Abstract The optimal ship hull form in contemporary design practice primarily consists of three parts: hull form modification, performance prediction, and optimization. Hull form modification is a crucial step to affect optimization efficiency because the baseline hull form is varied to search for performance improvements. The conventional hull form modification methods mainly rely on human decisions and intervention. As a direct expression of the three-dimensional hull form, the lines are not appropriate for machine learning techniques. This is because they do not explicitly express a meaningful performance metric despite their relatively large data dimension. To solve this problem and develop a novel machine-based hull form design technique, an autoencoder, which is a dimensional reduction technique based on an artificial neural network, was created in this study. Specifically, a convolutional autoencoder was designed; firstly, a convolutional neural network (CNN) preprocessor was used to effectively train the offsets, which are the half-width coordinate values on the hull surface, to extract feature maps. Secondly, the stacked encoder compressed the feature maps into an optimal lower dimensional-latent vector. Finally, a transposed convolution layer restored the dimension of the lines. In this study, 21 250 hull forms belonging to three different ship types of containership, LNG carrier, and tanker, were used as training data. To describe the hull form in more detail, each was divided into several zones, which were then input into the CNN preprocessor separately. After the training, a low-dimensional manifold consisting of the components of the latent vector was derived to represent the distinctive hull form features of the three ship types considered. The autoencoder technique was then combined with another novel approach of the surrogate model to form an objective function neural network. Further combination with the deterministic particle swarm optimization method led to a successful hull form optimization example. In summary, the present convolutional autoencoder has demonstrated its significance within the machine learning-based design process for ship hull forms.

A-MKMC: An effective adaptive-based multilevel K-means clustering with optimal centroid selection using hybrid heuristic approach for handling the incomplete data

In general, clustering is defined as partitioning similar and dissimilar objects into several groups. It has been widely used in applications like pattern recognition, image processing, and data analysis. When the dataset contains some missing data or value, it is termed incomplete data. In such implications, the incomplete dataset issue is untreatable while validating the data. Due to these flaws, the quality or standard level of the data gets an impact. Hence, the handling of missing values is done by influencing the clustering mechanisms for sorting out the missing data. Yet, the traditional clustering algorithms fail to combat the issues as it is not supposed to maintain large dimensional data. It is also caused by errors of human intervention or inaccurate outcomes. To alleviate the challenging issue of incomplete data, a novel clustering algorithm is proposed. Initially, incomplete or mixed data is garnered from the five different standard data sources. Once the data is to be collected, it is undergone the pre-processing phase, which is accomplished using data normalization. Subsequently, the final step is processed by the new clustering algorithm that is termed Adaptive centroid based Multilevel K-Means Clustering (A-MKMC), in which the cluster centroid is optimized by integrating the two conventional algorithms such as Border Collie Optimization (BCO) and Whale Optimization Algorithm (WOA) named as Hybrid Border Collie Whale Optimization (HBCWO). Therefore, the validation of the novel clustering model is estimated using various measures and compared against traditional mechanisms. From the overall result analysis, the accuracy and precision of the designed HBCWO-A-MKMC method attain 93 % and 95 %. Hence, the adaptive clustering process exploits the higher performance that aids in sorting out the missing data issuecompared to the other conventional methods.

Network Traffic Anomaly Detection Model Based on Feature Reduction and Bidirectional LSTM Neural Network Optimization

Aiming at the problems of large data dimension, more redundant data, and low accuracy in network traffic anomaly detection, a network traffic anomaly detection model (FR-APPSO BiLSTM) based on feature reduction and bidirectional long short-term memory (LSTM) neural network optimization is proposed. First, the feature dimensions are divided by hierarchical clustering according to the similarity distance between data features, and the features with high correlation are divided into the same feature subset. Second, an automatic encoder is used to reduce each feature subset, eliminating redundant information, and reducing the computational complexity of the detection data. Then, a particle swarm optimization algorithm based on adaptive updating of variables and dynamic adjustment of parameters (APPSO) is proposed, which is used to optimize the parameters of the bidirectional LSTM neural network (BiLSTM). Finally, the optimized BiLSTM is used as a classifier to model network traffic anomaly detection using the reduced feature data. Experiments based on NSL-KDD, UNSW-NB15, and CICIDS-2017 datasets show that the proposed FR-APPSO-BiLSTM model can effectively reduce data features, improve the accuracy of detection, and the performance of network traffic anomaly detection.

Target PCA: Transfer learning large dimensional panel data

This paper develops a novel method to estimate a latent factor model for a large target panel with missing observations by optimally using the information from auxiliary panel data sets. We refer to our estimator as target-PCA. Transfer learning from auxiliary panel data allows us to deal with a large fraction of missing observations and weak signals in the target panel. We show that our estimator is more efficient and can consistently estimate weak factors, which are not identifiable with conventional methods. We provide the asymptotic inferential theory for target-PCA under very general assumptions on the approximate factor model and missing patterns. In an empirical study of imputing data in a mixed-frequency macroeconomic panel, we demonstrate that target-PCA significantly outperforms all benchmark methods.

Robust estimation in regression and classification methods for large dimensional data

Robust estimation in regression and classification methods for large dimensional data

An entropy-based density peak clustering for numerical gene expression datasets

In molecular biology, gene expression analysis is one of the important research areas which deals with identifying the genes having similar functionality known as co-expressed genes. Data mining techniques like clustering are frequently employed for grouping gene expressions with similar functional characteristics. Numerous such clustering techniques are available for gene expression analysis. Usually, gene expression datasets are a result of millions of measurements due to which they possess high dimensionality and noise which makes the conventional distance measures ineffective. On the other hand, entropy-based distance computation is much more efficient to capture the inhomogeneity in large dimensional data and is also quite insensitive to noise. To exploit these advantages, we propose a novel method to compute the density distribution of data points in high-dimensional and noisy gene expression datasets using the concept of entropy. After obtaining the density distribution, an existing technique known as “Extreme Clustering” is used to obtain the desired clusters present in the gene expressions dataset. The proposed technique is implemented and evaluated on diversified microarray gene expression datasets. Experiment results show that the proposed technique outperforms other popular density-based techniques in terms of cluster quality, robustness against noise, and biological significance of the genes within the clusters.

CausNet: generational orderings based search for optimal Bayesian networks via dynamic programming with parent set constraints

BackgroundFinding a globally optimal Bayesian Network using exhaustive search is a problem with super-exponential complexity, which severely restricts the number of variables that can feasibly be included. We implement a dynamic programming based algorithm with built-in dimensionality reduction and parent set identification. This reduces the search space substantially and can be applied to large-dimensional data. We use what we call ‘generational orderings’ based search for optimal networks, which is a novel way to efficiently search the space of possible networks given the possible parent sets. The algorithm supports both continuous and categorical data, as well as continuous, binary and survival outcomes.ResultsWe demonstrate the efficacy of our algorithm on both synthetic and real data. In simulations, our algorithm performs better than three state-of-art algorithms that are currently used extensively. We then apply it to an Ovarian Cancer gene expression dataset with 513 genes and a survival outcome. Our algorithm is able to find an optimal network describing the disease pathway consisting of 6 genes leading to the outcome node in just 3.4 min on a personal computer with a 2.3 GHz Intel Core i9 processor with 16 GB RAM.ConclusionsOur generational orderings based search for optimal networks is both an efficient and highly scalable approach for finding optimal Bayesian Networks and can be applied to 1000 s of variables. Using specifiable parameters—correlation, FDR cutoffs, and in-degree—one can increase or decrease the number of nodes and density of the networks. Availability of two scoring option—BIC and Bge—and implementation for survival outcomes and mixed data types makes our algorithm very suitable for many types of high dimensional data in a variety of fields.

A dual perspective inflation analysis of China with large dimensional data: an application of large VARs model

ABSTRACT The inflation rate was an important indicator for monitoring macroeconomic operations. In this study, we analysed the reasons for the divergence of consumer-side inflation (CPI) and production-side inflation (PPI) trends through large-dimensional data, so as to consider the impact of various macroeconomic factors on the inflation. From the perspective of the diversity of inflation rate formation mechanisms and model fitted performance, we used the large BVAR/large TVP-VAR models for analysis, considered the analysis capability and time-varying adaptability to large dimensional data. In view of the impulse response, we found that the impulse responses of CPI were relatively flat, and the responses of inflation were not exactly the same in both perspectives. The effects of various variables on the inflation in China were highly time-varying. In summary, the macro factors that led to significantly different responses and should be focused on energy prices, chemical prices, food prices, 7-day Inter Bank Offered Rate, narrow measure of money supply and the interaction between PPI and CPI. Accordingly, policymakers could develop more thoughtful and detailed scenarios from our findings. Moreover, the impulse response mechanism of each indicator to the inflation has validated the applicability of large dimensional analysis in macroeconomic studies.

Sparse Estimation in Finite Mixture of Accelerated Failure Time and Mixture of Regression Models with R Package fmrs

Variable selection in large-dimensional data has been extensively studied in different settings over the past decades. In a recent article, Shokoohi et. al. [29, DOI:10.1214/18-AOAS1198] proposed a method for variable selection in finite mixture of accelerated failure time regression models for studies on time-to-event data to capture heterogeneity within the population and account for censoring. In this paper, we introduce the fmrs package, which implements the variable selection methodology for such models. Furthermore, as a byproduct, the fmrs package facilitates variable selection in finite mixture regression models. The package also incorporates a tuning parameter selection mechanism based on component-wise bic. Commonly used penalties, such as Least Absolute Shrinkage and Selection Operator, and Smoothly Clipped Absolute Deviation, are integrated into fmrs. Additionally, the package offers an option for non-mixture regression models. The C language is chosen to boost the optimization speed. We provide an overview of the fmrs principles and the strategies employed for optimization. Hands-on illustrations are presented to help users get acquainted with fmrs. Finally, we apply fmrs to a lung cancer dataset and observe that a two-component mixture model reveals a subgroup with a more aggressive form of the disease, displaying a lower survival time.

Internet Appendix for Target PCA: Transfer Learning Large Dimensional Panel Data

Internet Appendix for Target PCA: Transfer Learning Large Dimensional Panel Data