High Dimensional Sparse Dataset Research Articles

Clustering statistical method of high dimensional sparse data based on fuzzy data

Abstract Developing effective clustering and statistical methods for high-dimensional sparse data presents unique challenges compared to traditional low-dimensional data. To address this, a novel approach is proposed, leveraging fuzzy data principles to enhance the clustering and statistical performance of high-dimensional sparse datasets. The method builds upon the fuzzy C-means clustering algorithm, introducing key modifications for better suitability to high-dimensional sparse data. One crucial enhancement involves tackling the local optimization problem by optimizing the initial clustering center, significantly reducing clustering statistical time. Replacing the original Euclidean distance with cosine distance improves the clustering and statistical performance of high-dimensional sparse data. Experimental results have shown that this method has superior clustering statistical performance when the data dimensions are different. When the data dimension is low, and the blocking ratio is 10%, the clustering statistical effect is optimal. When the data dimension is high, and the blocking ratio is 40%, the clustering statistical effect is optimal. This method has higher hit rates and clustering statistical efficiency at different sparsity levels.

Efficient traffic-based IoT device identification using a feature selection approach with Lévy flight-based sine chaotic sub-swarm binary honey badger algorithm

Internet of Things (IoT) refers to the various devices connected to the Internet, enabling them to communicate and transmit data with each other. The rapid development of the IoT also brings security and other problems in cyberspace. In this case, device identification is a crucial tool for IoT security issues, which can detect and prevent cyber-attacks. However, device identification has some challenges on IoT traffic datasets, such as large-scale and high-dimensional sparse datasets, features prone to security vulnerabilities, and identification of IP and non-IP devices, which affects the performance of classifiers. Feature selection can be seen as an effective data preprocessing technique in IoT device identification, which may improve the performance of classification and reduce the computational complexity of IoT device identification. In this paper, we propose a traffic-based IoT device identification model using a novel wrapper feature selection approach based on an improved and efficient method, which we call Lévy flight-based sine chaotic sub-swarm binary honey badger algorithm (LS2-BHBA). Specifically, four improved factors are employed in LS2-BHBA to expand the search scope, balance the exploration and exploitation phases, and enhance the search capability. In addition, a binary mechanism is implemented to enhance the suitability of the proposed LS2-BHBA for feature selection in IoT device identification. The experimental results on several real IoT traffic datasets denote that LS2-BHBA can reduce the number of features to 10%, and the classification accuracy can reach 98%, which outperforms some classical and latest comparison algorithms in the feature selection of IoT device identification.

High-dimensional sparse vine copula regression with application to genomic prediction.

High-dimensional data sets are often available in genome-enabled predictions. Such data sets include nonlinear relationships with complex dependence structures. For such situations, vine copula-based (quantile) regression is an important tool. However, the current vine copula-based regression approaches do not scale up to high and ultra-high dimensions. To perform high-dimensional sparse vine copula-based regression, we propose 2 methods. First, we show their superiority regarding computational complexity over the existing methods. Second, we define relevant, irrelevant, and redundant explanatory variables for quantile regression. Then, we show our method's power in selecting relevant variables and prediction accuracy in high-dimensional sparse data sets via simulation studies. Next, we apply the proposed methods to the high-dimensional real data, aiming at the genomic prediction of maize traits. Some data processing and feature extraction steps for the real data are further discussed. Finally, we show the advantage of our methods over linear models and quantile regression forests in simulation studies and real data applications.

Implementation of Personalized Scenic Spot Recommendation Algorithm Based on Generalized Regression Neural Network for 5G Smart Tourism System.

On the basis of the analysis of the evolution dynamics and the process of smart tourism service, this paper constructs the evolutionary game model of smart tourism service and reveals the evolution mechanism of smart tourism service based on the network platform. Based on the strategic main line of “advantages,” it proposes the design ideas and overall framework of the smart tourism service model based on the network platform, including the smart tourism information interactive service model, the element collaborative service model, and the value cocreation service model. The comparison of recommendation results shows that the recommendation error of the genetically improved generalized regression neural network algorithm is reduced, and the recommendation accuracy is better than that of the unimproved generalized regression neural network algorithm. In the recommendation scenario of click-through rate recommendation, the existing recommendation models are difficult to meet the functions of memory and generalization at the same time and cannot fully mine and combine low-level features, and the model parameters of the deep learning model are difficult to learn under the high-dimensional sparse data set of the recommendation system. To solve the problem of generalization, this paper proposes a deep CTR recommendation model based on the gradient boosting tree and factorization machine. It can fully mine low-level feature information and automatically realize low-level feature combination, which can better learn model parameters on high-dimensional sparse data sets, and the recommendation results are no longer overgeneralized. In this paper, simulation experiments are carried out on the data set, and the related recommendation models are compared. The experimental results show that the model proposed in this paper achieves better results in both the AUC (area under ROC curve) evaluation index and the cross-entropy evaluation index.

Mapping High Dimensional Sparse Customer Requirements into Product Configurations

Mapping customer requirements into product configurations is a crucial step for product design, while, customers express their needs ambiguously and locally due to the lack of domain knowledge. Thus the data mining process of customer requirements might result in fragmental information with high dimensional sparsity, leading the mapping procedure risk uncertainty and complexity. The Expert Judgment is widely applied against that background since there is no formal requirements for systematic or structural data. However, there are concerns on the repeatability and bias for Expert Judgment. In this study, an integrated method by adjusted Local Linear Embedding (LLE) and Naïve Bayes (NB) classifier is proposed to map high dimensional sparse customer requirements to product configurations. The integrated method adjusts classical LLE to preprocess high dimensional sparse dataset to satisfy the prerequisite of NB for classifying different customer requirements to corresponding product configurations. Compared with Expert Judgment, the adjusted LLE with NB performs much better in a real-world Tablet PC design case both in accuracy and robustness.