Problem Of Data Redundancy Research Articles

Optimizing Storage Efficiency Through Pipelining and Parallelization for Enhanced Data Deduplication

Nowadays, huge devices are connected to the internet to deal with many cloud environments for storing data efficiently. Also, it is used to store big data determined in business activities. However, the data storage in the cloud avoids duplicate copies of files and reduces complexity issues. However, this platform is utilized by many users for storing the file in cloud and this creates a data redundancy problem. Also, fingerprint indexing is a major issue in data storage in the cloud. So in order to overcome this issue a novel Enhanced Arithmetic Optimization Algorithm with Mixed-Kernel-based Extreme Learning Machine-based Random Forest (EAOA-MKELM-RF) method is proposed that overcome the deduplication issues. In spite, the scalability issues are solved, and the limitations determined in fingerprint indexing. Further, the pipelining and parallelizing hash function values are utilized to address the scalability issue in the deduplication process. The data storage container organizes the programs and files into data blocks, metadata files and nonduplicate blocks. However, the proposed methods determine the similarity issues and accelerate the fingerprint queries for better scalability. The datasets employed to evaluate the proposed approach are CorrAL-100 dataset, Madelon dataset, XOR-100 dataset, Infocom’05 dataset and Linux kernel dataset. The experimentation results demonstrated that the proposed method attained a superior accuracy by 96.8% in terms of restoration accuracy, robustness and delivery ratio. Additionally, the proposed approach reduces buffer time, improving overall system performance. This validation optimized the storage resources, diminished storage costs and improved overall data management.

A new IPCA-CPSO-BP Model for Predicting Gas Emission in Underground Coal Mines

Gas has become the primary factor restricting the safe and efficient production of coal mines. Gas emission prediction model is very important for mine gas emission prediction and gas disaster prevention. The improved principal component analysis (IPCA) was used to reduce the dimension of 13 influencing factors of gas emission, and the CPSO-BP neural network prediction model was built with MATLAB software to predict the absolute gas emission of Zhongtai Mining.The results show that the cumulative contribution rate of principal components is not less than 95%, and the number of principal components after improvement is reduced from the original 6 to 3, which effectively solves the problem of excessive principal components and data redundancy caused by the correlation difference between various influencing factors and gas emission amount. At the same time, the optimized particle swarm optimization algorithm alleviates the trouble that the particle swarm optimization algorithm is prone to fall into the local optimal solution. By coupling the improved particle swarm optimization algorithm with BP neural network, the problem of BP neural network's over-dependence on the initial value of weights and thresholds is solved, and the optimal initial weights and thresholds are provided for BP neural network, improving the accuracy of model prediction results. The average relative error of the original BP neural network is 1.6638%, the normalized mean square error is 0.3155, and the regression correlation index is 0.8157. The IPCA-CPSO-BP prediction model decreased to 0.5749%, 0.1143 and 0.9758, respectively. IPCA improves the data dimensionality reduction ability of principal component analysis, IPCA-CPSO-BP enhances the stability of particle swarm optimization algorithm, and significantly improves the prediction accuracy of BP model. The prediction trend is highly consistent with the actual value, which verifies the reliability of the model and provides strong data support for mine gas prevention and control.

A High-Precision Monitoring Method Based on SVM Regression for Multivariate Quantitative Analysis of PID Response to VOC Signals

In the moist environment of soil-water-air, there is a problem of low accuracy in monitoring volatile organic compounds (VOCs) using a photoionization detector (PID). This study is based on the PID water-soil-gas VOC online monitor developed by this group, online monitoring of the concentration of different constituents of VOCs in different production enterprises of the petroleum and chemical industries in Shandong Province, with the concentration of the laboratory test, to build a relevant model. The correlation coefficient about the PID test concentration and the actual concentration correlation coefficient was obtained through the collection of a large number of data trainings. Based on the application of PID in VOC monitoring, the establishment of a PID high-precision calibration model is important for the precise monitoring of VOCs. In this paper, multiple quantitative analyses were conducted, based on SVM regression of PID response to VOC signals, to study the high-precision VOC monitoring method. To select the response signals of PID under different concentrations of environmental VOCs measured by the research group, first, the PID response to VOC signals was modeled using the support vector machine principle to verify the effect of traditional SVM regression. For the problem of raw data redundancy, calculate the time-domain and frequency-domain characteristics of the PID signal, and conduct the principal component analysis of the time-domain of the PID signal. In order to make the SVM regression more generalized and robust, the selection of kernel function parameters and penalty factor of SVM is optimized by genetic algorithm. By comparing the accuracy of PID calibration models such as PID signal feature extraction, SVM regression, and principal component analysis SVM regression, the superiority of photoionization detector using the signal feature extraction PCA-GA-SVM method to monitor VOCs is verified.

Optimized clustering sample selection for spectral reflectance recovery

The accuracy of spectral recovery depends heavily on the selection of an appropriate sample set, so the optimized sample selection by clustering strategy can improve the spectral recovery results. This paper presents a sample optimization method that combines hierarchical clustering and K-mean angle similar clustering to achieve this process. The proposed method employs the hierarchical clustering to divide the training sample dataset into 15 subspaces and obtain 15 subspace centroids. The similarity distance is then calculated between the testing sample and each subspace samples, and the subspace with the sample having the smallest distance is selected. The testing sample is utilized as a priori centroid, which clusters the optimal subspace by competition with the centroid of the subspace selected. This iterative process continues until the centroid of the subspace remains unaltered. Finally, the training samples within the optimal subspace use to recover spectral reflectance through Euclidean distance weighting. Experimental results demonstrate that the proposed method outperforms existing methods in terms of spectral and colorimetric accuracy, as well as stability and robustness. This research provides a solution to the problem of data redundancy in the spectral recovery process and enhances the accuracy and efficiency of spectral recovery.

Rating of Stability of Russian Companies in Oil and Gas and Electric Power Industries Based on Interval Volatility

The authors propose an instrumental apparatus for calculating the ratings of Russian companies in the oil and gas and electric power industries based on a weighting method, risk assessment using the minimax criterion and an intellectual tree structure. The relevance of the developed system is justified by the need to create Russian rating systems for companies that will represent their actual state and place in the analyzed group. The problem of data redundancy has been solved by using the hierarchical principle for the isolated indexing of absolute and relative indicators from the financial statements of the companies in question into sub-indexes, with further integral indexing and correction for the volatility of changes over the past three years. The authors used a financial analytics apparatus based on the regular financial (accounting) statements of companies according to accepted forms, and balance sheets and reports on financial results were applied. The authors developed and tested a methodology for sub-indexing important indicators of financial statements: capital structure—equity, debt capital, highly liquid assets (cash and settlement accounts in reliable banks, short-term financial investments) and net profit. Based on the results of the analysis, recommendations are provided for the long-term development of the energy business.

Combined Feature Selection Scheme for Banking Modeling

Machine learning methods have been successful in various aspects of bank lending. Banks have accumulated huge amounts of data about borrowers over the years of application. On the one hand, this made it possible to predict borrower behavior more accurately, on the other, it gave rise to the problem a problem of data redundancy, which greatly complicates the model development. Methods of feature selection, which allows to improve the quality of models, are apply to solve this problem. Feature selection methods can be divided into three main types: filters, wrappers, and embedded methods. Filters are simple and time-efficient methods that may help discover one-dimensional relations. Wrappers and embedded methods are more effective in feature selection, because they account for multi-dimensional relationships, but these methods are resource-consuming and may fail to process large samples with many features. In this article, the authors propose a combined feature selection scheme (CFSS), in which the first stages of selection use coarse filters, and on the final — wrappers for high-quality selection. This architecture lets us increase the quality of selection and reduce the time necessary to process large multi-dimensional samples, which are used in the development of industrial models. Experiments conducted by authors for four types of bank modelling tasks (survey scoring, behavioral scoring, customer response to cross-selling, and delayed debt collection) have shown that the proposed method better than classical methods containing only filters or only wrappers.

Skewed Evolving Data Streams Classification with Actionable Knowledge Extraction using Data Approximation and Adaptive Classification Framework

Skewed evolving data stream (SEDS) classification is a challenging research problem for online streaming data applications. The fundamental challenges in streaming data classification are class imbalance and concept drift. However, recently, either independently or together, the two topics have received enough attention; the data redundancy while performing stream data mining and classification remains unexplored. Moreover, the existing solutions for the classification of SEDSs have focused on solving concept drift and/or class imbalance problems using the sliding window mechanism, which leads to higher computational complexity and data redundancy problems. To end this, we propose a novel Adaptive Data Stream Classification (ADSC) framework for solving the concept drift, class imbalance, and data redundancy problems with higher computational and classification efficiency. Data approximation, adaptive clustering, classification, and actionable knowledge extraction are the major phases of ADSC. For the purpose of approximating unique items in the data stream with data pre-processing during the data approximation phase, we develop the Flajolet Martin (FM) algorithm. The periodically approximated tuples are grouped into distinct classes using an adaptive clustering algorithm to address the problem of concept drift and class imbalance. In the classification phase, the supervised classifiers are employed to classify the unknown incoming data streams into either of the classes discovered by the adaptive clustering algorithm. We then extract the actionable knowledge using classified skewed evolved data stream information for the end user decision-making process. The ADSC framework is empirically assessed utilizing two streaming datasets regarding classification and computing efficiency factors. The experimental results shows the better efficiency of the proposed ADSC framework as compared with existing classification methods.

A Novel Image Encryption Technique Using Multi-Coupled Map Lattice System with Generalized Symmetric Map and Adaptive Control Parameter

Image and video data make up a significant portion of the content shared over the Internet and social media. The use of image and video communication allows more information to be shared while simultaneously presenting higher risks in terms of data security. The traditional encryption schemes are general purpose; however, to encrypt image and video data, application-specific encryption solutions are needed. An image or a video frame comprises a two-dimensional matrix where pixel intensity values are integers in range [0,255], leading to data redundancy problems. Moreover, the bulk amount of image and video data adds another challenge when deploying security primitives. In this paper, a novel coupled map lattice system-based image cryptosystem has been proposed that uses generalised symmetric maps for generation of pseudo-random sequences. The generalization of symmetric maps allows the user to choose the source of pseudo-random sequence generation by varying a single control parameter. Other adaptive control parameters ensure an adequate degree of randomness in the generated sequences. The proposed encryption system relies on three independent sources of pseudo-random sequence generators, which are further re-randomized before the final encryption process. Comprehensive experimentation has been performed to test the proposed system against various attack models on publicly available datasets. A detailed comparative analysis has also been conducted with existing state-of-the-art image encryption techniques. Results show that the proposed algorithm provides high information entropy, negative correlation, large key space, and high sensitivity to key variations, and is resistant to various types of attacks, including chosen-text, statistical, and differential attacks.

Knowledge Graph-Based Assembly Resource Knowledge Reuse towards Complex Product Assembly Process

Assembly process designers typically confront the challenge of seeking information out of large volumes of non-structured files with a view to supporting the decision-making to be made. It is a leading concern that embedding data in text documents can hardly be retrieved semantically in order to facilitate decision-making with timely support. For tackling this gap, we propose in this paper a knowledge graph-based approach used to merge and retrieve information decided to be relevant within an engineering context. The proposed approach is to establish a multidimensional integrated assembly resource knowledge graph (ARKG) based on the structure of function-structure-assembly procedure-assembly resource, and this multidimensional integrated structure can well accomplish the retrieval of related knowledge. The upper semantic framework of ARKG is established by the assembly resource ontology model, which is a semantic-type framework involving multiple domains of knowledge to create instantiated data reflecting the full profile of the assembly resource for obtaining structured data of ARKG while avoiding the data redundancy problem. The ARKG method is validated through assembly scenario of the aircraft, and the results show the effectiveness and accuracy of the ARKG used by the assembly process designer in the assembly process design phase for retrieving the target knowledge of the assembly resources.

Obtaining Pedestrian Interest Regions Using Experimental Methods

In pedestrian simulation, using the region of interest of pedestrians can solve the problem of data redundancy and complexity and lack of representation in the pedestrian saccade range, and establish an efficient pedestrian collision avoidance behavior model. The real-time eye movement data of unidirectional and bidirectional pedestrian flow were collected by eye tracker, and the mathematical models of pedestrian flow density, radius and angle of each region of interest were established. Analyzing the experimental data, it is found that the range of pedestrian interest region is affected by pedestrian density. The angle of the pedestrian interest region is generally not more than 90°, and the radius of the high-interest region and the low-interest region are negatively correlated with the pedestrian density. When the pedestrian density reaches a certain degree, the radius of the high-interest region and the low-interest region approach the fixed value respectively.

Embedded parallel Fourier ptychographic microscopy reconstruction system.

Fourier ptychographic microscopy (FPM) has attracted a wide range of focus for its ability of large space-bandwidth product and quantitative phase imaging. It is a typical computational imaging technique that jointly optimizes imaging hardware and reconstruction algorithms. The data redundancy and inverse problem algorithms are the sources of FPM's excellent performance. But at the same time, this large amount of data processing and complex algorithms also evidently reduce the imaging speed. To accelerate the FPM reconstruction speed, we proposed a fast FPM reconstruction framework consisting of three levels of parallel computation and implemented it with an embedded computing module. In the conventional FPM framework, the sample image is divided into multiple sub-regions to process separately because the illumination angles and defocus distances for different sub-regions may also be different. Our parallel framework first performs digital refocusing and high-resolution reconstruction for each sub-region separately and then stitches the complex sub-regions together to obtain the final high-resolution complex image. The feasibility of the proposed parallel FPM reconstruction framework is verified with different experimental results acquired with the system we built.

Wetland mapping based on feature optimization and Google Earth Engine in Zhalong Nature Reserve

Wetlands have very important and irreplaceable ecological service functions in protecting biodiversity and maintaining ecological balance. Due to the unique hydrological characteristics of wetlands, large-scale wetland mapping methods are usually time-consuming, labor intensive, and expensive. With the rise of big data and cloud computing platforms, there are opportunities for long time series and large-scale spatial data processing that improve large-scale wetland mapping. In this study, the Sentinel-1, Sentinel-2, and digital elevation model derived from the Google Earth Engine were used to extract the wetland extent in the Zhalong Nature Reserve based on the Jeffries–Matusita (JM) distance for feature optimization and random forest. Research shows the following: (1) The JM distance for each feature showed that the optical features had the highest separation in all five features and (2) the classification after feature optimization performed best. User accuracy (UA) and producer accuracy (PA) for wetland reached 89.64% and 83.00%, respectively, and overall accuracy reached 90.76%. After feature optimization, PA and UA of wetland were each increased 4.26% and 1.07%, respectively, and the number of features was reduced from 46 to 28. The method used in this study is highly accurate in wetland mapping and can effectively avoid the problem of data redundancy.

Intelligent Multi-AUG Ocean Data Collection Scheme in Maritime Wireless Communication Network

The new generation of wireless networks is strengthening ocean development capabilities. The deep-sea detection ability of the offshore platform is being improved by Ocean Internet of Everything (OIoE) system formed by various equipment. This paper studies the application of a new generation of Artificial Intelligence (AI) wireless communication technology in marine monitoring platforms. The purpose is to solve the problem of high data redundancy and high energy consumption in multi-Automatic Underwater Glider (AUG) monitoring. In response to these problems, this paper proposes a matrix completion-based Sampling Points Selection joint Multi-AUG Path Planning (SPS-MAPP) scheme, which includes AUGs path planning and collection information matrix completion. In parts of AUGs path planning, the SPS-MAPP designed a multi-AUG trajectory optimization method using intermittent satellite communication. It proposes a reinforcement learning reward rule based on the idea of artificial potential field, which can effectively increase the coverage of AUGs trajectory. In parts of information processing, the SPS-MAPP proposes a 3-D matrix completion method, which searches for key nodes and edge nodes by designing regular sample points and operators to recover the entire space information. The experimental results show that the proposed SPS-MAPP scheme can effectively improve the subsea monitoring capability of multi-AUG.

Opening the black box: interpretable machine learning for predictor finding of metabolic syndrome

ObjectiveThe internal workings ofmachine learning algorithms are complex and considered as low-interpretation "black box" models, making it difficult for domain experts to understand and trust these complex models. The study uses metabolic syndrome (MetS) as the entry point to analyze and evaluate the application value of model interpretability methods in dealing with difficult interpretation of predictive models.MethodsThe study collects data from a chain of health examination institution in Urumqi from 2017 ~ 2019, and performs 39,134 remaining data after preprocessing such as deletion and filling. RFE is used for feature selection to reduce redundancy; MetS risk prediction models (logistic, random forest, XGBoost) are built based on a feature subset, and accuracy, sensitivity, specificity, Youden index, and AUROC value are used to evaluate the model classification performance; post-hoc model-agnostic interpretation methods (variable importance, LIME) are used to interpret the results of the predictive model.ResultsEighteen physical examination indicators are screened out by RFE, which can effectively solve the problem of physical examination data redundancy. Random forest and XGBoost models have higher accuracy, sensitivity, specificity, Youden index, and AUROC values compared with logistic regression. XGBoost models have higher sensitivity, Youden index, and AUROC values compared with random forest. The study uses variable importance, LIME and PDP for global and local interpretation of the optimal MetS risk prediction model (XGBoost), and different interpretation methods have different insights into the interpretation of model results, which are more flexible in model selection and can visualize the process and reasons for the model to make decisions. The interpretable risk prediction model in this study can help to identify risk factors associated with MetS, and the results showed that in addition to the traditional risk factors such as overweight and obesity, hyperglycemia, hypertension, and dyslipidemia, MetS was also associated with other factors, including age, creatinine, uric acid, and alkaline phosphatase.ConclusionThe model interpretability methods are applied to the black box model, which can not only realize the flexibility of model application, but also make up for the uninterpretable defects of the model. Model interpretability methods can be used as a novel means of identifying variables that are more likely to be good predictors.

Multi-keyword searchable encryption for smart grid edge computing

Searchable encryption scheme allows users to search for encrypted data containing some keywords without decrypting data. It protects the data privacy of data owners while promoting efficient data access. The searchable encryption scheme only supports a single keyword, which is easy to cause the problem of data redundancy. It makes users have to filter again in a large amount of data. In order to achieve the search goal more accurately, researchers proposed multi-keyword searchable encryption scheme. In addition, searchable encryption schemes that only support a single keyword are vulnerable to the disclosure of user access and search pattern. Based on these two reasons, we propose a searchable encryption scheme based on multi- keyword, which supports users to input more complex retrieval requests, and users can get a better sense of search experience. At the same time, it reduces the possibility of leakage of user access and search pattern to a certain extent. We also define the security model and prove that the scheme can resist the internal keyword guessing attack under the standard model. Finally, we also compare theoretical performance with other schemes and evaluate the experimental performance of our scheme.

A novel image encryption scheme based on chaotic apertured fractional Mellin transform and its filter bank

A novel image encryption scheme based on chaotic apertured fractional Mellin transform and its filter bank has been proposed in this paper. A nonlinear chaotic apertured fractional Mellin transform (ACFrMT) and its filter bank (ACFrMT-FB) are proposed, where a pair of complementary chaotic apertures is introduced into FrMT to form filter bank and a special algorithm is employed to solve the problem of data redundancy in filter banks. In order to improve the sensitivity of chaotic aperture, two random phases based on chaotic aperture (RPCA) are generated as phase masks. The proposed encryption algorithm not only provides huge key space, but also greatly improves the sensitivities of fractional order p and wavelength λ. The maximum values of those sensitivity can reach 10-16 and 10-22. The experiment results prove that the proposed encryption scheme is efficient and secure. The results are: sensitivity attack analysis (> 94.77%), differential attack analysis (average NPCR =99.6120% and average UACI = 33.4888%), entropy values (average value 7.99928), correlation coefficient values (almost 0), and cipher key space analysis (at least 1.677×2341).

User‐optimized data transmission scheduling based on edge community service in opportunistic social network

In order to improve transmission performance, opportunistic social networks are widely used in 5G mobile network communication for network congestion. However, transferring large amounts of data in an instant will lead to data redundancy problems, which will affect the transmission quality. The authors propose a method called user-optimized data transmission scheduling based on edge community service in opportunistic social networks (ECSUO). This method constructs the edge community service model by introducing mobile edge computing into the opportunistic social network. Then several scheduling factors are set to coordinate the dynamic priority assessment. The simulation results show that ECSUO performs better than other classical opportunistic complex network algorithms in a variety of transmission quality evaluation indexes, and this method also has good applicability when applied to different transmission models.

Perceived Integrity of Distributed Streaming Media Based on AWTC-TT Algorithm Optimization

With the development of economy, more and more attention has been paid to the monitoring system, which provides a reliable and powerful guarantee for people’s daily life, property security, and national security. The intelligent video surveillance introduces computer vision-related technologies into traditional video surveillance and realizes the analysis and understanding of video data without artificial dependence to obtain valuable target information in the perceived video data. On this basis, functions such as abnormal event monitoring and real-time alarm are realized. Distributed streaming media monitoring has changed the manual-based monitoring and content analysis modes of traditional monitoring, but the high-complexity calculations such as motion estimation and motion compensation in the encoding process increase the burden of monitoring and sensing equipment. Especially with the development of wireless multimedia technology, the traditional video coding has been unable to meet the requirements of monitoring and sensing equipment in the monitoring system based on wireless technology. This paper proposes an adaptive weighted tensor completion algorithm to complete the repair of streaming media data perceived by ordinary sensing devices. In the proposed algorithm, considering the unbalanced information distribution and data redundancy problems that may exist in the data, the tensor data is adjusted according to the approximate solution algorithm to obtain tensor data that only retains important information and the information distribution is more balanced and reasonable. In the iterative solution process, in order to better map the impact of each dimension of data in the repair process, an adaptive weighting mechanism is proposed according to the data characteristics to obtain the corresponding weight value of each dimension of data. Finally, the proposed approximate tensor solving algorithm and adaptive weighting mechanism are applied to a simple low-rank tensor completeness algorithm based on tensor columns to form the algorithm of this paper, and it is used to repair perceptual streaming media data with data missing problems. The experimental results show that the algorithm in this paper can improve the perceived streaming media data quality by 3% based on the known data information and maintain an advantage of 2% in average processing time. It avoids the replacement of sensing equipment and also provides data quality assurance for subsequent sensing streaming media content analysis. It has certain research significance for the development of monitoring system with artificial intelligence management for target perception and tracking.

Remaining useful life prediction for rolling bearings using multi-layer grid search and LSTM

Estimating the remaining useful life (RUL) of equipment is critical for ensuring the safe operation of machinery and reducing maintenance losses. For the existing RUL prediction, the problem of data redundancy and initial prediction time dramatically affects the prediction results. Therefore, this paper proposes a long short-term memory network (LSTM) RUL prediction algorithm that is based on multi-layer grid search (MLGS) optimization. This method integrates feature data and optimizes network parameters to ensure accuracy and effectively predict the non-stationary degradation of the bearing. Firstly, this paper uses a data fusion method to extract low dimensional feature vectors of running data, and the multi-feature fusion is performed to obtain the principal component index. Then, the MLGS is used to optimize the network overparameters. The effect of the initial measurement time on prediction results can be reduced, and the calculation speed can be improved. Finally, the IMS data set of NASA is used for verification and comparative testing. The test results show that the proposed RUL prediction algorithm can effectively reduce the influence of the initial prediction time on the prediction accuracy compared with other prediction methods.

Prediction of cement specific surface area in the cement mill grinding process: A method combining dual-frequency principal component analysis and extreme gradient boosting

The cement specific surface area is an important indicator of cement quality. The accurate prediction of the cement specific surface area aims to guide operators to control the cement grinding process to improve product quality while reducing system energy consumption. However, due to the complexity of the cement grinding process, the process variables have coupling, time-varying delay, nonlinear characteristics, and different sampling frequency. Herein, we proposed the specific surface area prediction model, which combined dual-frequency principal component analysis and extreme gradient boosting (DF-PCA-XGB). In order to solve the problem of difficulty in modeling due to different sampling intervals of related data, this paper analyzes the low-frequency sampling data and high-frequency sampling data under multiple working conditions, and establishes prediction models respectively. Aiming at the data redundancy problem of high-frequency and low-frequency variable data in the introduced time window, a method based on the combination of principal component analysis (PCA) and extreme gradient boosting (XGB) cross-validation is proposed to reduce data redundancy while retaining most of the characteristics of the data. The final specific surface area prediction results were obtained by weighting the high-frequency data model and the low-frequency data model. The simulation results showed that the prediction method in this paper can improve the prediction accuracy of the specific surface area of the finished cement product under multiple working conditions with high stability and has promising application in the cement manufacturing process.