Missing Ratio Research Articles

Beyond Belady to Attain a Seemingly Unattainable Byte Miss Ratio for Content Delivery Networks

Beyond Belady to Attain a Seemingly Unattainable Byte Miss Ratio for Content Delivery Networks

A novel Kalman smoothing (Ks) − Long Short-Term Memory (LSTM) hybrid model for filling in short- and long-term missing values in significant wave height

Continuous recordings of significant wave height (SWH) are of significant importance to facilitate the development and application of wave energy assessments. However, SWH recording is often interrupted by a variety of factors, including severe weather, apparatus failure, etc. The presence of short-term intermittent and long-term continuous missing values is recognized as a significant challenge for the accurate analysis and energy assessment of wave data. To effectively fill in the short- and long-term missing values simultaneously, a hybrid Kalman smoothing (KS) − Long Short-Term Memory (LSTM) model, which is applied to fill in SWH for the first time, is proposed. Initially, the short-term intermittent missing values, with missing ratios of 10%-50%, are filled by using the KS method, achieving a root mean square error (RMSE) as low as 0.082. Based on short-term missing values addressed by KS, the LSTM model is introduced to effectively predict the long-term continuous missing values. The maximum RMSE reduction rate of KS-LSTM is reduced by 49.6%, 59.4%, and 57.8% compared to KS-ARIMA, LSTM, and ARIMA, respectively, within the short-term intermittent missing ratios of 10%-50%. The maximum reduction in mean square error (MSE) is observed to be 71.4%, 83.6%, and 82.1%. Similarly, the maximum reduction in mean absolute error (MAE) is 74.6%, 61.1%, and 55.5%. The lowest prediction RMSE of long-term missing values by KS-LSTM is only 0.091, which demonstrates that the effectiveness of filling in both short-term and long-term missing values simultaneously by the KS-LSTM.

Optimised multiple data partitions for cluster-wise imputation of missing values in gene expression data

It is commonly agreed that the quality of data analysis may be degraded by the presence of missing data. In various domains such as bioinformatics, an effective tool is required for the discovery of knowledge from gene expression datasets. One may simply ignore defected samples, while others attempt to either make an algorithm robust to the problem or develop an imputation technique to fill in missing values. This research focuses on the latter and introduces a new hybridisation of cluster- and neighbour-based references to generate an accurate estimate. It also proposes a novel exploitation of multiple clusterings as the source of cluster-wise information, instead of a single data partition that has been studied by existing methods. These data partitions are selected from a pool of base clusterings with respect to both quality and diversity criteria. Another hybridisation is thus established between swarm intelligence and this search problem. In particular, the algorithm of artificial bee colony (ABC) is explored, with two new operators being invented to allow an evolution of solutions, or food sources for bees. Also, two different imputation strategies are provided to generate entries of missing entries in a data matrix, called cluster-only and cluster-neighbour. Based on published gene expression datasets and different experimental settings, the resulting models usually outperform their baselines and recent approaches, which make use of cluster analysis or devise an intelligent determination of nearest neighbours. Furthermore, they have proven competitive to a benchmark technique belonging to the global approach, especially with high missing ratios. Further extensions to iterative refinement and supervised imputation are discussed in addition to parameter analysis.

Advanced machine learning for missing petrophysical property imputation applied to improve the characterization of carbonate reservoirs

Missing data is a common cause of uncertainty in reservoir characterization, especially in core analysis of porosity and permeability. This is due to the high cost and time required to gather core samples while drilling through the entire reservoir thickness. Therefore, the ability to collect core samples and directly assess petrophysical parameters is restricted by the small proportion of the reservoir thickness that can be measured by the retrieved cores. As a result, the absence of data has the potential to lead to inconsistent and imprecise classification of reservoirs and geomodeling, thereby increasing reservoir uncertainty. Therefore, imputation algorithms are needed to estimate petrophysical properties for incomplete intervals. This study comprehensively compares seven imputation techniques for predicting missing horizontal and vertical core permeability and core porosity data in a well drilled through a carbonate reservoir in a southern Iraqi gas field. The performance of each method was assessed using relative bias (RB) and reasonable descriptive statistical distributions (histogram, log view, and box plots of data ratios to compare the imputed and original data), and robustness to outliers (RtO). The results revealed that the random imputation of missing data (RIMD) technique is the most effective algorithm in addressing the 30.5% missing data (vertical permeability), resulting in the attainment of the RB value closest to zero (0.04076191) compared to other algorithms. Principal component analysis (PCA) and random forest (RF) yielded the most favorable results in terms of handling missing values and outliers for the 15% missing ratio dataset (core porosity and horizontal permeability). The advantages of these approaches are related to their capacity to independently and accurately impute the missing data without requiring extra information, such as well logging records. Additionally, the multiple imputation and machine learning algorithms fill missing values with adequate numbers based on dataset variable distributions and correlations. Multiple plausible values quantify the uncertainty in estimating missing data, avoiding false precision (as with single imputation). Also, the novel workflow involving multiple imputation methods accurately take into account the correlations and variances between the variables. The novel workflow developed for missing data imputation has the potential to be applied to other clastic and carbonate reservoirs in oil and gas fields. The entire data imputation workflow was conducted, validated, visualized by codes in R, a robust open-source statistical computing software.

Low-Rank Kernel Tensor Learning for Incomplete Multi-View Clustering

Incomplete Multiple Kernel Clustering algorithms, which aim to learn a common latent representation from pre-constructed incomplete multiple kernels from the original data, followed by k-means for clustering. They have attracted intensive attention due to their high computational efficiency. However, our observation reveals that the imputation of these approaches for each kernel ignores the influence of other incomplete kernels. In light of this, we present a novel method called Low-Rank Kernel Tensor Learning for Incomplete Multiple Views Clustering (LRKT-IMVC) to address the above issue. Specifically, LRKT-IMVC first introduces the concept of kernel tensor to explore the inter-view correlations, and then the low-rank kernel tensor constraint is used to further capture the consistency information to impute missing kernel elements, thereby improving the quality of clustering. Moreover, we carefully design an alternative optimization method with promising convergence to solve the resulting optimization problem. The proposed method is compared with recent advances in experiments with different missing ratios on seven well-known datasets, demonstrating its effectiveness and the advantages of the proposed interpolation method.

Accelerating multi-tier storage cache simulations using knee detection

Storage cache hierarchies include diverse topologies, assorted parameters and policies, and devices with varied performance characteristics. Simulation enables efficient exploration of their configuration space while avoiding expensive physical experiments. Miss Ratio Curves (MRCs) efficiently characterize the performance of a cache over a range of cache sizes, revealing “key points” for cache simulation, such as knees in the curve that immediately follow sharp cliffs. Unfortunately, there are no automated techniques for efficiently finding key points in MRCs, and the cross-application of existing knee-detection algorithms yields inaccurate results.We present a multi-stage framework that identifies key points in any MRC, for both stack-based (e.g., LRU) and more sophisticated eviction algorithms (e.g., ARC). Our approach quickly locates candidates using efficient hash-based sampling, curve simplification, knee detection, and novel post-processing filters. We introduce Z-Method, a new multi-knee detection algorithm that employs statistical outlier detection to choose promising points robustly and efficiently.We evaluated our framework against seven other knee-detection algorithms, identifying key points in multi-tier MRCs with both ARC and LRU policies for 106 diverse real-world workloads. Compared to naïve approaches, our framework reduced the total number of points needed to accurately identify the best two-tier cache hierarchies by an average factor of approximately 5.5× for ARC and 7.7× for LRU.We also show how our framework can be used to seed the initial population for evolutionary algorithms. We ran 32,616 experiments requiring over three million cache simulations, on 151 samples, from three datasets, using a diverse set of population initialization techniques, evolutionary algorithms, knee-detection algorithms, cache replacement algorithms, and stopping criteria. Our results showed an overall acceleration rate of 34% across all configurations.

LIKFCM: Linear interpolation-based kernelized fuzzy C-means clustering imputation method for handling incomplete data

Addressing missing values is a persistent challenge in the field of data mining. The presence of incomplete data can significantly compromise the overall data quality. Consequently, it is crucial to handle incomplete data efficiently. This paper presents a novel approach for imputing missing values that incorporates Kernelized Fuzzy C-Means (KFCM) clustering and proposes a method termed LIKFCM, which combines its benefits with Linear Interpolation (LI). The proposed LIKFCM’s performance is assessed through a comparison against nine state-of-the-art imputation techniques (mean, median, LI, EMI, KNNI, KMI, FKMI, LIFCM, and LIPFCM) across ten widely used real-world datasets from the UCI repository with six combinations of missing ratios to assess the efficacy of the proposed imputation method. From the experimental results, it is evident that our proposed method outperforms the existing imputation methods with significant improvements in terms of RMSE & MAE for these datasets. Additionally, experiments examining the effect of missing values validate the robustness of the proposed approach by handling different missing ratios. The performance validation of the proposed approach against other state-of-the-art imputation methods has been conducted utilizing a Kendall’s W statistical test, involving a comparison of their mean ranks across different missing ratios. The outcomes indicate that LIKFCM has outperformed other imputation methods, attaining the highest rank in terms of different evaluation criteria.

Online Aware Synapse Weighted Autoencoder for Recovering Random Missing Data in Wastewater Treatment Process

Missing values in wastewater treatment process (WWTP) data hinder the monitoring and prediction of operational status. Therefore, various online imputation methods have been proposed to recover missing values from streaming data collected in WWTP in real time. However, existing methods tend to ignore previous learned knowledge. In this paper, an online aware synapse weighted autoencoder imputation method (OASI) is proposed to impute random missing values. First, an online stacked autoencoder (OSAE) framework is constructed to capture the nonlinear structure of the recently collected data. The framework decreases the computational and storage consumption of the model training. Second, an aware synapses weighted parameter regularization strategy is presented to guide the update of model parameters and alleviate the forgetting of historical information in an online continual setup. In this way, the learned features offer a more comprehensive representation of the overall information and help enhance imputation performance. Third, two real WWTP datasets with strong non-stationarity, high-noise level and high-dimensionality are used to validate the performance of the proposed OASI. Experimental results show that the proposed OASI achieves superior performances over the existing methods even in the presence of random missing values with different missing ratios, and only costs a short running time.

Optimizing Imputation for Educational Data: Exploring Training Partition and Missing Data Ratios

The integration of machine learning in educational data analysis presents challenges regarding the availability of sufficient training data, especially in the context of high missing data ratios. These challenges arise from data partitioning practices, resulting in smaller datasets and less precise models. Behavioral scientists have increasingly incorporated machine learning into propensity score estimation, necessitating investigations into the most effective training and testing partitioning methods for machine learning-based imputation. To address this gap in the literature, our Monte Carlo experiment examines the impact of partitioning methods and missing data ratios. Simulated datasets, featuring missing ratios of 10%, 30%, 50%, and 70%, are divided into training and testing sets, ranging from 80–20 to 20–80. Results indicate that each imputation method delivers highly accurate average treatment effects. However, in the context of maintaining covariate balance across diverse conditions, complex ensemble methods outperform artificial neural networks. A real-data comparison (Study II) further underscores that the adoption of sophisticated machine learning techniques significantly enhances covariate balance. This research contributes valuable insights into the development of machine learning-based imputation methods, with a specific focus on scenarios characterized by high missing data ratios, in educational data analysis.

Integrating k-means clustering algorithm for the symbiotic relationship of aesthetic community spatial science

Abstract In modern society, the scientific symbiotic relationship of aesthetic community space is crucial for the development of cities. This article aimed to explore the symbiotic relationship between aesthetic community space and science, and analyze this relationship based on the k-means clustering algorithm. The article analyzed the spatial characteristics of different types of aesthetic communities, as well as the commonalities and differences between aesthetic communities and science, including the complementary relationship between the two. The k-means clustering algorithm is integrated to classify community spatial features, thereby analyzing and optimizing the relationship between the two. The experiment selected a sample of an aesthetic community and divided it into three different types of spatial features: architectural area, landscape greening area, and other activity areas. This article also compared the clustering results of the k-means algorithm with the k-means algorithm based on three-dimensional grid space. The results showed that the k-means algorithm optimized by the principle of three-dimensional grid space had stronger clustering performance. In the experiment, four different types of clustering algorithms and multiple evaluation indicators were also selected for testing. The data showed that in the clustering of spatial features in building areas, the k-means average clustering error score was controlled at around 13–14 under different missing ratios; the minimum average clustering error score of the characteristic data of the landscape greening area was 12.1. The average clustering error score in spatial feature clustering of other activity areas reached a minimum value of 23.5 when the missing ratio was 10%, which was lower than other algorithms. The overall experimental results indicate that the fused k-means clustering algorithm can effectively partition the aesthetic community space and optimize scientific symbiotic relationships.

PARAFACM: A second-order calibration algorithm for handling data with missing values

In the process of collecting chemical data, missing values often occur due to some reasons. The missing values will destroy the multi-linear structure of the data, thus making the traditional multi-way calibration algorithm unable. Therefore, this work proposed a novel second-order calibration algorithm, PARAFAC for missing values (PARAFACM), which can directly handle the three-way data array with missing values. This proposed algorithm combines the idea of regarding multivariate linear regression as a set of individual univariate linear regressions with the iterative least squares principle. PARAFACM is open source and available from Supplementary materials. The proposed algorithm, as well as the previous three algorithms, including weighted PARAFAC (WPARAFAC), Incomplete Data PARAFAC (INDAFAC), and PARAFAC with single imputation (PARAFACSI), were used to handle the three-way simulation data array with three missing situations (random missing points, random missing channels, and scattering removal) at different missing ratios and noise levels. The comparison results show that the proposed algorithm is more universal when dealing with three missing situations and has a faster iteration speed. Moreover, its overall performance is better than the other algorithms when dealing with different missing ratios and noise levels. In addition, the proposed algorithm was also used to decompose the semi-simulated HPLC-DAD data array with missing time channels and the real EEM data array with scattering removal, and obtained satisfactory quantitative results. In general, the proposed algorithm can efficiently and accurately handle a variety of three-way data arrays with missing values, and extract the qualitative and quantitative information of components of interest in the presence of unknown interferences.

Deep Compressed Sensing based Data Imputation for Urban Environmental Monitoring

Data imputation is prevalent in crowdsensing, especially for Internet of Things (IoT) devices. On the one hand, data collected from sensors will inevitably be affected or damaged by unpredictability. On the other hand, extending the active time of sensor networks has urgently aspired environmental monitoring. Using neural networks to design a data imputation algorithm can take advantage of the prior information stored in the models. This paper proposes a preprocessing algorithm to extract a subset for training a neural network on an IoT dataset, including time window determination, sensor aggregation, sensor exclusion and data frame shape selection. Moreover, we propose a data imputation algorithm using deep compressed sensing with generative models. It explores novel representation matrices and can impute data in the case of a high missing ratio situation. Finally, we test our subset extraction algorithm and data imputation algorithm on the EPFL SensorScope dataset, respectively, and they effectively improve the accuracy and robustness even with extreme data loss.

Tensor-Based Temporal Control for Partially Observed High-Dimensional Streaming Data

In advanced manufacturing processes, high-dimensional (HD) streaming data (e.g., sequential images or videos) are commonly used to provide online measurements of product quality. Although there exist numerous research studies for monitoring and anomaly detection using HD streaming data, little research is conducted on feedback control based on HD streaming data to improve product quality, especially in the presence of incomplete responses. To address this challenge, this article proposes a novel tensor-based automatic control method for partially observed HD streaming data, which consists of two stages: offline modeling and online control. In the offline modeling stage, we propose a one-step approach integrating parameter estimation of the system model with missing value imputation for the response data. This approach (i) improves the accuracy of parameter estimation, and (ii) maintains a stable and superior imputation performance in a wider range of the rank or missing ratio for the data to be completed, compared to the existing data completion methods. In the online control stage, for each incoming sample, missing observations are imputed by balancing its low-rank information and the one-step-ahead prediction result based on the control action from the last time step. Then, the optimal control action is computed by minimizing a quadratic loss function on the sum of squared deviations from the target. Furthermore, we conduct two sets of simulations and one case study on semiconductor manufacturing to validate the superiority of the proposed framework.

Traffic‐aware rate control for mix‐flow in datacenter

AbstractDatacenter applications generate diverse flows, including deadline flows and non‐deadline flows. The deadline flows require to complete within strict deadline, while non‐deadline flows seek a shorter flow completion time. The state‐of‐the‐art deadline‐aware methods either transmit deadline flows with best‐effort at high priority, resulting in the starvation of non‐deadline flows, or blindly restrict the sending rates of deadline flows, leading to a high deadline missing ratio. To meet the different requirements of mix‐flows, a novel traffic‐aware rate control (TRC) method is proposed. TRC dynamically adjusts the sending rates of deadline flows according to their deadlines and the predicted future traffic patterns. If the intense competition is predicted among deadline flows, TRC will adopt a more aggressive manner to transmit the current deadline flows to avoid bandwidth contention in the future, reducing the deadline missing ratio. Otherwise, TRC will conservatively transmit deadline flows and complete these flows near their respective deadlines, relinquishing the excess bandwidth to non‐deadline flows. Meanwhile, TRC schedules non‐deadline flows in accordance with their sizes, minimizing the average FCT. The performance of TRC in large‐scale scenarios is evaluated through NS2 simulations. The test results show that TRC reduces the deadline missing ratio of deadline flows and the FCT of non‐deadline flows by up to 69.5% and 78.7% compared to the state‐of‐the‐art deadline‐aware schemes, respectively.

Super resolution direction finding technique of vortex electromagnetic wave radar in missing mode

Vortex electromagnetic waves have superior performance over electromagnetic waves. In order to improve the radar super-resolution lateralization technique in its missing modes, this study proposes to start from the perspective of mode missing. The missing modes are reconstructed using the Adaptive Step Size Gradient Descent (ASSGD) method by exploiting the features of the missing modes. The linear minimum mean square error (LMMSE) estimation method is also used to solve the problem of poor reconstruction accuracy due to the Missing Modes. The Missing Modes Iterative Adaptive Approach (MMIAA) algorithm and Missing Modes Sparse Learning via Iterative Minimization (MMSLIM) algorithm are then used. Minimization (MMSLIM) algorithm to recover missing modes. The results showed that the RMSEs of the recovery errors of MMSLIM, MMIAA and ASSGD were 0.16, 0.31 and 0.82 respectively at a modal missing ratio of 0.7, while ASSGD fails to recover the missing modal data at a modal missing ratio of 0.9. The overall data quality of the azimuthally estimated RMSE was average when the signal-to-noise ratio was at [–5, 10] dB. And the curve becomes flatter when it reaches 15 dB or more, indicating that MMSLIM, MMIAA has important theoretical and practical value.

Image reconstruction using superpixel clustering and tensor completion

This paper presents a pixel selection method for compact image representation based on superpixel segmentation and tensor completion. Our method divides the image into several regions that capture important textures or semantics and selects a representative pixel from each region to store. We experiment with different criteria for choosing the representative pixel and find that the centroid pixel performs the best. We also propose two smooth tensor completion algorithms that can effectively reconstruct different types of images from the selected pixels. Our experiments show that our superpixel-based method achieves better results than uniform sampling for various missing ratios.

Study of Landfalling Typhoon Potential Maximum Gale Forecasting in South China

Based on historical tropical cyclone (TC) tracking data and wind data from observation stations, four comparison experiments were designed that considered TC translation speed similarity and five new ensemble schemes in an improved Dynamical-Statistical-Analog Ensemble Forecast (DSAEF) model for Landfalling Typhoon Gale (LTG), which was tested in terms of forecast capability in South China. The results showed that the improved DSAEF_LTG model with the incorporation of TC translation speed and a new ensemble scheme could improve the forecast threat score (TS) and reduce both the false alarm ratio and the missing ratio in comparison with corresponding values attained before the improvement. The TS of the new ensemble scheme model (DLTG_3) was 0.34 at threshold above Beaufort Scale 7, which was 31% better than that of the unimproved model (DLTG_1). At a threshold above Beaufort Scale 10, the TS of DLTG_3 indicated even greater improvement, reaching 0.25, i.e., 127% higher than that of DLTG_1. The results of the experiments illustrated the marked improvement achievable when using the new ensemble scheme. The reasons for the differences in the DSAEF_LTG model forecasts before and after the introduction of TC translation speed and the new ensemble scheme were analyzed for the cases of Typhoon Haima and Typhoon Hato.

A Rolling Real-Time Correction Method for Minute Precipitation Forecast Based on Weather Radars

The quantitative precipitation estimation by weather radar plays an important role in observations and forecasts of meteorological processes. The National Minute Quantitative Precipitation Forecast system of China (MQPF), providing location-based refined short-term and imminent precipitation forecasting services, filled the gap in the official minute precipitation service products in China’s meteorological field. However, due to the technical limitations of radar itself and the complexity of the atmosphere, the corresponding relationship between radar echoes and surface precipitation is unstable. Based on radar and precipitation data from meteorological stations, a rolling real-time correction method is proposed to improve precipitation prediction accuracy through rolling correction of spatial and temporal structural errors in MQPF products. The results show the following: (1) Although this method may lead to a certain increase in the missing ratio, the significant improvement in the false alarm ratio after rolling correction has a positive guiding effect on short-term public meteorological services. (2) Regarding the time to complete rolling correction, the longest and shortest times appear in April and December, respectively. The mean running time to achieve correction of spatial and temporal error corrections ranges from 3.8 s to 6.4 s and 7.7 s to 11.5 s, respectively, which fully meets the real-time operational requirements of radar business.

Internal pump leakage detection of the hydraulic systems with highly incomplete flow data

Poor working condition makes internal pump leakage become one of the most frequent faults of hydraulic systems in industrial and agricultural production. The faults detection of the internal pump leakage not only suffers from the feature engineering but also the data acquisition and transmission. The latter may lead to the highly (over 50%) incomplete data problem which is deadly for fault diagnosis. Hence, the highly incomplete data problem is defined in this study. Then a two-stage fault diagnosis method based on the flow data is proposed. In the first stage, a Denoising Auto-Encoder with the conditional mask is used to complement the incomplete flow data. The conditional mask helps the model get extra information at a high missing ratio. In the second stage, with the help of masked noise and the Mask Attention mechanism, a classifier orients to the completed data is trained. These modifications help the classifier pay more attention to the remaining parts of flow data. The proposed method achieves the accuracy of 97% and 96% on the flow data which is missing by 60% and 70%. All the improvement measures are justified by the ablation and comparison experiments.

Physics-Informed Deep Learning for Reconstruction of Spatial Missing Climate Information in the Antarctic

Understanding the influence of the Antarctic on the global climate is crucial for the prediction of global warming. However, due to very few observation sites, it is difficult to reconstruct the rational spatial pattern by filling in the missing values from the limited site observations. To tackle this challenge, regional spatial gap-filling methods, such as Kriging and inverse distance weighted (IDW), are regularly used in geoscience. Nevertheless, the reconstructing credibility of these methods is undesirable when the spatial structure has massive missing pieces. Inspired by image inpainting, we propose a novel deep learning method that demonstrates a good effect by embedding the physics-aware initialization of deep learning methods for rapid learning and capturing the spatial dependence for the high-fidelity imputation of missing areas. We create the benchmark dataset that artificially masks the Antarctic region with ratios of 30%, 50% and 70%. The reconstructing monthly mean surface temperature using the deep learning image inpainting method RFR (Recurrent Feature Reasoning) exhibits an average of 63% and 71% improvement of accuracy over Kriging and IDW under different missing rates. With regard to wind speed, there are still 36% and 50% improvements. In particular, the achieved improvement is even better for the larger missing ratio, such as under the 70% missing rate, where the accuracy of RFR is 68% and 74% higher than Kriging and IDW for temperature and also 38% and 46% higher for wind speed. In addition, the PI-RFR (Physics-Informed Recurrent Feature Reasoning) method we proposed is initialized using the spatial pattern data simulated by the numerical climate model instead of the unified average. Compared with RFR, PI-RFR has an average accuracy improvement of 10% for temperature and 9% for wind speed. When applied to reconstruct the spatial pattern based on the Antarctic site observations, where the missing rate is over 90%, the proposed method exhibits more spatial characteristics than Kriging and IDW.