Problem Of Missing Values Research Articles

Predicting the Optimal Treatment for Diseases Using Whale Optimization Algorithm

Recently, there has been a significant research focus on addressing disease prevalence, especially in dealing with the complexities of big data associated with disease data. The challenge is achieving high accuracy due to missing value problems in big data. This study aims to use AI techniques to develop a system that predicates the optimal solutions for disease, regardless of the type of disease, i.e. the system can be applied to any type of disease. The approach involves handling missing values and normalizing disease datasets. The Whale Optimization Algorithm (WOA) will be used to improve predictions for effective disease treatments. We obtained good results for predicting the appropriate treatment for the disease in the proposed research, compared to the results obtained when applying the PSO algorithm before development in state of earlier, where the results obtained in the proposed research had higher accuracy than the results in in state of earlier at high iterations starting from 200 iterations and also had a lower error rate.:

Q-triplet characterization of atmospheric time series at Antofagasta: A missing values problem

Abstract Located in northern Chile (23.7°S, 70.4°W), Antofagasta has an exceptionally arid and stable climate characterized by minimal precipitation and consistent weather patterns. Nevertheless, despite these climate conditions being meaningful for several research and practical applications, our understanding of weather dynamics remains limited. The available meteorological data from 1969 to 2016 is analogical, which presents a significant challenge to analyze because these records are riddled with missing values, some measurements were taken at irregular measuring intervals, making it an interesting puzzle to grasp the Antofagasta’s climate scenario. To overcome this issue, we present a comprehensive statistical analysis of atmospheric temperature, pressure, and humidity time series. Our analytical approach involves the q-triplet calculation method, serving as a powerful tool to identify distinctive behavior within systems under non-equilibrium states. Our results suggest that, in general, the q-triplet values satisfy the condition q sens < 1 < q stat < q rel, a pattern that has been observed in previous studies.

A novel data-characteristic-driven modeling approach for imputing missing value in industrial statistics: A case study of China electricity statistics

As a direct reference tool to reflect the operational status and development level of national industry, industrial statistics hold significant value for numerous systematic studies. Nevertheless, it is crucial to recognize that the quality of these statistics can be compromised by the common occurrence of missing value. This issue poses a substantial challenge for analyzing and utilizing industrial statistics, impeding progress in tasks reliant upon them. Given the severity of the missing value problem in industrial statistical databases and the limitations of existing literatures on missing value imputation in terms of research objects and modeling approaches, this paper proposes a novel missing value imputation modeling approach for single-indicator panels of industrial statistics based on the idea of data-characteristic-driven (DCD). Accordingly, taking the inter-provincial “monthly power generation” data from China as an example, the imputation model was constructed and its validity was tested under different imputed objects (Jiangsu and Jilin), different missing types (continuous and discrete), and different missing rates (5%, 10% and 20%) respectively. The results indicate that the proposed DCD modeling approach in this paper exhibits excellent efficacy. The imputation model, constructed based on the data characteristic of the imputed object, demonstrates clear advantages in handling missing value with different missing types and rates. This is evident in its superior consideration of numerical accuracy, directional accuracy, and imputation stability, resulting in an outstanding comprehensive imputation effect.

Middle School Students’ Proportional Reasoning Ability in Solving Proportional and Non-proportional Problem

The objective of this descriptive qualitative study is to describe students' proportional reasoning ability in solving each type of proportional problem, namely missing-value problem, numerical comparison problem, and qualitative comparison problem. A non-proportional problem was also included in this study to assess students' ability in one of the indicators used to measure proportional reasoning ability. The instrument used in this study consists of mathematical word problems on the proportion concept. The participants involved nine eighth-grade students, who were chosen from a total of 49 eighth-grade students at a school in Bandung City. These students were selected using a technique known as purposive sampling, by looking at the answers of students who best represent the answers of other students. The gathered data were analyzed using content analysis and narrative analysis techniques. According to the results, students appeared to have difficulty solving proportional problems of all types using proportional reasoning. Students are still struggling with distinguishing between proportional and non-proportional situations, as well as direct and inverse proportions. Furthermore, students often encounter difficulties when attempting to solve numerical comparison and qualitative comparison problems. This might be a consequence of students' lack of experience in solving these types of problems. Another tendency is the use of the cross-multiplication algorithm in solving missing-value problems without knowing the purpose of using the algorithm.

Solving the missing value problem in PCA by Orthogonalized-Alternating Least Squares (O-ALS)

Dealing with missing data poses a challenge in Principal Component Analysis (PCA) since the most common algorithms are not designed to handle them. Several approaches have been proposed to solve the missing value problem in PCA, such as Imputation based on SVD (I-SVD), where missing entries are filled by imputation and updated in every iteration until convergence of the PCA model, and the adaptation of the Nonlinear Iterative Partial Least Squares (NIPALS) algorithm, able to work skipping the missing entries during the least-squares estimation of scores and loadings. However, some limitations have been reported for both approaches. On the one hand, convergence of the I-SVD algorithm can be very slow for data sets with a high percentage of missing data. On the other hand, the orthogonality properties among scores and loadings might be lost when using NIPALS.To solve these issues and perform PCA of data sets with missing values without the need of imputation steps, a novel algorithm called Orthogonalized-Alternating Least Squares (O-ALS) is proposed. The O-ALS algorithm is an alternating least-squares algorithm that estimates the scores and loadings subject to the Gram-Schmidt orthogonalization constraint. The way to estimate scores and loadings is adapted to work only with the available information.In this study, the performance of O-ALS is tested and compared with NIPALS and I-SVD in simulated data sets and in a real case study. The results show that O-ALS is an accurate and fast algorithm to analyze data with any percentage and distribution pattern of missing entries, being able to provide correct scores and loadings in cases where I-SVD and NIPALS do not perform satisfactorily.

Incomplete Contrastive Multi-View Clustering with High-Confidence Guiding

Incomplete multi-view clustering becomes an important research problem, since multi-view data with missing values are ubiquitous in real-world applications. Although great efforts have been made for incomplete multi-view clustering, there are still some challenges: 1) most existing methods didn't make full use of multi-view information to deal with missing values; 2) most methods just employ the consistent information within multi-view data but ignore the complementary information; 3) For the existing incomplete multi-view clustering methods, incomplete multi-view representation learning and clustering are treated as independent processes, which leads to performance gap. In this work, we proposed a novel Incomplete Contrastive Multi-View Clustering method with high-confidence guiding (ICMVC). Firstly, we proposed a multi-view consistency relation transfer plus graph convolutional network to tackle missing values problem. Secondly, instance-level attention fusion and high-confidence guiding are proposed to exploit the complementary information while instance-level contrastive learning for latent representation is designed to employ the consistent information. Thirdly, an end-to-end framework is proposed to integrate multi-view missing values handling, multi-view representation learning and clustering assignment for joint optimization. Experiments compared with state-of-the-art approaches demonstrated the effectiveness and superiority of our method. Our code is publicly available at https://github.com/liunian-Jay/ICMVC. The version with supplementary material can be found at http://arxiv.org/abs/2312.08697.

A Car-Following Model Considering Missing Data Based on TransGAN Networks

Car-following behavior is closely related to the longitudinal control of the vehicle, affecting the safety of the vehicle and traffic flow stability. In order to interact with the preceding vehicle, the target vehicle usually collects the driving data of the preceding vehicle. However, data acquisition devices often face malfunctions caused by various unpredictable disruptions, resulting in missing value problems. This may cause the target vehicle to make wrong control decisions. Given this situation, a new car-following(CF) model considering missing data based on Transformer-Generative Adversarial Networks (TransGAN) is proposed. Firstly, Transformer Network with multi-head attention is used to deeply extract the potential features from incomplete vehicle state data, which can filter important information from the input and focus on these, while capturing long distance dependencies. Secondly, a Generative Adversarial Network is constructed. The Generator generates the future multi-step control states of the target vehicle based on the features extracted by Transformer Network. The Discriminator with a fully connected network is applied to simultaneously ensure the generation accuracy. Finally, our proposed model was trained and tested on a publicly available NGSIM I-80 dataset. Compared with other existing advanced works, our model can fit the actual control states of the target vehicle with higher accuracy under different data missing rates of the preceding vehicle, which demonstrates that the proposed method effectively improves the robustness of vehicle longitudinal car-following control under missing data.

Deep Imputation Bi-stochastic Graph Regularized Matrix Factorization for Clustering Single-cell RNA-sequencing Data.

By generating massive gene transcriptome data and analyzing transcriptomic variations at the cell level, single-cell RNA-sequencing (scRNA-seq) technology has provided new way to explore cellular heterogeneity and functionality. Clustering scRNA-seq data could discover the hidden diversity and complexity of cell populations, which can aid to the identification of the disease mechanisms and biomarkers. In this paper, a novel method (DSINMF) is presented for single cell RNA sequencing data by using deep matrix factorization. Our proposed method comprises four steps: first, the feature selection is utilized to remove irrelevant features. Then, the dropout imputation is used to handle missing value problem. Further, the dimension reduction is employed to preserve data characteristics and reduce noise effects. Finally, the deep matrix factorization with bi-stochastic graph regularization is used to obtain cluster results from scRNA-seq data. We compare DSINMF with other state-of-the-art algorithms on nine datasets and the results show our method outperformances than other methods.

Student Proportional Reasoning on Missing Value Problems Based on Multiplicative Concepts

Proportional reasoning involves an understanding of the relationship of the overall multiplicative partitions between ratios and proportions. However, the problem that is often faced by students in proportional reasoning is that students tend to use additive strategies which can cause students to get incorrect answers. This study aims to analyze and describe students' proportional reasoning in solving the missing value problem based on the multiplicative concept. The research design used a qualitative approach with exploratory descriptive type. The research subjects were 3 students of grade VII. Research subjects were selected based on their oral and written communication skills and the considerations of mathematics teachers. Collecting data using the missing value problem solving test and interview guidelines. Data analysis based on the results of student work and the results of interviews directed to determine the research focus in the form of characteristics and indicators of proportional reasoning. The results of the study found three categories of students' proportional reasoning in solving the missing value problem, namely the "additive" category, students observing objects using the sum concept. The multiplicative category of the "within ratio" type, students observe objects using the multiplicative concept between elements in the same ratio. Multiplicative category of type "between ratios", students observe objects using the concept of multiplication between elements in different ratios.

Comparison of imputation methods for missing production data of dairy cattle

Nowadays, vast amounts of data representing feed intake, growth, and environmental impact of individual animals are being recorded in on-farm settings. Despite their apparent use, data collected in real-world applications often have missing values in one or several variables, due to reasons including human error, machine error, or sampling frequency misalignment across multiple variables. Since incomplete datasets are less valuable for downstream data analysis, it is important to address the missing value problem properly. One option may be to reduce the dataset to a subset that contains only complete data, but considerable data may be lost via this process. The current study aimed to compare imputation methods for the estimation of missing values in a raw dataset of dairy cattle including 454 553 records collected from 629 cows between 2009 and 2020. The dataset was subjected to a cleaning process that reduced its size to 437 075 observations corresponding to 512 cows. Missing values were present in four variables: concentrate DM intake (CDMI, missing percentage = 2.30%), forage DM intake (FDMI, 8.05%), milk yield (MY, 15.12%), and BW (64.33%). After removing all missing values, the resulting dataset (n = 129 353) was randomly sampled five times to create five independent subsets that exhibit the same missing data percentages as the cleaned dataset. Four univariate and nine multivariate imputation methods (eight machine learning methods and the MissForest method) were applied and evaluated on the five repeats, and average imputation performance was reported for each repeat. The results showed that Random Forest was overall the best imputation method for this type of data and had a lower mean squared prediction error and higher concordance correlation coefficient than the other imputation methods for all imputed variables. Random Forest performed particularly well for imputing CDMI, MY, and BW, compared to imputing FDMI.

Manufacturing industry based on dynamic soft sensors in integrated with feature representation and classification using fuzzy logic and deep learning architecture.

Soft sensors are data-driven devices that allow for estimates of quantities that are either impossible to measure or prohibitively expensive to do so. DL (deep learning) is a relatively new feature representation method for data with complex structures that has a lot of promise for soft sensing of industrial processes. One of the most important aspects of building accurate soft sensors is feature representation. This research proposed novel technique in automation of manufacturing industry where dynamic soft sensors are used in feature representation and classification of the data. Here the input will be data collected from virtual sensors and their automation-based historical data. This data has been pre-processed to recognize the missing value and usual problems like hardware failures, communication errors, incorrect readings, and process working conditions. After this process, feature representation has been done using fuzzy logic-based stacked data-driven auto-encoder (FL_SDDAE). Using the fuzzy rules, the features of input data have been identified with general automation problems. Then, for this represented features, classification process has been carried out using least square error backpropagation neural network (LSEBPNN) in which the mean square error while classification will be minimized with loss function of the data. The experimental results have been carried out for various datasets in automation of manufacturing industry in terms of computational time of 34%, QoS of 64%, RMSE of 41%, MAE of 35%, prediction performance of 94%, and measurement accuracy of 85% by proposed technique.

Missing Road Condition Imputation Using a Multi-View Heterogeneous Graph Network From GPS Trajectory

How to generate road conditions from urban GPS trajectory is an important problem in transportation systems. However, this computation process usually suffers from serious missing value problem due to the observation uncertainty or limited reports from crowdsourcing systems. Conventional tensor factorization approaches learn the spatio-temporal dependencies in a collaborative filtering way, which ignores the complex road network structure information and temporal heterogeneity. In this study, we propose a multi-view model with multiple aspects of prior knowledge to impute traffic state computed from a real-world trajectory dataset. More specifically, in the spatial view, rather than focusing on a specific type of road segment, we take the heterogeneity of road network into consideration and model the multiple relations of adjacent road segments. Meanwhile, the temporal pattern is also viewed as a heterogeneous graphical structure that discriminates the weekly/hourly adjacency in the temporal view. Finally, we fuse the above spatio-temporal features to provide a robust estimation under different sparse conditions. Intensive experiments on two types of missing scenarios (i.e., random and non-random) demonstrate that the proposed imputation method outperforms all the other state-of-the-art approaches. In addition, our model represents interpretable patterns for spatio-temporal graph analysis.

Tensor completion via convolutional sparse coding with small samples-based training

Tensor data often suffer from missing value problems due to the complex high-dimensional structure while acquiring them. To complete the missing information, lots of Low-Rank Tensor Completion (LRTC) methods have been proposed, most of which depend on the low-rank property of tensor data. In this way, the low-rank component of the original data could be recovered roughly. However, the shortcoming is that the detailed information can not be fully restored, no matter the Sum of the Nuclear Norm (SNN) nor the Tensor Nuclear Norm (TNN) based methods. On the contrary, in the field of signal processing, Convolutional Sparse Coding (CSC) can provide a good representation of the high-frequency component of the image, which is generally associated with the detail component of the data. To this end, we propose two novel methods, LRTC-CSC-I and LRTC-CSC-II, which adopt CSC as a supplementary regularization for LRTC to capture the high-frequency components. Therefore, the LRTC-CSC methods can not only solve the missing value problem but also recover the details. Moreover, the regularizer CSC can be trained with small samples due to the sparsity characteristic. Extensive experiments show the effectiveness of LRTC-CSC methods, and quantitative evaluation indicates that the performance of our models are superior to state-of-the-art methods.

Analisis Kinerja Algoritma K-Nearest Neighbor Imputation (KNNI) Untuk Missing Value Pada Klasifikasi Data Mining

At this time many researchers use datasets for research, in the dataset there is a lot of important information provided to make it easier for researchers to process the data, but there are obstacles when trying to process the data, namely some data values are lost, or even damaged (incompatible) with other values. The number of attributes and data samples in the dataset is unlimited which makes it difficult for researchers to find important information based on the goals of each researcher. The missing or damaged values are known as missing values and this often happens when researchers process data taken from datasets. When researchers classify a dataset, of course it is difficult to classify because if there is missing data, the classification process cannot be carried out completely. Of course this will greatly affect the results obtained and also the results of the accuracy of the process. The problem of missing data (missing value) must be resolved, one way to solve this problem is by using the K-Nearest Neighbor Imputation (K-NNI) algorithm. The KNNI algorithm is one of the data mining algorithms that can restore the missing value problem by performing pattern recognition of the nearest data from the missing data. The final result after applying KNNI in fixing the missing value is to obtain a weight value for each K closest observation of 51.6161 on K5. Datasets that experience missing values have been adjusted based on the weight values and the implementation of the KNNI algorithm with the rapidminer application. By returning data using the KNNI algorithm, it is hoped that it can help solve these problems so that the results obtained from the classification process are more accurate and the results of greater accuracy

MISNN: Multiple Imputation via Semi-parametric Neural Networks.

Multiple imputation (MI) has been widely applied to missing value problems in biomedical, social and econometric research, in order to avoid improper inference in the downstream data analysis. In the presence of high-dimensional data, imputation models that include feature selection, especially regularized regression (such as Lasso, adaptive Lasso, and Elastic Net), are common choices to prevent the model from underdetermination. However, conducting MI with feature selection is difficult: existing methods are often computationally inefficient and poor in performance. We propose MISNN, a novel and efficient algorithm that incorporates feature selection for MI. Leveraging the approximation power of neural networks, MISNN is a general and flexible framework, compatible with any feature selection method, any neural network architecture, high/low-dimensional data and general missing patterns. Through empirical experiments, MISNN has demonstrated great advantages over state-of-the-art imputation methods (e.g. Bayesian Lasso and matrix completion), in terms of imputation accuracy, statistical consistency and computation speed.

Urban traffic flow prediction: a dynamic temporal graph network considering missing values

Accurate traffic flow prediction on the urban road network is an indispensable function of Intelligent Transportation Systems (ITS), which is of great significance for urban traffic planning. However, the current traffic flow prediction methods still face many challenges, such as missing values and dynamic spatial relationships in traffic flow. In this study, a dynamic temporal graph neural network considering missing values (D-TGNM) is proposed for traffic flow prediction. First, inspired by the Bidirectional Encoder Representations from Transformers (BERT), we extend the classic BERT model, called Traffic BERT, to learn the dynamic spatial associations on the road structure. Second, we propose a temporal graph neural network considering missing values (TGNM) to mine traffic flow patterns in missing data scenarios for traffic flow prediction. Finally, the proposed D-TGNM model can be obtained by integrating the dynamic spatial associations learned by Traffic BERT into the TGNM model. To train the D-TGNM model, we design a novel loss function, which considers the missing values problem and prediction problem in traffic flow, to optimize the proposed model. The proposed model was validated on an actual traffic dataset collected in Wuhan, China. Experimental results showed that D-TGNM achieved good prediction results under four missing data scenarios (15% random missing, 15% block missing, 30% random missing, and 30% block missing), and outperformed ten existing state-of-the-art baselines.

Machine Learning Prediction Models for Postoperative Stroke in Elderly Patients: Analyses of the MIMIC Database.

ObjectiveWith the aging of populations and the high prevalence of stroke, postoperative stroke has become a growing concern. This study aimed to establish a prediction model and assess the risk factors for stroke in elderly patients during the postoperative period.MethodsML (Machine learning) prediction models were applied to elderly patients from the MIMIC (Medical Information Mart for Intensive Care)-III and MIMIC-VI databases. The SMOTENC (synthetic minority oversampling technique for nominal and continuous data) balancing technique and iterative SVD (Singular Value Decomposition) data imputation method were used to address the problem of category imbalance and missing values, respectively. We analyzed the possible predictive factors of stroke in elderly patients using seven modeling approaches to train the model. The diagnostic value of the model derived from machine learning was evaluated by the ROC curve (receiver operating characteristic curve).ResultsWe analyzed 7,128 and 661 patients from MIMIC-VI and MIMIC-III, respectively. The XGB (extreme gradient boosting) model got the highest AUC (area under the curve) of 0.78 (0.75–0.81), making it better than the other six models, Besides, we found that XGB model with databalancing was better than that without data balancing. Based on this prediction model, we found hypertension, cancer, congestive heart failure, chronic pulmonary disease and peripheral vascular disease were the top five predictors. Furthermore, we demonstrated that hypertension predicted postoperative stroke is much more valuable.ConclusionStroke in elderly patients during the postoperative period can be reliably predicted. We proved XGB model is a reliable predictive model, and the history of hypertension should be weighted more heavily than the results of laboratory tests to prevent postoperative stroke in elderly patients regardless of gender.

Jackknife winsorized variance estimator under imputed data

Abstract In the present study, we consider the problem of missing and extreme values for the estimation of population variance. The presence of extreme values either in the study variable, or the auxiliary variable, or in both of them, can adversely affect the performance of the estimation procedure. We consider three different situations for the presence of extreme values and also consider jackknife variance estimators for the population variance by handling these extreme values under stratified random sampling. Bootstrap technique ABB is carried out to understand the relative relationship more precisely.

Feature Optimization Method of Material Identification for Loose Particles Inside Sealed Relays.

Existing material identification for loose particles inside sealed relays focuses on the selection and optimization of classification algorithms, which ignores the features in the material dataset. In this paper, we propose a feature optimization method of material identification for loose particles inside sealed relays. First, for the missing value problem, multiple methods were used to process the material dataset. By comparing the identification accuracy achieved by a Random-Forest-based classifier (RF classifier) on the different processed datasets, the optimal direct-discarding method was obtained. Second, for the uneven data distribution problem, multiple methods were used to process the material dataset. By comparing the achieved identification accuracy, the optimal min–max standardization method was obtained. Then, for the feature selection problem, an innovative multi-index–fusion feature selection method was designed, and its superiority was verified through several tests. Test results show that the identification accuracy achieved by RF classifier on the dataset was improved from 59.63% to 63.60%. Test results of ten material verification datasets show that the identification accuracies achieved by RF classifier were greatly improved, with an average improvement of 3.01%. This strongly promotes research progress in loose particle material identification and is an important supplement to existing loose particle detection research. This is also the highest loose particle material identification accuracy achieved to in aerospace engineering, which has important practical value for improving the reliability of aerospace systems. Theoretically, it can be applied to feature optimization in machine learning.

Goodness-of-fit tests for a logistic regression model with missing covariates

Model checking for logistic regression with covariates missing at random is considered. Based on the ideas of Copas (1989) and Osius and Rojek (1992) and studies of Homser et al. (1997), proposed are the two-type goodness-of-fit tests, Pearson chi-squared and unweighted residual sum-of-squares tests, in which their test statistics are centralized by subtracting their estimated mean to be mean-zero-form test statistics via the inverse probability weighting (IPW) and nonparametric multiple imputation (MI) methods to solve the missing value problem. The asymptotic properties of these test statistics are established under the null hypothesis and some regularity conditions. The test statistics conducted by using the IPW and MI estimators are asymptotically equivalent. Proposed are the IPW method and two bootstrap re-sampling approaches for estimation of the variances of the proposed test statistics to solve the issue of underestimating their variances by the MI method of Rubin (1987). Simulation studies are carried out to assess the finite-sample power performances of these proposed tests. Two real data examples are used to illustrate the applicability of the proposed tests.