Analysis Of Incomplete Data Research Articles

A State-Migration Particle Swarm Optimizer for Adaptive Latent Factor Analysis of High-Dimensional and Incomplete Data

A State-Migration Particle Swarm Optimizer for Adaptive Latent Factor Analysis of High-Dimensional and Incomplete Data

Method for Incomplete and Imbalanced Data Based on Multivariate Imputation by Chained Equations and Ensemble Learning.

The classification analysis of incomplete and imbalanced data is still a challenging task since these issues could negatively impact the training of classifiers, which were also found in our study on the physical fitness assessments of patients. And in fields such as healthcare, there are higher requirements for the accuracy of the generated imputation values. To train a high-performance classifier and pursue high accuracy, we attempted to resolve any potential negative impact by using a novel algorithmic approach based on the combination of multivariate imputation by chained equations and the ensemble learning method (MICEEN), which can solve the two problems simultaneously. We used multivariate imputation by chained equations to generate more accurate imputation values for the training set passed to ensemble learning to build a predictor. On the other hand, missing values were introduced into minority classes and used them to generate new samples belonging to the minority classes in order to balance the distribution of classes. On real-world datasets, we perform extensive experiments to assess our method and compare it to other state-of-the-art approaches. The advantages of the proposed method are demonstrated by experimental results for the benchmark datasets and self-collected datasets of physical fitness assessment of tumor patients with varying missing rates.

Encoding prior knowledge in ensemble refinement.

The proper balancing of information from experiment and theory is a long-standing problem in the analysis of noisy and incomplete data. Viewed as a Pareto optimization problem, improved agreement with the experimental data comes at the expense of growing inconsistencies with the theoretical reference model. Here, we propose how to set the exchange rate a priori to properly balance this trade-off. We focus on gentle ensemble refinement, where the difference between the potential energy surfaces of the reference and refined models is small on a thermal scale. By relating the variance of this energy difference to the Kullback-Leibler divergence between the respective Boltzmann distributions, one can encode prior knowledge about energy uncertainties, i.e., force-field errors, in the exchange rate. The energy uncertainty is defined in the space of observables and depends on their type and number and on the thermodynamic state. We highlight the relation of gentle refinement to free energy perturbation theory. A balanced encoding of prior knowledge increases the quality and transparency of ensemble refinement. Our findings extend to non-Boltzmann distributions, where the uncertainty in energy becomes an uncertainty in information.

Parallel Adaptive Stochastic Gradient Descent Algorithms for Latent Factor Analysis of High-Dimensional and Incomplete Industrial Data

Latent factor analysis (LFA) is efficient in knowledge discovery from a high-dimensional and incomplete (HDI) matrix frequently encountered in industrial big data-related applications. A stochastic gradient descent (SGD) algorithm is commonly adopted as a learning algorithm for LFA owing to its high efficiency. However, its sequential nature makes it less scalable when processing large-scale data. Although alternating SGD decouples an LFA process to achieve parallelization, its performance relies on its hyper-parameters that are highly expensive to tune. To address this issue, this paper presents three extended alternating SGD algorithms whose hyper-parameters are made adaptive through particle swarm optimization. Correspondingly, three Parallel Adaptive LFA (PAL) models are proposed and achieve highly efficient latent factor acquisition from an HDI matrix. Experiments have been conducted on four HDI matrices collected from industrial applications, and the benchmark models are LFA models based on state-of-the-art parallel SGD algorithms including the alternative SGD, Hogwild!, distributed gradient descent, and sparse matrix factorization parallelization. The results demonstrate that compared with the benchmarks, with 32 threads, the proposed PAL models achieve much speedup gain. They achieve the highest prediction accuracy for missing data on most cases. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —HDI data are commonly encountered in many industrial big data-related applications, where rich knowledge and patterns can be extracted efficiently. An SGD based-LFA model is popular in addressing HDI data due to its efficiency. Yet when dealing with large-scale HDI data, its serial nature greatly reduces its scalability. Although alternating SGD can decouple an LFA process to implement parallelization, its performance depends on its hyper-parameter whose tuning is tedious. To address this vital issue, this study proposes three extended alternating SGD algorithms whose hyper-parameters are made via through a particle swarm optimizer. Based on them, three models are realized, which are able to efficiently obtain latent factors from HDI matrices. Compared with the existing and state-of-the-art models, they enjoy their hyper-parameter-adaptive learning process, as well as highly competitive computational efficiency and representation learning ability. Hence, they provide practitioners with more scalable solutions when addressing large HDI data from industrial applications.

Noisy Tensor Completion via Low-Rank Tensor Ring.

Tensor completion is a fundamental tool for incomplete data analysis, where the goal is to predict missing entries from partial observations. However, existing methods often make the explicit or implicit assumption that the observed entries are noise-free to provide a theoretical guarantee of exact recovery of missing entries, which is quite restrictive in practice. To remedy such drawback, this article proposes a novel noisy tensor completion model, which complements the incompetence of existing works in handling the degeneration of high-order and noisy observations. Specifically, the tensor ring nuclear norm (TRNN) and least-squares estimator are adopted to regularize the underlying tensor and the observed entries, respectively. In addition, a nonasymptotic upper bound of estimation error is provided to depict the statistical performance of the proposed estimator. Two efficient algorithms are developed to solve the optimization problem with convergence guarantee, one of which is specially tailored to handle large-scale tensors by replacing the minimization of TRNN of the original tensor equivalently with that of a much smaller one in a heterogeneous tensor decomposition framework. Experimental results on both synthetic and real-world data demonstrate the effectiveness and efficiency of the proposed model in recovering noisy incomplete tensor data compared with state-of-the-art tensor completion models.

On subset multiple correspondence analysis for incomplete multivariate categorical data

Multiple imputation is considered as a superior data analysis practice to deal with missing values. This is, in particular, also true in the important case of categorical variables. However, imputation techniques could be a daunting task for the less technical data practitioner. In the case of categorical variables, imputation can be avoided by recoding the indicator matrix of the data set before analysis. New category levels are created for each variable to which missing values are assigned. The recoded indicator matrix enables the separation of missing and observed responses. Subset multiple correspondence analysis (sMCA) allows a focused analysis on either of the missing or non-missing subsets of the recoded indicator matrix while preserving the original variation of the data. The main purpose of this article is to evaluate the sMCA approach using simulated data which incorporate various parameters. In addition, the visualization of multivariate categorical data containing missing values that follows from the sMCA approach is presented. This article could aid as a guide when deciding whether the sMCA approach is suitable for the analysis of incomplete multivariate categorical data. To ensure effortless application of the presented methodology, a complete R script with commented functions is available as supplementary material.

The Analysis of Incomplete Business Data

Missing values can dramatically reduce the accuracy and availability of missing data, especially when analyzing business data. A common method to deal with the missing data is simply deleting the samples containing missing attributes. However, this will lead to bias and invalid conclusions since some data are too important to be omitted easily. Therefore, we should use certain methods to complete the data set instead of deleting data with missing values. In this paper, we compared several data imputation methods by adopting them to deal with six benchmark business data sets. The result provides us with guidance when dealing with incomplete business data.

Nonparametric Analysis of Balanced Incomplete Block Rank Data

Traditional nonparametric analysis of balanced incomplete block rank data usually involves the Durbin test. Here we give an alternative adjustment for the Durbin statistic for when there are ties and mid-ranks are used. The adjusted Durbin statistic is shown to be simply related to the ANOVA F statistic on the ranks, so that the corresponding tests are, in a sense, equivalent. This means both are tests of equality of mean treatment ranks. A simulation study compares the size and power performances of the competing tests. We also apply the nonparametric ANOVA methodology to give tests for univariate moment effects and, when treatments are ordered, for bivariate moment effects. The latter includes umbrella tests.

Glaucoma knowledge of patients in Abuja, Nigeria

This year, 2012, marks the fifteenth consecutive year of publication of the Africa Journal of Nursing and Midwifery (AJNM) and its fifth year of publication as an academic journal accredited by South Africa's Department of Higher Education and Training (DoHET). This accreditation is a major accomplishment. Retaining its status as an academic accredited journal is the major focus of the journal's editors and editorial board, and poses challenges for exceeding the minimum expected standards of every DoHET audit. Meeting these expectations necessitates repeated revisions of most manuscripts and the rejection of some manuscripts. The critical factor for retaining its DoHET accreditation is the quality of every article published by the AJNM, rather than the number of articles or the number of issues published during any year. The major reasons for manuscripts being rejected include the manuscripts being based on outdated data, outdated references, incorrect statistics, incomplete qualitative data analysis or aspects irrelevant to the nurses and midwives of the 52 countries of the World Health Organization (WHO) Africa region. This is a non-negotiable requirement because the AJNM strives to provide a publication and communication platform for the nurses and midwives from these African countries.

Analysis of the lockdown effects due to the COVID-19 on air pollution in Brescia (Lombardy)

SARS-CoV-2 virus (COVID-19) pandemic has impacted several countries, with also some differences at local levels. When lockdown restrictions were imposed, the concentrations of some air pollutants were reduced, as reported in some other cities in the world. This was often considered a positive by-product of the pandemic. However, often literature reporting the connection of air quality (AQ) and lockdown, suffers of limited and incomplete data analysis, not considering, for example, some confounding factors.This work presents a methodology, and the results of its application, to assess the impact of pandemic restrictions on AQ (in particular nitrogen oxides, NO2 and particulate matter, PM10) in spring 2020 in Brescia, located in one of the most affected areas in terms of virus diffusion and in one of the most polluted areas in Europe (Po Valley, Italy).In particular, the proposed methodology integrates data and AQ modelling simulations to distinguish between the changes in the PM10 and NO2 pollutants concentration that occurred due to the restriction measures and due to other factors, like spatial-temporal characteristics (for example the seasonality), meteorological factors, and governmental actions that were introduced in the past to improve the air quality.Results show that NO2 is strongly dependent to traffic emission. On the contrary, although the expected decrease in PM10 concentrations, the results highlight that the reduction of transport emission would not help to avoid severe air pollution, due to the other pollution sources that contribute to its origin.The results presented for the first time in this work are of particular interest because they may be used as a basis to investigate in more details the sources that can impact on the air quality in Brescia, with the aim to propose effective measures able to reduce it.

Nearest neighbor imputation for categorical data by weighting of attributes

Missing values are a common phenomenon in modern medical research of complex diseases. The data often contains nominal or categorical variables, for example, single nucleotide polymorphisms (SNPs) in genetic studies. If the missing values are not handled properly, the downstream statistical analysis of incomplete data may be biased. While various imputation methods are available for metrically scaled variables, methods for categorical data are scarce. An imputation method that has been shown to work well for high dimensional metrically scaled variables is the imputation by nearest neighbor methods. In this paper, we propose a weighted nearest neighbors approach to impute missing values in categorical variables in high dimensional datasets. The proposed method explicitly uses the information on the association among attributes. Using different simulation settings, the performance is compared with available imputation methods. A variety of real data sets, containing heart, DNA, and lymphatic cancer, is also used to support the results obtained by simulations. The results show that the weighting of attributes yields smaller imputation errors than existing approaches like random forest and MICE.

Fuzzy Clustering Analysis of Power Incomplete Data based on Improved IVAEGAN Model

The scale of data generated by the complex and huge power system during operation is also very large. With the data acquisition of various information systems, it is easy to form the situation of incomplete power data information, which cannot guarantee the efficiency and quality of work, and reduce the security and reliability of the entire power grid. When incomplete data and incomplete data sets are caused by data storage failure or data acquisition errors, fuzzy clustering of data will face great difficulties. The fuzzy clustering of incomplete data of the power equipment is divided into the processing of incomplete data and the clustering analysis of "recovered" complete data. This paper proposes an IVAEGAN-IFCM interval fuzzy clustering algorithm, which uses interval data sets to fill in the incomplete data, and then completes the clustering of interval data. At the same time, the whole numerical data set is transformed into a complete interval data set. The final clustering result is obtained by interval fuzzy mean clustering analysis of the whole interval data set. Finally, the algorithm proposed in this paper and other machine learning training data sets is made for experimental analysis. The experimental results show that the algorithm proposed in this paper can complete incomplete data sets with high precision clustering. Compared with other contrast methods, it shows higher clustering accuracy. Compared with the numerical clustering algorithm, the clustering accuracy is improved by more than 4.3%, and it has better robustness. It also shows better generalization on the artificial data sets and other complex data sets. It is helpful to improve the technical level of the existing power grid and has important theoretical research value and engineering practice significance.

A global assessment of COVID-19 diffusion based on a single indicator: Some considerations about air pollution and COVID-19 spread.

The extraordinariness of COVID-19 occurred in a world that was completely unprepared to face it. To justify this, sometimes literature proposes positive associations between concentrations of some air pollutants and SARS-CoV-2 mortality and infectivity. However, several of these studies are affected by incomplete data analysis and/or incorrect accounts of spread dynamics that can be attributed to respiratory viruses.Based on separate analyses involving all the USA states and globally all the world countries suffering from the pandemic, this communication shows that commercial trade seems to be a good indicator of virus spread, being proposed as a surrogate of human-to-human interactions.The results of this study strongly support the conclusion that this new indicator could result fundamental to model (and avoid) possible future pandemics, strongly suggesting dedicated studies devoted to better investigate its significance.

98Framework for the Treatment And Reporting of Missing data in Observational Studies: The TARMOS framework

Abstract Focus of presentation Missing data are ubiquitous in medical research. Although there is increasing guidance on how to handle missing data, practice is changing slowly and misapprehensions abound, particularly in observational research. We present a practical framework for handling and reporting the analysis of incomplete data in observational studies, which we illustrate using a case study from the Avon Longitudinal Study of Parents and Children. Findings The framework consists of three steps: 1) Develop an analysis plan specifying the analysis model and how missing data are going to be addressed. Important considerations are whether a complete records analysis is likely to be valid, whether multiple imputation or an alternative approach is likely to offer benefits, and whether a sensitivity analysis regarding the missingness mechanism is required. 2) Explore the data, checking the methods outlined in the analysis plan are appropriate, and conduct the pre-planned analysis. 3) Report the results, including a description of the missing data, details on how missing data were addressed, and the results from all analyses, interpreted in light of the missing data and clinical relevance. Conclusions/Implications This framework encourages researchers to think carefully about their missing data and be transparent about the potential effect on the study results. This will increase confidence in the reliability and reproducibility of results from published papers. Key messages Researchers need to develop a plan for missing data prior to conducting their analysis, and be transparent about how they handled the missing data and its potential effect when reporting their results.

98Framework for the Treatment And Reporting of Missing data in Observational Studies: The TARMOS framework

Abstract Focus of presentation Missing data are ubiquitous in medical research. Although there is increasing guidance on how to handle missing data, practice is changing slowly and misapprehensions abound, particularly in observational research. We present a practical framework for handling and reporting the analysis of incomplete data in observational studies, which we illustrate using a case study from the Avon Longitudinal Study of Parents and Children. Findings The framework consists of three steps: 1) Develop an analysis plan specifying the analysis model and how missing data are going to be addressed. Important considerations are whether a complete records analysis is likely to be valid, whether multiple imputation or an alternative approach is likely to offer benefits, and whether a sensitivity analysis regarding the missingness mechanism is required. 2) Explore the data, checking the methods outlined in the analysis plan are appropriate, and conduct the pre-planned analysis. 3) Report the results, including a description of the missing data, details on how missing data were addressed, and the results from all analyses, interpreted in light of the missing data and clinical relevance. Conclusions/Implications This framework encourages researchers to think carefully about their missing data and be transparent about the potential effect on the study results. This will increase confidence in the reliability and reproducibility of results from published papers. Key messages Researchers need to develop a plan for missing data prior to conducting their analysis, and be transparent about how they handled the missing data and its potential effect when reporting their results.

Framework for the treatment and reporting of missing data in observational studies: The Treatment And Reporting of Missing data in Observational Studies framework

Missing data are ubiquitous in medical research. Although there is increasing guidance on how to handle missing data, practice is changing slowly and misapprehensions abound, particularly in observational research. Importantly, the lack of transparency around methodological decisions is threatening the validity and reproducibility of modern research. We present a practical framework for handling and reporting the analysis of incomplete data in observational studies, which we illustrate using a case study from the Avon Longitudinal Study of Parents and Children. The framework consists of three steps: 1) Develop an analysis plan specifying the analysis model and how missing data are going to be addressed. An important consideration is whether a complete records’ analysis is likely to be valid, whether multiple imputation or an alternative approach is likely to offer benefits and whether a sensitivity analysis regarding the missingness mechanism is required; 2) Examine the data, checking the methods outlined in the analysis plan are appropriate, and conduct the preplanned analysis; and 3) Report the results, including a description of the missing data, details on how the missing data were addressed, and the results from all analyses, interpreted in light of the missing data and the clinical relevance. This framework seeks to support researchers in thinking systematically about missing data and transparently reporting the potential effect on the study results, therefore increasing the confidence in and reproducibility of research findings.

Principal component analysis of incomplete data – A simple solution to an old problem

A long-standing problem in biological data analysis is the unintentional absence of values for some observations or variables, preventing the use of standard multivariate exploratory methods, such as principal component analysis (PCA). Solutions include deleting parts of the data by which information is lost, data imputation, which is always arbitrary, and restriction of the analysis to either the variables or observations, thereby losing the advantages of biplot diagrams. We describe a minor modification of eigenanalysis-based PCA in which correlations or covariances are calculated using different numbers of observations for each pair of variables, and the resulting eigenvalues and eigenvectors are used to calculate component scores such that missing values are skipped. This procedure avoids artificial data imputation, exhausts all information from the data and allows the preparation of biplots for the simultaneous display of the ordination of variables and observations. The use of the modified PCA, called InDaPCA (PCA of Incomplete Data) is demonstrated on actual biological examples: leaf functional traits of plants, functional traits of invertebrates, cranial morphometry of crocodiles and fish hybridization data – with biologically meaningful results. Our study suggests that it is not the percentage of missing entries in the data matrix that matters; the success of InDaPCA is mostly affected by the minimum number of observations available for comparing a given pair of variables. In the present study, interpretation of results in the space of the first two components was not hindered, however.

Attribute‐Associated Neuron Modeling and Missing Value Imputation for Incomplete Data

The imputation of missing values is an important research content in incomplete data analysis. Based on the auto associative neural network (AANN), this paper conducts regression modeling for incomplete data and imputes missing values. Since the AANN can estimate missing values in multiple missingness patterns efficiently, we introduce incomplete records into the modeling process and propose an attribute cross fitting model (ACFM) based on AANN. ACFM reconstructs the path of data transmission between output and input neurons and optimizes the model parameters by training errors of existing data, thereby improving its own ability to fit relations between attributes of incomplete data. Besides, for the problem of incomplete model input, this paper proposes a model training scheme, which sets missing values as variables and makes missing value variables update with model parameters iteratively. The method of local learning and global approximation increases the precision of model fitting and the imputation accuracy of missing values. Finally, experiments based on several datasets verify the effectiveness of the proposed method.

Classification for the analysis of incomplete medical data

Missing data is common in life. The processing of missing data is the key to classification. Therefore, using the existing reliable data set to complete the missing data is a common and necessary method. These methods have an important impact on dealing with the fuzziness and uncertainty in the data set. Therefore, it is necessary and effective to use accurate data and attribution methods to attribute missing data sets. This paper presents a new missing data classification method. Firstly, the center vector representing each class is calculated by using the training samples. The missing values are then estimated using the center of each class. By comparing the performance of three different interpolation methods in different test data sets, the final results show that the proposed method performs best in general.

K-means clustering for the analysis of incomplete business data

Missing values can significantly reduce the accuracy and availability of business data. Usually, when clustering incomplete data, the data with missing values are deleted, and only the complete data are analyzed. However, this often leads to significant loss or deviation of information. This paper mainly studies how to use unsupervised machine learning techniques to deal with missing values. The combination of imputation method and clustering technology forms a new method to deal with missing values, which is helpful to overcome the problem of missing data. We propose a strategy based on the combination of K-means, big data K-means, p-k-means, and mean imputation method, singular value decomposition imputation method, k-nearest neighbor imputation method. By comparing the performance of nine methods in different business data sets. The experimental analysis was carried out on four benchmark data sets. The effectiveness of K-means clustering and imputation methods is verified on different data sets, and the results also have a certain application prospect.