Value Imputation Research Articles

Economics of Subsistence Farming: A Comparative Analysis of Paddy-wheat and Maize-wheat Rotation Cropping System in Himachal Pradesh

Background: Marginalisation of Indian agriculture, due to increase in marginal and small agricultural households, brings the focus back on subsistence agriculture. Among other things, understanding the economics of subsistence agriculture will go a long way in making its practice viable by making sure the help that these subsistence farmers need are provided. Therefore, the present study attempts to study the economics of paddy-wheat and maize-wheat rotation cropping system practiced by subsistence farmers of Himachal Pradesh whose agriculture sector is known for its diversification towards high value crops. Methods: The study has used survey data collected from 60 randomly selected households from Rait block of district Kangra, Himachal Pradesh. 30 households practicing paddy-wheat rotating cultivation are selected from Boh and Drini village and 30 households doing Maize-Wheat rotating cultivation are from Machhlini, Panjol, Khridi, Rulede and Lahri village. The study is based on primary data which has been collected through a pre-tested schedule. Using the primary data, we have brought out factors other than self-consumption needs that determines choice between two farming systems and also two group of farmers are compared in terms of productivity, cost of cultivation and net returns from cultivation. Result: Our study found that irrigation facilities play an important role in the choice between the two cropping systems. Return from cultivation is negative for both groups of farmers if imputed value of family labour is included. Performance of paddy-wheat farmers is significantly better than maize-wheat farmers in economic terms in the study area.

A real data-driven simulation strategy to select an imputation method for mixed-type trait data.

Missing observations in trait datasets pose an obstacle for analyses in myriad biological disciplines. Considering the mixed results of imputation, the wide variety of available methods, and the varied structure of real trait datasets, a framework for selecting a suitable imputation method is advantageous. We invoked a real data-driven simulation strategy to select an imputation method for a given mixed-type (categorical, count, continuous) target dataset. Candidate methods included mean/mode imputation, k-nearest neighbour, random forests, and multivariate imputation by chained equations (MICE). Using a trait dataset of squamates (lizards and amphisbaenians; order: Squamata) as a target dataset, a complete-case dataset consisting of species with nearly complete information was formed for the imputation method selection. Missing data were induced by removing values from this dataset under different missingness mechanisms: missing completely at random (MCAR), missing at random (MAR), and missing not at random (MNAR). For each method, combinations with and without phylogenetic information from single gene (nuclear and mitochondrial) or multigene trees were used to impute the missing values for five numerical and two categorical traits. The performances of the methods were evaluated under each missing mechanism by determining the mean squared error and proportion falsely classified rates for numerical and categorical traits, respectively. A random forest method supplemented with a nuclear-derived phylogeny resulted in the lowest error rates for the majority of traits, and this method was used to impute missing values in the original dataset. Data with imputed values better reflected the characteristics and distributions of the original data compared to complete-case data. However, caution should be taken when imputing trait data as phylogeny did not always improve performance for every trait and in every scenario. Ultimately, these results support the use of a real data-driven simulation strategy for selecting a suitable imputation method for a given mixed-type trait dataset.

Improving Groundwater Imputation through Iterative Refinement Using Spatial and Temporal Correlations from In Situ Data with Machine Learning

Obtaining and managing groundwater data is difficult as it is common for time series datasets representing groundwater levels at wells to have large gaps of missing data. To address this issue, many methods have been developed to infill or impute the missing data. We present a method for improving data imputation through an iterative refinement model (IRM) machine learning framework that works on any aquifer dataset where each well has a complete record that can be a mixture of measured and input values. This approach corrects the imputed values by using both in situ observations and imputed values from nearby wells. We relied on the idea that similar wells that experience a similar environment (e.g., climate and pumping patterns) exhibit similar changes in groundwater levels. Based on this idea, we revisited the data from every well in the aquifer and “re-imputed” the missing values (i.e., values that had been previously imputed) using both in situ and imputed data from similar, nearby wells. We repeated this process for a predetermined number of iterations—updating the well values synchronously. Using IRM in conjuncture with satellite-based imputation provided better imputation and generated data that could provide valuable insight into aquifer behavior, even when limited or no data were available at individual wells. We applied our method to the Beryl-Enterprise aquifer in Utah, where many wells had large data gaps. We found patterns related to agricultural drawdown and long-term drying, as well as potential evidence for multiple previously unknown aquifers.

Data imputation strategies for process optimization of laser powder bed fusion of Ti6Al4V using machine learning

A database linking process parameters and material properties for additive manufacturing enables the performance of the material to be determined based on the process parameters, which are useful in the design and fabrication stage of a product. The data, however, are often incomplete as each individual research work focused on certain process parameters and material properties due to the wide range of variables available. Imputation of missing data is thus required to complete the material library. In this work, we attempt to collate the data of Ti6Al4V, a popular alloy used in aerospace and biomedical industries, fabricated using powder bed fusion, or commonly known as selective laser melting (SLM). Various imputation techniques of missing data of the SLM Ti6Al4V dataset, such as the k-nearest neighbor (kNN), multivariate imputation by chained equations, and graph imputation neural network (GINN) are investigated in this article. It was observed that kNN performed better in imputing variables related to process parameters, whereas GINN performed better in variables related to material properties. To further improve the quality of imputation, a strategy to use the median of the imputed values obtained from the three models has resulted in significant improvement in terms of the relative mean square error. Self-organizing map was used to visualize the relationship among the process parameters and the material properties.

Bayesian semiparametric joint modeling of a count outcome and inconveniently timed longitudinal predictors.

The Women's Health Initiative (WHI) Life and Longevity After Cancer (LILAC) study is an excellent resource for studying the quality of life following breast cancer treatment. At study entry, women were asked about new symptoms that appeared following their initial cancer treatment. In this article, we were interested in using regression modeling to estimate associations of clinical and lifestyle factors at cancer diagnosis (independent variables) with the number of new symptoms (dependent variable). Although clinical and lifestyle data were collected longitudinally, few measurements were obtained at diagnosis or at a consistent timepoint prior to diagnosis, which complicates the analysis. Furthermore, parametric count models, such as the Poisson and negative binomial, do not fit the symptom data well. Thus, motivated by the issues encountered in LILAC, we propose two Bayesian joint models for longitudinal data and a count outcome. Our two models differ according to the assumption on the outcome distribution: one uses a negative binomial (NB) distribution and the other a nonparametric rounded mixture of Gaussians (RMG). The mean of each count distribution is dependent on imputed values of continuous, binary, and ordinal variables at a time point of interest (e.g. diagnosis). To facilitate imputation, longitudinal variables are modeled jointly using a linear mixed model for a latent underlying normal random variable, and a Dirichlet process prior is assigned to the random subject-specific effects to relax distribution assumptions. In simulation studies, the RMG joint model exhibited superior power and predictive accuracy over the NB model when the data were not NB. The RMG joint model also outperformed an RMG model containing predictors imputed using the last value carried forward, which generated estimates that were biased toward the null. We used our models to examine the relationship between sleep health at diagnosis and the number of new symptoms following breast cancer treatment in LILAC.

Imputed mean tensor regression for near-sited spatial temporal data

Modern spatial temporal data are often collected from sensor networks. Missing data problems are common to this kind of data. Making accurate imputation is important for many applications. In the unsupervised setting, one technique is to minimize the rank of a tensor or matrix. If we add related covariates, can we get more accurate imputation results? To address this, we transform the original sensor×time measurements to high order tensors by adding additional temporal dimensions and then integrate tensor regression with tensor completion using nuclear norm penalty. One advantage is we can simultaneously estimate parameters and impute missing values due to clear spatial consistency for near-sited spatial-temporal data. The proposed method doesn't assume missing mechanism of the response. Theoretical properties of the proposed estimator are investigated. Simulation studies and real data analysis are conducted to verify the efficiency of the estimation procedure.

Quantile Regression for Count Data as a Robust Alternative to Negative Binomial Regression

The study investigated the robustness of Quantile regression of count data over negative binomial regression, when there is overdispersion and presence of outlier. The study made use of a complete data and the data with 30% missing data which was imputed using Multiple Imputation by Chain Equation (MICE) in R and also an outlier was injected into the data during imputation of missing values. The Quantile Regression and Negative Binomial Regression estimates were compared and their model fits were also compared. Results showed that the quantile regression for count data provided a better model estimate with both complete data and data with multiple imputed value with comparison to the negative binomial regression in terms of AIC, BIC RMSE and MSE. Hence, Quantile Regression is better than the negative binomial regression when the researcher is interested in the effect of the independent variable on different points of the distribution of the response variable and when there is overdispersion and presence of an outlier.

Simultaneous adjustment of uncontrolled confounding, selection bias and misclassification in multiple-bias modelling.

Adjusting for multiple biases usually involves adjusting for one bias at a time, with careful attention to the order in which these biases are adjusted. A novel, alternative approach to multiple-bias adjustment involves the simultaneous adjustment of all biases via imputation and/or regression weighting. The imputed value or weight corresponds to the probability of the missing data and serves to 'reconstruct' the unbiased data that would be observed based on the provided assumptions of the degree of bias. We motivate and describe the steps necessary to implement this method. We also demonstrate the validity of this method through a simulation study with an exposure-outcome relationship that is biased by uncontrolled confounding, exposure misclassification, and selection bias. The study revealed that a non-biased effect estimate can be obtained when correct bias parameters are applied. It also found that incorrect specification of every bias parameter by +/-25% still produced an effect estimate with less bias than the observed, biased effect. Simultaneous multi-bias analysis is a useful way of investigating and understanding how multiple sources of bias may affect naive effect estimates. This new method can be used to enhance the validity and transparency of real-world evidence obtained from observational, longitudinal studies.

Digital assets, crypto-assets and digitаl currencies: Economic content and potential of convergence

The article is devoted to researching the nature of new economic categories (digital assets, cryptoassets and digital currencies) and identifying their essential features and functional relationships between them. The study developed an original interpretation and classification of cryptoassets and digital currencies, proposed a decision tree for attributing a digital asset to a particular type, identified the features of issuance and circulation of various cryptoassets and digital currencies. The study showed that not all digital assets are cryptoassets or digital currencies. Cryptoassets should be considered as the most important and most numerous type of digital assets. According to the author’s classification, cryptoassets can be categorized into two main types: virtual currencies and digital tokens. The former perform mainly a payment and/or savings function, while the latter perform an investment and/or utilitarian function. The two main subspecies of virtual currencies are cryptocurrencies and stablecoins. Cryptocurrencies have neither an identifiable issuer nor collateral, but are characterized by intrinsic value (bitcoin) or imputed value (altcoins). Stablecoins typically have an identifiable issuer and use various reserve assets to maintain a stable market value. In terms of functionality, virtual currencies, like new digital forms of fiat money issued by central banks (central bank digital currencies) or commercial banks (tokenised deposits), can perform all or some monetary functions. The latter becomes possible due to the public consensus reached by users even in the absence of regulation of virtual currency circulation. According to the authors, modern digital currencies exist in four main forms: central bank digital currencies, tokenised deposits, stablecoins and cryptocurrencies. The potential of convergence between crypto assets and digital currencies is revealed at the instrumental, infrastructural, consumer, institutional and regulatory levels.

Adaptive Feature Projection with Distribution Alignment for Deep Incomplete Multi-view Clustering.

Incomplete multi-view clustering (IMVC) analysis, where some views of multi-view data usually have missing data, has attracted increasing attention. However, existing IMVC methods still have two issues: (1) they pay much attention to imputing or recovering the missing data, without considering the fact that the imputed values might be inaccurate due to the unknown label information, (2) the common features of multiple views are always learned from the complete data, while ignoring the feature distribution discrepancy between the complete and incomplete data. To address these issues, we propose an imputation-free deep IMVC method and consider distribution alignment in feature learning. Concretely, the proposed method learns the features for each view by autoencoders and utilizes an adaptive feature projection to avoid the imputation for missing data. All available data are projected into a common feature space, where the common cluster information is explored by maximizing mutual information and the distribution alignment is achieved by minimizing mean discrepancy. Additionally, we design a new mean discrepancy loss for incomplete multi-view learning and make it applicable in mini-batch optimization. Extensive experiments demonstrate that our method achieves the comparable or superior performance compared with state-of-the-art methods.

Machine learning approaches for the prediction of serious fluid leakage from hydrocarbon wells

AbstractThe exploitation of hydrocarbon reservoirs may potentially lead to contamination of soils, shallow water resources, and greenhouse gas emissions. Fluids such as methane or CO2may in some cases migrate toward the groundwater zone and atmosphere through and along imperfectly sealed hydrocarbon wells. Field tests in hydrocarbon-producing regions are routinely conducted for detecting serious leakage to prevent environmental pollution. The challenge is that testing is costly, time-consuming, and sometimes labor-intensive. In this study, machine learning approaches were applied to predict serious leakage with uncertainty quantification for wells that have not been field tested in Alberta, Canada. An improved imputation technique was developed by Cholesky factorization of the covariance matrix between features, where missing data are imputed via conditioning of available values. The uncertainty in imputed values was quantified and incorporated into the final prediction to improve decision-making. Next, a wide range of predictive algorithms and various performance metrics were considered to achieve the most reliable classifier. However, a highly skewed distribution of field tests toward the negative class (nonserious leakage) forces predictive models to unrealistically underestimate the minority class (serious leakage). To address this issue, a combination of oversampling, undersampling, and ensemble learning was applied. By investigating all the models on never-before-seen data, an optimum classifier with minimal false negative prediction was determined. The developed methodology can be applied to identify the wells with the highest likelihood for serious fluid leakage within producing fields. This information is of key importance for optimizing field test operations to achieve economic and environmental benefits.

Expanding Tidy Data Principles to Facilitate Missing Data Exploration, Visualization and Assessment of Imputations

Despite the large body of research on missing value distributions and imputation, there is comparatively little literature with a focus on how to make it easy to handle, explore, and impute missing values in data. This paper addresses this gap. The new methodology builds upon tidy data principles, with the goal of integrating missing value handling as a key part of data analysis workflows. We define a new data structure, and a suite of new operations. Together, these provide a connected framework for handling, exploring, and imputing missing values. These methods are available in the R package naniar.

The Net Benefits and Residual Cost from U.S. Border Management of the Initially Inadmissible

AbstractBorder management is a government activity affecting immigration and the economy. Benefit–cost and equivalent decision analyses are used to evaluate U.S. border management for 2017. Controversial issues arise. Among these are the issue of standing and the values of asylum, a criminal career, child custodial care, foreign deaths, fiscal and labor market effects, and distributional weighting. Sixteen unique shadow prices (imputed marginal value) are computed. Those shadow pries are combined with proportions and levels of border management outcomes. The aggregate result is not only a large expected present value net benefit per year from managed outcomes of $46.6 billion but also a large residual unmanaged annual cost of $23.7 billion. Significant uncertainty exists, but estimated net benefits remain positive.

Modeling Variation in Mobile Download Speed in Presence of Missing Samples

A stably fast mobile broadband connectivity is key to customer retention. Mobile networks, however, suffer unpredictability in performance. Analyzing variability in network speed is, therefore, challenging since it tends to exhibit patterns at several time scales. Additionally, frequently monitoring it over time, is costly. In this paper, we analyze speed measurements from 78 stationary probes, spread across Norway. Monitoring was performed thrice per day across the year, to assess performance of the two largest network operators. Despite being unique, the dataset involves a non-trivial extent of missing data. This study investigates the effect of missing data on the extracted performance patterns. We capture patterns with tensor factorizations, that show that missing data at random has a minimal effect on the identified patterns, and that depending upon the determinism of an operator's performance, the acceptable size and structure of missing data varies. Our analysis shows that, for a probe, the difference in speed variation between real and imputed speed values can be around 7% for up to 40% missing data. We also identify that congestion, routine maintenance and sub-optimal network configuration cause high speed variability. These findings can help operators improving their offerings and deciding on optimal performance monitoring frequency.

Clustering-Based Imputation for Dropout Buyers in Large-Scale Online Experimentation

In online experimentation, appropriate metrics (e.g., purchase) provide strong evidence to support hypotheses and enhance the decision-making process. However, incomplete metrics are frequently occurred in the online experimentation, making the available data to be much fewer than the planned online experiments (e.g., A/B testing). In this work, we introduce the concept of dropout buyers and categorize users with incomplete metric values into two groups: visitors and dropout buyers. For the analysis of incomplete metrics, we propose a clustering-based imputation method using k-nearest neighbors. Our proposed imputation method considers both the experiment-specific features and users’ activities along their shopping paths, allowing different imputation values for different users. To facilitate efficient imputation of large-scale data sets in online experimentation, the proposed method uses a combination of stratification and clustering. The performance of the proposed method is compared to several conventional methods in both simulation studies and a real online experiment at eBay.

Missing values imputation using Fuzzy K-Top Matching Value

Missing data occurs when variables or observations are missing. Researchers exclude or impute influenced variables and data. This study proposes Fuzzy K-Top Matching Value (FKTM) for missing value imputation. It imputes missing numerical and categorical data with intelligent estimates based on similar records, decreasing bias. Expectation-maximization is used, where it employs fuzzy clustering to find a group of similar data and estimate them. We compare FKTM with original datasets on Immunotherapy and Cryotherapy. Multiple classification techniques are used on the imputed datasets. Random Forest achieved the best, with 93.3% for cryotherapy and 85.6% for Immunotherapy. The proposed approach is compared with Multivariate Imputation by Chained Equations (MICE) utilizing a Support Vector Machine. The proposed approach beats MICE with 82.2% accuracy. On the Cryotherapy dataset, the proposed approach surpasses existing strategies with 86.6% accuracy. Levene and Shapiro-Wilk were used to examine the homoscedasticity and normality of data after imputation. The proposed imputation procedure has no detrimental influence on the dataset. Finally, execution time and RMSE of imputed values are determined for three datasets with varied sample sizes and data dimensions. The proposed system exhibits a fast execution time and low RMSE. The proposed FKTM works well in experiments and looks promising.

PP92 Should Missing Data Be Multiply Imputed Prior To Longitudinal Linear Mixed-Model Analyses In Trial-Based Economic Evaluations?

IntroductionFor the analysis of clinical effects, multiple imputation (MI) of missing data was shown to be unnecessary when using longitudinal linear mixed-models (LLM). It remains unclear whether this also applies to cost estimates from trial-based economic evaluations, that are generally right-skewed. Therefore, this study aimed to assess whether MI is required prior to LLM when analyzing longitudinal cost-effectiveness data.MethodsTwo-thousand complete datasets were simulated containing five time points. Incomplete datasets were generated with 10 percent, 25 percent, and 50 percent missing data in costs and effects, assuming a Missing At Random (MAR) mechanism. Statistical performance of six different methodological strategies was compared in terms of empirical bias (EB), root-mean-squared error (RMSE), and coverage rate (CR). Six strategies were compared: (i) LLM (LLM), (ii) MI prior to LLM (MI-LLM), (iii) mean imputation prior to LLM (M-LLM), (iv) complete-case analysis prior to seemingly unrelated regression (CCA-SUR), (v) MI prior to SUR (MI-SUR), and (vi) mean imputation prior to SUR (M-SUR). To evaluate the impact on the probability of cost-effectiveness at different willingness-to-pay [WTPs] thresholds, cost-effectiveness analyses were performed by applying the six strategies to two empirical datasets with 9% and 50% of missing data, respectively.ResultsFor costs and effects, LLM, MI-LLM, and MI-SUR performed better than M-LLM, CCA-SUR, and M-SUR, as indicated by smaller EBs and RMSEs, as well as CRs closer to the nominal levels of 0.95. However, even though LLM, MI-LLM, and MI-SUR performed equally well for effects, MI-LLM and MI-SUR were found to perform better than LLM for costs at 10 percent and 25 percent missing data. At 50 percent missing data, all strategies resulted in relatively high EBs and RMSEs for costs. In both empirical datasets, LLM, MI-LLM, and MI-SUR all resulted in similar probabilities of cost-effectiveness at different WTPs.ConclusionsWhen opting for using LLM for analyzing trial-based economic evaluation data, researchers are advised to multiply impute missing values first. Otherwise, MI-SUR may also be used.

Hierarchical, multiclass prediction of Alzheimer’s clinical diagnosis using imputed, multimodal NACC data

AbstractBackgroundEarly detection and diagnosis are critical for effective treatment of Alzheimer’s disease (AD). Statistical machine learning holds promise for disease prediction using large AD repositories, such as the National Alzheimer’s Coordinating Center (NACC) dataset. Past work has extensively explored binary classifications of AD diagnosis. Here, we extended this to address two major, remaining challenges: data imputation and multiclass classification (e.g., AD vs Mild Cognitive Impairment, MCI vs. Healthy Cognition, HC).MethodFor imputation, we tested single‐value (baseline), k‐NN, and Bayesian imputation on a subset of complete training data with artificially removed values (completely at random). Per modality of interest, a systematic search was conducted for the best input feature set to impute the missing values. The mean absolute deviation and mismatch rate between the imputed and true values were used to evaluate the methods on a held‐out test set. For classification, we then implemented both “flat” multiclass and hierarchical classification, an extension of multiclass classification that organizes classes into a tree, to predict clinical diagnosis using multimodal inputs. Several pipelines with combinations of imputation methods, classifiers, and input modalities were tested on different hierarchical classification strategies, e.g., HC vs [AD vs MCI], compared with the flat, multiclass baseline (AD vs MCI vs HC).ResultCompared to the baseline methods for both imputation and classification, the proposed alternatives performed better, as measured by the predetermined imputation and classification metrics evaluated on the test set. For imputation, behavioral and neuropsychiatric features were more easily and accurately imputed while ApoE genotype was more difficult to impute. For classification, using a combination of classifiers and input features in the hierarchy outperformed the baseline method and support the notion that select biomarkers are better able to characterize AD subpopulations within the hierarchy.ConclusionEstablishing a systematic imputation method tailored to AD biomarker modalities fills a critical gap in achieving more robust disease prediction. Building a hierarchical classifier parallels working through a differential diagnosis for AD, which offers greater translational value than binary classification of AD. In combination, these results form an important step toward developing reliable, holistic classifiers of AD disease status.

A Bootstrap Method for a Multiple-Imputation Variance Estimator in Survey Sampling

Rubin’s variance estimator of the multiple imputation estimator for a domain mean is not asymptotically unbiased. Kim et al. derived the closed-form bias for Rubin’s variance estimator. In addition, they proposed an asymptotically unbiased variance estimator for the multiple imputation estimator when the imputed values can be written as a linear function of the observed values. However, this needs the assumption that the covariance of the imputed values in the same imputed dataset is twice that in the different imputed datasets. In this study, we proposed a bootstrap variance estimator that does not need this assumption. Both theoretical argument and simulation studies show that it was unbiased and asymptotically valid. The new method was applied to the Hox pupil popularity data for illustration.

Imputation Strategies for Clustering Mixed-Type Data with Missing Values

Incomplete data sets with different data types are difficult to handle, but regularly to be found in practical clustering tasks. Therefore in this paper, two procedures for clustering mixed-type data with missing values are derived and analyzed in a simulation study with respect to the factors of partition, prototypes, imputed values, and cluster assignment. Both approaches are based on the k-prototypes algorithm (an extension of k-means), which is one of the most common clustering methods for mixed-type data (i.e., numerical and categorical variables). For k-means clustering of incomplete data, the k-POD algorithm recently has been proposed, which imputes the missings with values of the associated cluster center. We derive an adaptation of the latter and additionally present a cluster aggregation strategy after multiple imputation. It turns out that even a simplified and time-saving variant of the presented method can compete with multiple imputation and subsequent pooling.