Nearest-neighbor Imputation Research Articles

Variance estimation for nearest neighbor imputation for US Census long form data

Variance estimation for estimators of state, county, and school district quantities derived from the Census 2000 long form are discussed. The variance estimator must account for (1) uncertainty due to imputation, and (2) raking to census population controls. An imputation procedure that imputes more than one value for each missing item using donors that are neighbors is described and the procedure using two nearest neighbors is applied to the Census long form. The Kim and Fuller [Biometrika 91 (2004) 559--578] method for variance estimation under fractional hot deck imputation is adapted for application to the long form data. Numerical results from the 2000 long form data are presented.

Statistical file matching of flow cytometry data

Flow cytometry is a technology that rapidly measures antigen-based markers associated to cells in a cell population. Although analysis of flow cytometry data has traditionally considered one or two markers at a time, there has been increasing interest in multidimensional analysis. However, flow cytometers are limited in the number of markers they can jointly observe, which is typically a fraction of the number of markers of interest. For this reason, practitioners often perform multiple assays based on different, overlapping combinations of markers. In this paper, we address the challenge of imputing the high-dimensional jointly distributed values of marker attributes based on overlapping marginal observations. We show that simple nearest neighbor based imputation can lead to spurious subpopulations in the imputed data and introduce an alternative approach based on nearest neighbor imputation restricted to a cell’s subpopulation. This requires us to perform clustering with missing data, which we address with a mixture model approach and novel EM algorithm. Since mixture model fitting may be ill-posed in this context, we also develop techniques to initialize the EM algorithm using domain knowledge. We demonstrate our approach on real flow cytometry data.

Mapping gradients of community composition with nearest‐neighbour imputation: extending plot data for landscape analysis

AbstractQuestion: How can nearest‐neighbour (NN) imputation be used to develop maps of multiple species and plant communities?Location: Western and central Oregon, USA, but methods are applicable anywhere.Methods: We demonstrate NN imputation by mapping woody plant communities for >100 000 km2 of diverse forests and woodlands. Species abundances on ∼25 000 plots were related to spatial predictors (rasters) describing climate, topography, soil and geographic location using constrained ordination (CCA). Species data from the nearest plot in multi‐dimensional CCA space were imputed to each map pixel. Maps of multiple individual species and community types were constructed from the single imputed surface. We computed a variety of diagnostics to characterize different qualities of the imputed (mapped) community data.Results: Community composition gradients were strongly associated with climate and elevation, and less so with topography and soil. Accuracy of the imputation model for presence/absence of 150 species varied widely (kappa 0.00 to 0.80). Omission error rates were higher than commission rates due to low species prevalence, and areal representation of species was only slightly inflated. A map of 78 community types was 41% correct and 78% fuzzy correct. Errors of omission and commission were balanced, and areal representation of both rare and abundant communities was accurate. Map accuracy may be lower for some species than with other methods, but areal representation of species and communities across large landscapes is preserved. Because imputed vegetation surfaces are developed for all species simultaneously, map units contain suites of species known to co‐occur in nature. Maps of individual species, and of community types derived from them, will be internally consistent at map locations.Conclusions: NN imputation is a useful modelling approach where maps of multiple species and plant communities are needed, such as in natural resource management and conservation planning or models that project landscape change under alternative disturbance or climate scenarios. More research is needed to evaluate other ordination methods for NN imputation of plant communities.

Incomplete-case nearest neighbor imputation in software measurement data

k nearest neighbor imputation (kNNI) is one of the most popular methods in empirical software engineering for imputing missing values. kNNI typically uses only complete cases as possible donors for imputation (called complete case kNNI or CCkNNI). Though it often produces reasonable results, CCkNNI is severely limited when the amount of missing data is large (and hence the number of complete cases is small). In response, a variant of CCkNNI called incomplete case k nearest neighbor imputation (ICkNNI) has been proposed as an attractive alternative. This work presents a detailed simulation comparing CCkNNI and ICkNNI using two different software measurement datasets. The empirical results show that using incomplete cases often increases the effectiveness of nearest neighbor imputation (especially at higher missingness levels), regardless of the type of missingness (i.e., the distribution of missing values in the data).

Predicting potential and actual distribution of sudden oak death in Oregon: Prioritizing landscape contexts for early detection and eradication of disease outbreaks

An isolated outbreak of the emerging forest disease sudden oak death was discovered in Oregon forests in 2001. Despite considerable control efforts, disease continues to spread from the introduction site due to slow and incomplete detection and eradication. Annual field surveys and laboratory tests between 2001 and 2009 confirmed a total of 802 infested locations. Here, we apply two invasive species distribution models (iSDMs) of sudden oak death establishment and spread risk to target early detection and control further disease spread in Oregon forests. The goal was to develop (1) a model of potential distribution that estimates the level and spatial variability of disease establishment and spread risk for western Oregon, and (2) a model of actual distribution that quantifies the relative likelihood of current invasion in the quarantine area. Our predictions were based on four groups of primary parameters that vary in space and time: climate conditions, topographical factors, abundance and susceptibility of host vegetation, and dispersal pressure. First, we used multi-criteria evaluation to identify large-scale areas at potential risk of infection. We mapped and ranked host abundance and susceptibility using geospatial vegetation data developed with gradient nearest neighbor imputation. The host vegetation and climate variables were parameterized in accordance to their epidemiological importance and the final appraisal scores were summarized by month to represent a cumulative spread risk index, standardized as five categories from very low to very high risk. Second, using the field data for calibration we applied the machine-learning method, maximum entropy, to predict the actual distribution of the sudden oak death epidemic. The dispersal pressure incorporated in the statistical model estimates the force of invasion at all susceptible locations, allowing us to quantify the relative likelihood of current disease incidence rather than its potential distribution. Our predictions show that 65 km 2 of forested land was invaded by 2009, but further disease spread threatens more than 2100 km 2 of forests across the western region of Oregon (very high and high risk). Areas at greatest risk of disease spread are concentrated in the southwest region of Oregon where the highest densities of susceptible host species exist. This research identifies high priority locations for early detection and invasion control and illustrates how iSDMs can be used to analyze the actual versus potential distribution of emerging infectious disease in a complex, heterogeneous ecosystem.

Imputation of single-tree attributes using airborne laser scanning-based height, intensity, and alpha shape metrics

Forest inventories based on single-tree interpretation of airborne laser scanning (ALS) data often rely on an allometric estimation chain in which inaccuracies in the estimates of the diameter at breast height (DBH) propagate to other characteristics of interest such as the stem volume. Our purpose was to test nearest neighbor imputation by the k-Most Similar Neighbor ( k-MSN) and the Random Forest (RF) methods for the simultaneous estimation of species, DBH, height and stem volume using ALS data. The predictors included computational alpha shape metrics and variables based on the height and intensity distributions in the ALS data. Separate data sets covering 1898 and 1249 dominant to intermediate trees in a typical Scandinavian stand structure were used for training and validation, respectively. RF proved to be a flexible method with an ability to handle 1846 predictors with no need for their reduction. Classification of Scots pine, Norway spruce and deciduous trees showed an accuracy of 78%, and the estimates of DBH, height and volume had root mean square errors of 13%, 3%, and 31%, respectively, when evaluated against the validation data. The two selection strategies implemented here reduced the number of candidate variables effectively without any substantial effect on the accuracy relative to the use of all predictors. Differences in k-MSN and RF imputations were marginal when the reduced sets of variables were used. Estimation accuracies could be maintained practically unchanged with only 12.5% of the initial reference data (237 trees), provided the distribution of the observations was similar in the reference and target data. Since we used information collected in the field for extracting the ALS point clouds for individual trees, our results represent an optimal case and should nevertheless be validated against automated tree delineation.

Quantification of live aboveground forest biomass dynamics with Landsat time-series and field inventory data: A comparison of empirical modeling approaches

Spatially and temporally explicit knowledge of biomass dynamics at broad scales is critical to understanding how forest disturbance and regrowth processes influence carbon dynamics. We modeled live, aboveground tree biomass using Forest Inventory and Analysis (FIA) field data and applied the models to 20+ year time-series of Landsat satellite imagery to derive trajectories of aboveground forest biomass for study locations in Arizona and Minnesota. We compared three statistical techniques (Reduced Major Axis regression, Gradient Nearest Neighbor imputation, and Random Forests regression trees) for modeling biomass to better understand how the choice of model type affected predictions of biomass dynamics. Models from each technique were applied across the 20+ year Landsat time-series to derive biomass trajectories, to which a curve-fitting algorithm was applied to leverage the temporal information contained within the time-series itself and to minimize error associated with exogenous effects such as biomass measurements, phenology, sun angle, and other sources. The effect of curve-fitting was an improvement in predictions of biomass change when validated against observed biomass change from repeat FIA inventories. Maps of biomass dynamics were integrated with maps depicting the location and timing of forest disturbance and regrowth to assess the biomass consequences of these processes over large areas and long time frames. The application of these techniques to a large sample of Landsat scenes across North America will facilitate spatial and temporal estimation of biomass dynamics associated with forest disturbance and regrowth, and aid in national-level estimates of biomass change in support of the North American Carbon Program.

Mapping wildland fuels and forest structure for land management: a comparison of nearest neighbor imputation and other methods

Land managers need consistent information about the geographic distribution of wildland fuels and forest structure over large areas to evaluate fire risk and plan fuel treatments. We compared spatial predictions for 12 fuel and forest structure variables across three regions in the western United States using gradient nearest neighbor (GNN) imputation, linear models (LMs), classification and regression trees (CART), and geostatistical methods (kriging and universal kriging (UK)). Local-scale map accuracy varied considerably across sites, variables, and methods. GNN performed best for forest structure variables in Oregon, but LMs and UK were better for canopy variables and for forest structure variables in Washington and California. Kriging performed poorly throughout, and kriging did not improve prediction accuracy when added to LMs as UK. GNN outperformed CART in predicting vegetation classes and fuel models, complex variables defined by multiple attributes. Regional distributions of vegetation classes and fuel models were accurately represented by GNN and very poorly by CART and LMs. Despite their often limited accuracy at the local scale, GNN maps are useful when information on a wide range of forest attributes is needed for analysis and forest management at intermediate, i.e., landscape to ecoregional, scales.

Estimating cavity tree and snag abundance using negative binomial regression models and nearest neighbor imputation methods

Cavity tree and snag abundance data are highly variable and contain many zero observations. We predict cavity tree and snag abundance from variables that are readily available from forest cover maps or remotely sensed data using negative binomial (NB), zero-inflated NB, and zero-altered NB (ZANB) regression models as well as nearest neighbor (NN) imputation methods. The models were developed and fit to data collected by the Forest Inventory and Analysis program of the US Forest Service in Washington, Oregon, and California. For predicting cavity tree and snag abundance per stand, all three NB regression models performed better in terms of mean square prediction error than the NN imputation methods. The most similar neighbor imputation, however, outperformed the NB regression models in predicting overall cavity tree and snag abundance.

The roles of nearest neighbor methods in imputing missing data in forest inventory and monitoring databases

Almost universally, forest inventory and monitoring databases are incomplete, ranging from missing data for only a few records and a few variables, common for small land areas, to missing data for many observations and many variables, common for large land areas. For a wide variety of applications, nearest neighbor (NN) imputation methods have been developed to fill in observations of variables that are missing on some records (Y-variables), using related variables that are available for all records (X-variables). This review attempts to summarize the advantages and weaknesses of NN imputation methods and to give an overview of the NN approaches that have most commonly been used. It also discusses some of the challenges of NN imputation methods. The inclusion of NN imputation methods into standard software packages and the use of consistent notation may improve further development of NN imputation methods. Using X-variables from different data sources provides promising results, but raises the issue of spatial and temporal registration errors. Quantitative measures of the contribution of individual X-variables to the accuracy of imputing the Y-variables are needed. In addition, further research is warranted to verify statistical properties, modify methods to improve statistical properties, and provide variance estimators.

Imputing mean annual change to estimate current forest attributes

When a temporal trend in forest conditions is present, standard estimates from paneled forest inventories can be biased. Thus methods that use more recent remote sensing data to improve estimates are desired. Paneled inventory data from national forests in Oregon and Washington, U.S.A., were used to explore three nearest neighbor imputation methods to estimate mean annual change of four forest attributes (basal area/ha, stems/ha, volume/ha, biomass/ha). The randomForest imputation method outperformed the other imputation approaches in terms of root mean square error. The imputed mean annual change was used to project all panels to a common point in time by multiplying the mean annual change with the length of the growth period between measurements and adding the change estimate to the previously observed measurements of the four forest attributes. The resulting estimates of the mean of the forest attributes at the current point in time outperformed the estimates obtained from the national standard estimator.

Spatially consistent nearest neighbor imputation of forest stand data

This study suggests a method for improving spatial consistency in the estimation of forest stand data. Traditional nearest neighbor imputation can preserve between-variable consistency within a unit, but not between geographically nearby units. The lack of spatial consistency may cause problems when data are used for purposes of forestry planning or scenario analysis. In spatially consistent nearest neighbor imputation, adjacent units are considered in the estimation. The first step of the method is a k-Nearest Neighbor imputation. Secondly, based on the initial imputation an optimization algorithm, Simulated Annealing, is applied in order to reach certain spatial variation targets. The proposed method was tested in a case study where tree stem volume data were imputed to each unit (pixel) of forest stands, using satellite digital numbers as carrier data. The spatial variation measures used were between-pixel correlation and short-range variance. In addition, accuracy of the estimated stand level mean volume was used as a target in order to avoid drifts in mean volume during the optimization. The method was successful in three out of four stands where it resulted in imputations corresponding exactly to the target spatial variation measures. In the fourth stand it was not possible to find an exact solution. However, in this case the two spatial variation targets were reached whereas the mean stem volume was slightly overestimated (stem volume of 375 m 3 ha − 1 rather than 336 m 3 ha − 1 ).

Imputation techniques for multivariate missingness in software measurement data

The problem of missing values in software measurement data used in empirical analysis has led to the proposal of numerous potential solutions. Imputation procedures, for example, have been proposed to `fill-in' the missing values with plausible alternatives. We present a comprehensive study of imputation techniques using real-world software measurement datasets. Two different datasets with dramatically different properties were utilized in this study, with the injection of missing values according to three different missingness mechanisms (MCAR, MAR, and NI). We consider the occurrence of missing values in multiple attributes, and compare three procedures, Bayesian multiple imputation, k Nearest Neighbor imputation, and Mean imputation. We also examine the relationship between noise in the dataset and the performance of the imputation techniques, which has not been addressed previously. Our comprehensive experiments demonstrate conclusively that Bayesian multiple imputation is an extremely effective imputation technique.

Efficient and Doubly Robust Imputation for Covariate-Dependent Missing Responses

In this article we study a well-known response missing-data problem. Missing data is an ubiquitous problem in medical and social science studies. Imputation is one of the most popular methods for dealing with missing data. The most commonly used imputation that makes use of covariates is regression imputation, in which the regression model can be parametric, semiparametric, or nonparametric. Parametric regression imputation is efficient but is not robust against misspecification of the regression model. Although nonparametric regression imputation (such as nearest-neighbor imputation and kernel regression imputation) is model-free, it is not efficient, especially if the dimension of covariate vector is high (the well-known problem of curse of dimensionality). Semiparametric regression imputation (such as partially linear regression imputation) can reduce the dimension of the covariate in nonparametric regression fitting but is not robust against misspecification of the linear component in the regression. Assuming that the missing mechanism is covariate-dependent and that the propensity function can be specified correctly, we propose a regression imputation method that has good efficiency and is robust against regression model misspecification. Furthermore, our method is valid as long as either the regression model or the propensity model is correct, a property known as the double-robustness property. We show that asymptotically the sample mean based on our imputation achieves the semiparametric efficient lower bound if both regression and propensity models are specified correctly. Our simulation results demonstrate that the proposed method outperforms many existing methods for handling missing data, especially when the regression model is misspecified. As an illustration, an economic observational data set is analyzed.

Nearest neighbor imputation of species-level, plot-scale forest structure attributes from LiDAR data

Meaningful relationships between forest structure attributes measured in representative field plots on the ground and remotely sensed data measured comprehensively across the same forested landscape facilitate the production of maps of forest attributes such as basal area (BA) and tree density (TD). Because imputation methods can efficiently predict multiple response variables simultaneously, they may be usefully applied to map several structural attributes at the species-level. We compared several approaches for imputing the response variables BA and TD, aggregated at the plot-scale and species-level, from topographic and canopy structure predictor variables derived from discrete-return airborne LiDAR data. The predictor and response variables were associated using imputation techniques based on normalized and unnormalized Euclidean distance, Mahalanobis distance, Independent Component Analysis (ICA), Canonical Correlation Analysis (aka Most Similar Neighbor, or MSN), Canonical Correspondence Analysis (aka Gradient Nearest Neighbor, or GNN), and Random Forest (RF). To compare and evaluate these approaches, we computed a scaled Root Mean Square Distance (RMSD) between observed and imputed plot-level BA and TD for 11 conifer species sampled in north-central Idaho. We found that RF produced the best results overall, especially after reducing the number of response variables to the most important species in each plot with regard to BA and TD. We concluded that RF was the most robust and flexible among the imputation methods we tested. We also concluded that canopy structure and topographic metrics derived from LiDAR surveys can be very useful for species-level imputation.

YaImpute: AnRPackage forkNN Imputation

This article introduces yaImpute, an R package for nearest neighbor search and imputation. Although nearest neighbor imputation is used in a host of disciplines, the methods implemented in the yaImpute package are tailored to imputation-based forest attribute estimation and mapping. The impetus to writing the yaImpute is a growing interest in nearest neighbor imputation methods for spatially explicit forest inventory, and a need within this research community for software that facilitates comparison among different nearest neighbor search algorithms and subsequent imputation techniques. yaImpute provides directives for defining the search space, subsequent distance calculation, and imputation rules for a given number of nearest neighbors. Further, the package offers a suite of diagnostics for comparison among results generated from different imputation analyses and a set of functions for mapping imputation results.

Estimating cavity tree abundance using Nearest Neighbor Imputation methods for western Oregon and Washington forests

Cavity trees contribute to diverse forest structure and wildlife habitat. For a given stand, the size and density of cavity trees indicate its diversity, complexity, and suitability for wildlife habitat. Size and density of cavity trees vary with stand age, density, and structure. Using Forest Inventory and Analysis (FIA) data collected in western Oregon and western Washington, we applied correlation analysis and graphical approaches to examine relationships between cavity tree abundance and stand characteristics. Cavity tree abundance was negatively correlated with site index and percent composition of conifers, but positively correlated with stand density, quadratic mean diameter, and percent composition of hardwoods. Using FIA data, we examined the performance of Most Similar Neighbor (MSN), k nearest neighbor, and weighted MSN imputation with three variable transformations (regular, square root, and logarithmic) and Classification and Regression Tree with MSN imputation to estimate cavity tree abundance from stand attributes. There was a large reduction in mean root mean square error from 20% to 50% reference sets, but very little reduction in using the 80% reference sets, corresponding to the decreases in mean distances. The MSN imputation using square root transformation provided better estimates of cavity tree abundance for western Oregon and western Washington forests. We found that cavity trees were only 0.25 percent of live trees and 13.8 percent of dead trees in the forests of western Oregon and western Washington, thus rarer and more difficult to predict than many other forest attributes. Potential applications of MSN imputation include selecting and modeling wildlife habitat to support forest planning efforts, regional inventories, and evaluation of different management scenarios.

Multiple imputation using an iterative hot‐deck with distance‐based donor selection

Hot-deck imputation offers advantages in reflecting salient features of data distributions in missing-data problems, but previous implementations have lacked the appeal associated with modern Bayesian statistical-computing techniques. We outline a strategy of iterative hot-deck multiple imputation with distance-based donor selection. With distance defined as a monotonic function of the difference in predictive means between cases, donors are chosen with probability inversely proportional to their distance from the donee. This method retains the implementation ease of ad hoc techniques, while incorporating the desirable features of Bayesian approaches. Special cases of our method include nearest-neighbor imputation and a simple random hot-deck. Iterating the procedure provides an analogy to Markov Chain Monte Carlo methods and is intended to mitigate dependence on starting values. Results from imputing missing values in a longitudinal depression treatment trial as well as a simulation study are presented. We evaluate how different definitions of distance, choices of starting values, the order in which variables are chosen for imputation, and the number of iterations impact inferences. We show that our measure of distance controls the tradeoff between bias and variance of our estimates. We find that inferences from the depression treatment trial are not sensitive to most definitions of distance. In addition, while differences exist between 1 iteration and 10 iterations, there are no meaningful differences between inferences based on 10 iterations and those based on 500 iterations. The choice of starting value did not have an impact on inferences but the order in which the variables were chosen for imputation was significant even after iteration.

Analysis of a Pilot Study for Amelioration of Itching in Liver Disease

This article presents an analysis of a small “failed clinical trial” that shows how the combination of solid statistical theory and methods and expert collaboration from other disciplines can lead to new areas of research. A double-blind placebo-controlled pilot study of 15 subjects was conducted in order to examine the efficacy of gabapentin in reducing itching and scratching secondary to liver disease. The data from the study contained missing values due to a number of mechanisms. In addition, the measurement instrument produced highly variable data, including large outliers. This article describes how these issues were resolved for this trial in an expedient manner, and particular attention is paid to a modification of nearest-neighbor hot-deck imputation. Although the study results were inconclusive with respect to the efficacy of gabapentin on itching, when they are considered in the light of other known properties of the drug, the outcome points to an exciting new line of inquiry on the role of gabapentin in regulating the placebo effect itself.

Effects of replacing the unreliable cDNA microarray measurements on the disease classification based on gene expression profiles and functional modules

Microarrays datasets frequently contain a large number of missing values (MVs), which need to be estimated and replaced for subsequent data mining. The focus of the paper is to study the effects of different MV treatments for cDNA microarray data on disease classification analysis. By analyzing five datasets, we demonstrate that among three kinds of classifiers evaluated in this study, support vector machine (SVM) classifiers are robust to varied MV imputation methods [e.g. replacing MVs by zero, K nearest-neighbor (KNN) imputation algorithm, local least square imputation and Bayesian principal component analysis], while the classification and regression tree classifiers are sensitive in terms of classification accuracy. The KNNclassifiers built on differentially expressed genes (DEGs) are robust to the varied MV treatments, but the performances of the KNN classifiers based on all measured genes can be significantly deteriorated when imputing MVs for genes with larger missing rate (MR) (e.g. MR > 5%). Generally, while replacing MVs by zero performs relatively poor, the other imputation algorithms have little difference in affecting classification performances of the SVM or KNN classifiers. We further demonstrate the power and feasibility of our recently proposed functional expression profile (FEP) approach as means to handle microarray data with MVs. The FEPs, which are derived from the functional modules that are enriched with sets of DEGs and thus can be consistently identified under varied MV treatments, achieve precise disease classification with better biological interpretation. We conclude that the choice of MV treatments should be determined in context of the later approaches used for disease classification. The suggested exclusion criterion of ignoring the genes with larger MR (e.g. >5%), while justifiable for some classifiers such as KNN classifiers, might not be considered as a general rule for all classifiers.