Issue Of Missing Data Research Articles

Stability enhanced variable selection for a semiparametric model with flexible missingness mechanism and its application to the ChAMP study

ABSTRACT This paper is motivated by the analytical challenges we encounter when analyzing the ChAMP (Chondral Lesions And Meniscus Procedures) study, a randomized controlled trial to compare debridement to observation of chondral lesions in arthroscopic knee surgery. The main outcome, WOMAC (Western Ontario and McMaster Universities Osteoarthritis Index) pain score, is derived from the patient's responses to the questionnaire collected in the study. The major goal is to identify potentially important variables that contribute to this outcome. In this paper, the model of interest is a semiparametric model for the pain score. To address the missing data issue, we adopt a flexible missingness mechanism that is much more versatile in practice than a single parametric model. Then we propose a pairwise conditional likelihood approach to estimate the unknown parameter in the semiparametric model without the need of modeling its nonparametric counterpart nor the missingness mechanism. For variable selection, we apply a regularization approach with a variety of stability enhanced tuning parameter selection methods. We conduct comprehensive simulation studies to evaluate the performance of the proposed method. We also apply the proposed method to the ChAMP study to demonstrate its usefulness.

Hollow-tree: a metric access method for data with missing values

Similarity search is fundamental to store and retrieve large volumes of complex data required by many real world applications. A useful mechanism for such concept is the query-by-similarity. Based on their topological properties, metric similarity functions can be used to index sets of data which can be queried effectively and efficiently by the so-called metric access methods. However, data produced by various application domains and the varying data types handled often lead to missing data, hence, they do not follow the metric similarity requirements. As a consequence, missing data cause distortions in the index structure and yield bias in the query answer. In this paper, we propose the Hollow-tree, a novel access method aimed at successfully retrieving data with missing attribute values. It employs new strategies for indexing and searching data elements, capable of handling the missing data issues when the cause of missingness is ignorable. The indexing strategy is based on a family of distance functions that allow measuring the distance between elements with missing values, along with a set of policies able to organize the elements in the index without causing distortions to its internal structure. The searching strategy employs fractal dimension property of the data to achieve accurate query answer while considering data with missing values part of the response. Results from experiments performed on a variety of real and synthetic data sets showed that, while other metric access methods deteriorate with small amounts of missing values, the Hollow-tree maintains a remarkable performance with almost 100% of precision and recall for range queries and more than 90% for k-nearest neighbor queries, for up to 40% of missing values.

A Standardized Atmospheric Measurement Data Archive for Distributed Cloud and Precipitation Process-Oriented Observations in Central Europe

AbstractCentral Europe has a vital and extensive meteorological research community comprising national weather services, universities, and research organizations and institutes. Nearly all of them are involved in the open scientific questions regarding clouds and precipitation processes. The research activities include observations (from in situ ground-based remote sensing radio soundings to satellite-based observations), model development on all scales (from direct numerical simulations to global climate models), and other activities. With Germany as an example our first objective is to show the large amount and the diversity of observations regarding clouds and precipitation. The goal is to give an overview of existing measurements and datasets to show the benefit of combining the different information from a variety of observations. Up to now the access to and the usage of these datasets from different sources was not straightforward, due to the issue of missing data and archiving standards for observational data. This then motivates our second objective, which is to introduce our solution for this issue—the novel Standardized Atmospheric Measurement Data archive (SAMD). SAMD is one of the outcomes of the German research initiative High Definition Clouds and Precipitation for Advancing Climate Prediction [HD(CP)2]. The goal of SAMD is an easy-to-use approach for both data producers and archive users. Therefore the archive provides observational data in the common Climate Forecast (CF) Conventions format and makes it available to the broader public. SAMD offers highly standardized quality-controlled data and metadata for a wide range of instruments, with open access, which makes this novel archive important for the research community.

Comparative study of two imputation methods in R package for RGB color histogram data

Multiple imputation (MI) is a powerful tool in handling missing data issue. This paper provides a comparison of the multiple imputation method in Amelia II package and MICE package in R. Both packages are well-known and incredible to conduct the missing data research in numerous domains. There are very limited researches comparing the multiple imputation combined with other techniques in the image data context. We employ the mean absolute error (MAE) error metric to evaluate the accuracy on the predicted values based on 20% and 50% of missing data percentages. Although the implementation of MICE is time consuming, the result shows that MICE can deal with large amount of missing values while Amelia II is only capable to deal up 20% amount of missing values. Based on the MAE result, both packages show that they are superior on the particular variables.

Missing Data Management Practices in L2 Research: The Good, The Bad and The Ugly

Missing data are one of the frequently encountered problems in quantitative research. When neglected or handled improperly, this problem can have adverse impact on research results. However, the issue of missing data in quantitative second language (L2) research has largely been ignored when compared to the other sister disciplines such as education and psychology. The purpose of this methodological synthesis was, therefore, to investigate the issue of missing data in L2 research, with a particular focus on L2 researchers’ current missing data management practices. A total of 143 studies published in six leading L2 journals were reviewed in this synthesis. The results indicated that missing data were indeed quite common in L2 research in that 41% of the studies indicated evidence of missing data, but L2 researchers’ management and reporting of missing data was often less than optimal. In light of the results, several directed suggestions were made to improve the rigor and quality of L2 research.

Central nervous system melioidosis: A systematic review of individual participant data of case reports and case series.

BackgroundCentral nervous system (CNS) melioidosis is rare. However, delayed diagnosis and treatment could lead to fatality. To identify knowledge of CNS melioidosis, we systematically review case reports and case series.Methodology/Principal findingsWe searched through PubMed, Web of Science and Thai-Journal Citation Index databases as well as Google Scholar with the last date on July 10, 2018. The diagnosis of CNS melioidosis had to be confirmed with culture, serology or polymerase chain reaction. We excluded the animal cases and the studies that the clinical data were not available. We identified 1170 relevant studies, while 70 studies with a total of 120 patients were analyzed. Ninety-three percent of patients were reported from the endemic area of melioidosis. Median age was 40 years (IQR 18–53), and 70% were men. A total of 60% had one or more risk factors for melioidosis. The median duration from clinical onset to diagnosis was ten days (IQR 5–25). Fever (82%), headache (54%), unilateral weakness (57%) and cranial nerve deficits (52%) are among the prominent presentation. Most patient (67%) had at least one extraneurological organ involvement. The CSF profile mostly showed mononuclear pleocytosis (64%), high protein (93%) and normal glucose (66%). The rim-enhancing pattern (78%) is the most frequent neuroimaging finding in encephalomyelitis and brain abscess patients. Both brainstem (34%) and frontal lobe (34%) are the most affected locations. Mortality rate was 20%.Conclusions/SignificanceThis study is the most extensive systematic review of case reports and case series of CNS melioidosis in all age groups. However, the results should be cautiously interpreted due to the missing data issue. The propensity of brainstem involvement which correlates with prominent cranial nerve deficits is the characteristic of CNS melioidosis especially encephalomyelitis type. The presenting features of fever and neurological deficits (especially cranial nerve palsies) along with the mononuclear CSF pleocytosis in a patient who lives in the endemic area and also has the risk factor for melioidosis should raise the CNS melioidosis as the differential diagnosis.

A vision-based marker-less pose estimation system for articulated construction robots

Abstract The prospect of human-robot collaborative work on construction sites introduces new workplace hazards that must be mitigated to ensure safety. Human workers working on tasks alongside construction robots must perceive the interaction to be safe in order to ensure team identification and trust. Detecting the robot pose in real-time is thus an essential requirement to inform the workers and to enable autonomous operation. Vision-based (marker-based, marker-less) and sensor-based are two of the primary methods for estimating robot pose. The marker-based and sensor-based methods require some additional preinstalled sensors or markers, whereas the marker-less method only requires an on-site camera system, which is common on today's construction sites. This research developed a marker-less pose estimation system for on-site articulated construction robots, which is based on a deep convolutional network human pose estimation algorithm: stacked hourglass network. Both 2D and 3D pose are estimated. The system is trained with image datasets collected from a robotic excavator and annotations of excavator pose, as well as conventional excavators working on construction sites. A KUKA robot arm with a bucket mounted on the end-effector was used to represent the robotic excavator in the experiments. The marker-less 3D method was evaluated, and the results were compared with the sensor-based results and the robot's ground truth pose. The results demonstrated that the marker-less 2D and 3D pose estimation methods are capable of performing proximity detection and object tracking on construction sites and can overcome the missing data issues encountered in the sensor-based method. However, the lower accuracy of the bucket pose estimation due to occlusion highlights the need for modifying the network and collecting additional datasets for training in future work.

Relationship between sociodemographic factors and specialty destination of UK trainee doctors: a national cohort study

ObjectivesMany countries are driving forward policies to widen the socioeconomic profile of medical students and to train more medical students for certain specialties. However, little is known about how socioeconomic...

Parameter Estimation of Heavy-Tailed AR Model With Missing Data Via Stochastic EM

The autoregressive (AR) model is a widely used model to understand time series data. Traditionally, the innovation noise of the AR is modeled as Gaussian. However, many time series applications, for example, financial time series data, are non-Gaussian, therefore, the AR model with more general heavy-tailed innovations is preferred. Another issue that frequently occurs in time series is missing values, due to system data record failure or unexpected data loss. Although there are numerous works about Gaussian AR time series with missing values, as far as we know, there does not exist any work addressing the issue of missing data for the heavy-tailed AR model. In this paper, we consider this issue for the first time, and propose an efficient framework for parameter estimation from incomplete heavy-tailed time series based on a stochastic approximation expectation maximization (SAEM) coupled with a Markov Chain Monte Carlo (MCMC) procedure. The proposed algorithm is computationally cheap and easy to implement. The convergence of the proposed algorithm to a stationary point of the observed data likelihood is rigorously proved. Extensive simulations and real datasets analyses demonstrate the efficacy of the proposed framework.

Improvement of a Deep Learning Algorithm for Total Electron Content Maps: Image Completion

AbstractObservations with a global coverage are very important for space physics research and space weather monitoring. However, due to the technical limitations, it would be very expensive or even impossible to achieve a seamless global coverage even with advanced observational devices. It would be useful to fill missing data gaps to create a global map from the available data, but up until now this has been very challenging. Fortunately, the deep learning method, a recent breakthrough in artificial intelligence, may provide an effective way to solve this problem by making full use of data from reliable observations. In this paper, a promising deep learning algorithm, deep convolutional generative adversarial network (DCGAN), is investigated to fill the missing data of total electron content (TEC) map images. The direct use of DCGAN fails to fill missing data for the completion of TEC maps because there are always missing TEC data in some regions, such as oceans, where the features vary with time and geophysical conditions. Thus, no useful information can be utilized by DCGAN to achieve a meaningful image completion. In order to overcome this shortcoming of the original DCGAN method, a novel regularized DCGAN (R‐DCGAN) is proposed by adding an extra discriminator and some widely used reference TEC maps from the International Global Navigation Satellite Systems Service Ionosphere Working Group. The proposed R‐DCGAN method generates satisfactory ionospheric peak structures at different times and geomagnetic conditions, which demonstrate its effectiveness on filling the missing data in TEC maps. The proposed R‐DCGAN framework can be readily extended to a broad application in other fields of space sciences, particularly for addressing the missing observation data issues.

Data Mining of Spatio-Temporal Variability of Chlorophyll-a Concentrations in a Portion of the Western Atlantic with Low Performance Hardware

The contemporary scientific literature that deals with the dynamics of marine chlorophyll-a concentration is already customarily employing data mining techniques in small geographic areas or regional samples. However, there is little focus on the issue of missing data related to chlorophyll-a concentration estimated by remote sensors. Intending to provide greater scope to the identification of the spatiotemporal distribution patterns of marine chlorophyll-a concentrations, and to improve the reliability of results, this study presents a data mining approach to cluster similar chlorophyll-a concentration behaviors while implementing an iterative spatiotemporal interpolation technique for missing data inference. Although some dynamic behaviors of said concentrations in specific areas are already known by specialists, systematic studies in large geographical areas are still scarce due to the computational complexity involved. For this reason, this study analyzed 18 years of NASA satellite observations in one portion of the Western Atlantic Ocean, totaling more than 60 million records. Additionally, performance tests were carried out in low-cost computer systems to check the accessibility of the proposal implemented for use in computational structures of different sizes. The approach was able to identify patterns with high spatial resolution, accuracy and reliability, rendered in low-cost computers even with large volumes of data, generating new and consistent patterns of spatiotemporal variability. Thus, it opens up new possibilities for data mining research on a global scale in this field of application.

Medical card data imputation and patient psychological and behavioral profile construction

Missing data is a typical problem for many hands-on tasks and researches, which has required human intervention and contributed to an increase in errors during algorithms application that demand for a large number of metrics. Solving this particular problem is essential for medicine and healthcare, because it allows more easily diagnosing certain types of diseases, improving medical service quality, etc. The main approach for medical data imputation is to automate this process at all stages, beginning from finding the NA (Not Available) or missing Data, to the completion of the analysis and insertion of lost information entity. The proposed methods of mathematical computing and modeling, statistical functions, data flow diagrams during the imputation, and the use of computer programming tools should be implemented in the medical field to improve and address the missing data issue. The evaluation of key characteristics (algorithm’s error, number of imputed data, datasets dimensionality) helped to determine the factors for obtaining the most accurate result with the help of various algorithms and functions. The study is useful for the medical industry in general, since it will eliminate the missing data values in patient medical records by applying statistical methods and artificial intelligence, which will significantly shorten the automation of large datasets processing and facilitate their descriptive and exploratory analysis during further data discovery to identify certain patterns and features.

Investigating the missing data effect on credit scoring rule based models: The case of an Iranian bank

Credit risk management is a process in which banks estimate probability of default (PD) for each loan applicant. Data sets of previous loan applicants are built by gathering their data, and these internal data sets are usually completed using external credit bureau’s data and finally used for estimating PD in banks. There is also a continuous interest for bank to use rule based classifiers to build their default prediction models. However, in practice the data records are usually incomplete and have some missing values and this make problems for banks, especially in credit risk portfolios which are low default and makes model rule based building complex. Several strategies could be used in order to handle the missing data issue. This paper used five missing value handling strategies including; ignoring, replacing with random, mean, C&R tree induced values and elimination strategies in a real credit scoring dataset. Experimental results show that ignoring strategy consistently outperforms other methods on test data set, and suggest that the CHAID is a useful classifier for handling low default portfolios with missing value.

A novel predictive algorithm for double difference observations of obstructed BeiDou geostationary earth orbit (GEO) satellites

Transmission link disturbances and device failure cause global navigation satellite system (GNSS) receivers to miss observations, leading to poor accuracy in real-time kinematic (RTK) positioning. Previously described solutions for this problem are influenced by the length of the prediction period, or are unable to account for changes in receiver state because they use information from previous epochs to make predictions. We propose an algorithm for predicting double difference (DD) observations of obstructed BeiDou navigation system (BDS) GEO satellites. Our approach adopts the first-degree polynomial function for predicting missing observations. We introduce a Douglas-Peucker algorithm to judge the state of the rover receiver to reduce the impact of predictive biases. Static and kinematic experiments were carried out on BDS observations to evaluate the proposed algorithm. The results of our navigation experiment demonstrate that RTK positioning accuracy is improved from meter to decimeter level with fixed ambiguity (horizontal < 2 cm, vertical < 18 cm). Horizontal accuracy is improved by over 50%, and the vertical accuracies of the results of the static and kinematic experiments are increased by 47% and 27% respectively, compared with the results produced by the classical approach. Though as the baseline becomes longer, the accuracy is weakened, our predictive algorithm is an improvement over existing approaches to overcome the issue of missing data.

Infant Brain Development Prediction With Latent Partial Multi-View Representation Learning.

The early postnatal period witnesses rapid and dynamic brain development. However, the relationship between brain anatomical structure and cognitive ability is still unknown. Currently, there is no explicit model to characterize this relationship in the literature. In this paper, we explore this relationship by investigating the mapping between morphological features of the cerebral cortex and cognitive scores. To this end, we introduce a multi-view multi-task learning approach to intuitively explore complementary information from different time-points and handle the missing data issue in longitudinal studies simultaneously. Accordingly, we establish a novel model, latent partial multi-view representation learning. Our approach regards data from different time-points as different views and constructs a latent representation to capture the complementary information from incomplete time-points. The latent representation explores the complementarity across different time-points and improves the accuracy of prediction. The minimization problem is solved by the alternating direction method of multipliers. Experimental results on both synthetic and real data validate the effectiveness of our proposed algorithm.

High-Dimensional Probabilistic Fingerprinting in Wireless Sensor Networks Based on a Multivariate Gaussian Mixture Model.

The extensive deployment of wireless infrastructure provides a low-cost way to track mobile users in indoor environment. This paper demonstrates a prototype model of an accurate and reliable room location awareness system in a real public environment in which three typical problems arise. Firstly, a massive number of access points (APs) can be sensed leading to a high-dimensional classification problem. Secondly, heterogeneous devices record different received signal strength (RSS) levels because of the variations in chip-set and antenna attenuation. Thirdly, APs are not necessarily visible in every scanning cycle leading to missing data issue. This paper presents a probabilistic Wi-Fi fingerprinting method in a hidden Markov model (HMM) framework for mobile user tracking. To account for spatial correlation of the signal strengths from multiple APs, a Multivariate Gaussian Mixture Model (MVGMM) was fitted to model the probability distribution of RSS measurements in each cell. Furthermore, the unseen property of invisible AP was investigated in this research, and demonstrated the efficiency as a beneficial information to differentiate between cells. The proposed system is able to achieve comparable localisation performance. Filed test results achieve a reliable 97% localisation room level accuracy of multiple mobile users in a real university campus Wi-Fi network.

Considerations of multiple imputation approaches for handling missing data in clinical trials

Missing data exist in all clinical trials and missing data issue is a very serious issue in terms of the interpretability of the trial results. There is no universally applicable solution for all missing data problems. Methods used for handling missing data issue depend on the circumstances particularly the assumptions on missing data mechanisms. In recent years, if the missing at random mechanism cannot be assumed, conservative approaches such as the control-based and returning to baseline multiple imputation approaches are applied for dealing with the missing data issues. In this paper, we focus on the variability in data analysis of these approaches. As demonstrated by examples, the choice of the variability can impact the conclusion of the analysis. Besides the methods for continuous endpoints, we also discuss methods for binary and time to event endpoints as well as consideration for non-inferiority assessment.

A Kriging based spatiotemporal approach for traffic volume data imputation.

Along with the rapid development of Intelligent Transportation Systems, traffic data collection technologies have progressed fast. The emergence of innovative data collection technologies such as remote traffic microwave sensor, Bluetooth sensor, GPS-based floating car method, and automated license plate recognition, has significantly increased the variety and volume of traffic data. Despite the development of these technologies, the missing data issue is still a problem that poses great challenge for data based applications such as traffic forecasting, real-time incident detection, dynamic route guidance, and massive evacuation optimization. A thorough literature review suggests most current imputation models either focus on the temporal nature of the traffic data and fail to consider the spatial information of neighboring locations or assume the data follow a certain distribution. These two issues reduce the imputation accuracy and limit the use of the corresponding imputation methods respectively. As a result, this paper presents a Kriging based data imputation approach that is able to fully utilize the spatiotemporal correlation in the traffic data and that does not assume the data follow any distribution. A set of scenarios with different missing rates are used to evaluate the performance of the proposed method. The performance of the proposed method was compared with that of two other widely used methods, historical average and K-nearest neighborhood. Comparison results indicate that the proposed method has the highest imputation accuracy and is more flexible compared to other methods.

Missingness in the Setting of Competing Risks: from missing values to missing potential outcomes.

The setting of competing risks in which there is an event that precludes the event of interest from occurring is prevalent in epidemiological research. Unless studying all-cause mortality, any study following up individuals is subject to having a competing risk should individuals die during time period that the study covers. While there are prior papers discussing the need for competing risk methods in epidemiologic research, we are not aware of any review that discusses issues of missing data in a competing risk setting. We provide an overview of causal inference in competing risks as potential outcomes are missing, provide some strategies in dealing with missing (or misclassified) event type, and missing covariate data in competing risks. The strategies presented are specifically focused on those that may easily be implemented in standard statistical packages. There is ongoing work in terms of causal analyses, dealing with missing event type information, and missing covariate values specific to competing risk analyses. Competing events are common in epidemiologic research. While there has been a focus on why one should conduct a proper competing risk analysis, a perhaps unrecognized issue is in terms of missingness. Strategies exist to minimize the impact of missingness in analyses of competing risks.

Estimating Size of Drug Users in Macau: an Open Population Capture-Recapture Model with Data Augmentation Using Public Registration Data

Successful drug intervention policies rely on timely assessment of size of drug users and close monitoring of trend regarding drug abuse. Taking advantage of data collected by the Central Registration System for Drug Abusers of Macau (CRSDAM), this study applied open population capture-recapture models with data augmentation techniques to estimate the size of drug users and to identify influential factors on capture and survival probabilities from 2009 to 2014. In particular, the data augmentation technique was used to address missing data issues. The estimated size of drug users has been slowly declining from 2442 (95% Bayesian credible interval [BCI] 2042–2914) in 2009 to 807 (95% BCI 654–1000) in 2014 with a small fluctuation of 2251 (95% BIC 1950–2599) in 2011, and the estimated cumulative size of drug users is 6199 (95% BCI 5651–6873), which is correspondent to a prevalence rate of 1.20% of the current population aged 15 to 54. Although the estimated sizes of total, narcotic, and other drug users were declining, the size of stimulant users might be increasing. People who used narcotics as their first drug and who were reported by governmental agencies were more likely to stay in the registration system, while those who used needle injection were less likely to stay. Governmental agencies, higher education, and using needle injection were negatively associated with the probability of capture over time, while using narcotics as the first drug was positively associated with it.