Missing Data Samples Research Articles

The Impact of Missing Continuous Blood Glucose Samples on Machine Learning Models for Predicting Postprandial Hypoglycemia: An Experimental Analysis

This study investigates how missing data samples in continuous blood glucose data affect the prediction of postprandial hypoglycemia, which is crucial for diabetes management. We analyzed the impact of missing samples at different times before meals using two datasets: virtual patient data and real patient data. The study uses six commonly used machine learning models under varying conditions of missing samples, including custom and random patterns reflective of device failures and arbitrary data loss, with different levels of data removal before mealtimes. Additionally, the study explored different interpolation techniques to counter the effects of missing data samples. The research shows that missing samples generally reduce the model performance, but random forest is more robust to missing samples. The study concludes that the adverse effects of missing samples can be mitigated by leveraging complementary and informative non-point features. Consequently, our research highlights the importance of strategically handling missing data, selecting appropriate machine learning models, and considering feature types to enhance the performance of postprandial hypoglycemia predictions, thereby improving diabetes management.

Recovery of missing samples in Orthogonal Frequency Division Multiplexing signals with optimisation using data carriers

AbstractA method is proposed for reconstructing an Orthogonal Frequency Division Multiplexing (OFDM) signal that contains data gaps, with the aim to improve demodulation. The main objective is to use the method in a passive radar application with missing data samples and to improve target detection. The OFDM signal is assumed to comply with the Digital Video Broadcasting Terrestrial standard. The proposed recovery method is based on optimisation of a novel objective function, which consists of two parts. The first part is a function of the energy in the out‐of‐band frequencies, whereas the second, and novel part, uses the location of data carriers in the constellation diagram. The method is evaluated using both simulations and real data. The authors show that the proposed method significantly improves the OFDM signal in just a few iteration steps. The proposed method improved the condition number more than a factor ten thousand millions compared to using the least square method on the out‐of‐band frequencies only. The authors also decode the symbols with the Viterbi decoding algorithm and show how the required number of iterations with the proposed algorithm depends on the amount of missing samples and on the Signal‐to‐Noise Ratio in order to achieve a Bit Error Rate of less than one in one hundred thousand millions.

The Effectiveness of the StaySafe Intervention Using a Paradigm for Predicting Missing Outcome Data.

Sample attrition is a confounding issue in the analysis of data collected in follow-up studies. The present study uses a regression procedure that includes a propensity score as a predictor in estimating imputed data. The utility of the procedure was addressed by comparing results from this augmented data with those from the original data. Data were from a randomized controlled study testing the utility of a tablet-based intervention designed to improve decision-making with respect to health risk behaviors. Outcomes included self-reported testing for HIV, STD, and hepatitis. Two samples were used (163 in community facilities and 348 in residential facilities). Seventy-eight in the community sample and 238 in the residential sample completed follow-up surveys. Propensity scores based on a stepwise logistic regression were used to make the calibration sample and the missing data sample as close as possible. Multilevel analysis was performed for each outcome and multiple imputation compared estimated mean differences for the augmented and original analyses. The model imputing missing data was effective for the three outcomes and increased power. Least square mean differences between augmented and original data appeared to be essentially the same for most of the outcomes. This protocol has been registered with https://www.clinicaltrials.gov/(NCT02777086).

Investigating on the robustness of flow-based intrusion detection system against adversarial samples using Generative Adversarial Networks

Recently, Software Defined Networking (SDN) has emerged as the key technology in programming and orchestrating security policy in the security operations centers (SOCs) for heterogeneous networks. Typically, machine learning-based intrusion detection systems (ML-IDS) have been deployed and associated with SDN to leverage the features of a programmable network to defend against sophisticated cyberattacks in anomaly detection. Unfortunately, such ML-based IDSs are easily vulnerable to adversarial attacks due to the lack of diverse forms of malicious records in the training dataset. The missing data sample in the training phase can lead to a lower detection rate in real-world scenarios with adversarial settings. In this paper, we explore the ability of Wasserstein Generative Adversarial Networks with Gradient Penalty (WGAN-GP), WGAN-GP with two timescale update rule (WGAN-GP TTUR), and AdvGAN in generating perturbed attack samples to bypass attack detectors. Then, this approach is used to continuously evaluate the robustness of ML-based IDSs and then upgrade them as a service in SDN. The experimental results on CICIDS2018 and InSDN datasets demonstrate that generated adversarial samples can be used to fool targeted IDS. Later, those created samples can supplement the original ones in retraining IDS to improve the resilience of the attack detector.

Frequency Response Function Identification from Incomplete Data: A Wavelet-based Approach

Frequency Response Function (FRF) identification plays a crucial role in the design, the control, and the analysis of complex dynamical systems, including thermal and motion systems. Especially for applications that require long measurements, missing data samples, e.g., due to interruptions in the data transmission or sensor failure, often occur. The aim of this paper is to accurately identify nonparametric FRF models of periodically excited systems from noisy output measurements with missing samples. The presented method employs a wavelet-based transformation to address the identification problem in the time-frequency plane. A simulation example confirms that the developed techniques produce accurate estimates, even when many samples are missing.

Bootstrap joint prediction regions for sequences of missing values in spatio-temporal datasets

Missing data reconstruction is a critical step in the analysis and mining of spatio-temporal data. However, few studies comprehensively consider missing data patterns, sample selection and spatio-temporal relationships. To take into account the uncertainty in the point forecast, some prediction intervals may be of interest. In particular, for (possibly long) missing sequences of consecutive time points, joint prediction regions are desirable. In this paper we propose a bootstrap resampling scheme to construct joint prediction regions that approximately contain missing paths of a time components in a spatio-temporal framework, with global probability 1-alpha . In many applications, considering the coverage of the whole missing sample-path might appear too restrictive. To perceive more informative inference, we also derive smaller joint prediction regions that only contain all elements of missing paths up to a small number k of them with probability 1-alpha . A simulation experiment is performed to validate the empirical performance of the proposed joint bootstrap prediction and to compare it with some alternative procedures based on a simple nominal coverage correction, loosely inspired by the Bonferroni approach, which are expected to work well standard scenarios.

Impact of Female Executive on Investment Efficiency of Listed Companies in Shanghai Stock Market: A Case of Over-Investment

The purpose of this study is to analyze the relationships between the presence of female in top management teams and over-investment behavior of listed companies. This study draws on Richardson’s (2006) model and selects A-share listed companies from the Shanghai Stock Exchange from 2017 to 2018 as a research sample. After eliminating the financial listed companies and missing data samples, 725 sample companies were used for statistical analysis. The result shows that 1) the existence of female top management can effectively inhibit the occurrence of over-investment behavior of the company. 2) The higher the proportion of female senior executives, the less likely the listed company’s over-investment behavior will occur. The research in this paper will provide new ideas for the effectiveness of China’s listed companies to strengthen the internal governance mechanism, provide a broader direction for the study of investment theory, and also promote the transformation of the company’s human capital management practices.

Low-Rank Hankel Matrix Completion for Robust Time-Frequency Analysis

In this paper, we develop a novel method to enable robust sparsity-based time-frequency representation of multi-component frequency modulated signals in the presence of burst missing samples, where the amplitudes of the different signal components are generally different. Unlike existing methods which require cross-term presence to be sparse in the time-frequency domain, the proposed method permits effective time-frequency representation reconstruction even when undesired cross-terms take a high occupancy. A key enabling procedure is the high-fidelity missing entry recovery of the instantaneous autocorrelation function that is insensitive to cross-terms. By designing instantaneous autocorrelation function patches such that their Doppler-frequency domain representation is sparse, we formulate the instantaneous autocorrelation function recovery problem as a patch-based low-rank block Hankel matrix completion problem. This approach effectively suppresses the effects of burst missing data samples and is robust to the amplitude differences. A data-adaptive time-frequency kernel is then applied to further mitigate the undesired cross-terms and the residual artifacts due to the burst missing samples. We prove the superiority of the proposed method over the state of the art for both multi-component linear and nonlinear frequency modulated signals. Simulation results confirm that the proposed method outperforms the state of the art for different types of frequency modulated signals with varying signal-to-noise ratios and missing sample rates.

An adaptive algorithm for fast and reliable online saccade detection.

To investigate visual perception around the time of eye movements, vision scientists manipulate stimuli contingent upon the onset of a saccade. For these experimental paradigms, timing is especially crucial, because saccade offset imposes a deadline on the display change. Although efficient online saccade detection can greatly improve timing, most algorithms rely on spatial-boundary techniques or absolute-velocity thresholds, which both suffer from weaknesses: late detections and false alarms, respectively. We propose an adaptive, velocity-based algorithm for online saccade detection that surpasses both standard techniques in speed and accuracy and allows the user to freely define the detection criteria. Inspired by the Engbert-Kliegl algorithm for microsaccade detection, our algorithm computes two-dimensional velocity thresholds from variance in the preceding fixation samples, while compensating for noisy or missing data samples. An optional direction criterion limits detection to the instructed saccade direction, further increasing robustness. We validated the algorithm by simulating its performance on a large saccade dataset and found that high detection accuracy (false-alarm rates of < 1%) could be achieved with detection latencies of only 3 ms. High accuracy was maintained even under simulated high-noise conditions. To demonstrate that purely intrasaccadic presentations are technically feasible, we devised an experimental test in which a Gabor patch drifted at saccadic peak velocities. Whereas this stimulus was invisible when presented during fixation, observers reliably detected it during saccades. Photodiode measurements verified that-including all system delays-the stimuli were physically displayed on average 20 ms after saccade onset. Thus, the proposed algorithm provides a valuable tool for gaze-contingent paradigms.

Missing observation approximation for spatio-temporal profile reconstruction in participatory sensor networks

Purpose Participatory wireless sensor networks (PWSN) is an emerging paradigm that leverages existing sensing and communication infrastructures for the sensing task. Various environmental phenomenon – P monitoring applications dealing with noise pollution, road traffic, requiring spatio-temporal data samples of P (to capture its variations and its profile construction) in the region of interest – can be enabled using PWSN. Because of irregular distribution and uncontrollable mobility of people (with mobile phones), and their willingness to participate, complete spatio-temporal (CST) coverage of P may not be ensured. Therefore, unobserved data values must be estimated for CST profile construction of P and presented in this paper. Design/methodology/approach In this paper, the estimation of these missing data samples both in spatial and temporal dimension is being discussed, and the paper shows that non-parametric technique – Kernel Regression – provides better estimation compared to parametric regression techniques in PWSN context for spatial estimation. Furthermore, the preliminary results for estimation in temporal dimension have been provided. The deterministic and stochastic approaches toward estimation in the context of PWSN have also been discussed. Findings For the task of spatial profile reconstruction, it is shown that non-parametric estimation technique (kernel regression) gives a better estimation of the unobserved data points. In case of temporal estimation, few preliminary techniques have been studied and have shown that further investigations are required to find out best estimation technique(s) which may approximate the missing observations (temporally) with considerably less error. Originality/value This study addresses the environmental informatics issues related to deterministic and stochastic approaches using PWSN.

Sparsity-based time-frequency representation of FM signals with burst missing samples

Abstract In this paper, we present an effective time-frequency (TF) analysis of non-stationary frequency modulated (FM) signals in the presence of burst missing data samples. The key concept of the proposed work lies in the reliable sparse recovery of non-parametric FM signals in the joint-variable domains. Specifically, by utilizing the one-dimensional Fourier relationship between the instantaneous auto-correlation function (IAF) and the TF representation (TFR), the proposed approach iteratively recovers missing samples in the IAF domain through sparse reconstruction using, e.g., the orthogonal matching pursuit (OMP) method, while maintaining the TF-domain sparsity. The proposed method, referred to as missing data iterative sparse reconstruction (MI-SR), achieves reliable TFR recovery from the observed data with a high proportion of burst missing samples. This is in contrast to the existing sparse TFR recovery methods which work well only for random missing data samples. In particular, when applied in conjunction with signal-adaptive TF kernels, the proposed method achieves effective suppression of both cross-terms and artifacts due to burst missing samples. The superiority of the proposed technique is verified through analytical results and numerical examples.

An Effective Crowdsourcing Data Reporting Scheme to Compose Cloud-Based Services in Mobile Robotic Systems

The smart device combined with artificial intelligence can act as robot system to perform data collection task. To minimize the data collection cost and to guarantee the quality of service (QoS) of tasks are two vital issue in such mobile robot system. Data collection platform and data reporter often needs to negotiate with each other before start of data collection which will generate a certain cost. Once the platform and the data reporter agree to the cooperation, data reporter will collect and report data for a period. However, in previous researches, it was often considered that data reporters can report data at any time without considering the cost of interaction and negotiation, which is not suitable for the practice. In this paper, we propose an efficiency cost data collection scheme (ECDCS) in which the data reporter is selected according to the contribution that all the data it collects have on the whole system rather than a single data samples. Because there exists correlation in data, matrix completion technology can be adopted to recover the missing data samples with partial data while guarantee the QoS of the task. So, a data reporter selection scheme ECDCS based on the matrix completion technology is proposed in which the selection is in terms of the cooperation effect of the reporters rather than a single data sample. The main goal is to select the reporter set with low cost and high QoS which has the best cooperative effect. By doing so, in the proposed data collection scheme, the missing of partial data can be tolerated which can reduce data collection cost while guarantee the QoS. The extensive experiments results indicate that the proposed scheme can effectively reduce the data cost while maintain the QoS of application.

Replacing Missing Data Between Airborne SAR Coherent Image Pairs

For synthetic aperture radar systems, missing data samples can cause severe image distortion. When multiple, coherent data collections exist and the missing data samples do not overlap between collections, there exists the possibility of replacing data samples between collections. For airborne radar, the known and unknown motions of the aircraft prevent direct data sample replacement to repair image features. This paper presents a method to calculate the necessary phase corrections to enable data sample replacement using only the collected radar data.

Sparsity-Aware Adaptive Directional Time–Frequency Distribution for Source Localization

Multi-component characteristics and missing data samples introduce artifacts and cross-terms in quadratic time–frequency distributions, thus affecting their readability. In this study, we propose a new time–frequency method that employs directional smoothing and compressive sensing to reduce cross-terms and mitigate artifacts associated with missing samples. The efficacy of the proposed time–frequency distribution for solving real-life problems is illustrated by employing it to estimate direction of arrival of sparsely sampled sources in under-determined scenario. Numerical results show that the proposed method is superior to other state-of-the-art methods both in terms of obtaining clear time–frequency representation and accurately estimating direction of arrival.

A Two-Step Method for Missing Spatio-Temporal Data Reconstruction

Missing data reconstruction is a critical step in the analysis and mining of spatio-temporal data; however, few studies comprehensively consider missing data patterns, sample selection and spatio-temporal relationships. As a result, traditional methods often fail to obtain satisfactory accuracy or address high levels of complexity. To combat these problems, this study developed an effective two-step method for spatio-temporal missing data reconstruction (ST-2SMR). This approach includes a coarse-grained interpolation method for considering missing patterns, which can successfully eliminate the influence of continuous missing data on the overall results. Based on the results of coarse-grained interpolation, a dynamic sliding window selection algorithm was implemented to determine the most relevant sample data for fine-grained interpolation, considering both spatial and temporal heterogeneity. Finally, spatio-temporal interpolation results were integrated by using a neural network model. We validated our approach using Beijing air quality data and found that the proposed method outperforms existing solutions in term of estimation accuracy and reconstruction rate.

Mitigating Effects of Missing Data for SAR Coherent Images

Missing samples within synthetic aperture radar data result in image distortions. For coherent data products, such as coherent change detection and interferometric processing, the image distortion can be devastating to these second-order products, resulting in missed detections, and inaccurate height maps. Previous approaches to repair the coherent data products focus upon reconstructing the missing data samples. This paper demonstrates that reconstruction is not necessary to restore the quality of the coherent data products.

Online Method to Impute Missing Loop Detector Data for Urban Freeway Traffic Control

The dual loop detector provides input data for real-time traffic control. However, the common scenario of missing or invalid loop data samples negatively impacts the performance of the subsequent traffic control or traffic prediction model. This paper describes a method that detects blank data samples when a new data record is listed, then imputes missing data samples to form a complete data record in real time. One three-lane loop detector station on Whitemud Drive in Edmonton, Alberta, Canada, is used as a case study to verify the algorithms. The imputation scope for this study involves diagnosing and filling in the missing data of one lane for at least one whole day at a specific loop station. The diagnosis of missing data for both volume and speed measurements is based on data records from upstream and downstream stations, rather than from single samples. The imputation algorithm for volume and density data models the relationship of one loop detector with all other loop detectors at the same station as linear. The multiple linear regression algorithm is applied to full historic data offline to learn the linear relationship between loops at the same station, and the online data imputation is based on the equations learned. Two other commonly used imputation techniques—pairwise linear regression and the average of the surrounding detectors—are also conducted for comparison. The proposed diagnosis and imputation method enables the loop data input to support control models with high frequency and large data requirements.

Process Monitoring Based on Recursive Probabilistic PCA for Multi-mode Process

A recursive probabilistic principal component analysis (PPCA) based data-driven fault identification method is proposed to handle the missing data samples and the mode transition in multi-mode process. This model is recursively obtained by using the increasing number of normal observations with partly missing data. First, based on the singular value of historic data matrix, the whole process is divided into different steady modes and mode transitions. For steady modes, the conventional PPCA is used to obtain the principal components, and to impute the missing data. When the mode is a mode transition, the proposed recursive PPCA is applied, which can actually reveal the between-mode dynamics for process monitoring and fault detection. After that, in order to identify the faults, a contribution analysis method is developed and used to identify the variables which make the major contributions to the occurrence of faults. The effectiveness of the proposed approach is demonstrated by the Tennessee Eastman chemical process. The results show that the presented approach can accurately detect abnormal events, identify the faults, and it is also robust to mode transitions.

Fault detection and diagnosis for missing data systems with a three time-slice dynamic Bayesian network approach

A multi-time-slice dynamic Bayesian network with a mixture of the Gaussian output (MT-DBNMG) based data-driven fault identification method is proposed to handle the missing data samples and the non-Gaussian process data. First, via introducing more time slices, a new dynamic Bayesian network structure with multi-time-slice is constructed which can describe the dependence between the current state and historic states. Second, a parameter learning strategy based on expectation maximization algorithm is deduced, from the complete historical data with the non-Gaussianity, to train the parameters of MT-DBNMG. Subsequently, for the missing measurements, an online non-imputation inference method for MT-DBNMG is proposed to conduct fault detection and identification. The effectiveness of the proposed approach is demonstrated by the continuous stirred tank reactor system and the Tennessee Eastman chemical process. The results show that the presented approach can accurately detect abnormal events, identify the fault, and is also robust to unknown noise.

Missing samples analysis in signals for applications to L-estimation and compressive sensing

This paper provides statistical analysis for efficient detection of signal components when missing data samples are present. This analysis is important for both the areas of L-statistics and compressive sensing. In both cases, few samples are available due to either noisy sample elimination or random undersampling signal strategies. The analysis enables the determination of the sufficient number of observation and as such the minimum number of missing samples which still allow proper signal detection. Both single component and multicomponent signals are considered. The results are verified by computer simulations using different component frequencies and under various missing-available samples scenarios.