Imperfect Data Research Articles

Trans and gender diverse identities in adolescent health research: making the most of imperfect data

Advancing adolescent health research necessitates deliberate design and analysis that accurately captures the rapidly evolving world in which adolescents live and the ways in which they understand and express themselves...

Computational reaching of quantified consequences from imperfect initial data

Usually, it is not direct to translate theoretical results to practice. One of the challenges is the real computation of solutions to proposed problems, when infinite numbers (such as real numbers or the unit interval) are considered. The same is true when consequences (information) from a data set modeled by logical rules need to be obtained. In this framework, this information is obtained from the immediate consequences operator and ensuring its continuity is fundamental for making sure the computation of this information in a countable number of iterations. Recently, a versatile operator using generalized quantifiers has been introduced. This note will advance in the knowledge introducing necessary and sufficient conditions to ensure the continuity of this new flexible operator.

Deep-Learning-Based Automated Building Construction Progress Monitoring for Prefabricated Prefinished Volumetric Construction.

Prefabricated prefinished volumetric construction (PPVC) is a relatively new technique that has recently gained popularity for its ability to improve flexibility in scheduling and resource management. Given the modular nature of PPVC assembly and the large amounts of visual data amassed throughout a construction project today, PPVC building construction progress monitoring can be conducted by quantifying assembled PPVC modules within images or videos. As manually processing high volumes of visual data can be extremely time consuming and tedious, building construction progress monitoring can be automated to be more efficient and reliable. However, the complex nature of construction sites and the presence of nearby infrastructure could occlude or distort visual data. Furthermore, imaging constraints can also result in incomplete visual data. Therefore, it is hard to apply existing purely data-driven object detectors to automate building progress monitoring at construction sites. In this paper, we propose a novel 2D window-based automated visual building construction progress monitoring (WAVBCPM) system to overcome these issues by mimicking human decision making during manual progress monitoring with a primary focus on PPVC building construction. WAVBCPM is segregated into three modules. A detection module first conducts detection of windows on the target building. This is achieved by detecting windows within the input image at two scales by using YOLOv5 as a backbone network for object detection before using a window detection filtering process to omit irrelevant detections from the surrounding areas. Next, a rectification module is developed to account for missing windows in the mid-section and near-ground regions of the constructed building that may be caused by occlusion and poor detection. Lastly, a progress estimation module checks the processed detections for missing or excess information before performing building construction progress estimation. The proposed method is tested on images from actual construction sites, and the experimental results demonstrate that WAVBCPM effectively addresses real-world challenges. By mimicking human inference, it overcomes imperfections in visual data, achieving higher accuracy in progress monitoring compared to purely data-driven object detectors.

Attempt to Explore Ozone Mixing Ratio Data from Reanalyses for Trend Studies

In this paper, we use ozone mixing ratio data from the MERRA-2, ERA-5 and JRA-55 reanalyses from 500 hPa to 1 hPa in the period 1980–2020 with the aim of assessing their suitability for trend analysis. We found that these data are not suitable for trend studies due to huge differences in trend values and large differences in the variance of the ozone mixing ratio between reanalyses, and due to strong discrepancies between the ozone mixing ratio from reanalyses and that from the reliable ozonesonde at Hohenpeissenberg. These large differences can be caused by satellite replacement or by the assimilation of imperfect homogeneous data.

Enhanced robustness in machine learning: Application of an adaptive robust loss function

In the field of machine learning, the selection of a loss function plays a pivotal role in determining the training dynamics and generalization capability of models. Traditional loss functions such as mean square error (MSE) and cross-entropy are often not equipped to handle noisy and anomalous data effectively, which can result in models that perform poorly in practical scenarios. To address these shortcomings, this study introduces a novel adaptive robust loss function, enhanced by a tunable parameter , which allows for flexibility in adjusting the robustness of the function according to the nature of the data being processed. Our research demonstrates that this new loss function significantly improves the performance of linear regression and multilayer perceptron models, particularly in environments laden with noisy and anomalous data. By adapting the parameter , the function can cater to varying levels of data irregularities, thus enhancing the models accuracy and reliability across diverse and complex data environments. This adaptive mechanism not only offers a substantial theoretical contribution to the understanding of robust loss functions but also provides a practical tool for machine learning practitioners to develop models that are resilient to data imperfections. The implications of this research are profound, suggesting a shift towards more adaptive and robust approaches in machine learning model development.

Deep Reinforcement Learning for personalized diagnostic decision pathways using Electronic Health Records: A comparative study on anemia and Systemic Lupus Erythematosus

Background:Clinical diagnoses are typically made by following a series of steps recommended by guidelines that are authored by colleges of experts. Accordingly, guidelines play a crucial role in rationalizing clinical decisions. However, they suffer from limitations, as they are designed to cover the majority of the population and often fail to account for patients with uncommon conditions. Moreover, their updates are long and expensive, making them unsuitable for emerging diseases and new medical practices. Methods:Inspired by guidelines, we formulate the task of diagnosis as a sequential decision-making problem and study the use of Deep Reinforcement Learning (DRL) algorithms to learn the optimal sequence of actions to perform in order to obtain a correct diagnosis from Electronic Health Records (EHRs), which we name a diagnostic decision pathway. We apply DRL to synthetic yet realistic EHRs and develop two clinical use cases: Anemia diagnosis, where the decision pathways follow a decision tree schema, and Systemic Lupus Erythematosus (SLE) diagnosis, which follows a weighted criteria score. We particularly evaluate the robustness of our approaches to noise and missing data, as these frequently occur in EHRs. Results:In both use cases, even with imperfect data, our best DRL algorithms exhibit competitive performance compared to traditional classifiers, with the added advantage of progressively generating a pathway to the suggested diagnosis, which can both guide and explain the decision-making process. Conclusion:DRL offers the opportunity to learn personalized decision pathways for diagnosis. Our two use cases illustrate the advantages of this approach: they generate step-by-step pathways that are explainable, and their performance is competitive when compared to state-of-the-art methods.

Research on Flexible Data Model of MES System based on Time-series Database

In the process of advancing towards digitization and intelligence in manufacturing, Manufacturing Execution Systems (MES) have gradually become the core of production processes. The design of the data model is crucial as it affects the scalability, maintainability, and stability of the system. This study focuses on addressing issues in MES systems such as high investment costs, imperfect data models, lack of flexibility, and inefficient data flow. It proposes a novel flexible data model based on time-series databases.This paper introduces the definition and characteristics of time-series databases, analyzes the shortcomings of existing MES system data models, and proposes a novel flexible MES data model based on time-series databases. Finally, the practicality and efficiency of the new data model are validated through experiments.

Towards explainable oral cancer recognition: Screening on imperfect images via Informed Deep Learning and Case-Based Reasoning

Oral squamous cell carcinoma recognition presents a challenge due to late diagnosis and costly data acquisition. A cost-efficient, computerized screening system is crucial for early disease detection, minimizing the need for expert intervention and expensive analysis. Besides, transparency is essential to align these systems with critical sector applications. Explainable Artificial Intelligence (XAI) provides techniques for understanding models. However, current XAI is mostly data-driven and focused on addressing developers’ requirements of improving models rather than clinical users’ demands for expressing relevant insights. Among different XAI strategies, we propose a solution composed of Case-Based Reasoning paradigm to provide visual output explanations and Informed Deep Learning (IDL) to integrate medical knowledge within the system. A key aspect of our solution lies in its capability to handle data imperfections, including labeling inaccuracies and artifacts, thanks to an ensemble architecture on top of the deep learning (DL) workflow. We conducted several experimental benchmarks on a dataset collected in collaboration with medical centers. Our findings reveal that employing the IDL approach yields an accuracy of 85%, surpassing the 77% accuracy achieved by DL alone. Furthermore, we measured the human-centered explainability of the two approaches and IDL generates explanations more congruent with the clinical user demands.

Neural networks for quantum state tomography with constrained measurements

Quantum state tomography (QST) aiming at reconstructing the density matrix of a quantum state plays an important role in various emerging quantum technologies. Recognizing the challenges posed by imperfect measurement data, we develop a unified neural network (NN)-based approach for QST under constrained measurement scenarios, including limited measurement copies, incomplete measurements, and noisy measurements. Through comprehensive comparison with other estimation methods, we demonstrate that our method improves the estimation accuracy in scenarios with limited measurement resources, showcasing notable robustness in noisy measurement settings. These findings highlight the capability of NNs to enhance QST with constrained measurements.

Pathways to Enhancing the Quality of Agricultural Statistical Services in the Context of Rural Revitalization

In the strategic context of rural revitalization, optimizing the quality of agricultural statistical services is a crucial element for advancing agricultural modernization and sustainable rural economic development. This paper focuses on the significance of enhancing agricultural statistical service quality under the backdrop of rural revitalization. It addresses current issues such as inadequate implementation of agricultural statistical survey systems, an imperfect data quality control system, and a shortage of statistical service personnel. Proposals are made to improve the statistical survey system, enhance the data quality control framework, and strengthen personnel training. These pathways offer references for elevating the quality of agricultural statistical services and implementing the rural revitalization strategy in the new era.

Predicting Patient No-Shows: Situated Machine Learning with Imperfect Data.

Patients who do not show up for scheduled appointments are a considerable cost and concern in healthcare. In this study, we predict patient no-shows for outpatient surgery at the endoscopy ward of a hospital in Denmark. The predictions are made by training machine leaning (ML) models on available data, which have been recorded for purposes other than ML, and by doing situated work in the hospital setting to understand local data practices and fine-tune the models. The best performing model (XGBoost with oversampling) predicts no-shows at sensitivity = 0.97, specificity = 0.66, and accuracy = 0.95. Importantly, the situated work engaged local hospital staff in the design process and led to substantial quantitative improvements in the performance of the models. We consider the results promising but acknowledge that they are from a single ward. To transfer the no-show models to other wards and hospitals, the situated work must be redone.

Particle filter data assimilation for ubiquitous unstable trajectories of two-dimensional three-state cellular automata

Estimating the states of error-growing (sensitive to initial state) cellular automata (CA) based on noisy imperfect data is challenging due to the discreteness of the dynamical system. This paper proposes particle filter (PF)–based data assimilation (DA) for three-state error-growing CA and demonstrates that the PF-based DA can predict the present and future state even with noisy and sparse observations. The error-growing CA used in the present study comprised a competitive system of land, grass, and sheep. To the best of the authors’ knowledge, this is the first application of DA to such CA. The performance of DA for different observation sets was evaluated in terms of observational error, density, and frequency, and a series of sensitivity tests of the internal parameters in the DA was conducted. The inflation and localization parameters were tuned according to the sensitivity tests.

A robust operators’ cognitive workload recognition method based on denoising masked autoencoder

Identifying the cognitive workload of operators is crucial in complex human-automation collaboration systems. An excessive workload can lead to fatigue or accidents, while an insufficient workload may diminish situational awareness and efficiency. However, existing supervised learning-based methods for workload recognition are ineffective when dealing with imperfect input data, such as missing or noisy data, which is not practical in real applications. This study introduces a robust Electroencephalogram (EEG)-enabled cognitive workload recognition model using self-supervised learning. The proposed method, DMAEEG, combines the training strategies of denoising autoencoders and masked autoencoders, demonstrating strong robustness against noisy and incomplete data. More specifically, we adopt the temporal convolutional network and multi-head self-attention mechanisms as the backbone, effectively capturing both the temporal and spatial features from EEG. Extensive experiments are conducted to verify the effectiveness and robustness of the proposed method on an open dataset and a self-collected dataset. The results indicate that DMAEEG performs superior to other state-of-the-art across various evaluation metrics. Moreover, DMAEEG maintains high accuracy in workload inference even when EEG signals are corrupted with a high masking ratio or strong noises. This signifies its superiority in capturing robust intrinsic patterns from imperfect EEG data. The proposed method significantly contributes to decoding EEG signals for workload recognition in real-world applications, thereby enhancing the safety and reliability of human-automation interactions.

Robust online active learning with cluster-based local drift detection for unbalanced imperfect data

With the rapid development of data-driven technologies, a massive amount of actual data emerges from industrial systems, forming data stream. Their data distribution may change over time and outliers may be generated as unbalanced imperfect data due to time-varying working condition, aging equipment, etc. Previous methods struggle with the dual challenges of concept drift and unbalance, however, fail to efficiently distinguishing outliers from a drift under the limited labeling budget, causing the performance degradation. To address the issue, robust online active learning with cluster-based local drift detection is proposed to classify unbalanced imperfect data stream with the above characteristics. The cluster-based local drift detection is first designed to capture a new concept and recognize the corresponding drifted regions. Following that, an improved active learning mechanism is presented to distinguish outliers from a drift, and select most valuable instances for labeling and updating ensemble classifier. Experimental results for eight synthetic and four real-world data streams show that the proposed method outperforms seven comparative methods on classification accuracy and robustness.

Understanding world models through multi-step pruning policy via reinforcement learning

In model-based reinforcement learning, the conventional approach to addressing world model bias is to use gradient optimization methods. However, using a singular policy from gradient optimization methods in response to a world model bias inevitably results in an inherently biased policy. This is because of constraints on the imperfect and dynamic data of state-action pairs. The gap between the world model and the real environment can never be completely eliminated. This article introduces a novel approach that explores a variety of policies instead of focusing on either world model bias or singular policy bias. Specifically, we introduce the Multi-Step Pruning Policy (MSPP), which aims to reduce redundant actions and compress the action and state spaces. This approach encourages a different perspective within the same world model. To achieve this, we use multiple pruning policies in parallel and integrate their outputs using the cross-entropy method. Additionally, we provide a convergence analysis of the pruning policy theory in tabular form and an updated parameter theoretical framework. In the experimental section, the newly proposed MSPP method demonstrates a comprehensive understanding of the world model and outperforms existing state-of-the-art model-based reinforcement learning baseline techniques.

The Materiality of Data: Practices of Coping with Imperfect Data for Machine Learning Development

The Materiality of Data: Practices of Coping with Imperfect Data for Machine Learning Development

Spatio-temporal ecological models via physics-informed neural networks for studying chronic wasting disease

To mitigate the negative effects of emerging wildlife diseases in biodiversity and public health it is critical to accurately forecast pathogen dissemination while incorporating relevant spatio-temporal covariates. Forecasting spatio-temporal processes can often be improved by incorporating scientific knowledge about the dynamics of the process using physical models. Ecological diffusion equations are often used to model epidemiological processes of wildlife diseases where environmental factors play a role in disease spread. Physics-informed neural networks (PINNs) are deep learning algorithms that constrain neural network predictions based on physical laws and therefore are powerful forecasting models useful even in cases of limited and imperfect training data. In this paper, we develop a novel ecological modeling tool using PINNs, which fits a feedforward neural network and simultaneously performs parameter identification in a partial differential equation (PDE) with varying coefficients. We demonstrate the applicability of our model by comparing it with the commonly used Bayesian stochastic partial differential equation method and traditional machine learning approaches, showing that our proposed model exhibits superior prediction and forecasting performance when modeling chronic wasting disease in deer in Wisconsin. Furthermore, our model provides the opportunity to obtain scientific insights into spatio-temporal covariates affecting spread and growth of diseases. This work contributes to future machine learning and statistical methodology development by studying spatio-temporal processes enhanced by prior physical knowledge.

A Heat Method for Generalized Signed Distance

We introduce a method for approximating the signed distance function (SDF) of geometry corrupted by holes, noise, or self-intersections. The method implicitly defines a completed version of the shape, rather than explicitly repairing the given input. Our starting point is a modified version of the heat method for geodesic distance, which diffuses normal vectors rather than a scalar distribution. This formulation provides robustness akin to generalized winding numbers (GWN) , but provides distance function rather than just an inside/outside classification. Our formulation also offers several features not common to classic distance algorithms, such as the ability to simultaneously fit multiple level sets, a notion of distance for geometry that does not topologically bound any region, and the ability to mix and match signed and unsigned distance. The method can be applied in any dimension and to any spatial discretization, including triangle meshes, tet meshes, point clouds, polygonal meshes, voxelized surfaces, and regular grids. We evaluate the method on several challenging examples, implementing normal offsets and other morphological operations directly on imperfect curve and surface data. In many cases we also obtain an inside/outside classification dramatically more robust than the one obtained provided by GWN.

PCDMD: Physics-constrained dynamic mode decomposition for accurate and robust forecasting of dynamical systems with imperfect data and physics

The dynamic mode decomposition (DMD) method has attracted widespread attention as a representative modal-decomposition method and can build a predictive model. However, DMD may give predicted results that deviate from physical reality in some scenarios, such as dealing with translation problems or noisy data. Here, we propose a physics-constrained DMD (PCDMD) method to address this issue. The proposed PCDMD method first employs a data-driven model using DMD, then calculates the residual of the physical equations, and finally corrects the predicted results using Kalman filter and gain coefficients. In this way, the PCDMD method can integrate the physics-informed equations with the data-driven model generated by DMD. Numerical experiments are conducted using PCDMD, including the Allen–Cahn, advection-diffusion, Burgers' equations and lid-driven cavity flow. The results demonstrate that the proposed PCDMD method can reduce the reconstruction and prediction errors by 1%-10% by incorporating physical constraints. Regarding noisy datasets and imperfect physical constraints, PCDMD can still ensure that the predicted results satisfy the physical constraints, thereby reducing errors. Program summaryProgram title: PCDMDDataset link:https://github.com/YinYuhuiTJU/PCDMDLicensing provisions: MITProgramming language: Python

How to Be Certain: Using Known Relations and Trust Discount to Determine Confidence About the Degree of Uncertainty

Ensuring certainty in the rapidly evolving digital twin environments amidst inherent uncertainty is paramount in decision-making for critical engineering systems. This paper presents a novel approach for quantifying and tracking confidence levels within a network of sensors and structural models, specifically addressing uncertainty arising from faulty sensors. By leveraging known relations and implementing a trust discount mechanism, our methodology offers a fundamental framework for navigating in the presence of uncertainties. A quasi-real-time case study on a cantilever beam model is simulated to demonstrate the efficacy of the proposed approach. We showcase the ability of our method to accurately assess and adapt to varying levels of uncertainty introduced by faulty sensors. Our findings highlight the importance of incorporating trust dynamics and established relationships within digital twin environments to achieve improved certainty despite imperfect data. This research contributes to the theoretical underpinnings of digital twin technology and offers practical insights for its application across diverse domains.