Drift Monitoring Research Articles

Efficient and scalable covariate drift detection in machine learning systems with serverless computing

As machine learning models are increasingly deployed in production, robust monitoring and detection of concept and covariate drift become critical. This paper addresses the gap in the widespread adoption of drift detection techniques by proposing a serverless-based approach for batch covariate drift detection in ML systems. Leveraging the open-source OSCAR framework and the open-source Frouros drift detection library, we develop a set of services that enable parallel execution of two key components: the ML inference pipeline and the batch covariate drift detection pipeline. To this end, our proposal takes advantage of the elasticity and efficiency of serverless computing for ML pipelines, including scalability, cost-effectiveness, and seamless integration with existing infrastructure. We evaluate this approach through an edge ML use case, showcasing its operation on a simulated batch covariate drift scenario. Our research highlights the importance of integrating drift detection as a fundamental requirement in developing robust and trustworthy AI systems and encourages the adoption of these techniques in ML deployment pipelines. In this way, organizations can proactively identify and mitigate the adverse effects of covariate drift while capitalizing on the benefits offered by serverless computing.

System for continuous metabolic monitoring of mechanically ventilated patients.

In clinical settings, due largely to the cost, size and calibration complexity of existing indirect calorimetry systems, there is seldom instrumentation available to provide reliable, continuous tracking of a mechanically ventilated patient's metabolic output in support of proper nutrition. The atypical metabolisms associated with critically ill patients are difficult to predict and both underfeeding and overfeeding lead to negative impacts on both mortality and the recovery and healing processes. With these issues in mind, a novel ventilator-agnostic indirect calorimetry sensor design, prototype development, and validation were undertaken with the goal of enabling 24/7 metabolic monitoring of mechanically ventilated patients by means of a passive, rate-proportional side-stream sampling scheme and miniature mixing chamber. The miniature mixing chamber enables the use of small, low-cost gas concentration and flow sensing components to ensure the affordability of commercial design-for-manufacture implementations of the prototype sensor. In addition to continuous measurement of patient metabolism, the prototype sensor also enables autonomous monitoring and detection of calibration drift in the gas measurement sensors without disrupting the patient ventilation.

Feature-based analyses of concept drift

Feature selection is one of the most relevant preprocessing and analysis techniques in machine learning. It can dramatically increase the performance of learning algorithms and at the same time provide relevant information on the data. In the scenario of online and stream learning, concept drift, i.e., changes in the underlying distribution over time, can cause significant problems for learning models and data analysis. While there do exist feature selection methods for online learning, none of the methods targets feature selection for drift detection, i.e., the challenge to increase the performance of drift detectors by analyzing the drift rather than increasing model accuracy. However, this challenge is particularly relevant for common unsupervised scenarios. In this work, we study feature selection for drift detection and drift monitoring. We develop a formal definition for a feature-wise notion of drift that allows semantic interpretation. Besides, we derive an efficient algorithm by reducing the problem to classical feature selection and analyze the applicability of our approach to feature selection for drift detection on a theoretical level. Finally, we empirically show the relevance of our considerations on several benchmarks.

1D interferometric Rayleigh scattering velocimetry and thermometry using VIPA

The work introduces a VIPA-based interferometric Rayleigh scattering instrument for tracer-free, simultaneous temperature and velocity measurements along a 1D volume. A virtually imaged phased array (VIPA) replaces the Fabry-Perot etalon conventionally used in interferometric Rayleigh scattering, allowing the extension of the technique from 0D (point or multi-point) to 1D. The Rayleigh-Brillouin spectrum is a function of pressure and temperature and can be used for temperature diagnostics in isobaric flows. A reference leg based on a Fabry-Perot (FP) etalon provides real-time monitoring of the laser wavelength drift during the experiment. The accuracy and precision of the measurements are estimated from measurements in laminar flows, and the technique is then demonstrated in a heated turbulent jet of air.

Artificial Intelligence and Educational Measurement: Opportunities and Threats

I review opportunities and threats that widely accessible Artificial Intelligence (AI)-powered services present for educational statistics and measurement. Algorithmic and computational advances continue to improve approaches to item generation, scale maintenance, test security, test scoring, and score reporting. Predictable misuses of AI for these purposes will result in biased scores, construct underrepresentation, and differential impact over time. Recent efforts to develop standards for AI use in testing like those of Burstein are promising. I argue that similar efforts to develop AI standards for educational measurement will benefit from increased attention to the context of test use and explicit commitment to ongoing monitoring of bias and scale drift over time.

On-farm NIR sensor for milk analysis: Exploitation of bias monitoring and bias correction

Long-term studies have shown a bias drift over time in the prediction performance of near-infrared spectroscopy measurement systems. This bias drift generally requires extra laboratory reference measurements to detect and correct for this bias. Since these reference measurements are expensive and time consuming, there is a need for advanced methodologies for bias drift monitoring and correction without the need for taking extra samples. In this study, we propose and validate a method to monitor the bias drift and two methods to tackle it. The first method requires no extra measurements and uses a modified version of Partial Least Squares Regression to estimate and correct the bias. This method is based on the assumption that the mean concentration of the predicted component remains constant over time. The second method uses regular bulk milk measurements as a reference for bias correction. This method compares the measured concentrations of the bulk milk to the volume-weighted average concentrations of individual milk samples predicted by the sensor. Any difference between the actual and calculated bulk milk composition is then used to perform a bias correction on the predictions by the sensor system. The effectiveness of these methods to improve the component prediction was evaluated on data originating from a custom-built sensor that automatically measures the NIR reflectance and transmittance spectra of raw milk on the farm. We evaluate the practical use case where models for predicting the milk composition are trained upon installation of the sensor at the farm, and later used to predict the composition of subsequent samples over a period of more than 6 months. The effectiveness of the fully unsupervised method was confirmed when the mean concentration of the milk samples remained constant, while the effectiveness reduced when this was not the case. The bulk milk correction method was effective when all relevant samples for the component were measured by the sensor and included in the analyzed bulk milk, but is less effective when samples included in the bulk which are not measured by the sensor system. When the necessary conditions are met, these methods can be used to extend the lifetime of deployed prediction models by significantly reducing the bias on the predicted values.

Temperature-insensitive cascaded micro-ring resonator silicon photoelectric sensor based on wavelength interrogation for HSA detection

Silicon-based optical biosensors have garnered significant interest since their inception. They are, however, susceptible to temperature fluctuations in a controlled environment. In order to address this issue, we provide a solution including the utilization of a silicon-on-insulator (SOI) cascaded micro-ring resonators (CMRR) biosensor that is designed to be insensitive to temperature changes. It consists of two ring resonators exposed to the reference solution and analyte, respectively, their radius and waveguide width are designed to meet the temperature-insensitivity criterion of the CMRR sensor. The results of experiments indicate that by real-time monitoring of the envelope peak drift in the transmission spectrum of the sensor, the bulk refractive index (RI) sensitivity is 2940.71 nm/RIU, the limit of detection (LOD) is 3.1×10−5 RIU, and the temperature sensitivity is less than 40 pm/℃. Furthermore, the sensor was applied for HSA detection, with a calculated LOD of 18.09 ng/mL. The effect of temperature on HSA detection is much smaller than that of a CMRR sensor without athermal design. The sensor demonstrates high sensitivity and temperature insensitivity, and the absence of negative thermo-optic coefficient materials or supplementary temperature control mechanisms simplifies its practical applications. It can be made by a large array to detect different type protein molecules for high throughput multiparameter analysis.

A Novel Object-Oriented Rotated Intensity Matching Method for Iceberg Drift Monitoring With SAR Images

A Novel Object-Oriented Rotated Intensity Matching Method for Iceberg Drift Monitoring With SAR Images

Detecting changes in the performance of a clinical machine learning tool over time

Excessive use of blood cultures (BCs) in Emergency Departments (EDs) results in low yields and high contamination rates, associated with increased antibiotic use and unnecessary diagnostics. Our team previously developed and validated a machine learning model to predict BC outcomes and enhance diagnostic stewardship. While the model showed promising initial results, concerns over performance drift due to evolving patient demographics, clinical practices, and outcome rates warrant continual monitoring and evaluation of such models. A real-time evaluation of the model's performance was conducted between October 2021 and September 2022. The model was integrated into Amsterdam UMC's Electronic Health Record system, predicting BC outcomes for all adult patients with BC draws in real time. The model's performance was assessed monthly using metrics including the Area Under the Curve (AUC), Area Under the Precision-Recall Curve (AUPRC), and Brier scores. Statistical Process Control (SPC) charts were used to monitor variation over time. Across 3.035 unique adult patient visits, the model achieved an average AUC of 0.78, AUPRC of 0.41, and a Brier score of 0.10 for predicting the outcome of BCs drawn in the ED. While specific population characteristics changed over time, no statistical points outside the statistical control range were detected in the AUC, AUPRC, and Brier scores, indicating stable model performance. The average BC positivity rate during the study period was 13.4%. Despite significant changes in clinical practice, our BC stewardship tool exhibited stable performance, suggesting its robustness to changing environments. Using SPC charts for various metrics enables simple and effective monitoring of potential performance drift. The assessment of the variation of outcome rates and population changes may guide the specific interventions, such as intercept correction or recalibration, that may be needed to maintain a stable model performance over time. This study suggested no need to recalibrate or correct our BC stewardship tool. No funding to disclose.

Amplitude Stability Detection of Sinusoidal Excitation Signal for High Precision Sensors

This study presents a high-precision detection method based on biased operational rectifier (BOR) and tracking analog-to-digital converter (TADC) for real-time monitoring of the amplitude drift of sinusoidal excitation signals, which significantly reduces the requirement for ADC's measurement accuracy and improves the signal-to-noise ratio (SNR). BOR modules are proposed innovatively to make the amplitude drift can be detected easily by an ADC with poor precision. Combining with the tracking ADC, the proposed detector could suppress the noise near the cut-off frequency and thus improve SNR. An amplitude drift detector and a lock-in amplifier are designed and tested experimentally. The results show that the voltage resolution of the proposed detector could reach 0.12 ppm for a sinusoidal source with an amplitude of 2.5 V. The SNR is measured 88 dB, which is 8 dB higher than that of the lock-in amplifier. The proposed detector is also applied in a precision capacitive displacement sensor. With the amplitude drift detector, the drift of the capacitive sensor caused by the signal source's amplitude drift could be significantly reduced from 16 nm/°C to 2 nm/°C. Furthermore, this method is also applicable to other amplitude-modulated types of high-precision sensors.

In Situ Drift Monitoring and Calibration of Field-Deployed Potentiometric Sensors Using Temperature Supervision.

Potentiometric ion-selective electrodes (ISEs) have broad applications in personalized healthcare, smart agriculture, oil/gas exploration, and environmental monitoring. However, high-precision potentiometric sensing is difficult with field-deployed sensors due to time-dependent voltage drift and the need for frequent calibration. In the laboratory setting, these issues are resolved by repeated calibration by measuring the voltage response at multiple standard solutions at a constant temperature. For field-deployed sensors, it is difficult to frequently interrupt operation and recalibrate with standard solutions. Moreover, the constant surrounding temperature constraint imposed by the traditional calibration process makes it unsuitable for temperature-varying field use. To address the challenges of traditional calibration for field-deployed sensors, in this study, we propose a novel in situ calibration approach in which we use natural/external temperature variation in the field to obtain the time-varying calibration parameters, without having to relocate the sensors or use any complex system. We also develop a temperature-supervised monitoring method to detect the drift of the sensor during operation. Collectively, the temperature-based drift monitoring and in situ calibration methods allow us to monitor the drift of sensors and correct them periodically to achieve high-precision sensing. We demonstrate our approach in three testbeds: (1) under controlled temperature variation in the lab, (2) under natural temperature variation in a greenhouse, and (3) in the field to monitor nitrate activity of an agricultural site. In the laboratory study, we validate that the calibration parameters of printed nitrate ISEs can be reproduced by our proposed calibration process; therefore, it can serve as an alternative to traditional calibration processes. In the greenhouse, we show the use of natural temperature variation to calibrate the sensors and detect the drift in a fixed concentration nitrate solution. Finally, we demonstrate the use of the method to monitor the nitrate activity of an agricultural field within 10% of laboratory-based measurements (i.e., a sensitivity of 0.03 mM) for a period of 22 days. The findings highlight the prospect of temperature-based calibration and drift monitoring for high-precision sensing with field-deployed ISEs.

From Staple Food to Scarce Resource: The Population Status of an Endangered Striped Catfish Pangasianodon hypothalamus in the Mekong River, Cambodia

Striped catfish Pangasianodon hypopthalmus (Sauvage, 1878) is a flagship catfish species of the Mekong River region, a commercially valuable food fish that is important in freshwater fisheries, and a popular aquaculture species in many Asian countries. The species was assessed as “Endangered” by the International Union for Conservation of Nature (IUCN) due to range contraction and declining abundance, though the status of the species’ wild population in Cambodia, a critical habitat for the species, is not well understood. Here, we assess the population status of the striped catfish in Cambodia using multiple sources, including time-series catch data and length frequency distribution data from a commercial fishery (stationary trawl bagnet or dai) operated in the Tonle Sap River from 1998/99 to 2017/18 and larval drift data monitored in the Mekong River in Phnom Penh from 2004 to 2018. We found that there was a significant decline (R2 = 0.54, p = 0.0002) in the catch (metric tonnes) of the striped catfish from the commercial dai fishery over the last two decades. Similarly, length-based indicator analysis indicates that striped catfish mean length and abundance have both declined over the study period, raising concerns about the sustainability of river catfish fisheries. Moreover, long-term larval drift monitoring in Mekong River shows that there was a marginally significant decline in the quantity of striped catfish larvae/juvenile drifting downstream to the lower floodplain over the last decade. Changes in flood index (extent and duration of flood) in the Tonle Sap floodplain affected by the Mekong’s flow are likely key factors driving the decline of the wild populations of the striped catfish. Both larval fish abundance and floodplain fish harvests have a significant positive relationship with Mekong flow and flood extent. Indiscriminate fishing exacerbates pressures on striped catfish stocks. Therefore, actions such as maintaining natural seasonal flows (flood timing, extent, and duration) to the Tonle Sap floodplain and protecting migratory fish stocks from overharvest and habitat fragmentation are essential to the persistence of stocks of striped catfish and other large-bodied migratory fishes that utilize both the Cambodian Mekong and Tonle Sap floodplains.

Combined use of multiple external and internal standards in LA-ICP-MS analysis of bulk geological samples using lithium borate fused glass

We describe a method for laser ablation inductively coupled plasma mass spectrometry (LA-ICP-MS) elemental analysis of geological materials using low-dilution Li-borate fused glass wavelength dispersive X-ray fluorescence (WDXRF) pellets, with samples, drift monitor and 18 reference materials (RMs) identically prepared. After analysis for 46 elements by WDXRF, LA-ICP-MS intensities from samples and RMs are collected, and background corrected with Iolite software. HALite , a new software application, was developed to derive the elemental compositions from the LA-ICP-MS net signals. In HALite , elements are drift-corrected using polynomial functions, and flux-fused RM element sensitivities are calculated from known mass fractions. Multiple internal standard (IS) elements are used to model each sample's laser response. Analyte mass fractions in unknowns are determined using the calibrated sensitivity correlation models for multiple IS elements. Either the WDXRF mass fractions or the initial round of calculated LA-ICP-MS mass fractions are used to calculate weighted mean sensitivities. Validation experiments with flux-fused RMs run as unknowns yield results with less than 5–10% total relative uncertainty for most analytes. We derive equations which allow calculation of the precision and total uncertainty as a function of mass fraction for each analyte element. Supplementary material : Table 1 - RMs used; Table 2 - Operating parameters; Table 3 - Model v. accepted mass fractions; Table 4 - NIST 610 v. multiple IS models; Table 5 - Fitting parameters; Appendix 1 - HALite description; Appendix A - Summary calibration graphs; Appendix B - Validation results; Appendix C - WDXRF comparisons; Appendix D - Repeatability uncertainties; Appendix E - RM uncertainties; and Appendix F - Total uncertainties for this article are available at https://doi.org/10.6084/m9.figshare.c.6639885

Monitoring of NOx sensor drift in automotive fleets in a cloud/edge framework

Nitrogen oxides (NOx) are one of the main pollutants from both SI and CI engines. In recent years, regulation focus has moved from type-approval certification over known driving cycles to real-life verification of the emission level. In this paper, a system for NOx in-service emission monitoring based on ASSIST-IoT cloud/edge architecture is presented, and the effect of sensor bias on the emission level estimate discussed. The use of an additional high-fidelity sensor in a sample of the fleet is proposed as a method for estimating series sensor drift through a distributed learning approach, able to provide local and global models for the sensor.

Vision-based thermal drift monitoring method for machine tools

A method is presented to measure machine tool thermal drift for error compensation. A wireless microscope within a tool holder in the spindle is used to capture videos of image targets attached to the worktable. For each target, one video is captured during spindle rotation orthogonal to the worktable and another video is captured during axis translation orthogonal to the worktable. Data are collected periodically so that the three-dimensional thermal error at each target location is determined via image analysis. Experiments verify that the method measures micrometer-level tool-to-workpiece thermal drift for error model development and thermal drift compensation.

Concept Drift Monitoring and Diagnostics of Supervised Learning Models via Score Vectors

Supervised learning models are one of the most fundamental classes of models. Viewing supervised learning from a probabilistic perspective, the set of training data to which the model is fitted is usually assumed to follow a stationary distribution. However, this stationarity assumption is often violated in a phenomenon called concept drift, which refers to changes over time in the predictive relationship between covariates X and a response variable Y and can render trained models suboptimal or obsolete. We develop a comprehensive and computationally efficient framework for detecting, monitoring, and diagnosing concept drift. Specifically, we monitor the Fisher score vector, defined as the gradient of the log-likelihood for the fitted model, using a form of multivariate exponentially weighted moving average, which monitors for general changes in the mean of a random vector. In spite of the substantial performance advantages that we demonstrate over popular error-based methods, a score-based approach has not been previously considered for concept drift monitoring. Advantages of the proposed score-based framework include applicability to broad classes of parametric models, more powerful detection of changes as shown in theory and experiments, and inherent diagnostic capabilities for helping to identify the nature of the changes.

Development and comparative analysis of ANN and SVR-based models with conventional regression models for predicting spray drift.

As monitoring of spray drift during application can be expensive, time-consuming, and labor-intensive, drift predicting models may provide a practical complement. Several mechanistic models have been developed as drift prediction tool for various types of application equipment. Nevertheless, mechanistic models are quite often intricate and complex with a large number of input parameters required. Quite often, the detailed data needed for such models are not readily available. In this study, two advanced machine learning models (artificial neural network (ANN) and support vector regression (SVR)) were developed for pesticide drift prediction and compared with three conventional regression-based models: multiple linear regression (MLR), generalized linear model (GLM), and generalized nonlinear least squares (GNLS). The models were evaluated in fivefold cross-validation and by external validation using the coefficient of determination (R2), root mean square error (RMSE), mean absolute error (MAE), and mean absolute bias (MAB). From regression-based models, GLM and GNLS models performed very well when evaluated by cross-validation with R2 = 0.96 and 0.95 and RMSE = 0.70 and 0.82 respectively, while MLR performed less with R2 of 0.65 and RMSE of 2.25. Simultaneously, ANN and SVR models performed very well with R2 = 0.98 and 0.97 and RMSE = 0.58 and 0.71 respectively. Overall, ANN model performed best compared to the other four models followed by SVR. A comparison was also made between the high-performing model, ANN, and two previously published empirical models. The ANN model outperformed the two previously published empirical models and can be used to predict pesticide drift. Therefore, the ANN model is a potentially promising new approach for predicting ground drift that merits further study. In conclusion, our work demonstrated that the new approach, ANN and SVR-based models, for pesticide drift modeling has better predictive power than conventional regression models. Their ability to model complex relationships is a clear benefit in pesticide drift modeling where the variability in pesticide drift is often affected by a number of variables and the relationships between drift and predictors are very complicated. We believe such insights will pave better way for the application of machine learning towards spray drift modeling.

Thermal imaging drift monitoring of Doppler asymmetric spatial heterodyne spectroscopy for wind measurement based on segmented edge fitting

Doppler asymmetric spatial heterodyne spectroscopy is recently developed for spaceborne measurement of middle and upper atmospheric wind field, which relies on the accurate inverse of interferogram phase to calculate the Doppler shift of airglow emission lines. The change of temperature leads the optical and mechanical components to thermally deformed, causing the imaging plane to thermally drift relative to the detector, changing the distribution of interferogram phase on pixels, and directly introducing phase errors to affect the wind speed inversion. In order to reduce the influence of imaging thermal drift on phase inversion, the segmented fitting method is used in this paper to detect the sub-pixel edges of notch patterns and monitor imaging thermal drift accordingly. In the thermal stability test of a near-infrared Doppler asymmetric spatial heterodyne interferometer prototype, the thermal imaging drifts and ambient temperature show a high consistency in the trend of high-frequency oscillation, and the correlation coefficient can reach 0.86 after removing the baseline. After phase correct by using the thermal imaging shift, the high-frequency oscillation of interferogram phase shift is also greatly suppressed. In order to further verify the accuracy of the algorithm, the influence of the data signal-to-noise ratio and the data distribution characteristic parameter errors used in the fitting on the edge detection are simulated. The results show that the edge detection accuracy is restricted mainly by the data signal-to-noise ratio and the accuracy of the fringe frequency parameters. When the error of the fringe frequency parameter used for fitting is less than 0.5%, the error of other data distribution characteristic parameters is less than 5%, and the data signal-to-noise ratio is enhanced more than 35 times, the algorithm in this paper can achieve a detection accuracy higher than 0.05 pixels.

Segment Drift Control with a Supervision Mechanism for Autonomous Vehicles

Stable maneuverability is extremely important for the overall safety and robustness of autonomous vehicles under extreme conditions, and automated drift is able to ensure the widest possible range of maneuverability. However, due to the strong nonlinearity and fast vehicle dynamics occurring during the drift process, drift control is challenging. In view of the drift parking scenario, this paper proposes a segmented drift parking method to improve the handling ability of vehicles under extreme conditions. The whole process is divided into two parts: the location approach part and the drift part. The model predictive control (MPC) method was used in the approach to achieve consistency between the actual state and the expected state. For drift, the open-loop control law was designed on the basis of drift trajectories obtained by professional drivers. The drift monitoring strategy aims to monitor the whole drift process and improve the success rate of the drift. A simulation and an actual vehicle test platform were built, and the test results show that the proposed algorithm can be used to achieve accurate vehicle drift to the parking position.

Direct multiplex PCR-NALFIA to inform marine conservation: Use of an innovative diagnostic tool for the detection of Ostrea edulis larvae

The European oyster Ostrea edulis played a key role in the North Sea by providing several ecosystem functions and services. Today, O. edulis is classified as severely degraded or functionally extinct in Europe. Marine conservation is focusing on biogenic reef restoration, namely the restoration of O. edulis in Natura 2000 sites of the North Sea. The identification of oyster larvae related to natural spatfalls of restored reefs and monitoring of larval drift is a key aspect of marine protected area management. Morphological identification and distinction from other abundant bivalve larvae using microscopy is difficult. Existing molecular biological methods are expensive and bound to stationary laboratory equipment, or are inadequate in the visualization. In this study, we identified nucleic acid lateral flow immunoassay (NALFIA), a well-established tool in human pathogen diagnostics, as an efficient approach for point-of-care (POC) testing in marine monitoring. Based on the genetic sequence of the mitochondrial cytochrome b of O. edulis, forward and reverse primers were developed. The reverse primer was labelled with fluorescent dye FITC, forward primer with biotin. Reaction on the lateral flow stripe could be realized with a single O. edulis larva in direct PCR with multiplex primers in a portable PCR-cycler. The established NALFIA system can distinguish O. edulis larvae from Crassostrea gigas and Mytilus edulis larvae, respectively. This method offers new approaches in POC testing in marine research and monitoring. It gives quick and clear results, is inexpensive, and could be easily adapted to other species of interest.