Sources Of Error Research Articles

Understanding reliability of the observer-reported communication ability measure within Angelman syndrome through the lens of generalizability theory

AimsCaregivers rate improved communication ability as one of the most desired outcomes for successful interventions for individuals with Angelman syndrome (AS). When measuring communication ability in clinical trials, the reliability of such measures is critical for detecting significant changes over time. This study examined the reliability of the Observed-Reported Communication Ability (ORCA) measure completed by caregivers of individuals with AS.MethodsThe ORCA measure was completed by 249 caregivers with 170 caregivers completing the ORCA measure again after 5–12 days. Generalizability theory was used to examine the following sources of measurement error in ORCA scores: concepts, subdomains, assessment points, and the interactions among those facets and the object of measurement: communication ability. Three generalizability studies were conducted to understand the reliability of the ORCA measure for different measurement designs. Decision studies were carried out to demonstrate the optimization of measurement procedures of the ORCA measure.ResultsG and Phi coefficients of the original measurement design exceeded the 0.80 threshold considered sufficiently reliable to make relative and absolute decisions about the communication ability of individuals with AS based on their caregivers’ observed scores. The optimization procedures indicated that increasing the number of communication concepts and/or assessment points leads to more reliable estimates of communication.ConclusionThe ORCA measure was able to reliably distinguish different levels of communication ability among individuals with AS. Multiple assessment points and or more concepts would provide more precise estimates of an individual’s communication ability but at the cost of survey fatigue.

A comprehensive investigation of uncertainties in cavitating waterjet experiments

This paper systematically examines the sources of uncertainty in cavitating waterjet test setups and provides illustrative case studies to conduct thorough uncertainty analyses for erosion tests using cavitating waterjet method. Within this study, utilizing examples of uncertainty analysis presented by the International Towing Tank Conference (ITTC) for various testing techniques, a novel uncertainty analysis implementation suitable for cavitating waterjet tests has been conducted. Firstly, cavitation erosion experiments were carried out on samples aluminum (Al 6063) material under cavitation number conditions (σ = 0.02 to σ = 0.033 range) as part of the validation study. Subsequently, within the scope of uncertainty analysis calculations, five erosion tests were conducted for each of the two different cavitation numbers (σ = 0.03 and σ = 0.05). The study aims to examine the bias and precision uncertainty components separately for the test results of erosion rate and establish a reliable and repeatable procedure for predicting the uncertainty level of cavitating waterjet tests. For the first time in the literature, the uncertainty components of cavitating waterjet tests were investigated in detail and the uncertainty level in cavitating waterjet tests, including bias and precision errors, was predicted in accordance with specific ITTC guidelines and procedures. Based on the results, the primary bias error source was identified as the pressure inside the cavitation chamber, with the nozzle pressure being the second most significant bias error source. Consequently, it is recommended to perform the bias error prediction based on the ratio of these two, known as the cavitation number. Precision error, crucial for uncertainty, was estimated using five sets of erosion rate values, emphasizing its importance in cavitating waterjet tests. It is anticipated that the suggested methodology will serve as a benchmark for predicting uncertainty levels in cavitating waterjet tests.

Preliminary sensitivity study for a gravitational redshift measurement with China's Lunar Exploration Project

Abstract General relativity (GR) is a highly successful theory that describes gravity as a geometric phenomenon. The gravitational redshift, a classic test of GR, can potentially be violated in alternative gravity theories, and experimental tests on this effect are crucial for our understanding of gravity. In this paper, considering the space-ground clock comparisons with free-space links, we discuss a high-precision Doppler cancellation-based measurement model for testing gravitational redshift. This model can effectively reduce various sources of error and noise, reducing the influences of the first-order Doppler effect, atmospheric delay, Shapiro delay, etc. China's Lunar Exploration Project (CLEP) is proposed to equip the deep-space H maser with a daily stability of $2\times10^{-15}$, which provides an approach for testing gravitational redshift. Based on the simulation, we analyze the space-ground clock comparison experiments of the CLEP experiment, and simulation analysis demonstrates that under ideal condition of high-precision measurement of the onboard H-maser frequency offset and drift, the CLEP experiment may reach the uncertainty of $3.7\times10^{-6}$ after a measurement session of 60 days. Our results demonstrate that if the issue of frequency offset and drift is solved, CLEP missions have a potential of testing the gravitational redshift with high accuracy.

Taxonomy of performance shaping factors in manufacturing: A systematic literature review

AbstractHuman error in manufacturing can have substantial consequences, including loss of life, injuries, productivity, and financial losses. Human reliability analysis (HRA) methods can be used to evaluate the likelihood of human error in manufacturing tasks and identify potential sources of error. Performance shaping factors (PSFs) are internal and external factors that influence human performance and can affect the likelihood of human reliability estimates in HRA methods. Understanding the impact of PSFs on human performance in manufacturing is essential for developing effective strategies to minimize the likelihood of human error and improve the safety and efficiency of manufacturing processes. This systematic review scrutinizes the literature on PSFs within manufacturing, highlighting HRA applications. Using the PRISMA protocol, studies from 2000 to 2024 across engineering and psychology were examined, culminating in the analysis of 35 pertinent works. The review identifies and contrasts various PSF taxonomies from established HRA methods like SPAR‐H, HEART, CREAM, and THERP, revealing their diverse applications in different manufacturing settings. The review also uncovers a tendency to devise taxonomies through the lens of experts' domain knowledge, particularly tailored to discrete manufacturing contexts. A critical gap is observed in the lack of a uniform PSF framework, with the current literature reflecting a disparate understanding of PSFs' roles, definitions, and interrelations. This absence is further pronounced by the inadequate integration of human factors in the dialogue surrounding Industry 4.0. The analysis points to the necessity of harmonizing PSFs to better assess human reliability amid technological integration. The findings emphasize the need for an industry‐specific PSF framework that aligns with the intricacies of manufacturing operations, thus enabling more accurate HRA outcomes and informing strategies for error reduction and process optimization.

Improving the Precision of Classroom Observation Scores Using a Multi-Rater and Multi-Timepoint Item Response Theory Model

ABSTRACT Classroom observational protocols, in which raters observe and score the quality of teachers’ instructional practices, are often used to evaluate teachers for consequential purposes despite evidence that scores from such protocols are frequently driven by factors, such as rater and temporal effects, that have little to do with teacher quality. In this study, we present an item response theory model that improves precision of observation scores by accounting for multiple sources of measurement error. Unlike the sum-scoring approach typically used in practice, the model presented here identifies raters and protocol items that are more reliable and should therefore be weighted more in scores. We then show that teachers identified as substandard using our model versus the sum-scoring approach are oftentimes very different. Our results suggest that accountability systems could improve observation scores by increasing the scoring weights of items and raters that provide greater information.

Japanese tort-case dataset for rationale-supported legal judgment prediction

AbstractThis paper presents the first dataset for Japanese Legal Judgment Prediction (LJP), the Japanese Tort-case Dataset (JTD), which features two tasks: tort prediction and its rationale extraction. The rationale extraction task identifies the court’s accepting arguments from alleged arguments by plaintiffs and defendants, which is a novel task in the field. JTD is constructed based on annotated 3477 Japanese Civil Code judgments by 41 legal experts, resulting in 7978 instances with 59,697 of their alleged arguments from the involved parties. Our baseline experiments show the feasibility of the proposed two tasks, and our error analysis by legal experts identifies sources of errors and suggests future directions of the LJP research.

Error estimation in current noisy quantum computers

One of the main important features of the noisy intermediate-scale quantum (NISQ) era is the correct evaluation and consideration of errors. In this paper, we analyse the main sources of errors in current (IBM) quantum computers and we present a useful tool (TED-qc) designed to facilitate the total error probability expected for any quantum circuit. We propose this total error probability as the best way to estimate a lower bound for the fidelity in the NISQ era, avoiding the necessity of comparing the quantum calculations with any classical one. In order to contrast the robustness of our tool we compute the total error probability that may occur in three different quantum models: 1) the Ising model, 2) the Quantum-Phase Estimation (QPE), and 3) the Grover’s algorithm. For each model, the main quantities of interest are computed and benchmarked against the reference simulator’s results as a function of the error probability for a representative and statistically significant sample size. The analysis is satisfactory in more than the 99%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$99\\%$$\\end{document} of the cases. In addition, we study how error mitigation techniques are able to eliminate the noise induced during the measurement. These results have been calculated for the IBM quantum computers, but both the tool and the analysis can be easily extended to any other quantum computer.

NMR Precision Metabolomics: Dynamic Peak Sum Thresholding and Navigators for Highly Standardized and Reproducible Metabolite Profiling of Clinical Urine Samples

Metabolomics, especially urine-based studies, offers incredible promise for the discovery and development of clinically impactful biomarkers. However, due to the unique challenges of urine, a highly precise and reproducible workflow for NMR-based urine metabolomics is lacking. Using 1D and 2D non-uniform sampled (NUS) 1H-13C NMR spectroscopy, we systematically explored how changes in hydration or specific gravity (SG) and pH can impact biomarker discovery. Further, we examined additional sources of error in metabolomics studies and identified Navigator molecules that could monitor for those biases. Adjustment of SG to 1.002–1.02 coupled with a dynamic sum-based peak thresholding eliminates false positives associated with urine hydration and reduces variation in chemical shift. We identified Navigator molecules that can effectively monitor for inconsistencies in sample processing, SG, protein contamination, and pH. The workflow described provides quality assurance and quality control tools to generate high-quality urine metabolomics data, which is the first step in biomarker discovery.

Local primordial non-Gaussianity from the large-scale clustering of photometric DESI luminous red galaxies

Abstract We use angular clustering of luminous red galaxies from the Dark Energy Spectroscopic Instrument (DESI) imaging surveys to constrain the local primordial non-Gaussianity parameter fNL. Our sample comprises over 12 million targets, covering 14,000 square degrees of the sky, with redshifts in the range 0.2 < z < 1.35. We identify Galactic extinction, survey depth, and astronomical seeing as the primary sources of systematic error, and employ linear regression and artificial neural networks to alleviate non-cosmological excess clustering on large scales. Our methods are tested against simulations with and without fNL and systematics, showing superior performance of the neural network treatment. The neural network with a set of nine imaging property maps passes our systematic null test criteria, and is chosen as the fiducial treatment. Assuming the universality relation, we find $f_{\rm NL} = 34^{+24(+50)}_{-44(-73)}$ at 68 per cent(95 per cent) confidence. We apply a series of robustness tests (e.g. cuts on imaging, declination, or scales used) that show consistency in the obtained constraints. We study how the regression method biases the measured angular power-spectrum and degrades the fNL constraining power. The use of the nine maps more than doubles the uncertainty compared to using only the three primary maps in the regression. Our results thus motivate the development of more efficient methods that avoid over-correction, protect large-scale clustering information, and preserve constraining power. Additionally, our results encourage further studies of fNL with DESI spectroscopic samples, where the inclusion of 3D clustering modes should help separate imaging systematics and lessen the degradation in the fNL uncertainty.

Influence of Supraglacial Lakes on Accuracy of Inversion of Greenland Ice Sheet Surface Melt Data in Different Passive Microwave Bands

The occurrence of Supraglacial Lakes (SGLs) may influence the signals acquired with microwave radiometers, which may result in a degree of uncertainty when employing microwave radiometer data for the detection of surface melt. Accurate monitoring of surface melting requires a reasonable assessment of this uncertainty. However, there is a scarcity of research in this field. Therefore, in this study, we computed surface melt in the vicinity of Automatic Weather Stations (AWSs) by employing Defense Meteorological Satellite Program (DMSP) Ka-band data and Soil Moisture and Ocean Salinity (SMOS) satellite L-band data and extracted SGL pixels by utilizing Sentinel-2 data. A comparison between surface melt results derived from AWS air temperature estimates and those obtained with remote sensing inversion in the two different bands was conducted for sites below the mean snowline elevation during the summers of 2016 to 2020. Compared with sites with no SGLs, the commission error (CO) of DMSP morning and evening data at sites where these water bodies were present increased by 36% and 30%, respectively, and the number of days with CO increased by 12 and 3 days, respectively. The omission error (OM) of SMOS morning and evening data increased by 33% and 32%, respectively, and the number of days with OM increased by 17 and 21 days, respectively. Identifying the source of error is a prerequisite for the improvement of surface melt algorithms, for which this study provides a basis.

P-chromophore stability of Mehlich-1 and Mehlich-3 under Braga & Defelipo or Murphy & Riley dosing methods

ABSTRACT The performance of colorimetric methods for P quantification has been overlooked by researchers for decades. We investigated the performance of two blue colorimetric methods for P quantification, Braga and Defelipo (B&D) and Murphy and Riley (M&R), using two commonly soil P extractants, Mehlich-1 (M1) and Mehlich-3 (M3). Specifically, we evaluated the color development time and its stability in different soil extractant solution proportions (1:1 or 1:4 v/v), the optimum wavelength, limits of detection, and element quantification. Our results indicate that M3 leads to lower limits of detection and quantification for both colorimetric methods, particularly for B&D. For M1, the volumetric ratio (1:1 or 1:4) did not influence color development and both B&D and M&R methods showed a fast color development. However, B&D showed greater color stability (from 5 to 600 min) and an optimum wavelength of 711 nm, while M&R was stable from 27 to 600 min an optimum wavelength of 889 nm. For M3 soil extractant, there are important issues, such as M&R being unstable and B&D presenting slow color development. In addition, the spectral profile obtained from soil extractant was different from the one obtained by the calibration curve without soil for both M&R and B&D. Therefore, the adoption of original methods on P quantification in M3 extracting solution represents a potential source of error, leading to wrong P fertilizer recommendations. Thus, the best option for P quantification in M3 extracts seems to be the use of Inductively Coupled Plasma Atomic Emission Spectrometry (ICP-AES).

SWAN: A multihead autoregressive attention model for solar wind speed forecasting

High speed streams of solar wind can impact Earth’s magnetic environment. Current state of the art geomagnetic effect forecasting methods can predict perturbations up to two hours in advance by using solar wind measurements made in the L1 Lagrange point. By forecasting these solar wind parameters, it should be possible to extend the time range for which said methods can be used.We present Solar Wind Attention Network (SWAN), an attention-based autoregressive model for the forecasting of daily average bulk speed values. We apply a novel automatic design method to create its architecture by developing a pipeline of hyperparameter optimization techniques as a proof of concept. Usage of this pipeline leads to a robust, reliable architecture which we empirically verify the convergence of in a variety of problem configurations regarding solar wind speed forecasting.An improvement in terms of root mean squared error (RMSE) is found when compared to WindNet and convolutional neural networks (CNNs) expanding it, even when SWAN does not incorporate images of the solar disk into the input. SWAN’s RMSE of 68.79 ± 1.05 km s−1 with a lead time of three days compares to RMSE values in the order of 80.28 ± 3.05, 76.30 ± 1.87 and 71.93 ± 0.40 km s−1. From these results, we discuss the possibility of incorporating both autoregressive and image information as input to enhance predictive capability of newly developed models. Furthermore, we analyze the resulting model’s behavior to characterize its main error sources, and find its performance to be at its best in scenarios not involving newly formed coronal holes or coronal mass ejections. It is found that this behavior matches well with a priori expectations derived from the underlying physics. Finally, we briefly discuss the need for an in-depth study of different evaluation metrics to provide a more thorough understanding of modeling results on the field.

Reanalysis of multi-year high-resolution X-band weather radar observations in Hamburg

Abstract. This paper presents an open-access data set of reanalysed radar reflectivities and rainfall rates at sub-kilometre spatial and minute temporal scales. Variability at these scales is a blind spot for both operational rain gauge networks and operational radar networks. In the urban area of Hamburg, precipitation measurements of a single-polarized X-band weather radar operating at high temporal (30 s), range (60 m), and azimuthal sampling (1°) resolutions are made available for a period of more than 8 years. We describe in detail the reanalysis of the raw radar data, outline the radar performance for the years 2013 to 2021, and discuss open issues and limitations of the data set. Several sources of radar-based errors were adjusted gradually, affecting the radar reflectivity and rainfall measurements, e.g. noise, alignment, non-meteorological echoes, radar calibration, and attenuation. The deployment of additional vertically pointing micro rain radars yields drop size distributions at the radar beam height, which effectively reduces errors concerning the radar calibration and attenuation correction and monitors the radar data quality. A statistical evaluation revealed that X-band radar reflectivities and rainfall rates are in very good agreement with the micro rain radar measurements. Moreover, the analyses of rainfall patterns shown for an event and accumulated rainfall of several months prove the quality of the data set. The provided radar reflectivities facilitate studies on attenuation correction and the derivation of further weather radar products, like an improved rainfall rate. The rainfall rates themselves can be used for studies on the spatial and temporal scales of precipitation and hydrological research, e.g. input data for high-resolution modelling, in an urban area. The radar reflectivities and rainfall rates are available at https://doi.org/10.26050/WDCC/LAWR_UHH_HHG_v2 (Burgemeister et al., 2024).

A novel approach to model nance drying considering its shrinkage

Air drying is a viable preservation method to boost the accessibility of nance, a seasonal drupe with very short shelf life; however, information about nance drying is scarce, especially from a mass transfer (MT) standpoint. Besides, the lack of simple models considering the stone fraction (radius of inner stone/radius of nance) and the mesocarp shrinkage in drupes has caused most investigations rely on sphere and cube geometries, contributing to the unreliable estimation of MT properties. This study developed a discrete state-space (SS) formulation with analytical solution to describe nance drying which, unlike a variable separation (VS) solution, can consider a variable stone fraction, caused by outer boundary moving while flesh-pit interface remains still. Nance fruits were air dried (60, 70 and 80 °C) to obtain their moisture kinetics and shrinking behavior. Drying data were used to estimate water diffusivity (D) under rigid (VS and SS solutions) and shrinkable (SS and numerical solution) mesocarp assumptions. Nances lost about 60% of their initial volume at the end of drying, while stone fraction increased from 0.44 to 0.62. D values ranged from 5.3 to 6.7 (×10–10) m2/s with SS approach and variable stone fraction. The major source of error in D was not the formulation (SS or VS) when a rigid mesocarp was considered, but the values assigned to the nance radius (length for diffusion) and stone fraction, an initial or average value along process. SS diffusivities did not differ to those estimated with the full numerical solution under variable stone fraction (p>0.05), while VS parameters were overestimated by 44–46 % as a minimum. SS formulation can be applied to hollow cylinders and other geometries with 2D or 3D MT, while able to include non-isotropic shrinkage and evolving external conditions, a problem normally reserved to pure numerical implementations, offering substantial advantages over VS equations.

A Machine Learning-Based Tropospheric Prediction Approach for High-Precision Real-Time GNSS Positioning

For high-precision positioning applications, various GNSS errors need to be mitigated, including the tropospheric error, which remains a significant error source as it can reach up to a few meters. Although some commercial GNSS correction data providers, such as the Quasi-Zenith Satellite System (QZSS) Centimeter Level Augmentation Service (CLAS), have developed real-time precise regional troposphere products, the service is available only in limited regional areas. The International GNSS Service (IGS) has provided precise troposphere correction data in TRO format post-mission, but its long latency of 1 to 2 weeks makes it unable to support real-time applications. In this work, a real-time troposphere prediction method based on the IGS post-processing products was developed using machine learning techniques to eliminate the long latency problem. The test results from tropospheric predictions over a year using the proposed method indicate that the new method can achieve a prediction accuracy (RMSE) of 2 cm, making it suitable for real-time applications.

Error analysis of wave intercomparison based on sampling variability

Quantitative analysis of error sources in wave measurement intercomparison under real-world conditions remains challenging due to various factors such as spatial and temporal offsets in the field, diverse configurations of wave sensors and platforms, and distinct measurement algorithms. This study introduces a modified error analysis method based on sampling variability. Measurements from two co-deployed Datawell Waverider buoys at the National Marine Test Site (Weihai) in China offer an opportunity to assess the spatial variability error. The results reveal that for non-directional wave parameters, like significant wave height and mean zero-crossing period, the wave conditions in the field can be considered stationary, with random errors primarily arising from inherent sampling variability. Spatial offsets significantly impact directional wave parameters at peak frequencies, with large deviations (greater than 20°) in peak wave direction primarily attributed to bimodal distribution. For data with significant wave heights exceeding 0.5 m, random errors in the mean wave direction at the same frequency can be mainly attributed to sampling variability. The bulk directional wave parameters weighted by the energy spectrum demonstrate lower sensitivity to spatial offsets, and there is excellent agreement between the bulk mean directions when considering only the unimodal distribution data.

Simulations of Superconducting Quantum Gates by Digital Flux Tuner for Qubits

Abstract The interconnection bottleneck caused by limitations of cable number, inner space and cooling power of dilution refrigerators has been an outstanding challenge for building scalable superconducting quantum computers with the increasing number of qubits in quantum processors. To surmount such an obstacle, it is desirable to integrate qubits with quantum-classical interface (QCI) circuits based on rapid single flux quantum (RSFQ) circuits. In this work, a digital flux tuner for qubits (DFTQ) is proposed for manipulating flux of qubits as a crucial part of the interface circuit. A schematic diagram of the DFTQ is presented, consisting of a coarse tuning unit and a fine-tuning unit for providing magnetic flux with different precision to qubits. The method of using DFTQ to provide flux for gate operations is discussed from the optimization of circuit design and input signal. To verify the effectiveness of the method, simulations of a single DFTQ and quantum gates including a Z gate and an iSWAP gate with DFTQs are performed for flux-tunable transmons. The quantum process tomography corresponding to the two gates is also carried out to analyze the sources of gate error. The results of tomography show that the gate fidelities independent on initial states of the Z gate and the iSWAP gate are 99.935% and 99.676%, respectively. With DFTQs inside, the QCI would be a powerful tool for building large-scale quantum computers.

Evaluating error sources to improve precision in the co-registration of underwater 3D models

Change detection is an essential and widely used approach for investigating ecosystem dynamics. Multi-temporal 3D models increasingly underpin photogrammetry-based analyses of change for many ecologically relevant attributes. To detect change, it is necessary to accurately align 3D models collected at different times using a process referred to as co-registration. However, achieving precise co-registration is difficult in underwater habitats due to practical challenges intrinsic to surveying them. These include a lack of accurate georeferencing information, variable light, turbidity and weather conditions, and diving restrictions dictated by the diver's pressure exposure over time. Here we present an efficient co-registration workflow for 3D models that directly addresses these challenges, derived from underwater structure-from-motion methods. To test our approach, we used 3D models from across a wide range of coral reef habitats covering all those that one may encounter in shallow reefs (15 m depth and above). We then identified and empirically estimated four key sources of error: co-registration, 3D processing, image acquisition, and reference and scaling features (RSF) placement, and quantified their relative contributions to the overall error. Our proposed co-registration workflow had a mean precision of 1.37 ± 16.55 mm. Image acquisition and RSF placement errors contributed the most to the total workflow error (37% and 53%, respectively), while the contribution of co-registration and 3D processing errors was minimal (3% and 7%, respectively). As a result of our analysis, we provide ‘good practice’ guidelines to reduce errors associated with photogrammetric workflows and to facilitate efficient and reliable detection of 3D change in complex underwater ecosystems.

Development and implementation of the external quality assessment program for erythrocyte sedimentation rate in hematology laboratories in Thailand

Background: The erythrocyte sedimentation rate (ESR) remains the most widely used screening test for monitoring the course of infections, inflammatory diseases, and some types of cancer. Proficiency Testing (PT) programs are unavailable in Thailand for ESR. Objective: This study aimed to develop and initiate the PT program to improve testing quality through evaluation and analysis of the laboratory test results for ESR. Materials and methods: The PT program for ESR was established at the AMS CMU EQA Unit by the ISO/IEC17043:2010 requirements. The unit produced and sent the ESR control materials to participants for laboratory analysis; the test results were returned to the unit. The PT program was carried out in two cycles per year, with two samples in each cycle. The performance of each laboratory was assessed using robust Z score and performance score. Results: ESR control materials presented with satisfactory homogeneity and stability according to ISO 13528: 2015. One hundred and four laboratories were enrolled in the PT program. Of these laboratories, 96 (92.3%) returned their data: 40 (42.7%) used the modified Westergren method, 29 (29.2%), 14 (15.7%), and 13 (12.4%) used Microsed, Mixrate, and other methods, respectively. Satisfactory and questionable performance was obtained in 82/96 (85.4.%) and 8/96 (8.3%), respectively. Unsatisfactory performance was noted at 6/96 (6.3%). The level of excellent performance was achieved by 82 (approx. 85%) of these laboratories. Two main types of errors found from analyzing the received data were (i) specimen processing and (ii) incorrect identification of the analytical method. Conclusion: Our results indicate that the advantage of participation in the PT program for ESR is that the laboratories can monitor and investigate the source of laboratory errors. Therefore, the PT program for ESR testing should become an integral part of the quality assurance system in the laboratory.

Analysis of machine learning prediction reliability based on sampling distance evaluation with feature decorrelation

Despite successful use in a wide variety of disciplines for data analysis and prediction, machine learning (ML) methods suffer from a lack of understanding of the reliability of predictions due to the lack of transparency and black-box nature of ML models. In materials science and other fields, typical ML model results include a significant number of low-quality predictions. This problem is known to be particularly acute for target systems which differ significantly from the data used for ML model training. However, to date, a general method for uncertainty quantification (UQ) of ML predictions has not been available. Focusing on the intuitive and computationally efficient similarity-based UQ, we show that a simple metric based on Euclidean feature space distance and sampling density together with the decorrelation of the features using Gram–Schmidt orthogonalization allows effective separation of the accurately predicted data points from data points with poor prediction accuracy. To demonstrate the generality of the method, we apply it to support vector regression models for various small data sets in materials science and other fields. We also show that this metric is a more effective UQ tool than the standard approach of using the average distance of k nearest neighbors (k = 1–10) in features space for similarity evaluation. Our method is computationally simple, can be used with any ML learning method and enables analysis of the sources of the ML prediction errors. Therefore, it is suitable for use as a standard technique for the estimation of ML prediction reliability for small data sets and as a tool for data set design.