Calibration Framework Research Articles

Empirical Evaluations of Framework for Adaptive Trust Calibration in Human-AI Cooperation

Recent advances in AI technologies are dramatically changing the world and impacting our daily life. However, human users still essentially need to cooperate with AI systems to complete tasks as such technologies are never perfect. For optimal performance and safety in human-AI cooperation, human users must appropriately adjust their level of trust to the actual reliability of AI systems. Poorly calibrated trust can be a major cause of serious issues with safety and efficiency. Previous works on trust calibration have emphasized the importance of system transparency for avoiding trust miscalibration. Measuring and influencing trust are still challenging issues; consequently, not many studies have focused on how to detect improper trust calibration nor how to mitigate it. We approach these research challenges with a behavior-based approach to capture the status of calibration. A framework of adaptive trust calibration is proposed, including a formal definition of improper trust calibration called “a trust equation”. It involves cognitive cues called “trust calibration cues (TCCs)” and a conceptual entity called “trust calibration AI” (TCAI), which supervises the status of trust calibration. We conducted empirical evaluations using a simulated drone environment with two types of cooperative tasks: a visual search task and a real-time navigation task. We designed trust changing scenarios and evaluated our framework. The results demonstrated that adaptively presenting a TCC could promote trust calibration more effectively than a traditional system transparency approach.

Alignment for the first precision measurements at Belle II

On March 25th 2019, the Belle II detector recorded the first collisions delivered by the SuperKEKB accelerator. This marked the beginning of the physics run with vertex detector. The vertex detector was aligned initially with cosmic ray tracks without magnetic field simultaneously with the drift chamber. The alignment method is based on Millepede II and the General Broken Lines track model and includes also the muon system or primary vertex position alignment. To control weak modes, we employ sensitive validation tools and various track samples can be used as alignment input, from straight cosmic tracks to mass-constrained decays. With increasing luminosity and experience, the alignment is approaching the target performance, crucial for the first physics analyses in the era of Super-BFactories. We will present the software framework for the detector calibration and alignment, the results from the first physics run and the prospects in view of the experience with the first data.

A General Framework for a Joint Calibration of VIX and VXX Options

We analyze the VIX futures market with a focus on the exchange-traded notes written on such contracts, in particular we investigate the VXX notes tracking the short-end part of the futures term structure. Inspired by recent developments in commodity smile modelling, we present a multi-factor stochastic-local volatility model that is able to jointly calibrate plain vanilla options both on VIX futures and VXX notes, thus going beyond the failure of purely stochastic or simply local volatility models. We discuss numerical results on real market data by highlighting the impact of model parameters on implied volatilities.

A High-Quality VR Calibration and Real-Time Stitching Framework Using Preprocessed Features

Virtual Reality (VR) contents include $360^{\circ }\times 180^{\circ }$ seamless panoramic videos, stitched from multiple overlapping video streams. Many commercial VR devices use a two-camera rig to capture VR content. These devices have increased radial distortion effects along the seams of the stitching boundary. Moreover, a fixed number of cameras in the camera rig makes the VR system non-scalable. Since the VR experience is directly related to the quality of VR content, it is desirable to create a VR framework that is scalable in terms of the number of cameras attached to the camera-rig and has better geometric and photometric quality. In this paper, we propose an end-to-end VR system for stitching full spherical content. The VR system is composed of camera rig calibration and stitching modules. The calibration module performs a geometric alignment of the camera rig. The stitching module transforms texture from the camera or video stream into the VR stream using lookup tables (LUTs) and blend masks (BMs). In this work, our main contribution is the improvement of stitching quality. First, we propose a feature preprocessing method that filters out inconsistent, error-prone features. Secondly, we propose a geometric alignment method that outperforms state-of-the-art VR stitching solutions. We tested our system on diverse image sets and obtained state-of-the-art geometric alignment. Moreover, we achieved real-time stitching of camera and video streams up to 120 fps at 4K resolution. After stitching, we encode VR content for IP multicasting.

Improving the Predictive Skill of a Distributed Hydrological Model by Calibration on Spatial Patterns With Multiple Satellite Data Sets

AbstractHydrological model calibration combining Earth observations and in situ measurements is a promising solution to overcome the limitations of the traditional streamflow‐only calibration. However, combining multiple data sources in model calibration requires a meaningful integration of the data sets, which should harness their most reliable contents to avoid accumulation of their uncertainties and mislead the parameter estimation procedure. This study analyzes the improvement of model parameter selection by using only the spatial patterns of satellite remote sensing data, thereby ignoring their absolute values. Although satellite products are characterized by uncertainties, their most reliable key feature is the representation of spatial patterns, which is a unique and relevant source of information for distributed hydrological models. We propose a novel multivariate calibration framework exploiting spatial patterns and simultaneously incorporating streamflow and three satellite products (i.e., Global Land Evaporation Amsterdam Model [GLEAM] evaporation, European Space Agency Climate Change Initiative [ESA CCI] soil moisture, and Gravity Recovery and Climate Experiment [GRACE] terrestrial water storage). The Moderate Resolution Imaging Spectroradiometer (MODIS) land surface temperature data set is used for model evaluation. A bias‐insensitive and multicomponent spatial pattern matching metric is developed to formulate a multiobjective function. The proposed multivariate calibration framework is tested with the mesoscale Hydrologic Model (mHM) and applied to the poorly gauged Volta River basin located in a predominantly semiarid climate in West Africa. Results of the multivariate calibration show that the decrease in performance for streamflow (−7%) and terrestrial water storage (−6%) is counterbalanced with an increase in performance for soil moisture (+105%) and evaporation (+26%). These results demonstrate that there are benefits in using satellite data sets, when suitably integrated in a robust model parametrization scheme.

Bayesian calibration and sensitivity analysis of heat transfer models for fire insulation panels

A common approach to assess the performance of fire insulation panels is the component additive method (CAM). The parameters of the CAM are based on the temperature-dependent thermal material properties of the panels. These material properties can be derived by calibrating finite element heat transfer models using experimentally measured temperature records. In the past, the calibration of the material properties was done manually by trial and error approaches, which was inefficient and prone to error.In this contribution, the calibration problem is reformulated in a probabilistic setting and solved using the Bayesian model calibration framework. This not only gives a set of best-fit parameters but also confidence bounds on the latter. To make this framework feasible, the procedure is accelerated through the use of advanced surrogate modelling techniques: polynomial chaos expansions combined with principal component analysis. This surrogate modelling technique additionally allows one to conduct a variance-based sensitivity analysis at no additional cost by giving access to the Sobol’ indices. The calibration is finally validated by using the calibrated material properties to predict the temperature development in different experimental setups.

A Multisignal Wavelet Variance-Based Framework for Inertial Sensor Stochastic Error Modeling

The calibration of low-cost inertial sensors has become increasingly important over the last couple of decades, especially when dealing with sensor stochastic errors. This procedure is commonly performed on a single error measurement from an inertial sensor taken over a certain amount of time, although it is extremely frequent for different replicates to be taken for the same sensor, thereby delivering important information which is often left unused. In order to address the latter problem, this paper presents a general wavelet variance-based framework for multisignal inertial sensor calibration, which can improve the modeling and model selection procedures of sensor stochastic errors using all replicates from a calibration procedure and allows to understand the properties, such as stationarity, of these stochastic errors. The applications using microelectromechanical system inertial measurement units confirm the importance of this new framework, and a new graphical user interface makes these tools available to the general user. The latter is developed based on an R package called mgmwm and allows the user to select a type of sensor for which different replicates are available and to easily make use of the approaches presented in this paper in order to carry out the appropriate calibration procedure.

A novel framework for a rational, fully-automatised calibration routine for DEM models of cohesive powders

A new framework for calibration of DEM models of cohesive materials is presented. DEM simulations are a widespread numerical tool for modelling granular assemblies and related processes, but require careful calibration to give realistic predictive results. To this aim routine small-scale laboratory tests are used to investigate powders rheology while limiting resource investment, in conjunction with a rational approach to reduce computational costs as well. A specific order in numerical tests and calibrated variables is defined, enabling the implementation of an automatic iterative routine to calibrate material parameters from a given dataset. This method is successfully applied to calibrate four common increasingly cohesive powders of pharmaceutical relevance. Our framework saves an important amount of computational expense when compared to traditional parameters design space approaches, and proved to be able to calibrate very cohesive powder exhibiting plastic bulk volume loss of up to 25% (after uniaxial consolidation up to 10kPa).

Automatic Multi-Camera Extrinsic Parameter Calibration Based on Pedestrian Torsors.

Extrinsic camera calibration is essential for any computer vision task in a camera network. Typically, researchers place a calibration object in the scene to calibrate all the cameras in a camera network. However, when installing cameras in the field, this approach can be costly and impractical, especially when recalibration is needed. This paper proposes a novel, accurate and fully automatic extrinsic calibration framework for camera networks with partially overlapping views. The proposed method considers the pedestrians in the observed scene as the calibration objects and analyzes the pedestrian tracks to obtain extrinsic parameters. Compared to the state of the art, the new method is fully automatic and robust in various environments. Our method detect human poses in the camera images and then models walking persons as vertical sticks. We apply a brute-force method to determines the correspondence between persons in multiple camera images. This information along with 3D estimated locations of the top and the bottom of the pedestrians are then used to compute the extrinsic calibration matrices. We also propose a novel method to calibrate the camera network by only using the top and centerline of the person when the bottom of the person is not available in heavily occluded scenes. We verified the robustness of the method in different camera setups and for both single and multiple walking people. The results show that the triangulation error of a few centimeters can be obtained. Typically, it requires less than one minute of observing the walking people to reach this accuracy in controlled environments. It also just takes a few minutes to collect enough data for the calibration in uncontrolled environments. Our proposed method can perform well in various situations such as multi-person, occlusions, or even at real intersections on the street.

Automatic real‐time calibration, assessment, and maintenance of generic Raman models for online monitoring of cell culture processes

Raman spectroscopy is a multipurpose analytical technology that has found great utility in real-time monitoring and control of critical performance parameters of cell culture processes. As a process analytical technology (PAT) tool, the performance of Raman spectroscopy relies on chemometric models that correlate Raman signals to the parameters of interest. The current calibration techniques yield highly specific models that are reliable only on the operating conditions they are calibrated in. Furthermore, once models are calibrated, it is typical for the model performance to degrade over time due to various recipe changes, raw material variability, and process drifts. Maintaining the performance of industrial Raman models is further complicated due to the lack of a systematic approach to assessing the performance of Raman models. In this article, we propose a real-time just-in-time learning (RT-JITL) framework for automatic calibration, assessment, and maintenance of industrial Raman models. Unlike traditional models, RT-JITL calibrates generic models that can be reliably deployed in cell culture experiments involving different modalities, cell lines, media compositions, and operating conditions. RT-JITL is a first fully integrated and fully autonomous platform offering a self-learning approach for calibrating and maintaining industrial Raman models. The efficacy of RT-JITL is demonstrated on experimental studies involving real-time predictions of various cell culture performance parameters, such as metabolite concentrations, viability, and viable cell density. RT-JITL framework introduces a paradigm shift in the way industrial Raman models are calibrated, assessed, and maintained, which to the best of authors' knowledge, have not been done before.

Automatic calibration of a whole-of-basin water accounting model using a comprehensive learning particle swarm optimiser

We present a two-step framework for calibrating complex, many-parameter hydrological models at basin-scale. The framework first calibrates parameters for each catchment/sub-basin sequentially and then fine-tunes parameters as needed. We implemented a comprehensive learning particle swarm optimiser (CLPSO) as the calibrator and applied the two-step CLPSO tool in calibrating parameters of a water accounting model for the Murray-Darling Basin, Australia. The visual and quantitative results indicated that our tool produced satisfactory calibration and prediction outcomes for the model’s intended purpose. The comparison experiments demonstrated that the calibration framework and the CLPSO were competent in calibrating large-scale hydrological models. This framework can guarantee spatial coherence, balance objective trade-offs among all catchments, and calibrate many parameters at a low computational cost. By providing better parameter estimates in complex whole-of-basin hydrological models, our calibration tool has the potential to increase the development and application of these models, and thereby improve the management of large river basins.

Multifidelity and Multiscale Bayesian Framework for High-Dimensional Engineering Design and Calibration

AbstractThis paper proposes a machine learning–based multifidelity modeling (MFM) and information-theoretic Bayesian optimization approach where the associated models can have complex discrepancies among each other. Advantages of MFM-based optimization over a single-fidelity surrogate, specifically under complex constraints, are discussed with benchmark optimization problems involving noisy data. The MFM framework, based on modeling of the varied fidelity information sources via Gaussian processes, is augmented with information-theoretic active learning strategies that involve sequential selection of optimal points in a multiscale architecture. This framework is demonstrated to exhibit improved efficiency on practical engineering problems like high-dimensional design optimization of compressor rotor via implementing its multiscale architecture and calibration of expensive microstructure prediction model. From the perspective of the machine learning–assisted design of multiphysics systems, advantages of the proposed framework have been presented with respect to accelerating the search of optimal design conditions under budget constraints.

Calibration of linear contact stiffnesses in discrete element models using a hybrid analytical-computational framework

Efficient selections of particle-scale contact parameters in discrete element modelling remain an open question. The aim of this study is to provide a hybrid calibration framework to estimate linear contact stiffnesses (normal and tangential) for both two-dimensional and three-dimensional simulations. Analytical formulas linking macroscopic parameters (Young's modulus, Poisson's ratio) to mesoscopic particle parameters for granular systems are derived based on statistically isotropic packings under small-strain isotropic stress conditions. By taking the derived analytical solutions as initial approximations, the gradient descent algorithm automatically obtains a reliable numerical estimation. The proposed framework is validated with several numerical cases including randomly distributed monodisperse and polydisperse packings. The results show that this hybrid method practically reduces the time for artificial trials and errors to obtain reasonable stiffness parameters. The proposed framework can also be extended to other parameter calibration problems in DEM.

A feasible experimental framework for field calibration of portable light-scattering aerosol monitors: Case of TSI DustTrak

Portable light-scattering aerosol monitors (PLSAMs) can supplement existing air quality monitoring networks through measuring air pollutant exposure concentrations at high spatiotemporal resolution. However, data collected by PLSAMs are often subject to the simplicity of measurement principle which may lead to errors compared to the regulatory data observed at fixed-site air quality monitoring stations. The main objective of this study was to develop a feasible experimental framework to assess the influence of key factors (e.g., relative humidity (RH)) on the performance of PLSAMs in the real-world conditions. Following the proposed framework, the accuracy and precision of the TSI DustTrak aerosol monitor were evaluated through side-by-side comparison with the stationary reference instruments (SRIs) while taking characteristics of particles, RH, and the concentration range into consideration. DustTrak generally demonstrated low accuracy but high precision in measuring PM2.5 concentrations at the two selected stations. Three calibration models between DustTrak and the SRIs were used to bias correct the DustTrak PM2.5 measurements. The RH-adjusted linear regression calibration method led to better calibration results than the simple linear regression method and the RH-adjusted empirical method, with CV R2 values higher than 0.97, root mean square error less than 1.0 μg/m3, and accuracy values at 3% for two DustTraks. The proposed experimental framework can be extended to field calibration of various types of PLSAMs, and the obtained calibration results can promote a more accurate investigation of particle air pollution using these PLSAMs.

A neural network-based framework for financial model calibration

A data-driven approach called CaNN (Calibration Neural Network) is proposed to calibrate financial asset price models using an Artificial Neural Network (ANN). Determining optimal values of the model parameters is formulated as training hidden neurons within a machine learning framework, based on available financial option prices. The framework consists of two parts: a forward pass in which we train the weights of the ANN off-line, valuing options under many different asset model parameter settings; and a backward pass, in which we evaluate the trained ANN-solver on-line, aiming to find the weights of the neurons in the input layer. The rapid on-line learning of implied volatility by ANNs, in combination with the use of an adapted parallel global optimization method, tackles the computation bottleneck and provides a fast and reliable technique for calibrating model parameters while avoiding, as much as possible, getting stuck in local minima. Numerical experiments confirm that this machine-learning framework can be employed to calibrate parameters of high-dimensional stochastic volatility models efficiently and accurately.

Swaption Portfolio Risk Management: Optimal Model Selection in Different Interest Rate Regimes

The authors formulate a risk-based swaption portfolio management framework for a profit-and-loss (P&L) explanation. They analyze the implication of using the right volatility backbone in the pricing model from a hedging perspective and demonstrate the importance of incorporating a stability and robustness measure as part of the calibration process for optimal model selection. They also derive a displaced-diffusion stochastic volatility model with a closed-form analytical expression to handle negative interest rates. Finally, they show that their framework is able to identify the optimal pricing model, which leads to a superior P&L explanation and hedging performance. TOPICS:Risk management, portfolio management/multi-asset allocation, derivatives, factor-based models Key Findings • A holistic, risk-based calibration framework allows one to select the optimal pricing model with superior P&L explanation performance. • A displaced-diffusion stochastic volatility model with closed-form expression provides a means to price swaptions efficiently under both positive and negative interest rate regimes while capturing the volatility backbone. • Using the Herfindahl index to measure the concentration of hedging performance, we show that the optimal model exhibits stability and robustness of model parameters, along with the economy of the explanatory power of daily P&L movement.

EHT-HOPS Pipeline for Millimeter VLBI Data Reduction

Abstract We present the design and implementation of an automated data calibration and reduction pipeline for very long baseline interferometric (VLBI) observations taken at millimeter wavelengths. These short radio wavelengths provide the best imaging resolution available from ground-based VLBI networks such as the Event Horizon Telescope (EHT) and the Global Millimeter VLBI Array (GMVA) but require specialized processing owing to the strong effects from atmospheric opacity and turbulence, as well as the heterogeneous nature of existing global arrays. The pipeline builds on a calibration suite (HOPS) originally designed for precision geodetic VLBI. To support the reduction of data for astronomical observations, we have developed an additional framework for global phase and amplitude calibration that provides output in a standard data format for astronomical imaging and analysis. The pipeline was successfully used toward the reduction of 1.3 mm observations from the EHT 2017 campaign, leading to the first image of a black hole “shadow” at the center of the radio galaxy M87. In this work, we analyze observations taken at 3.5 mm (86 GHz) by the GMVA, joined by the phased Atacama Large Millimeter/submillimeter Array in 2017 April, and demonstrate the benefits from the specialized processing of high-frequency VLBI data with respect to classical analysis techniques.

A Bayesian model calibration framework to evaluate brain tissue characterization experiments

A Bayesian model calibration framework is implemented to determine the posterior parameter distributions of a hyper-viscoelastic constitutive model using mechanical testing data of brain tissue. Despite the fact that brain tissue exhibits large property variability and uncertainty, least squares model calibration remains common practice. It is shown that through Bayesian calibration, the capability of an experimental design to identify parameter values can be quantitatively evaluated. A large range of experimental designs (i.e., loading modes) are considered, including rate dependent, cyclic loading through compression, shear, and tension, as well as shear and compression relaxation loading. An implementation of the nested sampling algorithm is utilized to compute the joint posterior probability distributions of the constitutive parameters based on the data considered. The joint probability distribution of the parameters enables assessment of parameter sensitivity, uncertainty, correlation, and model calibration error as a function of experimental design. By calibrating simultaneously to numerous combinations of loading modes, redundancy in loading mode as well as parameter convergence can be evaluated based on model calibration error and parameter variability. The results of this study verify that the inclusion of more loading modes in the calibration increases the accuracy of parameter identification, while only one or two loading modes may provide almost no information for identifying certain parameters.

A season-specific, multi-site calibration strategy to study the hydrological cycle and the impact of extreme-flow events along an urban-to-agricultural gradient

Extreme-flow events have been a central point of interest in hydrology and recent developments suggest that their role could be critical in the overall functioning of watershed systems. The development of modelling techniques that can effectively reproduce the hydrological dynamics of watersheds during extreme events is one of the challenges of the contemporary modelling practice. In this study, we present a season-specific, multi-site calibration framework that accommodates the variability in the hydrological responses induced by the agricultural landscape changes during different periods of the year. Our case study is the Hamilton Harbour watershed in southern Ontario, Canada, where discharge data from three main tributaries and six gauging stations were available to constrain the Soil and Water Assessment Tool (SWAT) model. Two distinct values are assigned to the Curve Number for antecedent runoff condition II (CN2), relating surface runoff to precipitation as a function of land uses and soil characteristics, in order to reproduce flow patterns during the growing and dormant seasons. Overall, notwithstanding the model performance decline in the most urbanized catchment of Redhill Creek, our season-specific calibration strategy improved the predictive capacity of the model in the predominantly agricultural catchments of Grindstone Creek and Spencer Creek. Counter to our season-specific process characterization, an alternative calibration strategy postulating a differential watershed response, depending on the magnitude of individual precipitation events, appears to fair better in urbanized settings. The water budget displays significant disparities among the different land uses in the Hamilton Harbour watershed. Our analysis also suggests that the runoff generated from urban sites during the growing season (May–October), i.e., the period when the receiving waterbody is most sensitive to eutrophication, is six times higher than the average contribution from pasture, forest, and cropland areas. The potential implications of the on-going land-use and climate-change trends for the water cycle at both local and catchment scales are also discussed.

A multi-objective ensemble approach to hydrological modelling in the UK: an application to historic drought reconstruction

Abstract. Hydrological models can provide estimates of streamflow pre- and post-observations, which enable greater understanding of past hydrological behaviour, and potential futures. In this paper, a new multi-objective calibration method was derived and tested for 303 catchments in the UK, and the calibrations were used to reconstruct river flows back to 1891, in order to provide a much longer view of past hydrological variability, given the brevity of most UK river flow records which began post-1960. A Latin hypercube sample of 500 000 parameterisations for the GR4J model for each catchment were evaluated against six evaluation metrics covering all aspects of the flow regime from high, median, and low flows. The results of the top ranking model parameterisation (LHS1), and also the top 500 (LHS500), for each catchment were used to provide a deterministic result whilst also accounting for parameter uncertainty. The calibrations are generally good at capturing observed flows, with some exceptions in heavily groundwater-dominated catchments, and snowmelt and artificially influenced catchments across the country. Reconstructed flows were appraised over 30-year moving windows and were shown to provide good simulations of flow in the early parts of the record, in cases where observations were available. To consider the utility of the reconstructions for drought simulation, flow data for the 1975–1976 drought event were explored in detail in nine case study catchments. The model's performance in reproducing the drought events was found to vary by catchment, as did the level of uncertainty in the LHS500. The Standardised Streamflow Index (SSI) was used to assess the model simulations' ability to simulate extreme events. The peaks and troughs of the SSI time series were well represented despite slight over- or underestimations of past drought event magnitudes, while the accumulated deficits of the drought events extracted from the SSI time series verified that the model simulations were overall very good at simulating drought events. This paper provides three key contributions: (1) a robust multi-objective model calibration framework for calibrating catchment models for use in both general and extreme hydrology; (2) model calibrations for the 303 UK catchments that could be used in further research, and operational applications such as hydrological forecasting; and (3) ∼ 125 years of spatially and temporally consistent reconstructed flow data that will allow comprehensive quantitative assessments of past UK drought events, as well as long-term analyses of hydrological variability that have not been previously possible, thus enabling water resource managers to better plan for extreme events and build more resilient systems for the future.