Decision Threshold Research Articles

Evaluation of a Model to Target High-risk Psychiatric Inpatients for an Intensive Postdischarge Suicide Prevention Intervention

The months after psychiatric hospital discharge are a time of high risk for suicide. Intensive postdischarge case management, although potentially effective in suicide prevention, is likely to be cost-effective only if targeted at high-risk patients. A previously developed machine learning (ML) model showed that postdischarge suicides can be predicted from electronic health records and geospatial data, but it is unknown if prediction could be improved by adding additional information. To determine whether model prediction could be improved by adding information extracted from clinical notes and public records. Models were trained to predict suicides in the 12 months after Veterans Health Administration (VHA) short-term (less than 365 days) psychiatric hospitalizations between the beginning of 2010 and September 1, 2012 (299 050 hospitalizations, with 916 hospitalizations followed within 12 months by suicides) and tested in the hospitalizations from September 2, 2012, to December 31, 2013 (149 738 hospitalizations, with 393 hospitalizations followed within 12 months by suicides). Validation focused on net benefit across a range of plausible decision thresholds. Predictor importance was assessed with Shapley additive explanations (SHAP) values. Data were analyzed from January to August 2022. Suicides were defined by the National Death Index. Base model predictors included VHA electronic health records and patient residential data. The expanded predictors came from natural language processing (NLP) of clinical notes and a social determinants of health (SDOH) public records database. The model included 448 788 unique hospitalizations. Net benefit over risk horizons between 3 and 12 months was generally highest for the model that included both NLP and SDOH predictors (area under the receiver operating characteristic curve range, 0.747-0.780; area under the precision recall curve relative to the suicide rate range, 3.87-5.75). NLP and SDOH predictors also had the highest predictor class-level SHAP values (proportional SHAP = 64.0% and 49.3%, respectively), although the single highest positive variable-level SHAP value was for a count of medications classified by the US Food and Drug Administration as increasing suicide risk prescribed the year before hospitalization (proportional SHAP = 15.0%). In this study, clinical notes and public records were found to improve ML model prediction of suicide after psychiatric hospitalization. The model had positive net benefit over 3-month to 12-month risk horizons for plausible decision thresholds. Although caution is needed in inferring causality based on predictor importance, several key predictors have potential intervention implications that should be investigated in future studies.

Calculation of decision thresholds according to the standard ISO 11929-3 in case of presence of the peaked background

The method of calculation of the decision threshold with the Least Squares Method, described in the standard ISO 11929, is presented for the case when the sources of peaked background contribute to the peak holding the indication. The decision threshold is calculated from spectral data corresponding to the indication zero; therefore, the observed indication must be removed from the spectrum. When the peaked background is present, the indication completely overlaps with the peaked background, so it can't be unfolded directly. Therefore, two steps are needed in the calculation: the unfolding of the peak, housing the indication, from the continuous background and the possible overlapping peaks, and separating of the indication from the peaked background using the background data obtained from separate calculations and measurements. In this article it is shown that the method of least squares is flexible enough to accommodate all sources of uncertainty into the uncertainty matrix of input quantities. Its derivation is presented in detail and the calculation of the indication corresponding to the decision threshold is described. As a proof of the concept an example of calculating the number of counts corresponding to the decision threshold as a function of the indication is presented. The method of calculation and the results of the calculation are briefly discussed.

Revisiting the color-motion asynchrony.

Color-motion asynchrony (CMA) refers to an illusion in which we perceive a change in color earlier than a change in motion direction when the two changes occur simultaneously. This phenomenon may indicate that color is processed earlier than motion in the visual system. However, the very existence of CMA is under question owing to contradictory findings and methodological deficits in previous studies. Here, we used both the motion and color correspondence tasks (experiment 1) and the temporal order judgment (TOJ) task (experiment 2) to re-examine CMA. Colored dots moved in one direction and changed their color/direction at some time, whereas the relative timing between color and direction changes varied across trials. In the correspondence task, participants reported which direction/color of dots with a particular color/direction lasted longer, the one before or after the change? In the TOJ task, participants reported whether the change in color or the change in motion direction occurred earlier. Results indicated that participants perceived the change in color earlier than the change in motion direction in either the motion or color correspondence task, with a stronger asynchrony in the former. In the TOJ task, although participants showed no difference in psychophysical measures, they responded faster when the change in color occurred before (versus after) the change in direction. Drift-diffusion modeling (DDM) revealed a lower decision threshold when the change in color occurred before (versus after) the change in direction, indicating less cautiousness was excised in judgment when the color changed earlier. These results confirmed the veracity of CMA in different tasks and point to the viability of analyzing response times in traditional psychophysical studies.

Evidence That Event Boundaries Are Access Points for Memory Retrieval

When recalling memories, we often scan information-rich continuous episodes, for example, to find our keys. How does our brain access and search through those memories? We suggest that high-level structure, marked by event boundaries, guides us through this process: In our computational model, memory scanning is sped up by skipping ahead to the next event boundary upon reaching a decision threshold. In adult Mechanical Turk workers from the United States, we used a movie (normed for event boundaries; Study 1, N = 203) to prompt memory scanning of movie segments for answers (Study 2, N = 298) and mental simulation (Study 3, N = 100) of these segments. Confirming model predictions, we found that memory-scanning times varied as a function of the number of event boundaries within a segment and the distance of the search target to the previous boundary (the key diagnostic parameter). Mental simulation times were also described by a skipping process with a higher skipping threshold than memory scanning. These findings identify event boundaries as access points to memory.

Widespread adoption of AI risk models for maternal hemorrhage requires mitigation of bias

Accurate risk stratification for maternal hemorrhage could lead to improved outcomes. However, the performance of risk stratification tools across populations has not been extensively explored. Therefore, we aim to examine the fairness of machine learning methods for predicting the risk of maternal hemorrhage. A retrospective study was performed using the U.S. Consortium for Safe Labor (2002–2008) dataset. Severe maternal hemorrhage was defined based on receiving at least 1 unit of blood transfusion. The Random Forest algorithm was adopted to model a patient’s risk using eighty-six clinical variables. Model training was done using patients’ data from 2002 - 2006 and was tested on the following two years while excluding patients with missing transfusion records. The model evaluation was conducted using the area under the receiver operating characteristic (AUROC) and precision-recall curves(AUPRC). In addition, following the recommendations from the Aequitas fairness group, we assessed the bias using: (1) equality of opportunity criteria that aims to match errors in the hemorrhage predictions outcomes for the different subgroups, (2) positive predictive value rate that aims to evaluate whether the predicted outcome equally matches the observed outcome for all subgroups. Of the 142,137 births, 3.7% required at least 1 unit of blood transfusion. The model had good performance concerning AUROC (0.80) and AUPRC (0.11), representing a nearly three-fold improvement compared to the prior. However, when comparing the model performance and fairness metrics with respect to race (AUROC: W:.75, B:0.83, H:0.75, A:0.85, AUPRC: W:0.11 B:0.11, H:0.08, A:0.06), the model underperformed in minority populations compared to the white patients as the reference(Figure 1). The Widespread adoption of machine learning-based risk assessments requires careful evaluation of their performance across various sub-populations, choosing appropriate decision thresholds and developing methods for mitigating potential bias.View Large Image Figure ViewerDownload Hi-res image Download (PPT)

Optimal strategy of taxi drivers at airports by a stochastic programming approach: Evidence from airports in China

It is applicable to address how taxi drivers make optimal choices at airports by mathematical modelling owing to its impacts of service quality. This study formulates such a problem by a new stochastic 0-1 integer programming model such that the research gap of the literature is filled. Specifically, when net gains of the taxi service in urban areas are random, the downside semi-deviation risk of the total profit is considered in this model, and on the basis of its property analysis, a number of useful decision thresholds are derived. A time threshold associated with price differences is defined and when it is greater than the required actual time of waiting for passengers, waiting is the optimal choice. Appropriate conditions are given to answer when the decision-making mode of the developed model is either risk-taking or risk-averse. The derived expected-profit-maximization and risk-minimization model by expectation method is proved to be risk-neutral. Main findings include: The provided thresholds are of easy use for taxi drivers to earn the maximum profit, while the stochastic model is more applicable; Optimal strategy of taxi drivers at airports depends on the service periods, the needed time of driving on freeway, the relative difference of net gains earned by driving on freeway and in city, and the critical number of queuing taxis at airports; By our model, different optimal decision in line with service periods are provided; Except two earlier periods, shortening expended time of boarding a taxi can greatly attract taxi drivers' waiting at airports.

Optimizing a de novo artificial intelligence-based medical device under a predetermined change control plan: Improved ability to detect or rule out pediatric autism

A growing number of artificial intelligence-based medical devices are receiving clearance from the Food and Drug Administration (FDA). Debate has arisen about best practices for the regulation and safe oversight of such devices whose capabilities, if “unlocked”, include iterative learning and adaptation with exposure to new data. One regulatory mechanism proposed by the FDA is the predetermined change control plan (PCCP). This analysis provides an example of how a PCCP could be leveraged to improve the performance of a de novo autism diagnostic device in practice. Following model update procedures drafted in the marketing authorization of the first generation of the device (“algorithm V1”), we conducted an algorithmic threshold optimization procedure to improve the device's ability to detect or rule out autism in children ages 18–72 months without changing the accuracy or intended use of the device. Decision threshold optimization was achieved using a repeated train/test validation procedure on a dataset of 722 children with concern for developmental delay, aged 18–72 months (28% autism, 22% neurotypical, 50% other developmental delay, mean age 3.6 years, 39% female). In 1000 repeats, 70% of samples were selected for threshold optimization and 30% for evaluation, ensuring that no training data appeared in the test set. Out-of-sample performance was estimated by evaluating the selected threshold pair on the test set and comparing the performance metrics of the new pair to the corresponding V1 metrics on the same test set. The device, with optimized decision thresholds, produced a determinate output for 66.5% (95% CI, 62.5–71.0) of children. Positive Predictive Value (PPV) and Negative Predictive Value (PPV) were 87.5% (95% CI, 82.5–96.7) and 95.6% (95% CI, 93.7–97.9) respectively. Threshold optimization improved the device's ability to accurately detect or rule out autism in a greater proportion of children. Given the current waitlist crisis for autism evaluations in the United States, the potential increase in coverage offered by the optimized thresholds is promising and emphasizes the value of regulatory mechanisms that allow software as medical devices to adapt safely and appropriately given real world data.

Improving performance of decision threshold moving-based strategies by integrating density-based clustering technique

<abstract> <p>Class imbalance learning (CIL), which aims to addressing the performance degradation problem of traditional supervised learning algorithms in the scenarios of skewed data distribution, has become one of research hotspots in fields of machine learning, data mining, and artificial intelligence. As a postprocessing CIL technique, the decision threshold moving (DTM) has been verified to be an effective strategy to address class imbalance problem. However, no matter adopting random or optimal threshold designation ways, the classification hyperplane could be only moved parallelly, but fails to vary its orientation, thus its performance is restricted, especially on some complex and density variable data. To further improve the performance of the existing DTM strategies, we propose an improved algorithm called CDTM by dividing majority training instances into multiple different density regions, and further conducting DTM procedure on each region independently. Specifically, we adopt the well-known DBSCAN clustering algorithm to split training set as it could adapt density variation well. In context of support vector machine (SVM) and extreme learning machine (ELM), we respectively verified the effectiveness and superiority of the proposed CDTM algorithm. The experimental results on 40 benchmark class imbalance datasets indicate that the proposed CDTM algorithm is superior to several other state-of-the-art DTM algorithms in term of G-mean performance metric.</p> </abstract>

ИССЛЕДОВАНИЕ ПРОЦЕССА ФУНКЦИОНИРОВАНИЯ КОРРЕЛЯЦИОННОГО ПРИЕМНИКА С КВАДРАТУРНЫМИ КАНАЛАМИ АВТОМАТИЧЕСКОЙ ЛОКОМОТИВНОЙ СИГНАЛИЗАЦИИ НЕПРЕРЫВНОГО ТИПА В НОМИНАЛЬНЫХ УСЛОВИЯХ ЭКСПЛУАТАЦИИ

Development and implementation of receivers of CWACS channels on a microelectronic elemental base with digital processing of signals allowed using complicated methods (for example, correlation, non-linear) that couldn’t be realized on an analogue elemental base. One of the devices, in which the similar methods are applied, is a correlation receiver with quadrature channels. The paper considers its operation in nominal conditions. The study was carried out by simulation modeling in the Simulink environment. An experiment on the model showed that the normative operation of the receiver in some cases is violated because of the imperfection of an algorithm for calculation of a floating threshold of decision making. Adjustment of the algorithm provided correct operation of the receiver in the considered conditions. Moreover, the authors have revealed that real sensitivity of the receiver match normative requirements.

Fast Adaptive Spectrum Sensing Using Hardware Optimized, Cell Averaging Estimation for Cognitive Radio and Radar Applications

This article describes a technique that enables real-time spectrum sharing by rapidly sensing the spectrum and selecting available frequencies that a system may use for operation. Hardware optimized, cell averaging estimation (HO-CAE) is a method that uses a snapshot of the spectral environment to calculate a dynamic decision threshold that can be used by the fast spectrum sensing (FSS) algorithm to detect available spectrum. This energy detection technique is adaptive as it calculates a new threshold for each spectrum snapshot. A statistical derivation of HO-CAE will be provided with an analysis of probability of false alarm and probability of detection performance for noise only cases and in the presence of a 5 G downlink signal. Efficient field programmable gate array (FPGA) implementations of HO-CAE and FSS that enable real-time operation are presented.

A method for quantifying individual decision thresholds of latent print examiners

In recent years, ‘black box’ studies in forensic science have emerged as the preferred way to provide information about the overall validity of forensic disciplines in practice. These studies provide aggregated error rates over many examiners and comparisons, but errors are not equally likely on all comparisons. Furthermore, inconclusive responses are common and vary across examiners and comparisons, but do not fit neatly into the error rate framework. This work introduces Item Response Theory (IRT) and variants for the forensic setting to account for these two issues. In the IRT framework, participant proficiency and item difficulty are estimated directly from the responses, which accounts for the different subsets of items that participants often answer. By incorporating a decision-tree framework into the model, inconclusive responses are treated as a distinct cognitive process, which allows inter-examiner differences to be estimated directly. The IRT-based model achieves superior predictive performance over standard logistic regression techniques, produces item effects that are consistent with common sense and prior work, and demonstrates that most of the variability among fingerprint examiner decisions occurs at the latent print evaluation stage and as a result of differing tendencies to make inconclusive decisions.

Deep learning assisted holography microscopy for in-flow enumeration of tumor cells in blood.

Currently, detection of circulating tumor cells (CTCs) in cancer patient blood samples relies on immunostaining, which does not provide access to live CTCs, limiting the breadth of CTC-based applications. Here, we take the first steps to address this limitation, by demonstrating staining-free enumeration of tumor cells spiked into lysed blood samples using digital holographic microscopy (DHM), microfluidics and machine learning (ML). A 3D-printed module for laser assembly was developed to simplify the optical set up for holographic imaging of cells flowing through a sheath-based microfluidic device. Computational reconstruction of the holograms was performed to localize the cells in 3D and obtain the plane of best focus images to train deep learning models. We developed a custom-designed light-weight shallow Network dubbed s-Net and compared its performance against off-the-shelf CNN models including ResNet-50. The accuracy, sensitivity and specificity of the s-Net model was found to be higher than the off-the-shelf ML models. By applying an optimized decision threshold to mixed samples prepared in silico, the false positive rate was reduced from 1 × 10-2 to 2.77 × 10-4. Finally, the developed DHM-ML framework was successfully applied to enumerate spiked MCF-7 breast cancer cells and SkOV3 ovarian cancer cells from lysed blood samples containing white blood cells (WBCs) at concentrations typical of label-free enrichment techniques. We conclude by discussing the advances that need to be made to translate the DHM-ML approach to staining-free enumeration of actual CTCs in cancer patient blood samples.

Acquisition Time and Probabilities of Detection and False Alarm in Direct Sequence Code Division Multiple Access Systems

Multiple access interference (MAI) is one of the most important issues to consider in the design, implementation and operation of 5G systems using code division multiple access (CDMA) technique based on spread spectrum that offers a more flexible solution, compared to other techniques. Especially, it permits a much higher user bit rate, due to the bandwidth allocated to the emitted signal; thus, allowing very attractive multimedia services. In this paper, we develop the expressions of the detection probability, false alarm probability and the average acquisition time - in a direct sequence CDMA (DS-CDMA) transmission system - using a serial search and taking into consideration MAI. The evolution of these parameters according to the signal- to-noise ratio (SNR) and the decision threshold leads to study performances of the system in terms of detection and average acquisition time. The obtained results reveal that the acquisition time decreases when the SNR increases. Additionally, it is found that the probability of detection increases when the threshold decreases, and increasing SNR it reaches a constant value for an SNR/chip around 0 dB for all thresholds.

Dictionary Attacks on Speaker Verification

In this paper, we propose dictionary attacks against speaker verification-a novel attack vector that aims to match a large fraction of speaker population by chance. We introduce a generic formulation of the attack that can be used with various speech representations and threat models. The attacker uses adversarial optimization to maximize raw similarity of speaker embeddings between a seed speech sample and a proxy population. The resulting master voice successfully matches a non-trivial fraction of people in an unknown population. Adversarial waveforms obtained with our approach can match on average 69% of females and 38% of males enrolled in the target system at a strict decision threshold calibrated to yield false alarm rate of 1%. By using the attack with a black-box voice cloning system, we obtain master voices that are effective in the most challenging conditions and transferable between speaker encoders. We also show that, combined with multiple attempts, this attack opens even more to serious issues on the security of these systems.

Using Artificial Neural Networks to Couple Satellite C-Band Synthetic Aperture Radar Interferometry and Alpine3D Numerical Model for the Estimation of Snow Cover Extent, Height, and Density

This work presents a new approach for the estimation of snow extent, height and density in complex orography regions, which combines differential interferometric synthetic-aperture-radar (DInSAR) data and snowpack numerical model data through artificial neural networks (ANNs). The estimation method, subdivided into classification and estimation, is based on two artificial neural networks trained by a DInSAR response model coupled with Alpine3D snow cover numerical model outputs. Auxiliary satellite training data from satellite visible-infrared MODIS imager as well as digital elevation and land cover models are used to discriminate wet and dry snow areas. For snow cover classification the ANN-based estimation methodology is combined with fuzzy-logic and compared with a consolidated decision threshold approach using C-band SAR backscattering information. For snow height and density estimation, the proposed methodology is compared with an analytical inverse method and two model-based statistical techniques (linear regression and maximum likelihood). The validation is carried out in Central Apennines, a mountainous area in Italy with an extension of about 104 km <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> and peaks up to 2912 m, using in situ data collected between December 2018 and February 2019. Results show that the ANN-based technique has a snow cover area classification accuracy of more than 80% when compared MODIS maps. Estimation bias and root mean square error are equal to about 0.5 cm and 20 cm for snow height and to 5 kg/m <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> and 80 kg/m <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> for snow density. As expected, worse results are associated to low DInSAR coherence between two repeat passes and to snow melting periods.

Analytical Study and Noise Mitigation for Complex-Valued Optical DSB Transmission With Carrier-Assisted Differential Detection Receiver

A thorough analysis of the system model for complex-valued optical double-sideband (DSB) transmission with carrier-assisted differential detection (CADD) receiver under the influence of laser phase noise and fiber chromatic dispersion (CD) is presented. It is shown that the interaction of laser phase noise and CD leads to the phenomenon of phase-to-amplitude (P2A) noise and equalization-enhanced phase noise (EEPN). The achievable performance of the complex-valued optical DSB transmission with CADD receiver under the influence of EEPN and P2A noise is estimated by a closed-form expression, and it is verified by simulation that the results show an inaccuracy of less than 1 dB for a wide range of system parameters. In addition, in order to mitigate the EEPN and P2A noise, we propose a pre-decision aid-based simplified blind phase search (BPS) algorithm, commonly referred to as PDA-BPS algorithm. The PDA-BPS algorithm allows for reduced complexity by decreasing the number of symbols entering the BPS algorithm. Theoretical derivation and simulation verification illustrate that the PDA-BPS algorithm can achieve similar mitigation performance to the BPS algorithm with lower complexity. Specifically, under the condition that classification decision threshold R is equal to 0.25, the complexity can be reduced by 40%/16.6% with 1 MHz/7 MHz linewidth for 16-QAM and 29%/10.9% with 0.5 MHz/3 MHz linewidth for 64-QAM.

Human optional stopping in a heteroscedastic world.

When making decisions, animals must trade off the benefits of information harvesting against the opportunity cost of prolonged deliberation. Deciding when to stop accumulating information and commit to a choice is challenging in natural environments, where the reliability of decision-relevant information may itself vary unpredictably over time (variable variance or "heteroscedasticity"). We asked humans to perform a categorization task in which discrete, continuously valued samples (oriented gratings) arrived in series until the observer made a choice. Human behavior was best described by a model that adaptively weighted sensory signals by their inverse prediction error and integrated the resulting quantities with a linear urgency signal to a decision threshold. This model approximated the output of a Bayesian model that computed the full posterior probability of a correct response, and successfully predicted adaptive weighting of decision information in neural signals. Adaptive weighting of decision information may have evolved to promote optional stopping in heteroscedastic natural environments. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Speed-accuracy tradeoffs in decision making: Perception shifts and goal activation bias decision thresholds.

A fundamental aspect of decision making is the speed-accuracy tradeoff (SAT): slower decisions tend to be more accurate, but because time is a scarce resource people prefer to conclude decisions more quickly. The current research adds to the SAT literature by documenting two previously unrecognized influences on the SAT: perception shifts and goal activation. Decision makers' perceptions of what constitutes a fast or a slow decision, and what constitutes an accurate or inaccurate decision, are based on prior experience, and these perceptions influence decision speed. Similarly, previous experience in a decision context associates the context with a particular decision goal. Thus, in later decisions the decision context will activate this goal, and influence decision speed. Both of these mechanisms contribute to a specific decision bias: decision speeds are biased toward original decision speeds in a decision context. Four experiments provide evidence for the bias and the two contributing mechanisms. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Mixed-attitude three-way decision model for aerial targets: Threat assessment based on IF-VIKOR-GRA method.

Assessing potential threats typically necessitates the use of a robust mathematical model, a comprehensive evaluation method and universal decision rules. A novel approach is utilized in this study to optimize existing threat assessment (TA) algorithms and three-way decision models (3WDMs) are leveraged that incorporate decision-theoretic rough sets (DTRSs) within dynamic intuitionistic fuzzy (IF) environments to create a mixed-attitude 3WDM based on the IF-VIKOR-GRA method in the context of aviation warfare. The primary objectives of this study include determining conditional probabilities for IF three-way decisions (3WDs) and establishing mixed-attitude decision thresholds. Both the target attribute and loss function are expressed in the form of intuitionistic fuzzy numbers (IFNs). To calculate these conditional probabilities, an IF technique is used to combine the multi-attribute decision-making (MADM) method VIKOR and the grey relational analysis (GRA) method, while also taking into account the risk-related preferences of decision-makers (DMs). Optimistic and pessimistic 3WDMs are developed from the perspectives of membership degree and non-membership degree, then subsequently integrated into the comprehensive mixed-attitude 3WDM. The feasibility and effectiveness of this methodology are demonstrated through a numerical example and by comparison to other existing approaches.

Deep ROC Analysis and AUC as Balanced Average Accuracy, for Improved Classifier Selection, Audit and Explanation.

Optimal performance is desired for decision-making in any field with binary classifiers and diagnostic tests, however common performance measures lack depth in information. The area under the receiver operating characteristic curve (AUC) and the area under the precision recall curve are too general because they evaluate all decision thresholds including unrealistic ones. Conversely, accuracy, sensitivity, specificity, positive predictive value and the F1 score are too specific-they are measured at a single threshold that is optimal for some instances, but not others, which is not equitable. In between both approaches, we propose deep ROC analysis to measure performance in multiple groups of predicted risk (like calibration), or groups of true positive rate or false positive rate. In each group, we measure the group AUC (properly), normalized group AUC, and averages of: sensitivity, specificity, positive and negative predictive value, and likelihood ratio positive and negative. The measurements can be compared between groups, to whole measures, to point measures and between models. We also provide a new interpretation of AUC in whole or part, as balanced average accuracy, relevant to individuals instead of pairs. We evaluate models in three case studies using our method and Python toolkit and confirm its utility.