Desired Confidence Research Articles

A fast bootstrap algorithm for causal inference with large data.

Estimating causal effects from large experimental and observational data has become increasingly prevalent in both industry and research. The bootstrap is an intuitive and powerful technique used to construct standard errors and confidence intervals of estimators. Its application however can be prohibitively demanding in settings involving large data. In addition, modern causal inference estimators based on machine learning and optimization techniques exacerbate the computational burden of the bootstrap. The bag of little bootstraps has been proposed in non-causal settings for large data but has not yet been applied to evaluate the properties of estimators of causal effects. In this article, we introduce a new bootstrap algorithm called causal bag of little bootstraps for causal inference with large data. The new algorithm significantly improves the computational efficiency of the traditional bootstrap while providing consistent estimates and desirable confidence interval coverage. We describe its properties, provide practical considerations, and evaluate the performance of the proposed algorithm in terms of bias, coverage of the true 95% confidence intervals, and computational time in a simulation study. We apply it in the evaluation of the effect of hormone therapy on the average time to coronary heart disease using a large observational data set from the Women's Health Initiative.

Model Selection Path and Construction of Model Confidence Set under High-Dimensional Variables

Model selection uncertainty has drawn a lot of attention from academics recently because it significantly affects parameter estimation and prediction. Scholars are currently addressing and quantifying uncertainty in model selection by concentrating on model combining and model confidence sets. In this paper, we present a new approach for building model confidence sets, which we call AMac. We provide a theoretical lower bound on the degree of confidence in the model confidence sets that AMac has built. Furthermore, we discuss how the implementation of current model confidence set construction methods becomes difficult when dealing with high-dimensional variables. To address this problem, we suggest building model selection paths (MSP) as a solution. We develop an algorithm for building MSP and show its effectiveness by utilizing the theories of adaptive lasso and lars. We perform an extensive set of simulation experiments to compare the performances of Mac and AMac methods. According to the results, AMac is more stable when there are fluctuations in noise levels. The model confidence sets built by AMac, in particular, achieve coverage rates that are closer to the desired confidence level, especially in the presence of high noise levels. To further confirm that MSP can successfully generate model confidence sets that maintain the given confidence level as the sample size increases, we conduct extensive simulation tests with high-dimensional variables. Ultimately, we hope that the strategies and concepts discussed in this work will improve results in subsequent research on the uncertainty of model selection.

Climate-Adaptive Transmission Network Expansion Planning Considering Evolutions of Resources

Weather-sensitive resources are the main source of uncertainties in power systems. However, the unpredictable climate change further introduces ambiguity (i.e., unknown probability distribution) into the system, since the weather-sensitive resources would evolve with the climate and gradually exhibit a different probability distribution from the past in an uncertain manner. Lack of considering this climate-induced ambiguity in transmission network expansion planning (TNEP) may cause misunderstanding of future operational scenarios. Aiming at a higher security operation level under climate change yet less line investment, this paper proposes a climate-adaptive TNEP, which is essentially a robust TNEP equipped with a climate-adaptive uncertainty set (CUS) to embody injections from the weather-sensitive resources that have high probabilities in the target year despite the climate-induced ambiguity. Determination of the CUS involves three steps. First, model future unknown distribution under climate change. Specifically, the climate-driven evolution in distributions is quantified by an evolutionary distance between historical and future true distributions, whose upper bound is derived from practical data, while the future unknown distribution is then modeled by a distance-based ambiguity set; Second, determine the CUS which has a minimal volume yet a desired confidence level in the face of the ambiguous future distribution. To that end, a parametric Wasserstein distance-based distributionally robust optimization ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">p</i> -WDRO) is developed over the ambiguity set; Third, solve the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">p</i> -WDRO by a data-clustering-incorporated reformulation. After the CUS is determined, the overall climate-adaptive TNEP is solved by a column-and-constraint-generation method with an inner multi-loop algorithm tailored for the CUS. Simulations are conducted on three test systems with data from Australia and Coupled Model Intercomparison Project Phase 6 (CMIP6) under scenarios issued by Intergovernmental Panel on Climate Change (IPCC), which demonstrate that the climate-adaptive TNEP can improve operational security under climate change while reducing investment costs.

Integrated process design and self-optimizing control: A probabilistic approach

This work introduces a probabilistic-based integrated process design and self-optimizing control approach grounded in a steady-state model, aiming to strike a balance between solution reliability and optimality. The proposed approach offers a practical and cost-effective solution by considering disturbances as random variables with known probability distribution functions. To propagate these disturbances to process outputs, the unscented transform method is utilized alongside rigorous first-principles process models. Additionally, probability levels are assigned to process constraints based on user-defined specifications, providing flexibility to tailor the design to desired risk confidence levels. The effectiveness and benefits of the approach are demonstrated through an application on an extractive distillation process with pre-concentration.

Nonparametric bootstrap methods for interval estimation of the area under the ROC curve with correlated diagnostic test data: application to whole-virus ELISA testing in swine.

Developing and evaluating novel diagnostic assays are crucial components of contemporary diagnostic research. The receiver operating characteristic (ROC) curve and the area under the ROC curve (AUC) are frequently used to evaluate diagnostic assays' performance. The variation in AUC estimation can be quantified nonparametrically using resampling methods, such as bootstrapping, and then used to construct interval estimation for the AUC. When multiple observations are observed from the same subject, which is very common in veterinary diagnostic tests evaluation experiments, a traditional bootstrap-based method can fail to provide valid interval estimations of AUC. In particular, the traditional method does not account for the correlation among data observations and could result in interval estimation that fails to cover the true AUC adequately at the desired confidence level. In this paper, we proposed two novel methods to calculate the confidence interval of the AUC for correlated diagnostic test data based on cluster bootstrapping and hierarchical bootstrapping, respectively. Our simulation studies showed that both proposed methods had adequate coverage probabilities which were higher than the existing traditional method when there were intra-subject correlations. We also discussed applying the proposed methods to evaluate a novel whole-virus ELISA (wv-ELISA) diagnostic assay in detecting porcine parainfluenza virus type-1 antibodies in swine serum.

Combining formal methods and Bayesian approach for inferring discrete-state stochastic models from steady-state data

Stochastic population models are widely used to model phenomena in different areas such as cyber-physical systems, chemical kinetics, collective animal behaviour, and beyond. Quantitative analysis of stochastic population models easily becomes challenging due to the combinatorial number of possible states of the population. Moreover, while the modeller easily hypothesises the mechanistic aspects of the model, the quantitative parameters associated to these mechanistic transitions are difficult or impossible to measure directly. In this paper, we investigate how formal verification methods can aid parameter inference for population discrete-time Markov chains in a scenario where only a limited sample of population-level data measurements—sample distributions among terminal states—are available. We first discuss the parameter identifiability and uncertainty quantification in this setup, as well as how the existing techniques of formal parameter synthesis and Bayesian inference apply. Then, we propose and implement four different methods, three of which incorporate formal parameter synthesis as a pre-computation step. We empirically evaluate the performance of the proposed methods over four representative case studies. We find that our proposed methods incorporating formal parameter synthesis as a pre-computation step allow us to significantly enhance the accuracy, precision, and scalability of inference. Specifically, in the case of unidentifiable parameters, we accurately capture the subspace of parameters which is data-compliant at a desired confidence level.

Spatial confidence regions for combinations of excursion sets in image analysis.

The analysis of excursion sets in imaging data is essential to a wide range of scientific disciplines such as neuroimaging, climatology, and cosmology. Despite growing literature, there is little published concerning the comparison of processes that have been sampled across the same spatial region but which reflect different study conditions. Given a set of asymptotically Gaussian random fields, each corresponding to a sample acquired for a different study condition, this work aims to provide confidence statements about the intersection, or union, of the excursion sets across all fields. Such spatial regions are of natural interest as they directly correspond to the questions 'Where do all random fields exceed a predetermined threshold?', or 'Where does at least one random field exceed a predetermined threshold?'. To assess the degree of spatial variability present, our method provides, with a desired confidence, subsets and supersets of spatial regions defined by logical conjunctions (i.e. set intersections) or disjunctions (i.e. set unions), without any assumption on the dependence between the different fields. The method is verified by extensive simulations and demonstrated using task-fMRI data to identify brain regions with activation common to four variants of a working memory task.

Estimation of SARS-CoV-2 Seroprevalence in Central North Carolina: Accounting for Outcome Misclassification in Complex Sample Designs.

Population-based seroprevalence studies are crucial to understand community transmission of COVID-19 and guide responses to the pandemic. Seroprevalence is typically measured from diagnostic tests with imperfect sensitivity and specificity. Failing to account for measurement error can lead to biased estimates of seroprevalence. Methods to adjust seroprevalence estimates for the sensitivity and specificity of the diagnostic test have largely focused on estimation in the context of convenience sampling. Many existing methods are inappropriate when data are collected using a complex sample design. We present methods for seroprevalence point estimation and confidence interval construction that account for imperfect test performance for use with complex sample data. We apply these methods to data from the Chatham County COVID-19 Cohort (C4), a longitudinal seroprevalence study conducted in central North Carolina. Using simulations, we evaluate bias and confidence interval coverage for the proposed estimator compared with a standard estimator under a stratified, three-stage cluster sample design. We obtained estimates of seroprevalence and corresponding confidence intervals for the C4 study. SARS-CoV-2 seroprevalence increased rapidly from 10.4% in January to 95.6% in July 2021 in Chatham County, North Carolina. In simulation, the proposed estimator demonstrates desirable confidence interval coverage and minimal bias under a wide range of scenarios. We propose a straightforward method for producing valid estimates and confidence intervals when data are based on a complex sample design. The method can be applied to estimate the prevalence of other infections when estimates of test sensitivity and specificity are available.

Uncertainty quantification of the convolutional neural networks on permeability estimation from micro-CT scanned sandstone and carbonate rock images

Rock permeability is one of the most crucial properties affecting subsurface fluid flow behaviors. To accurately and robustly estimate the permeability, Digital Rock Physics, including micro-CT scanning technology and direct flow simulations on scanned images, has prevailed in recent years. Besides, machine learning techniques such as convolutional neural networks (CNNs) have been widely adopted and achieved success in permeability estimations directly from rock images. However, existing ML methods used for permeability estimation from rock images lack uncertainty quantification that causes unreliable predictions and overconfident estimations on out-of-distribution (OOD) samples. In this work, we propose a PI3NN-CNN framework to address this problem. PI3NN-CNN consists of a CNN model for absolute permeability estimation and a PI3NN method to quantify the estimation uncertainty. It is able to quantify the uncertainty for in-distribution (InD) data with a desired confidence level, and identify OOD samples to avoid overconfident predictions. We demonstrate the method using micro-CT scanned images from two sandstone and two carbonate rocks. We found that PI3NN-CNN generates accurate predictions for InD samples, while producing high-quality prediction uncertainties regardless of the prediction accuracy. Meanwhile, PI3NN-CNN identifies OOD samples using its special network initialization scheme. The unique feature of PI3NN-CNN makes it applicable to more complex real-world image-based data for robust learning and predictions without overconfident estimations when the ground-truth information is unavailable.

INCREASING STUDENT CONFIDENCE THROUGH EXPERIENTIAL DESIGN EXERCISES IN ENGINEERING SCIENCE COURSES

AbstractThis study investigates the impact on student confidence of completing in-lecture engineering design activities focused on the application of specific engineering science topics within a materials engineering course. Many times, engineering science courses are taught with the expectation that the course content can be easily translated by students at a later time to apply in engineering design activities. By measuring student self-reported confidence across several related topics before and after completion of the in-lecture design exercises the impact of the exercises on student confidence has been quantified. On average, students have a lower than desired confidence in applying the specific materials engineering topics to a design problem after completing only the course content on the subject. Following completion of the related seventy-five minute design exercise, student confidence increased by a statistically significant degree. These results suggest that close integration of topical content learning with design application activity may be a useful method to improve engineering student confidence and, by extension, retention.

PENGARUH ASUPAN GIZI TERHADAP KEJADIAN OBESITAS PADA LANSIA DIWILAYAH KERJA PUSKESMAS BAITUSSALAM, ACEH BESAR

The population growth of the elderly above 60 years has increased rapidly. In 2015, there were 901 million people aged 60 years or over, comprising 12% of the global population. Elderly in Indonesia who are in a state of malnutrition as much as 3.4%, underweight 28.3%, ideal body weight is 42.4%, overweight is 6.7% and obesity is 3.4%. Nutritional problems in the elderly arise due to wrong eating behavior, namely an imbalance between nutritional consumption and the recommended nutritional adequacy. The research objective was to determine the effect of nutritional intake on the incidence of obesity in the elderly in the working area of the Baitussalam Community Health Center, Aceh Besar District. This research is analytic observational. The number of samples was 68 elderly (34 elderly in the obese group and 34 in the non-obese group). The research variables consisted of nutritional intake and nutritional status of the elderly. The study was conducted in January 2020, the data analysis used the chi-square test with the desired confidence level of 95%. The research data are presented in tabular and textual form. From the results of statistical tests using the chi-square test, there is no effect between nutrient intake on obesity in the elderly with the p-value on energy intake = 0.726, p-value on carbohydrate intake = 0.454, p-value on protein intake = 0.445, and p-value on fat intake = 550. In conclusion, there is no relationship between energy intake, KH intake of protein intake and fat intake on obesity in the elderly. Keywords: Energy intake, carbohydrate intake, fat intake, protein intake and elderly obesity.

A framework for mediation analysis with massive data

During the past few years, mediation analysis has gained increasing popularity across various research fields. The primary objective of mediation analysis is to examine the direct impact of exposure on outcome, as well as the indirect effects that occur along the pathways from exposure to outcome. There has been a great number of articles that applied mediation analysis to data from hundreds or thousands of individuals. With the rapid development of technology, the volume of avaliable data increases exponentially, which brings new challenges to researchers. Directly conducting statistical analysis for large datasets is often computationally infeasible. Nonetheless, there is a paucity of findings regarding mediation analysis in the context of big data. In this paper, we propose utilizing subsampled double bootstrap and divide-and-conquer algorithms to conduct statistical mediation analysis on large-scale datasets. The proposed algorithms offer a significant enhancement in computational efficiency over traditional bootstrap confidence interval and Sobel test, while simultaneously ensuring desirable confidence interval coverage and power. We conducted extensive numerical simulations to evaluate the performance of our method. The practical applicability of our approach is demonstrated through two real-world data examples.

Bridging the False Discovery Gap.

Controlling the false discovery rate (FDR) among discoveries from a tandem mass spectrometry proteomics experiment using target decoy competition (TDC) controls only the proportion of false discoveries in an average sense. Thus, for any particular analysis, even with a valid FDR control procedure, the proportion of false discoveries (the FDP) may be higher than the specified FDR threshold. We demonstrate this phenomenon using real data and describe two recently developed methods that help bridge the gap between controlling the expected or average rate of false discoveries and the empirical rate (FDP). The FDP Stepdown method controls the FDP at any desired confidence level, and the TDC Uniform Band provides a confidence, or upper prediction bound, on the FDP in TDC's list of discoveries.

Privacy-Preserved Framework for Short-Term Probabilistic Net Energy Forecasting

This paper develops a differential privacy (DP) model for short-term probabilistic energy forecasting at the low-aggregate level. The method first takes probabilistic forecasting elements from a predictor and injects noise into the mean of the forecast based on the Laplace mechanism to guarantee privacy on the mean. The standard deviation is then carefully perturbed (enlarged) to ensure the forecast to be released contains a desirable confidence interval (CI), here 95%. By doing so, customers' privacy is protected, while the user, such as grid operators or retail providers, receives the forecast containing 95% CI of the original forecast. The predictor used to capture the model and data uncertainties is based on a Bayesian neural network (BNN), which is also benchmarked against a Gaussian Process (GP). The simulations are carried out for different levels of privacy, <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><inline-formula><tex-math notation="LaTeX">$\varepsilon$</tex-math></inline-formula></b> , and the obtained trend provides decision-makers with a clear idea of determining the appropriate <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><inline-formula><tex-math notation="LaTeX">$\varepsilon$</tex-math></inline-formula></b> . The study found that the forecasting error is smoothed out on the used dataset for <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><inline-formula><tex-math notation="LaTeX">$\varepsilon \geq 0.6$</tex-math></inline-formula></b> . The proposed model is an output perturbation approach. Accordingly, the obtained results are compared with an input perturbation approach, showing that the proposed DP model gives the best accuracy/privacy trade-off for users. Furthermore, this study includes rooftop PV generation behind the meter, which is one of the recent transitioning challenges in energy forecasting. Detailed analysis reveals that forecasting is more challenging for periods before sunset when PV generation drop coincides with the common changing point of households' activities.

Formal Verification of Unknown Discrete- and Continuous-Time Systems: A Data-Driven Approach

This paper is concerned with a formal verification scheme for both discrete- and continuous-time deterministic systems with unknown mathematical models. The main target is to verify the safety of unknown systems based on the construction of barrier certificates via a set of data collected from trajectories of systems while providing an a-priori guaranteed confidence on the safety. In our proposed framework, we first cast the original safety problem as a robust convex program (RCP). Solving the proposed RCP is not tractable in general since the unknown model appears in one of the constraints. Instead, we collect finite numbers of data from trajectories of the system and provide a scenario convex program (SCP) corresponding to the original RCP. We then establish a probabilistic closeness between the optimal value of SCP and that of RCP, and as a result, we formally quantify the safety guarantee of unknown systems based on the number of data points and a required level of confidence. We propose our framework in both discrete-time and continuous-time settings. We illustrate the effectiveness of our proposed results by first applying them to an unknown continuous-time room temperature system. We verify that the temperature of the room maintains in a comfort zone with some desirable confidence by collecting data from trajectories of the system. To show the applicability of our techniques to <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">higher dimensional</i> systems with <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">nonlinear</i> dynamics, we then apply our results to a continuous-time <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">nonlinear</i> jet engine compressor and a discrete-time DC motor.

Data-Driven Distributionally Robust Electric Vehicle Balancing for Autonomous Mobility-on-Demand Systems Under Demand and Supply Uncertainties

Electric vehicles (EVs) are being rapidly adopted due to their economic and societal benefits. Autonomous mobility-on-demand (AMoD) systems also embrace this trend. However, the long charging time and high recharging frequency of EVs pose challenges to efficiently managing EV AMoD systems. The complicated dynamic charging and mobility process of EV AMoD systems makes the demand and supply uncertainties significant when designing vehicle balancing algorithms. In this work, we design a data-driven distributionally robust optimization (DRO) approach to balance EVs for both the mobility service and the charging process. The optimization goal is to minimize the worst-case expected cost under both passenger mobility demand uncertainties and EV supply uncertainties. We then propose a novel distributional uncertainty sets construction algorithm that guarantees the produced parameters are contained in desired confidence regions with a given probability. To solve the proposed DRO AMoD EV balancing problem, we derive an equivalent computationally tractable convex optimization problem. Based on real-world EV data of a taxi system, we show that with our solution the average total balancing cost is reduced by 14.49%, and the average mobility fairness and charging fairness are improved by 15.78% and 34.51%, respectively, compared to solutions that do not consider uncertainties.

Uncertainty quantification of machine learning models to improve streamflow prediction under changing climate and environmental conditions

Machine learning (ML) models, and Long Short-Term Memory (LSTM) networks in particular, have demonstrated remarkable performance in streamflow prediction and are increasingly being used by the hydrological research community. However, most of these applications do not include uncertainty quantification (UQ). ML models are data driven and can suffer from large extrapolation errors when applied to changing climate/environmental conditions. UQ is required to quantify the influence of data noises on model predictions and avoid overconfident projections in extrapolation. In this work, we integrate a novel UQ method, called PI3NN, with LSTM networks for streamflow prediction. PI3NN calculates Prediction Intervals by training 3 Neural Networks. It can precisely quantify the predictive uncertainty caused by the data noise and identify out-of-distribution (OOD) data in a non-stationary condition to avoid overconfident predictions. We apply the PI3NN-LSTM method in the snow-dominant East River Watershed in the western US and in the rain-driven Walker Branch Watershed in the southeastern US. Results indicate that for the prediction data which have similar features as the training data, PI3NN precisely quantifies the predictive uncertainty with the desired confidence level; and for the OOD data where the LSTM network fails to make accurate predictions, PI3NN produces a reasonably large uncertainty indicating that the results are not trustworthy and should avoid overconfidence. PI3NN is computationally efficient, robust in performance, and generalizable to various network structures and data with no distributional assumptions. It can be broadly applied in ML-based hydrological simulations for credible prediction.

Sensor Management Method of Giving Priority to Confirmed Identified Targets.

The optimization objective function of sensor management for target identification is commonly established based on information theory indicators such as information gain, discrimination, discrimination gain, and quadratic entropy, which can control the sensors to reduce the overall uncertainty of all targets to be identified but ignores the speed of target being confirmed as identified. Therefore, inspired by the maximum posterior criterion of target identification and the target identification confirmation mechanism, we study a sensor management method that preferentially allocates resources to identifiable targets. Firstly, in the distributed target identification framework based on Bayesian theory, an improved identification probability prediction method that provides feedback the global identification results to local classifiers is proposed, which can improve the accuracy of identification probability prediction. Secondly, an effective sensor management function based on information entropy and expected confidence level is proposed to optimize the identification uncertainty itself rather than its variation, which can increase the priority of targets that satisfy the desired confidence level. In the end, the sensor management for target identification is modeled as a sensor allocation problem, and the optimization objective function based on the effective function is constructed, which can improve the target identification speed. The experimental results show that the correct identification rate of the proposed method is comparable to the methods based on information gain, discrimination, discrimination gain, and quadratic entropy in different scenarios, but the average time to confirm the identification is the shortest.

Formal Estimation of Collision Risks for Autonomous Vehicles: A Compositional Data-Driven Approach

In this work, we propose a compositional data-driven approach for the formal estimation of collision risks for autonomous vehicles (AVs) with black-box dynamics while acting in a stochastic multi-agent framework. The proposed approach is based on the construction of sub-barrier certificates for each stochastic agent via a set of data collected from its trajectories while providing an a priori guaranteed confidence on the data-driven estimation. In our proposed setting, we first cast the original collision risk problem for each agent as a robust optimization program (ROP). Solving the acquired ROP is not tractable due to an unknown model that appears in one of its constraints. To tackle this difficulty, we collect finite numbers of data from trajectories of each agent and provide a scenario optimization program (SOP) corresponding to the original ROP. We then establish a probabilistic bridge between the optimal value of SOP and that of ROP, and accordingly, we formally construct a sub-barrier certificate for each unknown agent based on the number of data and a required level of confidence. We then propose a compositional technique based on small-gain reasoning to quantify the collision risk for multi-agent AVs with some desirable confidence based on sub-barrier certificates of individual agents constructed from data. For the case that the proposed compositionality conditions are not satisfied, we provide a relaxed version of compositional results without requiring any compositionality conditions but at the cost of providing a potentially conservative collision risk. Eventually, we also present our approaches for <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">non-stochastic</i> multi-agent AVs. We demonstrate the effectiveness of our proposed results by applying them to a vehicle platooning consisting of 100 vehicles with 1 leader and 99 followers. We formally estimate the collision risk for the whole network by collecting sampled data from trajectories of each agent.

Sample Size Analysis for Machine Learning Clinical Validation Studies.

Before integrating new machine learning (ML) into clinical practice, algorithms must undergo validation. Validation studies require sample size estimates. Unlike hypothesis testing studies seeking a p-value, the goal of validating predictive models is obtaining estimates of model performance. There is no standard tool for determining sample size estimates for clinical validation studies for machine learning models. Our open-source method, Sample Size Analysis for Machine Learning (SSAML) was described and was tested in three previously published models: brain age to predict mortality (Cox Proportional Hazard), COVID hospitalization risk prediction (ordinal regression), and seizure risk forecasting (deep learning). Minimum sample sizes were obtained in each dataset using standardized criteria. SSAML provides a formal expectation of precision and accuracy at a desired confidence level. SSAML is open-source and agnostic to data type and ML model. It can be used for clinical validation studies of ML models.