Sample Size Scenarios Research Articles

Do errors on classic decision biases happen fast or slow? Numeracy and decision time predict probability matching, sample size neglect, and ratio bias

Higher numeracy is associated with better comprehension and use of numeric information as well as reduced susceptibility to some decision biases. We extended this line of work by showing that increased numeracy predicted probability maximizing (versus matching) as well as a better appreciation of large sample sizes. At the same time, we replicated the findings that the more numerate were less susceptible to the ratio bias and base rate neglect phenomena. Decision time predicted accuracy for the ratio bias, probability matching, and sample size scenarios, but not the base rate scenarios. Interestingly, this relationship between decision time and accuracy was positive for the ratio bias problems, but negative for the probability matching and sample size scenarios. Implications for research on cognitive ability and decision biases are discussed.

Cauchy-Schwarz bounded trade-off weighting for causal inference with small sample sizes

The difficulty of causal inference for small-sample-size data lies in the issue of inefficiency that the variance of the estimators may be large. Some existing weighting methods adopt the idea of bias-variance trade-off, but they require manual specification of the trade-off parameters. To overcome this drawback, in this article, we propose a Cauchy-Schwarz Bounded Trade-off Weighting (CBTW) method, in which the trade-off parameter is theoretically derived to guarantee a small Mean Square Error (MSE) in estimation. We theoretically prove that optimizing the objective function of CBTW, which is the Cauchy-Schwarz upper-bound of the MSE for causal effect estimators, contributes to minimizing the MSE. Moreover, since the upper-bound consists of the variance and the squared ℓ2-norm of covariate differences, CBTW can not only estimate the causal effects efficiently, but also keep the covariates balanced. Experimental results on both simulation data and real-world data show that the CBTW outperforms most existing methods especially under small sample size scenarios.

A knowledge-transfer-based approach for combining ordinal regression and medical scoring system in the early prediction of sepsis with electronic health records

Objective:The prediction of sepsis, especially early diagnosis, has received a significant attention in biomedical research. In order to improve current medical scoring system and overcome the limitations of class imbalance and sample size of local EHR (electronic health records), we propose a novel knowledge-transfer-based approach, which combines a medical scoring system and an ordinal logistic regression model. Materials and Methods:Medical scoring systems (i.e. NEWS, SIRS and QSOFA) are generally robust and useful for sepsis diagnosis. With local EHR, machine-learning-based methods have been widely used for building prediction models/methods, but they are often impacted by class imbalance and sample size. Knowledge distillation and knowledge transfer have recently been proposed as a combination approach for improving the prediction performance and model generalization. In this study, we developed a novel knowledge-transfer-based method for combining a medical scoring system (after a proposed score transformation) and an ordinal logistic regression model. We mathematically confirmed that it was equivalent to a specific form of the weighted regression. Furthermore, we theoretically explored its effectiveness in the scenario of class imbalance. Results:For the local dataset and the MIMIC-IV dataset, the VUS (the volume under the multi-dimensional ROC surface, a generalization measure of AUC-ROC for ordinal categories) of the knowledge-transfer-based model (ORNEWS) based on the NEWS scoring system were 0.384 and 0.339, respectively, while the VUS of the traditional ordinal regression model (OR) were 0.352 and 0.322, respectively. Consistent analysis results were also observed for the knowledge-transfer-based models based on the SIRS/QSOFA scoring systems in the ordinal scenarios. Additionally, the predicted probabilities and the binary classification ROC curves of the knowledge-transfer-based models indicated that this approach enhanced the predicted probabilities for the minority classes while reducing the predicted probabilities for the majority classes, which improved AUCs/VUSs on imbalanced data. Discussion:Knowledge transfer, which combines a medical scoring system and a machine-learning-based model, improves the prediction performance for early diagnosis of sepsis, especially in the scenarios of class imbalance and limited sample size.

A non-monotone trust-region method with noisy oracles and additional sampling

In this work, we introduce a novel stochastic second-order method, within the framework of a non-monotone trust-region approach, for solving the unconstrained, nonlinear, and non-convex optimization problems arising in the training of deep neural networks. The proposed algorithm makes use of subsampling strategies that yield noisy approximations of the finite sum objective function and its gradient. We introduce an adaptive sample size strategy based on inexpensive additional sampling to control the resulting approximation error. Depending on the estimated progress of the algorithm, this can yield sample size scenarios ranging from mini-batch to full sample functions. We provide convergence analysis for all possible scenarios and show that the proposed method achieves almost sure convergence under standard assumptions for the trust-region framework. We report numerical experiments showing that the proposed algorithm outperforms its state-of-the-art counterpart in deep neural network training for image classification and regression tasks while requiring a significantly smaller number of gradient evaluations.

Does Sample Size, Sampling Strategy, or Handling of Concentrations Below the Lower Limit of Quantification Matter When Externally Evaluating Population Pharmacokinetic Models?

Precision dosing requires selecting the appropriate population pharmacokinetic model, which can be assessed through external evaluations (EEs). The lack of understanding of how different study design factors influence EE study outcomes makes it challenging to select the most suitable model for clinical use. This study aimed to evaluate the impact of sample size, sampling strategy, and handling of concentrations below thelowerlimit of quantification (BLQ) on the outcomes of EE for four population pharmacokinetic models using vancomycin and tobramycin as examples. Three virtual patient populations undergoing vancomycin or tobramycin therapy were simulated with varying sample size and sampling scenarios. The three approaches used to handle BLQ data were to (1) discard them, (2) impute them as LLOQ/2, or (3) use a likelihood-based approach. EEs were performed with NONMEM and R. Sample size did not have an important impact on the EE results for a given scenario. Increasing the number of samples per patient did not improve predictive performance for two out of the three evaluated models. Evaluating a model developed with rich sampling did not result in better performance than those developed with regular therapeutic drug monitoring. A likelihood-based method to handle BLQ samples impacted the outcomes of the EE with lower bias for predicted troughs. This study suggests that a large sample size may not be necessary for an EE study, and models selected based on TDM may be more generalizable. The study highlights the need for guidelines for EE of population pharmacokinetic models for clinical use.

Military Unmanned Equipment Image Target Recognition Method based on Improved Deep Learning

Abstract Military unmanned equipment target recognition is currently a research hotspot and trend in the field of military intelligence. The small sample size and complex recognition scenarios of military unmanned equipment image datasets result in low recognition accuracy. It is proposed a military unmanned equipment image target recognition method based on improved deep learning, which set Faster R-CNN as the network framework for target recognition, used the Kmeans++ algorithm to label boxes on customized datasets, and then added OHEM to the framework to improve the network’s recognition accuracy for difficult to recognize samples. The accuracy of the algorithm proposed in this article reaches 93.8%, which is 2.8% higher than the YOLOv5 algorithm, providing an improved deep learning method for military unmanned equipment image target recognition.

A Bayesian Bernoulli-Exponential joint model for binary longitudinal outcomes and informative time with applications to bladder cancer recurrence data

BackgroundA variety of methods exist for the analysis of longitudinal data, many of which are characterized with the assumption of fixed visit time points for study individuals. This, however is not always a tenable assumption. Phenomenon that alter subject visit patterns such as adverse events due to investigative treatment administered, travel or any other emergencies may result in unbalanced data and varying individual visit time points. Visit times can be considered informative, because subsequent or current subject outcomes can change or be adapted due to previous subject outcomes.MethodsIn this paper, a Bayesian Bernoulli-Exponential model for analyzing joint binary outcomes and exponentially distributed informative visit times is developed. Via statistical simulations, the influence of controlled variations in visit patterns, prior and sample size schemes on model performance is assessed. As an application example, the proposed model is applied to a Bladder Cancer Recurrence data.Results and conclusionsResults from the simulation analysis indicated that the Bayesian Bernoulli-Exponential joint model converged in stationarity, and performed relatively better for small to medium sample size scenarios with less varying time sequences regardless of the choice of prior. In larger samples, the model performed better for less varying time sequences. This model’s application to the bladder cancer data showed a statistically significant effect of prior tumor recurrence on the probability of subsequent recurrences.

Abstract PR017: Enabling broad population access to facilitate gut microbiome discoveries

Abstract Microbial associations with colorectal cancer (CRC) are well recognized as well as the potential of using microbiome-based signals for broad-scale, non-invasive screening programs, particularly in underserved communities. Variability within microbiome workflows including sample collection is a contributing factor to inconsistent and non-translatable microbiome results. This limits microbiome discoveries associated with CRC and the development of such screening programs. In addition, ensuring participant and patient adherence can be a challenging prospect with fecal sample collections, and non-adherence (i.e., patient dropout, collection errors) can increasingly strain resources. An intuitively designed sample collection device should increase adherence across the general population while minimizing failed collections. Enabling the at-home collection of gut microbiome samples removes barriers between patients and testing while providing the scalability and flexibility needed for large studies or screening programs. Across two studies, 152 naïve participants were recruited to evaluate comprehension of and adherence to the OMNIgene™•GUT Dx device Instructions for Use (IFU). Both studies were designed with key considerations related to user group selection, sample size, population demographics, education level, and relevant real-world scenarios. In the first study (n=118), each participant was observed collecting one simulated “fecal” sample with representation across the Bristol scale (Type 1 to 7). The study observer conducted a follow-up interview with the participant regarding incorrect actions during collection. In the second study (n=34), participants collected one real fecal sample at home and returned it to the laboratory using standard postal services. Samples were inspected at the laboratory for collection adherence with the IFU by evaluating device closure, sufficient mixing, and DNA yield. DNA was extracted using the QIAamp PowerFecal Pro kit (QIAGEN) and quantified using PicoGreen (Invitrogen). In both studies, participants were asked to complete comprehension questionnaires to test understanding of the IFU used during collection. Participants demonstrated high rates of comprehension by answering the questions correctly 94% of the time. Participants observed collecting simulated samples demonstrated success rates of 87% for all tasks overall. Samples evaluated from the second study demonstrated participant adherence where 100% of samples were correctly returned via standard postal systems supporting at-home direct-to-consumer settings; 100% of participants properly closed the tube ensuring no loss of sample during transport; 97% of participants adequately mixed the sample; and 97% of participant samples provided sufficient DNA (≥ 120 ng) per extraction aliquot for downstream analysis. Results from these usability studies demonstrate that the OMNIgene™•GUT Dx device provides an easy-to-use at-home collection solution for diverse populations, especially at-risk, minority populations without easy access to clinical services. Citation Format: Ashlee Brown, Savannah Colameco, Tara Crawford Parks. Enabling broad population access to facilitate gut microbiome discoveries [abstract]. In: Proceedings of the 16th AACR Conference on the Science of Cancer Health Disparities in Racial/Ethnic Minorities and the Medically Underserved; 2023 Sep 29-Oct 2;Orlando, FL. Philadelphia (PA): AACR; Cancer Epidemiol Biomarkers Prev 2023;32(12 Suppl):Abstract nr PR017.

Inclusion of binary proxy variables in logistic regression improves treatment effect estimation in observational studies in the presence of binary unmeasured confounding variables.

We present a simulation study and application that shows inclusion of binary proxy variables related to binary unmeasured confounders improves the estimate of a related treatment effect in binary logistic regression. The simulation study included 60,000 randomly generated parameter scenarios of sample size 10,000 across six different simulation structures. We assessed bias by comparing the probability of finding the expected treatment effect relative to the modeled treatment effect with and without the proxy variable. Inclusion of a proxy variable in the logistic regression model significantly reduced the bias of the treatment or exposure effect when compared to logistic regression without the proxy variable. Including proxy variables in the logistic regression model improves the estimation of the treatment effect at weak, moderate, and strong association with unmeasured confounders and the outcome, treatment, or proxy variables. Comparative advantages held for weakly and strongly collapsible situations, as the number of unmeasured confounders increased, and as the number of proxy variables adjusted for increased.

Incorporating causal inference perspectives into psychoneuroimmunology: A simulation study highlighting concerns about controlling for adiposity in immunopsychiatry

Psychoneuroimmunology and immunopsychiatry are quickly approaching a critical point where the clinical translatability of their evidence base will be tested. To maximize chances for translational success, we believe researchers must adopt causal inference techniques that augment the causal relevance of estimates given theorized causal structures. To illustrate the utility of incorporating causal inference perspectives into psychoneuroimmunology, we applied directed acyclic graphs and a combination of empirical and simulated data to demonstrate the consequences of controlling for adiposity when testing the association between inflammation and depression under the plausible causal structure of increases in adipose tissue leading to greater inflammation that in turn promotes depression. Effect size estimates were pulled from a dataset combining the Midlife in the United States 2 (MIDUS-2) and MIDUS Refresher datasets. Data were extracted and used to simulate data reflecting an adiposity → inflammation → depression causal structure. Next, a Monte Carlo simulation study with 1,000 iterations and three sample size scenarios (Ns = 100, 250, and 500) was conducted testing whether controlling for adiposity when estimating the relation between inflammation and depression influenced the precision of this estimate. Across all simulation scenarios, controlling for adiposity reduced precision of the inflammation → depression estimate, suggesting that researchers primarily interested in quantifying inflammation → depression associations should not control for adiposity. This work thus underscores the importance of incorporating causal inference approaches into psychoneuroimmunological research.

A non-parametric statistical inference framework for Deep Learning in current neuroimaging

Deep Learning (DL) predictions are uncertain; but how uncertain? Statistical inference estimates the probabilities of uncertainty from a sample drawn from a population. Assessing the statistical significance of accuracies reported by DL remains largely unexplored. A framework to do so would usefully support a range of applications, and in particular group classifications from neuroimages where, for operational reasons, sample sizes are necessary limited and thus often do not generalise well. We applied a random-effects inference based on a label permutation test to calculate the statistical significance of K-fold cross-validation (CV) from statistical power and Type-I error rates. Our hypothesis is that in low sample size scenarios, the use of resubstitution with upper bound correction (RUB) as a validation would mitigate the debate on the generalisation ability of DL models. The derived framework enables testing such generalisation ability of DL models as feature extraction methods. A combination of autoencoders and support vector machines as feature extraction and classification models is evaluated in a case-control analysis of Alzheimer’s disease with well-established outcomes. We found that RUB slightly outperforms K-fold CV as a validation method, especially estimating statistical power in the most heterogeneous samples. Therefore, we suggest RUB as potent and valid method for DL with neuroimages in terms of bias, variance and computational demand.

Use of management strategy evaluation to understand the value of citizen science in managing an iconic California recreational fishery

Data-limited fisheries present significant challenges to fisheries management around the world. Landings data represent the simplest and most common source of fishery information, but length data of harvested species can offer particularly useful insight into the strength of fishing pressure and the status of stocks. However, unbiased length data can be difficult and costly to collect, which highlights a need for understanding the utility of non-conventional data collection programs, such as those generated by community members and citizen science programs to improve management decision-making. In this study, our objective was to understand the value of citizen science for management using the iconic California recreational red abalone (Haliotis rufescens) dive fishery. To accomplish this, we employed management strategy evaluation (MSE) to simulate this benthic invertebrate fishery that was sampled by two different fishery independent survey protocols: a community science survey program led by trained volunteers and a professional science survey program led by a state agency. We subjected the data to a range of uncertainties, by including time-varying life history parameters, environmental variability, and effective sample size scenarios, to understand the management level consequences of each survey methodology. Our results show that community science program data collection can serve as a high-quality data source, can be linked directly to fisheries management decisions via a Harvest Control Rule (HCR), and can be a useful information source even in the absence of conventional data sources. Our results are specific to the California red abalone fishery, but they also highlight the potential broader value of community science data collection programs for improving information content of small-scale and recreational fisheries.

Parameter Estimation of Exponentiated Half-Logistic Distribution for Left-Truncated and Right-Censored Data

Left-truncated and right-censored data are widely used in lifetime experiments, biomedicine, labor economics, and actuarial science. This article discusses how to resolve the problems of statistical inferences on the unknown parameters of the exponentiated half-logistic distribution based on left-truncated and right-censored data. In the beginning, maximum likelihood estimations are calculated. Then, asymptotic confidence intervals are constructed by using the observed Fisher information matrix. To cope with the small sample size scenario, we employ the percentile bootstrap method and the bootstrap-t method for the establishment of confidence intervals. In addition, Bayesian estimations under both symmetric and asymmetric loss functions are addressed. Point estimates are computed by Tierney–Kadane’s approximation and importance sampling procedure, which is also applied to establishing corresponding highest posterior density credible intervals. Lastly, simulated and real data sets are presented and analyzed to show the effectiveness of the proposed methods.

A comparison of methods for multiple degree of freedom testing in repeated measures RNA-sequencing experiments

BackgroundAs the cost of RNA-sequencing decreases, complex study designs, including paired, longitudinal, and other correlated designs, become increasingly feasible. These studies often include multiple hypotheses and thus multiple degree of freedom tests, or tests that evaluate multiple hypotheses jointly, are often useful for filtering the gene list to a set of interesting features for further exploration while controlling the false discovery rate. Though there are several methods which have been proposed for analyzing correlated RNA-sequencing data, there has been little research evaluating and comparing the performance of multiple degree of freedom tests across methods.MethodsWe evaluated 11 different methods for modelling correlated RNA-sequencing data by performing a simulation study to compare the false discovery rate, power, and model convergence rate across several hypothesis tests and sample size scenarios. We also applied each method to a real longitudinal RNA-sequencing dataset.ResultsLinear mixed modelling using transformed data had the best false discovery rate control while maintaining relatively high power. However, this method had high model non-convergence, particularly at small sample sizes. No method had high power at the lowest sample size. We found a mix of conservative and anti-conservative behavior across the other methods, which was influenced by the sample size and the hypothesis being evaluated. The patterns observed in the simulation study were largely replicated in the analysis of a longitudinal study including data from intensive care unit patients experiencing cardiogenic or septic shock.ConclusionsMultiple degree of freedom testing is a valuable tool in longitudinal and other correlated RNA-sequencing experiments. Of the methods that we investigated, linear mixed modelling had the best overall combination of power and false discovery rate control. Other methods may also be appropriate in some scenarios.

Testing the effect of bioturbation and species abundance upon discrete-depth individual foraminifera analysis

Abstract. We used a single foraminifera enabled, holistic hydroclimate-to-sediment transient modelling approach to fundamentally evaluate the efficacy of discrete-depth individual foraminifera analysis (IFA) for reconstructing past sea surface temperature (SST) variability from deep-sea sediment archives, a method that has been used, amongst other applications, for reconstructing El Niño–Southern Oscillation (ENSO). The computer model environment allows us to strictly control for variables such as SST, foraminifera species abundance response to SST, as well as depositional processes such as sediment accumulation rate (SAR) and bioturbation depth (BD) and subsequent laboratory processes such as sample size and machine error. Examining a number of best-case scenarios, we find that IFA-derived reconstructions of past SST variability are sensitive to all of the aforementioned variables. Running 100 ensembles for each scenario, we find that the influence of bioturbation upon IFA-derived SST reconstructions, combined with typical samples sizes employed in the field, produces noisy SST reconstructions with poor correlation to the original SST distribution in the water. This noise is especially apparent for values near the tails of the SST distribution, which is the distribution region of particular interest in the case of, e.g. ENSO. The noise is further increased in the case of increasing machine error, decreasing SAR and decreasing sample size. We also find poor agreement between ensembles, underscoring the need for replication studies in the field to confirm findings at particular sites and time periods. Furthermore, we show that a species abundance response to SST could in theory bias IFA-derived SST reconstructions, which can have consequences when comparing IFA-derived SST distributions from markedly different mean climate states. We provide a number of idealised simulations spanning a number of SAR, sample size, machine error and species abundance scenarios, which can help assist researchers in the field to determine under which conditions they could expect to retrieve significant results.

Should I use fixed effects or random effects when I have fewer than five levels of a grouping factor in a mixed-effects model?

As linear mixed-effects models (LMMs) have become a widespread tool in ecology, the need to guide the use of such tools is increasingly important. One common guideline is that one needs at least five levels of the grouping variable associated with a random effect. Having so few levels makes the estimation of the variance of random effects terms (such as ecological sites, individuals, or populations) difficult, but it need not muddy one’s ability to estimate fixed effects terms—which are often of primary interest in ecology. Here, I simulate datasets and fit simple models to show that having few random effects levels does not strongly influence the parameter estimates or uncertainty around those estimates for fixed effects terms—at least in the case presented here. Instead, the coverage probability of fixed effects estimates is sample size dependent. LMMs including low-level random effects terms may come at the expense of increased singular fits, but this did not appear to influence coverage probability or RMSE, except in low sample size (N = 30) scenarios. Thus, it may be acceptable to use fewer than five levels of random effects if one is not interested in making inferences about the random effects terms (i.e. when they are ‘nuisance’ parameters used to group non-independent data), but further work is needed to explore alternative scenarios. Given the widespread accessibility of LMMs in ecology and evolution, future simulation studies and further assessments of these statistical methods are necessary to understand the consequences both of violating and of routinely following simple guidelines.

Conditional Inference in Small Sample Scenarios Using a Resampling Approach

This paper discusses a non-parametric resampling technique in the context of multidimensional or multiparameter hypothesis testing of assumptions of the Rasch model. It is based on conditional distributions and it is suggested in small sample size scenarios as an alternative to the application of asymptotic or large sample theory. The exact sampling distribution of various well-known chi-square test statistics like Wald, likelihood ratio, score, and gradient tests as well as others can be arbitrarily well approximated in this way. A procedure to compute the power function of the tests is also presented. A number of examples of scenarios are discussed in which the power function of the test does not converge to 1 with an increasing deviation of the true values of the parameters of interest from the values specified in the hypothesis to be tested. Finally, an attempt to modify the critical region of the tests is made aiming at improving the power and an R package is provided.

Exploring the sample size and replications scenarios effect on spatial prediction of flood, using MARS and MaxEnt methods case study: saliantape catchment, Golestan, Iran

Exploring the sample size and replications scenarios effect on spatial prediction of flood, using MARS and MaxEnt methods case study: saliantape catchment, Golestan, Iran

Evaluating performance of covariate-constrained randomization (CCR) techniques under misspecification of cluster-level variables in cluster-randomized trials

Covariate constrained randomization (CCR) is a method of controlling imbalance in important baseline covariates in cluster-randomized trials (CRT). We use simulated CRTs to investigate the performance (control of imbalance) of CCR relative to simple randomization (SR) under conditions of misspecification of the cluster-level variable used in the CCR algorithm.We use data from a Patient-Centered Outcomes Research Institute (PCORI)-funded CRT evaluating the Mothers and Babies (MB) intervention (AD-1507-31,473). CCR methodology was used in the MB study to control imbalance in, among other baseline variables, the percent minority (i.e., non-White) participants at each study site. Simulation schemes explored variation in degree of misspecification in the baseline covariate of interest, and include correct report, observed misspecification, and a range of simulated misspecification for intervals within and beyond that observed in the MB study. We also consider three within-site sample size scenarios: that observed in the MB study, small (mean 10) and large (mean 50). Simulations at every level of baseline covariate misspecification suggest that use of the CCR strategy provides between-arm imbalance that is simultaneously lower and less variable, on average, than that produced from the SR strategy. We find that the gains to using CCR over SR are nearly twice as high with accurate reporting (Δ = −5.33) compared to the observed study-level misspecification (Δ = −3.03). Although CCR still outperforms SR as the level of misspecification increases, the gains to using CCR over SR decrease; thus, every effort should still be made to obtain high-quality baseline data.

Folded LDA: Extending the Linear Discriminant Analysis Algorithm for Feature Extraction and Data Reduction in Hyperspectral Remote Sensing

The rich spectral information provided by hyperspectral imaging (HSI) has made this technology very useful in the classification of remotely sensed data. However, classification of hyperspectral data is typically affected by noise and the Hughes phenomenon due to the presence of hundreds of spectral bands and correlation among them, with usually a limited number of samples for training. Linear Discriminant Analysis (LDA) is a well-known technique that has been widely used for supervised dimensionality reduction of hyperspectral data. However, the use of LDA in hyperspectral remote sensing is limited due to 1) its poor performance on small training datasets and 2) the limited number of features that can be selected i.e. c-1 where c is the number of classes in the data. To solve these problems, this work presents a Folded LDA (F-LDA) for dimensionality reduction of remotely sensed HSI data in Small Sample Size (SSS) scenarios. The proposed approach allows many more discriminant features to be selected in comparison to the conventional LDA since the selection is no longer bound by the limiting factor, leading to significantly higher accuracy in the classification of pixels under SSS restrictions. The proposed approach is evaluated on five different datasets, where the experimental results demonstrate the superiority of the F-LDA to the conventional LDA in terms of not only higher classification accuracy but also reduced computational complexity, and reduced contiguous memory requirements.