Sample Size Requirements For Estimation Research Articles

Abstract B065: FabricaTM: A large-scale data simulation platform isolates tumor signal from cell-free DNA and improves tissue of origin prediction accuracy

Abstract Cell-free DNA (cfDNA) liquid biopsy is a promising non-invasive method for disease detection, but data availability and the complexity of cfDNA composition pose barriers to model development. Cancer prediction models trained on cfDNA data from clinically collected blood samples often struggle to isolate tumor-derived signals from confounding factors, and tissue of origin (TOO) models suffer from small sample sizes of rarer tumor types. To address these challenges, we developed FabricaTM, a platform to generate diverse, large-scale, high-fidelity simulated cfDNA datasets for robust and accurate machine learning (ML) model training. A key advantage of FabricaTM is its ability to enhance ML models by matching confounding variables between non-cancer and cancer samples. We first synthesized age-balanced non-cancer samples, then simulated tumor DNA shedding into circulation from 502 reference biopsy samples across 16 indications, optimizing for final tumor content distributions ranging from 0.001 to 14.2%. All reference non-cancer and biopsy samples consisted of aligned bisulfite sequencing reads from a custom target hybrid capture panel. This approach expanded a non-cancer dataset of 177 samples to 1,212 simulated samples and generated 6,400 simulated cancer cfDNA samples matching the non-cancers in age distribution. We then assessed this dataset for equivalence with clinical data and ability to improve cancer prediction. Non-linear dimensionality reduction and clustering of methylation signal showed simulated samples cluster indistinguishably from 2,290 samples (1,149 non-cancer, 1,141 treatment-naïve cancer) from the CORE-HH clinical study (NCT05435066). A binary cancer prediction model trained on the FabricaTM-generated dataset and evaluated on the CORE-HH dataset showed reduced reliance on age-associated signal (R2=0.15) and increased tuning towards tumor content, with minor cost to sensitivity (<4% at 95% specificity), relative to a model trained and cross-validated on the CORE-HH data (R2=0.60). To evaluate the benefit of these data for TOO prediction, we trained multiclass TOO models using FabricaTM's tumor biopsy reference data with and without the addition of our simulated data for 10 target indication groupings and benchmarked their performance on our clinical dataset. Training with both data types yielded a 10% increase in balanced accuracy. Moreover, peak performance was achieved at different training sample numbers per indication, illustrating FabricaTM's potential to estimate sample size requirements during study design. Our findings suggest FabricaTM-generated data can augment limited datasets to train ML models to identify tumor-specific biomarkers obscured by technical and demographic biases in real-world data. The age-balancing approach is readily applied to other confounders to help models learn true disease signatures. The platform is also adaptable to other diseases and biofluids (e.g., Alzheimer's disease, urine), allowing for extension to diagnostic and screening development beyond cancer and blood across the care spectrum. Citation Format: Kade Pettie, Shiva Farashahi, Jackson Killian, Dorna Kashef, Josh Hubbell, Feras Hantash, Jocelyn Charlton, Kieran Chacko. FabricaTM: A large-scale data simulation platform isolates tumor signal from cell-free DNA and improves tissue of origin prediction accuracy [abstract]. In: Proceedings of the AACR Special Conference: Liquid Biopsy: From Discovery to Clinical Implementation; 2024 Nov 13-16; San Diego, CA. Philadelphia (PA): AACR; Clin Cancer Res 2024;30(21_Suppl):Abstract nr B065.

Examining the effect of sample size on the estimation of low-frequency signals in tree-ring chronologies

Dendrochronology utilizes mean values of individual tree-ring indices to average out noise and strengthen common signals, typically associated with climate. Expressed Population Signal (EPS) is commonly used to assess the degree to which a chronology represents the common signal. However, there is a lack of studies on how EPS reflects low-frequency signals. In this study, we propose a frequency-dependent EPS (FEPS) approach to evaluating the low-frequency signal strength. Extensive tests were conducted using tree rings from Chinese Loess Plateau. We found that as timescales lengthen, EPS decreases due to declining inter-series correlations, thus demanding larger sample sizes. This finding remains robust across different detrending methods, filtering methods, filter orders, and treatments for end effects. Determining sample sizes by standard EPS (common use of EPS with unfiltered series) has a limited impact on multi-year frequencies, but considerably influences timescales exceeding a decade. Our study reveals a prevalent underestimation of sample size requirements for robust multi-year and interdecadal signals estimation, and the FEPS approach is expected to potentially enhance our understanding of long-term climate dynamics.

Assessing Impact of Nutrition Care by Registered Dietitian Nutritionists on Patient Medical and Treatment Outcomes in Outpatient Cancer Clinics: A Cohort Feasibility Study

More information is needed about the impact of outpatient nutrition care from a registered dietitian nutritionist (RDN) on patient outcomes. This study aimed to assess the feasibility of a cohort study design to evaluate impact of RDN nutrition care on patient outcomes, describe clinic malnutrition screening practices, and estimate statistical parameters for a larger study. Seventy-seven patients with lung, esophageal, colon, rectal, or pancreatic cancer from six facilities were included (41 received RDN care and 36 did not). RDN nutrition care was prospectively documented for six months and documented emergency room visits, unplanned hospitalizations and treatment changes were retrospectively abstracted from medical records. Most facilities used the Malnutrition Screening Tool (MST) to determine malnutrition risk. Patients receiving RDN care had, on average, five, half hour visits and had more severe disease and higher initial malnutrition risk, although this varied across sites. Documented medical and treatment outcomes were relatively rare and similar between groups. Estimated sample size requirements varied from 113 to 5856, depending on tumor type and outcome, and intracluster correlation coefficients (ICCs) ranged from 0 to 0.47. Overall, the methods used in this study are feasible but an interventional or implementation design might be advantageous for a larger study.

Hierarchical drift diffusion modeling uncovers multisensory benefit in numerosity discrimination tasks.

Studies of accuracy and reaction time in decision making often observe a speed-accuracy tradeoff, where either accuracy or reaction time is sacrificed for the other. While this effect may mask certain multisensory benefits in performance when accuracy and reaction time are separately measured, drift diffusion models (DDMs) are able to consider both simultaneously. However, drift diffusion models are often limited by large sample size requirements for reliable parameter estimation. One solution to this restriction is the use of hierarchical Bayesian estimation for DDM parameters. Here, we utilize hierarchical drift diffusion models (HDDMs) to reveal a multisensory advantage in auditory-visual numerosity discrimination tasks. By fitting this model with a modestly sized dataset, we also demonstrate that large sample sizes are not necessary for reliable parameter estimation.

Improved convergence guarantees for learning Gaussian mixture models by EM and gradient EM

We consider the problem of estimating the parameters a Gaussian Mixture Model with K components of known weights, all with an identity covariance matrix. We make two contributions. First, at the population level, we present a sharper analysis of the local convergence of EM and gradient EM, compared to previous works. Assuming a separation of Ω( logK), we prove convergence of both methods to the global optima from an initialization region larger than those of previous works. Specifically, the initial guess of each component can be as far as (almost) half its distance to the nearest Gaussian. This is essentially the largest possible contraction region. Our second contribution are improved sample size requirements for accurate estimation by EM and gradient EM. In previous works, the required number of samples had a quadratic dependence on the maximal separation between the K components, and the resulting error estimate increased linearly with this maximal separation. In this manuscript we show that both quantities depend only logarithmically on the maximal separation.

Short-term integrated rehabilitation for people with newly diagnosed thoracic cancer: a multi-centre randomized controlled feasibility trial

Objectives: The main objective of this study is to determine the feasibility of recruiting and retaining patients recently diagnosed with thoracic cancer to a trial of short-term integrated rehabilitation; evaluate uptake of theoretically informed components targeting physical function, symptom self-management and participation; estimate sample size requirements for an efficacy trial. Design: Parallel group randomized controlled feasibility trial. Setting: Three U.K. hospitals. Participants: Patients ⩽eight weeks of thoracic cancer diagnosis, Eastern Cooperative Oncology Group Performance Status 0–3, any cancer stage and treatment plan. Interventions: Participants randomly allocated (1:1) to short-term integrated rehabilitation and standard care or standard care alone over 30 days. Main measures: Primary: participant recruitment and retention, targeting ⩾30% of eligible patients enrolling and ⩾50% of participants reporting outcomes at 30 days. Secondary: intervention fidelity; missing data and performance of outcome measures for self-efficacy, symptoms, physical activity and health-related quality of life. Results: Of 159 eligible patients approached, 54 (34%) were recruited. A total of 44 (82%) and 39 (72%) participants reported outcomes at 30 and 60 days, respectively. Intervention fidelity was high. Rehabilitation was delivered across 3 (1–3) sessions over 32 (22–45) days (median (range)). Changes in clinical outcomes were modest but most apparent at 60 days for health-related quality of life: Functional Assessment of Cancer Therapy Lung Cancer score median (interquartile range) change 9.7 (−12.0 to 16.0) rehabilitation versus 2.3 (−15.0 to 14.5) standard care. Conclusion: A trial to examine efficacy of short-term integrated rehabilitation for people newly diagnosed with thoracic cancer is feasible. A sample of 336 participants could detect a meaningful effect on health-related quality of life as the primary outcome.

Blood neurofilament light as a potential endpoint in Phase 2 studies in MS.

ObjectivesTo assess whether neurofilament light chain (NfL) could serve as an informative endpoint in Phase 2 studies in patients with relapsing–remitting multiple sclerosis (RRMS) and estimate the sample size requirements with NfL as the primary endpoint.MethodsUsing data from the Phase 3 FREEDOMS study, we evaluated correlation of NfL at Month 6 with 2‐year outcomes: relapses, confirmed disability worsening (CDW), new or enlarging T2 lesions (active lesions), and brain volume loss (BVL). We compared the proportion of treatment effect (PTE) on 2‐year relapses and BVL explained by 6‐month log‐transformed NfL levels with the PTE explained by the number of active lesions over 6 months. We estimated sample size requirements for different treatment effects.ResultsAt Month 6, blood NfL levels (pg/mL, median [range]) were lower in the fingolimod arm (fingolimod (n = 132) 18 [8–247]; placebo (n = 114) 26 [8–159], P < 0.001). NfL at 6 months correlated with number of relapses (r = 0.25, P < 0.001), 6‐month CDW (hazard ratio 1.83, P = 0.012), active lesions (r = 0.46, P < 0.001), and BVL (r = −0.41, P < 0.001) at Month‐24. The PTE (95% CI) on 24‐month relapses and BVL explained by 6‐month NfL was 25% (8–60%) and 60% (32–132%), and by 6‐month active lesions was 28% (11–66%) and 45% (18–115%), respectively. Assuming a 20–40% treatment‐related reduction in NfL levels, 143‐28 patients per arm will be required.ConclusionsBlood NfL may qualify as an informative and easy‐to‐measure endpoint for future Phase 2 clinical studies that captures both inflammatory‐ and noninflammatory‐driven neuroaxonal injury in RRMS.

A Comparison of Accelerated and Non-accelerated MRI Scans for Brain Volume and Boundary Shift Integral Measures of Volume Change: Evidence from the ADNI Dataset

The aim of this study was to assess whether the use of accelerated MRI scans in place of non-accelerated scans influenced brain volume and atrophy rate measures in controls and subjects with mild cognitive impairment and Alzheimer’s disease. We used data from 861 subjects at baseline, 573 subjects at 6 months and 384 subjects at 12 months from the Alzheimer’s Disease Neuroimaging Initiative (ADNI). We calculated whole-brain, ventricular and hippocampal atrophy rates using the k-means boundary shift integral (BSI). Scan quality was visually assessed and the proportion of good quality accelerated and non-accelerated scans compared. We also compared MMSE scores, vascular burden and age between subjects with poor quality scans with those with good quality scans. Finally, we estimated sample size requirements for a hypothetical clinical trial when using atrophy rates from accelerated scans and non-accelerated scans. No significant differences in whole-brain, ventricular and hippocampal volumes and atrophy rates were found between accelerated and non-accelerated scans. Twice as many non-accelerated scan pairs suffered from at least some motion artefacts compared with accelerated scan pairs (p ≤ 0.001), which may influence the BSI. Subjects whose accelerated scans had significant motion had a higher mean vascular burden and age (p ≤ 0.05) whilst subjects whose non-accelerated scans had significant motion had poorer MMSE scores (p ≤ 0.05). No difference in estimated sample size requirements was found when using accelerated vs. non-accelerated scans. Accelerated scans reduce scan time and are better tolerated. Therefore it may be advantageous to use accelerated over non-accelerated scans in clinical trials that use ADNI-type protocols, especially in more cognitively impaired subjects.

Sample Size Requirements for Estimation of Item Parameters in the Multidimensional Graded Response Model.

Likert types of rating scales in which a respondent chooses a response from an ordered set of response options are used to measure a wide variety of psychological, educational, and medical outcome variables. The most appropriate item response theory model for analyzing and scoring these instruments when they provide scores on multiple scales is the multidimensional graded response model (MGRM) A simulation study was conducted to investigate the variables that might affect item parameter recovery for the MGRM. Data were generated based on different sample sizes, test lengths, and scale intercorrelations. Parameter estimates were obtained through the flexMIRT software. The quality of parameter recovery was assessed by the correlation between true and estimated parameters as well as bias and root-mean-square-error. Results indicated that for the vast majority of cases studied a sample size of N = 500 provided accurate parameter estimates, except for tests with 240 items when 1000 examinees were necessary to obtain accurate parameter estimates. Increasing sample size beyond N = 1000 did not increase the accuracy of MGRM parameter estimates.

Acute optic neuritis: a clinical paradigm for evaluation of neuroprotective and restorative strategies?

What makes the acute optic neuritis model unique in an era of trials for neuroprotective and myelin repair agents? Acute optic neuritis (AON) is a common, and often the earliest manifestation of central nervous system (CNS) inflammatory demyelinating disorders like multiple sclerosis (MS) and neuromyelitis optica (NMO). It affects at least half the patients with MS and is the presenting feature in 15–20% of patients. Vision loss in AON secondary to NMO tends to be more severe with less potential for recovery and greater axonal degeneration, as measured by optical coherence tomography (OCT) (Bennett et al., 2014). Several unique features of the afferent visual pathway (AVP) make the AON model an ideal system to study disease pathogenesis and evaluate potential neuroprotective and myelin repair strategies.

Pooling prospective studies to investigate the etiology of second cancers.

With over 13 million cancer survivors in the United States today, second cancers are of rapidly growing importance. However, data on nontreatment risk factors for second cancers are sparse. We explored the feasibility of pooling data from cohort studies of cancer incidence to investigate second cancer etiology. We combined data from five prospective studies including more than 800,000 individuals. We compared study designs and populations; evaluated availability of and ability to harmonize risk factor data; compared incidence and survival for common first primary malignancies and incidence of second primary malignancies; and estimated sample size requirements. Overall, 96,513 incident, first primary malignancies were diagnosed during 1985 to 2009. Incidence rates and survival following the first primary varied among the cohorts, but most of the heterogeneity could be explained by characteristics of the study populations (age, sex, smoking, and screening rates). A total of 7,890 second primary cancers (excluding original primary site) were identified, yielding sufficient statistical power (≥80%) for detecting modest associations with risk of all second cancers among survivors of common first primary malignancies (e.g., colorectal cancer); however, there were insufficient events for studying survivors of rarer cancers or identifying risk factors for specific second cancers. Pooling data from cohort studies to investigate nontreatment risk factors for second primary cancers seems feasible but there are important methodologic issues-some of which are barriers to specific research questions-that require special attention. Increased understanding of nontreatment risk factors for second cancers will provide valuable prevention and surveillance information.

Heritability estimation of osteoarthritis in the pig-tailed macaque (Macaca nemestrina) with a look toward future data collection.

We examine heritability estimation of an ordinal trait for osteoarthritis, using a population of pig-tailed macaques from the Washington National Primate Research Center (WaNPRC). This estimation is non-trivial, as the data consist of ordinal measurements on 16 intervertebral spaces throughout each macaque’s spinal cord, with many missing values. We examine the resulting heritability estimates from different model choices, and also perform a simulation study to compare the performance of heritability estimation with these different models under specific known parameter values. Under both the real data analysis and the simulation study, we find that heritability estimates from an assumption of normality of the trait differ greatly from those of ordered probit regression, which considers the ordinality of the trait. This finding indicates that some caution should be observed regarding model selection when estimating heritability of an ordinal quantity. Furthermore, we find evidence that our real data have little information for valid heritability estimation under ordered probit regression. We thus conclude with an exploration of sample size requirements for heritability estimation under this model. For an ordinal trait, an incorrect assumption of normality can lead to severely biased heritability estimation. Sample size requirements for heritability estimation of an ordinal trait under the threshold model depends on the pedigree structure, trait distribution and the degree of relatedness between each phenotyped individual. Our sample of 173 monkeys did not have enough information from which to estimate heritability, but estimable heritability can be obtained with as few as 180 related individuals under certain scenarios examined here.

Sample Size Requirements for Vehicle's Speed Data Collection Using Global Positioning System

In recent years, Global Positioning System (GPS) device is widely used for collecting speed, position, travel time and delay data of various vehicle types. GPS device is capable of recording speed data at sub-second accuracy and position data to sub-meter accuracy. In order to have appropriate results, it is necessary to have sample size more than the minimum requirements for various vehicles types. With advent of GPS technology and its widespread use for traffic data collection, it is required to examine sample sizes recommended by Institute of Transportation Engineer's Manual of Transportation Engineering Studies and reported in literature by various other researchers.Reported sample size depends, either on standard deviation of maximum speed attained by vehicle or on speed range of vehicle. These two parameters change with speeding and accelerating capability of vehicle and hence change with vehicle type. This study aims at estimating sample size requirements for GPS speed data of various vehicles types (such as truck, car, motorized three and two wheeler, etc.) plying on Indian roads using various methods reported in literature.In acceleration/deceleration related studies at intersection or at control section where vehicles’ speed varies from zero to maximum (or otherwise), use of standard deviation of running speed (as specified in ITE Manual) or standard deviation of range of speed (Quiroga3) may not be appropriate to determine the sample size for GPS experiments. In such cases, use of standard deviation of free flow speed of test runs are more useful, (Li, S. et al., 2002). Since the standard deviation of free flow speed of vehicles varies with vehicle type (among other factors), the sample size requirements differ with the vehicle type.Experiments are conducted on truck, motorized three wheeler and two wheeler and car for evaluating their sample size requirements using GPS device. Sample size is evaluated on the basis of 5 test runs conducted initially for these vehicle types. Number of initial test runs were decided on the basis of comparison of tα/2 values with Zα/2 values. It is found that after 4 test runs, tα/2 and Zα/2 are equal at 95% level of confidence (Li, S. et al., 2002). Various methods (ITE Manual, 2000, Quiroga, 1998 and Li, S. et al., 2002) are used to decide sample size requirements of different vehicle types. It indicates that ITE manual2 method underestimates the sample size as compared to Quiroga's hybrid method and Li, S.’s modified method. It is also found that Li, S.’s method is best suited to estimate sample size requirement of different vehicle types.

Sample size requirements for interval estimation of the strength of association effect sizes in multiple regression analysis

Effect size reporting and interpreting practices have been extensively recommended in academic journals when analyzing primary outcomes of all empirical studies. Accordingly, the sample squared multiple correlation coefficient is the commonly reported strength of association index in practical applications of multiple linear regression. This paper examines the sample size procedures proposed by Bonett and Wright for precise interval estimation of the squared multiple correlation coefficient. The simulation results showed that their simple method for attaining the desired precision of expected width provides satisfactory results only when sample sizes are large. Moreover, the suggested sample size formula for achieving the designated assurance probability is inaccurate and problematic. According to these findings, their sample size procedures are not recommended.

Reliability of left ventricular noncompaction imaging criteria - the fractal facts

Background Left ventricular noncompaction (LVNC) is a cardiomyopathy with important prognostic implications. Identification of patients with LVNC using current criteria is challenging. Fractal analysis is a novel approach which summarises global LV trabecular complexity as a continuous variable, the fractal dimension, enabling the establishment of ethnically and disease-appropriate reference ranges for trabeculation. Methods We aimed to determine the reproducibility and accuracy of the fractal method compared to other LVNC criteria. We used these results to estimate sample size requirements for population studies of trabeculation. Two independent observers analyzed CMR data from 20 nonselected LVNC cases (mean age 47±13, 11 men) and 40 non-selected healthy volunteers (healthy whites, n=20; healthy blacks, n=20, mean age 45±14, 23 men). Datasets were evaluated using the fractal method (Figure 1) and two other CMR criteria (Petersen and Jacquier). Factorial repeated measures analysis of variance was performed to estimate within-subject variance components and total variability to be used in sample size calculations for the CMR criteria. Results The fractal approach was reproducible (for intra and inter-observer readings: repeatability coefficients, 0.059 and 0.067; intraclass correlation coefficients, 0.98 and 0.97; coefficients of variation, 5%, both). This reproducibility was greater than that calculated for the other LVNC analysis techniques (Figure 2a). Likelihood ratio graphs (Figure 2b) showed superior accuracy for the fractal method over the comparator techniques based on a composite of sensitivity, specificity and positive predictive values. These results translate into superior ability to detect difference and more powerful clinical trials - the minimum number of individuals needed to detect a clinically significant difference in trabeculation between diseased and healthy study populations would be 26 for the fractal method; 60 for the Petersen and 94 for the Jacquier technique (assuming power, 0.90; two-tailed a, 0.05). Conclusions Fractal analysis of LV trabeculae demonstrates good reproducibility and is more accurate in diagnosing LVNC than other techniques. The fractal dimension, if used in clinical trials would more than halve the number of patients required to detect clinically relevant differences in trabecular complexity.

Integrating Mortality and Morbidity Outcomes

Outcome measures that integrate mortality and morbidity, like quality-adjusted life years (QALYs), have been proposed for critical care clinical trials. We sought to describe the distribution of QALYs in critically ill patients and estimate sample size requirements for a hypothetical trial using QALYs as the primary outcome. We used data from a prospective cohort study of survivors of acute respiratory distress syndrome to generate utility values and calculate QALYs at 6 and 12 months. Using multiple simulations, we estimated the required sample sizes for multiple outcome scenarios in a hypothetical trial, including a base-case wherein the intervention improved both mortality and QALYs among survivors. From 195 enrolled patients, follow-up was sufficient to generate QALY outcomes for 168 (86.2%) at 6 months and 159 (81.5%) at 1 year. For a hypothetical intervention that reduced mortality from 48 to 44% and improved QALYs by 0.025 in survivors at 6 months, the required per-group sample size was 571 (80% power; two-sided α = 0.05), compared with 2,436 patients needed for a comparison focusing on mortality alone. When only mortality or QALY in survivors (but not both) showed improvement by these amounts, 3,426 and 1,827 patients per group were needed, respectively. When mortality and morbidity effects moved in opposite directions, simulation results became impossible to interpret. QALYs may be a feasible outcome in critical care trials yielding a patient-centered result and major gains in statistical power under certain conditions, but this approach is susceptible to several threats, including loss to follow-up.

Sample Size Requirements in Single- and Multiphase Growth Mixture Models: A Monte Carlo Simulation Study

Just as growth mixture models are useful with single-phase longitudinal data, multiphase growth mixture models can be used with multiple-phase longitudinal data. One of the practically important issues in single- and multiphase growth mixture models is the sample size requirements for accurate estimation. In a Monte Carlo simulation study, the sample sizes required for using these models are investigated under various theoretical and realistic conditions. In particular, the relationship between the sample size requirement and the number of indicator variables is examined, because the number of indicators can be relatively easily controlled by researchers in many multiphase data collection settings such as ecological momentary assessment. The findings not only provide tangible information about required sample sizes under various conditions to help researchers, but they also increase understanding of sample size requirements in single- and multiphase growth mixture models.

Sample Size Requirements for Interval Estimation of the Kappa Statistic for Interobserver Agreement Studies with a Binary Outcome and Multiple Raters

Sample size requirements that achieve a prespecified expected lower limit for a confidence interval about the intraclass kappa statistic are supplied for the case of multiple raters and a binary outcome variable. The expected lower confidence limit achievable for a given number of subjects and raters is also presented. These results should be useful in the planning stages of an interobserver agreement study in which the focus is on interval estimation rather than hypothesis-testing.

Reliability of Medical Group and Physician Performance Measurement in the Primary Care Setting

Performance reporting is increasingly focused on physician practice sites and individual physicians. To assess the reliability of performance measurement for practice sites and individual physicians. We used data collected across multiple payers as part of a statewide measurement collaborative to evaluate the observed measure reliability and sample size requirements to achieve acceptable reliability of 4 Health Care Effectiveness Data and Information Set measures of preventive care and 10 Health Care Effectiveness Data and Information Set measures of chronic care across 334 practice sites. We conducted a parallel set of physician-level analyses using data across 118 primary physicians practicing within a large multispecialty group. Observed reliabilities and estimated sample size requirements to achieve reliability ≥0.70. At the practice site level, sample sizes required to achieve a reliability of 0.70 were less than 200 patients per site for all 4 measures of preventive care, all 4 process measures of diabetes care, and 2 outcomes measures of diabetes care. Larger samples were required to achieve reliability for cholesterol screening in the presence of cardiovascular disease (n = 249) and use of appropriate asthma medications (n = 351). At the physician level, less than 200 patients were required for all 4 measures of preventive care, but for many chronic care measures the samples of patients available per physician were not sufficient to achieve a reliability of 0.70. In a multipayer collaborative, sample sizes were adequate to reliably assess clinical process and outcome measures at the practice site level. For individual physicians, sample sizes proved adequate to reliably measure preventive care, but may not be feasible for chronic care assessment.

Effects of Importance and Detectability of Usability Problems on Sample Size Requirements

Existing models of usability evaluation estimate sample size requirements based on goal discovery rate and detectability of potential problems without consideration of the importance of the problem and an index of usability, which allows the comparison of the effectiveness of usability of information technology products and services across different domains. This article presents a methodology which accounts for the aforementioned limitations in usability evaluation. A case study is presented, which illustrates the utility of the proposed methodology and indicates that the current methods underestimate the number of participants in usability evaluation studies.