Nominal Error Rate Research Articles

Heterogeneity in effect size estimates

A typical empirical study involves choosing a sample, a research design, and an analysis path. Variation in such choices across studies leads to heterogeneity in results that introduce an additional layer of uncertainty, limiting the generalizability of published scientific findings. We provide a framework for studying heterogeneity in the social sciences and divide heterogeneity into population, design, and analytical heterogeneity. Our framework suggests that after accounting for heterogeneity, the probability that the tested hypothesis is true for the average population, design, and analysis path can be much lower than implied by nominal error rates of statistically significant individual studies. We estimate each type's heterogeneity from 70 multilab replication studies, 11 prospective meta-analyses of studies employing different experimental designs, and 5 multianalyst studies. In our data, population heterogeneity tends to be relatively small, whereas design and analytical heterogeneity are large. Our results should, however, be interpreted cautiously due to the limited number of studies and the large uncertainty in the heterogeneity estimates. We discuss several ways to parse and account for heterogeneity in the context of different methodologies.

Investigating Structural Model Fit Evaluation

A full structural equation model (SEM) typically consists of both a measurement model (describing relationships between latent variables and observed scale items) and a structural model (describing relationships among latent variables). However, often researchers are primarily interested in testing hypotheses related to the structural model while treating the measurement model as a necessary but not primary focus of the overall model. In this case, researchers often wish to isolate and just evaluate the fit of the structural model. In our research, we examine a two-stage approach that can compute the chi-square statistic and fit indices for evaluating only the fit of the structural model in a full SEM. We call these the structural chi-square statistic and structural fit indices. For structural fit indices, we focused on the root mean square error of approximation (RMSEA), comparative fit index (CFI), and standardized root mean square residual (SRMR). We developed several new versions of the structural chi-square statistic, structural fit indices, and confidence intervals (CIs) of the structural fit indices. Through a simulation study, we demonstrated that several versions of our newly developed structural chi-square statistic yielded the nominal Type-I error rate; and the same versions of the structural fit indices exhibited low bias and their corresponding CIs had high coverage rates. Therefore, we recommend researchers use these versions of the structural chi-square test of fit alongside the structural fit indices when evaluating the fit of the structural model.

Smoothed Bootstrap Methods for Hypothesis Testing

This paper demonstrates the application of smoothed bootstrap methods and Efron’s methods for hypothesis testing on real-valued data, right-censored data and bivariate data. The tests include quartile hypothesis tests, two sample medians and Pearson and Kendall correlation tests. Simulation studies indicate that the smoothed bootstrap methods outperform Efron’s methods in most scenarios, particularly for small datasets. The smoothed bootstrap methods provide smaller discrepancies between the actual and nominal error rates, which makes them more reliable for testing hypotheses.

Nonparametric bootstrap methods for hypothesis testing in the event of double-censored data

<abstract><p>This paper illustrated how nonparametric bootstrap methods for double-censored data can be used to conduct some hypothesis tests, such as quartiles' hypothesis tests. Through simulation studies, the smoothed bootstrap (SB) method performed better results than Efron's method in most scenarios, particularly for small datasets. The SB method provided smaller discrepancies between the actual and nominal error rates.</p></abstract>

Permutation-based multiple testing corrections for -values and confidence intervals for cluster randomized trials.

In this article, we derive and compare methods to derive P-values and sets of confidence intervals with strong control of the family-wise error rates and coverage for estimates of treatment effects in cluster randomized trials with multiple outcomes. There are few methods for P-value corrections and deriving confidence intervals, limiting their application in this setting. We discuss the methods of Bonferroni, Holm, and Romano and Wolf and adapt them to cluster randomized trial inference using permutation-based methods with different test statistics. We develop a novel search procedure for confidence set limits using permutation tests to produce a set of confidence intervals under each method of correction. We conduct a simulation-based study to compare family-wise error rates, coverage of confidence sets, and the efficiency of each procedure in comparison to no correction using both model-based standard errors and permutation tests. We show that the Romano-Wolf type procedure has nominal error rates and coverage under non-independent correlation structures and is more efficient than the other methods in a simulation-based study. We also compare results from the analysis of a real-world trial.

Stimulus-specific random effects inflate false-positive classification accuracy in multivariate-voxel-pattern-analysis: A solution with generalized mixed-effects modelling

When conducting multivariate-voxel pattern analysis (MVPA), researchers typically compute the average accuracy for each subject and statistically test if the average accuracy is different from the chance level across subjects (by-subject analysis). We argue that this traditional by-subject analysis leads to inflated Type-1 error rates, regardless of the type of machine learning method used (e.g., support vector machine). This is because by-subject analysis does not consider the variance attributed to the idiosyncratic features of the stimuli that have a common influence on all subjects (i.e., the random stimulus effect). As a solution, we proposed the use of generalized linear mixed-effects modelling to evaluate average accuracy. This method only requires post-classification data (i.e., it does not consider the type of classification methods used) and is easily implemented in the analysis pipeline with common statistical software (SPSS, R, Python, etc.). Using both statistical simulation and real fMRI data analysis, we demonstrated that the traditional by-subject method indeed increases Type-1 error rates to a considerable degree, while generalized mixed-effects modelling that incorporates random stimulus effects can indeed maintain the nominal Type-1 error rates.

Inference of gene-environment interaction from heterogeneous case-parent trios.

Introduction: In genetic epidemiology, log-linear models of population risk may be used to study the effect of genotypes and exposures on the relative risk of a disease. Such models may also include gene-environment interaction terms that allow the genotypes to modify the effect of the exposure, or equivalently, the exposure to modify the effect of genotypes on the relative risk. When a measured test locus is in linkage disequilibrium with an unmeasured causal locus, exposure-related genetic structure in the population can lead to spurious gene-environment interaction; that is, to apparent gene-environment interaction at the test locus in the absence of true gene-environment interaction at the causal locus. Exposure-related genetic structure occurs when the distributions of exposures and of haplotypes at the test and causal locus both differ across population strata. A case-parent trio design can protect inference of genetic main effects from confounding bias due to genetic structure in the population. Unfortunately, when the genetic structure is exposure-related, the protection against confounding bias for the genetic main effect does not extend to the gene-environment interaction term. Methods: We show that current methods to reduce the bias in estimated gene-environment interactions from case-parent trio data can only account for simple population structure involving two strata. To fill this gap, we propose to directly accommodate multiple population strata by adjusting for genetic principal components (PCs). Results and Discussion: Through simulations, we show that our PC adjustment maintains the nominal type-1 error rate and has nearly identical power to detect gene-environment interaction as an oracle approach based directly on population strata. We also apply the PC-adjustment approach to data from a study of genetic modifiers of cleft palate comprised primarily of case-parent trios of European and East Asian ancestry. Consistent with earlier analyses, our results suggest that the gene-environment interaction signal in these data is due to the self-reported European trios.

A Highly Stable Physically Unclonable Function Using Algorithm-Based Mismatch Hardening Technique in 28-nm CMOS

Physically unclonable function (PUF) is an emerging security solution for Internet of Things (IoT) devices. However, PUF faces a critical design challenge: responses should remain the same regardless of environmental conditions. This article presents an algorithm-based mismatch hardening technique that provides an effective and efficient solution to stabilizing PUF responses, which could be applied to various PUF circuits. The proposed stabilization technique combines multiple mismatches to achieve high reliability with minimum loss of utility. Moreover, the proposed approach requires only the available digitized PUF responses, avoiding any auxiliary measurement circuit to minimize additional implementation and testing overhead. The PUF test chip implemented in 28-nm CMOS technology shows that the proposed stabilization technique achieves a highly stable performance, lowering the nominal bit error rate (BER) to 0.0016%. It also exhibits excellent reliability with a worst-case BER of 0.16% across 0.7 to 1.2 V and −50 to 125°C, proving to be a promising candidate for security primitive for IoT applications.

LmerSeq: an R package for analyzing transformed RNA-Seq data with linear mixed effects models

BackgroundStudies that utilize RNA Sequencing (RNA-Seq) in conjunction with designs that introduce dependence between observations (e.g. longitudinal sampling) require specialized analysis tools to accommodate this additional complexity. This R package contains a set of utilities to fit linear mixed effects models to transformed RNA-Seq counts that properly account for this dependence when performing statistical analyses.ResultsIn a simulation study comparing lmerSeq and two existing methodologies that also work with transformed RNA-Seq counts, we found that lmerSeq was comprehensively better in terms of nominal error rate control and statistical power.ConclusionsExisting R packages for analyzing transformed RNA-Seq data with linear mixed models are limited in the variance structures they allow and/or the transformation methods they support. The lmerSeq package offers more flexibility in both of these areas and gave substantially better results in our simulations.

Develop Control Architectures to Enhance Soft Actuator Motion and Force

Study: Soft robots can achieve the desired range of motion for finger movement to match their axis of rotation with the axis of rotation of the human hand. The iterative design has been used to achieve data that makes the movement smooth and the range of movement wider, and the validity of the design has been confirmed through practical experiments. Limitation: The challenges facing this research are to reach the most significant inclined angle and increase the force generated by the actuator, which is the most complicated matter while maintaining the desired control accuracy. The motion capture system verifies the actual movement of the soft pneumatic actuator (SPA). A tracking system has been developed for SPA in action by having sensors to know the position and strength of the SPA. Results: The novelty of this research is that it gave better control of soft robots by selecting the proportional, integral, and derivative (PID) controller. The parameters were tuned using three different methods: ZN (Ziegler Nichols Method), GA (Genetic Algorism), and PSO (Particle Swarm Optimization). The optimization techniques were used in Methods 2 and 3 in order to reach the nominal error rate (0.6) and minimum overshoot (0.1%) in the shortest time (2.5 s). Impact: The effect of the proposed system in this study is to provide precise control of the actuator, which helps in medical and industrial applications, the most important of which are the transfer of things from one place to another and the process of medical rehabilitation for patients with muscular dystrophy. A doctor who treats finger muscle insufficiency can monitor a patient’s ability to reach a greater angle of flexion or increase strength by developing three treatment modalities to boost strength: Full Assisted Movement (FAM), Half Assisted Movement (HAM), and Resistance Movement (RM).

Outage and Error-Rate Analysis of NOMA-Inspired Multicarrier Underlay Cognitive Radio System

In this paper, the performance of a multicarrier underlay cognitive radio system is analyzed considering that the successive interference cancellation (SIC) method is used for detection. In the proposed system model, a multicarrier-based secondary user (SU) uses all the subcarriers of the primary user (PU) in underlay mode, guaranteeing the quality of service (QoS) to the primary system. The power allocation strategy is modified accordingly. With a fixed PU transmit power, power is allocated to the SU subcarriers to maximize its throughput while ensuring minimum throughput to PU. The system is analyzed for its performance considering an L-tap multipath Rayleigh fading channel. Analytical expression for outage probability at SU is derived in closed-form. Also, analytical expressions and closed-form approximations for average symbol error rates (SER) per subcarrier of PU and SU are obtained. The derived expressions are validated with the Monte-Carlo (MC) simulations. Further, the proposed scheme was implemented in an orthogonal frequency division multiplexing (OFDM)-based system with different detection schemes used at the receiver. SERs were compared through extensive simulations and it was noticed that SIC with channel estimation based on minimum-mean-squared-error (MMSE) often performed better than that based on least squares (LS) estimation. Observations reveal that the nominal error rates were achieved on the optimal choice of the parameters. The proposed system ensures sufficient throughput to SU, meanwhile eliminating the need for opportunistic sensing.

Ranking procedures for repeated measures designs with missing data: Estimation, testing and asymptotic theory.

We develop purely nonparametric methods for the analysis of repeated measures designs with missing values. Hypotheses are formulated in terms of purely nonparametric treatment effects. In particular, data can have different shapes even under the null hypothesis and therefore, a solution to the nonparametric Behrens-Fisher problem in repeated measures designs will be presented. Moreover, global testing and multiple contrast test procedures as well as simultaneous confidence intervals for the treatment effects of interest will be developed. All methods can be applied for the analysis of metric, discrete, ordinal, and even binary data in a unified way. Extensive simulation studies indicate a satisfactory control of the nominal type-I error rate, even for small sample sizes and a high amount of missing data (up to 30%). We apply the newly developed methodology to a real data set, demonstrating its application and interpretation.

A two-stage drop-the-losers design for time-to-event outcome using a historical control arm.

Phase II immuno-oncology clinical trials screen for efficacy an increasing number of treatments. In rare cancers, using historical control data is a pragmatic approach for speeding up clinical trials. The drop-the-losers design allows dropping off ineffective arms at interim analyses. We extended the original drop-the-losers design for a time-to-event outcome using a historical control through the one-sample log-rank statistic. Simulated trials featured three arms at the first stage, one at the second stage, nine scenarios, eight sample sizes with 5%- and 10%- nominal family-wise error rate (FWER). A numerical algorithm is provided to solve power calculations at the design stage. Our design was compared with a group of three independent single-arm trials (fixed design) with and without correction for multiplicity. Our design allowed strict control of the FWER at nominal levels while the misspecification of survival distribution and fixed design inflated the FWER up to three times the nominal level. The empirical power of our design increased with the sample size, the treatment effect and the number of effective treatments and dropped when more patients were recruited at the second stage. The fixed design with correction showed comparable power, while our design advantageously included more patients to the most promising arm. Recommendations for future applications are given. By taking advantage of the use of historical control data and a time-to-event outcome, the drop-the-losers design is a promising tool to meet the challenge of improving phase II clinical trials in immuno-oncology.

MCMSeq: Bayesian hierarchical modeling of clustered and repeated measures RNA sequencing experiments

BackgroundAs the barriers to incorporating RNA sequencing (RNA-Seq) into biomedical studies continue to decrease, the complexity and size of RNA-Seq experiments are rapidly growing. Paired, longitudinal, and other correlated designs are becoming commonplace, and these studies offer immense potential for understanding how transcriptional changes within an individual over time differ depending on treatment or environmental conditions. While several methods have been proposed for dealing with repeated measures within RNA-Seq analyses, they are either restricted to handling only paired measurements, can only test for differences between two groups, and/or have issues with maintaining nominal false positive and false discovery rates. In this work, we propose a Bayesian hierarchical negative binomial generalized linear mixed model framework that can flexibly model RNA-Seq counts from studies with arbitrarily many repeated observations, can include covariates, and also maintains nominal false positive and false discovery rates in its posterior inference.ResultsIn simulation studies, we showed that our proposed method (MCMSeq) best combines high statistical power (i.e. sensitivity or recall) with maintenance of nominal false positive and false discovery rates compared the other available strategies, especially at the smaller sample sizes investigated. This behavior was then replicated in an application to real RNA-Seq data where MCMSeq was able to find previously reported genes associated with tuberculosis infection in a cohort with longitudinal measurements.ConclusionsFailing to account for repeated measurements when analyzing RNA-Seq experiments can result in significantly inflated false positive and false discovery rates. Of the methods we investigated, whether they model RNA-Seq counts directly or worked on transformed values, the Bayesian hierarchical model implemented in the mcmseq R package (available at https://github.com/stop-pre16/mcmseq) best combined sensitivity and nominal error rate control.

Multiple comparison correction methods for whole-body magnetic resonance imaging.

.Purpose: Voxel-level hypothesis testing on images suffers from test multiplicity. Numerous correction methods exist, mainly applied and evaluated on neuroimaging and synthetic datasets. However, newly developed approaches like Imiomics, using different data and less common analysis types, also require multiplicity correction for more reliable inference. To handle the multiple comparisons in Imiomics, we aim to evaluate correction methods on whole-body MRI and correlation analyses, and to develop techniques specifically suited for the given analyses.Approach: We evaluate the most common familywise error rate (FWER) limiting procedures on whole-body correlation analyses via standard (synthetic no-activation) nominal error rate estimation as well as smaller prior-knowledge based stringency analysis. Their performance is compared to our anatomy-based method extensions.Results: Results show that nonparametric methods behave better for the given analyses. The proposed prior-knowledge based evaluation shows that the devised extensions including anatomical priors can achieve the same power while keeping the FWER closer to the desired rate.Conclusions: Permutation-based approaches perform adequately and can be used within Imiomics. They can be improved by including information on image structure. We expect such method extensions to become even more relevant with new applications and larger datasets.

Analyzing the Fit of IRT Models With the Hausman Test.

In this manuscript, the applicability of the Hausman test to the evaluation of item response models is investigated. The Hausman test is a general test of model fit. The test assesses whether for a model in question the parameter estimates of two different estimators coincide. The test can be implemented for item response models by comparing the parameter estimates of the marginal maximum likelihood estimator with the corresponding parameter estimates of a limited information estimator. For a correctly specified item response model, the difference of the two estimates is normally distributed around zero. The Hausman test can be used for the evaluation of item fit and global model fit. The performance of the test is evaluated in a simulation study. The simulation study suggests that the implemented versions of the test adhere to the nominal Type-I error rate well in samples of 1000 test takers and more. The test is also capable to detect misspecified item characteristic functions, but lacks power to detect violations of the conditional independence assumption.

Unweighted regression models perform better than weighted regression techniques for respondent-driven sampling data: results from a simulation study

BackgroundIt is unclear whether weighted or unweighted regression is preferred in the analysis of data derived from respondent driven sampling. Our objective was to evaluate the validity of various regression models, with and without weights and with various controls for clustering in the estimation of the risk of group membership from data collected using respondent-driven sampling (RDS).MethodsTwelve networked populations, with varying levels of homophily and prevalence, based on a known distribution of a continuous predictor were simulated using 1000 RDS samples from each population. Weighted and unweighted binomial and Poisson general linear models, with and without various clustering controls and standard error adjustments were modelled for each sample and evaluated with respect to validity, bias and coverage rate. Population prevalence was also estimated.ResultsIn the regression analysis, the unweighted log-link (Poisson) models maintained the nominal type-I error rate across all populations. Bias was substantial and type-I error rates unacceptably high for weighted binomial regression. Coverage rates for the estimation of prevalence were highest using RDS-weighted logistic regression, except at low prevalence (10%) where unweighted models are recommended.ConclusionsCaution is warranted when undertaking regression analysis of RDS data. Even when reported degree is accurate, low reported degree can unduly influence regression estimates. Unweighted Poisson regression is therefore recommended.

Phylogenetic tree-based microbiome association test.

Ecological patterns of the human microbiota exhibit high inter-subject variation, with few operational taxonomic units (OTUs) shared across individuals. To overcome these issues, non-parametric approaches, such as the Mann-Whitney U-test and Wilcoxon rank-sum test, have often been used to identify OTUs associated with host diseases. However, these approaches only use the ranks of observed relative abundances, leading to information loss, and are associated with high false-negative rates. In this study, we propose a phylogenetic tree-based microbiome association test (TMAT) to analyze the associations between microbiome OTU abundances and disease phenotypes. Phylogenetic trees illustrate patterns of similarity among different OTUs, and TMAT provides an efficient method for utilizing such information for association analyses. The proposed TMAT provides test statistics for each node, which are combined to identify mutations associated with host diseases. Power estimates of TMAT were compared with existing methods using extensive simulations based on real absolute abundances. Simulation studies showed that TMAT preserves the nominal type-1 error rate, and estimates of its statistical power generally outperformed existing methods in the considered scenarios. Furthermore, TMAT can be used to detect phylogenetic mutations associated with host diseases, providing more in-depth insight into bacterial pathology. The 16S rRNA amplicon sequencing metagenomics datasets for colorectal carcinoma and myalgic encephalomyelitis/chronic fatigue syndrome are available from the European Nucleotide Archive (ENA) database under project accession number PRJEB6070 and PRJEB13092, respectively. TMAT was implemented in the R package. Detailed information is available at http://healthstat.snu.ac.kr/software/tmat. Supplementary data are available at Bioinformatics online.

The Behrens–Fisher problem with covariates and baseline adjustments

The Welch-Satterthwaite t-test is one of the most prominent and often used statistical inference method in applications. The method is, however, not flexible with respect to adjustments for baseline values or other covariates, which may impact the response variable. Existing analysis of covariance methods are typically based on the assumption of equal variances across the groups. This assumption is hard to justify in real data applications and the methods tend to not control the type-1 error rate satisfactorily under variance heteroscedasticity. In the present paper, we tackle this problem and develop unbiased variance estimators of group specific variances, and especially of the variance of the estimated adjusted treatment effect in a general analysis of covariance model. These results are used to generalize the Welch-Satterthwaite t-test to covariates adjustments. Extensive simulation studies show that the method accurately controls the nominal type-1 error rate, even for very small sample sizes, moderately skewed distributions and under variance heteroscedasticity. A real data set motivates and illustrates the application of the proposed methods.

Confidence intervals for directly standardized rates using mid-p gamma intervals.

Directly standardized rates continue to be an integral tool for presenting rates for diseases that are highly dependent on age, such as cancer. Statistically, these rates are modeled as a weighted sum of Poisson random variables. This is a difficult statistical problem, because there are k observed Poisson variables and k unknown means. The gamma confidence interval has been shown through simulations to have at least nominal coverage in all simulated scenarios, but it can be overly conservative. Previous modifications to that method have closer to nominal coverage on average, but they do not achieve the nominal coverage bound in all situations. Further, those modifications are not central intervals, and the upper coverage error rate can be substantially more than half the nominal error. Here we apply a mid-p modification to the gamma confidence interval. Typical mid-p methods forsake guaranteed coverage to get coverage that is sometimes higher and sometimes lower than the nominal coverage rate, depending on the values of the parameters. The mid-p gamma interval does not have guaranteed coverage in all situations; however, in the (not rare) situations where the gamma method is overly conservative, the mid-p gamma interval often has at least nominal coverage. The mid-p gamma interval is especially appropriate when one wants a central interval, since simulations show that in many situations both the upper and lower coverage error rates are on average less than or equal to half the nominal error rate.