Equivalence Testing Procedure Research Articles

Expression of cystic fibrosis lung disease modifier genes in human airway models

BackgroundVariation in respiratory response to cystic fibrosis (CF) small molecule therapies is due in part to the contribution of CF lung disease modifier genes. Cultured human bronchial epithelia (HBE) is the gold standard respiratory model for assessing CF therapeutic efficacy but it is hard to access. Cultured human nasal epithelia (HNE) is proposed as a more accessible surrogate model but it is unknown whether the expression profile of the modifier genes are comparable between HNE and HBE which we assess here. MethodsRNA-sequencing was conducted on paired cultured and fresh HNE and HBE (n = 71 samples) collected from 21 individuals with CF. Genome-wide gene expression was first compared between cultured and fresh cells and then between cultured HNE and HBE based on an equivalence testing procedure we implemented. The co-expression relationships of CFTR and CF lung disease modifier genes were compared between cultured HNE and HBE to determine equivalent interactions. ResultsThe culturing process had little impact on the expression level of CF lung disease modifier genes. Over 90% of expressed genes showed significant equivalent expression level across cultured HNE and HBE (expression fold-change<2, FDR<0.1), including CFTR and CF lung disease modifier genes. The difference in co-expression relationships among these genes was not significant (p-value=0.99), suggesting their functional interactions are likely to be consistent in the two models. ConclusionsCultured HNE recapitulates the expression profile of CF lung disease modifier genes in cultured HBE, suggesting the biological processes involving these genes are likely to be consistent across the two models.

B-value and empirical equivalence bound: A new procedure of hypothesis testing.

In this study, we propose a two-stage procedure for hypothesis testing, where the first stage is conventional hypothesis testing and the second is an equivalence testing procedure using an introduced empirical equivalence bound (EEB). In 2016, the American Statistical Association released a policy statement on P-values to clarify the proper use and interpretation in response to the criticism of reproducibility and replicability in scientific findings. A recent solution to improve reproducibility and transparency in statistical hypothesis testing is to integrate P-values (or confidence intervals) with practical or scientific significance. Similar ideas have been proposed via the equivalence test, where the goal is to infer equality under a presumption (null) of inequality of parameters. However, the definition of scientific significance/equivalence can sometimes be ill-justified and subjective. To circumvent this drawback, we introduce the B-value and the EEB, which are both estimated from the data. Performing a second-stage equivalence test, our procedure offers an opportunity to improve the reproducibility of findings across studies.

Cross-scale quality assessment of a mechanistic cation exchange chromatography model.

Cation exchange chromatography (CEX) is an essential part of most monoclonal antibody (mAb) purification platforms. Process characterization and root cause investigation of chromatographic unit operations are performed using scale down models (SDM). SDM chromatography columns typically have the identical bed height as the respective manufacturing-scale, but a significantly reduced inner diameter. While SDMs enable process development demanding less material and time, their comparability to manufacturing-scale can be affected by variability in feed composition, mobile phase and resin properties, or dispersion effects depending on the chromatography system at hand. Mechanistic models can help to close gaps between scales and reduce experimental efforts compared to experimental SDM applications. In this study, a multicomponent steric mass-action (SMA) adsorption model was applied to the scale-up of a CEX polishing step. Based on chromatograms and elution pool data ranging from laboratory- to manufacturing-scale, the proposed modeling workflow enabled early identification of differences between scales, for example, system dispersion effects or ionic capacity variability. A multistage model qualification approach was introduced to measure the model quality and to understand the model's limitations across scales. The experimental SDM and the in silico model were qualified against large-scale data using the identical state of the art equivalence testing procedure. The mechanistic chromatography model avoided limitations of the SDM by capturing effects of bed height, loading density, feed composition, and mobile phase properties. The results demonstrate the applicability of mechanistic chromatography models as a possible alternative to conventional SDM approaches.

Equivalence Testing and the Second Generation P-Value

&#x0D; &#x0D; &#x0D; To move beyond the limitations of null-hypothesis tests, statistical approaches have been developed where the observed data are compared against a range of values that are equivalent to the absence of a meaningful effect. Specifying a range of values around zero allows researchers to statistically reject the presence of effects large enough to matter, and prevents practically insignificant effects from being interpreted as a statistically significant difference. We compare the behavior of the recently proposed second generation p-value (Blume, D’Agostino McGowan, Dupont, &amp; Greevy, 2018) with the more established Two One-Sided Tests (TOST) equivalence testing procedure (Schuirmann, 1987). We show that the two approaches yield almost identical results under optimal conditions. Under suboptimal conditions (e.g., when the confidence interval is wider than the equivalence range, or when confidence intervals are asymmetric) the second generation p-value becomes difficult to interpret. The second generation p-value is interpretable in a dichotomous manner (i.e., when the SGPV equals 0 or 1 because the confidence intervals lies completely within or outside of the equivalence range), but this dichotomous interpretation does not require calculations. We conclude that equivalence tests yield more consistent p-values, distinguish between datasets that yield the same second generation p-value, and allow for easier control of Type I and Type II error rates.&#x0D; &#x0D; &#x0D;

Assessment of the Equivalence of Low-Cost Sensors with the Reference Method in Measuring PM10 Concentration Using Selected Correction Functions

The aim of the work is to demonstrate the possibility of building models to correct the results of measurements of particulate matter PM10 concentrations obtained using low-cost devices. Such devices apply the optical method to values comparable with those obtained using the reference gravimetric method. An additional goal is to show that the results corrected in this way can be used to carry out the procedure for testing equivalence of these methods. The study used generalized regression models (GRMs) to construct corrective functions. The constructed models were assessed using the coefficients of determination and the methodology of calculating the measurement uncertainty of the device. Measurement data from the two tested devices and the reference method were used to estimate model parameters. The measurement data were collected on a daily basis from 1 February to 30 June 2018 in Nowy Sącz. Regression allowed building multiple models with various functional forms and very promising statistical properties as well as good ability to describe the variability of reference measurements. These models also had very low values of measurement uncertainty. Of all the models constructed, a linear model using the original PM10 concentrations from the tested devices, air humidity, and wind speed was chosen as the most accurate and simplest model. Apart from the coefficient of determination, expanded relative uncertainty served as the measure of quality of the obtained model. Its small value, much lower than 25%, indicates that after correcting the results it is possible to carry out the equivalence testing procedure for the low-cost devices and confirm the equivalence of the tested method with the reference method.

Equivalence Testing of a Youth Risk and Needs Assessment

Truancy can have a detrimental impact on student outcomes. Risk assessments are used to identify behavioral and emotional problems associated with school truancy. Although imperative for informing decisions about student’s welfare, risk assessments generally lack substantial validity evidence. Specifically, supporting score interpretation across cultural groups through measurement invariance (MI) is needed for such assessments. This study examined MI, specifically factor invariance (FI), of the Washington Assessment of the Risks and Needs of Students (WARNS) across African American, Latinx, and Caucasian students using confirmatory factor analysis with traditional FI criteria and the equivalence testing (ET) procedure. Results from traditional criteria suggested that the factor structure is similar across groups. The ET procedure demonstrated minor model misspecifications.

Robustness of statistical methods when measure is affected by ceiling and/or floor effect.

A simulation study investigated how ceiling and floor effect (CFE) affect the performance of Welch's t-test, F-test, Mann-Whitney test, Kruskal-Wallis test, Scheirer-Ray-Hare-test, trimmed t-test, Bayesian t-test, and the "two one-sided tests" equivalence testing procedure. The effect of CFE on the estimate of group difference and on its confidence interval, and on Cohen's d and on its confidence interval was also evaluated. In addition, the parametric methods were applied to data transformed with log or logit function and the performance was evaluated. The notion of essential maximum from abstract measurement theory is used to formally define CFE and the principle of maximum entropy was used to derive probability distributions with essential maximum/minimum. These distributions allow the manipulation of the magnitude of CFE through a parameter. Beta, Gamma, Beta prime and Beta-binomial distributions were obtained in this way with the CFE parameter corresponding to the logarithm of the geometric mean. Wald distribution and ordered logistic regression were also included in the study due to their measure-theoretic connection to CFE, even though these models lack essential minimum/maximum. Performance in two-group, three-group and 2 × 2 factor design scenarios was investigated by fixing the group differences in terms of CFE parameter and by adjusting the base level of CFE. In general, bias and uncertainty increased with CFE. Most problematic were occasional instances of biased inference which became more certain and more biased as the magnitude of CFE increased. The bias affected the estimate of group difference, the estimate of Cohen's d and the decisions of the equivalence testing methods. Statistical methods worked best with transformed data, albeit this depended on the match between the choice of transformation and the type of CFE. Log transform worked well with Gamma and Beta prime distribution while logit transform worked well with Beta distribution. Rank-based tests showed best performance with discrete data, but it was demonstrated that even there a model derived with measurement-theoretic principles may show superior performance. Trimmed t-test showed poor performance. In the factor design, CFE prevented the detection of main effects as well as the detection of interaction. Irrespective of CFE, F-test misidentified main effects and interactions on multiple occasions. Five different constellations of main effect and interactions were investigated for each probability distribution, and weaknesses of each statistical method were identified and reported. As part of the discussion, the use of generalized linear models based on abstract measurement theory is recommended to counter CFE. Furthermore, the necessity of measure validation/calibration studies to obtain the necessary knowledge of CFE to design and select an appropriate statistical tool, is stressed.

Testing for goodness rather than lack of fit of an X-chromosomal SNP to the Hardy-Weinberg model.

The problem of checking the genotype distribution obtained for some diallelic marker for compatibility with the Hardy-Weinberg equilibrium (HWE) condition arises also for loci on the X chromosome. The possible genotypes depend on the sex of the individual in this case: for females, the genotype distribution is trinomial, as in the case of an autosomal locus, whereas a binomial proportion is observed for males. Like in genetic association studies with autosomal SNPs, interest is typically in establishing approximate compatibility of the observed genotype frequencies with HWE. This requires to replace traditional methods tailored for detecting lack of fit to the model with an equivalence testing procedure to be derived by treating approximate compatibility with the model as the alternative hypothesis. The test constructed here is based on an upper confidence bound and a simple to interpret combined measure of distance between true and HWE conforming genotype distributions in female and male subjects. A particular focus of the paper is on the derivation of the asymptotic distribution of the test statistic under null alternatives which is not of the usual Gaussian form. A closed sample size formula is also provided and shown to behave satisfactorily in terms of the approximation error.

Response Rates for Surveys Completed With Paper-and-Pencil and Computers: Using Meta-Analysis to Assess Equivalence

The increasing number of self-report surveys being collected using computers has led to a body of literature examining the response rates for computerized surveys compared with the more traditional paper-and-pencil method. However, results from individual studies have been inconsistent, and the meta-analyses available on this topic have included studies from a restricted range of years and did not use proper statistical procedures for examining comparability. Consequently, we conducted a meta-analysis with 96 independent effect sizes spanning over two decades of studies; we also assessed potential moderators. Comparability was determined using confidence interval equivalence testing procedures. The meta-analysis indicated nonequivalence, with those in the paper-and-pencil condition being almost twice as likely to return surveys as those in the computer condition. There was large heterogeneity of variance, and 11 of the 18 potential moderators were significant. Two meta-regressions yielded only two significant unique moderators: population and type of measure. Results highlighted issues within the response rate literature that can be addressed in future studies, as well as provided an example of using equivalence testing in meta-analyses.

A Simulation Investigation of the Performance of Invariance Assessment Using Equivalence Testing Procedures

Factorial invariance assessment is central in the development of educational and psychological assessments. Establishing invariance of factor structures is key for building a strong score and inference validity argument and assists in establishing the fairness of score use. Fit indices and guidelines for judging a lack of invariance is an ongoing line of research. In this study, the authors examined the performance of the root mean squared error of approximation equivalence testing approach described by Yuan and Chan in the context of measurement invariance assessment. This investigation was completed through a simulation study in which several factors were varied, including sample size, type of invariance tested, and magnitude and percent of a lack of invariance. The findings generally support the use of equivalence testing for situations in which the indicator variables were normally distributed, particularly for total sample sizes of 200 or more.

Small sample equivalence tests for exponentiality

ABSTRACTWe consider small sample equivalence tests for exponentialy. Statistical inference in this setting is particularly challenging since equivalence testing procedures typically require much larger sample sizes, in comparison with classical “difference tests,” to perform well. We make use of Butler's marginal likelihood for the shape parameter of a gamma distribution in our development of small sample equivalence tests for exponentiality. We consider two procedures using the principle of confidence interval inclusion, four Bayesian methods, and the uniformly most powerful unbiased (UMPU) test where a saddlepoint approximation to the intractable distribution of a canonical sufficient statistic is used. We perform small sample simulation studies to assess the bias of our various tests and show that all of the Bayes posteriors we consider are integrable. Our simulation studies show that the saddlepoint-approximated UMPU method performs remarkably well for small sample sizes and is the only method that consistently exhibits an empirical significance level close to the nominal 5% level.

Equivalence Testing With Particle Size Distribution Data: Methods and Applications in the Development of Inhalative Drugs

ABSTRACTKey criteria of the quality of inhalative drugs are assessed in experiments generating so-called particle size distributions as data. Many experiments of that kind are carried out to demonstrate that necessary modifications to whatever part of the manufacturing process do not substantially change basic characteristics of an inhalable drug product. The equivalence testing procedures we derive for that purpose rely on different models accommodating the specific structure of such data and on different ways of specifying the region of nonrelevant differences. For each hypotheses formulation, three different tests are derived (two parametric and one asymptotically distribution-free procedures) and compared in terms of level and power. The results support the conclusion that the asymptotically distribution-free procedure exhibits surprisingly favorable properties. Supplementary materials for this article are available online.

Agreement Between VO2peak Predicted From PACER and One-Mile Run Time-Equated Laps

Purpose: This study examined the agreement between estimated peak oxygen consumption (VO2peak) obtained from the Progressive Aerobic Cardiovascular Endurance Run (PACER) fitness test and equated PACER laps derived from One-Mile Run time (MR). Methods: A sample of 680 participants (324 boys and 356 girls) in Grades 7 through 12 completed both the PACER and the MR assessments. MR time was converted to PACER laps (PACER-MEQ) using previously developed conversion algorithms. Agreement between PACER and PACER-MEQ VO2peak was examined using Pearson correlations, mean absolute percent error (MAPE), and equivalence testing procedures. Classification agreement based on health-related standards was examined using sensitivity, specificity, and Kappa statistics. Results: Overall agreement between estimated VO2peak obtained from the PACER and PACER-MEQ was high in boys, r(324) = .79, R2 = .63, and moderate in girls, r(356) = .57, R2 = .33. The MAPE for estimates obtained from PACER-MEQ was 10.3% and estimates were deemed equivalent to the PACER (43.1 ± 6.9 mL/kg/min vs. 44.6 ± 0.3 mL/kg/min). Classification agreement as illustrated by sensitivity and specificity ranged from 20.4% to 90.2% and was higher for classifications in the Healthy Fitness Zone (HFZ). Kappa statistics ranged from .14 to .51 and were also higher for the HFZ. Conclusions: Equated PACER laps can be used to obtain equivalent estimates of PACER VO2peak in groups of adolescents, but some disparities can be found when students' scores are classified into the Needs Improvement Zone.

Multivariate Equivalence Tests for Use in Pharmaceutical Development

Statistical equivalence analyses are well-established parts of many studies in the biomedical sciences. Also in pharmaceutical development and manufacturing equivalence testing methods are required in order to statistically establish similarities between machines, process components, or complete processes. This article presents a choice of multivariate equivalence testing procedures for normally distributed data as generalizations of existing univariate methods. In all derived methods, variability is interpreted as nuisance parameter. The use of the proposed methods in pharmaceutical development is demonstrated with a comparative analysis of dissolution profiles.

Range statistics and equivalence tests

This paper investigates range statistics and its application on the hypothesis testing of equivalence of mean of multiple systems. We develop an equivalence-testing procedure that allows unequal variances among systems and discuss how to estimate the required critical constant, that is, quantiles of a range. A table of critical constants is provided for implementing the procedure. An experimental performance evaluation demonstrates the validity and efficiency of the procedure.

Nonparametric equivalence testing with respect to the median difference

The problem of comparing two independent groups of univariate data in the sense of testing for equivalence is considered for a fully nonparametric setting. The distribution of the data within each group may be a mixture of both a continuous and a discrete component, and no assumptions are made regarding the way in which the distributions of the two groups of data may differ from each other - in particular, the assumption of a shift model is avoided. The proposed equivalence testing procedure for this scenario refers to the median of the independent difference distribution, i.e. to the median of the differences between independent observations from the test group and the reference group, respectively. The procedure provides an asymptotic equivalence test, which is symmetric with respect to the roles of 'test' and 'reference'. It can be described either as a two-one-sided-tests (TOST) approach, or equivalently as a confidence interval inclusion rule. A one-sided variant of the approach can be applied analogously to non-inferiority testing problems. The procedure may be generalised to equivalence testing with respect to quantiles other than the median, and is closely related to tolerance interval type inference.

Testing model nesting and equivalence.

When using existing technology, it can be hard or impossible to determine whether two structural equation models that are being considered may be nested. There is also no routine technology for evaluating whether two very different structural models may be equivalent. A simple nesting and equivalence testing (NET) procedure is proposed that uses random sample and model-reproduced moment matrices to evaluate both model nesting and equivalence. The analysis is "local" rather than "global" in nature, but its use with simulation or bootstrapping can imply global conclusions. Two standard applications of NET are to verify whether or not two proposed models are equivalent and whether a baseline model used in an incremental fit index is appropriately nested.

Equivalence Testing the Easy Way

The purpose of the article is to demonstrate an equivalence testing software application written under SAS^1 Institute software designed for use by pharmaceutical and other medical research professionals. Besides making the entire equivalence testing procedure easier and more efficient, the application “EquivEasy” offers three main advantages over similar software: a) testing for 3 x 3 in addition to 2 x 2 crossover designs, b) familiar SAS user environment, and c) export flexibility (MS Word, PDF, HTML). Two case studies are presented with report results provided in tabular and graphical form.

Equivalence Testing in Dose-Response Studies

Some developments are described in the case of multidose equivalence studies (multiple doses of the test substance and one dose of the reference). If equivalence cannot be proven for the test substance at one of the doses studied, the procedure uses linear interpolation to examine intermediate doses. This greatly increases the power of the equivalence testing procedure, provided that equivalence for an intermediate dose only is considered an acceptable result of the study. The closed test procedure for equivalence testing of the doses studied is also described in order to clarify methods presented in other statistical publications. It is often difficult to specify the equivalence criterion (delta) precisely, although it is taken as a known constant in many statistical publications. It is recommended that results be presented in a form that allows them to be checked against several different values of delta.

In vitro--in vivo correlation of dissolution, a time scaling problem? Transformation of in vitro results to the in vivo situation, using theophylline as a practical example.

Two principal approaches to demonstrating the continuous in vivo relevance of an in vitro dissolution test are outlined. The first uses the convolution technique to predict the concentration-time course in vivo; the second uses deconvolution as a mathematical tool to estimate the in vivo dissolution profile. The weighting function must be known to utilise either technique. Defined by the aim of the analysis the dose-normalized response to the oral solution is regarded as the weighting function (Impulse Response). In both cases the essential step is continuous comparison of the predicted time dependent data with actual readings of the same class. To permit the prediction of concentration-time data from in vitro dissolution data the basic equations for the transformation of the time base from in vitro to in vivo conditions are developed. The transformation is essential, since one cannot assume that the time scales for the in vitro and the in vivo experiment are definitely the same. The estimated in vivo dissolution profile using the deconvolution technique gives a hypothetical image of the true in vivo dissolution curve. Comparison with in vitro dissolution test results, using one of the equivalence testing procedures, reveals how closely and for how long the in vitro dissolution test simulates the in vivo dissolution process. For the formulation of theophylline studied, equivalence of the in vitro and the estimated in vivo dissolution profiles was not confirmed for the entire period of observation, but it was demonstrated for approximately the first 5 h. The later inequivalence is not due to possible non-linear or time-dependent kinetics of theophylline. There is a discussion of whether a change in pH, agitation of the formulation, diffusion conditions or the absorption rate constant along the gastrointestinal tract might explain the biphasic linear correlation of the in vitro and in vivo data observed.