Sequential Testing Procedure Research Articles

A Practical Framework for the Integration of Structural Data Into Perimetric Examinations.

We sought to develop and evaluate a practical framework that supports structurally enhanced perimetric examinations. Two perimetric strategies were compared: standard Zippy Estimation through Sequential Testing (ZEST) procedure, a traditional visual field test with population-based prior distributions, and structural-ZEST (S-ZEST), enhanced with individual optical coherence tomography data to determine the starting parameters. The integration and collection of data was facilitated by a bespoke application developed in Shiny R (R Studio). The test was implemented using the Open Perimetry Interface on the Compass perimeter (CentreVue-iCare, Italy). The strategies were evaluated via simulations and on 10 visually healthy participants. The usability of the application was assessed in a simulated environment with 10 test users. In simulations, the S-ZEST improved test speed in patients with glaucoma. In the practical implementation, there was a statistically significant decrease in the testing time (approximately 26%) and in the number of presentations per test with S-ZEST (P < 0.001). The structure-function relationship was similar between the two strategies. The time taken for users to complete the sequence of actions on the application was 52.9 ± 11.5 seconds (mean ± standard deviation). Structurally enhanced perimetric examination can significantly improve test time in healthy subjects and can be delivered through a user-friendly interface. Further testing will need to assess feasibility and performance of S-ZEST in patients with glaucoma. We have developed a user-friendly web application based within the Shiny environment for R, which implements an automated extraction of optical coherence tomography data from raw files and performs real-time calculations of structural features to inform the perimetric strategy.

Effects of exercise or metformin on myokine concentrations in patients with breast and colorectal cancer: A phase II multi-centre factorial randomized trial.

Physical activity and metformin pharmacotherapy are associated with improved clinical outcomes in breast and colorectal cancer survivors. Myokines are cytokines secreted from skeletal muscle that may mediate these associations. This hypothesis-generating analysis used biospecimens collected from a multi-centre 2×2 factorial randomized design of 116 patients with stage I-III breast and colorectal cancer who were randomized to 12weeks of (1) aerobic exercise (moderate intensity titrated to 220min/week); (2) metformin (850mg daily for 2weeks and then titrated to 850mg twice per day); (3) aerobic exercise and metformin; or (4) control. Fourteen myokines were quantified using a multiplex panel. Myokine concentrations were log-transformed, and main effects analyses were conducted using linear mixed-effects regression models. The type I error rate was controlled with the Holm sequential testing procedure. Randomization to exercise increased leukaemia inhibitory factor (1.26pg/mL, 95% confidence interval [CI]: 0.69, 1.84; adjusted P=0.001) and interleukin-15 (2.23pg/mL, 95% CI: 0.87, 3.60; adjusted P=0.013) compared with randomization to no exercise. Randomization to metformin decreased apelin (-2.69pg/mL, 95% CI: -4.31, -1.07; adjusted P=0.014) and interleukin-15 (-1.74pg/mL, 95% CI: -2.79, -0.69; adjusted P=0.013) compared with randomization to no metformin. Metformin decreased myostatin, irisin, oncostatin M, fibroblast growth factor 21 and osteocrin; however, these changes were not statistically significant after correction for multiple comparisons. This pilot study demonstrates that randomization to exercise and metformin elicit unique effects on myokine concentrations in cancer patients. This hypothesis-generating observation warrants further basic, translational and clinical investigation and replication.

Adaptive two-stage seamless sequential design for clinical trials

ABSTRACT We propose an adaptive sequential testing procedure for the selection and testing of multiple treatment options, such as dose/regimen, different drugs, sub-populations, endpoints, or a mixture of them in a seamlessly combined phase II/III trial. The selection is to be made at the end of phase 2 stage. Unlike in many of the published literature, the selection rule is not required to be to “select the best”, and does not need to be pre-specified, which provides flexibility and allows the trial investigators to use any efficacy and safety information/criteria, or surrogate or intermediate endpoint to make the selection. Sample size and power calculations are provided. The calculations have been confirmed to be accurate by simulations. Interim analysis can be performed after the selection, sample size can be modified if the observed efficacy deviates from the assumed. Inference after the trial, including p-value, median unbiased point estimate and confidence intervals, are provided. By applying a dominance theorem, the procedure can be applied to normal, binary, Poisson, negative binomial distributed endpoints and time-to-event endpoints, and a mixture of these distributions (in trials involving endpoint selection).

Parametric risk-neutral density estimation via finite lognormal-Weibull mixtures

This paper proposes a new parametric risk-neutral density (RND) estimator based on a finite lognormal-Weibull mixture (LWM) density. We establish the consistency and asymptotic normality of the LWM method in a general misspecified parametric framework. Based on the theoretical results, we propose a sequential test procedure to evaluate the goodness-of-fit of the LWM model, which leads to an adaptive choice for the number and type of mixture components. Our simulation results show that, in finite samples with various observation error specifications, the LWM method can approximate complex RNDs generated by state-of-the-art multi-factor stochastic volatility models with a few (typically less than 4) mixtures. Application of the LWM model on index options confirms its reliability in recovering empirical RNDs with a heavy left tail or bimodality, which can be incorrectly identified as bimodality or a heavy left tail by existing (semi)-nonparametric methods if the goodness-of-fit to the observed data is ignored.

Sequential hypothesis testing for selecting the number of changepoints in segmented regression models

Segmented regression is widely used in many disciplines, especially when dealing with environmental data. This paper deals with the problem of selecting the correct number of changepoints in segmented regression models. A review of the usual selection criteria, namely information criteria and hypothesis testing, is provided. We enhance the latter method by proposing a novel sequential hypothesis testing procedure to address this problem. Our sequential procedure’s performance is compared to methods based on information-based criteria through simulation studies. The results show that our proposal performs similarly to its competitors for the Gaussian, Binomial, and Poisson cases. Finally, we present two applications to environmental datasets of crime data in Valencia and global temperature land data.

Using sequential statistical tests for efficient hyperparameter tuning

Hyperparameter tuning is one of the most time-consuming parts in machine learning. Despite the existence of modern optimization algorithms that minimize the number of evaluations needed, evaluations of a single setting may still be expensive. Usually a resampling technique is used, where the machine learning method has to be fitted a fixed number of k times on different training datasets. The respective mean performance of the k fits is then used as performance estimator. Many hyperparameter settings could be discarded after less than k resampling iterations if they are clearly inferior to high-performing settings. However, resampling is often performed until the very end, wasting a lot of computational effort. To this end, we propose the sequential random search (SQRS) which extends the regular random search algorithm by a sequential testing procedure aimed at detecting and eliminating inferior parameter configurations early. We compared our SQRS with regular random search using multiple publicly available regression and classification datasets. Our simulation study showed that the SQRS is able to find similarly well-performing parameter settings while requiring noticeably fewer evaluations. Our results underscore the potential for integrating sequential tests into hyperparameter tuning.

A study on permanent deformation and fatigue damage interaction in asphalt concrete

The dominant load-induced damages, permanent deformation (PD) and fatigue cracking (F), are traditionally predicted separately by taking temperature as a variable of departure. This paper presents an experimental study of the two damages and interaction using sequential test procedure (STP) based on ‘sequential damage’. The STP is conducted in PD-F and F-PD sequences on each specimen using a creep-recovery and cyclic fatigue tests. The effect of strain hardening on fatigue cracking is studied in the PD-F sequence, and the impact of fatigue cracking on permanent deformation is explored using the F-PD sequence. First, the shear deformation in tertiary stage of creep recovery is investigated using the dissipated energy ratio criterion and a new fourth creep phase is obtained. Following the PD-F sequence, strain-hardening is found a significant accelerator of fatigue damage rate, particularly on aged and laboratory produced mixes. The fatigue tests without considering the strain-hardening effect underestimate fatigue damage rate. In the F-PD sequence, the effect of pre-existing crack (up to 40% modulus reduction) on permanent deformation is found marginal. The sequential test and damage approach is an effective way to analyze interaction between damage modes and evaluate asphalt mixtures.

Adaptive Multiple Comparison Sequential Design (AMCSD) for clinical trials

ABSTRACT We propose an adaptive sequential testing procedure for clinical trials that test the efficacy of multiple treatment options, such as dose/regimen, different drugs, sub-populations, endpoints, or a mixture of them in one trial. At any interim analyses, sample size re-estimation can be conducted, and any option can be dropped for lack of efficacy or unsatisfactory safety profile. Inference after the trial, including p-value, conservative point estimate and confidence intervals, are provided.

Selecting a significance level in sequential testing procedures for community detection

While there have been numerous sequential algorithms developed to estimate community structure in networks, there is little available guidance and study of what significance level or stopping parameter to use in these sequential testing procedures. Most algorithms rely on prespecifiying the number of communities or use an arbitrary stopping rule. We provide a principled approach to selecting a nominal significance level for sequential community detection procedures by controlling the tolerance ratio, defined as the ratio of underfitting and overfitting probability of estimating the number of clusters in fitting a network. We introduce an algorithm for specifying this significance level from a user-specified tolerance ratio, and demonstrate its utility with a sequential modularity maximization approach in a stochastic block model framework. We evaluate the performance of the proposed algorithm through extensive simulations and demonstrate its utility in controlling the tolerance ratio in single-cell RNA sequencing clustering by cell type and by clustering a congressional voting network.

Testing for the appropriate level of clustering in linear regression models

The overwhelming majority of empirical research that uses cluster-robust inference assumes that the clustering structure is known, even though there are often several possible ways in which a dataset could be clustered. We propose two tests for the correct level of clustering in regression models. One test focuses on inference about a single coefficient, and the other on inference about two or more coefficients. We provide both asymptotic and wild bootstrap implementations. The proposed tests work for a null hypothesis of either no clustering or “fine” clustering against alternatives of “coarser” clustering. We also propose a sequential testing procedure to determine the appropriate level of clustering. Simulations suggest that the bootstrap tests perform very well under the null hypothesis and can have excellent power. An empirical example suggests that using the tests leads to sensible inferences.

Multi‐purpose open‐end monitoring procedures for multivariate observations based on the empirical distribution function

We propose non‐parametric open‐end sequential testing procedures that can detect all types of changes in the contemporary distribution function of possibly multivariate observations. Their asymptotic properties are theoretically investigated under stationarity and under alternatives to stationarity. Monte Carlo experiments reveal their good finite‐sample behavior in the case of continuous univariate, bivariate and trivariate observations. A short data example concludes the work.

A seasonality-adjusted sequential test for vaccine safety surveillance.

Post-market active safety monitoring is important for the timely capture of safety signals associated with exposure to a new vaccine or drug. The group sequential analysis is a common method employed in safety surveillance. Specifically, it compares the post-vaccination incidence of adverse event (AE) in a vaccinated population with a pre-specified reference level by sequentially conducting hypothesis testing during the surveillance. When the number of AEs is "too high", a safety signal is identified. If the null hypothesis is never rejected, the vaccine is considered safe. Such an approach does not account for either the variation in determining the reference risk from a control population or the seasonality effect. Furthermore, not rejecting the null could be due to a lack of power and cannot always be interpreted as proof of safety. In this paper, we proposed a new group sequential test procedure fully accounting for both seasonality and variation from the historical controls. More importantly, we proposed to construct a confidence interval for the relative AE risk between the exposed and control groups at the end of the study, which can be used to quantify the safety of the vaccine. The proposed method is illustrated via real-data examples on anaphylaxis and examined by extensive simulationstudies.

Inference in Heavy-Tailed Nonstationary Multivariate Time Series

We study inference on the common stochastic trends in a nonstationary, N-variate time series yt , in the possible presence of heavy tails. We propose a novel methodology which does not require any knowledge or estimation of the tail index, or even knowledge as to whether certain moments (such as the variance) exist or not, and develop an estimator of the number of stochastic trends m based on the eigenvalues of the sample second moment matrix of yt . We study the rates of such eigenvalues, showing that the first m ones diverge, as the sample size T passes to infinity, at a rate faster by O ( T ) than the remaining N – m ones, irrespective of the tail index. We thus exploit this eigen-gap by constructing, for each eigenvalue, a test statistic which diverges to positive infinity or drifts to zero according to whether the relevant eigenvalue belongs to the set of the first m eigenvalues or not. We then construct a randomized statistic based on this, using it as part of a sequential testing procedure, ensuring consistency of the resulting estimator of m. We also discuss an estimator of the common trends based on principal components and show that, up to a an invertible linear transformation, such estimator is consistent in the sense that the estimation error is of smaller order than the trend itself. Importantly, we present the case in which we relax the standard assumption of iid innovations, by allowing for heterogeneity of a very general form in the scale of the innovations. Finally, we develop an extension to the large dimensional case. A Monte Carlo study shows that the proposed estimator for m performs particularly well, even in samples of small size. We complete the article by presenting two illustrative applications covering commodity prices and interest rates data. Supplementary materials for this article are available online.

Order selection with confidence for finite mixture models

The determination of the number of mixture components (the order) of a finite mixture model has been an enduring problem in statistical inference. We prove that the closed testing principle leads to a sequential testing procedure (STP) that allows for confidence statements to be made regarding the order of a finite mixture model. We construct finite sample tests, via data splitting and data swapping, for use in the STP, and we prove that such tests are consistent against fixed alternatives. Simulation studies and real data examples are used to demonstrate the performance of the finite sample tests-based STP, yielding practical recommendations of their use as confidence estimators in combination with point estimates such as the Akaike information or Bayesian information criteria. In addition, we demonstrate that a modification of the STP yields a method that consistently selects the order of a finite mixture model, in the asymptotic sense. Our STP is not only applicable for order selection of finite mixture models, but is also useful for making confidence statements regarding any sequence of nested models.

SEQUENTIALLY ESTIMATING THE STRUCTURAL EQUATION BY POWER TRANSFORMATION

This study provides an econometric methodology to test a linear structural relationship among economic variables. We propose the so-called distance-difference (DD) test and show that it has omnibus power against arbitrary nonlinear structural relationships. If the DD-test rejects the linear model hypothesis, a sequential testing procedure assisted by the DD-test can consistently estimate the degree of a polynomial function that arbitrarily approximates the nonlinear structural equation. Using extensive Monte Carlo simulations, we confirm the DD-test’s finite sample properties and compare its performance with the sequential testing procedure assisted by the J-test and moment selection criteria. Finally, through investigation, we empirically illustrate the relationship between the value-added and its production factors using firm-level data from the United States. We demonstrate that the production function has exhibited a factor-biased technological change instead of Hicks-neutral technology presumed by the Cobb–Douglas production function.

Assessment of a Full-Scale Unreinforced Stone Masonry Building Tested on a Shaking Table by Inverse Engineering

The material deterioration of an unreinforced stone masonry (URSM) building, due to subsequent dynamic loadings of increasing intensity on a shaking table, is investigated by means of inverse engineering, i.e. calibrating a finite element (FE) model to the experimental response data. The mechanical properties of the structure were initially estimated by preliminary characterisation tests. A two-storey full scale URSM building was tested on a shaking table using a sequential testing procedure of stationary and strong motion vibrations. The building was submitted to five uniaxial time-histories with gradually increasing intensity on a shaking table at the EUCENTRE laboratory (Pavia, Italy) up to a near collapse damage state, each one followed by a stationary vibration test. A frequency domain calibration was carried out to extract the mechanical properties of the equivalent elastic model. To this end, the stationary measurements were used to build up the state-space model. On the other hand, a recognition model was employed using the finite element method (FEM), whose stiffness and mass matrices were used to derive the corresponding analytical state-space model, which was compared to the experimental one. The calibration of the model against the experimental dynamic results includes increased complexity and high computational effort. Through an iterative optimisation trial and error procedure, the mechanical properties of masonry and the shear modulus of the flexible diaphragm of the structure for each test phase were derived. It is shown that the deterioration is more intense for the shear modulus of the walls compared to their elastic modulus. The ratio of the in-plane shear to the elastic modulus decreases substantially. The deterioration of the shear modulus of the timber floors is comparable with those of masonry walls.

CUSUMIN: A cumulative sum interval design for cancer phase I dose finding studies.

Recently, model-assisted designs, including the Bayesian optimal interval (BOIN) design with optimal thresholds to determine the dose for the next cohort, have been proposed for cancer phase I studies. Model-assisted designs are useful because of their good performance as model-based designs in addition to their algorithm-based simplicity. In BOIN, escalation and de-escalation based on boundaries can be understood as a type of change point detection based on a sequential test procedure. Notably, the sequential test procedure is used in a wide range of fields and is known for its application to control charts, statistical monitoring methods used for detecting abnormalities in manufacturing processes. In control charts, abnormalities are detected if the control chart statistics are observed to be outside of the optimal boundaries. The cumulative sum (CUSUM) statistic, which is developed for control chart applications, derives higher power under the same erroneous judgment rate. Hence, it is expected that a more efficient model-assisted design can be achieved by the application of CUSUM statistics. In this study, a model-assisted design based on the CUSUM statistic is proposed. In the proposed design, the dose for the next cohort is decided by CUSUM statistics calculated from the counts of the dose-limiting toxicity and pre-defined boundaries, based on the CUSUM control chart scheme. Intensive simulation shows that our proposed method performs better than BOIN, and other representative model-assisted designs, including modified toxicity probability interval (mTPI) and Keyboard, in terms of controlling over-dosing rates while maintaining similar performance in the determination of maximum tolerated dose.

A sequential test for assessing the effectiveness of response strategies during an emerging epidemic.

In epidemiology, the fatality rate is an important indicator of disease severity and has been used to evaluate the effects of new treatments. During an emerging epidemic with limited resources, monitoring the changes in fatality rate can also provide signals on the evaluation of government policies and healthcare quality, which helps to guide public health decision. A statistical test is developed in this paper to detect changes in fatality rate over time during the course of an emerging infectious disease. A major advantage of the proposed test is that it only requires the regularly reported numbers of deaths and recoveries, which meets the actual need as detailed surveillance data are hard to collect during the course of an emerging epidemic especially the deadly infectious diseases with large magnitude. In addition, with the sequential testing procedure, the effective measures can be detected at the earliest possible time to provide guidance to policymakers for swift action. Simulation studies showed that the proposed test performs well and is sensitive in picking up changes in the fatality rate. The test is applied to the 2014-2016 Ebola outbreak in Sierra Leone forillustration.

Monitoring Value-at-Risk and Expected Shortfall Forecasts

This paper introduces formal monitoring procedures as a risk-management tool. Continuously monitoring risk forecasts allows practitioners to swiftly review and update their forecasting procedures as soon as forecasts turn inadequate. Similarly, regulators may take timely action in case reported risk forecasts become poor. Extant (one-shot) backtests require, however, that all data are available prior to testing and are not informative of when inadequacies might have occurred. To monitor value-at-risk and expected shortfall forecasts “online”—that is, as new observations become available—we construct sequential testing procedures. We derive the exact finite-sample distributions of the proposed procedures and discuss the suitability of asymptotic approximations. Simulations demonstrate good behavior of our exact procedures in finite samples. An empirical application to major stock indices during the COVID-19 pandemic illustrates the economic benefits of our monitoring approach. This paper was accepted by Agostino Capponi, finance. Funding: Y. Hoga gratefully acknowledges support of the German Research Foundation [Grant HO 6305/1-1]. Supplemental Material: The online appendix and data are available at https://doi.org/10.1287/mnsc.2022.4460 .

ON MULTIPLE STRUCTURAL BREAKS IN DISTRIBUTION: AN EMPIRICAL CHARACTERISTIC FUNCTION APPROACH

We estimate and test for multiple structural breaks in distribution via an empirical characteristic function approach. By minimizing the sum of squared generalized residuals, we can consistently estimate the break fractions. We propose a sup-F type test for structural breaks in distribution as well as an information criterion and a sequential testing procedure to determine the number of breaks. We further construct a class of derivative tests to gauge possible sources of structural breaks, which is asymptotically more powerful than the smoothed nonparametric tests for structural breaks. Simulation studies show that our method performs well in determining the appropriate number of breaks and estimating the unknown breaks. Furthermore, the proposed tests have reasonable size and excellent power in finite samples. In an application to exchange rate returns, our tests are able to detect structural breaks in distribution and locate the break dates. Our tests also indicate that the documented breaks appear to occur in variance and higher-order moments, but not so often in mean.