Adaptive Testing Procedure Research Articles

Adaptive two-stage seamless sequential design for clinical trials

ABSTRACT We propose an adaptive sequential testing procedure for the selection and testing of multiple treatment options, such as dose/regimen, different drugs, sub-populations, endpoints, or a mixture of them in a seamlessly combined phase II/III trial. The selection is to be made at the end of phase 2 stage. Unlike in many of the published literature, the selection rule is not required to be to “select the best”, and does not need to be pre-specified, which provides flexibility and allows the trial investigators to use any efficacy and safety information/criteria, or surrogate or intermediate endpoint to make the selection. Sample size and power calculations are provided. The calculations have been confirmed to be accurate by simulations. Interim analysis can be performed after the selection, sample size can be modified if the observed efficacy deviates from the assumed. Inference after the trial, including p-value, median unbiased point estimate and confidence intervals, are provided. By applying a dominance theorem, the procedure can be applied to normal, binary, Poisson, negative binomial distributed endpoints and time-to-event endpoints, and a mixture of these distributions (in trials involving endpoint selection).

Adaptive bootstrap tests for composite null hypotheses in the mediation pathway analysis.

Mediation analysis aims to assess if, and how, a certain exposure influences an outcome of interest through intermediate variables. This problem has recently gained a surge of attention due to the tremendous need for such analyses in scientific fields. Testing for the mediation effect (ME) is greatly challenged by the fact that the underlying null hypothesis (i.e. the absence of MEs) is composite. Most existing mediation tests are overly conservative and thus underpowered. To overcome this significant methodological hurdle, we develop an adaptive bootstrap testing framework that can accommodate different types of composite null hypotheses in the mediation pathway analysis. Applied to the product of coefficients test and the joint significance test, our adaptive testing procedures provide type I error control under the composite null, resulting in much improved statistical power compared to existing tests. Both theoretical properties and numerical examples of the proposed methodology are discussed.

An enhanced outlier processing approach based on the resilient mathematical model compensation in GNSS precise positioning and navigation

The abnormal measurements are widely existent in Global Navigation Satellite System (GNSS) precise positioning and navigation mainly because of the diffraction, reflection, refraction, and even non-line-of-sight reception. However, when multiple outliers exist in GNSS measurements, traditional methods including test procedure or robust estimation usually cannot work well. This study proposed an enhanced outlier processing approach based on the resilient mathematical model compensation. Specifically, first, to avoid excessive deletion, the total number of measurements is considered in the adaptive test procedure with the help of a scale factor. Second, in adaptive robust estimation, the total number of remaining measurements is also considered, thus making it more compatible with the adaptive test procedure. In addition, to overcome the potential inappropriate reweighting operator, different shrinking factors are adopted for code and phase measurements according to their precision, respectively. To verify the effectiveness of the proposed method, one static monitoring experiment and one kinematic vehicle experiment were conducted, where the method without outlier processing, traditional test procedure, traditional robust estimation, and the proposed method were all used. For the static experiment, the ambiguity resolution and positioning solutions of the proposed method perform best. The positioning accuracy of the float and fixed solutions can be improved by approximately 67.4% and 77.6% on average under challenging environments, respectively. For the kinematic experiment, the performance is also the best in terms of positioning availability and accuracy by using the proposed method.

Adaptive Multiple Testing Procedure for Clinical Trials with Urn Allocation

This work combines the Urn allocation and O’Brien and Fleming multiple testing procedure to compare two treatments in clinical trials in a novel way. It is shown that this approach overcomes the constraints that previously made it challenging to apply the original adaptive design to clinical trials. The method provides unique flexibility, enabling trials to be stopped early if one treatment shows it is superior without compromising the efficiency of the original multiple testing procedure in terms of type I error rate and power. Experimental data and simulated case examples are used to illustrate the efficacy and robustness of this original approach and its potential for usage in a variety of clinical settings.

Adaptive Multiple Comparison Sequential Design (AMCSD) for clinical trials

ABSTRACT We propose an adaptive sequential testing procedure for clinical trials that test the efficacy of multiple treatment options, such as dose/regimen, different drugs, sub-populations, endpoints, or a mixture of them in one trial. At any interim analyses, sample size re-estimation can be conducted, and any option can be dropped for lack of efficacy or unsatisfactory safety profile. Inference after the trial, including p-value, conservative point estimate and confidence intervals, are provided.

Enhancing Computerized Adaptive Testing with Batteries of Unidimensional Tests.

The article presents a new computerized adaptive testing (CAT) procedure for use with batteries of unidimensional tests. At each step of testing, the estimate of a certain ability is updated on the basis of the response to the latest administered item and the current estimates of all other abilities measured by the battery. The information deriving from these abilities is incorporated into an empirical prior that is updated each time that new estimates of the abilities are computed. In two simulation studies, the performance of the proposed procedure is compared with that of a standard procedure for CAT with batteries of unidimensional tests. The proposed procedure yields more accurate ability estimates in fixed-length CATs, and a reduction of test length in variable-length CATs. These gains in accuracy and efficiency increase with the correlation between the abilities measured by the batteries.

Effects of mode and medium in reading comprehension tests on cognitive load

Digital media are becoming increasingly prevalent in the assessment of student achievement. Assessments that use digital media rather than pencil-and-paper (PPT) present the test materials on a screen (computer-based test: CBT) and can apply adaptive testing procedures (computer adaptive test: CAT). Based on cognitive load theory, presentation differences between the screen and paper might impact the cognitive load experienced during a reading comprehension test. In addition, working memory resource depletion might be indicative of an accumulation of cognitive load over the course of a test. Due to their active alignment of item difficulty and person ability, cognitive load might increase more in CATs than in PPTs or CBTs. Research has mostly compared test scores resulting from these different test formats, while systematic differences in the testing experience, such as experienced cognitive load, have received less scholarly attention. This study investigated how the three test formats of PPT, CBT, CAT affect examinees' level of experienced cognitive load in the middle and at the end of a reading comprehension test. In a between- and within-subject design, 212 German fourth graders (age: M = 9.44, SD = 0.59) were randomly assigned to a standardized reading comprehension test administered in one of the three test formats (between-subject: CBT, CAT, or PPT). Linear mixed-effects models revealed no significant mean differences in experienced cognitive load between the three test conditions, but a higher rate of increase in cognitive load in the adaptive test, although the effects were relatively small. Implications for future reading comprehension assessments using digital media are discussed.

Automated Speech Audiometry for Integrated Voice Over Internet Protocol Communication Services.

Problems in speech recognition are often apparent in telecommunication situations. For ecologically valid assessments of such conditions, it is important to quantify the impact of real environments including acoustic conditions at a far-end communication device and all paths of transmission degradation. This study presents an automated matrix sentence test procedure based on automatic speech recognition (ASR) integrated in a Voice over Internet Protocol (VoIP) infrastructure and compares the individual effects of transmission degradations with results from laboratory measurements. Speech recognition thresholds (SRTs) were measured in 16 normal-hearing subjects in four test conditions: (a) a laboratory condition guided by a human experimenter, (b) a laboratory condition with reduced bandwidth and (c) additionally reduced headset quality to simulate typical communication systems, and (d) an automated, ASR-controlled adaptive test procedure over a real VoIP infrastructure. Errors of the ASR system were analyzed to show possible effects on measurement outcome Results: Measured SRTs showed a highly significant correlation (r = .93) between the fully automatic and "laboratory" conditions, with a constant bias of about 1 dB indicating a linear shift of the data without affecting the distribution around the mean. The individual impact of the different system degradations on SRTs could be quantified Conclusions: This study provides a proof of concept for automated ASR-based SRT measurements over VoIP systems for speech audiometric testing in real communication systems, as it produced results comparable to traditional laboratory settings for this group of 16 normal-hearing subjects. This makes VoIP services a promising candidate for speech audiometric testing in real communication systems.

Estimation of the proportion of true null hypotheses under sparse dependence: Adaptive FDR controlling in microarray data.

The proportion of non-differentially expressed genes is an important quantity in microarray data analysis and an appropriate estimate of the same is used to construct adaptive multiple testing procedures. Most of the estimators for the proportion of true null hypotheses based on the thresholding, maximum likelihood and density estimation approaches assume independence among the gene expressions. Usually, sparse dependence structure is natural in modelling associations in microarray gene expression data and hence it is necessary to develop methods for accommodating the sparse dependence well within the framework of existing estimators. We propose a clustering based method to put genes in the same group that are not coexpressed using the estimated high dimensional correlation structure under sparse assumption as dissimilarity matrix. This novel method is applied to three existing estimators for the proportion of true null hypotheses. Extensive simulation study shows that the proposed method improves an existing estimator by making it less conservative and the corresponding adaptive Benjamini-Hochberg algorithm more powerful. The proposed method is applied to a microarray gene expression dataset of colorectal cancer patients and the results show gain in terms of number of differentially expressed genes. The R code is available at https://github.com/aniketstat/Proportiontion-of-true-null-under-sparse-dependence-2021.

Language-Independent Hearing Screening - Increasing the Feasibility of a Hearing Screening Self-Test at School-Entry.

A tablet-based language-independent self-test involving the recognition of ecological sounds in background noise, the Sound Ear Check (SEC), was adapted to make it feasible for young children. Two experiments were conducted. The first experiment investigated the SEC‘s feasibility, as well as its sensitivity and specificity for detecting childhood hearing loss with a monaural adaptive test procedure. In the second experiment, the SEC sounds, noise, and test format were adapted based on the findings of the first experiment. The adaptations were combined with three test procedures, one similar to the one used in Experiment 1, one presenting the sounds dichotically in diotic noise, and one presenting all the sounds with a fixed signal-to-noise ratio and a stopping rule. Results in young children show high sensitivity and specificity to detect different grades of conductive and sensorineural hearing loss (70–90%). When using an adaptive, monaural procedure, the test duration was approximately 6 min, and 17% of the results obtained were unreliable. Adaptive staircase analyses showed that the unreliable results probably occur due to attention/motivation loss. The test duration could be reduced to 3-4 min with adapted test formats without decreasing the test-retest reliability. The unreliable test results could be reduced from 17% to as low as 5%. However, dichotic presentation requires longer training, reducing the dichotic test format‘s feasibility.

Shortening the Leuven Perceptual Organization Screening Test with item response theory and confirmatory factor analysis.

The Leuven Perceptual Organization Screening Test (L-POST) is a test for midlevel visual perceptual deficits after brain injury. Previous research demonstrated reliable test scores and valid test score interpretations. However, the test's duration (20-35 min) is too long for a screening approach for all patients in clinical practice. Our aim was to shorten L-POST to 10-15 min based on statistical criteria of the items. Data from 3,391 participants who completed L-POST were analyzed. L-POST consists of 15 subtests with five items each. Item Response Theory (IRT) was applied to guide the selection of the items. The highest correlation with full-scale subtest scores was observed when two items were selected for each subtest following an adaptive testing procedure. A pilot validation in a subsample of participants with low abilities demonstrated adaptive testing has reasonable sensitivity (80%) but limited specificity (55%) in classifying participants with impaired and unimpaired abilities. Subsequently, we reduced the number of subtests through factor analysis. We showed that the subtests using Radial Frequency Patterns as stimuli were redundant and could be combined in one subtest. We conclude that L-POST can be shortened to 26 items (7-14 min) and an adaptive item selection procedure could prove particularly useful for screening purposes. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Comparing the Speech Perception of Cochlear Implant Users with Three Different Finnish Speech Intelligibility Tests in Noise.

Background: A large number of different speech-in-noise (SIN) tests are available for testing cochlear implant (CI) recipients, but few studies have compared the different tests in the same patient population to assess how well their results correlate. Methods: A clinically representative group of 80 CI users conducted the Finnish versions of the matrix sentence test, the simplified matrix sentence test, and the digit triplet test. The results were analyzed for correlations between the different tests and for differences among the participants, including age and device modality. Results: Strong and statistically significant correlations were observed between all of the tests. No floor or ceiling effects were observed with any of the tests when using the adaptive test procedure. Age or the length of device use showed no correlation to SIN perception, but bilateral CI users showed slightly better results in comparison to unilateral or bimodal users. Conclusions: Three SIN tests that differ in length and complexity of the test material provided comparable results in a diverse CI user group.

Bias-corrected estimators for proportion of true null hypotheses: application of adaptive FDR-controlling in segmented failure data

Two recently introduced model-based bias-corrected estimators for proportion of true null hypotheses () under multiple hypotheses testing scenario have been restructured for random observations under a suitable failure model, available for each of the common hypotheses. Based on stochastic ordering, a new motivation behind formulation of some related estimators for is given. The reduction of bias for the model-based estimators are theoretically justified and algorithms for computing the estimators are also presented. The estimators are also used to formulate a popular adaptive multiple testing procedure. Extensive numerical study supports superiority of the bias-corrected estimators. The necessity of the proper distributional assumption for the failure data in the context of the model-based bias-corrected method has been highlighted. A case-study is done with a real-life dataset in connection with reliability and warranty studies to demonstrate the applicability of the procedure, under a non-Gaussian setup. The results obtained are in line with the intuition and experience of the subject expert. An intriguing discussion has been attempted to conclude the article that also indicates the future scope of study.

A Review of Standardised Assessment Development Procedure and Algorithms for Computer Adaptive Testing: Applications and Relevance for Fourth Industrial Revolution

Teaching and learning have gone online in response to the pandemic, which reveals the need for accurately tailored educational assessments to ascertain the extent to which learning outcomes or objectives are achieved. Computer Adaptive Testing (CAT) is a technology-driven form of assessment that tailors items to a candidate's ability level with empirically proven benefits over the fixed-form computer based test. A systematic review was employed which shows that item bank is a key requirement for CAT and the items must through a rigorous item development process to ensure and maintain quality in terms of content, criterion constructs and internal consistency, determining the psychometric validation of behavioural measures while leveraging on variances of Item Response Theory (IRT). Following the item development stage is the need to compile validated items into administrable forms using advanced computer software for automatic test assembly and administration, such as FastTest which allows specifying empirically tried algorithms for CAT from start to termination of the test. This helps to ensure that assessment properly leverages the advantages that CAT holds. Furthermore, the review revealed that CAT has been widely applied with large-scale testing in various fields by educational, health and psychological professionals utilising different IRT models; however only in developed countries. This brings to bear the need for adoption in other parts of the world, for improvements in educational assessments. The interjections of 4IR with AI considering emerging technology aids the CAT algorithm for achieving expert and knowledge-based systems, being a requirement for survival in today’s world.

Computer Adaptive Test Development To Assess Students’ Psychology

Stress becomes a significantly serious issue among university students and we need efficient tools to understand it more. The aim of present study is to develop a Computerized Adaptive Testing (CAT) to measure the mentioned stress, as pioneer project in Vietnam. In this vein, an item bank of 68 items has been constructed, which is based on Likert Polytomous Scales through five subdomains: behavior, academic performance, family, lecturer and finance. The sampling of the survey is large. It has assessed 2,085 students (704 males and 1,381 females). Multidimensional Random Coefficients Multinomial Logit (MRCML) Model is applied to develop Multidimensional Stress Scales and Computerized Adaptive Testing procedure. The result findings indicate that Multidimensional Random Coefficients Multinomial Logit (MRCML) can be used to develop new scale with psychometric properties. Indicated by various fit criteria MNSQ, standard errors, Z (t-test) implemented in software ConQuest. The subdomain has a good reliability (from .857 to .798). As respect to CATs, a simulated experiment based on the empirical data is applied to evaluate the performance of the proposed computerized adaptive testing. The standard error of the estimated stress proficiencies are reported in this study. The 68 items stress data appropriate fit the Multidimensional model applied.

Nonparametric Testing Under Randomized Sketching

A common challenge in nonparametric inference is its high computational complexity when data volume is large. In this paper, we develop computationally efficient nonparametric testing by employing a random projection strategy. In the specific kernel ridge regression setup, a simple distance-based test statistic is proposed. Notably, we derive the minimum number of random projections that is sufficient for achieving testing optimality in terms of the minimax rate. An adaptive testing procedure is further established without prior knowledge of regularity. One technical contribution is to establish upper bounds for a range of tail sums of empirical kernel eigenvalues. Simulations and real data analysis are conducted to support our theory.

Method to Investigate Multi-Axis Release Action of Ski Safety Bindings: A New Approach for Testing in Research and Development.

The authors developed and elaborated on a new method to release ski bindings by utilizing an industrial robot to simulate release movement showing a spatial repeatability of ± 0.06 mm. The parametric programming of the release parameters gave free control while executing repeatable release tests. A series of different motion patterns were performed, on the one hand, to test the applicability of the setup to the simulation of motion patterns and, on the other, to check for the impact of the ski deformations like ski deflections within the range of −5 mm to −85 mm, on the safety bindings' release forces. As certain falling mechanisms are related to knee injury, which is the most common severe injury in alpine skiing, this testing method can be used to develop related displacement movements in future. This movements do not necessarily accord with the directional release mechanics of safety ski bindings. The authors specify the developed testing apparatus as device for force measurements in 3D with an accuracy of ± 0.5% in boot-sole-plane. The intention behind this development is to enable faster, more versatile and adaptive testing procedures in R&D.

ASYMPTOTICALLY INDEPENDENT U-STATISTICS IN HIGH-DIMENSIONAL TESTING.

Many high-dimensional hypothesis tests aim to globally examine marginal or low-dimensional features of a high-dimensional joint distribution, such as testing of mean vectors, covariance matrices and regression coefficients. This paper constructs a family of U-statistics as unbiased estimators of the ℓ p -norms of those features. We show that under the null hypothesis, the U-statistics of different finite orders are asymptotically independent and normally distributed. Moreover, they are also asymptotically independent with the maximum-type test statistic, whose limiting distribution is an extreme value distribution. Based on the asymptotic independence property, we propose an adaptive testing procedure which combines p-values computed from the U-statistics of different orders. We further establish power analysis results and show that the proposed adaptive procedure maintains high power against various alternatives.

Adaptive spatial working memory assessments for aging pet dogs.

Assessments for spatial working memory (SWM) in pet dogs that can detect age-related cognitive deficits in a single session may aid in diagnosing canine dementia and may facilitate translational research on Alzheimer's disease in humans. Adaptive testing procedures are widely used in single-session assessments for humans with diverse cognitive abilities. In this study, we designed and deployed two up-down staircase assessments for SWM in which 26 pet dogs were required to recall the location of a treat hidden behind one of two identical boxes following delays of variable length. In the first experiment, performance tended to decline with age but few dogs completed the test (n = 10). However, all of the dogs that participated in the second experiment (n = 24) completed the assessment and provided reliable evidence of learning and retaining the task. Delay length and age significantly predicted performance supporting the validity of this assessment. The relationships between age and performance were described by inverted U-shaped functions as both old and young dogs displayed deficits in weighted cumulative-scores and trial-by-trial performance. Thus, SWM in pet dogs may develop until midlife and decline thereafter. Exploratory analyses of non-mnemonic fixation strategies, sustained engagement, inhibitory control, and potential improvements for future SWM assessments which adopt this paradigm are also discussed.

Prediction of Individual Cochlear Implant Recipient Speech Perception With the Output Signal to Noise Ratio Metric.

A cochlear implant (CI) implements a variety of sound processing algorithms that seek to improve speech intelligibility. Typically, only a small number of parameter combinations are evaluated with recipients but the optimal configuration may differ for individuals. The present study evaluates a novel methodology which uses the output signal to noise ratio (OSNR) to predict complete psychometric functions that relate speech recognition to signal to noise ratio for individual CI recipients. Speech scores from sentence-in-noise tests in a "reference" condition were mapped to OSNR and a psychometric function was fitted. The reference variability was defined as the root mean square error between the reference scores and the fitted curve. To predict individual scores in a different condition, OSNRs in that condition were calculated and the corresponding scores were read from the reference psychometric function. In a retrospective experiment, scores were predicted for each condition and subject in three existing data sets of sentence scores. The prediction error was defined as the root mean square error between observed and predicted scores. In data set 1, sentences were mixed with 20 talker babble or speech weighted noise and presented at 65 dB sound pressure level (SPL). An adaptive test procedure was used. Sound processing was advanced combinatorial encoding (ACE, Cochlear Limited) and ACE with ideal binary mask processing, with five different threshold settings. In data set 2, sentences were mixed with speech weighted noise, street-side city noise or cocktail party noise and presented at 65 dB SPL. An adaptive test procedure was used. Sound processing was ACE and ACE with two different noise reduction schemes. In data set 3, sentences were mixed with four-talker babble at two input SNRs and presented at levels of 55-89 dB SPL. Sound processing utilised three different automatic gain control configurations. For data set 1, the median of individual prediction errors across all subjects, noise types and conditions, was 12% points, slightly better than the reference variability. The OSNR prediction method was inaccurate for the specific condition with a gain threshold of +10 dB. For data set 2, the median of individual prediction errors was 17% points and the reference variability was 11% points. For data set 3, the median prediction error was 9% points and the reference variability was 7% points. A Monte Carlo simulation found that the OSNR prediction method, which used reference scores and OSNR to predict individual scores in other conditions, was significantly more accurate (p < 0.01) than simply using reference scores as predictors. The results supported the hypothesis that the OSNR prediction method could accurately predict individual recipient scores for a range of algorithms and noise types, for all but one condition. The medians of the individual prediction errors for each data set were accurate within 6% points of the reference variability and compared favourably with prediction methodologies in other recent studies. Overall, the novel OSNR-based prediction method shows promise as a tool to assist researchers and clinicians in the development or fitting of CI sound processors.