Unbalanced Random Effects Model Research Articles

Building reliable and generalizable clerkship competency assessments: Impact of ‘hawk-dove’ correction

Purpose Systematic differences among raters’ approaches to student assessment may result in leniency or stringency of assessment scores. This study examines the generalizability of medical student workplace-based competency assessments including the impact of rater-adjusted scores for leniency and stringency. Methods Data were collected from summative clerkship assessments completed for 204 students during 2017–2018 the clerkship at a single institution. Generalizability theory was used to explore variance attributed to different facets (rater, learner, item, and competency domain) through three unbalanced random-effects models by clerkship including applying assessor stringency-leniency adjustments. Results In the original assessments, only 4–8% of the variance was attributed to the student with the remainder being rater variance and error. Aggregating items to create a composite score increased variability attributable to the student (5–13% of variance). Applying a stringency-leniency (‘hawk-dove’) correction substantially increased the variance attributed to the student (14.8–17.8%) and reliability. Controlling for assessor leniency/stringency reduced measurement error, decreasing the number of assessments required for generalizability from 16–50 to 11–14. Conclusions Similar to prior research, most of the variance in competency assessment scores was attributable to raters, with only a small proportion attributed to the student. Making stringency-leniency corrections using rater-adjusted scores improved the psychometric characteristics of assessment scores.

A Question of Scale? Generalizability of the Ottawa and Chen Scales to Render Entrustment Decisions for the Core EPAs in the Workplace.

Assessments of the Core Entrustable Professional Activities (Core EPAs) are based on observations of supervisors throughout a medical student's progression toward entrustment. The purpose of this study was to compare generalizability of scores from 2 entrustment scales: the Ottawa Surgical Competency Operating Room Evaluation (Ottawa) scale and an undergraduate medical education supervisory scale proposed by Chen and colleagues (Chen). A secondary aim was to determine the impact of frequent assessors on generalizability of the data. For academic year 2019-2020, the Virginia Commonwealth University School of Medicine modified a previously described workplace-based assessment (WBA) system developed to provide feedback for the Core EPAs across clerkships. The WBA scored students' performance using both Ottawa and Chen scales. Generalizability (G) and decision (D) studies were performed using an unbalanced random-effects model to determine the reliability of each scale. Secondary G- and D-studies explored whether faculty who rated more than 5 students demonstrated better reliability. The Phi-coefficient was used to estimate reliability; a cutoff of at least 0.70 was used to conduct D-studies. Using the Ottawa scale, variability attributable to the student ranged from 0.8% to 6.5%. For the Chen scale, student variability ranged from 1.8% to 7.1%. This indicates the majority of variation was due to the rater (42.8%-61.3%) and other unexplained factors. Between 28 and 127 assessments were required to obtain a Phi-coefficient of 0.70. For 2 EPAs, using faculty who frequently assessed the EPA improved generalizability, requiring only 5 and 13 assessments for the Chen scale. Both scales performed poorly in terms of learner-attributed variance, with some improvement in 2 EPAs when considering only frequent assessors using the Chen scale. Based on these findings in conjunction with prior evidence, the authors provide a root cause analysis highlighting challenges with WBAs for Core EPAs.

Wherever You Go, No Matter the Weather, Always Bring Your Own Sunshine: Revisiting Thermal Climate Models of Trust with Cross-National Panel Data, 1981-2009

Recent theory predicts that climatic demands in conjunction with wealth-based resources serve to enhance socio-psychological functioning and facilitate the development of cognitive processes such as generalized trust. Past research, however, has provided only cross-sectional evidence to support this theory. In this study, I analyzed a repeated cross-sectional data set that included representative data from 123 societies spread over a 29-year time period. Unbalanced random-effects models and ordinary least squares regression showed that thermal climate and wealth-based resources interacted in their influence on generalized trust. Although the observed associations were robust to potential sources of bias, conditional marginal effect sizes for thermal climate were significantly reduced with the inclusion of confounding control variables. The findings support climatic demands -- resource theory of generalized trust, invite new research directions, and yield important implications for trust research and theory.

Climate, Affluence, and Trust

Recent theory predicts that climatic demands in conjunction with wealth-based resources serve to enhance socio-psychological functioning and facilitate the development of cognitive processes such as generalized trust. Past research, however, has provided only cross-sectional evidence to support this theory. In this study, I analyzed a repeated cross-sectional data set that included representative data from 123 societies spread over a 29-year time period. Unbalanced random-effects models and ordinary least squares regression showed that thermal climate and wealth-based resources interacted in their influence on generalized trust. Although the observed associations were robust to potential sources of bias, conditional marginal effect sizes for thermal climate were significantly reduced with the inclusion of confounding control variables. The findings support climatic demands–resource theory of generalized trust, invite new research directions, and yield important implications for trust research and theory.

Unilateral Conformance Proportions in Balanced and Unbalanced Normal Random Effects Models

Conformance proportion is the proportion of a performance characteristic of interest that falls within a prespecified acceptance region, which has been used in various applications. In this article, a simple closed-form interval estimation, based on Student’s t statistic, is proposed for unilateral conformance proportions in balanced and unbalanced random-effects models. Two real datasets are analyzed to illustrate the proposed method, whose performance is also evaluated through detailed simulation studies. The simulation results reveal that the empirical coverage probabilities for upper confidence limits of the method are sufficiently close to the nominal values, but those for lower confidence limits appear to be slightly less than the nominal level. Furthermore, a bootstrap-based calibration for both upper and lower confidence limits is provided to have empirical coverage probabilities closer to the nominal level. This article has supplementary material online.

One-Sided Tolerance Limits for Balanced and Unbalanced Random Effects Models

A simple analytical approach for constructing one-sided β-content, γ-confidence tolerance limits is proposed for general random effects models with normal data in both balanced and unbalanced data scenarios. The approach is based on an approximation to the noncentral t distribution and modified large sample methods for constructing confidence bounds on functions of variance components. An alternative bootstrap-adjusted limit is also proposed. The performance of the analytical and bootstrap-adjusted limits is assessed via simulation techniques. The results indicate that the analytical limit is generally somewhat conservative, but is often less conservative than an existing analytical approach and may provide substantially shorter interval lengths, particularly for smaller sample sizes and larger values of γ. The bootstrap-adjusted limit generally maintains the nominal confidence level and yields shorter interval lengths, though it can be anticonservative for small sample sizes. This article has supplementary material online.

One-sided tolerance limits for unbalanced one-way random effects models: a generalized Mee and Owen procedure

This article proposes a new procedure for obtaining one-sided tolerance limits in unbalanced random effects models. The procedure is a generalization of that proposed by Mee and Owen for the balanced situation, and can be easily implemented, because it only needs a non-central-t table. Two simulation studies are carried out to assess the performance of the new procedure and to compare it with one of the other procedures laid out in previous statistical literature. The article findings show that the new procedure is much simpler to compute and performs better than the previous ones, having inferior values of the gamma bias in a wide range of situations, representative of many actual industrial applications, and behaving also reasonably well in more extreme sampling situations. The use of the new limits is illustrated by an application to an actual example from the steel industry.

The pigeonhole bootstrap

Recently there has been much interest in data that, in statistical language, may be described as having a large crossed and severely unbalanced random effects structure. Such data sets arise for recommender engines and information retrieval problems. Many large bipartite weighted graphs have this structure too. We would like to assess the stability of algorithms fit to such data. Even for linear statistics, a naive form of bootstrap sampling can be seriously misleading and McCullagh [Bernoulli 6 (2000) 285--301] has shown that no bootstrap method is exact. We show that an alternative bootstrap separately resampling rows and columns of the data matrix satisfies a mean consistency property even in heteroscedastic crossed unbalanced random effects models. This alternative does not require the user to fit a crossed random effects model to the data.

The Expected Sample Variance of Uncorrelated Random Variables with a Common Mean and Some Applications in Unbalanced Random Effects Models

There is a little-known but very simple generalization of the standard result that for uncorrelated random variables with common mean and variance , the expected value of the sample variance is . The generalization justifies the use of the usual standard error of the sample mean in possibly heteroscedastic situations, and motivates elementary estimators in even unbalanced linear random effects models. The latter both provides nontrivial examples and exercises concerning method-of-moments estimation, and also helps “demystify” the whole matter of variance component estimation. This is illustrated in general for the simple one-way context and for a specific unbalanced two-factor hierarchical data structure.

Do Reuters spreads reflect currencies' differences in global trading activity?

A new estimation of the long-run impact of trading activity on bid–ask spreads in the foreign exchange markets is undertaken with a short panel containing around-the-clock Reuters quotes and global transaction volumes. Individual and time effects are accounted for in an unbalanced random effects model. In line with liquidity effect explanations the volume parameter is found to have a (weakly) significant negative sign, whereas the volatility parameter is positive. Structural parameters are stable over time, while residuals are groupwise heteroscedastic. Reuters quoting (`tick') frequency is also tested as a measure of trading activity with very similar results.)

Harmonic mean approach to uunbalanced random effects models under heteroscedasticity

In this paper we consider unbalanced random effects models under heteroscedastic variances. By using' the harmonic mean approach, it is shown that the problems are analogous to those from balanced random effects models under horaoscedastic variances. Thus, by using the harmonic mean approach, statistical inferences about variance components are derived by using procedures from balanced models under homoscedastic variances. Laguerre polynomial expansion is used to approximate the sampling distributions of relevant statistics.

Cumulants of bilinear forms and quadratic forms and their applications

In this paper, mixed cumulants of general bilinear forms, quadratic forms and linear forms are given up to order 4. These results are useful in deriving approximations to the distributions of sums of squares and F-ratios from unbalanced analysis of variance models from non-normal universes. As an illustration, the results are applied to derive mixed cumulants of sums of squares in a three stages nested unbalanced random effects models from non-normal universes.

On Approximating the Null and Nonnull Distributions of theFRatio in Unbalanced Random-Effect Models from Nonnormal Universes

Abstract In this article we consider the unbalanced random-effects model as given in (1.1) with random variables from nonormal universes. Assuming that the αi 's are independently distributed of the eij 's, an approximation to the joint distribution of the mean squares (and hence the F ratio) by using Laguerre polynomials is derived. These results are then used to examine the effects of departure from normality on the traditional F-ratio distribution.

Note on Testing Hypotheses in an Unbalanced Random Effects Model

Note on Testing Hypotheses in an Unbalanced Random Effects Model

Note on testing hypotheses in an unbalanced random effects model

Note on testing hypotheses in an unbalanced random effects model