Extraneous Variation Research Articles

FieldSimR: an R package for simulating plot data in multi-environment field trials.

This paper presents a general framework for simulating plot data in multi-environment field trials with one or more traits. The framework is embedded within the R package FieldSimR, whose core function generates plot errors that capture global field trend, local plot variation, and extraneous variation at a user-defined ratio. FieldSimR's capacity to simulate realistic plot data makes it a flexible and powerful tool for a wide range of improvement processes in plant breeding, such as the optimisation of experimental designs and statistical analyses of multi-environment field trials. FieldSimR provides crucial functionality that is currently missing in other software for simulating plant breeding programmes and is available on CRAN. The paper includes an example simulation of field trials that evaluate 100 maize hybrids for two traits in three environments. To demonstrate FieldSimR's value as an optimisation tool, the simulated data set is then used to compare several popular spatial models for their ability to accurately predict the hybrids' genetic values and reliably estimate the variance parameters of interest. FieldSimR has broader applications to simulating data in other agricultural trials, such as glasshouse experiments.

The Questionable Practice of Partialing to Refine Scores on and Inferences About Measures of Psychological Constructs.

Partialing is a statistical approach researchers use with the goal of removing extraneous variance from a variable before examining its association with other variables. Controlling for confounds through analysis of covariance or multiple regression analysis and residualizing variables for use in subsequent analyses are common approaches to partialing in clinical research. Despite its intuitive appeal, partialing is fraught with undesirable consequences when predictors are correlated. After describing effects of partialing on variables, we review analytic approaches commonly used in clinical research to make inferences about the nature and effects of partialed variables. We then use two simulations to show how partialing can distort variables and their relations with other variables. Having concluded that, with rare exception, partialing is ill-advised, we offer recommendations for reducing or eliminating problematic uses of partialing. We conclude that the best alternative to partialing is to define and measure constructs so that it is not needed.

Experimental design in practice: The importance of blocking and treatment structures

SummaryExperimental design and analysis has evolved substantially over the last 100 years, driven to a large extent by the power and availability of the computer. To demonstrate this development and encourage the use of experimental design in practice, three experiments from different research areas are presented. In these examples multiple blocking factors have been employed and they show how extraneous variation can be accommodated and interpreted. The examples are used to discuss the importance of blocking and treatment structures in the conduct of designed experiments.

Transcriptome analyses of 7-day-old zebrafish larvae possessing a familial Alzheimer\u2019s disease-like mutation in psen1 indicate effects on oxidative phosphorylation, ECM and MCM functions, and iron homeostasis

BackgroundEarly-onset familial Alzheimer’s disease (EOfAD) is promoted by dominant mutations, enabling the study of Alzheimer’s disease (AD) pathogenic mechanisms through generation of EOfAD-like mutations in animal models. In a previous study, we generated an EOfAD-like mutation, psen1Q96_K97del, in zebrafish and performed transcriptome analysis comparing entire brains from 6-month-old wild type and heterozygous mutant fish. We identified predicted effects on mitochondrial function and endolysosomal acidification. Here we aimed to determine whether similar effects occur in 7 day post fertilization (dpf) zebrafish larvae that might be exploited in screening of chemical libraries to find ameliorative drugs.ResultsWe generated clutches of wild type and heterozygous psen1Q96_K97del 7 dpf larvae using a paired-mating strategy to reduce extraneous genetic variation before performing a comparative transcriptome analysis. We identified 228 differentially expressed genes and performed various bioinformatics analyses to predict cellular functions.ConclusionsOur analyses predicted a significant effect on oxidative phosphorylation, consistent with our earlier observations of predicted effects on ATP synthesis in adult heterozygous psen1Q96_K97del brains. The dysregulation of minichromosome maintenance protein complex (MCM) genes strongly contributed to predicted effects on DNA replication and the cell cycle and may explain earlier observations of genome instability due to PSEN1 mutation. The upregulation of crystallin gene expression may be a response to defective activity of mutant Psen1 protein in endolysosomal acidification. Genes related to extracellular matrix (ECM) were downregulated, consistent with previous studies of EOfAD mutant iPSC neurons and postmortem late onset AD brains. Also, changes in expression of genes controlling iron ion transport were observed without identifiable changes in the prevalence of transcripts containing iron responsive elements (IREs) in their 3′ untranslated regions (UTRs). These changes may, therefore, predispose to the apparent iron dyshomeostasis previously observed in 6-month-old heterozygous psen1Q96_K97del EOfAD-like mutant brains.

Optimisation of elastomeric bearings’ vulcanisation process using response surface methodology and desirability function approach

In this study, response surface methodology and desirability function approach (DFA) were used to optimise the vulcanisation process of elastomeric bearings to obtain better quality products. For this purpose, compressive stiffness and vertical deflection under maximum load were determined as response variables with a target at 198 kN/mm and 3.46 mm, respectively, in accordance with customer’s request. Time, temperature and pressure were considered as control variables of the process. Experimental combinations were determined using Box–Behnken design and two repetitions were made in each combination. The experimental design was randomised for removing bias and other uncontrollable sources of extraneous variation. As a result of the statistical analysis, linear, quadratic, and interaction effects of factors on response variables were obtained. For simultaneous optimisation of multiple responses, the DFA was used and optimum factor levels were obtained. Finally, the optimum conditions were verified by confirmation experiments.

The MED scale: regional variation in values and beliefs, evidence from Italy

PurposeThe purpose of this paper is to provide a measurement scale that captures elements of Mediterranean identity.Design/methodology/approachA scale was developed from items drawn from existing scales and from a qualitative study that examined how respondents viewed their national identity as well as the concept of a Mediterranean identity. The literature pertaining to Mediterranean identity, culture and the influence of contextual factors on its development were reviewed. The scale revealed four dimensions of Mediterranean identity: traditions, relationships, ethnocentrism and a bond with the Sea.FindingsThe scale successfully discriminated between those living by the Sea and those living inland. At the same time, it did not differentiate between respondents living in the north from those living in the south. The study suggests that context, specifically proximity to the Sea, helps shape some of the regional variation in values and beliefs.Research limitations/implicationsThe geographic composition of the sample limits the generalizability of the results. A sample across additional countries would provide greater confidence in the results, although much of the theoretical work on Mediterranean thinking has focused on Italy. Further, it was desirable to keep as many factors as possible constant to limit extraneous variation. Inclusion of respondents from countries with different levels of wealth, religious and political ideologies would likely uncover additional facets of Mediterranean identity.Practical implicationsThe results have broader implications for identifying transnational segments that may exist around the Mediterranean. A key implication relates to the extent that similar values and beliefs appear to be associated with proximity to the Mediterranean Sea. It also highlights the importance of considering contextual factors when developing an understanding of consumer behavior.Originality/valueMediterranean identity has been examined conceptually, but a systematic means to measure is lacking. The essence of Mediterranean thinking embodies the importance individuals place on adhering to traditions, the value they attach to personal relationship, a certain degree of ethnocentricism, and of course the bond with the Sea. The results also suggest that differences that have been attributed to a north/south divide, needs to be reexamined.

Examining ecological validity in social interaction: problems of visual fidelity, gaze, and social potential.

Social interaction is an essential part of the human experience, and much work has been done to study it. However, several common approaches to examining social interactions in psychological research may inadvertently either unnaturally constrain the observed behaviour by causing it to deviate from naturalistic performance, or introduce unwanted sources of variance. In particular, these sources are the differences between naturalistic and experimental behaviour that occur from changes in visual fidelity (quality of the observed stimuli), gaze (whether it is controlled for in the stimuli), and social potential (potential for the stimuli to provide actual interaction). We expand on these possible sources of extraneous variance and why they may be important. We review the ways in which experimenters have developed novel designs to remove these sources of extraneous variance. New experimental designs using a ‘two-person’ approach are argued to be one of the most effective ways to develop more ecologically valid measures of social interaction, and we suggest that future work on social interaction should use these designs wherever possible.

Modelling heterogeneity in longitudinal binomial responses by generalized estimating equations

ABSTRACTExtra-binomial variation in longitudinal/clustered binomial data is frequently observed in biomedical and observational studies. The usual generalized estimating equations method treats the extra-binomial parameter as a constant across all subjects. In this paper, a two-parameter variance function modelling the extraneous variance is proposed to account for heterogeneity among subjects. The new approach allows modelling the extra-binomial variation as a function of the mean and binomial size.

Relationship between pitch and rhythm perception with tonal sequences

Past work has shown varying degrees of relationship between pitch and rhythm perception. Current work investigated the relationship between pitch and rhythm processing in a four-tone sequence-reconstruction task which places additional demands on short-term memory. Sequence tones either had a fixed duration (212 ms) with frequency randomly selected from a logarithmically scaled distribution (400–1750 Hz), a fixed frequency (837 Hz) with a randomly selected log scaled duration (75–600 ms), or a random frequency and duration. In initial conditions, the task was to assemble sequence elements to recreate the target sequence for each of the three sequence types. To evaluate effect of extraneous randomization, both frequency and duration were randomized in the final two conditions with only one of the two attributes defining the target sequence. When only one stimulus attribute was randomized, performance was significantly better with sequences defined by pitch rather than rhythmic variation. Combining pitch and rhythmic variations led to a slight improvement in performance, while the introduction of extraneous variation had little to no effect when either pitch or rhythm defined the target sequence. Overall, there was a wide performance range across listeners with listeners clustered primarily by ability to use pitch information. [Work supported by NIH.]

Does the uncanny valley exist? An empirical test of the relationship between eeriness and the human likeness of digitally created faces

The uncanny valley theory (UVT) (Mori, 1970) proposes that when stimuli are defined by a near-perfect resemblance to humans they cause people to experience greater negative affect relative to when they have perfect human likeness (HL) or little to no HL. Empirical research to support this non-linear relationship between negative affect and HL has been inconclusive, however, and a satisfactory causal explanation has not yet emerged to explain existing findings. In two studies, we examined the relationship between HL and eeriness using digital human faces. First, we examined the relationship between HL and eeriness while controlling for extraneous variation in stimulus appearance. We created two HL continua by manipulating the facial proportions and polygon count of several digital human models. Second, we proposed and tested two causal hypotheses regarding the uncanny valley phenomenon that we refer to as category conflict and feature atypicality. We created two additional HL continua by manipulating the skin coloration and category membership of models. Across these continua we introduced an atypical feature. Our results suggest that HL is linearly related to emotional response, except under conditions where HL varies by category membership, suggesting that previous empirical findings might be explained as a category conflict.

Perfectionistic Thoughts, Personal Standards, and Evaluative Concerns: Further Investigating Relationships to Psychological Distress

This study investigated relationships between perfectionism and psychological distress in a sample of first year college students further divided between those who were enrolled in an honors curriculum (n = 60) and those who were not (n = 53). Participants completed the Perfectionism Cognitions Inventory (PCI), the Frost Multidimensional Perfectionism Scale (MPS), and the profile of mood states (POMS) at the start of the semester and then again 6 weeks later. Results revealed that one dimension of the MPS—maladaptive evaluative concerns (EC)—was positively associated with tension-anxiety and depression-dejection scores, while a second factor—pure personal standards (PS)—was negatively related to distress after extraneous variance was removed. Perfectionism cognitions as measured by the PCI either fully or partially mediated most relationships. Honors students reported higher personal standards than did controls, a difference that was consistent at 6-week follow-up.

Relationship of Creatinine and Nutrition with Arsenic Metabolism

Basu et al. (2011) reported the associations of both dietary and blood nutrient measures, as well as urinary creatinine (uCr), with arsenic (As) methylation capacity, as assessed by the proportions of urinary inorganic, monomethyl, and dimethyl As metabolites. One finding was that uCr was the strongest predictor of As methylation; participants with higher uCr concentrations had a higher percentage of total urinary As as dimethylarsinic acid (DMA) compared to those with lower uCr. This is consistent with what we have previously reported in Bangladeshi adults and children (Gamble et al. 2005; Ahsan et al. 2007; Hall et al. 2009), and is an interesting and potentially very important observation. Approximately 40% of S-adenosylmethionine (SAM)-derived methyl groups are devoted to the biosynthesis of creatine, the precursor of creatinine (Brosnan et al. 2011; Mudd and Poole 1975). At high levels of As exposure (500–1,000 µg/L), based on one-carbon kinetics (Schalinske and Steele 1989), we estimated that methylation of 80% of a daily dose of inorganic As (InAs) to DMA would require approximately 50 µmol SAM, thus consuming approximately 2–4% of the SAM normally turning over in a well-nourished adult per day. Low dietary creatine intake associated with low-protein or vegetarian diets places an increased demand for SAM for creatine biosynthesis (Brosnan 2011). This could potentially reduce the availability of SAM for As methylation, providing a plausible mechanism underlying this highly reproducible observation. This assumes that uCr reflects, to some extent, dietary creatine intake, as we have observed (Gamble M, unpublished data). Conversely, dietary creatine intake and/or creatine supplementation down-regulates endogenous creatine biosynthesis, potentially sparing SAM for methylation of other substrates such as As. We are currently testing this hypothesis in a randomized controlled trial of creatine supplementation. In addition, as Basu et al. (2011) noted, and as we have previously reported (Gamble and Liu 2005), one implication of the observed association between uCr and As methylation capacity is that urinary As should not be expressed per gram creatinine to correct for urine concentration. Rather, uCr should be included as a covariate in regression models. One concerning aspect of the study by Basu et al. (2011) is the handling of blood samples used for nutrient measurements. As noted by Basu et al. and in a previous publication on these same participants (Chung et al. 2006), the blood samples were stored in an ice chest in the field for up to 24 hr before processing. This 24-hr delay can be problematic for some nutrients, especially folate, which is extremely sensitive to oxidative degradation (Drammeh et al. 2008). Basu et al. (2011) reported that in univariate analyses, they observed higher urinary percentages of InAs in individuals with higher serum folate concentrations. This finding is contrary to our previous findings that folate facilitates As methylation (Gamble et al. 2005, 2006, 2007; Hall et al. 2007, 2009). This discrepancy might be explained by differences in sample processing. Basu et al. (2011) also reported associations between dietary intake of several nutrients (assessed using a modified 24-hr recall) and As methylation capacity. One of the most critical and widely discussed issues in nutritional epidemiology is the method used to adjust for total energy intake (TEI) (Willett et al. 1997). The main reasons to adjust for TEI are to a) adjust for potential confounding by TEI, b) remove extraneous variation in nutrient intakes that is due only to their correlation with TEI, and c) simulate a dietary intervention. What is often most relevant is diet composition, or nutrient intake in relation to TEI (Willett et al. 1997). Several methods are available to adjust for TEI, and the best approach can vary depending on the nutrient and question of interest. Basu et al. (2011) adjusted for TEI by dividing each nutrient intake by TEI (nutrient density method). While this approach is appealing because of its simplicity, in reality it can create a complex variable (Willett and Stampfer 1998). For example, when TEI is related to the outcome of interest, the use of nutrient densities can actually induce confounding in the opposite direction. Although we cannot determine from Basu et al.’s article whether TEI measured by the 24-hr recall was associated with As methylation, in theory, an association seems plausible. Also, because their statistical analysis tested for associations between multiple nutrients and urinary As metabolites, it is best to acknowledge that some of the statistically significant associations might be due to chance alone.

Modeling Errors in Physical Activity Recall Data

Physical activity recall instruments provide an inexpensive method of collecting physical activity patterns on a sample of individuals, but they are subject to systematic and random measurement error. Statistical models can be used to estimate measurement error in activity recalls and provide more accurate estimates of usual activity parameters for a population. We develop a measurement error model for a short-term activity recall that describes the relationship between the recall and an individual's usual activity over a long period of time. The model includes terms for systematic and random measurement errors. To estimate model parameters, the design should include replicate observations of a concurrent activity recall and an objective monitor measurement on a subsample of respondents. We illustrate the approach with preliminary data from the Iowa Physical Activity Measurement Study. In this dataset, recalls tend to overestimate actual activity, and measurement errors greatly increase the variance of recalls relative to the person-to-person variation in usual activity. Statistical adjustments are used to remove bias and extraneous variation in estimating the usual activity distribution. Modeling measurement error in recall data can be used to provide more accurate estimates of long-term activity behavior.

Numerical estimation of air gaps’ influence on the insulating performance of multilayer thermal insulation

This study aims to provide insights regarding the benefits of multilayer thermal insulations in building applications placed in combination with two air gaps. For this purpose, a numerical approach was developed to determine the influence of the air gap thickness on the overall thermal resistance of a composite wall. A validated combined radiation/conduction heat transfer numerical model was employed to predict the temperature distribution and heat transfer in typical multilayer insulation complexes comprised of insulating materials separated by multiple reflecting foils. The radiation scheme was based on the two-flux approximation, in order to model both optically thick and optically thin fibrous materials. The heat transfer equation was solved explicitly for a composite wall. A basic methodology for designing simulation scenarios was employed in order to reduce the number of simulated cases. Furthermore, strict entry requirements were provided in order to reduce extraneous variation due to the complexity of physical phenomena that should be investigated in such a case study. Finally, total nine different configurations were selected as initial state, where the theoretical thermal resistances were calculated and compared, leading towards an empirical polynomial equation that calculates the overall thermal resistance of such a composite wall complex departing from the two air gap thickness values.

Optimized Preprocessing of Ultra-Performance Liquid Chromatography/Mass Spectrometry Urinary Metabolic Profiles for Improved Information Recovery

Ultra-performance liquid chromatography coupled to mass spectrometry (UPLC/MS) has been used increasingly for measuring changes of low molecular weight metabolites in biofluids/tissues in response to biological challenges such as drug toxicity and disease processes. Typically samples show high variability in concentration, and the derived metabolic profiles have a heteroscedastic noise structure characterized by increasing variance as a function of increased signal intensity. These sources of experimental and instrumental noise substantially complicate information recovery when statistical tools are used. We apply and compare several preprocessing procedures and introduce a statistical error model to account for these bioanalytical complexities. In particular, the use of total intensity, median fold change, locally weighted scatter plot smoothing, and quantile normalizations to reduce extraneous variance induced by sample dilution were compared. We demonstrate that the UPLC/MS peak intensities of urine samples should respond linearly to variable sample dilution across the intensity range. While all four studied normalization methods performed reasonably well in reducing dilution-induced variation of urine samples in the absence of biological variation, the median fold change normalization is least compromised by the biologically relevant changes in mixture components and is thus preferable. Additionally, the application of a subsequent log-based transformation was successful in stabilizing the variance with respect to peak intensity, confirming the predominant influence of multiplicative noise in peak intensities from UPLC/MS-derived metabolic profile data sets. We demonstrate that variance-stabilizing transformation and normalization are critical preprocessing steps that can benefit greatly metabolic information recovery from such data sets when widely applied chemometric methods are used.

CUSUM charts for monitoring clinical practice quality using primary care prescribing data: a case study of an initiative to encourage generic prescribing of proton pump inhibitors

Expectations on organizations to monitor quality of care are growing. Whilst relevant data are increasingly becoming available it is, in many cases, difficult to distinguish real effects from background variation. Here, the potential usefulness of cumulative sum (CUSUM) charts for monitoring the use of medicines is explored through a case study of an initiative to encourage the prescribing of lowest cost proton pump inhibitors (PPI) in the context of implementation of national guidelines for the management of dyspepsia. This was a longitudinal study involving analysis of routinely collected prescribing data, set in all 12 primary care trusts (PCT) in the North East Strategic Health Authority. In it, comparison (by subtraction) of the time-series of the percentage of generic PPI prescription items for Gateshead with the mean of the other 11 PCTs was used to reduce both variation and bias. This was followed by the construction of a CUSUM chart displaying the effect of the Gateshead initiative. The simple process of comparison was very successful both in removing extraneous trends and reducing background variation, and the CUSUM highly effective for displaying the evidence for the hypothesized step-change in prescribing behaviour consequent on the Gateshead initiative. The effectiveness of the CUSUM here is strongly linked to the success of the preliminary comparison step. CUSUM, a statistical process control technique, has already been tested as a tool for interpreting hospital and general practice mortality rates. Here, its potential for more general applications in quality monitoring is demonstrated using routinely collected prescribing data. Such data contain valuable information about changes in clinical practice and CUSUM charts, when coupled with the idea of removing time trends and extraneous variation by reference to average behaviour, can provide a simple but effective technique for extracting it.

Increased preconception omega-3 polyunsaturated fatty acid intake improves embryo morphology

The association between preconception dietary intake of the polyunsaturated fatty acids (LC-PUFAs) omega-6 and omega-3 and the E(2) levels and IVF/intracytoplasmic sperm injection (ICSI) outcome were investigated in women in a prospective study. It revealed that high intakes of omega-3 LC-PUFA alpha-linolenic acid increase baseline E(2), high intakes of eicosapentaenoic acid and docosahexaenoic acid reduce E(2) response and the number of follicles after ovarian stimulation, and total omega-3 intake, in particular alpha-linolenic acid and docosahexaenoic acid, improve embryo morphology.

Prognostic Differences for Functional Recovery After Major Lower Limb Amputation: Effects of the Timing and Type of Inpatient Rehabilitation Services in the Veterans Health Administration

To compare the recovery of mobility and self-care functions among veteran amputees according to the timing and type of rehabilitation services received. Observational study of inpatient rehabilitation care patterns of 2 types (specialized and consultative) with 2 timings (early and late). Data from inpatient specialized rehabilitation units (SRUs) and consultative services within 95 Veterans Affairs Medical Centers across the United States during fiscal years 2003 to 2004. Medical records of 1502 patients who received early or late consultative or specialized rehabilitation. Hypotheses were established and general categories of negative and positive risk factors specified a priori from available clinical characteristics. Linear mixed effects models were used to model motor Functional Independence Measure (FIM) gain scores on patient-level variables accounting for the correlation within the same facility. Recovery of activities of daily living (ADLs) and mobility (physical functioning) expressed as the magnitudes of gains in motor FIM scores achieved by rehabilitation discharge. After adjustment, amputees who received specialized rehabilitation had motor FIM gains that were on average 8.0 points greater than those for amputees who received consultative rehabilitation. Although patients whose rehabilitation was delayed until after discharge from the index surgical stay tended to be more clinically complex, they had gains comparable to those of patients who received early rehabilitation. Advanced age, transfemoral amputation, paralysis, serious nutritional compromise, and psychosis were associated with lower motor FIM gains. The variance for the random effect for facility was statistically significant, suggesting extraneous variation within facility that was not explainable by observed patient-level variables. On the basis of this analysis, those patients who receive specialized rehabilitation can be expected to make comparatively greater gains than patients who receive consultative services, regardless of timing and clinical complexity. Findings highlight the need for clinicians to adjust prognostic expectations to both clinical severity and the type of rehabilitation that patients receive.

Naturalistic changes in insomnia symptoms and pain in temporomandibular joint disorder: A cross-lagged panel analysis

An increasing number of prospective studies suggest a bi-directional association between the pain and sleep quality. Few of these investigations have controlled for synchronous correlations, an important source of extraneous variance in lagged associations, which may have confounded conclusions of prior investigations. Despite high rates of insomnia in temporomandibular joint disorders (TMD), no studies have examined temporal associations between naturalistic fluctuations in insomnia and pain in TMD. We conducted cross-lagged panel analysis to examine reciprocal temporal associations between 1-month changes in insomnia symptom severity and self-reported pain over 3 months among 53 TMD patients. This rigorous analytic strategy represents a comprehensive method to explore possible reciprocal temporal associations between insomnia and pain that controls for both auto- and synchronous correlations. Analyses revealed that initial-month increases in insomnia were associated with next-month increases in average daily pain, but not vice versa. The direction of the effect was such that initial-month increases in insomnia symptom severity were associated with next-month increases in average daily pain. These data suggest that naturally occurring fluctuations in insomnia symptom severity are prospectively associated with fluctuations in daily pain experience for persons with TMD. Potential mechanisms by which insomnia might influence pain in TMD and therapeutic implications of these findings are discussed.

Generalisability in unbalanced, uncrossed and fully nested studies

There is growing interest in multi-source, multi-level feedback for measuring the performance of health care professionals. However, data are often unbalanced (e.g. there are different numbers of raters for each doctor), uncrossed (e.g. raters rate the doctor on only one occasion) and fully nested (e.g. raters for a doctor are unique to that doctor). Estimating the true score variance among doctors under these circumstances is proving a challenge. Extensions to reliability and generalisability (G) formulae are introduced to handle unbalanced, uncrossed and fully nested data to produce coefficients that take into account variances among raters, ratees and questionnaire items at different levels of analysis. Decision (D) formulae are developed to handle predictions of minimum numbers of raters for unbalanced studies. An artificial dataset and two real-world datasets consisting of colleague and patient evaluations of doctors are analysed to demonstrate the feasibility and relevance of the formulae. Another independent dataset is used for validating D predictions of G coefficients for varying numbers of raters against actual G coefficients. A combined G coefficient formula is introduced for estimating multi-sourced reliability. The results from the formulae indicate that it is possible to estimate reliability and generalisability in unbalanced, fully nested and uncrossed studies, and to identify extraneous variance that can be removed to estimate true score variance among doctors. The validation results show that it is possible to predict the minimum numbers of raters even if the study is unbalanced. Calculating G and D coefficients for psychometric data based on feedback on doctor performance is possible even when the data are unbalanced, uncrossed and fully nested, provided that: (i) variances are separated at the rater and ratee levels, and (ii) the average number of raters per ratee is used in calculations for deriving these coefficients.