Precision Of Effect Estimates Research Articles

High quality (certainty) evidence changes less often than low-quality evidence, but the magnitude of effect size does not systematically differ between studies with low versus high-quality evidence.

Rationale, Aims, and ObjectivesIt is generally believed that evidence from low quality of evidence generate inaccurate estimates about treatment effects more often than evidence from high (certainty) quality evidence (CoE). As a result, we would expect that (a) estimates of effects of health interventions initially based on high CoE change less frequently than the effects estimated by lower CoE (b) the estimates of magnitude of effect size differ between high and low CoE. Empirical assessment of these foundational principles of evidence‐based medicine has been lacking.MethodsWe reviewed the Cochrane Database of Systematic Reviews from January 2016 through May 2021 for pairs of original and updated reviews for change in CoE assessments based on the Grading of Recommendations Assessment, Development and Evaluation (GRADE) method. We assessed the difference in effect sizes between the original versus updated reviews as a function of change in CoE, which we report as a ratio of odds ratio (ROR). We compared ROR generated in the studies in which CoE changed from very low/low (VL/L) to moderate/high (M/H) versus M/H to VL/L. Heterogeneity and inconsistency were assessed using the tau and I 2 statistic. We also assessed the change in precision of effect estimates (by calculating the ratio of standard errors) (seR), and the absolute deviation in estimates of treatment effects (aROR).ResultsFour hundred and nineteen pairs of reviews were included of which 414 (207 × 2) informed the CoE appraisal and 384 (192 × 2) the assessment of effect size. We found that CoE originally appraised as VL/L had 2.1 [95% confidence interval (CI): 1.19–4.12; p = 0.0091] times higher odds to be changed in the future studies than M/H CoE. However, the effect size was not different (p = 1) when CoE changed from VL/L → M/H [ROR = 1.02 (95% CI: 0.74–1.39)] compared with M/H → VL/L (ROR = 1.02 [95% CI: 0.44–2.37]). Similar overlap in aROR between the VL/L → M/H versus M/H → VL/L subgroups was observed [median (IQR): 1.12 (1.07–1.57) vs. 1.21 (1.12–2.43)]. We observed large inconsistency across ROR estimates (I 2 = 99%). There was larger imprecision in treatment effects when CoE changed from VL/L → M/H (seR = 1.46) than when it changed from M/H → VL/L (seR = 0.72).ConclusionsWe found that low‐quality evidence changes more often than high CoE. However, the effect size did not systematically differ between the studies with low versus high CoE. The finding that the effect size did not differ between low and high CoE indicate urgent need to refine current EBM critical appraisal methods.

Consequences of ignoring clustering in linear regression

BackgroundClustering of observations is a common phenomenon in epidemiological and clinical research. Previous studies have highlighted the importance of using multilevel analysis to account for such clustering, but in practice, methods ignoring clustering are often employed. We used simulated data to explore the circumstances in which failure to account for clustering in linear regression could lead to importantly erroneous conclusions.MethodsWe simulated data following the random-intercept model specification under different scenarios of clustering of a continuous outcome and a single continuous or binary explanatory variable. We fitted random-intercept (RI) and ordinary least squares (OLS) models and compared effect estimates with the “true” value that had been used in simulation. We also assessed the relative precision of effect estimates, and explored the extent to which coverage by 95% confidence intervals and Type I error rates were appropriate.ResultsWe found that effect estimates from both types of regression model were on average unbiased. However, deviations from the “true” value were greater when the outcome variable was more clustered. For a continuous explanatory variable, they tended also to be greater for the OLS than the RI model, and when the explanatory variable was less clustered. The precision of effect estimates from the OLS model was overestimated when the explanatory variable varied more between than within clusters, and was somewhat underestimated when the explanatory variable was less clustered. The cluster-unadjusted model gave poor coverage rates by 95% confidence intervals and high Type I error rates when the explanatory variable was continuous. With a binary explanatory variable, coverage rates by 95% confidence intervals and Type I error rates deviated from nominal values when the outcome variable was more clustered, but the direction of the deviation varied according to the overall prevalence of the explanatory variable, and the extent to which it was clustered.ConclusionsIn this study we identified circumstances in which application of an OLS regression model to clustered data is more likely to mislead statistical inference. The potential for error is greatest when the explanatory variable is continuous, and the outcome variable more clustered (intraclass correlation coefficient is ≥ 0.01).

Bottleneck analysis: Simple prediction of the precision of a planned case-control or cohort study based on healthcare registers.

In pharmacoepidemiological studies, the precision of effect estimates usually depends on the lowest number in the underlying two by two table. We denote this the "bottleneck count" (BNC). We describe how to translate the BNC into an achievable precision and provide empirical examples. First, we derive a theoretical prediction of the precision in a study where only the BNC determines precision. As an illustration, we calculated the expected precision of a null-effect study on retinoids and peptic ulcer bleeding, expressed as the upper/lower confidence limit ratio (ULCLR). Finally, we reviewed 126 effect estimates from the literature, analyzing the relationship between the predicted and achieved precision. The log-log transformed ULCLR was shown to be a simple linear function of log(BNC). The expected annual number of retinoid-users experiencing a peptic ulcer bleeding was 9.8, yielding an estimated ULCLR for a 1-year study of 3.84. The literature review showed an inverse linear relationship between the logarithmic BNC and the log-log transformed ULCLR, which was largely independent of study design, effect measure and category of BNC. Achieved precision deviated little from predictions but was usually lower than predicted, particularly with low BNC. The precision of a study can be predicted simply and with good accuracy from the BNC, which is useful for determining whether a study is worth pursuing or not.

„Übertragbarkeit‟ Ist GRADE die Lösung?

Biological and social factors, such as age, comorbidities and the care system, may - as well-established effect-modifiers - limit the transferability of study results to populations with dissimilar characteristics. In order to enable transparent and evidence-based decisions in systematic reviews and guidelines targeting subpopulations that are little or not represented in the study landscape, the GRADE approach is a valid tool to assess the certainty of the evidence. GRADE provides a structured methodology that covers all steps, from developing a precise question, prioritizing patient-relevant outcomes and assessing the available evidence to derive recommendations for practice, among other things. Evaluating confidence in a body of evidence comprises judgments on risk of bias, study heterogeneity, directness, including comparability between study population and target population, precision of effect estimates and publication bias. Because GRADE demands transparent decisions about the applicability of study results from the study population to the target population, gaps in the evidence landscape can be uncovered. Overall, the approach cannot solve the problem of the transferability of study results. It does, however, support the explicit handling of applicability and can give impetus to targeted research gaps.

Amphetamines for attention deficit hyperactivity disorder (ADHD) in adults.

Attention Deficit Hyperactivity Disorder (ADHD) is a childhood onset psychiatric disorder that can persist into adulthood in up to 50% of patients. From a clinical point of view, ADHD is characterized by hyperactivity, mood instability, irritability, difficulties in maintaining attention, lack of organization and impulsive behaviours. The presence of other disorders occurring at the the same time is also common, especially mood disorders and substance abuse. It seems that amphetamines could reverse the underlying neurological problems that feature in ADHD, and so improve ADHD symptoms. We found seven studies, which enrolled 1091 patients. These studies compared amphetamines to placebo and three of them also compared amphetamines with other drugs: guanfacine, modafinil and paroxetine. Three amphetamine derivatives were investigated: dexamphetamine, lisdexamphetamine and mixed amphetamine salts (MAS). Treatment length ranged from two to 20 weeks. All amphetamines improved ADHD symptoms but overall they did not make people more likely to stay in treatment and were associated with a higher risk of treatment ending early due to adverse events. One type of amphetamine, mixed amphetamine salts, did, however, increase retention in treatment. We found no evidence that higher doses worked better than lower ones. We did not find any difference in effectiveness between immediate-release and sustained-release formulations. Therefore, it appears that short-term treatment with amphetamines reduces ADHD symptoms, but studies assessing the effects of amphetamines for longer periods of time are needed.

PROTOCOL: Strategies for scaling up the implementation of interventions in social welfare: protocol for a systematic review.

PROTOCOL: Strategies for scaling up the implementation of interventions in social welfare: protocol for a systematic review.

PROTOCOL: Effects of interventions for infant and young child feeding (IYCF) promotion on optimal IYCF practices, nutrition, growth and health in low- and middle-income countries: a systematic review.

PROTOCOL: Effects of interventions for infant and young child feeding (IYCF) promotion on optimal IYCF practices, nutrition, growth and health in low- and middle-income countries: a systematic review.

Comparing antibiotic treatment for leptospirosis using network meta-analysis: a tutorial

BackgroundNetwork meta-analysis consists of simultaneous analysis of both direct comparisons of interventions within randomized controlled trials and indirect comparisons across trials based on a common comparator. In this paper, we aimed to characterise the conceptual understanding and the rationale for the use of network meta-analysis in assessing drug efficacy.MethodsWe selected randomized controlled trials, assessing efficacy of antibiotics for the treatment of leptospirosis as a case study. A pairwise meta-analysis was conducted using a random effect model, assuming that different studies assessed different but related treatment effects. The analysis was then extended to a network meta-analysis, which consists of direct and indirect evidence in a network of antibiotics trials, using a suite of multivariate meta-analysis routines of STATA (mvmeta command). We also assessed an assumption of ‘consistency’ that estimates of treatment effects from direct and indirect evidence are in agreement.ResultsSeven randomised controlled trials were identified for this analysis. These RCTs assessed the efficacy of antibiotics such as penicillin, doxycycline and cephalosporin for the treatment of human leptospirosis. These studies made comparisons between antibiotics (i.e. an antibiotic versus alternative antibiotic) in the primary study and a placebo, except for cephalosporin. These studies were sufficient to allow the creation of a network for the network meta-analysis; a closed loop in which three comparator antibiotics were connected to each other through a polygon. The comparison of penicillin versus the placebo has the largest contribution to the entire network (31.8%). The assessment of rank probabilities indicated that penicillin presented the greatest likelihood of improving efficacy among the evaluated antibiotics for treating leptospirosis.ConclusionsFindings suggest that network meta-analysis, a meta-analysis comparing multiple treatments, is feasible and should be considered as better precision of effect estimates for decisions when several antibiotic options are available for the treatment of leptospirosis.

Evaluation of contaminated drinking water and male breast cancer at Marine Corps Base Camp Lejeune, North Carolina: a case control study.

BackgroundSolvents contaminated drinking water supplies at Marine Corps Base Camp Lejeune during 1950s-1985.MethodsWe conducted a case–control study among Marines to evaluate associations between residential exposure to contaminated drinking water at Camp Lejeune and male breast cancer risk. The study included 71 male breast cancer cases and 373 controls identified from the Department of Veteran’s Affairs (VA) cancer registry whose military personnel records were available. Controls were selected from cancers not known to be associated with solvent exposure and included 270 skin cancers, 71 mesotheliomas, and 32 bone cancers. Base assignment and risk factor information came from military personnel and VA records. Groundwater contaminant fate/transport and distribution system models provided monthly estimated residential contaminant levels. We conducted exact logistic regression using the 50th percentile level among exposed controls to create low and high exposure categories. We calculated 95 % confidence intervals (CIs) to indicate precision of effect estimates. Exploratory analyses used proportional hazards methods to evaluate associations between exposures and age at diagnosis.ResultsAfter adjusting for age at diagnosis, race, and service in Vietnam, the odds ratio (OR) for ever stationed at Camp Lejeune was 1.14 (95 % CI: 0.65, 1.97). Adjusted ORs for high residential cumulative exposures to tetrachloroethylene (PCE), t-1,2 dichloroethylene (DCE), and vinyl chloride were 1.20 [95 % CI: 0.16-5.89], 1.50 [95 % CI: 0.30-6.11], 1.19 [95 % CI: 0.16-5.89], respectively, with a monotonic exposure response relationship for PCE only. However these results were based on two or three cases in the high cumulative exposure categories. Ever stationed at Camp Lejeune and high cumulative exposures to trichloroethylene (TCE), PCE, DCE and vinyl chloride were associated with earlier age at onset for male breast cancer; hazard ratios ranged from 1.4-2.7 with wide confidence intervals for cumulative exposure variables.ConclusionFindings suggested possible associations between male breast cancer and being stationed at Camp Lejeune and cumulative exposure to PCE, DCE, and vinyl chloride. TCE, PCE, DCE and vinyl chloride cumulative exposures showed possible associations with earlier age at onset of male breast cancer. However, this study was limited by small numbers of cases in high exposure categories.Electronic supplementary materialThe online version of this article (doi:10.1186/s12940-015-0061-4) contains supplementary material, which is available to authorized users.

The GRADE system

The Grades of Recommendation, Assessment, Development and Evaluation Working Group (GRADE Working Group) has developed a system for grading the quality of evidence. Over 20 organizations including the World Health Organization (WHO) have adopted it. The quality of a body of evidence involves consideration of within-study risk of bias (methodological quality), directness of evidence, heterogeneity, precision of effect estimates and risk of publication bias, and the system considers the assessment of the quality of a body of evidence for each individual outcome. The GRADE approach specifies four levels of quality. Randomized trials, usually the highest grade of evidence, can be downgraded depending on the presence of the methodological problems. A brief overview of the grade system is presented.

Comparing the cohort design and the nested case–control design in the presence of both time‐invariant and time‐dependent treatment and competing risks: bias and precision

PurposeObservational studies using electronic administrative healthcare databases are often used to estimate the effects of treatments and exposures. Traditionally, a cohort design has been used to estimate these effects, but increasingly, studies are using a nested case–control (NCC) design. The relative statistical efficiency of these two designs has not been examined in detail.MethodsWe used Monte Carlo simulations to compare these two designs in terms of the bias and precision of effect estimates. We examined three different settings: (A) treatment occurred at baseline, and there was a single outcome of interest; (B) treatment was time varying, and there was a single outcome; and C treatment occurred at baseline, and there was a secondary event that competed with the primary event of interest. Comparisons were made of percentage bias, length of 95% confidence interval, and mean squared error (MSE) as a combined measure of bias and precision.ResultsIn Setting A, bias was similar between designs, but the cohort design was more precise and had a lower MSE in all scenarios. In Settings B and C, the cohort design was more precise and had a lower MSE in all scenarios. In both Settings B and C, the NCC design tended to result in estimates with greater bias compared with the cohort design.ConclusionsWe conclude that in a range of settings and scenarios, the cohort design is superior in terms of precision and MSE. Copyright © 2012 John Wiley & Sons, Ltd.

Effects of Adjusting for Instrumental Variables on Bias and Precision of Effect Estimates

Recent theoretical studies have shown that conditioning on an instrumental variable (IV), a variable that is associated with exposure but not associated with outcome except through exposure, can increase both bias and variance of exposure effect estimates. Although these findings have obvious implications in cases of known IVs, their meaning remains unclear in the more common scenario where investigators are uncertain whether a measured covariate meets the criteria for an IV or rather a confounder. The authors present results from two simulation studies designed to provide insight into the problem of conditioning on potential IVs in routine epidemiologic practice. The simulations explored the effects of conditioning on IVs, near-IVs (predictors of exposure that are weakly associated with outcome), and confounders on the bias and variance of a binary exposure effect estimate. The results indicate that effect estimates which are conditional on a perfect IV or near-IV may have larger bias and variance than the unconditional estimate. However, in most scenarios considered, the increases in error due to conditioning were small compared with the total estimation error. In these cases, minimizing unmeasured confounding should be the priority when selecting variables for adjustment, even at the risk of conditioning on IVs.

Model selection and its consequences for different split‐plot designs with spatial covariance and trend

With 1 figure and 10 tables AbstractSplit‐plot designs play an important role in agricultural field trials, particularly in plant breeding. Analysis of split‐plot experiments may be performed by classical randomization‐based anova‐type models. In addition, spatial covariance can be modelled to improve precision. This study investigates the properties of alternative variance–covariance structures (random blocks, spatial covariance) using Monte‐Carlo simulation for design scenarios commonly found in plant breeding and variety testing. We investigate the efficiency of model selection by information criteria (AICC, BIC). Our results indicate that efficient model selection can be expected when there are marked differences among models with respect to the empirical error rates (type I error and power) as well as the precision of effect estimates (MSE). A comparison of approximations to the denominator degrees of freedom shows that for spatial models use of the Kenward–Roger method gives reasonable control of the nominal type I error rate.

Meta-Analysis and Causal Inference: A Case Study of Benzene and Non-Hodgkin Lymphoma

Meta-analysis is an important method in the practice of occupational epidemiology, with a legitimate, but limited role to play in causal inference. Meta-analysis provides an assessment of consistency-one of several classic causal criteria-through tests of heterogeneity and an assessment of differences across studies. It can also provide an increase in the precision of effect estimates, including the precision of dose response relationships. Causal inference, however, involves much more: a complete assessment of the classic causal criteria, for example. Causal claims, therefore, should not emerge from meta-analyses as such. A recent meta-analysis of epidemiological studies of benzene exposure and non-Hodgkin lymphoma (NHL), however, does exactly that. Using studies from a previous narrative review in which the authors made no causal claim, the same authors performed a meta-analysis and concluded that it represented new evidence that benzene causes NHL. Despite a lack of consistency (i.e., significant heterogeneity), weak associations, no evidence of dose-response, no effort to provide an assessment of biological plausibility, and no new epidemiological evidence, the authors, nevertheless, changed their conclusion from association to causation. By using case study as an illustrative platform, this commentary provides cautionary and critical comments about the use of meta-analysis and causal inference in occupational epidemiology.

Propensity score methods to adjust for confounding in assessing treatment effects: bias and precision

There is an increasing interest in the use of propensity score (PS) methods for confounding control, with generally three ways of estimating adjusted treatment effects in pharmacoepidemiological studies: 1) stratification on PS, 2) matching on PS and 3) using PS as a covariate. To assess bias and precision of different methods, we conducted simulations in three scenarios: 1) treatment had no effect but the crude estimate showed a protective effect; 2) treatment was protective and the crude estimate was more extreme; and 3) treatment increased the risk but the crude estimate showed protective. Adjusting for confounders in all methods shifted the effect estimates toward the true values. Adjusted odds ratios using the PS stratification and the method using PS as a covariate were biased due to either residual confounding or over-adjustment. Matching on PS produced less biased average estimates than other methods but the precision of effect estimates was lower. --------------------------------------------------------------------------------

Response

Response

Completeness and Accuracy of Interview Data from Proxy Respondents

To evaluate the quality of exposure data provided by proxy respondents, we used a dual interview protocol in a case-control study of subarachnoid hemorrhage. All control subjects and their proxy respondents were interviewed (N = 283 control-proxy pairs), as were the cases who were able to provide their own information and their proxy respondents (N = 68 case-proxy pairs). The reliability of proxy-derived data was excellent for demographic and body habitus measures (kappa or intraclass correlation range = 0.86-0.99), and all aspects of cigarette smoking history (range = 0.79-0.93). Proxy reliability was somewhat lower for questions regarding medications and hormone preparations (range = 0.55-0.88), alcohol consumption (range = 0.52-0.82), and recreational physical activity (range = 0.55-0.67). Proxy reliability varied according to the relationship of the proxy to the index subject. Relative to the index subjects, proxy respondents tended to underreport the presence or level of exposure. For most exposures, odds ratios computed with proxy-derived data were similar in magnitude to odds ratios obtained with index subject data; important bias due to differential nonresponse or differential misclassification was suggested only for questions regarding hormone replacement therapy. Epidemiologic studies that rely on proxy respondents may require more subjects to offset the effect of nondifferential nonresponse and misclassification on the precision of effect estimates.