Bias In Animal Studies Research Articles

Comparative efficacy of the five most common traditional Chinese medicine monomers in reducing intimal hyperproliferation in arterial balloon injury models: A network meta-analysis

ObjectiveThis study utilized network meta-analysis (NMA) to compare the efficacy of five commonly used traditional Chinese medicine monomers in reducing intimal hyperproliferation in arterial balloon injury models. MethodsRelevant literature up to January 2024 was systematically retrieved from seven major databases. The intima-to-media (I/M) ratio was chosen as the primary outcome measure. The risk of bias in animal studies was assessed using the SYstematic Review Centre for Laboratory Animal Experimentation (SYRCLE) tool. Statistical analysis was conducted using Stata 17 software. ResultsA total of 43 studies were included in this meta-analysis. NMA results showed that in the rat model, compared to the control group, GS (SMD: 0.99, 95%CI: 1.25 to −0.73), ASIV (SMD: 1.16, 95%CI: 1.65 to −0.67), TMP (SMD: 0.68, 95%CI: 1.31 to −0.05), and TPNS (SMD: 1.36, 95%CI: 1.91 to −0.80) exhibited inhibitory effects on postoperative intimal hyperproliferation, reducing the I/M ratio. In the rabbit model, compared to the control group, TPNS (SMD: 1.23, 95%CI: 1.97 to −0.49) inhibited postoperative intimal hyperproliferation and reduced the I/M ratio. Superiority ranking analysis suggested that total Panax notoginseng saponin (TPNS) might be the most effective traditional Chinese medicine monomer in reducing intimal hyperproliferation in arterial balloon injury models, lowering the I/M ratio. ConclusionNMA indicates that traditional Chinese medicine monomers can effectively reduce postoperative intimal hyperproliferation in arterial balloon injury models, lowering the I/M ratio, with TPNS showing optimal efficacy. However, the research on TIIA is insufficient, and the limited sample size may affect the robustness of the results. Furthermore, the majority of research on traditional Chinese medicine monomers is currently limited to the experimental stage, lacking further clinical validation. Conducting standardized animal experiments and reporting their findings can enhance the quality of evidence from animal studies, laying the foundation for future clinical trials.

Methodological and ethical challenges in the use of focused ultrasound for blood–brain barrier disruption in neuro-oncology

BackgroundFocused ultrasound (FUS) shows promise for enhancing drug delivery to the brain by temporarily opening the blood–brain barrier (BBB), and it is increasingly used in the clinical setting to treat brain tumours. It remains however unclear whether FUS is being introduced in an ethically and methodologically sound manner. The IDEAL-D framework for the introduction of surgical innovations and the SYRCLE and ROBINS-I tools for assessing the risk of bias in animal studies and non-randomized trials, respectively, provide a comprehensive evaluation for this.Objectives and methodsA comprehensive literature review on FUS in neuro-oncology was conducted. Subsequently, the included studies were evaluated using the IDEAL-D framework, SYRCLE, and ROBINS-I tools.ResultsIn total, 19 published studies and 12 registered trials were identified. FUS demonstrated successful BBB disruption, increased drug delivery, and improved survival rates. However, the SYRCLE analysis revealed a high risk of bias in animal studies, while the ROBINS-I analysis found that most human studies had a high risk of bias due to a lack of blinding and heterogeneous samples. Of the 15 pre-clinical stage 0 studies, only six had formal ethical approval, and only five followed animal care policies. Both stage 1 studies and stage 1/2a studies failed to provide information on patient data confidentiality. Overall, no animal or human study reached the IDEAL-D stage endpoint.ConclusionFUS holds promise for enhancing drug delivery to the brain, but its development and implementation must adhere to rigorous safety standards using the established ethical and methodological frameworks. The complementary use of IDEAL-D, SYRCLE, and ROBINS-I tools indicates a high risk of bias and ethical limitations in both animal and human studies, highlighting the need for further improvements in study design for a safe implementation of FUS in neuro-oncology.

Mesenchymal stem cell-derived extracellular vesicles therapy in traumatic central nervous system diseases: a systematic review and meta-analysis.

Although there are challenges in treating traumatic central nervous system diseases, mesenchymal stem cell-derived extracellular vesicles (MSC-EVs) have recently proven to be a promising non-cellular therapy. We comprehensively evaluated the efficacy of mesenchymal stem cell-derived extracellular vesicles in traumatic central nervous system diseases in this meta-analysis based on preclinical studies. Our meta-analysis was registered at PROSPERO (CRD42022327904, May 24, 2022). To fully retrieve the most relevant articles, the following databases were thoroughly searched: PubMed, Web of Science, The Cochrane Library, and Ovid-Embase (up to April 1, 2022). The included studies were preclinical studies of mesenchymal stem cell-derived extracellular vesicles for traumatic central nervous system diseases. The Systematic Review Centre for Laboratory Animal Experimentation (SYRCLE)'s risk of bias tool was used to examine the risk of publication bias in animal studies. After screening 2347 studies, 60 studies were included in this study. A meta-analysis was conducted for spinal cord injury (n = 52) and traumatic brain injury (n = 8). The results indicated that mesenchymal stem cell-derived extracellular vesicles treatment prominently promoted motor function recovery in spinal cord injury animals, including rat Basso, Beattie and Bresnahan locomotor rating scale scores (standardized mean difference [SMD]: 2.36, 95% confidence interval [CI]: 1.96-2.76, P < 0.01, I2 = 71%) and mouse Basso Mouse Scale scores (SMD = 2.31, 95% CI: 1.57-3.04, P = 0.01, I2 = 60%) compared with controls. Further, mesenchymal stem cell-derived extracellular vesicles treatment significantly promoted neurological recovery in traumatic brain injury animals, including the modified Neurological Severity Score (SMD = -4.48, 95% CI: -6.12 to -2.84, P < 0.01, I2 = 79%) and Foot Fault Test (SMD = -3.26, 95% CI: -4.09 to -2.42, P = 0.28, I2 = 21%) compared with controls. Subgroup analyses showed that characteristics may be related to the therapeutic effect of mesenchymal stem cell-derived extracellular vesicles. For Basso, Beattie and Bresnahan locomotor rating scale scores, the efficacy of allogeneic mesenchymal stem cell-derived extracellular vesicles was higher than that of xenogeneic mesenchymal stem cell-derived extracellular vesicles (allogeneic: SMD = 2.54, 95% CI: 2.05-3.02, P = 0.0116, I2 = 65.5%; xenogeneic: SMD: 1.78, 95%CI: 1.1-2.45, P = 0.0116, I2 = 74.6%). Mesenchymal stem cell-derived extracellular vesicles separated by ultrafiltration centrifugation combined with density gradient ultracentrifugation (SMD = 3.58, 95% CI: 2.62-4.53, P < 0.0001, I2 = 31%) may be more effective than other EV isolation methods. For mouse Basso Mouse Scale scores, placenta-derived mesenchymal stem cell-derived extracellular vesicles worked better than bone mesenchymal stem cell-derived extracellular vesicles (placenta: SMD = 5.25, 95% CI: 2.45-8.06, P = 0.0421, I2 = 0%; bone marrow: SMD = 1.82, 95% CI: 1.23-2.41, P = 0.0421, I2 = 0%). For modified Neurological Severity Score, bone marrow-derived MSC-EVs worked better than adipose-derived MSC-EVs (bone marrow: SMD = -4.86, 95% CI: -6.66 to -3.06, P = 0.0306, I2 = 81%; adipose: SMD = -2.37, 95% CI: -3.73 to -1.01, P = 0.0306, I2 = 0%). Intravenous administration (SMD = -5.47, 95% CI: -6.98 to -3.97, P = 0.0002, I2 = 53.3%) and dose of administration equal to 100 μg (SMD = -5.47, 95% CI: -6.98 to -3.97, P < 0.0001, I2 = 53.3%) showed better results than other administration routes and doses. The heterogeneity of studies was small, and sensitivity analysis also indicated stable results. Last, the methodological quality of all trials was mostly satisfactory. In conclusion, in the treatment of traumatic central nervous system diseases, mesenchymal stem cell-derived extracellular vesicles may play a crucial role in promoting motor function recovery.

Bias in animal studies of estrogen effects on cardiovascular disease: A systematic review and meta‐analysis

BackgroundRandomized controlled trials on menopausal hormone therapy in humans have not confirmed the benefit of estrogens on cardiovascular disease found in animal studies. Flawed methodology or publication bias in animal studies may explain the dicrepancy. ObjectivesThe aim of this study was to investigate whether publication of the randomized controlled trials Heart and Estrogen/Progestin Replacement Study and Women’s Health Initiative influenced study authors’ assessment of research findings (confirmation bias) as well as to investigate publication bias and small‐study effects in animal studies of estrogen effects on atherosclerosis. MethodsThe data source for this study was PubMed from inception to 2018. We selected animal studies with cardiovascular outcomes comparing 17‐β‐estradiol, its natural metabolites, or conjugated equine estrogen with control. Qualitative data were extracted on authors’ conclusions about estrogen effects on cardiovascular disease, as well as quantitative data for atherosclerosis outcomes. Fixed‐ and random‐effects meta‐analyses were used. Publication bias/small‐study effects were assessed using funnel plots and Egger’s regression. Trim‐and‐fill plots and extrapolation from Egger’s regression were used to adjust for publication bias. The main outcomes and measures were the primary study authors' interpretation of their own results, and estrogen effects on cardiovascular disease in general before and after publication of the Women’s health Initiative study (2003). The effects of estrogens on atherosclerosis were measured as standardized mean difference between intervention and control. ResultsOf 1925 studies retrieved, 360 were eligible for analyses. Study‐specific statements concluded that estrogens were protective against cardiovascular disease in 75% of studies before 2003 and 78% after, but the percentage of general statements about estrogens being cardioprotective changed from 70% to 40%. Meta‐analyses showed less atherosclerosis in estrogen‐treated animals. Extremely skewed funnel plots and P < .01 in Egger’s regression suggested publication bias and/or exaggerated effects in small studies, which was more pronounced after 2002. There was substantial heterogeneity of effects (I2 = 68%‐86%) overall and in all subgroups except cynomolgus monkeys (I2 = 9%), the only animal subgroup without clear bias. Adjusting for publication bias, overall estimates of estrogen effects on atherosclerosis were close to null effect. ConclusionsWe found substantial evidence of publication bias but not of confirmation bias. Publication bias and flawed small studies may partly explain why findings differ between animal studies and clinical trials on the effect of estrogens on cardiovascular disease.

Platelet Features and Derivatives in Osteoporosis: A Rational and Systematic Review on the Best Evidence

Background: With the increase in aging population, the rising prevalence of osteoporosis (OP) has become an important medical issue. Accumulating evidence showed a close relationship between OP and hematopoiesis and emerging proofs revealed that platelets (PLTs), unique blood elements, rich in growth factors (GFs), play a critical role in bone remodeling. The aim of this review was to evaluate how PLT features, size, volume, bioactive GFs released, existing GFs in PLTs and PLT derivatives change and behave during OP. Methods: A systematic search was carried out in PubMed, Scopus, Web of Science Core Collection and Cochrane Central Register of Controlled Trials databases to identify preclinical and clinical studies in the last 10 years on PLT function/features and growth factor in PLTs and on PLT derivatives during OP. The methodological quality of included studies was assessed by QUIPS tool for assessing risk of bias in the clinical studies and by the SYRCLE tool for assessing risk of bias in animal studies. Results: In the initial search, 2761 studies were obtained, only 47 articles were submitted to complete reading, and 23 articles were selected for the analysis, 13 on PLT function/features and growth factor in PLTs and 10 on PLT derivatives. Risk of bias of almost all animal studies was high, while the in the clinical studies risk of bias was prevalently moderate/low for the most of the studies. The majority of the evaluated studies highlighted a positive correlation between PLT size/volume and bone mineralization and an improvement in bone regeneration ability by using PLTs bioactive GFs and PLT derivatives. Conclusions: The application of PLT features as OP markers and of PLT-derived compounds as therapeutic approach to promote bone healing during OP need to be further confirmed to provide clear evidence for the real efficacy of these interventions and to contribute to the clinical translation.

Mullerian inhibiting substance, sex hormone binding globulin and sex hormone levels in stimulant-naïve, first-diagnosed prepubertal boys with attention-deficit/hyperactivity disorder: comparison with matched healthy controls as well as before and after oros-methylpenidate treatment

Objectives: Attention-Deficit/Hyperactivity Disorder (ADHD) is a complex neurodevelopmental disorder with strong male predominance. Since Müllerian Inhibiting Substance (MIS) produces sex-linked bias in animal studies, we aimed to investigate the role of MIS, Sex Hormone Binding Globulin (SHBG) and sex hormone levels in boys with ADHD.Methods: We compared prepubertal, psychostimulant-naïve boys with ADHD with age-matched healthy control boys (HCs). Patients were re-evaluated after 30 days of methylphenidate treatment assessing ADHD severity, and serum MIS, testosterone, estradiol, and albumin concentrations.Results: Compared to 30 HCs, with ADHD (n = 49, age = 6.9 ± 0.2 years) had lower SHBG (p = .014), and higher free testosterone (p = 0.006) and bioavailable testosterone (p = .002) percentages. Methylphenidate improved ADHD measures (all p < .0001) and abnormal baseline hormonal levels, increasing SHBG levels (p = .024), and lowering free (p = .001) and bioavailable testosterone (p = .016) percentages so that only free testosterone percentages remained higher versus HCs post-treatment (p = .02).Conclusions: Compared to age- and sex-matched HCs, prepubertal, stimulant-naïve boys with ADHD had significantly lower SHBG and higher free and bioavailable testosterone percentages, suggesting a possible contribution of sex hormones to ADHD. Osmotic-release oral system methylphenidate treatment for 30 days significantly improved ADHD symptoms and abnormal sex hormone levels, normalizing SHBG and bioavailable testosterone percentages that were similar to HCs while free testosterone remained elevated versus HCs.Key pointsCompare to healthy matched controls prepubertal stimulant-naïve boys with ADHD had significantly lower SHBG and higher free and bioavailable testosterone percentages, suggesting a possible effect on sex hormones to ADHD.After 30-day methylphenidate treatment, ADHD symptoms significantly improved, and SHBG and bioavailable testosterone percentages normalized which were similar to HCs, while free testosterone remained elevated versus HCs.We found a negative relationship between MIS levels and hyperactivity scores in ADHD boys. This finding suggests that MIS may contribute to hyperactivity symptoms, either directly by affecting behavior or indirectly by affecting sex hormone levels.

Repurposing Proteostasis-Modifying Drugs to Prevent or Treat Age-Related Dementia: A Systematic Review.

Background: Dementia has a significant impact on quality of life of older individuals. Impaired proteostasis has been implicated as a potential cause of dementia, that can be therapeutically targeted to improve patient outcomes. This review aimed to collate all current evidence of the potential for targeting proteostasis with repurposed drugs as an intervention for age-related dementia and cognitive decline.Methods: PubMed, Web of Science and Embase databases were searched from inception until 4th July 2017 for studies published in English. Interventional studies of repurposed proteostasis-modifying drugs in Alzheimer's disease (AD), Parkinson's disease (PD), Lewy Body disease, vascular dementia, and cognitive aging, in either animal models or humans with change in cognition as the outcome were included. The SYRCLE and Cochrane tools were used to assess risk of bias for included studies.Results: Overall 47 trials, 38 animal and 9 human, were isolated for inclusion in this review. Drugs tested in animals and humans included lithium, rapamycin, rifampicin, and tyrosine kinase inhibitors. Drugs tested only in animals included Macrophage and Granulocyte-Macrophage Colony Stimulating Factors, methylene blue, dantrolene, geranylgeranylacetone, minocycline and phenylbutyric acid. Lithium (n = 10 animal, n = 6 human) and rapamycin (n = 12 animal, n = 1 human) were the most studied proteostasis modifying drugs influencing cognition. Nine of ten animal studies of lithium showed a statistically significant benefit in Alzheimer's models. Rapamycin demonstrated a significant benefit in models of vascular dementia, aging, and Alzheimer's, but may not be effective in treating established Alzheimer's pathology. Lithium and nilotinib had positive outcomes in human studies including Alzheimer's and Parkinson's patients respectively, while a human study of rifampicin in Alzheimer's failed to demonstrate benefit. Microdose lithium showed a strongly significant benefit in both animals and humans. While the risk of bias was relatively low in human studies, the risk of bias in animal studies was largely unclear.Conclusion: Overall, the collective findings support the hypothesis that targeting proteostasis for treatment of dementia may be beneficial, and therefore future studies in humans with repurposed proteostasis modifying drugs are warranted. Larger human clinical trials focusing on safety, efficacy, tolerability, and reproducibility are required to translate these therapeutics into clinical practice.

The relationship between study sponsorship, risks of bias, and research outcomes in atrazine exposure studies conducted in non-human animals: Systematic review and meta-analysis

BackgroundA critical component of systematic review methodology is the assessment of the risks of bias of studies that are included in the review. There is controversy about whether funding source should be included in a risk of bias assessment of animal toxicology studies. ObjectiveTo determine whether industry research sponsorship is associated with methodological biases, the results, or conclusions of animal studies examining the effect of exposure to atrazine on reproductive or developmental outcomes. MethodsWe searched multiple electronic databases and the reference lists of relevant articles to identify original research studies examining the effect of any dose of atrazine exposure at any life stage on reproduction or development in non-human animals. We compared methodological risks of bias, the conclusions of the studies, the statistical significance of the findings, and the magnitude of effect estimates between industry sponsored and non-industry sponsored studies. ResultsFifty-one studies met the inclusion criteria. There were no differences in methodological risks of bias in industry versus non-industry sponsored studies. 39 studies tested environmentally relevant concentrations of atrazine (11 industry sponsored, 24 non-industry sponsored, 4 with no funding disclosures). Non-industry sponsored studies (12/24, 50.0%) were more likely to conclude that atrazine was harmful compared to industry sponsored studies (2/11, 18.1%) (p value=0.07). A higher proportion of non-industry sponsored studies reported statistically significant harmful effects (8/24, 33.3%) compared to industry-sponsored studies (1/11; 9.1%) (p value=0.13). The association of industry sponsorship with decreased effect sizes for harm outcomes was inconclusive. ConclusionOur findings support the inclusion of research sponsorship as a risk of bias criterion in tools used to assess risks of bias in animal studies for systematic reviews. The reporting of other empirically based risk of bias criteria for animal studies, such as blinded outcome assessment, randomization, and all animals included in analyses, needs to improve to facilitate the assessment of studies for systematic reviews.

Industry sponsorship and publication bias among animal studies evaluating the effects of statins on atherosclerosis and bone outcomes: a meta-analysis.

BackgroundThe effect that sponsorship has on publication rates or overall effect estimates in animal studies is unclear, though methodological biases are prevalent in animal studies of statins and there may be differences in efficacy estimates between industry and non-industry sponsored studies. In the present analysis, we evaluated the impact of funding source on publication bias in animal studies estimating the effect of statins on atherosclerosis and bone outcomes.MethodsWe conducted two independent systematic reviews and meta-analyses identifying animal studies evaluating the effect of statins on reducing the risk of atherosclerosis outcomes (n = 49) and increasing the likelihood of beneficial bone outcomes (n = 45). After stratifying the included studies within each systematic review by funding source, three separate analyses were employed to assess publication bias in these meta-analyses—funnel plots, Egger’s Linear Regression, and the Trim and Fill methods.ResultsWe found potential evidence of publication bias, primarily in non-industry sponsored studies. In all 3 assessments of publication bias, we found evidence of publication bias in non-industry sponsored studies, while in industry-sponsored studies publication bias was not evident in funnel plots and Egger’s regression tests. We also found that inadequate reporting of sponsorship in animal studies is still exceedingly common.ConclusionsIn meta-analyses assessing the effects of statins on atherosclerosis and bone outcomes in animal studies, we found evidence of publication bias, though small numbers of industry-sponsored studies limit the interpretation of the trim-and-fill results. This publication bias is more prominent in non-industry sponsored studies. Industry and non-industry funded researchers may have different incentives for publication. Industry may have a financial interest to publish all preclinical animal studies to maximize the success of subsequent trials in humans, whereas non-industry funded academics may prefer to publish high impact statistically significant results only. Differences in previously published effect estimates between industry- and non-industry sponsored animal studies may be partially explained by publication bias.Electronic supplementary materialThe online version of this article (doi:10.1186/s12874-015-0008-z) contains supplementary material, which is available to authorized users.

Misleading Results: Don't Blame the Mice

In the news focus story “when mice mislead” (22 November 2013, p. [922][1]), J. Couzin-Frankel raises concerns about practices that give rise to misleading scientific findings. The case studies cited demonstrate procedural errors that can occur in the design or execution of experiments. These unfortunate mistakes can happen in any area of research and are not the exclusive domain of mice or animal models; clinical research is certainly not immune to error. Even well-executed studies can give rise to misleading results for reasons that are insidious and hard to control ([ 1 ][2]). With regard to mice, the first “risk of bias question” that a reviewer should ask is, “Was the result replicated in more than one genetic background?” Idiosyncrasies of particular mouse strains can be misleading ([ 2 ][3]). Ignoring the effect of genetic background is the most pervasive source of bias in animal studies. To reduce the problem of false or irreproducible scientific findings, we need to address two root causes. Science today is driven by an incentive system that often rewards precedence and impact over quality of the work. Statistical training of scientists often emphasizes analytical techniques over experimental design and quantitative reasoning. These are systemic problems that will not change without substantial effort. However, the message in this News Focus story does suggest some simple advice that can redress many of the pitfalls of experimental studies: Be wise, randomize. 1. [↵][4]1. J. P. A. Ioannidis , PLOS Med. 2, e124 (2005). [OpenUrl][5][CrossRef][6][PubMed][7] 2. [↵][8]1. V. Kumar 2. et al ., Science 342, 1508 (2013). [OpenUrl][9][Abstract/FREE Full Text][10] [1]: /lookup/doi/10.1126/science.342.6161.922 [2]: #ref-1 [3]: #ref-2 [4]: #xref-ref-1-1 View reference 1 in text [5]: {openurl}?query=rft.stitle%253DPLoS%2BMed%26rft.aulast%253DIoannidis%26rft.auinit1%253DJ.%2BP.%26rft.volume%253D2%26rft.issue%253D8%26rft.spage%253De124%26rft.epage%253De124%26rft.atitle%253DWhy%2Bmost%2Bpublished%2Bresearch%2Bfindings%2Bare%2Bfalse.%26rft_id%253Dinfo%253Adoi%252F10.1371%252Fjournal.pmed.0020124%26rft_id%253Dinfo%253Apmid%252F16060722%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [6]: /lookup/external-ref?access_num=10.1371/journal.pmed.0020124&link_type=DOI [7]: /lookup/external-ref?access_num=16060722&link_type=MED&atom=%2Fsci%2F343%2F6169%2F370.1.atom [8]: #xref-ref-2-1 View reference 2 in text [9]: {openurl}?query=rft.jtitle%253DScience%26rft.stitle%253DScience%26rft.aulast%253DKumar%26rft.auinit1%253DV.%26rft.volume%253D342%26rft.issue%253D6165%26rft.spage%253D1508%26rft.epage%253D1512%26rft.atitle%253DC57BL%252F6N%2BMutation%2Bin%2BCytoplasmic%2BFMRP%2Binteracting%2Bprotein%2B2%2BRegulates%2BCocaine%2BResponse%26rft_id%253Dinfo%253Adoi%252F10.1126%252Fscience.1245503%26rft_id%253Dinfo%253Apmid%252F24357318%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [10]: /lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6Mzoic2NpIjtzOjU6InJlc2lkIjtzOjEzOiIzNDIvNjE2NS8xNTA4IjtzOjQ6ImF0b20iO3M6MjQ6Ii9zY2kvMzQzLzYxNjkvMzcwLjEuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9

Bias Detection: Study Identifies Instruments for Evaluating Animal Studies

No good scientist wants to produce, or be accused of producing, a poorly conducted study. Even so, toxicologists conducting animal studies are not widely expected to document the steps they take to ensure internal validity—that is, to prevent their results from being skewed due to a methodological issue (a concept known as risk of bias). This is likely to change soon, and a team of researchers at the University of California, San Francisco (UCSF) has taken a first step toward a solution by conducting a systematic literature review that identifies 30 instruments for evaluating risk of bias in animal research.1 “Risk of bias” refers specifically to the introduction of systematic errors as a result of the way a study was conducted; it is unrelated to prejudice or manipulation of results to achieve a desired outcome. Other aspects of study quality include external validity (the extent to which results can be generalized) and reporting quality (descriptions of the experiment design, conduct, and analysis).2 Tools to assess these other aspects of study quality are familiar to toxicologists,3,4,5 but the concept of risk of bias is relatively new in environmental health, says Kris Thayer, director of the National Toxicology Program’s Office of Health Assessment and Translation (OHAT), who was not involved in the new review. Thayer’s group is leading efforts to incorporate systematic review methodology into OHAT evaluations.6 She says it’s important, when conducting these evaluations, to clearly define which aspects of study quality are being considered and to present them as discrete elements. She lauds the new review for providing a starting point for determining how to evaluate the risk of bias in animal studies. The review presents an inventory of different approaches for assessing existing study-quality tools and summarizes what is known about specific factors as sources of systematic bias. “Although there is a well-developed and empirically based literature on how to evaluate the risk of bias of randomized controlled clinical trials, less is known about how to do this for animal studies,” wrote the team of UCSF researchers, who were led by Lisa Bero, an expert in systematic review methods and evidence-based health care. Bero and her coauthors combed through 45 years of MEDLINE publications and found 3,731 articles describing instruments for assessing risk of bias in animal studies. They identified 30 instruments that assess anywhere from 2 to 25 criteria associated with the risk of bias, methodological, or reporting criteria. Rather than pinpointing a single “best instrument,” the authors attempt to identify the core set of procedures important for evaluating risk of bias in animal studies, Bero says. These procedures include some process of random allocation to experimental groups; a blinded assessment of outcome; statistical tools to be sure that studies aren’t underpowered; recording both inclusion and exclusion criteria to show that the animals were appropriate for the study; and ensuring that all collected data were reported. Figure 1 Bias, or the introduction of errors into experiments, may be one explanation for equivocal research findings. The authors acknowledge that they may not have identified all published assessment instruments because they searched only for English-language publications using just one database. “However,” they wrote, “both our consultation with a librarian and the large pool of studies identified through the electronic search suggests that it was comprehensive.”1 Study coauthor Tracey Woodruff of UCSF’s Department of Obstetrics, Gynecology and Reproductive Sciences and Program on Reproductive Health and the Environment, calls the review “a foundational piece of a broader method that is going to greatly improve our ability to evaluate environmental chemicals.” Woodruff’s group is working to establish how the systematic reviews initially developed for use in the decades-old field of evidence-based medicine can be applied to environmental health and has developed a strategy known as the Navigation Guide.7 Woodruff’s overarching goal is to encourage medical practitioners to educate their patients about hazardous environmental exposures, particularly ones that may affect their reproductive health.8 Malcolm R. Macleod, a professor at the University of Edinburgh’s Centre for Clinical Brain Sciences, who was not involved in the study, calls the review a small but “really important” contribution toward reducing the risk of bias, says. Macleod’s interest in the subject sprang from his work on clinical trials in which drug candidates recommended by animal stroke studies ended up failing in humans. His subsequent analysis9 demonstrated how study quality and publication bias in those animal studies led to a major overstatement of the drugs’ efficacy. His publications and others document that lack of randomization or blinding, failure to specify inclusion/exclusion criteria or use of comorbid animals, and lack of statistical power in animal studies have inflated the estimates of the effects of pharmaceutical interventions for a number of human maladies.10,11,12,13,14 They inspired high-profile pleas to establish better reporting criteria that incorporates risk-of-bias assessments.15,16

Evaluation of excess significance bias in animal studies of neurological diseases.

Author SummaryStudies have shown that the results of animal biomedical experiments fail to translate into human clinical trials; this could be attributed either to real differences in the underlying biology between humans and animals, to shortcomings in the experimental design, or to bias in the reporting of results from the animal studies. We use a statistical technique to evaluate whether the number of published animal studies with “positive” (statistically significant) results is too large to be true. We assess 4,445 animal studies for 160 candidate treatments of neurological disorders, and observe that 1,719 of them have a “positive” result, whereas only 919 studies would a priori be expected to have such a result. According to our methodology, only eight of the 160 evaluated treatments should have been subsequently tested in humans. In summary, we judge that there are too many animal studies with “positive” results in the neurological disorder literature, and we discuss the reasons and potential remedies for this phenomenon.

Season change bias in animal studies

Season change bias in animal studies