Quotation Errors Research Articles

Where did this come from? When (not how) to cite sources in scientific publications

In the scientific literature, the link between an idea and its source is its reference information that allows the source to be identified and located. Not so obvious is where the source is cited in the text. Although authors are given extensive details on how to format references, they are not always taught when to cite them. Further, many are not vigilant in confirming the accuracy of the quoted information against the source or in verifying the associated reference information. In fact, discrepancies between the information cited in the text and the information actually presented in the source are common and often serious. Further, inaccuracies in references that break the link between the citation and its source are even more common. These discrepancies and inaccuracies affect the integrity of science and the validity of the citation metrics (for example, the Journal Impact Factor) that are used, rightly or wrongly, to evaluate the importance of journals and authors. Here, I discuss factors affecting when and where sources should be cited. I also consider factors that can bias the selection of sources and so interfere with the validity of citation analyses, review some considerations for evaluating a source, call attention to citation and quotation error rates, and review some strategies for reducing these errors. Finally, I summarise the most common recommendations for when, what, where, and why sources should or should not be cited.

Assessing citation integrity in biomedical publications: corpus annotation and NLP models.

Citations have a fundamental role in scholarly communication and assessment. Citation accuracy and transparency is crucial for the integrity of scientific evidence. In this work, we focus on quotation errors, errors in citation content that can distort the scientific evidence and that are hard to detect for humans. We construct a corpus and propose natural language processing (NLP) methods to identify such errors in biomedical publications. We manually annotated 100 highly-cited biomedical publications (reference articles) and citations to them. The annotation involved labeling citation context in the citing article, relevant evidence sentences in the reference article, and the accuracy of the citation. A total of 3063 citation instances were annotated (39.18% with accuracy errors). For NLP, we combined a sentence retriever with a fine-tuned claim verification model to label citations as ACCURATE, NOT_ACCURATE, or IRRELEVANT. We also explored few-shot in-context learning with generative large language models. The best performing model-which uses citation sentences as citation context, the BM25 model with MonoT5 reranker for retrieving top-20 sentences, and a fine-tuned MultiVerS model for accuracy label classification-yielded 0.59 micro-F1 and 0.52 macro-F1 score. GPT-4 in-context learning performed better in identifying accurate citations, but it lagged for erroneous citations (0.65 micro-F1, 0.45 macro-F1). Citation quotation errors are often subtle, and it is currently challenging for NLP models to identify erroneous citations. With further improvements, the models could serve to improve citation quality and accuracy. We make the corpus and the best-performing NLP model publicly available at https://github.com/ScienceNLP-Lab/Citation-Integrity/.

Do “Ten simple rules for neuroimaging meta-analysis” receive equal attention and accurate quotation? An examination on the quotations to an influential neuroimaging meta-analysis guideline

The collection of recommendations and guidelines for conducting and reporting neuroimaging meta-analyses, called “Ten simple rules for neuroimaging meta-analysis” by Müller et al., has been published for a few years. Here, the papers citing this reference were examined to evaluate the rationale of the quotations and what quotation errors existed. In May 2023, an online query via Scopus identified 386 papers citing this reference, 2 of which were inaccessible. The resultant 384 papers were checked to identify the total number of quotations to the reference, the exact quotations, which of the ten recommendations/rules was concerned by each quotation, and if any quotation error existed. Results found that the reference by Müller et al. were quoted 804 times by the 384 papers, meaning an average of 2.1 quotations per paper. Out of the 804 quotations, the three rules that the researchers most frequently referred were the power of the meta-analysis (Rule #2, 14.1%), the consistency of the search coverage and reference space (Rule #4, 13.8%), and the statistical threshold (Rule #8, 10.2%). Overall, 63 quotations from 51 papers contained some errors. In other words, 7.8% (63/804) of the quotations contained errors and they involved 13.3% (51/384) of the papers. The commonest quotation errors were dealing with a failure to substantiate the assertion, unrelated to the assertion, and oversimplification of the original notion. Some notable quotation error examples were to quote Müller et al. to substantiate the assertion of having at least 10 datasets to be considered to have adequate power for ES-SDM meta-analysis (no such recommendation), and having a misquoted primary cluster-forming threshold of p < 0.05 or p < 0.005 (should be p < 0.001). The neuroscience community should be cautious and double-check the accuracy of assertions, even with a quotation.

Some consideration triggered by misquotation of Temkin model and the derivation of its correct form

Accuracy of quotations is very important for the transmission of scientific knowledge, whereas the misquotations may significantly diminish the academic value of the paper. Recently, Sulyman et al. (2021) published a high academic paper to reveal the adsorption of crystal violet from aqueous solution by a new composite adsorbent in Arabian Journal of Chemistry (2021) 14, 103115. However, the Temkin isotherm was misquoted, which triggered some consideration on the correct form of Temkin model. The present study reviews the original Temkin model proposed by Mikhail I. Temkin in the 1930s, and derives the correct Temkin model by using the adsorption theory. The most striking feature of the Temkin model in its correct form is that the dimensions on both sides are consistent and dimensionless, and the unit of adsorption heat parameter ( b T ) is J/mol. Unfortunately, the dimensions of the Temkin model widely cited in the literature are imbalanced, and the unit of b T is confusing. If there is a quotation error in a published paper, it may mislead readers. This comment can deepen readers' understanding of the Temkin model, avoiding the misuse and propagation of its wrong form.

Quotation errors related to the wound management of open lower limb fractures (WOLLF) randomized clinical trial.

We assessed the accuracy of quotations in the published literature of the Wound Management of Open Lower Limb Fractures (WOLLF) Randomized Clinical Trial. A literature search was performed to identify all citations of WOLLF from 2018 to 2021. Duplicate papers, non-English language papers, textbooks, technical tip videos, Ph.D. theses, and other grey literature were excluded from analysis. Eligible publications had their full text assessed by 2 independent reviewers who used a validated framework of error classification. We identified 62 original papers that met our eligibility criteria and referenced WOLLF. Of the 62 papers, 12 contained a quotation error (12/62, 19%). Errors were classified into major and minor. There were 7 major errors and 5 minor errors. The majority of quotation errors (7/12, 58%) occurred due to multi-referencing, where groups of references were used to support single assertions. There was substantial agreement between the two independent reviewers as determined by a Kappa coefficient of 0.761. Our study demonstrated a 19% quotation error rate in the literature regarding WOLLF. The majority were due to multi-referencing. We suggest that both authors and reviewers carefully check the source literature of key references.

Quotation Accuracy of Systematic Review and Meta-Analysis Protocols on Acupuncture.

Currently, published systematic review protocols (SR protocols) have increasingly become a new trend in fields such as acupuncture and are therefore a new source of quotations in these fields. Systematic reviews are considered the pinnacle of the evidence pyramid as they embody comprehensive literature searching. Quotations are key elements to achieve this goal as they can support the assertions of the original authors, but the ‘misquotation’ exists, too, and they can be misleading to the reader. The aim of this study was to examine the quotation accuracy of SR protocols in a meta-analysis on acupuncture research. We searched SCOPUS through 31 December, 2020, and each protocol and its citations were analyzed and classified as correct or incorrect. We used descriptive statistics to report the quotation errors and characteristics of the included protocols. The results showed 248 SR protocols, where 124 protocols received quotations and 38 quotations (31.4%) were erroneous. Only 11 (4.4%) of the published SRs and SR protocols had been published previously. Furthermore, the scientific journal in which the most SR protocols were published was Medicine (193; 77.8%), followed by BMJ Open (39; 15.7%). Authors from China (86.5%) were the most productive in publishing SRs and SR protocols. Finally, we concluded that the number of SR protocols and meta-analyses published in scientific journals and indexed by databases exceeds the publication capacity of the SRs associated with them, generating scientific literature that does not make any novel contribution to knowledge.

Quotation errors related to the Distal Radius Acute Fracture Fixation Trial paper.

Quotation error is an inaccuracy in the assertions made by authors when referencing another's work. This study aimed to assess the quotation errors in articles referencing the Distal Radius Acute Fracture Fixation Trial (DRAFFT). A literature search was performed to identify all citations of DRAFFT from 2014 to 2020. The relevant publications were assessed by two reviewers using a validated framework of error classification. There were 83 articles containing references to DRAFFT. There was substantial agreement between the two reviewers (Kappa coefficient 0.66). We found 22/83 (28%) of articles contained an error, with one article containing two errors. There were 12 major errors, which were not substantiated by, were unrelated to or contradicted the findings of DRAFFT, and 11 minor errors, including numerical inaccuracies, oversimplification or generalization. This study highlights that a significant number of articles inaccurately quote DRAFFT. Authors and journals should consider checking the accuracy of key referenced statements.

Quotation Accuracy Matters: An Examination of How an Influential Meta-Analysis on Active Learning Has Been Cited

When previous research is cited incorrectly, misinformation can infiltrate scientific discourse and undermine scholarly knowledge. One of the more damaging citation issues involves incorrectly citing article content (called quotation errors); therefore, investigating quotation accuracy is an important research endeavor. One field where quotation accuracy is needed is in the learning sciences given its impact on pedagogy. An integral article in pedagogical discussions surrounding how to teach at the college level is the meta-analysis on active learning by Freeman et al. The Freeman et al. meta-analysis compared active learning to traditional lecture in terms of its effects on student learning and has been important in national initiatives on STEM (science, technology, engineering, and mathematics) reform. Given its influence coupled with the impact quotation errors could have in scientific discourse, we used citation context analysis to analyze whether assertions in the citing text that related to the efficacy of lecture and active learning were supported by what was explicitly stated in the cited meta-analysis. Assertions were analyzed under supported, unsupported, or irrelevant for purposes of study categories. The most prevalent supported category related to active learning being more effective than lecture; the most prevalent unsupported category related to the effectiveness of specific activities/approaches other than the general approach of active learning. Overall, the percentage of supported assertions was 47.67%, and the percentage of unsupported assertions was 26.01%. Furthermore, the percentage of articles containing at least one unsupported assertion was 34.77%. Proactive measures are needed to reduce the incidence of quotation errors to ensure robust scientific integrity.

Quotation Errors in High-Impact-Factor Orthopaedic and Sports Medicine Journals.

Background:Inappropriate referencing of the existing literature has the potential to propagate false information. Quotation errors are defined as citations in which the referenced article fails to substantiate the authors’ claims. The aim of this study was to determine the rate of quotation errors in high-impact general orthopaedic and sports medicine journals and to determine whether there are article or journal-related factors that are related to the rate of inaccuracies.Methods:A total of 250 citations from the 5 orthopaedic and sports medicine journals with the highest impact factors in 2019 (per Journal Citation Reports) were chosen using a random sequence generator. Reviewers rated the chosen citations by comparing the claims made by the authors with the data and conclusions of the referenced source to determine whether quotation errors were present. Logistic regression was utilized to assess for article- and journal-related factors related to the rate of quotation errors.Results:The overall quotation error rate was 13.6%. A total of 2.8% of the claims were completely unsubstantiated. The number of quotation errors did not significantly differ between the included journals. Single citations were significantly more likely than string citations to result in citations that could not be fully substantiated (χ2 = 4.57; odds ratio = 2.22; 95% confidence interval = 1.06 to 4.66; p = 0.03). No relationship was found between the rate of quotation errors and the total number of citations in the article, study type, or the graded level of evidence of the article.Conclusions:Quotation errors in high-impact factor orthopaedic and sports medicine journals are common. This is particularly important given the higher likelihood that studies in these journals are cited elsewhere, thus propagating the inaccuracies. Efforts from both authors and journals are needed to reduce quotation errors in the orthopaedic literature.

Quotation errors in general science journals

Due to the incremental nature of scientific discovery, scientific writing requires extensive referencing to the writings of others. The accuracy of this referencing is vital, yet errors do occur. These errors are called ‘quotation errors’. This paper presents the first assessment of quotation errors in high-impact general science journals. A total of 250 random citations were examined. The propositions being cited were compared with the referenced materials to verify whether the propositions could be substantiated by those materials. The study found a total error rate of 25%. This result tracks well with error rates found in similar studies in other academic fields. Additionally, several suggestions are offered that may help to decrease these errors and make similar studies more feasible in the future.

Reference accuracy in spine surgery.

The references list is an important part of a scientific article that serves to confirm the accuracy of the authors' statements. The goal of this study was to evaluate the reference accuracy in the field of spine surgery. Four major peer-reviewed spine surgery journals were chosen for this study based on their subspecialty clinical impact factors. Sixty articles per journal were selected from 12 issues each of The Spine Journal, Spine, and Journal of Neurosurgery: Spine, and 40 articles were selected from 8 issues of Global Spine Journal, for a total of 220 articles. All the articles were published in 2019 and were selected using computer-generated numbers. From the references list of each article, one reference was again selected by using a computer-generated number and then checked for citation or quotation errors. The results indicate that 84.1% of articles have a minor citation error, 4.5% of articles have a major citation error, 9.5% of articles have a minor quotation error, and 9.1% of articles have a major quotation error. Journal of Neurosurgery: Spine had the fewest citation errors compared with the other journals evaluated in this study. Using chi-square analysis, no association was determined between the occurrence of errors and potential markers of reference mistakes. Still, statistical significance was found between the occurrence of citation errors and the spine journals tested. In order to advance medical treatment and patient care in spine surgery, detailed documentation and attention to detail are necessary. The results from this study illustrate that improved reference accuracy is required.

How accurate is the neurosurgery literature? A review of references.

The reference list is an important part of academic manuscripts. The goal of this study is to evaluate the reference accuracy in the field of neurosurgery. This study examines four major peer-reviewed neurosurgery journals, chosen based on their clinical impact factor: Neurosurgery, J Neurosurg, World Neurosurg, and Acta Neurochir. For each of the four journals, five articles from each of the journal's 12 issues published in 2019 were randomly selected using an online generator. This resulted in a total of 240 articles, 60 from each journal. Additionally, from each article's list of references, one reference was again randomly selected and checked for a citation or quotation error. The chi-square test was used to analyze the association between the occurrence of citation and quotation errors and the presence of hypothesized risk factors that could impact reference accuracy. 62.1% of articles had a minor citation error, 8.33% had a major citation error, 12.1% had a minor quotation error, and 5.8% of articles had a major quotation error. Overall, Acta Neurochir presented with the fewest quotation errors compared with the other journals evaluated. The only association between the frequency of errors and potential markers of reference mistakes was with the length of the bibliography. Surprisingly, this correlation indicated that the articles with longer reference lists had fewer citation errors (p < 0.01). Statistical significance was found between the occurrence of citation errors and the journals of publication (p < 0.01). In order to advance medical treatment and patient care in neurosurgery, detailed documentation and attention to detail are necessary. The results from this analysis illustrate that improved reference accuracy is required.

Quotation errors related to the Proximal Fracture of the Humerus Evaluation by Randomization (ProFHER) study.

The aim of this study was to assess the accuracy of quotations of the Proximal Fracture of the Humerus Evaluation by Randomization (ProFHER) study in the published literature. A literature search was performed from March 2015 to November 2019 to identify all papers that reference ProFHER since its publication. Full text articles were reviewed by two independent reviewers using a validated framework of assessing quotation errors. A kappa co-efficient was calculated to assess interobserver reliability of the reviewers. There were 260 individual ProFHER quoted references within the 138 included articles. We identified 35/260 quotation errors (13%). Of these, 10/35 (29%) were major quotation errors and 25/35 (71%) minor quotation errors. There was substantial interobserver agreement when errors were classified. Of the 10 major errors, six quotations were not substantiated by the results of ProFHER and three were unrelated to ProFHER. One paper contained a quotation error that contradicted the results of ProFHER. Of the 25 minor errors, 19 oversimplified or generalised the conclusions of ProFHER and six contained numerical or grammatical errors. The current study demonstrated substantial inaccuracies in quotations of the Proximal Fracture of the Humerus Evaluation by Randomization study. Vigilance is recommended when quoting the literature and reviewing submitted papers in order to prevent the perpetuation of misquoted data.

Importance, Errors, and Patterns of Quotations to Psychiatric Original Articles.

A substantial rate of quotation errors has been reported in medical journal publications: about 25% of all quotations are wrong. It is, however, entirely unclear how important quotation errors are for the message of quoting articles. This is a case study in form of a retrospective quotation analysis of a cohort of 72 psychiatric original articles (index articles) from 5 German-language general psychiatric journals. Main outcomes were importance and accuracy of quotations from the 2 calendar years following the publication of index articles. Fifty-one index articles were quoted 235 times in 109 quoting articles. Almost all quotations were of medium (76% [95% CI: 70%; 81%]) or high (20% [15%, 25%]) importance for the message of the quoting paper. Regarding quotation accuracy, 44 quotations (19% [14%; 24%]) were rated as minor, and 51 (22% [17%; 27%]) as major errors. In multivariable analyses, no statistically significant and practically relevant factors associated with quotation inaccuracy emerged, such as self-quotation, impact factor of the quoting journal, or importance. Among quoting articles, 7 (6% [3%; 13%]) showed a pattern of predominantly quoting index articles from the time span relevant to the calculation of the impact factor. Quotations are important for the message of the quoting paper. Therefore, quotation errors may be detrimental to scientific reasoning and may undermine public trust in medical science. The present investigation is a case study, and its results are exploratory. While it is plausible that the findings translate into other environments, independent replication is needed.

When “Normal” Becomes Normative: A Case Study of Researchers’ Quotation Errors When Referring to a Focus Group Sample Size Study

In 2011, we published a review exploring how researchers report and justify their focus group sample sizes. We concluded that sample sizes vary widely and that most researchers give no explanation for their sample size. The aim of our 2011 study was to describe practice rather than develop guidance. However, after our study was published, we noticed that new researchers were using our information about typical sample sizes as justification for their own sample size. In other words, practice that we had presented as typical or “normal” but generally lacking in justification was being used as normative. The current study aims to explore the misrepresentation of descriptive information as normative. Specifically, we map this type of quotation error in references to our 2011 study. Using Google Scholar, we identified all articles referencing our study. We then extracted quotations where the researchers had referred to our study and categorized these as follows: (a) quotations where the researchers had used the descriptive information from our study to justify their sample size and (b) quotations where the researchers had referred to our study for other purposes or where the purpose was unclear. We assessed 205 articles that had referred to our 2011 study. We identified the type of quotation error we were interested in, namely the misrepresentation of descriptive information as normative, in 50.7% of the included articles. Our study shows very high rates of one type of quotation error: the misrepresentation of descriptive information about focus group sample size as normative. Researchers referring to other researchers’ work carry most of the responsibility for ensuring that they do this appropriately. However, the authors of the research being referred to also need to consider how they can make their results clearer. We offer suggestions as to how this might be achieved.

Massive citations to misleading methods and research tools: Matthew effect, quotation error and citation copying.

Massive citations to misleading methods and research tools: Matthew effect, quotation error and citation copying.

Case study in major quotation errors: a critical commentary on the Newcastle-Ottawa scale.

The Newcastle-Ottawa scale (NOS) is one of many scales used to judge the quality of observational studies in systematic reviews. It was criticized for its arbitrary definitions of quality items in a commentary in 2010 in this journal. That commentary was cited 1,250 times through December 2016. We examined the citation history of this commentary in a random sample of 100 full papers citing it, according to the Web of Science. Of these, 96 were systematic reviews, none of which quoted the commentary directly. All but 2 of the 96 indirect quotations (98%) portrayed the commentary as supporting use of the NOS in systematic reviews when, in fact, the opposite was the case. It appears that the vast majority of systematic review authors who cited this commentary did not read it. Journal reviewers and editors did not recognize and correct these major quotation errors. Authors should read each source they cite to make sure their direct and indirect quotations are accurate. Reviewers and editors should do a better job of checking citations and quotations for accuracy. It might help somewhat for commentaries to include abstracts, so that the basic content can be conveyed by PubMed and other bibliographic resources.

Reference Errors in Otolaryngology-Head and Neck Surgery Literature.

Objective Proper use of citation and quotation is crucial to the integrity of the medical literature. The purpose of this study was to determine the prevalence of quotation and citation errors in otolaryngology-head and neck surgery (OHNS) journals and how they have changed over time. Study Design Literature review. Methods Fifty references were randomly selected from the first published issue of 2017 for 8 leading OHNS journals. These were analyzed for errors in citation (data elements by which the article is referenced) and quotation (factual inaccuracies of the reference). Citation errors were categorized as major, intermediate, or minor. Quotation errors were categorized as major or minor. Results were compared with data from 1997 articles. Results Citation errors occurred in 17% of all references studied, with 34% classified as major. Quotation errors occurred in 9%, with 69% classified as major. There was no association between journal impact factor and total number of errors ( r = -0.33, P = .42). This compares with a 37% citation error rate (32% major) and 17% quotation error rate (65% major) from 1997. Conclusion Citation and quotation errors are still prevalent in the OHNS literature albeit decreased from previously reported data. Improvement in citation errors may be due to technological improvements in reference management. However, it is the continued responsibility of the authors, reviewers, and editors to further reduce error rates to maintain the integrity of our publications.

Normalization of Deviation: Quotation Error in Human Factors.

Objective The objective of this paper is to examine quotation error in human factors. Background Science progresses through building on the work of previous research. This requires accurate quotation. Quotation error has a number of adverse consequences: loss of credibility, loss of confidence in the journal, and a flawed basis for academic debate and scientific progress. Quotation error has been observed in a number of domains, including marine biology and medicine, but there has been little or no previous study of this form of error in human factors, a domain that specializes in the causes and management of error. Methods A study was conducted examining quotation accuracy of 187 extracts from 118 published articles that cited a control article (Vaughan's 1996 book: The Challenger Launch Decision: Risky Technology, Culture, and Deviance at NASA). Results Of extracts studied, 12.8% ( n = 24) were classed as inaccurate, with 87.2% ( n = 163) being classed as accurate. A second dimension of agreement was examined with 96.3% ( n = 180) agreeing with the control article and only 3.7% ( n = 7) disagreeing. The categories of accuracy and agreement form a two by two matrix. Conclusion Rather than simply blaming individuals for quotation error, systemic factors should also be considered. Vaughan's theory, normalization of deviance, is one systemic theory that can account for quotation error. Application Quotation error is occurring in human factors and should receive more attention. According to Vaughan's theory, the normal everyday systems that promote scholarship may also allow mistakes, mishaps, and quotation error to occur.

Accuracy of cited "facts" in medical research articles: A review of study methodology and recalculation of quotation error rate.

Previous reviews estimated that approximately 20 to 25% of assertions cited from original research articles, or “facts,” are inaccurately quoted in the medical literature. These reviews noted that the original studies were dissimilar and only began to compare the methods of the original studies. The aim of this review is to examine the methods of the original studies and provide a more specific rate of incorrectly cited assertions, or quotation errors, in original research articles published in medical journals. Additionally, the estimate of quotation errors calculated here is based on the ratio of quotation errors to quotations examined (a percent) rather than the more prevalent and weighted metric of quotation errors to the references selected. Overall, this resulted in a lower estimate of the quotation error rate in original medical research articles. A total of 15 studies met the criteria for inclusion in the primary quantitative analysis. Quotation errors were divided into two categories: content ("factual") or source (improper indirect citation) errors. Content errors were further subdivided into major and minor errors depending on the degree that the assertion differed from the original source. The rate of quotation errors recalculated here is 14.5% (10.5% to 18.6% at a 95% confidence interval). These content errors are predominantly, 64.8% (56.1% to 73.5% at a 95% confidence interval), major errors or cited assertions in which the referenced source either fails to substantiate, is unrelated to, or contradicts the assertion. Minor errors, which are an oversimplification, overgeneralization, or trivial inaccuracies, are 35.2% (26.5% to 43.9% at a 95% confidence interval). Additionally, improper secondary (or indirect) citations, which are distinguished from calculations of quotation accuracy, occur at a rate of 10.4% (3.4% to 17.5% at a 95% confidence interval).