Can ChatGPT Recognize Its Own Writing in Scientific Abstracts?

  • Abstract
  • Literature Map
  • Similar Papers
Abstract
Translate article icon Translate Article Star icon
Take notes icon Take Notes

Background: With the growing use of generative AI in scientific writing, distinguishing between AI-generated and human-authored content has become a pressing challenge. It remains unclear whether ChatGPT (OpenAI, San Francisco, CA) can accurately and consistently recognize its own output.Methods: We randomly selected 100 research articles published in 2000, before the advent of generative AI, from 10 high-impact internal medicine journals. For each article, a structured abstract was generated using ChatGPT-4.0 based on the full PDF. The original and AI-generated abstracts (n = 200) were then evaluated twice by ChatGPT-4.0, which was asked to rate the likelihood of authorship on a 0-10 scale (0 = definitely human, 10 = definitely ChatGPT, 5 = undetermined). Classifications of 0-4 were considered human, and 6-10 were considered AI generated.Results: Misclassification rates were high in both rounds (49% and 47.5%). No abstract received a score of 5. Score distributions overlapped substantially between groups, with no statistically significant difference (Wilcoxon p-value = 0.93 and 0.21). Cohen’s kappa for binary classification was 0.33 (95% CI: 0.19-0.46) and weighted kappa on the 0-10 scale was 0.24 (95% CI: 0.15-0.34), both reflecting poor agreement.Conclusion: ChatGPT-4.0 cannot reliably identify whether a scientific abstract was written by itself or by humans. More robust external tools are needed to ensure transparency in academic authorship.

Similar Papers
  • Research Article
  • Cite Count Icon 58
  • 10.1136/bmj.k3654
Collaboration between academics and industry in clinical trials: cross sectional study of publications and survey of lead academic authors
  • Oct 3, 2018
  • BMJ
  • Kristine Rasmussen + 4 more

ObjectivesTo determine the role of academic authors, funders, and contract research organisations in industry funded trials of vaccines, drugs, and devices and to determine lead academic authors’ experiences with industry...

  • Research Article
  • 10.4081/gc.2020.9227
Honorary authorship in Geriatric Literature: do authors adhere to the ICMJE-guidelines?
  • Oct 1, 2020
  • Geriatric Care
  • Alex Verhemel + 3 more

To protect appropriate authorship, the International Committee of Medical Journal Editors (ICMJE) formulated a guideline on authorship. Researchers not fulfilling these criteria and still enlisted as author are seen as honorary authors (HA). The objective of this study is to assess authorship decision making and the proportion of HA in journals in the field of geriatrics and gerontology. Corresponding authors of six highimpact journals in geriatrics and gerontology were sent a survey. The survey consisted of three parts: i) demographics of the respondent; ii) awareness of authorship guidelines; and iii) authorship decisions made for the paper they are authors of. Respondents were also asked if one of their coauthors does not deserve authorship. This is defined as self-perceived HA. Furthermore, respondents were asked if any of their co-authors only performed tasks from a list of non-authorship tasks. This is defined as ICMJE-defined HA. Of the 1592 contacted authors, 528 filled in a survey (response rate 33.2%). 84.4% was aware of the ICMJE-guidelines, but 44.2% was unaware of the issue of HA. The proportion of self-perceived HA was 12.7%. Independent factors associated with more self-perceived HA were having a senior member automatically enlisted as coauthor [odds ratio (OR) 3.4, 95%confidence interval (CI) 1.8 to 6.4] and have gotten the suggestion to include an HA (OR 11.1, 95% CI 4.4 to 27.9). The proportion of ICMJE defined HA was 39.3%. The journal surveyed (OR 1.2, 1.0 to 1.3) was associated with more, and awareness of the ICMJE-guidelines (OR 0.5, 95% CI 0.3 to 0.9) was associated with less ICMJE-defined HA respectively. Having a senior member automatically enlisted as co-author (OR 2.1 95% CI 1.3 to 3.4) and having anyone suggest to include an HA (OR 4.8 95% CI 1.8 to 12.8) were also associated with more ICMJE-defined HA. More than one out of ten of the corresponding authors thinks that based on the ICMJE-guidelines, one or more of their coauthors did not deserve authorship. A stricter journal policy and more awareness of the ICMJE-guidelines could help reduce the proportion of HA.

  • Research Article
  • Cite Count Icon 21
  • 10.1192/bjo.2017.8
Trends in female authorship in research papers on eating disorders: 20-year bibliometric study.
  • Feb 22, 2018
  • BJPsych open
  • Mattias Strand + 1 more

There is a clear gender gap in scientific authorship. Although the proportions of female authors in medicine and psychiatry have increased over the past decades, women are still underrepresented. Aims To analyse authorship gender trends in eating disorder research. First and last author gender in research articles on eating disorders during the period 1997-2016 were assessed in eating disorder specialty journals, high-impact psychiatry journals and high-impact clinical psychology journals. The total number of papers on eating disorders increased substantially over the observation period, although a decrease was observed in high-impact psychiatry journals. Female authorship increased in both specialty journals and high-impact psychiatry journals. Authors were significantly less likely to be female in high-impact psychiatry and clinical psychology journals than in speciality journals. Eating disorder research has been increasingly allocated to specialty journals over the past 20 years. A consistent gender gap between specialty and high-impact journals exists. Declaration of interest C.M.B is a grant recipient from Shire Pharmaceuticals, Inc. and has participated as a member of their scientific advisory board. These positions are unrelated to the content of this article.

  • Research Article
  • Cite Count Icon 48
  • 10.1111/jnu.12093
Honorary and Ghost Authorship in Nursing Publications
  • Jun 13, 2014
  • Journal of Nursing Scholarship
  • Maureen Shawn Kennedy + 2 more

The purposes of this study were to (a) assess the prevalence of articles with honorary authors and ghost authors in 10 leading peer-reviewed nursing journals between 2010 to 2012; (b) compare the results to prevalence reported by authors of articles published in high-impact medical journals; and (c) assess the experiences of editors in the International Academy of Nursing Editors with honorary and guest authorship. Corresponding authors of articles published in 10 nursing journals between 2010 and 2012 were invited to complete an online survey about the contributions of coauthors to see if the International Committee of Medical Journal Editors () criteria for authorship were met. Additionally, members of the International Academy of Nursing Editors were invited to complete an online survey about their experiences in identifying honorary or ghost authors in articles submitted for publication. The prevalence of articles published in 10 nursing journals with honorary authors was 42%, and the prevalence of ghost authorship was 27.6%. This is a greater prevalence than what has been reported among medical journals. Qualitative data yielded five themes: lack of awareness around the rules for authorship; acknowledged need for debate, discussion, and promotion of ethical practice; knowingly tolerating, and sometimes deliberately promoting, transgressions in practice; power relations and expectations; and avoiding scrutiny. Among the 60 respondents to the editor survey, 22 (36.7%) reported identifying honorary authors and 13 (21.7%) reported ghost authors among papers submitted to their publications. Inappropriate authorship is a significant problem among scholarly nursing publications. If nursing scholarship is to maintain integrity and be considered trustworthy, and if publications are to be a factor in professional advancement, editors, nursing leaders, and faculty need to disseminate and adhere to ethical authorship practices.

  • Research Article
  • Cite Count Icon 2
  • 10.17294/2330-0698.1735
Financial Conflicts of Interest Change After a High-Impact Clinical Trial Publication in Oncology.
  • Jul 27, 2020
  • Journal of Patient-Centered Research and Reviews
  • Craig L Cambridge + 3 more

Because financial conflicts of interest (FCOIs) may potentially influence patient care, hospital drug formularies, and treatment guidelines, it is important that these are disclosed. The purpose of this observational study was to quantify the changes in FCOI among U.S.-based academic authors in industry-sponsored oncology trials after a high-impact publication. A list of all U.S.-based academic authors (authors) of industry-sponsored solid tumor clinical trials published between August 1, 2014, and December 31, 2015, in 6 high-impact journals (New England Journal of Medicine, Nature, Science, Lancet Oncology, Journal of Clinical Oncology, and Cancer Discovery) was assembled. Studies were limited to solid tumor oncology trials. After all authors were identified, direct and research funding was tabulated from CMS Open Payments for the year prior (Ypre) and the first 3 years following publication (Y1, Y2, Y3) in the high-impact journal. Summary statistics were tabulated and repeated-measures linear mixed-effects regression models were fit to examine changes after publication. A total of 102 publications with a total of 620 authors were identified. No FCOI was declared by 11, 12, 21, and 24 authors in Ypre, Y1, Y2, and Y3, respectively. In Ypre, Y1, Y2, and Y3: median FCOI for direct payments was $16,702 (range: $0-$3,180,356), $20,830 (range: $0-$3,180,356), $22,031 (range: $0-$920,746), and $21,356 (range: $0-$920,707), respectively; while median research funding was $559,202 (range: $0-$19,973,818), $505,031 (range $0-$19,920,452), $502,726 (range: $0-$15,729,776), and $497,342 (range: $0-$43,036,716), respectively. There were nonsignificant increases in total direct payments and total direct payments received from the sponsor (P>0.0125 for both) and statistically significant decreases in total associated research funding and total research funding from the research sponsor in Y1, Y2, and Y3 as compared to Ypre (P<0.0001 for both). After publication of an industry-sponsored solid tumor clinical trial in a high-impact journal, authors had statistically significant decreases in research funding FCOI in the first 3 years postpublication compared to the year prior.

  • PDF Download Icon
  • Research Article
  • Cite Count Icon 13
  • 10.1186/s13643-020-01318-5
Reporting of financial conflicts of interest in meta-analyses of drug trials published in high-impact medical journals: comparison of results from 2017 to 2018 and 2009
  • Apr 8, 2020
  • Systematic Reviews
  • Carla Benea + 6 more

BackgroundA previous study found that 2 of 29 (6.9%) meta-analyses published in high-impact journals in 2009 reported included drug trials’ funding sources, and none reported trial authors’ financial conflicts of interest (FCOIs) or industry employment. It is not known if reporting has improved since 2009. Our objectives were to (1) investigate the extent to which pharmaceutical industry funding and author-industry FCOIs and employment from included drug trials are reported in meta-analyses published in high-impact journals and (2) compare current reporting with results from 2009.MethodsWe searched PubMed (January 2017–October 2018) for systematic reviews with meta-analyses including ≥ 2 randomized controlled trials (RCTs) of patented drugs. We included 3 meta-analyses published January 2017–October 2018 from each of 4 high-impact general medicine journals, high-impact journals from 5 specialty areas, and the Cochrane Database of Systematic Reviews, as in the previous study.ResultsAmong 29 meta-analyses reviewed, 13 of 29 (44.8%) reported the funding source of included trials compared to 2 of 29 (6.9%) in 2009, a difference of 37.9% (95% confidence interval, 15.7 to 56.3%); this included 7 of 11 (63.6%) from general medicine journals, 3 of 15 (20.0%) from specialty medicine journals, and 3 of 3 (100%) Cochrane reviews. Only 2 of 29 meta-analyses (6.9%) reported trial author FCOIs, and none reported trial author-industry employment.Protocol PublicationA protocol was uploaded to the Open Science Framework prior to initiating the study. https://osf.io/8xt5p/LimitationsWe examined only a relatively small number of meta-analyses from selected high-impact journals and compared results to a similarly small sample from an earlier time period.ConclusionsReporting of drug trial sponsorship and author FCOIs in meta-analyses published in high-impact journals has increased since 2009 but is still suboptimal. Standards on reporting of trial funding described in the forthcoming revised PRISMA statement should be adapted and enforced by journals to improve reporting.

  • Research Article
  • 10.1056/wh200609070000004
Women and Academic Authorship: Advancement Tempered by Challenge
  • Jan 1, 2006
  • NEJM Journal Watch
  • Andrew M Kaunitz

In the U.S., women constitute 49% of medical students and 32% of full-time medical faculty. To quantify the representation of the sexes in academic medical authorship, the investigators assessed original articles published in 1970, 1980, 1990, 2000, and 2004 in six journals that cover four core specialties and that are generally regarded as prestigious: Annals of Internal Medicine, Annals of Surgery, Journal of …

  • Research Article
  • Cite Count Icon 74
  • 10.1016/j.jclinepi.2012.10.005
Randomized trials published in higher vs. lower impact journals differ in design, conduct, and analysis
  • Jan 22, 2013
  • Journal of Clinical Epidemiology
  • Malgorzata M Bala + 19 more

Randomized trials published in higher vs. lower impact journals differ in design, conduct, and analysis

  • Front Matter
  • Cite Count Icon 31
  • 10.1053/j.gastro.2013.02.013
Bibliometrics of the Top 100 Clinical Articles in Digestive Disease
  • Feb 21, 2013
  • Gastroenterology
  • Dustin Edward Loomes + 1 more

Bibliometrics of the Top 100 Clinical Articles in Digestive Disease

  • PDF Download Icon
  • Research Article
  • Cite Count Icon 9
  • 10.1186/s13613-021-00954-x
A multivariate model for successful publication of intensive care medicine randomized controlled trials in the highest impact factor journals: the SCOTI score
  • Nov 27, 2021
  • Annals of Intensive Care
  • Joris Pensier + 6 more

BackgroundCritical care randomized controlled trials (RCTs) are often published in high-impact journals, whether general journals [the New England Journal of Medicine (NEJM), The Lancet, the Journal of the American Medical Association (JAMA)] or critical care journals [Intensive Care Medicine (ICM), the American Journal of Respiratory and Critical Care Medicine (AJRCCM), Critical Care Medicine (CCM)]. As rejection occurs in up to 97% of cases, it might be appropriate to assess pre-submission probability of being published. The objective of this study was to develop and internally validate a simplified score predicting whether an ongoing trial stands a chance of being published in high-impact general journals.MethodsA cohort of critical care RCTs published between 1999 and 2018 in the three highest impact medical journals (NEJM, The Lancet, JAMA) or the three highest impact critical care journals (ICM, AJRCCM, CCM) was split into two samples (derivation cohort, validation cohort) to develop and internally validate the simplified score. Primary outcome was journal of publication assessed as high-impact general journal (NEJM, The Lancet, JAMA) or critical care journal (ICM, AJRCCM, CCM).ResultsA total of 968 critical care RCTs were included in the predictive cohort and split into a derivation cohort (n = 510) and a validation cohort (n = 458). In the derivation cohort, the sample size (P value < 0.001), the number of centers involved (P value = 0.01), mortality as primary outcome (P value = 0.002) or a composite item including mortality as primary outcome (P value = 0.004), and topic [ventilation (P value < 0.001) or miscellaneous (P value < 0.001)] were independent factors predictive of publication in high-impact general journals, compared to high-impact critical care journals. The SCOTI score (Sample size, Centers, Outcome, Topic, and International score) was developed with an area under the ROC curve of 0.84 (95% Confidence Interval, 0.80–0.88) in validation by split sample.ConclusionsThe SCOTI score, developed and validated by split sample, accurately predicts the chances of a critical care RCT being published in high-impact general journals, compared to high-impact critical care journals.

  • Research Article
  • Cite Count Icon 395
  • 10.1136/bmj.d6128
Honorary and ghost authorship in high impact biomedical journals: a cross sectional survey
  • Oct 25, 2011
  • BMJ
  • Joseph S Wislar + 3 more

Objectives To assess the prevalence of honorary and ghost authors in six leading general medical journals in 2008 and compare this with the prevalence reported by authors of articles published...

  • Front Matter
  • Cite Count Icon 16
  • 10.1097/aln.0000000000003830
Authorship and Publication Matters: Credit and Credibility.
  • May 27, 2021
  • Anesthesiology
  • Evan D Kharasch + 11 more

Authorship and Publication Matters: Credit and Credibility.

  • Front Matter
  • Cite Count Icon 1
  • 10.1053/j.jvca.2019.01.035
From the Brontës to J.K. Rowling and Beyond – Have We Hit a Wall? The Status of Women Authors
  • Jan 16, 2019
  • Journal of Cardiothoracic and Vascular Anesthesia
  • Mariya Geube + 1 more

From the Brontës to J.K. Rowling and Beyond – Have We Hit a Wall? The Status of Women Authors

  • Research Article
  • Cite Count Icon 86
  • 10.1136/bmjopen-2017-020089
Gender disparities in high-quality dermatology research: a descriptive bibliometric study on scientific authorships
  • Apr 1, 2018
  • BMJ Open
  • Michael H K Bendels + 5 more

ObjectiveThe present study aims to elucidate the state of gender equality in high-quality dermatological research by analysing the representation of female authorships from January 2008 to May 2017.DesignRetrospective, descriptive study.Setting113...

  • Research Article
  • Cite Count Icon 10
  • 10.1016/j.euf.2022.04.009
A Seat at the Table: The Correlation Between Female Authorship and Urology Journal Editorial Board Membership
  • May 6, 2022
  • European Urology Focus
  • Megan Prunty + 9 more

A Seat at the Table: The Correlation Between Female Authorship and Urology Journal Editorial Board Membership

Save Icon
Up Arrow
Open/Close
  • Ask R Discovery Star icon
  • Chat PDF Star icon

AI summaries and top papers from 250M+ research sources.

Search IconWhat is the difference between bacteria and viruses?
Open In New Tab Icon
Search IconWhat is the function of the immune system?
Open In New Tab Icon
Search IconCan diabetes be passed down from one generation to the next?
Open In New Tab Icon