Credibility Assessment Research Articles

Assessment of the reliability of sociological research on the Internet

The purpose of research. The main goal of assessing reliability is to quantitatively substantiate the measure of confidence in a particular process. The article discusses the methodological and practical aspects of assessing the reliability of sociological research conducted on the Internet. Emphasis is placed on analyzing the specific problems encountered when using online surveys and other digital data collection methods. Issues of sample representativeness and a number of other quantitative indicators that affect the reliability of the assessment of sociological research are discussed.Methods. The article is devoted to methodological and practical aspects of assessing the reliability of sociological research conducted on the Internet. Issues of sample representativeness, weighting coefficients, data analysis, testing for systematic errors, and calculation of Cronbach’s alpha coefficient are discussed.Results. The main steps for calculating the reliability of sociological research conducted on the Internet are presented. Key steps include proper sample selection and adjustment, use of weights, performing statistical analysis, and estimating confidence intervals. These methods help minimize errors and increase the reliability of the data obtained.Conclusion. The proposed methods for assessing the reliability of the Internet will help improve the quality of data and the reliability of the results, namely, Online surveys can be subject to various types of biases, including self-selection of participants, when only certain groups of people decide to participate in the survey. Credibility assessment will promote fair and transparent research, which strengthens the trust of respondents and the public in research projects.

Verifying the robustness of automatic credibility assessment

Verifying the robustness of automatic credibility assessment

The Testimony or the Eyewitness? Factors That Influence Eyewitness Credibility Assessment Conducted by (Future) Fact-Finders and Laypeople

ABSTRACT Purpose This study examined how the (in)consistency of eyewitness testimony and their relationship with the defendant affect assessments of eyewitness credibility and defendant culpability, along with decision confidence. Methods Judicial apprentices (N = 125) and students (N = 115) rated the guilt and credibility of testimony after reading a case summary. Results Eyewitness-defendant relationships influenced credibility judgements but not guilt assessments. Reminiscent details reduced credibility ratings and increased culpability judgements. Future judges exhibited greater confidence than laypeople. Conclusions The findings emphasize the need for ecologically valid research involving legal professionals to better understand evidence credibility and verdict decision-making.

Effect of Explanation Conceptualisations on Trust in AI-assisted Credibility Assessment

As misinformation increasingly proliferates on social media platforms, it has become crucial to explore how to best convey automated news credibility assessments to end-users, and foster trust in fact-checking AIs. In this paper, we investigate how model-agnostic, natural language explanations influence trust and reliance on a fact-checking AI. We construct explanations from four Conceptualisation Validations (CVs) - namely consensual, expert, internal (logical), and empirical - which are foundational units of evidence that humans utilise to validate and accept new information. Our results show that providing explanations significantly enhances trust in AI, even in a fact-checking context where influencing pre-existing beliefs is often challenging, with different CVs causing varying degrees of reliance. We find consensual explanations to be the least influential, with expert, internal, and empirical explanations exerting twice as much influence. However, we also find that users could not discern whether the AI directed them towards the truth, highlighting the dual nature of explanations to both guide and potentially mislead. Further, we uncover the presence of automation bias and aversion during collaborative fact-checking, indicating how users' previously established trust in AI can moderate their reliance on AI judgements. We also observe the manifestation of a 'boomerang'/backfire effect often seen in traditional corrections to misinformation, with individuals who perceive AI as biased or untrustworthy doubling down and reinforcing their existing (in)correct beliefs when challenged by the AI. We conclude by presenting nuanced insights into the dynamics of user behaviour during AI-based fact-checking, offering important lessons for social media platforms.

Clinical risk factors for pancreatic cancer: protocol for an umbrella review

IntroductionIdentifying cancer earlier can help save lives. An increasingly popular approach to diagnosing cancer earlier is in the development of risk prediction models to be applied to the electronic healthcare...

When Automated Assessment Meets Automated Content Generation: Examining Text Quality in the Era of GPTs

The use of machine learning (ML) models to assess and score textual data has become increasingly pervasive in an array of contexts including natural language processing, information retrieval, search and recommendation, and credibility assessment of online content. A significant disruption at the intersection of ML and text are text-generating large-language models (LLMs) such as generative pre-trained transformers (GPTs). We empirically assess the differences in how ML-based scoring models trained on human content assess the quality of content generated by humans versus GPTs. To do so, we propose an analysis framework that encompasses essay scoring ML-models, human and ML-generated essays, and a statistical model that parsimoniously considers the impact of type of respondent, prompt genre, and the ML model used for assessment model. A rich testbed is utilized that encompasses 18,460 human-generated and GPT-based essays. Results of our benchmark analysis reveal that LLMs and transformer pretrained language models (PLMs) more accurately score human essay quality as compared to CNN/RNN and feature-based ML methods. Interestingly, we find that LLMs and transformer PLMs tend to score GPT-generated text 10-20% higher on average, relative to human-authored documents. Conversely, traditional deep learning and feature-based ML models score human text considerably higher. Further analysis reveals that even though the LLMs and transformer PLMs are exclusively fine-tuned on human text, they more prominently attend to certain tokens appearing only in GPT-generated text, possibly (in part) due to familiarity/overlap in pre-training. Our framework and results have implications for text classification settings where automated scoring of text is likely to be disrupted by generative AI.

Beyond Text: Multimodal Credibility Assessment Approaches for Online User-Generated Content

User-generated content (UGC) is increasingly becoming prevalent on various digital platforms. The content generated on social media, review forums, and question–answer platforms impacts a larger audience and influences their political, social, and other cognitive abilities. Traditional credibility assessment mechanisms involve assessing the credibility of the source and the text. However, with the increase in how user content can be generated and shared (audio, video, and images), multimodal representation of UGC has become increasingly popular. This article reviews the credibility assessment of UGC in various domains, particularly identifying fake news, suspicious profiles, and fake reviews and testimonials, focusing on both textual content and the source of the content creator. Next, the concept of multimodal credibility assessment is presented, which also includes audio, video, and images in addition to text. After that, the article presents a systematic review and comprehensive analysis of work done in the credibility assessment of UGC considering multimodal features. Additionally, the article provides extensive details on the publicly available multimodal datasets for the credibility assessment of UGC. In the end, the research gaps, challenges, and future directions in assessing the credibility of multimodal UGC are presented.

A 28-time-point cropland area change dataset in Northeast China from 1000 to 2020

Abstract. Based on historical documents, population data, published results, remote sensing data products, statistical data, and survey data, this study reconstructed the cropland area and the spatial pattern changes at 28 time points from 1000 to 2020 in Northeast China. The period from 1000 to 1600 corresponds to historical provincial-level administrative districts, while the period from 1700 to 2020 corresponds to modern county-level administrative districts. The main findings are as follows: (1) the cropland in Northeast China exhibited phase changes of expansion–reduction–expansion over the past millennium. (2) The cropland area in Northeast China increased from 0.55×104 km2 in 1000 to 37.90×104 km2 in 2020, and the average cropland fraction increased from 0.37 % to 26.27 %; (3) from 1000 to 1200, the cropland area exhibited an increasing trend, which peaked in 1200. The scope of land reclamation was comparable to modern times, but the overall cropland fraction remained low. The cropland area significantly decreased between 1300 and 1600, with the main land reclamation area being reduced southward into Liaoning province. From 1700 to 1850, the cropland area increased slowly and the agricultural reclamation gradually expanded northward. After 1850, there was almost exponential growth, with the cropland area continuously expanding to the whole study area, and this growth trend persists until 2020; (4) the dataset of changes in the cropland of administrative districts in Northeast China, reconstructed based on multiple data sources and improved historical cropland reconstruction methods, significantly enhances time resolution and reliability. Additionally, the dataset shows relatively better credibility assessment results, which can provide a refined database for historical land use and land cover change (LUCC) dataset reconstruction, carbon emission estimation, climate data construction, etc. The dataset can be downloaded from https://doi.org/10.6084/m9.figshare.25450468.v2 (Jia et al., 2024).

DOMAIN: Explainable credibility assessment tools for empowering online readers coping with misinformation

Despite all the fact-checking initiatives on news and social media aimed at countering misinformation, they remain insufficient to promptly address the wide array of misleading information disseminated by both news and social media outlets. Rather than attempting to identify or filter misleading information, this work advocates new tools for assisting online readers in identifying misinformation among the massive online content pushed every day through multiple platforms. We introduce DOMAIN, an article assessment resource bundle comprising a multidimensional indicator to categorize articles into different types (hard news, soft news, opinion, satire, and conspiracy), a set of explanatory metrics to help users understand the results, a tool for verifying the reliability of the article’s source, and a text summary of the assessment. This work also studies how DOMAIN tools impact online readers, specifically focusing on i) understanding the extent to which computer-generated assessments influence human perceptions of credibility; ii) evaluating the effectiveness of automatic article categorization in human assessment of credibility; and iii) identifying the most relevant explanatory metrics for promoting informed and critical consumption of information.

Trust at the border: identifying risk and assessing credibility on reality television

AbstractEvery day, officers working at international airports investigate potential risks to state safety and security. But how do they decide who they can trust, and also ensure that the broader public trusts them to conduct this work? This article explores these questions through an examination of the reality television show Border Security: Australia's Front Line. Through critical discourse analysis of a collection of 108 televised airport encounters, we explore the aspects of communication, behaviour, and identity made salient in officers’ evaluations of passengers’ credibility and critically examine the assumptions underlying them. Further, we consider how power and role divisions are implicated in the construction of passenger and officer credibility, both within border encounters and in discourses about them. Our analysis makes a novel contribution to the literature on credibility assessments in intercultural communication, demonstrating how an institutional and social ‘culture of disbelief’ is constructed vis‐à‐vis certain groups through seemingly banal border work.

Epistemological Misunderstandings of the German Federal Court of Justice in Criminal Cases Regarding the Null Hypothesis: Verification in the Credibility Assessment

Epistemological Misunderstandings of the German Federal Court of Justice in Criminal Cases Regarding the Null Hypothesis: Verification in the Credibility Assessment Abstract: Background: The article deals with the decision of the German Federal Court of Justice (Bundesgerichtshof, BGH) in criminal matters regarding credibility assessment dated 30 July 1999 (1 StR 618/98, BGHSt 45, 164). Regarding criminal matters, the BGH formulated specific requirements for credibility assessments based on two published scientific expert reports. Method: We analyzed conflicting postulates of scientific theory in the expert reports and the reception of these principles in the BGH judgment by examining the original quotes. Results: Given the central importance of this BGH decision, we analyzed the original expert reports for their epistemological content. The BGH formulated the scientific approach of starting from the assumption that the statement is untrue - the so-called "null hypothesis". In doing so, it referred to Popper's deductivism, albeit without addressing the rules of hypothesis testing. Based on the second expert report, which argues for inductive.

Internet as a Source of Public Health Information on Acupuncture for Pain Relief: Credibility Assessment and Readability Analysis

Internet as a Source of Public Health Information on Acupuncture for Pain Relief: Credibility Assessment and Readability Analysis

Augmenting Multimodal Content Representation with Transformers for Misinformation Detection

Information sharing on social media has become a common practice for people around the world. Since it is difficult to check user-generated content on social media, huge amounts of rumors and misinformation are being spread with authentic information. On the one hand, most of the social platforms identify rumors through manual fact-checking, which is very inefficient. On the other hand, with an emerging form of misinformation that contains inconsistent image–text pairs, it would be beneficial if we could compare the meaning of multimodal content within the same post for detecting image–text inconsistency. In this paper, we propose a novel approach to misinformation detection by multimodal feature fusion with transformers and credibility assessment with self-attention-based Bi-RNN networks. Firstly, captions are derived from images using an image captioning module to obtain their semantic descriptions. These are compared with surrounding text by fine-tuning transformers for consistency check in semantics. Then, to further aggregate sentiment features into text representation, we fine-tune a separate transformer for text sentiment classification, where the output is concatenated to augment text embeddings. Finally, Multi-Cell Bi-GRUs with self-attention are used to train the credibility assessment model for misinformation detection. From the experimental results on tweets, the best performance with an accuracy of 0.904 and an F1-score of 0.921 can be obtained when applying feature fusion of augmented embeddings with sentiment classification results. This shows the potential of the innovative way of applying transformers in our proposed approach to misinformation detection. Further investigation is needed to validate the performance on various types of multimodal discrepancies.

Toward widespread use of virtual trials in medical imaging innovation and regulatory science.

The rapid advancement in the field of medical imaging presents a challenge in keeping up to date with the necessary objective evaluations and optimizations for safe and effective use in clinical settings. These evaluations are traditionally done using clinical imaging trials, which while effective, pose several limitations including high costs, ethical considerations for repetitive experiments, time constraints, and lack of ground truth. To tackle these issues, virtual trials (aka in silico trials) have emerged as a promising alternative, using computational models of human subjects and imaging devices, and observer models/analysis to carry out experiments. To facilitate the widespread use of virtual trials within the medical imaging research community, a major need is to establish a common consensus framework that all can use. Based on the ongoing efforts of an AAPM Task Group (TG387), this article provides a comprehensive overview of the requirements for establishing virtual imaging trial frameworks, paving the way toward their widespread use within the medical imaging research community. These requirements include credibility, reproducibility, and accessibility. Credibility assessment involves verification, validation, uncertainty quantification, and sensitivity analysis, ensuring the accuracy and realism of computational models. A proper credibility assessment requires a clear context of use and the questions that the study is intended to objectively answer. For reproducibility and accessibility, this article highlights the need for detailed documentation, user-friendly software packages, and standard input/output formats. Challenges in data and software sharing, including proprietary data and inconsistent file formats, are discussed. Recommended solutions to enhance accessibility include containerized environments and data-sharing hubs, along with following standards such as CDISC (Clinical Data Interchange Standards Consortium). By addressing challenges associated with credibility, reproducibility, and accessibility, virtual imaging trials can be positioned as a powerful and inclusive resource, advancing medical imaging innovation and regulatory science.

Intelligent and Secure Cloud–Edge Collaborative Industrial Information Encryption Strategy Based on Credibility Assessment

As industries develop and informatization accelerates, enterprise collaboration is increasing. However, current architectures face malicious attacks, data tampering, privacy issues, and security and efficiency problems in information exchange and enterprise credibility. Additionally, the complexity of cyber threats requires integrating intelligent security measures to proactively defend against sophisticated attacks. To address these challenges, this paper introduces an intelligent and secure cloud–edge collaborative industrial information encryption strategy based on credibility assessment. The proposed strategy incorporates adaptive encryption specifically designed for cloud–edge and edge–edge architectures and utilizes attribute encryption to control access to user-downloaded data, ensuring secure information exchange. A mechanism for assessing enterprise credibility over a defined period helps maintain a trusted collaborative environment, crucial for identifying and mitigating risks from potentially malicious or unreliable entities. Furthermore, integrating intelligent threat detection and response systems enhances overall security by continuously monitoring and analyzing network traffic for anomalies. Experimental analysis evaluates the security of communication paths and examines how enterprise integrity influences collaboration outcomes. Simulation results show that this approach enhances enterprise integrity, reduces losses caused by harmful actors, and promotes efficient collaboration without compromising security. This intelligent and secure strategy not only safeguards sensitive data but also ensures the resilience and trustworthiness of the collaborative network.

PBBM Considerations for Base Models, Model Validation, and Application Steps: Workshop Summary Report.

The proceedings from the 30th August 2023 (Day 2) of the workshop "Physiologically Based Biopharmaceutics Models (PBBM) Best Practices for Drug Product Quality: Regulatory and Industry Perspectives" are provided herein. Day 2 covered PBBM case studies from six regulatory authorities which provided considerations for model verification, validation, and application based on the context of use (COU) of the model. PBBM case studies to define critical material attribute (CMA) specification settings, such as active pharmaceutical ingredient (API) particle size distributions (PSDs) were shared. PBBM case studies to define critical quality attributes (CQAs) such as the dissolution specification setting or to define the bioequivalence safe space were also discussed. Examples of PBBM using the credibility assessment framework, COU and model risk assessment, as well as scientific learnings from PBBM case studies are provided. Breakout session discussions highlighted current trends and barriers to application of PBBMs including: (a) PBBM credibility assessment framework and level of validation, (b) use of disposition parameters in PBBM and points to consider when iv data are not available, (c) conducting virtual bioequivalence trials and dealing with variability, (d) model acceptance criteria, and (e) application of PBBMs for establishing safe space and failure edges.

Exploring the effects of location information on perceptions of news credibility and sharing intention

In recent years, the integration of location-based services into social media platforms has seen a significant surge, coinciding with the growing challenges posed by the proliferation of fake news online. However, the influence of location data on readers’ perceptions of online news credibility, particularly in relation to the reporters’ whereabouts, remains unclear. To investigate this relationship, we conducted a 3 (Topics: crime, science, health) × 2 (Location anchor: event-anchored or participant-anchored) × 4 (Proximity to location anchor - no, same, close-by or faraway location) mixed-method online study (N = 288) on Prolific. Our data collection involved presenting participants with news articles and assessing their credibility assessments and sharing intentions based on the proximity of those disseminating the news to both the subject matter of the news and the audience consuming it. Our findings reveal that the proximity of the reporter’s location to the readers’ location had a noticeable adverse impact on perceptions of news credibility and the likelihood of sharing it. Furthermore, we also identified a weak positive correlation between sharing intentions and trust in social media platforms. In addition, we observed that crime news were generally perceived as less credible compared to health and science news. Our research contributes significantly to a nuanced understanding of how location-based cues impact user behaviour when interacting with online news articles. Furthermore, it provides design insights for social media platforms aiming to enhance user trust and promote pro-social behaviours.

The Influence of “Body Language” on the Assessment of Witness Credibility in Australian Written Court Judgments

ABSTRACT This article examines how “body language” has been used in Australian court judgments when assessing witness credibility, and how it can influence legal proceedings. We first address the use (and misuse) of nonverbal communication. Then, we address the guidelines of the High Court of Australia regarding nonverbal behavior, also known as demeanor within the legal community. The methodology of the doctrinal analysis we conducted and the results are presented. Their implications considering research on nonverbal communication are discussed, and we conclude with a call for legal scholars and practitioners to further address the influence of “body language” in legal proceedings.

The impact of descriptor identicalness on investors' judgements of managers’ opportunistic estimation choices

We conduct two experiments to examine how descriptor identicalness influences investors' assessments of managers' opportunistic estimation choices. We manipulate the identicalness of descriptors (identical versus non-identical) and managers' opportunistic estimation choices (more versus less opportunistic). We find that investors are better able to integrate information when managers use identical descriptors in their disclosures than when managers use non-identical descriptors. We also find that investors who observe identical descriptors tend to make less favorable investment judgments when managers' estimation choices are more (versus less) opportunistic, whereas those presented with non-identical descriptors are less likely to discern between more and less opportunistic estimation choices. Lastly, our supplemental analyses indicate that descriptor identicalness facilitates information integration, which in turn moderates the impact of managers' estimation choices on investors’ credibility assessments and subsequent investment judgments. Our findings have important implications for regulators, managers, and investors, shedding light on how investors use and incorporate accounting disclosures into their investment decision-making processes.

The impact of socio-demographic factors on web credibility assessment

The aim of this study is to investigate the relationship between web credibility evaluation and socio-demographic variables. A sample of 380 students from Ferdowsi University of Mashhad, Iran, was selected using stratified sampling methods. Data was collected through a questionnaire encompassing socio-demographic information and Choi items. The collected data was analysed using Spearman and Kruskal–Wallis non-parametric tests, guided by the data's characteristics (type and distribution). The findings reveal a significant association between interest, experience and academic level in determining web credibility assessments. Conversely, expertise, income, gender and age were not found to be influential factors. This is a first attempt to determine the impact of socio-demographic variables on web credibility assessment. The findings suggest that individuals from diverse age, gender and income groups could benefit from acquiring knowledge pertaining to web credibility and engaging in workshops related to this domain. Additionally, those with lower levels of experience, education and interest may derive greater value from participating in web-credibility-evaluation workshops.