Credibility Judgments Research Articles

CREDBANK: A Large-Scale Social Media Corpus With Associated Credibility Annotations

Social media has quickly risen to prominence as a news source, yet lingering doubts remain about its ability to spread rumor and misinformation. Systematically studying this phenomenon, however, has been difficult due to the need to collect large-scale, unbiased data along with in-situ judgements of its accuracy. In this paper we present CREDBANK, a corpus designed to bridge this gap by systematically combining machine and human computation. Specifically, CREDBANK is a corpus of tweets, topics, events and associated human credibility judgements. It is based on the real-time tracking of more than 1 billion streaming tweets over a period of more than three months, computational summarizations of those tweets, and intelligent routings of the tweet streams to human annotators — within a few hours of those events unfolding on Twitter. In total CREDBANK comprises more than 60 million tweets grouped into 1049 real-world events, each annotated by 30 human annotators. As an example, with CREDBANK one can quickly calculate that roughly 24% of the events in the global tweet stream are not perceived as credible. We have made CREDBANK publicly available, and hope it will enable new research questions related to online information credibility in fields such as social science, data mining and health.

The effect of boilerplate language on nonprofessional investors’ judgments

Most companies disclose risk factors using vague, boilerplate language. Regulators are concerned that this vagueness reduces the decision-usefulness of the information; hence, they are encouraging companies to be more specific rather than generic. However, little is known about the impact of specificity on investment judgments. The results of this experimental study suggest that regulators’ concern may be justified. Non-professional investors who read a generic disclosure react less strongly immediately after reading it than those who read a more specific disclosure when prior information about the disclosed risk factor is available in their memory immediately before reading the risk disclosure. In addition, on realisation of the risk, they are more surprised than their counterparts who read a more specific disclosure, and lower their credibility judgments accordingly. These investors correct their judgments after the risk realisation to a greater extent than those who have read a more specific disclosure. The study has implications for regulators, managers, non-professional investors and researchers.

Perceived suspect credibility: a brief report on the association between suspect and observer gender

ABSTRACT The current study examined the influence of gender on perceived credibility in an international drug trafficking setting. Two hundred law students (approximately half female) assessed a suspect’s credibility in a drug trafficking investigation. The participants read a transcript provided by either a male or a female suspect and rated a suspect as truthful/untruthful and their confidence level in this rating. The combined score represents credibility judgement. ANCOVA indicated a significant main effect of participant gender and a significant interaction between participant gender and suspect gender. Men considered the suspect as more credible than women, but this difference manifested only in the case of a male suspect. In the case of a female suspect, there was no significant gender difference between the participants. These findings highlight the dynamic interplay of the gender of both participants and suspects in the judgement bias field and are discussed in the context of defensive attribution theory. The applied implications address investigations, interrogations and other processes that surround them.

Factors influencing the perceived credibility of children alleging physical abuse

Adults’ assessments of the credibility of children’s reports are affected by factors including the frequency of abuse, reporting delays and the child’s age. The present study examined whether similar factors affect the perceived credibility of children reporting physical abuse, which is more common than sexual abuse. Two hundred and eight mock jurors read a simulated transcript of a child reporting physical abuse to police and made credibility ratings. Within each transcript, abuse frequency (once, repeated), reporting timing (recent, delayed), police question type (open, closed) and child age (6 or 10 years) were manipulated. The child was considered more credible when the abuse was only experienced once and reported shortly after it occurred, and when prompted with open questions. The child’s age did not affect credibility judgments. Current findings support recommendations to prioritise open questions with children and provide evidence for extension of the benefits of open questions to children’s credibility.

Sourcing on the internet: Examining the relations among different phases of online inquiry

This study examined students’ engagement in sourcing throughout online inquiry, that is, when they specified the information need, formulated search queries, evaluated online texts, and composed a written product. Participants were 167 upper secondary school students. Students completed an online inquiry task in a restricted online environment that utilized authentic online texts. Students’ prior topic knowledge and reading fluency was measured and controlled for in the analysis. The results showed that students engaged in sourcing even in the earliest phases of online inquiry. A sequential regression analysis indicated that the more frequently students engaged in sourcing in specifying the information need and in search querying, the more frequently they engaged in sourcing when judging the credibility of online texts. Further, the more frequently students engaged in sourcing in their credibility judgments, the more frequently sourcing also was observed in their written products. The results suggest that students would benefit from instruction emphasizing that sourcing is a continuous process throughout online inquiry.

The Relationships between Web-Based Information Credibility Judgment, Critical Thinking, and Learning Styles of Iranian EFL University Students

The advent of Web 2 tools and their features have allowed Internet users not only to seek information but also to generate and edit on-line information. However, excessive reliance of students, particularly EFL students, on the Internet has raised the issue of information credibility or quality of the information found online. This study sought to investigate the relationship between Iranian EFL university students’ credibility judgment and their learning styles and critical thinking. It also intended to examine the extent to which the learners employ credibility judgment strategies to verify web-based information. To this end, 212 Iranian EFL students, 165 female(s) and 47 male(s) (or add) students, received three questionnaires: Honey’s (2004) Critical Thinking questionnaire, Kolb's (1984) Learning Style inventory, and the adapted version of Credibility Judgment questionnaire originally developed by Metzger, Flanagin, and Zwarun (2003). The results of descriptive as well as bivariate and multivariate correlation analysis revealed that, first, Iranian EFL students employ credibility judgment strategies at a moderate level, and second, critical thinking is a better predictor for credibility judgment behavior of EFL students than learning style. Moreover, three learning styles were significantly related to the currency sub-scale of credibility judgment. The findings can assist curriculum planners, policymakers, and instructors to develop plans for empowering EFL students with the skills they need to evaluate on-line information for accuracy, objectivity, authority, and currency.

Patient Perceptions of Physician Burden in the Treatment of Chronic Pain

While patient perceptions of burden to caregivers is of recognized clinical significance among people with chronic pain, perceived burden to treating physicians has not been studied. This study examined how people with chronic pain perceived levels of medical evidence (low vs high) and pain severity (4,6,8/10) to influence physician burden and how burden then mediated expected clinical judgments. 476 people with chronic pain read vignettes describing a hypothetical patient with varying levels of medical evidence and pain severity from the perspective of a treating physician, rated the burden that patient care would pose, and made a range of clinical judgments. The effect of pain severity on clinical judgments was expected to interact with medical evidence and be conditionally mediated by burden. Although no associations with burden were found for the pain severity x medical evidence interaction or for pain severity alone, low levels of supporting medical evidence yielded higher burden ratings. Burden significantly mediated medical evidence effects on judgments of symptom credibility, clinical improvement, and psychosocial dysfunction. Results indicate that perceived physician burden negatively influenced judgments of patients with chronic pain, beyond the direct effects of medical evidence. Implications are discussed for clinical practice, as well as future research. Perspective: People with chronic pain expect physicians to view the care of patients without supporting medical evidence as burdensome. Higher burden is associated with less symptom credibility, more psychosocial dysfunction, and less treatment benefit. Perceived physician burden appears to impact how patients approach treatment, with potentially adverse implications for clinical practice.

The Impact of Social Endorsement Cues and Manipulability Concerns on Perceptions of News Credibility.

Social endorsement cues (SEC) offer information about how online users have engaged and evaluated online content. Some view that SEC thus can serve as useful heuristics when users evaluate the credibility of news content on social media. At the same time, SEC can be manipulated by a variety of commercial and political actors on social media. This study examines whether SEC influence individuals' credibility judgments of political news on social media, and how the salience of concerns that SEC can be manipulated by others can undermine the perceived credibility. Using an experiment, we found that SEC had a negative influence on news credibility, regardless of whether or not SEC manipulability concerns were primed. An independent effect of SEC manipulability concerns was also found, such that priming thoughts about the manipulability of SEC led participants to rate the news post as less credible, regardless of whether that post included SEC. These results suggest a spillover effect whereby concerns over the manipulation of SEC can create doubt about the authenticity of other cues from the news (e.g., source and message), and lead to perceptions that news shared on social media can be manipulated more generally.

The Goldilocks zone: young adults’ credibility perceptions of online news articles based on visual appearance

ABSTRACT Credibility judgments of online news are affected greatly by perceived expertise and trustworthiness, but users encounter an article’s visual appearance before its content, and yet visual appearance has not been studied in isolation. We conduct two studies of news article visual appearance. The first was with 31 undergraduates who rated the credibility of synthetic newslike articles containing only “lorem ipsum” text, indistinct videos and images, non-functional hyperlinks, and various fonts. The second study was with 30 different university students who rated the credibility of news articles from popular web outlets, half credible and half not. The articles were presented at 5600 words per minute, or 20 times faster than typical reading speeds, enabling only judgments of appearance, not substance. Findings show that credibility is affected by article length, image count and density, and font face and size. These factors interact to yield differential effects on perceived credibility. Articles that struck a balance among factors were most credible, giving rise to the notion of a “Goldilocks zone”, where credibility is highest. Interviews from both studies also revealed that perceived credibility was highest for articles that struck a balance among factors. This work has implications for visual information design, especially for online news.

Credibility and Enjoyment through Data? Effects of Statistical Information and Data Visualizations on Message Credibility and Reading Experience

ABSTRACT News organizations, journalists, and scholars are searching for ways to increase the perceived message credibility of the stories they produce. With the aim to evaluate the potentials of data-based storytelling formats, we build on message credibility research and investigate how features of message content such as the use of different presentation formats of statistical information and data visualizations affect news users’ message credibility judgements as well as their reading experience. We conducted two online experiments to detect the influences of percentages (e.g., 33%) and verbalized ratios (e.g., one out of three) as well as static and interactive data visualizations. Results of both studies indicate that the format of statistical information and the interactivity of data visualizations do not affect message credibility and reading experience. However, the perceived interactivity of the graphic has an impact on the reading experience. One conclusion is that the extra work involved in creating interactive graphics does not seem to pay off directly in terms of message credibility and improving the reading experience.

Linking perceived political network homogeneity with political social media use via perceived social media news credibility

ABSTRACT In this hyperpartisan political landscape, scholars have emphasized the importance of examining how perceptions of political network homogeneity on social media affect political outcomes. However, little research has examined how perceived news credibility mediates the relationship between perceived political network homogeneity and political social media use. Social media platforms aggregate and display news items from many sources, enabling people to make generalized judgments about the overall credibility of news items. Research has suggested that perceived political network homogeneity can bias such collective news credibility judgments. In turn, such credibility perceptions can affect political social media use. Lagged and autoregressive analyses of panel survey data gathered in 2012 and 2016 consistently show that perceived news credibility mediates the positive relationship between perceived political network homogeneity and expressive but not informational political social media use. Perceived political network homogeneity positively predicted news credibility, which in turn increased expressive political social media use. Furthermore, I show that the indirect effects outlined above tend to be stronger among Democrats than Republicans.

How humans impair automated deception detection performance

BackgroundDeception detection is a prevalent problem for security practitioners. With a need for more large-scale approaches, automated methods using machine learning have gained traction. However, detection performance still implies considerable error rates. Findings from different domains suggest that hybrid human-machine integrations could offer a viable path in detection tasks. MethodWe collected a corpus of truthful and deceptive answers about participants' autobiographical intentions (n = 1640) and tested whether a combination of supervised machine learning and human judgment could improve deception detection accuracy. Human judges were presented with the outcome of the automated credibility judgment of truthful or deceptive statements. They could either fully overrule it (hybrid-overrule condition) or adjust it within a given boundary (hybrid-adjust condition). ResultsThe data suggest that in neither of the hybrid conditions did the human judgment add a meaningful contribution. Machine learning in isolation identified truth-tellers and liars with an overall accuracy of 69%. Human involvement through hybrid-overrule decisions brought the accuracy back to chance level. The hybrid-adjust condition did not improve deception detection performance. The decision-making strategies of humans suggest that the truth bias - the tendency to assume the other is telling the truth - could explain the detrimental effect. ConclusionsThe current study does not support the notion that humans can meaningfully add the deception detection performance of a machine learning system. All data are available at https://osf.io/45z7e/.

Feedback from health professionals in postgraduate medical education: Influence of interprofessional relationship, identity and power.

Capitalising on direct workplace observations of residents by interprofessional team members might be an effective strategy to promote formative feedback in postgraduate medical education. To better understand how interprofessional feedback is conceived, delivered, received and used, we explored both feedback provider and receiver perceptions of workplace feedback. We conducted 17 individual interviews with residents and eight focus groups with health professionals (HPs) (two nurses, two rehabilitation therapists, two pharmacists and two social workers), for a total of 61 participants. Using a constructivist grounded theory approach, data collection and analysis proceeded as an iterative process using constant comparison to identify and explore themes. Conceptualisations and content of feedback were dependent on whether the resident was perceived as a learner or a peer within the interprofessional relationship. Residents relied on interprofessional role understanding to determine how physician competencies align with HP roles. The perceived alignment was unique to each profession and influenced feedback credibility judgements. Residents prioritised feedback from physicians or within the Medical Expertise domain-a role that HPs felt was over-valued. Despite ideal opportunities for direct observation, operational enactment of feedback was influenced by power differentials between the professions. Our results illuminate HPs' conceptualisation of feedback for residents and the social constructs influencing how their feedback is disseminated. Professional identity and social categorisation added complexity to feedback acceptance and incorporation. To ensure that interprofessional feedback can achieve desired outcomes, education programmes should implement strategies to help mitigate intergroup bias and power imbalance.

How heuristic credibility cues affect credibility judgments and decisions.

We investigated how heuristic credibility cues affected credibility judgments and decisions. Participants saw advice in comments in a simulated online health forum. Each comment was accompanied by credibility cues, including author expertise and peer reputation ratings (by forum members) of comments and authors. In Experiment 1, participants' credibility judgments of comments and authors increased with expertise and increased with the number of reputation ratings for supportive ratings and decreased with number of ratings for disconfirmatory ratings. Also, results suggested that the diagnosticity (informativeness) of credibility cues influenced credibility judgments. Using the same credibility cues and task context, Experiment 2 found that when high-utility choices had low credibility, participants often chose alternatives with lower utility but higher credibility. They did this more often when less utility had to be sacrificed and when more credibility was gained. The influence of credibility and utility information on participants' choices was mediated by their explicit credibility judgments. These findings supported the predictions of a Bayesian belief-updating model and an elaboration of Prospect Theory (Budescu, Kuhn, Kramer, & Johnson, 2002). This research provides novel insights into how cues including valence and relevance influence credibility judgments and how utility and credibility trade off during decision making. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

1EMF Hearing Is Believing: A Qualitative Exploration of Trust and Credibility Judgements in Educational Podcasts

Podcasts have become increasingly popular platforms for knowledge synthesis and translation. Trainees now report spending more time with podcasts than any other educational resource, including textbooks and journals. Though almost two thirds of residents report podcast listening changes their clinical practice, there is uncertainty over the quality and influence of podcasts. Given the broad use of podcasts among emergency medicine (EM) trainees, there is a need to better understand the processes by which they sort, interpret, and judge information as they learn. What is not known is how EM residents make credibility judgements about podcast content, how their judgements compare to the judgements of attending physicians, and how those credibility decisions relate to other learning modalities. The objective was to explore the processes by which podcasts are weighed, valued, and judged relative to one another, and relative to other learning modalities. We performed a multi-center qualitative thematic analysis based on a constructivist grounded theory approach by conducting 11 semi-structured interviews with resident and attending physicians from three North American teaching institutions from January 2020 to June 2020. Narrative transcripts were coded line-by-line using constant comparative analysis to organize transcripts into focused codes, key conceptual categories, and then major themes. Three authors met regularly during the analysis to develop the coding schema, resolve discrepancies, and discuss themes. We identified four broad themes related to credibility judgements and educational podcasts: trust in source, congruence of content, triangulation of references, and application context. Participants had a baseline level of trust in a podcast resource based on popularity, recommendations from colleagues, format, Web site, and speaker credentials. When listening to podcast content, participants’ levels of scrutiny varied based on the type of material (core content vs. cutting-edge) and level of agreement of the content with their existing knowledge. When considering incongruent or cutting-edge information, participants triangulated the podcast content with their experience, understanding of physiology, content of other podcasts and online resources, reading the primary literature, and conversations with attending physicians. When applying information gleaned from podcasts, participants yielded to local practice contexts and, for residents, their attendings' judgments. When listening to educational podcasts, resident and attending physicians made a series of complex credibility judgements that weighed trust in the source, congruence of the content, triangulation of references, and the context of application.

Do We Trust the Crowd? Effects of Crowdsourcing on Perceived Credibility of Online Health Information

ABSTRACT Crowdsourcing websites such as Wikipedia have become go-to places for health information. To what extent do we trust such health content that is generated by other Internet users? Will it make a difference if such entries are curated by medical professionals? Does the affordance of crowdsourcing make users feel like they themselves could be contributors, and does that influence their credibility judgments? We explored these questions with a 2 (Crowdsourcing: absence vs. presence) × 2 (Professional source: absence vs. presence) × 2 (Message: sunscreen vs. milk) between-subjects experiment (N = 189). Two indirect paths for crowdsourcing effects were found. The crowd-as-source path suggests that crowdsourcing negatively affects content credibility through decreased source trustworthiness and information completeness. In contrast, the self-as-source path indicates that crowdsourcing elevates source trustworthiness via heightened interactivity and sense of control. Although the additional professional source raises perceived gatekeeping on the site, it does not have substantial influence on credibility judgments. Theoretical and practical implications of the findings are discussed.

Should Graphic Warning Labels Proposed for Cigarette Packages Sold in the United States Mention the Food and Drug Administration?

Under the US Family Smoking Prevention and Tobacco Control Act, the US Food and Drug Administration (FDA) has the authority to implement graphic warning labels (GWLs) on cigarette packages. Neither the original labels proposed by the FDA nor the revised labels include a source to indicate sponsorship of the warnings. This study tests the potential impact of adding a sponsor to the content of GWLs. We recruited adult smokers (N = 245) and middle-school youth (N = 242) from low-income areas in the Northeastern US. We randomly assigned participants to view one of three versions of the original FDA-proposed warning labels in a between-subjects experiment: no sponsor, "US Food and Drug Administration," or "American Cancer Society" sponsor. We tested the effect of varying sponsorship on source attribution and source credibility. Compared to unsponsored labels, FDA sponsorship increased source attributions that the FDA sponsored the labels among both middle-school, largely nonsmoking youth and adult smokers. However, sponsorship had no effect on source credibility among either population. We found no evidence that adding FDA as the source is likely to boost source credibility judgments, at least in the short term; though doing so would not appear to have adverse effects on credibility judgments. As such, our data are largely consistent with the Tobacco Control Act's provisions that allow, but do not require, FDA sponsorship on the labels. This study addresses the FDA's regulatory efforts by informing the possible design and content of future cigarette warning labels. Our results do not offer compelling evidence that adding the FDA name on GWLs will directly increase source credibility. Future work may test more explicit FDA source labeling and continue to examine the credibility of tobacco message content among high-priority populations.

‘There were some clues’: a qualitative study of heuristics used by parents of adolescents to make credibility judgements of online health news articles citing research

ObjectiveTo identify how parents judge the credibility of online health news stories with links to scientific research.DesignThis qualitative study interviewed parents who read online stories about e-cigarettes and human papillomavirus...

Source memory for advertisements: The role of advertising message credibility

Advertising is seen as an untrustworthy source because of the perceived self-interest of the advertisers in presenting product information in a biased or misleading way. Regulations require advertising messages in print and online media to be labeled as advertisements to allow recipients to take source information into account when judging the credibility of the messages. To date, little is known about how these source tags are remembered. Research within the source-monitoring framework suggests that source attributions are not only based on veridical source memory but are often reconstructed through schematic guessing. In two experiments, we examined how the credibility of advertising messages affects these source attribution processes. The source of the messages affected judgments of credibility at the time of encoding, but the source tags were forgotten after a short period of time. Retrospective source attributions in the absence of memory for the source tags were strongly influenced by the a priori credibility of the messages: Statements with a low a priori credibility were more likely to be (mis)attributed to advertising than statements with high a priori credibility. These findings suggest that the mere labeling of untrustworthy sources is of limited use because source information is quickly forgotten and memory-based source attributions are strongly biased by schematic influences.

Truthiness and law: Nonprobative photos bias perceived credibility in forensic contexts

SummaryNonprobative but related photos can increase the perceived truth value of statements relative to when no photo is presented (truthiness). In two experiments, we tested whether truthiness generalizes to credibility judgments in a forensic context. Participants read short vignettes in which a witness viewed an offence. The vignettes were presented with or without a nonprobative, but related photo. In both experiments, participants gave higher witness credibility ratings to photo‐present vignettes compared to photo‐absent vignettes. In Experiment 2, half the vignettes included additional nonprobative information in the form of text. We replicated the photo presence effect in Experiment 2, but the nonprobative text did not significantly alter witness credibility. The results suggest that nonprobative photos can increase the perceived credibility of witnesses in legal contexts.