Seeing isn't always believing: the effects of misinformation modality and debunking on message credibility and memory recall

All you need is a (heuristic) cue?: An Empirical Investigation of the Use of Social Media Cues and Features and Underlying Mechanisms for Credibility Judgments of News and Political Communication

News organizations, journalists, and scholars are searching for ways to increase the perceived message credibility of the stories they produce. With the aim to evaluate the potentials of data-based storytelling formats, we build on message credibility research and investigate how features of message content such as the use of different presentation formats of statistical information and data visualizations affect news users’ message credibility judgements as well as their reading experience. We conducted two online experiments to detect the influences of percentages (e.g., 33%) and verbalized ratios (e.g., one out of three) as well as static and interactive data visualizations. Results of both studies indicate that the format of statistical information and the interactivity of data visualizations do not affect message credibility and reading experience. However, the perceived interactivity of the graphic has an impact on the reading experience. One conclusion is that the extra work involved in creating interactive graphics does not seem to pay off directly in terms of message credibility and improving the reading experience.

The current study tested the potential effect of regional-accented speech on perceived credibility. Canarian and Madrid listeners were presented with a series of audio recordings in which speakers read out loud news items with either a Canarian or a Madrid accent, and they were tasked to rate the credibility of each news item. The within-subject manipulation of accent demonstrated a small but significant effect on credibility judgment, which was not moderated by listener’s origin. Specifically, in line with socio-linguistic stereotyping, news items presented in a Canarian accent were judged as less credible on average than news items presented in a Madrid accent. These findings are discussed both within the perspective of cognitive-linguistic theory, and within a sociological perspective.

Auditory, visual and auditory-visual memory and sequencing performance in typically developing children

It has been demonstrated that the spatial representation of time is psychologically real, and the experience responsible for the formation of the "Time is Space" conceptual metaphor is associated with reading/writing directionality. Perceptual symbol theory assures that the spatial mapping of time derived from reading/writing is perceptual. That is to say, the temporal-spatial metaphor is grounded on modality- specific systems. Previous studies have shown that facilitation for the association of past with left and future with right was found only in the visual and motor modality, while none in the auditory modality. The difference of time metaphor between the visual and motor modality remains largely unexplored. In this study, three experiments were conducted to compare the spatial mapping of time between the visual and motor modality. In each experiment, a spatial Stroop task was carried out. Time series, which were displayed on the middle of the screen, were used as clues. Arrows, as targets, were displayed on the left, right, top or bottom of the screen, pointing to four directions. Arrow location and pointing direction were completely orthogonal to temporal reference. The participants were asked to indicate the direction to which the arrow pointed. Experiment 1 used a square as clues to explore whether the horizontal and vertical facilitation effects were modality-specific. The results proved that the horizontal and vertical facilitation effects existed in the visual and motor modality. Experiment 2 explored the difference of horizontal time metaphor between the visual and motor modality by Mandarin Speakers. A 2×2×2 repeated measure design was adopted with independent variables of temporal reference(before/after), target location(left /right) and response side(left / right).The results showed horizontal metaphoric representation of time was observed both at the visual and motor level of Mandarin Speakers. Processing time series affected the orienting of visual attention and the activation of motor responses. Participants were faster both to identify targets and to respond to the left when the cue was a word in a forward position of time series, and the opposite was true when the cue was a word in a afterward position of time series. Experiment 3 adopted the same task as Experiment 2, except changing target location and response side into top/ bottom. Experiment 3 observed the same facilitation effect in the visual modality as Experiment 2. But facilitation effect in the motor modality was not complete. Participants were faster to respond to the top only when the cue was a word in a forward position of time series. Processing words in a forward position of time series could not accelerate the response to the bottom. Experiment 4 increased the number of target location and response side. A 2×4×4 repeated measure design was adopted with independent variables of temporal reference(before/after), target location(left /right/top/bottom) and response side(left /right/top/bottom). Results showed facilitation effect of vertical direction in the motor modality which was not complete in Experiment 3 totally disappeared. It indicated that vertical metaphoric representation of time in the visual modality was stronger than the one in the motor modality. The explanation could be that Mandarin speakers' former top-down reading/writing directionality has been changed into "from left to right". The vertical sensori-motor experience from the visual and motor modality were unequal, which led to the incomplete time metaphor in the motor modality. In a word, Spatial information activated when processing time series was perceptual, which was different from the one activated by the spatial clue. In different modalities, time metaphor was different.

Following the work of Nairne and McNabb (1985), some previously unpublished data on recency and suffix effects in the visual and tactual modalities are presented. These data and the further analysis of some previously published data clarify the complicated picture of research in this area. The implications of the presence of a new “modality effect, ” involving the tactual and visual modalities, are discussed for the various theories proposed to explain recency, suffix effects, and modality effects.

Lateral Differences in Face Processing: Task and Modality Effects

We propose word-region alignment-guided multimodal neural machine translation (MNMT), a novel model for MNMT that links the semantic correlation between textual and visual modalities using word-region alignment (WRA). Existing studies on MNMT have mainly focused on the effect of integrating visual and textual modalities. However, they do not leverage the semantic relevance between the two modalities. We advance the semantic correlation between textual and visual modalities in MNMT by incorporating WRA as a bridge. This proposal has been implemented on two mainstream architectures of neural machine translation (NMT): the recurrent neural network (RNN) and the transformer. Experiments on two public benchmarks, English–German and English–French translation tasks using the Multi30k dataset and English–Japanese translation tasks using the Flickr30kEnt-JP dataset prove that our model has a significant improvement with respect to the competitive baselines across different evaluation metrics and outperforms most of the existing MNMT models. For example, 1.0 BLEU scores are improved for the English–German task and 1.1 BLEU scores are improved for the English–French task on the Multi30k test2016 set; and 0.7 BLEU scores are improved for the English–Japanese task on the Flickr30kEnt-JP test set. Further analysis demonstrates that our model can achieve better translation performance by integrating WRA, leading to better visual information use.

Ten experimental conditions were used to study the effects of auditory and visual (printed words, uncolored and colored pictures) modalities and their various combinations with college students. A recall paradigm was employed in which subjects responded in a written test. Analysis of data showed the auditory modality was superior to visual (pictures) ones but was not significantly different from visual (printed words) modality. In visual modalities, printed words were superior to colored pictures. Generally, conditions with multiple modes of representation of stimuli were significantly higher than for conditions with single modes. Multiple modalities, consisting of two or three modes, did not differ significantly from each other. It was concluded that any two modalities of the stimuli presented simultaneously were just as effective as three in recall of stimulus words.

Sharing breathing signals has the capacity to provide insights into hidden experiences and enhance interpersonal communication. However, it remains unclear how the modality of breath signals (visual, haptic) is socially interpreted during collaborative tasks. In this mixed-methods study, we design and evaluate BreatheWithMe, a prototype for real-time sharing and receiving of breathing signals through visual, vibrotactile, or visual-vibrotactile modalities. In a within-subjects study (15 pairs), we investigated the effects of modality on breathing synchrony, social presence, and overall user experience. Key findings showed: (a) there were no significant effects of visualization modality on breathing synchrony, only on deliberate music-driven synchronization; (b) visual modality was preferred over vibrotactile feedback, despite no differences across social presence dimensions; (c) BreatheWithMe was perceived to be an insightful window into others, however included data exposure and social acceptability concerns. We contribute insights into the design of multi-modal real-time breathing visualization systems for colocated, collaborative tasks.

This study aimed to develop and psychometrically evaluate an instrument to measure three key constructs in religious crisis communication during the COVID-19 pandemic: communication style, message credibility, and public trust in Islamic religious leaders (ulama). Using a quantitative approach, data were collected through a closed-ended questionnaire from a purposive sample of 180 respondents. The instrument was designed to capture three dimensions of communication style (authoritative–directive, empathetic–collaborative, analytical–integrative), three dimensions of message credibility (expertise, trustworthiness, goodwill), and overall public trust. Item validity was examined using item–total correlations, and all items exceeded the minimum r-table threshold, indicating that each item adequately represented its intended construct. Reliability was assessed using Cronbach’s Alpha for each variable, and the coefficients obtained fell within the acceptable range for social science research, demonstrating satisfactory internal consistency. As such, the instrument can be considered both valid and reliable for use in subsequent empirical studies. The primary contribution of this research is methodological: it provides a preliminary, psychometrically tested measurement tool for examining faith-based crisis communication in Muslim contexts. Future research is encouraged to employ this instrument to test structural relationships among communication style, message credibility, public trust, and related behavioral outcomes.

Various studies have demonstrated an advantage of auditory over visual text modality when learning with texts and pictures. To explain this modality effect, two complementary assumptions are proposed by cognitive theories of multimedia learning: first, the visuospatial load hypothesis, which explains the modality effect in terms of visuospatial working memory overload in the visual text condition; and second, the temporal contiguity assumption, according to which the modality effect occurs because solely auditory texts and pictures can be attended to simultaneously. The latter explanation applies only to simultaneous presentation, the former to both simultaneous and sequential presentation. This paper introduces a third explanation, according to which parts of the modality effect are due to early, sensory processes. This account predicts that-for texts longer than one sentence-the modality effect with sequential presentation is restricted to the information presented most recently. Two multimedia experiments tested the influence of text modality across three different conditions: simultaneous presentation of texts and pictures versus sequential presentation versus presentation of text only. Text comprehension and picture recognition served as dependent variables. An advantage for auditory texts was restricted to the most recent text information and occurred under all presentation conditions. With picture recognition, the modality effect was restricted to the simultaneous condition. These findings clearly support the idea that the modality effect can be attributed to early processes in perception and sensory memory rather than to a working memory bottleneck.

Social media platform's lack of control over its content made way to the fundamental problem of misinformation. As users struggle with determining the truth, social media platforms should strive to empower users to make more accurate credibility judgements. A good starting point is a more accurate perception of the credibility of the message's source. Two pre-registered online experiments (N = 525;N = 590) were conducted to investigate how verified source information affects perceptions of Tweets (study 1) and generic social media posts (study 2). In both studies, participants reviewed posts by an unknown author and rated source and message credibility, as well as likelihood of sharing. Posts varied by the information provided about the account holder: (1) none, (2) the popular method of verified source identity, or (3) verified credential of the account holder (e.g., employer, role), a novel approach. The credential was either relevant to the content of the post or not. Study 1 presented the credential as a badge, whereas study 2 included the credential as both a badge and a signature. During an initial intuitive response, the effects of these cues were generally unpredictable. Yet, after explanation how to interpret the different source cues, two prevalent reasoning errors surfaced. First, participants conflated source authenticity and message credibility. Second, messages from sources with a verified credential were perceived as more credible, regardless of whether this credential was context relevant (i.e., virtual lab coat effect). These reasoning errors are particularly concerning in the context of misinformation. In sum, credential verification as tested in this paper seems ineffective in empowering users to make more accurate credibility judgements. Yet, future research could investigate alternative implementations of this promising technology.

Modality effects in implicit artificial grammar learning: An EEG study

While several studies suggest that the nature and properties of the input have significant effects on statistical learning, they have rarely been investigated systematically. In order to understand how input characteristics and their interactions impact statistical learning, we explored the effects of modality (auditory vs. visual), presentation type (serial vs. simultaneous), domain (linguistic vs. non-linguistic), and training type (random, starting small, starting big) on artificial grammar learning in young adults (N = 360). With serial presentation of stimuli, learning was more effective in the auditory than in the visual modality. However, with simultaneous presentation of visual and serial presentation of auditory stimuli, the modality effect was not present. We found a significant domain effect as well: a linguistic advantage over nonlinguistic material, which was driven by the domain effect in the auditory modality. Overall, the auditory linguistic condition had an advantage over other modality-domain types. Training types did not have any overall effect on learning; starting big enhanced performance only in the case of serial visual presentation. These results show that input characteristics such as modality, presentation type, domain and training type influence statistical learning, and suggest that their effects are also dependent on the specific stimuli and structure to be learned.