Verbal Modes Research Articles

Students' Meaning‐Making of Nature of Science: Interaction Between Visual, Verbal, and Written Modes of Representation

ABSTRACTStudents' understanding of nature of science (NOS) has been largely examined primarily in written or verbal modes. The visual, verbal, and written modes are essential for students' meaning‐making of NOS. However, research has sidelined the interaction among these three modes in understanding students' collaborative discourse of NOS. Informed by theories of multimodality and social semiotics, this paper investigates the interactions between the visual, verbal, and written modes as groups of students engaged in explicit‐reflective multimodal representation during NOS instruction. Utilizing a collective case study approach, we planned NOS instruction with teachers, and videotaped how each focal group of students in two grade seven classes in Hong Kong constructed multimodal representations of NOS. Multimodal discourse analysis revealed that the three modes fulfill various purposes during students' co‐construction of multimodal representations of NOS. The interaction between the three modes facilitates meaning‐making of NOS in four ways: (a) students' re‐semiotization of discursive scientific practices into their multimodal ensembles; (b) bridging students' writing of scientific reports to scientists' social certification and dissemination; (c) connecting students' decontextualized meaning‐making to contextualized meaning‐making of methods and methodological rules; and (d) facilitating students' embodied semiosis in social organizations and interactions of science. Focusing on four episodes of co‐constructing multimodal representations of NOS, we illustrate how students' meaning‐making of NOS is multimodal in nature and how various modes have their own affordances. We discuss future research directions on how multimodality can facilitate students' meaning‐making of NOS.

The challenge of dissecting the frog: cartoonists analyze their creative process

Abstract This article delves into two intrinsic tensions present in humor and cartoons: funniness-seriousness and repetition-novelty. The focus is on the perspectives of cartoonists regarding the semiotic and rhetorical resources they put into play to create humor and satire. In this way, the study echoes a key question in semiotics, and particularly in multimodal studies: how image and words interplay in meaning-making. With that aim, a questionnaire was designed and filled out by 100 cartoonists from 22 different countries. The responses were analyzed applying mixed methods (content analysis, non-parametric statistical techniques, and network analysis). Based on the findings, four distinct profiles of cartoonists were identified, ranging from those who prioritize multimodality (the largest group) to those who favor the verbal mode. Cartoonists who prioritize the visual mode and skeptics to humor analysis fall in between. The article also discusses the various rhetorical resources that cartoonists declare to use to create humor, with irony and sarcasm being the most frequent ones, while puns are the least common. The results are examined in relation to the cartoonists’ conceptions of humor, satire, and the cartoon genre, as well as the different dimensions they considered when discussing their humor style. The study contributes to gaining a deeper understanding of the ways cartoonists use image and writing to create humor and satire and sheds light on an under-researched area in this field.

Using multimodal resources to design EFL classroom lead-ins—A multimodal pedagogical stylistics perspective

The study explores the ways EFL teachers use multimodal resources to design classroom lead-ins on the basis of multimodal stylistic analyses of the lead-ins in six award-winning classroom demonstrations. Results show that verbal mode and image mode are mainly used in the lead-ins to establish cognitive framework for the new teaching topics, and body language, such as eye movements, facial expressions, and gestures, is particularly important in attracting students’ attention and motivate their participation. The foregrounded complementary reinforcement relationships within the multimodal ensembles in the lead-ins play significant roles in designing students’ learning experiences. Discussions on the multimodal stylistic features and their teaching effects also indicate that topic relevance, closeness to life, mode conciseness, proper use of body language, complementary reinforcement within mode ensembles, and dynamicity of mode choices are general guidelines for the design of effective multimodal classroom lead-ins. The study not only verifies the effectiveness of the multimodal pedagogical stylistic theory but also provides feasible implications for the design and implementation of multimodal classroom lead-ins in EFL teaching.

A Study of Multimodal Communicative Acts in Selected Nollywood Film Advertisement Posters

Movie posters employ multiple modes of communication in their display to communicate with the audience. Identifying the modes and the meanings realised through them as projected in most scholarly works on film posters are not sufficient in interpreting the intentional goals of the image designers or advertisers. It is against this backdrop that this article articulates the pragmatic functions of Nollywood film posters’ advertising language as a form of language in action and intention in the situational context of users. Eight (8) Nollywood film advertisement posters which were purposively selected were analysed using insights from van Leeuwen’s (2005) theory of multimodal communicative acts and Bach and Harnish’s (1979) speech acts theory. The findings from the analysis reveal that communicative acts of informing, stating, expressing, describing and asserting are frequently demonstrated via visual and verbal modes in the advertisement posters. The less employed multimodal communicative acts are questioning and ordering. These multimodal communicative acts utilised demonstrate the persuasive illocutionary and communicative intentions of Nollywood film advertisers which is primarily aimed at enhancing viewing patronage. The article concludes that the adaptability of Nollywood advertisement posters to the context and situation of users is considered pertinent to the appropriate interpretation of meanings realised from the complex modes of communication employed in them.

Recharacterising Confucius in Multimodal Translation: From Analects to Comics

ABSTRACT This study aims to explore how Confucius is recharacterised in the multimodal translation of The Analects from verbal (analects) to verbal-visual (comics) texts. The focus is on the interpersonal/interactive meaning in Systemic Functional Linguistics and visual social semiotics. Parameters of MOOD and MODALITY are used in discussing the verbal mode in both source and target texts, and the system of CONTACT, SOCIAL DISTANCE, INVOLVEMENT, POWER, AFFECT, PROXIMITY and MASS/AMOUNT are employed in describing features of the visual mode in comics. The quantitative results show that Confucius has been recharacterised from an authoritative teacher in the source text into different images in four comic adaptions, i.e. a detached truth transmitter in Mori [2002. Dongfang Zhidian Shengshu: Manhua Lunyu. Beijing: Jiuzhou Press], an affable wiser in Tsai [2005. Confucius Speaks: The Message of the Benevolent. Beijing: China Modern Publishing House Ltd.], a superior mentor in Zhang [2007. Manhua LunYu. Hangzhou: Zhejiang People’s Publishing House.], and a popularised teacher in Chen [2021. Banxiaoshi Manhua Lunyu. Shanghai: Wenhui Press]. It is argued that the different images of Confucius reconstrued in four translations can be better understood in terms of the context of translation, in which translators’ identity, translation purpose, target reader and historically accepted image of Confucius are finally explored.

The place of feelings

Abstract This article describes the semiotic resources deployed during the creation, in 2021, of the Alphabet Workers Union (AWU): the trade union that represents the workforce of the tech giant Alphabet Inc. The focus is on how the union shapes and develops its identity as one of “empathetic workers who believe in social justice” (Alphabet Workers Union 2021). The analysis of the homepage of AWU’s website through the prism of the textual metafunction in the visual mode (Kress and van Leeuwen 2001, 2006, 2020) together with the analysis of the interpersonal metafunction in the verbal mode according to the “appraisal” paradigm (Martin and White 2005), shows how the union assigns a central position to positive feelings of togetherness.

DECODING THE SEMANTIC VOLUME OF NOUNS WITH THE VERBAL MODALITY MÜSSEN: THE SYNERGISTIC DIMENSION OF LEAST EFFORT

The presented work is one of the stages of noun research within a certain verbal modality. The focus of the article lies in the study of the generating noun content in relation to the verbal modality müssen in German from the perspective of linguosynergetic scientific paradigm, which considers language as a complex dynamic system capable of self-organization and self-regulation. In this study methodologically important is the position that the synergetic law of making the least effort and conservation of speech energy is the driving factor of language development. In a synergetic sense, the author of the article draws parallels between the dictionary as a representative of the linguistic generalization of the structured amount of knowledge of the learned extraverbal reality and the mental lexicon, which is not an arbitrary accumulation of contributions, but constitutes a structured hierarchical system of such contributions. If lexical units in the synergetic sense and their meanings at the language level contain coded structured information, then in the same aspect the paper considers the implementation of the word in one of its meanings at the speech level as information decoding. The linguosynergetic format of the research proposed in this article assumes that language as a self-regulating system under the influence of external energy and information has developed a mechanism of different representations of nouns in syntactic constructions with different verbal modality in German-language fiction and press. The results obtained on the basis of the synergetic-quantitative approach are extrapolated to linguosynergetic models of self-organization of noun names according to the principle of minimization of efforts, which directs the human mental lexicon to optimal decoding of semantic volume of polysemic or monosemic nouns in relation to the verbal modality müssen in German-language fiction and press. It was proved that since the closest distances in the lexical structure of a word at the linguistic level are the closest in comparison with the structures of knowledge of the human mental lexicon, the construction of noun content in modern German on the synergetic principle of conservation of speech energy and minimization of efforts is the most optimal way of directing the mental lexicon of a person to decoding the semantic scope of the closets (main) meanings of the polysemic and unambiguous monosemic word models.

L’altérité: une composante clé de l'apprentissage par l'expérience?

Taking the view that learning is only learning when it is accompanied by experience, this research focuses on the identifiable variables of this experiential dimension. To do this, the researchers followed learners as they discovered cultural otherness through the Dialogue en route project (https://enroute.ch/fr/), which involves a dual encounter with a cultural site and a witness to that site. To reach the learners' semiosis, the researchers documented their learning activity using two types of trace, visual and verbal. Because of their fundamentally different intrinsic characteristics, the two types of trace opened up a process-space of intelligibility that enabled the researchers to empirically document the complexity of meaning-making, its situated nature and its potential impact on learning. To this end, the temporality of trace production was used as an instrument, since the learners photographed moments in the double encounter situation and then captioned them afterwards. Reconstructing the situation as it unfolded turned the visual and verbal modalities into semiotic resources that the learners used to reconstruct their potential experience. For the researchers, the study of the relationships between these two modal traces makes it possible to qualify the experience lived by the learners. The limited number of so-called 'rupture' relationships between the images and their verbal captions, revealing a singular experience with a high learning potential, raises questions about both the pedagogical device and the emergence of collectively constructed knowledge, captured here in its 'average' restitution rather than in its negotiation.

A Comparative Multimodal Stylistics Analysis of Selected Newspapers Cartoons

This study focuses on the comparative multimodal stylistic analysis of selected newspapers cartoons. A comparative study of newspapers cartoon juxtaposes two or more newspaper cartoon brands. Multimodality entails the mode(s) employed in a cartoon to portray meaning. The data for the study comprises ten cartoons selected through the purposive random sampling technique between 2020 and 2021 from The Punch, The Nation and The Guardian newspapers respectively. Data were analysed using Halliday’s meta-function of language in his Systemic Functional Grammar. It was discovered that framing, salience and linking layout are handy analytical tools in carrying out a multimodal analysis; each brand communicates its thoughts in a unique way with salient similarities and it was discovered that newspaper cartoons also lament the current tragic occurrence in society. This study concludes that visual mode is the lifeblood of a newspaper cartoon while the verbal mode only complements cartoon but cannot stand alone as a newspaper cartoon.

Exploring the technological affordances of SimpleMind in supported academic writing: A mind-mapping software analysis

This research investigates EFL undergraduate students’ utilization of cross-platform mind-mapping software and the factors shaping their choice of devices and timing when employing SimpleMind to enhance their academic writing skills. Grounded in Mayer’s (2019) cognitive science principles of learning, this study posits that offering EFL learners dual channels for processing information, encompassing both visual and verbal modalities, enhances information retention and reduces cognitive load. SimpleMind, a digital graphic software, has the potential to support learners in this dual coding process of information, provided it is used thoughtfully. Through the framework of the affordance-actualization theory, this research advocates for empowering language learners to proactively discover and leverage SimpleMind’s affordances to craft a customized strategy aligned with their specific academic writing learning environment. This paper presents preliminary findings from a larger study involving 71 Japanese first-year students. They explored SimpleMind’s features to cater to their individualized academic writing learning preferences.

Multimodal metaphors in a Sino-British co-produced documentary

Metaphor is an important tool for people to use in their perception of the world, but its representational forms vary across genres. Using Nvivo 12 plus as a tool, this study employs a combination of quantitative and qualitative methods to investigate the multimodal metaphorical representations, modal characteristics and cognitive rationales in the Sino-British co-produced documentary Through the Seasons: China. It has been discovered that: (1) Multimodal metaphorical representations in documentaries are found to be divided into two primary categories and five sub-categories. Implicit source-verbal and pictorial target is the primary way of representation for documentaries. (2) Various modes, including verbal mode, pictorial mode, verbal and pictorial mode, and implicit mode are employed in documentaries. However, the most commonly used modes are verbal and pictorial mode as well as implicit mode. The relationship between verbal and pictorial modes in the documentary is primarily characterised by juxtaposition and interpretation. (3) The documentary’s genre attributes and purpose, as well as the target audience’s physical and cultural experiences, are important cognitive justifications for the multimodal approach to metaphorical representation. This study further enriches the study of multimodal metaphorical representations and contributes to the theoretical refinement of multimodal metaphors. Additionally, it offers a theoretical reference for the development of dynamic multimodal discourses, such as documentaries, and aids in the improvement of audiences’ multimodal literacy.

Analyzing metaphor patterns in COVID-19 news pictures: A critical study in China.

Drawing on Critical Metaphor Analysis, this study investigates major metaphors of the COVID-19 pandemic used by the Chinese government in the specific genre of news picture. It examines patterns of metaphor use in the first year of the pandemic in China and explains how and why the government employs the identified metaphors in the Chinese political context. Results reveal that pandemic metaphors (45%) are not as widely used in news pictures as presumed, the vast majority (95%) are rendered in verbal mode, and the most salient metaphors used in news pictures are the UP/DOWN (spatial), WAR, FAMILY, and COMPETITION metaphors. This study then addresses how COVID-19 metaphors are used in the Chinese political context and claims that the Chinese government uses specific metaphors with persuasive and ideological functions. The WAR metaphor aids comprehension of abstract concepts of the pandemic treatment, the FAMILY metaphor fosters empathy among Chinese individuals to counter blame and discrimination in society, UP/DOWN (spatial) and COMPETITION metaphors stimulate action to overcome the common "enemy." WAR and FAMILY metaphors also contribute to the construction of a greater sense of collectivism and play a crucial role in fostering a positive national identity. Implications, limitations and some directions for future research are suggested.

DUGAAN UJARAN PENGHINAAN DAN PENGANCAMAN DI MEDIA SOSIAL: KAJIAN LINGUISTIK FORENSIK

This study analyzes the content of Facebook status comprising of the written status and its picture assumed to use insulting and threatening words toward Indonesian President Joko Widodo. This study uses a descriptive qualitative method by using Searle’s (1976) speech act theory as well as Kress and Leeuwen’s (2021) multimodality. The analyzes shows that there are 4 representative speech acts, 3 expressive speech acts, and two directives speech acts on the Facebook status. Moreover, this study also shows that the verbal element extends the visual element, providing additional information about the main object being discussed. The verbal mode is not dependent on the visual mode so the written status on the Facebook can be reported to the police as an offence warranting complaint based on the verbal mode only. However, the visual mode cannot be carried out to the authorities and reported as an offence warranting complaint because the visual mode has does not correspond with the verbal mode.

New Working Memory Paradigm for Neuroimaging Testing of Visual and Verbal Modality under Different Attentional Involvement

Working memory (WM) is a cognitive function essential for short-term maintenance of information in a highly accessible state to support goal-directed behavior. The classical behavioral model of WM includes a visuospatial sketchpad, a phonological loop and the central executive. Neuroimaging studies selectively targeted the activity associated with maintenance and processing of modality specific information. However, an experimental design is still missing that would enable the assessment of all components of WM drawing a holistic neuroimaging model. In this study, we propose a modified paradigm based on the classical retro-cue task, which allows disentangling the activity of all WM components, and in particular of the central executive. This paradigm consists of five conditions: passive perception, simple verbal storage, simple visual storage, alphabetical reordering (complex verbal) and mental rotation (complex visual). Testing on a cohort of 35 healthy adults, we obtained a similar workload for simple storage conditions with a low engagement of the central executive. A different workload was verified between the simple and complex conditions in both verbal and visual modalities. This experimental design provides the framework to assess the neural activity associated with the central executive components in different modalities and to address the question of a unitary or modality-specific central executive nature. Therefore, the paradigm is suitable for utilization in neuroimaging to potentially advance our comprehension of the WM organization.

The relationship of aerobic fitness with verbal and spatial working memory: An ERP study.

Working memory (WM) plays an important role in daily life and is known to correlated with aerobic fitness. However, whether the relationship between aerobic fitness and WM is dependent on the stimulus modality or is associated with one or multiple subprocesses involved in WM remains unknown. Accordingly, this study utilized event-related potentials (ERPs) to comprehensively examine the encoding, preparation, and retrieval processes during verbal and spatial WM performance. Eighty-eight young adults aged 18-30years were recruited to participate in two laboratory visits on separate days. On day 1, aerobic fitness was assessed by maximum oxygen consumption (V˙O2max) during a treadmill-based graded exercise test. On day 2, participants completed verbal and spatial WM tasks while P2, contingent negative voltage (CNV), and P3 components of ERP were recorded during the encoding, preparatory, and retrieval stages of WM, respectively. Results of hierarchical regression analysis showed that V˙O2max was positively correlated with response accuracy during the high-demanding condition of spatial WM after controlling for age, sex, and self-reported physical activity. Additionally, a higher level of V˙O2max was associated with larger terminal CNV amplitude at the Cz electrode during the high-demanding condition of spatial WM. These findings suggest that aerobic fitness may have selective beneficial associations with the motor preparatory process and subsequent task performance requiring a greater amount of spatial information but not the encoding and retrieval stages nor the verbal modality of WM.

Multimodal Discourse Analysis of Figures Used in the English Textbook “Get Smart Plus”

The study analyzed the multimodality of the EFL textbook for elementary school students. A qualitative descriptive study has been employed to carry out this study. The source of the data is the textbook, namely Get Smart Plus. The data was visual and verbal elements within the textbook, including any text and images that explain each other. The study used the multimodal framework proposed by Krees Van Leuween’s visual grammar and Halliday’s transitivity system to analyze the data. The finding shows that the visual elements of the textbook contain representation, composition, and relational patterns. In addition, the dominant process in the textbook is the verbal mode of communication (42%). However, the material process based on the analysis of Halliday’s theory was not represented in the book. The visual and verbal elements relate to each other and can potentially support students’ learning and understanding of the content. Therefore, the textbook can be a multimodal resource for students’ English learning. This study hopes to benefit textbook developers and designers by improving the use of visual and verbal elements of the textbook to support students’ cognitive development. Further research can be conducted on the impact of textbook multimodality on students’ learning achievement to contribute more to the study of discourse.
 Keywords: Multimodal Discourse Analysis, EFL, English textbook

Home Automation for Disable Person Using Gesture Control and Artificial Intelligence

For an extended time, gestures communique has performed a giant function in conveying information. Before the evolution of human beings, while no verbal mode of communique changed into the present, gesture communique changed into the simplest manner to interact with different human fellows. After evolution, with language improvement, the want of gestures for brief variety communique changed into removed. However, it became nonetheless giant in conveying messages at large distances, i.e., by lightening the hearth place at a far-off place. The modern generation is gaining this mode of communique, and it has correctly translated those gestures into the virtual world. The improvement of augmented truth is an instance of it. With the arrival of deep gaining knowledge of algorithms like Artificial Neural Networks (ANN), Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN), the issue in records evaluation and sample recognition has come to be very smooth and robust. In this undertaking, we've advanced a mind gestures (signals) reputation machine that controls home automation home equipment for bodily impaired people.

Questions directed to children with diverse communicative competencies in paediatric healthcare consultations

ObjectiveThis paper examines question-response sequences, in which clinicians asked questions to child patients who appear to interact using means other than the verbal mode of communication. MethodsConversation Analysis methods were used to study questions in 46 paediatric palliative care consultations. These questions were directed towards children who observably used vocalisations and embodied modes of communication (e.g., gaze, gesture and facial expressions) but did not appear to use the verbal mode. ResultsMost questions asked children either about their willingness and preferences for a proposed next activity, or their current feelings, experiences or intentions. Questions involved children by foregrounding their preferences and feelings. These questions occasioned contexts where the child’s vocal or embodied conduct could be treated as a relevant response. ConclusionThis paper demonstrates how questions are used to involve children in consultations about their own healthcare, and how their views come to be understood by clinicians and family members, even when children interact using means other than the verbal mode of communication. Practice ImplicationsQuestions can be asked of both children who do and do not verbally communicate. When asking questions, clinicians should be mindful of the modes of communication an individual child uses to consider how the child might meaningfully respond.

Ventral and dorsal aspects of the inferior frontal-occipital fasciculus support verbal semantic access and visually-guided behavioural control

The Inferior Frontal Occipital Fasciculus (IFOF) is a major anterior-to-posterior white matter pathway in the ventral human brain that connects parietal, temporal and occipital regions to frontal cortex. It has been implicated in a range of functions, including language, semantics, inhibition and the control of action. The recent research shows that the IFOF can be sub-divided into a ventral and dorsal branch, but the functional relevance of this distinction, as well as any potential hemispheric differences, are poorly understood. Using DTI tractography, we investigated the involvement of dorsal and ventral subdivisions of the IFOF in the left and right hemisphere in a response inhibition task (Go/No-Go), where the decision to respond or to withhold a prepotent response was made on the basis of semantic or non-semantic aspects of visual inputs. The task also varied the presentation modality (whether concepts were presented as written words or images). The results showed that the integrity of both dorsal and ventral IFOF in the left hemisphere were associated with participants’ inhibition performance when the signal to stop was meaningful and presented in the verbal modality. This effect was absent in the right hemisphere. The integrity of dorsal IFOF was also associated with participants’ inhibition efficiency in difficult perceptually guided decisions. This pattern of results indicates that left dorsal IFOF is implicated in the domain-general control of visually-guided behaviour, while the left ventral branch might interface with the semantic system to support the control of action when the inhibitory signal is based on meaning.

A Comparative Study of Intersemiotic Complementarity of Visual and Verbal Modes in Three English School Textbooks from Iran and Malaysia

A Comparative Study of Intersemiotic Complementarity of Visual and Verbal Modes in Three English School Textbooks from Iran and Malaysia