Intersemiotic Relations Research Articles

The genre of PechaKucha presentations: Analysis and implications for enhancing multimodal literacy at university

ESP students need to develop their multimodal literacy to become literate in today's professional spaces. For this purpose, ESP teachers should revisit pedagogy practices to best engage students in the navigation and construction of multimodal genres. As a case in point, we explore PechaKucha (PK) presentations. This multimodal genre consists of 20 slides, which are automatically advanced every 20 s. PK presentations entail a complex format that requires speakers to choose how to convey content, design suitable visuals, and engage audiences. The dataset for the study consists of 7 PK presentations delivered during a social event at an architecture conference. Adopting a multimodal discourse analysis lens, we analyse this set of PK presentations in terms of rhetorical structure and the way in which intersemiotic relations unfolded (synchronisation between speech and visuals and the modal density of slides). The analysis demonstrates that PK presentations entail an intricate multimodal composition consisting of three moves in which professional and personal narratives intertwine. The examination of intersemiotic relations reveals how speech and visuals interplay effectively to transmit meaning and engage the audience. The results of this study provide critical information to design a research-informed pedagogy to enhance ESP students' multimodal literacy.

Computer-assisted multimodal analysis research: A technical review of GRAPE-MARS

Multimodal (discourse) analysis is about analyzing and theorizing individual semiotic resources (e.g., language, image, audio, space, gesture, facial expression) and inter-semiotic relations to make meaning of multimodal phenomena in context. Conducting multimodal analyses can be challenging because of its complexity. Computer-based software can support multimodal analysis by automating or semi-automating stages of annotating and analyzing, thus reducing the extensive time and labor required. The designed affordances of the software are aimed at helping researchers establish links between low-level features of multimodal texts and higher-order multimodal semantic meanings by following specific workflows. Amongst many popular multimodal transcription tools (e.g., ELAN, CLAN, ChronoViz), a new open-source multimodal annotation software for video analysis, GRAPE-MARS has been recently launched. This technical review will first describe the organization and affordances of GRAPE-MARS by illustrating a multimodal analysis of a video via this tool. We will then highlight the principal functionalities that make this software efficient in supporting multimodal analysis. Finally, the limitations and future possible applications of GRAPE-MARS will be discussed.

Polycode text in digital communications of the university: Сontent, graphics, illustrations

The article considers the structuring regularities of the polycode text functioning in the news feeds of the technical universities websites. The relevance of the research is due to the need for a linguistic description of various formats and genres of digital communication. The novelty of the research lies in the comprehensive analysis of the scientific news feed of technical universities. The purpose of the study is to analyze the verbal (verbal media topics, themes) and non-verbal (graphics, illustrations, visual media topics) components of a polycode text. The research methods are content analysis, graphical and semiotic analysis. The multidimensional characterization of the various components of the text of scientific content makes it possible to identify the thematic dominants of university content, as well as a system of paraverbal techniques for its design. The discourse analysis makes it possible to establish inter-semiotic relations that reflect the distribution of meaning at the level of the verbal and visual code of the text. The image of the technical university as a research structure in the relevant fields of science and production dominates at the level of verbal codes. Herewith, the scientist (as a person) is on the periphery, but it provides the research process. Visual media topic is dominated by the image of a male scientist (as a person), which is a more significant component in the structure of the scientific process than the process itself and the scientific community. The visual components of the text (graphics and illustrations) are distinguished by a narrow repertoire of techniques. The analysis of verbal and visual media topics showed a logical inversion. The visual code complements and compensates for the verbal code, and vice versa. These data are consistent with the established dominant function of illustrations in the news text: the function of a supplement.

Book Covers and State Translation Programs: A Multimodal Approach to Book Covers in the Panda Books Series

In the era of globalization, institutional translation has become increasingly important. The Panda Books Series (PBS) is an exemplar of institutional translation that translates and disseminates Chinese literature to other parts of the world. The present study provides a multimodal analysis of the book covers in the PBS in terms of their visual meanings and intersemiotic relations. Analysis of 129 front covers and 64 back covers reveals that a positive and non-threatening image of China is constructed by characterizing China as a gender-equal, peaceful, and rural country, while a modern and approachable image of Chinese writers is created to highlight the authorship. The verbal and visual modes that are present on the covers often repeat the same information to reinforce positive images of China and Chinese writers, which is instrumental in realizing the goal of the PBS. It is argued that the agency of subjects at different levels, that is, the political ideology of the state, the governmental institutions involved, and translators and editors, all combine to shape the book cover designs of the PBS. This study contributes to research on institutional translation, especially state translation program, from a multimodal perspective.

Representational and Ideational Meanings of Images and Texts of Tourism Promotion on Instagram

Tourism is one of prospective sectors to increase Indonesian economic level. It is crucial to develop effective promotion media of tourist destinations in Indonesia. One of the recent potential media is Instagram. This paper aims to investigate the representational and ideational meanings of visual and verbal modes, including their intersemiotic relations, in promoting Taman Nasional Komodo Labuan Bajo, focused on @btn_komodo as an official Instagram account of Balai Taman Nasional Komodo. The destination is one of the 5 super priority destinations in Indonesia. By applying the Visual Grammar framework (Kress & Van Leeuwen, 2006), this study views how the visual and verbal modes construe the meaning for the purpose of tourism promotion on the Instagram. The data was captured from Instagram posts in the form of visual texts (photos) and verbal texts (captions). The posts are predominantly communicated through the visual text, then completed, clarified and confirmed by the accompanying verbal text. Multimodal discourse analysis of the Instagram posts shows that both visual and verbal texts contribute the values of promotion. The result implies that future research should include more complete material of semiotic resources and implement more comprehensive analysis related to multimodal discourse.

Representing cityscape through texts and images: translations of multimodal public notices in Macao

ABSTRACT This study investigates the translation of multimodal public notices in Macao from the perspective of visual social semiotics. With a self-built database, the study first conducts a detailed textual analysis to examine the linguistic features and translations of verbal texts in the multimodal public notices in Macao, and then discusses different types of multimodal representations and text-image relations in the public notices through a multimodal analysis. The research results show that Chinese poetic forms and Cantonese expressions are mainly used in multimodal public notices in Macao; the realization of appellative function remains a pivotal issue in public notice translation. Meanwhile, the photograph, cartoon and comic images in public notices interplay with the texts and produce diversified multimodal representations and intersemiotic relations, which contributes to the publicity in multimodal public notices and represents the cityscape of Macao as an inclusive city with multifaceted publicity forms and multicultural expressions.

An Intersemiotic Analysis of the Arabic Dubbed Version of Disney’s Frozen Let it Go

: In today’s digital environment, verbal, visual and sound resources constantly collaborate to construct meaning in the audiovisual mediascape. Audiovisual Translation (AVT) studies have thus shifted focus to examine how such multimodal orchestration and meaning shifts are captured and transferred from one culture to another. Songs’ AVT in animated musicals imposes (non)verbal constraints on translators who have to adapt/adjust to transfer not only the verbal and visual codes but also the musical. This article examines the intersemiotic relations among the three different semiotic modes when dubbing Let it Go from English into Arabic. It builds on Reus’ (2020a/b) Triangle of Aspects to fit both academic and practical purposes. The results suggest that dubbing a song mainly relies on a quadrangular of parameters: meaning-making, technical, interpretative, and performative elements of translation. Verbally, the findings indicated that the translator oscillated between full adherence to disregard of the semantic meaning. Nonverbally, the findings revealed mixed results regarding the OS-DS (in)congruencies in which the translator foregrounded and/or backgrounded certain aspects to conform to imposed constraints. Four different intermodal relations were identified: addition, enhancement, modification, and deletion. This study offers a methodological contribution to the AVT scholarship, positing a framework that can be systematically followed in future research.

A Multimodal Discourse Analysis of BBC Documentary Chinese New Year: The Biggest Celebration on Earth

With the rapid development of multimedia and the internet, traditional discourse analysis, which only studies text, is gradually transferred to multimodal discourse analysis (MDA), studying image, language, color, sound, etc. Informed by Systemic Functional Linguistics, this paper tries making a multimodal discourse analysis of the BBC documentary Chinese New Year: The Biggest Celebration on Earth (CNY). Based on Kress & Van Leeuwen’s Visual Grammar and Zhang Delu’s visual-verbal Inter-semiotic relations, the paper examines how different semiotic modes combine to provide meaning in a documentary. After a synthesis analysis, it is concluded that the employment of various modes in this dynamic discourse contributes to the spreading of Chinese culture and the representation of “Cultural China” image. It is hoped that this paper will provide a new perspective for the semiotic studies of documentaries. The study's importance lies in demonstrating how numerous modes and sub-modes are employed in the source domains as part of a larger persuasion campaign to skew public perceptions of China. Additionally, we look at the metaphorical implications. Keywords: multimodal discourse analysis, systemic functional linguistics, BBC documentary. DOI: https://doi.org/10.55463/hkjss.issn.1021-3619.60.40

The Co-construction of News Values on News Magazine Covers: A Corpus-assisted Multimodal Discourse Analysis (CAMDA)

ABSTRACT Extant research from various perspectives has provided valuable insights into what news values are more frequently foregrounded across news types, topics, and outlets; and how they are constructed through verbal and visual resources. However, no research so far has examined how specific news values may be frequently co-constructed to package an individual news story and make it more appealing to potential readers. This research adopts the discursive news values analysis (DNVA) approach to examine the co-construction of news values in the multimodal discourse of TIME magazine covers (n = 300). To identify such co-construction, this study adopts cluster analysis, which has yet been deployed to study news values. The use of cluster analysis has identified six different combinations of news values repeatedly been observed together in clusters of covers. These combinations may represent what the magazine perceives as most attractive to the target audience. This study has also revealed intersemiotic relations in news values construction. While some news values are often reinforced across the verbal (i.e., cover line) and visual (i.e., cover image) semiotic modes, others rely more on the verbal mode. This study crosses disciplinary boundaries by bringing corpus linguistics techniques into journalism studies to understand a previously under-explored news genre.

Cognitive and Intersemiotic Model of the Visual and Verbal Modes in a Screen Adaptation to Literary Texts

The aim of the study is to examine screen adaptations from the perspective of cognitive and intersemiotic models of the visual and verbal modes. The purpose of the study is to express the specificity of a screen text which is defined as a combination of three media: speech, image, and music. The scope is to demonstrate the general framework of an intersemiotic translation from a new point of view – like a transliteration. The method of the research refers to semiotic and stylistic analyzes – methods of transformation from one sign system into another from prose works with regard to their cognitive as well as narrative and stylistic features (Zhong, Chen, & Xuan, 2021). Thus, the study analyses such specific relations between the verbal and visual modes in film adaptations of prose literature as a more detailed description of event episodes, events’ temporal structure, presentation of author’s thoughts and characters’ thoughts; their mental activity formulated indirect speech and inner speech that is shown only by the actor’s intonation. The results of the study made possible to show the types of inner speech in their adaptations: author’s thoughts, characters’ thoughts which are presented only by the verbal mode, and visual modes’ inner speeches that combine the modes of character’s voice and image. One can conclude, that taking into account intersemiotic relations between the visual and verbal spaces, it is possible to explain, for instance, how the words of characters are replaced by their facial expressions, gestures, or intonations.

INTRODUCTION

INTRODUCTION

A Survey on Intertextual and Intersemiotic Relations between the Sechischti Lyrics and the Kilims of the North Khorasan’s Kurmanj Nomads

A Survey on Intertextual and Intersemiotic Relations between the Sechischti Lyrics and the Kilims of the North Khorasan’s Kurmanj Nomads

Second language learners’ reading strategies. The case of intersemiotic relations

Second language learners’ reading strategies. The case of intersemiotic relations

Wstęp. Przekład intersemiotyczny w poezji Europy Środkowej w XX wieku

For years, Porówania has focused on diverse correspondences between cultures, ethnic literatures, areas of art, inhabitants of various geographical spaces, imagined communities, contemporary discourses, etc. This volume is dedicated to the conversational strategies artists established between language and the visual arts, but also to the broader, and often variously defined, area of Central Europe. We were interested in the relationship between different (thematically and functionally) intersemiotic translation strategies with regard to cultural events and changes in the socio-political life of Central European nations, dominant communities and ethnic minorities. We also wanted to address the similarities and differences visible in this area on the basis of the different literatures and art (domestic and emigration; high-art and popular art). The selected case studies gathered in this volume are meant at least initially to outline a map of diverse intersemiotic relations, the coordinates of which are determined, on the one hand, by the broadly understood category of visibility and, on the other, by the cultural, social, political changes imposed on the Central European nations and communities they helped form. We wanted to ask the following question: does the changeability of the intersemiotic phenomena and theories justify the strength of the caesura, which we used to take for granted in ethnic literatures of this part of Europe? Does it help to better describe the “isms,” which have been developed by modern art and literature?

Multimodal approach to analysing big social and news media data

Multimodal analysis traditionally involves conceptualising abstract frameworks for language, images, and other resources and their intersemiotic relations (e.g. text and image relations) and then demonstrating these frameworks with some examples. This scenario has changed with the recent move towards multimodal approaches to big data analytics which will involve empirically testing and validating multimodal theory and frameworks through the analysis of large data sets. However, large training sets of analysed texts are required to develop computational models based on multimodal theory. Therefore, an alternative approach which involves integrating multimodal frameworks with existing computational models for big data, cloud computing, natural language processing, image processing, video processing, and contextual metadata is proposed. The integration of these disparate fields has the potential to dramatically improve computational tools and techniques, thus placing multimodality at the forefront of research aimed at mapping and understanding multimodal communication. As a step forward in this direction, we explore how existing computational tools and approaches can be integrated into a multimodal analysis platform (MAP) with facilities for searching, storing and analysing text, images and videos in online media, together with dashboards for visualising the results. Preliminary analyses and classifications of text and images about COVID-19 and George Floyd in five online newspapers and Twitter postings show how media patterns can be studied using existing computational tools. The study highlights (a) the benefits and current limitations of big data approach to multimodal discourse analysis and (b) the need to incorporate knowledge about language, images, metadata, and other resources as semiotic systems (rather simply sets of symbols and pixels) to improve computational techniques for big data analytics.

Паремії як засоби реалізації категорії інтертекстуальності в драматичному тексті буковинських письменників

Paremic units as a means of intertextuality in the dramatic text of Bukovynian writers of the second half of the XIX – early XX centuries (Yu. Fedkovych, S. Vorobkevych, S. Yarychevsky) are analyzed in the article. Functional groups of paremias as a kind of communicative code of Bukovynians are singled out. The peculiarities of the use of set folk expressions in the dramatic works of Bukovynian writers in the corresponding communicative situation are examined. The system-forming category of a dramatic text is intertextuality – a versatile connection of the text with other texts in terms of content, genre and stylistic features, structure, formal and symbolic expression. The most realized intersemiotic relations in dramaturgy are the relations between verbal texts and texts that represent other semiotic systems. Within the theatrical performance there is a complementarity of verbal and musical within the song, painting and words, cinematic adaptation of the literary text, in particular, the transmission through visual images of verbal significance. In the DT of Bukovynian writers we pay special attention to paremias as a means of intertextuality. Paremia is the history of the Ukrainian people that is contained in concise but wise expressions. Paremias fully reflect all stages of the development of the nation's consciousness and historical events in the society. This is an example of the linguistic perfection of mankind. Paremias focus on the whole cognitive essence of man: history, experience, memory, imagination, worldview, sorrow and joy, suffering and happiness. Paremias are the highest manifestation of the folk spirit, the depths of folk philosophy, ethics and aesthetics in the dramatic works of Bukovynian writers. Drama (in comparison with other art forms) reproduces the vernacular, in which paremias are realized the most. In the works of Bukovynian playwrights we have focused on proverbs and sayings, distinguishing them into certain functional types of paremias that regulate relations between people in society; reveal popular ideas and observations on the development of society; characterize the nature of human speech; paremias, expressing peculiar moral and social precepts for person; paremia-assessment. On the other hand, paremic units recorded in the dramatic works of Bukovynian writers are a manifestation of the intertextuality of DT and certify the perception, understanding and knowledge of the human world. Paremias are the result of the collective speech activity of the people and they represent group and individual forms of human communicative behavior. They are kind of spiritual code of Bukovinians which regulates the life of each individual.

El audiovisual en escena

El teatro contemporáneo recurre con frecuencia al uso de audiovisuales, práctica que tiene su origen en la misma invención del cinematógrafo. Las relaciones entre cine y teatro, además de trasponer códigos del uno al otro, han permitido transferir también su semántica, al punto de configurarse la videoescena como un elemento dramatúrgico de la escenificación. Este trabajo busca analizar cómo tienen lugar las relaciones intersemióticas entre el lenguaje audiovisual y el escénico cuando se incluye la videoescena en la dramaturgia teatral. Para ello se revisa una breve historia del audiovisual en el teatro y la terminología existente, que permiten extraer conclusiones tomando en consideración espectáculos de referencia, tanto de ámbito español como internacional.

Tertium comparationis w badaniach nad przekładem filmowym

The paper attempts to present a review of the term tertium comparationis (TerComp) in the studies of film translation. For many years this issue has attracted the attention of linguists and translators trying to determine a common reference platform for both the compared elements of linguistic systems and the realization of those systems in specific speech acts (texts). In the studies of film translation as a polysemiotic text, TerComp requires separate attention. The starting point for this study is the same assumption as in A. Boguslawski that the superior TerComp is a concrete situation of the use of linguistic signs. For the purpose of the studies of film translation, this situation was specified more precisely. It was determined that in the case of film translation, a two-type situation should be considered: external, i.e. a communication situation in which the compared elements of the text were used and a situation within a film as polysemiotic communication, in which only verbal messages – which are in different intersemiotic relations with the visual code, also with the audio code – are translated and, at the same time, remain in interlingual relations with the original verbal text – in the case of voice-over and subtitles. TerComp in film translation is an important criterion in translation choices, determining the equivalence between the original and the translation, helping to identify translation units and to search for pairs of translation equivalents.

Texto e imagem: a complementaridade intersemiótica em dicionários escolares Tipo 2

O texto dos dicionários escolares se utiliza de ilustrações para complementar as definições. Nosso objetivo com este artigo é investigar e descrever os tipos de relações semânticas intersemióticas existentes entre texto e imagem nos verbetes ilustrados dos dicionários escolares tipo 2. Adotamos o modelo proposto por Royce (1998) para o estudo das relações intersemióticas entre texto e imagem. Desenhamos uma pesquisa quati-qualitativa em que verificamos a ocorrência ou não da complementaridade intersemiótica entre o texto do verbete e a ilustração por meio das seguintes relações semânticas: repetição, sinonímia, meronímia, hiponímia, antonímia e colocação. Os resultados deste estudo demostram que a complementaridade intersemiótica ocorre em quase 90% dos verbetes ilustrados, destacando-se a relação semântica de sinonímia.

When the classic speaks for children

AbstractBob Dylan has significantly influenced American song tradition and popular music for more than five decades. As his songs are famous for the creative poetic expressions, they are not merely song lyrics, but also can be regarded as lyric poems. This paper aims to investigate the retranslation of Bob Dylan’s songs in bilingual picture books to explore how his classic musical works are repackaged both verbally and visually for contemporary children. The data for analysis are selected from two bilingual picture books on Dylan’s songs published in China in2018. Firstly, it conducts a detailed textual analysis of the English and Chinese song lyrics to analyze the appropriate translation strategies and methods for song translation. Secondly, based on visual narratives (Painter, Martin and Unsworth 2013), it analyzes the intersemiotic relations between texts and images in picture books to discuss how the emotions and narratives in Dylan’s songs are visually represented.