Impact of verbal comment on interpretation of visual message: Multimodal approach

Abstract

The paper discusses several advantages of multimodal approach to the analysis of nature and specifics of media audience’s reception of texts. The authors base their ideas on the principally multimodal nature of media communication, which reflects in the way the recipients interpret texts. Thus, various factors impact on the interpretation of the initialized text. Moreover, such factors predominately include other semiotic resources that are not formally affiliated with the basic text. In the research, the hypothesis was about the impact of verbal comments on interpretation of visual objects by recipients. According to this goal, the authors selected two groups of respondents — a control and an experimental one, which were offered a photograph and were asked to describe it verbally in a free manner. The experimental group of respondents was offered the same photograph with motivating verbal comment, including the general information about the origins and topics of the photograph. The authors compared the results of the both groups descriptions by matching verbalized categories of representation, individual evaluations of the photograph and “the syntax” of verbal description of the visual media text. The research results proved the general supposition: the multimodal approach, being aimed at finding, describing and explicating the meaning effects of semiotic ensembles, contributes to understanding the features of interpretation of visual objects under the influence of other semiotic (here — verbal) resources. Thus, the results show that the interpretation of visual objects is motivated by the verbal comment: it topicalizes and contextualizes the visual reception and interpretation of the basic message. In the conclusion, the authors define the perspectives both of the research and the multimodal approach to media texts study.