Caption Text Research Articles

AN ANALYSIS OF SUBTITLING SLANG EXPRESSIONS AND READABILITY IN PURPLE HEART MOVIE FROM ENGLISH INTO INDONESIAN

Subtitling strategies is part of translation process to gain a good result. The purpose of this study is to identify the slang expressions and analyze the subtitling strategies, slang types, and the readability rating of subtitle in Purple Heart movie. The data of this study is the subtitle text that contains slang expressions in Purple Heart movie. The data of this study is collected by searching and downloading, identifying, and classifying the data that contains slang expressions to the subtitling strategy and slang types. This study also uses a method with a questionnaire test on 11 respondents to determine the level of readability of the translation results. The writer analyzes the data using qualitative research according to Miles and Huberman's techniques, such as data reduction, data display, and conclusion. The result of this study shows that there are 73 data of slang expressions. The result of slang types shows that general slang with 44 data and specific slang with 29 data. The subtitling strategies applied in the data is only 5 strategy out of ten they are transfer with 34 data, paraphrase with 23 data, deletion with 9 data, condensation with 4 data, and expansion with 3 data. The result of the readability rating in the translation of slang expressions in the subtitle of Purple Heart movie shows that there are 803 data. The most dominant used is the readable category with 558 data, less readable with 173 data, and unreadable with 72 data.

Transformative Fusion: Vision Transformers and GPT-2 Unleashing New Frontiers in Image Captioning within Image Processing

In the ever-evolving digital landscape, this paper presents an innovative Image Caption Generator that seamlessly merges Vision Transformers (ViT) and GPT-2. By combining the strengths of computer vision and natural language processing (NLP), our paper aims to extract significant image features using ViT and generate contextual, human-like descriptions through GPT-2. The resultant system boasts an intuitive interface, allowing users to effortlessly receive coherent captions for uploaded images. This ground breaking technology holds immense potential for the visually impaired community, enhancing image-based content accessibility and overall user experiences. The primary objective of our image caption generator paper is to develop a sys-tem that automates the generation of descriptive and coherent textual captions for images. This endeavor involves the integration of computer vision and NLP techniques, enabling the system to analyze the content of an image and produce relevant and meaningful textual descriptions. The broader goal is to improve the accessibility of visual content, enhance image search capabilities, and facilitate applications such as automated content tagging. Furthermore, the paper addresses the needs of visually impaired individuals by providing assistive technology that interprets and communicates image content effectively. This paper exemplifies the symbiotic relationship between computer vision and NLP, illustrating how their integration can pave the way for transformative AI applications. The resulting synergy not only contributes to the development of advanced image captioning systems but also opens avenues for innovative applications across diverse domains. The conference presentation will delve into the technical aspects of our approach, showcasing the significance of this integration and its potential impact on the future of AI applications.

Writing on screens: (Re-)mediating music and sound through captions

The article focuses on the screenplay’s ‘afterlife’, as a (re-)creative product of captioners and a text for reading by the d/Deaf and Hard of Hearing (DHH) audience. In particular, it explores captioning practices that textualize aspects of the soundtrack crucial to screenplay meanings. Close study of horror series Stranger Things (ST, Netflix) and The Last of Us (TLoU, HBO) reveals how their closed captions represent the end in a unique chain of mediated translations between the script’s written word, the media form’s soundtrack and the captions’ screen text. Comparing ST Season 4, Episode 9 with TLoU Season 1, Episodes 3 and 6 uncovers the different approaches to captioning music and sound effects adopted by captioners. Moreover, juxtaposing the ST Episode 9 music and sound captions with its screenplay by the Duffer Brothers discloses the considerable gap between screenplay and captioned text, which argues for the significant contributions of captioners to media meanings initially created by screenwriters.

Automatic Intelligence Caption Generator

An Image Caption Generator is a sophisticated AI system that combines computer vision and natural language processing to automatically create descriptive textual captions for images. This technology utilizes deep learning, particularly Convolutional Neural Networks (CNNs), to analyze and extract meaningful visual features from the input image. These features capture details about the objects, scenes, and elements within the image. Subsequently, a natural language processing model, often built on Recurrent Neural Networks (RNNs) or Transformers, processes these visual features and generates coherent, contextually relevant captions. Post-processing steps may be applied to enhance the quality of the generated text. The primary aim of Image Caption Generators is to facilitate image understanding, improve accessibility, and enhance content search ability by providing human-readable descriptions for visual content. This technology is instrumental in various fields, including content tagging, accessibility tools for the visually impaired, and enhancing user experiences in multimedia content management systems, ultimately bridging the gap between visual and textual information for a more comprehensive and human-like interpretation of image. Key Words:Image Recognition, Internet, Image-To-Caption, Contextual Understanding,Image Captioning.

Cross-modal fine-grained alignment and fusion network for multimodal aspect-based sentiment analysis

Multi-modal Aspect-based Sentiment Analysis (MABSA) aims to forecast the polarity of sentiment concerning aspects within a given sentence based on the correlation between the sentence and its accompanying image. Comprehending multi-modal sentiment expression requires strong cross-modal alignment and fusion ability. Previous state-of-the-art (SOTA) models fail to explicitly align valuable visual clues with aspect and sentiment information in textual representations and overlook the utilization of syntactic dependency information in the accompanying text modality. We present CoolNet (Cross-modal Fine-grained Alignment and Fusion Network) to boost the performance of visual-language models in seamlessly integrating vision and language information. Specifically, CoolNet starts by transforming an image into a textual caption and a graph structure, then dynamically aligns the semantic and syntactic information from both the input sentence and the generated caption, as well as models the object-level visual features. Finally, a cross-modal transformer is employed to fuse and model the inter-modality dynamics.This network boasts advanced cross-modal fine-grained alignment and fusion capabilities. On standard benchmarks such as Twitter-2015 and Twitter-2017, CoolNet consistently outperforms state-of-the-art algorithm FITE with notable improvements in accuracy and Macro-F1 values. Specifically, we observe an improvement in accuracy and Macro-F1 values by 1.43% and 1.38% for Twitter-2015, and 0.74% and 0.88% for Twitter-2017, respectively, demonstrating the superiority of our CoolNet architecture.

Image Captioning Using Deep Learning

Abstract: This paper focuses on developing an image captioning system using deep learning techniques. The paper aims to generate descriptive textual captions for images, enabling machines to understand and communicate the content of visual data. The methodology involves leveraging convolutional neural networks (CNNs) for image feature extraction and recurrent neural networks (RNNs) for sequential language generation. The paper includes steps such as dataset collection, data preprocessing, CNN feature extraction, RNN-based captioning model implementation, model evaluation using metrics like BLEU score and METEOR, and presenting the results obtained. The expected deliverables include a functional image captioning system, comprehensive documentation, and a well-documented codebase. Through this paper, students gain practical experience in deep learning, computer vision, and natural language processing, contributing to advancements in image understanding and humanmachine interaction with visual data

Captioning Image Using Deep Learning: A Novel Approach

Abstract: Captioning image using deep learning is a technology that aims to generate descriptive and accurate textual descriptions for images. By using the power of deep neural networks, this approach enables computers to understand and interpret visual content bridging the gap between the visual and textual domains. This paper focuses on developing an image captioning system using deep learning techniques. The paper aims to generate descriptive textual captions for images, enabling machines to understand and communicate the content of visual data. The methodology involves leveraging convolutional neural networks (CNNs) for image feature extraction and recurrent neural networks (RNNs) for sequential language generation. Captioning images is a challenging task in computer vision that involves describing the content of an image in natural language. In recent years, deep learning techniques have shown remarkable success in various computer vision tasks, including image captioning.

COSMOS: Catching Out-of-Context Image Misuse Using Self-Supervised Learning

Despite the recent attention to DeepFakes, one of the most prevalent ways to mislead audiences on social media is the use of unaltered images in a new but false context. We propose a new method that automatically highlights out-of-context image and text pairs, for assisting fact-checkers. Our key insight is to leverage the grounding of images with text to distinguish out-of-context scenarios that cannot be disambiguated with language alone. We propose a self-supervised training strategy where we only need a set of captioned images. At train time, our method learns to selectively align individual objects in an image with textual claims, without explicit supervision. At test time, we check if both captions correspond to the same object(s) in the image but are semantically different, which allows us to make fairly accurate out-of-context predictions. Our method achieves 85% out-of-context detection accuracy. To facilitate benchmarking of this task, we create a large-scale dataset of 200K images with 450K textual captions from a variety of news websites, blogs, and social media posts

A Multimodal Emotion Perspective on Social Media Influencer Marketing: The Effectiveness of Influencer Emotions, Network Size, and Branding on Consumer Brand Engagement Using Facial Expression and Linguistic Analysis

Social media influencers rely on emotional connection to maintain and grow their followings and have value for brands. To date, however, no research has quantitatively examined the impact of emotion in the facial expressions and caption text that influencers use in their video posts on consumer engagement through likes, comments, and views of posts. Grounded in consumer brand engagement, psychological sense of community, and the behavior ecology view of facial displays, this study uses social media analytics, facial expression analysis, and computational linguistic analysis to assess the emotional substance of 402 video posts by prominent micro-, macro-, and mega-influencer mothers, known as InstaMoms, as exemplars of Instagram influencers. The study identifies that the amount of emotion used and specific discrete emotions have a meaningful influence on engagement, and both follower count and presence of branding saliently contribute to a more robust understanding of the relationship. Theoretical and practical implications are identified and discussed.

“DARI LUCU MENUJU KOMERSIAL” SEBUAH TINJAUAN SEMIOTIKA VISUAL KOMIK IKLAN TAHI LALATS

The Tahilalats advertising comics are a differentiator from works that only aim to convey humorous messages on Instagram. An overview of advertising comics with commercial purposes is interesting because the contents of commercial advertisements for a product become a new pattern of creativity for Tahilalats creators. The object of the problem taken from the work entitled “Surprise” offers a form of negotiation between the aim of being funny and commercial advertising The Ponds Men advertisement is a promotional idea for men’s skin care products which is humorously visualized in ten comic strip panels. The series of panels is a sign structure that will be dissected in order to find the right meaning to show the persuasion message of a commercial product. The accuracy of meaning can be seen from the visualization of the text or aspects of form and context or content as part of the marking system. The marking system will be reviewed based on the sign structure of Barthes’s semiotic theory, namely lexia. The meaning analysis begins by re-reading the nine comic panels into four lexias by naming them based on the logic of meaning, not chronology. In the lexia you will find code categories used as sign dynamic rules according to Barthes based on the perception and culture of the users as interpreters including hermeneutic codes, semimic or semantic codes, symbolic codes, proairetic or action codes, and cultural codes. The code becomes the main reference showing the context in the form of a persuasive message from a Ponds Men commercial advertisement. The context of being funny can be visualized with aspects of character shape, cartoon-style expressions that are simple and simple. The supporting theory is in the form of relays and intertextuality from Julia Kristeva to review the performance of the tagging system based on caption text in the form of titles and descriptions in the Instagram Tahilalats column. In this case, it can show that the commercial message of Ponds Men’s products can be combined appropriately according to Tahilalats’ cute characterization.Keyword: advertising, comic, tahilalats, visual semiotics

WCC-JC 2.0: A Web-Crawled and Manually Aligned Parallel Corpus for Japanese-Chinese Neural Machine Translation

Movie and TV subtitles are frequently employed in natural language processing (NLP) applications, but there are limited Japanese-Chinese bilingual corpora accessible as a dataset to train neural machine translation (NMT) models. In our previous study, we effectively constructed a corpus of a considerable size containing bilingual text data in both Japanese and Chinese by collecting subtitle text data from websites that host movies and television series. The unsatisfactory translation performance of the initial corpus, Web-Crawled Corpus of Japanese and Chinese (WCC-JC 1.0), was predominantly caused by the limited number of sentence pairs. To address this shortcoming, we thoroughly analyzed the issues associated with the construction of WCC-JC 1.0 and constructed the WCC-JC 2.0 corpus by first collecting subtitle data from movie and TV series websites. Then, we manually aligned a large number of high-quality sentence pairs. Our efforts resulted in a new corpus that includes about 1.4 million sentence pairs, an 87% increase compared with WCC-JC 1.0. As a result, WCC-JC 2.0 is now among the largest publicly available Japanese-Chinese bilingual corpora in the world. To assess the performance of WCC-JC 2.0, we calculated the BLEU scores relative to other comparative corpora and performed manual evaluations of the translation results generated by translation models trained on WCC-JC 2.0. We provide WCC-JC 2.0 as a free download for research purposes only.

Analysis of Translation Techniques and Ideology in Olaf's Frozen Adventure Movie Subtitles

This study aims to identify and analyze translation techniques and translation ideology used by translators in translating subtitle texts in the film Olaf's Frozen adventure. In this study using descriptive qualitative research with data in the form of words, phrases, clauses, sentences containing translation techniques in the subtitle text of the film Olaf's Frozen Adventure. The data collection technique in this study was to use documents and archives, by watching the film Olaf's Frozen Adventure and reading the available subtitles in both the source language and the target language and transcribing. In analyzing the data the steps taken by the researcher are watching and reading the subtitle text of the film Olaf's Frozen Adventure, analyzing translation techniques and translation ideology, and finally drawing conclusions.
 Based on the results of the analysis, there are 12 translation techniques used by translators in translating the subtitles of the film Olaf's Frozen Adventure from 133 data. Based on the frequency of use of these techniques are: reduction technique 17, borrowing 9, literal translation 27, equivalent 24, calculus 14, transposition 11, amplification 5, particulation 2, linguistic compression 17, modulation 5, adaptation 1, and generalization 1. applied by translators in translating the subtitles of the film Olaf's Frozen Adventure is the ideology of domestication.

«Киноатлас СССР»: концепция М. В. Налетного. По архивным материалам 1920-х гг.

The 1920s saw an increased general release of documentaries about life of peoples and regions in the USSR, opening an opportunity for mass audience to make virtual trips through multinational and diverse country. It was within the frameworks of the project “Cinema-Atlas of the USSR,” an ambitious state program launched under the auspices of the Central Executive Committee, which involved creation of a 150-episode thematic film almanac. The article is to introduce into scientific use the archival documentary film “To the shores of the Pacific Ocean” (1927) directed by M. V. Naletny, created during complex film expedition along the Trans-Siberian railway from Moscow to the Far East. To achieve this goal, not only the film itself is analyzed, but also socio-political, cultural, and ideological context of its creation, in line with the state program for creation of the Soviet Cine-Atlas. Little-known visual and text archives, as well as data from the Soviet periodical press of the 1920s, are used as a source base for the study. Due to specifics of silent cinema, this film is a film text of a kind, consisting of approximately equal number of alternating film frames and text captions. One can tell that, in accordance with general recommendations for film making in the USSR, based on Marxist theory, the director designed his full-length film as a series of visual essays suitable for the “Cinema-Atlas” developed at the time. The study makes it obvious that M. V. Naletny sought in his film to overcome the format of superficial “tourist” narrative, widespread at the time, detailing geographical features of regions, types of population, their economic structures, etc. M. V. Naletny’s methodological developments, which are considered cocurrently, clearly demonstrate party requirements to capacious resources of cinematography as a means of information and agitation, textbook, and means for popularization of scientific knowledge. It is concluded that M. V. Naletny’s legacy is of multi-layered popular science significance, having become a contribution to the on-screen chronicle of Soviet transformations on the ground. The studied complex of materials is a significant example of visual and anthropological searches of the early Soviet period, as well as an informative historical source that has not lost its relevance for modern scientific study.

Image-Text Embedding Learning via Visual and Textual Semantic Reasoning.

As a bridge between language and vision domains, cross-modal retrieval between images and texts is a hot research topic in recent years. It remains challenging because the current image representations usually lack semantic concepts in the corresponding sentence captions. To address this issue, we introduce an intuitive and interpretable model to learn a common embedding space for alignments between images and text descriptions. Specifically, our model first incorporates the semantic relationship information into visual and textual features by performing region or word relationship reasoning. Then it utilizes the gate and memory mechanism to perform global semantic reasoning on these relationship-enhanced features, select the discriminative information and gradually grow representations for the whole scene. Through the alignment learning, the learned visual representations capture key objects and semantic concepts of a scene as in the corresponding text caption. Experiments on MS-COCO [1] and Flickr30K [2] datasets validate that our method surpasses many recent state-of-the-arts with a clear margin. In addition to the effectiveness, our methods are also very efficient at the inference stage. Thanks to the effective overall representation learning with visual semantic reasoning, our methods can already achieve very strong performance by only relying on the simple inner-product to obtain similarity scores between images and captions. Experiments validate the proposed methods are more than 30-75 times faster than many recent methods with code public available. Instead of following the recent trend of using complex local matching strategies [3], [4], [5], [6] to pursue good performance while sacrificing efficiency, we show that the simple global matching strategy can still be very effective, efficient and achieve even better performance based on our framework.

An Efficient Deep Learning based Hybrid Model for Image Caption Generation

In the recent yeas, with the increase in the use of different social media platforms, image captioning approach play a major role in automatically describe the whole image into natural language sentence. Image captioning plays a significant role in computer-based society. Image captioning is the process of automatically generating the natural language textual description of the image using artificial intelligence techniques. Computer vision and natural language processing are the key aspect of the image processing system. Convolutional Neural Network (CNN) is a part of computer vision and used object detection and feature extraction and on the other side Natural Language Processing (NLP) techniques help in generating the textual caption of the image. Generating suitable image description by machine is challenging task as it is based upon object detection, location and their semantic relationships in a human understandable language such as English. In this paper our aim to develop an encoder-decoder based hybrid image captioning approach using VGG16, ResNet50 and YOLO. VGG16 and ResNet50 are the pre-trained feature extraction model which are trained on millions of images. YOLO is used for real time object detection. It first extracts the image features using VGG16, ResNet50 and YOLO and concatenate the result in to single file. At last LSTM and BiGRU are used for textual description of the image. Proposed model is evaluated by using BLEU, METEOR and RUGE score.

Effect of video tutorial delivery method on D/HH students' content comprehension.

Using video tutorials to deliver instructional content has become common practices nowadays. However, it is required to investigate how to implement new methods to deliver instructional content to deaf students to ensure success of their learning and reduce their reliance on personnel support or consultation from hearing peers. Therefore, and in light of cognitive theory of multimedia learning, this study experimented three different video tutorial methods to deliver instructional content that are tailored to deaf students' learning needs. The three methods included: (a) sign language only, (b) captioned text only, and (c) sign language and captioned text combined. The study applied a mixed methods research design using pretest-posttest quasi-experimental design (tests scores) and qualitative research design (interviews). Fifty-four undergraduate deaf students from a large university in Saudi Arabia participated in this study, and of those participants, fifteen deaf students participated in semistructured interviews. One-way analysis of variance results showed that using video tutorial that presents declarative content with captions only was significantly effective in comparison with the other methods. While the video tutorial that presents procedural content with sign language only was significantly effective in comparison with the other methods. Interview results confirmed the quantitative results. Practical and theoretical implications are discussed.

Strategy of Audiovisual Translation on Enola Holmes Netflix Movie Subtitle

The article discusses translation strategies of Audiovisual Translation engaged with the movie on Netflix. The research concentrates on translation strategies involved in the AVT subtitle case of an adventure mystery genre in the Enola Holmes Netflix movie to know the relation of translation strategies with AVT concept. This is an audiovisual translation research in a qualitative descriptive, along with data collecting of English and Indonesian subtitles of Enola Holmes Netflix movie. From the findings, it found nine strategies applied in translating the subtitle of Enola Holmes movie. The result reveals strategies found in translating the subtitle text into Indonesian shows the order from the dominant to the least which are condensation (27%), transfer (25%), paraphrase (24%), deletion (13%), expansion (5%), imitation (5%), resignation (1%), decimation (1%), and transcription (0%). Furthermore, it found two situations of the speech acts equivalency in the English into Indonesian subtitle text are equivalence (98%) and not equivalence (2%). Additionally, all of the translation strategies are helpful to produce the efficient subtitles and deliver the original message properly which also does not cross the line of the rules in AVT.

Gaze-assisted automatic captioning of fetal ultrasound videos using three-way multi-modal deep neural networks.

In this work, we present a novel gaze-assisted natural language processing (NLP)-based video captioning model to describe routine second-trimester fetal ultrasound scan videos in a vocabulary of spoken sonography. The primary novelty of our multi-modal approach is that the learned video captioning model is built using a combination of ultrasound video, tracked gaze and textual transcriptions from speech recordings. The textual captions that describe the spatio-temporal scan video content are learnt from sonographer speech recordings. The generation of captions is assisted by sonographer gaze-tracking information reflecting their visual attention while performing live-imaging and interpreting a frozen image. To evaluate the effect of adding, or withholding, different forms of gaze on the video model, we compare spatio-temporal deep networks trained using three multi-modal configurations, namely: (1) a gaze-less neural network with only text and video as input, (2) a neural network additionally using real sonographer gaze in the form of attention maps, and (3) a neural network using automatically-predicted gaze in the form of saliency maps instead. We assess algorithm performance through established general text-based metrics (BLEU, ROUGE-L, F1 score), a domain-specific metric (ARS), and metrics that consider the richness and efficiency of the generated captions with respect to the scan video. Results show that the proposed gaze-assisted models can generate richer and more diverse captions for clinical fetal ultrasound scan videos than those without gaze at the expense of the perceived sentence structure. The results also show that the generated captions are similar to sonographer speech in terms of discussing the visual content and the scanning actions performed.

Image Captioning Generator Text-to-Speech

A model is created for blind people that can guide and support them while traveling on the highways just with the help of a smartphone application. This can be accomplished by first converting the scene in front of the user into text and then converting text into voice output. Then a method for the generation of image legends based on deep neural networks. With an image as an entry, the method can display an English sentence describing the contents of the image. The user first provides a voice command, then a quick snapshot is captured by the camera or webcam. This image is then fed as input to the image caption generator template that generates a caption for the image. Next, this caption text is converted to speech, which gives rise to a voice message on the description of the image.

Multimodal detection of hateful memes by applying a vision-language pre-training model.

Detrimental to individuals and society, online hateful messages have recently become a major social issue. Among them, one new type of hateful message, "hateful meme", has emerged and brought difficulties in traditional deep learning-based detection. Because hateful memes were formatted with both text captions and images to express users' intents, they cannot be accurately identified by singularly analyzing embedded text captions or images. In order to effectively detect a hateful meme, the algorithm must possess strong vision and language fusion capability. In this study, we move closer to this goal by feeding a triplet by stacking the visual features, object tags, and text features of memes generated by the object detection model named Visual features in Vision-Language (VinVl) and the optical character recognition (OCR) technology into a Transformer-based Vision-Language Pre-Training Model (VL-PTM) OSCAR+ to perform the cross-modal learning of memes. After fine-tuning and connecting to a random forest (RF) classifier, our model (OSCAR+RF) achieved an average accuracy and AUROC of 0.684 and 0.768, respectively, on the hateful meme detection task in a public test set, which was higher than the other eleven (11) published baselines. In conclusion, this study has demonstrated that VL-PTMs with the addition of anchor points can improve the performance of deep learning-based detection of hateful memes by involving a more substantial alignment between the text caption and visual information.