Video Text Research Articles

Cross-modal adapter for vision–language retrieval

Vision–language retrieval is an important multi-modal learning topic, where the goal is to retrieve the most relevant visual candidate for a given text query. Recently, pre-trained models, e.g., CLIP, show great potential on retrieval tasks. However, as pre-trained models are scaling up, fully fine-tuning them on donwstream retrieval datasets has a high risk of overfitting. Moreover, in practice, it would be costly to train and store a large model for each task. To overcome the above issues, we present a novel Cross-Modal Adapter for parameter-efficient transfer learning. Inspired by adapter-based methods, we adjust the pre-trained model with a few parameterization layers. However, there are two notable differences. First, our method is designed for the multi-modal domain. Secondly, it allows encoder-level implicit cross-modal interactions between vision and language encoders. Although surprisingly simple, our approach has three notable benefits: (1) reduces the vast majority of fine-tuned parameters, (2) saves training time, and (3) allows all the pre-trained parameters to be fixed, enabling the pre-trained model to be shared across datasets. Extensive experiments demonstrate that, without bells and whistles, our approach outperforms adapter-based methods on image–text retrieval datasets (MSCOCO, Flickr30K) and video–text retrieval datasets (MSR-VTT, DiDeMo, and ActivityNet).

TRRHA: A two-stream re-parameterized refocusing hybrid attention network for synthesized view quality enhancement

In multi-view video systems, the decoded texture video and its corresponding depth video are utilized to synthesize virtual views from different perspectives using the depth-image-based rendering (DIBR) technology in 3D-high efficiency video coding (3D-HEVC). However, the distortion of the compressed multi-view video and the disocclusion problem in DIBR can easily cause obvious holes and cracks in the synthesized views, degrading the visual quality of the synthesized views. To address this problem, a novel two-stream re-parameterized refocusing hybrid attention (TRRHA) network is proposed to significantly improve the quality of synthesized views. Firstly, a global multi-scale residual information stream is applied to extract the global context information by using refocusing attention module (RAM), and the RAM can detect the contextual feature and adaptively learn channel and spatial attention feature to selectively focus on different areas. Secondly, a local feature pyramid attention information stream is used to fully capture complex local texture details by using re-parameterized refocusing attention module (RRAM). The RRAM can effectively capture multi-scale texture details with different receptive fields, and adaptively adjust channel and spatial weights to adapt to information transformation at different sizes and levels. Finally, an efficient feature fusion module is proposed to effectively fuse the extracted global and local information streams. Extensive experimental results show that the proposed TRRHA achieves significantly better performance than the state-of-the-art methods. The source code will be available at https://github.com/647-bei/TRRHA.

A Comprehensive Review on Text Detection and Recognition in Scene Images

Detecting and recognizing text in natural scene images and videos is vital for several real-world applications, such as in the analysis of Crime scene CCTV footage, sports videos, and autonomous driving, to name a few. Therefore, one can expect several challenges, namely, arbitrarily oriented and shaped Text detection and identification in movies and natural environments. Many methods have been developed in the past to address these challenges, including advanced deep-learning models and transformers. Due to several methods available in the literature, it is not so easy to understand the open challenges, applications, directions, scope, limitations, and weaknesses of the methods. Therefore, there is a need to write a survey/review to highlight and discuss the strengths and weaknesses of the developed methods. This survey/review presents different categories of work and discusses their importance, limitations, new challenges, applications, and, finally, directions such that readers can choose appropriate methods and directions to carry out research work in the field of text detection/recognition in the natural scene and videos.

Adaptive Video Text Tracking Based on Pixel-level Feature Extraction

Adaptive Video Text Tracking Based on Pixel-level Feature Extraction

AVaTER: Fusing Audio, Visual, and Textual Modalities Using Cross-Modal Attention for Emotion Recognition.

Multimodal emotion classification (MEC) involves analyzing and identifying human emotions by integrating data from multiple sources, such as audio, video, and text. This approach leverages the complementary strengths of each modality to enhance the accuracy and robustness of emotion recognition systems. However, one significant challenge is effectively integrating these diverse data sources, each with unique characteristics and levels of noise. Additionally, the scarcity of large, annotated multimodal datasets in Bangla limits the training and evaluation of models. In this work, we unveiled a pioneering multimodal Bangla dataset, MAViT-Bangla (Multimodal Audio Video Text Bangla dataset). This dataset, comprising 1002 samples across audio, video, and text modalities, is a unique resource for emotion recognition studies in the Bangla language. It features emotional categories such as anger, fear, joy, and sadness, providing a comprehensive platform for research. Additionally, we developed a framework for audio, video and textual emotion recognition (i.e., AVaTER) that employs a cross-modal attention mechanism among unimodal features. This mechanism fosters the interaction and fusion of features from different modalities, enhancing the model's ability to capture nuanced emotional cues. The effectiveness of this approach was demonstrated by achieving an F1-score of 0.64, a significant improvement over unimodal methods.

Swin transformer-based traffic video text tracking

Swin transformer-based traffic video text tracking

Video text tracking with transformer-based local search

Video text tracking is a highly significant branch in the field of multi-object tracking (MOT), aiming to detect all text instances in video frames and construct trajectories for each text. Conventional text tracking methods typically follow the Tracking-By-Detection (TbD) paradigm which involves two separate steps of detection and association. In complex scenarios, abundant detection omission of text instances is caused by exposure, occlusion, motion blur, and the like, leading to text trajectory breakpoints in the TbD methods. To this end, video text tracking with transformer-based local search (LSTrack) is proposed in this paper. We utilize historical trajectory information to estimate the approximate search areas where omitted texts are located. In this case, our local search tracker leverages the text images in historical trajectories as references to directly recall text instances in the search areas. In addition, our tracking framework can combine the explicit semantic information output from an OCR model to obtain the semantic version (SV) of LSTrack, which for the first time uses the text edit distance as the distance measurement in matching stages to achieve better text tracking results. Ultimately, our method achieves an advanced performance on several public benchmarks. In particular, LSTrack (SV) achieves state-of-the-art performance on the Minetto benchmark.

CLIP2TF:Multimodal video–text retrieval for adolescent education

With the rapid advancement of artificial intelligence technology, particularly within the sphere of adolescent education, a continual emergence of new challenges and opportunities is observed. The current educational system increasingly requires the automation of teaching activities detection and evaluation, offering fresh perspectives for enhancing the quality of adolescent education. Although large-scale models are receiving significant attention in educational research, their high demand for computational resources and limitations in specific applications constrain their widespread use in analyzing educational video content, especially when handling multimodal data. Current multimodal contrastive learning methods, which integrate video, audio, and text information, have achieved certain successes in video–text retrieval tasks. However, these methods typically employ simpler weighted fusion strategies and fail to avoid noise and information redundancy. Therefore, our study proposes a novel network framework, CLIP2TF, which includes an efficient audio–visual fusion encoder. It aims to dynamically interact and integrate visual and audio features, further enhancing the visual features that may be missing or insufficient in specific teaching scenarios while effectively reducing redundant information transfer during the modality fusion process. Through ablation experiments on the MSRVTT and MSVD datasets, we first demonstrate the effectiveness of CLIP2TF in video–text retrieval tasks. Subsequent tests on teaching video datasets further proves the applicability of the proposed method. This research not only showcases the potential of artificial intelligence in the automated assessment of teaching quality but also provides new directions for research in related fields studies.

End-to-End Video Text Spotting with Transformer

End-to-End Video Text Spotting with Transformer

Analysis of Common Verbal Errors in Chinese Video Texts

Analysis of Common Verbal Errors in Chinese Video Texts

A Comparison of ChatGPT and Human Questionnaire Evaluations of the Urological Cancer Videos Most Watched on YouTube

AimTo examine the reliability of ChatGPT in evaluating the quality of medical content of the most watched videos related to urological cancers on YouTube. Material and methodsIn March 2024 a playlist was created of the first 20 videos watched on YouTube for each type of urological cancer. The video texts were evaluated by ChatGPT and by a urology specialist using the DISCERN-5 and Global Quality Scale (GQS) questionnaires. The results obtained were compared using the Kruskal-Wallis test. ResultsFor the prostate, bladder, renal, and testicular cancer videos, the median (IQR) DISCERN-5 scores given by the human evaluator and ChatGPT were (Human: 4 [1], 3 [0], 3 [2], 3 [1], P = .11; ChatGPT: 3 [1.75], 3 [1], 3 [2], 3 [0], P = .4, respectively) and the GQS scores were (Human: 4 [1.75], 3 [0.75], 3.5 [2], 3.5 [1], P = .12; ChatGPT: 4 [1], 3 [0.75], 3 [1], 3.5 [1], P = .1, respectively), with no significant difference determined between the scores. The repeatability of the ChatGPT responses was determined to be similar at 25 % for prostate cancer, 30 % for bladder cancer, 30 % for renal cancer, and 35 % for testicular cancer (P = .92). No statistically significant difference was determined between the median (IQR) DISCERN-5 and GQS scores given by humans and ChatGPT for the content of videos about prostate, bladder, renal, and testicular cancer (P > .05). ConclusionAlthough ChatGPT is successful in evaluating the medical quality of video texts, the results should be evaluated with caution as the repeatability of the results is low.

Fame-Led to Sympathy: Content Analysis of Felicya Angelista’s Statement Related to the Israeli-Palestinian Conflict Using Roland Barthes Semiotic Approach

The video of Felicya Angelista discussing the Israeli-Palestinian conflict sparked controversy, with criticism directed at her for seemingly supporting Israel. The use of the word "war" without context and footage of Hamas attacking Israeli citizens contributed to this backlash, leading to calls to boycott Scarlett products. This research, utilizing Roland Barthes’ Semiotic Theory, aimed to analyze the denotation, connotation, and myths within the video's text, assessing its impact on Felicya's reputation and Scarlett products. Qualitative content analysis was conducted using data from Felicya's Instagram videos, netizen comments, and online news reports. The findings revealed denotatively that Felicya is crying and wiping her tears, Israeli civilians attacked by Hamas, and Felicya stood on humanity and peace. The denotational meanings were Felicya expressed sadness for the victims especially parents and children, and Felicya appearing neutral. However, connotatively, she was perceived to support Israel. Myths surrounding the expression of sadness, artists' involvement in the conflict, and the conflict's religious nature were also identified. The video faced criticism from netizens, leading to calls for boycotting Scarlett products due to Felicya's perceived alignment with Israel. In summary, while the video portrayed Felicya's neutrality denotatively, it connotatively suggested support for Israel, contributing to backlash and calls for boycotts.

Developing a Feature Set from Scene and Texture Features for Detecting Neural Texture Videos Using Boosted Decision Trees

Developing a Feature Set from Scene and Texture Features for Detecting Neural Texture Videos Using Boosted Decision Trees

Video text rediscovery: Predicting and tracking text across complex scenes

AbstractDynamic texts in scene videos provide valuable insights and semantic cues crucial for video applications. However, the movement of this text presents unique challenges, such as blur, shifts, and blockages. While efficient in tracking text, state‐of‐the‐art systems often need help when text becomes obscured or complicated scenes. This study introduces a novel method for detecting and tracking video text, specifically designed to predict the location of obscured or occluded text in subsequent frames using a tracking‐by‐detection paradigm. Our approach begins with a primary detector to identify text within individual frames, thus enhancing tracking accuracy. Using the Kalman filter, Munkres algorithm, and deep visual features, we establish connections between text instances across frames. Our technique works on the concept that when text goes missing in a frame due to obstructions, we use its previous speed and location to predict its next position. Experiments conducted on the ICDAR2013 Video and ICDAR2015 Video datasets confirm our method's efficacy, matching or surpassing established methods in performance.

An empirical study of excitation and aggregation design adaptions in CLIP4Clip for video–text retrieval

CLIP4Clip model transferred from the CLIP has been the de-factor standard to solve the video clip retrieval task from frame-level input, triggering the surge of CLIP4Clip-based models in the video–text retrieval domain. In this work, we rethink the inherent limitation of widely-used mean pooling operation in the frame features aggregation and investigate the adaptions of excitation and aggregation design for discriminative video representation generation. We present a novel excitation-and-aggregation design, including (1) The excitation module is available for capturing non-mutually-exclusive relationships among frame features and achieving frame-wise features recalibration, and (2) The aggregation module is applied to learn exclusiveness used for frame representations aggregation. Similarly, we employ the cascade of sequential module and aggregation design to generate discriminative video representation in the sequential type. Besides, we adopt the excitation design in the tight type to obtain representative frame features for multi-modal interaction. The proposed modules are evaluated on three benchmark datasets of MSR-VTT, ActivityNet and DiDeMo, achieving MSR-VTT (43.9 R@1), ActivityNet (44.1 R@1) and DiDeMo (31.0 R@1). They outperform the CLIP4Clip results by +1.2% (+0.5%), +4.5% (+1.9%) and +9.5% (+2.7%) relative (absolute) improvements, demonstrating the superiority of our proposed excitation and aggregation designs. We hope our work will serve as an alternative for frame representations aggregation and facilitate future research.

FaceFolds: Meshed Radiance Manifolds for Efficient Volumetric Rendering of Dynamic Faces

3D rendering of dynamic face captures is a challenging problem, and it demands improvements on several fronts---photorealism, efficiency, compatibility, and configurability. We present a novel representation that enables high-quality volumetric rendering of an actor's dynamic facial performances with minimal compute and memory footprint. It runs natively on commodity graphics soft- and hardware, and allows for a graceful trade-off between quality and efficiency. Our method utilizes recent advances in neural rendering, particularly learning discrete radiance manifolds to sparsely sample the scene to model volumetric effects. We achieve efficient modeling by learning a single set of manifolds for the entire dynamic sequence, while implicitly modeling appearance changes as temporal canonical texture. We export a single layered mesh and view-independent RGBA texture video that is compatible with legacy graphics renderers without additional ML integration. We demonstrate our method by rendering dynamic face captures of real actors in a game engine, at comparable photorealism to state-of-the-art neural rendering techniques at previously unseen frame rates.

교양교육에서 AI 창작품을 활용한 융복합 시 교육 실제

This study aims to explore and argue the significance of using AI and new me dia to promote new reading and creative expression among liberal arts education learners. After exposing liberal arts education learners to the AI poetry book “Why Write Poetry” and the AI poetry drama “Paphos,” this study was designed to create “video poems” using new media based on the above experiences. This is because it is assumed that through AI creations, liberal arts education learners will approach works using AI with interest, and furthermore, they will continue to ‘write’ their thoughts and imagination as an opportunity to explore ‘me’ by analyzing the mean ing and expression inherent in AI creations. More specifically, this study asked learners who read AI creations to create poetry texts based on their impressions and to write and revise video poetry plans using new media. We chose video poetry production using new media for the following reasons. The learners were exposed to A.I. poetry and poetry theater, and understood the im ages in poetic texts and stage performances. However, it is not possible to produce enough poetry texts and poetry plays for learners to gain educational efficacy im mediately based on the contents of the class. Therefore, this study introduced a new creative method that incorporates video. Since learners have had many opportuni ties to encounter video texts through social media and online, the teaching method of creating video poems using new media is a new writing method designed in line with the modern IT era that allows learners to take a break from traditional literary writing and gain a sense of accomplishment. To summarize the results of this class In this study, liberal arts education learn-ers were asked to write an essay through the production and presentation of video poems using A.I. creations, and analyzing their essays, many changes occurred, unlike before when they were not interested in poetry or poetry plays. Furthermore, the learners were able to share the video poems with their fellow learners’ inner concerns, and sincerely acquire interest and enjoyment in the medium and genre of literature. Ultimately, through the above liberal arts education, learners were able to break away from their self-centered perspective, understand the feelings of others, and develop a mutual respectful attitude through other people’s writings and videos. The educational process designed in this study is a method designed to empa thize and communicate with each other’s inner ‘subjectr’ and ‘valuesr’ despite the fact that the learners are from various majors. In particular, what is expected from this study is that it can help the development of liberal arts education by realizing creative, fusion and convergent educational theories and analyses in response to the future society that has already arrived, and realizing them in university education classes.

Robust and efficient airplane cockpit video coding leveraging temporal redundancy

AbstractAirplane cockpit screens consist of virtual instruments where characters, numbers, and graphics are overlaid on a black or natural background. Recording the cockpit screen allows one to log vital plane data, as aircraft manufacturers do not offer direct access to raw data. However, traditional video codecs struggle at preserving character readability at the required low bit-rates. We showed in a previous work that large rate-distortion gains can be achieved if the characters are encoded as text rather than as pixels. We now leverage temporal redundancy to both achieve robust character recognition and improve encoding efficiency. A convolutional neural network is trained for character classification over synthetic samples augmented with occlusions to gain robustness against overlapping graphics. Further robustness to background occlusions is brought by a probabilistic framework that error-corrects the output of the convolutional neural network. Next, we propose a predictive text coding technique specifically tailored for text in cockpit videos that achieves competitive performance over commodity lossless methods. Experiments with real cockpit video footage show large rate-distortion gains for the proposed method with respect to three different video compression standards. Notably, the H.264/AVC codec retrofitted with our method outperforms H.265/HEVC-SCC and is competitive with the much more complex H.266/VVC while preserving text and graphics. The entire pipeline described in this work has been implemented at Safran Electronics as an embedded avionics system drawing just 2W of power thanks to a combination of software and FPGA implementation.

The Value of Folk Belief in Chinese Films

Folk belief is closely related to people's life, and it is the value crystallization of the agglomeration and evolution of human society and culture. Folk beliefs are also presented in literary and artistic works, and diversified social and cultural values are constructed together with the texts, showing the cultural pattern based on folk beliefs. In Chinese films, the folk custom also constructs the special value and cultural connotation of the film art based on the video text, and the folk belief in the film art echoes each other, establishing the dual value of the film art and the folk belief.

Video–text retrieval via multi-modal masked transformer and adaptive attribute-aware graph convolutional network

Video–text retrieval via multi-modal masked transformer and adaptive attribute-aware graph convolutional network