Video Clip Retrieval Research Articles

An empirical study of excitation and aggregation design adaptions in CLIP4Clip for video–text retrieval

CLIP4Clip model transferred from the CLIP has been the de-factor standard to solve the video clip retrieval task from frame-level input, triggering the surge of CLIP4Clip-based models in the video–text retrieval domain. In this work, we rethink the inherent limitation of widely-used mean pooling operation in the frame features aggregation and investigate the adaptions of excitation and aggregation design for discriminative video representation generation. We present a novel excitation-and-aggregation design, including (1) The excitation module is available for capturing non-mutually-exclusive relationships among frame features and achieving frame-wise features recalibration, and (2) The aggregation module is applied to learn exclusiveness used for frame representations aggregation. Similarly, we employ the cascade of sequential module and aggregation design to generate discriminative video representation in the sequential type. Besides, we adopt the excitation design in the tight type to obtain representative frame features for multi-modal interaction. The proposed modules are evaluated on three benchmark datasets of MSR-VTT, ActivityNet and DiDeMo, achieving MSR-VTT (43.9 R@1), ActivityNet (44.1 R@1) and DiDeMo (31.0 R@1). They outperform the CLIP4Clip results by +1.2% (+0.5%), +4.5% (+1.9%) and +9.5% (+2.7%) relative (absolute) improvements, demonstrating the superiority of our proposed excitation and aggregation designs. We hope our work will serve as an alternative for frame representations aggregation and facilitate future research.

Basketball Object Extraction Method Based on Image Segmentation Algorithm

Finding your favorite videos from massive sports video data has become a big demand for users, accurate sports videos can better help people learn sports content, and the traditional data management and retrieval methods using text identifiers are difficult to meet the needs of users, so the research on the extraction of sports objects in sports videos is of great significance. This paper mainly studies and proposes the basketball object extraction method based on image segmentation algorithm and can accurately analyze the trajectory of the basketball target. By modeling the video frame of basketball game, the basketball object is selected for segmentation and extraction. The extracted basketball object can be used for tracking the target in the basketball video clip retrieval system. At the same time, the segmentation and extraction of the basketball object are also the core part in the basketball video clip retrieval framework. Combined with the characteristics of basketball video images in the database, the algorithm extracts the image block variance and contrast to form the training feature vector, and the correct segmentation rate on the database is higher than 95.2%. The results show that this method has a good effect on the segmentation and extraction of basketball objects in basketball videos.

CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioning

Video clip retrieval and captioning tasks play an essential role in multimodal research and are the fundamental research problem for multimodal understanding and generation. The CLIP (Contrastive Language-Image Pre-training) model has demonstrated the power of visual concepts learning from web collected image-text datasets. In this paper, we propose a CLIP4Clip model to transfer the knowledge of the image-text pretrained CLIP model to video-text tasks in an end-to-end manner. Furthermore, we conduct several empirical studies including 1) Whether image feature is enough for video-text retrieval and captioning? 2) How a post-pretraining on a large-scale video-text dataset based on the CLIP affect the performance? 3) What is the practical mechanism to model temporal dependency between video frames? And 4) The Hyper-parameters sensitivity of the model. Extensive experimental results present that the CLIP4Clip model transferred from the CLIP can achieve SOTA results on various video-text datasets, including MSR-VTT, MSVD, LSMDC, and DiDeMo for multimodal understanding and generation tasks.

A multi-embedding neural model for incident video retrieval

Many internet search engines have been developed, however, the retrieval of video clips remains a challenge. This paper considers the retrieval of incident videos, which may contain more spatial and temporal semantics. We propose an encoder-decoder ConvLSTM model that explores multiple embeddings of a video to facilitate comparison of similarity between a pair of videos. The model is able to encode a video into an embedding that integrates both its spatial information and temporal semantics. Multiple video embeddings are then generated from coarse- and fine-grained features of a video to capture high- and low-level meanings. Subsequently, a learning-based comparative model is proposed to compare the similarity of two videos based on their embeddings. Extensive evaluations are presented and show that our model outperforms state-of-the-art methods for several video retrieval tasks on the FIVR-200K, CC_WEB_VIDEO, and EVVE datasets.

Key Frame Extraction Method of Music and Dance Video Based on Multicore Learning Feature Fusion

The purpose of video key frame extraction is to use as few video frames as possible to represent as much video content as possible, reduce redundant video frames, and reduce the amount of computation, so as to facilitate quick browsing, content summarization, indexing, and retrieval of videos. In this paper, a method of dance motion recognition and video key frame extraction based on multifeature fusion is designed to learn the complicated and changeable dancer motion recognition. Firstly, multiple features are fused, and then the similarity is measured. Then, the video sequences are clustered by the clustering algorithm according to the scene. Finally, the key frames are extracted according to the minimum amount of motion. Through the quantitative analysis and research of the simulation results of different models, it can be seen that the model proposed in this paper can show high performance and stability. The breakthrough of video clip retrieval technology is bound to effectively promote the inheritance and development of dance, which is of great theoretical significance and practical value.

Using semantic annotations on political debate videos for building open government based lawmaking

AbstractThe traditional concept and practice of lawmaking is being transformed by the open government paradigm. It involves a new digital model that links three principles: transparency, participation and collaboration. However, the information system‐enabled solutions in this context are still in their infancy, which none available that address the three crucial challenges: (1) providing the functionalities for semantic search and retrieval of video clips on political debates about legislative proposals, (2) support of open participation and collaboration with contextual information in lawmaking processes and, (3) automatic generation of outcomes achieved in the lawmaking lifecycle. In this article, we describe the underlying methods and techniques leading to address these challenges when a semantic‐driven approach is applied for the development and validation of the real‐world scenario of open government‐based lawmaking systems. To evaluate the value created by using a semantic‐driven approach as computational backbone, the development and testing of a software platform as case study was carried out together with experts from a parliamentary institution and analysed the user experience in parliamentary settings with end‐users. The results show how the use of the appropriate techniques for the adoption of a semantic assumption is a key enabler for the management and retrieval of information tasks relating to the lawmaking lifecycle providing a transparent, participatory and collaborative approach in a holistic way.

Enhanced reinstatement of naturalistic event memories due to hippocampal-network-targeted stimulation

Episodic memory involves the reinstatement of distributed patterns of brain activity present when events were initially experienced. The hippocampus is thought to coordinate reinstatement via its interactions with a network of brain regions, but this hypothesis has not been causally tested in humans. The current study directly tested the involvement of the hippocampal network in reinstatement using network-targeted noninvasive stimulation. We measured reinstatement of multi-voxel patterns of functional magnetic resonance imaging (fMRI) activity during encoding and retrieval of naturalistic video clips depicting everyday activities. Reinstatement of video-specific activity patterns was robust in posterior parietal and occipital areas previously implicated in event reinstatement. Theta-burst stimulation targeting the hippocampal network increased video-specific reinstatement of fMRI activity patterns in occipital cortex and improved memory accuracy relative to stimulation of a control out-of-network location. Furthermore, stimulation targeting the hippocampal network influenced the trial-by-trial relationship between hippocampal activity during encoding and later reinstatement in occipital cortex. These findings implicate the hippocampal network in the reinstatement of spatially distributed patterns of event-specific activity and identify a role for the hippocampus in encoding complex naturalistic events that later undergo cortical reinstatement.

Enhanced Reinstatement of Naturalistic Event Memories Due to Hippocampal-Network-Targeted Stimulation

Episodic memory involves the reinstatement of distributed patterns of brain activity present when events were initially experienced. The hippocampus is thought to coordinate reinstatement via its interactions with a network of brain regions, but this hypothesis has not been causally tested in humans. The current study directly tested the involvement of the hippocampal network in reinstatement using network-targeted noninvasive stimulation. We measured reinstatement of multi-voxel patterns of fMRI activity during encoding and retrieval of naturalistic video clips depicting everyday activities. Reinstatement of video-specific activity patterns was robust in posterior-parietal and occipital areas previously implicated in event reinstatement. Theta-burst stimulation targeting the hippocampal network increased video-specific reinstatement of fMRI activity patterns in occipital cortex and improved memory accuracy relative to stimulation of a control out-of-network location. Furthermore, stimulation targeting the hippocampal network influenced the trial-by-trial relationship between hippocampal activity during encoding and later reinstatement in occipital cortex. These findings implicate the hippocampal network in the reinstatement of spatially distributed patterns of event-specific activity, and identify a role for the hippocampus in encoding complex naturalistic events that later undergo cortical reinstatement.

A semantic parliamentary multimedia approach for retrieval of video clips with content understanding

Digital videos of parliamentary activity play an important role in enhancing transparency and accountability for open e-government. The rapid growth in these videos and the lack of semantic annotations and relationships between video and knowledge resources make it increasingly difficult to find accurate video clips with contextual information for content understanding. To overcome this problem, we highlight the need for building multimedia systems based on a semantic vision. With this aim, we focus on (1) how to address the knowledge representation for automatic extraction of contextual information for video content understanding; (2) how to link the parliamentary knowledge structures within video resources to provide accurate video clips retrieval; and (3) how to perform semantic annotation on video resources. The methodology applied is focused on a systematic approach that uses techniques from ontology engineering. This approach is based on the definition of two models: the semantic model and the reference architecture. The semantic model is composed of a reference ontology and a semantic video annotation framework. The ontology provides the support for video content understanding and the semantic vocabulary for annotating video resources. The video annotation framework is based on an RDF-powered semantic video annotation to effectively relate low- and mid-level visual features, corresponding to speakers’ interventions, to high-level parliamentary concepts. To evaluate the proposed system, a prototype for the Canary Islands Parliament (Spain) has been carried out. The results show how semantic enhancement is a key enabler for improved video retrieval on parliamentary multimedia content.

Football Extraction Method Based on Image Segmentation Algorithm

This paper proposes a football object extraction method based on image segmentation algorithm in football video. The football segmentation Interactive operation method with the advantages of high accuracy segmentation, can be applied to other ball games Object (such as volleyball, soccer) segmentation and extraction. In the next step of the study, in a Step to improve the extraction method of the football video frames in image segmentation accuracy ,Application of football and the extracted from football video segment retrieval system for target tracking, to provide an effective method for basketball video clip retrieval.

Summarizing high-level scene behavior

We present several novel techniques to summarize the high-level behavior in surveillance video. Our proposed methods can employ either optical flow or trajectories as input, and incorporate spatial and temporal information together, which improve upon existing approaches for summarization. To begin, we extract common pathway regions by performing graph-based clustering on similarity matrices describing the relationships between location/orientation states. We then employ the activities along the pathway regions to extract the aggregate behavioral patterns throughout scenes. We show how our summarization methods can be applied to detect anomalies, retrieve video clips of interest, and generate adaptive-speed summary videos. We examine our approaches on multiple complex urban scenes and present experimental results.

K-Partite graph reinforcement and its application in multimedia information retrieval

In many example-based information retrieval tasks, example query actually contains multiple sub-queries. For example, in 3D object retrieval, the query is an object described by multiple views. In content-based video retrieval, the query is a video clip that contains multiple frames. Without prior knowledge, the most intuitive approach is to treat the sub-queries equally without difference. In this paper, we propose a k-partite graph reinforcement approach to fuse these sub-queries based on the to-be-retrieved database. The approach first collects the top retrieved results. These results are regarded as pseudo-relevant samples and then a k-partite graph reinforcement is performed on these samples and the query. In the reinforcement process, the weights of the sub-queries are updated by an iterative process. We present experiments on 3D object retrieval and content-based video clip retrieval, and the results demonstrate that our method effectively boosts retrieval performance.

Influences of users' familiarity with visual search topics on interactive video digital libraries

AbstractUser‐centered analysis can benefit the development of interactive video digital libraries. Findings from this study support the idea that having additional understanding about the intended users of video digital libraries can help researchers match system designs with the envisioned use of prototype systems. This study examines one user‐centered factor specifically, familiarity with visual search topics, to explore if and how this may be associated with other factors within an interactive video retrieval context. Twenty‐eight users from the field of science education were recruited to complete six visual search topics using a prototype system to retrieve video clips from a collection of NASA Science Education Programs. Analysis revealed that topic familiarity was associated with other factors that were examined throughout this study, including user‐assessed and experimenter‐assessed topic completion ratios, opinions of the prototype system, and interaction behaviors. Such results can have a variety of implications for developing video digital libraries, especially those designed to support queries and interactions of knowledgeable users from a defined domain.

Efficient video retrieval using index structure

AbstractVideo retrieval remains a challenging problem since most of traditional query algorithms are ineffectual and time‐consuming. In this article, we proposed a new video retrieval method, which segments the video stream by visual similarity between neighboring frames, and adopt the high‐dimensional index structure to organize segments. Furthermore, a new similarity measure is brought forward to improve the query accuracy by synthetically taking into account the visual similarity and temporal order among video segments. Based on the similarity measure, we propose a novel video clip retrieval algorithm which achieves high query efficiency by using restricted sliding window to construct candidate video clips. Experimental results show that the proposed video retrieval method is efficient and effective. © 2008 Wiley Periodicals, Inc. Int J Imaging Syst Technol, 18, 113–123, 2008

The description and retrieval of a sequence of moving objects using a shape variation map

Motion information in a video clip plays an important role in characterizing the content of the clip. In this paper, we propose the shape variation descriptor to describe and retrieve video clips based on the shape variation caused by object movement along time. Similar to accumulator cells, a shape variation map was first constructed by cumulating binary shape images in the sequence of the video clip. A shape descriptor called ART, obtained from the map, then becomes a feature set for the shape variation of the clip. Experimental results over hundreds of video and animation clips show that our approach of retrieving video clips based on shape variation compares favorably against an alternative edge based method.

Developing an object‐oriented framework for content‐based image retrieval

AbstractContent‐based image retrieval (CBIR) is a process of retrieving images from an image database by exploiting the content of the images (typically the querying of an image). CBIR avoids many problems associated with traditional ways of retrieving images by keywords. Thus, a growing interest in the area of CBIR has been established in recent years. In this paper, a novel object‐oriented framework (CBIRFrame) is built for CBIR applications development. We discuss the motivations for CBIRFrame before discussing its design in detail. Two applications of CBIRFrame are also briefly discussed to show the effectiveness of applying CBIRFrame to real applications. Finally, we outline the possible uses of the design of CBIRFrame for other types of domains, such as content‐based retrieval of video clips. Copyright © 2003 John Wiley & Sons, Ltd.

Shape-sequence-based key image generation algorithm for browsing and retrieval of video clips

A new shape-sequence-based algorithm that can effectively generate key images from video clips is introduced. Generated key images can be used as the feature information in the browsing and retrieval of video clips from a multimedia database. Experiments with MPEG-7 data sets were performed, and the results are compared with existing methods.